This aligns to the behaviour of OneToOneField and ForeignKey fields. Thanks Jonny Park for the initial patch.