The better analogy is that people live in houses and houses have addresses, and I can use an address to find someone's house.
Whether the pointed data by the pointer is valid or not is... not the point. In all languages I can think of, dereferencing an invalid pointer like a pointer to the wrong address per the type and alignment is never valid. Your analogy does not improve on historical analogies and it is wrong.