"A family in our neighborhood recently moved overseas and their house is listed with a real estate/rental agent. The family left house keys with several friends who have since entered the empty house for various reasons, ranging from picking up appliances to letting in a carpet cleaner. Now one of these “friends” wants to pass a key to me, saying the owners wanted me to have it. I refused it on the grounds that I have no business with a key to an empty house that isn’t mine. Isn’t the Realtor the only one with documented authority to enter the property? With all these keys in different hands, I smell big potential for trouble here."
There’s no reason for you to have a key if you don’t want it, but stop trying to control what your neighbors do with their property. Truth is, the owners can give keys to whomever they want and their Realtor is simply the person they’ve hired to rent the house. If you snoop and see anything illegal, call the police.

