You're not alone John, but that's why zoning laws and city ordinances were established. If your scenario were true everywhere, a farmer could buy up land next to a beautiful custom home development and put in a hog farm-stinking up the entire place and ruining property values across the board. For that reason, I'm glad there are some measures in place. The real problem comes when the hog farmer of three decades gets shut down because some developer builds the custom home subdivision next to him and the incoming residents complain about the smell/property values. That's pretty much what Pomona is dealing with. Most of the most vocal opponents have lived there less than ten years.
I guess that would depend how it was zoned to start with. Build a custom home next farm land you better like the smell.