Actually, there have been large scale attacks made on the residents, on multiple occasions, though it has been some time since one occurred. Yet that is no reason to be unwary.

People's disappearances are another matter. They return home. I believe that is a result of the Gardens' inherent instability, not the Queen's doing.