Neighbours in Swindon have raised concerns about plans to replace a bungalow and its garden with new homes.