Locals have described plans to build a pair of homes near Dereham as "an accident waiting to happen".