Darren Summers has said he and his family have thoroughly enjoyed their summer at the seafront mansion but would like to downsize