You can only see earthworks of the former castle today