The listed college building is said to have "gradually deteriorated" due to age and weathering