A Government inspector rejected suggestions it was unsuitable for the townscape