One councillor argued "wool is being pulled over our eyes" on affordable housing