'I could not give a **** who you vote for' said the owner after he reiterated the ban was not political