Norwich Muslim Centre claims it has "come across only three people" who used to regularly visit a former pub which it wants to turn into a mosque.