Peter Cardwell: Those in Birmingham were convinced. The wider British public is not