Exclusive: The Prime Minister branded ‘ludicrous’ the claim from ex-Home Secretary Mr Cleverly that he is anti-London