It comes after he received criticism last week when, during a visit to Belfast, he told media that a referendum on a united Ireland was “off the table”