The move would require renegotiating the UK's Brexit deal and could risk stripping British expats living in the EU of similar rights.