The elimination of the 'golden visa' would only affect 1%