The right of consenting individuals to be left in peace by the government is the heart of freedom. It doesn’t matter if the individuals are white or black, men or women, Christian or atheist; consenting adults of all stripes have the right to engage in consensual capitalist acts.
The case for open borders begins with a follow-up question: If race, gender, and religion don’t matter here, why should nationality? Suppose I want to hire a Chinese citizen to work in my factory, and he wants to work in my factory. Or suppose I want to rent my apartment to a Romanian citizen, and she wants to accept my offer. It seems like government should leave us alone, too. If it did, open borders – a world where every non-criminal is free to live and work in any country on earth – would result.