Racial prejudice worked to perpetuate American slavery, even if it was not essential to sustain the institution. Slavery, serfdom, and peonage had existed elsewhere without racial connotations. Indeed, bondage had been so historically ubiquitous one might well ask why, by the 1760’s, it had come to trouble so many white Americans so much. The answer lies in part — and this part help explain why people like Mason did not act more aggressively on their concerns — in the reservations many whites felt about living alongside members of a supposedly inferior race, whether slave or free. The problem was inherent in American slavery, and emancipation, by undermining white control, would only make it worse.
Alexander, William (Lord Stirling)
George Mason: Forgotten Founder (2006)