I think Adam Smith would have a lot to say about this. Specifically, he would probably point out that the slave states really had an awfully small economy compared to the free states, and that most of the wealth generation which occurred in the US occurred due to productivity gains driven by technological innovations which were most aggressively exploited in the north. In the long run, few people could claim to have really benefitted noticeably from american slavery - it was just a shitty thing to do for no reason.

Some historians appear to agree with additional support. During the 18th century, the slave economy was significant in the expansion of industry & commerce, but its value declined & was no longer needed by the 19th century. Slave-intensive sugar production that dominated the 18th century became less important in the 19th century as shipment of cotton products to international markets grew in significance. Unlike sugar, cotton had less need for slaves, and early cotton growers used slaves primarily because they were already slave-owners.

Insurgent scholars known as New Historians of Capitalism argue that slavery, specifically slave-grown cotton, was critical for the rise of the U.S. economy in the 19th century. In contrast, I argued that although industrial capitalism needed cheap cotton, cheap cotton did not need slavery. Unlike sugar, cotton required no large investments of fixed capital and could be cultivated efficiently at any scale, in locations that would have been settled by free farmers in the absence of slavery. Early mainland cotton growers deployed slave labour not because of its productivity or aptness for the new crop, but because they were already slave owners, searching for profitable alternatives to tobacco, indigo, and other declining crops. Slavery was, in effect, a ‘pre-existing condition’ for the 19th-century American South.

Slavery restrained economic development of the south, causing it to underperform economically: while it unevenly concentrated the lesser wealth produced there, the lesser wealth produced there benefitted the rest of the economy less than it could have. Free states didn’t benefit from the less wealth concentrated elsewhere.

To be sure, U.S. cotton did indeed rise ‘on the backs of slaves’, and no cliometric counterfactual can gainsay this brute fact of history. But it is doubtful that this brutal system served the long-run interests of textile producers in Lancashire and New England, as many of them recognized at the time. As argued here, the slave South underperformed as a world cotton supplier, for three distinct though related reasons: in 1807 the region closed the African slave trade, yet failed to recruit free migrants, making labour supply inelastic; slave owners neglected transportation infrastructure, leaving large sections of potential cotton land on the margins of commercial agriculture; and because of the fixed-cost character of slavery, even large plantations aimed at self-sufficiency in foodstuffs, limiting the region’s overall degree of market specialization. The best evidence that slavery was not essential for cotton supply is demonstrated by what happened when slavery ended. After war and emancipation, merchants and railroads flooded into the southeast, enticing previously isolated farm areas into the cotton economy. Production in plantation areas gradually recovered, but the biggest source of new cotton came from white farmers in the Piedmont. When the dust settled in the 1880s, India, Egypt, and slave-using Brazil had retreated from world markets, and the price of cotton in Liverpool returned to its antebellum level. See Figure 2.