Let’s not trivialize slavery by saying that everyone else was doing it.

As far as the deeper cuts go, it was about overt racism as well. Slavery in the US was justified by saying that the slaves were less than people (or even human) because they weren’t white. Slavery may have been abolished, but that justification is still strong if you go to the wrong parts of the US today.