Over at the Atlantic, an interview by Conor Friedersdorf with a Nigerian writer. From it I learned learned about the Nigerian caliphate, and its involvement in slavery. If you can trust wikipedia, it may have been the second largest slave society (after US) at any time, though I don't believe it. Brazil likely is similar in size and the Roman empire may well have had more slaves than the US. Other empires likely also had numbers in the millions. The problem though is we're likely comparing apples and oranges, since the rules about slavery were different in different societies.
Nevertheless, an interesting take.