From The Number’s Game: the commonsense guide to understanding numbers in the news, in politics and life, by Michael Blastland and Andrew Dilnot; on wealth and averages:

A Dutch economist, Jan Pen, famously imagined a procession of the worlds population where people were as tall as they were rich, everyone’s height proportional to their wealth (note wealth, not income). A person of average wealth would be of average height. The procession starts with the poorest (and shortest) person first and ends, one hour later, with the richest (and tallest). Not until twenty minutes into the procession do we see anyone at all. So far, they’ve had either negative net worth (owing more than they own) or no wealth at all, and so have no height. It’s a full thirty minutes before we begin to see dwarfs about six inches tall. And the dwarfs keep coming. It is not until forty-eight minutes have passed that we see the first person of average height and average wealth, when more than three quarters of the world’s population has already gone by.

What delays the average so long after the majority have passed? The answer lies in the effect of those who come next. “In the last few minutes,” wrote Pen, “giants loom up… a lawyer, not exceptionally succesful, eighteen feet tall.” As the hour approaches, the very last people in the procession are so tall we can’t see their heads. Last of all, sid Pen (at a time before the fully formed fortunes of Bill Gates and Warren Buffett), we se John Paul Getty. His height is breathtaking, perhaps ten miles, perhaps twice as much.

One millionaire can shift the average more than many hundreds of poor people, one billionaire a thousand times more. They have not this effect to the extent that 80 percent of the world’s population has less than average.

In everyday speech, “average” is a word meaning low or disdained. With incomes, the average is high. The colloquial use, being blunt, thoughtless, and bordering on a term of abuse, distorts the statistical one, which might, according to the distribution, be high, or low, or in the middle, or altogether irrelevant. It is worth knowing which. If only one thought survives about averages, let it be that they are not necessarily anywhere near the middle, nor representative of what’s typical, and that places often called “the middle” by politicians or the media may be far removed. These ideas have been lazily hitched together for too long. It is time for a divorce.

A more narrative explanation is available from The Atlantic.