Long Tails, Black Swans, and 3-Day Tennis Matches

I've read a fair bit about the subject.  As a fast summary, the bell curve distribution of events is what we think of as "normal".  But in reality some distributions have a very long tail, the graph extends very far to the right.  This is what Mr. Talibi calls a "black swan" event.  And we've just had an instance of it in the 3-day tennis match at Wimbleton.  It's much, much longer than any previous match.  Don't know what the British bookies would have given as odds, before this week, because no one ever considered the possibility of such an event.  Just as no one really considered the possibility of a big eruption from a broken oil well in the Gulf.

What's Up with Turkmenistan?

Via Matt Yglesias, Gallup has a global survey of nations which are thriving, struggling, or suffering.  It's color-coded.  Briefly US, Canada, UK, and Scandinavia are thriving, the rest of the world not so much.  But Turkmenistan sticks out like a sore thumb in having a higher score than any country within thousands of miles of it.  What's that about?

ERS Food Environment Atlas

ERS has an interactive map here.  (Warning, I had a problem with Firefox 3.5.3, but not with Chrome or Firefox 3.6. And you may have to "reset the map")  It displays state level, and some county-level data, on various parameters relating to food: availability of supermarkets, availability of fast food, demographics, consumption per capita of various kinds of food, etc. etc.

I think I owe a hat tip to Obamafoodorama.

The Problem With Statistics--Crime Waves and Falls [Revised]

Mathew Blake at Understanding Government links to a Wall Street Journal piece on the fall of crime rates. The piece quotes some experts who say that the graying of the boomer generation is partly responsible, since people over 50 are unlikely to commit violent crime (as opposed to financial crimes, perhaps?) and that age group  is the most rapidly growing.  Mathew has some doubts, but I have a different question. Suppose we went back to the high crime days of the 70's and 80's and normalized the statistics for the age distribution--what would the statistics look like?  I'm sure it would reduce the amplitude of the the peaks, but by how much?  And if public discussion and the media had used those statitstics rather than the gross figures they did use, would California today be spending more on public schools and less on criminal schools (i.e., prisons). That's a factoid which caught my attention today--Schwartzenegger wants to decrease the money spent on prisons because they do spend more on prisons than schools.

[Revision: thinking more about this issue, it strikes me the statistical adjustment would need to be tricky, not just for the number of young in the population, but probably also for the percentage of young.  Seems to me the boomers in the 70's gained confidence from being such a large generation, while those of us in the "silent" generation realized we were outnumbered. I suspect everyone has experienced the intimidation factor of a group of young, rowdy males on a public street where there's only a handful of adults around.  Conversely, a large crowd of middle-aged and old people can establish a standard of behavior that cows a handful of teens/twenties.  So the question would be, in this hypothesis, how often each scenario occurred. ]

Now USDA Messes With the Definition of a "Month"

Not content with defining "beef" and "veal", the USDA decides there are 13 months:

"Please be advised that 2008 13th month data has been applied to the FAS U.S. Foreign Agricultural Trade Database "

(The Foreign Ag Service has redone their statistics database here and "13th month" is a term for a catchall of corrections and late reports.
[Updated with the link I intended]

Surprising Stat--Lefties

No, I'm not talking about my fellow liberals but about baseball.

From a Times article on the scarcity of left-handed catchers in baseball:

But right-handed catchers do not seem to struggle throwing past lefties; besides, while right-handed hitters made 62 percent of major league plate appearances 50 years ago, it is now almost even, 56 percent to 44.

I know Mickey Mantle wasn't the first switch hitter in baseball, but he was the greatest one .

From a human perspective it's a lesson in how adaptable people can be, particularly when they have large financial incentives.

Unemployment Statistics

Are out today, and are bad. Some discussion in the blogosphere (Brad DeLong and TPM among them, I think) about comparisons with the past. As between now and 1930, say, I think the following are true:

• we may have fewer (proportionately) people institutionalized for mental problems
• we definitely have more people imprisoned (there's an interesting argument that since the 1950's we've moved people out of mental hospitals and into jails, keeping the proportion in some sort of involuntary confinement roughly the same)
• we have many more people in educational institutions
• we have more women working outside the home
• we have more people working inside the home (i.e., by computer, call-centers)
• we have more temporary workers.
• we have more older people able to work (i.e., better health and longer lived)
• we have fewer old people working (Social Security)
• we have more people in the military
• we have more people in the government

Definitions Matter: What Is a Farm

A former employee of USDA's Economic Research Service elucidates the definition of a "farm" in the 2007 Ag Census. It's a reminder that statistics are usually tricky to use, because the users aren't familiar with how the data was obtained and massaged.

Elsewhere he hits more strongly on the fact that farm prices increased dramatically between 2002 and 2007, which would affect farm numbers.

Are Farmers Rich?

All depends on your definition. If you look at Census data, there are about 1 million people employed in farming, forestry, and fishing with a median earning of %16,700, which is above the $11K for food service but below the rates for other occupations.

(It probably all is a matter of definitions, with Census and ERS definitions differing.)

Graphs Are Not Facts

I stumbled across a graph of mile driven in a green site. Here Then there was one here.

And here from the gov. And this one. All the same subject, but not the same statistic, and giving different impressions of reality. Reminds me of a classic book on How to Lie With Statistics, which everyone should read around freshman year in high school.

Sec. Spellings Gets My Praise

This week the head of the Department of Education announced a long-overdue change in statistics: standardizing the process by which the high school graduation rate is computed. I love it. The only way to discuss issues intelligently is if everyone is using the same words with the same meaning. Currently, states use different processes to compute a graduation rate. (If I remember correctly, my high school class had about 56 kids in 9th grade, by graduation we had 37 or so. The issue is the extent to which the rate accounts for dropouts. Because we don't have a system for tracking every child, that's difficult for the school bureaucracies--one system's dropout can be another system's in-transfer. I'll be interested to see how accurate the statistics can get.)

I have to say, this is a change that GW should have insisted on in the "No Child Left Behind" legislation. But then, his first Education secretary had played games with statistics in Houston, so Bush understandably didn't want to draw attention to statistics, particularly if they might undermine his major claim as Texas governor. (There--I had been too silent on GWB for a while--nice to get some criticism off my chest.)

But, progress is made in steps, and this is better late than never. Sec. Spellings should be commended.