May 2008
Sun Mon Tue Wed Thu Fri Sat
        1 2 3
4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31

Authors' Committee

Chair:

Andy Eggers (Gov)

Members:

Weihua An (Soc)
Kevin Bartz (Stats)
Sebastian Bauhoff (HealthPol)
John Graves (HealthPol)
Justin Grimmer (Gov)
Jens Hainmueller (Gov)
Mike Kellermann (Gov)
Ellie Powell (Gov)
Gary King (Gov)

Weekly Research Workshop Sponsors

Alberto Abadie, Lee Fleming, Adam Glynn, Guido Imbens, Gary King, Kevin Quinn, Jamie Robins, Don Rubin, Chris Winship

Recent Comments

Recent Entries

Categories

Blogroll

Brad DeLong
Cognitive Daily
Complexity & Social Networks
Developing Intelligence
EconLog
The Education Wonks
Empirical Legal Studies
Free Exchange
Freakonomics
Health Care Economist
Junk Charts
Language Log
Law & Econ Prof Blog
Machine Learning (Theory)
Marginal Revolution
Mixing Memory
Mystery Pollster
New Economist
Political Arithmetik
Political Science Methods
Pure Pedantry
Science & Law Blog
Simon Jackman
Social Science++
Statistical modeling, causal inference, and social science

Archives

Notification

Powered by
Movable Type 3.34


« Applied Statistics - Guido Imbens | Main | Judge Alito & Statistics »

31 October 2005

The Value of Control Groups in Causal Inference (and Breakfast Cereal)

Gary King

A few years ago, I taught the following lesson in my daughter's kindergarden class and my graduate methods class in the same week. It worked pretty well in both. Anyone who has a kid in kindergarten, some good graduate students, or both, might want to try this. It was especially fun for the instructor.

To start, I hold up some nails and ask "does everyone likes to eat nails?" The kindergarten kids scream, "Nooooooo." The graduate students say "No," trying to look cool. I say I'm going to convince them otherwise.

I hand out a little magnet to everyone. I ask the class to figure out what it sticks to and what it doesn't stick to. After a few minutes running around the classroom, the kindergardners figure out that magnets stick to stuff with iron in it, and anything without iron in it doesn't stick. The graduate students sit there looking cool.

From behind the table, I pull out a box of Total Cereal (teaching is just like doing magic tricks, except that you get paid more as a magician). I show them the list of ingredients; "iron, 100 percent" is on the list. I ask by a show of hands whether this is the same iron as in the nails. 3 of 23 kindergarten kids say "yes"; 5 of 44 Harvard graduate students say "yes" (almost the same percent in both classes!).

I show the students that the box is sealed (and I have nothing up my sleeves), Then, I open the box, spill some cereal on a cutting board, and smash it up into tiny pieces with a rolling pin. I take the pile of cereal around the room and let the kids put their magnet next to it and see whether the cereal sticks to the magnet. To everyone's amazement, it sticks!

Then I ask, are we now convinced that the iron in the nails is the same iron as in the cereal? All the kids in kindergarten and all the graduate students say "yes."

I respond by saying "but how do you know the cereal stuck to the magnet because it had iron in it? Maybe it was just sticky, like gum or tape." Now that I finally have their attention (not a minor matter with kindergartners), I get to explain to them what a control group is. And from behind the table, I pull out a box of Rice Krispies (which are made of nothing). We examine the side of the box to verify the lack of (much) iron, and then I smash up the Rice Krispies, and let them see if their magnet sticks. It doesn't stick!

Everyone gets to take home a cool fact (they love to eat the stuff in nails), I get to convey the point of the lesson in a way they won't forget (the central role of control groups in causal inference), and everyone gets a free magnet.

Posted by Gary King at October 31, 2005 2:18 AM

Comments

I was there when Gary did this (at Harvard, not in the kindergarten class). It was very effective, especially with the patter he describes above. I remember the grad students were spooked when Gary asked if the iron in Total cereal was the same as the iron in nails. No one wanted to get that question wrong!

Posted by: Alexis Diamond at October 31, 2005 1:39 PM

I can attest that Alexis did look cool tho! :-)

Does anyone have any unusual teaching experiences (kids or grad students) they might tell us about? I will say that teaching little kids is a lot harder than grad students and requires extremely detailed preparation. Imagine what your lecture would be like if every time you paused (or otherwise lost the attention of some in your audience), someone got up and ran around the classroom screaming or laughing.

Posted by: Gary King [TypeKey Profile Page] at October 31, 2005 1:52 PM

Teaching little kids is easier for me than Harvard graduate students, but that's because Gary spoils them! ;-)

1) Where can one buy 30 cheap magnets?

2) I tried to replicate Fisher's famous lady-drinking-tea experiment in GOV 1000 a few years ago. Instead of tea, I used Coke and Diet Coke. This was a mistake since many people can (too) easily tell them apart. I furthered my error by asking for a volunteer to test the drinks, thereby guaranteeing that I would have a student who was --- or at least thought of herself as --- an expert. Sure enough, the student determined the correct answer for 8 cups. Damn you Olivia!

;-)

Alas, having 8 picks correct is the least interesting data result for testing the hypothesis that the student can differentiate between the two.

In retrospect, the whole thing might have been a huge mistake, even if I set it up correctly, since the students were already in open revolt over the huge workload for the course and other matters too painful to recall.

Posted by: David Kane at November 1, 2005 12:17 AM

Gary,

Wow, that's a great demo! I'm a little disturbed that most of the grad students thought that the iron in the cereal was not "the same iron" as in the magnet. Didn't they learn the periodic table in high school??

Also, I don't have your experience as a magician, but I agree that "teaching is just like doing magic tricks." But do you really think magicians get paid more than teachers? I would guess that magicians get paid more per hour but less per year.

Posted by: Andrew [TypeKey Profile Page] at November 1, 2005 8:14 PM

Dave, Cheap magnets are available from hardware stores. It costs a few dollars. Easy to find. Just make sure they're strong enough (bring some smashed up Total!).

Andrew's referring to the fact that I paid for college by being a magician. Its certainly true that on average magicians don't make more than professors, but I found that it took a lot of years as a professor before at least I felt as flush.

Finally, to answer my question above, anyone who wants some great teaching tips shouldn't miss Andrew's book, Teaching Statistics: A Bag of Tricks.

Posted by: Gary King [TypeKey Profile Page] at November 1, 2005 10:25 PM

At least by grade school, I was the annoying student teachers loved and hated because I would then point out that we still didn't like to eat nails, because "all nails contain iron" does not lead to "all things with iron are nails."

Come to think of it, I'm still pretty much that way, except I figured out how to get paid for it.

But it's a very good demo. I might have to keep some of that stuff on hand for the next time the lunch discussion comes to some silly untested health claim and I need to explain why traditional medicine really is more reliable. Dunno if holistic medicine afficionados are as teachable as kindergarten or graduate students, though.

Posted by: Sandy [TypeKey Profile Page] at November 2, 2005 9:20 AM

Hope you didn't convince any kids (or grad students) to eat nails!

Posted by: Zaoem at November 2, 2005 9:59 AM

I use a similar "playing to student coolness" scenario, but it deals with spatial statistics.

In our discussion of the spatial mean (and its limitations of description), I have the students help me calculate "party central" on campus. I have them identify where most parties on campus take place (dorm buildings) and we plot their centroids on a map. I then say that if we plot the mean center, we should be in a place where all the action is (right?), and hang out there on a Friday night.

As it happens, the mean center is on the walkway between our building and the library.

I tell them that I would like to attribute this to geography being cool, I eventually pull an essential fact from the students: we have clusters of dorms on two ends of campus and the mean center is between the two.

By treating the two clusters of dorms separately, we get to a more reasonable answer of waiting for parties either in the parking lot, or in a wooded area (much more reasonable).

Posted by: Rich Crepeau at November 2, 2005 11:03 AM

I use a similar "playing to student coolness" scenario, but it deals with spatial statistics.

In our discussion of the spatial mean (and its limitations of description), I have the students help me calculate "party central" on campus. I have them identify where most parties on campus take place (dorm buildings) and we plot their centroids on a map. I then say that if we plot the mean center, we should be in a place where all the action is (right?), and hang out there on a Friday night.

As it happens, the mean center is on the walkway between our building and the library.

I tell them that I would like to attribute this to geography being cool, I eventually pull an essential fact from the students: we have clusters of dorms on two ends of campus and the mean center is between the two.

By treating the two clusters of dorms separately, we get to a more reasonable answer of waiting for parties either in the parking lot, or in a wooded area (much more reasonable).

Posted by: Rich Crepeau at November 2, 2005 11:04 AM

An excellent anecdote! I went to school in India and Im astonished that university kids didnt know that the iron in cereal is the same iron that is naturally occurring. I think this speaks to the quality of science education in the US K-12 system. I eventually became an accountant, but I still had to sit though balancing chemistry equations and studying the periodic table in the 6th grade.

Posted by: Continental Drift at November 2, 2005 11:18 AM

Great demo. Another control might be a lump of non-magnetized metal. I wasn't too surprised that only 5 of 44 grad students said the iron was the "same". It's an ambiguous question, especially sice the presence of a fortified cereal probably cued the concept that the chemical/molecular form in which the micronutrient appears is important to its nutritional effectiveness (so they might have been thinking "by 'same', does he mean just the same element, or the same element in the same chemical form?").

Posted by: Tony Plate at November 2, 2005 11:55 AM

I want to reinforce Tony's point: Asking if the iron is the "same" is not just ambiguous, but misleading. Syaing "the nitrogen in the air you breathe is the same as the nitrogen in cyanide poison" is correct in the same sense, but hardly useful in most contexts outside the lab or lecture hall. Dietary Iron comes in forms chemically distinct from the elemental iron and ferrous oxides present nails -- which is rather more relevant to most Americans and their dietary needs than the more esoteric point of elements being nature's buidling blocks.

Continental Drift - I think you have missed a key cultural point: The word iron is frquently prefaced with "dietary" in nutritional contexts. This vernacular would lead most Americans to assume (correctly, for their nutritional purposes) that the iron in cereal is different from iron in nails.

Posted by: tylerh at November 2, 2005 2:25 PM

I'm surprised Total -- even crushed Total -- has any particles sufficiently rich in iron to be attracted to a magnet. I'll have to try that for myself some time.

Posted by: Steven Jens at November 3, 2005 12:21 AM

Surely I am not the only one looking forward to a series of similar posts from Gary on this topic. After all, his daughter is now in the fourth grade! What other cool demonstrations have you done?

Posted by: David Kane at November 3, 2005 9:53 PM

Hey Gary, I stumbled in here...
If you give each student a (small) bag of skittles and look at the distribution of flavors in the bags, you can try to make inferences about the overall distribution of flavors. In how many bags do you need to see a disproportionately large number of grape Skittles to believe that the big Skittle DGP makes more Grape ones? This is also a good Central LT exercise since the distribution of flavors should be uniform (except for the extra grape!) but the number of each flavor per bag is distributed normally.
Matt

Posted by: Matt Lebo at November 8, 2005 8:51 PM

I agree with Tony Plate that non-magnetized metal is a better control group. Using Rice Crispies as a control only tells us that Rice Crispies is not sticky AND has very little to no iron, it does not rule out the possibility that Total is not sticky.

Posted by: tommie z at November 12, 2005 8:23 AM

Yes, I agree, and even better from a scientific perspective would be to manufacture Total Cereal without iron as the control group. But think about the audience here and the pedagogical purposes. Kindergarteners know what Rice Crispies Cereal is, but aren't even so clear on what magnets are supposed to stick to. Sometimes it makes sense to ignore the fine points in teaching, and then if the audience is capable of it to bring in the good points you raise after the fact.

Posted by: Gary King [TypeKey Profile Page] at November 12, 2005 10:30 AM

Gary, I couldn't agree more. Sometimes you need to keep the pedagogical point in the foreground.

Posted by: tommie z at November 12, 2005 1:21 PM

I'd like to point out that not all molecules which contain iron are ferromagnetic. That is, not every substance which contains iron will stick to a magnet. Is the iron atom at the center of a hemoglobin molecule the same element as other iron? Of course. Will a hemoglobin molecule be attracted to a magnet? No.

Also, ferromagnetic substances exist which do not contain iron, so it's not strictly true that "magnets stick to stuff with iron in it, and anything without iron in it doesn't stick." Maybe you don't have a lot of cobalt or rare earth elements lying around the classroom, but chances are that there are things made of nickel present.

Posted by: Edward at November 13, 2005 4:41 PM

". . . I show them the list of ingredients; "iron, 100 percent [of RDA]" . . ."

The recommended daily allowance (RDA) of Fe is 18 mg. If a serving of Total cereal is on the order of 100 g (4 oz), the iron content of Total would be on the order of 1 part in 5,000 (by weight)! It is highly unlikely that it is the iron in Total that was causing it to be attracted to a magnet.

Posted by: anon at November 13, 2005 5:02 PM

I won't eat Total because it tastes like, well, nails....

Posted by: tigger [TypeKey Profile Page] at November 13, 2005 10:43 PM

In 10th grade chemistry our teacher did a little experiment. She dumped a box of Total into a big beaker and filled it with water. At the bottom of the beaker was a mechanical stirrer (a magnet that is induced to rotate).

The next day we came to class she turned off the stirrer and removed the magnet. It was covered in iron filings.

Posted by: Scott de B. at November 14, 2005 10:37 AM

Sounds like your 10th grade teacher did a great job, but she forgot the control group! There should have been a 2nd big beaker filled with water and something else (like Rice Crispies) next to the one with Total, doing the same thing.
Gary

Posted by: Gary King [TypeKey Profile Page] at November 30, 2005 10:09 AM

Do you know what a problem for this science experiment is? I would like to know before Monday, March 27,2006.
Thank You So Much If You Tell Me.
Donna

Posted by: Donna Joel at March 25, 2006 9:26 AM

If you mean you'd like a homework assignment, then you could have the kids look through the other cereals at home and find which ones have 100% Iron and which have relatively little. Then you can ask them whether they can tell how much iron they have from just the name alone. In fact, you could collect the info yourself and have them guess.

Posted by: Gary King [TypeKey Profile Page] at March 25, 2006 9:51 AM

Notification

Enter e-mail address to receive notification of new comments to this entry

Post a comment




Remember Me?

(you may use HTML tags for style)