Harlem Children’s Zone
Via Steve Sailer and Half Sigma, we have this New York Times op-ed by David Brooks on work (pdf) by Will Dobbie and Roland Fryer on the Harlem Children’s Zone (HCZ). Brooks writes:
The fight against poverty produces great programs but disappointing results. You go visit an inner-city school, job-training program or community youth center and you meet incredible people doing wonderful things. Then you look at the results from the serious evaluations and you find that these inspiring places are only producing incremental gains.
That’s why I was startled when I received an e-mail message from Roland Fryer, a meticulous Harvard economist. It included this sentence: “The attached study has changed my life as a scientist.”
No one else seems to have linked to (read?) the study itself. Here are the key graphics:
Extremely impressive, if true.
Note, however, that there is no way (that I could find) to tell from the paper just how many observations make up the blue and red dots for 8th grade mean math scores in Fig 3A. Key paragraph:
We use two separate statistical strategies to account for the fact that students who attend HCZ schools are not likely to be a random sample. First, we exploit the fact that HCZ charter schools are required to select students by lottery when the number of applicants exceeds the number of available slots for admission. In this scenario, the treatment group is composed of students who are lottery winners and the control group consists of students who are lottery losers. This allows us to provide a set of causal estimates of the effect of being offered admission into the HCZ charter schools on a range of outcomes, including test scores, attendance, and grade completion.
Using a lottery as a method of randomly assigning students to treatment and control groups is far-and-away the best method for estimating causal effects. Their second statistical strategy, instrumental variables, is much less reliable. If the authors were merely reporting some regression-based estimates, few would take the results that seriously. Teasing out causal effects from a regression is very hard. That the authors do not use a propensity score approach (at least as a check against their estimates) makes me doubt their statistical chops.
Anyway, the lottery aspect is key. To their credit, the authors are upfront in admitting that:
[T]he HCZ middle school was not significantly oversubscribed in their first year of operation, and the HCZ elementary schools have never been significantly oversubscribed, making it more difficult to estimate the effect of being offered admission for these groups.
I think that the first year of operation refers to 2005, so the number of observations in the 8th grade loser category might be very low. Still, the authors report that “The effect of receiving a winning lottery number is generally larger for students in the 2006 cohort, though we only observe sixth and seventh grade scores for these students and so decided not to show it in our figures.” So, I expect that the 8th grade numbers reported here are not a fluke.
If you really started with 1,000 5th graders, randomly assigned 500 to HCZ and 500 to their local (lousy) public schools and then saw these huge differences in math scores, you would have discovered just about the biggest causal effect in the history of education research.
Have Dobbie and Fryer made that discovery? I don’t know. Their write-up and tables make it very hard to understand what is going on. What is the mean difference (without any “adjustments”) in 8th grade math scores between students who won the lottery and those who did not? It would certainly be useful if someone were to replicate these results.
The notes to Table 2 report that “Each regression controls for the gender, race, lunch status, and predetermined values of the dependent variable.” How do you control for “predetermined values of the dependent variable” in a regression? I have no idea.
Summary: There are many subtle issues in any study like this one. How do you handle missing data? What about students who win the lottery but decide, for whatever reason, not to attend an HCZ school? The authors mention several of these issues and their approach is reasonable. Still, a lot more focus on the lottery results and a lot less of the instrumental variables would have made for a stronger paper.
No one else seems to have linked to (read?) the study itself.
You really need to peek out of the Sailersphere sometimes (I keed ;) My favourite links:
http://gothamschools.org/2009/05/08/just-how-gullible-is-david-brooks/
http://www.prometheus6.org/node/24232
http://www.prometheus6.org/node/24255
(That prometheus6 website was an interesting find btw)
This piece by Thomas Sowell may be worth looking into:
“…there are schools where low-income and minority students do in fact score well on standardized tests. These students are like the bumblebees who supposedly should not be able to fly, according to the theories of aerodynamics, but who fly anyway, in disregard of those theories.
While there are examples of schools where this happens in our own time– both public and private, secular and religious– we can also go back nearly a hundred years and find the same phenomenon. Back in 1899, in Washington, D. C., there were four academic public high schools– one black and three white.1 In standardized tests given that year, students in the black high school averaged higher test scores than students in two of the three white high schools.2
This was not a fluke. It so happens that I have followed 85 years of the history of this black high school– from 1870 to 1955 –and found it repeatedly equalling or exceeding national norms on standardized tests.3 In the 1890s, it was called The M Street School and after 1916 it was renamed Dunbar High School but its academic performances on standardized tests remained good on into the mid-1950sÂ….
But the history of this black high school in Washington likewise shows a pay-off for solid academic preparation and the test scores that result from it. Over the entire 85-year history of academic success of this school, from 1870 to 1955, most of its 12,000 graduates went on to higher education.7 This was very unusual for either black or white high-school graduates during this era. Because these were low-income students, most went to a local free teachers college but significant numbers won scholarships to leading colleges and universities elsewhere.8
Some M Street School graduates began going to Harvard and other academically elite colleges in the early twentieth century. As of 1916, there were nine black students, from the entire country, attending Amherst College. Six were from the M Street School. During the period from 1918 to 1923, graduates of this school went on to earn 25 degrees from Ivy League colleges, Amherst, Williams, and Wesleyan. Over the period from 1892 to 1954, Amherst admitted 34 graduates of the M Street School and Dunbar. Of these, 74 percent graduated and more than one-fourth of these graduates were Phi Beta Kappas.9
No systematic study has been made of the later careers of the graduates of this school. However, when the late black educator Horace Mann Bond studied the backgrounds of blacks with Ph.D.s, he discovered that more of them had graduated from M Street-Dunbar than from any other black high school in the country.”
http://www.tsowell.com/speducat.html
If you have an IQ of 100, to what extent can you not handle Algebra and Geometry given drilling and pushing?
Does that mean you can handle differential equations and so forth?
There have been schools where API scores looked very good, except that after monitors were sent to each and every class in that school, the scores mysteriously went down.
By the way, how confident are we that “lottery” means “random selection with no peeking”? I applied my kid to a charter school founded by teachers from my kid’s middle school. There rule was that parents had to drop off the application in person and find out the results of the lottery in person. So, I went to see if my son’s name had been picked in the lottery. The teacher picked up the list of winners and said, “What was your son’s name?”
“Sailer.”
“You mean the kid who got a 5 on the AP Biology test in 7th Grade?”
“Uh, yes.”
He put the list down without looking at it. “He’s in.”
“Was his name picked in the random draw?”
“Don’t worry about it. He’s IN.”
There’s good reason to spend time on the IV strategy. The “random sample” may suffer from selection bias. Kids that enter the lottery may be different than the ones that don’t. That identification strategy is only estimating the effect of treatment on the kids that get into the lottery; treatment may have different effects on everybody else.
Why do you want the mean scores? The regressions show significant differences between the treated and untreated. Also, “predetermined values” are the pre-lottery test scores.
There’s also very good reasons not to use propensity scoring. Data limitations (those often found in economic analysis) greatly reduce that methods reliability. See Smith and Todd (2001).
I haven’t put in the time to really follow this story, but I did notice that on one test of the 8th graders, given on the same day, 2 of 77 students were absent for the reading portion vs. 20 for the math. Here is the link:
http://schools.nyc.gov/NR/rdonlyres/E54DBC8D-CEAC-41F2-A491-105A3FA25949/0/HCZPAI.pdf
Middle school is probably the most overlooked division of education. As more years were added to high school, less real academic growth was required in the middle years. For example in the local middle school, the promotion standard to go to 6th grade is a grade equivalent of 3.6 on the reading section of the Stanford Achievement test. That means the student must be able to read at the 3rd grade plus six months level or attend summer school in order to enter 6th grade. The interesting part is that the promotion standard for 7th grade is exactly the same. No growth is required in 6th grade.
The middle grades are where smart kids take more electives like foreign language and get involved in clubs and fun classes like drama etc. That is good, however schools where parents insist that achievement not take a backseat to enrichment, there is also Algebra in 7th grade, because parents demand students be given the opportunity to work up to potential. The school has to offer those classes because enough parents of strong students will take their kids out for it to wreck the school’s overall averages.
The achievement on the graphs are averages. It looks to me like they gave the kids the opportunity to do more, and enough of them were able to do it to bring up the overall average.
Kids (just like adults) have to be geniuses to come up with knowledge without instruction. However, they don’t have to be geniuses to learn with instruction. Too much of reform focuses on methods not on content.
The point about fuzzy data is not insignificant. I remember reading an analysis of some cancer research someone claimed showed there was no difference between two groups. When I went back to the original, I found that statement, no difference in incidence after adjusting for known risk factors. Well yea, one of the differences between the two groups was a known risk factor!
That’s why I was startled when I received an e-mail message from Roland Fryer, a meticulous Harvard economist. It included this sentence: “The attached study has changed my life as a scientist.”
Anyone whose life is changed over one study is an idiot.
Steve links to a Slate article from 2008 (http://www.slate.com/id/2198864/pagenum/all/)
about a book about the HCZ, which appears to cover the founding through 2007, a period when HCZ’s middle school “students do considerably worse on citywide exams during its first two years, Tough reports, than comparable kids in many regular public schools” and Canada is facing “impatient board members and financial backers”. And then, a miracle occurs. Hmmm….
Anyone whose life is changed over one study is an idiot.
Depends on the study, surely? E.g. Francis Galton reading Darwin’s Origin.
If I had to bet, I’d bet that this result was mistaken/exaggerated and/or fraudulent. That has been the case for a number of similar reports in the past – for _all_ similar reports in the past.
If it were true in a strong sense – if intelligence levels were the same in blacks and whites, while we’re at it in all human populations – it’d be hard to explain why any group had a brain size above the lowest human population average. But in fact, population averages vary by two standard deviations or thereabouts. There would certainly be immense practical advantages to smaller brains if they performed just as well – ask any woman who has given birth.
If memory and Google serve, someone named Steven Levitt took a look at cheating on high-stakes tests in the Chicago Public Schools.
It appears to have been fairly common – a minimum of 4 or 5%. One strong sign was a sudden large increase in standardized scores from one school year to the next: “one signal of cheating is unusually large gains in test scores for students in the year the cheating occurs, followed by very small test score gains (or even declines) the following year. ” – http://www.chicagobooth.edu/capideas/may05/cheating.html
Steve et al,
Right, they’re not publishing the IQ’s of the students (or their parents)involved versus those in the same community that weren’t involved.
And if they find IQ testing “distasteful”, as it’s non-PC, then they don’t account for the work ethic of students involved, versus those not.
Decades ago in Ireland I coached a poor student in Math, whose average grade was E – failing grade – and took him to an A in about 2 months. However, though he came from an impoverished family, and was poorly motivated, he was very bright and was the class smart-ass.
pconroy: Makes me wonder how he would have scored on a standard IQ test before and after “re-motivation”…
Foxp2,
Thomas Sowell has been going on and on about Dunbar high school for, like, ever.
Dunbar was the school of the Washington “colored” elite. Also, I don’t know what “national norms” meant in 1912.
What does “leading colleges and universities elsewhere” mean?
Sowell is not a hard scientist. He’s a good writer though.
“Sowell is not a hard scientist.”
That’s one way of putting it.
Toto,
I would say he would have scored highly before and after on IQ tests.
I was dating his sister at the time, and she was very bright, great at math and music – had perfect pitch, was something of a virtuoso at orchastration. Though his dad fitted carpet for a living, he was very quick witted. Meanwhile his uncle was something of a self-taught reclusive genius type, probably with Aspergers – he worked as a shop assistant by day, read advanced science books by night.
Just in relation to the brain size comment, there is a recent summary by Rushton & Ankney:
Rushton, J. P., & Ankney, C. D. (2009). Whole-brain size and general mental ability: A review. International Journal of Neuroscience, 119, 691-731.
“That’s one way of putting it.”
Yes indeed.
“If I had to bet, I’d bet that this result was mistaken/exaggerated and/or fraudulent.”
There’s loads of fraud in the standardized testing racket. In fact, the entire education industry is a racket. See this:
“In the late ’80s, the Beckley Register/Herald ran a story stating that all 55 West Virginia counties were above the national average on commercial elementary standardized achievement tests after which the local newspaper ran an editorial congratulating both state and local officials. Cannell read the stories and decided to do some investigating.
Upon researching “normed referenced” commercial elementary achievement tests, Cannell discovered that the commercial publishers are free to choose their own “norm group” (a group of students said to represent the average which are given the test without any test preparation). The publishers sell the test booklets, the norms, and the answer keys to school officials who reuse the same booklets (with the same questions) each year. Teachers told Cannell that any administrator seeking a promotion will have his teachers handle the booklets so that they can teach the answers to their students.”
Etc.
(Link: http://www.vitamindcouncil.org/cannellBiography.shtml)
Not-completely-off-topic: that vitamin D website in diana’s post seems a bit on the crackpotish side (vitamin D deficiency as the single cause of autism?), but it makes you wonder… Could this kind of trivial, skin-colour-dependent factor play a role in the B-W differences?
I’m thinking of the well-known Harrell, Woodyard and Gates study, where giving multi-vitamins to pregnant black mothers led to an 8-point IQ advantage in children at age 4 (against children from mothers who took a placebo). The result is generally interpreted in terms of nutrition, and thus of environment. But vitamin D is also directly dependent on skin colour.
The concept is obvious enough that somebody must have published a review of the evidence, so if someone has any pointers I’d be glad to know about it.
toto,
Did this study of Black mother’s account for the IQ of those who volunteered versus those that didn’t?
Daily Mail (UK) today has this article – Middle-class children have better genes!
I didn’t give the link because I buy everything that it claims about Vitamin D, I just thought the guy made some good points about the funny bidness of testing.
My own feeling is that we should take a look at that gigantic aptitude test called Life. Do you see any group differences?
“something of a virtuoso at orchastration”: bloody hell, then you were a brave fellow.
pconroy: the comparison was between placebo and vitamin takers, so no selection bias there.
bioIgnoramus,
Hmmm, what do you mean?
What I meant was that she could take any piece of sheet music, say for violin, and at an almost normal writing pace, re-write it for a different instrument, say piano, flute or viola, to accompany another instrument, or for a chamber quartet, as opposed to a full orchestra. She didn’t need to hear the music to know, almost intuitively, how it would sound in different settings, played on different instruments or groups of instruments, and at different tempos.
pconroy,
You probably meant “something of a virtuoso at orchestration“. Think about the meaning of the word you actually wrote :-)
Oh, you mean “orchestrate”? Much less painful, and easier to reverse.
Regarding the middle class genes, that is well known to anyone familiar with twin studies. Sandra Scarr, after conducting the Minnesota Transracial Adoption Study: “Within the range of ‘humane environments,’variations in family socioeconomic characteristics and in child-rearing practices have little or no effect on IQ measured in adolescence.” P. 476 ‘The G Factor’
“There is simply no good evidence that social environmental factors have a large effect on IQ, particularly in adolescence and beyond, except in cases of extreme environmental deprivation.” P. 476
Recent study discussed in New Scientist:
It is clear that intelligence is at least partly genetically determined. This was supported by the discovery in 2001 that the volume of the brain’s grey matter, made up of “processor” cells, is heritable and correlates with certain elements of IQ (Nature Neuroscience, DOI: 10.1038/nn758). The amount of white matter, which provides the connections between these processors, has since been shown to be heritable too (Journal of Neuroscience, vol 26, p 10235).
Now it seems that the quality of these connections, which is governed by the integrity of the protective myelin sheath that encases them, is also largely genetic, and correlates with IQ.
Paul Thompson and colleagues at the University of California, Los Angeles, scanned the brains of 23 sets of identical twins and the same number of fraternal twins, using a type of magnetic resonance imaging called HARDI. MRI scans typically show the volumes of different tissues in the brain by measuring the amount of water present. HARDI measures the amount of water that is diffusing through white matter, a measure of the integrity of myelin sheathing, and therefore the speed of nerve impulses. “It’s like a picture of your mental speed,” says Thompson.
http://www.newscientist.com/article/mg20126993.300-highspeed-brains-are-in-the-genes.html
Regarding the middle class genes, that is well known to anyone familiar with twin studies. Sandra Scarr, after conducting the Minnesota Transracial Adoption Study: “Within the range of ‘humane environments,’variations in family socioeconomic characteristics and in child-rearing practices have little or no effect on IQ measured in adolescence.” P. 476 ‘The G Factor’
“There is simply no good evidence that social environmental factors have a large effect on IQ, particularly in adolescence and beyond, except in cases of extreme environmental deprivation.” P. 476
Recent study discussed in New Scientist:
It is clear that intelligence is at least partly genetically determined. This was supported by the discovery in 2001 that the volume of the brain’s grey matter, made up of “processor” cells, is heritable and correlates with certain elements of IQ (Nature Neuroscience, DOI: 10.1038/nn758). The amount of white matter, which provides the connections between these processors, has since been shown to be heritable too (Journal of Neuroscience, vol 26, p 10235).
Now it seems that the quality of these connections, which is governed by the integrity of the protective myelin sheath that encases them, is also largely genetic, and correlates with IQ.
Paul Thompson and colleagues at the University of California, Los Angeles, scanned the brains of 23 sets of identical twins and the same number of fraternal twins, using a type of magnetic resonance imaging called HARDI. MRI scans typically show the volumes of different tissues in the brain by measuring the amount of water present. HARDI measures the amount of water that is diffusing through white matter, a measure of the integrity of myelin sheathing, and therefore the speed of nerve impulses. “It’s like a picture of your mental speed,” says Thompson.
http://www.newscientist.com/article/mg20126993.300-highspeed-brains-are-in-the-genes.html
Just a heads up that Rushton & Jensen have released a working paper reviewing Nisbett’s recent book on IQ. Comments are invited.
Working Paper, May 10, 2009,
Rushton, J. P., & Jensen, A. R. (2009). Race and IQ: A Theory-Based Review of the Research in Richard Nisbett?s Intelligence and How to Get It. Working paper, Department of Psychology, University of Western Ontario, London, Canada.
http://psychology.uwo.ca/faculty/rushton_pubs.htm
E-mail: rushton@uwo.ca
Nesnejanda@aol.com
The Moldbuggian Universalists at Metafilter have linked here. Quintessential douchebag of the site ‘Delmoi’ also took a jab at this site and is running his mouth of again about stuff he has very little knowledge about, which is about par for his contributions. OMG PEOPLE WHO TALK ABOUT GENETICS ARE RACIST.