Friday, September 08, 2006

Standardized test score distributions-what are they measuring?   posted by rosko @ 9/08/2006 11:58:00 PM

I know this has nothing intrinsically to do with genetics, but it has something to do with cognitive ability testing, which I know is a favorite topic of discussion on GNXP. I was inspired to write a post on this since I am in the process of preparing for the GRE. Among the preparation materials that ETS provides are the distributions of verbal and quantitative test scores by graduate field of study.

Out of curiosity I looked through these, and found some interesting results. In particular, the shape of the distribution of quantitative test scores varies drastically by intended field of study. It is approximately normal for the humanities and some of the biological sciences, but is extremely left-skewed for physics, math and engineering majors, having a peak at around 700 but rising again near a perfect score of 800. In some cases this ceiling effect in quantitative scores is so severe that it's impossible to score above the 90th percentile!

The verbal scores show a very different pattern. The humanities and social sciences mostly show normal distributions, whereas the hard sciences typically show a bimodal distribution but with a similar mean. There is no evidence, though, in any major for a ceiling effect even remotely resembling that for the quantitative scores. The difference is such that, even though I scored a full 80 points lower on the verbal section than the quantitative section of a practice test, I didn't see a single major where I would have been at a higher quantitative than verbal percentile. This raises the question of what the verbal test is actually measuring, and why its mean seems to vary so much less by major. One explanation I came up with is that verbal tests measure mostly how much education you have had, and how much reading you have done, rather than intellectual aptitude. This is because they ask you for things like antonyms of quite uncommon words. In any case, I doubt that it measures the type of verbal ability factor that has been shown to trade off with visuospatial ability in many studies, including those assessing gender differences. If it were so, I would expect the scores to vary more, with the less spatially-oriented fields showing more left-skewed distributions. Also, I have always done well on verbal tests although I am predominantly a visuospatial thinker, and have always found analyzing literature to be one of the most difficult tasks. Any thoughts?