Generation Me?
In a recent post, Agnostic dismissed Jean Twenge’s thesis that narcissism has increased over the last couple of decades. Twenge has been on my reading list for a while, so this intrigued me. Not feeling knowledgeable enough to play devil’s advocate against agnostic, I sent Professor Twenge an email inviting her to join the thread. She does a pretty good job of defending her thesis against agnostic’s criticisms, in my opinion. I invite anyone who’s interested to check out the thread, read the studies, and share their own two cents.
Labels: Personality
She can’t defend against the criticism that convenience samples don’t allow inferences about a population. That’s the key thing — the claim about increasing narcissism isn’t about the particular individuals who took the test (that’s a boring claim), but who they are supposed to represent, whether young people in general or just the subset who are college students (that’s the interesting claim).
Without a probability sample, we have no way of estimating the mean, variance, or other aspects of the distribution for the population. As a result, we can’t say how the mean (etc.) have changed over time. Since choosing the members of a convenience sample didn’t involve chance, we can’t use probability theory to compute sampling error — what if the error bars (confidence intervals) for each year more or less overlap, despite an apparent upward trend in the mean? We can’t answer this if the C.I.s can’t be computed to begin with.
This is why people get paid to design opinion polls like the GSS or exit polls, to estimate the prevalence of an infectious disease, to see whether teenagers are having more or fewer sex partners than before, etc. It’s hard and expensive.
Because we can’t travel back in time, we have no way of getting probability samples from college students in the past, and thus will never be able to directly test the hypothesis. Still, some sample of data is better than none — we just won’t be able to answer too much with it. So, if the presentation was: “We have this really crummy data that was not designed to answer our question, and which statistics tells us probably cannot answer, but let’s see what it says anyway, taking it with a grain of salt,” then I’d be fine with it.
But it’s becoming the conventional wisdom that people just run with, ignoring that the empirical basis is pretty shoddy. Jesus, whenever some blogger reports a time trend in some variable from GSS data, does the news catch on? Not really — even though it’s from a probability sample, and so infinitely more reliable than purported trends gleaned from convenience samples.
That’s what really worries me: it’s news that lots of people are thirsty for — those damned selfish kids! — so the chance of them performing a check of the evidence is minimal.
And just on a priori grounds, the hypothesis doesn’t sound plausible — today’s 20 year-olds more narcissistic than those from the Me Decade or the Decade of Greed? Gimme a break.
If the effect were huge — imagine she found an increase of 12 inches in student height (4 S.D.s) in 30 years — then I’d be more inclined to not worry about convenience sampling. But when changes are expected to be subtle from year to year, as with personality traits, you absolutely need a probability sample.
In her May 2008 article she responds to the criticism you’re making (about convenience samples):
We are uncertain why Trzesniewski et al. (this issue) claimed that our analysis relied on convenience sampling. This is a term with an inexact definition because it is used differently across fields and is a matter of degree?perfectly random samples of people are virtually nonexistent. In psychology, it is most often applied to shopping mall surveys with low response rates, or to samples of one’s friends, and not to samples of college students from subject pools. However, it is important to consider whether our meta-analysis is a representative sample of the data available on college students’ NPI scores. It is. Reporting means in articles is not systematically biased by the level of the mean or how the mean has changed over time.
A separate issue is the data available on college student NPI scores. Although only some researchers study narcissism, there is no biased or nonrandom relationship between the location of the researchers and the campus’s mean NPI score. Thus a random sample of the population of college students at 4-year universities would likely yield similar results.
…
Even if we apply the term “convenience” only to less representative samples like college students, the vast majority of both descriptive and experimental psychology research uses such samples. The authors of these studies routinely generalize from college students to entire populations (e.g., Meston & Buss, 2007; Terracciano et al., 2005). If it were truly “Epidemiology 101″ not to use nonrepresentative samples in descriptive studies, our psychology journals would be nearly empty. So would the vitas of the authors of the comment.
…
Finally, college students are an important group to study. Two-thirds of high school graduates enroll in college (a much more meaningful indicator than the percentage of people age 18?24 in college, the statistic used by Trzesniewski et al., which includes people who have been in college in the past or may enroll in the future). College students, particularly those at 4-year universities, are also their generation’s future professionals and leaders. Thus, examining college students is central to a discussion of generational change.
Is her data really so biased that it doesn’t make the narcissism hypothesis more likely even if only for *college students*?
I guess the question is how biased her samples are..?
Here’s a handy discussion from an epidemiology textbook on sampling methods:
2convenience+samples%22+ %2Bepidemiology&source=web&ots=FKW4wGSTUC&sig=V_FP CiKitv28wJh7HSfcnHUmvkk&hl=en&sa=X&oi=book_result&resnum=4&ct=result
http://books.google.com/books?id=FaKaTvflBGQC&pg=PA203&lpg=PA203&dq=%2B%2
The reason why it’s so important in this case is that usually psychologists aren’t trying to estimate parameters in a population. They are trying to see whether variation in X is associated with variation in Y, hence all the correlations reported.
For example, do taller men have more sexual partners? Here, you’re not trying to estimate the mean, variance, etc., in male height or in number of sexual partners among the population of students, young people, or whoever. You just want to see if the taller guys have more partners. There could still be problems here — like, maybe the range is restricted and every guy you studied was either 5’10 or 5’11. But this isn’t the same as representativeness, which refers to whether we can estimate a parameter in the population using our sample.
Suppose you wanted to ask if young people or college students were getting taller — then you are trying to use a sample to estimate the mean height in the population. You’d need to design the study to ensure the sample is representative. The CDC has estimates of male and female height for different ages and races in the US, and these result from the kind of design I’m talking about.
In short, the trouble with the narcissism increasing research is that it wandered off the beaten path of correlational and experimental work, which is what most of psychology is, and into the territory of estimating population parameters and tracking them over time. There, convenience samples are no good. They might suggest a pattern to follow up on with a study designed to estimate parameters, but on their own they don’t tell us about the population.
She can’t defend against the criticism that convenience samples don’t allow inferences about a population
I think she is claiming that college going population is a statistically valid subset of the entire population at that age, and not a convenience sample. Given how large it is, it is hard to disagree. In fact, she said that the majority of the published pychological and sociologicl literatire would have to be abandonded if we started to assume differently now.
You like to boldify stuff but it doesnt really imprees
off the beaten path of correlational and experimental work
to
he territory of estimating population parameters and tracking them over time
These are your particular bug bears. Why wouldnt the pyschological literature measure differences in population over time ( and also be corrrlational and experimental, at the same time? )
The original piece you wrote “contradicted” these findings with anecdotes. We only see girls go wild during ovulation ( not proven, not universal, and doesnt explain why everybody ovulates at spring break) and a friend of yours posts sexier photos on facebook only when ovulating. That’s a lovely anecdote. But it is not a statistic ( and needless to say doesnt explain differences in intergenerational behaviour – since women have always ovulated).
You did admit this was anecdotal, but it is worse than that. Irrelevant.
If you’re too stupid to read the link to the epidemiology book that I put in the above comment, then don’t comment anymore.
You haven’t taken a statistics course, or even read a Wikipedia entry or anything, or else what I said would at least ring a bell. But let’s try one more time:
Estimating parameters in a population requires collecting a representative sample — period. For example, estimating the mean level of narcissism among any population, and then tracking this estimate over time.
Correlational work does not try to estimate parameters — only whether two or more variables are associated with each other.
Experimental work does not try to estimate parameters — only whether some hypothesized cause produces a hypothesized effect.
The confusion arises because “representative” has many ordinary language meanings. In statistics, it’s used to say that you can infer something about a population based on estimates from a sample you’ve taken from that population. It does not mean whether or not your findings generalize or can be projected onto some larger population that your subjects came from. I think “unbiased” is a better word to use in that case.
My anecdote did not contradict any findings — because there aren’t any. The burden of proof is on the person who says something is increasing. Particularly in light of the fact that everyone calls the 1970s the Me Decade, and the 1980s as the Decade of Greed, while in the ’90s the popular view is that everyone was too mopey and self-effacing. How true these popular views are, we don’t know, but they’re a priori grounds to suspect no increase in narcissism.
And they haven’t designed the study properly to convince anyone of the increase. There’s a suggestive pattern, but that’s it — there could be all kinds of problems with inferring that this shows narcissism is increasing among young people, because the samples were not representative (in the above strict sense).
My anecdote was to show how the popular conception of “kids these days are more narcissistic” could be wrong. The secular change in this perception is explained by the simple fact that MySpace and YouTube didn’t exist — did you not know that? I don’t know what being horny on spring break has to do with narcissism — i.e., thinking you’re better than others, having an overly inflated ego, etc.
Again — you’re not allowed to comment until you demonstrate that you’ve absorbed the distinctions I’ve been drawing, explaining at length, and providing links to free textbooks for further study.
In your post at Dusk in Autumn you write: “The data Twenge looked at weren’t representative samples, while the ones showing no change were.”
Then you say “My anecdote did not contradict any findings — because there aren’t any.” So what kind of no evidence is there?
Ah, I should’ve clarified that, and will change the entry sometime soon. The counter-Twenge authors had a large probability sample (not convenience sample) on high schoolers, looking at their GPA and self-perceived intelligence. If narcissism is increasing, they reasoned, high school students’ perception of their smarts should be increasingly outta whack with their GPA. But it wasn’t.
So, that wasn’t about the counter-Twenge authors’ NPI data.