Social Science Statistics Blog

I am a postdoc research psychology fellow cross-training in public health. This is a fascinating and important area, and I have a couple thoughts.

Obviously it is often difficult or impossible to obtain anything more than self-rated health outcomes in large scale research (ie, epidemiologic surveys). Vignettes may possibly correct this under some conditions. However, my immediate impression is that there are some limitations it is important to be aware of.

In the paper on King's web site, he admits 2 key assumptions of the vignette method (p. 4): response consistency, or the the notion that "each respondent uses the survey response categories in the same way to answer the anchoring vignette and self-assessment questions" and vignette equivalence, or "the assumption that...levels of the variable represented in the vignette is understood by all respondent in the same way apart from random measurement error."

These are huge assumptions! People bring all sorts of idiosyncracy in perception to all survey items, including the vignettes proposed to correct initial items. Since people often employ different criteria to evaluate themselves vs. others, the first assumption that the response scale is being used similarly may well be violated in any given individual. Some are likely to apply the anchors more favorably to their own version of the question than the vignette (self-serving bias) while the reverse will be true in others (for instance, depressed people or those with self-denigrating tendencies will evaluate themselves worse, using the same set of anchors, than a hypothetical external person in the vignette).

Regarding the second assumption that respondents will perceive the vignette equivalently, this suffers from the initial problematic assumption that respondents will view the initial survey item in the same way! Simply wording the vignette with precise details does not help because people differ in the meaning and importance they attach to the same detail, or even the same descriptive word used. What is one supposed to do--develop another vignette to anchor the anchoring vignette?

These assumptions may be like ever-elusive normality assumptions in parametric models--rarely met. Having said this, perhaps this method is robust to violations of these 2 assumptions. I will certainly consider its applications in our group's research.

I also wonder to what extent this improves over a well-worded, precisely operationally defined survey item. So it may be more useful for ambiguous survey items, like the famous General Health item "In general, my health is...poor, fair, good, etc."

In the end, psychologists have been studying response biases and how to model them for decades using classical test theory, and more recently using item response theory. But these are typically back-end statistical methods of correcting for, modeling, or otherwise quantifying response biases in a set of already-collected questionnaire items. I think the potential contribution here is the possibility of front-end design which might alleviates some differential item functioning. It is interesting, and ultimately an empirical question whether this will help.