WP 2014-05 Hypothetical Surveys or Incentivized Scoring Rules for Eliciting Subjective Belief Distributions?

AUTHOR: Glenn W. Harrison

ABSTRACT. Is there a difference between the elicited subjective belief distribution obtained from hypothetical surveys or from incentivized scoring rules? If so, are they correlated? What is the interaction between responses to unverifiable events and comparable, verifiable events when one interacts the use of incentives? We address these questions with controlled experiments. The relationship between the inferences made from subjective beliefs elicited with hypothetical surveys and incentivized scoring rules is complex. It is easy to find examples where the two generate results that are different, and in a statistically significant manner. In that sense, one can reject the naive claim that there is no difference between hypothetical and incentivized responses. On the other hand, there are important inferences for which it does not matter. If someone was in front a jury arguing that this sample generally underestimates the effect of smoking on the risk of dying from cancer, the differences in hypothetical and incentivized responses does not matter. Nor do we generally see that the differences in hypothetical and incentivized beliefs applies equally to everyone or across the demographic board. What we do see is that hypothetical bias varies significantly for particular demographic sub-samples, and not systematically across questions. Thus the absence of an overall effect is due to “offsetting biases” for demographic sub-samples. If one actually wanted to make inferences about the average belief, this would provide some confidence in hypothetical surveys to provide a reliable measure. However, one should not cite this “aggregate, net non-result” and then use individual data on hypothetical beliefs as if it were the same thing as incentivized beliefs: that is simply a non sequitur.