Why Surveys Cannot Be Trusted

Being human means being quite curious about other humans. Specifically, we’re very interested in why people do the things they do. The better we understand each other’s motivations, the better we can serve each other. We know that at the most basic level, beliefs and preferences determine to a great extent what people do. These are the informational building blocks that shape human decision-making.

(Un)Fortunately, beliefs and preferences are hidden by default. Despite all the technological advances, humanity hasn’t even come close to inventing a device that would allow you to read anyone’s mind. So, instead, we resort to what seems to be the next best thing: ask people about what they believe and value, what researchers call survey research.

Survey research only makes sense if and only if people honestly report their beliefs and preferences. The value of survey research is directly linked to this fundamental assumption. It’s a fact of life, however, that we have the ability to misrepresent ourselves. And often, there are reasons to do so. For example, your willingness to pay for a new luxury watch will probably depend on who’s asking. You might overstate the amount to impress friends while you would downplay it to negotiate a good deal with the salesman.

Because people are free to misrepresent themselves, it raises the question of whether surveys can provide an accurate view of what people truly believe and value. Sadly, there’s solid evidence that surveys are unreliable and give a skewed picture. The problem is so systematic that there’s a whole body of scientific studies focused on what’s called the hypothetical bias.

The root of this problem is that talk is cheap. In a survey, there are no consequences to misrepresenting yourself. The problem worsens because we like to tell what people want to hear, also known as the Hawthorne effect. The end result is that survey measurements of beliefs and preferences are often significantly biased. Compare this to making a purchase. If you buy something that you don’t want, you’re going to regret making that decision. So, there’s a strong incentive to make decisions that correspond to your true beliefs and preferences. Actions speak louder than words.

This doesn’t mean that all survey questions cannot be trusted. There are no reasons to misrepresent, for example, your gender or highest completed level of education. In fact, the answers to these types of questions can be verified objectively. However, questions that require value judgments or reporting beliefs are susceptible to bias because these are inherently subjective.

The science of humans is the only field in which the subject matter is able to talk back. So it’s quite tempting to trust what people claim about themselves. This shouldn’t, however, prevent researchers from maintaining a high standard to get a reliable view of the world.