Survey instructions influence reported perceptions of environmental risks

This Bulletin summaries the findings from: Timmons, S., Papadopoulos, A. & Lunn, P. D. (2024). Survey instructions bias perceptions of environmental health risks. Journal of Risk Research. Available at: https://doi.org/10.1080/13669877.2024.2421006

Introduction

Measuring public perceptions of environmental hazards is important for effective mitigation policy, but doing so accurately is not straightforward. Irrelevant features of surveys, such as the order of the questions, can alter reported attitudes and beliefs. Our primary aim was to measure public perceptions of a relatively benign hazard that nevertheless receives media attention: electromagnetic fields (EMFs). EMF exposure levels in public spaces in Ireland are far below recommended thresholds of harm, but standard surveys suggest many people are worried about EMFs. However, we hypothesised that such survey responses could be biased upwards by survey instructions that mention the specific risk, increasing the salience of EMFs. We used a controlled experiment to test this hypothesis and compared generic survey instructions to instructions that highlighted the hazard of interest (EMFs) and instructions that highlighted an alternative hazard (carbon monoxide). 

Data and methods 

A sample of 800 adults took part in the online experiment. Participants were selected at random to read one of three sets of instructions: that the survey was about (1) “possible environmental risks”, (2) “electromagnetic fields… and other possible environmental risks”, or (3) “carbon monoxide… and other possible environmental risks”. Everything that followed in the survey was identical.

Participants completed a series of tasks designed to assess perceived risk. The first task required them to allocate a hypothetical budget across four environmental health hazards (selected from a pool of seven: carbon monoxide, E. coli, EMFs, lead in drinking water, microplastics, nitrous oxide, particulate matter). The logic of this task was that participants would allocate more budget to hazards they perceive as riskier. 

Participants then rated each hazard using seven-point scales for four dimensions of risk: perceived probability of being exposed to the risk, perceived severity of the consequences if exposed, general negative feelings towards the hazard and how often they think about the hazard in daily life. 

Results

The perceived risk from hazards did indeed depend on the stated purpose of the survey. Instructions that highlighted EMFs led to significantly higher budget allocation to mitigating EMFs compared to the generic instructions and the instructions that highlighted carbon monoxide. Compared to participants who were told that the focus of the survey was carbon monoxide, participants who read the EMF-focused instructions were more likely to report high probability of exposure to EMFs (a 43% increase), more severe consequences of exposure (24% increase), very negative feelings (a 21% increase) and more frequently thinking about EMFs (48% increase). Similarly, the instructions that mentioned carbon monoxide amplified the perceived risk from carbon monoxide (which was very high even in the control group), while diminishing perceived risk of the other recorded hazards. 

Conclusions

Drawing attention to a specific hazard in survey instructions can artificially amplify perceived risk estimates. This effect may explain apparent high levels of concern about EMFs in previous surveys. Future investigations of public perceptions of individual hazards are likely to be more accurate if they are embedded within larger studies on a broad range of hazards, with no individual hazard highlighted to the survey respondents.