It depends on how you ask: measuring bias in population surveys of compliance with COVID-19 public health guidance

Objective Accurate measurement of compliance with COVID-19 guidance is important for public health policy and communications. Responses to surveys, however, are susceptible to psychological biases, including framing effects and social desirability. Our aim was to measure the effects of these biases on estimates of compliance with public health guidance (eg, hand-washing, social distancing).

Design We conducted two online experiments (n=1800) and varied whether questions were framed positively or negatively (eg, ‘I always wash my hands…’ vs ‘I don’t always wash my hands…’). We also varied the degree to which anonymity was assured, via a ‘list’ experiment.

Results Reported compliance, despite being generally high, was reduced by negatively framing questions and increasing anonymity using a list experiment technique. Effect sizes were large: compliance estimates diminished by up to 17% points and 10% points, respectively.

Conclusion Estimates of compliance with COVID-19 guidance vary substantially with how the question is asked. Standard tracking surveys tend to pose questions in ways that lead to higher estimates than alternative approaches. Experimental tests of these surveys offer public health officials greater insight into the range of likely compliance estimates to better inform policy and communications.