Measuring participation or participating in measurement? the cautionary tale of an accidental experiment in survey accuracy

While survey results are always subject to measurement error, it is generally assumed that surveys of cultural participation are no less accurate than surveys in other areas of social science. The present chapter casts doubt on this assumption via a cautionary tale of events that befell an established national survey in Ireland. An organisational change led the survey to be conducted via the same method but using a different set of interviewers. The result was a surprising and dramatic increase in the headline figures, which related to participation in sport, both active and social. Subsequent data pointed to a systematic relationship between the decision to participate in the different surveys and in the activity being measured. The implication is that surveys of cultural participation may be subject to a specific form of selection bias. Furthermore, the effect size reported here suggests that this bias may be discomfortingly large.