New research outlines challenges of evaluating the impact of community development programmes

SICAP aims to tackle poverty, social exclusion and long-term unemployment through partnerships between disadvantaged individuals, community organisations and public sector agencies. This report focuses on interventions with LCGs with the overall goal to “support and promote the community engagement of disadvantaged communities and individuals across the life cycle”. The type of LCGs who receive support from SICAP are varied. Some examples include men’s sheds, youth groups, drugs task force, traveller women’s groups, mental health groups, community garden networks, and active retirement groups.

This report looks at how best to record the work of SICAP with these groups and the effect they are having. The research finds that it is not practical to measure the influence of SICAP on broad community level outcomes, such as poverty rates or improved education/income levels, because disadvantaged communities are typically receiving a range of supports and services as well as SICAP, making it difficult to isolate the effect of SICAP. Instead, the report proposes a way of assessing how groups are progressing in relation to the variety of goals they set for themselves.

This research highlighted many reasons why it is challenging to assess outcomes:

• Numerous national agencies simultaneously implementing policies that affect such broad outcomes making it difficult to isolate the impacts of one particular policy;

• Local organisations targeting specific communities often receive funding from multiple sources, making it impossible to measure the impact of a particular funding stream even in instances where the community-level outcome measures are narrowly defined and identifiable;

• It may be more feasible to focus on more narrow outcomes for evaluating the impact of funding to community level organisations. However, local community organisations tend to be highly diverse in nature with differing objectives. This makes it very complex to identify a set of specific community-level outcome measures relevant to the activities of all LCGs.

• The identification of comparison groups at a community level who have not been subject to any policy interventions against which to measure the impact of the intervention is also extremely difficult. 

The research proposed a number of approaches with potential to monitor SICAP community development interventions effectively.

Effective monitoring of community development initiatives within the SICAP model is possible with the collection of broad metrics relevant to all local community groups for use within a Logic Model framework. This framework outlined in the research shows the logical relationships among the resources that are invested, the activities that take place, and the benefits or changes that result.

Two suggestions for effective monitoring seem to warrant further consideration: (i) a ‘distance-travelled’ measurement tool for LCGs, which documents the progress they have made in relation to the goals they have set and to assist self-assessment by LCG members and local community workers, and (ii) regular national-level qualitative reports. These reports can focus on specific impacts of funding to particular target groups, such as lone parents, those with disabilities or those facing homelessness. Such qualitative approaches allow for a more systematic and in-depth analysis of the impact of SICAP on community level outcomes.

Michael Ring TD, Minister for Rural and Community Development, the Department with responsibility for SICAP said: 

“I welcome this constructive approach to assessing the impact of SICAP for communities and individuals throughout Ireland. I know that the programme has an extremely positive effect at ground level in communities. Effective assessment can positively inform the future direction of the programme and help ensure that SICAP supports those that need it most.”

Author of the report and ESRI Research Officer Adele Whelan, commented,

“Community development programmes, similar to all government-funded activities, require adequate monitoring and measurement to ensure value for money. Programme evaluation helps us to answer key questions for evidence-based policymaking: what works, what does not, and for how much? However, this research report highlights several complexities in demonstrating a causal link between community development interventions and changes in broad measures of community well-being and suggests a number of approaches that could effectively monitor outcomes in future.”