Evaluation of Social Mobilization Network (SM Net): Final Evaluation Report

In 2013, scattered polio outbreaks in Somalia prompted a proactive vaccination response. As part of this, the Social Mobilisation Network (SM Net) was established in 2013 by the United Nations Children's Fund (UNICEF) to raise awareness of polio and increase coverage of polio vaccination in difficult-to-reach locations in Somalia. (See Related Summaries, below, for more on SMNet.) In 2016, Kimetrica was contracted to conduct an evaluation of the outcomes and impacts of SM Net on activities, including coverage of polio immunisation activities, community-level support for immunisation, and local perceptions of trust in health service delivery. Commissioned by UNICEF office for Somalia, the evaluation was also designed to explore SM Net's ability to deliver on other child survival and development interventions.

Using a mixed-methods approach, the Kimetrica team collected primary data in collaboration with their partner Forcier Consulting, through in-depth interviews (IDIs) and focus group discussions (FGDs) with the following groups of stakeholders: SM Net partners (Group 1), coordinators and implementing partners (Group 2), vaccinators and District Field Assistants (DFAs) (Group 3), community mobilisers (CMs) (Group 4), and community members (Group 5). These interviews were conducted February through April 2017 in the 14 SM Net districts spread across Puntland, Somaliland, and South Central Somalia. Secondary data from Independent Monitoring (IM) by the World Health Organization (WHO) provided information on changes in vaccination coverage and refusal rates from 2013 through 2016.

Summary of overall findings in terms of:
Relevance:

• Most respondents said that programme needs were being addressed, but improvements could be made in improving the number of CMs, and the training and resources provided to them, particularly in rural areas.
• Most respondents thought that the SM Net campaign was relevant to local contexts - in particular, its use of CMs from the community, engagement with local leaders, and the respect of their religious customs and practices.
• The key constraints and difficulties in implementation related to access, particularly in insecure areas of South Central, and mistrust. Mistrust has decreased over time, and is helped by engaging CMs from local communities and working together with religious and other respected members of the community.
• Even though the focus of SM Net activities changed over time (e.g., initially focusing on raising awareness, then the vaccination of missed children, and now reaching the nomadic populations), the ability to respond to these changes was not well articulated, though an obvious gap in addressing the needs of people during the current drought was raised.

Effectiveness:

• Knowledge and awareness of polio, immunisation, and the local polio campaign amongst community members, including hard-to-reach groups improved since the start of the campaign, though there are still people sho reject the vaccine, mainly fathers and some religious groups, and some people who have had little access to awareness such as internally displaced persons (IDPs), nomads, and those in remote or insecure locations.
• SM Net changed attitudes towards the polio vaccine, with communities actively seeking out the polio vaccine and other health care, though negative attitudes still exist, especially for those still not being adequately reached by the CMs, such as nomads and IDPs.
• The most effective external collaborators were identified as religious and traditional leaders, and the government, and the low performance of SM Net in establishing effective collaborations in Somaliland was attributed to the poor relationships with the relevant Ministries.
• The partners and coordinators were satisfied with the SM Net management and coordination structure, though some identified weaknesses in the budget, the monitoring systems, and the lack of government ownership.

Efficiency:

• House-to-house visits by CMs were perceived to be the most efficient activity, and the perceived efficiency and value for money of the programme varied by zone.
• SM Net's efficiency in reaching hard-to-reach groups was rated low, and the proposed alternative approaches were expected to be more expensive.

Impact:

• Whereas most partners and coordinators rated the impact of SM Net in increasing coverage as good or very good, refusal rates were still seen as a problem.
• There was consensus that SM Net had increased the profile of polio and the prioritisation of polio eradication in the region, though there are still problems in accessing and influencing the hard-to-reach groups.

Sustainability:

• Respondents confirmed that SM Net has built trust and acceptance amongst community members in the frontline workers, and communities felt involved in the network, though some distrust in the vaccine remains.
• The communities were satisfied with the information and support provided, though there remained some isolated groups where the experience with the CMs and vaccinators had fallen short of expectations, and some areas for improvement in the service provided.
• The frontline workers were satisfied with the work that they do, but raised concerns about the resources provided to them to carry out this work.
• The approaches used by CMs (house-to-house visits and loudspeakers on cars, rather than mass media) were considered to be the best sources of information, though specific hard-to-reach groups may require different approaches. For nomadic groups, for example, these might include radio messaging and scheduled awareness-raising at key water points.
• The majority of respondents thought the SM Net approach was sustainable, particularly the use of CMs and house-to-house visits, though improvements could be made. There was consensus that the network could be used for other health campaign messaging.

Primary and secondary data support the claim that "the SM Net programme has been a great success in the communities it was able to reach. Initial hostility and mistrust have generally been overcome, coverage rates have increased, and the number of unvaccinated children has decreased. For example, overall coverage rates increased by 7 percent, from 90 percent in 2013 to 97 percent in 2016....Furthermore, whereas initially 1.2 percent of parents refused to let their children be vaccinated, by 2016, this fell to 0.6 percent."

Looking ahead, recommendations offered through the evaluation process include:

1. Support CMs with more resources - To optimise their work, it is recommended that: CMs are chosen from the same community in which they work; they are well-resourced in terms of training, incentives, and access to transport; there are sufficient CMs, and adequate time is allocated to cover the assigned communities; and teams are comprised of both males and females to facilitate communication with household members.
2. Engage more with external stakeholders - Given the importance of religion in Somali life, religious leaders were key to ensuring community trust and acceptance of SM Net activities. The evaluation finds that nurturing these relationships will be essential for winning over the remaining vaccination refusals, and in extending coverage to hard-to-reach groups. Going forward, it is recommended that other respected authority figures, such as doctors, traditional medicine practitioners, and teachers, are also included, and that stronger relationships are built with the Ministry of Religious Affairs, Ministry of Education, Ministry of Health (MoH), and local government.
3. Tackle refusals by learning why they persist - Further research is required to understand why certain groups of people (such as fathers and some religious groups) continue to refuse the vaccine.
4. Tackle hard-to-reach groups with novel approaches and more local engagement - In remote rural areas, it is recommended that in addition to training select local individuals in a central location to go back home and spread the word, more resources should be directed to reaching these locations, and novel approaches to raising awareness in public forums should be explored. With both nomads and those in remote areas, it is essential to engage local leaders and traditional doctors, and develop appropriate visual tools as literacy is low.

As noted above, all stakeholders agreed that the SM Net approach should be used for other health campaigns. However, more research is needed to identify how this would fit into existing health care activities, and what additional resources, such as outreach services, would be required to ensure access to interventions.