Mapping Competencies for Communication for Development and Social Change: Appendix C-1: Preliminary Results of Delphi Study

- from Mapping Competencies for Communication for Development and Social Change: Turning Knowledge, Skills, and Attitudes Into Action

Appendix C-1: Preliminary Results of the CHANGE/Communication Initiative Delphi Study of Competencies


Prepared for the Bellagio Conference: Competencies: Communication for Development/Social Change

January 2002

A Presentation to the Conference by Dana M. Faulkner, the CHANGE Project;

Prepared by Susan Zimicki, Director, the CHANGE Project

What Is a Delphi study?

• Means of obtaining consensus on issues
• Usually a multi-round survey of experts
• Minimizes response bias (no one knows what anyone else's responses are)
• Frequently three rounds, but this study was two

A Delphi is a very specific kind of study. It doesn't aim to get a sample of representative opinion, but to build an unbiased consensus based on expert opinion. The way that bias is avoided is by asking opinions in a way that maintains privacy: no one giving an opinion knows what any one else's opinion is. Classically, Delphi studies have been carried out by mail, but people are now using the Internet.

Because it involves narrowing down choices, a Delphi study usually involves several rounds of getting and refining opinions.

The first round of this study was used to generate a list of categories; in the second round, people were asked to rate the importance of each of the categories. We didn't have time for more than two rounds, but in fact the results show such a high consensus that there is no need for a third round.

Innovative Aspects of This Study

• Open to all through the Communication Initiative Web sites rather than to a preselected group of experts
• In English and French on www.comminit.com
• In Spanish on www.comminit.com/la/

Classically, Delphi studies involve up to 30 respondents, who are selected because those running the study consider them to be experts in the field. This study was different: we wanted to hear from practitioners. We posted the questionnaires on the Communication Initiative Web site. All Drum Beat subscribers were invited to respond. Thus respondents were self-selected.

In the case of the first round, it is possible that some people who wanted to respond but had bad computer connections might have given up because of a bug in the questionnaire program - if they hit "enter" after typing in something, the program terminated the questionnaire, and to get the rest of their responses recorded they would have to start all over again.

There didn't seem to be any way to get rid of the bug, so we posted a notice on the questionnaire.

However, we are aware that some potential respondents may have gotten discouraged.

Purpose of This Delphi Study

• Allow a group of active practitioners to voice their opinions concerning the important skills, knowledge and attitudes needed by practitioners of communication for development/social change
• Inform the discussion at the Bellagio meeting

It is important to note that the results of this study cannot be considered representative of the opinions of all practitioners. We don't have any idea about the proportion of all practitioners who get Drum Beat or visit the Communication Initiative Website. However, the results reflect the opinions of more than 300 people, and we think will usefully inform the Bellagio discussion about competencies.

First Round Questions

• Respondents were asked to identify up to five SKILLS, prompted by the statement: "A competent communicator for development/ social change knows how to..."
• Similarly, they were asked to identify TYPES OF KNOWLEDGE: "...has a basic understanding of..."
• and ATTITUDES: "... believes that/believes in..."

This slide shows the core set of questions that were asked about the skills, areas of knowledge, and attitudes that characterize a competent practitioner of communication for social change/development.

We asked a second set of questions, trying to focus people's responses on people who work in development but whose main focus is not communication. It was clear from people's responses that the distinction between this group of people and "competent communicators for development/social change" was not always clear. Thus this presentation discusses only the results we obtained concerning practitioners whose main focus is communication for development/social change.

First Round respondents

• 315 respondents
• 65.1% answered in English, 25.4% Spanish, 7.3% French
• 29.4% from North America, 27.3% from Latin America/Caribbean, 15.9% Europe, 11.4% Africa, 6.7% Asia, 1.6% Middle East, 1.6% Australia/New Zealand

Just over 300 people responded to the first survey. This was a lot more than we anticipated, and made the task of analysis much harder.

As expected, most of the responses were entered in the English version of the questionnaire. However, because of the posting on the Latin American CI Web site, fully one-quarter of the responses were entered in the Spanish version. Only 7% ofrespondents completed the French version.

Almost equal proportions of respondents came from (were born in) North America and Latin America; 16% were born in Europe, 11% in Africa and about 7% in Asia.

First Round Analysis

• Answers were categorized by three independent raters
• Categories were reviewed and discussed and a general coding scheme generated
• Answers were coded according to this scheme
• Frequencies of responses in each category were then calculated

People were allowed to write in responses that were up to 60 characters long. The coding proved to be very difficult because there was such a great variety of responses and because people expressed themselves very differently. In addition, in some cases, people included two responses in the same statement.

The three independent coders each sorted the first response to each question into as many categories as they thought necessary. Cindy Rider, a Program Associate with the CHANGE project, produced a concordance between the three sets of categories, which was used as the basis of discussions about the choice of final code categories.

Once the coding scheme was finalized, she and others used it to code all the responses. This was a huge task - with 5 responses to each category and 315 respondents, they coded nearly 5,000 items.

We recommend that in the future, the category-generation phase of a Delphi should be limited to a core group of people.

Second Round Questions

• The form listed the top 12 categories of skills, knowledge and attitudes most frequently mentioned in the first round
• Respondents were asked to rate each category from 1 (not important) to 10 (extremely important)
• Respondents were also asked to choose the single most important category of skills, knowledge and attitudes

The second round of the Delphi was much easier to answer than the first round, because people could simply record their ratings as numbers.

Although we were still plagued by the same "enter" bug as we had in the first questionnaire, fewer people seemed to have problems with it, perhaps because the responses were so much shorter that they didn't forget and hit "enter."

We did, however, have a new bug: the default response to the "most important" question was set to "a", the first response in each category. Thus if a person skipped the "indicate the single most important skill" question, "a" would be recorded.

We corrected for this by excluding the people who had "a" as the chosen category for skills, knowledge and attitudes.

Second Round Respondents

• 311 respondents
• Slightly more skewed towards English-speakers than the first round:
  
  • 71.1% answered in English
  • 20.6% Spanish
  • 8.4% French
• Analysis still in progress

Interestingly, about the same number of people responded to the second questionnaire as to the first. These were not all the same people: 178 of the 311 had answered the first questionnaire, but the rest answered only the second.

We asked some information about each respondent's experience. However, because coding the first set of responses took so long, analysis of respondents' profession and training has not yet been completed.

Second Round Results: Skills

• Understand target audience, find out about context/ culture, analyze the situation: 9.32
• Listen, observe 9.20
• Communicate clearly and effectively 9.01
• Use participatory methods, give communities a voice 8.98
• Relate with people, empathize, communicate and work across cultures 8.87
• Motivate, mobilize, engage people 8.81
• Build partnerships (negotiate, build trust), collaborate, make linkages 8.84
• Develop communications and intervention strategies 8.56

Now let's look at what we've found. Here are the results for skills.

It is important to note that most people gave items high ratings. The average rating for all the skill items was about 8.5.

This slide lists the skills that received above-average ratings. They are listed in the order of their average standardized rating (that is, correcting for the tendency of people to rate things high or low). The number in bold is the simple average of all the ratings that skill received. Thus, on average, respondents gave understanding the target audience a score of 9.32 out of 10.

Respondents also scored listening and observing very high.

What is most impressive though is the strong emphasis that respondents put on participating, engaging, and negotiating. The only "desk work" skill that received more than an average rating is developing strategies.

Second Round Results: Skills

RECEIVED A LOWER THAN AVERAGE RATING:

• Manage (coordinate tasks, prioritize, organize, problem solve) 7.97
• Plan research, conduct research, use research, evaluate 7.73
• Produce and use media, e.g. radio programs, printed materials 7.78
• Organize meetings and facilitate discussion 7.64

This slide lists the skills that received ratings below the average for the skills category.It's interesting that managing, organizing meetings, planning research and producing media are all skills that imply some distance between the communicator and people.

Most Important Skills

• Understand the target audience, find out about context/culture, analyze the situation 23.2%
• Motivate, mobilize, engage people 18.9%
• Use participatory methods, give communities a voice 9.4%
• Build partnerships, collaborate, make linkages 9.4%

Finally, remember that we asked respondents to indicate the most important single skill.Here are the skills that were mentioned the most - by about 10% or more of the respondents. All of them reflect a central concern with the community.

Second Round Results: Knowledge

• Local conditions, including culture and language, community issues, cross-cultural issues 8.80
• Communication (including theory) 8.16
• Program planning and implementation, communication planning 8.03
• Development 7.87
• Use different media, including mass media 7.80

The average rating for areas of knowledge was 7.70.

This slide shows the areas of knowledge that received higher than average ratings.

Second Round Results:

Knowledge RECEIVED A LOWER THAN AVERAGE RATING:

• Organizational behavior, organizational change, group dynamics 7.64
• Behavior change theory, human behavior, social psychology, behavioral psychology 7.54
• Social sciences, social change theory 7.51
• Education principles and practice, training, adult education 7.47
• Research methods, evaluation 7.37
• Technical proficiency in special topic areas such as gender, HIV/AIDS, reproductive health 6.99
• Political science, political issues, policy, history 6.93

And here are the areas of knowledge that received ratings that were below the average for the category. Note, however, that the lowest rating is close to 7 - still extremely high! What characterizes these categories of knowledge is that they mainly concern areas that deepen understanding of social change and development processes; they are less central than the areas of knowledge rated most highly.

Most Important Knowledge

• Local conditions, including culture and language, community issues, cross-cultural issues 24.1%
• Social sciences, anthropology, sociology, social change theory 14.1%
• Communication (including theory) 13.5%
• Behavior change theory, human behavior, social psychology, behavioral psychology 13.5%

These are the four categories that were considered the single most important ones by at least 10% of respondents.

The importance giving to knowing local conditions complements the importance given to the skills of understanding the audience. The importance given to knowing about communication is self-evident.

What's interesting is that the other two categories - social sciences and behavior change theory - had overall ratings lower than 7.7, the average across all knowledge categories. Social sciences may have benefited from being category "a" - that is, because of a program bug it would get the respondent's vote even if s/he opted not to indicate a most important category. However, even when we excluded people who arguable did skip the questions about which categories they considered important, social science got more than 10% of the vote.

It appears that respondents thought that understanding the basis of social and individual change is important.

Second Round Results: Attitudes

• Respect for human and cultural diversity, tolerance 9.27
• Importance of participation (listening and dialogue) 9.22
• Value of local people and resources 9.09
• Honesty, openness, truth, integrity 9.00
• Teamwork, collaboration, sharing 9.05
• Communication can make a difference 8.93
• Equity, social justice, human rights 8.74
• Change/social change is possible, people can change 8.89

Finally, all the attitude categories were also rated high: the average across all attitude categories was about 8.70. The emphasis on tolerance, valuing local people, participation and collaboration echoes the highest skill and knowledge categories.

Second Round Results: Attitudes

RECEIVED A LOWER THAN AVERAGE RATING:

• Importance of learning 8.64
• Democracy, right to self-determination, local control 8.27
• Change involves a comprehensive approach 8.22
• Human goodness, humanitarianism, helping 7.78

Here are the categories that received below-average ratings.

Most Important Attitudes

• Importance of participation (listening and dialogue) 13.2%
• Respect for human and cultural diversity, tolerance 11.5%
• Value of local people and resources 11.8%
• Change/social change is possible, people can change 10.8%
• Communication can make a difference 10.1%

The most important attitudes emphasize participation and appreciation of the value of local people and resources as well as the importance of belief in the communication for social change approach.

Conclusion

• The Delphi produced a list of 12 skills, areas of knowledge, and attitudes that should be seriously considered for inclusion in lists of competencies
• There is a notable triangulation between the top skills, areas of knowledge, and attitudes considered to be the "single most important" ones reflecting a consensus on the overriding importance of the community

Compilation of responses about the skills, knowledge, and attitudes needed, according to respondents of the Delphi survey. NOTE: The double lines in each table indicate the average rating across all categories.