A comprehensive study of the Open Jirga program, gauging its reach and resonance among diverse Afghan audiences, to align program content with audience insights, ensuring effective communication and engagement.
BBC Media Action, renowned for its discussion and debate program Open Jirga, has been actively providing a platform for people across Afghanistan to engage with government officials on crucial governance issues. Another popular program, New Home, New Life, a cultural radio show, reaches 24% of the audience and remains highly acclaimed. Open Jirga, broadcasted through television, radio, Facebook, and YouTube, involves round table discussions on peace negotiations and the peace process. The Facebook Live program mirrors Open Jirga but is exclusively broadcast on the Open Jirga Facebook page. New Home, New Life is a socially relevant drama broadcasted on BBC radio and other local radio stations.
Despite their popularity, there was a recognized need to assess the impact of these programs on audience knowledge, understanding, and awareness. To address this need, the Evaluation Study of Discussions on Democracy in Afghanistan Project was initiated. Adroit Associates launched a survey targeted the audience members of Open Jirga, New Home, New Life, and Facebook Life programs across 16 provinces.
The study aimed to understand the engagement and preferences of the audience for these programs, assess their impact on the audience’s understanding of peace talks, the peace process, and local accountability mechanisms, and evaluate their influence on the political participation of women and men. Considering the project’s vast scope and contextual challenges, Adroit Associates’ evaluation study provided valuable insights into the complexities and learnings associated with evaluating the impact of media programs on public awareness and attitudes in challenging contexts like Afghanistan.
Project Goals/Objectives
The project aimed to:
- Measure audience engagement and preferences for Open Jirga and New Home, New Life.
- Evaluate the impact on the audience’s knowledge and understanding of peace talks, processes, and local accountability mechanisms.
- Assess the impact of Open Jirga and New Home, New Life on women’s and men’s political participation.
- Facilitate increased representation and participation of women in public life.
- Determine if Open Jirga provides a secure platform for various discussions and debates.
Our Approach
Adroit Associates initiated the data collection process through a comprehensive training session for enumerators. The training, spanning one day, focused on familiarizing enumerators with the intricacies of the data collection tool and elucidating the methodology. The data collection unfolded in two distinct phases: first, the implementation of a screening questionnaire to compile a roster of eligible respondents based on the sample plan and exposure to BBC MA programs; and second, the use of an evaluation questionnaire with the identified eligible respondents. Simultaneous execution of both steps aimed to minimize respondent dropout.
The data collection unfolded across three stages. The initial phase involved a dry run of the data collection application, during which enumerators provided detailed feedback that led to significant modifications in both the screening and main questionnaires. Special attention was dedicated to the educational background of respondents, particularly those with extensive madarsa education, who were considered educated for the purposes of the study. Alignment of the sample with UNICEF’s country-level education data was deemed crucial.
In the second stage, data was gathered from 100 respondents, and subsequent findings prompted further refinements to the data collection application. Issues with skip logics were identified and addressed through ongoing discussions between the BBC MA and Adroit Associates teams. Once the application was tested and deemed to meet the study’s requirements and BBC MA’s expectations, data collection resumed to complete all surveys. This meticulous approach ensured the reliability and integrity of the collected data throughout the survey process.
The study conducted as part of the BBC Discussions on Democracy project involved interviews with a total of 1163 individuals. Out of these, 701 interviews were conducted with male respondents, and 462 interviews were with female respondents. Geographically, 771 individuals were interviewed in rural areas, while 392 were from urban areas.
The project posed several challenges, but valuable lessons were learned. These included gaining increased experience in remote data collection, optimizing the management of resources for efficient data collection, improving communication skills to engage female respondents, and employing the snowballing method in remote data collection. The importance of assuring respondents’ confidentiality in the survey was emphasized, and enumerators and field officers became more skilled and adopted a professional approach in data collection. Language skills in Pashto were recognized as crucial for field officers operating in specific regions, and flexibility in interview scheduling was highlighted. Good communication skills for enumerators were deemed essential for encouraging respondent participation, especially considering the difficulties associated with conducting surveys over the phone compared to face-to-face interviews, despite potential cost savings.
The outcomes of this assessment have significantly influenced upcoming programming and initiatives in the region. The study has proven instrumental in offering valuable insights into the intricate dynamics and lessons learned from evaluating the impact of media programs on public awareness and attitudes, particularly in challenging contexts such as Afghanistan.
