Remote monitoring and evaluation involve qualitative or quantitative data collection using a phone or mobile for monitoring or evaluation purposes (Hensen et al., 2020). Remote monitoring and data collection can utilize methods such as online or phone-based interviews, focus group discussions, audio-diary methods, photo voice, video documenting, documentary analysis of social media, auto-ethnography, and mobile phone surveys (ibid). The main components of remote monitoring include timelines, approvals, instrument design, coding surveys, and enumerator training (The World Bank, 2021).
Throughout the international development and humanitarian community, COVID-19 and conflict situations (more from the point of view of countries such as Afghanistan) have forced organizations to adapt their research, monitoring and evaluation to ensure high-quality data collection remotely (Elliott et al., 2020). COVID-19 has increased the requirement for remote data collection throughout the world in recent times. However, in a country like Afghanistan, increasing conflict in addition to COVID-19 makes remote data collection mandatory in various parts of the country.
Preparing for Remote Monitoring and Evaluation
Experiential knowledge shows that preparation for remote monitoring and evaluation in terms of developing sampling frames, sample selection, ensuring training of data enumerators, and fine-tuning data collection tools and applications are critical in ensuring successful remote interviews. However, these activities include several sub-activities and take lots of detailed discussions and planning, including revisits, re-strategizing, critical reflection, regular debriefings and feedback, and continuously improving the process of data collection. For example, while undertaking sampling for a nationwide survey of respondents exposed to radio and television programs, the sample was distributed based on the proportion of the population.
The sampling was conducted region-wise, province-wise, gender-wise and by educational attainment. Through discussions with the client, it was agreed that respondents who spent considerable years in Madrasa education should be treated as being educated. Additionally, the sample was to be aligned as much as possible with the country-level data on education as provided by the client. All this was done to make the sample as representative as possible. Nevertheless, this increased the challenge of reaching the respondents who fit into the required sampling frame. Further, an eligible respondent was one who had previous exposure to relevant program content on radio, television, or social media.
A screening questionnaire was used to recruit eligible respondents using a database received from the client and the proprietary database of Adroit Associates. However, the database shared by the client was provided to them by another research agency from a previous research assignment and most of the phone numbers were not relevant to the survey. In order to increase the numbers, which could have a higher possibility of fitting into the inclusion criteria for the survey, exponential non-discriminative snowball sampling was used. One of the major challenges was ensuring women’s representation in the sample. To address this challenge, snowball sampling proved to be effective. To ensure transparency and accountability in remote monitoring and evaluation, the client was provided with daily reports which included data about screening and final successful interviews completed.
The client also kept a keen eye on the data quality as the data was collected through the KoBo application and the KoBo database was a shared one between the client and Adroit Associates. Talking about the KoBo application was developed by the client but was fine-tuned by Adroit Associates. In order to fine-tune the application (both screening and main questionnaire), the application was subjected to a dry run of 10 samples and then 100 samples. Findings from the dry run were used to fine-tune the application to make the interviewing process more efficient (less time-consuming). However, this exercise proved helpful in training the enumerators as they experienced real-life situations while undertaking the pilot interviews. In order to reduce dropouts, screening and main interviews were conducted simultaneously. Further, more female enumerators were recruited as experiential knowledge shows that respondents are keener to talk to a female enumerator. Female enumerators also have higher interview completion rates than male enumerators during remote monitoring and evaluation.
The Process of Remote Monitoring and Evaluation
Enumerators were supervised by a research specialist who provided each of the enumerators (10 in this case) with a list of telephone numbers. The research specialist kept track of the numbers assigned to each of the enumerators and, at the end of the day, collected daily reports from them on the number of calls made, number of dropouts, and number of successful calls. A consolidated daily analytical report was provided to the client based on these reports. Regular meetings with the client focal point ensured that challenges such as lack of telephone numbers to call were addressed quickly.
The client also provided a number of Community Development Committee (CDC) members so that more respondents could be reached. Enumerators were not only present in Kabul but were present across Afghanistan in various provinces. Coordinating with them on a daily basis was challenging and time-consuming. However, placing enumerators in different regions ensured that the enumerators were aware of the local context and the cultural nuances. Enumerators operating from the Kabul office of Adroit Associates were asked to choose the quietest corners of the office to conduct the interviews so that they were not disturbed by background noise and voices. While undertaking interviews, the enumerators got to know that some of the respondents were not able to answer their phones while they were at work. Therefore, the enumerators scheduled an appointment with them later the same day and called them up at the scheduled time.
This ensured that eligible respondents were not missed out due to their time limitations. It was ensured that the enumerators took a very humane approach to interviewing and went beyond official working hours to ensure an interview was successfully completed. The enumerators were trained to be respectful and calm even if the respondents were unkind and abusive.
Conclusion
Remote monitoring and evaluation provide challenges as well as opportunities. It has become ubiquitous lately due to the COVID-19 pandemic; however, remote data collection methods have to be further experimented with to understand how effective they are in different contexts and in addressing different research needs. Moreover, remote data collection can present critical ethical challenges that researchers and enumerators should be aware of, particularly in identifying and recruiting study participants. Furthermore, the need for the use of snowballing in remote data collection has its own ethical issues and limitations. Adroit Associates’ experience in conducting remote monitoring and evaluation has been that the role of enumerators is pivotal in the entire process. The success of the remote interview heavily depends on the experience and skills of the enumerator.
Know more about our Monitoring, Evaluation, and Learning Services: