Usability testing to transform research impact: Lessons from Guatemala

Sometimes, as researchers, we unknowingly make assumptions that can cause our innovations to fail. This is especially true when it comes to scale out a technology; what works in one context may not in another. Here, we share how usability testing can provide crucial insights that challenge our assumptions, from our experience of developing a digital data collection tool in Guatemala.

First, let’s talk about what we mean by usability testing.

The only way to reliably develop innovations that work well for the intended users is to involve them in the process with a human-centered design approach. Usability testing is a critical method which involves observing real users as they interact with a technology. This allows the research team to identify usability issues and opportunities to improve their product. Just as importantly, it is a shared learning experience for the team that helps build consensus around the way forward.

In this sense, usability testing can be seen as another research method that complements the research being applied, whether providing agronomic recommendations to farmers or, in this case, a system to crowdsource data on diet quality. Usability testing requires some investment, especially to incentivize participants to attend, but it can identify critical issues that may be the difference between success and failure.

In practice: the Diet Quality Monitoring System in Guatemala

In Rwanda, the CGIAR Digital Innovation Initiative has successfully piloted a cellphone-based tool to crowdsource diet quality information at high frequency, in near real time. Partnering with an established cellphone services provider (Viamo), subscribers complete the Diet Quality Questionnaire (a survey already applied in 50 countries) via text message, in exchange for US $0.30 in free airtime.

To bring a similar approach to Guatemala, a Digital Innovation team is developing the “Live Food System monitor” in partnership with the UN World Food Programme (WFP) and Action Against Hunger (AAH). However, the telecommunications infrastructure in Guatemala does not support surveys by text message, nor is there a partner such as Viamo to support with message delivery.

Instead, the team is developing MeMo, a tool to enable crowdsourced data collection. So first we test it in the department of Chiquimula, part of Guatemala’s dry corridor. Real-time data on diet quality in this region would make for better policies and quick action in the case of rising hunger.

Due to low levels of literacy in Guatemala, the team assumed that interactive voice response (IVR) – verbally responding to a questionnaire over the phone – would be the best replacement for SMS. As the project is being developed with human design principles, usability tests were carried out with IVR and the second option of WhatsApp messages, with surprising results.

For the study, participants in WFP and AAH projects were recruited with the incentive of free mobile airtime. The study group included adults and young women between the ages of 17 and 42, with most participants being women as men were more likely to be engaged in the field work. The testing took place in-person, in a group environment, and in two scenarios.

Usability Test for IVR system with women from La Palmilla , Chiquimula. Credit: Luis Melgar

In the first scenario, the team provided an internet connection for the group to take the survey via WhatsApp. As predicted, a quarter were unable to complete the questionnaire due to factors such as literacy, the time demands of childcare and low uptake of WhatsApp among phone users.

The second test compared WhatsApp against interactive voice response, the WhatsApp questionnaire was sent beforehand while the IVR call was scheduled for when the group convened. Surprisingly, IVR did not outperform WhatsApp for usability: a lack of cellphone coverage in the area made it difficult to sustain a call, and the calls took on average 15 minutes to complete. Just 3% of voice calls were completed compared to 43% of the WhatsApp surveys sent out.

In this case, usability testing immediately delivered a result that confounded initial expectations and highlighted several other challenges to be overcome. Low cellphone battery, poor signal, infrequent purchases of internet airtime and high rotation of phone numbers make it difficult to send surveys to the database participants. A high rate of phone extorsion in Guatemala also means that end users are likely to mistrust surveys sent from unknown numbers.

Our experience in Guatemala shows the importance of working directly with users to develop products, with a skilled team able to ask the right questions. While our initial ideas might seem strong and solutions plausible, conducting such tests and gathering user feedback offers a reality check that no other design method can offer. Despite accompanying costs and challenges, the advantages of early identification and resolution of usability issues during the design phase significantly outweigh any drawbacks.