Evaluating Food Delivery in a Pandemic: Usability & Ethnographic Principles to Guide Consumers

For almost 85 years, Consumer Reports has been detecting and anticipating shifts in consumer need for products and services so that we can guide consumer choice with rigorous research and testing. When the COVID-19 pandemic started to peak in March 2020 in the U.S., our ability to access the 63 product testing labs at our site changed. These labs house specialized equipment, such as a pressurized water dunk tank that is used to simulate conditions that electronics like phones and watches must be able to withstand if their manufacturers claim they are waterproof. And CR’s anechoic chamber removes all reflective sound signals, allowing a clean read on noise levels emitted by products. Such specialized equipment allows CR’s product testing experts to conduct repeatable and accurate testing across a large number of competing models in any given product category.

Like most other organizations, we had to quickly pivot. To the extent possible, our product testers set up makeshift labs in their homes, but at a time when availability of products was limited, shipping times were delayed, and stores were closed, we needed other ways to assess products and help differentiate among models. This is where CR’s Consumer Experience & Usability Research team was able to offer an alternative by evaluating products via user experience and our human factors expertise. Our team jumped in, armed with our remote research toolkit consisting of videoconferencing platforms, respondent-recruiting databases, and remote user testing and diary platforms.

We knew that these studies needed to offer deliverables in line with what CR’s audience expects: independent, rigorous, comparative, model-specific guidance on products. Our approach was to blend concepts from usability testing with ethnographic principles. We compared product models via users’ experiences and provided insights through natural context and use as opposed to testing the product performance with meters and monitors. In some cases, we reached out to current users of the products and services, while in others, we provided participants with a product to try over a period of time.

Learning About Food Delivery Services from Current Users

One of our first studies was on food delivery services. Because restaurants were closed for dining and people were overwhelmed with cooking so many meals at home, the option of getting food delivered to the door offered some relief, particularly for those under stay-at-home orders or in high-risk groups. We wondered how the experience varied between services and what, if anything, had changed as a result of the pandemic. It was important to offer the results from this study quickly when consumers were faced with so much uncertainty, but as UX researchers, we also saw the need for deeper understanding. A digital diary with some structured tasks offered the best combination of methods on a tight timeline. We used an online platform for both the recruitment of participants and administering the study.

Test Overview

Our insights were gathered through the experiences of 24 participants across the country. Each had primarily used one of the four major food delivery services (DoorDash, Grubhub, Postmates, and Uber Eats) for at least a year.

The study was set up in four parts, with a combination of video, written, and multiple-choice questions. Part 1 started with prompts to glean motivation for using the service, past experience, level of satisfaction, and impacts of the pandemic on their day-to-day food situation. In parts 2 and 3, participants were asked to document two separate instances of using the services to order food as they typically would. To gain structured measurements, scaled experience questions were posed at key moments in the journey:

1. When placing an order
2. While waiting for an order
3. When receiving the order

To understand their context, we asked participants to record videos when placing and receiving the order. After both deliveries were documented, participants completed Part 4, in which they could rate their overall experience and the system usability, and share thoughts on whether their opinions about the service had changed.

What We Learned

We learned that when we asked participants to assign a numeric value to their experiences, the scores across the services did not vary much. Participants were generally positive in their scoring, which is not surprising because they were evaluating the service that they primarily use. We found some finer differences when reviewing the qualitative and ethnographic data, but more notable were the similar themes across the four services:

Most participants were using multiple food delivery providers but listed one or two as their primary, go-to providers. The key reasons for their choices were:

• Availability of the providers’ services in their region.
• Ample restaurant selection (for some that meant variety, while for others it meant local and favorite restaurants).
• Promotions, discounts, and rewards.
• Brand loyalty.

When participants placed their orders, we noticed quirky things like these:

• Sometimes the service provider combined menu items and offered them as a package, but at times the items lacked appropriate customization options, e.g., a burger combo meal for two that doesn’t allow separate customization of the two burgers or sides.
• Menus were sometimes not reflective of changes that restaurants made in their offerings at the start of the pandemic.

While waiting for orders:

• Many participants wanted to be aware of the progress, but that sometimes made them more sensitive to shifting delivery times.
• Communication was often lacking between the provider and the participant, so in-the-moment decisions about things that might have varied from the restaurant menu were sometimes made by the provider alone.

When receiving orders:

• Drivers often did not pay attention to the notes regarding delivery location, gate code, entry process, etc.
• Contactless delivery instructions provided by participants were not always followed.
• Participants were most content when food packages were stapled or secured well.
• Deliveries were sometimes delayed by 5 to 30 minutes.

From a pandemic reality perspective:

• Most participants said that the delivery experience was the same, maybe even better with the addition of new restaurants and promotions.
• The only real difference noted was the addition of contact-free delivery.
• During the course of the study, many experienced delays and food packages left in unexpected places. One had missing items, and another had her order canceled.
• Some are still choosing hand-delivery because they:
  • don’t like food being left at the door or on a porch.
  • are concerned that food left outside their building could be grabbed by someone else.
  • worry that food can be left outside the wrong door.

Sharing Insights with Consumers

We shared our detailed research report with a Consumer Reports writer. For some of the issues that our report surfaced, she investigated further with the providers and experts in the field. The combination of human experiences and facts captured in an article gave people deeper information for using these services during a social and health crisis.

In Conclusion

For the Food Delivery Services study, the quick immersion into a discrete experience provided enough information for the type of guidance our audience was seeking. For some studies, we had to dial back on the duration of the project and try to achieve similar clarity by relying on people’s recall of the specific stages of use and the challenges they experienced in those moments (for example, bidets, hair dyes, meal kit delivery services, and electric toothbrush subscriptions). For others, we have allocated more time and provided participants with a new product or model that they have not used before (for example, beard trimmers). No matter what the balance, using a combination of usability and ethnographic principles has given us a tangible deliverable that aligns with our business goals to partner with and guide consumers. Furthermore, extending our capabilities during the pandemic has carved a path for our team and we now continue to offer insights directly to CR’s readers.

Images:”Philippine’s Sili (chili)” by Wayne S. Grazio via Flickr https://flic.kr