Just Add Water: Lessons Learned from Mixing Data Science and Design Research Methods to Improve Customer Service

BACKGROUND AND CONTEXT

Anyone who has ever called a call center more than once knows the process can be painful. Customers call with a quick question only to be forced to provide a slew of non-relevant information. Most often they’re forced to give information to automated systems. Once their question goes unanswered; only then do they get to talk to a person. But by then they’re forced to repeat the irrelevant information to a call center agent before they can actually ask a question.

Even getting a human agent doesn’t guarantee the question gets answered if the information needed isn’t at the agent’s fingertips. Good luck if that call gets interrupted. The customer calls back and the frustrating process begins again. The process can be equally unpleasant for call center employees. Call center workers routinely complain of customer aggression and one study shows 1 out of 5 calls to contact employees are from angry customers, an average of 10 calls a day. The aggression from customers has risen over the years, so much so that the turnover rate within call centers has exploded from 19% in 2008 to 24% and rising today. (Dixon, Ponomareff & Turner 2017)

For every call center interaction between an agent and a customer, there’s an equal and corresponding data creation and storage action. Each phone call a customer makes. Every email an agent sends to a customer. Each time a customer goes to a website and enters the search words of some product they want or exotic local they want to visit data collected and often stored. All of these actions create data points. These data points, or digital bits as they’re known by the industry, make up what’s called the digital universe – all the bits of data that are created, replicated, and consumed by people and businesses. (Gert & Reinsel 2013) But even though all this data available, call centers can’t seem to recognize who is calling, why they’re calling or how to more quickly solve their problem. This not only frustrates customers but annoys agents as well. It’s like there’s data everywhere but not a drop that’s useful. It’s a common problem across industries. A lot of data is being created, collected and stored but that data is incredibly difficult to track back to individuals. There’s many reasons for this, but one of the central reasons is that realistically, 91 percent of companies globally report they have inaccurate data, according to a survey of more than 1,200 global companies on data quality commissioned by Experian^. (SourceMedia Research 2018) The most common ingredients that lead to data inaccuracies include incomplete or missing data, outdated data and wrong data.

In addition to the data being notoriously inaccurate, data on individuals are collected through multiple channels that are rarely connected. A company may have data on one individual collected from at least three different channels, including email, a mobile application, a phone call or a text. A customer’s information is split piece by piece into three, four, five, even 10 different databases, never giving companies a true “360-degree” view of the customer.

Some Level-Setting Definitions

Before explaining the methods used during this particular design project it’s helpful to do some level-setting definitions. For this paper, let’s define data science and data scientist as it is practiced at IDEO. Data scientists at IDEO are adept at shaping data as a resource for human-centered design. They sketch in pencil and in code, shape and reshape data, know what data can and can’t tell us, help design feedback mechanisms, and prototype machine learning algorithms. They do all this to improve human experiences through the design of intelligent products, services, and systems. Data is the paint they use in the art of creating intelligent products.

The most easily understandable example of how data scientists and designers come together to create products is The Nest. This intelligent thermostat system developed by famed Apple designer Tony Fadell, and now owned by Google, is a smart device. The Nest processes data from a number of places including the Internet, the body heat of people inside a room, and adjusts a home’s temperature accordingly. To create such smart devices, a company needs people who understand and can develop devices that use data to sense, listen (adjust) and then act. These people are data scientists. Generally, data scientists are seen as people who can wrangle large data sets. But data scientists can also be designers, people who use data as their art to create a different, more improved world. And yet, like most designers, data scientists often find it can be difficult to know where to start, because without context, millions of data bytes sit dormant unable to be rendered useful. That’s where ethnographic research can illuminate pathways forward.

For this paper let’s define ethnographic research, in the context of design research. In this context, ethnographic research is the study of how people live their lives in order to better understand their behavior, motivations, needs and aspirational wants to inspire new design. This approach is comprised of various methods including interviews, observations, role-playing games and journey mapping. This paper will show that using ethnographic research at the very beginning of the project, transformed the data scientist’s initial assumptions about how to use data to solve the problem.

METHODOLOGY

To better prepare the team for upcoming field research, the design researcher on the project did a literature review about the cruising industry and cruiser. Using that information, she created an empathy exercise to help the team understand the reality of people who booked cruises over the phone.

Only two out of the team’s six members had ever been on a cruise. And no team member had gone through the entire cruise booking process. Among the team, and in the broader media, the typical persona of someone who goes on cruises is a silver-haired, often retired, affluent couple looking to relax. A quick literature review of the cruise industry painted a very different picture what team members had envisioned. The average cruiser is a 49-year-old employed, married, college graduate with a six-figure income. (Cruise Lines International Association 2017) In addition, “Generation Xers,” and “Millennials,” were fast outpacing Baby Boomers as cruisers. In one study, Millennials were twice as likely to have taken a cruise than their Baby Boomer parents. (Sheivachman 2018)

Repeatedly, in articles about cruising the idea of seeing multiple locations without ever having to unpack a bag, made cruising appealing to all age groups. Exploring the data collected by the client really couldn’t get to the central question, why was booking a cruise so darn difficult in the first place? Once that question was answered, the team felt confident they could design a satisfactory solution.

IDEO conducted two large-scale projects, Project Traveler* and Project Board* (not the projects real names), in multiple phases and with multiple design teams, to diagnose, prototype and design solutions to create a better cruise booking experience. There was 27 weeks of work though the projects spanned more than a year. Only two designers remained constant throughout all phases – the data scientist and the design researcher. The design teams used iterative research and design approaches that included a variety of methods. The project essentially had two user groups: customers who were booking cruises and call center agents who were helping them to book the cruise. The complexity and multi-variant challenge to the problem required an extensive array of design and data science research methods to illuminate a path toward design solutions.

GENERAL FINDINGS

The team’s in-depth interviews quickly lead them to discover a universal truth–no one cruises alone. The complexity of the booking processes compounds this truth. Even if a person is going on the cruise solo, he or she usually has to consult someone else to make the decision. One person may be on the phone, but to make the final decision it could be more than 10 people involved. The team imagined the cruise booking process was simpler than it actually was. After conducting interviews with both first-time cruisers and those who had not been on a cruise yet but were actively looking, it became apparent that cruising ain’t easy. Take Michelle (not her real name). Michelle is a 37-year-old mother of two who lives off the Florida coast. The team interviewed her at her home just as she was in the midst of booking her next cruise. Even though she has been cruising since she was a teen-ager, Michelle says she’s still a novice when it comes to booking a cruise.

“While I’m booking [a cruise], I’m stressed,” she said softly as the team sat at her dining room table in rattan chairs. The stress of the process weighing on her face as she dictated all the decisions she had to make. “I need to figure out the date, the room type, do they have it, is it the right port that I want to go to, so it’s kind of stressful. “I’m not an expert,” she continued. “I’ve done a lot of cruising, but not so much recently. I know about the tipping procedures and what you can bring on board and what you can’t. I mean, I think I do. Who knows? It might have changed since then.”

Even though Michelle had been through the booking process repeatedly, she had been on nine cruises, she still felt like a novice. Her insight that the booking process is ever changing making it difficult for anyone rookie or not to navigate was particularly inspiring to the team. The team sat in Michelle’s living room, outfitted with rattan chairs and a beautiful glass table, covered with a white sheet to protect its shine, as she made a phone call to a cruise line. She said she wanted to call two different cruise lines to get pricing and availability for her next cruise. On the day the team went to interview Michelle, she called the client’s cruise line. (Incidentally, researchers knew about the call before and asked Michelle’s permission to listen to it. It was totally coincidental that Michelle called the client’s cruise line.) The call was painful. Cruise line stakeholders visibly cringed as they listened to the Michelle struggle to get her questions answered. She had to repeat her destination desires multiple times and never really did get the answers she was seeking. Shortly after that call, she called another cruise line, the client’s competition. The call went better, Michelle thought, but she was still unsatisfied.

“I feel okay. I don’t feel fantastic with either of them. I guess I feel better from this call. The other guy seemed just kind of like, you know, wanting to talk about his own trip. … But both of them, I don’t have all my questions answered, so…” and her voice trailed off in soft disappointment.

Ethnographic Methods Makes Intelligent World Understandable

Walking into the bright, spacious building of the cruise line’s call center was akin to jumping on a merry-go-round going full tilt. Dozens, and dozens of call center agents sat row after row, some standing, others leaning on their chairs some bouncing even, as they cheerily but assertively, talked to customers about booking cruises. There were shouts, and bells ringing when cabins were booked and sold. Leaderboards adorned the entire back wall as names of top sales agents beckoned the competition to come and get them. Sales coaches and managers paced the floors walking up and down the cubicle aisles giving advice. It was a call center, filled with extroverts, deal-making and fun, LOTS of fun. Timing and scheduling forced the team to be on site at the call center for just three days. Yet, that was enough time for center’s infectious energy to engulf the team. Being on-site at the call center allowed the team to see the company’s culture, understand employee incentives and see where service played a part in the sales journey. Listening to frontline agents take and make calls to customers was, by far, one of the most lucrative research activity the team did. The phone calls told a truth that was buried, hidden and obscured by disparate systems, human nature and a lack of transparency. After a couple of hours of listening to phone calls; it was clear that calling into a cruise line’s 1-800 number was an adventure. But not a fun one. It didn’t take long listening to agent’s phone calls with real customers to hear echoes of the frustrations expressed by cruisers during our earlier field research phase.

Team members noticed that the calls weren’t great for agents either. When the customer called in often agents didn’t know anything about the customer, beyond name, address and phone number. Agents would have no clue the customer called in yesterday asking questions about a Caribbean cruise. They wouldn’t know that the customer was afraid of flying and booked a hotel two days in advance from their cruise date and needed directions from the nearest train station.

Ethnographic Methods Makes Data Science More Powerful

Adhering to the research plan which prompted team members to document information exchange and transfer between agents, customers and systems, members noticed agents would have to search through multiple software systems to retrieve crucial information about a customer’s history with the company. This lack of understanding about who was calling into the call center and why was a major roadblock to the customer experience. It led to long wait times as agents sifted through a half a dozen software applications to find the exact information needed to help the customer. It was also obvious that enterprise systems were a barrier to agents having information readily accessible to help customers.

During these observations, the team also discovered something that never would have been found through data exploration alone–the sheer amount of relevant customer data that was being captured by agents but not stored by the cruise line. The team observed frontline agents repeatedly recording information from customers in a notebook or on their desktops in Notepad or Word documents, only to delete or throw away that information after a call was completed.

Sometimes the information was thought to be unnecessary. Other times the agent didn’t really know what to do with the information. And sometimes the agent didn’t see the information as valuable because it didn’t lead to a sale. We also saw agents using these notebooks to “double check,” their sales. Though all sales were marked in their system when a customer booked, some agents weren’t sure they were getting credit for all their sales. So, they recorded every sale in their own notebooks. This let the team know that agents didn’t trust their systems to record data accurately. Coincidentally, many customers didn’t either.

This observation was something the data science designer found invaluable to see before crafting an intelligent model using data for a solution. Such human data exclusion and manipulations are virtually undetectable during a data science usual methodology for dissecting and analyzing data. Data scientists usually only see the output of data. They see data without context. Seeing how data is created and transformed by humans is essential to devising useful data-driven solutions for businesses. Paring what the team saw in observation with the data mapping exercise allowed the team to see opportunities for design. This map pairing allowed the team to see the system barriers agents and customers had when creating or extracting information for use. Mixing the traditional data mapping exercise with the human data interaction exercise allowed the team to actually “see,” how data began and ended in the client’s technology systems. It was also great way to see how user data was actually used by agents. Visualizing this journey was key to finding the root cause of the pain points both agents and customers felt during calls:

• Agents spent a lot of time on calls manually pulling customer information from multiple applications
• Agents struggled to build rapport with customers because personal information about customers was missing or incomplete
• Agents struggled with better visibility into customers’ past interactions with the company forcing customers to repeat information frequently
• Agents wanted to take ownership of their guest relationships but feel they had to depend upon systems they didn’t wholly trust.

After the observations, the team developed a “behavioral data map,” that allowed it to see the connective tissue the customer’s information traveled through during a phone call. The team discovered that the customer’s information was split between three main information systems and it took considerable amount of effort and system know-how to connect them to give a coherent and contextual understanding of a customer’s need, if the agent could do that at all. The team could pinpoint design opportunities to leverage existing data to the benefit of the customer. Without the on-site observations, the data flow mapping would have given an incomplete picture to the team.

“Mapping the people to the data and the journey both take with data makes it easier to identify data priorities, redundancies and how data is transformed and changed throughout the system,” said Ann*, the data scientist on the project. “This gives data scientists the edge they need to focus on the information that really matters and find opportunities to augment human intelligence, instead of just taking a bunch of data without context and plugging it into some new technology model and seeing what comes out.”

Frontline inbound sales agents, who were the agents that answered the cruise line’s 1-800 number were often the ones who recorded the data. It was incredible the amount of personal details these agents were capturing on customers that never ended up in the cruise line’s databases. Information gathered on calls where a customer did not actually book a cruise was just “lost,” preventing other agents from “picking up where another agent left off,” when a customer called back.

“For a company that is very focused on conversion and selling quickly for them to have an agent to capture that information and take the time out to capture that information is huge,” Cheryl*, the project lead said. “As designers, we need to show them the value, to show them how much that information is worth and why it’s worth collecting.”

Using Empathy for Paradigm Shifts

Because the client collected a lot of data about its current and prospective customers–basically anyone who contacts them through their call center or website–the team determined that leveraging that data that lay dormant and often inaccessible to agents could go a long way in shoring up some of their customer service issues. But just like there was a separation of sales and service agents, there was a separation of sales and service data. This left the data the company collected on consumers as fragmented as the service experience.

Still one truth remained–ethnographic research revealed that leveraging data alone wouldn’t solve most of the customer service problems. Optimizing the company’s ability to use data without changes to the culture and organizational structure wouldn’t give agents the freedom and incentive to use that data to serve customer. Infusing agents who were focused just on sales and not service with customer data might benefit the agents, but it would do nothing to improve the customer experience. But how to tell a client expecting some tweaks to a script or a series of new training modules that they needed to change their entire employee organizational structure before they could give their information data systems a modern facelift to serve customers better? Not an easy task. But the team decided that building empathy for the customer’s pain would give the client a sense of urgency that made them conducive to large-scale change.

After returning from the research trip in the field the team holed up in a project space to process what they learned. The team had just a few weeks to synthesize what we had learned. Plastering quotes and notes from our field debriefing sessions the team holed up on their project space to find themes and patterns. In addition, the team’s researcher branched off to read transcripts and listen to audio from various observation sessions the team witnessed. This dissection of transcripts lead to the “Anatomy of the Call,” and exercise that broke down customer calls to agents, section by section marking information and data used during those called. This allowed the team to see what data was most frequently requested and used during phone calls. Subsequent interviews with agents and data analysts and software engineers confirmed the team’s hunches on information most used by agents and most requested by customers. In addition to synthesizing data capture, the team also worked through patterns found in interviews from customers and listening in on calls. Clustering quotes and deconstructing their meaning, the team centered around four actionable insights.

• People want support that acknowledges who they are and adapts to where they are in the journey.
• People seek confidence in their decisions, and rushing to commit before they’re ready makes them feel insecure.
• People are delighted by a streamlined process and deflated by repetition.
• People naturally collaborate during the travel planning process, but they feel overwhelmed by managing this task

The team felt the insights lead to good design opportunities and was excited to share these with the client. But team members knew it would be difficult for the client to act upon the insights without winning the hearts and mind argument. The idea of sales and service being separate was entrenched in the client’s culture. The team had to figure out a way to get the clients to see how disruptive this split was to the customer experience. The stakeholders needed to see their roles and the roles of the agents within their organizations differently, from the customer’s view. Sharing what customers said was one way, but the design researcher and the team wanted their clients to actually feel the barriers that guest felt when accessing their call center. So in true design research form, the design team designed a series of empathy exercises that put the client core team through a one-hour timed game that simulated the way their customers felt. In this newly stylized version of television gameshow “The $10,000 Pyramid,” players were positioned back-to-back and were asked to guess words based upon clues given by their partner.

One partner could see the word and had to give clues to the other partner, while not saying the actual word. On average, there were three to six words to guess. Like a phone call to a call center, partners couldn’t see each other. Each round got harder as the rules changed. The “rules,” mimicked many of the barriers customers encountered when calling the call center, the communication constraints (can only give three clues), the system constraints (players couldn’t skip words) and, of course, time constraints. Players were rushed to make decisions. The giggles were equally matched by groans as player after player failed to win. Older players used clues that younger players didn’t understand. “Who is Mick Jagger?” One player went down in flames because he couldn’t think of the word “Limbo,” though he kept slinking down in his chair as if sliding under a phantom pole. As the words got increasingly abstract the players sounded like they were speaking foreign languages to each other. Only one pair guessed all the words in the allotted time. It was a fun, but poignant way to have them walk in their customer’s shoes. Through the exercise the executives felt much like their customers did on the phone, frustrated, misunderstood and ignored. It was a powerful illustration of just how frustrating the entire booking process could be. And it exhibited the true uniqueness of design research. The client had collected raw data about customer satisfaction. But having to go through an exercise where they were treated like their customers helped them to understand the feelings behind the data. From one executive after the meeting,

“I feel like the shackle has fallen from my eyes. No matter what becomes of this project, I know that I will not think the way I did in the past again. This process has changed my whole outlook.” – Cruise Executive

This meeting was a turning point for the project. It marked the point where the client core team fully came on board. The sincere desire to change exhibited by executive leadership at that meeting gave the team license to expand design opportunities in new and innovative ways. It allowed the team to take some bold design risks including live prototyping a new organizational structure that teamed up sales and service agents to take calls together. But by far the most exciting change the client glommed onto was an opportunity to leverage existing data to make its call center agents more knowledgeable and responsive to customers’ need in a quick and efficient way. Though the one design recommendation the client wanted to implement right away was a data-driven solution, it was the tried-and-true ethnographic method of empathy-building that created the opportunity for this bold new design.

LESSONS LEARNED

Convinced in the new opportunities presented, the client embraced the expansive design solutions proposed. The team landed on four design recommendations to help the company improve their customer experience.

Three of them focused on organizational and communications changes. For example, the team’s research suggested that the way inbound agents were incentivized was a huge barrier to the customer experience. So, a design recommendation devised a different way to organize the two sales divisions and one service department to better meet customers’ needs when they call in. However, there was one design recommendation that sprang solely from collaboration with a data scientist. This design concept, created to envision a new way to organize and surface customer information to agents was designed to make customers feel heard, understood and served and is called “The Board.

Through design research, the team had mapped three parallel journeys – the customer’s, data and systems. Using text from customer phone calls, the team matched all the information that customers talked about during a phone call to the software or enterprise system where that data was retrieved or stored. The team did this to see if there was a better and quicker way to surface this information during phone calls to help eliminate the frustrations customers felt during calls.

DATA AND ETHNOGRAPHY SHARE EQUAL BILLING IN PROJECT OUTCOMES

It took about eight months and another project to bring the “Dashboard,” into fruition, but the seed of this new software application was planted a year before in a small project room. That’s when a design researcher, a data scientist, an environments designer, a communication designer, an organizational designer and a business designer all poured over transcripts of customer calls to the call center.

They all felt the pain of both agents and customers with the current systems and processes, and a data scientist saw that same pain and offered a way to leverage data to help make the process better. And an entire design team worked together to create an innovative software product that the client could implement into existing systems to get customer service value right away. There’s no way to know of course if the results would have been as good without the mix of a multidisciplinary team. But it’s a certainty that it took both ethnographic research and data science to pull off the right solution.

“That’s where the design research comes in,” the data scientist said. “We got an accurate deep learning of what people need today and what the pie in the sky was for tomorrow and we worked backward from there. We created the first version given the data limitations they have today but we created it in such a way they can get further and more sophisticated in their data use over time. We created the foundation for an intelligent dashboard. And it’s not just a dashboard that surfaces basic customer information, it’s collecting that information such that when the company is ready to invest in something that’s more intelligent, that uses machine learning for example, they have the foundation to do so.”

Project Board is an example of pairing the highly effective skill of call center agents with the processing power of software programs to get a better result than either alone. For this reason alone, not the data that is integral to the design, but the people who will make use of this surfaced information, Project Board is a good example of the future of human-centered AI. It’s not about the data. It’s about the people, and how surfacing the data made people better at their job, happier doing their job and customers happier when interacting with customer service agents.

Ovetta Sampson is a design research lead at IDEO in Chicago where she uses a variety of qualitative and quantitative methods to inspire innovation in various industries including IoT, transportation, and service. She works with clients to visualize ways to ethically and with compassion serve people through digital and intelligent products. She also an adjunct professor in the School of Design at DePaul University in Chicago. Contact her at ovetta@ovetta-sampson.com.

NOTES

The paper’s author would like to acknowledge the many people who helped make this publication possible. This includes the leader of the project and data scientists who worked on both projects, both of whom are not named at the request of the author’s employer. In addition, the author would like to acknowledge the many data scientist and design research colleagues at IDEO and at DePaul University who helped her highlight the most illuminating and innovative parts of this very complex and unique project. This includes Jess Freaner, Dan Perkel, Lisa Nash, and many others. Special thanks to EPIC Mentor Ryoko Imani for your selfless commitment to this first-time EPIC author and your endless patience answering all of my incessant questions.

The information included is in this paper is accurate but does not represent the official case study about the project from the author’s employer IDEO.