Sustaining Stories: The Versatile Life of Sustained, In-house, Ethnographic Practice in a Global Software Company

Vignette I. Make the Unseen Visible, but Losing it from Sight: The Woeful Pie Chart

After having expressed the wish to bring her anthropological training to bear on her work at SAP, Natalie was given the opportunity to conduct a research study as a ‘proof-of-concept’ for what might be possible. That study has come to mark the beginning of the User Experience function in SAP’s Business Operations group. At the time, the operations function in the U.S. was focused almost exclusively on the sales line of business. The team’s charter was to increase the productivity of sales people, as measured by (among other things) license sales revenue per sales representative. It is therefore not surprising that this first research project involved shadowing revenue-generating employees. The research was justified to an executive team on the basis of the opportunities it brought to understanding sales activities, specifically looking for opportunities to increase their productivity.

At that moment, Natalie was the sole researcher who handled almost every aspect of the study– recruiting, scheduling, capturing field notes, data analysis, and the reporting of results. The final deliverables included a presentation that was given to the Senior Vice President (SVP) of Business Operations (the project sponsor) with a list of possible action items, including business scenarios illustrated with quotes from study participants, and a series of recommendations. This type of work is perhaps the standard of ethnographic practice at the service of business strategy in which the “unseen” of work practices is reconstructed and made accessible to decision makers who might be unaware or distant from it. Messaging such findings involves the creation of reports, diagrams and other artifacts that attempt to serve as boundary objects (Star & Griesemer, 1989) between recipients and researchers.

In this case, the presentation materials produced included a slide that attempted to visualize the frequency with which certain key activities occurred. It was a pie chart based on the coded data, intended to provide visual impact and designed as an anchor point for discussion with executives who might not tolerate the narrative detail in the findings. When the study results were presented to the Senior Vice President of Business Operations, this pie chart was used in addition to informal, verbal findings; the complete presentation materials were delivered later to an entire management team.

At the time, this pie chart in particular appeared to be extremely effective in stimulating dialog with the sponsor and others about the complex stories and data behind it. In fact, it effectively enabled messaging of complex findings all the way to the executive level, a success rarely experienced for this kind of work. Without it, the research and its outcomes may have not had the same lasting success. At this point, it appears to be the only artifact from the research that is still circulating. To the best of our knowledge, none of the detailed business scenarios (or even the sales people screen shots) have been distributed further, despite those being, from our point of view, the most interesting and valuable aspects of that research. We have come to call it the “woeful” pie chart, the one which has been used and re-used the most, and in many cases, unfortunately, misused and surely misinterpreted. This might be an inescapable fact of ethnographic practice. Or perhaps the artifact itself, with its simplified and attractive appearance, affords the twisting and positioning to suit the needs of the speaker.

As professional practitioners and as members of our particular organization, we are still learning and re-learning that it is the contextual interpretation and ongoing analysis that makes field data useful—the nuances are not made visible except in situated conversations. Partly in response to this, we have made some changes in how we handle our reports today. For example, we insist on providing readouts before we distribute the soft copy of a report, and we provide private readouts to help interpret or expand on key topics. We hope that this approach will help our audience understand the richness and complexity of the findings, so that they will come back to us again and again, rather than assuming all the data they need is in the final deliverables. What we have come to learn from this experience illustrates the subtle ways in which we adapt our professional practices to suit the contexts in which we operate.

There had been a few other studies conducted during a similar time frame, carried out mostly by outside vendors. The first had yielded very general findings, and had not provided any significant new insights that could be used to drive productivity improvements. The second study had been conducted by a usability testing firm, and its results had limited value to the operations management team because the outcomes were narrowly focused largely on ways to improve the intranet. At the time, the research Natalie conducted had succeeded in looking for ‘white space’ (opportunities to improve productivity that might not have been uncovered otherwise) and that the research approach and final deliverables permitted the management team to more deeply perceive the problem areas, the frequency with which they were occurring, and to begin to understand the real impact from a salesperson’s point of view. The research also made visible problem areas that were known but were previously not well understood. The detailed findings and recommendations permitted a level of visibility on a core process that had previously not existed, and specifically showed the impact of those process breakdowns on sales productivity.

Overall, this initial work was perceived as innovative and adding value, and it contributed to the increased understanding of the management team of how anthropology and ethnographic practices could be blended with work in business operations. This initial success began to make ethnographic research methods visible within the organization, and opened the door for future engagements. However, as a junior member of the operations team, Natalie had very little opportunity to drive uptake of the findings, and almost no visibility into what was done to address the issues uncovered. Lack of direct access to members of the executive team, lack of influence in general, and lack of resources severely limited what could be done to extend and act on the findings. At least in part, some of these challenges related from “losing sight” of research findings are derived from the position that ethnographers might occupy in the organizational landscape, and the visibility and access that such position might afford them. At the time of our study, we didn’t have enough visibility and access to the corporate strategy and direction, which in turn made it challenging for us to message the findings in ways that would be compelling to senior management. Several more years and projects had to pass before a team dedicated to similar work was constituted and our work better positioned in a way that provided us visibility and the means to manage uptake.

In addition to these challenges, we also face the problem of being disconnected from the results of our work. We know that our findings have been used to build a number of business cases, but for the most part we have learned about them after the fact. Not too long ago, we learned that the COO was speaking with enthusiasm about a pie chart that showed where sales representatives were spending their time. A member of the leadership had to explain to the COO that the person who had done that research worked in their group. On one hand, it is fantastic to see that the research has had such lasting value; because so little has changed since for the salespeople, so most of the findings remain quite valid. One of the interesting things here is that the company has changed, sponsors and stakeholders have changed, but much of the day to day work that was originally the subject of study has not changed all that much in the intervening years. What is bothersome is that the team is not being recognized or acknowledged. As individual practitioners gain recognition and teams dedicated to similar practices emerge, it is common that resources need to be justified on a regular basis; having our work mis-interpreted (or not getting recognized for our work) can present a significant long-term risk for resource justification. Therefore, in such situations one has to be even more careful to ensure that proper work recognition is assured.

However, at that time, the entire concept of User Experience was still being proven. As such, the team’s primary charter was in technical realms, for example bridging between the business and IT or standard development. These factors influenced the way we thought of the solution space and its presentation – i.e. many of the recommendations were of technical nature and presented in that way. Some of the findings were in fact used to design and implement a small internal software application to support important but previously unidentified concerns of sales personnel. However, without significant resources at our disposal, it was extremely challenging to demonstrate the ways in which technical recommendations could have larger business impact. Perhaps most importantly, having a sole researcher on the project made it extremely difficult to manage the presentation of outcomes in ways that would be simple and compelling enough to engage an executive audience and sustain the visibility of the work. In hindsight, it is fortunate that our work was able to have any impact at all, even if in some cases such impact was not completely aligned with our intentions as researchers.

Vignette II. Projecting retrospective inquiry as relevant to the present and to a set of envisioned futures

Despite the troubled visibility of some of the artifacts produced from the initial study, it cannot be denied that eventually that research and other changes played a significant role in making our expertise visible and opening opportunities for further ethnographic work. A few years later, the regional Chief Information Officer came to us to find out what further research, if any, had been done on sales personnel. His interest was around the ways that the existing Customer Relationship Management (CRM) software was used by sales personnel to keep track of activities with their customers and prospects, communicate projected revenue by quarter, etc. Although we had conducted numerous user research studies within the sales and marketing organization (a few of them using ethnographic methods) none of them directly answered his questions. However, exploring the relevance of prior insights in new situations has become a way of opening up new opportunities for organizational contributions and, naturally, we were not going to miss an opportunity to present our work to the CIO! As a result, research findings from multiple prior studies were used to prepare a summary of what had been learned about the CRM implementation. In a personal meeting with the CIO, a general overview presentation of all the research projects was provided and then, through the course of the discussions, specific topic areas and supporting materials were reviewed based on what appeared most relevant for the questions at hand. While the overview presentation provided the framework for discussion, however, the majority of the data and the rich stories were exchanged during that meeting were anecdotal, drawn from memory and reconstructed based on the questions the executive was asking.

As a result of this “retrospective” presentation and the dialog that ensued, we had a chance to learn about a new program intended to bring improvements to the internal implementation of CRM. As it turns out, many of the areas that had been identified in the original ethnographic research on salespeople continued to be a challenge several years later. Most importantly, through our retrospective review, we had succeeded in making the sales point-of-view visible for the CIO and his project team. Inclusion of the users’ perspective at the outset of the project would be critical to ensure uptake on the improvements that were planned for the system. Some of our prior research, for instance, had shown that the sales people actually spent very little time online in front of a computer; they were hyper-connected, but it was largely via a Blackberry or mobile phone, and not using a computer browser. Even when sitting at their desks (which were equipped with land lines), they more often than not opted to use their cell phones. Of the time they spent in front of their computer, most of it was spent looking at market news and trends through a personalized service like Yahoo! Finance, and only a very short amount of time each week was spent in the CRM application itself. One insight we were able to bring to the conversation at the time was that the sales representative perceived the tool as a vehicle for management reporting, and therefore only maintained the fields that they knew would end up on an executive report, or that impacted how and how much they were being paid for the software licenses they sold.

Using historical information, we were able to present the system user’s point of view in a way that was unique inside SAP. Prospectively, we were able to position not only our expertise but the ethnographic approach as strategic and useful. After years of practice talking about the findings, we were finally confident that we were both sought for and able to speak to that point of view in a way that was compelling at the executive level. It was more than two years after that initial study, but it was the first time that we had been approached by a member of the executive team to provide insights on users at the outset of the project. Even more importantly, the research was now being used to enable strategy, prioritization, and funding decisions, and not simply system usability.

Vignette III. About Design: Prospective but Partially Blinded

A recent team engagement represents a clear example of how our trajectory of work has opened up significant avenues for key contributions of our ethnographic work as well as positioned us closer to the challenging area of organizational design. Motivated by cross-company tensions about the value being derived from a particular operations service offering and rather than continuing to point fingers, two former project sponsors recommended our services to another executive. We were asked to explore and analyze the field’s point of view on the service in question through a series of in-depth interviews.

As we were working on the interview protocol, we gained familiarity with the various parties involved. Recognizing the seniority of the individuals involved (and the tensions), the involvement of senior members of our team became essential. Through interactions with the various stakeholders, we learned that there were at least three distinct research agendas or objectives at play, and that – in order to ensure maximum uptake of our findings – we should do our best to accommodate all of them in the methods, the protocol, the findings report, and in the verbal delivery of findings. This is not an uncommon situation in which organizational ethnographers find themselves but little in our collective knowledge seems to speak to ways of managing such a challenge.

The main project sponsor appeared strongly committed to the ethnographic approach, very interested in hearing the field’s point of view. He enlisted our support to understand the issues, and identify areas where his global organization needed to change. The secondary sponsor was a regional executive who was interested in the same issues, but who also explicitly stated his interest in proposing a new regional organizational model based, perhaps, on his expectations of the findings. Finally, a third sponsor (to whom the first directly reported), was interested in ensuring that his organization remained aligned to the field’s priorities, thereby ensuring it’s ongoing relevance to the corporation. Identifying such diverse perspectives in the course of research planning helped in selecting the right approach to the design of the interviews but, naturally, some compromises were necessary. Thanks to prior research efforts and personal work of some team members as part of a regional sales operations group, we already had a fair amount of visibility into the challenges existing with the service in question. Specifically, it was clear that most of the formal processes were not well perceived in the field because they were slow and ineffective, and that as a result, personal networks and informal processes prevailed as a way to get the tasks completed. Considering all of this, we were able to formulate a protocol that pursued organizational issues directly. Rather than asking ‘how are these [formal] processes working for you?’, we asked more open-ended questions such as ‘how are you getting this task done?’ The process of defining the protocol itself involved extensive collaboration with some of the sponsors to validate and explain our approach.

The findings were not completely surprising and they did identify numerous areas of improvement. As a neutral third party were able to provide feedback to the stakeholders involved, well-grounded in data from their internal customers. In our professional commitment to make the context of our studies as visible and accessible to our sponsors as possible, we transcribe and code all of our interviews and create reports that are heavily enriched with direct quotes, making it quite explicit that we are presenting on behalf of the end-users. We take special precautions to make sure that the identity of our research participants does not get compromised in order to comply both to our ethical commitments as well as with the labor laws of the countries and regions in which we operate. As with our prior experiences, instead of disseminating reports indiscriminately we were especially careful to manage the issues uncovered as a dialog with all of the direct and indirect sponsors. This helps us to ensure that the data is made visible only to those that have requested the research. Not only does this practice ameliorate the possibility of misuse or misinterpretation of our findings but it establishes a strong partnership of collaboration with our sponsors where the interpretive sense-making is perceived as a shared responsibility. When we are successful in collaborating in this way, it helps to ensure that we remain engaged as experts on the data and its interpretation. In that way, we can help guide the development of an actionable plan that truly reflects the findings.

This type of relationship is especially key when we leave the problem space and attempt to reason through a possible set of solutions.

In this particular case, we were especially cautious about making explicit recommendations in the research report as we typically do for research on, for instance, information system design and user interfaces. When research is done in the service of organizational design, we have found that we can never put enough information into a written report. Even with a rigorous discovery process, the political landscape or objectives shift in the course of the research. As a result, we are always somewhat blind to how the research may be used, and whom it will impact. In other words, we have learned that trying to manage the future meaning of our work represents a significant challenge that often results in unintended and unplanned uses for the artifacts we produce. We try to keep our written reports tightly focused on the data. As much as possible, we then ask for a seat at the table to help with the interpretation of findings, and how they can be applied to solve the business challenges at hand. This type of conversation allows us not only to manage the way our inquiry serves the organizational needs at hand but also increases the visibility of our approach and the overall understanding of our potential roles in the future.