Ethnographic Agency in a Data Driven World

INTRODUCTION: A SOCIOCULTURAL ANTHROPOLOGIST AMONGST THE ENGINEERS

The world of modern technology companies, similar to other contemporary business environments, is full of numbers. These environments are made up of and thrive on data, which takes the form of measurements collected by apps and phones, survey results, metrics, monetary revenue, and key performance indicators (KPIs). In data-driven business environments, ethnographers often work alongside teams of engineers and data scientists who communicate with statistics, or in organizations where large-scale patterns drive decision making. Doing ethnography in these contexts means, increasingly, getting familiar with quantitative data¹ (Knox and Nafus 2018).

And yet, ethnographers often lack a fluency and familiarity with quantitative data, which can limit their engagement with numbers and the social world in which they a pervasive. In the academic world, social anthropological training—in contrast to other social science disciplines like economics or psychology—often includes little to no exposure to statistics or programming languages. In business environments, ethnographers face large barriers to learning quantitative skills like statistics and coding. Researchers in the already liminal field of user experience research must produce results on fast timelines, and may not be able to justify the time and space needed to learn new quantitative skills. As a result, ethnographers often have less familiarity with quantitative data. They may lack the skills to analyze quantitative data or write code. They may struggle to interpret quantitative data in the form of database tables, statistical models, or charts and graphs. Or, they might not have the deep understanding of data that is required to use quantitative data and insights to drive strategic organizational change.

This distance from data is further evidenced in the anthropological and social studies of science (STS) literature on quantitative data, which has tended to focus on its end products—papers and visualizations, societal implications (Miller 2015; Dougherty 2015)—instead of the everyday practices that go into negotiating and making sense of quantitative data (Levin 2014a; Starosielski 2015; Buur, Mosleh, and FYHN 2018). This has left ethnographers, in both industry and academic settings, with few playbooks for engaging critically with quantitative data, or for doing “participant observation” on the practices that shape how quantitative data are made and used.

This paper takes as its problematic the author’s three years of experience working in a large, data-intensive organization, where ethnographers often struggle to gain traction with their work because of an inherent bias towards quantification and numbers (Maiers 2018). By providing an autoethnography of my experiences learning quantitative methods (Jones, Adams, and Ellis 2016), I argue that quantitative literacy—a deep familiarity with data, as well as knowledge of how to use quantitative skills to shape an organization—can give ethnographers expanded agency² to do research in and impact data-intensive businesses. To be effective industry researchers, ethnographers must not only deliver insights into their subject area, but must also understand and move within the larger systems in which these insights operate (Cefkin 2010). Fluency with coding and statistics can give ethnographers increased independence and authority within organizations, by enabling them to speak the same language as their stakeholders, and by helping them more clearly articulating how quantitative and qualitative research can complement each other. This ability to not only work, but also tap into the social power of numbers, becomes a powerful tool for gaining ethnographic agency.

However, quantitative literacy not only gives ethnographers new agency, but can also enable them to imbue quantitative data itself with new forms of agency and meaning. Quantitative skills can empower ethnographers to ask crucial questions about how data is used to make decisions in organizations. This can, for example, highlight crucial gaps in datasets, making space to question whether business outcomes are measured with the correct metrics. Quantitative literacy can give ethnographers the agency not just to critique institutions, but also to change them from within. With a greater understanding of how quantitative data is made and used, ethnographers can ensure that data is collected in representative ways, point out the limitations of existing metrics, and argue for new ways of measuring and understanding social life. With quantitative literacy, anthropologists can gain the tools to re-negotiate and restructure the quantitative environment around them, by changing the processes through which data shape and have power in the world.

And yet, it might not always be possible, necessary, or ideal for researchers trained in ethnography to pursue quantitative literacy. If researchers do not have the skills, opportunity, or desire to develop quantitative literacy, does their impact suffer? Does this emphasis on quantitative literacy create false expectations that everyone can and should be able to tackle both the quantitative and the qualitative? Do qualitative approaches become relegated to the less influential projects and parts of the company? Might promoting a more quantitative way of viewing users and technology make it unintentionally harder in the long run to get buy-in for qualitative approaches and insights?

Ultimately, this paper asks, how can we better equip ethnographers to have more agency as they enter this world of quantitative data, statistics, and algorithms? What are the best approaches and theories to help ethnographers work and succeed in data-intensive environments? How can the agency that comes with quantitative literacy help ethnographers have more impact, by enabling them question and influence the structures and values surrounding data-driven decision making? And ultimately, what does this need to have quantitative literacy in order to gain a “voice” within technology companies say about the current culture of user experience research, or the current culture of knowledge and values within data-driven organizations?

PART 1: GAINING AGENCY BY LEARNING THE LANGUAGE AND SKILLS OF QUANTITATIVE DATA

In this first section of the paper, I reflect on my experiences becoming fluent in creating, transforming, understanding, and using large datasets at a well-known tech company. I describe the process of carrying out a large-scale survey project, where I learned how different sampling methods impacted survey data, wrote code to analyze this data with statistics, and told stories about this data to stakeholders to influence company strategy. Through my work, I came to do participant observation with quantitative data not in a cursory, surface-level way—as can happen through reading papers or relying on second-hand accounts in interviews—but by deeply engaging in the creation, analysis, and socialization of quantitative data. Drawing on this experience, I reflect on how quantitative literacy can help ethnographers have more autonomy and influence in an organization, and can simultaneously uplift the status of qualitative research methods and insights in data-driven environments.

For my PhD, I did an ethnography of how scientists worked with large datasets and statistics. I began my research at a laboratory at Imperial College London thinking that I would study academic-industry collaborations, and ended up focusing on how scientists were trying to understand the complex system of metabolism with multivariate statistics (Levin 2014a). I carried out participant observation with scientists in the field of “metabolomics,” as they attempted to understand the role that metabolism played in enabling living beings to interact with their environment over time. I watched scientists and clinicians put samples of urine, blood, and tissue into mass spectrometry and nuclear magnetic resonance machines, and also observed how they analyze the ensuing datasets—which contained hundreds of thousands of data points, and could be several gigabytes large—with statistics and algorithms.

Ultimately, these statistics and algorithms became the focus of my participant observation with scientists. Although ethnography is typically thought of as a distinctly qualitative methodology, anthropology, it turns out, has always had a relationship to numbers (Curran 2013). Adam Kuper writes, for example, that early British anthropology had an “overriding concern with the accumulation of data” (Kuper 1977, 5), and that Malinowksi looked to collect “statistical documentation through concrete evidence” as part of his ethnographies (Kuper 1977, 15).

Consequently, as I carried out participant observation with scientists who were using complex, black-boxed machine learning algorithms (Eubanks 2018), I came up with particular ways to engage with these data practices. I shadowed researchers as they did lab experiments and analyzed data in MATLAB and other statistical software. I pored over scientific literature and attended training courses and seminars. Even though I had no formal training in statistics, I learned to “speak the language of data” by familiarizing myself with the theory behind principal components analysis, supervised learning techniques, and neural networks (Levin 2014a). By gaining a deeper understanding of virtual and intangible data-rich systems, I was able to reflect on how quantitative data was reshaping concepts like metabolism and health, creating friction between scientists and clinicians (Levin 2014b), and shaping notions of “persons” and “populations” in healthcare systems.

Although I spent much of my PhD thinking and writing about how data was impacting society (Levin 2018), in academia, I did not need to learn how to actually do data analysis. To write an ethnography about data and statistics, it was enough to shadow scientists, to understand the theory behind statistics, and then to talk to participants about how data was impacting their understandings of metabolism and health. My success as an academic was in no way tied to my ability to do quantitative work, and as such, I had no incentive or reason to learn to do data analysis.

In my industry job, however, I found myself in an environment where having quantitative skills seemed to unlock a number of doors and opportunities. Before I began working in the tech industry, newspapers and magazines seemed to portray ethnography in business world as an almost mystical tool for unearthing consumer insights (Wood 2013; Singer 2014) or for enacting organizational change (Huhman 2018). In my own experiences, however, most of my colleagues—other user researchers included—did not understand the nature or value of qualitative data. They did not understand which business questions would best benefit from ethnographic inquiries, or that qualitative insights were never meant to be “representative” (Maiers 2018). As a result, business decisions were still largely driven by quantitative insights from surveys and by behavioral shifts seen through the lens of log data.

In my everyday world, qualitative data was often seen as a storytelling tool, and little more. “Qual,” as it was colloquially called, was relegated to the “human” or “ethical” dimension of big data, rather than existing as an equal form of data in and of itself (Arora et al. 2018). I often advocated that qualitative data could be used to come up with new product directions, or to develop principles and values for product design. But because I was not producing data that would neatly fit with existing metrics—in the format of a survey that said “4% of people thought X with product Y”—many of my stakeholders did not know how to operationalize my qualitative insights.

After about a year in my job, as a mostly qualitative researcher whose interaction with data was limited to an occasional analysis of survey data in Microsoft Excel, I realized that my lack of quantitative expertise was preventing me from engaging in strategic conversations in the company. I struggled to engage with data scientists and engineers, as they talked about the numeric results of A/B experiments. I also found it challenging to engage with other user researchers who came from more quantitative social psychology backgrounds, as they talked about complex survey analysis in the programming language R, with which I had no familiarity.

I found myself caught in a double bind. I wanted to advocate for qualitative methods within the organization, but I realized that I needed to become more adept with quantitative methods to do so. On the one hand, I wanted to grow my identity within the organization as an ethnographer and anthropologist, but on the other hand, I wanted to have access to new projects, relationships with stakeholders, and forms of impact. I started to wonder if a deeper understanding of the ways that quantitative data was being created, manipulated, and used would help me understand and influence the data-driven organization in which I worked.

Working in a fast-paced company, I did not have the luxury of continuing my doctoral research, by doing extensive interviews or participant observation as the primary method of becoming more fluent with quantitative data. Instead, I needed a more practical approach. I began to learn how to conduct more complicated survey projects, as a way to develop a greater familiarity and understanding of quantitative data, and also as a way to strategically advance my position within the organization. But as I started to improve my knowledge of survey design and sampling, I still had to rely on data scientists and quantitative researchers for help with querying databases, or with figuring out the right statistical tests to use on my data.

At the time, I was still using Excel to analyze my data. I had learned this computer program during my undergraduate studies, as I had taken classes in biology or chemistry that only required simple data manipulation like adding or calculating p values. But with survey data, I began running into an increasing number of problems with data manipulation, particularly as the datasets became bigger and more complex. Each time I analyzed a subset of my survey data or carried out a new type of analysis, I had to create a new tab within an Excel workbook. This led to a proliferation of tabs and hand-coded calculations, leading to issues with version control and mistakes with calculations.

Observing that the “expert” quantitative researchers throughout the company were analyzing their data in R, an open source statistical software package, I decided to follow in their footsteps. To learn R, which has a notoriously complex syntax (Machlis 2017), I took advantage of the several in-person and online training courses that my company offered. But learning R syntax abstractly, without concrete datasets to solve for, was challenging. As a result, I designed a moderately complicated survey, and started practicing data analysis with my own datasets. Knowing that I wanted to answer specific questions about the data, I was able to translate my working process in Excel into R, by looking at “R Cheat Sheets” (https://www.rstudio.com/resources/cheatsheets/), drawing out visual diagrams for how the data should be manipulated, and by debugging issues on the website Stack Exchange (https://stackexchange.com/).

As I engaged more deeply with quantitative methods than I had during my PhD, I went through a process of becoming fluent not just with the language, but also in the skills of quantitative data. Beyond a high-level, theoretical understanding of statistics, I learned how to conduct representative sampling with large surveys, how to effectively structure survey questions to control for response bias, how to join survey data to other data in our databases, how to write code to analyze data quickly and efficiently, and how to tell stories with numbers through graphs and other visuals. As I became more fluent in the skills needed to manipulate quantitative data, I also developed a greater understanding of the types of research questions that would best benefit from qualitative versus quantitative approaches, as well as how the two methods could be combined to drive the greatest impact. I could more clearly identify when research—take for example a study to understand which strategies people were using to learn new things—would benefit from a survey rather than a qualitative study.

As a culmination of my efforts to learn more about quantitative data, I completed a large survey project, which delivered a number of insights that shaped company strategy. I used stratified sampling and weighting—key strategies for minimizing bias—to illuminate the complexities of the product’s user base. This caused stakeholders to question their assumptions about how and by whom the product was being used. Instead of presenting the survey results as an average, which would have lumped the experiences of different populations into one number, I showed how the survey results varied depending on where the user lived, how old they were, and whether they used an android or apple phone. The “user,” which had formerly been an amorphous concept (Amirebrahimi 2016), was suddenly anchored in rich contextual information. As a result, my stakeholders were forced to consider how social environments shaped peoples’ interactions with technology, because there was incontrovertible quantitative evidence that gave texture and shape to a formerly “average” user.

These kinds of insights, however, would likely not have emerged had I partnered with a quantitative researcher or data scientist instead of doing the work myself. Because I was no longer reliant on others to work with data—to query databases or carry out statistical analyses—I gained the autonomy and freedom to approach data with my own unique perspective. Once I understood the common ways that researchers approached surveys, I began to question the decisions that were made about which differences in data to highlight, or about how data should be visualized. I approached my survey data with the eyes of an ethnographer, with a view towards drawing on the multitude of dimensions in the data, in order to highlight cultural, social, and regional differences.

As my quantitative work became more visible in the organization, somewhat unexpectedly, more stakeholders began to pay attention to my ethnographic work. As I paired qualitative and quantitative approaches within larger projects, the people I worked with began to understand how in-depth interviews and ethnographic insights were both valuable types of data, which were part of a larger story that could be told about a problem space. By uncovering a number of interesting trends in the survey, I had created new opportunities to do ethnographic research with specific populations. While the quantitative survey data provided the “what,” qualitative methods like ethnography helped uncover the “why,” the reasons underling the differences in the data.

Doing quantitative work ultimately became a way to elevate the status of my qualitative work throughout the organization. Because my ethnographic analysis became data-driven, my stakeholders perceived it to be more rigorous and high quality. The close relationship between quantitative and qualitative research helped to circumvent the all-too-common criticism that qualitative research lacked statistical validity or situational generalizability (Maiers 2018), Here, ethnography was not just a way to give texture to quantitative data. Instead, quantitative methods emerged as a way to give new value and life to ethnography and qualitative data itself, by leveraging “big data” to open up opportunities to explain cultural differences (Curran 2013).

My experiences with quantitative research not only led to new opportunities for qualitative research, but also transformed my role and status in the research organization. Following this survey project, I was given license to do more strategic projects—and even assumed a new role as a “pathfinding” researcher, focused on the future of the business—because I had learned to deliver insights in a shape and format that the organization recognized and understood. I was able to take on projects that were bigger in scope and spanned longer timelines, as I was no longer classified as a “qualitative researcher” and could now address complex topics using whatever method I needed. As a result, researchers in the organization began to solicit my help and advice with tackling complex problems, transforming me into a trusted thought-partner for driving company strategy.

My push to create more impact by upskilling in quantitative methods did, however, have some unintentional consequences for ethnography and ethnographers within the organization. Research leadership began to promote the hybrid quantitative-ethnography approach that I had developed as an “ideal” model for other researchers. And yet, other ethnographers who had less exposure to quantitative environments or less flexibility to pursue skill development in their free time, struggled to adopt this model. By emphasizing the intertwined nature of surveys and ethnography, I had helped to create false expectations that anyone, regardless of their background or resources, could and should gain experience in quantitative methods as a pathway to having greater impact. While quantitative literacy increased exposure to and understanding of ethnography in some ways, it also devalued the method and placed it in a more precarious situation in other ways.

In summary, this section of the paper shows how by developing quantitative literacy, ethnographers can gain the ability and agency to gain more autonomy and influence in data-driven organizations. While organizational cultures and social norms—like the dominance of quantitative data and reasoning—can create power asymmetries that make it difficult for qualitative research to have impact, ethnographers are not helpless. Just as patients with mood disorders can repurpose “constraining” technologies like in-vitro fertilization or brain imaging for their own strategic ends (Lock and Kaufert 1998; Cohn 2004), ethnographers can “hijack” quantitative methods for their own strategic ends within organizations.

Here, for example, quantitative literacy can become a strategic tool to help ethnographers gain back some of the agency that was lost when ethnography was fit into the user-experience framework (Amirebrahimi 2016). When this occurred, the focus on and language of “the user” flattened research into the binary of the user and the used, removing much of the richness of peoples’ local, social, and culturally-specific engagements with technology. While ethnography’s multifaceted engagement with culture and power is often reduced to the individual usage of a device, quantitative data can help bring context, specificity, and place to qualitative data, by showing how technology usage varies by dimensions like age, gender, and country.

Ultimately, in “expert” environments like tech companies, ethnographers can more critically and meaningfully engage with technologies like databases and algorithms by becoming fluent in the language and end-to-end processes of data. While such expert knowledge may not be necessary in the context of the quantified-self movement (Nafus 2016), a lack of expert knowledge in tech companies can preclude ethnographers from participating fully in the social life and strategic decisions of organizations. In this way, possessing certain skillsets, or not possessing others, can alter power dynamics and disrupt the so-called “big data divide” that exists internally in organizations.

PART 2: IMBUING QUANTITATIVE DATA WITH NEW AGENCY, BY SHAPING THE NORMS, VALUES, AND POLITICS OF NUMBERS

In this second section of the paper, I reflect on the process of driving impact and decision-making with a large-scale, hybrid ethnographic and quantitative project to measure the relationship between digital skills and product usage. I talk about how doing quantitative work, rather than just observing it, can give ethnographers critical insight into the politics of data in a large institution: into what is and is not being measured with quantitative data (Crawford 2013, 2016), into how the contingency of data is negotiated in decision making (Latour and Woolgar 1986), and into the ways that certain forms of data come to be valued and have power (Räsänen and Nyce 2013; Biruk 2018; Rajan and Leonelli 2013).

During my PhD, I had used my understanding of the entire lifecycle of metabolic data to develop a theoretical toolkit for approaching data practices in the laboratory. By carrying out ethnography in a data-intensive environment, which might appear off-limits or intimidating to anthropologists, I came to see how numbers were not “stable and objective measures of reality” (Biruk 2018), but had complex social lives and were embroiled in power dynamics. In this way, I began to reflect on the social aspects of how data were made, reasoned through, and used, and how these “data practices” shaped how people and societies functioned. My work with numbers gave me insight into the politics of data (Boyd and Crawford 2012), and gave me a theoretical toolkit (Levin 2018) for analyzing the claims to objectivity (Daston and Galison 2007), newness (Boellstorff and Maurer 2015), and accuracy that surround quantitative data (Gitelman 2013).

CONCLUSION: ETHNOGRAPHIC EMPOWERMENT IN DATA-INTENSIVE ENVIRONMENTS

This paper shows how quantitative literacy—the ability not just to produce and understand quantitative data, but also the ability to use and apply it strategically within organizations—can give ethnographers the ability not just to critique institutions, but also to change them from within. By restructuring who has access to and can generate narratives about data, quantitative literacy enables ethnographers to renegotiate and restructure power relations in data-driven environments. As ethnographers learn how to do data-related tasks like running surveys, interpreting metrics and models, and writing code, they can challenge the epistemic authority of other disciplines that typically produce and control narratives about data. In doing so, they can gain a seat at the decision-making table to discuss important issues and tradeoffs in company strategy.

Having more of a strategic voice within an organization, and restructuring international power relations, is a difficult undertaking. The reality is that unless ethnographers learn to speak the language of and deeply understand quantitative data, they will struggle to enact institutional change, and to shift the power and value afforded to qualitative research. Knowing a system, rather than just the users of that system, allows for changes at the level of values and norms rather than products. This kind of thinking can reveal how and why complex social problems cannot be addressed with new metrics or algorithms alone. If the “smartness” of AI lies, as Clare Elish writes, in its power to process patterns and numbers with statistics (Elish 2018), then anthropologists need to play a role in the creation and deployment of statistical systems. Anthropologists must widen their horizons to focus not just on users and designs, but also on the machine learning algorithms, data architectures, and institutional hierarchies that make up data-driven organizations.

Ultimately, it is possible that promoting the adoption of quantitative methods and skills amongst ethnographers will increase the precarity of ethnography as a method and approach. The goal of this article is not to argue that all ethnographers should gain quantitative literacy, but rather that they could, as an avenue towards effecting institutional change. Quantitative literacy is one of many possible avenues that ethnographers can take, as they push institutions to reflect on whether data are made and used in the best and most ethical ways.

Nadine Levin is a senior researcher at Facebook, who conducts strategic work on under-represented populations, including older adults and those with low digital skills. You can contact her at nslevin87@gmail.com with any thoughts.

NOTES

I would like to give thanks to my colleague and fellow anthro evangelist Abbas Jaffer, who read and provided comments on this paper, and has been a sounding board for these ideas throughout our time together at Facebook. I would also like to thank my husband Andrew Symington for always reading and being a tireless advocate of my work.

1. Here, I use “data” as an umbrella term encompassing a variety of numeric, quantitative inputs and outputs, ranging from the datasets on which AI are trained, to the metrics companies use to make product decisions.

2. Here, I use agency to refer to the socioculturally mediated capacity to act (Ahearn 2001).

3. Common examples of this are: (1) using the “Like” button to add heart and other reactions to Facebook posts, or (2) tapping and holding an Instagram story to pause the progression to other stories.

REFERENCES CITED

Ahearn, Laura M. 2001. “Language and Agency.” Annual review of anthropology 30 (1): 109-137.

Amirebrahimi, Shaheen. 2016. “The Rise of the User and the Fall of People: Ethnographic Cooptation and a New Language of Globalization.” Ethnographic Praxis in Industry Conference Proceedings. https://www.epicpeople.org/ethnographic-cooptation/

Arora, Millie P., Mikkel Krenchel, Jacob McAuliffe, and Poornima Ramaswamy. 2018. “Contextual Analytics: Towards a Practical Integration of Human and Data Science Approaches in the Development of Algorithms.” Ethnographic Praxis in Industry Conference Proceedings. https://www.epicpeople.org/contextual-analytics/

Biruk, Crystal. 2018. Cooking Data: Culture and Politics in an African Research World. Duke University Press.

Boellstorff, Tom, and Bill Maurer. 2015. Data: Now Bigger and Better! Chicago: University of Chicago Press.

Boyd, Danah, and Kate Crawford. 2012. “Critical Questions for Big Data: Provocations for a Cultural, Technological, and Scholarly Phenomenon.” Information, Communication & Society 15 (5): 662-679.

Buur, Jacob, Sara Said Mosleh, and Christina Fyhn. 2018. “Physicalizations of Big Data in Ethnographic Context.” Ethnographic Praxis in Industry Conference Proceedings. https://www.epicpeople.org/physicalizations-big-data/

Callon, Michel. 1984. “Some elements of a sociology of translation: domestication of the scallops and the fishermen of St Brieuc Bay.” The sociological review 32 (1_suppl): 196-233.

Cefkin, Melissa. 2010. Ethnography and the Corporate Encounter: Reflections on Research in and of Corporations. Vol. 5. Berghahn Books.

Cohn, S. 2004. “Increasing Resolution, Intensifying Ambiguity: An Ethnographic Account of Seeing Life in Brain Scans.” Economy & Society 33 (1): 52-76.

Crawford, Kate. 2013. “The Hidden Biases in Big Data.” HBR Blog Network. https://hbr.org/2013/04/the-hidden-biases-in-big-data.

—. 2016. “Can an Algorithm be Agonistic? Ten Scenes from Life in Calculated Publics.” Science, technology & human values 14 (1).