Flexibility and the Curatorial Eye: Why and How Well-Documented Fieldwork Sustains Value Over Time

Program Flexibility

By program flexibility, we mean the ability to re-direct the course of a project while conducting the research. In design-oriented ethnography, the research process is iterative, with the discussion evolving to take advantage of what the researcher learns during the observation or interview. For example, interview protocols may evolve through the course of a project, if necessitated by emerging understandings of the situated practices under investigation. Changes made in response to observations, often at fast pace and in situ, require a reconfiguration of the data gathering instruments—which in ethnographic work are the ethnographers themselves. The reason for this flexibility is ideological and rooted in the validity of open-ended conversations that vary individual to individual, rather than an insistence on a consistently applied instrument administered to every participant. By contrast, experimental methods and survey methods embody predetermined assertions about construct relevance and are less suited to the making of modifications while a program is running. The tight relationship between data collection and analysis better equips ethnographic methods to respond quickly to emergent directions during research.

Data Flexibility

By data flexibility, we mean the revisiting and reviewing of data from previous projects for new insights, which can lead to the derivation of previously unseen connections between areas. The very ‘messiness’ of ethnography’s systematic, but open approach to data collection and agnosticism about data formats is its strength for providing ongoing long-term value. Design ethnographers generate written notes, photographs, video and films, written and drawn sketches, and produce recreations of relevant artifacts. As a result of the richness of the collected data, datasets contain not just the answers to the questions posed, where analysis can focus on predetermined or emerging issues in relevant frames, but also contain rich contextual, “extraneous” data. Qualitative, field-based data collection, therefore, yields highly rich and nuanced data that carry contextual meta-data that enable the setting and activities to be re-experienced through different lenses. Data from other forms of empirical enquiry are often not so rich and not as facile; experimental control implies data can seldom be reviewed for new insights except within the confines of the experimental questions that were initially posed. The nature of evidence and its form, constrains what questions can be asked of it. By maintaining an open mind, field researchers often collect additional contextual data that later can yield great gains.

These are central areas of value that have to date been undersold, usually being subjugated to the ‘product’ of the field program – usually design recommendations, guidelines and prototypes. Both of these capabilities imply a particular set of practices around program management, including data gathering, anonymisation, storage, and curation. These have, to date, been taken as necessary skills for an ethnographer, but have not been identified as providing particular value in an organization, in part as a result of the focus on outcome/product rather than process. We believe it is important to highlight these two skill-sets. Firstly, qualitative researchers are trained to be reflective about the specific program details (Are we collecting the right data? Are we using the right methods to gather those data?) and also trained to be reflective at a strategic level (Why are we conducting this research? What are the implications of using the current methods?). Researchers are also trained to be critically reflective of programs that are not delivering results to questions posed. Strategic insights can result from such critical reflections, but credit may not be accorded to the field research team for driving these insights. Worse, if a program is halted or realigned during a field trip, on occasion the assumption is that the original field program was a failure – not that the orienting assumptions at strategy level were identified as being flawed, and that investment has been protected through reorientation. Secondly, while qualitative data is acknowledged to be ‘rich’, two critiques that are levied are that it is more time consuming to collect and less able to be generalized. We believe that the additional contextual detail means qualitative data can often be re-visited for new insights, and this can save time in project planning downstream. It is important to understand that investment in a field project can continue to have pay-offs much later. Data storage, organization, indexing, and reanalysis are key skills of good qualitative researchers. Indeed, an essential part of our craft is the storage and curation of field results, and well-trained field researchers are careful to incorporate data storage and indexing as part of the program planning and ongoing program management.

Case Study 1: Drinking Water and Storage in India

In January 2009, an IDEO research team spent three weeks in India collaborating with Acumen Fund on the Ripple Effect, an eighteen-month project backed by the Gates Foundation to increase access to drinking water by the poor. The strong social mission of this project mandated a very lean budget, and the team was challenged to deliver maximum business value with limited financial outlay. The three-week duration of this field visit was a hard constraint determined by the budget, and the team was responsible for reporting opportunities from diverse water contexts throughout India during that time. There were significant logistical challenges to accessing our field sites in Gujarati villages, slums in Bangalore, and peri-urban villages outside Hyderabad. The team traveled approximately 4,000 km/2,485 miles around India by plane, car, and rickshaw in a program designed to balance breadth, or reporting needs from multiple contexts, and depth, or taking the time to cultivate and maintain the personal relationships essential to working in India.

Recruiting participants started with Acumen Fund’s network of social enterprises. We relied on their introductions to local water organizations and used those organizations to get access to communities. Our partner organizations live and work in the communities we visited, so we deferred to them about how to navigate community politics. Despite our intense time constraints, we spent up to one third of our time on visits to the water organizations’ regional offices and in introductory meetings with community leadership. Once in the field, getting access to appropriate participants was still a challenge since local leaders sometimes had strong opinions about who would show the community in the best light, putting intense pressure on getting the most useful responses during precious time with our participants.

Program flexibility allowed us to respond to emerging information in the field and to deliver maximum business value for Acumen Fund and Gates Foundation without a need to extend the time in the field. One example of in situ adaptation during this program was guiding the snowball sampling method to appropriate respondents. For example, if participants were embarrassed to discuss something as personal as water use in front of strangers, or over-dramatizing their difficulties, we moved on. Because of our flexibility, including the opportunistic decisions to divide into two parallel tracks or respectfully cut short awkward encounters, we were able to meet the program’s goals and interview a mix of families including those without access to treated drinking water (11 families), those who purchased treated drinking water (5 families), and those who had access to treated drinking water but chose not to purchase it (5 families).

Over the course of this fieldwork, our team made changes to the protocol that allowed our team to deliver maximum business value under intense time constraints. We knew from previous field research in the developing world (see Human Centered Design Toolkit in Web Resources), that supporting interviews with physical artifacts is effective. We went into the field with photo cards and sketches to help communicate, but made a decision to add a new activity in response to emerging needs. Because water is relatively cheap for its weight and bulk, we experimented with light-weight, high-value items that could be bundled with water by showing participants products from local kiosks such as singe shampoos, incense, and fruit that a door-to-door water peddler could also sell to make a water delivery route economically viable for the peddler while keeping water affordable for customers. This activity effectively helped the team gather more specific feedback, and was a nice complement to the photo sorting and discussion cards we used throughout the fieldwork.

A second change to the program demonstrates an agile re-framing of the design challenge while in the field. We were contracted to explore community water storage solutions including shared tanks, and our research plan dedicated some of our very limited time to that activity. However, in the field we quickly learned that shared tanks are undesirable because the origin and freshness of the water inside is questionable. We made an in situ decision to divert time away from community storage solutions towards a new activity better able to serve our ultimate objective of increasing access to treated water by the very poor. We were surprised that groups of slum families pooled their resources to order an entire tanker truck of water because our earlier research suggested it was not feasible. As soon as the team identified the mistaken assumption about the desirability of shared community storage early in the research, and learned that tankers we feasible, we were able to respond by recruiting a tanker truck for a visit and interview the driver. The flexibility to adapt the program while it was running led our team to gather meaningful insights with business value in a limited time.

Case Study 2: Working With Mobility

From 1999 until May 2000 a three-phase project was conducted to address changes in work-based communications resulting from the introduction of mobile technologies. This period of time saw an explosion of the use of mobile technologies such as cell phones, laptops, mobile email, and SMS providers, as well as the convergence of walkie-talkie functionality with cellular telephones. The project was developed to more deeply understand the implications of the introduction of these technologies into the workplace, and consisted of three parts: a diary study of the use of the RIM Blackberry 850 email/pager by two collaborating co-workers, a field-based interview study of 17 sales people from a multi-national printer company in the US, and a field-based study of 11 mobile phone users in the UK, looking at patterns of recreational and work use. In all three phases, data gathered were text, photographs, drawings, representations of regularly used artifacts, and, where possible, video recordings. In the case of the field studies rather than the diary study, field interviews were conducted and, where possible, interviewees were shadowed between several hours to a day. The study results focused primarily on people’s management of data flows in collaborations and on development of applications for graceful management of service interruptions. Key findings from the study with respect to use of mobile technologies were reported internally, and a framework for design was developed and published (Churchill and Wakeford 2001).

For the purposes of this paper we will focus on the second phase of the study of sales people. Whilst the study was predominantly about the use of mobile technologies, two themes emerged at the time and were also documented initially in the form of ‘asides’—reflecting our point in the previous section of this paper about program flexibility. The first theme was the effect of “hot-desking” that was intimately linked to the provision of laptops and cell phones to all personnel, replacing desktop computers and phones. The second related observation was the use of fliers and posters in the physical space as a means of communication between members of the sales team. Both of these issues were pursued as side issues to the primary data collection. These insights from this phase of the project, together with supporting data, were subsequently used on three occasions. First to drive proposals for the development of digital bulletin boards with a back-end social networking site that is also accessible from cell phones (Churchill and Nelson 2009). Secondly the data and frameworks were used to outline design opportunities for mobile applications for recreational use, and to drive the project ideation phase of a project conducted in Japan, comparatively addressing infrastructure, hardware, and service provision in Japan, the US and Europe; data from the earlier study were used as a primer and to drive research questions prior to the project launch. Finally, the data and frameworks were adapted in developing a program addressing the practices of remote work in homes and in Internet cafés, conducted in 2004.

In all instances, the original data were mined for potential insights, the original framework was questioned and elaborated, and new questions were generated to illuminate what factors did and did not generalize in the face of market changes. A key factor in the success of this data reuse was that data collected were systematically organized according to a scheme agreed upon by the team. Following Emerson et al.’s schema, during the original studies we made “jottings”, that is recording activities in terms of ‘what, how, where and when’ on paper; we made drawings of the space and took photographs of spaces used and of artifacts; we wrote extensive notes following time in the field in the form of multi-media documents depicting the settings and specific scenes; and we generated characterizations of the people involved and sketches of incidents, combining all of these into episodes and tales with dialogue (Emerson et al. 1995). In all instances, while generating detailed representations for the project at hand, we were also generating rich data capable of providing opportunities for re-experiencing, collecting, and storing data with the curatorial eye of a documentarian. It is this focus that distinguishes qualitative work from other forms of research—the focus on providing sufficient data to enable re-experiencing.