The Advertising Products research team at Yahoo! is building an internal research practice within an organization that is user-centered, but optimized for consumer product development. While our fellow researchers observe millions of consumers on our websites, we study our coworkers: their experiences with the tools of online advertising, and how those experiences shape the service that our advertiser customers receive. Adopting methods such as task-oriented interviewing and extended observation, we are reconnecting with a tradition of ethnographic inquiry in the workplace that is largely unknown at consumer Internet companies. This paper describes how we have re-learned and built company support for this approach. I describe our work with Yahoo!’s advertising sales and operations staff, highlighting the structural challenges of conducting and applying this research. I conclude by reflecting on how qualitative research can help a company bridge the gap between product design capacity and the ability to produce great services.
INTRODUCTION: YAHOO!’S OTHER USERS
As one of the world’s most recognized internet companies, Yahoo! is known for its large and broad user base. In press coverage and in its marketing materials, Yahoo!’s users are identified as global, representing a range of ages and backgrounds, and numbering in the hundreds of millions. Yahoo! has long valued and aspired to a deep understanding of these users, attained through investment in many forms of research. User researchers—variously called Design Researchers, User Experience Researchers, Customer Insights Researchers and recently Insights Researchers—have worked at the company since at least 1999.
1 UERs (as the author and her researcher colleagues will be called hereafter) are responsible for the qualitative and behavioral study of Yahoo!’s users and potential users, with User Experience Designers (UEDs) and product managers as our primary stakeholders. Most research done by UERs is focused on those legions of consumers who come to Yahoo! sites to check their email, read the news, share photos, and do many other things. The methods employed are various and constantly changing, but they include lab-based usability testing and eye-tracking studies, field visits, focus groups and remote qualitative research, concept testing and other participatory methods, and survey research. UERs also work together with market researchers and web analytics specialists to build insights from qualitative and quantitative data. Through extended engagement with a specific Yahoo! site or product, the typical UER develops deep knowledge about their site’s users: everything from the demographics of the user base, to their perceptions of competitor sites, to the needs, aspirations and goals they typically address through their use of the site.
Important as they are, however, most consumer users provide no direct revenue to the company. The vast majority of revenue comes instead from the advertisers who run marketing campaigns on Yahoo! and partner sites (Yahoo! 2009, pp. 30-31). This much smaller, but still diverse group includes individuals posting classified ads, small business owners, search engine marketing professionals, Chief Marketing Officers at companies of all sizes, and a host of brand managers, strategists, creatives, producers, media planners, buyers and analysts at thousands of corporate marketing departments, advertising agencies and agency holding companies. The most lucrative advertisers are global corporations, who may pay Yahoo! millions of dollars in a year.
Consumers experience Yahoo! as a network of websites and communication systems—essentially, online software products. For major advertisers, however, Yahoo! provides a service whose touchpoints can include sales calls and pitches, expert consultation, custom configuration of advertising programs, on-demand technical support, creative services, data feeds and even conferences and events. Yahoo! software interfaces also play an important role in this service, but in the case of large display advertisers, the interface is not directly used by the customer. Rather, it is experienced by internal users: Yahoo!’s front-line sales and advertising operations staff, who book orders for advertising on behalf of customers and deliver the ads on Yahoo! and partner sites.
Yahoo!’s Advertising Products UER team, which includes the author, was formed in 2006 to inform the design of advertisers’ experiences.2 As with other UER teams at Yahoo!, the primary objective of this team was to improve users’ experience of Yahoo! software by developing and sharing insights about its users—whether these users were external to Yahoo! (as with Yahoo!’s self-service products) or internal. This definition of our work has two implications, which frame the discussion here: First, we are the only UER team at Yahoo! to regularly conduct studies of Yahoo! employees, as well as advertisers. Second, while our mission is to improve the entire experience of advertising with Yahoo!, we are primarily responsible to a product organization that owns the software component.
The first section of this paper provides context about the business of online advertising, and reviews the methods the Advertising Products UER team developed to study the online advertising domain. Through these methods, UERs came to engage internal users as partners in product development. The second section of the paper looks at the limitations of this partnership and of a product-based approach to the design of the advertising user experience. This discussion concludes with the team’s emerging conception of design research in our domain, as a process that informs every touchpoint in a continuous and coherent advertiser service.
APPROACHING ONLINE ADVERTISING
Yahoo! sells many types of internet advertising—including search advertising, classified ads, promotional services, and recruitment ads—but the work I will discuss here focused on the display advertising business. Display ads consist of images, interactive graphics and/or video rather than text, and are typically more expensive than text ads. Unlike most buyers of text ads, display advertisers typically have sufficient advertising budgets for multi-media campaigns. That means that Yahoo! competes for their dollars not only with other online publishers, but with print, broadcast and outdoor media. Compared to these ‘traditional’ forms, online advertising suffers from the perception that it is poor at communicating an advertiser’s brand and reliably reaching desired audiences (Knoll 2010).
FIGURE 1. High-level process description of a display advertising campaign. Diagram created by the author based on research by Yahoo! UERs Frances Karandy and Michael Bartholomew.
Figure 1 describes generally how a major advertiser or its ad agency runs a display advertising campaign on a designated set of Yahoo! websites. Large advertisers running such campaigns typically work with two Yahoo! salespeople, an Account Executive who manages the overall customer relationship, and an Account Manager who works with the customer on specific online ad campaigns and places orders for advertising. In addition, accounts are assigned a Media Delivery Coordinator, who is part of Yahoo!’s Ad Operations team. This person’s job is to receive the creative materials that will be used in the advertisements, and to make sure the ads appear on our websites to the consumers, at the frequency, and in the time frame specified in the order placed by the Account Manager. To do their jobs, Account Executives, Account Managers and Media Delivery Coordinators rely on software tools developed and maintained by Yahoo!.
Beginning in 2006, Yahoo! made significant investments in redesigning these systems, as part of the introduction of a new online ad-serving platform, now known as Apt From Yahoo!. UEDs and UERs have taken part in these efforts from the start, though our activities have changed as the Apt strategy and target user base have evolved. Our current objectives are to improve our coworkers’ and partners’ experience of the systems they are obliged to use; to increase Yahoo!’s efficiency and accuracy in booking, serving, optimizing and measuring online ad campaigns; and to make it easier for advertisers to do business with us and to shift ad dollars from traditional media to our network.
With training and background similar to that of other UERs at Yahoo!, our team had to climb a significant learning curve to understand the specific research requirements of building software for online advertising sales and delivery. Most of us needed to learn the fundamentals of the ad business, the rapidly evolving role of digital advertising, and the conditions specific to online ad sales at our company. We learned partly by taking industry training and consulting various secondary sources, but as one might expect, our richest sources of information were the Sales and Ad Operations staff themselves, along with their trainers and managers.
It was during this ramping-up phase that our overall approach to the research took shape. Our methodology differed from that of consumer-facing UERs at Yahoo! in three important aspects:
- We created business process models, to capture our synthesis of findings from users in different jobs.
- We did workplace observation due to the intense and goal-oriented nature of system usage and the need to capture variation in use cases.
- Our small pool of internal users—a few hundred people in the U.S.—and the need for iterative, expert feedback led to repeated engagement with key informants.
In short, we adopted ethnographic and business analysis methods that were characteristic of systems design research during its formative years at the IT workplaces where it began, but that have faded from use in design research at consumer internet companies like ours. This approach has brought us unique insights, but also challenged common understandings of the place of research in software development at Yahoo!, held by both our stakeholders and our research participants. In the next three sections, I will describe the practices we developed, and the successes and setbacks that followed.
Business Process Modeling
The purchase of online display advertising is a complex transaction, occurring in a fast-moving marketplace: both the ads and the spaces they fill are intangible and subject to frequent change. Every ad campaign involves a diverse group of actors over time, working on behalf of the advertiser, their ad agency, or the publishers of the ads, in this case Yahoo! and its partners. No one person, or even company, has a complete view of the process. To develop a thorough understanding of how display ad campaigns happen on Yahoo!, our team gathered data from dozens of advertising professionals inside and outside the company, in the form of semi-structured interviews and workplace observation (see next section), over approximately 18 months.
As the data accumulated and the complexity and variation in the process became evident, process modeling emerged organically as a way to synthesize our findings and make them legible to stakeholders in UED, product management and engineering. UERs created several flow diagrams depicting the digital ad campaign ‘lifecycle’, from the pre-proposal stage through billing. Each model was based on field research within a particular organization (or type of organization): Yahoo! itself, a partner publisher that sells advertising on Yahoo!, or advertising agencies that develop online ad campaigns. Consequently, the diagrams focused on the work being done within each organization and the immediate inputs and outputs of that work to and from others. Figure 2 is an excerpt from one of these diagrams.
FIGURE 2. Initial page of a two-page workflow diagram created by UER Michael Bartholomew and UEDs Summer Bedard and Eric Thomason, showing the steps required to book and fulfill a guaranteed-placement display ad campaign on Yahoo!, from Yahoo!’s perspective. Activities shown above the timeline occur at Yahoo!; those below the line, at the ad agency working on the campaign. The complete diagram follows the campaign through execution, optimization, reporting and billing, depicting a total of 33 steps. This diagram and the processes it describes are the property of Yahoo! and are reproduced here under permission of the owner.
Diagrams of this sort are more visually complex than the typical user research presentation at Yahoo!, where less data-dense PowerPoint decks are standard. This presented an information design challenge: depicting both the overall campaign lifecycle and all the component workflows; notating actors and software systems involved in each step; and overlaying trend data (such as the current proportion of advertising orders requiring revisions). UERs collaborated with UEDs to develop representational schemes like the one in Figure 2, which succeeded in simultaneously showing the sequence, the locus, and the principal actors in all activities.
Presented to our stakeholders in slide decks, handouts and posters, these diagrams were immediately useful as a way of demonstrating UERs’ mastery of complex advertising business processes. This step was essential given the purpose of our research: to inform the design of a new software suite that would consolidate the functions of multiple existing tools. Only by documenting the current state of things could we gain the authority to make recommendations about the new system. In the longer term, however, our process modeling work allowed us to participate in a larger discussion about the display advertising market, taking place in sales operations, customer advocacy and at executive levels, as well as among our immediate stakeholders.
Many advertiser-facing employees had heard from customers that online display advertising was difficult to understand and to buy, compared to both traditional-media advertising and online text advertising. Furthermore, salespeople had complained that bureaucratic and administrative overhead made it difficult to sell in a way that was responsive enough to customer needs. One executive labeled this pervasive difficulty as “friction” in the display market—a term that subsequently gained currency across the company. Implicit in this term was the view that if the friction points could be located and smoothed out, display advertising sales would increase. Indeed, this is the premise of business process modeling as practiced by systems engineers: examine the entire process, identify the poorly performing parts, and target them for change (Havey 2005, pp. 3-8).
Because the UER team used ethnographic methods to gather data for our models, our conclusions were very different. In interviews and observations of online advertising professionals and through secondary research, we learned that there were many types of uncertainty inherent in the business: At the time the ad space was booked, neither the buyer nor the seller might know exactly how many times the ad would be shown, which web pages or application views would display it, what would appear alongside it, how many people would see it, what portion of them would be members of the advertiser’s desired audience, and how much the campaign would cost, let alone how many new customers or sales it would yield.
In field visits to Yahoo! and partner offices, our team observed the work Yahoo!’s sales and ad operations staff did to reduce customer uncertainty. This work leveraged proprietary technology, such as ad targeting logic that can restrict an ad to site visitors of a certain gender, occupation or consumer interest. It also made use of rhetorical tools: Salespeople presented Yahoo! as a “trusted partner” in marketing and tried to cultivate a “consulting relationship” with advertisers. These workers experienced their greatest dissatisfaction with Yahoo!’s ad booking and delivery system when it prevented them from offering certain assurances to customers. For example, in an interview conducted by UER Lee McNeill, one salesperson attributed a lost sale to the failure of an inventory forecasting function that determines how much ad space will be available for a future campaign:
I lost $15K in commission because [the system] couldn’t deliver…[it] was not predicting inventory and my client felt it was unreliable. (Sales representative, Yahoo! partner publisher.)
Drawing upon this fieldwork, McNeill and UER Frances Karandy created a process model that incorporated several extra steps taken by salespeople in this organization to assure clients that the exact ad space they wanted would be available. These steps included creating a “fake” order during the proposal stage, in order to locate inventory in the system that could then be offered to the advertiser.
Similarly to the salespeople, Yahoo! ad operations staff expressed an overwhelming need to know in advance how the system would behave. In their case this knowledge was needed in order to head off any technical problems that could arise with the “creative” (that is, the images or media files) that the advertiser provided. Even through they were not involved in negotiating the sale of the advertising, they reported feeling responsible for executing the ad campaign to the specifications promised by the salespeople. They were particularly dissatisfied with technical limitations in the ad previewing tools, which did not catch certain issues that could cause an ad to display incorrectly. A Media Delivery Coordinator the author interviewed described these failures as a series of harsh lessons: after placing confidence in a diagnostic tool that then fails, “you learn never to do that again.”
While evidence of specific failure points—such as those with inventory forecasting and ad previewing—did emerge during our interviews and observations, we also discovered cumulative effects that could not have been predicted by these instances alone. Overall, the existing software system did not support the relationships that our sales and ad operations staff aspired to have with customers. This suggested certain foundational principles for the system redesign, such as enabling our sales and ad operations staff to reduce customer uncertainty and build trust in their expertise and in our advertising service.
The discipline of process modeling was critical to developing the insights and recommendations the team produced. At the same time, it was a challenge to advocate for these recommendations using the workflow diagrams that were generated. Each of the diagrams commanded stakeholder attention by revealing a proliferation of steps that prolonged the time required to get a sale made or a campaign run, with evident costs for the business. In the months after Bartholomew presented the workflow diagram excerpted in Figure 2, his “33-step process” became known among stakeholders as emblematic of a situation that needed fixing. What was harder to convey was that a better process would not merely consist of fewer steps, but would embody design principles that aligned the system with its users’ ideals of customer service. The complexity of the process models made the case for change, but this same complexity undermined the team’s recommendation of comprehensive change by encouraging stakeholders to think about the most accessible, piecemeal improvements—or, to use a common expression at Yahoo!, “the low-hanging fruit.”
Workplace Observation
Observational research at work sites is a core activity of applied ethnography. In the information systems context, best practices in workplace ethnography have been a rich topic of discussion among practitioners since at least the early 1990s, with the publication of pioneering work on ethnographically informed systems design (for example, Bentley et al 1992). Rarely, however, have these studies and discourse influenced the practices of user researchers at large internet companies. A notable exception is the Contextual Design methodology developed by Hugh Beyer and Karen Holtzblatt. In their 1998 book Contextual Design: Defining Customer-Centered Systems and in several related publications, Beyer and Holzblatt instruct would-be practitioners in a research process that blends ethnography and business analysis, and allows team members of diverse expertise to collaborate in developing and acting upon insights from the field.
The years following the book’s publication coincided with the first internet boom. As the applications of online systems grew along with internet businesses, Contextual Design provided web development teams with a systematic way to derive product requirements from direct encounters with users. Today, Contextual Design—especially its user-led observation sessions and group synthesis of data through affinity diagramming—is considered an essential part of the user researcher’s toolkit (Kuniavsky 2003, pp. 160-192). For example, Google has shared stories of field research on the contextual inquiry model in its official blog (Russell 2008).
In recent years, however, contextual inquiry and other field study methods have been applied increasingly selectively at Yahoo!. Some reasons for this are practical. Shorter software development cycles create pressure to produce findings more quickly. User bases have become more global while travel budgets have been reduced. More to the point, different research techniques have evolved that may be better suited to studying consumer website use. Remote and asynchronous methods, such as diary studies, capture user impressions of interactions over multiple sessions that may share an overarching goal or theme. Eye-tracking studies delve deeply into the micro-level of user experience: the second-to-second distribution of a website visitor’s attention. Any of these methods may be combined with retrospective interviewing to focus on the experience of specific sites or tools, rather than the holistic experience studied through contextual inquiry or ethnography.
The Advertising Products UER team is one of the few at Yahoo! that employ field research as the predominant means of data collection. This is largely due to the team’s current focus on the experiences of internal users in the workplace. As mentioned, Yahoo!’s sales and ad operations staff is a relatively small population. They are concentrated in a few locations; they do stationary, desk-based work that involves intensive use of the software systems we are researching. Furthermore, much of the users’ work involves real-time collaboration with colleagues, which can only be captured in the field. For example, in a recent field visit, the author observed an impromptu problem-solving session among several colleagues located at two offices, after a software component stopped working:
Oh no, [a coworker at another office] is having that syncing problem… Did loading metadata work for you? (Media Delivery Coordinator, to another MDC in an adjacent cubicle)
In other instances, insights into the work process came from observing the way users employed personal information management applications on their computer desktops, and paper notes, calculators and other objects on their physical desktops, to organize their tasks in ways not supported by the ad booking and delivery system. For example, the author observed one Media Delivery Coordinator who used email “flags” of five different colors to indicate different message content, with yellow, the most prominent color, as a signal to “do something.”
While visits like these yielded valuable data, they changed our customary relationship with participants and stakeholders. Over time, it became clear that internal user observations introduced an element of trust and implied obligation that is not part of the relationship with external research participants. When we visited Account Executives, Account Managers, and Media Delivery Coordinators, we explained our connection with product development and our objective of improving their tools. As a result, some internal users appeared to view UERs as conduits for problems and feature requests to be fed to the product team and then fixed. This assumption could create a mistrust of research efforts when the issues identified were not solved in the next release of the software.
Other interpersonal issues arose when data from these sessions were shared with product teams. Some observation sessions included very blunt criticism of the software systems by the user. This material appeared to have a deeper significance and potency for developers and managers because it came from coworkers. In some cases, stakeholders responded to harsh criticism by invoking their own user outreach projects, reporting that they had asked the same users and the issue had not come up. Interpretational conflicts like these were an unfamiliar and often difficult experience for UERs who were used to doing consumer research.
To convince product teams to accept findings from field observations, we found it helpful to conduct field visits in parallel with quantitative studies that can set a baseline for user experience and measure progress. The UER team established a regular schedule of time studies and satisfaction surveys for this purpose. Using both in-experience and retrospective measures allowed us to report on which parts of the workflow were getting better or worse (supported by the analysis done while building process models), while ongoing field research helps us to offer accounts of why this is happening. While this approach may unfortunately lead product teams to view observational findings through the lens of the metrics, this disadvantage is offset by the benefits of benchmarking the experience of a recent software release against previous versions, a comparison that could not be easily made with observational research alone.
Repeated Engagements with Users
In consumer research, working repeatedly with the same participants is studiously avoided. In our research with internal Sales and Ad Operations staff, it was both hard to avoid and potentially beneficial. Some repeat engagements happened unintentionally, as a result of small user populations being targeted for outreach by sales operations or product management as well as UER. In many cases, however, the UER team intentionally worked with participants who could offer unique perspectives due to their position, degree of experience, or familiarity with a given use case. Key informant relationships developed, in which research participants became self-aware about their roles as proxy users and actively collaborated with UERs, and UEDs.
Key informant relationships were driven by our need to learn from users who were themselves in the middle of an intensive learning process. Yahoo!’s new ad booking and delivery platform was being gradually rolled out to different sales and ad operations teams. To meet the various business requirements of those teams, the engineers were adding substantial new feature sets to the system with each release. Since the old software system had taken months for users to learn, we wanted to be able to provide recommendations about the optimal approach to training; this meant following early adopters through their learning process. In addition, early users naturally acted as beta testers, who helped to validate and modify features over multiple iterations.
A few internal users have now worked with us for months or years, and their long-term participation has given them the status of trusted contributors, known to product managers and engineers as well as researchers and designers. Their collaboration allows us to iterate not just the product, but the product development process. Some features of these relationships are uniquely feasible because researchers and users work for the same company. Their status as fellow employees permits them a view of the development process that mere “users” are unlikely to have. Nonetheless, the experience of working with key informants offers motivation to think about all design research as having the potential to incorporate co-design. Observing users interacting with a changing software system over time, we are made aware of design research as an arrangement between people, and not a knowledge extraction operation (Suchman 2002). Recalling Granka, et al’s invocation of ethnographers as “brokers of access to ‘real’,” we have learned that our “real” is not a static body of knowledge that we curate and broker; it is the flow of knowledge and experience through our organization (Granka et al, 2008).
TRANSFORMING INTERNAL RESEARCH: FROM PRODUCT TO SERVICE DESIGN
As proclaimed in numerous mission statements and job descriptions, the normative role of the corporate user researcher is to advocate for the users of a product throughout its development; to argue for both the design decisions that benefit users and the allocation of sufficient time and resources to provide a good user experience. The Advertising Products UER team has frequently played this role with respect to internal users. The current advertising platform has had several releases, and each has included user interface improvements grounded in user research. By demonstrating the severity and business impact of certain user experience issues, UERs have been able to elevate the priority of those issues over others and get them addressed earlier than might otherwise have occurred.
The research described here, however, aspires to a two-tiered model. Rather that simply advocate for users, we seek to partner with internal users in order to better advocate for our external customers, the advertisers. We hope to do this by maintaining ongoing relationships with internal users, which helps us develop focused insights that improve through iteration, as do the products. By also working closely with UEDs, we gain the ability to not just learn from, but co-create design solutions with internal users. We can then assess these solutions against research on advertiser needs, and prioritize the measures that will ultimately have the most positive impact on customers.
Alignment between UERs, UEDs and internal users could provide the foundation for a holistic view of Yahoo!’s advertising service as a unified customer experience, rather than a series of interactions with websites, data, documents and people. In this vision, design research could be as valuable an input to employee education and training, staffing levels, or back-end system architectures as it is to user interface design. This will require design research with an expanded scope and new patterns for involving both internal users and external customers. In working toward these changes, we inevitably involve ourselves in corporate politics, and contend with the historical association of UER and UED at Yahoo! with interface design innovation, rather than with innovation more broadly. We must also contend with imbalances in our own training and knowledge: the typical UER at Yahoo! has long experience translating research insights into recommendations for better button placement in a user interface, more clearly worded labels, or more appealing content offerings. He or she is less likely to know the language of process improvement: how to recommend changes to policy, incentives, or training.
As design researchers, how do we get from internet-based product design to internet-enabled-service design? From having a well-defined set of stakeholders who have ‘bought into’ the value of user research, to informing constituents across an organization, who may have no starting assumptions about the value of our work? It remains to be seen whether research teams like ours, embedded in large, established user experience divisions, can be nimble enough to forge new paths of influence within our companies, as described by Hanson & Sarmiento (2008). Based on our team’s experience, we would do well to continue to draw lessons from workplace ethnographers: not just from their data collection methods, but from their flexible and opportunistic approach to making organizational change.
Andrea Moed has over fifteen years’ experience designing and evaluating interactive systems. As a Senior Insights Researcher at Yahoo!, she leads user research on social media tools for advertisers. Andrea has master’s degrees from the Interactive Telecommunications Program at New York University and the UC Berkeley School of Information.
NOTES
This paper references and builds upon work by the author’s fellow Advertising Products User Experience Researchers from 2007 to 2010: Michael Bartholomew, JonDelina Buckley, Frances Karandy, Lee McNeill, Erik Troberg and director Jill Strawbridge. The author wishes to thank all of them for their indispensable guidance and support, and their help in reviewing this paper. Thanks also to the designers of the Advertising Products User Experience Design group, whose partnership was critical to the research described here. Finally, thanks to Elizabeth Churchill at Yahoo! Labs for her comments and feedback on the paper proposal.
1 Email exchange with Tracy Beasley, June, 2010. Beasley has been employed at Yahoo! since 2003 as a participant recruiter, and worked with some of the first UERs.
2 Conversation with Michael Bartholomew, July 2010. Bartholomew was one of the first UERs to be part of the Advertising Products team.
REFERENCES CITED
Bentley, Richard, Hughes, John A., Randall, David, Rodden, Tom, Sawyer, Pete, Shapiro, Dan and Sommerville, Ian
1992 Ethnographically-Informed Systems Design for Air Traffic Control. Proceedings of the 1992 ACM Conference on Computer-Supported Cooperative Work, November 1992, Toronto, Ontario, Canada.
Beyer, Hugh, and Karen Holtzblatt
1998 Contextual design: defining customer-centered systems. London: Academic Press.
Granka, Laura, Larvie, Patrick, and Riegelsberger, Jens
2008 Now You See It, Now You Don’t: Ethnography and Selective Visibility in the Technology Sector. Ethnographic Praxis in Industry Conference Proceedings, Copenhagen, Denmark. Arlington, VA: American Anthropological Association.
Hanson, Natalie and Johan Sarmiento
2008 Sustaining Stories: The Versatile Life of Sustained In-House Ethnographic Practice in a Global Software Company. Ethnographic Praxis in Industry Conference Proceedings, Copenhagen, Denmark. Arlington, VA: American Anthropological Association.
Havey, Michael
2005 Essential business process modeling. Sebastopol, California: O’Reilly Media.
Kuniavsky, Mike
2003 Observing the user experience: a practitioner’s guide to user research. San Francisco: Morgan Kaufman Publishers.
Suchman, Lucy
2002 Located Accountabilities in Technology Production. Scandinavian Journal of Information Systems, 14(2), pp. 91-105.
Web resources
Knoll, Scott, “Yes, Web Advertising Scales – If You Measure It Right.” Advertising Age, Crain Communications, July 6, 2010. http://adage.com/digitalnext/post?article_id=144812, accessed 19 July 2010.
Russell, Dan, “The Art of the Field Study.” The Official Google Blog, November 6, 2008. http://googleblog.blogspot.com/2008/11/art-of-field-study.html, accessed 19 July 2010.
Yahoo! 2009 Annual Report. Downloaded from http://yhoo.client.shareholder.com/annuals.cfm, accessed 1 July 2010.