Little Data, Big Data and Design at LinkedIn

INTRODUCTION

LinkedIn’s UED (User Experience Design) Research team is “little” (8) relative to the company (6000+). The qualitative data we gather is even more vastly dwarfed, by orders of magnitude. A UED Researcher might target and interview 12 people for a project, while her counterparts in Product and Marketing are measuring interactions in the millions. Like many global companies, our product design and development process is built on a foundation of behavioral data, real-time metrics, algorithms and predictive models. In a word (or two) “big data.” In this environment, our team has established a strong foothold for our work alongside our big data counterparts, squarely in the realm of design.

Big data holds undeniable power. It informs and increasingly shapes the products and services that make up our world. In this paper, we use the term “little” playfully, in the David and Goliath sense. In practice, it’s not us against them, and it’s not even a battle. Our small team of qualitative researchers has an outsized impact because of the inherent power of the approach, and our single-minded focus on impacting design. Because we sit within the UED group, alongside Interaction Designers, Web Developers and Writers/Editors, our ability to create impact in design is secured. Our team’s is a success story, but not without the occasional plot twist. We would like to boast that collaborations always result in mutual alleluias and allegiance. In reality, however, at times both sides must exercise diplomacy at the border where big and little data meet. Though at first our work may seem to defy logic to those worshipping at the big data altar, and vice versa, open minds benefit greatly from an approach that has room for both.

DATA AND DESIGN

Big data is gathered, discussed, debated and leveraged in a number of ways.

Big data is traded as an asset or a commodity. Abby Margolis talks about our “digital exhaust” being traded and monetized, suggesting that many people are “convinced that big data will become the world’s most important resource, the fuel for the next economy.” (Margolis, 32) This is most often experienced as targeted advertising.

Another way big data impacts our lives is in the increasingly mainstream “quantified self” movement. In 2010, Gary Wolf talked about the proliferation of mobile devices and biometric sensors, which allow us to increasingly capture data about our habits and behavior to “reflect, learn, remember and improve.” (Wolf)

He also suggested that, beyond contributing to self-knowledge, these personal data sets combine with social networks and distribution platforms to create interesting opportunities to contribute to public health research and biometric security.

Perhaps the most interesting consequences of the self-tracking movement will come when its adherents merge their findings into databases. The Zeo, for example, gives its users the option of making anonymized data available for research; the result is a database orders of magnitude larger than any other repository of information on sleep stages…this type of database could help to redefine healthy sleep behavior. (Singer)

We observed a recent example of what this looks like after the recent South Napa earthquake on August 24^th. Data Scientists from Jawbone aggregated data from thousands of local UP users who were tracking their sleep that night.

Our data science team wanted to quantify its effect on sleep…Napa, Sonoma, Vallejo, and Fairfield were less than 15 miles from the epicenter. Almost all (93%) of the UP wearers in these cities suddenly woke up at 3:20AM when the quake struck. Farther from the epicenter, the impact was weaker and more people slept through the shaking. In San Francisco and Oakland, slightly more than half (55%) woke up. (Mandel)

THE LION AND THE MOUSE

Just as qualitative practitioners can be quick to point out the limits of big data, big data adherents sometimes doubt the validity of little data. We encounter those who do not immediately perceive “little data” as actual data. Scale trumps story; insights based on ethnographic research run the risk of seeming anecdotal.

Aesop’s fable, The Lion and the Mouse, nicely illustrates a useful approach for us. The story goes like this: the lion is going to kill and eat the mouse. The mouse begs for its life, promising to help the lion someday. The lion is so amused by this notion that it spares the mouse’s life. Soon after, the lion becomes caught in a hunter’s net. The mouse gnaws through the net, freeing the lion and proving its worth.

One “net” that big data gets caught in is negative space. One must think about what’s not in the data set as much as what is. Kate Crawford referred to this effect as “signal problems,” or biases. She suggested that, “with every big data set, we need to ask which people are excluded.” (Crawford) We regularly (but gently) remind teams that behavioral data is absent context and culture – and, by nature, reactive. We can only click on what is there. By the same token the less active a LinkedIn member is, the more of a mystery they are to us. We are left to fill in the data vacuum with assumptions, guesses and inferences.

Another “net” we can help free the lion from is abstraction. Large data sets are abstractions of humanity, and proxies for assumed emotion. Click-throughs and return visits that resemble addiction are interpreted as positive emotional experiences.

Ajran Haring, a technologist and behavioral scientist, put it this way in an interview in the web magazine, Medium:

Companies started to use ‘user engagement’ as the core metric that they built around. Engagement is all about usage: how often someone uses the product and how long they use it for… getting people in your product as many times as possible, and for as long as possible, became the barometer of success. (Hreha)

But what does this sort of engagement feel like for the user? Our team works to add depth of understanding, culture, context and, ultimately, the “why” behind the numbers. On that note, however, Curran warns us against naively assuming that uncovering the “why” is the exclusive jurisdiction of the qualitative researcher: it would be a mistake to believe that “qualitative approaches are positioned within an elite creating game changing insights while Big Data is less capable of doing this.” (Curran, 68) Understanding the “why” must be seen as a collaborative endeavor, as should the process of defining success metrics.

Simply asking Product Managers, Data Scientists, Designers and other stakeholders in a genuinely curious manner what they believe lies behind the numbers, and what their questions are – in other words, where they are trapped – can create openings. We sometimes frame these as data “mysteries.” People start to then wonder how they might solve these mysteries, and we are there to apply our methods, provide an additional level of insight, and point to opportunities.

Profile Photo – One example of such a mystery arose in regard to the LinkedIn profile photo. The project team set a goal of increasing the percentage of members who upload one. Having a photo materially increases a LinkedIn member’s odds of being viewed by a recruiter. Improving this number would result in a win for the member, and a win for our ecosystem. The team hypothesized that usability issues were the key to improving this number, and so the process was made simpler, more elegant and optimized. While these were notable improvements, the expected changes in the numbers were slower to follow.

The UED Researcher embedded with this team, Elysa Soffer, started asking what else we know about why some people don’t upload photos, and the team was left to guessing. Hypotheses lacked a deep understanding of the core reasons. She interviewed a small number of our members, at a very low investment, and detected patterns suggesting that the problem went beyond usability. Study participants were carefully targeted to represent people who seemed very much like members who might or should upload a photo. Yet, they hadn’t. Research revealed a host of very human reasons why not. A lack confidence in appearance. No suitable photo. Uncertain what is appropriate. Doesn’t want to open himself or herself to bias, discrimination or stalking.

These actionable insights fundamentally reframed the way the team approached the feature. Designers incorporated image previews, while writers found ways to address members conversationally, and with more empathy throughout the upload process, to guide them and clearly spell out the benefits. This investigation led to further explorations and a deeper understanding of the barriers preventing people from making a variety of updates to their LinkedIn Profile, which are currently informing long-term product strategy.

THE TROJAN HORSE

In The Odyssey, Homer told the tale of the Trojan Horse, the ultimate subterfuge. A greatly outnumbered battalion gains entry into a city protected by numerous troops and an unassailable wall by pretending to be a gift.

The UED Research team has been known to employ this tactic. Earlier this year, a team engaged Researcher Yoni Karpfen as they redesigned a webpage to better communicate our suite of subscription-based products to LinkedIn members. This is not typically a project that we would take on, as we generally focus more on front-end initiatives, but we sensed an opportunity. The team’s goal was to organize the page to optimize click-through and drive conversion to paid products. We knew that deeper issues would be surfaced through conversations around this design. Yoni partnered with the designer to set up exploratory research wrapped in concept testing. After just a handful of interviews, the patterns were strong enough for the team to begin rethinking their approach and even revisit the suite of products entirely. The project evolved from prescriptive to existential. Months later, the design that resulted is outperforming designs derived from big-data-driven optimizations.

DOUBTING THOMAS

A Doubting Thomas cannot rely on faith. He must see with his own eyes to believe. A big data adherent can be convinced of its limitations only through examples.

The Research team dabbles in big data ourselves from time to time, most often for targeting and recruiting participants. Data gurus help us reach out to targeted members to participate in research. They extract data from our enormous database of members.¹ Sometimes these efforts wind up almost comically off base, failing to reflect intent, context, or even in some cases the most basic demographic information. These failures become examples to mention in passing to a Doubting Thomas.

A couple of examples from the recruiting front lines: we recruited people who had been members for less than thirty days for a New Member Experience study, only to find that a handful of participants had actually been members for years, but recently inadvertently created a duplicate account. For a separate study, to better understand the experiences of people actively looking for work, we recruited members who appeared to be extraordinarily active job seekers based on their behavior on the site, only to find that some were looking for jobs for their spouse or child.

This type of data disconnect does not inform product design or strategy, necessarily, but it does provide us with surprising examples to sprinkle into conversations. When we expose these stories to product teams, a seed of healthy skepticism and/or curiosity germinates. People start to realize that we can’t rely on big data or metrics wholesale to define and understand people. We are serving people, not numbers, and people are complicated. As much data as we have about people, people defy being defined by them.

Global Navigation – When LinkedIn set out to redesign its global navigation system in 2012, our team supplemented A/B testing with a series of group interviews, which were essentially in-person A/B tests. While variants of the design were tested on the live site, Researcher Julie Norvaisas met face-to-face with members for purposes of comparison. For each of these supplemental groups she recruited three members with very different profiles (i.e.: a senior executive, a person in their first job, and a student), in order to provide contrasting perceptions and engender conversations. Each participant took a turn signing in to their LinkedIn account; we gave them temporary permissions to play with each of the new navigation bar designs. As participants explored and experienced the designs, which were projected onto a large monitor in the room, we discussed reactions and relative merits as a group.

The global navigation is particularly interesting territory organizationally, because teams vie for top tier representation, and feel strongly about the terminology, order and hierarchy. Many of our colleagues watch these metrics very closely. All of the stakeholders convened to review the results – both quantitative and qualitative. Some team members were very interested in our companion qualitative work, others less so. But everyone perked up when Julie shared one basic but important piece of data that was not found in A/B test results: people liked the new simplified designs compared to the current design at that time. Both of the designs – A and B! This proved helpful in framing the numbers, as we weren’t in the dark about the emotional reaction to the new designs. We could talk about performance and feelings at the same time. Having established that, we were able to contribute additional insights about responses to changes in wording, search features and notifications in both designs.

That was an experimental approach, which turned doubters into believers. One Product Manager remarked later that he would not have known that he was missing a layer in his interpretation of the metrics had we not been involved. Because we were able to provide our inputs simultaneously with the quantitative A/B testing results; discussion and decision-making were equally influenced by both.

Endorsements – Another example: our team did work on LinkedIn Endorsements that proved useful in turning stubborn minds on to the value of qualitative research. Endorsements performed extremely well when the feature launched in 2012. Based on engagement with the feature, LinkedIn members were quite keen to publicly vouch for one another on the basis of individual skills. Endorsements rapidly multiplied as members paid forward the social proof of each other’s skills. Within just a few months, over a billion of these gestures were registered across the network. Measured in clicks, satisfaction with the feature was through the roof!

At the same time, all of the Researchers on the team started hearing unsolicited criticisms of Endorsements in our work on unrelated topics. The fact that Endorsements were so engaging was contributing to their credibility issue. Members reported receiving Endorsements for skills they did not feel they possessed; for which they did not want to be known; or from people without the requisite knowledge to endorse them for a given skill. Interviews with Recruiters confirmed our suspicions: while they viewed LinkedIn as a key source of information about candidates’ skills, the rapid proliferation of Endorsements caused them to question their trustworthiness.

Colleagues following metrics and engagement numbers were celebrating Endorsements success as we conducted a study and drew our conclusions. The data were telling two very different stories. When critical stories emerged in the press, and Endorsements made it onto the Meh List of the New York Times Magazine, (Staley) we were ready with answers and ideas. Findings led the Endorsement team to implement a series of improvements and downshifts to the experience. In some cases these changes had a negative impact on the numbers; our team’s work helped us become more comfortable with that outcome as an organization.

DAVID AND GOLIATH

We started our paper with a reference to this Biblical tale, and we will return to it for our last story. If Goliath is big data, and the UED Research team is David, the rock in our slingshot is empathy.

Field Day – The majority of our non-research colleagues rarely have an opportunity to observe or engage with users of the products they create. It is easy for teams to lose touch with the needs and experiences of their customers. Once a quarter, we assemble fifteen teams of our colleagues to venture out into the wild to interview users in their homes or workplaces. Small teams from across the organization – Product, Marketing, Business Development, Finance, Sales, Legal, Customer Service and beyond – spend the morning interviewing members, followed by an afternoon exchanging stories and developing themes. We call this quarterly event “Field Day.” It’s not uncommon for our colleagues to tell us, at the conclusion of a Field Day, that this was the first time they’ve ever spoken directly to a member (at least, a member who is not related or well known to them) about the product, or seen a member interact with a product they work on.

These empathy exercises extend to every level of the organization. In late 2013, LinkedIn began exploring concepts for a mobile application devoted exclusively to job searching.² As a professional network, we were perfectly positioned to launch such a product. Yet our experience and historical data provided little guidance on how to best address our members’ needs for such a narrow use case, and exclusively on mobile devices.

Given the strategic importance of this project, our executive team took a particular interest in its execution. As a result, we decided to focus our Field Day for that quarter on mobile job search, and dubbed it “Executive Field Day.” Every member of the executive team attended, providing them with a rare glimpse into the daily lives of our members.

At a company All-Hands meeting later that month, our CEO, Jeff Weiner – who is known to be relentlessly data-driven – emotionally emphasized the value he got out of the interview he attended. The team he went out into the field with, led by Julie, interviewed a young professional woman in San Jose, California. They had met the participant in her grandparent’s home, where she lived with them. Jeff shared with the company that, during this structured 2-hour interview, he gained invaluable insight into the young woman’s career ambitions, her skills, and her perceptions of LinkedIn. He spoke about how experiences like this help us question our assumptions, and how important that is. He had internalized her struggles, related to her ambition, gained respect for her sophistication as a user. He left energized to continue to build products that serve her well, with a renewed sense that a seemingly small thing can actually have big impact. That energy was transmitted to the company, with Jeff encouraging every employee to take part in a Field Day. We now have our Principal Researcher, Donna Driscoll, dedicated to running Field Days, so that we can include as many of our colleagues as possible.

CONCLUSION

On a daily basis, our team’s aim is to prove measurable product success based on our work. We’ve made progress as change-agents by working collaboratively; engaging product managers, marketers, developers, writers, data scientists and designers at a grassroots level; developing iterative design processes; telling compelling stories; and organizing cross-functional events that “go viral” within the organization. Often, research success, while contributing to business goals, has the added benefit of introducing empathy and removing people from the comfort of their “bubbles.”

The little guy besting the behemoth is an ancient trope. It’s hard to feel anything less than honored, and a bit amused, to contribute to this narrative. We’ve had fun with this construct during this paper. But in reality, keeping up the deceit required setting up a dichotomy that is a bit disingenuous. The David and Goliath metaphor actually falls apart upon examination, for our aim with our empathy missiles is not to take big data down. The effect on our colleagues, teams and executives is not harmful at all, nor is it experienced or celebrated at the expense of the value of quantitative data and behavioral metrics.

The truth is that, in close collaboration with our sophisticated big data colleagues, we continue to evolve the interplay between qualitative, quantitative and big data as an organization. One of LinkedIn’s Principal Data Scientists, Xin Fu, himself a former User Experience Researcher, put it well: our collective goal is to create “successful end-to-end stories that demonstrate the power when you combine the best of both camps.”

The success of our UED Research team in a data-driven organization ultimately requires shifting the narrative to one of data-parity. Moreover, this shift is required to enable our design team to function at maximum potential, to best serve our members. We must understand, and we must measure.

Being open and a bit self-deprecating ultimately creates an environment friendlier to our work, so that when we detect that teams are making too many assumptions or missing opportunities we can play that card, and work with teams to uncover mysteries, and discover surprises and insights that big data alone can’t address. We anticipate with glee the companion paper from one of our Data Sciences colleagues, describing their close encounters with folks on our team, and techniques they use to bring us around.

Julie Norvaisas is the Manager of User Experience Research at LinkedIn. She is focused on growing the team and developing a member-centered practice that informs, infuses and inspires designs to connect the world’s professionals to make them more successful and productive. Before joining LinkedIn she applied the principles and practices of design thinking around the world, delivering insights to teams working on products that ranged from toilet paper to hospital medication management distribution systems. She majored in Art History.

Jonathan (Yoni) Karpfen is a Senior User Experience Researcher at LinkedIn. He leads UX research for the company’s monetization group, collaborating with product and marketing teams to deliver business solutions for enterprise hiring, marketing and selling needs, as well as consumer solutions for job seekers and aspirational professionals. Yoni’s consumer insights and user experience work has spanned diverse topics and industries, from federal government policy making to online and mobile gaming.

NOTES

The views expressed in this paper do not represent the official position of LinkedIn Corporation.

¹ LinkedIn Research and Marketing functions operate under very strict privacy and terms of service standards when contacting our members to participate in studies.

² The LinkedIn Job Search App officially launched in the Apple App Store in June 2014