Product teams, including those I work with, struggle to connect the challenges observed in prior research to issues that endure in the field and market space. As a shortcut for efficiency gains, product partners rely on researchers to succinctly summarize deep insights, sometimes preferring reductive quantitative interpretations to enable a bias toward action in product development cycles. Challenges facing researchers in product development include maintaining the relevance of prior research, providing a way to make it evergreen and accessible, and building on it to deepen and expand an existing model of behavior. This case introduces the concept of Research Amnesia, which poses a threat to organizational resilience. Using core ethnographic methods, a strategic methodological approach is outlined to frameshift the value of existing research within a company to develop new insights, bring together disparate analyses and teams, and propel product partners forward by offering more questions as a means to answers. Keywords: mixed-methods, organizational culture, resilience, strategy, research amnesia, Reddit
INTRODUCTION
During the past four decades of technology product development advances, competitive market dynamics have demanded that product teams navigate the delivery of high-quality, low-cost, differentiated products with increasing speed and flexibility (Takeuchi & Nonaka, 1986). With mantras like “move fast and break things” (Zuckerburg, 2012) in the background, product leaders and teams at software technology companies seek to make efficiency gains where possible to reduce costs, either materially or in time spent. The first step of the product strategy process is to identify and understand customer needs using existing research and it is sometimes overlooked in the effort to develop and deploy a competitive solution quickly. The public’s increasing intolerance for consumer product innovations that ignore societal ramifications, and the paradigm shift away from minimum viable products to minimum virtuous products (Taneja, 2019), require that product development companies genuinely understand their target consumers and market niche.
Using previous user insights in a company’s memory avoids ‘reinventing the wheel’ for product solution development as a viable alternative to what are considered slow, overly complex, and resource-heavy new user research projects. A challenge that product development teams sometimes experience is Research Amnesia, where team members forget user knowledge that has been developed or do not know how to apply insights across studies on a current topic facing the team. As Pollitt (2000) suggests,
in the world of management, stress has been placed on innovation and change rather than stability and precedent, on creativity rather than experience, on envisioning the future rather than studying the past, on sound bites and keywords rather than full texts (p. 5-6).
With inherent complexities to deliver a successful product that departs from the norm, teams may prioritize other aspects of the product development process that privilege innovation rather than optimization and skip exploring or refreshing available user knowledge in the company.
Product launches can fail when they proceed without either drawing from existing research to support the product-market fit or conducting research to test the product concept iteration with consumers (Schneider & Hall, 2011). The impact of misunderstanding the user from knowledge already gleaned can equate to substantial missed revenue (Manning, 2016) and product failure (Schmidt, Lyytinen, Keil, & Cule, 2015; Dwivedi, Ravichandran, Williams, Miller, et al., 2013). For instance, there are several consumer technology products that support messaging. Meta developed Facebook Messenger and acquired WhatsApp, Apple developed iMessage, Salesforce acquired Slack, Tencent developed WeChat, and Microsoft developed Teams, and yet Google does not own a competitive product offering (CNBCTV18.com, 2022). During the past 17 years, however, Google launched and sunsetted a number of messaging applications, including Wave, YouTube Messages, Allo, Spaces, Talk, Hangout, Meebo, and Buzz (Amadeo, 2021; Ogden, 2022). Misunderstanding the users’ needs, behaviors, or motivations can induce product failures. More granularly, on a product feature level, Meta’s reversal of full-screen videos and images on the Instagram feed could be seen as a case of Research Amnesia. Instagram’s substantiated value proposition with users is for social photo posts. The full-screen video changes to Instagram’s feed were made in a wave of design modifications to reflect explicit knowledge around shifting user behavior toward watching video and to compete with TikTok (Newton, 2022; The Wall Street Journal, 2022). Adam Mosseri, Head of Instagram and trained designer, programmer, and product manager, stated: “For the new feed designs, people are frustrated and the usage data isn’t great…I think that we need to take a big step back, regroup, and figure out how we want to move forward” (Newton, 2022). Launch failures can result from lack of evaluation research or forgetting existing research that reinforces a product’s value to users. Consumer backlash and a product rollback may force a team to reinterpret signals around usage patterns in conversation with existing research.
Challenges that face researchers as partners in product development include maintaining the relevance of prior research, providing a way to make it evergreen and accessible, and building on it to deepen and expand an existing model of behavior. As a means to assist in preventing such catastrophic ends to the expensive undertaking of product development for hypergrowth organizations (Izosimov, 2008), this paper proposes a template of cost-effective, in-depth, and resilient research. A strategic mixed-method approach and report structure is outlined to frameshift the value of existing research to develop new insights without reducing depth, bring together disparate analyses and teams in service of product strategy, and propel multiple product partners forward by provoking curious inquiry to design successful products.
Product Development in Software Companies
In lean startup culture (Olsen, 2015), new product and business priorities emerge quickly and can fade the relevancy of data and stories from the product development mindset. Product managers have several factors and partners to consider when managing a team and timeline to launch a product. Product team communication with team members or partners, the pace of work needed to compete, and company structures all contribute to the context within which research investment, both in terms of time and money, is resourced.
Product development teams make trade-offs between user experience, technology capabilities, and the company’s business goals to achieve success (Eriksson, 2016; Banfield, Eriksson, Walkingshaw, 2017). Within user experience, product managers collaborate with user researchers, data scientists, and designers to understand user needs and conceptualize solutions that provide value. Product managers work with individual contributors (ICs) and people managers across roles like software engineer, systems architect, quality assurance analyst, and program manager, to implement product requirements into code. The company’s business goals as well as compliance with regulations require that product managers communicate across product marketing management, data analysts, finance, legal, and sales to launch a product or feature to the intended market. All in all, product managers make compromises with cross-functional partners across several expertise areas to successfully deliver a product or feature and contribute to a company’s goals.
For social media companies, the pressure to be a pioneer and act on first-mover competitive advantage, and the requisite speed of product launches is intense (Going, 2017), though second-mover advantages include reduced costs due to lessons learned from others’ mistakes (Shankar & Carpenter, 2013). The pace by which a product team might complete product milestones depends substantially on where a company is in its stage of growth, how well the team understands the user needs, the product strategy for solutions, and the velocity of its developers. Developer velocity is a metric for work done in agile software development (Rubin, 2013), measuring the speed of progress in a development cycle as a result of empowerment, an environment for innovation, and the removal of points of friction. Increased velocity is often a measure of success for a well-performing product development team, and according to a study by McKinsey, has an outsized impact on a company’s profit margin (Srivastava, Trehan, Wagle, and Wang, 2020). Velocity tracking does not actually reflect efficiency or success, however, and solely represents the amount of work accomplished (Agile Alliance, n.d.). Developer velocity could increase as a result of additional people without improving the product outcomes or process efficiency.
A company’s structure is critical to consider in tandem with the process of product development and consequent research resourcing. As Conway (1968) described, the structure of a product mirrors the communication and social boundaries of the people that produce them. A company’s life cycle and evolution can be viewed in many ways, though the growth progression of organizational structures moves from startup to emerging to enterprise (Banfield, Eriksson, Walkingshaw, 2017; Wylonis, 2021). New companies require product development teams to adapt to a rapidly changing environment as the company matures from its founding to demonstrate traction with a viable product-market fit for its product. In an emerging company, product teams coalesce around scaling and coordinate as quickly as possible for growth, perhaps amidst a series of funding rounds and the expansion of teams. Established corporations offer a balanced environment of innovating new products and optimizing existing products to meet customer needs, sometimes acquiring other companies as the complexity of the company’s structure accelerates. Technology products, similar to company structures, also rarely stay the same during a company’s evolution. Features of technologies adapt or reify structures over time in response to the way that people use them together (DeSanctis & Poole, 1994). As a demonstration of Conway’s law, product technologies shift as a company undergoes reorganizations. Within hypergrowth companies, constant structural changes create a complex environment for strategic research and its impact.
Organizational Memory as Change Agent
Organizations evolve by creating new knowledge (Chen & Edgington, 2005; Nonaka, Toyama, & Konmo, 2000; Rusaw, 2005), and organizations can learn by reflecting on memories of specific events and contexts (Rowlinson et al., 2010; Walsh & Ungson, 1991). Knowledge is a type of organizational memory (Rowlinson et al., 2010; Walsh & Ungson, 1991) and the result of amalgamating distinct data into useful information and insights through analysis (Tuomi, 1999). Knowledge can be explicit, which explains when, what, or how much and is easily quantified and recorded, or tacit, which explains how or why and is harder to quantify or record because it includes experiences, learning by doing, or personal beliefs (Anand, Manz, & Glick, 1998; Feldman & Feldman, 2006; Goh, 2002; Nonaka, 1994; Olivera, 2000). Organizational memory is shared understandings and beliefs (Tuomi, 1995) that are stored in individuals, culture, transformations, structures, ecology, and external archives (Walsh & Ungson, 1991). Product teams and software organizations can draw on product knowledge from memories of past research to face new challenges and facilitate learning (Coffey & Hoffman, 2003; Pollitt, 2000; Tuomi, 1995; Walsh & Ungson, 1991). Organizational amnesia, or the declining ability and willingness to make use of possibly relevant past experiences (Pollitt, 2000, p. 6), can inhibit a company’s evolution. Pollitt (2000) outlines a range of four situational types for this forgetfulness: (1) significant data or decisions are not documented, (2) records are lost, (3) archives cannot be accessed quickly, and (4) records are available and accessible but no one thinks of using them, partly due to attitude or mentality against recourse to the past. Pollitt’s four reasons assume good intent possibly because of his focus on the public sector. This paper builds on Pollitt’s foundational situations for forgetfulness by adding another: (5) willful misdirection that conveniently obfuscates or overlooks past records for purposes of subterfuge.
Managers and producers of research within a company control how memory is recorded, disseminated, and used (Casey & Olivera, 2011). If memories are not effectively maintained, they will likely be lost (Pollitt, 2000) and the organization is unlikely to generate new knowledge (Bhardwaj & Monin, 2006; El Sawy, Gomes, & Gonzalez, 1986; Chinying Lang, 2001; Walsh & Ungson, 1991). Therefore, to progress toward its growth potential, it is a company’s imperative and management’s prerogative to build on existing knowledge in its memory. Though there is debate around whether a product manager should be framed as the CEO of a product (Horowitz, 2012; Eriksson, 2017), the product manager is centrally responsible for leading a product team through a product or feature’s development. Product managers gather resources to identify the customer needs and larger business goals that a product or feature will fulfill, articulate a plan for success, and coordinate a team to realize that vision (Mansour, n.d.). People of several different expertise areas work together under the guidance of a product manager in technology companies to “discover a product that is valuable, usable and feasible” (Cagan, 2017). They are also the co-managers with researchers of prioritizing organizational memories, choosing between what might be applicable or not to the product development.
Product Development Knowledge. Product managers may choose to compromise on investments in decision-making when faced with the pressure of rapid delivery and the competing complexities of working across multiple groups to launch a product. Product teams may follow different paths of varying complexity through identifiable phases when aligning on the product strategy, including problem identification, solution development, and solution selection (Burnstein & Berbaum, 1983). Structures that impact product direction can be team members’ understandings of the product problem, its scope, the approaches to make the best decision under the circumstances (i.e., decision logics), and external factors such as timing or resources (Poole, 2013; Poole, 1985; Poole & Doelger, 1986). The steps of deciding on product strategy as a team (Poole & Doelger, 1989) allow for multiple combinations and paths to be taken by groups in their decision-making to solve the problems they face (Poole & Roth, 1989). The role of the researcher in technology product development teams is essentially to retrieve or produce information to assist with decision making, including generatively around problem discovery or solution discovery, or evaluatively around solution selection. Within the frame of decision making models, researchers essentially perform but are not limited to the functions of problem analysis and evaluation to address customer needs and product solution guidance, elaboration, and evaluation. Product managers and researchers co-own the responsibility to incorporate research into product development knowledge, and researchers aid in surfacing relevant knowledge during the product development process for broader organizational learning.
Research Amnesia. Research Amnesia describes the behavior of fast-moving product teams that do not consistently remember research or revisit research, or are ill-equipped to apply the relevance of existing research in problem-solving efforts at a point sometime after the research was produced. An organization may not support infrastructure that perpetuates research, such as the technical service of a centralized knowledge base with clusters of information (Ackehurst & Polvere, 2020), the value of people supporting each other in a reference culture (Wilson, 1999), or even the practice of citing previous documents or artifacts. Teams may misremember or partially recall a finding from past research and move on with faulty assumptions. The sheer number of moving parts in the product development process pushes product teams to overlook or miss valuable learnings and common themes from previous research findings, and instead, new research is requested (see Figure 1). The outcome of Research Amnesia potentially threatens the resilience of the organization as a whole with the propagation of short-term thinking alongside an overcorrection into total innovation and a rejection of existing ideas or knowledge, rather than optimization and repurposing of existing insights. When teams operate with Research Amnesia, it represents an underlying confusion of product innovation with research innovation.
Figure 1. The Attractiveness of New Research vs. Old’ Research Meme (Hoang, 2021).
© David Hoang, used with permission.
In seeking to understand customer needs, teams and companies may face obstacles to generating knowledge from research in the organization’s memory. Following Pollitt’s (2000) four circumstances of memory loss, insights may not be documented and the team may not cull tacit knowledge of customers’ experiences from more tenured team members (Alwis & Hartmann, 2008). Prior research on user experience could be unavailable or lost. The team may not have access to an efficient system or repository to transfer formal insights about user experiences (Pollitt, 2000). Fourth, the product team may not use available research because they perceive existing findings as irrelevant or inapplicable to the new context, or because they believe they already have the answer when they may have misremembered or partially remembered the findings.
For non-researchers on the team, prior research findings may also become fixed to a certain, outdated time and place that product objectives have moved past as the company focuses on several factors in scaling for the future (Barquin, Dreischmeier, Hertli, Köningsfeld, & Roth, 2020). Similarly, the common perception of software documentation as outdated among software engineers and product managers might have carry-over effects to other forms of documentation, like research. In one study, nearly 70 percent of participants agreed or somewhat agreed with the statement “Documentation is always outdated relative to the current state of a software system” (Lethbridge, Singer, & Forward, 2003, p. 36). The meaningfulness of user stories is unremembered or untranslatable to new contexts. Teams struggle to connect challenges that surfaced in prior research to the current context and these insights are viewed as inapplicable.
Another pernicious influence working against the use of available research, and a fifth situation of forgetfulness introduced in this paper, may be the researcher themselves occluding information for a perceived research opportunity. For instance, if a researcher discovers an existing set of findings that thoroughly answer the research questions for the team, their duty to cite the findings for team progress may conflict with their desire to carry out their planned study for a variety of reasons, including method skill attainment, visibility, career progression, travel, or other outcomes of self-interest. Superfluous citation manipulation as a practice has been well documented (Fong & Wilhite, 2017). Less explored are the murky ethics of omitted citations (Penders, 2018) as deceit along the way to achieving a goal. In environments that foster Research Amnesia, and where researchers alone are the purveyors of insights, the control of information may create opportunities too tempting to resist the pursuit of personal gain, deviating from the team’s outcomes. In the information flows typified in Table 1 by the Luft and Ingham’s Johari window (1955) and an organizational analysis of facts and risks in Table 2 presented by Rumsfeld (US Department of Defense, 2002; Krogerus & Tschäppeler, 2018), a researcher may project a façade of “unknown knowns” and create research questions to support their goals, when in reality, the team could progress with an arena of open knowledge or “known knowns.”
Table 1. The Johari Window: Types of information flows in relationship to the self and others.
| Known to self | Not known to self | |
|---|---|---|
| Known to others | Open | Blindspot |
| Not known to others | Façade/ Hidden | Unknown |
Table 2. The Rumsfeld Matrix: Analysis of facts and risks in organizations.
| Known | Unknown | |
|---|---|---|
| Known | Known-Knowns/ Facts and Requirements: things we are aware of and understand |
Known-Unknowns/ Known Risks: things we are aware of but don’t understand |
| Unknown | Unknown-Knowns/ Hidden Facts: things we understand but are not aware of |
Unknown-Unknowns/ Unknown Risks: Things we are neither aware of nor understand |
On the rare occasion that past research is remembered accurately by product development teams, quick translations via reductive, quantitative interpretations are preferred to extrapolate previous findings to their new context and motivate swift action. Quick translations to new contexts create opportunities for further muddling and misremembering of research later. Regarding Research Amnesia, researchers face challenges such as maintaining the relevance and depth of prior research, producing research that remains evergreen and accessible, and the effort of building on prior research to deepen and expand models of behavior. The following case provides an illustration of one approach at Reddit known as The Book of Insights project that documented analyses and provided a narrative of understanding, brought together records that might have been lost, made those analyses accessible to all, and socialized the new tools to heighten the value of past findings for future action in the hands of our partners.
RESEARCH APPROACH
Reddit, an emerging social media company, experienced hypergrowth in 2020. The company needed a way to document analyses and highlight the rush of insights produced by multiple teams, and a way to place these findings in dialogue with each other to generate knowledge and advance the organization’s evolution. At the time, Reddit had a number of groups and independent contributors across the company generating analyses without coordination, sometimes repeating the same questions. Some of these groups included Advertising Products, Communications, Community, Data, Design, Marketing, Product, Safety, and User Research teams. Without the ability to access insights across groups, significant findings were forgotten by the organization. Results were trapped behind team-specific documentation processes, not written down, not vetted to a standard degree of execution, and rarely did product teams chase down the findings, leading to an increasingly alarming trend of product teams not utilizing the findings. Additionally, some teams were convinced the older findings did not coherently translate to the current context they worked in, and the analyses were overlooked.
Under the banner of The Book of Insights, a research team of three – a product management intern, an early career user researcher, and a senior manager of user research (the author) – was established. The mission of the team was to iterate on the existing idea of The Book of Insights, which in the previous three versions had selectively gathered a few research studies and was led by a central product manager. The expectation from the Executive sponsor of this project, a new version of The Book of Insights led by User Research, was for an expansive and comprehensive summary of insights from 23 teams across the company featuring takeaways. As a company of 500 employees doubling in size, Reddit demonstrated Pollitt’s (2000) four situations of forgetfulness, including a lack of documentation, lost records, inaccessible archives, and an attitude preferring fresh insights over evergreen insights. When we began the project, we were not yet sure that subterfuge played a role in obfuscated archives. The intended outcome of The Book of Insights was to reduce the chance of Research Amnesia at Reddit and herald evidence-based product decisions that evolved rather than reinvented the wheel. The Book of Insights set out to accomplish four goals:
- Bring insights from analyses across the company that all employees can use to make informed decisions.
- Create a centralized repository of research completed between February 2020 through June 2020 across several teams producing insights.
- Identify insights from multiple sources, including analyses, academic publications, logged data, news articles, survey data, and interview data.
- Create a thematic analysis of all reports and identify emergent insights.
We collected and reviewed 123 analyses across five months from 57 internal authors and synthesized their analyses into eight major insights with associated artifacts and “how might we?” questions for teams. The Book of Insights used a combination of five data gathering methods in phases, including archival data collection, a survey questionnaire, open-ended interviews, a literature review, and logged data analysis. Our assessment of insights was determined through archival sources analyses, academic publications, logged data, news articles, survey data, and interview data (see Table 3). A qualitative thematic analysis was performed to produce emergent themes and categories of insights (Saldana, 2014). Corresponding original sources were compiled into a newly established knowledge repository.
Table 3. Mixed-methods approaches for collecting and aggregating diverse insights across Reddit for product strategy research in The Book of Insights.
| Research Approach | Goal | Impact |
|---|---|---|
| Archival research | Collect analyses completed between late 2019 through June 2020 across teams, including Advertising Products, Communications, Community, Data, Design, Marketing, Product, Safety, and User Research teams. | The Resource Matrix, a comprehensive and centralized repository of previous research and knowledge management source for the entire Reddit organization, provided a source of truth to find previous research. |
| Survey | Efficiently collect information on analyses gathered in standardized documentation that asked about the impact, value, and relevance of findings and supported shared ownership of accuracy. | Identification of prior work most valuable for making decisions. |
| In-depth interviews | Interviews were scheduled at the request of employees who had either produced multiple reports or otherwise made the request in lieu of the survey. For those employees with multiple analyses, an in-depth interview format provided the opportunity to seek depth across the various analyses they had produced. | Deepened the insights for identifying more valuable research, and reaffirmed the context of previous work. |
| Literature review | Academic publications that mapped to insight categories were condensed into academic insights and rephrased into the narrative. | Increased credibility of research insights by grounding the takeaways in the science of human behavior, needs, and motivations. |
| Logged data | Logged data was pulled from telemetry to contextualize the insights and highlight the growth Reddit had experienced between the first half of 2019 and the first half 2020. | Aligned teams on relevant outcome metrics and provided a frame for qualitative insights in relationship to quantitative data. |
Phase I: Archival Data
First, we began by gathering analyses produced by partners across the company from February 2020 when the previous volume was published through the end of June 2020. In exploring the analyses, we realized that there were relevant analyses prior to that timeframe that had not been captured in other volumes. Consequently, our selection criteria expanded to be inclusive of analyses released from late 2019 forward. We searched for and collected analyses within the following 11 internal company sources, such as reports from the Experiments forum, the Consumer Product organization, the Design organization, and the Community Initiatives team; a Safety database; the User Research shared folder; internal subreddits for the User Research team, the Analytics team, and Reddit; emails; internal chat posts, and other miscellaneous resources.
Following archival data collection, we conducted an initial round of analysis of these digital artifacts (Boellstorff, Nardi, Pearce, & Taylor, 2012; Ladner, 2014). We performed a qualitative grounded thematic analysis and used the constant comparative technique to categorize the findings and create themes of insights. We initially grouped linked analyses with notes under initial themes by product area (e.g., Search), product feature (e.g., push notifications), role (e.g., Moderators), or event (e.g., the COVID-19 pandemic).
Phase II: Survey Data
As a means of facilitating further data collection, we sent a survey to nine cross-functional teams across the company, including Ads Product, Comms, Community, Data, Design, Marketing, Product, Safety, and User Research, for a few reasons:
- Efficiency: Lots of partners created analyses—we were collecting information on the initial analyses gathered, a total of 69 reports by 29 employees.
- Standardized documentation: Analyses varied in depth and explanation.
- To ask partners about impact, value, and relevance: These takeaways were sometimes excluded in the analyses, given their original purpose.
- Supporting accuracy: A survey provided partners the opportunity to voice directly the contextual meaningfulness of their analyses.
The survey asked for the name of the respondent, a link to the analysis, and ten questions (see Table 4). We encouraged partners to copy analysis report information into the survey item text fields as a shortcut, and also to confirm the analyses’ findings. The survey was sent to 29 employees who had created 69 analyses that were found during the initial archival data phase and warranted more information to effectively interpret the reports. We received survey responses from 18 employees regarding 26 analyses from across the company, amounting to an initial 62% response rate and a 38% collection rate. During ensuing conversations with employees, we also discovered several more existing analyses in the initial round of survey recruitment and distributed subsequent surveys beyond the initial 29 employees to capture additional data.
Table 4: Data Collection Questions in Reddit’s Book of Insights Survey and In-depth Interview
| Survey Questions | In-Depth Interview Questions |
|---|---|
|
|
Phase III: In-Depth Interview Data
Given the launch of a new method of information collection for The Book of Insights, and an initial low collection rate, we adapted our methodology to include a round of interviews. Interviews were scheduled at the request of employees who wanted a conversation in lieu of completing the survey, which could have been willful obfuscation of previous archives, though our aim was the collection of information rather than assessment of intents in refusing the survey. We also initiated scheduling interviews with employees who had produced multiple reports because the format provided the opportunity to seek depth across the various analyses they had produced. To standardize the analysis of Reddit-wide insights, a truncated version of the survey items was represented in a questionnaire with only six items. Nine interviews were conducted with employees across the nine teams, including Data, Product, Community, Safety, International, and Marketing, and 54 additional analyses were reviewed in total. Interviews were recorded and discussion notes were taken to clarify points of data and interpretation and used in the later thematic analysis as reference.
After the survey responses and in-depth interviews were complete, we added several more rounds of analysis by incorporating extensive notes and details and began to write. We created accompanying documents with relevant meeting notes, survey responses, and summaries. The team combed through the gathered details and sought to concisely summarize each analysis’ artifacts. We conducted qualitative thematic coding, examining nuances in the report findings, refining categories, and crafting a more cohesive narrative that brought together distinct insights across different teams (Saldana, 2021). We reexamined the placement of analyses within initial themes, repositioning and creating new, emergent themes along the way.
The new themes aimed to synthesize company business unit functions and areas around a user-experience focus, and better represent insights for Reddit as a whole. Insights such as, “great products are often inspired by users, not requested by them” and “the business and the users move together” were used to frame large bodies of research insights. As a knowledge piece of organizational memory, the framework employed to shape The Book of Insights was crucial to develop. I wrote our headline insights by juxtaposing the themes to values that drive newsworthiness in journalism (Walsh, 2017), contextualizing their merit against timeliness of current events, proximity to company values, impact to users, unexpectedness, relevance to company strategy, and conflict of perspectives. We affinitized research findings around common themes that supported the company’s direction, emphasizing insights in alignment with company values, and providing multiple groups visibility in the top insights. The first draft of The Book of Insights was complete (see Figure 2).
Figure 2. Example insight from The Book of Insights. © Reddit, used with permission.
Phase IV: Literature Review
User Research summarized daily and shared externally-produced academic publications on internal research chat posts to the company about product considerations to assist in driving evidence-based product development. Academic publications related to Reddit, Reddit-related product offerings, and user attitudes and behaviors were distributed with a link to their PDFs on an internal cloud-based drive. We integrated academic insights into The Book of Insights by:
- Identifying academic publications that mapped to insight categories
- Reviewing publications and drafting condensed academic insights
- Rephrasing insights into the Reddit insights narrative
The academic insights process was iterative as The Book of Insights draft evolved over time. All academic papers referenced in The Book of Insights are available through the User Research library, a company-wide resource that was updated on a weekly basis.
Phase V: Logged Data
In an effort to contextualize the insights showcased in The Book of Insights, and highlight the growth Reddit had experienced between the first half of 2019 and the first half of 2020 (i.e., January through June each year) and to coincide with the date range of analyses included in The Book of Insights, logged data was collected from eight teams and their dashboards. The goal of using logged data to begin The Book of Insights was to introduce new metrics of growth and demonstrate the relevance of the insights to these product areas. Previously, these growth metrics had not been brought together in one place, and gathering them together helped to emphasize the potential impact and relevance of the findings to business objectives for product managers. Each team was asked for a topline descriptive statistic relevant to their product area, and a corresponding data dashboard that could be accessed by all employees. The consolidated logged data snapshot included user-centric (e.g., the number of average daily active users or commenters) and activity-centric statistics (e.g., the number of posts or searches made). For each of these fields, we calculated the year-over-year (YoY) percentage increase seen between 2019 and 2020. Logged data anchored The Book of Insights in the magnitude of growth that Reddit has supported across a spectrum of initiatives and products.
To finish our analysis and additions to the report, we added the academic insights to the narrative already crafted, referenced the insights from previous volumes of The Book of Insights, and outlined the topline product area statistics. Through iterative analysis, we created four drafts of The Book of Insights before arriving at the final draft. In the drafting process, we met with product leadership and shared in-progress work. Feedback was incorporated into the intermediate drafts, along with accompanying recategorizations and refining of the larger narrative.
BUSINESS IMPACT
The new version of The Book of Insights changed the way that the Reddit departments producing analyses worked together to retain knowledge. We presented on The Book of Insights, posted the artifacts in announcements on our internal company subreddit, an email, and a chat system, and provided links to the report documents, the repository, the presentation slides, and the presentation recording. Product became more vested in understanding patterns of findings across research studies and seeking out multiple methods to validate assumptions about user needs or behaviors themselves. Rather than forgetting what research had come before, product teams had two resources that helped re-circulate and build on existing research at Reddit. The Book of Insights Vol. 4 shed light on an extended discussion of eight key insights supported by a variety of data sources and built a model for the next volume. A few insights were commonplace by product development standards but were responsible for radically shifting the product development mindset at the company.
The Resource Matrix, as an ongoing repository of analyses at the company, provided an evergreen resource. The Resource Matrix is a comprehensive and centralized knowledge management source for the entire Reddit organization. The Resource Matrix sheet is a sortable, exportable database that can be referenced in future versions of The Book of Insights. The Resource Matrix brings together valuable analyses in one accessible place to empower decision-making and conversation with partners across the company. The Resource Matrix itself identifies and consolidates links to internal analyses, academic publications, news articles, and internal team hubs across all four volumes of Books of Insights. Internal analyses and academic publications tabs provide the following information fields for each entry:
- Date distributed or published
- Title
- Team or journal/publication
- Owner or author
- Book of Insights volume
- Insight number
- Insight [text]
- Link
Analyses reviewed for future Books of Insights will be included in the Resource Matrix or its next iteration as part of maintaining this consolidated source of company-wide insights and knowledge.
For the first time within Reddit, research across the company was brought together in a cohesive and understandable way via ethnographic methodology. Product partners referenced the eight larger insights with supporting evidence to develop product roadmap strategies. Conversations began among teams about the company’s current state, its trajectory, and users in this journey, improving organizational resiliency during a time of remote, distributed work-from-home culture.
Reflection
Upon reflection, the two months of work undertaken to produce these insights posed some challenges. First, there was no direct, centralized way to access information produced by different teams at the company. Teams that produce analyses prefer different methods of storing and sharing outputs, do not have a normalization of unprompted sharing of information between teams beyond formalized settings, and use different places and tools to store information, which creates implications for access control. These pose central challenges to the relevance of research and organizational resiliency to solve problems and move on from them. The company benefited from a central resource of insights, though its ongoing maintenance required further investment and helped to usher in the establishment of Research Operations at Reddit.
A persistent myth that should be addressed directly is the fantasy of the research repository – or the naïve idea that a centralized resource to retrieve reports and analyses will solve all organizational knowledge problems, provide an easily discernable roster on what is known or not known, and serve as an endless source for an organization to draw from to propel its own evolution. Overall, the question of whether a research repository like the Resource Matrix is a wise investment depends on who will use it and how it will be used. Occasionally research leaders determine little value in establishing a research repository because teams default to asking research colleagues to function as librarians, retrieving findings for them as a shortcut rather than searching for information in a repository themselves. The Resource Matrix has been useful as a catalog repository for researchers, and it has aided in the ability to cite across projects and advance teams past basic product questions. The centralized repository has assisted the researchers at Reddit in staving off Research Amnesia and in their roles as partners to keep the organization evolving and learning. A research repository on its own will not save an organization from Research Amnesia. Implementing a repository at an organization is as much about introducing a new tool as it is about establishing boundaries, expectations, and roles and investing in cultural change around the process of using research effectively. Similar to any system built for success, a research repository must have early champions and quick wins to illustrate the larger, long-term value in its vision. Partners must understand and feel confident enough to self-service retrieval of information, comprehend what they discover, and then enact critical thinking to enable application and propel action.
The second challenge arose when many partners who created analyses realized the obstacles around communicating meaningfulness and impact in their own analyses as we attempted to collect and understand them. Devoid of context, the gravity of particular analyses and their contributions to the company outcomes or goals could be overlooked. The executive sponsor of The Book of Insights team asked us to provide feedback to other teams about how to structure their individual reports to enable faster and deeper insights going forward.
The final challenge came about during the process itself: The Book of Insights requires a substantial amount of work in partnership negotiation as well as research analysis. Establishing a serial Book of Insights program with multiple volumes on a cadence that adds value to the company to guide product development strategy can drain the energy of the researchers for other projects. The synthesis task can overwhelm researchers, but its cost in effort should be balanced with its ability to provide thought leadership without costly research investments in new data collection.
Although this case prioritizes impact on the product development teams, The Book of Insights fostered other teams’ use of the findings and takeaways for other goals that Reddit pursued, including some in Communications (such as the Year in Review publication), Marketing, Sales, and other departments. In terms of impact, one product director commented,
You all crushed it! I was very proud of the team. The reception from the company was fantastic, and it was wonderful to see so many enthusiastic questions. Let me know how I can be supportive of this effort in the future.
As a team, we were excited to learn that the success of the project also supported the conversion of one of our teammates from an intern to a full-time employee position at the company. By distributing the documents in an accessible repository, presenting the findings, and empowering partners to ask questions, The Book of Insights project elevated the entire research team enterprise as a powerful voice for interpreting findings across the company into guidance for product and business strategy.
CONCLUSION
The Book of Insights case provides a template for preserving and documenting findings from multiple analyses produced across Reddit. This paper outlines the context of product development in hypergrowth tech companies and the challenges faced by product teams and researchers in these contexts. Research Amnesia sets in and teams forget or believe previous research does not apply to contexts facing the product team. Companies undergoing change may exhibit forgetfulness around research due to a lack of documentation around data or decisions, missing records, archives that cannot be accessed, or simply not thinking to use artifacts from the past (Pollitt, 2000), or as this paper contributes, subterfuge for reasons of opportunistic gain. With limited time to allocate during product development, partners seek time-saving shortcuts to bypass deeper investments required for strategic research. Commonly perceived time-saving shortcuts in research include quantitative summaries of existing, rich qualitative research; new research on a timeline that affords immediate tactical value rather than necessary strategic depth; nonexperts undertaking basic research (Takeuchi & Nonaka, 1986), which has most recently been coined as research democratization (Pernice, 2022); or reliance on anecdata (i.e., anecdotal evidence) in lieu of research altogether. The gains in a product development timeline from neglecting existing research perhaps allows for a quicker iteration of solutions and launch of a novel product to market, or an increase in the number of products launched, often a marker of success. To prevent Research Amnesia, The Book of Insights documented analyses and provided a narrative of understanding; the Resource Matrix collated records that might have been lost and made them accessible to all; and the process of presenting and socializing The Book of Insights and Resource Matrix changed the way partners understood the value of past findings for future action and reduced the potential for willful misdirection. Research Amnesia is a common problem among organizations of all types, and The Book of Insights project provides one approach to fighting the inner voice who says upon hearing a new challenge, “we don’t know anything” to moving to explore what we do know and confidently move beyond starting at square one.
Kristen Guth, Ph.D. is the Principal of Product Research for Reddit, Inc. She built the Reddit User Research organization and conducts mixed-methods research to create user experiences that align product and business goals. Kristen is a social scientist research leader focused on change at the intersections of strategy, technology, innovation, and the digital space. kristen.guth@reddit.com
NOTES
The author would like to acknowledge the assistance of team members Megan Lee and Trisha Shetty, a product manager (an intern at the time on this work) and a user researcher at Reddit, who worked with the author to execute The Book of Insights project at Reddit. The author would also like to thank Product and Design leadership, including Alex Le and Lowell Goss, for their sponsorship of this project at Reddit.
REFERENCES CITED
Ackehurst, Maree, and Rose-Anne Polvere. 2020. “Evolutional Librarianship: From Supermarket to Smorgasbord.” Journal of Web Librarianship 14, no. 3-4: 74-85. https://doi.org/10.1080/19322909.2020.1823299
Agile Alliance. n.d. “Velocity.” Agile Alliance website. Accessed September 20, 2022. https://www.agilealliance.org/glossary/velocity
Alwis, Ragna Seidler-de, and Evi Hartmann. 2008. “The Use of Tacit Knowledge Within Innovative Companies: Knowledge Management in Innovative Enterprises.” Journal of Knowledge Management 12, no. 1: 133-147. https://doi.org/10.1108/13673270810852449
Amadeo, Ron. 2021. “A Decade and a Half of Instability: The History of Google Messaging Apps.” Ars Technica website, August 25. Accessed September 20, 2022. https://arstechnica.com/gadgets/2021/08/a-decade-and-a-half-of-instability-the-history-of-google-messaging-apps/
Anand, Vikas, Charles C. Manz, and William H. Glick. 1998. “An Organizational Memory Approach to Information Management.” The Academy of Management Review 23, no. 4: 796-809. https://doi.org/10.2307/259063
Banfield, Richard, Martin Eriksson, and Nate Walkingshaw. 2017. Product Leadership: How Top Product Managers Launch Awesome Products and Build Successful Teams. Sebastopol, CA: O’Reilly Media.
Barquin, Sonia, Ralf Dreischmeier, Sascha Hertli, Jerome Köningsfeld, and Andrew Roth. 2020. “The Big Boost: How Incumbents Successfully Scale their New Businesses.” McKinsey Digital website, August 27. Accessed September 20, 2022. https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/the-big-boost-how-incumbents-successfully-scale-their-new-businesses
Bhardwaj, Meeta and John Monin. 2006. “Tacit to Explicit: An Interplay Shaping Organizational Knowledge.” Journal of Knowledge Management 10, no. 3: 72-85. https://doi.org/10.1108/13673270610670867
Boellstorff, Tom, Bonnie Nardi, Celia Pearce, and T.L. Taylor. 2012. Ethnography and Virtual Worlds: A Handbook of Methods. Princeton, NJ: Princeton University Press.
Burnstein, Eugene, and Michael L. Berbaum. 1983. Stages in group decision Making: The Decomposition of Historical Narratives. Political Psychology 4, no. 3: 531-561. https://doi.org/10.2307/3790872
Cagan, Marty. 2017. Inspired: How to Create Tech Products Customers Love. Hoboken, NJ: John Wiley & Sons.
Casey, Andrea J., and Fernando Olivera. 2011. “Reflections on Organizational Memory and Forgetting.” Journal of Management Inquiry 20, no.3: 305-310. https://doi.org/10.1177/1056492611408264
Chen, Andrew N. K. and Theresa M. Edgington. 2005. “Assessing Value in Organizational Knowledge Creation: Considerations for Knowledge Workers.” MIS Quarterly 29, no. 2: 279-309. https://doi.org/10.2307/25148680
Chinying Lang, Josephine. 2001. “Managerial Concerns in Knowledge Management.” Journal of Knowledge Management 5, no. 1: 43-59. https://doi.org/10.1108/13673270110384392
CNBCTV18.com. 2022. “The Google Graveyard: Why Almost All of the Tech Giant’s Messaging Apps End up Dead.” CNBC18 website, January 13. Accessed September 20, 2022. https://www.cnbctv18.com/technology/the-google-graveyard-why-almost-all-of-the-tech-giants-messaging-appsend-up-dead-12110792.htm
Coffey, John W., and Robert R. Hoffman. 2003. “Knowledge Modeling for the Preservation of Institutional Memory.” Journal of Knowledge Management 7, no. 3:38-52. https://doi.org/10.1108/13673270310485613
Conway, Melvin E. 1968. “How do Committees Invent?” Datamation 14, no. 4: 28-31. https://www.melconway.com/Home/pdf/committees.pdf
DeSanctis, Geraldine, and M. Scott Poole. 1994. “Capturing the Complexity in Advanced Technology Use: Adaptive Structuration Theory.” Organization Science 5, no. 2: 121-147. https://www.jstor.org/stable/2635011
Dumitrescu, Sorin. 2021. “What is Developer Velocity and How Do You Measure It?” Bunnyshell.com website, August 13. Accessed September 20, 2022. https://www.bunnyshell.com/blog/what-development-velocity
Dwivedi, Yogesh K., Karthik Ravichandran, Michael D. Williams, Siân Miller, Banita Lal, George V. Antony, and Muktha Kartik. 2013. “IS/IT Project Failures: A Review of the Extant Literature for Deriving a Taxonomy of Failure Factors.” In: Grand Successes and Failures in IT. Public and Private Sectors, eds. Yogesh K. Dwivedi, Helle Zinner Henriksen, David Wastell, and Rahul De’. IFIP Advances in Information and Communication Technology, vol 402, 73-88. Berlin: Springer. https://doi.org/10.1007/978-3-642-38862-0_5
El Sawy, Omar A., Glenn M. Gomes, and Manolete V. Gonzalez. 1986. “Preserving Institutional Memory: The Management of History as an Organizational Resource.” Academy of Management Best Papers Proceedings: 118-122. https://doi.org/10.5465/ambpp.1986.4980227
Eriksson, Martin. 2016. “What is a Product Manager?” Medium website, February 18. Accessed September 20, 2022. https://medium.com/@bfgmartin/what-is-a-product-manager-ce0efdcf114c
Eriksson, Martin. 2017. “Product Managers – You are Not the CEO.” Medium website, May 9. Accessed September 20, 2022. https://medium.com/@bfgmartin/product-managers-you-are-not-the-ceo-3441cb3914e4
Feldman, Regina M., and Steven P. Feldman. 2006. “What Links the Chain: An Essay on Organizational Remembering as a Practice.” Organization 13, no. 6: 861-887. https://doi.org/10.1177/1350508406068500
Fong, Eric A. and Allen W. Wilhite. 2017. “Authorship and Citation Manipulation in Academic Research.” PLoS One 12, no. 12. https://doi.org/10.1371/journal.pone.0187394
Going, Gretel. 2017. “Too Soon? How to Avoid a Tech Product Launch Fail.” Forbes website, April 14. Accessed September 20, 2022. https://www.forbes.com/sites/forbesagencycouncil/2017/04/14/too-soon-how-to-avoid-a-tech-product-launch-fail/
Goh, Swee C. 2002. “Managing Effective Knowledge Transfer: An Integrative Framework and Some Practice Implications.” Journal of Knowledge Management 6, no. 1: 23-30. https://doi.org/10.1108/13673270210417664
Horowitz, Ben. 2012. Good Product Manager/Bad Product Manager. Andreessen Horowitz website, June 15. Accessed September 20, 2022. https://a16z.com/2012/06/15/good-product-managerbad-product-manager/
Hoang, David. 2021. There are not enough user research memes. Twitter website, March 12. Accessed September 20, 2022. https://twitter.com/davidhoang/status/1370473756141875203
Izosimov, Alexander V. 2008. “Managing Hypergrowth.” Harvard Business Review website, April. Accessed September 20, 2022. https://hbr.org/2008/04/managing-hypergrowth
Krogerus, Mikael and Roman Tschäppeler. 2018. The Decision Book: Fifty Models for Strategic Thinking. New York, NY: W. W. Norton & Company.
Ladner, Sam. 2014. Practical Ethnography: A Guide to Doing Ethnography in the Private Sector. New York, NY: Routledge.
Lethbridge, Timothy C., Janice Singer, and Andrew Forward. 2003. “How Software Engineers Use Documentation: The State of the Practice.” IEEE Software 20, no. 6: 35-39. https://doi.org/10.1109/MS.2003.1241364
Literally Media, LTD. 2022. “Distracted Boyfriend.” Know Your Meme website. Accessed October 2, 2022. https://knowyourmeme.com/memes/distracted-boyfriend
Luft, Joe and Harry Ingham. 1955. “The Johari Window, A Graphic Model of Interpersonal Awareness.” Proceedings of the Western Training Laboratory in Group Development. Los Angeles, CA: University of California, Los Angeles.
Manning, Harley. 2016. “Customer Experience Drives Revenue Growth.” Forrester website, June 21. Accessed September 20, 2022. https://www.forrester.com/blogs/16-06-21-customer_experience_drives_revenue_growth_2016/
Mansour, Sherif. n.d. “Product Manager: The Role and Best Practices for Beginners.” Atlassian website. Accessed September 20, 2022. https://www.atlassian.com/agile/product-management/product-manager
Newton, Casey. 2022. “Instagram is Walking Back Its Changes for Now – Adam Mosseri Explains Why.” The Verge website, July 28. Accessed September 20, 2022. https://www.theverge.com/2022/7/28/23282682/instagram-rollback-tiktok-feed-recommendations-interview-adam-mosseri
Nonaka, Ikujiro. 1994. “A Dynamic Theory of Organizational Knowledge Creation.” Organization Science 5, no. 1: 14-37. http://www.jstor.org/stable/2635068
Nonaka, Ikujiro, Ryoko Toyama, and Noboru Konno. 2000. “SECI, Ba and Leadership: A Unified Model of Dynamic Knowledge Creation.” Long Range Planning 33, no. 1: 5-34. https://doi.org/10.1016/S0024-6301(99)00115-6
Ogden, Cody. 2022. “Killed by Google.” Killed by Google website. Accessed September 20, 2022. https://killedbygoogle.com/
Olivera, Fernando. 2000. “Memory Systems in Organizations: An Empirical Investigation of Mechanisms for Knowledge Collection, Storage and Access.” Journal of Management Studies 37, no. 6: 811-832. https://doi.org/10.1111/1467-6486.00205
Olsen, Dan. 2015. The Lean Product Playbook. Hoboken, NJ: John Wiley & Sons.
Penders, Bart. 2018. “Ten Simple Rules for Responsible Referencing.” PLoS Computational Biology 14, no. 4. https://doi.org/10.1371/journal.pcbi.1006036
Pernice, Kara. 2022. “Democratize User Research in 5 Steps.” Nielsen Norman Group website, June 12. Accessed September 20, 2022. https://www.nngroup.com/articles/democratize-user-research/
Pollitt, Christopher. 2000. “Institutional Amnesia: A Paradox of the ‘Information Age’?” Prometheus 18, no. 1:5-16. https://doi.org/10.1080/08109020050000627
Poole, Marshall Scott. 1985. Tasks and Interaction Sequences: A Theory of Coherence in Group Decision Making Interaction. In R. L. Street & J. N. Capella (Eds.), Sequence and Pattern in Communicative Behavior (pp. 206-224). London: Edwards Arnold.
Poole, Marshall Scott. 2013. Structuration Research on Group Communication. Management Communication Quarterly 27, no. 4: 607-614. https://doi.org/10.1177/0893318913506265
Poole, Marshall Scott, and Joel A. Doelger. 1986. Developmental Processes in Group Decision Making. In Randy Y. Hirokawa & M. S. Poole (Eds.), Communication and Group Decision Making (pp. 35-62). Beverly Hills, CA: Sage.
Poole, Marshall Scott, and Joel A. Doelger. 1989. “Decision Functions Coding System.” University of Illinois IDEALS website. Accessed September 20, 2022. https://hdl.handle.net/2142/14542
Poole, Marshall Scott, and Jonelle Roth. 1989. Decision Development in Small Groups IV: A Typology of Group Decision Paths. Human Communication Research 15, no. 3: 323-356. https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1468-2958.1989.tb00188.x
Rowlinson, Michael, Charles Booth, Peter Clark, Agnes Delahaye, and Stephen Procter. 2010. “Social Remembering and Organizational Memory.” Organization Studies 31, no. 1: 69-87. https://doi.org/10.1177%2F0170840609347056
Rubin, Kenneth. 2013. Essential Scrum. A Practical Guide to the Most Popular Agile Process. Upper Saddle River, NJ: Addison-Wesley.
Department of Defense. 2002. “DoD News Briefing – Secretary Rumsfeld and Gen. Myers.” Archive.defense.gov website. Accessed September 20, 2022. https://archive.ph/20180320091111/http://archive.defense.gov/Transcripts/Transcript.aspx
Rusaw, A. Carol. 2005. “How Downsizing Affects Organizational Memory in Government: Some Implications for Professional and Organizational Development.” Public Administration Quarterly 28, no. 3/4: 482-500. https://www.jstor.org/stable/41288231
Saldana, Johnny. 2021. The coding manual for qualitative researchers, 4th ed. Thousand Oaks, CA: Sage.
Schmidt, Roy, Kalle Lyytinen, Mark Keil, and Paul Cule. 2015. “Identifying Software Project Risks: An International Delphi Study.” Journal of Management Information Systems 17, no. 4: 5-36. http://www.jstor.org/stable/40398503
Shankar, Venkatesh and Gregory Carpenter. 2013. “The Second-Mover Advantage.” Kellogg Insight website, November 4. Accessed September 20, 2022. https://insight.kellogg.northwestern.edu/article/the_second_mover_advantage
Srivastava, Shivam, Kartik Trehan, Dilip Wagle, and Jane Wang. 2020. “Developer velocity: How excellence fuels business performance.” McKinsey & Company website, April 20. Accessed September 20, 2022. https://www.mckinsey.com/industries/technology-media-and-telecommunications/our-insights/developer-velocity-how-software-excellence-fuels-business-performance
Takeuchi, Hirotaka, and Ikujiro Nonaka. 1986. “The New New Product Development Game.” Harvard Business Review website, January. Accessed September 20, 2022. https://hbr.org/1986/01/the-new-new-product-development-game
Taneja, Hemant. 2019. “The Era of ‘Move Fast and Break Things’ is Over.” Harvard Business Review website, January 22. Accessed September 20, 2022. https://hbr.org/2019/01/the-era-of-move-fast-and-break-things-is-over
Tuomi, Ilkka. 1995. “Abstraction and History: From Institutional Amnesia to Organizational Memory.” Proceedings of the Hawaii International Conference on Systems Sciences 28//V4:303-312. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.96.4594&rep=rep1&type=pdf
Tuomi, Ilkka. “Data Is More Than Knowledge: Implications of the Reversed Knowledge Hierarchy for Knowledge Management and Organizational Memory.” Journal of Management Information Systems 16, no. 3: 103-117. https://doi.org/10.1080/07421222.1999.11518258
The Wall Street Journal. 2022. “Watch: Instagram Head Adam Mosseri Responds to Users Decrying Changes.” The Wall Street Journal website, July 26. Accessed September 20, 2022. https://www.wsj.com/video/watch-instagram-head-adam-mosseri-responds-to-users-decrying-changes/1C571337-D33D-4D46-82CF-2FA7EDF0CC51.html
Walsh, James P., and Gerardo Rivera Ungson. 1991. “Organizational Memory.” Academy of Management Review 16, no. 1:57-91. https://doi.org/10.2307/258607
Walsh, Jim. 2017. “These 7 News Values Can Help You Make Smarter Content Choices.” Medium website, September 15. Accessed September 20, 2022. https://medium.com/atlantic-57/these-7-news-values-can-help-you-make-smarter-content-choices-6b3cbc0bddf7
Wilson, Myoung C. 1999. “Post Amazon.com Reference Culture: The Work of Reference Librarians in the Age of Smart Machines: A Preliminary Working Paper.” https://doi.org/10.7282/T3XP7399
Wylonis, Christian. 2021. “The Startup Org Chart from Founding to IPO.” The Org website, January 26. Accessed September 20, 2022. https://theorg.com/iterate/the-startup-org-chart-from-founding-to-ipo
Zuckerburg, Mark. 2012. “Mark Zuckerberg’s Letter to Investors: ‘The Hacker Way’.” WIRED website, February 1. Accessed September 20, 2022. https://www.wired.com/2012/02/zuck-letter/
