The use of meta-analytic studies has grown steadily in recent decades as a means of establishing greater confidence and robustness of social science findings, but such approaches remain rare in the business world. This paper offers two inter-linked qualitative meta-analytic approaches for business: one that both draws on pre-existing data to gain insight into new strategic questions and reaches across multiple studies to achieve greater generalizability and robustness, and a second that studies researchers and research practice as a means of reflecting on and improving methodology in particular organizations or research groups. Drawing on an in-house study the authors conducted for a Fortune 500 corporation, this paper articulates these two approaches and points to potential dangers and opportunities in applying them in other settings. In a moment in which researchers are increasingly called upon to do more with less, our approach provides flexibility and adaptability to environments inhospitable to marshalling resources to new original research.
INTRODUCTION: MASTERY OF THE PAST TO INFORM THE PRESENT
Ethnography and ethnographic mastery are generally assumed to require original research: new fieldwork and an influx of new data. This paper suggests that mastery might profitably include reflection on and analysis of past work, not in the form of literature reviews but as a basis for generating new and actionable insights. We offer a reflection on the potential, and potential pitfalls, of meta-analytical approaches to past projects, taking as a point of departure our effort to consolidate data from more than ten previous research projects conducted over a six-year period by the research division of a Fortune 500 company. The meta-analytic method we adopted was triple-pronged: 1) analysis across multiple projects aimed at generating insights with a greater degree of empirical support; 2) re-analysis of data based on new questions outside of the original projects’ scope; 3) what we call an ethnography of ethnographers, which can facilitate higher order reflection on ethnographic practice in the company and in the field more generally.
The reuse and reevaluation of past work is a key element of every research field. In the social sciences, literature reviews are a typical form of demonstrating knowledge of prior art, the researcher’s own skills, and the novelty of the direction presented by the author (cf. Hart 1998). Ethnographers in industry have long moved beyond the use of published literature. This stems in part from a relative lack of pertinent published studies compared to academic peers, as well as the need to produce research tailored to the demands of providing actionable results for clients and senior management. In past years, essays published in EPIC proceedings have taken advantage of data from earlier studies produced in the authors’ companies to develop a richer understanding of phenomena under observation (e.g., Asokan [2008] on privacy in shared spaces in India; Cefkin [2007] on expressions and performance in everyday work routines). Taking up the use of past work a bit further, Churchill and Elliott (2009) have underlined the importance of managing, storing, and “curating” data over time so that it can be useful in the future.
In contrast to both literature reviews and typical mining of past projects, the approach we discuss here is meant to reach across multiple projects and data sets simultaneously, against the grain of the intent of the original researchers. What we suggest is not simply a review of the final output of past studies (the final report, the published study), nor is it simply a re-purposing of data. It is a new look at what data might offer when new guiding questions are driving the research team, particularly more general and open-ended ones. Clearly, attention to data curating and archiving can facilitate such an approach, but it is not a prerequisite.
Meta-analyses are commonly associated with quantitative research. Glass introduced the term meta-analysis in 1976 to indicate an approach for developing statistically sound conclusions achieved by combining and analyzing the results of large numbers of individual studies that treat similar research questions. Since that time, meta-studies have gained steadily in popularity for two main reasons. First, they are a response to the expansion of literatures in many fields to such a degree that it is practically impossible for a single researcher or team to comprehend or evaluate their content and significance (Darity: 2008). Second, in a climate in which statistical reliability has become increasingly paramount, meta-analyses are assumed to be more generalizable and statistically sound than single studies. As a result, meta-studies are now employed for a wide range of applications. In health care research, for instance, the approach has emerged as a means of testing with greater confidence the impact of different interventions. In the policy world, meta-studies are used to make a strong case for favoring one policy over another.
But meta-studies are rarely used in business. They are generally neither appreciated as a standard method or tool, nor considered as contributors to more reflective methodology. As Sam Ladner put it recently on the anthrodesign discussion forum: anthrodesigners are strong on method and weak on methodology(http://tech.groups.yahoo.com/group/anthrodesign, April 6, 2010). That is, existing tools are strong for the purposes they are put to, but there is relatively little reflection on their underlying rationale, philosophical assumptions, or link to questions of validity or proof. Whatever one’s views on anthrodesign research practice, in describing and reflecting on meta-analytical approaches and their possible applications, we aim to contribute to what we hope will be a growing conversation on methodology.
The approach we offer is qualitative, not quantitative. The difference between these two approaches mostly depends on the particular understanding at play of the difference between qualitative and quantitative research. In their seminal article on meta-ethnography, Noblitt and Hare rather dismissively wrote that “[a]ny similarity [between qualitative and quantitative approaches] lies only in a shared interest in synthesizing empirical studies” (1988:10). But the contrast here may be exaggerated. Both types of meta-analysis entail: 1) higher order analysis than found in individual studies; 2) potentially more robust or statistically sound conclusions than smaller studies; 3) possible mining of data sets for different questions (though, once again, quantitative approaches usually analyze studies on the same question). The key difference between the two approaches turns on the nature of the material under analysis and the role the analyst plays in treating that material. A qualitative meta-analysis, like any qualitative analysis, is an interpretive exercise: in this case, an interpretation of interpretations. It may take up ethnographic material, published articles, or some other manner of comparative analysis, synthesis, or “translation” of individual studies. As with other interpretive approaches, the role of the analyst is crucial, but in interpretive meta-studies it may be even more crucial—and at the same time, under-appreciated. Indeed, as we discuss below, there is a danger that the aura of greater robustness associated with meta-studies might lead audiences to discount possible analyst bias.
To give a sense of what qualitative meta-analysis might look like in a business setting, the next section (II) describes the initial context in which we developed a meta-analytical approach and the specific project we completed, including the basic steps required (III). The project stimulated a reflection on methodology and what we see as a new tool, a twist on qualitative meta-analysis that we call ethnography of ethnographers (i.e. meta-study of researchers and research practice), which can be deployed for auditing and improving research practices in particular organizations, projects, or teams (section IV). These two approaches carry some cautions as well as significant possibilities for application, which we discuss in sections V and VI.
WHAT IT MEANS TO BE A SMALL BUSINESS (OR WHY WE NEEDED A META-ANALYSIS)
As a full-time employee and a consultant for the research division of Pitney Bowes (PBI), we were tasked with bringing together the division’s collective knowledge of small businesses in order to help inform broader corporate strategy. While the company’s primary marketing focus had been on larger businesses and enterprise customers, Pitney Bowes produces postage meters designed for small businesses and, at the time of writing, had over 800,000 small business customers using those products. The business units saw potential in this large base of small business customers, were motivated to provide them with solutions beyond postages meters, and, at the time of our engagement, were actively developing a strategy for this market.
In the preceding six years the research group had conducted 10 research initiatives involving small businesses. These were based on interviews, site visits, and focus groups that spanned a range of verticals—retail, medical, financial services, insurance, travel, professional services, non-profit—and included a mix of Pitney Bowes customers and non-customers. The foci of each of these projects centered on aspects of Pitney Bowes’s core businesses—primarily mailstream-related products and services such as shipping and marketing—and extant documentation (final reports, executive presentations) reflected these concerns.
As an aggregate, we felt that the data gathered in these projects formed a body of in-house knowledge about small businesses, but it had not been brought together as a coherent whole, nor was it separated from the specific contexts of mailing, shipping, and marketing. We believed that doing so could help form a picture of what it means to operate as a small business, with mindsets, self-perceptions, challenges, and priorities distinct from larger businesses.
One challenge of our meta-analytic effort was thus to return to the original data and extract new insights with different research goals in mind. Analysis would need to be directed away from the areas in the company’s core business, and toward a broader understanding of overall work practices in small businesses. Our driving assumption was that a new perspective or strategic question on the data would lead to new insights. This meant we needed to: obtain the original data from relevant projects, not just the analyzed results; ground new analysis in areas relevant to our current focus; and find data not recorded by the researchers because it fell outside the original project scope. Our approach was to engage the three-pronged meta-analysis mentioned above. First, we collected all available data across the projects, both written-up in the form of final reports and executive presentations, and unpolished notes and working documents. Second, we developed a new analytic frame that was based around the life cycle of a small business. Third, we interviewed the researchers themselves in order to extract knowledge and insights from their observations of small business.
DEVELOPING THE APPROACH
Criteria for inclusion, guiding research questions, and defining the object of study
The first step was to determine criteria for inclusion in the meta-study. Which of the many dozens of projects previously completed by the research organization would be most useful for understanding small businesses? Had other researchers already conducted secondary research we could take advantage of? We wanted to identify projects that might have collected data relevant to the understanding of small businesses, regardless if their original aim was to understand other matters.
Discussions of criteria for inclusion in the published literature on meta-analyses express concerns about studies employing different methods, theoretical perspectives, subject matter, levels of quality, or even different standpoints on the nature of data and the position of the researcher (Atkins et al 2008, Doyle 2003). Where there is little theoretical development or interpretation and what is presented is largely descriptive, these concerns probably exert only a minimal effect (ibid). This was the case for the in-house studies that we decided to revisit: they were mostly descriptive, attempting to influence management decision-making by offering a few conclusions from the data, and leaving any theoretical orientation implicit. Since our mission was to inform corporate strategy, we assumed that studies conducted with the interests of Pitney Bowes specifically in mind were more likely to contain helpful material, and so a large proportion of the research we included was in-house studies. We also searched for broader studies on the subject in academic and industry literature. It had been more than five years since some of our in-house studies were completed, so we felt it was important to include more recent research in order to account for socio-economic and demographic shifts that had taken place since that time.
The most crucial consideration we faced in determining which studies to include was the question of consistent definition, in this case of “small businesses.” There was at the time no consistent definition of small business across the research organization of Pitney Bowes, nor across the secondary literature for that matter. Some studies defined by size, some by revenue. This meant we had to ask if observations about companies that had, say, 50 employees were relevant to our notion of “small businesses” in the same way as those about companies with less than 10 or even less than two employees. Rather than focus on size, we attempted instead to highlight some of the key qualitative characteristics. What distinguishes small businesses from other businesses? What do they care about? What motivates them? How do they see themselves as different from big businesses? For our purposes, a key point became how and why the work small businesses do might matter for understanding other dimensions of their business (staffing decisions, customer orientation, supply chain management, etc).1
Reanalysis
The next step was to collect existing documentation from each project, code relevant material by categories, and create new categories as they emerged in the course of analysis. Since the aim was to articulate the company’s store of insights about small business as a means of informing strategy, our approach was to separate the data about small businesses from each project’s particular focus. To organize the data in the course of analysis, we created an initial categorization of elements of a small businesses lifecycle, based in part on our previous knowledge of the data set. These were working categories that we assumed would shift and change from the bottom up as the data was analyzed. The starting categories included items such as Start up, Staffing, Growth, Supply Chain, Financing, Customers, and Partnerships. To redirect the analysis away from earlier project concerns, we deliberately excluded categories specifically focusing on mail, shipping, or marketing—the primary subject areas of the studies we were re-analyzing.
In order to develop this analysis, we needed to get as close to the data as possible. Existing documentation ranged from final reports and final presentations, description of concepts, interview notes, and audio recordings. Since final reports aim to deliver actionable messages to management, they tend to offer interpretations oriented toward the focus of the project and relatively little in the way of original data. We expected that much of the data we were seeking would not have been of primary interest to those collecting and interpreting the data for the original projects. A researcher looking at mailing and shipping, for example, may not delve deeply into the issues a business had at start-up. At the same time, we assumed that many other dimensions to small businesses were observed during site visits than the shipping and marketing concerns that were the driving research concerns of the original studies. This was, in effect, the “extra data,” that McCracken has argued that ethnographers should not lose, noting the wide net we cast in research, with only a narrow portion used in the results of a particular project (2006: 1).
To supplement the documentation we collected on the projects, we decided to interview the original researchers themselves. We asked them to consider what they knew about small businesses that was not explicitly a part of their past projects. To jog their memories, we gave them a list of topics to mull over in advance. And since many of the interviews took place over multiple sessions, interviewees had plenty of time to reflect and remember.
The new data gleaned from the interviews was coded and clustered with other material. We then cross-analyzed and synthesized the results, using insights gained about the research processes to influence our interpretations. This led us to revise, combine, and remove insight categories for a draft report. We then circulated the draft to interviewees, inviting their feedback, comments, and critiques—the main outcome of which was that our interim findings were largely vetted by the original researchers.
ETHNOGRAPHY OF THE ETHNOGRAPHERS: META-STUDIES OF RESEARCHERS AND RESEARCH PRACTICE
Besides uncovering rich insights and observations not captured in project documentation and vetting our own analysis, interviewing the original researchers also enabled us to undertake an “ethnography of the ethnographers.” This meta-ethnographic endeavor turned out to be a “reflexive ethnography” literally of a higher order: rather than a single ethnographer reflecting on the biases or limits of his or her own method or positionality, it offered insight into the biases and approach of multiple members of a research division over several years. As a new tool with broad potential application, it offers an opportunity to reflect on, adjust, and hopefully improve, research practice—how a particular organization’s researchers have tended to conduct research, come to insights, and make conclusions.
Researcher interviews allow for the possibility of experiencing what it is like to be an interviewee. In this and other ways, they are an opportunity to reflect on interview practice and to integrate insights into future practice. The expanding use of researcher interviews is also a response to a “perceived lack of clarity regarding the appropriate criteria for appraising qualitative studies” (Bryman and Cassel 2006: 43) and has emerged against the backdrop of perceived relative consensus in appraising quantitative studies. Thus, like other meta-analytic approaches, researcher interviews are a means of establishing confidence in the conduct and findings of qualitative research. 2 In focusing on the practice of interviews, however, Bryman and Cassel seem to miss more crucial aspects of a reevaluation of research practice in applied settings—aspects like synthesis, interpretation and presentation of results, not to mention reception of those results by various clients, audiences, and stakeholders. Our ethnography of ethnographers approach is thus more comprehensive than simply engaging in researcher interviews.
There are at least two basic ways of approaching this sort of meta-study. One is to ask interviewees to reflect on their own research practice. Another is to take an anthropology or sociology of science approach in which the analyst is embedded in and yet not officially a part of the practices which he studies (he is not a physicist, for example, but spends an extended period in a physics lab). The former is likely to be subject to bias in self-presentation. The latter is likely to be too costly in resources for most applied and business settings. We therefore used the occasion of interviews to reflect on how other researchers conduct research and come to insights and conclusions. We asked researchers what they knew about subjects that were not explicitly a part of their past projects—as opposed to asking them directly about their approach to research and analysis.
Interviewing researchers directly about the practice of research might contribute to a desire to tweak their representation of the process more than when researchers are asked to provide more background and insights about aspects of a project that were not reported on in the written output. Researcher interviews, like any social interaction, are an instance of the performance of identity (Goffman 1967)— in this case, the interviewees are expert researchers themselves and so are under pressure to display that identity in the course of the interaction. In our meta-study, this was especially evident in the case of interviewees known for their intelligence and expertise and their penchant for displaying it. And just as some interviewees may feel the need to perform their identity as experts, the same is true for interviewers, since their skills as interviewers are on display in the very moment of interviewing other researchers. Moreover, interviewers’ own biases about how research should be conducted could influence the second-order reporting.
All this suggests that care should be taken to put the results of ethnographic study of ethnographers in perspective. Especially if conducted by an in-house expert, that expert is likely to be under pressure to demonstrate expertise, insight into the very practices he has been tasked with observing—and improving. At the same time, it can be a powerful tool for delving deeper into prior research, not only for unearthing “hidden” data and shedding light on earlier interpretations, but also for understanding (and improving) research practice.
METHODOLOGICAL CONSIDERATIONS: CAUTIONS
In this section we reflect on our methods and how they may or may not hold in other circumstances. While our adaptation of qualitative meta-analysis was in a sense ad hoc—it developed out of a specific instance and need—we believe it offers significant advantages, as well as potential for wider application. But it also comes with some dangers. Perhaps the greatest risk of meta-analyses lies in the allure of possible greater robustness of insights and confidence in results. This is because a poorly executed meta-study or a meta-study based on poorly designed studies will yield misleading or poor results that are nevertheless assumed to be more robust.
Despite assumptions of their authority and robustness, meta-studies are subject to various forms of distortion, bias, and misleading or inaccurate conclusions. These dangers turn on similar considerations of researcher savvy, slant, or bias; the quality and validity of research design, data, and interpretation of studies employed for the meta-analysis; decisions about which studies to include; and possible loss of original context or nuance.
The prevailing wisdom on meta-analyses, particularly quantitative ones, is that a good meta-study of badly executed or designed studies yields bad results.3 How much this is a danger in qualitative meta-studies arguably depends on the researcher. In any qualitative study, analysts need to evaluate other researchers’ second- or third-order insight with a degree of skepticism; they should not simply accept or apply prior claims. Similarly, it is up to the meta-analyst to carefully evaluate the merit of the data and interpretations on which he bases the meta-study. In this sense, perhaps paradoxically, a qualitative meta-study may be at least as dependent on the skills (and slant and style) of the researcher conducting it as is an individual study. Interpretation is the core of the very exercise and yet both the analyst and the ultimate audience are likely to believe that the output of a meta-study is more authoritative than any individual study. Analysts may paradoxically appear as interchangeable, their role downplayed by the very assumption of generalizability or reproducibility of findings.
This raises a concern about researcher savvy, skill, or bias which cuts across the different techniques we employed (repurposing and reimagining pre-existing data, analysis across past projects, ethnography of ethnographers). The meta-analytic researcher plays a crucial role in evaluating the validity or generalizability of whatever material they are analyzing—whether it happens to be the “data”, interpretation, or research design. Even what “data” is included in a study, what notes are recorded and presented, and certainly what appears in reports—are all a matter of interpretation. Since a meta-analysis aims at generalizing across studies, extra care must be taken to consider the validity of the studies themselves, especially in terms of their application beyond their original contexts. Sufficiently savvy qualitative analysts should be able to pull out data and interpretations that are relevant to the specific question driving their meta-analysis, and naturally exclude the rest; this need not mean an exclusion of entire studies that are aimed at different questions to begin with.
Nevertheless, there are considerations that may challenge even the most skilled analysts. Original interpretations are made by researchers other than the meta-analyst and in a context likely to be unknown to them. The difference between an interpretation and an observation may seem unclear. Moreover, because the analyst is pulling strands out of many studies at the same time, he is likely to be more attuned to the goal of a greater synthesis, of producing conclusions that appear to be of interest to the current concerns of the company or the sponsor, than he is to evaluating the original interpretations that are the fodder for this exercise. These concerns also point to practical difficulties in establishing a baseline for the interpretive re-analysis. Researchers undertaking a meta-analysis will have varying degrees of familiarity with the original projects and data and this will influence how they initially frame guiding questions and categories for their project. These and other differences in researcher orientation will need to be taken into account in designing and evaluating the credibility of a given meta-study.
Generalizability
In re-analyzing the material from past projects, we needed to carefully judge how much the observations we were reviewing—concerning customers or partnerships, staffing decisions, and the like—might apply in different contexts (geographical locations, different time periods, and so on). This is a consideration that social researchers always face: to what extent are specific data unique to the instance observed or generalizable to other instances, and in which instances. We took pains to disregard any data, insights, or interpretations that appeared to be time or location dependent, and weighted more heavily what was supported across multiple projects, locations, and times.
The aim for greater generalizability, like the act of comparison across qualitative cases, by its nature privileges higher order interpretations or conclusions and downplays the nuance of particular cases. This is perhaps Noblitt and Hare’s main worry in their classic piece on meta-ethnography. But it is a false dichotomy: more comparability does not necessarily mean less nuance (especially in instances in which the original studies are not deeply contextual to begin with); it means retaining nuance that supports the claims that hold across cases. The claims that do not hold across cases are simply not of interest in a meta-study. There is no reason to assume that this skews results.
Old data
The question arises whether using old data might bias results. This is linked to another question: does a project’s success depend on gathering new data? In industry, the latter question is often answered in the affirmative because sponsors are frequently convinced that only new data (fresh from “the field”) can be of value. To the former question, we assert that old data bias results when the old data are no longer valid. In this sense, old data are no different from bad data. There is nothing inherently advantageous about new or old data; what matters is whether they continue to hold. But the return to apparently “old” data has another possible advantage—that characterized by the “historical” turn in the social sciences—the possibility of demonstrating that the observation or data point is not just a single point in time but more than one, or a continuous one. For applied purposes, especially business, what is of interest is the indication that what has been observed is enduring and thus, perhaps, more closely held, less likely to change, more dear to the community in question. Whatever the particular case, a thoughtful reflection on the quality of the data, as well as the quality of the study, is a necessary step in meta-analysis.
Increasing confidence in the validity of meta-studies
Besides the above tactics, a number of measures can be taken to increase confidence in the results of a meta-study, among them, as noted above, interviewing the researchers themselves as a way of both pulling out data that were not recorded or retained in the individual studies and circulating the conclusions among researchers who were involved in the past projects studied. Results may also be vetted with relevant stakeholders and research subjects. In our case, for example, we solicited (and obtained) feedback from senior managers in other parts of the company with knowledge of small businesses. Finally, due to issues raised by researcher-researcher interactions and the crucial role the meta-analyst plays in the process, the recording and review of researcher interviews may be crucial. An interviewer is unlikely to be fully aware of the influence they are exerting in the moment of the interview; this may be better gleaned by listening to the recordings after the fact, and perhaps by a third party.
BROADER APPLICABILITY: OPPORTUNITIES
All this points to a larger question about when meta-studies can be used most profitably. One of the key opportunities we see is to conduct meta-studies as a new tool for auditing research practices in a wide range of settings (e.g., organizations, research divisions, consulting practices). Like any ethnographic audit, a meta-study could provide much needed opportunity to reflect on, improve, and refine established research practices in a given operation—including how questions are framed, data collected, insights arrived upon, and conclusions determined. Rather than a single ethnographer reflecting on the biases or limits of his or her own method or positionality, it offers insight into the strengths and weaknesses of research practice in specific settings over many years and among many researchers.
What we have proposed throughout is a new look at what reviewing existing data might offer when new guiding questions are driving a research team, particularly more general and open-ended questions. Advantages include the ability to:
- Create a larger, more robust data set from many smaller studies. By drawing on different researchers, projects, and field research settings, they may offer more generalizable insights than one-off ethnographic approaches typical of projects geared towards a specific question
- Take advantage of available in-house data to address new strategic questions or provide conclusions with more robust empirical backing
- Review beyond what is published, reaching back to data, enabling more insights than can be gleaned from analyzing results and analysis only
- “Do more with less” (i.e., save considerable human resources by not conducting new empirical research)
Clearly meta-studies cannot be a stand-in for new research. What they offer is distinct. What is to be avoided in any case is a false confidence in the output of a meta-study. Any study deserves follow up for further confirmation. Researchers need always to be on the lookout for errors or misleading suggestions in their past work as they conduct new work. The aura of robustness and confidence of meta-studies should not lead researchers or their audiences to forget this.
Josh Kaplan is an anthropologist and research fellow at Yale University, Institution for Social and Policy Studies. He consults for organizations ranging from grassroots NGOs to Fortune 500 companies and is looking out for better ways to make use of anthropological expertise. At Yale, he is finishing a book on the social and political dimensions to international lawmaking.
Alexandra Mack is a Workplace Anthropologist in Pitney Bowes’s Strategic Technology and Innovation Center. Her work is focused on developing ideas for new products, services, and technologies based on a deep understanding of work practice. She is currently focused new innovation practices.
NOTES
1 Before the results of our study were circulated, small businesses at Pitney Bowes were typically thought of as businesses below ‘x’ in sales or ‘y’ in amount mailed—as just another segment for selling meters and mail support. Sharing the results of the meta-study was a way to begin to shift the internal orientation to the concerns and priorities of small businesses. In very brief form, two insights from these results should begin to illustrate both the natural of this internal shift and, more generally, the output of a meta-approach based on new questions. Most fundamentally, small business are not smaller versions of big businesses. In fact, they define themselves against big businesses (which they often call “big boxes.”) This means that small businesses do not simply want to grow (which had been the prevailing internal assumption)—they want to get better, not just big bigger. Second, running a small business is “personal.” Notice how many small businesses carry the name of their owner; their name is the owner’s name. Owners have a personal stake: it’s their quality, their reputation, their name; the reputation on the line is their reputation.
2 There is a growing use of what has been called the “researcher interview” among social scientists (Bryman and Cassel 2006). This practice has precedent in the social studies of science, the anthropology and sociology of science (Latour and Law 1988), and reflexive sociology more generally (Bourdieu 1993, Bourdieu and Wacquant 1992). An aim common to all of these is to better understand the role of researchers, scientists, and analysts in the practice and production of science or social science. Indeed, reflexivity in social science has itself been described as “a sensitivity to the significance of the researcher for the research process” (Bryman and Cassel: 45). We might think of this as one of a series of reverberating effects from the insights that originally came from Heisenberg’s uncertainty principle in physics (if we know that the researcher looking for waves finds waves than we know that the researcher plays a fundamental role in shaping the output of science). These insights found their way into the reflexive turn in anthropology and science studies in the 1980s, which questioned the role of anthropologists and ethnographic authority in anthropology, and the role of the scientists in science more generally.
3 Of course, in an industry setting a meta-analyst is likely to have relatively few studies to consider and so may not have the luxury of excluding ones that were hastily conceived or executed. In such cases, this may be a point in favor of attempting to include relevant published studies. In any case, concerns about selection criteria are likely to be an important consideration in applications outside business.
REFERENCES
Asokan, A.
2008 The Space Between Mine and Ours: Exploring the Subtle Spaces Between the Private and the Shared in India. Ethnographic Praxis In Industry Conference Proceedings, (November 2008): 175-191
Atkins, Lewins, et al
2008. Conducting a meta-ethnography of qualitative literature: Lessons learnt. BMC Medical Research Methodology. 8:21
Bourdieu, P.
1993 Sociology in Question. Thousand Oaks, CA: Sage Publications.
Bourdieu. P and L. Wacquant.
1992 An Invitation to Reflexive Sociology. Chicago: University of Chicago Press.
Britten N, Campbell R, Pope C, Donovan J, Morgan M, Pill R
2002 Using meta ethnography to synthesise qualitative research: a worked example. J Health Serv Res Policy 7:209-215.
Bryman, A. and Cassell, C.
2006 The Researcher Interview: A Reflexive Perspective. Qualitative Research in Organizations and Management: An International Journal 1 (1): 41-55.
Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M, Donovan J:
2003 Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. Social Science Medicine, 56:671-684
Cefkin. M.
2007 Numbers may speak louder than words, but is anyone listening? The rhythmscape and sales pipeline management. Ethnographic Praxis In Industry Conference Proceedings, (October 2007): 187-199.
Churchill E. F., and Elliott, A.
2009 Flexibility and the Curatorial Eye: Why and How Well-Documented Fieldwork Sustains Value Over Time. Ethnographic Praxis In Industry Conference Proceedings, (September 2009): 72-80.
Donmoyer, R.
1990 Generalizability and the Single-case Study, in E.W. Eisner and A. Peshkin (eds) Qualitative Inquiry in Education: The Continuing Debate, pp. 175–200. New York: Teachers College Press.
Doyle, L.
2003 Synthesis through Meta-Ethnography: Paradoxes, Enhancements, and Possibilities. Qualitative Research. 3(3):321-344.
Eisenhart, M.
1998 On the Subject of Interpretive Reviews, Review of Educational Research68(4): 391–400.
Frake, W. B., and Kang, K.
2005 Software Reuse Research: Status and Future. IEEE Transactions on Software Engineering 31 (7): 529-536
Darity, W. Jr.
2008 Meta-Analysis. Gale Encyclopedia of the Social Sciences, Vol. 5, 2nd Ed. Detroit: Macmillan Reference.
Glass, G.V.
1976 Primary, Secondary, and Meta-analysis Research, Educational Researcher5(10): 3–8.
Goffman, E.
1967 Interaction Ritual. New York: Anchor.
Hart, C.
1998 Doing a Literature Review: Releasing the Social Science Research Imagination. Sage Publications, London.
Latour, B.
1988 The Pasteurization of France. Cambridge: Harvard University Press.
Light, RJ and DB Pillemer.
1984 Summing Up: the Science of Reviewing Research, Harvard University Press, Cambridge, MA.
McCracken, G.
2006 Keynote Address: Ethnography and the “Extra Data” Opportunity. Ethnographic Praxis In Industry Conference Proceedings, (September): 1-3.
Meacham, S.J.
1998 Threads of a New Language: A Response to Eisenhart’s “On the Subject of Interpretive Review”, Review of Educational Research68(4): 401–8.
Noblit, G. and D. Hare.
1988 Meta-Ethnography: Synthesizing Qualitative Studies. Qualitative Research Methods Series 11. A Sage University Paper. Newbury Park, CA and London, UK: Sage Publications.
Rossman, G.B.
1993 Building Explanations across Case Studies: A Framework for Synthesis, paper prepared for the Curriculum Reform Project. Colorado University: Boulder. (ERIC Document Reproduction Service No. ED 373 115.)
Strike, K. and Posner, G.
1983 Types of Synthesis and their Criteria, in S. Ward and L. Reed (eds) Knowledge Structure and Use: Implications for Synthesis and Interpretation, pp. 343–62. Philadelphia: Temple University Press.
Tweedie, R.L.
2004 Meta-Analysis: Overview. Elsiever International Encyclopedia of the Social Sciences, pp. 9717-9724
Ward, S. and Reed, L.
1983 Introduction, in S. Ward and L. Reed (eds) Knowledge Structure and Use: Implications for Synthesis and Interpretation, pp. 1–18. Philadelphia: Temple University Press.
Wiles, R. Charles, V., Crow, G., and Heath, S.
2004 Researching Researchers: Lessons for Research Ethics. Paper presented to the BSA Medical Sociology Conference, York.