Ethnographic Inspection Identifying Project Risks

Episode2: Lack of evidence for shared understanding

An inspector pointed out a communication issue among project members. The project manager under inspection could not understand the problem because the inspector could not provide the details of the problem.

Inspector: “We did not get the frequency, but there are some communication errors from team A to team B that fully depend on oral communication and e-mail.”

Project manager: “I could not figure out what was happening. Communication errors are important problems; however, it is of no use if you can not give us more detailed information.”

Episode3: Difficulty of relaxing wariness

As their general policy, the inspection teams tried not to be auditors but helpers. However, a project manager and a leader received the suggestions from the inspection teams as criticisms and rejected the team’s suggestions.

Inspector: “Some of members of your project have been in your project for many years. There are possibilities that certain skills belong to individuals and are not shared with others.”

Project manager: “I could not figure out what kinds of skills are not shared. Specific individual skills are needed to some extent. However, we rotate members inside our project, assuming there is a risk that some members will leave the team. It is basic of project management. There would be no problem even if someone passed away.”

Inspector: “We raised this issue as a suggestion from a third party’s point of view. I hope you will regard our suggestions as positive feedback.”

Episode4: Questions based on past experience of failures

Typical questions of the inspectors we observed are questions based on past experience of failures. Most of them are questions that confirm how they take actions for some risk factors.

“What actions do you plan for transferring skills of experienced members who are going to retire?”

“What is being done for variation of work load?”

“How frequently do you report to your client about the work progress?”

Episode5: Difficulty of uncovering real problems

The inspectors were concerned that they failed to put project members at ease and, therefore, could not uncover the real problems the members faced. The result of the questionnaire from 101 inspectors showed that 25% of inspectors described the difficulties of uncovering real problems.

“I always face difficulty in making project members feel at ease enough to speak about real problems.”

“Project members would not tell us about their real problems. I don’t think they like to be told of problems by inspectors since that will produce re-work.”

“Before inspections, some projects create documents that are rarely used in their projects.”

On the other hand, some of the questionnaire data showed that there are episodes that inspectors sometimes ignored problems, considering the burdens of project members.

“Considering the burdens of project members, I sometimes do not dare to point out problems that may be hard to tackle.”

“The leader of my inspection team told me, you should not describe problems that are hard to solve.”

Discussion

The inspection teams tried not to be auditors but instead to be helpers as their policy. However, guideline-based inspection does not always fit the project context, and sometimes they failed to obtain shared understanding of problems because they lacked evidence, or caused wariness among project members. Questions raised by inspectors tend to be closed questions confirming how they take actions for some risk factors that are invoked by past experience. We also found evidence in the questionnaire responses that guideline-based inspection produces the side effect of generating unanticipated “work-arounds” as Jordan and Putz pointed out (Jordan 2004). Moreover, considering the burdens of project members, inspectors sometimes did not dare to point out problems that were hard to tackle.

Guideline-based inspection is a good tool for checking whether projects follow basic standard rules or not. However, the evidence showed that guideline-based inspection is not always effective in uncovering the real problems that projects face.

Guideline

We developed guidelines for ethnographic inspection for the IT experts by articulating the essence of ethnography. The following is an overview of our guidelines.

Describe problems from the workers’ perspective – To describe problems from the workers’ perspective, we developed the following guidelines:

1. Do not attribute causes of problems to workers’ mistakes.
   You should look for causes of problems in the workers’ environment, such as tools, organizational structures, policy, or pressure from outsiders. Try not to judge a problem in such a way that makes the workers feel their performance is insufficient to accomplish their tasks. Instead, try to describe the workers’ difficulties in accomplishing their goal or fitting their work to the standard processes.
2. Do not use evaluative questions. Get episodes of everyday work practice.
   You should encourage informants to tell you about their everyday work practices. Ask probing questions to elicit more detail about the episodes. When your informants talk about their problems, you should not evaluate them using standard models.

Describe episodes in detail – To encourage description of problems that would be like fieldnotes, we developed the following guidelines:

3. Describe problem episodes in detail to create a common ground.
   Do not describe problems in abstract terms. Describe concrete episodes that occurred in the field. It allows many stakeholders to discuss problems from a common ground.
4. Do not introduce outsiders’ standards uncritically.
   Problems pointed out by inspectors tend to introduce outsiders’ points of view uncritically. These points sometimes cause conflicts between inspection teams and project members, whose perspectives may be different. Describe a problem by dividing it into three parts: 1) the name of the problem, 2) the episodes that illustrate the problem, and 3) discussions about the problem. Refer to descriptions of the episode in which you do not to introduce outsiders’ point of view.

Analyze patterns and structures – It is important for the ethnographic method to focus on what people take for granted and rarely discuss (Jordan 2006). We developed guidelines for helping inspectors discover important patterns and structures from everyday work practice:

5. Using heuristics for detecting patterns.
   Jordan and Dalal argued that ethnographic study requires years of theoretically grounded training and practical experience. The special skill of trained ethnographers is the ability to look for patterns (Jordan 2006). To help IT experts who have no experience in fieldwork, we provided heuristics for detecting patterns. When inspectors analyze filed memos, they consult the heuristics to find interesting patterns. We provide heuristics written in short messages that are easy to remember. We developed these heuristics based on papers by Hughes (1997) and Martin (2004). For example, these papers described heuristics such as “working division of labor” and “distributed coordination” to find interesting patterns.
6. Analyzing problems as systems.
   We provided a process for analysis by integrating several episodes to determine the tacit structural problems that were deeply rooted in organizational structure and culture. After describing several episodes of problems, inspectors created causal loop diagrams (Senge 1990) discussing the casual relationships among episodes.

Procedure

Ethnographic inspection is conducted in the following six steps. It takes approximately three weeks from step 1 to step 3 for a few days’ observation and a few members’ interviews. The time needed for steps 4, 5, and 6 depend on the difficulty of the problems.

Step 1: Setting up the fieldwork theme – Interview the manager of the target project, and then discuss the theme and target tasks and workers. Examples of themes are; “Why does the same kind of mistake frequently occur?”, “Why do they have difficulty following the standard process?”

Step 2: Interview and observation – Get agreement on conducting ethnographic inspection with informants. Interview and observe informants whose work is related to the theme, and collect episodes that illustrate the problems.

Step 3: Co-analysis with informants – Show the collected episodes to the inspection team and discuss them with both inspectors and informants. Confirm the validity of the episodes and try to get a deeper understanding of the problems.

Step 4: Analyzing patterns and structures – Based on the collected episodes from multiple observations and informants, analyze the patterns and structures of the problems.

Step 5: Co-design with stakeholders – Feedback the problems based on episodes from interviews and observations, and create shared understanding of problems among stakeholders. Then discuss a vision for resolution of the problem and actions to solve the problem. Participants in these discussions are not limited to project members, but also include executive managers, clients, and experts from other projects.

Step 6: Following up actions – Observe how actions are executed, and what the effects are of the actions. Continue these steps repeatedly if new important issues emerge in the process.

Case 1: Organizational structures and culture behind human errors

One of the projects that we inspected had approximately 50 project members. Several executive managers perceived that this project produced more system troubles than other projects, and wanted to know the underlying problems behind the recurring troubles. On the other hand, the project manager perceived that human errors were unavoidable to some extent. He was unsure how he could reduce human errors.

Inspectors using guideline-based inspection pointed out problems in several documents. Descriptions of work distribution with their client in the contract document were too ambiguous, and lacked description of the escalation process and rules. They also suggested that the project manager should get agreement with the client and make the project members aware of the new rules. In contrast , the inspectors judged that the project provided documents properly for quality management that were in alignment with the standard model. We interviewed one of the inspectors about the recurring troubles. He suspected that the recent replacement of the project manager might be a cause of the problem. He suspected that the relationships with customers and sub-contractors might not be going well.

We conducted fieldwork to determine the underlying causes of the recurring troubles. We observed several meetings such as progress reporting and the issues management meeting, and took notes during the meetings. We interviewed several members of projects sharing with them descriptions of episodes that had occurred in the meetings.

We obtained several episodes from meetings that indicated a leader of the project had just tried to publicize the standard model rules, but that he did not make sure that his members rigidly follow the rules. Here is a excerpt of a conversation that occurred in a progress meeting:

Project manager: “What did you do for this issue?”

Leader: “I just told them again and again that they should follow the standard rules.”

We interviewed several leaders by sharing episodes to understand the stories behind them. Then we found a leader who perceived that the project team members were overloaded:

Leader: “We have many things to do. We don’t have enough time to pay attention to each incident. I asked for too much work from my members.”

Moreover, we got episodes illustrating that work was increasing unexpectedly due to the retirement of experienced members from the information system division of their customers:

“There have been a number of experienced customers in the information system division who got involved in building the current system from the scratch. Those experienced customers are gradually retiring. Recently, two experienced customers retired and were replaced by young people. “

“When I asked a customer about their work process, they directed me to investigate the work ourselves. I think they did not know their work flow.”

“Customers of different divisions tend to impose additional work on each other. They don’t like to get additional work. They only know their related tasks. Currently no one knows the whole system and can not coordinate such conflicts. There is no way, except that we are taking on this role. ”.

Team members are willing to do such additional hidden work because of their cooperate culture:

“Even if we defined detailed work distribution with customers in the contract, maybe we can’t say no to our customers. This may be our corporate culture.”

From analysis of these observation and interviews, underlying problems of recurring human errors emerged. There are organizational and cultural issues such as retirement of experienced customers in the information system division that is producing unexpected hidden work, and there is willingness to do overloaded work for their customers. The executive manager thought similar problems might also be emerging in other projects, and began discussion with members of his department to overcome these problems.

Case 2: Problems behind multi-layered business

The second project that we inspected had approximately 300 project members in the development phase, and the project downsized to 18 members in the maintenance phase after years of development. This project is part of a multiple systems integration project. Three system integrators were participating in the project. One of these system integrators was a first-tier company, and others, including the project we inspected, were second-tier companies.

The problems pointed out by inspectors using guideline-based inspection were similar to Case 1. They pointed out that descriptions of the distribution of work with customers in the contract document were too ambiguous, and there was a lack of documents that described the escalation process and rules. In addition, they suggested adding a cross-review process to the library management process. The project manager appreciated the suggestions of the inspectors.

We interviewed the project manager to discuss the purpose of the fieldwork. He was not concerned with system problems. His primary concern was how they were to handover the knowledge of the system during the transition from the development phase to the maintenance phase. We focused on meetings that were related to handover of system knowledge.

We uncovered episodes that the inspectors who followed guideline-based inspection had not pointed out. The project team had difficulty in handing over system operations work to the first-tier company. The responsibility of the first-tier company was not clearly defined, and they had to do additional work for their real customers. Here is a sample of comments that illustrate the episodes:

“Mr. Suzuki who participated in the meeting (a meeting for handover) is a newcomer. He has not been here before. We did handover the process with Mr. Sato who accompanied the three operators. But they were missing. “

We reported these episodes to the executive manager as well as to the project manager. They recognized the problem well. However, the discussion based on these episodes created reflection. They said they should clearly define work distribution and responsibility for the maintenance phase in the system proposal phase. They appreciated the results of our inspection:

“It is important to describe such tacit problems that occur in everyday work to discuss standard process. It might be effective to create new guidelines for inspections.”

Ethnographic inspection as a tool for organizational learning

We have described the situation in which guideline-based inspections do not take into account the project context, and sometimes these inspections failed to obtain shared understanding of problems, and caused wariness among project members. Questions raised by inspectors tended to be closed-ended and simply reconfirmed, actions invoked in response to some risk factors encountered in past experience. Moreover, the primary focus of the inspectors in our study was correcting work practices to create a better fit to standard models. Our result suggested that guideline-based inspection is not appropriate for pointing out emerging and hidden issues that trigger reviewing company policy and standard models.

On the other hand, ethnographic inspection successfully uncovered organizational issues that were not pointed out by guideline-based inspection, such as retirement of experienced customers in the information system division that produced unexpected hidden work, and the ambiguous scope of responsibility in a multi-layered business. The executive manager perceived these problems not as a project matter, but as general issues that should be addressed by the organization. Our results suggest that ethnographic inspections could be a tool for reviewing organizational policy and standard models.

We obtained some evidence that ethnographic inspection could capture not only risk factors, but also risk-mitigating best practices devised in the field. Externalizing and sharing tacit knowledge is an important practice for organizational learning (Nonaka 1995). Ethnographic inspection might also useful to create a shared vision, another important factor for organizational learning (Senge 1990). Executive managers and project managers might have personal visions that never get translated into shared visions and, there, cannot be of value in guiding and encouraging project members. Policy and standard models that are reviewed based on episodes taken from many projects can reflect these many contexts and might strongly support project members for overcoming difficult issues. We will continue to explore ethnographic inspection as an organizational learning tool.

Cooperation and diffusion

It is important to develop tools for IT experts that they can easily and effectively use to conduct ethnographic inspection and penetrate work practices organizational wide. Four IT experts who work as inspectors participated to our ethnographic inspection following the guidelines and the procedures that we illustrated in this paper. At the start of the project, participants were accustomed to established practices for evaluating project work, and tended to violate the guidelines we provided. They evaluated the work practices based on their own observations and perspectives rather than describing problems from the workers’ points of view. We reminded them repeatedly about the guidelines and procedures for an ethnographic approach, and they gradually understood the value of approach. By the conclusion of the project, the guidelines for ethnographic interviewing seemed very useful to them. Some inspectors said they enjoyed doing ethnographic interviews because they could get various interesting and informative episodes from project members.

However, the descriptions of episodes were sometimes insufficient. The members of the inspection team who were outside the project teams had difficulty understanding what was going on in the projects. Professional ethnographers sometimes pointed out that the relationship of the episodes to the discussions was unclear. It took many hours to review episodes before people outside the work were able to easily understand the project contexts. Analysis also required many hours. We totaled the time we needed for each step. We found that the analysis step was the most time consuming task. Moreover, some inspectors were concerned that there would be a risk that they would not get interesting episodes within the time allotted for inspection, while they could achieve their goal of checking all guidelines in a guideline-based inspection within established time limits. In the future, we need to develop support tools for IT experts to conduct ethnographic inspection more effectively if we are to diffuse this practice company-wide.

Acknowledgments –We thank Jack Whalen, Erik Vinkhuyzen, Nozomi Ikeya, and Yutaka Yamauchi for basic training of ethnography and useful comments on our research. We also thank Julia Gluesing and anonymous reviewers for useful comments on the draft version of our paper.

Akihiko Obata is a senior researcher at Fujitsu Laboratories LTD. His current research focuses on knowledge management and organizational learning based on ethnography. His research interest also includes CSCW, and human interface.

Shigeru Yamada is a researcher at Fujitsu Laboratories LTD. His research area is from Organizational Development and Self-organized Meeting, Combination of Coaching and Ethnography, Usability Engineering and User Interface to Product Design.

Hiroaki Harada is a senior research fellow at Fujitsu Laboratories LTD. His research focuses on knowledge management and organizational learning based on ethnography.

Sadayo Hirata is a system engineer at Fujitsu Limited. Her research focuses on knowledge management and organizational learning based on ethnography.

Seisuke Ito is a system engineer at Fujitsu Limited. His research focuses on knowledge management and organizational learning based on ethnography.