Harmonizing Human Eyes with Digital Sensors

INTRODUCTION: COULD GROCERY SHOPPING EXPERIENCES BE MORE ATTRACTIVE?

The clouds have begun to gather over food retailing industry. In recent years conventional supermarkets in the United States have been losing their market share. Instead superstores, that distinguish themselves by enormous inventories with low-pricing strategy and organic food stores selling quality foods with relatively higher prices, in comparison, expand their market share and with a large variety of business formats, therefore, unfurl fierce competition in the industry (Food Marketing Institute, 2006). This trend forces the industry to operate at minimum profit margins (Roussos, 2004). Quite similar situations can be noticed in many developed nations such as Japan, Germany, and the United Kingdom.

One more serious issue in the industry is not just that quite a few consumers have been getting bored with shopping, they have, worse yet, unambiguously come to say “I hate shopping!” (Reid and Brown, 1996). Grocery shopping is the strongest contender for their vexation, so that experiences at grocery store, including supermarket, superstore, and even organic food store, are no longer the ones that may hold any sort of promises. In other words, grocery shopping has got to be one of chores.

Against this backdrop, we encounter various important questions: “Is grocery shopping a mere effort for consumers just to replenish supplies?” “Will grocery shopping at a physical store, eventually be replaced by home delivery in order to save hassles?” We believe grocery shopping is not the one “ticky-tacky” experience, rather, it consists of multi-strata experiences, i.e. each shopper has the opportunity to experience painful and/or delightful moments during the purchasing process (even during the same shopping trip).

As for our technology development project called “Smart Shopping-Aid”, we aimed to enhance shopper’s experiences by introducing new media equipments and information-communication systems at a physical grocery shopping environment. Our principal technological challenge was not to improve logistics or store operations, which was likely to be employed at similar technology development projects (for an overview of the area, see Krafft and Mantrala, 2006), but rather to ease shopper’s sore experiences as well as to expand capabilities for him/her to encounter enjoyable moments; this should be truly a human-centered approach.

For these reasons, we had to understand shopper’s experiences at a physical storefront as the first outset. Some basic questions arose here: How did he/she conduct everyday grocery shopping? What did he/she feel in the course of his/her shopping trip? What were the painful or delightful experiences and why? And, then, what did he/she expect explicitly and implicitly when grocery shopping?

OUR APPROACH: COMBINING PARTICIPANT OBSERVATION WITH DIGITAL-SENSOR OBSERVATION

To address the questions posed above, we paid particular attention to a holistic grocery shopping process because the shopping experience is, as we suggest, more than a composition of discrete actions and/or feelings towards a shopping arena. Rather it is a type of narrative featuring various vignettes.

But how should we approach this seemingly complex process with limited time and budget? Besides, our project team consisted of solely tech-heads: an HCI researcher with an artificial intelligence background (first author), a system engineering researcher (second author), and an interaction designer, and none were familiar with fieldwork¹ at the time.

Along with learning how to conduct fieldwork, we, as tech-heads, came up with an idea that we could use digital sensors as complementary means to observe shopper’s experiences. We already knew of earlier studies that utilized digital sensors to learn about shopper’s experiences in retail arena. Sorensen’s work is a prominent one: He installed location sensors in shopping carts and baskets, and observed more than 200,000 shopping trips in a laboratory store (Sorensen, 2003). Sorensen conducted a series of experiments to know “shopper behavior on a wall-to-wall, entrance-to-exit basis,” and clarified “the impact of location on shopper behavior, independent of the products in front of them.” Not only his discovery, but the approach itself was very attractive for us, one of the concerns for us was however that Sorensen broke “the entire shopping process into component parts.” As we mentioned earlier, we were interested in the entire shopping process because the purpose of our work was to know the truth about shopping, not the store layout’s.

We, therefore, reexamined his methods and framed our methods: to build complementary relationships between participant observation and digital-sensor observation. As for participant observation, we assumed we could take down detailed behaviors and utterances with relatively small number of shoppers. As for digital-sensor observation, we would be able to gather behavioral data with a much larger number of shoppers through multiple sensors embedded in shopping carts. We were convinced that each means had its respective advantages. Participant observation can preserve shopper’s behaviors and utterances that encapsulate a variety of contexts without spoiling these mutual relationships. On the other hand, digital-sensor observation enables us to record shopper’s entire moment-to-moment behaviors with unified metrics, i.e. it can easily bring about answers to such the questions: “How fast does a shopper walk around in the store in the course of his/her shopping trip?” “How many times does he/she stop by at the aisles?” We expected digital sensors could complement our weak point such that human perceptions were not always stable; under the condition accompanying time-space transition, observers face difficulties in becoming aware of subtle changes or differences of informant’s behaviors and emotions, although these sometimes come to be valuable resources for our critical insights.

By referring to the data derived from both means, we believed we could eventually obtain robust insights into shopping. We will discuss now these operational details in the following section.

RESEARCH SETTINGS

Research Procedure

As our lively research venue we chose an outlet of a regional supermarket chain³ based in Fukuoka City, which is one of ten largest cities in Japan. The outlet of the chain is medium sized and has about 12,000 SKUs lined up. It is located at a new residential area popular among wealthy young families and is recognized as a high-class supermarket sufficient for the area, i.e. it provides high-quality grocery foods with relatively higher prices, like organic food stores in the U.S.

As for participant observation, we had nine informants (two of whom were for pilot studies) chosen from the customer base of the outlet. All of them were female homemakers ranging from 30s to 50s and residing within walking distance from the store. The observations were conducted one by one with each informant from August to September 2005, for about two weeks. The procedure consisted of the four following steps:

1. Pre-interview at home (30-60 min.): interview with an informant regarding her rituals and attitudes of everyday grocery shopping; checking whether she had her shopping plan or not; and observing her preparation for the expedition, i.e. looking in the refrigerator,
2. Participant observation at the storefront (20–50 min): accompanying a participant on her shopping trip on an entrance-to-exit basis,
3. Participant observation at home (30–120 min.): describing the participant in the home-situation regarding storing, using, processing, and cooking purchased items, and
4. Post-interview at home (30-60 min.): debriefing what was not clarified during the previous steps.

We also required each informant to wear a heart rate meter to measure the informant’s heart beat interval, so called r-r interval or RRI⁴, so as to infer informant’s emotional state with uniform metrics. This is a precedent trial part of the digital-sensor observation that is described below.

As for the digital-sensor observation, we introduced five “SmartCart (Figure-1),” each of which was equipped with multiple sensors⁵ including a three-axle accelerometer, a two-axle gyrocompass, an RF-tag reader, three pressure sensors with a handgrip, and two digital video cameras. Corresponding to the RF-tag reader, about a hundred RF-tags, each of which emits its location-ID at every five seconds, were installed wall-to-wall within the store. As for the records of these sensor readings, we expected that we could infer multiple aspects of shopper’s behavior (see details at Table-1). We assumed video data would be the key for analyses because it represents shoppers’ integrated behaviors. When the other sensor readings would be analyzed with the video data, we could associate each extracted pattern from the readings with shopper’s particular behavior with real-life meanings. We will explain the actual process next.

SmartCarts were open to the public from February to March 2006 for about a month. About 120 customers used SmartCarts during this period, and about half permitted us to use their data for analyses.

TABLE-1

Specifications of sensor units equipped with SmartCart are shown.

FIGURE-1 A SmartCart used for our digital-sensor observation is shown. Besides multiple sensor units, it is also equipped with content delivery functions, although a very limited number of these were activated during the study.

ANALYSES AND INSIGHTS

Although preliminary analysis was begun during the participant observations period, we started a series of full-fledged analyses after finishing the digital-sensor observation. We introduced a very simple analytical framework that consisted of reciprocal reflections: one for the participant observation, and the other for the digital-sensor observation. From the former we obtained qualitative interpretations by understanding meanings of shoppers’ behaviors and utterances and such relationships. From the latter we learned the larger context of relationships of behaviors situated in the process: as we have already stated, unified metrics of the sensors could help us understanding subtle changes and/or differences of them according to larger time-space scales. Through the iterative analyses, we actually yielded plenty of substantial insights into shoppers.

Here we bring in a discovery process called “Three-Phase Model,” or TPM, which was employed as our core user-model while we developed new media systems described in the next section. To put it bluntly, TPM is a shopper’s cognitive process in the course of his/her shopping trip from-entrance-to-exit basis. There are three consecutive phases in his/her trip, each of which has totally different characteristics with each other. As such we were able to reveal the secrets through the process of analyses described below.

1. Noticing transformations of planned/unplanned purchase ratio: Through the analyses of the participant observation data, we noticed that the planned and unplanned purchase ratio tended to change steadily as the informants’ shopping proceeded: planned purchase was likely to occur in the anterior half and unplanned purchase, in comparison, increased in the posterior half regardless of their shopping paths nor of items bought. It is rational that an informant uttered in her shopping, “I basically start shopping from what I don’t have to forget to buy.” (a homemaker in her 40’s)
2. Discovering shopping process being consisted of three phases: Then we tackled with the sensor readings and found that there were two common points of variation in behavioral patterns in shopping cart orientation (Figure-2) and state of handgrip pressure (Figure-3). As for shopping cart orientation, the value tended to continuously decrease⁶ for a certain time at first, then it stopped decreasing and kept a solid state for a while, to finally start increasing till the end, even though the shoppers’ paths varied. This means shoppers tended to move counterclockwise till the first point of variation. It was followed by the solid state and then switched to move clockwise after the second point of variation. As for state of handgrip pressure, there appeared to be two states of shoppers frequently and repeatedly grasping and releasing handgrips of the carts. (We can also see a third state after 20 min. at Figure-3 but it was excluded because it clearly corresponded to behaviors preparing for checkout which were less related to the process accompanying shopper’s decision making.) Figure-3 shows shoppers were apt to keep grasping handgrips until the first point of variation. This was followed by a period when they often unlinked their hands from the handgrips and then, after the second point of variation, they shifted to alternately grasping and releasing the handgrips for a longer time. Interestingly, the timings of when these two points of variation arose are very consistent in the both sensor readings. We then investigated in multiple samples of video data and found that there were different types of shopper-floor interaction in each of the three phases, which were divided at the two points of variation. In the first phase, or Phase-1, shoppers had a tendency to stop at each floor for a short while and to choose items quickly. In the second phase, or Phase-2, they took a turn to stop at each visited floor for a longer period and to examine items carefully. In the third phase, or Phase-3, they were likely to go back and forth at the storefront and to reflect on items between aisles. These observations were accurately consistent with the patterns appeared in the two sensor readings.
3. Understanding characteristics of each phase: How different are the items bought in each of the three consecutive phases? We turned back to analysis of the participant observation data and broke down “planned/unplanned purchases” into five categories each of which was labeled with the informants’ intent based on actual usage of the items: these were 1) replenishment with a plan, 2) replenishment without a plan, 3) selection for main dish (dinner menu of the day), 4) selection for side dish (dinner menu of the day), and 5) other (e.g. impulse buy). When these labels (attributes) were applied to each informant’s shopping process, we could understand that most informants exhibited the patterns showed in Figure-4. That is, they were likely to fulfill the first intent (replenishment with plan) and to plan the dinner menu of the day⁷, especially for the main dish, in Phase-1, to fulfill the third and fourth intents (selection for main dish and for side dish) simultaneously in Phase-2, and to fulfill the second and fifth intents (replenishment without plan and other) simultaneously in Phase-3.
4. Understanding shopper’s state of mind in each phase: Through the analyses of RRI, we could know that each informant tended to decrease the rate of RRI as the phase proceeded. This implied shoppers got more relaxed with the proceeding phase. We could estimate that shoppers felt tense while performing two disparate tasks simultaneously in Phase-1, concentrated on items by performing two associated tasks in Phase-2, and felt relaxed to perform tasks without consistency and obligation in Phase-3 respectively
5. Constructing TPM: Synthesizing above analyses, we could obtain TPM which could adequately describe shopper’s cognitive processes in the course of her shopping trip at the storefront (see Figure-5).

FIGURE-2 A typical transition of shopping cart orientation is shown.

FIGURE-4 A typical shopping process consisted of five layers (categories) based on usage of the items bought/considered to buy is shown. Each mark denotes type of actions to the item: ○ denotes ”picked it and put it into basket,” □, ”picked it but returned it to shelf,” ◇, ”gazed at it,” and △, ”glanced it respectively.

FIGURE-5 Three-Phase Model (TPM) as one of major discoveries from iterative analyses with our framework

INFORMING DESIGN

While we developed new media systems integrated with shopping carts, we, consequently, developed two service applications making use of TPM: as a shopper’s context transforms, a particular service exerts never uniform, but should be dynamically shifted as well. We assessed the best allocation and configuration of the service types by examining when and how the each service type should be activated. The brief service concept of each is as follows:

• Recipe Recommendations: As recipes are effective information for a shopper’s meal arrangement, they will tend to be used in an earlier part of the process. Since we knew that a shopper usually plans his/her main dish in the first phase and side dish in the second phase, an appropriate series of contents could be provided at each respective phase.
• Product Recommendations: Although product information seems valuable throughout the process, each phase will favor different kinds of information. In the first phase, information regarding a fresh food sale may help a shopper to develop his/her plan for his/her dinner arrangement, especially for his/her main dish. In the second phase, information that enables a shopper to compare prices, qualities, and features of an array of choices may help a shopper to fulfill his/her plan for a main dish and side dish. In the third phase, information regarding new and luxury articles may help a shopper to suffice his/her curiosity about grocery items.

In fact, these services could gain a welcome acceptance while the operational test took place at the store in the early autumn in 2006. Although I should withhold detailed results of the test here, one important lesson from the test was that these services were effective especially in the third phase. As we learned that a shopper tended to feel relaxed in the phase, he/she was likely to take new information as a part of his/her shopping experience during the term. We think it is an important implication and we have to think of new ideas to make shopping experiences more attractive.

FURTHER REFLECTIONS

We have reported our process in which we tried to build complementary relationships between participant observation and digital-sensor observation. Generally speaking, we recognized that our aim was successfully accomplished. Most of all, we could conduce shopper’s latent behavioral patterns under longer time-space transition by introducing digital-sensor observation, and we could further obtain deep insights by giving interpretations from the analyses on participant observation to the patterns.

Besides, we could also bring about instrumental outcomes. In the domain of system design, “design implication,” which has been a practical form of ethnographic outcome, has often left hard-to-traverse boundaries concerning design practices, i.e. it leaves unsolved problems to identify critical system parameters. However, our method includes the discovery of parameters into an ethnographic research preventing any sort of the boundaries arising. In fact, we could serve a location detector using RF-tags, a gyro compass, and pressure sensors with a handgrip as the system input for our specification.

On the other hand, problems regarding types of sensors to use still exist. Reflecting on the lessons to be learned from the research, we have come to believe that adequate choice, configuration, and measurement of sensors can go together with open-ended qualitative analyses by giving adequate research focuses based on a broad range of human perspectives. In that sense, concurrent collection of video data is a key factor, i.e. matching between recordings of behaviors and/or utterances of an informant and patterns included in sensor readings accommodates exhaustive investigation in human experience; although it brought us much more work as well! As a future technical contribution, a data mining method dealing with stream sensor readings and video data will play a critical role in the research process.

Our method is still in an early phase of development, and therefore, needs more time to be adapted to gain in our everyday lives from digital sensors. We believe however, that our aim is a valuable venture where ethnography praxis is likely to penetrate into the industrial world more deeply and reliably.

Acknowledgments – We’d like to extend our deepest gratitude to Bonrepas Corporation at first. Without its dedicated contribution including providing a store for our longitudinal research base, recruiting informants from her customer base, and gathering and editing vast product information, we could not realize the research. We’re also grateful for Yasuhisa Ito and Naoko Okuizumi, both of who are hard-to-find professional field-researchers in Japan. Due to their generous cooperation, we could blaze a trail to our research process. While working on this manuscript, attentive help from Rich Radka, our session chair of EPIC 2007, has been a mariner’s compass for us. We appreciate his thoughtfulness and patience.

Hiroshi Tamura Research Director, Hakuhodo Inc. Received BA (Cognitive Psychology, 1994) and MS (Interdisciplinary Information Studies, 2003) from The Univ. of Tokyo. He has researched creativity support systems as an interdisciplinary study of artificial intelligence, cognitive science and interaction design. He co-chairs User-study Forum at Information Processing Society of Japan, or IPSJ, since 2006.

Tamami Sugasaka Researcher, Fujitsu Laboratories Ltd. Received BS (Computer Science, 1988) from Waseda Univ. She has researched broad-ranging applications of artificial intelligence including neural-network, software agent, and intelligent robot. Her current interest focuses on bringing emerging technologies into society. She is a core member of User-study Forum at IPSJ since 2006.

NOTES

¹ It is unfortunate to say that field research industry is immature in Japan. Field research suffers from untrained designers and engineers. One of the problems is that they tend to gather instrumental data for design and development activities, and do not deeply understand users and/or settings.

² Our previous survey, which was conducted in Tokyo metropolitan area in 2005, clarified that more than 80 percent of respondents went grocery shopping once in less than a few days.

³ This supermarket chain recently experienced some difficulty by a decrease in its variety and breadth of offerings. Thanks to the development and spread of information technology as well as those of physical distribution technology, competitors could easily respond. The supermarket chain therefore, was looking for a new differentiation strategy and expected our project might bring such opportunities.

⁴ RRI is the time duration between two consecutive peak waves (R waves) of the ECG (electrocardiogram) and is an established index of variables to measure user’s mental workload. In general, RRI decrease is more sensitive to increases in workload than is RRI increase.

⁵ These sensor units were chosen by referring to a number of previous studies in the area of context-aware computing. As a matter of fact, many researchers currently use all sorts of readings derived from these sensor units to infer a variety of user behaviors and contexts (for an overview of the area, see Chen and Kotz, 2000). In particular, angular velocity and pressure value (with the handgrip) could well contribute to our analyses described in the next section.

⁶ Decreasing value in shopping cart orientation meant the cart is rotating in counterclockwise direction, and the reverse meant clockwise rotation.

⁷ In Phase-1, most informants expressed that they were thinking of their dinner menu and had not yet decided. Among interesting behaviors observed was that an informant directly went to the deli floor right after starting to shop, closely looking at some items, saying “they’re very helpful to plan my dinner menu as well as to know how to cook (the menu)”.