Patent No. 6611196 System and method for providing audio augmentation of a physical environment

 

Patent No. 6611196

System and method for providing audio augmentation of a physical environment (Mynatt, et al., Aug 26, 2003)

Abstract

A system and method are provided for implementing the transmission of information to users--via peripheral, or background, auditory cues--in response to the physical action of the users in a particular environment, e.g., the workplace. The system combines three known technologies: active badges, distributed systems, and digital audio delivered via portable wireless headphones.

Notes:

 

BACKGROUND OF THE INVENTION

This invention relates to a system for providing unique audio augmentation of a physical environment to users. More particularly, the invention is directed to an apparatus and method implementing the transmission of information to the users--via peripheral, or background, auditory cues in response to the physical but implicit or natural action of the users in a particular environment, e.g., the workplace. The system in its preferred form combines three known technologies: active badges, distributed systems, and digital audio delivered via portable wireless headphones.

While the invention is particularly directed to the art of audio augmentation of the physical workplace, and will be thus described with specific reference thereto, it will be appreciated that the invention may have usefulness in other fields and applications.

Considering the richness and variety of activities in the typical workplace, interaction with computers is relatively limited and explicit. Such interaction is primarily limited to typing and mousing into a box while seated at a desk. The dialogue with the computer is explicit. That is, we enter in commands and the computer responds.

Part of the reason that interaction with computers is relatively mundane is that computers are not particularly well designed to match the variety of activities of the typical human being. For example, we walk around, get coffee, retrieve the mail, go to lunch, go to conference rooms and visit the offices of coworkers. Although some computers are now small enough to travel with users, such computers do not take advantage of physical actions.

It would be advantageous to leverage everyday physical activities. For example, an opportune time to provide serendipitous, yet useful, information by way of peripheral audio is when a person is walking down the hallway. If the person is concentrating on their current task, he/she will likely not even notice or attend to the peripheral audio display. If, however, the person is less focused on a particular task, he/she will naturally notice the audio display and perhaps decide to attend to information posted thereon.

Additionally, it would be advantageous if physical actions could guide the information content. For example, a pause at a coworker's empty office is an opportune time for the user to hear whether their coworker has been in the office earlier that day.

Unfortunately, known systems do not provide for these types of interactions with computer systems. Most work in augmented reality systems has focused on augmenting visual information by overlaying a visual image of the environment with additional information, usually presented as text. A common configuration of these systems is a hand-held device that can be pointed at objects in the environment. A video image with overlays is displayed in a small window.

These types of hand-held systems have two primary disadvantages. First, users must actively probe the environment. The everyday pattern of walking through an office does not trigger the delivery of useful information. Second, users only view a representation of the physical world, and cannot continue to interact with the physical world.

Providing auditory cues based on the motion of users in a physical environment has also been explored by researchers and artists, and is currently used for gallery and museum tours. These include a system described by Bederson, et al., "Computer Augmented Environments: New Places to Learn, Work and Play", in Advances in Human Computer Interaction, Vol. 5, Ablex Press. Here, a linear, usually cassette-based audio tour is replaced by a non-linear sensor-based digital audio tour, allowing the visitor to choose their own path through a museum. A commercial version of the Bederson system is believed to be produced under the name Antenna Galley Circle.TM..

Several disadvantages of this system exist. First, in Bederson's system, users must carry the digital audio with them, imposing an obvious constraint on the range and generation of audio cues that can be presented. Second, Bederson's system is unidirectional. It does not send information from a user to the environment such as the identity, location, or history of the particular user.

Other investigations into audio awareness include Hudson, et al., "Electronic Mail Previews Using Non-Speech Audio", CHI '96 Conference Companion, ACM, pp. 237-238, who demonstrated providing iconic auditory summaries of newly arrived e-mail when a user flashed a colored card while walking by a sensor. This system still required active input from the user and only explored one use of audio in contrast to creating an additional auditory environment that does not require user input.

Explorations in providing awareness data and other forms of serendipitous information illustrate additional possible scenarios in this design space. Ishii et al.'s "Tangible Bits: Towards Seamless Interfaces Between People, Bits and Atoms", in Proc. CHI'97, ACM, March 1997, focuses on surrounding people in their office with a wealth of background awareness cues using light, sound and touch. This system does not follow the user outside of their office and does not provide for the triggering of awareness cues based on the activities of the user.

Gaver et al., "Effective Sound in Complex Systems: The ARKola Simulation", Proc. CHI'91, ACM Press, pp. 85-90, explored using auditory cues in monitoring the state of a mock bottling plant. Pederson et al., "AROMA: Abstract Representation of Presence Supporting Mutual Awareness", Pro. CHI'97, ACM Press, 51-58, has also explored using awareness cues to support awareness of other people.

Another area of computing that relates generally to electronically monitoring information concerning users and machines, including state and locational or proximity information, is called "ubiquitous" computing. The ubiquitous computing known, however, does not take advantage of audio cues on the periphery of the perception of humans.

The following U.S. patents commonly owned by the assignee of the present invention generally relating to ubiquitous computing are incorporated herein by reference:

U.S. Pat. No. Inventor Issue Date 5,485,634 Weiser et al. Jan. 16, 1996 5,530,235 Stefik et al. Jun. 25, 1996 5,544,321 Theimer et al. Aug. 6, 1996 5,555,376 Theimer et al. Sep. 10, 1996 5,564,070 Want et al. Oct. 8, 1996 5,603,054 Theimer et al. Feb. 11, 1997 5,611,050 Theimer et al. Mar. 11, 1997 5,627,517 Theimer et al. May 6, 1997

Therefore, it would be advantageous if a system was provided that: 1) transmitted useful information to a user via peripheral audio cues, such transmission being triggered by the passive interaction of the user in, for example, the workplace, 2) allowed the user to continue to interact in the physical environment, physically uninterrupted by the transmission, 3) allowed the user to carry only lightweight communication hardware such as badges and wireless headphones or earphones instead of more constraining devices such as hand held processors or CD players and the like, and 4) accomplished and manipulated bidirectional communication between the user and the system.

The present invention contemplates a new audio augmentation system which achieves the above-referenced advantages, and others, and resolves appurtenant difficulties.

SUMMARY OF THE INVENTION

In the subject invention, audio is used to provide information that lies on the edge of background awareness. Humans naturally use their sense of hearing to monitor the environment, e.g., hearing someone approaching, hearing someone saying a name, and hearing that a computer's disk drive is spinning. While in the midst of some conscious action, ears are gathering information that persons may or may not need to comprehend.

Accordingly, audio (primarily non-speech audio) is a natural medium to create a peripheral display in the human mind. A goal of the subject invention is thus to leverage these natural abilities and create an interface that enriches the physical world without being distracting to the user.

The subject invention is also designed to be serendipitous. That is, the information is such that one appreciates it when heard, but does not necessarily rely on it in the same way that one relies on receiving a meeting reminder or an urgent page. The reason for this distinction should be clear. Information that one relies on must penetrate beyond a user's peripheral perceptions to ensure that it has been perceived. This, of course, does not imply that serendipitous information is not of value. Conversely, many of our actions are guided by the wealth of background information in our environment. Whether we are reminded of something to do, warned of difficulty along a potential path, or simply provided the spark of a new idea, opportunistic use of serendipitous information makes lives more efficient and rich. The goal of the subject invention is to provide useful, serendipitous information to users by augmenting the environment via audio cues in the workplace.

Thus, in accordance with the present invention, a system and method for providing unique audio augmentation of a physical environment is implemented. An active badge is worn by a user to repeatedly emit a unique infrared signal detected by a low cost network of infrared sensors placed strategically around a workplace. The information from the infrared sensors is collected and combined with other data sources, such as on-line calendars and e-mail cues. Audio cues are triggered by changes in the system (e.g. movement of the user from one room to another) and sent to the user's wireless headphones.

Further scope of the applicability of the present invention will become apparent from the detailed description provided below. It should be understood, however, that the detailed description and specific examples, while indicating preferred embodiments of the invention, are given by way of illustration only, since various changes and modifications within the spirit of the scope of the invention will become apparent to those skilled in the art.