Patent No. 6614408 Eye-tap for electronic newsgathering, documentary video, photojournalism, and personal safety

 

Patent No. 6614408

Eye-tap for electronic newsgathering, documentary video, photojournalism, and personal safety (Mann, Sep 2, 2003)

Abstract

A novel system for a new kind of electronic news gathering and videography is described. In particular, a camera that captures light passing through the center of a lens of an eye of the user is described. Such an electronic newsgathering system allows the eye itself to, in effect, function as a camera. In wearable embodiments of the invention, a journalist wearing the apparatus becomes, after adaptation, an entity that seeks, without conscious thought or effort, an optimal point of vantage and camera orientation. Moreover, the journalist can easily become part of a human intelligence network, and draw upon the intellectual resources and technical photographic skills of a large community. Because of the journalist's ability to constantly see the world through the apparatus of the invention, which may also function as an image enhancement device, the apparatus behaves as a true extension of the journalist's mind and body, giving rise to a new genre of documentary video. In this way, it functions as a seamless communications medium that uses a reality-based user-interface.

Notes:

BACKGROUND OF THE INVENTION

In photojournalism, electronic news gathering (ENG), and in movie or video production, it is desirable to capture events in a natural manner with minimal disturbance to the subject matter being captured. Current state-of-the-art newsgathering apparatus creates a visual disturbance to others and attracts considerable attention on account of the camera crew, sound crew, the bulky equipment, and the manner in which it is deployed and used. Even when a single camera operator is assigned to a newsgathering task, the simple gesture of bringing or holding the camera up to the eye can disrupt the event being reported. Even if the size of the camera could be reduced to the point of being negligible (e.g. no bigger than the eyecup of a typical camera viewfinder, for example), the very gesture of bringing a device up to the eye is unnatural and attracts considerable attention, especially in establishments such as gambling casinos or department stores where photography is often prohibited, yet where newsworthy events frequently happen. Although there exist a variety of covert cameras which are frequently used in investigative journalism, such as a camera concealed beneath the jewel of a necktie clip, cameras concealed in baseball caps, and cameras concealed in eyeglasses, these cameras tend to produce inferior images, not just because of the technical limitations imposed by their small size, but, more importantly because they lack a means of viewing the image, as well as a means of having others view the image. In typical newsgathering scenarios, one person operates the camera while another carries a monitor to observe the technical quality of the video images. Often there is a news truck equipped with instrumentation so that technical staff can monitor the quality of the video signal, and report back to the camera crew by telephone or the like. Because of the lack of viewfinder means, and the lack of analysis/communications means between the camera operator and remote test equipment/technical staff, investigative video and photojournalism made with concealed cameras of the prior art suffers from poor composition and poor image/sound quality.

Recent experiments, as conducted by and reported by Mann, in a Massachusetts Institute of Technology (M.I.T.) technical report titled Mediated Reality (M.R.), Vision and Modeling Group TR260, (1994), which is available online at http://wearcam.org/mr.htm, show that moderate transformations such as rotation by a few degrees or moderate image displacements, often give rise to a reversed aftereffect that is more rapidly assimilated by the wearer than either very large or very small transformations. It is also shown that effects of moderate transformations can often have a more detrimental effect on performing other tasks through the camera as well as detrimental flashbacks upon removal of the camera, than that which would arise from either extreme or negligible transformations. This work also looked into the effects of using wireless communications with a remote image processing system as a means of mediated reality. These findings suggest that merely mounting a conventional camera such as a small 35 mm rangefinder camera or a small video camcorder to a helmet, so that one can look through the viewfinder and use it it hands-free while performing other tasks, will result in poor performance at doing those tasks while looking through the camera viewfinder, in addition to the obvious shortcoming of not having technical staff or other remote collaborators available.

Part of the reason for poor performance associated with simply attaching a conventional camera to a helmet is the induced noncollinearity (failure to provide a truly orthoscopic view). Even viewfinders which correct for parallax, as described in U.S. Pat. No. 5,692,227 in which a rangefinder is coupled to a parallax error compensating mechanism, only correct for parallax between the viewfinder and the camera lens that is taking the picture, but do not correct for noncollinearity between rays of light passing through the viewfinder and those that would be observed with the naked eye while not looking through the camera.

An object of the invention is to provide a viewfinder means that is suitable for long-term telepresence, computer supported collaborative photojournalism etc., suitable when wearing the camera for an entire day, looking through it all the while.

An important aspect of the invention is the capability of the apparatus to mediate (augment, diminish, or otherwise alter) the visual perception of reality. Traditional camera viewfinders often include the ability to overlay virtual objects, such as camera shutter speed, or the like, on top of reality, as described in U.S. Pat. No. 5,664,244 which describes a viewfinder with additional information display capability.

This electronic news gathering invention is related to known displays that are used in the field of Virtual Reality (VR) in the sense that both are wearable. However, an important difference is that embodiments of the invention allow the wearer to continue to see the real world, while VR displays block out the ability to see the real world.

Displays for helmet mounted aircraft weapons aiming applications have been developed, as described in U.S. Pat. Nos. 3,697,154, 3,833,300, 4,081,209, 4,220,400. Such displays do not directly incorporate a camera. Although they could be used to display the output image from an electronic camera (hand-held or perhaps mounted to the same helmet), the above-mentioned problems would still exist.

U.S. Pat. No. 4,806,011 describes an eyeglass-based display of a clock or the like.

Open-air viewfinders are often used on extremely low cost cameras, as well as on some professional cameras for use at night when the light levels would be too low to tolerate any optical loss in the viewfinder. Examples of open-air viewfinders used on professional cameras, in addition to regular viewfinders, include those used on the Grafflex press cameras of the 1940s (which had three different kinds of viewfinding means), as well as those used on some twin-lens reflex cameras. While such viewfinders could be used in the context of the invention, and would have the advantage of not inducing the problems such as flashback effects described above, they also fail to provide an electronically mediated reality.

Wearable display devices have been described, such as in U.S. Pat. Nos. 5,546,099, 5,708,449, 5,331,333, 4,636,866, but have no light sensing apparatus or wireless communications capability.

U.S. Pat. No. 5,640,221 also proposes an eye-tracking device which may be used in the context of the invention.

DESCRIPTION OF THE INVENTION

The apparatus of the invention can allow visual reality to be mediated in order to make certain that exposure is correct as well as to keep the wearer of the apparatus in the feedback loop of the photo compositional process by constantly providing the wearer with a video stream. Moreover, it is desired that the apparatus will allow the wearer to experience a computationally mediated visual reality, and for that experience to be shared through wireless communications networks so that the wearer may receive additional visual information, as well as be aware of modifications to visual reality that might arise, for example, as part of a communications process in a shared virtual environment. For such compositional and interactional capabilities, a simple air-based viewfinder is inadequate.

The invention facilitates a new form of visual art, in which the artist may capture, with relatively little effort, a visual experience as viewed from his or her own perspective. With some practice, it is possible to develop a very steady body posture and mode of movement that best produces video of the genre pertaining to this invention. Because the apparatus is lightweight and close to the head, there is not the protrusion associated with carrying a hand-held camera. Also because components of the apparatus of the invention are mounted very close to the head, in a manner that balances the weight distribution as well as minimizes the moment of inertia about the rotational axis of the neck, the head can be turned quickly while wearing the apparatus. This allows one to record the experiences of ordinary day-to-day activities from a first-person perspective, and, because of a communications infrastructure, to convey these experiences to a remote entity. Moreover, because both hands are free, much better balance and posture is possible while using the apparatus. Anyone skilled in the arts of body movement control as is learned in the martial arts such as karate, as well as in dance, most notably ballet, will have little difficulty capturing exceptionally high quality video using the apparatus of the invention.

With the prior art, the best camera operators tend to be very large people who have trained for many years in the art of smooth control of the cumbersome video or motion picture film cameras used. In addition to requiring a very large person to optimally operate such cameras, various stabilization devices are often used, which make the apparatus even more cumbersome. The apparatus of the invention may be optimally operated by people of any size. Even young children can become quite proficient in the use of some embodiments of the invention, and could make excellent photojournalists, especially if given the capabilities of a network of adult experts.

A typical embodiment of the invention comprises one or two spatial light modulators or other display means built into a pair of eyeglasses together with one or more light sensor arrays, a body-worn computer and image processing system, and a high-speed wireless communications link. Typically one or more CCD (charge coupled device) image sensor arrays and appropriate optical elements comprise the camera portion of the invention. Typically a beamsplitter or a mirror silvered on both sides is used to combine the image of the viewfinder with the apparent position of the camera. The viewfinder is not just a means of determining the extent of coverage of the camera in a natural manner, but it is also a communications means, so that, for example, a remote expert may provide advice in the form of text superimposed inside the viewfinder. In this manner the viewfinder functions as a teleprompter, as well as a means for judging photographic or video composition. Moreover, one or more remote experts may superimpose graphical elements in the viewfinder, such as a cursor or arrow that points to an object as depicted in the viewfinder. In this way the wearer may collaborate with one or more remote experts on matters such as scene content, or, for example, specific details of a photographic composition. Finally, the collaboration may take the form of one or more computer graphics renderings inserted into the viewfinder, together with the images of real objects. This allows the user to experience a computer-mediated reality in which there is a much more rich form of collaborative potential.

In some embodiments of the invention the viewfinder has a focusing mechanism that is coupled to a focusing mechanism of a camera system. In such embodiments, when the camera is focused on a particular object, the viewfinder also presents that object in a manner such that when the apparatus moves relative to the user's eye, the object appears to neither move with or against the movement of the eye, e.g. rays of light entering the eye are approximately collinear to corresponding rays of light that would be present if the apparatus were not present. This focusing of the camera may be manual or automatic, but in both cases, it is preferable that the viewfinder be either automatic in tracking the camera (whether manual or auto focus), or that it have sufficient depth of focus to allow a lens of an eye of the wearer itself to function as the focal selector, or that it be responsive to the focal state of a lens of an eye of the wearer. A viewfinder meeting one of these criteria will be referred to as an autofocus viewfinder.

A viewfinder-like function may come from various forms of devices that generate a visible image or a visual perception thereof. In some embodiments, a viewfinder which is a small television tube or LCD screen with appropriate optics so that the eye can focus on it as if it were far away, is used. In other embodiments the viewfinder-like function takes the form of an image created directly upon the retina of an eye of the wearer of the apparatus. A viewfinder device, or other kind of device that makes the eye perceive a picture, or creates a picture visible to the eye, or inside the eye, does the opposite of what the camera does, in the sense that it turns electrical signals into pictures or a visible perception like seeing a picture, while the camera turns pictures (light falling on an image sensor) into electrical signals. Because the viewfinder device, or equivalent device that turns electrical signals into pictures or visible perceptions equivalent to seeing or experiencing pictures, does the opposite of what a camera does, I will call the device an aremac. This word "aremac" is simply the word "camera" spelled backwards.

Preferably the viewfinder is a laser-based aremac, or other kind of aremac that has either infinite depth of focus or a sufficient depth of focus that there is a perceived unity between virtual and real objects for real objects at any depth plane. Alternatively, the aremac may be an automatic focus viewfinder, or other device where the limited depth of focus is mitigated by an automatic focusing system.

Preferably a collinearity criterion is satisfied, e.g. a wearable camera system with aremac is arranged so that the aremac displays video from the camera in such a way that all rays of light from the aremac that enter the eye appear to emanate from essentially the same direction as they would have had the apparatus not been worn.

Preferably the wearer experiences additional information overlaid on top of his or her visual field of view such that the information is relevant to the imagery being viewed, and also such that the virtual objects appear in the same depth plane as the real objects.

The apparatus of the invention gives a photojournalist a means of determining the composition of a picture from a display device that is located such that only the photojournalist can see the display device, and so that the photojournalist can ascertain the composition of a picture and take a picture or video and transmit these to one or more remote locations without the knowledge of others in the immediate environment.

The camera viewfinder can also function as a way of providing a photojournalist with the ability to collaborate with one or more remote entities, in matters pertaining to composition of a picture or video or in matters pertaining to an interview with one or more subjects.

In some embodiments, the viewfinder has a focusing mechanism coupled with the focus of a camera, such that both can be operated remotely with a single control, by a technician or camera operator at a remote location.

Some embodiments of the invention also provide a means by which a technician or camera operator at a remote location can signal to the wearer of the viewfinder which direction in which to turn, for best picture. These signals are typically vibrotactile or facilitated by direct electrical stimulation of the actual muscles needed to effect the actual change (e.g. a message to turn to the right is felt as a tug of the neck muscles that actually turn the head to the right).

In some embodiments a technician or camera operator at a remote location can signal to the wearer of a wearable camera which has no local viewfinder, which direction in which to turn, for best picture, while the remote technician or camera operator remotely monitors the video signal from the wearable camera.

Multiple photojournalists using the invention, at the same location, can also collaborate in such a way that multiple camera viewpoints may be shared among the photojournalists so that they can advise each other on matters such as composition, or so that one or more experts at remote locations can advise one or more of the photojournalists on matters such as composition or camera angle.

Certain embodiments of the invention provide a new genre of journalism in which content may be produced and consumed using the same wearable camera and display system. This content sharing capacity allows for such collaboration among photojournalists and others.

Typically embodiments of the invention allow a photojournalist to wear the apparatus continuously and therefore always end up with the ability to produce a picture from something that was seen a couple of minutes ago, through a retroactive record function into a circular buffer. For example, a "begin recording from 5 minutes ago" button works well if the apparatus is always worn and always ready.

Moreover, as an artistic tool for first-person perspective photojournalism, the apparatus allows the photojournalist to record, from a first-person-perspective, experiences that have been difficult to so record in the past. For example, a photojournalist might be able to record the experience of looking through binoculars while riding horseback, or the experience of waterskiing, rope climbing, or the like. Such experiences captured from a first-person perspective provide a new genre of video by way of a wearable camera system with viewfinder means that goes beyond current state-of-the-art point of view sports videos (such as created by cameras mounted in sports helmets which have no viewfinder means).