Apple Patents Mixed Reality Headset With Face Tracking

Apple has recently filed a patent that describes a mixed reality headset with face tracking capabilities.

A new patent filing describes Apple’s plans to create a mixed reality headset that not only gives its wearer a view of the world in 3D and augmented with virtual objects -- it can also monitor the wearer’s eyes and facial expressions as well.

The patent “Display System Having Sensors,” first spotted by Patently Apple, describes a head-mounted display (HMD) that uses a variety of outward-facing cameras or sensors. These sensors may include those that capture depth or range information, lighting information, as well as those that can capture information about the user's position, orientation, and motion in the environment.

These sensors will take the information in, process it, and then present it to the user on a display. This 3D view of the world can then be augmented with virtual reality objects. Users will be able to interact with the 3D environment, including the virtual objects, making for a totally immersive experience.

The patent also described how the HMD will use sensors to take user information in. Some embodiments may include “one or more sensors” that keep track of the wearer’s eye position and movements, the expressions of the user’s eyebrows and forehead, the expressions or movements of the user’s mouth or jaw.

Some embodiments also said that the HMD might include some sensors that will track the position, movement and gestures of the wearer’s hands, fingers, and/or arms.

Uses?

Apple has already developed facial tracking software for use on Animoji. This software makes use of an iPhone’s selfie camera to track a user’s facial expressions and then translate that into corresponding animation.

The new technologies described in the patent, particularly the eyebrow and jaw sensors, will be very important for Apple’s Animoji and Memoji avatars, The Verge noted. Their importance will be made even more apparent should the avatars be ported into a headset.

Variety noted that according to the patent, the sensors used to track the user’s eyes and face can also be used for biometric applications. The sensors that keep track of a user’s hand, finger and arm movements can also be used for gesture tracking.

So far, Apple hasn’t announced anything about its mixed reality headset. Previous reports said it was terminated, but patents keep surfacing, indicating that it could be coming.