Apple’s Mixed Reality Headset Features Advanced RGB-D Sensors In New Patent

A new patent reveals Apple’s plans for a mixed-reality headset that features advanced sensors used in producing accurate 3D models of a real-life scene.

According to a patent published by the U.S. Patent and Trademark Office, Apple plans to use RGB depth sensors to allow future headsets to produce accurate 3D models of real-life scenes. The patent, first spotted by Patently Apple, was filed earlier this year.

The patent titled “Reconstructing Views Of Real World 3D Scenes” described how devices such as mixed reality headsets can make use of RGB-D sensors, such as RGB depth cameras, to construct 3D renderings based on real-world scenes captured by the RGB-D sensor.

One example implementation described in the patent involved the device or headset receiving multiple frames of a real world scene well within a camera’s field of view (FOV) at multiple instances. The frames received include color values and depth values for the pixels for the camera’s FOV.

The device will then select keyframes from the multiple frames it received. The keyframes are associated with camera poses defined or in relation to a three dimensional (3D) coordinate system.

After that, the device will receive another frame showing the current real-world scene, also found within the camera’s FOV. This frame also includes color values and current depth values for the pixels for the camera’s FOV, all at a current time. The device will determine the current camera pose of the camera in the 3D coordinate system as well.

The result is that the device will then be able to provide a virtual or 3D representation of the current frame based on the current camera pose and two or more keyframes. This representation, or model, will be based on the color values and the depth values of two or more keyframes.

RGB-D sensors

RGB-D sensors aren’t new technology. These sensors have been used in the Xbox Kinect as well as the Asus Xtion Pro - two devices that have been available in the market for years. Apple’s new patent, Patently Apple noted, simply shrinks the tech for smaller devices such as a future headset.