The Future Of Virtual Reality And Eye-Tracking VR Technology?

This question originally appeared on Quora. Answer by Kynan Eng.

I think that most of the first uses of AR/VR eye tracking will be to improve general AR/VR headset comfort and usability. Why? As a direct input device, eye tracking is actually fairly frustrating and useless. However, as a general contextual signal indicating possible user intent or attentional focus, it is quite useful. Many use cases for eye tracking will work in the background, and will probably include the following:

• Graphics rendering resource allocation: If a person is looking somewhere, more graphics rendering resources can be allocated in that general direction. This can provide better-quality output for a given amount of rendering power.
• Data pre-fetch: Some VR data operations require time to complete, e.g. looking up something in an online database. If a person glances in a particular direction, data fetching can begin in the background even before the person selects an item to interact with. This improves the perceived responsiveness of the VR environment, which can be especially useful e.g. over mobile data networks.
• Multi-modal smart 3D object selection: In VR, pointing at a small object in a cluttered environment can be quite difficult. Eye tracking can be used to help disambiguate the object that the user intends to select by combining the information with the controller input.
• Automatic headset calibration: A headset which knows where the user’s eyes are can better adjust its own image output parameters for optimal user comfort.
• Balance manipulation: The vestibulo–ocular reflex is a well-known automatic effect linking eye movements to changes in the vestibular system. Knowing eye movements as well as those of the headset (via accelerometers) allow deductions about the likely state of the user’s vestibular system, and thus enables systematic manipulations to heighten changes in balance or possibly to reduce the effects of motion sickness during VR use.

What all of these use cases have in common is that, when working well, you don’t notice that they are doing anything. In fact, I would go as far as to guess that some of these eye tracking use cases are essential for enabling truly useable VR/AR for mass-market applications.