3D video content has a significant limitation, one that is not trivial to solve. Video captured by a camera — even one with high resolution and a very wide field of view — still records a scene as a flat plane, from a fixed point of view. The limitation this brings will be familiar to anyone who has watched a 3D video (or “360 video”) in VR and moved their head the wrong way. In these videos one is free to look around, but may not change the position of their head in the process. Put another way, pivoting one’s head to look up, down, left, or right is fine. Moving one’s head higher, lower, closer, further, or to the side? None of that works. Natural movements like trying to peek over an object, or moving slightly to the side for a better view simply do not work.
Light field video changes that. It is captured using a device like the one in the image above, and Google has a resource page giving an excellent overview of what light field video is, what it can look like, and how they are doing it. That link covers recent improvements to their camera apparatus as well as to video encoding and rendering, but serves as a great show-and-tell of what light fields are and what they can do.
The meta-camera is a hemisphere just under one meter in diameter that contains an array of 46 time-synchronized action cameras pointed outwards. On the software end, camera input is used to reconstruct the scene and create a 6 DoF volumetric video. In other words, the perspective of the video correctly changes depending on a user moving their point of view (within an area very roughly corresponding to the size of the camera device, anyway.)
The other significant improvement is in compression and rendering of the resulting video. By reducing the video down to a small, fixed number of depth layers to represent the light field content, conventional video encoding and compression can be leveraged to deliver lightweight representations that can render easily on just about any platform. A picture is worth a thousand words, so here is a short animation showing a light field image. The point of view moves in a figure eight, and the perspectives and sight lines all change exactly as one would expect them to. The animation also briefly peeks behind the curtain, showing the color-coded depth layers that the software uses to decide what belongs where.
You can download the PDF of the SIGGRAPH 2020 technical paper, or browse the DeepView resource page hosted on GitHub for plenty of in-browser demos and a downloadable VR headset demo. The team’s video presentation is also embedded below, and gives an excellent overview.
Light fields don’t have to be complex affairs, and there is plenty of room for curious hackers to explore. Interested? [Alex Hornstein] shared a fascinating presentation on working with light fields in his 2018 Hackaday Superconference talk, so check it out.
Light Fields: Missing Ingredient for Immersive 3D Video Gets Improved
Source: HackADay
