Support NeoGAF

[Raticus79]

Happening now through July 28th:

NVIDIA blog: https://blogs.nvidia.com/blog/2016/07/21/rendering-foveated-vr/
Research paper: https://research.nvidia.com/publication/perceptually-based-foveated-virtual-reality

Ideally, foveated rendering should be undetectable by the user. They found that as they progressively lowered the resolution of the periphery and increased the peripheral blur (achieving greater rendering efficiency), people could tell that something was different because contrast was being lost. The innovation here is increasing the contrast of the periphery to compensate - so they can get those efficiencies without any perceivable difference to the user. There's a video of this in action on their blog (direct link).

Older posts with a bit more background info on foveated rendering itself:
https://www.vrfocus.com/2016/07/nvidia-and-smi-to-demo-foveated-rendering-at-siggraph-2016/
http://uploadvr.com/nvidia-smi-siggraph-foveated-rendering-eye-tracking/

One of the next big leaps in VR technology will arrive when headsets come with eye trackers. The addition will allow the computer to optimize the river of images floating into your eyes so only the most detail is shown in the places where you are actually focused. The technique, called foveated rendering, will open the door to higher quality VR on lower cost hardware. It may be a key requirement of second generation VR headsets if the technology is ever to become mainstream.

Regarding the cost to add eye tracking hardware to a headset: http://uploadvr.com/smi-hands-on-250hz-eye-tracking/

“The total cost of all the hardware at scale is in the single digits,” says Villwock, “a few dollars.”

 

[ps3ud0]

My VR savior!

EDIT: Just need Leapmotions hand tracking and then some form of physical feedback and we are cooking

ps3ud0 8)

 

[majik13]

video from the article (old)
https://www.youtube.com/watch?v=Qq09BTmjzRs

 

[qcf x2]

The cost bit is the best news of all.

 

[Manoko]

I knew I did the right thing by deciding to wait for the second generation of VR HMD.

It's crazy how foveated rendering changes VR as a whole, by bypassing the extreme cost of entry and thus allowing much more graphically demanding games and applications available to the masses.

VR without foveated rendering just seems like a half-assed idea. Second generation will be head and shoulders above the first, with both foveated rendering and improved hand/body tracking.

 

[Muskieratboi]

Here's a link to the Actual SIGGRAPH Demo from Nvidia:

https://www.youtube.com/watch?v=L_I5Dxdbjqg

 

[Raticus79]

Updated OP with the links, thanks

 

[UncleSporky]

I knew I did the right thing by deciding to wait for the second generation of VR HMD.

It's crazy how foveated rendering changes VR as a whole, by bypassing the extreme cost of entry and thus allowing much more graphically demanding games and applications available to the masses.

VR without foveated rendering just seems like a half-assed idea. Second generation will be head and shoulders above the first, with both foveated rendering and improved hand/body tracking.

I'm happy to get in on the ground floor and then wait for third gen. Get to take part in the bleeding edge and presumably retain support through the next gen, and then I'm not hemming and hawing over whether the following upgrade is worth it. It will be, by that point, but perhaps not to a 2nd gen owner.

 

[Noobcraft]

Just curious, would well implemented foveated rendering actually lower the performance cost beneath running the game on a standard TV? It seems like a process that could focus system resources on only the most relevant areas could potentially achieve that.

 

[UncleSporky]

Just curious, would well implemented foveated rendering actually lower the performance cost beneath running the game on a standard TV? It seems like a process that could focus system resources on only the most relevant areas could potentially achieve that.

The TV would have to fill a lot more of your vision in order to be worth it. You'd have to be really close.

 

[Alexlf]

Just curious, would well implemented foveated rendering actually lower the performance cost beneath running the game on a standard TV? It seems like a process that could focus system resources on only the most relevant areas could potentially achieve that.

Kinda? If it had good eye tracking you could only render where the eye is looking, but I can't imagine it will really be worth it outside of VR, where the effects would be IMMENSELY more obvious and useful because of the FOV.

 

[ps3ud0]

Just curious, would well implemented foveated rendering actually lower the performance cost beneath running the game on a standard TV? It seems like a process that could focus system resources on only the most relevant areas could potentially achieve that.

I expect there will be some cross-over where people use a VR headset to play non-VR games because of the significant reduction in spec needed to render the game with foveated rendering opposed to playing on a monitor...

Eek out that power before your next GFX card purchase

ps3ud0 8)

 

[Noobcraft]

I expect there will be some cross-over where people use a VR headset to play non-VR games because of the significant reduction in spec needed to render the game with foveated rendering opposed to playing on a monitor...

Eek out that power before your next GFX card purchase

ps3ud0 8)

Yeah that's what I was thinking. It would be interesting to see games that actually perform better in VR than they do with standard displays because of foveated rendering.

 

[Nillansan]

Will Tom be there to demo it at SIGGRAPH?

 

[Raticus79]

I expect there will be some cross-over where people use a VR headset to play non-VR games because of the significant reduction in spec needed to render the game with foveated rendering opposed to playing on a monitor...

Eek out that power before your next GFX card purchase

ps3ud0 8)

Yeah, it should be possible to do that in a VR cinema mode, for example. I had asked around about that before and unfortunately it's not something that can just be added on to an existing engine via injection/etc, so the target game's engine needs to have some support for it.

Just curious, would well implemented foveated rendering actually lower the performance cost beneath running the game on a standard TV? It seems like a process that could focus system resources on only the most relevant areas could potentially achieve that.

I think camera resolution becomes the problem there. With a camera sitting next to the TV, as your distance to the panel increases, the camera needs higher resolution to achieve the same tracking accuracy it was achieving for the VR panel a few cm away. Add on to that the fact that your eyes can move relative to the TV (rather than just changing orientation) and that there can be multiple viewers and I'd put that way down the road.

 

[Randdalf]

Do they actually give any performance information for this i.e. is this foveated rendering technique actually cheaper, or that just taken for granted?

 

[Durante]

If it actually works imperceptibly and the cost part is true, then what we need now are 8k 200 Hz displays.

I knew I did the right thing by deciding to wait for the second generation of VR HMD.

I knew I did the right thing by getting a -1 generation, 0th generation and first generation VR HMD

Do they actually give any performance information for this i.e. is this foveated rendering technique actually cheaper, or that just taken for granted?

It doesn't even matter much if it's significantly cheaper than current VR rendering, what's important is that it should allow for 4k or higher physical resolution (so actually 6k+ per eye rendering resolution for good quality), which is extremely hard to achieve with conventional full frame rendering.

 

[Manoko]

I knew I did the right thing by getting a -1 generation, 0th generation and first generation VR HMD

I'll be honest, if I had the means, I would have been all over this VR train since the Oculus kickstarter campaign.

 

[UncleSporky]

Do they actually give any performance information for this i.e. is this foveated rendering technique actually cheaper, or that just taken for granted?

They assumed it would help but didn't take the time to see if it actually did, just moved full steam ahead with the project anyway?

 

[wurzle]

Just curious, would well implemented foveated rendering actually lower the performance cost beneath running the game on a standard TV? It seems like a process that could focus system resources on only the most relevant areas could potentially achieve that.

I suspect that there is some lower bound on acceptable framerate to avoid producing noticeable artifacts when your eyes move around the image. At 30fps there will be a pretty significant delay between eye movement and the corresponding screen update. Even 60fps sounds low for this.

 

[ps3ud0]

This is going to bug me but is this technique also useful for holographic 3d or is that just a function of ray tracing?

EDIT:

Yeah, it should be possible to do that in a VR cinema mode, for example. I had asked around about that before and unfortunately it's not something that can just be added on to an existing engine via injection/etc, so the target game's engine needs to have some support for it..

Thanks for the clarification - hopefully its something added to the popular engines so its becomes a stealth standard.

ps3ud0 8)

 

[Randdalf]

It doesn't even matter much if it's significantly cheaper than current VR rendering, what's important is that it should allow for 4k or higher physical resolution (so actually 6k+ per eye rendering resolution for good quality), which is extremely hard to achieve with conventional full frame rendering.

But isn't the ability to render at that resolution only possible because it's cheaper to render than anything else? Not that it's technically rendering at that resolution, that's the whole point of foveated rendering, but you know what I mean.

 

[Hayvic]

I hope you have Nvidia hardware cause this baby is gonna be propriatary!

Spoiler

As a 970 owner I hope I'm only joking

 

[Jinkies]

Yeah that's what I was thinking. It would be interesting to see games that actually perform better in VR than they do with standard displays because of foveated rendering.

I wonder how the same technique would work with existing eye-tracking solutions like TrackIR.

 

[collige]

If it actually works imperceptibly and the cost part is true, then what we need now are 8k 200 Hz displays.

I knew I did the right thing by getting a -1 generation, 0th generation and first generation VR HMD

It doesn't even matter much if it's significantly cheaper than current VR rendering, what's important is that it should allow for 4k or higher physical resolution (so actually 6k+ per eye rendering resolution for good quality), which is extremely hard to achieve with conventional full frame rendering.

Does this have bandwidth implications? I was under the impression that pushing 4-6k per eye at 90fps isn't feasible with any interface we have for the next few years. Do you imagine the headsets have special hardware for properly upscaling foveated images or will the data sent down the wire have to be at native resolution?

 

[Shao Kahn Brewing a Stew]

This would either help me improve my focus and attention, or fuck my OCD (and eyesight) to oblivion.

 

[DieH@rd]

AMD will also use foveated rendering as a part of Liquid VR SDK. There is already talk they are working on 4K 120Hz headset support.

 

[Aztechnology]

I truly believe and have been stating incessantly that foveated rendering is a big key to reaching a much much wider audience. When you tell people on the PC/console that they may be able to get performance at 4k for the cost of 1080p suddenly even people who don't care much for VR becoming interested even just as a display device with head tracking disabled for non VR based games.

Nvidia showed early tests of 2-3x performance, but others on unoptimized extremely early tests were showing upwards of 4x performance or greater. The potential scalability is huge for performance and might let us get the headsets we truly want but we're untenable because of performance limitations before.

https://youtu.be/GKR8tM28NnQ

 

[EmSeta]

This could have done wonders for the PSVR, right?

 

[Teknopathetic]

"Does this have bandwidth implications? I was under the impression that pushing 4-6k per eye at 90fps isn't feasible with any interface we have for the next few years. Do you imagine the headsets have special hardware for properly upscaling foveated images or will the data sent down the wire have to be at native resolution?"

IIRC, Displayport 1.4 can drive at least 4k at 120hz. Realistically, by time the headsets actually have 4k panels, there'll be graphics cards with interfaces that can support the bandwidth required.

 

[VindicatorZ]

Will this help with an increase in FOV in the future headsets? That's what I care about the most.

 

[Durante]

Does this have bandwidth implications? I was under the impression that pushing 4-6k per eye at 90fps isn't feasible with any interface we have for the next few years. Do you imagine the headsets have special hardware for properly upscaling foveated images or will the data sent down the wire have to be at native resolution?

That's a good question. For what is possible in terms of display technology in the short term the latest DisplayPort should be sufficient. But for that 8k 200 FPS future you'd clearly only want to send the foveated data over the bus, which would require some custom signaling standard.

I truly believe and have been stating incessantly that foveated rendering is a big key to reaching a much much wider audience. When you tell people on the PC/console that they may be able to get performance at 4k for the cost of 1080p suddenly even people who don't care much for VR becoming interested even just as a display device with head tracking disabled for non VR based games.

You probably don't want that. Even for a 2D/cinema use case you'd want a virtual screen which is tracked.

Will this help with an increase in FOV in the future headsets? That's what I care about the most.

Only in so far as it alleviates rendering requirements. For further increasing FoV optics, size and weight are also significant hurdles.

 

[Aztechnology]

Will this help with an increase in FOV in the future headsets? That's what I care about the most.

Definitely would, immensly. The performance cost would be minimal as it's particularly low on its rendering needs.

That's a good question. For what is possible in terms of display technology in the short term the latest DisplayPort should be sufficient. But for that 8k 200 FPS future you'd clearly only want to send the foveated data over the bus, which would require some custom signaling standard.

You probably don't want that. Even for a 2D/cinema use case you'd want a virtual screen which is tracked.

Only in so far as it alleviates rendering requirements. For further increasing FoV optics, size and weight are also significant hurdles.

I guess I meant more along the lines of the Cinema/Big Picture mode the PSVR touts. So yea a virtual screen.

 

[Shifty]

Just curious, would well implemented foveated rendering actually lower the performance cost beneath running the game on a standard TV? It seems like a process that could focus system resources on only the most relevant areas could potentially achieve that.

I suspect that there is some lower bound on acceptable framerate to avoid producing noticeable artifacts when your eyes move around the image. At 30fps there will be a pretty significant delay between eye movement and the corresponding screen update. Even 60fps sounds low for this.

Framerate aside there's the fact that unlike a VR HMD, TVs don't fill most of your field of view. Since foveated rendering is dependent on the size of the screen relative to your fovea (a.k.a. the sharp, high-detail area of your view), you have a lot less down-rezzable space to work with when sat on the other side of a room. Unless you're gaming on an IMAX screen, that is.

Not to mention that you'd need fast, precise eye tracking for it to work. No way current tech can do that from across a room with highly variable lighting conditions.

 

[Karak]

Loved testing out the quick version with the tobieyex and the Solus Project. It was almost impossible to tell it was happening and it wasn't a perfect implementation.

 

[Aztechnology]

This could have done wonders for the PSVR, right?

IMO the single most important thing, second becomes getting cost down but you need to implement that feature and higher res panels before you can start working to get the prices down because many like myself, feel the tech isn't there yet in the headsets.

 

[pj]

"Does this have bandwidth implications? I was under the impression that pushing 4-6k per eye at 90fps isn't feasible with any interface we have for the next few years. Do you imagine the headsets have special hardware for properly upscaling foveated images or will the data sent down the wire have to be at native resolution?"

IIRC, Displayport 1.4 can drive at least 4k at 120hz. Realistically, by time the headsets actually have 4k panels, there'll be graphics cards with interfaces that can support the bandwidth required.

I think it makes more sense to put some kind of image recombination hw in the headset. You could do 180hz video with the low res and high res areas sent as separate frames, or 90hz with packed frames. If 75% of the image is rendered sub-native, what's the point of upscaling it before transmission?

Keeping the bandwidth lower would also make it easier to do wireless.

 

[RoboPlato]

This is one of those rare rendering breakthroughs that will be a real gamechanger if it works as intended. Huge gains for VR if done right.

 

[Oxn]

Man I really want VR now, but this is really discouraging me from buying anything now given the pricepoint and tech involved.

 

[pj]

Man I really want VR now, but this is really discouraging me from buying anything now given the pricepoint and tech involved.

Second generation headsets are probably 2 years away. If you can wait that long, do it. Current VR is fun but has lots of limitations and not a tremendous amount of content.

I expect they will be 4k per eye + eye tracking. Even with foveated rendering it's going to be a while before there's a large enough market of capable computers.

 

[Maximo]

Good thing I'm poor this year and means I can wait for the second generation of VR.

 

[mrklaw]

Could this even be used for monitor gaming?

 

[Blanquito]

It's an interesting find that increasing the contrast outside the area of focus helps decrease how much a person notices the lower resolution. I wonder what other strange combinations will be found to help trick the mind when it comes to VR.

Would things like Foveated rendering be something standard across games on certain hardware (AMD/Nvidia, or even as part of PS4 dev environment), or would each game have to implement it on their own with their own settings as far as how big each circle is, how much lower the outer circles' resolution is, etc.?

 

[spekkeh]

Regarding the cost to add eye tracking hardware to a headset:

“The total cost of all the hardware at scale is in the single digits,” says Villwock, “a few dollars.”

http://uploadvr.com/smi-hands-on-250hz-eye-tracking/

Lol. We paid you 30,000 just two years ago, fuck you very much too, SMI.

 

[Karak]

Could this even be used for monitor gaming?

You can see it work in a rough implementation in Solus Project if you have an eye tracker.

 

[Fafalada]

If it actually works imperceptibly and the cost part is true, then what we need now are 8k 200 Hz displays.

I think there'll be a fair amount of work adopting the various rendering pipelines to this - not an unsolvable problem - but it will serve to further widen the growing gap between 2D and VR game codebases, and cost competition between them.

On the flip-side, I'd be happy to jump on-to such a display for productivity purposes and media watching alone.

 

[Business]

Could this even be used for monitor gaming?

Don't see why not. With two people watching is a problem but even that could be sorted in the future at the expense of some more resources (but still saving compared to full frame).

 

[Aztechnology]

Could this even be used for monitor gaming?

I mean sure. If you play like this guy.

I'm joking, but yea it would work, to a limited degree. Depends on a number of factors including monitor size, distance etc.

 

[Alexlf]

Lol. We paid you 30,000 just two years ago, fuck you very much too, SMI.

R&D is expensive. If they discover a way to do it for dollars then that's awesome.

 

[bj00rn_]

I mean sure. If you play like this guy.

I'm joking, but yea it would work, to a limited degree. Depends on a number of factors including monitor size, distance etc.

It's not a joke though. That is literally how you need to sit in front of the display for foveated rendering to be useful. Thus it doesn't work. With that afterthought in mind foveated rendering on a fixed display becomes an absurd idea for 99% of people out there with conventional configurations.