Support NeoGAF

[GuinGuin]

You need ridiculously low latency for VR. It will NEVER work with a cloud service. The speed of light won't allow it. That's why VR is the true future of gaming and streaming will be a footnote.

 

[reksveks]

Sadly ShadowPC shat the bed and went bust and are now pivoting to enterprise.

I do wonder what happened in a world where apple didn't force them to change their app behaviour

 

[Fahdis]

Don't worry about the naysayers OP. In the next 5 years, they will be doing exactly what some of these plebs are calling you a schizo for.

 

[jigglet]

I dont think VR will be a big 'anything' in gaming until VR headsets look like this -

One and done. There's no reason to doubt we won't get there eventually with the rate technology is evolving.

 

[McCheese]

You need ridiculously low latency for VR. It will NEVER work with a cloud service. The speed of light won't allow it. That's why VR is the true future of gaming and streaming will be a footnote.

Booted my laptop just to reply.

I also thought this, it seemed obvious VR and streaming would be a terrible combination, and I figured people were full of shit when they said it worked well.

Then ended up having to eat crow after trying it for myself and being blown away. Weirdly enough, it turns out VR is actually really well suited for streaming.

So first off you have re-projection / space-warping, this technique effectively detaches the framerate of the game from the display of the headset, so when you turn your head it's always whatever fps (i.e. 90hz on Quest 2), this was developed so that when games have slowdown (i.e. loading screens, or just sections which tank the framerate) it doesn't affect the responsiveness of headset tracking.

As these techniques got smarter they even started to do stuff like mapping the rendered stereoscopic game image to a low-poly 3D mesh rendered natively in the headset, and increasing the rendered FoV to wider than the actual headset so you can even turn your head left and right slightly during a freeze and you won't see the "edge of" the virtual screen. Stuff like Virtual Desktop use this system for streaming in the VR game remotely, it's not just streaming directly to your screen, and it gives it a ton of wiggle room in terms or latency and inconsistent frame-timings.

This works incredibly, you may notice things "within" the game being affected by low frame-rates (or in the case of streaming, lost packets) but the actual feeling of being in VR is almost entirely unaffected by it unless you are turning your head 180 degrees rapidly. For games where the world is mostly static (i.e. Google Earth VR) then playing it streaming in VR, even with a shitty connection, works perfectly. For FPS titles where the world is constantly moving around you, other than the odd hiccup it's completely playable and on-par with a wired connection 98% of the time, but you can notice the world "stutter" when you have connection problems. But it feels like the world is stuttering, not your headset or your view of the world. And if you get a good router you can easily enjoy a 3-hour session without any problems at all.

The other factor to take into account is that VR screens have super fast response times and run at much higher refresh rates than your standard non-gaming monitor/TV, so they add very little in terms of their own latency. I've seen some expensive ass 4k HDTVs that add more latency than the streaming does, so your VR headset is probably the best screen you have in your house when it comes to streaming content.

TLDR; Streaming VR works really well.

 

[clintar]

Booted my laptop just to reply.

I also thought this, it seemed obvious VR and streaming would be a terrible combination, and I figured people were full of shit when they said it worked well.

Then ended up having to eat crow after trying it for myself and being blown away. Weirdly enough, it turns out VR is actually really well suited for streaming.

So first off you have re-projection / space-warping, this technique effectively detaches the framerate of the game from the display of the headset, so when you turn your head it's always whatever fps (i.e. 90hz on Quest 2), this was developed so that when games have slowdown (i.e. loading screens, or just sections which tank the framerate) it doesn't affect the responsiveness of headset tracking.

As these techniques got smarter they even started to do stuff like mapping the rendered stereoscopic game image to a low-poly 3D mesh rendered natively in the headset, and increasing the rendered FoV to wider than the actual headset so you can even turn your head left and right slightly during a freeze and you won't see the "edge of" the virtual screen. Stuff like Virtual Desktop use this system for streaming in the VR game remotely, it's not just streaming directly to your screen, and it gives it a ton of wiggle room in terms or latency and inconsistent frame-timings.

This works incredibly, you may notice things "within" the game being affected by low frame-rates (or in the case of streaming, lost packets) but the actual feeling of being in VR is almost entirely unaffected by it unless you are turning your head 180 degrees rapidly. For games where the world is mostly static (i.e. Google Earth VR) then playing it streaming in VR, even with a shitty connection, works perfectly. For FPS titles where the world is constantly moving around you, other than the odd hiccup it's completely playable and on-par with a wired connection 98% of the time, but you can notice the world "stutter" when you have connection problems. But it feels like the world is stuttering, not your headset or your view of the world. And if you get a good router you can easily enjoy a 3-hour session without any problems at all.

The other factor to take into account is that VR screens have super fast response times and run at much higher refresh rates than your standard non-gaming monitor/TV, so they add very little in terms of their own latency. I've seen some expensive ass 4k HDTVs that add more latency than the streaming does, so your VR headset is problem the best screen you have in your house when it comes to streaming content.

TLDR; Streaming VR works really well.

Very nice info there. I could see in a future where everyone has no data caps, all consoles/PCs being able to be servers in the cloud where people could earn income for providing their systems so others close by can use them when not in use. Latency would be much lower in a setup like that I would think, if the consoles were pretty ubiquitous. It would take tons of external storage to keep a lot of games downloaded or really fast internet for that to work well, but seems like a cool idea.

 

[DeltaPolarBear]

And that’s right now. Give it a few years, and…

I think there’s a lot of people in this thread in a bit of denial about the future. Cloud gaming is an inevitability. It will make high end gaming cheaper, more accessible and ubiquitous. There isn’t a games company on earth that doesn’t want all of those things.

And high end VR already exists. The people in here banging on about it remaining a small niche haven’t spent eight hours playing Half Life Alyx. It’s transformative. And that’s a game that’ll seem basic and rudimentary compared to what will come in the next few years.

This whole conversation is going to be very different in five years.

Make no mistake, I am a full supporter of high end VR.

What i am disagreeing with is the very idea of using Streaming to run VR, which is the antithesis of the Virtual Reality experience. VR is about having low latency powerful hardware as close to yourself as possible. And it is doing just fine without being handcuffed to the impossible task of breaking the speed of light through streaming.

You know what? If you can solve VR streaming, you should sell the idea to NASA. NASA had been stuck with drones on Mars with a one way communication lag of 12 minutes from Earth. Go and earn billions of government contracts rather than trying to make Streamer money.

EDIT: And the post above me is still trying to argue that fake pretend graphics could paper over the latency. Look, you are going to need good hardware to run VR, there is no reason to have part of that hardware being in a different building across town.

 

[SF Kosmo]

I dont think VR will be a big 'anything' in gaming until VR headsets look like this -

Are people really taking a pass on VR because the headsets aren't glasses? I really doubt it.

I think cost and content are still the biggest barriers. But also to a lesser extent the concept itself which is inherently physical. Games are entertainment but VR is an activity, it involves getting up and exercising which people don't always want to do.

I can see a time when VR and cloud gaming work well enough together. But I am not sure that barrier will be crossed before a lot of these problems are already solved offline.

 

[Wonko_C]

OP what have you done? Now you're going to have posts after you crying out: "It's impossible without going beyond the speed of light!", when ShadowPC has already proven that's not needed for it to work:



It will also work wirelessly with 5G. I'm pretty sure the engineers working on these technologies at companies like Qualcomm, Nvidia, Huawei, etc. are juuuust a bit more knowledgeable than your average forum user.

 

[alabtrosMyster]

Cloud gaming is nothing but a delivery mechanism, a poor one at that, especially for VR where the level of interaction needs to be even more immediate.

That has to be the worst take I have seen in years.

 

[K2D]

Name on instance where cloud gaming (as opposed to general online and mmo) has brought anything of value, anything pivotal, anything game changing..

Next Gen VR will be great, but not because of the magical cloud..

 

[MetalRain]

I can see future where most of processing happens on someone else's hardware.

If you think that your device could have 20ms latency from click to network call, then networking takes 16ms to server and then server takes 16ms to process next frame, 16ms back from server to your device and then 20ms to display that. It's about 90ms of latency, it's fine. Looks like this is quite close to reality based on GameStar measurements from February 2020 where local PC was about 40ms and streaming services (Stadia & Geforce Now) 70-100ms.

But I'm still not sold on VR becoming more prominent style of gaming, people don't have room to play in VR and sitting VR experiences are quite limited. Also VR makes latency requirements more strict.

However, I would not be surprised if Microsoft tried to convert old Xbox users to stream instead of buying new hardware in few years.

 

[DeltaPolarBear]

I can see future where most of processing happens on someone else's hardware.

If you think that your device could have 20ms latency from click to network call, then networking takes 16ms to server and then server takes 16ms to process next frame, 16ms back from server to your device and then 20ms to display that. It's about 90ms of latency, it's fine. Looks like this is quite close to reality based on GameStar measurements from February 2020 where local PC was about 40ms and streaming services (Stadia & Geforce Now) 70-100ms.

But I'm still not sold on VR becoming more prominent style of gaming, people don't have room to play in VR and sitting VR experiences are quite limited. Also VR makes latency requirements more strict.

However, I would not be surprised if Microsoft tried to convert old Xbox users to stream instead of buying new hardware in few years.

If local PC offer 40ms latency, then the exact same processing on Streamintg would be 90m of traveling PLUS the 40ms latenmcy of the PC in the streaming node. Either way you have a PC doing the processing, that doesn't go away just because you are streaming.

 

[FunkMiller]

OP what have you done? Now you're going to have posts after you crying out: "It's impossible without going beyond the speed of light!", when ShadowPC has already proven that's not needed for it to work:



It will also work wirelessly with 5G. I'm pretty sure the engineers working on these technologies at companies like Qualcomm, Nvidia, Huawei, etc. are juuuust a bit more knowledgeable than your average forum user.

I do find it a bit weird that so many people are determined that the latency for cloud VR gaming will always be too high… when you can stream VR right now with acceptably low latency. Shadow is imperfect certainly, but it’s proof of concept, if nothing else.

And that’s only going to get better. Don’t see latency as an issue at all moving forward into the near future.

The issues are getting cloud infrastructure and internet speeds up to a level high enough across the board (a good few years away probably) and, as somebody pointed out, that VR is an active pursuit more than pancake gaming, and requires more space.

But then the Wii also required a lot of space, and that did pretty well for itself.

 

[MetalRain]

If local PC offer 40ms latency, then the exact same processing on Streamintg would be 90m of traveling PLUS the 40ms latenmcy of the PC in the streaming node. Either way you have a PC doing the processing, that doesn't go away just because you are streaming.

Yes, I was just trying to gauge how much more latency does streaming add and if that would be acceptable. Many 30 FPS console games have about 100ms end-to-end latency or more. So streaming doesn't feel like it's significantly worse than before in that sense.

But there is still way to go until we all are streaming instead of buying expensive hardware every other year. One thing that comes to my mind is separating game UI from video stream so that you can have compression artifact free UI, even at higher resolution than rest of the stream.

 

[ABnormal]

I’d be interested to see a detailed breakdown of latency between PC and Quest 2, via Air Link, and how that compares to potential latency over high speed internet.

I can’t say I ever have an issue with Air Link, and there‘s definitely some latency there. Most people report 40 - 50 ms from the brief search I’ve done.

Whereas the median latency on my fibre connection is 15ms.

You can't do cloud computing for VR with a latency like that. With the bare minimum acceptable frame rate (60 fps), you still have 16 milliseconds just to render the frame. To that you have to add input latency, the whole pipeline for data management and rendering, then the output latency and screen latency. All this MUST stay at least below 20 milliseconds from input to screen output, to have a SUFFICIENT perception of correlation between your head movement and the movement of the virtual world.
If you increase the frame rate you can gain up to 8 ms (for 120 frames), but still, having 15 ms of just server latency, it completely kills the chance to render VR via cloud.
You have to have a direct fiber to fiber connection, in order to make it feasible. Or wait for quantum entanglement connection.

 

[MetalRain]

You can't do cloud computing for VR with a latency like that. With the bare minimum acceptable frame rate (60 fps), you still have 16 milliseconds just to render the frame. To that you have to add input latency, the whole pipeline for data management and rendering, then the output latency and screen latency. All this MUST stay at least below 20 milliseconds from input to screen output, to have a SUFFICIENT perception of correlation between your head movement and the movement of the virtual world.

What if end user device would not stream video, but graphics primitives from server? So server sends culled geometry data and matching (baked) texture data then client stores that locally and can render that even with relatively low power GPU which means user can move relatively freely with good framerate, but size and detail of scene would be restricted by what is being streamed in.

 

[ABnormal]

What if end user device would not stream video, but graphics primitives from server? So server sends culled geometry data and matching (baked) texture data then client stores that locally and can render that even with relatively low power GPU which means user can move relatively freely with good framerate, but size and detail of scene would be restricted by what is being streamed in.

It would save some rendering time (and you have to compute it in some hardware in the headset itself like the quest), but that's it. All the other steps in the pipeline remain the same.

 

[Agent X]

I think we’re talking next gen, personally. The streaming but is the issue. The VR bit is pretty much done now.

From the perspective of "capability", both VR and streaming are "capable" right now. The largest obstacle is streaming, which isn't ubiquitous or reliable enough to be used for streaming VR.

I think that post #55 from McCheese did a good job of highlighting both the pros and cons of streaming through VR. It shows how it could work right now for certain users under excellent conditions, but at the same time it's far from ready for mass consumption.

I also thought this, it seemed obvious VR and streaming would be a terrible combination, and I figured people were full of shit when they said it worked well.

Then ended up having to eat crow after trying it for myself and being blown away. Weirdly enough, it turns out VR is actually really well suited for streaming.

Good to know. Thanks for sharing your experience!

So first off you have re-projection / space-warping, this technique effectively detaches the framerate of the game from the display of the headset, so when you turn your head it's always whatever fps (i.e. 90hz on Quest 2), this was developed so that when games have slowdown (i.e. loading screens, or just sections which tank the framerate) it doesn't affect the responsiveness of headset tracking.

As these techniques got smarter they even started to do stuff like mapping the rendered stereoscopic game image to a low-poly 3D mesh rendered natively in the headset, and increasing the rendered FoV to wider than the actual headset so you can even turn your head left and right slightly during a freeze and you won't see the "edge of" the virtual screen. Stuff like Virtual Desktop use this system for streaming in the VR game remotely, it's not just streaming directly to your screen, and it gives it a ton of wiggle room in terms or latency and inconsistent frame-timings.

This is a very cool technological application, and goes with what I said earlier in post #23. The cloud server wouldn't necessarily be sending a pure video stream into the headset (like current streaming video game services), but it would send some data about the game environment. You'd have a mixture of the headset rendering some portion of the image locally, and fetching a video stream for other parts that would be too taxing for your local hardware to handle by itself.

This works incredibly, you may notice things "within" the game being affected by low frame-rates (or in the case of streaming, lost packets) but the actual feeling of being in VR is almost entirely unaffected by it unless you are turning your head 180 degrees rapidly. For games where the world is mostly static (i.e. Google Earth VR) then playing it streaming in VR, even with a shitty connection, works perfectly. For FPS titles where the world is constantly moving around you, other than the odd hiccup it's completely playable and on-par with a wired connection 98% of the time, but you can notice the world "stutter" when you have connection problems. But it feels like the world is stuttering, not your headset or your view of the world. And if you get a good router you can easily enjoy a 3-hour session without any problems at all.

Unfortunately, having the world "stutter" even momentarily could be off-putting for VR applications. Dropped frames (either from frame rate degradation within the game, or connection issues for streaming) are tolerable to an extent with traditional "flat screen" gaming, because most games treat the player as a mere observer of the on-screen action. By comparison, VR is meant to be an immersive experience, where the player feels he is situated in the game world. These types of inconsistencies could wreck the sense of immersion, providing a substandard experience.

If you're working with stereoscopic imagery, then you really need both frames to be consistently reliable representations of the "world" at all times. If one eye gets the correct image, while a network transmission error causes the other eye to receive an improperly rendered frame (visible artifacting or other glitches in the image) or miss the frame altogether (falling back onto the previous frame), then this could cause serious problems for the user's perception.

I feel you've done a good job of describing the potential for the combination of VR and streaming, but I still believe that it isn't viable enough to work on a large scale for today's mass market consumers.

 

[ABnormal]

One and done. There's no reason to doubt we won't get there eventually with the rate technology is evolving.

This is a working prototype by Panasonic, which already is very small, with hi-res dual oleds, 10 bit, HDR, 120 frames, 6DOF:

CES: occhiali VR Panasonic HDR OLED 2,6k

Panasonic introduce i nuovi VR glasses 6DOF, dotati di display OLED HDR 120 Hz da 2245ppi e sistema audio ottimizzato Technics, utilizzabili con PC e smartphone 5G

www.avmagazine.it

 

[Matt_Fox]

Are people really taking a pass on VR because the headsets aren't glasses? I really doubt it.

I think cost and content are still the biggest barriers. But also to a lesser extent the concept itself which is inherently physical. Games are entertainment but VR is an activity, it involves getting up and exercising which people don't always want to do.

I can see a time when VR and cloud gaming work well enough together. But I am not sure that barrier will be crossed before a lot of these problems are already solved offline.

Good post, and I agree. VR is gaming as 'an activity'.

Eyetoy Play...

Kinect...

Wii Sports...

VR.

I think the brightest future for VR isn't for everyone to have a headset in the home as a replacement to traditional gaming, but rather as an 'activity' to visit in laser quest style businesses. The kids party market, stag dos, interactive events, escape rooms - that type of thing.

 

[nemiroff]

When I look at the state of 2D game streaming today I'd say it'll take another 10-20 years, if ever for VR streaming to come to a place where I'm going to be able to stomach the lower image quality and higher latency (not with today's technology).

Also, even as a VR enthusiast for decades I have to say that even though form factor has slightly improved through the last few years, smaller and lighter, the usability of today's VR headsets are still completely idiotic. It'll take time - But at least we're on the way, and on the right track.

 

[Danknugz]

The way I see it, the major hurdle to Vr adoption is motion sickness. Latency, fidelity and software quality all can be solved with time and in the right hands. A solution for motion sickness (outside of just getting used to it) has yet to be found. Without it, a lot of casual players will be turned off by this factor.

 

[FunkMiller]

Good post, and I agree. VR is gaming as 'an activity'.

Eyetoy Play...

Kinect...

Wii Sports...

VR.

I think the brightest future for VR isn't for everyone to have a headset in the home as a replacement to traditional gaming, but rather as an 'activity' to visit in laser quest style businesses. The kids party market, stag dos, interactive events, escape rooms - that type of thing.

How much have you used VR? Because while some games are very much activity games (Beat Saber, Pistol Whip, Superhot etc.) there are an awful lot that don't require anything like as much movement or activity. In fact, there are plenty that you can sit down and play quite happily. I went through Resident Evil 7 sat on my arse, in VR, and it was one of my absolute favourite gaming experiences, simply due to the heightened level of immersion.

And even if you are talking about VR purely as an activity type gaming platform, the Wii sold 100 million units. There is a potentially strong and very large VR market out there... once headsets become small and cheap enough (conceivably well under 200 hundred dollars for a headset that can play high end VR) - hence why I think the convergence with streaming technology is important.

 

[FunkMiller]

This is a working prototype by Panasonic, which already is very small, with hi-res dual oleds, 10 bit, HDR, 120 frames, 6DOF:

CES: occhiali VR Panasonic HDR OLED 2,6k

Panasonic introduce i nuovi VR glasses 6DOF, dotati di display OLED HDR 120 Hz da 2245ppi e sistema audio ottimizzato Technics, utilizzabili con PC e smartphone 5G

www.avmagazine.it

Yep. That's the kind of way VR headsets will go if they only stream games, instead of running them locally. Much more streamlined, lighter, better looking and easier to use. The big stupid bricks we've got on our faces at the moment are going to look like a Virtual Boy by comparison.

 

[Romulus]

I think the maturity of foveated rendering is the next big thing for VR.
If they push that technology, you can have better visuals in VR than with monitor/TVs because the headset can track your eyes and limit quality and resolution everywhere you're not directly looking. This doesn't work on a TV.

 

[McCheese]

If you're working with stereoscopic imagery, then you really need both frames to be consistently reliable representations of the "world" at all times. If one eye gets the correct image, while a network transmission error causes the other eye to receive an improperly rendered frame (visible artifacting or other glitches in the image) or miss the frame altogether (falling back onto the previous frame), then this could cause serious problems for the user's perception.

What happens is a sort of 'time freeze' effect, where you can still look around (to a degree) and see your controllers etc, but the world itself (enemies, audio, your weapon etc) just freeze temporarily. It's weirdly surreal when your connection dies completely and you are just stuck in this frozen world, but still able to look around to some degree.

 

[TheThreadsThatBindUs]

We talk a lot round here about the future of video gaming, the path the industry will go down, and what advancements will drive it.

For me, the two advancements are VR and cloud gaming... but only when they mature and converge.

Right now, VR is imperfect for many reasons. You either have to own an expensive PC or console, as well as an expensive headset for high fidelity VR gaming. Or you own a standalone headset for lower quality gaming.

Yes. Also, the latency issue is important. Wireless headsets are ideal, but wireless technologies introduce too much latency, meaning the processing hardware has to be mounted in the headset, leading to poorer quality hardware and thus VR gaming experience.

We need more powerful mobile tech, alongside new innovations in solid-state battery technology to facilitate the requisite dramatic advances in the former.

The biggest issue in VR gaming is the locomotion problem, which seems at least at the moment an unsolvable problem. VR will then only ever be constrained to producing alternative gaming experiences to regular gaming. It can never replace regular gaming.

Cloud gaming is still in its infancy, and requires enough people to have fast, unlimited and stable internet for it to function properly.

It requires much more than that to be successful. It needs features that separate it from home console/PC gaming. It need to be able to deliver superior experiences for a lower cost of entry.

It needs to solve the latency problems. It also needs to solve the business model problem..... this is the biggest challenge for cloud gaming.

So right now, neither thing is a threat to the status quo of home console or PC gaming.

BUT.

At some point, cloud gaming will reach the point of no return, where it starts to become far more widespread, popular, and ubiquitous (especially when 5G actually gets going in a meaningful way). Once enough people have the right internet connection across the world, gaming will be in the cloud.

Cloud gaming needs to provide any reasonable appeal at all to the broader gaming market. It's not just about consumer internet connections and 5G won't solve shit when mobile bandwidth throttling is even more severe than stationary and data caps continue to remain a thing.

Cloud also needs to solve all the issues I already mentioned above, chief of which is the business model issue, which is critical.

If it can't do more than stationary gaming at a lower price, it's never going to grow to reach mass-market scale.

At that time, a VR headset will be released that only streams games via the cloud. This will be smaller, cheaper and lighter than any other before it, because it won't require anything in it needed to power the actual game. It'll also be truly standalone, and able to play games at the highest level of fidelity (current best example being Half Life Alyx). Have a look at PlutoSphere for the first indicator of what will be coming down the pipe.

Internet latency and general unreliability make this entire proposition dead in the water.

VR combined with the cloud sounds novel in theory, but in practice is worse than both stationary non-VR gaming, as well as stationary non-cloud VR gaming.

How does the worst of both worlds somehow become the future of gaming?

An untethered, cloud-based VR headset will have unconscionably bad latency and will be super unreliable. Even if you figure out a way to somehow overcome the laws of physics to solve the latency issue (with say some neural network-based predictive algorithm), the infinite variability in consumer's home internet networks and router quality will categorically ruin the experience.

What you're suggesting is a pipe dream at best.

That moment is when the video game industry undergoes its biggest change for decades. To me, as someone who enjoys VR gaming now, it feels

Nah.

 

[FunkMiller]

I think the maturity of foveated rendering is the next big thing for VR.
If they push that technology, you can have better visuals in VR than with monitor/TVs because the headset can track your eyes and limit quality and resolution everywhere you're not directly looking. This doesn't work on a TV.

Really good point. And it'd probably help solve a lot of the issues with streaming as well. The less data and power needed, the better.

 

[MetalRain]

I think the maturity of foveated rendering is the next big thing for VR.
If they push that technology, you can have better visuals in VR than with monitor/TVs because the headset can track your eyes and limit quality and resolution everywhere you're not directly looking. This doesn't work on a TV.

I think problem with foveated rendering is speed of eye tracking vs speed of eye. Saccades can move like 700 decrees per second. If human vision covers about 120 decrees then focus of our vision can move full range in about 1/(700/120) ~ 170 milliseconds.

Would foveated rendering be able to keep up rendering sharpish images through eye movements and then really good picture once big movement is over? I also wonder what guides our eyes within those movements, would lower class of rendering hinder our eye movements since there is no detail to be seen before eye tracking sees eye movement and dispatches higher quality rendering?

Foveated rendering is interesting and I hope we get there. I have no idea about current capability of state of the art VR tech maybe they are quite capable already?

 

[GuinGuin]

Booted my laptop just to reply.

I also thought this, it seemed obvious VR and streaming would be a terrible combination, and I figured people were full of shit when they said it worked well.

Then ended up having to eat crow after trying it for myself and being blown away. Weirdly enough, it turns out VR is actually really well suited for streaming.

So first off you have re-projection / space-warping, this technique effectively detaches the framerate of the game from the display of the headset, so when you turn your head it's always whatever fps (i.e. 90hz on Quest 2), this was developed so that when games have slowdown (i.e. loading screens, or just sections which tank the framerate) it doesn't affect the responsiveness of headset tracking.

As these techniques got smarter they even started to do stuff like mapping the rendered stereoscopic game image to a low-poly 3D mesh rendered natively in the headset, and increasing the rendered FoV to wider than the actual headset so you can even turn your head left and right slightly during a freeze and you won't see the "edge of" the virtual screen. Stuff like Virtual Desktop use this system for streaming in the VR game remotely, it's not just streaming directly to your screen, and it gives it a ton of wiggle room in terms or latency and inconsistent frame-timings.

This works incredibly, you may notice things "within" the game being affected by low frame-rates (or in the case of streaming, lost packets) but the actual feeling of being in VR is almost entirely unaffected by it unless you are turning your head 180 degrees rapidly. For games where the world is mostly static (i.e. Google Earth VR) then playing it streaming in VR, even with a shitty connection, works perfectly. For FPS titles where the world is constantly moving around you, other than the odd hiccup it's completely playable and on-par with a wired connection 98% of the time, but you can notice the world "stutter" when you have connection problems. But it feels like the world is stuttering, not your headset or your view of the world. And if you get a good router you can easily enjoy a 3-hour session without any problems at all.

The other factor to take into account is that VR screens have super fast response times and run at much higher refresh rates than your standard non-gaming monitor/TV, so they add very little in terms of their own latency. I've seen some expensive ass 4k HDTVs that add more latency than the streaming does, so your VR headset is probably the best screen you have in your house when it comes to streaming content.

TLDR; Streaming VR works really well.

Are you talking about a streaming service like Stadia or streaming locally from your computer? Two totally different things.

 

[Robb]

Maybe, we'll see. I personally don't belive in VR being the next 'big thing'. Not that VR isn't neat, but I don't think it'll be a revolution for gaming specifically.

I do think it'll grow and have a lot of different applications in other areas though.

 

[Tygeezy]

VR is about powerful local hardware. Streaming is about not having hardware.

By the time we could have VR streaming, we would have broken speed of light and invented time-travel as a side effect. At that point gaming would be unrecognizable. Streaming is not magic, it is just a computer that you rent somewhere in your neighborhood. Having the computer in your own house is what allows VR to work properly.

Vr is about vr. Streaming is already very good using airlink. You get lower latency streaming from your pc than a lot of local gaming on console. My buddy kings los Angela’s from Sacramento and under 15 ms with his sober line. So even from that distance is very workable. It would be under 10 ms from San Francisco.

 

[McCheese]

Are you talking about a streaming service like Stadia or streaming locally from your computer? Two totally different things.

Talking about using ShadowPC with Virtual Desktop (aka vrdesktop) to stream PC VR games from Steam/Oculus to my Oculus Quest 2 connected via Wifi, the ShadowPC itself is in Paris, so streaming from another country (albeit a close one).

 

[GuinGuin]

Talking about using ShadowPC with Virtual Desktop (aka vrdesktop) to stream PC VR games from Steam/Oculus to my Oculus Quest 2 connected via Wifi, the ShadowPC itself is in Paris, so streaming from another country (albeit a close one).

Yeah then I don't believe you that it is even a halfway decent experience.

 

[DarthBuzzer]

VR isolate people (it's difficult, even dangerous, to play VR if you are a parent with kids in home, for example) and it causes sickness in a lot of users, me for example. In the eighties VR was my most beloved dream, but now, after several months of use I think it never will reach true mass market like TVs, it's simply too uncomfortable and not enough immersive to compensate its problems.

If it's too uncomfortable and not immersive enough, why would you assume it won't solve those? That seems a given.

Isolation is something that I can understand being impossible to solve, but it actually is solvable as well - as VR and AR merge.

 

[DarthBuzzer]

Are people really taking a pass on VR because the headsets aren't glasses? I really doubt it.

I think cost and content are still the biggest barriers. But also to a lesser extent the concept itself which is inherently physical. Games are entertainment but VR is an activity, it involves getting up and exercising which people don't always want to do.

I can see a time when VR and cloud gaming work well enough together. But I am not sure that barrier will be crossed before a lot of these problems are already solved offline.

I have no doubts that the main barrier to VR is the size. It puts people off immensely, seeing the bulky box on your face, and it stops most people who use it from being able to wear it for hours.

The concept of VR is not inherently physical. It's however physical or laid back you want to make it. I play some VR games seated, motion controls or gamepad. I have weekly movie nights where I lay in bed with friends in a virtual theater. I even play some VR games in bed too.

 

[Tygeezy]

You need ridiculously low latency for VR. It will NEVER work with a cloud service. The speed of light won't allow it. That's why VR is the true future of gaming and streaming will be a footnote.

You need 20 ms for head rotation and that’s done at all times using active time warp(atw) to interpolate frames. For basic controls with the controllers that isn’t true. People stream vr now at 40-50 ms and it works perfectly because the head rotation is still sub 20 ms.

There is no doubt in my mind that you will be able to cloud stream vr lower than 50 ms (which still beats most local console games by the way) in the very near future.

 

[Agent X]

What happens is a sort of 'time freeze' effect, where you can still look around (to a degree) and see your controllers etc, but the world itself (enemies, audio, your weapon etc) just freeze temporarily. It's weirdly surreal when your connection dies completely and you are just stuck in this frozen world, but still able to look around to some degree.

That's interesting. That appears to be a good way to deal with network data hiccups or interruptions, as opposed to sending mismatched images to each eye.

 

[Tygeezy]

I have no doubts that the main barrier to VR is the size. It puts people off immensely, seeing the bulky box on your face, and it stops most people who use it from being able to wear it for hours.

The concept of VR is not inherently physical. It's however physical or laid back you want to make it. I play some VR games seated, motion controls or gamepad. I have weekly movie nights where I lay in bed with friends in a virtual theater. I even play some VR games in bed too.

Vr is about choice really. You can play in a very wide space like a tennis court at night using natural location.



Or you can play seated in a swivel chair, or whatever you like. There are an awful lot of people with an agenda against vr and streaming that have never tried it.

 

[Romulus]

I think problem with foveated rendering is speed of eye tracking vs speed of eye. Saccades can move like 700 decrees per second. If human vision covers about 120 decrees then focus of our vision can move full range in about 1/(700/120) ~ 170 milliseconds.

Would foveated rendering be able to keep up rendering sharpish images through eye movements and then really good picture once big movement is over? I also wonder what guides our eyes within those movements, would lower class of rendering hinder our eye movements since there is no detail to be seen before eye tracking sees eye movement and dispatches higher quality rendering?

Foveated rendering is interesting and I hope we get there. I have no idea about current capability of state of the art VR tech maybe they are quite capable already?

It's full steam ahead right now in terms of development. PSVR2 has in implemented and Facebook is dumping tons of money into it. So, it's just a matter of time.

It's a combination of tricks to get around the speed of the eye to constantly project image cones that the user cannot discern the changes in quality and/or resolution.

I think what we will see in the short term these initial rendering saving tech that saves around 20-40% resolution loads but are "safe" in that the image gives the image construction plenty of leeway so that the quality is not disrupted.

It's only a matter of time. In a mere 3 years, huge strides have happened and the human eye is a static baseline, it cannot get faster, but technology will. I would say in less than 8 years we'll be rendering 1/10 the resolution load that TVs/monitors need. I think1/20 is possible considering how narrow the FOV of the pupil can see the highest resolutions.

I think it's the holy grail for VR development, and there's already a solid baseline to improve on.

 

[nemiroff]

It's full steam ahead right now in terms of development. PSVR2 has in implemented and Facebook is dumping tons of money into it. So, it's just a matter of time.

There's already a PC VR headset coming out these days with 200HZ eye tracking for foveated rendering, and it also has hand tracking and a Lidar sensor (!) :

Varjo XR-4 Series

The world’s highest resolution mixed reality headsets for ultra-high fidelity enterprise VR, XR and Spatial Computing.

varjo.com

For shame it's ~3000 dollars though

 

[Romulus]

There's already a PC VR headset coming out these days with 200HZ eye tracking for foveated rendering, and it also has hand tracking and a Lidar sensor (!) :

Varjo XR-4 Series

The world’s highest resolution mixed reality headsets for ultra-high fidelity enterprise VR, XR and Spatial Computing.

varjo.com

For shame it's ~3000 dollars though

Yup, just a matter of time before that tech is more mainstream. I wouldn't be shocked if one of the next few Quest iterations has something so robust that you can run mid-tier PC games without a PC because of the rendering-saving technologies. Seems to me that's a big part of their goal.

 

[Kamina]

The next big leap for gaming will be when the industry stops promoting gaas, streaming, cloud and VR, just like they did with 3D, and return to making good games.

 

[WitchHunter]

We talk a lot round here about the future of video gaming, the path the industry will go down, and what advancements will drive it.

For me, the two advancements are VR and cloud gaming... but only when they mature and converge.

Right now, VR is imperfect for many reasons. You either have to own an expensive PC or console, as well as an expensive headset for high fidelity VR gaming. Or you own a standalone headset for lower quality gaming.

Cloud gaming is still in its infancy, and requires enough people to have fast, unlimited and stable internet for it to function properly.

So right now, neither thing is a threat to the status quo of home console or PC gaming.

BUT.

At some point, cloud gaming will reach the point of no return, where it starts to become far more widespread, popular, and ubiquitous (especially when 5G actually gets going in a meaningful way). Once enough people have the right internet connection across the world, gaming will be in the cloud.

At that time, a VR headset will be released that only streams games via the cloud. This will be smaller, cheaper and lighter than any other before it, because it won't require anything in it needed to power the actual game. It'll also be truly standalone, and able to play games at the highest level of fidelity (current best example being Half Life Alyx). Have a look at PlutoSphere for the first indicator of what will be coming down the pipe.

That moment is when the video game industry undergoes its biggest change for decades. To me, as someone who enjoys VR gaming now, it feels

I can't wait to surf with a Beluga inside the VR.

 

[FearlessBelgian]

This will never be possible because for VR you need an almost instantaneous response time/input lag in terms of video rendering otherwise it will cause nausea and inner ear problems ...

 

[Tygeezy]

This will never be possible because for VR you need an almost instantaneous response time/input lag in terms of video rendering otherwise it will cause nausea and inner ear problems ...

Did you not read my post just a few posts up?

edit:


You need 20 ms for head rotation and that’s done at all times using active time warp(atw) to interpolate frames. For basic controls with the controllers that isn’t true. People stream vr now at 40-50 ms and it works perfectly because the head rotation is still sub 20 ms.

There is no doubt in my mind that you will be able to cloud stream vr lower than 50 ms (which still beats most local console games by the way) in the very near future.

 

[DarthBuzzer]

The next big leap for gaming will be when the industry stops promoting gaas, streaming, cloud and VR, just like they did with 3D, and return to making good games.

Good games wouldn't exist in the first place if all developers had this mindset. We'd be stuck in the Atari era permanently.

We already have plenty of good games releasing each year, the 2021 delays being an exception due to exceptional circumstances.

Let's have good games and advancements in these areas.

 

[junguler]

no thank you, i'll stay with the traditional games and the normal means to play them forever.

 

[Larogue]

Do you realize how much bandwidth needed to drive an HDMI 2.1 output ?

48Gbps

No 5G or even home fiber can feed such demands.

And it always gets doubled every 2-3 years with higher resolution and refresh rates standards.