Sadly ShadowPC shat the bed and went bust and are now pivoting to enterprise.
I dont think VR will be a big 'anything' in gaming until VR headsets look like this -
You need ridiculously low latency for VR. It will NEVER work with a cloud service. The speed of light won't allow it. That's why VR is the true future of gaming and streaming will be a footnote.
Very nice info there. I could see in a future where everyone has no data caps, all consoles/PCs being able to be servers in the cloud where people could earn income for providing their systems so others close by can use them when not in use. Latency would be much lower in a setup like that I would think, if the consoles were pretty ubiquitous. It would take tons of external storage to keep a lot of games downloaded or really fast internet for that to work well, but seems like a cool idea.Booted my laptop just to reply.
I also thought this, it seemed obvious VR and streaming would be a terrible combination, and I figured people were full of shit when they said it worked well.
Then ended up having to eat crow after trying it for myself and being blown away. Weirdly enough, it turns out VR is actually really well suited for streaming.
So first off you have re-projection / space-warping, this technique effectively detaches the framerate of the game from the display of the headset, so when you turn your head it's always whatever fps (i.e. 90hz on Quest 2), this was developed so that when games have slowdown (i.e. loading screens, or just sections which tank the framerate) it doesn't affect the responsiveness of headset tracking.
As these techniques got smarter they even started to do stuff like mapping the rendered stereoscopic game image to a low-poly 3D mesh rendered natively in the headset, and increasing the rendered FoV to wider than the actual headset so you can even turn your head left and right slightly during a freeze and you won't see the "edge of" the virtual screen. Stuff like Virtual Desktop use this system for streaming in the VR game remotely, it's not just streaming directly to your screen, and it gives it a ton of wiggle room in terms or latency and inconsistent frame-timings.
This works incredibly, you may notice things "within" the game being affected by low frame-rates (or in the case of streaming, lost packets) but the actual feeling of being in VR is almost entirely unaffected by it unless you are turning your head 180 degrees rapidly. For games where the world is mostly static (i.e. Google Earth VR) then playing it streaming in VR, even with a shitty connection, works perfectly. For FPS titles where the world is constantly moving around you, other than the odd hiccup it's completely playable and on-par with a wired connection 98% of the time, but you can notice the world "stutter" when you have connection problems. But it feels like the world is stuttering, not your headset or your view of the world. And if you get a good router you can easily enjoy a 3-hour session without any problems at all.
The other factor to take into account is that VR screens have super fast response times and run at much higher refresh rates than your standard non-gaming monitor/TV, so they add very little in terms of their own latency. I've seen some expensive ass 4k HDTVs that add more latency than the streaming does, so your VR headset is problem the best screen you have in your house when it comes to streaming content.
TLDR; Streaming VR works really well.
Make no mistake, I am a full supporter of high end VR.And that’s right now. Give it a few years, and…
I think there’s a lot of people in this thread in a bit of denial about the future. Cloud gaming is an inevitability. It will make high end gaming cheaper, more accessible and ubiquitous. There isn’t a games company on earth that doesn’t want all of those things.
And high end VR already exists. The people in here banging on about it remaining a small niche haven’t spent eight hours playing Half Life Alyx. It’s transformative. And that’s a game that’ll seem basic and rudimentary compared to what will come in the next few years.
This whole conversation is going to be very different in five years.
I dont think VR will be a big 'anything' in gaming until VR headsets look like this -
If local PC offer 40ms latency, then the exact same processing on Streamintg would be 90m of traveling PLUS the 40ms latenmcy of the PC in the streaming node. Either way you have a PC doing the processing, that doesn't go away just because you are streaming.I can see future where most of processing happens on someone else's hardware.
If you think that your device could have 20ms latency from click to network call, then networking takes 16ms to server and then server takes 16ms to process next frame, 16ms back from server to your device and then 20ms to display that. It's about 90ms of latency, it's fine. Looks like this is quite close to reality based on GameStar measurements from February 2020 where local PC was about 40ms and streaming services (Stadia & Geforce Now) 70-100ms.
But I'm still not sold on VR becoming more prominent style of gaming, people don't have room to play in VR and sitting VR experiences are quite limited. Also VR makes latency requirements more strict.
However, I would not be surprised if Microsoft tried to convert old Xbox users to stream instead of buying new hardware in few years.
OP what have you done? Now you're going to have posts after you crying out: "It's impossible without going beyond the speed of light!", when ShadowPC has already proven that's not needed for it to work:
It will also work wirelessly with 5G. I'm pretty sure the engineers working on these technologies at companies like Qualcomm, Nvidia, Huawei, etc. are juuuust a bit more knowledgeable than your average forum user.
Yes, I was just trying to gauge how much more latency does streaming add and if that would be acceptable. Many 30 FPS console games have about 100ms end-to-end latency or more. So streaming doesn't feel like it's significantly worse than before in that sense.If local PC offer 40ms latency, then the exact same processing on Streamintg would be 90m of traveling PLUS the 40ms latenmcy of the PC in the streaming node. Either way you have a PC doing the processing, that doesn't go away just because you are streaming.
You can't do cloud computing for VR with a latency like that. With the bare minimum acceptable frame rate (60 fps), you still have 16 milliseconds just to render the frame. To that you have to add input latency, the whole pipeline for data management and rendering, then the output latency and screen latency. All this MUST stay at least below 20 milliseconds from input to screen output, to have a SUFFICIENT perception of correlation between your head movement and the movement of the virtual world.I’d be interested to see a detailed breakdown of latency between PC and Quest 2, via Air Link, and how that compares to potential latency over high speed internet.
I can’t say I ever have an issue with Air Link, and there‘s definitely some latency there. Most people report 40 - 50 ms from the brief search I’ve done.
Whereas the median latency on my fibre connection is 15ms.
What if end user device would not stream video, but graphics primitives from server? So server sends culled geometry data and matching (baked) texture data then client stores that locally and can render that even with relatively low power GPU which means user can move relatively freely with good framerate, but size and detail of scene would be restricted by what is being streamed in.You can't do cloud computing for VR with a latency like that. With the bare minimum acceptable frame rate (60 fps), you still have 16 milliseconds just to render the frame. To that you have to add input latency, the whole pipeline for data management and rendering, then the output latency and screen latency. All this MUST stay at least below 20 milliseconds from input to screen output, to have a SUFFICIENT perception of correlation between your head movement and the movement of the virtual world.
It would save some rendering time (and you have to compute it in some hardware in the headset itself like the quest), but that's it. All the other steps in the pipeline remain the same.What if end user device would not stream video, but graphics primitives from server? So server sends culled geometry data and matching (baked) texture data then client stores that locally and can render that even with relatively low power GPU which means user can move relatively freely with good framerate, but size and detail of scene would be restricted by what is being streamed in.
I think we’re talking next gen, personally. The streaming but is the issue. The VR bit is pretty much done now.
I also thought this, it seemed obvious VR and streaming would be a terrible combination, and I figured people were full of shit when they said it worked well.
Then ended up having to eat crow after trying it for myself and being blown away. Weirdly enough, it turns out VR is actually really well suited for streaming.
So first off you have re-projection / space-warping, this technique effectively detaches the framerate of the game from the display of the headset, so when you turn your head it's always whatever fps (i.e. 90hz on Quest 2), this was developed so that when games have slowdown (i.e. loading screens, or just sections which tank the framerate) it doesn't affect the responsiveness of headset tracking.
As these techniques got smarter they even started to do stuff like mapping the rendered stereoscopic game image to a low-poly 3D mesh rendered natively in the headset, and increasing the rendered FoV to wider than the actual headset so you can even turn your head left and right slightly during a freeze and you won't see the "edge of" the virtual screen. Stuff like Virtual Desktop use this system for streaming in the VR game remotely, it's not just streaming directly to your screen, and it gives it a ton of wiggle room in terms or latency and inconsistent frame-timings.
This works incredibly, you may notice things "within" the game being affected by low frame-rates (or in the case of streaming, lost packets) but the actual feeling of being in VR is almost entirely unaffected by it unless you are turning your head 180 degrees rapidly. For games where the world is mostly static (i.e. Google Earth VR) then playing it streaming in VR, even with a shitty connection, works perfectly. For FPS titles where the world is constantly moving around you, other than the odd hiccup it's completely playable and on-par with a wired connection 98% of the time, but you can notice the world "stutter" when you have connection problems. But it feels like the world is stuttering, not your headset or your view of the world. And if you get a good router you can easily enjoy a 3-hour session without any problems at all.
This is a working prototype by Panasonic, which already is very small, with hi-res dual oleds, 10 bit, HDR, 120 frames, 6DOF:One and done. There's no reason to doubt we won't get there eventually with the rate technology is evolving.
Are people really taking a pass on VR because the headsets aren't glasses? I really doubt it.
I think cost and content are still the biggest barriers. But also to a lesser extent the concept itself which is inherently physical. Games are entertainment but VR is an activity, it involves getting up and exercising which people don't always want to do.
I can see a time when VR and cloud gaming work well enough together. But I am not sure that barrier will be crossed before a lot of these problems are already solved offline.
Good post, and I agree. VR is gaming as 'an activity'.
Eyetoy Play...
Kinect...
Wii Sports...
VR.
I think the brightest future for VR isn't for everyone to have a headset in the home as a replacement to traditional gaming, but rather as an 'activity' to visit in laser quest style businesses. The kids party market, stag dos, interactive events, escape rooms - that type of thing.
This is a working prototype by Panasonic, which already is very small, with hi-res dual oleds, 10 bit, HDR, 120 frames, 6DOF:
CES: occhiali VR Panasonic HDR OLED 2,6k
Panasonic introduce i nuovi VR glasses 6DOF, dotati di display OLED HDR 120 Hz da 2245ppi e sistema audio ottimizzato Technics, utilizzabili con PC e smartphone 5Gwww.avmagazine.it
If you're working with stereoscopic imagery, then you really need both frames to be consistently reliable representations of the "world" at all times. If one eye gets the correct image, while a network transmission error causes the other eye to receive an improperly rendered frame (visible artifacting or other glitches in the image) or miss the frame altogether (falling back onto the previous frame), then this could cause serious problems for the user's perception.
We talk a lot round here about the future of video gaming, the path the industry will go down, and what advancements will drive it.
For me, the two advancements are VR and cloud gaming... but only when they mature and converge.
Right now, VR is imperfect for many reasons. You either have to own an expensive PC or console, as well as an expensive headset for high fidelity VR gaming. Or you own a standalone headset for lower quality gaming.
Cloud gaming is still in its infancy, and requires enough people to have fast, unlimited and stable internet for it to function properly.
So right now, neither thing is a threat to the status quo of home console or PC gaming.
BUT.
At some point, cloud gaming will reach the point of no return, where it starts to become far more widespread, popular, and ubiquitous (especially when 5G actually gets going in a meaningful way). Once enough people have the right internet connection across the world, gaming will be in the cloud.
At that time, a VR headset will be released that only streams games via the cloud. This will be smaller, cheaper and lighter than any other before it, because it won't require anything in it needed to power the actual game. It'll also be truly standalone, and able to play games at the highest level of fidelity (current best example being Half Life Alyx). Have a look at PlutoSphere for the first indicator of what will be coming down the pipe.
That moment is when the video game industry undergoes its biggest change for decades. To me, as someone who enjoys VR gaming now, it feels
I think the maturity of foveated rendering is the next big thing for VR.
If they push that technology, you can have better visuals in VR than with monitor/TVs because the headset can track your eyes and limit quality and resolution everywhere you're not directly looking. This doesn't work on a TV.
I think problem with foveated rendering is speed of eye tracking vs speed of eye. Saccades can move like 700 decrees per second. If human vision covers about 120 decrees then focus of our vision can move full range in about 1/(700/120) ~ 170 milliseconds.I think the maturity of foveated rendering is the next big thing for VR.
If they push that technology, you can have better visuals in VR than with monitor/TVs because the headset can track your eyes and limit quality and resolution everywhere you're not directly looking. This doesn't work on a TV.
Booted my laptop just to reply.
I also thought this, it seemed obvious VR and streaming would be a terrible combination, and I figured people were full of shit when they said it worked well.
Then ended up having to eat crow after trying it for myself and being blown away. Weirdly enough, it turns out VR is actually really well suited for streaming.
So first off you have re-projection / space-warping, this technique effectively detaches the framerate of the game from the display of the headset, so when you turn your head it's always whatever fps (i.e. 90hz on Quest 2), this was developed so that when games have slowdown (i.e. loading screens, or just sections which tank the framerate) it doesn't affect the responsiveness of headset tracking.
As these techniques got smarter they even started to do stuff like mapping the rendered stereoscopic game image to a low-poly 3D mesh rendered natively in the headset, and increasing the rendered FoV to wider than the actual headset so you can even turn your head left and right slightly during a freeze and you won't see the "edge of" the virtual screen. Stuff like Virtual Desktop use this system for streaming in the VR game remotely, it's not just streaming directly to your screen, and it gives it a ton of wiggle room in terms or latency and inconsistent frame-timings.
This works incredibly, you may notice things "within" the game being affected by low frame-rates (or in the case of streaming, lost packets) but the actual feeling of being in VR is almost entirely unaffected by it unless you are turning your head 180 degrees rapidly. For games where the world is mostly static (i.e. Google Earth VR) then playing it streaming in VR, even with a shitty connection, works perfectly. For FPS titles where the world is constantly moving around you, other than the odd hiccup it's completely playable and on-par with a wired connection 98% of the time, but you can notice the world "stutter" when you have connection problems. But it feels like the world is stuttering, not your headset or your view of the world. And if you get a good router you can easily enjoy a 3-hour session without any problems at all.
The other factor to take into account is that VR screens have super fast response times and run at much higher refresh rates than your standard non-gaming monitor/TV, so they add very little in terms of their own latency. I've seen some expensive ass 4k HDTVs that add more latency than the streaming does, so your VR headset is probably the best screen you have in your house when it comes to streaming content.
TLDR; Streaming VR works really well.
Vr is about vr. Streaming is already very good using airlink. You get lower latency streaming from your pc than a lot of local gaming on console. My buddy kings los Angela’s from Sacramento and under 15 ms with his sober line. So even from that distance is very workable. It would be under 10 ms from San Francisco.VR is about powerful local hardware. Streaming is about not having hardware.
By the time we could have VR streaming, we would have broken speed of light and invented time-travel as a side effect. At that point gaming would be unrecognizable. Streaming is not magic, it is just a computer that you rent somewhere in your neighborhood. Having the computer in your own house is what allows VR to work properly.
Are you talking about a streaming service like Stadia or streaming locally from your computer? Two totally different things.
Talking about using ShadowPC with Virtual Desktop (aka vrdesktop) to stream PC VR games from Steam/Oculus to my Oculus Quest 2 connected via Wifi, the ShadowPC itself is in Paris, so streaming from another country (albeit a close one).
If it's too uncomfortable and not immersive enough, why would you assume it won't solve those? That seems a given.VR isolate people (it's difficult, even dangerous, to play VR if you are a parent with kids in home, for example) and it causes sickness in a lot of users, me for example. In the eighties VR was my most beloved dream, but now, after several months of use I think it never will reach true mass market like TVs, it's simply too uncomfortable and not enough immersive to compensate its problems.
I have no doubts that the main barrier to VR is the size. It puts people off immensely, seeing the bulky box on your face, and it stops most people who use it from being able to wear it for hours.Are people really taking a pass on VR because the headsets aren't glasses? I really doubt it.
I think cost and content are still the biggest barriers. But also to a lesser extent the concept itself which is inherently physical. Games are entertainment but VR is an activity, it involves getting up and exercising which people don't always want to do.
I can see a time when VR and cloud gaming work well enough together. But I am not sure that barrier will be crossed before a lot of these problems are already solved offline.
You need 20 ms for head rotation and that’s done at all times using active time warp(atw) to interpolate frames. For basic controls with the controllers that isn’t true. People stream vr now at 40-50 ms and it works perfectly because the head rotation is still sub 20 ms.You need ridiculously low latency for VR. It will NEVER work with a cloud service. The speed of light won't allow it. That's why VR is the true future of gaming and streaming will be a footnote.
What happens is a sort of 'time freeze' effect, where you can still look around (to a degree) and see your controllers etc, but the world itself (enemies, audio, your weapon etc) just freeze temporarily. It's weirdly surreal when your connection dies completely and you are just stuck in this frozen world, but still able to look around to some degree.
Vr is about choice really. You can play in a very wide space like a tennis court at night using natural location.I have no doubts that the main barrier to VR is the size. It puts people off immensely, seeing the bulky box on your face, and it stops most people who use it from being able to wear it for hours.
The concept of VR is not inherently physical. It's however physical or laid back you want to make it. I play some VR games seated, motion controls or gamepad. I have weekly movie nights where I lay in bed with friends in a virtual theater. I even play some VR games in bed too.
I think problem with foveated rendering is speed of eye tracking vs speed of eye. Saccades can move like 700 decrees per second. If human vision covers about 120 decrees then focus of our vision can move full range in about 1/(700/120) ~ 170 milliseconds.
Would foveated rendering be able to keep up rendering sharpish images through eye movements and then really good picture once big movement is over? I also wonder what guides our eyes within those movements, would lower class of rendering hinder our eye movements since there is no detail to be seen before eye tracking sees eye movement and dispatches higher quality rendering?
Foveated rendering is interesting and I hope we get there. I have no idea about current capability of state of the art VR tech maybe they are quite capable already?
It's full steam ahead right now in terms of development. PSVR2 has in implemented and Facebook is dumping tons of money into it. So, it's just a matter of time.
There's already a PC VR headset coming out these days with 200HZ eye tracking for foveated rendering, and it also has hand tracking and a Lidar sensor (!) :
Varjo XR-4 Series
The world’s highest resolution mixed reality headsets for ultra-high fidelity enterprise VR, XR and Spatial Computing.varjo.com
For shame it's ~3000 dollars though
I can't wait to surf with a Beluga inside the VR.We talk a lot round here about the future of video gaming, the path the industry will go down, and what advancements will drive it.
For me, the two advancements are VR and cloud gaming... but only when they mature and converge.
Right now, VR is imperfect for many reasons. You either have to own an expensive PC or console, as well as an expensive headset for high fidelity VR gaming. Or you own a standalone headset for lower quality gaming.
Cloud gaming is still in its infancy, and requires enough people to have fast, unlimited and stable internet for it to function properly.
So right now, neither thing is a threat to the status quo of home console or PC gaming.
BUT.
At some point, cloud gaming will reach the point of no return, where it starts to become far more widespread, popular, and ubiquitous (especially when 5G actually gets going in a meaningful way). Once enough people have the right internet connection across the world, gaming will be in the cloud.
At that time, a VR headset will be released that only streams games via the cloud. This will be smaller, cheaper and lighter than any other before it, because it won't require anything in it needed to power the actual game. It'll also be truly standalone, and able to play games at the highest level of fidelity (current best example being Half Life Alyx). Have a look at PlutoSphere for the first indicator of what will be coming down the pipe.
That moment is when the video game industry undergoes its biggest change for decades. To me, as someone who enjoys VR gaming now, it feels
Did you not read my post just a few posts up?This will never be possible because for VR you need an almost instantaneous response time/input lag in terms of video rendering otherwise it will cause nausea and inner ear problems ...
Good games wouldn't exist in the first place if all developers had this mindset. We'd be stuck in the Atari era permanently.The next big leap for gaming will be when the industry stops promoting gaas, streaming, cloud and VR, just like they did with 3D, and return to making good games.