Support NeoGAF

[IbizaPocholo]

https://videocardz.com/newz/nvidia-details-dlss-5-three-models-per-object-controls-and-single-gpu-support

NVIDIA has revealed that DLSS 5 will give developers fine-grained control over AI-enhanced rendering through multiple AI models, per-object customization, and adjustable visual intensity, aiming to preserve artistic intent while improving image quality on a single GPU.

Summary

• NVIDIA provided the first detailed look at DLSS 5 during SIGGRAPH, explaining how developers will be able to control the technology.
• DLSS 5 is designed to address criticism that AI-generated rendering could alter a game's intended artistic style by giving developers extensive customization options.
• The technology introduces a generative rendering stage after the game engine has produced a conventionally rendered frame.
• The original game renderer continues to determine:
  • Geometry.
  • Camera positioning.
  • Lighting setup.
  • Scene composition.
• DLSS 5 then enhances the rendered frame by improving:
  • Materials.
  • Shadows.
  • Reflections.
  • Surface detail.
  • Other visual effects.
• NVIDIA says DLSS 5 is intended to complement, not replace:
  • Rasterized rendering.
  • Ray tracing.
  • Path tracing.
• The company describes generative rendering as a third neural rendering category, alongside reconstruction and function approximation.

Three major technical challenges

• NVIDIA outlined three primary engineering challenges addressed by DLSS 5:
  1. Preserving developers' artistic intent.
  2. Maintaining temporal stability.
  3. Delivering acceptable performance.

Preserving artistic intent

• AI image generation can unintentionally change important visual details such as:
  • Facial features.
  • Scars.
  • Clothing details.
  • Character identity.
• To prevent this, DLSS 5 relies on:
  • The original rendered frame.
  • Albedo buffers.
  • Surface normals.
  • Lighting data.
  • Other engine-generated information.
• These inputs help keep AI-generated enhancements faithful to the original artwork.

Temporal stability

• Unlike traditional video-generation AI models that process multiple future frames, games require immediate responses to player input.
• DLSS 5 operates causally, processing one frame at a time without future-frame information.
• Motion vectors and additional engine data are used to reduce:
  • Shimmering.
  • Image drifting.
  • Swimming artifacts.
  • Frame-to-frame instability.

Performance

• NVIDIA says it compressed knowledge from larger generative AI models into a compact one-step pixel-space diffusion transformer.
• The model is optimized specifically for enhancing rendered game frames rather than generating entirely new images.
• NVIDIA also indicated DLSS 5 is designed to operate on a single GPU with efficient VRAM usage.
• The company has not yet confirmed:
  • Hardware compatibility across the entire RTX 50-series lineup.
  • Whether every DLSS 5 model will run on all supported GPUs.

Three selectable AI models

• DLSS 5 includes three separate AI models, demonstrated as:
  • Model A.
  • Model B.
  • Model C.
• Each model produces a different visual interpretation of the same rendered scene.
• Developers are not restricted to a single model throughout a game.
• Different models can be assigned to:
  • Individual scenes.
  • Environments.
  • Cutscenes.
  • Other specific portions of a game.
• Models can be mixed throughout gameplay to achieve different artistic goals.

Adjustable intensity controls

• Developers receive two independent adjustment sliders:
  • Structure Intensity
  • Tone Intensity
• Structure Intensity primarily affects high-frequency visual detail, including:
  • Ambient occlusion.
  • Reflections.
  • Subsurface scattering.
  • Fine material detail.
• Tone Intensity controls broader visual characteristics, including:
  • Lighting.
  • Color grading.
  • Overall scene appearance.

Per-object customization

• One of DLSS 5's most significant additions is per-object control.
• The system supports automatic AI-generated masks that can identify characters independently from the rest of the environment.
• Developers can:
  • Apply DLSS 5 only to selected characters.
  • Exclude characters entirely.
  • Use different enhancement strengths for different elements.
• Additional masks can also be created manually within the game engine.
• NVIDIA demonstrated separate controls for objects such as:
  • Bottles.
  • Grapes.
  • Pitchers.
  • Other environmental props.
• Every masked object can have its own:
  • Structure Intensity.
  • Tone Intensity.
• This enables developers to selectively strengthen, weaken, or completely disable DLSS 5 processing on individual objects.

Release plans

• NVIDIA says DLSS 5 will continue evolving based on feedback from:
  • Developers.
  • Artists.
• The technology is scheduled to launch this fall.
• The presentation focused on providing developers with creative control to ensure AI enhancements improve image quality without overriding the original artistic vision.

 

[jufonuk]

Switch 2 to have Arkham knight working well then?

 

[Trilobit]

It will feel really weird to throw a realistic human to the alligators in RDR3.

 

[Gonzito]

That's cool

 

[nemiroff]

That's pretty cool.

Developer options are always a good thing as they preserve the artist's intent (developer end=better).

(But with that said. even if as an additional end user overlay, who the f cares what people do on their 'puters).

 

[Gameplay Gods Bless]

https://videocardz.com/newz/nvidia-details-dlss-5-three-models-per-object-controls-and-single-gpu-support

So hyped for this. Hopefully it runs alright on a 5080 at 1080p, I realize this tech may be very demanding despite Nvidias relentless attempts at optimization. Also the 60 series will likely greatly excel at DLSS5, can't wait to see what 6080 and 6090 can do with it.

 

[Buggy Loop]

I want to see the benchmarks and what GPU it'll need as they optimize for release, but it looks really good.

 

[dgrdsv]

Switch 2 to have Arkham knight working well then?

Nintendo decided to use 2020 h/w for this one so no.

 

[//DEVIL//]

I want to see the benchmarks and what GPU it'll need as they optimize for release, but it looks really good.

Them not talking about GPU requirements makes me think only the 5090 will run this. Maybe the 5080 and 5070 Ti Super with 24 GB.

They would have mentioned the GPUs by now if it was compatible with 16 GB cards. Or maybe it's at very low resolution like full HD, and they don't want bad marketing, as in a 5080 is a full HD card now without people seeing the difference with their own eyes.

This should be coming soonish around fall. I expected more to be honest.

 

[CGNoire]

Please work with the 4090.

 

[Gonzito]

Please work with the 4090.

Same here, i am not buying at 5000 series for this

 

[Black_Stride]

Is Remedy going to be first again with this tech?

Control Resonant could easily work well with this considering they use scans for many of their NPCs.

 

[Wolzard]

IbizaPocholo If you want to add more details to the post:

DLSS 5: Three AI models, single-GPU operation and full control

What DLSS 5 means for gamers and developers: sharper 4K visuals, stable frame-to-frame output and more control over AI rendering on Geforce RTX 5000.

www.pcgameshardware.de

 

[Buggy Loop]

Them not talking about GPU requirements makes me think only the 5090 will run this. Maybe the 5080 and 5070 Ti Super with 24 GB.

They would have mentioned the GPUs by now if it was compatible with 16 GB cards. Or maybe it's at very low resolution like full HD, and they don't want bad marketing, as in a 5080 is a full HD card now without people seeing the difference with their own eyes.

This should be coming soonish around fall. I expected more to be honest.

This makes me hopeful :

• NVIDIA also indicated DLSS 5 is designed to operate on a single GPU with efficient VRAM usage.

Single GPU was of course known since early on as dual 5090 was probably a very early proof of concept, one rendering, the other inferring, probably because it was not ready to be in a single pipeline.

Nvidia has surprised me with legacy GPU support, but I think Ampere will be cutoff here though, not enough AI TOPs.

 

[JimmyRustler]

So it's not just a full screen filter after all... all the crying when it was announced in vain.

 

[Laptop1991]

Interesting tech,, if it makes games better and there isn't too much overhead of resources, i am all for it.

 

[dgrdsv]

I think Ampere will be cutoff here though, not enough AI TOPs

Ampere isn't far behind Lovelace in TOPs with Blackwell also being close.
There are three possibilities for h/w support IMO:
1. They'll support all RTX h/w, as with other DLSS features bar FG. 20 series will probably be too slow to run it though, considering that 2080Ti is about equal to 5060.
2. They'll support only FP8 capable RTX h/w which would mean Ada and Blackwell only.
3. They'll require whatever multiprocessor changes were implemented in Blackwell specifically meaning only 50 series supporting it.
The latter is a bit unlikely though cause I would expect them to tell this already by now.
Leaning towards 2 more personally.

 

[RafterXL]

Hasn't even been 6 months and all the whining about this announcement has already proven to be just noise. It's crazy how few people can see past the exact moment they are in and lack any ability to perceive the future. Like I said back in March, the tech will only ever get better and this is as bad as it will ever look. This tech is exciting, already looks significantly better than it did, the only question at this point is performance and whether or not we'll be seeing this on anything lower than a 50 series card.

 

[Cumulus]

So it's not just a full screen filter after all... all the crying when it was announced in vain.

You have a bad memory....

We already knew about the masking possibilities back then.
The Issues however are still the same, it`s screen space information and nothing more. It doesn`t know geometry or lightsources, it`s all just guesswork and if the devs aren´t very thorough with the manual masking etc we`ll get the exact same issue people complained about at the first showing.....
It´s still just a glorified instagram filter.

 

[Buggy Loop]

Hasn't even been 6 months and all the whining about this announcement has already proven to be just noise. It's crazy how few people can see past the exact moment they are in and lack any ability to perceive the future. Like I said back in March, the tech will only ever get better and this is as bad as it will ever look. This tech is exciting, already looks significantly better than it did, the only question at this point is performance and whether or not we'll be seeing this on anything lower than a 50 series card.

Yup, and AI tech iterates so fast that any glitch or imperfections would be either fixed by launch or some 5.x version down the line.

We went from crappy DLSS 1 February 2019 → DLSS 2 April 2020 which its baseline basically defined the AI upscalers of the past years.

 

[ResurrectedContrarian]

link to the actual video is in other thread

E

Post in thread 'Nvidia at Live GTC : DLSS 5'

Today at 3:51 AM

DLSS 5 is back and it looks incredible!

• EternalTruth

• 

 

[//DEVIL//]

This makes me hopeful :

• NVIDIA also indicated DLSS 5 is designed to operate on a single GPU with efficient VRAM usage.

Single GPU was of course known since early on as dual 5090 was probably a very early proof of concept, one rendering, the other inferring, probably because it was not ready to be in a single pipeline.

Nvidia has surprised me with legacy GPU support, but I think Ampere will be cutoff here though, not enough AI TOPs.

I think it was confirmed this is for 5000 series only and above.

But I am just wondering if 16gb cards are enough For 2k dlss 5. Because there is no way you will be able to get it to run it at 4k when some games native are close to that and dlss 5 still need VRAM anyway.

 

[Hoddi]

I think it was confirmed this is for 5000 series only and above only.

But I am just wondering if 16gb cards are enough For 2k dlss 5. Because there is no way you will be able to get it to run it at 4k when some games native are close to that and dlss 5 still need VRAM anyway.

We still don't know the model sizes. RTX 4000-series added FP8 support which needs much less than memory than FP16 and also runs much faster. The 5000-series then added support for FP4 which may or may not be required for DLSS 5.

It won't be free but it might only be a few hundred megabytes.

 

[Esppiral]

Nintendo decided to use 2013 h/w for this one so no.

FFY

 

[viveks86]

Ampere isn't far behind Lovelace in TOPs with Blackwell also being close.
There are three possibilities for h/w support IMO:
1. They'll support all RTX h/w, as with other DLSS features bar FG. 20 series will probably be too slow to run it though, considering that 2080Ti is about equal to 5060.
2. They'll support only FP8 capable RTX h/w which would mean Ada and Blackwell only.
3. They'll require whatever multiprocessor changes were implemented in Blackwell specifically meaning only 50 series supporting it.
The latter is a bit unlikely though cause I would expect them to tell this already by now.
Leaning towards 2 more personally.

There is the possibility of using NVFP4 as that could be well suited for generative AI. Or cooperative vectors may be how they are able to parallelize this enough to run on a single GPU. It's likely Ada and Blackwell or just Blackwell imo.

 

[lughnasadh123]

Let me adjust things to have mixed art styles within the same games ala Smiling Friends.

 

[Buggy Loop]

You have a bad memory....

We already knew about the masking possibilities back then.
The Issues however are still the same, it`s screen space information and nothing more. It doesn`t know geometry or lightsources, it`s all just guesswork and if the devs aren´t very thorough with the manual masking etc we`ll get the exact same issue people complained about at the first showing.....
It´s still just a glorified instagram filter.

Trying to the boogeyman tactic for peoples who have no idea how a GPU pipeline works.

It has access to albedos buffer, G-buffer and all its material properties, including surface normals and depth and lighting accumulation, for storing light samples.



Rasterization is by definition in "screen space". The moment 3D geometric shapes (triangles) are converted to a 2D grid of pixels (rasterization), you're in screen space (insert boogeyman). Most shaders are in screen space. Ambient occlusions are in screen space. Subsurface scattering is in screen space. Ray marching 3D clouds in Red Dead Redemption 2 in screen space. PBR materials, etc.

Every deferred rendering such as deferred lighting and deferred shading, is done in screen space. The principles of it is to remove the dependance between meshes and lights. Welcome to most of the game rendering for the past 20 years and all modern graphic engines. The list of effects done in screen space is so long that it's basically the principles of modern rendering.

id engine, Valve, Forza engine and some indies I guess, are basically the last bastion of forward rendering. That's basically it. And I'm not gonna start arguments on which are better in the end, just that a crushing majority of games nowadays are defered rendering. Or at the very least, moved to hybrid pipelines, especially for transparencies.

Same as ray reconstruction, see 4.0 tag all required resources, basically a match of DLSS 5.

Streamline/docs/ProgrammingGuideDLSS_RR.md at main · NVIDIA-RTX/Streamline

Streamline Integration Framework. Contribute to NVIDIA-RTX/Streamline development by creating an account on GitHub.

github.com

Unreal engine Megalights? Hybrid of a simplified ray tracing scene and then completes it with deferred lighting (screen space), that's how you get hundreds of lights

MegaLights in Unreal Engine | Unreal Engine 5.8 Documentation | Epic Developer Community

A whole new direct lighting path to place many dynamic and shadowed area lights in Unreal Engine.

dev.epicgames.com

RTXDI ? Ray tracing is dispatched from G-buffer and does light accumulation in G-buffer. Multiple bounces? Multi-sample accumulation buffers. Almost every modern lighting techniques are an hybrid of techniques because performances would tank.

RTXDI/Doc/Integration.md at main · NVIDIA-RTX/RTXDI

Contribute to NVIDIA-RTX/RTXDI development by creating an account on GitHub.

github.com

The primary use case for RTXDI is sampling direct lights for primary surfaces, available in the G-buffer, and resampling those lights using the motion vectors and the previous G-buffer

• ReSTIR DI, short for Reservoir SpatioTemporal Importance Resampling for Direct Illumination, is a screen space light sampling technique used for illuminating primary surfaces during path tracing. It cheaply selects several lights from an initial distribution, such as a uniform or power-based one, then selects one light of that group based on a more expensive distribution, such as one including geometry, BRDF, and visibility terms, and uses that one for illumination. These samples are then resampled over time and across neighboring pixels to increase effective sample count.
• ReGIR, short for Reservoir-based Grid Importance Sampling, is a world space light sampling technique that can be used for illuminating primary or subsequent surfaces. It samples lights from an initial distribution, such as a uniform or power-based one, for each grid cell. Other algorithms, such as ReSTIR DI and ReSTIR GI, can sample lights from this grid if the surface they are illuminating falls in a grid cell.
• ReSTIR GI, short for Reservoir SpatioTemporal Importance Resampling for Global Illumination, is a screen space light sampling technique used for illuminating secondary surfaces during path tracing. ReSTIR GI requires an initial light sampling technique be implemented by the path tracer for the secondary surface. In the full-sample project, this method is the initial sampling pass from ReSTIR DI, although any technique may be used. This secondary surface sample is then resampled across other pixels in space and time.
• ReSTIR PT, short for Reservoir SpatioTemporal Importance Resampling for Path Tracing, is a screen space path resampling technique used for illuminating secondary surfaces. ReSTIR PT requires an integration of the RTXDI_PathTracingContext into a path tracer that calculates indirect lighting starting from the primary hit.

Star Wars Outlaws uses short screen space rays to handle geometry and lighting immediately in front of the surface, if the ray fails to hit anything then they switch to a hardware ray-tracing pass with BVH.

Mommy, screen space is scary! /s

The whole past 2 decades of rendering and ~98% of games out there (or anything accessing the G-buffer) is guess work and instagram filter according to you. Jesus fucking Christ.

 

[DarkLordOtaku]

I just really want them to release it at this point, so we can either validate or put to rest all of the speculation around this.

 

[Cumulus]

Trying to the boogeyman tactic for peoples who have no idea how a GPU pipeline works.

It has access to albedos buffer, G-buffer and all its material properties, including surface normals and depth and lighting accumulation, for storing light samples.



Rasterization is by definition in "screen space". The moment 3D geometric shapes (triangles) are converted to a 2D grid of pixels (rasterization), you're in screen space (insert boogeyman). Most shaders are in screen space. Ambient occlusions are in screen space. Subsurface scattering is in screen space. Ray marching 3D clouds in Red Dead Redemption 2 in screen space. PBR materials, etc.

Every deferred rendering such as deferred lighting and deferred shading, is done in screen space. The principles of it is to remove the dependance between meshes and lights. Welcome to most of the game rendering for the past 20 years and all modern graphic engines. The list of effects done in screen space is so long that it's basically the principles of modern rendering.

id engine, Valve, Forza engine and some indies I guess, are basically the last bastion of forward rendering. That's basically it. And I'm not gonna start arguments on which are better in the end, just that a crushing majority of games nowadays are defered rendering. Or at the very least, moved to hybrid pipelines, especially for transparencies.

Same as ray reconstruction, see 4.0 tag all required resources, basically a match of DLSS 5.

Streamline/docs/ProgrammingGuideDLSS_RR.md at main · NVIDIA-RTX/Streamline

Streamline Integration Framework. Contribute to NVIDIA-RTX/Streamline development by creating an account on GitHub.

github.com

Unreal engine Megalights? Hybrid of a simplified ray tracing scene and then completes it with deferred lighting (screen space), that's how you get hundreds of lights

MegaLights in Unreal Engine | Unreal Engine 5.8 Documentation | Epic Developer Community

A whole new direct lighting path to place many dynamic and shadowed area lights in Unreal Engine.

dev.epicgames.com

RTXDI ? Ray tracing is dispatched from G-buffer and does light accumulation in G-buffer. Multiple bounces? Multi-sample accumulation buffers. Almost every modern lighting techniques are an hybrid of techniques because performances would tank.

RTXDI/Doc/Integration.md at main · NVIDIA-RTX/RTXDI

Contribute to NVIDIA-RTX/RTXDI development by creating an account on GitHub.

github.com

Star Wars Outlaws uses short screen space rays to handle geometry and lighting immediately in front of the surface, if the ray fails to hit anything then they switch to a hardware ray-tracing pass with BVH.

Mommy, screen space is scary! /s

The whole past 2 decades of rendering and ~98% of games out there (or anything accessing the G-buffer) is guess work and instagram filter according to you. Jesus fucking Christ.

The issue stays the same so....

and wildly inaccurate in parts on top of it....

 

[Buggy Loop]

Weakest of potential replies. What a disappointment. So you're shitposting because "AI" with no understanding of the pipeline. Glad that's settled.

 

[Cumulus]

Weakest of potential replies. What a disappointment. So you're shitposting because "AI" with no understanding of the pipeline. Glad that's settled.

Lol If I engaged every time some wannabe expert started spewing generalized nonsense in a discussion I´d probably not have time to sleep anymore....

 

[Buggy Loop]

Lol If I engaged every time some wannabe expert started spewing generalized nonsense in a discussion I´d probably not have time to sleep anymore....

It's in the papers, SDK kit information, siggraph discussions, and more importantly, 2 decades of deferred rendering teaching in schools, graphic engine documentation. Not "generalized nonsense". What a stupid cope out that you don't know shit

 

[Cumulus]

It's in the papers, SDK kit information, siggraph discussions, and more importantly, 2 decades of deferred rendering teaching in schools, graphic engine documentation. Not "generalized nonsense". What a stupid cope out that you don't know shit

Yeah,utter bullshit. But if you feel better now that's ok.

 

[Buggy Loop]

Yeah,utter bullshit. But if you feel better now that's ok.

Utter bullshit , all graphic engines, SDKs, siggraphs, universities, all bullshit

You're heading towards flat earth levels of disconnect with how graphic engines work, ffs.

All rasterization/shader is guesswork according to you, the entire graphic engine industry is utter bullshit /s

We went from "it's a post process effect" clowns to screenspace bogeymans. While it has access to everything that ray reconstruction does and is directly in the pipeline. Continue believing your little world of nonsense. Don't ever open a graphic engine and read the documentation, it'll blow your mind.

 

[dgrdsv]

There is the possibility of using NVFP4 as that could be well suited for generative AI. Or cooperative vectors may be how they are able to parallelize this enough to run on a single GPU. It's likely Ada and Blackwell or just Blackwell imo.

FP4 is likely to be too low for any graphical work, and coop vectors (or whatever they are called now) are supported on all RTX GPUs.

 

[Cumulus]

Utter bullshit , all graphic engines, SDKs, siggraphs, universities, all bullshit

You're heading towards flat earth levels of disconnect with how graphic engines work, ffs.

All rasterization/shader is guesswork according to you, the entire graphic engine industry is utter bullshit /s

We went from "it's a post process effect" clowns to screenspace bogeymans. While it has access to everything that ray reconstruction does and is directly in the pipeline. Continue believing your little world of nonsense. Don't ever open a graphic engine and read the documentation, it'll blow your mind.

Nothing but nonsense generalisations and absolutes that could come straight from a kindergartner since your first reply as if you`d have a discussion with yourself in your head, lol

Peak Dunning Kruger

 

[Buggy Loop]

Nothing but nonsense generalisations and absolutes that could come straight from a kindergartner,lol

Peak dunning Kruger right here.

Ah yes, SDK documentation straight from a kindergartner called Nvidia and Epic
See this clown viveks86 ?

And you keep just repeating the same shit over and over without bringing anything to the table. Full blown ad hominem fallacy. Weakest of replies.

 

[geary]

Let's drop the whole "you're stupid, no you're stupid" back-and-forth. I'm genuinely interested in discussion about the technology behind DLSS5, not the petty jabs between AI enthusiasts and detractors.

 

[Cumulus]

Ah yes, SDK documentation straight from a kindergartner called Nvidia and Epic
See this clown viveks86 ?

And you keep just repeating the same shit over and over without bringing anything to the table. Full blown ad hominem fallacy. Weakest of replies.

oh jesus. I´d recommend to take a step back and review your replies.
Holy delusions, batman....

 

[viveks86]

FP4 is likely to be too low for any graphical work, and coop vectors (or whatever they are called now) are supported on all RTX GPUs.

Yeah I'm skeptical about FP4 too, but who knows? It's certainly not sufficient for upscaling, but for hidden neural layers where it's all about moving data, it could be something. The theoretical boosts could be massive. Something that a realtime generative pipeline would need.

Didn't know coop vectors were viable before Ada. Has it been tested in any exploratory work?

 

[dgrdsv]

Didn't know coop vectors were viable before Ada. Has it been tested in any exploratory work?

I think I've seen tests of NTC with them on Turing and Ampere?
Coop vectors were just an API construct mapping the DX s/w to the AI h/w, any RTX GPU could've run them, any GPU could've run them in fact - provided the drivers would support the feature - it's just that performance would be low if such GPU wouldn't have specialized matrix h/w for such AI workloads.
MS has folded coop vectors into a bigger/wider Linear Algebra API now but both of its parts should also work on all RTX GPUs as they just add more API interfaces to the same tensor h/w any RTX GPU has.

Blackwell should be better at handling the last option in this image as well as the parts where ML workload is inside a shader:

But the extent to which it will be better in practice isn't known yet. DLSS NR should be the first real application where we might see 50 series running better than previous RTX GPUs.