Support NeoGAF

[Tripolygon]

I remember in the start of the generation when Sony said PS5 is capable of running machine learning inference for use in games. We have seen a few Sony studios use machine learning inference at runtime in their games. Spiderman for muscle deformation, Horizon FW in their temporal upsampling and now God of War in texture compression and upsampling. Remember BC-Pack that microsoft was advertising during the launch of Series X, think something like that but more advanced using neural network.

Every frame, the texture streaming system detects textures that require upsampling , and sends them over to the upsampling system.

Goal.

We hoped that upsampling textures at run-time would help us save disk space – Artists author textures for PS4, and we upsample these textures on PS5 while keeping roughly the same package size.

We wanted to use a single network to handle both upsampling and compression, and output directly to BC. The player is always moving from one level to another, and upsampling must keep up.

Design

Result

In 10 minutes, the system produced 760 million BC1 and BC7 blocks with both our method and the method provided by Sony, which is around 10 gigabytes of BC1 and BC7 blocks if no other form of compression is used. Around 70% of the 10 gigabytes of data is produced by neural networks, which is equivalent to around 42 4k textures per minute.

On PS5 fp16 is used.

One of the simplest optimization is adopting 16-bit floats, also known as half-precision floats or halves

It is expensive in terms of cost but worth it. That is why they use spare resources to run it during gameplay.

The way we determine how much BC7 blocks to upsample is by tracking running averages of excess frame time and the duration of one evaluation of any specific network.

it upsamples one 2k texture to 4k in around 9.5 milliseconds, shaving off over a whole millisecond

Issues

Since BC7 works with 4x4 pixel blocks, the networks sometimes produce block artifacts. This is especially visible for highly specular surfaces under direct lighting. We attempted to fix this by training with four neighboring blocks at the same time and adding first and second order gradients to the loss function. These helped somewhat but the issue was still visible.

Machine Learning Summit: Real-Time Neural Texture Upsampling in 'God of War Ragnarok'

This talk gives a detailed breakdown of the run-time neural texture upsampling system in God of War Ragnarkincluding how the networks are designed to directly output BC compressed image blocks, how the process is optimized to take full advantage...

www.gdcvault.com

 

[sankt-Antonio]

not seeing a difference on my screen with these comparisons :/

 

[M1chl]

That's really cool. Wonder how much of GPU it needs to do this.

 

[Daneel Elijah]

not seeing a difference on my screen with these comparisons :/

The images were used in a projector I think. So a lot bigger than our TVs/ monitors. There is more of it in the link. What seems to be nice here is that the PS4 textures simply get upsampled without manual work? And with a gain compared to other methods. Only loss is that it is using computational power. So CPU. But there is so much more of it than the PS4 than for GOW this is not a problem. What is nice too is that if they are showing this at GDC, they have talked about it with others Sony studios too. Will wait for someone more knowlegable of the tech to explain a little more why it is a good thing.

 

[MasterCornholio]

Thanks for thr article and the summary. Really cool to read about tech that's used in that system.

 

[Raven77]

Great write up.

Now looking at the comparisons...

 

[SJRB]

I legit feel like I'm being trolled.

 

[Black_Stride]

Well thats one way to make the up port easy.

Though id think, it would have been easier to have your high res textures and then downres them for the PS4.
Did they lose all their high res maps?


EDIT: Ohh this is being done at runtime?
Okay thats cool, I can respect that as one way to keep file sizes low.

 

[Shh]

not seeing a difference on my screen with these comparisons :/

It helps if your eyelids aren't closed.

 

[Trogdor1123]

Can you point the differences out please?

 

[GHG]

That's the purpose. Read the "design goals" section.

 

[CGNoire]

Most of those before and after shots look the same. In the end it still looked like PS4 textures.

 

[Bo_Hazem]

BRB trying to enhance my concentration to notice the difference.

 

[onQ123]

Fp16 Bitches!

 

[Raonak]

That's very cool tech, ML upscaling on the actual game assets.

Comparisons can't really be noticed without direct 4K screenshots.

 

[onQ123]

The smart thing for next gen would be to have a common pool of textures already on the console & distribute really small games that create full size textures on the fly using the commonly used data that's on every console.

 

[lh032]

mommy i cant find this hidden sauce where did you put it.

 

[TheRedRiders]

I can't notice much of a difference in the screenshots, however one of the first things you'll notice on the PS5 version of Ragnarok is the very high texture resolution. It's one of the most impressive things about the game.

 

[mrmustard]

Is this one of those?

 

[ToTTenTranz]

So Nvidia's Revolutionary Neural Textures presented for the first time two weeks ago in tech demos that were going to change the world... had actually been preceded by a software implementation from Santa Monica and put into a shipping title half a year before, running on RDNA2 hardware without the need for dedicated tensor units.

Oops.


About this thread:

Some users: But I cAn'T sEe aNy dIfFeReNcE...

Yes, that was the point.
The "no upscale" pictures take a lot more disk space, over 3x more, and with this you can't notice the difference.

 

[sankt-Antonio]

So Nvidia's Revolutionary Neural Textures presented for the first time two weeks ago in tech demos that were going to change the world... had actually been preceded by a software implementation from Santa Monica and put into a shipping title half a year before, running on RDNA2 hardware without the need for dedicated tensor units.

Oops.


About this thread:

Some users: But I cAn'T sEe aNy dIfFeReNcE...

Yes, that was the point.
The "no upscale" pictures take a lot more disk space, over 3x more, and with this you can't notice the difference.

I thoughts it’s saving disc space by using the low res PS4 textures and then upscale it “live” to a better quality. Instead of doing higher res textures. But at the end the textures need to look better then the base PS4 textures to make sense because this process isn’t free. You give away CPU performance for Disk space and less dev time for cross games.

I just don’t see the benefit in the pictures.

 

[winjer]

not seeing a difference on my screen with these comparisons :/

I think the goal they had was to have lower disk and vram usage, while retaining the same quality.
And in this regard, it seems they were very successful.

 

[sankt-Antonio]

I think the goal they had was to have lower disk and vram usage, while retaining the same quality.
And in this regard, it seems they were very successful.

Not lower, but same disc space as the PS4 version while apparently looking better. I just don’t see it doing that in the pics. I’m sure it’s working in game perfectly fine.

 

[Bo_Hazem]

mommy i cant find this hidden sauce where did you put it.

Yeah seems like that's what it's meant for, to not notice a difference while using a lower quality texture AFAIK.

 

[winjer]

Not lower, but same disc space as the PS4 version while apparently looking better. I just don’t see it doing that in the pics. I’m sure it’s working in game perfectly fine.

I meant for the PS5. They could have made higher resolution textures for the PS5 without this compression system, using more vram and disk space.
This way not only did they save time and money, by using the original PS4 textures, but also vram and space. It's a win all around.

 

[sankt-Antonio]

I meant for the PS5. They could have made higher resolution textures for the PS5 without this compression system, using more vram and disk space.
This way not only did they save time and money, by using the original PS4 textures, but also vram and space. It's a win all around.

At the cost of CPU performance.

 

[winjer]

At the cost of CPU performance.

But it's using the GPU FP16 to do the upscaling.

 

[Mephisto40]

 

[sankt-Antonio]

But it's using the GPU FP16 to do the upscaling.

And? It’s still using the CPU for a task not needed of the textured had been higher res from the start. It’s a trade off.

 

[oldergamer]

Didn't Microsoft demo this same tech before the launch of series x. They just called it ml texture upscaling or something like that. They already had some developers working on using it. Not sure the new fancy name was needed.

 

[Killer8]

The difference is more noticeable when it's not being show in a tiny image:

Lossless comparison:

God of War Texture Upscaling - Imgsli

 

[winjer]

And? It’s still using the CPU for a task not needed of the textured had been higher res from the start. It’s a trade off.

Where did you see the CPU being used?

 

[CGNoire]

So Nvidia's Revolutionary Neural Textures presented for the first time two weeks ago in tech demos that were going to change the world... had actually been preceded by a software implementation from Santa Monica and put into a shipping title half a year before, running on RDNA2 hardware without the need for dedicated tensor units.

Oops.


About this thread:

Some users: But I cAn'T sEe aNy dIfFeReNcE...

Yes, that was the point.
The "no upscale" pictures take a lot more disk space, over 3x more, and with this you can't notice the difference.

No the "no upscale" is the lower res texture before any processing which means for this whole process to be even worth it you should see a difference.
The fact that you cant means it wasnt worth the time and makes no sense in showcasing like this since they could have just left the "lower res" textures as is and the disk space would never get gobbled up to begin with and you would end up with the same look.

 

[CGNoire]

The difference is more noticeable when it's not being show in a tiny image:

Lossless comparison:

http://[URL]https://imgsli.com/MTc4NDk1

[/URL]

nice catch. it does look a little bit better....but also a little artificial as well.

 

[sankt-Antonio]

Where did you see the CPU being used?

Cons: “More wasted computation” in the slides or the OP

 

[winjer]

Cons: “More wasted computation” in the slides or the OP

That is GPU computation. FP16, Wave32, etc.

 

[ToTTenTranz]

And? It’s still using the CPU for a task not needed of the textured had been higher res from the start. It’s a trade off.

It's obviously using the GPU. They say so right at the start of the presentation and later on proceed to show a bunch of methods they used to fit the operations within the GPU's L1 cache.
It makes little sense to run NN inference for textures on the CPU. The textures go to the GPU's cache and the upscaling is done locally, without traveling to the RAM.

No the "no upscale" is the lower res texture before any processing which means for this whole process to be even worth it you should see a difference.
The fact that you cant means it wasnt worth the time and makes no sense in showcasing like this since they could have just left the "lower res" textures as is and the disk space would never get gobbled up to begin with and you would end up with the same look.

The images on the first post compare big size textures with small size + neural upscaling, which is what the author shows at the end of the presentation. The point is exactly to show how little difference there is, while saving a lot of data in the whole pipeline.

For those still insisting on this "not being worth the time" because they couldn't grasp the goals of image compression, the author does show a couple examples between bilinear upscaling and NN upscaling from the same small texture. In page 25:

Alas, this is not the point of the work. The goal in texture compression is always to preserve image quality while reducing file size, hence the comparisons showing similar quality.

 

[sankt-Antonio]

That is GPU computation. FP16, Wave32, etc.

Mixed up CPU/GPU. It’s a trade off with GPU computation.
So not just a positive on all fronts.

 

[LordOfChaos]

 

[TheThreadsThatBindUs]

Mixed up CPU/GPU. It’s a trade off with GPU computation.
So not just a positive on all fronts.

Everything in computer graphics and rendering is a trade-off. You're not really arguing anything meaningful.

 

[TheRedRiders]

Everything in computer graphics and rendering is a trade-off. You're not really arguing anything meaningful.

Took the words right out of my mouth lol

 

[zeroluck]

So Nvidia's Revolutionary Neural Textures presented for the first time two weeks ago in tech demos that were going to change the world... had actually been preceded by a software implementation from Santa Monica and put into a shipping title half a year before, running on RDNA2 hardware without the need for dedicated tensor units.

Oops.


About this thread:

Some users: But I cAn'T sEe aNy dIfFeReNcE...

Yes, that was the point.
The "no upscale" pictures take a lot more disk space, over 3x more, and with this you can't notice the difference.

Apple to orange, one is upsamping to higher res then upload to the GPU(more VRAM) and another is storing the NN compressed textures on the GPU bypassing the BC compression entirely(less VRAM).

 

[Insane Metal]

Damn!!!! That looks... exactly the same!

 

[SlimySnake]

About this thread:

Some users: But I cAn'T sEe aNy dIfFeReNcE...

Yes, that was the point.
The "no upscale" pictures take a lot more disk space, over 3x more, and with this you can't notice the difference.

Isnt the point to use better quality textures without them costing disk space?

If we arent seeing better textures then whats the point? Just use the same textures, they wouldve taken the same amount of space.

 

[Tripolygon]

This thread serves as a reminder that about 90% of people who post in this forum know fuck all about the process of making video games even though they spend most of their life talking about it or playing it.

Oh but I can't see any difference, no shit, you are looking at a compressed screenshot measuring ~960x530 taken using windows snipping tool from a pdf opened in google chrome window less than 1080p then uploaded to imgur which further compresses it viewed from no doubt your 6.5 inch phone screen. When aligned side by side of course you won't see much difference, but if superimposed like Killer8 did, you will see a difference. Which might not seem like a lot but in terms of storage space saved, it is a lot. Also the normals is what is being mostly compressed and upsampled, normals are used to define light bounces on a surface which create detail where there is none in the underlying geometry.

 

[ToTTenTranz]

Mixed up CPU/GPU. It’s a trade off with GPU computation.
So not just a positive on all fronts.

No, according to the presentation there are no additional CPU cycles involved in this approach.
A smaller texture gets sent to the VRAM and the GPU runs portions of the texture through a NN for upscaling.

IIRC the Spider Man Miles Morales' muscle deformation NN does run on the CPU, it's a geometry transformation.
This one does not.

Apple to orange, one is upsamping to higher res then upload to the GPU(more VRAM) and another is storing the NN compressed textures on the GPU bypassing the BC compression entirely(less VRAM).

Well it's apples to apples in the sense that there's no perceivable visual difference between both solutions.

Isnt the point to use better quality textures without them costing disk space?

If we arent seeing better textures then whats the point? Just use the same textures, they wouldve taken the same amount of space.

The advantages in the smaller size of the textures are actually multifold:

1 - Less storage space. Installation sizes get smaller. For example, GoW Ragnarok occupies 107GB on the PS4 version and 86GB on the PS5 version, despite the latter showing better texture detail.

2 - Less VRAM occupation per texture. Meaning you can fill the RAM with other stuff, like e.g. BVH trees for raytracing, or simply more cached assets to prevent stuttering.

3 - Lower bandwidth requirements for the same texture quality. This also means that, for ISO texture quality there's less data exchanged between GPU caches and RAM/VRAM, leading to lower power consumption. And in cases where the GPU clock is limited by power consumption (like pretty much all modern GPUs and iGPUs), this may lead to higher core clocks.


The only "downside" is the fact that there's higher ALU utilization. However, it's been hard for game developers to keep the consoles' compute utilization at acceptably high levels. even on the PS5.
I imagine that in solutions where the compute throughput power far exceeds a balanced approach (too much compute power, too little memory bandwidth), like e.g. AMD's 7840U / Z1 Extreme in the ROG Ally, techniques like this should be a godsend.

 

[hlm666]

So Nvidia's Revolutionary Neural Textures presented for the first time two weeks ago in tech demos that were going to change the world... had actually been preceded by a software implementation from Santa Monica and put into a shipping title half a year before, running on RDNA2 hardware without the need for dedicated tensor units.

Judging by the video in your link nvidia are upscaling from lower res assets (so smaller file sizes) and doing it in 1.x ms vs 9.x ms with maybe better end results texture detail wise.

 

[deriks]

I legit feel like I'm being trolled.

I was going to use this meme

@thread
I know that has a lot of computer shit on it, but man, only the water reflections I saw some difference, and like those Digital Foundry analysis, I bet that it was zoomed, so I don't care that much. But if helps performance, I'm all for

 

[DeepEnigma]

Machine learning, eh?

Another FUD from the Next-Gen OT scratched off the list.

 

[zeroluck]

No, according to the presentation there are no additional CPU cycles involved in this approach.
A smaller texture gets sent to the VRAM and the GPU runs portions of the texture through a NN for upscaling.

IIRC the Spider Man Miles Morales' muscle deformation NN does run on the CPU, it's a geometry transformation.
This one does not.





Well it's apples to apples in the sense that there's no perceivable visual difference between both solutions.




The advantages in the smaller size of the textures are actually multifold:

1 - Less storage space. Installation sizes get smaller. For example, GoW Ragnarok occupies 107GB on the PS4 version and 86GB on the PS5 version, despite the latter showing better texture detail.

2 - Less VRAM occupation per texture. Meaning you can fill the RAM with other stuff, like e.g. BVH trees for raytracing, or simply more cached assets to prevent stuttering.

3 - Lower bandwidth requirements for the same texture quality. This also means that, for ISO texture quality there's less data exchanged between GPU caches and RAM/VRAM, leading to lower power consumption. And in cases where the GPU clock is limited by power consumption (like pretty much all modern GPUs and iGPUs), this may lead to higher core clocks.


The only "downside" is the fact that there's higher ALU utilization. However, it's been hard for game developers to keep the consoles' compute utilization at acceptably high levels. even on the PS5.
I imagine that in solutions where the compute throughput power far exceeds a balanced approach (too much compute power, too little memory bandwidth), like e.g. AMD's 7840U / Z1 Extreme in the ROG Ally, techniques like this should be a godsend.

Nvidia paper has NN compression looking better than BC compression by 16x(540p to 4K) at the same memory footprint, this has it looking worse than BC compression at the same memory footprint. Its purpose is to saves disk space for a slight quality loss.