Texture dumping feature dumps video frames

[vlbastos]

Hello all. I've been trying to use the texture dumping feature here, but it keeps dumping frames from FMVs, which causes the creation of an absurd ammount of files in the dumped textures folder. My objective is to upscale the dumped textures in realtime using a batch script to waifu2x-caffe.exe them, and then reload textures ingame.

It works nicely in PCSX2, but in Dolphin, I keep getting these FMV frames in the textures folder, basically killing my NVME softly and adding an absurdly huge ammount of wasted processing power and disk space by upscaling those.

I can't even detect those textures and delete them before upscaling, because they have no unique aspect I could use to detect them.

Is there a way to get rid of those frames, as in, not dumping them? I tried messing with the Hacks\Store EFB/XFB Copies to Texture Only, but no luck. Is there a way to disable EFB to texture when playing FMVs?

   

[JosJuice]

Is there a way to get rid of those frames, as in, not dumping them?

No, Dolphin doesn't have any options for this.

Is there a way to disable EFB to texture when playing FMVs?

No. That doesn't affect this anyway.

[vlbastos]

- Any possibility of implementing a "Don't dump FMV frames as textures" option?

- Any tips for detecting these dumped frames, so I can delete them before upscaling?

[iwubcode]

- Any possibility of implementing a "Don't dump FMV frames as textures" option?

I'd like to provide a way to know when a FMV occurs and then dump it as say a video or something that is more intuitive and can be replaced. However, it's no more than an idea at this point. Auto detection is a challenge. I've considered providing manual cues on a game-by-game basis but it's not ideal.

As an aside, another user has requested a way to skip any textures in general (for dynamic textures). See issue 12905. I might implement that at some point but it's not available now. The feature, like the manual FMV detection, seems unfortunate to me and dynamic textures outside FMV are rather rare.

- Any tips for detecting these dumped frames, so I can delete them before upscaling?

Not that I'm aware of. Honestly, if we could detect it easily we would have already added an option.

My objective is to upscale the dumped textures in realtime using a batch script to waifu2x-caffe.exe them, and then reload textures ingame.

I personally am not a fan of these kinds of solutions as it doesn't give back to the community (though upscaled textures are a legal grey area). However, this idea has been done before, see: https://github.com/WalkerMx/AutoCrispy

[Venomalia]

- Any tips for detecting these dumped frames, so I can delete them before upscaling?

You can recognize them very well by the size and format
the resolution is usually 304x224, 320x240, 608x448, 640x336 or 640x480.
the format is always "I8" so the texture name ends with _1.
together it looks like this.
tex1_320x240_XXXXXXXXXXXXXXXX_1.png

Is there a way to get rid of those frames, as in, not dumping them?

You can use the TextureExtraction tool for some games, it does not dump video frames.

My objective is to upscale the dumped textures in realtime using a batch script to waifu2x-caffe.exe them, and then reload textures ingame.

waifu2x-caffe has incredibly poor quality on most Gamecube and Wii textures and is outdated.
if your focus is on speed, there are ESRGAN-lite, Compact models these are slightly faster and produce very good results.

[vlbastos]

I'd like to provide a way to know when a FMV occurs and then dump it as say a video or something that is more intuitive and can be replaced.  However, it's no more than an idea at this point.  Auto detection is a challenge.  I've considered providing manual cues on a game-by-game basis but it's not ideal.

As an aside, another user has requested a way to skip any textures in general (for dynamic textures).  See issue 12905.  I might implement that at some point but it's not available now.  The feature, like the manual FMV detection, seems unfortunate to me and dynamic textures outside FMV are rather rare.


Not that I'm aware of.  Honestly, if we could detect it easily we would have already added an option.


I personally am not a fan of these kinds of solutions as it doesn't give back to the community (though upscaled textures are a legal grey area).  However, this idea has been done before, see:  https://github.com/WalkerMx/AutoCrispy

I got into making this batch after I realized AutoCrispy has some limitations on features and performance, and I couldn't contact the author for some suggestions:
1. AutoCrispy can't run waifu2x-caffe.exe on FOLDERS, only on individual files, and that's a HUGE performance penalty for waifu2x-caffe.exe.
2. AutoCrispy isn't a set-and-forget solution, you need to configure the input and output paths everytime you change the game.
3. AutoCrispy can't output intermediary 2x textures to emulator folder when you chain, say, 2x then 2x again (it gets ONE file, scales 2x, doesn't output it, then it scales 2x again, and only then you get the output).
4. AutoCrispy can't send hotkeys for toggling external textures on and off or reload external textures to the emulator.

I addressed all of those with a few batches. You can copy my launcher batch for each emu, set that emu's configurations (executable, paths, hotkeys), then forget it forever. Launch the emu with the batch, and it will work with any game you play in it.

It will:
1. move the dumped textures to a temp folder (I STRONGLY recommed you set the temp folder to a RAM drive with a good size);
2. junction the dump folder to the temp folder (which causes the dumped textures to be saved directly to RAM drive);
3. separate XX textures to be processed at the same time (I'm using 32 now, but 64 gives good results too);
4. upscale the textures to 2x (in RAM drive) and copy them to emu folder (so you can see the 2x textures faster ingame);
5. send the "reload" or the "toggle" hotkey to the emu window to refresh the textures ingame;
6. upscale the 2x textures to 4x (in RAM drive) and move them to emu folder;
7. refresh the textures again;
8. start it all over again.

Since it's just a batch, you can easily edit it for any backend you want, not only waifu2x-caffe.exe. I will try the ESRGAN backends later.

I started making those with PCSX2, but having Dolphin in mind. I had to adapt a few things for Dolphin, and it's working, although the hotkey part didn't work when I tested it. Will check it soon.

Talking about "giving back to the community", here it goes:

  WaifuLauncher.zip (Size: 4.99 KB / Downloads: 61)

Try with PCSX2 and games like God of War, Shadow of the Colossus, Genji, Klonoa 2 (set the "Toggle External Textures" hotkey to 0 (the zero key) and the "Reload Textures" hotkey to 9 (the nine key)).... DON'T TRY WITH GAMES WITH WEIRD DYNAMIC/PROCEDURAL TEXTURES LIKE TALES OF LEGENDIA, these produce infinite textures while you play, and the batch will keep upscaling these random textures until your PC melts. I will try to detect those weird textures and delete them instead of upscale them in the future.

For Dolphin, I only tried Super Mario Sunshine so far. And it has the weird textures problem. So it's not a good start, I will try to fix this soon.

[vlbastos]

You can recognize them very well by the size and format
the resolution is usually 304x224, 320x240, 608x448, 640x336 or 640x480.
the format is always "I8" so the texture name ends with _1.
together it looks like this.
tex1_320x240_XXXXXXXXXXXXXXXX_1.png


You can use the TextureExtraction tool for some games, it does not dump video frames.


waifu2x-caffe has incredibly poor quality on most Gamecube and Wii textures and is outdated.
if your focus is on speed, there are ESRGAN-lite, Compact models these are slightly faster and produce very good results.

Thanks a lot for these tips!!! I, too, thought about the resolution part. So it's a good direction. I'll certainly delete textures with those sizes+formats, and will make some kind of universal "blacklist" for textures. I noticed that I can blacklist too small and too big textures.

TextureExtractioin tool seems awesome, will try it too. But what I'm really trying to do is an universal realtime upscaler for any game and any emulator, because I'm lazy and I just want to remaster all old games to 4k in realtime while I play them! 

I'm using waifu2x-caffe.exe because it's the only one that didn't break the alpha in textures in the games I tried. After tinkering a lot with it, I got a configuration that runs quite fast (on folders, not on individual files), and it's fast enough to be used while playing. But I'm REALLY interested into trying these ESRGAN you suggested. Can you recommend one for realtime upscaling? I don't have the exact numbers, but when playing Shadow of the Colossus on PCSX2, I could get something like 64 textures upscaled to 2x then 4x in 16 seconds or so (5 seconds for 2x, then 11 seconds for 4x) (SotC uses thousands of textures in just a few minutes of gameplay).

[Venomalia]

I'm using waifu2x-caffe.exe because it's the only one that didn't break the alpha in textures in the games I tried. After tinkering a lot with it, I got a configuration that runs quite fast (on folders, not on individual files), and it's fast enough to be used while playing. But I'm REALLY interested into trying these ESRGAN you suggested. Can you recommend one for realtime upscaling? I don't have the exact numbers, but when playing Shadow of the Colossus on PCSX2, I could get something like 64 textures upscaled to 2x then 4x in 16 seconds or so (5 seconds for 2x, then 11 seconds for 4x) (SotC uses thousands of textures in just a few minutes of gameplay).

alpha in ESRGAN depends very much on the implementation, and the behavior of the model since most models use 3 channels and do not support alpha natively...
You will get the best quality if you separate the color and alpha channels and upscale them separately, but it will cost even more time.

Tensorrt-Docker should be the fastest ESRGAN inference backend and the fastest model i have seen so far is sudo_UltraCompact_2x_G, however it is optimized for animes.
I think the best overall result would be to find a fast model architecture and make a lite version based on my HDcube mode.
I have never dealt much with realtime upscaling, the discord server "Enhance Everything" should be a good contact point for you.

[vlbastos]

alpha in ESRGAN depends very much on the implementation, and the behavior of the model since most models use 3 channels and do not support alpha natively...
You will get the best quality if you separate the color and alpha channels and upscale them separately, but it will cost even more time.

Tensorrt-Docker should be the fastest ESRGAN inference backend and the fastest model i have seen so far is sudo_UltraCompact_2x_G, however it is optimized for animes.
I think the best overall result would be to find a fast model architecture and make a lite version based on my HDcube mode.
I have never dealt much with realtime upscaling, the discord server "Enhance Everything" should be a good contact point for you.

I tested your HDCube 4x model with AutoCrispy's ESRGAN setup (Which WalkerMx kindly provided a prebuilt, "portable" 6GB installation so we don't need to worry about Python installs and etc), and.... OH MY GOD. The quality upgrade made me WOW just like the first time I upscaled from original to Waifu2x 4x. PERFECT DETAILS. PERFECT OUTLINES. Holy cow.

Unfortunally, the performance crawls, as expected. I've never seen a laptop get so hot in my life. It's totally not suitable for emulating and upscaling at the same time, and it's not because of speed alone, it brings the entire PC to it's knees. I don't know if it's AutoCrispy's fault, because I still didn't check if I can upscale folders instead of individual files.

I stepped further and discovered Real-ESRGAN. Xinntao made a portable realesrgan-ncnn-vulkan.exe for Windows, which is BLAZING FAST, and guess what? It can also use your HDCube 4x model. FAST. REAL FAST. Faster than the waifu2x-caffe.exe I was using. No need to install Python, Docker, anything like that. Just unzip and use. Got 2 GPUs in your laptop? Turn on a single switch and use both at the same time to upscale. ALMOST perfect.

As he explains in his README_windows.md: "it may introduce block inconsistency (and also generate slightly different results from the PyTorch implementation), because this executable file first crops the input image into several tiles, and then processes them separately, finally stitches together." And it does. Lots of tiling artifacts in some textures. Also, the outlines aren't as good as AutoCrispy+Python's results. I tried increasing tile size to 512, got tons of runtime errors (vram stuff), then tried 256, managed to upscale everything, but the tiling artifacts still remained. Sucks.

But the overall results are VERY good. VERY comparable to Autocrispy+Python. At the naked eye, sometimes even better (in big textures with more details). Next, I'm gonna try using the "portable" WalkerMx Python install with Real-ESRGAN or maybe do a real installation and try the hard way, to see if I can get better quality outlines and tiles, coupled with better performance than AutoCrispy.

[Venomalia]

PyTorch needs a lot of resources but the quality is good, needs a nvidia card and NCNN is very solid but sometimes has quality drops. you could also try Tensorrt-Docker, as mentioned it is designed for speed and has good quality.