Custom Texture Tool PS v50.1

[Bighead]

(08-29-2021, 04:53 AM)Kupo Wrote: Btw I recently updated Lubuntu (20.10 -> 21.04) and now dolphin always crashs if texture replacement is enabled.
Is there any setting which I have to take care of? Because it worked fine with my old setup and I'm unsure about my old settings.

This is a known issue with linux when prefetching is enabled. Whether or not it happens when it's disabled, I don't know since I don't use linux. It was brought to my attention not too long ago (a few pages back on this thread) with Hypatia's Wind Waker pack. The workaround was to use a downscaled DDS pack or disable prefetching.

Edit: The issue report is here:
https://bugs.dolphin-emu.org/issues/11994

[Kupo]

Thank you, disabling prefatching fixed it for me Although I thought I had it enabled in the past, but maybe I'm wrong

So, I converted my PNGs to DDS and yea it's stunning how fast they load. It also looks good to me ingame, but my DDS are smaller than my optimized PNGs, what makes me sceptical, especially since I can't open the files in GIMP, so I can't compare them to the PNGs.

I used texconv to convert them to BC7

[Bighead]

Yeah that's the unfortunate thing about BC7, it does not have the greatest support outside of using it for textures. But alas, when it comes to the DDS container, BC7 is unmatched for general purpose RGBA. There are other, possibly superior block compressed formats like ASTC but it was never really picked up for PC gaming (competition from MS I imagine). There is a plugin for GIMP to open up older DDS formats, not sure if there is one for BC7. It amazes me to this day that windows still cannot create thumbnails or view the DX10 DDS formats, considering that both were created by Microsoft... but considering that is also still true for uncompressed DDS textures which come from the DX9 era, I don't have much hope that it will change.

Anyway, don't take literally too much about the file size part of what I said previously, my old write-up was way better. It was rather misleading to say that PNG will always be smaller than DDS in file size, as there are times when DDS files will be smaller in file size than PNG, it depends on how well the pixels can be compressed in the image. PNG images with a lot of diversity in colors are not going to compress as well as say, a purely green image. If all of your PNG files are larger than DDS in file size, then yeah that is a bit strange. I guess the thing to take away is that DDS file sizes are "fixed", every 32x32 texture will be 65kb, every 64x64 texture will be 257KB, etc. When it comes to an entire pack, (usually) on disk PNG will be smaller, DDS larger. In VRAM, PNG will be larger, DDS smaller. The program "SageThumbs" can view uncompressed DDS, but I don't think it has support for BC7 (yet), despite having it requested years ago.



Lastly, TexConv is fine for creating BC7 textures, it has the advantage that it's really fast at doing so and still has decent image quality. Compressonator has the ability to create higher quality DDS textures. Depending on how much quality you want to squeeze out of it, it can take anywhere from 5 seconds to 5 hours to convert a single texture. To strike a balance between speed/quality in CTT-PS, I created the "Auto" setting that adjusts settings based on resolution and alpha/opaque. Smaller textures and/or textures with alpha get slightly higher quality than larger and/or opaque images.

[Kupo]

Okay, I think I will just offer my texture pack as PNG and as DDS. And if somebody notices issues with the DDS pack he hopefully will tell me
Thanks again for your time

[Bighead]

Yeah it's not likely that most people will notice a difference. IMO the performance of DDS will always trump the quality of PNG. And if there are textures you absolutely do not want to lose quality (like smaller UI textures), you can always use ARGB instead of BC7 for those. Just remember they are huge in file size, 4x larger than BC7, but equal to PNG size in VRAM.

I'll eventually have to do another write-up of DDS vs. PNG file sizes because it has come up a lot over the years. Looking back at my previous posts I generalized a lot, some of the info is inaccurate, but the points and the calculations are at least accurate.

[Bighead]

New update which I highly suggest grabbing if you use this to create DDS textures with Compressonator:
https://www.mediafire.com/file/d65bux18a99jfq9/Custom_Texture_Tool_PS_v48.0.zip/file

It turns out that I made a rather substantial mistake which lead to Compressonator never actually creating textures, the script was always falling back to TexConv, hence the full version bump. While textures created with TexConv are fine, they don't match the quality of textures created with Compressonator. The problem was due to the new options that are present when clicking [Configure Tool] under the DDS options. More specifically, both commands were always being added when creating DDS textures which caused the failure, and the correct command was never being added when converting DDS to PNG.

Spoiler: (Show Spoiler)

The two options which caused the issue were "Image Compression" and "Image Decompression", which I have renamed to Compression Encoder and Compression Decoder. These allow selecting what to use when either creating DDS images from PNG images, or when creating PNG images from DDS images. I didn't do adequate research into what each option does in the last version, but now I have a somewhat better understanding.

Compression Encoder:
CPU - Compress using only the CPU.
HPC - Compress using CPU High Performance Compute.
GPU - Compress using OpenGL compute.
DXC - Compress using Direct-X compute.
VLK - Compress using Vulkan compute.
OCL - Compress using OpenCL compute.

In short, this option uses either the CPU or GPU when compressing DDS textures and allows selecting which API to use when choosing the GPU. As expected, CPU is the slowest of the options, while HPC which supposedly also uses the CPU is the fastest for me for some reason. I have an overclocked Ryzen 3700x which has 8 cores and 16 threads, which seems to outperform creating BC7 textures using any of the GPU variants. I'm not sure why this is, it could be my GPU (GTX 1060) is old now, and it hasn't been easy finding an upgrade for a reasonable price. So I'm stuck with it for awhile...

Edit: I'm not sure Compressonator is even using the GPU at all. Running some tests and my GPU usage is always idling at 0-1% when running Compressonator no matter which option I use. When using TexConv, the GPU is clearly being used as GPU usage sits at a constant 100%. So I'm not really sure what is going on at the moment. I suggest just trying each option and see which one works best for you.

When decoding DDS textures (meaning, converting DDS to PNG), the option Compression Decoder comes into play. This only applies when entering the Additional Options menu and selecting "Compressonator" as the BC7 to PNG Converter.

Spoiler: (Show Spoiler)

Here you can choose from the usual suspects: CPU, OpenGL, DirectX, and Vulkan. I really didn't notice much difference between them, maybe a slight boost when using OpenGL and DirectX. When choosing Vulkan, Compressonator fails to generate a PNG image and instead the script falls back to TexConv. So I do not suggest using it, it may be an issue with Compressonator itself since my GPU definitely supports Vulkan.

---

Aside from that, I also added a quality adjustment when creating JPG images on the Additional Options menu which ranges between 0-100. Most programs that create JPG offer this, so I'm not sure why I never thought to add it before. The value represents a percentage, lower values result in lower quality, higher values result in higher quality.

Spoiler: (Show Spoiler)

I also made it so that the PowerShell window shows temporarily when checking/installing dependencies for Material Map Generator instead of just doing it in the background. The user should be informed of what is going on, so this window shows what is being downloaded and installed. Or it just shows that it's all there if it already is.

Spoiler: (Show Spoiler)

For the Custom CLI Executable operation, I noticed it was kind of a pain to remove commands from the list. When just deleting the text, it did not actually remove how many commands are applied until the script was restarted, which added blank commands. Or something, I don't think it caused problems but it didn't feel correct in its implementation. So two new buttons have been added above the command list [ - ] and [ x ] to remove the selected command or clear all commands from the list. There is no currently no confirmation when clearing them all, but I'll probably add one next version.

Spoiler: (Show Spoiler)

And that's about it outside of a few tooltip fixes. I do apologize to anyone who used this script to create BC7 textures since v47.0 was released as they probably weren't getting the quality they thought they were. If you want the highest possible quality, I do suggest regenerating your packs with this version. And do grab the latest version of Compressonator or the CTT-PS Programs Pack as older versions did not work with the -Performance command which does affect quality!

Always verify that the script is actually creating the texture with Compressonator, it will say which tool was actually used in the "Message" field.

[Bighead]

The next version has been released:
https://www.mediafire.com/file/c6tpirrnb1370rs/Custom_Texture_Tool_PS_v49.0.7z/file

Full changelog can be viewed here:
http://bhemuhelp.unaux.com/other/ctt/CTT-PS%20Changelog.txt

The DDS Options section has been changed to contain the options for all file formats. Just click on a "tab" to open the corresponding options. Truth be told, these are not real tabs but buttons cleverly disguised as tabs that auto resize and blend into the background when clicked. How it works isn't really important, just that it does and the illusion is convincing. Since all of these options were previously found on the Additional Options window, it has been removed along with the buttons that opened it.

Image Format Options: (Show Spoiler)

Text justification is now applied to all operation description labels, as well as a few other places in the GUI. This has been on my to-do list for awhile. Unfortunately, justified text is not something you can just choose to do by using default .NET functionality. This method is using imported code that gives access to 2 new controls: transparent label and advanced rich textbox. The textbox has its properties altered to appear like a label, and the transparent label overlaps it completely so it can not be clicked on. Another successful illusion.

Justified vs. Unjustified Text: (Show Spoiler)

The option to toggle HD Example Preview Images for the operation to generate Metroid Prime 2 fonts now no longer requires a restart of the script to take effect. Now it can be toggled back and forth at will.

Metroid Previews: (Show Spoiler)

When using the Combine/Split Texture operations, a file with the extension (.ctt) is generated when textures are combined. This file contains information about the images that went into the combined texture such as their file names, the row/columns used, and various other properties. Concerning this change, if the values set for pixel overlap or garbage pixels exceed zero, it will now toggle the appropriate checkboxes automatically when a CTT file is loaded into the script.

Pixel Overlap and Garbage: (Show Spoiler)

---

So while I usually cover the big changes first, this time I decided to save it for last. The big changes in this version concern the Combine/Split texture options. Admentus reached out to me on behalf of the Zelda retexture projects, which these options are very useful for and have had many features implemented just for Ocarina of Time and Majora's Mask. So where to begin...

The N64 Zelda games have static background images that are composed of 40 strips of images stacked on top of each other. CTT-PS can be used to combine these images into a single image. Each strip has a few rows and columns of "garbage pixels" that can automatically be cropped out and reinserted using the appropriate options. While someday I should merge all this information into a single post for the sake of posterity, if you really want to know more about all of this you can start here and read the many, many exchanges between Admentus and I while it was being developed.

But for now, let's start with an image that shows how the 40x strips appear when combining:

Spoiler: (Show Spoiler)

As it turns out, this is not the total sum of the images used in these background. There can actually be up to 6 "screen shake" textures that dump when Link bonks into the edge of the screen by doing a roll. You can get an idea of how these textures are used with this animation. Pay attention to the top and bottom of the image.

Spoiler: (Show Spoiler)

So the goal was, to somehow store these textures in the CTT file, and be able to automatically crop them from the combined image. This came with many, many challenges. First was to understand how the textures are actually being used in-game. After careful analysis, that was understood. Then it appears that these textures also have their own garbage pixels. Since this is a special case situation, these "garbage" values are hardcoded into the script. The amount of garbage depends on which texture we are referring to.

Spoiler: (Show Spoiler)

By clicking the "Triforce" button on the Combine Textures window, a new window opens to select these textures.

Spoiler: (Show Spoiler)

These textures are then stored in the CTT file when the image is combined, and cropped out when the image is split.

Spoiler: (Show Spoiler)

Edit: A day late, but I forgot to mention the final "Textures" line where a texture is "duplicated" with an alternate name. Some backgrounds may have an alternate texture dump that contains different garbage pixels when entering from an alternative entrance. Since these pixels don't matter, the same texture can be cropped from the same location and can serve as both textures. While there is currently no way to do this in the GUI, it is possible to edit the CTT file and manually enter the alternate name by using the "=" sign.

So even with this, I am greatly oversimplifying this entire process. It took several days to get right, but in short, it's now possible to add the screen shake textures when combining, upscale/rework the combined image, then crop out all the necessary textures when splitting. Hundreds of man hours cut down to a matter of minutes.

Last but not least, an option was added to Split Combined Multi-Texture to batch combined images found in the Input Path. The images must have a CTT file included or they will be ignored. This is because the various properties stored in the CTT files are required to properly split the images.

Spoiler: (Show Spoiler)

While all of this is pretty niche, it should greatly help anyone who wants to retexture the N64 VC versions of Ocarina of Time and Majora's Mask. And that's all for now!

[Florian199607073]

I'm new to the forum so have pity. Where can I ask for help if I want to create my own texture pack? I would like to create a texture pack for pokemon xd gale of darkness and have already started with it, I have dumped over thirty-five thousand textures and I will upscale all textures in waifu2x coffee with the ultra preset (esrgan) and 8x scale and then with the custom- convert texture-tool i convert them to dds (bc7), it would be nice if someone either helps me or tells me what I can do better, if someone agrees to take care of me, then I will show the already upscaled textures to them so that they can work together

[Swift016]

Update: I have reinstalled CUDA and pretty much everything else in the chain. Even now, running SFTGAN seems to take an inordinate amount of time. I am doing a test batch of 15 textures (one 4096, the rest are 2048) and after 23 minutes the progress bar is at 6% and has spit out only one image. Is this normal behaviour?

Original post:

I can't get SFTGAN to run at all. I have textures in PNG format that I ran through ESRGAN a while ago that I'd like to put through SFTGAN as well.

I am able to run ESRGAN without issues (using [Joey's version of upscale.py](https://github.com/JoeyBallentine/ESRGAN)), so I know that I have the correct versions of PyTorch, CUDA, etc. I have not been able to get test_segmentation.py and test_sftgan.py to work on any images other than the default brick wall and cat photos that it comes with (the console spits out some Python errors that I can share). I have given CTT a try to see if it's an issue with having the right version of everything in the chain but have had no success; CTT hangs at "Filtering Textures" when I try to run the SFTGAN upscale option (have tried 1x and 2x). Right now I am at 18 minutes elapsed and no progress.

[ghyujkilop]

What's your way to extract .arc files? I've tried extracting the audiostr.arc file (which was already extracted from The Urbz: Sims in the city DATA folder using its ISO) where I used Freearc & Wiiexplorer to attempt extraction but I pretty much get nowhere. Anybody else have success or wanna try it? My purpose is to upscale dem textures.