Custom Texture Tool PS v50.1

[Ridiculous_Death]

(11-15-2019, 02:24 PM)Bighead Wrote: Sorry I had to take a break from this due to lack of time (and really frustration). Many of the changes I made will most likely be scrapped as the idea is apparently not working as intended. My goal was to try to get images to look identical to the original, but I apparently have a lack of understanding of how TGA and TIFF files work. On top of that, while all the programs this script supports can make both, they all have very different results so I have no idea what is "right".

TexConv/Compressonator can create corrupted images when working with certain types of TGA. I also get corruption when working with some TIFF images. ImageMagick appears to get all of them right, but only when converting to PNG and it is cropping alpha channels? I have no idea why TGA has so many orientations and the impact it has if it is changed. I have very little understanding of TIFF images in general. It also doesn't help that I have zero test cases (are these two image formats used in games?), so I have no idea if the images created are as intended.

Since I've never had to work with these image formats before, and since they've been around for awhile, I blindly assumed the programs would just know what to do when going between formats. I guess there is a lot I just don't understand. This is the first I've seen "interleaved rgbrgb" so I don't don't know what that is, the point of it, or what advantages it has. I have no idea what "opaque" alpha is, do you mean a value of 255 for all pixels in the alpha channel? In that case, what would be the point of an alpha channel? I'm not sure what you mean by "TIFF behaves like PNG (does not store alpha channel)" as PNG can have an alpha channel with a separate alpha value per pixel.

I'm not a graphics artist outside of hobby (which I basically lost interest in), nor am I a programmer, so I don't have a deeper understanding of the image formats and terminology used. This script is first and foremost just a frontend for the programs it supports (with a few of my own twists to manipulate images). The only format I know well is DDS, and that is only because I spent months learning and experimenting with it in my free time. And that actually has great program support, but still had many issues I had to work around. I say all this because in the end, getting this to work may be completely over my head. At least for now.

Don't worry, you already did tremendous work! At the least now I have beta 3 to use compressonator with tiffs and texconv with tgas! Before it was impossible to automate it.
I'm too just an enthusiast and learning about all of this. I think "opaque" alpha is used to hide part of the texture without destroying it (in the example 1 first texture with "opaque" alpha has moving door in the hidden part) and transparent is used for translucency, reflections and such(and it needs separate alpha channel to achieve this maybe?). I "mean by behaves like png" that in your script tiff converts alpha channel to png transparency(example 2) thus cuts parts of the image. And in conversion by hand in PS or xnconvert alpha channel remains separated.

Quote:>nor am I a programmer

If that's the case...WOW.

[Bighead]

Thanks, those screenshots alone really puts the problem into perspective. As for the non-programmer thing, maybe I'm being humble but I honestly wish I had the aptitude, the time, but mostly the ambition to learn beyond PowerShell. C# is very similar and I still have major hangups translating code between the two, and most examples I find of tasks I want to perform are usually in C#. The reason this script has come so far is dedication rather than brilliance, and perhaps a bit of passion for the personal challenge. And of course it's nice to have a swiss army knife for images. I think my search engine skills of figuring out issues in my code by entering the right key words, is greater than my ability to code.

Some backstory: This started as a batch script a few years ago to create mipmaps, and evolved from there (I still give credit to degasus because I probably would have never attempted it otherwise). I loved to automate tasks with Batch files since the Windows 98 days, and I loved Blizzard games. I spent an unknown amount of time modding them and making custom maps/scenarios when I was younger (StarCraft 1/2 WarCraft 2/3). Fortunately, WarCraft III JASS (the scripting language I spent the most time with) was eventually extended through a preprocessor called "vJASS", and this is where I learned most of the basic concepts and even some advanced ones. This is relevant because after discovering PowerShell, I quickly realized it has similar syntax, so I was able to pick it up fast. It's pure bonus that PS can make almost full use of .NET which makes the GUI for this tool possible. CTT-PS (powershell) was once CTT (batch), and knowing almost nothing about PowerShell, it only took about 2 weeks to convert the entire script to PowerShell. End rant, I was in the mood for story telling.

Anyway, I wanted to say I'm not gonna give up on this, it's just going to take me longer than I had hoped. My original intention was to run every image format (aside from DDS) through the same functions with arguments to tweak the output. I may have to handle TGA and TIF differently like I do DDS. I need to download more example cases and run even more tests. The biggest roadblock I was facing really is TexConv/Compressonator failing with so many images but working with others even though they are all TGA, just different kinds of TGA. There was also the image being flipped issue, but I think that's only with ImageMagick screwing up the orientation. I spent way too much time trying to create work-arounds which was a bad idea to begin with, so before my next attempt I'm going to make sure I know what I'm doing before going in.

[General_Han_Solo]

Hey Bighead, decided to come back after a long break, and re-render my original PNG pack into DDS again, using v44 of your texture tool (including the latest updates on all the tools that it uses, like ImageMagick and Compressonator, etc). I am noticing some bad color banding. Here's two sky textures to illustrate my point. This is on the Kothlis level for Rogue Leader. Just to confirm, I am converting PNG to DDS using Compressonator, with BC3 compression, and everything is greyed out on Auto. I've included the original PNG file to test.

[Bighead]

Some amount of banding is to be expected with any DDS format, especially BC1-BC3 (DXT1-DXT5). An old example, but still relevant:
https://imgur.com/m3x9NEO

Here is an example of my own conversion of the texture (BC3). Is the banding you are getting any worse than this?
https://www.mediafire.com/file/pjy82rp0r3gl238/tex1_8x256_1674684a527cf56a_6.dds/file

And here is an example of PNG>BC7>PNG (so its viewable without special software).
https://i.imgur.com/FVSzYfZ.png

Is there a reason why you chose BC3 over BC7? If it's because of that issue where it puts a white box around certain textures when loaded in-game, don't forget there is a work-around of putting a nearly invisible pixel in the top-left corner.

Or if you wanted specific textures to be a specific format, you could use the script's DDS Format: Search Flags option, set BC7 as the DDS Fallback, and flag the problematic textures (or the folder they are in) with _BC3 or _DXT5 (for example: tex1_8x256_1674684a527cf56a_6_BC3.png). When they are converted, the flags are automatically removed from both the original textures and the new textures, depending on the setting in Flag Removal. More info can be found here.

Edit: Oh I almost forgot. The latest Compressonator does not work with this script, in fact I couldn't get the CLI version to work with the command line at all. This can explain why the banding is significantly worse, the script is most likely falling back to TexConv or ImageMagick. I think for the next update I'm going to add which program was actually used in the "Message" section of the texture list, so it won't be required to enable the PS console to figure out which one is actually being used.

This is the last working version:
https://github.com/GPUOpen-Tools/Compressonator/releases/tag/v3.0.3707

Alternatively, I have a compiled pack of all the necessary programs here. You can just install it and check Tools Master Path and the script will fill in the rest:
https://www.mediafire.com/file/oczxb5ljbh428b3/CTT-PS_Programs_%25287-7-2019%2529.7z/file

Unless these programs either fall out of compatibility, or newer versions have increased image quality, there is no real need to update them. If and when I think an update would be necessary, I'll update that pack.

[General_Han_Solo]

Thanks Bighead! Will re-install everything as per your recommendation and do some testing. In regards to BC3 and not BC7, well, the option isn't available for me, even when I choose Compressonator? Maybe it's because I don't have the version installed (3.0.3707)? I am able to select Compressonator, but the option for BC7 doesn't highlight. It only goes up to BC3. Will do some tests tonight.

Edit: All I did was re-install Compressonator to the version you recommended, and the banding was nearly gone! However, the initial old DDS render I did is still superior, and I am presuming that's because I utilized BC7 for it. I only was able to do BC3, and I don't understand why the BC7 option is not optional for me.

[Bighead]

BC7 is hidden unless you have TexConv linked to the script as well, as Compressonator can fail for some images. You can also force it visible in the debug options, but that may result in missing textures or mipmaps.

I cover it a bit more here and link to a build of TexConv:
https://forums.dolphin-emu.org/Thread-custom-texture-tool-ps-v44-0?pid=499577#pid499577

[General_Han_Solo]

Bighead, decided to go with your compiled pack (removed everything else) and it's working perfectly! Thanks!

[Bighead]

Great to hear! Just an FYI, you can also remove some things from that folder if you never intend to make use of them, I just include everything to cover all bases. Most of the installs are as minimal as they can be, ESRGAN and SFTGAN make up the bulk (over 1GB because of included models). I also made it so the folder names do not matter, the programs are scanned for in the master path when the script starts. So for example if you wanted a newer ImageMagick, you could delete the "ImageMagick" folder from "CTT-PS Programs", install a newer version to that folder using any install name (like ImageMagick-7.0.9-5-Q16), and the script will still find it without having to change anything. I know programs get updates (ImageMagick, OptiPNG, waifu2x, etc.), and I wanted a seamless way to update them: just delete the old version and drop in the new. And if you already have a 7-Zip install somewhere else, you can delete 7-Zip from the CTT-PS Programs folder and the script will still automatically pick up the "Program Files" install. All in all it's just an alternate way to link the programs to the script.

[Bighead]

@Ridiculous_Death: If you happen to check this thread again, I have rebooted my work on TGA/TIFF and have been slowly making progress. Rather than try to tackle both TIFF and TGA at the same time, I am starting with just TIFF. After much research, testing, and headaches, I have a beta ready.

https://www.mediafire.com/file/4ubiculti1ivf0c/Custom_Texture_Tool_PS_v45.0.b1.zip/file
https://www.mediafire.com/file/yd32ox6ipxzlg14/Custom_Texture_Tool_PS_v45.0.b2.zip/file

Compressonator is not used at all for TIFF images, so TexConv is required. There are several reasons for this:
- There is a bug that can swap red/blue color channels.
- "Opaque" alpha as you call it is lost in the conversion.
- It cannot resize images, so an intermediate format must be used (like PNG). This causes information to be lost.
- TexConv just works.

So TexConv is used when doing the actual writing. I have also extended TIFF support using .NET to handle multi-page TIFF images. An input TIFF will have its internal images read as mipmap layers, and an output TIFF will have mipmaps appended as additional images. This can be disabled by disabling mipmaps. The caveat is that each additional image will be half the resolution of the previous, so multi-page TIFF images can not have multiple pages with the same resolution.

So whether creating TIFF from another format, or creating another format from TIFF, everything should hopefully work correctly now. TGA images can still be processed in this beta, but cannot be created at the moment. I have no additional support for TGA yet, so when it is used as a source image, there is probably still some issues that I haven't worked around yet.

Edit: Update to b2. Fixes some issues with TIFF to PNG as well as other minor bugs I've long overlooked. Also should finally fix the long standing TexConv "darkening" issue that affected images at random. Last version fixed the "brightening" issue when converting BC7 to PNG. So both of those issues should finally be fixed.

[MosAnted]

I'm trying to upscale a few textures to twice their initial size (Skyrim DDS) and everytime I keep getting a "Texture Failed Creation" error. I'm using the full CTT Package install, though it's installed to a different drive (I made sure all the tool paths were correct though)

Edit: Went back to an older version of the script I had around (36.3) and it worked without issue.