Custom Texture Tool PS v50.1

**Bighead** · (This post was last modified: 11-07-2019, 05:22 PM by Bighead.)

This beta adds support for TIF/TIFF and TGA. It is not complete, but should allow converting between the formats.

http://www.mediafire.com/file/akhpz3yh1okiafi/Custom_Texture_Tool_PS_v44.1.b1.zip/file

Some things to note:
- Input images can now be TGA, TIF, or TIFF. All operations now support them, but only ImageMagick upscaling filters are currently supported (Point/Cubic).
- Output images can now be TGA or TIFF when in Expert Mode. Outputting TIF is not possible through GUI, although I can add it if necessary.
- JPG was also moved to Expert Mode for convert/rescale. IIRC Dolphin doesn't support JPG anymore, nor should it be used for texture packs anyway.
- Multi-image TIF/TIFF files may not be currently supported or work correctly.

Things to look out for:
- Any bugs or unexpected behavior. There are about 6 places in the script that reference image formats, and I'm known to break stuff.
- ImageMagick by default flips TGA files. I have a work-around in place (-flip parameter), but I don't know if this was local to me, a bug, or expected behavior.

@Ridiculous_Death: After reading your post more closely, I have some questions and some things to mention.

(11-06-2019, 12:22 AM)Ridiculous_Death Wrote: As 32bit transparency mask is damaged on converting/upscaling with PNG...

I'm not sure what you mean by this, can you explain in a bit more detail? Whether you mean a separate texture for the mask, or a texture with alpha, all PNG textures are created with 32-bit depth. An example image or screenshot of the damage would be useful.

(11-06-2019, 12:22 AM)Ridiculous_Death Wrote: ...and textures becoming unusable with missing parts.

Again not sure what you mean by "missing parts". Again an example image or screenshot would be very useful.

And one thing to keep in mind: this tool is limited by the programs it uses. The current beta is as far as it goes for natively supporting TIFF and TGA. ImageMagick, Compressonator, and TexConv all natively support both formats, so its quite simple to convert between them. Converting/Rescaling operations should now work when converting any format to any format.

As for other the upscaling programs: xBRZ, Waifu2x, ESRGAN, and SFTGAN, I have not tested most of these but I know xBRZ does not support either TIFF or TGA. So in this situation the image must be converted to PNG before upscaling. If the other upscaling programs do not support these formats natively, then the same method needs to be applied. Which means if converting TIFF or TGA to PNG loses data (I don't know if it does), there really isn't much I can do about that.

Last, if you can send me a few TGA and TIFF images so I can get an idea of what you are working with, I would appreciate it!

@djneo: If you're still around, just out of curiosity, have you tried generating those problematic textures as BC7 without mipmaps and testing in-game?
I noticed it has "m" in the name: tex1_1024x1024_m_e96aa72b1a32f7f3_14.png

[Image: UwIYPOD.png]

This may cause shimmering in the distance, but it might just be weird enough to fix the issue. Alternatively, if shimmering is bad, you can try less levels like 1 or 2 and see if that works.

[Image: WHQbFhV.png]

If it doesn't fix the issue, I do suggest lowering the levels even for ARGB8 as each level will add size to the texture, which partially explains why they are so large. You may need to experiment.

Ridiculous_Death · (This post was last modified: 11-07-2019, 07:52 PM by Ridiculous_Death.)

(11-07-2019, 02:10 PM)Bighead Wrote: This beta adds support for TIF/TIFF and TGA. It is not complete, but should allow converting between the formats.

http://www.mediafire.com/file/akhpz3yh1okiafi/Custom_Texture_Tool_PS_v44.1.b1.zip/file

Some things to note:
- Input images can now be TGA, TIF, or TIFF. All operations now support them, but only ImageMagick upscaling filters are currently supported (Point/Cubic).
- Output images can now be TGA or TIFF when in Expert Mode. Outputting TIF is not possible through GUI, although I can add it if necessary.
- JPG was also moved to Expert Mode for convert/rescale. IIRC Dolphin doesn't support JPG anymore, nor should it be used for texture packs anyway.
- Multi-image TIF/TIFF files may not be currently supported or work correctly.

Things to look out for:
- Any bugs or unexpected behavior. There are about 6 places in the script that reference image formats, and I'm known to break stuff.
- ImageMagick by default flips TGA files. I have a work-around in place (-flip parameter), but I don't know if this was local to me, a bug, or expected behavior.

@Ridiculous_Death: After reading your post more closely, I have some questions and some things to mention.

I'm not sure what you mean by this, can you explain in a bit more detail? Whether you mean a separate texture for the mask, or a texture with alpha, all PNG textures are created with 32-bit depth. An example image or screenshot of the damage would be useful.

Again not sure what you mean by "missing parts". Again an example image or screenshot would be very useful.

And one thing to keep in mind: this tool is limited by the programs it uses. The current beta is as far as it goes for natively supporting TIFF and TGA. ImageMagick, Compressonator, and TexConv all natively support both formats, so its quite simple to convert between them. Converting/Rescaling operations should now work when converting any format to any format.

As for other the upscaling programs: xBRZ, Waifu2x, ESRGAN, and SFTGAN, I have not tested most of these but I know xBRZ does not support either TIFF or TGA. So in this situation the image must be converted to PNG before upscaling. If the other upscaling programs do not support these formats natively, then the same method needs to be applied. Which means if converting TIFF or TGA to PNG loses data (I don't know if it does), there really isn't much I can do about that.

Last, if you can send me a few TGA and TIFF images so I can get an idea of what you are working with, I would appreciate it!

@djneo: If you're still around, just out of curiosity, have you tried generating those problematic textures as BC7 without mipmaps and testing in-game?
I noticed it has "m" in the name: tex1_1024x1024_m_e96aa72b1a32f7f3_14.png

This may cause shimmering in the distance, but it might just be weird enough to fix the issue. Alternatively, if shimmering is bad, you can try less levels like 1 or 2 and see if that works.

If it doesn't fix the issue, I do suggest lowering the levels even for ARGB8 as each level will add size to the texture, which partially explains why they are so large. You may need to experiment.

Wow!Thank you for such a quick response and changes!I will now test a new version, but I'll show you what happens in older versions. I think the problem was that "middle man" in conversions\upscalings was PNG(?) and it can not save transparency mask(alpha channel) in the right way. This is the simplest texture(this is normal map but its happening on any texture with solid "opaque" alpha) to show this "bug" but it'll happen on any texture with 32bit transparency it will cut out parts or corrupt the texture.

Ridiculous_Death · (This post was last modified: 11-07-2019, 07:53 PM by Ridiculous_Death.)

Also, I've tested other texture, and it was corrupted on older versions of the script(sadly I dot remember exact version number), with version 44 when upscaling from dds to dds it's now working correctly(maybe its linked to versions of compressonator\imagemagick?)!But with dds to png it stayed the same.
Now I'll test beta.

Ridiculous_Death · (This post was last modified: 11-07-2019, 09:02 PM by Ridiculous_Death.)

Here are the results of the test with beta converting. From original dds to tga it still cuts "opaque" alpha, fine with transparent. Tiff strangely behaves the same as PNG and cuts alpha channel entirely(but it is preserving alpha channel same as TGA when I use Photoshop). When converting from TGA dds gets fliped as you said, and others behave the same. Link to original textures( normal is from Condemned and weapon is from Stalker, but no matter the game results are the same). Also here's original files Weapon Normal. Also there is tool called IEU with similar functionality as yours, and it leaves opaque alpha channel(as in door normal texture) alone when converting but it has far worse conversion quality to DDS than your tool with compressonator.

**Bighead** · 11-07-2019, 09:42 PM **#895**

Thanks, this should be enough info. I think I may understand what is going on. Hopefully I'll have something to test later on.

Ridiculous_Death · (This post was last modified: 11-07-2019, 11:47 PM by Ridiculous_Death.)

(11-07-2019, 09:42 PM)Bighead Wrote: Thanks, this should be enough info. I think I may understand what is going on. Hopefully I'll have something to test later on.

I did some more tests if it can help, with beta44 and standalone texconv and compressonator, and results are somewhat funny. Mimaps are NOT generated and using BC1\BC3 format.
Standalone Compressonator 3.1.4 : Tiff>DDS=texture chopped TGA>DDS=texture as original
44BetaCompressonator 3.0.3: TIFF>DDS=texture as original TGA>DDS=texture gets chopped,image flipped.
Standalone texconv(same as in script): TIFF>DDS=texture as original but colors are corrupted TGA>DDS=texture as original
44Beta texconv: TIFF>DDS=texture as original but colors are corrupted TGA>DDS=texture as original

Also tested in reverse in beta44. Original DDS>TGA and TIF, and in this case no matter what I use (compress,dxtex,imagemagick) texture always result in chopped parts.
But Standalone Compressonator 3.1.4 : DDS>TGA=texture as original DDS>TIFF=chopped, alpha channel deleted like in png.
Standalone texconv(same as in script) : DDS>TGA=texture as original DDS>TIFF=chopped, alpha channel deleted like in png.

**Bighead** · (This post was last modified: 11-08-2019, 10:15 AM by Bighead.)

Thanks for all the info, it's very helpful. TIFF seems pretty simple to work with (it's a plus that .NET supports it as a bitmap), but TGA is a bit more complex. There's a few problems with my past approaches: things that worked with the image formats that were supported, aren't always working with TIFF/TGA so I have to work on changing some things. There has always been a very blurry line between "image has alpha channel" vs. "image has transparency" in my script. While this didn't really matter for PNG/DDS, it's increasingly important to distinguish the differences here. For example, there are places that may strip the alpha channel if the image doesn't actually contain transparency.

It may take me a few days/tries to get everything right (mostly due to lack of free time/energy). I've been reading deeper into the TGA and TIFF formats, and noticed there are a few nuances to these images and strange behaviors in the programs that support them, so I need to run many tests to figure all of that out. I can make a good guess that image formats were not designed around the idea of having a convenient and automated way of converting between them, and relying on the default behavior of the available tools to get what we want every time is dubious at best.

**Bighead** · (This post was last modified: 11-11-2019, 07:12 AM by Bighead.)

@Ridiculous_Death: Can you please test converting between formats with this version?
~~http://www.mediafire.com/file/k49y6i6yvholnx0/Custom_Texture_Tool_PS_v44.1.b2.zip/file~~
http://www.mediafire.com/file/07qve8pqm3mzu5h/Custom_Texture_Tool_PS_v44.1.b3.zip/file

Alpha channels should hopefully now be preserved between all formats, and almost all transparency checks were removed. The only time an actual transparency check will ever be performed is if a user wants to remove alpha channels from non-transparent images using the respective operation, or if excluding images that contain transparency (this is not the same as excluding images with an alpha channel).

TGA images have been a tricky beast. Pixels can be in any number of orientations (topleft, topright, bottomleft, bottomright, righttop, rightbottom, lefttop, leftbottom, probably more...). Currently I check only for "TopLeft" and force a flip when this orientation is found. I only have a few different kinds to test, but so far this has resulted in the proper orientation in all my test cases when converting to/from TGA and any other format (PNG, TIFF, DDS BC3, DDS BC7, ARGB8, RGBA8, and so on).

Upscaling filters may or may not work, I have not tested them yet. For now I am mostly interested if basic conversions are working as intended.

Edit 10/10/19: Updated to beta 3. I reworked much of the texture generation logic to be a bit more friendly towards additional image formats that are not DDS (as DDS formats have their own special functions for everything). This also inadvertently fixed a few other issues when converting between formats, and allows direct conversion without using PNG as a middle-man.

TGA continues to be a pain, there are several issues when converting the various forms of it to other formats. Even converting TGA to TGA does not work without issues depending on the type. Converting to PNG does seem to produce a correct result every time, so in the end, PNG may need serve as a mediator. Whether or not this loses some form of "data" I have no idea.

Spoiler: (Show Spoiler)

Unrelated: I discovered I didn't really fix the TexConv BC7>PNG lighter/darker issue, it was just not using it anymore and falling back to Compressonator. After deeper analysis it seems all the pixel data is correct, the issue may lie in the header data. So as a workaround, the resulting PNG of TexConv converting from DDS BC7 is loaded into .NET as raw pixel data and then rewritten as PNG. Fortunately this is very fast even for large images, so the time added is mostly negligible, and this should hopefully finally "fix" this issue once and for all.

I will continue to work on this until I feel that I am getting the results that I expect. So if you are still around, and don't want to test anything yet, that is fine. The upcoming beta 4 will probably be the one actually begins to get everything right in one way or another.

Ridiculous_Death · 11-11-2019, 11:23 PM **#899**

Hi!Thank you for working on this! I did some tests with beta 2 and 3. Mimaps are NOT generated and using BC1\BC3 format.
Beta 2:
Compressonator: TIFF>DDS=textures with all alpha types as original TGA>DDS=images with "transparent alpha" slightly corrupted, texture with "opaque" alpha chopped
texconv: TIFF>DDS=texture as original but colors are corrupted TGA>DDS=textures as original

Original DDS>TGA and TIF, and in this case no matter what I use(compress, dxtex, imagemagick) in debug I see that it's always using Imagemagick.
DDS>TGA=texture with "opaque" alpha chopped DDS>TIFF=chopped, alpha channel deleted like in png.

Beta 3:
Compressonator: TIFF>DDS=textures with all alpha types as original TGA>DDS=texture with "opaque" alpha chopped
texconv: TIFF>DDS=texture as original but colors are corrupted TGA>DDS=textures as original

Original DDS>TGA and TIF, and in this case no matter what I use(compress, dxtex, imagemagick) in debug I see that it's always using Imagemagick.
DDS>TGA=texture with "opaque" alpha chopped DDS>TIFF=chopped, alpha channel deleted like in png.

TGA>TIFF=alpha channes deleted, images flipped upside down TIFF>TGA=opaque alpha chopped

Some other things I've noted. If I convert tga texture to tiff and then use that TIFF with compressonator to make DDS: if I used photoshop to convert(no compression, interleaved rgbrgb) > compressonator works OK. If I use xnconvert(no compression) debug shows "identify: Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples.. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/1029." and then passes work to imagemagick(?). Mostly happns with textures that contain alpha.

At the time, in most scenarios, TIFF in the script behaves more like PNG after conversions(does not store alpha channel). And with "opaque" alpha channels> when there's a problem(compressonator tga>dds, texconv tiff>dds), something in the pipeline does not recognize parts of the image covered by "opaque" alpha and just deletes it(or converts to black color?)

**Bighead** · 11-15-2019, 02:24 PM **#900**

Sorry I had to take a break from this due to lack of time (and really frustration). Many of the changes I made will most likely be scrapped as the idea is apparently not working as intended. My goal was to try to get images to look identical to the original, but I apparently have a lack of understanding of how TGA and TIFF files work. On top of that, while all the programs this script supports can make both, they all have very different results so I have no idea what is "right".

TexConv/Compressonator can create corrupted images when working with certain types of TGA. I also get corruption when working with some TIFF images. ImageMagick appears to get all of them right, but only when converting to PNG and it is cropping alpha channels? I have no idea why TGA has so many orientations and the impact it has if it is changed. I have very little understanding of TIFF images in general. It also doesn't help that I have zero test cases (are these two image formats used in games?), so I have no idea if the images created are as intended.

Since I've never had to work with these image formats before, and since they've been around for awhile, I blindly assumed the programs would just know what to do when going between formats. I guess there is a lot I just don't understand. This is the first I've seen "interleaved rgbrgb" so I don't don't know what that is, the point of it, or what advantages it has. I have no idea what "opaque" alpha is, do you mean a value of 255 for all pixels in the alpha channel? In that case, what would be the point of an alpha channel? I'm not sure what you mean by "TIFF behaves like PNG (does not store alpha channel)" as PNG can have an alpha channel with a separate alpha value per pixel.

I'm not a graphics artist outside of hobby (which I basically lost interest in), nor am I a programmer, so I don't have a deeper understanding of the image formats and terminology used. This script is first and foremost just a frontend for the programs it supports (with a few of my own twists to manipulate images). The only format I know well is DDS, and that is only because I spent months learning and experimenting with it in my free time. And that actually has great program support, but still had many issues I had to work around. I say all this because in the end, getting this to work may be completely over my head. At least for now.