Custom Texture Tool PS v50.1

[Bighead]

(10-02-2021, 01:03 AM)Swift016 Wrote: Update: I have reinstalled CUDA and pretty much everything else in the chain. Even now, running SFTGAN seems to take an inordinate amount of time. I am doing a test batch of 15 textures (one 4096, the rest are 2048) and after 23 minutes the progress bar is at 6% and has spit out only one image. Is this normal behaviour?

Original post:

I can't get SFTGAN to run at all. I have textures in PNG format that I ran through ESRGAN a while ago that I'd like to put through SFTGAN as well.

I am able to run ESRGAN without issues (using [Joey's version of upscale.py](https://github.com/JoeyBallentine/ESRGAN)), so I know that I have the correct versions of PyTorch, CUDA, etc. I have not been able to get test_segmentation.py and test_sftgan.py to work on any images other than the default brick wall and cat photos that it comes with (the console spits out some Python errors that I can share). I have given CTT a try to see if it's an issue with having the right version of everything in the chain but have had no success; CTT hangs at "Filtering Textures" when I try to run the SFTGAN upscale option (have tried 1x and 2x). Right now I am at 18 minutes elapsed and no progress.

Unfortunately this is normal, SFTGAN is incredibly slow especially if you have to divide the image into segments. It's a bit hard to explain, and this is how I understand it anyway. ESRGAN upscales an image using the original size, so if you are upscaling 256x256 to 1024x1024, ESRGAN will work with the 256x256 image and output a 1024x1024 image. SFTGAN on the other hand, works on the upscale. CTT-PS upscales the image using point filtering by the upscale amount before being fed to SFTGAN. So the input texture into SFTGAN is the 1024x1024 texture. Basically, SFTGAN is working with 4x the pixels as ESRGAN.

(10-03-2021, 08:12 AM)ghyujkilop Wrote: What's your way to extract .arc files? I've tried extracting the audiostr.arc file (which was already extracted from The Urbz: Sims in the city DATA folder using its ISO) where I used Freearc & Wiiexplorer to attempt extraction but I pretty much get nowhere. Anybody else have success or wanna try it? My purpose is to upscale dem textures.

I have no idea what ".arc" files are, and this has nothing to do with this tool. You would be better off asking in the general help or somewhere else.

[Bighead]

New beta version available:
https://www.mediafire.com/file/jdw32k0oq1qdzcd/Custom_Texture_Tool_PS_v50.0.b02.7z/file

First thing I should mention is that the script now comes in a double compressed file, first it is zipped, then it is 7-zipped (zip file inside a 7z file). Mediafire has been flagging my script as a virus, and this seems to work around it, for now at least. Discord is also flagging it as well when trying to share it there. And I promise everyone that it is not a virus, it has zero malicious code, and all of the code is available for all to see simply by right clicking it and editing it with any notepad (I use Notepad++). I do apologize for this inconvenience, but there's not much I can do about it since corporations want to police the internet and control the flow of software in the name of "security". I'll reserve the rest of my thoughts on that...

A major bug has been fixed when working on The Legend of Heroes: Trails of Cold Steel textures. There was a bug with updating the "asset_D3D11.xml" file that caused newly created PKG files to crash. If files in the archive shared the same name as one of the textures, it would also replace those in the XML file. While this is a rare situation, I'm sure there are more than one PKG file that could have this issue. More on that can be read here:
https://steamcommunity.com/app/538680/discussions/0/2579854400755735260/?ctp=23#c3083268548817743634

Windows 11 is now officially out, and I decided to make the switch. I figured I would be the guinea pig for my friends, but I also did it to make sure this script works with it without issue. So far, so good.. I had to add a way to detect Windows 11 in the script, and everything works except Microsoft has removed the registry entry that made it possible to open PowerShell scripts with a double click. Unfortunately, it means I now disable that option in the options menu because all it will do is add an entry to the registry that does nothing. This won't hurt anything by adding it, but it's pointless to do since it now does nothing, so... yeah. Hopefully I figure something out, because its really annoying to have to right click and choose "Run with PowerShell" every time I want to run it.

Spoiler: (Show Spoiler)

Finally I added the file format "ICO" to Convert Textures to Another Format while in Expert Mode. This allows generating an icon from an image file. The script will pack the icon with multiple sizes of the image into the icon at resolutions of 16x16, 24x24, 32x32, 48x48, 64x64, 128x128, and 256x256. For best results, use an image that is 256x256 or larger. Support for this is still very immature, and I haven't tested it much, but it does seem to work if you want to quickly generate an icon for something.

[stealthii]

I've been using this tool on Windows 11 and so far its flexibility and support is great! Awesome work.

I've one tool with an issue though. The Material Map Generator doesn't seem to execute properly, failing to run the main python loop. From the logs when run with CTT-PS debug:

Code:

C:\Users\dprei\AppData\Local\Programs\Python\Python39\python.exe: can't find '__main__' module in

'C:\\Users\\dprei\\Apps\\CTT-PS\\CTT-PS Data\\CTT-PS Programs\\Material-Map-Generator'
    + CategoryInfo          : NotSpecified: (C:\Users\dprei\...-Map-Generator':String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
    + PSComputerName        : localhost

The tool runs fine directly from the command line:

Code:

PS C:\Users\dprei> cd '.\Apps\CTT-PS\CTT-PS Data\CTT-PS Programs\Material-Map-Generator\'
PS C:\Users\dprei\Apps\CTT-PS\CTT-PS Data\CTT-PS Programs\Material-Map-Generator> C:\Users\dprei\AppData\Local\Programs\Python\Python39\python.exe .\generate.py --help
usage: generate.py [-h] [--input INPUT] [--output OUTPUT] [--reverse] [--tile_size TILE_SIZE] [--seamless] [--mirror]
                  [--replicate] [--cpu] [--ishiiruka] [--ishiiruka_texture_encoder]

optional arguments:
 -h, --help            show this help message and exit
 --input INPUT         Input folder
 --output OUTPUT       Output folder
 --reverse             Reverse Order
 --tile_size TILE_SIZE
                       Tile size for splitting
 --seamless            Seamless upscaling
 --mirror              Mirrored seamless upscaling
 --replicate           Replicate edge pixels for padding
 --cpu                 Use CPU instead of CUDA
 --ishiiruka           Save textures in the format used in Ishiiruka Dolphin material map texture packs
 --ishiiruka_texture_encoder
                       Save textures in the format used by Ishiiruka Dolphin's Texture Encoder tool

Other python tools such as ESRGAN and SFTGAN work fine, from similar directory structures.

EDIT: I was able to fix this - the CTT-PS tool sets 'MatMapGenPath' as the directory, similar to other Python tools, but assumes later in the code that it's a direct string to the 'generator.py' executable.

To maintain consistency with the other Python tools CTT-PS calls, I've updated the MatMapGen function to use this variable as the path directly, and call 'generator.py' from within that directory.

Code:

--- Custom Texture Tool PS v50.0.b02.ps1    2021-10-13 12:00:10.000000000 +0100
+++ Custom Texture Tool PS v50.0.b02.fix.ps1    2021-10-13 15:52:39.000000000 +0100
@@ -3505,14 +3505,11 @@
    # Everything that is passed to Waifu2x needs to be copied over.
    Param ([string]$Python, [string]$MatMapGenPath, [string[]]$Arguments)

-    # Get the python script as an item to extract the path.
-    $MatMapGenDir = (Get-Item -LiteralPath $MatMapGenPath).DirectoryName
-
    # The python script will look for external files so move to it's home folder.
-    Push-Location -LiteralPath $MatMapGenDir
+    Push-Location -LiteralPath $MatMapGenPath

    # Run the script through python.
-    & $Python $MatMapGenPath $Arguments
+    & $Python 'generate.py' $Arguments

    # Return to wherever we were.
    Pop-Location

[Bighead]

You are right, good catch. I think at some point the stored directory was pointing to the full path of "generate.py", but maybe not. I've tested it before, and generated material maps several times, so I'm not sure how it was working at some point. Either way, I'll have this fixed in the next version.

Edit: Looked back at the beta version it was added (47.0.b30) and the script always stored it this way. Thinking about it, I think I manually entered the path into the INI file when I first added it to the script. I usually do it this way so I can test the tools before I add the ability to add the path via "Options". So it was always working for me despite being wrong since "generate.py" was a part of my stored path. I probably meant to change that code you linked above at some point, but completely forgot. Anyway, it's been fixed now for everyone else, no need for you to update since you already fixed it locally. Thanks for the head up.

https://www.mediafire.com/file/uxxgzotjqf2mcjs/Custom_Texture_Tool_PS_v50.0.b03.7z/file

[Bighead]

New beta version:
https://www.mediafire.com/file/jaifnxc2hfp0b1f/Custom_Texture_Tool_PS_v50.0.b04.7z/file

For the longest time, the script had a huge bottleneck when counting textures before the loop begins. A HUGE bottleneck, so bad that I included an option to skip the count altogether if dealing with a large number of files. As an example, to count 130k files took nearly two and a half minutes! The code that counts files hasn't really changed much since it was first created. Since then I've learned a great deal more, so I thought it's finally time to see if I can write something that is a bit more performant. Fixing this has always been a long term goal that kept getting put off, but I figured I would finally tackle it.

As for why it was so slow, I used to blame the amount of filtering that is done when counting. For example, counting textures omits files that are not processed, but rather supplement the main texture such as files that are not textures, mipmaps, material maps, and materials (that combine into material maps). Depending on the operation, it will also look for other file types such as (.pkg, .pka, .ctt). The script also skips paths and files that contain the tilde (~) character, this has been a feature since the beginning. So I always assumed all of this was why it was slow.

Without going into too much detail, it turns out it was just bad code. I found an awesome method for file enumeration on this post which allows specifying multiple filters which greatly sped up the process, but even with that it was still slow. The main problem was storing each sub-folder in an array then adding them together. This is extremely slow in Powershell. I tinkered with quite a few different methods and slowly I managed to get more and more speed. How much?



Counting/analyzing 130k files used to take almost two and a half minutes. After the big rewrite using the new enumeration method, I got that down to 17 seconds. A few more tweaks shaved off 4 more seconds. Finally, I found the last real bottleneck and its down to 4 seconds! This is around 38 times faster than before. While most people probably won't be processing this many files in one go, this should finally prevent the GUI from freezing up when pressing "Start" when working with a few hundred or thousand textures.

[Bighead]

New beta version. There's still a few things left I want to fix, but it will take me some more time and there's a number of bugs in the last beta that I feel should be addressed sooner rather than later.
https://www.mediafire.com/file/6n4p6fd6gnux48p/Custom_Texture_Tool_PS_v50.0.b05.7z/file

First thing to mention, I created some new icons for the CTT-PS windows. Unfortunately at 20x20 pixels, they look almost identical to the old ones. Maybe I'll find some use for them in the future.


Next up, using the new EnumerateFiles and EnumerateFolders methods over using Get-ChildItem to build collections has been applied to many other places in the script. I had over 50 instance of Get-ChildItem, that has been knocked down to 14. The most important function this has been applied to was the Master Loop. This has given another large speed increase overall when working with hundreds/thousands of textures. Replacing all these cases has the potential to create new problems if I missed anything, so please be aware of any bugs and let me know if you find them. The new methods create collections only using the entire path, while Get-ChildItem creates an "Item" that allows accessing properties such as Name, BaseName, Extension, Directory, etc... Places where it was replaced, the solution is to use Get-Item so these properties can be referenced. I think I managed to find them all, but there is always the chance I missed something.

Counting textures has a few fixes, now other file types will now be properly counted such as BMP, JPG, TIF/TIFF, and TGA. I also fixed an issue that could generate some false counts. An example would be if the path to the file has a folder with the name These_Mipmaps, the "_Mip" section of that would trigger the detection for a mipmap texture and mess up the counts. Also fixed an issue with counting "phyre" textures when in Phyre Mode. I can't remember what the issue was specifically but they are now counted correctly.

Sizes can now be set for icons when creating them. Separate the sizes using a comma. Each value entered into the textbox is treated as a square, so 32 is [32x32], and 64 is [64x64]. The maximum size of an icon is 256x256. Invalid sizes/characters are ignored/auto-fixed.



In the image above you may notice a new option Disable Sharpening. This can be found in all three of the texture creation operations: Convert/Rescale/Resize. CTT-PS has always applied a slight sharpening filter to textures whenever they are resized for any reason, which includes generated mipmaps. Over the years, options to enable/disable/adjust the sharpening came and went, and eventually I created a formula that adjusts the sharpening based on the amount of change in resolution. I deemed this "good enough" to apply to all textures all the time. Well, I changed my mind on this again. When creating some icons, the sharpening filter had a negative impact on how they appeared. So now it can be disabled again, and will probably stay for good this time. It will be up to the user if they want this auto-sharpening which works well on stuff like environment textures, but may not work well on smaller, user interface textures.

Next, some bugs have been fixed with the Search Flags/Flag Removal options.



One such fix is to merge folders when flags are removed if there is a folder that already has a name that matches the folder having a flag removed. For example, a folder is named Interface, and in the same path, a temp folder is created to set a flag Interface_ARGB8. After the flag is removed, the contents of "Interface_ARGB8" will be merged with "Interface", instead of ending up in the folder itself "Interface/Interface_ARGB8".

The last flag removal fix is to remove flags from mipmaps. Flags can be applied directly to textures, such as tex1_64x64_7bb8d8c77712c8a7_14_BC7.png. Generated mipmaps would be created with the flag, but it was never removed from them. There is still a few issues with flag removal, if too many flags are used on the same path, some may not get removed. Up to three flags deep should work, such as "C:\Textures\Environment_BC7\Woods_BC3\sometex_BC1.png", but really there is a limit to how far this can go. It should not be abused in this way to begin with. Flags can be applied to a single folder with a name like "_ARGB8", all textures in that path can be dumped into it, and when flags are removed, that folder is removed and all contents merged back one folder.

Next, a few beta versions ago, I disabled the option to launch PowerShell scripts with a double click on Windows 11. It turns out that this also caused a bug on Windows 10. Every time the script is launched, it disables the registry hack on Windows 10 despite still being checked. This was caused due to some bad code ordering, and has now been fixed. I still don't have a workaround to launch PowerShell scripts with a double click on Windows 11, but I have some ideas brewing.



Finally I have completely rewritten the material/material map generation code. This is still a WIP, and has not been fully tested. At the moment, "emissive" textures are not supported (material with "_lum" flag) but will be in the next beta. Ishiiruka Tool has changed quite a bit since I last touched this code, and I haven't updated CTT-PS to reflect those changes. There are a few bugs in the Ishiiruka Tool, most of them I can work around. I will dig a little deeper into these in a bit.

The operation to generate material textures using Material Map Generator now has a new option to convert directly to material maps. This removes the "middle man" of having to convert to materials, then convert those materials into material maps. This requires a valid path to Ishiiruka Tool, the option will be grayed out if it can't be found.



Onto the bugs with Ishiiruka Tool. The first bug to mention is that certain commands that are entered via command line seem to be broken. The option "-savecolor" seems to act as the "-usebc7" command, while the latter doesn't seem to do anything. Fortunately the other commands all seem to work as intended, and I have added both in the script. If Ishiiruka Tool ever has this bug fixed, everything should still work as expected.

The second bug I noticed is that Ishiiruka Tool is setting bad flags in the DDS headers, for both BC3 and BC7 textures. This has always been the case. While this does not affect how they work, it is still not correct and many image editing programs will not even recognize them as textures. So after they are generated, CTT-PS will apply the correct flags in the header, and they will now be recognized as images in any program that supports them.



The third bug I noticed is that since "-savecolor" is acting as "-usebc7", the color texture is never actually saved. So I work around this by converting the color texture manually. Ishiiruka Tool doesn't generate mipmaps down to 1x1, Tino always saw this as "wasteful". While I respectfully disagree, the color texture should have the same number of mipmaps as the material map. So after the material map is generated, the number of mipmaps is counted, and only that number of mipmaps is applied to the color texture when created.

The fourth bug I noticed is when working with existing material maps. Once materials are generated into a material map (with the ".mat") flag, I rely on Ishiiruka Tool to apply additional work to them. For example, you want to downscale a pack with these textures to reduce VRAM consumption. For the most part, Ishiiruka Tool handles them well, and creates downscaled material maps, except when the material map is BC7. Unfortunately, it does not currently work with BC7 material maps, but it does work with PNG and BC3/DXT5 material maps. As a workaround, if a BC7 material map is found when regenerating a pack, it will simply copy the texture and material.

Last thing to mention is the script currently will not work with "_lum" materials. I think it might work and generate the material map, but I have not tested it at all. IIRC, the emissive texture is applied to the color texture. This could be a huge problem since the "-savecolor" texture is not working as expected. Again I have no idea, I haven't tested it at all since Material Map Generator does not create these textures. Next beta I will have more information and hopefully some fixes to this. I will also let Tino know of the bugs with his tool in his Ishiiruka thread as soon as I can figure out all the issues and come up with a comprehensive list.

[Snowblind]

Hello Bighead! Thanks for this GREAT tool. I really liked it.

I was wondering why you didn't include in the Waifu2x-Caffe tool the CUnet model and the option to choose between "Denoise & Magnify" and "Magnify & Auto Denoise" conversion modes. Thank you!

[Bighead]

(11-04-2021, 11:01 AM)Blindsnow Wrote: Hello Bighead! Thanks for this GREAT tool. I really liked it.

I was wondering why you didn't include in the Waifu2x-Caffe tool the CUnet model and the option to choose between "Denoise & Magnify" and "Magnify & Auto Denoise" conversion modes. Thank you!

It's been a long time since I updated waifu2x, loaded its GUI, or read its documentation. I'll check it out soon, and if I can I'll add that stuff to the next release.

[Snowblind]

(11-04-2021, 07:05 PM)Bighead Wrote: It's been a long time since I updated waifu2x, loaded its GUI, or read its documentation. I'll check it out soon, and if I can I'll add that stuff to the next release.

Thank you, Bighead! I appreciate it.

If you need help testing something, let me know.

[Bighead]

New beta version, mostly just waifu2x changes:
https://www.mediafire.com/file/nz6rwd7asa7c3fb/Custom_Texture_Tool_PS_v50.0.b06.7z/file

The script now comes with a launcher written in C# that simply launches the script with PowerShell. Simply keep it in the same folder as the script, and it will launch the first version it finds. It took me several hours to figure out how to write a few lines of code to pull this off, but it had the added benefit that Admentus was able to also include a variation of it with his Patcher64+ Tool which has now had a few days of testing. Unlike previous attempts of converting PowerShell scripts to executable files, since this is written in C#, most anti-virus programs probably won't immediately start screaming about it, although Windows Smart Screen will probably pop up asking if you want to run it. Over time that may go away if enough people report that it's not a virus here:
https://www.microsoft.com/en-us/wdsi/filesubmission



So you may wonder what purpose it serves? Well, there are a few advantages it can offer:
- It was created as a work-around since Windows 11 prevents the registry hack to launch PowerShell scripts with a double click.
- A shortcut can be created to the launcher, which can be placed on desktop or pinned to the start menu or taskbar.
- It bypasses the current setting for PowerShell execution policy for those who don't feel comfortable changing it just to run this script.
- It has an icon. Wheeee.

Edit: Figured I'd post it just in case anyone wanted to see the source code behind it. It's nothing too special but it gets the job done and should be easy to modify for other purposes. The included "compile.ps1" Powershell script uses "csc.exe" which should be found in almost all Windows installations with .NET Framework.
https://www.mediafire.com/file/x58hpbg19h9hdam/CTT-PS_Launcher_Source.7z/file

And the CTT-PS icon files I created that I talked about here. Maybe you want to use a different color (or something).
https://www.mediafire.com/file/4w8a2xif14gieuu/CTT-PS_Icons.7z/file

Aside from that, I made the requested changes to Waifu2x. Two models that were missing have now been added: 2D Illust (UpResNet10) and 2D Illust (CUnet). Both supposedly offer better quality than 2D Illust (UpRGB), with the CUnet claimed to be the best choice. I also changed the conversion mode names to mostly match how they are found on the waifu2x-caffe GUI, with the difference of using "Scale" instead of "Magnify". No reason other than I prefer the former and it's more consistent with the rest of CTT-PS. I also added the missing conversion mode: Scale & Auto-Denoise, and increased the size of the window a bit to better fit the options.



That's it for now. I was going to hold off on a new beta version until more work on material maps was done, but I am currently hoping/waiting for Tino to fix the bugs I mentioned in his Ishiiruka Tool. At the moment the options are mostly usable, it is possible to create material maps from materials if using an older version of the tool (0.9.10), as opposed to trying to use (1.0.0) which creates material maps at the resolution the textures were dumped with.