New version, enormous speed increase, lots of changes, lots to cover.
v29.0 : Download - Mirror
Alpha vs. Transparency
Let me start out by saying I don't always make the best decisions. Before BC7 support, there were only two DDS formats this script created: DXT1 and DXT5. Because I wanted the script to automatically figure out which format to choose, a method was created to detect if PNG images had an alpha channel. At first, I just checked for the channel, but soon realized this can miss some textures because images can have an alpha channel but not actually contain transparent pixels. So then I used ImageMagick which can scan an image for transparency, but this method was slow. I eventually researched my own method to loop through every pixel and read the alpha value. This was much faster, but of course, it was still slow. This is the method that has remained for ages, and eventually creeped its way into other options of the script.
Much time has passed, my prospects change, new stuff has been implemented, and I no longer think its as important to know whether or not images have transparent pixels. The method for checking for alpha pixels is far too slow, so I have gone back to just testing for alpha channels. This has a few drawbacks: fully opaque PNG images with an alpha channel will be created as 32-bit RGBA instead of 24-bit RGB, and DXT5 images will be created instead of DXT1 if using those formats and the option to auto-select the format. Neither of these situations are important enough for the time it takes to actually test transparency in images using PowerShell. I thought about trying to use memory pointers which is supposedly much faster than copying to byte arrays, but apparently PowerShell does not support this, and for good reason (people could use it maliciously to alter RAM contents). C# is allowed to to this with "unsafe" code, and it's possible to use C# code blocks in PowerShell, but I'd rather not go down that alley especially since my knowledge of it is still limited.
So a bit more important, alpha check is also used to know which quality to pick for BC7 images using Compressonator, so very large opaque images may take longer to generate if they have an alpha channel. It is possible to avoid this by using the option to remove alpha channels from opaque images, which I will talk more about in a minute. The same situation holds true when applying upscaling filters. The seamless method by default is only chosen for images without an alpha channel unless it is forced to all images. This means the seamless method may not get applied to some images that should have it applied to keep the seamless effect. This can be worked around by forcing the seamless method for all textures, or fixing them to not have alpha when they shouldn't. Again, neither of these situations seems important enough to justify the slowdown caused by checking every pixel.
The whole checking for alpha pixels was basically me trying to fix problems created by others, but to be fair, it's an easy mistake to make, and in the real world has little repercussions outside of increased image size. Even I accidentally keep alpha channels in opaque images from time to time. The transparent pixel check is not completely gone from the script, it is still used in the option Remove Alpha Channel From Opaque Textures which has been around for awhile now and has been reworked a bit. So if you absolutely want all textures to be perfect and not contain unnecessary data, this option will do exactly what it says. This only needs to be done once, and it only works for PNG images. This will ensure all the potential issues I mentioned above won't ever be an issue. It's unfortunate that there really is no faster way to test images for transparency, at least not using PowerShell. My method is as fast as it can possibly get, but it's still far too slow.
Now that the transparency check has been replaced with just a check for alpha channels, I decided to do a basic issue scan on the Paper Mario: TTYD pack. This is always my go pack for the reason it has over 4000 textures of various types and sizes, covering a large spectrum of what I need to test.
00:27:15.28 - Before
00:03:06.48 - After
What took nearly a half hour can now be done in 3 minutes. Like I said, I do not always make the best decisions. This time was added to almost every single option including DDS conversions, as a texture is always scanned before anything is done with it. This is a huge speed up everywhere, and I regret not "fixing" this sooner.
DDS Fast Mipmaps vs. Custom Mipmaps
I've always had two ways of generating DDS mipmaps: fast and custom. The "fast" method tells whatever program is being used to automatically generate all mipmaps when creating the texture. This method was used if the pack did not include mipmaps for the texture, or if the user forced new ones. The "custom" method generates all mipmaps as individual images, using either the top level (base texture) or mipmap levels already included in the pack. It then splices the generated versions together into a single image file and sets up some header data, which allows creating arbitrary mipmap effects in DDS textures which almost no programs offer. The custom method used to be much slower, but it's now pretty well optimized so there is almost no difference anymore.
I recently discovered when using the fast method that Compressonator is not creating all mipmap levels (its stopping at like 8x8 and 4x4), and Nvidia Texture Tools does not allow defining the number of mipmaps to generate. While this is not a big concern, I see this as a bug. It combats my option to define the number of mipmaps to generate, and it nags at my OCD to create images in the way that I want them to be created. So the fast mipmaps method has been scrapped and the script will always use the custom mipmap method. This way I know that they are always being created correctly, and it also allows me to always apply a sharpening filter if the dimensions change. I couldn't do this with the fast method, so mipmaps always had slightly higher quality using the custom method.
Mipmap Base Image
When a mipmap is generated, it needs to be created from somewhere, so a "base image" must be used. Now that custom mipmaps is the only method, I thought this could use improvements. Basically here's what is happening. If an image does not have mipmaps in the pack, or new mipmaps are forced, the first mipmap is created from the top level, and every mipmap after that is created from the previously generated mipmap. This means each mipmap will be generated from a smaller image. The result is basically the same, except now each mipmap does not need to be calculated from the top level. This offers a nice speed up especially for large images, and mipmap image quality as far as I can tell, is the same.
Next situation is the pack has mipmaps included. If they exist, the script uses those, no changes there. But, if it contains an incomplete chain, it will first use the mipmaps it can find. After the final included mipmap is used, the next mipmap is then created from the previous mipmap, and it will continue to use the previously generated mipmap until the chain is finished. If the user forces missing mipmaps from the top level, it will use all the mipmaps it can find, then generate the next level from the top level, and again use the previously generated mipmap from there on out.
If none of this makes sense, the only thing to take away from this is mipmap generation is now faster when mipmaps are not provided. No difference in generation speed if they are provided, slight speed up if the included mipmap chain is incomplete.
Other Changes
So that was the big stuff, but there are a bunch of other little things that don't require as much depth to explain. So here's the list:
• Converting BC7 to PNG now uses Compressonator. TexConv was used before, but as discussed earlier in this thread, it has issues.
• Image viewer improvements. A bug was fixed if "Force New Mipmaps" was checked. It caused DDS mipmaps not to show up. Also the viewer will now show the image format, which is useful to distinguish between BC7 and ARGB32 without opening up a hex editor.
• Fix up some of the more uncommonly used options like "Remove Invalid Mipmaps", which now only works on Dolphin "tex1" textures.
• Remove Alpha Channel From Opaque Textures now only works on PNG images. It used to work for DDS images, but this was to convert DXT5 textures to DXT1 if they did not have transparent pixels. As far as I'm concerned, this is now useless because of BC7.
• A small speed up when error checking DDS textures (which is an uncommon and discouraged procedure). There exists a check to calculate the number of mipmaps the texture should have, count the number of internal mipmaps it actually has, and compare. I realized I can just pull the number from the header, rather than extract them all and count them, which saves both time and hard drive life.
A lot of stuff was changed everywhere, rewritten, or scrapped completely which was not worth mentioning since functionality should not have change. I'd be shocked if I didn't cause any bugs with all these changes. So as always, if anyone finds an issue, please let me know.
v29.0 : Download - Mirror
Alpha vs. Transparency
Let me start out by saying I don't always make the best decisions. Before BC7 support, there were only two DDS formats this script created: DXT1 and DXT5. Because I wanted the script to automatically figure out which format to choose, a method was created to detect if PNG images had an alpha channel. At first, I just checked for the channel, but soon realized this can miss some textures because images can have an alpha channel but not actually contain transparent pixels. So then I used ImageMagick which can scan an image for transparency, but this method was slow. I eventually researched my own method to loop through every pixel and read the alpha value. This was much faster, but of course, it was still slow. This is the method that has remained for ages, and eventually creeped its way into other options of the script.
Much time has passed, my prospects change, new stuff has been implemented, and I no longer think its as important to know whether or not images have transparent pixels. The method for checking for alpha pixels is far too slow, so I have gone back to just testing for alpha channels. This has a few drawbacks: fully opaque PNG images with an alpha channel will be created as 32-bit RGBA instead of 24-bit RGB, and DXT5 images will be created instead of DXT1 if using those formats and the option to auto-select the format. Neither of these situations are important enough for the time it takes to actually test transparency in images using PowerShell. I thought about trying to use memory pointers which is supposedly much faster than copying to byte arrays, but apparently PowerShell does not support this, and for good reason (people could use it maliciously to alter RAM contents). C# is allowed to to this with "unsafe" code, and it's possible to use C# code blocks in PowerShell, but I'd rather not go down that alley especially since my knowledge of it is still limited.
So a bit more important, alpha check is also used to know which quality to pick for BC7 images using Compressonator, so very large opaque images may take longer to generate if they have an alpha channel. It is possible to avoid this by using the option to remove alpha channels from opaque images, which I will talk more about in a minute. The same situation holds true when applying upscaling filters. The seamless method by default is only chosen for images without an alpha channel unless it is forced to all images. This means the seamless method may not get applied to some images that should have it applied to keep the seamless effect. This can be worked around by forcing the seamless method for all textures, or fixing them to not have alpha when they shouldn't. Again, neither of these situations seems important enough to justify the slowdown caused by checking every pixel.
The whole checking for alpha pixels was basically me trying to fix problems created by others, but to be fair, it's an easy mistake to make, and in the real world has little repercussions outside of increased image size. Even I accidentally keep alpha channels in opaque images from time to time. The transparent pixel check is not completely gone from the script, it is still used in the option Remove Alpha Channel From Opaque Textures which has been around for awhile now and has been reworked a bit. So if you absolutely want all textures to be perfect and not contain unnecessary data, this option will do exactly what it says. This only needs to be done once, and it only works for PNG images. This will ensure all the potential issues I mentioned above won't ever be an issue. It's unfortunate that there really is no faster way to test images for transparency, at least not using PowerShell. My method is as fast as it can possibly get, but it's still far too slow.
Now that the transparency check has been replaced with just a check for alpha channels, I decided to do a basic issue scan on the Paper Mario: TTYD pack. This is always my go pack for the reason it has over 4000 textures of various types and sizes, covering a large spectrum of what I need to test.
00:27:15.28 - Before
00:03:06.48 - After
What took nearly a half hour can now be done in 3 minutes. Like I said, I do not always make the best decisions. This time was added to almost every single option including DDS conversions, as a texture is always scanned before anything is done with it. This is a huge speed up everywhere, and I regret not "fixing" this sooner.
DDS Fast Mipmaps vs. Custom Mipmaps
I've always had two ways of generating DDS mipmaps: fast and custom. The "fast" method tells whatever program is being used to automatically generate all mipmaps when creating the texture. This method was used if the pack did not include mipmaps for the texture, or if the user forced new ones. The "custom" method generates all mipmaps as individual images, using either the top level (base texture) or mipmap levels already included in the pack. It then splices the generated versions together into a single image file and sets up some header data, which allows creating arbitrary mipmap effects in DDS textures which almost no programs offer. The custom method used to be much slower, but it's now pretty well optimized so there is almost no difference anymore.
I recently discovered when using the fast method that Compressonator is not creating all mipmap levels (its stopping at like 8x8 and 4x4), and Nvidia Texture Tools does not allow defining the number of mipmaps to generate. While this is not a big concern, I see this as a bug. It combats my option to define the number of mipmaps to generate, and it nags at my OCD to create images in the way that I want them to be created. So the fast mipmaps method has been scrapped and the script will always use the custom mipmap method. This way I know that they are always being created correctly, and it also allows me to always apply a sharpening filter if the dimensions change. I couldn't do this with the fast method, so mipmaps always had slightly higher quality using the custom method.
Mipmap Base Image
When a mipmap is generated, it needs to be created from somewhere, so a "base image" must be used. Now that custom mipmaps is the only method, I thought this could use improvements. Basically here's what is happening. If an image does not have mipmaps in the pack, or new mipmaps are forced, the first mipmap is created from the top level, and every mipmap after that is created from the previously generated mipmap. This means each mipmap will be generated from a smaller image. The result is basically the same, except now each mipmap does not need to be calculated from the top level. This offers a nice speed up especially for large images, and mipmap image quality as far as I can tell, is the same.
Next situation is the pack has mipmaps included. If they exist, the script uses those, no changes there. But, if it contains an incomplete chain, it will first use the mipmaps it can find. After the final included mipmap is used, the next mipmap is then created from the previous mipmap, and it will continue to use the previously generated mipmap until the chain is finished. If the user forces missing mipmaps from the top level, it will use all the mipmaps it can find, then generate the next level from the top level, and again use the previously generated mipmap from there on out.
If none of this makes sense, the only thing to take away from this is mipmap generation is now faster when mipmaps are not provided. No difference in generation speed if they are provided, slight speed up if the included mipmap chain is incomplete.
Other Changes
So that was the big stuff, but there are a bunch of other little things that don't require as much depth to explain. So here's the list:
• Converting BC7 to PNG now uses Compressonator. TexConv was used before, but as discussed earlier in this thread, it has issues.
• Image viewer improvements. A bug was fixed if "Force New Mipmaps" was checked. It caused DDS mipmaps not to show up. Also the viewer will now show the image format, which is useful to distinguish between BC7 and ARGB32 without opening up a hex editor.
• Fix up some of the more uncommonly used options like "Remove Invalid Mipmaps", which now only works on Dolphin "tex1" textures.
• Remove Alpha Channel From Opaque Textures now only works on PNG images. It used to work for DDS images, but this was to convert DXT5 textures to DXT1 if they did not have transparent pixels. As far as I'm concerned, this is now useless because of BC7.
• A small speed up when error checking DDS textures (which is an uncommon and discouraged procedure). There exists a check to calculate the number of mipmaps the texture should have, count the number of internal mipmaps it actually has, and compare. I realized I can just pull the number from the header, rather than extract them all and count them, which saves both time and hard drive life.
A lot of stuff was changed everywhere, rewritten, or scrapped completely which was not worth mentioning since functionality should not have change. I'd be shocked if I didn't cause any bugs with all these changes. So as always, if anyone finds an issue, please let me know.
I prefer to keep the backwards compatibility so I'll replace that check. I need to get my VMs back up and running so I can make sure this isn't the only thing. About 2 weeks ago the hard drive that my virtual disks were on started acting funky so I pulled it. I'll try to plug it back in and copy them over to another drive.
![[Image: se2eRuo.png]](https://i.imgur.com/se2eRuo.png)
