Custom Texture Tool PS v50.1

**Admentus** · (This post was last modified: 07-10-2017, 04:03 AM by Admentus.)

Awesome update Bighead! Back to porting textures again for me.

A bit off-topic, but is your head really that big? I have always wondered about that since my time on the Dolphin forums.

Techie Android · 07-10-2017, 05:35 AM **#312**

@Bighead
Nice! I was just wondering what would be the best way to execute this within Linux specifically Ubuntu. I wonder if WineHQ supports this kind of code... I'll be checking on that but if anyone knows off hand please let me know.

**Bighead** · (This post was last modified: 07-10-2017, 06:16 AM by Bighead.)

If you throw a piece of popcorn at my face, it will never make it because it will get caught in orbit.

In all seriousness, no, my head is not that big, and it's certainly not a name I originally chose for myself. Here's the not very interesting story if you really want to know.

Spoiler: (Show Spoiler)

(07-10-2017, 05:35 AM)Techie Android Wrote: @Bighead
Nice! I was just wondering what would be the best way to execute this within Linux specifically Ubuntu. I wonder if WineHQ supports this kind of code... I'll be checking on that but if anyone knows off hand please let me know.

You could probably set up a Windows VM and create a shared folder between it and the host. I often test the script on older windows versions this way, and it works just as well on the guest as it does on the host.

General_Han_Solo · 07-10-2017, 08:41 AM **#314**

Bighead, if you have time, check out my Star Wars texture packs re-rendered with v. 23.2 Big Grin

**Bighead** · 07-10-2017, 10:00 PM **#315**

@General_Han_Solo: All your textures look amazing, I don't know how you do it. I looked through your images and found no issues with anything. I did notice some textures with their original resolution scattered about, I'm assuming they are improved over the originals. If they are not custom and are "as-is" dumps from the game, converting them to DDS would actually be a quality drop over not loading any texture at all.

@Techie Android: I forgot to mention earlier the VM I use is VirtualBox. But that might not even be needed. I was looking into it and it seems PowerShell is now open source and available for OSX and Linux Ubuntu. I have no idea of if/how it works or how to set it up since I know next to nothing about other operating systems (the last time I used another OS was QBASIC). If you decide to try it out, please share your experiences: https://github.com/PowerShell/PowerShell/releases

Now that I know this exists I might even be temped to set up Ubuntu in a VM just to play around with it.

Techie Android · (This post was last modified: 07-11-2017, 02:47 AM by Techie Android.)

(07-10-2017, 10:00 PM)Bighead Wrote: @Techie Android: I forgot to mention earlier the VM I use is VirtualBox. But that might not even be needed. I was looking into it and it seems PowerShell is now open source and available for OSX and Linux Ubuntu. I have no idea of if/how it works or how to set it up since I know next to nothing about other operating systems (the last time I used another OS was QBASIC). If you decide to try it out, please share your experiences: https://github.com/PowerShell/PowerShell/releases

Now that I know this exists I might even be temped to set up Ubuntu in a VM just to play around with it.

Cool I'm looking into this now. Ubuntu is really great you should go ahead and try it, it's worth it. I use WineHQ for pretty much anything I'd need in Windows so there's little to no sacrifice.

Edit: I tried first with the package and it didn't install so now it looks like I'll have to build it from source. I don't know if it will work even after that though because I can't get libicu55 installed. I do wonder though if/when I get this working what is the best way to benchmark the speed of your script? I want to give you the results when I have them, to compare to the Windows PS speeds.

Edit-2: Got it setup now.

It spit this out when I tried running it:

Spoiler: (Show Spoiler)

I'm afraid that the best thing to do would be to just convert it to bash. I think it's having problems because the environment is simply not the same as it is in Windows. I know there's 13,220 lines of code, I would be glad to help convert it over.

**Bighead**

I honestly don't have the will to convert it to another language, or learn one for that matter, but anyone is free to do so. I heavily documented it just in case that rare situation would happen. This would not be an easy task, there are some things I didn't consider before as to why this wouldn't work on other OS or through that PS distro.

1.) The script uses GUI code built on .NET framework forms namespace. This is mostly the source of all those errors. A cross platform version of .NET Framework exists called Mono, but again I know nothing about this. If there is a way to get linux Powershell and Mono to play nice together, it might work? But you won't get far because of 2.

2.) The script is just a script, there is not much fancy going on under the hood. Think of it as just a GUI/automation tool for CLI based external programs. DDS Utilities is not cross platform, nor is xBRZ ScalerTest, or waifu2x-caffe, or Tino's Ishiiruka Tool. ImageMagick is cross platform, so some stuff could technically be easily ported. But only basic stuff, no fancy upscalers, no converting to DDS, no working with material maps. New programs would have to be found, or replaced entirely with native code to handle images (which would be a huge undertaking). I also know nothing about the way linux sets up files and folder in its system, and a some code checks install paths or values in the Windows registry.

It would probably be better to create a brand new project if cross platform was desired. I have no qualms of using the script as a reference if there is anything worthwhile to take from it. I have a lot of edge cases documented, but most of them are nuances to the specific program it was designed around. A new project could be built to not require external tools, and have its own image libraries (like libpng). But this would be significantly more work than a script file. I don't have the skills or time to consider it, I already had to learn way more than I ever wanted to to get this far with it.

The only reasonable option I can think I already suggested, setting up Windows in a VM. Virtual Box is really easy to use and it takes about an hour to install windows on it. All programs and dependencies the script needs can be installed on the virtual Windows and everything works as it would on a real install.

There is something that can resize textures on linux by leolam you can find here.

Techie Android · 07-11-2017, 08:18 AM **#318**

(07-11-2017, 04:06 AM)Bighead Wrote: I honestly don't have the will to convert it to another language, or learn one for that matter, but anyone is free to do so. I heavily documented it just in case that rare situation would happen. This would not be an easy task, there are some things I didn't consider before as to why this wouldn't work on other OS or through that PS distro.

1.) The script uses GUI code built on .NET framework forms namespace. This is mostly the source of all those errors. A cross platform version of .NET Framework exists called Mono, but again I know nothing about this. If there is a way to get linux Powershell and Mono to play nice together, it might work? But you won't get far because of 2.

2.) The script is just a script, there is not much fancy going on under the hood. Think of it as just a GUI/automation tool for CLI based external programs. DDS Utilities is not cross platform, nor is xBRZ ScalerTest, or waifu2x-caffe, or Tino's Ishiiruka Tool. ImageMagick is cross platform, so some stuff could technically be easily ported. But only basic stuff, no fancy upscalers, no converting to DDS, no working with material maps. New programs would have to be found, or replaced entirely with native code to handle images (which would be a huge undertaking). I also know nothing about the way linux sets up files and folder in its system, and a some code checks install paths or values in the Windows registry.

It would probably be better to create a brand new project if cross platform was desired. I have no qualms of using the script as a reference if there is anything worthwhile to take from it. I have a lot of edge cases documented, but most of them are nuances to the specific program it was designed around. A new project could be built to not require external tools, and have its own image libraries (like libpng). But this would be significantly more work than a script file. I don't have the skills or time to consider it, I already had to learn way more than I ever wanted to to get this far with it.

The only reasonable option I can think I already suggested, setting up Windows in a VM. Virtual Box is really easy to use and it takes about an hour to install windows on it. All programs and dependencies the script needs can be installed on the virtual Windows and everything works as it would on a real install.

There is something that can resize textures on linux by leolam you can find here.

Yeah that's pretty much what I was think about too, without support for those features it would be even more difficult to pull off. I'll probably just setup a VM with Windows in it and share the folder.

**Bighead** · (This post was last modified: 07-13-2017, 11:30 PM by Bighead.)

Nothing new this post, just some pointless blabbing because I'm bored at the moment. My most recent quest has been to better understand DDS headers. As far as I know, there are two important types: the widely recognized D3D9 header, and the DX10/DX11 extended header which adds a bit more information. So far, I have not yet fully looked into the latter, but it will be next since I want the script to be able to support textures that are compressed with DXT1/DXT5 that have DX10 headers (and maybe eventually BC7 if its worth it). This is most likely a pointless task since 99.99% of people probably don't create DDS textures with DX10 headers, but I am OCD about being thorough and supporting every possible case since Dolphin does support these headers. I have looked over several documents and pages that explain the header in detail, but it wasn't really clicking in my head because it was hard to see how it all ties together. This led me to creating a visual representation that dissects what all the information is, and slightly describes each section of it.

Spoiler: (Show Spoiler)

I was hoping to find an image like this online which would have saved me much time and hassle, but no luck. Much of what I have written down is backwards compared to what the documents state since DDS headers are little endian, and I wanted the information to translate literally into what a hex editor sees / how the image translates into a byte array. Not everything may be correct, and a lot of info is missing, but it's enough to start understanding what exactly is going on in those first 128 bytes.

Before I relied mostly on ImageMagick to get data from a DDS image. A few versions back I did manage to get the dimensions from the header which provided a nice speed up, but that's it. Having to rely on ImageMagick is the main reason its much slower to process DDS packs over PNG packs using any option. So the second advantage to being able to get this information directly from the header is it will lead to a further speed-up in a future version when working with DDS textures (which, *should be* a rare situation since we should be working with PNG textures, but that's not always possible).

So now I have the information I needed to try to figure out an issue with Ishiiruka textures that has bugged me for over a year: they don't work in several programs, and I never could figure out why. Most importantly, they don't work with ImageMagick which is the backbone of this script, so in that broken state they are completely unsupported since I can't do anything with them.

Spoiler: (Show Spoiler)

When the script finds an Ishiiruka color texture + material map, the only tool that is used in an attempt to modify those textures is Ishiiruka Tool. But what if the material map becomes lost? We would be left with a broken color texture that few programs can make use of, and this doesn't help that it defeats the automation of this script if yet another tool that does work has to be found. Even the master branch of Dolphin could not load them for the longest time until stenzek added support for loading compressed textures which inadvertenly "fixed" it. A long time ago, I did come up with a workaround in a specific scenario when a "broken" texture was found which the current version of the script is still using.

1.) All DDS files have an "identify" check ran on them using ImageMagick when validating the image.
2.) If the DDS file fails to be identified, the script would assume it was an Ishiiruka color texture with a missing material map.
3.) A fake material map is created so it can be ran through Ishiiruka Tool, which is used to convert the image to PNG.
4.) The new PNG color texture is then converted back to DDS with DDS Utilities, and the fake material is discarded.
5.) If this process failed for any reason, the DDS file was considered "corrupt".

This is an ugly workaround that made use of 3 programs. So, what exactly is Ishiiruka Tool doing? Referring to the DDS Header image above, there are actually two headers: The main header, and the Pixel Format header. Both headers have various flags that can be set, that either set certain properties or let the program utilizing them know that there is valid data in other parts of the header. The pixel format header has a flag called "DDPF_RGB" that when set, says there is valid data in the BitCount and BitMask sections. Ishiiruka Tool is setting this flag, but it's not actually creating any data in these sections.

Spoiler: (Show Spoiler)

Changing the "4" to a "0" fixes the problem and all programs now recognize the image. I don't know if this is a bug in Ishiiruka Tool, if Tino is setting this flag on purpose for some reason, or if I'm totally off base. But now instead of that mess of a work-around, I can simply change a single value in the file and the issue is fixed. This will be performed automatically, but only on textures that are missing a material map since I don't know the reason that flag is set in the first place. Ishiiruka Tool can handle these images just fine, and its the only tool that is used on them when working with material maps, so there is no reason to modify that byte unless its absolutely necessary.

Edit: Success in getting DX10 headers to work when texture is using a DXT# block compression. I realized directly supporting DX10 headers is impossible due to some programs not knowing what they are, and I would hate to limit the script to just Compressonator/TexConv to work with these specifically. So now they are automatically converted to the D3D9 version. Converting the header is simple enough. The block compression is now stored in the DX10 section of the header, which can be retrieved, translated, and copied into the FourCC value. The 20 bytes that make up the DX10 header were then eliminated. This was not as complicated of a task as I thought it would be, although it was something I wasn't able to do until now.

Spoiler: (Show Spoiler)

jvook · 07-19-2017, 01:52 AM **#320**

(07-10-2017, 06:14 AM)Bighead Wrote:

Spoiler: (Show Spoiler)

There were a few weeks in high school I had a lot of hair which made it look like I had a big head. As a dumb joke, I tried to grow an afro but it never quite worked out so it was only for a very, very short period. I never would have guessed it would have a life lasting effect. A guy in my group of friends at the time was calling me "cabeza grande" which is Spanish for big head. People would then ask when hearing it what it meant, so "big head" was being tossed around in random conversations. Something to keep in mind, the average IQ of where I live is probably in the double digits. Knicknames are very common and spread like fire on a dry gas covered field. For example, a dude from Texas moved here, his knickname ended up becoming "Tex", very original and well thought out. Let me assure you that you never wanted to meet "Booger".

It felt like it happened over a weekend. Monday morning came, a new day in school, and I was forever dubbed "Bighead" somehow... Even by people who weren't part of my click, ones who barely knew me. Many had their own variation, just combine any word for "big" with a random body part and let the imagination go wild. I wasn't a fan of it at first, although it didn't really bother me, so it wasn't long before I just starting rolling with it. Fast forward almost 2 decades, most people around here now think I got this knickname because they think I'm smart (I 'm not stupid, but saying I'm smart is a stretch). And well, that's about it. When the internet became a thing, I used it as my screen name and just never changed it. Online games were the exception, I went by "StRoNgFoE" for years but long retired it. Too many alter egos, I was afraid I would start having conversations with myself.

No chance you're from OH right? I knew a booger (named because apparently looked like a booger when born -- terrible nickname to choose for your own kid!) and a bighead (with an actual big head) 20 years ago.