Hey there Bighead. Firstly, I want to say I'm amazed at how relatively quickly you've managed to add ESRGAN and SFTGAN support to your texture tool, even with all the hiccups left to iron out. I've been jumping down the ESRGAN rabbit hole myself recently while working on one of my texture packs. It may not help any of your current issues, but I thought I would share some useful information and resources I've come across. First up, here's a link to the initial reddit thread (for context) that started the ball rolling for me: Metroid Prime 2 upscaled with a neural network
The important part is the OP's link to a very useful set of scripts and tools by rapka: https://github.com/rapka/dolphin-textures
This is the primary toolset I've been using to upscale with. The big points are that rapka uses his own forked branch of ESRGAN in conjunction with the manga109 model and his own scripts geared to upscale textures for dolphin. Out of those scripts, the biggest feature I've come across is scripts for handling images with alpha transparency. They separate, upscale, and re-integrate the alpha channel into the upscaled transparent image. I've also found that ESRGAN's ability to upscale is solely dependent on the model it uses, and the manga109 model is one of the most impressive I've seen so far for textures. I put together a quick comparison between models used on a 128x128 texture (manga109, PSNR_x4, ESRGAN_x4, and an interpolation between PSNR and ESRGAN), plus an upscaled text sheet with transparency:
I hope these links might be of use, or at least interest. Unforunately, I don't have any helpful advice for the texture size/VRAM issue, other than to confirm that trying to upscale a 1024x1024 texture fails for me with a GTX 1080 and 8 GB of vram. 512x512 is my limit (I haven't tested any sizes in-between).
The important part is the OP's link to a very useful set of scripts and tools by rapka: https://github.com/rapka/dolphin-textures
This is the primary toolset I've been using to upscale with. The big points are that rapka uses his own forked branch of ESRGAN in conjunction with the manga109 model and his own scripts geared to upscale textures for dolphin. Out of those scripts, the biggest feature I've come across is scripts for handling images with alpha transparency. They separate, upscale, and re-integrate the alpha channel into the upscaled transparent image. I've also found that ESRGAN's ability to upscale is solely dependent on the model it uses, and the manga109 model is one of the most impressive I've seen so far for textures. I put together a quick comparison between models used on a 128x128 texture (manga109, PSNR_x4, ESRGAN_x4, and an interpolation between PSNR and ESRGAN), plus an upscaled text sheet with transparency:
Spoiler: