Very nice update, perfect in fact. Unfortunately I'm getting a crash when it's modifying the maps.
You make a valid point when it comes to texture sizes. There was a time when I used to host a "performance" pack where I would limit the maximum texture sizes to 2048x2048, and also have "UltraHD" environments as a separate download. 2048x2048 can still be rather big at 1080p (depending on where they are being used), but not over the top like some of the Tephra Cave textures currently are. Releasing so many packs became a maintenance burden since I was releasing both PNG and DDS packs at multiple sizes. Including core packs, controller packs, and UltraHD environments, at one point there was a whopping 44 different packs! I've eliminated the downscaled textures and PNG packs, and introduced "compatibility packs" for older hardware which cut that number by about half (currently at 20 packs).
I have no issue generating packs, that's just a few button pushes with my script. So this is probably part me being lazy, part trying to save myself from uploading gigabytes of data every update, and part me wanting to keep the textures at the highest possible quality thinking they can always be downscaled later. But it brings up the question of who should be the one downscaling them? It would be silly for me to expect users to do that, most are just going to use the downloads that are offered. Some textures in the current core pack do have shimmering issues at 1080p, so this is a failure on my part of providing enough options for a better experience.
I wouldn't go so far as to say I was disappointed. Annoyed is probably a bit more accurate, but that's only because of having to choose a Dolphin folder and it wanted to download a lot of stuff. I have to watch every GB so I don't go over my limit. My ISP throttles my connection speed by 90% if I exceed it which is very annoying, and my area does not have choices. So the issue is more with my ISP than the installer, choosing the Dolphin folder seemed unnecessary but manageable, but.. data limitations is a huge issue for me.
I did handpick which ones to compress, but the easiest way would be to just detect if a texture has transparency or not. If it does, ARGB32, if not, BC7. For most textures, BC7 is fine but it does cause artifacts around edges which is inherent to all DDS compression. BC7 is nearly indistinguishable from PNG when it comes to opaque images. Currently I am only doing ARGB32 for UI textures. So grass and flower textures and the like are using BC7 just to save VRAM or RAM when prefetching. So I can see the transparency check failing for this and creating them as ARGB32. Sure they would look better, but it's easier to dismiss artifacts in the wilderness where everything blends together, as opposed to the UI which is over top of everything.
Custom mipmaps are fairly simple to create. Since no tools handle custom mipmaps for BC7, I just generate all the levels as individual textures then splice them together. BC7 is simple as you can just assume a block of 16 pixels = 16 bytes, except on the smallest mipmap layers. A block must consist of 16 bytes, so even though a mipmap level can be 2x2 or 1x1, it still takes 16 bytes to represent that block. For ARGB32 it's 64 bytes per block (the term "block" here is used loosely as ARGB32 is an uncompressed format).
Another valid point. It's been my goal for the longest time now to have some kind of online repository, which is why I originally created the archive on Google Drive. Unfortunately I have not found a way to download a folder from Google Drive without actually having to press the "Download All" button that appears on the page. This makes it difficult (maybe even impossible) to download a folder without using a browser. The Download All button zips the folder then downloads it, and there doesn't seem to be a way to create a direct link to this function. My idea has always been to have an online master location for the latest pack, but I never fully brought it to fruition yet. I want a repository that both users and tools like your installer can easily download from. Anyone who wants to make meaningful contributions can have access to modify it, and of course personal backups would be made to protect against sabotage. With this the latest pack is always online, anyone who wants to contribute can easily do so, tools like your installer could grab the files, all the textures are viewable online, and it could just be downloaded directly for those who like to do stuff manually. It would save tons of future work and make life easier for everyone. I would probably still provide packs on file hosts like I do now just for sake of convenience and a smaller download compressed with 7z, but I wouldn't need to worry about mirrors.
It would probably make sense to have multiple packs in this repository, PNG, DDS and a 1080p DDS pack. This way your installer could just grab a pack instead of having to convert from PNG, resulting in a smaller download and a much faster installation. So this would mean at least 3 core packs minimum which is simple enough to maintain. Three packs is more work to contributors, which is why I liked Google Drive logging all changes. Contributors could just modify the PNG pack, the changes are viewable, and I (or anyone with access) would know to do all the resizing and conversion junk for the other packs. But I don't think Google Drive is sufficient for the reason it's not easy to download a folder. Links must remain static or the installer would need to be updated and that's no good. Having the textures in a zip file would defeat being able to modify the pack online by multiple contributors from multiple locations (unless modifying zip contents directly online through some interface is possible?). Github could maybe work, but it's confusing to noobs like me and I don't know if something like your installer could download from it. I haven't put much effort into figuring Github out, and there's the fact others who want to contribute would have to also figure out how to make pull requests. I'm going to look more into it, and I'm open to any ideas on alternate sites that have a bit more power than Google Drive. Maybe even a FTP server would be good enough.
Spoiler:
I have no issue generating packs, that's just a few button pushes with my script. So this is probably part me being lazy, part trying to save myself from uploading gigabytes of data every update, and part me wanting to keep the textures at the highest possible quality thinking they can always be downscaled later. But it brings up the question of who should be the one downscaling them? It would be silly for me to expect users to do that, most are just going to use the downloads that are offered. Some textures in the current core pack do have shimmering issues at 1080p, so this is a failure on my part of providing enough options for a better experience.
(11-20-2017, 07:38 PM)Rickv123 Wrote: I'm sorry that I disappointed you with the latest version.
I wouldn't go so far as to say I was disappointed. Annoyed is probably a bit more accurate, but that's only because of having to choose a Dolphin folder and it wanted to download a lot of stuff. I have to watch every GB so I don't go over my limit. My ISP throttles my connection speed by 90% if I exceed it which is very annoying, and my area does not have choices. So the issue is more with my ISP than the installer, choosing the Dolphin folder seemed unnecessary but manageable, but.. data limitations is a huge issue for me.
(11-20-2017, 07:38 PM)Rickv123 Wrote: The installer does not yet have a proper way to tell whether a texture needs BC7 or ARGB32, other that a little list of texture names that look "horrible" with BC7 textures, perhaps I can to them based on archive name but I'll figure something out.
I did handpick which ones to compress, but the easiest way would be to just detect if a texture has transparency or not. If it does, ARGB32, if not, BC7. For most textures, BC7 is fine but it does cause artifacts around edges which is inherent to all DDS compression. BC7 is nearly indistinguishable from PNG when it comes to opaque images. Currently I am only doing ARGB32 for UI textures. So grass and flower textures and the like are using BC7 just to save VRAM or RAM when prefetching. So I can see the transparency check failing for this and creating them as ARGB32. Sure they would look better, but it's easier to dismiss artifacts in the wilderness where everything blends together, as opposed to the UI which is over top of everything.
(11-20-2017, 07:38 PM)Rickv123 Wrote: About the internal mip-maps, yeah I will admit, there I went the lazy route. With the new release I mainly focused on the modding department. Creating an actual logical way of loading the game's files instead of the previous method of read this value and change to that value at this position and shift this number to there. Internal mip-maps were not my biggest fear at the time. However I'll fix that in future update.
Custom mipmaps are fairly simple to create. Since no tools handle custom mipmaps for BC7, I just generate all the levels as individual textures then splice them together. BC7 is simple as you can just assume a block of 16 pixels = 16 bytes, except on the smallest mipmap layers. A block must consist of 16 bytes, so even though a mipmap level can be 2x2 or 1x1, it still takes 16 bytes to represent that block. For ARGB32 it's 64 bytes per block (the term "block" here is used loosely as ARGB32 is an uncompressed format).
(11-20-2017, 07:38 PM)Rickv123 Wrote: I know the .png versions are archived in the google-drive directory. However when downloading large files from google drive it required to accept cookies and whatever. However in the new hotfix I released I managed to accept the cookie confirmation for the installer itself and continue to download large files from google drive. If you were to archive them in a .zip rather then I directly connect the texture pack to that version of the installer.
Another valid point. It's been my goal for the longest time now to have some kind of online repository, which is why I originally created the archive on Google Drive. Unfortunately I have not found a way to download a folder from Google Drive without actually having to press the "Download All" button that appears on the page. This makes it difficult (maybe even impossible) to download a folder without using a browser. The Download All button zips the folder then downloads it, and there doesn't seem to be a way to create a direct link to this function. My idea has always been to have an online master location for the latest pack, but I never fully brought it to fruition yet. I want a repository that both users and tools like your installer can easily download from. Anyone who wants to make meaningful contributions can have access to modify it, and of course personal backups would be made to protect against sabotage. With this the latest pack is always online, anyone who wants to contribute can easily do so, tools like your installer could grab the files, all the textures are viewable online, and it could just be downloaded directly for those who like to do stuff manually. It would save tons of future work and make life easier for everyone. I would probably still provide packs on file hosts like I do now just for sake of convenience and a smaller download compressed with 7z, but I wouldn't need to worry about mirrors.
It would probably make sense to have multiple packs in this repository, PNG, DDS and a 1080p DDS pack. This way your installer could just grab a pack instead of having to convert from PNG, resulting in a smaller download and a much faster installation. So this would mean at least 3 core packs minimum which is simple enough to maintain. Three packs is more work to contributors, which is why I liked Google Drive logging all changes. Contributors could just modify the PNG pack, the changes are viewable, and I (or anyone with access) would know to do all the resizing and conversion junk for the other packs. But I don't think Google Drive is sufficient for the reason it's not easy to download a folder. Links must remain static or the installer would need to be updated and that's no good. Having the textures in a zip file would defeat being able to modify the pack online by multiple contributors from multiple locations (unless modifying zip contents directly online through some interface is possible?). Github could maybe work, but it's confusing to noobs like me and I don't know if something like your installer could download from it. I haven't put much effort into figuring Github out, and there's the fact others who want to contribute would have to also figure out how to make pull requests. I'm going to look more into it, and I'm open to any ideas on alternate sites that have a bit more power than Google Drive. Maybe even a FTP server would be good enough.