Well yes, technically if your sticks are dual channel (meant to be run in packs of two) there could be problems trying to run four simultaneously, but it depends on the memory. I would suggest googling if someone has had any success with the memory that you have.
In addition, it seems that your motherboard doesn't even support quad channel memory, so your only choices would be to run two compatible dual channel sets, or to replace the current ones with two 8GB sticks.
*IMPORTANT* The "Pre-fetch custom textures" option is meant for machines with a monstrous amount of RAM, so it's no wonder it fills 8GB. Basically, it loads all of the textures into system memory when Dolphin is opened so that there is no texture pop-up or stutter when gaming. If you have Dolphin on all the time, the textures are taking up space from other things that could be there. I don't know if this is one of the reasons for your problem, but you should try disabling the setting, rebooting and seeing if the problem persists.
Thread on pre-fetch.
In addition, it seems that your motherboard doesn't even support quad channel memory, so your only choices would be to run two compatible dual channel sets, or to replace the current ones with two 8GB sticks.
*IMPORTANT* The "Pre-fetch custom textures" option is meant for machines with a monstrous amount of RAM, so it's no wonder it fills 8GB. Basically, it loads all of the textures into system memory when Dolphin is opened so that there is no texture pop-up or stutter when gaming. If you have Dolphin on all the time, the textures are taking up space from other things that could be there. I don't know if this is one of the reasons for your problem, but you should try disabling the setting, rebooting and seeing if the problem persists.
Thread on pre-fetch.