Check how much RAM your system is using before you launch Dolphin. Unless you're doing heavy video-editing (or playing Crysis at 2560*1600) in the background I don't know what could be using up all of that RAM.
Windows 7 RC Build 7100 x64 || Windows Vista SP2 x64
Intel Core 2 Duo E8500 (3.16Ghz)
ATI Radeon HD 4870 512MB (750Mhz/900Mhz)
OCZ Vista Gold 4GB RAM (2x2GB) DDR2-800
Asus P5QL-E
OCZ GameXStream 600W
Intel Core 2 Duo E8500 (3.16Ghz)
ATI Radeon HD 4870 512MB (750Mhz/900Mhz)
OCZ Vista Gold 4GB RAM (2x2GB) DDR2-800
Asus P5QL-E
OCZ GameXStream 600W
