A month ago, NaturalViolence told me it was kind of sad Dolphin devs didn't write a lot about how emulation works and how they try to improve it. Recently I've been working on a rewrite of the old DSP high-level emulation (HLE). Few people actually know about how sound emulation works in Dolphin, even among the developers, so this is an interesting topic to explain in detail IMO. I'll mostly talk about GC here, but Wii sound handling works almost the same so most of this post should apply to Wii as well.
An edited version of this article is available here: http://blog.lse.epita.fr/articles/38-emulating-the-gamecube-audio-processing-in-dolphin.html
This is a three parts article:
Part 1: how sound is processed in a Gamecube
Part 2: AX features and DSP/CPU communication
Part 3: AX HLE in Dolphin, previous vs. new
Part 1: how sound is processed in a Gamecube
There are three main internal components related to sound in a Gamecube: the ARAM, the AI and the DSP:
ARAM and AI are not that hard to emulate: once you understand how they work, they are both simple chips which can only perform one function and don't communicate a lot with the CPU. You just need to have a precise enough timing for AI emulation, and everything is fine.
DSP is a lot harder to emulate properly, for two reasons I have not mentioned yet. First, it is a programmable CPU. All the mixing, filtering, etc. are part of a program that is sent to the DSP by the game, and the DSP behavior varies depending on the program it receives. For example, the DSP is not only used for sound processing, but also to unlock memory cards, and to crypt/decrypt data sent to a GBA using the official link cable. Even for sound processing, not every game uses the same DSP code. The second reason is that it can communicate with the main Gamecube CPU, read RAM and ARAM and write to RAM. This allows games to use a complicated communication protocol between CPU and DSP.
We call a program running on the DSP a UCode ("microcode"). Because the DSP is programmable, it would seem like the only way to emulate it properly is to use low level emulation: running instructions one by one from a program to reproduce accurately what the real DSP does. However, while it is programmable, there are actually very few different UCodes used by games. On Gamecube, there are only 3 UCodes we know of: the AX UCode (used in most games because it is distributed with Nintendo's SDK), the Zelda UCode (called that way because it's used in Zelda games, but it is also used for some Mario games and some other first party games), and the JAC UCode (early version of the Zelda UCode, used in the Gamecube IPL/BIOS as well as Luigi's Mansion). That means if we can reproduce the behavior of these three UCodes, we can emulate the audio processing in most games without having to emulate the DSP instructions.
I started working on AX HLE last week because I want to play Skies of Arcadia Legends and Tales of Symphonia, which are two games that had completely broken audio with the previous AX HLE implementation. I added a new hack to "fix" the bug that caused bad music emulation, but fixing this made me even more interested in rewriting the whole thing to make it cleaner. I wasn't following Dolphin development when the current AX HLE was developed. However, it looks to me as if it was written without actually looking at the DSP code, only looking at what is sent to the DSP and what comes out. I don't know if people at the time had the capability to disassemble DSP code, but it is a very bad way to emulate AX anyway: some parts of the emulation code are completely WTF, and the more you understand how AX works the less you understand how the current AX HLE emulation was able to work and output sound in most cases. That's why, 3 days ago I decided I should start from scratch and re-implement AX HLE.
I'm not completely done with it yet: there are still a few bugs and desyncs, which are mostly identified. Still, the current new-ax-hle audio is largely better than the audio on latest master revisions. Small comparison for old HLE, new HLE and LLE on Tales of Symphonia main menu.
In a next article I'll talk about how audio is emulated in Dolphin, how AX works internally and why the previous AX implementation shouldn't actually be working that well
An edited version of this article is available here: http://blog.lse.epita.fr/articles/38-emulating-the-gamecube-audio-processing-in-dolphin.html
This is a three parts article:
Part 1: how sound is processed in a Gamecube
Part 2: AX features and DSP/CPU communication
Part 3: AX HLE in Dolphin, previous vs. new
Part 1: how sound is processed in a Gamecube
There are three main internal components related to sound in a Gamecube: the ARAM, the AI and the DSP:
- ARAM is an auxiliary memory which is used to store sound data. The CPU cannot access ARAM directly, it can only read/write blocks of data from RAM to ARAM (or from ARAM to RAM) using DMA requests. As ARAM is quite large, games often use it to store more than sound data: for example, WWE Day of Reckoning 2 uses it to store animation data (and a bug in DMA handling causes a crash because the data it writes is corrupted).
- The AI (Audio Interface) is responsible for getting sound data from RAM and sending it to your TV. It performs an optional sample rate conversion (32KHz -> 48KHz) and converts the data to an analog signal that is sent through the cables to your audio output device. The input data is read at a regular interval from RAM (not ARAM), usually every 0.25ms 32 bytes of input data is read (each sound sample is 2 bytes, so 32 bytes is 16 sound samples, which is 8 stereo sound samples, and 8 samples every 0.25ms == 32KHz sound).
- The DSP is what processes all the sounds a game wants to play and outputs a single stereo stream. Its job is to perform volume changes on the sounds, sample rate conversion (converting 4KHz sounds which take less space to 32KHz sounds - this is needed because you can't mix together sounds that are not the same rate). It can optionally do a lot of other stuff with the sounds (delaying to simulate 3D sound, filtering, handling surround sound, etc.).
ARAM and AI are not that hard to emulate: once you understand how they work, they are both simple chips which can only perform one function and don't communicate a lot with the CPU. You just need to have a precise enough timing for AI emulation, and everything is fine.
DSP is a lot harder to emulate properly, for two reasons I have not mentioned yet. First, it is a programmable CPU. All the mixing, filtering, etc. are part of a program that is sent to the DSP by the game, and the DSP behavior varies depending on the program it receives. For example, the DSP is not only used for sound processing, but also to unlock memory cards, and to crypt/decrypt data sent to a GBA using the official link cable. Even for sound processing, not every game uses the same DSP code. The second reason is that it can communicate with the main Gamecube CPU, read RAM and ARAM and write to RAM. This allows games to use a complicated communication protocol between CPU and DSP.
We call a program running on the DSP a UCode ("microcode"). Because the DSP is programmable, it would seem like the only way to emulate it properly is to use low level emulation: running instructions one by one from a program to reproduce accurately what the real DSP does. However, while it is programmable, there are actually very few different UCodes used by games. On Gamecube, there are only 3 UCodes we know of: the AX UCode (used in most games because it is distributed with Nintendo's SDK), the Zelda UCode (called that way because it's used in Zelda games, but it is also used for some Mario games and some other first party games), and the JAC UCode (early version of the Zelda UCode, used in the Gamecube IPL/BIOS as well as Luigi's Mansion). That means if we can reproduce the behavior of these three UCodes, we can emulate the audio processing in most games without having to emulate the DSP instructions.
I started working on AX HLE last week because I want to play Skies of Arcadia Legends and Tales of Symphonia, which are two games that had completely broken audio with the previous AX HLE implementation. I added a new hack to "fix" the bug that caused bad music emulation, but fixing this made me even more interested in rewriting the whole thing to make it cleaner. I wasn't following Dolphin development when the current AX HLE was developed. However, it looks to me as if it was written without actually looking at the DSP code, only looking at what is sent to the DSP and what comes out. I don't know if people at the time had the capability to disassemble DSP code, but it is a very bad way to emulate AX anyway: some parts of the emulation code are completely WTF, and the more you understand how AX works the less you understand how the current AX HLE emulation was able to work and output sound in most cases. That's why, 3 days ago I decided I should start from scratch and re-implement AX HLE.
I'm not completely done with it yet: there are still a few bugs and desyncs, which are mostly identified. Still, the current new-ax-hle audio is largely better than the audio on latest master revisions. Small comparison for old HLE, new HLE and LLE on Tales of Symphonia main menu.
In a next article I'll talk about how audio is emulated in Dolphin, how AX works internally and why the previous AX implementation shouldn't actually be working that well
