So I have a general idea as to why there is a outline still going on with the shadows in Star Fox Adventures. While I don't have programming knowledge on the issue, this is extremely similar to something I run into when I use a art program like GIMP, and it involves Anti-Aliasing and Masking.
So here is what I think is going on. It's masking out the unused shadows with a exact copy of those shadows, subtracting the correct shadow from the mask. Let me help you visualize it.
Here are three overlapped shapes, a Triangle, a Square and a Hexagon. While with shadows they are black, I color coded them to be easily visualized.
For the masking it selects the combined shape, and then subtracts the shape it wants, in this case the Hexagon. It then creates a mask (black and white) of the shape.
It then applies this mask image to the mass to remove everything but the hexagon.
However when you look closely, you can see the outline where the cutout shapes were
And this works no matter which shape we want to visual
The solution I use for this is when I'm doing art is to have the mask be several pixels larger than the original. This prevents the outline issue.
The reason this occurs is simple. Let's say we have a square that is pure solid black, except along it's edge is one pixel wide border where it's 50% opacity. When you copy and apply itself as a mask on the original, the square that was solid black (100% opacity) is completely gone. However with the 50% opacity border, it gets applied with a 50% opacity mask, making the border still visible, but only at 25% opacity.
This is why in the example image above I gave the outline for the bottom of the triangle is not there, because it was at 100% since it didn't have any Anti-aliasing.
Honestly I don't think there is a problem with how the emulator is handling the shadows. This is a problem with the game itself. At lower resolutions, and without AA, this issue was not a visible issue for the developers.
The only solution for this would be if someone came up with a game specific gamehack for it to have masking be larger. Or probably more simple, have masking be a rectangle/square that excludes the wanted shadow. That way it doesn't have to take in account of the other shadows, and only focuses on the needed one. I am curious as to why that isn't what is happening. Is it something to do with LogicOp? A case where the devs over thought a solution? I don't know, since as previously stated I don't have the programming knowledge to know for sure, so I could be wrong in what's actually going on. But this is how it happens in the various art programs out there.
Though I do find it odd that when you compare the two Star Fox Adventure images, the shadow where Fox's head is is messed up in the 'before fix' image. Maybe caused by layering multiple masks being applied to each other instead of the shadow. Like if you used a free select tool, and had it cross over itself multiple times, creating negative spots within a positive area.