Dolphin, the GameCube and Wii emulator - Forums - Shader compilation still failing under Linux

Mentioned this bug a couple of revisions ago. Tested again and still the same error. Emulator under Linux unusable since no graphics can be rendered:

Code:
29:10:448 Source/Plugins/Plugin_VideoOGL/Src/PixelShaderCache.cpp:240 E[Video]: Failed to compile ps (0) : warning C6502: Profile option 'NumInstructionSlots' value (-1) too small; clamped to 0

(0) : warning C6502: Profile option 'NumTexInstructionSlots' value (-1) too small; clamped to 0

(0) : warning C6502: Profile option 'NumMathInstructionSlots' value (-1) too small; clamped to 0

(0) : warning C6503: Profile option 'MaxDrawBuffers' value (8) too large; clamped to 4

(0) : warning C6502: Profile option 'NumInstructionSlots' value (-1) too small; clamped to 0

(0) : warning C6502: Profile option 'NumTexInstructionSlots' value (-1) too small; clamped to 0

(0) : warning C6502: Profile option 'NumMathInstructionSlots' value (-1) too small; clamped to 0

(0) : warning C6503: Profile option 'MaxDrawBuffers' value (8) too large; clamped to 4

(0) : error C6002: Instruction limit of 0 exceeded; 10 instructions needed to compile program

:                                    

29:10:448 Source/Plugins/Plugin_VideoOGL/Src/PixelShaderCache.cpp:241 E[Video]: uniform samplerRECT samp0 : register(s0);

void main(                                  

  out float4 ocol0 : COLOR0,                    

  in float2 uv0 : TEXCOORD0)                           

{                                    

  float2 uv1 = float2(uv0.x + 1.0f, uv0.y);   

  float3 c0 = texRECT(samp0, uv0).rgb;           

  float3 c1 = texRECT(samp0, uv1).rgb;           

  float3 y_const = float3(0.257f,0.504f,0.098f);                            

  float3 u_const = float3(-0.148f,-0.291f,0.439f);                             

  float3 v_const = float3(0.439f,-0.368f,-0.071f);                             

  float4 const3 = float4(0.0625f,0.5f,0.0625f,0.5f);                               

  float3 c01 = (c0 + c1) * 0.5f;          

  ocol0 = float4(dot(c1,y_const),dot(c01,u_const),dot(c0,y_const),dot(c01, v_const)) + const3;

}

OpenGL caps-flags are still queried blindly as can be seen from the "value (-1) too small;" lines. Would be nice if somebody could fix this as it turns the Linux build literally into a paper weight.

EDIT: Forgot to say. All shaders fail to compile due to this error.

EDIT: EDIT: Besides... why the hell are you using Cg? If you are using OpenGL use GLSL instead as this does not require an error-prone shader compiler at run-time.

Took a closer look at the problem. In the code cgGLSetOptimalOptions is used to set profile options. This is bad since the Cg compiler is known to be buggy. A bug was introduced in Cg2.1's handling of very large profile option values so this will not work on ATI. ATI returns MAXINT = 2147483647 (0x7fffffff) which is a correct value in OpenGL but Cg fails to handle it properly. As a result -1 is used by Cg resulting (signed-ness bug most probably) and compilation fails. To get around this problem this function call should not be used. Furthermore the value queried manually has to be checked and adjusted. I'm using these hacks and got it working. Mind this is a hack but Cg by itself is one huge hack so breaking it more than it is already broken is not that much of a problem:

Code:
// PixelShaderCache.cpp@88 ; insert

    s_nMaxPixelInstructions = 4096; // bugged, MAXINT not supported

Code:
// VertexShaderCache.cpp@125 ; insert

    s_nMaxVertexInstructions = 4096; // bugged, MAXINT not supported

Code:
//Render.cpp@365 ; replace two lines

    // A bug was introduced in Cg2.1's handling of very large profile option values

    // so this will not work on ATI. ATI returns MAXINT = 2147483647 (0x7fffffff)

    // which is correct in OpenGL but Cg fails to handle it properly. As a result

    // -1 is used by Cg resulting (signedness incorrect) and compilation fails.

    //cgGLSetOptimalOptions(g_cgvProf);

    //cgGLSetOptimalOptions(g_cgfProf);

A better solution would be to check if the value is above 4096 (attention, it's unsigned int!) and to clamp it to this value. I doubt any shader is going to require more than this number of instructions.

Would be nice if some dever can state if they took notice of this problem. It's a show stopper and for the time being this hack would be enough until a better fix is found.

(04-17-2010, 08:42 AM)Dragonlord Wrote: [ -> ]Took a closer look at the problem. In the code cgGLSetOptimalOptions is used to set profile options. This is bad since the Cg compiler is known to be buggy. A bug was introduced in Cg2.1's handling of very large profile option values so this will not work on ATI. ATI returns MAXINT = 2147483647 (0x7fffffff) which is a correct value in OpenGL but Cg fails to handle it properly. As a result -1 is used by Cg resulting (signed-ness bug most probably) and compilation fails. To get around this problem this function call should not be used. Furthermore the value queried manually has to be checked and adjusted. I'm using these hacks and got it working. Mind this is a hack but Cg by itself is one huge hack so breaking it more than it is already broken is not that much of a problem:

Code:
// PixelShaderCache.cpp@88 ; insert     s_nMaxPixelInstructions = 4096; // bugged, MAXINT not supported
Code:
// VertexShaderCache.cpp@125 ; insert     s_nMaxVertexInstructions = 4096; // bugged, MAXINT not supported
Code:
//Render.cpp@365 ; replace two lines     // A bug was introduced in Cg2.1's handling of very large profile option values     // so this will not work on ATI. ATI returns MAXINT = 2147483647 (0x7fffffff)     // which is correct in OpenGL but Cg fails to handle it properly. As a result     // -1 is used by Cg resulting (signedness incorrect) and compilation fails.     //cgGLSetOptimalOptions(g_cgvProf);     //cgGLSetOptimalOptions(g_cgfProf);

A better solution would be to check if the value is above 4096 (attention, it's unsigned int!) and to clamp it to this value. I doubt any shader is going to require more than this number of instructions.

Hello DragonLord,
Code evolves, bugs remain. I'm trying to apply your patches but I'm not sure certain if the line numbers are still the same in the last builds.
Could you expose the code context (method names, 2 lines before and after the modification) ?
thx

Why don't you update to CG 2.2 ?
This particular bug is now fixed.

Sorry, didn't see the topic update (so much for e-mail notification <.=.<).

@sl1nk3
CG 2.2 is not stable and not in portage. The last unstable version is 2.1.17 . As far as I know though the bug is still present in the newer versions. Game-Dev forums recommend to simply not use cgGLSetOptimalOptions not only for this bug but also for other reasons. I don't know more in detail about the problem since I use only standardized libs (aka GLSL) not vendor specific hack-libs. I could only get dolphin running on all test systems I could get my hand on by compiling SVN with the hacks applied.

@yomat
I don't have the source on this machine. One you can find by grep since the lines are unique. The others are right after the OpenGL parameters are queried. So they are a hardcoded upper bound. Should be done with an if-clause properly.

Dragonlord: I committed your fix as r5830, since other people were also experiencing that problem. Please test it when you got some time.

Dolphin, the GameCube and Wii emulator - Forums

Dragonlord

Dragonlord

Dragonlord

yomat

sl1nk3

Dragonlord

Jack Frost