I'm trying to do some image processing on the Nokia N9 (SGX 530). In a simple first step I want to convert a RGB image to grayscale with the following fragment shader:
uniform sampler2D rgbImage;
varying highp vec2 texCoord0;
highp vec4 color = texture2D(rgbImage, texCoord0);
highp float luminance = 0.23 * color.r + 0.59 * color.g + 0.11 * color.b;
gl_FragColor = vec4(luminance, 0.0, 0.0, 0.0);
However, probably due to lower precision some pixels of the resulting grayscale image differ from the result of the same shader running in PVR GLES2 emulation on a desktop GPU. The differences are -1 or +1. If I leave out the add operations and only inspect how e.g. the red channel is modified by 0.23 * color.r, similar errors occur. I also tried converting and rounding the texel values to integers in range [0 255] before multiplying the constant, multiplying then, rounding to the nearest integer again, and then dividing by 255 before setting gl_FragColor. Still, those +/-1 errors kept occuring.
Since highp floats are supposed to have 32 bit precision , this sort of confuses me. Am I missing something here? Is this expected behavior?
This is what glGetShaderPrecisionFormat() returns for GL_FRAGMENT_SHADER on the device:
GL_LOW_FLOAT: range: [1 1], precision: 8
GL_MEDIUM_FLOAT: range: [14 14], precision: 10
GL_HIGH_FLOAT: range: [126 126], precision: 23
Alright, this seems to be expected behavior. The PVR GLES2 emulator kind of reports 24 bits of precision for my desktop GPU. I reproduced the problem calculating the values in question by hand, each for a 23 and for a 24 bit significand, and it really came down to rounding errors. After all, that's obvious, but since I'm not too savvy with floating point arithmetic I couldn't believe it first...