I'm trying to do some image processing on the Nokia N9 (SGX 530). In a simple first step I want to convert a RGB image to grayscale with the following fragment shader:
uniform sampler2D rgbImage;
varying highp vec2 texCoord0;
void main()
{
highp vec4 color = texture2D(rgbImage, texCoord0);
highp float luminance = 0.23 * color.r + 0.59 * color.g + 0.11 * color.b;
gl_FragColor = vec4(luminance, 0.0, 0.0, 0.0);
}
However, probably due to lower precision some pixels of the resulting grayscale image differ from the result of the same shader running in PVR GLES2 emulation on a desktop GPU. The differences are -1 or +1. If I leave out the add operations and only inspect how e.g. the red channel is modified by 0.23 * color.r, similar errors occur. I also tried converting and rounding the texel values to integers in range [0 255] before multiplying the constant, multiplying then, rounding to the nearest integer again, and then dividing by 255 before setting gl_FragColor. Still, those +/-1 errors kept occuring.
Since highp floats are supposed to have 32 bit precision [1], this sort of confuses me. Am I missing something here? Is this expected behavior?
This is what glGetShaderPrecisionFormat() returns for GL_FRAGMENT_SHADER on the device:
GL_LOW_FLOAT: range: [1 1], precision: 8
GL_MEDIUM_FLOAT: range: [14 14], precision: 10
GL_HIGH_FLOAT: range: [126 126], precision: 23
[1] https://www.imgtec.com/powervr/insider/docs/PowerVR%20SGX.OpenGL%20ES%202.0%20Application%20Development%20Recommendations.1.8f.External.pdf
bert2011-11-16 22:24:33
Alright, this seems to be expected behavior. The PVR GLES2 emulator kind of reports 24 bits of precision for my desktop GPU. I reproduced the problem calculating the values in question by hand, each for a 23 and for a 24 bit significand, and it really came down to rounding errors. After all, that's obvious, but since I'm not too savvy with floating point arithmetic I couldn't believe it first...
bert2011-11-16 23:35:43
