Imagination PowerVR SDK Blog

Fragment shader: highp precision issue


I'm trying to do some image processing on the Nokia N9 (SGX 530). In a simple first step I want to convert a RGB image to grayscale with the following fragment shader:

    uniform sampler2D rgbImage;

    varying highp vec2 texCoord0;

    void main()


        highp vec4 color = texture2D(rgbImage, texCoord0);

        highp float luminance = 0.23 * color.r + 0.59 * color.g + 0.11 * color.b;

        gl_FragColor = vec4(luminance, 0.0, 0.0, 0.0);


However, probably due to lower precision some pixels of the resulting grayscale image differ from the result of the same shader running in PVR GLES2 emulation on a desktop GPU. The differences are -1 or +1. If I leave out the add operations and only inspect how e.g. the red channel is modified by 0.23 * color.r, similar errors occur. I also tried converting and rounding the texel values to integers in range [0 255] before multiplying the constant, multiplying then, rounding to the nearest integer again, and then dividing by 255 before setting gl_FragColor. Still, those +/-1 errors kept occuring.

Since highp floats are supposed to have 32 bit precision [1], this sort of confuses me. Am I missing something here? Is this expected behavior?

This is what glGetShaderPrecisionFormat() returns for GL_FRAGMENT_SHADER on the device:

    GL_LOW_FLOAT:    range: [1 1], precision: 8
    GL_MEDIUM_FLOAT: range: [14 14], precision: 10
    GL_HIGH_FLOAT:   range: [126 126], precision: 23

bert2011-11-16 22:24:33


Alright, this seems to be expected behavior. The PVR GLES2 emulator kind of reports 24 bits of precision for my desktop GPU. I reproduced the problem calculating the values in question by hand, each for a 23 and for a 24 bit significand, and it really came down to rounding errors. After all, that's obvious, but since I'm not too savvy with floating point arithmetic I couldn't believe it first...
bert2011-11-16 23:35:43