Imagination PowerVR SDK Blog

Video textures in iOS 4?


#1

Hi!

I learned about the GL_APPLE_rgb_422 OpenGL ES extension by Apple ( http://www.opengl.org/registry/specs/APPLE/rgb_422.txt ),  which is apparently supported by iOS 4 and which should support video texture decoding. Thus, I got curious and learned on this page:

http://developer.apple.com/iphone/library/documentation/GraphicsImaging/Reference/CoreVideoRef/Reference/reference.html

that CoreVideo is supported by iOS 4.

Now, I never have done any Apple development, but according to this page:

http://developer.apple.com/mac/library/documentation/GraphicsImaging/Conceptual/CoreVideo/CVProg_Intro/CVProg_Intro.html

Core Video allows you to send CVImageBuffers to OpenGL textures. (And with functions like captureStillImageAsynchronouslyFromConnection in AVCaptureStillImageOutput you apparently can get at least still images from an input source.)

I know that this is out of the scope of PowerVR, but can anyone confirm that it is already possible to get video images into OpenGL ES on iOS 4 devices?

Thanks


#2

Just a follow-up to myself:

The issue is also discussed here (including most of the links below): https://www.imgtec.com/forum/forum_posts.asp?TID=1015

If supported (not on iOS 4.1), the best way to go is to use GL_IMG_texture_stream2. But I couldn’t find any official documentation of it. Texas Instruments is providing code that uses it under Linux: http://software-dl.ti.com/dsps/dsps_public_sw/apps_processors/OMAP35x_AM35x_Video_Texture_Streaming/1_0/index_FDS.html

On iOS, one apparently has to get a pointer to the video data in main memory (see here: http://cmgresearch.blogspot.com/2010/10/augmented-reality-on-iphone-with-ios40.html or here: http://developer.apple.com/library/ios/#qa/qa2010/qa1702.html ) and then load it into a texture with glTexImage2D and possibly do any conversion in a fragment shader (see here: http://developer.bada.com/blog/?p=373 but that particular implementation appears to be quite inefficient since it uses several calls to glTexImage2D )

Thus, currently there are no video textures on iOS 4.1 but the performance might still be OK because it boils down to a call to glTexImage2D. Things would probably be much easier if Apple would implement the full CoreVideo framework on iOS (the OpenGL part is missing on iOS) or integrate the GL_IMG_texture_stream2 extension.