I've two textures 1024 * 1024. One is stored as RGB (~3MB) and second is stored as
palletised texture (1024 * 1024 indices + 512 palette (u16, R5G6B5), ~1MB). My question is
how much video memory would be used to store each of this textures (3mb + 1mb or 3mb + 3mb)?
Both texture formats (RGB8 and palette) are not natively supported by POWERVR MBX. The driver will most likely convert the RGB8 texture to RGBA8, thus using 4 MiB for a 1024x1024 mip level. The palletised texture will be expanded to RGB565 and thus take up 2 MiB for a 1024x1024 mip level. So in total they should consume 6 MiB of video memory.
To save texture memory you should use PVRTC compressed textures wherever possible and only use 32-bit textures if 16-bit doesn’t yield acceptable quality.