Pixel10Pro vkCreateGraphicsPipelines very slow and cause a crash

1

When running our game on the Pixel 10 Pro, we encounter a situation where vkCreateGraphicsPipelines is extremely time-consuming and slow. Moreover, during PSO (Pipeline State Object) pre-compilation, memory usage keeps rising, eventually causing an abnormal crash at the stack trace shown below.

Below is the timing data viewed using Unreal Engine’s UnrealInsight tool. As you can see, vkCreateGraphicsPipelines takes over 400ms per call, making the entire pre-compilation process extremely slow.

image
image1920×1080 781 KB

Below is the crash stack trace — the crash occurs while compiling the 11,445th PipelineStateObject (out of a total of 22,035), at which point memory usage has reached 4,137 MB.

 	libc.so!abort()	unknown
 	libc.so!scudo::die()	unknown
 	libc.so!scudo::reportRawError()	unknown
 	libc.so!scudo::reportMapError()	unknown
 	libc.so!scudo::MemMapLinux::remapImpl()	unknown
 	libc.so!scudo::MapAllocator<scudo::SecondaryConfig<scudo::AndroidNormalConfig>>::allocate()	unknown
 	libc.so!scudo::Allocator<scudo::AndroidNormalConfig, &scudo_malloc_postinit>::allocate()	unknown
 	libc.so!scudo_malloc()	unknown
 	libc.so!malloc()	unknown
 	libusc.so!UscAllocGeneric()	unknown
 	libusc.so!FastAlloc()	unknown
 	libusc.so!ResizeArray()	unknown
 	libusc.so!SetDestCount()	unknown
 	libusc.so!SetOpcodeAndDestCount()	unknown
 	libusc.so!EmitTwoSrcInst()	unknown
 	libusc.so!ExtractField.llvm.9224253951859868103()	unknown
 	libusc.so!ExtractImageDimensions()	unknown
 	libusc.so!ConvertSamplerInstructionCore()	unknown
 	libusc.so!ConvertSamplerInstructionF32()	unknown
 	libusc.so!DoConvert()	unknown
 	libusc.so!BuildCFG()	unknown
 	libusc.so!ConvertToIntermediate()	unknown
 	libusc.so!PVRUniFlexCompileToHw()	unknown
 	vulkan.powervr.so!PipelineCompileFragmentStage()	unknown
 	vulkan.powervr.so!CompileGraphicsPipelinesShaderState(_DEVICE*, VkGraphicsPipelineCreateInfo const*, _PIPELINE_CACHE*, _ALLOCATION_CONTEXT*, _COMPILER_CONTEXT*, _GRAPHICS_PIPELINE*, VkPipelineCreationFeedbackCreateInfo*) ()	unknown
 	vulkan.powervr.so!CreateGraphicsPipeline(_DEVICE*, _ALLOCATION_CONTEXT*, _ALLOCATION_CONTEXT*, _COMPILER_CONTEXT*, _PIPELINE_CACHE*, VkGraphicsPipelineCreateInfo const*, _PIPELINE_LAYOUT*, _PIPELINE_LAYOUT*, _GRAPHICS_PIPELINE**) ()	unknown
 	vulkan.powervr.so!IMG_vkCreateGraphicsPipelines()	unknown
 	libUE4.so!FVulkanPipelineStateCacheManager::CreateVKPipeline(FVulkanRHIGraphicsPipelineState*, FVulkanShader**, VkGraphicsPipelineCreateInfo const&, bool, bool)::$_27::operator()(FVulkanRHIGraphicsPipelineState*, VkPipelineCache_T*) const(const (unnamed class) * this, FVulkanRHIGraphicsPipelineState * PSO, VkPipelineCache InPipelineCache) Line 1594	c++
 	libUE4.so!decltype(Forward<FVulkanPipelineStateCacheManager::CreateVKPipeline(FVulkanRHIGraphicsPipelineState*, FVulkanShader**, VkGraphicsPipelineCreateInfo const&, bool, bool)::$_27&>(fp)(Forward<FVulkanRHIGraphicsPipelineState*>(fp0), Forward<VkPipelineCache_T*>(fp0))) Invoke<FVulkanPipelineStateCacheManager::CreateVKPipeline(FVulkanRHIGraphicsPipelineState*, FVulkanShader**, VkGraphicsPipelineCreateInfo const&, bool, bool)::$_27&, FVulkanRHIGraphicsPipelineState*, VkPipelineCache_T*>((unnamed class) & Func, FVulkanRHIGraphicsPipelineState *&& Args, VkPipelineCache_T *&& Args) Line 50	c++
 	libUE4.so!UE4Function_Private::TFunctionRefCaller<FVulkanPipelineStateCacheManager::CreateVKPipeline(FVulkanRHIGraphicsPipelineState*, FVulkanShader**, VkGraphicsPipelineCreateInfo const&, bool, bool)::$_27, VkResult (FVulkanRHIGraphicsPipelineState*, VkPipelineCache_T*)>::Call(void * Obj, FVulkanRHIGraphicsPipelineState *& Params, VkPipelineCache_T *& Params) Line 538	c++
 	libUE4.so!VkResult FVulkanPipelineCacheChunk::CreatePSO<FVulkanRHIGraphicsPipelineState>(FVulkanPipelineCacheChunk * this, FVulkanRHIGraphicsPipelineState * PipelineState, FVulkanPipelineCacheChunk::EPSOCacheAccessType AccessType, TUniqueFunction<VkResult (FVulkanRHIGraphicsPipelineState *, VkPipelineCache_T *)> PSOCreateFunc)	c++
 	libUE4.so!VkResult FVulkanChunkedPipelineCacheManagerImpl::CreatePSO<FVulkanRHIGraphicsPipelineState>(FVulkanChunkedPipelineCacheManagerImpl * this, FVulkanRHIGraphicsPipelineState * PipelineState, bool bIsPrecompile, bool bAllowSkipDraw, TUniqueFunction<VkResult (FVulkanRHIGraphicsPipelineState *, VkPipelineCache_T *)> PSOCreateFunc) Line 1409	c++
 	libUE4.so!VkResult FVulkanChunkedPipelineCacheManager::CreatePSO<FVulkanRHIGraphicsPipelineState>(FVulkanChunkedPipelineCacheManager * this, FVulkanRHIGraphicsPipelineState * PipelineState, bool bIsPrecompileJob, bool bAllowSkipDraw, TUniqueFunction<VkResult (FVulkanRHIGraphicsPipelineState *, VkPipelineCache_T *)> PSOCreateFunc) Line 3184	c++
 	libUE4.so!FVulkanPipelineStateCacheManager::CreateVKPipeline(FVulkanPipelineStateCacheManager * this, FVulkanRHIGraphicsPipelineState * PSO, FVulkanShader ** Shaders, const VkGraphicsPipelineCreateInfo & PipelineInfo, bool bIsPrecompile, bool bAllowSkipDraw) Line 1587	c++
 	libUE4.so!FVulkanPipelineStateCacheManager::CreateGfxPipelineFromEntry(FVulkanPipelineStateCacheManager * this, FVulkanRHIGraphicsPipelineState * PSO, FVulkanShader ** Shaders, VkPipeline * Pipeline, bool bIsPrecompile, bool bAllowSkipDraw) Line 1550	c++
 	libUE4.so!FVulkanPipelineStateCacheManager::RHICreateGraphicsPipelineState(FVulkanPipelineStateCacheManager * this, const FGraphicsPipelineStateInitializer & Initializer) Line 2369	c++
>	libUE4.so!FVulkanDynamicRHI::RHICreateGraphicsPipelineState(FVulkanDynamicRHI * this, const FGraphicsPipelineStateInitializer & PSOInitializer) Line 2476	c++
 	libUE4.so!RHICreateGraphicsPipelineState(const FGraphicsPipelineStateInitializer & Initializer) Line 1767	c++
 	libUE4.so!FCompilePipelineStateTask::CompilePSO(FCompilePipelineStateTask * this) Line 2044	c++

Why is the compilation so slow? and Why does it trigger a Malloc crash? This phone has 16 GB of RAM.

2

Hello @shaoshuai ,

It is definitely strange that the device is crashing in this fashion. From the backtrace, it is clear that it is stuck on the compiler trying to digest the shaders and something is going wrong.

In order to debug this properly, I would recommend you to use our tools, specifically PVRTune and PVRCarbon, to diagnose and fix this issue.

  1. Use PVRCarbon to record a trace of the game on a device where this game build works properly, you can use the trimming feature to narrow it down to one or two frames, thus creating a deterministic workload and removing the engine component our of the equation.
  2. Use PVRCarbon playback to run the trace on GP10+, confirming that the issue is replicated properly.
  3. Use PVRCarbon analysis to inspect all the shaders that are used on this two frames. You can export them and create simplified versions of them to allow you discriminate which shader is actually causing the issue (sometimes is not as obvious).
  4. Onced you’ve identified the shader that causes the crash, investigate what line is exactly causing the issue, this is probably due to a certain expression/s that combined together are making our LLVM optimisation stages to iterate too much, or bounce back and forth, causing the malloc issue.
  5. Try to change the expression, or perhaps simplify it.

While doing all this, you can use PVRTune to asses what the driver is doing internally (user mode, kernel mode and firmware). Note on here, if you are part of a game company (even a small studio), consider getting a NDA with us, you can then have access to PVRTune Complete version for free, to have an absolutely full picture of what the GPU is doing.

Hope this helps,

Javi.