CPU friendly texture loading / acquire texture multithreading?

[csyshing]

Hi,

I am doing experiment in maya-usd to see if we can load textures in multithreading way, the first problem I encountered is that when calling MHWRender::MTextureManager::acquireTiledTexture() in multithreading, Maya seems always crash with message like this:

V2Renderer : OGS WARNING:  param types don't match:  roughness   0
Prevented attempt to load texture "USD_ALab_0730/fragment/look/surfacing/chemistry_test_tube01/render_high/texture/look_surfacing_chemistry_test_tube01_render_high_texture/chemistry_test_tube01_BaseColor.1001.jpg" due to lack of GPU memory.
Prevented attempt to load texture "USD_ALab_0730/fragment/look/surfacing/electronics_portable_tv01/render_high/texture/look_surfacing_electronics_portable_tv01_render_high_texture/electronics_portable_tv01_BaseColor.1001.jpg" due to lack of GPU memory.
Prevented attempt to load texture "USD_ALab_0730/fragment/look/surfacing/electronics_portable_tv01/render_high/texture/look_surfacing_electronics_portable_tv01_render_high_texture/electronics_portable_tv01_BaseColor.1002.jpg" due to lack of GPU memory.
...
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff7371e700 (LWP 138734)]
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007ffff76413d6 in OGSTargetManager::ReleaseTarget(OGS::Devices::ATarget*, bool, bool) () at /local/S/Maya_2022PFix_DI/src/Maya/src/Components/OGS/SDK/Include/internal/OGS/Objects/ObjectModel.h:2275
#2  0x00007ffff7c0e950 in OGSMayaFileTexturePacker::pack(TfileTexturePackerInput&, TfileTexturePackerData&, bool) () at /local/S/Maya_2022PFix_DI/src/Maya/src/OGSMayaBridge/Utilities/OGSMayaFileTexturePacker.cpp:659
#3  0x00007fffa64f9d27 in Autodesk::Maya::OpenMaya20220000::MHWRender::MTextureManager::acquireTiledTexture(Autodesk::Maya::OpenMaya20220000::MString, Autodesk::Maya::OpenMaya20220000::MStringArray const&, Autodesk::Maya::OpenMaya20220000::MFloatArray const&, Autodesk::Maya::OpenMaya20220000::MColor const&, unsigned int, unsigned int, Autodesk::Maya::OpenMaya20220000::MStringArray&, Autodesk::Maya::OpenMaya20220000::MFloatArray&) () at /local/S/Maya_2022PFix_DI/src/Maya/src/OpenMayaRender/Render/MTextureManager.cpp:1279
#4  0x00007ffe062f0f0c in _LoadUdimTexture (uvScaleOffset=..., isColorSpaceSRGB=<synthetic pointer>, path="USD_ALab_0730/fragment/look/surfacing/chemistry_test_tube03/render_high/texture/look_surfacing_chemistry_test_tube03_render_high_texture/chemistr"..., 
    this=<optimized out>) at maya-usd/lib/mayaUsd/render/vp2RenderDelegate/material.cpp:830
#5  _LoadTexture (node=..., uvScaleOffset=..., isColorSpaceSRGB=<synthetic pointer>, path="USD_ALab_0730/fragment/look/surfacing/chemistry_test_tube03/render_high/texture/look_surfacing_chemistry_test_tube03_render_high_texture/chemistr"..., this=<optimized out>)
    at maya-usd/lib/mayaUsd/render/vp2RenderDelegate/material.cpp:854
#6  pxrInternal_v0_21__pxrReserved__::HdVP2Material::_AcquireTexture (this=this@entry=0x16a1ef50, path="USD_ALab_0730/fragment/look/surfacing/chemistry_test_tube03/render_high/texture/look_surfacing_chemistry_test_tube03_render_high_texture/chemistr"..., node=...)
    at maya-usd/lib/mayaUsd/render/vp2RenderDelegate/material.cpp:2334
#7  0x00007ffe062f1946 in pxrInternal_v0_21__pxrReserved__::HdVP2Material::_UpdateShaderInstance (this=this@entry=0x16a1ef50, mat=...) at maya-usd/lib/mayaUsd/render/vp2RenderDelegate/material.cpp:2229
#8  0x00007ffe062f5ed0 in pxrInternal_v0_21__pxrReserved__::HdVP2Material::Sync (this=0x16a1ef50, sceneDelegate=0x166ff890, dirtyBits=0x167b9c48) at maya-usd/lib/mayaUsd/render/vp2RenderDelegate/material.cpp:1361
....

I am not familiar with MTextureManage class, I have a few questions:

• Is MHWRender::MTextureManager::acquireTiledTexture() thread-safe?
• For the crash above, is it because of GPU memory reached the limit thus does not allow running more that one acquireTiledTexture() in the same time?
• If there was a CPU friendly version of acquireTiledTexture(), would that be enough to run multithreading?

Thanks,
Zhicheng

[williamkrick]

Hi @csyshing,

MHWRender::MTextureManager::acquireTiledTexture() is not thread safe. In general none of Maya's API functions are thread safe, and this particular function is especially not thread safe. I don't think there is a way to safely load textures in parallel through Maya's API.

The crash occurs when trying to clean up some temporary targets created when loading the tiled texture.

A CPU friendly version of acquireTiledTexture() would remove a major source of thread-unsafety from the code, accessing the underlying graphics API, so it would be a step towards a multithreaded texture load. However, speculating about that hypothetical API is not really possible because of how different it would be from existing APIs.

Is this work an effort to improve the performance of loading ALab?

[csyshing]

Yes @williamkrick , I was trying to improve the performance of loading ALab, the trace shows that most of the time is on running acquireTiledTexture(), attempting to call acquireTiledTexture() in secondary thread would cause Maya to crash immediately.

[williamkrick]

Unfortunately we don't have a simple answer for #1624. Working on this is near the top of our priority list so I'm hopefully we can spend some time on it soon.

There is a related issue I logged against USD PixarAnimationStudios/OpenUSD#1597 which would alleviate some of the problem loading the scene. Loading the textures for the render materials will still be slow but we'd only have to load those textures when the user chooses to see the render version of the objects.

[csyshing]

@williamkrick , I am doing tests that run acquireTiledTexture() and acquireTexture() in low priority via MGlobal::executeTaskOnIdle(), that seems to be able to load textures in background without crashing Maya while keeping Maya interactive in foreground, thinking it might be an alternative to improve user experience? Any potential technical problem/risk you can think of?

[williamkrick]

That is a great idea and it should work fine. The main thing to watch out for would be if the interactive session modifies the scene while the low priority task is still running then don't crash. For example if a file -new occurs, or the plugin gets unloaded, or the rprim gets renamed or re-parented or disabled etc. then the background task needs to understand the changed state and do the right thing.

[csyshing]

I implemented a prototype like this:

• load textures in background
• once finished, update the texture object MHWRender::MTexture* in _textureMap in HdVP2Material class
• mark the sprim (the material) as dirty:

sceneDelegate->GetRenderIndex().GetChangeTracker().MarkSprimDirty(sprimId, HdMaterial::DirtyResource)

• the corresponding rprim (the mesh) would be marked as dirty later on:

  maya-usd/lib/mayaUsd/render/vp2RenderDelegate/material.cpp

  Lines 1271 to 1274 in 43fe072

  	_UpdateShaderInstance(bxdfNet);
  	#ifdef HDVP2_MATERIAL_CONSOLIDATION_UPDATE_WORKAROUND
  	_MaterialChanged(sceneDelegate);
  	#endif

But it does not seem to render the prims as expected:

When I switch the purpose to "render" (around 0:38 second in the GIF above), I do see the textures showing up correctly, switching back to "proxy" does not seem to trigger any redrawing.

Then I realized that, ALab put the prims as instances, e.g. fragment/assembly/modelling/thelab01/assembly_modelling_thelab01.usda :

def Xform "root" (
    kind = "assembly"
)
{
    def Xform "workbench_grp" (
        kind = "group"
    )
    {
        def Xform "furniture_workbench01_0001" (
            instanceable = true
            prepend references = @../../../../entity/furniture_workbench01/furniture_workbench01.usda@
        )
        {
            double3 xformOp:translate = (0, 3.088914144757659, 0)
            uniform token[] xformOpOrder = ["xformOp:translate"]
        }

        def Xform "electronics_oscilloscope01_0001" (
            instanceable = true
            prepend references = @../../../../entity/electronics_oscilloscope01/electronics_oscilloscope01.usda@
        )
        {
            float3 xformOp:rotateXYZ = (-16.271652, -55.71754, 1.1477762)
            double3 xformOp:translate = (152.53024886222218, 111.14867513515668, 20.336049721017872)
            uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:rotateXYZ"]
        }
....
....

I manually edited the usda to turn off the instanceable scene to be false: instanceable = false, and tried again:

The textures once finished loading, seem to show up one by one as expected (starting around 0:21 second in GIF above).

So looks like there is a step missing to inform maya-usd to refresh/reload the material for instances but I am not sure where it could be, @williamkrick , do you have any hints I could look into? Thanks!

[williamkrick]

Cool!

What version of Maya are you working with? HDVP2_MATERIAL_CONSOLIDATION_UPDATE_WORKAROUND is true for 2022 but not for 2023, so there could be different behavior in those two versions.

Assuming you are using 2022, my first guess would be that there is a breakdown in communication between the instancer and the instance prototype. The HdVP2Material holds a list of _materialSubscriptions, which appears to be a list of rprims that the material is assigned to. In the instanced case is it the path to the HdVP2Instancer that is in there, or the path to the instance prototype? Maybe the dirtiness is getting to the HdVP2Instancer and is not passed on to the prototype correctly.

Do we need the rprim sync for the drawing to correctly update? Something simple like modifying the diffuse color of a material shouldn't require an rprim sync, but I'm sure we could create a shader where changing a parameter did require a sync.

If you are using 2023 there is a system in place in Maya which, after a call to MShaderInstance::setParameter, will trigger all the shaders to update. But I don't see how that would tie back in to MayaUSD and trigger an rprim Sync. Maybe something is missing there. If modifying the material changed the required geometry streams I don't see an opportunity for it to be added. It could be there and I'm just missing it.

I guess my advice is focus on HdVP2Instancer. That's what is different in MayaUSD when instancing is on, so maybe there is a problem there.

[csyshing]

Hi @williamkrick , I am using Maya 2022, I think I found the bug that instances do not get updated:

maya-usd/lib/mayaUsd/render/vp2RenderDelegate/mesh.cpp

Lines 1862 to 1865 in 14b93b8

	if (shader != nullptr && shader != drawItemData._shader) {
	drawItemData._shader = shader;
	drawItemData._shaderIsFallback = false;
	stateToCommit._shader = shader;

The if condition tests against the drawItemData._shader but I believe the intention is to reset both drawItemData._shader (L1863) and stateToCommit._shader = shader (L1865), so it's better to count in stateToCommit._shader, I created a draft PR-1980 for discussion, please have a look and let me know!

[williamkrick]

I think I see what you are saying. Is it true that when we hit line 1862 the state is drawItemData._shader matches shader, but stateToCommit._shader has something in it which doesn't match, and so we end up with unexpected behavior? If that is the case then your change makes sense.

I'm looking at the rest of the draft review now.