Fine terrain crashes when using cuda_terrain.gin

[lorjul]

Describe the bug

Adding cuda_terrain.gin to the pipeline configs results in a crash during fine_terrain, regardless of the seed.

Steps to Reproduce

What version of the code were you using?

commit 5968578213b24a6666bb3170836abe3f29a45275 (HEAD -> main, tag: v1.12.0, origin/main, origin/HEAD)
Merge: e254428b 7744dc98
Author: Alex Raistrick <[email protected]>
Date:   Sun Dec 22 08:47:54 2024 +0000

    Merge pull request #333 from princeton-vl/develop

    Release v1.11.4 and v1.12.0 to public main

What command did you run?

I installed Infinigen like this:

git clone https://github.com/princeton-vl/infinigen.git
cd infinigen
conda create --name infinigen python=3.11
conda activate infinigen

pip install -e ".[terrain,vis]"

bash scripts/install/compile_terrain.bash

Initially, I used the manage_jobs script but the following two lines reproduce the same error:

python -m infinigen_examples.generate_indoors -- --output_folder /data/out/debug/3aca81f2/coarse --seed 3aca81f2 --task coarse --task_uniqname coarse -g coast singleroom.gin -p LOG_DIR='/data/out/debug/3aca81f2/logs' camera.spawn_camera_rigs.n_camera_rigs=30 compose_indoors.restrict_single_supported_roomtype=True 'execute_tasks.frame_range=[1, 48]' 'execute_tasks.camera_id=[0, 0]'

python -m infinigen_examples.generate_indoors -- --input_folder /data/out/debug/3aca81f2/coarse --output_folder /data/out/debug/3aca81f2/fine --seed 3aca81f2 --task fine_terrain --task_uniqname fineterrain -g coast singleroom.gin -p LOG_DIR='/data/out/debug/3aca81f2/logs' Terrain.device='cuda' camera.spawn_camera_rigs.n_camera_rigs=30 compose_indoors.restrict_single_supported_roomtype=True 'execute_tasks.frame_range=[1, 48]' 'execute_tasks.camera_id=[0, 0]'

What are your FULL output logs?

stderr output from the second command: fineterrain.err.txt

The exception excerpt from the log file:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/data/user/isg/infinigen/infinigen_examples/generate_indoors.py", line 573, in <module>
    main(args)
  File "/data/user/isg/infinigen/infinigen_examples/generate_indoors.py", line 509, in main
    execute_tasks.main(
  File "/data/user/isg/infinigen/infinigen/core/execute_tasks.py", line 368, in main
    execute_tasks(
  File "/data/user/isg/igconda/lib/python3.11/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/data/user/isg/igconda/lib/python3.11/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/data/user/isg/igconda/lib/python3.11/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/user/isg/infinigen/infinigen/core/execute_tasks.py", line 275, in execute_tasks
    terrain.fine_terrain(
  File "/data/user/isg/igconda/lib/python3.11/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/data/user/isg/igconda/lib/python3.11/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/data/user/isg/igconda/lib/python3.11/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/user/isg/infinigen/infinigen/terrain/core.py", line 470, in fine_terrain
    fine_meshes, _ = self.export(mesher_backend=mesher_backend, cameras=cameras)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/user/isg/infinigen/infinigen/terrain/core.py", line 322, in export
    surface_kernel(meshes_dict[mesh_name])
  File "/data/user/isg/infinigen/infinigen/terrain/surface_kernel/core.py", line 134, in __call__
    ret[var] *= params.vertex_attributes[self.attribute].reshape(
                ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'beach'
  In call to configurable 'fine_terrain' (<function Terrain.fine_terrain at 0x7923ba130ea0>)
  In call to configurable 'execute_tasks' (<function execute_tasks at 0x7923b4729800>)

Platform

• OS & OS Version: Ubuntu 24.04
• GPU: 1 RTX 4090
• GPU Driver Version: 550.127.08
• RAM (GB): 64 GB

Additional context

The described bug occurs with any seed. If I disable cuda for terrain generation, it works. Note that rendering using CUDA works; only terrain seems to be a problem.

[lorjul]

I just discovered that this bug was already reported in #370 (although the importance of cuda_terrain.gin was not highlighted). I hope this report provides some additional information.
Feel free to close this issue if you want to track it only once.