Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

An error when training #6

Open
Scxw010516 opened this issue Apr 17, 2023 · 0 comments
Open

An error when training #6

Scxw010516 opened this issue Apr 17, 2023 · 0 comments

Comments

@Scxw010516
Copy link

Thank you for your great work!
When I run the training code, I get an error like the following

Traceback (most recent call last):
  File "C:\Apps\Python39\lib\site-packages\torch\multiprocessing\spawn.py", line 69, in _wrap
    fn(i, *args)
  File "D:\Projects\GRAM\train.py", line 33, in train
    training_process(rank, world_size, opt, device)
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "C:\Apps\Python39\lib\site-packages\torch\nn\parallel\distributed.py", line 1110, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])  # type: ignore[index]
  File "C:\Apps\Python39\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Projects\GRAM\generators\generators.py", line 112, in forward
    img, _ = self.renderer.render(self._intersections, self._volume(z, truncation_psi), img_size, camera_origin, camera_pos, fov, ray_start, ray_end, z.device)
  File "D:\Projects\GRAM\generators\renderers\manifold_renderer.py", line 146, in render
    transformed_points_sample, transformed_ray_directions, transformed_ray_origins, _ = transform_sampled_points(points_cam, rays_d_cam, camera_origin, camera_pos, device=device)
  File "D:\Projects\GRAM\generators\renderers\manifold_renderer.py", line 92, in transform_sampled_points
    cam2world_matrix = create_cam2world_matrix(forward_vector, camera_origin, device=device)
  File "D:\Projects\GRAM\generators\renderers\manifold_renderer.py", line 123, in create_cam2world_matrix
    cam2world = translation_matrix @ rotation_matrix
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`

Please help solve,thank you very much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant