Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RuntimeError: There were no tensor arguments to this function #218

Open
DongGuangchang opened this issue Mar 22, 2022 · 1 comment
Open

Comments

@DongGuangchang
Copy link

Traceback (most recent call last):
File "tools/train.py", line 125, in
main()
File "tools/train.py", line 121, in main
timestamp=timestamp)
File "/home/dy/dong/instanceSegmentation/SOLO-master/mmdet/apis/train.py", line 111, in train_detector
timestamp=timestamp)
File "/home/dy/dong/instanceSegmentation/SOLO-master/mmdet/apis/train.py", line 297, in _non_dist_train
runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
File "/home/dy/software/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/runner.py", line 364, in run
epoch_runner(data_loaders[i], **kwargs)
File "/home/dy/software/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/runner/runner.py", line 268, in train
self.model, data_batch, train_mode=True, **kwargs)
File "/home/dy/dong/instanceSegmentation/SOLO-master/mmdet/apis/train.py", line 78, in batch_processor
losses = model(**data)
File "/home/dy/software/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/dy/software/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/dy/software/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/dy/dong/instanceSegmentation/SOLO-master/mmdet/core/fp16/decorators.py", line 49, in new_func
return old_func(*args, **kwargs)
File "/home/dy/dong/instanceSegmentation/SOLO-master/mmdet/models/detectors/base.py", line 142, in forward
return self.forward_train(img, img_meta, **kwargs)
File "/home/dy/dong/instanceSegmentation/SOLO-master/mmdet/models/detectors/single_stage_ins.py", line 78, in forward_train
*loss_inputs, gt_bboxes_ignore=gt_bboxes_ignore)
File "/home/dy/dong/instanceSegmentation/SOLO-master/mmdet/models/anchor_heads/solov2_head.py", line 249, in loss
loss_ins = torch.cat(loss_ins).mean()
RuntimeError: There were no tensor arguments to this function (e.g., you passed an empty list of Tensors), but no fallback function is registered for schema aten::_cat. This usually means that this function requires a non-empty list of Tensors. Available functions are [CUDATensorId, CPUTensorId, VariableTensorId]

I encountered a problem during training. How can I solve it? thanks!

@biendltb
Copy link

biendltb commented Jun 14, 2022

I hit the same issue. The problem is that there were empty predictions from the model for that batch which returns an empty list of losses. Not sure how this could be solved.
This is a temporary solution that you could try (added to SOLO/mmdet/models/anchor_heads/solov2_head.py:249):

        if len(loss_ins) == 0:
            loss_ins = torch.tensor(0.0, device=gt_bbox_list[0].device)
        else:
            loss_ins = torch.cat(loss_ins).mean()
            loss_ins = loss_ins * self.ins_loss_weight

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants