scipy.spatial.qhull.QhullError: QH6019 qhull input error

[lingchen1991]

[ ] 0/1496, elapsed: 0s, ETA:Traceback (most recent call last):
File "tools/train.py", line 143, in
main()
File "tools/train.py", line 139, in main
meta=meta)
File "/home/endlessplato/Dense-RepPoints/mmdet/apis/train.py", line 102, in train_detector
meta=meta)
File "/home/endlessplato/Dense-RepPoints/mmdet/apis/train.py", line 181, in _dist_train
runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
File "/home/endlessplato/anaconda3/envs/mmdetection/lib/python3.7/site-packages/mmcv/runner/runner.py", line 383, in run
epoch_runner(data_loaders[i], **kwargs)
File "/home/endlessplato/anaconda3/envs/mmdetection/lib/python3.7/site-packages/mmcv/runner/runner.py", line 292, in train
self.call_hook('after_train_epoch')
File "/home/endlessplato/anaconda3/envs/mmdetection/lib/python3.7/site-packages/mmcv/runner/runner.py", line 245, in call_hook
getattr(hook, fn_name)(self)
File "/home/endlessplato/Dense-RepPoints/mmdet/core/evaluation/eval_hooks.py", line 41, in after_train_epoch
gpu_collect=self.gpu_collect)
File "/home/endlessplato/Dense-RepPoints/mmdet/apis/test.py", line 58, in multi_gpu_test
result = model(return_loss=False, rescale=True, **data)
File "/home/endlessplato/anaconda3/envs/mmdetection/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/home/endlessplato/anaconda3/envs/mmdetection/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 442, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/home/endlessplato/anaconda3/envs/mmdetection/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/home/endlessplato/Dense-RepPoints/mmdet/core/fp16/decorators.py", line 49, in new_func
return old_func(*args, **kwargs)
File "/home/endlessplato/Dense-RepPoints/mmdet/models/detectors/base.py", line 149, in forward
return self.forward_test(img, img_metas, **kwargs)
File "/home/endlessplato/Dense-RepPoints/mmdet/models/detectors/base.py", line 130, in forward_test
return self.simple_test(imgs[0], img_metas[0], **kwargs)
File "/home/endlessplato/Dense-RepPoints/mmdet/models/detectors/dense_reppoints_detector.py", line 62, in simple_test
self.test_cfg, ori_shape, scale_factor, rescale)
File "/home/endlessplato/Dense-RepPoints/mmdet/models/detectors/dense_reppoints_detector.py", line 117, in get_seg_masks
bbox_mask = scipy.interpolate.griddata(im_pts, im_pts_score, grids)
File "/home/endlessplato/anaconda3/envs/mmdetection/lib/python3.7/site-packages/scipy/interpolate/ndgriddata.py", line 221, in griddata
rescale=rescale)
File "interpnd.pyx", line 248, in scipy.interpolate.interpnd.LinearNDInterpolator.init
File "qhull.pyx", line 1839, in scipy.spatial.qhull.Delaunay.init
File "qhull.pyx", line 357, in scipy.spatial.qhull._Qhull.init
scipy.spatial.qhull.QhullError: QH6019 qhull input error (qh_scalelast): can not scale last coordinate to [ 0, 0]. Input is cocircular or cospherical. Use option 'Qz' to add a point at infinity.

While executing: | qhull d Qbb Qc Qt Q12 Qz
Options selected for Qhull 2019.1.r 2019/06/21:
run-id 1267007510 delaunay Qbbound-last Qcoplanar-keep Qtriangulate
Q12-allow-wide Qz-infinity-point _pre-merge _zero-centrum Qinterior-keep
Pgood _maxoutside 0

==============================================

I use dist_train.sh to train the model on 5 GPUs. It worked well on first several epochs. But at the beginning of the forth epoch's validation, this error jump up.

May I ask for some suggetions to sovle it?