training more iterations crash

[qbwdp]

If I train mscnn_8s_768_trainval model in second stage for more iterations with lr=0.0005, it will crash at about 80000+ iteration as following :

I0323 16:15:44.452236 18769 sgd_solver.cpp:106] Iteration 81875, lr = 0.0005
I0323 16:17:07.957013 18769 proposal_target_layer.cpp:181] sampled rois: 51, random rois: 77
F0323 16:17:07.964038 18782 math_functions.cu:79] Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered
*** Check failure stack trace: ***
F0323 16:17:07.964064 18769 math_functions.cu:79] Check failed: error == cudaSuccess (77 vs. 0) an illegal memory access was encountered
*** Check failure stack trace: ***
@ 0x7fef502c2daa (unknown)
@ 0x7fef502c2daa (unknown)
@ 0x7fef502c2ce4 (unknown)
@ 0x7fef502c2ce4 (unknown)
@ 0x7fef502c26e6 (unknown)
@ 0x7fef502c26e6 (unknown)
@ 0x7fef502c5687 (unknown)
@ 0x7fef502c5687 (unknown)
@ 0x7fef50acf078 caffe::caffe_gpu_memcpy()
@ 0x7fef50acf078 caffe::caffe_gpu_memcpy()
@ 0x7fef5091ce1e caffe::SyncedMemory::to_gpu()
@ 0x7fef5091c870 caffe::SyncedMemory::cpu_data()
@ 0x7fef5091c2c9 caffe::SyncedMemory::gpu_data()
@ 0x7fef5091db12 caffe::Blob<>::cpu_data()
@ 0x7fef5091dda2 caffe::Blob<>::gpu_data()

I think ROIPooling has illegal memory access, how to solve this problem?