Bug 运行久了出现了错误

[fiancee123]

问题描述 / Problem Description

onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running ReorderOutput node. Name:'ReorderOutput_token_221' Status Message: GetElementType is not implemented

result_ocr, elapse = yolomodels.OCRengine(image)
File "/usr/local/python311/lib/python3.11/site-packages/rapidocr_onnxruntime
dt_boxes, det_elapse = self.auto_text_det(img)
File "/usr/local/python311/lib/python3.11/site-packages/rapidocr_onnxruntime /main.py", line 156, in auto_text_det
dt_boxes, det_elapse = self.text_det(img)
File "/usr/local/python311/lib/python3.11/site-packages/rapidocr_onnxruntime/ch_ppocr_det/text_detect.py", line 56, in call
Nov 12 06:40:12 VM-12-17-centos uwsgi[613753]: preds = self.infer(prepro_img)[0]
Nov 12 06:40:12 VM-12-17-centos uwsgi[613753]: ^^^^^^^^^^^^^^^^^^^^^^
Nov 12 06:40:12 VM-12-17-centos uwsgi[613753]: File "/usr/local/python311/lib/python3.11/site-packages/rapidocr_onnxruntime/utils/infer_engine.py", line 199, in call
Nov 12 06:40:12 VM-12-17-centos uwsgi[613753]: raise ONNXRuntimeError(error_info) from e
Nov 12 06:40:12 VM-12-17-centos uwsgi[613753]: rapidocr_onnxruntime.utils.infer_engine.ONNXRuntimeError: Traceback (most recent call last):
Nov 12 06:40:12 VM-12-17-centos uwsgi[613753]: File "/usr/local/python311/lib/python3.11/site-packages/rapidocr_onnxruntime/utils/infer_engine.py", line 196, in call
Nov 12 06:40:12 VM-12-17-centos uwsgi[613753]: return self.session.run(self.get_output_names(), input_dict)
Nov 12 06:40:12 VM-12-17-centos uwsgi[613753]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 12 06:40:12 VM-12-17-centos uwsgi[613753]: File "/usr/local/python311/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 220, in run
Nov 12 06:40:12 VM-12-17-centos uwsgi[613753]: return self._sess.run(output_names, input_feed, run_options)
Nov 12 06:40:12 VM-12-17-centos uwsgi[613753]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 12 06:40:12 VM-12-17-centos uwsgi[613753]: onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running ReorderOutput node. Name:'ReorderOutput_token_221' Status Message: GetElementType is not implemented

运行环境 / Runtime Environment

python 3.11 Linux

复现代码 / Reproduction Code

可能解决方案 / Possible solutions

[SWHL]

请给出具体运行环境和运行代码，越详细越好

[fiancee123]

代码：
@api_bp.route("/rapidocr", methods=['POST'])
@jwt_required(fresh=True)
def doOcr():
data = json.loads(request.data)
pic = data.get("pic", "")
image = decode_base64_image(pic)
result_ocr, elapse = yolomodels.OCRengine(image)
result_data = []
if result_ocr:
for ocr_res in result_ocr:
xxyy = ocr_res[0]
centerX = (xxyy[0][0] + xxyy[2][0]) / 2
centerY = (xxyy[0][1] + xxyy[2][1]) / 2
result_data.append({"center": (int(centerX), int(centerY)), "text": ocr_res[1], "Confidence": str(ocr_res[2])})
return {'code': 200, 'data': result_data}

运行在 centos 9 stream ， python 3.11， 运行了一天之后出现错误:onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running ReorderOutput node. Name:'ReorderOutput_token_221' Status Message: GetElementType is not implemented,

[SWHL]

看一下环境下onnxruntime版本，如果是1.19.x，建议降到1.18试试。
相关issue #22075

[fiancee123]

还有一个问题 运行久了 内存一直飙升是什么问题

[fiancee123]

准确说是内存缓慢上升

[SWHL]

这个是已知问题，暂时定时重启程序试试

[fiancee123]

请给出具体运行环境和运行代码，越详细越好

Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: The above exception was the direct cause of the following exception:
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: Traceback (most recent call last):
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: File "/usr/local/python311/lib/python3.11/site-packages/flask/app.py", line 1473, in wsgi_app
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: response = self.full_dispatch_request()
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: File "/usr/local/python311/lib/python3.11/site-packages/flask/app.py", line 882, in full_dispatch_request
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: rv = self.handle_user_exception(e)
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: File "/usr/local/python311/lib/python3.11/site-packages/flask_cors/extension.py", line 194, in wrapped_function
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: return cors_after_request(app.make_response(f(*args, **kwargs)))
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: ^^^^^^^^^^^^^^^^^^
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: File "/usr/local/python311/lib/python3.11/site-packages/flask/app.py", line 880, in full_dispatch_request
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: rv = self.dispatch_request()
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: ^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: File "/usr/local/python311/lib/python3.11/site-packages/flask/app.py", line 865, in dispatch_request
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: File "/usr/local/python311/lib/python3.11/site-packages/flask_cors/decorator.py", line 130, in wrapped_function
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: resp = make_response(f(*args, **kwargs))
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: ^^^^^^^^^^^^^^^^^^
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: File "/usr/local/python311/lib/python3.11/site-packages/flask_limiter/extension.py", line 1303, in __inner
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: return cast(R, flask.current_app.ensure_sync(obj)(*a, **k))
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: File "/usr/local/python311/lib/python3.11/site-packages/flask_jwt_extended/view_decorators.py", line 170, in decorator
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: return current_app.ensure_sync(fn)(*args, **kwargs)
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: File "/usr/app/view.py", line 651, in doOcr
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: result_ocr, elapse = OCRengine(image)
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: File "/usr/local/python311/lib/python3.11/site-packages/rapidocr_onnxruntime/main.py", line 96, in call
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: dt_boxes, det_elapse = self.auto_text_det(img)
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: ^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: File "/usr/local/python311/lib/python3.11/site-packages/rapidocr_onnxruntime/main.py", line 156, in auto_text_det
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: dt_boxes, det_elapse = self.text_det(img)
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: ^^^^^^^^^^^^^^^^^^
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: File "/usr/local/python311/lib/python3.11/site-packages/rapidocr_onnxruntime/ch_ppocr_det/text_detect.py", line 56, in call
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: preds = self.infer(prepro_img)[0]
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: ^^^^^^^^^^^^^^^^^^^^^^
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: File "/usr/local/python311/lib/python3.11/site-packages/rapidocr_onnxruntime/utils/infer_engine.py", line 199, in call
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: raise ONNXRuntimeError(error_info) from e
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: rapidocr_onnxruntime.utils.infer_engine.ONNXRuntimeError: Traceback (most recent call last):
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: File "/usr/local/python311/lib/python3.11/site-packages/rapidocr_onnxruntime/utils/infer_engine.py", line 196, in call
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: return self.session.run(self.get_output_names(), input_dict)
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: File "/usr/local/python311/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 220, in run
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: return self._sess.run(output_names, input_feed, run_options)
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Conv node. Name:'batch_norm_0.tmp_4_nchwc' Status Message: GetElementType is not implemented
Nov 14 00:01:06 VM-12-17-centos uwsgi[3112532]: [pid: 3112532|app: 0|req: 1151/4927] 114.132.238.39 () {38 vars in 972 bytes} [Thu Nov 14 00:01:02 2024] POST /ocr => generated 265 bytes in 3602 msecs (HTTP/1.0 500) 3 headers in 131 bytes (2 switches on core 98)
Nov 14 00:01:07 VM-12-17-centos uwsgi[3112530]: worker 4 buried after 2 seconds
Nov 14 00:01:07 VM-12-17-centos uwsgi[3112530]: worker 5 buried after 2 seconds

[SWHL]

方便的话，请在环境下单独跑RapidOCR程序，不要跑Flask包装的，看看是否还有问题。