Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[MACE v1.0.0] rpcmem_init failed on GPU #709

Closed
gasgallo opened this issue Jan 25, 2021 · 5 comments · May be fixed by #710
Closed

[MACE v1.0.0] rpcmem_init failed on GPU #709

gasgallo opened this issue Jan 25, 2021 · 5 comments · May be fixed by #710

Comments

@gasgallo
Copy link
Contributor

gasgallo commented Jan 25, 2021 •
edited
Loading

Before you open an issue, please make sure you have tried the following steps:

  1. Make sure your environment is the same with (https://mace.readthedocs.io/en/latest/installation/env_requirement.html).
  2. Have you ever read the document for your usage?
  3. Check if your issue appears in HOW-TO-DEBUG or FAQ.
  4. The form below must be filled.

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
  • NDK version(e.g., 15c): 19c
  • GCC version(if compiling for host, e.g., 5.4.0): 5.4.0
  • MACE version (Use the command: git describe --long --tags): 1.0.0
  • Python version(2.7): 3.6
  • Bazel version (e.g., 0.13.0): 0.16.0
  • CMake version: 3.16.0

Model deploy file (*.yml)

library_name: model
target_abis: [arm64-v8a, armeabi-v7a]
model_graph_format: code
model_data_format: code

models:
  model:
    platform: onnx
    model_file_path: "/models/cb487a80da4a4431a846be8a0d73e5b5.onnx"
    model_sha256_checksum: ba2fa1b0d4850a1831327c98c56c97acc5bce949fd2a344b2713d4ee1e35a742
    subgraphs:
      - input_tensors:
          - data
        input_shapes:
          - 1,3,192,192
        input_data_formats:
          - NCHW
        output_tensors:
          - mobilepose0_conv1_fwd
        output_shapes:
          - 1,128,6,6
    obfuscate: 0
    runtime: cpu+gpu
    winograd: 2

Describe the problem

Model fails to run on GPU with openCL with below error.

To Reproduce

Steps to reproduce the problem:

1. cd /path/to/mace
2. python3 tools/converter.py convert --config=/path/to/yml
3. python3 tools/converter.py run --config=/path/to/yml

Error information / logs

Please include the full log and/or traceback here.

WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading: 
Loading: 0 packages loaded
WARNING: The major revision of the Android NDK referenced by android_ndk_repository rule 'androidndk' is 19. The major revisions supported by Bazel are [10, 11, 12, 13, 14, 15, 16]. Bazel will attempt to treat the NDK as if it was r16. This may cause compilation and linkage problems. Please download a supported NDK version.
Analyzing: target //mace/tools:mace_run_static (2 packages loaded)
Analyzing: target //mace/tools:mace_run_static (11 packages loaded)
INFO: Analysed target //mace/tools:mace_run_static (33 packages loaded).
INFO: Found 1 target...
[0 / 13] [-----] BazelWorkspaceStatusAction stable-status.txt
[29 / 205] no action
[191 / 237] [-----] Creating source manifest for //mace/tools:mace_run_static
[207 / 718] Compiling mace/ops/reverse.cc; 0s local ... (6 actions running)
[212 / 718] Compiling mace/ops/group_norm.cc; 2s local ... (6 actions running)
[242 / 718] Compiling mace/ops/ref/deconv_2d.cc; 1s local ... (3 actions, 2 running)
[573 / 718] Compiling mace/ops/opencl/buffer/softmax.cc; 1s local ... (5 actions running)
[581 / 718] Compiling mace/ops/opencl/image/resize_bicubic.cc; 2s local ... (5 actions running)
[591 / 718] Compiling mace/ops/opencl/image/extract_image_patches.cc; 2s local ... (3 actions running)
[598 / 718] Compiling mace/ops/opencl/image/eltwise.cc; 2s local ... (6 actions running)
[610 / 718] Compiling mace/ops/opencl/buffer/conv_2d_general.cc; 1s local ... (6 actions running)
[629 / 718] Compiling mace/ops/opencl/image/depthwise_deconv2d.cc; 1s local ... (5 actions running)
[650 / 718] Compiling mace/ops/arm/base/gemm.cc; 1s local ... (5 actions running)
[709 / 721] Compiling mace/libmace/mace.cc; 1s local ... (6 actions, 5 running)
[729 / 740] Compiling mace/ops/addn.cc; 2s local ... (5 actions running)
[754 / 762] Compiling mace/ops/eltwise.cc; 4s local ... (5 actions running)
[780 / 781] [-----] Linking mace/tools/mace_run_static
Target //mace/tools:mace_run_static up-to-date:
  bazel-bin/mace/tools/mace_run_static
INFO: Elapsed time: 76.118s, Critical Path: 6.67s
INFO: 191 processes: 191 local.
INFO: Build completed successfully, 194 total actions
INFO: Build completed successfully, 194 total actions
CMD> bazel version
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Build label: 0.16.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Jul 31 17:01:24 2018 (1533056484)
Build timestamp: 1533056484
Build timestamp as int: 1533056484
CMD> bazel build //mace/proto:mace_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //mace/proto:mace_py (12 packages loaded)
INFO: Analysed target //mace/proto:mace_py (17 packages loaded).
INFO: Found 1 target...
[2 / 9] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //mace/proto:mace_py up-to-date:
bazel-genfiles/mace/proto/mace_pb2.py
INFO: Elapsed time: 1.681s, Critical Path: 0.03s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
CMD> cp -f bazel-genfiles/mace/proto/mace_pb2.py tools/python/py_proto
CMD> bazel build //mace/proto:micro_mem_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //mace/proto:micro_mem_py (12 packages loaded)
INFO: Analysed target //mace/proto:micro_mem_py (17 packages loaded).
INFO: Found 1 target...
[0 / 4] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //mace/proto:micro_mem_py up-to-date:
bazel-genfiles/mace/proto/micro_mem_pb2.py
INFO: Elapsed time: 1.648s, Critical Path: 0.05s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
CMD> cp -f bazel-genfiles/mace/proto/micro_mem_pb2.py tools/python/py_proto
CMD> bazel build //third_party/caffe:caffe_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //third_party/caffe:caffe_py (12 packages loaded)
INFO: Analysed target //third_party/caffe:caffe_py (17 packages loaded).
INFO: Found 1 target...
[0 / 5] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //third_party/caffe:caffe_py up-to-date:
bazel-genfiles/third_party/caffe/caffe_pb2.py
INFO: Elapsed time: 1.709s, Critical Path: 0.05s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
CMD> cp -f bazel-genfiles/third_party/caffe/caffe_pb2.py tools/python/py_proto
tools/converter.py:319: Downloading model, please wait ...
tools/converter.py:289: 
Downloaded successfully.
* Build //mace/tools:mace_run_static with ABI arm64-v8a
('build', '//mace/tools:mace_run_static', '--config', 'android', '--cpu=arm64-v8a', '--define', 'neon=true', '--define', 'opencl=true', '--define', 'quantize=false', '--define', 'bfloat16=false', '--define', 'fp16=false', '--define', 'rpcmem=true', '--define', 'hexagon=false', '--define', 'hta=false', '--define', 'apu=false', '--config', 'optimization', '--config', 'symbol_hidden', '--per_file_copt=mace/tools/mace_run.cc@-DMODEL_GRAPH_FORMAT_CODE')
('build', '//mace/tools:mace_run_static', '--config', 'android', '--cpu=arm64-v8a', '--define', 'neon=true', '--define', 'opencl=true', '--define', 'quantize=false', '--define', 'bfloat16=false', '--define', 'fp16=false', '--define', 'rpcmem=true', '--define', 'hexagon=false', '--define', 'hta=false', '--define', 'apu=false', '--config', 'optimization', '--config', 'symbol_hidden', '--per_file_copt=mace/tools/mace_run.cc@-DMODEL_GRAPH_FORMAT_CODE')
Build done!
******************************************************
          Run model model on TPS980P          
******************************************************
Generate input file:  build/model/_tmp/model/52ccb254544535bc260dea8cc3fcac30/TPS980P_rk3399/arm64-v8a/model_input_data
Generate input file done.
* Run 'model' with round=100, restart_round=1, tuning=False, out_of_range_check=False, num_threads=(-1,), cpu_affinity_policy=(1,), gpu_perf_hint=(3,), gpu_priority_hint=(3,)
Push build/model/_tmp/model/52ccb254544535bc260dea8cc3fcac30/TPS980P_rk3399/arm64-v8a/model_input_data to /data/local/tmp/mace_run
Push third_party/nnlib/arm64-v8a/libhexagon_controller.so to /data/local/tmp/mace_run
Push build/model/_tmp/arm64-v8a/mace_run_static to /data/local/tmp/mace_run
Push /tmp/cmd_file-model-1610601805.2930682 to /data/local/tmp/mace_run/cmd_file-model-1610601805.2930682
I mace/tools/mace_run.cc:544] model name: model
I mace/tools/mace_run.cc:545] mace version: v1.0.0-0-g0945e49
I mace/tools/mace_run.cc:546] input node: data
I mace/tools/mace_run.cc:547] input shape: 1,3,192,192
I mace/tools/mace_run.cc:548] output node: mobilepose0_conv1_fwd
I mace/tools/mace_run.cc:549] output shape: 1,128,6,6
I mace/tools/mace_run.cc:550] input_file: /data/local/tmp/mace_run/model_input
I mace/tools/mace_run.cc:551] output_file: /data/local/tmp/mace_run/model_out
I mace/tools/mace_run.cc:552] input dir:
I mace/tools/mace_run.cc:553] output dir:
I mace/tools/mace_run.cc:554] model_data_file:
I mace/tools/mace_run.cc:555] model_file:
I mace/tools/mace_run.cc:556] apu_cache_policy: 0
I mace/tools/mace_run.cc:557] apu_binary_file:
I mace/tools/mace_run.cc:558] apu_storage_file:
I mace/tools/mace_run.cc:559] device: CPU
I mace/tools/mace_run.cc:560] round: 100
I mace/tools/mace_run.cc:561] restart_round: 1
I mace/tools/mace_run.cc:562] gpu_perf_hint: 3
I mace/tools/mace_run.cc:563] gpu_priority_hint: 3
I mace/tools/mace_run.cc:564] num_threads: -1
I mace/tools/mace_run.cc:565] cpu_affinity_policy: 1
I mace/tools/mace_run.cc:568] limit_opencl_kernel_time: 0
I mace/tools/mace_run.cc:573] opencl_queue_window_size: 0
I mace/libmace/mace.cc:558] Creating MaceEngine, MACE version: v1.0.0-0-g0945e49
I mace/libmace/mace.cc:628] Initializing MaceEngine
I mace/libmace/mace.cc:790] Destroying MaceEngine
I mace/tools/mace_run.cc:616] restart round 0
I mace/libmace/mace.cc:558] Creating MaceEngine, MACE version: v1.0.0-0-g0945e49
I mace/libmace/mace.cc:628] Initializing MaceEngine
I mace/tools/mace_run.cc:286] Create Mace Engine latency: 49.954 ms
I mace/tools/mace_run.cc:293] Total init latency: 50.098 ms
I mace/tools/mace_run.cc:387] Warm up run
I mace/tools/mace_run.cc:423] 1st warm up run latency: 106.728 ms
I mace/tools/mace_run.cc:431] Run model
I mace/tools/mace_run.cc:493] Average latency: 65.5742 ms
I mace/tools/mace_run.cc:508] Write output file /data/local/tmp/mace_run/model_out_mobilepose0_conv1_fwd with size 18432 done.
========================================================
     capability(CPU)        init      warmup     run_avg
========================================================
time          52.470      50.098     106.728      65.574
I mace/libmace/mace.cc:790] Destroying MaceEngine
Running finished!
Dana service is not available.
* Run 'model' with round=100, restart_round=1, tuning=False, out_of_range_check=False, num_threads=(-1,), cpu_affinity_policy=(1,), gpu_perf_hint=(3,), gpu_priority_hint=(3,)
Push build/landmark/_tmp/model/52ccb254544535bc260dea8cc3fcac30/TPS980P_rk3399/arm64-v8a/model_input_data to /data/local/tmp/mace_run
Push third_party/nnlib/arm64-v8a/libhexagon_controller.so to /data/local/tmp/mace_run
Push build/landmark/_tmp/arm64-v8a/mace_run_static to /data/local/tmp/mace_run
Push /tmp/cmd_file-model-1610601817.087663 to /data/local/tmp/mace_run/cmd_file-model-1610601817.087663
I mace/tools/mace_run.cc:544] model name: model
I mace/tools/mace_run.cc:545] mace version: v1.0.0-0-g0945e49
I mace/tools/mace_run.cc:546] input node: data
I mace/tools/mace_run.cc:547] input shape: 1,3,192,192
I mace/tools/mace_run.cc:548] output node: mobilepose0_conv1_fwd
I mace/tools/mace_run.cc:549] output shape: 1,128,6,6
I mace/tools/mace_run.cc:550] input_file: /data/local/tmp/mace_run/model_input
I mace/tools/mace_run.cc:551] output_file: /data/local/tmp/mace_run/model_out
I mace/tools/mace_run.cc:552] input dir:
I mace/tools/mace_run.cc:553] output dir:
I mace/tools/mace_run.cc:554] model_data_file:
I mace/tools/mace_run.cc:555] model_file:
I mace/tools/mace_run.cc:556] apu_cache_policy: 0
I mace/tools/mace_run.cc:557] apu_binary_file:
I mace/tools/mace_run.cc:558] apu_storage_file:
I mace/tools/mace_run.cc:559] device: GPU
I mace/tools/mace_run.cc:560] round: 100
I mace/tools/mace_run.cc:561] restart_round: 1
I mace/tools/mace_run.cc:562] gpu_perf_hint: 3
I mace/tools/mace_run.cc:563] gpu_priority_hint: 3
I mace/tools/mace_run.cc:564] num_threads: -1
I mace/tools/mace_run.cc:565] cpu_affinity_policy: 1
I mace/tools/mace_run.cc:568] limit_opencl_kernel_time: 0
I mace/tools/mace_run.cc:573] opencl_queue_window_size: 0
I mace/libmace/mace.cc:558] Creating MaceEngine, MACE version: v1.0.0-0-g0945e49
I mace/libmace/mace.cc:628] Initializing MaceEngine
I mace/libmace/mace.cc:790] Destroying MaceEngine
I mace/tools/mace_run.cc:616] restart round 0
W ./mace/utils/tuner.h:201] Failed to read tuned param file: /data/local/tmp/mace_run/model_tuned_opencl_parameter.device.rk3399.bin
I mace/libmace/mace.cc:558] Creating MaceEngine, MACE version: v1.0.0-0-g0945e49
W mace/core/kv_storage.cc:109] Failed to read kv store file: /data/local/tmp/mace_run/interior//mace_cl_compiled_program.bin
W mace/core/runtime/opencl/opencl_runtime.cc:442] Load OpenCL cached compiled kernel file failed. Please make sure the storage directory exist and you have Write&Read permission
F mace/core/rpcmem.cc:23] Check failed: rm.flag == 0 rpcmem_init failed!
F mace/core/rpcmem.cc:23] backtrace:
F mace/core/rpcmem.cc:23]  pc 0x58cdeee2c4
F mace/core/rpcmem.cc:23]  pc 0x58cdef03a0
F mace/core/rpcmem.cc:23]  pc 0x58cdef0354
F mace/core/rpcmem.cc:23]  pc 0x58cdef0550
F mace/core/rpcmem.cc:23]  pc 0x58cdef05ac
F mace/core/rpcmem.cc:23]  pc 0x58cdee1148
F mace/core/rpcmem.cc:23]  pc 0x58cded0164
F mace/core/rpcmem.cc:23]  pc 0x58cdecff30
F mace/core/rpcmem.cc:23]  pc 0x58cde0ab68
F mace/core/rpcmem.cc:23]  pc 0x58cde0edec
F mace/core/rpcmem.cc:23]  pc 0x58cde0edb8
F mace/core/rpcmem.cc:23]  pc 0x58cddd10e0
F mace/core/rpcmem.cc:23]  pc 0x58cddd1a04
F mace/core/rpcmem.cc:23]  pc 0x58cddd422c
F mace/core/rpcmem.cc:23]  pc 0x58cddd43c4
F mace/core/rpcmem.cc:23]  pc 0x7d26d04590 __libc_init
F mace/core/rpcmem.cc:23]  pc 0x58cddd0f40
Aborted

Additional context

Add any other context about the problem here, e.g., what you have modified about the code.
@lee-bin
Copy link
Member

lee-bin commented Jan 25, 2021

@gasgallo rpcmem is a qualcomm library that is enabled by default, which should not happen on other soc. You can disable it by modifying this line https://github.com/XiaoMi/mace/blob/v1.0.0/tools/sh_commands.py#L275

BTW, if the model's output_data_formats is NCHW, it should be specified in the yml.

@gasgallo
Copy link
Contributor Author

@lee-bin thanks for the hint! So enable_rpcmem=True should default to False instead? Is this a bug?

@lee-bin
Copy link
Member

lee-bin commented Jan 25, 2021

Yes, it should be False by default to run on all the platforms. You can consider it a bug, we seldom test android devices other than Qualcomm or MTK.

@gasgallo gasgallo mentioned this issue Jan 26, 2021
Open
@gasgallo
Copy link
Contributor Author

@lee-bin can you check #710?

@lu229
Copy link
Collaborator

lu229 commented Mar 8, 2021

@gasgallo We have reconstructed the MACE framework and resolve this bug, thanks for your contribution!

@lu229 lu229 closed this as completed Mar 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Default rpc_mem to false for bazel gasgallo/mace
3 participants
@lu229 @lee-bin @gasgallo