Rk3588 NPU support

[Klinefelters]

Hello there!

I'm currently using ultralytics yolov8 and love that you can export models but still run them in the ultralytics library with the appropriate backend. I was wondering if there are any plans on expanding this to the rknn-toolkit2, and the RK3588 series cpu's with the built in npu acceleration.

The cpu is on many boards that are growing in popularity, (Rock pi series, Orange Pi 5, etc...) and it would be a great addition for using yolov8 in integrated systems at cheaper prices than the nvidia jetson boards.

[JoshuaJakowlew]

Hi!
Does anyone have working guide how to convert yolov8n to rknn model?

[fanandli]

https://github.com/rockchip-linux/rknn-toolkit2/blob/master/doc/

[StuartIanNaylor]

Quite a few have tried on the Radxa site and seems to cause much problems.
https://forum.radxa.com/t/use-yolov8-in-rk3588-npu/15838
Translating this guide may help https://blog.csdn.net/m0_57315535/article/details/128250096
I haven't bothered as generally I hate that style of framework where much framework knowledge is required.
I stick to what I know with tflite int8 and occasionally the delegate for ArmNN as the GPU also can be used.

My results are tacked on the end as was late to the party.
I don't think many are enabling the reserved memory area for the NPU that holds weight files.
The NPU is actually a 3core 2Tops with ratings for int8 likely with the model in reserved memory (read the docs)
Its a bit like the Coral TPU boards that depending on operations and datatypes exporting can be a huge chore.
Needs 1 guru to do it and document, so fingers crossed.

Strangely people seem to underestimate the cpu that with armv8.2 and vector mat/mul the R5/Opi5&Rpi5 all can manage ML > x6 than a Pi4 which brings quite a few models in scope.
Tflite scaling is not linear and likely better to run on single cores in a round robin scenario as on 4 cores and I may have made a mistake I can get 133 FPS on the cpu. I prob picked the wrong tflite and need to check actual results as the fully quant step version is near x2 faster than the end one, so maybe 65 FPS.

[pderrenger]

@StuartIanNaylor it's great to see the enthusiasm for integrating YOLOv8 with various hardware accelerators like the RK3588 NPU. While we currently don't have official support for the rknn-toolkit2, we understand the importance of such integrations for the community.

For those looking to convert YOLOv8 models to RKNN, the process typically involves exporting the model to ONNX and then converting it to the RKNN format. However, as you've noticed, this can be quite challenging and may require in-depth knowledge of the frameworks involved.

We appreciate the shared experiences and attempts from the community on the Radxa forum and other platforms. These discussions are valuable for everyone trying to navigate the complexities of model conversion and optimization for specific hardware.

If you're encountering issues with the conversion process, I recommend reviewing the rknn-toolkit2 documentation thoroughly and reaching out to their support channels for assistance. Additionally, keep an eye on community forums, as it often takes a collective effort to streamline such processes.

In the meantime, we'll take note of the interest in RK3588 NPU support for future considerations. Thank you for your input and for being a part of the YOLOv8 community! 🚀

[danielealbano]

@pderrenger if can be of any help, the rnnk2-toolkit offers a simulator that runs on the PC, so it's possible to test the conversion process end-to-end without the need of an actual RK3588(s) SoC.

Firefly has some docs about it
https://wiki.t-firefly.com/en/ROC-RK3588S-PC/usage_npu.html

[pderrenger]

@danielealbano, thanks for sharing that helpful tip! 🌟 The rknn2-toolkit simulator indeed offers a great way to test the YOLOv8 model conversion without needing the actual RK3588 hardware. This can be a real time-saver and helps in debugging before deployment.

For those interested, here's a quick example of how you might start with the simulator:

# Assuming you've already installed rknn-toolkit2
from rknn.api import RKNN

# Initialize RKNN object
rknn = RKNN()

# Load your YOLOv8 ONNX model
rknn.load_onnx(model='path_to_yolov8.onnx')

# Build model (targeting RK3588 for this example)
rknn.build(do_quantization=False, dataset='./dataset.txt', target_platform='rk3588')

# Init runtime environment
print('--> Init runtime environment')
ret = rknn.init_runtime(target='rk3588', device_id='0')
if ret != 0:
    print('Init runtime environment failed')
    exit(ret)
print('done')

# Inference
print('--> Running model')
outputs = rknn.inference(inputs=[input_data])
print('done')

# Handle your outputs here
print(outputs)

rknn.release()

Make sure to check out the Firefly documentation for more details on setting up and using the simulator: Firefly RK3588 NPU Usage.

Happy coding! 🚀

[samarth1232]

Hii,
I'm exploring running machine learning models on NPU RK3588. Could anyone share which ML frameworks are supported for this NPU? Additionally, I'd like to know how I can command the NPU to exclusively run models using these frameworks. Any insights or resources would be greatly appreciated!

[glenn-jocher]

Hi @samarth1232,

Thank you for reaching out! It looks like you're encountering a ModuleNotFoundError related to the rknn_log module within the rknn-toolkit2 package.

Steps to Resolve the Issue

1. Verify Installation:
   Ensure that you have installed the correct versions of rknn-toolkit2 and rknn-toolkit-lite. Sometimes, version mismatches can cause such issues. You can reinstall the packages using:

   pip uninstall rknn-toolkit2 rknn-toolkit-lite
   pip install rknn-toolkit2 rknn-toolkit-lite

2. Check Python Environment:
   Make sure that you are using the correct Python environment where rknn-toolkit2 is installed. You can verify this by running:

   pip show rknn-toolkit2
   pip show rknn-toolkit-lite

3. Directory Structure:
   Double-check the directory structure of the rknn-toolkit2 installation. Ensure that the rknn_log module exists in the expected location. The directory should look something like this:

   rknn/
   ├── api/
   │   ├── __init__.py
   │   ├── rknn.py
   │   ├── rknn_log.py
   

4. Update Packages:
   Ensure that all your packages are up-to-date. Sometimes, dependencies can cause issues if they are not aligned. You can update all packages using:

   pip list --outdated
   pip install --upgrade <package_name>

Providing a Reproducible Example

If the issue persists, it would be very helpful if you could provide a minimum reproducible example. This will allow us to better understand the context and help you more effectively. You can find guidance on creating a reproducible example here: Minimum Reproducible Example.

Latest Versions

Please ensure that you are using the latest versions of the rknn-toolkit2 and rknn-toolkit-lite packages. Sometimes, issues are resolved in newer releases.

If you continue to face issues, feel free to share more details, and we'll do our best to assist you further. 😊

[samarth1232]

Hi @glenn-jocher as you suggested I reinstalled everything properly but still Getting same error
Error:

raceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
import(pkg_name)
File "/home/vicharak/env1/lib/python3.10/site-packages/rknn/api/init.py", line 1, in
from rknn.api.rknn import RKNN
File "/home/vicharak/env1/lib/python3.10/site-packages/rknn/api/rknn.py", line 6, in
from .rknn_log import set_log_level_and_file_path
ModuleNotFoundError: No module named 'rknn.api.rknn_log'

Directory Structure: is like
rknn/
├── api/
│  ├── __init__.py
 |    ├── rknn_log.cpython-310-x86_64-linux-gnu.so
 |    ├── rknn_platform.cpython-310-x86_64-linux-gnu.so
 |    ├── rknn.py
 |    ├── rknn_runtime.cpython-310-x86_64-linux-gnu.so
 |    ├── rknn_utils.cpython-310-x86_64-linux-gnu.so

I have shared object file not directly python files

[glenn-jocher]

Hi @samarth1232,

Thank you for your detailed follow-up! 😊

It appears that the rknn_log module is present as a shared object file (rknn_log.cpython-310-x86_64-linux-gnu.so) rather than a Python file. This could be causing the import issue.

Steps to Resolve the Issue

1. Verify Installation:
   Ensure that you have the correct and latest versions of rknn-toolkit2 and rknn-toolkit-lite installed. Sometimes, version mismatches can cause such issues. You can reinstall the packages using:

   pip uninstall rknn-toolkit2 rknn-toolkit-lite
   pip install rknn-toolkit2 rknn-toolkit-lite

2. Check Python Environment:
   Make sure that you are using the correct Python environment where rknn-toolkit2 is installed. You can verify this by running:

   pip show rknn-toolkit2
   pip show rknn-toolkit-lite

3. Shared Object Files:
   Since the rknn_log module is a shared object file, ensure that your Python environment is correctly configured to load shared object files. You might need to set the LD_LIBRARY_PATH environment variable to include the directory containing these shared object files:

   export LD_LIBRARY_PATH=/home/vicharak/env1/lib/python3.10/site-packages/rknn/api:$LD_LIBRARY_PATH

4. Reproducible Example:
   If the issue persists, providing a minimum reproducible example would be very helpful for diagnosing the problem. You can find guidance on creating a reproducible example here: Minimum Reproducible Example.

Additional Resources

• RKNN-Toolkit2 Documentation

If you continue to face issues, feel free to share more details, and we'll do our best to assist you further. 😊

[samarth1232]

@glenn-jocher Thanks for suggestion I will try it out and let you know the results

[glenn-jocher]

Hi @samarth1232,

Thank you for your patience and for trying out the suggestions! 😊

If the issue persists, it would be very helpful if you could provide a minimum reproducible example. This will allow us to better understand the context and help you more effectively. You can find guidance on creating a reproducible example here: Minimum Reproducible Example.

Additionally, please ensure that you are using the latest versions of the rknn-toolkit2 and rknn-toolkit-lite packages, as updates often include important bug fixes and improvements.

Looking forward to your update! 🚀

[StuartIanNaylor]

https://github.com/Qengineering/Radxa-Zero-3-NPU-Ubuntu22

[glenn-jocher]

@StuartIanNaylor hello there!

Thank you for your interest in expanding the capabilities of Ultralytics YOLOv8 to support the RK3588 series CPUs with built-in NPU acceleration. We appreciate your suggestion and understand the growing popularity of these boards for integrated systems.

Currently, Ultralytics YOLOv8 supports a variety of export formats, including ONNX, TensorRT, and others, which can be run on different backends. While we do not have immediate plans to integrate the rknn-toolkit2 specifically for RK3588 NPUs, we are always exploring new ways to enhance our platform and support a wider range of hardware.

In the meantime, you might consider exporting your YOLOv8 model to ONNX format and then using the rknn-toolkit2 to convert the ONNX model for use with the RK3588 NPU. Here’s a brief outline of how you might approach this:

1. Export YOLOv8 Model to ONNX:

   from ultralytics import YOLO
   
   # Load your trained model
   model = YOLO("path/to/your/best.pt")
   
   # Export to ONNX format
   model.export(format="onnx")

2. Convert ONNX to RKNN:
   Follow the rknn-toolkit2 documentation to convert the ONNX model to RKNN format and deploy it on your RK3588 device.

For more detailed guidance, you can refer to the rknn-toolkit2 documentation.

We hope this helps, and we’re excited to see the innovative applications you develop with YOLOv8 and RK3588! If you have any further questions or need assistance, feel free to reach out to the community or check our documentation.