Unsupported Scalar Type 5? -- Portable/optimized ops don't consistently support half/bfloat16

[bluejack]

🐛 Describe the bug

After exporting a model to pte form and running it through executor_runner, I get:

E 00:00:02.220756 executorch:inputs_portable.cpp:45] Unsupported scalar type 5

I believe this is the "Half" type, or float16

Does that simply mean executor_runner does not support float16? Or does the whole framework not support float16?

Noted that when I run some investigation on the file using a python script, I get as far as sending it my float16 tensors, but it still fails to execute with a similar error:

[op_native_layer_norm.cpp:169] In function operator()(), assert failed (false): Unhandled dtype Half for native_layer_norm.out

I'm including the versions below, but note that this is using executorch built from head, rather than the last release. Should I expect the framework to support float16? And look to my own code for the error?

Versions

PyTorch version: 2.6.0.dev20250104
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 14.6.1 (arm64)
GCC version: Could not collect
Clang version: 16.0.0 (clang-1600.0.26.4)
CMake version: version 3.31.4
Libc version: N/A

Python version: 3.12.7 | packaged by Anaconda, Inc. | (main, Oct 4 2024, 08:22:19) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-14.6.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M3 Pro

Versions of relevant libraries:
[pip3] executorch==0.6.0a0+cd0e584
[pip3] flake8==7.0.0
[pip3] mypy==1.11.2
[pip3] mypy-extensions==1.0.0
[pip3] numpy==2.0.0
[pip3] numpydoc==1.7.0
[pip3] torch==2.6.0.dev20250104
[pip3] torchao==0.8.0+git2e032c6b
[pip3] torchaudio==2.6.0.dev20250104
[pip3] torchsr==1.0.4
[pip3] torchvision==0.22.0.dev20250104
[conda] executorch 0.6.0a0+cd0e584 pypi_0 pypi
[conda] numpy 2.0.0 pypi_0 pypi
[conda] numpydoc 1.7.0 py312hca03da5_0
[conda] torch 2.6.0.dev20250104 pypi_0 pypi
[conda] torchao 0.8.0+git2e032c6b pypi_0 pypi
[conda] torchaudio 2.6.0.dev20250104 pypi_0 pypi
[conda] torchsr 1.0.4 pypi_0 pypi
[conda] torchvision 0.22.0.dev20250104 pypi_0 pypi

[swolchok]

Does that simply mean executor_runner does not support float16?

It looks like this particular function does not support float16. I've just sent #7750 to fix it.

Or does the whole framework not support float16?

We are capable of supporting it, but it looks like portable ops coverage is spotty. I'll send a fix for native_layer_norm and as many other places as I can find.

[swolchok]

By the way, if you're running on your Mac, you might want to enable the XNNPACK delegate when exporting; there's a good chance you will get both better performance and a workaround for the remaining instance of this issue I haven't got PRs out for yet (though I don't know whether XNNPACK has layer norm off the top of my head).

[bluejack]

By the way, if you're running on your Mac, you might want to enable the XNNPACK delegate when exporting; there's a good chance you will get both better performance and a workaround for the remaining instance of this issue I haven't got PRs out for yet (though I don't know whether XNNPACK has layer norm off the top of my head).

Ok, I will look at this option, thanks for the tip.