[QST]Error installing transformers4rec: Failed building wheel for tokenizers

[MarRazane]

❓ Questions & Help

Details

Description: I encountered an error while trying to install transformers4rec. The installation fails when attempting to build the wheel for the tokenizers package, showing a "Failed building wheel for tokenizers" error. This prevents me from successfully installing and using the package.

Steps to Reproduce:

Create a new Conda environment with python=3.8.
Run pip install transformers4rec[torch].
The installation process begins but fails while building the tokenizers package, throwing an error.
Expected Behavior: I expected the transformers4rec package to be installed without errors, and for all dependencies (including tokenizers) to be installed successfully.

Actual Behavior: The installation process fails with the following error message:

ERROR: Failed building wheel for tokenizers
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tokenizers)

The installation terminates without completing, and transformers4rec is not installed.
System Information:

OS: Windows 11
Python version: Python 3.12.3
Pip version: pip 24.2
Rust version: rustc 1.57.0
transformers4rec version: Attempted to install the latest version

Attempted Solutions: I tried the following solutions, but none of them resolved the issue:

Upgraded pip, setuptools, and wheel to the latest versions.
Installed Rust as a compiler for the tokenizers package.
Tried installing tokenizers separately using pip install tokenizers==0.13.3.
Attempted to install transformers4rec within a fresh Conda environment.
Tried using both TensorFlow and PyTorch backends.

Request for Help: Could you please assist in resolving this issue or guide on what might be causing the error during installation? Any help would be appreciated!

[rnyak]

@MarRazane the recommended way to install transformers4rec is via docker image. could you use our pytorch docker image?

nvcr.io/nvidia/merlin/merlin-pytorch:23.12