FP8 support

[markoarnauto]

No description provided.

[btakeya]

Hi coreweave team, could you please give any consideration on it? I've just faced unknown quantization type error as below:

ValueError: Unknown quantization type, got fp8 - supported types are: ['awq', 'bitsandbytes_4bit', 'bitsandbytes_8bit', 'gptq', 'aqlm', 'quanto', 'eetq', 'hqq', 'compressed-tensors', 'fbgemm_fp8', 'torchao', 'bitnet']

It'd be helped a lot if fp8 would be supported. Thanks in advance.

[sangstar]

Hi @btakeya

Where did you get that error from? Can you send the full traceback? This looks like it might be an error from transformers.

[btakeya]

@sangstar Thanks for your attention!
I just tried with this code -- used my private fp8-quantized model (changed L16 with its hf repo name)
attached full traceback as well:

Traceback (most recent call last):
  File "tensorize.py", line 40, in <module>
    model = original_model(model_ref)
  File "tensorize.py", line 20, in original_model
    return AutoModelForCausalLM.from_pretrained(ref)
  File "/home/juhwan/venv/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/home/juhwan/venv/lib/python3.8/site-packages/transformers/modeling_utils.py", line 3647, in from_pretrained
    config.quantization_config = AutoHfQuantizer.merge_quantization_configs(
  File "/home/juhwan/venv/lib/python3.8/site-packages/transformers/quantizers/auto.py", line 173, in merge_quantization_configs
    quantization_config = AutoQuantizationConfig.from_dict(quantization_config)
  File "/home/juhwan/venv/lib/python3.8/site-packages/transformers/quantizers/auto.py", line 97, in from_dict
    raise ValueError(
ValueError: Unknown quantization type, got fp8 - supported types are: ['awq', 'bitsandbytes_4bit', 'bitsandbytes_8bit', 'gptq', 'aqlm', 'quanto', 'eetq', 'hqq', 'compressed-tensors', 'fbgemm_fp8', 'torchao', 'bitnet']

(seems transformer as you've expected)