GPTQ Support #421

maktukmak · 2024-10-23T21:57:05Z

This PR enables loading GPTQ quantized models and running weight-only quantized inference on HPU.

Currently, it works only for BF16 inference due to kernel torch.ops.hpu.convert_from_uint4 not supporting FP16.

Tested on TheBloke/Llama-2-70B-GPTQ with TP=1, 2, 4, and all passed.

… to all logging messages (vllm-project#9590)

Signed-off-by: luka <[email protected]>

…#9555)

…der + fix quant args (vllm-project#9217) Co-authored-by: Isotr0py <[email protected]>

…vllm-project#9612) Signed-off-by: Alex-Brooks <[email protected]>

Co-authored-by: Isotr0py <[email protected]>

…#9620) Signed-off-by: youkaichao <[email protected]>

) Signed-off-by: Tyler Michael Smith <[email protected]>

…llm-project#8914)

…-project#9614)

…#9393) Signed-off-by: Alex-Brooks <[email protected]> Co-authored-by: Cyrus Leung <[email protected]>

Co-authored-by: DarkLight1337 <[email protected]>

…ti-image (vllm-project#9626) Signed-off-by: mgoin <[email protected]>

…t#9628) Signed-off-by: mgoin <[email protected]>

Signed-off-by: Vinay Damodaran <[email protected]>

Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Cyrus Leung <[email protected]>

Signed-off-by: Woosuk Kwon <[email protected]>

Co-authored-by: Zhuohan Li <[email protected]>

…-project#9639) Signed-off-by: youkaichao <[email protected]> Co-authored-by: youkaichao <[email protected]>

Signed-off-by: Jee Jee Li <[email protected]>

…-project#9637) Signed-off-by: youkaichao <[email protected]> Co-authored-by: youkaichao <[email protected]>

…odels (vllm-project#9653)

…workflow (vllm-project#9661) Signed-off-by: Harry Mellor <[email protected]>

…-project#6143) Signed-off-by: yuwenzho <[email protected]> Signed-off-by: Chendi.Xue <[email protected]> Signed-off-by: Bob Zhu <[email protected]> Signed-off-by: zehao-intel <[email protected]> Signed-off-by: Konrad Zawora <[email protected]> Co-authored-by: Kunshang Ji <[email protected]> Co-authored-by: Sanju C Sudhakaran <[email protected]> Co-authored-by: Michal Adamczyk <[email protected]> Co-authored-by: Marceli Fylcek <[email protected]> Co-authored-by: Himangshu Lahkar <[email protected]> Co-authored-by: Vivek Goel <[email protected]> Co-authored-by: yuwenzho <[email protected]> Co-authored-by: Dominika Olszewska <[email protected]> Co-authored-by: barak goldberg <[email protected]> Co-authored-by: Michal Szutenberg <[email protected]> Co-authored-by: Jan Kaniecki <[email protected]> Co-authored-by: Agata Dobrzyniewicz <[email protected]> Co-authored-by: Krzysztof Wisniewski <[email protected]> Co-authored-by: Dudi Lester <[email protected]> Co-authored-by: Ilia Taraban <[email protected]> Co-authored-by: Chendi.Xue <[email protected]> Co-authored-by: Michał Kuligowski <[email protected]> Co-authored-by: Jakub Maksymczuk <[email protected]> Co-authored-by: Tomasz Zielinski <[email protected]> Co-authored-by: Sun Choi <[email protected]> Co-authored-by: Iryna Boiko <[email protected]> Co-authored-by: Bob Zhu <[email protected]> Co-authored-by: hlin99 <[email protected]> Co-authored-by: Zehao Huang <[email protected]> Co-authored-by: Andrzej Kotłowski <[email protected]> Co-authored-by: Yan Tomsinsky <[email protected]> Co-authored-by: Nir David <[email protected]> Co-authored-by: Yu-Zhou <[email protected]> Co-authored-by: Ruheena Suhani Shaik <[email protected]> Co-authored-by: Karol Damaszke <[email protected]> Co-authored-by: Marcin Swiniarski <[email protected]> Co-authored-by: Woosuk Kwon <[email protected]> Co-authored-by: Jacek Czaja <[email protected]> Co-authored-by: Jacek Czaja <[email protected]> Co-authored-by: Yuan <[email protected]>

Signed-off-by: Woosuk Kwon <[email protected]>

…naAI#452)

Signed-off-by: Max de Bayser <[email protected]> Signed-off-by: Max de Bayser <[email protected]> Signed-off-by: Joe Runde <[email protected]> Signed-off-by: Russell Bryant <[email protected]> Signed-off-by: Thomas Parnell <[email protected]> Signed-off-by: Russell Bryant <[email protected]> Signed-off-by: Varad Ahirwadkar <[email protected]> Signed-off-by: Wallas Santos <[email protected]> Signed-off-by: Travis Johnson <[email protected]> Signed-off-by: Rafael Vasquez <[email protected]> Signed-off-by: Yuan Zhou <[email protected]> Signed-off-by: luka <[email protected]> Signed-off-by: Alex-Brooks <[email protected]> Signed-off-by: youkaichao <[email protected]> Signed-off-by: Tyler Michael Smith <[email protected]> Signed-off-by: mgoin <[email protected]> Signed-off-by: Vinay Damodaran <[email protected]> Signed-off-by: Woosuk Kwon <[email protected]> Signed-off-by: Jee Jee Li <[email protected]> Signed-off-by: Harry Mellor <[email protected]> Signed-off-by: charlifu <[email protected]> Signed-off-by: Sam Stoelinga <[email protected]> Signed-off-by: Vasily Alexeev <[email protected]> Signed-off-by: Kevin-Yang <[email protected]> Signed-off-by: Abatom <[email protected]> Signed-off-by: Bill Nell <[email protected]> Signed-off-by: wangshuai09 <[email protected]> Signed-off-by: Qishuai [email protected] Signed-off-by: yuze.zyz <[email protected]> Signed-off-by: Yannick Schnider <[email protected]> Signed-off-by: Kunjan Patel <[email protected]> Signed-off-by: simon-mo <[email protected]> Signed-off-by: kevin <[email protected]> Signed-off-by: YiSheng5 <[email protected]> Signed-off-by: yan ma <[email protected]> Signed-off-by: Went-Liang <[email protected]> Signed-off-by: Roger Wang <[email protected]> Signed-off-by: sasha0552 <[email protected]> Signed-off-by: mzusman <[email protected]> Signed-off-by: Prashant Gupta <[email protected]> Signed-off-by: André Jonasson <[email protected]> Signed-off-by: Gene Su <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Peter Salas <[email protected]> Signed-off-by: Nick Hill <[email protected]> Signed-off-by: Nick Hill <[email protected]> Signed-off-by: Michael Green <[email protected]> Signed-off-by: Shanshan Wang <[email protected]> Signed-off-by: Gregory Shtrasberg <[email protected]> Signed-off-by: daitran2k1 <[email protected]> Signed-off-by: MengqingCao <[email protected]> Signed-off-by: chaunceyjiang <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: Hissu Hyvarinen <[email protected]> Signed-off-by: [email protected] <[email protected]> Signed-off-by: Linkun Chen <[email protected]> Signed-off-by: Tomer Asida <[email protected]> Signed-off-by: DarkLight1337 <[email protected]> Co-authored-by: sasha0552 <[email protected]> Co-authored-by: Woosuk Kwon <[email protected]> Co-authored-by: Li, Jiang <[email protected]> Co-authored-by: Kuntai Du <[email protected]> Co-authored-by: Daniele <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Luka Govedič <[email protected]> Co-authored-by: bnellnm <[email protected]> Co-authored-by: Kai Wu <[email protected]> Co-authored-by: Isotr0py <[email protected]> Co-authored-by: Shashwat Srijan <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: Andrew Feldman <[email protected]> Co-authored-by: afeldman-nm <[email protected]> Co-authored-by: laishzh <[email protected]> Co-authored-by: Max de Bayser <[email protected]> Co-authored-by: Max de Bayser <[email protected]> Co-authored-by: Dipika Sikka <[email protected]> Co-authored-by: Joe Runde <[email protected]> Co-authored-by: Haoyu Wang <[email protected]> Co-authored-by: Russell Bryant <[email protected]> Co-authored-by: Nick Hill <[email protected]> Co-authored-by: tomeras91 <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> Co-authored-by: Michael Goin <[email protected]> Co-authored-by: Kunjan <[email protected]> Co-authored-by: Kunjan Patel <kunjanp_google_com@vllm.us-central1-a.c.kunjanp-gke-dev-2.internal> Co-authored-by: Cody Yu <[email protected]> Co-authored-by: Thomas Parnell <[email protected]> Co-authored-by: Chih-Chieh Yang <[email protected]> Co-authored-by: Yue Zhang <[email protected]> Co-authored-by: Chen Zhang <[email protected]> Co-authored-by: Andy Dai <[email protected]> Co-authored-by: Dhia Eddine Rhaiem <[email protected]> Co-authored-by: yudian0504 <[email protected]> Co-authored-by: Varad Ahirwadkar <[email protected]> Co-authored-by: youkaichao <[email protected]> Co-authored-by: Baoyuan Qi <[email protected]> Co-authored-by: Wallas Henrique <[email protected]> Co-authored-by: Travis Johnson <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: ngrozae <[email protected]> Co-authored-by: Falko1 <[email protected]> Co-authored-by: Rafael Vasquez <[email protected]> Co-authored-by: chenqianfzh <[email protected]> Co-authored-by: wangshuai09 <[email protected]> Co-authored-by: Jee Jee Li <[email protected]> Co-authored-by: xendo <[email protected]> Co-authored-by: Jerzy Zagorski <[email protected]> Co-authored-by: gopalsarda <[email protected]> Co-authored-by: Yuan <[email protected]> Co-authored-by: Gubrud, Aaron D <[email protected]> Co-authored-by: adgubrud <[email protected]> Co-authored-by: Yuhong Guo <[email protected]> Co-authored-by: Yuhong Guo <[email protected]> Co-authored-by: Ronen Schaffer <[email protected]> Co-authored-by: Aurick Qiao <[email protected]> Co-authored-by: Jeremy Arnold <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> Co-authored-by: yulei <[email protected]> Co-authored-by: Seth Kimmel <[email protected]> Co-authored-by: Kaunil Dhruv <[email protected]> Co-authored-by: Flex Wang <[email protected]> Co-authored-by: Mengqing Cao <[email protected]> Co-authored-by: Alex Brooks <[email protected]> Co-authored-by: Yongzao <[email protected]> Co-authored-by: Yunfei Chu <[email protected]> Co-authored-by: Vinay R Damodaran <[email protected]> Co-authored-by: Yan Ma <[email protected]> Co-authored-by: Zhuohan Li <[email protected]> Co-authored-by: litianjian <[email protected]> Co-authored-by: Harry Mellor <[email protected]> Co-authored-by: Charlie Fu <[email protected]> Co-authored-by: Kevin H. Luu <[email protected]> Co-authored-by: Will Johnson <[email protected]> Co-authored-by: pavlo-ruban <[email protected]> Co-authored-by: Sam Stoelinga <[email protected]> Co-authored-by: ErkinSagiroglu <[email protected]> Co-authored-by: Vasiliy Alekseev <[email protected]> Co-authored-by: kakao-kevin-us <[email protected]> Co-authored-by: Kevin-Yang <[email protected]> Co-authored-by: 科英 <[email protected]> Co-authored-by: madt2709 <[email protected]> Co-authored-by: litianjian <[email protected]> Co-authored-by: Zhong Qishuai <[email protected]> Co-authored-by: tastelikefeet <[email protected]> Co-authored-by: Sven Seeberg <[email protected]> Co-authored-by: yannicks1 <[email protected]> Co-authored-by: Junichi Sato <[email protected]> Co-authored-by: Kunjan <[email protected]> Co-authored-by: Will Eaton <[email protected]> Co-authored-by: Simon Mo <[email protected]> Co-authored-by: Lily Liu <[email protected]> Co-authored-by: YiSheng5 <[email protected]> Co-authored-by: Went-Liang <[email protected]> Co-authored-by: Elfie Guo <[email protected]> Co-authored-by: Harsha vardhan manoj Bikki <[email protected]> Co-authored-by: Guillaume Calmettes <[email protected]> Co-authored-by: Roger Wang <[email protected]> Co-authored-by: Alexei-V-Ivanov-AMD <[email protected]> Co-authored-by: Mor Zusman <[email protected]> Co-authored-by: Prashant Gupta <[email protected]> Co-authored-by: Patrick von Platen <[email protected]> Co-authored-by: André Jonasson <[email protected]> Co-authored-by: Pavani Majety <[email protected]> Co-authored-by: Gene Der Su <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Peter Salas <[email protected]> Co-authored-by: sroy745 <[email protected]> Co-authored-by: Michael Green <[email protected]> Co-authored-by: Nick Hill <[email protected]> Co-authored-by: Nikita Furin <[email protected]> Co-authored-by: shanshan wang <[email protected]> Co-authored-by: Roger Wang <[email protected]> Co-authored-by: Gregory Shtrasberg <[email protected]> Co-authored-by: Yang Zheng <[email protected]> Co-authored-by: Yang Zheng(SW)(Alex) <[email protected]> Co-authored-by: Tran Quang Dai <[email protected]> Co-authored-by: Chauncey <[email protected]> Co-authored-by: hissu-hyvarinen <[email protected]> Co-authored-by: lkchen <[email protected]> Co-authored-by: Linkun Chen <[email protected]> Co-authored-by: Linkun Chen <[email protected]> Co-authored-by: Gene Der Su <[email protected]>

Reverts HabanaAI#439

This PR adds all commits before vllm-project#6143 without vllm-project#6143.

michalkuligowski · 2024-11-06T13:34:50Z

vllm/_core_ext.py

@@ -131,10 +131,37 @@ def is_ieee_754(self) -> bool:
                not self._finite_values_only

        def __str__(self) -> str:
-            raise NotImplementedError
+            """
+            naming generally follows: https://github.com/jax-ml/ml_dtypes


You are right. Vllm was rebased and this methods do have those defitnitions now (its now under scalar_type.py name). Please rebase

vllm/_custom_ops.py

vllm/model_executor/layers/quantization/gptq_hpu.py

vllm-project#6143 got merged, but it's based on an older revision of HPU components. This PR aligns the two.

) Accuracy fix for multi-step scheduling. This code solves the problem of a wrong second token when HPU Graphs are used. --------- Co-authored-by: Libin Tang <[email protected]>

…o gptq_hpu

maktukmak · 2024-11-06T20:08:11Z

@michalkuligowski, I did rebasing but seems like something might go wrong. Let me know if merging creates a problem.

michalkuligowski · 2024-11-08T11:46:07Z

@maktukmak something went wrong with the rebase, you should not have all those commit visible. Maybe you should create a new PR on the latest habana_main. Also please extract hpu specific classes and push to vllm-hpu-extension as in #421 (comment) as this cant be merged into habana_main in present form.

maktukmak · 2024-11-08T22:40:13Z

@maktukmak something went wrong with the rebase, you should not have all those commit visible. Maybe you should create a new PR on the latest habana_main. Also please extract hpu specific classes and push to vllm-hpu-extension as in #421 (comment) as this cant be merged into habana_main in present form.

@michalkuligowski, I created a new PR #481 from habana_main. I also pushed hpu specific classes to vllm-hpu-extension (HabanaAI/vllm-hpu-extension#28).

michalkuligowski · 2024-11-20T09:14:34Z

Closing this on, continued in #481

sfc-gh-zhwang and others added 22 commits October 23, 2024 11:17

[Misc] Add an env var VLLM_LOGGING_PREFIX, if set, it will be prepend…

29061ed

… to all logging messages (vllm-project#9590)

[Model] Support E5-V (vllm-project#9576)

831540c

[Build] Fix FetchContent multiple build issue (vllm-project#9596)

51c24c9

Signed-off-by: luka <[email protected]>

[Hardware][XPU] using current_platform.is_xpu (vllm-project#9605)

2394962

[Model] Initialize Florence-2 language backbone support (vllm-project…

3ff57eb

…#9555)

[VLM] Enable overriding whether post layernorm is used in vision enco…

c18e1a3

…der + fix quant args (vllm-project#9217) Co-authored-by: Isotr0py <[email protected]>

[Model] Add min_pixels / max_pixels to Qwen2VL as mm_processor_kwargs (…

31a08f5

…vllm-project#9612) Signed-off-by: Alex-Brooks <[email protected]>

[Bugfix] Fix _init_vision_model in NVLM_D model (vllm-project#9611)

e7116c0

Co-authored-by: Isotr0py <[email protected]>

[misc] comment to avoid future confusion about baichuan (vllm-project…

dbdd3b5

…#9620) Signed-off-by: youkaichao <[email protected]>

[Bugfix] Fix divide by zero when serving Mamba models (vllm-project#9617

e5ac6a4

) Signed-off-by: Tyler Michael Smith <[email protected]>

[Misc] Separate total and output tokens in benchmark_throughput.py (v…

fd0e2cf

…llm-project#8914)

[torch.compile] Adding torch compile annotations to some models (vllm…

9013e24

…-project#9614)

[Frontend] Enable Online Multi-image Support for MLlama (vllm-project…

150b779

…#9393) Signed-off-by: Alex-Brooks <[email protected]> Co-authored-by: Cyrus Leung <[email protected]>

[Model] Add Qwen2-Audio model support (vllm-project#9248)

fc6c274

Co-authored-by: DarkLight1337 <[email protected]>

gptq hpu support added

d1fbc94

[CI/Build] Add bot to close stale issues and PRs (vllm-project#9436)

b548d7a

[Bugfix][Model] Fix Mllama SDPA illegal memory access for batched mul…

bb01f29

…ti-image (vllm-project#9626) Signed-off-by: mgoin <[email protected]>

[Bugfix] Use "vision_model" prefix for MllamaVisionModel (vllm-projec…

b7df53c

…t#9628) Signed-off-by: mgoin <[email protected]>

[Bugfix]: Make chat content text allow type content (vllm-project#9358)

33bab41

Signed-off-by: Vinay Damodaran <[email protected]>

[XPU] avoid triton import for xpu (vllm-project#9440)

056a68c

Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Cyrus Leung <[email protected]>

[Bugfix] Fix PP for ChatGLM and Molmo (vllm-project#9422)

836e8ef

[V1][Bugfix] Clean up requests when aborted (vllm-project#9629)

3770071

Signed-off-by: Woosuk Kwon <[email protected]>

michalkuligowski marked this pull request as draft October 24, 2024 06:47

youkaichao and others added 7 commits October 24, 2024 00:16

[core] simplify seq group code (vllm-project#9569)

4fdc581

Co-authored-by: Zhuohan Li <[email protected]>

[torch.compile] Adding torch compile annotations to some models (vllm…

8a02cd0

…-project#9639) Signed-off-by: youkaichao <[email protected]> Co-authored-by: youkaichao <[email protected]>

[Kernel] add kernel for FATReLU (vllm-project#9610)

295a061

Signed-off-by: Jee Jee Li <[email protected]>

[torch.compile] expanding support and fix allgather compilation (vllm…

ad6f780

…-project#9637) Signed-off-by: youkaichao <[email protected]> Co-authored-by: youkaichao <[email protected]>

[Doc] Move additional tips/notes to the top (vllm-project#9647)

b979143

[Bugfix]Disable the post_norm layer of the vision encoder for LLaVA m…

f584549

…odels (vllm-project#9653)

Increase operation per run limit for "Close inactive issues and PRs" …

de662d3

…workflow (vllm-project#9661) Signed-off-by: Harry Mellor <[email protected]>

kzawora-intel and others added 13 commits November 6, 2024 01:09

[Hotfix] Fix ruff errors (vllm-project#10073)

6a585a2

Signed-off-by: Woosuk Kwon <[email protected]>

[BugFix][Habana_main][Multistep]Fix multistep deepcopy overhead (Haba…

c3c0e90

…naAI#452)

Set vllm-hpu-extension to 0063520 (HabanaAI#455)

dc5cdfb

Revert "Oct 28 rebase" (HabanaAI#466)

07a6441

Reverts HabanaAI#439

Oct 28 rebase - attempt 2 (HabanaAI#467)

5812cb6

Merge commit 'a5fda50a10641e47c0c290907f30ef2add6d4e7a' into HEAD

40882f3

format.sh

8e62377

Nov 6 rebase (sans vllm-project#6143) (HabanaAI#468)

5eb7f3d

This PR adds all commits before vllm-project#6143 without vllm-project#6143.

Fix missed conflict (HabanaAI#469)

0a17a2e

Merge commit 'a02a50e' into HEAD

b91403a

Merge commit '6a585a2' into HEAD

843ae37

michalkuligowski requested changes Nov 6, 2024

View reviewed changes

michalkuligowski requested a review from kzawora-intel November 6, 2024 13:37

michalkuligowski and others added 9 commits November 6, 2024 14:43

Align fork with HPU upstream code (HabanaAI#465)

60b981e

vllm-project#6143 got merged, but it's based on an older revision of HPU components. This PR aligns the two.

The output tensor from sampling is the input_tokens to the (HabanaAI#471

3c39626

) Accuracy fix for multi-step scheduling. This code solves the problem of a wrong second token when HPU Graphs are used. --------- Co-authored-by: Libin Tang <[email protected]>

gptq hpu support added

66a67fc

row vs column paralel fix

f9cf700

g_idx check added

cbcba5d

exllama state removed

a6ab053

removed custom ops check

9b323b5

format fixes

7077a99

Merge branch 'gptq_hpu' of https://github.com/maktukmak/vllm-fork int…

4ef6b7e

…o gptq_hpu

maktukmak mentioned this pull request Nov 8, 2024

GPTQ Support [Cont.] #481

Open

michalkuligowski closed this Nov 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GPTQ Support #421

GPTQ Support #421

maktukmak commented Oct 23, 2024 •

edited

Loading

michalkuligowski Nov 6, 2024 •

edited

Loading

maktukmak commented Nov 6, 2024 •

edited

Loading

michalkuligowski commented Nov 8, 2024

maktukmak commented Nov 8, 2024 •

edited

Loading

michalkuligowski commented Nov 20, 2024

GPTQ Support #421

GPTQ Support #421

Conversation

maktukmak commented Oct 23, 2024 • edited Loading

michalkuligowski Nov 6, 2024 • edited Loading

Choose a reason for hiding this comment

maktukmak commented Nov 6, 2024 • edited Loading

michalkuligowski commented Nov 8, 2024

maktukmak commented Nov 8, 2024 • edited Loading

michalkuligowski commented Nov 20, 2024

maktukmak commented Oct 23, 2024 •

edited

Loading

michalkuligowski Nov 6, 2024 •

edited

Loading

maktukmak commented Nov 6, 2024 •

edited

Loading

maktukmak commented Nov 8, 2024 •

edited

Loading