Code

Multi models support for upstream#590

Code

Open

xuechendi wants to merge 211 commits intoHabanaAI:multi_models_upstreamHabanaAI/vllm-fork:multi_models_upstreamfrom xuechendi:multi_models_rebasexuechendi/vllm-fork:multi_models_rebaseCopy head branch name to clipboard

+25,211-8,904

Commits

Commits on Nov 18, 2024

• [Model][LoRA]LoRA support added for glm-4v (vllm-project#10418)

  Show description for 5be4e52
  B-201

  authored
  5be4e52

  Copy full SHA for 5be4e52

• [Model] Remove transformers attention porting in VITs (vllm-project#10414)

  Show description for e7ebb66
  Isotr0py

  authored
  e7ebb66

  Copy full SHA for e7ebb66

• [Doc] Update doc for LoRA support in GLM-4V (vllm-project#10425)

  Show description for 4186be8
  B-201

  authored
  4186be8

  Copy full SHA for 4186be8

• [5/N][torch.compile] torch.jit.script --> torch.compile (vllm-project#10406)

  Show description for 7851b45
  youkaichao

  authored
  7851b45

  Copy full SHA for 7851b45

• [Doc] Add documentation for Structured Outputs (vllm-project#9943)

  Show description for 31894a2
  ismael-dm

  authored
  31894a2

  Copy full SHA for 31894a2

• Fix open_collective value in FUNDING.yml (vllm-project#10426)

  Show description for 4f686d1
  andrew

  authored
  4f686d1

  Copy full SHA for 4f686d1

• [Model][Bugfix] Support TP for PixtralHF ViT (vllm-project#10405)

  Show description for 281cc4b
  mgoin

  authored
  281cc4b

  Copy full SHA for 281cc4b

• [Hardware][XPU] AWQ/GPTQ support for xpu backend (vllm-project#10107)

  Show description for 6b2d25e
  yma11

  authored
  6b2d25e

  Copy full SHA for 6b2d25e

• [Kernel] Explicitly specify other value in tl.load calls (vllm-project#9014)

  Show description for c2170a5
  angusYuhao

  authored
  c2170a5

  Copy full SHA for c2170a5

• [Kernel] Initial Machete W4A8 support + Refactors (vllm-project#9855)

  Show description for 96d999f
  LucasWilkinson

  authored
  96d999f

  Copy full SHA for 96d999f

• [3/N][torch.compile] consolidate custom op logging (vllm-project#10399)

  Show description for a03ea40
  youkaichao

  authored
  a03ea40

  Copy full SHA for a03ea40

• [ci][bugfix] fix kernel tests (vllm-project#10431)

  Show description for 2298e69
  youkaichao

  authored
  2298e69

  Copy full SHA for 2298e69

• [misc] partial prefix & random input generation benchmark (vllm-project#9929)

  Show description for 90a6c75
  rickyyx

  authored
  90a6c75

  Copy full SHA for 90a6c75

Commits on Nov 19, 2024

• [ci/build] Have dependabot ignore all patch update (vllm-project#10436)

  Show description for 284203f
  khluu

  authored
  284203f

  Copy full SHA for 284203f

• [Bugfix]Fix Phi-3 BNB online quantization (vllm-project#10417)

  Show description for 7eb719d
  jeejeelee

  authored
  7eb719d

  Copy full SHA for 7eb719d

• [Platform][Refactor] Extract func get_default_attn_backend to Platform (vllm-project#10358)

  Show description for 8c1fb50
  MengqingCao

  authored
  8c1fb50

  Copy full SHA for 8c1fb50

• Add openai.beta.chat.completions.parse example to structured_outputs.rst (vllm-project#10433)

  mgoin

  authored
  74f8c2c

  Copy full SHA for 74f8c2c

• [Bugfix] Guard for negative counter metrics to prevent crash (vllm-project#10430)

  Show description for 272e31c
  tjohnson31415

  authored
  272e31c

  Copy full SHA for 272e31c

• [Misc] Avoid misleading warning messages (vllm-project#10438)

  Show description for 382b6a4
  jeejeelee

  authored
  382b6a4

  Copy full SHA for 382b6a4

• [Doc] Add the start of an arch overview page (vllm-project#10368)

  russellb

  authored
  5390d66

  Copy full SHA for 5390d66

• [misc][plugin] improve plugin loading (vllm-project#10443)

  Show description for 25f9c78
  youkaichao

  authored
  25f9c78

  Copy full SHA for 25f9c78

• [CI][CPU] adding numa node number as container name suffix (vllm-project#10441)

  Show description for b461465
  zhouyuan

  authored
  b461465

  Copy full SHA for b461465

• [BugFix] Fix hermes tool parser output error stream arguments in some cases (vllm-project#10395) (vllm-project#10398)

  Show description for f028dff
  xiyuan-lee

  authored
  f028dff

  Copy full SHA for f028dff

• [Pixtral-Large] Pixtral actually has no bias in vision-lang adapter (vllm-project#10449)

  patrickvonplaten

  authored
  11fd7ea

  Copy full SHA for 11fd7ea

• Fix: Build error seen on Power Architecture (vllm-project#10421)

  Show description for 1ea291a
  

  17 peopleauthored
  1ea291a

  Copy full SHA for 1ea291a

• [Doc] fix link for page that was renamed (vllm-project#10455)

  Show description for fd9f124
  russellb

  authored
  fd9f124

  Copy full SHA for fd9f124

• [6/N] torch.compile rollout to users (vllm-project#10437)

  Show description for 803f37e
  youkaichao

  authored
  803f37e

  Copy full SHA for 803f37e

• [Core] Avoid metrics log noise when idle (vllm-project#8868)

  Show description for efa9084
  russellb

  authored
  efa9084

  Copy full SHA for efa9084

• [Model][Quantization] HQQ support through Marlin kernel expansion (vllm-project#9766)

  Show description for b00b33d
  ElizaWszola

  authored
  b00b33d

  Copy full SHA for b00b33d

Commits on Nov 20, 2024

• Change granite chat template to keep json list formatting for tool calls (vllm-project#10452)

  Show description for a324d3a
  maxdebayser

  authored
  a324d3a

  Copy full SHA for a324d3a

• [CI/Build] Update Dockerfile.rocm (vllm-project#10434)

  Show description for d5b68ab
  Alexei-V-Ivanov-AMD

  authored
  d5b68ab

  Copy full SHA for d5b68ab

• [Bugfix] Marlin 2:4 temp fix for large M dim (>256) (vllm-project#10464)

  Show description for d200972
  LucasWilkinson

  authored
  d200972

  Copy full SHA for d200972

• [Misc] Add __setitem__ for LazyDict (vllm-project#10469)

  Show description for 9e05252
  liuyanyi

  authored
  9e05252

  Copy full SHA for 9e05252

• [Bugfix] Fix Mamba model initialization and MLP Speculator weights loading (vllm-project#10456)

  Show description for ad44437
  Isotr0py

  authored
  ad44437

  Copy full SHA for ad44437

• [Bugfix] Enforce no chunked prefill for embedding models (vllm-project#10470)

  Show description for b4be5a8
  DarkLight1337

  authored
  b4be5a8

  Copy full SHA for b4be5a8

• [CI/Build] Add sphinx/rst linter for docs (vllm-project#10366)

  rafvasq

  authored
  709c9f1

  Copy full SHA for 709c9f1

• [CI/Build] Support compilation with local cutlass path (vllm-project#10423) (vllm-project#10424)

  wchen61

  authored
  7629a9c

  Copy full SHA for 7629a9c

• [ci/build] Combine nightly and optional (vllm-project#10465)

  khluu

  authored
  ed701ca

  Copy full SHA for ed701ca

• [model] Reduce medusa weight (vllm-project#10454)

  Show description for 343041c
  skylee-01

  authored
  343041c

  Copy full SHA for 343041c

• [Bugfix] Handle conflicts between modern and legacy fields (vllm-project#10471)

  Show description for 09dbf9f
  DarkLight1337

  authored
  09dbf9f

  Copy full SHA for 09dbf9f

• [Platforms] Refactor xpu code (vllm-project#10468)

  Show description for d5b2844
  MengqingCao

  authored
  d5b2844

  Copy full SHA for d5b2844

• [Hardware][CPU] Support chunked-prefill and prefix-caching on CPU (vllm-project#10355)

  Show description for 63f1fde
  bigPYJ1151

  authored
  63f1fde

  Copy full SHA for 63f1fde

• [platforms] restore xpu check for parallel config (vllm-project#10479)

  Show description for 772a667
  youkaichao

  authored
  772a667

  Copy full SHA for 772a667

• [perf bench] H200 development (vllm-project#9768)

  Show description for 5f1d6af
  simon-mo

  authored
  5f1d6af

  Copy full SHA for 5f1d6af

• [7/N] torch.compile, reduce compilation time (vllm-project#10460)

  Show description for 0cd3d97
  youkaichao

  authored
  0cd3d97

  Copy full SHA for 0cd3d97

• [Bugfix]: allow extra fields in requests to openai compatible server (vllm-project#10463)

  Show description for c68f7ed
  gcalmettes

  authored
  c68f7ed

  Copy full SHA for c68f7ed

• [TPU] Implement prefix caching for TPUs (vllm-project#10307)

  Show description for 2f77b6c
  WoosukKwon

  authored
  2f77b6c

  Copy full SHA for 2f77b6c

Commits on Nov 21, 2024

• [torch.compile] limit inductor threads and lazy import quant (vllm-project#10482)

  Show description for 388ee3d
  youkaichao

  authored
  388ee3d

  Copy full SHA for 388ee3d

• [Core] Add Sliding Window Support with Flashinfer (vllm-project#10462)

  Show description for 6c1208d
  pavanimajety

  authored
  6c1208d

  Copy full SHA for 6c1208d

• [Platforms] Add device_type in Platform (vllm-project#10508)

  Show description for 9d82717
  MengqingCao

  authored
  9d82717

  Copy full SHA for 9d82717

• [torch.compile] Inductor code caching fix (vllm-project#10273)

  Show description for 8b0fe06
  ProExpertProg

  authored
  8b0fe06

  Copy full SHA for 8b0fe06

• [Misc] Increase default video fetch timeout (vllm-project#10495)

  Show description for 3430857
  DarkLight1337

  authored
  3430857

  Copy full SHA for 3430857

• [platforms] improve error message for unspecified platforms (vllm-project#10520)

  Show description for aaddce5
  youkaichao

  authored
  aaddce5

  Copy full SHA for aaddce5

• [Doc] fix a small typo in docstring of llama_tool_parser (vllm-project#10513)

  FerdinandZhong

  authored
  f0e0238

  Copy full SHA for f0e0238

• [Model] Add Support for Multimodal Granite Models (vllm-project#10291)

  Show description for 1cfde82
  

  alex-jw-brooks

  and

  DarkLight1337

  authored
  1cfde82

  Copy full SHA for 1cfde82

• fix the issue that len(tokenizer(prompt)["input_ids"]) > prompt_len (vllm-project#10524)

  Show description for 8a93a59
  sywangyi

  authored
  8a93a59

  Copy full SHA for 8a93a59

• [Model] Expose dynamic_image_size as mm_processor_kwargs for InternVL2 models (vllm-project#10518)

  Show description for d5ec121
  Isotr0py

  authored
  d5ec121

  Copy full SHA for d5ec121

• [Bugfix] Embedding model pooling_type equals ALL and multi input's bug (vllm-project#10494)

  BBuf

  authored
  4d676f0

  Copy full SHA for 4d676f0

• [Bug]: When apply continue_final_message for OpenAI server, the "echo":false is ignored (vllm-project#10180)

  Show description for da7e702
  chaunceyjiang

  authored
  da7e702

  Copy full SHA for da7e702

• [Kernel] Register punica ops directly (vllm-project#10522)

  Show description for 2385b60
  jeejeelee

  authored
  2385b60

  Copy full SHA for 2385b60

• [Misc] Suppress duplicated logging regarding multimodal input pipeline (vllm-project#10530)

  Show description for c51e397
  ywang96

  authored
  c51e397

  Copy full SHA for c51e397

• [Bugfix] Allow token ID-only inputs in Qwen2-Audio (vllm-project#10536)

  Show description for e7a8341
  DarkLight1337

  authored
  e7a8341

  Copy full SHA for e7a8341

• [8/N] enable cli flag without a space (vllm-project#10529)

  Show description for 7560ae5
  youkaichao

  authored
  7560ae5

  Copy full SHA for 7560ae5

• [V1] Fix Compilation config & Enable CUDA graph by default (vllm-project#10528)

  Show description for f9310cb
  WoosukKwon

  authored
  f9310cb

  Copy full SHA for f9310cb

• [CI][Installation] Avoid uploading CUDA 11.8 wheel (vllm-project#10535)

  Show description for edec338
  

  cermeng

  and

  simon-mo

  authored
  edec338

  Copy full SHA for edec338

• [misc] improve error message (vllm-project#10553)

  Show description for cf656f5
  youkaichao

  authored
  cf656f5

  Copy full SHA for cf656f5

• [Minor] Revert change in offline inference example (vllm-project#10545)

  Show description for 46fe9b4
  WoosukKwon

  authored
  46fe9b4

  Copy full SHA for 46fe9b4

• Add small example to metrics.rst (vllm-project#10550)

  mgoin

  authored
  9afa014

  Copy full SHA for 9afa014

Commits on Nov 22, 2024

• [Benchmark] Add new H100 machine (vllm-project#10547)

  simon-mo

  authored
  aed0748

  Copy full SHA for aed0748

• [9/N] torch.compile LLM usage (vllm-project#10552)

  Show description for 33e0a25
  youkaichao

  authored
  33e0a25

  Copy full SHA for 33e0a25

• [Minor] Fix line-too-long (vllm-project#10563)

  Show description for 446c780
  WoosukKwon

  authored
  446c780

  Copy full SHA for 446c780

• [platforms] absorb worker cls difference into platforms folder (vllm-project#10555)

  Show description for a111d01
  

  youkaichao

  and

  njhill

  authored
  a111d01

  Copy full SHA for a111d01

• [Bugfix] Fix Phi-3 BNB quantization with tensor parallel (vllm-project#9948)

  Show description for b6374e0
  Isotr0py

  authored
  b6374e0

  Copy full SHA for b6374e0

• Remove token-adding chat embedding params (vllm-project#10551)

  Show description for 11fcf0e
  noamgat

  authored
  11fcf0e

  Copy full SHA for 11fcf0e

• [bugfix] fix full graph tests (vllm-project#10581)

  Show description for db100c5
  youkaichao

  authored
  db100c5

  Copy full SHA for db100c5

• [torch.compile] support all attention backends (vllm-project#10558)

  Show description for eebad39
  youkaichao

  authored
  eebad39

  Copy full SHA for eebad39

• [v1] Refactor KVCacheManager for more hash input than token ids (vllm-project#10507)

  Show description for 97814fb
  

  rickyyx

  and

  comaniac

  authored
  97814fb

  Copy full SHA for 97814fb

Commits on Nov 23, 2024

• support bitsandbytes quantization with qwen model (vllm-project#10549)

  Show description for 948c859
  zixuanzhang226

  authored
  948c859

  Copy full SHA for 948c859

• [Core] remove temporary local variables in LLMEngine.__init__ (vllm-project#10577)

  Show description for 28598f3
  russellb

  authored
  28598f3

  Copy full SHA for 28598f3

• [V1] EngineCore supports profiling (vllm-project#10564)

  Show description for d345f40
  Abatom

  authored
  d345f40

  Copy full SHA for d345f40

• [bugfix] fix cpu tests (vllm-project#10585)

  Show description for d559979
  youkaichao

  authored
  d559979

  Copy full SHA for d559979

• [Bugfix][Frontend] Update Llama Chat Templates to also support Non-Tool use (vllm-project#10164)

  Show description for 9195dbd
  tjohnson31415

  authored
  9195dbd

  Copy full SHA for 9195dbd

• [Core] Fix broken log configuration (vllm-project#10458)

  Show description for ebda519
  russellb

  authored
  ebda519

  Copy full SHA for ebda519

• [Misc] Add pynccl wrappers for all_gather and reduce_scatter (vllm-project#9432)

  tlrmchlsmth

  authored
  978b397

  Copy full SHA for 978b397

• [core] gemma2 full context length support (vllm-project#10584)

  Show description for 4aba6e3
  youkaichao

  authored
  4aba6e3

  Copy full SHA for 4aba6e3

• [Bugfix] Internal Server Error when tool_choice is incorrect. (vllm-project#10567)

  Show description for 7d8ffb3
  shenoyvvarun

  authored
  7d8ffb3

  Copy full SHA for 7d8ffb3

• [Model] Fix Baichuan BNB online quantization (vllm-project#10572)

  Show description for cfea9c0
  CNTRYROA

  authored
  cfea9c0

  Copy full SHA for cfea9c0

• Update default max_num_batch_tokens for chunked prefill to 2048 (vllm-project#10544)

  mgoin

  authored
  02a43f8

  Copy full SHA for 02a43f8

• [AMD] Add support for GGUF quantization on ROCm (vllm-project#10254)

  kliuae

  authored
  7c25fe4

  Copy full SHA for 7c25fe4

• Prefix Cache Aware Scheduling [1/n] (vllm-project#10128)

  Show description for 4634a89
  rickyyx

  authored
  4634a89

  Copy full SHA for 4634a89

• [2/N] handling placeholders in merged multi-modal processor (vllm-project#10485)

  Show description for c8acd80
  DarkLight1337

  authored
  c8acd80

  Copy full SHA for c8acd80

• [Bugfix] multi_modal_kwargs broadcast for CPU tensor parallel (vllm-project#10541)

  Show description for 4cfe5d2
  Isotr0py

  authored
  4cfe5d2

  Copy full SHA for 4cfe5d2

• [Platforms] Refactor openvino code (vllm-project#10573)

  Show description for 86a44fb
  ji-huazhong

  authored
  86a44fb

  Copy full SHA for 86a44fb

• For ppc64le, disabled tests for now and addressed space issues (vllm-project#10538)

  npanpaliya

  authored
  651f6c3

  Copy full SHA for 651f6c3

• [Bugfix] Avoid import AttentionMetadata explicitly in Mllama (vllm-project#10593)

  Show description for 04668eb
  Isotr0py

  authored
  04668eb

  Copy full SHA for 04668eb

Commits on Nov 24, 2024

• [bugfix] Fix example/tensorize_vllm_model tests (vllm-project#10595)

  Show description for 17d8fc1
  jeejeelee

  authored
  17d8fc1

  Copy full SHA for 17d8fc1

• [Bugfix] Fix LoRA weight sharding (vllm-project#10450)

  Show description for 1700c54
  

  jeejeelee

  and

  DarkLight1337

  authored
  1700c54

  Copy full SHA for 1700c54

• [CI/Build] Print running script to enhance CI log readability (vllm-project#10594)

  Show description for 1c445dc
  jeejeelee

  authored
  1c445dc

  Copy full SHA for 1c445dc

• Revert "Print running script to enhance CI log readability" (vllm-project#10601)

  youkaichao

  authored
  eda2b35

  Copy full SHA for eda2b35

• [model][utils] add extract_layer_index utility function (vllm-project#10599)

  Show description for c055747
  youkaichao

  authored
  c055747

  Copy full SHA for c055747

• [doc] update the code to add models (vllm-project#10603)

  Show description for e4fbb14
  

  youkaichao

  and

  DarkLight1337

  authored
  e4fbb14

  Copy full SHA for e4fbb14

Commits on Nov 25, 2024

• [Doc] Update README.md with Ray Summit talk links (vllm-project#10610)

  zhuohan123

  authored
  49628fe

  Copy full SHA for 49628fe

• Support Cross encoder models (vllm-project#10400)

  Show description for 214efc2
  

  maxdebayser

  and

  flaviabeo

  authored
  214efc2

  Copy full SHA for 214efc2

• [Refactor][MISC] del redundant code in ParallelConfig.postinit (vllm-project#10614)

  Show description for 7ea3cd7
  MengqingCao

  authored
  7ea3cd7

  Copy full SHA for 7ea3cd7

• [torch.compile] support encoder based models (vllm-project#10613)

  Show description for 571841b
  youkaichao

  authored
  571841b

  Copy full SHA for 571841b

• [Doc] Add encoder-based models to Supported Models page (vllm-project#10616)

  Show description for a30a605
  DarkLight1337

  authored
  a30a605

  Copy full SHA for a30a605

• [torch.compile] force inductor threads (vllm-project#10620)

  Show description for 7c2134b
  jeejeelee

  authored
  7c2134b

  Copy full SHA for 7c2134b

• [torch.compile] add warning for unsupported models (vllm-project#10622)

  Show description for 6581378
  youkaichao

  authored
  6581378

  Copy full SHA for 6581378

• [misc] add torch.compile compatibility check (vllm-project#10618)

  Show description for 25d806e
  youkaichao

  authored
  25d806e

  Copy full SHA for 25d806e

• [misc] move functions to config.py (vllm-project#10624)

  Show description for 05d1f8c
  youkaichao

  authored
  05d1f8c

  Copy full SHA for 05d1f8c

• [Model] Support is_causal HF config field for Qwen2 model (vllm-project#10621)

  Show description for ed46f14
  DarkLight1337

  authored
  ed46f14

  Copy full SHA for ed46f14

• Super tiny little typo fix (vllm-project#10633)

  fzyzcjy

  authored
  2b0879b

  Copy full SHA for 2b0879b

• [Bug]: Authorization ignored when root_path is set (vllm-project#10606)

  Show description for d04b13a
  chaunceyjiang

  authored
  d04b13a

  Copy full SHA for d04b13a

• [Bugfix] Fix chunked prefill with model dtype float32 on Turing Devices (vllm-project#9850)

  Show description for c27df94
  

  wallashss

  and

  mgoin

  authored
  c27df94

  Copy full SHA for c27df94

• [Docs] Add Snowflake Slides (vllm-project#10641)

  Show description for 452a4e8
  simon-mo

  authored
  452a4e8

  Copy full SHA for 452a4e8

• [Model]: Add support for Aria model (vllm-project#10514)

  Show description for b1d9205
  

  xffxff

  and

  Isotr0py

  authored
  b1d9205

  Copy full SHA for b1d9205

• [Model] Enable optional prefix when loading embedding models (vllm-project#10639)

  Show description for cf73f0c
  DarkLight1337

  authored
  cf73f0c

  Copy full SHA for cf73f0c

• [Doc] Fix typos in docs (vllm-project#10636)

  Show description for 1b583cf
  DarkLight1337

  authored
  1b583cf

  Copy full SHA for 1b583cf

• [Model] Add OLMo November 2024 model (vllm-project#10503)

  2015aroras

  authored
  9db713a

  Copy full SHA for 9db713a

Commits on Nov 26, 2024

• [misc] do not read HOST_IP (vllm-project#10644)

  Show description for 6e9ff05
  youkaichao

  authored
  6e9ff05

  Copy full SHA for 6e9ff05

• [bugfix] fix aria model and add torch.compile (vllm-project#10645)

  Show description for 45ac4ff
  youkaichao

  authored
  45ac4ff

  Copy full SHA for 45ac4ff

• [Feature] vLLM ARM Enablement for AARCH64 CPUs (vllm-project#9228)

  Show description for a6760f6
  

  3 peopleauthored
  a6760f6

  Copy full SHA for a6760f6

• [v1] EngineArgs for better config handling for v1 (vllm-project#10382)

  Show description for 519e8e4
  rickyyx

  authored
  519e8e4

  Copy full SHA for 519e8e4

• custom allreduce + torch.compile (vllm-project#10121)

  Show description for 9a88f89
  

  SageMoore

  and

  youkaichao

  authored
  9a88f89

  Copy full SHA for 9a88f89

• [Misc] Remove outdated init protocols (vllm-project#10655)

  Show description for 9406353
  DarkLight1337

  authored
  9406353

  Copy full SHA for 9406353

• [ci] add vllm_test_utils (vllm-project#10659)

  Show description for 334d64d
  youkaichao

  authored
  334d64d

  Copy full SHA for 334d64d

• [V1] Enable profile for LLMEngine (vllm-project#10665)

  jikunshang

  authored
  1f6584e

  Copy full SHA for 1f6584e

• [Bugfix] Fix for Spec model TP + Chunked Prefill (vllm-project#10232)

  Show description for db66e01
  

  andoorve

  and

  sroy745

  authored
  db66e01

  Copy full SHA for db66e01

• [Hardware][NVIDIA] Add non-NVML CUDA mode for Jetson (vllm-project#9735)

  Show description for f5792c7
  conroy-cheers

  authored
  f5792c7

  Copy full SHA for f5792c7

• [Bugfix] Fix using -O[0,3] with LLM entrypoint (vllm-project#10677)

  Show description for 9a99273
  mgoin

  authored
  9a99273

  Copy full SHA for 9a99273

• [Bugfix] Check bnb_4bit_quant_storage for bitsandbytes (vllm-project#10642)

  mgoin

  authored
  7576cd3

  Copy full SHA for 7576cd3

• [V1] Refactor model executable interface for multimodal models (vllm-project#10570)

  Show description for 2f0a0a1
  ywang96

  authored
  2f0a0a1

  Copy full SHA for 2f0a0a1

Commits on Nov 27, 2024

• Remove hard-dependencies of Speculative decode to CUDA workers (vllm-project#10587)

  Show description for 0a71900
  xuechendi

  authored
  0a71900

  Copy full SHA for 0a71900

• [V1] Update interface for idefics3 (vllm-project#10680)

  Show description for 0a4d968
  ywang96

  authored
  0a4d968

  Copy full SHA for 0a4d968

• [Bugfix][SpecDecode] apply sampling parameters to target probabilities for consistency in rejection sampling. (vllm-project#10198)

  Show description for 1bf905d
  jeongin601

  authored
  1bf905d

  Copy full SHA for 1bf905d

• [bugfix] fix the default value of llm_int8_threshold in BitsAndBytesConfig (vllm-project#10657)

  yansh97

  authored
  cfb3bf2

  Copy full SHA for cfb3bf2

• [Hardware][Gaudi]add get_name method for HPUAttentionBackend (vllm-project#10667)

  Show description for e85250b
  jikunshang

  authored
  e85250b

  Copy full SHA for e85250b

• [Misc]Further reduce BNB static variable (vllm-project#10597)

  Show description for 15cc2a9
  jeejeelee

  authored
  15cc2a9

  Copy full SHA for 15cc2a9

• [Kernel] Remove if-else with identical branches in marlin 2:4 (vllm-project#10687)

  Show description for e225110
  tlrmchlsmth

  authored
  e225110

  Copy full SHA for e225110

• [Model] Support telechat2 (vllm-project#10311)

  Show description for 1209261
  

  3 peopleauthored
  1209261

  Copy full SHA for 1209261

• [Bugfix][Hardware][CPU] Fix intel-omp version to avoid segfault (vllm-project#10700)

  Show description for 418cb3b
  bigPYJ1151

  authored
  418cb3b

  Copy full SHA for 418cb3b

• [V1] Update interface for mistral-format Pixtral (vllm-project#10703)

  Show description for 9e0a147
  ywang96

  authored
  9e0a147

  Copy full SHA for 9e0a147

• [ci] fix slow tests (vllm-project#10698)

  Show description for 308cc5e
  youkaichao

  authored
  308cc5e

  Copy full SHA for 308cc5e

• [torch.compile] fix shape specialization (vllm-project#10722)

  Show description for c411def
  youkaichao

  authored
  c411def

  Copy full SHA for c411def

• [Bugfix] Fix GGUF inference with FP16 unquantized checkpoint (vllm-project#10675)

  Show description for b98c62b
  Isotr0py

  authored
  b98c62b

  Copy full SHA for b98c62b

• [Bugfix][Mamba] Fix Multistep on Mamba-like models (vllm-project#10705)

  Show description for 197b448
  mzusman

  authored
  197b448

  Copy full SHA for 197b448

• [Bugfix] Ignore lm_head when loading embedding models (vllm-project#10719)

  Show description for 9b4b150
  DarkLight1337

  authored
  9b4b150

  Copy full SHA for 9b4b150

• [Frontend] don't block event loop in tokenization (preprocess) in OpenAI compatible server (vllm-project#10635)

  Show description for 395b1c7
  tomeras91

  authored
  395b1c7

  Copy full SHA for 395b1c7

Commits on Nov 28, 2024

• [misc] upgrade filelock version (vllm-project#10731)

  Show description for cb4e1c3
  youkaichao

  authored
  cb4e1c3

  Copy full SHA for cb4e1c3

• [Model] support bitsandbytes quantization with minicpm3 model (vllm-project#10682)

  Show description for 70dc14f
  zixuanzhang226

  authored
  70dc14f

  Copy full SHA for 70dc14f

• [Doc] Update model in arch_overview.rst to match comment (vllm-project#10701)

  Show description for 278be67
  spacewander

  authored
  278be67

  Copy full SHA for 278be67

• [Bug][CLI] Allow users to disable prefix caching explicitly (vllm-project#10724)

  Show description for d9b4b3f
  rickyyx

  authored
  d9b4b3f

  Copy full SHA for d9b4b3f

• [V1] Do not allocate beyond the max_model_len (vllm-project#10730)

  Show description for a79b122
  WoosukKwon

  authored
  a79b122

  Copy full SHA for a79b122

• [Kernel] Update vllm-flash-attn version (vllm-project#10736)

  Show description for 9a8bff0
  WoosukKwon

  authored
  9a8bff0

  Copy full SHA for 9a8bff0

• [TPU] Update requirements-tpu (vllm-project#10726)

  Show description for 3ed5e73
  richardsliu

  authored
  3ed5e73

  Copy full SHA for 3ed5e73

• [Model] Added GLM-4 series hf format model support vllm==0.6.4 (vllm-project#10561)

  Show description for 5fc5ce0
  

  3 peopleauthored
  5fc5ce0

  Copy full SHA for 5fc5ce0

• [Kernel] Update vllm-flash-attn version to reduce CPU overheads (vllm-project#10742)

  Show description for 8c1e77f
  WoosukKwon

  authored
  8c1e77f

  Copy full SHA for 8c1e77f

• [V1] Optimize the CPU overheads in FlashAttention custom op (vllm-project#10733)

  Show description for 98f47f2
  WoosukKwon

  authored
  98f47f2

  Copy full SHA for 98f47f2

• [Model] Add Internlm2 LoRA support (vllm-project#5064)

  Show description for c83919c
  Isotr0py

  authored
  c83919c

  Copy full SHA for c83919c

Commits on Nov 29, 2024

• [Model] Clean up MiniCPMV (vllm-project#10751)

  Show description for fa6ecb9
  DarkLight1337

  authored
  fa6ecb9

  Copy full SHA for fa6ecb9

• [Misc] typo find in sampling_metadata.py (vllm-project#10740)

  noooop

  authored
  c82b432

  Copy full SHA for c82b432

• [Bugfix] Fix Idefics3 bug (vllm-project#10778)

  Show description for 3132aac
  jeejeelee

  authored
  3132aac

  Copy full SHA for 3132aac

• [platform] Add verify_quantization in platform. (vllm-project#10757)

  Show description for 661175b
  wangxiyuan

  authored
  661175b

  Copy full SHA for 661175b

Commits on Nov 30, 2024

• [Bugfix] Fix OpenVino/Neuron driver_worker init (vllm-project#10779)

  Show description for 40bc242
  

  NickLucche

  and

  DarkLight1337

  authored
  40bc242

  Copy full SHA for 40bc242

• [Model] Refactor Molmo weights loading to use AutoWeightsLoader (vllm-project#10771)

  Show description for 16ee07f
  Isotr0py

  authored
  16ee07f

  Copy full SHA for 16ee07f

• [Interleaved ATTN] Support for Mistral-8B (vllm-project#10591)

  Show description for e7cfc4e
  

  patrickvonplaten

  and

  youkaichao

  authored
  e7cfc4e

  Copy full SHA for e7cfc4e

• [doc] format fix (vllm-project#10789)

  Show description for 7e4bbda
  wangxiyuan

  authored
  7e4bbda

  Copy full SHA for 7e4bbda

Commits on Dec 1, 2024

• [Model] Replace embedding models with pooling adapter (vllm-project#10769)

  Show description for 1337071
  DarkLight1337

  authored
  1337071

  Copy full SHA for 1337071

• [Misc] Improve type annotations for support_torch_compile (vllm-project#10763)

  Show description for f877a7d
  DarkLight1337

  authored
  f877a7d

  Copy full SHA for f877a7d

• [Misc] Rename embedding classes to pooling (vllm-project#10801)

  Show description for d2f058e
  DarkLight1337

  authored
  d2f058e

  Copy full SHA for d2f058e

• [doc] add warning about comparing hf and vllm outputs (vllm-project#10805)

  Show description for 169a0ff
  youkaichao

  authored
  169a0ff

  Copy full SHA for 169a0ff

• [Misc] Adding MMMU-Pro vision dataset to serving benchmark (vllm-project#10804)

  Show description for c11f172
  

  3 peopleauthored
  c11f172

  Copy full SHA for c11f172

Commits on Dec 2, 2024

• [Core] Implement disagg prefill by StatelessProcessGroup (vllm-project#10502)

  Show description for 0590ec3
  KuntaiDu

  authored
  0590ec3

  Copy full SHA for 0590ec3

• [Model] Add BNB support to Llava and Pixtral-HF (vllm-project#10795)

  Show description for b18c9bb
  Isotr0py

  authored
  b18c9bb

  Copy full SHA for b18c9bb

• [core] Avoid metrics log noise when idle - include speculative decodi… (vllm-project#10809)

  cduk

  authored
  b795477

  Copy full SHA for b795477

• [Kernel] Use out arg in flash_attn_varlen_func (vllm-project#10811)

  Show description for 073a4bd
  WoosukKwon

  authored
  073a4bd

  Copy full SHA for 073a4bd

• Fill TorchSDPAAttentionMetadata seq_lens_field for prefill (vllm-project#10799)

  Show description for e25810a
  maxdebayser

  authored
  e25810a

  Copy full SHA for e25810a

• [misc] remove xverse modeling file (vllm-project#10814)

  Show description for 63a1641
  youkaichao

  authored
  63a1641

  Copy full SHA for 63a1641

• [doc]Update config docstring (vllm-project#10732)

  Show description for 995a148
  wangxiyuan

  authored
  995a148

  Copy full SHA for 995a148

• [Model]: add some tests for aria model (vllm-project#10770)

  Show description for ef31eab
  

  xffxff

  and

  Isotr0py

  authored
  ef31eab

  Copy full SHA for ef31eab

• [CI/Build] Update mistral_common version for tests and docs (vllm-project#10825)

  Show description for e95f275
  DarkLight1337

  authored
  e95f275

  Copy full SHA for e95f275

• [misc] use out argument for flash attention (vllm-project#10822)

  Show description for a4c4daf
  youkaichao

  authored
  a4c4daf

  Copy full SHA for a4c4daf

• [Misc][LoRA] Move the implementation of lora bias to punica.py (vllm-project#10829)

  Show description for b45f0d7
  jeejeelee

  authored
  b45f0d7

  Copy full SHA for b45f0d7

• [Misc][XPU] Avoid torch compile for XPU platform (vllm-project#10747)

  Show description for 519cc6c
  

  yma11

  and

  youkaichao

  authored
  519cc6c

  Copy full SHA for 519cc6c

• Fix openvino on GPU (vllm-project#10793)

  janimo

  authored
  9b14d97

  Copy full SHA for 9b14d97

• [Model] Add TP and BNB quantization support to LlavaMultiModalProjector (vllm-project#10834)

  Show description for 4c05edb
  

  Isotr0py

  and

  DarkLight1337

  authored
  4c05edb

  Copy full SHA for 4c05edb

Commits on Dec 3, 2024

• [Bugfix] Prevent benchmark_throughput.py from using duplicated random prompts (vllm-project#10753)

  mgoin

  authored
  4433195

  Copy full SHA for 4433195

• [Model] support bitsandbytes quantization with minicpm model (vllm-project#10842)

  Show description for d746268
  zixuanzhang226

  authored
  d746268

  Copy full SHA for d746268

• [Bugfix] Fix QKVParallelLinearWithShardedLora bias bug (vllm-project#10844)

  Show description for a4cf256
  jeejeelee

  authored
  a4cf256

  Copy full SHA for a4cf256

• [core][distributed] add pynccl broadcast (vllm-project#10843)

  Show description for 21fe7b4
  youkaichao

  authored
  21fe7b4

  Copy full SHA for 21fe7b4

• [torch.compile] remove compilation_context and simplify code (vllm-project#10838)

  Show description for dc5ce86
  youkaichao

  authored
  dc5ce86

  Copy full SHA for dc5ce86

• [Doc] Add github links for source code references (vllm-project#10672)

  Show description for ef51831
  

  russellb

  and

  DarkLight1337

  authored
  ef51831

  Copy full SHA for ef51831

• [Misc] Remove deprecated names (vllm-project#10817)

  Show description for 3257d44
  DarkLight1337

  authored
  3257d44

  Copy full SHA for 3257d44

• [Core][Performance] Add XGrammar support for guided decoding and set it as default (vllm-project#10785)

  Show description for 9323a31
  

  aarnphm

  and

  mgoin

  authored
  9323a31

  Copy full SHA for 9323a31

• [Speculative Decoding] Move indices to device before filtering output (vllm-project#10850)

  Show description for f6084f6
  

  zhengy001

  and

  Yang Zheng(SW)(Alex)

  authored
  f6084f6

  Copy full SHA for f6084f6

• [V1] VLM - Run the mm_mapper preprocessor in the frontend process (vllm-project#10640)

  Show description for 3bc94ca
  

  3 peopleauthored
  3bc94ca

  Copy full SHA for 3bc94ca

• [MISC][XPU] quick fix for XPU CI (vllm-project#10859)

  Show description for 2f2cdc7
  yma11

  authored
  2f2cdc7

  Copy full SHA for 2f2cdc7

• [Bugfix] Only require XGrammar on x86 (vllm-project#10865)

  Show description for 7090c27
  mgoin

  authored
  7090c27

  Copy full SHA for 7090c27

• [Frontend] correctly record prefill and decode time metrics (vllm-project#10853)

  Show description for 7c32b68
  tomeras91

  authored
  7c32b68

  Copy full SHA for 7c32b68

• [Build][Bugfix] Using the correct type hint (vllm-project#10866)

  Show description for a061fe6
  gshtras

  authored
  a061fe6

  Copy full SHA for a061fe6

Commits on Dec 4, 2024

• [Benchmark] Benchmark structured output with datasets (vllm-project#10557)

  Show description for 381ac93
  

  xuechendi

  and

  aarnphm

  authored
  381ac93

  Copy full SHA for 381ac93

• [CI/Build] Replace mean with torch.all in test_pynccl.py (vllm-project#10876)

  Show description for d2bd88b
  tlrmchlsmth

  authored
  d2bd88b

  Copy full SHA for d2bd88b

• Drop ROCm load format check (vllm-project#10767)

  Show description for b5b647b
  wangxiyuan

  authored
  b5b647b

  Copy full SHA for b5b647b

• [ci/build] Change queue name for Release jobs (vllm-project#10875)

  khluu

  authored
  fa2dea6

  Copy full SHA for fa2dea6

• [ci/build] Job to build and push release image (vllm-project#10877)

  khluu

  authored
  c9ca4fc

  Copy full SHA for c9ca4fc

• [bugfix] fixed parameter “n” when set parameter “bestof” > 1 (vllm-project#10854)

  Show description for 8db957e
  o2363286

  authored
  8db957e

  Copy full SHA for 8db957e

• [ci/build] Update vLLM postmerge ECR repo (vllm-project#10887)

  khluu

  authored
  c92acb9

  Copy full SHA for c92acb9

Commits on Dec 5, 2024

• copy origin file to mm_ files

  

  jikunshang

  authored and

  xuechendi

  committed
  41685cb

  Copy full SHA for 41685cb

• support multi models

  Show description for 883fac0
  

  jikunshang

  authored and

  xuechendi

  committed
  883fac0

  Copy full SHA for 883fac0

• add model arg to original client

  Show description for 655b610
  xuechendi

  committed
  655b610

  Copy full SHA for 655b610

• fix GPU memory allocation for multi_models

  Show description for e4ad1c5
  xuechendi

  committed
  e4ad1c5

  Copy full SHA for e4ad1c5