Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nov 6 rebase (sans vllm-project#6143) #468

Merged
merged 25 commits into from
Nov 6, 2024
Merged

Nov 6 rebase (sans vllm-project#6143) #468

merged 25 commits into from
Nov 6, 2024

Conversation

kzawora-intel
Copy link

This PR adds all commits before vllm-project#6143 without vllm-project#6143.

russellb and others added 24 commits November 5, 2024 07:50
@russellb
[Frontend] Fix tcp port reservation for api server (vllm-project#10012)
5952d81 
Signed-off-by: Russell Bryant <[email protected]>
@richardsliu
Refactor TPU requirements file and pin build dependencies (vllm-proje…
cd34029 
…ct#10010)

Signed-off-by: Richard Liu <[email protected]>
@yangalan123 @youkaichao
[Misc] Add logging for CUDA memory (vllm-project#10027)
09d3550 
Signed-off-by: Chenghao Yang <[email protected]>
Signed-off-by: youkaichao <[email protected]>
Co-authored-by: Chenghao Yang <[email protected]>
Co-authored-by: youkaichao <[email protected]>
@russellb
[CI/Build] Limit github CI jobs based on files changed (vllm-project#…
731aec5 
…9928)

Signed-off-by: Russell Bryant <[email protected]>
@mgoin
[Model] Support quantization of PixtralHFTransformer for PixtralHF (v…
a53046b 
…llm-project#9921)

Signed-off-by: mgoin <[email protected]>
@lk-chen
[Feature] Update benchmark_throughput.py to support image input (vllm…
d2e8033 
…-project#9851)

Signed-off-by: Linkun Chen <[email protected]>
Co-authored-by: Linkun Chen <[email protected]>
@jeejeelee
[Misc] Modify BNB parameter name (vllm-project#9997)
b9c64c0 
Signed-off-by: Jee Jee Li <[email protected]>
@mgoin
[CI] Prune tests/models/decoder_only/language/* tests (vllm-project#9940
0246246 
)

Signed-off-by: mgoin <[email protected]>
@mgoin
[CI] Prune back the number of tests in tests/kernels/* (vllm-project#…
235366f 
…9932)

Signed-off-by: mgoin <[email protected]>
@youkaichao
[bugfix] fix weak ref in piecewise cudagraph and tractable test (vllm…
ca9844b 
…-project#10048)

Signed-off-by: youkaichao <[email protected]>
@zifeitong
[Bugfix] Properly propagate trust_remote_code settings (vllm-project#…
43300bd 
…10047)

Signed-off-by: Zifei Tong <[email protected]>
@wallashss
[Bugfix] Fix pickle of input when async output processing is on (vllm…
966e316 
…-project#9931)

Signed-off-by: Wallas Santos <[email protected]>
@llsj14 @LiuXiaoxuanPKU
[Bugfix][SpecDecode] kv corruption with bonus tokens in spec decode (v…
0c63c34 
…llm-project#9730)

Co-authored-by: LiuXiaoxuanPKU <[email protected]>
@youkaichao
[v1] reduce graph capture time for piecewise cudagraph (vllm-project#…
c4cacba 
…10059)

Signed-off-by: youkaichao <[email protected]>
@DarkLight1337
[Misc] Sort the list of embedding models (vllm-project#10037)
82bfc38 
Signed-off-by: DarkLight1337 <[email protected]>
@petersalas
[Model][OpenVINO] Fix regressions from vllm-project#8346 (vllm-projec…
ffc0f2b 
…t#10045)

Signed-off-by: Peter Salas <[email protected]>
@tjohnson31415
[Bugfix] Fix edge-case crash when using chat with the Mistral Tekken …
2bcbae7 
…Tokenizer (vllm-project#10051)

Signed-off-by: Travis Johnson <[email protected]>
@arakowsk-amd
[Bugfix] Gpt-j-6B patch kv_scale to k_scale path (vllm-project#10063)
ea928f6 
Signed-off-by: Alex Rakowski <[email protected]>
Signed-off-by: Alex Rakowski <[email protected]>
@zifeitong
[Bugfix] Remove CustomChatCompletionContentPartParam multimodal input…
9d59b75 
… type (vllm-project#10054)

Signed-off-by: Zifei Tong <[email protected]>
@WoosukKwon
[V1] Integrate Piecewise CUDA graphs (vllm-project#10058)
4089985 
Signed-off-by: Woosuk Kwon <[email protected]>
@youkaichao
[distributed] add function to create ipc buffers directly (vllm-proje…
4be3a45 
…ct#10064)

Signed-off-by: youkaichao <[email protected]>
@aarnphm
[CI/Build] drop support for Python 3.8 EOL (vllm-project#8464)
21063c1 
Signed-off-by: Aaron Pham <[email protected]>
@Isotr0py
[CI/Build] Fix large_gpu_mark reason (vllm-project#10070)
a5fda50 
Signed-off-by: Isotr0py <[email protected]>
@kzawora-intel
Merge commit 'a5fda50a10641e47c0c290907f30ef2add6d4e7a' into HEAD
40882f3
github-advanced-security[bot]
github-advanced-security bot found potential problems Nov 6, 2024
View reviewed changes
vllm/entrypoints/openai/api_server.py
@@ -569,7 +569,8 @@
# This avoids race conditions with ray.
# see https://github.com/vllm-project/vllm/issues/8204
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(("", args.port))
sock.bind((args.host or "", args.port))

Check warning

Code scanning / CodeQL

Binding a socket to all network interfaces Medium

'' binds a socket to all interfaces.

Copilot Autofix AI 2 months ago

To fix the problem, we need to bind the socket to a specific network interface instead of all interfaces. This can be achieved by modifying the sock.bind call to use a specific IP address. The best way to do this without changing existing functionality is to use the IP address provided in the args.host parameter. If args.host is not provided, we should raise an error or use a default secure IP address.

  1. Modify the sock.bind call on line 572 to use args.host directly.
  2. Ensure that args.host is not an empty string or 0.0.0.0. If it is, raise an error or use a default secure IP address.
Suggested changeset 1
vllm/entrypoints/openai/api_server.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/vllm/entrypoints/openai/api_server.py b/vllm/entrypoints/openai/api_server.py
--- a/vllm/entrypoints/openai/api_server.py
+++ b/vllm/entrypoints/openai/api_server.py
@@ -571,3 +571,5 @@
     sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-    sock.bind((args.host or "", args.port))
+    if not args.host or args.host in ["", "0.0.0.0"]:
+        raise ValueError("Invalid host address. Please provide a specific IP address.")
+    sock.bind((args.host, args.port))
     sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
EOF
@@ -571,3 +571,5 @@
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((args.host or "", args.port))
if not args.host or args.host in ["", "0.0.0.0"]:
raise ValueError("Invalid host address. Please provide a specific IP address.")
sock.bind((args.host, args.port))
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
Copilot is powered by AI and may make mistakes. Always verify output.
Positive Feedback
Negative Feedback

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Please select one or more of the options
@kzawora-intel kzawora-intel merged commit 5eb7f3d into habana_main Nov 6, 2024
6 checks passed
@michalkuligowski michalkuligowski deleted the private/kzawora/nov_6_rebase branch November 6, 2024 13:09
@kzawora-intel kzawora-intel added the habana Issues or PRs submitted by Habana Labs label Nov 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@michalkuligowski michalkuligowski michalkuligowski approved these changes

Assignees
No one assigned
Labels
habana Issues or PRs submitted by Habana Labs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.