Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add padding to encoder_seq_lens #610

Merged

Conversation

kdamaszk
Copy link

@kdamaszk kdamaszk commented Dec 10, 2024

Without this change we can observe below error:

[rank0]:   File "/software/users/kdamaszke/repos/vllm-fork/vllm/model_executor/models/mllama.py", line 959, in forward
[rank0]:     full_text_row_masked_out_mask = full_text_row_masked_out_mask.view(
[rank0]: RuntimeError: shape '[4, -1, 1]' is invalid for input of size 3

It occurs when one of the requests is removed from the batch earlier. In that case, language model is still working on the shapes padded to the bucketed batch size, while encoder input doesn't. This change is aligning the batch size on encoder_seq_lens to the expected one.

@kdamaszk kdamaszk requested a review from jkaniecki December 10, 2024 16:26
@yisonzhu
Copy link

LGTM.

@michalkuligowski michalkuligowski merged commit 449a89d into habana_main Dec 12, 2024
10 checks passed
@michalkuligowski michalkuligowski deleted the dev/kdamaszke/fix-padding-on-encoder-seq-lens branch December 12, 2024 08:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants