Intern2 habana #489
Merged
Intern2 habana #489
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 3 commits
November 12, 2024 23:13
changing view to shape in split_qvk changing view to shape in split_qvk changing view to shape in split_qvk internlm2 habana compatibility
kzawora-intel
requested changes
Nov 21, 2024
| @@ -144,20 +144,23 @@ def __init__( | |||
| ) | |||
|
|
|||
| def split_qkv(self, qkv: torch.Tensor): | |||
| seq_len = qkv.shape[0] | |||
| batch_size, seq_len, _ = qkv.shape | |||
There was a problem hiding this comment.
This will break all non-HPU accelerators. Can we make this more generic? E.g. pass *qkv.shape[:-1] into reshape, so that 2D tensors will work as well?
kzawora-intel
approved these changes
Nov 26, 2024
michalkuligowski
requested changes
Nov 26, 2024
|
|
||
| if self.tp_size > 1: | ||
| splitter = partial(split_tensor_along_last_dim, | ||
| num_partitions=self.tp_size) | ||
| q = splitter(q)[self.tp_rank] | ||
| k = splitter(k)[self.tp_rank] | ||
| v = splitter(v)[self.tp_rank] | ||
|
|
There was a problem hiding this comment.
Yapf fails on this empty line, please remove
| @@ -144,27 +144,33 @@ def __init__( | |||
| ) | |||
|
|
|||
| def split_qkv(self, qkv: torch.Tensor): | |||
| seq_len = qkv.shape[0] | |||
| # Unpack all dimensions except the last one | |||
| *batch_dims, last_dim = qkv.shape | |||
There was a problem hiding this comment.
last_dim seems that is not used, please replace with _
There was a problem hiding this comment.
The "Trigger Jenkins Tests" failure can be ignored now
michalkuligowski
approved these changes
Nov 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Making sure the model runs on habana devices. Original code did not run due to error in the split_qkv code as param unpacking was assuming lack of batch dimension. Tested inference with the changes and InternLM2 works on Gaudi2 as expected.