feat: Refactor post-processing of adapters #345
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Sukriti-Sharma4 <[email protected]>
|
Thanks for making a pull request! 😃 |
Signed-off-by: Sukriti-Sharma4 <[email protected]>
Signed-off-by: Sukriti-Sharma4 <[email protected]>
Ssukriti
changed the title
Signed-off-by: Sukriti-Sharma4 <[email protected]>
Signed-off-by: Sukriti-Sharma4 <[email protected]>
Signed-off-by: Sukriti-Sharma4 <[email protected]>
Ssukriti
added a commit
that referenced
this pull request
Sep 23, 2024
* refactor saving tokens metadata Signed-off-by: Sukriti-Sharma4 <[email protected]> * remove extra check Signed-off-by: Sukriti-Sharma4 <[email protected]> * post processing script Signed-off-by: Sukriti-Sharma4 <[email protected]> * post processing script Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix: unit test args Signed-off-by: Sukriti-Sharma4 <[email protected]> * undo post_process_vLLm flag Signed-off-by: Sukriti-Sharma4 <[email protected]> --------- Signed-off-by: Sukriti-Sharma4 <[email protected]>
Ssukriti
added a commit
that referenced
this pull request
Sep 25, 2024
* utilities to post process checkpoint for LoRA Signed-off-by: Sukriti-Sharma4 <[email protected]> * improve code comments Signed-off-by: Sukriti-Sharma4 <[email protected]> * Add unit test and fix some lint errors Signed-off-by: Angel Luu <[email protected]> * lint: fix more fmt errors Signed-off-by: Angel Luu <[email protected]> * feat: Add post_process_vLLM_adapters_new_tokens function to main Signed-off-by: Will Johnson <[email protected]> * fmt Signed-off-by: Will Johnson <[email protected]> * fix: Add post processing flag so post processing is only done for vLLM Signed-off-by: Will Johnson <[email protected]> * fix: get num_added_tokens from resize function (#344) * get num_added_tokens Signed-off-by: Sukriti-Sharma4 <[email protected]> * remove extra code Signed-off-by: Sukriti-Sharma4 <[email protected]> --------- Signed-off-by: Sukriti-Sharma4 <[email protected]> * Ran fmt and also removed unneccessary files from test artifact Signed-off-by: Angel Luu <[email protected]> * fix: unit tests Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix: Adding tokens in special_tokens_dict Signed-off-by: Abhishek <[email protected]> * fix: Add additional arg to tests to reflect new flag post_process_vllm Signed-off-by: Will Johnson <[email protected]> * fmt Signed-off-by: Will Johnson <[email protected]> * feat: Refactor post-processing of adapters (#345) * refactor saving tokens metadata Signed-off-by: Sukriti-Sharma4 <[email protected]> * remove extra check Signed-off-by: Sukriti-Sharma4 <[email protected]> * post processing script Signed-off-by: Sukriti-Sharma4 <[email protected]> * post processing script Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix: unit test args Signed-off-by: Sukriti-Sharma4 <[email protected]> * undo post_process_vLLm flag Signed-off-by: Sukriti-Sharma4 <[email protected]> --------- Signed-off-by: Sukriti-Sharma4 <[email protected]> * add test for LoRA tuning from main Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix formatting Signed-off-by: Sukriti-Sharma4 <[email protected]> * correcting post processing script Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix:post-process in place Signed-off-by: Sukriti-Sharma4 <[email protected]> * update documentation for post-processing Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix:formatting Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix:linting Signed-off-by: Sukriti-Sharma4 <[email protected]> * more warnings /exceptions in script Signed-off-by: Sukriti-Sharma4 <[email protected]> * check for no tokens added Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix:linting Signed-off-by: Sukriti-Sharma4 <[email protected]> * additional unit test Signed-off-by: Sukriti-Sharma4 <[email protected]> * add more tests Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix:tokenizer test Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix:linting and docstrings Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix:return type of trainer Signed-off-by: Sukriti-Sharma4 <[email protected]> * test: enable tests and fix copytree Signed-off-by: Anh Uong <[email protected]> * use copy function from build Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix:linting and formatting Signed-off-by: Sukriti-Sharma4 <[email protected]> * make build a module Signed-off-by: Sukriti-Sharma4 <[email protected]> * add back old copy function Signed-off-by: Sukriti-Sharma4 <[email protected]> --------- Signed-off-by: Sukriti-Sharma4 <[email protected]> Signed-off-by: Angel Luu <[email protected]> Signed-off-by: Will Johnson <[email protected]> Signed-off-by: Abhishek <[email protected]> Signed-off-by: Anh Uong <[email protected]> Co-authored-by: Angel Luu <[email protected]> Co-authored-by: Will Johnson <[email protected]> Co-authored-by: Abhishek <[email protected]> Co-authored-by: Anh Uong <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of the change
Related issue number
How to verify the PR
Was the PR tested