refactor: move tokenizer_data_utils with the rest of utils, add further unit testing.#348

Merged

willmj merged 7 commits intofoundation-model-stack:mainfrom willmj:1344

Oct 8, 2024

+75-22

Commits on Sep 25, 2024

cleanup: Move tokenizer_data_utils to /utils to /data, change imports
willmj
committed
tests: Add additional tests for test_embedding_resize to check resize with special tokens, resize multiple of. fmt
willmj
committed

Commits on Oct 4, 2024

Commits on Oct 7, 2024

fix: more thorough testing from output of function
willmj
committed

Commits on Oct 8, 2024