Our library @lenml/llama2-tokenizer has been deprecated. We are excited to introduce our new library @lenml/tokenizers as its replacement, offering a broader set of features and an enhanced experience.
- Fully Compatible with
transformers.jsInterfaces: Seamlessly supports all interfaces defined intransformers.js, making migration and integration effortless. - Support for a Wide Range of Models: Regardless of which model you need, our new library supports it, ensuring broader coverage.
- Rich Feature Implementation: Includes a complete implementation of chat templates and normalizers to better serve your text processing and tokenization needs.
check out lenML/tokenizers.
here is demo for llama2-tokenizer.js package
deploy on github pages
- dark theme
- tokenizer comparer
- fix react-highlight-within-textarea
text style(when paste) issue