GitHub - ZhuoruLin/fast-wordpiece: Implementation of fast wordpiece tokenization algorithm from https://aclanthology.org/2021.emnlp-main.160.pdf in rust

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
src		src
.gitignore		.gitignore
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml

About

Implementation of fast wordpiece tokenization algorithm from https://aclanthology.org/2021.emnlp-main.160.pdf in rust

No releases published

No packages published