-
Notifications
You must be signed in to change notification settings - Fork 289
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Summary: To 2.1 Unfortunately, I can't remove the old version of the crate. The problem is that it appears in the [public API](https://docs.rs/tiktoken-rs/latest/tiktoken_rs/struct.CoreBPE.html#method.new) of `tiktoken_rs`, which we also import. That crate has a PR to upgrade its dependency [here](zurawiki/tiktoken-rs#95) but that has not been merged. Alternatives here are: 1. Fork `tiktoken_rs`. This seems worse. 2. It's strictly speaking possible to use generics combined with some other tricks to avoid needing to name v1 of `FxHashMap` in fbsource. However, that makes that code extraordinarily hard to change and work with, and doesn't seem all that clearly motivated. If there's something else I should be doing here, all ears Reviewed By: zertosh Differential Revision: D68064854 fbshipit-source-id: bd1c96abbc0ebaa178a55826bc85c61eb35e8301
- Loading branch information
1 parent
de68be9
commit 29bb259
Showing
4 changed files
with
5 additions
and
9 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters