BPETokenizer splits grapheme clusters rather than code points -- crashes in Gemma merges

[davidkoski]

The BPETokenizer currently has code like this:

            let tuple = item.split(separator: " ").map { String($0) }

That doesn't split some merges correctly, e.g. in Gemma it has ▁ह ै. In swift the split() on String splits on grapheme clusters, not code points. As-is, this will crash on the Gemma merges.

I think you want to use:

String.UnicodeScalarView.split(separator:)

to perform this split on the code points instead.

[pcuenca]

Thanks @davidkoski, I think you are right! I'll fix this issue and then work on #42, it's becoming high priority!

[pcuenca]

Fixed in #52, thanks again for reporting!