Multibyte characters (like Emojis) corrupt the file on disk

[avioli]

How to reproduce:

1. Open a file with content
2. Add a couple of emojis
3. Add a non-emoji character, eg. single-byte
4. Open your file locally - it will be corrupt in an unexpected way
5. (Extra) Delete the emojis
6. Reopen/update the locally opened file - it will have more characters removed than just the emojis.

The only way to "fix" this is to reopen the file within leaps, but after it was cleaned locally.

[Jeffail]

Hey @avioli, thanks for raising this. Can you paste a sample of the emojis that cause the corruption? I think I have this reproduced but just want to be sure.

I believe this issue is with UTF-16 surrogate pairs within the ACE editor ajaxorg/ace#1153, if you could post any logged output from leaps that could help confirm it.

I can probably find a fix for this by changing how leaps creates the transforms from ACE input. I'll update as soon as I find out more. Thanks again!

[Jeffail]

Hey @avioli, so ES6 is fairly common place in modern browsers and therefore I've managed to create a client library that supports emojis (or any >2 byte characters). The caveat here is that old browsers will fallback on a client that doesn't support this fix. You can either pull the master branch and build, or grab a binary from https://github.com/Jeffail/leaps/releases/tag/v0.6.0 to test it.