OpenGL* renderer: UTF-8 multi-byte symbols are not rendering

[Megaxela]

Issue:

• Operating System (name & version): Linux (Archlinux)
• Compiler (name & version): gcc 8.2
• Target renderer: OpenGL /OpenGL Core Profile
• cmake command-line used: cmake -DRENDER_OPENGL=true ..

See title.
I guess I can begin research about this issue is the near future.

[billyquith]

May be because the STB pre-renderer characters don't include those outside ASCII range?

[Megaxela]

Yes, that may be true. But also `typedef` for `UnicodeCharacter` defined as 1 byte `char` may be a part of a problem.

…

On Sat, May 19, 2018, 18:16 Bill Quith ***@***.***> wrote: May be because the STB pre-renderer characters don't include those outside ASCII range? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#80 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABY5eKBEgVINVyWzgbNWhaa5de4LinHYks5t0DdggaJpZM4UFuvp> .

[billyquith]

I think this is your problem because pre-rendered text

[billyquith]

Yes, that may be true. But also typedef for UnicodeCharacter defined as
1 byte char may be a part of a problem.

All the text is UTF-8, but yes, I haven't dealt with multi-byte character encodings on all platforms. I don't know how STB font render works with UTF-8 multi-byte codes.

[Megaxela]

Ok, I see issue, I will try to figure out how stb works with multibyte symbols as soon as possible. And maybe only required fix will be changing rendering character range and changing of UnicodeCharacter type.

…

On Sat, May 19, 2018, 18:21 Bill Quith ***@***.***> wrote: All the text is UTF-8, but yes, I haven't dealt with multi-byte character encodings on all platforms. I don't know how STB font render works with UTF-8 multi-byte codes. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#80 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABY5eMHvrCm_R3e2NjthgtTz57NvSeDxks5t0DiJgaJpZM4UFuvp> .

[Megaxela]

I researched problem and found several places.

1. In PlatformType.h type UnicodeChar is char so useful data will be cut out.
2. No conversion from UTF-8 to unicode values (in renderers). It can be fixed with utfcpp library.
3. stb library can bake Unicode characters, but it can accept only first character unicode index and number of character to render. It has no any interface to perform more flexible baking. It may be a problem if required to render only specific sets of glyphs. Maybe project requires another (or modified) system of font baking. (I guess, just copy and modify stbs stbtt_BakeFontBitmap function can be enough. Also, this function is rendering all unknown symbols multiple times, instead of just once. (see img))
   

[billyquith]

In PlatformType.h type UnicodeChar is char so useful data will be cut out.

Ok, that can be made an int.

No conversion from UTF-8 to unicode values (in renderers). It can be fixed with utfcpp library.

I don't really want to have dependencies on many external libraries. Is there specific functionality that is required? I think UTF-8 to int may be all that is necessary and that is a one function that could go in Utility.

stb library can bake Unicode characters, ...

Yep. I think Allegro just bakes characters on the fly, whatever they are. I just added this range to get this to work. I mainly use Allegro so that is probably the most complete solution here.

Does STB remember the baked information? I.e. can you later bake more characters into the same texture? In which case you could just cache them like Allegro. And render from the cache.

[Megaxela]

Ok, that can be made an int.

I guess uint32_t would be better. But yes, 32 bit variable will be enough.

I don't really want to have dependencies on many external libraries.

It's pretty small, and contains of only several headers. But I'm agree, that dependencies are bad.

Is there specific functionality that is required?

I guess yes, because utf-8 is a specification, where one symbol can be described in range of 1 to 6 bytes, but after decoding it will describe maximum 4 byte variable. But I think I can implement it myself without any particular problems.

Does STB remember the baked information?

Nope, It's just saving 1 byte per pixel image to buffer. It's not caching glyphs. I guess I can cache buffer for font and if required - add symbols to it, then update texture in GPU and continue rendering. This implementation also requires custom baking function.

[billyquith]

I guess uint32_t would be better.

Agreed.

But I think I can implement it myself without any particular problems.

If the cpplib license is liberal you could copy the function and attribute it? I think a lot of the other functionality should be handled by C++1x.

I guess I can cache buffer for font and if required...

TBH I don't know what is required here. Maybe have a look at the Allegro strategy. I think they have multiple texture glyph atlases. New one added when it fills up. Each new font/glyph/size is cached when it is used. Not sure how easy this would be in STB. Perhaps need a rect packer (which I think STB has).

[billyquith]

Perhaps this is what is needed: https://github.com/memononen/fontstash

Font stash is light-weight online font texture atlas builder written in C. It uses stb_truetype to render fonts on demand to a texture atlas.

The code is split in two parts, the font atlas and glyph quad generator fontstash.h, and an example OpenGL backend (glstash.h).

• 1 file dependency for fontstash and one per backend. 👍

This modification sounds interesting: https://github.com/akrinke/Font-Stash/ from memononen/fontstash#24.

The idea with the ResourceLoader is that there could be different flavours and users can choose an appropriate one. Likely, they would write their own because most games have their own resource management. Perhaps another one could be added that supports FontStash, and/or its variant (@akrinke).

[Megaxela]

I checked out https://github.com/akrinke/Font-Stash/. It's pretty interesting, but it's renderer uses OpenGL Compat Profile, that's not usable with OpenGL Core Profile. So font rendering algorithm will be reimplemented for OpenGL Core Profile. Also there is an stb as dependency, and GWork has it already.

ResourceLoader may be used with sth_add_font_from_memory.

[billyquith]

I checked out https://github.com/akrinke/Font-Stash/. It's pretty interesting

I guess a concern is that it looks like a significant fork from fontstash, and that receives bug fixes and new features, so this fork would get less/no support. But I guess if it's all functional and looks reasonable to maintain then it will effectively just become part of GWork.

[billyquith]

Might be useful https://github.com/DuffsDevice/tinyutf8

[Megaxela]

Hmm. This is extremely interesting library. First of all it has to be tested a little bit and if everything will be fine, it can replace standard string without any problems, I guess.
The only question to that library: why is this library using any .cpp files, if it would be better to implement it like "single header" library.

[billyquith]

if it would be better to implement it like "single header" library.

Ah. I thought is was single header as no source directory (cpp in "Lib"!).

License is liberal. If necessary you could take what you need and attribute?

[topblast]

I was playing with https://github.com/memononen/fontstash on the DirectX11 renderer. This method is a better solution than the hack of expanding the limited character range.

static const wchar_t BeginCharacter = L' '; // First Character of Wide Character Table
static const wchar_t LastCharacter = 0x2FFF; // Last Character of Wide Character Table
static const wchar_t NewLineCharacter = L'\n'; // New Line Character

topblast@e2b3e6e
I am planning to update the DirectX11 renderer to use a similar implementation that fits better into GWork; it should be portable into OpenGL* and possibly other renderers.

[billyquith]

I'm gong to turn this into a bug and open a new issue for the "font module" #88 new feature.