I see a few potential sources of issues with Gwen's UnicodeString typedef:
1) It's a wstring. On Windows, this would be 16-bit characters with UTF16LE encoding. On most other systems, this would be 32-bit characters with implied UTF-32 encoding (although standard C and C++ functions may not actually think its Unicode -- some systems have locale-dependent wide character encodings).
2) ctype::narrow/ctype::widen (and thus, the UnicodeToString and StringToUnicode utility functions) are defined as simply being "the simplest possible transformation", which I take to mean that the characters in the wstring are simply truncated or the characters in the string are simply expanded; although being a locale facet maybe there is actually an encoding conversion when appropriate. But even if that's the case, since it is a locale facet, outside localization code might cause unexpected behavior.
This could be fixed by changing the typedef to std::basic_string<uint32_t> for UTF-32 or std::basic_string<uint16_t> for UTF-16; or even by simply treating an std::string as a UTF-8 encoded string (note: you can never trust basic_string<T>::length if you're using it for Unicode). My personal recommendation would be UTF-8: it's endian-independent, most Windowing and Font APIs either support or expect it already, and conversion to wchar_t on Windows is simple enough. A major advantage of UTF-8 is that ASCII requires no conversion, so string literals can be used directly in function calls.
you should write a letter to garry.
Or a pm, or just write a fix and request a pull.
Hi, as the Unicode expert of Facepunch (lol) I opened this issue a while back: [url]https://github.com/garrynewman/GWEN/issues/3[/url]
To my knowledge BlackPhoenix has completed an almost entire UTF-8 port of GWEN, which I've been pestering him about. Recently he stopped returning my PMs and switched his entire project to Qt. I presume he's MarbleHornets-style burned the source code.
If you want to discuss this further, feel free to PM me as I seem to be the only one who cares at this point.
Sorry, you need to Log In to post a reply to this thread.
