History log of /haiku/src/kits/locale/TextEncoding.cpp
Revision Date Author Comments
# f638c82a 29-Sep-2016 Adrien Destugues <pulkomandy@pulkomandy.tk>

StyledText: fix reading of utf-8 documents

BNode::ReadAttrString does not check for the attribute to actually be a
string. It will return B_OK (and useless data) if called on a
B_INT32_TYPE attribute. Check the attribute type before reading.

Also remove various debug ouput leftovers.

May fix #12976.


# a71b10d7 27-Sep-2016 Adrien Destugues <pulkomandy@pulkomandy.tk>

BTextEncoding: fix naming and move to BPrivate namespace.


# 08e52491 25-Sep-2016 Adrien Destugues <pulkomandy@pulkomandy.tk>

TextEncoding: add Encode/Decode methods.

This replaces convert_from/to_utf8 in a way that doesn't leak memory.

Use the new method in StyledEdit.

Fixes #6252, #3065.


# fbb725bb 25-Sep-2016 Adrien Destugues <pulkomandy@pulkomandy.tk>

Add text encoding detection to the Locale Kit.

As usual we ask ICU to do the actual work. The TextEncoding constructor
is fed with a sample of the text to identify (ICU docs recommend a few
hundred bytes). The text is analyzed in various ways (bytes patterns
such as UTF-8 escaping schemes, common letter sequences from known
languages, byte order marks) and an encoding is determined.

Replace code in StyledEdit by this new implementation.

Note that ICU seems to always return some valid encoding, even with fed
with obviously non-text data. This makes StyledEdit open the files no
matter what, where it would error out before.

Fixes #9395.