STXTTranslator: Fix use-after-free of encoding name.

The name retrieved from GetName() on BTextEncoding is owned by that
instance which goes out of scope when identify_text() returns. A const
char pointer to it can therefore not be used as an output variable. Use
a BString instead which ensures an appropriate reference is acquired.

Change-Id: Ib87fc9879d29e927bb41d1a9165ca7599b74a3bf
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3115
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

StyledText translator: use a larger buffer for encoding detection

It may be the case that a file is largely ASCII with few UTF-8 or
similar characters (typically, english text mentionning names in other
languages). It turns out 2 kilobytes of text is not enough for a safe
guess in that case.

Fixes #14674

Change-Id: Id894bebb821c22a17b776a71822d011abc795dea
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2628
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

STXTTranslator: Fix dead initialization

* Fix dead initialization at line 596. Pointed out
by clang static analyzer.
* As a result, 'BMallocIO encodingIO' at line 524
becomes unused, so delete it and header.

Change-Id: Ia62184fe0682a2eedeee42a513faccabdd3d1b59
Reviewed-on: https://review.haiku-os.org/c/1079
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

STXTTranslator: Fix truncated "be:encoding" attribute write.

Same issue as the mail_util fix.

Also while I'm here, alter behavior so that the Be-style attribute
gets overwritten with the Haiku-style one, instead of getting left
as is.

Should potentially help with #14674.

StyledText: fix reading of utf-8 documents

BNode::ReadAttrString does not check for the attribute to actually be a
string. It will return B_OK (and useless data) if called on a
B_INT32_TYPE attribute. Check the attribute type before reading.

Also remove various debug ouput leftovers.

May fix #12976.

BTextEncoding: fix naming and move to BPrivate namespace.

TextEncoding: add Encode/Decode methods.

This replaces convert_from/to_utf8 in a way that doesn't leak memory.

Use the new method in StyledEdit.

Fixes #6252, #3065.

Add text encoding detection to the Locale Kit.

As usual we ask ICU to do the actual work. The TextEncoding constructor
is fed with a sample of the text to identify (ICU docs recommend a few
hundred bytes). The text is analyzed in various ways (bytes patterns
such as UTF-8 escaping schemes, common letter sequences from known
languages, byte order marks) and an encoding is determined.

Replace code in StyledEdit by this new implementation.

Note that ICU seems to always return some valid encoding, even with fed
with obviously non-text data. This makes StyledEdit open the files no
matter what, where it would error out before.

Fixes #9395.

Harmonized translator titles in DataTranslations.

Added translator add-ons.

More catalog-related cleanup.

* rename B_TRANSLATE_CONTEXT to B_TRANSLATION_CONTEXT and
B_TRANSLATE_WITH_CONTEXT to B_TRANSLATE_CONTEXT, squashing a TODO
* adjust all uses of both macros in Haiku's source tree
* use correct header guard for collecting/Catalog.h

The renamed macros require adjustments to all external applications
using catalogs.

Replace inappropriate usage of strncpy by strlcpy

Prevent string overflow by replacing usage of strcpy by strncpy

CID 8951, CID 10733, CID 10734, CID 10735, CID 10736, CID 10737, CID 10738,
CID 10739

Applied following patches proposed by Jorma Karvonen:
#7135 #7140 #7141 #7145 #7186 #7188 #7191
#7136 #7187 #7184 #7185 #7192 #7138 #7139

with some changes and exclusions:
- all attempts to localize "fprintf(stderr,..." and "printf(..."
replaced by _untranslated_ "syslog(LOG_ERR ...";
- following *Translator.rdef files, that were not added in mentioned patches
were additionally created:
SGI, TIFF, RAW, RTF, PPM, WebP, EXR, STXT, WonderBrush, GIF, TGA;
- some small fixes for consistent catalogs building.

Thank you, Jorma! Please check. ;-)



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40357 a95241bf-73f2-0310-859d-f6bbb57e9c96

Made local variables actually local.
Both JPEGTranslator and JPEG200Translator were "exporting" gSettings symbol,
which when loaded within a app (like Paladin) exporting the same global symbol
was leading to a symbol resolution error. See #7114 for details.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40245 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch by Humdiner:
Changed strings in add-ons to sentence case. This is case-add-ons.diff
from #5169.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35046 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fixing all headers/namespaces/name lookup problems that hinder a GCC 4.3.2
build. I sure hope that this doesn't break the build for anyone else.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@28992 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added a more generic subtype for the source code files for automatic
MIME type guessing.
* Now, if the specific MIME type exists, it will be chosen, if it doesn't,
the generic MIME type is chosen. If that does not exist either, the most
generic is chosen (a.k.a. "text/plain").


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@24187 a95241bf-73f2-0310-859d-f6bbb57e9c96

* remove superfluous tab


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22898 a95241bf-73f2-0310-859d-f6bbb57e9c96

* patch by vzsolt which really fixes the problem with zero byte buffers


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22895 a95241bf-73f2-0310-859d-f6bbb57e9c96

* this should cover the case where the buffer is one byte long but is _not_
a text file


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22893 a95241bf-73f2-0310-859d-f6bbb57e9c96

* fixed a bug researched by Varadi Zsolt Gyula, if the buffer is one byte long
or empty, assume text anyways, fixes Styled Edit opening empty files (#1610)


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22892 a95241bf-73f2-0310-859d-f6bbb57e9c96

I accidently prevented the STXTTranslator from adding the styles section to the styled text output.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@19932 a95241bf-73f2-0310-859d-f6bbb57e9c96

Minor fixes:
* the "state" variable was reset for each loop cycle.
* since the inner loop tested for "bytes" instead of "bytesLeft", convert_to_utf8()
was called twice for each chunk read.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@19846 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added the file_ascmagic() stuff from our text sniffer add-on (originally taken from
BSD's file utility) to identify text files.
* The translator now automatically decodes files based "be:encoding" (which can now
also be a string type attribute), as well as whatever file_ascmagic() recognizes.
IOW we can now finally read all text files we recognize as text files.
* Added an ioExtension to force a specific encoding for Translate() (called "be:encoding"
as well).
* Enlarged the read buffer to 32768 bytes (instead of just 2048).
* Now fails when reading/writing files >4GB in STXT format (which doesn't support those)
* Translate() now writes the "be:encoding" attribute in case it detected the encoding
itself.
* Cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@19842 a95241bf-73f2-0310-859d-f6bbb57e9c96

Bug fix for my last commit: all of a sudden, a text would need more than one
bad character to be accepted.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@18708 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Extended the text recognition to UTF-8 characters - if a text contains too many
bad characters, it will be rejected, at least after bug #675 has been fixed.
* In the future, we should use the same mechanism as the TextSnifferAddon of the
registrar; it would also be nice to use additional charset information (at least
differentiating between latin-1 and UTF-8).
* Minor cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@18703 a95241bf-73f2-0310-859d-f6bbb57e9c96

removed the redundant "translator" from the folder names, fixed some warnings in the JPEG2000Translator

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@14449 a95241bf-73f2-0310-859d-f6bbb57e9c96

Harmonized translator titles in DataTranslations.

Added translator add-ons.

More catalog-related cleanup.

* rename B_TRANSLATE_CONTEXT to B_TRANSLATION_CONTEXT and
B_TRANSLATE_WITH_CONTEXT to B_TRANSLATE_CONTEXT, squashing a TODO
* adjust all uses of both macros in Haiku's source tree
* use correct header guard for collecting/Catalog.h

The renamed macros require adjustments to all external applications
using catalogs.

Replace inappropriate usage of strncpy by strlcpy

Prevent string overflow by replacing usage of strcpy by strncpy

CID 8951, CID 10733, CID 10734, CID 10735, CID 10736, CID 10737, CID 10738,
CID 10739

Applied following patches proposed by Jorma Karvonen:
#7135 #7140 #7141 #7145 #7186 #7188 #7191
#7136 #7187 #7184 #7185 #7192 #7138 #7139

with some changes and exclusions:
- all attempts to localize "fprintf(stderr,..." and "printf(..."
replaced by _untranslated_ "syslog(LOG_ERR ...";
- following *Translator.rdef files, that were not added in mentioned patches
were additionally created:
SGI, TIFF, RAW, RTF, PPM, WebP, EXR, STXT, WonderBrush, GIF, TGA;
- some small fixes for consistent catalogs building.

Thank you, Jorma! Please check. ;-)



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40357 a95241bf-73f2-0310-859d-f6bbb57e9c96

Made local variables actually local.
Both JPEGTranslator and JPEG200Translator were "exporting" gSettings symbol,
which when loaded within a app (like Paladin) exporting the same global symbol
was leading to a symbol resolution error. See #7114 for details.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40245 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch by Humdiner:
Changed strings in add-ons to sentence case. This is case-add-ons.diff
from #5169.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35046 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fixing all headers/namespaces/name lookup problems that hinder a GCC 4.3.2
build. I sure hope that this doesn't break the build for anyone else.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@28992 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added a more generic subtype for the source code files for automatic
MIME type guessing.
* Now, if the specific MIME type exists, it will be chosen, if it doesn't,
the generic MIME type is chosen. If that does not exist either, the most
generic is chosen (a.k.a. "text/plain").


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@24187 a95241bf-73f2-0310-859d-f6bbb57e9c96

* remove superfluous tab


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22898 a95241bf-73f2-0310-859d-f6bbb57e9c96

* patch by vzsolt which really fixes the problem with zero byte buffers


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22895 a95241bf-73f2-0310-859d-f6bbb57e9c96

* this should cover the case where the buffer is one byte long but is _not_
a text file


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22893 a95241bf-73f2-0310-859d-f6bbb57e9c96

* fixed a bug researched by Varadi Zsolt Gyula, if the buffer is one byte long
or empty, assume text anyways, fixes Styled Edit opening empty files (#1610)


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@22892 a95241bf-73f2-0310-859d-f6bbb57e9c96

I accidently prevented the STXTTranslator from adding the styles section to the styled text output.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@19932 a95241bf-73f2-0310-859d-f6bbb57e9c96

Minor fixes:
* the "state" variable was reset for each loop cycle.
* since the inner loop tested for "bytes" instead of "bytesLeft", convert_to_utf8()
was called twice for each chunk read.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@19846 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added the file_ascmagic() stuff from our text sniffer add-on (originally taken from
BSD's file utility) to identify text files.
* The translator now automatically decodes files based "be:encoding" (which can now
also be a string type attribute), as well as whatever file_ascmagic() recognizes.
IOW we can now finally read all text files we recognize as text files.
* Added an ioExtension to force a specific encoding for Translate() (called "be:encoding"
as well).
* Enlarged the read buffer to 32768 bytes (instead of just 2048).
* Now fails when reading/writing files >4GB in STXT format (which doesn't support those)
* Translate() now writes the "be:encoding" attribute in case it detected the encoding
itself.
* Cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@19842 a95241bf-73f2-0310-859d-f6bbb57e9c96

Bug fix for my last commit: all of a sudden, a text would need more than one
bad character to be accepted.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@18708 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Extended the text recognition to UTF-8 characters - if a text contains too many
bad characters, it will be rejected, at least after bug #675 has been fixed.
* In the future, we should use the same mechanism as the TextSnifferAddon of the
registrar; it would also be nice to use additional charset information (at least
differentiating between latin-1 and UTF-8).
* Minor cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@18703 a95241bf-73f2-0310-859d-f6bbb57e9c96

removed the redundant "translator" from the folder names, fixed some warnings in the JPEG2000Translator

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@14449 a95241bf-73f2-0310-859d-f6bbb57e9c96