Replace each occurence of an invalide character. Before all invalid
characters where omitted and the substitute character was append at the end
of the input text.
Added comment how the continuation of incomplete multibyte sequences
could be solved.
Please review.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35875 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Removed a superfluous allocation (even without nothrow!), and a superfluous
const_cast.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@28645 a95241bf-73f2-0310-859d-f6bbb57e9c96

delete -> delete[]. CID 121

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27521 a95241bf-73f2-0310-859d-f6bbb57e9c96

While investigating why our UTF-16 files are broken (with a smaller output buffer),
I noticed that "state" was never set, but as expected this doesn't help at all:
we just can't use iconv() this way, I'll open a bug for this.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@19844 a95241bf-73f2-0310-859d-f6bbb57e9c96

Unlike what shatty wrote in his commit messages, convert_from_utf8() does return
B_OK here when called with an empty string.
Since this is the natural thing to expect anyway, I changed our version to return
a sane value here as well - if this causes an error within our StyledEdit, as
shatty writes, we should just fix our StyledEdit.
Cleanup to match our style guides a bit better, added license.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@16328 a95241bf-73f2-0310-859d-f6bbb57e9c96

as on R5, return an error when *srcLen == 0 and set dstLen to 0
fix bug #120


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@16294 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fixes that make Haiku build with gcc 4. Mainly out of the following
categories:
* Missing includes (like <stdlib.h> and <string.h>).
* Linking against $(TARGET_LIBSTDC++) instead of libstdc++.r4.so.
* Local variables shadowing parameters.
* Default parameters in function definitions (as opposed to function
declarations).
* All C++ stuff (nothrow, map, set, vector, min, max,...) must be imported
explicitly from the std:: namespace now.
* "new (sometype)[...]" must read "new sometype[...]", even if sometype is
something like "const char *".
* __FUNCTION__ is no longer a string literal (but a string expression), i.e.
'printf(__FUNCTION__ ": ...\n")' is invalid code.
* A type cast results in a non-lvalue. E.g. "(char *)buffer += bytes"
is an invalid expression.
* "friend class SomeClass" only works when SomeClass is known before.
Otherwise the an inner class with that name is considered as friend.
gcc 4 is much pickier about scopes.
* gcc 4 is generally stricter with respect to type conversions in C.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@14878 a95241bf-73f2-0310-859d-f6bbb57e9c96

use DEBUG_CONV to print some output not the global DEBUG

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@14797 a95241bf-73f2-0310-859d-f6bbb57e9c96

gracefully tolerate coder passing NULL for state pointer


git-svn-id: file:///srv/svn/repos/haiku/trunk/current@9471 a95241bf-73f2-0310-859d-f6bbb57e9c96

fix stupid looping bug on large files - thanks kuye, Tim


git-svn-id: file:///srv/svn/repos/haiku/trunk/current@5823 a95241bf-73f2-0310-859d-f6bbb57e9c96

implemented substitution behavior by using non-posix iconvctl. also implemented bad input bytes tolerance (via ignore)


git-svn-id: file:///srv/svn/repos/haiku/trunk/current@5822 a95241bf-73f2-0310-859d-f6bbb57e9c96

MOVED:
---------------------------------------------------------------------------
RCS file: /cvsroot/open-beos/current/src/kits/support/utf8_conversions.cpp,v
Working file: utf8_conversions.cpp
head: 1.8
branch:
locks: strict
access list:
symbolic names:
keyword substitution: kv
total revisions: 8; selected revisions: 8
description:
----------------------------
revision 1.8
date: 2003/08/13 05:38:07; author: shatty; state: Exp; lines: +24 -4
refine the error handling behavior. note: we depart from the bebook specification for returning B_ERROR when no characters are converted. we do this in exactly one situation: when there are no bytes in the input. this behavior is the behavior given by the R5 libs themselves. not having this behavior caused an error in our stylededit as well. stylededit has been fixed to not exercise this functionality. also added in the two most popular chinese encodings for my own evil purposes. GB18030 support is required to legally sell an operating system in mainland china as well. GB18030 support encompasses GBK and GB2312, additionally.
----------------------------
revision 1.7
date: 2003/08/02 09:01:13; author: shatty; state: Exp; lines: +1 -1
no const for now
----------------------------
revision 1.6
date: 2003/07/31 07:34:30; author: shatty; state: Exp; lines: +0 -1
dunno when beos changes the state, but it seems to leave at 0 for a while so I am going to have it be consistent that way
----------------------------
revision 1.5
date: 2003/07/31 07:18:15; author: shatty; state: Exp; lines: +0 -3
remove debugging print things
----------------------------
revision 1.4
date: 2003/07/31 07:17:30; author: shatty; state: Exp; lines: +35 -30
new and better working implementations for conversion functions, including better abstraction
----------------------------
revision 1.3
date: 2003/07/31 04:57:37; author: shatty; state: Exp; lines: +6 -2
added iconv_close to free resources and made a new input_buffer_t typedef to ease switching iconv implementations
----------------------------
revision 1.2
date: 2003/07/31 00:04:53; author: shatty; state: Exp; lines: +43 -4
written to use iconv.h
----------------------------
revision 1.1
date: 2003/07/26 21:28:02; author: shatty; state: Exp;
utf8 conversions file with stub implementations of convert_to_utf8 and convert_from_utf8


git-svn-id: file:///srv/svn/repos/haiku/trunk/current@5810 a95241bf-73f2-0310-859d-f6bbb57e9c96

Replace each occurence of an invalide character. Before all invalid
characters where omitted and the substitute character was append at the end
of the input text.
Added comment how the continuation of incomplete multibyte sequences
could be solved.
Please review.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35875 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Removed a superfluous allocation (even without nothrow!), and a superfluous
const_cast.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@28645 a95241bf-73f2-0310-859d-f6bbb57e9c96

delete -> delete[]. CID 121

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27521 a95241bf-73f2-0310-859d-f6bbb57e9c96

While investigating why our UTF-16 files are broken (with a smaller output buffer),
I noticed that "state" was never set, but as expected this doesn't help at all:
we just can't use iconv() this way, I'll open a bug for this.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@19844 a95241bf-73f2-0310-859d-f6bbb57e9c96

Unlike what shatty wrote in his commit messages, convert_from_utf8() does return
B_OK here when called with an empty string.
Since this is the natural thing to expect anyway, I changed our version to return
a sane value here as well - if this causes an error within our StyledEdit, as
shatty writes, we should just fix our StyledEdit.
Cleanup to match our style guides a bit better, added license.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@16328 a95241bf-73f2-0310-859d-f6bbb57e9c96

as on R5, return an error when *srcLen == 0 and set dstLen to 0
fix bug #120


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@16294 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fixes that make Haiku build with gcc 4. Mainly out of the following
categories:
* Missing includes (like <stdlib.h> and <string.h>).
* Linking against $(TARGET_LIBSTDC++) instead of libstdc++.r4.so.
* Local variables shadowing parameters.
* Default parameters in function definitions (as opposed to function
declarations).
* All C++ stuff (nothrow, map, set, vector, min, max,...) must be imported
explicitly from the std:: namespace now.
* "new (sometype)[...]" must read "new sometype[...]", even if sometype is
something like "const char *".
* __FUNCTION__ is no longer a string literal (but a string expression), i.e.
'printf(__FUNCTION__ ": ...\n")' is invalid code.
* A type cast results in a non-lvalue. E.g. "(char *)buffer += bytes"
is an invalid expression.
* "friend class SomeClass" only works when SomeClass is known before.
Otherwise the an inner class with that name is considered as friend.
gcc 4 is much pickier about scopes.
* gcc 4 is generally stricter with respect to type conversions in C.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@14878 a95241bf-73f2-0310-859d-f6bbb57e9c96

use DEBUG_CONV to print some output not the global DEBUG

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@14797 a95241bf-73f2-0310-859d-f6bbb57e9c96

gracefully tolerate coder passing NULL for state pointer


git-svn-id: file:///srv/svn/repos/haiku/trunk/current@9471 a95241bf-73f2-0310-859d-f6bbb57e9c96

fix stupid looping bug on large files - thanks kuye, Tim


git-svn-id: file:///srv/svn/repos/haiku/trunk/current@5823 a95241bf-73f2-0310-859d-f6bbb57e9c96

implemented substitution behavior by using non-posix iconvctl. also implemented bad input bytes tolerance (via ignore)


git-svn-id: file:///srv/svn/repos/haiku/trunk/current@5822 a95241bf-73f2-0310-859d-f6bbb57e9c96

MOVED:
---------------------------------------------------------------------------
RCS file: /cvsroot/open-beos/current/src/kits/support/utf8_conversions.cpp,v
Working file: utf8_conversions.cpp
head: 1.8
branch:
locks: strict
access list:
symbolic names:
keyword substitution: kv
total revisions: 8; selected revisions: 8
description:
----------------------------
revision 1.8
date: 2003/08/13 05:38:07; author: shatty; state: Exp; lines: +24 -4
refine the error handling behavior. note: we depart from the bebook specification for returning B_ERROR when no characters are converted. we do this in exactly one situation: when there are no bytes in the input. this behavior is the behavior given by the R5 libs themselves. not having this behavior caused an error in our stylededit as well. stylededit has been fixed to not exercise this functionality. also added in the two most popular chinese encodings for my own evil purposes. GB18030 support is required to legally sell an operating system in mainland china as well. GB18030 support encompasses GBK and GB2312, additionally.
----------------------------
revision 1.7
date: 2003/08/02 09:01:13; author: shatty; state: Exp; lines: +1 -1
no const for now
----------------------------
revision 1.6
date: 2003/07/31 07:34:30; author: shatty; state: Exp; lines: +0 -1
dunno when beos changes the state, but it seems to leave at 0 for a while so I am going to have it be consistent that way
----------------------------
revision 1.5
date: 2003/07/31 07:18:15; author: shatty; state: Exp; lines: +0 -3
remove debugging print things
----------------------------
revision 1.4
date: 2003/07/31 07:17:30; author: shatty; state: Exp; lines: +35 -30
new and better working implementations for conversion functions, including better abstraction
----------------------------
revision 1.3
date: 2003/07/31 04:57:37; author: shatty; state: Exp; lines: +6 -2
added iconv_close to free resources and made a new input_buffer_t typedef to ease switching iconv implementations
----------------------------
revision 1.2
date: 2003/07/31 00:04:53; author: shatty; state: Exp; lines: +43 -4
written to use iconv.h
----------------------------
revision 1.1
date: 2003/07/26 21:28:02; author: shatty; state: Exp;
utf8 conversions file with stub implementations of convert_to_utf8 and convert_from_utf8


git-svn-id: file:///srv/svn/repos/haiku/trunk/current@5810 a95241bf-73f2-0310-859d-f6bbb57e9c96