MFC: r286490,r286491,r287125

Per rfc3629 value greater than 0x10ffff should be rejected (r286490,r286491)

Make UTF-8 parsing and generation more strict. (r287125 by ed)

- in mbrtowc() we need to disallow codepoints above 0x10ffff.
- In wcrtomb() we need to disallow codepoints between 0xd800 and 0xdfff.

MFC r268272:

minor perf enhancement for UTF-8

Reduce some duplicate code.

Reference:
https://www.illumos.org/issues/628

Obtained from: Illumos

MFC r265095, r265167;

citrus: Avoid invalid code points.

The UTF-8 decoder should not accept byte sequences which decode to
unicode code positions U+D800 to U+DFFF (UTF-16 surrogates).[1]

Contrary to the original OpenBSD patch, we do pass U+FFFE and U+FFFF,
both values are valid "non-characters" [2] and must be mapped through
UTFs.

[1] http://www.cl.cam.ac.uk/~mgk25/unicode.html#utf-8
[2] http://www.unicode.org/faq/private_use.html

Reported by: Stefan Sperling [1]
Thanks to: jilles [2]
Obtained from: OpenBSD

MFC: r286490,r286491,r287125

Per rfc3629 value greater than 0x10ffff should be rejected (r286490,r286491)

Make UTF-8 parsing and generation more strict. (r287125 by ed)

- in mbrtowc() we need to disallow codepoints above 0x10ffff.
- In wcrtomb() we need to disallow codepoints between 0xd800 and 0xdfff.

MFC r268272:

minor perf enhancement for UTF-8

Reduce some duplicate code.

Reference:
https://www.illumos.org/issues/628

Obtained from: Illumos

MFC r265095, r265167;

citrus: Avoid invalid code points.

The UTF-8 decoder should not accept byte sequences which decode to
unicode code positions U+D800 to U+DFFF (UTF-16 surrogates).[1]

Contrary to the original OpenBSD patch, we do pass U+FFFE and U+FFFF,
both values are valid "non-characters" [2] and must be mapped through
UTFs.

[1] http://www.cl.cam.ac.uk/~mgk25/unicode.html#utf-8
[2] http://www.unicode.org/faq/private_use.html

Reported by: Stefan Sperling [1]
Thanks to: jilles [2]
Obtained from: OpenBSD