MFC r324103:

__setrunelocale: Fix asprintf(3) failure not returning an error.

Revert r330897:

This was intended to be a non-functional change. It wasn't. The commit
message was thus wrong. In addition it broke arm, and merged crypto
related code.

Revert with prejudice.

This revert skips files touched in r316370 since that commit was since
MFCed. This revert also skips files that require $FreeBSD$ property
changes.

Thank you to those who helped me get out of this mess including but not
limited to gonzo, kevans, rgrimes.

Requested by: gjb (re)

Partial merge of the SPDX changes

These changes are incomplete but are making it difficult
to determine what other changes can/should be merged.

No objections from: pfg

MFC r322925:
libc: minor indent(1) cleanups.

Illumos and Schillix are adopting some of the locale code and our style(9)
sometimes matches the Solaris cstyle, so the changes are also useful as a
way to reduce diffs.

No functional change.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

locales: Enforce US-ASCII encoding (limited to 7-bit)

The US-ASCII format was getting treated identically to POSIX. It is
supposed to throw an ILSEQ errno if a value of 0x80 or greater is
encountered, so let's bring back the "ASCII" handling.

While here, change nl_codeset to return US-ASCII only when the encoding
really is "US-ASCII". Before "C" and "POSIX" encoding returned this
string, so now they return "POSIX".

Discussed with: ache
Submitted by: marino
Obtained from: DragonflyBSD

Improve collation string and locales support

Merge collation support from Illumos and DragonflyBSD.

Locales are now generated with the new localedef(1) tool from CLDR POSIX files.
The generated files are now identified as "BSD 1.0" format.

The libc now only read "BSD 1.0" locales definitions, all other version will be
set to "C"
The localedef(1) tool has been imported from Illumos and modified to use tree(3)
instead of the CDDL avl(3)
A set of tool created by edwin@ and extended by marino@ for dragonfly has been
added to be able to generate locales and the Makefiles from the vanilla CLDR
unicode databases + a universal UTF-8 charmap (by marino@)
Update the locales to unicode v27
Given our regex(3) does not support multibyte (yet) it has been forced to always
use locale C
Remove now unused colldef(1) and mklocale(1)
Finish implementing the numeric BSD extension for ctypes
The number of supported locales has grown from 175 to 250 locales. Among the new
locales: 6 Arabic locales (AE EG JO MA QA SA), Different variations of spanish
locales.
Added new 3 components locales for mn_Cyrl_MN, sr_Cyrl_RS sr_Latn_RS,
zh_Hans_CN, zh_Hant_HK and zh_Hant_TW. Some aliases has been for 2 components
version when possible.

Thanks: Garrett D'Amore (Illumos) who made sure all his work was done under
BSD license!, Edwin Groothuis (edwin@) for the work he made on tools to be able
to generate locales definition usable in freebsd sources out of vanilla CLDR
definitions, John Marino (DragonflyBSD) who first merge the Illumos work into
Dragonfly and spent hours tracking down bugs.

Fix an issue where the locale and rune locale could become out of sync,
causing mb* functions (and similar) to be called with the wrong data
(possibly a null pointer, causing a crash).

PR: standards/188036
MFC after: 1 week

libc: Make various internal file descriptors close-on-exec.

These are obtained via fopen().

Improve style(9) compliance of function declarations.

Restore the __collate_load_error global that was accidentally removed in the
xlocale refactoring.

MFC after: 1 week

Fix a leak when setting the global character locale to "C" from something else.

Reported by: mm

Reapply 227753 (xlocale cleanup), plus some fixes so that it passes build
universe with gcc.

Approved by: dim (mentor)

Revert r231673 and r231682 for now, until we can run a full make
universe with them. Sorry for the breakage.

Pointy hat to: me and brooks

Fix a misplaced __NO_TLS locations, and change a GNUism to a C11ism for
consistency.

Approved by: brooks (mentor)

Cleanup of xlocale:

- Address performance regressions encountered by das@ by caching per-thread
data in TLS where available.
- Add a __NO_TLS flag to cdefs.h to indicate where not available.
- Reorganise the xlocale.h definitions into xlocale/*.h so that they can be
included from multiple places.
- Export the POSIX2008 subset of xlocale when POSIX2008 says it should be
exported, independently of whether xlocale.h is included.
- Fix the bug where programs using ctype functions always assumed ASCII unless
recompiled.
- Fix some style(9) violations.

Reviewed by: brooks (mentor)
Approved by: dim (mentor)

Implement xlocale APIs from Darwin, mainly for use by libc++. This adds a
load of _l suffixed versions of various standard library functions that use
the global locale, making them take an explicit locale parameter. Also
adds support for per-thread locales. This work was funded by the FreeBSD
Foundation.

Please test any code you have that uses the C standard locale functions!

Reviewed by: das (gdtoa changes)
Approved by: dim (mentor)

Fix longstanding mb/wc functions segfault if error occurse
inside _<encoding>_init().
Currently _EUC_init() only was affected.

Better fix for longstanding segfault. Don't touch current locale at all
on unknown encoding. Previous fix resets it to POSIX.

1) Add (void) cast to _none_init() (while I am here)
2) Fix longstanding segfault in mb/wc code when unknown encoding is specified
in the locale file (mb/wc functions becomes NULL in that case).

Introduce new encoding: "ASCII"
It differs from default C/POSIX "NONE" mainly by stricter 8bit check
for mb*towc*/wc*tomb* family, returning EILSEQ

The problem is: currently our single byte ctype(3) functions are broken
for wide characters locales in the argument range >= 0x80 - they may
return false positives.

Example 1: for UTF-8 locale we currently have:
iswspace(0xA0)==1 and isspace(0xA0)==1
(because iswspace() and isspace() are the same code)
but must have
iswspace(0xA0)==1 and isspace(0xA0)==0
(because there is no such character and all others in the range
0x80..0xff for the UTF-8 locale, it keeps ASCII only in the single byte
range because our internal wchar_t representation for UTF-8 is UCS-4).

Example 2: for all wide character locales isalpha(arg) when arg > 0xFF may
return false positives (must be 0).
(because iswalpha() and isalpha() are the same code)

This change address this issue separating single byte and wide ctype
and also fix iswascii() (currently iswascii() is broken for
arguments > 0xFF).
This change is 100% binary compatible with old binaries.

Reviewied by: i18n@

Per Regents of the University of Calfornia letter, remove advertising
clause.

# If I've done so improperly on a file, please let me know.

. Static'ize functions exported via function reference variables only.
. Replace inclusion of sys/param.h to sys/cdefs.h and sys/types.h where
appropriate.
. move _*_init() prototypes to mblocal.h, and remove these prototypes
from .c files
. use _none_init() in __setrunelocale() instead of duplicating code
. move __mb* variables from table.c to none.c allowing us to not to
export _none_*() externs, and appropriately remove them from mblocal.h

Ok'ed by: tjr

Remove setrunelocale()

Remove the obsolete <rune.h> interface.

Remove support for the obsolete UTF2 encoding.

Change "deprecated" in link-time warnings about various rune functions
to "obsolete".

Implement the GNU extensions of mbsnrtowcs() and wcsnrtombs(). These are
convenient when the source string isn't null-terminated.

Implement the other conversion functions (mbstowcs(), mbsrtowcs(), wcstombs(),
wcsrtombs()) in terms of these new functions.

Prefix the names of members of _RuneLocale and its sub-structures
with ``__'' to avoid polluting the namespace. This doesn't change the
documented rune interface at all, but breaks applications that accessed
_RuneLocale directly.

Allow encoding modules to override the default implementations of
mbsrtowcs() and wcsrtombs(). Provide a fast implementation for the
trivial "NONE" encoding.

Move prototypes of various encoding-related functions into a new header
file to avoid extern'ing them all over the place.

Allow partial multibyte characters to accumulate in conversion state
objects passed to mbrtowc(), mbsrtowcs(), and mbrlen(), as required
by C99.

Remove support for emulating mbrtowc() and wcrtomb() in terms of the
old rune interface now that it is no longer needed.

Set __mbrtowc and __wcrtomb correctly when changing to the C/POSIX locale.
Save __mbrtowc and __wcrtomb and restore them when changing back to
the cached locale.

Reported by: perky

Add gb2312 encoding.

Allow mbrtowc() and wcrtomb() to be implemented directly, instead of
as wrappers around the deprecated 4.4BSD rune functions. This paves the
way for state-dependent encodings, which the rune API does not support.
- Add __emulated_sgetrune() and __emulated_sputrune(), which are
implementations of sgetrune() and sputrune() in terms of
mbrtowc() and wcrtomb().
- Rename the old rune-wrapper mbrtowc() and wcrtomb() functions to
__emulated_mbrtowc() and __emulated_wcrtomb().
- Add __mbrtowc and __wcrtomb function pointers, which point to the
current locale's conversion functions, or the __emulated versions.
- Implement mbrtowc() and wcrtomb() as calls to these function pointers.
- Make the "NONE" encoding implement mbrtowc() and wcrtomb() directly.

All of this emulation mess will be removed, together with rune support,
in FreeBSD 6.

Add support for gb18030 encoding

PR: 51729
Submitted by: Kang Liu <liukang@bjpu.edu.cn>

Add const to __setrunelocale prototype

Reorganize wrapper around setrunelocale() to mark it as deprecated
in FreeBSD 6

Reduce code duplication by separating _PathLocle detection code into
internal helper function.

Add GBK encoding

PR: 51504
Submitted by: Statue <statue@freebsd.sinica.edu.tw>

Add a UTF-8 encoding method, which will eventually replace the antique
"UTF2" method. Although UTF-8 and the old UTF2 encoding are compatible
for 16-bit characters, the new UTF-8 implementation is much more strict
about rejecting malformed input and also handles the full 31 bit range
of characters.

Add safeguards to never use errno == 0 as setrunelocale() error return code

Rewrite locale loading procedures, so any load failure will not affect
currently cached data. It allows a number of nice things, like: removing
fallback code from single locale loading, remove memory leak when LC_CTYPE
data loaded again and again, efficient cache use, not only for
setlocale(locale1); setlocale(locale1), but for setlocale(locale1);
setlocale("C"); setlocale(locale1) too (i.e. data file loaded only once).

Reset __mb_cur_max to 1 when "C" or "POSIX" locales loaded after multibyte one

Catch empty encoding name too

Return errno provided by fopen, not always ENOENT.
Return EFTYPE instead of EINVAL for wrong locale file format.
Whitespaces.

Check encoding for ".", ".." and / inside

Return EINVAL for NULL or too long encoding, not EFAULT

Return ENAMETOOLONG for long PATH_LOCALE, not EFAULT

Fix the style of the SCM ID's.
I believe have made all of libc .c's as consistent as possible.

Remove __P() usage.

Megre XPG4 code into libc

I have added the support for BIG5 encoding into libc/libxpg4/mklocale.
the diff is attached below. This is done on the 3.0 source-tree.
I have test this on 2.2-stable before, but I don't have a 3.0 machine
right now.

This patch is mainly to make libc support BIG5 encoding, thus add
zh_TW.BIG5 locale to 3.0.

Submitted by: Chen Hsiung Chan <frankch@waru.life.nthu.edu.tw>

Add #ifndef __NETBSD_SYSCALLS around calls to issetugid() which
do not exist in NetBSD 1.3.

Move MSKanji under XPG4 define

Submitted by: Sin'ichiro MIYATANI / Phase One, Inc <siu@phaseone.co.jp>
Basic support for the Shift JIS encoding of japanese.
(and one tiny typo fixed in a comment)

Restore PATH_LOCALE functionality using issetugid() call now

Use symbolic constants instead of hardcoded digits
Add range check for setrunelocale since it can be called
directly.
Remove _startup_setlocale compatibility function

Should go into 2.2

Update the comment why range checking not needed

Fix setrunelocale fail if called directly without prior setlocale
call

Should go in 2.2

Add comment that range checking is already done at upper level
Kill snprintf left in collate.c from previous backout

Should go in 2.2

Back out rev 1.5: the overflow condition is already handled elsewhere.

Fix yet another buffer overflow. :-(

Vulnerable: all programs that use setlocale(LC_COLLATE),
setlocale(LC_CTYPE), or setlocale(LC_ALL). The only setuid/setgid
binary i've found for this is w(1).

Should go into 2.2.

PATH_LOCALE: use this non-standard env variable first time only, i.e.
strdup() it to prevent unsetenv() or setenv() effects. Check its length to
not allow user to overflow internal locale buffer. Move PATH_LOCALE
handling code into one place.

POSIX: make better stub for LC_MONETARY & LC_NUMERIC, now it check
locale directory existance instead of refusing all non-C non-POSIX
locales. POSIX treats empty locale env variable as unset variable
while our old code treats it as "C" locale, fix it. Implement previous locale
restoring, if locale setting fails. Old code assumes success if some
of LC_ALL subset is successed even other fails, POSIX treats it as
failure with previous locale restoring, fix it.

Remove unneccessary length checking in currentlocale()

Move more stuff out to XPG4
Handle negative chars inside runetype/tolower/toupper

Treat empty encoding as "C" encoding

Migrate from XPG4 to XPG3 (libxpg4 will be added soon)
Remove big part of my startup_setlocale hack.
Add missing manpage links.