Revert r330897:

This was intended to be a non-functional change. It wasn't. The commit
message was thus wrong. In addition it broke arm, and merged crypto
related code.

Revert with prejudice.

This revert skips files touched in r316370 since that commit was since
MFCed. This revert also skips files that require $FreeBSD$ property
changes.

Thank you to those who helped me get out of this mess including but not
limited to gonzo, kevans, rgrimes.

Requested by: gjb (re)

Partial merge of the SPDX changes

These changes are incomplete but are making it difficult
to determine what other changes can/should be merged.

No objections from: pfg

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Improve collation string and locales support

Merge collation support from Illumos and DragonflyBSD.

Locales are now generated with the new localedef(1) tool from CLDR POSIX files.
The generated files are now identified as "BSD 1.0" format.

The libc now only read "BSD 1.0" locales definitions, all other version will be
set to "C"
The localedef(1) tool has been imported from Illumos and modified to use tree(3)
instead of the CDDL avl(3)
A set of tool created by edwin@ and extended by marino@ for dragonfly has been
added to be able to generate locales and the Makefiles from the vanilla CLDR
unicode databases + a universal UTF-8 charmap (by marino@)
Update the locales to unicode v27
Given our regex(3) does not support multibyte (yet) it has been forced to always
use locale C
Remove now unused colldef(1) and mklocale(1)
Finish implementing the numeric BSD extension for ctypes
The number of supported locales has grown from 175 to 250 locales. Among the new
locales: 6 Arabic locales (AE EG JO MA QA SA), Different variations of spanish
locales.
Added new 3 components locales for mn_Cyrl_MN, sr_Cyrl_RS sr_Latn_RS,
zh_Hans_CN, zh_Hant_HK and zh_Hant_TW. Some aliases has been for 2 components
version when possible.

Thanks: Garrett D'Amore (Illumos) who made sure all his work was done under
BSD license!, Edwin Groothuis (edwin@) for the work he made on tools to be able
to generate locales definition usable in freebsd sources out of vanilla CLDR
definitions, John Marino (DragonflyBSD) who first merge the Illumos work into
Dragonfly and spent hours tracking down bugs.

Correctly expose xlocale functions if people include the headers in the wrong
order (as some ports apparently do).

Approved by: dim (mentor)

Reapply 227753 (xlocale cleanup), plus some fixes so that it passes build
universe with gcc.

Approved by: dim (mentor)

Revert r231673 and r231682 for now, until we can run a full make
universe with them. Sorry for the breakage.

Pointy hat to: me and brooks

Cleanup of xlocale:

- Address performance regressions encountered by das@ by caching per-thread
data in TLS where available.
- Add a __NO_TLS flag to cdefs.h to indicate where not available.
- Reorganise the xlocale.h definitions into xlocale/*.h so that they can be
included from multiple places.
- Export the POSIX2008 subset of xlocale when POSIX2008 says it should be
exported, independently of whether xlocale.h is included.
- Fix the bug where programs using ctype functions always assumed ASCII unless
recompiled.
- Fix some style(9) violations.

Reviewed by: brooks (mentor)
Approved by: dim (mentor)

Hide some more macros that will break C++ when compiling in C++ mode.

Approved by: dim (mentor)

Remove the Berkeley clause 3's.
Add a few $FreeBSD$

Don't prototype _tolower() and _toupper(). They're not supposed to be
functions, and there's no implementation of them in any case.

The problem is: currently our single byte ctype(3) functions are broken
for wide characters locales in the argument range >= 0x80 - they may
return false positives.

Example 1: for UTF-8 locale we currently have:
iswspace(0xA0)==1 and isspace(0xA0)==1
(because iswspace() and isspace() are the same code)
but must have
iswspace(0xA0)==1 and isspace(0xA0)==0
(because there is no such character and all others in the range
0x80..0xff for the UTF-8 locale, it keeps ASCII only in the single byte
range because our internal wchar_t representation for UTF-8 is UCS-4).

Example 2: for all wide character locales isalpha(arg) when arg > 0xFF may
return false positives (must be 0).
(because iswalpha() and isalpha() are the same code)

This change address this issue separating single byte and wide ctype
and also fix iswascii() (currently iswascii() is broken for
arguments > 0xFF).
This change is 100% binary compatible with old binaries.

Reviewied by: i18n@

Move some internal macros and inlines from ctype.h to a new file, _ctype.h,
which has been repo-copied from ctype.h. This will allow us to remove
namespace pollution from <wctype.h> and to make wcwidth() an inline function
without introducing more pollution.

Now that <runetype.h> no longer brings in namespace pollution,
bring back the inline functions for the !__BSD_VISIBLE case.

Prefix the names of members of _RuneLocale and its sub-structures
with ``__'' to avoid polluting the namespace. This doesn't change the
documented rune interface at all, but breaks applications that accessed
_RuneLocale directly.

Make isblank() visible in the C99 namespace.

PR: 63371
Submitted by: Stefan Farfeleder

Solve the <runetype.h> pollution problem by disabling inline
optimizations when a standard has been requested, except when the
inline optimizations are also specifically requested.

o Fix namespace scope issues in <ctype.h> by using the relatively new
visibility primitives.
o Implement _tolower() and _toupper() POSIX.1-2001 (XSI) macros in
<ctype.h>.
o Reduce pollution in <runetype.h> by removing typedefs and using
implementation namespaced types.
o Add a typedef in <rune.h> to compensate for <runetype.h> losing its
typedefs.

Reviewed by: bde

o Merge <machine/ansi.h> and <machine/types.h> into a new header
called <machine/_types.h>.
o <machine/ansi.h> will continue to live so it can define MD clock
macros, which are only MD because of gratuitous differences between
architectures.
o Change all headers to make use of this. This mainly involves
changing:
#ifdef _BSD_FOO_T_
typedef _BSD_FOO_T_ foo_t;
#undef _BSD_FOO_T_
#endif
to:
#ifndef _FOO_T_DECLARED
typedef __foo_t foo_t;
#define _FOO_T_DECLARED
#endif

Concept by: bde
Reviewed by: jake, obrien

Move internal defines from here to libc/locale/wcwidth.c

Properly define SWIDTH1, add autowidth (was SWIDTH1)

Add swidth definition for wchar stuff

Submitted by: clkao@clkao.org
Reviewed by: keichii
Obtained from: NetBSD
MFC after: 1 month

Breath deep and take __P out of the system include files.

# This appears to not break X11, but I'm having problems compiling the
# glide part of the server with or without this patch, so I can't tell
# for sure.

Makes __istype() an inline (and a non inlined) function to avoid C++
compile issues. std::isspace(' ') was expanding to std::(!!_maskrune...)
which would cause a C++ compile error. Making __istype() an inline
causes the expansion to be std::__istype() instead, which is valid.

Reviewed by: jkh

There is a problem in that one cannot use ctype.h at the same time as parts
of the C++ stdlib. Our ctype.h uses symbols of the form _<X> to denote the
various character classes. Our ctype.h also extends the usual ctype.h
offering by adding the "_T" (special) class. Problem is parts of the STL
also use the symbol "_T" as its parameterized type. These two uses are
incompatible.

Thus change the form of the symbols used in ctype to something that fixes
the current problem and is less likely to cause conflicts in the future.

Requested by: Tomoaki NISHIYAMA <tomoaki@biol.s.u-tokyo.ac.jp>
Ok'ed by: JKH

Back up following macros by functions: ishexnumber, isideogram, isnumber,
isphonogram, isrune, isspecial. Fix ordering.

Reviewed by: bde

Oops, fix typo in istype definition

1) Bring '!!' back since we wan't overflow even a char
2) Revive __maskrune, just return mask now
3) Express __istype via !!__maskrune
4) Use __maskrune in digittoint

Oops, wrong version of digittoint fix was commited, step back to v1.8
instead just remove '!!' to make digittoint() work and re-arrange
things a little

Fix digittoint broken long ago
Redesign to allow digittoint work for runes too
__maskrune removed, __istype become a macro, __runeflags added

Bandaid for the build-breaking reference to _BSD_RUNE_T_. I'm not sure
that this is right, but the old reference defaintely was not.

Submitted by: Sin'ichiro MIYATANI / Phase One, Inc <siu@phaseone.co.jp>
Basic support for the Shift JIS encoding of japanese.
(and one tiny typo fixed in a comment)

Fixed longstanding namespace convolution involving rune_t vs wchar_t.
If _ANSI_SOURCE or _POSIX_SOURCE is defined, then <ctype.h> had to
be included before <stddef.h> or <stdlib.h> to get rune_t declared.
Now rune_t is declared perfectly bogusly in all cases when <ctype.h>
is included.

This change breaks similar (but more convoluted) convolutions in the
stddef.h in gcc distributions. Ports of gcc should avoid using the
gcc headers.

Slightly simplify inlined functions

Fix isspecial/isphonogram, they was swapped
Remove EOF hack, now it is recognized per ANSI/POSIX
Add upper bounds check
Handle all negative chars inside locale functions

Reviewed by: ache and wollman (long ago)

Fix numerous ANSI conformance bugs and other nits.

ctype.h:
o There were no prototypes behind the macros (conformance bug).
o isascii() didn't have enough parentheses (plain bug).
o tolower() and toupper were always static inline (conformance
bug? You could undef them and take their address, but this
gave different addresses in different modules. You couldn't
undef them and declare them (correctly) again). <stdio.h>'s
treatment of putc() shows one way to handle this problem,
but it only works because the putc() macro is allowed to
reevaluate its args. I used a hack controlled by
_EXTERNALIZE_CTYPE_INLINES_ to get <ctype.h> to generate the
code (the previous hack involving _ANSI_LIBRARY_ goes away).
This has the advantage that the core of the functions is only
written down once and the disadvantage that another layer of
functions is required. The extra layer goes away if inline
functions are used, leaving only the problem of understanding
why there are functions named toupper(), __toupper and
___toupper() as well as a macro named toupper.
o Nothing seems to define _USE_CTYPE_LIBRARY_. Eliminate it
o Let the user set _USE_CTYPE_INLINE_ and _DONT_USE_CTYPE_INLINE_
for full control over inlining.
o The args for the inline functions didn't have enough
underscores (conformance bug).
o The formatting and ordering was inconsistent (style bug).
o TODO: fix conformance bugs brought by including <runetype.h>.

Move inline args out of user namespace.
Obtained from: 1.x

Remove EOF handling after Bruce explanation. This step returns
to 4.4 way to not allow EOF in ctype and now all signed chars
(including '\377' which becomes EOF) converted to (unsigned char) properly.

Handle EOF case in all macros by ANSI standard.
Cast all ints < 0 to (unsigned char) to fix common problem
with sign extention on signed char.

This commit was generated by cvs2svn to compensate for changes in r1539,
which included commits to RCS files with non-trunk default branches.

BSD 4.4 Lite Include Sources