MFC r353127:

Pre-generate Big5 charmap from CLDR data.

The one used previously was missing the characters in 0-127 range,
making various tools try to escape them in output.

PR: 235100
Reviewed by: bapt
Tested by: Ting-Wei Lan <lantw44@gmail.com>
Differential Revision: https://reviews.freebsd.org/D21794

MFC r339311, r339313:
Restore some of the ctype definitions reported in the PR from pre-CLDR
data, namely 0xE000-0xF8FF private use area, and 0xFF00-0xFFF half- and
fullwidth punctuation.

While here, update tools/tools/locale/README based on my experience
rebuilding the locale data.

PR: 225692
Reviewed by: bapt, cem (previous version)
Differential Revision: https://reviews.freebsd.org/D17471

MFC: 325359

Upgrade to Unicode 10.0.0

MFC r310201:

Update locales cldr to v30.0.3 and unicode to 9.0.0

Relnotes: yes

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Posixify the locales name for variants

For all locales with variants:
- if no ambiguity on the locale (only one variant) just use the regular name
- if ambiguity, pick one as default and append @<variant> to the others
respecting POSIX

As a result:
- All the 3 components locales added recently are renamed to the usual 2
components version for all but sr_RS.UTF-8
- Set sr_RS.UTF-8 to the cyrillic variant
- Add sr_RS.UTF-8@latin
- Remove the symlinks aliases they were created to represent the 2 components
version as aliasas and are now useless
- Update the OptionalObsoleteFiles.inc and ObsoleteFiles.inc to reflect those
changes

Discussed with: ache@
Approved by: re@ (gjb)

Fix generation of locales with multiple variants

Serbian locales have triple components to represent the 2 variations of the
locale: Latin and Cyrillic. Previously the tools generatic the locale were
appending both definitions instead of differentiating them.

Reported by: ache
Approved by: re (gjb)

- Prefer to use %d over %e where the day of the month should be zero
filled.
- Since %e means the day of the month as well, regard %e as same as %d
in md_order.

Reported by: vangyzen
Approved by: re (gjb)

Add more hacks to correct CLDR data

For Russian:
- Convert AM/PM which are badly formatted in CLDR to replace it by the proper
cyrillic
- Add a dependency on Text::Iconv so non unicode get the proper encoding for
AM/PM
- fix the date format having 'r.,' and convert it to 'r.' (also fixed in Bulgarian)

For All:
- Use complete Day of Week instead of the abbreviated one

Reported by: ache

Add a hack to readd the day of weeks in default date formats

tools: minor spelling fix in locales template.

No functional change.

Remove leftover from Big5HKSCS removal

Do not forget to set the destination directory for the new collation

Readd the zh_hant_HK.UTF-8 removed by accident

Only Big5HKSCS as been removed.

Really stop converting map from UTF-8 to GB2312

Replace generated maps with maps extracted from CLDR for GB3212 and eucCN

Rework collation generation:

When building collation database for non unicode encodings use the proper
unicode mapping (this fixes collation not working properly for those encodings)

For locales where new characters are added but only for unicode, stop trying to
map the new characters, directly extract from CLDR the collation files for the
said encoding

Stop trying to generate encoding map from unicode version for GB2312 and encCN
It was not reliable. Instead use the map provide by the CLDR project

Reported by: ache

Restore a day of the week in date(1) output for Japanese locales.
It was lost by recent locale change.

Fix indentation to make the script more readable

Initialize SRCOUT3 as it may be used unitilizard for certains types of locales

Use :E instead of regex in ctypedef Makefiles

Simplify the locale generated Makefiles by using bmake multi variables for loops

Bring back a couple of ISO8859-15 locales:
- af_ZA.ISO8859-15
- en_AU.ISO8859-15
- en_CA.ISO8859-15
- en_NZ.ISO8859-15
- en_US.ISO8859-15
- fr_CA.ISO8859-15

Generate in the FreeBSD keyword when generating the Makefiles

Add ISO8859-1 everywhere ISO8859-15 exists

Allow to generate the locale when the source directory is not /usr/src

Use explicit filename when creating locale symlinks to avoid creating a
directory symlink when the target directory does not exist. This will
cause an error instead of a broken setup.

Sponsored by: EMC / Isilon Storage Division

Add a README to explain how to generate the locales from CLDR data

Modify the tool that generates the locales so that the generated CJK locales
are padded as expected by the users

Submitted by: marino
Obtained from: DragonflyBSD

Improve collation string and locales support

Merge collation support from Illumos and DragonflyBSD.

Locales are now generated with the new localedef(1) tool from CLDR POSIX files.
The generated files are now identified as "BSD 1.0" format.

The libc now only read "BSD 1.0" locales definitions, all other version will be
set to "C"
The localedef(1) tool has been imported from Illumos and modified to use tree(3)
instead of the CDDL avl(3)
A set of tool created by edwin@ and extended by marino@ for dragonfly has been
added to be able to generate locales and the Makefiles from the vanilla CLDR
unicode databases + a universal UTF-8 charmap (by marino@)
Update the locales to unicode v27
Given our regex(3) does not support multibyte (yet) it has been forced to always
use locale C
Remove now unused colldef(1) and mklocale(1)
Finish implementing the numeric BSD extension for ctypes
The number of supported locales has grown from 175 to 250 locales. Among the new
locales: 6 Arabic locales (AE EG JO MA QA SA), Different variations of spanish
locales.
Added new 3 components locales for mn_Cyrl_MN, sr_Cyrl_RS sr_Latn_RS,
zh_Hans_CN, zh_Hant_HK and zh_Hant_TW. Some aliases has been for 2 components
version when possible.

Thanks: Garrett D'Amore (Illumos) who made sure all his work was done under
BSD license!, Edwin Groothuis (edwin@) for the work he made on tools to be able
to generate locales definition usable in freebsd sources out of vanilla CLDR
definitions, John Marino (DragonflyBSD) who first merge the Illumos work into
Dragonfly and spent hours tracking down bugs.

Make generated makefiles respects ${SHAREDIR}

Synchronize locale generation tools with dragonfly

generates the makefiles and the sources

Add FreeBSD keyword on generated Makefiles

Add set of tools created by edwin@ long time ago to generate locales
Those tools were modified and extended by John Marino <draco@marino.st>
Add the generated charmaps and maps for LC_CTYPE generation

Obtained from: Dragonfly