MFC r310201:

Update locales cldr to v30.0.3 and unicode to 9.0.0

Relnotes: yes

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Posixify the locales name for variants

For all locales with variants:
- if no ambiguity on the locale (only one variant) just use the regular name
- if ambiguity, pick one as default and append @<variant> to the others
respecting POSIX

As a result:
- All the 3 components locales added recently are renamed to the usual 2
components version for all but sr_RS.UTF-8
- Set sr_RS.UTF-8 to the cyrillic variant
- Add sr_RS.UTF-8@latin
- Remove the symlinks aliases they were created to represent the 2 components
version as aliasas and are now useless
- Update the OptionalObsoleteFiles.inc and ObsoleteFiles.inc to reflect those
changes

Discussed with: ache@
Approved by: re@ (gjb)

Regenerate the Makefile so that actually the new generated collation are
properly installed

Regenerates locales Makefiles and locales

Simplify the locale generated Makefiles by using bmake multi variables for loops

Bring back a couple of ISO8859-15 locales:
- af_ZA.ISO8859-15
- en_AU.ISO8859-15
- en_CA.ISO8859-15
- en_NZ.ISO8859-15
- en_US.ISO8859-15
- fr_CA.ISO8859-15

Regenerate locales after readding ISO8859-1 for locales that have ISO8859-15

Requested by: arche

Use explicit filename when creating locale symlinks to avoid creating a
directory symlink when the target directory does not exist. This will
cause an error instead of a broken setup.

Sponsored by: EMC / Isilon Storage Division

Improve collation string and locales support

Merge collation support from Illumos and DragonflyBSD.

Locales are now generated with the new localedef(1) tool from CLDR POSIX files.
The generated files are now identified as "BSD 1.0" format.

The libc now only read "BSD 1.0" locales definitions, all other version will be
set to "C"
The localedef(1) tool has been imported from Illumos and modified to use tree(3)
instead of the CDDL avl(3)
A set of tool created by edwin@ and extended by marino@ for dragonfly has been
added to be able to generate locales and the Makefiles from the vanilla CLDR
unicode databases + a universal UTF-8 charmap (by marino@)
Update the locales to unicode v27
Given our regex(3) does not support multibyte (yet) it has been forced to always
use locale C
Remove now unused colldef(1) and mklocale(1)
Finish implementing the numeric BSD extension for ctypes
The number of supported locales has grown from 175 to 250 locales. Among the new
locales: 6 Arabic locales (AE EG JO MA QA SA), Different variations of spanish
locales.
Added new 3 components locales for mn_Cyrl_MN, sr_Cyrl_RS sr_Latn_RS,
zh_Hans_CN, zh_Hant_HK and zh_Hant_TW. Some aliases has been for 2 components
version when possible.

Thanks: Garrett D'Amore (Illumos) who made sure all his work was done under
BSD license!, Edwin Groothuis (edwin@) for the work he made on tools to be able
to generate locales definition usable in freebsd sources out of vanilla CLDR
definitions, John Marino (DragonflyBSD) who first merge the Illumos work into
Dragonfly and spent hours tracking down bugs.

Use more appropriate ${SHAREDIR} rather than /usr/share.

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Ensure SYMLINKS fully specified

Differential Revision: D2748
Reviewed by: brooks

Add lv_LV, move lt_LT to common part

Croatian locale support for hr_HR.ISO8859-2

Without this patch /usr/share/locale/hr_HR.ISO8859-2/LC_COLLATE
is a link to the ../la_LN.ISO8859-2/LC_COLLATE. It means
that there is no support for Croatian locale.

PR: conf/120113
Submitted by: Simun Mikecin <numisemis@yahoo.com>
MFC after: 1 week

add ca_AD, ca_FR and ca_IT locales

Catalan language is not only spoken in Spain (ca_ES), but also
in Andorra, France and Italy. In Andorra it is the official
language.

(see http://en.wikipedia.org/wiki/Catalan_language#Geographic_distribution)

Add a bunch of symlinks to between ca_ES and ca_AD, ca_FR and ca_IT.

PR: conf/92541
Submitted by: <rmh@io.debian.net>
MFC after: 1 week

Add collation sequence for no_NO. The same sequence applies to nb_NO
and nn_NO, which are symlinked to no_NO.

The patch in the PR contained a number of errors apparently based on
(sometimes incorrect) pronunciation; for instance, v and w are
distinct letters and should be collated in that order, even if they
are pronounced the same, while <u:> should be collated with u, even
though it is often mispronounced as y. For lack of a solid reference,
I have taken sv_SE and simply changed the last three letters of the
alphabet.

PR: conf/51920
MFC after: 2 weeks

Add Turkish collate

PR: 118976
Submitted by: Ismail YENIGUL <ismail.yenigul@endersys.com.tr>

Add locales for nb_NO and nn_NO.

Reviewed by: philip

Add mn_MN.UTF-8

New Ukrainian locale: uk_UA.CP1251.

Submitted by: Alexander Peresunko

Utilize FILES and SYMLINKS, other simplifications.

Add Basque (Spain) locales: eu_ES.ISO8859-1, eu_ES.ISO8859-15, eu_ES.UTF-8.

(This differs somewhat from the version originally submitted - any mistakes
are my own.)

PR: 68524
Submitted by: J. Vicente Carrasco -Bixen-

Add an Israel Hebrew locale: he_IL.UTF-8.

PR: 65826
Submitted by: Hye-Shik Chang

Add data for missing categories to zh_HK.Big5HKSCS locale.

Add UTF-8 versions of all the currently supported system locales. Most of
the hard work was done by Hye-Shik Chang in the misc/utf8locale port; I made
a few minor adjustments and merged the makefiles.

PR: 44307

Add kk_KZ.PT154

Submitted by: Birsh T <tim@zhezu.kz>

Add be_BY.*

Submitted by: Yury Tarasievich <grog@grsu.by>

Enable GB2312 locale.

Fix sl_SI.ISO8859-2

PR: 50160
Submitted by: Simon 'Zekar <simon@entered.net>

Add ko_KR.CP949 locale

PR: 55341
submitted by: Hye-Shik Chang <perky@freebsd.org>

Support for the Amharic locale.

Add GB18030 locale

PR: 51729
Submitted by: Kang Liu <liukang@bjpu.edu.cn>

Add ARMSCSII-8 locale

Submitted by: Vahe Khachikyan <vahe@khachikyan.de>
PR: 53944

Add ru_RU.CP1251

Add zh_CN.GBK to ASCIILINKS

Add ca_ES locale

PR: 45874
Submitted by: Joan Picanyol i Puig <joan-dev@biaix.org>

Add lt_LT.ISO8859-13

PR: 44268
Submitted by: Kestutis Paulikas <kestas@elen.ktu.lt>

Add sr_YU locales
Submitted by: Toni Andjelkovic <toni@soth.at>

Drop support for COPY, -c has been the default mode of install(1)
for a long time now.

Approved by: bde

s/${INSTALL} -c/${INSTALL} ${COPY}/

Fix pl_PL collating table

PR: 39408
Submitted by: Michal Pasternak <doc@pasternak.w.lub.pl>

la_LN.ISO8859-2 depends on map.ISO8859-2. Reflect it in dependence rules

Associate tr_TR.ISO8859-9's LC_COLLATE with US-ASCII's one

Add pt_BR.ISO8859-1 (Brazilian Portuguese) locale

Submitted (partly) by: dcs

Add ro_RO

PR: 33343
Submitted by: Adrian Penisoara <ady@warpnet.ro>

Optimize locale data Makefiles as well as make their style more consistent.

Add uk_UA.ISO8859-5 locale

PR: 32450
Submitted by: partially by Alexey Klimov <kao@wiuu.kiev.ua>

Add hi_IN.ISCII-DEV

Add bg_BG.CP1251

PR: 25472
Submitted by: Peter Pentchev <roam@ringlet.net>

Add more *.US-ASCII entries

Switch to new locale names

Add sk_SK

PR:
Submitted by:
Reviewed by:
Approved by:
Obtained from:

Add et_EE

PR: 25495
Submitted by: Vallo Kallaste <vallo@matti.ee>

DIS_8859-15 -> ISO_8859-15 rename

Add Greek collate

PR: 24324
Submitted by: past@netmode.ntua.gr

Get rid of deprecated ru_SU.* locales. ru_RU.* will be used instead.

Add 8859-4 and lt_LT
Add missing dependance to cs_CZ

Submitted by: Ricardas Cepas <rch@writeme.com>

lt_LN->la_LN

Add cs_CZ

Submitted by: Cejka Rudolf <cejkar@dcse.fee.vutbr.cz>

Add uk_UA.KOI8-U

Submitted by: Alexey Zelkin <phantom@cris.net>

Add 8859-5, remove BUILD_TOOLS ifdef

Remove one of the locales which is currently breaking the build.

sv_SE collate
add one missing dependance

PR: 13928
Submitted by: Palle Girgensohn <girgen@partitur.se>

ISO8859-5 collate

Submitted by: Nikolai Saoukh <nms@ethereal.ru>

$Id$ -> $FreeBSD$

Change locale name of Traditional Chinese : zh_TW.BIG5 -> zh_TW.Big5.
It makes compatible with IANA charset defination, and let existent I18N
app happier.

Ref: <http://www.isi.edu/in-notes/iana/assignments/character-sets>

Support the IANA definition of Shift Jis nameley ja_JP.Shift_JIS
as well as the X11 version ja_JP.SJIS

Correct a link problem with zh_TW.BIG5, make the display reasonable.

Supporting locale for Chinese Big5 completely.

Fix disorder introduced by me in the Euro part 2 locale commit.

Submited by: bde
(This was uncommitted for too long in my tree, but is well tested)

back out obj/colldef PATH hack, it breaks cross-compiling per Bruce
There is _BUILD_TOOLS hack for. Plase all targets under it.

syntax change: allow symbolic names as substitute first arg
use this to substitute <ss>

Euro support, part 2.

This should be merged into RELENG_3 and a similar patch may be needed
for RELENG_2_2, should that deemed necessary.

Make world succeeded with these patches in my tree.

Submitted by: "Kaleb S. KEITHLEY" <kaleb@ics.com>

I have been running with these locale data for a while now and
I hope some other people might find them useful. They are for
zh_CN.EUC (GB) only. I'm not familiar with the BIG5 encoding,
so I could only hope someone else would fill the gap.

PR: 7310
Submitted by: Luoqi Chen <luoqi@chen.ml.org>

add Hungarian clooate

Use the installed version of colldef for `make all'. It is now
bootstrapped by `make world'. The version just built in ".."
normally won't work if the target system is not binary compatible.

Don't build or install anything if _BUILD_TOOLS is defined. Then
we only want to build and install the colldef binary, but the layout
of the colldef tree forces recursing to colldef/data for at least
the obj target even when _BUILD_TOOLS is defined.

Add Slovene link

Add Polish link
Submitted by: Andrzej Bialecki <abial@warman.org.pl>

put in the link for the dummy collation module for Shift-JIS

Add ASCII collate mainly for fallback for EUC-type locales

Add generic colldef for ISO-8859-2.
Submitted by: Slaven Reziæ (Tomiæ) <eserte@cs.tu-berlin.de>

Use '.for' loop from make(1), not sh(1) 'for' loops.

Revert $FreeBSD$ to $Id$

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

icelandic collation
(non-icelandic characters in latin1 order)

Add de and es collates
Submitted by: joerg

Save half of LC_COLLATE space
ASCIIfy data files, use RFC1345 symbolic names instead of hex codes
Improve parser and manpage

Bring in my changes for removing the pestilent obj links (unless you
really want them) from /usr/src. This is the final version of the
patches, incorporating the feedback I've received from -current.

Cleanup make process per Bruce suggestions