Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

locale: update to CLDR 43 and Unicode 15

Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D41151

locales: Update to CLDR 41.0 and Unicode 14.0

Relnotes: yes

locale: Add Farsi/Persian locales

MFC after: 3 weeks
Reviewed by: farrokhi
Differential Revision: https://reviews.freebsd.org/D24359

locales: prepare CLDR update to v39.0 and Unicode to 13.0.0

Note that rebuilding the cldr tool is not necessary anymore, also note
that the sources are not anymore provided along with the other distfiles

locales: only generate unicode locales

Fix generation of colldef source files for non-UTF-8 locales

- Files for colldef were generated by duplicating UTF-8 collation files
for each language and included invalid characters in the non-UTF-8
encodings. localedef(1) does not allow those characters.
cldr2def.pl now checks if the characters are valid based on charmap files.

TODO: ja_JP.UTF-8 locale should not be generated solely from CLDR because
it was standardized in a document "UI-OSF Application Platform Profile for
Japanese Environment" which was incompatible with information in CLDR.
Most of commercial Unix vendors adopt this pre-Unicode-era document
as the reference even for UTF-8 locale. Newer versions of Solaris have
added a CLDR version as ja_JP.UTF-8@cldr, and IBM AIX has used
JA_JP.UTF-8 for the UI-OSF specification and ja_JP.UTF-8 for CLDR.

Note that this commit does not change generation of ja_JP.UTF-8.
Changes related to this issue will be committed separately later.

- Generate POSIX charamap UTF-32 as a reference. It was confusing that
charmap.xml used Unicode names defined in UnicodeData.txt though POSIX
charmap used slightly different names for the same code points.
cldr2def.pl now uses UTF-32.cm as single information source for Unicode
symbol names and code points. Charset.xml is also updated to use them.

- Fix a bug in get_encodings() in cldr2def.pl which did not understand
0x00+0x00 notation correctly in charmaps/ISCII-DEV.TXT.

- Do not regenerate posix/xx_Comm_C.UTF-8.src every time when doing
"make build".

Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D27809

tools/tools/locale: fix static-colldef

tools/tools/locale: install generated files in current src checkout

locale: fix mode for installed files to be 644, not 755

While here, drop '-c' flag to install as it's default and provided for
backward compatibility only.

Fix the source directory when installing the results. The install
target did not install them actually.

Spotted by: Thomas Munro, bapt, yuripv

update wcwidth data from utf8proc

Character width data being out of date is a constant source
of weird rendering issues and wasted time trying to diagnose
those, e.g. as reported by Jeremy Chadwick:

https://gitlab.com/muttmua/mutt/-/issues/67

Sadly, there is no real ("standard") wcwidth data source, so
this tries to rectify the problem using the utf8proc one (through
its C API) which would hopefully benefeat both FreeBSD and
utf8proc through bug reports (if any).

Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D27259

Add collation version support to querylocale(3).

Provide a way to ask for an opaque version string for a locale_t, so
that potential changes in sort order can be detected. Similar to
ICU's ucol_getVersion() and Windows' GetNLSVersionEx(), this API is
intended to allow databases to detect when text order-based indexes
might need to be rebuilt.

The CLDR version is extracted from CLDR source data by the Makefile
under tools/tools/locale, written into the machine-generated Makefile
under shared/colldef, passed to localedef -V, and then written into
LC_COLLATE file headers. The initial version is 34.0.
tools/tools/locale was recently updated to pull down 35.0, but the
output hasn't been committed under share/colldef yet, so that will
provide the first observable change when it happens. Other versioning
schemes are possible in future, because the format is unspecified.

Reviewed by: bapt, 0mp, kib, yuripv (albeit a long time ago)
Differential Revision: https://reviews.freebsd.org/D17166

- Update to Unicode CLDR v35 (Unicode 12.0).

- Update tools/tools/locale to add make targets to automatically
generate locale source files. With this change, just typing
"make obj && make -j4" will rebuild them. Check README for more details.

- Fix issues in ja_JP ctypedef and range specification support
in utf8-rollup.pl.

- Add a temporary patch for UnicodeData.txt to fix code ranges of
CJK Ideograph Extension A and Extension B.

- tools/cldr2def.pl:
Use eucJP for ja_JP ctypedef because eucJP is not compatible with UTF-8.

- tools/convert_map.pl:
Add a verbose error message.

- tools/utf8-rollup.pl:
Normalize entries to use Unicode, not UTF-8.

Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D25503

tools/tools/locale: allow POSIX target to be built in parallel

While it's rarely used target, more so a one not used during the
buildworld, it helps when it's not taking hours (literally).

Add ga_IE.UTF-8 locale.

PR: 228587
Submitted by: Micil <micheal.maclochlainn@oegaillimh.ie> (LC_TIME)
Reviewed by: bapt
Approved by: kib (mentor, implicit)
MFC after: 3 days
Differential revision: https://reviews.freebsd.org/D17997

Use UnicodeData.txt to create UTF-8 ctype map.

This should provide more complete coverage of currently defined Unicode
characters as compared to manually assembled one we use currently.

Comparison of original and new UTF-8 ctype maps by character class:

TYPE ORIG NEW
alnum 94229 126029
alpha 93557 125419
blank 4 2
cntrl 73 137685
digit 469 622
graph 109615 137203
lower 1478 2145
print 109641 137222
punct 3428 797
rune 110481 274907
space 33 24
upper 983 1781
xdigit 469 622

Large number of added cntrl definitions is due to the fact that private-use
planes are currently defined as such, this can change in the future.

Discussed with: bapt
Approved by: kib (mentor, implicit)
MFC after: 1 month
Differential revision: https://reviews.freebsd.org/D17842

Cleanup locale tools:

- Simplify the source dir specification, and update README
appropriately
- Drop the LC (doonly) processing, it's broken, and even if fixed, not
really useful
- Don't remove the target directories while installing new data as it
removes Makefile.depend which we don't manage; only rm the files we
are going to add/replace/delete instead
- Restrict adding bsd.endian.mk to colldef and ctypedef Makefiles, it's
not needed in other (text-only) categories
- GC unused scripts; they don't seem to be particularly helpful standalone
as well

Reviewed by: bapt
Approved by: kib (mentor, implicit)
Differential Revision: https://reviews.freebsd.org/D17858

Add hybrid C.UTF-8 locale being identical to default C locale except
that it uses the same ctype maps and functions as other UTF-8 locales.

Reviewed by: bapt, cem, eadler
Approved by: kib (mentor, implicit)
Differential Revision: https://reviews.freebsd.org/D17833

Update to CLDR 34 and UNICODE 11.

Discussed with: bapt
Approved by: kib (mentor)
MFC after: 1 month

Readd Big5: some large databases setup are still requiring it.

Reported by: "張君天(Chun-Tien Chang)" <tcs@kitty.2y.idv.tw>

Prepare the removal of the zh_TW.Big5 encoding

Disable timedef generation, cldr data are too far from freebsd extensions

In order to prepare cldr v30.0.3 and unicode 9.0.0,
more locales are removed from automatic mapping from unicode as their
encoding does not supports new characters available in unicode

Prepare import of CLDR v30.3 and unicode 9.0.0

Upstream kk_Cyrl_KZ has been renamed kk_KZ
Upstream mn_Cyrl_MN has been renamed mn_MN

For ru_RU: the default currency for unicode is now ₽ which is not mapped to
other encoding, add charmaps entries to be able to generate them

Fix an obvious typo.

Allow to install various components of the locales independantly

Rework collation generation:

When building collation database for non unicode encodings use the proper
unicode mapping (this fixes collation not working properly for those encodings)

For locales where new characters are added but only for unicode, stop trying to
map the new characters, directly extract from CLDR the collation files for the
said encoding

Stop trying to generate encoding map from unicode version for GB2312 and encCN
It was not reliable. Instead use the map provide by the CLDR project

Reported by: ache

Allow to generate the locale when the source directory is not /usr/src

Synchronize locale generation tools with dragonfly

generates the makefiles and the sources

Add set of tools created by edwin@ long time ago to generate locales
Those tools were modified and extended by John Marino <draco@marino.st>
Add the generated charmaps and maps for LC_CTYPE generation

Obtained from: Dragonfly