MFC r360952: Generate zlib.pc from source.

MFC r347244:

Move contrib/zlib to sys/contrib/zlib so that we can use it in kernel.
This is a prerequisite of unifying kernel zlib instances.

Submitted by: Yoshihiro Ota <ota at j.email.ne.jp>
Approved by: re (kib)

MFC r316649 (kan):

Add missing double quote to fix r316635 commit.

MFC r316635:

Enable 16-bit longest_match for x86.

This gives a ~2% improvement in compression tests.

MFC r311285,312335: zlib 1.2.11.

MFC r311275: Restructure libz.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Merge the projects/release-pkg branch to head.

This allows packaging the base system with pkg(8), including
but not limited to providing the ability to provide upstream
binary update possibilities for non-tier-1 architectures.

This merge is a requirement of the 11.0-RELEASE, and as such,
thank you to everyone that has tested the project branch.

Documentation in build(7) etc. is still somewhat sparse, but
updates to those parts will follow.

Sponsored by: The FreeBSD Foundation

Prune bogus svn:mergeinfo from directores on which it should not ever
be present.

Sponsored by: The FreeBSD Foundation

Implement (ACFLAGS|CFLAGS|CXXFLAGS).SRC globally.

Sponsored by: EMC / Isilon Storage Division

META MODE: Prefer INSTALL=tools/install.sh to lessen the need for xinstall.host.

This both avoids some dependencies on xinstall.host and allows
bootstrapping on older releases to work due to lack of at least 'install -l'
support.

Sponsored by: EMC / Isilon Storage Division

Improve collation string and locales support

Merge collation support from Illumos and DragonflyBSD.

Locales are now generated with the new localedef(1) tool from CLDR POSIX files.
The generated files are now identified as "BSD 1.0" format.

The libc now only read "BSD 1.0" locales definitions, all other version will be
set to "C"
The localedef(1) tool has been imported from Illumos and modified to use tree(3)
instead of the CDDL avl(3)
A set of tool created by edwin@ and extended by marino@ for dragonfly has been
added to be able to generate locales and the Makefiles from the vanilla CLDR
unicode databases + a universal UTF-8 charmap (by marino@)
Update the locales to unicode v27
Given our regex(3) does not support multibyte (yet) it has been forced to always
use locale C
Remove now unused colldef(1) and mklocale(1)
Finish implementing the numeric BSD extension for ctypes
The number of supported locales has grown from 175 to 250 locales. Among the new
locales: 6 Arabic locales (AE EG JO MA QA SA), Different variations of spanish
locales.
Added new 3 components locales for mn_Cyrl_MN, sr_Cyrl_RS sr_Latn_RS,
zh_Hans_CN, zh_Hant_HK and zh_Hant_TW. Some aliases has been for 2 components
version when possible.

Thanks: Garrett D'Amore (Illumos) who made sure all his work was done under
BSD license!, Edwin Groothuis (edwin@) for the work he made on tools to be able
to generate locales definition usable in freebsd sources out of vanilla CLDR
definitions, John Marino (DragonflyBSD) who first merge the Illumos work into
Dragonfly and spent hours tracking down bugs.

Merge mpsutil(8) branch

mpsutil(8)/mprutil(8) are new utilities for managing LSI Fusion-MPT
2/3 controllers (mps(4) and mpr(4))

For now only informational commands have been implemented.

This utility has been written by scottl@ [1] and polished by myself[2]

Submitted by: scottl
Discussed with: scottl
Relnotes: yes
Sponsored by: Netflix [1]
Sponsored by: Gandi.net [2]

Upgrade our copies of clang, llvm, lldb, compiler-rt and libc++ to 3.7.0
release.

Please note that from 3.5.0 onwards, clang, llvm and lldb require C++11
support to build; see UPDATING for more information.

Release notes for llvm and clang can be found here:
<http://llvm.org/releases/3.7.0/docs/ReleaseNotes.html>
<http://llvm.org/releases/3.7.0/tools/clang/docs/ReleaseNotes.html>

Thanks to Ed Maste, Andrew Turner and Antoine Brodin for their help.

Exp-run: antoine
Relnotes: yes

Remove redundant COMPAT_32BIT guard on pkgconfig files. This is already handled
by the LIBRARIES_ONLY mechanism protecting FILES.

Sponsored by: EMC / Isilon Storage Division

Replace beforeinstall: handling with FILES.

This actually fixes some cases to respect LIBRARIES_ONLY.

Sponsored by: EMC / Isilon Storage Division

In libz's inflateMark(), avoid left-shifting a negative integer, which
is undefined.

Reviewed by: delphij
Differential Revision: https://reviews.freebsd.org/D3344
MFC after: 3 days

new depends

Revert r284417 it is not necessary anymore

Enforce overwritting SHLIBDIR

Since METAMODE has been added, sys.mk loads bsd.mkopt.mk which ends load loading
bsd.own.mk which then defines SHLIBDIR before all the Makefile.inc everywhere.

This makes /lib being populated again.

Reported by: many

Add META_MODE support.

Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.

Still need to add real targets under targets/ to build packages.

Differential Revision: D2796
Reviewed by: brooks imp

Upgrade our copy of clang, llvm and lldb to 3.5.0 release.

Please note that this version now requires C++11 support to build; see
UPDATING for more information.

Release notes for llvm and clang can be found here:
<http://llvm.org/releases/3.5.0/docs/ReleaseNotes.html>
<http://llvm.org/releases/3.5.0/tools/clang/docs/ReleaseNotes.html>

Thanks to Ed Maste, Roman Divacky, Andrew Turner, Justin Hibbits and
Antoine Brodin for their invaluable help with this import.

Approved by: portmgr (antoine)
MFC after: 1 month

only install .pc files when we are not installing 32bit compat libs...

This fixes the problem of installing the .pc files multiple times...

Fix path for zlib examples, this have no runtime effect and was
overlooked when I was doing zlib update.

MFC after: 2 weeks

Add a zlib pkg-config file (more and more ports requires it)

Approved by: delphij
MFC after: 1 week

Add a manual page for zopen(3) provided by our libz.so.

Fix up FreeBSD tag for files not from a vendor branch

Unexpand the tag, remove the fbsd:nokeywords property and add the
svn:keywords property. This should eliminate the gratuituous diffs
that appear on these files in projects branches.

Sponsored by: The FreeBSD Foundation

Make it possible to seek within a gzip stream.

MFV: Update zlib to 1.2.8.

MFC after: 1 month

- Correct mispellings of the word occurrence

Submitted by: Christoph Mallon <christoph.mallon@gmx.de> (via private mail)

MFV: Update zlib to 1.2.7.

(x86 assembler optimization disabled for now because it
requires the new .cfi_* directives that is not supported
by base system binutils).

MFC after: 1 week

Define several extra macros in bsd.sys.mk and sys/conf/kern.pre.mk, to
get rid of testing explicitly for clang (using ${CC:T:Mclang}) in
individual Makefiles.

Instead, use the following extra macros, for use with clang:
- NO_WERROR.clang (disables -Werror)
- NO_WCAST_ALIGN.clang (disables -Wcast-align)
- NO_WFORMAT.clang (disables -Wformat and friends)
- CLANG_NO_IAS (disables integrated assembler)
- CLANG_OPT_SMALL (adds flags for extra small size optimizations)

As a side effect, this enables setting CC/CXX/CPP in src.conf instead of
make.conf! For clang, use the following:

CC=clang
CXX=clang++
CPP=clang-cpp

MFC after: 2 weeks

Disable gvmat64.S, the assembler version of longest_match for now.

PR: kern/154073
MFC after: 3 days
Approved by: re (kib)

With retirement of cpumask_t and usage of cpuset_t for representing a
mask of CPUs, pc_other_cpus and pc_cpumask become highly inefficient.

Remove them and replace their usage with custom pc_cpuid magic (as,
atm, pc_cpumask can be easilly represented by (1 << pc_cpuid) and
pc_other_cpus by (all_cpus & ~(1 << pc_cpuid))).

This change is not targeted for MFC because of struct pcpu members
removal and dependency by cpumask_t retirement.

MD review by: marcel, marius, alc
Tested by: pluknet
MD testing by: marcel, marius, gonzo, andreast

etire the cpumask_t type and replace it with cpuset_t usage.

This is intended to fix the bug where cpu mask objects are
capped to 32. MAXCPU, then, can now arbitrarely bumped to whatever
value. Anyway, as long as several structures in the kernel are
statically allocated and sized as MAXCPU, it is suggested to keep it
as low as possible for the time being.

Technical notes on this commit itself:
- More functions to handle with cpuset_t objects are introduced.
The most notable are cpusetobj_ffs() (which calculates a ffs(3)
for a cpuset_t object), cpusetobj_strprint() (which prepares a string
representing a cpuset_t object) and cpusetobj_strscan() (which
creates a valid cpuset_t starting from a string representation).
- pc_cpumask and pc_other_cpus are target to be removed soon.
With the moving from cpumask_t to cpuset_t they are now inefficient
and not really useful. Anyway, for the time being, please note that
access to pcpu datas is protected by sched_pin() in order to avoid
migrating the CPU while reading more than one (possible) word
- Please note that size of cpuset_t objects may differ between kernel
and userland. While this is not directly related to the patch itself,
it is good to understand that concept and possibly use the patch
as a reference on how to deal with cpuset_t objects in userland, when
accessing kernland members.
- KTR_CPUMASK is changed and now is represented through a string, to be
set as the example reported in NOTES.

Please additively note that no MAXCPU is bumped in this patch, but
private testing has been done until to MAXCPU=128 on a real 8x8x2(htt)
machine (amd64).

Please note that the FreeBSD version is not yet bumped because of
the upcoming pcpu changes. However, note that this patch is not
targeted for MFC.

People to thank for the time spent on this patch:
- sbruno, pluknet and Nicholas Esborn (nick AT desert DOT net) tested
several revision of the patches and really helped in improving
stability of this work.
- marius fixed several bugs in the sparc64 implementation and reviewed
patches related to ktr.
- jeff and jhb discussed the basic approach followed.
- kib and marcel made targeted review on some specific part of the
patch.
- marius, art, nwhitehorn and andreast reviewed MD specific part of
the patch.
- marius, andreast, gonzo, nwhitehorn and jceel tested MD specific
implementations of the patch.
- Other people have made contributions on other patches that have been
already committed and have been listed separately.

Companies that should be mentioned for having participated at several
degrees:
- Yahoo! for having offered the machines used for testing on big
count of CPUs.
- The FreeBSD Foundation for having sponsored my devsummit attendance,
which has been instrumental.
- Sandvine for having offered offices and infrastructure during
development.

(I really hope I didn't forget anyone, if it happened I apologize in
advance).

Upgrade our copy of llvm/clang to r126079, from upstream's trunk.

This contains many improvements, primarily better C++ support, an
integrated assembler for x86 and support for -pg.

Fix some leftover binaries and shared libraries in the system that still
have an executable stack, due to linking in hand-assembled .S or .s
files, that have no .GNU-stack sections:

RWX --- --- /lib/libcrypto.so.6
RWX --- --- /lib/libmd.so.5
RWX --- --- /lib/libz.so.6
RWX --- --- /lib/libzpool.so.2
RWX --- --- /usr/lib/liblzma.so.5

These were found using scanelf, from the sysutils/pax-utils port.

Reviewed by: kib

Revert 212517 to restore pristine state of this file

Include FreeBSD svn tag

Merger of the quota64 project into head.

This joint work of Dag-Erling Smørgrav and myself updates the
FFS quota system to support both traditional 32-bit and new 64-bit
quotas (for those of you who want to put 2+Tb quotas on your users).

By default quotas are not compiled into the kernel. To include them
in your kernel configuration you need to specify:

options QUOTA # Enable FFS quotas

If you are already running with the current 32-bit quotas, they
should continue to work just as they have in the past. If you
wish to convert to using 64-bit quotas, use `quotacheck -c 64';
if you wish to revert from 64-bit quotas back to 32-bit quotas,
use `quotacheck -c 32'.

There is a new library of functions to simplify the use of the
quota system, do `man quotafile' for details. If your application
is currently using the quotactl(2), it is highly recommended that
you convert your application to use the quotafile interface.
Note that existing binaries will continue to work.

Special thanks to John Kozubik of rsync.net for getting me
interested in pursuing 64-bit quota support and for funding
part of my development time on this project.

MFV: zlib 1.2.5.

Expose a few symbols as public interface rather than private.

Note: the *64 interfaces are no longer exposed via zlib.h but were keep
as public interfaces;

Note 2: this commit would break applications that uses the moved symbols
directly.

MFV: libz 1.2.4.3.

Remove _LARGEFILE64_SOURCE==1 case which is supposed to be always false
on FreeBSD.

Pointed out by: dougb

For now, hardcode FreeBSD configuration that will never change on zlib.h:

unifdef -D_FILE_OFFSET_BITS=64 -U_LFS64_LARGEFILE -U_LARGEFILE64_SOURCE

Update to 1.2.4.1 (beta).

Expand $FreeBSD$.

Note that we are the same with the vendor tree.

Update to zlib 1.2.4 and add versioned symbols to the
library.

Sponsored by: iXsystems, Inc.

Bootstrap the merge history for zlib.

Build lib/ with WARNS=6 by default.

Similar to libexec/, do the same with lib/. Make WARNS=6 the norm and
lower it when needed.

I'm setting WARNS?=0 for secure/. It seems secure/ includes the
Makefile.inc provided by lib/. I'm not going to touch that directory.
Most of the code there is contributed anyway.

Set magic fbsd:nokeywords property that allows files to bypass
keyword expansion. (file-specific replacement for CVSROOT/exclude)

Resolve conflicts.

This commit was generated by cvs2svn to compensate for changes in r157043,
which included commits to RCS files with non-trunk default branches.

Correct a buffer overflow which can occur when decompressing a
carefully crafted deflated data stream. [1]

Correct problems in the AES-XCBC-MAC IPsec authentication algorithm. [2]

Submitted by: suz [2]
Security: FreeBSD-SA-05:18.zlib [1], FreeBSD-SA-05:19.ipsec [2]

Correct a buffer overflow which occurred in the handling of some
particularly corrupt deflated data streams.

Security: FreeBSD-SA-05:16.zlib
Approved by: re (security blanket)

Remove libz and pppd redundancy

Document zlib update procedure and provide future maintainers a
snapshot of the current state of the import.

MFC after: 2 weeks

Since the FreeBSD local modifications are mostly trivial (consisting
primarily of pointless $FreeBSD$ tags), sync most files in HEAD with
those in the ZLIB branch. This minimizes the differences between
HEAD and ZLIB and should simplify future imports.

After this, there are only three files with local modifications
(gzio.c, minigzip.c, and zconf.h) and two non-vendor files
(Makefile, zopen.c). The rest exactly match the vendor distribution.

PR: i386/76294
MFC after: 2 weeks

Update zconf.h and gzio.c with changes from ZLib 1.2.1 to ZLib 1.2.2.

These are two of the three files that have non-trivial differences from
the vendor branch. minigzip.c is the third, but there were no changes
from ZLib 1.2.1 to ZLib 1.2.2 in that file.

The rest of the files I intend to get reverted back to the vendor
branch (with cooperation of cvsadmin@).

PR: i386/76294

This commit was generated by cvs2svn to compensate for changes in r145474,
which included commits to RCS files with non-trunk default branches.

Correct a denial-of-service vulnerability in zlib.
For reference, this bug was first reported at
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=252253

Submitted by: "Dmitry V. Levin" <ldv@altlinux.org>

Update makefile for zlib 1.2.1.

Resolve conflicts.

This commit was generated by cvs2svn to compensate for changes in r131377,
which included commits to RCS files with non-trunk default branches.

Stage 3 of dynamic root support. Make all the libraries needed to run
binaries in /bin and /sbin installed in /lib. Only the versioned files
reside in /lib, the .so symlink continues to live /usr/lib so the
toolchain doesn't need to be modified.

Fix URL of zlib home page: It's hosted at gzip.org now.

In src/lib/libz/gzio.c the function gzprintf does not check if the
amount of bytes (supposed to be) written by vsnprintf exceeds the
size of the buffer.

PR: bin/48844
Submitted by: Peter A Jonsson <pj@ludd.luth.se>
Obtained from: OpenBSD
MFC after: 1 month

Actually implement gzip's -c option. minigzip now also doubles
as zcat properly.

PR: 13043
Submitted by: Toshihiko ARAI <toshi@jp.freebsd.org>
Jonathan Towne <jontow@twcny.rr.com>

Approved by: peter, sheldonh (mentor)
MFC After: 2 weeks

Fix an easy warning in a local addition.

Work around a gcc bug which causes zlib to dump core in certain
situations. This can be removed once gcc is fixed.

Submitted by: bde

Remove some files we don't want.

s/nilename/filename/

Fix conflicts.

This commit was generated by cvs2svn to compensate for changes in r92111,
which included commits to RCS files with non-trunk default branches.

This commit was generated by cvs2svn to compensate for changes in r92109,
which included commits to RCS files with non-trunk default branches.

Error handling fixes for inflate.

Add huff __FBSDID()s puff to whew! libz

Add zopen(), a stdio wrapper for gzipped data streams.

Obtained from: NetBSD

Remove whitespace at EOL.

MAN[1-9] -> MAN.

Fix typo: compatability -> compatibility.

Compatability is not an existing english word.

Add appropriate defines to use snprintf/vsnprintf instead of
sprintf/vsprintf.

Approved by: peter

sprintf -> snprintf

Prevent buffer overflows.

Remove single-space hard sentence breaks. These degrade the quality
of the typeset output, tend to make diffs harder to read and provide
bad examples for new-comers to mdoc.

Replace beforeinstall target with new variables used by .mk system.

Reviewed by: marcel, and make world

Hard-configure z_off_t as long. Soft configuration using HAVE_UNISTD_H
just breaks the prototypes for gzseek() and gztell() when an application
defines HAVE_UNISTD_H before including <zlib.h>. z_off_t was always
long for compiling zlib, but was sometimes off_t for compiling
applications, e.g., Ethereal 0.7.5.

This "fix" preserves bug for bug binary compatibility. z_off_t should
be off_t for everything, but zlib doesn't support off_t being longer
than long, so using the correct type without fixing zlib's internals
would at best break binary compatibility. This "fix" also make the
namespace problems for HAVE_UNISTD_H no worse than hundreds of other
namespace prooblems in zconf.h. I'll wait for a new release of zlib
for proper fixes.

Reported by: Guy Harris via jkh

Add $FreeBSD$ lines to man pages that are missing them to make it
easier for translation teams.

PR: docs/13418
Submitted by: Alexey Zelkin <phantom@cris.net>

$Id$ -> $FreeBSD$

Merge changes from vendor branch (1.1.1 -> 1.1.3) into mainline.

This commit was generated by cvs2svn to compensate for changes in r42468,
which included commits to RCS files with non-trunk default branches.

Allow minigzip to be invoked more like gzip, checking to see
if we're being called as gunzip as well as dealing with the -c
flag.
Submitted by: jhay

Backup out the last commit, it was already there.

Noticed by: bde

Manpages not installed
PR: 6327
Reviewed by: phk
Submitted by: Chia-liang Kao <clkao@cirx.org>

Install manpage too

foo.gz is created by 'make test' so add it to CLEANFILES.

algorithm.doc renamed to algorithm.txt in new distribution.

Merge conflicts.

This commit was generated by cvs2svn to compensate for changes in r33904,
which included commits to RCS files with non-trunk default branches.

This commit was generated by cvs2svn to compensate for changes in r29747,
which included commits to RCS files with non-trunk default branches.

Back out an easy $FreeBSD$

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

Clean up the beforeinstall target a bit more, bde pointed out that the old
one was a mess. (I was in a 'talk' session at the time, can you tell? :-)

oops.. attempted to install header files from wrong directory..

Add a bmakefile for zlib.

This commit was generated by cvs2svn to compensate for changes in r17651,
which included commits to RCS files with non-trunk default branches.