MFC r356212,r356366,r356416,r357785
Update libarchive to version 3.4.2

Relevant vendor changes (r356212):
Issue #351: Refactor and implement private state logic for write filters
PR #1252: RAR5 reader - verify window size for solid files (OSS-Fuzz 15482)
PR #1255: zip writer - don't append unused NUL for directories
PR #1260: Fix sparse file offset overflow on 32-bit systems
PR #1263: UNICODE filename support for reading lha/lzh format
Issue #1276: Bugfix and optimize archive_wstring_append_from_mbs()
PR #1288: Add the "xattrhdr" option to pax write options
PR #1295: 7z reader - fix reading archives with digests in PackInfo
PR #1296: RAR5 reader - verify window size for multivolume archives
PR #1297: ZIP reader - support LZMA_STREAM_END marker in 'lzma alone' files
Issue #1298: Fix a heap-buffer-overflow in archive_string_append_from_wcs()
OSS-Fuzz 19360, 19362: LHA reader - plug two memory leaks on error
Fix possible off-by-one when dealing with readlink(2)

Relevant vendor changes (r356366):
Issue #1302: Plug memory leak on failure of archive_write_client_open()

Relevant vendor changes (r356416):
Issue #1302: Re-do fix for archive_write_client_open()

Relevant vendor changes (r357785):
PR #1289: atomic extraction support (bsdtar -x --safe-writes)
PR #1308: big endian fix for UTF16 support in LHA reader
PR #1326: reject RAR5 files that declare invalid header flags
Issue #987: fix support 7z archive entries with Delta filter
Issue #1317: fix compression output buffer handling in XAR writer
Issue #1319: fix uname or gname longer than 32 characters in pax writer
Issue #1325: fix use after free when archiving hardlinks in ISO9660 or XAR
Use localtime_r() and gmtime_r() instead of localtime() and gmtime()

MFC r347990:
Sync libarchive with vendor.

Relevant vendor changes:
Issue #795: XAR - do not try to add xattrs without an allocated name
PR #812: non-recursive option for extract and list
PR #958: support reading metadata from compressed files
PR #999: add --exclude-vcs option to bsdtar
Issue #1062: treat empty archives with a GNU volume header as valid
PR #1074: Handle ZIP files with trailing 0s in the extra fields
(Android APK archives)
PR #1109: Ignore padding in Zip extra field data (Android APK archives)
PR #1167: fix problems related to unreadable directories
Issue #1168: fix handling of strtol() and strtoul()
PR #1172: RAR5 - fix invalid window buffer read in E8E9 filter
PR #1174: ZIP reader - fix of MSZIP signature parsing
PR #1175: gzip filter - fix reading files larger than 4GB from memory
PR #1177: gzip filter - fix memory leak with repeated header reads
PR #1180: ZIP reader - add support for Info-ZIP Unicode Path Extra Field
PR #1181: RAR5 - fix merge_block() recursion
(OSS-Fuzz 12999, 13029, 13144, 13478, 13490)
PR #1183: fix memory leak when decompressing ZIP files with LZMA
PR #1184: fix RAR5 OSS-Fuzz issues 12466, 14490, 14491, 12817
OSS-Fuzz 12466: RAR5 - fix buffer overflow when parsing huffman tables
OSS-Fuzz 14490, 14491: RAR5 - fix bad shift-left operations
OSS-Fuzz 12817: RAR5 - handle a case with truncated huffman tables
PR #1186: RAR5 - fix invalid type used for dictionary size mask
(OSS-Fuzz 14537)
PR #1187: RAR5 - fix integer overflow (OSS-Fuzz 14555)
PR #1190: RAR5 - RAR5 don't try to unpack entries marked as directories
(OSS-Fuzz 14574)
PR #1196: RAR5 - fix a potential SIGSEGV on 32-bit builds
OSS-Fuzz 2582: RAR - fix use after free if there is an invalid entry
OSS-Fuzz 14331: RAR5 - fix maximum owner name length
OSS-Fuzz 13965: RAR5 - use unsigned int for volume number + range check

Additional RAR5 reader changes:
- support symlinks, hardlinks, file owner, file group, versioned files
- change ARCHIVE_FORMAT_RAR_V5 to 0x100000
- set correct mode for readonly directories
- support readonly, hidden and system Windows file attributes

MFC r347999:
Install missing data file for
lib.libarchive.functional_test.test_read_format_zip_utf8_paths

Approved by: re (gjb blanket)

MFH r336801,r336854:

MFH r336801 (cem):
Cherry-pick upstream 2c8c83b9

Relevant vendor changes:
Fix issue #948: out-of-bounds read in lha_read_data_none()

MFH r336854:
Sync libarchive with vendor.

Important vendor changes:
PR #993: Chdir to -C directory for metalog processing
OSS-Fuzz #4969: Check size of the extended time field in zip archives
PR #973: Record informational compression level in gzip header

amdbugs: 877
Security: CVE-2017-14503

MFH r324148:
Sync libarchive with vendor.

Relevant vendor changes:
PR #905: Support for Zstandard read and write filters
PR #922: Avoid overflow when reading corrupt cpio archive
Issue #935: heap-based buffer overflow in xml_data (CVE-2017-14166)
OSS-Fuzz 2936: Place a limit on the mtree line length
OSS-Fuzz 2394: Ensure that the ZIP AES extension header is large enough
OSS-Fuzz 573: Read off-by-one error in RAR archives (CVE-2017-14502)

Security: CVE-2017-14166, CVE-2017-14502

MFC r315636,315876,316095:
Sync libarchive with vendor

Vendor changes/bugfixes (FreeBSD-related):
r315636:
PR 867 (bsdcpio): show numeric uid/gid when names are not found
PR 870 (seekable zip): accept files with valid ZIP64 EOCD headers
PR 880 (pax): Fix handling of "size" pax header keyword
PR 887 (crypto): Discard 3072 bytes instead of 1024 of first keystream
OSS-Fuzz issue 806 (mtree): rework mtree_atol10 integer parser
Break ACL read/write code into platform-specific source files

r315876:
Store extended attributes with extattr_set_link() if no fd is provided
Add extended attribute tests to libarchive and bsdtar
Fix tar's test_option_acls
Support the UF_HIDDEN file flag

r316095:
Constify variables in several places
Unify platform ACL code in a single source file
Fix unused variable if compiling on FreeBSD without NFSv4 ACL support

MFC r314571:
Update libarchive to version 3.3.1 (and sync with latest vendor dist)

Notable vendor changes:
PR #501: improvements in ACL path handling
PR #724: fix hang when reading malformed cpio files
PR #864: fix out of bounds read with malformed GNU tar archives
Documentation, style, test suite improvements and typo fixes.

New options to bsdtar that enable or disable reading and/or writing of:
Access Control Lists (--acls, --no-acls)
Extended file flags (--fflags, --no-fflags)
Extended attributes (--xattrs, --no-xattrs)
Mac OS X metadata (Mac OS X only) (--mac-metadata, --no-mac-metadata)

MFC r303804:

Fix building usr.bin/tar/tests with PIE symbol building enabled by
removing CFLAGS+= -static

`CFLAGS+= -static` was a carryover from pre-r289195 with
usr.bin/tar/test/Makefile that should have been specified in LDFLAGS
There doesn't seem to be an apparent need for static compilation
of the test binaries.

Obtained-from: opBSD (418a491eed20d2603ddd1f1bd92c2c0d95094002)

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFV r302003,r302037,r302038,r302056:

Update libarchive to 3.2.1 (bugfix and security fix release)

List of vendor fixes:
- fix exploitable heap overflow vulnerability in Rar decompression
(vendor issue 719, CVE-2016-4302, TALOS-2016-0154)
- fix exploitable stack based buffer overflow vulnebarility in mtree
parse_device functionality (vendor PR 715, CVE-2016-4301, TALOS-2016-0153)
- fix exploitable heap overflow vulnerability in 7-zip read_SubStreamsInfo
(vendor issue 718, CVE-2016-4300, TALOS-2016-152)
- fix integer overflow when computing location of volume descriptor
(vendor issue 717)
- fix buffer overflow when reading a crafred rar archive (vendor issue 521)
- fix possible buffer overflow when reading ISO9660 archives on machines
where sizeof(int) < sizeof(size_t) (vendor issue 711)
- tar and cpio should fail if an input file named on the command line is
missing (vendor issue 708)
- fix incorrect writing of gnutar filenames that are exactly 512 bytes
long (vendor issue 682)
- allow tests to be run from paths that are equal or longer than 128
characters (vendor issue 657)
- add memory allocation errors in archive_entry_xattr.c (vendor PR 603)
- remove dead code in archive_entry_xattr_add_entry() (vendor PR 716)
- fix broken decryption of ZIP files (vendor issue 553)
- manpage style, typo and description fixes

Post-3.2.1 vendor fixes:
- fix typo in cpio version reporting (Vendor PR 725, 726)
- fix argument range of ctype functions in libarchive_fe/passphrase.c
- fix ctype use and avoid empty loop bodies in WARC reader

MFC after: 1 week
Security: CVE-2016-4300, CVE-2016-4301, CVE-2016-4302
Approved by: re (kib)

MFV r299425:

Update libarchive to 3.2.0

New features:
- new bsdcat command-line utility
- LZ4 compression (in src only via external utility from ports)
- Warc format support
- 'Raw' format writer
- Zip: Support archives >4GB, entries >4GB
- Zip: Support encrypting and decrypting entries
- Zip: Support experimental streaming extension
- Identify encrypted entries in several formats
- New --clear-nochange-flags option to bsdtar tries to remove noschg and
similar flags before deleting files
- New --ignore-zeros option to bsdtar to handle concatenated tar archives
- Use multi-threaded LZMA decompression if liblzma supports it
- Expose version info for libraries used by libarchive

Patched files (fixed compiler warnings):

contrib/libarchive/cat/bsdcat.c (vendor PR #702)
contrib/libarchive/cat/bsdcat.h (vendor PR #702)
contrib/libarchive/libarchive/archive_read_support_format_mtree.c (PR #701)
contrib/libarchive/libarchive_fe/err.c (vendor PR #703)

MFC after: 1 month
Relnotes: yes

Merge ^/user/ngie/release-pkg-fix-tests to unbreak how test files are installed
after r298107

Summary of changes:

- Replace all instances of FILES/TESTS with ${PACKAGE}FILES. This ensures that
namespacing is kept with FILES appropriately, and that this shouldn't need
to be repeated if the namespace changes -- only the definition of PACKAGE
needs to be changed
- Allow PACKAGE to be overridden by callers instead of forcing it to always be
`tests`. In the event we get to the point where things can be split up
enough in the base system, it would make more sense to group the tests
with the blocks they're a part of, e.g. byacc with byacc-tests, etc
- Remove PACKAGE definitions where possible, i.e. where FILES wasn't used
previously.
- Remove unnecessary TESTSPACKAGE definitions; this has been elided into
bsd.tests.mk
- Remove unnecessary BINDIRs used previously with ${PACKAGE}FILES;
${PACKAGE}FILESDIR is now automatically defined in bsd.test.mk.
- Fix installation of files under data/ subdirectories in lib/libc/tests/hash
and lib/libc/tests/net/getaddrinfo
- Remove unnecessary .include <bsd.own.mk>s (some opportunistic cleanup)

Document the proposed changes in share/examples/tests/tests/... via examples
so it's clear that ${PACKAGES}FILES is the suggested way forward in terms of
replacing FILES. share/mk/bsd.README didn't seem like the appropriate method
of communicating that info.

MFC after: never probably
X-MFC with: r298107
PR: 209114
Relnotes: yes
Tested with: buildworld, installworld, checkworld; buildworld, packageworld
Sponsored by: EMC / Isilon Storage Division

Merge the projects/release-pkg branch to head.

This allows packaging the base system with pkg(8), including
but not limited to providing the ability to provide upstream
binary update possibilities for non-tier-1 architectures.

This merge is a requirement of the 11.0-RELEASE, and as such,
thank you to everyone that has tested the project branch.

Documentation in build(7) etc. is still somewhat sparse, but
updates to those parts will follow.

Sponsored by: The FreeBSD Foundation

DIRDEPS_BUILD: Connect MK_TESTS.

Sponsored by: EMC / Isilon Storage Division

Don't override LIB*DIR variables from src.libnames.mk.

In some cases switch to the LIB*SRCDIR value.

These recently were defined in r291327 and r291619.

Sponsored by: EMC / Isilon Storage Division

Remove redundant DPSRCS which were already in SRCS.

DPSRCS already contains all of SRCS.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Integrate the tests from lib/libarchive, usr.bin/cpio, and usr.bin/tar in to
the FreeBSD test suite

functional_test.sh was ported from bin/sh/tests/functional_test.sh, as a
small wrapper around libarchive_test, bsdcpio_test, and bsdtar_test provided
by upstream.

A handful of testcases in lib/libarchive/tests have been disabled as they
were failing when run with kyua test (see BROKEN_TESTS in
lib/libarchive/tests/Makefile)

As a sidenote: this removes the check/test targets from the Makefiles as they
don't match the pattern used in the rest of the FreeBSD test suite.

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Revert r289133; retry the merge

Integrate the rest of the pieces from libarchive into the
FreeBSD test suite (cpio, tar)

Re-branch because apparently resyncing from head has svn issues with missing
revisions from ^/user/ngie/more-tests...

#idontknowwhatevenanymore #howilearnedtogiveupsvnandacceptmydvcsoverlords

NO_MAN= has been deprecated in favor of MAN= for some time, go ahead
and finish the job. ncurses is now the only Makefile in the tree that
uses it since it wasn't a simple mechanical change, and will be
addressed in a future commit.

MFV r248590,248594:
Update libarchive to 3.1.2

Some of new features:
- support for lrzip and grzip compression
- support for writing tar v7 format
- b64encode and uuencode filters
- support for __MACOSX directory in Zip archives
- support for lzop compresion (external utility)

Update libarchive to 3.0.4

Update libarchive to 3.0.3

Some of new features:
- New readers: RAR, LHA/LZH, CAB reader, 7-Zip
- New writers: ISO9660, XAR
- Improvements to many formats, especially including ISO9660 and Zip
- Stackable write filters to write, e.g., tar.gz.uu in a single pass
- Exploit seekable input; new "seekable" Zip reader can exploit the Zip
Central Directory when it's available; the old "streamable" Zip reader
is still fully supported for cases where seeking is not possible.

Full release notes available at:
https://github.com/libarchive/libarchive/wiki/ReleaseNotes

Use contrib sources for building libarchive, tar and cpio.
Make "make test" fully operational.

MFC after: 2 weeks

Update bsdtar to 2.8.4
Use common code from lib/libarchive/libarchive_fe

Approved by: kientzle
MFC after: 2 weeks

Enable liblzma support in libarchive
Adjust dependencies for programs using libarchive
Add xz and linkage against liblzma to rescue system

Approved by: kientzle, delphij (mentor)
MFC after: 2 weeks

Fix common misspelling of hierarchy

Pointed out by: bf1783 at gmail
Approved by: np (cxgb), kientzle (tar, etc.), philip (mentor)

Build usr.bin/ with WARNS=6 by default.

Also add some missing $FreeBSD$ to keep svn happy.

Merge from libarchive.googlecode.com: Numerous Windows-specific build tweaks.

Match a comment to reduce differences with libarchive.googlecode.com.

Merge r709,r710 from libarchive.googlecode.com: More work on
Windows support.

Merge r687-689,691,693-701,720 from libarchive.googlecode.com:
Translate getdate.y into C for portability. Make the get_date()
function easier to test as well:
* Have it accept a time_t "now" to use as a reference so that test
code can verify relative time specifications against known starting
points.
* Set up default date after parsing the string so that we
can use the specified timezone (if any) instead of the local
default. Otherwise, local DST makes it almost impossible to
reliably test time specifications such as "sunday UTC"

Merger r629-631,633-646,648,654,678,681,682 from libarchive.googlecode.com:
Many changes for Windows compatibility. bsdtar_test now runs successfully
on both POSIX platforms and Windows.

Merge r368,496,625,626 from libarchive.googlecode.com: A number of
style and portability tweaks to the test harness. Most significantly,
don't use getopt().

Merge r435,r443 from libarchive.googlecode.com: Let the compiler options
determine how to read config.h.

Merge r374 from libarchive.googlecode.com: Stupid typo in open() call. <sigh>

Merge r369 from libarchive.googlecode.com: Test -s option.

Merge r278 from libarchive.googlecode.com: Reduce the number of
patterns tested here from 200 to 170, which seems to be the
most that Cygwin can handle.

Merge r273 from libarchive.googlecode.com: Use open() correctly.

Include more detailed explanation of this case, since it's pretty
subtle why it comes out the way it does. Once you realize that it
depends on the archiving order, it's also important to realize that
filesystem differences aren't going to break this case. (Some of the
other tests have had to be extensively rewritten to make them
independent of the order in which a particular filesystem returns file
entries.)

(This commit also serves to note the PR number that I accidentally
omitted from the previous commit.)

PR: bin/128562
MFC after: 30 days

Test --strip-components and fix it to actually work. Jaakko did a
good job writing this test; it exercises a lot of subtle cases. The
trickiest one is that a hardlink to something that didn't get
extracted should not itself be extracted. In some sense, this is not
the desired behavior (we'd rather restore the file), but it's the best
you can do in a single-pass restore of a tar archive.

The test here should be extended to exercise cpio and newc formats as
well, since their hardlink models are different, which will lead to
different handling of some of these edge cases.

Submitted by: Jaakko Heinonen
MFC after: 30 days

When comparing, cast to the larger size, off_t in this case.
Once we know which one is smaller, then we cast to the smaller size.

Thanks to Xin Li (delphij@)
Pointy hat: /me

Fix compile warnings building on amd64. This is modified slightly
from Jaakko's original patch: I have misgivings about the portability
of the 'z' printf modifier so opted to cast the arguments to (int)
instead.

PR: bin/128561
Submitted by: Jaakko Heinonen
MFC after: 30 days

Test handling of restores relative to symlinks.
In particular:
* tar -x -P follows symlinks to existing dirs, but not without -P
* symlinks to files are always replaced
* broken symlinks are always replaced

Minor cleanup of the -q test: Assert that stdout/stderr are empty for each
extraction.

Explain how the test_option_q test works.

Test for -q (aka --fast-read).
Fix the error uncovered by this test.

The results for test 2 here are short enough to just be included inline.
There's no need to go through the hassle of having a checked-in uuencoded
reference file for comparison.

Add some more tests to verify that "./foo" matches "foo" but "/foo" does not.

Always display the unedited pathname in -t output.
I would like to provide a way to preview the effects of pathname edits,
but pattern selection has to happen against the unedited path, so it
seems that we have to show people the unedited path to help in
designing selection patterns.

Better comment the pattern tests; adjust the filenames for the
reference files to match the corresponding source.

MFC after: 3 days

Test updates: Handling of patterns on command line, error messages.

MfP4: test harness cleanup.

MFp4: bsdtar 2.5.4b

In addition to a number of bug fixes and minor changes:
* --numeric-owner (ignore user/group names on create and extract)
* -S (sparsify files on extraction)
* -s (regex filename substitutions)
* Use new libarchive 'linkify' to get correct hardlink handling for
both old and new cpio formats
* Rework 'copy' test to be insensitive to readdir() filename ordering

Most of the credit for this work goes to Joerg Sonnenberger, who
has been duplicating features from NetBSD's 'pax' program.

New bsdtar test harness. Still rather skimpy, but a lot easier
to run and maintain than the old scripts that used to be here.