Fix bspatch heap overflow vulnerability. [SA-16:29]

Fix multiple portsnap vulnerabilities. [SA-16:30]

Fix multiple libarchive vulnerabilities. [SA-16:31]

Approved by: so

Copy stable/10@r272459 to releng/10.1 as part of
the 10.1-RELEASE process.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFV r248590,248594:
Update libarchive to 3.1.2

Some of new features:
- support for lrzip and grzip compression
- support for writing tar v7 format
- b64encode and uuencode filters
- support for __MACOSX directory in Zip archives
- support for lzop compresion (external utility)

Backport NFSv4 ACL fix from libarchive master branch.

Source:
https://github.com/libarchive/libarchive/commit/f67370d5

Obtained from: libarchive (master branch)

Update libarchive to 3.0.4

Update libarchive to 3.0.3

Some of new features:
- New readers: RAR, LHA/LZH, CAB reader, 7-Zip
- New writers: ISO9660, XAR
- Improvements to many formats, especially including ISO9660 and Zip
- Stackable write filters to write, e.g., tar.gz.uu in a single pass
- Exploit seekable input; new "seekable" Zip reader can exploit the Zip
Central Directory when it's available; the old "streamable" Zip reader
is still fully supported for cases where seeking is not possible.

Full release notes available at:
https://github.com/libarchive/libarchive/wiki/ReleaseNotes

Use contrib sources for building libarchive, tar and cpio.
Make "make test" fully operational.

MFC after: 2 weeks

Update libarchive, tar and cpio to version 2.8.5

The following additional vendor revisions are applied:

Revision 3740:
Use archive_clear_error() to clear the error markers.

Obtained from: http://code.google.com/p/libarchive
MFC after: 2 weeks

Add compatibility for ISO images created with unfixed makefs that
violated ECMA-119 (ISO9660): allow reserved4 to be 0x20 in PVD.
This allows tar to read FreeBSD distribution ISO images created
with makefs prior to NetBSD bin/45217 bugfix (up to 9.0-BETA1).

In addition, merge following important bugfixes from
libarchive's release/2.8 branch:

Revision 2812:
Merge 2811 from trunk: Don't try to verify that compression-level=0
produces larger results than the default compression, since this isn't
true for all versions of liblzma.

Revision 2817:
Merge 2814 from trunk: Fix Issue 121 (mtree parser error)
http://code.google.com/p/libarchive/issues/detail?id=121

Revision 2820:
Fix issue 119.
Change the file location check that a file location does not exceed
volume block. New one is that a file content does not exceed volume
block(end of an ISO image). It is better than previous check even
if the issue did not happen.

While reading an ISO image generated by an older version of mkisofs
utility, a file location indicates the end the ISO image if its file
size is zero and it is the last file of all files of the ISO image,
so it is possible that the location value is the same as the number
of the total block of the ISO image.

http://code.google.com/p/libarchive/issues/detail?id=119

Revision 2955:
Issue 134: Fix libarchive 2.8 crashing in archive_write_finish() when
the open has failed and we're trying to write Zip format.

http://code.google.com/p/libarchive/issues/detail?id=134

Revision 2958:
Followup on Issue 134:
1) Port test_open_failure to libarchive 2.8 branch to test
the problem reported in Issue 134.
This test also shows that archive_read_open() sometimes
fails to report open errors correctly.
2) Fix the bug in archive_read.c
3) Comment out the tests that close functions are invoked
promptly when open fails; that's fully fixed in libarchive 3.0,
but I don't think it's worth fixing here.

Revision 3484:
Use uintmax_t with %ju

Revision 3487:
Fix issue 163.
Correctly allocate enough memory for a input buffer saved.

http://code.google.com/p/libarchive/issues/detail?id=163

Revision 3542:
Merge 2516, 2536 from trunk: Allow path table offset values of
0 and 18, which are used by some ISO writers.

Reviewed by: kientzle
Approved by: re (kib)
MFC after: 3 days

- Update libarchive to 2.8.4
- Add support for extracting xar and rpm archives
- Add libarchive_fe subdir (common code for tar and cpio)

Approved by: kientzle
MFC after: 2 weeks

Don't write data into an empty "file."

In particular, this check avoids a warning when
extracting directory entries from certain GNU tar
archives that store directory contents.

MFC after: 3 days

Remove redundant WARNS?=6 overrides and inherit the WARNS setting from
the toplevel directory.

This does not change any WARNS level and survives a make universe.

Approved by: ed (co-mentor)

UU decoder. Now that libarchive can recursively taste input streams,
you can do things like this: tar xvf archive.tar.gz.uu

A raft of test changes and improvements from the Googlecode repository.
In particular, this includes tests for the new features I've merged
over the last few days.

Merge r990,r1044 from libarchive.googlecode.com:
read_support_format_raw() allows people to exploit libarchive's
automatic decompression support by simply stubbing out the
archive format handler.
The raw handler is not enabled by support_format_all(), of course.
It bids 1 on any non-empty input and always returns a single
entry named "data" with no properties set.

Merge r1061,r1062,r1063 from libarchive.googlecode.com:
Fix reading big-endian binary cpio archives, and add a test.
While I'm here, add a note about Solaris ACL extension for cpio,
which should be relatively straightforward to support.

Thanks to: Edward Napierala, who sent me a big-endian cpio archive
from a Solaris system he's been playing with.
Pointy hat: me

Merge r1053,r1055,r1056,r1057,r1065 from libarchive.googlecode.com:
* Fix parsing of POSIX.1e ACLs from Solaris tar archives
* Test the above
* Preserve the order of POSIX.1e ACL entries
* Update tests whose results depended on the order of ACL entries
* Identify NFSv4 ACLs in Solaris tar archives and warn that
they're not yet supported. (In particular, don't try to parse
them as POSIX.1e ACLs.)

Thanks to: Edward Napierala sent me some Solaris 10 tar archives to test

Merge lots of test suite updates from libarchive.googlecode.com.

Merge r658 from libarchive.googlecode.com: Only flush and close the
file if it was actually opened. Test for this case.

Merge r416 from libarchive.googlecode.com:

Restoring POSIX.1e Extended Attributes on FreeBSD, part 1

This implements the basic ability to restore extended attributes
on FreeBSD, including a test suite.

Merge r399,401,402,405,415,430,440,452,453,458,506,533,536,538,544,590
from libarchive.googlecode.com: Add a new "archive_read_disk" API
that provides the important service of reading metadata from the
disk. In particular, this will make it possible to remove all
knowledge of extended attributes, ACLs, etc, from clients such
as bsdtar and bsdcpio.

Closely related, this API also provides pluggable uid->uname
and gid->gname lookup and caching services similar to
the uname->uid and gname->gid services provided by archive_write_disk.
Remember this is also required for correct ACL management.

Documentation is still pending...

Fix ARCHIVE_EXTRACT_SPARSE handling in libarchive.
Add a test to exercise this feature.
This should fix --sparse/-S support in tar.

Thanks to: Daichi GOTO
MFC after: 1 week

Merge-from-Googlecode r419

In archive_write_disk: If archive_write_header() fails to create
the file, that's a failure and should return ARCHIVE_FAILED.
Metadata restore failures still return ARCHIVE_WARN, because
that's non-critical. Fix test_write_disk_secure test to
verify the correct return code in one case; add test_write_disk_failures
to do another very simple test of restore failure.

This should fix cpio coredumping when it tries to restore to
a write-protected directory.

Thanks to: Giorgos Keramidas
MFC after: 30 days

New tests:
* support for bzip2 file with multiple concatenated bzip2 streams
* support for bzip2 file with junk after bzip2 stream
* support for gzip file with junk after gzip stream
* "fuzz" tester randomly modifies a bunch of input files in order to try
to crash libarchive (this found an amusing hang in the ISO9660 code
when trying to read images that advertised a zero blocksize).

This test is implemented, but commented out for now:
* support for gzip file with multiple concatenated gzip streams

Add test to verify ACL restores on FreeBSD, verify kern/128203 (mtime
restore fails if ACL provided), apply patch from kern/128203, and verify fix.

PR: kern/128203
Submitted by: Udo Schweigert
MFC after: 3 days

MfP4: restore birth time data to disk + more thorough tests for
time restore to disk.

MFC after: 30 days

MfP4: Store/read birthtime data in pax format.

Submitted by: Pedro Giffuni
MFC after: 30 days

Fix a type-aliasing problem in the libarchive test suite,
restore WARNS=6.

Alphabetize the list of tests.

Some tests won't build at WARNS level 6 due to aliasing violations.
Add missing -I. so the tests will build when ${.OBJDIR} != ${.CURDIR}.
${.OBJDIR} does not need to be spelled out.

Allow the tests to build without libdmalloc.

archive.h is no longer constructed from archive.h.in,
so we can rename it and drop some no-longer-necessary
build magic from the Makefile.

Until the old archive.h.in gets renamed to archive.h in the repository,
we still need some Makefile trickery to ensure archive.h is
correctly built for the test harness.

MFp4: libarchive 2.5.4b. (Still 'b' until I get a bit more
feedback, but the 2.5 branch is shaping up nicely.)

In addition to many small bug fixes and code improvements:
* Another iteration of versioning; I think I've got it right now.
* Portability: A lot of progress on Windows support (though I'm
not committing all of the Windows support files to FreeBSD CVS)
* Explicit tracking of MBS, WCS, and UTF-8 versions of strings
in archive_entry; the archive_entry routines now correctly return
NULL only when something is unset, setting NULL properly clears
string values. Most charset conversions have been pushed down to
archive_string.
* Better handling of charset conversion failure when writing or
reading UTF-8 headers in pax archives
* archive_entry_linkify() provides multiple strategies for
hardlink matching to suit different format expectations
* More accurate bzip2 format detection
* Joerg Sonnenberger's extensive improvements to mtree support
* Rough support for self-extracting ZIP archives. Not an ideal
approach, but it works for the archives I've tried.
* New "sparsify" option in archive_write_disk converts blocks of nulls
into seeks.
* Better default behavior for the test harness; it now reports
all failures by default instead of coredumping at the first one.

GC a reference to the defunct TESTFILES variable.

A subtle point: "pax interchange format" mandates that all strings
(including pathname, gname, uname) be stored in UTF-8. This usually
doesn't cause problems on FreeBSD because the "C" locale on FreeBSD
can convert any byte to Unicode/wchar_t and from there to UTF-8. In
other locales (including the "C" locale on Linux which is really
ASCII), you can get into trouble with pathnames that cannot be
converted to UTF-8.

Libarchive's pax writer truncated pathnames and other strings at the
first nonconvertible character. (ouch!) Other archivers have worked
around this by storing unconvertible pathnames as raw binary, a
practice which has been sanctioned by the Austin group. However,
libarchive's pax reader would segfault reading headers that weren't
proper UTF-8. (ouch!) Since bsdtar defaults to pax format, this
affects bsdtar rather heavily.

To correctly support the new "hdrcharset" header that is going into
SUS and to handle conversion failures in general, libarchive's pax reader
and writer have been overhauled fairly extensively. They used to do
most of the pax header processing using wchar_t (Unicode); they now do
most of it using char so that common logic applies to either UTF-8 or
"binary" strings.

As a bonus, a number of extraneous conversions to/from wchar_t have
been eliminated, which should speed things up just a tad.

Thanks to: Bjoern Jacke for originally reporting this to me
Thanks to: Joerg Sonnenberger for noting a bad typo in my first draft of this
Thanks to: Gunnar Ritter for getting the standard fixed
MFC after: 5 days

Support for writing 'compress' format, thanks to Joerg Sonnenberger.

Portability: Eliminate the need for uudecode by incorporating
uudecode into the main test driver and invoking it just-in-time
within the various tests.

Also, incorporate a number of improvements to the main test support
code that have proven useful on other projects where I've used this
framework.

Add a test to verify compatibility with archives with
odd hardlinks. I need to extend this to test pax extended
archives with bodies attached to hardlinks and other less-common cases.

Issues with hardlinks in newc-format files prompted me to
write a new test to exercise the hardlink strategies used
by different archive formats (tar, old cpio, new cpio).
This uncovered two problems, both fixed by this commit:

1) Enforce file size when writing files to disk.

2) When restoring hardlink entries, if they have data associated, go
ahead and open the file so we can write the data.

In particular, this fixes bsdtar/bsdcpio extraction of new cpio
formats where the "original" is empty and the subsequent "hardlink"
entry actually carries the data. It also provides correct behavior
for old cpio archives where hardlinked entries have their bodies
stored multiple times in the archive; the last body should always be
the one that ends up in the final file. The new pax format also
permits (but does not require) hardlinks to carry file data; again,
the last contents should always win.

Note that with any of these, a size of zero on a hardlink simply means
that the hardlink carries no data; it does not mean that the file has
zero size. A non-zero size on a hardlink does provide the file size.

Thanks to: John Baldwin, for reminding me about this long-standing bug
and sending me a simple example archive that prompted this test case

Extensive improvements to the libarchive_test test program that
exercises and verifies the libarchive APIs:

* Improved error reporting; hexdumps are now provided for
many file/memory content differences.
* Overall status more clearly counts "tests" and "assertions"
* Reference files can now be stored on disk instead of having
to be compiled into the test program itself. A couple of
tests have been converted to this more natural structure.
* Several memory leaks corrected so that leaks within libarchive
itself can be more easily detected and diagnosed.
* New test: GNU tar compatibility
* New test: Zip compatibility
* New test: Zero-byte writes to a compressed archive entry
* New test: archive_entry_strmode() format verification
* New test: mtree reader
* New test: write/read of large (2G - 1TB) entries to tar archives
(thanks to recent performance work, this test only requires a few seconds)
* New test: detailed format verification of cpio odc and newc writers
* Many minor additions/improvements to existing tests as well.

New test suite test_read_pax_truncated probes libarchive
behavior with truncated or damaged pax archives. This
tests most of the cases covered by the recent security advisory.

Approved by: re (blanket, libarchive test suite)

New file "read_open_memory.c" is a custom variant of
archive_read_open_memory.c that tries to test border
cases. In particular, it copies over each returned block
so that formats or decompressors that read past the end
of a returned block will break.

Approved by: re (blanket, libarchive test suite)

Ouch. I partially screwed up the last commit by
enabling a test that's not ready yet. <sigh>

Pointy hat: /me
Approved by: re@

Support for writing the 'newc' cpio format, plus a minimal test harness
for the cpio formats.

Thanks to: Rudolf Marek
Approved by: re@

Read support for the new GNU tar sparse formats added in gtar 1.15 and
gtar 1.16.

Options spring cleanup:
- Add and document the KVM and KVM_SUPPORT options that
are needed for the ifmcstats(3) makefile
- Garbage collect unused variables
- Add missing inclusion of bsd.own.mk where needed

Approved by: kan (mentor)
Reviewed by: ru

libarchive 2.2.3
* "compression_program" support uses an external program
* Portability: no longer uses "struct stat" as a primary
data interchange structure internally
* Part of the above: refactor archive_entry to separate
out copy_stat() and stat() functions
* More complete tests for archive_entry
* Finish archive_entry_clone()
* Isolate major()/minor()/makedev() in archive_entry; remove
these from everywhere else.
* Bug fix: properly handle decompression look-ahead at end-of-data
* Bug fixes to 'ar' support
* Fix memory leak in ZIP reader
* Portability: better timegm() emulation in iso9660 reader
* New write_disk flags to suppress auto dir creation and not
overwrite newer files (for future cpio front-end)
* Simplify trailing-'/' fixup when writing tar and pax
* Test enhancements: fix various compiler warnings, improve
portability, add lots of new tests.
* Documentation: document new functions, first draft of
libarchive_internals.3

MFC after: 14 days
Thanks to: Joerg Sonnenberger (compression_program)
Thanks to: Kai Wang (ar)
Thanks to: Colin Percival (many small fixes)
Thanks to: Many others who sent me various patches and problem reports.

Conventionally, tar archives have always included a trailing '/'
for directories. bsdtar used to add this, but that recently got
lost somehow. So now I'm adding it back in libarchive.
The only odd part of doing this in libarchive: Adding a directory to
a tar archive and then reading it back again can yield a different name.

Add a test case to exercise some boundary conditions with
tar filenames and ensure that trailing slashes are added to
dir names only as necessary.

Thanks to: Oliver Lehmann for bringing this regression to my attention.

Enable 'ar' support; hook it up to the build and
enable it with _read_support_format_all().

New tests for system-independent ACL support.
These tests verify that archive_entry objects can store and return
ACL data and that pax format archives can read and write ACL
information. These do not (yet) test that ACL data is read or
written to disk correctly. (And hence would not have caught the
recent snafu about ACL read-from-disk being turned off.)

libarchive 2.0
* libarchive_test program exercises many of the core features
* Refactored old "read_extract" into new "archive_write_disk", which
uses archive_write methods to put entries onto disk. In particular,
you can now use archive_write_disk to create objects on disk
without having an archive available.
* Pushed some security checks from bsdtar down into libarchive, where
they can be better optimized.
* Rearchitected the logic for creating objects on disk to reduce
the number of system calls. Several common cases now use a
minimum number of system calls.
* Virtualized some internal interfaces to provide a clearer separation
of read and write handling and make it simpler to override key
methods.
* New "empty" format reader.
* Corrected return types (this ABI breakage required the "2.0" version bump)
* Many bug fixes.