Remove svn:mergeinfo carried over from stable/9.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r228797:
Use contrib sources for building libarchive, tar and cpio.
Make "make test" fully operational.

MFC r228744, r228745, r228748:

MFC r228744 [1]:
Merge vendor revision 3723:
Fixes extraction of Zip entries that use length-at-end without specifying
either the compressed or uncompressed length. In particular, fixes bsdtar
extraction of such files.

Reported by: Patrick Lamaiziere <patfbsd@davenulle.org> (freebsd-stable@)

MFC r228745:
Update libarchive, tar and cpio to version 2.8.5

The following additional vendor revisions are applied:

Revision 3740:
Use archive_clear_error() to clear the error markers.

Obtained from: http://code.google.com/p/libarchive

MFC r228748:
Sync libarchive with vendor branch release/2.8:

3730:
Fix issue 174 (Windows path names, not relevant for FreeBSD)

3734:
Merge r1989: archive_clear_error should set errno to 0.

3735:
Merge r3247 from trunk: Clear errors before returning
from archive_read_support_format_all()

3799:
Check the position before dereferencing the pointer.
This avoids dereferencing one byte past the end of a string

3824:
Merge r3823 from trunk for issue 199 (hang in iso9660 reading)

Obtained from: http://code.google.com/p/libarchive

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Fix cpio on ARM.

PR: bin/160430
Submitted by: Ian Lepore
Approved by: re (Kostik Belousov)
MFC after: 7 days

Import additional bugfix for reading and extracting makefs-created
ISO images with tar.

Vendor revision 3648 (merge of 3647):
Additional fix to issue 168 because the change of r3642 was not sufficient.
- Make sure "CL" entry appear after its "RE" entry which the "CL" entry
should be connected with.
- Give consideration to the case that the top level "RE" entry has
already been exposed outside before its tree.

Approved by: re (kib)
Obtained from: libarchive (release/2.8, svn rev 3648)
MFC after: 3 days

Import bugfix for reading and extracting of FreeBSD ISO images with tar.

Upstream revision 3645 (merge of 3642):
Change the mechanism handling a rr_moved directory,
which is Rockridge extension that can exceed the limitation of
a maximum directory depth of ISO 9660.
- Stop reading all entries at a time.
- Connect "CL" entry to "RE" entry dynamically, which "CL" and "RE"
have information to rebuild a full directory tree.
- Tweak some related tests since we use Headsort for re-ordering
entries and it cannot make a steady order when the keies of
the entries are the same.

http://code.google.com/p/libarchive/issues/detail?id=168

Reviewed by: kientzle
Approved by: re (kib)
Obtained from: libarchive (release/2.8, svn rev 3645)
MFC after: 3 days

Merge revision 3554 from libarchive's release/2.8 branch:

Partial merge of 2431 from trunk: Retry writes on EINTR.
This should fix the SIGINT handler in bsdtar.
Note: The rest of r2431 can't be merged, since it interacts
with a big write-side rearchitecture.

PR: bin/149409
Reviewed by: kientzle
Approved by: re (kib)
MFC after: 3 days

Add compatibility for ISO images created with unfixed makefs that
violated ECMA-119 (ISO9660): allow reserved4 to be 0x20 in PVD.
This allows tar to read FreeBSD distribution ISO images created
with makefs prior to NetBSD bin/45217 bugfix (up to 9.0-BETA1).

In addition, merge following important bugfixes from
libarchive's release/2.8 branch:

Revision 2812:
Merge 2811 from trunk: Don't try to verify that compression-level=0
produces larger results than the default compression, since this isn't
true for all versions of liblzma.

Revision 2817:
Merge 2814 from trunk: Fix Issue 121 (mtree parser error)
http://code.google.com/p/libarchive/issues/detail?id=121

Revision 2820:
Fix issue 119.
Change the file location check that a file location does not exceed
volume block. New one is that a file content does not exceed volume
block(end of an ISO image). It is better than previous check even
if the issue did not happen.

While reading an ISO image generated by an older version of mkisofs
utility, a file location indicates the end the ISO image if its file
size is zero and it is the last file of all files of the ISO image,
so it is possible that the location value is the same as the number
of the total block of the ISO image.

http://code.google.com/p/libarchive/issues/detail?id=119

Revision 2955:
Issue 134: Fix libarchive 2.8 crashing in archive_write_finish() when
the open has failed and we're trying to write Zip format.

http://code.google.com/p/libarchive/issues/detail?id=134

Revision 2958:
Followup on Issue 134:
1) Port test_open_failure to libarchive 2.8 branch to test
the problem reported in Issue 134.
This test also shows that archive_read_open() sometimes
fails to report open errors correctly.
2) Fix the bug in archive_read.c
3) Comment out the tests that close functions are invoked
promptly when open fails; that's fully fixed in libarchive 3.0,
but I don't think it's worth fixing here.

Revision 3484:
Use uintmax_t with %ju

Revision 3487:
Fix issue 163.
Correctly allocate enough memory for a input buffer saved.

http://code.google.com/p/libarchive/issues/detail?id=163

Revision 3542:
Merge 2516, 2536 from trunk: Allow path table offset values of
0 and 18, which are used by some ISO writers.

Reviewed by: kientzle
Approved by: re (kib)
MFC after: 3 days

- Update libarchive to 2.8.4
- Add support for extracting xar and rpm archives
- Add libarchive_fe subdir (common code for tar and cpio)

Approved by: kientzle
MFC after: 2 weeks

libarchive is mixing libmd and libcrypto -- correct to use one or the other.
[mixing the two can be quite bad -- they define the same context structures,
but with differing structure members (and sizes)]

Update the hash function support comments, and update config_freebsd.h
to match.

Approved by: kientzle

Don't write data into an empty "file."

In particular, this check avoids a warning when
extracting directory entries from certain GNU tar
archives that store directory contents.

MFC after: 3 days

If the Zip reader doesn't see a PK signature block
because there's inter-entry garbage, just scan forward
to find the next one. This allows us to handle a lot
of Zip archives that have been modified in-place.

Thanks to: Gleb Kurtsou for sending me a sample archive

Fix manpage markup.

Clarify the naming: Methods that free an object should
be called "free". Retain the old "finish" names to preserve
source compatibility for now.

mdoc: drop redundant .Pp and .LP calls

They have no effect when coming in pairs, or before .Bl/.Bd

Fix typo.

s/an struct/a struct

PR: docs/148660
Submitted by: Warren Block (wblock at wonkity dot com)
Reviewed by: Tim Kientzle
Mentored by: jkois
MFC after: 7 days

mdoc nitpicking: the title argument shall be uppercase

mdoc: consistently spell our email addresses <foo@FreeBSD.org>

Reviewed by: ru

Retry reads that fail with EINTR. This fixes a problem
with bsdtar failing on SIGINT.

Fix build.

Reorganize slightly in preparation for making lzma and bz2 support conditional.

mdoc: move CAVEATS, BUGS and SECURITY CONSIDERATIONS sections to the
bottom of the manpages and order them consistently.

GNU groff doesn't care about the ordering, and doesn't even mention
CAVEATS and SECURITY CONSIDERATIONS as common sections and where to put
them.

Found by: mdocml lint run
Reviewed by: ru

Enable liblzma support in libarchive
Adjust dependencies for programs using libarchive
Add xz and linkage against liblzma to rescue system

Approved by: kientzle, delphij (mentor)
MFC after: 2 weeks

Remove redundant WARNS?=6 overrides and inherit the WARNS setting from
the toplevel directory.

This does not change any WARNS level and survives a make universe.

Approved by: ed (co-mentor)

Fix common misspelling of hierarchy

Pointed out by: bf1783 at gmail
Approved by: np (cxgb), kientzle (tar, etc.), philip (mentor)

Add missing newline in last line of file.

Uncovered via: fromcvs vs. svn
Approved by: ed (co-mentor)

Don't try to hardlink block or character device nodes that come out
of cpio archives.

Use a simpler memory-management strategy for the file objects.
Instead of trying to reference-count them and free them as soon
as they are no longer needed, we now just keep them around and free
them all when we release the archive object. This fixes a number
of minor memory leaks, especially when reading damaged archives.

If we can't stat a file, return the correct ARCHIVE_FAILED (this entry can't
be processed any further) and a suitable error string.
In particular, this improves the error-reporting when cpio -o is
given a nonexistent filename.

Initialize the allocated 'struct zip'.

Fix a memory leak when a filter fails to initialize.

Build lib/ with WARNS=6 by default.

Similar to libexec/, do the same with lib/. Make WARNS=6 the norm and
lower it when needed.

I'm setting WARNS?=0 for secure/. It seems secure/ includes the
Makefile.inc provided by lib/. I'm not going to touch that directory.
Most of the code there is contributed anyway.

Remove some test files that are no longer used.

Reserve constants for RPM unwrapper and XAR reader.

Libarchive in FreeBSD is now synced up with 2.7.901a from
libarchive.googlecode.com

UU decoder. Now that libarchive can recursively taste input streams,
you can do things like this: tar xvf archive.tar.gz.uu

A raft of test changes and improvements from the Googlecode repository.
In particular, this includes tests for the new features I've merged
over the last few days.

Merge Michihiro NAKAJIMA's significant work on the ISO9660 reader
from googlecode:
* Support for zisofs compressed entries
* Support for relocated deep directories
* Direct calculation of link counts for accurate nlink values
even on images that lack Rockridge extensions
* Faster handling of the internal file lists.
* Better detection of ISO variants

Rationalize the support for cryptographic hashes.

Essentially eliminate problems with false hardlinks in POSIX cpio
archives by generating synthetic ino values and mapping values
from disk into the new values.

Clear extraneous error messages.

Zip write support for libarchive.

The initial implementation was developed by Anselm Strauss as part of
Google Summer of Code 2008, then completed by Joerg Sonnenberger.

If we're linking against liblzma, allow the decompressor to use up to
1<<30 bytes of memory, which is required for some streams.

Also, try to make the taster more discriminating about raw lzma streams.
The detection here is still really weak, though; please use xz instead
of lzma.

Minor style fixes plus a fix for an obscure crash on certain malformed
mtree files.

Be a little more skeptical of dev/ino matches when reading cpio files.
This eliminates some false-positives in the hardlink detection logic.

Style and portability fixes for pax writer. Mostly very routine, except for:
* Use ino64 where appropriate
* Don't pass atime or nsec to ustar formatter

Style fixes to tar reader:
For portability, prefer int64_t to off_t.
Improve numeric overflow handling when parsing.
Fix some variable types.
Eliminate some unused results.

Various style fixes to the 'newc' cpio writer:
* Warn about truncation of ino
* Use a macro to hide the somewhat obscure pad-to-multiple-of-4 calculation
* Eliminate some unused variables

Various fixes when creating objects on disk:
* Write xattrs last instead of first (required on platforms that use
system xattrs for security attributes)
* Better handling of chdir() failures
* Don't bother trying to shorten files via seek()/write()
* Fix build on systems that lack link()/symlink()/mknod()
* Prefer futimens()/utimensat() when they're present

Catch decompression failures earlier. In particular, this gives
immediate feedback if we fail to fork an external decompression program.

Various style and portability fixes, including:
* Enforce option interface can only be used before the archive is opened
* Correctly handle large skips on platforms with 32-bit off_t
* Use int64_t instead of off_t

64-bit ino support and discourage bad use of this header.

Portability.

Remove some dead assignments, fix some declarations.

Clarify an example so Linux folks won't keep getting confused by this point.

Compatibility with old systems with non-POSIX getpwuid_r/getgrgid_r.

Fix some unused variables and dead assignments.

Remove an unused variable and an unreachable statement.

Fix line endings.

Discourage external use of this header.

Remove unused variable.

O_BINARY for Windows.

Handle Zip CRC validation for uncompressed Zip archives even on
platforms that lack zlib.

Include limits.h; remove unused variable.

Use ino64 interface.

Use new ino64 interface.

New archive_file_count() utility.

Portability fixes.

Various portability fixes, plus:
* New "ino64" field.
* New UTF8 interfaces for hardlink/symlink updates

Remove an unused variable and an unnecessary increment.

Eliminate an unused assignment.

Portability: Set the file descriptor to binary mode on Win32

Discourage outside use of this internal header.

Remove dead assignment.

Portability:
* some non-FreeBSD platforms do actually define __FBSDID() in sys/cdefs.h
* Borland C <sigh>

Portability: terminate abnormally via abort() instead of segfault,
watch the return value from write(), and avoid signed arithmetic on
unsigned values.

Allow more formats to be active at a time; discourage outside use.

Enforce this as an internal-only header.

Discourage people from using this header outside of libarchive.

Compatibility: I found some more compilers that don't like 'inline' keyword.
Protection: Discourage people from using this header outside libarchive.

Sync with googlecode some changes that have no impact on FreeBSD.

Compatibility fix for some older systems with non-POSIX getgrnam_r/getpwnam_r
and a minor style fix for the hash function.

Catch certain gzip failures at close time.

Remove a dead assignment.

Remove some unused variables and dead assignments.

Update the hard-coded configuration for libarchive.

Update format manpages for libarchive.

Merge two cpio fixes from libarchive.googlecode.com:
1) Avoid an infinite loop in the header resync for certain malformed
archives.
2) Don't try to match hardlinks if the nlinks count is < 2. This
reduces the likelihood of a false hardlink match due to ino truncation.

MFC after: 7 days

Fiz /usr/bin/unzip: A bug deep in libarchive's read-ahead logic
(incorrect handling of zero-length reads before the copy buffer is
allocated) is masked by the iso9660 taster. Tar and cpio both enable
that taster so were protected from the bug; unzip is susceptible.

This both fixes the bug and updates the test harness to exercise
this case.

Submitted by: Ed Schouten diagnosed the bug and drafted a patch
MFC after: 7 days

Update tests to match r195873, which corrected how hardlinked files
on iso9660 images were returned. While I'm poking around, update
some comments around this area to try to clarify what's going on and
what still remains to be improved.

The parser for Rockridge symlinks tended to insert
extra slashes at the beginning of absolute targets.

Thanks to Jung-uk Kim for pointing this out to me.

Approved by: re (kib)

Libarchive recognizes hardlinked files on ISO images,
but returned them incorrectly, causing tar to actually
erase the resulting file while trying to restore the
link. This one-line fix corrects the hardlink descriptions
to avoid this problem.

Thanks to Jung-uk Kim for pointing this out.

Approved by: re (kib)

Bump the version of all non-symbol-versioned shared libraries in
preparation for 8.0-RELEASE. Add the previous version of those
libraries to ObsoleteFiles.inc and bump __FreeBSD_Version.

Reviewed by: kib
Approved by: re (rwatson)

Fix "tar --options=iso9660:joliet" and other uses
of format-specific options.

Include libmd and libcrypto in DPADD

Link libarchive against -lmd and -lcrypto.

Thanks to Ed Schouten for the clue.

Include the 2 byte length field for the optional "extra data"
field when computing the length of the gzip header.

Thanks to Dag-Erling for pointing me to the OpenSSH tarballs,
which are the first files I've seen that actually used this field.

Eliminate duplicate error messages from "tar c".

Reported by: pav@

Partially revert r191171, which went too far in trying
to eliminate some duplicated code. In particular,
archive_read_open_filename() has different close
handling than archive_read_open_fd(), so delegating
the former to the latter in the degenerate case
(a NULL filename is treated as stdin) broke reading
from pipelines. In particular, this fixes occasional
port failures that were seen when using "gunzip | tar"
pipelines under /bin/csh.

Thanks to Alexey Shuvaev for reporting this failure and
patiently helping me to track down the cause.

Document the liblzma support.
Unfortunately, liblzma itself is GPLed, so unlikely to become part of
the FreeBSD base system.
However, the core lzma compression/decompression code is public
domain, so it should be feasible for someone to create a compatible
library without the GPL strings.

Symlink some additional man page entries.

Merge r991 from libarchive.googlecode.com: Various updates
to archive_read.3 manpage, including documentation for the
new "raw" handler.

Merge r990,r1044 from libarchive.googlecode.com:
read_support_format_raw() allows people to exploit libarchive's
automatic decompression support by simply stubbing out the
archive format handler.
The raw handler is not enabled by support_format_all(), of course.
It bids 1 on any non-empty input and always returns a single
entry named "data" with no properties set.

Merge r1061,r1062,r1063 from libarchive.googlecode.com:
Fix reading big-endian binary cpio archives, and add a test.
While I'm here, add a note about Solaris ACL extension for cpio,
which should be relatively straightforward to support.

Thanks to: Edward Napierala, who sent me a big-endian cpio archive
from a Solaris system he's been playing with.
Pointy hat: me

Merge r1034 from libarchive.googlecode.com:
Put a much larger file on the reference ISO for this test.

Merge r1058 from libarchive.googlecode.com: Require that each
test source file has exactly one DEFINE_TEST().

ino_t varies across platforms; casting (int) here avoids
various pointless complaints.

Merge r1032 from libarchive.googlecode.com:
Make test_fuzz a bit more sensitive by actually reading the body
of each entry instead of skipping it.
While I'm here, move the "UnsupportedCompress" macro into the
only file that still uses it.

Merge r1054,r1060 from libarchive.googlecode.com:
* assertEqualMem() now takes void * arguments
* Be a little smarter about what we hexdump when assertEqualMem() fails

Merge r1021 from libarchive.googlecode.com:
If we know it's a socket, say so.

Merge r1052,r1055 from libarchive.googlecode.com:
Clear the error flag on entry to a few more API functions.

Merge r1053,r1055,r1056,r1057,r1065 from libarchive.googlecode.com:
* Fix parsing of POSIX.1e ACLs from Solaris tar archives
* Test the above
* Preserve the order of POSIX.1e ACL entries
* Update tests whose results depended on the order of ACL entries
* Identify NFSv4 ACLs in Solaris tar archives and warn that
they're not yet supported. (In particular, don't try to parse
them as POSIX.1e ACLs.)

Thanks to: Edward Napierala sent me some Solaris 10 tar archives to test

Merge r988,r1064 from libarchive.googlecode.com:
* Split whiny skip function to create a new best-effort skip_lenient()
* Correctly increment the top-level file position only for the top filter
* Simulate skip by reading against the current filter, not the top filter

The latter two bugs aren't currently visible because no existing
filter delegates skip operations.

Reading an mtree file is supposed to provide
access to the file data (if the file exists on
disk). This was broken for the first regular
file; fix it and add a test so it won't break again.

In particular, this fixes the following idiom for creating
a tar archive in which every file is owned by root:

tar cf - --format=mtree . \
| sed -e 's/uname=[a-z]*/uname=root/' -e 's/uid=[0-9]*/uid=0/' \
| tar cf - @-

Various improvements to the tar.5 manpage, including
descriptions of the GNU tar "posix-style" sparse format,
clarification of the Solaris tar ACL storage,
and a few comments about Mac OS X tar's resource storage.

Exit with ARCHIVE_FATAL if the ISO image is truncated.

Remove an unused variable.

Thanks to: Christoph Mallon

Rely on OpenSSL bits only if we're building a system with OpenSSL.
Also, adjust the MD5 calls to rely on libmd instead of libcrypto,
so we keep MD5 support even in the !OpenSSL case.

Correct and update the manpage to include
more details about some of the formats and
to briefly describe the mtree writing capability.

Disabling the crypto bits should quiet tinderbox while I
track down the library dependencies that are screwing up the /rescue build.

Publicize a handful of new functions and bump the version number
to indicate that this is synchronized with the recent libarchive 2.7.0
release.

Merge lots of test suite updates from libarchive.googlecode.com.

FreeBSD has a lot of crypto functions used by the recent mtree writer updates.

Ensure that the option setters return OK (option used) even
for options that don't change the list of keywords.

Fix a memory leak: Release the current list of /set keywords
even when we exit on an error.

Document the new read options interface and the new read_header2() call.

Dynamically size the buffer we pass to getgrgid_r() and getpwuid_r().
Keep the buffer in the cache object so we don't have to keep doing this.

Don't match an empty file on a read error.

When pulling metadata from disk, lookup the user and group name at the same time.

LZW bugfix: when we hit end-of-file, return an invalid code.

Minor fix: some platforms require both inttypes.h and stdint.h.

Implement command-line fallbacks for gzip and bzip2 decompression as well.
Not an issue for FreeBSD, since the base system has the necessary libraries.
Since all decompressors are always available now, we can unconditionally
enable them in archive_read_support_compression_all().

Merge new xz/lzma support from libarchive.googlecode.com.
Since FreeBSD doesn't have liblzma in the base system, the
read side will always fall back to the unxz/unlzma commands for now.
(Which will in turn fail if those commands are not currently
installed.) The write side does not yet have a fallback, so
that will just fail.

Don't use the open callback, which is deprecated (because it's
never necessary). Also, simplify just a tad by delegating
to read_open_fd() when we know the file descriptor, instead
of duplicating that logic.

Accept empty options, add a new read_next_header2() which is more
efficient for some uses.

Properly clone and free the recently-added "sourcepath" field.

Use thread-safe getgrnam_r() and getpwnam_r(); dynamically size
the buffer used by this.

Fix a minor memory leak.

Merge from libarchive.googlecode.com: Numerous fixes to the
write options handling, including documentation.

Merge from libarchive.googlecode.com: If we're
given an empty filename, just invoke write_open_fd()
instead of re-implementing the code to use stdout.

Re-enable backing up extended attributes, as the ZFS bug this
triggered seems to have been fixed by John Baldwin's commit
r189967.

Merge from libarchive.googlecode.com r791, r879, r884, r948: Various
fixes to read_support_compression_program. In particular, failure of
the external program is detected a lot earlier, which gives much more
reasonable error handling.

Thanks to Christoph Mallon for pointing out the dead variable here.
Also, rework this a little to make the logic excruciatingly clear.

Merge from libarchive.googlecode.com: Mostly a bunch of
corrections to the Windows support to reconcile differences
between Visual Studio and Cygwin. Includes parts of
revisions 757, 774, 787, 815, 817, 819, 820, 844, and 886.

Of particular note, r886 overhauled the UTF-8/Unicode conversions to
work correctly regardless of whether the local system uses 16-bit
or 32-bit wchar_t. (I assume that systems with 16-bit wchar_t
use UTF-16 and those with 32-bit wchar_t use UCS-4.) This revision
also added a preference for wcrtomb() (which is thread-safe) on
platforms that support it.

Merge from libarchive.googlecode.com r756,r761:
Document the new archive_read_disk API.

Merge from libarchive.googlecode.com:
r751: Change __archive_strncat() to use a void * source, which reduces
the amount of casting needed to use this with "char", "signed char"
and "unsigned char".
r752: Use additions instead of multiplications when growing buffer;
faster and less chance of overflow.

Merge r881 from libarchive.googlecode.com: The "empty" format
should not be recognized if there is a read error.

Temporary hack to unbreak user/group lookups;
use a larger buffer for getpwuid_r/getgrgid_r.
This needs to be dynamically sized.

Hack: *Temporarily* disable reading extended attributes from disk, as
it seems to be badly broken on ZFS.

Set version to 2.6.901a to indicate this now matches
libarchive.googlecode.com r745. (Except for the lzma/xz support,
which needs a little more attention before it can be merged.)

Merge a bunch of changes through r722 from libarchive.googlecode.com:
mtree writer now supports a variety of checksum keys; it also provides
option hooks to set what keys get written.

Merge r718 from libarchive.googlecode.com: Some additional
tests of restoring files to disk with unusual characters, specifically
to exercise Windows issues.

Fix spelling.

Merge r348 from libarchive.googlecode.com: Suppress testing invalid
conversions if there aren't any. In particular, Cygwin's "C" locale
has no invalid inputs for wctomb().

Merge r596,r690 from libarchive.googlecode.com: Minor style and compile
warning fixes for test_read_pax_truncated.c.

Merge r335,653,676 from libarchive.googlecode.com: Instead of
conditioning tests on HAVE_ZLIB, etc, just ask libarchive for the
service and handle the failure coming back from libarchive. This
gives us better test coverage of common client usage where clients
simply try to use libarchive services and handle the errors coming
back instead of trying to second-guess which libarchive services are
compiled in.

Merge r280,281,496,595,675,712 from libarchive.googlecode.com: Various
test improvements, including some work on Windows compatibility and an
extra check to verify that no test leaves open file descriptors
around.

Merge r540 from libarchive.googlecode.com: Interix doesn't have inttypes.h
either. <sigh>

Merge r723 from libarchive.googlecode.com: Don't try to restore
owner or SUID bits on Windows; just ignore them.

Merge r511,r513,r607 from libarchive.googlecode.com: Mtree reader
tweaks: Support nanosecond timestamps, handle attributes broken
across multiple lines.

Merge r585,r669 from libarchive.googlecode.com: If zlib is unavailable,
use external "gunzip" instead. With this in place, we can unconditionally
enable gzip read support.

Merge r550,584,587,609,647,674 from libarchive.googlecode.com:
Refactor the read_compression_program to add two new abilities:
* Public API: You can now include a signature string when you
register a program; the program will run only on input that
matches the signature string.
* Internal API: You can use the init() function to instantiate
an external program as part of a filter pipeline. This
can be used for graceful fallback (if zlib is unavailable, use
external gzip instead) and to use external programs with
bidders that are more sophisticated than a static signature check.

Merge r591 from libarchive.googlecode.com: signed/unsigned fixes.

Merge r558,567,569,571,581,582,583,598 from libarchive.googlecode.com:
Support Joliet extensions. This currently ignores Rockridge extensions
if both exist on the same disk unless the '!joliet' option is provided.
e.g.: tar -xvf example.iso --options '!joliet'
Thanks to: Andreas Henriksson

Merge r658 from libarchive.googlecode.com: Only flush and close the
file if it was actually opened. Test for this case.

Merge r714,r715 from libarchive.googlecode.com: Fix Debian bug #516577.
Don't crash if client does not provide a skip function. Extend one
of the test cases to use archive_read_open2() with only a read callback.

Merge r608 from libarchive.googlecode.com: Clear a newly-allocated
bidder object.

Merge r564,r566 from libarchive.googlecode.com: Fix segfault when
specifying an option and the current format doesn't have an
options handler.

Merge the rest of r453 from libarchive.googlecode.com: Test using
libarchive APIs to read extended attributes from disk on FreeBSD.

Minor whitespace adjustment to reduce diffs with libarchive.googlecode.com.

Merge r552,r559 from libarchive.googlecode.com: Support high-resolution
timestamps on Tru64, AIX, and GNU Hurd. Thanks to Björn Jacke.

Merge r283,r529 from libarchive.googlecode.com: Fix ext2_fs.h includes
for Linux.

Merge r659 from libarchive.googlecode.com: Correctly report "none"
as the compression name when no other read filter bid. Add some
assertions to various tests to verify that read filters are properly
setting the textual name as well as the compression code.

Merge r668 from libarchive.googlecode.com: Style correction to the
'ar' reader: Don't redefine 'isdigit' and don't create a macro that's
only used once.

Merge r541,r542 from libarchive.googlecode.com: Two sign mismatches
in the Zip reader.

Merge r491,493,500,507,510,530,543 from libarchive.googlecode.com:
This implements the new generic options framework that provides a way
to override format- and compression-specific parameters.

Merge r448 from libarchive.googlecode.com: Suppress testing write_disk
failures on Windows for now. Someday this will be revisited.

Merge r403,702,721 from libarchive.googlecode.com: Handle odd
pathnames on Windows by mapping '\\' to '/' and converting
illegal characters to '_'.

Merge r505 from libarchive.googlecode.com: Fix %ju support. Simplify
the code here a bit by making the int formatting functions static to
archive_string_sprintf.c, which is the only place this has ever been
used.

Merge r457 from libarchive.googlecode.com: Stop appending strerror()
information to error strings. This caused a lot of unnecessary
duplication in error messages; in particular, there are a few cases
where error messages get copied from one archive object to another
and this would cause the strerror() info to get appended each time.

Merge r420,r494 from libarchive.googlecode.com: Prettify the test
harness a bit: remove a dead comment, tweak the wording of the
summary report.

Merge r435,r443 from libarchive.googlecode.com: Read config files from
include path; this makes it easier to support multiple build frameworks.

Merge r416 from libarchive.googlecode.com:

Restoring POSIX.1e Extended Attributes on FreeBSD, part 1

This implements the basic ability to restore extended attributes
on FreeBSD, including a test suite.

Merge r407,r508 from libarchive.googlecode.com: Correctly handle
Zip entries that are zero length but stored with deflate. This
is arguably a silly thing to do (deflating a zero-length file actually
makes it bigger) but apparently quite a few Zip writers do this.
This was broken in two places: archive_write_disk disliked being asked
to write data to zero-length files (even if the write was zero-length)
and zip_read_file_header tripped over itself when non-regular files
had compressed bodies.

Merge r399,401,402,405,415,430,440,452,453,458,506,533,536,538,544,590
from libarchive.googlecode.com: Add a new "archive_read_disk" API
that provides the important service of reading metadata from the
disk. In particular, this will make it possible to remove all
knowledge of extended attributes, ACLs, etc, from clients such
as bsdtar and bsdcpio.

Closely related, this API also provides pluggable uid->uname
and gid->gname lookup and caching services similar to
the uname->uid and gname->gid services provided by archive_write_disk.
Remember this is also required for correct ACL management.

Documentation is still pending...

Merge r398 from libarchive.googlecode.com: Check that bidder
object was allocated before we try to use it.

Merge r394,r396 from libarchive.googlecode.com: Plug some memory
leaks in the ACL test, correctly mark that FreeBSD has acl_get_perm_np().

Merge r390,r391,r392,r397 from libarchive.googlecode.com: Virtualize
"close" and "finish" across both read and write interfaces.
(Someday, "finish" should be renamed to "free" to better reflect
what it actually does...)

Merge r389 from libarchive.googlecode.com: Fix a memory
leak in ISO9660 handler structure whenever a file entry
has a nonsensical CE offset.

Merge r386,r395,r451 from libarchive.googlecode.com: On Windows, break
into the debugger on test setup failures (otherwise, the console window
just goes away and you can't see what went wrong). On all platforms,
clean up a stray buffer before exiting.

Argh. r189389 was supposed to include r539 from libarchive.googlecode.com
but those compile fixes somehow got lost. This should fix the build.

Merge r551,r561 from libarchive.googlecode.com: Update gzip read filter
to fully take advantage of the new peek/consume I/O support.
In particular, this now properly handles concatenated gzip streams.

Merge r364, r378, r379, r393, and r539 from libarchive.googlecode.com:
This is the last phase of the "big decompression refactor" that
puts a lazy reblocking layer between each pair of read filters.
I've also changed the terminology for this area---the two kinds
of objects are now called "read filters" and "read filter bidders"---and
moved ownership of these objects to the archive_read core.

This greatly simplifies implementing new read filters, which
can now use peek/consume I/O semantics both for bidding (arbitrary
look-ahead!) and for reading streams (look-ahead simplifies handling
concatenated streams, for instance).

The first merge here is the overhaul proper; the remainder are small
fixes to correct errors in the initial implementation.

Correct r189383, which mis-merged a change from libarchive.googlecode.com.

Merge r356 and r358 from libarchive.googlecode.com: Remove a Windows
special case from archive_entry.c, add one to archive_check_magic.c.

Merge r357 from libarchive.googlecode.com: bzip2 compression
support can always be enabled even if bzlib doesn't exist on
this platform; don't give up until we fail to open the file.

Merge r362 from libarchive.googlecode.com: Minor fix to
custom argument parser.

Merge r342 from libarchive.googlecode.com: Remove some
Windows special casing.

Merge r344 from libarchive.googlecode.com: __LA_INT64_T and __LA_SSIZE_T
are part of the public API and therefore need to be exposed. This is
ugly; I'd like to find a better solution for this.

Merge r341,r345,r346,347 from libarchive.googlecode.com: Style
fixes to test harness and a few extra guards to detect tests
that can't succeed on certain platforms.

Merge r340 from libarchive.googlecode.com: If zlib/bzlib aren't available,
we can still detect gzip/bzip2 compressed streams, we just can't
decompress them.

Merge r294:337,r348:350 from libarchive.googlecode.com: A lot
of work to make libarchive work on Windows.

Merge r294 from libarchive.googlecode.com: Skip testing for
locale-based failures on systems where the "C" locale is so permissive
that it cannot possibly fail. In particular, this fixes a test
problem on Cygwin.

Merge r282 from libarchive.googlecode.com: Close multiple filters
by walking the filter list in archive_read_close().

Merge r272 from libarchive.googlecode.com: Fix building on MSVC6.

Permit ` as a pad character in the filename table.
This seems to fix the devel/zziplib port, which distributes
its man pages in an ar archive.

Fix ARCHIVE_EXTRACT_SPARSE handling in libarchive.
Add a test to exercise this feature.
This should fix --sparse/-S support in tar.

Thanks to: Daichi GOTO
MFC after: 1 week

Merge-from-Googlecode r419

In archive_write_disk: If archive_write_header() fails to create
the file, that's a failure and should return ARCHIVE_FAILED.
Metadata restore failures still return ARCHIVE_WARN, because
that's non-critical. Fix test_write_disk_secure test to
verify the correct return code in one case; add test_write_disk_failures
to do another very simple test of restore failure.

This should fix cpio coredumping when it tries to restore to
a write-protected directory.

Thanks to: Giorgos Keramidas
MFC after: 30 days

"The first part is just to give more info, the latter part fixes
an error to read files past the 32bit byte offset, for instance
on DVDs."

Submitted by: phk@
MFC after: 10 days

Don't try to read the next Gzip header after we reach the
end of the compressed stream. This is desirable behavior,
but the implementation here is very broken and causes strange
problems, so disable it for now.

Thanks to Simon L. Nielsen for reporting this problem.

If conversion from UTF8 fails, don't mark Unicode text as available.

Submitted by: Michihiro NAKAJIMA
MFC after: 30 days

Teach get_refdir() about FreeBSD's /usr/obj convention.
In development, I run libarchive_test frequently by hand
and it gets tedious having to specify a suitable -r path
all of the time.

Fill in a missing const

Exit earlier on failure in this test.

Once the test has failed, exit before the segfault.

Update the ar write test to give more detailed information about failures.

Clarify an 'ar' error message.

Obey the TMPDIR, TMP, TEMP, or TEMPDIR environment variables
when choosing a scratch directory for the tests. Fallback
to "/tmp", of course.

New tests:
* support for bzip2 file with multiple concatenated bzip2 streams
* support for bzip2 file with junk after bzip2 stream
* support for gzip file with junk after gzip stream
* "fuzz" tester randomly modifies a bunch of input files in order to try
to crash libarchive (this found an amusing hang in the ISO9660 code
when trying to read images that advertised a zero blocksize).

This test is implemented, but commented out for now:
* support for gzip file with multiple concatenated gzip streams

Strip ";1" and trailing "." from ISO9660 entries.
This seems a better match for people's expectations.

General improvements to Rockridge parsing and ISO9660 format detection.

Conditionalize a bunch of debugging messages; this also
eliminates what should be the only remaining stdio dependency.

MfP4: Big read filter refactoring.

This is an attempt to eliminate a lot of redundant
code from the read ("decompression") filters by
changing them to juggle arbitrary-sized blocks
and consolidate reblocking code at a single point
in archive_read.c.

Along the way, I've changed the internal read/consume
API used by the format handlers to a slightly
different style originally suggested by des@. It
does seem to simplify a lot of common cases.

The most dramatic change is, of course, to
archive_read_support_compression_none(), which
has just evaporated into a no-op as the blocking
code this used to hold has all been moved up
a level.

There's at least one more big round of refactoring
yet to come before the individual filters are as
straightforward as I think they should be...

Style: tabs after #define

Use GCC-specific dead function markers only when using GCC.

Structure declarations can't be dllexported.

Style fixes:
* Wrap long declarations to fit 80 chars
* #undef macros that shouldn't be exported
* Organize the version-dependent conditionals a
bit more consistently

Speculative:
* libarchive 3.0 will (eventually) use int64_t
instead of off_t. This is an attempt to avoid
some the headaches caused by Linux LFS. (I'll
still have to do ugly things for the struct stat
references in archive_entry.h, of course.)

A couple of portability fixes from Joerg Sonnenberger

Recover after test failure.

Style: skipping() function correctly counts this as a skipped test.

Style improvements.

Better reporting of test failures.

Recover from a test failure here.

New internal archive_string_concat utility function.

Minor portability: Declare 'r' only within the conditional
block where it's used; S_ISLNK is more portable than S_ISLINK.

Windows build now uses PLATFORM_CONFIG_H macro, so we can eliminate the
special case for it.

2 is more portable than STDERR_FILENO

When running on a filesystem that lacks ACL support,
just SKIP the test, don't report a test failure.

Correct the documented declaration of the archive_write_callback to
match the code.

PR: docs/128089
Submitted by: Mel
MFC after: 3 days

Move va_end() up

If we don't know the file size, leave the file size unset.
If it's not a regular file, don't return any data, even if the size is unknown.

Update the Zip test with a hand-tweaked Zip archive that has a
directory (with length-at-end set), a regular file without
length-at-end set, and a regular file with length-at-end set and a bad
CRC. Update the test code to verify that the file size is unset
for the regular file with length-at-end.

MFC after: 7 days

Add test to verify ACL restores on FreeBSD, verify kern/128203 (mtime
restore fails if ACL provided), apply patch from kern/128203, and verify fix.

PR: kern/128203
Submitted by: Udo Schweigert
MFC after: 3 days

Style: Mark private data 'static', correct an old comment.

MFC after: 3 days

Clean out some empty mergeinfo records, presumably by people doing local
cp/mv operations. The full repo-relative URL should be specified for the
source in these cases.

Verify high-res birthtime support on FreeBSD.

MFC after: 30 days

MfP4: restore birth time data to disk + more thorough tests for
time restore to disk.

MFC after: 30 days

MfP4: Store/read birthtime data in pax format.

Submitted by: Pedro Giffuni
MFC after: 30 days

MfP4: Support for storing birthtime in archive_entry objects.

Submitted by: Pedro Giffuni
MFC after: 30 days

MfP4: Generalize int64_t in the public headers to support compilers
which have a 64-bit integer type but don't provide standard C99 definitions.

Trying to distinguish different "versions" of mtree files was a silly idea.

Submitted by: Joerg Sonnenberger

Choose a flag-handling strategy based on platform capabilities
rather than on platform.

Portability: Don't use symbolic constants when the values are more
standard than the names are. Remove some trailing whitespace.

Portability: Not everyone is lucky enough to have ftruncate()

Portability: Support platforms that lack (struct stat).st_blksize or SSIZE_MAX
Of course, FreeBSD has both.

Windows compatibility: The stub replacement for compression_program
(which always returns an error when invoked) needs its prototype.

Fix a type-aliasing problem in the libarchive test suite,
restore WARNS=6.

Correct a leaking archive_entry in the test suite.

Correct the comments here.

PR: bin/126849
MFC after: 3 days

When restoring a directory, allow symlinks to be followed. The full
logic here gets a little complex, but the net effect is that the
SECURE_SYMLINKS flag will prevent us from ever following a symlink.
Without it, we'll only follow symlinks to dirs. bsdtar specifies
SECURE_SYMLINKS by default, suppresses it for -P.

I've also beefed up the write_disk_secure test to verify this
behavior.

PR: bin/126849

The link resolver now unsets the size to mark a hardlink
with no body instead of setting it to zero.

MfP4: Handle entries with unset size properly: Regular files with
unspecified size are "unlimited" (required by Zip reader, which
sometimes does not know the uncompressed size of an entry until it
gets to the end). Also, hardlinks with unspecified (or zero) size do
not overwrite the data on disk nor do they set metadata. This is
compatible with GNU tar and NetBSD pax behavior.

MfP4: Use the set/unset tracking to determine when mtime or atime
is unavailable. Use start_time as a substitute when one is missing;
if both are missing, skip the call to {f,l,}utime{s,}() entirely.

Markup fix.

MFC after: 3 days

Style: Use ARCHIVE_VERSION_NUMBER to conditionalize tests on
the particular libarchive version being tested instead of the
deprecated ARCHIVE_API_VERSION and ARCHIVE_VERSION_STAMP macros.

MfP4: set/unset tracking for atime, ctime, mtime, and size fields.
This generalizes the existing set/unset tracking for hardlink/symlink
fields and extends it to cover non-string fields. Eventually, this
will be further extended to cover most fields.

In particular, this is needed to correctly detect when time fields
are missing (for example, reading ustar archives doesn't set atime or
ctime) for proper time restore and is helpful when trying to determine
whether to overwrite data when restoring hardlinks.

This commit updates the tests but not the docs.

Sort the list of format names.

Use the "start_time" (time at which the restore process began)
to fill in a missing atime instead of substituting mtime.

PR: bin/124915

New mtree writer, thanks to Joerg Sonnenberger.

Obtained from: Joerg Sonnenberger

Joerg Sonnenberger's improvements to the shar writer.
Significant performance improvements, better quoting of file names, etc.

Make libarchive_test a little smarter about trying to find
its reference files when you don't specify -r. It now checks
a couple of likely nearby directories to see if any of them
have a particular known file.

When atime wasn't specified, it should get set == mtime.
This is much stronger than just asserting that it's not zero.

If no atime was specified (for example, when extracting from ustar
archives), set atime == mtime. Before this, atime would get restored
to 0.

Alphabetize the list of tests.

Update the total archive byte counters when writing entries to disk using
archive_write_disk.
Update cpio to use this to emit block counts in -p mode.
Update cpio tests to verify these block counts.

Remove the stillborn attempt to cleanup tests as well as the build dir.

svn:ignore built files

Fail copying path/.. only if SECURE_NODOTDOT was specified.
Since we already warn for any '..' elements in that case,
the extra "lastdotdot" tracking turns out to be unnecessary.

PR: bin/124924

Fix the cpio bug tested for by r182092 of usr.bin/cpio/test.
Since various 'find' incantations can emit container directories
in various orders, we cannot refuse to update a dir because it's
apparently the same age.

MFC after: 3 days

Fix up test suite so it works again with the libarchive 1.x API.
In particular, FreeBSD 6 still uses the libarchive 1.x API and
this correction will permit MFCing new libarchive features back
to FreeBSD 6.

Correct test_tar_large so it works correctly with the libarchive 1.x API.

Explain why two apparently-identical archive_read_data() calls
really should have different return values.

MfP4: Joerg Sonnenberger's fixes to make 'ar' and 'pax' writers
robust against multiple calls to their destroy() functions.

Mark functions as __dead2 in order to help the LLVM static checker
understand which code paths aren't possible.

This commit eliminates 117 false positive bug reports of the form
"allocate memory; error out if pointer is NULL; use pointer".

Garbage collect a variable which is assigned a value once but otherwise
is never used.

Found by: LLVM/Clang Static Analyzer

libarchive in FreeBSD-CURRENT is now synched with libarchive-portable 2.5.5.

Remove a couple of capability definitions that are never used.

When ARCHIVE_EXTRACT_PERM is requested (e.g., by "tar -p"), always
schedule a chmod() fixup for directories. In particular, this fixes
sgid handling on systems where the sgid bit is inherited from the
parent directory (which means that the actual mode of the dir
does not match the mode used in the mkdir() system call.

It may be possible to tighten this condition a bit. In
working through this, I also found a few other places where
it looks like we can avoid a redundant syscall or two. I've
commented those here but not yet tried to address them.

Don't use the deprecated linux/ext2_fs.h include file.

Exclude .cvsignore files from $FreeBSD$ checking

Add $FreeBSD$ to the uuencoded test files that didn't have them already.

More explicit error message.

Add missing $FreeBSD$

Some tests won't build at WARNS level 6 due to aliasing violations.
Add missing -I. so the tests will build when ${.OBJDIR} != ${.CURDIR}.
${.OBJDIR} does not need to be spelled out.

Add regression test for CRC32 check. The test file has been modified to
include an invalid checksum for file2.

Approved by: kientzle

Implement CRC32 verification. Note that you have to read until EOF to
trigger the check.

Requested by: ache
Approved by: kientzle

Allow the tests to build without libdmalloc.

Split out the reference zip file for ease of maintenance.

Refactor one of the ISO extraction tests: Move the reference
file into a separate file (instead of embedding it in the C code)
and use later timestamps (timestamps too close to the Epoch fail
predictably on systems that lack timegm(), whose mktime() doesn't
support dates before the Epoch and which are running in timezones
with negative offsets from GMT). The goal here is to test the ISO
extraction, not the local platform's time support.

MfP4: Joerg Sonnenberg's extensions to the mtree support for more
complete quoting. In particular, this handles escaped newlines
and common C-style escapes.

A security-check failure here should be ARCHIVE_FAILED (cannot continue this
operation) and not ARCHIVE_WARN, since we don't actually open the file.
Both bsdtar and bsdcpio will try to copy file contents after an ARCHIVE_WARN,
which will fail loudly.

Rework the my_mbtowc_utf8() support function to fully match the mbtowc()
calling convention, not the mbrtowc() convention.

Since wctomb() returns int, temporaries should be int, not size_t.

A number of minor corrections to the support for external compression
programs:
* Support platforms that have fork() but not vfork()
* Don't write(), select(), or poll() against closed file descriptors

Be a little more careful about closing file descriptors that may not exist.

MfP4: test harness improvements.

Fix a bad cast.

Fix reading TOC from zip archives with unsupported
compression. We can't read the body, but we
shouldn't try to skip the body twice.

Mark hardlinks that lack any other interesting filetype information
with an 'h'.

Portability: We can get away with the older and better-supported
wctomb() here; we don't need wcrtomb(). In particular, this fixes
libarchive building on FreeBSD 4.

Portability: wchar_t is defined in stdlib.h on some systems.

archive.h is no longer constructed from archive.h.in,
so we can rename it and drop some no-longer-necessary
build magic from the Makefile.

Fix the new generic link resolver in libarchive to never match
dirs as hardlinks. In particular, this fixes some recent ports
build failures.

Thanks to: Kris Kennaway

Don't store a pointer in "next" if we're never going to use it.

Fix typo in checking results of strchr.

Found by: Coverity Prevent

Minor code hardening: Verify the final bytes of the string
are actually accessible before trying to use them.

Until the old archive.h.in gets renamed to archive.h in the repository,
we still need some Makefile trickery to ensure archive.h is
correctly built for the test harness.

MFp4: libarchive 2.5.4b. (Still 'b' until I get a bit more
feedback, but the 2.5 branch is shaping up nicely.)

In addition to many small bug fixes and code improvements:
* Another iteration of versioning; I think I've got it right now.
* Portability: A lot of progress on Windows support (though I'm
not committing all of the Windows support files to FreeBSD CVS)
* Explicit tracking of MBS, WCS, and UTF-8 versions of strings
in archive_entry; the archive_entry routines now correctly return
NULL only when something is unset, setting NULL properly clears
string values. Most charset conversions have been pushed down to
archive_string.
* Better handling of charset conversion failure when writing or
reading UTF-8 headers in pax archives
* archive_entry_linkify() provides multiple strategies for
hardlink matching to suit different format expectations
* More accurate bzip2 format detection
* Joerg Sonnenberger's extensive improvements to mtree support
* Rough support for self-extracting ZIP archives. Not an ideal
approach, but it works for the archives I've tried.
* New "sparsify" option in archive_write_disk converts blocks of nulls
into seeks.
* Better default behavior for the test harness; it now reports
all failures by default instead of coredumping at the first one.

Check that lseek(2) succeeds and puts us where we expect. [1]

While we're here, fix a long-standing bug in the handling of write(2)
errors: The API changed from "return # of bytes written" to "return
status code" almost 4 years ago, so instead of returning (-1) we need
to return ARCHIVE_FATAL.

Found by: Coverity Prevent [1]

We only use the string "?rwxrwxrwx " once, so inline it rather than
declaring a variable which points to it. Aside from eliminating a
line of code and one level of unnecessary indirection, this eliminates
a false positive in Coverity.

Return ARCHIVE_FATAL if we can't allocate memory instead of going ahead and
dereferencing NULL.

Found by: Coverity Prevent

Include an extra byte for the trailing NUL. <sigh>

Pointy hat: Me

Add MLINK for archive_write_close.

Approved by: jkoshy(mentor), kientzle

Update a comment: the format bid only runs once per archive; it no
longer runs once per entry.

Free up the entry objects allocated during this test.

Release the buffers used for exercising the compress code.

Remove the duplicate "archive_format" and "archive_format_name" fields
from the private archive_write structure and fix up all writers to use
the format fields in the base "archive" structure. This error made it
impossible to query the format after setting up a writer because the
write format was stored in an inaccessible place.

Correct a sign mismatch that only showed up on 64-bit systems.

Pointy hat: me

Refactor the mtree code a bit to make the layering clearer: Each
"file" is described by multiple "lines" each possibly containing
multiple "keywords." Incorporate some additions from Joerg Sonnenberger
to handle linked files and correctly deal with backing files on disk.

FreeBSD does have fstat().
Correct the nasty typo this uncovers.

Testability is more important than standards conformance.
Disable the use of PaxHeader.<pid> for the fake pax extension pathname
until I can make the name here settable. Otherwise, tests that try
to compare output to static pre-generated reference files break.

Ignore a few more common files.

Resolve a minor nit in SUS compliance by including the PID in the
fake directory name used for pax extended headers.

GC a reference to the defunct TESTFILES variable.

A subtle point: "pax interchange format" mandates that all strings
(including pathname, gname, uname) be stored in UTF-8. This usually
doesn't cause problems on FreeBSD because the "C" locale on FreeBSD
can convert any byte to Unicode/wchar_t and from there to UTF-8. In
other locales (including the "C" locale on Linux which is really
ASCII), you can get into trouble with pathnames that cannot be
converted to UTF-8.

Libarchive's pax writer truncated pathnames and other strings at the
first nonconvertible character. (ouch!) Other archivers have worked
around this by storing unconvertible pathnames as raw binary, a
practice which has been sanctioned by the Austin group. However,
libarchive's pax reader would segfault reading headers that weren't
proper UTF-8. (ouch!) Since bsdtar defaults to pax format, this
affects bsdtar rather heavily.

To correctly support the new "hdrcharset" header that is going into
SUS and to handle conversion failures in general, libarchive's pax reader
and writer have been overhauled fairly extensively. They used to do
most of the pax header processing using wchar_t (Unicode); they now do
most of it using char so that common logic applies to either UTF-8 or
"binary" strings.

As a bonus, a number of extraneous conversions to/from wchar_t have
been eliminated, which should speed things up just a tad.

Thanks to: Bjoern Jacke for originally reporting this to me
Thanks to: Joerg Sonnenberger for noting a bad typo in my first draft of this
Thanks to: Gunnar Ritter for getting the standard fixed
MFC after: 5 days

Ignore some built files.

Don't lie. If a string can't be converted to a wide (Unicode) string,
return a NULL instead of an incomplete string. Expand the test coverage
to verify the correct behavior here.

Don't advertise the default block size as a constant; don't
rely on a deprecated value to set the default. This is also
related to a longer-term goal of setting the default block
size based on format and possibly other factors, which makes
it a bad idea to tie this to a published constant.

New public functions archive_entry_copy_link() and archive_entry_copy_link_w()
override the currently set link value, whether that's a hardlink
or a symlink. Plus documentation update and tests.

Update some comments, comment out argument names to guard against
namespace problems.

Since "length" computes the length of a string and is used as an
argument to malloc(3), it should be size_t, not int.

Let archive_entry_clear() accept a NULL pointer and simply do nothing.
In particular, this allows archive_entry_free() to work correctly
for a NULL pointer, which makes it parallel with free(3).

Rework the versioning implementation and test to match the
new interface. Mark the functions that are going away in
libarchive 3.0.

In particular, archive_version_string() now computes the
string rather than assuming that it will be created by the
build infrastructure. Eventually, this will allow some
simplification of the build infrastructure.

Rework the versioning information, hopefully for the last time.
* There are now only two public version identifiers: "number" is
a single integer that combines Major/minor/release in a single
value of the form Mmmmrrr. This is easy to compare against for
checking feature support. "string" is a displayable text string
of the form "libarchive M.mm.rr".
* The number is present both as a macro (version of the installed header)
and a function (version of the shared library). The string form
is available only as a function.
* Retain the older version definitions for now, but mark them all
as deprecated, to disappear in libarchive 3.0 (whenever that happens).
* Rework the various deprecation conditionals to use ARCHIVE_VERSION_NUMBER.

An ancillary goal is to reduce the number of @...@ substitutions that
are required. Someday, I might even be able to avoid build-time
processing of archive.h entirely.

Add a useful sprintf()-style wrapper around
archive_string_vsprintf(). (Which is built
on top of libarchive's internal resizable string
support.)

Support for writing 'compress' format, thanks to Joerg Sonnenberger.

A block in a tar file is 512 bytes. Period.
Remove the entirely pointless symbolic constant
and sizeof(unsigned char). (The constant
here is doubly wrong, since not only does
it obscure a basic format constant, it was
never intended to be a tar-specific value,
so could conceivably be changed at some point
in the future.)

Current 'ar' read support in libarchive can only handle a GNU/SVR4
filename table whose size is less than 65536 bytes.

The original intention was to not consume the filename table, so the
client will have a chance to look at it. To achieve that, the library
call decompressor->read_ahead to read(look ahead) but do not call
decompressor->consume to consume the data, thus a limit was raised
since read_ahead call can only look ahead at most BUFFER_SIZE(65536)
bytes at the moment, and you can not "look any further" before you
consume what you already "saw".

This commit will turn GNU/SVR4 filename table into "archive format
data", i.e., filename table will be consumed by libarchive, so the
65536-bytes limit will be gone, but client can no longer have access
to the content of filename table.

'ar' support test suite is changed accordingly. BSD ar(1) is not
affected by this change since it doesn't look at the filename table.

Reported by: erwin
Discussed with: jkoshy, kientzle
Reviewed by: jkoshy, kientzle
Approved by: jkoshy(mentor), kientzle

Portability: Eliminate the need for uudecode by incorporating
uudecode into the main test driver and invoking it just-in-time
within the various tests.

Also, incorporate a number of improvements to the main test support
code that have proven useful on other projects where I've used this
framework.

Remove some unused fields from the private archive_read structure
(left over from when the unified read/write structure was copied
to form separate read and write structures) and eliminate the
pointless initialization of a couple of the unused fields.

Tighten up the semantics of acl_next() and xattr_next() when you
hit the end of the ACL or xattr list.

Thanks to: Jeff Johnson for pointing out the obvious typo

Typo, thanks to: Jeff Johnson.
MFC after: 3 days

Use .Fo/.Fc and .Xo/.Xc to bring the line widths below 79 columns.
Correct a typo [a misplaced comma].

Reviewed by: ru

Use .Fo/.Fc and .Xo/.Xc to bring the line widths below 79 columns.

Reviewed by: ru

Spelling correction, thanks to Joerg Sonnenberger.

Optimize skipping over Zip entries.

Thanks to: Dan Nelson, who sent me the patch
MFC after: 7 days

Rename the archive_endian.h functions to avoid name clashes
with NetBSD's sys/endian.h file.

Pointed out by: Joerg Sonnenberger

Sanity-check the block size.

Thanks to: Joerg Sonnenberger
MFC after: 7 days

Include O_BINARY in open() calls on platforms that support it.

Another tiny, tiny step towards Windows support. No, I don't plan to
ever commit the Windows support files to FreeBSD CVS. That would just
be wrong.

Someday I might forgive the standards bodies for omitting timegm().
Maybe. In the meantime, my workarounds for trying to coax UTC without
timegm() are getting uglier and uglier. Apparently, some systems
don't support setenv()/unsetenv(), so you can't set the TZ env var and
hope thereby to coax mktime() into generating UTC. Without that, I
don't see a really good alternative to just giving up and converting to
localtime with mktime(). (I suppose I should research the Perl library
approach for computing an inverse function to gmtime(); that might
actually be simpler than this growing list of hacks.)

Simplify file type setting.

The test_assert() function that backs my custom assert() macro
now returns a value, which supports such convenient
constructs as:
if (assert(NULL != foo())) {
}

Also be careful to setlocale("C") for each new test to
avoid locale pollution.

Also a couple of minor portability enhancements.

Portability: Since the values are fixed and the symbolic names
are only present on some platforms, just use the values directly.

Portability: Include O_BINARY if the local platform defines it.

Correct a compile error when libbz2/zlib are unavailable.

Mark a few additional functions that are/are not available on FreeBSD.

Portability improvements:
* If the platform can't restore char nodes, block nodes, or fifos,
don't try and just return error.
* Include O_BINARY in most open() calls (define O_BINARY to 0 if the
platform doesn't provide a definition already)
* Refactor the ownership restore to more cleanly support platforms
that don't have any form of {l,f,}chown() call.
* Comment a lingering issue with older Unix-like systems that allow
root to hose the filesystem. I don't (yet) have a good solution for
this, but I expect it will require adding more redundant stat()
calls. <sigh>

MFC after: 14 days

Add hook routine archive_write_ar_finish() which writes the 'ar'
global header if nothing else has been written before the closing of
the archive. This will change the behaviour when creating archives
without members, i.e., instead of generating a 0-size archive file, an
archive with just the global header (8 bytes in total) will be created
and it is indeed a valid archive by the definition of libarchive, thus
subsequent operation on this archive will be accepted. This especially
solves the failure caused by following sequence: (several ports do)

% ar cru libfoo.a # without specifying obj files
% ranlib libfoo.a

Reviewed by: kientzle, jkoshy
Approved by: kientzle
Approved by: jkoshy (mentor)
Reported by: erwin
MFC after: 1 month

Add a test to verify compatibility with archives with
odd hardlinks. I need to extend this to test pax extended
archives with bodies attached to hardlinks and other less-common cases.

Tighten up the heuristic that decides whether or not we should
obey or ignore the size field on a hardlink entry. In particular,
if we're reading a non-POSIX archive, we should always ignore
the size field.

This should fix both the audio/xmcd port and the math/unixstat port.

Thanks to: Pav Lucistnik for pointing these two ports out to me.
MFC after: 7 days

Track version # from the portable release.

Explain a subtle API change that was made recently.
Even though I believe this is a good change, it does
have the potential to break certain clients, so it's
good to document the reasoning behind the change.

Properly pad symlinks when writing cpio "newc" format.
Thanks to: Jesse Barker for reporting this.
MFC after: 7 days

I misread the Tinderbox error; this should really unbreak 64-bit builds.

Pointy hats, yep, keep 'em coming. ;-/

Fix 64-bit build after my last commit. <sigh>

The previous commit caused the archive_write_disk interface to
start obeying filesize limits; this test wasn't properly setting
file sizes before trying to write file data.

Issues with hardlinks in newc-format files prompted me to
write a new test to exercise the hardlink strategies used
by different archive formats (tar, old cpio, new cpio).
This uncovered two problems, both fixed by this commit:

1) Enforce file size when writing files to disk.

2) When restoring hardlink entries, if they have data associated, go
ahead and open the file so we can write the data.

In particular, this fixes bsdtar/bsdcpio extraction of new cpio
formats where the "original" is empty and the subsequent "hardlink"
entry actually carries the data. It also provides correct behavior
for old cpio archives where hardlinked entries have their bodies
stored multiple times in the archive; the last body should always be
the one that ends up in the final file. The new pax format also
permits (but does not require) hardlinks to carry file data; again,
the last contents should always win.

Note that with any of these, a size of zero on a hardlink simply means
that the hardlink carries no data; it does not mean that the file has
zero size. A non-zero size on a hardlink does provide the file size.

Thanks to: John Baldwin, for reminding me about this long-standing bug
and sending me a simple example archive that prompted this test case

Reconnect the progress callback. It may not get called
as often as you might expect, but at least it will get called
now.

Thanks to: David Topham for asking how this got disconnected.

Handle Zip archives that are "multi-part archives with only
one part" by simply ignoring the marker at the beginning
of the file. (Zip archivers reserve four bytes at the beginning
of each part of a multi-part archive, if it happens to only
require one part, those four bytes get filled with a placeholder
that can be ignored.)

Thanks to: Marius Nuennerich,
for pointing me to a Zip archive that libarchive couldn't handle
MFC after: 7 days

Support uppercase hex digits in cpio archives.

Thanks to: Joshua Kwan
MFC after: 7 days

Since the tar bidder can never get called more than once, it
doesn't need to compensate for this situation.

While here, fix a minor longstanding bug that empty tar archives
(which begin with at least 512 zero bytes) never properly reported
their format. In particular, this fixes the output of:
bsdtar tvvf /dev/zero

And, of course, a new test to verify that libarchive correctly
recognizes the format of such files.

Crib {be,le}{16,32,64}{dec,enc} from src/sys/sys/endian.h and use it instead
of home-rolled [iu][248] in the ZIP support code.

Approved by: kientzle

Add an internal utility function to simplify the many, many places where
the number of bytes read is actually not important as long as we have at
least what we ask for. Illustrate its benefits by using it throughout
the ZIP support code, except for the few cases where it doesn't apply.

Approved by: kientzle

Extensive improvements to the libarchive_test test program that
exercises and verifies the libarchive APIs:

* Improved error reporting; hexdumps are now provided for
many file/memory content differences.
* Overall status more clearly counts "tests" and "assertions"
* Reference files can now be stored on disk instead of having
to be compiled into the test program itself. A couple of
tests have been converted to this more natural structure.
* Several memory leaks corrected so that leaks within libarchive
itself can be more easily detected and diagnosed.
* New test: GNU tar compatibility
* New test: Zip compatibility
* New test: Zero-byte writes to a compressed archive entry
* New test: archive_entry_strmode() format verification
* New test: mtree reader
* New test: write/read of large (2G - 1TB) entries to tar archives
(thanks to recent performance work, this test only requires a few seconds)
* New test: detailed format verification of cpio odc and newc writers
* Many minor additions/improvements to existing tests as well.

The mtree.5 file has been moved to src/usr.sbin/mtree.

Update libarchive to 2.4.10. This includes a number of improvements
that I've been working on but put off committing until after the
RELENG_7 branch, including:

* New manpages: cpio.5 mtree.5
* New archive_entry_strmode()
* New archive_entry_link_resolver()
* New read support: mtree format
* Internal API change: read format auction only runs once
* Running the auction only once allowed simplifying a lot of bid logic.
* Cpio robustness: search for next header after a sync error
* Support device nodes on ISO9660 images
* Eliminate a lot of unnecessary copies for uncompressed archives
* Corrected handling of new GNU --sparse --posix formats
* Correctly handle a zero-byte write to a compressed archive
* Fixed memory leaks

Many of these improvements were motivated by the upcoming bsdcpio
front-end.

There have also been extensive improvements to the libarchive_test
test harness, which I'll commit separately.

If a Zip entry has the "length at end" flag set, then just ignore
a length field of zero; it does not mean the body is empty.

Thanks to: Lapo Luchini for sending me a JAR archive that demonstrated this bug
MFC after: 3 days

When skipping input data, don't overflow a 32-bit size_t.
This can only happen on 32-bit systems when you're reading
an uncompressed archive and the skip request is an exact
multiple of 4G (e.g., skipping a tar entry with an 8G body).

The symptom is that the read_ahead() ends up returning zero
bytes, and the extraction stops with a premature end-of-file.

Using '1' here is more correct anyway, as it allows read_ahead()
to function opportunistically and minimize copying.

MFC after: 5 days

Fix reading of files that use pax 'size' attribute to store size.
In particular, bsdtar uses the pax 'size' attribute for any file
over 8G.

MFC after: 3 days

Correct the cpio writers to not accept data for non-regular files.
In particular, the previous code led to archives that had
non-empty bodies following directory entries. Not a fatal
problem, as bsdtar and GNU cpio are both happy to just skip
this bogus data, but it still shouldn't be there.

MFC after: 3 days

Correct the return values of the final zero-length block at EOF.
Return EOF immediately if an entry in a ZIP archive has no body.
In particular, the latter issue was causing bsdtar to emit spurious
warnings when extracting directory entries from ZIP archives.

MFC after: 3 days

Fix the archive_write_data() function so it always returns
number of bytes written, even when used to write files to
disk. Extend the test suite to verify the correct return
values for archive_write_data() and archive_write_data_block().

Thanks to: Bruce Mah, for stepping in promptly to back out the
earlier broken version of this fix
Thanks to: Colin Percival, for pointing out the correct fix
MFC after: 5 days
Approved by: re (ksmith)
Pointy hat: \me

Fill in a missing 'e'

Thanks to: Kai Wang, for pointing this out
Approved by: re (bmah)
MFC after: 3 days

Revert the last commit to libarchive. It introduced some regresssions,
most noticably the incorrect extraction of files by bsdtar.

This commit reverts:

src/lib/libarchive/archive_write_disk.c 1.15
src/lib/libarchive/test/test_write_disk.c 1.4

Approved by: re (implicitly)

Correct the return value from archive_write_data()
(when used to restore files to disk) to match:
* The documentation
* The return values of this function when used
to write files into an archive.

Approved by: re (bmah)
Pointy hat: \me
MFC after: 5 days

This commit updates libarchive to be compatible with
GNU tar 1.17's implementation of --posix --sparse,
at the cost of losing compatibility with GNU tar 1.16.
Fortunately, the 1.17 implementation actually makes sense,
so the libarchive code is now a bit more straightforward
than before.

Background: GNU tar 1.16 defined a new way to store
sparse files in --posix archives. Unfortunately,
the implementation incorrectly inserted several
blocks of null padding after each such entry.
As a result, non-GNU tar implementations saw the
archive as truncated after any sparse entry.
This was fixed in GNU tar 1.17 at the cost of
losing compatibility with GNU tar 1.16 for this
new format (which is not the default, so hopefully
rarely used). Libarchive recently gained support
for reading the GNU tar 1.16 formats; this commit
updates it to read the GNU tar 1.17 variant instead.

Approved by: re (ksmith for libarchive portion)
Approved by: re (blanket for libarchive_test portion)
MFC after: 5 days

Make suid/sgid restore be "opportunistic" if
owner restore is not requested. If you ask
for permissions to be restored but not owner,
you will now get no error if suid/sgid bits
cannot be set. (It's a security hole to restore
suid/sgid bits if the owner/group aren't restored.)

This fixes an obscure problem where a simple
"tar -xf" with no other options will sometimes
fail gratuitously because of suid/sgid bits.
This is causing occasional problems for people
using bsdtar as a drop-in replacement for
"that other tar program." ;-)

Note: If you do ask for owner restore, then suid/sgid
restore failures still issue an error. This
only suppresses the error in the case where an
suid/sgid bit restore fails because of an owner
mismatch and owner restore was not requested.

Approved by: re (bmah)
MFC after: 7 days

Update the tests for reading the various GNU tar sparse formats.
In particular:
* Include a second entry in all of the test archives (to catch errors
with intermediate padding)
* Test the GNU tar 1.17 version of "posix sparse format 1.0"
instead of the GNU tar 1.16 version (the latter is no longer
supported by GNU tar).

Right now, libarchive fails this test because I originally
implemented the GNU tar 1.16 version of "posix sparse format 1.0".
I'll fix libarchive shortly.

Approved by: re (blanket, libarchive testing)

Two minor nits:
* Allow libarchive_test to compile on Interix again.
* Track the test name (not just line number) when counting skipped tests.

Thanks to: Joerg Sonnenberger
Approved by: re (blanket; libarchive testing)

Fix a strict aliasing warning from GCC 4.1.

Thanks to: Joerg Sonnenberger
Approved by: re (hrs)
MFC after: 3 days

archive_string_ensure() used to call exit(3) if it
couldn't allocate more memory for a string. Change
this so it returns NULL in that case, and update
all of its callers to handle the error. Some of
those callers can now return errors back to the
client instead of calling exit(3).

Approved by: re (bmah)

Add archive_entry_copy_gname() and archive_entry_copy_uname()
functions.

Approved by: re (bmah)
MFC after: 3 days

Clarify one test.

Approved by: re (blanket, libarchive testing)

Use an uncompressed test archive for gtar sparse format 1.0 format,
as that better exercises some internal read-combining logic than the
compressed archive.

Approved by: re (blanket, libarchive testing)

Report each failed test once, but keep a count and report that count
if there was more than one. In particular, this simplifies
test_tar_filenames.c, which has a tendency to be very noisy otherwise.

Approved by: re (blanket, libarchive testing)

Restore the 'break' that was inadvertently removed in 1.57 of this file.
Without this, hardlinks get returned as symlinks.

Approved by: re (Ken Smith)
MFC after: 2 days

Extend the basic tar reading test to exercise most types of
entries. This doesn't cover everything yet, but it's a big improvement.

Approved by: re (blanket, libarchive testing)

Make the test for reading gtar sparse entries more robust;
it now verifies that the returned blocks have the correct data
at the correct file offsets, ignoring any null padding that
may exist.

Approved by: re (blanket, libarchive test suite)

New test suite test_read_pax_truncated probes libarchive
behavior with truncated or damaged pax archives. This
tests most of the cases covered by the recent security advisory.

Approved by: re (blanket, libarchive test suite)

New file "read_open_memory.c" is a custom variant of
archive_read_open_memory.c that tries to test border
cases. In particular, it copies over each returned block
so that formats or decompressors that read past the end
of a returned block will break.

Approved by: re (blanket, libarchive test suite)

Fix running individual tests via "libarchive_test <number> <number> ..."

Approved by: re (blanket)

Correct multiple security issues in how libarchive handles corrupt
tar archives, including a potentially exploitable buffer overflow.

Approved by: re (kensmith, security blanket)
Reviewed by: kientzle
Security: FreeBSD-SA-07:05.libarchive

Fix installworld: /usr/bin/printf isn't available then,
so use awk's printf for the formatting here instead.

Pointy hat: Yours Truly
Approved by: re

Make test suite work with libarchive 1.3.1: Take advantage of
ARCHIVE_VERSION_STAMP to selectively disable tests that don't
apply to that version; new "skipping()" function reports skipped
tests; modify final summary to report component test failures and
skips.

Note: I don't currently intend to MFC the test suite itself;
anyone interested should just checkout and use this version
of the test suite, which should work for any library version.

Approved by: re (Ken Smith, blanket)

New "version stamp" simplifies determining the exact version
of libarchive being used. I've been taking advantage of this
with a recent round of updates to libarchive_test so that it
can test older and newer versions of the library.

Approved by: re (Ken Smith)

Fix 'bsdtar -t' on tape drives. Libarchive uses the
skip() callback to skip over data when reading uncompressed
archives. This gets invoked, for example, during tar -t
or tar -x with a filename argument. The revised code
only calls [lf]seek() on regular files, instead of depending
on the kernel to return an error.

Thanks to: bde for explaining the implementation of lseek()
Thanks to: Daniel O'Connor for testing
Approved by: re (Ken Smith)
MFC after: 5 days

Ouch. I partially screwed up the last commit by
enabling a test that's not ready yet. <sigh>

Pointy hat: /me
Approved by: re@

Support for writing the 'newc' cpio format, plus a minimal test harness
for the cpio formats.

Thanks to: Rudolf Marek
Approved by: re@

Track whether the current read stream supports seek(). For now, we
assume yes unless seek has previously failed, but I fear I'll have to
avoid seeks under other circumstances. (For instance, tape drives on
FreeBSD seem to return garbage from lseek().) Also, optimize away
zero-byte skips.

Make 'ar' write test a tad more portable.

Read support for the new GNU tar sparse formats added in gtar 1.15 and
gtar 1.16.

Add some options to libarchive_test:
-k: like make -k, try to keep going after errors.
-q: quiet

Fix a broken function declaration.

Options spring cleanup:
- Add and document the KVM and KVM_SUPPORT options that
are needed for the ifmcstats(3) makefile
- Garbage collect unused variables
- Add missing inclusion of bsd.own.mk where needed

Approved by: kan (mentor)
Reviewed by: ru

Don't lose leading '/' for pathnames exactly 101 bytes long.
Also, update the test harness to exercise this case.

libarchive 2.2.3
* "compression_program" support uses an external program
* Portability: no longer uses "struct stat" as a primary
data interchange structure internally
* Part of the above: refactor archive_entry to separate
out copy_stat() and stat() functions
* More complete tests for archive_entry
* Finish archive_entry_clone()
* Isolate major()/minor()/makedev() in archive_entry; remove
these from everywhere else.
* Bug fix: properly handle decompression look-ahead at end-of-data
* Bug fixes to 'ar' support
* Fix memory leak in ZIP reader
* Portability: better timegm() emulation in iso9660 reader
* New write_disk flags to suppress auto dir creation and not
overwrite newer files (for future cpio front-end)
* Simplify trailing-'/' fixup when writing tar and pax
* Test enhancements: fix various compiler warnings, improve
portability, add lots of new tests.
* Documentation: document new functions, first draft of
libarchive_internals.3

MFC after: 14 days
Thanks to: Joerg Sonnenberger (compression_program)
Thanks to: Kai Wang (ar)
Thanks to: Colin Percival (many small fixes)
Thanks to: Many others who sent me various patches and problem reports.

Replace "sizeof(struct bucket[cache_size])" with
"cache_size * sizeof(struct bucket)". The former is valid in C99 but can
confuse earlier compilers, while the latter is a standard idiom which all
C compilers understand.

Approved by: kientzle

Don't test for NULL when it is both unnecessary (the pointer is checked
against NULL when it is first allocated) and pointless (we've already
dereferenced the pointer several times).

Found by: Coverity Prevent(tm)
CID: 3204

Remove pointless code: Don't assign a value to a variable when we're
going to overwrite it with a new value a few lines later.

Visual inspection of the surrounding code indicates that the code does
what it's supposed to do; i.e., the pointless code wasn't supposed to
be doing something other than what it was doing.

CID: 3323
Found by: Coverity Prevent(tm)

Don't forget to free a string allocated by strdup where necessary.

Found by: Coverity Prevent
Coverity ID: CID 3132

s/@VERSION@/@ARCHIVE_VERSION@/

This is a no-op as far as FreeBSD is concerned, but makes libarchive
more autoconf-friendly.

Approved by: kientzle

Fix a memory leak in the uname/gname lookup cache.
Thanks to: VMiklos

In libarchive: Downgrade ARCHIVE_FATAL and ARCHIVE_FAILED errors which
occur on the write side of extracting a file to ARCHIVE_WARN errors
when returning them from archive_read_extract.

In bsdtar: Use the return code from archive_read_data_into_fd and
archive_read_extract to determine whether we should continue trying to
extract an archive after one of the entries fails.

This commit makes extracting a truncated tarball complain once about
the archive being truncated, instead of complaining twice (once when
trying to extract an entry, and once when trying to seek to the next
entry).

Discussed with: kientzle

Properly cleanup the UID/GID lookup data.

Don't free a NULL pointer.

Clean up a lot of memory leaks in the libarchive test harness.

Thanks to Colin for catching my mixup. The original problem wasn't
suppressing the second error, it was failing to address the first.

Consolidate numeric limit macros in one place; include them
only on platforms that need them. FreeBSD doesn't.

Pass through error message if any operation fails, not just the last one.

Overhaul of 'ar' support:
* use "AR_GNU" as the format name instead of AR_SVR4 (it's what everyone is going to call it anyway)
* Simplify numeric parsing to unsigned (none of the numeric values should ever be negative); don't run off end of numeric fields.
* Finish parsing the common header fields before the next I/O request (which might dump the contents)
* Be smarter about format guessing and trimming filenames.
* Most of the magic values are only used in one place, so just inline them.
* Many more comments.
* Be smarter about handling damaged entries; return something reasonable.
* Call it a "filename table" instead of a "string table"
* Update tests.

Enable selection of 'ar', 'arbsd', and 'argnu' formats by name
(this allows bsdtar to create ar format archives).

The 'ar' writer still needs some work; it should reject
entries that aren't regular files and should probably also
strip leading paths from filenames.

Invoke utime(2) properly. (It's only used on platforms
that lack utimes(2).)

Fix the build by temporarily disabling 'ar' support until
I can clean it up.

Fix the build.
N.B. 'ar' format support is broken right now, it's not
passing tests. If I can't find the problem soon, I'll
back out the last commit.

Fixes from Joerg Sonnenberger, reviewed by Kai Wang.

Conventionally, tar archives have always included a trailing '/'
for directories. bsdtar used to add this, but that recently got
lost somehow. So now I'm adding it back in libarchive.
The only odd part of doing this in libarchive: Adding a directory to
a tar archive and then reading it back again can yield a different name.

Add a test case to exercise some boundary conditions with
tar filenames and ensure that trailing slashes are added to
dir names only as necessary.

Thanks to: Oliver Lehmann for bringing this regression to my attention.

Portability: Don't use mkdtemp() when mkdir() will suffice.
If we can't create the dir, just give up.

More portability improvements from Martin Koeppe:
conditionally use utime() when utimes() is not available;
allow the most common wide-char functions to be replaced
when local alternatives are lacking.

Portability.

Now that libarchive is being built in more environments,
factor out the platform-specific configuration header a bit
more cleanly.
Suggested by: Joerg Sonnenberger

Make Lint happier.

Fix build on Solaris.

Enable 'ar' support; hook it up to the build and
enable it with _read_support_format_all().

Clarification: Point people to archive_read_data(), which
should be used instead of archive_read_data_into_buffer().

When copying data from one archive to another, only set the error
message in the reader to the error message from the writer if the
error which occurred was in the writer. This avoids error messages
of "Empty error message" when extracting truncated archives.

Move archive_read_data_into_buffer into archive_read.c, simplify its
implementation, and mark it as deprecated. It will be removed entirely
in libarchive 3.0 (in FreeBSD 8.0?) but there's no reason for anyone to
use it instead of archive_read_data.

Approved by: kientzle

More corrections from Joerg Sonnenberger.

Style fixes from Joerg Sonnenberger: use correct types,
spell lint(1) comments correctly.

From Joerg Sonnenberger: Fix a number of style gaffes,
including type puns and avoidable casts.

Wordsmithing.

Parse SCHILY.dev and SCHILY.ino fields. These are ignored when extracting
files, but used during archive creation. This change unbreaks
# tar -cf rcp.tar /bin/rcp
# tar -cf rcp-copy.tar @rcp.tar
# cmp rcp.tar rcp-copy.tar

'ar' format support for libarchive, contributed by Kai Wang.

Now that there is always a compression-layer skip function available,
skip over the end-of-entry padding instead of reading and discarding
it.

Considering that tar files normally have a block size of 10kB, this
isn't likely to avoid reading any data, but at least it makes the code
simpler and clearer.

Fix type-punned pointer, minor style fixes.
Thanks to: Joerg Sonnenberger

Remove unused variable; use consistent types.
Thanks to: Joerg Sonnenberger

Be consistent: file flags are unsigned bitmaps.
Thanks to: Joerg Sonnenberger

Don't compare a signed char to 0xFF.
Thanks to: Joerg Sonnenberger

Avoid a potential overflow when 'skip' is larger than a pointer.
Thanks to: Joerg Sonnenberger

Style fix: Use the correct type for 'bytes_to_write'.

Thanks to: Joerg Sonnenberger

Style: bare "unsigned" is deprecated, use "unsigned int" instead.

Thanks to: Joerg Sonnenberger

Remove some unused fields from archive_read internal structure.
(Left over from when read and write used to share this structure.)

Provide a dummy compression-layer skip function which just reads data and
discards it, for use when the compression layer code doesn't know how to
skip data (e.g., everything other than the "none" compressor). This makes
format level code simpler because that code can now assume that the
compression layer always knows how to skip and will always skip exactly
the requested number of bytes.

Discussed with: kientzle (3 months ago)

Notice when mkdir() fails.
Don't change permissions on an existing dir unless _EXTRACT_PERM
is requested.

In particular, bsdtar -x should not edit mode of existing dirs
now; bsdtar -xp will.

Fix compile error when libbz2 is unavailable.

Don't forget to increment the raw_position (bytes written) counter, even
when operating in non-buffered mode.

Pointy hat to: cperciva
MFC after: 3 days

When ARCHIVE_EXTRACT_UNLINK is requested:
* Only try to remove the existing item if we're not restoring a directory.
* If unlink fails, try rmdir next.
This should fix the broken --unlink option in bsdtar.

Thanks again to: Kris Kennaway, for beating up bsdtar on pointyhat.

Libarchive 2.0.23:
* The ACL formatter was mis-formatting entries which had a
user/group ID but no name. Make the parser tolerant of
these, so that old archives can be correctly restored;
fix the formatter to generate correct entries.
* Fix overwrite detection by introducing a new "FAILED" return
code that indicates the current entry cannot be continued
but the archive as a whole is still sound.
* Header cleanup: Remove some unused headers, add some that
are required with new Linux systems.

New tests for system-independent ACL support.
These tests verify that archive_entry objects can store and return
ACL data and that pax format archives can read and write ACL
information. These do not (yet) test that ACL data is read or
written to disk correctly. (And hence would not have caught the
recent snafu about ACL read-from-disk being turned off.)

Distinguish between the end of ACL data and an error in pulling
ACL data from the archive entry. This doesn't impact
archive_read_extract or archive_write_disk since they only
check for != ARCHIVE_OK when calling this function. (Though
they should be more careful.)

libarchive 2.0
* libarchive_test program exercises many of the core features
* Refactored old "read_extract" into new "archive_write_disk", which
uses archive_write methods to put entries onto disk. In particular,
you can now use archive_write_disk to create objects on disk
without having an archive available.
* Pushed some security checks from bsdtar down into libarchive, where
they can be better optimized.
* Rearchitected the logic for creating objects on disk to reduce
the number of system calls. Several common cases now use a
minimum number of system calls.
* Virtualized some internal interfaces to provide a clearer separation
of read and write handling and make it simpler to override key
methods.
* New "empty" format reader.
* Corrected return types (this ABI breakage required the "2.0" version bump)
* Many bug fixes.

Force the umask to something predictable while creating objects
on disk. In particular, this fixes tar -xp restore of mode bits.

Because the buffer gets released immediately, I need to
copy the symlink target name, not just copy the reference.
This problem sometimes caused crashes when extracting
symlinks from ISO9660 images.

Thanks to: Diego "Flameeyes" Pettenò

If (a == NULL), don't dereference (a) to record an error message. [1]

Fallout from changing the skip API to use off_t instead of size_t: Print
the skip length using %jd and cast to (intmax_t) instead of %d / (int),
and if ARCHIVE_API_VERSION >= 2, allow the client skipper to be called
for requests longer than SSIZE_MAX. [2]

Approved by: kientzle
Pointy hats to: kientzle [1], cperciva [2]
MFC after: 3 days

Add support for a new archive format "empty" that reads empty files.

Correct the int->text conversion. <sigh>
MFC after: 3 days

Fix the copyright notice; it was always intended to be
a vanilla 2-clause BSD license, but somehow some confusing
extra verbage get copied from somewhere.

Also, update the copyright dates to 2007 for all of the files.

Prompted by: several questions about what those extra words really mean

Special case to support hard-coded configuration for building on Windows.

Change the client skipper API to use off_t instead of size_t/ssize_t; but
wrap this within #if/#else/#endif so that it will only take effect once
ARCHIVE_API_VERSION is increased (which should happen on HEAD some time
between now and when RELENG_7 is branched).

Convert compression_skip from taking a size_t skip length request and
returning the length skipped in a ssize_t to using off_t for both. This
does not break any A[BP]Is, since compression_skip is entirely internal
to libarchive.

If a skip request is > SSIZE_MAX, don't pass it down to the client layer
skip function, since those still uses size_t / ssize_t. Instead, just
read the data and throw it away.

With this commit, libarchive/bsdtar should now successfully skip archive
entries of >2GB on 32-bit systems, but does so slower than necessary.
The performance will improve with a future A[BP]I breaking commit which
makes client layer skip functions use off_t.

Discussed with: kientzle
MFC after: 1 week

Rewrite and simplify archive_read_format_tar_skip. Compression-layer skip
functions are required to skip the requested distance, so we can avoid
lots of bookkeeping which would otherwise be necessary.

Reviewed by: kientzle
MFC after: 1 week

Quiet a compiler warning where (off_t + size_t) has a different sign
from (off_t).

Submitted by: delphij

Insert zero-padding between sparse blocks in archive_read_data(). This
fixes "tar -c @foo.tar" where "foo.tar" contains sparse entries.

MFC after: 1 week

Correct spelling of "accommodate", "guarantee", "guaranteed", "guard",
"return", "ridiculous", and "success".

MFC after: 3 days

Minor refactoring; move the FreeBSD-specific config info into
config_freebsd.h. archive_platform.h decides which config file
to bring in and uses some of those selectors to define wrapper
macros and other compatibility glue.

Argh. Restore a stat() call that was erroneously removed.

Thanks to: WATANABE, Kazuo
Pointy hat: me, from the handy dispenser I keep nearby.

Improve support for large ISOs:
* Correct a signed/unsigned problem that broke handling of files >2G.
* Implement "skip" support for much faster "tar -t".

Thanks to: Robert Sciuk for sending me a DVD that illustrated the first problem