libarchive: import changes from upstream

Libarchive 3.5.2

New features:
PR #1502: Support for PWB and v7 binary cpio formats
PR #1509: Support of deflate algorithm in symbolic link decompression
for ZIP archives

Important bugfixes:
IS #1044: fix extraction of hardlinks to symlinks
PR #1480: Fix truncation of size values during 7zip archive
extraction on 32bit architectures
PR #1504: fix rar header skiming
PR #1514: ZIP excessive disk read - fix location of central directory
PR #1520: fix double-free in CAB reader
PR #1521: Fixed leak of rar before ending with error
PR #1530: Handle short writes from archive_write_callback
PR #1532: 7zip: Use compression settings from file also for file header
IS #1566: do not follow symlinks when processing the fixup list

Relnotes: yes

(cherry picked from commit ddce862ad8594542e1fa1af9ffae7264e12ffd27)

Git Hash: fd9825481d30a1a083219d610ecaaa152fceb21b
Git Author: mm@FreeBSD.org

MFC r368207,368607:

MFC r368207:
Update libarchive to 3.5.0

Relevant vendor changes:
Issue #1258: add archive_read_support_filter_by_code()
PR #1347: mtree digest reader support
Issue #1381: skip hardlinks pointing to itself on extraction
PR #1387: fix writing of cpio archives with hardlinks without file type
PR #1388: fix rdev field in cpio format for device nodes
PR #1389: completed support for UTF-8 encoding conversion
PR #1405: more formats in archive_read_support_format_by_code()
PR #1408: fix uninitialized size in rar5_read_data
PR #1409: system extended attribute support
PR #1435: support for decompression of symbolic links in zipx archives
Issue #1456: memory leak after unsuccessful archive_write_open_filename

MFC r368607:
Sync libarchive with vendor.

Vendor changes:
Issue #1461: Unbreak build without lzma
Issue #1462: warc reader: Fix build with gcc11
Issue #1463: Fix code compatibility in test_archive_read_support.c
Issue #1464: Use built-in strnlen on platforms where not available
Issue #1465: warc reader: fix undefined behaviour in deconst() function

MFC r347990:
Sync libarchive with vendor.

Relevant vendor changes:
Issue #795: XAR - do not try to add xattrs without an allocated name
PR #812: non-recursive option for extract and list
PR #958: support reading metadata from compressed files
PR #999: add --exclude-vcs option to bsdtar
Issue #1062: treat empty archives with a GNU volume header as valid
PR #1074: Handle ZIP files with trailing 0s in the extra fields
(Android APK archives)
PR #1109: Ignore padding in Zip extra field data (Android APK archives)
PR #1167: fix problems related to unreadable directories
Issue #1168: fix handling of strtol() and strtoul()
PR #1172: RAR5 - fix invalid window buffer read in E8E9 filter
PR #1174: ZIP reader - fix of MSZIP signature parsing
PR #1175: gzip filter - fix reading files larger than 4GB from memory
PR #1177: gzip filter - fix memory leak with repeated header reads
PR #1180: ZIP reader - add support for Info-ZIP Unicode Path Extra Field
PR #1181: RAR5 - fix merge_block() recursion
(OSS-Fuzz 12999, 13029, 13144, 13478, 13490)
PR #1183: fix memory leak when decompressing ZIP files with LZMA
PR #1184: fix RAR5 OSS-Fuzz issues 12466, 14490, 14491, 12817
OSS-Fuzz 12466: RAR5 - fix buffer overflow when parsing huffman tables
OSS-Fuzz 14490, 14491: RAR5 - fix bad shift-left operations
OSS-Fuzz 12817: RAR5 - handle a case with truncated huffman tables
PR #1186: RAR5 - fix invalid type used for dictionary size mask
(OSS-Fuzz 14537)
PR #1187: RAR5 - fix integer overflow (OSS-Fuzz 14555)
PR #1190: RAR5 - RAR5 don't try to unpack entries marked as directories
(OSS-Fuzz 14574)
PR #1196: RAR5 - fix a potential SIGSEGV on 32-bit builds
OSS-Fuzz 2582: RAR - fix use after free if there is an invalid entry
OSS-Fuzz 14331: RAR5 - fix maximum owner name length
OSS-Fuzz 13965: RAR5 - use unsigned int for volume number + range check

Additional RAR5 reader changes:
- support symlinks, hardlinks, file owner, file group, versioned files
- change ARCHIVE_FORMAT_RAR_V5 to 0x100000
- set correct mode for readonly directories
- support readonly, hidden and system Windows file attributes

MFC r347999:
Install missing data file for
lib.libarchive.functional_test.test_read_format_zip_utf8_paths

Approved by: re (gjb blanket)

MFC r344063,r344088:

MFC r344063:
Sync libarchive with vendor.

Relevant vendor changes:
PR #1085: Fix a null pointer dereference bug in zip writer
PR #1110: ZIP reader added support for XZ, LZMA, PPMD8 and BZIP2
decopmpression
PR #1116: Add support for 64-bit ar format
PR #1120: Fix a 7zip crash [1] and a ISO9660 infinite loop [2]
PR #1125: RAR5 reader - fix an invalid read and a memory leak
PR #1131: POSIX reader - do not fail when tree_current_lstat() fails
due to ENOENT [3]
PR #1134: Delete unnecessary null pointer checks before calls of free()
OSS-Fuzz 10843: Force intermediate to uint64_t to make UBSAN happy.
OSS-Fuzz 11011: Avoid buffer overflow in rar5 reader

MFC r344088:
archive_read_disk_posix.c: initialize delayed_errno

PR: 233006 [3]
Security: CVE-2019-1000019 [1], CVE-2019-1000020 [2]

MFC r339746,339751,339794,340866,340939,342042:
Sync libarchive with vendor.

Relevant vendor changes:
PR #1013: Add missing h_base offset when performing absolute seeks in
xar decompression
PR #1023: Support extracting extattrs as non-root on non-user-writeable
files
PR #1061: Add support for extraction of RAR v5 archives
PR #1066: Fix out of bounds read on empty string filename for gnutar, pax
and v7tar
PR #1067: Fix temporary file path buffer overflow in tests
IS #1068: Correctly process and verify integer arguments passed to
bsdcpio and bsdtar
PR #1070: Don't default XAR entry atime/mtime to the current time
PR #1080: Spelling fixes
PR #1084: RAR5 reader bugfixes
PR #1091: fix use-after-free in delayed newc link processing
PR #1092: Fix a few obvious resource leaks and strcpy() misuses
IS #1096: Support extracting ACLs with in-entry comments (GNU tar)
PR #1102: RAR5 reader - fix big-endian problems
PR #1105: Fix various crash, memory corruption and infinite loop conditions

RAR5 reader: FreeBSD build platform fixes for powerpc(64), mips(64),
sparc64 and riscv64
RAR5 reader: more maybe-uninitialized size_t fixes for riscv64
FreeBSD build

MFH r324148:
Sync libarchive with vendor.

Relevant vendor changes:
PR #905: Support for Zstandard read and write filters
PR #922: Avoid overflow when reading corrupt cpio archive
Issue #935: heap-based buffer overflow in xml_data (CVE-2017-14166)
OSS-Fuzz 2936: Place a limit on the mtree line length
OSS-Fuzz 2394: Ensure that the ZIP AES extension header is large enough
OSS-Fuzz 573: Read off-by-one error in RAR archives (CVE-2017-14502)

Security: CVE-2017-14166, CVE-2017-14502

MFC r315636,315876,316095:
Sync libarchive with vendor

Vendor changes/bugfixes (FreeBSD-related):
r315636:
PR 867 (bsdcpio): show numeric uid/gid when names are not found
PR 870 (seekable zip): accept files with valid ZIP64 EOCD headers
PR 880 (pax): Fix handling of "size" pax header keyword
PR 887 (crypto): Discard 3072 bytes instead of 1024 of first keystream
OSS-Fuzz issue 806 (mtree): rework mtree_atol10 integer parser
Break ACL read/write code into platform-specific source files

r315876:
Store extended attributes with extattr_set_link() if no fd is provided
Add extended attribute tests to libarchive and bsdtar
Fix tar's test_option_acls
Support the UF_HIDDEN file flag

r316095:
Constify variables in several places
Unify platform ACL code in a single source file
Fix unused variable if compiling on FreeBSD without NFSv4 ACL support

MFC r312452-r312512:

r312452-r312512:

- Use SRCTOP-relative paths to other directories instead of .CURDIR-relative ones

This simplifies pathing in make/displayed output

- Use .CURDIR:H instead of .CURDIR to simplify pathing in output, etc

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFV r299425:

Update libarchive to 3.2.0

New features:
- new bsdcat command-line utility
- LZ4 compression (in src only via external utility from ports)
- Warc format support
- 'Raw' format writer
- Zip: Support archives >4GB, entries >4GB
- Zip: Support encrypting and decrypting entries
- Zip: Support experimental streaming extension
- Identify encrypted entries in several formats
- New --clear-nochange-flags option to bsdtar tries to remove noschg and
similar flags before deleting files
- New --ignore-zeros option to bsdtar to handle concatenated tar archives
- Use multi-threaded LZMA decompression if liblzma supports it
- Expose version info for libraries used by libarchive

Patched files (fixed compiler warnings):

contrib/libarchive/cat/bsdcat.c (vendor PR #702)
contrib/libarchive/cat/bsdcat.h (vendor PR #702)
contrib/libarchive/libarchive/archive_read_support_format_mtree.c (PR #701)
contrib/libarchive/libarchive_fe/err.c (vendor PR #703)

MFC after: 1 month
Relnotes: yes

Merge the projects/release-pkg branch to head.

This allows packaging the base system with pkg(8), including
but not limited to providing the ability to provide upstream
binary update possibilities for non-tier-1 architectures.

This merge is a requirement of the 11.0-RELEASE, and as such,
thank you to everyone that has tested the project branch.

Documentation in build(7) etc. is still somewhat sparse, but
updates to those parts will follow.

Sponsored by: The FreeBSD Foundation

Don't override LIB*DIR variables from src.libnames.mk.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Integrate the tests from lib/libarchive, usr.bin/cpio, and usr.bin/tar in to
the FreeBSD test suite

functional_test.sh was ported from bin/sh/tests/functional_test.sh, as a
small wrapper around libarchive_test, bsdcpio_test, and bsdtar_test provided
by upstream.

A handful of testcases in lib/libarchive/tests have been disabled as they
were failing when run with kyua test (see BROKEN_TESTS in
lib/libarchive/tests/Makefile)

As a sidenote: this removes the check/test targets from the Makefiles as they
don't match the pattern used in the rest of the FreeBSD test suite.

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Drop libarchive.pc

We want to ensure we always use libarchive from ports in the ports tree.
It simplifies ports maintainance and anyway libarchive.pc was not reflecting the
different way libarchive can be built in base

Remove the const qualifier from iconv(3) to comply with POSIX:
http://pubs.opengroup.org/onlinepubs/9699919799/functions/iconv.html

Adjust all code that calls iconv.

PR: 199099
Exp-run by: antoine
MFC after: 2 weeks

Convert libraries to use LIBADD
While here reduce a bit overlinking

Add pkg-config file for libarchive

Requested by: bapt
MFC after: 1 week

Remove ia64.

This includes:
o All directories named *ia64*
o All files named *ia64*
o All ia64-specific code guarded by __ia64__
o All ia64-specific makefile logic
o Mention of ia64 in comments and documentation

This excludes:
o Everything under contrib/
o Everything under crypto/
o sys/xen/interface
o sys/sys/elf_common.h

Discussed at: BSDcan

Use src.opts.mk in preference to bsd.own.mk except where we need stuff
from the latter.

Temporarily disable iconv for non-shared library builds. The dynamic
loading of conversation table is not yet compatible with static builds.

Approved by: re (gjb)

MFV r248590,248594:
Update libarchive to 3.1.2

Some of new features:
- support for lrzip and grzip compression
- support for writing tar v7 format
- b64encode and uuencode filters
- support for __MACOSX directory in Zip archives
- support for lzop compresion (external utility)

Remove several MLINKS that were listed twice.

Backport NFSv4 ACL fix from libarchive master branch.

Source:
https://github.com/libarchive/libarchive/commit/f67370d5

Obtained from: libarchive (master branch)

Update libarchive to 3.0.4

Silence alignment warnings from clang in libarchive's ppmd code (which
is actually third-party code). Clang even warns about alignment issues
on x86, and the warnings are not needed there.

Update libarchive to 3.0.3

Some of new features:
- New readers: RAR, LHA/LZH, CAB reader, 7-Zip
- New writers: ISO9660, XAR
- Improvements to many formats, especially including ISO9660 and Zip
- Stackable write filters to write, e.g., tar.gz.uu in a single pass
- Exploit seekable input; new "seekable" Zip reader can exploit the Zip
Central Directory when it's available; the old "streamable" Zip reader
is still fully supported for cases where seeking is not possible.

Full release notes available at:
https://github.com/libarchive/libarchive/wiki/ReleaseNotes

Use contrib sources for building libarchive, tar and cpio.
Make "make test" fully operational.

MFC after: 2 weeks

- Update libarchive to 2.8.4
- Add support for extracting xar and rpm archives
- Add libarchive_fe subdir (common code for tar and cpio)

Approved by: kientzle
MFC after: 2 weeks

libarchive is mixing libmd and libcrypto -- correct to use one or the other.
[mixing the two can be quite bad -- they define the same context structures,
but with differing structure members (and sizes)]

Update the hash function support comments, and update config_freebsd.h
to match.

Approved by: kientzle

Fix build.

Reorganize slightly in preparation for making lzma and bz2 support conditional.

Enable liblzma support in libarchive
Adjust dependencies for programs using libarchive
Add xz and linkage against liblzma to rescue system

Approved by: kientzle, delphij (mentor)
MFC after: 2 weeks

Build lib/ with WARNS=6 by default.

Similar to libexec/, do the same with lib/. Make WARNS=6 the norm and
lower it when needed.

I'm setting WARNS?=0 for secure/. It seems secure/ includes the
Makefile.inc provided by lib/. I'm not going to touch that directory.
Most of the code there is contributed anyway.

UU decoder. Now that libarchive can recursively taste input streams,
you can do things like this: tar xvf archive.tar.gz.uu

Zip write support for libarchive.

The initial implementation was developed by Anselm Strauss as part of
Google Summer of Code 2008, then completed by Joerg Sonnenberger.

New archive_file_count() utility.

Various portability fixes, plus:
* New "ino64" field.
* New UTF8 interfaces for hardlink/symlink updates

Bump the version of all non-symbol-versioned shared libraries in
preparation for 8.0-RELEASE. Add the previous version of those
libraries to ObsoleteFiles.inc and bump __FreeBSD_Version.

Reviewed by: kib
Approved by: re (rwatson)

Include libmd and libcrypto in DPADD

Link libarchive against -lmd and -lcrypto.

Thanks to Ed Schouten for the clue.

Document the liblzma support.
Unfortunately, liblzma itself is GPLed, so unlikely to become part of
the FreeBSD base system.
However, the core lzma compression/decompression code is public
domain, so it should be feasible for someone to create a compatible
library without the GPL strings.

Symlink some additional man page entries.

Merge r990,r1044 from libarchive.googlecode.com:
read_support_format_raw() allows people to exploit libarchive's
automatic decompression support by simply stubbing out the
archive format handler.
The raw handler is not enabled by support_format_all(), of course.
It bids 1 on any non-empty input and always returns a single
entry named "data" with no properties set.

Rely on OpenSSL bits only if we're building a system with OpenSSL.
Also, adjust the MD5 calls to rely on libmd instead of libcrypto,
so we keep MD5 support even in the !OpenSSL case.

Merge new xz/lzma support from libarchive.googlecode.com.
Since FreeBSD doesn't have liblzma in the base system, the
read side will always fall back to the unxz/unlzma commands for now.
(Which will in turn fail if those commands are not currently
installed.) The write side does not yet have a fallback, so
that will just fail.

Merge from libarchive.googlecode.com r756,r761:
Document the new archive_read_disk API.

Merge r399,401,402,405,415,430,440,452,453,458,506,533,536,538,544,590
from libarchive.googlecode.com: Add a new "archive_read_disk" API
that provides the important service of reading metadata from the
disk. In particular, this will make it possible to remove all
knowledge of extended attributes, ACLs, etc, from clients such
as bsdtar and bsdcpio.

Closely related, this API also provides pluggable uid->uname
and gid->gname lookup and caching services similar to
the uname->uid and gname->gid services provided by archive_write_disk.
Remember this is also required for correct ACL management.

Documentation is still pending...

New mtree writer, thanks to Joerg Sonnenberger.

Obtained from: Joerg Sonnenberger

Remove the stillborn attempt to cleanup tests as well as the build dir.

Fail copying path/.. only if SECURE_NODOTDOT was specified.
Since we already warn for any '..' elements in that case,
the extra "lastdotdot" tracking turns out to be unnecessary.

PR: bin/124924

archive.h is no longer constructed from archive.h.in,
so we can rename it and drop some no-longer-necessary
build magic from the Makefile.

MFp4: libarchive 2.5.4b. (Still 'b' until I get a bit more
feedback, but the 2.5 branch is shaping up nicely.)

In addition to many small bug fixes and code improvements:
* Another iteration of versioning; I think I've got it right now.
* Portability: A lot of progress on Windows support (though I'm
not committing all of the Windows support files to FreeBSD CVS)
* Explicit tracking of MBS, WCS, and UTF-8 versions of strings
in archive_entry; the archive_entry routines now correctly return
NULL only when something is unset, setting NULL properly clears
string values. Most charset conversions have been pushed down to
archive_string.
* Better handling of charset conversion failure when writing or
reading UTF-8 headers in pax archives
* archive_entry_linkify() provides multiple strategies for
hardlink matching to suit different format expectations
* More accurate bzip2 format detection
* Joerg Sonnenberger's extensive improvements to mtree support
* Rough support for self-extracting ZIP archives. Not an ideal
approach, but it works for the archives I've tried.
* New "sparsify" option in archive_write_disk converts blocks of nulls
into seeks.
* Better default behavior for the test harness; it now reports
all failures by default instead of coredumping at the first one.

Add MLINK for archive_write_close.

Approved by: jkoshy(mentor), kientzle

New public functions archive_entry_copy_link() and archive_entry_copy_link_w()
override the currently set link value, whether that's a hardlink
or a symlink. Plus documentation update and tests.

Support for writing 'compress' format, thanks to Joerg Sonnenberger.

Track version # from the portable release.

The mtree.5 file has been moved to src/usr.sbin/mtree.

Update libarchive to 2.4.10. This includes a number of improvements
that I've been working on but put off committing until after the
RELENG_7 branch, including:

* New manpages: cpio.5 mtree.5
* New archive_entry_strmode()
* New archive_entry_link_resolver()
* New read support: mtree format
* Internal API change: read format auction only runs once
* Running the auction only once allowed simplifying a lot of bid logic.
* Cpio robustness: search for next header after a sync error
* Support device nodes on ISO9660 images
* Eliminate a lot of unnecessary copies for uncompressed archives
* Corrected handling of new GNU --sparse --posix formats
* Correctly handle a zero-byte write to a compressed archive
* Fixed memory leaks

Many of these improvements were motivated by the upcoming bsdcpio
front-end.

There have also been extensive improvements to the libarchive_test
test harness, which I'll commit separately.

Add archive_entry_copy_gname() and archive_entry_copy_uname()
functions.

Approved by: re (bmah)
MFC after: 3 days

Correct multiple security issues in how libarchive handles corrupt
tar archives, including a potentially exploitable buffer overflow.

Approved by: re (kensmith, security blanket)
Reviewed by: kientzle
Security: FreeBSD-SA-07:05.libarchive

Fix installworld: /usr/bin/printf isn't available then,
so use awk's printf for the formatting here instead.

Pointy hat: Yours Truly
Approved by: re

New "version stamp" simplifies determining the exact version
of libarchive being used. I've been taking advantage of this
with a recent round of updates to libarchive_test so that it
can test older and newer versions of the library.

Approved by: re (Ken Smith)

Support for writing the 'newc' cpio format, plus a minimal test harness
for the cpio formats.

Thanks to: Rudolf Marek
Approved by: re@

libarchive 2.2.3
* "compression_program" support uses an external program
* Portability: no longer uses "struct stat" as a primary
data interchange structure internally
* Part of the above: refactor archive_entry to separate
out copy_stat() and stat() functions
* More complete tests for archive_entry
* Finish archive_entry_clone()
* Isolate major()/minor()/makedev() in archive_entry; remove
these from everywhere else.
* Bug fix: properly handle decompression look-ahead at end-of-data
* Bug fixes to 'ar' support
* Fix memory leak in ZIP reader
* Portability: better timegm() emulation in iso9660 reader
* New write_disk flags to suppress auto dir creation and not
overwrite newer files (for future cpio front-end)
* Simplify trailing-'/' fixup when writing tar and pax
* Test enhancements: fix various compiler warnings, improve
portability, add lots of new tests.
* Documentation: document new functions, first draft of
libarchive_internals.3

MFC after: 14 days
Thanks to: Joerg Sonnenberger (compression_program)
Thanks to: Kai Wang (ar)
Thanks to: Colin Percival (many small fixes)
Thanks to: Many others who sent me various patches and problem reports.

s/@VERSION@/@ARCHIVE_VERSION@/

This is a no-op as far as FreeBSD is concerned, but makes libarchive
more autoconf-friendly.

Approved by: kientzle

Consolidate numeric limit macros in one place; include them
only on platforms that need them. FreeBSD doesn't.

Overhaul of 'ar' support:
* use "AR_GNU" as the format name instead of AR_SVR4 (it's what everyone is going to call it anyway)
* Simplify numeric parsing to unsigned (none of the numeric values should ever be negative); don't run off end of numeric fields.
* Finish parsing the common header fields before the next I/O request (which might dump the contents)
* Be smarter about format guessing and trimming filenames.
* Most of the magic values are only used in one place, so just inline them.
* Many more comments.
* Be smarter about handling damaged entries; return something reasonable.
* Call it a "filename table" instead of a "string table"
* Update tests.

Enable selection of 'ar', 'arbsd', and 'argnu' formats by name
(this allows bsdtar to create ar format archives).

The 'ar' writer still needs some work; it should reject
entries that aren't regular files and should probably also
strip leading paths from filenames.

Fix the build by temporarily disabling 'ar' support until
I can clean it up.

Fix the build.
N.B. 'ar' format support is broken right now, it's not
passing tests. If I can't find the problem soon, I'll
back out the last commit.

Conventionally, tar archives have always included a trailing '/'
for directories. bsdtar used to add this, but that recently got
lost somehow. So now I'm adding it back in libarchive.
The only odd part of doing this in libarchive: Adding a directory to
a tar archive and then reading it back again can yield a different name.

Add a test case to exercise some boundary conditions with
tar filenames and ensure that trailing slashes are added to
dir names only as necessary.

Thanks to: Oliver Lehmann for bringing this regression to my attention.

Now that libarchive is being built in more environments,
factor out the platform-specific configuration header a bit
more cleanly.
Suggested by: Joerg Sonnenberger

Enable 'ar' support; hook it up to the build and
enable it with _read_support_format_all().

Move archive_read_data_into_buffer into archive_read.c, simplify its
implementation, and mark it as deprecated. It will be removed entirely
in libarchive 3.0 (in FreeBSD 8.0?) but there's no reason for anyone to
use it instead of archive_read_data.

Approved by: kientzle

Notice when mkdir() fails.
Don't change permissions on an existing dir unless _EXTRACT_PERM
is requested.

In particular, bsdtar -x should not edit mode of existing dirs
now; bsdtar -xp will.

When ARCHIVE_EXTRACT_UNLINK is requested:
* Only try to remove the existing item if we're not restoring a directory.
* If unlink fails, try rmdir next.
This should fix the broken --unlink option in bsdtar.

Thanks again to: Kris Kennaway, for beating up bsdtar on pointyhat.

Libarchive 2.0.23:
* The ACL formatter was mis-formatting entries which had a
user/group ID but no name. Make the parser tolerant of
these, so that old archives can be correctly restored;
fix the formatter to generate correct entries.
* Fix overwrite detection by introducing a new "FAILED" return
code that indicates the current entry cannot be continued
but the archive as a whole is still sound.
* Header cleanup: Remove some unused headers, add some that
are required with new Linux systems.

libarchive 2.0
* libarchive_test program exercises many of the core features
* Refactored old "read_extract" into new "archive_write_disk", which
uses archive_write methods to put entries onto disk. In particular,
you can now use archive_write_disk to create objects on disk
without having an archive available.
* Pushed some security checks from bsdtar down into libarchive, where
they can be better optimized.
* Rearchitected the logic for creating objects on disk to reduce
the number of system calls. Several common cases now use a
minimum number of system calls.
* Virtualized some internal interfaces to provide a clearer separation
of read and write handling and make it simpler to override key
methods.
* New "empty" format reader.
* Corrected return types (this ABI breakage required the "2.0" version bump)
* Many bug fixes.

Add support for a new archive format "empty" that reads empty files.

Bump the libarchive version number, correct the shell hackery to break
the full version down into major/minor values.

Write-blocking cleanup, largely thanks to Colin Percival (cperciva@).
* If write block size is zero, don't block at all.
This supports the unusual requirement of applications
that need "no-delay" writes.
* Expose _write_finish_entry() to give such applications more
control over write boundaries. (Normal applications do not
need this, as entries are completed automatically.)
* Correct the type of write callbacks; this is a minor API
change that does not affect the ABI.
* Correct the error handling in _write_next_header() around
completing the previous entry.
* Correct the documentation for block-size markers: Remove
docs for the long-defunct _read_set_block_size(); document
all of the write block size manipulators.

MFC after: 14 days

Unbreak libarchive on arm. Two parts of libarchive relied on a
traditional shortcut of defining on-disk layouts using structures of
character arrays. Unfortunately, as recently discussed on cvs-all@,
this usage is not actually sanctioned by the standards and
specifically fails on GCC/arm (unless your data structures happen to
be "naturally aligned").

The new code defines offsets/sizes for data fields and accesses
them using explicit pointer arithmetic, instead of casting to
a structure and accessing structure fields. In particular,
the new code is now clean with WARNS=6 on arm.

MFC after: 14 days

Connect four new files to the build.
PR: bin/86742

Officially rename archive_{read,write}_open_file() to
archive_{read,write}_open_filename():
* Update Makefile to build the files using the new name.
* Update docs to document the new names, mentioning the
old ones as "deprecated synonyms."
* The old filenames will be reconnected to the build soon;
I'll soon recyce those files for a slightly different purpose.

Eliminate documentation references to a non-existent function.

Computing SHLIB_MAJOR is not a good idea. It's really a FreeBSD
system value that has no real relation to the libarchive version.
(Except, of course, that any ABI breakage will force both to be
incremented.)

Some minor corrections:
* Expose functions for setting the "skip file" dev/ino information
* Expose functions for setting/querying the block size on reads
* Correctly propagate errors out of archive_read_close/archive_write_close
* Update manpage with information about new functions

Simplify some of the wide-character handling, inspired
in part by OpenBSD's not-quite-standard-compliant
standard libraries. (No loss of functionality,
just minor recoding to not rely on certain "standard"
facilities that weren't actually needed.)

POSIX.1e-style Extended Attribute support

This commit implements storing/reading POSIX.1e-style extended
attribute information in "pax" format archives. An outline of the
storage format is in the tar.5 manpage. The archive_read_extract()
function has code to restore those archives to disk for Linux; FreeBSD
implementation is forthcoming.

Many thanks to Jaakko Heinonen for finding flaws in earlier
proposals and doing the bulk of the coding in this work.

The idea of supporting 'tp' was a fun one, but it is
really not worth the effort to develop and maintain
support for a format that hasn't been used for 30 years. ;-/

Remove configure.ac.in and reorganize a few other things. This is
part of a program to remove the non-FreeBSD autoconf/automake build
system for libarchive from the FreeBSD source tree.

Fix an aliasing error in the new TP support and reenable it in the build.

Disable "tp" support until I figure out why it's breaking the build. <sigh>

Add support for "tp" format. tp was the standard system
archiver for Fourth Edition through Sixth Edition Unix; it was
replaced by tar in Seventh Edition. (First Edition through
Third Edition used "tap.")

Unfortunately, tp was not so very standard; there were a
few different variants. The code here attempts to support
what I believe were the most common variants.

tp support is not yet enabled by archive_read_support_format_all(),
as I'm not yet entirely comfortable with the detection
heuristics. People interested in experimenting can
add archive_read_support_format_tp() just after any calls
to archive_read_support_format_all() in bsdtar to see how
well this works.

TODO: tp format is roughly similar in structure to dump/restore
archive formats used by many systems. It should be possible
to generalize this code to handle many dump/restore variants.
Format detection heuristics are going to be rough, though.

Thanks to: Warren Toomey, whose very basic tp extraction programs
and documentation made this possible.

Add missing shared library interdependencies.

Fix installworld breakage. <sigh>

expr and printf are not available during installworld, so
use /bin/sh arithmetic expansion instead of expr and simply
give up on vanity formatting. ;-)

1) Use GNU libtool to build shared libraries on non-FreeBSD
systems (or on FreeBSD systems when using ports).

2) Overhaul the versioning logic. In particular,
SHLIB_MAJOR number is now computed as "major+minor",
which ensures library versions are the same for
the FreeBSD build system and the portable
libtool/autoconf/automake build system.

Add a lot of error checks, based on the patches provided by Dan Lukes.
Also fixes a memory leak reported by Andrew Turner.

PR: bin/83476
Thanks to: Dan Lukes, Andrew Turner

Style issue: Don't include <wchar.h> where it is not actually needed.
(wchar_t is defined in stddef.h, and only two files need more than that.)

Portability: Since the wchar requirements are really quite modest,
it's easy to define basic replacements for wcslen, wcscmp, wcscpy,
etc, for use on systems that lack <wchar.h>. In particular, this allows
libarchive to be used on older OpenBSD systems.

Bump the shared library version number of all libraries that have not
been bumped since RELENG_5.

Reviewed by: ru
Approved by: re (not needed for commit check but in principle...)

Minor clean up for flags restoration: Use fchflags/lchflags when
available, stub out flags restore on platforms that don't support it,
update autoconf to probe for fchflags and lchflags support.

Start to address the race issue between restoring a file's contents
and restoring the metadata. In particular, the metadata-restore
functions now all accept a file descriptor and a pathname. If the
file descriptor is set and the platform supports the appropriate
syscall, restore the metadata through the file descriptor. Otherwise,
restore it through the pathname. This is complicated by varying
syscall support (FreeBSD has an fchmod(2) but no fchflags(2), for
example) and because non-file entries don't have an fd to use in
restoring attributes (for example, mknod(2) doesn't return a file
handle).

MFC after: 14 days

Fix broken ACL configuration on FreeBSD 4 and Linux.

Thanks to: Greg Lewis, Juergen Lock, Jaakko Heinonen for reporting and testing

Update "make distfile" to use newest automake/autoconf from ports.

Thanks to: Juergen Lock

Clean up the support for extracting very long pathnames.

Support extracting entries with pathnames longer than PATH_MAX. In
testing, I've archived and restored dir trees with ~1MB pathnames.
Most formats, of course, have much smaller limits.

Fix a buffer overflow in the "none" decompression handler that
occurred with large read-ahead requests. This only affected
formats that incorrectly make large requests (ZIP did this until
recently) or with block sizes over 32k.

Basic support for ZIP archives.
Only supports "deflate" and "none" compression for now.

Also, add a few clarifications to the archive_read.3 manpage as
requested by William Dean DeVries.

Documentation improvements. In particular,
expand and clarify the description of the client
callback functions and how they should handle errors.

Thanks to: Antony Dovgal

First cut support for extracting from ISO9660 disk images.
This seems to be able to extract a TOC and extract files from
the couple of ISO images I've tested it with.

Treat this as experimental proof-of-concept code for the
moment. There are still a bunch of debug messages (there
are a few oddities in ISO9660 that I haven't yet figured
out how to handle), a lot of bugs to be addressed (this
code leaks memory very badly), and a lot of missing features (no
Rockridge support, in particular). I'd appreciate
feedback from anyone who understands ISO9660 format
better than I do. ;-)

Suggested by: Robert Watson

Makefile tweaks:
* Update Version
* Add a missing MLINK
* Fix 'distfile' target so it works from a clean checkout

Revert 1.24: Brain glitch. <sigh>

archive.h gets built in ${.OBJDIR}

Build as a shared lib again.

Discussed with: kientzle

Forced commit to remind me to...
MFC after: 3 days

Per Ruslan, bsd.lib.mk already has support for dynamically-generated
.h files. This simplifies the Makefile here a bit and makes it behave
better in a couple of situations. While I'm here, clean up some comments
and try to improve the organization a bit.

Thanks to: Ruslan Ermilov (The Marvelous Makefile Guru)

Eliminate reliance on non-portable <err.h> by implementing a very
simple errx() function.
Improve behavior when bzlib/zlib are missing by detecting and
issuing an error message on attempts to read gzip/bzip2 compressed
archives.

Split archive_{read,write}_finish into separate "close" (finish the archive
and close it) and "finish" (destroy the object) functions. For backwards
compat and simplicity, have "finish" invoke "close" transparently if needed.
This allows clients to close the archive and check end-of-operation
statistics before destroying the object.

Fix the calculation of the most negative int64_t value, which
is used on systems that lack C99 headers (such as FreeBSD 4).

libarchive now has two complete build systems. The usual "Makefile"
is present for FreeBSD. If you "make distfile" on FreeBSD, you will
soon have a tar.gz file suitable for deploying to other systems
(complete with the expected "configure" script, etc). This latter
relies (at least for now) on the GNU auto??? tools. (I like autoconf
okay, but someday I hope to write a custom Makefile.in and dispense
with automake, which is somewhat odious.)

As part of this, I've cleaned up some of the conditional
compilation options, added make-foo to construct archive.h dynamically
(it now contains some version constants), and added some useful
informational files.

For the "portable" distribution, the configure script will overwrite
"Makefile," so I'm moving all the FreeBSD build machinery to
"Makefile.freebsd", with the default "Makefile" containing a single
include.

When writing "pax" format, readers are supposed to ignore fields
in the regular ustar header that are overridden by the pax
extended attributes. As a result, it makes perfect sense to
use numeric extensions in the regular ustar header so that readers
that don't understand pax extensions but do understand some other
extensions can still get useful information out of it.

This is especially important for filesizes, as the failure to
read a file size correctly can get the reader out of sync.

This commit introduces a "non-strict" option into the internal
function to format a ustar header. In non-strict mode, the formatter
will use longer octal values (overwriting terminators) or binary
("base-256") values as needed to ensure that large file sizes,
negative mtimes, etc, have the correct values stored in the regular
ustar header.

Define the PACKAGE_NAME and PACKAGE_VERSION macros.

Add two new convenience functions to query the uid/gid from an
archive_entry.

Update the Makefile MLINKS and manpage to bring it up-to-date with
the current status of archive_entry. At least the manpage actually
lists all of the functions now, even if it doesn't really yet explain
them all.

Add MLINKS for new API functions.

Connect libarchive decompress support to the build.
Also, add it to archive_read_support_compression_all()
so that typical clients get it pulled in by default.

Use WARNS?=3 for these in the arm case for now, due to toolchain issues.

Add MLINK for newly-added archive_read_extract_set_progress_callback(3).

Add the libarchive-formats.5 manpage. This summarizes the various
archive formats supported by libarchive, with some information about
the relative strengths and weaknesses of each format and notes about
issues with libarchive's support for those formats.

This page should make it unnecessary to list all of the libarchive
formats in the manpage of each program that uses libarchive.
Such programs can simply refer to libarchive-formats(5).

Update file flag handling.

The new fflags support in archive_entry supports Linux and FreeBSD
file flags and is a bit more gracious about unrecognized flag names
than strtofflags(3). This involves some minor API breakage.

The default tar format ("restricted pax") now enables pax extensions
when archiving files that have flags. In particular, copying dir
heirarchies with 'bsdtar cf - -C src . | bsdtar xpf - -C dest' now
preserves file flags. (Note the "p" on extract!)

While I'm here, fill in some additional explanation in the
archive_entry.3 manpage, fill in some missing MLINKS, mark some
overlooked internal functions 'static', and make a few minor style
fixes.

More work on ACLs: fix error in archive_entry's ACL parsing code,
try to set ACLs even if fflag restore fails, first cut at reading
Solaris tar ACLs

Code improvement: merge gnu tar read support into main tar reader;
this eliminates a lot of duplicate code and generalizes the tar
reader to handle formats with GNU-like extensions.

Style: Makefile cleanup, eliminate 'dmalloc' references, remove 'tartype'
from archive_entry (this makes archive_entry more format-agnostic)

Thanks to: David Magda for providing Solaris tar test files

Overhauled ACL support. This makes us compatible
with 'star' ACL handling, though there's still a
bit more work needed in this area.

Added 'write_open_fd' and 'read_open_fd' to simplify, e.g.,
tar's u and r modes. Eliminated old 'write_open_file_position'
as a bad idea. (It required closing/reopening files to
do updates, which led to unpleasant implications.)

Various other minor fixes, API tweaks, etc.

Many fixes:

* Disabled shared-library building, as some API breakage is
still likely. (I didn't realize it was turned on by default.) If
you have an existing /usr/lib/libarchive.so.2, I recommend deleting it.
* Pax interchange format now correctly stores and reads UTF8
for extended attributes. In particular, pax format can portably
handle arbitrarily long pathnames containing arbitrary characters.
* Library compiles cleanly at -O2, -O3, and WARNS=6 on all
FreeBSD-CURRENT platforms.
* Minor portability improvements inspired by Juergen Lock
and Greg Lewis. (Less reliance on stdint.h, isolating of
various portability-challenged constructs.)
* archive_entry transparently converts multi-byte <-> wide character
strings, allowing clients and format handlers to deal with either
one, as appropriate.
* Support for reading 'L' and 'K' entries in standard tar archives
for star compatibility.
* Recognize (but don't yet handle) ACL entries from Solaris tar.
* Pushed format-specific data for format readers down into
format-specific storage and out of library-global storage. This
should make it easier to maintain individual formats without mucking
with the core library management.
* Documentation updates to track the above changes.
* Updates to tar.5 to correct a few mistakes and add some additional
information about GNU tar and Solaris tar formats.

Notes:
* The basic 'tar' reader is getting more general; there's not much
point in keeping the 'gnutar' reader separate. Merging the two
would lose a bunch of duplicate code.
* The libc ACL support is looking increasingly inadequate for my needs
here. I might need to assemble some fairly significant code for
parsing and building ACLs. <sigh>

Many fixes.

Portability: Thanks to Juergen Lock, libarchive now compiles cleanly
on Linux. Along the way, I cleaned up a lot of error return codes and
reorganized some code to simplify conditional compilation of certain
sections.

Bug fixes:
* pax format now actually stores filenames that are 101-154
characters long.
* pax format now allows newline characters in extended attributes
(this fixes a long-standing bug in ACL handling)
* mtime/atime are now restored for directories
* directory list is now sorted prior to fix-up to permit
correct restore of non-writable dir heirarchies

libarchive itself is now completely clean at WARNS=10 on all architectures.

Unfortunately, the stock zlib.h is not:
line 885: 'err' parameter shadows global 'err' definition from <err.h>

Back the WARNS level down to 3 to accomodate borked zlib.h.

Initial import of libarchive.

What it is:
A library for reading and writing various streaming archive
formats, especially tar and cpio. Being a library, it should
be easy to incorporate into pkg_* tools, sysinstall, and any
other place that needs to read or write such archives.

Features:
* Full automatic detection of both compression and archive format.
* Extensible internal architecture to make it easy to add new formats.
* Support for "pax interchange format," a new POSIX-standard tar format
that eliminates essentially all of the restrictions of historic formats.
* BSD license

Thanks to: jkh for pushing me to start this work, gordon for
encouraging me to commit it, bde for answering endless style
questions, and many others for feedback and encouragement.

Status: Pretty good overall, though there are still a few rough edges and
the library could always use more testing. Feedback eagerly solicited.