Remove ZSTD_ENABLED checks from the Package Kit.

BZstdCompressionAlgorithm already has these #ifdefs and will
return error codes appropriately if built without libzstd,
so we do not need to check again inside the Package Kit.

This should not break the no-zstd build (well, it is broken
somewhat right now anyway, but this will not break it further),
and it simplifies logic somewhat.

packagefs: Remove some more HPKGv1-related classes.

These were now unreferenced following the removal of GlobalFactory
and the other v1-related code, so this commit should not be any
functional change.

packagefs: Remove support for HPKGv1.

The current version is "v2", and it has been since the final PM branch
was merged back in 2013, so it's unclear if anyone actually uses this
file format. The "package" command can still of course read these
files, it's just that packagefs will now not be able to.

It's very unlikely anyone is impacted by this change.

packagefs: Move DebugSupport up into the file_systems shared folder.

ramfs will find it useful also.

Package Kit: add Zstd compression.

Change-Id: Idbdb7cf1bde659046a88ea69a76e3b5fc4cd7013
Reviewed-on: https://review.haiku-os.org/323
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

NaturalCompare: implement using locale kit

The non-locale aware version is kept in src/build/libshared for use on
the host system and in packagefs (kernel add-on). In both cases, ICU is
not available.

Fixes #8192

Use libstdc++, libsupc++ and libgcc from gcc_syslibs.

* Instead of faking libstdc++.so from libstdc++.a, use libstdc++.so
from the gcc_syslibs build feature for everything except x86_gcc2.
* Use libgcc_s.so from the gcc_syslibs build feature for everything but
x86_gcc2 (which still carries libgcc as part of libroot.so).
* Drop filtering of libgcc objects for libroot, as that is no longer
necessary since we're only using libgcc-as-single-object for libroot
with x86_gcc2, where the filtered object file doesn't exist. Should
the objects that used to be filtered cause any problems as part of
libgcc_s.so, we can always filter them as part of the gcc build.
* Use libsupc++.so from the gcc_syslibs build feature for everything but
x86_gcc2.
* Adjust all Jamfiles accordingly.
* Deactivate building of faked libstdc++.so for non-x86-gcc2. For
x86_gcc2, we still build libstdc++.so from the sources in the Haiku
source tree as part of the Haiku build .
* Put gcc_syslibs package onto the image, when needed.

Switch package file accessor classes to use BPositionIO

* PackageFileHeap{Reader,Writer} as well as Package{Reader,Writer} and
their implementation and super classes do now internally use a
BPositionIO instead of a FD to access the package file. This provides
more flexibility needed for features to come.
* BPackageReader has already grown a new Init() version with a
BPositionIO* parameter.

Switch package kit to BZlibCompressionAlgorithm

... and remove the Zlib{Compressor,Decompressor} API.

Move package kit Zlib* classes to support kit

Also move to B* namespace and no longer expose the zlib dependency in
the headers.

Remove BPackageKit::BDataOutput

Use BDataIO instead.

Revert "Move ZlibDecompressor to libshared"

This reverts commit 9af2105d36856be60528c361edf14a263f242338.

Conflicts:
src/kits/package/Jamfile

Revert "Fix zlib-dependencies of (boot & standard) packagefs."

This reverts commit 06ed40580339e21f65c24b3d210c880dcf2ec739.

Fix zlib-dependencies of (boot & standard) packagefs.

* Some files that depend on zlib.h weren't declared as such, which
could cause multijob builds to fail.

packagefs: Move Volume::PackagesDirectory to top level

Also make it a BReferenceable and fix Volume::fPackagesDirectory leak.

Move ZlibDecompressor to libshared

* This will be used to implement compressed http streams
* Remove the custom BDataOutput class, and use BDataIO instead, for
easier integration with existing code.

packagefs: Refactor UnpackingAttributeDirectoryCookie

Pull out a reusable base class AutoPackageAttributeDirectoryCookie.

packagefs: Support blacklisting entries in packages

In each installation location, it is now possible to create a settings
file "packages" that allows to blacklist entries contained in packages.
The format is:

Package <package name> {
EntryBlacklist {
<entry path>
...
}
}
...

<package name> is the base name (no version) of the respective package
(e.g. "haiku"), <entry path> is an installation location relative path
(e.g. "add-ons/Translators/FooTranslator").

Blacklisted entries will be ignored by packagefs, i.e. they won't appear
in the file system. This addresses the issue that it may be necessary to
remove a problematic file (e.g. driver, add-on, or library), which would
otherwise require editing the containing package file.

The settings file is not not "live". Changes take effect only after
reboot (respectively when remounting the concerned packagefs volume).

Revert "Fix build problems of btrfs and packagefs for x86_64."

This reverts commit f272e63f393449d3a816970ca632764895a78cd6.

Fix build problems of btrfs and packagefs for x86_64.

* Both filesystems used to link to a static kernel-zlib, which
was being built with -fno-pic. This doesn't work on x86_64 as the
filesystem add-ons are meant to be relocatable, which requires their
code to be compiled as position independent.
Solve that by moving zlib into the kernel, so any add-on can just use
it from there (packagefs is mandatory, so we can't really do without
zlib anyway).

More work towards hybrid support

* All packaging architecture dependent variables do now have a
respective suffix and are set up for each configured packaging
architecture, save for the kernel and boot loader variables, which
are still only set up for the primary architecture.
For convenience TARGET_PACKAGING_ARCH, TARGET_ARCH, TARGET_LIBSUPC++,
and TARGET_LIBSTDC++ are set to the respective values for the primary
packaging architecture by default.
* Introduce a set of MultiArch* rules to help with building targets for
multiple packaging architectures. Generally the respective targets are
(additionally) gristed with the packaging architecture. For libraries
the additional grist is usually omitted for the primary architecture
(e.g. libroot.so and <x86>libroot.so for x86_gcc2/x86 hybrid), so that
Jamfiles for targets built only for the primary architecture don't
need to be changed.
* Add multi-arch build support for all targets needed for the stage 1
cross devel package as well as for libbe (untested).

packagefs: Add caching for the package file heap reader

* ReaderImplBase:
- Add virtual CreateCachedHeapReader() which can create a cached
reader based on the given heap reader.
- Rename HeapReader() to RawHeapReader() and add HeapReader() for the
cached heap reader.
- Add DetachHeapReader() to allow a clients to remove the heap
reader(s) after deleting the ReaderImplBase object.
* packagefs:
- Add CachedDataReader class, which wraps a given
BAbstractBufferedDataReader and provides caching for it using a
VMCache. The implementation is based on the IOCache implementation.
- Use CachedDataReader to wrap the heap reader. For file data that
means they are cached twice -- in the heap reader cache and in the
file cache -- but due to the heap reader using a VMCache as well,
the pages will be recycled automatically anyway. For attribute data
the cache should be very helpful, since they weren't cached at all
before.

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.

Rename BBlockBufferCache and friends to *Pool*

Also move BBlockBufferPoolNoLock into BHPKG namespace with the other
classes. Not sure why it wasn't there before.

packagefs: Add support for hpkg format V1

* Add union-like class PackageData which wraps the V1 and V2
BPackageData classes.
* GlobalFactory: Create a data reader depending on the data format
version.
* Package: Add a loader for V1 format and try that, if the other one
fails.

package kit: some fixes for multi-version support

* Use enums/constants/functions instead of preprocessor macros.
* Missing include in PackageInfoAttributeValue.h.
* PackageReaderImpl::Init(): Check version before header size and
return B_MISMATCHED_VALUES instead of B_BAD_DATA, if the version
doesn't match. This allows callers to determine the condition and
try a reader for a different version. A more flexible interface for
that case would be nice, but since we want to support the old package
version only temporarily, the current solution should be good enough.

packagefs: Share string instances globally

Introduce a class String which refers to shared private data that is
registered in a global hash table (in class StringPool) and use the
class consequently. This eliminates duplicate allocations for the same
string and also speeds up tests for equality. There's quite a bit
overhead for the management structures (apparently even more than for
the string data itself), but due to almost all strings being used
multiple times this still almost halves the memory usage for string
data.

packagefs: Organize sources into subdirectories

packagefs: Remove no longer working userland build support

Fix race condition during build of some file systems.

* make sure the zlib package is available before trying to use zlib.h

Complete outsourcing of zlib

* Add optional packages Zlib and Zlib-devel.
* Simplify the build feature section for zlib and also extract the
source package.
* Replace all remaining references to the zlib instance in the tree and
remove it.

packagefs: Remove support for multiple package domains per volume

That also get rid of the job stuff and the package loader thread.

packagefs: Add ioctl to get basic volume information

Also rename the MountType enum and members, since they are no longer
packagefs private.

Build a separate libsupc++ for the kernel with correct flags.

Kernel mode code on x86_64 needs to be built with -mno-red-zone as
interrupts would corrupt the red zone if it were in use. However, the
kernel is linked with libsupc++, which was not compiled with
-mno-red-zone. If an interrupt occurred in libsupc++ code the red zone
would get corrupted. This was causing random panics, particularly under
heavy system load. Therefore, on x86_64 a separate build of libsupc++
with -mno-red-zone is now done for the kernel to use. Note: this commit
will require a rerun of configure and rebuild of cross tools.

Attribute index support for non-special attributes

* Add AttributeIndex class.
* Each attribute does now have an attribute index cookie. The new
attribute index service methods Node::IndexAttribute() and
IndexCookieForAttribute() create+set/retrieve the cookie. The cookie
is actually the attribute index's tree node.
* Add OldNodeAttribute::IndexCookieForAttribute() so the cookie is
available when the node changes.

Move helper function out of QueryParser.h

They live in the new QueryParserUtils.{h,cpp} now.

Add last modified index support

* Add class LastModifiedIndex.
* Create and add an instance of it in Volume::Mount().

Add size index support

* Add class SizeIndex.
* Create and add an instance of it in Volume mount.

NodeListener::NodeChanged(): Provide old value

* Add interface OldNodeAttributes an instance of which is passed to
NodeListener::NodeChanged() to provide the old attribute values
(currently only modification time and file size).
* Also extend PackageLinksListener::PackageLinkNodeChanged() with a
OldNodeAttributes parameter.
* Add OldNodeAttributes implementations for PackageLinkSymlink (inner
class OldAttributes) and UnpackingNode (OldUnpackageNodeAttributes).

Add Query class and query management in Volume

Add NameIndex class and create instance in Volume

Add Index base class and index table in Volume

Add a global node listener mechanism

Support for auto-generated package node attributes

* Add utility class AutoPackageAttributes and AttributeCookie subclass
AutoPackageAttributeCookie.
* UnpackingAttributeCookie::Open() does now try to create a
AutoPackageAttributeCookie, when it doesn't find the attribute in the
PackageNode.
* Adjust UnpackingAttributeDirectoryCookie to also list auto-generated
attributes.

Currently the only supported attribute is "SYS:PACKAGE", which is the
file name of the package containing the node.

Move PackageLinkDirectory::Link to own file

And rename it to PackageLinkSymlink.

WIP towards dependency package links

* Add ResolvableFamily and DependencyFamily classes for grouping equally
named resolvables/dependencies.
* Add hash tables to PackageFSRoot to track resolvables and
dependencies. That is done in {Add,Remove}Package.
* Resolvable does now have a list of Dependency objects, the ones it
satisfies.

Add package link directories to system volume

* Add PackageLinksListener interface. It is used by
PackageLink[s]Directory to notify a listener about changes.
* Volume does now implement PackageLinksListener and recursively adds/
removes package link directories.

This makes the package link directories appear at least. They are still
empty.

WIP: Create empty package links directories

* Rename PackageFamily to PackageLinkDirectory and derive from
Directory.
* PackageLinksDirectory: Add the PackageLinkDirectory objects as
children and remove the no longer needed hash table (we use the
Directory's child hash table).

Move EmptyAttributeDirectoryCookie to own files

Add yet empty package-links directory

* Add PackageLinksDirectory Directory subclass. Currently not doing
anything.
* PackageFSRoot: Create a PackageLinksDirectory.
* Volume: Add/remove the package links directory for the system volume.

Pull derived UnpackingDirectory out of Directory

Move all package specifics from now abstract Directory to new derived
class UnpackingDirectory and adjust the Volume implementation
accordingly. This concludes the Node/Directory refactoring. Neither
class is aware of packages anymore.

Rename LeafNode to UnpackingLeafNode

Remove package related dependencies from Node

* Move package related methods from Node to new interface UnpackingNode.
* LeafNode and Directory derive from UnpackingNode now.
* Adjust Volume implementation accordingly.

Attribute dir access kernel interface abstraction

* Introduce interface AttributeDirectoryCookie and currently only
implementation UnpackingAttributeDirectoryCookie. This is an interface
for reading/rewinding an attribute directory.
* Add abstract virtual Node::OpenAttributeDirectory() method that
returns an AttributeDirectoryCookie and implement it for derived
classes.
* In the kernel interface attribute directory hooks use
AttributeDirectoryCookie now.

Kernel interface abstraction for attribute access

* Introduce interface AttributeCookie and currently only implementation
UnpackingAttributeCookie. This is an interface for reading/stat()ing
an attribute.
* Add abstract virtual Node::OpenAttribute() method that returns an
AttributeCookie and implemented it for derived classes.
* In the kernel interface attribute hooks use AttributeCookie now. The
attribute directory hooks are unchanged.

WIP towards packagefs package links support

* Introduce Version class representing a version.
* Introduce Dependency and Resolvable class and add lists of either to
Package.
* Parse package attributes and add dependencies and resolvables to
Package.
* Add a mount type to Volume and add a respective mount parameter
"mount-type" (values "system", "common", "home", "custom"). Also
implies the shine-through type, if that's not given.
* Introduce class PackageFamily which groups equally named and versioned
packages.
* Add class PackageFSRoot. Each instance represents a possible file
system root (separate roots for different chroot environments). Tracks
Volumes belonging to the same root and their packages.

Split package kit DataReader.cpp

Pull AttributeDataReader and FDDataReader implementations out of
DataReader.cpp into own source files. Thus we can avoid dependencies
(e.g. to fs_attr code) we don't need/want.

Attribute index support for non-special attributes

* Add AttributeIndex class.
* Each attribute does now have an attribute index cookie. The new
attribute index service methods Node::IndexAttribute() and
IndexCookieForAttribute() create+set/retrieve the cookie. The cookie
is actually the attribute index's tree node.
* Add OldNodeAttribute::IndexCookieForAttribute() so the cookie is
available when the node changes.

Move helper function out of QueryParser.h

They live in the new QueryParserUtils.{h,cpp} now.

Add last modified index support

* Add class LastModifiedIndex.
* Create and add an instance of it in Volume::Mount().

Add size index support

* Add class SizeIndex.
* Create and add an instance of it in Volume mount.

NodeListener::NodeChanged(): Provide old value

* Add interface OldNodeAttributes an instance of which is passed to
NodeListener::NodeChanged() to provide the old attribute values
(currently only modification time and file size).
* Also extend PackageLinksListener::PackageLinkNodeChanged() with a
OldNodeAttributes parameter.
* Add OldNodeAttributes implementations for PackageLinkSymlink (inner
class OldAttributes) and UnpackingNode (OldUnpackageNodeAttributes).

Add Query class and query management in Volume

Add NameIndex class and create instance in Volume

Add Index base class and index table in Volume

Add a global node listener mechanism

Support for auto-generated package node attributes

* Add utility class AutoPackageAttributes and AttributeCookie subclass
AutoPackageAttributeCookie.
* UnpackingAttributeCookie::Open() does now try to create a
AutoPackageAttributeCookie, when it doesn't find the attribute in the
PackageNode.
* Adjust UnpackingAttributeDirectoryCookie to also list auto-generated
attributes.

Currently the only supported attribute is "SYS:PACKAGE", which is the
file name of the package containing the node.

Move PackageLinkDirectory::Link to own file

And rename it to PackageLinkSymlink.

WIP towards dependency package links

* Add ResolvableFamily and DependencyFamily classes for grouping equally
named resolvables/dependencies.
* Add hash tables to PackageFSRoot to track resolvables and
dependencies. That is done in {Add,Remove}Package.
* Resolvable does now have a list of Dependency objects, the ones it
satisfies.

Add package link directories to system volume

* Add PackageLinksListener interface. It is used by
PackageLink[s]Directory to notify a listener about changes.
* Volume does now implement PackageLinksListener and recursively adds/
removes package link directories.

This makes the package link directories appear at least. They are still
empty.

WIP: Create empty package links directories

* Rename PackageFamily to PackageLinkDirectory and derive from
Directory.
* PackageLinksDirectory: Add the PackageLinkDirectory objects as
children and remove the no longer needed hash table (we use the
Directory's child hash table).

Move EmptyAttributeDirectoryCookie to own files

Add yet empty package-links directory

* Add PackageLinksDirectory Directory subclass. Currently not doing
anything.
* PackageFSRoot: Create a PackageLinksDirectory.
* Volume: Add/remove the package links directory for the system volume.

Pull derived UnpackingDirectory out of Directory

Move all package specifics from now abstract Directory to new derived
class UnpackingDirectory and adjust the Volume implementation
accordingly. This concludes the Node/Directory refactoring. Neither
class is aware of packages anymore.

Rename LeafNode to UnpackingLeafNode

Remove package related dependencies from Node

* Move package related methods from Node to new interface UnpackingNode.
* LeafNode and Directory derive from UnpackingNode now.
* Adjust Volume implementation accordingly.

Attribute dir access kernel interface abstraction

* Introduce interface AttributeDirectoryCookie and currently only
implementation UnpackingAttributeDirectoryCookie. This is an interface
for reading/rewinding an attribute directory.
* Add abstract virtual Node::OpenAttributeDirectory() method that
returns an AttributeDirectoryCookie and implement it for derived
classes.
* In the kernel interface attribute directory hooks use
AttributeDirectoryCookie now.

Kernel interface abstraction for attribute access

* Introduce interface AttributeCookie and currently only implementation
UnpackingAttributeCookie. This is an interface for reading/stat()ing
an attribute.
* Add abstract virtual Node::OpenAttribute() method that returns an
AttributeCookie and implemented it for derived classes.
* In the kernel interface attribute hooks use AttributeCookie now. The
attribute directory hooks are unchanged.

WIP towards packagefs package links support

* Introduce Version class representing a version.
* Introduce Dependency and Resolvable class and add lists of either to
Package.
* Parse package attributes and add dependencies and resolvables to
Package.
* Add a mount type to Volume and add a respective mount parameter
"mount-type" (values "system", "common", "home", "custom"). Also
implies the shine-through type, if that's not given.
* Introduce class PackageFamily which groups equally named and versioned
packages.
* Add class PackageFSRoot. Each instance represents a possible file
system root (separate roots for different chroot environments). Tracks
Volumes belonging to the same root and their packages.

Split package kit DataReader.cpp

Pull AttributeDataReader and FDDataReader implementations out of
DataReader.cpp into own source files. Thus we can avoid dependencies
(e.g. to fs_attr code) we don't need/want.

Pulled reusable stuff from PackageReaderImpl into new class ReaderImplBase.

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40485 a95241bf-73f2-0310-859d-f6bbb57e9c96

Refactored hpkg implementation to provide some separation between
public and private API (still far from ideal, but a start):
* moved several HPKG-classes into the public namespace BPackageKit::HPKG
* added fImpl-wrappers around PackageReader and PackageWriter to hide
most of the gory details
* adjusted 'package'-binary and packagefs accordingly


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40320 a95241bf-73f2-0310-859d-f6bbb57e9c96

More work on package kit:
* added class BPackageInfo, which contains packaging attributes of
a package (the values relevant for package management)
* implemented parser (mostly) for reading a BPackageInfo from a config
file (.PackageInfo) in order to pass them on to the PackageWriter
when creating a package
* pulled hpkg-related stuff from bin/package into the package kit
* adjusted packagefs-Volume to skip .PackageInfo files when populating
the mountpoint, as those files shouldn't appear as part of an
activated package


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40301 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Extract the attributes from the package files and attach them as
PackageNodeAttribute to the PackageNodes.
* Implemented the attribute FS interface hooks.
* Made Package and PackageNode BReferenceable.
* Fixed the reference management for Node.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34119 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added new interface BufferCache -- basically a memory allocator with the
option to cache a freed buffer -- and implementations
BlockBufferCache{NoLock,Kernel}.
* ZlibPackageDataReader does now dynamically get its read and uncompress
buffers from a provided BufferCache when needed.
* Allocating the buffers once and keeping them over the whole life time was a
bit too memory heavy, since we create a reader for every file for which a
vnode is created. A FS module global factory provides a buffer cache.
* Added a mutex to PackageFile::DataAccessor which guards the access to the
data reader which isn't thread safe.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34115 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Implemented intermediate base class PackageLeafNode for PackageFile and
PackageSymlink with a "fat" interface.
* Replaced File and Symlink by fat class LeafNode.
* Added Package* object management to Directory and LeafNode.
* Implemented packagefs_read_symlink(), so symlinks work now.
* Added some missing locking the kernel interface.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34096 a95241bf-73f2-0310-859d-f6bbb57e9c96

Work in progress:
* The FS parses package files from "/boot/common/packages" and builds an
node tree representation of the contained entries. Merging directories should
work. Lots of other stuff is missing yet, though.
* Implemented the hooks for directory entry lookup and iteration. So it's
possible to see the contents of the FS at least.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34088 a95241bf-73f2-0310-859d-f6bbb57e9c96

Implemented the most basic functionality required to cleanly mount and unmount
(at least in userlandfs).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34069 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added Jamfile for building for the userlandfs.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34065 a95241bf-73f2-0310-859d-f6bbb57e9c96

The very beginning of a package FS: Empty stubs for a minimal read-only file
system.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34064 a95241bf-73f2-0310-859d-f6bbb57e9c96

Use libstdc++, libsupc++ and libgcc from gcc_syslibs.

* Instead of faking libstdc++.so from libstdc++.a, use libstdc++.so
from the gcc_syslibs build feature for everything except x86_gcc2.
* Use libgcc_s.so from the gcc_syslibs build feature for everything but
x86_gcc2 (which still carries libgcc as part of libroot.so).
* Drop filtering of libgcc objects for libroot, as that is no longer
necessary since we're only using libgcc-as-single-object for libroot
with x86_gcc2, where the filtered object file doesn't exist. Should
the objects that used to be filtered cause any problems as part of
libgcc_s.so, we can always filter them as part of the gcc build.
* Use libsupc++.so from the gcc_syslibs build feature for everything but
x86_gcc2.
* Adjust all Jamfiles accordingly.
* Deactivate building of faked libstdc++.so for non-x86-gcc2. For
x86_gcc2, we still build libstdc++.so from the sources in the Haiku
source tree as part of the Haiku build .
* Put gcc_syslibs package onto the image, when needed.

Switch package file accessor classes to use BPositionIO

* PackageFileHeap{Reader,Writer} as well as Package{Reader,Writer} and
their implementation and super classes do now internally use a
BPositionIO instead of a FD to access the package file. This provides
more flexibility needed for features to come.
* BPackageReader has already grown a new Init() version with a
BPositionIO* parameter.

Switch package kit to BZlibCompressionAlgorithm

... and remove the Zlib{Compressor,Decompressor} API.

Move package kit Zlib* classes to support kit

Also move to B* namespace and no longer expose the zlib dependency in
the headers.

Remove BPackageKit::BDataOutput

Use BDataIO instead.

Revert "Move ZlibDecompressor to libshared"

This reverts commit 9af2105d36856be60528c361edf14a263f242338.

Conflicts:
src/kits/package/Jamfile

Revert "Fix zlib-dependencies of (boot & standard) packagefs."

This reverts commit 06ed40580339e21f65c24b3d210c880dcf2ec739.

Fix zlib-dependencies of (boot & standard) packagefs.

* Some files that depend on zlib.h weren't declared as such, which
could cause multijob builds to fail.

packagefs: Move Volume::PackagesDirectory to top level

Also make it a BReferenceable and fix Volume::fPackagesDirectory leak.

Move ZlibDecompressor to libshared

* This will be used to implement compressed http streams
* Remove the custom BDataOutput class, and use BDataIO instead, for
easier integration with existing code.

packagefs: Refactor UnpackingAttributeDirectoryCookie

Pull out a reusable base class AutoPackageAttributeDirectoryCookie.

packagefs: Support blacklisting entries in packages

In each installation location, it is now possible to create a settings
file "packages" that allows to blacklist entries contained in packages.
The format is:

Package <package name> {
EntryBlacklist {
<entry path>
...
}
}
...

<package name> is the base name (no version) of the respective package
(e.g. "haiku"), <entry path> is an installation location relative path
(e.g. "add-ons/Translators/FooTranslator").

Blacklisted entries will be ignored by packagefs, i.e. they won't appear
in the file system. This addresses the issue that it may be necessary to
remove a problematic file (e.g. driver, add-on, or library), which would
otherwise require editing the containing package file.

The settings file is not not "live". Changes take effect only after
reboot (respectively when remounting the concerned packagefs volume).

Revert "Fix build problems of btrfs and packagefs for x86_64."

This reverts commit f272e63f393449d3a816970ca632764895a78cd6.

Fix build problems of btrfs and packagefs for x86_64.

* Both filesystems used to link to a static kernel-zlib, which
was being built with -fno-pic. This doesn't work on x86_64 as the
filesystem add-ons are meant to be relocatable, which requires their
code to be compiled as position independent.
Solve that by moving zlib into the kernel, so any add-on can just use
it from there (packagefs is mandatory, so we can't really do without
zlib anyway).

More work towards hybrid support

* All packaging architecture dependent variables do now have a
respective suffix and are set up for each configured packaging
architecture, save for the kernel and boot loader variables, which
are still only set up for the primary architecture.
For convenience TARGET_PACKAGING_ARCH, TARGET_ARCH, TARGET_LIBSUPC++,
and TARGET_LIBSTDC++ are set to the respective values for the primary
packaging architecture by default.
* Introduce a set of MultiArch* rules to help with building targets for
multiple packaging architectures. Generally the respective targets are
(additionally) gristed with the packaging architecture. For libraries
the additional grist is usually omitted for the primary architecture
(e.g. libroot.so and <x86>libroot.so for x86_gcc2/x86 hybrid), so that
Jamfiles for targets built only for the primary architecture don't
need to be changed.
* Add multi-arch build support for all targets needed for the stage 1
cross devel package as well as for libbe (untested).

packagefs: Add caching for the package file heap reader

* ReaderImplBase:
- Add virtual CreateCachedHeapReader() which can create a cached
reader based on the given heap reader.
- Rename HeapReader() to RawHeapReader() and add HeapReader() for the
cached heap reader.
- Add DetachHeapReader() to allow a clients to remove the heap
reader(s) after deleting the ReaderImplBase object.
* packagefs:
- Add CachedDataReader class, which wraps a given
BAbstractBufferedDataReader and provides caching for it using a
VMCache. The implementation is based on the IOCache implementation.
- Use CachedDataReader to wrap the heap reader. For file data that
means they are cached twice -- in the heap reader cache and in the
file cache -- but due to the heap reader using a VMCache as well,
the pages will be recycled automatically anyway. For attribute data
the cache should be very helpful, since they weren't cached at all
before.

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.

Rename BBlockBufferCache and friends to *Pool*

Also move BBlockBufferPoolNoLock into BHPKG namespace with the other
classes. Not sure why it wasn't there before.

packagefs: Add support for hpkg format V1

* Add union-like class PackageData which wraps the V1 and V2
BPackageData classes.
* GlobalFactory: Create a data reader depending on the data format
version.
* Package: Add a loader for V1 format and try that, if the other one
fails.

package kit: some fixes for multi-version support

* Use enums/constants/functions instead of preprocessor macros.
* Missing include in PackageInfoAttributeValue.h.
* PackageReaderImpl::Init(): Check version before header size and
return B_MISMATCHED_VALUES instead of B_BAD_DATA, if the version
doesn't match. This allows callers to determine the condition and
try a reader for a different version. A more flexible interface for
that case would be nice, but since we want to support the old package
version only temporarily, the current solution should be good enough.

packagefs: Share string instances globally

Introduce a class String which refers to shared private data that is
registered in a global hash table (in class StringPool) and use the
class consequently. This eliminates duplicate allocations for the same
string and also speeds up tests for equality. There's quite a bit
overhead for the management structures (apparently even more than for
the string data itself), but due to almost all strings being used
multiple times this still almost halves the memory usage for string
data.

packagefs: Organize sources into subdirectories

packagefs: Remove no longer working userland build support

Fix race condition during build of some file systems.

* make sure the zlib package is available before trying to use zlib.h

Complete outsourcing of zlib

* Add optional packages Zlib and Zlib-devel.
* Simplify the build feature section for zlib and also extract the
source package.
* Replace all remaining references to the zlib instance in the tree and
remove it.

packagefs: Remove support for multiple package domains per volume

That also get rid of the job stuff and the package loader thread.

packagefs: Add ioctl to get basic volume information

Also rename the MountType enum and members, since they are no longer
packagefs private.

Build a separate libsupc++ for the kernel with correct flags.

Kernel mode code on x86_64 needs to be built with -mno-red-zone as
interrupts would corrupt the red zone if it were in use. However, the
kernel is linked with libsupc++, which was not compiled with
-mno-red-zone. If an interrupt occurred in libsupc++ code the red zone
would get corrupted. This was causing random panics, particularly under
heavy system load. Therefore, on x86_64 a separate build of libsupc++
with -mno-red-zone is now done for the kernel to use. Note: this commit
will require a rerun of configure and rebuild of cross tools.

Attribute index support for non-special attributes

* Add AttributeIndex class.
* Each attribute does now have an attribute index cookie. The new
attribute index service methods Node::IndexAttribute() and
IndexCookieForAttribute() create+set/retrieve the cookie. The cookie
is actually the attribute index's tree node.
* Add OldNodeAttribute::IndexCookieForAttribute() so the cookie is
available when the node changes.

Move helper function out of QueryParser.h

They live in the new QueryParserUtils.{h,cpp} now.

Add last modified index support

* Add class LastModifiedIndex.
* Create and add an instance of it in Volume::Mount().

Add size index support

* Add class SizeIndex.
* Create and add an instance of it in Volume mount.

NodeListener::NodeChanged(): Provide old value

* Add interface OldNodeAttributes an instance of which is passed to
NodeListener::NodeChanged() to provide the old attribute values
(currently only modification time and file size).
* Also extend PackageLinksListener::PackageLinkNodeChanged() with a
OldNodeAttributes parameter.
* Add OldNodeAttributes implementations for PackageLinkSymlink (inner
class OldAttributes) and UnpackingNode (OldUnpackageNodeAttributes).

Add Query class and query management in Volume

Add NameIndex class and create instance in Volume

Add Index base class and index table in Volume

Add a global node listener mechanism

Support for auto-generated package node attributes

* Add utility class AutoPackageAttributes and AttributeCookie subclass
AutoPackageAttributeCookie.
* UnpackingAttributeCookie::Open() does now try to create a
AutoPackageAttributeCookie, when it doesn't find the attribute in the
PackageNode.
* Adjust UnpackingAttributeDirectoryCookie to also list auto-generated
attributes.

Currently the only supported attribute is "SYS:PACKAGE", which is the
file name of the package containing the node.

Move PackageLinkDirectory::Link to own file

And rename it to PackageLinkSymlink.

WIP towards dependency package links

* Add ResolvableFamily and DependencyFamily classes for grouping equally
named resolvables/dependencies.
* Add hash tables to PackageFSRoot to track resolvables and
dependencies. That is done in {Add,Remove}Package.
* Resolvable does now have a list of Dependency objects, the ones it
satisfies.

Add package link directories to system volume

* Add PackageLinksListener interface. It is used by
PackageLink[s]Directory to notify a listener about changes.
* Volume does now implement PackageLinksListener and recursively adds/
removes package link directories.

This makes the package link directories appear at least. They are still
empty.

WIP: Create empty package links directories

* Rename PackageFamily to PackageLinkDirectory and derive from
Directory.
* PackageLinksDirectory: Add the PackageLinkDirectory objects as
children and remove the no longer needed hash table (we use the
Directory's child hash table).

Move EmptyAttributeDirectoryCookie to own files

Add yet empty package-links directory

* Add PackageLinksDirectory Directory subclass. Currently not doing
anything.
* PackageFSRoot: Create a PackageLinksDirectory.
* Volume: Add/remove the package links directory for the system volume.

Pull derived UnpackingDirectory out of Directory

Move all package specifics from now abstract Directory to new derived
class UnpackingDirectory and adjust the Volume implementation
accordingly. This concludes the Node/Directory refactoring. Neither
class is aware of packages anymore.

Rename LeafNode to UnpackingLeafNode

Remove package related dependencies from Node

* Move package related methods from Node to new interface UnpackingNode.
* LeafNode and Directory derive from UnpackingNode now.
* Adjust Volume implementation accordingly.

Attribute dir access kernel interface abstraction

* Introduce interface AttributeDirectoryCookie and currently only
implementation UnpackingAttributeDirectoryCookie. This is an interface
for reading/rewinding an attribute directory.
* Add abstract virtual Node::OpenAttributeDirectory() method that
returns an AttributeDirectoryCookie and implement it for derived
classes.
* In the kernel interface attribute directory hooks use
AttributeDirectoryCookie now.

Kernel interface abstraction for attribute access

* Introduce interface AttributeCookie and currently only implementation
UnpackingAttributeCookie. This is an interface for reading/stat()ing
an attribute.
* Add abstract virtual Node::OpenAttribute() method that returns an
AttributeCookie and implemented it for derived classes.
* In the kernel interface attribute hooks use AttributeCookie now. The
attribute directory hooks are unchanged.

WIP towards packagefs package links support

* Introduce Version class representing a version.
* Introduce Dependency and Resolvable class and add lists of either to
Package.
* Parse package attributes and add dependencies and resolvables to
Package.
* Add a mount type to Volume and add a respective mount parameter
"mount-type" (values "system", "common", "home", "custom"). Also
implies the shine-through type, if that's not given.
* Introduce class PackageFamily which groups equally named and versioned
packages.
* Add class PackageFSRoot. Each instance represents a possible file
system root (separate roots for different chroot environments). Tracks
Volumes belonging to the same root and their packages.

Split package kit DataReader.cpp

Pull AttributeDataReader and FDDataReader implementations out of
DataReader.cpp into own source files. Thus we can avoid dependencies
(e.g. to fs_attr code) we don't need/want.

Attribute index support for non-special attributes

* Add AttributeIndex class.
* Each attribute does now have an attribute index cookie. The new
attribute index service methods Node::IndexAttribute() and
IndexCookieForAttribute() create+set/retrieve the cookie. The cookie
is actually the attribute index's tree node.
* Add OldNodeAttribute::IndexCookieForAttribute() so the cookie is
available when the node changes.

Move helper function out of QueryParser.h

They live in the new QueryParserUtils.{h,cpp} now.

Add last modified index support

* Add class LastModifiedIndex.
* Create and add an instance of it in Volume::Mount().

Add size index support

* Add class SizeIndex.
* Create and add an instance of it in Volume mount.

NodeListener::NodeChanged(): Provide old value

* Add interface OldNodeAttributes an instance of which is passed to
NodeListener::NodeChanged() to provide the old attribute values
(currently only modification time and file size).
* Also extend PackageLinksListener::PackageLinkNodeChanged() with a
OldNodeAttributes parameter.
* Add OldNodeAttributes implementations for PackageLinkSymlink (inner
class OldAttributes) and UnpackingNode (OldUnpackageNodeAttributes).

Add Query class and query management in Volume

Add NameIndex class and create instance in Volume

Add Index base class and index table in Volume

Add a global node listener mechanism

Support for auto-generated package node attributes

* Add utility class AutoPackageAttributes and AttributeCookie subclass
AutoPackageAttributeCookie.
* UnpackingAttributeCookie::Open() does now try to create a
AutoPackageAttributeCookie, when it doesn't find the attribute in the
PackageNode.
* Adjust UnpackingAttributeDirectoryCookie to also list auto-generated
attributes.

Currently the only supported attribute is "SYS:PACKAGE", which is the
file name of the package containing the node.

Move PackageLinkDirectory::Link to own file

And rename it to PackageLinkSymlink.

WIP towards dependency package links

* Add ResolvableFamily and DependencyFamily classes for grouping equally
named resolvables/dependencies.
* Add hash tables to PackageFSRoot to track resolvables and
dependencies. That is done in {Add,Remove}Package.
* Resolvable does now have a list of Dependency objects, the ones it
satisfies.

Add package link directories to system volume

* Add PackageLinksListener interface. It is used by
PackageLink[s]Directory to notify a listener about changes.
* Volume does now implement PackageLinksListener and recursively adds/
removes package link directories.

This makes the package link directories appear at least. They are still
empty.

WIP: Create empty package links directories

* Rename PackageFamily to PackageLinkDirectory and derive from
Directory.
* PackageLinksDirectory: Add the PackageLinkDirectory objects as
children and remove the no longer needed hash table (we use the
Directory's child hash table).

Move EmptyAttributeDirectoryCookie to own files

Add yet empty package-links directory

* Add PackageLinksDirectory Directory subclass. Currently not doing
anything.
* PackageFSRoot: Create a PackageLinksDirectory.
* Volume: Add/remove the package links directory for the system volume.

Pull derived UnpackingDirectory out of Directory

Move all package specifics from now abstract Directory to new derived
class UnpackingDirectory and adjust the Volume implementation
accordingly. This concludes the Node/Directory refactoring. Neither
class is aware of packages anymore.

Rename LeafNode to UnpackingLeafNode

Remove package related dependencies from Node

* Move package related methods from Node to new interface UnpackingNode.
* LeafNode and Directory derive from UnpackingNode now.
* Adjust Volume implementation accordingly.

Attribute dir access kernel interface abstraction

* Introduce interface AttributeDirectoryCookie and currently only
implementation UnpackingAttributeDirectoryCookie. This is an interface
for reading/rewinding an attribute directory.
* Add abstract virtual Node::OpenAttributeDirectory() method that
returns an AttributeDirectoryCookie and implement it for derived
classes.
* In the kernel interface attribute directory hooks use
AttributeDirectoryCookie now.

Kernel interface abstraction for attribute access

* Introduce interface AttributeCookie and currently only implementation
UnpackingAttributeCookie. This is an interface for reading/stat()ing
an attribute.
* Add abstract virtual Node::OpenAttribute() method that returns an
AttributeCookie and implemented it for derived classes.
* In the kernel interface attribute hooks use AttributeCookie now. The
attribute directory hooks are unchanged.

WIP towards packagefs package links support

* Introduce Version class representing a version.
* Introduce Dependency and Resolvable class and add lists of either to
Package.
* Parse package attributes and add dependencies and resolvables to
Package.
* Add a mount type to Volume and add a respective mount parameter
"mount-type" (values "system", "common", "home", "custom"). Also
implies the shine-through type, if that's not given.
* Introduce class PackageFamily which groups equally named and versioned
packages.
* Add class PackageFSRoot. Each instance represents a possible file
system root (separate roots for different chroot environments). Tracks
Volumes belonging to the same root and their packages.

Split package kit DataReader.cpp

Pull AttributeDataReader and FDDataReader implementations out of
DataReader.cpp into own source files. Thus we can avoid dependencies
(e.g. to fs_attr code) we don't need/want.

Pulled reusable stuff from PackageReaderImpl into new class ReaderImplBase.

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40485 a95241bf-73f2-0310-859d-f6bbb57e9c96

Refactored hpkg implementation to provide some separation between
public and private API (still far from ideal, but a start):
* moved several HPKG-classes into the public namespace BPackageKit::HPKG
* added fImpl-wrappers around PackageReader and PackageWriter to hide
most of the gory details
* adjusted 'package'-binary and packagefs accordingly


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40320 a95241bf-73f2-0310-859d-f6bbb57e9c96

More work on package kit:
* added class BPackageInfo, which contains packaging attributes of
a package (the values relevant for package management)
* implemented parser (mostly) for reading a BPackageInfo from a config
file (.PackageInfo) in order to pass them on to the PackageWriter
when creating a package
* pulled hpkg-related stuff from bin/package into the package kit
* adjusted packagefs-Volume to skip .PackageInfo files when populating
the mountpoint, as those files shouldn't appear as part of an
activated package


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40301 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Extract the attributes from the package files and attach them as
PackageNodeAttribute to the PackageNodes.
* Implemented the attribute FS interface hooks.
* Made Package and PackageNode BReferenceable.
* Fixed the reference management for Node.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34119 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added new interface BufferCache -- basically a memory allocator with the
option to cache a freed buffer -- and implementations
BlockBufferCache{NoLock,Kernel}.
* ZlibPackageDataReader does now dynamically get its read and uncompress
buffers from a provided BufferCache when needed.
* Allocating the buffers once and keeping them over the whole life time was a
bit too memory heavy, since we create a reader for every file for which a
vnode is created. A FS module global factory provides a buffer cache.
* Added a mutex to PackageFile::DataAccessor which guards the access to the
data reader which isn't thread safe.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34115 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Implemented intermediate base class PackageLeafNode for PackageFile and
PackageSymlink with a "fat" interface.
* Replaced File and Symlink by fat class LeafNode.
* Added Package* object management to Directory and LeafNode.
* Implemented packagefs_read_symlink(), so symlinks work now.
* Added some missing locking the kernel interface.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34096 a95241bf-73f2-0310-859d-f6bbb57e9c96

Work in progress:
* The FS parses package files from "/boot/common/packages" and builds an
node tree representation of the contained entries. Merging directories should
work. Lots of other stuff is missing yet, though.
* Implemented the hooks for directory entry lookup and iteration. So it's
possible to see the contents of the FS at least.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34088 a95241bf-73f2-0310-859d-f6bbb57e9c96

Implemented the most basic functionality required to cleanly mount and unmount
(at least in userlandfs).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34069 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added Jamfile for building for the userlandfs.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34065 a95241bf-73f2-0310-859d-f6bbb57e9c96

The very beginning of a package FS: Empty stubs for a minimal read-only file
system.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34064 a95241bf-73f2-0310-859d-f6bbb57e9c96