packagefs: Use WeakReference in PackageNode.

This allows us to stop storing the package flags, which saves 4
bytes per package node (a value that really adds up when there
are thousands upon thousands of PackageNodes), at the cost of an extra
sizeof(int32) allocation for the WeakPointer object per-package (of
which there are are much fewer, of course.)

This also is safer overall, as now consumers of GetPackage() or VFSInit()
will now hit a NULL dereference if they have failed to check if the
package still exists, instead of a use-after-free.

Change-Id: Iea97ffcd491c6e2da7093730a7fa951b84dcefdf
Reviewed-on: https://review.haiku-os.org/c/haiku/+/4478
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>
Reviewed-by: Alex von Gluck IV <kallisti5@unixzen.com>

packagefs: system_packages take precedence over non-system packages.

Affects *all* nodes, i.e., both directories and files. It is still possible
to override these, but it must be done in another system_package, not just by
mounting a package with newer datestamps on the conflicting files.

This is the pure version of axeld's proposal 2(d) from #10071. However,
as the haiku_secondary packages contain application mime_db entires and
are also 'system_package's, it does not fully solve that ticket.

More useless inclusions of khash.h

Adjust packagefs ioctl interface to support old states

* PackageFSVolumeInfo: Add the directories for all relevant states.
* PackageFSPackageInfo: Include the package file's parent directory node
ref.

Package daemon and package kit still don't support old states yet.

packagefs: Package: Use PackagesDirectory directly

... instead of Volume. Necessary to support using packages from
different directories.

packagefs: Support blacklisting entries in packages

In each installation location, it is now possible to create a settings
file "packages" that allows to blacklist entries contained in packages.
The format is:

Package <package name> {
EntryBlacklist {
<entry path>
...
}
}
...

<package name> is the base name (no version) of the respective package
(e.g. "haiku"), <entry path> is an installation location relative path
(e.g. "add-ons/Translators/FooTranslator").

Blacklisted entries will be ignored by packagefs, i.e. they won't appear
in the file system. This addresses the issue that it may be necessary to
remove a problematic file (e.g. driver, add-on, or library), which would
otherwise require editing the containing package file.

The settings file is not not "live". Changes take effect only after
reboot (respectively when remounting the concerned packagefs volume).

packagefs: Package: Add versionedName property

packagefs: Add caching for the package file heap reader

* ReaderImplBase:
- Add virtual CreateCachedHeapReader() which can create a cached
reader based on the given heap reader.
- Rename HeapReader() to RawHeapReader() and add HeapReader() for the
cached heap reader.
- Add DetachHeapReader() to allow a clients to remove the heap
reader(s) after deleting the ReaderImplBase object.
* packagefs:
- Add CachedDataReader class, which wraps a given
BAbstractBufferedDataReader and provides caching for it using a
VMCache. The implementation is based on the IOCache implementation.
- Use CachedDataReader to wrap the heap reader. For file data that
means they are cached twice -- in the heap reader cache and in the
file cache -- but due to the heap reader using a VMCache as well,
the pages will be recycled automatically anyway. For attribute data
the cache should be very helpful, since they weren't cached at all
before.

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.

packagefs: Add support for hpkg format V1

* Add union-like class PackageData which wraps the V1 and V2
BPackageData classes.
* GlobalFactory: Create a data reader depending on the data format
version.
* Package: Add a loader for V1 format and try that, if the other one
fails.

packagefs: Share string instances globally

Introduce a class String which refers to shared private data that is
registered in a global hash table (in class StringPool) and use the
class consequently. This eliminates duplicate allocations for the same
string and also speeds up tests for equality. There's quite a bit
overhead for the management structures (apparently even more than for
the string data itself), but due to almost all strings being used
multiple times this still almost halves the memory usage for string
data.

packagefs: Organize sources into subdirectories

More useless inclusions of khash.h

Adjust packagefs ioctl interface to support old states

* PackageFSVolumeInfo: Add the directories for all relevant states.
* PackageFSPackageInfo: Include the package file's parent directory node
ref.

Package daemon and package kit still don't support old states yet.

packagefs: Package: Use PackagesDirectory directly

... instead of Volume. Necessary to support using packages from
different directories.

packagefs: Support blacklisting entries in packages

In each installation location, it is now possible to create a settings
file "packages" that allows to blacklist entries contained in packages.
The format is:

Package <package name> {
EntryBlacklist {
<entry path>
...
}
}
...

<package name> is the base name (no version) of the respective package
(e.g. "haiku"), <entry path> is an installation location relative path
(e.g. "add-ons/Translators/FooTranslator").

Blacklisted entries will be ignored by packagefs, i.e. they won't appear
in the file system. This addresses the issue that it may be necessary to
remove a problematic file (e.g. driver, add-on, or library), which would
otherwise require editing the containing package file.

The settings file is not not "live". Changes take effect only after
reboot (respectively when remounting the concerned packagefs volume).

packagefs: Package: Add versionedName property

packagefs: Add caching for the package file heap reader

* ReaderImplBase:
- Add virtual CreateCachedHeapReader() which can create a cached
reader based on the given heap reader.
- Rename HeapReader() to RawHeapReader() and add HeapReader() for the
cached heap reader.
- Add DetachHeapReader() to allow a clients to remove the heap
reader(s) after deleting the ReaderImplBase object.
* packagefs:
- Add CachedDataReader class, which wraps a given
BAbstractBufferedDataReader and provides caching for it using a
VMCache. The implementation is based on the IOCache implementation.
- Use CachedDataReader to wrap the heap reader. For file data that
means they are cached twice -- in the heap reader cache and in the
file cache -- but due to the heap reader using a VMCache as well,
the pages will be recycled automatically anyway. For attribute data
the cache should be very helpful, since they weren't cached at all
before.

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.

packagefs: Add support for hpkg format V1

* Add union-like class PackageData which wraps the V1 and V2
BPackageData classes.
* GlobalFactory: Create a data reader depending on the data format
version.
* Package: Add a loader for V1 format and try that, if the other one
fails.

packagefs: Share string instances globally

Introduce a class String which refers to shared private data that is
registered in a global hash table (in class StringPool) and use the
class consequently. This eliminates duplicate allocations for the same
string and also speeds up tests for equality. There's quite a bit
overhead for the management structures (apparently even more than for
the string data itself), but due to almost all strings being used
multiple times this still almost halves the memory usage for string
data.

packagefs: Organize sources into subdirectories