packagefs: Use WeakReference in PackageNode.

This allows us to stop storing the package flags, which saves 4
bytes per package node (a value that really adds up when there
are thousands upon thousands of PackageNodes), at the cost of an extra
sizeof(int32) allocation for the WeakPointer object per-package (of
which there are are much fewer, of course.)

This also is safer overall, as now consumers of GetPackage() or VFSInit()
will now hit a NULL dereference if they have failed to check if the
package still exists, instead of a use-after-free.

Change-Id: Iea97ffcd491c6e2da7093730a7fa951b84dcefdf
Reviewed-on: https://review.haiku-os.org/c/haiku/+/4478
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>
Reviewed-by: Alex von Gluck IV <kallisti5@unixzen.com>

fix build after MethodDeleter interface change

Change-Id: Iadb9e37772fd6588636085944c1455249ac0f926
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3457
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

packagefs: Use object caches for the most used node classes.

Slab statistics from KDL show that on a stock nightly image
(i.e. no additional packages besides the standards installed)
there are 43848 *objects* (not bytes) in the PackageNodeAttribute
cache, and 25090 in the PackageFile cache, so this seems more than
worth it.

The last commit seems to reduce memory usage at boot by about 1%,
this commit seems to not affect it at all; but it is a significant
performance optimization and on systems with more packages installed
the effect may be very noticeable.

Change-Id: I676a642ed6003f82b14396e1f02684575d899362

file_systems: Remove now-redundant bounds checks before file_cache_read().

Change-Id: Iafb7d188c7e7cb4406d924eb3354a7ede40c6641
Reviewed-on: https://review.haiku-os.org/c/1421
Reviewed-by: waddlesplash <waddlesplash@gmail.com>
Reviewed-by: Michael Lotz <mmlr@mlotz.ch>

packagefs: Remove another reference to GlobalFactory.h.

Missed in previous push.

Remove BPackageKit::BDataOutput

Use BDataIO instead.

Revert "Move ZlibDecompressor to libshared"

This reverts commit 9af2105d36856be60528c361edf14a263f242338.

Conflicts:
src/kits/package/Jamfile

Revert "Fix build."

This reverts commit 00dfae0bce7e63243180e89847df647cfa57be85.

Revert "Fix more issues detected by gcc4 compiler:"

This reverts commit 19f3bae0716c4eb7dd2a5716778560580be68071.

Revert "build fix follow up on status_t/ssize_t changes."

This reverts commit 7202cfb455cae89faa6e2fa3d9a9d28c3068fa6b.

build fix follow up on status_t/ssize_t changes.

* added a TODO in packagefs.

Fix more issues detected by gcc4 compiler:

More ssize_t/status_t mixup.

Fix build.

Sorry, renaming headers isn't detected by Jam and not everything was
rebuilt.

Move ZlibDecompressor to libshared

* This will be used to implement compressed http streams
* Remove the custom BDataOutput class, and use BDataIO instead, for
easier integration with existing code.

packagfs: fix header order

Fix reading attributes of directories

When reading an attribute of a directory there was no guarantee that the
underlying package would be open. When it wasn't reading an attribute
would fail, unless the attribute data were already cached. The reasons
for this are:
* UnpackingDirectory didn't forward the {Init,Uninit}VFS() calls to the
underlying PackageDirectory.
* Only PackageFile was actually opening the package in InitVFS().

Now we forward the {Init,Uninit}VFS() calls in all cases -- even in
{Add,Remove}PackageNode(), when the active package node changes -- and
opening/closing the package is now done in
PackageNode::{Init,Uninit}VFS().

packagefs: PackageFile::DataAccessor: Avoid unnecessary locking

... in case of version 2 package files.

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.

packagefs: Add support for hpkg format V1

* Add union-like class PackageData which wraps the V1 and V2
BPackageData classes.
* GlobalFactory: Create a data reader depending on the data format
version.
* Package: Add a loader for V1 format and try that, if the other one
fails.

package kit: internalize BPackageDataReader

It is no longer public (or even private) API. BPackageDataReaderFactory
returns a BAbstractBufferedDataReader instead. The advantage is that
the latter doesn't have hpkg format specific dependencies.

packagefs: Organize sources into subdirectories

Remove BPackageKit::BDataOutput

Use BDataIO instead.

Revert "Move ZlibDecompressor to libshared"

This reverts commit 9af2105d36856be60528c361edf14a263f242338.

Conflicts:
src/kits/package/Jamfile

Revert "Fix build."

This reverts commit 00dfae0bce7e63243180e89847df647cfa57be85.

Revert "Fix more issues detected by gcc4 compiler:"

This reverts commit 19f3bae0716c4eb7dd2a5716778560580be68071.

Revert "build fix follow up on status_t/ssize_t changes."

This reverts commit 7202cfb455cae89faa6e2fa3d9a9d28c3068fa6b.

build fix follow up on status_t/ssize_t changes.

* added a TODO in packagefs.

Fix more issues detected by gcc4 compiler:

More ssize_t/status_t mixup.

Fix build.

Sorry, renaming headers isn't detected by Jam and not everything was
rebuilt.

Move ZlibDecompressor to libshared

* This will be used to implement compressed http streams
* Remove the custom BDataOutput class, and use BDataIO instead, for
easier integration with existing code.

packagfs: fix header order

Fix reading attributes of directories

When reading an attribute of a directory there was no guarantee that the
underlying package would be open. When it wasn't reading an attribute
would fail, unless the attribute data were already cached. The reasons
for this are:
* UnpackingDirectory didn't forward the {Init,Uninit}VFS() calls to the
underlying PackageDirectory.
* Only PackageFile was actually opening the package in InitVFS().

Now we forward the {Init,Uninit}VFS() calls in all cases -- even in
{Add,Remove}PackageNode(), when the active package node changes -- and
opening/closing the package is now done in
PackageNode::{Init,Uninit}VFS().

packagefs: PackageFile::DataAccessor: Avoid unnecessary locking

... in case of version 2 package files.

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.

packagefs: Add support for hpkg format V1

* Add union-like class PackageData which wraps the V1 and V2
BPackageData classes.
* GlobalFactory: Create a data reader depending on the data format
version.
* Package: Add a loader for V1 format and try that, if the other one
fails.

package kit: internalize BPackageDataReader

It is no longer public (or even private) API. BPackageDataReaderFactory
returns a BAbstractBufferedDataReader instead. The advantage is that
the latter doesn't have hpkg format specific dependencies.

packagefs: Organize sources into subdirectories