Package Kit: Scale compression levels to the method's levels.

Zlib has (for now?) 1-9 as its compression levels, but Zstd has
1-22, so we need to scale and round based on *_COMPRESSION_LEVEL_BEST.

Remove ZSTD_ENABLED checks from the Package Kit.

BZstdCompressionAlgorithm already has these #ifdefs and will
return error codes appropriately if built without libzstd,
so we do not need to check again inside the Package Kit.

This should not break the no-zstd build (well, it is broken
somewhat right now anyway, but this will not break it further),
and it simplifies logic somewhat.

package kit: Add pre-uninstall scripts feature.

Just like post-install scripts which run shortly after a package is
installed, pre-uninstall scripts are run just before a package is
removed. Implements enhancement #13427

* Fix script exit code handling vs script launch errors.
* Bump package and repo file version numbers due to new attribute,
unfortunately makes new .hpkg files not backwards compatible.
* Add pre-uninstall functionality, mostly cloning the post-install
except in a few places.
* Discover that _RunQueuedScripts() is never called, a future TODO:?
* Update package documentation for pre-uninstall scripts, and use of
the boot/post-install directory.

Change-Id: I45596255ce74bc102f6e5b606cbf83e4e4347a17
Reviewed-on: https://review.haiku-os.org/c/haiku/+/1504
Reviewed-by: Alexander G. M. Smith <agmsmith@ncf.ca>
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

Package Kit: add Zstd compression.

Change-Id: Idbdb7cf1bde659046a88ea69a76e3b5fc4cd7013
Reviewed-on: https://review.haiku-os.org/323
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

BPackageWriter: Add BPositionIO support

package kit: Actually add support for B_HPKG_COMPRESSION_NONE

Until now we always declared in the HPKG header that the package file is
zlib compressed. For uncompressed files we would just store all
individual chunks uncompressed. Now we handle completely uncompressed
files slightly differently: We don't write the redundant chunk size
table anymore. The size savings are minor, but it makes the uncompressed
format read-streamable which may be handy.

Switch package file accessor classes to use BPositionIO

* PackageFileHeap{Reader,Writer} as well as Package{Reader,Writer} and
their implementation and super classes do now internally use a
BPositionIO instead of a FD to access the package file. This provides
more flexibility needed for features to come.
* BPackageReader has already grown a new Init() version with a
BPositionIO* parameter.

Switch package kit to BZlibCompressionAlgorithm

... and remove the Zlib{Compressor,Decompressor} API.

Move package kit Zlib* classes to support kit

Also move to B* namespace and no longer expose the zlib dependency in
the headers.

Remove BPackageKit::[...]::AbstractDataWriter and friends

* FDDataWriter and ZlibDataWriter weren't used anymore.
* AbstractDataWriter was implemented only by PackageFileHeapWriter,
which was only used by WriterImplBase.
* Add a PackageFileHeapWriter::AddDataThrows() which has semantics
equivalent to the previously inherited WriteDataThrows().

Repository file format: group attributes of a package

* Introduce new package attribute B_HPKG_ATTRIBUTE_ID_PACKAGE (valid
only in a repository file) to group the attributes belonging to a
package.
* BRepositoryContentHandler:
- No longer derive from BPackageContentHandler.
- Add hooks HandlePackage() and HandlePackageDone() that bracket the
attributes for a package. This is more explicit and robust than
handlers having to guess when one package ended and the next began.
* BRepositoryCache: Make use of BPackageInfoContentHandler. No need to
duplicate the code for reading a package info from package info
attributes.

repository writer: fix use of incorrect header size

The header size of the package file header was used, which resulted in
broken repository files.

hpkg format: Generalize user/global settings file attributes

* For all identifiers: Rename global settings file to global writable
file. We want to use the respective attribute also for other writable
files, not only settings files.
* User settings file/global writable file info/attribute: Add
isDirectory property/child attribute. This allows declaring global/
user settings directories associated with the package.

WriterImplBase: Add _AddStringAttributeList()

... and simplify some code by using it.

hpkg format: Add attribute for declaring post install scripts

WriterImplBase: Some simplifications

... using the recently introduced _AddStringAttribute[IfNotEmpty]().

hpkg format: Add attributes for declaring users and groups

hpkg format: Add attributes for declaring settings files

Global and user settings files can be declared. For global ones an
update policy can be specified. If not specified, the settings file is
not included in the package, but created by the program (or user) later.
If an update type is specified, it defines what to do with the settings
file when updating the package to a newer version.

User settings files are never included in the package; they are always
created by the program or the user. If the package contains a template/
default settings file, it can be declared, but for informative purposes
only.

Allow specifying the compression level for package creation

* Introduce BPackageWriterParameters which comprises all parameters
for package creation, currently flags and compression level. Such an
object can be passed to BPackageWriter::Init() and is passed on to
PackageWriterImpl and WriterImplBase.
* PackageFileHeapWriter: Add compressionLevel property and pass the
value on to ZlibCompressor.
* package add/create: Add options -0 ... -9 to specify the compression
level to be used.

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.

Some comment updates

Add "base package" package attribute

Remove resolvable provides type

* Remove BPackageResolvableType and respective property from
BPackageResolvable.
* Remove package attribute B_HPKG_ATTRIBUTE_ID_PACKAGE_PROVIDES_TYPE.

package kit: some fixes for multi-version support

* Use enums/constants/functions instead of preprocessor macros.
* Missing include in PackageInfoAttributeValue.h.
* PackageReaderImpl::Init(): Check version before header size and
return B_MISMATCHED_VALUES instead of B_BAD_DATA, if the version
doesn't match. This allows callers to determine the condition and
try a reader for a different version. A more flexible interface for
that case would be nice, but since we want to support the old package
version only temporarily, the current solution should be good enough.

BPackageVersion: Rename property release to revision

* ... to avoid confusion with the preRelease property. It's also called
"revision" in the HaikuPorts recipes.
* Update libsolv package. Was necessary due to the BPackageVersion
change, but also includes a few more changes.

Use BStringList in package kit

Replace all instances of BObjectList<BString> by BStringList.

Add a package info "install-path" attribute

The attribute is intended for simplifying package building. The
package's install path will be used for the package's .self package
symlink, allowing installation to a temporary directory when building
the package.

Add support for adding/updating package entries

Add flags parameter to BPackageWriter::Init() (and the private
implementation classes) to indicate that an existing package file shall
be updated instead of created. Currently that always happens in-place.

Support package info [source] URL attributes

Write resolvable compatible version to the package

To avoid a clash with the regular version or an extra attribute level we
use the "package:provides.compatible" package attribute instead of
"package:version.major".

Introduce a pre-release version component

* The version string pattern is now:
<major>[.<minor>[.<micro>]][-<pre>][-<release>]
* Introduce B_HPKG_ATTRIBUTE_ID_PACKAGE_VERSION_PRE_RELEASE package
attribute.
* Add "preRelease" field to BPackageVersionData.
* Add "preRelease" property to BPackageVersion and packagefs's Version.
* Adjust package reader and writer code accordingly.

Use BStringList in package kit

Replace all instances of BObjectList<BString> by BStringList.

Add a package info "install-path" attribute

The attribute is intended for simplifying package building. The
package's install path will be used for the package's .self package
symlink, allowing installation to a temporary directory when building
the package.

Add support for adding/updating package entries

Add flags parameter to BPackageWriter::Init() (and the private
implementation classes) to indicate that an existing package file shall
be updated instead of created. Currently that always happens in-place.

Support package info [source] URL attributes

Write resolvable compatible version to the package

To avoid a clash with the regular version or an extra attribute level we
use the "package:provides.compatible" package attribute instead of
"package:version.major".

Introduce a pre-release version component

* The version string pattern is now:
<major>[.<minor>[.<micro>]][-<pre>][-<release>]
* Introduce B_HPKG_ATTRIBUTE_ID_PACKAGE_VERSION_PRE_RELEASE package
attribute.
* Add "preRelease" field to BPackageVersionData.
* Add "preRelease" property to BPackageVersion and packagefs's Version.
* Adjust package reader and writer code accordingly.

Largish adjustments to PackagerReaderImpl and package attribute
handling:
* package attributes are now compatible with the low level attribute
handling of other HPKG attributes (such that 'package dump' now shows
package attributes, too)
* dropped type names from hpkg format, the attributes were identified
by IDs already and this simplifies the code considerably. Type names
are now handled in BLowLevelPackageHandler only.
* instead of rolling their own mechanism, high-level package attributes
handling is now implemented via a corresonding set of
AttributeHandler-subclasses
* adjusted package writer to only write package attributes that are
needed (empty ones are left out)


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40466 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fix problems in repository writer
* actually write the checksum
* compress the repository info archive, too (handle it as separate
section)


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40426 a95241bf-73f2-0310-859d-f6bbb57e9c96

* implemented checking of licenses, such that it is no longer possible
to create a package that refers to a license that is not a system
license or contained in the package
* added package-info flags (currently only approve_license and
system_package)
* adjusted package-reader/writer and pkgman accordingly

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40393 a95241bf-73f2-0310-859d-f6bbb57e9c96

Cleanup:
* PackageWriterImpl no longer accesses members of WriterImplBase
directly, but uses getters/setters instead
* package attribute registration has been moved from PackageWriterImpl
to WriterImplBase, as it will be used in the same manner by the
RepositoryWriter


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40388 a95241bf-73f2-0310-859d-f6bbb57e9c96

Refactoring step towards implementation of RepositoryWriter:
* pulled commonly useful parts out of PackageWriterImpl into
WriterImplBase
* moved CachedStringTable and related methods into a separate class,
StringCache, in order to support having more than one string cache
per package file
* made package attribute section use a string cache, too, as that's
going to be very useful for repositories
* instead of writing package attributes directly, we now collect
corresponding PackageAttributes and write those later
* adjusted package reader accordingly


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40376 a95241bf-73f2-0310-859d-f6bbb57e9c96

BPackageWriter: Add BPositionIO support

package kit: Actually add support for B_HPKG_COMPRESSION_NONE

Until now we always declared in the HPKG header that the package file is
zlib compressed. For uncompressed files we would just store all
individual chunks uncompressed. Now we handle completely uncompressed
files slightly differently: We don't write the redundant chunk size
table anymore. The size savings are minor, but it makes the uncompressed
format read-streamable which may be handy.

Switch package file accessor classes to use BPositionIO

* PackageFileHeap{Reader,Writer} as well as Package{Reader,Writer} and
their implementation and super classes do now internally use a
BPositionIO instead of a FD to access the package file. This provides
more flexibility needed for features to come.
* BPackageReader has already grown a new Init() version with a
BPositionIO* parameter.

Switch package kit to BZlibCompressionAlgorithm

... and remove the Zlib{Compressor,Decompressor} API.

Move package kit Zlib* classes to support kit

Also move to B* namespace and no longer expose the zlib dependency in
the headers.

Remove BPackageKit::[...]::AbstractDataWriter and friends

* FDDataWriter and ZlibDataWriter weren't used anymore.
* AbstractDataWriter was implemented only by PackageFileHeapWriter,
which was only used by WriterImplBase.
* Add a PackageFileHeapWriter::AddDataThrows() which has semantics
equivalent to the previously inherited WriteDataThrows().

Repository file format: group attributes of a package

* Introduce new package attribute B_HPKG_ATTRIBUTE_ID_PACKAGE (valid
only in a repository file) to group the attributes belonging to a
package.
* BRepositoryContentHandler:
- No longer derive from BPackageContentHandler.
- Add hooks HandlePackage() and HandlePackageDone() that bracket the
attributes for a package. This is more explicit and robust than
handlers having to guess when one package ended and the next began.
* BRepositoryCache: Make use of BPackageInfoContentHandler. No need to
duplicate the code for reading a package info from package info
attributes.

repository writer: fix use of incorrect header size

The header size of the package file header was used, which resulted in
broken repository files.

hpkg format: Generalize user/global settings file attributes

* For all identifiers: Rename global settings file to global writable
file. We want to use the respective attribute also for other writable
files, not only settings files.
* User settings file/global writable file info/attribute: Add
isDirectory property/child attribute. This allows declaring global/
user settings directories associated with the package.

WriterImplBase: Add _AddStringAttributeList()

... and simplify some code by using it.

hpkg format: Add attribute for declaring post install scripts

WriterImplBase: Some simplifications

... using the recently introduced _AddStringAttribute[IfNotEmpty]().

hpkg format: Add attributes for declaring users and groups

hpkg format: Add attributes for declaring settings files

Global and user settings files can be declared. For global ones an
update policy can be specified. If not specified, the settings file is
not included in the package, but created by the program (or user) later.
If an update type is specified, it defines what to do with the settings
file when updating the package to a newer version.

User settings files are never included in the package; they are always
created by the program or the user. If the package contains a template/
default settings file, it can be declared, but for informative purposes
only.

Allow specifying the compression level for package creation

* Introduce BPackageWriterParameters which comprises all parameters
for package creation, currently flags and compression level. Such an
object can be passed to BPackageWriter::Init() and is passed on to
PackageWriterImpl and WriterImplBase.
* PackageFileHeapWriter: Add compressionLevel property and pass the
value on to ZlibCompressor.
* package add/create: Add options -0 ... -9 to specify the compression
level to be used.

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.

Some comment updates

Add "base package" package attribute

Remove resolvable provides type

* Remove BPackageResolvableType and respective property from
BPackageResolvable.
* Remove package attribute B_HPKG_ATTRIBUTE_ID_PACKAGE_PROVIDES_TYPE.

package kit: some fixes for multi-version support

* Use enums/constants/functions instead of preprocessor macros.
* Missing include in PackageInfoAttributeValue.h.
* PackageReaderImpl::Init(): Check version before header size and
return B_MISMATCHED_VALUES instead of B_BAD_DATA, if the version
doesn't match. This allows callers to determine the condition and
try a reader for a different version. A more flexible interface for
that case would be nice, but since we want to support the old package
version only temporarily, the current solution should be good enough.

BPackageVersion: Rename property release to revision

* ... to avoid confusion with the preRelease property. It's also called
"revision" in the HaikuPorts recipes.
* Update libsolv package. Was necessary due to the BPackageVersion
change, but also includes a few more changes.

Use BStringList in package kit

Replace all instances of BObjectList<BString> by BStringList.

Add a package info "install-path" attribute

The attribute is intended for simplifying package building. The
package's install path will be used for the package's .self package
symlink, allowing installation to a temporary directory when building
the package.

Add support for adding/updating package entries

Add flags parameter to BPackageWriter::Init() (and the private
implementation classes) to indicate that an existing package file shall
be updated instead of created. Currently that always happens in-place.

Support package info [source] URL attributes

Write resolvable compatible version to the package

To avoid a clash with the regular version or an extra attribute level we
use the "package:provides.compatible" package attribute instead of
"package:version.major".

Introduce a pre-release version component

* The version string pattern is now:
<major>[.<minor>[.<micro>]][-<pre>][-<release>]
* Introduce B_HPKG_ATTRIBUTE_ID_PACKAGE_VERSION_PRE_RELEASE package
attribute.
* Add "preRelease" field to BPackageVersionData.
* Add "preRelease" property to BPackageVersion and packagefs's Version.
* Adjust package reader and writer code accordingly.

Use BStringList in package kit

Replace all instances of BObjectList<BString> by BStringList.

Add a package info "install-path" attribute

The attribute is intended for simplifying package building. The
package's install path will be used for the package's .self package
symlink, allowing installation to a temporary directory when building
the package.

Add support for adding/updating package entries

Add flags parameter to BPackageWriter::Init() (and the private
implementation classes) to indicate that an existing package file shall
be updated instead of created. Currently that always happens in-place.

Support package info [source] URL attributes

Write resolvable compatible version to the package

To avoid a clash with the regular version or an extra attribute level we
use the "package:provides.compatible" package attribute instead of
"package:version.major".

Introduce a pre-release version component

* The version string pattern is now:
<major>[.<minor>[.<micro>]][-<pre>][-<release>]
* Introduce B_HPKG_ATTRIBUTE_ID_PACKAGE_VERSION_PRE_RELEASE package
attribute.
* Add "preRelease" field to BPackageVersionData.
* Add "preRelease" property to BPackageVersion and packagefs's Version.
* Adjust package reader and writer code accordingly.

Largish adjustments to PackagerReaderImpl and package attribute
handling:
* package attributes are now compatible with the low level attribute
handling of other HPKG attributes (such that 'package dump' now shows
package attributes, too)
* dropped type names from hpkg format, the attributes were identified
by IDs already and this simplifies the code considerably. Type names
are now handled in BLowLevelPackageHandler only.
* instead of rolling their own mechanism, high-level package attributes
handling is now implemented via a corresonding set of
AttributeHandler-subclasses
* adjusted package writer to only write package attributes that are
needed (empty ones are left out)


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40466 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fix problems in repository writer
* actually write the checksum
* compress the repository info archive, too (handle it as separate
section)


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40426 a95241bf-73f2-0310-859d-f6bbb57e9c96

* implemented checking of licenses, such that it is no longer possible
to create a package that refers to a license that is not a system
license or contained in the package
* added package-info flags (currently only approve_license and
system_package)
* adjusted package-reader/writer and pkgman accordingly

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40393 a95241bf-73f2-0310-859d-f6bbb57e9c96

Cleanup:
* PackageWriterImpl no longer accesses members of WriterImplBase
directly, but uses getters/setters instead
* package attribute registration has been moved from PackageWriterImpl
to WriterImplBase, as it will be used in the same manner by the
RepositoryWriter


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40388 a95241bf-73f2-0310-859d-f6bbb57e9c96

Refactoring step towards implementation of RepositoryWriter:
* pulled commonly useful parts out of PackageWriterImpl into
WriterImplBase
* moved CachedStringTable and related methods into a separate class,
StringCache, in order to support having more than one string cache
per package file
* made package attribute section use a string cache, too, as that's
going to be very useful for repositories
* instead of writing package attributes directly, we now collect
corresponding PackageAttributes and write those later
* adjusted package reader accordingly


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40376 a95241bf-73f2-0310-859d-f6bbb57e9c96