libpackage_build: Add FetchUtils.

Change-Id: I2394d35c520c6cdde6e7cb09039d0d8f5689f19f
Reviewed-on: https://review.haiku-os.org/c/haiku/+/4731
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

Remove Curl, Switch to NetApi in Package Kit

Change-Id: I92213ab60dc987175c323d1d9ed11ac8b3517f2f
Reviewed-on: https://review.haiku-os.org/475
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

Move SHA256 class to libroot instead of linking libshared into libroot.

Discussed with PulkoMandy and on the haiku-commits mailing list.

Thanks to @jessicah for compile-testing and fixing 2 small issues I missed.

BPackageInfo::Parser: Validate URL strings.

Fixes #12710.

Signed-off-by: Augustin Cavalier <waddlesplash@gmail.com>
I fixed the modifications to the Jamfiles in src/bin, they were all wrong
in the patch.

Moved BJob, and JobQueue into the support kit.

* Put it in the BSupportKit namespace, following the style introduced
with the package kit for now.
* The BSupportKit::BJob class no longer knows about the package kit's
Context class. However, the BPackageKit::BJob class does.
* Due to the namespace juggling, a lot of files had to be touched.
* The JobQueue class remains private.
* Due to the way Haiku is built on itself, you cannot build this change
under Haiku with an older release.

Move package kit Zlib* classes to support kit

Also move to B* namespace and no longer expose the zlib dependency in
the headers.

Remove BPackageKit::[...]::AbstractDataWriter and friends

* FDDataWriter and ZlibDataWriter weren't used anymore.
* AbstractDataWriter was implemented only by PackageFileHeapWriter,
which was only used by WriterImplBase.
* Add a PackageFileHeapWriter::AddDataThrows() which has semantics
equivalent to the previously inherited WriteDataThrows().

Remove BPackageKit::BDataOutput

Use BDataIO instead.

Revert "Move ZlibDecompressor to libshared"

This reverts commit 9af2105d36856be60528c361edf14a263f242338.

Conflicts:
src/kits/package/Jamfile

package daemon: Rework error and issue propagation to client

* BDaemonClient: Move inner class BCommitTransactionResult to top level
and make it public.
* BCommitTransactionResult:
- Add a whole bunch of specific error code enum values. Such an error
code is now the primary error, as opposed to before where we would
mix status_t and enum value errors. There's a systemError property
of type status_t which may provide additional information, though
(depending on the primary error type).
- Remove the errorMessage property. Due to mapping all errors to the
specific error codes this is no longer necessary. Mixing such a
message with another error description is also not very helpful when
it comes to localization (still not supported, though).
- Add several properties (paths, strings, error codes) that serve as
arguments to the primary error and are used by FullErrorMessage().
- Add issues property, a list of instances of new class
BTransactionIssue. Those describe non-critical issues (e.g. failed
update of a settings file) that occurred in the process of
committing the transaction. Those issues should be presented to the
user by the package management program.
* Exception: Adjust to transport the BCommitTransactionResult
properties.
* CommitTransactionHandler, FsTransactions, Root, Volume: Adjust to
BCommitTransactionResult/Exception changes.
* CommitTransactionHandler: Now requires a BCommitTransactionResult to
which it adds the issues it encounters. The reply BMessage is no
longer needed, though.
* Volume: Refactor common code from the three methods that use
CommitTransactionHandler into new method _CommitTransaction.

Move ZlibDecompressor to libshared

* This will be used to implement compressed http streams
* Remove the custom BDataOutput class, and use BDataIO instead, for
easier integration with existing code.

Actually fix the build

Looks like I was too quick on previous commit.

It seems -pthread is actually not that much needed on Linux though...
although the manpage says "compile and link with -pthread". Go figure.

Now dlopen() uses RTLD_LAZY | RTLD_LOCAL for Linux, which seems
to work here.

Clean up FreeBSD build support as suggested by Ingo.

libpackage_build: Fix build on FreeBSD.

FreeBSD's gcc doesn't include /usr/local/{include,lib} in its default
search paths, but libcurl gets installed there.

libpackage_build: Fix build on gcc2 haiku.

- gcc2 lacks the -iquote option, and side effects of some of the other flags
we use with it during the build cause some of libcurl's includes to fail.
This temporarily works around the problem until gcc2 can be suitably fixed.

libpackage: Use libcurl directly for downloads.

- Initial step towards allowing listening for download progress.

libpackage_build: Fix build for paths with spaces

libpackage_build: Make sure built-in data dir path is absolute

Fixes #10022.

Refactor reusable pkgman code into libpackage

* Move RepositoryBuilder class to libpackage and add B* prefix to name.
* Pull BPackageManager class out of PackageManager and move to
libpackage. The base class is customizable via three handler objects
responsible for transaction handling, request execution, respectively
user interaction.
* Reorganize _ApplyPackageChanges(): Now we first prepare the
transactions for all affected installation locations (downloading
files etc.) and then commit them.

BRepositoryConfig::SetTo(): fix use after deallocation

The strings retrieved from the driver settings were used after
unload_driver_settings(). Use BDriverSettings which also simplifies
things quite a bit.

Switch build system from optional package to repositories

* Build libsolv and the dependency solver part of the package kit for
the build platform.
* Add build tool get_package_dependencies. Given a list of package files
and a list of repository files it determines the additional packages
that need to be retrieved from the repositories and prints their URLs.
* Add rules to work with external repositories in the build system
(build/jam/RepositoryRules):
- PackageRepository declares an external repository with all its
packages. The URL of the repository file isn't specified. It is
computed from a given base URL and the SHA256 hash of the list of
package files.
- GeneratedRepositoryPackageList generates a file containing the file
names of all packages in a repository.
- IsPackageAvailable returns whether a package is available in any
repository.
- PackageURL returns the URL for a package.
* Declare the HaikuPorts repository for x86_gcc2
(build/jam/repositories/HaikuPorts/x86_gcc2).
* Add rule AddHaikuImagePackages to add a package to the image and rule
IsHaikuImagePackageAdded to determine whether a package has been
added.
* OptionalPackages: Remove all entries that just downloaded and
installed an external package. AddHaikuImagePackages can be used
instead and is used in the remaining entries. Also move the remaining
optional package dependency declarations from
OptionalPackageDependencies here.
* ExtractBuildFeatureArchives: Instead of the URL parameter a package
name must be specified now. This allows to simplify BuildFeatures
significantly, since there's no dealing with URLs anymore. "if" out
the entries that aren't supported yet.
* build_haiku_image: For the packages installed in system and common
resolve their dependencies and download and install them as well.

Repository file format: group attributes of a package

* Introduce new package attribute B_HPKG_ATTRIBUTE_ID_PACKAGE (valid
only in a repository file) to group the attributes belonging to a
package.
* BRepositoryContentHandler:
- No longer derive from BPackageContentHandler.
- Add hooks HandlePackage() and HandlePackageDone() that bracket the
attributes for a package. This is more explicit and robust than
handlers having to guess when one package ended and the next began.
* BRepositoryCache: Make use of BPackageInfoContentHandler. No need to
duplicate the code for reading a package info from package info
attributes.

hpkg format: Generalize user/global settings file attributes

* For all identifiers: Rename global settings file to global writable
file. We want to use the respective attribute also for other writable
files, not only settings files.
* User settings file/global writable file info/attribute: Add
isDirectory property/child attribute. This allows declaring global/
user settings directories associated with the package.

hpkg format: Add attributes for declaring users and groups

Move package info parser to its own file

hpkg format: Add attributes for declaring settings files

Global and user settings files can be declared. For global ones an
update policy can be specified. If not specified, the settings file is
not included in the package, but created by the program (or user) later.
If an update type is specified, it defines what to do with the settings
file when updating the package to a newer version.

User settings files are never included in the package; they are always
created by the program or the user. If the package contains a template/
default settings file, it can be declared, but for informative purposes
only.

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.

Rename BBlockBufferCache and friends to *Pool*

Also move BBlockBufferPoolNoLock into BHPKG namespace with the other
classes. Not sure why it wasn't there before.

Clone read support functionality for HPKG format version 1

It uses sub-namespace BPackage::BHPKG::V1. Unlike the one for the
current format version, the V1 version of BPackageInfoContentHandler
lives in BHPKG(::V1) sub-namespace and is private.

package kit: some fixes for multi-version support

* Use enums/constants/functions instead of preprocessor macros.
* Missing include in PackageInfoAttributeValue.h.
* PackageReaderImpl::Init(): Check version before header size and
return B_MISMATCHED_VALUES instead of B_BAD_DATA, if the version
doesn't match. This allows callers to determine the condition and
try a reader for a different version. A more flexible interface for
that case would be nice, but since we want to support the old package
version only temporarily, the current solution should be good enough.

Use libbe_build also on Haiku

This means the build tools will no longer be built against the host
platform's libbe, which avoids compatibility problems -- e.g. an
older Haiku host libbe may not have certain features the build tools
require -- and also makes the build behave more similiar on Haiku and
other platforms. The host libroot dependency still remains and is not
easy to get rid of.

Also remove some bits of BeOS/Dano/Zeta build support.

Add DownloadFileRequest

Downloads a file and optionally checks its checksum.

package daemon: Add support for activation change request

* daemon: Handle new request B_MESSAGE_COMMIT_TRANSACTION. It activates
and deactivates given sets of packages. The new packages must be
placed in a directory in the administrative directory. The daemon
moves them to the packages directory and the deactivated packages to
a subdirectory it creates. It also save the old activation state
there.
* Add private BActivationTransaction, describing an activation change
transaction.
* BDaemonClient: Add CommitTransaction(), which sends a given
BActivationTransaction as a B_MESSAGE_COMMIT_TRANSACTION request to
the daemon.

Completely untested yet.

Add BInstallationLocationInfo

Fix the libpackage_build build

* The recently introduced {No,Standard}ErrorOutput.cpp were still
missing.
* Add hack to allow building more correctly on Haiku. We were using
the installed package kit headers, not the ones from the working
directory.

Add class BPackageInfoContentHandler

A BPackageContentHandler subclass that initializes a BPackageInfo from
the read package attributes. Pulled out of RepositoryWriterImpl's
PackageContentHandler.

Pull class BPackageInfoSet out of BRepositoryCache

Split package kit DataReader.cpp

Pull AttributeDataReader and FDDataReader implementations out of
DataReader.cpp into own source files. Thus we can avoid dependencies
(e.g. to fs_attr code) we don't need/want.

<build>package: attribute iteration, license dir

PackageWriterImpl:
* Iterate through attributes using fs_read_attr_dir() instead of
readdir(). Makes it work correctly on the build platform.
* On the build platform look up the system licenses in their source
directory rather than based on find_directory().

Build libpackage for the build platform

Add class BPackageInfoContentHandler

A BPackageContentHandler subclass that initializes a BPackageInfo from
the read package attributes. Pulled out of RepositoryWriterImpl's
PackageContentHandler.

Pull class BPackageInfoSet out of BRepositoryCache

Split package kit DataReader.cpp

Pull AttributeDataReader and FDDataReader implementations out of
DataReader.cpp into own source files. Thus we can avoid dependencies
(e.g. to fs_attr code) we don't need/want.

<build>package: attribute iteration, license dir

PackageWriterImpl:
* Iterate through attributes using fs_read_attr_dir() instead of
readdir(). Makes it work correctly on the build platform.
* On the build platform look up the system licenses in their source
directory rather than based on find_directory().

Build libpackage for the build platform

Moved BJob, and JobQueue into the support kit.

* Put it in the BSupportKit namespace, following the style introduced
with the package kit for now.
* The BSupportKit::BJob class no longer knows about the package kit's
Context class. However, the BPackageKit::BJob class does.
* Due to the namespace juggling, a lot of files had to be touched.
* The JobQueue class remains private.
* Due to the way Haiku is built on itself, you cannot build this change
under Haiku with an older release.

Move package kit Zlib* classes to support kit

Also move to B* namespace and no longer expose the zlib dependency in
the headers.

Remove BPackageKit::[...]::AbstractDataWriter and friends

* FDDataWriter and ZlibDataWriter weren't used anymore.
* AbstractDataWriter was implemented only by PackageFileHeapWriter,
which was only used by WriterImplBase.
* Add a PackageFileHeapWriter::AddDataThrows() which has semantics
equivalent to the previously inherited WriteDataThrows().

Remove BPackageKit::BDataOutput

Use BDataIO instead.

Revert "Move ZlibDecompressor to libshared"

This reverts commit 9af2105d36856be60528c361edf14a263f242338.

Conflicts:
src/kits/package/Jamfile

package daemon: Rework error and issue propagation to client

* BDaemonClient: Move inner class BCommitTransactionResult to top level
and make it public.
* BCommitTransactionResult:
- Add a whole bunch of specific error code enum values. Such an error
code is now the primary error, as opposed to before where we would
mix status_t and enum value errors. There's a systemError property
of type status_t which may provide additional information, though
(depending on the primary error type).
- Remove the errorMessage property. Due to mapping all errors to the
specific error codes this is no longer necessary. Mixing such a
message with another error description is also not very helpful when
it comes to localization (still not supported, though).
- Add several properties (paths, strings, error codes) that serve as
arguments to the primary error and are used by FullErrorMessage().
- Add issues property, a list of instances of new class
BTransactionIssue. Those describe non-critical issues (e.g. failed
update of a settings file) that occurred in the process of
committing the transaction. Those issues should be presented to the
user by the package management program.
* Exception: Adjust to transport the BCommitTransactionResult
properties.
* CommitTransactionHandler, FsTransactions, Root, Volume: Adjust to
BCommitTransactionResult/Exception changes.
* CommitTransactionHandler: Now requires a BCommitTransactionResult to
which it adds the issues it encounters. The reply BMessage is no
longer needed, though.
* Volume: Refactor common code from the three methods that use
CommitTransactionHandler into new method _CommitTransaction.

Move ZlibDecompressor to libshared

* This will be used to implement compressed http streams
* Remove the custom BDataOutput class, and use BDataIO instead, for
easier integration with existing code.

Actually fix the build

Looks like I was too quick on previous commit.

It seems -pthread is actually not that much needed on Linux though...
although the manpage says "compile and link with -pthread". Go figure.

Now dlopen() uses RTLD_LAZY | RTLD_LOCAL for Linux, which seems
to work here.

Clean up FreeBSD build support as suggested by Ingo.

libpackage_build: Fix build on FreeBSD.

FreeBSD's gcc doesn't include /usr/local/{include,lib} in its default
search paths, but libcurl gets installed there.

libpackage_build: Fix build on gcc2 haiku.

- gcc2 lacks the -iquote option, and side effects of some of the other flags
we use with it during the build cause some of libcurl's includes to fail.
This temporarily works around the problem until gcc2 can be suitably fixed.

libpackage: Use libcurl directly for downloads.

- Initial step towards allowing listening for download progress.

libpackage_build: Fix build for paths with spaces

libpackage_build: Make sure built-in data dir path is absolute

Fixes #10022.

Refactor reusable pkgman code into libpackage

* Move RepositoryBuilder class to libpackage and add B* prefix to name.
* Pull BPackageManager class out of PackageManager and move to
libpackage. The base class is customizable via three handler objects
responsible for transaction handling, request execution, respectively
user interaction.
* Reorganize _ApplyPackageChanges(): Now we first prepare the
transactions for all affected installation locations (downloading
files etc.) and then commit them.

BRepositoryConfig::SetTo(): fix use after deallocation

The strings retrieved from the driver settings were used after
unload_driver_settings(). Use BDriverSettings which also simplifies
things quite a bit.

Switch build system from optional package to repositories

* Build libsolv and the dependency solver part of the package kit for
the build platform.
* Add build tool get_package_dependencies. Given a list of package files
and a list of repository files it determines the additional packages
that need to be retrieved from the repositories and prints their URLs.
* Add rules to work with external repositories in the build system
(build/jam/RepositoryRules):
- PackageRepository declares an external repository with all its
packages. The URL of the repository file isn't specified. It is
computed from a given base URL and the SHA256 hash of the list of
package files.
- GeneratedRepositoryPackageList generates a file containing the file
names of all packages in a repository.
- IsPackageAvailable returns whether a package is available in any
repository.
- PackageURL returns the URL for a package.
* Declare the HaikuPorts repository for x86_gcc2
(build/jam/repositories/HaikuPorts/x86_gcc2).
* Add rule AddHaikuImagePackages to add a package to the image and rule
IsHaikuImagePackageAdded to determine whether a package has been
added.
* OptionalPackages: Remove all entries that just downloaded and
installed an external package. AddHaikuImagePackages can be used
instead and is used in the remaining entries. Also move the remaining
optional package dependency declarations from
OptionalPackageDependencies here.
* ExtractBuildFeatureArchives: Instead of the URL parameter a package
name must be specified now. This allows to simplify BuildFeatures
significantly, since there's no dealing with URLs anymore. "if" out
the entries that aren't supported yet.
* build_haiku_image: For the packages installed in system and common
resolve their dependencies and download and install them as well.

Repository file format: group attributes of a package

* Introduce new package attribute B_HPKG_ATTRIBUTE_ID_PACKAGE (valid
only in a repository file) to group the attributes belonging to a
package.
* BRepositoryContentHandler:
- No longer derive from BPackageContentHandler.
- Add hooks HandlePackage() and HandlePackageDone() that bracket the
attributes for a package. This is more explicit and robust than
handlers having to guess when one package ended and the next began.
* BRepositoryCache: Make use of BPackageInfoContentHandler. No need to
duplicate the code for reading a package info from package info
attributes.

hpkg format: Generalize user/global settings file attributes

* For all identifiers: Rename global settings file to global writable
file. We want to use the respective attribute also for other writable
files, not only settings files.
* User settings file/global writable file info/attribute: Add
isDirectory property/child attribute. This allows declaring global/
user settings directories associated with the package.

hpkg format: Add attributes for declaring users and groups

Move package info parser to its own file

hpkg format: Add attributes for declaring settings files

Global and user settings files can be declared. For global ones an
update policy can be specified. If not specified, the settings file is
not included in the package, but created by the program (or user) later.
If an update type is specified, it defines what to do with the settings
file when updating the package to a newer version.

User settings files are never included in the package; they are always
created by the program or the user. If the package contains a template/
default settings file, it can be declared, but for informative purposes
only.

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.

Rename BBlockBufferCache and friends to *Pool*

Also move BBlockBufferPoolNoLock into BHPKG namespace with the other
classes. Not sure why it wasn't there before.

Clone read support functionality for HPKG format version 1

It uses sub-namespace BPackage::BHPKG::V1. Unlike the one for the
current format version, the V1 version of BPackageInfoContentHandler
lives in BHPKG(::V1) sub-namespace and is private.

package kit: some fixes for multi-version support

* Use enums/constants/functions instead of preprocessor macros.
* Missing include in PackageInfoAttributeValue.h.
* PackageReaderImpl::Init(): Check version before header size and
return B_MISMATCHED_VALUES instead of B_BAD_DATA, if the version
doesn't match. This allows callers to determine the condition and
try a reader for a different version. A more flexible interface for
that case would be nice, but since we want to support the old package
version only temporarily, the current solution should be good enough.

Use libbe_build also on Haiku

This means the build tools will no longer be built against the host
platform's libbe, which avoids compatibility problems -- e.g. an
older Haiku host libbe may not have certain features the build tools
require -- and also makes the build behave more similiar on Haiku and
other platforms. The host libroot dependency still remains and is not
easy to get rid of.

Also remove some bits of BeOS/Dano/Zeta build support.

Add DownloadFileRequest

Downloads a file and optionally checks its checksum.

package daemon: Add support for activation change request

* daemon: Handle new request B_MESSAGE_COMMIT_TRANSACTION. It activates
and deactivates given sets of packages. The new packages must be
placed in a directory in the administrative directory. The daemon
moves them to the packages directory and the deactivated packages to
a subdirectory it creates. It also save the old activation state
there.
* Add private BActivationTransaction, describing an activation change
transaction.
* BDaemonClient: Add CommitTransaction(), which sends a given
BActivationTransaction as a B_MESSAGE_COMMIT_TRANSACTION request to
the daemon.

Completely untested yet.

Add BInstallationLocationInfo

Fix the libpackage_build build

* The recently introduced {No,Standard}ErrorOutput.cpp were still
missing.
* Add hack to allow building more correctly on Haiku. We were using
the installed package kit headers, not the ones from the working
directory.

Add class BPackageInfoContentHandler

A BPackageContentHandler subclass that initializes a BPackageInfo from
the read package attributes. Pulled out of RepositoryWriterImpl's
PackageContentHandler.

Pull class BPackageInfoSet out of BRepositoryCache

Split package kit DataReader.cpp

Pull AttributeDataReader and FDDataReader implementations out of
DataReader.cpp into own source files. Thus we can avoid dependencies
(e.g. to fs_attr code) we don't need/want.

<build>package: attribute iteration, license dir

PackageWriterImpl:
* Iterate through attributes using fs_read_attr_dir() instead of
readdir(). Makes it work correctly on the build platform.
* On the build platform look up the system licenses in their source
directory rather than based on find_directory().

Build libpackage for the build platform

Add class BPackageInfoContentHandler

A BPackageContentHandler subclass that initializes a BPackageInfo from
the read package attributes. Pulled out of RepositoryWriterImpl's
PackageContentHandler.

Pull class BPackageInfoSet out of BRepositoryCache

Split package kit DataReader.cpp

Pull AttributeDataReader and FDDataReader implementations out of
DataReader.cpp into own source files. Thus we can avoid dependencies
(e.g. to fs_attr code) we don't need/want.

<build>package: attribute iteration, license dir

PackageWriterImpl:
* Iterate through attributes using fs_read_attr_dir() instead of
readdir(). Makes it work correctly on the build platform.
* On the build platform look up the system licenses in their source
directory rather than based on find_directory().

Build libpackage for the build platform