Packages: signal added|removed packages

Update the package server so that it reports the
added and removed packages when it commits a
change so that clients are able to pickup and
act on the changes without iterating over all of
the packages.

Change-Id: I6feb52c34fc51a78e2282d8d5ca6cb6775b221ca
Reviewed-on: https://review.haiku-os.org/c/haiku/+/4141
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

package kit: Add pre-uninstall scripts feature.

Just like post-install scripts which run shortly after a package is
installed, pre-uninstall scripts are run just before a package is
removed. Implements enhancement #13427

* Fix script exit code handling vs script launch errors.
* Bump package and repo file version numbers due to new attribute,
unfortunately makes new .hpkg files not backwards compatible.
* Add pre-uninstall functionality, mostly cloning the post-install
except in a few places.
* Discover that _RunQueuedScripts() is never called, a future TODO:?
* Update package documentation for pre-uninstall scripts, and use of
the boot/post-install directory.

Change-Id: I45596255ce74bc102f6e5b606cbf83e4e4347a17
Reviewed-on: https://review.haiku-os.org/c/haiku/+/1504
Reviewed-by: Alexander G. M. Smith <agmsmith@ncf.ca>
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

package_daemon: queue post-install scripts to run after reboot

* Creates symlinks to the post-install scripts into the
administrative/queued-scripts directory.
* Upon reboot, post-install scripts in the queued-scripts
are run, and then the symlinks to the scripts removed.
* This solves the issue of post-install scripts not being
run when the package is installed/upgraded along with the
system packages, as experienced with the latest upgrade
to the bash package.

package daemon: Rework error and issue propagation to client

* BDaemonClient: Move inner class BCommitTransactionResult to top level
and make it public.
* BCommitTransactionResult:
- Add a whole bunch of specific error code enum values. Such an error
code is now the primary error, as opposed to before where we would
mix status_t and enum value errors. There's a systemError property
of type status_t which may provide additional information, though
(depending on the primary error type).
- Remove the errorMessage property. Due to mapping all errors to the
specific error codes this is no longer necessary. Mixing such a
message with another error description is also not very helpful when
it comes to localization (still not supported, though).
- Add several properties (paths, strings, error codes) that serve as
arguments to the primary error and are used by FullErrorMessage().
- Add issues property, a list of instances of new class
BTransactionIssue. Those describe non-critical issues (e.g. failed
update of a settings file) that occurred in the process of
committing the transaction. Those issues should be presented to the
user by the package management program.
* Exception: Adjust to transport the BCommitTransactionResult
properties.
* CommitTransactionHandler, FsTransactions, Root, Volume: Adjust to
BCommitTransactionResult/Exception changes.
* CommitTransactionHandler: Now requires a BCommitTransactionResult to
which it adds the issues it encounters. The reply BMessage is no
longer needed, though.
* Volume: Refactor common code from the three methods that use
CommitTransactionHandler into new method _CommitTransaction.

package daemon: Move volume change counter back to Volume

package daemon: Initial old packages state support

* VolumeState: Move locking to Volume.
* Package: Pull the file related functionality into new class
PackageFile. A Package refers to a PackageFile. A PackageFile can be
referenced by multiple Packages. PackageFiles are managed by the new
PackageFileManager.
* CommitTransactionHandler: Clone the passed volume state.
* Volume now manages two VolumeStates: A state reflecting the currently
active packages and the latest state (i.e. the one reflecting the
situation of the packages directory). Usually the two are the same,
unless an old state has been booted.
* The client interface hasn't been adjusted yet. Clients only see the
latest state.

package daemon: Retrieve all package directories from packagefs

package daemon: Pass volume state CommitTransactionHandler

... and remove the getter from Volume again.

package daemon: Move pending job accounting back to Volume

package daemon: Sever Volume-CommitTransactionHandler friendship

Move quite a bit of transaction related methods from Volume to
CommitTransactionHandler.

Move Volume::CommitTransactionHandler to top level

Also move constant definitions to Constants.h/cpp.

package daemon: Move Volume::State to top level

... and rename it to VolumeState.

package daemon: Handle location info request in app thread

* ... instead of queuing it for the job thread. The advantage is that
the request will be handled immediately and clients won't have to wait
for transactions (which may even require user feedback) to finish. It
complicates Volume a bit, since there are now two threads that may
access it. The shared data have been moved to a State object which is
protected by a lock.
* For commit transaction requests check whether another package request
is already pending/in progress before queuing a job. Fail immediately,
if there is.

Fixes bug #10039.

Initial changes to remove /boot/common

* find_directory() and hard-coded paths use /boot/system instead of
/boot/common.
* The build system creates the writable directories in /boot/system
instead of /boot/common.
* The build system no longer installs any packages in /boot/common.

package daemon: Implement writable files handling

* Move Volume::Exception to top level and into own files.
* Add utility class FSUtils, move Volume::RelativePath there and add a
bunch of FS utility functionality.
* Add FSTransaction, a helper class to record FS operations and revert
them.
* When activating a package we now extract the writable
files/directories it declares. The handling is not quite complete:
- We don't handle merges yet. I.e. the user will have to do that
manually for now.
- We don't propagate issues/infos regarding the writable files (e.g.
that a manual intervention is required) to the user yet.

package daemon: Implement dependency resolution support

... for package changes performed manually by the user (i.e.
adding/removing files in a packages directory). Currently only done for
the system root. Alternative roots should still work as before, so this
shouldn't affect haikuporter (not tested yet).

Needs some more work (e.g. the GUI part).

package daemon: Add support for activation change request

* daemon: Handle new request B_MESSAGE_COMMIT_TRANSACTION. It activates
and deactivates given sets of packages. The new packages must be
placed in a directory in the administrative directory. The daemon
moves them to the packages directory and the deactivated packages to
a subdirectory it creates. It also save the old activation state
there.
* Add private BActivationTransaction, describing an activation change
transaction.
* BDaemonClient: Add CommitTransaction(), which sends a given
BActivationTransaction as a B_MESSAGE_COMMIT_TRANSACTION request to
the daemon.

Completely untested yet.

Implement getting a BInstallationLocationInfo

* Rename PackageDaemonDefs.h to DaemonDefs.h.
* Replace the MESSAGE_GET_PACKAGES by the new
B_MESSAGE_GET_INSTALLATION_LOCATION_INFO, which not only returns the
packages, but also other information about the installation location.
* daemon: Volume: Implement a change count which is bumped whenever
packages are activated/deactivated/added/removed. Cache the reply
for a location info request, using the change count to check whether
it is still up-to-date.
* Add private BDaemonClient for communication with the daemon.
* BRoster:
- Add GetInstallationLocationInfo() using BDaemonClient.
- Reimplement GetActivePackages(), using
GetInstallationLocationInfo().

package kit: get active packages list from daemon

* daemon: Implement private message protocol to retrieve the active
packages.
* BPackageRoster::GetActivePackages(): Get the active packages list
from the daemon.

package daemon: Properly qualify private package headers

... instead of adding the private package header directory to the
include paths.

package daemon: On changes write a file with the active packages

package daemon: Check volume package dependencies initially

Dumps the result (i.e. found problems and solutions) to the syslog.
Eventually the user should be asked what to do when inconsistencies are
encountered.

package daemon: De/-activate all changed packages together

* We first process the node monitoring events, collecting the required
package activation changes, then apply all changes together.
* Change the PackageFSActivationChangeItem/-Request structs. The former
is no longer variable in size, which makes it easier to work with.

package daemon: Add a Listener interface for Volume

* This way, Volume doesn't have to know Root.
* Suppress listener notifications for node monitoring events when there
are were already events pending.

package daemon: Make all work with the packages asynchronous

There's now a worker thread per Root that does all the work that can
take time. Node monitoring notifications received in the main thread are
just pushed into the worker's job queue, so the application looper
remains responsive.

package daemon: Add package monitoring and de-/activation

* packagefs: Disable (comment out) node monitoring of the packages
directory.
* package daemon:
- When a packagefs volume is added load the respective packages
directory and get from the volume which of the packages are
activated.
- Add node monitoring for the packages directory and
activate/deactivate packages as packages are added/removed.

Add the humble beginnings of the package daemon

It doesn't really do anything yet save for tracking what packagefs
volumes are mounted and unmounted.

package_daemon: queue post-install scripts to run after reboot

* Creates symlinks to the post-install scripts into the
administrative/queued-scripts directory.
* Upon reboot, post-install scripts in the queued-scripts
are run, and then the symlinks to the scripts removed.
* This solves the issue of post-install scripts not being
run when the package is installed/upgraded along with the
system packages, as experienced with the latest upgrade
to the bash package.

package daemon: Rework error and issue propagation to client

* BDaemonClient: Move inner class BCommitTransactionResult to top level
and make it public.
* BCommitTransactionResult:
- Add a whole bunch of specific error code enum values. Such an error
code is now the primary error, as opposed to before where we would
mix status_t and enum value errors. There's a systemError property
of type status_t which may provide additional information, though
(depending on the primary error type).
- Remove the errorMessage property. Due to mapping all errors to the
specific error codes this is no longer necessary. Mixing such a
message with another error description is also not very helpful when
it comes to localization (still not supported, though).
- Add several properties (paths, strings, error codes) that serve as
arguments to the primary error and are used by FullErrorMessage().
- Add issues property, a list of instances of new class
BTransactionIssue. Those describe non-critical issues (e.g. failed
update of a settings file) that occurred in the process of
committing the transaction. Those issues should be presented to the
user by the package management program.
* Exception: Adjust to transport the BCommitTransactionResult
properties.
* CommitTransactionHandler, FsTransactions, Root, Volume: Adjust to
BCommitTransactionResult/Exception changes.
* CommitTransactionHandler: Now requires a BCommitTransactionResult to
which it adds the issues it encounters. The reply BMessage is no
longer needed, though.
* Volume: Refactor common code from the three methods that use
CommitTransactionHandler into new method _CommitTransaction.

package daemon: Move volume change counter back to Volume

package daemon: Initial old packages state support

* VolumeState: Move locking to Volume.
* Package: Pull the file related functionality into new class
PackageFile. A Package refers to a PackageFile. A PackageFile can be
referenced by multiple Packages. PackageFiles are managed by the new
PackageFileManager.
* CommitTransactionHandler: Clone the passed volume state.
* Volume now manages two VolumeStates: A state reflecting the currently
active packages and the latest state (i.e. the one reflecting the
situation of the packages directory). Usually the two are the same,
unless an old state has been booted.
* The client interface hasn't been adjusted yet. Clients only see the
latest state.

package daemon: Retrieve all package directories from packagefs

package daemon: Pass volume state CommitTransactionHandler

... and remove the getter from Volume again.

package daemon: Move pending job accounting back to Volume

package daemon: Sever Volume-CommitTransactionHandler friendship

Move quite a bit of transaction related methods from Volume to
CommitTransactionHandler.

Move Volume::CommitTransactionHandler to top level

Also move constant definitions to Constants.h/cpp.

package daemon: Move Volume::State to top level

... and rename it to VolumeState.

package daemon: Handle location info request in app thread

* ... instead of queuing it for the job thread. The advantage is that
the request will be handled immediately and clients won't have to wait
for transactions (which may even require user feedback) to finish. It
complicates Volume a bit, since there are now two threads that may
access it. The shared data have been moved to a State object which is
protected by a lock.
* For commit transaction requests check whether another package request
is already pending/in progress before queuing a job. Fail immediately,
if there is.

Fixes bug #10039.

Initial changes to remove /boot/common

* find_directory() and hard-coded paths use /boot/system instead of
/boot/common.
* The build system creates the writable directories in /boot/system
instead of /boot/common.
* The build system no longer installs any packages in /boot/common.

package daemon: Implement writable files handling

* Move Volume::Exception to top level and into own files.
* Add utility class FSUtils, move Volume::RelativePath there and add a
bunch of FS utility functionality.
* Add FSTransaction, a helper class to record FS operations and revert
them.
* When activating a package we now extract the writable
files/directories it declares. The handling is not quite complete:
- We don't handle merges yet. I.e. the user will have to do that
manually for now.
- We don't propagate issues/infos regarding the writable files (e.g.
that a manual intervention is required) to the user yet.

package daemon: Implement dependency resolution support

... for package changes performed manually by the user (i.e.
adding/removing files in a packages directory). Currently only done for
the system root. Alternative roots should still work as before, so this
shouldn't affect haikuporter (not tested yet).

Needs some more work (e.g. the GUI part).

package daemon: Add support for activation change request

* daemon: Handle new request B_MESSAGE_COMMIT_TRANSACTION. It activates
and deactivates given sets of packages. The new packages must be
placed in a directory in the administrative directory. The daemon
moves them to the packages directory and the deactivated packages to
a subdirectory it creates. It also save the old activation state
there.
* Add private BActivationTransaction, describing an activation change
transaction.
* BDaemonClient: Add CommitTransaction(), which sends a given
BActivationTransaction as a B_MESSAGE_COMMIT_TRANSACTION request to
the daemon.

Completely untested yet.

Implement getting a BInstallationLocationInfo

* Rename PackageDaemonDefs.h to DaemonDefs.h.
* Replace the MESSAGE_GET_PACKAGES by the new
B_MESSAGE_GET_INSTALLATION_LOCATION_INFO, which not only returns the
packages, but also other information about the installation location.
* daemon: Volume: Implement a change count which is bumped whenever
packages are activated/deactivated/added/removed. Cache the reply
for a location info request, using the change count to check whether
it is still up-to-date.
* Add private BDaemonClient for communication with the daemon.
* BRoster:
- Add GetInstallationLocationInfo() using BDaemonClient.
- Reimplement GetActivePackages(), using
GetInstallationLocationInfo().

package kit: get active packages list from daemon

* daemon: Implement private message protocol to retrieve the active
packages.
* BPackageRoster::GetActivePackages(): Get the active packages list
from the daemon.

package daemon: Properly qualify private package headers

... instead of adding the private package header directory to the
include paths.

package daemon: On changes write a file with the active packages

package daemon: Check volume package dependencies initially

Dumps the result (i.e. found problems and solutions) to the syslog.
Eventually the user should be asked what to do when inconsistencies are
encountered.

package daemon: De/-activate all changed packages together

* We first process the node monitoring events, collecting the required
package activation changes, then apply all changes together.
* Change the PackageFSActivationChangeItem/-Request structs. The former
is no longer variable in size, which makes it easier to work with.

package daemon: Add a Listener interface for Volume

* This way, Volume doesn't have to know Root.
* Suppress listener notifications for node monitoring events when there
are were already events pending.

package daemon: Make all work with the packages asynchronous

There's now a worker thread per Root that does all the work that can
take time. Node monitoring notifications received in the main thread are
just pushed into the worker's job queue, so the application looper
remains responsive.

package daemon: Add package monitoring and de-/activation

* packagefs: Disable (comment out) node monitoring of the packages
directory.
* package daemon:
- When a packagefs volume is added load the respective packages
directory and get from the volume which of the packages are
activated.
- Add node monitoring for the packages directory and
activate/deactivate packages as packages are added/removed.

Add the humble beginnings of the package daemon

It doesn't really do anything yet save for tracking what packagefs
volumes are mounted and unmounted.