Package server: turn down logging a bit.

This prints two quite long lists of packages that are mixed up with
other logs, making them annoying to read. It is also not very useful now
that the package server is working quite reliably.

Change-Id: Ibf9d5fc3eadc5cef42a78aea831e0a035e248de3
Reviewed-on: https://review.haiku-os.org/c/haiku/+/7366
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

integrate AutoDeleter's into pointers

Change-Id: I6c3925a7aec4d0647c76c2a03aad7b08985d7166
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3490
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

Packages: signal added|removed packages

Update the package server so that it reports the
added and removed packages when it commits a
change so that clients are able to pickup and
act on the changes without iterating over all of
the packages.

Change-Id: I6feb52c34fc51a78e2282d8d5ca6cb6775b221ca
Reviewed-on: https://review.haiku-os.org/c/haiku/+/4141
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

Package Kit: Proper Installation for First Boot Packages

Do the final installation operations for all the packages in the
/system/packages directory when the OS is booted for the first time.

This will run their post-install scripts, create users, groups and generate
settings files (marked with a package version attribute). Previously we just
ran all the shell scripts found in the /system/boot/post-install directory
(don't do that as much now).

Fixes bug #14382

This patch has simpler code flow in CommitTransactionHandler::_ApplyChanges
Tested on 32 and 64 bit systems. Once it's official, need to remove the
open_ssh redundant post-install script that creates users etc. from HaikuPorts.
Now we can notice bugs like package version attributes on settings files aren't
fully working. :-)

Didn't remove special case for add_catalog_entry_attributes.sh since it
still does stuff that the build system doesn't do. Might be able to add
that script as part of the Haiku.hpkg. See change 3751 for removing it,
https://review.haiku-os.org/c/haiku/+/3751

Change-Id: I3807b78042fdb70e5a79eca2e2a45816ece0236f
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2342
Reviewed-by: Alexander G. M. Smith <agmsmith@ncf.ca>
Reviewed-by: Niels Sascha Reedijk <niels.reedijk@gmail.com>
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

package_daemon: fix path of activated-package files

Until now the activated-packages file was not used for anything, because
it was not looked for in the correct place.

Change-Id: Ib38f28b52e85af6900c0e2b61a9621734428ad25
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2341
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

package_daemon: Actually return transaction errors instead of OK always.

This "fixes" #10959, though now the varying potential causes of that
will now print actual errors earlier instead of the error message
in that ticket.

package_daemon: fix an issue with package overwrite

When overwriting a package the sequence is: remove old, add new package.
On remove we add package to ToBeDeactivated set. This is ok.
On adding new package there is a check if package exists or not. On true
return. But package still in ToBeDeactivated set.

Fixes #10038

Signed-off-by: Augustin Cavalier <waddlesplash@gmail.com>
Signed-off-by: Axel Dörfler <axeld@pinc-software.de>

package_daemon: fix crash on system update

We can't delete old volume state while it's still active and can be
accessed until reboot.

Fixes #10898

Signed-off-by: Augustin Cavalier <waddlesplash@gmail.com>
Signed-off-by: Axel Dörfler <axeld@pinc-software.de>

package_daemon: queue post-install scripts to run after reboot

* Creates symlinks to the post-install scripts into the
administrative/queued-scripts directory.
* Upon reboot, post-install scripts in the queued-scripts
are run, and then the symlinks to the scripts removed.
* This solves the issue of post-install scripts not being
run when the package is installed/upgraded along with the
system packages, as experienced with the latest upgrade
to the bash package.

Add watching support for installation location package changes

Can be requested/stopped via BPackageRoster::{Start,Stop}Watching().
The notification message has the what code B_PACKAGE_UPDATE and contains
fields "event", "location", and "change count".

package daemon: Rework error and issue propagation to client

* BDaemonClient: Move inner class BCommitTransactionResult to top level
and make it public.
* BCommitTransactionResult:
- Add a whole bunch of specific error code enum values. Such an error
code is now the primary error, as opposed to before where we would
mix status_t and enum value errors. There's a systemError property
of type status_t which may provide additional information, though
(depending on the primary error type).
- Remove the errorMessage property. Due to mapping all errors to the
specific error codes this is no longer necessary. Mixing such a
message with another error description is also not very helpful when
it comes to localization (still not supported, though).
- Add several properties (paths, strings, error codes) that serve as
arguments to the primary error and are used by FullErrorMessage().
- Add issues property, a list of instances of new class
BTransactionIssue. Those describe non-critical issues (e.g. failed
update of a settings file) that occurred in the process of
committing the transaction. Those issues should be presented to the
user by the package management program.
* Exception: Adjust to transport the BCommitTransactionResult
properties.
* CommitTransactionHandler, FsTransactions, Root, Volume: Adjust to
BCommitTransactionResult/Exception changes.
* CommitTransactionHandler: Now requires a BCommitTransactionResult to
which it adds the issues it encounters. The reply BMessage is no
longer needed, though.
* Volume: Refactor common code from the three methods that use
CommitTransactionHandler into new method _CommitTransaction.

package daemon: Volume: Fix destruction order

We must destroy the VolumeStates before destroying the
PackageFileManager that manages their packages.

Kudos to Rene for analyzing.

package daemon: Fix manual package [de/]activation

CommitTransactionHandler: Since the VolumeState is now always copied
(including the contained packages), we can't just use the passed in
packagesAlready{Added,Removed} sets, since they refer to the original
packages.

BInstallationLocationInfo: Provide more info

* If the active state is not the latest state, also provide a package
set for it.
* If an old state has been booted, provide its name.

package daemon: Move volume change counter back to Volume

package daemon: Initial old packages state support

* VolumeState: Move locking to Volume.
* Package: Pull the file related functionality into new class
PackageFile. A Package refers to a PackageFile. A PackageFile can be
referenced by multiple Packages. PackageFiles are managed by the new
PackageFileManager.
* CommitTransactionHandler: Clone the passed volume state.
* Volume now manages two VolumeStates: A state reflecting the currently
active packages and the latest state (i.e. the one reflecting the
situation of the packages directory). Usually the two are the same,
unless an old state has been booted.
* The client interface hasn't been adjusted yet. Clients only see the
latest state.

Fix #10804: package_daemon crashing in Volume destructor.

* use delete [] for fPackagesDirectories, as it has been allocated
via array new

packagefs: Extend PACKAGE_FS_OPERATION_GET_PACKAGE_INFOS ioctl

Also return the package file names.

package daemon: Retrieve all package directories from packagefs

package daemon: Pass volume state CommitTransactionHandler

... and remove the getter from Volume again.

package daemon: Move pending job accounting back to Volume

package daemon: Sever Volume-CommitTransactionHandler friendship

Move quite a bit of transaction related methods from Volume to
CommitTransactionHandler.

Move Volume::CommitTransactionHandler to top level

Also move constant definitions to Constants.h/cpp.

package daemon: Move Volume::State to top level

... and rename it to VolumeState.

Adjust packagefs ioctl interface to support old states

* PackageFSVolumeInfo: Add the directories for all relevant states.
* PackageFSPackageInfo: Include the package file's parent directory node
ref.

Package daemon and package kit still don't support old states yet.

package daemon: Handle location info request in app thread

* ... instead of queuing it for the job thread. The advantage is that
the request will be handled immediately and clients won't have to wait
for transactions (which may even require user feedback) to finish. It
complicates Volume a bit, since there are now two threads that may
access it. The shared data have been moved to a State object which is
protected by a lock.
* For commit transaction requests check whether another package request
is already pending/in progress before queuing a job. Fail immediately,
if there is.

Fixes bug #10039.

package daemon: Fix handling of multiple writable files

When attempting to extract the second writable file from the package,
we noticed that the root directory already existed and assumed
everything had been already extracted. Now we first extract all files
and process them one by one afterwards. Fixes #10131.

package daemon: Fix error handling of manually added packages

Since we don't move those packages out of the "packages" directory, we
don't want to remove their respective object from the hash tables
either. Fixes #10102.

package daemon: Fix typo in error message

Remove /boot/common for good

* Remove support for the "common" installation location from packagefs,
package kit, package daemon, package managers.
* Rename the B_COMMON_*_DIRECTORY constants referring to writable
directories to B_SYSTEM_*_DIRECTORY.
* Remove/adjust the use of various B_COMMON_*_DIRECTORY constants.
I'm sure some occurrence still remain. They can be adjusted when the
remaining B_COMMON_*_DIRECTORY constants are removed.

Initial changes to remove /boot/common

* find_directory() and hard-coded paths use /boot/system instead of
/boot/common.
* The build system creates the writable directories in /boot/system
instead of /boot/common.
* The build system no longer installs any packages in /boot/common.

package daemon: Implement writable files handling

* Move Volume::Exception to top level and into own files.
* Add utility class FSUtils, move Volume::RelativePath there and add a
bunch of FS utility functionality.
* Add FSTransaction, a helper class to record FS operations and revert
them.
* When activating a package we now extract the writable
files/directories it declares. The handling is not quite complete:
- We don't handle merges yet. I.e. the user will have to do that
manually for now.
- We don't propagate issues/infos regarding the writable files (e.g.
that a manual intervention is required) to the user yet.

package daemon: Handle post-installation scripts, users/groups

... specified by a package when it is going to be activated. We don't
try to remove users/groups when deactivating packages yet, nor is the
user properly identified in all error cases.

package daemon: Use CommitTransactionHandler in all cases

... also when only activating/deactivating already moved packages.

package daemon: Fix use of destroyed objects

Fixes #9957.

package daemon: Ignore only entry events of packages to remove

package daemon: Implement dependency resolution support

... for package changes performed manually by the user (i.e.
adding/removing files in a packages directory). Currently only done for
the system root. Alternative roots should still work as before, so this
shouldn't affect haikuporter (not tested yet).

Needs some more work (e.g. the GUI part).

package daemon: Delay handling node monitor events to group them

package daemon: Volume: fix package removal after transaction

Volume::_ChangePackageActivation(): Refactoring oversight:
fPackagesToBeDeactivated was still used instead of the
packagesToDeactivate parameter, so the packages wouldn't be removed from
the hash tables when invoked from the CommitTransactionHandler. Fixes
#9921.

package_daemon: create the administrative dir when mounted

package daemon: Fix Volume::_WriteActivationFile()

* Wasn't using the toActivate and toDeactivate parameters, but the
attributes. Thus failed when called from the
CommitTransactionHandler.
* Volume::_ChangePackageActivation(): Fix debug output.

package daemon: Volume::_WriteActivationFile(): Init _entry

package daemon: Fix check in Volume::_PackagesEntryCreated()

package daemon: Add support for activation change request

* daemon: Handle new request B_MESSAGE_COMMIT_TRANSACTION. It activates
and deactivates given sets of packages. The new packages must be
placed in a directory in the administrative directory. The daemon
moves them to the packages directory and the deactivated packages to
a subdirectory it creates. It also save the old activation state
there.
* Add private BActivationTransaction, describing an activation change
transaction.
* BDaemonClient: Add CommitTransaction(), which sends a given
BActivationTransaction as a B_MESSAGE_COMMIT_TRANSACTION request to
the daemon.

Completely untested yet.

Implement getting a BInstallationLocationInfo

* Rename PackageDaemonDefs.h to DaemonDefs.h.
* Replace the MESSAGE_GET_PACKAGES by the new
B_MESSAGE_GET_INSTALLATION_LOCATION_INFO, which not only returns the
packages, but also other information about the installation location.
* daemon: Volume: Implement a change count which is bumped whenever
packages are activated/deactivated/added/removed. Cache the reply
for a location info request, using the change count to check whether
it is still up-to-date.
* Add private BDaemonClient for communication with the daemon.
* BRoster:
- Add GetInstallationLocationInfo() using BDaemonClient.
- Reimplement GetActivePackages(), using
GetInstallationLocationInfo().

Rename "packages" subdir "config" to "administrative"

... to avoid the impression that it contains something the user can
play with.

package kit: get active packages list from daemon

* daemon: Implement private message protocol to retrieve the active
packages.
* BPackageRoster::GetActivePackages(): Get the active packages list
from the daemon.

package daemon: Properly qualify private package headers

... instead of adding the private package header directory to the
include paths.

package daemon: On changes write a file with the active packages

package daemon: Check volume package dependencies initially

Dumps the result (i.e. found problems and solutions) to the syslog.
Eventually the user should be asked what to do when inconsistencies are
encountered.

package daemon: Volume::InitPackages(): Fix error case

Don't assign fListener, when starting node monitoring fails.

package daemon: De/-activate all changed packages together

* We first process the node monitoring events, collecting the required
package activation changes, then apply all changes together.
* Change the PackageFSActivationChangeItem/-Request structs. The former
is no longer variable in size, which makes it easier to work with.

packagefs/daemon: Ignore packages/ entries without .hpkg suffix

package daemon: Volume: Make sure move entry events don't get split

package daemon: Add a Listener interface for Volume

* This way, Volume doesn't have to know Root.
* Suppress listener notifications for node monitoring events when there
are were already events pending.

package daemon: Make all work with the packages asynchronous

There's now a worker thread per Root that does all the work that can
take time. Node monitoring notifications received in the main thread are
just pushed into the worker's job queue, so the application looper
remains responsive.

package daemon: Add package monitoring and de-/activation

* packagefs: Disable (comment out) node monitoring of the packages
directory.
* package daemon:
- When a packagefs volume is added load the respective packages
directory and get from the volume which of the packages are
activated.
- Add node monitoring for the packages directory and
activate/deactivate packages as packages are added/removed.

Add the humble beginnings of the package daemon

It doesn't really do anything yet save for tracking what packagefs
volumes are mounted and unmounted.

package_daemon: queue post-install scripts to run after reboot

* Creates symlinks to the post-install scripts into the
administrative/queued-scripts directory.
* Upon reboot, post-install scripts in the queued-scripts
are run, and then the symlinks to the scripts removed.
* This solves the issue of post-install scripts not being
run when the package is installed/upgraded along with the
system packages, as experienced with the latest upgrade
to the bash package.

Add watching support for installation location package changes

Can be requested/stopped via BPackageRoster::{Start,Stop}Watching().
The notification message has the what code B_PACKAGE_UPDATE and contains
fields "event", "location", and "change count".

package daemon: Rework error and issue propagation to client

* BDaemonClient: Move inner class BCommitTransactionResult to top level
and make it public.
* BCommitTransactionResult:
- Add a whole bunch of specific error code enum values. Such an error
code is now the primary error, as opposed to before where we would
mix status_t and enum value errors. There's a systemError property
of type status_t which may provide additional information, though
(depending on the primary error type).
- Remove the errorMessage property. Due to mapping all errors to the
specific error codes this is no longer necessary. Mixing such a
message with another error description is also not very helpful when
it comes to localization (still not supported, though).
- Add several properties (paths, strings, error codes) that serve as
arguments to the primary error and are used by FullErrorMessage().
- Add issues property, a list of instances of new class
BTransactionIssue. Those describe non-critical issues (e.g. failed
update of a settings file) that occurred in the process of
committing the transaction. Those issues should be presented to the
user by the package management program.
* Exception: Adjust to transport the BCommitTransactionResult
properties.
* CommitTransactionHandler, FsTransactions, Root, Volume: Adjust to
BCommitTransactionResult/Exception changes.
* CommitTransactionHandler: Now requires a BCommitTransactionResult to
which it adds the issues it encounters. The reply BMessage is no
longer needed, though.
* Volume: Refactor common code from the three methods that use
CommitTransactionHandler into new method _CommitTransaction.

package daemon: Volume: Fix destruction order

We must destroy the VolumeStates before destroying the
PackageFileManager that manages their packages.

Kudos to Rene for analyzing.

package daemon: Fix manual package [de/]activation

CommitTransactionHandler: Since the VolumeState is now always copied
(including the contained packages), we can't just use the passed in
packagesAlready{Added,Removed} sets, since they refer to the original
packages.

BInstallationLocationInfo: Provide more info

* If the active state is not the latest state, also provide a package
set for it.
* If an old state has been booted, provide its name.

package daemon: Move volume change counter back to Volume

package daemon: Initial old packages state support

* VolumeState: Move locking to Volume.
* Package: Pull the file related functionality into new class
PackageFile. A Package refers to a PackageFile. A PackageFile can be
referenced by multiple Packages. PackageFiles are managed by the new
PackageFileManager.
* CommitTransactionHandler: Clone the passed volume state.
* Volume now manages two VolumeStates: A state reflecting the currently
active packages and the latest state (i.e. the one reflecting the
situation of the packages directory). Usually the two are the same,
unless an old state has been booted.
* The client interface hasn't been adjusted yet. Clients only see the
latest state.

Fix #10804: package_daemon crashing in Volume destructor.

* use delete [] for fPackagesDirectories, as it has been allocated
via array new

packagefs: Extend PACKAGE_FS_OPERATION_GET_PACKAGE_INFOS ioctl

Also return the package file names.

package daemon: Retrieve all package directories from packagefs

package daemon: Pass volume state CommitTransactionHandler

... and remove the getter from Volume again.

package daemon: Move pending job accounting back to Volume

package daemon: Sever Volume-CommitTransactionHandler friendship

Move quite a bit of transaction related methods from Volume to
CommitTransactionHandler.

Move Volume::CommitTransactionHandler to top level

Also move constant definitions to Constants.h/cpp.

package daemon: Move Volume::State to top level

... and rename it to VolumeState.

Adjust packagefs ioctl interface to support old states

* PackageFSVolumeInfo: Add the directories for all relevant states.
* PackageFSPackageInfo: Include the package file's parent directory node
ref.

Package daemon and package kit still don't support old states yet.

package daemon: Handle location info request in app thread

* ... instead of queuing it for the job thread. The advantage is that
the request will be handled immediately and clients won't have to wait
for transactions (which may even require user feedback) to finish. It
complicates Volume a bit, since there are now two threads that may
access it. The shared data have been moved to a State object which is
protected by a lock.
* For commit transaction requests check whether another package request
is already pending/in progress before queuing a job. Fail immediately,
if there is.

Fixes bug #10039.

package daemon: Fix handling of multiple writable files

When attempting to extract the second writable file from the package,
we noticed that the root directory already existed and assumed
everything had been already extracted. Now we first extract all files
and process them one by one afterwards. Fixes #10131.

package daemon: Fix error handling of manually added packages

Since we don't move those packages out of the "packages" directory, we
don't want to remove their respective object from the hash tables
either. Fixes #10102.

package daemon: Fix typo in error message

Remove /boot/common for good

* Remove support for the "common" installation location from packagefs,
package kit, package daemon, package managers.
* Rename the B_COMMON_*_DIRECTORY constants referring to writable
directories to B_SYSTEM_*_DIRECTORY.
* Remove/adjust the use of various B_COMMON_*_DIRECTORY constants.
I'm sure some occurrence still remain. They can be adjusted when the
remaining B_COMMON_*_DIRECTORY constants are removed.

Initial changes to remove /boot/common

* find_directory() and hard-coded paths use /boot/system instead of
/boot/common.
* The build system creates the writable directories in /boot/system
instead of /boot/common.
* The build system no longer installs any packages in /boot/common.

package daemon: Implement writable files handling

* Move Volume::Exception to top level and into own files.
* Add utility class FSUtils, move Volume::RelativePath there and add a
bunch of FS utility functionality.
* Add FSTransaction, a helper class to record FS operations and revert
them.
* When activating a package we now extract the writable
files/directories it declares. The handling is not quite complete:
- We don't handle merges yet. I.e. the user will have to do that
manually for now.
- We don't propagate issues/infos regarding the writable files (e.g.
that a manual intervention is required) to the user yet.

package daemon: Handle post-installation scripts, users/groups

... specified by a package when it is going to be activated. We don't
try to remove users/groups when deactivating packages yet, nor is the
user properly identified in all error cases.

package daemon: Use CommitTransactionHandler in all cases

... also when only activating/deactivating already moved packages.

package daemon: Fix use of destroyed objects

Fixes #9957.

package daemon: Ignore only entry events of packages to remove

package daemon: Implement dependency resolution support

... for package changes performed manually by the user (i.e.
adding/removing files in a packages directory). Currently only done for
the system root. Alternative roots should still work as before, so this
shouldn't affect haikuporter (not tested yet).

Needs some more work (e.g. the GUI part).

package daemon: Delay handling node monitor events to group them

package daemon: Volume: fix package removal after transaction

Volume::_ChangePackageActivation(): Refactoring oversight:
fPackagesToBeDeactivated was still used instead of the
packagesToDeactivate parameter, so the packages wouldn't be removed from
the hash tables when invoked from the CommitTransactionHandler. Fixes
#9921.

package_daemon: create the administrative dir when mounted

package daemon: Fix Volume::_WriteActivationFile()

* Wasn't using the toActivate and toDeactivate parameters, but the
attributes. Thus failed when called from the
CommitTransactionHandler.
* Volume::_ChangePackageActivation(): Fix debug output.

package daemon: Volume::_WriteActivationFile(): Init _entry

package daemon: Fix check in Volume::_PackagesEntryCreated()

package daemon: Add support for activation change request

* daemon: Handle new request B_MESSAGE_COMMIT_TRANSACTION. It activates
and deactivates given sets of packages. The new packages must be
placed in a directory in the administrative directory. The daemon
moves them to the packages directory and the deactivated packages to
a subdirectory it creates. It also save the old activation state
there.
* Add private BActivationTransaction, describing an activation change
transaction.
* BDaemonClient: Add CommitTransaction(), which sends a given
BActivationTransaction as a B_MESSAGE_COMMIT_TRANSACTION request to
the daemon.

Completely untested yet.

Implement getting a BInstallationLocationInfo

* Rename PackageDaemonDefs.h to DaemonDefs.h.
* Replace the MESSAGE_GET_PACKAGES by the new
B_MESSAGE_GET_INSTALLATION_LOCATION_INFO, which not only returns the
packages, but also other information about the installation location.
* daemon: Volume: Implement a change count which is bumped whenever
packages are activated/deactivated/added/removed. Cache the reply
for a location info request, using the change count to check whether
it is still up-to-date.
* Add private BDaemonClient for communication with the daemon.
* BRoster:
- Add GetInstallationLocationInfo() using BDaemonClient.
- Reimplement GetActivePackages(), using
GetInstallationLocationInfo().

Rename "packages" subdir "config" to "administrative"

... to avoid the impression that it contains something the user can
play with.

package kit: get active packages list from daemon

* daemon: Implement private message protocol to retrieve the active
packages.
* BPackageRoster::GetActivePackages(): Get the active packages list
from the daemon.

package daemon: Properly qualify private package headers

... instead of adding the private package header directory to the
include paths.

package daemon: On changes write a file with the active packages

package daemon: Check volume package dependencies initially

Dumps the result (i.e. found problems and solutions) to the syslog.
Eventually the user should be asked what to do when inconsistencies are
encountered.

package daemon: Volume::InitPackages(): Fix error case

Don't assign fListener, when starting node monitoring fails.

package daemon: De/-activate all changed packages together

* We first process the node monitoring events, collecting the required
package activation changes, then apply all changes together.
* Change the PackageFSActivationChangeItem/-Request structs. The former
is no longer variable in size, which makes it easier to work with.

packagefs/daemon: Ignore packages/ entries without .hpkg suffix

package daemon: Volume: Make sure move entry events don't get split

package daemon: Add a Listener interface for Volume

* This way, Volume doesn't have to know Root.
* Suppress listener notifications for node monitoring events when there
are were already events pending.

package daemon: Make all work with the packages asynchronous

There's now a worker thread per Root that does all the work that can
take time. Node monitoring notifications received in the main thread are
just pushed into the worker's job queue, so the application looper
remains responsive.

package daemon: Add package monitoring and de-/activation

* packagefs: Disable (comment out) node monitoring of the packages
directory.
* package daemon:
- When a packagefs volume is added load the respective packages
directory and get from the volume which of the packages are
activated.
- Add node monitoring for the packages directory and
activate/deactivate packages as packages are added/removed.

Add the humble beginnings of the package daemon

It doesn't really do anything yet save for tracking what packagefs
volumes are mounted and unmounted.