HaikuDepot : Refactor Screenshots

Rework the way in which the application deals
with screenshots so that there is a repository
of screenshots that is referenced from
identifiers on the package model and not that
the screenshot bitmaps are stored with the
model and not cleaned up.

Change-Id: I632def96852a518b3f6b1c8b81909de117bbbc54
Reviewed-on: https://review.haiku-os.org/c/haiku/+/7296
Reviewed-by: Andrew Lindesay <apl@lindesay.co.nz>

HaikuDepot : Break Out Pkg Model

The data model for packages contains a number
of components which all reside in a single
source header and implementation file making
maintenance and modifications difficult. This
change will break the sub-models out into
separate files without making any functional
changes.

Change-Id: I9c2557bc1e5a33661145c55290a0baad4084fd49
Reviewed-on: https://review.haiku-os.org/c/haiku/+/7181
Reviewed-by: Adrien Destugues <pulkomandy@pulkomandy.tk>
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>

HaikuDepot: Token Based Authentication

This switches the application over from using basic
authentication to using token-based authentication in
preparation for later using Open-ID based
authentication flows. The application version is also
bumped in order that the server can detect this version
at some later date in the future when it no longer
supports basic authentication itself.

Change-Id: I7addde1d57503c58d6bcd54908f22f66830c0c59
Reviewed-on: https://review.haiku-os.org/c/haiku/+/6944
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>
Reviewed-by: Jérôme Duval <jerome.duval@gmail.com>

HaikuDepot: reflect package install changes

This update will mean that HaikuDepot is able to detect
install / uninstall actions on packages at the system
level and reflect this in the user interface.

Fixes #15879, #15994 and #15046.
Relates to #11674.

Also cleans up disused code in the WorkStatusView.

Change-Id: Ide2d5d93c2f71915bc7cfe7d3628e5f343e43f35
Reviewed-on: https://review.haiku-os.org/c/haiku/+/4145
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>

HaikuDepot: Increment Pkg View Counter

Closes #16814

Change-Id: Idf451628b4680fb33563dbf4817bd11049c326b5
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3803
Reviewed-by: Jérôme Duval <jerome.duval@gmail.com>

HaikuDepot: Remove Custom List

Further removal of the use of custom list class;
this time with the package lists.

Relates To #15534
Change-Id: I1f01ed9d5ddbd7754097ce0adbf505d6ba17fd2f
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3732
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

HaikuDepot: Remove Custom List

Further removal of the use of custom list class.

Relates To #15534

Change-Id: I1d84b562b334e5e52ed4772bad3a6aea7b715562
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3657
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

HaikuDepot: Lists; Categories + Stabilities

Remove use of custom list class where it is not
really required in the area of Categories. Also
introduces stabilities relayed over from the
Server and used in the HD user interface instead
of being hard-coded.

Relates To #15534

Change-Id: Ib71141e71cd4a0b4882827e2e59b62072de01b4b
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3331
Reviewed-by: Jérôme Duval <jerome.duval@gmail.com>

HaikuDepot: Custom List Removal (Depots)

Remove use of custom list class where it is not
really required in the area of Depots. Also
convert the use of Depots to be wrapped in
BReference to match other model objects. At the
same time some data-loading logic has been
simplified.

Relates To #15534

Change-Id: Ie6fcc35f258a0c69c44990e4b09f6c32ec79945d
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3225
Reviewed-by: Rene Gollent <rene@gollent.com>

HaikuDepot: Icon / Tarball Handling

Instead of exploding the tar-ball of icons from
HDS, keep it as a tar-ball, index it and read
data directly out from specific indicies on demand.

This will speed up the process of downloading
the icons by removing the unpack. Also updates
will be faster by avoiding the need to delete the
old icon files.

Because icons are loaded on-demand, the start
time is faster by avoiding all the icon loads.
There are also savings on memory consumption.

Indexing on each load is surprisingly fast so
no external index is maintained. Likewise for
the tar-balls's meta-data.

This commit does not cover the implementation
of an LRU cache of the icons in memory.

Relates to #15370

Change-Id: Ia1647d8c805be89618f493d2592bf7877fca3f14
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3205
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

HaikuDepot: Languages List

Abstacts the list of supported languages into
the LanguageModel class preventing use of
List. Also; fix a few cases where newer
logging techniques may have caused incorrect
logic flow.

Relates To #15534

Change-Id: I144fe4788abdaf0d93e53eeabc97b3f7aa2ec710
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3085
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

HaikuDepot: Url -> Identifier

This change will rename the confusing "url" within
HaikuDepot to be "identifier" in line with
corresponding changes in pkg kit and HDS. Also at
the same time support is introduced for HDS
repos' meta-data to artificially match against
multiple repos; as requested for the future R1B3
release process. Some tidy-ups and extensions have
been made to the JSON schema-to-model and the
schema-to-parser scripts.

Change-Id: I402e7d610986039f58d72028bda7de977e9115e2
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2986
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

HaikuDepot: Performance

This change removes two mistakes I made a long time ago
that caused unnecessarily copying of lists of data. This
fix speeds up the UI alot.

This change also clears data in UI list elements when a
bulk load is requested. It stops clearing otherwise and
instead uses "add" and "remove" operations in the lists
which is OK now because the UI list elements are much
faster than they have been in the past. This removes
the strange clean-and-reload that was visible in the UI
previously.

A threaded package loading system was put in place a long
time ago, but with these performance improvements this
mechanism is no longer necessary; it has been removed to
simplify the code.

Fixes #16012

Change-Id: I393cee929695726539602b51630ae285fb8384f1
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2748
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

HaikuDepot: Fixes for Tabs

Cleans up the code around tab-switching and
also improves some logic around inserting
packages into the list of 'prominent'
packages by using a binary search.

If Haiku is installed into an environment with no
networking then it won't be able to talk to HDS
and so won't know which packages are promoted.
In this case switch the user to the "all packages"
tab so they are not shown a blank panel by default.

Relates to #14675, #14927

Change-Id: I14dd3be4af09a98245ddd0a9704bd8d53ed64a53
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2478
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

HaikuDepot: Check Working Files Writable

At start time, it is possible that the local disk
system is not able to be written to where the
application is trying to store working files. This
change introduces a check at startup that ensures
working files can be written to and warns the user
if this is not the case.

Change-Id: I907bf41a3b4eceb0083119a082fd5e68e4d534c1
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2397
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

HaikuDepot: Display User's Usage Conditions

A HaikuDepotServer may have previously agreed to
user usage conditions. If this is the case then
they are now able to view those conditions. There
were some problems with date formatting in this
change and so the date formatting logic has moved
into the HaikuDepot source code temporarily until
issues with the BDateTime class can be resolved;
most likely as a separate piece of work.

Relates to 15209

Change-Id: Ic3e5413d9139f410d7f7e8b566d4c56352dd2778
Reviewed-on: https://review.haiku-os.org/c/haiku/+/1870
Reviewed-by: Stephan Aßmus <superstippi@gmx.de>

HaikuDepot: Categories from Server

The list of categories for packages is currently
hard-coded into the HaikuDepot desktop
application. This change will change that so that
the list is obtained from the HaikuDepot Server
system and is always up to date with the server's
list of categories.

Change-Id: I757732f4d771e1599d6ad9c85cd65905640de928
Reviewed-on: https://review.haiku-os.org/c/1478
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

HaikuDepot: Load Languages from HDS

The HaikuDepot application has, thus far had its
own hard-coded list of languages that the user is
able to choose when (a) creating a new account or
(b) creating a user-rating. This change will mean
that those languages are loaded from the HDS
server dynamically and in this way the user can
choose from the full list. There have also been
improvements to the way in which the languages are
displayed in the menu as well.

Change-Id: If7cb7b87f348ca59d503d276a22444e72d0e6168
Reviewed-on: https://review.haiku-os.org/c/1425
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

HaikuDepot: Process and Data-loading Improvements

This change is a reshuffle of the backend processing involved in the
aquisition of data from servers including the pull-down and load of
HPKR data as well as the pull-down and load of data from the
HaikuDepotServer (HDS) system. The driver for this change is to
implement an initial implementation of a progress bar for the
loading of data as the application starts.

The following are notable changes;

* Removed some previously attempted 'functional style' logic in the
model which didn't fit well with C++
* Use of the base-url in the logical mapping from HDS and HD data is no
longer required and has been removed
* Some logging has been improved making it clearer which part of HD
is producing the logging which in turn helps with debugging issues
* List class has been modified to more cleanly support sorted lists
and binary searches; tests have also be updated accordingly
* Reorganise and tidy-up of the data-loading processes' structures
* The local repository update (HPKR) and data-load occur in background
processes now in the same system as the HDS data-load - this has been
crudely shifted from the MainWindow to new Processes and incorporated
into the background processing system
* The 'state-machine' background process runner is now replaced with a
'coordinator' style approach that can more easily handle the new
processes related to HPKR loading.
* Progress for loading processes is shown in the main window in the
WorkStatusView - this is flickering a bit, but basically works
* Added some documentation regarding how Processes work in the system
* The "Refresh Repositories" menu item now also updates data from HDS
* The "Refresh Repositories" menu item is disabled when the background
processes are running that update the repository data

Some further refinement would be good, but this change is large enough
for one round of improvements. There is an issue that the status bar
is used for screenshot display as well as this data-loading, but that
was the case before so it is something that can be dealt with later if
it is a problem.

Change-Id: I7668307645e3aabaf7e4a6e37e2cca80cc0f489e
Reviewed-on: https://review.haiku-os.org/770
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

HaikuDepot / PackageKit: Repositories 'Identifier' URL

Repositories are identified with a 'url' in the
remote 'repo.info' file. There is also a
'base url' which is the URL locally with which
the system is able to access the repository
data on. There is some confusion between these
two terms in the source. This change aims to
separate the two out and consistently name them.
The settings for the repository locally also was
not storing these values and that has been fixed.
Debug info about the repositories also did not
display the two urls consistently and will now
also do so. Finally, HaikuDepot now correlates
locally configured repositories with the data in
HaikuDepotServer using the identifier URL; this
makes the use of mirrors with HaikuDepot possible.

Fixes #13888
Change-Id: I66dfe589b05c24e1ab123a6945352e0f24b60bf1

HaikuDepot : Changelog, Remove old API and Version Bump

This change re-instates the change-log function (TRAC#13827) that was
dropped in earlier changes. Also now-disused API handling logic is
removed. A small code-style fix is made and the version of the
application has been incremented so that this version's logic can later
be detected by the HDS application server.

HaikuDepot: speed up package change handling

* During package population, when the repository is being
refreshed, thousands of package change messages arrive
in MainWindow while the PackageInfo fields are filled with
data.

Every such message caused a full generation of a package
list (including applying filters to all known packages),
which is quite slow, and caused the freezing from bug
#13823. The list was only used temporarily to figure out
if the package should be visible now or not.

* Instead of generating a new package list, we now only
match the single updated PackageInfo against the current
filters.

* Fixes #13823

* To reduce CPU usage even more, it would be better to not
generate at all, or not subscribe to, package changes while the
list is being initially populated, and/or to coalesce all the
changes to each package info into one update message (instead
of one for title, one for prominence, etc).
Deferring subscription requires some additional thought on when
to subscribe then, and how to avoid leaving a "hole" where
changes would go unnoticed.

HaikuDepot : More Backend Communications Improvements

* Further improves the logging and provides some
basic performance numbers.
* Moves the bulk-load logic out of the data-model
class.
* Introduces a state-machine for the bulk-load
process so that it will be more easily able to be
shifted to non-blocking IO when the HTTP libraries
can do that.
* Implements concurrent loading of the bulk-data to
hopefully improve lead time for icons and meta-data.
* Loads data to a temporary file and then moves to the
final location in order to avoid partially written
data in the cache.
* Handles situations where no network is available;
prevents attempt to access the network.
* Allows bulk-load processes to be cancelled when the
application quits.
* Introduces command-line arguments to help simulate
scenarios to help with testing performance and
network absence.
* Implements ordered insert and binary search in the
'List' class + basic unit test.

HaikuDepot: New Mechanic for Package Data Download

The application was previously making a number of different
requests to the application server in order to pull-down
meta-data (ratings, localization etc...) for packages. This
will now happen in one request per repo. The data is
also cached locally and freshness of the data is checked
back with the server before new data is pulled down.

HaikuDepot: Change communication mechanism with server for repos

(last commit with same title only included new files - added those now)

Previously the desktop application would make a number of JSON-RPC calls
over HTTP to get the repositories. Now it will make a single call to get
the repositories and cache the result. This uses standard HTTP cache
signalling techniques and allows the server-side the ability to cache
the generated data as well. Note that the model classes and parse-
related classes are generated and may not be code-style compliant. They
are generated from JSON schema files in the server-side project.
Information about this as well as the python files used to generate the
C++ classes and headers are included in the server-side project.

HaikuDepot: Improve icon download handling performance

Previously each icon would launch an independent HTTP request to
pull down the HVIF icon data. This change means that the data
will be pulled down in bulk across all packages as a .tgz and
will then be kept in a cache locally. The client-server logic
will use standard "If-Modified-Since" headers to check for
updates each time the HaikuDepot desktop application starts up.
This arrangement will bring down the HVIF as well as bitmap
icons and use the best representation it can.

Additionally, it is possible from a command-line option to log
HTTP traffic verbosely and it is also possible to use an "-h"
flag to display help on command-line arguments.

The code-structure around this change also anticipates some
future extensions to handle other client-server improvements.

Fixes #11804

HaikuDepot: Handling for communications with application server over multiple repositories

This change allows for the HaikuDepot desktop application to query the server application
over HTTP for data regarding packages across the various repositories that the have been
configured on the desktop environment. Previously it was 'hard-coded' to only
communicate about the HaikuPorts repository.

HaikuDepot: Implemented light-weight package manager update

No, not updating packages via HaikuDepot, but a light-weight synchronization
between the system package manager list and the HaikuDepot model. After
installing or removing a package, HaikuDepot performed a very costly update
of all information by basically throwing everything away and reloading it.
Implements #11786.

HaikuDepot: Update package info after rating package

Let Model::PopulatePackage() have a mode where it forces the update on a
package that has already been populated. Trigger populating from success
code path in RatePackageWindow. Fixes #11638.

HaikuDepot: Search all packages when showing featured

The intention was that typing search terms searches all packages, not only
the featured ones. This didn't actually work in Model::CreatePackageList().

HaikuDepot: Change when featured packages are shown

* Renamed the "Option" menu to "Show", removed the now redundant "Show" from
the items in that menu.
* Add an "Only featured packages" item in the Show menu. This item is disabled
as soon as the user types something in the search terms field. Othetwise,
this setting is respected, i.e. one can combine the featured setting with
other settings, i.e. show all featured packages of a certain depot and in a
certain category.
* Added PackageInfo::IsProminent() and removed the respective method from
the MainWindow (which only considered a package prominent if it was not
yet installed).
* "Show -> Installed packages" now defaults to on. There have been complaints
on Trac and I guess rightfully so. The use-cases in which one starts
HaikuDepot and wants to know about installed packages are just as likely
as the use-case where one wants to install new software.

HaikuDepot: Add some getters to Model.

For category, depot and search terms filter.

HaikuDepot: Enable caching of info retrieved from web-app

HaikuDepot: Added WebAppInterface getter to Model

HaikuDepot: Restore logged in user across sessions.

* Store the username in the app settings. The password was already stored
in the keyring. Restore the username upon next launch.
* Display the logged in user in the main window.
* Added "Log out" menu entry.
* When the password could not be retrieved from the keyring (also because
the user rejects the keyring dialog), unset the username.
* Allow unsetting the username by passing an empty name.

HaikuDepot: Put supported languages into a StringList

* Added TODO to retrieve the list from the web-app, there is already an
API for it. For now, the hard-coded list matches the web-app's and the
next change in this regard will be to support any known language.
* Make the list accessible.

HaikuDepot: Organized source into sub-folders

HaikuDepot: Implemented light-weight package manager update

No, not updating packages via HaikuDepot, but a light-weight synchronization
between the system package manager list and the HaikuDepot model. After
installing or removing a package, HaikuDepot performed a very costly update
of all information by basically throwing everything away and reloading it.
Implements #11786.

HaikuDepot: Update package info after rating package

Let Model::PopulatePackage() have a mode where it forces the update on a
package that has already been populated. Trigger populating from success
code path in RatePackageWindow. Fixes #11638.

HaikuDepot: Search all packages when showing featured

The intention was that typing search terms searches all packages, not only
the featured ones. This didn't actually work in Model::CreatePackageList().

HaikuDepot: Change when featured packages are shown

* Renamed the "Option" menu to "Show", removed the now redundant "Show" from
the items in that menu.
* Add an "Only featured packages" item in the Show menu. This item is disabled
as soon as the user types something in the search terms field. Othetwise,
this setting is respected, i.e. one can combine the featured setting with
other settings, i.e. show all featured packages of a certain depot and in a
certain category.
* Added PackageInfo::IsProminent() and removed the respective method from
the MainWindow (which only considered a package prominent if it was not
yet installed).
* "Show -> Installed packages" now defaults to on. There have been complaints
on Trac and I guess rightfully so. The use-cases in which one starts
HaikuDepot and wants to know about installed packages are just as likely
as the use-case where one wants to install new software.

HaikuDepot: Add some getters to Model.

For category, depot and search terms filter.

HaikuDepot: Enable caching of info retrieved from web-app

HaikuDepot: Added WebAppInterface getter to Model

HaikuDepot: Restore logged in user across sessions.

* Store the username in the app settings. The password was already stored
in the keyring. Restore the username upon next launch.
* Display the logged in user in the main window.
* Added "Log out" menu entry.
* When the password could not be retrieved from the keyring (also because
the user rejects the keyring dialog), unset the username.
* Allow unsetting the username by passing an empty name.

HaikuDepot: Put supported languages into a StringList

* Added TODO to retrieve the list from the web-app, there is already an
API for it. For now, the hard-coded list matches the web-app's and the
next change in this regard will be to support any known language.
* Make the list accessible.

HaikuDepot: Organized source into sub-folders