HaikuDepot: Buffered Reads for JSON Parse

Reduce the number of small reads into a file
for JSON parse by buffering the data.

Change-Id: Iae8030a72daec358968cd2edf5f95510194a2f68
Reviewed-on: https://review.haiku-os.org/c/haiku/+/7096
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>
Reviewed-by: Adrien Destugues <pulkomandy@pulkomandy.tk>

HaikuDepot: Avoid Corrupt Username (2)

Previous changes removed the ability to supply
control characters at the start and end of a
nickname; this change removes them from anywhere
in the nickname when supplied.

Change-Id: I0631fffa8aaf2c0e267c777892044c53faa4334e
Reviewed-on: https://review.haiku-os.org/c/haiku/+/7071
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>
Reviewed-by: Jérôme Duval <jerome.duval@gmail.com>

Replace usages of B_FILE_NOT_FOUND in non-BeOS-API code.

B_FILE_NOT_FOUND was deprecated in BeOS R5 in favor of B_ENTRY_NOT_FOUND,
but it remained in Haiku and was never removed even conditionally, so
we have accumulated a number of usages of it.

This commit changes all the usages of it in new code, applications,
or anything else that BeOS applications will otherwise never see,
and so should be relatively safe.

libbnetapi: BUrlRequest now outputs to BDataIO

Previously, BUrlRequest returns data received via a callback that can't
return any value. This approach have several issues:

- It's not possible to signify failures to the request.
- Users have to implement custom listeners just to handle the common
case of outputting to a buffer/file/etc.
- The received data has to be serialized into BMessage when
BUrlProtocolDispatchingListener is employed. This can cause a
noticible slowdown in real-world scenarios as evident by #10748.

With this change, BUrlRequest will output directly into a BDataIO, which
exposes a richer API for request handlers to work with (for example a
BitTorrent client can request a BPositionIO for non-linear data
delivery), as well as simplifying common cases for users.

The adaptation only requires one additional API:
BHttpRequest::SetStopOnError(). This API simply instructs the HTTP
request handler to cancel the request if an HTTP error is occurred.

Change-Id: I4160884d77bff0e7678e0a623e2587987704443a
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3084
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

haikudepot: switch to libnetservices.a

Switch to the experimental services API as the copy in libbnetapi.so is
deprecated.

Change-Id: Ic5e2c23a1be598be3057ea913c420c5346f9007f
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3669
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

libbnetapi: Disallow instantiation of BUrlRequest subclasses directly

This API change forces all creation of BUrlRequest to be done via
BUrlProtocolRoster::MakeRequest(). This allows the structure of protocol
addons to be altered without breaking ABI for client applications.

Change-Id: I1785c9136c50d19eaa9e57cb9d259ed8d88a5b56
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3080
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

HaikuDepot: Icon / Tarball Handling

Instead of exploding the tar-ball of icons from
HDS, keep it as a tar-ball, index it and read
data directly out from specific indicies on demand.

This will speed up the process of downloading
the icons by removing the unpack. Also updates
will be faster by avoiding the need to delete the
old icon files.

Because icons are loaded on-demand, the start
time is faster by avoiding all the icon loads.
There are also savings on memory consumption.

Indexing on each load is surprisingly fast so
no external index is maintained. Likewise for
the tar-balls's meta-data.

This commit does not cover the implementation
of an LRU cache of the icons in memory.

Relates to #15370

Change-Id: Ia1647d8c805be89618f493d2592bf7877fca3f14
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3205
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

HaikuDepot: Languages List

Abstacts the list of supported languages into
the LanguageModel class preventing use of
List. Also; fix a few cases where newer
logging techniques may have caused incorrect
logic flow.

Relates To #15534

Change-Id: I144fe4788abdaf0d93e53eeabc97b3f7aa2ec710
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3085
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

HaikuDepot: Better Logging

Changes the logging in HD to use a macro for the
various log levels to declutter the code and to
make it easier to more systematically log.

Change-Id: I025970707a0a11e7e3aaa4b52fc91288af0183f5
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3018
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

HaikuDepot: Process and Data-loading Improvements

This change is a reshuffle of the backend processing involved in the
aquisition of data from servers including the pull-down and load of
HPKR data as well as the pull-down and load of data from the
HaikuDepotServer (HDS) system. The driver for this change is to
implement an initial implementation of a progress bar for the
loading of data as the application starts.

The following are notable changes;

* Removed some previously attempted 'functional style' logic in the
model which didn't fit well with C++
* Use of the base-url in the logical mapping from HDS and HD data is no
longer required and has been removed
* Some logging has been improved making it clearer which part of HD
is producing the logging which in turn helps with debugging issues
* List class has been modified to more cleanly support sorted lists
and binary searches; tests have also be updated accordingly
* Reorganise and tidy-up of the data-loading processes' structures
* The local repository update (HPKR) and data-load occur in background
processes now in the same system as the HDS data-load - this has been
crudely shifted from the MainWindow to new Processes and incorporated
into the background processing system
* The 'state-machine' background process runner is now replaced with a
'coordinator' style approach that can more easily handle the new
processes related to HPKR loading.
* Progress for loading processes is shown in the main window in the
WorkStatusView - this is flickering a bit, but basically works
* Added some documentation regarding how Processes work in the system
* The "Refresh Repositories" menu item now also updates data from HDS
* The "Refresh Repositories" menu item is disabled when the background
processes are running that update the repository data

Some further refinement would be good, but this change is large enough
for one round of improvements. There is an issue that the status bar
is used for screenshot display as well as this data-loading, but that
was the case before so it is something that can be dealt with later if
it is a problem.

Change-Id: I7668307645e3aabaf7e4a6e37e2cca80cc0f489e
Reviewed-on: https://review.haiku-os.org/770
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

HaikuDepot : 'Old Client' Handling

The server side has the ability to reject client
requests where the client is no longer supported
because it is presumably too old. This change
will inform the user when this happens and will
prevent the client from attempting further server
communications within this execution of the
application.

HaikuDepot : More Backend Communications Improvements

Fixes for x86_64 build.

HaikuDepot : More Backend Communications Improvements

* Further improves the logging and provides some
basic performance numbers.
* Moves the bulk-load logic out of the data-model
class.
* Introduces a state-machine for the bulk-load
process so that it will be more easily able to be
shifted to non-blocking IO when the HTTP libraries
can do that.
* Implements concurrent loading of the bulk-data to
hopefully improve lead time for icons and meta-data.
* Loads data to a temporary file and then moves to the
final location in order to avoid partially written
data in the cache.
* Handles situations where no network is available;
prevents attempt to access the network.
* Allows bulk-load processes to be cancelled when the
application quits.
* Introduces command-line arguments to help simulate
scenarios to help with testing performance and
network absence.
* Implements ordered insert and binary search in the
'List' class + basic unit test.

HaikuDepot: Handling Corrupted Download Data

Damage in latter portions of the downloaded package data can cause the
meta-data for the caching to load OK, but the actual data is unable to
be parsed. In this case, the application can fail to refresh the
broken data. This change will mean that the data is "moved aside" into
another file so that on the next run, the application will load fresh
data in place.

HaikuDepot: Improved logging control

The application had previously no means to control the verbosity of
logging. This excessive logging meant that it was bit hard to
understand what is going on. Now it is possible to define the level
of logging that is output; especially as the application pulls-down
data from the remote server system.

HaikuDepot: Change communication mechanism with server for repos

(last commit with same title only included new files - added those now)

Previously the desktop application would make a number of JSON-RPC calls
over HTTP to get the repositories. Now it will make a single call to get
the repositories and cache the result. This uses standard HTTP cache
signalling techniques and allows the server-side the ability to cache
the generated data as well. Note that the model classes and parse-
related classes are generated and may not be code-style compliant. They
are generated from JSON schema files in the server-side project.
Information about this as well as the python files used to generate the
C++ classes and headers are included in the server-side project.

HaikuDepot: Improve icon download handling performance

Previously each icon would launch an independent HTTP request to
pull down the HVIF icon data. This change means that the data
will be pulled down in bulk across all packages as a .tgz and
will then be kept in a cache locally. The client-server logic
will use standard "If-Modified-Since" headers to check for
updates each time the HaikuDepot desktop application starts up.
This arrangement will bring down the HVIF as well as bitmap
icons and use the best representation it can.

Additionally, it is possible from a command-line option to log
HTTP traffic verbosely and it is also possible to use an "-h"
flag to display help on command-line arguments.

The code-structure around this change also anticipates some
future extensions to handle other client-server improvements.

Fixes #11804