libnetservices: fix handling of HEAD requests and 204 responses

We were incorrectly reporting a B_IO_ERROR for these requests because we
could not read the content after the headers. There is no content in
these cases.

Add an unit test for both HEAD and 204 status, checking that there is no
content and the headers are correct.

Fixes #16885.

Change-Id: I98fefc5c604253bb2545b50395b7af9f8834def0
Reviewed-on: https://review.haiku-os.org/c/haiku/+/4142
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>
Reviewed-by: Niels Sascha Reedijk <niels.reedijk@gmail.com>

libbnetapi: BUrlRequest now outputs to BDataIO

Previously, BUrlRequest returns data received via a callback that can't
return any value. This approach have several issues:

- It's not possible to signify failures to the request.
- Users have to implement custom listeners just to handle the common
case of outputting to a buffer/file/etc.
- The received data has to be serialized into BMessage when
BUrlProtocolDispatchingListener is employed. This can cause a
noticible slowdown in real-world scenarios as evident by #10748.

With this change, BUrlRequest will output directly into a BDataIO, which
exposes a richer API for request handlers to work with (for example a
BitTorrent client can request a BPositionIO for non-linear data
delivery), as well as simplifying common cases for users.

The adaptation only requires one additional API:
BHttpRequest::SetStopOnError(). This API simply instructs the HTTP
request handler to cancel the request if an HTTP error is occurred.

Change-Id: I4160884d77bff0e7678e0a623e2587987704443a
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3084
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

libbnetservices: BUrlResult is no longer a BArchivable

The switch to make BUrlResult serializable was debuted in
f9e1854f198d4200f21a9cbe29fdfb0fabbe192f with the rationale is that
BHttpRequest auto-redirection might cause the headers to become
obsolete by the time a client process the BMessage received from
BUrlProtocolDispatchingListener.

With the change to BHttpRequest to not notify listeners when
auto-redirection is enabled, this is no longer the case and the
serialization code can go away now. This simplifies BUrlResult and its
subclasses, and gain us some performance for clients using
BUrlProtocolDispatchingListener as the result object no longer has to be
serialized.

This also change the ABI of BUrlProtocolListener::HeadersReceived to no
longer passing a BUrlResult.

Additionally, BUrlResult and BHttpResult now express the size of the content
as an off_t, thus allowing results larger than 4 GB.

Change-Id: I9dd29a8b26fdd9aa8e5bbad8d1728084f136312d
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3082
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

HttpRequest: Don't trigger listener on auto redirect

When the user enable auto-redirection, what they meant is that they do
not want to handle redirections, thus we should take total control of this
step and hide it from the user. In fact, a majority of in-tree users
write code to disable their listener when a redirection happen.

This should also allow us to simplify BUrlResult, which has been turned
into a BArchivable for the sole reason of "preserving" headers when
auto redirect is enabled when used with BUrlDispatchingListener, which
has been shown to have little (if any) practical usage.

Change-Id: I9b10b81de0a13edbaec25f6b48ed7a4335ea691a
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3081
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

libnetservices.a: fix unit test builds after hrev54923

libbnetapi: Disallow instantiation of BUrlRequest subclasses directly

This API change forces all creation of BUrlRequest to be done via
BUrlProtocolRoster::MakeRequest(). This allows the structure of protocol
addons to be altered without breaking ABI for client applications.

Change-Id: I1785c9136c50d19eaa9e57cb9d259ed8d88a5b56
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3080
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

tests/HttpTest: Fix build on x86_gcc2

* std::istreambuf_iterator<T> template isn't available until C++11.
* std::vector<T>::cbegin() is not available
* Add missing include of errno.h

Change-Id: Ice344f6b0f93bf72d9120674607878c4c3e8ef54
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2515
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

tests/net: HTTP proxy client test

With this patch, ProxyTest is implemented and all of the tests in
HttpTest are enabled.

Adding a transparent proxy server implementation proxy.py. Like
testserver.py, this can be provided a socket file descriptor and port
via command-line arguments.

TestServer was refactored to extract ChildProcess and
RandomTCPServerPort, which are now also used by TestProxyServer.

ProxyTest starts TestProxyServer and validates that the request is
sent to the proxy and is routed to the appropriate endpoint of the
downstream server.

The template which adds common tests between HttpTest and HttpsTest
was changed slightly to just take a BThreadedTestCaller<T>&, which
made it simpler to add additional test cases to one suite which are
not appropriate to the other. There wasn't much point in keeping that
template as a member function so I moved it into HttpTest.cpp as a
free function template.

Change-Id: Ied32d6e10bb195d111cae7bbcf0e93168118088b
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2291
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

tests/net: Implement testserver.py TLS for HttpsTests

This patch is a followup to 0dde5052b which added testserver.py, a
HTTP echo server for the HttpTests and HttpsTests in the ServicesKit
test suite. This patch implements `testserver.py --use-tls` which
allows for re-enabling HttpsTests.

If `--use-tls` is used, then a self-signed TLS certificate is
generated in a temporary directory which is used by the test
server. This option is used when running HttpsTests.

There doesn't seem to be a good way to have these tests trust the
certificate generated by this test at the moment. Until that API
exists I've just made these tests ignore certificate validation. We'll
want to resolve this and update these tests to actually verify that
validation works as expected.

Some minor tweaks had to be made to testserver.py to take care of
differences in the response body when serving HTTP and HTTPS requests.

Some additional changes:
* Don't depend on any files outside of src/tests/kits/net/service for
these tests. UploadTest was uploading a file from /boot, but I
copied it into the test source directory to avoid having these tests
break if someone makes an unrelated change. It doesn't really matter
what the contents of this file is as long as it doesn't change.
* Use BThreadedTestCase. This speeds up the tests considerably, mostly
because it means that the different test cases can share the same
HttpTest instance, which means there is only a single TestServer
instance, and it takes around half a second to bootstrap the test
server on my system, and even longer if --use-tls is used.

Change-Id: I6d93d390ebd56115365a85109140d175085e1f01
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2260
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

tests/net: Working integration tests for HTTP client

This patch is part 1 of 3 with the goal of having a working
integration test harness for BHttpRequest. In this patch the existing
test cases were expanded and fixed for HTTP. In followup patches the
test harness will be updated to support HTTPS and reverse proxies.

Before this patch the tests for BHttpRequest had hard dependencies on
the external services httpbin.org and portquiz.net. These tests
eventually stopped working because the owner of those services made
changes, causing the assertions in these tests to fail.

The goal of these patches is to make a test harness that allows for
the same kinds of end-to-end integration tests but without any
external dependencies.

The test suite now includes a Python script called testserver.py which
is a HTTP echo server of sorts. When it receives a request, it will
echo the request headers and request body back to the client as a
text/plain response body.

The TestServer class manages the lifecycle of this testserver.py
process. Each test case calls Start() on the server to start a new
instance, and then it is shut down when the destructor is called. On
each invocation a random port is assigned by the kernel in TestServer,
and that socket file descriptor is provided to the child testserver.py
script.

Authorization tests are supported, currently implementing Basic and
Digest auth. If the test server receives a request for a path
/auth/<auth-scheme>/<expected-username>/<expected-password>, then the
appropriate authorization scheme will be employed. For example, if
/auth/basic/foo/bar is used as the path, then the server will expect
the Authorization header to contain an appropriate Basic auth
payload.

The tests now perform a bit more validation than before, validating
the expected HTTP headers and response body is returned from the
server.

The following tests are not fixed yet or were removed:
* PortTest was removed entirely since I'm not sure of the point of this
test, and that functionality seems to be covered by the existing tests
anyway.
* HTTPS tests are not functional yet, but will be in a followup
patch. THis requires updating testserver.py to generate a
self-signed TLS cert if --use-tls is provided.
* ProxyTest was disabled before this patch, but can be enabled in a
followup patch by providing a reverse proxy in the test harness.

Change-Id: Ia201ef4583b7636c61e77072a03db936cb0092be
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2243
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

Revive unit tests

* Fix build on 64-bit.
* Test suite can now run from start to finish.
* Changes in OutlineListView test need an explanation:
for some reason when BApplication is created on heap, be_app is not
reset. This causes other tests to crash due to second BApplication
object being created.

Change-Id: Ic7d55f4622192f83572bfd83fa37f1f5fd641e22
Reviewed-on: https://review.haiku-os.org/465
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

Add HTTP proxy support.

* Move default context management to BUrlRequest since some code
(including the testsuite) bypass the BUrlProtocolRoster.
* Introduce proxy host and port in BUrlContext
* Have BHttpRequest use the proxy when making requests

HttpTest: use BReferenceable APIs for the BUrlContext.

Test for the "Host" header containing the port.

* Also tests that receiving data using a listener works.

Make HttpRequest tests use the test framework

* Use httpbin.org as a server, rather than a local machine.
* Also add an HTTPS version

Add HTTP proxy support.

* Move default context management to BUrlRequest since some code
(including the testsuite) bypass the BUrlProtocolRoster.
* Introduce proxy host and port in BUrlContext
* Have BHttpRequest use the proxy when making requests

HttpTest: use BReferenceable APIs for the BUrlContext.

Test for the "Host" header containing the port.

* Also tests that receiving data using a listener works.

Make HttpRequest tests use the test framework

* Use httpbin.org as a server, rather than a local machine.
* Also add an HTTPS version