libnetservices: fix handling of HEAD requests and 204 responses

We were incorrectly reporting a B_IO_ERROR for these requests because we
could not read the content after the headers. There is no content in
these cases.

Add an unit test for both HEAD and 204 status, checking that there is no
content and the headers are correct.

Fixes #16885.

Change-Id: I98fefc5c604253bb2545b50395b7af9f8834def0
Reviewed-on: https://review.haiku-os.org/c/haiku/+/4142
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>
Reviewed-by: Niels Sascha Reedijk <niels.reedijk@gmail.com>

HttpRequest: Don't trigger listener on auto redirect

When the user enable auto-redirection, what they meant is that they do
not want to handle redirections, thus we should take total control of this
step and hide it from the user. In fact, a majority of in-tree users
write code to disable their listener when a redirection happen.

This should also allow us to simplify BUrlResult, which has been turned
into a BArchivable for the sole reason of "preserving" headers when
auto redirect is enabled when used with BUrlDispatchingListener, which
has been shown to have little (if any) practical usage.

Change-Id: I9b10b81de0a13edbaec25f6b48ed7a4335ea691a
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3081
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

tests/net: HTTP proxy client test

With this patch, ProxyTest is implemented and all of the tests in
HttpTest are enabled.

Adding a transparent proxy server implementation proxy.py. Like
testserver.py, this can be provided a socket file descriptor and port
via command-line arguments.

TestServer was refactored to extract ChildProcess and
RandomTCPServerPort, which are now also used by TestProxyServer.

ProxyTest starts TestProxyServer and validates that the request is
sent to the proxy and is routed to the appropriate endpoint of the
downstream server.

The template which adds common tests between HttpTest and HttpsTest
was changed slightly to just take a BThreadedTestCaller<T>&, which
made it simpler to add additional test cases to one suite which are
not appropriate to the other. There wasn't much point in keeping that
template as a member function so I moved it into HttpTest.cpp as a
free function template.

Change-Id: Ied32d6e10bb195d111cae7bbcf0e93168118088b
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2291
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

tests/net: Implement testserver.py TLS for HttpsTests

This patch is a followup to 0dde5052b which added testserver.py, a
HTTP echo server for the HttpTests and HttpsTests in the ServicesKit
test suite. This patch implements `testserver.py --use-tls` which
allows for re-enabling HttpsTests.

If `--use-tls` is used, then a self-signed TLS certificate is
generated in a temporary directory which is used by the test
server. This option is used when running HttpsTests.

There doesn't seem to be a good way to have these tests trust the
certificate generated by this test at the moment. Until that API
exists I've just made these tests ignore certificate validation. We'll
want to resolve this and update these tests to actually verify that
validation works as expected.

Some minor tweaks had to be made to testserver.py to take care of
differences in the response body when serving HTTP and HTTPS requests.

Some additional changes:
* Don't depend on any files outside of src/tests/kits/net/service for
these tests. UploadTest was uploading a file from /boot, but I
copied it into the test source directory to avoid having these tests
break if someone makes an unrelated change. It doesn't really matter
what the contents of this file is as long as it doesn't change.
* Use BThreadedTestCase. This speeds up the tests considerably, mostly
because it means that the different test cases can share the same
HttpTest instance, which means there is only a single TestServer
instance, and it takes around half a second to bootstrap the test
server on my system, and even longer if --use-tls is used.

Change-Id: I6d93d390ebd56115365a85109140d175085e1f01
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2260
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

tests/net: Working integration tests for HTTP client

This patch is part 1 of 3 with the goal of having a working
integration test harness for BHttpRequest. In this patch the existing
test cases were expanded and fixed for HTTP. In followup patches the
test harness will be updated to support HTTPS and reverse proxies.

Before this patch the tests for BHttpRequest had hard dependencies on
the external services httpbin.org and portquiz.net. These tests
eventually stopped working because the owner of those services made
changes, causing the assertions in these tests to fail.

The goal of these patches is to make a test harness that allows for
the same kinds of end-to-end integration tests but without any
external dependencies.

The test suite now includes a Python script called testserver.py which
is a HTTP echo server of sorts. When it receives a request, it will
echo the request headers and request body back to the client as a
text/plain response body.

The TestServer class manages the lifecycle of this testserver.py
process. Each test case calls Start() on the server to start a new
instance, and then it is shut down when the destructor is called. On
each invocation a random port is assigned by the kernel in TestServer,
and that socket file descriptor is provided to the child testserver.py
script.

Authorization tests are supported, currently implementing Basic and
Digest auth. If the test server receives a request for a path
/auth/<auth-scheme>/<expected-username>/<expected-password>, then the
appropriate authorization scheme will be employed. For example, if
/auth/basic/foo/bar is used as the path, then the server will expect
the Authorization header to contain an appropriate Basic auth
payload.

The tests now perform a bit more validation than before, validating
the expected HTTP headers and response body is returned from the
server.

The following tests are not fixed yet or were removed:
* PortTest was removed entirely since I'm not sure of the point of this
test, and that functionality seems to be covered by the existing tests
anyway.
* HTTPS tests are not functional yet, but will be in a followup
patch. THis requires updating testserver.py to generate a
self-signed TLS cert if --use-tls is provided.
* ProxyTest was disabled before this patch, but can be enabled in a
followup patch by providing a reverse proxy in the test harness.

Change-Id: Ia201ef4583b7636c61e77072a03db936cb0092be
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2243
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>