Avoid a NULL dereference when handling a malformed fastcgi request.

Rework the hack to avoid a use-after-free in the fastcgi code.
Since server_fcgi() can be called by server_read_httpcontent() we
can't set clt_fcgi_error to NULL. Instead, we implement a simple
reference count to track when a fastcgi session is in progress to
avoid closing the http session prematurely on fastcgi error.
Based on a diff from and OK by tb@. Reported by Ben Kallus.

Use EVBUFFER_DATA instead of reaching into struct evbuffer. ok tb

Work around use after free in httpd(8)

A malformed HTTP request can cause httpd in fastcgi mode to crash due to a
use-after-free. This is an awful hack, but it's good enough until someone
figures out the correct way of dealing with server_close() here.

"this will do the trick for now" claudio
ok beck deraadt

branches: 1.126.6; 1.126.10;
Remove unneeded calls to tls_init(3)

As per the manual and lib/libtls/tls.c revision 1.79 from 2018
"Automatically handle library initialisation for libtls." initialisation
is handled automatically by other tls_*(3) functions.

Remove explicit tls_init() calls from base to not give the impression of
it being needed.

Feedback tb
OK Tests mestre

Do not compare TLS config params for non-TLS servers. This allows to
mix 'listen * port 80' and 'listen * tls port 443' in one server block.
Also the last argument of server_tls_cmp - match_keypair - is always 0
so remove this code.
OK florian@ tb@ some long time ago

Pull tls_close() and tls_free() further up, so tls_free() will already
be in the right spot once tls_close() is handled by libevent.

suggested by jsing

Call tls_close() before closing the underlying socket

In order to end a TLS connection regularly, an implementation MUST send a
close_notify alert. libtls does this in tls_close() via SSL_shutdown(),
so the socket had better still be open.

The incorrect order in server_close() caused a leak on each tls connection
due to a bug in libssl (fixed in tls_record_layer.c r1.56).

As pointed out by claudio, tls_close() should really be handled from the
main event loop. This will be addressed in a later commit.

ok claudio florian jsing

Don't leak the log message in server_sendlog

While there, use the length calculated by vasprintf() instead of using
strlen needlessly.

ok claudio florian

Handle absence of TLS certs while parsing the config

There is a soft fail mechanism to handle missing certs for seamless
interaction with acme-client. Move this to the config parser. This is
simpler than server.c r1.117 and avoids a crash due to listening on
port 443 without having set up the TLS context first. More precisely,
the crash happens if a server with missing certificate is visited via
https in a configuration where there is a second server with valid
certificate and key.

From Joshua Sing (joshua at hypera dot dev)

ok benno

httpd(8) sent a 408 response every time a connection request timeout
was reached. This is not what other servers are doing, it leads to
ugly log messages and might confuse some clients.
benno@ analyzed that the correct behavior is (probably) to send a 408
when we are in the middle of receiving headers and time out there and
just close the connection in all other cases.
In particular, if a connection gets opened and no request is received
at all just close the connection. If a connection is set to keep-alive
and a request was handled and no further request is coming in just
close the connection. The later is the usual cause for spurious log
messages and client confusion.

Reported over the years by many.
Input, explanations and OK benno

When system calls indicate an error they return -1, not some arbitrary
value < 0. errno is only updated in this case. Change all (most?)
callers of syscalls to follow this better, and let's see if this strictness
helps us in the future.

httpd(8): add support for setting custom FastCGI parameters.

This commit extends the existing grammar by adding the param option
to the fastcgi directive: fastcgi param name value.

Example usage:
fastcgi param VAR1 hello
fastcgi param VAR2 world

With help and OK florian@
Rogue manpage bits, feel free to modify them.

Allow httpd(8) to start when TLS is configured but a cert is not yet
available.
Assuming a httpd.conf based on /etc/examples/httpd.conf, httpd(8)
will only listen on port 80 and serve the acme-challenge directory
for acme-client(1).
The workflow to get a certificate then becomes
acme-client -vAD example.com && rcctl reload httpd
Without the need to edit the httpd.conf yet again. Once the cert
is in place and httpd is reloaded it starts to serve on port 443.

Idea, tweaks & OK deraadt, OK benno

Backout my previous commit:

date: 2018/10/01 19:24:09; author: benno; state: Exp; lines: +7 -1;
commitid: 0O8fyHPNvPd8rvYU;
Only send 408 Timeout responses when we have seen at least part of a
request. Without a request, just close the connection when we hit
request timeout.
Prompted by a bug report from Nikola Kolev, thanks.
ok reyk@ and some suggestions from claudio@ and bluhm@

Mark Patruck (mark AT wrapped DOT cx) found a problem with it, thanks
for the report.

ok reyk@ bluhm@ sthen@ deraadt@

Only send 408 Timeout responses when we have seen at least part of a
request. Without a request, just close the connection when we hit
request timeout.
Prompted by a bug report from Nikola Kolev, thanks.
ok reyk@ and some suggestions from claudio@ and bluhm@

Add support for client certificate authentication to httpd.

From Jack Burton <jack at saosce dot com dot au> - thanks!

Also tested by Jan Klemkow <j.klemkow at wemelug dot de>.

ok beck@ reyk@

Don't do OCSP stapling only if the staple file is 0 length.

This allows something external (like ocspcheck) to disable the stapling
deliberatly if it can not retreive a valid staple by truncating the
staple file to indicate "do not provide a staple", while the file not
existin will still be treated as a configuration error
ok claudio@ florian@, and prompted by @jsing

Disable oscp stapling on invalid staple, rather than failing to start.
ok claudio@ florian@

Convert httpd to tls_config_set_ecdhecurves(), allowing a list of curves
to be specified, rather than a single curve.

ok beck@

Rework the way that TLS configuration is sent/received via imsgs, so that
are no longer limited by the 16KB maximum size of a single imsg.
Configuration data that is larger than a single message is now chunked and
sent via multiple imsgs.

Prompted by a diff from Jack Burton <jack at saosce dot com dot au>.

ok reyk@

some freezero() calls

Implement TLS ticket support in httpd. Off by default. Use
tls ticket lifetime default
to turn it on with a 2h ticket lifetime.
Rekeying happens after a quarter of that time.
OK reky@ and bob@

/tmp/cvsspEkok

Fix support for HTTP pipelining by handling all requests in the buffer.

Tested & OK jung@

remove extra call setting OCSP staple now that it is done above
using keypair_ocsp..
ok reyk@

Correct mistake I made when converting this to new funciton

Add tls_config_[add|set]keypair_ocsp functions so that ocsp staples may be
added associated to a keypair used for SNI, and are usable for more than
just the "main" certificate. Modify httpd to use this.
Bump libtls minor.

ok jsing@

Do not set EVBUFFER_EOF on read/write errors and handle EOF correctly.

Either libevent or the TLS callback can trigger an EOF when the
connection is closed.

OK sunil@ jung@ benno@

Stop accessing verbose and debug variables from log.c directly.

This replaces log_verbose() and "extern int verbose" with the two functions
log_setverbose() and log_getverbose().

Pointed out by benno@
OK krw@ eric@ gilles@ (OK gilles@ for the snmpd bits as well)

Check the return value of tls_config_set_protocols(), now that it returns
an int.

Move OCSP loading into a separate function - it is not part of the keypair
and this way we can give a separate specific error message.

ok beck@ reyk@

Fix tcp ip ttl / minttl on IPv6 sockets.

ok florian@

conditionalize ocsp load properly
ok jsing@

Add OCSP stapling support to httpd
ok jsing@ bcook@

Kill (remove) the ps_pid from privsep struct since it is not being used
anymore. Also fix the process initialization prototypes.

ok reyk@

Kill p_instance from proc.c and remove static proc_id unused variables.

To keep the debug functionality intact and correct we'll use the pid
field in the imsg header to pass the instance number. Remember to always
pass 'ps_instance + 1' otherwise libutil will fill imsg header pid field
with the imsgbuf pid (which is the current process pid).

ok reyk@

Replace the static env variables with a single global variable.

ok reyk@

Enable SNI support in httpd(8).

ok reyk@

stop including sys/param.h for nitems. define locally as needed.
ok natano reyk

Turn "TLS handshake failed -" log message into a debug message - it
happens way too often and does not provide much information.

OK jung@

Rename server_handshake_tls() to server_tls_handshake() to align with
the other server_tls_* functions (and I like the prefix notation
better). No functional change.

Move server_match() from parse.y to server.c; use env instead of conf,
which is actually the same thing (cluebat from reyk@).

Use lowercase 'tls' in debug and log messages for consistency.

Requested by reyk@

Make httpd stricter with respect to TLS configuration - in particular, do
not allow TLS and non-TLS to be configured on the same port, do not allow
TLS options to be specified without a TLS listener and ensure that the TLS
options are the same when a server is specified on the same address/port.
Currently, these configurations are permitted but do not work as intended.

Also factor out and reuse the server matching code, which was previously
duplicated.

ok reyk@

Include the TLS configuration errors in log messages. Also set the
certificate and private key at the same time.

Use log_warnx() instead of log_warn() when the failure will not have
resulted in errno being set.

ok reyk@

sync with relayd, use proc_compose()

Retire socket_set_blockmode() in favor of the SOCK_NONBLOCK type flag.
As done in iked and snmpd.

OK jung@

pledge(2) for httpd.

1) The main process listens on sockets and accepts connections. It
creates and opens log files, creates and kills child processes. On
start up and on receiving a HUP signal it parses the configuration. It
passes on file descriptors for logging or requests to it's children.
2) The logger process writes log messages to a file descriptor passed
in from the main process.
3) The server process reads the request from a file descriptor passed
in from the main process. It reads a file or creates a directory index
to send a response.
Additionally this process handles fastcgi requests. It connects to
AF_UNIX, AF_INET or AF_INET6 sockets. A re-factoring might make it
possible to drop the additional fastcgi privileges when only static
files are served.

with deraadt@ some time ago
prodding & OK deraadt@
tweaks and OK reyk@

Fix server_handshake_tls() - we should only call server_input() in the case
where the handshake has successfully completed.

ok beck@

fix return type for tls_read/write
jointly with jsing@

fix after libtls api changes
ok jsing@

Update httpd to call tls_handshake() after tls_accept_socket().

ok beck@

Fix a regression that was introduced with server.c r1.64: Do NOT free
srv_conf->auth in serverconfig_free() because it was not allocated in
config_getserver() but assigned as a reference by id from a global
list that is maintained independently. This fixes a potential
double-free. This fix also makes srv_conf->auth "const" to emphasize
that the read-only auth pointer was not allocated here.

OK jsing@

Change httpd(8) to use C99-style fixed-width integers (uintN_t instead
of u_intN_t) and replace u_int with unsigned int. Mixing both
variants is a bad style and most contributors seem to prefer this
style; it also helps us to get used to it, portability, and
standardization.

Theoretically no binary change, except one in practice: httpd.o has a
different checksum because gcc with -O2 pads/optimizes "struct
privsep" differently when using "unsigned int" instead "u_int" for the
affected members. "u_int" is just a typedef of "unsigned int", -O0
doesn't build the difference and clang with -O2 doesn't do it either -
it is just another curiosity from gcc-land.

OK semarie@

Fix rev 1.70 of server.c by only re-enabling the bufferevent if we
previously disabled it because we were reading to fast (from disk).
Problem noted and tracked down to that commit by weerd@ and
independently by stsp@.
Tested by weerd@, stsp@, reyk@
OK bluhm@, reyk@

backout the previous: it broke wordpress somehow.
we need more care to find a proper fix for the fastcgi headers.

acknowledged by deraadt@

Read fcgi response records until we have the whole http header and can
parse it. Otherwise http headers can leak into the body.
Pointed out by Jean-Philippe Ouellet on bugs@ Thanks!
OK reyk, commit ASAP deraadt@

libtls has been changed to set SSL_MODE_ENABLE_PARTIAL_WRITE and
SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER by default. This gives
tls_write() a similar short write semantics as write(2) and a
workaround in httpd to cope with the previous differences can be
removed. Specifically, httpd can stop copying data into a local
buffer that was used to keep it around for repeated writes.

OK bluhm@

If we can read faster from disk than send data to the client stop
reading from disk when we hold a certain amount of data in
RAM. Re-enable reading once we send enough data to the
client. Otherwise we might end up with the whole file (which can be
huge) in RAM.
Reported by Matthew Martin ( matt.a.martin AT gmail ) on bugs@,
thanks!
OK reyk@, benno@

Escape the message in server_log() as well.

OK benno@

Close connections that fail to complete a TLS handshake.

Based on a diff from Jack Burton <jack at saosce dot com dot au>.

ok reyk@

Fix typo in comment.

Send the TLS certificate and key via separate imsgs, rather than
including them in the IMSG_CFG_SERVER imsg. This allows the certificate
and key to each be almost 16KB (the maximum size for an imsg), rather than
having a combined total of less than 16KB (which can be reached with large
keys, certificate bundles or by including text versions of certificates).

ok reyk@

Explicitly check for and handle EOF on a TLS connection.

ok reyk@

Fix memory leaks that can occur when config_getserver() fails.

config.c r1.34 and r1.30 introduced potential memory leaks for auth and
return_uri when config_getserver fails. Fix this by switching to
serverconfig_free() and adding the missing free for srv_conf->auth.
While here, make serverconfig_free() a little more bulletproof by
explicit_bzero()ing key material.

ok reyk@

We cannot log errors with server_close() before allocating
clt_log evbuffer.
server_close() calls server_log() which uses ctl_log.
Crash reported by Daniel Jakots <vigdis AT chown DOT me>, thanks!
OK benno

Always check the return value of proc_composev_imsg() and handle failures
appropriately. Otherwise imsg construction can silently fail, resulting in
non-obvious problems.

Found the hard way by Theodore Wynnychenko.

ok doug@ florian@

Prevent use after free.
While here unconditionally free clt and move declaration of
server_inflight_dec() into server.c
Found while investigating if (foo != NULL) free(foo) patterns pointed
out by Markus Elfring.
OK reyk

branches: 1.60.2;
Add return_uri to serverconfig_reset() to avoid using garbage from the
imsg buffer.

Debugging & OK halex@

Allow TLS protocols to be specified via a "tls protocols" configuration
option.

ok reyk@

Change TLS_PROTOCOLS_DEFAULT to be TLSv1.2 only. Add a TLS_PROTOCOLS_ALL
that includes all currently supported protocols (TLSv1.0, TLSv1.1 and
TLSv1.2). Change all users of libtls to use TLS_PROTOCOLS_ALL so that they
maintain existing behaviour.

Discussed with tedu@ and reyk@.

Remove server_load_file() in favor of tls_load_file(3)

Add httpd configuration options to allow the specification of DHE
parameters and the ECDHE curve. This primarily allows for DHE cipher suites
to be enabled.

ok reyk@

Add support for blocking, dropping, and redirecting requests.

OK florian@

httpd is based on relayd and had included many headers that are only
needed by its ancestor. jsg@, include-what-you-use, and some manual
review helped to cleanup the headers (take iwyu with a grain of salt).
Based on common practice, httpd.h now also includes the necessary
headers for itself.

OK florian@

Decouple auth parameters from struct server_config into struct auth.

OK florian@

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

bump copyright year

Only open a socket once for each unique "listen on" statement. This
prevents running out of file descriptors when loading a configuration
with many aliases.

OK florian@

Stop pulling in <arpa/inet.h> or <arpa/nameser.h> when unnecessary.
*Do* pull it in when in_{port,addr}_h is needed and <netinet/in.h> isn't.

ok reyk@

Like previously done in relayd, change the keyword "ssl" to "tls" to
reflect reality.

OK benno@

stop viral header propagation. none of this code uses sys/hash.h
from Max Fillinger

Update httpd(8) to use libtls instead of libressl.

Remove unnecessary netinet/in_systm.h include.

ok millert@

Update ressl configuration to handle recent changes in the library.

ok tedu@

Fix an error case that was never handled ending up in an endless event
loop that could eat all CPU. I thought that the previous (correct)
commit fixed it which wasn't the case. But this one is obvious.

ok florian@

Remove a limitation that only allowed to specify a server name once.
The key has been changed to server name + address + port and now it is
possible to use the same server name for multiple servers with
different addresses, eg. http://www.example.com and
https://www.example.com/.

OK doug@ florian@

FastCGI did not support persistent connections. Add initial support
for persistent connections with FastCGI by implementing chunked
Transfer-Encoding. This only works with HTTP/1.1.

With input and help from florian@ who found some FastCGI edge cases.

OK florian@

Write all data before closing the server socket if the output buffer
is not empty. This fixes a bug of short responses that could happen
with large files or fcgi data on connections with a higher latency.

OK florian@

branches: 1.39.2;
Use memset(buf instead of memset(&buf.

Pointed out by deraadt@

Also clean up the public key when it is no longer needed.

ok deraadt@ reyk@

Configure the default SSL ciphers as HIGH:!aNULL.

ok deraadt@ reyk@

http POST support
with & OK reyk@

spacing

The watermark exposed a bug in server_write that broke keep-alive
support. Instead of calling server_close from server_write, we have
to proceed to the next connection by calling the error handler.

OK jsg@

Bring back the last read (done) / last write (done) messages instead of just
"done" to simplify connection debugging.

Adjust the read/write watermarks according to the TCP send buffer.
This fixes sending of large files. Previously, httpd was reading the
input file too quickly and could run out of memory when filling the
input buffer.

Found by jsg@
OK florian@

add missing va_start/va_end calls
ok deraadt@ guenther@

Load the SSL public/private keys in the parent process, then provide them
to the privsep process via imsg. This allows the keys to be moved out of
the chroot (now /etc/ssl/server.crt, /etc/ssl/private/server.key).

ok reyk@

Improve logging to allow per- server/location log files. The log
files can also be owned by root now: they're opened by the parent and
send to the logger process with fd passing. This also works with reload.

ok deraadt@

Temporarily move the default location of the SSL/TLS server key and
certificate from /var/www/ to /var/www/conf/. Don't get scared - this
will be changed soon! They're currently located in the chroot
directory but will be moved outside as soon as we adopted some of the
key privsep from relayd in ressl/httpd.

Proxy commit for jsing@:
"Add TLS/SSL support to httpd, based on the recent ressl commits."

From jsing@
ok reyk@

Add initial support for log files in /var/www/logs/. Logging with
syslog is still supported but disabled by default.

ok deraadt@

httpd doesn't support SSL/TLS yet, remove the remaining bits.
The secrect plan is to add it later using the ressl wrapper library.

spacing

Add another log mode "connection" for a relayd(8)-style log entry after
each connection, not every request. The code was already there and enabled
on debug, I just turned it into an alternative log format.

Allow to specify a FastCGI TCP socket on localhost (eg. :9000). Used
for debugging, you should prefer local UNIX sockets, but it helped to
find an issue that will be fixed with the next commit.

OK florian@

Use the log buffer to defer the logging until the connection is closed
or the request completed. Turn the old log message into a debug message.

ok doug@

remove the global "log updates/all" option that came from relayd.

Correctly parse fcgi records if we don't get the whole record in one
bufferevent_read().
Input/OK reyk@

One bufferevent can be shared by file and fcgi.

Make "location" work with name-based virtual servers.

Add "location" keyword to specify path-specific configuration in
servers, for example auto index for a sub-directory only. Internally,
a "location" is just a special type of a "virtual" server.

The inflight decremented message should only be printed with DEBUG.

Move configurable TCP options into struct server_config.

Differentiate servers by address and port, not just by address.

It is recommended to use a URL in the Location header of 3xx
responses. To accomplish this, add some semantics to retrieve the
server host name of a connection: either IP, IP:PORT (if not 80) or
[IP6]:PORT, or Host value (if valid).

Add support for "virtual hosts" aka. server blocks aka. multiple
servers with the same or "overlapping" IP address but a different name.

ok beck@

Split server and server_config.

Rename a field, needed later, no functional change.

Plug a memleak by correctly free'ing the HTTP descriptor that contains
all the headers etc. of a connection.

Correctly shutdown the servers when the process is terminating;
prevents a crash on exit. With debugging help from blambert@.

Implement file descriptor accounting. The concept was taken from
relayd but had to be adjusted for httpd. It now handles single-pass
HTTP connections, persistent connections with multiple requests, and
body-less HEAD requests. With input from benno@

first step towards keep-alive/persistent connections support

Finish writing the output before closing the connection
(adopted from relayd).

Close the connection after the response is completed (no Keepalive yet).

Add support for media types (aka. MIME types): the types section is
compatible to nginx' mime.types file which can be included directly.
If not present, use a few built-in defaults for html, css, txt, jpeg,
gif, png, and js.

Add httpd(8), an attempt to turn the relayd(8) codebase into a simple
web server. It is not finished yet and I just started it today, but
the goal is to provide an HTTP server that a) provides minimal
features, b) serves static files, c) provides FastCGI support, and d)
follows common coding practices of OpenBSD.

It will neither support plugins, nor custom memory allocators, EBCDIC
support, PCRE or any other things that can be found elsewhere.
httpd(8) is not intended to provide a fully-featured replacement for
nginx(8) or the Apache, but it will provide enough functionality that
is needed in the OpenBSD base system.

ok deraadt@

Use EVBUFFER_DATA instead of reaching into struct evbuffer. ok tb

Work around use after free in httpd(8)

A malformed HTTP request can cause httpd in fastcgi mode to crash due to a
use-after-free. This is an awful hack, but it's good enough until someone
figures out the correct way of dealing with server_close() here.

"this will do the trick for now" claudio
ok beck deraadt

branches: 1.126.6; 1.126.10;
Remove unneeded calls to tls_init(3)

As per the manual and lib/libtls/tls.c revision 1.79 from 2018
"Automatically handle library initialisation for libtls." initialisation
is handled automatically by other tls_*(3) functions.

Remove explicit tls_init() calls from base to not give the impression of
it being needed.

Feedback tb
OK Tests mestre

Do not compare TLS config params for non-TLS servers. This allows to
mix 'listen * port 80' and 'listen * tls port 443' in one server block.
Also the last argument of server_tls_cmp - match_keypair - is always 0
so remove this code.
OK florian@ tb@ some long time ago

Pull tls_close() and tls_free() further up, so tls_free() will already
be in the right spot once tls_close() is handled by libevent.

suggested by jsing

Call tls_close() before closing the underlying socket

In order to end a TLS connection regularly, an implementation MUST send a
close_notify alert. libtls does this in tls_close() via SSL_shutdown(),
so the socket had better still be open.

The incorrect order in server_close() caused a leak on each tls connection
due to a bug in libssl (fixed in tls_record_layer.c r1.56).

As pointed out by claudio, tls_close() should really be handled from the
main event loop. This will be addressed in a later commit.

ok claudio florian jsing

Don't leak the log message in server_sendlog

While there, use the length calculated by vasprintf() instead of using
strlen needlessly.

ok claudio florian

Handle absence of TLS certs while parsing the config

There is a soft fail mechanism to handle missing certs for seamless
interaction with acme-client. Move this to the config parser. This is
simpler than server.c r1.117 and avoids a crash due to listening on
port 443 without having set up the TLS context first. More precisely,
the crash happens if a server with missing certificate is visited via
https in a configuration where there is a second server with valid
certificate and key.

From Joshua Sing (joshua at hypera dot dev)

ok benno

httpd(8) sent a 408 response every time a connection request timeout
was reached. This is not what other servers are doing, it leads to
ugly log messages and might confuse some clients.
benno@ analyzed that the correct behavior is (probably) to send a 408
when we are in the middle of receiving headers and time out there and
just close the connection in all other cases.
In particular, if a connection gets opened and no request is received
at all just close the connection. If a connection is set to keep-alive
and a request was handled and no further request is coming in just
close the connection. The later is the usual cause for spurious log
messages and client confusion.

Reported over the years by many.
Input, explanations and OK benno

When system calls indicate an error they return -1, not some arbitrary
value < 0. errno is only updated in this case. Change all (most?)
callers of syscalls to follow this better, and let's see if this strictness
helps us in the future.

httpd(8): add support for setting custom FastCGI parameters.

This commit extends the existing grammar by adding the param option
to the fastcgi directive: fastcgi param name value.

Example usage:
fastcgi param VAR1 hello
fastcgi param VAR2 world

With help and OK florian@
Rogue manpage bits, feel free to modify them.

Allow httpd(8) to start when TLS is configured but a cert is not yet
available.
Assuming a httpd.conf based on /etc/examples/httpd.conf, httpd(8)
will only listen on port 80 and serve the acme-challenge directory
for acme-client(1).
The workflow to get a certificate then becomes
acme-client -vAD example.com && rcctl reload httpd
Without the need to edit the httpd.conf yet again. Once the cert
is in place and httpd is reloaded it starts to serve on port 443.

Idea, tweaks & OK deraadt, OK benno

Backout my previous commit:

date: 2018/10/01 19:24:09; author: benno; state: Exp; lines: +7 -1;
commitid: 0O8fyHPNvPd8rvYU;
Only send 408 Timeout responses when we have seen at least part of a
request. Without a request, just close the connection when we hit
request timeout.
Prompted by a bug report from Nikola Kolev, thanks.
ok reyk@ and some suggestions from claudio@ and bluhm@

Mark Patruck (mark AT wrapped DOT cx) found a problem with it, thanks
for the report.

ok reyk@ bluhm@ sthen@ deraadt@

Only send 408 Timeout responses when we have seen at least part of a
request. Without a request, just close the connection when we hit
request timeout.
Prompted by a bug report from Nikola Kolev, thanks.
ok reyk@ and some suggestions from claudio@ and bluhm@

Add support for client certificate authentication to httpd.

From Jack Burton <jack at saosce dot com dot au> - thanks!

Also tested by Jan Klemkow <j.klemkow at wemelug dot de>.

ok beck@ reyk@

Don't do OCSP stapling only if the staple file is 0 length.

This allows something external (like ocspcheck) to disable the stapling
deliberatly if it can not retreive a valid staple by truncating the
staple file to indicate "do not provide a staple", while the file not
existin will still be treated as a configuration error
ok claudio@ florian@, and prompted by @jsing

Disable oscp stapling on invalid staple, rather than failing to start.
ok claudio@ florian@

Convert httpd to tls_config_set_ecdhecurves(), allowing a list of curves
to be specified, rather than a single curve.

ok beck@

Rework the way that TLS configuration is sent/received via imsgs, so that
are no longer limited by the 16KB maximum size of a single imsg.
Configuration data that is larger than a single message is now chunked and
sent via multiple imsgs.

Prompted by a diff from Jack Burton <jack at saosce dot com dot au>.

ok reyk@

some freezero() calls

Implement TLS ticket support in httpd. Off by default. Use
tls ticket lifetime default
to turn it on with a 2h ticket lifetime.
Rekeying happens after a quarter of that time.
OK reky@ and bob@

/tmp/cvsspEkok

Fix support for HTTP pipelining by handling all requests in the buffer.

Tested & OK jung@

remove extra call setting OCSP staple now that it is done above
using keypair_ocsp..
ok reyk@

Correct mistake I made when converting this to new funciton

Add tls_config_[add|set]keypair_ocsp functions so that ocsp staples may be
added associated to a keypair used for SNI, and are usable for more than
just the "main" certificate. Modify httpd to use this.
Bump libtls minor.

ok jsing@

Do not set EVBUFFER_EOF on read/write errors and handle EOF correctly.

Either libevent or the TLS callback can trigger an EOF when the
connection is closed.

OK sunil@ jung@ benno@

Stop accessing verbose and debug variables from log.c directly.

This replaces log_verbose() and "extern int verbose" with the two functions
log_setverbose() and log_getverbose().

Pointed out by benno@
OK krw@ eric@ gilles@ (OK gilles@ for the snmpd bits as well)

Check the return value of tls_config_set_protocols(), now that it returns
an int.

Move OCSP loading into a separate function - it is not part of the keypair
and this way we can give a separate specific error message.

ok beck@ reyk@

Fix tcp ip ttl / minttl on IPv6 sockets.

ok florian@

conditionalize ocsp load properly
ok jsing@

Add OCSP stapling support to httpd
ok jsing@ bcook@

Kill (remove) the ps_pid from privsep struct since it is not being used
anymore. Also fix the process initialization prototypes.

ok reyk@

Kill p_instance from proc.c and remove static proc_id unused variables.

To keep the debug functionality intact and correct we'll use the pid
field in the imsg header to pass the instance number. Remember to always
pass 'ps_instance + 1' otherwise libutil will fill imsg header pid field
with the imsgbuf pid (which is the current process pid).

ok reyk@

Replace the static env variables with a single global variable.

ok reyk@

Enable SNI support in httpd(8).

ok reyk@

stop including sys/param.h for nitems. define locally as needed.
ok natano reyk

Turn "TLS handshake failed -" log message into a debug message - it
happens way too often and does not provide much information.

OK jung@

Rename server_handshake_tls() to server_tls_handshake() to align with
the other server_tls_* functions (and I like the prefix notation
better). No functional change.

Move server_match() from parse.y to server.c; use env instead of conf,
which is actually the same thing (cluebat from reyk@).

Use lowercase 'tls' in debug and log messages for consistency.

Requested by reyk@

Make httpd stricter with respect to TLS configuration - in particular, do
not allow TLS and non-TLS to be configured on the same port, do not allow
TLS options to be specified without a TLS listener and ensure that the TLS
options are the same when a server is specified on the same address/port.
Currently, these configurations are permitted but do not work as intended.

Also factor out and reuse the server matching code, which was previously
duplicated.

ok reyk@

Include the TLS configuration errors in log messages. Also set the
certificate and private key at the same time.

Use log_warnx() instead of log_warn() when the failure will not have
resulted in errno being set.

ok reyk@

sync with relayd, use proc_compose()

Retire socket_set_blockmode() in favor of the SOCK_NONBLOCK type flag.
As done in iked and snmpd.

OK jung@

pledge(2) for httpd.

1) The main process listens on sockets and accepts connections. It
creates and opens log files, creates and kills child processes. On
start up and on receiving a HUP signal it parses the configuration. It
passes on file descriptors for logging or requests to it's children.
2) The logger process writes log messages to a file descriptor passed
in from the main process.
3) The server process reads the request from a file descriptor passed
in from the main process. It reads a file or creates a directory index
to send a response.
Additionally this process handles fastcgi requests. It connects to
AF_UNIX, AF_INET or AF_INET6 sockets. A re-factoring might make it
possible to drop the additional fastcgi privileges when only static
files are served.

with deraadt@ some time ago
prodding & OK deraadt@
tweaks and OK reyk@

Fix server_handshake_tls() - we should only call server_input() in the case
where the handshake has successfully completed.

ok beck@

fix return type for tls_read/write
jointly with jsing@

fix after libtls api changes
ok jsing@

Update httpd to call tls_handshake() after tls_accept_socket().

ok beck@

Fix a regression that was introduced with server.c r1.64: Do NOT free
srv_conf->auth in serverconfig_free() because it was not allocated in
config_getserver() but assigned as a reference by id from a global
list that is maintained independently. This fixes a potential
double-free. This fix also makes srv_conf->auth "const" to emphasize
that the read-only auth pointer was not allocated here.

OK jsing@

Change httpd(8) to use C99-style fixed-width integers (uintN_t instead
of u_intN_t) and replace u_int with unsigned int. Mixing both
variants is a bad style and most contributors seem to prefer this
style; it also helps us to get used to it, portability, and
standardization.

Theoretically no binary change, except one in practice: httpd.o has a
different checksum because gcc with -O2 pads/optimizes "struct
privsep" differently when using "unsigned int" instead "u_int" for the
affected members. "u_int" is just a typedef of "unsigned int", -O0
doesn't build the difference and clang with -O2 doesn't do it either -
it is just another curiosity from gcc-land.

OK semarie@

Fix rev 1.70 of server.c by only re-enabling the bufferevent if we
previously disabled it because we were reading to fast (from disk).
Problem noted and tracked down to that commit by weerd@ and
independently by stsp@.
Tested by weerd@, stsp@, reyk@
OK bluhm@, reyk@

backout the previous: it broke wordpress somehow.
we need more care to find a proper fix for the fastcgi headers.

acknowledged by deraadt@

Read fcgi response records until we have the whole http header and can
parse it. Otherwise http headers can leak into the body.
Pointed out by Jean-Philippe Ouellet on bugs@ Thanks!
OK reyk, commit ASAP deraadt@

libtls has been changed to set SSL_MODE_ENABLE_PARTIAL_WRITE and
SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER by default. This gives
tls_write() a similar short write semantics as write(2) and a
workaround in httpd to cope with the previous differences can be
removed. Specifically, httpd can stop copying data into a local
buffer that was used to keep it around for repeated writes.

OK bluhm@

If we can read faster from disk than send data to the client stop
reading from disk when we hold a certain amount of data in
RAM. Re-enable reading once we send enough data to the
client. Otherwise we might end up with the whole file (which can be
huge) in RAM.
Reported by Matthew Martin ( matt.a.martin AT gmail ) on bugs@,
thanks!
OK reyk@, benno@

Escape the message in server_log() as well.

OK benno@

Close connections that fail to complete a TLS handshake.

Based on a diff from Jack Burton <jack at saosce dot com dot au>.

ok reyk@

Fix typo in comment.

Send the TLS certificate and key via separate imsgs, rather than
including them in the IMSG_CFG_SERVER imsg. This allows the certificate
and key to each be almost 16KB (the maximum size for an imsg), rather than
having a combined total of less than 16KB (which can be reached with large
keys, certificate bundles or by including text versions of certificates).

ok reyk@

Explicitly check for and handle EOF on a TLS connection.

ok reyk@

Fix memory leaks that can occur when config_getserver() fails.

config.c r1.34 and r1.30 introduced potential memory leaks for auth and
return_uri when config_getserver fails. Fix this by switching to
serverconfig_free() and adding the missing free for srv_conf->auth.
While here, make serverconfig_free() a little more bulletproof by
explicit_bzero()ing key material.

ok reyk@

We cannot log errors with server_close() before allocating
clt_log evbuffer.
server_close() calls server_log() which uses ctl_log.
Crash reported by Daniel Jakots <vigdis AT chown DOT me>, thanks!
OK benno

Always check the return value of proc_composev_imsg() and handle failures
appropriately. Otherwise imsg construction can silently fail, resulting in
non-obvious problems.

Found the hard way by Theodore Wynnychenko.

ok doug@ florian@

Prevent use after free.
While here unconditionally free clt and move declaration of
server_inflight_dec() into server.c
Found while investigating if (foo != NULL) free(foo) patterns pointed
out by Markus Elfring.
OK reyk

branches: 1.60.2;
Add return_uri to serverconfig_reset() to avoid using garbage from the
imsg buffer.

Debugging & OK halex@

Allow TLS protocols to be specified via a "tls protocols" configuration
option.

ok reyk@

Change TLS_PROTOCOLS_DEFAULT to be TLSv1.2 only. Add a TLS_PROTOCOLS_ALL
that includes all currently supported protocols (TLSv1.0, TLSv1.1 and
TLSv1.2). Change all users of libtls to use TLS_PROTOCOLS_ALL so that they
maintain existing behaviour.

Discussed with tedu@ and reyk@.

Remove server_load_file() in favor of tls_load_file(3)

Add httpd configuration options to allow the specification of DHE
parameters and the ECDHE curve. This primarily allows for DHE cipher suites
to be enabled.

ok reyk@

Add support for blocking, dropping, and redirecting requests.

OK florian@

httpd is based on relayd and had included many headers that are only
needed by its ancestor. jsg@, include-what-you-use, and some manual
review helped to cleanup the headers (take iwyu with a grain of salt).
Based on common practice, httpd.h now also includes the necessary
headers for itself.

OK florian@

Decouple auth parameters from struct server_config into struct auth.

OK florian@

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

bump copyright year

Only open a socket once for each unique "listen on" statement. This
prevents running out of file descriptors when loading a configuration
with many aliases.

OK florian@

Stop pulling in <arpa/inet.h> or <arpa/nameser.h> when unnecessary.
*Do* pull it in when in_{port,addr}_h is needed and <netinet/in.h> isn't.

ok reyk@

Like previously done in relayd, change the keyword "ssl" to "tls" to
reflect reality.

OK benno@

stop viral header propagation. none of this code uses sys/hash.h
from Max Fillinger

Update httpd(8) to use libtls instead of libressl.

Remove unnecessary netinet/in_systm.h include.

ok millert@

Update ressl configuration to handle recent changes in the library.

ok tedu@

Fix an error case that was never handled ending up in an endless event
loop that could eat all CPU. I thought that the previous (correct)
commit fixed it which wasn't the case. But this one is obvious.

ok florian@

Remove a limitation that only allowed to specify a server name once.
The key has been changed to server name + address + port and now it is
possible to use the same server name for multiple servers with
different addresses, eg. http://www.example.com and
https://www.example.com/.

OK doug@ florian@

FastCGI did not support persistent connections. Add initial support
for persistent connections with FastCGI by implementing chunked
Transfer-Encoding. This only works with HTTP/1.1.

With input and help from florian@ who found some FastCGI edge cases.

OK florian@

Write all data before closing the server socket if the output buffer
is not empty. This fixes a bug of short responses that could happen
with large files or fcgi data on connections with a higher latency.

OK florian@

branches: 1.39.2;
Use memset(buf instead of memset(&buf.

Pointed out by deraadt@

Also clean up the public key when it is no longer needed.

ok deraadt@ reyk@

Configure the default SSL ciphers as HIGH:!aNULL.

ok deraadt@ reyk@

http POST support
with & OK reyk@

spacing

The watermark exposed a bug in server_write that broke keep-alive
support. Instead of calling server_close from server_write, we have
to proceed to the next connection by calling the error handler.

OK jsg@

Bring back the last read (done) / last write (done) messages instead of just
"done" to simplify connection debugging.

Adjust the read/write watermarks according to the TCP send buffer.
This fixes sending of large files. Previously, httpd was reading the
input file too quickly and could run out of memory when filling the
input buffer.

Found by jsg@
OK florian@

add missing va_start/va_end calls
ok deraadt@ guenther@

Load the SSL public/private keys in the parent process, then provide them
to the privsep process via imsg. This allows the keys to be moved out of
the chroot (now /etc/ssl/server.crt, /etc/ssl/private/server.key).

ok reyk@

Improve logging to allow per- server/location log files. The log
files can also be owned by root now: they're opened by the parent and
send to the logger process with fd passing. This also works with reload.

ok deraadt@

Temporarily move the default location of the SSL/TLS server key and
certificate from /var/www/ to /var/www/conf/. Don't get scared - this
will be changed soon! They're currently located in the chroot
directory but will be moved outside as soon as we adopted some of the
key privsep from relayd in ressl/httpd.

Proxy commit for jsing@:
"Add TLS/SSL support to httpd, based on the recent ressl commits."

From jsing@
ok reyk@

Add initial support for log files in /var/www/logs/. Logging with
syslog is still supported but disabled by default.

ok deraadt@

httpd doesn't support SSL/TLS yet, remove the remaining bits.
The secrect plan is to add it later using the ressl wrapper library.

spacing

Add another log mode "connection" for a relayd(8)-style log entry after
each connection, not every request. The code was already there and enabled
on debug, I just turned it into an alternative log format.

Allow to specify a FastCGI TCP socket on localhost (eg. :9000). Used
for debugging, you should prefer local UNIX sockets, but it helped to
find an issue that will be fixed with the next commit.

OK florian@

Use the log buffer to defer the logging until the connection is closed
or the request completed. Turn the old log message into a debug message.

ok doug@

remove the global "log updates/all" option that came from relayd.

Correctly parse fcgi records if we don't get the whole record in one
bufferevent_read().
Input/OK reyk@

One bufferevent can be shared by file and fcgi.

Make "location" work with name-based virtual servers.

Add "location" keyword to specify path-specific configuration in
servers, for example auto index for a sub-directory only. Internally,
a "location" is just a special type of a "virtual" server.

The inflight decremented message should only be printed with DEBUG.

Move configurable TCP options into struct server_config.

Differentiate servers by address and port, not just by address.

It is recommended to use a URL in the Location header of 3xx
responses. To accomplish this, add some semantics to retrieve the
server host name of a connection: either IP, IP:PORT (if not 80) or
[IP6]:PORT, or Host value (if valid).

Add support for "virtual hosts" aka. server blocks aka. multiple
servers with the same or "overlapping" IP address but a different name.

ok beck@

Split server and server_config.

Rename a field, needed later, no functional change.

Plug a memleak by correctly free'ing the HTTP descriptor that contains
all the headers etc. of a connection.

Correctly shutdown the servers when the process is terminating;
prevents a crash on exit. With debugging help from blambert@.

Implement file descriptor accounting. The concept was taken from
relayd but had to be adjusted for httpd. It now handles single-pass
HTTP connections, persistent connections with multiple requests, and
body-less HEAD requests. With input from benno@

first step towards keep-alive/persistent connections support

Finish writing the output before closing the connection
(adopted from relayd).

Close the connection after the response is completed (no Keepalive yet).

Add support for media types (aka. MIME types): the types section is
compatible to nginx' mime.types file which can be included directly.
If not present, use a few built-in defaults for html, css, txt, jpeg,
gif, png, and js.

Add httpd(8), an attempt to turn the relayd(8) codebase into a simple
web server. It is not finished yet and I just started it today, but
the goal is to provide an HTTP server that a) provides minimal
features, b) serves static files, c) provides FastCGI support, and d)
follows common coding practices of OpenBSD.

It will neither support plugins, nor custom memory allocators, EBCDIC
support, PCRE or any other things that can be found elsewhere.
httpd(8) is not intended to provide a fully-featured replacement for
nginx(8) or the Apache, but it will provide enough functionality that
is needed in the OpenBSD base system.

ok deraadt@

Work around use after free in httpd(8)

A malformed HTTP request can cause httpd in fastcgi mode to crash due to a
use-after-free. This is an awful hack, but it's good enough until someone
figures out the correct way of dealing with server_close() here.

"this will do the trick for now" claudio
ok beck deraadt

branches: 1.126.6; 1.126.10;
Remove unneeded calls to tls_init(3)

As per the manual and lib/libtls/tls.c revision 1.79 from 2018
"Automatically handle library initialisation for libtls." initialisation
is handled automatically by other tls_*(3) functions.

Remove explicit tls_init() calls from base to not give the impression of
it being needed.

Feedback tb
OK Tests mestre

Do not compare TLS config params for non-TLS servers. This allows to
mix 'listen * port 80' and 'listen * tls port 443' in one server block.
Also the last argument of server_tls_cmp - match_keypair - is always 0
so remove this code.
OK florian@ tb@ some long time ago

Pull tls_close() and tls_free() further up, so tls_free() will already
be in the right spot once tls_close() is handled by libevent.

suggested by jsing

Call tls_close() before closing the underlying socket

In order to end a TLS connection regularly, an implementation MUST send a
close_notify alert. libtls does this in tls_close() via SSL_shutdown(),
so the socket had better still be open.

The incorrect order in server_close() caused a leak on each tls connection
due to a bug in libssl (fixed in tls_record_layer.c r1.56).

As pointed out by claudio, tls_close() should really be handled from the
main event loop. This will be addressed in a later commit.

ok claudio florian jsing

Don't leak the log message in server_sendlog

While there, use the length calculated by vasprintf() instead of using
strlen needlessly.

ok claudio florian

Handle absence of TLS certs while parsing the config

There is a soft fail mechanism to handle missing certs for seamless
interaction with acme-client. Move this to the config parser. This is
simpler than server.c r1.117 and avoids a crash due to listening on
port 443 without having set up the TLS context first. More precisely,
the crash happens if a server with missing certificate is visited via
https in a configuration where there is a second server with valid
certificate and key.

From Joshua Sing (joshua at hypera dot dev)

ok benno

httpd(8) sent a 408 response every time a connection request timeout
was reached. This is not what other servers are doing, it leads to
ugly log messages and might confuse some clients.
benno@ analyzed that the correct behavior is (probably) to send a 408
when we are in the middle of receiving headers and time out there and
just close the connection in all other cases.
In particular, if a connection gets opened and no request is received
at all just close the connection. If a connection is set to keep-alive
and a request was handled and no further request is coming in just
close the connection. The later is the usual cause for spurious log
messages and client confusion.

Reported over the years by many.
Input, explanations and OK benno

When system calls indicate an error they return -1, not some arbitrary
value < 0. errno is only updated in this case. Change all (most?)
callers of syscalls to follow this better, and let's see if this strictness
helps us in the future.

httpd(8): add support for setting custom FastCGI parameters.

This commit extends the existing grammar by adding the param option
to the fastcgi directive: fastcgi param name value.

Example usage:
fastcgi param VAR1 hello
fastcgi param VAR2 world

With help and OK florian@
Rogue manpage bits, feel free to modify them.

Allow httpd(8) to start when TLS is configured but a cert is not yet
available.
Assuming a httpd.conf based on /etc/examples/httpd.conf, httpd(8)
will only listen on port 80 and serve the acme-challenge directory
for acme-client(1).
The workflow to get a certificate then becomes
acme-client -vAD example.com && rcctl reload httpd
Without the need to edit the httpd.conf yet again. Once the cert
is in place and httpd is reloaded it starts to serve on port 443.

Idea, tweaks & OK deraadt, OK benno

Backout my previous commit:

date: 2018/10/01 19:24:09; author: benno; state: Exp; lines: +7 -1;
commitid: 0O8fyHPNvPd8rvYU;
Only send 408 Timeout responses when we have seen at least part of a
request. Without a request, just close the connection when we hit
request timeout.
Prompted by a bug report from Nikola Kolev, thanks.
ok reyk@ and some suggestions from claudio@ and bluhm@

Mark Patruck (mark AT wrapped DOT cx) found a problem with it, thanks
for the report.

ok reyk@ bluhm@ sthen@ deraadt@

Only send 408 Timeout responses when we have seen at least part of a
request. Without a request, just close the connection when we hit
request timeout.
Prompted by a bug report from Nikola Kolev, thanks.
ok reyk@ and some suggestions from claudio@ and bluhm@

Add support for client certificate authentication to httpd.

From Jack Burton <jack at saosce dot com dot au> - thanks!

Also tested by Jan Klemkow <j.klemkow at wemelug dot de>.

ok beck@ reyk@

Don't do OCSP stapling only if the staple file is 0 length.

This allows something external (like ocspcheck) to disable the stapling
deliberatly if it can not retreive a valid staple by truncating the
staple file to indicate "do not provide a staple", while the file not
existin will still be treated as a configuration error
ok claudio@ florian@, and prompted by @jsing

Disable oscp stapling on invalid staple, rather than failing to start.
ok claudio@ florian@

Convert httpd to tls_config_set_ecdhecurves(), allowing a list of curves
to be specified, rather than a single curve.

ok beck@

Rework the way that TLS configuration is sent/received via imsgs, so that
are no longer limited by the 16KB maximum size of a single imsg.
Configuration data that is larger than a single message is now chunked and
sent via multiple imsgs.

Prompted by a diff from Jack Burton <jack at saosce dot com dot au>.

ok reyk@

some freezero() calls

Implement TLS ticket support in httpd. Off by default. Use
tls ticket lifetime default
to turn it on with a 2h ticket lifetime.
Rekeying happens after a quarter of that time.
OK reky@ and bob@

/tmp/cvsspEkok

Fix support for HTTP pipelining by handling all requests in the buffer.

Tested & OK jung@

remove extra call setting OCSP staple now that it is done above
using keypair_ocsp..
ok reyk@

Correct mistake I made when converting this to new funciton

Add tls_config_[add|set]keypair_ocsp functions so that ocsp staples may be
added associated to a keypair used for SNI, and are usable for more than
just the "main" certificate. Modify httpd to use this.
Bump libtls minor.

ok jsing@

Do not set EVBUFFER_EOF on read/write errors and handle EOF correctly.

Either libevent or the TLS callback can trigger an EOF when the
connection is closed.

OK sunil@ jung@ benno@

Stop accessing verbose and debug variables from log.c directly.

This replaces log_verbose() and "extern int verbose" with the two functions
log_setverbose() and log_getverbose().

Pointed out by benno@
OK krw@ eric@ gilles@ (OK gilles@ for the snmpd bits as well)

Check the return value of tls_config_set_protocols(), now that it returns
an int.

Move OCSP loading into a separate function - it is not part of the keypair
and this way we can give a separate specific error message.

ok beck@ reyk@

Fix tcp ip ttl / minttl on IPv6 sockets.

ok florian@

conditionalize ocsp load properly
ok jsing@

Add OCSP stapling support to httpd
ok jsing@ bcook@

Kill (remove) the ps_pid from privsep struct since it is not being used
anymore. Also fix the process initialization prototypes.

ok reyk@

Kill p_instance from proc.c and remove static proc_id unused variables.

To keep the debug functionality intact and correct we'll use the pid
field in the imsg header to pass the instance number. Remember to always
pass 'ps_instance + 1' otherwise libutil will fill imsg header pid field
with the imsgbuf pid (which is the current process pid).

ok reyk@

Replace the static env variables with a single global variable.

ok reyk@

Enable SNI support in httpd(8).

ok reyk@

stop including sys/param.h for nitems. define locally as needed.
ok natano reyk

Turn "TLS handshake failed -" log message into a debug message - it
happens way too often and does not provide much information.

OK jung@

Rename server_handshake_tls() to server_tls_handshake() to align with
the other server_tls_* functions (and I like the prefix notation
better). No functional change.

Move server_match() from parse.y to server.c; use env instead of conf,
which is actually the same thing (cluebat from reyk@).

Use lowercase 'tls' in debug and log messages for consistency.

Requested by reyk@

Make httpd stricter with respect to TLS configuration - in particular, do
not allow TLS and non-TLS to be configured on the same port, do not allow
TLS options to be specified without a TLS listener and ensure that the TLS
options are the same when a server is specified on the same address/port.
Currently, these configurations are permitted but do not work as intended.

Also factor out and reuse the server matching code, which was previously
duplicated.

ok reyk@

Include the TLS configuration errors in log messages. Also set the
certificate and private key at the same time.

Use log_warnx() instead of log_warn() when the failure will not have
resulted in errno being set.

ok reyk@

sync with relayd, use proc_compose()

Retire socket_set_blockmode() in favor of the SOCK_NONBLOCK type flag.
As done in iked and snmpd.

OK jung@

pledge(2) for httpd.

1) The main process listens on sockets and accepts connections. It
creates and opens log files, creates and kills child processes. On
start up and on receiving a HUP signal it parses the configuration. It
passes on file descriptors for logging or requests to it's children.
2) The logger process writes log messages to a file descriptor passed
in from the main process.
3) The server process reads the request from a file descriptor passed
in from the main process. It reads a file or creates a directory index
to send a response.
Additionally this process handles fastcgi requests. It connects to
AF_UNIX, AF_INET or AF_INET6 sockets. A re-factoring might make it
possible to drop the additional fastcgi privileges when only static
files are served.

with deraadt@ some time ago
prodding & OK deraadt@
tweaks and OK reyk@

Fix server_handshake_tls() - we should only call server_input() in the case
where the handshake has successfully completed.

ok beck@

fix return type for tls_read/write
jointly with jsing@

fix after libtls api changes
ok jsing@

Update httpd to call tls_handshake() after tls_accept_socket().

ok beck@

Fix a regression that was introduced with server.c r1.64: Do NOT free
srv_conf->auth in serverconfig_free() because it was not allocated in
config_getserver() but assigned as a reference by id from a global
list that is maintained independently. This fixes a potential
double-free. This fix also makes srv_conf->auth "const" to emphasize
that the read-only auth pointer was not allocated here.

OK jsing@

Change httpd(8) to use C99-style fixed-width integers (uintN_t instead
of u_intN_t) and replace u_int with unsigned int. Mixing both
variants is a bad style and most contributors seem to prefer this
style; it also helps us to get used to it, portability, and
standardization.

Theoretically no binary change, except one in practice: httpd.o has a
different checksum because gcc with -O2 pads/optimizes "struct
privsep" differently when using "unsigned int" instead "u_int" for the
affected members. "u_int" is just a typedef of "unsigned int", -O0
doesn't build the difference and clang with -O2 doesn't do it either -
it is just another curiosity from gcc-land.

OK semarie@

Fix rev 1.70 of server.c by only re-enabling the bufferevent if we
previously disabled it because we were reading to fast (from disk).
Problem noted and tracked down to that commit by weerd@ and
independently by stsp@.
Tested by weerd@, stsp@, reyk@
OK bluhm@, reyk@

backout the previous: it broke wordpress somehow.
we need more care to find a proper fix for the fastcgi headers.

acknowledged by deraadt@

Read fcgi response records until we have the whole http header and can
parse it. Otherwise http headers can leak into the body.
Pointed out by Jean-Philippe Ouellet on bugs@ Thanks!
OK reyk, commit ASAP deraadt@

libtls has been changed to set SSL_MODE_ENABLE_PARTIAL_WRITE and
SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER by default. This gives
tls_write() a similar short write semantics as write(2) and a
workaround in httpd to cope with the previous differences can be
removed. Specifically, httpd can stop copying data into a local
buffer that was used to keep it around for repeated writes.

OK bluhm@

If we can read faster from disk than send data to the client stop
reading from disk when we hold a certain amount of data in
RAM. Re-enable reading once we send enough data to the
client. Otherwise we might end up with the whole file (which can be
huge) in RAM.
Reported by Matthew Martin ( matt.a.martin AT gmail ) on bugs@,
thanks!
OK reyk@, benno@

Escape the message in server_log() as well.

OK benno@

Close connections that fail to complete a TLS handshake.

Based on a diff from Jack Burton <jack at saosce dot com dot au>.

ok reyk@

Fix typo in comment.

Send the TLS certificate and key via separate imsgs, rather than
including them in the IMSG_CFG_SERVER imsg. This allows the certificate
and key to each be almost 16KB (the maximum size for an imsg), rather than
having a combined total of less than 16KB (which can be reached with large
keys, certificate bundles or by including text versions of certificates).

ok reyk@

Explicitly check for and handle EOF on a TLS connection.

ok reyk@

Fix memory leaks that can occur when config_getserver() fails.

config.c r1.34 and r1.30 introduced potential memory leaks for auth and
return_uri when config_getserver fails. Fix this by switching to
serverconfig_free() and adding the missing free for srv_conf->auth.
While here, make serverconfig_free() a little more bulletproof by
explicit_bzero()ing key material.

ok reyk@

We cannot log errors with server_close() before allocating
clt_log evbuffer.
server_close() calls server_log() which uses ctl_log.
Crash reported by Daniel Jakots <vigdis AT chown DOT me>, thanks!
OK benno

Always check the return value of proc_composev_imsg() and handle failures
appropriately. Otherwise imsg construction can silently fail, resulting in
non-obvious problems.

Found the hard way by Theodore Wynnychenko.

ok doug@ florian@

Prevent use after free.
While here unconditionally free clt and move declaration of
server_inflight_dec() into server.c
Found while investigating if (foo != NULL) free(foo) patterns pointed
out by Markus Elfring.
OK reyk

branches: 1.60.2;
Add return_uri to serverconfig_reset() to avoid using garbage from the
imsg buffer.

Debugging & OK halex@

Allow TLS protocols to be specified via a "tls protocols" configuration
option.

ok reyk@

Change TLS_PROTOCOLS_DEFAULT to be TLSv1.2 only. Add a TLS_PROTOCOLS_ALL
that includes all currently supported protocols (TLSv1.0, TLSv1.1 and
TLSv1.2). Change all users of libtls to use TLS_PROTOCOLS_ALL so that they
maintain existing behaviour.

Discussed with tedu@ and reyk@.

Remove server_load_file() in favor of tls_load_file(3)

Add httpd configuration options to allow the specification of DHE
parameters and the ECDHE curve. This primarily allows for DHE cipher suites
to be enabled.

ok reyk@

Add support for blocking, dropping, and redirecting requests.

OK florian@

httpd is based on relayd and had included many headers that are only
needed by its ancestor. jsg@, include-what-you-use, and some manual
review helped to cleanup the headers (take iwyu with a grain of salt).
Based on common practice, httpd.h now also includes the necessary
headers for itself.

OK florian@

Decouple auth parameters from struct server_config into struct auth.

OK florian@

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

bump copyright year

Only open a socket once for each unique "listen on" statement. This
prevents running out of file descriptors when loading a configuration
with many aliases.

OK florian@

Stop pulling in <arpa/inet.h> or <arpa/nameser.h> when unnecessary.
*Do* pull it in when in_{port,addr}_h is needed and <netinet/in.h> isn't.

ok reyk@

Like previously done in relayd, change the keyword "ssl" to "tls" to
reflect reality.

OK benno@

stop viral header propagation. none of this code uses sys/hash.h
from Max Fillinger

Update httpd(8) to use libtls instead of libressl.

Remove unnecessary netinet/in_systm.h include.

ok millert@

Update ressl configuration to handle recent changes in the library.

ok tedu@

Fix an error case that was never handled ending up in an endless event
loop that could eat all CPU. I thought that the previous (correct)
commit fixed it which wasn't the case. But this one is obvious.

ok florian@

Remove a limitation that only allowed to specify a server name once.
The key has been changed to server name + address + port and now it is
possible to use the same server name for multiple servers with
different addresses, eg. http://www.example.com and
https://www.example.com/.

OK doug@ florian@

FastCGI did not support persistent connections. Add initial support
for persistent connections with FastCGI by implementing chunked
Transfer-Encoding. This only works with HTTP/1.1.

With input and help from florian@ who found some FastCGI edge cases.

OK florian@

Write all data before closing the server socket if the output buffer
is not empty. This fixes a bug of short responses that could happen
with large files or fcgi data on connections with a higher latency.

OK florian@

branches: 1.39.2;
Use memset(buf instead of memset(&buf.

Pointed out by deraadt@

Also clean up the public key when it is no longer needed.

ok deraadt@ reyk@

Configure the default SSL ciphers as HIGH:!aNULL.

ok deraadt@ reyk@

http POST support
with & OK reyk@

spacing

The watermark exposed a bug in server_write that broke keep-alive
support. Instead of calling server_close from server_write, we have
to proceed to the next connection by calling the error handler.

OK jsg@

Bring back the last read (done) / last write (done) messages instead of just
"done" to simplify connection debugging.

Adjust the read/write watermarks according to the TCP send buffer.
This fixes sending of large files. Previously, httpd was reading the
input file too quickly and could run out of memory when filling the
input buffer.

Found by jsg@
OK florian@

add missing va_start/va_end calls
ok deraadt@ guenther@

Load the SSL public/private keys in the parent process, then provide them
to the privsep process via imsg. This allows the keys to be moved out of
the chroot (now /etc/ssl/server.crt, /etc/ssl/private/server.key).

ok reyk@

Improve logging to allow per- server/location log files. The log
files can also be owned by root now: they're opened by the parent and
send to the logger process with fd passing. This also works with reload.

ok deraadt@

Temporarily move the default location of the SSL/TLS server key and
certificate from /var/www/ to /var/www/conf/. Don't get scared - this
will be changed soon! They're currently located in the chroot
directory but will be moved outside as soon as we adopted some of the
key privsep from relayd in ressl/httpd.

Proxy commit for jsing@:
"Add TLS/SSL support to httpd, based on the recent ressl commits."

From jsing@
ok reyk@

Add initial support for log files in /var/www/logs/. Logging with
syslog is still supported but disabled by default.

ok deraadt@

httpd doesn't support SSL/TLS yet, remove the remaining bits.
The secrect plan is to add it later using the ressl wrapper library.

spacing

Add another log mode "connection" for a relayd(8)-style log entry after
each connection, not every request. The code was already there and enabled
on debug, I just turned it into an alternative log format.

Allow to specify a FastCGI TCP socket on localhost (eg. :9000). Used
for debugging, you should prefer local UNIX sockets, but it helped to
find an issue that will be fixed with the next commit.

OK florian@

Use the log buffer to defer the logging until the connection is closed
or the request completed. Turn the old log message into a debug message.

ok doug@

remove the global "log updates/all" option that came from relayd.

Correctly parse fcgi records if we don't get the whole record in one
bufferevent_read().
Input/OK reyk@

One bufferevent can be shared by file and fcgi.

Make "location" work with name-based virtual servers.

Add "location" keyword to specify path-specific configuration in
servers, for example auto index for a sub-directory only. Internally,
a "location" is just a special type of a "virtual" server.

The inflight decremented message should only be printed with DEBUG.

Move configurable TCP options into struct server_config.

Differentiate servers by address and port, not just by address.

It is recommended to use a URL in the Location header of 3xx
responses. To accomplish this, add some semantics to retrieve the
server host name of a connection: either IP, IP:PORT (if not 80) or
[IP6]:PORT, or Host value (if valid).

Add support for "virtual hosts" aka. server blocks aka. multiple
servers with the same or "overlapping" IP address but a different name.

ok beck@

Split server and server_config.

Rename a field, needed later, no functional change.

Plug a memleak by correctly free'ing the HTTP descriptor that contains
all the headers etc. of a connection.

Correctly shutdown the servers when the process is terminating;
prevents a crash on exit. With debugging help from blambert@.

Implement file descriptor accounting. The concept was taken from
relayd but had to be adjusted for httpd. It now handles single-pass
HTTP connections, persistent connections with multiple requests, and
body-less HEAD requests. With input from benno@

first step towards keep-alive/persistent connections support

Finish writing the output before closing the connection
(adopted from relayd).

Close the connection after the response is completed (no Keepalive yet).

Add support for media types (aka. MIME types): the types section is
compatible to nginx' mime.types file which can be included directly.
If not present, use a few built-in defaults for html, css, txt, jpeg,
gif, png, and js.

Add httpd(8), an attempt to turn the relayd(8) codebase into a simple
web server. It is not finished yet and I just started it today, but
the goal is to provide an HTTP server that a) provides minimal
features, b) serves static files, c) provides FastCGI support, and d)
follows common coding practices of OpenBSD.

It will neither support plugins, nor custom memory allocators, EBCDIC
support, PCRE or any other things that can be found elsewhere.
httpd(8) is not intended to provide a fully-featured replacement for
nginx(8) or the Apache, but it will provide enough functionality that
is needed in the OpenBSD base system.

ok deraadt@

Remove unneeded calls to tls_init(3)

As per the manual and lib/libtls/tls.c revision 1.79 from 2018
"Automatically handle library initialisation for libtls." initialisation
is handled automatically by other tls_*(3) functions.

Remove explicit tls_init() calls from base to not give the impression of
it being needed.

Feedback tb
OK Tests mestre

Do not compare TLS config params for non-TLS servers. This allows to
mix 'listen * port 80' and 'listen * tls port 443' in one server block.
Also the last argument of server_tls_cmp - match_keypair - is always 0
so remove this code.
OK florian@ tb@ some long time ago

Pull tls_close() and tls_free() further up, so tls_free() will already
be in the right spot once tls_close() is handled by libevent.

suggested by jsing

Call tls_close() before closing the underlying socket

In order to end a TLS connection regularly, an implementation MUST send a
close_notify alert. libtls does this in tls_close() via SSL_shutdown(),
so the socket had better still be open.

The incorrect order in server_close() caused a leak on each tls connection
due to a bug in libssl (fixed in tls_record_layer.c r1.56).

As pointed out by claudio, tls_close() should really be handled from the
main event loop. This will be addressed in a later commit.

ok claudio florian jsing

Don't leak the log message in server_sendlog

While there, use the length calculated by vasprintf() instead of using
strlen needlessly.

ok claudio florian

Handle absence of TLS certs while parsing the config

There is a soft fail mechanism to handle missing certs for seamless
interaction with acme-client. Move this to the config parser. This is
simpler than server.c r1.117 and avoids a crash due to listening on
port 443 without having set up the TLS context first. More precisely,
the crash happens if a server with missing certificate is visited via
https in a configuration where there is a second server with valid
certificate and key.

From Joshua Sing (joshua at hypera dot dev)

ok benno

httpd(8) sent a 408 response every time a connection request timeout
was reached. This is not what other servers are doing, it leads to
ugly log messages and might confuse some clients.
benno@ analyzed that the correct behavior is (probably) to send a 408
when we are in the middle of receiving headers and time out there and
just close the connection in all other cases.
In particular, if a connection gets opened and no request is received
at all just close the connection. If a connection is set to keep-alive
and a request was handled and no further request is coming in just
close the connection. The later is the usual cause for spurious log
messages and client confusion.

Reported over the years by many.
Input, explanations and OK benno

When system calls indicate an error they return -1, not some arbitrary
value < 0. errno is only updated in this case. Change all (most?)
callers of syscalls to follow this better, and let's see if this strictness
helps us in the future.

httpd(8): add support for setting custom FastCGI parameters.

This commit extends the existing grammar by adding the param option
to the fastcgi directive: fastcgi param name value.

Example usage:
fastcgi param VAR1 hello
fastcgi param VAR2 world

With help and OK florian@
Rogue manpage bits, feel free to modify them.

Allow httpd(8) to start when TLS is configured but a cert is not yet
available.
Assuming a httpd.conf based on /etc/examples/httpd.conf, httpd(8)
will only listen on port 80 and serve the acme-challenge directory
for acme-client(1).
The workflow to get a certificate then becomes
acme-client -vAD example.com && rcctl reload httpd
Without the need to edit the httpd.conf yet again. Once the cert
is in place and httpd is reloaded it starts to serve on port 443.

Idea, tweaks & OK deraadt, OK benno

Backout my previous commit:

date: 2018/10/01 19:24:09; author: benno; state: Exp; lines: +7 -1;
commitid: 0O8fyHPNvPd8rvYU;
Only send 408 Timeout responses when we have seen at least part of a
request. Without a request, just close the connection when we hit
request timeout.
Prompted by a bug report from Nikola Kolev, thanks.
ok reyk@ and some suggestions from claudio@ and bluhm@

Mark Patruck (mark AT wrapped DOT cx) found a problem with it, thanks
for the report.

ok reyk@ bluhm@ sthen@ deraadt@

Only send 408 Timeout responses when we have seen at least part of a
request. Without a request, just close the connection when we hit
request timeout.
Prompted by a bug report from Nikola Kolev, thanks.
ok reyk@ and some suggestions from claudio@ and bluhm@

Add support for client certificate authentication to httpd.

From Jack Burton <jack at saosce dot com dot au> - thanks!

Also tested by Jan Klemkow <j.klemkow at wemelug dot de>.

ok beck@ reyk@

Don't do OCSP stapling only if the staple file is 0 length.

This allows something external (like ocspcheck) to disable the stapling
deliberatly if it can not retreive a valid staple by truncating the
staple file to indicate "do not provide a staple", while the file not
existin will still be treated as a configuration error
ok claudio@ florian@, and prompted by @jsing

Disable oscp stapling on invalid staple, rather than failing to start.
ok claudio@ florian@

Convert httpd to tls_config_set_ecdhecurves(), allowing a list of curves
to be specified, rather than a single curve.

ok beck@

Rework the way that TLS configuration is sent/received via imsgs, so that
are no longer limited by the 16KB maximum size of a single imsg.
Configuration data that is larger than a single message is now chunked and
sent via multiple imsgs.

Prompted by a diff from Jack Burton <jack at saosce dot com dot au>.

ok reyk@

some freezero() calls

Implement TLS ticket support in httpd. Off by default. Use
tls ticket lifetime default
to turn it on with a 2h ticket lifetime.
Rekeying happens after a quarter of that time.
OK reky@ and bob@

/tmp/cvsspEkok

Fix support for HTTP pipelining by handling all requests in the buffer.

Tested & OK jung@

remove extra call setting OCSP staple now that it is done above
using keypair_ocsp..
ok reyk@

Correct mistake I made when converting this to new funciton

Add tls_config_[add|set]keypair_ocsp functions so that ocsp staples may be
added associated to a keypair used for SNI, and are usable for more than
just the "main" certificate. Modify httpd to use this.
Bump libtls minor.

ok jsing@

Do not set EVBUFFER_EOF on read/write errors and handle EOF correctly.

Either libevent or the TLS callback can trigger an EOF when the
connection is closed.

OK sunil@ jung@ benno@

Stop accessing verbose and debug variables from log.c directly.

This replaces log_verbose() and "extern int verbose" with the two functions
log_setverbose() and log_getverbose().

Pointed out by benno@
OK krw@ eric@ gilles@ (OK gilles@ for the snmpd bits as well)

Check the return value of tls_config_set_protocols(), now that it returns
an int.

Move OCSP loading into a separate function - it is not part of the keypair
and this way we can give a separate specific error message.

ok beck@ reyk@

Fix tcp ip ttl / minttl on IPv6 sockets.

ok florian@

conditionalize ocsp load properly
ok jsing@