Avoid a NULL dereference when handling a malformed fastcgi request.

Rework the hack to avoid a use-after-free in the fastcgi code.
Since server_fcgi() can be called by server_read_httpcontent() we
can't set clt_fcgi_error to NULL. Instead, we implement a simple
reference count to track when a fastcgi session is in progress to
avoid closing the http session prematurely on fastcgi error.
Based on a diff from and OK by tb@. Reported by Ben Kallus.

Work around use after free in httpd(8)

A malformed HTTP request can cause httpd in fastcgi mode to crash due to a
use-after-free. This is an awful hack, but it's good enough until someone
figures out the correct way of dealing with server_close() here.

"this will do the trick for now" claudio
ok beck deraadt

branches: 1.162.4;
remove unused references to httpd.sock; found by dante catalfamo
ok florian

branches: 1.161.2;
Neither clt_descreq nor clt_descresp in struct client need to be void *.
They both are only used as struct http_descriptor.
OK tb@

Nothing uses kv_flags.

John (j AT bitminer.ca) pointed out that we didn't correctly
initialize struct kv and might use slower KV_FLAG_GLOBBING path in
kv_find depending on stack garbage. Instead of fixing the
initialization just delete kv_flags from struct kv.

OK claudio, tb

Add gzip-static option to httpd. This allows to deliver precompressed
files with content-encoding gzip.
from prx at si3t dot ch; OK tracey@

Add httpd custom error page facility. Adapted by me from
https://github.com/mpfr/httpd-plus.
Improvements from & (earlier version) reads fine to tracey@;
improvements & OK this version benno@, florian@. Thanks.

Do not try to chunk encode an empty http body coming from an fcgi
upstream.

Found the hard way by Chris Narkiewicz who tracked failing uploads in
the nextcloud mobile app down to httpd(8) trying to chunk encode a
"204 No Content" resonse.

Testing by Steve Williams
Testing & OK stsp

Move TAILQ initialization to files where they are used.

These priv-sep daemons all follow a similar design and use TAILQs
for tracking control process connections. In most cases, the TAILQs
are initialized separate from where they are used. Since the scope
of use is generally confined to a specific control process file,
this commit also removes any extern definitions and exposing the
TAILQ structures to other compilation units.

ok bluhm@, tb@

Do not compare TLS config params for non-TLS servers. This allows to
mix 'listen * port 80' and 'listen * tls port 443' in one server block.
Also the last argument of server_tls_cmp - match_keypair - is always 0
so remove this code.
OK florian@ tb@ some long time ago

these programs (with common ancestry) had a -fno-common problem related
to privsep_procid.
ok mortimer

Add location (found|not found) option to test for resource path existence.

Diff by Matthias Pressfreund <mpfr @ fn de>. Thanks.

typo; pointed out by Matthias (mpfr AT fn.de), thanks!

Set fastcgi socket default on server and location.
This allows "fastcgi" directly inside of a server directive without
giving specifying socket.
OK tracey

Add support for non-localhost fastcgi sockets.

Lots of review time kn@
Lots of review time, tweaks, and ok florian@

remove unused functions
from Ross L Richardson <openbsd AT rlr DOT id DOT au>, Thanks
ok claudio@

Remove the unused function canonicalize_host(), it was copied from relayd.
Found by Ross L Richardson, Thanks.

remove unsused enum key_type, copied initialy from relays.
found by Ross L Richardson, thanks!
ok deraadt@

Implement "strip" option for fastcgi to be able to have multiple chroots
under /var/www for FastCGI servers.
From Nazar Zhuk (nazar AT zhuk DOT online), thanks!
Ok benno

spacing

Add a "forwarded" log format that extends the "combined" log format
by including the contents of the X-Forwarded-For and X-Forwarded-Port
headers. If httpd(8) runs behind a proxy like relayd(8), this allows
tracking the origin of the requests. The format is compatible with
log analyzers such as GoAccess and Webalizer.

Patch from Bruno Flueckiger

ok benno, reyk

httpd(8): add support for setting custom FastCGI parameters.

This commit extends the existing grammar by adding the param option
to the fastcgi directive: fastcgi param name value.

Example usage:
fastcgi param VAR1 hello
fastcgi param VAR2 world

With help and OK florian@
Rogue manpage bits, feel free to modify them.

Backout my previous commit:

date: 2018/10/01 19:24:09; author: benno; state: Exp; lines: +7 -1;
commitid: 0O8fyHPNvPd8rvYU;
Only send 408 Timeout responses when we have seen at least part of a
request. Without a request, just close the connection when we hit
request timeout.
Prompted by a bug report from Nikola Kolev, thanks.
ok reyk@ and some suggestions from claudio@ and bluhm@

Mark Patruck (mark AT wrapped DOT cx) found a problem with it, thanks
for the report.

ok reyk@ bluhm@ sthen@ deraadt@

Only send 408 Timeout responses when we have seen at least part of a
request. Without a request, just close the connection when we hit
request timeout.
Prompted by a bug report from Nikola Kolev, thanks.
ok reyk@ and some suggestions from claudio@ and bluhm@

During the fork+exec implementation, daemon(3) was moved after
proc_init(). As a consequence httpd(8) and relayd(8) child processes
did not detach from the terminal anymore. Dup /dev/null to the
stdio file descriptors in the children.
OK benno@

double the allowed length for the 'tls ciphers' option

for example now it can hold the recommended cipher list from the mozilla
ssl config generator rather than failing with a "ciphers too long" error.

ok benno@ sthen@ tb@

Add support for simple one-off internal rewrites.

For example:

location match "/page/(%d+)/.*" {
request rewrite "/static/index.php?id=%1&$QUERY_STRING"
}

Requested by many.

Ok benno@

Add support for client certificate authentication to httpd.

From Jack Burton <jack at saosce dot com dot au> - thanks!

Also tested by Jan Klemkow <j.klemkow at wemelug dot de>.

ok beck@ reyk@

Make httpd listen on all IPv4 and IPv6 addresses for
"listen on * port 80".
While here accept up to 16 addresses from DNS or interface groups.

requested by & "lovely" deraadt@
OK kn@

NAME_MAX is the length of the thing between / / in a path *without*
the terminating NUL.
Do not use it for a "small string" or a "probably short path". Replace
it with new defines or PATH_MAX.
It also makes the life easier for people auditing the tree for real
usage of NAME_MAX.
OK deraadt, benno

Convert httpd to tls_config_set_ecdhecurves(), allowing a list of curves
to be specified, rather than a single curve.

ok beck@

Rework the way that TLS configuration is sent/received via imsgs, so that
are no longer limited by the 16KB maximum size of a single imsg.
Configuration data that is larger than a single message is now chunked and
sent via multiple imsgs.

Prompted by a diff from Jack Burton <jack at saosce dot com dot au>.

ok reyk@

use __func__ in log messages.
From Hiltjo Posthuma hiltjo -AT codemadness -DOT- org, thanks!
ok florian, claudio

Implement TLS ticket support in httpd. Off by default. Use
tls ticket lifetime default
to turn it on with a 2h ticket lifetime.
Rekeying happens after a quarter of that time.
OK reky@ and bob@

/tmp/cvsspEkok

Stop assuming that in_{addr,port}_t are typedefed in <sys/types.h> and
instead pull in <netinet/in.h> or <arpa/inet.h> when those are needed.

ok florian@ beck@ millert@

Fix support for HTTP pipelining by handling all requests in the buffer.

Tested & OK jung@

Reimplement httpd's support for byte ranges.

The previous implementation loaded all the output into a single output
buffer and used its size to determine the Content-Length of the body.

The new implementation calculates the body length first and writes the
individual ranges in an async way using the bufferevent mechanism.

This prevents httpd from using too much memory and applies the
watermark and throttling mechanisms to range requests.

Problem reported by Pierre Kim (pierre.kim.sec at gmail.com)

OK benno@ sunil@

The variable clt_done is used in too many places.

Introduce a new variable clt_headersdone in the async HTTP parser.

OK sunil@ benno@

Stop accessing verbose and debug variables from log.c directly.

This replaces log_verbose() and "extern int verbose" with the two functions
log_setverbose() and log_getverbose().

Pointed out by benno@
OK krw@ eric@ gilles@ (OK gilles@ for the snmpd bits as well)

Move OCSP loading into a separate function - it is not part of the keypair
and this way we can give a separate specific error message.

ok beck@ reyk@

Add OCSP stapling support to httpd
ok jsing@ bcook@

Prevent fd exhaustion in the parent when loading the listening server
sockets by sending the fd one-by-one. This allows to start httpd with
max 32 server instances and many server sockets without changing the
default rlimits in any way.

OK rzalamena@

sync proc.c with vmd: add p_pw to specify a non-standard user for a process.

OK rzalamena@

sync proc.c from switchd, includes minor cast qual fix and removal of p_env.

Replace [RELAY|SERVER]_MAXPROC with the new PROC_MAX_INSTANCES
variable and limit it from 128 to 32 instances (the old value).
While here, move a few PROC_ defines around.

OK rzalamena@

proc.c tweaks: Rename proc_listento() to proc_accept() as it is the
receiving side of proc_connect(). Move some code from main into
proc_init(), the function is now called by parent and children, not
just the parent and it is less copy + paste for other daemons.

OK florian@

struct client starts to become the kitchen sink. Move fastcgi data to
its own struct. Requested by and OK reyk@

spacing

Teach httpd/proc.c how to fork+exec.

This commit implemented the basic functions to proc.c to make it not rely
on global variables, malloc()ed memory and CLOEXEC pipes.

Fix child proc titles from reyk@
ok reyk@, florian@

Kill (remove) the ps_pid from privsep struct since it is not being used
anymore. Also fix the process initialization prototypes.

ok reyk@

Remove duplicated prototypes from header.

"Looks good to me" natano@

Do not assume that the full http response header is in the first
fastcgi stdout record. Keep processing stdout records until we found
the header / body separator and only then generate the header
response.
Problem reported by many.

OK jung@

Kill p_instance from proc.c and remove static proc_id unused variables.

To keep the debug functionality intact and correct we'll use the pid
field in the imsg header to pass the instance number. Remember to always
pass 'ps_instance + 1' otherwise libutil will fill imsg header pid field
with the imsgbuf pid (which is the current process pid).

ok reyk@

Kill the ps_ninstances from proc.c.

We got the same information in ps_instances[proc] (more accurate) and
we avoid allocating unnecessary memory for pipe storage.

ok reyk@

Replace the static env variables with a single global variable.

ok reyk@

Enable SNI support in httpd(8).

ok reyk@

stop including sys/param.h for nitems. define locally as needed.
ok natano reyk

Move server_match() from parse.y to server.c; use env instead of conf,
which is actually the same thing (cluebat from reyk@).

Make httpd stricter with respect to TLS configuration - in particular, do
not allow TLS and non-TLS to be configured on the same port, do not allow
TLS options to be specified without a TLS listener and ensure that the TLS
options are the same when a server is specified on the same address/port.
Currently, these configurations are permitted but do not work as intended.

Also factor out and reuse the server matching code, which was previously
duplicated.

ok reyk@

Adjust existing tls_config_set_cipher() callers for TLS cipher group
changes - map the previous configuration to the equivalent in the new
groups. This will be revisited post release.

Discussed with beck@

Simplify TLS configuration handling. Instead of matching by address/port,
match by configuration ID. This also prevents a memory leak when there are
multiple certificates specified for the same server.

ok beck@

sync with relayd, use proc_compose()

Retire socket_set_blockmode() in favor of the SOCK_NONBLOCK type flag.
As done in iked and snmpd.

OK jung@

Update log.c: change fatal() and fatalx() into variadic functions,
include the process name, and replace all calls of fatal*(NULL) with
fatal(__func__) for better debugging.

OK benno@

Move local logging functions into httpd.c, and sync log.c with relayd -
both daemons are now sharing the same file. No functional changes.

Fix a regression that was introduced with server.c r1.64: Do NOT free
srv_conf->auth in serverconfig_free() because it was not allocated in
config_getserver() but assigned as a reference by id from a global
list that is maintained independently. This fixes a potential
double-free. This fix also makes srv_conf->auth "const" to emphasize
that the read-only auth pointer was not allocated here.

OK jsing@

Change httpd(8) to use C99-style fixed-width integers (uintN_t instead
of u_intN_t) and replace u_int with unsigned int. Mixing both
variants is a bad style and most contributors seem to prefer this
style; it also helps us to get used to it, portability, and
standardization.

Theoretically no binary change, except one in practice: httpd.o has a
different checksum because gcc with -O2 pads/optimizes "struct
privsep" differently when using "unsigned int" instead "u_int" for the
affected members. "u_int" is just a typedef of "unsigned int", -O0
doesn't build the difference and clang with -O2 doesn't do it either -
it is just another curiosity from gcc-land.

OK semarie@

Fix rev 1.70 of server.c by only re-enabling the bufferevent if we
previously disabled it because we were reading to fast (from disk).
Problem noted and tracked down to that commit by weerd@ and
independently by stsp@.
Tested by weerd@, stsp@, reyk@
OK bluhm@, reyk@

repair hsts header output, wrong format strings caused broken
Strict-Transport-Security headers. Add __format__ attribute to
kv_set() and kv_setkey() to make it easier to spot such problems.

Found by and fix from Donovan Watteau <tsoomi -AT- gmail -DOT- com>,
thanks for your help.

ok deraadt@

backout the previous: it broke wordpress somehow.
we need more care to find a proper fix for the fastcgi headers.

acknowledged by deraadt@

Read fcgi response records until we have the whole http header and can
parse it. Otherwise http headers can leak into the body.
Pointed out by Jean-Philippe Ouellet on bugs@ Thanks!
OK reyk, commit ASAP deraadt@

For the completeness of HSTS, add the non-standard preload option.

OK florian@

libtls has been changed to set SSL_MODE_ENABLE_PARTIAL_WRITE and
SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER by default. This gives
tls_write() a similar short write semantics as write(2) and a
workaround in httpd to cope with the previous differences can be
removed. Specifically, httpd can stop copying data into a local
buffer that was used to keep it around for repeated writes.

OK bluhm@

Allow to change the default media type globally or per-location,
eg. default type text/html.

OK florian@

Implement HTTP Strict Transport Security (HSTS).
Input & OK reyk

If we can read faster from disk than send data to the client stop
reading from disk when we hold a certain amount of data in
RAM. Re-enable reading once we send enough data to the
client. Otherwise we might end up with the whole file (which can be
huge) in RAM.
Reported by Matthew Martin ( matt.a.martin AT gmail ) on bugs@,
thanks!
OK reyk@, benno@

VIS_QUOTE is not there yet, unbreak the tree. Noticed by semarie@

Escape the message in server_log() as well.

OK benno@

Send the TLS certificate and key via separate imsgs, rather than
including them in the IMSG_CFG_SERVER imsg. This allows the certificate
and key to each be almost 16KB (the maximum size for an imsg), rather than
having a combined total of less than 16KB (which can be reached with large
keys, certificate bundles or by including text versions of certificates).

ok reyk@

Add initial support for pattern matching using Lua's pattern matching code.

With important help on the pattern matcher from semarie@

OK semarie@

Use off_t instead of size_t to pass file size and print it using %lld when
constructing the Content-Length header field. Should fix some, but probably
not all, problems with serving files bigger than 2G on 32-bit architectures.

ok reyk@, florian@

Prevent use after free.
While here unconditionally free clt and move declaration of
server_inflight_dec() into server.c
Found while investigating if (foo != NULL) free(foo) patterns pointed
out by Markus Elfring.
OK reyk

Allow to specify CGI variables as macros in redirection strings, eg.
block return 301 "http://www.example.com/$REQUEST_URI"

OK tedu@ florian@

Rename escape_uri() to url_encode() because it is the opposite of
url_decode(). No functional change.

Allow TLS protocols to be specified via a "tls protocols" configuration
option.

ok reyk@

Encode directory listings.
Problem pointed out by remco AT d-compu.dyndns.org some time ago.
Input / OK reyk@

Remove server_load_file() in favor of tls_load_file(3)

Add httpd configuration options to allow the specification of DHE
parameters and the ECDHE curve. This primarily allows for DHE cipher suites
to be enabled.

ok reyk@

Add support for blocking, dropping, and redirecting requests.

OK florian@

Fix log options in locations.

Reported and tested by Markus Bergkvist
OK florian@

Ooops, no need to include sys/cdefs.h.

Pointed out by florian@

httpd is based on relayd and had included many headers that are only
needed by its ancestor. jsg@, include-what-you-use, and some manual
review helped to cleanup the headers (take iwyu with a grain of salt).
Based on common practice, httpd.h now also includes the necessary
headers for itself.

OK florian@

s/clt_fcgi_remote_user/clt_remote_user/
OK reyk@

Decouple auth parameters from struct server_config into struct auth.

OK florian@

First stab at implementing basic auth.
Currently the htpasswd file needs to be in the chroot; will hopefully
improved soonish.
Based on a diff from Oscar Linderholm many months ago but turned into
a complete rewrite.
input/OK reyk@

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

bump copyright year

add new url stripping option:

strip number
Strip number path components from the beginning of the
request URI before looking up the stripped-down URI at
the document root.


reviewed with much patience and OK by reyk@

Bump config flags field to 32bits. Makes room for future changes -
but no functional change yet.

Like previously done in relayd, change the keyword "ssl" to "tls" to
reflect reality.

OK benno@

Allow the log directory to be configurable in the config file, rather than
fixed as /logs within the chroot. As this httpd is properly privesp'ed this
has the nice property of allowing us to put the logs outside the chroot if
we want to.
ok reyk@

Convert the logic in yyerror(). Instead of creating a temporary
format string, create a temporary message.
OK deraadt@

Update httpd(8) to use libtls instead of libressl.

URL-decode the request path.

Tested by ajacoutot@ and others
OK doug@

Handle different possible variations of the Host header (eg.
www.example.com, www.example.com:80, [2001:db8::1], [2001:db8::1]:80).
The port is optional and is typically used on non-default ports. If
the server name is a plain IPv6 address, it is commonly specified in
square brackets.

Makes ajacoutot@ happy
OK florian@

Remove a limitation that only allowed to specify a server name once.
The key has been changed to server name + address + port and now it is
possible to use the same server name for multiple servers with
different addresses, eg. http://www.example.com and
https://www.example.com/.

OK doug@ florian@

FastCGI did not support persistent connections. Add initial support
for persistent connections with FastCGI by implementing chunked
Transfer-Encoding. This only works with HTTP/1.1.

With input and help from florian@ who found some FastCGI edge cases.

OK florian@

Replace the code to get the FastCGI Status header with a proper way to
parse and write the headers using the http response descriptor. This
allows to add other tweaks, like support for chunked encoding, later.

OK florian@

Use two instead of one http descriptor for request and response.

OK chrisz@

Add Last-Modified: HTTP header.

OK reyk@

Provide a failsafe version of the path_info() function that doesn't
need a temporary path variable. Based on an initial diff from
chrisz@.

"Commit any failsafe version and I'm ok with it" chrisz@

When opening directories, re-match the location after the index file
has been appended. This allows to use a fastcgi target as the default
index, for example index.php.

OK florian@

branches: 1.51.2;
Limit the body size in client requests (eg. POST data) to 1M by default;
add a configuration option to change the limit.

ok florian@

Configure the default SSL ciphers as HIGH:!aNULL.

ok deraadt@ reyk@

http POST support
with & OK reyk@

Adjust the read/write watermarks according to the TCP send buffer.
This fixes sending of large files. Previously, httpd was reading the
input file too quickly and could run out of memory when filling the
input buffer.

Found by jsg@
OK florian@

Load the SSL public/private keys in the parent process, then provide them
to the privsep process via imsg. This allows the keys to be moved out of
the chroot (now /etc/ssl/server.crt, /etc/ssl/private/server.key).

ok reyk@

Add configuration options for the most-important connection limits:
max requests (per connection) and timeout. We don't want to add too
many button, and there are good defaults, but these ones are kind of
mandatory.

Limit the number of (Keep-Alive) requests per connection to 100.
(Same default as in nginx and Apache).

Improve logging to allow per- server/location log files. The log
files can also be owned by root now: they're opened by the parent and
send to the logger process with fd passing. This also works with reload.

ok deraadt@

add a config option to specify the chroot directory
ok reyk@

Temporarily move the default location of the SSL/TLS server key and
certificate from /var/www/ to /var/www/conf/. Don't get scared - this
will be changed soon! They're currently located in the chroot
directory but will be moved outside as soon as we adopted some of the
key privsep from relayd in ressl/httpd.

Proxy commit for jsing@:
"Add TLS/SSL support to httpd, based on the recent ressl commits."

From jsing@
ok reyk@

Add initial support for log files in /var/www/logs/. Logging with
syslog is still supported but disabled by default.

ok deraadt@

Implement PATH_INFO and add DOCUMENT_ROOT.
PATH_INFO was requested by naddy@ who successfully tested it with "cvsweb".

ok naddy@

whitespace

Dynamically pass HTTP request headers as protocol-specific HTTP_* CGI
meta-variables.

ok florian@

Add function to iterate all headers. No functional change.

Split fastcgi socket path and document root option and add the
SCRIPT_FILENAME CGI param with a prepended root. This fixes php-fpm
that expects SCRIPT_FILENAME and also works with slowcgi if you
configure the root correctly. For example, if SCRIPT_NAME and
REQUEST_URI are /php/index.php, root is /htdocs, SCRIPT_FILENAME will
be /htdocs/php/index.php. As tested and discussed with florian@

Add another log mode "connection" for a relayd(8)-style log entry after
each connection, not every request. The code was already there and enabled
on debug, I just turned it into an alternative log format.

Locations now inherit access log settings from the server.

Add log to the server flags.

input/"Looks ok" reyk@

Padding of fcgi records is optional, but if we receive padding data we
should read it.

Allow to specify a FastCGI TCP socket on localhost (eg. :9000). Used
for debugging, you should prefer local UNIX sockets, but it helped to
find an issue that will be fixed with the next commit.

OK florian@

Use the log buffer to defer the logging until the connection is closed
or the request completed. Turn the old log message into a debug message.

ok doug@

remove the global "log updates/all" option that came from relayd.

Add common and combined access logging to httpd.

ok reyk@

Correctly parse fcgi records if we don't get the whole record in one
bufferevent_read().
Input/OK reyk@

Only write the HTTP header for the first fastcgi chunk.

some fastcgi improvements:
- DPRINTF instead of log_info for internal debugging.
- submit QUERY_STRING, if it exists
- use a proper function to create an HTTP header.
- use server_file_error() to detect EOF and fastcgi stream errors.
- disable keep-alive/persist for now until we have a reliable way to
get the content length from the cgi response or support chunked
encoding.

"Cool, jep" florian@

One bufferevent can be shared by file and fcgi.

Allow to specify a non-default fastcgi socket.

Rename the "docroot" variable to "path" because it will be used for
either files or the fastcgi socket (and there's no need to use a union yet).

Add a configuration variable "fastcgi" to enable it per server or location.

Put in first stab at fastcgi. Very early work in progress. Putting it
in now so that we can quickly work on it in tree. Requested by reyk@.
deraadt@ is OK with this according to reyk@.

Make "location" work with name-based virtual servers.

Add "location" keyword to specify path-specific configuration in
servers, for example auto index for a sub-directory only. Internally,
a "location" is just a special type of a "virtual" server.

Add extended directory index options: "[no] index" and "[no] auto index".
The option "directory auto index" implements basic directory listing
and is turned off by default.

ok deraadt@

Move configurable TCP options into struct server_config.

Differentiate servers by address and port, not just by address.

It is recommended to use a URL in the Location header of 3xx
responses. To accomplish this, add some semantics to retrieve the
server host name of a connection: either IP, IP:PORT (if not 80) or
[IP6]:PORT, or Host value (if valid).

Canonicalize the request path once without the docroot and prepend the
docroot only only when it's needed. Suggested by deraadt@.

Add support for "virtual hosts" aka. server blocks aka. multiple
servers with the same or "overlapping" IP address but a different name.

ok beck@

Split server and server_config.

Rename a field, needed later, no functional change.

Move the docroot into the server block.

Remove unused fields from structure

Add canonicalize_path() to canonicalize the requested URL path.

Correctly shutdown the servers when the process is terminating;
prevents a crash on exit. With debugging help from blambert@.

always enable DPRINTF with compiled with DEBUG

Implement file descriptor accounting. The concept was taken from
relayd but had to be adjusted for httpd. It now handles single-pass
HTTP connections, persistent connections with multiple requests, and
body-less HEAD requests. With input from benno@

first step towards keep-alive/persistent connections support

Add support for media types (aka. MIME types): the types section is
compatible to nginx' mime.types file which can be included directly.
If not present, use a few built-in defaults for html, css, txt, jpeg,
gif, png, and js.

Add httpd(8), an attempt to turn the relayd(8) codebase into a simple
web server. It is not finished yet and I just started it today, but
the goal is to provide an HTTP server that a) provides minimal
features, b) serves static files, c) provides FastCGI support, and d)
follows common coding practices of OpenBSD.

It will neither support plugins, nor custom memory allocators, EBCDIC
support, PCRE or any other things that can be found elsewhere.
httpd(8) is not intended to provide a fully-featured replacement for
nginx(8) or the Apache, but it will provide enough functionality that
is needed in the OpenBSD base system.

ok deraadt@

Work around use after free in httpd(8)

A malformed HTTP request can cause httpd in fastcgi mode to crash due to a
use-after-free. This is an awful hack, but it's good enough until someone
figures out the correct way of dealing with server_close() here.

"this will do the trick for now" claudio
ok beck deraadt

branches: 1.162.4;
remove unused references to httpd.sock; found by dante catalfamo
ok florian

branches: 1.161.2;
Neither clt_descreq nor clt_descresp in struct client need to be void *.
They both are only used as struct http_descriptor.
OK tb@

Nothing uses kv_flags.

John (j AT bitminer.ca) pointed out that we didn't correctly
initialize struct kv and might use slower KV_FLAG_GLOBBING path in
kv_find depending on stack garbage. Instead of fixing the
initialization just delete kv_flags from struct kv.

OK claudio, tb

Add gzip-static option to httpd. This allows to deliver precompressed
files with content-encoding gzip.
from prx at si3t dot ch; OK tracey@

Add httpd custom error page facility. Adapted by me from
https://github.com/mpfr/httpd-plus.
Improvements from & (earlier version) reads fine to tracey@;
improvements & OK this version benno@, florian@. Thanks.

Do not try to chunk encode an empty http body coming from an fcgi
upstream.

Found the hard way by Chris Narkiewicz who tracked failing uploads in
the nextcloud mobile app down to httpd(8) trying to chunk encode a
"204 No Content" resonse.

Testing by Steve Williams
Testing & OK stsp

Move TAILQ initialization to files where they are used.

These priv-sep daemons all follow a similar design and use TAILQs
for tracking control process connections. In most cases, the TAILQs
are initialized separate from where they are used. Since the scope
of use is generally confined to a specific control process file,
this commit also removes any extern definitions and exposing the
TAILQ structures to other compilation units.

ok bluhm@, tb@

Do not compare TLS config params for non-TLS servers. This allows to
mix 'listen * port 80' and 'listen * tls port 443' in one server block.
Also the last argument of server_tls_cmp - match_keypair - is always 0
so remove this code.
OK florian@ tb@ some long time ago

these programs (with common ancestry) had a -fno-common problem related
to privsep_procid.
ok mortimer

Add location (found|not found) option to test for resource path existence.

Diff by Matthias Pressfreund <mpfr @ fn de>. Thanks.

typo; pointed out by Matthias (mpfr AT fn.de), thanks!

Set fastcgi socket default on server and location.
This allows "fastcgi" directly inside of a server directive without
giving specifying socket.
OK tracey

Add support for non-localhost fastcgi sockets.

Lots of review time kn@
Lots of review time, tweaks, and ok florian@

remove unused functions
from Ross L Richardson <openbsd AT rlr DOT id DOT au>, Thanks
ok claudio@

Remove the unused function canonicalize_host(), it was copied from relayd.
Found by Ross L Richardson, Thanks.

remove unsused enum key_type, copied initialy from relays.
found by Ross L Richardson, thanks!
ok deraadt@

Implement "strip" option for fastcgi to be able to have multiple chroots
under /var/www for FastCGI servers.
From Nazar Zhuk (nazar AT zhuk DOT online), thanks!
Ok benno

spacing

Add a "forwarded" log format that extends the "combined" log format
by including the contents of the X-Forwarded-For and X-Forwarded-Port
headers. If httpd(8) runs behind a proxy like relayd(8), this allows
tracking the origin of the requests. The format is compatible with
log analyzers such as GoAccess and Webalizer.

Patch from Bruno Flueckiger

ok benno, reyk

httpd(8): add support for setting custom FastCGI parameters.

This commit extends the existing grammar by adding the param option
to the fastcgi directive: fastcgi param name value.

Example usage:
fastcgi param VAR1 hello
fastcgi param VAR2 world

With help and OK florian@
Rogue manpage bits, feel free to modify them.

Backout my previous commit:

date: 2018/10/01 19:24:09; author: benno; state: Exp; lines: +7 -1;
commitid: 0O8fyHPNvPd8rvYU;
Only send 408 Timeout responses when we have seen at least part of a
request. Without a request, just close the connection when we hit
request timeout.
Prompted by a bug report from Nikola Kolev, thanks.
ok reyk@ and some suggestions from claudio@ and bluhm@

Mark Patruck (mark AT wrapped DOT cx) found a problem with it, thanks
for the report.

ok reyk@ bluhm@ sthen@ deraadt@

Only send 408 Timeout responses when we have seen at least part of a
request. Without a request, just close the connection when we hit
request timeout.
Prompted by a bug report from Nikola Kolev, thanks.
ok reyk@ and some suggestions from claudio@ and bluhm@

During the fork+exec implementation, daemon(3) was moved after
proc_init(). As a consequence httpd(8) and relayd(8) child processes
did not detach from the terminal anymore. Dup /dev/null to the
stdio file descriptors in the children.
OK benno@

double the allowed length for the 'tls ciphers' option

for example now it can hold the recommended cipher list from the mozilla
ssl config generator rather than failing with a "ciphers too long" error.

ok benno@ sthen@ tb@

Add support for simple one-off internal rewrites.

For example:

location match "/page/(%d+)/.*" {
request rewrite "/static/index.php?id=%1&$QUERY_STRING"
}

Requested by many.

Ok benno@

Add support for client certificate authentication to httpd.

From Jack Burton <jack at saosce dot com dot au> - thanks!

Also tested by Jan Klemkow <j.klemkow at wemelug dot de>.

ok beck@ reyk@

Make httpd listen on all IPv4 and IPv6 addresses for
"listen on * port 80".
While here accept up to 16 addresses from DNS or interface groups.

requested by & "lovely" deraadt@
OK kn@

NAME_MAX is the length of the thing between / / in a path *without*
the terminating NUL.
Do not use it for a "small string" or a "probably short path". Replace
it with new defines or PATH_MAX.
It also makes the life easier for people auditing the tree for real
usage of NAME_MAX.
OK deraadt, benno

Convert httpd to tls_config_set_ecdhecurves(), allowing a list of curves
to be specified, rather than a single curve.

ok beck@

Rework the way that TLS configuration is sent/received via imsgs, so that
are no longer limited by the 16KB maximum size of a single imsg.
Configuration data that is larger than a single message is now chunked and
sent via multiple imsgs.

Prompted by a diff from Jack Burton <jack at saosce dot com dot au>.

ok reyk@

use __func__ in log messages.
From Hiltjo Posthuma hiltjo -AT codemadness -DOT- org, thanks!
ok florian, claudio

Implement TLS ticket support in httpd. Off by default. Use
tls ticket lifetime default
to turn it on with a 2h ticket lifetime.
Rekeying happens after a quarter of that time.
OK reky@ and bob@

/tmp/cvsspEkok

Stop assuming that in_{addr,port}_t are typedefed in <sys/types.h> and
instead pull in <netinet/in.h> or <arpa/inet.h> when those are needed.

ok florian@ beck@ millert@

Fix support for HTTP pipelining by handling all requests in the buffer.

Tested & OK jung@

Reimplement httpd's support for byte ranges.

The previous implementation loaded all the output into a single output
buffer and used its size to determine the Content-Length of the body.

The new implementation calculates the body length first and writes the
individual ranges in an async way using the bufferevent mechanism.

This prevents httpd from using too much memory and applies the
watermark and throttling mechanisms to range requests.

Problem reported by Pierre Kim (pierre.kim.sec at gmail.com)

OK benno@ sunil@

The variable clt_done is used in too many places.

Introduce a new variable clt_headersdone in the async HTTP parser.

OK sunil@ benno@

Stop accessing verbose and debug variables from log.c directly.

This replaces log_verbose() and "extern int verbose" with the two functions
log_setverbose() and log_getverbose().

Pointed out by benno@
OK krw@ eric@ gilles@ (OK gilles@ for the snmpd bits as well)

Move OCSP loading into a separate function - it is not part of the keypair
and this way we can give a separate specific error message.

ok beck@ reyk@

Add OCSP stapling support to httpd
ok jsing@ bcook@

Prevent fd exhaustion in the parent when loading the listening server
sockets by sending the fd one-by-one. This allows to start httpd with
max 32 server instances and many server sockets without changing the
default rlimits in any way.

OK rzalamena@

sync proc.c with vmd: add p_pw to specify a non-standard user for a process.

OK rzalamena@

sync proc.c from switchd, includes minor cast qual fix and removal of p_env.

Replace [RELAY|SERVER]_MAXPROC with the new PROC_MAX_INSTANCES
variable and limit it from 128 to 32 instances (the old value).
While here, move a few PROC_ defines around.

OK rzalamena@

proc.c tweaks: Rename proc_listento() to proc_accept() as it is the
receiving side of proc_connect(). Move some code from main into
proc_init(), the function is now called by parent and children, not
just the parent and it is less copy + paste for other daemons.

OK florian@

struct client starts to become the kitchen sink. Move fastcgi data to
its own struct. Requested by and OK reyk@

spacing

Teach httpd/proc.c how to fork+exec.

This commit implemented the basic functions to proc.c to make it not rely
on global variables, malloc()ed memory and CLOEXEC pipes.

Fix child proc titles from reyk@
ok reyk@, florian@

Kill (remove) the ps_pid from privsep struct since it is not being used
anymore. Also fix the process initialization prototypes.

ok reyk@

Remove duplicated prototypes from header.

"Looks good to me" natano@

Do not assume that the full http response header is in the first
fastcgi stdout record. Keep processing stdout records until we found
the header / body separator and only then generate the header
response.
Problem reported by many.

OK jung@

Kill p_instance from proc.c and remove static proc_id unused variables.

To keep the debug functionality intact and correct we'll use the pid
field in the imsg header to pass the instance number. Remember to always
pass 'ps_instance + 1' otherwise libutil will fill imsg header pid field
with the imsgbuf pid (which is the current process pid).

ok reyk@

Kill the ps_ninstances from proc.c.

We got the same information in ps_instances[proc] (more accurate) and
we avoid allocating unnecessary memory for pipe storage.

ok reyk@

Replace the static env variables with a single global variable.

ok reyk@

Enable SNI support in httpd(8).

ok reyk@

stop including sys/param.h for nitems. define locally as needed.
ok natano reyk

Move server_match() from parse.y to server.c; use env instead of conf,
which is actually the same thing (cluebat from reyk@).

Make httpd stricter with respect to TLS configuration - in particular, do
not allow TLS and non-TLS to be configured on the same port, do not allow
TLS options to be specified without a TLS listener and ensure that the TLS
options are the same when a server is specified on the same address/port.
Currently, these configurations are permitted but do not work as intended.

Also factor out and reuse the server matching code, which was previously
duplicated.

ok reyk@

Adjust existing tls_config_set_cipher() callers for TLS cipher group
changes - map the previous configuration to the equivalent in the new
groups. This will be revisited post release.

Discussed with beck@

Simplify TLS configuration handling. Instead of matching by address/port,
match by configuration ID. This also prevents a memory leak when there are
multiple certificates specified for the same server.

ok beck@

sync with relayd, use proc_compose()

Retire socket_set_blockmode() in favor of the SOCK_NONBLOCK type flag.
As done in iked and snmpd.

OK jung@

Update log.c: change fatal() and fatalx() into variadic functions,
include the process name, and replace all calls of fatal*(NULL) with
fatal(__func__) for better debugging.

OK benno@

Move local logging functions into httpd.c, and sync log.c with relayd -
both daemons are now sharing the same file. No functional changes.

Fix a regression that was introduced with server.c r1.64: Do NOT free
srv_conf->auth in serverconfig_free() because it was not allocated in
config_getserver() but assigned as a reference by id from a global
list that is maintained independently. This fixes a potential
double-free. This fix also makes srv_conf->auth "const" to emphasize
that the read-only auth pointer was not allocated here.

OK jsing@

Change httpd(8) to use C99-style fixed-width integers (uintN_t instead
of u_intN_t) and replace u_int with unsigned int. Mixing both
variants is a bad style and most contributors seem to prefer this
style; it also helps us to get used to it, portability, and
standardization.

Theoretically no binary change, except one in practice: httpd.o has a
different checksum because gcc with -O2 pads/optimizes "struct
privsep" differently when using "unsigned int" instead "u_int" for the
affected members. "u_int" is just a typedef of "unsigned int", -O0
doesn't build the difference and clang with -O2 doesn't do it either -
it is just another curiosity from gcc-land.

OK semarie@

Fix rev 1.70 of server.c by only re-enabling the bufferevent if we
previously disabled it because we were reading to fast (from disk).
Problem noted and tracked down to that commit by weerd@ and
independently by stsp@.
Tested by weerd@, stsp@, reyk@
OK bluhm@, reyk@

repair hsts header output, wrong format strings caused broken
Strict-Transport-Security headers. Add __format__ attribute to
kv_set() and kv_setkey() to make it easier to spot such problems.

Found by and fix from Donovan Watteau <tsoomi -AT- gmail -DOT- com>,
thanks for your help.

ok deraadt@

backout the previous: it broke wordpress somehow.
we need more care to find a proper fix for the fastcgi headers.

acknowledged by deraadt@

Read fcgi response records until we have the whole http header and can
parse it. Otherwise http headers can leak into the body.
Pointed out by Jean-Philippe Ouellet on bugs@ Thanks!
OK reyk, commit ASAP deraadt@

For the completeness of HSTS, add the non-standard preload option.

OK florian@

libtls has been changed to set SSL_MODE_ENABLE_PARTIAL_WRITE and
SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER by default. This gives
tls_write() a similar short write semantics as write(2) and a
workaround in httpd to cope with the previous differences can be
removed. Specifically, httpd can stop copying data into a local
buffer that was used to keep it around for repeated writes.

OK bluhm@

Allow to change the default media type globally or per-location,
eg. default type text/html.

OK florian@

Implement HTTP Strict Transport Security (HSTS).
Input & OK reyk

If we can read faster from disk than send data to the client stop
reading from disk when we hold a certain amount of data in
RAM. Re-enable reading once we send enough data to the
client. Otherwise we might end up with the whole file (which can be
huge) in RAM.
Reported by Matthew Martin ( matt.a.martin AT gmail ) on bugs@,
thanks!
OK reyk@, benno@

VIS_QUOTE is not there yet, unbreak the tree. Noticed by semarie@

Escape the message in server_log() as well.

OK benno@

Send the TLS certificate and key via separate imsgs, rather than
including them in the IMSG_CFG_SERVER imsg. This allows the certificate
and key to each be almost 16KB (the maximum size for an imsg), rather than
having a combined total of less than 16KB (which can be reached with large
keys, certificate bundles or by including text versions of certificates).

ok reyk@

Add initial support for pattern matching using Lua's pattern matching code.

With important help on the pattern matcher from semarie@

OK semarie@

Use off_t instead of size_t to pass file size and print it using %lld when
constructing the Content-Length header field. Should fix some, but probably
not all, problems with serving files bigger than 2G on 32-bit architectures.

ok reyk@, florian@

Prevent use after free.
While here unconditionally free clt and move declaration of
server_inflight_dec() into server.c
Found while investigating if (foo != NULL) free(foo) patterns pointed
out by Markus Elfring.
OK reyk

Allow to specify CGI variables as macros in redirection strings, eg.
block return 301 "http://www.example.com/$REQUEST_URI"

OK tedu@ florian@

Rename escape_uri() to url_encode() because it is the opposite of
url_decode(). No functional change.

Allow TLS protocols to be specified via a "tls protocols" configuration
option.

ok reyk@

Encode directory listings.
Problem pointed out by remco AT d-compu.dyndns.org some time ago.
Input / OK reyk@

Remove server_load_file() in favor of tls_load_file(3)

Add httpd configuration options to allow the specification of DHE
parameters and the ECDHE curve. This primarily allows for DHE cipher suites
to be enabled.

ok reyk@

Add support for blocking, dropping, and redirecting requests.

OK florian@

Fix log options in locations.

Reported and tested by Markus Bergkvist
OK florian@

Ooops, no need to include sys/cdefs.h.

Pointed out by florian@

httpd is based on relayd and had included many headers that are only
needed by its ancestor. jsg@, include-what-you-use, and some manual
review helped to cleanup the headers (take iwyu with a grain of salt).
Based on common practice, httpd.h now also includes the necessary
headers for itself.

OK florian@

s/clt_fcgi_remote_user/clt_remote_user/
OK reyk@

Decouple auth parameters from struct server_config into struct auth.

OK florian@

First stab at implementing basic auth.
Currently the htpasswd file needs to be in the chroot; will hopefully
improved soonish.
Based on a diff from Oscar Linderholm many months ago but turned into
a complete rewrite.
input/OK reyk@

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

bump copyright year

add new url stripping option:

strip number
Strip number path components from the beginning of the
request URI before looking up the stripped-down URI at
the document root.


reviewed with much patience and OK by reyk@

Bump config flags field to 32bits. Makes room for future changes -
but no functional change yet.

Like previously done in relayd, change the keyword "ssl" to "tls" to
reflect reality.

OK benno@

Allow the log directory to be configurable in the config file, rather than
fixed as /logs within the chroot. As this httpd is properly privesp'ed this
has the nice property of allowing us to put the logs outside the chroot if
we want to.
ok reyk@

Convert the logic in yyerror(). Instead of creating a temporary
format string, create a temporary message.
OK deraadt@

Update httpd(8) to use libtls instead of libressl.

URL-decode the request path.

Tested by ajacoutot@ and others
OK doug@

Handle different possible variations of the Host header (eg.
www.example.com, www.example.com:80, [2001:db8::1], [2001:db8::1]:80).
The port is optional and is typically used on non-default ports. If
the server name is a plain IPv6 address, it is commonly specified in
square brackets.

Makes ajacoutot@ happy
OK florian@

Remove a limitation that only allowed to specify a server name once.
The key has been changed to server name + address + port and now it is
possible to use the same server name for multiple servers with
different addresses, eg. http://www.example.com and
https://www.example.com/.

OK doug@ florian@

FastCGI did not support persistent connections. Add initial support
for persistent connections with FastCGI by implementing chunked
Transfer-Encoding. This only works with HTTP/1.1.

With input and help from florian@ who found some FastCGI edge cases.

OK florian@

Replace the code to get the FastCGI Status header with a proper way to
parse and write the headers using the http response descriptor. This
allows to add other tweaks, like support for chunked encoding, later.

OK florian@

Use two instead of one http descriptor for request and response.

OK chrisz@

Add Last-Modified: HTTP header.

OK reyk@

Provide a failsafe version of the path_info() function that doesn't
need a temporary path variable. Based on an initial diff from
chrisz@.

"Commit any failsafe version and I'm ok with it" chrisz@

When opening directories, re-match the location after the index file
has been appended. This allows to use a fastcgi target as the default
index, for example index.php.

OK florian@

branches: 1.51.2;
Limit the body size in client requests (eg. POST data) to 1M by default;
add a configuration option to change the limit.

ok florian@

Configure the default SSL ciphers as HIGH:!aNULL.

ok deraadt@ reyk@

http POST support
with & OK reyk@

Adjust the read/write watermarks according to the TCP send buffer.
This fixes sending of large files. Previously, httpd was reading the
input file too quickly and could run out of memory when filling the
input buffer.

Found by jsg@
OK florian@

Load the SSL public/private keys in the parent process, then provide them
to the privsep process via imsg. This allows the keys to be moved out of
the chroot (now /etc/ssl/server.crt, /etc/ssl/private/server.key).

ok reyk@

Add configuration options for the most-important connection limits:
max requests (per connection) and timeout. We don't want to add too
many button, and there are good defaults, but these ones are kind of
mandatory.

Limit the number of (Keep-Alive) requests per connection to 100.
(Same default as in nginx and Apache).

Improve logging to allow per- server/location log files. The log
files can also be owned by root now: they're opened by the parent and
send to the logger process with fd passing. This also works with reload.

ok deraadt@

add a config option to specify the chroot directory
ok reyk@

Temporarily move the default location of the SSL/TLS server key and
certificate from /var/www/ to /var/www/conf/. Don't get scared - this
will be changed soon! They're currently located in the chroot
directory but will be moved outside as soon as we adopted some of the
key privsep from relayd in ressl/httpd.

Proxy commit for jsing@:
"Add TLS/SSL support to httpd, based on the recent ressl commits."

From jsing@
ok reyk@

Add initial support for log files in /var/www/logs/. Logging with
syslog is still supported but disabled by default.

ok deraadt@

Implement PATH_INFO and add DOCUMENT_ROOT.
PATH_INFO was requested by naddy@ who successfully tested it with "cvsweb".

ok naddy@

whitespace

Dynamically pass HTTP request headers as protocol-specific HTTP_* CGI
meta-variables.

ok florian@

Add function to iterate all headers. No functional change.

Split fastcgi socket path and document root option and add the
SCRIPT_FILENAME CGI param with a prepended root. This fixes php-fpm
that expects SCRIPT_FILENAME and also works with slowcgi if you
configure the root correctly. For example, if SCRIPT_NAME and
REQUEST_URI are /php/index.php, root is /htdocs, SCRIPT_FILENAME will
be /htdocs/php/index.php. As tested and discussed with florian@

Add another log mode "connection" for a relayd(8)-style log entry after
each connection, not every request. The code was already there and enabled
on debug, I just turned it into an alternative log format.

Locations now inherit access log settings from the server.

Add log to the server flags.

input/"Looks ok" reyk@

Padding of fcgi records is optional, but if we receive padding data we
should read it.

Allow to specify a FastCGI TCP socket on localhost (eg. :9000). Used
for debugging, you should prefer local UNIX sockets, but it helped to
find an issue that will be fixed with the next commit.

OK florian@

Use the log buffer to defer the logging until the connection is closed
or the request completed. Turn the old log message into a debug message.

ok doug@

remove the global "log updates/all" option that came from relayd.

Add common and combined access logging to httpd.

ok reyk@

Correctly parse fcgi records if we don't get the whole record in one
bufferevent_read().
Input/OK reyk@

Only write the HTTP header for the first fastcgi chunk.

some fastcgi improvements:
- DPRINTF instead of log_info for internal debugging.
- submit QUERY_STRING, if it exists
- use a proper function to create an HTTP header.
- use server_file_error() to detect EOF and fastcgi stream errors.
- disable keep-alive/persist for now until we have a reliable way to
get the content length from the cgi response or support chunked
encoding.

"Cool, jep" florian@

One bufferevent can be shared by file and fcgi.

Allow to specify a non-default fastcgi socket.

Rename the "docroot" variable to "path" because it will be used for
either files or the fastcgi socket (and there's no need to use a union yet).

Add a configuration variable "fastcgi" to enable it per server or location.

Put in first stab at fastcgi. Very early work in progress. Putting it
in now so that we can quickly work on it in tree. Requested by reyk@.
deraadt@ is OK with this according to reyk@.

Make "location" work with name-based virtual servers.

Add "location" keyword to specify path-specific configuration in
servers, for example auto index for a sub-directory only. Internally,
a "location" is just a special type of a "virtual" server.

Add extended directory index options: "[no] index" and "[no] auto index".
The option "directory auto index" implements basic directory listing
and is turned off by default.

ok deraadt@

Move configurable TCP options into struct server_config.

Differentiate servers by address and port, not just by address.

It is recommended to use a URL in the Location header of 3xx
responses. To accomplish this, add some semantics to retrieve the
server host name of a connection: either IP, IP:PORT (if not 80) or
[IP6]:PORT, or Host value (if valid).

Canonicalize the request path once without the docroot and prepend the
docroot only only when it's needed. Suggested by deraadt@.

Add support for "virtual hosts" aka. server blocks aka. multiple
servers with the same or "overlapping" IP address but a different name.

ok beck@

Split server and server_config.

Rename a field, needed later, no functional change.

Move the docroot into the server block.

Remove unused fields from structure

Add canonicalize_path() to canonicalize the requested URL path.

Correctly shutdown the servers when the process is terminating;
prevents a crash on exit. With debugging help from blambert@.

always enable DPRINTF with compiled with DEBUG

Implement file descriptor accounting. The concept was taken from
relayd but had to be adjusted for httpd. It now handles single-pass
HTTP connections, persistent connections with multiple requests, and
body-less HEAD requests. With input from benno@

first step towards keep-alive/persistent connections support

Add support for media types (aka. MIME types): the types section is
compatible to nginx' mime.types file which can be included directly.
If not present, use a few built-in defaults for html, css, txt, jpeg,
gif, png, and js.

Add httpd(8), an attempt to turn the relayd(8) codebase into a simple
web server. It is not finished yet and I just started it today, but
the goal is to provide an HTTP server that a) provides minimal
features, b) serves static files, c) provides FastCGI support, and d)
follows common coding practices of OpenBSD.

It will neither support plugins, nor custom memory allocators, EBCDIC
support, PCRE or any other things that can be found elsewhere.
httpd(8) is not intended to provide a fully-featured replacement for
nginx(8) or the Apache, but it will provide enough functionality that
is needed in the OpenBSD base system.

ok deraadt@

remove unused references to httpd.sock; found by dante catalfamo
ok florian

Neither clt_descreq nor clt_descresp in struct client need to be void *.
They both are only used as struct http_descriptor.
OK tb@

Nothing uses kv_flags.

John (j AT bitminer.ca) pointed out that we didn't correctly
initialize struct kv and might use slower KV_FLAG_GLOBBING path in
kv_find depending on stack garbage. Instead of fixing the
initialization just delete kv_flags from struct kv.

OK claudio, tb

Add gzip-static option to httpd. This allows to deliver precompressed
files with content-encoding gzip.
from prx at si3t dot ch; OK tracey@

Add httpd custom error page facility. Adapted by me from
https://github.com/mpfr/httpd-plus.
Improvements from & (earlier version) reads fine to tracey@;
improvements & OK this version benno@, florian@. Thanks.

Do not try to chunk encode an empty http body coming from an fcgi
upstream.

Found the hard way by Chris Narkiewicz who tracked failing uploads in
the nextcloud mobile app down to httpd(8) trying to chunk encode a
"204 No Content" resonse.

Testing by Steve Williams
Testing & OK stsp

Move TAILQ initialization to files where they are used.

These priv-sep daemons all follow a similar design and use TAILQs
for tracking control process connections. In most cases, the TAILQs
are initialized separate from where they are used. Since the scope
of use is generally confined to a specific control process file,
this commit also removes any extern definitions and exposing the
TAILQ structures to other compilation units.

ok bluhm@, tb@

Do not compare TLS config params for non-TLS servers. This allows to
mix 'listen * port 80' and 'listen * tls port 443' in one server block.
Also the last argument of server_tls_cmp - match_keypair - is always 0
so remove this code.
OK florian@ tb@ some long time ago

these programs (with common ancestry) had a -fno-common problem related
to privsep_procid.
ok mortimer

Add location (found|not found) option to test for resource path existence.

Diff by Matthias Pressfreund <mpfr @ fn de>. Thanks.

typo; pointed out by Matthias (mpfr AT fn.de), thanks!

Set fastcgi socket default on server and location.
This allows "fastcgi" directly inside of a server directive without
giving specifying socket.
OK tracey

Add support for non-localhost fastcgi sockets.

Lots of review time kn@
Lots of review time, tweaks, and ok florian@

remove unused functions
from Ross L Richardson <openbsd AT rlr DOT id DOT au>, Thanks
ok claudio@

Remove the unused function canonicalize_host(), it was copied from relayd.
Found by Ross L Richardson, Thanks.

remove unsused enum key_type, copied initialy from relays.
found by Ross L Richardson, thanks!
ok deraadt@

Implement "strip" option for fastcgi to be able to have multiple chroots
under /var/www for FastCGI servers.
From Nazar Zhuk (nazar AT zhuk DOT online), thanks!
Ok benno

spacing

Add a "forwarded" log format that extends the "combined" log format
by including the contents of the X-Forwarded-For and X-Forwarded-Port
headers. If httpd(8) runs behind a proxy like relayd(8), this allows
tracking the origin of the requests. The format is compatible with
log analyzers such as GoAccess and Webalizer.

Patch from Bruno Flueckiger

ok benno, reyk

httpd(8): add support for setting custom FastCGI parameters.

This commit extends the existing grammar by adding the param option
to the fastcgi directive: fastcgi param name value.

Example usage:
fastcgi param VAR1 hello
fastcgi param VAR2 world

With help and OK florian@
Rogue manpage bits, feel free to modify them.

Backout my previous commit:

date: 2018/10/01 19:24:09; author: benno; state: Exp; lines: +7 -1;
commitid: 0O8fyHPNvPd8rvYU;
Only send 408 Timeout responses when we have seen at least part of a
request. Without a request, just close the connection when we hit
request timeout.
Prompted by a bug report from Nikola Kolev, thanks.
ok reyk@ and some suggestions from claudio@ and bluhm@

Mark Patruck (mark AT wrapped DOT cx) found a problem with it, thanks
for the report.

ok reyk@ bluhm@ sthen@ deraadt@

Only send 408 Timeout responses when we have seen at least part of a
request. Without a request, just close the connection when we hit
request timeout.
Prompted by a bug report from Nikola Kolev, thanks.
ok reyk@ and some suggestions from claudio@ and bluhm@

During the fork+exec implementation, daemon(3) was moved after
proc_init(). As a consequence httpd(8) and relayd(8) child processes
did not detach from the terminal anymore. Dup /dev/null to the
stdio file descriptors in the children.
OK benno@

double the allowed length for the 'tls ciphers' option

for example now it can hold the recommended cipher list from the mozilla
ssl config generator rather than failing with a "ciphers too long" error.

ok benno@ sthen@ tb@

Add support for simple one-off internal rewrites.

For example:

location match "/page/(%d+)/.*" {
request rewrite "/static/index.php?id=%1&$QUERY_STRING"
}

Requested by many.

Ok benno@

Add support for client certificate authentication to httpd.

From Jack Burton <jack at saosce dot com dot au> - thanks!

Also tested by Jan Klemkow <j.klemkow at wemelug dot de>.

ok beck@ reyk@

Make httpd listen on all IPv4 and IPv6 addresses for
"listen on * port 80".
While here accept up to 16 addresses from DNS or interface groups.

requested by & "lovely" deraadt@
OK kn@

NAME_MAX is the length of the thing between / / in a path *without*
the terminating NUL.
Do not use it for a "small string" or a "probably short path". Replace
it with new defines or PATH_MAX.
It also makes the life easier for people auditing the tree for real
usage of NAME_MAX.
OK deraadt, benno

Convert httpd to tls_config_set_ecdhecurves(), allowing a list of curves
to be specified, rather than a single curve.

ok beck@

Rework the way that TLS configuration is sent/received via imsgs, so that
are no longer limited by the 16KB maximum size of a single imsg.
Configuration data that is larger than a single message is now chunked and
sent via multiple imsgs.

Prompted by a diff from Jack Burton <jack at saosce dot com dot au>.

ok reyk@

use __func__ in log messages.
From Hiltjo Posthuma hiltjo -AT codemadness -DOT- org, thanks!
ok florian, claudio

Implement TLS ticket support in httpd. Off by default. Use
tls ticket lifetime default
to turn it on with a 2h ticket lifetime.
Rekeying happens after a quarter of that time.
OK reky@ and bob@

/tmp/cvsspEkok

Stop assuming that in_{addr,port}_t are typedefed in <sys/types.h> and
instead pull in <netinet/in.h> or <arpa/inet.h> when those are needed.

ok florian@ beck@ millert@

Fix support for HTTP pipelining by handling all requests in the buffer.

Tested & OK jung@

Reimplement httpd's support for byte ranges.

The previous implementation loaded all the output into a single output
buffer and used its size to determine the Content-Length of the body.

The new implementation calculates the body length first and writes the
individual ranges in an async way using the bufferevent mechanism.

This prevents httpd from using too much memory and applies the
watermark and throttling mechanisms to range requests.

Problem reported by Pierre Kim (pierre.kim.sec at gmail.com)

OK benno@ sunil@

The variable clt_done is used in too many places.

Introduce a new variable clt_headersdone in the async HTTP parser.

OK sunil@ benno@

Stop accessing verbose and debug variables from log.c directly.

This replaces log_verbose() and "extern int verbose" with the two functions
log_setverbose() and log_getverbose().

Pointed out by benno@
OK krw@ eric@ gilles@ (OK gilles@ for the snmpd bits as well)

Move OCSP loading into a separate function - it is not part of the keypair
and this way we can give a separate specific error message.

ok beck@ reyk@

Add OCSP stapling support to httpd
ok jsing@ bcook@

Prevent fd exhaustion in the parent when loading the listening server
sockets by sending the fd one-by-one. This allows to start httpd with
max 32 server instances and many server sockets without changing the
default rlimits in any way.

OK rzalamena@

sync proc.c with vmd: add p_pw to specify a non-standard user for a process.

OK rzalamena@

sync proc.c from switchd, includes minor cast qual fix and removal of p_env.

Replace [RELAY|SERVER]_MAXPROC with the new PROC_MAX_INSTANCES
variable and limit it from 128 to 32 instances (the old value).
While here, move a few PROC_ defines around.

OK rzalamena@

proc.c tweaks: Rename proc_listento() to proc_accept() as it is the
receiving side of proc_connect(). Move some code from main into
proc_init(), the function is now called by parent and children, not
just the parent and it is less copy + paste for other daemons.

OK florian@

struct client starts to become the kitchen sink. Move fastcgi data to
its own struct. Requested by and OK reyk@

spacing

Teach httpd/proc.c how to fork+exec.

This commit implemented the basic functions to proc.c to make it not rely
on global variables, malloc()ed memory and CLOEXEC pipes.

Fix child proc titles from reyk@
ok reyk@, florian@

Kill (remove) the ps_pid from privsep struct since it is not being used
anymore. Also fix the process initialization prototypes.

ok reyk@

Remove duplicated prototypes from header.

"Looks good to me" natano@

Do not assume that the full http response header is in the first
fastcgi stdout record. Keep processing stdout records until we found
the header / body separator and only then generate the header
response.
Problem reported by many.

OK jung@

Kill p_instance from proc.c and remove static proc_id unused variables.

To keep the debug functionality intact and correct we'll use the pid
field in the imsg header to pass the instance number. Remember to always
pass 'ps_instance + 1' otherwise libutil will fill imsg header pid field
with the imsgbuf pid (which is the current process pid).

ok reyk@

Kill the ps_ninstances from proc.c.

We got the same information in ps_instances[proc] (more accurate) and
we avoid allocating unnecessary memory for pipe storage.

ok reyk@

Replace the static env variables with a single global variable.

ok reyk@

Enable SNI support in httpd(8).

ok reyk@

stop including sys/param.h for nitems. define locally as needed.
ok natano reyk

Move server_match() from parse.y to server.c; use env instead of conf,
which is actually the same thing (cluebat from reyk@).

Make httpd stricter with respect to TLS configuration - in particular, do
not allow TLS and non-TLS to be configured on the same port, do not allow
TLS options to be specified without a TLS listener and ensure that the TLS
options are the same when a server is specified on the same address/port.
Currently, these configurations are permitted but do not work as intended.

Also factor out and reuse the server matching code, which was previously
duplicated.

ok reyk@

Adjust existing tls_config_set_cipher() callers for TLS cipher group
changes - map the previous configuration to the equivalent in the new
groups. This will be revisited post release.

Discussed with beck@

Simplify TLS configuration handling. Instead of matching by address/port,
match by configuration ID. This also prevents a memory leak when there are
multiple certificates specified for the same server.

ok beck@

sync with relayd, use proc_compose()

Retire socket_set_blockmode() in favor of the SOCK_NONBLOCK type flag.
As done in iked and snmpd.

OK jung@

Update log.c: change fatal() and fatalx() into variadic functions,
include the process name, and replace all calls of fatal*(NULL) with
fatal(__func__) for better debugging.

OK benno@

Move local logging functions into httpd.c, and sync log.c with relayd -
both daemons are now sharing the same file. No functional changes.

Fix a regression that was introduced with server.c r1.64: Do NOT free
srv_conf->auth in serverconfig_free() because it was not allocated in
config_getserver() but assigned as a reference by id from a global
list that is maintained independently. This fixes a potential
double-free. This fix also makes srv_conf->auth "const" to emphasize
that the read-only auth pointer was not allocated here.

OK jsing@

Change httpd(8) to use C99-style fixed-width integers (uintN_t instead
of u_intN_t) and replace u_int with unsigned int. Mixing both
variants is a bad style and most contributors seem to prefer this
style; it also helps us to get used to it, portability, and
standardization.

Theoretically no binary change, except one in practice: httpd.o has a
different checksum because gcc with -O2 pads/optimizes "struct
privsep" differently when using "unsigned int" instead "u_int" for the
affected members. "u_int" is just a typedef of "unsigned int", -O0
doesn't build the difference and clang with -O2 doesn't do it either -
it is just another curiosity from gcc-land.

OK semarie@

Fix rev 1.70 of server.c by only re-enabling the bufferevent if we
previously disabled it because we were reading to fast (from disk).
Problem noted and tracked down to that commit by weerd@ and
independently by stsp@.
Tested by weerd@, stsp@, reyk@
OK bluhm@, reyk@

repair hsts header output, wrong format strings caused broken
Strict-Transport-Security headers. Add __format__ attribute to
kv_set() and kv_setkey() to make it easier to spot such problems.

Found by and fix from Donovan Watteau <tsoomi -AT- gmail -DOT- com>,
thanks for your help.

ok deraadt@

backout the previous: it broke wordpress somehow.
we need more care to find a proper fix for the fastcgi headers.

acknowledged by deraadt@

Read fcgi response records until we have the whole http header and can
parse it. Otherwise http headers can leak into the body.
Pointed out by Jean-Philippe Ouellet on bugs@ Thanks!
OK reyk, commit ASAP deraadt@

For the completeness of HSTS, add the non-standard preload option.

OK florian@

libtls has been changed to set SSL_MODE_ENABLE_PARTIAL_WRITE and
SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER by default. This gives
tls_write() a similar short write semantics as write(2) and a
workaround in httpd to cope with the previous differences can be
removed. Specifically, httpd can stop copying data into a local
buffer that was used to keep it around for repeated writes.

OK bluhm@

Allow to change the default media type globally or per-location,
eg. default type text/html.

OK florian@

Implement HTTP Strict Transport Security (HSTS).
Input & OK reyk

If we can read faster from disk than send data to the client stop
reading from disk when we hold a certain amount of data in
RAM. Re-enable reading once we send enough data to the
client. Otherwise we might end up with the whole file (which can be
huge) in RAM.
Reported by Matthew Martin ( matt.a.martin AT gmail ) on bugs@,
thanks!
OK reyk@, benno@

VIS_QUOTE is not there yet, unbreak the tree. Noticed by semarie@

Escape the message in server_log() as well.

OK benno@

Send the TLS certificate and key via separate imsgs, rather than
including them in the IMSG_CFG_SERVER imsg. This allows the certificate
and key to each be almost 16KB (the maximum size for an imsg), rather than
having a combined total of less than 16KB (which can be reached with large
keys, certificate bundles or by including text versions of certificates).

ok reyk@

Add initial support for pattern matching using Lua's pattern matching code.

With important help on the pattern matcher from semarie@

OK semarie@

Use off_t instead of size_t to pass file size and print it using %lld when
constructing the Content-Length header field. Should fix some, but probably
not all, problems with serving files bigger than 2G on 32-bit architectures.

ok reyk@, florian@

Prevent use after free.
While here unconditionally free clt and move declaration of
server_inflight_dec() into server.c
Found while investigating if (foo != NULL) free(foo) patterns pointed
out by Markus Elfring.
OK reyk

Allow to specify CGI variables as macros in redirection strings, eg.
block return 301 "http://www.example.com/$REQUEST_URI"

OK tedu@ florian@

Rename escape_uri() to url_encode() because it is the opposite of
url_decode(). No functional change.

Allow TLS protocols to be specified via a "tls protocols" configuration
option.

ok reyk@

Encode directory listings.
Problem pointed out by remco AT d-compu.dyndns.org some time ago.
Input / OK reyk@

Remove server_load_file() in favor of tls_load_file(3)

Add httpd configuration options to allow the specification of DHE
parameters and the ECDHE curve. This primarily allows for DHE cipher suites
to be enabled.

ok reyk@

Add support for blocking, dropping, and redirecting requests.

OK florian@

Fix log options in locations.

Reported and tested by Markus Bergkvist
OK florian@

Ooops, no need to include sys/cdefs.h.

Pointed out by florian@

httpd is based on relayd and had included many headers that are only
needed by its ancestor. jsg@, include-what-you-use, and some manual
review helped to cleanup the headers (take iwyu with a grain of salt).
Based on common practice, httpd.h now also includes the necessary
headers for itself.

OK florian@