plug fd leak introduced in "avoid toctu" rewrite.

Issue reported by job
OK tb, job

Prevent toctu issues in static file serving and auto index generation.

This fixes a problem in passing, reported by matthieu@ where httpd
would return 500 Internal Server Error when it could stat(2) but not
open(2) a file. The correct error code is 403.

testing matthieu
ok tobhe, tl;dr ok stsp
input & OK deraadt

put the real sizes into the "title" attribute so that hovering shows the
exact value.

adjust/refactor javascript sorter accordingly

make auto-index better
- make it an actual table
- use "human readable sizes" for the file sizes
- add some decoration and javascript to be able to sort it per-column
(client side) (this means some extra column attribute)
- add glue to facilitate embedding js + css directly in the program
- add some graphical indication for directories
- should still validate as proper html everywhere (custom properties
need to be called data-* for this!)

Work with claudio@ and tb@, many thanks to claudio@ for some of the finer
points of css handling, and tb@ for some fine spaces fixes.

I've tried it with lynx as well, shows up correctly.

One big plus is that the size of columns work as utf-8, so you can expose
filenames without any problems (I've tried it with non-js text navigators
as well as firefox, chromium and friends)

And it looks slightly less yahoo ca. 1995.

It's still "one size fits all". If people object to the current look, adding
httpd.conf(5) properties to override the default css should be easy.

okay claudio@, tb@

don't add values we won't print, use scandir properly

okay tb@

(logic NOT simplified because I don't want to risk introducing bugs,
and the compiler knows as much boolean logic as me)

plug some memory leaks in server_file_index when failures occur

namelist and its entries are not freed if escape_html fails or if we
fail in the inner loop. Move scandir later so it's closer to the for
loop and handle escape_html and url_encode failures.

With lots of help from tb, thanks!

ok tb@

Only provide the .gz file if timestamp is >= the non-gzip file. Make
sure it is a regular file. Use the timestamp of the non-gzip file as
the last-modified timestamp, as proposed by claudio.
ok claudio bluhm millert

struct stat from early file inspection was being used after actual file
open() which means the stat could refer to the wrong file. Mostly this
relates to st_size use. This bug could mean that httpd sends new files
truncated to the old length, saying "I am sure you have the correct file now"?
Could have other bad effects.
ok tb millert bluhm

Simplify .gz handling a bit

Combine strlcpy + strlcat into a single snprintf and remove a few
unnecessary parentheses.

ok deraadt millert

Add gzip-static option to httpd. This allows to deliver precompressed
files with content-encoding gzip.
from prx at si3t dot ch; OK tracey@

Use relative reference URIs in Location header on directory redirects.

This adds support for front-ending httpd(8) with a TLS-terminating
gateway like relayd(8) that forwards unencrypted http traffic.

Previously httpd(8) would use a full URL in the Location header in 301
redirects when a user-agent requests a directory but without the
trailing '/'. If the user-agent originally connected with https, this
caused the redirected url to be http.

This change conforms to RFC7231 section 7.1.2.

Reported by Vincent Lee.

OK claudio@

A socket buffer is not the best size to read from a disk.
Use st_blksize to set high water mark; florian@

Use the simpler HTML5 idiom to declare charset in autogenerated pages.

This came from a suggestion by Andras Farkas to replace use of
XHTML self-closing tags.

ok cwen@ danj@ florian@

httpd: add a "dark mode" in directory listings and error pages

Introduce a new "dark mode" for httpd(8) directory listings and error pages,
using the prefers-color-scheme css function. It uses the colorscheme already
used in OpenBSD project websites.

This version will only be displayed if the user's browser specifically sends
a "prefer darker themes if available" request.

Input from florian@, danj@ and clematis.

OK florian@, danj@ (on a previous version with the wrong background color)

Fix 304 Not Modified response: don't send a body, use the correct MIME type.

Reported by Hidvegi Gabor gaborca websivision hu

Fix found by anton@

OK anton@

Fix support for HTTP pipelining by handling all requests in the buffer.

Tested & OK jung@

Reimplement httpd's support for byte ranges.

The previous implementation loaded all the output into a single output
buffer and used its size to determine the Content-Length of the body.

The new implementation calculates the body length first and writes the
individual ranges in an async way using the bufferevent mechanism.

This prevents httpd from using too much memory and applies the
watermark and throttling mechanisms to range requests.

Problem reported by Pierre Kim (pierre.kim.sec at gmail.com)

OK benno@ sunil@

Fix error path of range requests, found while reviewing byte range support.

OK jsg@

branches: 1.62.2;
Repair some file descriptor leaks.
ok beck krw millert

Set content charset for auto index generated page.
Pointed out and diff by dhill, thanks!
Tweaks and same change for error documents by me.

branches: 1.60.2;
Fix rev 1.70 of server.c by only re-enabling the bufferevent if we
previously disabled it because we were reading to fast (from disk).
Problem noted and tracked down to that commit by weerd@ and
independently by stsp@.
Tested by weerd@, stsp@, reyk@
OK bluhm@, reyk@

ensure http_path is escaped before using it in Location redirection.

OK reyk@

Fix check against NULL which was reverted by accident in r1.56.

ok reyk@

Allow to change the default media type globally or per-location,
eg. default type text/html.

OK florian@

Adjust server_file_modified_since() to our style. Please keep httpd clean.

spacing

Implement If-Modified-Since.
From Kyle Thompson <jmp AT giga DOT moe>.
Tweaks by me.
OK benno@

Implement byte ranges.
From Sunil Nimmagadda <sunil At nimmagadda DOT net>
OK benno@

Prepend files or directories containing ":" with "./" in directory
indexes as per RFC 3986:
A path segment that contains a colon character (e.g., "this:that")
cannot be used as the first segment of a relative-path reference, as
it would be mistaken for a scheme name. Such a segment must be
preceded by a dot-segment (e.g., "./this:that") to make a relative-
path reference.

While here add a "/" to the end of directory names, this saves us one
redirect round trip.

Found the hard way & "functionality wise, OK" ajacoutot@
RFC pointer & OK benno@

Rename escape_uri() to url_encode() because it is the opposite of
url_decode(). No functional change.

Encode directory listings.
Problem pointed out by remco AT d-compu.dyndns.org some time ago.
Input / OK reyk@

spacing

httpd is based on relayd and had included many headers that are only
needed by its ancestor. jsg@, include-what-you-use, and some manual
review helped to cleanup the headers (take iwyu with a grain of salt).
Based on common practice, httpd.h now also includes the necessary
headers for itself.

OK florian@

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

bump copyright year

Make httpd return "404 not found" if an intermediate component of a requested
file path does not exist rather than returning "500 internal server error".
ok reyk

add new url stripping option:

strip number
Strip number path components from the beginning of the
request URI before looking up the stripped-down URI at
the document root.


reviewed with much patience and OK by reyk@

Use the HTML5 doctype for error and auto index pages because it is
shorter, newer, and the recommendation. From James Jerkins.

Exclude the charset for now because it is not explicitly handled by httpd.

OK validator.w3.org (This document was successfully checked as HTML5!)

Stop pulling in <arpa/inet.h> or <arpa/nameser.h> when unnecessary.
*Do* pull it in when in_{port,addr}_h is needed and <netinet/in.h> isn't.

ok reyk@

Like previously done in relayd, change the keyword "ssl" to "tls" to
reflect reality.

OK benno@

stop viral header propagation. none of this code uses sys/hash.h
from Max Fillinger

Remove unnecessary netinet/in_systm.h include.

ok millert@

Rework the error message a little bit: Do not send details of the
error. Traditionally, web servers responsed with the request path on
40x errors which could be abused to inject JavaScript etc. Instead of
sanitizing the path, we just don't reprint it. Also modify the style
a little bit but keep Comic Sans.

With input from Jonas Lindemann and doug@

Fix an error case that was never handled ending up in an endless event
loop that could eat all CPU. I thought that the previous (correct)
commit fixed it which wasn't the case. But this one is obvious.

ok florian@

In addition to READ, disable WRITE events when closing the file
descriptor of the file I/O bufferevent. This fixes a potential event flood.

OK florian@

Use two instead of one http descriptor for request and response.

OK chrisz@

Add Last-Modified: HTTP header.

OK reyk@

Remove obsolete struct stat parameters.

ok reyk@

When opening directories, re-match the location after the index file
has been appended. This allows to use a fastcgi target as the default
index, for example index.php.

OK florian@

branches: 1.31.4;
The watermark exposed a bug in server_write that broke keep-alive
support. Instead of calling server_close from server_write, we have
to proceed to the next connection by calling the error handler.

OK jsg@

Adjust the read/write watermarks according to the TCP send buffer.
This fixes sending of large files. Previously, httpd was reading the
input file too quickly and could run out of memory when filling the
input buffer.

Found by jsg@
OK florian@

Redirect to https:// if SSL/TLS is enabled.

httpd doesn't support SSL/TLS yet, remove the remaining bits.
The secrect plan is to add it later using the ressl wrapper library.

Only allow GET and HEAD for static files or return 405.

ok florian@

Also write log messages, like 404 Not Found, on error. This is a bit
tricky because we couldn't guarantee a sane state after
server_response_http() so fail hard afterwards and close the connection.

ok doug@

Split fastcgi socket path and document root option and add the
SCRIPT_FILENAME CGI param with a prepended root. This fixes php-fpm
that expects SCRIPT_FILENAME and also works with slowcgi if you
configure the root correctly. For example, if SCRIPT_NAME and
REQUEST_URI are /php/index.php, root is /htdocs, SCRIPT_FILENAME will
be /htdocs/php/index.php. As tested and discussed with florian@

spacing

scandir(3)-based directory auto index didn't work on NFS because the
file system is not filling in d_type properly. Using st_mode from the
stat call fixes the problem, eg. S_ISDIR(st.st_mode) instead of
dp->d_type == DT_DIR. Pointed out by pelikan@

some fastcgi improvements:
- DPRINTF instead of log_info for internal debugging.
- submit QUERY_STRING, if it exists
- use a proper function to create an HTTP header.
- use server_file_error() to detect EOF and fastcgi stream errors.
- disable keep-alive/persist for now until we have a reliable way to
get the content length from the cgi response or support chunked
encoding.

"Cool, jep" florian@

One bufferevent can be shared by file and fcgi.

Rename the "docroot" variable to "path" because it will be used for
either files or the fastcgi socket (and there's no need to use a union yet).

Reserve an extra file descriptor per connection instead of per
request. This fixes fd accounting with persistent connections and
reduces the complexity of the implementation.

ok benno@

Add extended directory index options: "[no] index" and "[no] auto index".
The option "directory auto index" implements basic directory listing
and is turned off by default.

ok deraadt@

Remove redundant slash

It is recommended to use a URL in the Location header of 3xx
responses. To accomplish this, add some semantics to retrieve the
server host name of a connection: either IP, IP:PORT (if not 80) or
[IP6]:PORT, or Host value (if valid).

Canonicalize the request path once without the docroot and prepend the
docroot only only when it's needed. Suggested by deraadt@.

Don't leak docroot in the error message if the default index file is missing.

OK florian@

Split server and server_config.

Rename a field, needed later, no functional change.

Move the docroot into the server block.

The default index page shouldn't be a directory. It's a 500.

Don't expose the docroot on error.

First attempt at verifying the request path and the access
permissions. We also have to redirect with 301 if a directory name
was requested without the trailing slash.

Add canonicalize_path() to canonicalize the requested URL path.

Implement file descriptor accounting. The concept was taken from
relayd but had to be adjusted for httpd. It now handles single-pass
HTTP connections, persistent connections with multiple requests, and
body-less HEAD requests. With input from benno@

don't diplay the full path in error messages

first step towards keep-alive/persistent connections support

Finish writing the output before closing the connection
(adopted from relayd).

Add support for media types (aka. MIME types): the types section is
compatible to nginx' mime.types file which can be included directly.
If not present, use a few built-in defaults for html, css, txt, jpeg,
gif, png, and js.

Add httpd(8), an attempt to turn the relayd(8) codebase into a simple
web server. It is not finished yet and I just started it today, but
the goal is to provide an HTTP server that a) provides minimal
features, b) serves static files, c) provides FastCGI support, and d)
follows common coding practices of OpenBSD.

It will neither support plugins, nor custom memory allocators, EBCDIC
support, PCRE or any other things that can be found elsewhere.
httpd(8) is not intended to provide a fully-featured replacement for
nginx(8) or the Apache, but it will provide enough functionality that
is needed in the OpenBSD base system.

ok deraadt@

Prevent toctu issues in static file serving and auto index generation.

This fixes a problem in passing, reported by matthieu@ where httpd
would return 500 Internal Server Error when it could stat(2) but not
open(2) a file. The correct error code is 403.

testing matthieu
ok tobhe, tl;dr ok stsp
input & OK deraadt

put the real sizes into the "title" attribute so that hovering shows the
exact value.

adjust/refactor javascript sorter accordingly

make auto-index better
- make it an actual table
- use "human readable sizes" for the file sizes
- add some decoration and javascript to be able to sort it per-column
(client side) (this means some extra column attribute)
- add glue to facilitate embedding js + css directly in the program
- add some graphical indication for directories
- should still validate as proper html everywhere (custom properties
need to be called data-* for this!)

Work with claudio@ and tb@, many thanks to claudio@ for some of the finer
points of css handling, and tb@ for some fine spaces fixes.

I've tried it with lynx as well, shows up correctly.

One big plus is that the size of columns work as utf-8, so you can expose
filenames without any problems (I've tried it with non-js text navigators
as well as firefox, chromium and friends)

And it looks slightly less yahoo ca. 1995.

It's still "one size fits all". If people object to the current look, adding
httpd.conf(5) properties to override the default css should be easy.

okay claudio@, tb@

don't add values we won't print, use scandir properly

okay tb@

(logic NOT simplified because I don't want to risk introducing bugs,
and the compiler knows as much boolean logic as me)

plug some memory leaks in server_file_index when failures occur

namelist and its entries are not freed if escape_html fails or if we
fail in the inner loop. Move scandir later so it's closer to the for
loop and handle escape_html and url_encode failures.

With lots of help from tb, thanks!

ok tb@

Only provide the .gz file if timestamp is >= the non-gzip file. Make
sure it is a regular file. Use the timestamp of the non-gzip file as
the last-modified timestamp, as proposed by claudio.
ok claudio bluhm millert

struct stat from early file inspection was being used after actual file
open() which means the stat could refer to the wrong file. Mostly this
relates to st_size use. This bug could mean that httpd sends new files
truncated to the old length, saying "I am sure you have the correct file now"?
Could have other bad effects.
ok tb millert bluhm

Simplify .gz handling a bit

Combine strlcpy + strlcat into a single snprintf and remove a few
unnecessary parentheses.

ok deraadt millert

Add gzip-static option to httpd. This allows to deliver precompressed
files with content-encoding gzip.
from prx at si3t dot ch; OK tracey@

Use relative reference URIs in Location header on directory redirects.

This adds support for front-ending httpd(8) with a TLS-terminating
gateway like relayd(8) that forwards unencrypted http traffic.

Previously httpd(8) would use a full URL in the Location header in 301
redirects when a user-agent requests a directory but without the
trailing '/'. If the user-agent originally connected with https, this
caused the redirected url to be http.

This change conforms to RFC7231 section 7.1.2.

Reported by Vincent Lee.

OK claudio@

A socket buffer is not the best size to read from a disk.
Use st_blksize to set high water mark; florian@

Use the simpler HTML5 idiom to declare charset in autogenerated pages.

This came from a suggestion by Andras Farkas to replace use of
XHTML self-closing tags.

ok cwen@ danj@ florian@

httpd: add a "dark mode" in directory listings and error pages

Introduce a new "dark mode" for httpd(8) directory listings and error pages,
using the prefers-color-scheme css function. It uses the colorscheme already
used in OpenBSD project websites.

This version will only be displayed if the user's browser specifically sends
a "prefer darker themes if available" request.

Input from florian@, danj@ and clematis.

OK florian@, danj@ (on a previous version with the wrong background color)

Fix 304 Not Modified response: don't send a body, use the correct MIME type.

Reported by Hidvegi Gabor gaborca websivision hu

Fix found by anton@

OK anton@

Fix support for HTTP pipelining by handling all requests in the buffer.

Tested & OK jung@

Reimplement httpd's support for byte ranges.

The previous implementation loaded all the output into a single output
buffer and used its size to determine the Content-Length of the body.

The new implementation calculates the body length first and writes the
individual ranges in an async way using the bufferevent mechanism.

This prevents httpd from using too much memory and applies the
watermark and throttling mechanisms to range requests.

Problem reported by Pierre Kim (pierre.kim.sec at gmail.com)

OK benno@ sunil@

Fix error path of range requests, found while reviewing byte range support.

OK jsg@

branches: 1.62.2;
Repair some file descriptor leaks.
ok beck krw millert

Set content charset for auto index generated page.
Pointed out and diff by dhill, thanks!
Tweaks and same change for error documents by me.

branches: 1.60.2;
Fix rev 1.70 of server.c by only re-enabling the bufferevent if we
previously disabled it because we were reading to fast (from disk).
Problem noted and tracked down to that commit by weerd@ and
independently by stsp@.
Tested by weerd@, stsp@, reyk@
OK bluhm@, reyk@

ensure http_path is escaped before using it in Location redirection.

OK reyk@

Fix check against NULL which was reverted by accident in r1.56.

ok reyk@

Allow to change the default media type globally or per-location,
eg. default type text/html.

OK florian@

Adjust server_file_modified_since() to our style. Please keep httpd clean.

spacing

Implement If-Modified-Since.
From Kyle Thompson <jmp AT giga DOT moe>.
Tweaks by me.
OK benno@

Implement byte ranges.
From Sunil Nimmagadda <sunil At nimmagadda DOT net>
OK benno@

Prepend files or directories containing ":" with "./" in directory
indexes as per RFC 3986:
A path segment that contains a colon character (e.g., "this:that")
cannot be used as the first segment of a relative-path reference, as
it would be mistaken for a scheme name. Such a segment must be
preceded by a dot-segment (e.g., "./this:that") to make a relative-
path reference.

While here add a "/" to the end of directory names, this saves us one
redirect round trip.

Found the hard way & "functionality wise, OK" ajacoutot@
RFC pointer & OK benno@

Rename escape_uri() to url_encode() because it is the opposite of
url_decode(). No functional change.

Encode directory listings.
Problem pointed out by remco AT d-compu.dyndns.org some time ago.
Input / OK reyk@

spacing

httpd is based on relayd and had included many headers that are only
needed by its ancestor. jsg@, include-what-you-use, and some manual
review helped to cleanup the headers (take iwyu with a grain of salt).
Based on common practice, httpd.h now also includes the necessary
headers for itself.

OK florian@

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

bump copyright year

Make httpd return "404 not found" if an intermediate component of a requested
file path does not exist rather than returning "500 internal server error".
ok reyk

add new url stripping option:

strip number
Strip number path components from the beginning of the
request URI before looking up the stripped-down URI at
the document root.


reviewed with much patience and OK by reyk@

Use the HTML5 doctype for error and auto index pages because it is
shorter, newer, and the recommendation. From James Jerkins.

Exclude the charset for now because it is not explicitly handled by httpd.

OK validator.w3.org (This document was successfully checked as HTML5!)

Stop pulling in <arpa/inet.h> or <arpa/nameser.h> when unnecessary.
*Do* pull it in when in_{port,addr}_h is needed and <netinet/in.h> isn't.

ok reyk@

Like previously done in relayd, change the keyword "ssl" to "tls" to
reflect reality.

OK benno@

stop viral header propagation. none of this code uses sys/hash.h
from Max Fillinger

Remove unnecessary netinet/in_systm.h include.

ok millert@

Rework the error message a little bit: Do not send details of the
error. Traditionally, web servers responsed with the request path on
40x errors which could be abused to inject JavaScript etc. Instead of
sanitizing the path, we just don't reprint it. Also modify the style
a little bit but keep Comic Sans.

With input from Jonas Lindemann and doug@

Fix an error case that was never handled ending up in an endless event
loop that could eat all CPU. I thought that the previous (correct)
commit fixed it which wasn't the case. But this one is obvious.

ok florian@

In addition to READ, disable WRITE events when closing the file
descriptor of the file I/O bufferevent. This fixes a potential event flood.

OK florian@

Use two instead of one http descriptor for request and response.

OK chrisz@

Add Last-Modified: HTTP header.

OK reyk@

Remove obsolete struct stat parameters.

ok reyk@

When opening directories, re-match the location after the index file
has been appended. This allows to use a fastcgi target as the default
index, for example index.php.

OK florian@

branches: 1.31.4;
The watermark exposed a bug in server_write that broke keep-alive
support. Instead of calling server_close from server_write, we have
to proceed to the next connection by calling the error handler.

OK jsg@

Adjust the read/write watermarks according to the TCP send buffer.
This fixes sending of large files. Previously, httpd was reading the
input file too quickly and could run out of memory when filling the
input buffer.

Found by jsg@
OK florian@

Redirect to https:// if SSL/TLS is enabled.

httpd doesn't support SSL/TLS yet, remove the remaining bits.
The secrect plan is to add it later using the ressl wrapper library.

Only allow GET and HEAD for static files or return 405.

ok florian@

Also write log messages, like 404 Not Found, on error. This is a bit
tricky because we couldn't guarantee a sane state after
server_response_http() so fail hard afterwards and close the connection.

ok doug@

Split fastcgi socket path and document root option and add the
SCRIPT_FILENAME CGI param with a prepended root. This fixes php-fpm
that expects SCRIPT_FILENAME and also works with slowcgi if you
configure the root correctly. For example, if SCRIPT_NAME and
REQUEST_URI are /php/index.php, root is /htdocs, SCRIPT_FILENAME will
be /htdocs/php/index.php. As tested and discussed with florian@

spacing

scandir(3)-based directory auto index didn't work on NFS because the
file system is not filling in d_type properly. Using st_mode from the
stat call fixes the problem, eg. S_ISDIR(st.st_mode) instead of
dp->d_type == DT_DIR. Pointed out by pelikan@

some fastcgi improvements:
- DPRINTF instead of log_info for internal debugging.
- submit QUERY_STRING, if it exists
- use a proper function to create an HTTP header.
- use server_file_error() to detect EOF and fastcgi stream errors.
- disable keep-alive/persist for now until we have a reliable way to
get the content length from the cgi response or support chunked
encoding.

"Cool, jep" florian@

One bufferevent can be shared by file and fcgi.

Rename the "docroot" variable to "path" because it will be used for
either files or the fastcgi socket (and there's no need to use a union yet).

Reserve an extra file descriptor per connection instead of per
request. This fixes fd accounting with persistent connections and
reduces the complexity of the implementation.

ok benno@

Add extended directory index options: "[no] index" and "[no] auto index".
The option "directory auto index" implements basic directory listing
and is turned off by default.

ok deraadt@

Remove redundant slash

It is recommended to use a URL in the Location header of 3xx
responses. To accomplish this, add some semantics to retrieve the
server host name of a connection: either IP, IP:PORT (if not 80) or
[IP6]:PORT, or Host value (if valid).

Canonicalize the request path once without the docroot and prepend the
docroot only only when it's needed. Suggested by deraadt@.

Don't leak docroot in the error message if the default index file is missing.

OK florian@

Split server and server_config.

Rename a field, needed later, no functional change.

Move the docroot into the server block.

The default index page shouldn't be a directory. It's a 500.

Don't expose the docroot on error.

First attempt at verifying the request path and the access
permissions. We also have to redirect with 301 if a directory name
was requested without the trailing slash.

Add canonicalize_path() to canonicalize the requested URL path.

Implement file descriptor accounting. The concept was taken from
relayd but had to be adjusted for httpd. It now handles single-pass
HTTP connections, persistent connections with multiple requests, and
body-less HEAD requests. With input from benno@

don't diplay the full path in error messages

first step towards keep-alive/persistent connections support

Finish writing the output before closing the connection
(adopted from relayd).

Add support for media types (aka. MIME types): the types section is
compatible to nginx' mime.types file which can be included directly.
If not present, use a few built-in defaults for html, css, txt, jpeg,
gif, png, and js.

Add httpd(8), an attempt to turn the relayd(8) codebase into a simple
web server. It is not finished yet and I just started it today, but
the goal is to provide an HTTP server that a) provides minimal
features, b) serves static files, c) provides FastCGI support, and d)
follows common coding practices of OpenBSD.

It will neither support plugins, nor custom memory allocators, EBCDIC
support, PCRE or any other things that can be found elsewhere.
httpd(8) is not intended to provide a fully-featured replacement for
nginx(8) or the Apache, but it will provide enough functionality that
is needed in the OpenBSD base system.

ok deraadt@

put the real sizes into the "title" attribute so that hovering shows the
exact value.

adjust/refactor javascript sorter accordingly

make auto-index better
- make it an actual table
- use "human readable sizes" for the file sizes
- add some decoration and javascript to be able to sort it per-column
(client side) (this means some extra column attribute)
- add glue to facilitate embedding js + css directly in the program
- add some graphical indication for directories
- should still validate as proper html everywhere (custom properties
need to be called data-* for this!)

Work with claudio@ and tb@, many thanks to claudio@ for some of the finer
points of css handling, and tb@ for some fine spaces fixes.

I've tried it with lynx as well, shows up correctly.

One big plus is that the size of columns work as utf-8, so you can expose
filenames without any problems (I've tried it with non-js text navigators
as well as firefox, chromium and friends)

And it looks slightly less yahoo ca. 1995.

It's still "one size fits all". If people object to the current look, adding
httpd.conf(5) properties to override the default css should be easy.

okay claudio@, tb@

don't add values we won't print, use scandir properly

okay tb@

(logic NOT simplified because I don't want to risk introducing bugs,
and the compiler knows as much boolean logic as me)

plug some memory leaks in server_file_index when failures occur

namelist and its entries are not freed if escape_html fails or if we
fail in the inner loop. Move scandir later so it's closer to the for
loop and handle escape_html and url_encode failures.

With lots of help from tb, thanks!

ok tb@

Only provide the .gz file if timestamp is >= the non-gzip file. Make
sure it is a regular file. Use the timestamp of the non-gzip file as
the last-modified timestamp, as proposed by claudio.
ok claudio bluhm millert

struct stat from early file inspection was being used after actual file
open() which means the stat could refer to the wrong file. Mostly this
relates to st_size use. This bug could mean that httpd sends new files
truncated to the old length, saying "I am sure you have the correct file now"?
Could have other bad effects.
ok tb millert bluhm

Simplify .gz handling a bit

Combine strlcpy + strlcat into a single snprintf and remove a few
unnecessary parentheses.

ok deraadt millert

Add gzip-static option to httpd. This allows to deliver precompressed
files with content-encoding gzip.
from prx at si3t dot ch; OK tracey@

Use relative reference URIs in Location header on directory redirects.

This adds support for front-ending httpd(8) with a TLS-terminating
gateway like relayd(8) that forwards unencrypted http traffic.

Previously httpd(8) would use a full URL in the Location header in 301
redirects when a user-agent requests a directory but without the
trailing '/'. If the user-agent originally connected with https, this
caused the redirected url to be http.

This change conforms to RFC7231 section 7.1.2.

Reported by Vincent Lee.

OK claudio@

A socket buffer is not the best size to read from a disk.
Use st_blksize to set high water mark; florian@

Use the simpler HTML5 idiom to declare charset in autogenerated pages.

This came from a suggestion by Andras Farkas to replace use of
XHTML self-closing tags.

ok cwen@ danj@ florian@

httpd: add a "dark mode" in directory listings and error pages

Introduce a new "dark mode" for httpd(8) directory listings and error pages,
using the prefers-color-scheme css function. It uses the colorscheme already
used in OpenBSD project websites.

This version will only be displayed if the user's browser specifically sends
a "prefer darker themes if available" request.

Input from florian@, danj@ and clematis.

OK florian@, danj@ (on a previous version with the wrong background color)

Fix 304 Not Modified response: don't send a body, use the correct MIME type.

Reported by Hidvegi Gabor gaborca websivision hu

Fix found by anton@

OK anton@

Fix support for HTTP pipelining by handling all requests in the buffer.

Tested & OK jung@

Reimplement httpd's support for byte ranges.

The previous implementation loaded all the output into a single output
buffer and used its size to determine the Content-Length of the body.

The new implementation calculates the body length first and writes the
individual ranges in an async way using the bufferevent mechanism.

This prevents httpd from using too much memory and applies the
watermark and throttling mechanisms to range requests.

Problem reported by Pierre Kim (pierre.kim.sec at gmail.com)

OK benno@ sunil@

Fix error path of range requests, found while reviewing byte range support.

OK jsg@

branches: 1.62.2;
Repair some file descriptor leaks.
ok beck krw millert

Set content charset for auto index generated page.
Pointed out and diff by dhill, thanks!
Tweaks and same change for error documents by me.

branches: 1.60.2;
Fix rev 1.70 of server.c by only re-enabling the bufferevent if we
previously disabled it because we were reading to fast (from disk).
Problem noted and tracked down to that commit by weerd@ and
independently by stsp@.
Tested by weerd@, stsp@, reyk@
OK bluhm@, reyk@

ensure http_path is escaped before using it in Location redirection.

OK reyk@

Fix check against NULL which was reverted by accident in r1.56.

ok reyk@

Allow to change the default media type globally or per-location,
eg. default type text/html.

OK florian@

Adjust server_file_modified_since() to our style. Please keep httpd clean.

spacing

Implement If-Modified-Since.
From Kyle Thompson <jmp AT giga DOT moe>.
Tweaks by me.
OK benno@

Implement byte ranges.
From Sunil Nimmagadda <sunil At nimmagadda DOT net>
OK benno@

Prepend files or directories containing ":" with "./" in directory
indexes as per RFC 3986:
A path segment that contains a colon character (e.g., "this:that")
cannot be used as the first segment of a relative-path reference, as
it would be mistaken for a scheme name. Such a segment must be
preceded by a dot-segment (e.g., "./this:that") to make a relative-
path reference.

While here add a "/" to the end of directory names, this saves us one
redirect round trip.

Found the hard way & "functionality wise, OK" ajacoutot@
RFC pointer & OK benno@

Rename escape_uri() to url_encode() because it is the opposite of
url_decode(). No functional change.

Encode directory listings.
Problem pointed out by remco AT d-compu.dyndns.org some time ago.
Input / OK reyk@

spacing

httpd is based on relayd and had included many headers that are only
needed by its ancestor. jsg@, include-what-you-use, and some manual
review helped to cleanup the headers (take iwyu with a grain of salt).
Based on common practice, httpd.h now also includes the necessary
headers for itself.

OK florian@

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

bump copyright year

Make httpd return "404 not found" if an intermediate component of a requested
file path does not exist rather than returning "500 internal server error".
ok reyk

add new url stripping option:

strip number
Strip number path components from the beginning of the
request URI before looking up the stripped-down URI at
the document root.


reviewed with much patience and OK by reyk@

Use the HTML5 doctype for error and auto index pages because it is
shorter, newer, and the recommendation. From James Jerkins.

Exclude the charset for now because it is not explicitly handled by httpd.

OK validator.w3.org (This document was successfully checked as HTML5!)

Stop pulling in <arpa/inet.h> or <arpa/nameser.h> when unnecessary.
*Do* pull it in when in_{port,addr}_h is needed and <netinet/in.h> isn't.

ok reyk@

Like previously done in relayd, change the keyword "ssl" to "tls" to
reflect reality.

OK benno@

stop viral header propagation. none of this code uses sys/hash.h
from Max Fillinger

Remove unnecessary netinet/in_systm.h include.

ok millert@

Rework the error message a little bit: Do not send details of the
error. Traditionally, web servers responsed with the request path on
40x errors which could be abused to inject JavaScript etc. Instead of
sanitizing the path, we just don't reprint it. Also modify the style
a little bit but keep Comic Sans.

With input from Jonas Lindemann and doug@

Fix an error case that was never handled ending up in an endless event
loop that could eat all CPU. I thought that the previous (correct)
commit fixed it which wasn't the case. But this one is obvious.

ok florian@

In addition to READ, disable WRITE events when closing the file
descriptor of the file I/O bufferevent. This fixes a potential event flood.

OK florian@

Use two instead of one http descriptor for request and response.

OK chrisz@

Add Last-Modified: HTTP header.

OK reyk@

Remove obsolete struct stat parameters.

ok reyk@

When opening directories, re-match the location after the index file
has been appended. This allows to use a fastcgi target as the default
index, for example index.php.

OK florian@

branches: 1.31.4;
The watermark exposed a bug in server_write that broke keep-alive
support. Instead of calling server_close from server_write, we have
to proceed to the next connection by calling the error handler.

OK jsg@

Adjust the read/write watermarks according to the TCP send buffer.
This fixes sending of large files. Previously, httpd was reading the
input file too quickly and could run out of memory when filling the
input buffer.

Found by jsg@
OK florian@

Redirect to https:// if SSL/TLS is enabled.

httpd doesn't support SSL/TLS yet, remove the remaining bits.
The secrect plan is to add it later using the ressl wrapper library.

Only allow GET and HEAD for static files or return 405.

ok florian@

Also write log messages, like 404 Not Found, on error. This is a bit
tricky because we couldn't guarantee a sane state after
server_response_http() so fail hard afterwards and close the connection.

ok doug@

Split fastcgi socket path and document root option and add the
SCRIPT_FILENAME CGI param with a prepended root. This fixes php-fpm
that expects SCRIPT_FILENAME and also works with slowcgi if you
configure the root correctly. For example, if SCRIPT_NAME and
REQUEST_URI are /php/index.php, root is /htdocs, SCRIPT_FILENAME will
be /htdocs/php/index.php. As tested and discussed with florian@

spacing

scandir(3)-based directory auto index didn't work on NFS because the
file system is not filling in d_type properly. Using st_mode from the
stat call fixes the problem, eg. S_ISDIR(st.st_mode) instead of
dp->d_type == DT_DIR. Pointed out by pelikan@

some fastcgi improvements:
- DPRINTF instead of log_info for internal debugging.
- submit QUERY_STRING, if it exists
- use a proper function to create an HTTP header.
- use server_file_error() to detect EOF and fastcgi stream errors.
- disable keep-alive/persist for now until we have a reliable way to
get the content length from the cgi response or support chunked
encoding.

"Cool, jep" florian@

One bufferevent can be shared by file and fcgi.

Rename the "docroot" variable to "path" because it will be used for
either files or the fastcgi socket (and there's no need to use a union yet).

Reserve an extra file descriptor per connection instead of per
request. This fixes fd accounting with persistent connections and
reduces the complexity of the implementation.

ok benno@

Add extended directory index options: "[no] index" and "[no] auto index".
The option "directory auto index" implements basic directory listing
and is turned off by default.

ok deraadt@

Remove redundant slash

It is recommended to use a URL in the Location header of 3xx
responses. To accomplish this, add some semantics to retrieve the
server host name of a connection: either IP, IP:PORT (if not 80) or
[IP6]:PORT, or Host value (if valid).

Canonicalize the request path once without the docroot and prepend the
docroot only only when it's needed. Suggested by deraadt@.

Don't leak docroot in the error message if the default index file is missing.

OK florian@

Split server and server_config.

Rename a field, needed later, no functional change.

Move the docroot into the server block.

The default index page shouldn't be a directory. It's a 500.

Don't expose the docroot on error.

First attempt at verifying the request path and the access
permissions. We also have to redirect with 301 if a directory name
was requested without the trailing slash.

Add canonicalize_path() to canonicalize the requested URL path.

Implement file descriptor accounting. The concept was taken from
relayd but had to be adjusted for httpd. It now handles single-pass
HTTP connections, persistent connections with multiple requests, and
body-less HEAD requests. With input from benno@

don't diplay the full path in error messages

first step towards keep-alive/persistent connections support

Finish writing the output before closing the connection
(adopted from relayd).

Add support for media types (aka. MIME types): the types section is
compatible to nginx' mime.types file which can be included directly.
If not present, use a few built-in defaults for html, css, txt, jpeg,
gif, png, and js.

Add httpd(8), an attempt to turn the relayd(8) codebase into a simple
web server. It is not finished yet and I just started it today, but
the goal is to provide an HTTP server that a) provides minimal
features, b) serves static files, c) provides FastCGI support, and d)
follows common coding practices of OpenBSD.

It will neither support plugins, nor custom memory allocators, EBCDIC
support, PCRE or any other things that can be found elsewhere.
httpd(8) is not intended to provide a fully-featured replacement for
nginx(8) or the Apache, but it will provide enough functionality that
is needed in the OpenBSD base system.

ok deraadt@

don't add values we won't print, use scandir properly

okay tb@

(logic NOT simplified because I don't want to risk introducing bugs,
and the compiler knows as much boolean logic as me)

plug some memory leaks in server_file_index when failures occur

namelist and its entries are not freed if escape_html fails or if we
fail in the inner loop. Move scandir later so it's closer to the for
loop and handle escape_html and url_encode failures.

With lots of help from tb, thanks!

ok tb@

Only provide the .gz file if timestamp is >= the non-gzip file. Make
sure it is a regular file. Use the timestamp of the non-gzip file as
the last-modified timestamp, as proposed by claudio.
ok claudio bluhm millert

struct stat from early file inspection was being used after actual file
open() which means the stat could refer to the wrong file. Mostly this
relates to st_size use. This bug could mean that httpd sends new files
truncated to the old length, saying "I am sure you have the correct file now"?
Could have other bad effects.
ok tb millert bluhm

Simplify .gz handling a bit

Combine strlcpy + strlcat into a single snprintf and remove a few
unnecessary parentheses.

ok deraadt millert

Add gzip-static option to httpd. This allows to deliver precompressed
files with content-encoding gzip.
from prx at si3t dot ch; OK tracey@

Use relative reference URIs in Location header on directory redirects.

This adds support for front-ending httpd(8) with a TLS-terminating
gateway like relayd(8) that forwards unencrypted http traffic.

Previously httpd(8) would use a full URL in the Location header in 301
redirects when a user-agent requests a directory but without the
trailing '/'. If the user-agent originally connected with https, this
caused the redirected url to be http.

This change conforms to RFC7231 section 7.1.2.

Reported by Vincent Lee.

OK claudio@

A socket buffer is not the best size to read from a disk.
Use st_blksize to set high water mark; florian@

Use the simpler HTML5 idiom to declare charset in autogenerated pages.

This came from a suggestion by Andras Farkas to replace use of
XHTML self-closing tags.

ok cwen@ danj@ florian@

httpd: add a "dark mode" in directory listings and error pages

Introduce a new "dark mode" for httpd(8) directory listings and error pages,
using the prefers-color-scheme css function. It uses the colorscheme already
used in OpenBSD project websites.

This version will only be displayed if the user's browser specifically sends
a "prefer darker themes if available" request.

Input from florian@, danj@ and clematis.

OK florian@, danj@ (on a previous version with the wrong background color)

Fix 304 Not Modified response: don't send a body, use the correct MIME type.

Reported by Hidvegi Gabor gaborca websivision hu

Fix found by anton@

OK anton@

Fix support for HTTP pipelining by handling all requests in the buffer.

Tested & OK jung@

Reimplement httpd's support for byte ranges.

The previous implementation loaded all the output into a single output
buffer and used its size to determine the Content-Length of the body.

The new implementation calculates the body length first and writes the
individual ranges in an async way using the bufferevent mechanism.

This prevents httpd from using too much memory and applies the
watermark and throttling mechanisms to range requests.

Problem reported by Pierre Kim (pierre.kim.sec at gmail.com)

OK benno@ sunil@

Fix error path of range requests, found while reviewing byte range support.

OK jsg@

branches: 1.62.2;
Repair some file descriptor leaks.
ok beck krw millert

Set content charset for auto index generated page.
Pointed out and diff by dhill, thanks!
Tweaks and same change for error documents by me.

branches: 1.60.2;
Fix rev 1.70 of server.c by only re-enabling the bufferevent if we
previously disabled it because we were reading to fast (from disk).
Problem noted and tracked down to that commit by weerd@ and
independently by stsp@.
Tested by weerd@, stsp@, reyk@
OK bluhm@, reyk@

ensure http_path is escaped before using it in Location redirection.

OK reyk@

Fix check against NULL which was reverted by accident in r1.56.

ok reyk@

Allow to change the default media type globally or per-location,
eg. default type text/html.

OK florian@

Adjust server_file_modified_since() to our style. Please keep httpd clean.

spacing

Implement If-Modified-Since.
From Kyle Thompson <jmp AT giga DOT moe>.
Tweaks by me.
OK benno@

Implement byte ranges.
From Sunil Nimmagadda <sunil At nimmagadda DOT net>
OK benno@

Prepend files or directories containing ":" with "./" in directory
indexes as per RFC 3986:
A path segment that contains a colon character (e.g., "this:that")
cannot be used as the first segment of a relative-path reference, as
it would be mistaken for a scheme name. Such a segment must be
preceded by a dot-segment (e.g., "./this:that") to make a relative-
path reference.

While here add a "/" to the end of directory names, this saves us one
redirect round trip.

Found the hard way & "functionality wise, OK" ajacoutot@
RFC pointer & OK benno@

Rename escape_uri() to url_encode() because it is the opposite of
url_decode(). No functional change.

Encode directory listings.
Problem pointed out by remco AT d-compu.dyndns.org some time ago.
Input / OK reyk@

spacing

httpd is based on relayd and had included many headers that are only
needed by its ancestor. jsg@, include-what-you-use, and some manual
review helped to cleanup the headers (take iwyu with a grain of salt).
Based on common practice, httpd.h now also includes the necessary
headers for itself.

OK florian@

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

bump copyright year

Make httpd return "404 not found" if an intermediate component of a requested
file path does not exist rather than returning "500 internal server error".
ok reyk

add new url stripping option:

strip number
Strip number path components from the beginning of the
request URI before looking up the stripped-down URI at
the document root.


reviewed with much patience and OK by reyk@

Use the HTML5 doctype for error and auto index pages because it is
shorter, newer, and the recommendation. From James Jerkins.

Exclude the charset for now because it is not explicitly handled by httpd.

OK validator.w3.org (This document was successfully checked as HTML5!)

Stop pulling in <arpa/inet.h> or <arpa/nameser.h> when unnecessary.
*Do* pull it in when in_{port,addr}_h is needed and <netinet/in.h> isn't.

ok reyk@

Like previously done in relayd, change the keyword "ssl" to "tls" to
reflect reality.

OK benno@

stop viral header propagation. none of this code uses sys/hash.h
from Max Fillinger

Remove unnecessary netinet/in_systm.h include.

ok millert@

Rework the error message a little bit: Do not send details of the
error. Traditionally, web servers responsed with the request path on
40x errors which could be abused to inject JavaScript etc. Instead of
sanitizing the path, we just don't reprint it. Also modify the style
a little bit but keep Comic Sans.

With input from Jonas Lindemann and doug@

Fix an error case that was never handled ending up in an endless event
loop that could eat all CPU. I thought that the previous (correct)
commit fixed it which wasn't the case. But this one is obvious.

ok florian@

In addition to READ, disable WRITE events when closing the file
descriptor of the file I/O bufferevent. This fixes a potential event flood.

OK florian@

Use two instead of one http descriptor for request and response.

OK chrisz@

Add Last-Modified: HTTP header.

OK reyk@

Remove obsolete struct stat parameters.

ok reyk@

When opening directories, re-match the location after the index file
has been appended. This allows to use a fastcgi target as the default
index, for example index.php.

OK florian@

branches: 1.31.4;
The watermark exposed a bug in server_write that broke keep-alive
support. Instead of calling server_close from server_write, we have
to proceed to the next connection by calling the error handler.

OK jsg@

Adjust the read/write watermarks according to the TCP send buffer.
This fixes sending of large files. Previously, httpd was reading the
input file too quickly and could run out of memory when filling the
input buffer.

Found by jsg@
OK florian@

Redirect to https:// if SSL/TLS is enabled.

httpd doesn't support SSL/TLS yet, remove the remaining bits.
The secrect plan is to add it later using the ressl wrapper library.

Only allow GET and HEAD for static files or return 405.

ok florian@

Also write log messages, like 404 Not Found, on error. This is a bit
tricky because we couldn't guarantee a sane state after
server_response_http() so fail hard afterwards and close the connection.

ok doug@

Split fastcgi socket path and document root option and add the
SCRIPT_FILENAME CGI param with a prepended root. This fixes php-fpm
that expects SCRIPT_FILENAME and also works with slowcgi if you
configure the root correctly. For example, if SCRIPT_NAME and
REQUEST_URI are /php/index.php, root is /htdocs, SCRIPT_FILENAME will
be /htdocs/php/index.php. As tested and discussed with florian@

spacing

scandir(3)-based directory auto index didn't work on NFS because the
file system is not filling in d_type properly. Using st_mode from the
stat call fixes the problem, eg. S_ISDIR(st.st_mode) instead of
dp->d_type == DT_DIR. Pointed out by pelikan@

some fastcgi improvements:
- DPRINTF instead of log_info for internal debugging.
- submit QUERY_STRING, if it exists
- use a proper function to create an HTTP header.
- use server_file_error() to detect EOF and fastcgi stream errors.
- disable keep-alive/persist for now until we have a reliable way to
get the content length from the cgi response or support chunked
encoding.

"Cool, jep" florian@

One bufferevent can be shared by file and fcgi.

Rename the "docroot" variable to "path" because it will be used for
either files or the fastcgi socket (and there's no need to use a union yet).

Reserve an extra file descriptor per connection instead of per
request. This fixes fd accounting with persistent connections and
reduces the complexity of the implementation.

ok benno@

Add extended directory index options: "[no] index" and "[no] auto index".
The option "directory auto index" implements basic directory listing
and is turned off by default.

ok deraadt@

Remove redundant slash

It is recommended to use a URL in the Location header of 3xx
responses. To accomplish this, add some semantics to retrieve the
server host name of a connection: either IP, IP:PORT (if not 80) or
[IP6]:PORT, or Host value (if valid).

Canonicalize the request path once without the docroot and prepend the
docroot only only when it's needed. Suggested by deraadt@.

Don't leak docroot in the error message if the default index file is missing.

OK florian@

Split server and server_config.

Rename a field, needed later, no functional change.

Move the docroot into the server block.

The default index page shouldn't be a directory. It's a 500.

Don't expose the docroot on error.

First attempt at verifying the request path and the access
permissions. We also have to redirect with 301 if a directory name
was requested without the trailing slash.

Add canonicalize_path() to canonicalize the requested URL path.

Implement file descriptor accounting. The concept was taken from
relayd but had to be adjusted for httpd. It now handles single-pass
HTTP connections, persistent connections with multiple requests, and
body-less HEAD requests. With input from benno@

don't diplay the full path in error messages

first step towards keep-alive/persistent connections support

Finish writing the output before closing the connection
(adopted from relayd).

Add support for media types (aka. MIME types): the types section is
compatible to nginx' mime.types file which can be included directly.
If not present, use a few built-in defaults for html, css, txt, jpeg,
gif, png, and js.

Add httpd(8), an attempt to turn the relayd(8) codebase into a simple
web server. It is not finished yet and I just started it today, but
the goal is to provide an HTTP server that a) provides minimal
features, b) serves static files, c) provides FastCGI support, and d)
follows common coding practices of OpenBSD.

It will neither support plugins, nor custom memory allocators, EBCDIC
support, PCRE or any other things that can be found elsewhere.
httpd(8) is not intended to provide a fully-featured replacement for
nginx(8) or the Apache, but it will provide enough functionality that
is needed in the OpenBSD base system.

ok deraadt@

plug some memory leaks in server_file_index when failures occur

namelist and its entries are not freed if escape_html fails or if we
fail in the inner loop. Move scandir later so it's closer to the for
loop and handle escape_html and url_encode failures.

With lots of help from tb, thanks!

ok tb@

Only provide the .gz file if timestamp is >= the non-gzip file. Make
sure it is a regular file. Use the timestamp of the non-gzip file as
the last-modified timestamp, as proposed by claudio.
ok claudio bluhm millert

struct stat from early file inspection was being used after actual file
open() which means the stat could refer to the wrong file. Mostly this
relates to st_size use. This bug could mean that httpd sends new files
truncated to the old length, saying "I am sure you have the correct file now"?
Could have other bad effects.
ok tb millert bluhm

Simplify .gz handling a bit

Combine strlcpy + strlcat into a single snprintf and remove a few
unnecessary parentheses.

ok deraadt millert

Add gzip-static option to httpd. This allows to deliver precompressed
files with content-encoding gzip.
from prx at si3t dot ch; OK tracey@

Use relative reference URIs in Location header on directory redirects.

This adds support for front-ending httpd(8) with a TLS-terminating
gateway like relayd(8) that forwards unencrypted http traffic.

Previously httpd(8) would use a full URL in the Location header in 301
redirects when a user-agent requests a directory but without the
trailing '/'. If the user-agent originally connected with https, this
caused the redirected url to be http.

This change conforms to RFC7231 section 7.1.2.

Reported by Vincent Lee.

OK claudio@

A socket buffer is not the best size to read from a disk.
Use st_blksize to set high water mark; florian@

Use the simpler HTML5 idiom to declare charset in autogenerated pages.

This came from a suggestion by Andras Farkas to replace use of
XHTML self-closing tags.

ok cwen@ danj@ florian@

httpd: add a "dark mode" in directory listings and error pages

Introduce a new "dark mode" for httpd(8) directory listings and error pages,
using the prefers-color-scheme css function. It uses the colorscheme already
used in OpenBSD project websites.

This version will only be displayed if the user's browser specifically sends
a "prefer darker themes if available" request.

Input from florian@, danj@ and clematis.

OK florian@, danj@ (on a previous version with the wrong background color)

Fix 304 Not Modified response: don't send a body, use the correct MIME type.

Reported by Hidvegi Gabor gaborca websivision hu

Fix found by anton@

OK anton@

Fix support for HTTP pipelining by handling all requests in the buffer.

Tested & OK jung@

Reimplement httpd's support for byte ranges.

The previous implementation loaded all the output into a single output
buffer and used its size to determine the Content-Length of the body.

The new implementation calculates the body length first and writes the
individual ranges in an async way using the bufferevent mechanism.

This prevents httpd from using too much memory and applies the
watermark and throttling mechanisms to range requests.

Problem reported by Pierre Kim (pierre.kim.sec at gmail.com)

OK benno@ sunil@

Fix error path of range requests, found while reviewing byte range support.

OK jsg@

branches: 1.62.2;
Repair some file descriptor leaks.
ok beck krw millert

Set content charset for auto index generated page.
Pointed out and diff by dhill, thanks!
Tweaks and same change for error documents by me.

branches: 1.60.2;
Fix rev 1.70 of server.c by only re-enabling the bufferevent if we
previously disabled it because we were reading to fast (from disk).
Problem noted and tracked down to that commit by weerd@ and
independently by stsp@.
Tested by weerd@, stsp@, reyk@
OK bluhm@, reyk@

ensure http_path is escaped before using it in Location redirection.

OK reyk@

Fix check against NULL which was reverted by accident in r1.56.

ok reyk@

Allow to change the default media type globally or per-location,
eg. default type text/html.

OK florian@

Adjust server_file_modified_since() to our style. Please keep httpd clean.

spacing

Implement If-Modified-Since.
From Kyle Thompson <jmp AT giga DOT moe>.
Tweaks by me.
OK benno@

Implement byte ranges.
From Sunil Nimmagadda <sunil At nimmagadda DOT net>
OK benno@

Prepend files or directories containing ":" with "./" in directory
indexes as per RFC 3986:
A path segment that contains a colon character (e.g., "this:that")
cannot be used as the first segment of a relative-path reference, as
it would be mistaken for a scheme name. Such a segment must be
preceded by a dot-segment (e.g., "./this:that") to make a relative-
path reference.

While here add a "/" to the end of directory names, this saves us one
redirect round trip.

Found the hard way & "functionality wise, OK" ajacoutot@
RFC pointer & OK benno@

Rename escape_uri() to url_encode() because it is the opposite of
url_decode(). No functional change.

Encode directory listings.
Problem pointed out by remco AT d-compu.dyndns.org some time ago.
Input / OK reyk@

spacing

httpd is based on relayd and had included many headers that are only
needed by its ancestor. jsg@, include-what-you-use, and some manual
review helped to cleanup the headers (take iwyu with a grain of salt).
Based on common practice, httpd.h now also includes the necessary
headers for itself.

OK florian@

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

bump copyright year

Make httpd return "404 not found" if an intermediate component of a requested
file path does not exist rather than returning "500 internal server error".
ok reyk

add new url stripping option:

strip number
Strip number path components from the beginning of the
request URI before looking up the stripped-down URI at
the document root.


reviewed with much patience and OK by reyk@

Use the HTML5 doctype for error and auto index pages because it is
shorter, newer, and the recommendation. From James Jerkins.

Exclude the charset for now because it is not explicitly handled by httpd.

OK validator.w3.org (This document was successfully checked as HTML5!)

Stop pulling in <arpa/inet.h> or <arpa/nameser.h> when unnecessary.
*Do* pull it in when in_{port,addr}_h is needed and <netinet/in.h> isn't.

ok reyk@

Like previously done in relayd, change the keyword "ssl" to "tls" to
reflect reality.

OK benno@

stop viral header propagation. none of this code uses sys/hash.h
from Max Fillinger

Remove unnecessary netinet/in_systm.h include.

ok millert@

Rework the error message a little bit: Do not send details of the
error. Traditionally, web servers responsed with the request path on
40x errors which could be abused to inject JavaScript etc. Instead of
sanitizing the path, we just don't reprint it. Also modify the style
a little bit but keep Comic Sans.

With input from Jonas Lindemann and doug@

Fix an error case that was never handled ending up in an endless event
loop that could eat all CPU. I thought that the previous (correct)
commit fixed it which wasn't the case. But this one is obvious.

ok florian@

In addition to READ, disable WRITE events when closing the file
descriptor of the file I/O bufferevent. This fixes a potential event flood.

OK florian@

Use two instead of one http descriptor for request and response.

OK chrisz@

Add Last-Modified: HTTP header.

OK reyk@

Remove obsolete struct stat parameters.

ok reyk@

When opening directories, re-match the location after the index file
has been appended. This allows to use a fastcgi target as the default
index, for example index.php.

OK florian@

branches: 1.31.4;
The watermark exposed a bug in server_write that broke keep-alive
support. Instead of calling server_close from server_write, we have
to proceed to the next connection by calling the error handler.

OK jsg@

Adjust the read/write watermarks according to the TCP send buffer.
This fixes sending of large files. Previously, httpd was reading the
input file too quickly and could run out of memory when filling the
input buffer.

Found by jsg@
OK florian@

Redirect to https:// if SSL/TLS is enabled.

httpd doesn't support SSL/TLS yet, remove the remaining bits.
The secrect plan is to add it later using the ressl wrapper library.

Only allow GET and HEAD for static files or return 405.

ok florian@

Also write log messages, like 404 Not Found, on error. This is a bit
tricky because we couldn't guarantee a sane state after
server_response_http() so fail hard afterwards and close the connection.

ok doug@

Split fastcgi socket path and document root option and add the
SCRIPT_FILENAME CGI param with a prepended root. This fixes php-fpm
that expects SCRIPT_FILENAME and also works with slowcgi if you
configure the root correctly. For example, if SCRIPT_NAME and
REQUEST_URI are /php/index.php, root is /htdocs, SCRIPT_FILENAME will
be /htdocs/php/index.php. As tested and discussed with florian@

spacing

scandir(3)-based directory auto index didn't work on NFS because the
file system is not filling in d_type properly. Using st_mode from the
stat call fixes the problem, eg. S_ISDIR(st.st_mode) instead of
dp->d_type == DT_DIR. Pointed out by pelikan@

some fastcgi improvements:
- DPRINTF instead of log_info for internal debugging.
- submit QUERY_STRING, if it exists
- use a proper function to create an HTTP header.
- use server_file_error() to detect EOF and fastcgi stream errors.
- disable keep-alive/persist for now until we have a reliable way to
get the content length from the cgi response or support chunked
encoding.

"Cool, jep" florian@

One bufferevent can be shared by file and fcgi.

Rename the "docroot" variable to "path" because it will be used for
either files or the fastcgi socket (and there's no need to use a union yet).

Reserve an extra file descriptor per connection instead of per
request. This fixes fd accounting with persistent connections and
reduces the complexity of the implementation.

ok benno@

Add extended directory index options: "[no] index" and "[no] auto index".
The option "directory auto index" implements basic directory listing
and is turned off by default.

ok deraadt@

Remove redundant slash

It is recommended to use a URL in the Location header of 3xx
responses. To accomplish this, add some semantics to retrieve the
server host name of a connection: either IP, IP:PORT (if not 80) or
[IP6]:PORT, or Host value (if valid).

Canonicalize the request path once without the docroot and prepend the
docroot only only when it's needed. Suggested by deraadt@.

Don't leak docroot in the error message if the default index file is missing.

OK florian@

Split server and server_config.

Rename a field, needed later, no functional change.

Move the docroot into the server block.

The default index page shouldn't be a directory. It's a 500.

Don't expose the docroot on error.

First attempt at verifying the request path and the access
permissions. We also have to redirect with 301 if a directory name
was requested without the trailing slash.

Add canonicalize_path() to canonicalize the requested URL path.

Implement file descriptor accounting. The concept was taken from
relayd but had to be adjusted for httpd. It now handles single-pass
HTTP connections, persistent connections with multiple requests, and
body-less HEAD requests. With input from benno@

don't diplay the full path in error messages

first step towards keep-alive/persistent connections support

Finish writing the output before closing the connection
(adopted from relayd).

Add support for media types (aka. MIME types): the types section is
compatible to nginx' mime.types file which can be included directly.
If not present, use a few built-in defaults for html, css, txt, jpeg,
gif, png, and js.

Add httpd(8), an attempt to turn the relayd(8) codebase into a simple
web server. It is not finished yet and I just started it today, but
the goal is to provide an HTTP server that a) provides minimal
features, b) serves static files, c) provides FastCGI support, and d)
follows common coding practices of OpenBSD.

It will neither support plugins, nor custom memory allocators, EBCDIC
support, PCRE or any other things that can be found elsewhere.
httpd(8) is not intended to provide a fully-featured replacement for
nginx(8) or the Apache, but it will provide enough functionality that
is needed in the OpenBSD base system.

ok deraadt@

Only provide the .gz file if timestamp is >= the non-gzip file. Make
sure it is a regular file. Use the timestamp of the non-gzip file as
the last-modified timestamp, as proposed by claudio.
ok claudio bluhm millert

struct stat from early file inspection was being used after actual file
open() which means the stat could refer to the wrong file. Mostly this
relates to st_size use. This bug could mean that httpd sends new files
truncated to the old length, saying "I am sure you have the correct file now"?
Could have other bad effects.
ok tb millert bluhm

Simplify .gz handling a bit

Combine strlcpy + strlcat into a single snprintf and remove a few
unnecessary parentheses.

ok deraadt millert

Add gzip-static option to httpd. This allows to deliver precompressed
files with content-encoding gzip.
from prx at si3t dot ch; OK tracey@

Use relative reference URIs in Location header on directory redirects.

This adds support for front-ending httpd(8) with a TLS-terminating
gateway like relayd(8) that forwards unencrypted http traffic.

Previously httpd(8) would use a full URL in the Location header in 301
redirects when a user-agent requests a directory but without the
trailing '/'. If the user-agent originally connected with https, this
caused the redirected url to be http.

This change conforms to RFC7231 section 7.1.2.

Reported by Vincent Lee.

OK claudio@

A socket buffer is not the best size to read from a disk.
Use st_blksize to set high water mark; florian@

Use the simpler HTML5 idiom to declare charset in autogenerated pages.

This came from a suggestion by Andras Farkas to replace use of
XHTML self-closing tags.

ok cwen@ danj@ florian@

httpd: add a "dark mode" in directory listings and error pages

Introduce a new "dark mode" for httpd(8) directory listings and error pages,
using the prefers-color-scheme css function. It uses the colorscheme already
used in OpenBSD project websites.

This version will only be displayed if the user's browser specifically sends
a "prefer darker themes if available" request.

Input from florian@, danj@ and clematis.

OK florian@, danj@ (on a previous version with the wrong background color)

Fix 304 Not Modified response: don't send a body, use the correct MIME type.

Reported by Hidvegi Gabor gaborca websivision hu

Fix found by anton@

OK anton@

Fix support for HTTP pipelining by handling all requests in the buffer.

Tested & OK jung@

Reimplement httpd's support for byte ranges.

The previous implementation loaded all the output into a single output
buffer and used its size to determine the Content-Length of the body.

The new implementation calculates the body length first and writes the
individual ranges in an async way using the bufferevent mechanism.

This prevents httpd from using too much memory and applies the
watermark and throttling mechanisms to range requests.

Problem reported by Pierre Kim (pierre.kim.sec at gmail.com)

OK benno@ sunil@

Fix error path of range requests, found while reviewing byte range support.

OK jsg@

branches: 1.62.2;
Repair some file descriptor leaks.
ok beck krw millert

Set content charset for auto index generated page.
Pointed out and diff by dhill, thanks!
Tweaks and same change for error documents by me.

branches: 1.60.2;
Fix rev 1.70 of server.c by only re-enabling the bufferevent if we
previously disabled it because we were reading to fast (from disk).
Problem noted and tracked down to that commit by weerd@ and
independently by stsp@.
Tested by weerd@, stsp@, reyk@
OK bluhm@, reyk@

ensure http_path is escaped before using it in Location redirection.

OK reyk@

Fix check against NULL which was reverted by accident in r1.56.

ok reyk@

Allow to change the default media type globally or per-location,
eg. default type text/html.

OK florian@

Adjust server_file_modified_since() to our style. Please keep httpd clean.

spacing

Implement If-Modified-Since.
From Kyle Thompson <jmp AT giga DOT moe>.
Tweaks by me.
OK benno@

Implement byte ranges.
From Sunil Nimmagadda <sunil At nimmagadda DOT net>
OK benno@

Prepend files or directories containing ":" with "./" in directory
indexes as per RFC 3986:
A path segment that contains a colon character (e.g., "this:that")
cannot be used as the first segment of a relative-path reference, as
it would be mistaken for a scheme name. Such a segment must be
preceded by a dot-segment (e.g., "./this:that") to make a relative-
path reference.

While here add a "/" to the end of directory names, this saves us one
redirect round trip.

Found the hard way & "functionality wise, OK" ajacoutot@
RFC pointer & OK benno@

Rename escape_uri() to url_encode() because it is the opposite of
url_decode(). No functional change.

Encode directory listings.
Problem pointed out by remco AT d-compu.dyndns.org some time ago.
Input / OK reyk@

spacing

httpd is based on relayd and had included many headers that are only
needed by its ancestor. jsg@, include-what-you-use, and some manual
review helped to cleanup the headers (take iwyu with a grain of salt).
Based on common practice, httpd.h now also includes the necessary
headers for itself.

OK florian@