Avoid a NULL dereference when handling a malformed fastcgi request.

Rework the hack to avoid a use-after-free in the fastcgi code.
Since server_fcgi() can be called by server_read_httpcontent() we
can't set clt_fcgi_error to NULL. Instead, we implement a simple
reference count to track when a fastcgi session is in progress to
avoid closing the http session prematurely on fastcgi error.
Based on a diff from and OK by tb@. Reported by Ben Kallus.

Work around use after free in httpd(8)

A malformed HTTP request can cause httpd in fastcgi mode to crash due to a
use-after-free. This is an awful hack, but it's good enough until someone
figures out the correct way of dealing with server_close() here.

"this will do the trick for now" claudio
ok beck deraadt

branches: 1.95.2; 1.95.6;
Neither clt_descreq nor clt_descresp in struct client need to be void *.
They both are only used as struct http_descriptor.
OK tb@

For FCGI_END_REQUEST reset the clt struct similar to what is done in the
file and other cases. Especially when the session uses keep-alive it is
important to set TOREAD_HTTP_HEADER so that the state machine knows what's
next.
OK op@

Use break instead of return so that a HEAD request still consumes all data.
OK op@

fix regression introduced in previous commit. HEAD replies don't have a
body so server_fcgi_error shouldn't print the end marker.

OK claudio@

correctly handle an abnormal fastcgi termination. httpd handles the
disconnection from the fastcgi application via server_file_error which
assumes that the reply was completey done. However, if the fastcgi
reply wasn't complete (e.g. because slowcgi hit the timeout) the HTTP
client are left "hanging" and waiting for a reply until they give up.

This adds a server_fcgi_error callback to handle the "no headers" and
"incomplete data" cases and properly close the reply before falling back
to server_file_error.

OK claudio@

Nothing uses kv_flags.

John (j AT bitminer.ca) pointed out that we didn't correctly
initialize struct kv and might use slower KV_FLAG_GLOBBING path in
kv_find depending on stack garbage. Instead of fixing the
initialization just delete kv_flags from struct kv.

OK claudio, tb

* stop sending the content for head requests, even when its supplied by the
fcgi. Required by RFC 7231 and RFC 3875 section 4.3.2.
* If the client sends an empty body without a Content-Lenght:
do not add the Content-Lenght if it's a HEAD request.
If it's a HEAD request, the Content-Lenght should show the size of the
equivalent GET request, but we don't know how much that will be so
don't lie.

found by and fix suggested by Ross L Richardson, Thanks!

Additionally:

* when the fcgi supplies a Content-Length header, do not remove it and
set Transfer-Encoding: chunked. Instead, leave the Content-Lenght
header in place, as obviously the fcgi knows how much data will come.

ok claudio@

Fix previous.
Only set Content-Length when we no the body is empty and we disable
chunked encoding. Otherwise we break the nextcloud app again :/
Pointed out by Matthias Pressfreund, thanks!

When we disable "Transfer-Encoding: chunked" in the fastcgi backend
because we are going to send an empty body we have to provide
"Content-Length: 0" otherwise some browsers (Firefox, Safari) just
hang until httpd(8) closes the connection.
Problem reported by Matthias Pressfreund, debugged with weerd@ who
pointed out that the problem is browser dependent.
OK tracey

Do not try to chunk encode an empty http body coming from an fcgi
upstream.

Found the hard way by Chris Narkiewicz who tracked failing uploads in
the nextcloud mobile app down to httpd(8) trying to chunk encode a
"204 No Content" resonse.

Testing by Steve Williams
Testing & OK stsp

Remove outdated comment. We got all httpd headers from the fcgi server
at this point.

Use the original requested URI for REQUEST_URI.

ok millert florian

Add support for non-localhost fastcgi sockets.

Lots of review time kn@
Lots of review time, tweaks, and ok florian@

remove unused assignment
from Ross L Richardson <openbsd AT rlr DOT id DOT au>, Thanks
ok claudio@

Implement "strip" option for fastcgi to be able to have multiple chroots
under /var/www for FastCGI servers.
From Nazar Zhuk (nazar AT zhuk DOT online), thanks!
Ok benno

Set the REQUEST_URI CGI variable to the requested URI and query string
instead of the rewritten path and query string.

Patch from Tim Baumgard, reminded by Mischa Peters.

ok benno, reyk

spacing

httpd(8): add support for setting custom FastCGI parameters.

This commit extends the existing grammar by adding the param option
to the fastcgi directive: fastcgi param name value.

Example usage:
fastcgi param VAR1 hello
fastcgi param VAR2 world

With help and OK florian@
Rogue manpage bits, feel free to modify them.

Omit HSTS headers over unencrypted connections, per RFC 6797.

ok florian@

Add support for client certificate authentication to httpd.

From Jack Burton <jack at saosce dot com dot au> - thanks!

Also tested by Jan Klemkow <j.klemkow at wemelug dot de>.

ok beck@ reyk@

Don't set HTTP date header if already set.

Thanks Nick Owens

OK florian@

The POSIX APIs that that sockaddrs all ignore the s*_len field in the
incoming socket, so userspace doesn't need to set it unless it has its
own reasons for tracking the size along with the sockaddr.

ok phessler@ deraadt@ florian@

The strchr() call either returns a NULL pointer, on which the code will
break out of the loop, or a pointer to ':'. Thus the extra check for
':' is unnecessary and can be removed.

ok jung@

Empty lines cause server_fcgi_getheaders() to immediately return.
Unfortunately in that case the line was not freed. This lead to a
memleak on each request. Thus, save the return value prior to
returning, free the line and return the saved value.

ok jung@

struct client starts to become the kitchen sink. Move fastcgi data to
its own struct. Requested by and OK reyk@

spacing

Do not assume that the full http response header is in the first
fastcgi stdout record. Keep processing stdout records until we found
the header / body separator and only then generate the header
response.
Problem reported by many.

OK jung@

Always pass QUERY_STRING variable.
According to the RFC it is empty when no query string was found.
From Tim Baumgard <openbsd@bmgrd.com>o

ok florian@

Retire socket_set_blockmode() in favor of the SOCK_NONBLOCK type flag.
As done in iked and snmpd.

OK jung@

fix an fd leak if socket connection fails; from Carlin Bingham
ok reyk@

fix a typo; from Carlin Bingham

Change httpd(8) to use C99-style fixed-width integers (uintN_t instead
of u_intN_t) and replace u_int with unsigned int. Mixing both
variants is a bad style and most contributors seem to prefer this
style; it also helps us to get used to it, portability, and
standardization.

Theoretically no binary change, except one in practice: httpd.o has a
different checksum because gcc with -O2 pads/optimizes "struct
privsep" differently when using "unsigned int" instead "u_int" for the
affected members. "u_int" is just a typedef of "unsigned int", -O0
doesn't build the difference and clang with -O2 doesn't do it either -
it is just another curiosity from gcc-land.

OK semarie@

Fix rev 1.70 of server.c by only re-enabling the bufferevent if we
previously disabled it because we were reading to fast (from disk).
Problem noted and tracked down to that commit by weerd@ and
independently by stsp@.
Tested by weerd@, stsp@, reyk@
OK bluhm@, reyk@

repair hsts header output, wrong format strings caused broken
Strict-Transport-Security headers. Add __format__ attribute to
kv_set() and kv_setkey() to make it easier to spot such problems.

Found by and fix from Donovan Watteau <tsoomi -AT- gmail -DOT- com>,
thanks for your help.

ok deraadt@

backout the previous: it broke wordpress somehow.
we need more care to find a proper fix for the fastcgi headers.

acknowledged by deraadt@

Read fcgi response records until we have the whole http header and can
parse it. Otherwise http headers can leak into the body.
Pointed out by Jean-Philippe Ouellet on bugs@ Thanks!
OK reyk, commit ASAP deraadt@

add HSTS to fcgi responses
OK reyk

handle error returns from bufferevent_write()

ok florian@

remove XXX and handle error return from evbuffer_add()

ok florian@

treat asprintf failure in REQUEST_URI case as a fatal error

ok florian@

According to RFC 3875 PATH_INFO should either contain a full path or
be empty (""). It was not set at all when there is nothing to set
which caused problems with some FastCGI applications (like
Flask/Python through uWSGI).

From hrkfdn via github

plug fd leak found by Todd Mortimer

ok claudio deraadt florian

Allow more characters in CGI environment variables as specified by RFC
7230 and RFC 3875.
sthen@ suggested to add a comment to explain where the list of
characters is coming from.
Found the hard way and initial diff from Tim van der Molen (tbvdm at
xs4all), thanks! Some more allowed characters added by me.
OK sthen@

Use the rewritten (index file appended) uri as DOCUMENT_URI.

OK florian@

Ooops, no need to include sys/cdefs.h.

Pointed out by florian@

httpd is based on relayd and had included many headers that are only
needed by its ancestor. jsg@, include-what-you-use, and some manual
review helped to cleanup the headers (take iwyu with a grain of salt).
Based on common practice, httpd.h now also includes the necessary
headers for itself.

OK florian@

s/clt_fcgi_remote_user/clt_remote_user/
OK reyk@

Decouple auth parameters from struct server_config into struct auth.

OK florian@

First stab at implementing basic auth.
Currently the htpasswd file needs to be in the chroot; will hopefully
improved soonish.
Based on a diff from Oscar Linderholm many months ago but turned into
a complete rewrite.
input/OK reyk@

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

Abort if fcgi_chunked is not true to avoid sending additional garbage
after the response.

Found by Erik Lax

ok florian@

add new url stripping option:

strip number
Strip number path components from the beginning of the
request URI before looking up the stripped-down URI at
the document root.


reviewed with much patience and OK by reyk@

Stop pulling in <arpa/inet.h> or <arpa/nameser.h> when unnecessary.
*Do* pull it in when in_{port,addr}_h is needed and <netinet/in.h> isn't.

ok reyk@

Like previously done in relayd, change the keyword "ssl" to "tls" to
reflect reality.

OK benno@

stop viral header propagation. none of this code uses sys/hash.h
from Max Fillinger

Remove unnecessary netinet/in_systm.h include.

ok millert@

whitespace spotted while studying the code

FastCGI did not support persistent connections. Add initial support
for persistent connections with FastCGI by implementing chunked
Transfer-Encoding. This only works with HTTP/1.1.

With input and help from florian@ who found some FastCGI edge cases.

OK florian@

Don't pass the local buffer array by reference.

OK florian@

Replace the code to get the FastCGI Status header with a proper way to
parse and write the headers using the http response descriptor. This
allows to add other tweaks, like support for chunked encoding, later.

OK florian@

Use two instead of one http descriptor for request and response.

OK chrisz@

Add Last-Modified: HTTP header.

OK reyk@

For a non-existent root we don't want the root prefix to show up in
PATH_INFO.
Therefore put a lower bound of strlen(root) on scriptlen.
This makes perfect sense for virtual FastCGI scripts which run chrooted
in another directory from httpd.


ok reyk@

Provide a failsafe version of the path_info() function that doesn't
need a temporary path variable. Based on an initial diff from
chrisz@.

"Commit any failsafe version and I'm ok with it" chrisz@

make a few variables more local

When opening directories, re-match the location after the index file
has been appended. This allows to use a fastcgi target as the default
index, for example index.php.

OK florian@

branches: 1.29.2;
Don't try to ouput FCGI_STDERR into error.log if there is no data.
Problem noticed by naddy@, OK reyk@

Opportunistically try to parse "Status: $code" in the very first
response from the fcgi daemon and use that code as HTTP response
code. If it doesn't work out fall back to code 200.
This might fix naddy@'s issue with redirects in cvsweb.
To be revisited after unlock.
Discussed with & grudgingly OK reyk@

Write STDERR from the CGI to the web server error log as intended.

OK florian@

If the very first fcgi STDOUT record has length 0 the cgi script
didn't send anything back. This is an internal server error.
OK reyk@

Always zero-out the fcgi record header for STDIN data.

OK florian@

Use memset(buf instead of memset(&buf.

Pointed out by deraadt@

http POST support
with & OK reyk@

Content-Length and Content-Type are transmitted as CONTENT_LENGTH and
CONTENT_TYPE environment variables to cgi scripts, without the HTTP_
prefix.
OK reyk@

Add HTTPS = on CGI variable.

Implement PATH_INFO and add DOCUMENT_ROOT.
PATH_INFO was requested by naddy@ who successfully tested it with "cvsweb".

ok naddy@

httpd doesn't support SSL/TLS yet, remove the remaining bits.
The secrect plan is to add it later using the ressl wrapper library.

c-type functions / makros need a cast to unsigned char, not int
"feel free to commit" reyk@

spacing

Dynamically pass HTTP request headers as protocol-specific HTTP_* CGI
meta-variables.

ok florian@

Add function to iterate all headers. No functional change.

Split fastcgi socket path and document root option and add the
SCRIPT_FILENAME CGI param with a prepended root. This fixes php-fpm
that expects SCRIPT_FILENAME and also works with slowcgi if you
configure the root correctly. For example, if SCRIPT_NAME and
REQUEST_URI are /php/index.php, root is /htdocs, SCRIPT_FILENAME will
be /htdocs/php/index.php. As tested and discussed with florian@

Add missing log call for FastCGI requests.

don't leak fcgi fd

Padding of fcgi records is optional, but if we receive padding data we
should read it.

We need to read from the fcgi bufferevent until it's empty because the
event handler will not be called again if no new data arrives.
Debugged with and OK reyk@

Allow to specify a FastCGI TCP socket on localhost (eg. :9000). Used
for debugging, you should prefer local UNIX sockets, but it helped to
find an issue that will be fixed with the next commit.

OK florian@

spacing

Rewrite fcgi_add_param and hand over a lot more http headers etc. to
the cgi script.
OK reyk@
"blanket OK" for changes in httpd for the time beeing from deraadt@

Correctly parse fcgi records if we don't get the whole record in one
bufferevent_read().
Input/OK reyk@

Only write the HTTP header for the first fastcgi chunk.

some fastcgi improvements:
- DPRINTF instead of log_info for internal debugging.
- submit QUERY_STRING, if it exists
- use a proper function to create an HTTP header.
- use server_file_error() to detect EOF and fastcgi stream errors.
- disable keep-alive/persist for now until we have a reliable way to
get the content length from the cgi response or support chunked
encoding.

"Cool, jep" florian@

One bufferevent can be shared by file and fcgi.

Allow to specify a non-default fastcgi socket.

Put in first stab at fastcgi. Very early work in progress. Putting it
in now so that we can quickly work on it in tree. Requested by reyk@.
deraadt@ is OK with this according to reyk@.

Work around use after free in httpd(8)

A malformed HTTP request can cause httpd in fastcgi mode to crash due to a
use-after-free. This is an awful hack, but it's good enough until someone
figures out the correct way of dealing with server_close() here.

"this will do the trick for now" claudio
ok beck deraadt

branches: 1.95.2; 1.95.6;
Neither clt_descreq nor clt_descresp in struct client need to be void *.
They both are only used as struct http_descriptor.
OK tb@

For FCGI_END_REQUEST reset the clt struct similar to what is done in the
file and other cases. Especially when the session uses keep-alive it is
important to set TOREAD_HTTP_HEADER so that the state machine knows what's
next.
OK op@

Use break instead of return so that a HEAD request still consumes all data.
OK op@

fix regression introduced in previous commit. HEAD replies don't have a
body so server_fcgi_error shouldn't print the end marker.

OK claudio@

correctly handle an abnormal fastcgi termination. httpd handles the
disconnection from the fastcgi application via server_file_error which
assumes that the reply was completey done. However, if the fastcgi
reply wasn't complete (e.g. because slowcgi hit the timeout) the HTTP
client are left "hanging" and waiting for a reply until they give up.

This adds a server_fcgi_error callback to handle the "no headers" and
"incomplete data" cases and properly close the reply before falling back
to server_file_error.

OK claudio@

Nothing uses kv_flags.

John (j AT bitminer.ca) pointed out that we didn't correctly
initialize struct kv and might use slower KV_FLAG_GLOBBING path in
kv_find depending on stack garbage. Instead of fixing the
initialization just delete kv_flags from struct kv.

OK claudio, tb

* stop sending the content for head requests, even when its supplied by the
fcgi. Required by RFC 7231 and RFC 3875 section 4.3.2.
* If the client sends an empty body without a Content-Lenght:
do not add the Content-Lenght if it's a HEAD request.
If it's a HEAD request, the Content-Lenght should show the size of the
equivalent GET request, but we don't know how much that will be so
don't lie.

found by and fix suggested by Ross L Richardson, Thanks!

Additionally:

* when the fcgi supplies a Content-Length header, do not remove it and
set Transfer-Encoding: chunked. Instead, leave the Content-Lenght
header in place, as obviously the fcgi knows how much data will come.

ok claudio@

Fix previous.
Only set Content-Length when we no the body is empty and we disable
chunked encoding. Otherwise we break the nextcloud app again :/
Pointed out by Matthias Pressfreund, thanks!

When we disable "Transfer-Encoding: chunked" in the fastcgi backend
because we are going to send an empty body we have to provide
"Content-Length: 0" otherwise some browsers (Firefox, Safari) just
hang until httpd(8) closes the connection.
Problem reported by Matthias Pressfreund, debugged with weerd@ who
pointed out that the problem is browser dependent.
OK tracey

Do not try to chunk encode an empty http body coming from an fcgi
upstream.

Found the hard way by Chris Narkiewicz who tracked failing uploads in
the nextcloud mobile app down to httpd(8) trying to chunk encode a
"204 No Content" resonse.

Testing by Steve Williams
Testing & OK stsp

Remove outdated comment. We got all httpd headers from the fcgi server
at this point.

Use the original requested URI for REQUEST_URI.

ok millert florian

Add support for non-localhost fastcgi sockets.

Lots of review time kn@
Lots of review time, tweaks, and ok florian@

remove unused assignment
from Ross L Richardson <openbsd AT rlr DOT id DOT au>, Thanks
ok claudio@

Implement "strip" option for fastcgi to be able to have multiple chroots
under /var/www for FastCGI servers.
From Nazar Zhuk (nazar AT zhuk DOT online), thanks!
Ok benno

Set the REQUEST_URI CGI variable to the requested URI and query string
instead of the rewritten path and query string.

Patch from Tim Baumgard, reminded by Mischa Peters.

ok benno, reyk

spacing

httpd(8): add support for setting custom FastCGI parameters.

This commit extends the existing grammar by adding the param option
to the fastcgi directive: fastcgi param name value.

Example usage:
fastcgi param VAR1 hello
fastcgi param VAR2 world

With help and OK florian@
Rogue manpage bits, feel free to modify them.

Omit HSTS headers over unencrypted connections, per RFC 6797.

ok florian@

Add support for client certificate authentication to httpd.

From Jack Burton <jack at saosce dot com dot au> - thanks!

Also tested by Jan Klemkow <j.klemkow at wemelug dot de>.

ok beck@ reyk@

Don't set HTTP date header if already set.

Thanks Nick Owens

OK florian@

The POSIX APIs that that sockaddrs all ignore the s*_len field in the
incoming socket, so userspace doesn't need to set it unless it has its
own reasons for tracking the size along with the sockaddr.

ok phessler@ deraadt@ florian@

The strchr() call either returns a NULL pointer, on which the code will
break out of the loop, or a pointer to ':'. Thus the extra check for
':' is unnecessary and can be removed.

ok jung@

Empty lines cause server_fcgi_getheaders() to immediately return.
Unfortunately in that case the line was not freed. This lead to a
memleak on each request. Thus, save the return value prior to
returning, free the line and return the saved value.

ok jung@

struct client starts to become the kitchen sink. Move fastcgi data to
its own struct. Requested by and OK reyk@

spacing

Do not assume that the full http response header is in the first
fastcgi stdout record. Keep processing stdout records until we found
the header / body separator and only then generate the header
response.
Problem reported by many.

OK jung@

Always pass QUERY_STRING variable.
According to the RFC it is empty when no query string was found.
From Tim Baumgard <openbsd@bmgrd.com>o

ok florian@

Retire socket_set_blockmode() in favor of the SOCK_NONBLOCK type flag.
As done in iked and snmpd.

OK jung@

fix an fd leak if socket connection fails; from Carlin Bingham
ok reyk@

fix a typo; from Carlin Bingham

Change httpd(8) to use C99-style fixed-width integers (uintN_t instead
of u_intN_t) and replace u_int with unsigned int. Mixing both
variants is a bad style and most contributors seem to prefer this
style; it also helps us to get used to it, portability, and
standardization.

Theoretically no binary change, except one in practice: httpd.o has a
different checksum because gcc with -O2 pads/optimizes "struct
privsep" differently when using "unsigned int" instead "u_int" for the
affected members. "u_int" is just a typedef of "unsigned int", -O0
doesn't build the difference and clang with -O2 doesn't do it either -
it is just another curiosity from gcc-land.

OK semarie@

Fix rev 1.70 of server.c by only re-enabling the bufferevent if we
previously disabled it because we were reading to fast (from disk).
Problem noted and tracked down to that commit by weerd@ and
independently by stsp@.
Tested by weerd@, stsp@, reyk@
OK bluhm@, reyk@

repair hsts header output, wrong format strings caused broken
Strict-Transport-Security headers. Add __format__ attribute to
kv_set() and kv_setkey() to make it easier to spot such problems.

Found by and fix from Donovan Watteau <tsoomi -AT- gmail -DOT- com>,
thanks for your help.

ok deraadt@

backout the previous: it broke wordpress somehow.
we need more care to find a proper fix for the fastcgi headers.

acknowledged by deraadt@

Read fcgi response records until we have the whole http header and can
parse it. Otherwise http headers can leak into the body.
Pointed out by Jean-Philippe Ouellet on bugs@ Thanks!
OK reyk, commit ASAP deraadt@

add HSTS to fcgi responses
OK reyk

handle error returns from bufferevent_write()

ok florian@

remove XXX and handle error return from evbuffer_add()

ok florian@

treat asprintf failure in REQUEST_URI case as a fatal error

ok florian@

According to RFC 3875 PATH_INFO should either contain a full path or
be empty (""). It was not set at all when there is nothing to set
which caused problems with some FastCGI applications (like
Flask/Python through uWSGI).

From hrkfdn via github

plug fd leak found by Todd Mortimer

ok claudio deraadt florian

Allow more characters in CGI environment variables as specified by RFC
7230 and RFC 3875.
sthen@ suggested to add a comment to explain where the list of
characters is coming from.
Found the hard way and initial diff from Tim van der Molen (tbvdm at
xs4all), thanks! Some more allowed characters added by me.
OK sthen@

Use the rewritten (index file appended) uri as DOCUMENT_URI.

OK florian@

Ooops, no need to include sys/cdefs.h.

Pointed out by florian@

httpd is based on relayd and had included many headers that are only
needed by its ancestor. jsg@, include-what-you-use, and some manual
review helped to cleanup the headers (take iwyu with a grain of salt).
Based on common practice, httpd.h now also includes the necessary
headers for itself.

OK florian@

s/clt_fcgi_remote_user/clt_remote_user/
OK reyk@

Decouple auth parameters from struct server_config into struct auth.

OK florian@

First stab at implementing basic auth.
Currently the htpasswd file needs to be in the chroot; will hopefully
improved soonish.
Based on a diff from Oscar Linderholm many months ago but turned into
a complete rewrite.
input/OK reyk@

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

Abort if fcgi_chunked is not true to avoid sending additional garbage
after the response.

Found by Erik Lax

ok florian@

add new url stripping option:

strip number
Strip number path components from the beginning of the
request URI before looking up the stripped-down URI at
the document root.


reviewed with much patience and OK by reyk@

Stop pulling in <arpa/inet.h> or <arpa/nameser.h> when unnecessary.
*Do* pull it in when in_{port,addr}_h is needed and <netinet/in.h> isn't.

ok reyk@

Like previously done in relayd, change the keyword "ssl" to "tls" to
reflect reality.

OK benno@

stop viral header propagation. none of this code uses sys/hash.h
from Max Fillinger

Remove unnecessary netinet/in_systm.h include.

ok millert@

whitespace spotted while studying the code

FastCGI did not support persistent connections. Add initial support
for persistent connections with FastCGI by implementing chunked
Transfer-Encoding. This only works with HTTP/1.1.

With input and help from florian@ who found some FastCGI edge cases.

OK florian@

Don't pass the local buffer array by reference.

OK florian@

Replace the code to get the FastCGI Status header with a proper way to
parse and write the headers using the http response descriptor. This
allows to add other tweaks, like support for chunked encoding, later.

OK florian@

Use two instead of one http descriptor for request and response.

OK chrisz@

Add Last-Modified: HTTP header.

OK reyk@

For a non-existent root we don't want the root prefix to show up in
PATH_INFO.
Therefore put a lower bound of strlen(root) on scriptlen.
This makes perfect sense for virtual FastCGI scripts which run chrooted
in another directory from httpd.


ok reyk@

Provide a failsafe version of the path_info() function that doesn't
need a temporary path variable. Based on an initial diff from
chrisz@.

"Commit any failsafe version and I'm ok with it" chrisz@

make a few variables more local

When opening directories, re-match the location after the index file
has been appended. This allows to use a fastcgi target as the default
index, for example index.php.

OK florian@

branches: 1.29.2;
Don't try to ouput FCGI_STDERR into error.log if there is no data.
Problem noticed by naddy@, OK reyk@

Opportunistically try to parse "Status: $code" in the very first
response from the fcgi daemon and use that code as HTTP response
code. If it doesn't work out fall back to code 200.
This might fix naddy@'s issue with redirects in cvsweb.
To be revisited after unlock.
Discussed with & grudgingly OK reyk@

Write STDERR from the CGI to the web server error log as intended.

OK florian@

If the very first fcgi STDOUT record has length 0 the cgi script
didn't send anything back. This is an internal server error.
OK reyk@

Always zero-out the fcgi record header for STDIN data.

OK florian@

Use memset(buf instead of memset(&buf.

Pointed out by deraadt@

http POST support
with & OK reyk@

Content-Length and Content-Type are transmitted as CONTENT_LENGTH and
CONTENT_TYPE environment variables to cgi scripts, without the HTTP_
prefix.
OK reyk@

Add HTTPS = on CGI variable.

Implement PATH_INFO and add DOCUMENT_ROOT.
PATH_INFO was requested by naddy@ who successfully tested it with "cvsweb".

ok naddy@

httpd doesn't support SSL/TLS yet, remove the remaining bits.
The secrect plan is to add it later using the ressl wrapper library.

c-type functions / makros need a cast to unsigned char, not int
"feel free to commit" reyk@

spacing

Dynamically pass HTTP request headers as protocol-specific HTTP_* CGI
meta-variables.

ok florian@

Add function to iterate all headers. No functional change.

Split fastcgi socket path and document root option and add the
SCRIPT_FILENAME CGI param with a prepended root. This fixes php-fpm
that expects SCRIPT_FILENAME and also works with slowcgi if you
configure the root correctly. For example, if SCRIPT_NAME and
REQUEST_URI are /php/index.php, root is /htdocs, SCRIPT_FILENAME will
be /htdocs/php/index.php. As tested and discussed with florian@

Add missing log call for FastCGI requests.

don't leak fcgi fd

Padding of fcgi records is optional, but if we receive padding data we
should read it.

We need to read from the fcgi bufferevent until it's empty because the
event handler will not be called again if no new data arrives.
Debugged with and OK reyk@

Allow to specify a FastCGI TCP socket on localhost (eg. :9000). Used
for debugging, you should prefer local UNIX sockets, but it helped to
find an issue that will be fixed with the next commit.

OK florian@

spacing

Rewrite fcgi_add_param and hand over a lot more http headers etc. to
the cgi script.
OK reyk@
"blanket OK" for changes in httpd for the time beeing from deraadt@

Correctly parse fcgi records if we don't get the whole record in one
bufferevent_read().
Input/OK reyk@

Only write the HTTP header for the first fastcgi chunk.

some fastcgi improvements:
- DPRINTF instead of log_info for internal debugging.
- submit QUERY_STRING, if it exists
- use a proper function to create an HTTP header.
- use server_file_error() to detect EOF and fastcgi stream errors.
- disable keep-alive/persist for now until we have a reliable way to
get the content length from the cgi response or support chunked
encoding.

"Cool, jep" florian@

One bufferevent can be shared by file and fcgi.

Allow to specify a non-default fastcgi socket.

Put in first stab at fastcgi. Very early work in progress. Putting it
in now so that we can quickly work on it in tree. Requested by reyk@.
deraadt@ is OK with this according to reyk@.

Neither clt_descreq nor clt_descresp in struct client need to be void *.
They both are only used as struct http_descriptor.
OK tb@

For FCGI_END_REQUEST reset the clt struct similar to what is done in the
file and other cases. Especially when the session uses keep-alive it is
important to set TOREAD_HTTP_HEADER so that the state machine knows what's
next.
OK op@

Use break instead of return so that a HEAD request still consumes all data.
OK op@

fix regression introduced in previous commit. HEAD replies don't have a
body so server_fcgi_error shouldn't print the end marker.

OK claudio@

correctly handle an abnormal fastcgi termination. httpd handles the
disconnection from the fastcgi application via server_file_error which
assumes that the reply was completey done. However, if the fastcgi
reply wasn't complete (e.g. because slowcgi hit the timeout) the HTTP
client are left "hanging" and waiting for a reply until they give up.

This adds a server_fcgi_error callback to handle the "no headers" and
"incomplete data" cases and properly close the reply before falling back
to server_file_error.

OK claudio@

Nothing uses kv_flags.

John (j AT bitminer.ca) pointed out that we didn't correctly
initialize struct kv and might use slower KV_FLAG_GLOBBING path in
kv_find depending on stack garbage. Instead of fixing the
initialization just delete kv_flags from struct kv.

OK claudio, tb

* stop sending the content for head requests, even when its supplied by the
fcgi. Required by RFC 7231 and RFC 3875 section 4.3.2.
* If the client sends an empty body without a Content-Lenght:
do not add the Content-Lenght if it's a HEAD request.
If it's a HEAD request, the Content-Lenght should show the size of the
equivalent GET request, but we don't know how much that will be so
don't lie.

found by and fix suggested by Ross L Richardson, Thanks!

Additionally:

* when the fcgi supplies a Content-Length header, do not remove it and
set Transfer-Encoding: chunked. Instead, leave the Content-Lenght
header in place, as obviously the fcgi knows how much data will come.

ok claudio@

Fix previous.
Only set Content-Length when we no the body is empty and we disable
chunked encoding. Otherwise we break the nextcloud app again :/
Pointed out by Matthias Pressfreund, thanks!

When we disable "Transfer-Encoding: chunked" in the fastcgi backend
because we are going to send an empty body we have to provide
"Content-Length: 0" otherwise some browsers (Firefox, Safari) just
hang until httpd(8) closes the connection.
Problem reported by Matthias Pressfreund, debugged with weerd@ who
pointed out that the problem is browser dependent.
OK tracey

Do not try to chunk encode an empty http body coming from an fcgi
upstream.

Found the hard way by Chris Narkiewicz who tracked failing uploads in
the nextcloud mobile app down to httpd(8) trying to chunk encode a
"204 No Content" resonse.

Testing by Steve Williams
Testing & OK stsp

Remove outdated comment. We got all httpd headers from the fcgi server
at this point.

Use the original requested URI for REQUEST_URI.

ok millert florian

Add support for non-localhost fastcgi sockets.

Lots of review time kn@
Lots of review time, tweaks, and ok florian@

remove unused assignment
from Ross L Richardson <openbsd AT rlr DOT id DOT au>, Thanks
ok claudio@

Implement "strip" option for fastcgi to be able to have multiple chroots
under /var/www for FastCGI servers.
From Nazar Zhuk (nazar AT zhuk DOT online), thanks!
Ok benno

Set the REQUEST_URI CGI variable to the requested URI and query string
instead of the rewritten path and query string.

Patch from Tim Baumgard, reminded by Mischa Peters.

ok benno, reyk

spacing

httpd(8): add support for setting custom FastCGI parameters.

This commit extends the existing grammar by adding the param option
to the fastcgi directive: fastcgi param name value.

Example usage:
fastcgi param VAR1 hello
fastcgi param VAR2 world

With help and OK florian@
Rogue manpage bits, feel free to modify them.

Omit HSTS headers over unencrypted connections, per RFC 6797.

ok florian@

Add support for client certificate authentication to httpd.

From Jack Burton <jack at saosce dot com dot au> - thanks!

Also tested by Jan Klemkow <j.klemkow at wemelug dot de>.

ok beck@ reyk@

Don't set HTTP date header if already set.

Thanks Nick Owens

OK florian@

The POSIX APIs that that sockaddrs all ignore the s*_len field in the
incoming socket, so userspace doesn't need to set it unless it has its
own reasons for tracking the size along with the sockaddr.

ok phessler@ deraadt@ florian@

The strchr() call either returns a NULL pointer, on which the code will
break out of the loop, or a pointer to ':'. Thus the extra check for
':' is unnecessary and can be removed.

ok jung@

Empty lines cause server_fcgi_getheaders() to immediately return.
Unfortunately in that case the line was not freed. This lead to a
memleak on each request. Thus, save the return value prior to
returning, free the line and return the saved value.

ok jung@

struct client starts to become the kitchen sink. Move fastcgi data to
its own struct. Requested by and OK reyk@

spacing

Do not assume that the full http response header is in the first
fastcgi stdout record. Keep processing stdout records until we found
the header / body separator and only then generate the header
response.
Problem reported by many.

OK jung@

Always pass QUERY_STRING variable.
According to the RFC it is empty when no query string was found.
From Tim Baumgard <openbsd@bmgrd.com>o

ok florian@

Retire socket_set_blockmode() in favor of the SOCK_NONBLOCK type flag.
As done in iked and snmpd.

OK jung@

fix an fd leak if socket connection fails; from Carlin Bingham
ok reyk@

fix a typo; from Carlin Bingham

Change httpd(8) to use C99-style fixed-width integers (uintN_t instead
of u_intN_t) and replace u_int with unsigned int. Mixing both
variants is a bad style and most contributors seem to prefer this
style; it also helps us to get used to it, portability, and
standardization.

Theoretically no binary change, except one in practice: httpd.o has a
different checksum because gcc with -O2 pads/optimizes "struct
privsep" differently when using "unsigned int" instead "u_int" for the
affected members. "u_int" is just a typedef of "unsigned int", -O0
doesn't build the difference and clang with -O2 doesn't do it either -
it is just another curiosity from gcc-land.

OK semarie@

Fix rev 1.70 of server.c by only re-enabling the bufferevent if we
previously disabled it because we were reading to fast (from disk).
Problem noted and tracked down to that commit by weerd@ and
independently by stsp@.
Tested by weerd@, stsp@, reyk@
OK bluhm@, reyk@

repair hsts header output, wrong format strings caused broken
Strict-Transport-Security headers. Add __format__ attribute to
kv_set() and kv_setkey() to make it easier to spot such problems.

Found by and fix from Donovan Watteau <tsoomi -AT- gmail -DOT- com>,
thanks for your help.

ok deraadt@

backout the previous: it broke wordpress somehow.
we need more care to find a proper fix for the fastcgi headers.

acknowledged by deraadt@

Read fcgi response records until we have the whole http header and can
parse it. Otherwise http headers can leak into the body.
Pointed out by Jean-Philippe Ouellet on bugs@ Thanks!
OK reyk, commit ASAP deraadt@

add HSTS to fcgi responses
OK reyk

handle error returns from bufferevent_write()

ok florian@

remove XXX and handle error return from evbuffer_add()

ok florian@

treat asprintf failure in REQUEST_URI case as a fatal error

ok florian@

According to RFC 3875 PATH_INFO should either contain a full path or
be empty (""). It was not set at all when there is nothing to set
which caused problems with some FastCGI applications (like
Flask/Python through uWSGI).

From hrkfdn via github

plug fd leak found by Todd Mortimer

ok claudio deraadt florian

Allow more characters in CGI environment variables as specified by RFC
7230 and RFC 3875.
sthen@ suggested to add a comment to explain where the list of
characters is coming from.
Found the hard way and initial diff from Tim van der Molen (tbvdm at
xs4all), thanks! Some more allowed characters added by me.
OK sthen@

Use the rewritten (index file appended) uri as DOCUMENT_URI.

OK florian@

Ooops, no need to include sys/cdefs.h.

Pointed out by florian@

httpd is based on relayd and had included many headers that are only
needed by its ancestor. jsg@, include-what-you-use, and some manual
review helped to cleanup the headers (take iwyu with a grain of salt).
Based on common practice, httpd.h now also includes the necessary
headers for itself.

OK florian@

s/clt_fcgi_remote_user/clt_remote_user/
OK reyk@

Decouple auth parameters from struct server_config into struct auth.

OK florian@

First stab at implementing basic auth.
Currently the htpasswd file needs to be in the chroot; will hopefully
improved soonish.
Based on a diff from Oscar Linderholm many months ago but turned into
a complete rewrite.
input/OK reyk@

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

Abort if fcgi_chunked is not true to avoid sending additional garbage
after the response.

Found by Erik Lax

ok florian@

add new url stripping option:

strip number
Strip number path components from the beginning of the
request URI before looking up the stripped-down URI at
the document root.


reviewed with much patience and OK by reyk@

Stop pulling in <arpa/inet.h> or <arpa/nameser.h> when unnecessary.
*Do* pull it in when in_{port,addr}_h is needed and <netinet/in.h> isn't.

ok reyk@

Like previously done in relayd, change the keyword "ssl" to "tls" to
reflect reality.

OK benno@

stop viral header propagation. none of this code uses sys/hash.h
from Max Fillinger

Remove unnecessary netinet/in_systm.h include.

ok millert@

whitespace spotted while studying the code

FastCGI did not support persistent connections. Add initial support
for persistent connections with FastCGI by implementing chunked
Transfer-Encoding. This only works with HTTP/1.1.

With input and help from florian@ who found some FastCGI edge cases.

OK florian@

Don't pass the local buffer array by reference.

OK florian@

Replace the code to get the FastCGI Status header with a proper way to
parse and write the headers using the http response descriptor. This
allows to add other tweaks, like support for chunked encoding, later.

OK florian@

Use two instead of one http descriptor for request and response.

OK chrisz@

Add Last-Modified: HTTP header.

OK reyk@

For a non-existent root we don't want the root prefix to show up in
PATH_INFO.
Therefore put a lower bound of strlen(root) on scriptlen.
This makes perfect sense for virtual FastCGI scripts which run chrooted
in another directory from httpd.


ok reyk@

Provide a failsafe version of the path_info() function that doesn't
need a temporary path variable. Based on an initial diff from
chrisz@.

"Commit any failsafe version and I'm ok with it" chrisz@

make a few variables more local

When opening directories, re-match the location after the index file
has been appended. This allows to use a fastcgi target as the default
index, for example index.php.

OK florian@

branches: 1.29.2;
Don't try to ouput FCGI_STDERR into error.log if there is no data.
Problem noticed by naddy@, OK reyk@

Opportunistically try to parse "Status: $code" in the very first
response from the fcgi daemon and use that code as HTTP response
code. If it doesn't work out fall back to code 200.
This might fix naddy@'s issue with redirects in cvsweb.
To be revisited after unlock.
Discussed with & grudgingly OK reyk@

Write STDERR from the CGI to the web server error log as intended.

OK florian@

If the very first fcgi STDOUT record has length 0 the cgi script
didn't send anything back. This is an internal server error.
OK reyk@

Always zero-out the fcgi record header for STDIN data.

OK florian@

Use memset(buf instead of memset(&buf.

Pointed out by deraadt@

http POST support
with & OK reyk@

Content-Length and Content-Type are transmitted as CONTENT_LENGTH and
CONTENT_TYPE environment variables to cgi scripts, without the HTTP_
prefix.
OK reyk@

Add HTTPS = on CGI variable.

Implement PATH_INFO and add DOCUMENT_ROOT.
PATH_INFO was requested by naddy@ who successfully tested it with "cvsweb".

ok naddy@

httpd doesn't support SSL/TLS yet, remove the remaining bits.
The secrect plan is to add it later using the ressl wrapper library.

c-type functions / makros need a cast to unsigned char, not int
"feel free to commit" reyk@

spacing

Dynamically pass HTTP request headers as protocol-specific HTTP_* CGI
meta-variables.

ok florian@

Add function to iterate all headers. No functional change.

Split fastcgi socket path and document root option and add the
SCRIPT_FILENAME CGI param with a prepended root. This fixes php-fpm
that expects SCRIPT_FILENAME and also works with slowcgi if you
configure the root correctly. For example, if SCRIPT_NAME and
REQUEST_URI are /php/index.php, root is /htdocs, SCRIPT_FILENAME will
be /htdocs/php/index.php. As tested and discussed with florian@

Add missing log call for FastCGI requests.

don't leak fcgi fd

Padding of fcgi records is optional, but if we receive padding data we
should read it.

We need to read from the fcgi bufferevent until it's empty because the
event handler will not be called again if no new data arrives.
Debugged with and OK reyk@

Allow to specify a FastCGI TCP socket on localhost (eg. :9000). Used
for debugging, you should prefer local UNIX sockets, but it helped to
find an issue that will be fixed with the next commit.

OK florian@

spacing

Rewrite fcgi_add_param and hand over a lot more http headers etc. to
the cgi script.
OK reyk@
"blanket OK" for changes in httpd for the time beeing from deraadt@

Correctly parse fcgi records if we don't get the whole record in one
bufferevent_read().
Input/OK reyk@

Only write the HTTP header for the first fastcgi chunk.

some fastcgi improvements:
- DPRINTF instead of log_info for internal debugging.
- submit QUERY_STRING, if it exists
- use a proper function to create an HTTP header.
- use server_file_error() to detect EOF and fastcgi stream errors.
- disable keep-alive/persist for now until we have a reliable way to
get the content length from the cgi response or support chunked
encoding.

"Cool, jep" florian@

One bufferevent can be shared by file and fcgi.

Allow to specify a non-default fastcgi socket.

Put in first stab at fastcgi. Very early work in progress. Putting it
in now so that we can quickly work on it in tree. Requested by reyk@.
deraadt@ is OK with this according to reyk@.

Neither clt_descreq nor clt_descresp in struct client need to be void *.
They both are only used as struct http_descriptor.
OK tb@

For FCGI_END_REQUEST reset the clt struct similar to what is done in the
file and other cases. Especially when the session uses keep-alive it is
important to set TOREAD_HTTP_HEADER so that the state machine knows what's
next.
OK op@

Use break instead of return so that a HEAD request still consumes all data.
OK op@

fix regression introduced in previous commit. HEAD replies don't have a
body so server_fcgi_error shouldn't print the end marker.

OK claudio@

correctly handle an abnormal fastcgi termination. httpd handles the
disconnection from the fastcgi application via server_file_error which
assumes that the reply was completey done. However, if the fastcgi
reply wasn't complete (e.g. because slowcgi hit the timeout) the HTTP
client are left "hanging" and waiting for a reply until they give up.

This adds a server_fcgi_error callback to handle the "no headers" and
"incomplete data" cases and properly close the reply before falling back
to server_file_error.

OK claudio@

Nothing uses kv_flags.

John (j AT bitminer.ca) pointed out that we didn't correctly
initialize struct kv and might use slower KV_FLAG_GLOBBING path in
kv_find depending on stack garbage. Instead of fixing the
initialization just delete kv_flags from struct kv.

OK claudio, tb

* stop sending the content for head requests, even when its supplied by the
fcgi. Required by RFC 7231 and RFC 3875 section 4.3.2.
* If the client sends an empty body without a Content-Lenght:
do not add the Content-Lenght if it's a HEAD request.
If it's a HEAD request, the Content-Lenght should show the size of the
equivalent GET request, but we don't know how much that will be so
don't lie.

found by and fix suggested by Ross L Richardson, Thanks!

Additionally:

* when the fcgi supplies a Content-Length header, do not remove it and
set Transfer-Encoding: chunked. Instead, leave the Content-Lenght
header in place, as obviously the fcgi knows how much data will come.

ok claudio@

Fix previous.
Only set Content-Length when we no the body is empty and we disable
chunked encoding. Otherwise we break the nextcloud app again :/
Pointed out by Matthias Pressfreund, thanks!

When we disable "Transfer-Encoding: chunked" in the fastcgi backend
because we are going to send an empty body we have to provide
"Content-Length: 0" otherwise some browsers (Firefox, Safari) just
hang until httpd(8) closes the connection.
Problem reported by Matthias Pressfreund, debugged with weerd@ who
pointed out that the problem is browser dependent.
OK tracey

Do not try to chunk encode an empty http body coming from an fcgi
upstream.

Found the hard way by Chris Narkiewicz who tracked failing uploads in
the nextcloud mobile app down to httpd(8) trying to chunk encode a
"204 No Content" resonse.

Testing by Steve Williams
Testing & OK stsp

Remove outdated comment. We got all httpd headers from the fcgi server
at this point.

Use the original requested URI for REQUEST_URI.

ok millert florian

Add support for non-localhost fastcgi sockets.

Lots of review time kn@
Lots of review time, tweaks, and ok florian@

remove unused assignment
from Ross L Richardson <openbsd AT rlr DOT id DOT au>, Thanks
ok claudio@

Implement "strip" option for fastcgi to be able to have multiple chroots
under /var/www for FastCGI servers.
From Nazar Zhuk (nazar AT zhuk DOT online), thanks!
Ok benno

Set the REQUEST_URI CGI variable to the requested URI and query string
instead of the rewritten path and query string.

Patch from Tim Baumgard, reminded by Mischa Peters.

ok benno, reyk

spacing

httpd(8): add support for setting custom FastCGI parameters.

This commit extends the existing grammar by adding the param option
to the fastcgi directive: fastcgi param name value.

Example usage:
fastcgi param VAR1 hello
fastcgi param VAR2 world

With help and OK florian@
Rogue manpage bits, feel free to modify them.

Omit HSTS headers over unencrypted connections, per RFC 6797.

ok florian@

Add support for client certificate authentication to httpd.

From Jack Burton <jack at saosce dot com dot au> - thanks!

Also tested by Jan Klemkow <j.klemkow at wemelug dot de>.

ok beck@ reyk@

Don't set HTTP date header if already set.

Thanks Nick Owens

OK florian@

The POSIX APIs that that sockaddrs all ignore the s*_len field in the
incoming socket, so userspace doesn't need to set it unless it has its
own reasons for tracking the size along with the sockaddr.

ok phessler@ deraadt@ florian@

The strchr() call either returns a NULL pointer, on which the code will
break out of the loop, or a pointer to ':'. Thus the extra check for
':' is unnecessary and can be removed.

ok jung@

Empty lines cause server_fcgi_getheaders() to immediately return.
Unfortunately in that case the line was not freed. This lead to a
memleak on each request. Thus, save the return value prior to
returning, free the line and return the saved value.

ok jung@

struct client starts to become the kitchen sink. Move fastcgi data to
its own struct. Requested by and OK reyk@

spacing

Do not assume that the full http response header is in the first
fastcgi stdout record. Keep processing stdout records until we found
the header / body separator and only then generate the header
response.
Problem reported by many.

OK jung@

Always pass QUERY_STRING variable.
According to the RFC it is empty when no query string was found.
From Tim Baumgard <openbsd@bmgrd.com>o

ok florian@

Retire socket_set_blockmode() in favor of the SOCK_NONBLOCK type flag.
As done in iked and snmpd.

OK jung@

fix an fd leak if socket connection fails; from Carlin Bingham
ok reyk@

fix a typo; from Carlin Bingham

Change httpd(8) to use C99-style fixed-width integers (uintN_t instead
of u_intN_t) and replace u_int with unsigned int. Mixing both
variants is a bad style and most contributors seem to prefer this
style; it also helps us to get used to it, portability, and
standardization.

Theoretically no binary change, except one in practice: httpd.o has a
different checksum because gcc with -O2 pads/optimizes "struct
privsep" differently when using "unsigned int" instead "u_int" for the
affected members. "u_int" is just a typedef of "unsigned int", -O0
doesn't build the difference and clang with -O2 doesn't do it either -
it is just another curiosity from gcc-land.

OK semarie@

Fix rev 1.70 of server.c by only re-enabling the bufferevent if we
previously disabled it because we were reading to fast (from disk).
Problem noted and tracked down to that commit by weerd@ and
independently by stsp@.
Tested by weerd@, stsp@, reyk@
OK bluhm@, reyk@

repair hsts header output, wrong format strings caused broken
Strict-Transport-Security headers. Add __format__ attribute to
kv_set() and kv_setkey() to make it easier to spot such problems.

Found by and fix from Donovan Watteau <tsoomi -AT- gmail -DOT- com>,
thanks for your help.

ok deraadt@

backout the previous: it broke wordpress somehow.
we need more care to find a proper fix for the fastcgi headers.

acknowledged by deraadt@

Read fcgi response records until we have the whole http header and can
parse it. Otherwise http headers can leak into the body.
Pointed out by Jean-Philippe Ouellet on bugs@ Thanks!
OK reyk, commit ASAP deraadt@

add HSTS to fcgi responses
OK reyk

handle error returns from bufferevent_write()

ok florian@

remove XXX and handle error return from evbuffer_add()

ok florian@

treat asprintf failure in REQUEST_URI case as a fatal error

ok florian@

According to RFC 3875 PATH_INFO should either contain a full path or
be empty (""). It was not set at all when there is nothing to set
which caused problems with some FastCGI applications (like
Flask/Python through uWSGI).

From hrkfdn via github

plug fd leak found by Todd Mortimer

ok claudio deraadt florian

Allow more characters in CGI environment variables as specified by RFC
7230 and RFC 3875.
sthen@ suggested to add a comment to explain where the list of
characters is coming from.
Found the hard way and initial diff from Tim van der Molen (tbvdm at
xs4all), thanks! Some more allowed characters added by me.
OK sthen@

Use the rewritten (index file appended) uri as DOCUMENT_URI.

OK florian@

Ooops, no need to include sys/cdefs.h.

Pointed out by florian@

httpd is based on relayd and had included many headers that are only
needed by its ancestor. jsg@, include-what-you-use, and some manual
review helped to cleanup the headers (take iwyu with a grain of salt).
Based on common practice, httpd.h now also includes the necessary
headers for itself.

OK florian@

s/clt_fcgi_remote_user/clt_remote_user/
OK reyk@

Decouple auth parameters from struct server_config into struct auth.

OK florian@

First stab at implementing basic auth.
Currently the htpasswd file needs to be in the chroot; will hopefully
improved soonish.
Based on a diff from Oscar Linderholm many months ago but turned into
a complete rewrite.
input/OK reyk@

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

Abort if fcgi_chunked is not true to avoid sending additional garbage
after the response.

Found by Erik Lax

ok florian@

add new url stripping option:

strip number
Strip number path components from the beginning of the
request URI before looking up the stripped-down URI at
the document root.


reviewed with much patience and OK by reyk@

Stop pulling in <arpa/inet.h> or <arpa/nameser.h> when unnecessary.
*Do* pull it in when in_{port,addr}_h is needed and <netinet/in.h> isn't.

ok reyk@

Like previously done in relayd, change the keyword "ssl" to "tls" to
reflect reality.

OK benno@

stop viral header propagation. none of this code uses sys/hash.h
from Max Fillinger

Remove unnecessary netinet/in_systm.h include.

ok millert@

whitespace spotted while studying the code

FastCGI did not support persistent connections. Add initial support
for persistent connections with FastCGI by implementing chunked
Transfer-Encoding. This only works with HTTP/1.1.

With input and help from florian@ who found some FastCGI edge cases.

OK florian@

Don't pass the local buffer array by reference.

OK florian@

Replace the code to get the FastCGI Status header with a proper way to
parse and write the headers using the http response descriptor. This
allows to add other tweaks, like support for chunked encoding, later.

OK florian@

Use two instead of one http descriptor for request and response.

OK chrisz@

Add Last-Modified: HTTP header.

OK reyk@

For a non-existent root we don't want the root prefix to show up in
PATH_INFO.
Therefore put a lower bound of strlen(root) on scriptlen.
This makes perfect sense for virtual FastCGI scripts which run chrooted
in another directory from httpd.


ok reyk@

Provide a failsafe version of the path_info() function that doesn't
need a temporary path variable. Based on an initial diff from
chrisz@.

"Commit any failsafe version and I'm ok with it" chrisz@

make a few variables more local

When opening directories, re-match the location after the index file
has been appended. This allows to use a fastcgi target as the default
index, for example index.php.

OK florian@

branches: 1.29.2;
Don't try to ouput FCGI_STDERR into error.log if there is no data.
Problem noticed by naddy@, OK reyk@

Opportunistically try to parse "Status: $code" in the very first
response from the fcgi daemon and use that code as HTTP response
code. If it doesn't work out fall back to code 200.
This might fix naddy@'s issue with redirects in cvsweb.
To be revisited after unlock.
Discussed with & grudgingly OK reyk@

Write STDERR from the CGI to the web server error log as intended.

OK florian@

If the very first fcgi STDOUT record has length 0 the cgi script
didn't send anything back. This is an internal server error.
OK reyk@

Always zero-out the fcgi record header for STDIN data.

OK florian@

Use memset(buf instead of memset(&buf.

Pointed out by deraadt@

http POST support
with & OK reyk@

Content-Length and Content-Type are transmitted as CONTENT_LENGTH and
CONTENT_TYPE environment variables to cgi scripts, without the HTTP_
prefix.
OK reyk@

Add HTTPS = on CGI variable.

Implement PATH_INFO and add DOCUMENT_ROOT.
PATH_INFO was requested by naddy@ who successfully tested it with "cvsweb".

ok naddy@

httpd doesn't support SSL/TLS yet, remove the remaining bits.
The secrect plan is to add it later using the ressl wrapper library.

c-type functions / makros need a cast to unsigned char, not int
"feel free to commit" reyk@

spacing

Dynamically pass HTTP request headers as protocol-specific HTTP_* CGI
meta-variables.

ok florian@

Add function to iterate all headers. No functional change.

Split fastcgi socket path and document root option and add the
SCRIPT_FILENAME CGI param with a prepended root. This fixes php-fpm
that expects SCRIPT_FILENAME and also works with slowcgi if you
configure the root correctly. For example, if SCRIPT_NAME and
REQUEST_URI are /php/index.php, root is /htdocs, SCRIPT_FILENAME will
be /htdocs/php/index.php. As tested and discussed with florian@

Add missing log call for FastCGI requests.

don't leak fcgi fd

Padding of fcgi records is optional, but if we receive padding data we
should read it.

We need to read from the fcgi bufferevent until it's empty because the
event handler will not be called again if no new data arrives.
Debugged with and OK reyk@

Allow to specify a FastCGI TCP socket on localhost (eg. :9000). Used
for debugging, you should prefer local UNIX sockets, but it helped to
find an issue that will be fixed with the next commit.

OK florian@

spacing

Rewrite fcgi_add_param and hand over a lot more http headers etc. to
the cgi script.
OK reyk@
"blanket OK" for changes in httpd for the time beeing from deraadt@

Correctly parse fcgi records if we don't get the whole record in one
bufferevent_read().
Input/OK reyk@

Only write the HTTP header for the first fastcgi chunk.

some fastcgi improvements:
- DPRINTF instead of log_info for internal debugging.
- submit QUERY_STRING, if it exists
- use a proper function to create an HTTP header.
- use server_file_error() to detect EOF and fastcgi stream errors.
- disable keep-alive/persist for now until we have a reliable way to
get the content length from the cgi response or support chunked
encoding.

"Cool, jep" florian@

One bufferevent can be shared by file and fcgi.

Allow to specify a non-default fastcgi socket.

Put in first stab at fastcgi. Very early work in progress. Putting it
in now so that we can quickly work on it in tree. Requested by reyk@.
deraadt@ is OK with this according to reyk@.

Use break instead of return so that a HEAD request still consumes all data.
OK op@

fix regression introduced in previous commit. HEAD replies don't have a
body so server_fcgi_error shouldn't print the end marker.

OK claudio@

correctly handle an abnormal fastcgi termination. httpd handles the
disconnection from the fastcgi application via server_file_error which
assumes that the reply was completey done. However, if the fastcgi
reply wasn't complete (e.g. because slowcgi hit the timeout) the HTTP
client are left "hanging" and waiting for a reply until they give up.

This adds a server_fcgi_error callback to handle the "no headers" and
"incomplete data" cases and properly close the reply before falling back
to server_file_error.

OK claudio@

Nothing uses kv_flags.

John (j AT bitminer.ca) pointed out that we didn't correctly
initialize struct kv and might use slower KV_FLAG_GLOBBING path in
kv_find depending on stack garbage. Instead of fixing the
initialization just delete kv_flags from struct kv.

OK claudio, tb

* stop sending the content for head requests, even when its supplied by the
fcgi. Required by RFC 7231 and RFC 3875 section 4.3.2.
* If the client sends an empty body without a Content-Lenght:
do not add the Content-Lenght if it's a HEAD request.
If it's a HEAD request, the Content-Lenght should show the size of the
equivalent GET request, but we don't know how much that will be so
don't lie.

found by and fix suggested by Ross L Richardson, Thanks!

Additionally:

* when the fcgi supplies a Content-Length header, do not remove it and
set Transfer-Encoding: chunked. Instead, leave the Content-Lenght
header in place, as obviously the fcgi knows how much data will come.

ok claudio@

Fix previous.
Only set Content-Length when we no the body is empty and we disable
chunked encoding. Otherwise we break the nextcloud app again :/
Pointed out by Matthias Pressfreund, thanks!

When we disable "Transfer-Encoding: chunked" in the fastcgi backend
because we are going to send an empty body we have to provide
"Content-Length: 0" otherwise some browsers (Firefox, Safari) just
hang until httpd(8) closes the connection.
Problem reported by Matthias Pressfreund, debugged with weerd@ who
pointed out that the problem is browser dependent.
OK tracey

Do not try to chunk encode an empty http body coming from an fcgi
upstream.

Found the hard way by Chris Narkiewicz who tracked failing uploads in
the nextcloud mobile app down to httpd(8) trying to chunk encode a
"204 No Content" resonse.

Testing by Steve Williams
Testing & OK stsp

Remove outdated comment. We got all httpd headers from the fcgi server
at this point.

Use the original requested URI for REQUEST_URI.

ok millert florian

Add support for non-localhost fastcgi sockets.

Lots of review time kn@
Lots of review time, tweaks, and ok florian@

remove unused assignment
from Ross L Richardson <openbsd AT rlr DOT id DOT au>, Thanks
ok claudio@

Implement "strip" option for fastcgi to be able to have multiple chroots
under /var/www for FastCGI servers.
From Nazar Zhuk (nazar AT zhuk DOT online), thanks!
Ok benno

Set the REQUEST_URI CGI variable to the requested URI and query string
instead of the rewritten path and query string.

Patch from Tim Baumgard, reminded by Mischa Peters.

ok benno, reyk

spacing

httpd(8): add support for setting custom FastCGI parameters.

This commit extends the existing grammar by adding the param option
to the fastcgi directive: fastcgi param name value.

Example usage:
fastcgi param VAR1 hello
fastcgi param VAR2 world

With help and OK florian@
Rogue manpage bits, feel free to modify them.

Omit HSTS headers over unencrypted connections, per RFC 6797.

ok florian@

Add support for client certificate authentication to httpd.

From Jack Burton <jack at saosce dot com dot au> - thanks!

Also tested by Jan Klemkow <j.klemkow at wemelug dot de>.

ok beck@ reyk@

Don't set HTTP date header if already set.

Thanks Nick Owens

OK florian@

The POSIX APIs that that sockaddrs all ignore the s*_len field in the
incoming socket, so userspace doesn't need to set it unless it has its
own reasons for tracking the size along with the sockaddr.

ok phessler@ deraadt@ florian@

The strchr() call either returns a NULL pointer, on which the code will
break out of the loop, or a pointer to ':'. Thus the extra check for
':' is unnecessary and can be removed.

ok jung@

Empty lines cause server_fcgi_getheaders() to immediately return.
Unfortunately in that case the line was not freed. This lead to a
memleak on each request. Thus, save the return value prior to
returning, free the line and return the saved value.

ok jung@

struct client starts to become the kitchen sink. Move fastcgi data to
its own struct. Requested by and OK reyk@

spacing

Do not assume that the full http response header is in the first
fastcgi stdout record. Keep processing stdout records until we found
the header / body separator and only then generate the header
response.
Problem reported by many.

OK jung@

Always pass QUERY_STRING variable.
According to the RFC it is empty when no query string was found.
From Tim Baumgard <openbsd@bmgrd.com>o

ok florian@

Retire socket_set_blockmode() in favor of the SOCK_NONBLOCK type flag.
As done in iked and snmpd.

OK jung@

fix an fd leak if socket connection fails; from Carlin Bingham
ok reyk@

fix a typo; from Carlin Bingham

Change httpd(8) to use C99-style fixed-width integers (uintN_t instead
of u_intN_t) and replace u_int with unsigned int. Mixing both
variants is a bad style and most contributors seem to prefer this
style; it also helps us to get used to it, portability, and
standardization.

Theoretically no binary change, except one in practice: httpd.o has a
different checksum because gcc with -O2 pads/optimizes "struct
privsep" differently when using "unsigned int" instead "u_int" for the
affected members. "u_int" is just a typedef of "unsigned int", -O0
doesn't build the difference and clang with -O2 doesn't do it either -
it is just another curiosity from gcc-land.

OK semarie@

Fix rev 1.70 of server.c by only re-enabling the bufferevent if we
previously disabled it because we were reading to fast (from disk).
Problem noted and tracked down to that commit by weerd@ and
independently by stsp@.
Tested by weerd@, stsp@, reyk@
OK bluhm@, reyk@

repair hsts header output, wrong format strings caused broken
Strict-Transport-Security headers. Add __format__ attribute to
kv_set() and kv_setkey() to make it easier to spot such problems.

Found by and fix from Donovan Watteau <tsoomi -AT- gmail -DOT- com>,
thanks for your help.

ok deraadt@

backout the previous: it broke wordpress somehow.
we need more care to find a proper fix for the fastcgi headers.

acknowledged by deraadt@