remove ssl_init()

it's a noop; nowadays both LibreSSL and OpenSSL libcrypto and libssl
initialize themselves automatically before doing anything.

ok tb

relayd/ssl.c: Remove a workaround that uses a copy of the old
certificate instead of using it directly because BIO_new_mem_buf()
used to take an non-const buffer. This was changed in 2018, so we
can now remove an XXX and simplify the code.

ok bluhm

remove bogus key hack now that it's handled by libtls

no objection claudio@
ok tb@ jsing@

Always calculate the hash value of the x509 cert in ssl_load_pkey().
Check whether TLS server object is available before using it. With
these fixes the ssl inspect regress test just fails and does not
crash relayd.
OK claudio@

use __func__ in log messages. fix some whitespace while here.
From Hiltjo Posthuma hiltjo -AT codemadness -DOT- org, thanks!
ok florian, claudio

Migrate relayd to use libtls for TLS. Still does the TLS privsep via the
engine but at least we can use a sane API for new features.
Going in now so it is possible to work with this in tree.
General agreement at d2k17.

Stop accessing verbose and debug variables from log.c directly.

This replaces log_verbose() and "extern int verbose" with the two functions
log_setverbose() and log_getverbose().

Pointed out by benno@
OK krw@ eric@ gilles@ (OK gilles@ for the snmpd bits as well)

now that ibuf_free() checks for null, we can remove the check here.
ok mmcc@ millert@

Missing free(3) in error path
ok benno@

branches: 1.28.2;
Clean up the relayd headers with help of include-what-you-use and some
manual review. Based on common practice, relayd.h now includes the
necessary headers for itself.

OK benno@

Adapt to <limits.h> universe.
ok millert

Change the keyword "ssl" to "tls" to reflect reality since we
effectively disabled support for the SSL protocols. SSL remains a
common term describing SSL/TLS, there is some controvery about this
change, and the name really doesn't matter, but I feel confident about
it now.

(btw., sthen@ pointed out some historical context:
http://tim.dierks.org/2014/05/security-standards-and-name-changes-in.html)

OK benno@, with input from tedu@

add additional includes required to build with -DOPENSSL_NO_DEPRECATED
ok reyk@

Unify the SSL privsep key loading functions.

ok eric@

Fix two memory leaks: EVP_PKEY_get1_RSA() returns a referenced key
that requires to call RSA_free() to dereference it after use. Also
free a temporary key that was read by PEM_read_PrivateKey() and
immediately written into a bio.

ok markus@

Support the CA key for SSL inspection in the ca process. Instead of
looking up the keys by relay id, add all keys to a list and look them
up by key id.

ok benno@

Use RSA_set_ex_data()/RSA_get_ex_data() directly instead of
the undocumented RSA_set_app_data()/RSA_get_app_data() wrappers.

Introduce privsep for private keys:

- Move RSA private keys to a new separate process instead of copying
them to the relays. A custom RSA engine is used by the SSL/TLS code
of the relay processes to send RSA private key encryption/decryption
(also used for sign/verify) requests to the new "ca" processes instead
of operating on the private key directly.

- Each relay process gets its own related ca process. Setting
"prefork 5" in the config file will spawn 10 processes (5 relay, 5
ca). This diff also reduces the default number of relay processes
from 5 to 3 which should be suitable in most installations without a
very heavy load.

- Don't keep text versions of the keys in memory, parse them once and
keep the binary representation. This might still be the case in
OpenSSL's internals but will be fixed in the library.

This diff doesn't prevent something like "heartbleed" but adds an
additional mitigation to prevent leakage of the private keys from the
processes doing SSL/TLS.

With feedback from many
ok benno@

unset cte->buf after free, fix double free via tcp_close().
ok reyk@

Support SSL inspection, the ability to transparently filter in SSL/TLS
connections (eg. HTTPS) by using a local CA that is accepted by the
clients. See the "SSL RELAYS" and "EXAMPLES" sections in the
relayd.conf(5) manpage for more details.

ok benno@, manpage bits jmc@

reorder the variables a bit, no functionaly change.

Rename some imsg bits to make namespace collisions less likely buf to
ibuf, buf_read to ibuf_read, READ_BUF_SIZE to IBUF_READ_SIZE.

ok henning gilles claudio jacekm deraadt

Keep around the SSL session for each checked host. This way SSL
caching can kick in on subsequent checks, making them faster and
lighter on the server.

From camield, closes PR 6137 (modified diff)

change the way relayd reports check results: instead of logging an
arbitrary string in debugging mode, it will store an error code
(HCE_*) for each host. the error code can be translated to a string
(in log.c) for debugging but it will also be passed to relayctl via
the control socket.

from a user point of view, this will print a human-readable error
message in the "relayctl show hosts" output if a host is down because
the check failed. the relayctl(8) manpage includes detailed
explanations of the error messages including mitigations for the
most-common problems.

ok jmc@ (manpages)
ok phessler@

hoststated gets renamed to relayd. easier to type, and actually says
what the daemon does - it is a relayer that pays attention to the status
of pools of hosts; not a status checkers that happens to do some relaying

fix fd/ssl leak in hoststated ssl checks

From Przemyslaw Frasunek, closes PR 5671

sort includes, adjust to style(9)

Correct my mail address.

Second step towards hoststated reload:
First split out hosts, tables and services into to structs, one that
contains the runtime fields and one (inside the runtime) that contains
mostly static fields that will be sent over the socket during reload.

Also move the demoted field of tables inside the flags field as its
just a boolean.
ok reyk@

spacing

Add layer 7 functionality to hoststated used for layer 7
loadbalancing, SSL acceleration, general-purpose TCP relaying, and
transparent proxying.

see hoststated.conf(5) and my upcoming article on undeadly.org for
details.

ok to commit deraadt@ pyr@

carefully check some return values and make lint happier. never pass
any truncated strings (table names/anchors/tags/...) to pf and the
kernel.

ok pyr@

remove superfluos log_debug() messages

ok pyr@

update some more debug messages and fix typos. also remove some extra
debug noise from ssl.c.

declare the function ssl_error() globally

provide more accurate error messages.
ok reyk@

Add SSL support to hoststated.
with help and OK reyk@
with help and advice by claudio@ and Srebrenko Sehic

relayd/ssl.c: Remove a workaround that uses a copy of the old
certificate instead of using it directly because BIO_new_mem_buf()
used to take an non-const buffer. This was changed in 2018, so we
can now remove an XXX and simplify the code.

ok bluhm

remove bogus key hack now that it's handled by libtls

no objection claudio@
ok tb@ jsing@

Always calculate the hash value of the x509 cert in ssl_load_pkey().
Check whether TLS server object is available before using it. With
these fixes the ssl inspect regress test just fails and does not
crash relayd.
OK claudio@

use __func__ in log messages. fix some whitespace while here.
From Hiltjo Posthuma hiltjo -AT codemadness -DOT- org, thanks!
ok florian, claudio

Migrate relayd to use libtls for TLS. Still does the TLS privsep via the
engine but at least we can use a sane API for new features.
Going in now so it is possible to work with this in tree.
General agreement at d2k17.

Stop accessing verbose and debug variables from log.c directly.

This replaces log_verbose() and "extern int verbose" with the two functions
log_setverbose() and log_getverbose().

Pointed out by benno@
OK krw@ eric@ gilles@ (OK gilles@ for the snmpd bits as well)

now that ibuf_free() checks for null, we can remove the check here.
ok mmcc@ millert@

Missing free(3) in error path
ok benno@

branches: 1.28.2;
Clean up the relayd headers with help of include-what-you-use and some
manual review. Based on common practice, relayd.h now includes the
necessary headers for itself.

OK benno@

Adapt to <limits.h> universe.
ok millert

Change the keyword "ssl" to "tls" to reflect reality since we
effectively disabled support for the SSL protocols. SSL remains a
common term describing SSL/TLS, there is some controvery about this
change, and the name really doesn't matter, but I feel confident about
it now.

(btw., sthen@ pointed out some historical context:
http://tim.dierks.org/2014/05/security-standards-and-name-changes-in.html)

OK benno@, with input from tedu@

add additional includes required to build with -DOPENSSL_NO_DEPRECATED
ok reyk@

Unify the SSL privsep key loading functions.

ok eric@

Fix two memory leaks: EVP_PKEY_get1_RSA() returns a referenced key
that requires to call RSA_free() to dereference it after use. Also
free a temporary key that was read by PEM_read_PrivateKey() and
immediately written into a bio.

ok markus@

Support the CA key for SSL inspection in the ca process. Instead of
looking up the keys by relay id, add all keys to a list and look them
up by key id.

ok benno@

Use RSA_set_ex_data()/RSA_get_ex_data() directly instead of
the undocumented RSA_set_app_data()/RSA_get_app_data() wrappers.

Introduce privsep for private keys:

- Move RSA private keys to a new separate process instead of copying
them to the relays. A custom RSA engine is used by the SSL/TLS code
of the relay processes to send RSA private key encryption/decryption
(also used for sign/verify) requests to the new "ca" processes instead
of operating on the private key directly.

- Each relay process gets its own related ca process. Setting
"prefork 5" in the config file will spawn 10 processes (5 relay, 5
ca). This diff also reduces the default number of relay processes
from 5 to 3 which should be suitable in most installations without a
very heavy load.

- Don't keep text versions of the keys in memory, parse them once and
keep the binary representation. This might still be the case in
OpenSSL's internals but will be fixed in the library.

This diff doesn't prevent something like "heartbleed" but adds an
additional mitigation to prevent leakage of the private keys from the
processes doing SSL/TLS.

With feedback from many
ok benno@

unset cte->buf after free, fix double free via tcp_close().
ok reyk@

Support SSL inspection, the ability to transparently filter in SSL/TLS
connections (eg. HTTPS) by using a local CA that is accepted by the
clients. See the "SSL RELAYS" and "EXAMPLES" sections in the
relayd.conf(5) manpage for more details.

ok benno@, manpage bits jmc@

reorder the variables a bit, no functionaly change.

Rename some imsg bits to make namespace collisions less likely buf to
ibuf, buf_read to ibuf_read, READ_BUF_SIZE to IBUF_READ_SIZE.

ok henning gilles claudio jacekm deraadt

Keep around the SSL session for each checked host. This way SSL
caching can kick in on subsequent checks, making them faster and
lighter on the server.

From camield, closes PR 6137 (modified diff)

change the way relayd reports check results: instead of logging an
arbitrary string in debugging mode, it will store an error code
(HCE_*) for each host. the error code can be translated to a string
(in log.c) for debugging but it will also be passed to relayctl via
the control socket.

from a user point of view, this will print a human-readable error
message in the "relayctl show hosts" output if a host is down because
the check failed. the relayctl(8) manpage includes detailed
explanations of the error messages including mitigations for the
most-common problems.

ok jmc@ (manpages)
ok phessler@

hoststated gets renamed to relayd. easier to type, and actually says
what the daemon does - it is a relayer that pays attention to the status
of pools of hosts; not a status checkers that happens to do some relaying

fix fd/ssl leak in hoststated ssl checks

From Przemyslaw Frasunek, closes PR 5671

sort includes, adjust to style(9)

Correct my mail address.

Second step towards hoststated reload:
First split out hosts, tables and services into to structs, one that
contains the runtime fields and one (inside the runtime) that contains
mostly static fields that will be sent over the socket during reload.

Also move the demoted field of tables inside the flags field as its
just a boolean.
ok reyk@

spacing

Add layer 7 functionality to hoststated used for layer 7
loadbalancing, SSL acceleration, general-purpose TCP relaying, and
transparent proxying.

see hoststated.conf(5) and my upcoming article on undeadly.org for
details.

ok to commit deraadt@ pyr@

carefully check some return values and make lint happier. never pass
any truncated strings (table names/anchors/tags/...) to pf and the
kernel.

ok pyr@

remove superfluos log_debug() messages

ok pyr@

update some more debug messages and fix typos. also remove some extra
debug noise from ssl.c.

declare the function ssl_error() globally

provide more accurate error messages.
ok reyk@

Add SSL support to hoststated.
with help and OK reyk@
with help and advice by claudio@ and Srebrenko Sehic

remove bogus key hack now that it's handled by libtls

no objection claudio@
ok tb@ jsing@

Always calculate the hash value of the x509 cert in ssl_load_pkey().
Check whether TLS server object is available before using it. With
these fixes the ssl inspect regress test just fails and does not
crash relayd.
OK claudio@

use __func__ in log messages. fix some whitespace while here.
From Hiltjo Posthuma hiltjo -AT codemadness -DOT- org, thanks!
ok florian, claudio

Migrate relayd to use libtls for TLS. Still does the TLS privsep via the
engine but at least we can use a sane API for new features.
Going in now so it is possible to work with this in tree.
General agreement at d2k17.

Stop accessing verbose and debug variables from log.c directly.

This replaces log_verbose() and "extern int verbose" with the two functions
log_setverbose() and log_getverbose().

Pointed out by benno@
OK krw@ eric@ gilles@ (OK gilles@ for the snmpd bits as well)

now that ibuf_free() checks for null, we can remove the check here.
ok mmcc@ millert@

Missing free(3) in error path
ok benno@

branches: 1.28.2;
Clean up the relayd headers with help of include-what-you-use and some
manual review. Based on common practice, relayd.h now includes the
necessary headers for itself.

OK benno@

Adapt to <limits.h> universe.
ok millert

Change the keyword "ssl" to "tls" to reflect reality since we
effectively disabled support for the SSL protocols. SSL remains a
common term describing SSL/TLS, there is some controvery about this
change, and the name really doesn't matter, but I feel confident about
it now.

(btw., sthen@ pointed out some historical context:
http://tim.dierks.org/2014/05/security-standards-and-name-changes-in.html)

OK benno@, with input from tedu@

add additional includes required to build with -DOPENSSL_NO_DEPRECATED
ok reyk@

Unify the SSL privsep key loading functions.

ok eric@

Fix two memory leaks: EVP_PKEY_get1_RSA() returns a referenced key
that requires to call RSA_free() to dereference it after use. Also
free a temporary key that was read by PEM_read_PrivateKey() and
immediately written into a bio.

ok markus@

Support the CA key for SSL inspection in the ca process. Instead of
looking up the keys by relay id, add all keys to a list and look them
up by key id.

ok benno@

Use RSA_set_ex_data()/RSA_get_ex_data() directly instead of
the undocumented RSA_set_app_data()/RSA_get_app_data() wrappers.

Introduce privsep for private keys:

- Move RSA private keys to a new separate process instead of copying
them to the relays. A custom RSA engine is used by the SSL/TLS code
of the relay processes to send RSA private key encryption/decryption
(also used for sign/verify) requests to the new "ca" processes instead
of operating on the private key directly.

- Each relay process gets its own related ca process. Setting
"prefork 5" in the config file will spawn 10 processes (5 relay, 5
ca). This diff also reduces the default number of relay processes
from 5 to 3 which should be suitable in most installations without a
very heavy load.

- Don't keep text versions of the keys in memory, parse them once and
keep the binary representation. This might still be the case in
OpenSSL's internals but will be fixed in the library.

This diff doesn't prevent something like "heartbleed" but adds an
additional mitigation to prevent leakage of the private keys from the
processes doing SSL/TLS.

With feedback from many
ok benno@

unset cte->buf after free, fix double free via tcp_close().
ok reyk@

Support SSL inspection, the ability to transparently filter in SSL/TLS
connections (eg. HTTPS) by using a local CA that is accepted by the
clients. See the "SSL RELAYS" and "EXAMPLES" sections in the
relayd.conf(5) manpage for more details.

ok benno@, manpage bits jmc@

reorder the variables a bit, no functionaly change.

Rename some imsg bits to make namespace collisions less likely buf to
ibuf, buf_read to ibuf_read, READ_BUF_SIZE to IBUF_READ_SIZE.

ok henning gilles claudio jacekm deraadt

Keep around the SSL session for each checked host. This way SSL
caching can kick in on subsequent checks, making them faster and
lighter on the server.

From camield, closes PR 6137 (modified diff)

change the way relayd reports check results: instead of logging an
arbitrary string in debugging mode, it will store an error code
(HCE_*) for each host. the error code can be translated to a string
(in log.c) for debugging but it will also be passed to relayctl via
the control socket.

from a user point of view, this will print a human-readable error
message in the "relayctl show hosts" output if a host is down because
the check failed. the relayctl(8) manpage includes detailed
explanations of the error messages including mitigations for the
most-common problems.

ok jmc@ (manpages)
ok phessler@

hoststated gets renamed to relayd. easier to type, and actually says
what the daemon does - it is a relayer that pays attention to the status
of pools of hosts; not a status checkers that happens to do some relaying

fix fd/ssl leak in hoststated ssl checks

From Przemyslaw Frasunek, closes PR 5671

sort includes, adjust to style(9)

Correct my mail address.

Second step towards hoststated reload:
First split out hosts, tables and services into to structs, one that
contains the runtime fields and one (inside the runtime) that contains
mostly static fields that will be sent over the socket during reload.

Also move the demoted field of tables inside the flags field as its
just a boolean.
ok reyk@

spacing

Add layer 7 functionality to hoststated used for layer 7
loadbalancing, SSL acceleration, general-purpose TCP relaying, and
transparent proxying.

see hoststated.conf(5) and my upcoming article on undeadly.org for
details.

ok to commit deraadt@ pyr@

carefully check some return values and make lint happier. never pass
any truncated strings (table names/anchors/tags/...) to pf and the
kernel.

ok pyr@

remove superfluos log_debug() messages

ok pyr@

update some more debug messages and fix typos. also remove some extra
debug noise from ssl.c.

declare the function ssl_error() globally

provide more accurate error messages.
ok reyk@

Add SSL support to hoststated.
with help and OK reyk@
with help and advice by claudio@ and Srebrenko Sehic

Always calculate the hash value of the x509 cert in ssl_load_pkey().
Check whether TLS server object is available before using it. With
these fixes the ssl inspect regress test just fails and does not
crash relayd.
OK claudio@

use __func__ in log messages. fix some whitespace while here.
From Hiltjo Posthuma hiltjo -AT codemadness -DOT- org, thanks!
ok florian, claudio

Migrate relayd to use libtls for TLS. Still does the TLS privsep via the
engine but at least we can use a sane API for new features.
Going in now so it is possible to work with this in tree.
General agreement at d2k17.

Stop accessing verbose and debug variables from log.c directly.

This replaces log_verbose() and "extern int verbose" with the two functions
log_setverbose() and log_getverbose().

Pointed out by benno@
OK krw@ eric@ gilles@ (OK gilles@ for the snmpd bits as well)

now that ibuf_free() checks for null, we can remove the check here.
ok mmcc@ millert@

Missing free(3) in error path
ok benno@

branches: 1.28.2;
Clean up the relayd headers with help of include-what-you-use and some
manual review. Based on common practice, relayd.h now includes the
necessary headers for itself.

OK benno@

Adapt to <limits.h> universe.
ok millert

Change the keyword "ssl" to "tls" to reflect reality since we
effectively disabled support for the SSL protocols. SSL remains a
common term describing SSL/TLS, there is some controvery about this
change, and the name really doesn't matter, but I feel confident about
it now.

(btw., sthen@ pointed out some historical context:
http://tim.dierks.org/2014/05/security-standards-and-name-changes-in.html)

OK benno@, with input from tedu@

add additional includes required to build with -DOPENSSL_NO_DEPRECATED
ok reyk@

Unify the SSL privsep key loading functions.

ok eric@

Fix two memory leaks: EVP_PKEY_get1_RSA() returns a referenced key
that requires to call RSA_free() to dereference it after use. Also
free a temporary key that was read by PEM_read_PrivateKey() and
immediately written into a bio.

ok markus@

Support the CA key for SSL inspection in the ca process. Instead of
looking up the keys by relay id, add all keys to a list and look them
up by key id.

ok benno@

Use RSA_set_ex_data()/RSA_get_ex_data() directly instead of
the undocumented RSA_set_app_data()/RSA_get_app_data() wrappers.

Introduce privsep for private keys:

- Move RSA private keys to a new separate process instead of copying
them to the relays. A custom RSA engine is used by the SSL/TLS code
of the relay processes to send RSA private key encryption/decryption
(also used for sign/verify) requests to the new "ca" processes instead
of operating on the private key directly.

- Each relay process gets its own related ca process. Setting
"prefork 5" in the config file will spawn 10 processes (5 relay, 5
ca). This diff also reduces the default number of relay processes
from 5 to 3 which should be suitable in most installations without a
very heavy load.

- Don't keep text versions of the keys in memory, parse them once and
keep the binary representation. This might still be the case in
OpenSSL's internals but will be fixed in the library.

This diff doesn't prevent something like "heartbleed" but adds an
additional mitigation to prevent leakage of the private keys from the
processes doing SSL/TLS.

With feedback from many
ok benno@

unset cte->buf after free, fix double free via tcp_close().
ok reyk@

Support SSL inspection, the ability to transparently filter in SSL/TLS
connections (eg. HTTPS) by using a local CA that is accepted by the
clients. See the "SSL RELAYS" and "EXAMPLES" sections in the
relayd.conf(5) manpage for more details.

ok benno@, manpage bits jmc@

reorder the variables a bit, no functionaly change.

Rename some imsg bits to make namespace collisions less likely buf to
ibuf, buf_read to ibuf_read, READ_BUF_SIZE to IBUF_READ_SIZE.

ok henning gilles claudio jacekm deraadt

Keep around the SSL session for each checked host. This way SSL
caching can kick in on subsequent checks, making them faster and
lighter on the server.

From camield, closes PR 6137 (modified diff)

change the way relayd reports check results: instead of logging an
arbitrary string in debugging mode, it will store an error code
(HCE_*) for each host. the error code can be translated to a string
(in log.c) for debugging but it will also be passed to relayctl via
the control socket.

from a user point of view, this will print a human-readable error
message in the "relayctl show hosts" output if a host is down because
the check failed. the relayctl(8) manpage includes detailed
explanations of the error messages including mitigations for the
most-common problems.

ok jmc@ (manpages)
ok phessler@

hoststated gets renamed to relayd. easier to type, and actually says
what the daemon does - it is a relayer that pays attention to the status
of pools of hosts; not a status checkers that happens to do some relaying

fix fd/ssl leak in hoststated ssl checks

From Przemyslaw Frasunek, closes PR 5671

sort includes, adjust to style(9)

Correct my mail address.

Second step towards hoststated reload:
First split out hosts, tables and services into to structs, one that
contains the runtime fields and one (inside the runtime) that contains
mostly static fields that will be sent over the socket during reload.

Also move the demoted field of tables inside the flags field as its
just a boolean.
ok reyk@

spacing

Add layer 7 functionality to hoststated used for layer 7
loadbalancing, SSL acceleration, general-purpose TCP relaying, and
transparent proxying.

see hoststated.conf(5) and my upcoming article on undeadly.org for
details.

ok to commit deraadt@ pyr@

carefully check some return values and make lint happier. never pass
any truncated strings (table names/anchors/tags/...) to pf and the
kernel.

ok pyr@

remove superfluos log_debug() messages

ok pyr@

update some more debug messages and fix typos. also remove some extra
debug noise from ssl.c.

declare the function ssl_error() globally

provide more accurate error messages.
ok reyk@

Add SSL support to hoststated.
with help and OK reyk@
with help and advice by claudio@ and Srebrenko Sehic