drop engine support

diff originally by tb@, tweaked to apply after the useless logging
methods removal.

ok tb

remove the useless logging methods

Instead of wrapping all the methods of the RSA and ECDSA ENGINE,
duplicate the default and override only the ones that are actually
needed for the privsep crypto engine.

part of a larger diff that's ok tb@

remove ca_verify_cb(). was initially used for debugging, then the
logging went away but the no-op callback remained.

noticed by tb@

smtpd: switch ECDSA_METHOD usage to EC_KEY_METHOD

smtpd and the bits it needs in libtls are the only consumer left of
ECDSA_METHOD, which is long deprecated. This paves the way for the
removal in libcrypto.

The diff is from gilles' work on OpenSMTPD-portable, with minor changes
by me.

ok tb@, jsing@

Another missing #include <openssl/err.h>

Revert changes to use the new libtls signer api
There are bugs in the new libtls signer that can lead to a crash.
OK tb@ jsing@

use new libtls signer api

ok tb@

add required headers for smtpd.h and remove unnecessary ones in other files.

ok jung@

replaces calls to err(3)/errx(3) with fatal()/fatalx() from log.c
for code that runs in the daemon.

ok florian@ millert@

Start porting smtpd to libtls.

Note that it changes the way SNI works: The certificate to use is now
selected by looking at the names found in the certificates themselves,
rather than the names of the pki entries in the configuration file.
The set of certificates for a tls listener must be defined explicitly by
using the pki listener option multiple times.

ok tb@

Rename the pony process to dispatcher and klondike to crypto.

From gilles@
OK millert@ giovanni@

properly initialize errstr before going to fail label.

ok gilles@

errstr may be uninitialized in error code path

assume RSA_METHOD is opaque and only access members through setters/getters

a long long time ago, in a galaxy quite close actually, reyk@ introduced an
RSA privsep engine to isolate private keys in the ca process. ECDSA support
in smtpd is become a frequent request so here's an ECDSA privsep engine and
the code required for smtpd to load ECDSA certificates and use them.

assume X509_STORE_CTX is opaque, don't access ->error but use the
X509_STORE_CTX_get_error() function instead

mechanical change to dynamically allocate rsae_method

remove useless check, it's never been and will never be hit

switch smtpd to new grammar

ok eric@

no need to check the sending process in imsg handlers when there is no
ambiguity: just use a single switch.

ok gilles@ sunil@

Introduce more use of freezero(). Also, remove ptr conditionals before
many functions which are free(NULL)-compat
ok gilles

smtpd joins the 7 other daemons that share the same log.c file.

The only major difference was the "log_trace" concept that is only
used by smtpd - move it from log.c into util.c and make it a local
concept. This also needed to rename the global "verbose" variable to
"tracing" in a few places.

OK krw@ gilles@ eric@

Streamline the daemon shutdown sequence.

Only the parent process handles SIGTERM and SIGINT. Upon receiving one
of those, it closes all imsg sockets and waitpid() for the children. It
fatal()s if one of the sockets is closed unexpectedly. Other processes
exit() "normally" when one of the imsg sockets is closed.

ok gilles@ sunil@

The smtpd processes are not expected to ever leave their event loop.
So stop pretending that the *_shutdown() functions could ever be called
in this context, and just fatal() if event_dispatch() returns.

ok gilles@ sunil@ giovanni@

remove noop function

ok sunil@

Implement the fork+exec pattern in smtpd.

The parent process forks child processes and re-exec each of them with
an additional "-x <proc>" argument. During the early setup phase, the
parent process sends ipc socket pairs to interconnect the child
processes as needed, and it passes the queue encryption key to the
queue if necessary. When this is done, all processes have their
environment set as in the fork-only case, and they can start doing
their work as before.

ok gilles@ jung@

in RSA privsep engine, do not provide methods for rsa_sign / rsa_verify,
they are unused in OpenSMTPD and lead to crashes in -portable when we're
linked to OpenSSL starting with 1.0.2f

ok reyk@

remove spaces after '!'

no binary change

ok millert

EAGAIN handling for imsg_read. OK henning@ benno@

replace u_char and u_int* with standard stdint.h types to ease portable version
also remove trailing whitespaces while here

no binary change

ok sunil millert gilles

KNF

remove a handful of log_warn that we should handle at a different place to
make them really useful

pledge("stdio") the RSA-privsep process

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

missing include

no need to set the same field NULL twice ;-)

ok reyk@

add additional includes required to build with -DOPENSSL_NO_DEPRECATED
ok eric@ gilles@

make the control process broadcast verbose/profile admin requests directly,
rather than going through the parent process. simplify code in the meantime.

fatalx(errorstr) -> fatalx("%s", errorstr)
add missing include and remove redundant debug trace while here.

Create a new default RSA engine instead of patching the existing one
if none is available. Fixes SSL/TLS and a possible fatalx() on
machines without a default RSA engine.

Thanks to Bjorn Ketelaars for reporting and testing.

ok gilles@ (for the relayd part)

Move RSA keys from "lka" to a new dedicated "ca" process because lka
is handling some async requests and shouldn't be busy with sync RSA.

ok gilles@

The RSA engine (used by pony) has to wait for a response from the
privileged process (lka) and receive the imsgs in a while loop
synchronously. But the lka also sends other imsgs (DNS etc.) that can
still be queued up in the buffer when waiting for the RSA response.
This only happens under load with many concurrent connections. For
now, we just call the pony imsg handler for non-RSA imsgs that are
already in the buffer.

ok gilles@ eric@ blambert@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

fail if lka can't load cert file

Report the ssl certificate verification status in the mail header.
Log ssl certificate validation errors.
Fix several ssl-related leaks.

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

remove ca_verify_cb(). was initially used for debugging, then the
logging went away but the no-op callback remained.

noticed by tb@

smtpd: switch ECDSA_METHOD usage to EC_KEY_METHOD

smtpd and the bits it needs in libtls are the only consumer left of
ECDSA_METHOD, which is long deprecated. This paves the way for the
removal in libcrypto.

The diff is from gilles' work on OpenSMTPD-portable, with minor changes
by me.

ok tb@, jsing@

Another missing #include <openssl/err.h>

Revert changes to use the new libtls signer api
There are bugs in the new libtls signer that can lead to a crash.
OK tb@ jsing@

use new libtls signer api

ok tb@

add required headers for smtpd.h and remove unnecessary ones in other files.

ok jung@

replaces calls to err(3)/errx(3) with fatal()/fatalx() from log.c
for code that runs in the daemon.

ok florian@ millert@

Start porting smtpd to libtls.

Note that it changes the way SNI works: The certificate to use is now
selected by looking at the names found in the certificates themselves,
rather than the names of the pki entries in the configuration file.
The set of certificates for a tls listener must be defined explicitly by
using the pki listener option multiple times.

ok tb@

Rename the pony process to dispatcher and klondike to crypto.

From gilles@
OK millert@ giovanni@

properly initialize errstr before going to fail label.

ok gilles@

errstr may be uninitialized in error code path

assume RSA_METHOD is opaque and only access members through setters/getters

a long long time ago, in a galaxy quite close actually, reyk@ introduced an
RSA privsep engine to isolate private keys in the ca process. ECDSA support
in smtpd is become a frequent request so here's an ECDSA privsep engine and
the code required for smtpd to load ECDSA certificates and use them.

assume X509_STORE_CTX is opaque, don't access ->error but use the
X509_STORE_CTX_get_error() function instead

mechanical change to dynamically allocate rsae_method

remove useless check, it's never been and will never be hit

switch smtpd to new grammar

ok eric@

no need to check the sending process in imsg handlers when there is no
ambiguity: just use a single switch.

ok gilles@ sunil@

Introduce more use of freezero(). Also, remove ptr conditionals before
many functions which are free(NULL)-compat
ok gilles

smtpd joins the 7 other daemons that share the same log.c file.

The only major difference was the "log_trace" concept that is only
used by smtpd - move it from log.c into util.c and make it a local
concept. This also needed to rename the global "verbose" variable to
"tracing" in a few places.

OK krw@ gilles@ eric@

Streamline the daemon shutdown sequence.

Only the parent process handles SIGTERM and SIGINT. Upon receiving one
of those, it closes all imsg sockets and waitpid() for the children. It
fatal()s if one of the sockets is closed unexpectedly. Other processes
exit() "normally" when one of the imsg sockets is closed.

ok gilles@ sunil@

The smtpd processes are not expected to ever leave their event loop.
So stop pretending that the *_shutdown() functions could ever be called
in this context, and just fatal() if event_dispatch() returns.

ok gilles@ sunil@ giovanni@

remove noop function

ok sunil@

Implement the fork+exec pattern in smtpd.

The parent process forks child processes and re-exec each of them with
an additional "-x <proc>" argument. During the early setup phase, the
parent process sends ipc socket pairs to interconnect the child
processes as needed, and it passes the queue encryption key to the
queue if necessary. When this is done, all processes have their
environment set as in the fork-only case, and they can start doing
their work as before.

ok gilles@ jung@

in RSA privsep engine, do not provide methods for rsa_sign / rsa_verify,
they are unused in OpenSMTPD and lead to crashes in -portable when we're
linked to OpenSSL starting with 1.0.2f

ok reyk@

remove spaces after '!'

no binary change

ok millert

EAGAIN handling for imsg_read. OK henning@ benno@

replace u_char and u_int* with standard stdint.h types to ease portable version
also remove trailing whitespaces while here

no binary change

ok sunil millert gilles

KNF

remove a handful of log_warn that we should handle at a different place to
make them really useful

pledge("stdio") the RSA-privsep process

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

missing include

no need to set the same field NULL twice ;-)

ok reyk@

add additional includes required to build with -DOPENSSL_NO_DEPRECATED
ok eric@ gilles@

make the control process broadcast verbose/profile admin requests directly,
rather than going through the parent process. simplify code in the meantime.

fatalx(errorstr) -> fatalx("%s", errorstr)
add missing include and remove redundant debug trace while here.

Create a new default RSA engine instead of patching the existing one
if none is available. Fixes SSL/TLS and a possible fatalx() on
machines without a default RSA engine.

Thanks to Bjorn Ketelaars for reporting and testing.

ok gilles@ (for the relayd part)

Move RSA keys from "lka" to a new dedicated "ca" process because lka
is handling some async requests and shouldn't be busy with sync RSA.

ok gilles@

The RSA engine (used by pony) has to wait for a response from the
privileged process (lka) and receive the imsgs in a while loop
synchronously. But the lka also sends other imsgs (DNS etc.) that can
still be queued up in the buffer when waiting for the RSA response.
This only happens under load with many concurrent connections. For
now, we just call the pony imsg handler for non-RSA imsgs that are
already in the buffer.

ok gilles@ eric@ blambert@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

fail if lka can't load cert file

Report the ssl certificate verification status in the mail header.
Log ssl certificate validation errors.
Fix several ssl-related leaks.

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

Another missing #include <openssl/err.h>

Revert changes to use the new libtls signer api
There are bugs in the new libtls signer that can lead to a crash.
OK tb@ jsing@

use new libtls signer api

ok tb@

add required headers for smtpd.h and remove unnecessary ones in other files.

ok jung@

replaces calls to err(3)/errx(3) with fatal()/fatalx() from log.c
for code that runs in the daemon.

ok florian@ millert@

Start porting smtpd to libtls.

Note that it changes the way SNI works: The certificate to use is now
selected by looking at the names found in the certificates themselves,
rather than the names of the pki entries in the configuration file.
The set of certificates for a tls listener must be defined explicitly by
using the pki listener option multiple times.

ok tb@

Rename the pony process to dispatcher and klondike to crypto.

From gilles@
OK millert@ giovanni@

properly initialize errstr before going to fail label.

ok gilles@

errstr may be uninitialized in error code path

assume RSA_METHOD is opaque and only access members through setters/getters

a long long time ago, in a galaxy quite close actually, reyk@ introduced an
RSA privsep engine to isolate private keys in the ca process. ECDSA support
in smtpd is become a frequent request so here's an ECDSA privsep engine and
the code required for smtpd to load ECDSA certificates and use them.

assume X509_STORE_CTX is opaque, don't access ->error but use the
X509_STORE_CTX_get_error() function instead

mechanical change to dynamically allocate rsae_method

remove useless check, it's never been and will never be hit

switch smtpd to new grammar

ok eric@

no need to check the sending process in imsg handlers when there is no
ambiguity: just use a single switch.

ok gilles@ sunil@

Introduce more use of freezero(). Also, remove ptr conditionals before
many functions which are free(NULL)-compat
ok gilles

smtpd joins the 7 other daemons that share the same log.c file.

The only major difference was the "log_trace" concept that is only
used by smtpd - move it from log.c into util.c and make it a local
concept. This also needed to rename the global "verbose" variable to
"tracing" in a few places.

OK krw@ gilles@ eric@

Streamline the daemon shutdown sequence.

Only the parent process handles SIGTERM and SIGINT. Upon receiving one
of those, it closes all imsg sockets and waitpid() for the children. It
fatal()s if one of the sockets is closed unexpectedly. Other processes
exit() "normally" when one of the imsg sockets is closed.

ok gilles@ sunil@

The smtpd processes are not expected to ever leave their event loop.
So stop pretending that the *_shutdown() functions could ever be called
in this context, and just fatal() if event_dispatch() returns.

ok gilles@ sunil@ giovanni@

remove noop function

ok sunil@

Implement the fork+exec pattern in smtpd.

The parent process forks child processes and re-exec each of them with
an additional "-x <proc>" argument. During the early setup phase, the
parent process sends ipc socket pairs to interconnect the child
processes as needed, and it passes the queue encryption key to the
queue if necessary. When this is done, all processes have their
environment set as in the fork-only case, and they can start doing
their work as before.

ok gilles@ jung@

in RSA privsep engine, do not provide methods for rsa_sign / rsa_verify,
they are unused in OpenSMTPD and lead to crashes in -portable when we're
linked to OpenSSL starting with 1.0.2f

ok reyk@

remove spaces after '!'

no binary change

ok millert

EAGAIN handling for imsg_read. OK henning@ benno@

replace u_char and u_int* with standard stdint.h types to ease portable version
also remove trailing whitespaces while here

no binary change

ok sunil millert gilles

KNF

remove a handful of log_warn that we should handle at a different place to
make them really useful

pledge("stdio") the RSA-privsep process

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

missing include

no need to set the same field NULL twice ;-)

ok reyk@

add additional includes required to build with -DOPENSSL_NO_DEPRECATED
ok eric@ gilles@

make the control process broadcast verbose/profile admin requests directly,
rather than going through the parent process. simplify code in the meantime.

fatalx(errorstr) -> fatalx("%s", errorstr)
add missing include and remove redundant debug trace while here.

Create a new default RSA engine instead of patching the existing one
if none is available. Fixes SSL/TLS and a possible fatalx() on
machines without a default RSA engine.

Thanks to Bjorn Ketelaars for reporting and testing.

ok gilles@ (for the relayd part)

Move RSA keys from "lka" to a new dedicated "ca" process because lka
is handling some async requests and shouldn't be busy with sync RSA.

ok gilles@

The RSA engine (used by pony) has to wait for a response from the
privileged process (lka) and receive the imsgs in a while loop
synchronously. But the lka also sends other imsgs (DNS etc.) that can
still be queued up in the buffer when waiting for the RSA response.
This only happens under load with many concurrent connections. For
now, we just call the pony imsg handler for non-RSA imsgs that are
already in the buffer.

ok gilles@ eric@ blambert@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

fail if lka can't load cert file

Report the ssl certificate verification status in the mail header.
Log ssl certificate validation errors.
Fix several ssl-related leaks.

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

Revert changes to use the new libtls signer api
There are bugs in the new libtls signer that can lead to a crash.
OK tb@ jsing@

use new libtls signer api

ok tb@

add required headers for smtpd.h and remove unnecessary ones in other files.

ok jung@

replaces calls to err(3)/errx(3) with fatal()/fatalx() from log.c
for code that runs in the daemon.

ok florian@ millert@

Start porting smtpd to libtls.

Note that it changes the way SNI works: The certificate to use is now
selected by looking at the names found in the certificates themselves,
rather than the names of the pki entries in the configuration file.
The set of certificates for a tls listener must be defined explicitly by
using the pki listener option multiple times.

ok tb@

Rename the pony process to dispatcher and klondike to crypto.

From gilles@
OK millert@ giovanni@

properly initialize errstr before going to fail label.

ok gilles@

errstr may be uninitialized in error code path

assume RSA_METHOD is opaque and only access members through setters/getters

a long long time ago, in a galaxy quite close actually, reyk@ introduced an
RSA privsep engine to isolate private keys in the ca process. ECDSA support
in smtpd is become a frequent request so here's an ECDSA privsep engine and
the code required for smtpd to load ECDSA certificates and use them.

assume X509_STORE_CTX is opaque, don't access ->error but use the
X509_STORE_CTX_get_error() function instead

mechanical change to dynamically allocate rsae_method

remove useless check, it's never been and will never be hit

switch smtpd to new grammar

ok eric@

no need to check the sending process in imsg handlers when there is no
ambiguity: just use a single switch.

ok gilles@ sunil@

Introduce more use of freezero(). Also, remove ptr conditionals before
many functions which are free(NULL)-compat
ok gilles

smtpd joins the 7 other daemons that share the same log.c file.

The only major difference was the "log_trace" concept that is only
used by smtpd - move it from log.c into util.c and make it a local
concept. This also needed to rename the global "verbose" variable to
"tracing" in a few places.

OK krw@ gilles@ eric@

Streamline the daemon shutdown sequence.

Only the parent process handles SIGTERM and SIGINT. Upon receiving one
of those, it closes all imsg sockets and waitpid() for the children. It
fatal()s if one of the sockets is closed unexpectedly. Other processes
exit() "normally" when one of the imsg sockets is closed.

ok gilles@ sunil@

The smtpd processes are not expected to ever leave their event loop.
So stop pretending that the *_shutdown() functions could ever be called
in this context, and just fatal() if event_dispatch() returns.

ok gilles@ sunil@ giovanni@

remove noop function

ok sunil@

Implement the fork+exec pattern in smtpd.

The parent process forks child processes and re-exec each of them with
an additional "-x <proc>" argument. During the early setup phase, the
parent process sends ipc socket pairs to interconnect the child
processes as needed, and it passes the queue encryption key to the
queue if necessary. When this is done, all processes have their
environment set as in the fork-only case, and they can start doing
their work as before.

ok gilles@ jung@

in RSA privsep engine, do not provide methods for rsa_sign / rsa_verify,
they are unused in OpenSMTPD and lead to crashes in -portable when we're
linked to OpenSSL starting with 1.0.2f

ok reyk@

remove spaces after '!'

no binary change

ok millert

EAGAIN handling for imsg_read. OK henning@ benno@

replace u_char and u_int* with standard stdint.h types to ease portable version
also remove trailing whitespaces while here

no binary change

ok sunil millert gilles

KNF

remove a handful of log_warn that we should handle at a different place to
make them really useful

pledge("stdio") the RSA-privsep process

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

missing include

no need to set the same field NULL twice ;-)

ok reyk@

add additional includes required to build with -DOPENSSL_NO_DEPRECATED
ok eric@ gilles@

make the control process broadcast verbose/profile admin requests directly,
rather than going through the parent process. simplify code in the meantime.

fatalx(errorstr) -> fatalx("%s", errorstr)
add missing include and remove redundant debug trace while here.

Create a new default RSA engine instead of patching the existing one
if none is available. Fixes SSL/TLS and a possible fatalx() on
machines without a default RSA engine.

Thanks to Bjorn Ketelaars for reporting and testing.

ok gilles@ (for the relayd part)

Move RSA keys from "lka" to a new dedicated "ca" process because lka
is handling some async requests and shouldn't be busy with sync RSA.

ok gilles@

The RSA engine (used by pony) has to wait for a response from the
privileged process (lka) and receive the imsgs in a while loop
synchronously. But the lka also sends other imsgs (DNS etc.) that can
still be queued up in the buffer when waiting for the RSA response.
This only happens under load with many concurrent connections. For
now, we just call the pony imsg handler for non-RSA imsgs that are
already in the buffer.

ok gilles@ eric@ blambert@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

fail if lka can't load cert file

Report the ssl certificate verification status in the mail header.
Log ssl certificate validation errors.
Fix several ssl-related leaks.

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

use new libtls signer api

ok tb@

add required headers for smtpd.h and remove unnecessary ones in other files.

ok jung@

replaces calls to err(3)/errx(3) with fatal()/fatalx() from log.c
for code that runs in the daemon.

ok florian@ millert@

Start porting smtpd to libtls.

Note that it changes the way SNI works: The certificate to use is now
selected by looking at the names found in the certificates themselves,
rather than the names of the pki entries in the configuration file.
The set of certificates for a tls listener must be defined explicitly by
using the pki listener option multiple times.

ok tb@

Rename the pony process to dispatcher and klondike to crypto.

From gilles@
OK millert@ giovanni@

properly initialize errstr before going to fail label.

ok gilles@

errstr may be uninitialized in error code path

assume RSA_METHOD is opaque and only access members through setters/getters

a long long time ago, in a galaxy quite close actually, reyk@ introduced an
RSA privsep engine to isolate private keys in the ca process. ECDSA support
in smtpd is become a frequent request so here's an ECDSA privsep engine and
the code required for smtpd to load ECDSA certificates and use them.

assume X509_STORE_CTX is opaque, don't access ->error but use the
X509_STORE_CTX_get_error() function instead

mechanical change to dynamically allocate rsae_method

remove useless check, it's never been and will never be hit

switch smtpd to new grammar

ok eric@

no need to check the sending process in imsg handlers when there is no
ambiguity: just use a single switch.

ok gilles@ sunil@

Introduce more use of freezero(). Also, remove ptr conditionals before
many functions which are free(NULL)-compat
ok gilles

smtpd joins the 7 other daemons that share the same log.c file.

The only major difference was the "log_trace" concept that is only
used by smtpd - move it from log.c into util.c and make it a local
concept. This also needed to rename the global "verbose" variable to
"tracing" in a few places.

OK krw@ gilles@ eric@

Streamline the daemon shutdown sequence.

Only the parent process handles SIGTERM and SIGINT. Upon receiving one
of those, it closes all imsg sockets and waitpid() for the children. It
fatal()s if one of the sockets is closed unexpectedly. Other processes
exit() "normally" when one of the imsg sockets is closed.

ok gilles@ sunil@

The smtpd processes are not expected to ever leave their event loop.
So stop pretending that the *_shutdown() functions could ever be called
in this context, and just fatal() if event_dispatch() returns.

ok gilles@ sunil@ giovanni@

remove noop function

ok sunil@

Implement the fork+exec pattern in smtpd.

The parent process forks child processes and re-exec each of them with
an additional "-x <proc>" argument. During the early setup phase, the
parent process sends ipc socket pairs to interconnect the child
processes as needed, and it passes the queue encryption key to the
queue if necessary. When this is done, all processes have their
environment set as in the fork-only case, and they can start doing
their work as before.

ok gilles@ jung@

in RSA privsep engine, do not provide methods for rsa_sign / rsa_verify,
they are unused in OpenSMTPD and lead to crashes in -portable when we're
linked to OpenSSL starting with 1.0.2f

ok reyk@

remove spaces after '!'

no binary change

ok millert

EAGAIN handling for imsg_read. OK henning@ benno@

replace u_char and u_int* with standard stdint.h types to ease portable version
also remove trailing whitespaces while here

no binary change

ok sunil millert gilles

KNF

remove a handful of log_warn that we should handle at a different place to
make them really useful

pledge("stdio") the RSA-privsep process

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

missing include

no need to set the same field NULL twice ;-)

ok reyk@

add additional includes required to build with -DOPENSSL_NO_DEPRECATED
ok eric@ gilles@

make the control process broadcast verbose/profile admin requests directly,
rather than going through the parent process. simplify code in the meantime.

fatalx(errorstr) -> fatalx("%s", errorstr)
add missing include and remove redundant debug trace while here.

Create a new default RSA engine instead of patching the existing one
if none is available. Fixes SSL/TLS and a possible fatalx() on
machines without a default RSA engine.

Thanks to Bjorn Ketelaars for reporting and testing.

ok gilles@ (for the relayd part)

Move RSA keys from "lka" to a new dedicated "ca" process because lka
is handling some async requests and shouldn't be busy with sync RSA.

ok gilles@

The RSA engine (used by pony) has to wait for a response from the
privileged process (lka) and receive the imsgs in a while loop
synchronously. But the lka also sends other imsgs (DNS etc.) that can
still be queued up in the buffer when waiting for the RSA response.
This only happens under load with many concurrent connections. For
now, we just call the pony imsg handler for non-RSA imsgs that are
already in the buffer.

ok gilles@ eric@ blambert@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

fail if lka can't load cert file

Report the ssl certificate verification status in the mail header.
Log ssl certificate validation errors.
Fix several ssl-related leaks.

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

add required headers for smtpd.h and remove unnecessary ones in other files.

ok jung@

replaces calls to err(3)/errx(3) with fatal()/fatalx() from log.c
for code that runs in the daemon.

ok florian@ millert@

Start porting smtpd to libtls.

Note that it changes the way SNI works: The certificate to use is now
selected by looking at the names found in the certificates themselves,
rather than the names of the pki entries in the configuration file.
The set of certificates for a tls listener must be defined explicitly by
using the pki listener option multiple times.

ok tb@

Rename the pony process to dispatcher and klondike to crypto.

From gilles@
OK millert@ giovanni@

properly initialize errstr before going to fail label.

ok gilles@

errstr may be uninitialized in error code path

assume RSA_METHOD is opaque and only access members through setters/getters

a long long time ago, in a galaxy quite close actually, reyk@ introduced an
RSA privsep engine to isolate private keys in the ca process. ECDSA support
in smtpd is become a frequent request so here's an ECDSA privsep engine and
the code required for smtpd to load ECDSA certificates and use them.

assume X509_STORE_CTX is opaque, don't access ->error but use the
X509_STORE_CTX_get_error() function instead

mechanical change to dynamically allocate rsae_method

remove useless check, it's never been and will never be hit

switch smtpd to new grammar

ok eric@

no need to check the sending process in imsg handlers when there is no
ambiguity: just use a single switch.

ok gilles@ sunil@

Introduce more use of freezero(). Also, remove ptr conditionals before
many functions which are free(NULL)-compat
ok gilles

smtpd joins the 7 other daemons that share the same log.c file.

The only major difference was the "log_trace" concept that is only
used by smtpd - move it from log.c into util.c and make it a local
concept. This also needed to rename the global "verbose" variable to
"tracing" in a few places.

OK krw@ gilles@ eric@

Streamline the daemon shutdown sequence.

Only the parent process handles SIGTERM and SIGINT. Upon receiving one
of those, it closes all imsg sockets and waitpid() for the children. It
fatal()s if one of the sockets is closed unexpectedly. Other processes
exit() "normally" when one of the imsg sockets is closed.

ok gilles@ sunil@

The smtpd processes are not expected to ever leave their event loop.
So stop pretending that the *_shutdown() functions could ever be called
in this context, and just fatal() if event_dispatch() returns.

ok gilles@ sunil@ giovanni@

remove noop function

ok sunil@

Implement the fork+exec pattern in smtpd.

The parent process forks child processes and re-exec each of them with
an additional "-x <proc>" argument. During the early setup phase, the
parent process sends ipc socket pairs to interconnect the child
processes as needed, and it passes the queue encryption key to the
queue if necessary. When this is done, all processes have their
environment set as in the fork-only case, and they can start doing
their work as before.

ok gilles@ jung@

in RSA privsep engine, do not provide methods for rsa_sign / rsa_verify,
they are unused in OpenSMTPD and lead to crashes in -portable when we're
linked to OpenSSL starting with 1.0.2f

ok reyk@

remove spaces after '!'

no binary change

ok millert

EAGAIN handling for imsg_read. OK henning@ benno@

replace u_char and u_int* with standard stdint.h types to ease portable version
also remove trailing whitespaces while here

no binary change

ok sunil millert gilles

KNF

remove a handful of log_warn that we should handle at a different place to
make them really useful

pledge("stdio") the RSA-privsep process

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

missing include

no need to set the same field NULL twice ;-)

ok reyk@

add additional includes required to build with -DOPENSSL_NO_DEPRECATED
ok eric@ gilles@

make the control process broadcast verbose/profile admin requests directly,
rather than going through the parent process. simplify code in the meantime.

fatalx(errorstr) -> fatalx("%s", errorstr)
add missing include and remove redundant debug trace while here.

Create a new default RSA engine instead of patching the existing one
if none is available. Fixes SSL/TLS and a possible fatalx() on
machines without a default RSA engine.

Thanks to Bjorn Ketelaars for reporting and testing.

ok gilles@ (for the relayd part)

Move RSA keys from "lka" to a new dedicated "ca" process because lka
is handling some async requests and shouldn't be busy with sync RSA.

ok gilles@

The RSA engine (used by pony) has to wait for a response from the
privileged process (lka) and receive the imsgs in a while loop
synchronously. But the lka also sends other imsgs (DNS etc.) that can
still be queued up in the buffer when waiting for the RSA response.
This only happens under load with many concurrent connections. For
now, we just call the pony imsg handler for non-RSA imsgs that are
already in the buffer.

ok gilles@ eric@ blambert@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

fail if lka can't load cert file

Report the ssl certificate verification status in the mail header.
Log ssl certificate validation errors.
Fix several ssl-related leaks.

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

replaces calls to err(3)/errx(3) with fatal()/fatalx() from log.c
for code that runs in the daemon.

ok florian@ millert@

Start porting smtpd to libtls.

Note that it changes the way SNI works: The certificate to use is now
selected by looking at the names found in the certificates themselves,
rather than the names of the pki entries in the configuration file.
The set of certificates for a tls listener must be defined explicitly by
using the pki listener option multiple times.

ok tb@

Rename the pony process to dispatcher and klondike to crypto.

From gilles@
OK millert@ giovanni@

properly initialize errstr before going to fail label.

ok gilles@

errstr may be uninitialized in error code path

assume RSA_METHOD is opaque and only access members through setters/getters

a long long time ago, in a galaxy quite close actually, reyk@ introduced an
RSA privsep engine to isolate private keys in the ca process. ECDSA support
in smtpd is become a frequent request so here's an ECDSA privsep engine and
the code required for smtpd to load ECDSA certificates and use them.

assume X509_STORE_CTX is opaque, don't access ->error but use the
X509_STORE_CTX_get_error() function instead

mechanical change to dynamically allocate rsae_method

remove useless check, it's never been and will never be hit

switch smtpd to new grammar

ok eric@

no need to check the sending process in imsg handlers when there is no
ambiguity: just use a single switch.

ok gilles@ sunil@

Introduce more use of freezero(). Also, remove ptr conditionals before
many functions which are free(NULL)-compat
ok gilles

smtpd joins the 7 other daemons that share the same log.c file.

The only major difference was the "log_trace" concept that is only
used by smtpd - move it from log.c into util.c and make it a local
concept. This also needed to rename the global "verbose" variable to
"tracing" in a few places.

OK krw@ gilles@ eric@

Streamline the daemon shutdown sequence.

Only the parent process handles SIGTERM and SIGINT. Upon receiving one
of those, it closes all imsg sockets and waitpid() for the children. It
fatal()s if one of the sockets is closed unexpectedly. Other processes
exit() "normally" when one of the imsg sockets is closed.

ok gilles@ sunil@

The smtpd processes are not expected to ever leave their event loop.
So stop pretending that the *_shutdown() functions could ever be called
in this context, and just fatal() if event_dispatch() returns.

ok gilles@ sunil@ giovanni@

remove noop function

ok sunil@

Implement the fork+exec pattern in smtpd.

The parent process forks child processes and re-exec each of them with
an additional "-x <proc>" argument. During the early setup phase, the
parent process sends ipc socket pairs to interconnect the child
processes as needed, and it passes the queue encryption key to the
queue if necessary. When this is done, all processes have their
environment set as in the fork-only case, and they can start doing
their work as before.

ok gilles@ jung@

in RSA privsep engine, do not provide methods for rsa_sign / rsa_verify,
they are unused in OpenSMTPD and lead to crashes in -portable when we're
linked to OpenSSL starting with 1.0.2f

ok reyk@

remove spaces after '!'

no binary change

ok millert

EAGAIN handling for imsg_read. OK henning@ benno@

replace u_char and u_int* with standard stdint.h types to ease portable version
also remove trailing whitespaces while here

no binary change

ok sunil millert gilles

KNF

remove a handful of log_warn that we should handle at a different place to
make them really useful

pledge("stdio") the RSA-privsep process

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

missing include

no need to set the same field NULL twice ;-)

ok reyk@

add additional includes required to build with -DOPENSSL_NO_DEPRECATED
ok eric@ gilles@

make the control process broadcast verbose/profile admin requests directly,
rather than going through the parent process. simplify code in the meantime.

fatalx(errorstr) -> fatalx("%s", errorstr)
add missing include and remove redundant debug trace while here.

Create a new default RSA engine instead of patching the existing one
if none is available. Fixes SSL/TLS and a possible fatalx() on
machines without a default RSA engine.

Thanks to Bjorn Ketelaars for reporting and testing.

ok gilles@ (for the relayd part)

Move RSA keys from "lka" to a new dedicated "ca" process because lka
is handling some async requests and shouldn't be busy with sync RSA.

ok gilles@

The RSA engine (used by pony) has to wait for a response from the
privileged process (lka) and receive the imsgs in a while loop
synchronously. But the lka also sends other imsgs (DNS etc.) that can
still be queued up in the buffer when waiting for the RSA response.
This only happens under load with many concurrent connections. For
now, we just call the pony imsg handler for non-RSA imsgs that are
already in the buffer.

ok gilles@ eric@ blambert@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

fail if lka can't load cert file

Report the ssl certificate verification status in the mail header.
Log ssl certificate validation errors.
Fix several ssl-related leaks.

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

Start porting smtpd to libtls.

Note that it changes the way SNI works: The certificate to use is now
selected by looking at the names found in the certificates themselves,
rather than the names of the pki entries in the configuration file.
The set of certificates for a tls listener must be defined explicitly by
using the pki listener option multiple times.

ok tb@

Rename the pony process to dispatcher and klondike to crypto.

From gilles@
OK millert@ giovanni@

properly initialize errstr before going to fail label.

ok gilles@

errstr may be uninitialized in error code path

assume RSA_METHOD is opaque and only access members through setters/getters

a long long time ago, in a galaxy quite close actually, reyk@ introduced an
RSA privsep engine to isolate private keys in the ca process. ECDSA support
in smtpd is become a frequent request so here's an ECDSA privsep engine and
the code required for smtpd to load ECDSA certificates and use them.

assume X509_STORE_CTX is opaque, don't access ->error but use the
X509_STORE_CTX_get_error() function instead

mechanical change to dynamically allocate rsae_method

remove useless check, it's never been and will never be hit

switch smtpd to new grammar

ok eric@

no need to check the sending process in imsg handlers when there is no
ambiguity: just use a single switch.

ok gilles@ sunil@

Introduce more use of freezero(). Also, remove ptr conditionals before
many functions which are free(NULL)-compat
ok gilles

smtpd joins the 7 other daemons that share the same log.c file.

The only major difference was the "log_trace" concept that is only
used by smtpd - move it from log.c into util.c and make it a local
concept. This also needed to rename the global "verbose" variable to
"tracing" in a few places.

OK krw@ gilles@ eric@

Streamline the daemon shutdown sequence.

Only the parent process handles SIGTERM and SIGINT. Upon receiving one
of those, it closes all imsg sockets and waitpid() for the children. It
fatal()s if one of the sockets is closed unexpectedly. Other processes
exit() "normally" when one of the imsg sockets is closed.

ok gilles@ sunil@

The smtpd processes are not expected to ever leave their event loop.
So stop pretending that the *_shutdown() functions could ever be called
in this context, and just fatal() if event_dispatch() returns.

ok gilles@ sunil@ giovanni@

remove noop function

ok sunil@

Implement the fork+exec pattern in smtpd.

The parent process forks child processes and re-exec each of them with
an additional "-x <proc>" argument. During the early setup phase, the
parent process sends ipc socket pairs to interconnect the child
processes as needed, and it passes the queue encryption key to the
queue if necessary. When this is done, all processes have their
environment set as in the fork-only case, and they can start doing
their work as before.

ok gilles@ jung@

in RSA privsep engine, do not provide methods for rsa_sign / rsa_verify,
they are unused in OpenSMTPD and lead to crashes in -portable when we're
linked to OpenSSL starting with 1.0.2f

ok reyk@

remove spaces after '!'

no binary change

ok millert

EAGAIN handling for imsg_read. OK henning@ benno@

replace u_char and u_int* with standard stdint.h types to ease portable version
also remove trailing whitespaces while here

no binary change

ok sunil millert gilles

KNF

remove a handful of log_warn that we should handle at a different place to
make them really useful

pledge("stdio") the RSA-privsep process

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

missing include

no need to set the same field NULL twice ;-)

ok reyk@

add additional includes required to build with -DOPENSSL_NO_DEPRECATED
ok eric@ gilles@

make the control process broadcast verbose/profile admin requests directly,
rather than going through the parent process. simplify code in the meantime.

fatalx(errorstr) -> fatalx("%s", errorstr)
add missing include and remove redundant debug trace while here.

Create a new default RSA engine instead of patching the existing one
if none is available. Fixes SSL/TLS and a possible fatalx() on
machines without a default RSA engine.

Thanks to Bjorn Ketelaars for reporting and testing.

ok gilles@ (for the relayd part)

Move RSA keys from "lka" to a new dedicated "ca" process because lka
is handling some async requests and shouldn't be busy with sync RSA.

ok gilles@

The RSA engine (used by pony) has to wait for a response from the
privileged process (lka) and receive the imsgs in a while loop
synchronously. But the lka also sends other imsgs (DNS etc.) that can
still be queued up in the buffer when waiting for the RSA response.
This only happens under load with many concurrent connections. For
now, we just call the pony imsg handler for non-RSA imsgs that are
already in the buffer.

ok gilles@ eric@ blambert@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

fail if lka can't load cert file

Report the ssl certificate verification status in the mail header.
Log ssl certificate validation errors.
Fix several ssl-related leaks.

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

Rename the pony process to dispatcher and klondike to crypto.

From gilles@
OK millert@ giovanni@

properly initialize errstr before going to fail label.

ok gilles@

errstr may be uninitialized in error code path

assume RSA_METHOD is opaque and only access members through setters/getters

a long long time ago, in a galaxy quite close actually, reyk@ introduced an
RSA privsep engine to isolate private keys in the ca process. ECDSA support
in smtpd is become a frequent request so here's an ECDSA privsep engine and
the code required for smtpd to load ECDSA certificates and use them.

assume X509_STORE_CTX is opaque, don't access ->error but use the
X509_STORE_CTX_get_error() function instead

mechanical change to dynamically allocate rsae_method

remove useless check, it's never been and will never be hit

switch smtpd to new grammar

ok eric@

no need to check the sending process in imsg handlers when there is no
ambiguity: just use a single switch.

ok gilles@ sunil@

Introduce more use of freezero(). Also, remove ptr conditionals before
many functions which are free(NULL)-compat
ok gilles

smtpd joins the 7 other daemons that share the same log.c file.

The only major difference was the "log_trace" concept that is only
used by smtpd - move it from log.c into util.c and make it a local
concept. This also needed to rename the global "verbose" variable to
"tracing" in a few places.

OK krw@ gilles@ eric@

Streamline the daemon shutdown sequence.

Only the parent process handles SIGTERM and SIGINT. Upon receiving one
of those, it closes all imsg sockets and waitpid() for the children. It
fatal()s if one of the sockets is closed unexpectedly. Other processes
exit() "normally" when one of the imsg sockets is closed.

ok gilles@ sunil@

The smtpd processes are not expected to ever leave their event loop.
So stop pretending that the *_shutdown() functions could ever be called
in this context, and just fatal() if event_dispatch() returns.

ok gilles@ sunil@ giovanni@

remove noop function

ok sunil@

Implement the fork+exec pattern in smtpd.

The parent process forks child processes and re-exec each of them with
an additional "-x <proc>" argument. During the early setup phase, the
parent process sends ipc socket pairs to interconnect the child
processes as needed, and it passes the queue encryption key to the
queue if necessary. When this is done, all processes have their
environment set as in the fork-only case, and they can start doing
their work as before.

ok gilles@ jung@

in RSA privsep engine, do not provide methods for rsa_sign / rsa_verify,
they are unused in OpenSMTPD and lead to crashes in -portable when we're
linked to OpenSSL starting with 1.0.2f

ok reyk@

remove spaces after '!'

no binary change

ok millert

EAGAIN handling for imsg_read. OK henning@ benno@

replace u_char and u_int* with standard stdint.h types to ease portable version
also remove trailing whitespaces while here

no binary change

ok sunil millert gilles

KNF

remove a handful of log_warn that we should handle at a different place to
make them really useful

pledge("stdio") the RSA-privsep process

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

missing include

no need to set the same field NULL twice ;-)

ok reyk@

add additional includes required to build with -DOPENSSL_NO_DEPRECATED
ok eric@ gilles@

make the control process broadcast verbose/profile admin requests directly,
rather than going through the parent process. simplify code in the meantime.

fatalx(errorstr) -> fatalx("%s", errorstr)
add missing include and remove redundant debug trace while here.

Create a new default RSA engine instead of patching the existing one
if none is available. Fixes SSL/TLS and a possible fatalx() on
machines without a default RSA engine.

Thanks to Bjorn Ketelaars for reporting and testing.

ok gilles@ (for the relayd part)

Move RSA keys from "lka" to a new dedicated "ca" process because lka
is handling some async requests and shouldn't be busy with sync RSA.

ok gilles@

The RSA engine (used by pony) has to wait for a response from the
privileged process (lka) and receive the imsgs in a while loop
synchronously. But the lka also sends other imsgs (DNS etc.) that can
still be queued up in the buffer when waiting for the RSA response.
This only happens under load with many concurrent connections. For
now, we just call the pony imsg handler for non-RSA imsgs that are
already in the buffer.

ok gilles@ eric@ blambert@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

fail if lka can't load cert file

Report the ssl certificate verification status in the mail header.
Log ssl certificate validation errors.
Fix several ssl-related leaks.

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

properly initialize errstr before going to fail label.

ok gilles@

errstr may be uninitialized in error code path

assume RSA_METHOD is opaque and only access members through setters/getters

a long long time ago, in a galaxy quite close actually, reyk@ introduced an
RSA privsep engine to isolate private keys in the ca process. ECDSA support
in smtpd is become a frequent request so here's an ECDSA privsep engine and
the code required for smtpd to load ECDSA certificates and use them.

assume X509_STORE_CTX is opaque, don't access ->error but use the
X509_STORE_CTX_get_error() function instead

mechanical change to dynamically allocate rsae_method

remove useless check, it's never been and will never be hit

switch smtpd to new grammar

ok eric@

no need to check the sending process in imsg handlers when there is no
ambiguity: just use a single switch.

ok gilles@ sunil@

Introduce more use of freezero(). Also, remove ptr conditionals before
many functions which are free(NULL)-compat
ok gilles

smtpd joins the 7 other daemons that share the same log.c file.

The only major difference was the "log_trace" concept that is only
used by smtpd - move it from log.c into util.c and make it a local
concept. This also needed to rename the global "verbose" variable to
"tracing" in a few places.

OK krw@ gilles@ eric@

Streamline the daemon shutdown sequence.

Only the parent process handles SIGTERM and SIGINT. Upon receiving one
of those, it closes all imsg sockets and waitpid() for the children. It
fatal()s if one of the sockets is closed unexpectedly. Other processes
exit() "normally" when one of the imsg sockets is closed.

ok gilles@ sunil@

The smtpd processes are not expected to ever leave their event loop.
So stop pretending that the *_shutdown() functions could ever be called
in this context, and just fatal() if event_dispatch() returns.

ok gilles@ sunil@ giovanni@

remove noop function

ok sunil@

Implement the fork+exec pattern in smtpd.

The parent process forks child processes and re-exec each of them with
an additional "-x <proc>" argument. During the early setup phase, the
parent process sends ipc socket pairs to interconnect the child
processes as needed, and it passes the queue encryption key to the
queue if necessary. When this is done, all processes have their
environment set as in the fork-only case, and they can start doing
their work as before.

ok gilles@ jung@

in RSA privsep engine, do not provide methods for rsa_sign / rsa_verify,
they are unused in OpenSMTPD and lead to crashes in -portable when we're
linked to OpenSSL starting with 1.0.2f

ok reyk@

remove spaces after '!'

no binary change

ok millert

EAGAIN handling for imsg_read. OK henning@ benno@

replace u_char and u_int* with standard stdint.h types to ease portable version
also remove trailing whitespaces while here

no binary change

ok sunil millert gilles

KNF

remove a handful of log_warn that we should handle at a different place to
make them really useful

pledge("stdio") the RSA-privsep process

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

Replace <sys/param.h> with <limits.h> and other less dirty headers where
possible. Annotate <sys/param.h> lines with their current reasons. Switch
to PATH_MAX, NGROUPS_MAX, HOST_NAME_MAX+1, LOGIN_NAME_MAX, etc. Change
MIN() and MAX() to local definitions of MINIMUM() and MAXIMUM() where
sensible to avoid pulling in the pollution. These are the files confirmed
through binary verification.
ok guenther, millert, doug (helped with the verification protocol)

missing include

no need to set the same field NULL twice ;-)

ok reyk@

add additional includes required to build with -DOPENSSL_NO_DEPRECATED
ok eric@ gilles@

make the control process broadcast verbose/profile admin requests directly,
rather than going through the parent process. simplify code in the meantime.

fatalx(errorstr) -> fatalx("%s", errorstr)
add missing include and remove redundant debug trace while here.

Create a new default RSA engine instead of patching the existing one
if none is available. Fixes SSL/TLS and a possible fatalx() on
machines without a default RSA engine.

Thanks to Bjorn Ketelaars for reporting and testing.

ok gilles@ (for the relayd part)

Move RSA keys from "lka" to a new dedicated "ca" process because lka
is handling some async requests and shouldn't be busy with sync RSA.

ok gilles@

The RSA engine (used by pony) has to wait for a response from the
privileged process (lka) and receive the imsgs in a while loop
synchronously. But the lka also sends other imsgs (DNS etc.) that can
still be queued up in the buffer when waiting for the RSA response.
This only happens under load with many concurrent connections. For
now, we just call the pony imsg handler for non-RSA imsgs that are
already in the buffer.

ok gilles@ eric@ blambert@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

fail if lka can't load cert file

Report the ssl certificate verification status in the mail header.
Log ssl certificate validation errors.
Fix several ssl-related leaks.

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@