remove ssl_init()

it's a noop; nowadays both LibreSSL and OpenSSL libcrypto and libssl
initialize themselves automatically before doing anything.

noticed by jsing, ok tb

Revert changes to use the new libtls signer api
There are bugs in the new libtls signer that can lead to a crash.
OK tb@ jsing@

use new libtls signer api

ok tb@

remove unused code

ok millert@

do not build unused code and remove uneeded dependency on libm.

ok tb@

Start porting smtpd to libtls.

Note that it changes the way SNI works: The certificate to use is now
selected by looking at the names found in the certificates themselves,
rather than the names of the pki entries in the configuration file.
The set of certificates for a tls listener must be defined explicitly by
using the pki listener option multiple times.

ok tb@

Implement server certificate validation in smtp(1).
Check certificate against MX name in smtpd(8) mta.

ok gilles@

Use automatic DH parameters, instead of fixed ones. Also disable DHE by
default since it is computationally expensive and a potential DoS vector.

ok gilles@

refactor a bit to move the SNI handling away from smtp_session into smtp

ok sunil@, jung@

remove CA from pki and no longer allow specifying a CA with 'pki' keyword.
introduce 'ca' keyword to allow specifying a custom CA.
making CA part of pki was a bad idea and several people hit use-cases that
plain couldn't work.

instead of:
pki foobar.org ca "/etc/mail/CA.pem"

use now:
ca foobar.org certificate "/etc/mail/CA.pem"


ok sunil@, jung@

allow overriding the default cipher-suite

ok jung@, sunil@, millert@

bump DH params to 2048, it's been part of smtpd releases for a long time
and I've been running with it since June with no side-effect

ok sunil@, jung@, millert@

pki name is a hostname not a path

whitespaces

prepare the ground for the CA certificate handling refactor, this commit
adds the parse.y bit + structures & members needed but does not make use
of it yet

Use SSL_CTX_set_ecdh_auto() instead of rolling our own version.

ok gilles@

LibreSSL now supports loading of CA certificates from memory, replace
the internal and long-serving ssl_ctx_load_verify_memory() function
with a call to the SSL_CTX_load_verify_mem() API function. The
ssl_privsep.c file with hacks for using OpenSSL in privsep'ed
processes can now go away; portable versions of smtpd and relayd
should start depending on LibreSSL or they have to carry ssl_privsep.c
in openbsd-compat to work with legacy OpenSSL. No functional change.

Based on previous discussions with gilles@ bluhm@ and many others
OK bluhm@ (as part of the libcrypto/libssl/libtls diff)

SSL_CTX_use_certificate_chain() has been added to LibreSSL and there
is no need to keep a local copy in ssl_privsep.c. This adds a little
burden on OpenSMTPD-portable because it will have to put it in
openbsd-compat for compatibility with legacy OpenSSL.

OK gilles@

Unify the SSL privsep key loading functions.

ok eric@

Deep down inside OpenSSL, err... LibreSSL, RSA_set_ex_data attempts to
free() the external data when releasing the RSA object. The
RSA_GET_EX_NEW_INDEX(3) manual page doesn't mention that this is the
default behaviour - it just describes the possible free_func()
callback - and the code path in libcrypto is hiding the fact behind
layers of abstraction.

Fix possible double free by allocating and copying the external data
reference that is used for RSA privsep (pkiname in smtpd's case).

ok eric@ gilles@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

It is only required to load the keys and certs into the same SSL
context once. Simplify the code path by moving the loading from three
different places into ssl_ctx_create():

ok gilles@

pki code cleanup

- rename "struct ssl" and "cert" to "struct pki" and "cert" to "pki_name"
- inherit pki conf on fork instead of passing it through imsg at startup
- implement SNI on smtp listeners

fix loading of passphrase-protected keys.

Much much improved config parser and related changes.
Simplify code and do not impose an order on conditions and rule options.

Format changes that may require smtpd.conf update for some setups:

- SSL certificates are no longer automatically loaded, but must be
explicitely declared using the "pki" keyword.
- "certificate" option becomes "pki" in listener and accept rules.
- "ssl://" becomes "secure://" in relay via rules.
- "helo" becomes "hostnames" in relay rules

New features:

- accept rules do not need an explicit action, in which case alias table
or .forward must provide one.
- new "forward-only" action to force relaying and reject rcpts that expand
as local delivery.
- "!" (negation) modifier on rule matching conditions.
- new "recipient" rule matching condition.
- new "verify" option on listeners and relay rules to reject invalid
certificates.

Other changes:

- remember the helo name advertised on incoming mail and use it for sending
bounces.
- bump envelope version (existing envelopes are updated on-the-fly).

tls perfect forward secrecy with ecdhe

suggested by djm@ on hackers@, diff ok djm@

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

Revert changes to use the new libtls signer api
There are bugs in the new libtls signer that can lead to a crash.
OK tb@ jsing@

use new libtls signer api

ok tb@

remove unused code

ok millert@

do not build unused code and remove uneeded dependency on libm.

ok tb@

Start porting smtpd to libtls.

Note that it changes the way SNI works: The certificate to use is now
selected by looking at the names found in the certificates themselves,
rather than the names of the pki entries in the configuration file.
The set of certificates for a tls listener must be defined explicitly by
using the pki listener option multiple times.

ok tb@

Implement server certificate validation in smtp(1).
Check certificate against MX name in smtpd(8) mta.

ok gilles@

Use automatic DH parameters, instead of fixed ones. Also disable DHE by
default since it is computationally expensive and a potential DoS vector.

ok gilles@

refactor a bit to move the SNI handling away from smtp_session into smtp

ok sunil@, jung@

remove CA from pki and no longer allow specifying a CA with 'pki' keyword.
introduce 'ca' keyword to allow specifying a custom CA.
making CA part of pki was a bad idea and several people hit use-cases that
plain couldn't work.

instead of:
pki foobar.org ca "/etc/mail/CA.pem"

use now:
ca foobar.org certificate "/etc/mail/CA.pem"


ok sunil@, jung@

allow overriding the default cipher-suite

ok jung@, sunil@, millert@

bump DH params to 2048, it's been part of smtpd releases for a long time
and I've been running with it since June with no side-effect

ok sunil@, jung@, millert@

pki name is a hostname not a path

whitespaces

prepare the ground for the CA certificate handling refactor, this commit
adds the parse.y bit + structures & members needed but does not make use
of it yet

Use SSL_CTX_set_ecdh_auto() instead of rolling our own version.

ok gilles@

LibreSSL now supports loading of CA certificates from memory, replace
the internal and long-serving ssl_ctx_load_verify_memory() function
with a call to the SSL_CTX_load_verify_mem() API function. The
ssl_privsep.c file with hacks for using OpenSSL in privsep'ed
processes can now go away; portable versions of smtpd and relayd
should start depending on LibreSSL or they have to carry ssl_privsep.c
in openbsd-compat to work with legacy OpenSSL. No functional change.

Based on previous discussions with gilles@ bluhm@ and many others
OK bluhm@ (as part of the libcrypto/libssl/libtls diff)

SSL_CTX_use_certificate_chain() has been added to LibreSSL and there
is no need to keep a local copy in ssl_privsep.c. This adds a little
burden on OpenSMTPD-portable because it will have to put it in
openbsd-compat for compatibility with legacy OpenSSL.

OK gilles@

Unify the SSL privsep key loading functions.

ok eric@

Deep down inside OpenSSL, err... LibreSSL, RSA_set_ex_data attempts to
free() the external data when releasing the RSA object. The
RSA_GET_EX_NEW_INDEX(3) manual page doesn't mention that this is the
default behaviour - it just describes the possible free_func()
callback - and the code path in libcrypto is hiding the fact behind
layers of abstraction.

Fix possible double free by allocating and copying the external data
reference that is used for RSA privsep (pkiname in smtpd's case).

ok eric@ gilles@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

It is only required to load the keys and certs into the same SSL
context once. Simplify the code path by moving the loading from three
different places into ssl_ctx_create():

ok gilles@

pki code cleanup

- rename "struct ssl" and "cert" to "struct pki" and "cert" to "pki_name"
- inherit pki conf on fork instead of passing it through imsg at startup
- implement SNI on smtp listeners

fix loading of passphrase-protected keys.

Much much improved config parser and related changes.
Simplify code and do not impose an order on conditions and rule options.

Format changes that may require smtpd.conf update for some setups:

- SSL certificates are no longer automatically loaded, but must be
explicitely declared using the "pki" keyword.
- "certificate" option becomes "pki" in listener and accept rules.
- "ssl://" becomes "secure://" in relay via rules.
- "helo" becomes "hostnames" in relay rules

New features:

- accept rules do not need an explicit action, in which case alias table
or .forward must provide one.
- new "forward-only" action to force relaying and reject rcpts that expand
as local delivery.
- "!" (negation) modifier on rule matching conditions.
- new "recipient" rule matching condition.
- new "verify" option on listeners and relay rules to reject invalid
certificates.

Other changes:

- remember the helo name advertised on incoming mail and use it for sending
bounces.
- bump envelope version (existing envelopes are updated on-the-fly).

tls perfect forward secrecy with ecdhe

suggested by djm@ on hackers@, diff ok djm@

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

use new libtls signer api

ok tb@

remove unused code

ok millert@

do not build unused code and remove uneeded dependency on libm.

ok tb@

Start porting smtpd to libtls.

Note that it changes the way SNI works: The certificate to use is now
selected by looking at the names found in the certificates themselves,
rather than the names of the pki entries in the configuration file.
The set of certificates for a tls listener must be defined explicitly by
using the pki listener option multiple times.

ok tb@

Implement server certificate validation in smtp(1).
Check certificate against MX name in smtpd(8) mta.

ok gilles@

Use automatic DH parameters, instead of fixed ones. Also disable DHE by
default since it is computationally expensive and a potential DoS vector.

ok gilles@

refactor a bit to move the SNI handling away from smtp_session into smtp

ok sunil@, jung@

remove CA from pki and no longer allow specifying a CA with 'pki' keyword.
introduce 'ca' keyword to allow specifying a custom CA.
making CA part of pki was a bad idea and several people hit use-cases that
plain couldn't work.

instead of:
pki foobar.org ca "/etc/mail/CA.pem"

use now:
ca foobar.org certificate "/etc/mail/CA.pem"


ok sunil@, jung@

allow overriding the default cipher-suite

ok jung@, sunil@, millert@

bump DH params to 2048, it's been part of smtpd releases for a long time
and I've been running with it since June with no side-effect

ok sunil@, jung@, millert@

pki name is a hostname not a path

whitespaces

prepare the ground for the CA certificate handling refactor, this commit
adds the parse.y bit + structures & members needed but does not make use
of it yet

Use SSL_CTX_set_ecdh_auto() instead of rolling our own version.

ok gilles@

LibreSSL now supports loading of CA certificates from memory, replace
the internal and long-serving ssl_ctx_load_verify_memory() function
with a call to the SSL_CTX_load_verify_mem() API function. The
ssl_privsep.c file with hacks for using OpenSSL in privsep'ed
processes can now go away; portable versions of smtpd and relayd
should start depending on LibreSSL or they have to carry ssl_privsep.c
in openbsd-compat to work with legacy OpenSSL. No functional change.

Based on previous discussions with gilles@ bluhm@ and many others
OK bluhm@ (as part of the libcrypto/libssl/libtls diff)

SSL_CTX_use_certificate_chain() has been added to LibreSSL and there
is no need to keep a local copy in ssl_privsep.c. This adds a little
burden on OpenSMTPD-portable because it will have to put it in
openbsd-compat for compatibility with legacy OpenSSL.

OK gilles@

Unify the SSL privsep key loading functions.

ok eric@

Deep down inside OpenSSL, err... LibreSSL, RSA_set_ex_data attempts to
free() the external data when releasing the RSA object. The
RSA_GET_EX_NEW_INDEX(3) manual page doesn't mention that this is the
default behaviour - it just describes the possible free_func()
callback - and the code path in libcrypto is hiding the fact behind
layers of abstraction.

Fix possible double free by allocating and copying the external data
reference that is used for RSA privsep (pkiname in smtpd's case).

ok eric@ gilles@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

It is only required to load the keys and certs into the same SSL
context once. Simplify the code path by moving the loading from three
different places into ssl_ctx_create():

ok gilles@

pki code cleanup

- rename "struct ssl" and "cert" to "struct pki" and "cert" to "pki_name"
- inherit pki conf on fork instead of passing it through imsg at startup
- implement SNI on smtp listeners

fix loading of passphrase-protected keys.

Much much improved config parser and related changes.
Simplify code and do not impose an order on conditions and rule options.

Format changes that may require smtpd.conf update for some setups:

- SSL certificates are no longer automatically loaded, but must be
explicitely declared using the "pki" keyword.
- "certificate" option becomes "pki" in listener and accept rules.
- "ssl://" becomes "secure://" in relay via rules.
- "helo" becomes "hostnames" in relay rules

New features:

- accept rules do not need an explicit action, in which case alias table
or .forward must provide one.
- new "forward-only" action to force relaying and reject rcpts that expand
as local delivery.
- "!" (negation) modifier on rule matching conditions.
- new "recipient" rule matching condition.
- new "verify" option on listeners and relay rules to reject invalid
certificates.

Other changes:

- remember the helo name advertised on incoming mail and use it for sending
bounces.
- bump envelope version (existing envelopes are updated on-the-fly).

tls perfect forward secrecy with ecdhe

suggested by djm@ on hackers@, diff ok djm@

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

remove unused code

ok millert@

do not build unused code and remove uneeded dependency on libm.

ok tb@

Start porting smtpd to libtls.

Note that it changes the way SNI works: The certificate to use is now
selected by looking at the names found in the certificates themselves,
rather than the names of the pki entries in the configuration file.
The set of certificates for a tls listener must be defined explicitly by
using the pki listener option multiple times.

ok tb@

Implement server certificate validation in smtp(1).
Check certificate against MX name in smtpd(8) mta.

ok gilles@

Use automatic DH parameters, instead of fixed ones. Also disable DHE by
default since it is computationally expensive and a potential DoS vector.

ok gilles@

refactor a bit to move the SNI handling away from smtp_session into smtp

ok sunil@, jung@

remove CA from pki and no longer allow specifying a CA with 'pki' keyword.
introduce 'ca' keyword to allow specifying a custom CA.
making CA part of pki was a bad idea and several people hit use-cases that
plain couldn't work.

instead of:
pki foobar.org ca "/etc/mail/CA.pem"

use now:
ca foobar.org certificate "/etc/mail/CA.pem"


ok sunil@, jung@

allow overriding the default cipher-suite

ok jung@, sunil@, millert@

bump DH params to 2048, it's been part of smtpd releases for a long time
and I've been running with it since June with no side-effect

ok sunil@, jung@, millert@

pki name is a hostname not a path

whitespaces

prepare the ground for the CA certificate handling refactor, this commit
adds the parse.y bit + structures & members needed but does not make use
of it yet

Use SSL_CTX_set_ecdh_auto() instead of rolling our own version.

ok gilles@

LibreSSL now supports loading of CA certificates from memory, replace
the internal and long-serving ssl_ctx_load_verify_memory() function
with a call to the SSL_CTX_load_verify_mem() API function. The
ssl_privsep.c file with hacks for using OpenSSL in privsep'ed
processes can now go away; portable versions of smtpd and relayd
should start depending on LibreSSL or they have to carry ssl_privsep.c
in openbsd-compat to work with legacy OpenSSL. No functional change.

Based on previous discussions with gilles@ bluhm@ and many others
OK bluhm@ (as part of the libcrypto/libssl/libtls diff)

SSL_CTX_use_certificate_chain() has been added to LibreSSL and there
is no need to keep a local copy in ssl_privsep.c. This adds a little
burden on OpenSMTPD-portable because it will have to put it in
openbsd-compat for compatibility with legacy OpenSSL.

OK gilles@

Unify the SSL privsep key loading functions.

ok eric@

Deep down inside OpenSSL, err... LibreSSL, RSA_set_ex_data attempts to
free() the external data when releasing the RSA object. The
RSA_GET_EX_NEW_INDEX(3) manual page doesn't mention that this is the
default behaviour - it just describes the possible free_func()
callback - and the code path in libcrypto is hiding the fact behind
layers of abstraction.

Fix possible double free by allocating and copying the external data
reference that is used for RSA privsep (pkiname in smtpd's case).

ok eric@ gilles@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

It is only required to load the keys and certs into the same SSL
context once. Simplify the code path by moving the loading from three
different places into ssl_ctx_create():

ok gilles@

pki code cleanup

- rename "struct ssl" and "cert" to "struct pki" and "cert" to "pki_name"
- inherit pki conf on fork instead of passing it through imsg at startup
- implement SNI on smtp listeners

fix loading of passphrase-protected keys.

Much much improved config parser and related changes.
Simplify code and do not impose an order on conditions and rule options.

Format changes that may require smtpd.conf update for some setups:

- SSL certificates are no longer automatically loaded, but must be
explicitely declared using the "pki" keyword.
- "certificate" option becomes "pki" in listener and accept rules.
- "ssl://" becomes "secure://" in relay via rules.
- "helo" becomes "hostnames" in relay rules

New features:

- accept rules do not need an explicit action, in which case alias table
or .forward must provide one.
- new "forward-only" action to force relaying and reject rcpts that expand
as local delivery.
- "!" (negation) modifier on rule matching conditions.
- new "recipient" rule matching condition.
- new "verify" option on listeners and relay rules to reject invalid
certificates.

Other changes:

- remember the helo name advertised on incoming mail and use it for sending
bounces.
- bump envelope version (existing envelopes are updated on-the-fly).

tls perfect forward secrecy with ecdhe

suggested by djm@ on hackers@, diff ok djm@

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

do not build unused code and remove uneeded dependency on libm.

ok tb@

Start porting smtpd to libtls.

Note that it changes the way SNI works: The certificate to use is now
selected by looking at the names found in the certificates themselves,
rather than the names of the pki entries in the configuration file.
The set of certificates for a tls listener must be defined explicitly by
using the pki listener option multiple times.

ok tb@

Implement server certificate validation in smtp(1).
Check certificate against MX name in smtpd(8) mta.

ok gilles@

Use automatic DH parameters, instead of fixed ones. Also disable DHE by
default since it is computationally expensive and a potential DoS vector.

ok gilles@

refactor a bit to move the SNI handling away from smtp_session into smtp

ok sunil@, jung@

remove CA from pki and no longer allow specifying a CA with 'pki' keyword.
introduce 'ca' keyword to allow specifying a custom CA.
making CA part of pki was a bad idea and several people hit use-cases that
plain couldn't work.

instead of:
pki foobar.org ca "/etc/mail/CA.pem"

use now:
ca foobar.org certificate "/etc/mail/CA.pem"


ok sunil@, jung@

allow overriding the default cipher-suite

ok jung@, sunil@, millert@

bump DH params to 2048, it's been part of smtpd releases for a long time
and I've been running with it since June with no side-effect

ok sunil@, jung@, millert@

pki name is a hostname not a path

whitespaces

prepare the ground for the CA certificate handling refactor, this commit
adds the parse.y bit + structures & members needed but does not make use
of it yet

Use SSL_CTX_set_ecdh_auto() instead of rolling our own version.

ok gilles@

LibreSSL now supports loading of CA certificates from memory, replace
the internal and long-serving ssl_ctx_load_verify_memory() function
with a call to the SSL_CTX_load_verify_mem() API function. The
ssl_privsep.c file with hacks for using OpenSSL in privsep'ed
processes can now go away; portable versions of smtpd and relayd
should start depending on LibreSSL or they have to carry ssl_privsep.c
in openbsd-compat to work with legacy OpenSSL. No functional change.

Based on previous discussions with gilles@ bluhm@ and many others
OK bluhm@ (as part of the libcrypto/libssl/libtls diff)

SSL_CTX_use_certificate_chain() has been added to LibreSSL and there
is no need to keep a local copy in ssl_privsep.c. This adds a little
burden on OpenSMTPD-portable because it will have to put it in
openbsd-compat for compatibility with legacy OpenSSL.

OK gilles@

Unify the SSL privsep key loading functions.

ok eric@

Deep down inside OpenSSL, err... LibreSSL, RSA_set_ex_data attempts to
free() the external data when releasing the RSA object. The
RSA_GET_EX_NEW_INDEX(3) manual page doesn't mention that this is the
default behaviour - it just describes the possible free_func()
callback - and the code path in libcrypto is hiding the fact behind
layers of abstraction.

Fix possible double free by allocating and copying the external data
reference that is used for RSA privsep (pkiname in smtpd's case).

ok eric@ gilles@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

It is only required to load the keys and certs into the same SSL
context once. Simplify the code path by moving the loading from three
different places into ssl_ctx_create():

ok gilles@

pki code cleanup

- rename "struct ssl" and "cert" to "struct pki" and "cert" to "pki_name"
- inherit pki conf on fork instead of passing it through imsg at startup
- implement SNI on smtp listeners

fix loading of passphrase-protected keys.

Much much improved config parser and related changes.
Simplify code and do not impose an order on conditions and rule options.

Format changes that may require smtpd.conf update for some setups:

- SSL certificates are no longer automatically loaded, but must be
explicitely declared using the "pki" keyword.
- "certificate" option becomes "pki" in listener and accept rules.
- "ssl://" becomes "secure://" in relay via rules.
- "helo" becomes "hostnames" in relay rules

New features:

- accept rules do not need an explicit action, in which case alias table
or .forward must provide one.
- new "forward-only" action to force relaying and reject rcpts that expand
as local delivery.
- "!" (negation) modifier on rule matching conditions.
- new "recipient" rule matching condition.
- new "verify" option on listeners and relay rules to reject invalid
certificates.

Other changes:

- remember the helo name advertised on incoming mail and use it for sending
bounces.
- bump envelope version (existing envelopes are updated on-the-fly).

tls perfect forward secrecy with ecdhe

suggested by djm@ on hackers@, diff ok djm@

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

Start porting smtpd to libtls.

Note that it changes the way SNI works: The certificate to use is now
selected by looking at the names found in the certificates themselves,
rather than the names of the pki entries in the configuration file.
The set of certificates for a tls listener must be defined explicitly by
using the pki listener option multiple times.

ok tb@

Implement server certificate validation in smtp(1).
Check certificate against MX name in smtpd(8) mta.

ok gilles@

Use automatic DH parameters, instead of fixed ones. Also disable DHE by
default since it is computationally expensive and a potential DoS vector.

ok gilles@

refactor a bit to move the SNI handling away from smtp_session into smtp

ok sunil@, jung@

remove CA from pki and no longer allow specifying a CA with 'pki' keyword.
introduce 'ca' keyword to allow specifying a custom CA.
making CA part of pki was a bad idea and several people hit use-cases that
plain couldn't work.

instead of:
pki foobar.org ca "/etc/mail/CA.pem"

use now:
ca foobar.org certificate "/etc/mail/CA.pem"


ok sunil@, jung@

allow overriding the default cipher-suite

ok jung@, sunil@, millert@

bump DH params to 2048, it's been part of smtpd releases for a long time
and I've been running with it since June with no side-effect

ok sunil@, jung@, millert@

pki name is a hostname not a path

whitespaces

prepare the ground for the CA certificate handling refactor, this commit
adds the parse.y bit + structures & members needed but does not make use
of it yet

Use SSL_CTX_set_ecdh_auto() instead of rolling our own version.

ok gilles@

LibreSSL now supports loading of CA certificates from memory, replace
the internal and long-serving ssl_ctx_load_verify_memory() function
with a call to the SSL_CTX_load_verify_mem() API function. The
ssl_privsep.c file with hacks for using OpenSSL in privsep'ed
processes can now go away; portable versions of smtpd and relayd
should start depending on LibreSSL or they have to carry ssl_privsep.c
in openbsd-compat to work with legacy OpenSSL. No functional change.

Based on previous discussions with gilles@ bluhm@ and many others
OK bluhm@ (as part of the libcrypto/libssl/libtls diff)

SSL_CTX_use_certificate_chain() has been added to LibreSSL and there
is no need to keep a local copy in ssl_privsep.c. This adds a little
burden on OpenSMTPD-portable because it will have to put it in
openbsd-compat for compatibility with legacy OpenSSL.

OK gilles@

Unify the SSL privsep key loading functions.

ok eric@

Deep down inside OpenSSL, err... LibreSSL, RSA_set_ex_data attempts to
free() the external data when releasing the RSA object. The
RSA_GET_EX_NEW_INDEX(3) manual page doesn't mention that this is the
default behaviour - it just describes the possible free_func()
callback - and the code path in libcrypto is hiding the fact behind
layers of abstraction.

Fix possible double free by allocating and copying the external data
reference that is used for RSA privsep (pkiname in smtpd's case).

ok eric@ gilles@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

It is only required to load the keys and certs into the same SSL
context once. Simplify the code path by moving the loading from three
different places into ssl_ctx_create():

ok gilles@

pki code cleanup

- rename "struct ssl" and "cert" to "struct pki" and "cert" to "pki_name"
- inherit pki conf on fork instead of passing it through imsg at startup
- implement SNI on smtp listeners

fix loading of passphrase-protected keys.

Much much improved config parser and related changes.
Simplify code and do not impose an order on conditions and rule options.

Format changes that may require smtpd.conf update for some setups:

- SSL certificates are no longer automatically loaded, but must be
explicitely declared using the "pki" keyword.
- "certificate" option becomes "pki" in listener and accept rules.
- "ssl://" becomes "secure://" in relay via rules.
- "helo" becomes "hostnames" in relay rules

New features:

- accept rules do not need an explicit action, in which case alias table
or .forward must provide one.
- new "forward-only" action to force relaying and reject rcpts that expand
as local delivery.
- "!" (negation) modifier on rule matching conditions.
- new "recipient" rule matching condition.
- new "verify" option on listeners and relay rules to reject invalid
certificates.

Other changes:

- remember the helo name advertised on incoming mail and use it for sending
bounces.
- bump envelope version (existing envelopes are updated on-the-fly).

tls perfect forward secrecy with ecdhe

suggested by djm@ on hackers@, diff ok djm@

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

Implement server certificate validation in smtp(1).
Check certificate against MX name in smtpd(8) mta.

ok gilles@

Use automatic DH parameters, instead of fixed ones. Also disable DHE by
default since it is computationally expensive and a potential DoS vector.

ok gilles@

refactor a bit to move the SNI handling away from smtp_session into smtp

ok sunil@, jung@

remove CA from pki and no longer allow specifying a CA with 'pki' keyword.
introduce 'ca' keyword to allow specifying a custom CA.
making CA part of pki was a bad idea and several people hit use-cases that
plain couldn't work.

instead of:
pki foobar.org ca "/etc/mail/CA.pem"

use now:
ca foobar.org certificate "/etc/mail/CA.pem"


ok sunil@, jung@

allow overriding the default cipher-suite

ok jung@, sunil@, millert@

bump DH params to 2048, it's been part of smtpd releases for a long time
and I've been running with it since June with no side-effect

ok sunil@, jung@, millert@

pki name is a hostname not a path

whitespaces

prepare the ground for the CA certificate handling refactor, this commit
adds the parse.y bit + structures & members needed but does not make use
of it yet

Use SSL_CTX_set_ecdh_auto() instead of rolling our own version.

ok gilles@

LibreSSL now supports loading of CA certificates from memory, replace
the internal and long-serving ssl_ctx_load_verify_memory() function
with a call to the SSL_CTX_load_verify_mem() API function. The
ssl_privsep.c file with hacks for using OpenSSL in privsep'ed
processes can now go away; portable versions of smtpd and relayd
should start depending on LibreSSL or they have to carry ssl_privsep.c
in openbsd-compat to work with legacy OpenSSL. No functional change.

Based on previous discussions with gilles@ bluhm@ and many others
OK bluhm@ (as part of the libcrypto/libssl/libtls diff)

SSL_CTX_use_certificate_chain() has been added to LibreSSL and there
is no need to keep a local copy in ssl_privsep.c. This adds a little
burden on OpenSMTPD-portable because it will have to put it in
openbsd-compat for compatibility with legacy OpenSSL.

OK gilles@

Unify the SSL privsep key loading functions.

ok eric@

Deep down inside OpenSSL, err... LibreSSL, RSA_set_ex_data attempts to
free() the external data when releasing the RSA object. The
RSA_GET_EX_NEW_INDEX(3) manual page doesn't mention that this is the
default behaviour - it just describes the possible free_func()
callback - and the code path in libcrypto is hiding the fact behind
layers of abstraction.

Fix possible double free by allocating and copying the external data
reference that is used for RSA privsep (pkiname in smtpd's case).

ok eric@ gilles@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

It is only required to load the keys and certs into the same SSL
context once. Simplify the code path by moving the loading from three
different places into ssl_ctx_create():

ok gilles@

pki code cleanup

- rename "struct ssl" and "cert" to "struct pki" and "cert" to "pki_name"
- inherit pki conf on fork instead of passing it through imsg at startup
- implement SNI on smtp listeners

fix loading of passphrase-protected keys.

Much much improved config parser and related changes.
Simplify code and do not impose an order on conditions and rule options.

Format changes that may require smtpd.conf update for some setups:

- SSL certificates are no longer automatically loaded, but must be
explicitely declared using the "pki" keyword.
- "certificate" option becomes "pki" in listener and accept rules.
- "ssl://" becomes "secure://" in relay via rules.
- "helo" becomes "hostnames" in relay rules

New features:

- accept rules do not need an explicit action, in which case alias table
or .forward must provide one.
- new "forward-only" action to force relaying and reject rcpts that expand
as local delivery.
- "!" (negation) modifier on rule matching conditions.
- new "recipient" rule matching condition.
- new "verify" option on listeners and relay rules to reject invalid
certificates.

Other changes:

- remember the helo name advertised on incoming mail and use it for sending
bounces.
- bump envelope version (existing envelopes are updated on-the-fly).

tls perfect forward secrecy with ecdhe

suggested by djm@ on hackers@, diff ok djm@

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

Use automatic DH parameters, instead of fixed ones. Also disable DHE by
default since it is computationally expensive and a potential DoS vector.

ok gilles@

refactor a bit to move the SNI handling away from smtp_session into smtp

ok sunil@, jung@

remove CA from pki and no longer allow specifying a CA with 'pki' keyword.
introduce 'ca' keyword to allow specifying a custom CA.
making CA part of pki was a bad idea and several people hit use-cases that
plain couldn't work.

instead of:
pki foobar.org ca "/etc/mail/CA.pem"

use now:
ca foobar.org certificate "/etc/mail/CA.pem"


ok sunil@, jung@

allow overriding the default cipher-suite

ok jung@, sunil@, millert@

bump DH params to 2048, it's been part of smtpd releases for a long time
and I've been running with it since June with no side-effect

ok sunil@, jung@, millert@

pki name is a hostname not a path

whitespaces

prepare the ground for the CA certificate handling refactor, this commit
adds the parse.y bit + structures & members needed but does not make use
of it yet

Use SSL_CTX_set_ecdh_auto() instead of rolling our own version.

ok gilles@

LibreSSL now supports loading of CA certificates from memory, replace
the internal and long-serving ssl_ctx_load_verify_memory() function
with a call to the SSL_CTX_load_verify_mem() API function. The
ssl_privsep.c file with hacks for using OpenSSL in privsep'ed
processes can now go away; portable versions of smtpd and relayd
should start depending on LibreSSL or they have to carry ssl_privsep.c
in openbsd-compat to work with legacy OpenSSL. No functional change.

Based on previous discussions with gilles@ bluhm@ and many others
OK bluhm@ (as part of the libcrypto/libssl/libtls diff)

SSL_CTX_use_certificate_chain() has been added to LibreSSL and there
is no need to keep a local copy in ssl_privsep.c. This adds a little
burden on OpenSMTPD-portable because it will have to put it in
openbsd-compat for compatibility with legacy OpenSSL.

OK gilles@

Unify the SSL privsep key loading functions.

ok eric@

Deep down inside OpenSSL, err... LibreSSL, RSA_set_ex_data attempts to
free() the external data when releasing the RSA object. The
RSA_GET_EX_NEW_INDEX(3) manual page doesn't mention that this is the
default behaviour - it just describes the possible free_func()
callback - and the code path in libcrypto is hiding the fact behind
layers of abstraction.

Fix possible double free by allocating and copying the external data
reference that is used for RSA privsep (pkiname in smtpd's case).

ok eric@ gilles@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

It is only required to load the keys and certs into the same SSL
context once. Simplify the code path by moving the loading from three
different places into ssl_ctx_create():

ok gilles@

pki code cleanup

- rename "struct ssl" and "cert" to "struct pki" and "cert" to "pki_name"
- inherit pki conf on fork instead of passing it through imsg at startup
- implement SNI on smtp listeners

fix loading of passphrase-protected keys.

Much much improved config parser and related changes.
Simplify code and do not impose an order on conditions and rule options.

Format changes that may require smtpd.conf update for some setups:

- SSL certificates are no longer automatically loaded, but must be
explicitely declared using the "pki" keyword.
- "certificate" option becomes "pki" in listener and accept rules.
- "ssl://" becomes "secure://" in relay via rules.
- "helo" becomes "hostnames" in relay rules

New features:

- accept rules do not need an explicit action, in which case alias table
or .forward must provide one.
- new "forward-only" action to force relaying and reject rcpts that expand
as local delivery.
- "!" (negation) modifier on rule matching conditions.
- new "recipient" rule matching condition.
- new "verify" option on listeners and relay rules to reject invalid
certificates.

Other changes:

- remember the helo name advertised on incoming mail and use it for sending
bounces.
- bump envelope version (existing envelopes are updated on-the-fly).

tls perfect forward secrecy with ecdhe

suggested by djm@ on hackers@, diff ok djm@

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@