remove prototypes with no matching function
leave prototypes with functions in OpenSMTPD-extras
ok op@

actually honour the services supported by the proc tables

ok gilles@

change the smtpd table protocol

Using imsg for the "proc" table (external programs) has proven quite
painful in practice since a lot of smtpd internals (structs, enums,
etc..) have to be kept in sync with the various tables implementations.

Instead, a filter-like protocol for tables decouples the implementations
and allows to write and test tables easily.

The new text-based transport protocol is documented in the (added)
smtpd-tables(7) manpage.

The old imsg protocol is no longer supported and existing tables have to
be converted. In particular, users of opensmtpd-extras tables will need
install the new opensmtpd-table-* packages.

With lots of suggestions and improvements from gilles and a tweak
from Philipp (philipp+openbsd [at] bureaucracy [dot] de), thanks!

ok gilles

bump version to 7.5.0

unify smtpd and makemap table parser

These are supposed to parse the same file format but have subtle
difference in the handling of comments, continuation lines and escaping.

Converge both to the simpler smtpd parser which doesn't handle
continuation lines nor escaping, and support comments only at the start
of the line.

improvements and ok millert@

there's no good reason to allow smtpd to execute custom command set by root
in a .forward file so disallow custom commands and file reading, only allow
setting forward addresses and users.

as root is no longer allowed to run any MDA but mbox, we can be stricter on
the setup of the MDA process and refuse to exec anything that's not an mbox
dispatcher.

tested by op@ who edited a root envelope to simulate an exploit injecting a
custom command in a root envelope, smtpd refused to exec.

ok millert@ and op@

relax ORCPT syntax validation

We expected the ORCPT parameter to be a valid rfc822 address. This is
wrong on multiple levels:

- any other IANA-registered "addr-type" can be used
- the parameter may be encoded and we didn't decode it prior validation
- RFC3461 explicitly states that "[..] the address associated with the
ORCPT keyword is NOT constrained to conform to the syntax rules for
that 'addr-type'".

Instead, just validate the xtext and preserve the ORCPT value as-is.

Issue originally reported by Tim Kuijsten, Tassilo Philipp and others.

ok millert@

RFC 7505 ("Null MX") handling

mail delivery will not be attempted if a domain advertises a single MX
record with preference 0 and a zero-length label.

based on an initial diff from Philipp (philipp+openbsd [at] bureaucracy
[dot] de), thanks!

ok jung@

bump version to 7.4.0

bump version to 7.3.0

add missing include of time.h

spotted after a report on OpenSMTPD-portable. While here include
sys/time.h in smtpd.h, as noted in event_init(3), since it includes
event.h.

ok millert@

remove two unused defines

last PROC_COUNT use was removed with the switch to fork+exec by eric@ in
2016, CA_FILE with the removal of cert.c two years ago.

ok tb@, kn@

Revert changes to use the new libtls signer api
There are bugs in the new libtls signer that can lead to a crash.
OK tb@ jsing@

use new libtls signer api

ok tb@

Do not verify the cert or CA for a relay using opportunistic TLS.
If a relay is not explicitly configured to use TLS but the remote
side supports STARTTLS, we will try to use it. However, in this
case we should not verify the cert or CA (which may be self-signed).
This restores the relay behavior before the switch to libtls was made.
There is no change if the relay is explicitly configured to use TLS.
OK eric@

bump version to 7.0.0

add format attribute to vaararg functions.
millert@ thinks its useful.

add required headers for smtpd.h and remove unnecessary ones in other files.

ok jung@

unplug unused certificate verification code, now that this is done by libtls.

ok tb@ millert@

do not build unused code and remove uneeded dependency on libm.

ok tb@

bump smtpd version

allow to specify tls ciphers and protocols on listeners

ok tb@

turn log_trace() into a macro to prevent evaluating the format string
parameters when tracing is not enabled.

ok millert@

allow to specify tls protocols and ciphers on relay actions

ok espie@ sthen@ tb@

Start porting smtpd to libtls.

Note that it changes the way SNI works: The certificate to use is now
selected by looking at the names found in the certificates themselves,
rather than the names of the pki entries in the configuration file.
The set of certificates for a tls listener must be defined explicitly by
using the pki listener option multiple times.

ok tb@

Do the KAME embedded scope fixup in the two places where getifaddrs() is
used. With this there should be no more embedded scopes left and therefor
in6addr_to_text() can be removed. getnameinfo() will just do the right
thing now.
OK eric@

Rename the pony process to dispatcher and klondike to crypto.

From gilles@
OK millert@ giovanni@

Revert agentx support for now, we're too close to release.

requested by deraadt@

Add support for agentx to smtpd.

This is based around NETWORK-SERVICES-MIB from RFC2788 and MTA-MIB from
RFC2789, but does not export the full spec. Hopefully this will expand in
the future.

People who want to use this against net-snmp (currently the only option
known to me at the time of writing) may want to add -I -mta_sendmail to the
flags, so net-snmp doesn't throw garbage into the mib-2.28 subtree.

Add the admd keyword. This can be used by filters interested in the
Authentication-Results header.

OK giovanni@

bump smtpd version

Bump version to 6.6.4 for errata and to match -portable.

Fix two security vulnerabilities discovered by Qualys.
An out of bounds read in smtpd allows an attacker to inject arbitrary
commands into the envelope file which are then executed as root.
Separately, missing privilege revocation in smtpctl allows arbitrary
commands to be run with the _smtpq group.

now that mail.local(8) relies on lockspool(1) for mailbox locking, have the
mailbox created by smtpd for mbox before privileges are dropped then we can
call mail.local(8) with the recipient privileges.

ok millert@

introduce mda_mbox() to handle mbox delivery in its own code path, and make
it use execle() since we know all parameters and don't need command line to
be parsed.

ok millert@ and jung@

Bump smtpd version after recent changes

ok gilles@

allow using the session username in builtin filters when available

add FILTER_SUBSYSTEM_SMTP_OUT to filter_subsystem enum and add filter name
to struct dispatcher_remote, this will reduce the smtp-out reporting diff

do not pass rdns, fcrdns, ss_src and ss_dest with IMSG_FILTER_SMTP_BEGIN,
but gather the information from the link-connect reporting event instead.
this removes redundant code and makes it easier to prepare for smtp-out.

give a better name to a couple functions and struct fields related to
filters, no functional change

teach relay action how to do domain-based relay host, this allows declaring
a single relay action with a mapping of relay hosts per domain.

ok eric@

introduce a bypass keyword so that builtin filters can bypass processing of
a phase when a condition is met

suggested by several people including jung@, ok jung@

filter protocol has an initial handshake within which smtpd tells filters
about a few global configuration informations. this makes smtpd tell proc
filters for which subsystem they are registered allowing them to register
only events that are relevant.

store smtp session username in envelope and allow ruleset to match specific
users or mailaddr:

match auth "gilles@openbsd.org" [...]
match auth "@openbsd.org" [...]

ok eric@

6.6.0 -> 6.6.1

branches: 1.641.2;
Allow maildir and mbox MDAs to tempfail on situations that might be
resolved over time.

While here remove mkdirs component from utils.c, which isn't used anywhere.

OK gilles@, millert@

SRS uses base64 encoding for the checksum, however while this is ok when we
only have MTA in the loop, some implementations like Dovecot's LMTP dislike
finding '/' in an e-mail address. Since checksum is meant to be verified at
the MX that generated the SRS encoding, use alternate rfc354 base64 encode,
swapping '/' with '_' and '+' with '-'.

ok eric@ millert@

teach smtpd how to do SRS so hosts that act as forwarders don't break SPF.
this basic implementation does SRS0/SRS1 encoding/decoding, validating time
and checksums.

with insight from semarie@, ok eric@ and millert@

unescape / and ^ in the general delivery case, they only need to be for
maildir

ok eric@

Implement server certificate validation in smtp(1).
Check certificate against MX name in smtpd(8) mta.

ok gilles@

Sprinkle around some __attribute__((__format__ (printf(...))).

OK gilles@

Add support for filter-reports. These allow filters to send freetext
reports to other filters/report handlers.
Builtin filters work via the new "report" keyword.
Proc filters can send reports via:
"report|<seconds.<microseconds>|smtp-in|<reqid>|<message>"
Subscribing to these reports can be done via the the "filter-report"
keyword.
The reports themselves contain the usual elements followed by:
<type>|<name>|<message>
Type can be builtin or proc.
Name is the process name for type proc and the filter name for type builtin.

OK gilles@

introduce the 'junk' builtin filter action which marks a session or
transaction as junked when a filter matches. this with the maildir
junk option allows classifying messages in Spam folder instead of
rejecting/disconnecting.

ok semarie@, eric@, martijn@

Add a link-greeting report. This allows us to get the active domain name
in use for the current request.

OK gilles@

res_hnok() is too lenient wrt to acceptable domain name in mail addresses.
replace it with a valid_domainname() check that implements something closer
to RFC 5321, but still usable in real-life.

ok gilles@ millert@

this introduces experimental proxy v2 support which is fairly isolated to a
single proxy.c file, importing it to work in tree

initial work from Antoine Kaufmann <toni@famkaufmann.info>

bump version

even though RSET can be issued outside a tx, RFC states it's noop outside a
transaction so rename link-reset to tx-reset and only issue the smtp report
when a reset _actually_ has a side-effect.

note that rset is implicit on a message commit or rollback, so tx-reset get
issued even though there was no explicit RSET. the filters are MUCH simpler
to write when you don't need to track every event that can reset a tx :-)

introduce link-auth to the smtp reporting stream so that filters may know
if a link has been authenticated successfully or not and for which user

modify link-identify so it reports if HELO or EHLO was used

introduce link-reset to let smtpd report resets happening in a session

Allow filters to log information through stderr. This simplifies and
unifies the way filters need to get their logging to the right location.

Log-messages are read line by line and are logged at LOG_ERR level via
the lookup process.

OK gilles@

simplify the runq interface:

- remove (unused) per-job callback
- rename runq_schedule() to runq_schedule_at() and runq_delay()
to runq_schedule()
- remove unused runq_next()

ok sunil@ gilles@

extend the resolver interface to delegate res_query() calls to the lka.

ok gilles@ sunil@

a long long time ago, in a galaxy quite close actually, reyk@ introduced an
RSA privsep engine to isolate private keys in the ca process. ECDSA support
in smtpd is become a frequent request so here's an ECDSA privsep engine and
the code required for smtpd to load ECDSA certificates and use them.

branches: 1.621.2;
remove unused declarations

ok gilles@

bump smtpd version

on behalf of gilles@

revert previous commit, i wasn't happy with it and it probably came from a
misunderstanding.

don't be too strict with .forward permissions, it's ok to process it if the
group has write access, it's not ok if the world has write access.

ok eric@

revert this change, it was committed by accident

introduce smtp 'timeout' reporting event to notify filters that a timeout
occured during the smtp session

introduce table_dump() and tweak format

ok gilles@

remove the tag workaround for table_create() and table_find(),
now that static tables handle their updates internally.

ok gilles@

remove unused members in struct table

extract subaddress from last resolved node, not from dest or rcpt address
which was incorrect and can lead to ambiguous cases, this will affect the
people who were using subaddresses within aliases themselves AND expected
deliveries to a maildir subdir of the recipient user.

ok eric@

introduce dump() and add() table methods, only implemented for static tables.

ok gilles@

pass the table pointer to the lookup()/fecth() methods

ok gilles@

change the close() method to take the table pointer

ok gilles

Make the backend open method return an int to report success.
The implementation is responsible for setting the handle pointer
as needed.

ok gilles@

reorder parameters for consistency

introduce a table_match() function to check for a key in a table

ok gilles@

get rid of the unused dict argument in table lookup and fetch api.

ok gilles@

move the table backend name in the backend struct.
remove unused function.

ok gilles@

remove dead code

ok gilles@

Simplify the table backend interface: lookup results are returned
as strings, and parsing is handled by the upper layer.

ok gilles@

introduce 'rcpt-to' builtin filter, can only be used on 'rcpt-to' hook

introduce 'mail-from' builtin filter, can be applied on the 'mail-from',
'rcpt-to', 'data' and 'commit' phases.

introduce 'helo' builtin filter, can be used on any hook but 'connect'

introduce new matching criteria 'from rdns' to match sessions based on rDNS
of the client, works with literal and tables, both string and regex:

match from rdns "mx1.poolp.org" for any action blahblah

since we already support regex lookups in tables for builtin filters, let's
also support regex lookups in match rule criterias performing table lookups

ok millert@

implement some additional builtin filters:
check-src-{table,regex}, check-rdns-{table,regex}

make sure that these builtins may be used at all phases

bring in new grammar for filters, allowing filter chains and plugging of
different filters & chains on different interfaces.

in this diff, proc filters are still disabled as they're missing on very
important piece of logic.

ok eric@

in event reports, use a struct timeval instead of time_t since we want more
than second precision

discussed with eric@

remove unused prototypes

add tx-data reporting event

do some imsg renaming to make them more clear

remove unused imsg names

factor smtp-in and smtp-out reporting code

report filter responses to smtp

generate an event when a helo name identifies a link

Improve the cert_*() interface. Use the return value to tell whether
the request is pending (waiting for an async event) or not. Success
or failure is always reported through the callback function.

ok gilles@

add check-fcrdns builtin filter

ok eric@

no longer pass rdns in all filtering requests, they can be retrieved from
the filter session.

add client and listener address, as well as client rDNS and FCrDNS lookup
result to the filter_session structure upon filter session allocation. it
will allow me to simplify all filter hooks.

Use correct RFC 3464 specified values for Action field in a DSN.
error -> failed
success -> delivered

This fixes DSN parsing for Mailman. Issue reported by Cristiano
Costa on misc@opensmtpd.org.

While here, rename enums to reflect the intent and properly handle
envelope ascii load/dump to understand change in the values.

Suggestions and ok gilles@

Refactor certificate initialization and verification.
Factorize code duplicated in smtp_session.c and mta_session.c
Implement a simple callback interface, with proper request management
and simplified imsg protocol.

Only add the necessary parts for now.
Exisiting code path will be adapted later.

input from gilles@ sunil@
ok gilles@

link-connect event report had an empty fcrdns field, but now that eric@ has
plugged fcrdns in the smtp_session we can fill the field with a value

introduce tx-mail and tx-rcpt report events

allow passing data lines to proc filters

ok eric@

in mda variables expansions, do not consider empty strings as errors since
an empty %{sender} is really a mailer-daemon and not an error

reported and initial diff by Lauri Tirkkonen <lotheac@iki.fi>
commit is a revised version of the diff based on a discussion with eric@

bring the first bits of DATA filtering plumbing but bypass it for now

ok eric@

prepare for smtp-out reporting and while at it, make a few changes to the
report format

introduce FILTER_COMMIT which will allow taking a decision at DATA commit
time, unusable yet but necessary for the upcoming serie of diffs.

ok eric@

when reporting tx events, report tx id
when reporting tx commit, report data size
report tx-envelope events

only apply filter rules to filtered interfaces

check-rdns builtin filter, to be improved

bring plumbing for proc filters

ok millert@, eric@, jung@

bring plumbing for builtin filters

ok millert@, eric@, jung@

report rDNS in link connect event

pass struct sockaddr_storage instead of ss_to_text() in reporting

teach smtp process how to report smtp events to lka and teach lka how to
report these events to a proc

ok millert@

introduce K_REGEX table type and table_regex_match(), unused for now

ok eric@

allow smtpd to fork processes at startup and maintain a socketpair with
them.

ok jung@, eric@

add helper valid_smtp_response() to be used in upcoming commits

Allow to use the "tls" keyword on any relay action to force TLS, with
strict certificate validation. The "no-verify" becomes optional.

ok gilles@ millert@ semarie@

rename the ill-named "flags" member to "as_host" in domain structure.
remove yet another useless relay flag while there.

ok gilles@

simplify code path for backup relay and remove useless flag

ok gilles@

use symbolic integer values for the different tls options when relaying,
rather than a confusing set of flags.

ok gilles@

upon mda failure, smtpd would assume tempfail and retry. this is at odds
with the other MTA which assume a permfail unless the exit status is one
of a specific set. make smtpd honour the same exit statuses as postfix.

note that all errors that occur before the user mda is executed (fork, pipe
and related) are still considered tempfail, only errors coming from the mda
itself are handled as permfail.

this commit is a temporary solution as i believe the SIGCHLD handler is way
more complex than it should be and we'll simplify it after 6.4 is out.

ok eric@

switch to improved incoming message parser:
- simpler interface not using callbacks
- no hard-coded line length
- avoid unnecessary string copy

ok gilles@

Implement a generic interface to forward resolver queries to the lka
process. Use it for the reverse lookups required by smtp and mta.

Until now, DNS-related lookups were implemented using ad-hoc IMSGs
between the lka and other processes. It turns out to be confusing and
difficult to maintain/extend. So we want to replace this with a better
set of IMSGs matching the standard resolver interface.

ok gilles@

simplify parse_config() further so it no longer has any side effect outside
of parse.y, there's still work to be done but it's now able to run twice if
we want (we don't) without failing due to some global side-effect.

ok millert@

split smtp_accept() in two parts: the accept part, the session init part,
while at it allow smtp_session() to receive a pre-allocated struct io

ok millert@
diff contributed by Antoine Kaufmann

rework the table API so that it takes a struct smtpd * context in parameter
of functions creating, looking up or destroying tables.

this is a first step in cleaning up parse.y so it doesn't have side effects
outside of parse_config(), bringing nothing but making code cleaner.

ok millert@

remove unused flags and obsolete comments

ok gilles@

remove fields that are found in struct dispatcher from struct relayhost

ok gilles@

remove struct relayhost from struct envelope.

ok gilles@

add support for mda wrappers allowing postmaster to define command wrappers
that will be executed (with recipient privileges) before calling the users'
mail delivery agent

ok eric@

split forkmda() in two:
- forkmda() creates the process that will be used for the delivery and does
the switching of privileges then calls mda_unpriv()
- mda_unpriv() runs with privileges of the recipient, it expands variables,
sets up environment and executes the mda

ok millert@ and eric@

Require a valid certificate by default when relaying through a smarthost.
Add "tls no-verify" relay option to disable it.

suggested and initial diff by semarie@.

ok gilles@

remove 'where' parameter from all x*() functions in utils.c, it doesn't
really help us with anything, propagate the change in codebase

ok millert@

remove unused function

ok gilles@

no need to parse and dump the relayhost in the lookup process.

ok gilles@

provide mail user agents with the same environments as Postfix

ok millert@, eric@

bump version, this will be a big release when OpenBSD 6.4 is released :-)

switch smtpd to new grammar

ok eric@

kill corrupt / uncorrupt queue mechanism as it has never been usable and it
will be made irrelevant when the new config comes up soon

ok eric@

sync log.h with other daemons

ok gilles@

bump minor version just to be sure it makes release :-)

ok gilles@

we haven't updated the version in a while despite many commits which is
confusing for people running the portable version

remove more filter-related cruft

ok gilles@

bypass the filter code for incoming smtp sessions.
experimental support for filters has been removed from the config
parser already, and we want to get rid of the remaining code.

ok gilles@

the PURGE_EVERYTHING flag used to purge config bits was inaccurate

ok eric@

smtpctl(8): Use an int to determine mode instead of __progname.

Ok millert@ gilles@

move variables expansion out of lka_session into their own file, this is a
mechanical diff to simplify a bit the lka code and prepare for moving
variables outside of the lookup process into the chrooted mda process.
no functional change for now.

ok eric@

- filters are currently broken, do not allow using them until we're done

allow negation of authenticated keyword:
accept ! authenticated [...]

ok sunil@, jung@

Stop assuming that in_{addr,port}_t are typedefed in <sys/types.h> and
instead pull in <netinet/in.h> or <arpa/inet.h> when those are needed.

ok florian@ beck@ millert@

smtpd joins the 7 other daemons that share the same log.c file.

The only major difference was the "log_trace" concept that is only
used by smtpd - move it from log.c into util.c and make it a local
concept. This also needed to rename the global "verbose" variable to
"tracing" in a few places.

OK krw@ gilles@ eric@

remove unused iobuf helpers

make struct io opaque:

- move struct io definition to ioev.c
- replace io_init/io_clear with io_new/io_free
- allocate an iobuf for each new io internally
- use struct io pointer in the rest of the code
- remove remaining uses of iobuf_*

ok gilles@ sunil@

assign an id to each rule in the ruleset, first step towards an MTA layer
and scheduler simplification

ok eric@

Add io api functions for dealing with buffered data, as wrapper around
their iobuf counterparts.

ok gilles@ sunil@

Remove the "smtpctl stop" command.
The daemon is stopped with kill(1).

ok gilles@

get rid of the type-checking system on internal messages.
bump all imsg protocol versions since message format changed.

ok gilles@ sunil@

remove noop function

ok sunil@

get rid of the imsg buffer usage profiling code.

ok gilles@ jung@ sunil@

Remove dead code. queue_flow_control() has never been used and is
probably a bad idea.

ok gilles@

introduce "authenticated" parameter so rules may apply to authenticated
sessions specifically

ok eric@, sunil@, jung@

allow overriding the subaddressing delimiter with subaddressing-delimiter
keyword, the default is still +

ok eric@, sunil@

bump version

ok deraadt@

Implement the fork+exec pattern in smtpd.

The parent process forks child processes and re-exec each of them with
an additional "-x <proc>" argument. During the early setup phase, the
parent process sends ipc socket pairs to interconnect the child
processes as needed, and it passes the queue encryption key to the
queue if necessary. When this is done, all processes have their
environment set as in the fork-only case, and they can start doing
their work as before.

ok gilles@ jung@

Nuke session_socket_blockmode() and session_socket_linger(). Use
the identical io_set_blocking() and io_set_linger().

Since both are always called to turn off blocking or lingering,
nuke the parameter and associated enum in favour of "just doing the
right thing".

While passing remove the unneeded last parameter to the remaining
fcntl(F_GETFL).

Finally, rename the functions to io_set_nonblocking() and
io_set_nolinger() for clarity.

No functional change.

Started with a sweep of fcntl() usage inspired by guenther@.

ok gilles@

bump version

handle enqueuer socket as a regular listener that can be configured with
"listen on socket". this simplifies a bit of code, removes some special
cases and will allow attaching filters & masking source just as on lo0.

diff from Peter Bisroev <peter@int19h.net>
ok gilles@, jung@

remove no longer relevant ifndef

suggested by gilles

Check imsg data length before use.

Ok jung@ gilles@ eric@

switch to /usr/local/libexec when looking for -extras and drop loop iterating
paths

this effectively reverts table.c r1.21 which was mainly introduced for a smooth
transition in -current

ok gilles

smtpd is no longer 5.4.6

refactor a bit to move the SNI handling away from smtp_session into smtp

ok sunil@, jung@

allow overriding the default cipher-suite

ok jung@, sunil@, millert@

add bits so local enqueuer can run filters when they are enabled

expose foreground_log in smtpd.h so filters can inherit it

sender and recipient are mail addresses, not pathnames, use proper define

use smtpd specific define for table name sizes

pki name and ca name must match a hostname, use HOST_NAME_MAX

do not limit usernames to LOGIN_NAME_MAX in places where "virtual users"
may be used, in such places an email address may be specified.

rename an smtpd specific define

prepare some imsg structures for upcoming diff to support wildcard ca

rename field member + whitespaces

add filter.c prototypes, unused for now

prepare smtpd.h for masquerading

Merge makemap(8) into smtpctl(8).

Ok gilles@, jung@

introduce limit session keyword replacing fixed values

original diff from Renaud Allard

ok gilles

add received-auth parameter to listener to identify authenticated sessions
in locally appended Received header when enabled

ok millert@, jung@

add IMSG_SMTP_CHECK_SENDER in preparation for another diff

prepare the ground for the CA certificate handling refactor, this commit
adds the parse.y bit + structures & members needed but does not make use
of it yet

add ca_name field to some structures in preparation for a larger
diff to refactor handling of CA certificates

when looking up tables, start in /usr/local/libexec before /usr/libexec, so
ports/packages can be installed in the proper place

ok jung@

mechanical rename of some IMSG constants

ok sunil@, ok jung@

While delivering to lmtp or mda, accept optional "as user" parameter
whose privileges would be used instead of the default.

Ok gilles@ jung@

allow table API to lookup for mailaddr mappings

ok sunil@, ok jung@

Implement smtpctl uncorrupt <msgid>

"uncorrupt" moves envelopes from corrupt bucket back to the queue
for further discovery by the daemon.

After correcting the corrupt envelopes, admin could now...

# smtpctl uncorrupt msgid
# smtpctl discover msgid

to schedule the messages.

Ok gilles@

Implement smtpctl discover <evpid|msgid>.

discover subcommand schedules envelopes manually moved to the queue.
It triggers a queue walk searching for envelopes with the given id,
schedules them and informs the user number of envelopes scheduled.
Admins no longer would need to restart the daemon to discover
manually moved messages.

Ok gilles@

masquerade and senders map require being able to lookup mailaddr structures
in tables, prepare for such features by bringing the helpers + smtpd.h part

ok millert@

aliases support resolving to maildir:/path

ok sunil@ millert@

aliases_virtual_check() has been unneeded for a while

ok jung@, ok sunil@, ok millert@

Only enable SSL_VERIFY_PEER when the verify option is set on a listener.

Always enabling SSL_VERIFY_PEER unnecessarily increases the number of
messages/bytes in the TLS handshake and increases our attack surface,
since we request and then process client certificates.

ok gilles@

mailaddr_match() allows comparing two struct mailaddr taking into account
catchall and +-tags

ok millert@ and jung@ for util.c

smtpd in tree is no longer neither 5.4.4, nor 5.4.5, bump SMTPD_VERSION

turn our local enqueuer setgid _smtpq and restrict access to offline queue,
the enqueuer will revoke group and regain real gid right after mkstemp.

this would have prevented the symlink/hardlink attacks against offline, and
it will avoid having to deal with new ways users can mess with it.

ok eric@, ok millert@

when bypassing the enqueuer, insert Message-Id header if none was found and
the client has connected from a loopback interface.

ok millert@ eric@

Incorrect logic in smtpd(8) can lead to unexpected client disconnect, invalid
certificate in SNI negotiation or server crash.

spotted by Edwin Torok

branches: 1.473.2;
use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

bump version

these are no longer used, remove

Convert the logic in yyerror(). Instead of creating a temporary
format string, create a temporary message.
OK doug@

when From, To and Cc headers present users without domains, append the
listener hostname to avoid smtpd relaying a header that will be rewritten
by the destination MX.

ok eric@

branches: 1.468.4;
Improve the scheduler, better and simpler.

- Get rid of the scheduler_batch structure. The scheduler can now return
envelopes of different types in a single run, interlacing them to avoid
batch effects.

- Ask for an acknowledgement from the queue when removing or expiring
an envelope to benefit from the inflight envelope limitation mechanism.
This ensures that the scheduler always keeps sending envelopes at a rate
that the queue can sustain in all cases.

- Limit the number of envelopes in a holdq. When a holdq is full,
new envelopes are put back in the pending queue instead, with a
shorter retry time.

- Plumbing for proc-ified schedulers.

imsg version bump. smtpctl stop before updating.

ok gilles@

add a "no-dsn" listener option to disable DSN extension.

config parser improvements:

- fail if the same option is specified multiple times on a listener
- prompt for queue encryption key after config parsing, not during.
- add ip addresses to localnames table
- prepare for filters

update filter configuration parsing (not plugged yet)

get rid of mfa leftovers

various queue improvements:

- add a "close" hook to the backend API.
- improve the sync() pattern in queue_fs: only sync at commit
time and not for every envelope creation
- various fixes to the experimental external queue API.

Update the table API: lookup functions can take an optional parameters
dictionnary (currently not set). While there, add a helper for forking
external backends, and remove unused table functions.

ok gilles@

Create a new default RSA engine instead of patching the existing one
if none is available. Fixes SSL/TLS and a possible fatalx() on
machines without a default RSA engine.

Thanks to Bjorn Ketelaars for reporting and testing.

ok gilles@ (for the relayd part)

Move RSA keys from "lka" to a new dedicated "ca" process because lka
is handling some async requests and shouldn't be busy with sync RSA.

ok gilles@

when using maildir, do not create automatically create folders to match tag
in email address (ie: gilles+tag => ~/Maildir/.tag), instead use the folder
if it already exists and deliver to the mail Maildir otherwise.

ok eric@ and chl@

The RSA engine (used by pony) has to wait for a response from the
privileged process (lka) and receive the imsgs in a while loop
synchronously. But the lka also sends other imsgs (DNS etc.) that can
still be queued up in the buffer when waiting for the RSA response.
This only happens under load with many concurrent connections. For
now, we just call the pony imsg handler for non-RSA imsgs that are
already in the buffer.

ok gilles@ eric@ blambert@

Implement RSA privilege separation for OpenSMTPD, based on my previous
implementation for relayd(8). The smtpd(8) pony processes (mta
client, smtp server) don't keep the private keys in memory but send
their private key operations as imsgs to the "lookup"/mta process.
It's worth mentioning that this prevents acidental private key leakage
as it could have been caused by "Heartbleed".

ok gilles@

Remove unused arguments from ssl_smtp_init()

ok gilles@

certs are looked up by hostname, the size of the buffer should use the
max hostname len, not max pathname len as before

Zap the mfa process. It is not currently doing anything, and content filtering
will be done at session level anyway.

ok gilles@

remove useless define for banner

ok gilles@

Merge the mda, mta and smtp processes into a single unprivileged
process managing message reception, delivery and transfer. Mostly
mechanical, but very intrusive as it required to rewamp all IMSG to
fix ambiguities.

with and ok gilles@

disable the imsg buffers profiling code unless requested, this will prevent
all processes from waking up every second

branches: 1.450.2;
new "smtpctl show status" command to show if mta/mda/smtp are currently running or paused.

tweak usage() and bump version.

Add support for DSN and Enhanced Status Code

Allow the admin to pause relaying to a specific domain:
- smtpctl pause mta from <source> for <domain>
- smtpctl resume mta from <source> for <domain>
- smtpctl show mta paused

internal improvements and cleanups

- get rid of the whole penalty thing for failed envelopes in the mta and scheduler.
- do not disable routes on smtp errors
- try to schedule all types of envelopes on each scheduler frame.

pki code cleanup

- rename "struct ssl" and "cert" to "struct pki" and "cert" to "pki_name"
- inherit pki conf on fork instead of passing it through imsg at startup
- implement SNI on smtp listeners

extend allowed charset for email address, escape all potentially dangerous ones.

add base64_encode/base64_decode helpers

get rid of fdlimit()

now at 5.4.1

move defines around

Rework the envelope flushing loops in mta to avoid sending all delivery
notifications in one go to the queue. Simplify code in the process.

do not hardcode scheduler batch size, and reduce default limit to avoid
hammering effects.

limit the number of envelopes to recall in the hoststat cache.

Rework the mda and scheduler to use the holdq mechanism instead of
tempfail for limiting the number of pending deliveries to the same
user. This allows to reach optimal delivery time even in case of
burst, while keeping the number of inflight envelopes low.

Add a limit on the number of inflight envelopes. The scheduler suspends
scheduling of mta/mda envelopes until the number of inflight envelopes
falls below that line.

Allow overriding the local ca

Much much improved config parser and related changes.
Simplify code and do not impose an order on conditions and rule options.

Format changes that may require smtpd.conf update for some setups:

- SSL certificates are no longer automatically loaded, but must be
explicitely declared using the "pki" keyword.
- "certificate" option becomes "pki" in listener and accept rules.
- "ssl://" becomes "secure://" in relay via rules.
- "helo" becomes "hostnames" in relay rules

New features:

- accept rules do not need an explicit action, in which case alias table
or .forward must provide one.
- new "forward-only" action to force relaying and reject rcpts that expand
as local delivery.
- "!" (negation) modifier on rule matching conditions.
- new "recipient" rule matching condition.
- new "verify" option on listeners and relay rules to reject invalid
certificates.

Other changes:

- remember the helo name advertised on incoming mail and use it for sending
bounces.
- bump envelope version (existing envelopes are updated on-the-fly).

add "smtpctl show relays" and "smtpctl show hosts" commands

add missing heloname field for relayhost.
differenciate relays with different helotable/heloname.
improve code a bit.

use "/etc/mail/mailname" instead of "/etc/mailname" and make it a define.

Report mta sessions errors on the route rather than on the MX.
If a route has too many of these errors, disable it for a while.
Reset the error counter for a route when it is re-enabled or when
it could establish a connection successfully.

Implement a feedback mechanism which allows the mta to "hold" envelopes
in the scheduler when it has too many tasks for a given relay. The
envelopes are put on a wait queue, and are not scheduled again until
the mta "releases" some envelopes from that queue.

It prevents from having too many inflight envelopes, which are out of reach
for the admin.

Make the filter infrastructure move forward.
This is a work-in-progress and it's not supposed to be useable for now.

Create the control socket in the parent process to abort early if
another smtpd instance is running. Close the inherited socket in
every forked process but control.

Simplify code for loading and dumping envelopes. Makes it much easier
to deal with automatic upgrade between envelope versions at load time.

local enqueuer improvements:

- parse the whole input before trying to establish the connection
to the local socket: fixes timeout problems when reading the output
of a long running program.

- use sendmail(8)-like exit status.

We are basically at 5.4 now

Implement a scheduler_proc backend

Many MTA improvements:

- Better transient error handling logic: failing destinations are
automatically disabled for a while. When a destination is active
again, ask the scheduler to retry previous envelopes immediatly.
- More informative error report when all routes fail for a mail.
- Implement a "smtpctl show hoststats" command to get the latest stat
message per MX domain.
- Implement a "smtpctl show routes" command to show the state the
currently known routes to remote MXs.
- Implement a "smtpctl resume route" command to re-enable a route that
has been disabled.
- Do not hardcode limits
- Minor code improvements

Assorted queue improvements:
- cleanup the internal queue backend API and get rid of the QOP_* thing.
- implement a queue_proc backend
- rename queue_fsqueue.c to queue_fs
- enable support for queue encryption
- add an envelope cache
- better logging and error reporting

Add a table_proc backend for delegating table lookups to another
process. Stop building experimental table_sqlite and table_ldap as
they will be provided as external backends.

Move the filter infrastructure forward.

scheduler improvements:
- implement suspend/resume scheduling for individual envelopes or message,
with the associated smtpctl commands.
- allow the mta to request immediate scheduling of an envelope.
- on temporary failures a penalty can be given to further delay the next try.

New implementation for smtpctl and the command line parser. Allows
richer syntax, and makes the code way simpler to follow and extend
with new commands.

Get rid of env->sc_pw and env->sc_pwqueue. Early queue initialization
now happens in queue_init(), and backends take the queue passwd as
parameter in their init function.

Remove useless SMTPD_FILTER_USER while there.

Introduce expand string modifiers

Remove useless sc_pid from struct smtpd.

get rid of sa_set_port() and its awfully contorted implementation

we are at 5.3.3 now.

sync with OpenSMTPD 5.3.2

ok gilles@

replace MAX_LINE_SIZE and SMTP_LINE_MAX with SMTPD_MAXLINESIZE for
consistency and clarity. Remove useless and confusing extra byte in
a few arrays based on this define.

ok gilles@

as done in ospf{,6}d/relayd, sync yyerror in various other daemons with
that from bgpd, so that it logs to syslog when daemonized.

add missing bits for lmtp support (from Ashish SHUKLA).

ok gilles@

- log smtpd version at startup

ok eric@

- smtpctl trace expand, enables tracing of aliases expansion
- replace "users" keyword with "userbase" when providing alternate userbase
- disambiguise expansion nodes when expanding across domains and userbases
- allow use of '=' instead of '=>' when declaring a mapping

ok eric@

When getting the next batch of envelope to schedule, use an array to
store envelope ids, rather than a dynamic list.

ok gilles@

unbreak broken smtpctl table update

fix by eric and I, ok eric@

assorted fixes spotted by Coverity.
some log message updates.

ok gilles@

do not need to tweak the socket sndbuf, now that the envelopes are passed
in compressed form. reduce the default size for envelope messages.

ok gilles@

use a stripped-down mta_envelope structure in the mta process.
reduces memory footprint by a great deal when relaying lots of messages.

ok gilles@

- introduce 'smtpctl trace lookup' to trace lookup process
- improve logging of the transfer process

trace by me, logging by eric

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

Replace the qwalk API (to retreive on disk envelopes at runtime) with
a simple QOP_WALK queue operation. Some knf and formating fixes while
there.

ok gilles@

Allow "smtpctl show queue" to run in "online" mode if the smtpd server
is running. The scheduler sends the runtime state of each envelope to
the queue process which loads the envelope, fills the runtime bits and
sends the envelope back to the client. Iteration over the envelope set
happens in small chunks to make the request interruptible and to allow
the server to keep doing its job in the meantime.

Adpat "smtpctl schedule-all" to schedule the messages one by one using
the same iteration mechanism.

Document "smtpctl monitor" and "smtpctl show queue".

ok gilles@

Cleanups and improvements:

* Log more events (especially client session) and use a better scheme
for that: each messages is prefixed with a token to easily identify
its class:
- info/warn/debug: general server messages
- smtp-in: smtp client connections
- relay: status update for relayed messages
- delivery: status update for local deliveries

* Implement "smtpctl monitor" to display updates of selected internal
counters.

* When reloading the on-disk queue at startup do not commit a message
if no envelope was submitted for that message.

* Remove unused stuff in the config parser.

ok gilles@

Add a "kick counter" that gets incremented on each command, to detect
clients that don't do their best to do something useful, and just hog
the session. When that kick counter reaches the limit, the client is
disconnected. The counter is reset after the first HELO/EHLO command,
after tls is established, after a succesful authentication, and after
a message is accepted. It is decremented when a RCPT is accepted.

ok gilles@

Consistency and robustness improvements in mda:

- Introduce a mda_getlastline function(); improve the code to avoid
useless allocations and string formatting; make it return the last
line with content (skip trailing empty lines if found).
- Add a mechanism by which the mda can request the parent to abort a
local delivery by killing the process.
- Use ioev/iobuf for draining data to the delivery process.
- Make sure to catch all transient errors and make them result in a
tempfail rather than calling fatal().
- Make sure that the envelope status is properly set for all failures.
- Stop using SMTP response codes; it makes no sense in this context.

ok gilles@

Make counters more informative in the scheduler:

- Change the scheduler backend API a bit: commit() and rollback()
API calls return the number of envelopes added or canceled; put
the number of envelopes in the structure returned by batch().

- Properly report the number of incoming, registered, removed and
expired envelopes, as well as the outcome of deliveries.

ok gilles@

Limit the number of messages that can be enqueued on a single SMTP
connection, and the number of recipients in each of them.

ok gilles@ chl@

MAX_RULEBUFFER_LEN is too small, bump it.

discussed with gilles@

introduce map_file.c which will deprecate map_stdio.c

The idea is to have a file-backed map but to have smtpd(8) cache the maps
so that it cannot be partially read if edited while mail is received. The
file is read and converted to a static map (map_static.c), changes aren't
visible to smtpd until an explicit: smtpctl update map which reads file,
builds a new static map and invalidates the former.

partial-read issue discussed with beck@ and halex@
idea to convert internally to a static map by eric@

diff ok eric@ and chl@

Extend the "retry" field to 16 bits. The new quadratic retry formula
makes the maximum retry delay a bit to small on 8 bits.

ok gilles@ chl@

- map_create() takes a map_src not a map_kind

ok eric@ and chl@

- replace "from all" and "for all" with "from any" and "for any"

ok eric@, chl@

F_BACKUP and ROUTE_BACKUP must be sync-ed for now, otherwise smtpd won't
work as a backup MX ...

bug experienced by todd, verified and analyzed by eric

teach smtpctl how to display envelopes and messages using their id.
this allows an admin to inspect the queue without having to manually
extract bucket and find the path to an envelope or message.

diff by Sunil Nimmagadda <sunil@poolp.org>

ok eric@, chl@ and I

For each alias node, mark if it has been expanded from an alias map or
from a .forward file. Local deliveries for files and filters expanded
from an alias map are run as user _smtpd.

issue reported by tood@

ok gilles@ todd@

- allow a listen statement to impose tls on its clients;
- make listen statements impose authentication if 'auth' is specified and
to make it optional if 'auth-optional' is specified;
- sync documentation accordingly

with ideas and input from beck@ and halex@, ok eric@

disk space is cheap but we still want to limit the default size of a body
to a sane default for everyone.

Implement a simple wait queue API. The idea is to allow multiple "waiters"
to wait on the same "tag" for a deferred result.

A waiter is a callback and a void *argument. The first waiter (the one for
which waitq_wait() returns true) is supposed to run some code that leads to
waitq_run() being run, which will destroy that waitq and call all callbacks
in turn.

Not used at the moment, but will be soon.

ok gilles@ chl@

convert iobuf_queue()'s to iobuf_fqueue(). (idea from gilles@)
introduce iobuf_xinit() and iobuf_xfqueue(). (idea from eric@)

ok gilles@

we reintroduced a bug that was fixed 2 years ago with the aliases rewrite:

During the entire expansion process, a username may be larger than
MAXLOGNAME because it may be an alias going through another expansion.
We should use a buffer that's large enough to fit a mailaddr user-part so
we avoid hitting a truncation check leading to a fatal().

ok eric@, ok chl@

disallow root deliveries for "deliver to filename" and "deliver to mda"
rules, we only allow them for mbox and maildir though users should really
create a root alias ...

discussed with eric@ and chl@, ok both

- add decision to the rule so that we can actually perform a reject match
ie:

reject from 192.168.1.0/24 for domain "openbsd.org"
accept from 192.168.0.0/16 for domain "openbsd.org" deliver to mbox

it was documented but not working.

ok eric@ & chl@

some mfa_session cleanups.

- move mfa_session() prototype to smtpd.h
- make mfa session use a tree
- make static functions static
- merge mfa_session_init() into mfa_session()

ok chl@

finally remove rule member from struct envelope.

"wow!" gilles@

Remove support for "as user" for local deliveries.
It's not documented and not implemented.

ok gilles@

some smtpd.{c,h} cleanups:

- move struct child to smtpd.c
- make it use a tree keyed on the pid
- change child_add to take the title directly as a const char *
- remove useless child_lookup() and child_del()
- remove CHILD_INVALID

ok chl@ gilles@

Move mda_session to mda.c, and make it use a tree instead of a list,
but still use uint32_t keys since ithe key is used as peerid in msg.

ok gilles@

smtpd.h/control.c cleanups:

- move session_socket_* prototypes under util.c
- move struct ctl_conn in control.c
- make static functions static
- remove unused functions
- call unlink() in control_shutdown()
- make control_close() take a ctl_conn * instead of a fd

ok chl@ gilles@

clarify the alias expansion code.

The session manages a list of nodes to process. A node has a link to the
parent node from which it has been expanded, and a link to the rule that
led to its creation. Depending on its type and the associated rule, each
node is either "expanded" to create new nodes or "submitted" to create a
final envelope. Nodes which have already been seen, either processed or
not, are discarded to avoid loops.

The expansion process is bootstrapped by creating an EXPAND_ADDRESS node
from the original dest, with no rule and no parent. It is done when all
nodes have been expanded or if an error occurs before. The expand depth
is limited 5 levels. The whole expansion fails if the limit is reached.

While there, make sure that only one .forward file is queried at a time,
and only append the subfolder tag in the maildir case.

Fixe issues with some virtual map setups where the dest would get mixed
up, and make the whole expansion process generally easier to follow.

ok chl@ gilles@

Stop using the delivery_data union (field "to") in delivery_mda.
It's confusing and not necessary as it's only used for "buffer".
Instead, just add a "buffer" member in the structure and rename
"as_user" to "user".

The delivery_data union becomes an anonymous union in expandnode,
which is the only other place where it's used.

ok gilles@

make const arguments const, and static functions static.

ok chl@

Do not pass the username to forwards_get() which does not have to care about
this. Instead, set the username on the expand context, and copy it on the
expand nodes as they are inserted.

ok gilles@

wrap expandtree into a "struct expand".

ok gilles@

move struct lka_session definition in lka_session.c

ok gilles@

Add a log_envelope() function that log envelope status in a uniform way.
It automagically adds an rcpt=<user@domain> field if "dest" differs from
the original "rcpt". The function takes an "extra" parameter that allows
to add some specific info depending on the context.

ok gilles@

Move ruleset_match() prototype to smtpd.h and make the envelope const.
Adapt a lot of functions in chain to use const args where required.

ok gilles@

constify parameters that are supposed to be const.

ok gilles@

Remove DF_ENQUEUE flag. It is mostly unused and logically broken.
Ignore it in existing envelopes until it gets completely dropped.
Change "smtpctl show queue" to display the address family of the
envelope source instead of the ENQUEUE flag.

ok gilles@

remove IS_RELAY and IS_MAILBOX macros.

ok gilles@

Remove aliases_exists() and aliases_virtual_exists(). The corresponding
*_get() functions can be called directly.

ok gilles@

start cleaning the expansion code:

- change expandtree_* prefix to expand_ for better readability and
because the structure might change at some point
- rename <>_free_nodes() to <>_free()
- remove unused <>_remove_node()
- refcounting has no purpose at all; just remove it as well as the
decrement/increment functions, and replace the latter with <>_insert
- expandnode flags is only used to know if it's been processed or not,
don't make it a flag but a simple field with clear name.

ok gilles@ chl@

- add xmemdup() helper.
- remove useless block in switch.

ok gilles@

simple lka cleanups:

- fix lka* function prototypes in smtpd.h
- make static functions static
- merge lka_session_init() into lka_session()
- make lka_session.c use tree.c to store sessions

ok gilles@

this structure is not useful and ill-named. remove it.

ok gilles@

remove C_NET. it's not used and there is no plan for it at the moment.

ok gilles@

Add map_create() and map_add() helpers. Simplify the config parser by a
great deal.

While there, rename the default "localhost" map to "<localhost>" to make
it look more internal, and create a single "<anyhost>" map referenced by
"from all" rules, instead of creating a dynamic one for each of them.

ok gilles@ chl@

now that log_imsg() is only used in smtpd.c, set it as static.

ok gilles@

silent warnings

reported by ajacoutot@

ok gilles@ ajacoutot@

replace BSD-licensed mkdir_p() with ISC-licensed mkdirs(), this allows us
to avoid a dual-licensed util.c for no reason

ok chl@

Remove s_ssl from the smtp session since it is duplicated in the io struct.
Change ssl_session_init to ssl_smtp_init and make it simpler: only create
an SSL* from the SSL_CTX* passed as parameter, so it does not have to know
about the struct session itself. Kill some dead prototypes while there.

ok chl@ gilles@

Rework the scheduler internals. Fix some scheduling loop issues and
handle envelope scheduling/expiration better.

ok gilles@

remove unused flag

- remove crypto_backend
- remove support for encrypted queue, it will be reintroduced later after
pouring more thinking into it

if you had it enabled, flush your queue before updating

change crypto_setup() prototype to use const char * instead of uint8_t *

while there do some KNF:
- change 8 spaces to tab
- add/remove some missing/extra space after if's

ok gilles@

switch compress_backend to use FILE * instead of file descriptors, like
crypto_backend

ok gilles@

- import latest aldap.[ch] and ber.[ch] from ypldap
- revive map_ldap.c by updating it to the current API

diff by Mathieu Masson who played puzzle with an oooold changeset of mine,
this import is to let us work on it in tree, it won't work as is.

idea ok eric@ and chl@

Introduce the crypto_backend API and provide support for... encrypted queue
using the new API. By default, OpenSMTPD does not provide queue encryption,
but it can be enabled with "queue encryption [args]" and will transparently
encrypt/decrypt envelopes/messages as they hit the queue.

By default, it will use Blowfish in CBC mode with a different random IV for
each envelope and message. User provided key is expanded using sha256 but a
different cipher and digest may be specified in smtpd.conf

Queue encryption is compatible with compression and if both options are set
it will do them in correct order and transparently.

tested by chl@, a few users and myself
ok chl@ and I

remove encrypt.c prototypes leftovers

ok gilles@

- define ZLIB_BUFFER_SIZE instead of hardcoding 8192
- check gzdopen() failure
- call gzclose() whenever a failure occurs after gzdopen()
- simplify slightly some checks in compress/uncompress
- create PATH_TEMPORARY in /var/spool/smtpd, chmod 700, owned by _smtpd
- compress_zlib should use PATH_TEMPORARY instread of /tmp as we're
chrooted and this will otherwise lead to a fatal()

ok chl@

Add compress_backend, allowing compression of messages and envelopes in the queue.
To use it, just add "queue compress" in smtpd.conf. For now, only zlib is used.

lots of feedback from eric@ and gilles@

ok eric@ gilles@

- add myself to the copyright in control.c, i've done quite a few changes
there in the last few years ;-)
- get rid of availdesc(): getdtablecount() is so much more reliable
- get rid of env->sc_maxconn, we can be much smarter with getdtablecount()
and getdtablesize()
- disable accept when we hit the control process fd reserve
- disable accept when we fail
- enable accept when we're back below the limit

this is not the full fd exhaustion diff, i'll merge changes from relayd
tomorrow, this was only required to get rid of the env->sc_maxconn and
availdesc() mess

"reads alright" eric@

- stop accepting clients if we hit our fd reserve limit (or if we fail)
- resume if we go below the fd reserve

with feedback and ok eric@

- offline enqueue does not need to use the user_backend API, it relies on
system users ... use getpwuid() instead of ub->getbyuid()
- since that was the only caller, get rid of user_backend->getbyuid()

this is the first step towards removing the user_backend API and making
user lookups available through the maps API (yes, virtual user support ;)

ok eric@, ok chl@

- introduce TRACE_PROFILING
- when smtpd starts with -T profiling it will log_trace() some prof. info
- when smtpd starts with -T profstat, it will push them to stats API with
type STAT_TIMESPEC under key profiling.imsg.*

with this diff we can get live profiling of events with a very minimal
overhead :-)

ok chl@, ok eric@

- introduce struct stat_value
- statistics can now have a type (counter, timestamp, timeval, timespec and
possibly others in the future)
- stat_increment() / stat_decrement() now take an increment/decrement value
and are at the moment only of type counter
- stat_set() now takes a stat_value
- provide helpers to convert raw values to stat_value

ok eric@, ok chl@

while at it fix a rq_queue_dump() call using a bogus timestamp in scheduler
ramqueue.

In envelope ascii dump/load:
- remove loading of evpid.
- don't dump the msgid
- ignore msgid at load
- remove now unused functions ascii_{dump,load}_uint{32,64}_hex()

With inputs from eric@ and gilles@

ok gilles@ eric@

Don't pass struct envelope pointer in queue backend API, instead use envelope id and
an envelope ascii buffer.

ok eric@ gilles@

Remove the rq_host and rq_batch structures from the ramqueue scheduler.
The scheduler should only allow admin to schedule specific envelopes by
id, or msgid. More advanced scheduling (per host/route/whatever) should
be achieved using smtpctl schedule-id and proper filtering on the queue,
or using ad-hoc scheduler backend and tools.

ok gilles@ chl@

Allow smtpd to work as a backup MX, relaying only to MXs with higher
priority in the DNS record. For example:

accept for domain "foo.org" relay backup "mx3.foo.org"

will relay mails for "foo.org" using only hosts with higher priority
(i.e. lower value) than "mx3.foo.org", which is supposed to be the
current server.

If the specified backup MX is not found in the DNS record, relaying
works as normal.

ok gilles@

Re-enable loop detection, but in mta and mda this time.

ok gilles@

MAX_LINE_SIZE is supposed to define the max length of a SMTP line ...
... but SMTP_LINE_MAX *also* defines it ... with a different value ...
and did I mention both were too small anyway ?

quick fix until we kill one or the other: bump MAX_LINE_SIZE and define
SMTP_LINE_MAX to be MAX_LINE_SIZE. this fixes the immediate issue while
we decide which one bites the dust.

fixes the crashes and "line too long" errors spotted by todd@
ok todd@, ok chl@

coding style: replace all occurences of u_int* with uint*

ok eric@

Kill envelope_{dump,load}_file() and replace them with envelope_{dump,load}_buffer().

with input from eric@

ok eric@

fix an issue where too long lines were not spot properly.

issue reported by todd@

ok eric@

zap struct mta_batch. Only pass ids where needed.

ok gilles@

- introduce stat_backend, an API for pluggable statistic backends
> statistics are no longer static structures in shared memory
> statistics are only set, smtpd never uses them in its logic
> each statistic is a key/value where key can be any (dynamic) string
- convert all uses of the former API to use the new one
- implement stat_ramstat that keeps non-persistent stats in ram structure

ok eric@, ok chl@

Major update of the mta internals.

Add a mta_route structure which describes a route through which
outgoing mails are to be sent. This structure holds connection
parameters and limits. When an envelope is received in a batch,
the route for it is looked up, and the envelope is added to the
a list of envelope to be sent for this message on that route: a
task. When the batch is closed, each task is added to the list
of tasks for their respective route.

The routes are drained when new work can happen. The route will
create new mta sessions if necessary. When a session is up and
ready, it picks the first pending task on the route if any. In
the other case, it just closes the connection.

Errors on the connection are reported to the route, so that the
route could be flagged as broken. Currently, three errors on a
an attempt to open a route is reported as a failure for all pen-
ding tasks.

ok gilles@

Move mta and smtp specific defines into their own files.
Some formatting cleanups while there.

ok gilles@

Allow failure reports for different recipients of the same message
to be grouped into a single bounce message.

The bounce structure keeps a list of envelopes. For now, the list
is constructed by delaying the re-enqueuing of a bounce envelope a
bit, to wait for other bounces from the same message to be part of
the same report.

remove unused function and prototypes

Improve the message flows to completely isolate operations on the
queue backend within the queue process.

The scheduler sends envelope ids to the queue process which loads
the envelope and forward the request to the agent responsible for
the delivery. The result is sent by the agent to the queue which
updates the storage before notifying the scheduler.

Bounces are created and enqueued (from the client side) by the
queue process, rather than the scheduler.

ok gilles@

remove useless defines

ok gilles@ chl@

Improve the scheduler backend API.

New envelopes are pushed into the scheduler through the insert()
commit() rollback() transactional interface functions.

Worklists are pulled from the scheduler through a single batch()
interface function, which returns a list of envelope ids and the
type of processing. Envelopes returned in this batch are said to
be "in-flight", as opposed to "pending". They are supposed to be
processed in some way, and either updated() or deleted() at some
point.

The schedule()/remove() functions are used to alter the internal
state of "pending" envelopes to make them schedulable. The enve-
lopes will be part of a worklist on the next call to batch().

Rewrite the scheduler_ramqueue backend.

The initial queue loading in now done by the queue.

ok gilles@

Implement a set of tree_* functions for storing arbitrary pointers in splay
trees with uint64_t keys. Also add x{m,c}alloc and xstrdup helpers.

ok gilles@

- introduce xlowercase() and allow lowercase() to fail gracefully
- replace all calls to lowercase() with calls to xlowercase()
- in the format string expansion, lowercase() all formats

we will have to reassess all calls to xlowercase() even though it has never
triggered as far as I know, we can probably gracefully fail some of them.
right now we're just keeping former behaviour.

this commit fixes issue reported by Hugo Osvaldo Barrera where a %u format
could lead to a delivery failure (ie: GILLES@openbsd.org should be expanded
to gilles, not GILLES ... only for local deliveries).

ok chl@ on the idea, ok eric@ on the diff

get rid of A_INVALID.
little code cleanup while here.

ok gilles@

remove the session tree from the global env and move it to mta_session.c,
along with mta_relay and mta_session definition.

ok gilles@

add support for maildir tagging/folders.

ok gilles@
ok eric@ on previous versions of this patch

backout the:
- remove the /envelopes subdirectory, envelopes are at the same level than
the message file
- kill PATH_ENVELOPES define

but keep the:
- reduce the number of buckets from 0xfff to 0xff, this avoid performances
of the queue to decrease when we start having tons of buckets

ok eric@ gilles@

accept address literal for the recipient domain.
while there, change valid_{local,domain}part() prototypes to use const char *.

with input from gilles@ and eric@

ok gilles@ eric@

- simplify the scheduler loop logic further, it is ridiculously simple now
and I don't think we can do much better (at that level) :-p
- always break out of the handler after processing an envelope, this will
avoid a busy scheduler from not getting a chance to handle SIGTERM/SIGINT
YES we can now ctrl-c a maaaaad scheduler !

ok eric@, ok chl@

- introduce log_trace(TRACE_SCHEDULER, ...)
- simplify a tiny tiny bit the scheduler loop
- no functional change (yet)

- runner is the terminology we used back when we had runqueues, we no
longer have them and runner is actually a scheduler so rename.
- introduce scheduler_backend which does the same to scheduler than
queue_backend does to queue and map_backend does to maps
- remove all occurences of RUNNER and runner, replace them with SCHEDULER
and scheduler

ok eric@, ok chl@

first step of simplifying fsqueue:

- remove the /envelopes subdirectory, envelopes are at the same level than
the message file
- kill PATH_ENVELOPES define
- reduce the number of buckets from 0xfff to 0xff, this avoid performances
of the queue to decrease when we start having tons of buckets

this diff introduces a change to the queue layout, you will want to empty
your queue before updating. more cleanup to come

ok eric@, ok chl@

remove enum queue_kind from queue_fsqueue.c.
incoming messages are now always stored in /incoming, whatever the queue_backend is.
remove QOP_FD_RW and fsqueue_message_fd_rw().
while there check return value of generated paths before calling rmtree()

with advice from gilles@ and eric@

ok gilles@ eric@

Lookup queue and scheduler backends by name, rather than enum.
Add a command-line option to specify the backend to use at runtime.

ok gilles@

Finally get rid of the queue_kind enum in the queue API. Keep that
internally in fsqueue backend for now, and let the fsqueue_message()
and fsqueue_envelope() dispatchers do the right thing.

Based on a diff by chl@

ok chl@ gilles@

- introduce struct scheduler_info and the scheduler_info() function to fill
a struct scheduler_info given a struct envelope
- adapt the scheduler API and the scheduler_ramqueue backend to use the new
struct scheduler_info instead of a struct envelope

idea discussed with eric@ and chl@, mechanical diff, no functional change

- rename filter.h -> filter_api.h to be consistent with upcoming changes

Do not store the envelope id within the envelope, only the message id.
Make sure existing envelopes can be properly loaded.

ok chl@ gilles@

remove unused fields

ok eric@

move envelope dump/load functions to envelope.c

ok gilles@

- introduce map_static.c as a backend to static maps in parse.y, this has
the benefit that we no longer have two code paths whenever we deal with
maps, we can always use the backend mechanism.

I have not plugged this in yet, I'll do it in a later commit, just get it
out of my sandbox

- introduce text_to_relayhost() which converts an url into a relayhost.
urls are of the form: [schema://]host[:ip]

not used, yet other commits are following ;-)

- cleanup parse.y by removing lots of code that should not have been there,
but in ruleset.c and util.c instead.

- introduce the new map_compare() map API call to allow iterating over keys
and comparing them with provided key using provided function. this allows
checking a partial key in a key set, very useful for comparing an address
to a set of netmask.

- introduce new map kind K_NETADDR
- implement K_NETADDR for map_db and map_stdio
- teach ruleset checking how to use the map_compare() with K_NETADDR

we can now do the following:

map "srcaddr" source plain "/etc/mail/srcaddr.txt"

accept from map srcaddr for domain "openbsd.org" [...]

- remove unused sources S_EXT, S_DYN and S_EXT from enum map_src
- continue simplification of parse.y
- remove "for network", if we ever need it we can reimport, probably no
one knows of that undocumented strange feature ;-)
- change syntax for virtual domains configuration:

accept for virtual vmap [...] <- wrong
accept for virtual map vmap [...] <- right

the reason for this change is that we will soon implement relay rules
through maps and that keeping that syntax would make it inconsistent
with the other rules.

- update man pages for makemap and smtpd.conf to reflect changes

ok eric@, looks ok chl@

- simplify a bit maps by removing fields which are still unused years
after the initial ambitious implementation: byebye map type & map flags

- simplify a bit parse.y by removing assignations to these otherwise unused
fields

- remove the DNS map source, it may be a good idea, but we can just add it
when we plan to implement it (if we do)

- make the { } options in map declaration, it's been annoying me for a long
time now, this allows the following to work:

map "foobar" source plain "/etc/mail/foobar"

- update smtpd.conf.5 accordingly ;-)

Update the internal mta implementation so that a session now has a
list of messages to send to the remote smtp server over the same
connection. It's not currently used as the scheduler/runner is not
aware of this yet, and the imsg protocol would need to be updated.

ok gilles@

- rename all occurences of K_SECRET to K_CREDENTIALS
- rename all occurences of struct map_secret to map_credentials
- do not fatal if the credentials map has disappeared, instead make the
auth fail with a lookup failure. the mail will be temporary failed so
it stays in queue until admin fixes smtpd.conf, removes mail, or lets
it expires

split the session logic off mta.c into mta_session.c

ok gilles@

various reliability fixes:

- prevent queue_fsqueue from fatal() when it hits an ENOENT, it can happen
- change a bit the scheduler API to simplify it, fix runner accordingly

- we can't remove msg/batch from ramqueue while envelope is offloaded or
it will cause a double, instead we add refcnt to both msg/batch and
only free them when it hits 0

fix an issue observed this week-end while flooding ajacoutot@ :

we keep track of available fd's to prevent scheduling of messages if we
know that we are going to fail. however, since the envelope is not
removed from the scheduler, it will be rescheduled right away leading to
a busy loop in the scheduler. we know flag the mda/mta processes as BUSY
and do not schedule envelopes that target a BUSY process.

also, fix a potential bug that could lead to a use after free when doing
a batch/message/host traversal of schedulable envelopes.

while at it fix misuse of env->sc_opts as env->sc_flags, was not really
causing any issue as the misuse was constant ...

Rewrite io code in smtp and mta using the iobuf/ioev interface to have
a better separation between io and protocol logic. As a side-effect,
it fixes a couple of long-standing issues in the io path, and
hopefully add fresh ones instead. Kill client.c in the process.

ok gilles@

add optional display handler to scheduler_backend, if not NULL the handler
will be called for each iteration of the runner

implement a display handler for scheduler_ramqueue to display the entire
ramqueue (hosttree, msgtree and linear queue) in log_debug

remove useless state

ok gilles@

- introduce the scheduler_backend API
- introduce the scheduler_ramqueue backend
- remove all occurences of ramqueue outside of the ramqueue backend
- teach runner how to use the new API

it is now possible to write custom schedulers !

ok eric@, ok chl@

Add a parameter to the queue backend init() call to specify wether the
call is issued by smtpd or smtpctl. In the latter case, only perform
sanity checks and do not touch directories. A running server no
longer lose its "incoming/" directory each time smtpctl is called...

ok gilles@

Add new filters callbacks for:
- network events (CONNECT/CLOSE)
- commands (QUIT/RSET)

ok gilles@ eric@

queue_message_purge() and queue_message_delete() are actually the same
thing. Remove queue_message_purge() in favor of queue_message_delete
and simplify fsqueue_message_delete() implementation to move the
message dir to purge/

ok gilles@

remove the status field from struct envelope, move it to the smtp
session, and cleanup the DS_* flags.

ok gilles@ chl@

Stop using envelope->status to report delivery outcome to the
runner/queue. Instead, replace IMSG_QUEUE_MESSAGE_UPDATE with three
messages:

- IMSG_QUEUE_DELIVERY_OK
- IMSG_QUEUE_DELIVERY_TEMPFAIL
- IMSG_QUEUE_DELIVERY_PERMFAIL

1) it's less confusing as status is also used by smtp
2) it's easier to see what happens just looking at imsg traces
3) it makes the code path generally easier to follow
4) it's safer because it enforces clear semantics and intent, whereas
the status field is loosely defined and could carry bogus values.

ok gilles@ chl@

use mbox backend for mbox delivery.

ok gilles@

Remove dead code for config reloading for now. It is not functionnal
and confusing.

ok gilles@

remove envelope_get_errormsg() and move envelope_set_errormsg()
to envelope.c

ok gilles@

remove stateful iteration from ramqueue, if we ever need to reintroduce it
we'll do it, but it isn't used and causes potential bugs

idea by Nathanael Rensel, diff by me, ok eric@

implement an envelope_ascii API that's not tied to a specific queue_backend
simplify queue_fsqueue

Simplify runner/queue by getting rid of Q_PURGE. Instead, let smtpd
periodically clear the purge/ directory. At init time, the fsqueue
backend simply moves the existing incoming/ dir in purge/ to discard
aborted sessions.

ok gilles@ chl@

remove dead prototype

from Nathanael Rensen

ok gilles@