consider an MX of "localhost" as it were a "Null MX"

diff from Philipp (philipp+openbsd [at] bureaucracy [dot] de), thanks!

ok sthen@

RFC 7505 ("Null MX") handling

mail delivery will not be attempted if a domain advertises a single MX
record with preference 0 and a zero-length label.

based on an initial diff from Philipp (philipp+openbsd [at] bureaucracy
[dot] de), thanks!

ok jung@

add required headers for smtpd.h and remove unnecessary ones in other files.

ok jung@

Implement server certificate validation in smtp(1).
Check certificate against MX name in smtpd(8) mta.

ok gilles@

treat NOTIMP as NO_DATA in response to MX query: fallback to hostname lookup
instead of bouncing the mail.

ok gilles@

Implement a generic interface to forward resolver queries to the lka
process. Use it for the reverse lookups required by smtp and mta.

Until now, DNS-related lookups were implemented using ad-hoc IMSGs
between the lka and other processes. It turns out to be confusing and
difficult to maintain/extend. So we want to replace this with a better
set of IMSGs matching the standard resolver interface.

ok gilles@

remove 'where' parameter from all x*() functions in utils.c, it doesn't
really help us with anything, propagate the change in codebase

ok millert@

Move unpack functions into a seperate file.

Required for upcoming 'smtpctl spf walk'.
Ok eric@ gilles@ millert@

typo; from Edgar Pettijohn

dns_lookup_host() needs to remove brackets and IPv6: prefix when receiving
a text representation otherwise getaddrinfo_async() will choke

ok eric@

remove unused variables

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

fix whitespace and indentation, by Kyle Milz

restrict address lookups to configured address families.

ok gilles@

(void) cast this strlcpy(), it cannot truncate

these strlcpy can't truncate, the copy is from a buffer to a buffer of same
size and the first buffer handles the truncation already

Merge the mda, mta and smtp processes into a single unprivileged
process managing message reception, delivery and transfer. Mostly
mechanical, but very intrusive as it required to rewamp all IMSG to
fix ambiguities.

with and ok gilles@

use asr helpers after libevent update.

asr API is now public

update after asr API update

Integrate necessary dns packet parsing helpers from asr.
They are not supposed to be exposed.

ok gilles@

field rename

bcopy -> memmove
bzero -> memset

When looking up a MX, parse the address if the domain is a "[ipaddr]" string.

%i -> %d in format strings

update after asr changes.

sync with OpenSMTPD 5.3.2

ok gilles@

remove params after API change.

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

fix after asr update

knf

ok gilles@

Cleanups and improvements:

* Log more events (especially client session) and use a better scheme
for that: each messages is prefixed with a token to easily identify
its class:
- info/warn/debug: general server messages
- smtp-in: smtp client connections
- relay: status update for relayed messages
- delivery: status update for local deliveries

* Implement "smtpctl monitor" to display updates of selected internal
counters.

* When reloading the on-disk queue at startup do not commit a message
if no envelope was submitted for that message.

* Remove unused stuff in the config parser.

ok gilles@

skip RR if type is not MX. Use hostname if the list of MX is empty
after the loop.

spotted by huku at grhack.net

ok gilles@

when requesting MX entries, the result can be appear in random orders.

the logic for inserting them in a lka session when acting as backup MX did
not take account for one specific case that could lead to an early exit
without smtpd getting a chance to detect the entry corrsponds to itself.

in such case, a backup MX woud try to connect to itself and bounce in the
loop detection code ... or it would sometimes work.

use xmalloc()/xcalloc() helpers

while there unify usage of log_trace() in ramstat_set()

ok gilles@ eric@

- introduce struct stat_value
- statistics can now have a type (counter, timestamp, timeval, timespec and
possibly others in the future)
- stat_increment() / stat_decrement() now take an increment/decrement value
and are at the moment only of type counter
- stat_set() now takes a stat_value
- provide helpers to convert raw values to stat_value

ok eric@, ok chl@

while at it fix a rq_queue_dump() call using a bogus timestamp in scheduler
ramqueue.

Allow smtpd to work as a backup MX, relaying only to MXs with higher
priority in the DNS record. For example:

accept for domain "foo.org" relay backup "mx3.foo.org"

will relay mails for "foo.org" using only hosts with higher priority
(i.e. lower value) than "mx3.foo.org", which is supposed to be the
current server.

If the specified backup MX is not found in the DNS record, relaying
works as normal.

ok gilles@

Use TAILQ rather than array for mx list.

ok gilles@

dns sessions don't use lookup. no need to store them in a tree.

ok gilles@

coding style: replace all occurences of u_int* with uint*

ok eric@

- introduce stat_backend, an API for pluggable statistic backends
> statistics are no longer static structures in shared memory
> statistics are only set, smtpd never uses them in its logic
> each statistic is a key/value where key can be any (dynamic) string
- convert all uses of the former API to use the new one
- implement stat_ramstat that keeps non-persistent stats in ram structure

ok eric@, ok chl@

cleanup some old debug traces

ok gilles@ chl@

fix bogus permfail when no MX is defined on a valid domain.

ok gilles@

move to the new resolver implementation, with temporary glue to use
the relevant files from asr directly.

ok gilles@

improve readability

ok gilles@

Try to parse hostnames as IP addresses before resolving. This allows
relays to be given as IP address in the config file.

ok gilles@

Improve error reporting. Most errors during hostname lookup are
now correctly reported as temporary failures.

from Nathanael Rensen, tweaks by me.

Introduce a small set of functions to manage stat counters in a
simpler and hopefully saner way.

ok gilles@ chl@

Fix reporting of permanent/temporary failures for MX lookups.
Simplify code a bit while there.

ok gilles@

imsg.h requires sys/queue.h and sys/uio.h.

ok eric

move dns session specific structs and prototypes out of smtpd.h.

ok gilles@

the smtpd env is meant to be global, so do not pass it all around.

discussed with and ok gilles@

cleanups, cosmethic changes, functions that should be static are now static
no functionnal change

add stat counters for the lookup agent

ok gilles@

cleanup and simplification following the asr update.

- use a specific dispatch function for each type of query
- make the host handler work on a list of hosts by default (single host
queries are just a particular case) and use that to resolve the MX list
- various other code cleanup
- remove unused headers
- remove orphaned prototypes
- update copyright

ok gilles@

remove unused code now that reverse lookups are done through asr.

ok gilles@

make use the cname query interface from asr for reverse lookups

ok gilles@

add a function to factorize resetting of dnssession events.

ok gilles@

Fix the MX lookup process:

- the MX records were not always properly inserted into the sorted
array, which led to some MX being silently dropped.
- if an MX address could not be resolved, mail delivery would fail,
even though other valid MX exist for that domain. Now only report
the failure if no server address can be found at all.

grrrreeat gilles@

Tweak the asr API to make things a bit smoother on the user side.
Then asr_run() call now returns ASR_COND when a condition on a FD is
expected. The exact condition (readable or writeable) is specified in
the asr_result structure, along with the fd and timeout.

ok gilles@

use an index for iterating into the mx list.

ok gilles@

Simplify resolver initialization. This is done only once
since resolv.conf reloading is handled automatically by asr.

ok gilles@

make similar code .. more similar
ok gilles@

previous commit should have read:

when copying 'struct sockaddr' data, use sa_len not sizeof(struct sockaddr_in)
this fixes truncation of IPv6 addresses in the mail delivery path
ok gilles@

*** empty log message ***

If MX lookup fails, fallback to using the host itself. This has always been
the behavior but I introduced a regression when switching to ASR.

bug reported by jmc@, bugfix tested by jmc@ and I

use memcpy instead of a cast/deref dance that was reading past
the end of the buffer.

tested by and ok gilles@

replace the fork-based-non-blocking-resolver-hack by shiny async resolver
written by eric@. it is still experimental but still better than what we
had earlier so ... we'll improve in tree :)

diff by me with *lots* of help from eric@, tested by todd and I (and a
few people out there)

remove unused headers

a bit of .h cleanups, no functionnal change

we do dns resolutions in a separate process because we don't have an async
resolver. if we run scarce on resources and we cannot fork a separate dns
process or we cannot socketpair() tell the caller that we have a temporary
failure rather than issueing a fatal(). message will stay in queue and be
rescheduled later ...

bug reported and bugfix tested by Sacha El Masry <lists@devilray.eu>

force the dns buffers to be aligned using a union, until the retarded
"misalign strings on the stack" bug in gcc4 is fixed (even when that
is fixed this idiom is safer and quite common)
ok jacekm

check event_dispatch() return value

ok jacekm@

add missing header needed by signal()

ok gilles@

Fix previous. When configured to relay via IP address, MX lookup would fail
(NXDOMAIN), leading to a bounce. Precede the MX lookup with an attempt to
parse the relay as numeric string.

"reads ok" gilles@

Consider DNS lookups that result in NXDOMAIN to be a permanent failure.

ok gilles@ jacekm@

Introduce a 6yz status code, used internally to report permanent errors.
The 1yz and 6yz status codes are now removed prior to reporting the status
message in bounce messages, which provides an easy way to distinguish
between local and remote status messages. Initial diff from jacekm@

ok gilles@ jacekm@

imsg_get sets errno so use fatal instead of fatalx.

import some changes from portable smtpd to reduce the delta between both.
this commit contains mostly missing casts and cosmethic changes, do not
expect to build this anywhere but on OpenBSD, it does not contain any of
the portable glue.

Sync with relayd:
Stop pushing event handling in the imsg framework.
Instead, provide a small glue layer on top of both imsg and libevent.
This finally clearly separates event handling and imsg construction.

Sidetrack bonus: remove the mega-ugly hack of having a dummy imsg_event_add
stub in smtpctl.
ok jaceckm@

make smtpd's imsg lib ready, just like relayd and ospfd.
ok gilles@, jacekm@

Fix EV_READ/EV_WRITE testing inside IMSG handlers. Based on similar change
to the routing daemons by claudio@; ok gilles@

- New API to handle all DNS query types (A, MX, PTR) asynchronously.

- Improve RFC compliance: CNAMEs are resolved, equal preference MXs
are randomized, relaying via MX that has equal/lower preference
than local server is prevented, decision on when to treat domain
name as implicit MX is better.

ok gilles@

replace MAX* constants by sizeof where possible

ok jacekm@

If MX query fails due to DNS error, do not attempt more queries; ok gilles@

Fix a bug where list of 6 MXs or more was not handled correctly.
Pointed out by & ok sthen@, ok gilles@.

remove unnecessary includes; ok gilles@

- more err/errx -> fatal/fatalx, warn/warnx -> log_warn/log_warnx
contains bits based on an old diff from Jacek Masiulaniec and
other bits from me.

- in mxsort, fix type of loop counter, it will never be < 0 if it is
unsigned and when running out of luck it will cause the lookup
process to crash.

- move prototype to smtpd.h

insertion sort is faster than bubble sort. ok gilles

add a few missing id tags; there are a bunch of files, and developers
will probably miss this change when working on more important matters,
so it is probably better to sort them now. there is a risk of losing
the tags if a change needs to be reverted too.

written with excellent advice from jmc@

ok gilles@

smtpd is a smtp server implementation for OpenBSD. It is a work in progress
which still lacks many features. bringing it in tree will help working on it
more easily.

"at this stage it should go in" henning@, "move ahead" deraadt@

RFC 7505 ("Null MX") handling

mail delivery will not be attempted if a domain advertises a single MX
record with preference 0 and a zero-length label.

based on an initial diff from Philipp (philipp+openbsd [at] bureaucracy
[dot] de), thanks!

ok jung@

add required headers for smtpd.h and remove unnecessary ones in other files.

ok jung@

Implement server certificate validation in smtp(1).
Check certificate against MX name in smtpd(8) mta.

ok gilles@

treat NOTIMP as NO_DATA in response to MX query: fallback to hostname lookup
instead of bouncing the mail.

ok gilles@

Implement a generic interface to forward resolver queries to the lka
process. Use it for the reverse lookups required by smtp and mta.

Until now, DNS-related lookups were implemented using ad-hoc IMSGs
between the lka and other processes. It turns out to be confusing and
difficult to maintain/extend. So we want to replace this with a better
set of IMSGs matching the standard resolver interface.

ok gilles@

remove 'where' parameter from all x*() functions in utils.c, it doesn't
really help us with anything, propagate the change in codebase

ok millert@

Move unpack functions into a seperate file.

Required for upcoming 'smtpctl spf walk'.
Ok eric@ gilles@ millert@

typo; from Edgar Pettijohn

dns_lookup_host() needs to remove brackets and IPv6: prefix when receiving
a text representation otherwise getaddrinfo_async() will choke

ok eric@

remove unused variables

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

fix whitespace and indentation, by Kyle Milz

restrict address lookups to configured address families.

ok gilles@

(void) cast this strlcpy(), it cannot truncate

these strlcpy can't truncate, the copy is from a buffer to a buffer of same
size and the first buffer handles the truncation already

Merge the mda, mta and smtp processes into a single unprivileged
process managing message reception, delivery and transfer. Mostly
mechanical, but very intrusive as it required to rewamp all IMSG to
fix ambiguities.

with and ok gilles@

use asr helpers after libevent update.

asr API is now public

update after asr API update

Integrate necessary dns packet parsing helpers from asr.
They are not supposed to be exposed.

ok gilles@

field rename

bcopy -> memmove
bzero -> memset

When looking up a MX, parse the address if the domain is a "[ipaddr]" string.

%i -> %d in format strings

update after asr changes.

sync with OpenSMTPD 5.3.2

ok gilles@

remove params after API change.

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

fix after asr update

knf

ok gilles@

Cleanups and improvements:

* Log more events (especially client session) and use a better scheme
for that: each messages is prefixed with a token to easily identify
its class:
- info/warn/debug: general server messages
- smtp-in: smtp client connections
- relay: status update for relayed messages
- delivery: status update for local deliveries

* Implement "smtpctl monitor" to display updates of selected internal
counters.

* When reloading the on-disk queue at startup do not commit a message
if no envelope was submitted for that message.

* Remove unused stuff in the config parser.

ok gilles@

skip RR if type is not MX. Use hostname if the list of MX is empty
after the loop.

spotted by huku at grhack.net

ok gilles@

when requesting MX entries, the result can be appear in random orders.

the logic for inserting them in a lka session when acting as backup MX did
not take account for one specific case that could lead to an early exit
without smtpd getting a chance to detect the entry corrsponds to itself.

in such case, a backup MX woud try to connect to itself and bounce in the
loop detection code ... or it would sometimes work.

use xmalloc()/xcalloc() helpers

while there unify usage of log_trace() in ramstat_set()

ok gilles@ eric@

- introduce struct stat_value
- statistics can now have a type (counter, timestamp, timeval, timespec and
possibly others in the future)
- stat_increment() / stat_decrement() now take an increment/decrement value
and are at the moment only of type counter
- stat_set() now takes a stat_value
- provide helpers to convert raw values to stat_value

ok eric@, ok chl@

while at it fix a rq_queue_dump() call using a bogus timestamp in scheduler
ramqueue.

Allow smtpd to work as a backup MX, relaying only to MXs with higher
priority in the DNS record. For example:

accept for domain "foo.org" relay backup "mx3.foo.org"

will relay mails for "foo.org" using only hosts with higher priority
(i.e. lower value) than "mx3.foo.org", which is supposed to be the
current server.

If the specified backup MX is not found in the DNS record, relaying
works as normal.

ok gilles@

Use TAILQ rather than array for mx list.

ok gilles@

dns sessions don't use lookup. no need to store them in a tree.

ok gilles@

coding style: replace all occurences of u_int* with uint*

ok eric@

- introduce stat_backend, an API for pluggable statistic backends
> statistics are no longer static structures in shared memory
> statistics are only set, smtpd never uses them in its logic
> each statistic is a key/value where key can be any (dynamic) string
- convert all uses of the former API to use the new one
- implement stat_ramstat that keeps non-persistent stats in ram structure

ok eric@, ok chl@

cleanup some old debug traces

ok gilles@ chl@

fix bogus permfail when no MX is defined on a valid domain.

ok gilles@

move to the new resolver implementation, with temporary glue to use
the relevant files from asr directly.

ok gilles@

improve readability

ok gilles@

Try to parse hostnames as IP addresses before resolving. This allows
relays to be given as IP address in the config file.

ok gilles@

Improve error reporting. Most errors during hostname lookup are
now correctly reported as temporary failures.

from Nathanael Rensen, tweaks by me.

Introduce a small set of functions to manage stat counters in a
simpler and hopefully saner way.

ok gilles@ chl@

Fix reporting of permanent/temporary failures for MX lookups.
Simplify code a bit while there.

ok gilles@

imsg.h requires sys/queue.h and sys/uio.h.

ok eric

move dns session specific structs and prototypes out of smtpd.h.

ok gilles@

the smtpd env is meant to be global, so do not pass it all around.

discussed with and ok gilles@

cleanups, cosmethic changes, functions that should be static are now static
no functionnal change

add stat counters for the lookup agent

ok gilles@

cleanup and simplification following the asr update.

- use a specific dispatch function for each type of query
- make the host handler work on a list of hosts by default (single host
queries are just a particular case) and use that to resolve the MX list
- various other code cleanup
- remove unused headers
- remove orphaned prototypes
- update copyright

ok gilles@

remove unused code now that reverse lookups are done through asr.

ok gilles@

make use the cname query interface from asr for reverse lookups

ok gilles@

add a function to factorize resetting of dnssession events.

ok gilles@

Fix the MX lookup process:

- the MX records were not always properly inserted into the sorted
array, which led to some MX being silently dropped.
- if an MX address could not be resolved, mail delivery would fail,
even though other valid MX exist for that domain. Now only report
the failure if no server address can be found at all.

grrrreeat gilles@

Tweak the asr API to make things a bit smoother on the user side.
Then asr_run() call now returns ASR_COND when a condition on a FD is
expected. The exact condition (readable or writeable) is specified in
the asr_result structure, along with the fd and timeout.

ok gilles@

use an index for iterating into the mx list.

ok gilles@

Simplify resolver initialization. This is done only once
since resolv.conf reloading is handled automatically by asr.

ok gilles@

make similar code .. more similar
ok gilles@

previous commit should have read:

when copying 'struct sockaddr' data, use sa_len not sizeof(struct sockaddr_in)
this fixes truncation of IPv6 addresses in the mail delivery path
ok gilles@

*** empty log message ***

If MX lookup fails, fallback to using the host itself. This has always been
the behavior but I introduced a regression when switching to ASR.

bug reported by jmc@, bugfix tested by jmc@ and I

use memcpy instead of a cast/deref dance that was reading past
the end of the buffer.

tested by and ok gilles@

replace the fork-based-non-blocking-resolver-hack by shiny async resolver
written by eric@. it is still experimental but still better than what we
had earlier so ... we'll improve in tree :)

diff by me with *lots* of help from eric@, tested by todd and I (and a
few people out there)

remove unused headers

a bit of .h cleanups, no functionnal change

we do dns resolutions in a separate process because we don't have an async
resolver. if we run scarce on resources and we cannot fork a separate dns
process or we cannot socketpair() tell the caller that we have a temporary
failure rather than issueing a fatal(). message will stay in queue and be
rescheduled later ...

bug reported and bugfix tested by Sacha El Masry <lists@devilray.eu>

force the dns buffers to be aligned using a union, until the retarded
"misalign strings on the stack" bug in gcc4 is fixed (even when that
is fixed this idiom is safer and quite common)
ok jacekm

check event_dispatch() return value

ok jacekm@

add missing header needed by signal()

ok gilles@

Fix previous. When configured to relay via IP address, MX lookup would fail
(NXDOMAIN), leading to a bounce. Precede the MX lookup with an attempt to
parse the relay as numeric string.

"reads ok" gilles@

Consider DNS lookups that result in NXDOMAIN to be a permanent failure.

ok gilles@ jacekm@

Introduce a 6yz status code, used internally to report permanent errors.
The 1yz and 6yz status codes are now removed prior to reporting the status
message in bounce messages, which provides an easy way to distinguish
between local and remote status messages. Initial diff from jacekm@

ok gilles@ jacekm@

imsg_get sets errno so use fatal instead of fatalx.

import some changes from portable smtpd to reduce the delta between both.
this commit contains mostly missing casts and cosmethic changes, do not
expect to build this anywhere but on OpenBSD, it does not contain any of
the portable glue.

Sync with relayd:
Stop pushing event handling in the imsg framework.
Instead, provide a small glue layer on top of both imsg and libevent.
This finally clearly separates event handling and imsg construction.

Sidetrack bonus: remove the mega-ugly hack of having a dummy imsg_event_add
stub in smtpctl.
ok jaceckm@

make smtpd's imsg lib ready, just like relayd and ospfd.
ok gilles@, jacekm@

Fix EV_READ/EV_WRITE testing inside IMSG handlers. Based on similar change
to the routing daemons by claudio@; ok gilles@

- New API to handle all DNS query types (A, MX, PTR) asynchronously.

- Improve RFC compliance: CNAMEs are resolved, equal preference MXs
are randomized, relaying via MX that has equal/lower preference
than local server is prevented, decision on when to treat domain
name as implicit MX is better.

ok gilles@

replace MAX* constants by sizeof where possible

ok jacekm@

If MX query fails due to DNS error, do not attempt more queries; ok gilles@

Fix a bug where list of 6 MXs or more was not handled correctly.
Pointed out by & ok sthen@, ok gilles@.

remove unnecessary includes; ok gilles@

- more err/errx -> fatal/fatalx, warn/warnx -> log_warn/log_warnx
contains bits based on an old diff from Jacek Masiulaniec and
other bits from me.

- in mxsort, fix type of loop counter, it will never be < 0 if it is
unsigned and when running out of luck it will cause the lookup
process to crash.

- move prototype to smtpd.h

insertion sort is faster than bubble sort. ok gilles

add a few missing id tags; there are a bunch of files, and developers
will probably miss this change when working on more important matters,
so it is probably better to sort them now. there is a risk of losing
the tags if a change needs to be reverted too.

written with excellent advice from jmc@

ok gilles@

smtpd is a smtp server implementation for OpenBSD. It is a work in progress
which still lacks many features. bringing it in tree will help working on it
more easily.

"at this stage it should go in" henning@, "move ahead" deraadt@

add required headers for smtpd.h and remove unnecessary ones in other files.

ok jung@

Implement server certificate validation in smtp(1).
Check certificate against MX name in smtpd(8) mta.

ok gilles@

treat NOTIMP as NO_DATA in response to MX query: fallback to hostname lookup
instead of bouncing the mail.

ok gilles@

Implement a generic interface to forward resolver queries to the lka
process. Use it for the reverse lookups required by smtp and mta.

Until now, DNS-related lookups were implemented using ad-hoc IMSGs
between the lka and other processes. It turns out to be confusing and
difficult to maintain/extend. So we want to replace this with a better
set of IMSGs matching the standard resolver interface.

ok gilles@

remove 'where' parameter from all x*() functions in utils.c, it doesn't
really help us with anything, propagate the change in codebase

ok millert@

Move unpack functions into a seperate file.

Required for upcoming 'smtpctl spf walk'.
Ok eric@ gilles@ millert@

typo; from Edgar Pettijohn

dns_lookup_host() needs to remove brackets and IPv6: prefix when receiving
a text representation otherwise getaddrinfo_async() will choke

ok eric@

remove unused variables

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

fix whitespace and indentation, by Kyle Milz

restrict address lookups to configured address families.

ok gilles@

(void) cast this strlcpy(), it cannot truncate

these strlcpy can't truncate, the copy is from a buffer to a buffer of same
size and the first buffer handles the truncation already

Merge the mda, mta and smtp processes into a single unprivileged
process managing message reception, delivery and transfer. Mostly
mechanical, but very intrusive as it required to rewamp all IMSG to
fix ambiguities.

with and ok gilles@

use asr helpers after libevent update.

asr API is now public

update after asr API update

Integrate necessary dns packet parsing helpers from asr.
They are not supposed to be exposed.

ok gilles@

field rename

bcopy -> memmove
bzero -> memset

When looking up a MX, parse the address if the domain is a "[ipaddr]" string.

%i -> %d in format strings

update after asr changes.

sync with OpenSMTPD 5.3.2

ok gilles@

remove params after API change.

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

fix after asr update

knf

ok gilles@

Cleanups and improvements:

* Log more events (especially client session) and use a better scheme
for that: each messages is prefixed with a token to easily identify
its class:
- info/warn/debug: general server messages
- smtp-in: smtp client connections
- relay: status update for relayed messages
- delivery: status update for local deliveries

* Implement "smtpctl monitor" to display updates of selected internal
counters.

* When reloading the on-disk queue at startup do not commit a message
if no envelope was submitted for that message.

* Remove unused stuff in the config parser.

ok gilles@

skip RR if type is not MX. Use hostname if the list of MX is empty
after the loop.

spotted by huku at grhack.net

ok gilles@

when requesting MX entries, the result can be appear in random orders.

the logic for inserting them in a lka session when acting as backup MX did
not take account for one specific case that could lead to an early exit
without smtpd getting a chance to detect the entry corrsponds to itself.

in such case, a backup MX woud try to connect to itself and bounce in the
loop detection code ... or it would sometimes work.

use xmalloc()/xcalloc() helpers

while there unify usage of log_trace() in ramstat_set()

ok gilles@ eric@

- introduce struct stat_value
- statistics can now have a type (counter, timestamp, timeval, timespec and
possibly others in the future)
- stat_increment() / stat_decrement() now take an increment/decrement value
and are at the moment only of type counter
- stat_set() now takes a stat_value
- provide helpers to convert raw values to stat_value

ok eric@, ok chl@

while at it fix a rq_queue_dump() call using a bogus timestamp in scheduler
ramqueue.

Allow smtpd to work as a backup MX, relaying only to MXs with higher
priority in the DNS record. For example:

accept for domain "foo.org" relay backup "mx3.foo.org"

will relay mails for "foo.org" using only hosts with higher priority
(i.e. lower value) than "mx3.foo.org", which is supposed to be the
current server.

If the specified backup MX is not found in the DNS record, relaying
works as normal.

ok gilles@

Use TAILQ rather than array for mx list.

ok gilles@

dns sessions don't use lookup. no need to store them in a tree.

ok gilles@

coding style: replace all occurences of u_int* with uint*

ok eric@

- introduce stat_backend, an API for pluggable statistic backends
> statistics are no longer static structures in shared memory
> statistics are only set, smtpd never uses them in its logic
> each statistic is a key/value where key can be any (dynamic) string
- convert all uses of the former API to use the new one
- implement stat_ramstat that keeps non-persistent stats in ram structure

ok eric@, ok chl@

cleanup some old debug traces

ok gilles@ chl@

fix bogus permfail when no MX is defined on a valid domain.

ok gilles@

move to the new resolver implementation, with temporary glue to use
the relevant files from asr directly.

ok gilles@

improve readability

ok gilles@

Try to parse hostnames as IP addresses before resolving. This allows
relays to be given as IP address in the config file.

ok gilles@

Improve error reporting. Most errors during hostname lookup are
now correctly reported as temporary failures.

from Nathanael Rensen, tweaks by me.

Introduce a small set of functions to manage stat counters in a
simpler and hopefully saner way.

ok gilles@ chl@

Fix reporting of permanent/temporary failures for MX lookups.
Simplify code a bit while there.

ok gilles@

imsg.h requires sys/queue.h and sys/uio.h.

ok eric

move dns session specific structs and prototypes out of smtpd.h.

ok gilles@

the smtpd env is meant to be global, so do not pass it all around.

discussed with and ok gilles@

cleanups, cosmethic changes, functions that should be static are now static
no functionnal change

add stat counters for the lookup agent

ok gilles@

cleanup and simplification following the asr update.

- use a specific dispatch function for each type of query
- make the host handler work on a list of hosts by default (single host
queries are just a particular case) and use that to resolve the MX list
- various other code cleanup
- remove unused headers
- remove orphaned prototypes
- update copyright

ok gilles@

remove unused code now that reverse lookups are done through asr.

ok gilles@

make use the cname query interface from asr for reverse lookups

ok gilles@

add a function to factorize resetting of dnssession events.

ok gilles@

Fix the MX lookup process:

- the MX records were not always properly inserted into the sorted
array, which led to some MX being silently dropped.
- if an MX address could not be resolved, mail delivery would fail,
even though other valid MX exist for that domain. Now only report
the failure if no server address can be found at all.

grrrreeat gilles@

Tweak the asr API to make things a bit smoother on the user side.
Then asr_run() call now returns ASR_COND when a condition on a FD is
expected. The exact condition (readable or writeable) is specified in
the asr_result structure, along with the fd and timeout.

ok gilles@

use an index for iterating into the mx list.

ok gilles@

Simplify resolver initialization. This is done only once
since resolv.conf reloading is handled automatically by asr.

ok gilles@

make similar code .. more similar
ok gilles@

previous commit should have read:

when copying 'struct sockaddr' data, use sa_len not sizeof(struct sockaddr_in)
this fixes truncation of IPv6 addresses in the mail delivery path
ok gilles@

*** empty log message ***

If MX lookup fails, fallback to using the host itself. This has always been
the behavior but I introduced a regression when switching to ASR.

bug reported by jmc@, bugfix tested by jmc@ and I

use memcpy instead of a cast/deref dance that was reading past
the end of the buffer.

tested by and ok gilles@

replace the fork-based-non-blocking-resolver-hack by shiny async resolver
written by eric@. it is still experimental but still better than what we
had earlier so ... we'll improve in tree :)

diff by me with *lots* of help from eric@, tested by todd and I (and a
few people out there)

remove unused headers

a bit of .h cleanups, no functionnal change

we do dns resolutions in a separate process because we don't have an async
resolver. if we run scarce on resources and we cannot fork a separate dns
process or we cannot socketpair() tell the caller that we have a temporary
failure rather than issueing a fatal(). message will stay in queue and be
rescheduled later ...

bug reported and bugfix tested by Sacha El Masry <lists@devilray.eu>

force the dns buffers to be aligned using a union, until the retarded
"misalign strings on the stack" bug in gcc4 is fixed (even when that
is fixed this idiom is safer and quite common)
ok jacekm

check event_dispatch() return value

ok jacekm@

add missing header needed by signal()

ok gilles@

Fix previous. When configured to relay via IP address, MX lookup would fail
(NXDOMAIN), leading to a bounce. Precede the MX lookup with an attempt to
parse the relay as numeric string.

"reads ok" gilles@

Consider DNS lookups that result in NXDOMAIN to be a permanent failure.

ok gilles@ jacekm@

Introduce a 6yz status code, used internally to report permanent errors.
The 1yz and 6yz status codes are now removed prior to reporting the status
message in bounce messages, which provides an easy way to distinguish
between local and remote status messages. Initial diff from jacekm@

ok gilles@ jacekm@

imsg_get sets errno so use fatal instead of fatalx.

import some changes from portable smtpd to reduce the delta between both.
this commit contains mostly missing casts and cosmethic changes, do not
expect to build this anywhere but on OpenBSD, it does not contain any of
the portable glue.

Sync with relayd:
Stop pushing event handling in the imsg framework.
Instead, provide a small glue layer on top of both imsg and libevent.
This finally clearly separates event handling and imsg construction.

Sidetrack bonus: remove the mega-ugly hack of having a dummy imsg_event_add
stub in smtpctl.
ok jaceckm@

make smtpd's imsg lib ready, just like relayd and ospfd.
ok gilles@, jacekm@

Fix EV_READ/EV_WRITE testing inside IMSG handlers. Based on similar change
to the routing daemons by claudio@; ok gilles@

- New API to handle all DNS query types (A, MX, PTR) asynchronously.

- Improve RFC compliance: CNAMEs are resolved, equal preference MXs
are randomized, relaying via MX that has equal/lower preference
than local server is prevented, decision on when to treat domain
name as implicit MX is better.

ok gilles@

replace MAX* constants by sizeof where possible

ok jacekm@

If MX query fails due to DNS error, do not attempt more queries; ok gilles@

Fix a bug where list of 6 MXs or more was not handled correctly.
Pointed out by & ok sthen@, ok gilles@.

remove unnecessary includes; ok gilles@

- more err/errx -> fatal/fatalx, warn/warnx -> log_warn/log_warnx
contains bits based on an old diff from Jacek Masiulaniec and
other bits from me.

- in mxsort, fix type of loop counter, it will never be < 0 if it is
unsigned and when running out of luck it will cause the lookup
process to crash.

- move prototype to smtpd.h

insertion sort is faster than bubble sort. ok gilles

add a few missing id tags; there are a bunch of files, and developers
will probably miss this change when working on more important matters,
so it is probably better to sort them now. there is a risk of losing
the tags if a change needs to be reverted too.

written with excellent advice from jmc@

ok gilles@

smtpd is a smtp server implementation for OpenBSD. It is a work in progress
which still lacks many features. bringing it in tree will help working on it
more easily.

"at this stage it should go in" henning@, "move ahead" deraadt@

Implement server certificate validation in smtp(1).
Check certificate against MX name in smtpd(8) mta.

ok gilles@

treat NOTIMP as NO_DATA in response to MX query: fallback to hostname lookup
instead of bouncing the mail.

ok gilles@

Implement a generic interface to forward resolver queries to the lka
process. Use it for the reverse lookups required by smtp and mta.

Until now, DNS-related lookups were implemented using ad-hoc IMSGs
between the lka and other processes. It turns out to be confusing and
difficult to maintain/extend. So we want to replace this with a better
set of IMSGs matching the standard resolver interface.

ok gilles@

remove 'where' parameter from all x*() functions in utils.c, it doesn't
really help us with anything, propagate the change in codebase

ok millert@

Move unpack functions into a seperate file.

Required for upcoming 'smtpctl spf walk'.
Ok eric@ gilles@ millert@

typo; from Edgar Pettijohn

dns_lookup_host() needs to remove brackets and IPv6: prefix when receiving
a text representation otherwise getaddrinfo_async() will choke

ok eric@

remove unused variables

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

fix whitespace and indentation, by Kyle Milz

restrict address lookups to configured address families.

ok gilles@

(void) cast this strlcpy(), it cannot truncate

these strlcpy can't truncate, the copy is from a buffer to a buffer of same
size and the first buffer handles the truncation already

Merge the mda, mta and smtp processes into a single unprivileged
process managing message reception, delivery and transfer. Mostly
mechanical, but very intrusive as it required to rewamp all IMSG to
fix ambiguities.

with and ok gilles@

use asr helpers after libevent update.

asr API is now public

update after asr API update

Integrate necessary dns packet parsing helpers from asr.
They are not supposed to be exposed.

ok gilles@

field rename

bcopy -> memmove
bzero -> memset

When looking up a MX, parse the address if the domain is a "[ipaddr]" string.

%i -> %d in format strings

update after asr changes.

sync with OpenSMTPD 5.3.2

ok gilles@

remove params after API change.

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

fix after asr update

knf

ok gilles@

Cleanups and improvements:

* Log more events (especially client session) and use a better scheme
for that: each messages is prefixed with a token to easily identify
its class:
- info/warn/debug: general server messages
- smtp-in: smtp client connections
- relay: status update for relayed messages
- delivery: status update for local deliveries

* Implement "smtpctl monitor" to display updates of selected internal
counters.

* When reloading the on-disk queue at startup do not commit a message
if no envelope was submitted for that message.

* Remove unused stuff in the config parser.

ok gilles@

skip RR if type is not MX. Use hostname if the list of MX is empty
after the loop.

spotted by huku at grhack.net

ok gilles@

when requesting MX entries, the result can be appear in random orders.

the logic for inserting them in a lka session when acting as backup MX did
not take account for one specific case that could lead to an early exit
without smtpd getting a chance to detect the entry corrsponds to itself.

in such case, a backup MX woud try to connect to itself and bounce in the
loop detection code ... or it would sometimes work.

use xmalloc()/xcalloc() helpers

while there unify usage of log_trace() in ramstat_set()

ok gilles@ eric@

- introduce struct stat_value
- statistics can now have a type (counter, timestamp, timeval, timespec and
possibly others in the future)
- stat_increment() / stat_decrement() now take an increment/decrement value
and are at the moment only of type counter
- stat_set() now takes a stat_value
- provide helpers to convert raw values to stat_value

ok eric@, ok chl@

while at it fix a rq_queue_dump() call using a bogus timestamp in scheduler
ramqueue.

Allow smtpd to work as a backup MX, relaying only to MXs with higher
priority in the DNS record. For example:

accept for domain "foo.org" relay backup "mx3.foo.org"

will relay mails for "foo.org" using only hosts with higher priority
(i.e. lower value) than "mx3.foo.org", which is supposed to be the
current server.

If the specified backup MX is not found in the DNS record, relaying
works as normal.

ok gilles@

Use TAILQ rather than array for mx list.

ok gilles@

dns sessions don't use lookup. no need to store them in a tree.

ok gilles@

coding style: replace all occurences of u_int* with uint*

ok eric@

- introduce stat_backend, an API for pluggable statistic backends
> statistics are no longer static structures in shared memory
> statistics are only set, smtpd never uses them in its logic
> each statistic is a key/value where key can be any (dynamic) string
- convert all uses of the former API to use the new one
- implement stat_ramstat that keeps non-persistent stats in ram structure

ok eric@, ok chl@

cleanup some old debug traces

ok gilles@ chl@

fix bogus permfail when no MX is defined on a valid domain.

ok gilles@

move to the new resolver implementation, with temporary glue to use
the relevant files from asr directly.

ok gilles@

improve readability

ok gilles@

Try to parse hostnames as IP addresses before resolving. This allows
relays to be given as IP address in the config file.

ok gilles@

Improve error reporting. Most errors during hostname lookup are
now correctly reported as temporary failures.

from Nathanael Rensen, tweaks by me.

Introduce a small set of functions to manage stat counters in a
simpler and hopefully saner way.

ok gilles@ chl@

Fix reporting of permanent/temporary failures for MX lookups.
Simplify code a bit while there.

ok gilles@

imsg.h requires sys/queue.h and sys/uio.h.

ok eric

move dns session specific structs and prototypes out of smtpd.h.

ok gilles@

the smtpd env is meant to be global, so do not pass it all around.

discussed with and ok gilles@

cleanups, cosmethic changes, functions that should be static are now static
no functionnal change

add stat counters for the lookup agent

ok gilles@

cleanup and simplification following the asr update.

- use a specific dispatch function for each type of query
- make the host handler work on a list of hosts by default (single host
queries are just a particular case) and use that to resolve the MX list
- various other code cleanup
- remove unused headers
- remove orphaned prototypes
- update copyright

ok gilles@

remove unused code now that reverse lookups are done through asr.

ok gilles@

make use the cname query interface from asr for reverse lookups

ok gilles@

add a function to factorize resetting of dnssession events.

ok gilles@

Fix the MX lookup process:

- the MX records were not always properly inserted into the sorted
array, which led to some MX being silently dropped.
- if an MX address could not be resolved, mail delivery would fail,
even though other valid MX exist for that domain. Now only report
the failure if no server address can be found at all.

grrrreeat gilles@

Tweak the asr API to make things a bit smoother on the user side.
Then asr_run() call now returns ASR_COND when a condition on a FD is
expected. The exact condition (readable or writeable) is specified in
the asr_result structure, along with the fd and timeout.

ok gilles@

use an index for iterating into the mx list.

ok gilles@

Simplify resolver initialization. This is done only once
since resolv.conf reloading is handled automatically by asr.

ok gilles@

make similar code .. more similar
ok gilles@

previous commit should have read:

when copying 'struct sockaddr' data, use sa_len not sizeof(struct sockaddr_in)
this fixes truncation of IPv6 addresses in the mail delivery path
ok gilles@

*** empty log message ***

If MX lookup fails, fallback to using the host itself. This has always been
the behavior but I introduced a regression when switching to ASR.

bug reported by jmc@, bugfix tested by jmc@ and I

use memcpy instead of a cast/deref dance that was reading past
the end of the buffer.

tested by and ok gilles@

replace the fork-based-non-blocking-resolver-hack by shiny async resolver
written by eric@. it is still experimental but still better than what we
had earlier so ... we'll improve in tree :)

diff by me with *lots* of help from eric@, tested by todd and I (and a
few people out there)

remove unused headers

a bit of .h cleanups, no functionnal change

we do dns resolutions in a separate process because we don't have an async
resolver. if we run scarce on resources and we cannot fork a separate dns
process or we cannot socketpair() tell the caller that we have a temporary
failure rather than issueing a fatal(). message will stay in queue and be
rescheduled later ...

bug reported and bugfix tested by Sacha El Masry <lists@devilray.eu>

force the dns buffers to be aligned using a union, until the retarded
"misalign strings on the stack" bug in gcc4 is fixed (even when that
is fixed this idiom is safer and quite common)
ok jacekm

check event_dispatch() return value

ok jacekm@

add missing header needed by signal()

ok gilles@

Fix previous. When configured to relay via IP address, MX lookup would fail
(NXDOMAIN), leading to a bounce. Precede the MX lookup with an attempt to
parse the relay as numeric string.

"reads ok" gilles@

Consider DNS lookups that result in NXDOMAIN to be a permanent failure.

ok gilles@ jacekm@

Introduce a 6yz status code, used internally to report permanent errors.
The 1yz and 6yz status codes are now removed prior to reporting the status
message in bounce messages, which provides an easy way to distinguish
between local and remote status messages. Initial diff from jacekm@

ok gilles@ jacekm@

imsg_get sets errno so use fatal instead of fatalx.

import some changes from portable smtpd to reduce the delta between both.
this commit contains mostly missing casts and cosmethic changes, do not
expect to build this anywhere but on OpenBSD, it does not contain any of
the portable glue.

Sync with relayd:
Stop pushing event handling in the imsg framework.
Instead, provide a small glue layer on top of both imsg and libevent.
This finally clearly separates event handling and imsg construction.

Sidetrack bonus: remove the mega-ugly hack of having a dummy imsg_event_add
stub in smtpctl.
ok jaceckm@

make smtpd's imsg lib ready, just like relayd and ospfd.
ok gilles@, jacekm@

Fix EV_READ/EV_WRITE testing inside IMSG handlers. Based on similar change
to the routing daemons by claudio@; ok gilles@

- New API to handle all DNS query types (A, MX, PTR) asynchronously.

- Improve RFC compliance: CNAMEs are resolved, equal preference MXs
are randomized, relaying via MX that has equal/lower preference
than local server is prevented, decision on when to treat domain
name as implicit MX is better.

ok gilles@

replace MAX* constants by sizeof where possible

ok jacekm@

If MX query fails due to DNS error, do not attempt more queries; ok gilles@

Fix a bug where list of 6 MXs or more was not handled correctly.
Pointed out by & ok sthen@, ok gilles@.

remove unnecessary includes; ok gilles@

- more err/errx -> fatal/fatalx, warn/warnx -> log_warn/log_warnx
contains bits based on an old diff from Jacek Masiulaniec and
other bits from me.

- in mxsort, fix type of loop counter, it will never be < 0 if it is
unsigned and when running out of luck it will cause the lookup
process to crash.

- move prototype to smtpd.h

insertion sort is faster than bubble sort. ok gilles

add a few missing id tags; there are a bunch of files, and developers
will probably miss this change when working on more important matters,
so it is probably better to sort them now. there is a risk of losing
the tags if a change needs to be reverted too.

written with excellent advice from jmc@

ok gilles@

smtpd is a smtp server implementation for OpenBSD. It is a work in progress
which still lacks many features. bringing it in tree will help working on it
more easily.

"at this stage it should go in" henning@, "move ahead" deraadt@

treat NOTIMP as NO_DATA in response to MX query: fallback to hostname lookup
instead of bouncing the mail.

ok gilles@

Implement a generic interface to forward resolver queries to the lka
process. Use it for the reverse lookups required by smtp and mta.

Until now, DNS-related lookups were implemented using ad-hoc IMSGs
between the lka and other processes. It turns out to be confusing and
difficult to maintain/extend. So we want to replace this with a better
set of IMSGs matching the standard resolver interface.

ok gilles@

remove 'where' parameter from all x*() functions in utils.c, it doesn't
really help us with anything, propagate the change in codebase

ok millert@

Move unpack functions into a seperate file.

Required for upcoming 'smtpctl spf walk'.
Ok eric@ gilles@ millert@

typo; from Edgar Pettijohn

dns_lookup_host() needs to remove brackets and IPv6: prefix when receiving
a text representation otherwise getaddrinfo_async() will choke

ok eric@

remove unused variables

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

fix whitespace and indentation, by Kyle Milz

restrict address lookups to configured address families.

ok gilles@

(void) cast this strlcpy(), it cannot truncate

these strlcpy can't truncate, the copy is from a buffer to a buffer of same
size and the first buffer handles the truncation already

Merge the mda, mta and smtp processes into a single unprivileged
process managing message reception, delivery and transfer. Mostly
mechanical, but very intrusive as it required to rewamp all IMSG to
fix ambiguities.

with and ok gilles@

use asr helpers after libevent update.

asr API is now public

update after asr API update

Integrate necessary dns packet parsing helpers from asr.
They are not supposed to be exposed.

ok gilles@

field rename

bcopy -> memmove
bzero -> memset

When looking up a MX, parse the address if the domain is a "[ipaddr]" string.

%i -> %d in format strings

update after asr changes.

sync with OpenSMTPD 5.3.2

ok gilles@

remove params after API change.

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

fix after asr update

knf

ok gilles@

Cleanups and improvements:

* Log more events (especially client session) and use a better scheme
for that: each messages is prefixed with a token to easily identify
its class:
- info/warn/debug: general server messages
- smtp-in: smtp client connections
- relay: status update for relayed messages
- delivery: status update for local deliveries

* Implement "smtpctl monitor" to display updates of selected internal
counters.

* When reloading the on-disk queue at startup do not commit a message
if no envelope was submitted for that message.

* Remove unused stuff in the config parser.

ok gilles@

skip RR if type is not MX. Use hostname if the list of MX is empty
after the loop.

spotted by huku at grhack.net

ok gilles@

when requesting MX entries, the result can be appear in random orders.

the logic for inserting them in a lka session when acting as backup MX did
not take account for one specific case that could lead to an early exit
without smtpd getting a chance to detect the entry corrsponds to itself.

in such case, a backup MX woud try to connect to itself and bounce in the
loop detection code ... or it would sometimes work.

use xmalloc()/xcalloc() helpers

while there unify usage of log_trace() in ramstat_set()

ok gilles@ eric@

- introduce struct stat_value
- statistics can now have a type (counter, timestamp, timeval, timespec and
possibly others in the future)
- stat_increment() / stat_decrement() now take an increment/decrement value
and are at the moment only of type counter
- stat_set() now takes a stat_value
- provide helpers to convert raw values to stat_value

ok eric@, ok chl@

while at it fix a rq_queue_dump() call using a bogus timestamp in scheduler
ramqueue.

Allow smtpd to work as a backup MX, relaying only to MXs with higher
priority in the DNS record. For example:

accept for domain "foo.org" relay backup "mx3.foo.org"

will relay mails for "foo.org" using only hosts with higher priority
(i.e. lower value) than "mx3.foo.org", which is supposed to be the
current server.

If the specified backup MX is not found in the DNS record, relaying
works as normal.

ok gilles@

Use TAILQ rather than array for mx list.

ok gilles@

dns sessions don't use lookup. no need to store them in a tree.

ok gilles@

coding style: replace all occurences of u_int* with uint*

ok eric@

- introduce stat_backend, an API for pluggable statistic backends
> statistics are no longer static structures in shared memory
> statistics are only set, smtpd never uses them in its logic
> each statistic is a key/value where key can be any (dynamic) string
- convert all uses of the former API to use the new one
- implement stat_ramstat that keeps non-persistent stats in ram structure

ok eric@, ok chl@

cleanup some old debug traces

ok gilles@ chl@

fix bogus permfail when no MX is defined on a valid domain.

ok gilles@

move to the new resolver implementation, with temporary glue to use
the relevant files from asr directly.

ok gilles@

improve readability

ok gilles@

Try to parse hostnames as IP addresses before resolving. This allows
relays to be given as IP address in the config file.

ok gilles@

Improve error reporting. Most errors during hostname lookup are
now correctly reported as temporary failures.

from Nathanael Rensen, tweaks by me.

Introduce a small set of functions to manage stat counters in a
simpler and hopefully saner way.

ok gilles@ chl@

Fix reporting of permanent/temporary failures for MX lookups.
Simplify code a bit while there.

ok gilles@

imsg.h requires sys/queue.h and sys/uio.h.

ok eric

move dns session specific structs and prototypes out of smtpd.h.

ok gilles@

the smtpd env is meant to be global, so do not pass it all around.

discussed with and ok gilles@

cleanups, cosmethic changes, functions that should be static are now static
no functionnal change

add stat counters for the lookup agent

ok gilles@

cleanup and simplification following the asr update.

- use a specific dispatch function for each type of query
- make the host handler work on a list of hosts by default (single host
queries are just a particular case) and use that to resolve the MX list
- various other code cleanup
- remove unused headers
- remove orphaned prototypes
- update copyright

ok gilles@

remove unused code now that reverse lookups are done through asr.

ok gilles@

make use the cname query interface from asr for reverse lookups

ok gilles@

add a function to factorize resetting of dnssession events.

ok gilles@

Fix the MX lookup process:

- the MX records were not always properly inserted into the sorted
array, which led to some MX being silently dropped.
- if an MX address could not be resolved, mail delivery would fail,
even though other valid MX exist for that domain. Now only report
the failure if no server address can be found at all.

grrrreeat gilles@

Tweak the asr API to make things a bit smoother on the user side.
Then asr_run() call now returns ASR_COND when a condition on a FD is
expected. The exact condition (readable or writeable) is specified in
the asr_result structure, along with the fd and timeout.

ok gilles@

use an index for iterating into the mx list.

ok gilles@