add required headers for smtpd.h and remove unnecessary ones in other files.

ok jung@

replaces calls to err(3)/errx(3) with fatal()/fatalx() from log.c
for code that runs in the daemon.

ok florian@ millert@

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

Improve the scheduler, better and simpler.

- Get rid of the scheduler_batch structure. The scheduler can now return
envelopes of different types in a single run, interlacing them to avoid
batch effects.

- Ask for an acknowledgement from the queue when removing or expiring
an envelope to benefit from the inflight envelope limitation mechanism.
This ensures that the scheduler always keeps sending envelopes at a rate
that the queue can sustain in all cases.

- Limit the number of envelopes in a holdq. When a holdq is full,
new envelopes are put back in the pending queue instead, with a
shorter retry time.

- Plumbing for proc-ified schedulers.

imsg version bump. smtpctl stop before updating.

ok gilles@

internal improvements and cleanups

- get rid of the whole penalty thing for failed envelopes in the mta and scheduler.
- do not disable routes on smtp errors
- try to schedule all types of envelopes on each scheduler frame.

Rework the mda and scheduler to use the holdq mechanism instead of
tempfail for limiting the number of pending deliveries to the same
user. This allows to reach optimal delivery time even in case of
burst, while keeping the number of inflight envelopes low.

Implement a feedback mechanism which allows the mta to "hold" envelopes
in the scheduler when it has too many tasks for a given relay. The
envelopes are put on a wait queue, and are not scheduled again until
the mta "releases" some envelopes from that queue.

It prevents from having too many inflight envelopes, which are out of reach
for the admin.

Implement a scheduler_proc backend

scheduler improvements:
- implement suspend/resume scheduling for individual envelopes or message,
with the associated smtpctl commands.
- allow the mta to request immediate scheduling of an envelope.
- on temporary failures a penalty can be given to further delay the next try.

sync with OpenSMTPD 5.3.2

ok gilles@

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

replaces calls to err(3)/errx(3) with fatal()/fatalx() from log.c
for code that runs in the daemon.

ok florian@ millert@

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

Improve the scheduler, better and simpler.

- Get rid of the scheduler_batch structure. The scheduler can now return
envelopes of different types in a single run, interlacing them to avoid
batch effects.

- Ask for an acknowledgement from the queue when removing or expiring
an envelope to benefit from the inflight envelope limitation mechanism.
This ensures that the scheduler always keeps sending envelopes at a rate
that the queue can sustain in all cases.

- Limit the number of envelopes in a holdq. When a holdq is full,
new envelopes are put back in the pending queue instead, with a
shorter retry time.

- Plumbing for proc-ified schedulers.

imsg version bump. smtpctl stop before updating.

ok gilles@

internal improvements and cleanups

- get rid of the whole penalty thing for failed envelopes in the mta and scheduler.
- do not disable routes on smtp errors
- try to schedule all types of envelopes on each scheduler frame.

Rework the mda and scheduler to use the holdq mechanism instead of
tempfail for limiting the number of pending deliveries to the same
user. This allows to reach optimal delivery time even in case of
burst, while keeping the number of inflight envelopes low.

Implement a feedback mechanism which allows the mta to "hold" envelopes
in the scheduler when it has too many tasks for a given relay. The
envelopes are put on a wait queue, and are not scheduled again until
the mta "releases" some envelopes from that queue.

It prevents from having too many inflight envelopes, which are out of reach
for the admin.

Implement a scheduler_proc backend

scheduler improvements:
- implement suspend/resume scheduling for individual envelopes or message,
with the associated smtpctl commands.
- allow the mta to request immediate scheduling of an envelope.
- on temporary failures a penalty can be given to further delay the next try.

sync with OpenSMTPD 5.3.2

ok gilles@

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

Improve the scheduler, better and simpler.

- Get rid of the scheduler_batch structure. The scheduler can now return
envelopes of different types in a single run, interlacing them to avoid
batch effects.

- Ask for an acknowledgement from the queue when removing or expiring
an envelope to benefit from the inflight envelope limitation mechanism.
This ensures that the scheduler always keeps sending envelopes at a rate
that the queue can sustain in all cases.

- Limit the number of envelopes in a holdq. When a holdq is full,
new envelopes are put back in the pending queue instead, with a
shorter retry time.

- Plumbing for proc-ified schedulers.

imsg version bump. smtpctl stop before updating.

ok gilles@

internal improvements and cleanups

- get rid of the whole penalty thing for failed envelopes in the mta and scheduler.
- do not disable routes on smtp errors
- try to schedule all types of envelopes on each scheduler frame.

Rework the mda and scheduler to use the holdq mechanism instead of
tempfail for limiting the number of pending deliveries to the same
user. This allows to reach optimal delivery time even in case of
burst, while keeping the number of inflight envelopes low.

Implement a feedback mechanism which allows the mta to "hold" envelopes
in the scheduler when it has too many tasks for a given relay. The
envelopes are put on a wait queue, and are not scheduled again until
the mta "releases" some envelopes from that queue.

It prevents from having too many inflight envelopes, which are out of reach
for the admin.

Implement a scheduler_proc backend

scheduler improvements:
- implement suspend/resume scheduling for individual envelopes or message,
with the associated smtpctl commands.
- allow the mta to request immediate scheduling of an envelope.
- on temporary failures a penalty can be given to further delay the next try.

sync with OpenSMTPD 5.3.2

ok gilles@

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@