actually honour the services supported by the proc tables

ok gilles@

change the smtpd table protocol

Using imsg for the "proc" table (external programs) has proven quite
painful in practice since a lot of smtpd internals (structs, enums,
etc..) have to be kept in sync with the various tables implementations.

Instead, a filter-like protocol for tables decouples the implementations
and allows to write and test tables easily.

The new text-based transport protocol is documented in the (added)
smtpd-tables(7) manpage.

The old imsg protocol is no longer supported and existing tables have to
be converted. In particular, users of opensmtpd-extras tables will need
install the new opensmtpd-table-* packages.

With lots of suggestions and improvements from gilles and a tweak
from Philipp (philipp+openbsd [at] bureaucracy [dot] de), thanks!

ok gilles

fix IPv6 addresses table lookups

Rework parse_sockaddr() to not reach inet_pton() with a brace-wrapped
IPv6 address.

Issue reported by Kirill Miazine.
ok millert@

add required headers for smtpd.h and remove unnecessary ones in other files.

ok jung@

Use regfree after we're done with preg.

From gilles@

report errno in debug logs and other minor cleanups.

ok gilles@

introduce table_dump() and tweak format

ok gilles@

simplify code

ok gilles@

remove the tag workaround for table_create() and table_find(),
now that static tables handle their updates internally.

ok gilles@

remove unused members in struct table

table_fetch() always expect a valid dst pointer.

ok gilles@

introduce dump() and add() table methods, only implemented for static tables.

ok gilles@

pass the table pointer to the lookup()/fecth() methods

ok gilles@

change the close() method to take the table pointer

ok gilles

Make the backend open method return an int to report success.
The implementation is responsible for setting the handle pointer
as needed.

ok gilles@

reorder parameters for consistency

introduce a table_match() function to check for a key in a table

ok gilles@

get rid of the unused dict argument in table lookup and fetch api.

ok gilles@

move the table backend name in the backend struct.
remove unused function.

ok gilles@

Simplify the table backend interface: lookup results are returned
as strings, and parsing is handled by the upper layer.

ok gilles@

since we already support regex lookups in tables for builtin filters, let's
also support regex lookups in match rule criterias performing table lookups

ok millert@

add REG_NOSUB to flags in regcompile

suggested by jung@

introduce K_REGEX table type and table_regex_match(), unused for now

ok eric@

rework the table API so that it takes a struct smtpd * context in parameter
of functions creating, looking up or destroying tables.

this is a first step in cleaning up parse.y so it doesn't have side effects
outside of parse_config(), bringing nothing but making code cleaner.

ok millert@

remove 'where' parameter from all x*() functions in utils.c, it doesn't
really help us with anything, propagate the change in codebase

ok millert@

missing case

ok gilles@

no need to parse and dump the relayhost in the lookup process.

ok gilles@

zero memory before calling text_to_relayhost()

fix issue reported by semarie@

ok gilles@

switch smtpd to new grammar

ok eric@

in function used for tracing, display unknown lookup types as "???"

diff from Wolf480pl

switch to /usr/local/libexec when looking for -extras and drop loop iterating
paths

this effectively reverts table.c r1.21 which was mainly introduced for a smooth
transition in -current

ok gilles

remove spaces after '!'

no binary change

ok millert

when looking up tables, start in /usr/local/libexec before /usr/libexec, so
ports/packages can be installed in the proper place

ok jung@

allow table API to lookup for mailaddr mappings

ok sunil@, ok jung@

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

recipient and sender lists now support the user+TAG notation, allowing
among other things a secondary MX to filter recipients to be relayed to a
primary MX even if they are using tags. there are other nice things to do
with that feature, use your imagination.

tested and ok florian@, tested by several users for a few days too

Update the table API: lookup functions can take an optional parameters
dictionnary (currently not set). While there, add a helper for forking
external backends, and remove unused table functions.

ok gilles@

stop casting sizeof to int

these snprintf() calls can't possibly truncate because they copy data from
buffers that are already protected against truncation and that do not
exceed the destination buffer size when copied together ...

however, i think we should add checks here too because it'll help us catch
errors in table backends when adding new ones if we miss a truncation check
there.

(void) cast strlcat() and snprintf() that cannot truncate

be a bit more strict with an strlcat() truncation by causing it to fail in
table_create() instead of later in parse.y - in both cases, this would
cause smtpd to fatal() at startup if a table has a config file too large

bcopy -> memmove
bzero -> memset

allow subdomain matching in mailaddr tables

change dict_poproot() prototype: do not take key placeholder parameter as
it can't work that way.

accept credentials formated as "<user> <passwd>"

fix parsing of inet6 addresses when prefixed with "IPv6:"

%i -> %d in format strings

Update ldap and sqlite table backends and provide them as external backends.

Add a table_proc backend for delegating table lookups to another
process. Stop building experimental table_sqlite and table_ldap as
they will be provided as external backends.

lowercase the key when adding an entry to a static table, since the
lookups are done on lowercase strings. warn about duplicates entries
and avoid to leak memory.

ok gilles@

sync with OpenSMTPD 5.3.2

ok gilles@

unbreak broken smtpctl table update

fix by eric and I, ok eric@

- handle getaddrinfo() error as LKA_TEMPFAIL
- handle getsockname() error in smtp_connected()
- accept '/' as part of user-part, expand to ':' as done by qmail
- fix wrong check in mda leading to bogus Return-Path header
- fix aliases parsing when there's a white space between key and separator
- some cosmethic cleanup

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

change the smtpd table protocol

Using imsg for the "proc" table (external programs) has proven quite
painful in practice since a lot of smtpd internals (structs, enums,
etc..) have to be kept in sync with the various tables implementations.

Instead, a filter-like protocol for tables decouples the implementations
and allows to write and test tables easily.

The new text-based transport protocol is documented in the (added)
smtpd-tables(7) manpage.

The old imsg protocol is no longer supported and existing tables have to
be converted. In particular, users of opensmtpd-extras tables will need
install the new opensmtpd-table-* packages.

With lots of suggestions and improvements from gilles and a tweak
from Philipp (philipp+openbsd [at] bureaucracy [dot] de), thanks!

ok gilles

fix IPv6 addresses table lookups

Rework parse_sockaddr() to not reach inet_pton() with a brace-wrapped
IPv6 address.

Issue reported by Kirill Miazine.
ok millert@

add required headers for smtpd.h and remove unnecessary ones in other files.

ok jung@

Use regfree after we're done with preg.

From gilles@

report errno in debug logs and other minor cleanups.

ok gilles@

introduce table_dump() and tweak format

ok gilles@

simplify code

ok gilles@

remove the tag workaround for table_create() and table_find(),
now that static tables handle their updates internally.

ok gilles@

remove unused members in struct table

table_fetch() always expect a valid dst pointer.

ok gilles@

introduce dump() and add() table methods, only implemented for static tables.

ok gilles@

pass the table pointer to the lookup()/fecth() methods

ok gilles@

change the close() method to take the table pointer

ok gilles

Make the backend open method return an int to report success.
The implementation is responsible for setting the handle pointer
as needed.

ok gilles@

reorder parameters for consistency

introduce a table_match() function to check for a key in a table

ok gilles@

get rid of the unused dict argument in table lookup and fetch api.

ok gilles@

move the table backend name in the backend struct.
remove unused function.

ok gilles@

Simplify the table backend interface: lookup results are returned
as strings, and parsing is handled by the upper layer.

ok gilles@

since we already support regex lookups in tables for builtin filters, let's
also support regex lookups in match rule criterias performing table lookups

ok millert@

add REG_NOSUB to flags in regcompile

suggested by jung@

introduce K_REGEX table type and table_regex_match(), unused for now

ok eric@

rework the table API so that it takes a struct smtpd * context in parameter
of functions creating, looking up or destroying tables.

this is a first step in cleaning up parse.y so it doesn't have side effects
outside of parse_config(), bringing nothing but making code cleaner.

ok millert@

remove 'where' parameter from all x*() functions in utils.c, it doesn't
really help us with anything, propagate the change in codebase

ok millert@

missing case

ok gilles@

no need to parse and dump the relayhost in the lookup process.

ok gilles@

zero memory before calling text_to_relayhost()

fix issue reported by semarie@

ok gilles@

switch smtpd to new grammar

ok eric@

in function used for tracing, display unknown lookup types as "???"

diff from Wolf480pl

switch to /usr/local/libexec when looking for -extras and drop loop iterating
paths

this effectively reverts table.c r1.21 which was mainly introduced for a smooth
transition in -current

ok gilles

remove spaces after '!'

no binary change

ok millert

when looking up tables, start in /usr/local/libexec before /usr/libexec, so
ports/packages can be installed in the proper place

ok jung@

allow table API to lookup for mailaddr mappings

ok sunil@, ok jung@

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

recipient and sender lists now support the user+TAG notation, allowing
among other things a secondary MX to filter recipients to be relayed to a
primary MX even if they are using tags. there are other nice things to do
with that feature, use your imagination.

tested and ok florian@, tested by several users for a few days too

Update the table API: lookup functions can take an optional parameters
dictionnary (currently not set). While there, add a helper for forking
external backends, and remove unused table functions.

ok gilles@

stop casting sizeof to int

these snprintf() calls can't possibly truncate because they copy data from
buffers that are already protected against truncation and that do not
exceed the destination buffer size when copied together ...

however, i think we should add checks here too because it'll help us catch
errors in table backends when adding new ones if we miss a truncation check
there.

(void) cast strlcat() and snprintf() that cannot truncate

be a bit more strict with an strlcat() truncation by causing it to fail in
table_create() instead of later in parse.y - in both cases, this would
cause smtpd to fatal() at startup if a table has a config file too large

bcopy -> memmove
bzero -> memset

allow subdomain matching in mailaddr tables

change dict_poproot() prototype: do not take key placeholder parameter as
it can't work that way.

accept credentials formated as "<user> <passwd>"

fix parsing of inet6 addresses when prefixed with "IPv6:"

%i -> %d in format strings

Update ldap and sqlite table backends and provide them as external backends.

Add a table_proc backend for delegating table lookups to another
process. Stop building experimental table_sqlite and table_ldap as
they will be provided as external backends.

lowercase the key when adding an entry to a static table, since the
lookups are done on lowercase strings. warn about duplicates entries
and avoid to leak memory.

ok gilles@

sync with OpenSMTPD 5.3.2

ok gilles@

unbreak broken smtpctl table update

fix by eric and I, ok eric@

- handle getaddrinfo() error as LKA_TEMPFAIL
- handle getsockname() error in smtp_connected()
- accept '/' as part of user-part, expand to ':' as done by qmail
- fix wrong check in mda leading to bogus Return-Path header
- fix aliases parsing when there's a white space between key and separator
- some cosmethic cleanup

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

fix IPv6 addresses table lookups

Rework parse_sockaddr() to not reach inet_pton() with a brace-wrapped
IPv6 address.

Issue reported by Kirill Miazine.
ok millert@

add required headers for smtpd.h and remove unnecessary ones in other files.

ok jung@

Use regfree after we're done with preg.

From gilles@

report errno in debug logs and other minor cleanups.

ok gilles@

introduce table_dump() and tweak format

ok gilles@

simplify code

ok gilles@

remove the tag workaround for table_create() and table_find(),
now that static tables handle their updates internally.

ok gilles@

remove unused members in struct table

table_fetch() always expect a valid dst pointer.

ok gilles@

introduce dump() and add() table methods, only implemented for static tables.

ok gilles@

pass the table pointer to the lookup()/fecth() methods

ok gilles@

change the close() method to take the table pointer

ok gilles

Make the backend open method return an int to report success.
The implementation is responsible for setting the handle pointer
as needed.

ok gilles@

reorder parameters for consistency

introduce a table_match() function to check for a key in a table

ok gilles@

get rid of the unused dict argument in table lookup and fetch api.

ok gilles@

move the table backend name in the backend struct.
remove unused function.

ok gilles@

Simplify the table backend interface: lookup results are returned
as strings, and parsing is handled by the upper layer.

ok gilles@

since we already support regex lookups in tables for builtin filters, let's
also support regex lookups in match rule criterias performing table lookups

ok millert@

add REG_NOSUB to flags in regcompile

suggested by jung@

introduce K_REGEX table type and table_regex_match(), unused for now

ok eric@

rework the table API so that it takes a struct smtpd * context in parameter
of functions creating, looking up or destroying tables.

this is a first step in cleaning up parse.y so it doesn't have side effects
outside of parse_config(), bringing nothing but making code cleaner.

ok millert@

remove 'where' parameter from all x*() functions in utils.c, it doesn't
really help us with anything, propagate the change in codebase

ok millert@

missing case

ok gilles@

no need to parse and dump the relayhost in the lookup process.

ok gilles@

zero memory before calling text_to_relayhost()

fix issue reported by semarie@

ok gilles@

switch smtpd to new grammar

ok eric@

in function used for tracing, display unknown lookup types as "???"

diff from Wolf480pl

switch to /usr/local/libexec when looking for -extras and drop loop iterating
paths

this effectively reverts table.c r1.21 which was mainly introduced for a smooth
transition in -current

ok gilles

remove spaces after '!'

no binary change

ok millert

when looking up tables, start in /usr/local/libexec before /usr/libexec, so
ports/packages can be installed in the proper place

ok jung@

allow table API to lookup for mailaddr mappings

ok sunil@, ok jung@

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

recipient and sender lists now support the user+TAG notation, allowing
among other things a secondary MX to filter recipients to be relayed to a
primary MX even if they are using tags. there are other nice things to do
with that feature, use your imagination.

tested and ok florian@, tested by several users for a few days too

Update the table API: lookup functions can take an optional parameters
dictionnary (currently not set). While there, add a helper for forking
external backends, and remove unused table functions.

ok gilles@

stop casting sizeof to int

these snprintf() calls can't possibly truncate because they copy data from
buffers that are already protected against truncation and that do not
exceed the destination buffer size when copied together ...

however, i think we should add checks here too because it'll help us catch
errors in table backends when adding new ones if we miss a truncation check
there.

(void) cast strlcat() and snprintf() that cannot truncate

be a bit more strict with an strlcat() truncation by causing it to fail in
table_create() instead of later in parse.y - in both cases, this would
cause smtpd to fatal() at startup if a table has a config file too large

bcopy -> memmove
bzero -> memset

allow subdomain matching in mailaddr tables

change dict_poproot() prototype: do not take key placeholder parameter as
it can't work that way.

accept credentials formated as "<user> <passwd>"

fix parsing of inet6 addresses when prefixed with "IPv6:"

%i -> %d in format strings

Update ldap and sqlite table backends and provide them as external backends.

Add a table_proc backend for delegating table lookups to another
process. Stop building experimental table_sqlite and table_ldap as
they will be provided as external backends.

lowercase the key when adding an entry to a static table, since the
lookups are done on lowercase strings. warn about duplicates entries
and avoid to leak memory.

ok gilles@

sync with OpenSMTPD 5.3.2

ok gilles@

unbreak broken smtpctl table update

fix by eric and I, ok eric@

- handle getaddrinfo() error as LKA_TEMPFAIL
- handle getsockname() error in smtp_connected()
- accept '/' as part of user-part, expand to ':' as done by qmail
- fix wrong check in mda leading to bogus Return-Path header
- fix aliases parsing when there's a white space between key and separator
- some cosmethic cleanup

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

add required headers for smtpd.h and remove unnecessary ones in other files.

ok jung@

Use regfree after we're done with preg.

From gilles@

report errno in debug logs and other minor cleanups.

ok gilles@

introduce table_dump() and tweak format

ok gilles@

simplify code

ok gilles@

remove the tag workaround for table_create() and table_find(),
now that static tables handle their updates internally.

ok gilles@

remove unused members in struct table

table_fetch() always expect a valid dst pointer.

ok gilles@

introduce dump() and add() table methods, only implemented for static tables.

ok gilles@

pass the table pointer to the lookup()/fecth() methods

ok gilles@

change the close() method to take the table pointer

ok gilles

Make the backend open method return an int to report success.
The implementation is responsible for setting the handle pointer
as needed.

ok gilles@

reorder parameters for consistency

introduce a table_match() function to check for a key in a table

ok gilles@

get rid of the unused dict argument in table lookup and fetch api.

ok gilles@

move the table backend name in the backend struct.
remove unused function.

ok gilles@

Simplify the table backend interface: lookup results are returned
as strings, and parsing is handled by the upper layer.

ok gilles@

since we already support regex lookups in tables for builtin filters, let's
also support regex lookups in match rule criterias performing table lookups

ok millert@

add REG_NOSUB to flags in regcompile

suggested by jung@

introduce K_REGEX table type and table_regex_match(), unused for now

ok eric@

rework the table API so that it takes a struct smtpd * context in parameter
of functions creating, looking up or destroying tables.

this is a first step in cleaning up parse.y so it doesn't have side effects
outside of parse_config(), bringing nothing but making code cleaner.

ok millert@

remove 'where' parameter from all x*() functions in utils.c, it doesn't
really help us with anything, propagate the change in codebase

ok millert@

missing case

ok gilles@

no need to parse and dump the relayhost in the lookup process.

ok gilles@

zero memory before calling text_to_relayhost()

fix issue reported by semarie@

ok gilles@

switch smtpd to new grammar

ok eric@

in function used for tracing, display unknown lookup types as "???"

diff from Wolf480pl

switch to /usr/local/libexec when looking for -extras and drop loop iterating
paths

this effectively reverts table.c r1.21 which was mainly introduced for a smooth
transition in -current

ok gilles

remove spaces after '!'

no binary change

ok millert

when looking up tables, start in /usr/local/libexec before /usr/libexec, so
ports/packages can be installed in the proper place

ok jung@

allow table API to lookup for mailaddr mappings

ok sunil@, ok jung@

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

recipient and sender lists now support the user+TAG notation, allowing
among other things a secondary MX to filter recipients to be relayed to a
primary MX even if they are using tags. there are other nice things to do
with that feature, use your imagination.

tested and ok florian@, tested by several users for a few days too

Update the table API: lookup functions can take an optional parameters
dictionnary (currently not set). While there, add a helper for forking
external backends, and remove unused table functions.

ok gilles@

stop casting sizeof to int

these snprintf() calls can't possibly truncate because they copy data from
buffers that are already protected against truncation and that do not
exceed the destination buffer size when copied together ...

however, i think we should add checks here too because it'll help us catch
errors in table backends when adding new ones if we miss a truncation check
there.

(void) cast strlcat() and snprintf() that cannot truncate

be a bit more strict with an strlcat() truncation by causing it to fail in
table_create() instead of later in parse.y - in both cases, this would
cause smtpd to fatal() at startup if a table has a config file too large

bcopy -> memmove
bzero -> memset

allow subdomain matching in mailaddr tables

change dict_poproot() prototype: do not take key placeholder parameter as
it can't work that way.

accept credentials formated as "<user> <passwd>"

fix parsing of inet6 addresses when prefixed with "IPv6:"

%i -> %d in format strings

Update ldap and sqlite table backends and provide them as external backends.

Add a table_proc backend for delegating table lookups to another
process. Stop building experimental table_sqlite and table_ldap as
they will be provided as external backends.

lowercase the key when adding an entry to a static table, since the
lookups are done on lowercase strings. warn about duplicates entries
and avoid to leak memory.

ok gilles@

sync with OpenSMTPD 5.3.2

ok gilles@

unbreak broken smtpctl table update

fix by eric and I, ok eric@

- handle getaddrinfo() error as LKA_TEMPFAIL
- handle getsockname() error in smtp_connected()
- accept '/' as part of user-part, expand to ':' as done by qmail
- fix wrong check in mda leading to bogus Return-Path header
- fix aliases parsing when there's a white space between key and separator
- some cosmethic cleanup

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

Use regfree after we're done with preg.

From gilles@

report errno in debug logs and other minor cleanups.

ok gilles@

introduce table_dump() and tweak format

ok gilles@

simplify code

ok gilles@

remove the tag workaround for table_create() and table_find(),
now that static tables handle their updates internally.

ok gilles@

remove unused members in struct table

table_fetch() always expect a valid dst pointer.

ok gilles@

introduce dump() and add() table methods, only implemented for static tables.

ok gilles@

pass the table pointer to the lookup()/fecth() methods

ok gilles@

change the close() method to take the table pointer

ok gilles

Make the backend open method return an int to report success.
The implementation is responsible for setting the handle pointer
as needed.

ok gilles@

reorder parameters for consistency

introduce a table_match() function to check for a key in a table

ok gilles@

get rid of the unused dict argument in table lookup and fetch api.

ok gilles@

move the table backend name in the backend struct.
remove unused function.

ok gilles@

Simplify the table backend interface: lookup results are returned
as strings, and parsing is handled by the upper layer.

ok gilles@

since we already support regex lookups in tables for builtin filters, let's
also support regex lookups in match rule criterias performing table lookups

ok millert@

add REG_NOSUB to flags in regcompile

suggested by jung@

introduce K_REGEX table type and table_regex_match(), unused for now

ok eric@

rework the table API so that it takes a struct smtpd * context in parameter
of functions creating, looking up or destroying tables.

this is a first step in cleaning up parse.y so it doesn't have side effects
outside of parse_config(), bringing nothing but making code cleaner.

ok millert@

remove 'where' parameter from all x*() functions in utils.c, it doesn't
really help us with anything, propagate the change in codebase

ok millert@

missing case

ok gilles@

no need to parse and dump the relayhost in the lookup process.

ok gilles@

zero memory before calling text_to_relayhost()

fix issue reported by semarie@

ok gilles@

switch smtpd to new grammar

ok eric@

in function used for tracing, display unknown lookup types as "???"

diff from Wolf480pl

switch to /usr/local/libexec when looking for -extras and drop loop iterating
paths

this effectively reverts table.c r1.21 which was mainly introduced for a smooth
transition in -current

ok gilles

remove spaces after '!'

no binary change

ok millert

when looking up tables, start in /usr/local/libexec before /usr/libexec, so
ports/packages can be installed in the proper place

ok jung@

allow table API to lookup for mailaddr mappings

ok sunil@, ok jung@

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

recipient and sender lists now support the user+TAG notation, allowing
among other things a secondary MX to filter recipients to be relayed to a
primary MX even if they are using tags. there are other nice things to do
with that feature, use your imagination.

tested and ok florian@, tested by several users for a few days too

Update the table API: lookup functions can take an optional parameters
dictionnary (currently not set). While there, add a helper for forking
external backends, and remove unused table functions.

ok gilles@

stop casting sizeof to int

these snprintf() calls can't possibly truncate because they copy data from
buffers that are already protected against truncation and that do not
exceed the destination buffer size when copied together ...

however, i think we should add checks here too because it'll help us catch
errors in table backends when adding new ones if we miss a truncation check
there.

(void) cast strlcat() and snprintf() that cannot truncate

be a bit more strict with an strlcat() truncation by causing it to fail in
table_create() instead of later in parse.y - in both cases, this would
cause smtpd to fatal() at startup if a table has a config file too large

bcopy -> memmove
bzero -> memset

allow subdomain matching in mailaddr tables

change dict_poproot() prototype: do not take key placeholder parameter as
it can't work that way.

accept credentials formated as "<user> <passwd>"

fix parsing of inet6 addresses when prefixed with "IPv6:"

%i -> %d in format strings

Update ldap and sqlite table backends and provide them as external backends.

Add a table_proc backend for delegating table lookups to another
process. Stop building experimental table_sqlite and table_ldap as
they will be provided as external backends.

lowercase the key when adding an entry to a static table, since the
lookups are done on lowercase strings. warn about duplicates entries
and avoid to leak memory.

ok gilles@

sync with OpenSMTPD 5.3.2

ok gilles@

unbreak broken smtpctl table update

fix by eric and I, ok eric@

- handle getaddrinfo() error as LKA_TEMPFAIL
- handle getsockname() error in smtp_connected()
- accept '/' as part of user-part, expand to ':' as done by qmail
- fix wrong check in mda leading to bogus Return-Path header
- fix aliases parsing when there's a white space between key and separator
- some cosmethic cleanup

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

report errno in debug logs and other minor cleanups.

ok gilles@

introduce table_dump() and tweak format

ok gilles@

simplify code

ok gilles@

remove the tag workaround for table_create() and table_find(),
now that static tables handle their updates internally.

ok gilles@

remove unused members in struct table

table_fetch() always expect a valid dst pointer.

ok gilles@

introduce dump() and add() table methods, only implemented for static tables.

ok gilles@

pass the table pointer to the lookup()/fecth() methods

ok gilles@

change the close() method to take the table pointer

ok gilles

Make the backend open method return an int to report success.
The implementation is responsible for setting the handle pointer
as needed.

ok gilles@

reorder parameters for consistency

introduce a table_match() function to check for a key in a table

ok gilles@

get rid of the unused dict argument in table lookup and fetch api.

ok gilles@

move the table backend name in the backend struct.
remove unused function.

ok gilles@

Simplify the table backend interface: lookup results are returned
as strings, and parsing is handled by the upper layer.

ok gilles@

since we already support regex lookups in tables for builtin filters, let's
also support regex lookups in match rule criterias performing table lookups

ok millert@

add REG_NOSUB to flags in regcompile

suggested by jung@

introduce K_REGEX table type and table_regex_match(), unused for now

ok eric@

rework the table API so that it takes a struct smtpd * context in parameter
of functions creating, looking up or destroying tables.

this is a first step in cleaning up parse.y so it doesn't have side effects
outside of parse_config(), bringing nothing but making code cleaner.

ok millert@

remove 'where' parameter from all x*() functions in utils.c, it doesn't
really help us with anything, propagate the change in codebase

ok millert@

missing case

ok gilles@

no need to parse and dump the relayhost in the lookup process.

ok gilles@

zero memory before calling text_to_relayhost()

fix issue reported by semarie@

ok gilles@

switch smtpd to new grammar

ok eric@

in function used for tracing, display unknown lookup types as "???"

diff from Wolf480pl

switch to /usr/local/libexec when looking for -extras and drop loop iterating
paths

this effectively reverts table.c r1.21 which was mainly introduced for a smooth
transition in -current

ok gilles

remove spaces after '!'

no binary change

ok millert

when looking up tables, start in /usr/local/libexec before /usr/libexec, so
ports/packages can be installed in the proper place

ok jung@

allow table API to lookup for mailaddr mappings

ok sunil@, ok jung@

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

recipient and sender lists now support the user+TAG notation, allowing
among other things a secondary MX to filter recipients to be relayed to a
primary MX even if they are using tags. there are other nice things to do
with that feature, use your imagination.

tested and ok florian@, tested by several users for a few days too

Update the table API: lookup functions can take an optional parameters
dictionnary (currently not set). While there, add a helper for forking
external backends, and remove unused table functions.

ok gilles@

stop casting sizeof to int

these snprintf() calls can't possibly truncate because they copy data from
buffers that are already protected against truncation and that do not
exceed the destination buffer size when copied together ...

however, i think we should add checks here too because it'll help us catch
errors in table backends when adding new ones if we miss a truncation check
there.

(void) cast strlcat() and snprintf() that cannot truncate

be a bit more strict with an strlcat() truncation by causing it to fail in
table_create() instead of later in parse.y - in both cases, this would
cause smtpd to fatal() at startup if a table has a config file too large

bcopy -> memmove
bzero -> memset

allow subdomain matching in mailaddr tables

change dict_poproot() prototype: do not take key placeholder parameter as
it can't work that way.

accept credentials formated as "<user> <passwd>"

fix parsing of inet6 addresses when prefixed with "IPv6:"

%i -> %d in format strings

Update ldap and sqlite table backends and provide them as external backends.

Add a table_proc backend for delegating table lookups to another
process. Stop building experimental table_sqlite and table_ldap as
they will be provided as external backends.

lowercase the key when adding an entry to a static table, since the
lookups are done on lowercase strings. warn about duplicates entries
and avoid to leak memory.

ok gilles@

sync with OpenSMTPD 5.3.2

ok gilles@

unbreak broken smtpctl table update

fix by eric and I, ok eric@

- handle getaddrinfo() error as LKA_TEMPFAIL
- handle getsockname() error in smtp_connected()
- accept '/' as part of user-part, expand to ':' as done by qmail
- fix wrong check in mda leading to bogus Return-Path header
- fix aliases parsing when there's a white space between key and separator
- some cosmethic cleanup

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

introduce table_dump() and tweak format

ok gilles@

simplify code

ok gilles@

remove the tag workaround for table_create() and table_find(),
now that static tables handle their updates internally.

ok gilles@

remove unused members in struct table

table_fetch() always expect a valid dst pointer.

ok gilles@

introduce dump() and add() table methods, only implemented for static tables.

ok gilles@

pass the table pointer to the lookup()/fecth() methods

ok gilles@

change the close() method to take the table pointer

ok gilles

Make the backend open method return an int to report success.
The implementation is responsible for setting the handle pointer
as needed.

ok gilles@

reorder parameters for consistency

introduce a table_match() function to check for a key in a table

ok gilles@

get rid of the unused dict argument in table lookup and fetch api.

ok gilles@

move the table backend name in the backend struct.
remove unused function.

ok gilles@

Simplify the table backend interface: lookup results are returned
as strings, and parsing is handled by the upper layer.

ok gilles@

since we already support regex lookups in tables for builtin filters, let's
also support regex lookups in match rule criterias performing table lookups

ok millert@

add REG_NOSUB to flags in regcompile

suggested by jung@

introduce K_REGEX table type and table_regex_match(), unused for now

ok eric@

rework the table API so that it takes a struct smtpd * context in parameter
of functions creating, looking up or destroying tables.

this is a first step in cleaning up parse.y so it doesn't have side effects
outside of parse_config(), bringing nothing but making code cleaner.

ok millert@

remove 'where' parameter from all x*() functions in utils.c, it doesn't
really help us with anything, propagate the change in codebase

ok millert@

missing case

ok gilles@

no need to parse and dump the relayhost in the lookup process.

ok gilles@

zero memory before calling text_to_relayhost()

fix issue reported by semarie@

ok gilles@

switch smtpd to new grammar

ok eric@

in function used for tracing, display unknown lookup types as "???"

diff from Wolf480pl

switch to /usr/local/libexec when looking for -extras and drop loop iterating
paths

this effectively reverts table.c r1.21 which was mainly introduced for a smooth
transition in -current

ok gilles

remove spaces after '!'

no binary change

ok millert

when looking up tables, start in /usr/local/libexec before /usr/libexec, so
ports/packages can be installed in the proper place

ok jung@

allow table API to lookup for mailaddr mappings

ok sunil@, ok jung@

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

recipient and sender lists now support the user+TAG notation, allowing
among other things a secondary MX to filter recipients to be relayed to a
primary MX even if they are using tags. there are other nice things to do
with that feature, use your imagination.

tested and ok florian@, tested by several users for a few days too

Update the table API: lookup functions can take an optional parameters
dictionnary (currently not set). While there, add a helper for forking
external backends, and remove unused table functions.

ok gilles@

stop casting sizeof to int

these snprintf() calls can't possibly truncate because they copy data from
buffers that are already protected against truncation and that do not
exceed the destination buffer size when copied together ...

however, i think we should add checks here too because it'll help us catch
errors in table backends when adding new ones if we miss a truncation check
there.

(void) cast strlcat() and snprintf() that cannot truncate

be a bit more strict with an strlcat() truncation by causing it to fail in
table_create() instead of later in parse.y - in both cases, this would
cause smtpd to fatal() at startup if a table has a config file too large

bcopy -> memmove
bzero -> memset

allow subdomain matching in mailaddr tables

change dict_poproot() prototype: do not take key placeholder parameter as
it can't work that way.

accept credentials formated as "<user> <passwd>"

fix parsing of inet6 addresses when prefixed with "IPv6:"

%i -> %d in format strings

Update ldap and sqlite table backends and provide them as external backends.

Add a table_proc backend for delegating table lookups to another
process. Stop building experimental table_sqlite and table_ldap as
they will be provided as external backends.

lowercase the key when adding an entry to a static table, since the
lookups are done on lowercase strings. warn about duplicates entries
and avoid to leak memory.

ok gilles@

sync with OpenSMTPD 5.3.2

ok gilles@

unbreak broken smtpctl table update

fix by eric and I, ok eric@

- handle getaddrinfo() error as LKA_TEMPFAIL
- handle getsockname() error in smtp_connected()
- accept '/' as part of user-part, expand to ':' as done by qmail
- fix wrong check in mda leading to bogus Return-Path header
- fix aliases parsing when there's a white space between key and separator
- some cosmethic cleanup

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

simplify code

ok gilles@

remove the tag workaround for table_create() and table_find(),
now that static tables handle their updates internally.

ok gilles@

remove unused members in struct table

table_fetch() always expect a valid dst pointer.

ok gilles@

introduce dump() and add() table methods, only implemented for static tables.

ok gilles@

pass the table pointer to the lookup()/fecth() methods

ok gilles@

change the close() method to take the table pointer

ok gilles

Make the backend open method return an int to report success.
The implementation is responsible for setting the handle pointer
as needed.

ok gilles@

reorder parameters for consistency

introduce a table_match() function to check for a key in a table

ok gilles@

get rid of the unused dict argument in table lookup and fetch api.

ok gilles@

move the table backend name in the backend struct.
remove unused function.

ok gilles@

Simplify the table backend interface: lookup results are returned
as strings, and parsing is handled by the upper layer.

ok gilles@

since we already support regex lookups in tables for builtin filters, let's
also support regex lookups in match rule criterias performing table lookups

ok millert@

add REG_NOSUB to flags in regcompile

suggested by jung@

introduce K_REGEX table type and table_regex_match(), unused for now

ok eric@

rework the table API so that it takes a struct smtpd * context in parameter
of functions creating, looking up or destroying tables.

this is a first step in cleaning up parse.y so it doesn't have side effects
outside of parse_config(), bringing nothing but making code cleaner.

ok millert@

remove 'where' parameter from all x*() functions in utils.c, it doesn't
really help us with anything, propagate the change in codebase

ok millert@

missing case

ok gilles@

no need to parse and dump the relayhost in the lookup process.

ok gilles@

zero memory before calling text_to_relayhost()

fix issue reported by semarie@

ok gilles@

switch smtpd to new grammar

ok eric@

in function used for tracing, display unknown lookup types as "???"

diff from Wolf480pl

switch to /usr/local/libexec when looking for -extras and drop loop iterating
paths

this effectively reverts table.c r1.21 which was mainly introduced for a smooth
transition in -current

ok gilles

remove spaces after '!'

no binary change

ok millert

when looking up tables, start in /usr/local/libexec before /usr/libexec, so
ports/packages can be installed in the proper place

ok jung@

allow table API to lookup for mailaddr mappings

ok sunil@, ok jung@

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

recipient and sender lists now support the user+TAG notation, allowing
among other things a secondary MX to filter recipients to be relayed to a
primary MX even if they are using tags. there are other nice things to do
with that feature, use your imagination.

tested and ok florian@, tested by several users for a few days too

Update the table API: lookup functions can take an optional parameters
dictionnary (currently not set). While there, add a helper for forking
external backends, and remove unused table functions.

ok gilles@

stop casting sizeof to int

these snprintf() calls can't possibly truncate because they copy data from
buffers that are already protected against truncation and that do not
exceed the destination buffer size when copied together ...

however, i think we should add checks here too because it'll help us catch
errors in table backends when adding new ones if we miss a truncation check
there.

(void) cast strlcat() and snprintf() that cannot truncate

be a bit more strict with an strlcat() truncation by causing it to fail in
table_create() instead of later in parse.y - in both cases, this would
cause smtpd to fatal() at startup if a table has a config file too large

bcopy -> memmove
bzero -> memset

allow subdomain matching in mailaddr tables

change dict_poproot() prototype: do not take key placeholder parameter as
it can't work that way.

accept credentials formated as "<user> <passwd>"

fix parsing of inet6 addresses when prefixed with "IPv6:"

%i -> %d in format strings

Update ldap and sqlite table backends and provide them as external backends.

Add a table_proc backend for delegating table lookups to another
process. Stop building experimental table_sqlite and table_ldap as
they will be provided as external backends.

lowercase the key when adding an entry to a static table, since the
lookups are done on lowercase strings. warn about duplicates entries
and avoid to leak memory.

ok gilles@

sync with OpenSMTPD 5.3.2

ok gilles@

unbreak broken smtpctl table update

fix by eric and I, ok eric@

- handle getaddrinfo() error as LKA_TEMPFAIL
- handle getsockname() error in smtp_connected()
- accept '/' as part of user-part, expand to ':' as done by qmail
- fix wrong check in mda leading to bogus Return-Path header
- fix aliases parsing when there's a white space between key and separator
- some cosmethic cleanup

Sync with our smtpd repo:

* first bricks of ldap and sqlite support (not finished but both working)
* new table API to replace map API, all lookups are done through tables
* improved handling of temporary errors throughout the daemon
* improved scheduler and mta logic: connection reuse, optimizes batches
* improved queue: more tolerant to admin errors, new layout, less disk-IO
* improved memory usage under high load
* SSL certs/keys isolated to lookup process to avoid facing network
* VIRTUAL support improved, fully virtual setups possible now
* runtime tracing of processes through smtpctl trace
* ssl_privsep.c sync-ed with relayd
* ssl.c no longer contains smtpd specific interfaces
* smtpd-specific ssl bits moved to ssl_smtpd.c
* update mail address in copyright

FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE. FLUSH YOUR QUEUE.

smtpd.conf(5) simplified, it will require adaptations

ok eric@

introduce a table_match() function to check for a key in a table

ok gilles@

get rid of the unused dict argument in table lookup and fetch api.

ok gilles@

move the table backend name in the backend struct.
remove unused function.

ok gilles@

Simplify the table backend interface: lookup results are returned
as strings, and parsing is handled by the upper layer.

ok gilles@

since we already support regex lookups in tables for builtin filters, let's
also support regex lookups in match rule criterias performing table lookups

ok millert@

add REG_NOSUB to flags in regcompile

suggested by jung@

introduce K_REGEX table type and table_regex_match(), unused for now

ok eric@

rework the table API so that it takes a struct smtpd * context in parameter
of functions creating, looking up or destroying tables.

this is a first step in cleaning up parse.y so it doesn't have side effects
outside of parse_config(), bringing nothing but making code cleaner.

ok millert@

remove 'where' parameter from all x*() functions in utils.c, it doesn't
really help us with anything, propagate the change in codebase

ok millert@

missing case

ok gilles@

no need to parse and dump the relayhost in the lookup process.

ok gilles@

zero memory before calling text_to_relayhost()

fix issue reported by semarie@

ok gilles@

switch smtpd to new grammar

ok eric@

in function used for tracing, display unknown lookup types as "???"

diff from Wolf480pl

switch to /usr/local/libexec when looking for -extras and drop loop iterating
paths

this effectively reverts table.c r1.21 which was mainly introduced for a smooth
transition in -current

ok gilles

remove spaces after '!'

no binary change

ok millert

when looking up tables, start in /usr/local/libexec before /usr/libexec, so
ports/packages can be installed in the proper place

ok jung@

allow table API to lookup for mailaddr mappings

ok sunil@, ok jung@

use <limits.h> comprehensively. For now try to push <> includes to
each .c file, and out of the .h files. To avoid overinclude.
ok gilles, in principle. If this has been done right, -portable should
become easier to maintain.

recipient and sender lists now support the user+TAG notation, allowing
among other things a secondary MX to filter recipients to be relayed to a
primary MX even if they are using tags. there are other nice things to do
with that feature, use your imagination.

tested and ok florian@, tested by several users for a few days too

Update the table API: lookup functions can take an optional parameters
dictionnary (currently not set). While there, add a helper for forking
external backends, and remove unused table functions.

ok gilles@

stop casting sizeof to int

these snprintf() calls can't possibly truncate because they copy data from
buffers that are already protected against truncation and that do not
exceed the destination buffer size when copied together ...

however, i think we should add checks here too because it'll help us catch
errors in table backends when adding new ones if we miss a truncation check
there.

(void) cast strlcat() and snprintf() that cannot truncate

be a bit more strict with an strlcat() truncation by causing it to fail in
table_create() instead of later in parse.y - in both cases, this would
cause smtpd to fatal() at startup if a table has a config file too large

bcopy -> memmove
bzero -> memset

allow subdomain matching in mailaddr tables

change dict_poproot() prototype: do not take key placeholder parameter as
it can't work that way.

accept credentials formated as "<user> <passwd>"

fix parsing of inet6 addresses when prefixed with "IPv6:"

%i -> %d in format strings