Change adds a 'log' option to relayd.conf(5) rule. The relayd(8) then uses
the option to set corresponding `log` action in pf(4) rules it generates
to handle network traffic.

The patch comes from Giannis Kapetanakis (bilias _from_ edu.physics.uoc.gr).

OK sashan@

Unmention/don't explain SSL, drop 9y old "ssl" keyword/deprecation warning

Switch "ssl" to "tls" in relayd.conf(5) if you haven't done so in the last
ten years, "ssl" is now an error.

Say "TLS" not "SSL/TLS" and drop the primer in the TLS RELAYS section.

OK benno

Use ibuf_data() instead of accessing ibuf->buf directly.
OK tb@

Don't declare variables as "unsigned char *" that are passed to
functions that take "char *" arguments. Where such chars are
assigned to int or passed to ctype functions, explicitly cast them
to unsigned char.

For OpenBSD's clang, -Wpointer-sign has been disabled by default,
but when the parse.y code was built elsewhere, the compiler would
complain.

With help from millert@
ok benno@ deraadt@

Don't leak host address. Found with clang static analyzer.

OK tb@

Add 'strip' directive

Feedback by Olivier Cherrier, Hiltjo Posthuma, Mischa

OK benno@

getifaddrs() can return entries where ifa_addr is NULL. Check for this
before accessing anything in ifa_addr.
ok claudio@

Remove deprecated snmp keyword.

OK denis@

Let relayd make use of libagentx. No functional change intended.

OK tb@
Enthousiasm from Mischa Peters

Remove trailing spaces & tabs

Rewrite the agentx code of relayd. This new framework should allow us
to add new objects easier if so desired and should handle a lot more
corner-cases.

This commit should also fix the following:
- On most (all) tables it omits the *Entry elements, making it not map to
OPENBSD-RELAYD-MIB.txt.
- sstolen returns the size of the sockaddr_in{,6}, instead of the
sin{,6}_addr resulting in garbage data to be put in the ip-field.
- relaydSessionPortIn and relaydSessionPortOut are swapped
- relaydSessions only uses relaydSessionIndex, while
OPENBSD-RELAYD-MIB.txt says it should have 2 indices
- miscellaneous minor things related to the AGENTX-protocol, like wonky
index handeling and returning NOSUCHINSTANCE where NOSUCHOBJECT
should be returned, etc.

This commit does remove traps, but it's large enough as is and I intent
on adding it soon(tm). It also deprecates the snmp keyword in favour of
an agentx keyword. The snmp keyword is still available, but will be
removed in the future.

Tweaks and OK denis@ on the relayd parts
Tweaks and OK claudio@ on the agentx parts
"Get it in" deraadt@

Enable TLSv1.3 support in relayd(8)
with the help from tb@ jsing@; ok tb@

improve parsing of relay {} sections:
- Do not accept multiple protocol statements, as only one will be
used, it is better to error out if more arespecified.
- do not allow tcp and http options in dns protocol definitions, they
are ignored anyway.
Suggested by Nick (nick -AT- kousu -DOT- ca) thanks.
ok claudio@

remove old log options 'log update/all' that were replaced with 'log
state changes/host checks/connection' some time ago. ok reyk@
claudio@ on first version, kn@ noticed that the tokens could be
removed too.

Add support for binary protocol health checking. Feedback and guidance from
benno@ and reky@. Man page tweaks from jmc@.

ok benno@

fix error description on invalid forward ip.

OK benno@

Fix integer sizes in format strings and enable formatting warnings
for yyerror.

From Moritz Buhl
ok claudio@

When system calls indicate an error they return -1, not some arbitrary
value < 0. errno is only updated in this case. Change all (most?)
callers of syscalls to follow this better, and let's see if this strictness
helps us in the future.

Add support for SNI with new "tls keypair" option to load additional certs.

Tested by many (thanks!)

Feedback & OK rob@

Move the relay keys/certs into a separate global list and look them up by id.

Moving the certs out of the relay struct will help to add multiple SNI certs.

Tested by many users (thanks!)

Feedback & OK rob@

Fix the check if a relay has been specified twice

Relays cannot have the same name or listen address. If a listen
address is specified multiple times, the parser expands the
configuration into multiple relays automatically.

OK rob@

Move relay_load_*() functions into relayd.c

Pass the *env as an explicit argument instead of using the global
pointer: The relay_load_certfiles() function is called early before
the *env is set up. This does not change anything in the current code
as *env is not used by anything in the function (not even
ssl_load_key() that is taking it as an argument) but it will be needed
by upcoming changes for SNI.

Ok rob@

Add support for from/to in relay filter rules.

For example,
pass from 10.0.0.0/8 path "/hello/*" forward to <b>

Ok benno@

remove unused keyword "virtual".
ok gcc, claudio@ agrees

Support for rfc 6455 Websockets connection upgrade. Add a new protocol
option 'http { [no] websockets }' to allow such connections (default
is no). Original diff from Daniel Lamando (dan AT danopia DOT net),
option and header checks by me. suggestions and ok bluhm@ and earlier
diff claudio@

(unsigned) means (unsigned int) which on ptrdiff_t or size_t or other
larger types really is a range reduction...
Almost any cast to (unsigned) is a bug.
ok millert tb benno

- odd condition/test in PF lexer
(and other lexers too)

This commit rectifies earlier change:

in the lex... even inside quotes, a \ followed by space or tab should
expand to space or tab, and a \ followed by newline should be ignored
(as a line continuation). compatible with the needs of hoststated
(which has the most strict quoted string requirements), and ifstated
(where one commonly does line continuations in strings).

OK deraadt@, OK millert@

Make host_*() AF-agnostic

Merge host_v{4,6}() into much simpler host_ip() using just getaddrinfo().

With input & test by kn@ and benno@

OK benno@ kn@

replace malloc()+strlcpy() with strndup() in cmdline_symset().

"looks good" gilles@ halex@

replace the current log options

log updates|all

with

log state changes
log host checks
log connection [errors]

The first two control the logging of host check results: either changes in host state only or
all checks.

The third option controls logging of connections in relay mode:
Either log all connections, or only errors.

Additionaly, errors will be logged with LOG_WARN and good connections
will be logged with LOG_INFO, so they can be differentiated in syslog.

ok and feedback from claudio@

Do for most running out of memory err() what was done for most running
out of memory log_warn(). i.e. ("%s", __func__) instead of manual
function names and redundant verbiage about which wrapper detected the
out of memory condition.

ok henning@

No need to mention which memory allocation entry point failed (malloc,
calloc or strdup), we just need to log that we ran out of memory in a
particular function.

Recommended by florian@ and deraadt@

ok benno@ henning@ tb@

Fix an off-by-one line count when using include statements.

Thanks to otto@ for the initial diff.

OK benno@

Plug leak in error case of the common 'varset' implementations.

ok benno@

Remove RELAY_MAX_SESSIONS from relayd, there is no reason to limit relays
to 1024 session per process (esp. with keep-alive). Now the fd limit is
the new maximum and relayd will make sure to not accept too many sessions.
The tcp backlog config maximum is now 512, adjust manpage accordingly.
OK benno@ deraadt@

add options to specify the control socket in relayd and relayctl.
From Kapetanakis Giannis, thanks.
ok claudio@

Change the ecdhe curve configuration to the same way httpd is doing it.
This removes 'no ecdh' and renames 'ecdh curve auto' to ecdhe default.
The code uses now tls_config_set_ecdhecurves(3) so it is possible to
specify multiple curves now. If people specified curves in their config
they need to adjust their config now.
OK beck@

Use file descriptor passing to load certificates into the relays. Especially
the ca file (having all the trusted certs in them) can be so big that loading
via imsg fails.
OK beck@

Check that http options are only configured in http protocols.
OK benno@

make the maximum size of http headers configurable in the protocol.
ok bluhm@, >8k makes sense claudio@

65535 is a valid port to listen on.
Off-by-one pointed out by and diff from Kris Katterjohn katterjohn AT
gmail, thanks!
chris@ pointed out that more than httpd(8) is effected.
OK gilles@

Migrate relayd to use libtls for TLS. Still does the TLS privsep via the
engine but at least we can use a sane API for new features.
Going in now so it is possible to work with this in tree.
General agreement at d2k17.

Replace hand-rolled for(;;) emptying of 'symhead' TAILQ with more
modern TAILQ_FOREACH_SAFE().

No intentional functional change.

ok millert@ bluhm@ gilles@

Replace symset()'s hand-rolled for(;;) traversal of 'symhead' TAILQ
with more modern TAILQ_FOREACH(). This what symget() was already
doing.

Add paranoia '{}' around body of symget()'s TAILQ_FOREACH().

No intentional functional change.

ok bluhm@ otto@

spacing

Replace [RELAY|SERVER]_MAXPROC with the new PROC_MAX_INSTANCES
variable and limit it from 128 to 32 instances (the old value).
While here, move a few PROC_ defines around.

OK rzalamena@

Move snmp options into struct relayd_config and delay start of the
snmp subsystem until the configuration is done.

OK benno@ claudio@

Split "struct relayd" into two structs: "struct relayd" and "struct
relayd_config". This way we can send all the relevant global
configuration to the children, not just the flags and the opts.

With input from and
OK claudio@ benno@

Switch from the not really working session cache (because of the multiprocess
nature of relayd) to tls session tickets to do TLS session resumption.
TLS session tickets do not need to store SSL session data in the server but
instead send an encrypted ticket to the clients that allows to resume the
session. This is mostly stateless (apart from the encryption keys).
relayd now ensures that all relay processes use the same key to encrypt
the tickets. Keys are rotated every 2h and there is a primary and backup key.
The tls session timeout is set to 2h to hint to the clients how long the
session tickets is supposed to be alive.
Input and OK benno@, reyk@

do not allow whitespace in macro names, i.e. "this is" = "a variable".
change this in all config parsers in our tree that support macros.
problem reported by sven falempin.

feedback from henning@, stsp@, deraadt@
ok florian@ mikeb@

Update log.c: change fatal() and fatalx() into variadic functions,
include the process name, and replace all calls of fatal*(NULL) with
fatal(__func__) for better debugging.

OK benno@

stdlib.h is in scope; do not cast malloc/calloc/realloc*
ok millert krw

Fix obvious problems with relayd config reload.
- fix a TAILQ corruption because of a use after free
- do not reinit the SSL engine since that fails
OK sthen, benno

branches: 1.203.2;
Use AI_ADDRCONFIG when resolv hosts on startup.

OK henning@

Clean up the relayd headers with help of include-what-you-use and some
manual review. Based on common practice, relayd.h now includes the
necessary headers for itself.

OK benno@

Include <netinet/in.h> before <net/pfvar.h>. In a future change when
ports is ready, <net/pfvar.h> will stop including a pile of balony.

Adapt to <limits.h> universe.
ok millert

pf now supports source-hash and random with tables so we can allow it
in redirections. Thanks for help and input from jsg and yasuoka who
reminded me to dig out and update these old diffs for pf and relayd.

ok jsg@

Stop pulling in <arpa/inet.h> or <arpa/nameser.h> when unnecessary.
*Do* pull it in when in_{port,addr}_h is needed and <netinet/in.h> isn't.

ok reyk@

Update relayd to use siphash instead of sys/hash. The source-hash,
loadbalance and hash modes use a random key by default that can be
forced to be a static key with a new configuration argument.

With input from Max Fillinger.

ok tedu@

Change the keyword "ssl" to "tls" to reflect reality since we
effectively disabled support for the SSL protocols. SSL remains a
common term describing SSL/TLS, there is some controvery about this
change, and the name really doesn't matter, but I feel confident about
it now.

(btw., sthen@ pointed out some historical context:
http://tim.dierks.org/2014/05/security-standards-and-name-changes-in.html)

OK benno@, with input from tedu@

Don't allow embedded nul characters in strings.
Fixes a pfctl crash with an anchor name containing
an embedded nul found with the afl fuzzer.

pfctl parse.y patch from and ok deraadt@

Support exporting relayd statistics via AgentX/snmpd

This should be equivalent to the statistics available
via the various relaydctl show commands

okay benno@ reyk@

Remove the sslv2 option since LibreSSL has no SSLv2 support (however retain
SSL_OP_NO_SSLv2 in case you happen to be running relayd on another platform
with another SSL library). Also fix the SSLv3 handling so that 'no sslv3'
actually works as intended.

ok reyk@

Convert the logic in yyerror(). Instead of creating a temporary
format string, create a temporary message.
OK deraadt@

Remove the "interface" option from the "transparent forward" directive.
It was mandatory in the grammar but never used in the code.

A fully transparent relay can now be specified with the following
directive in a relay block: "transparent forward to destination".

OK sthen@

Disable SSLv3 by default.

OK sthen@ jsing@

revert previous; was based on a work-in-progress, as well
as being an incomplete and therefore incorrect adaptation

apologies to anybody who got bitten by this mistake

ok reyk@

Implement consistent host hashing for relayd, based on
work done by andre@

Re-add a randomized hash seed (which had apparently
gotten inadvertently removed in the past).

Allows for multiple relayd instances to be configured
to forward traffic to the same host, falling back to
the random seed when not explicitly configured to do so.

ok reyk@

Sometimes I just sort the tokens in parse.y

Add support for EDH to provide perfect forward secrecy for older SSL
clients. Additionally, add options for disallowing client-initiated
renegotiations and to prefer the server's cipher list over the
client's preferences.

This work is based on a diff by Markus Gebert at hostpoint.ch, and was
discussed with jsing@ resulting in a few different defaults.

ok benno@

Simplify the code that handles the HTTP headers by using an RB tree
with associated lists instead of the complicated lookup table and
"others" list. This might add a little malloc overhead for common
headers but also fixes some issues like the handling of repeated
headers - for example, handling of multiple "Set-Cookie" headers.

ok bluhm@ (regress part)
ok benno@

Replace the protocol directives for HTTP with a new generic filtering
language. The grammar is inspired by pf and allows to write versatile
last-matching filter rules in protocol sections starting with the
"pass", "block" or "match" keywords. This work was started almost two
years ago and replaces large parts of relayd(8)'s HTTP and filtering
code. The initial version reimplements and extends HTTP filtering,
but will be improved to support generic TCP and other protocols later.

With some testing, feedback, and help from benno@ and andre@.

OK benno@

sync copyright to reality according to my last changes

Fix a leak from a recent added mark/marked keywords check.

ok reyk

fail for unsupported node action/type combinations. Also fail for the
unsupported mark/marked combination in a single rule.

ok andre@

Support the CA key for SSL inspection in the ca process. Instead of
looking up the keys by relay id, add all keys to a list and look them
up by key id.

ok benno@

Add a few missing free's in the grammar.

Add a few more overflow checks for strlc* functions in parse.y

remove 'restricted' symbol from yacc parser, overlooked in previous
commit

found by andre@

Adapt relayd to use AgentX protocol to send traps

ok reyk@ benno@

relax the cfg file secrecy check slightly to allow group readability
default permissions and mtree NOT changed.
prodded by benno, ok phessler benno jmatthew theo pelikan florian

fix a a double free caused by a config with two listen on statements
in a relay (the first one with ssl).
found and fixed by Erik Lax <erik AT halon DOT se>
ok phessler

use u_char for buffers in yylex, for ctype calls
found by millert@, ok deraadt@

Add support for ECDHE (Elliptic curve Diffie-Hellman) to enable
TLS/SSL Perfect Forward Secrecy (PFS).

ok djm@

Support SSL inspection, the ability to transparently filter in SSL/TLS
connections (eg. HTTPS) by using a local CA that is accepted by the
clients. See the "SSL RELAYS" and "EXAMPLES" sections in the
relayd.conf(5) manpage for more details.

ok benno@, manpage bits jmc@

time_t 64bit fixes for relayd and relayctl:
- fix statistics
- set INT_MAX limit on session timeouts
- make sure we dont use to large session timeouts in pf redirects and
openssl

tested with old and new time_t

ok florian@

sync yyerror() with bgpd; use vlog() to log parser errors so they show in
logs if they occur when reloading. ok benno@

Support additional scheduling algorithms in the load balancer:
least-states, random, source-hash. least-states is currently only
supported for redirections and the other ones are currently only
supported by relays.

ok benno@

spacing

Inherit and pass the relay table flags correctly.

Support more than one relay backup table. Instead of duplicating the
code for main and backup table all over the place, turn the relay
tables into a list attached to the relay. This improves the code and
allows some other tricks with multiple tables later.

do not overwrite the table timeout with the global timeout when a
table timeout is set.
ok sthen@ giovanni@ henning@

fix "label string" in http protocol. problem found by giovanni.
ok giovanni@, henning@

fix some leaks
ok krw@

Only start the child processes after all of them reported to have loaded
the config. Solves a race at startup time where processes can send status
messages about hosts that other processes don't know about yet.
(and have relayd abort with "desynchronized" or "invalid host id")

ok henning pyr deraadt
solves the problem ok from benno todd

Remove global carp demote option. It is currently broken, but also flawed
by design.

ok henning pyr

During socket splicing the relayd session timeouts could not be
measured exactly in user land. Use the new idle timeout for socket
splicing in the kernel to make it correct. Also do splicing with
http if relayd does not check headers.
ok mikeb

Add additional check to prevent running scripts when not configured.

Support interface groups in address specifications for tables or
directives like "listen on egress".

Based on gilles@' code for smtpd and an idea from Mikolaj Kucharski.

Fix reload support in relayd(8) by reimplementing large parts of the
daemon infrastructure. The previous design made it fairly hard to
reload the complex data structures, especially relays and protocols.
One of the reasons was that the privsep'd relayd processes had two
ways of getting their configuration: 1) from memory after forking from
the parent process and 2) and (partially) via imsgs after reload. The
new implementation first forks the privsep'd children before the
parents loads the configuration and sends it via imsgs to them; so it
is only like 2) before. It is based on an approach that I first
implemented for iked(8) and I also fixed many bugs in the code.

Thanks to many testers including dlg@ sthen@ phessler@
ok pyr@ dlg@ sthen@

Reorganize the relayd code to use the proc.c privsep API/commodity
functions that are based on work for iked and smtpd. This simplifies
the setup of privsep processes and moves some redundant and repeated
code to a single place - which is always good from a quality and
security point of view. The relayd version of proc.c is different to
the current version in iked because it uses 1:N communications between
processes, eg. a single parent process is talking to many forked relay
children while iked only needs 1:1 communications.

ok sthen@ pyr@

Update all logging and debug functions to use the __func__ macro
instead of static function names. __func__ is C99 and perfectly fine
to use. It also avoids printing errors; for example if a statement
log_debug("foo:"..) was moved or copied from function foo() to bar()
and the log message was not updated...

Allow a user to specify the route priority

OK reyk@ claudio@ sthen@

Splicing is currently only supported for TCP relays, not for HTTP or
others (HTTP will need a more complicated splicing mechanism to switch
between headers and bodies in userland and kernel). Add the "no
splice" flag for non-TCP relays by default to indicate it in the debug
and status output.

update flags and printing of flags in debug mode, handle splicing flag.

Add support for divert-to which provides some benefits over rdr-to.

ok mikeb@

redirects are loaded as "pass in quick ... rdr-to" pf rules by default. In
some cases it is desired to load the rules as "match in" without "quick"
to allow additional filtering or applying additional rule/state options,
eg. to add an overload table for DOS mitigation. Add the optional "match"
keyword for the redirect "tag" option to change the pf rule type accordingly.

ok jsg@ mikeb@

Missing semicolon, ok henning@

remove trailing spaces and tabs from source code; no binary changes
(verified by both sthen@ and me).

ok sthen@; "just commit it" claudio@

Fix carp demotion on tables. For some reason the default values were
inherited from the table definition even though these values could
not be changed there. While there fix a memory leak in a rather strange
case.
OK phessler, jsg, pyr, sthen, deraadt

fix linecount bug with comments spanning multiple lines
problem reported with the obvious fix for bgpd by Sebastian Benoit
<benoit-lists at fb12.de>, also PR 6432
applied to all the others by yours truly. ok theo
isn't it amazing how far this parser (and more) spread?

Allow fallback tables for relays, not just redirections.
Seems reasonable to jsg, ok phessler, no response from reyk or pyr

Sync the list of initialisation steps done for the default
relay protocol with that done for specified relay protocols.

Makes it possible to use SSL for the default relay protocol.

From boudewijn@indes.com in pr 6316

allow to specify interface names as addresses, for example "listen on
em0". the implementation will lookup the first IPv4 address of an
interface before any other IPv4 and IPv6 addresses.

ok gilles@ (i got inspired by smtpd)

add new 'router' functionality to dynamically add or remove routes
based on health check results, using the existing table syntax. this
allows to maintain multiple (uplink) gateways to implement link
balancing or WAN link failover if no routing protocol or other
keepalive method is available. works fine with or without
net.inet.ip.multipath enabled.

ok pyr@, jmc@ for manpages

allow to modify the IP TTL value for host checks. this can be used to
check if the host is only n hops away and not re-routed over a longer
path.

add missing line

allow to specify host attributes in an arbitrary order (parent, retry) by
making the grammar a bit more flexible.

cosmetic change - move address into an own element

prevent configuration of relays listening to a single addr:port tuple twice

support multiple 'listen on' lines in a single relay block

tables don't need pf if running in "l7" relay mode

Allow UDP and/or TCP redirections instead of just TCP.

Thanks to Marek Grzybowski for feedback and testing.

ok jmc@ (manpage bits)

add "Connection: close" to HTTP check headers to deconfuse HTTP/1.1
servers claiming keepalive sessions.

From Camiel Dobbelaar

add support to specify a ca file (eg. /etc/ssl/cert.pem) to verify ssl
server certificates when connecting as an SSL client from relays. it
works so far, but needs more testing and is currently lacking support
for certificate revocation (like CRL or OCSP). the file ssl_privsep.c
is extended to implement more code that should be in openssl to allow
loading the ca from chroot...

fix an incorrect flag in route mode.

From Padcal Lalonde, closes PR 6114

Add support for client-side SSL connections from relays. relayd can
now sit between two SSL connections (Oitm - OpenBSD-in-the-middle),
accept SSL connections and forward to TCP, accept TCP connections and
forward to SSL, and do TCP to TCP of course.

This was tested by some people a while ago.

Fixed memory leaks which would occur if the second of two memory
allocations fails.

looks right deraadt, krw
ok henning

use HTTP/1.1 instead of HTTP/1.0 if a host header is specified.

spotted by phessler@

bring in the findeol() fix from pfctl. list of affected parsers by sthen

allow to load expect, filter, log, and remove keys from external files
just containing on key per line. this allows easier use of URL
white/blacklists from external sources.

sort tokens for better readability

allow to listen on a port range for redirections. this fixes
stickyness with web applications that cannot do the clustering on
their own and require stickyness with HTTP to HTTPS migration. this
is required in many cases; it is a true fact that we cannot always fix
the backend application in the real world.

Tested and requested by many

Add dynamic IPv6-to-IPv4 and IPv4-to-IPv6 translation inspired by
faithd(8) by doing a similar mapping of IPv4/6 addresses with
relayd(8) and pf(4) redirections without the need of the faith(4)
interface. The trick works in both directions, it can accept IPv6
connections and relay them to IPv4 hosts by extracting the last 4
octets from the IPv6 destination (like faithd(8)), and it can accept
IPv4 connections and relay them to IPv6 hosts by prepending the 4
octets of the original IPv4 destination to a configured IPv6 prefix.
An access list is not needed because the classification is done in
pf.conf(5). It helps to get more faith in relayd.

manpage bits ok jmc@
yes, sounds good todd@

no need for using a TAILQ queue for the host children list, use a
singly-linked SLIST instead. the only noticeable change is the
reversed order to notify the children but it does not really matter
here. also only walk through the children host list if the host
itself is a potential parent.

If the new 'parent' keyword is specified for a host in a table,
inherit the state from another host with the specified Id; no
additional check will be for the inheriting host. This helps in
scenarios with lots of IP aliases that all point to the same service
on the same host (like web hosting with many SSL domains).

discussed with pyr, tested in different setups

final reorder diff to use TAILQ_INSERT_TAIL instead of
TAILQ_INSERT_HEAD. now tables and relays are also matching the order
in the config file.

ok pyr@

add the hosts in order to get host ids that match the order in the config file. ok pyr@

give sane ids to hosts, tables, redirections, relays, etc. - start
counting at 1 and do not assign an id before inheriting a real table.
makes more sense in the relayctl output.

ok pyr@

use getaddrinfo/getnameinfo to parse ipv6 addresses instead of
inet_pton/inet_ntop to allow specifing and printing the IPv6 scope
identifier. synced host_v6() with ntpd's version to use getaddrinfo()
instead of inet_pton() - host_v4, host_v6, and host_dns could all use
getaddrinfo in a single function by specifing different flags but this
would diverge from the other daemons using this common interface so we
keep this little overhead.

discussed with henning@
ok pyr@

add support for "transparent" forwarding in relays: normally the l7
relay will connect to the target host with its own ip address, but
this mode will let it use the address of the client that is connecting
from the other side. for example, there is no need to add the
X-Forwarded-For HTTP headers for internal webservers in this mode
anymore since they magically see the remote client ip address in the
connection. it also allows to build fully-transparent ssl
encapsulation for tcp sessions and many other things...

based on an initial idea from dlg@ and pascoe@ (dlg's talk at opencon)
using the new BINDANY and divert-reply interfaces from markus@ (since n2k8)

ok markus@ pyr@

missed to set the default for tables to round-robin, so it was
loadbalance after the grammar change. make it round-robin again.

add an alternative "route to" mode to relayd redirections which maps
to pf route-to instead of the default rdr. it is a first steps towards
support for "direct server return" (dsr), an asynchronous mode where
the load balanced servers send the replies to a different gateway like
a l3 switch/router to handle higher amounts of return traffic.
because the state handling in pf isn't optimal for this case yet, it
just sees half of the TCP connection, the sessions are forced to time
out after fixed number of seconds.

discussed with many, thought about in the onsen

the manpage mentioned "timeout" in relay sections, while the grammar
expected the keywords "forward timeout". rename it to "session
timeout" and sync the documentation with reality.

Do not unconditionnaly load pf. If pf isn't required by the configuration
the initialisation isn't done properly.

Inherit global table options.

From Armin Wolfermann

ok pyr@ deraadt@

Unbreak parser by initializing topfile correctly.
I got fooled by patch(1). Sorry.

Have popfile() also close the main config file,
but only do the final popfile call after yyparse() is done.
This also fixes config reload on SIGHUP for some daemons.

Spotted by otto@. OK deraadt@

bump copyright

stylistic change: move code to add protonodes from the BNF into
seperate functions in relayd.c (protonode_add/protonode_header). this
code got to big to look nice in the BNF statements...

unbreak the dns protocol handler, closes pr 5717

Thanks to Nigel Taylor

Marry relayd with snmpd using new "send trap" option: Request to send
a SNMP trap when the state of a host changes. relayd(8) will try to
(re-)connect to snmpd(8) and request it to send a trap to the
registered trap receivers, see snmpd.conf(5) for more information
about the configuration.

ok pyr@ thib@

add prefixes to names of structure elements to make it easier to grep
for code, next struct relay. knf long line fixes will follow later.

ok thib@

add prefixes to names of structure elements to make it easier to grep
for code, start with struct relayd. finally.

ok thib@

Free resources now that we don't always exit after an unsuccessful
parsing of the configuration.
From Igor Zinovik <zinovik at cs.karelia.ru>
ok thib@ and me.

Rename everything which reffered to services refer to rdr for internals
(for instance: rename struct service to struct rdr), refer to redirects
otherwise (hoststatectl output).
ok reyk@

make the generic handler for TCP-based protocols the default (allows
to use "protocol foo" without defining a type).

some changes to the relayd.conf configuration language and grammar.

the tables will look more like pf tables, it is easier to re-use
tables with different options, "services" will become "redirections"
(they refer to rdr pf rules), sync configuration directives of
redirect (l3, ex-service) relay (l7) sections (for example "virtual
host" will become "listen on"), all target definitions will start with
"forward to", etc. pp. (see relay.conf(5) and etc/relayd.conf)

discussed with pyr and deraadt
ok pyr@

hoststated gets renamed to relayd. easier to type, and actually says
what the daemon does - it is a relayer that pays attention to the status
of pools of hosts; not a status checkers that happens to do some relaying

allow to add labels to protocol actions, they will be printed in http
error pages and can be used to refer to additional information.

ok pyr@

sort includes, adjust to style(9)

re-implement the "mark" action and document it in the manpage:
it is possible to attach a mark to a session based on matching an
entity (header, url, cookie, ...) and add conditional action for this
mark. it works a bit like the tag/tagged keywords in pf, but i decided
to pick a different name to avoid confusion.

ok pyr@ gilles@

add (new) "url" protocol action, this can be used to match/filter URL
suffix/prefix expressions like "example.com/index.html?args". a digest
mode allows to match against anonymized SHA1/MD5 digests of
suffix/prefix expressions.

spacing

extend action grammar with "filter value" and "expect value" as a
short form for "filter * from value" or "expect * from value".

move digest string handling into an extra function.

rename the "url" filter action to "query" to use the correct term.
please update your hoststated.conf configurations. also add more
examples to the manpage.

alright pyr@

allow the http digest type to be either SHA1 or MD5 determined by the
digest string length; it is compatible to any existing SHA1-only
configurations.

ok pyr@ gilles@

allow to use the "include" directive in tables, services, relays, and
protocols.

ok pyr@

it may be desirable to send a HTTP error page with error code and a
meaningful message if a HTTP/HTTPS relay closes the connection for
some reason. for example, a "403 Forbidden" if the request was
rejected by a filter. this will be enabled with the "return error"
option and is disabled by default, the standard behaviour is to
silently drop the connection; the browser may display an empty page in
this case. the look+feel of the HTTP error page can be customized with
a CSS style sheet, but we do not intend to allow customization of the
error page contents (hoststated is not a webserver!).

ok pyr@

Allow overriding the global interval in a table.
Table specific intervals must be multiples of the global interval.
help and ok reyk@

spacing

knf (replace some ';;' with a single ';')

rework the internal handling of protocol actions a little bit:

- allow to use a key for multiple times by appending a queue of
additional matches to the tree node. for example, this allows to
specify multiple "expect" or "filter" actions to white-/black-list
a list of HTTP-headers, URLs, ..

- prevent specifing an HTTP header for multiple times when using the
expect action.

- minor code shuffling

always enable "late connect" relay mode if an "expect" or "filter"
action has been specified for the protocol. late connect mode first
reads the complete request (HTTP header) before opening the inbound
connection instead of relaying it line-by-line.

Do not insert proto_default inside the dynamically alloced protocol queue.
Handle it as a special case in the one place where it actually matters
instead.

reset global variables everytime we enter parse_config.

make protos dynamic too

Remove space/tab compression function from lgetc() and replace
it with a simple filter in the yylex() loop.
The compression in lgetc() didn't happen for quoted strings,
thus creating a regression when tabs were used in variables.
Some testing by todd@ and pyr@
OK deraadt@

Not using all defined protocols is not a hard error. Just print a warning
but start anyway. OK reyk@

load certificates text at parse time. then load them in relay processes.
this separation will ease reload a bit more.

ok reyk@ who spotted a stupid mistake again...

add support for the include directive to the configuration file parser,
based on the existing hostapd/pfctl code.

ok pyr@

do not check the file secrecy of hoststated.conf, there is no need to
enforce the file ownership and permissions to root:wheel 0400 because
we have nothing to hide.

ok pyr@

print system error when fopen fails.

Do log initialisation correctly, like bgpd does.
This removes the double warn/log_warn madness i introduced yesterday.
This also keeps messages on stderr at startup and when running with -n.

Move relays from static TAILQs to allocated ones.
This syncs it with other hoststated entities and will make reload easier.
This is step 1 out of 7 for reload.

keep log_warn messages to be informed when a failure occurs during a reload.

unbreak tree

extra arg to warn slipped through.

repair file security warnings; ok pyr

Allow '=' to end a number in all lexers.
Requested and OK deraadt@

in the lex... even inside quotes, a \ followed by space or tab should
expand to space or tab, and a \ followed by newline should be ignored
(as a line continuation). compatible with the needs of hoststated
(which has the most strict quoted string requirements), and ifstated
(where one commonly does line continuations in strings).
pointed out by mpf, discussed with pyr

in all these programs using the same pfctl-derived parse.y, re-unify the
yylex implementation and the code which interacts with yylex. this also
brings the future potential for include support to all of the parsers.
in the future please do not silly modifications to one of these files
without checking if you are de-unifying the code.
checked by developers in all these areas.

next step in the yylex unification: handle quoted strings in a nicer fashion
as found in hoststated, and make all the code diff as clean as possible. a
few issues remain mostly surrounding include support, which will likely be
added to more of the grammers soon.
ok norby pyr, others

use macro argument

cast to an int

stop messing with lgetc to please hoststated's check/expect.
instead move some of the logic in yylex and do hoststated specific
translations into hoststated.c
ok gilles@

clean up merged code.

Add NUMBER to hoststated's lexer, very similar to what has gone in
in the other daemons recently. Prompted and based on work by deraadt@
proofread and ok gilles@

keep lines < 80.

Correct my mail address.

add support for relaying DNS traffic (with a little bit of packet
header randomization). this adds an infrastructure to support
UDP-based protocols.

ok gilles@, tested by some

store the table's port as the relay's dstport

store relay sessions in SPLAY trees instead of TAILQ lists. this will
be used for faster lookups of sessions based on different criteria.

ok pyr@

Add the ability to specify a host header when using http(s) check methods.
Prodded by me, done by Gille Chehade <veins@evilkittens.org>

ok reyk, jmc for the manpage bits.

use a more traditional while() instead of for() for getopt().
sync usage() to the man page.
format string fixes.
complain about failed calloc()'s instead of exiting silently.

ok pry@,reyk@

do not forget to store table and backup table ids in the service
configuration struct.

make sure object ids are reset before parsing the configuration file
again.

allocate table lists and service lists instead of using static structs.
split the code to start the event loop in two functions.
introduce merge_config which will be used later on.

do not start relay processes when no L7 load balancing is needed.
ok reyk@

add a new check method which allows to run external scripts/programs
for custom evaluations.

pyr agrees to put it in now but to do some improvements of the timeout
handling later.

move the ssl cipher suite string to a (small) static charbuf,
this will make it easier to send the struct over the socket.

move struct relay to the runtime + config scheme.
this time around, include hoststatectl changes too.

store the configuration file's path, this will be useful when reloading.

Second step towards hoststated reload:
First split out hosts, tables and services into to structs, one that
contains the runtime fields and one (inside the runtime) that contains
mostly static fields that will be sent over the socket during reload.

Also move the demoted field of tables inside the flags field as its
just a boolean.
ok reyk@

allow to specify table templates in the configuration file and to
inherit them from multiple services or relays. this is useful if you
want to use a table with the same list of hosts but different ports as
specified in the relay or service section.

this makes mcbride more happy
ok pyr@

first steps for implementing reload:
* make parse_config allocate the hoststated function by itself
* make as many sockets as necessary to talk to the relay children
* add send_all for talking to all children
with advise and ok reyk@

add a new relay 'path' action to filter the URL path and arguments.

ok pyr@

in addition to the host retry option in tables, add support for the
optional connection "retry" to the forward to, service, and nat lookup
options. for example, "nat lookup retry 3" is useful when running
hoststated as a transparent proxy when connecting to unreliable
frontend/backend servers.

ok pyr@

allow to specify the IP_TTL and IP_MINTTL options for the relays to
support the Generalized TTL Security Mechanism (GTSM) according to RFC
3682. this is especially useful with inbound connections and a fixed
distance to the backend servers.

ok pyr@

- fix the hoststatectl host disable/enable commands to work with relay
layer 7 loadbalancing.
- allow to run relays with tables without depending on services
- show hosts and tables assigned to relays in hoststatectl show commands

ok pyr@ deraadt@ with some input from mcbride@

add support for handling simple HTTP cookies (no per-path/domain
cookies yet), for example: cookie hash "JSESSIONID"

tested by some people
ok pyr@

in addition to actions on request headers, allow to define relay
actions on response headers (the reply sent by backend HTTP servers).
the default and slightly faster relay streaming mode will be used if
no actions are defined.

for example:
response change "Server" to "OpenBSD-hoststated/4.1"

ok pyr@

kill the ``use ssl'' directive for consistency across parser directives.
another heads up for testers: you need to change configuration files.
ok reyk@

remove HTTP and HTTPS tokens, makes for cleaner parser.
reorder other rules as well.
ok reyk@

solve some conflicts in the configuration parser.
configuration will need to be updated as some directives have changed.
manpage and examples bits coming up.
ok reyk@

KNF

Change the ``virtual ip'' directive to ``virtual host''.
You will need to update your configuration files accordingly.
"just do it", reyk@

- allow to specify the SSL cipher suite and the SSL protocols
(as required by the PCI DSS)
- increase the default listen backlog to 10, allow to modify the
backlog as a per-protocol tcp option to improve the performance
on busy systems (to get less connection failures on heavy load)
- close the connection if SSL_accept returned an error
- instead of logging _new_ relay sessions to syslog, log the
sessions in relay_close() after they have been _finished_.
this will allow to collect some additional information
- add a new log keyword to log specified header/url entities (useful
to track "bad guys" using many session ids or multiple user agents)
- some minor fixes, manpage bits, and bump the copyright (by some
reason, i didn't realize that we already have 2007...).

Add layer 7 functionality to hoststated used for layer 7
loadbalancing, SSL acceleration, general-purpose TCP relaying, and
transparent proxying.

see hoststated.conf(5) and my upcoming article on undeadly.org for
details.

ok to commit deraadt@ pyr@

unbreak the symset function

carefully check some return values and make lint happier. never pass
any truncated strings (table names/anchors/tags/...) to pf and the
kernel.

ok pyr@

add new "log (updates|all)" configuration option to log state
notifications after completed host checks. either only log the
"updates" to new states or log "all" state notifications, even if the
state didn't change. the log messages will be reported to syslog or to
stderr if the daemon is running in foreground mode.

ok claudio@ pyr@

remove unused functions and variables which have been copied from
ospfd(8) (can be re-imported later if required).

add the -D option to define macros on the command line (as found in
bgpd(8), hostapd(8), ipsecctl(8), pfctl(8), ...).

small memleak plugged and style changes.
ok reyk@

Add SSL support to hoststated.
with help and OK reyk@
with help and advice by claudio@ and Srebrenko Sehic

return 0, not NULL in a function returning int.

ok pyr@

Better handling of escaped CR-LF in the configuration file, commenting
them out was previously broken. This is needed for send/expect scripts.

ok claudio@

eliminate duplicate tcp read/write code.
ok claudio@, reyk@

Finish renaming hostated to hoststated.
Note to testers: the user the daemon changes its id to is now _hoststated,
don't forget to update master.passwd.
ok reyk@

adapt to rename

do NOT use the regexp interface. it is way to complicated, error-prone
and we don't know about all the possible security problems.

change the check send/expect code to use the fnmatch(3) interface
using shell globbing rules instead. this allows simple patterns like
"220 * ESMTP*" or "SSH-[12].??-*".

suggested by deraadt@ and otto@
ok Pierre-Yves Ritschard (pyr at spootnik dot org)

allow to use service names in addition to numerical port numbers in
the configuration file, eg. "real port http".

> From Pierre-Yves Ritschard (pyr at spootnik dot org)

ok claudio@

the timeout values are not allowed to exceed the global interval (i
figured this out while testing hostated against a stottering spamd
where the send/expect timeout needs be > 10 seconds). also use another
struct timeval to store the interval for easier handling in the code.

ok Pierre-Yves Ritschard (pyr at spootnik dot org)

remove unused token.

add a generic send/expect check using regular expression (see
regex(3)). this allows to define additional checks for other TCP
protocols.

From Pierre-Yves Ritschard (pyr at spootnik dot org)

spacing

allow the sticky-address option for round-robin pools.

From Pierre-Yves Ritschard (pyr at spootnik dot org)

fix the conversion from milliseconds to struct timeval, which uses
seconds (tv_sec) and microseconds (tv_usec), but the code assumed
seconds and milliseconds...

partial rewrite of the check_* routines to use libevent everywhere
instead of nested select() calls and to handle the non-blocking
sockets properly.

From Pierre-Yves Ritschard (pyr at spootnik dot org)
(with a little help by me)

typo

spacing

knf, spacing

please note that some editors will replace tabs with multiple spaces
if you cut & paste code from other sections. please try to keep the
tabs ;).

Import hostated, the host status daemon. This daemon will monitor
remote hosts and dynamically alter pf(4) tables and redirection rules
for active server load balancing. The daemon has been written by
Pierre-Yves Ritschard (pyr at spootnik.org) and was formerly known as
"slbd".

The daemon is fully functional but it still needs some work and
cleanup so we don't link it to the build yet. Some TODOs are a
partial rewrite of the check_* routines (use libevent whenever we
can), improvement of the manpages, and general knf and cleanup.

ok deraadt@ claudio@

Unmention/don't explain SSL, drop 9y old "ssl" keyword/deprecation warning

Switch "ssl" to "tls" in relayd.conf(5) if you haven't done so in the last
ten years, "ssl" is now an error.

Say "TLS" not "SSL/TLS" and drop the primer in the TLS RELAYS section.

OK benno

Use ibuf_data() instead of accessing ibuf->buf directly.
OK tb@

Don't declare variables as "unsigned char *" that are passed to
functions that take "char *" arguments. Where such chars are
assigned to int or passed to ctype functions, explicitly cast them
to unsigned char.

For OpenBSD's clang, -Wpointer-sign has been disabled by default,
but when the parse.y code was built elsewhere, the compiler would
complain.

With help from millert@
ok benno@ deraadt@

Don't leak host address. Found with clang static analyzer.

OK tb@

Add 'strip' directive

Feedback by Olivier Cherrier, Hiltjo Posthuma, Mischa

OK benno@

getifaddrs() can return entries where ifa_addr is NULL. Check for this
before accessing anything in ifa_addr.
ok claudio@

Remove deprecated snmp keyword.

OK denis@

Let relayd make use of libagentx. No functional change intended.

OK tb@
Enthousiasm from Mischa Peters

Remove trailing spaces & tabs

Rewrite the agentx code of relayd. This new framework should allow us
to add new objects easier if so desired and should handle a lot more
corner-cases.

This commit should also fix the following:
- On most (all) tables it omits the *Entry elements, making it not map to
OPENBSD-RELAYD-MIB.txt.
- sstolen returns the size of the sockaddr_in{,6}, instead of the
sin{,6}_addr resulting in garbage data to be put in the ip-field.
- relaydSessionPortIn and relaydSessionPortOut are swapped
- relaydSessions only uses relaydSessionIndex, while
OPENBSD-RELAYD-MIB.txt says it should have 2 indices
- miscellaneous minor things related to the AGENTX-protocol, like wonky
index handeling and returning NOSUCHINSTANCE where NOSUCHOBJECT
should be returned, etc.

This commit does remove traps, but it's large enough as is and I intent
on adding it soon(tm). It also deprecates the snmp keyword in favour of
an agentx keyword. The snmp keyword is still available, but will be
removed in the future.

Tweaks and OK denis@ on the relayd parts
Tweaks and OK claudio@ on the agentx parts
"Get it in" deraadt@

Enable TLSv1.3 support in relayd(8)
with the help from tb@ jsing@; ok tb@

improve parsing of relay {} sections:
- Do not accept multiple protocol statements, as only one will be
used, it is better to error out if more arespecified.
- do not allow tcp and http options in dns protocol definitions, they
are ignored anyway.
Suggested by Nick (nick -AT- kousu -DOT- ca) thanks.
ok claudio@

remove old log options 'log update/all' that were replaced with 'log
state changes/host checks/connection' some time ago. ok reyk@
claudio@ on first version, kn@ noticed that the tokens could be
removed too.

Add support for binary protocol health checking. Feedback and guidance from
benno@ and reky@. Man page tweaks from jmc@.

ok benno@

fix error description on invalid forward ip.

OK benno@

Fix integer sizes in format strings and enable formatting warnings
for yyerror.

From Moritz Buhl
ok claudio@

When system calls indicate an error they return -1, not some arbitrary
value < 0. errno is only updated in this case. Change all (most?)
callers of syscalls to follow this better, and let's see if this strictness
helps us in the future.

Add support for SNI with new "tls keypair" option to load additional certs.

Tested by many (thanks!)

Feedback & OK rob@

Move the relay keys/certs into a separate global list and look them up by id.

Moving the certs out of the relay struct will help to add multiple SNI certs.

Tested by many users (thanks!)

Feedback & OK rob@

Fix the check if a relay has been specified twice

Relays cannot have the same name or listen address. If a listen
address is specified multiple times, the parser expands the
configuration into multiple relays automatically.

OK rob@

Move relay_load_*() functions into relayd.c

Pass the *env as an explicit argument instead of using the global
pointer: The relay_load_certfiles() function is called early before
the *env is set up. This does not change anything in the current code
as *env is not used by anything in the function (not even
ssl_load_key() that is taking it as an argument) but it will be needed
by upcoming changes for SNI.

Ok rob@

Add support for from/to in relay filter rules.

For example,
pass from 10.0.0.0/8 path "/hello/*" forward to <b>

Ok benno@

remove unused keyword "virtual".
ok gcc, claudio@ agrees

Support for rfc 6455 Websockets connection upgrade. Add a new protocol
option 'http { [no] websockets }' to allow such connections (default
is no). Original diff from Daniel Lamando (dan AT danopia DOT net),
option and header checks by me. suggestions and ok bluhm@ and earlier
diff claudio@

(unsigned) means (unsigned int) which on ptrdiff_t or size_t or other
larger types really is a range reduction...
Almost any cast to (unsigned) is a bug.
ok millert tb benno

- odd condition/test in PF lexer
(and other lexers too)

This commit rectifies earlier change:

in the lex... even inside quotes, a \ followed by space or tab should
expand to space or tab, and a \ followed by newline should be ignored
(as a line continuation). compatible with the needs of hoststated
(which has the most strict quoted string requirements), and ifstated
(where one commonly does line continuations in strings).

OK deraadt@, OK millert@

Make host_*() AF-agnostic

Merge host_v{4,6}() into much simpler host_ip() using just getaddrinfo().

With input & test by kn@ and benno@

OK benno@ kn@

replace malloc()+strlcpy() with strndup() in cmdline_symset().

"looks good" gilles@ halex@

replace the current log options

log updates|all

with

log state changes
log host checks
log connection [errors]

The first two control the logging of host check results: either changes in host state only or
all checks.

The third option controls logging of connections in relay mode:
Either log all connections, or only errors.

Additionaly, errors will be logged with LOG_WARN and good connections
will be logged with LOG_INFO, so they can be differentiated in syslog.

ok and feedback from claudio@

Do for most running out of memory err() what was done for most running
out of memory log_warn(). i.e. ("%s", __func__) instead of manual
function names and redundant verbiage about which wrapper detected the
out of memory condition.

ok henning@

No need to mention which memory allocation entry point failed (malloc,
calloc or strdup), we just need to log that we ran out of memory in a
particular function.

Recommended by florian@ and deraadt@

ok benno@ henning@ tb@

Fix an off-by-one line count when using include statements.

Thanks to otto@ for the initial diff.

OK benno@

Plug leak in error case of the common 'varset' implementations.

ok benno@

Remove RELAY_MAX_SESSIONS from relayd, there is no reason to limit relays
to 1024 session per process (esp. with keep-alive). Now the fd limit is
the new maximum and relayd will make sure to not accept too many sessions.
The tcp backlog config maximum is now 512, adjust manpage accordingly.
OK benno@ deraadt@

add options to specify the control socket in relayd and relayctl.
From Kapetanakis Giannis, thanks.
ok claudio@

Change the ecdhe curve configuration to the same way httpd is doing it.
This removes 'no ecdh' and renames 'ecdh curve auto' to ecdhe default.
The code uses now tls_config_set_ecdhecurves(3) so it is possible to
specify multiple curves now. If people specified curves in their config
they need to adjust their config now.
OK beck@

Use file descriptor passing to load certificates into the relays. Especially
the ca file (having all the trusted certs in them) can be so big that loading
via imsg fails.
OK beck@

Check that http options are only configured in http protocols.
OK benno@

make the maximum size of http headers configurable in the protocol.
ok bluhm@, >8k makes sense claudio@

65535 is a valid port to listen on.
Off-by-one pointed out by and diff from Kris Katterjohn katterjohn AT
gmail, thanks!
chris@ pointed out that more than httpd(8) is effected.
OK gilles@

Migrate relayd to use libtls for TLS. Still does the TLS privsep via the
engine but at least we can use a sane API for new features.
Going in now so it is possible to work with this in tree.
General agreement at d2k17.

Replace hand-rolled for(;;) emptying of 'symhead' TAILQ with more
modern TAILQ_FOREACH_SAFE().

No intentional functional change.

ok millert@ bluhm@ gilles@

Replace symset()'s hand-rolled for(;;) traversal of 'symhead' TAILQ
with more modern TAILQ_FOREACH(). This what symget() was already
doing.

Add paranoia '{}' around body of symget()'s TAILQ_FOREACH().

No intentional functional change.

ok bluhm@ otto@

spacing

Replace [RELAY|SERVER]_MAXPROC with the new PROC_MAX_INSTANCES
variable and limit it from 128 to 32 instances (the old value).
While here, move a few PROC_ defines around.

OK rzalamena@

Move snmp options into struct relayd_config and delay start of the
snmp subsystem until the configuration is done.

OK benno@ claudio@

Split "struct relayd" into two structs: "struct relayd" and "struct
relayd_config". This way we can send all the relevant global
configuration to the children, not just the flags and the opts.

With input from and
OK claudio@ benno@

Switch from the not really working session cache (because of the multiprocess
nature of relayd) to tls session tickets to do TLS session resumption.
TLS session tickets do not need to store SSL session data in the server but
instead send an encrypted ticket to the clients that allows to resume the
session. This is mostly stateless (apart from the encryption keys).
relayd now ensures that all relay processes use the same key to encrypt
the tickets. Keys are rotated every 2h and there is a primary and backup key.
The tls session timeout is set to 2h to hint to the clients how long the
session tickets is supposed to be alive.
Input and OK benno@, reyk@

do not allow whitespace in macro names, i.e. "this is" = "a variable".
change this in all config parsers in our tree that support macros.
problem reported by sven falempin.

feedback from henning@, stsp@, deraadt@
ok florian@ mikeb@

Update log.c: change fatal() and fatalx() into variadic functions,
include the process name, and replace all calls of fatal*(NULL) with
fatal(__func__) for better debugging.

OK benno@

stdlib.h is in scope; do not cast malloc/calloc/realloc*
ok millert krw

Fix obvious problems with relayd config reload.
- fix a TAILQ corruption because of a use after free
- do not reinit the SSL engine since that fails
OK sthen, benno

branches: 1.203.2;
Use AI_ADDRCONFIG when resolv hosts on startup.

OK henning@

Clean up the relayd headers with help of include-what-you-use and some
manual review. Based on common practice, relayd.h now includes the
necessary headers for itself.

OK benno@

Include <netinet/in.h> before <net/pfvar.h>. In a future change when
ports is ready, <net/pfvar.h> will stop including a pile of balony.

Adapt to <limits.h> universe.
ok millert

pf now supports source-hash and random with tables so we can allow it
in redirections. Thanks for help and input from jsg and yasuoka who
reminded me to dig out and update these old diffs for pf and relayd.

ok jsg@

Stop pulling in <arpa/inet.h> or <arpa/nameser.h> when unnecessary.
*Do* pull it in when in_{port,addr}_h is needed and <netinet/in.h> isn't.

ok reyk@

Update relayd to use siphash instead of sys/hash. The source-hash,
loadbalance and hash modes use a random key by default that can be
forced to be a static key with a new configuration argument.

With input from Max Fillinger.

ok tedu@

Change the keyword "ssl" to "tls" to reflect reality since we
effectively disabled support for the SSL protocols. SSL remains a
common term describing SSL/TLS, there is some controvery about this
change, and the name really doesn't matter, but I feel confident about
it now.

(btw., sthen@ pointed out some historical context:
http://tim.dierks.org/2014/05/security-standards-and-name-changes-in.html)

OK benno@, with input from tedu@

Don't allow embedded nul characters in strings.
Fixes a pfctl crash with an anchor name containing
an embedded nul found with the afl fuzzer.

pfctl parse.y patch from and ok deraadt@

Support exporting relayd statistics via AgentX/snmpd

This should be equivalent to the statistics available
via the various relaydctl show commands

okay benno@ reyk@

Remove the sslv2 option since LibreSSL has no SSLv2 support (however retain
SSL_OP_NO_SSLv2 in case you happen to be running relayd on another platform
with another SSL library). Also fix the SSLv3 handling so that 'no sslv3'
actually works as intended.

ok reyk@

Convert the logic in yyerror(). Instead of creating a temporary
format string, create a temporary message.
OK deraadt@

Remove the "interface" option from the "transparent forward" directive.
It was mandatory in the grammar but never used in the code.

A fully transparent relay can now be specified with the following
directive in a relay block: "transparent forward to destination".

OK sthen@

Disable SSLv3 by default.

OK sthen@ jsing@

revert previous; was based on a work-in-progress, as well
as being an incomplete and therefore incorrect adaptation

apologies to anybody who got bitten by this mistake

ok reyk@

Implement consistent host hashing for relayd, based on
work done by andre@

Re-add a randomized hash seed (which had apparently
gotten inadvertently removed in the past).

Allows for multiple relayd instances to be configured
to forward traffic to the same host, falling back to
the random seed when not explicitly configured to do so.

ok reyk@

Sometimes I just sort the tokens in parse.y

Add support for EDH to provide perfect forward secrecy for older SSL
clients. Additionally, add options for disallowing client-initiated
renegotiations and to prefer the server's cipher list over the
client's preferences.

This work is based on a diff by Markus Gebert at hostpoint.ch, and was
discussed with jsing@ resulting in a few different defaults.

ok benno@

Simplify the code that handles the HTTP headers by using an RB tree
with associated lists instead of the complicated lookup table and
"others" list. This might add a little malloc overhead for common
headers but also fixes some issues like the handling of repeated
headers - for example, handling of multiple "Set-Cookie" headers.

ok bluhm@ (regress part)
ok benno@

Replace the protocol directives for HTTP with a new generic filtering
language. The grammar is inspired by pf and allows to write versatile
last-matching filter rules in protocol sections starting with the
"pass", "block" or "match" keywords. This work was started almost two
years ago and replaces large parts of relayd(8)'s HTTP and filtering
code. The initial version reimplements and extends HTTP filtering,
but will be improved to support generic TCP and other protocols later.

With some testing, feedback, and help from benno@ and andre@.

OK benno@

sync copyright to reality according to my last changes

Fix a leak from a recent added mark/marked keywords check.

ok reyk

fail for unsupported node action/type combinations. Also fail for the
unsupported mark/marked combination in a single rule.

ok andre@

Support the CA key for SSL inspection in the ca process. Instead of
looking up the keys by relay id, add all keys to a list and look them
up by key id.

ok benno@

Add a few missing free's in the grammar.

Add a few more overflow checks for strlc* functions in parse.y

remove 'restricted' symbol from yacc parser, overlooked in previous
commit

found by andre@

Adapt relayd to use AgentX protocol to send traps

ok reyk@ benno@

relax the cfg file secrecy check slightly to allow group readability
default permissions and mtree NOT changed.
prodded by benno, ok phessler benno jmatthew theo pelikan florian

fix a a double free caused by a config with two listen on statements
in a relay (the first one with ssl).
found and fixed by Erik Lax <erik AT halon DOT se>
ok phessler

use u_char for buffers in yylex, for ctype calls
found by millert@, ok deraadt@

Add support for ECDHE (Elliptic curve Diffie-Hellman) to enable
TLS/SSL Perfect Forward Secrecy (PFS).

ok djm@

Support SSL inspection, the ability to transparently filter in SSL/TLS
connections (eg. HTTPS) by using a local CA that is accepted by the
clients. See the "SSL RELAYS" and "EXAMPLES" sections in the
relayd.conf(5) manpage for more details.

ok benno@, manpage bits jmc@

time_t 64bit fixes for relayd and relayctl:
- fix statistics
- set INT_MAX limit on session timeouts
- make sure we dont use to large session timeouts in pf redirects and
openssl

tested with old and new time_t

ok florian@

sync yyerror() with bgpd; use vlog() to log parser errors so they show in
logs if they occur when reloading. ok benno@

Support additional scheduling algorithms in the load balancer:
least-states, random, source-hash. least-states is currently only
supported for redirections and the other ones are currently only
supported by relays.

ok benno@

spacing

Inherit and pass the relay table flags correctly.

Support more than one relay backup table. Instead of duplicating the
code for main and backup table all over the place, turn the relay
tables into a list attached to the relay. This improves the code and
allows some other tricks with multiple tables later.

do not overwrite the table timeout with the global timeout when a
table timeout is set.
ok sthen@ giovanni@ henning@

fix "label string" in http protocol. problem found by giovanni.
ok giovanni@, henning@

fix some leaks
ok krw@

Only start the child processes after all of them reported to have loaded
the config. Solves a race at startup time where processes can send status
messages about hosts that other processes don't know about yet.
(and have relayd abort with "desynchronized" or "invalid host id")

ok henning pyr deraadt
solves the problem ok from benno todd

Remove global carp demote option. It is currently broken, but also flawed
by design.

ok henning pyr

During socket splicing the relayd session timeouts could not be
measured exactly in user land. Use the new idle timeout for socket
splicing in the kernel to make it correct. Also do splicing with
http if relayd does not check headers.
ok mikeb

Add additional check to prevent running scripts when not configured.

Support interface groups in address specifications for tables or
directives like "listen on egress".

Based on gilles@' code for smtpd and an idea from Mikolaj Kucharski.

Fix reload support in relayd(8) by reimplementing large parts of the
daemon infrastructure. The previous design made it fairly hard to
reload the complex data structures, especially relays and protocols.
One of the reasons was that the privsep'd relayd processes had two
ways of getting their configuration: 1) from memory after forking from
the parent process and 2) and (partially) via imsgs after reload. The
new implementation first forks the privsep'd children before the
parents loads the configuration and sends it via imsgs to them; so it
is only like 2) before. It is based on an approach that I first
implemented for iked(8) and I also fixed many bugs in the code.

Thanks to many testers including dlg@ sthen@ phessler@
ok pyr@ dlg@ sthen@

Reorganize the relayd code to use the proc.c privsep API/commodity
functions that are based on work for iked and smtpd. This simplifies
the setup of privsep processes and moves some redundant and repeated
code to a single place - which is always good from a quality and
security point of view. The relayd version of proc.c is different to
the current version in iked because it uses 1:N communications between
processes, eg. a single parent process is talking to many forked relay
children while iked only needs 1:1 communications.

ok sthen@ pyr@

Update all logging and debug functions to use the __func__ macro
instead of static function names. __func__ is C99 and perfectly fine
to use. It also avoids printing errors; for example if a statement
log_debug("foo:"..) was moved or copied from function foo() to bar()
and the log message was not updated...

Allow a user to specify the route priority

OK reyk@ claudio@ sthen@

Splicing is currently only supported for TCP relays, not for HTTP or
others (HTTP will need a more complicated splicing mechanism to switch
between headers and bodies in userland and kernel). Add the "no
splice" flag for non-TCP relays by default to indicate it in the debug
and status output.

update flags and printing of flags in debug mode, handle splicing flag.

Add support for divert-to which provides some benefits over rdr-to.

ok mikeb@

redirects are loaded as "pass in quick ... rdr-to" pf rules by default. In
some cases it is desired to load the rules as "match in" without "quick"
to allow additional filtering or applying additional rule/state options,
eg. to add an overload table for DOS mitigation. Add the optional "match"
keyword for the redirect "tag" option to change the pf rule type accordingly.

ok jsg@ mikeb@

Missing semicolon, ok henning@

remove trailing spaces and tabs from source code; no binary changes
(verified by both sthen@ and me).

ok sthen@; "just commit it" claudio@

Fix carp demotion on tables. For some reason the default values were
inherited from the table definition even though these values could
not be changed there. While there fix a memory leak in a rather strange
case.
OK phessler, jsg, pyr, sthen, deraadt

fix linecount bug with comments spanning multiple lines
problem reported with the obvious fix for bgpd by Sebastian Benoit
<benoit-lists at fb12.de>, also PR 6432
applied to all the others by yours truly. ok theo
isn't it amazing how far this parser (and more) spread?

Allow fallback tables for relays, not just redirections.
Seems reasonable to jsg, ok phessler, no response from reyk or pyr

Sync the list of initialisation steps done for the default
relay protocol with that done for specified relay protocols.

Makes it possible to use SSL for the default relay protocol.

From boudewijn@indes.com in pr 6316

allow to specify interface names as addresses, for example "listen on
em0". the implementation will lookup the first IPv4 address of an
interface before any other IPv4 and IPv6 addresses.

ok gilles@ (i got inspired by smtpd)

add new 'router' functionality to dynamically add or remove routes
based on health check results, using the existing table syntax. this
allows to maintain multiple (uplink) gateways to implement link
balancing or WAN link failover if no routing protocol or other
keepalive method is available. works fine with or without
net.inet.ip.multipath enabled.

ok pyr@, jmc@ for manpages

allow to modify the IP TTL value for host checks. this can be used to
check if the host is only n hops away and not re-routed over a longer
path.

add missing line

allow to specify host attributes in an arbitrary order (parent, retry) by
making the grammar a bit more flexible.

cosmetic change - move address into an own element

prevent configuration of relays listening to a single addr:port tuple twice

support multiple 'listen on' lines in a single relay block

tables don't need pf if running in "l7" relay mode

Allow UDP and/or TCP redirections instead of just TCP.

Thanks to Marek Grzybowski for feedback and testing.

ok jmc@ (manpage bits)

add "Connection: close" to HTTP check headers to deconfuse HTTP/1.1
servers claiming keepalive sessions.

From Camiel Dobbelaar

add support to specify a ca file (eg. /etc/ssl/cert.pem) to verify ssl
server certificates when connecting as an SSL client from relays. it
works so far, but needs more testing and is currently lacking support
for certificate revocation (like CRL or OCSP). the file ssl_privsep.c
is extended to implement more code that should be in openssl to allow
loading the ca from chroot...

fix an incorrect flag in route mode.

From Padcal Lalonde, closes PR 6114

Add support for client-side SSL connections from relays. relayd can
now sit between two SSL connections (Oitm - OpenBSD-in-the-middle),
accept SSL connections and forward to TCP, accept TCP connections and
forward to SSL, and do TCP to TCP of course.

This was tested by some people a while ago.

Fixed memory leaks which would occur if the second of two memory
allocations fails.

looks right deraadt, krw
ok henning

use HTTP/1.1 instead of HTTP/1.0 if a host header is specified.

spotted by phessler@

bring in the findeol() fix from pfctl. list of affected parsers by sthen

allow to load expect, filter, log, and remove keys from external files
just containing on key per line. this allows easier use of URL
white/blacklists from external sources.

sort tokens for better readability

allow to listen on a port range for redirections. this fixes
stickyness with web applications that cannot do the clustering on
their own and require stickyness with HTTP to HTTPS migration. this
is required in many cases; it is a true fact that we cannot always fix
the backend application in the real world.

Tested and requested by many

Add dynamic IPv6-to-IPv4 and IPv4-to-IPv6 translation inspired by
faithd(8) by doing a similar mapping of IPv4/6 addresses with
relayd(8) and pf(4) redirections without the need of the faith(4)
interface. The trick works in both directions, it can accept IPv6
connections and relay them to IPv4 hosts by extracting the last 4
octets from the IPv6 destination (like faithd(8)), and it can accept
IPv4 connections and relay them to IPv6 hosts by prepending the 4
octets of the original IPv4 destination to a configured IPv6 prefix.
An access list is not needed because the classification is done in
pf.conf(5). It helps to get more faith in relayd.

manpage bits ok jmc@
yes, sounds good todd@

no need for using a TAILQ queue for the host children list, use a
singly-linked SLIST instead. the only noticeable change is the
reversed order to notify the children but it does not really matter
here. also only walk through the children host list if the host
itself is a potential parent.

If the new 'parent' keyword is specified for a host in a table,
inherit the state from another host with the specified Id; no
additional check will be for the inheriting host. This helps in
scenarios with lots of IP aliases that all point to the same service
on the same host (like web hosting with many SSL domains).

discussed with pyr, tested in different setups

final reorder diff to use TAILQ_INSERT_TAIL instead of
TAILQ_INSERT_HEAD. now tables and relays are also matching the order
in the config file.

ok pyr@

add the hosts in order to get host ids that match the order in the config file. ok pyr@

give sane ids to hosts, tables, redirections, relays, etc. - start
counting at 1 and do not assign an id before inheriting a real table.
makes more sense in the relayctl output.

ok pyr@

use getaddrinfo/getnameinfo to parse ipv6 addresses instead of
inet_pton/inet_ntop to allow specifing and printing the IPv6 scope
identifier. synced host_v6() with ntpd's version to use getaddrinfo()
instead of inet_pton() - host_v4, host_v6, and host_dns could all use
getaddrinfo in a single function by specifing different flags but this
would diverge from the other daemons using this common interface so we
keep this little overhead.

discussed with henning@
ok pyr@

add support for "transparent" forwarding in relays: normally the l7
relay will connect to the target host with its own ip address, but
this mode will let it use the address of the client that is connecting
from the other side. for example, there is no need to add the
X-Forwarded-For HTTP headers for internal webservers in this mode
anymore since they magically see the remote client ip address in the
connection. it also allows to build fully-transparent ssl
encapsulation for tcp sessions and many other things...

based on an initial idea from dlg@ and pascoe@ (dlg's talk at opencon)
using the new BINDANY and divert-reply interfaces from markus@ (since n2k8)

ok markus@ pyr@

missed to set the default for tables to round-robin, so it was
loadbalance after the grammar change. make it round-robin again.

add an alternative "route to" mode to relayd redirections which maps
to pf route-to instead of the default rdr. it is a first steps towards
support for "direct server return" (dsr), an asynchronous mode where
the load balanced servers send the replies to a different gateway like
a l3 switch/router to handle higher amounts of return traffic.
because the state handling in pf isn't optimal for this case yet, it
just sees half of the TCP connection, the sessions are forced to time
out after fixed number of seconds.

discussed with many, thought about in the onsen

the manpage mentioned "timeout" in relay sections, while the grammar
expected the keywords "forward timeout". rename it to "session
timeout" and sync the documentation with reality.

Do not unconditionnaly load pf. If pf isn't required by the configuration
the initialisation isn't done properly.

Inherit global table options.

From Armin Wolfermann

ok pyr@ deraadt@

Unbreak parser by initializing topfile correctly.
I got fooled by patch(1). Sorry.

Have popfile() also close the main config file,
but only do the final popfile call after yyparse() is done.
This also fixes config reload on SIGHUP for some daemons.

Spotted by otto@. OK deraadt@

bump copyright

stylistic change: move code to add protonodes from the BNF into
seperate functions in relayd.c (protonode_add/protonode_header). this
code got to big to look nice in the BNF statements...

unbreak the dns protocol handler, closes pr 5717

Thanks to Nigel Taylor

Marry relayd with snmpd using new "send trap" option: Request to send
a SNMP trap when the state of a host changes. relayd(8) will try to
(re-)connect to snmpd(8) and request it to send a trap to the
registered trap receivers, see snmpd.conf(5) for more information
about the configuration.

ok pyr@ thib@

add prefixes to names of structure elements to make it easier to grep
for code, next struct relay. knf long line fixes will follow later.

ok thib@

add prefixes to names of structure elements to make it easier to grep
for code, start with struct relayd. finally.

ok thib@

Free resources now that we don't always exit after an unsuccessful
parsing of the configuration.
From Igor Zinovik <zinovik at cs.karelia.ru>
ok thib@ and me.

Rename everything which reffered to services refer to rdr for internals
(for instance: rename struct service to struct rdr), refer to redirects
otherwise (hoststatectl output).
ok reyk@

make the generic handler for TCP-based protocols the default (allows
to use "protocol foo" without defining a type).

some changes to the relayd.conf configuration language and grammar.

the tables will look more like pf tables, it is easier to re-use
tables with different options, "services" will become "redirections"
(they refer to rdr pf rules), sync configuration directives of
redirect (l3, ex-service) relay (l7) sections (for example "virtual
host" will become "listen on"), all target definitions will start with
"forward to", etc. pp. (see relay.conf(5) and etc/relayd.conf)

discussed with pyr and deraadt
ok pyr@