Replace protocol literal strings and strlen() calls with defined constants

OK tb@ claudio@

Also download SPLs via rsync

OK tb@

Require files to be of a minimum size in the RRDP & RSYNC transports

Picked 100 bytes as a minimum, to accommodate future signature schemes
(such as the smaller P-256) and small files like empty CRLs.

With and OK claudio@ tb@

Don't set directory modtimes to match the source

When syncing against remote repositories, the modtimes of the
remote directories is irrelevant. In the RRDP protocol the directory
modtimes aren't signalled either. This should save some IOPS.

OK tb@

spelling fixes; from paul tagliamonte
any parts of his diff not taken are noted on tech

Only include assert.h if we call assert()

OK tb@

Add support for draft-ietf-sidrops-signed-tal-12

Add support validation of Signed Objects containing Trust Anchor Keys
(TAKs - aka 'Signed TALs'). Signed TALs provide a mechanism for RIRs
to distribute and sign the next Trust Anchor with the current Trust
Anchor. This might be an improvement over visiting RIR websites and
copy+pasting TAL data by hand.

OK tb@

extra newline

Rework the rsync proc code. Use a proper queue of requests and enforce
the limit on that queue instead of stopping to read new messages.
This is needed to implement an abort request.
"There is not enough RB_TREE in this diff" tb@

Make the http code respect MAX_CONN_TIMEOUT and fail connects once they
hit this timeout. This is in line with the rsync code.
OK tb@ job@

Unify the maximum idle IO timeout for RSYNC & HTTPS

OK claudio@

Set rsync connection timeout to 15 seconds.

OK sthen@

Introduce MAX_HTTP_REQUESTS and MAX_RSYNC_REQUESTS.
These just replace MAX_CONNECTIONS and MAX_RSYNC_PROCESSES to be more unified.
OK tb@

more whitespace cleanups

Fix various annoying whitespace errors.

Refactor on how the subprocesses are started.

Move the unveil and pledges to the actuall subprocesses and put all the
common code to start these into process_start(). Reduces the lenght of
main() a fair bit.
OK tb@

Change from a dynamic allocation for the process list to a static
array because the maximum size. The number of processes was already
limited by stopping to poll for new commands but this enforces it
even more. With this remove the FIXME comment since it is no longer
true.
OK tb@

Sync & permit ASPA objects to appear on Manifests

OK tb@ claudio@

Implement but don't use code to use rsync's --compare-dest feature.
One gotcha is that the path passed to --compare-dest needs to be relative
to the dst directory. rsync_fixup_dest() will prepend the necessary ../
for that by counting number of '/' in dst.
OK tb@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Limit the number of rsync processes being spawned by stopping to accept
new requests when over the limit. Use a generous limit of 16.
OK deraadt@

Don't fetch files larger than 2MB

OK claudio@

Rename io_buf_new to io_new_buffer and io_buf_close to io_close_buffer.
With this the write functions are all of the form io_xyz_buffer.
Remove some prototypes of functions I forgot to remove in previous commit.
OK benno@

Finnally move away from blocking reads in rpki-client. The code was a
mish mash of poll, non-blocking writes and blocking reads. Using the
introduced ibuf size header in io_buf_new()/io_buf_close() the read
side can be changed to pull in a full ibuf and only start the un-marshal
once all data has been read.
OK benno@

First step of cleanup in the io land. Introduce io_buf_new() and
io_buf_close(). These function will inject a size of the the buffer
at the beginning of the buffer and will allow the read size to be
switched to proper async IO.
OK benno@

branches: 1.25.4;
RPKI only cares about *.{cer,crl,gbr,mft,roa} files. Use rsync --include
and --exclude to only fetch those files from the CA repositories.
OK job@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.23.2;
Abate superfluous lines from remote servers

OK claudio@

Fail in rsync_base_uri() if the strdup calls fail. Do not bubble this
error upwards since a NULL return represents a bad-URI.
Diff originally from tb@

Use the same way to error out in out of memory situation.
Just use 'err(1, NULL);' there is no need to include the type of function
that failed since it is still impossible to locate the right call.
Just use a debugger in that case.
OK tb@ deraadt@

Remove NOTREACHED marker, it should be obvious when the code is:
exit(rc);
/* NOTREACHED */

Adjust pledge() and unveil() calls for proc_rsync() a bit. Since the
mkdir was moved to the main process there is no need for access to .
in the rsync process.
OK job@ deraadt@

Move the mkpath() call from the rsync path to the main process. This allows
to drop cpath from the rsync proc pledge (down to "stdio proc exec").
This will also make work easier with the upcoming http fetcher.
OK tb@

Rework the repository handling. Split the handling of trust anchors into
ta_lookup() while regular repositories (to fetch .mft files) are handled
by repo_lookup(). Also the cache directory layout changed; moving the
trust anchors to ./ta/{tal basename}/ the other repositories end up in
./rsync/
OK tb@

Use mkpath() == -1 to check for failure. No functional change.

Adjust the repository handling a bit. Instead of storing host/module pairs
store repo (rsync URI) and local (the local path to the repository).
Simplifies the the rsync handling a fair bit.
OK deraadt@

rsync is using buffered output now, so remove this FIXME comment

Start using the ibuf API (ibuf_dynamic, ibuf_add, ibuf_close) for writing
data between processes. This completely decouples the write side.
rpki-client can't really use the imsg framework but it can use the ibuf
bits wich imsg is built on.
OK benno@ job@

Now that a NULL string is marshalled as NULL again we can drop some
extra has_xyz integers to indicate if the following buffer is present
or not. At the same time sprinkle some asserts for strings which must
be not NULL.
OK tb@

Remove the last users of io_*_write functions that call io_simple_write()
internally. This is a step in direction of more async aware io in rpki-client.
Now everything uses a buffer which is then written.
OK tb@

Kill connection if rsync server stalls

OK deraadt@

Include openssl/x509.h in extern.h since it uses a few of the typedefs from
there in structs and prototypes. Remove the openssl/ssl.h and other strange
openssl includes in the .c files that don't use openssl specific functions.
OK beck@ and tb@

Move the proc_rsync and with that the rsync processing into rsync.c
main.c is too crowded
OK deraadt@

Handle the TAL files in the master process and pass them as buffer to the
parser process. This way the parser never needs to read outside of the
cache directory which makes the unveil simpler. Additionally rsync_uri_parse
no longer needs to know about .tal files so there is now no chance to sneak
in a .tal file later on.
OK deraadt@

use $OpenBSD$ headers

indentation adjustments, in particular near warn statements
ok claudio

swap comparisons

system includes first, always.

Don't do -portable in base. It is better done outside the tree.
Imagine if we did it throughout the tree, how many copies of strlcpy
would we have, and how much time would all the configure shell scripts
and includes take? It would be ludicrous.

branches: 1.1.1;
Initial revision

Also download SPLs via rsync

OK tb@

Require files to be of a minimum size in the RRDP & RSYNC transports

Picked 100 bytes as a minimum, to accommodate future signature schemes
(such as the smaller P-256) and small files like empty CRLs.

With and OK claudio@ tb@

Don't set directory modtimes to match the source

When syncing against remote repositories, the modtimes of the
remote directories is irrelevant. In the RRDP protocol the directory
modtimes aren't signalled either. This should save some IOPS.

OK tb@

spelling fixes; from paul tagliamonte
any parts of his diff not taken are noted on tech

Only include assert.h if we call assert()

OK tb@

Add support for draft-ietf-sidrops-signed-tal-12

Add support validation of Signed Objects containing Trust Anchor Keys
(TAKs - aka 'Signed TALs'). Signed TALs provide a mechanism for RIRs
to distribute and sign the next Trust Anchor with the current Trust
Anchor. This might be an improvement over visiting RIR websites and
copy+pasting TAL data by hand.

OK tb@

extra newline

Rework the rsync proc code. Use a proper queue of requests and enforce
the limit on that queue instead of stopping to read new messages.
This is needed to implement an abort request.
"There is not enough RB_TREE in this diff" tb@

Make the http code respect MAX_CONN_TIMEOUT and fail connects once they
hit this timeout. This is in line with the rsync code.
OK tb@ job@

Unify the maximum idle IO timeout for RSYNC & HTTPS

OK claudio@

Set rsync connection timeout to 15 seconds.

OK sthen@

Introduce MAX_HTTP_REQUESTS and MAX_RSYNC_REQUESTS.
These just replace MAX_CONNECTIONS and MAX_RSYNC_PROCESSES to be more unified.
OK tb@

more whitespace cleanups

Fix various annoying whitespace errors.

Refactor on how the subprocesses are started.

Move the unveil and pledges to the actuall subprocesses and put all the
common code to start these into process_start(). Reduces the lenght of
main() a fair bit.
OK tb@

Change from a dynamic allocation for the process list to a static
array because the maximum size. The number of processes was already
limited by stopping to poll for new commands but this enforces it
even more. With this remove the FIXME comment since it is no longer
true.
OK tb@

Sync & permit ASPA objects to appear on Manifests

OK tb@ claudio@

Implement but don't use code to use rsync's --compare-dest feature.
One gotcha is that the path passed to --compare-dest needs to be relative
to the dst directory. rsync_fixup_dest() will prepend the necessary ../
for that by counting number of '/' in dst.
OK tb@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Limit the number of rsync processes being spawned by stopping to accept
new requests when over the limit. Use a generous limit of 16.
OK deraadt@

Don't fetch files larger than 2MB

OK claudio@

Rename io_buf_new to io_new_buffer and io_buf_close to io_close_buffer.
With this the write functions are all of the form io_xyz_buffer.
Remove some prototypes of functions I forgot to remove in previous commit.
OK benno@

Finnally move away from blocking reads in rpki-client. The code was a
mish mash of poll, non-blocking writes and blocking reads. Using the
introduced ibuf size header in io_buf_new()/io_buf_close() the read
side can be changed to pull in a full ibuf and only start the un-marshal
once all data has been read.
OK benno@

First step of cleanup in the io land. Introduce io_buf_new() and
io_buf_close(). These function will inject a size of the the buffer
at the beginning of the buffer and will allow the read size to be
switched to proper async IO.
OK benno@

branches: 1.25.4;
RPKI only cares about *.{cer,crl,gbr,mft,roa} files. Use rsync --include
and --exclude to only fetch those files from the CA repositories.
OK job@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.23.2;
Abate superfluous lines from remote servers

OK claudio@

Fail in rsync_base_uri() if the strdup calls fail. Do not bubble this
error upwards since a NULL return represents a bad-URI.
Diff originally from tb@

Use the same way to error out in out of memory situation.
Just use 'err(1, NULL);' there is no need to include the type of function
that failed since it is still impossible to locate the right call.
Just use a debugger in that case.
OK tb@ deraadt@

Remove NOTREACHED marker, it should be obvious when the code is:
exit(rc);
/* NOTREACHED */

Adjust pledge() and unveil() calls for proc_rsync() a bit. Since the
mkdir was moved to the main process there is no need for access to .
in the rsync process.
OK job@ deraadt@

Move the mkpath() call from the rsync path to the main process. This allows
to drop cpath from the rsync proc pledge (down to "stdio proc exec").
This will also make work easier with the upcoming http fetcher.
OK tb@

Rework the repository handling. Split the handling of trust anchors into
ta_lookup() while regular repositories (to fetch .mft files) are handled
by repo_lookup(). Also the cache directory layout changed; moving the
trust anchors to ./ta/{tal basename}/ the other repositories end up in
./rsync/
OK tb@

Use mkpath() == -1 to check for failure. No functional change.

Adjust the repository handling a bit. Instead of storing host/module pairs
store repo (rsync URI) and local (the local path to the repository).
Simplifies the the rsync handling a fair bit.
OK deraadt@

rsync is using buffered output now, so remove this FIXME comment

Start using the ibuf API (ibuf_dynamic, ibuf_add, ibuf_close) for writing
data between processes. This completely decouples the write side.
rpki-client can't really use the imsg framework but it can use the ibuf
bits wich imsg is built on.
OK benno@ job@

Now that a NULL string is marshalled as NULL again we can drop some
extra has_xyz integers to indicate if the following buffer is present
or not. At the same time sprinkle some asserts for strings which must
be not NULL.
OK tb@

Remove the last users of io_*_write functions that call io_simple_write()
internally. This is a step in direction of more async aware io in rpki-client.
Now everything uses a buffer which is then written.
OK tb@

Kill connection if rsync server stalls

OK deraadt@

Include openssl/x509.h in extern.h since it uses a few of the typedefs from
there in structs and prototypes. Remove the openssl/ssl.h and other strange
openssl includes in the .c files that don't use openssl specific functions.
OK beck@ and tb@

Move the proc_rsync and with that the rsync processing into rsync.c
main.c is too crowded
OK deraadt@

Handle the TAL files in the master process and pass them as buffer to the
parser process. This way the parser never needs to read outside of the
cache directory which makes the unveil simpler. Additionally rsync_uri_parse
no longer needs to know about .tal files so there is now no chance to sneak
in a .tal file later on.
OK deraadt@

use $OpenBSD$ headers

indentation adjustments, in particular near warn statements
ok claudio

swap comparisons

system includes first, always.

Don't do -portable in base. It is better done outside the tree.
Imagine if we did it throughout the tree, how many copies of strlcpy
would we have, and how much time would all the configure shell scripts
and includes take? It would be ludicrous.

branches: 1.1.1;
Initial revision

Require files to be of a minimum size in the RRDP & RSYNC transports

Picked 100 bytes as a minimum, to accommodate future signature schemes
(such as the smaller P-256) and small files like empty CRLs.

With and OK claudio@ tb@

Don't set directory modtimes to match the source

When syncing against remote repositories, the modtimes of the
remote directories is irrelevant. In the RRDP protocol the directory
modtimes aren't signalled either. This should save some IOPS.

OK tb@

spelling fixes; from paul tagliamonte
any parts of his diff not taken are noted on tech

Only include assert.h if we call assert()

OK tb@

Add support for draft-ietf-sidrops-signed-tal-12

Add support validation of Signed Objects containing Trust Anchor Keys
(TAKs - aka 'Signed TALs'). Signed TALs provide a mechanism for RIRs
to distribute and sign the next Trust Anchor with the current Trust
Anchor. This might be an improvement over visiting RIR websites and
copy+pasting TAL data by hand.

OK tb@

extra newline

Rework the rsync proc code. Use a proper queue of requests and enforce
the limit on that queue instead of stopping to read new messages.
This is needed to implement an abort request.
"There is not enough RB_TREE in this diff" tb@

Make the http code respect MAX_CONN_TIMEOUT and fail connects once they
hit this timeout. This is in line with the rsync code.
OK tb@ job@

Unify the maximum idle IO timeout for RSYNC & HTTPS

OK claudio@

Set rsync connection timeout to 15 seconds.

OK sthen@

Introduce MAX_HTTP_REQUESTS and MAX_RSYNC_REQUESTS.
These just replace MAX_CONNECTIONS and MAX_RSYNC_PROCESSES to be more unified.
OK tb@

more whitespace cleanups

Fix various annoying whitespace errors.

Refactor on how the subprocesses are started.

Move the unveil and pledges to the actuall subprocesses and put all the
common code to start these into process_start(). Reduces the lenght of
main() a fair bit.
OK tb@

Change from a dynamic allocation for the process list to a static
array because the maximum size. The number of processes was already
limited by stopping to poll for new commands but this enforces it
even more. With this remove the FIXME comment since it is no longer
true.
OK tb@

Sync & permit ASPA objects to appear on Manifests

OK tb@ claudio@

Implement but don't use code to use rsync's --compare-dest feature.
One gotcha is that the path passed to --compare-dest needs to be relative
to the dst directory. rsync_fixup_dest() will prepend the necessary ../
for that by counting number of '/' in dst.
OK tb@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Limit the number of rsync processes being spawned by stopping to accept
new requests when over the limit. Use a generous limit of 16.
OK deraadt@

Don't fetch files larger than 2MB

OK claudio@

Rename io_buf_new to io_new_buffer and io_buf_close to io_close_buffer.
With this the write functions are all of the form io_xyz_buffer.
Remove some prototypes of functions I forgot to remove in previous commit.
OK benno@

Finnally move away from blocking reads in rpki-client. The code was a
mish mash of poll, non-blocking writes and blocking reads. Using the
introduced ibuf size header in io_buf_new()/io_buf_close() the read
side can be changed to pull in a full ibuf and only start the un-marshal
once all data has been read.
OK benno@

First step of cleanup in the io land. Introduce io_buf_new() and
io_buf_close(). These function will inject a size of the the buffer
at the beginning of the buffer and will allow the read size to be
switched to proper async IO.
OK benno@

branches: 1.25.4;
RPKI only cares about *.{cer,crl,gbr,mft,roa} files. Use rsync --include
and --exclude to only fetch those files from the CA repositories.
OK job@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.23.2;
Abate superfluous lines from remote servers

OK claudio@

Fail in rsync_base_uri() if the strdup calls fail. Do not bubble this
error upwards since a NULL return represents a bad-URI.
Diff originally from tb@

Use the same way to error out in out of memory situation.
Just use 'err(1, NULL);' there is no need to include the type of function
that failed since it is still impossible to locate the right call.
Just use a debugger in that case.
OK tb@ deraadt@

Remove NOTREACHED marker, it should be obvious when the code is:
exit(rc);
/* NOTREACHED */

Adjust pledge() and unveil() calls for proc_rsync() a bit. Since the
mkdir was moved to the main process there is no need for access to .
in the rsync process.
OK job@ deraadt@

Move the mkpath() call from the rsync path to the main process. This allows
to drop cpath from the rsync proc pledge (down to "stdio proc exec").
This will also make work easier with the upcoming http fetcher.
OK tb@

Rework the repository handling. Split the handling of trust anchors into
ta_lookup() while regular repositories (to fetch .mft files) are handled
by repo_lookup(). Also the cache directory layout changed; moving the
trust anchors to ./ta/{tal basename}/ the other repositories end up in
./rsync/
OK tb@

Use mkpath() == -1 to check for failure. No functional change.

Adjust the repository handling a bit. Instead of storing host/module pairs
store repo (rsync URI) and local (the local path to the repository).
Simplifies the the rsync handling a fair bit.
OK deraadt@

rsync is using buffered output now, so remove this FIXME comment

Start using the ibuf API (ibuf_dynamic, ibuf_add, ibuf_close) for writing
data between processes. This completely decouples the write side.
rpki-client can't really use the imsg framework but it can use the ibuf
bits wich imsg is built on.
OK benno@ job@

Now that a NULL string is marshalled as NULL again we can drop some
extra has_xyz integers to indicate if the following buffer is present
or not. At the same time sprinkle some asserts for strings which must
be not NULL.
OK tb@

Remove the last users of io_*_write functions that call io_simple_write()
internally. This is a step in direction of more async aware io in rpki-client.
Now everything uses a buffer which is then written.
OK tb@

Kill connection if rsync server stalls

OK deraadt@

Include openssl/x509.h in extern.h since it uses a few of the typedefs from
there in structs and prototypes. Remove the openssl/ssl.h and other strange
openssl includes in the .c files that don't use openssl specific functions.
OK beck@ and tb@

Move the proc_rsync and with that the rsync processing into rsync.c
main.c is too crowded
OK deraadt@

Handle the TAL files in the master process and pass them as buffer to the
parser process. This way the parser never needs to read outside of the
cache directory which makes the unveil simpler. Additionally rsync_uri_parse
no longer needs to know about .tal files so there is now no chance to sneak
in a .tal file later on.
OK deraadt@

use $OpenBSD$ headers

indentation adjustments, in particular near warn statements
ok claudio

swap comparisons

system includes first, always.

Don't do -portable in base. It is better done outside the tree.
Imagine if we did it throughout the tree, how many copies of strlcpy
would we have, and how much time would all the configure shell scripts
and includes take? It would be ludicrous.

branches: 1.1.1;
Initial revision

Don't set directory modtimes to match the source

When syncing against remote repositories, the modtimes of the
remote directories is irrelevant. In the RRDP protocol the directory
modtimes aren't signalled either. This should save some IOPS.

OK tb@

spelling fixes; from paul tagliamonte
any parts of his diff not taken are noted on tech

Only include assert.h if we call assert()

OK tb@

Add support for draft-ietf-sidrops-signed-tal-12

Add support validation of Signed Objects containing Trust Anchor Keys
(TAKs - aka 'Signed TALs'). Signed TALs provide a mechanism for RIRs
to distribute and sign the next Trust Anchor with the current Trust
Anchor. This might be an improvement over visiting RIR websites and
copy+pasting TAL data by hand.

OK tb@

extra newline

Rework the rsync proc code. Use a proper queue of requests and enforce
the limit on that queue instead of stopping to read new messages.
This is needed to implement an abort request.
"There is not enough RB_TREE in this diff" tb@

Make the http code respect MAX_CONN_TIMEOUT and fail connects once they
hit this timeout. This is in line with the rsync code.
OK tb@ job@

Unify the maximum idle IO timeout for RSYNC & HTTPS

OK claudio@

Set rsync connection timeout to 15 seconds.

OK sthen@

Introduce MAX_HTTP_REQUESTS and MAX_RSYNC_REQUESTS.
These just replace MAX_CONNECTIONS and MAX_RSYNC_PROCESSES to be more unified.
OK tb@

more whitespace cleanups

Fix various annoying whitespace errors.

Refactor on how the subprocesses are started.

Move the unveil and pledges to the actuall subprocesses and put all the
common code to start these into process_start(). Reduces the lenght of
main() a fair bit.
OK tb@

Change from a dynamic allocation for the process list to a static
array because the maximum size. The number of processes was already
limited by stopping to poll for new commands but this enforces it
even more. With this remove the FIXME comment since it is no longer
true.
OK tb@

Sync & permit ASPA objects to appear on Manifests

OK tb@ claudio@

Implement but don't use code to use rsync's --compare-dest feature.
One gotcha is that the path passed to --compare-dest needs to be relative
to the dst directory. rsync_fixup_dest() will prepend the necessary ../
for that by counting number of '/' in dst.
OK tb@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Limit the number of rsync processes being spawned by stopping to accept
new requests when over the limit. Use a generous limit of 16.
OK deraadt@

Don't fetch files larger than 2MB

OK claudio@

Rename io_buf_new to io_new_buffer and io_buf_close to io_close_buffer.
With this the write functions are all of the form io_xyz_buffer.
Remove some prototypes of functions I forgot to remove in previous commit.
OK benno@

Finnally move away from blocking reads in rpki-client. The code was a
mish mash of poll, non-blocking writes and blocking reads. Using the
introduced ibuf size header in io_buf_new()/io_buf_close() the read
side can be changed to pull in a full ibuf and only start the un-marshal
once all data has been read.
OK benno@

First step of cleanup in the io land. Introduce io_buf_new() and
io_buf_close(). These function will inject a size of the the buffer
at the beginning of the buffer and will allow the read size to be
switched to proper async IO.
OK benno@

branches: 1.25.4;
RPKI only cares about *.{cer,crl,gbr,mft,roa} files. Use rsync --include
and --exclude to only fetch those files from the CA repositories.
OK job@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.23.2;
Abate superfluous lines from remote servers

OK claudio@

Fail in rsync_base_uri() if the strdup calls fail. Do not bubble this
error upwards since a NULL return represents a bad-URI.
Diff originally from tb@

Use the same way to error out in out of memory situation.
Just use 'err(1, NULL);' there is no need to include the type of function
that failed since it is still impossible to locate the right call.
Just use a debugger in that case.
OK tb@ deraadt@

Remove NOTREACHED marker, it should be obvious when the code is:
exit(rc);
/* NOTREACHED */

Adjust pledge() and unveil() calls for proc_rsync() a bit. Since the
mkdir was moved to the main process there is no need for access to .
in the rsync process.
OK job@ deraadt@

Move the mkpath() call from the rsync path to the main process. This allows
to drop cpath from the rsync proc pledge (down to "stdio proc exec").
This will also make work easier with the upcoming http fetcher.
OK tb@

Rework the repository handling. Split the handling of trust anchors into
ta_lookup() while regular repositories (to fetch .mft files) are handled
by repo_lookup(). Also the cache directory layout changed; moving the
trust anchors to ./ta/{tal basename}/ the other repositories end up in
./rsync/
OK tb@

Use mkpath() == -1 to check for failure. No functional change.

Adjust the repository handling a bit. Instead of storing host/module pairs
store repo (rsync URI) and local (the local path to the repository).
Simplifies the the rsync handling a fair bit.
OK deraadt@

rsync is using buffered output now, so remove this FIXME comment

Start using the ibuf API (ibuf_dynamic, ibuf_add, ibuf_close) for writing
data between processes. This completely decouples the write side.
rpki-client can't really use the imsg framework but it can use the ibuf
bits wich imsg is built on.
OK benno@ job@

Now that a NULL string is marshalled as NULL again we can drop some
extra has_xyz integers to indicate if the following buffer is present
or not. At the same time sprinkle some asserts for strings which must
be not NULL.
OK tb@

Remove the last users of io_*_write functions that call io_simple_write()
internally. This is a step in direction of more async aware io in rpki-client.
Now everything uses a buffer which is then written.
OK tb@

Kill connection if rsync server stalls

OK deraadt@

Include openssl/x509.h in extern.h since it uses a few of the typedefs from
there in structs and prototypes. Remove the openssl/ssl.h and other strange
openssl includes in the .c files that don't use openssl specific functions.
OK beck@ and tb@

Move the proc_rsync and with that the rsync processing into rsync.c
main.c is too crowded
OK deraadt@

Handle the TAL files in the master process and pass them as buffer to the
parser process. This way the parser never needs to read outside of the
cache directory which makes the unveil simpler. Additionally rsync_uri_parse
no longer needs to know about .tal files so there is now no chance to sneak
in a .tal file later on.
OK deraadt@

use $OpenBSD$ headers

indentation adjustments, in particular near warn statements
ok claudio

swap comparisons

system includes first, always.

Don't do -portable in base. It is better done outside the tree.
Imagine if we did it throughout the tree, how many copies of strlcpy
would we have, and how much time would all the configure shell scripts
and includes take? It would be ludicrous.

branches: 1.1.1;
Initial revision

spelling fixes; from paul tagliamonte
any parts of his diff not taken are noted on tech

Only include assert.h if we call assert()

OK tb@

Add support for draft-ietf-sidrops-signed-tal-12

Add support validation of Signed Objects containing Trust Anchor Keys
(TAKs - aka 'Signed TALs'). Signed TALs provide a mechanism for RIRs
to distribute and sign the next Trust Anchor with the current Trust
Anchor. This might be an improvement over visiting RIR websites and
copy+pasting TAL data by hand.

OK tb@

extra newline

Rework the rsync proc code. Use a proper queue of requests and enforce
the limit on that queue instead of stopping to read new messages.
This is needed to implement an abort request.
"There is not enough RB_TREE in this diff" tb@

Make the http code respect MAX_CONN_TIMEOUT and fail connects once they
hit this timeout. This is in line with the rsync code.
OK tb@ job@

Unify the maximum idle IO timeout for RSYNC & HTTPS

OK claudio@

Set rsync connection timeout to 15 seconds.

OK sthen@

Introduce MAX_HTTP_REQUESTS and MAX_RSYNC_REQUESTS.
These just replace MAX_CONNECTIONS and MAX_RSYNC_PROCESSES to be more unified.
OK tb@

more whitespace cleanups

Fix various annoying whitespace errors.

Refactor on how the subprocesses are started.

Move the unveil and pledges to the actuall subprocesses and put all the
common code to start these into process_start(). Reduces the lenght of
main() a fair bit.
OK tb@

Change from a dynamic allocation for the process list to a static
array because the maximum size. The number of processes was already
limited by stopping to poll for new commands but this enforces it
even more. With this remove the FIXME comment since it is no longer
true.
OK tb@

Sync & permit ASPA objects to appear on Manifests

OK tb@ claudio@

Implement but don't use code to use rsync's --compare-dest feature.
One gotcha is that the path passed to --compare-dest needs to be relative
to the dst directory. rsync_fixup_dest() will prepend the necessary ../
for that by counting number of '/' in dst.
OK tb@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Limit the number of rsync processes being spawned by stopping to accept
new requests when over the limit. Use a generous limit of 16.
OK deraadt@

Don't fetch files larger than 2MB

OK claudio@

Rename io_buf_new to io_new_buffer and io_buf_close to io_close_buffer.
With this the write functions are all of the form io_xyz_buffer.
Remove some prototypes of functions I forgot to remove in previous commit.
OK benno@

Finnally move away from blocking reads in rpki-client. The code was a
mish mash of poll, non-blocking writes and blocking reads. Using the
introduced ibuf size header in io_buf_new()/io_buf_close() the read
side can be changed to pull in a full ibuf and only start the un-marshal
once all data has been read.
OK benno@

First step of cleanup in the io land. Introduce io_buf_new() and
io_buf_close(). These function will inject a size of the the buffer
at the beginning of the buffer and will allow the read size to be
switched to proper async IO.
OK benno@

branches: 1.25.4;
RPKI only cares about *.{cer,crl,gbr,mft,roa} files. Use rsync --include
and --exclude to only fetch those files from the CA repositories.
OK job@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.23.2;
Abate superfluous lines from remote servers

OK claudio@

Fail in rsync_base_uri() if the strdup calls fail. Do not bubble this
error upwards since a NULL return represents a bad-URI.
Diff originally from tb@

Use the same way to error out in out of memory situation.
Just use 'err(1, NULL);' there is no need to include the type of function
that failed since it is still impossible to locate the right call.
Just use a debugger in that case.
OK tb@ deraadt@

Remove NOTREACHED marker, it should be obvious when the code is:
exit(rc);
/* NOTREACHED */

Adjust pledge() and unveil() calls for proc_rsync() a bit. Since the
mkdir was moved to the main process there is no need for access to .
in the rsync process.
OK job@ deraadt@

Move the mkpath() call from the rsync path to the main process. This allows
to drop cpath from the rsync proc pledge (down to "stdio proc exec").
This will also make work easier with the upcoming http fetcher.
OK tb@

Rework the repository handling. Split the handling of trust anchors into
ta_lookup() while regular repositories (to fetch .mft files) are handled
by repo_lookup(). Also the cache directory layout changed; moving the
trust anchors to ./ta/{tal basename}/ the other repositories end up in
./rsync/
OK tb@

Use mkpath() == -1 to check for failure. No functional change.

Adjust the repository handling a bit. Instead of storing host/module pairs
store repo (rsync URI) and local (the local path to the repository).
Simplifies the the rsync handling a fair bit.
OK deraadt@

rsync is using buffered output now, so remove this FIXME comment

Start using the ibuf API (ibuf_dynamic, ibuf_add, ibuf_close) for writing
data between processes. This completely decouples the write side.
rpki-client can't really use the imsg framework but it can use the ibuf
bits wich imsg is built on.
OK benno@ job@

Now that a NULL string is marshalled as NULL again we can drop some
extra has_xyz integers to indicate if the following buffer is present
or not. At the same time sprinkle some asserts for strings which must
be not NULL.
OK tb@

Remove the last users of io_*_write functions that call io_simple_write()
internally. This is a step in direction of more async aware io in rpki-client.
Now everything uses a buffer which is then written.
OK tb@

Kill connection if rsync server stalls

OK deraadt@

Include openssl/x509.h in extern.h since it uses a few of the typedefs from
there in structs and prototypes. Remove the openssl/ssl.h and other strange
openssl includes in the .c files that don't use openssl specific functions.
OK beck@ and tb@

Move the proc_rsync and with that the rsync processing into rsync.c
main.c is too crowded
OK deraadt@

Handle the TAL files in the master process and pass them as buffer to the
parser process. This way the parser never needs to read outside of the
cache directory which makes the unveil simpler. Additionally rsync_uri_parse
no longer needs to know about .tal files so there is now no chance to sneak
in a .tal file later on.
OK deraadt@

use $OpenBSD$ headers

indentation adjustments, in particular near warn statements
ok claudio

swap comparisons

system includes first, always.

Don't do -portable in base. It is better done outside the tree.
Imagine if we did it throughout the tree, how many copies of strlcpy
would we have, and how much time would all the configure shell scripts
and includes take? It would be ludicrous.

branches: 1.1.1;
Initial revision

Only include assert.h if we call assert()

OK tb@

Add support for draft-ietf-sidrops-signed-tal-12

Add support validation of Signed Objects containing Trust Anchor Keys
(TAKs - aka 'Signed TALs'). Signed TALs provide a mechanism for RIRs
to distribute and sign the next Trust Anchor with the current Trust
Anchor. This might be an improvement over visiting RIR websites and
copy+pasting TAL data by hand.

OK tb@

extra newline

Rework the rsync proc code. Use a proper queue of requests and enforce
the limit on that queue instead of stopping to read new messages.
This is needed to implement an abort request.
"There is not enough RB_TREE in this diff" tb@

Make the http code respect MAX_CONN_TIMEOUT and fail connects once they
hit this timeout. This is in line with the rsync code.
OK tb@ job@

Unify the maximum idle IO timeout for RSYNC & HTTPS

OK claudio@

Set rsync connection timeout to 15 seconds.

OK sthen@

Introduce MAX_HTTP_REQUESTS and MAX_RSYNC_REQUESTS.
These just replace MAX_CONNECTIONS and MAX_RSYNC_PROCESSES to be more unified.
OK tb@

more whitespace cleanups

Fix various annoying whitespace errors.

Refactor on how the subprocesses are started.

Move the unveil and pledges to the actuall subprocesses and put all the
common code to start these into process_start(). Reduces the lenght of
main() a fair bit.
OK tb@

Change from a dynamic allocation for the process list to a static
array because the maximum size. The number of processes was already
limited by stopping to poll for new commands but this enforces it
even more. With this remove the FIXME comment since it is no longer
true.
OK tb@

Sync & permit ASPA objects to appear on Manifests

OK tb@ claudio@

Implement but don't use code to use rsync's --compare-dest feature.
One gotcha is that the path passed to --compare-dest needs to be relative
to the dst directory. rsync_fixup_dest() will prepend the necessary ../
for that by counting number of '/' in dst.
OK tb@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Limit the number of rsync processes being spawned by stopping to accept
new requests when over the limit. Use a generous limit of 16.
OK deraadt@

Don't fetch files larger than 2MB

OK claudio@

Rename io_buf_new to io_new_buffer and io_buf_close to io_close_buffer.
With this the write functions are all of the form io_xyz_buffer.
Remove some prototypes of functions I forgot to remove in previous commit.
OK benno@

Finnally move away from blocking reads in rpki-client. The code was a
mish mash of poll, non-blocking writes and blocking reads. Using the
introduced ibuf size header in io_buf_new()/io_buf_close() the read
side can be changed to pull in a full ibuf and only start the un-marshal
once all data has been read.
OK benno@

First step of cleanup in the io land. Introduce io_buf_new() and
io_buf_close(). These function will inject a size of the the buffer
at the beginning of the buffer and will allow the read size to be
switched to proper async IO.
OK benno@

branches: 1.25.4;
RPKI only cares about *.{cer,crl,gbr,mft,roa} files. Use rsync --include
and --exclude to only fetch those files from the CA repositories.
OK job@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.23.2;
Abate superfluous lines from remote servers

OK claudio@

Fail in rsync_base_uri() if the strdup calls fail. Do not bubble this
error upwards since a NULL return represents a bad-URI.
Diff originally from tb@

Use the same way to error out in out of memory situation.
Just use 'err(1, NULL);' there is no need to include the type of function
that failed since it is still impossible to locate the right call.
Just use a debugger in that case.
OK tb@ deraadt@

Remove NOTREACHED marker, it should be obvious when the code is:
exit(rc);
/* NOTREACHED */

Adjust pledge() and unveil() calls for proc_rsync() a bit. Since the
mkdir was moved to the main process there is no need for access to .
in the rsync process.
OK job@ deraadt@

Move the mkpath() call from the rsync path to the main process. This allows
to drop cpath from the rsync proc pledge (down to "stdio proc exec").
This will also make work easier with the upcoming http fetcher.
OK tb@

Rework the repository handling. Split the handling of trust anchors into
ta_lookup() while regular repositories (to fetch .mft files) are handled
by repo_lookup(). Also the cache directory layout changed; moving the
trust anchors to ./ta/{tal basename}/ the other repositories end up in
./rsync/
OK tb@

Use mkpath() == -1 to check for failure. No functional change.

Adjust the repository handling a bit. Instead of storing host/module pairs
store repo (rsync URI) and local (the local path to the repository).
Simplifies the the rsync handling a fair bit.
OK deraadt@

rsync is using buffered output now, so remove this FIXME comment

Start using the ibuf API (ibuf_dynamic, ibuf_add, ibuf_close) for writing
data between processes. This completely decouples the write side.
rpki-client can't really use the imsg framework but it can use the ibuf
bits wich imsg is built on.
OK benno@ job@

Now that a NULL string is marshalled as NULL again we can drop some
extra has_xyz integers to indicate if the following buffer is present
or not. At the same time sprinkle some asserts for strings which must
be not NULL.
OK tb@

Remove the last users of io_*_write functions that call io_simple_write()
internally. This is a step in direction of more async aware io in rpki-client.
Now everything uses a buffer which is then written.
OK tb@

Kill connection if rsync server stalls

OK deraadt@

Include openssl/x509.h in extern.h since it uses a few of the typedefs from
there in structs and prototypes. Remove the openssl/ssl.h and other strange
openssl includes in the .c files that don't use openssl specific functions.
OK beck@ and tb@

Move the proc_rsync and with that the rsync processing into rsync.c
main.c is too crowded
OK deraadt@

Handle the TAL files in the master process and pass them as buffer to the
parser process. This way the parser never needs to read outside of the
cache directory which makes the unveil simpler. Additionally rsync_uri_parse
no longer needs to know about .tal files so there is now no chance to sneak
in a .tal file later on.
OK deraadt@

use $OpenBSD$ headers

indentation adjustments, in particular near warn statements
ok claudio

swap comparisons

system includes first, always.

Don't do -portable in base. It is better done outside the tree.
Imagine if we did it throughout the tree, how many copies of strlcpy
would we have, and how much time would all the configure shell scripts
and includes take? It would be ludicrous.

branches: 1.1.1;
Initial revision

Add support for draft-ietf-sidrops-signed-tal-12

Add support validation of Signed Objects containing Trust Anchor Keys
(TAKs - aka 'Signed TALs'). Signed TALs provide a mechanism for RIRs
to distribute and sign the next Trust Anchor with the current Trust
Anchor. This might be an improvement over visiting RIR websites and
copy+pasting TAL data by hand.

OK tb@

extra newline

Rework the rsync proc code. Use a proper queue of requests and enforce
the limit on that queue instead of stopping to read new messages.
This is needed to implement an abort request.
"There is not enough RB_TREE in this diff" tb@

Make the http code respect MAX_CONN_TIMEOUT and fail connects once they
hit this timeout. This is in line with the rsync code.
OK tb@ job@

Unify the maximum idle IO timeout for RSYNC & HTTPS

OK claudio@

Set rsync connection timeout to 15 seconds.

OK sthen@

Introduce MAX_HTTP_REQUESTS and MAX_RSYNC_REQUESTS.
These just replace MAX_CONNECTIONS and MAX_RSYNC_PROCESSES to be more unified.
OK tb@

more whitespace cleanups

Fix various annoying whitespace errors.

Refactor on how the subprocesses are started.

Move the unveil and pledges to the actuall subprocesses and put all the
common code to start these into process_start(). Reduces the lenght of
main() a fair bit.
OK tb@

Change from a dynamic allocation for the process list to a static
array because the maximum size. The number of processes was already
limited by stopping to poll for new commands but this enforces it
even more. With this remove the FIXME comment since it is no longer
true.
OK tb@

Sync & permit ASPA objects to appear on Manifests

OK tb@ claudio@

Implement but don't use code to use rsync's --compare-dest feature.
One gotcha is that the path passed to --compare-dest needs to be relative
to the dst directory. rsync_fixup_dest() will prepend the necessary ../
for that by counting number of '/' in dst.
OK tb@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Limit the number of rsync processes being spawned by stopping to accept
new requests when over the limit. Use a generous limit of 16.
OK deraadt@

Don't fetch files larger than 2MB

OK claudio@

Rename io_buf_new to io_new_buffer and io_buf_close to io_close_buffer.
With this the write functions are all of the form io_xyz_buffer.
Remove some prototypes of functions I forgot to remove in previous commit.
OK benno@

Finnally move away from blocking reads in rpki-client. The code was a
mish mash of poll, non-blocking writes and blocking reads. Using the
introduced ibuf size header in io_buf_new()/io_buf_close() the read
side can be changed to pull in a full ibuf and only start the un-marshal
once all data has been read.
OK benno@

First step of cleanup in the io land. Introduce io_buf_new() and
io_buf_close(). These function will inject a size of the the buffer
at the beginning of the buffer and will allow the read size to be
switched to proper async IO.
OK benno@

branches: 1.25.4;
RPKI only cares about *.{cer,crl,gbr,mft,roa} files. Use rsync --include
and --exclude to only fetch those files from the CA repositories.
OK job@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.23.2;
Abate superfluous lines from remote servers

OK claudio@

Fail in rsync_base_uri() if the strdup calls fail. Do not bubble this
error upwards since a NULL return represents a bad-URI.
Diff originally from tb@

Use the same way to error out in out of memory situation.
Just use 'err(1, NULL);' there is no need to include the type of function
that failed since it is still impossible to locate the right call.
Just use a debugger in that case.
OK tb@ deraadt@

Remove NOTREACHED marker, it should be obvious when the code is:
exit(rc);
/* NOTREACHED */

Adjust pledge() and unveil() calls for proc_rsync() a bit. Since the
mkdir was moved to the main process there is no need for access to .
in the rsync process.
OK job@ deraadt@

Move the mkpath() call from the rsync path to the main process. This allows
to drop cpath from the rsync proc pledge (down to "stdio proc exec").
This will also make work easier with the upcoming http fetcher.
OK tb@

Rework the repository handling. Split the handling of trust anchors into
ta_lookup() while regular repositories (to fetch .mft files) are handled
by repo_lookup(). Also the cache directory layout changed; moving the
trust anchors to ./ta/{tal basename}/ the other repositories end up in
./rsync/
OK tb@

Use mkpath() == -1 to check for failure. No functional change.

Adjust the repository handling a bit. Instead of storing host/module pairs
store repo (rsync URI) and local (the local path to the repository).
Simplifies the the rsync handling a fair bit.
OK deraadt@

rsync is using buffered output now, so remove this FIXME comment

Start using the ibuf API (ibuf_dynamic, ibuf_add, ibuf_close) for writing
data between processes. This completely decouples the write side.
rpki-client can't really use the imsg framework but it can use the ibuf
bits wich imsg is built on.
OK benno@ job@

Now that a NULL string is marshalled as NULL again we can drop some
extra has_xyz integers to indicate if the following buffer is present
or not. At the same time sprinkle some asserts for strings which must
be not NULL.
OK tb@

Remove the last users of io_*_write functions that call io_simple_write()
internally. This is a step in direction of more async aware io in rpki-client.
Now everything uses a buffer which is then written.
OK tb@

Kill connection if rsync server stalls

OK deraadt@

Include openssl/x509.h in extern.h since it uses a few of the typedefs from
there in structs and prototypes. Remove the openssl/ssl.h and other strange
openssl includes in the .c files that don't use openssl specific functions.
OK beck@ and tb@

Move the proc_rsync and with that the rsync processing into rsync.c
main.c is too crowded
OK deraadt@

Handle the TAL files in the master process and pass them as buffer to the
parser process. This way the parser never needs to read outside of the
cache directory which makes the unveil simpler. Additionally rsync_uri_parse
no longer needs to know about .tal files so there is now no chance to sneak
in a .tal file later on.
OK deraadt@

use $OpenBSD$ headers

indentation adjustments, in particular near warn statements
ok claudio

swap comparisons

system includes first, always.

Don't do -portable in base. It is better done outside the tree.
Imagine if we did it throughout the tree, how many copies of strlcpy
would we have, and how much time would all the configure shell scripts
and includes take? It would be ludicrous.

branches: 1.1.1;
Initial revision

extra newline

Rework the rsync proc code. Use a proper queue of requests and enforce
the limit on that queue instead of stopping to read new messages.
This is needed to implement an abort request.
"There is not enough RB_TREE in this diff" tb@

Make the http code respect MAX_CONN_TIMEOUT and fail connects once they
hit this timeout. This is in line with the rsync code.
OK tb@ job@

Unify the maximum idle IO timeout for RSYNC & HTTPS

OK claudio@

Set rsync connection timeout to 15 seconds.

OK sthen@

Introduce MAX_HTTP_REQUESTS and MAX_RSYNC_REQUESTS.
These just replace MAX_CONNECTIONS and MAX_RSYNC_PROCESSES to be more unified.
OK tb@

more whitespace cleanups

Fix various annoying whitespace errors.

Refactor on how the subprocesses are started.

Move the unveil and pledges to the actuall subprocesses and put all the
common code to start these into process_start(). Reduces the lenght of
main() a fair bit.
OK tb@

Change from a dynamic allocation for the process list to a static
array because the maximum size. The number of processes was already
limited by stopping to poll for new commands but this enforces it
even more. With this remove the FIXME comment since it is no longer
true.
OK tb@

Sync & permit ASPA objects to appear on Manifests

OK tb@ claudio@

Implement but don't use code to use rsync's --compare-dest feature.
One gotcha is that the path passed to --compare-dest needs to be relative
to the dst directory. rsync_fixup_dest() will prepend the necessary ../
for that by counting number of '/' in dst.
OK tb@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Limit the number of rsync processes being spawned by stopping to accept
new requests when over the limit. Use a generous limit of 16.
OK deraadt@

Don't fetch files larger than 2MB

OK claudio@

Rename io_buf_new to io_new_buffer and io_buf_close to io_close_buffer.
With this the write functions are all of the form io_xyz_buffer.
Remove some prototypes of functions I forgot to remove in previous commit.
OK benno@

Finnally move away from blocking reads in rpki-client. The code was a
mish mash of poll, non-blocking writes and blocking reads. Using the
introduced ibuf size header in io_buf_new()/io_buf_close() the read
side can be changed to pull in a full ibuf and only start the un-marshal
once all data has been read.
OK benno@

First step of cleanup in the io land. Introduce io_buf_new() and
io_buf_close(). These function will inject a size of the the buffer
at the beginning of the buffer and will allow the read size to be
switched to proper async IO.
OK benno@

branches: 1.25.4;
RPKI only cares about *.{cer,crl,gbr,mft,roa} files. Use rsync --include
and --exclude to only fetch those files from the CA repositories.
OK job@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.23.2;
Abate superfluous lines from remote servers

OK claudio@

Fail in rsync_base_uri() if the strdup calls fail. Do not bubble this
error upwards since a NULL return represents a bad-URI.
Diff originally from tb@

Use the same way to error out in out of memory situation.
Just use 'err(1, NULL);' there is no need to include the type of function
that failed since it is still impossible to locate the right call.
Just use a debugger in that case.
OK tb@ deraadt@

Remove NOTREACHED marker, it should be obvious when the code is:
exit(rc);
/* NOTREACHED */

Adjust pledge() and unveil() calls for proc_rsync() a bit. Since the
mkdir was moved to the main process there is no need for access to .
in the rsync process.
OK job@ deraadt@

Move the mkpath() call from the rsync path to the main process. This allows
to drop cpath from the rsync proc pledge (down to "stdio proc exec").
This will also make work easier with the upcoming http fetcher.
OK tb@

Rework the repository handling. Split the handling of trust anchors into
ta_lookup() while regular repositories (to fetch .mft files) are handled
by repo_lookup(). Also the cache directory layout changed; moving the
trust anchors to ./ta/{tal basename}/ the other repositories end up in
./rsync/
OK tb@

Use mkpath() == -1 to check for failure. No functional change.

Adjust the repository handling a bit. Instead of storing host/module pairs
store repo (rsync URI) and local (the local path to the repository).
Simplifies the the rsync handling a fair bit.
OK deraadt@

rsync is using buffered output now, so remove this FIXME comment

Start using the ibuf API (ibuf_dynamic, ibuf_add, ibuf_close) for writing
data between processes. This completely decouples the write side.
rpki-client can't really use the imsg framework but it can use the ibuf
bits wich imsg is built on.
OK benno@ job@

Now that a NULL string is marshalled as NULL again we can drop some
extra has_xyz integers to indicate if the following buffer is present
or not. At the same time sprinkle some asserts for strings which must
be not NULL.
OK tb@

Remove the last users of io_*_write functions that call io_simple_write()
internally. This is a step in direction of more async aware io in rpki-client.
Now everything uses a buffer which is then written.
OK tb@

Kill connection if rsync server stalls

OK deraadt@

Include openssl/x509.h in extern.h since it uses a few of the typedefs from
there in structs and prototypes. Remove the openssl/ssl.h and other strange
openssl includes in the .c files that don't use openssl specific functions.
OK beck@ and tb@

Move the proc_rsync and with that the rsync processing into rsync.c
main.c is too crowded
OK deraadt@

Handle the TAL files in the master process and pass them as buffer to the
parser process. This way the parser never needs to read outside of the
cache directory which makes the unveil simpler. Additionally rsync_uri_parse
no longer needs to know about .tal files so there is now no chance to sneak
in a .tal file later on.
OK deraadt@

use $OpenBSD$ headers

indentation adjustments, in particular near warn statements
ok claudio

swap comparisons

system includes first, always.

Don't do -portable in base. It is better done outside the tree.
Imagine if we did it throughout the tree, how many copies of strlcpy
would we have, and how much time would all the configure shell scripts
and includes take? It would be ludicrous.

branches: 1.1.1;
Initial revision

Make the http code respect MAX_CONN_TIMEOUT and fail connects once they
hit this timeout. This is in line with the rsync code.
OK tb@ job@

Unify the maximum idle IO timeout for RSYNC & HTTPS

OK claudio@

Set rsync connection timeout to 15 seconds.

OK sthen@

Introduce MAX_HTTP_REQUESTS and MAX_RSYNC_REQUESTS.
These just replace MAX_CONNECTIONS and MAX_RSYNC_PROCESSES to be more unified.
OK tb@

more whitespace cleanups

Fix various annoying whitespace errors.

Refactor on how the subprocesses are started.

Move the unveil and pledges to the actuall subprocesses and put all the
common code to start these into process_start(). Reduces the lenght of
main() a fair bit.
OK tb@

Change from a dynamic allocation for the process list to a static
array because the maximum size. The number of processes was already
limited by stopping to poll for new commands but this enforces it
even more. With this remove the FIXME comment since it is no longer
true.
OK tb@

Sync & permit ASPA objects to appear on Manifests

OK tb@ claudio@

Implement but don't use code to use rsync's --compare-dest feature.
One gotcha is that the path passed to --compare-dest needs to be relative
to the dst directory. rsync_fixup_dest() will prepend the necessary ../
for that by counting number of '/' in dst.
OK tb@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Limit the number of rsync processes being spawned by stopping to accept
new requests when over the limit. Use a generous limit of 16.
OK deraadt@

Don't fetch files larger than 2MB

OK claudio@

Rename io_buf_new to io_new_buffer and io_buf_close to io_close_buffer.
With this the write functions are all of the form io_xyz_buffer.
Remove some prototypes of functions I forgot to remove in previous commit.
OK benno@

Finnally move away from blocking reads in rpki-client. The code was a
mish mash of poll, non-blocking writes and blocking reads. Using the
introduced ibuf size header in io_buf_new()/io_buf_close() the read
side can be changed to pull in a full ibuf and only start the un-marshal
once all data has been read.
OK benno@

First step of cleanup in the io land. Introduce io_buf_new() and
io_buf_close(). These function will inject a size of the the buffer
at the beginning of the buffer and will allow the read size to be
switched to proper async IO.
OK benno@

branches: 1.25.4;
RPKI only cares about *.{cer,crl,gbr,mft,roa} files. Use rsync --include
and --exclude to only fetch those files from the CA repositories.
OK job@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.23.2;
Abate superfluous lines from remote servers

OK claudio@

Fail in rsync_base_uri() if the strdup calls fail. Do not bubble this
error upwards since a NULL return represents a bad-URI.
Diff originally from tb@

Use the same way to error out in out of memory situation.
Just use 'err(1, NULL);' there is no need to include the type of function
that failed since it is still impossible to locate the right call.
Just use a debugger in that case.
OK tb@ deraadt@

Remove NOTREACHED marker, it should be obvious when the code is:
exit(rc);
/* NOTREACHED */

Adjust pledge() and unveil() calls for proc_rsync() a bit. Since the
mkdir was moved to the main process there is no need for access to .
in the rsync process.
OK job@ deraadt@

Move the mkpath() call from the rsync path to the main process. This allows
to drop cpath from the rsync proc pledge (down to "stdio proc exec").
This will also make work easier with the upcoming http fetcher.
OK tb@

Rework the repository handling. Split the handling of trust anchors into
ta_lookup() while regular repositories (to fetch .mft files) are handled
by repo_lookup(). Also the cache directory layout changed; moving the
trust anchors to ./ta/{tal basename}/ the other repositories end up in
./rsync/
OK tb@

Use mkpath() == -1 to check for failure. No functional change.

Adjust the repository handling a bit. Instead of storing host/module pairs
store repo (rsync URI) and local (the local path to the repository).
Simplifies the the rsync handling a fair bit.
OK deraadt@

rsync is using buffered output now, so remove this FIXME comment

Start using the ibuf API (ibuf_dynamic, ibuf_add, ibuf_close) for writing
data between processes. This completely decouples the write side.
rpki-client can't really use the imsg framework but it can use the ibuf
bits wich imsg is built on.
OK benno@ job@

Now that a NULL string is marshalled as NULL again we can drop some
extra has_xyz integers to indicate if the following buffer is present
or not. At the same time sprinkle some asserts for strings which must
be not NULL.
OK tb@

Remove the last users of io_*_write functions that call io_simple_write()
internally. This is a step in direction of more async aware io in rpki-client.
Now everything uses a buffer which is then written.
OK tb@

Kill connection if rsync server stalls

OK deraadt@

Include openssl/x509.h in extern.h since it uses a few of the typedefs from
there in structs and prototypes. Remove the openssl/ssl.h and other strange
openssl includes in the .c files that don't use openssl specific functions.
OK beck@ and tb@

Move the proc_rsync and with that the rsync processing into rsync.c
main.c is too crowded
OK deraadt@

Handle the TAL files in the master process and pass them as buffer to the
parser process. This way the parser never needs to read outside of the
cache directory which makes the unveil simpler. Additionally rsync_uri_parse
no longer needs to know about .tal files so there is now no chance to sneak
in a .tal file later on.
OK deraadt@

use $OpenBSD$ headers

indentation adjustments, in particular near warn statements
ok claudio

swap comparisons

system includes first, always.

Don't do -portable in base. It is better done outside the tree.
Imagine if we did it throughout the tree, how many copies of strlcpy
would we have, and how much time would all the configure shell scripts
and includes take? It would be ludicrous.

branches: 1.1.1;
Initial revision

Introduce MAX_HTTP_REQUESTS and MAX_RSYNC_REQUESTS.
These just replace MAX_CONNECTIONS and MAX_RSYNC_PROCESSES to be more unified.
OK tb@

more whitespace cleanups

Fix various annoying whitespace errors.

Refactor on how the subprocesses are started.

Move the unveil and pledges to the actuall subprocesses and put all the
common code to start these into process_start(). Reduces the lenght of
main() a fair bit.
OK tb@

Change from a dynamic allocation for the process list to a static
array because the maximum size. The number of processes was already
limited by stopping to poll for new commands but this enforces it
even more. With this remove the FIXME comment since it is no longer
true.
OK tb@

Sync & permit ASPA objects to appear on Manifests

OK tb@ claudio@

Implement but don't use code to use rsync's --compare-dest feature.
One gotcha is that the path passed to --compare-dest needs to be relative
to the dst directory. rsync_fixup_dest() will prepend the necessary ../
for that by counting number of '/' in dst.
OK tb@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Limit the number of rsync processes being spawned by stopping to accept
new requests when over the limit. Use a generous limit of 16.
OK deraadt@

Don't fetch files larger than 2MB

OK claudio@

Rename io_buf_new to io_new_buffer and io_buf_close to io_close_buffer.
With this the write functions are all of the form io_xyz_buffer.
Remove some prototypes of functions I forgot to remove in previous commit.
OK benno@

Finnally move away from blocking reads in rpki-client. The code was a
mish mash of poll, non-blocking writes and blocking reads. Using the
introduced ibuf size header in io_buf_new()/io_buf_close() the read
side can be changed to pull in a full ibuf and only start the un-marshal
once all data has been read.
OK benno@

First step of cleanup in the io land. Introduce io_buf_new() and
io_buf_close(). These function will inject a size of the the buffer
at the beginning of the buffer and will allow the read size to be
switched to proper async IO.
OK benno@

branches: 1.25.4;
RPKI only cares about *.{cer,crl,gbr,mft,roa} files. Use rsync --include
and --exclude to only fetch those files from the CA repositories.
OK job@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.23.2;
Abate superfluous lines from remote servers

OK claudio@

Fail in rsync_base_uri() if the strdup calls fail. Do not bubble this
error upwards since a NULL return represents a bad-URI.
Diff originally from tb@

Use the same way to error out in out of memory situation.
Just use 'err(1, NULL);' there is no need to include the type of function
that failed since it is still impossible to locate the right call.
Just use a debugger in that case.
OK tb@ deraadt@

Remove NOTREACHED marker, it should be obvious when the code is:
exit(rc);
/* NOTREACHED */

Adjust pledge() and unveil() calls for proc_rsync() a bit. Since the
mkdir was moved to the main process there is no need for access to .
in the rsync process.
OK job@ deraadt@

Move the mkpath() call from the rsync path to the main process. This allows
to drop cpath from the rsync proc pledge (down to "stdio proc exec").
This will also make work easier with the upcoming http fetcher.
OK tb@

Rework the repository handling. Split the handling of trust anchors into
ta_lookup() while regular repositories (to fetch .mft files) are handled
by repo_lookup(). Also the cache directory layout changed; moving the
trust anchors to ./ta/{tal basename}/ the other repositories end up in
./rsync/
OK tb@

Use mkpath() == -1 to check for failure. No functional change.

Adjust the repository handling a bit. Instead of storing host/module pairs
store repo (rsync URI) and local (the local path to the repository).
Simplifies the the rsync handling a fair bit.
OK deraadt@