Download new TA files into a temporary place (.ta/) so that the parser
can decide which of the two files to use.
With and OK tb@

Properly setup the fts_state in the FTS_D case.

At fts_level 1 the state needs to be fully reset since we most probably
exited from a directory at level 1 and entered a new dir at level 1.
Without this empty directories remained since the fts_state.type
was wrong for those entries.
Noticed by job@, OK tb@

Instead of tracking certificates by SKI track them by an internal identifier.

The certificate SKI is not strictly unique so using it as a unique id is
problematic. It is also not really needed to do that since in theory we
already know the path (but this got lost in the privsep communication).
So add a cert id and pass this id back and forth between main process and
the parser. With this id we can lookup the authentication chain in the
parser and this even works with multiple paths to the same resource.
Since we no longer lookup by SKI the valid_aki_ski function is replaced
by find_issuer() which does the lookup by certid.

The loop protection is now extended to allow each TAL to reach each file
once but still triggers if a file is reaccessed by the tree of a TAL.

In filemode the lookup now uses an AIA uri based lookup tree. Again this
replaces the SKI based lookups from before.

Done together with tb@
OK tb@ job@

P-256 support is experimental so require -x to enable it.

Also clean up the externs a little bit by moving experimental and noop
to extern.h.
Reminded by and OK tb@

Fix capping of VAPs

The previous approach introduced a cap, but it might not always be hit as
intended (I missed this on review). Fix this to check the cap after merging
an ASPA into an already existing VAP. Also free the list of providers since
nothing should be looking at it anymore.

Count VAPs that hit the limit with a new overflowed counter. There are
still a few aspects of the accounting that probably aren't entirely right.
This will be fixed at another point. It's just statistics after all.

with/ok claudio, ok job

Replace protocol literal strings and strlen() calls with defined constants

OK tb@ claudio@

Track the number of new files moving from 'staging' to 'validated cache'

The OpenMetrics output shows per-repository counters for new files
added, the main process and JSON output emit the sum of all new files.

OK claudio@

Add support for RPKI Signed Prefix Lists

Signed Prefix List are a CMS protected content type for use with the
RPKI to carry the complete list of prefixes which an Autonomous System
may originate to all or any of its routing peers. The validation of a
Signed Prefix List confirms that the holder of the listed ASN produced
the object, and that this list is a current, accurate and complete
description of address prefixes that may be announced into the routing
system originated by this AS.

https://datatracker.ietf.org/doc/html/draft-ietf-sidrops-rpki-prefixlist

with and OK claudio@ tb@

Refactor handling of stale manifests

No need to hoist a staleness indicator through the whole process and
count it explicitly.

OK tb@

Do not set O_EXCL on open() when a published file is on the withdraw list.

If during the process of applying RRDP deltas a object is published then
withdrawn and later published again the current code fails. This is because
published elements use O_EXCL in open() but in the case above the previous
file is still around since withdraws are delayed.

Problem noticed by job@
OK tb@

rpki-client: fix vap_pas stats

A small mistake in a diff broke the counters. Make them AFI agnostic and
adjust ometric output.

guidance & ok claudio

In rrdp_session_save() stop the loop over deltas once MAX_RRDP_DELTAS
have been processed.
With and OK tb@

Improve detection of RRDP session desynchronization

According to RFC 8182, a given session_id and serial number represent an
immutable record of the state of the Repository Server at a certain
point in time.

Add a check to the RRDP notification file processing to compare whether
the delta hashes associated to previously seen serials are different in
newly fetched notification files. Fall back to a snapshot if a difference
is detected, because such a mutation is a strong desynchronization
indicator.

Idea from Ties de Kock (RIPE NCC).
Based on a diff by job@
With and OK job@ tb@

Fixup file modification timestamps to optimize failover from RRDP to RSYNC

In the RSYNC protocol a file's last modification time and its size are
used to determine whether sending a (partial) copy over the wire is needed.
Previously, when RRDP data structures are serialized to disk, the mtime of
files in DIR_VALID ended up being UTIME_NOW.

Thus, the mtimes of files obtained through RRDP will never match the mtimes
of the same files available through RSYNC - causing each and every file to
be added to the file transfer list.

Instead, use the internal timestamps of RPKI files as the last modified
timestamp. Specifically, for Signed Objects (ROAs, MFTs, GBRs, TAKs, ASPAs)
the CMS signing-time, for .cer files the X.509 notBefore, and for .crl files
the CRL lastUpdate. This results in a surprising optimization for the number
files which have to be transfered.

OK claudio@

Fix repo_cleanup_entry() state machine so that the repository lookups
are done when the full repo path is available. Without this all repo
lookups returned NULL and the code did not work as intended.
OK tb@

RRDP snapshots should encode publish elements only once. If encountered
fail the transfer and fall back to rsync.

When more than one publish element for the same file exist the RP does
not know which one to choose. Lets fail the RRDP transfer in this case
and fall back to rsync. CA that publish a file more than once are buggy
and need to be fixed.
OK job@ tb@

Improve accounting by tracking things by repo and tal.
This fixes some wrong accounting for repositories that are referenced
from more than one TAL. It changes the ometric lable output a little bit
since there are repository metrics that no longer include the 'name' label.
OK tb@

Add the protocol used to sync the repository to the open-metric output.
OK tb@

Instead of storing the short TAL name as repouri use the first TAL URI
for that. This is more in line with the other uses of repouri.
OK tb@

spelling fixes; from paul tagliamonte
any parts of his diff not taken are noted on tech

Rework statistic collection to be per repository and add metric output option

Many statistic values are now accounted by repository via repo_stat_inc()
At end of the run sum_stats() accumulates these stats per TAL and globally.
The new output file metrics is written when the -m output flag is specified.
The metrics file is written in OpenMetrics format (with a few tweaks to
allow node_exporter to parse the file as well). The ometric code is a copy
from bgpctl(8) and should be kept in sync.
OK tb@

Introduce a deadline timer that aborts all repository syncs.
With this rpki-client has a chance to still finish and produce an output
even when a CA is excessivly slow and holds back progress.
With and OK benno@ tb@ and job@

Use the abort commands when a repo timeout happens. This is cleaner
then just failing the repo fetch but leaving the backends running.
OK tb@

Move mkpath logic after checking for 'noop' to prevent creation of directories in -n mode

OK claudio@

In RRDP mode it is possible that a file in the validated cache dir is
no longer valid while the newer file in the .rrdp directory is not (yet)
valid. In this case do not copy the old file over the newer file.
RRDP has no method to identiy inconsistent repos (unlike rsync) so if
the wrong file is put in .rrdp it is not noticed appart from the
verification process failing for that file.
OK tb@ job@

infromation -> information

More KNF and whitespace fixes.

Fix various annoying whitespace errors.

Adjust cache cleanup to the deficits of RRDP. Unlike rysnc, RRDP has no
method to sync the cache. It just depends on all deltas to work but has
no method to check if the result is consistent with the source.
Because of this do not unlink files which were fetched via RRDP. Instead
move them back to their .rrdp directory and hope the will be cleaned up.
This should help to keep the cache coherent in some edge cases.
OK tb@

Draw a bit more attention to transport switches between RRDP, RSYNC, and cache

OK claudio@ tb@

When a RRDP fetch failed not only clear the filesystem but also flush
the to delete list stored this RRDP node.
Noticed by Job on console.rpki-client.org with the help of idnic.net
OK tb@

Do not skip .rrdp cache cleanup if rrdp is off (option -R). The sync
via rsync alters the cache and rrdp can only recover by downloading
a snapshot. By doing the cleanup here it will make sure this happens.
Noticed by job@
OK tb@

Change the repository layout by removing the valid directory. Everything
that was in that directory is now in the chachedir root. The rsync and
rrdp directories are now .rsync/ and .rrdp/. The ta/ directory still
remains because TAs are special.
Idea and most of the diff from job@
OK tb@ and benno@

When rename fails show the source filename and not the destination.
The error should be more helpful thisway.
OK tb@

Handle EINTR the same way in all poll loops. In all cases restart the
poll loop. In the main process move the timeout handling for repositories
into a single function that does the timeouts and the calculation of the
timeout in one go.
OK tb@

Introduce a validated cache which holds all the files that have
successfully been verified by rpki-client.

With this the rsync and rrdp directories are more of a temporary storage
location. New files are downloaded there and then moved to the valid
directory at the end. In -n mode only the valid directory is looked at with
the exception of the ta directory holding the trust anchors.
A file can now be in two different locations so adjust all the code paths
that open files to check both locations.
One nice side-effect of this is that the RRDP handling in the main process
got simplified. There is no longer the need for temporary RRDP directories.
OK tb@

Move some functions around to reduce diff with an upcoming change.
OK tb@

Alter valid_filehash() to take a file descriptor instead of a path.
This is needed so that callers can allow a file to be in multiple
locations. Also move mft_check() from mft.c to parser.c.
OK tb@

Implement a RRDP_CLEAR message that instructs the parent to cleanup
the rrdp directory. This is used before a snapshot download to ensure
that the snapshot is applied to a clean repo.
Similar cleanup happens if the transfer fails. In that case remove the
temp directory contents only.
This uses a new function remove_contents() to remove everything below
a base directory (a bit like rm -r X/*).
OK tb@

Add the same entityq_flush() logic in ta_lookup() as it is done in
repo_lookup(). This fixes -n mode.
OK tb@

Change the way the parser accesses files. It now builds the file path
based on information from the repository, a local path and the filename.
This simplifies some code both in the main process and the parser.
For this to work repositories are passed to the parser before any other
entity of this repository is passed. Struct entity is extended to include
the repoid and the path along the file(name).
Input and OK tb@ & job@

Fix -n mode by setting repository states to REPO_DONE instead of the
more correct REPO_FAILED. The problem is that with REPO_FAILED the repo
no longer matches a subsequent rrdp_get lookup and as a result the repo
uses rsync which wrong in this case.
OK benno@ tb@

Merge multiple similar code bits together into repo_done(). Which currently
finds all repos that point to the passed backend and flushes their queues.
Also make sure that REPO_FAILED is set in cases where no sync happened and
also fall back from rrdp to rsync in any case of failure (including errors
merging the rrdp repo itself).
OK tb@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Simplify code a bit. There is only one TA per TAL and so only one
ta_lookup(). Implementing the talrepocnt limiter there makes little
sense and gains us nothing.
OK job@

Use one common function to build the base directory path for repositories.
OK benno@

Replace MAX_REPO_TIMEOUT with repo_timeout, which is set to 1/4th of
timeout, or if timeout is disabled set to 24 hours.

OK claudio@

Check for possible negative diff timeout. Clamp it to 0 in that case.
poll with negative timeout noticed by job@
OK job@

Adjust how the repository count limit works. Instead of failing hard just
fall back to a possible cache and try to validate what is available.
This still limits the number of repositories fetched but allows valid
repositories to finish with the available data.
OK job@

Limit the number of publication points under a given TAL.
Introduce an additional timeout for each publication point.

The limits are large enough to accomodate normal operating levels.

With and OK benno@ job@ tb@ beck@ deraadt@

Instead of creating a struct repo for each unique caRepository URI
use the rsync URI (a base version of caRepository) and the notify URI
to identify repositories. If both rsync URI and notify URI are the same
then the repo is the same. The notify URI is optional and can be NULL
so the lookup needs to be a bit careful.
This reduces the number of struct repos from 26k to around 50.
OK tb@

branches: 1.9.4;
Rework the logic for fetching TA files a bit. Move the logic that checks
if all possible sources have been used to ta_fetch() this simplifies the
logic in the rsync and http done cases.
Additionally when -R is used, skip https URI and only connect via rsync.
If the TAL file has no rsync URI then the system will fall back to the
cache and not try https instead.
This makes -R a only use rsync flag.
With and OK job@

Change errors for file manipulations from err(3) to warn(3) and adjust the
surrounding code to handle these errors gracefully. When rpki-client runs
out of space it will keep on running including the cleanup at the end.
This way the temporary and old files are removed hopfully giving back some
space. This affects mainly RRDP and the initial fetch of TA files.
OK job@

Similar to rsync repo delay deletes for rrdp repos to after processing
all files. Now RRDP has a weak sync mechanism and therefor not all
unreferenced files can be removed, instead look at the list of removes
which were part of the delta processing and remove those files.
Warn and keep the file if a delta specifies to remove a file which is
actually still referenced.
OK benno@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.5.2;
Before adding a file to the temporary rrdp repo remove it from the
deleted filepath set. A file can only be in one set (deleted or added)
but not on both.
OK benno@

When merging a repo even files to delete can be part of the temporary
work dir. So unlink can return an ENOENT error for the main repo. In
which case the temp dir should be tried.
Refactor this code a bit since there is no way rrdp_filename() should
fail in this part of the code.
OK tb@

fix typo + some whitespace

spelling

Initial commit of RRDP (The RPKI Repository Delta Protocol - RFC8182) support
in rpki-client. For now it is off by default.

All XML processing is done in its own process with minimal pledge rights.
It uses the already present https process to fetch the xml files and uses
the master porcess to handle the file IO into the repositories.
RRDP data is stored in the cache under ./rrdp/ and the first directory
is the SHA256 hash of the notify URI.

Fetching snapshots and deltas works to bring the cache up to date.
If something goes wrong rpki-client will fall back to rsync.

RRDP was implemented by Nils Fisher and integrated into rpki-client by myself.
"Time to get it in" deraadt@

Properly setup the fts_state in the FTS_D case.

At fts_level 1 the state needs to be fully reset since we most probably
exited from a directory at level 1 and entered a new dir at level 1.
Without this empty directories remained since the fts_state.type
was wrong for those entries.
Noticed by job@, OK tb@

Instead of tracking certificates by SKI track them by an internal identifier.

The certificate SKI is not strictly unique so using it as a unique id is
problematic. It is also not really needed to do that since in theory we
already know the path (but this got lost in the privsep communication).
So add a cert id and pass this id back and forth between main process and
the parser. With this id we can lookup the authentication chain in the
parser and this even works with multiple paths to the same resource.
Since we no longer lookup by SKI the valid_aki_ski function is replaced
by find_issuer() which does the lookup by certid.

The loop protection is now extended to allow each TAL to reach each file
once but still triggers if a file is reaccessed by the tree of a TAL.

In filemode the lookup now uses an AIA uri based lookup tree. Again this
replaces the SKI based lookups from before.

Done together with tb@
OK tb@ job@

P-256 support is experimental so require -x to enable it.

Also clean up the externs a little bit by moving experimental and noop
to extern.h.
Reminded by and OK tb@

Fix capping of VAPs

The previous approach introduced a cap, but it might not always be hit as
intended (I missed this on review). Fix this to check the cap after merging
an ASPA into an already existing VAP. Also free the list of providers since
nothing should be looking at it anymore.

Count VAPs that hit the limit with a new overflowed counter. There are
still a few aspects of the accounting that probably aren't entirely right.
This will be fixed at another point. It's just statistics after all.

with/ok claudio, ok job

Replace protocol literal strings and strlen() calls with defined constants

OK tb@ claudio@

Track the number of new files moving from 'staging' to 'validated cache'

The OpenMetrics output shows per-repository counters for new files
added, the main process and JSON output emit the sum of all new files.

OK claudio@

Add support for RPKI Signed Prefix Lists

Signed Prefix List are a CMS protected content type for use with the
RPKI to carry the complete list of prefixes which an Autonomous System
may originate to all or any of its routing peers. The validation of a
Signed Prefix List confirms that the holder of the listed ASN produced
the object, and that this list is a current, accurate and complete
description of address prefixes that may be announced into the routing
system originated by this AS.

https://datatracker.ietf.org/doc/html/draft-ietf-sidrops-rpki-prefixlist

with and OK claudio@ tb@

Refactor handling of stale manifests

No need to hoist a staleness indicator through the whole process and
count it explicitly.

OK tb@

Do not set O_EXCL on open() when a published file is on the withdraw list.

If during the process of applying RRDP deltas a object is published then
withdrawn and later published again the current code fails. This is because
published elements use O_EXCL in open() but in the case above the previous
file is still around since withdraws are delayed.

Problem noticed by job@
OK tb@

rpki-client: fix vap_pas stats

A small mistake in a diff broke the counters. Make them AFI agnostic and
adjust ometric output.

guidance & ok claudio

In rrdp_session_save() stop the loop over deltas once MAX_RRDP_DELTAS
have been processed.
With and OK tb@

Improve detection of RRDP session desynchronization

According to RFC 8182, a given session_id and serial number represent an
immutable record of the state of the Repository Server at a certain
point in time.

Add a check to the RRDP notification file processing to compare whether
the delta hashes associated to previously seen serials are different in
newly fetched notification files. Fall back to a snapshot if a difference
is detected, because such a mutation is a strong desynchronization
indicator.

Idea from Ties de Kock (RIPE NCC).
Based on a diff by job@
With and OK job@ tb@

Fixup file modification timestamps to optimize failover from RRDP to RSYNC

In the RSYNC protocol a file's last modification time and its size are
used to determine whether sending a (partial) copy over the wire is needed.
Previously, when RRDP data structures are serialized to disk, the mtime of
files in DIR_VALID ended up being UTIME_NOW.

Thus, the mtimes of files obtained through RRDP will never match the mtimes
of the same files available through RSYNC - causing each and every file to
be added to the file transfer list.

Instead, use the internal timestamps of RPKI files as the last modified
timestamp. Specifically, for Signed Objects (ROAs, MFTs, GBRs, TAKs, ASPAs)
the CMS signing-time, for .cer files the X.509 notBefore, and for .crl files
the CRL lastUpdate. This results in a surprising optimization for the number
files which have to be transfered.

OK claudio@

Fix repo_cleanup_entry() state machine so that the repository lookups
are done when the full repo path is available. Without this all repo
lookups returned NULL and the code did not work as intended.
OK tb@

RRDP snapshots should encode publish elements only once. If encountered
fail the transfer and fall back to rsync.

When more than one publish element for the same file exist the RP does
not know which one to choose. Lets fail the RRDP transfer in this case
and fall back to rsync. CA that publish a file more than once are buggy
and need to be fixed.
OK job@ tb@

Improve accounting by tracking things by repo and tal.
This fixes some wrong accounting for repositories that are referenced
from more than one TAL. It changes the ometric lable output a little bit
since there are repository metrics that no longer include the 'name' label.
OK tb@

Add the protocol used to sync the repository to the open-metric output.
OK tb@

Instead of storing the short TAL name as repouri use the first TAL URI
for that. This is more in line with the other uses of repouri.
OK tb@

spelling fixes; from paul tagliamonte
any parts of his diff not taken are noted on tech

Rework statistic collection to be per repository and add metric output option

Many statistic values are now accounted by repository via repo_stat_inc()
At end of the run sum_stats() accumulates these stats per TAL and globally.
The new output file metrics is written when the -m output flag is specified.
The metrics file is written in OpenMetrics format (with a few tweaks to
allow node_exporter to parse the file as well). The ometric code is a copy
from bgpctl(8) and should be kept in sync.
OK tb@

Introduce a deadline timer that aborts all repository syncs.
With this rpki-client has a chance to still finish and produce an output
even when a CA is excessivly slow and holds back progress.
With and OK benno@ tb@ and job@

Use the abort commands when a repo timeout happens. This is cleaner
then just failing the repo fetch but leaving the backends running.
OK tb@

Move mkpath logic after checking for 'noop' to prevent creation of directories in -n mode

OK claudio@

In RRDP mode it is possible that a file in the validated cache dir is
no longer valid while the newer file in the .rrdp directory is not (yet)
valid. In this case do not copy the old file over the newer file.
RRDP has no method to identiy inconsistent repos (unlike rsync) so if
the wrong file is put in .rrdp it is not noticed appart from the
verification process failing for that file.
OK tb@ job@

infromation -> information

More KNF and whitespace fixes.

Fix various annoying whitespace errors.

Adjust cache cleanup to the deficits of RRDP. Unlike rysnc, RRDP has no
method to sync the cache. It just depends on all deltas to work but has
no method to check if the result is consistent with the source.
Because of this do not unlink files which were fetched via RRDP. Instead
move them back to their .rrdp directory and hope the will be cleaned up.
This should help to keep the cache coherent in some edge cases.
OK tb@

Draw a bit more attention to transport switches between RRDP, RSYNC, and cache

OK claudio@ tb@

When a RRDP fetch failed not only clear the filesystem but also flush
the to delete list stored this RRDP node.
Noticed by Job on console.rpki-client.org with the help of idnic.net
OK tb@

Do not skip .rrdp cache cleanup if rrdp is off (option -R). The sync
via rsync alters the cache and rrdp can only recover by downloading
a snapshot. By doing the cleanup here it will make sure this happens.
Noticed by job@
OK tb@

Change the repository layout by removing the valid directory. Everything
that was in that directory is now in the chachedir root. The rsync and
rrdp directories are now .rsync/ and .rrdp/. The ta/ directory still
remains because TAs are special.
Idea and most of the diff from job@
OK tb@ and benno@

When rename fails show the source filename and not the destination.
The error should be more helpful thisway.
OK tb@

Handle EINTR the same way in all poll loops. In all cases restart the
poll loop. In the main process move the timeout handling for repositories
into a single function that does the timeouts and the calculation of the
timeout in one go.
OK tb@

Introduce a validated cache which holds all the files that have
successfully been verified by rpki-client.

With this the rsync and rrdp directories are more of a temporary storage
location. New files are downloaded there and then moved to the valid
directory at the end. In -n mode only the valid directory is looked at with
the exception of the ta directory holding the trust anchors.
A file can now be in two different locations so adjust all the code paths
that open files to check both locations.
One nice side-effect of this is that the RRDP handling in the main process
got simplified. There is no longer the need for temporary RRDP directories.
OK tb@

Move some functions around to reduce diff with an upcoming change.
OK tb@

Alter valid_filehash() to take a file descriptor instead of a path.
This is needed so that callers can allow a file to be in multiple
locations. Also move mft_check() from mft.c to parser.c.
OK tb@

Implement a RRDP_CLEAR message that instructs the parent to cleanup
the rrdp directory. This is used before a snapshot download to ensure
that the snapshot is applied to a clean repo.
Similar cleanup happens if the transfer fails. In that case remove the
temp directory contents only.
This uses a new function remove_contents() to remove everything below
a base directory (a bit like rm -r X/*).
OK tb@

Add the same entityq_flush() logic in ta_lookup() as it is done in
repo_lookup(). This fixes -n mode.
OK tb@

Change the way the parser accesses files. It now builds the file path
based on information from the repository, a local path and the filename.
This simplifies some code both in the main process and the parser.
For this to work repositories are passed to the parser before any other
entity of this repository is passed. Struct entity is extended to include
the repoid and the path along the file(name).
Input and OK tb@ & job@

Fix -n mode by setting repository states to REPO_DONE instead of the
more correct REPO_FAILED. The problem is that with REPO_FAILED the repo
no longer matches a subsequent rrdp_get lookup and as a result the repo
uses rsync which wrong in this case.
OK benno@ tb@

Merge multiple similar code bits together into repo_done(). Which currently
finds all repos that point to the passed backend and flushes their queues.
Also make sure that REPO_FAILED is set in cases where no sync happened and
also fall back from rrdp to rsync in any case of failure (including errors
merging the rrdp repo itself).
OK tb@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Simplify code a bit. There is only one TA per TAL and so only one
ta_lookup(). Implementing the talrepocnt limiter there makes little
sense and gains us nothing.
OK job@

Use one common function to build the base directory path for repositories.
OK benno@

Replace MAX_REPO_TIMEOUT with repo_timeout, which is set to 1/4th of
timeout, or if timeout is disabled set to 24 hours.

OK claudio@

Check for possible negative diff timeout. Clamp it to 0 in that case.
poll with negative timeout noticed by job@
OK job@

Adjust how the repository count limit works. Instead of failing hard just
fall back to a possible cache and try to validate what is available.
This still limits the number of repositories fetched but allows valid
repositories to finish with the available data.
OK job@

Limit the number of publication points under a given TAL.
Introduce an additional timeout for each publication point.

The limits are large enough to accomodate normal operating levels.

With and OK benno@ job@ tb@ beck@ deraadt@

Instead of creating a struct repo for each unique caRepository URI
use the rsync URI (a base version of caRepository) and the notify URI
to identify repositories. If both rsync URI and notify URI are the same
then the repo is the same. The notify URI is optional and can be NULL
so the lookup needs to be a bit careful.
This reduces the number of struct repos from 26k to around 50.
OK tb@

branches: 1.9.4;
Rework the logic for fetching TA files a bit. Move the logic that checks
if all possible sources have been used to ta_fetch() this simplifies the
logic in the rsync and http done cases.
Additionally when -R is used, skip https URI and only connect via rsync.
If the TAL file has no rsync URI then the system will fall back to the
cache and not try https instead.
This makes -R a only use rsync flag.
With and OK job@

Change errors for file manipulations from err(3) to warn(3) and adjust the
surrounding code to handle these errors gracefully. When rpki-client runs
out of space it will keep on running including the cleanup at the end.
This way the temporary and old files are removed hopfully giving back some
space. This affects mainly RRDP and the initial fetch of TA files.
OK job@

Similar to rsync repo delay deletes for rrdp repos to after processing
all files. Now RRDP has a weak sync mechanism and therefor not all
unreferenced files can be removed, instead look at the list of removes
which were part of the delta processing and remove those files.
Warn and keep the file if a delta specifies to remove a file which is
actually still referenced.
OK benno@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.5.2;
Before adding a file to the temporary rrdp repo remove it from the
deleted filepath set. A file can only be in one set (deleted or added)
but not on both.
OK benno@

When merging a repo even files to delete can be part of the temporary
work dir. So unlink can return an ENOENT error for the main repo. In
which case the temp dir should be tried.
Refactor this code a bit since there is no way rrdp_filename() should
fail in this part of the code.
OK tb@

fix typo + some whitespace

spelling

Initial commit of RRDP (The RPKI Repository Delta Protocol - RFC8182) support
in rpki-client. For now it is off by default.

All XML processing is done in its own process with minimal pledge rights.
It uses the already present https process to fetch the xml files and uses
the master porcess to handle the file IO into the repositories.
RRDP data is stored in the cache under ./rrdp/ and the first directory
is the SHA256 hash of the notify URI.

Fetching snapshots and deltas works to bring the cache up to date.
If something goes wrong rpki-client will fall back to rsync.

RRDP was implemented by Nils Fisher and integrated into rpki-client by myself.
"Time to get it in" deraadt@

Instead of tracking certificates by SKI track them by an internal identifier.

The certificate SKI is not strictly unique so using it as a unique id is
problematic. It is also not really needed to do that since in theory we
already know the path (but this got lost in the privsep communication).
So add a cert id and pass this id back and forth between main process and
the parser. With this id we can lookup the authentication chain in the
parser and this even works with multiple paths to the same resource.
Since we no longer lookup by SKI the valid_aki_ski function is replaced
by find_issuer() which does the lookup by certid.

The loop protection is now extended to allow each TAL to reach each file
once but still triggers if a file is reaccessed by the tree of a TAL.

In filemode the lookup now uses an AIA uri based lookup tree. Again this
replaces the SKI based lookups from before.

Done together with tb@
OK tb@ job@

P-256 support is experimental so require -x to enable it.

Also clean up the externs a little bit by moving experimental and noop
to extern.h.
Reminded by and OK tb@

Fix capping of VAPs

The previous approach introduced a cap, but it might not always be hit as
intended (I missed this on review). Fix this to check the cap after merging
an ASPA into an already existing VAP. Also free the list of providers since
nothing should be looking at it anymore.

Count VAPs that hit the limit with a new overflowed counter. There are
still a few aspects of the accounting that probably aren't entirely right.
This will be fixed at another point. It's just statistics after all.

with/ok claudio, ok job

Replace protocol literal strings and strlen() calls with defined constants

OK tb@ claudio@

Track the number of new files moving from 'staging' to 'validated cache'

The OpenMetrics output shows per-repository counters for new files
added, the main process and JSON output emit the sum of all new files.

OK claudio@

Add support for RPKI Signed Prefix Lists

Signed Prefix List are a CMS protected content type for use with the
RPKI to carry the complete list of prefixes which an Autonomous System
may originate to all or any of its routing peers. The validation of a
Signed Prefix List confirms that the holder of the listed ASN produced
the object, and that this list is a current, accurate and complete
description of address prefixes that may be announced into the routing
system originated by this AS.

https://datatracker.ietf.org/doc/html/draft-ietf-sidrops-rpki-prefixlist

with and OK claudio@ tb@

Refactor handling of stale manifests

No need to hoist a staleness indicator through the whole process and
count it explicitly.

OK tb@

Do not set O_EXCL on open() when a published file is on the withdraw list.

If during the process of applying RRDP deltas a object is published then
withdrawn and later published again the current code fails. This is because
published elements use O_EXCL in open() but in the case above the previous
file is still around since withdraws are delayed.

Problem noticed by job@
OK tb@

rpki-client: fix vap_pas stats

A small mistake in a diff broke the counters. Make them AFI agnostic and
adjust ometric output.

guidance & ok claudio

In rrdp_session_save() stop the loop over deltas once MAX_RRDP_DELTAS
have been processed.
With and OK tb@

Improve detection of RRDP session desynchronization

According to RFC 8182, a given session_id and serial number represent an
immutable record of the state of the Repository Server at a certain
point in time.

Add a check to the RRDP notification file processing to compare whether
the delta hashes associated to previously seen serials are different in
newly fetched notification files. Fall back to a snapshot if a difference
is detected, because such a mutation is a strong desynchronization
indicator.

Idea from Ties de Kock (RIPE NCC).
Based on a diff by job@
With and OK job@ tb@

Fixup file modification timestamps to optimize failover from RRDP to RSYNC

In the RSYNC protocol a file's last modification time and its size are
used to determine whether sending a (partial) copy over the wire is needed.
Previously, when RRDP data structures are serialized to disk, the mtime of
files in DIR_VALID ended up being UTIME_NOW.

Thus, the mtimes of files obtained through RRDP will never match the mtimes
of the same files available through RSYNC - causing each and every file to
be added to the file transfer list.

Instead, use the internal timestamps of RPKI files as the last modified
timestamp. Specifically, for Signed Objects (ROAs, MFTs, GBRs, TAKs, ASPAs)
the CMS signing-time, for .cer files the X.509 notBefore, and for .crl files
the CRL lastUpdate. This results in a surprising optimization for the number
files which have to be transfered.

OK claudio@

Fix repo_cleanup_entry() state machine so that the repository lookups
are done when the full repo path is available. Without this all repo
lookups returned NULL and the code did not work as intended.
OK tb@

RRDP snapshots should encode publish elements only once. If encountered
fail the transfer and fall back to rsync.

When more than one publish element for the same file exist the RP does
not know which one to choose. Lets fail the RRDP transfer in this case
and fall back to rsync. CA that publish a file more than once are buggy
and need to be fixed.
OK job@ tb@

Improve accounting by tracking things by repo and tal.
This fixes some wrong accounting for repositories that are referenced
from more than one TAL. It changes the ometric lable output a little bit
since there are repository metrics that no longer include the 'name' label.
OK tb@

Add the protocol used to sync the repository to the open-metric output.
OK tb@

Instead of storing the short TAL name as repouri use the first TAL URI
for that. This is more in line with the other uses of repouri.
OK tb@

spelling fixes; from paul tagliamonte
any parts of his diff not taken are noted on tech

Rework statistic collection to be per repository and add metric output option

Many statistic values are now accounted by repository via repo_stat_inc()
At end of the run sum_stats() accumulates these stats per TAL and globally.
The new output file metrics is written when the -m output flag is specified.
The metrics file is written in OpenMetrics format (with a few tweaks to
allow node_exporter to parse the file as well). The ometric code is a copy
from bgpctl(8) and should be kept in sync.
OK tb@

Introduce a deadline timer that aborts all repository syncs.
With this rpki-client has a chance to still finish and produce an output
even when a CA is excessivly slow and holds back progress.
With and OK benno@ tb@ and job@

Use the abort commands when a repo timeout happens. This is cleaner
then just failing the repo fetch but leaving the backends running.
OK tb@

Move mkpath logic after checking for 'noop' to prevent creation of directories in -n mode

OK claudio@

In RRDP mode it is possible that a file in the validated cache dir is
no longer valid while the newer file in the .rrdp directory is not (yet)
valid. In this case do not copy the old file over the newer file.
RRDP has no method to identiy inconsistent repos (unlike rsync) so if
the wrong file is put in .rrdp it is not noticed appart from the
verification process failing for that file.
OK tb@ job@

infromation -> information

More KNF and whitespace fixes.

Fix various annoying whitespace errors.

Adjust cache cleanup to the deficits of RRDP. Unlike rysnc, RRDP has no
method to sync the cache. It just depends on all deltas to work but has
no method to check if the result is consistent with the source.
Because of this do not unlink files which were fetched via RRDP. Instead
move them back to their .rrdp directory and hope the will be cleaned up.
This should help to keep the cache coherent in some edge cases.
OK tb@

Draw a bit more attention to transport switches between RRDP, RSYNC, and cache

OK claudio@ tb@

When a RRDP fetch failed not only clear the filesystem but also flush
the to delete list stored this RRDP node.
Noticed by Job on console.rpki-client.org with the help of idnic.net
OK tb@

Do not skip .rrdp cache cleanup if rrdp is off (option -R). The sync
via rsync alters the cache and rrdp can only recover by downloading
a snapshot. By doing the cleanup here it will make sure this happens.
Noticed by job@
OK tb@

Change the repository layout by removing the valid directory. Everything
that was in that directory is now in the chachedir root. The rsync and
rrdp directories are now .rsync/ and .rrdp/. The ta/ directory still
remains because TAs are special.
Idea and most of the diff from job@
OK tb@ and benno@

When rename fails show the source filename and not the destination.
The error should be more helpful thisway.
OK tb@

Handle EINTR the same way in all poll loops. In all cases restart the
poll loop. In the main process move the timeout handling for repositories
into a single function that does the timeouts and the calculation of the
timeout in one go.
OK tb@

Introduce a validated cache which holds all the files that have
successfully been verified by rpki-client.

With this the rsync and rrdp directories are more of a temporary storage
location. New files are downloaded there and then moved to the valid
directory at the end. In -n mode only the valid directory is looked at with
the exception of the ta directory holding the trust anchors.
A file can now be in two different locations so adjust all the code paths
that open files to check both locations.
One nice side-effect of this is that the RRDP handling in the main process
got simplified. There is no longer the need for temporary RRDP directories.
OK tb@

Move some functions around to reduce diff with an upcoming change.
OK tb@

Alter valid_filehash() to take a file descriptor instead of a path.
This is needed so that callers can allow a file to be in multiple
locations. Also move mft_check() from mft.c to parser.c.
OK tb@

Implement a RRDP_CLEAR message that instructs the parent to cleanup
the rrdp directory. This is used before a snapshot download to ensure
that the snapshot is applied to a clean repo.
Similar cleanup happens if the transfer fails. In that case remove the
temp directory contents only.
This uses a new function remove_contents() to remove everything below
a base directory (a bit like rm -r X/*).
OK tb@

Add the same entityq_flush() logic in ta_lookup() as it is done in
repo_lookup(). This fixes -n mode.
OK tb@

Change the way the parser accesses files. It now builds the file path
based on information from the repository, a local path and the filename.
This simplifies some code both in the main process and the parser.
For this to work repositories are passed to the parser before any other
entity of this repository is passed. Struct entity is extended to include
the repoid and the path along the file(name).
Input and OK tb@ & job@

Fix -n mode by setting repository states to REPO_DONE instead of the
more correct REPO_FAILED. The problem is that with REPO_FAILED the repo
no longer matches a subsequent rrdp_get lookup and as a result the repo
uses rsync which wrong in this case.
OK benno@ tb@

Merge multiple similar code bits together into repo_done(). Which currently
finds all repos that point to the passed backend and flushes their queues.
Also make sure that REPO_FAILED is set in cases where no sync happened and
also fall back from rrdp to rsync in any case of failure (including errors
merging the rrdp repo itself).
OK tb@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Simplify code a bit. There is only one TA per TAL and so only one
ta_lookup(). Implementing the talrepocnt limiter there makes little
sense and gains us nothing.
OK job@

Use one common function to build the base directory path for repositories.
OK benno@

Replace MAX_REPO_TIMEOUT with repo_timeout, which is set to 1/4th of
timeout, or if timeout is disabled set to 24 hours.

OK claudio@

Check for possible negative diff timeout. Clamp it to 0 in that case.
poll with negative timeout noticed by job@
OK job@

Adjust how the repository count limit works. Instead of failing hard just
fall back to a possible cache and try to validate what is available.
This still limits the number of repositories fetched but allows valid
repositories to finish with the available data.
OK job@

Limit the number of publication points under a given TAL.
Introduce an additional timeout for each publication point.

The limits are large enough to accomodate normal operating levels.

With and OK benno@ job@ tb@ beck@ deraadt@

Instead of creating a struct repo for each unique caRepository URI
use the rsync URI (a base version of caRepository) and the notify URI
to identify repositories. If both rsync URI and notify URI are the same
then the repo is the same. The notify URI is optional and can be NULL
so the lookup needs to be a bit careful.
This reduces the number of struct repos from 26k to around 50.
OK tb@

branches: 1.9.4;
Rework the logic for fetching TA files a bit. Move the logic that checks
if all possible sources have been used to ta_fetch() this simplifies the
logic in the rsync and http done cases.
Additionally when -R is used, skip https URI and only connect via rsync.
If the TAL file has no rsync URI then the system will fall back to the
cache and not try https instead.
This makes -R a only use rsync flag.
With and OK job@

Change errors for file manipulations from err(3) to warn(3) and adjust the
surrounding code to handle these errors gracefully. When rpki-client runs
out of space it will keep on running including the cleanup at the end.
This way the temporary and old files are removed hopfully giving back some
space. This affects mainly RRDP and the initial fetch of TA files.
OK job@

Similar to rsync repo delay deletes for rrdp repos to after processing
all files. Now RRDP has a weak sync mechanism and therefor not all
unreferenced files can be removed, instead look at the list of removes
which were part of the delta processing and remove those files.
Warn and keep the file if a delta specifies to remove a file which is
actually still referenced.
OK benno@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.5.2;
Before adding a file to the temporary rrdp repo remove it from the
deleted filepath set. A file can only be in one set (deleted or added)
but not on both.
OK benno@

When merging a repo even files to delete can be part of the temporary
work dir. So unlink can return an ENOENT error for the main repo. In
which case the temp dir should be tried.
Refactor this code a bit since there is no way rrdp_filename() should
fail in this part of the code.
OK tb@

fix typo + some whitespace

spelling

Initial commit of RRDP (The RPKI Repository Delta Protocol - RFC8182) support
in rpki-client. For now it is off by default.

All XML processing is done in its own process with minimal pledge rights.
It uses the already present https process to fetch the xml files and uses
the master porcess to handle the file IO into the repositories.
RRDP data is stored in the cache under ./rrdp/ and the first directory
is the SHA256 hash of the notify URI.

Fetching snapshots and deltas works to bring the cache up to date.
If something goes wrong rpki-client will fall back to rsync.

RRDP was implemented by Nils Fisher and integrated into rpki-client by myself.
"Time to get it in" deraadt@

P-256 support is experimental so require -x to enable it.

Also clean up the externs a little bit by moving experimental and noop
to extern.h.
Reminded by and OK tb@

Fix capping of VAPs

The previous approach introduced a cap, but it might not always be hit as
intended (I missed this on review). Fix this to check the cap after merging
an ASPA into an already existing VAP. Also free the list of providers since
nothing should be looking at it anymore.

Count VAPs that hit the limit with a new overflowed counter. There are
still a few aspects of the accounting that probably aren't entirely right.
This will be fixed at another point. It's just statistics after all.

with/ok claudio, ok job

Replace protocol literal strings and strlen() calls with defined constants

OK tb@ claudio@

Track the number of new files moving from 'staging' to 'validated cache'

The OpenMetrics output shows per-repository counters for new files
added, the main process and JSON output emit the sum of all new files.

OK claudio@

Add support for RPKI Signed Prefix Lists

Signed Prefix List are a CMS protected content type for use with the
RPKI to carry the complete list of prefixes which an Autonomous System
may originate to all or any of its routing peers. The validation of a
Signed Prefix List confirms that the holder of the listed ASN produced
the object, and that this list is a current, accurate and complete
description of address prefixes that may be announced into the routing
system originated by this AS.

https://datatracker.ietf.org/doc/html/draft-ietf-sidrops-rpki-prefixlist

with and OK claudio@ tb@

Refactor handling of stale manifests

No need to hoist a staleness indicator through the whole process and
count it explicitly.

OK tb@

Do not set O_EXCL on open() when a published file is on the withdraw list.

If during the process of applying RRDP deltas a object is published then
withdrawn and later published again the current code fails. This is because
published elements use O_EXCL in open() but in the case above the previous
file is still around since withdraws are delayed.

Problem noticed by job@
OK tb@

rpki-client: fix vap_pas stats

A small mistake in a diff broke the counters. Make them AFI agnostic and
adjust ometric output.

guidance & ok claudio

In rrdp_session_save() stop the loop over deltas once MAX_RRDP_DELTAS
have been processed.
With and OK tb@

Improve detection of RRDP session desynchronization

According to RFC 8182, a given session_id and serial number represent an
immutable record of the state of the Repository Server at a certain
point in time.

Add a check to the RRDP notification file processing to compare whether
the delta hashes associated to previously seen serials are different in
newly fetched notification files. Fall back to a snapshot if a difference
is detected, because such a mutation is a strong desynchronization
indicator.

Idea from Ties de Kock (RIPE NCC).
Based on a diff by job@
With and OK job@ tb@

Fixup file modification timestamps to optimize failover from RRDP to RSYNC

In the RSYNC protocol a file's last modification time and its size are
used to determine whether sending a (partial) copy over the wire is needed.
Previously, when RRDP data structures are serialized to disk, the mtime of
files in DIR_VALID ended up being UTIME_NOW.

Thus, the mtimes of files obtained through RRDP will never match the mtimes
of the same files available through RSYNC - causing each and every file to
be added to the file transfer list.

Instead, use the internal timestamps of RPKI files as the last modified
timestamp. Specifically, for Signed Objects (ROAs, MFTs, GBRs, TAKs, ASPAs)
the CMS signing-time, for .cer files the X.509 notBefore, and for .crl files
the CRL lastUpdate. This results in a surprising optimization for the number
files which have to be transfered.

OK claudio@

Fix repo_cleanup_entry() state machine so that the repository lookups
are done when the full repo path is available. Without this all repo
lookups returned NULL and the code did not work as intended.
OK tb@

RRDP snapshots should encode publish elements only once. If encountered
fail the transfer and fall back to rsync.

When more than one publish element for the same file exist the RP does
not know which one to choose. Lets fail the RRDP transfer in this case
and fall back to rsync. CA that publish a file more than once are buggy
and need to be fixed.
OK job@ tb@

Improve accounting by tracking things by repo and tal.
This fixes some wrong accounting for repositories that are referenced
from more than one TAL. It changes the ometric lable output a little bit
since there are repository metrics that no longer include the 'name' label.
OK tb@

Add the protocol used to sync the repository to the open-metric output.
OK tb@

Instead of storing the short TAL name as repouri use the first TAL URI
for that. This is more in line with the other uses of repouri.
OK tb@

spelling fixes; from paul tagliamonte
any parts of his diff not taken are noted on tech

Rework statistic collection to be per repository and add metric output option

Many statistic values are now accounted by repository via repo_stat_inc()
At end of the run sum_stats() accumulates these stats per TAL and globally.
The new output file metrics is written when the -m output flag is specified.
The metrics file is written in OpenMetrics format (with a few tweaks to
allow node_exporter to parse the file as well). The ometric code is a copy
from bgpctl(8) and should be kept in sync.
OK tb@

Introduce a deadline timer that aborts all repository syncs.
With this rpki-client has a chance to still finish and produce an output
even when a CA is excessivly slow and holds back progress.
With and OK benno@ tb@ and job@

Use the abort commands when a repo timeout happens. This is cleaner
then just failing the repo fetch but leaving the backends running.
OK tb@

Move mkpath logic after checking for 'noop' to prevent creation of directories in -n mode

OK claudio@

In RRDP mode it is possible that a file in the validated cache dir is
no longer valid while the newer file in the .rrdp directory is not (yet)
valid. In this case do not copy the old file over the newer file.
RRDP has no method to identiy inconsistent repos (unlike rsync) so if
the wrong file is put in .rrdp it is not noticed appart from the
verification process failing for that file.
OK tb@ job@

infromation -> information

More KNF and whitespace fixes.

Fix various annoying whitespace errors.

Adjust cache cleanup to the deficits of RRDP. Unlike rysnc, RRDP has no
method to sync the cache. It just depends on all deltas to work but has
no method to check if the result is consistent with the source.
Because of this do not unlink files which were fetched via RRDP. Instead
move them back to their .rrdp directory and hope the will be cleaned up.
This should help to keep the cache coherent in some edge cases.
OK tb@

Draw a bit more attention to transport switches between RRDP, RSYNC, and cache

OK claudio@ tb@

When a RRDP fetch failed not only clear the filesystem but also flush
the to delete list stored this RRDP node.
Noticed by Job on console.rpki-client.org with the help of idnic.net
OK tb@

Do not skip .rrdp cache cleanup if rrdp is off (option -R). The sync
via rsync alters the cache and rrdp can only recover by downloading
a snapshot. By doing the cleanup here it will make sure this happens.
Noticed by job@
OK tb@

Change the repository layout by removing the valid directory. Everything
that was in that directory is now in the chachedir root. The rsync and
rrdp directories are now .rsync/ and .rrdp/. The ta/ directory still
remains because TAs are special.
Idea and most of the diff from job@
OK tb@ and benno@

When rename fails show the source filename and not the destination.
The error should be more helpful thisway.
OK tb@

Handle EINTR the same way in all poll loops. In all cases restart the
poll loop. In the main process move the timeout handling for repositories
into a single function that does the timeouts and the calculation of the
timeout in one go.
OK tb@

Introduce a validated cache which holds all the files that have
successfully been verified by rpki-client.

With this the rsync and rrdp directories are more of a temporary storage
location. New files are downloaded there and then moved to the valid
directory at the end. In -n mode only the valid directory is looked at with
the exception of the ta directory holding the trust anchors.
A file can now be in two different locations so adjust all the code paths
that open files to check both locations.
One nice side-effect of this is that the RRDP handling in the main process
got simplified. There is no longer the need for temporary RRDP directories.
OK tb@

Move some functions around to reduce diff with an upcoming change.
OK tb@

Alter valid_filehash() to take a file descriptor instead of a path.
This is needed so that callers can allow a file to be in multiple
locations. Also move mft_check() from mft.c to parser.c.
OK tb@

Implement a RRDP_CLEAR message that instructs the parent to cleanup
the rrdp directory. This is used before a snapshot download to ensure
that the snapshot is applied to a clean repo.
Similar cleanup happens if the transfer fails. In that case remove the
temp directory contents only.
This uses a new function remove_contents() to remove everything below
a base directory (a bit like rm -r X/*).
OK tb@

Add the same entityq_flush() logic in ta_lookup() as it is done in
repo_lookup(). This fixes -n mode.
OK tb@

Change the way the parser accesses files. It now builds the file path
based on information from the repository, a local path and the filename.
This simplifies some code both in the main process and the parser.
For this to work repositories are passed to the parser before any other
entity of this repository is passed. Struct entity is extended to include
the repoid and the path along the file(name).
Input and OK tb@ & job@

Fix -n mode by setting repository states to REPO_DONE instead of the
more correct REPO_FAILED. The problem is that with REPO_FAILED the repo
no longer matches a subsequent rrdp_get lookup and as a result the repo
uses rsync which wrong in this case.
OK benno@ tb@

Merge multiple similar code bits together into repo_done(). Which currently
finds all repos that point to the passed backend and flushes their queues.
Also make sure that REPO_FAILED is set in cases where no sync happened and
also fall back from rrdp to rsync in any case of failure (including errors
merging the rrdp repo itself).
OK tb@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Simplify code a bit. There is only one TA per TAL and so only one
ta_lookup(). Implementing the talrepocnt limiter there makes little
sense and gains us nothing.
OK job@

Use one common function to build the base directory path for repositories.
OK benno@

Replace MAX_REPO_TIMEOUT with repo_timeout, which is set to 1/4th of
timeout, or if timeout is disabled set to 24 hours.

OK claudio@

Check for possible negative diff timeout. Clamp it to 0 in that case.
poll with negative timeout noticed by job@
OK job@

Adjust how the repository count limit works. Instead of failing hard just
fall back to a possible cache and try to validate what is available.
This still limits the number of repositories fetched but allows valid
repositories to finish with the available data.
OK job@

Limit the number of publication points under a given TAL.
Introduce an additional timeout for each publication point.

The limits are large enough to accomodate normal operating levels.

With and OK benno@ job@ tb@ beck@ deraadt@

Instead of creating a struct repo for each unique caRepository URI
use the rsync URI (a base version of caRepository) and the notify URI
to identify repositories. If both rsync URI and notify URI are the same
then the repo is the same. The notify URI is optional and can be NULL
so the lookup needs to be a bit careful.
This reduces the number of struct repos from 26k to around 50.
OK tb@

branches: 1.9.4;
Rework the logic for fetching TA files a bit. Move the logic that checks
if all possible sources have been used to ta_fetch() this simplifies the
logic in the rsync and http done cases.
Additionally when -R is used, skip https URI and only connect via rsync.
If the TAL file has no rsync URI then the system will fall back to the
cache and not try https instead.
This makes -R a only use rsync flag.
With and OK job@

Change errors for file manipulations from err(3) to warn(3) and adjust the
surrounding code to handle these errors gracefully. When rpki-client runs
out of space it will keep on running including the cleanup at the end.
This way the temporary and old files are removed hopfully giving back some
space. This affects mainly RRDP and the initial fetch of TA files.
OK job@

Similar to rsync repo delay deletes for rrdp repos to after processing
all files. Now RRDP has a weak sync mechanism and therefor not all
unreferenced files can be removed, instead look at the list of removes
which were part of the delta processing and remove those files.
Warn and keep the file if a delta specifies to remove a file which is
actually still referenced.
OK benno@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.5.2;
Before adding a file to the temporary rrdp repo remove it from the
deleted filepath set. A file can only be in one set (deleted or added)
but not on both.
OK benno@

When merging a repo even files to delete can be part of the temporary
work dir. So unlink can return an ENOENT error for the main repo. In
which case the temp dir should be tried.
Refactor this code a bit since there is no way rrdp_filename() should
fail in this part of the code.
OK tb@

fix typo + some whitespace

spelling

Initial commit of RRDP (The RPKI Repository Delta Protocol - RFC8182) support
in rpki-client. For now it is off by default.

All XML processing is done in its own process with minimal pledge rights.
It uses the already present https process to fetch the xml files and uses
the master porcess to handle the file IO into the repositories.
RRDP data is stored in the cache under ./rrdp/ and the first directory
is the SHA256 hash of the notify URI.

Fetching snapshots and deltas works to bring the cache up to date.
If something goes wrong rpki-client will fall back to rsync.

RRDP was implemented by Nils Fisher and integrated into rpki-client by myself.
"Time to get it in" deraadt@

Fix capping of VAPs

The previous approach introduced a cap, but it might not always be hit as
intended (I missed this on review). Fix this to check the cap after merging
an ASPA into an already existing VAP. Also free the list of providers since
nothing should be looking at it anymore.

Count VAPs that hit the limit with a new overflowed counter. There are
still a few aspects of the accounting that probably aren't entirely right.
This will be fixed at another point. It's just statistics after all.

with/ok claudio, ok job

Replace protocol literal strings and strlen() calls with defined constants

OK tb@ claudio@

Track the number of new files moving from 'staging' to 'validated cache'

The OpenMetrics output shows per-repository counters for new files
added, the main process and JSON output emit the sum of all new files.

OK claudio@

Add support for RPKI Signed Prefix Lists

Signed Prefix List are a CMS protected content type for use with the
RPKI to carry the complete list of prefixes which an Autonomous System
may originate to all or any of its routing peers. The validation of a
Signed Prefix List confirms that the holder of the listed ASN produced
the object, and that this list is a current, accurate and complete
description of address prefixes that may be announced into the routing
system originated by this AS.

https://datatracker.ietf.org/doc/html/draft-ietf-sidrops-rpki-prefixlist

with and OK claudio@ tb@

Refactor handling of stale manifests

No need to hoist a staleness indicator through the whole process and
count it explicitly.

OK tb@

Do not set O_EXCL on open() when a published file is on the withdraw list.

If during the process of applying RRDP deltas a object is published then
withdrawn and later published again the current code fails. This is because
published elements use O_EXCL in open() but in the case above the previous
file is still around since withdraws are delayed.

Problem noticed by job@
OK tb@

rpki-client: fix vap_pas stats

A small mistake in a diff broke the counters. Make them AFI agnostic and
adjust ometric output.

guidance & ok claudio

In rrdp_session_save() stop the loop over deltas once MAX_RRDP_DELTAS
have been processed.
With and OK tb@

Improve detection of RRDP session desynchronization

According to RFC 8182, a given session_id and serial number represent an
immutable record of the state of the Repository Server at a certain
point in time.

Add a check to the RRDP notification file processing to compare whether
the delta hashes associated to previously seen serials are different in
newly fetched notification files. Fall back to a snapshot if a difference
is detected, because such a mutation is a strong desynchronization
indicator.

Idea from Ties de Kock (RIPE NCC).
Based on a diff by job@
With and OK job@ tb@

Fixup file modification timestamps to optimize failover from RRDP to RSYNC

In the RSYNC protocol a file's last modification time and its size are
used to determine whether sending a (partial) copy over the wire is needed.
Previously, when RRDP data structures are serialized to disk, the mtime of
files in DIR_VALID ended up being UTIME_NOW.

Thus, the mtimes of files obtained through RRDP will never match the mtimes
of the same files available through RSYNC - causing each and every file to
be added to the file transfer list.

Instead, use the internal timestamps of RPKI files as the last modified
timestamp. Specifically, for Signed Objects (ROAs, MFTs, GBRs, TAKs, ASPAs)
the CMS signing-time, for .cer files the X.509 notBefore, and for .crl files
the CRL lastUpdate. This results in a surprising optimization for the number
files which have to be transfered.

OK claudio@

Fix repo_cleanup_entry() state machine so that the repository lookups
are done when the full repo path is available. Without this all repo
lookups returned NULL and the code did not work as intended.
OK tb@

RRDP snapshots should encode publish elements only once. If encountered
fail the transfer and fall back to rsync.

When more than one publish element for the same file exist the RP does
not know which one to choose. Lets fail the RRDP transfer in this case
and fall back to rsync. CA that publish a file more than once are buggy
and need to be fixed.
OK job@ tb@

Improve accounting by tracking things by repo and tal.
This fixes some wrong accounting for repositories that are referenced
from more than one TAL. It changes the ometric lable output a little bit
since there are repository metrics that no longer include the 'name' label.
OK tb@

Add the protocol used to sync the repository to the open-metric output.
OK tb@

Instead of storing the short TAL name as repouri use the first TAL URI
for that. This is more in line with the other uses of repouri.
OK tb@

spelling fixes; from paul tagliamonte
any parts of his diff not taken are noted on tech

Rework statistic collection to be per repository and add metric output option

Many statistic values are now accounted by repository via repo_stat_inc()
At end of the run sum_stats() accumulates these stats per TAL and globally.
The new output file metrics is written when the -m output flag is specified.
The metrics file is written in OpenMetrics format (with a few tweaks to
allow node_exporter to parse the file as well). The ometric code is a copy
from bgpctl(8) and should be kept in sync.
OK tb@

Introduce a deadline timer that aborts all repository syncs.
With this rpki-client has a chance to still finish and produce an output
even when a CA is excessivly slow and holds back progress.
With and OK benno@ tb@ and job@

Use the abort commands when a repo timeout happens. This is cleaner
then just failing the repo fetch but leaving the backends running.
OK tb@

Move mkpath logic after checking for 'noop' to prevent creation of directories in -n mode

OK claudio@

In RRDP mode it is possible that a file in the validated cache dir is
no longer valid while the newer file in the .rrdp directory is not (yet)
valid. In this case do not copy the old file over the newer file.
RRDP has no method to identiy inconsistent repos (unlike rsync) so if
the wrong file is put in .rrdp it is not noticed appart from the
verification process failing for that file.
OK tb@ job@

infromation -> information

More KNF and whitespace fixes.

Fix various annoying whitespace errors.

Adjust cache cleanup to the deficits of RRDP. Unlike rysnc, RRDP has no
method to sync the cache. It just depends on all deltas to work but has
no method to check if the result is consistent with the source.
Because of this do not unlink files which were fetched via RRDP. Instead
move them back to their .rrdp directory and hope the will be cleaned up.
This should help to keep the cache coherent in some edge cases.
OK tb@

Draw a bit more attention to transport switches between RRDP, RSYNC, and cache

OK claudio@ tb@

When a RRDP fetch failed not only clear the filesystem but also flush
the to delete list stored this RRDP node.
Noticed by Job on console.rpki-client.org with the help of idnic.net
OK tb@

Do not skip .rrdp cache cleanup if rrdp is off (option -R). The sync
via rsync alters the cache and rrdp can only recover by downloading
a snapshot. By doing the cleanup here it will make sure this happens.
Noticed by job@
OK tb@

Change the repository layout by removing the valid directory. Everything
that was in that directory is now in the chachedir root. The rsync and
rrdp directories are now .rsync/ and .rrdp/. The ta/ directory still
remains because TAs are special.
Idea and most of the diff from job@
OK tb@ and benno@

When rename fails show the source filename and not the destination.
The error should be more helpful thisway.
OK tb@

Handle EINTR the same way in all poll loops. In all cases restart the
poll loop. In the main process move the timeout handling for repositories
into a single function that does the timeouts and the calculation of the
timeout in one go.
OK tb@

Introduce a validated cache which holds all the files that have
successfully been verified by rpki-client.

With this the rsync and rrdp directories are more of a temporary storage
location. New files are downloaded there and then moved to the valid
directory at the end. In -n mode only the valid directory is looked at with
the exception of the ta directory holding the trust anchors.
A file can now be in two different locations so adjust all the code paths
that open files to check both locations.
One nice side-effect of this is that the RRDP handling in the main process
got simplified. There is no longer the need for temporary RRDP directories.
OK tb@

Move some functions around to reduce diff with an upcoming change.
OK tb@

Alter valid_filehash() to take a file descriptor instead of a path.
This is needed so that callers can allow a file to be in multiple
locations. Also move mft_check() from mft.c to parser.c.
OK tb@

Implement a RRDP_CLEAR message that instructs the parent to cleanup
the rrdp directory. This is used before a snapshot download to ensure
that the snapshot is applied to a clean repo.
Similar cleanup happens if the transfer fails. In that case remove the
temp directory contents only.
This uses a new function remove_contents() to remove everything below
a base directory (a bit like rm -r X/*).
OK tb@

Add the same entityq_flush() logic in ta_lookup() as it is done in
repo_lookup(). This fixes -n mode.
OK tb@

Change the way the parser accesses files. It now builds the file path
based on information from the repository, a local path and the filename.
This simplifies some code both in the main process and the parser.
For this to work repositories are passed to the parser before any other
entity of this repository is passed. Struct entity is extended to include
the repoid and the path along the file(name).
Input and OK tb@ & job@

Fix -n mode by setting repository states to REPO_DONE instead of the
more correct REPO_FAILED. The problem is that with REPO_FAILED the repo
no longer matches a subsequent rrdp_get lookup and as a result the repo
uses rsync which wrong in this case.
OK benno@ tb@

Merge multiple similar code bits together into repo_done(). Which currently
finds all repos that point to the passed backend and flushes their queues.
Also make sure that REPO_FAILED is set in cases where no sync happened and
also fall back from rrdp to rsync in any case of failure (including errors
merging the rrdp repo itself).
OK tb@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Simplify code a bit. There is only one TA per TAL and so only one
ta_lookup(). Implementing the talrepocnt limiter there makes little
sense and gains us nothing.
OK job@

Use one common function to build the base directory path for repositories.
OK benno@

Replace MAX_REPO_TIMEOUT with repo_timeout, which is set to 1/4th of
timeout, or if timeout is disabled set to 24 hours.

OK claudio@

Check for possible negative diff timeout. Clamp it to 0 in that case.
poll with negative timeout noticed by job@
OK job@

Adjust how the repository count limit works. Instead of failing hard just
fall back to a possible cache and try to validate what is available.
This still limits the number of repositories fetched but allows valid
repositories to finish with the available data.
OK job@

Limit the number of publication points under a given TAL.
Introduce an additional timeout for each publication point.

The limits are large enough to accomodate normal operating levels.

With and OK benno@ job@ tb@ beck@ deraadt@

Instead of creating a struct repo for each unique caRepository URI
use the rsync URI (a base version of caRepository) and the notify URI
to identify repositories. If both rsync URI and notify URI are the same
then the repo is the same. The notify URI is optional and can be NULL
so the lookup needs to be a bit careful.
This reduces the number of struct repos from 26k to around 50.
OK tb@

branches: 1.9.4;
Rework the logic for fetching TA files a bit. Move the logic that checks
if all possible sources have been used to ta_fetch() this simplifies the
logic in the rsync and http done cases.
Additionally when -R is used, skip https URI and only connect via rsync.
If the TAL file has no rsync URI then the system will fall back to the
cache and not try https instead.
This makes -R a only use rsync flag.
With and OK job@

Change errors for file manipulations from err(3) to warn(3) and adjust the
surrounding code to handle these errors gracefully. When rpki-client runs
out of space it will keep on running including the cleanup at the end.
This way the temporary and old files are removed hopfully giving back some
space. This affects mainly RRDP and the initial fetch of TA files.
OK job@

Similar to rsync repo delay deletes for rrdp repos to after processing
all files. Now RRDP has a weak sync mechanism and therefor not all
unreferenced files can be removed, instead look at the list of removes
which were part of the delta processing and remove those files.
Warn and keep the file if a delta specifies to remove a file which is
actually still referenced.
OK benno@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.5.2;
Before adding a file to the temporary rrdp repo remove it from the
deleted filepath set. A file can only be in one set (deleted or added)
but not on both.
OK benno@

When merging a repo even files to delete can be part of the temporary
work dir. So unlink can return an ENOENT error for the main repo. In
which case the temp dir should be tried.
Refactor this code a bit since there is no way rrdp_filename() should
fail in this part of the code.
OK tb@

fix typo + some whitespace

spelling

Initial commit of RRDP (The RPKI Repository Delta Protocol - RFC8182) support
in rpki-client. For now it is off by default.

All XML processing is done in its own process with minimal pledge rights.
It uses the already present https process to fetch the xml files and uses
the master porcess to handle the file IO into the repositories.
RRDP data is stored in the cache under ./rrdp/ and the first directory
is the SHA256 hash of the notify URI.

Fetching snapshots and deltas works to bring the cache up to date.
If something goes wrong rpki-client will fall back to rsync.

RRDP was implemented by Nils Fisher and integrated into rpki-client by myself.
"Time to get it in" deraadt@

Replace protocol literal strings and strlen() calls with defined constants

OK tb@ claudio@

Track the number of new files moving from 'staging' to 'validated cache'

The OpenMetrics output shows per-repository counters for new files
added, the main process and JSON output emit the sum of all new files.

OK claudio@

Add support for RPKI Signed Prefix Lists

Signed Prefix List are a CMS protected content type for use with the
RPKI to carry the complete list of prefixes which an Autonomous System
may originate to all or any of its routing peers. The validation of a
Signed Prefix List confirms that the holder of the listed ASN produced
the object, and that this list is a current, accurate and complete
description of address prefixes that may be announced into the routing
system originated by this AS.

https://datatracker.ietf.org/doc/html/draft-ietf-sidrops-rpki-prefixlist

with and OK claudio@ tb@

Refactor handling of stale manifests

No need to hoist a staleness indicator through the whole process and
count it explicitly.

OK tb@

Do not set O_EXCL on open() when a published file is on the withdraw list.

If during the process of applying RRDP deltas a object is published then
withdrawn and later published again the current code fails. This is because
published elements use O_EXCL in open() but in the case above the previous
file is still around since withdraws are delayed.

Problem noticed by job@
OK tb@

rpki-client: fix vap_pas stats

A small mistake in a diff broke the counters. Make them AFI agnostic and
adjust ometric output.

guidance & ok claudio

In rrdp_session_save() stop the loop over deltas once MAX_RRDP_DELTAS
have been processed.
With and OK tb@

Improve detection of RRDP session desynchronization

According to RFC 8182, a given session_id and serial number represent an
immutable record of the state of the Repository Server at a certain
point in time.

Add a check to the RRDP notification file processing to compare whether
the delta hashes associated to previously seen serials are different in
newly fetched notification files. Fall back to a snapshot if a difference
is detected, because such a mutation is a strong desynchronization
indicator.

Idea from Ties de Kock (RIPE NCC).
Based on a diff by job@
With and OK job@ tb@

Fixup file modification timestamps to optimize failover from RRDP to RSYNC

In the RSYNC protocol a file's last modification time and its size are
used to determine whether sending a (partial) copy over the wire is needed.
Previously, when RRDP data structures are serialized to disk, the mtime of
files in DIR_VALID ended up being UTIME_NOW.

Thus, the mtimes of files obtained through RRDP will never match the mtimes
of the same files available through RSYNC - causing each and every file to
be added to the file transfer list.

Instead, use the internal timestamps of RPKI files as the last modified
timestamp. Specifically, for Signed Objects (ROAs, MFTs, GBRs, TAKs, ASPAs)
the CMS signing-time, for .cer files the X.509 notBefore, and for .crl files
the CRL lastUpdate. This results in a surprising optimization for the number
files which have to be transfered.

OK claudio@

Fix repo_cleanup_entry() state machine so that the repository lookups
are done when the full repo path is available. Without this all repo
lookups returned NULL and the code did not work as intended.
OK tb@

RRDP snapshots should encode publish elements only once. If encountered
fail the transfer and fall back to rsync.

When more than one publish element for the same file exist the RP does
not know which one to choose. Lets fail the RRDP transfer in this case
and fall back to rsync. CA that publish a file more than once are buggy
and need to be fixed.
OK job@ tb@

Improve accounting by tracking things by repo and tal.
This fixes some wrong accounting for repositories that are referenced
from more than one TAL. It changes the ometric lable output a little bit
since there are repository metrics that no longer include the 'name' label.
OK tb@

Add the protocol used to sync the repository to the open-metric output.
OK tb@

Instead of storing the short TAL name as repouri use the first TAL URI
for that. This is more in line with the other uses of repouri.
OK tb@

spelling fixes; from paul tagliamonte
any parts of his diff not taken are noted on tech

Rework statistic collection to be per repository and add metric output option

Many statistic values are now accounted by repository via repo_stat_inc()
At end of the run sum_stats() accumulates these stats per TAL and globally.
The new output file metrics is written when the -m output flag is specified.
The metrics file is written in OpenMetrics format (with a few tweaks to
allow node_exporter to parse the file as well). The ometric code is a copy
from bgpctl(8) and should be kept in sync.
OK tb@

Introduce a deadline timer that aborts all repository syncs.
With this rpki-client has a chance to still finish and produce an output
even when a CA is excessivly slow and holds back progress.
With and OK benno@ tb@ and job@

Use the abort commands when a repo timeout happens. This is cleaner
then just failing the repo fetch but leaving the backends running.
OK tb@

Move mkpath logic after checking for 'noop' to prevent creation of directories in -n mode

OK claudio@

In RRDP mode it is possible that a file in the validated cache dir is
no longer valid while the newer file in the .rrdp directory is not (yet)
valid. In this case do not copy the old file over the newer file.
RRDP has no method to identiy inconsistent repos (unlike rsync) so if
the wrong file is put in .rrdp it is not noticed appart from the
verification process failing for that file.
OK tb@ job@

infromation -> information

More KNF and whitespace fixes.

Fix various annoying whitespace errors.

Adjust cache cleanup to the deficits of RRDP. Unlike rysnc, RRDP has no
method to sync the cache. It just depends on all deltas to work but has
no method to check if the result is consistent with the source.
Because of this do not unlink files which were fetched via RRDP. Instead
move them back to their .rrdp directory and hope the will be cleaned up.
This should help to keep the cache coherent in some edge cases.
OK tb@

Draw a bit more attention to transport switches between RRDP, RSYNC, and cache

OK claudio@ tb@

When a RRDP fetch failed not only clear the filesystem but also flush
the to delete list stored this RRDP node.
Noticed by Job on console.rpki-client.org with the help of idnic.net
OK tb@

Do not skip .rrdp cache cleanup if rrdp is off (option -R). The sync
via rsync alters the cache and rrdp can only recover by downloading
a snapshot. By doing the cleanup here it will make sure this happens.
Noticed by job@
OK tb@

Change the repository layout by removing the valid directory. Everything
that was in that directory is now in the chachedir root. The rsync and
rrdp directories are now .rsync/ and .rrdp/. The ta/ directory still
remains because TAs are special.
Idea and most of the diff from job@
OK tb@ and benno@

When rename fails show the source filename and not the destination.
The error should be more helpful thisway.
OK tb@

Handle EINTR the same way in all poll loops. In all cases restart the
poll loop. In the main process move the timeout handling for repositories
into a single function that does the timeouts and the calculation of the
timeout in one go.
OK tb@

Introduce a validated cache which holds all the files that have
successfully been verified by rpki-client.

With this the rsync and rrdp directories are more of a temporary storage
location. New files are downloaded there and then moved to the valid
directory at the end. In -n mode only the valid directory is looked at with
the exception of the ta directory holding the trust anchors.
A file can now be in two different locations so adjust all the code paths
that open files to check both locations.
One nice side-effect of this is that the RRDP handling in the main process
got simplified. There is no longer the need for temporary RRDP directories.
OK tb@

Move some functions around to reduce diff with an upcoming change.
OK tb@

Alter valid_filehash() to take a file descriptor instead of a path.
This is needed so that callers can allow a file to be in multiple
locations. Also move mft_check() from mft.c to parser.c.
OK tb@

Implement a RRDP_CLEAR message that instructs the parent to cleanup
the rrdp directory. This is used before a snapshot download to ensure
that the snapshot is applied to a clean repo.
Similar cleanup happens if the transfer fails. In that case remove the
temp directory contents only.
This uses a new function remove_contents() to remove everything below
a base directory (a bit like rm -r X/*).
OK tb@

Add the same entityq_flush() logic in ta_lookup() as it is done in
repo_lookup(). This fixes -n mode.
OK tb@

Change the way the parser accesses files. It now builds the file path
based on information from the repository, a local path and the filename.
This simplifies some code both in the main process and the parser.
For this to work repositories are passed to the parser before any other
entity of this repository is passed. Struct entity is extended to include
the repoid and the path along the file(name).
Input and OK tb@ & job@

Fix -n mode by setting repository states to REPO_DONE instead of the
more correct REPO_FAILED. The problem is that with REPO_FAILED the repo
no longer matches a subsequent rrdp_get lookup and as a result the repo
uses rsync which wrong in this case.
OK benno@ tb@

Merge multiple similar code bits together into repo_done(). Which currently
finds all repos that point to the passed backend and flushes their queues.
Also make sure that REPO_FAILED is set in cases where no sync happened and
also fall back from rrdp to rsync in any case of failure (including errors
merging the rrdp repo itself).
OK tb@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Simplify code a bit. There is only one TA per TAL and so only one
ta_lookup(). Implementing the talrepocnt limiter there makes little
sense and gains us nothing.
OK job@

Use one common function to build the base directory path for repositories.
OK benno@

Replace MAX_REPO_TIMEOUT with repo_timeout, which is set to 1/4th of
timeout, or if timeout is disabled set to 24 hours.

OK claudio@

Check for possible negative diff timeout. Clamp it to 0 in that case.
poll with negative timeout noticed by job@
OK job@

Adjust how the repository count limit works. Instead of failing hard just
fall back to a possible cache and try to validate what is available.
This still limits the number of repositories fetched but allows valid
repositories to finish with the available data.
OK job@

Limit the number of publication points under a given TAL.
Introduce an additional timeout for each publication point.

The limits are large enough to accomodate normal operating levels.

With and OK benno@ job@ tb@ beck@ deraadt@

Instead of creating a struct repo for each unique caRepository URI
use the rsync URI (a base version of caRepository) and the notify URI
to identify repositories. If both rsync URI and notify URI are the same
then the repo is the same. The notify URI is optional and can be NULL
so the lookup needs to be a bit careful.
This reduces the number of struct repos from 26k to around 50.
OK tb@

branches: 1.9.4;
Rework the logic for fetching TA files a bit. Move the logic that checks
if all possible sources have been used to ta_fetch() this simplifies the
logic in the rsync and http done cases.
Additionally when -R is used, skip https URI and only connect via rsync.
If the TAL file has no rsync URI then the system will fall back to the
cache and not try https instead.
This makes -R a only use rsync flag.
With and OK job@

Change errors for file manipulations from err(3) to warn(3) and adjust the
surrounding code to handle these errors gracefully. When rpki-client runs
out of space it will keep on running including the cleanup at the end.
This way the temporary and old files are removed hopfully giving back some
space. This affects mainly RRDP and the initial fetch of TA files.
OK job@

Similar to rsync repo delay deletes for rrdp repos to after processing
all files. Now RRDP has a weak sync mechanism and therefor not all
unreferenced files can be removed, instead look at the list of removes
which were part of the delta processing and remove those files.
Warn and keep the file if a delta specifies to remove a file which is
actually still referenced.
OK benno@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.5.2;
Before adding a file to the temporary rrdp repo remove it from the
deleted filepath set. A file can only be in one set (deleted or added)
but not on both.
OK benno@

When merging a repo even files to delete can be part of the temporary
work dir. So unlink can return an ENOENT error for the main repo. In
which case the temp dir should be tried.
Refactor this code a bit since there is no way rrdp_filename() should
fail in this part of the code.
OK tb@

fix typo + some whitespace

spelling

Initial commit of RRDP (The RPKI Repository Delta Protocol - RFC8182) support
in rpki-client. For now it is off by default.

All XML processing is done in its own process with minimal pledge rights.
It uses the already present https process to fetch the xml files and uses
the master porcess to handle the file IO into the repositories.
RRDP data is stored in the cache under ./rrdp/ and the first directory
is the SHA256 hash of the notify URI.

Fetching snapshots and deltas works to bring the cache up to date.
If something goes wrong rpki-client will fall back to rsync.

RRDP was implemented by Nils Fisher and integrated into rpki-client by myself.
"Time to get it in" deraadt@

Track the number of new files moving from 'staging' to 'validated cache'

The OpenMetrics output shows per-repository counters for new files
added, the main process and JSON output emit the sum of all new files.

OK claudio@

Add support for RPKI Signed Prefix Lists

Signed Prefix List are a CMS protected content type for use with the
RPKI to carry the complete list of prefixes which an Autonomous System
may originate to all or any of its routing peers. The validation of a
Signed Prefix List confirms that the holder of the listed ASN produced
the object, and that this list is a current, accurate and complete
description of address prefixes that may be announced into the routing
system originated by this AS.

https://datatracker.ietf.org/doc/html/draft-ietf-sidrops-rpki-prefixlist

with and OK claudio@ tb@

Refactor handling of stale manifests

No need to hoist a staleness indicator through the whole process and
count it explicitly.

OK tb@

Do not set O_EXCL on open() when a published file is on the withdraw list.

If during the process of applying RRDP deltas a object is published then
withdrawn and later published again the current code fails. This is because
published elements use O_EXCL in open() but in the case above the previous
file is still around since withdraws are delayed.

Problem noticed by job@
OK tb@

rpki-client: fix vap_pas stats

A small mistake in a diff broke the counters. Make them AFI agnostic and
adjust ometric output.

guidance & ok claudio

In rrdp_session_save() stop the loop over deltas once MAX_RRDP_DELTAS
have been processed.
With and OK tb@

Improve detection of RRDP session desynchronization

According to RFC 8182, a given session_id and serial number represent an
immutable record of the state of the Repository Server at a certain
point in time.

Add a check to the RRDP notification file processing to compare whether
the delta hashes associated to previously seen serials are different in
newly fetched notification files. Fall back to a snapshot if a difference
is detected, because such a mutation is a strong desynchronization
indicator.

Idea from Ties de Kock (RIPE NCC).
Based on a diff by job@
With and OK job@ tb@

Fixup file modification timestamps to optimize failover from RRDP to RSYNC

In the RSYNC protocol a file's last modification time and its size are
used to determine whether sending a (partial) copy over the wire is needed.
Previously, when RRDP data structures are serialized to disk, the mtime of
files in DIR_VALID ended up being UTIME_NOW.

Thus, the mtimes of files obtained through RRDP will never match the mtimes
of the same files available through RSYNC - causing each and every file to
be added to the file transfer list.

Instead, use the internal timestamps of RPKI files as the last modified
timestamp. Specifically, for Signed Objects (ROAs, MFTs, GBRs, TAKs, ASPAs)
the CMS signing-time, for .cer files the X.509 notBefore, and for .crl files
the CRL lastUpdate. This results in a surprising optimization for the number
files which have to be transfered.

OK claudio@

Fix repo_cleanup_entry() state machine so that the repository lookups
are done when the full repo path is available. Without this all repo
lookups returned NULL and the code did not work as intended.
OK tb@

RRDP snapshots should encode publish elements only once. If encountered
fail the transfer and fall back to rsync.

When more than one publish element for the same file exist the RP does
not know which one to choose. Lets fail the RRDP transfer in this case
and fall back to rsync. CA that publish a file more than once are buggy
and need to be fixed.
OK job@ tb@

Improve accounting by tracking things by repo and tal.
This fixes some wrong accounting for repositories that are referenced
from more than one TAL. It changes the ometric lable output a little bit
since there are repository metrics that no longer include the 'name' label.
OK tb@

Add the protocol used to sync the repository to the open-metric output.
OK tb@

Instead of storing the short TAL name as repouri use the first TAL URI
for that. This is more in line with the other uses of repouri.
OK tb@

spelling fixes; from paul tagliamonte
any parts of his diff not taken are noted on tech

Rework statistic collection to be per repository and add metric output option

Many statistic values are now accounted by repository via repo_stat_inc()
At end of the run sum_stats() accumulates these stats per TAL and globally.
The new output file metrics is written when the -m output flag is specified.
The metrics file is written in OpenMetrics format (with a few tweaks to
allow node_exporter to parse the file as well). The ometric code is a copy
from bgpctl(8) and should be kept in sync.
OK tb@

Introduce a deadline timer that aborts all repository syncs.
With this rpki-client has a chance to still finish and produce an output
even when a CA is excessivly slow and holds back progress.
With and OK benno@ tb@ and job@

Use the abort commands when a repo timeout happens. This is cleaner
then just failing the repo fetch but leaving the backends running.
OK tb@

Move mkpath logic after checking for 'noop' to prevent creation of directories in -n mode

OK claudio@

In RRDP mode it is possible that a file in the validated cache dir is
no longer valid while the newer file in the .rrdp directory is not (yet)
valid. In this case do not copy the old file over the newer file.
RRDP has no method to identiy inconsistent repos (unlike rsync) so if
the wrong file is put in .rrdp it is not noticed appart from the
verification process failing for that file.
OK tb@ job@

infromation -> information

More KNF and whitespace fixes.

Fix various annoying whitespace errors.

Adjust cache cleanup to the deficits of RRDP. Unlike rysnc, RRDP has no
method to sync the cache. It just depends on all deltas to work but has
no method to check if the result is consistent with the source.
Because of this do not unlink files which were fetched via RRDP. Instead
move them back to their .rrdp directory and hope the will be cleaned up.
This should help to keep the cache coherent in some edge cases.
OK tb@

Draw a bit more attention to transport switches between RRDP, RSYNC, and cache

OK claudio@ tb@

When a RRDP fetch failed not only clear the filesystem but also flush
the to delete list stored this RRDP node.
Noticed by Job on console.rpki-client.org with the help of idnic.net
OK tb@

Do not skip .rrdp cache cleanup if rrdp is off (option -R). The sync
via rsync alters the cache and rrdp can only recover by downloading
a snapshot. By doing the cleanup here it will make sure this happens.
Noticed by job@
OK tb@

Change the repository layout by removing the valid directory. Everything
that was in that directory is now in the chachedir root. The rsync and
rrdp directories are now .rsync/ and .rrdp/. The ta/ directory still
remains because TAs are special.
Idea and most of the diff from job@
OK tb@ and benno@

When rename fails show the source filename and not the destination.
The error should be more helpful thisway.
OK tb@

Handle EINTR the same way in all poll loops. In all cases restart the
poll loop. In the main process move the timeout handling for repositories
into a single function that does the timeouts and the calculation of the
timeout in one go.
OK tb@

Introduce a validated cache which holds all the files that have
successfully been verified by rpki-client.

With this the rsync and rrdp directories are more of a temporary storage
location. New files are downloaded there and then moved to the valid
directory at the end. In -n mode only the valid directory is looked at with
the exception of the ta directory holding the trust anchors.
A file can now be in two different locations so adjust all the code paths
that open files to check both locations.
One nice side-effect of this is that the RRDP handling in the main process
got simplified. There is no longer the need for temporary RRDP directories.
OK tb@

Move some functions around to reduce diff with an upcoming change.
OK tb@

Alter valid_filehash() to take a file descriptor instead of a path.
This is needed so that callers can allow a file to be in multiple
locations. Also move mft_check() from mft.c to parser.c.
OK tb@

Implement a RRDP_CLEAR message that instructs the parent to cleanup
the rrdp directory. This is used before a snapshot download to ensure
that the snapshot is applied to a clean repo.
Similar cleanup happens if the transfer fails. In that case remove the
temp directory contents only.
This uses a new function remove_contents() to remove everything below
a base directory (a bit like rm -r X/*).
OK tb@

Add the same entityq_flush() logic in ta_lookup() as it is done in
repo_lookup(). This fixes -n mode.
OK tb@

Change the way the parser accesses files. It now builds the file path
based on information from the repository, a local path and the filename.
This simplifies some code both in the main process and the parser.
For this to work repositories are passed to the parser before any other
entity of this repository is passed. Struct entity is extended to include
the repoid and the path along the file(name).
Input and OK tb@ & job@

Fix -n mode by setting repository states to REPO_DONE instead of the
more correct REPO_FAILED. The problem is that with REPO_FAILED the repo
no longer matches a subsequent rrdp_get lookup and as a result the repo
uses rsync which wrong in this case.
OK benno@ tb@

Merge multiple similar code bits together into repo_done(). Which currently
finds all repos that point to the passed backend and flushes their queues.
Also make sure that REPO_FAILED is set in cases where no sync happened and
also fall back from rrdp to rsync in any case of failure (including errors
merging the rrdp repo itself).
OK tb@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Simplify code a bit. There is only one TA per TAL and so only one
ta_lookup(). Implementing the talrepocnt limiter there makes little
sense and gains us nothing.
OK job@

Use one common function to build the base directory path for repositories.
OK benno@

Replace MAX_REPO_TIMEOUT with repo_timeout, which is set to 1/4th of
timeout, or if timeout is disabled set to 24 hours.

OK claudio@

Check for possible negative diff timeout. Clamp it to 0 in that case.
poll with negative timeout noticed by job@
OK job@

Adjust how the repository count limit works. Instead of failing hard just
fall back to a possible cache and try to validate what is available.
This still limits the number of repositories fetched but allows valid
repositories to finish with the available data.
OK job@

Limit the number of publication points under a given TAL.
Introduce an additional timeout for each publication point.

The limits are large enough to accomodate normal operating levels.

With and OK benno@ job@ tb@ beck@ deraadt@

Instead of creating a struct repo for each unique caRepository URI
use the rsync URI (a base version of caRepository) and the notify URI
to identify repositories. If both rsync URI and notify URI are the same
then the repo is the same. The notify URI is optional and can be NULL
so the lookup needs to be a bit careful.
This reduces the number of struct repos from 26k to around 50.
OK tb@

branches: 1.9.4;
Rework the logic for fetching TA files a bit. Move the logic that checks
if all possible sources have been used to ta_fetch() this simplifies the
logic in the rsync and http done cases.
Additionally when -R is used, skip https URI and only connect via rsync.
If the TAL file has no rsync URI then the system will fall back to the
cache and not try https instead.
This makes -R a only use rsync flag.
With and OK job@

Change errors for file manipulations from err(3) to warn(3) and adjust the
surrounding code to handle these errors gracefully. When rpki-client runs
out of space it will keep on running including the cleanup at the end.
This way the temporary and old files are removed hopfully giving back some
space. This affects mainly RRDP and the initial fetch of TA files.
OK job@

Similar to rsync repo delay deletes for rrdp repos to after processing
all files. Now RRDP has a weak sync mechanism and therefor not all
unreferenced files can be removed, instead look at the list of removes
which were part of the delta processing and remove those files.
Warn and keep the file if a delta specifies to remove a file which is
actually still referenced.
OK benno@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.5.2;
Before adding a file to the temporary rrdp repo remove it from the
deleted filepath set. A file can only be in one set (deleted or added)
but not on both.
OK benno@