Improve x509_get_purpose()

Instead of only differentiating between CA and BGPsec Router certs,
make it recognize TA and EE certs as well. TAs and CAs have the cA
boolean in the basic constraints, while EE and BGPsec router certs
do not.

TAs are self-signed, CAs not self-issued, all other certs with the
cA boolean are invalid. EE certs do not have an extended key usage
and BGPsec certs contain the id-kp-bgpsec-router OID.

Handle the new purposes where needed.
ok job

Download new TA files into a temporary place (.ta/) so that the parser
can decide which of the two files to use.
With and OK tb@

Instead of tracking certificates by SKI track them by an internal identifier.

The certificate SKI is not strictly unique so using it as a unique id is
problematic. It is also not really needed to do that since in theory we
already know the path (but this got lost in the privsep communication).
So add a cert id and pass this id back and forth between main process and
the parser. With this id we can lookup the authentication chain in the
parser and this even works with multiple paths to the same resource.
Since we no longer lookup by SKI the valid_aki_ski function is replaced
by find_issuer() which does the lookup by certid.

The loop protection is now extended to allow each TAL to reach each file
once but still triggers if a file is reaccessed by the tree of a TAL.

In filemode the lookup now uses an AIA uri based lookup tree. Again this
replaces the SKI based lookups from before.

Done together with tb@
OK tb@ job@

Fix capping of VAPs

The previous approach introduced a cap, but it might not always be hit as
intended (I missed this on review). Fix this to check the cap after merging
an ASPA into an already existing VAP. Also free the list of providers since
nothing should be looking at it anymore.

Count VAPs that hit the limit with a new overflowed counter. There are
still a few aspects of the accounting that probably aren't entirely right.
This will be fixed at another point. It's just statistics after all.

with/ok claudio, ok job

Don't emit Validated ASPAs for Customer ASIDs with more than MAX_ASPA_PROVIDERS

The number of providers in a single ASPA object already was limited to
MAX_ASPA_PROVIDERS, now also impose a limit on the total number of providers
across multiple ASPA objects. If the MAX_ASPA_PROVIDERS limit is hit, omit
the Customer ASID's entry from OpenBGPD and JSON output.

OK tb@

Replace protocol literal strings and strlen() calls with defined constants

OK tb@ claudio@

Lipstick on a pig: avoid comparing signed and unsigned

OK tb@ claudio@

Add -x to opt into experimental file formats

Instead of burning one letter for each new file format (sidrops is known
to crank out new things faster than a normal person can read), use -x to
opt into parsing and processing file formats that aren't yet considered
stable. This is currently only the Signed Prefix List. While a repetition
of the ASPA debacle, this code hasn't yet seen enough stress testing to be
enabled by default.

ok claudio job

Track the number of new files moving from 'staging' to 'validated cache'

The OpenMetrics output shows per-repository counters for new files
added, the main process and JSON output emit the sum of all new files.

OK claudio@

Add support for RPKI Signed Prefix Lists

Signed Prefix List are a CMS protected content type for use with the
RPKI to carry the complete list of prefixes which an Autonomous System
may originate to all or any of its routing peers. The validation of a
Signed Prefix List confirms that the holder of the listed ASN produced
the object, and that this list is a current, accurate and complete
description of address prefixes that may be announced into the routing
system originated by this AS.

https://datatracker.ietf.org/doc/html/draft-ietf-sidrops-rpki-prefixlist

with and OK claudio@ tb@

Group logx() getmonotime() and get_current_time() together

Fix their indent in extern.h, move the X509_TIME_* macros to main.c since
they aren't (and can't really be) used elsewhere, document the meaning of
the magic numbers. Also move get_current_time() out of the middle of entity
handlers.

ok claudio job

fix whitespace

Refactor handling of stale manifests

No need to hoist a staleness indicator through the whole process and
count it explicitly.

OK tb@

Allow imposing constraints on RPKI trust anchors

The ability to constrain a RPKI Trust Anchor's effective signing
authority to a limited set of Internet Number Resources allows
Relying Parties to enjoy the potential benefits of assuming trust,
within a bounded scope.

Some examples: ARIN does not support inter-RIR IPv6 transfers, so
it wouldn't make any sense to see a ROA subordinate to ARIN's trust
anchor covering RIPE-managed IPv6 space. Conversely, it wouldn't
make sense to observe a ROA covering ARIN-managed IPv6 space under
APNIC's, LACNIC's, or RIPE's trust anchor - even if a derived trust
arc (a cryptographically valid certificate path) existed. Along these
same lines, AFRINIC doesn't support inter-RIR transfers of any kind,
and none of the RIRs have authority over private resources like
10.0.0.0/8 and 2001:db8::/32.

For more background see:
https://datatracker.ietf.org/doc/draft-snijders-constraining-rpki-trust-anchors/
https://mailman.nanog.org/pipermail/nanog/2023-September/223354.html

With and OK tb@, OK claudio@

Constify argument to entity_write_repo()

OK tb@

Fix comments

OK tb@

rpki-client: fix vap_pas stats

A small mistake in a diff broke the counters. Make them AFI agnostic and
adjust ometric output.

guidance & ok claudio

Improve detection of RRDP session desynchronization

According to RFC 8182, a given session_id and serial number represent an
immutable record of the state of the Repository Server at a certain
point in time.

Add a check to the RRDP notification file processing to compare whether
the delta hashes associated to previously seen serials are different in
newly fetched notification files. Fall back to a snapshot if a difference
is detected, because such a mutation is a strong desynchronization
indicator.

Idea from Ties de Kock (RIPE NCC).
Based on a diff by job@
With and OK job@ tb@

Use new ibuf API in rpki-client.

Use ibuf_fd_get/set/avail in the file descriptor passing code.
Switch some ibuf_seek() calls to ibuf_set() and use ibuf_add_zero() instead
of ibuf_reserve().
OK tb@ job@

Fixup file modification timestamps to optimize failover from RRDP to RSYNC

In the RSYNC protocol a file's last modification time and its size are
used to determine whether sending a (partial) copy over the wire is needed.
Previously, when RRDP data structures are serialized to disk, the mtime of
files in DIR_VALID ended up being UTIME_NOW.

Thus, the mtimes of files obtained through RRDP will never match the mtimes
of the same files available through RSYNC - causing each and every file to
be added to the file transfer list.

Instead, use the internal timestamps of RPKI files as the last modified
timestamp. Specifically, for Signed Objects (ROAs, MFTs, GBRs, TAKs, ASPAs)
the CMS signing-time, for .cer files the X.509 notBefore, and for .crl files
the CRL lastUpdate. This results in a surprising optimization for the number
files which have to be transfered.

OK claudio@

Revert commitid ANSBO0rBvIUtTi45:
Make rpki-client choose the verification time of the time it is invoked
rather than always getting the current system time for every certificate
verification. This will result in output that is not variable on run-time.

Using the time of invocation does not work well with fast publishing CAs. It
can take a few minutes to reach a repo and that CA may have issued certificates
that are not yet valid if that startup time of rpki-client is used to validate.

This still keeps the -P option to specify a fixed validation time.
OK beck@ job@ tb@

Split cleanup into cleanup and repository cleanup and show how many files
are kept / removed in the repository temporary storage.
After a discussion with tb@ and job@

Properly account del_extra_files and add the value to the json header.
OK tb@ (from a larger diff)

Update usage add -P epoch

Make rpki-client choose the verification time of the time it is invoked
rather than always getting the current system time for every certificate
verification. This will result in output that is not variable on run-time.

ok tb@ claudio@

Add a -P option to rpki-client to specify the evaluation time

This is intended to be able to test rpki-client in a reproducable
way without worrying about the system time changing the results

ok claudio@

Improve accounting by tracking things by repo and tal.
This fixes some wrong accounting for repositories that are referenced
from more than one TAL. It changes the ometric lable output a little bit
since there are repository metrics that no longer include the 'name' label.
OK tb@

Check whether products listed on a manifest were issued by the same authority as the manifest itself

OK tb@

When parsing MFT the CRL referenced by the MFT is loaded and verified at
the same time. So in case of a valid crl pass the CRL filename as entity
message to the parent process together with the MFT. This way the MFT and
CRL end up both in the valid cache even if some files in the MFT are missing.
On severe errors (like X.509 verify errors) the CRL is not moved since it
is not considered valid.
With and OK job@, tb@

Add aspa-set to openbgpd config output.

Change the way the validated ASPA tree is built since OpenBGPD config
follows more the ASPA profile and puts the optional AFI to each provider
ASnum instead of duplicated everything into an IPv4 and IPv6 tree.
The JSON output of ASPA is still the same.

The inclusion of the aspa-set can currently be disabled by the -A flag.
OK tb@

Rework the main <-> parser/filemode communication protocol a bit.

Swap repo_id and filename to simplify the code in parser.c. In filemode
both repo_id and filename are ignored.
Additionally do not errx() in case of unknown file types. Instead send back
enough info that the code can move on.

OK tb@

Rework statistic collection to be per repository and add metric output option

Many statistic values are now accounted by repository via repo_stat_inc()
At end of the run sum_stats() accumulates these stats per TAL and globally.
The new output file metrics is written when the -m output flag is specified.
The metrics file is written in OpenMetrics format (with a few tweaks to
allow node_exporter to parse the file as well). The ometric code is a copy
from bgpctl(8) and should be kept in sync.
OK tb@

Switch to struct timespec for collecting stats. This allows to use
clock_gettime(CLOCK_MONOTONIC) for runtime calculation.
OK tb@

Remove unused ctype.h include

OK tb@

Only include stdarg.h, if we call any of va_{start,end}()

OK tb@

Reshuffle case a little bit. No functional change.

Move skiplist to main and merge with shortlist.

With/ok job

zap errant space in usage();

Schedule cleanup of '-r' now useless getopt option

OK tb@

Add shortlist functionality, a compagnion to the skiplist

If the operator specifies the '-H' option once (or more) followed by a FQDN,
the utility will *only* connect to those hosts and skip all others.

OK claudio@ tb@

Add support for draft-ietf-sidrops-signed-tal-12

Add support validation of Signed Objects containing Trust Anchor Keys
(TAKs - aka 'Signed TALs'). Signed TALs provide a mechanism for RIRs
to distribute and sign the next Trust Anchor with the current Trust
Anchor. This might be an improvement over visiting RIR websites and
copy+pasting TAL data by hand.

OK tb@

Move the repo lookup into queue_from_mft()
OK tb@

Introduce a deadline timer that aborts all repository syncs.
With this rpki-client has a chance to still finish and produce an output
even when a CA is excessivly slow and holds back progress.
With and OK benno@ tb@ and job@

Fix over long lines
OK tb@ job@

Use the abort commands when a repo timeout happens. This is cleaner
then just failing the repo fetch but leaving the backends running.
OK tb@

Avoid dead assignment

Add support for ASPA objects (draft-ietf-sidrops-aspa-profile-10)

ASPA objects are published in the RPKI and can be used to detect and
mitigate BGP route leaks. Validated ASPA Payloads are visible through
filemode (-f) and the JSON output format (-j).

With feedback from tb@

OK claudio@ tb@

Simplify load_skiplist()

Instead of copy-pasting valid_uri(), we can call the real thing. That's
also not ideal since we actually check for an FQDN, but it's a bit
simpler.

ok claudio

Tweaks in load_skiplist()

If there's an issue opening the default skip list file other than its
absence (most likely bad permissions), we should not silently ignore
that. Also, use err() instead of errx() to display the error.

Second, linelen, the return value of getline(), is not currently used.
Repurpose it to save the result of strcspn() which calculates the length
of the string we're interested in and use that instead of strlen().

ok claudio

Make PEM printing available through increased verbosity (-vvf)

Suggestion from claudio@

OK tb@

In verbose filemode print details about encapsulated certificates.
Add command line flag to print the certificate in PEM format.

OK tb@

Once all files from argv have been read the main process can unveil to
the cachedir and if output files are written to outputdir. In -f mode
the unveil can be read-only in normal operation rwc is required because
the main process writes the RRDP files and also does the cleanup at the
end of the run.
Input from tb@ and mestre@, OK tb@

Add skiplist option to steer clear of skiplisted hosts

Blocking outbound connections towards RPKI publication servers based
on IP or IPv6 address in external instrumentation like HTTP proxies
or pf(4) rules is somewhat unwieldy. It might be easier for operators
if we offer a mechanism that cuts at the CA cert SIA parsing step.

OK claudio@ tb@

Don't add unknown filetypes to the entity queue. Otherwise we bail out
without good reason. Regression introduced in a recent refactoring found
by job

ok claudio job

Remove a confusing comment that has been stale since e2k19 (r1.26).

There is no need to be quiet by default anymore and also stdout is
perfectly fine. So switch the stats output at the end of the run to
simply use printtf(3) and no longer depend on -v flag.
OK tb@

More KNF and whitespace fixes.

Improve control flow readability by removing 'else if' stanzas

OK tb@ claudio@

rpki-client: bypass timeout in file mode.

ok claudio

Tweak wording of a comment to reflect current reality better.

typo: exceded -> exceed

The filemode code is enough different from the regular parser code that it
makes sense to totally split it out. Duplicate proc_parser_cert_validate()
and proc_parser_root_cert() for now.

The valid_x509() plus the required static functions are moved to validate.c.
The crl_tree code moved into crl.c similar to the auth_tree handling in
cert.c. All the proc functions are now tagged with __attribute(noreturn)
which allows to remove the errx() after them.

OK tb@

Remove an unreachable error message

If timeout != 0 and 0 <= timeout <= 24*60*60, then timeout < 1 is
impossible.

ok

zap extra line break and extra empty line

Change type of talsz and dependent code from size_t to int. Tal ids are
already stored as int and and talsz is the limit for these ids.
OK tb@

Cosmetics, add missing space

Adjust on how CRL and MFT files are verified.

Verify the CRL referenced from the mft against the mft's fileAndHash info.
If the CRL matches then load it and use it to validate this mft. If the
mft validated OK add the now also valid CRL to the auth store for later use.

Before the newest CRL was always selected but that has negative consequences
because it is common practice to revoke the previous MFT's EE cert and with
that the cache is turned useless as soon as a new CRL is used. Also there
was a possibility that the CRL used for validation of the MFT was not the
one later used.

Both RFC6486 and draft-ietf-sidrops-6486bis are unclear about this part
of the validation process. We opted in favor of the chached MFT.

With and OK tb@

Refactor on how the subprocesses are started.

Move the unveil and pledges to the actuall subprocesses and put all the
common code to start these into process_start(). Reduces the lenght of
main() a fair bit.
OK tb@

Adjust cache cleanup to the deficits of RRDP. Unlike rysnc, RRDP has no
method to sync the cache. It just depends on all deltas to work but has
no method to check if the result is consistent with the source.
Because of this do not unlink files which were fetched via RRDP. Instead
move them back to their .rrdp directory and hope the will be cleaned up.
This should help to keep the cache coherent in some edge cases.
OK tb@

rpki-client: remove a stale FIXME

We fixed this back in January when we added rtype_from_mftfile().

ok claudio

Split certificate parsing in two steps. cert_parse_pre() which does
the parse and some checks and cert_parse() or ta_parse() to do the
additional checks for regular certs or TAs.
With this adjust the cert parser in -f mode to identify TAs (by checking
if it is self signed) and adjust the validation in that case. Now -f
should be able to parse and show all object correctly.
With and OK tb@

Improve free inodes check from fs.f_favail > 0 to fs.f_free > 0.

suggested by millert

Only show the 'not enough inodes' warning if the filesystem reports more
than zero free inodes. btrfs always reports zero inodes, so on linux
distros like fedora, users will always see the not enough space warning,
which it is a bit ugly.

fine with claudio

Properly handle .mft files as intended by the RFC. Instead of always
selecting the newest file this opens both the new (from rrdp or rsync)
and old (valid) MFT. It then compares the manifest number and based on
that the 'newer' MFT is selected.
The MFT file and hash check is also changed to always try both locations
and selecting whatever matches up with the hash. The selction is passed
back to the the main process and used later on to open exactly the same
file as was checked against the hash.
The MFT parsing code has been split up into multiple steps so that the
files can be parsed, compared and then fully validated.
In most cases this makes no difference but it prevents replay attacks
using old but still valid files.
With and OK tb@

Allow rsync:// URI as file in -f mode. This makes it easier to explore
rpki repositories by following AIA and manifest URIs.
Also stop checking the the loaded file is not part of the auth tree,
it is possible that this file was loaded before as a dependency.
OK tb@

Adjust code to handle unsupported file types a bit more graceful.
The file still needs to match its hash to make the MFT valid but then
there will only be a warning printed. Parsing of other files from that
MFT are not influenced.
OK tb@

rearrange SYNOPSIS/usage to be a bit clearer;
discussed with and ok claudio

Handle EINTR the same way in all poll loops. In all cases restart the
poll loop. In the main process move the timeout handling for repositories
into a single function that does the timeouts and the calculation of the
timeout in one go.
OK tb@

Allow rpki-client to display more than one file in -f mode.
Change -f to be a mode flag and pass one or multiple files as arguments
to rpki-client. Some extra checks need to be done to not load the same
certificate or CRL multiple times.
Input and OK tb@

Add function to determine the file type from the file name extension

rpki-client uses the same idiom to determine the file type in too many
places. Use one function that determines the appropriate RTYPE from the
file name. Add that type to struct mftfile and use this new member to
simplify queue_add_from_mft*().

input/ok claudio

Avoid out-of-bounds accesses in case the filename is too showrt to
contain an extension of length four.

ok claudio

Add -f to usage in main routine

OK tb@

Implement a filemode where the passed file from the command line is
printed in human readable form and is also verified against the valid
cache of rpki-client.

To validate the file the chain is explored backwards by looking at the
Authority Information Access URI and X509v3 CRL Distribution Point of
the cert. Once the trust anchor is found this chain can be verified.

Feedback and OK job@ tb@

rpki-client: don't leak nfile if no repo is found.

ok claudio

Introduce a validated cache which holds all the files that have
successfully been verified by rpki-client.

With this the rsync and rrdp directories are more of a temporary storage
location. New files are downloaded there and then moved to the valid
directory at the end. In -n mode only the valid directory is looked at with
the exception of the ta directory holding the trust anchors.
A file can now be in two different locations so adjust all the code paths
that open files to check both locations.
One nice side-effect of this is that the RRDP handling in the main process
got simplified. There is no longer the need for temporary RRDP directories.
OK tb@

Implement a RRDP_CLEAR message that instructs the parent to cleanup
the rrdp directory. This is used before a snapshot download to ensure
that the snapshot is applied to a clean repo.
Similar cleanup happens if the transfer fails. In that case remove the
temp directory contents only.
This uses a new function remove_contents() to remove everything below
a base directory (a bit like rm -r X/*).
OK tb@

Implement but don't use code to use rsync's --compare-dest feature.
One gotcha is that the path passed to --compare-dest needs to be relative
to the dst directory. rsync_fixup_dest() will prepend the necessary ../
for that by counting number of '/' in dst.
OK tb@

Change the way the parser accesses files. It now builds the file path
based on information from the repository, a local path and the filename.
This simplifies some code both in the main process and the parser.
For this to work repositories are passed to the parser before any other
entity of this repository is passed. Struct entity is extended to include
the repoid and the path along the file(name).
Input and OK tb@ & job@

Cleanup mft file handling, especially the stale mft bits.
Move staleness check up into mft_parse_econtent() to simplify code.
Remove the big FIXME bits since they are no longer needed. The parent
process will only process MFTs that are not stale.
Cleanup a few other bits mainly unneccessary else if cascades and
use valid_filename() to check if the filename embedded in the mft
fileandhash is sensible.
OK tb@

Pass the filename back from the parser to the parent.
The parent will then add the filename to the filepath tree instead
of doing that in entity_write_req(). In the parser pass the filename
instead of the full entity object to various proc_parser functions.
With this it will be possible to check more then one file in the parser.
OK tb@, earlyer version OK benno@

Cleanup struct entity. Remove the unneeded has_data field, the same
information can be figured out by looking at the data pointer itself.
It is NULL when there is no data and not-NULL if there is data.
OK benno@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Cleanup a few things while reading the code.
OK job@

Initialize the repo_timeout to timeout / 4 at start of main().
OK job@

Replace MAX_REPO_TIMEOUT with repo_timeout, which is set to 1/4th of
timeout, or if timeout is disabled set to 24 hours.

OK claudio@

Fix typo

Limit the number of publication points under a given TAL.
Introduce an additional timeout for each publication point.

The limits are large enough to accomodate normal operating levels.

With and OK benno@ job@ tb@ beck@ deraadt@

On errors related to the pipes to the childs don't error out right away.
Instead exit the main event loop and use waitpid to know why a child
went away. This should make it hopefully more clear when shit hits the fan.
OK tb@ deraadt@

Move and promote getmonotime() to an internal API function.

Instead of passing tal descriptions around just pass a tal id and
use a small lookup table to print the description in the output path.
OK tb@

Further simplify cert and auth handling. Move common code into auth_insert
and skip this distinction between invalid and failed certificates.
The difference between the to is getting more and more blurry.
OK tb@

Further improve load_file(). Always set an errno on errors and make sure
the errno is not clobbered in the error path. Check result from load_file()
and issue a warning and skip the file in queue_add_tal().
OK benno@ beck@

When converting the TAL parser to use load_file and pass the file via
data/datasz element to the parser process I forgot to remove the old
buf argument that abused descr for that. Because of this rpki-client
can crash.
Found by benno@, OK benno@ beck@

Sync output at end of run with output inside the openbgpd output file.
With and OK tb@

Move load_file() to encoding.c so that regress can use the function.

Refactor the tal parsing code to use the same load_file() and buffer
passing as done for the other parsers.
OK job@ tb@

For open/openat, if the flags parameter does not contain O_CREAT, the
3rd (variadic) mode_t parameter is irrelevant. Many developers in the past
have passed mode_t (0, 044, 0644, or such), which might lead future people
to copy this broken idiom, and perhaps even believe this parameter has some
meaning or implication or application. Delete them all.
This comes out of a conversation where tb@ noticed that a strange (but
intentional) pledge behaviour is to always knock-out high-bits from
mode_t on a number of system calls as a safety factor, and his bewilderment
that this appeared to be happening against valid modes (at least visually),
but no sorry, they are all irrelevant junk. They could all be 0xdeafbeef.
ok millert

Add my copyright to some files
OK job@

Rename io_buf_new to io_new_buffer and io_buf_close to io_close_buffer.
With this the write functions are all of the form io_xyz_buffer.
Remove some prototypes of functions I forgot to remove in previous commit.
OK benno@

Finnally move away from blocking reads in rpki-client. The code was a
mish mash of poll, non-blocking writes and blocking reads. Using the
introduced ibuf size header in io_buf_new()/io_buf_close() the read
side can be changed to pull in a full ibuf and only start the un-marshal
once all data has been read.
OK benno@

First step of cleanup in the io land. Introduce io_buf_new() and
io_buf_close(). These function will inject a size of the the buffer
at the beginning of the buffer and will allow the read size to be
switched to proper async IO.
OK benno@

Add support for BGPsec Router Certificates (RFC 8209)

BGPsec router keys are extracted from RPKI certificates and
emitted via the JSON output in base64 encoded form.

OK tb@ claudio@

Remove unused variable

Make style consistent and remove an unused code path

OK tb@

Add rudimentary support for BGPsec router certificates

OK claudio@

branches: 1.145.4;
Properly account entity_queue when the file has already been seen

Thanks to Ben Maddison for helping create a test case

OK claudio@

Flip the -r RRDP switch, which went through community testing for half a year.
Default is now to attempt RRDP first, then try RSYNC.
If problems show up, return to rsync-only behaviour with -R, and file a report.
ok claudio

Do a fstatvfs() call to figure out if the filesystem used for the cache
is large enough for the cache. People like to build VM images with way
too small filesystems and so warning about this situation should help.
With deraadt@ and job@

Use O_DIRECTORY when opening directories. This makes some errors a bit
clearer when using -d.
OK deraadt@

Make sure some variables are initialised since modern gcc warns about them.
Handle rrdppid like we do for all other child processes. The two warnings
in rrdp are probably false positives.
OK tb@

Arm the alarms in all childs so they terminate if the timeout is hit.
Also install the special signal handler around the main poll() loop.
The main process handles SIGALRM so it can log an error to the operator
before exiting.
With and OK deraadt@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.138.2;
Rework the http code to require poll() only when really needed.
Especially tls_read() and tls_write() do not map 1:1 to read() and write()
calls and so assuming that after a tls_read() one needs to poll for more
data is wrong. Instead call tls_read() until it returns a TLS_WANT_*
return.

While here also ignore SIGPIPE. It is almost impossible to properly guard
from SIGPIPE by looking at POLLHUP. Instead just let write() handle it and
return an error.

Putting this in now so this can be tested widely.

Make rpki-client -V output nicer.
OK job@, kn@, deraadt@

move the RPKI_VERSION define into its own version.h file, helps portable.
ok claudio@

Fix two typos in comments

sure deraadt@

Switch logic from != to ==. Makes the code easier to read.

More cleanup of the main function. Use a loop to collect all childs with
waitpid() and print if they exited non-zero or by a signal. Also adjust
the poll hangup case to exit the poll loop instead of erroring out. This
way a crashed child should be reported before exit.
OK job@ tb@

Missing setproctitle("rrdp") noticed by deraadt@

minor KNF found while hunting for a bug

Sort usage: rR -> Rr

Initial commit of RRDP (The RPKI Repository Delta Protocol - RFC8182) support
in rpki-client. For now it is off by default.

All XML processing is done in its own process with minimal pledge rights.
It uses the already present https process to fetch the xml files and uses
the master porcess to handle the file IO into the repositories.
RRDP data is stored in the cache under ./rrdp/ and the first directory
is the SHA256 hash of the notify URI.

Fetching snapshots and deltas works to bring the cache up to date.
If something goes wrong rpki-client will fall back to rsync.

RRDP was implemented by Nils Fisher and integrated into rpki-client by myself.
"Time to get it in" deraadt@

Do a better job at cleaning up. Remove empty directories, scan not only the
known repositories but also clean up no longer known repositories.
With this rpki-client keeps its cache nice and shiny.
With and OK job@

Set the process title for the rpki-client subprocesses so they can be
identified more easily.
OK deraadt@

minor KNF and consistancy
(rpki-client is pretty good code, couple hours of audit and I can only find
a few minor things)

Compare filepath with strcmp() and not strcasecmp(). The URI in RPKI are
case sensitive.
OK tb@

Rewrite poll loop a bit. Put the various message queues into an array
with the same order as the pollfds. This way simple for loops can be
used to setup and handle all cases but POLLIN.
OK tb@

Adjust HTTP client code a bit. Add support for 304 Not Modified responses,
remove handling of 206 Partial Content (the client does not use range
headers). Report the Last-Modified timestamp back to the requestor and
switch OK to a enum value for (FAIL, OK and NOT MODIFIED).
OK tb@

Add an -V option to show the version of rpki-client. For the base version
it will show just OpenBSD while -portable will show the portable version.
OK sthen@, tb@, kn@

Delay chdir to the cache directory to after parsing the tal files.
Using the -t option relative locations can be passed as tal locations
and so the process can not chdir until these files were read.
OK job@

Initialize rsyncpid and httppid in the noop case. It seem gcc is not able
to realize that the pids are initialized if !noop and not accessed if noop.
OK job@ tb@

Allocate the repo structs individually linked by a SLIST instead of using
an array that is reallocated during runtime. With this the entityq can
move back into struct repo.
OK tb@

Revert rev 1.116
The repo structs are reallocated during runtime and so the back pointers to
the head element of the TAILQ get corrupted.
Noticed by tb@

There is no need to revisit a file in the repo, so if the RB_INSERT fails
just drop the entity queue element.
OK benno@ tb@

There is no need for a global enity queue, instead use per repo queues.
Simplifies the code a fair bit.
OK tb@

When adding files from the manifest only the .crl need to be first, after
that the order does not matter so simplify the code and just walk the
list twice. Add the .crl first and then in the second round all other files.
OK job@

Add two error checks for strdup for consistency.

ok claudio

Last commit was done from a non-clean tree. Revert one line that broke
everything. Oups.

Use the same way to error out in out of memory situation.
Just use 'err(1, NULL);' there is no need to include the type of function
that failed since it is still impossible to locate the right call.
Just use a debugger in that case.
OK tb@ deraadt@

an -> and

Replace NOTREACHED with actuall errx() calls. I managed to return from
the proc_xyz() function at least in two occastions and it is not that
trivial to realize what goes wrong.

Implement a https client as a sub-process for rpki-client. This code will
be used to fetch TA certs and later on for RRDP. Kind of unreached for now
since the default TAL files don't include https URI.
The http client is fully asynchronous and can handle multiple downloads at
the same time. This code was based on the http client in ftp(1).
OK tb@, job@

There is no longer the need to mkpathat(), the main process runs now from
the cachedir.

Open both the cachedir and outputdir early and use fchdir(2) to
switch between the two.
OK deraadt@ job@

It is perfectly fine to POLLIN and POLLOUT at the same time. It may
actually make things a bit faster.
OK deraadt@

Adjust pledge() and unveil() calls for proc_rsync() a bit. Since the
mkdir was moved to the main process there is no need for access to .
in the rsync process.
OK job@ deraadt@

Rename mkpath() to mkpathat() since it uses mkdirat() internally.

Convert reallocarray + memset to recallocarray.

ok claudio

Move the mkpath() call from the rsync path to the main process. This allows
to drop cpath from the rsync proc pledge (down to "stdio proc exec").
This will also make work easier with the upcoming http fetcher.
OK tb@

All the code for output handling was moved to output.c some time ago.
Also move the license that came along with it to output.c.

Prevent double free by copying the URI string from struct tal to the repo.

Rework the repository handling. Split the handling of trust anchors into
ta_lookup() while regular repositories (to fetch .mft files) are handled
by repo_lookup(). Also the cache directory layout changed; moving the
trust anchors to ./ta/{tal basename}/ the other repositories end up in
./rsync/
OK tb@

Remove the not so periodic stats from rpki-client. The output is not very
useful and the way it works is a bit strange.
OK job@ tb@

The uri parameter of struct entity is actually a local file path.
Rename field to reduce confusion about what is what.
OK tb@

Define the msgbuf queues globably. Clean up the code since rsyncq and procq
no longer need to be passed all the way down anymore. Shuffle code a bit
to bring it into more order.
OK tb@

Fix an overly long line

Remove openssl includes and fcntl.h neither of those are used in main.c.
Add errno.h since this code makes direct use of errno.

Shuffle code around, move all the bits of proc_parser into parser.c.
OK tb@

Adjust the repository handling a bit. Instead of storing host/module pairs
store repo (rsync URI) and local (the local path to the repository).
Simplifies the the rsync handling a fair bit.
OK deraadt@

A while ago rpki-client was changed to validate the sha256 hashes of
files referenced in MFT files during the validation of the MFT file.
An MFT is only valid if all files are present and their hashes are valid.
Because of this there is no longer the need to check the hash when
these files are parsed later on. Remove these checks for CRT, ROA and CRL
files. Use the presence of the pkey when parsing cert files to decide
if it is a root cert or not.
OK tb@

Simplify the entity queue processing. There is no need to keep the entity
queue element around once it is sent to the parser process. Instead just
keep a count of pending entities and make the parser send back the entity
type instead of the id (since this is the only field needed to figure out
what to do).
OK job@

Start using the ibuf API (ibuf_dynamic, ibuf_add, ibuf_close) for writing
data between processes. This completely decouples the write side.
rpki-client can't really use the imsg framework but it can use the ibuf
bits wich imsg is built on.
OK benno@ job@

Now that a NULL string is marshalled as NULL again we can drop some
extra has_xyz integers to indicate if the following buffer is present
or not. At the same time sprinkle some asserts for strings which must
be not NULL.
OK tb@

Use asprintf with %.*s to construct the path based on the mft file
location and the filename of the referenced file. Use the same method
both in main.c and mft.c which is a fair bit simpler than using calloc
and strlcpy/strlcat to build the string by hand.
OK tb@

Validate ghostbuster records (RFC 6493) but for now do nothing with the
provided vcard payload. This change verifies the certificate of the .gbr
file and makes sure it is valid (like we do for e.g. .roa files).
OK job@

Remove the last users of io_*_write functions that call io_simple_write()
internally. This is a step in direction of more async aware io in rpki-client.
Now everything uses a buffer which is then written.
OK tb@

queue_add_from_cert() and queue_add_from_tal() are both called with properly
validated certs / tals and there is no need to double check if the URIs
point to the right resource again. rtype_resolve() is now unused and can
be removed.
OK tb@

Introduce repo_filename() a function that converts a URI into the local
path to that resource. This will be needed for future RRDP support.

Additionally support more then one TAL URI and select the rsync URI
in that list. Finally queue_add_from_cert() got modified to include
both the rsync URI and the RRDP notify URI (which is still unused).

OK benno@

Rewrite the signal handler to just toggle a flag and then exit asap
in the main loop. This removes a few portability issues.
OK benno@

Make the repotab (rt) a global instead of passing the value around as
an argument. Will help with further changes.
OK benno@

Use correct way to report error when strtonum fails.
Use errx(), remove the extra \n and use the correct argument (-s not -t).

Experienced a situation where (older code) rpki-client+openrsyncd got "stuck"
(for an amusing long time), so that the partnership with cron "~ -ns" could
not make progress ingesting new ROAs. Add a "-s timeout" feature (default 1H,
0 to disable) after which rpki-client gives up the ghost, allowing cron to
perform a fresh try. Log loudly when this occurs.
ok job claudio beck

Remove last two arguments from repo_lookup() since they are unused.

Move the proc_rsync and with that the rsync processing into rsync.c
main.c is too crowded
OK deraadt@

Delete unused strdup().

ok claudio@

Change the noop (-n) behaviour. When no network fetch is done there
is no reason to run the rsync subprocess. Just immediatly make the
repo valid when inserting it to the repo_list.
OK benno@ (some time ago)

Replace warnc() with warnx() + strerror() since the first is less portable.
Should hopefully fix an issue seen by Robert Scheck
OK deraadt@

-Separate out the adding of trust anchors into a separate function than
what is used for adding other certificates.
-Don't call X509_verify on trust anchors for no reason.
-Add basic checks for TA certificate (subject can parse, cert is not expired)
-Add some useful error reporting if the TA cert we fetched is no good.
ok claudio@

Remove -f (force) option.

The -f option existed for some initial debugging work.

Thanks Weerd for review

OK claudio@

Stop using rsync --delete when syncing up with the CA repos. Instead
use the files referenced in the manifests to build up a list of files
to keep and remove anything that is not in the list after doing the
full computation.
OK job@ benno@

Properly wait for exiting rsync processes. Since SIGCHILD is blocked
outside of ppoll() it is possible that multiple processes exited before
waitpid is called. Because of this all childs need to be picked up and
not only one. Fixes a hang seen more often now since there is a lot more
repositories to sync.
OK deraadt@ who came up with a very similar fix also OK job@ benno@

branches: 1.69.4;
rpki-client is only interested in real files. Don't tell rsync to
preserve symbolic links. Instead just ignore them.
OK benno@ deraadt@

Reword verbose log messages to clarify what's happening

OK deraadt@

Place elapsed, user, and system time for processing in the comment headers.
ok job benno claudio

Print statistics as comments at the top of the files which can take
comments.
ok claudio job

Always initialize cachedir and outputdir.
ok job@

improve whitespaces in usage

remove the libcrypto cleanup before exit. Most of the functions are
officially deprecated and even if not exit(3) will take care of freeing
all that memory anyway. Non of the functions do any cleanup beyond simple
memory deallocation.
OK benno@ deraadt@

Replace deprecated ERR_remove_state(0) with ERR_remove_thread_state(NULL);
OK tb@

Split the mft file and hash check from the mft parsing. This makes it easier
to check all files in a mft before failing and also the check is now done
after the embedded cert was checked.
This refactor was triggered because of a bug in mft_parse_econtent().
check_validity() altered rc but later failure code assumed that goto out
is good enough to return an error (rc == -1) but since rc was 1 success
was returned. This bug is now also fixed.
Bug report and OK job@

Better warning message. Show the filename of the CRL instead of the constant
function name. Also AKI (authority key identifier) should be capitalized
and use duplicated instead of dup. There was a case where this error got hit
so better make it pretty.

Change default table name for BIRD to something that is more easily identifable as user provided

Simplify and unify wording for the -I sourceaddr option in various places.
This is somewhat related to the "-b bind_addr" option some programs have,
which should get some cleanup also...
input florian claudio jmc

When rsync exits non zero because of network issues or because the
provided URI is bogous, rpki-client should try to work with the cache
it has and not exit with an error. This should help when URI to localhost
are published which happened at least 2 times already.
Agreed by deraadt@ and benno@

add -d to usage();

Don't hardcode the cache directory for rpki-client. If started as root
rpki-client will use the defaults for cache and output directory. If not
started as root users need to provide both directories as arguments.
While there switch from absolute path names to relative ones. For this
the parser and rsync process do a chdir(2) to the cache directory on startup.
OK benno@

No need to unveil(NULL, NULL) right before pledge "stdio rpath".
From George Brown.

ok benno

split output management code into seperate file. iterate over output
methods using a table. detect output truncation (for instance filesystem
full) and don't overwrite previous output
ok claudio

output_createtmp() remembers the true name, so output_finish() does not
need to replay it.

linewrap usage

missing | leads to incompete results

- no -r option
- man page is bgpd, not openbgpd

ok deraadt

Tweak rpki-client to create all 4 output file formats from a single
compute, based upon flags. OpenBGPD compatible format by default if
no options, to integrate with bgpd.conf and bgpctl reload. Adapt
mtree and stuff. This will receive further refactoring...
ok benno job

Use X509_STORE_CTX_set_flags() instead of doing the same thing by hand
in a much more complicated way.
OK benno@

Restore the previous behaiviour of not checking the crl for the mft.
Added back by accident when readding CRL checks.
ok claudio@

... and fix the compile.

bring back CRL checks, noted by claudio@

whitespaces

s/EXIT_SUCCESS/0/
ok "yeeees" claudio@

s/EXIT_FAILURE/1/
ok claudio@

normalize_name() is no longer needed

Use the AKI embedded in the CRL to store the crls. This way the AKI from
the cert can be used to load the correct CRL. This simplifies the code
even further.
OK benno@, beck@ agrees that this is the right way to do it

If run as root, priv-drop to _rpki-client.
If no output file is specified, output to the bgpd/bgpd.conf-compatible
/var/db/rpki-client/roa file.
ok claudio and benno

fchmod 0644 the output roa file, since it is public information and
this will work better with upcoming privdrop rpki-client and bgpd consumption
ok benno

Shuffle the desk chairs on this ocean liner

enable more warning flags and fix a few issues noticed.
ok claudio@ deraadt@

improve snprintf overflow test; noted by benno

Convert the auths array into an RB tree indexed by SKI. For fast lookups
the parent auth struct is stored as a pointer so walking up the certificate
chain remains simple. Doing this replaces a few slow lookups and speeds up
rpki-client by a factor of 2.
OK benno@

Output to a temporary file, which gets renamed on success. This will
work much better with bgpd.conf include automation.
ok claudio benno

convert the return code to be 0 for success and 1 for failure.
ok claudio@ deraadt@

intentionally fclose output file, rather than relying on exit() to do it.
benno and stsp stared...

add missing prototype

Remove -r flag and associated plumbing. We are always checking CRLs from now on.
ok claudio@

improve on previous to only use the required CRLs.
ok claudio@

To verify a manifest, a roa, or a certificate, we check its signature
against the CA one layer above it. This check just needs to use the
certificate revocation list published by that CA.

Instead of loading all CRLs we find into the trust store, we create a
STACK_OF(X509_CRL), push the onto it and use it with
X509_STORE_CTX_set0_crls(ctx, crls);

The x509_get_crl() function to retrieve a crl path from an X509 object
is written by claudio@.
ok claudio@

Only store ta certs in the trust store and build chains of the
intermediate certificates for use in X509_STORE_CTX_init(). To do that
save the X509 cert in struct cert.
claudio sees > 2x speed increase.

ok claudio@ and feedback from jsing@ and beck@.

rename a variable to show it's a CRL. ok claudio@

Move the define of TALSZ_MAX to top and add a description for
tal_load_default().
from kristaps@

Refactor tal code a bit. Move the file reader back into tal.c so that the
regress test is able to use it.
OK deraadt@

Handle the TAL files in the master process and pass them as buffer to the
parser process. This way the parser never needs to read outside of the
cache directory which makes the unveil simpler. Additionally rsync_uri_parse
no longer needs to know about .tal files so there is now no chance to sneak
in a .tal file later on.
OK deraadt@

tweak previous; help/ok claudio

Add an output format for bird and one doing CSV. Also update the manpage
to include all the changes.

Rewrite the output handling of rpki-client and add an option to dump the
data in JSON format. To make the JSON output the same as the output of the
RIPE rpki-validator the basename of the TAL had to be added and passed around
in rpki-client. Additinally the VRPs are now stored in an RB tree in the
main process instead of keeping them per ROA object. This changes the sort
order to be in network order and no longer just lexographical.
Agreed by job@ deraadt@

For whatever reason it is fairly common that a ROA object references a CRL
that is not in the store. Put this paricular error under verbose logging.
Agreed by deraadt@, job@

Change the arguments to rpki-client a bit. Instead of listing all TAL files
as arguments rpki-client will now load the TAL installed in /etc/rpki by
default. For debug reasons an option -t tal is added to pass in TAL files
by hand. The argument is now instead the filename of the output file.
Now `rpki-client roa.conf` will do what you need which is a lot nicer.
Agreed by deraadt@ job@ to be a step in the right direction.

Show the most common warnings only if verbose is set. Most of these warnings
were shown because of an inconsistent rpki database and is no real problem.
OK florian@

Stop pulling libssl into rpki-client.

None of this code actually does TLS, hence libssl is not needed. Instead,
pull in the correct headers and call the appropriate libcrypto
initialisation functions (even this is only necessary to support OpenSSL
prior to 1.1).

While here also remove libssl/libcrypto initialisation/uninitialisation
from main() - it should only be necessary in proc_parser().

ok deraadt@ job@

Add -b bind_addr as argument which is passed to rsync as --address to
use as source address for connecting to the rsync daemons.
OK sthen@ benno@

If the rsync of a repository fails print out the URI of the failed process.
Better error handling is needed but at least now it is clear which repository
fails.
OK deraadt@

When system calls indicate an error they return -1, not some arbitrary
value < 0. errno is only updated in this case. Change all (most?)
callers of syscalls to follow this better, and let's see if this strictness
helps us in the future.

The main process does not need to toggle the sockets from blocking to
nonblocking just for poll().
OK deraadt@

Close proc fd used to talk to proc_parser when spaning the rsync process
else the fd leaks into the rsync process.
OK deraadt@

use $OpenBSD$ headers

repair a cluster of "< 0" error checks for libc / syscalls, which
should be "== -1"

Improve terminology used

indentation adjustments, in particular near warn statements
ok claudio

swap comparisons

do not need to allocate memory for execvp arguments

Don't do -portable in base. It is better done outside the tree.
Imagine if we did it throughout the tree, how many copies of strlcpy
would we have, and how much time would all the configure shell scripts
and includes take? It would be ludicrous.

branches: 1.1.1;
Initial revision

Download new TA files into a temporary place (.ta/) so that the parser
can decide which of the two files to use.
With and OK tb@

Instead of tracking certificates by SKI track them by an internal identifier.

The certificate SKI is not strictly unique so using it as a unique id is
problematic. It is also not really needed to do that since in theory we
already know the path (but this got lost in the privsep communication).
So add a cert id and pass this id back and forth between main process and
the parser. With this id we can lookup the authentication chain in the
parser and this even works with multiple paths to the same resource.
Since we no longer lookup by SKI the valid_aki_ski function is replaced
by find_issuer() which does the lookup by certid.

The loop protection is now extended to allow each TAL to reach each file
once but still triggers if a file is reaccessed by the tree of a TAL.

In filemode the lookup now uses an AIA uri based lookup tree. Again this
replaces the SKI based lookups from before.

Done together with tb@
OK tb@ job@

Fix capping of VAPs

The previous approach introduced a cap, but it might not always be hit as
intended (I missed this on review). Fix this to check the cap after merging
an ASPA into an already existing VAP. Also free the list of providers since
nothing should be looking at it anymore.

Count VAPs that hit the limit with a new overflowed counter. There are
still a few aspects of the accounting that probably aren't entirely right.
This will be fixed at another point. It's just statistics after all.

with/ok claudio, ok job

Don't emit Validated ASPAs for Customer ASIDs with more than MAX_ASPA_PROVIDERS

The number of providers in a single ASPA object already was limited to
MAX_ASPA_PROVIDERS, now also impose a limit on the total number of providers
across multiple ASPA objects. If the MAX_ASPA_PROVIDERS limit is hit, omit
the Customer ASID's entry from OpenBGPD and JSON output.

OK tb@

Replace protocol literal strings and strlen() calls with defined constants

OK tb@ claudio@

Lipstick on a pig: avoid comparing signed and unsigned

OK tb@ claudio@

Add -x to opt into experimental file formats

Instead of burning one letter for each new file format (sidrops is known
to crank out new things faster than a normal person can read), use -x to
opt into parsing and processing file formats that aren't yet considered
stable. This is currently only the Signed Prefix List. While a repetition
of the ASPA debacle, this code hasn't yet seen enough stress testing to be
enabled by default.

ok claudio job

Track the number of new files moving from 'staging' to 'validated cache'

The OpenMetrics output shows per-repository counters for new files
added, the main process and JSON output emit the sum of all new files.

OK claudio@

Add support for RPKI Signed Prefix Lists

Signed Prefix List are a CMS protected content type for use with the
RPKI to carry the complete list of prefixes which an Autonomous System
may originate to all or any of its routing peers. The validation of a
Signed Prefix List confirms that the holder of the listed ASN produced
the object, and that this list is a current, accurate and complete
description of address prefixes that may be announced into the routing
system originated by this AS.

https://datatracker.ietf.org/doc/html/draft-ietf-sidrops-rpki-prefixlist

with and OK claudio@ tb@

Group logx() getmonotime() and get_current_time() together

Fix their indent in extern.h, move the X509_TIME_* macros to main.c since
they aren't (and can't really be) used elsewhere, document the meaning of
the magic numbers. Also move get_current_time() out of the middle of entity
handlers.

ok claudio job

fix whitespace

Refactor handling of stale manifests

No need to hoist a staleness indicator through the whole process and
count it explicitly.

OK tb@

Allow imposing constraints on RPKI trust anchors

The ability to constrain a RPKI Trust Anchor's effective signing
authority to a limited set of Internet Number Resources allows
Relying Parties to enjoy the potential benefits of assuming trust,
within a bounded scope.

Some examples: ARIN does not support inter-RIR IPv6 transfers, so
it wouldn't make any sense to see a ROA subordinate to ARIN's trust
anchor covering RIPE-managed IPv6 space. Conversely, it wouldn't
make sense to observe a ROA covering ARIN-managed IPv6 space under
APNIC's, LACNIC's, or RIPE's trust anchor - even if a derived trust
arc (a cryptographically valid certificate path) existed. Along these
same lines, AFRINIC doesn't support inter-RIR transfers of any kind,
and none of the RIRs have authority over private resources like
10.0.0.0/8 and 2001:db8::/32.

For more background see:
https://datatracker.ietf.org/doc/draft-snijders-constraining-rpki-trust-anchors/
https://mailman.nanog.org/pipermail/nanog/2023-September/223354.html

With and OK tb@, OK claudio@

Constify argument to entity_write_repo()

OK tb@

Fix comments

OK tb@

rpki-client: fix vap_pas stats

A small mistake in a diff broke the counters. Make them AFI agnostic and
adjust ometric output.

guidance & ok claudio

Improve detection of RRDP session desynchronization

According to RFC 8182, a given session_id and serial number represent an
immutable record of the state of the Repository Server at a certain
point in time.

Add a check to the RRDP notification file processing to compare whether
the delta hashes associated to previously seen serials are different in
newly fetched notification files. Fall back to a snapshot if a difference
is detected, because such a mutation is a strong desynchronization
indicator.

Idea from Ties de Kock (RIPE NCC).
Based on a diff by job@
With and OK job@ tb@

Use new ibuf API in rpki-client.

Use ibuf_fd_get/set/avail in the file descriptor passing code.
Switch some ibuf_seek() calls to ibuf_set() and use ibuf_add_zero() instead
of ibuf_reserve().
OK tb@ job@

Fixup file modification timestamps to optimize failover from RRDP to RSYNC

In the RSYNC protocol a file's last modification time and its size are
used to determine whether sending a (partial) copy over the wire is needed.
Previously, when RRDP data structures are serialized to disk, the mtime of
files in DIR_VALID ended up being UTIME_NOW.

Thus, the mtimes of files obtained through RRDP will never match the mtimes
of the same files available through RSYNC - causing each and every file to
be added to the file transfer list.

Instead, use the internal timestamps of RPKI files as the last modified
timestamp. Specifically, for Signed Objects (ROAs, MFTs, GBRs, TAKs, ASPAs)
the CMS signing-time, for .cer files the X.509 notBefore, and for .crl files
the CRL lastUpdate. This results in a surprising optimization for the number
files which have to be transfered.

OK claudio@

Revert commitid ANSBO0rBvIUtTi45:
Make rpki-client choose the verification time of the time it is invoked
rather than always getting the current system time for every certificate
verification. This will result in output that is not variable on run-time.

Using the time of invocation does not work well with fast publishing CAs. It
can take a few minutes to reach a repo and that CA may have issued certificates
that are not yet valid if that startup time of rpki-client is used to validate.

This still keeps the -P option to specify a fixed validation time.
OK beck@ job@ tb@

Split cleanup into cleanup and repository cleanup and show how many files
are kept / removed in the repository temporary storage.
After a discussion with tb@ and job@

Properly account del_extra_files and add the value to the json header.
OK tb@ (from a larger diff)

Update usage add -P epoch

Make rpki-client choose the verification time of the time it is invoked
rather than always getting the current system time for every certificate
verification. This will result in output that is not variable on run-time.

ok tb@ claudio@

Add a -P option to rpki-client to specify the evaluation time

This is intended to be able to test rpki-client in a reproducable
way without worrying about the system time changing the results

ok claudio@

Improve accounting by tracking things by repo and tal.
This fixes some wrong accounting for repositories that are referenced
from more than one TAL. It changes the ometric lable output a little bit
since there are repository metrics that no longer include the 'name' label.
OK tb@

Check whether products listed on a manifest were issued by the same authority as the manifest itself

OK tb@

When parsing MFT the CRL referenced by the MFT is loaded and verified at
the same time. So in case of a valid crl pass the CRL filename as entity
message to the parent process together with the MFT. This way the MFT and
CRL end up both in the valid cache even if some files in the MFT are missing.
On severe errors (like X.509 verify errors) the CRL is not moved since it
is not considered valid.
With and OK job@, tb@

Add aspa-set to openbgpd config output.

Change the way the validated ASPA tree is built since OpenBGPD config
follows more the ASPA profile and puts the optional AFI to each provider
ASnum instead of duplicated everything into an IPv4 and IPv6 tree.
The JSON output of ASPA is still the same.

The inclusion of the aspa-set can currently be disabled by the -A flag.
OK tb@

Rework the main <-> parser/filemode communication protocol a bit.

Swap repo_id and filename to simplify the code in parser.c. In filemode
both repo_id and filename are ignored.
Additionally do not errx() in case of unknown file types. Instead send back
enough info that the code can move on.

OK tb@

Rework statistic collection to be per repository and add metric output option

Many statistic values are now accounted by repository via repo_stat_inc()
At end of the run sum_stats() accumulates these stats per TAL and globally.
The new output file metrics is written when the -m output flag is specified.
The metrics file is written in OpenMetrics format (with a few tweaks to
allow node_exporter to parse the file as well). The ometric code is a copy
from bgpctl(8) and should be kept in sync.
OK tb@

Switch to struct timespec for collecting stats. This allows to use
clock_gettime(CLOCK_MONOTONIC) for runtime calculation.
OK tb@

Remove unused ctype.h include

OK tb@

Only include stdarg.h, if we call any of va_{start,end}()

OK tb@

Reshuffle case a little bit. No functional change.

Move skiplist to main and merge with shortlist.

With/ok job

zap errant space in usage();

Schedule cleanup of '-r' now useless getopt option

OK tb@

Add shortlist functionality, a compagnion to the skiplist

If the operator specifies the '-H' option once (or more) followed by a FQDN,
the utility will *only* connect to those hosts and skip all others.

OK claudio@ tb@

Add support for draft-ietf-sidrops-signed-tal-12

Add support validation of Signed Objects containing Trust Anchor Keys
(TAKs - aka 'Signed TALs'). Signed TALs provide a mechanism for RIRs
to distribute and sign the next Trust Anchor with the current Trust
Anchor. This might be an improvement over visiting RIR websites and
copy+pasting TAL data by hand.

OK tb@

Move the repo lookup into queue_from_mft()
OK tb@

Introduce a deadline timer that aborts all repository syncs.
With this rpki-client has a chance to still finish and produce an output
even when a CA is excessivly slow and holds back progress.
With and OK benno@ tb@ and job@

Fix over long lines
OK tb@ job@

Use the abort commands when a repo timeout happens. This is cleaner
then just failing the repo fetch but leaving the backends running.
OK tb@

Avoid dead assignment

Add support for ASPA objects (draft-ietf-sidrops-aspa-profile-10)

ASPA objects are published in the RPKI and can be used to detect and
mitigate BGP route leaks. Validated ASPA Payloads are visible through
filemode (-f) and the JSON output format (-j).

With feedback from tb@

OK claudio@ tb@

Simplify load_skiplist()

Instead of copy-pasting valid_uri(), we can call the real thing. That's
also not ideal since we actually check for an FQDN, but it's a bit
simpler.

ok claudio

Tweaks in load_skiplist()

If there's an issue opening the default skip list file other than its
absence (most likely bad permissions), we should not silently ignore
that. Also, use err() instead of errx() to display the error.

Second, linelen, the return value of getline(), is not currently used.
Repurpose it to save the result of strcspn() which calculates the length
of the string we're interested in and use that instead of strlen().

ok claudio

Make PEM printing available through increased verbosity (-vvf)

Suggestion from claudio@

OK tb@

In verbose filemode print details about encapsulated certificates.
Add command line flag to print the certificate in PEM format.

OK tb@

Once all files from argv have been read the main process can unveil to
the cachedir and if output files are written to outputdir. In -f mode
the unveil can be read-only in normal operation rwc is required because
the main process writes the RRDP files and also does the cleanup at the
end of the run.
Input from tb@ and mestre@, OK tb@

Add skiplist option to steer clear of skiplisted hosts

Blocking outbound connections towards RPKI publication servers based
on IP or IPv6 address in external instrumentation like HTTP proxies
or pf(4) rules is somewhat unwieldy. It might be easier for operators
if we offer a mechanism that cuts at the CA cert SIA parsing step.

OK claudio@ tb@

Don't add unknown filetypes to the entity queue. Otherwise we bail out
without good reason. Regression introduced in a recent refactoring found
by job

ok claudio job

Remove a confusing comment that has been stale since e2k19 (r1.26).

There is no need to be quiet by default anymore and also stdout is
perfectly fine. So switch the stats output at the end of the run to
simply use printtf(3) and no longer depend on -v flag.
OK tb@

More KNF and whitespace fixes.

Improve control flow readability by removing 'else if' stanzas

OK tb@ claudio@

rpki-client: bypass timeout in file mode.

ok claudio

Tweak wording of a comment to reflect current reality better.

typo: exceded -> exceed

The filemode code is enough different from the regular parser code that it
makes sense to totally split it out. Duplicate proc_parser_cert_validate()
and proc_parser_root_cert() for now.

The valid_x509() plus the required static functions are moved to validate.c.
The crl_tree code moved into crl.c similar to the auth_tree handling in
cert.c. All the proc functions are now tagged with __attribute(noreturn)
which allows to remove the errx() after them.

OK tb@

Remove an unreachable error message

If timeout != 0 and 0 <= timeout <= 24*60*60, then timeout < 1 is
impossible.

ok

zap extra line break and extra empty line

Change type of talsz and dependent code from size_t to int. Tal ids are
already stored as int and and talsz is the limit for these ids.
OK tb@

Cosmetics, add missing space

Adjust on how CRL and MFT files are verified.

Verify the CRL referenced from the mft against the mft's fileAndHash info.
If the CRL matches then load it and use it to validate this mft. If the
mft validated OK add the now also valid CRL to the auth store for later use.

Before the newest CRL was always selected but that has negative consequences
because it is common practice to revoke the previous MFT's EE cert and with
that the cache is turned useless as soon as a new CRL is used. Also there
was a possibility that the CRL used for validation of the MFT was not the
one later used.

Both RFC6486 and draft-ietf-sidrops-6486bis are unclear about this part
of the validation process. We opted in favor of the chached MFT.

With and OK tb@

Refactor on how the subprocesses are started.

Move the unveil and pledges to the actuall subprocesses and put all the
common code to start these into process_start(). Reduces the lenght of
main() a fair bit.
OK tb@

Adjust cache cleanup to the deficits of RRDP. Unlike rysnc, RRDP has no
method to sync the cache. It just depends on all deltas to work but has
no method to check if the result is consistent with the source.
Because of this do not unlink files which were fetched via RRDP. Instead
move them back to their .rrdp directory and hope the will be cleaned up.
This should help to keep the cache coherent in some edge cases.
OK tb@

rpki-client: remove a stale FIXME

We fixed this back in January when we added rtype_from_mftfile().

ok claudio

Split certificate parsing in two steps. cert_parse_pre() which does
the parse and some checks and cert_parse() or ta_parse() to do the
additional checks for regular certs or TAs.
With this adjust the cert parser in -f mode to identify TAs (by checking
if it is self signed) and adjust the validation in that case. Now -f
should be able to parse and show all object correctly.
With and OK tb@

Improve free inodes check from fs.f_favail > 0 to fs.f_free > 0.

suggested by millert

Only show the 'not enough inodes' warning if the filesystem reports more
than zero free inodes. btrfs always reports zero inodes, so on linux
distros like fedora, users will always see the not enough space warning,
which it is a bit ugly.

fine with claudio

Properly handle .mft files as intended by the RFC. Instead of always
selecting the newest file this opens both the new (from rrdp or rsync)
and old (valid) MFT. It then compares the manifest number and based on
that the 'newer' MFT is selected.
The MFT file and hash check is also changed to always try both locations
and selecting whatever matches up with the hash. The selction is passed
back to the the main process and used later on to open exactly the same
file as was checked against the hash.
The MFT parsing code has been split up into multiple steps so that the
files can be parsed, compared and then fully validated.
In most cases this makes no difference but it prevents replay attacks
using old but still valid files.
With and OK tb@

Allow rsync:// URI as file in -f mode. This makes it easier to explore
rpki repositories by following AIA and manifest URIs.
Also stop checking the the loaded file is not part of the auth tree,
it is possible that this file was loaded before as a dependency.
OK tb@

Adjust code to handle unsupported file types a bit more graceful.
The file still needs to match its hash to make the MFT valid but then
there will only be a warning printed. Parsing of other files from that
MFT are not influenced.
OK tb@

rearrange SYNOPSIS/usage to be a bit clearer;
discussed with and ok claudio

Handle EINTR the same way in all poll loops. In all cases restart the
poll loop. In the main process move the timeout handling for repositories
into a single function that does the timeouts and the calculation of the
timeout in one go.
OK tb@

Allow rpki-client to display more than one file in -f mode.
Change -f to be a mode flag and pass one or multiple files as arguments
to rpki-client. Some extra checks need to be done to not load the same
certificate or CRL multiple times.
Input and OK tb@

Add function to determine the file type from the file name extension

rpki-client uses the same idiom to determine the file type in too many
places. Use one function that determines the appropriate RTYPE from the
file name. Add that type to struct mftfile and use this new member to
simplify queue_add_from_mft*().

input/ok claudio

Avoid out-of-bounds accesses in case the filename is too showrt to
contain an extension of length four.

ok claudio

Add -f to usage in main routine

OK tb@

Implement a filemode where the passed file from the command line is
printed in human readable form and is also verified against the valid
cache of rpki-client.

To validate the file the chain is explored backwards by looking at the
Authority Information Access URI and X509v3 CRL Distribution Point of
the cert. Once the trust anchor is found this chain can be verified.

Feedback and OK job@ tb@

rpki-client: don't leak nfile if no repo is found.

ok claudio

Introduce a validated cache which holds all the files that have
successfully been verified by rpki-client.

With this the rsync and rrdp directories are more of a temporary storage
location. New files are downloaded there and then moved to the valid
directory at the end. In -n mode only the valid directory is looked at with
the exception of the ta directory holding the trust anchors.
A file can now be in two different locations so adjust all the code paths
that open files to check both locations.
One nice side-effect of this is that the RRDP handling in the main process
got simplified. There is no longer the need for temporary RRDP directories.
OK tb@

Implement a RRDP_CLEAR message that instructs the parent to cleanup
the rrdp directory. This is used before a snapshot download to ensure
that the snapshot is applied to a clean repo.
Similar cleanup happens if the transfer fails. In that case remove the
temp directory contents only.
This uses a new function remove_contents() to remove everything below
a base directory (a bit like rm -r X/*).
OK tb@

Implement but don't use code to use rsync's --compare-dest feature.
One gotcha is that the path passed to --compare-dest needs to be relative
to the dst directory. rsync_fixup_dest() will prepend the necessary ../
for that by counting number of '/' in dst.
OK tb@

Change the way the parser accesses files. It now builds the file path
based on information from the repository, a local path and the filename.
This simplifies some code both in the main process and the parser.
For this to work repositories are passed to the parser before any other
entity of this repository is passed. Struct entity is extended to include
the repoid and the path along the file(name).
Input and OK tb@ & job@

Cleanup mft file handling, especially the stale mft bits.
Move staleness check up into mft_parse_econtent() to simplify code.
Remove the big FIXME bits since they are no longer needed. The parent
process will only process MFTs that are not stale.
Cleanup a few other bits mainly unneccessary else if cascades and
use valid_filename() to check if the filename embedded in the mft
fileandhash is sensible.
OK tb@

Pass the filename back from the parser to the parent.
The parent will then add the filename to the filepath tree instead
of doing that in entity_write_req(). In the parser pass the filename
instead of the full entity object to various proc_parser functions.
With this it will be possible to check more then one file in the parser.
OK tb@, earlyer version OK benno@

Cleanup struct entity. Remove the unneeded has_data field, the same
information can be figured out by looking at the data pointer itself.
It is NULL when there is no data and not-NULL if there is data.
OK benno@

Replace two questionable size_t types. For the repo id use a unsigned int
and for the roa maxlength use unsigned char (like the prefixlen in struct
ip_addr).
With input and OK job@

Cleanup a few things while reading the code.
OK job@

Initialize the repo_timeout to timeout / 4 at start of main().
OK job@

Replace MAX_REPO_TIMEOUT with repo_timeout, which is set to 1/4th of
timeout, or if timeout is disabled set to 24 hours.

OK claudio@

Fix typo

Limit the number of publication points under a given TAL.
Introduce an additional timeout for each publication point.

The limits are large enough to accomodate normal operating levels.

With and OK benno@ job@ tb@ beck@ deraadt@

On errors related to the pipes to the childs don't error out right away.
Instead exit the main event loop and use waitpid to know why a child
went away. This should make it hopefully more clear when shit hits the fan.
OK tb@ deraadt@

Move and promote getmonotime() to an internal API function.

Instead of passing tal descriptions around just pass a tal id and
use a small lookup table to print the description in the output path.
OK tb@

Further simplify cert and auth handling. Move common code into auth_insert
and skip this distinction between invalid and failed certificates.
The difference between the to is getting more and more blurry.
OK tb@

Further improve load_file(). Always set an errno on errors and make sure
the errno is not clobbered in the error path. Check result from load_file()
and issue a warning and skip the file in queue_add_tal().
OK benno@ beck@

When converting the TAL parser to use load_file and pass the file via
data/datasz element to the parser process I forgot to remove the old
buf argument that abused descr for that. Because of this rpki-client
can crash.
Found by benno@, OK benno@ beck@

Sync output at end of run with output inside the openbgpd output file.
With and OK tb@

Move load_file() to encoding.c so that regress can use the function.

Refactor the tal parsing code to use the same load_file() and buffer
passing as done for the other parsers.
OK job@ tb@

For open/openat, if the flags parameter does not contain O_CREAT, the
3rd (variadic) mode_t parameter is irrelevant. Many developers in the past
have passed mode_t (0, 044, 0644, or such), which might lead future people
to copy this broken idiom, and perhaps even believe this parameter has some
meaning or implication or application. Delete them all.
This comes out of a conversation where tb@ noticed that a strange (but
intentional) pledge behaviour is to always knock-out high-bits from
mode_t on a number of system calls as a safety factor, and his bewilderment
that this appeared to be happening against valid modes (at least visually),
but no sorry, they are all irrelevant junk. They could all be 0xdeafbeef.
ok millert

Add my copyright to some files
OK job@

Rename io_buf_new to io_new_buffer and io_buf_close to io_close_buffer.
With this the write functions are all of the form io_xyz_buffer.
Remove some prototypes of functions I forgot to remove in previous commit.
OK benno@

Finnally move away from blocking reads in rpki-client. The code was a
mish mash of poll, non-blocking writes and blocking reads. Using the
introduced ibuf size header in io_buf_new()/io_buf_close() the read
side can be changed to pull in a full ibuf and only start the un-marshal
once all data has been read.
OK benno@

First step of cleanup in the io land. Introduce io_buf_new() and
io_buf_close(). These function will inject a size of the the buffer
at the beginning of the buffer and will allow the read size to be
switched to proper async IO.
OK benno@

Add support for BGPsec Router Certificates (RFC 8209)

BGPsec router keys are extracted from RPKI certificates and
emitted via the JSON output in base64 encoded form.

OK tb@ claudio@

Remove unused variable

Make style consistent and remove an unused code path

OK tb@

Add rudimentary support for BGPsec router certificates

OK claudio@

branches: 1.145.4;
Properly account entity_queue when the file has already been seen

Thanks to Ben Maddison for helping create a test case

OK claudio@

Flip the -r RRDP switch, which went through community testing for half a year.
Default is now to attempt RRDP first, then try RSYNC.
If problems show up, return to rsync-only behaviour with -R, and file a report.
ok claudio

Do a fstatvfs() call to figure out if the filesystem used for the cache
is large enough for the cache. People like to build VM images with way
too small filesystems and so warning about this situation should help.
With deraadt@ and job@

Use O_DIRECTORY when opening directories. This makes some errors a bit
clearer when using -d.
OK deraadt@

Make sure some variables are initialised since modern gcc warns about them.
Handle rrdppid like we do for all other child processes. The two warnings
in rrdp are probably false positives.
OK tb@

Arm the alarms in all childs so they terminate if the timeout is hit.
Also install the special signal handler around the main poll() loop.
The main process handles SIGALRM so it can log an error to the operator
before exiting.
With and OK deraadt@

code review results in KNF, and moving local variables into lowest scope
ok claudio

branches: 1.138.2;
Rework the http code to require poll() only when really needed.
Especially tls_read() and tls_write() do not map 1:1 to read() and write()
calls and so assuming that after a tls_read() one needs to poll for more
data is wrong. Instead call tls_read() until it returns a TLS_WANT_*
return.

While here also ignore SIGPIPE. It is almost impossible to properly guard
from SIGPIPE by looking at POLLHUP. Instead just let write() handle it and
return an error.

Putting this in now so this can be tested widely.

Make rpki-client -V output nicer.
OK job@, kn@, deraadt@

move the RPKI_VERSION define into its own version.h file, helps portable.
ok claudio@

Fix two typos in comments

sure deraadt@

Switch logic from != to ==. Makes the code easier to read.

More cleanup of the main function. Use a loop to collect all childs with
waitpid() and print if they exited non-zero or by a signal. Also adjust
the poll hangup case to exit the poll loop instead of erroring out. This
way a crashed child should be reported before exit.
OK job@ tb@

Missing setproctitle("rrdp") noticed by deraadt@

minor KNF found while hunting for a bug

Sort usage: rR -> Rr

Initial commit of RRDP (The RPKI Repository Delta Protocol - RFC8182) support
in rpki-client. For now it is off by default.

All XML processing is done in its own process with minimal pledge rights.
It uses the already present https process to fetch the xml files and uses
the master porcess to handle the file IO into the repositories.
RRDP data is stored in the cache under ./rrdp/ and the first directory
is the SHA256 hash of the notify URI.

Fetching snapshots and deltas works to bring the cache up to date.
If something goes wrong rpki-client will fall back to rsync.

RRDP was implemented by Nils Fisher and integrated into rpki-client by myself.
"Time to get it in" deraadt@

Do a better job at cleaning up. Remove empty directories, scan not only the
known repositories but also clean up no longer known repositories.
With this rpki-client keeps its cache nice and shiny.
With and OK job@

Set the process title for the rpki-client subprocesses so they can be
identified more easily.
OK deraadt@

minor KNF and consistancy
(rpki-client is pretty good code, couple hours of audit and I can only find
a few minor things)

Compare filepath with strcmp() and not strcasecmp(). The URI in RPKI are
case sensitive.
OK tb@

Rewrite poll loop a bit. Put the various message queues into an array
with the same order as the pollfds. This way simple for loops can be
used to setup and handle all cases but POLLIN.
OK tb@

Adjust HTTP client code a bit. Add support for 304 Not Modified responses,
remove handling of 206 Partial Content (the client does not use range
headers). Report the Last-Modified timestamp back to the requestor and
switch OK to a enum value for (FAIL, OK and NOT MODIFIED).
OK tb@

Add an -V option to show the version of rpki-client. For the base version
it will show just OpenBSD while -portable will show the portable version.
OK sthen@, tb@, kn@

Delay chdir to the cache directory to after parsing the tal files.
Using the -t option relative locations can be passed as tal locations
and so the process can not chdir until these files were read.
OK job@

Initialize rsyncpid and httppid in the noop case. It seem gcc is not able
to realize that the pids are initialized if !noop and not accessed if noop.
OK job@ tb@

Allocate the repo structs individually linked by a SLIST instead of using
an array that is reallocated during runtime. With this the entityq can
move back into struct repo.
OK tb@

Revert rev 1.116
The repo structs are reallocated during runtime and so the back pointers to
the head element of the TAILQ get corrupted.
Noticed by tb@

There is no need to revisit a file in the repo, so if the RB_INSERT fails
just drop the entity queue element.
OK benno@ tb@

There is no need for a global enity queue, instead use per repo queues.
Simplifies the code a fair bit.
OK tb@

When adding files from the manifest only the .crl need to be first, after
that the order does not matter so simplify the code and just walk the
list twice. Add the .crl first and then in the second round all other files.
OK job@

Add two error checks for strdup for consistency.

ok claudio

Last commit was done from a non-clean tree. Revert one line that broke
everything. Oups.

Use the same way to error out in out of memory situation.
Just use 'err(1, NULL);' there is no need to include the type of function
that failed since it is still impossible to locate the right call.
Just use a debugger in that case.
OK tb@ deraadt@

an -> and

Replace NOTREACHED with actuall errx() calls. I managed to return from
the proc_xyz() function at least in two occastions and it is not that
trivial to realize what goes wrong.

Implement a https client as a sub-process for rpki-client. This code will
be used to fetch TA certs and later on for RRDP. Kind of unreached for now
since the default TAL files don't include https URI.
The http client is fully asynchronous and can handle multiple downloads at
the same time. This code was based on the http client in ftp(1).
OK tb@, job@

There is no longer the need to mkpathat(), the main process runs now from
the cachedir.

Open both the cachedir and outputdir early and use fchdir(2) to
switch between the two.
OK deraadt@ job@

It is perfectly fine to POLLIN and POLLOUT at the same time. It may
actually make things a bit faster.
OK deraadt@

Adjust pledge() and unveil() calls for proc_rsync() a bit. Since the
mkdir was moved to the main process there is no need for access to .
in the rsync process.
OK job@ deraadt@

Rename mkpath() to mkpathat() since it uses mkdirat() internally.

Convert reallocarray + memset to recallocarray.

ok claudio

Move the mkpath() call from the rsync path to the main process. This allows
to drop cpath from the rsync proc pledge (down to "stdio proc exec").
This will also make work easier with the upcoming http fetcher.
OK tb@

All the code for output handling was moved to output.c some time ago.
Also move the license that came along with it to output.c.

Prevent double free by copying the URI string from struct tal to the repo.

Rework the repository handling. Split the handling of trust anchors into
ta_lookup() while regular repositories (to fetch .mft files) are handled
by repo_lookup(). Also the cache directory layout changed; moving the
trust anchors to ./ta/{tal basename}/ the other repositories end up in
./rsync/
OK tb@

Remove the not so periodic stats from rpki-client. The output is not very
useful and the way it works is a bit strange.
OK job@ tb@

The uri parameter of struct entity is actually a local file path.
Rename field to reduce confusion about what is what.
OK tb@

Define the msgbuf queues globably. Clean up the code since rsyncq and procq
no longer need to be passed all the way down anymore. Shuffle code a bit
to bring it into more order.
OK tb@