opencrypto: crypto_dispatch never fails now. Make it return void.

Same with crypto_kdispatch.

opencrypto: Rip out EAGAIN logic when unregistering crypto drivers.

I'm pretty sure this never worked reliably based on code inspection,
and it's unlikely to have ever been tested because it only applies
when unregistering a driver -- but we have no crypto drivers for
removable devices, so it would only apply if we went out of our way
to trigger detach with drvctl.

Instead, just make the operation fail with ENODEV, and remove all the
callback logic to resubmit the request on EAGAIN. (Maybe this should
be ENXIO, but crypto_kdispatch already does ENODEV.)

opencrypto: Make crypto_freesession return void.

No callers use the return value. It is not sensible to allow this to
fail.

netipsec: Nothing uses xf_zeroize return value. Nix it.

opencrypto: Make crp_callback, krp_callback return void.

Nothing uses the return values inside opencrypto, so let's stop
making users return them.

Fix ipsecif(4) IPV6_MINMTU does not work correctly.

make DPRINTF use varyadic cpp macros, and merge with IPSECLOG.

Merge the [pgoyette-compat] branch

branches: 1.106.2;
Constify ipseczeroes, and remove one use of it.

Correctly handle the padding for IPv6-AH, as specified by RFC4302. Seen in
a FreeBSD bug report, by Jason Mader.

The RFC specifies that under IPv6 the complete AH header must be 64bit-
aligned, and under IPv4 32bit-aligned. That's a rule we've never respected.
The other BSDs and MacOS never have either.

So respect it now.

This makes it possible to set up IPv6-AH between Linux and NetBSD, and also
probably between Windows and NetBSD.

Until now all the tests I made were between two *BSD hosts, and everything
worked "correctly" since both hosts were speaking the same non-standard
AHv6, so they could understand each other.

Tested with Fedora<->NetBSD, hmac-sha2-384.

Introduce ah_authsiz, which computes the length of the ICV only. Use it in
esp_hdrsiz, and clarify.

Until now we were using ah_hdrsiz, and were relying on the fact that the
size of the AH header happens to be equal to that of the ESP trailer.

Now the size of the ESP trailer is added manually. This also fixes one
branch in esp_hdrsiz: we always append an ESP trailer, so it must always
be taken into account, and not just when an ICV is here.

Strengthen and simplify, once more.

Fix non-INET6 builds

IP6_EXTHDR_GET -> M_REGION_GET, no functional change.

Remove unused calls to nat_t_ports_get.

ENOBUFS -> EACCES when updating the replay counter.

Remove unused 'mp' argument from all the xf_output functions. Also clean
up xform.h a bit.

When IP6_EXTHDR_GET fails, return ENOBUFS, and don't log an error (HDROPS
is not supposed to be used here).

When the replay check fails, return EACCES instead of ENOBUFS.

Remove IPSEC_SPLASSERT_SOFTNET, it has always been a no-op.

Remove the M_AUTHIPDGM flag. It is equivalent to M_AUTHIPHDR, both
are set in IPsec-AH, and they are always handled together.

Remove the kernel RH0 code. RH0 is deprecated by RFC5095, for security
reasons. RH0 was already removed in the kernel's input path, but some
parts were still present in the output path: they are now removed.

Sent on tech-net@ a few days ago.

Remove extra long file paths from the headers.

cosmetic

Simplify the IPv4 parser. Get the option length in 'optlen', and sanitize
it earlier. A new check is added (off + optlen > skip).

In the IPv6 parser we reuse 'optlen', and remove 'ad' as a result.

Remove dead code.

ok ozaki-r@

Remove duplicate, to better show that this place doesn't make a lot of
sense. The code should probably be removed, it's a leftover from when we
had #ifdef __FreeBSD__.

branches: 1.87.2;
Reinforce this area, make sure the length field fits the option. Normally
it always does because the options were already sanitized earlier.

Add [ah/esp/ipcomp]_enable sysctls, and remove the FreeBSD #ifdefs.
Discussed with ozaki-r@.

Remove some more FreeBSD sysctl declarations that already have NetBSD
counterparts. Discussed with ozaki-r@.

Fix buffer overflow on sending an IPv6 packet with large options

If an IPv6 packet has large options, a necessary space for evacuation can
exceed the expected size (ah_pool_item_size). Give up using the pool_cache
if it happens.

Pointed out by maxv@

Commonalize error paths (NFC)

style

Make sure the Authentication Header fits the mbuf chain, otherwise panic.

Fix use-after-free, 'ah' may not be valid after m_makewritable and
ah_massage_headers.

Fix kernel panic (assertion failure) on receiving an IPv6 packet with large options

If an IPv6 packet has large options, a necessary space for evacuation can
exceed the expected size (ah_pool_item_size). Give up using the pool_cache
if it happens.

Pointed out by maxv@

Don't relook up an SP/SA in opencrpyto callbacks

We don't need to do so because we have a reference to it. And also
relooking-up one there may return an sp/sav that has different
parameters from an original one.

Reinforce and clarify.

Fix a vulnerability in IPsec-IPv6-AH, that allows an attacker to remotely
crash the kernel with a single packet.

In this loop we need to increment 'ad' by two, because the length field
of the option header does not count the size of the option header itself.

If the length is zero, then 'count' is incremented by zero, and there's
an infinite loop. Beyond that, this code was written with the assumption
that since the IPv6 packet already went through the generic IPv6 option
parser, several fields are guaranteed to be valid; but this assumption
does not hold because of the missing '+2', and there's as a result a
triggerable buffer overflow (write zeros after the end of the mbuf,
potentially to the next mbuf in memory since it's a pool).

Add the missing '+2', this place will be reinforced in separate commits.

Revert a part of rev1.49 (six months ago). The pointer given to memcpy
was correct.

Discussed with Christos and Ryota.

Constify isr at many places (NFC)

Use pool_cache(9) instead of pool(9) for tdb_crypto objects

The change improves network throughput especially on multi-core systems.

MP-ify SAD (savlist)

localcount(9) is used to protect savlist of sah. The basic design is
similar to MP-ifications of SPD and SAD sahlist. Please read the
locking notes of SAD for more details.

Introduce KEY_SA_UNREF and replace KEY_FREESAV with it where sav will never be actually freed in the future

KEY_SA_UNREF is still key_freesav so no functional change for now.

This change reduces diff of further changes.

Make IPsec SPD MP-safe

We use localcount(9), not psref(9), to make the sptree and secpolicy (SP)
entries MP-safe because SPs need to be referenced over opencrypto
processing that executes a callback in a different context.

SPs on sockets aren't managed by the sptree and can be destroyed in softint.
localcount_drain cannot be used in softint so we delay the destruction of
such SPs to a thread context. To do so, a list to manage such SPs is added
(key_socksplist) and key_timehandler_spd deletes dead SPs in the list.

For more details please read the locking notes in key.c.

Proposed on tech-kern@ and tech-net@

Don't acquire global locks for IPsec if NET_MPSAFE

Note that the change is just to make testing easy and IPsec isn't MP-safe yet.

Use pool to allocate tdb_crypto

For ESP and AH, we need to allocate an extra variable space in addition
to struct tdb_crypto. The fixed size of pool items may be larger than
an actual requisite size of a buffer, but still the performance
improvement by replacing malloc with pool wins.

Dedup error paths (NFC)

Fix a debug message

Hold a reference to an SP during opencrypto processing

An SP has a list of isr (ipsecrequest) that represents a sequence
of IPsec encryption/authentication processing. One isr corresponds
to one opencrypto processing. The lifetime of an isr follows its SP.

We pass an isr to a callback function of opencrypto to continue
to a next encryption/authentication processing. However nobody
guaranteed that the isr wasn't freed, i.e., its SP wasn't destroyed.

In order to avoid such unexpected destruction of isr, hold a reference
to its SP during opencrypto processing.

Don't bother the case of crp->crp_buf == NULL in callbacks

Don't release sav if calling crypto_dispatch again

branches: 1.62.2;
Calculate ah_max_authsize on initialization as well as esp_max_ivlen

Prepare to stop using isr->sav

isr is a shared resource and using isr->sav as a temporal storage
for each packet processing is racy. And also having a reference from
isr to sav makes the lifetime of sav non-deterministic; such a reference
is removed when a packet is processed and isr->sav is overwritten by
new one. Let's have a sav locally for each packet processing instead of
using shared isr->sav.

However this change doesn't stop using isr->sav yet because there are
some users of isr->sav. isr->sav will be removed after the users find
a way to not use isr->sav.

Pass sav directly to opencrypto callback

In a callback, use a passed sav as-is by default and look up a sav
only if the passed sav is dead.

Fix header size calculation of ah in the case sav is NULL

This fix was also needed for esp.

Use explicit_memset to surely zero-clear key_auth and key_enc

Rename key_alloc* functions (NFC)

We shouldn't use the term "alloc" for functions that just look up
data and actually don't allocate memory.

Remove codes for PACKET_TAG_IPSEC_IN_CRYPTO_DONE

It seems that PACKET_TAG_IPSEC_IN_CRYPTO_DONE is for network adapters
that have IPsec accelerators; a driver sets the mtag to a packet
when its device has already encrypted the packet.

Unfortunately no driver implements such offload features for long
years and seems unlikely to implement them soon. (Note that neither
FreeBSD nor Linux doesn't have such drivers.) Let's remove related
(unused) codes and simplify the IPsec code.

Apply C99-style struct initialization to xformsw

branches: 1.54.2;
Make ipsec_address() and ipsec_logsastr() mpsafe.

branches: 1.53.2;
Retire ipsec_osdep.h

We don't need to care other OSes (FreeBSD) anymore.

Some macros are alive in ipsec_private.h.

Convert IPSEC_ASSERT to KASSERT or KASSERTMSG

IPSEC_ASSERT just discarded specified message...

Remove __FreeBSD__ and __NetBSD__ switches

No functional changes (except for a debug printf).

Note that there remain some __FreeBSD__ for sysctl knobs which counerparts
to NetBSD don't exist. And ipsec_osdep.h isn't touched yet; tidying it up
requires actual code changes.

cosmetic fixes:
- __func__ in printfs
- no space after sizeof
- eliminate useless casts
- u_intX_t -> uintX_t

- fix old refactoring which zeroed the wrong part of the buffer.
- simplify.

change into __func__

Redo the statistics through an indirection array and put the definitions
of the arrays in pfkeyv2.h so that they are next to the index definitions.
Remove "bogus" comment about compressing the statistics which is now fixed.

Fix that ah_algorithm_lookup and esp_algorithm_lookup don't handle some algorithms

Unrelated upper limit values, AH_ALG_MAX and ESP_ALG_MAX, prevented some
algorithms from being looked up.

Prepare netipsec for rump-ification

- Include "opt_*.h" only if _KERNEL_OPT is defined
- Allow encapinit to be called twice (by ifinit and ipe4_attach)
- ifinit didn't call encapinit if IPSEC is enabled (ipe4_attach called
it instead), however, on a rump kernel ipe4_attach may not be called
even if IPSEC is enabled. So we need to allow ifinit to call it anyway
- Setup sysctls in ipsec_attach explicitly instead of using SYSCTL_SETUP
- Call ip6flow_invalidate_all in key_spdadd only if in6_present
- It's possible that a rump kernel loads the ipsec library but not
the inet6 library

branches: 1.44.2; 1.44.4;
Tidy up opt_ipsec.h inclusions

Some inclusions of opt_ipsec.h were for IPSEC_NAT_T and are now unnecessary.
Add inclusions to some C files for IPSEC_DEBUG.

KNF

branches: 1.42.4; 1.42.6; 1.42.8; 1.42.12;
- apply some __diagused
- remove unused variables
- move some variables inside their relevant use #ifdef

Fix sense of consttime_memequal and update all callers.

Now it returns true (nonzero) to mean equal and false (zero) to mean
inequal, as the name suggests.

As promised on tech-userlevel back in June:

https://mail-index.netbsd.org/tech-userlevel/2013/06/24/msg007843.html

branches: 1.40.2;
Replace consttime_bcmp/explicit_bzero by consttime_memequal/explicit_memset.

consttime_memequal is the same as the old consttime_bcmp.
explicit_memset is to memset as explicit_bzero was to bcmp.

Passes amd64 release and i386/ALL, but I'm sure I missed some spots,
so please let me know.

PR/47886: Dr. Wolfgang Stukenbrock: IPSEC_NAT_T enabled kernels may access
outdated pointers and pass ESP data to UPD-sockets.
While here, simplify the code and remove the IPSEC_NAT_T option; always
compile nat-traversal in so that it does not bitrot.

branches: 1.38.2;
Add "consttime_bcmp" and "explicit_bzero" functions for both kernel
abd userland, as proposed on tech-security, with explicit_bzero using
a volatile function pointer as suggested by Alan Barrett.
Both do what the name says. For userland, both are prefixed by "__"
to keep them out of the user namespace.
Change some memset/memcmp uses to the new functions where it makes
sense -- these are just some examples, more to come.

branches: 1.37.2; 1.37.6; 1.37.8;
remove some DPRINTFs which are not just diagnostics but cause noise
even on regular operation

Make sure the mbufs in the input path (only the parts which we are going
to modify in the AH case) are writable/non-shared.
This addresses PR kern/33162 by Jeff Rizzo, and replaces the insufficient
patch from that time by a radical solution.
(The PR's problem had been worked around by rev.1.3 of xennetback_xenbus.c,
so it needs a network driver modification to reproduce it.)
Being here, clarify a bit of ipcomp -- uncompression is done in-place,
the header must be removed explicitly.

fix pointer/offset mistakes in handling of IPv4 options

add patch from Arnaud Degroote to handle IPv6 extended options with
(FAST_)IPSEC, tested lightly with a DSTOPTS header consisting
of PAD1

branches: 1.33.4; 1.33.8;
copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

As a first step towards more fine-grained locking, don't require
crypto_{new.free}session() to be called with the "crypto_mtx"
spinlock held.
This doesn't change much for now because these functions acquire
the said mutex first on entry now, but at least it keeps the nasty
locks local to the opencrypto core.

more "const"

sprinkle some "const", documenting that the SA is not supposed to
change during an xform operation

remove some unnecessary pointer typecasts
(one was wrong on BE systems, but was harmless here because the
result is effectively unused)

change locking order, to make sure the cpu is at splsoftnet()
before the softnet_lock (adaptive) mutex is acquired, from
Wolfgang Stukenbrock, should fix a recursive lock panic

-in opencrypto callbacks (which run in a kernel thread), pull softnet_lock
everywhere splsoftnet() was used before, to fix MP concurrency problems
-pull KERNEL_LOCK where ip(6)_output() is called, as this is what
the network stack (unfortunately) expects, in particular to avoid
races for packets in the interface send queues
From Wolfgang Stukenbrock per PR kern/44418, with the application
of KERNEL_LOCK to what I think are the essential points, tested
on a dual-core i386.

branches: 1.26.4; 1.26.6; 1.26.8;
Remove extra whitespace added by a stupid tool.
XXX: more in src/sys/arch

bcopy -> memcpy

bzero -> memset

bcmp -> memcmp

branches: 1.22.2;
kill MALLOC and FREE macros.

branches: 1.21.2; 1.21.10;
Make IPSEC and FAST_IPSEC stats per-cpu. Use <net/net_stats.h> and
netstat_sysctl().

branches: 1.20.6; 1.20.8;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

branches: 1.19.2;
The function ipsec4_get_ulp assumes that ip_off is in host order. This results
in IPsec processing that is dependent on protocol and/or port can be bypassed.

Bug report, analysis and initial fix from Karl Knutsson.
Final patch and ok from degroote@

branches: 1.18.6; 1.18.8; 1.18.12;
Add support for options IPSEC_NAT_T (RFC 3947 and 3948) for fast_ipsec(4).

No objection on tech-net@

Honor the ip4_ah_offsetmask bits (clear or not the ip->ip_off field for ah
processing).

Use ip4_ah_cleartos instead of ah_cleartos for consistency

branches: 1.15.2; 1.15.4; 1.15.6;
Remove useless cast
Use NULL instead of (void*) 0

Fix fallout from caddr_t changes

Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.12.2; 1.12.4; 1.12.8;
__unused removal on arguments; approved by core.

more __unused

branches: 1.10.8; 1.10.10;
Add two new sysctls protected under IPSEC_DEBUG:

net.inet.ipsec.test_replay - When set to 1, IPsec will send packets with
the same sequence number. This allows to verify if the other side
has proper replay attacks detection.

net.inet.ipsec.test_integrity - When set 1, IPsec will send packets with
corrupted HMAC. This allows to verify if the other side properly
detects modified packets.

(a message will be printed indicating when these sysctls changed)

By Pawel Jakub Dawidek <pjd@FreeBSD.org>.
Discussed with Christos Zoulas and Jonathan Stone.

branches: 1.9.4; 1.9.6; 1.9.8; 1.9.10; 1.9.12;
merge ktrace-lwp.

branches: 1.8.2; 1.8.4; 1.8.12; 1.8.14;
nuke trailing whitespace

branches: 1.7.2; 1.7.6; 1.7.8;
Commit an old diff for AH which has been in my personal tree since
August 2003:

On NetBSD, when we get to ah_massage_headers(), ip->ip_len is in
network byte order and includes all bytes in the input packet.
Therefore we don't need to byte-swap it or to add `skip' back in,
before verifying the receive-side hash.

With this change, AH transport mode works against FreeBSD 4.9 fast-ipsec
(which also works against Win2k, &c., &c.).

branches: 1.6.2;
sys/netinet6/ip6_ecn.h is reportedly a FreeBSD-ism; NetBSD has
prototypes for the IPv6 ECN ingress/egress functions in sys/netinet/ip_ecn.h,
inside an #ifdef INET6 wrapper. So, wrap sys/netipsec ocurrences of
#include <netinet6/ip6_ecn.h>
in #ifdef __FreeBSD__/#endif, until both camps can agree on this
teensy little piece of namespace. Affects:
ipsec_output.c xform_ah.c xform_esp.c xform_ipip.c

Cast an expression with sizeof() to long.

Reversion of "netkey merge", part 2 (replacement of removed files in the
repository by christos was part 1). netipsec should now be back as it
was on 2003-09-11, with some very minor changes:

1) Some residual platform-dependent code was moved from ipsec.h to
ipsec_osdep.h; without this, IPSEC_ASSERT() was multiply defined. ipsec.h
now includes ipsec_osdep.h

2) itojun's renaming of netipsec/files.ipsec to netipsec/files.netipsec has
been left in place (it's arguable which name is less confusing but the
rename is pretty harmless).

3) Some #endif TOKEN has been replaced by #endif /* TOKEN */; #endif TOKEN
is invalid and GCC 3 won't compile it.

An i386 kernel with "options FAST_IPSEC" and "options OPENCRYPTO" now
gets through "make depend" but fails to build with errors in ip_input.c.
But it's better than it was (thank heaven for small favors).

merge netipsec/key* into netkey/key*. no need for both.
change confusing filename

opt_inet6.h is FreeBSD-specific, so wrap it with #ifdef __FreeBSD__/#endif.

Initial import of Sam Leffler's `Fast-IPsec' from FreeBSD 4.
Fast-IPsec is a rework of the OpenBSD and KAME IPsec code, using the
OpenCryptoFramework (and thus hardware crypto accelerators) and
numerous detailed performance improvements.

This import is (aside from SPL-level names) the FreeBSD source,
imported ``as-is'' as a historical snapshot, for future maintenance
and comparison against the FreeBSD source. For now, several minor
kernel-API differences are hidden by macros a shim file, ipsec_osdep.h,
which (aside from SPL names) can be targeted at either NetBSD or FreeBSD.

Fix ipsecif(4) IPV6_MINMTU does not work correctly.

make DPRINTF use varyadic cpp macros, and merge with IPSECLOG.

Merge the [pgoyette-compat] branch

branches: 1.106.2;
Constify ipseczeroes, and remove one use of it.

Correctly handle the padding for IPv6-AH, as specified by RFC4302. Seen in
a FreeBSD bug report, by Jason Mader.

The RFC specifies that under IPv6 the complete AH header must be 64bit-
aligned, and under IPv4 32bit-aligned. That's a rule we've never respected.
The other BSDs and MacOS never have either.

So respect it now.

This makes it possible to set up IPv6-AH between Linux and NetBSD, and also
probably between Windows and NetBSD.

Until now all the tests I made were between two *BSD hosts, and everything
worked "correctly" since both hosts were speaking the same non-standard
AHv6, so they could understand each other.

Tested with Fedora<->NetBSD, hmac-sha2-384.

Introduce ah_authsiz, which computes the length of the ICV only. Use it in
esp_hdrsiz, and clarify.

Until now we were using ah_hdrsiz, and were relying on the fact that the
size of the AH header happens to be equal to that of the ESP trailer.

Now the size of the ESP trailer is added manually. This also fixes one
branch in esp_hdrsiz: we always append an ESP trailer, so it must always
be taken into account, and not just when an ICV is here.

Strengthen and simplify, once more.

Fix non-INET6 builds

IP6_EXTHDR_GET -> M_REGION_GET, no functional change.

Remove unused calls to nat_t_ports_get.

ENOBUFS -> EACCES when updating the replay counter.

Remove unused 'mp' argument from all the xf_output functions. Also clean
up xform.h a bit.

When IP6_EXTHDR_GET fails, return ENOBUFS, and don't log an error (HDROPS
is not supposed to be used here).

When the replay check fails, return EACCES instead of ENOBUFS.

Remove IPSEC_SPLASSERT_SOFTNET, it has always been a no-op.

Remove the M_AUTHIPDGM flag. It is equivalent to M_AUTHIPHDR, both
are set in IPsec-AH, and they are always handled together.

Remove the kernel RH0 code. RH0 is deprecated by RFC5095, for security
reasons. RH0 was already removed in the kernel's input path, but some
parts were still present in the output path: they are now removed.

Sent on tech-net@ a few days ago.

Remove extra long file paths from the headers.

cosmetic

Simplify the IPv4 parser. Get the option length in 'optlen', and sanitize
it earlier. A new check is added (off + optlen > skip).

In the IPv6 parser we reuse 'optlen', and remove 'ad' as a result.

Remove dead code.

ok ozaki-r@

Remove duplicate, to better show that this place doesn't make a lot of
sense. The code should probably be removed, it's a leftover from when we
had #ifdef __FreeBSD__.

branches: 1.87.2;
Reinforce this area, make sure the length field fits the option. Normally
it always does because the options were already sanitized earlier.

Add [ah/esp/ipcomp]_enable sysctls, and remove the FreeBSD #ifdefs.
Discussed with ozaki-r@.

Remove some more FreeBSD sysctl declarations that already have NetBSD
counterparts. Discussed with ozaki-r@.

Fix buffer overflow on sending an IPv6 packet with large options

If an IPv6 packet has large options, a necessary space for evacuation can
exceed the expected size (ah_pool_item_size). Give up using the pool_cache
if it happens.

Pointed out by maxv@

Commonalize error paths (NFC)

style

Make sure the Authentication Header fits the mbuf chain, otherwise panic.

Fix use-after-free, 'ah' may not be valid after m_makewritable and
ah_massage_headers.

Fix kernel panic (assertion failure) on receiving an IPv6 packet with large options

If an IPv6 packet has large options, a necessary space for evacuation can
exceed the expected size (ah_pool_item_size). Give up using the pool_cache
if it happens.

Pointed out by maxv@

Don't relook up an SP/SA in opencrpyto callbacks

We don't need to do so because we have a reference to it. And also
relooking-up one there may return an sp/sav that has different
parameters from an original one.

Reinforce and clarify.

Fix a vulnerability in IPsec-IPv6-AH, that allows an attacker to remotely
crash the kernel with a single packet.

In this loop we need to increment 'ad' by two, because the length field
of the option header does not count the size of the option header itself.

If the length is zero, then 'count' is incremented by zero, and there's
an infinite loop. Beyond that, this code was written with the assumption
that since the IPv6 packet already went through the generic IPv6 option
parser, several fields are guaranteed to be valid; but this assumption
does not hold because of the missing '+2', and there's as a result a
triggerable buffer overflow (write zeros after the end of the mbuf,
potentially to the next mbuf in memory since it's a pool).

Add the missing '+2', this place will be reinforced in separate commits.

Revert a part of rev1.49 (six months ago). The pointer given to memcpy
was correct.

Discussed with Christos and Ryota.

Constify isr at many places (NFC)

Use pool_cache(9) instead of pool(9) for tdb_crypto objects

The change improves network throughput especially on multi-core systems.

MP-ify SAD (savlist)

localcount(9) is used to protect savlist of sah. The basic design is
similar to MP-ifications of SPD and SAD sahlist. Please read the
locking notes of SAD for more details.

Introduce KEY_SA_UNREF and replace KEY_FREESAV with it where sav will never be actually freed in the future

KEY_SA_UNREF is still key_freesav so no functional change for now.

This change reduces diff of further changes.

Make IPsec SPD MP-safe

We use localcount(9), not psref(9), to make the sptree and secpolicy (SP)
entries MP-safe because SPs need to be referenced over opencrypto
processing that executes a callback in a different context.

SPs on sockets aren't managed by the sptree and can be destroyed in softint.
localcount_drain cannot be used in softint so we delay the destruction of
such SPs to a thread context. To do so, a list to manage such SPs is added
(key_socksplist) and key_timehandler_spd deletes dead SPs in the list.

For more details please read the locking notes in key.c.

Proposed on tech-kern@ and tech-net@

Don't acquire global locks for IPsec if NET_MPSAFE

Note that the change is just to make testing easy and IPsec isn't MP-safe yet.

Use pool to allocate tdb_crypto

For ESP and AH, we need to allocate an extra variable space in addition
to struct tdb_crypto. The fixed size of pool items may be larger than
an actual requisite size of a buffer, but still the performance
improvement by replacing malloc with pool wins.

Dedup error paths (NFC)

Fix a debug message

Hold a reference to an SP during opencrypto processing

An SP has a list of isr (ipsecrequest) that represents a sequence
of IPsec encryption/authentication processing. One isr corresponds
to one opencrypto processing. The lifetime of an isr follows its SP.

We pass an isr to a callback function of opencrypto to continue
to a next encryption/authentication processing. However nobody
guaranteed that the isr wasn't freed, i.e., its SP wasn't destroyed.

In order to avoid such unexpected destruction of isr, hold a reference
to its SP during opencrypto processing.

Don't bother the case of crp->crp_buf == NULL in callbacks

Don't release sav if calling crypto_dispatch again

branches: 1.62.2;
Calculate ah_max_authsize on initialization as well as esp_max_ivlen

Prepare to stop using isr->sav

isr is a shared resource and using isr->sav as a temporal storage
for each packet processing is racy. And also having a reference from
isr to sav makes the lifetime of sav non-deterministic; such a reference
is removed when a packet is processed and isr->sav is overwritten by
new one. Let's have a sav locally for each packet processing instead of
using shared isr->sav.

However this change doesn't stop using isr->sav yet because there are
some users of isr->sav. isr->sav will be removed after the users find
a way to not use isr->sav.

Pass sav directly to opencrypto callback

In a callback, use a passed sav as-is by default and look up a sav
only if the passed sav is dead.

Fix header size calculation of ah in the case sav is NULL

This fix was also needed for esp.

Use explicit_memset to surely zero-clear key_auth and key_enc

Rename key_alloc* functions (NFC)

We shouldn't use the term "alloc" for functions that just look up
data and actually don't allocate memory.

Remove codes for PACKET_TAG_IPSEC_IN_CRYPTO_DONE

It seems that PACKET_TAG_IPSEC_IN_CRYPTO_DONE is for network adapters
that have IPsec accelerators; a driver sets the mtag to a packet
when its device has already encrypted the packet.

Unfortunately no driver implements such offload features for long
years and seems unlikely to implement them soon. (Note that neither
FreeBSD nor Linux doesn't have such drivers.) Let's remove related
(unused) codes and simplify the IPsec code.

Apply C99-style struct initialization to xformsw

branches: 1.54.2;
Make ipsec_address() and ipsec_logsastr() mpsafe.

branches: 1.53.2;
Retire ipsec_osdep.h

We don't need to care other OSes (FreeBSD) anymore.

Some macros are alive in ipsec_private.h.

Convert IPSEC_ASSERT to KASSERT or KASSERTMSG

IPSEC_ASSERT just discarded specified message...

Remove __FreeBSD__ and __NetBSD__ switches

No functional changes (except for a debug printf).

Note that there remain some __FreeBSD__ for sysctl knobs which counerparts
to NetBSD don't exist. And ipsec_osdep.h isn't touched yet; tidying it up
requires actual code changes.

cosmetic fixes:
- __func__ in printfs
- no space after sizeof
- eliminate useless casts
- u_intX_t -> uintX_t

- fix old refactoring which zeroed the wrong part of the buffer.
- simplify.

change into __func__

Redo the statistics through an indirection array and put the definitions
of the arrays in pfkeyv2.h so that they are next to the index definitions.
Remove "bogus" comment about compressing the statistics which is now fixed.

Fix that ah_algorithm_lookup and esp_algorithm_lookup don't handle some algorithms

Unrelated upper limit values, AH_ALG_MAX and ESP_ALG_MAX, prevented some
algorithms from being looked up.

Prepare netipsec for rump-ification

- Include "opt_*.h" only if _KERNEL_OPT is defined
- Allow encapinit to be called twice (by ifinit and ipe4_attach)
- ifinit didn't call encapinit if IPSEC is enabled (ipe4_attach called
it instead), however, on a rump kernel ipe4_attach may not be called
even if IPSEC is enabled. So we need to allow ifinit to call it anyway
- Setup sysctls in ipsec_attach explicitly instead of using SYSCTL_SETUP
- Call ip6flow_invalidate_all in key_spdadd only if in6_present
- It's possible that a rump kernel loads the ipsec library but not
the inet6 library

branches: 1.44.2; 1.44.4;
Tidy up opt_ipsec.h inclusions

Some inclusions of opt_ipsec.h were for IPSEC_NAT_T and are now unnecessary.
Add inclusions to some C files for IPSEC_DEBUG.

KNF

branches: 1.42.4; 1.42.6; 1.42.8; 1.42.12;
- apply some __diagused
- remove unused variables
- move some variables inside their relevant use #ifdef

Fix sense of consttime_memequal and update all callers.

Now it returns true (nonzero) to mean equal and false (zero) to mean
inequal, as the name suggests.

As promised on tech-userlevel back in June:

https://mail-index.netbsd.org/tech-userlevel/2013/06/24/msg007843.html

branches: 1.40.2;
Replace consttime_bcmp/explicit_bzero by consttime_memequal/explicit_memset.

consttime_memequal is the same as the old consttime_bcmp.
explicit_memset is to memset as explicit_bzero was to bcmp.

Passes amd64 release and i386/ALL, but I'm sure I missed some spots,
so please let me know.

PR/47886: Dr. Wolfgang Stukenbrock: IPSEC_NAT_T enabled kernels may access
outdated pointers and pass ESP data to UPD-sockets.
While here, simplify the code and remove the IPSEC_NAT_T option; always
compile nat-traversal in so that it does not bitrot.

branches: 1.38.2;
Add "consttime_bcmp" and "explicit_bzero" functions for both kernel
abd userland, as proposed on tech-security, with explicit_bzero using
a volatile function pointer as suggested by Alan Barrett.
Both do what the name says. For userland, both are prefixed by "__"
to keep them out of the user namespace.
Change some memset/memcmp uses to the new functions where it makes
sense -- these are just some examples, more to come.

branches: 1.37.2; 1.37.6; 1.37.8;
remove some DPRINTFs which are not just diagnostics but cause noise
even on regular operation

Make sure the mbufs in the input path (only the parts which we are going
to modify in the AH case) are writable/non-shared.
This addresses PR kern/33162 by Jeff Rizzo, and replaces the insufficient
patch from that time by a radical solution.
(The PR's problem had been worked around by rev.1.3 of xennetback_xenbus.c,
so it needs a network driver modification to reproduce it.)
Being here, clarify a bit of ipcomp -- uncompression is done in-place,
the header must be removed explicitly.

fix pointer/offset mistakes in handling of IPv4 options

add patch from Arnaud Degroote to handle IPv6 extended options with
(FAST_)IPSEC, tested lightly with a DSTOPTS header consisting
of PAD1

branches: 1.33.4; 1.33.8;
copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

As a first step towards more fine-grained locking, don't require
crypto_{new.free}session() to be called with the "crypto_mtx"
spinlock held.
This doesn't change much for now because these functions acquire
the said mutex first on entry now, but at least it keeps the nasty
locks local to the opencrypto core.

more "const"

sprinkle some "const", documenting that the SA is not supposed to
change during an xform operation

remove some unnecessary pointer typecasts
(one was wrong on BE systems, but was harmless here because the
result is effectively unused)

change locking order, to make sure the cpu is at splsoftnet()
before the softnet_lock (adaptive) mutex is acquired, from
Wolfgang Stukenbrock, should fix a recursive lock panic

-in opencrypto callbacks (which run in a kernel thread), pull softnet_lock
everywhere splsoftnet() was used before, to fix MP concurrency problems
-pull KERNEL_LOCK where ip(6)_output() is called, as this is what
the network stack (unfortunately) expects, in particular to avoid
races for packets in the interface send queues
From Wolfgang Stukenbrock per PR kern/44418, with the application
of KERNEL_LOCK to what I think are the essential points, tested
on a dual-core i386.

branches: 1.26.4; 1.26.6; 1.26.8;
Remove extra whitespace added by a stupid tool.
XXX: more in src/sys/arch

bcopy -> memcpy

bzero -> memset

bcmp -> memcmp

branches: 1.22.2;
kill MALLOC and FREE macros.

branches: 1.21.2; 1.21.10;
Make IPSEC and FAST_IPSEC stats per-cpu. Use <net/net_stats.h> and
netstat_sysctl().

branches: 1.20.6; 1.20.8;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

branches: 1.19.2;
The function ipsec4_get_ulp assumes that ip_off is in host order. This results
in IPsec processing that is dependent on protocol and/or port can be bypassed.

Bug report, analysis and initial fix from Karl Knutsson.
Final patch and ok from degroote@

branches: 1.18.6; 1.18.8; 1.18.12;
Add support for options IPSEC_NAT_T (RFC 3947 and 3948) for fast_ipsec(4).

No objection on tech-net@

Honor the ip4_ah_offsetmask bits (clear or not the ip->ip_off field for ah
processing).

Use ip4_ah_cleartos instead of ah_cleartos for consistency

branches: 1.15.2; 1.15.4; 1.15.6;
Remove useless cast
Use NULL instead of (void*) 0

Fix fallout from caddr_t changes

Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.12.2; 1.12.4; 1.12.8;
__unused removal on arguments; approved by core.

more __unused

branches: 1.10.8; 1.10.10;
Add two new sysctls protected under IPSEC_DEBUG:

net.inet.ipsec.test_replay - When set to 1, IPsec will send packets with
the same sequence number. This allows to verify if the other side
has proper replay attacks detection.

net.inet.ipsec.test_integrity - When set 1, IPsec will send packets with
corrupted HMAC. This allows to verify if the other side properly
detects modified packets.

(a message will be printed indicating when these sysctls changed)

By Pawel Jakub Dawidek <pjd@FreeBSD.org>.
Discussed with Christos Zoulas and Jonathan Stone.

branches: 1.9.4; 1.9.6; 1.9.8; 1.9.10; 1.9.12;
merge ktrace-lwp.

branches: 1.8.2; 1.8.4; 1.8.12; 1.8.14;
nuke trailing whitespace

branches: 1.7.2; 1.7.6; 1.7.8;
Commit an old diff for AH which has been in my personal tree since
August 2003:

On NetBSD, when we get to ah_massage_headers(), ip->ip_len is in
network byte order and includes all bytes in the input packet.
Therefore we don't need to byte-swap it or to add `skip' back in,
before verifying the receive-side hash.

With this change, AH transport mode works against FreeBSD 4.9 fast-ipsec
(which also works against Win2k, &c., &c.).

branches: 1.6.2;
sys/netinet6/ip6_ecn.h is reportedly a FreeBSD-ism; NetBSD has
prototypes for the IPv6 ECN ingress/egress functions in sys/netinet/ip_ecn.h,
inside an #ifdef INET6 wrapper. So, wrap sys/netipsec ocurrences of
#include <netinet6/ip6_ecn.h>
in #ifdef __FreeBSD__/#endif, until both camps can agree on this
teensy little piece of namespace. Affects:
ipsec_output.c xform_ah.c xform_esp.c xform_ipip.c

Cast an expression with sizeof() to long.

Reversion of "netkey merge", part 2 (replacement of removed files in the
repository by christos was part 1). netipsec should now be back as it
was on 2003-09-11, with some very minor changes:

1) Some residual platform-dependent code was moved from ipsec.h to
ipsec_osdep.h; without this, IPSEC_ASSERT() was multiply defined. ipsec.h
now includes ipsec_osdep.h

2) itojun's renaming of netipsec/files.ipsec to netipsec/files.netipsec has
been left in place (it's arguable which name is less confusing but the
rename is pretty harmless).

3) Some #endif TOKEN has been replaced by #endif /* TOKEN */; #endif TOKEN
is invalid and GCC 3 won't compile it.

An i386 kernel with "options FAST_IPSEC" and "options OPENCRYPTO" now
gets through "make depend" but fails to build with errors in ip_input.c.
But it's better than it was (thank heaven for small favors).

merge netipsec/key* into netkey/key*. no need for both.
change confusing filename

opt_inet6.h is FreeBSD-specific, so wrap it with #ifdef __FreeBSD__/#endif.

Initial import of Sam Leffler's `Fast-IPsec' from FreeBSD 4.
Fast-IPsec is a rework of the OpenBSD and KAME IPsec code, using the
OpenCryptoFramework (and thus hardware crypto accelerators) and
numerous detailed performance improvements.

This import is (aside from SPL-level names) the FreeBSD source,
imported ``as-is'' as a historical snapshot, for future maintenance
and comparison against the FreeBSD source. For now, several minor
kernel-API differences are hidden by macros a shim file, ipsec_osdep.h,
which (aside from SPL names) can be targeted at either NetBSD or FreeBSD.

make DPRINTF use varyadic cpp macros, and merge with IPSECLOG.

Merge the [pgoyette-compat] branch

branches: 1.106.2;
Constify ipseczeroes, and remove one use of it.

Correctly handle the padding for IPv6-AH, as specified by RFC4302. Seen in
a FreeBSD bug report, by Jason Mader.

The RFC specifies that under IPv6 the complete AH header must be 64bit-
aligned, and under IPv4 32bit-aligned. That's a rule we've never respected.
The other BSDs and MacOS never have either.

So respect it now.

This makes it possible to set up IPv6-AH between Linux and NetBSD, and also
probably between Windows and NetBSD.

Until now all the tests I made were between two *BSD hosts, and everything
worked "correctly" since both hosts were speaking the same non-standard
AHv6, so they could understand each other.

Tested with Fedora<->NetBSD, hmac-sha2-384.

Introduce ah_authsiz, which computes the length of the ICV only. Use it in
esp_hdrsiz, and clarify.

Until now we were using ah_hdrsiz, and were relying on the fact that the
size of the AH header happens to be equal to that of the ESP trailer.

Now the size of the ESP trailer is added manually. This also fixes one
branch in esp_hdrsiz: we always append an ESP trailer, so it must always
be taken into account, and not just when an ICV is here.

Strengthen and simplify, once more.

Fix non-INET6 builds

IP6_EXTHDR_GET -> M_REGION_GET, no functional change.

Remove unused calls to nat_t_ports_get.

ENOBUFS -> EACCES when updating the replay counter.

Remove unused 'mp' argument from all the xf_output functions. Also clean
up xform.h a bit.

When IP6_EXTHDR_GET fails, return ENOBUFS, and don't log an error (HDROPS
is not supposed to be used here).

When the replay check fails, return EACCES instead of ENOBUFS.

Remove IPSEC_SPLASSERT_SOFTNET, it has always been a no-op.

Remove the M_AUTHIPDGM flag. It is equivalent to M_AUTHIPHDR, both
are set in IPsec-AH, and they are always handled together.

Remove the kernel RH0 code. RH0 is deprecated by RFC5095, for security
reasons. RH0 was already removed in the kernel's input path, but some
parts were still present in the output path: they are now removed.

Sent on tech-net@ a few days ago.

Remove extra long file paths from the headers.

cosmetic

Simplify the IPv4 parser. Get the option length in 'optlen', and sanitize
it earlier. A new check is added (off + optlen > skip).

In the IPv6 parser we reuse 'optlen', and remove 'ad' as a result.

Remove dead code.

ok ozaki-r@

Remove duplicate, to better show that this place doesn't make a lot of
sense. The code should probably be removed, it's a leftover from when we
had #ifdef __FreeBSD__.

branches: 1.87.2;
Reinforce this area, make sure the length field fits the option. Normally
it always does because the options were already sanitized earlier.

Add [ah/esp/ipcomp]_enable sysctls, and remove the FreeBSD #ifdefs.
Discussed with ozaki-r@.

Remove some more FreeBSD sysctl declarations that already have NetBSD
counterparts. Discussed with ozaki-r@.

Fix buffer overflow on sending an IPv6 packet with large options

If an IPv6 packet has large options, a necessary space for evacuation can
exceed the expected size (ah_pool_item_size). Give up using the pool_cache
if it happens.

Pointed out by maxv@

Commonalize error paths (NFC)

style

Make sure the Authentication Header fits the mbuf chain, otherwise panic.

Fix use-after-free, 'ah' may not be valid after m_makewritable and
ah_massage_headers.

Fix kernel panic (assertion failure) on receiving an IPv6 packet with large options

If an IPv6 packet has large options, a necessary space for evacuation can
exceed the expected size (ah_pool_item_size). Give up using the pool_cache
if it happens.

Pointed out by maxv@

Don't relook up an SP/SA in opencrpyto callbacks

We don't need to do so because we have a reference to it. And also
relooking-up one there may return an sp/sav that has different
parameters from an original one.

Reinforce and clarify.

Fix a vulnerability in IPsec-IPv6-AH, that allows an attacker to remotely
crash the kernel with a single packet.

In this loop we need to increment 'ad' by two, because the length field
of the option header does not count the size of the option header itself.

If the length is zero, then 'count' is incremented by zero, and there's
an infinite loop. Beyond that, this code was written with the assumption
that since the IPv6 packet already went through the generic IPv6 option
parser, several fields are guaranteed to be valid; but this assumption
does not hold because of the missing '+2', and there's as a result a
triggerable buffer overflow (write zeros after the end of the mbuf,
potentially to the next mbuf in memory since it's a pool).

Add the missing '+2', this place will be reinforced in separate commits.

Revert a part of rev1.49 (six months ago). The pointer given to memcpy
was correct.

Discussed with Christos and Ryota.

Constify isr at many places (NFC)

Use pool_cache(9) instead of pool(9) for tdb_crypto objects

The change improves network throughput especially on multi-core systems.

MP-ify SAD (savlist)

localcount(9) is used to protect savlist of sah. The basic design is
similar to MP-ifications of SPD and SAD sahlist. Please read the
locking notes of SAD for more details.

Introduce KEY_SA_UNREF and replace KEY_FREESAV with it where sav will never be actually freed in the future

KEY_SA_UNREF is still key_freesav so no functional change for now.

This change reduces diff of further changes.

Make IPsec SPD MP-safe

We use localcount(9), not psref(9), to make the sptree and secpolicy (SP)
entries MP-safe because SPs need to be referenced over opencrypto
processing that executes a callback in a different context.

SPs on sockets aren't managed by the sptree and can be destroyed in softint.
localcount_drain cannot be used in softint so we delay the destruction of
such SPs to a thread context. To do so, a list to manage such SPs is added
(key_socksplist) and key_timehandler_spd deletes dead SPs in the list.

For more details please read the locking notes in key.c.

Proposed on tech-kern@ and tech-net@

Don't acquire global locks for IPsec if NET_MPSAFE

Note that the change is just to make testing easy and IPsec isn't MP-safe yet.

Use pool to allocate tdb_crypto

For ESP and AH, we need to allocate an extra variable space in addition
to struct tdb_crypto. The fixed size of pool items may be larger than
an actual requisite size of a buffer, but still the performance
improvement by replacing malloc with pool wins.

Dedup error paths (NFC)

Fix a debug message

Hold a reference to an SP during opencrypto processing

An SP has a list of isr (ipsecrequest) that represents a sequence
of IPsec encryption/authentication processing. One isr corresponds
to one opencrypto processing. The lifetime of an isr follows its SP.

We pass an isr to a callback function of opencrypto to continue
to a next encryption/authentication processing. However nobody
guaranteed that the isr wasn't freed, i.e., its SP wasn't destroyed.

In order to avoid such unexpected destruction of isr, hold a reference
to its SP during opencrypto processing.

Don't bother the case of crp->crp_buf == NULL in callbacks

Don't release sav if calling crypto_dispatch again

branches: 1.62.2;
Calculate ah_max_authsize on initialization as well as esp_max_ivlen

Prepare to stop using isr->sav

isr is a shared resource and using isr->sav as a temporal storage
for each packet processing is racy. And also having a reference from
isr to sav makes the lifetime of sav non-deterministic; such a reference
is removed when a packet is processed and isr->sav is overwritten by
new one. Let's have a sav locally for each packet processing instead of
using shared isr->sav.

However this change doesn't stop using isr->sav yet because there are
some users of isr->sav. isr->sav will be removed after the users find
a way to not use isr->sav.

Pass sav directly to opencrypto callback

In a callback, use a passed sav as-is by default and look up a sav
only if the passed sav is dead.

Fix header size calculation of ah in the case sav is NULL

This fix was also needed for esp.

Use explicit_memset to surely zero-clear key_auth and key_enc

Rename key_alloc* functions (NFC)

We shouldn't use the term "alloc" for functions that just look up
data and actually don't allocate memory.

Remove codes for PACKET_TAG_IPSEC_IN_CRYPTO_DONE

It seems that PACKET_TAG_IPSEC_IN_CRYPTO_DONE is for network adapters
that have IPsec accelerators; a driver sets the mtag to a packet
when its device has already encrypted the packet.

Unfortunately no driver implements such offload features for long
years and seems unlikely to implement them soon. (Note that neither
FreeBSD nor Linux doesn't have such drivers.) Let's remove related
(unused) codes and simplify the IPsec code.

Apply C99-style struct initialization to xformsw

branches: 1.54.2;
Make ipsec_address() and ipsec_logsastr() mpsafe.

branches: 1.53.2;
Retire ipsec_osdep.h

We don't need to care other OSes (FreeBSD) anymore.

Some macros are alive in ipsec_private.h.

Convert IPSEC_ASSERT to KASSERT or KASSERTMSG

IPSEC_ASSERT just discarded specified message...

Remove __FreeBSD__ and __NetBSD__ switches

No functional changes (except for a debug printf).

Note that there remain some __FreeBSD__ for sysctl knobs which counerparts
to NetBSD don't exist. And ipsec_osdep.h isn't touched yet; tidying it up
requires actual code changes.

cosmetic fixes:
- __func__ in printfs
- no space after sizeof
- eliminate useless casts
- u_intX_t -> uintX_t

- fix old refactoring which zeroed the wrong part of the buffer.
- simplify.

change into __func__

Redo the statistics through an indirection array and put the definitions
of the arrays in pfkeyv2.h so that they are next to the index definitions.
Remove "bogus" comment about compressing the statistics which is now fixed.

Fix that ah_algorithm_lookup and esp_algorithm_lookup don't handle some algorithms

Unrelated upper limit values, AH_ALG_MAX and ESP_ALG_MAX, prevented some
algorithms from being looked up.

Prepare netipsec for rump-ification

- Include "opt_*.h" only if _KERNEL_OPT is defined
- Allow encapinit to be called twice (by ifinit and ipe4_attach)
- ifinit didn't call encapinit if IPSEC is enabled (ipe4_attach called
it instead), however, on a rump kernel ipe4_attach may not be called
even if IPSEC is enabled. So we need to allow ifinit to call it anyway
- Setup sysctls in ipsec_attach explicitly instead of using SYSCTL_SETUP
- Call ip6flow_invalidate_all in key_spdadd only if in6_present
- It's possible that a rump kernel loads the ipsec library but not
the inet6 library

branches: 1.44.2; 1.44.4;
Tidy up opt_ipsec.h inclusions

Some inclusions of opt_ipsec.h were for IPSEC_NAT_T and are now unnecessary.
Add inclusions to some C files for IPSEC_DEBUG.

KNF

branches: 1.42.4; 1.42.6; 1.42.8; 1.42.12;
- apply some __diagused
- remove unused variables
- move some variables inside their relevant use #ifdef

Fix sense of consttime_memequal and update all callers.

Now it returns true (nonzero) to mean equal and false (zero) to mean
inequal, as the name suggests.

As promised on tech-userlevel back in June:

https://mail-index.netbsd.org/tech-userlevel/2013/06/24/msg007843.html

branches: 1.40.2;
Replace consttime_bcmp/explicit_bzero by consttime_memequal/explicit_memset.

consttime_memequal is the same as the old consttime_bcmp.
explicit_memset is to memset as explicit_bzero was to bcmp.

Passes amd64 release and i386/ALL, but I'm sure I missed some spots,
so please let me know.

PR/47886: Dr. Wolfgang Stukenbrock: IPSEC_NAT_T enabled kernels may access
outdated pointers and pass ESP data to UPD-sockets.
While here, simplify the code and remove the IPSEC_NAT_T option; always
compile nat-traversal in so that it does not bitrot.

branches: 1.38.2;
Add "consttime_bcmp" and "explicit_bzero" functions for both kernel
abd userland, as proposed on tech-security, with explicit_bzero using
a volatile function pointer as suggested by Alan Barrett.
Both do what the name says. For userland, both are prefixed by "__"
to keep them out of the user namespace.
Change some memset/memcmp uses to the new functions where it makes
sense -- these are just some examples, more to come.

branches: 1.37.2; 1.37.6; 1.37.8;
remove some DPRINTFs which are not just diagnostics but cause noise
even on regular operation

Make sure the mbufs in the input path (only the parts which we are going
to modify in the AH case) are writable/non-shared.
This addresses PR kern/33162 by Jeff Rizzo, and replaces the insufficient
patch from that time by a radical solution.
(The PR's problem had been worked around by rev.1.3 of xennetback_xenbus.c,
so it needs a network driver modification to reproduce it.)
Being here, clarify a bit of ipcomp -- uncompression is done in-place,
the header must be removed explicitly.

fix pointer/offset mistakes in handling of IPv4 options

add patch from Arnaud Degroote to handle IPv6 extended options with
(FAST_)IPSEC, tested lightly with a DSTOPTS header consisting
of PAD1

branches: 1.33.4; 1.33.8;
copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

As a first step towards more fine-grained locking, don't require
crypto_{new.free}session() to be called with the "crypto_mtx"
spinlock held.
This doesn't change much for now because these functions acquire
the said mutex first on entry now, but at least it keeps the nasty
locks local to the opencrypto core.

more "const"

sprinkle some "const", documenting that the SA is not supposed to
change during an xform operation

remove some unnecessary pointer typecasts
(one was wrong on BE systems, but was harmless here because the
result is effectively unused)

change locking order, to make sure the cpu is at splsoftnet()
before the softnet_lock (adaptive) mutex is acquired, from
Wolfgang Stukenbrock, should fix a recursive lock panic

-in opencrypto callbacks (which run in a kernel thread), pull softnet_lock
everywhere splsoftnet() was used before, to fix MP concurrency problems
-pull KERNEL_LOCK where ip(6)_output() is called, as this is what
the network stack (unfortunately) expects, in particular to avoid
races for packets in the interface send queues
From Wolfgang Stukenbrock per PR kern/44418, with the application
of KERNEL_LOCK to what I think are the essential points, tested
on a dual-core i386.

branches: 1.26.4; 1.26.6; 1.26.8;
Remove extra whitespace added by a stupid tool.
XXX: more in src/sys/arch

bcopy -> memcpy

bzero -> memset

bcmp -> memcmp

branches: 1.22.2;
kill MALLOC and FREE macros.

branches: 1.21.2; 1.21.10;
Make IPSEC and FAST_IPSEC stats per-cpu. Use <net/net_stats.h> and
netstat_sysctl().

branches: 1.20.6; 1.20.8;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

branches: 1.19.2;
The function ipsec4_get_ulp assumes that ip_off is in host order. This results
in IPsec processing that is dependent on protocol and/or port can be bypassed.

Bug report, analysis and initial fix from Karl Knutsson.
Final patch and ok from degroote@

branches: 1.18.6; 1.18.8; 1.18.12;
Add support for options IPSEC_NAT_T (RFC 3947 and 3948) for fast_ipsec(4).

No objection on tech-net@

Honor the ip4_ah_offsetmask bits (clear or not the ip->ip_off field for ah
processing).

Use ip4_ah_cleartos instead of ah_cleartos for consistency

branches: 1.15.2; 1.15.4; 1.15.6;
Remove useless cast
Use NULL instead of (void*) 0

Fix fallout from caddr_t changes

Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.12.2; 1.12.4; 1.12.8;
__unused removal on arguments; approved by core.

more __unused

branches: 1.10.8; 1.10.10;
Add two new sysctls protected under IPSEC_DEBUG:

net.inet.ipsec.test_replay - When set to 1, IPsec will send packets with
the same sequence number. This allows to verify if the other side
has proper replay attacks detection.

net.inet.ipsec.test_integrity - When set 1, IPsec will send packets with
corrupted HMAC. This allows to verify if the other side properly
detects modified packets.

(a message will be printed indicating when these sysctls changed)

By Pawel Jakub Dawidek <pjd@FreeBSD.org>.
Discussed with Christos Zoulas and Jonathan Stone.

branches: 1.9.4; 1.9.6; 1.9.8; 1.9.10; 1.9.12;
merge ktrace-lwp.

branches: 1.8.2; 1.8.4; 1.8.12; 1.8.14;
nuke trailing whitespace

branches: 1.7.2; 1.7.6; 1.7.8;
Commit an old diff for AH which has been in my personal tree since
August 2003:

On NetBSD, when we get to ah_massage_headers(), ip->ip_len is in
network byte order and includes all bytes in the input packet.
Therefore we don't need to byte-swap it or to add `skip' back in,
before verifying the receive-side hash.

With this change, AH transport mode works against FreeBSD 4.9 fast-ipsec
(which also works against Win2k, &c., &c.).

branches: 1.6.2;
sys/netinet6/ip6_ecn.h is reportedly a FreeBSD-ism; NetBSD has
prototypes for the IPv6 ECN ingress/egress functions in sys/netinet/ip_ecn.h,
inside an #ifdef INET6 wrapper. So, wrap sys/netipsec ocurrences of
#include <netinet6/ip6_ecn.h>
in #ifdef __FreeBSD__/#endif, until both camps can agree on this
teensy little piece of namespace. Affects:
ipsec_output.c xform_ah.c xform_esp.c xform_ipip.c

Cast an expression with sizeof() to long.

Reversion of "netkey merge", part 2 (replacement of removed files in the
repository by christos was part 1). netipsec should now be back as it
was on 2003-09-11, with some very minor changes:

1) Some residual platform-dependent code was moved from ipsec.h to
ipsec_osdep.h; without this, IPSEC_ASSERT() was multiply defined. ipsec.h
now includes ipsec_osdep.h

2) itojun's renaming of netipsec/files.ipsec to netipsec/files.netipsec has
been left in place (it's arguable which name is less confusing but the
rename is pretty harmless).

3) Some #endif TOKEN has been replaced by #endif /* TOKEN */; #endif TOKEN
is invalid and GCC 3 won't compile it.

An i386 kernel with "options FAST_IPSEC" and "options OPENCRYPTO" now
gets through "make depend" but fails to build with errors in ip_input.c.
But it's better than it was (thank heaven for small favors).

merge netipsec/key* into netkey/key*. no need for both.
change confusing filename

opt_inet6.h is FreeBSD-specific, so wrap it with #ifdef __FreeBSD__/#endif.

Initial import of Sam Leffler's `Fast-IPsec' from FreeBSD 4.
Fast-IPsec is a rework of the OpenBSD and KAME IPsec code, using the
OpenCryptoFramework (and thus hardware crypto accelerators) and
numerous detailed performance improvements.

This import is (aside from SPL-level names) the FreeBSD source,
imported ``as-is'' as a historical snapshot, for future maintenance
and comparison against the FreeBSD source. For now, several minor
kernel-API differences are hidden by macros a shim file, ipsec_osdep.h,
which (aside from SPL names) can be targeted at either NetBSD or FreeBSD.

Merge the [pgoyette-compat] branch

Constify ipseczeroes, and remove one use of it.

Correctly handle the padding for IPv6-AH, as specified by RFC4302. Seen in
a FreeBSD bug report, by Jason Mader.

The RFC specifies that under IPv6 the complete AH header must be 64bit-
aligned, and under IPv4 32bit-aligned. That's a rule we've never respected.
The other BSDs and MacOS never have either.

So respect it now.

This makes it possible to set up IPv6-AH between Linux and NetBSD, and also
probably between Windows and NetBSD.

Until now all the tests I made were between two *BSD hosts, and everything
worked "correctly" since both hosts were speaking the same non-standard
AHv6, so they could understand each other.

Tested with Fedora<->NetBSD, hmac-sha2-384.

Introduce ah_authsiz, which computes the length of the ICV only. Use it in
esp_hdrsiz, and clarify.

Until now we were using ah_hdrsiz, and were relying on the fact that the
size of the AH header happens to be equal to that of the ESP trailer.

Now the size of the ESP trailer is added manually. This also fixes one
branch in esp_hdrsiz: we always append an ESP trailer, so it must always
be taken into account, and not just when an ICV is here.

Strengthen and simplify, once more.

Fix non-INET6 builds

IP6_EXTHDR_GET -> M_REGION_GET, no functional change.

Remove unused calls to nat_t_ports_get.

ENOBUFS -> EACCES when updating the replay counter.

Remove unused 'mp' argument from all the xf_output functions. Also clean
up xform.h a bit.

When IP6_EXTHDR_GET fails, return ENOBUFS, and don't log an error (HDROPS
is not supposed to be used here).

When the replay check fails, return EACCES instead of ENOBUFS.

Remove IPSEC_SPLASSERT_SOFTNET, it has always been a no-op.

Remove the M_AUTHIPDGM flag. It is equivalent to M_AUTHIPHDR, both
are set in IPsec-AH, and they are always handled together.

Remove the kernel RH0 code. RH0 is deprecated by RFC5095, for security
reasons. RH0 was already removed in the kernel's input path, but some
parts were still present in the output path: they are now removed.

Sent on tech-net@ a few days ago.

Remove extra long file paths from the headers.

cosmetic

Simplify the IPv4 parser. Get the option length in 'optlen', and sanitize
it earlier. A new check is added (off + optlen > skip).

In the IPv6 parser we reuse 'optlen', and remove 'ad' as a result.

Remove dead code.

ok ozaki-r@

Remove duplicate, to better show that this place doesn't make a lot of
sense. The code should probably be removed, it's a leftover from when we
had #ifdef __FreeBSD__.

branches: 1.87.2;
Reinforce this area, make sure the length field fits the option. Normally
it always does because the options were already sanitized earlier.

Add [ah/esp/ipcomp]_enable sysctls, and remove the FreeBSD #ifdefs.
Discussed with ozaki-r@.

Remove some more FreeBSD sysctl declarations that already have NetBSD
counterparts. Discussed with ozaki-r@.

Fix buffer overflow on sending an IPv6 packet with large options

If an IPv6 packet has large options, a necessary space for evacuation can
exceed the expected size (ah_pool_item_size). Give up using the pool_cache
if it happens.

Pointed out by maxv@

Commonalize error paths (NFC)

style

Make sure the Authentication Header fits the mbuf chain, otherwise panic.

Fix use-after-free, 'ah' may not be valid after m_makewritable and
ah_massage_headers.

Fix kernel panic (assertion failure) on receiving an IPv6 packet with large options

If an IPv6 packet has large options, a necessary space for evacuation can
exceed the expected size (ah_pool_item_size). Give up using the pool_cache
if it happens.

Pointed out by maxv@

Don't relook up an SP/SA in opencrpyto callbacks

We don't need to do so because we have a reference to it. And also
relooking-up one there may return an sp/sav that has different
parameters from an original one.

Reinforce and clarify.

Fix a vulnerability in IPsec-IPv6-AH, that allows an attacker to remotely
crash the kernel with a single packet.

In this loop we need to increment 'ad' by two, because the length field
of the option header does not count the size of the option header itself.

If the length is zero, then 'count' is incremented by zero, and there's
an infinite loop. Beyond that, this code was written with the assumption
that since the IPv6 packet already went through the generic IPv6 option
parser, several fields are guaranteed to be valid; but this assumption
does not hold because of the missing '+2', and there's as a result a
triggerable buffer overflow (write zeros after the end of the mbuf,
potentially to the next mbuf in memory since it's a pool).

Add the missing '+2', this place will be reinforced in separate commits.

Revert a part of rev1.49 (six months ago). The pointer given to memcpy
was correct.

Discussed with Christos and Ryota.

Constify isr at many places (NFC)

Use pool_cache(9) instead of pool(9) for tdb_crypto objects

The change improves network throughput especially on multi-core systems.

MP-ify SAD (savlist)

localcount(9) is used to protect savlist of sah. The basic design is
similar to MP-ifications of SPD and SAD sahlist. Please read the
locking notes of SAD for more details.

Introduce KEY_SA_UNREF and replace KEY_FREESAV with it where sav will never be actually freed in the future

KEY_SA_UNREF is still key_freesav so no functional change for now.

This change reduces diff of further changes.

Make IPsec SPD MP-safe

We use localcount(9), not psref(9), to make the sptree and secpolicy (SP)
entries MP-safe because SPs need to be referenced over opencrypto
processing that executes a callback in a different context.

SPs on sockets aren't managed by the sptree and can be destroyed in softint.
localcount_drain cannot be used in softint so we delay the destruction of
such SPs to a thread context. To do so, a list to manage such SPs is added
(key_socksplist) and key_timehandler_spd deletes dead SPs in the list.

For more details please read the locking notes in key.c.

Proposed on tech-kern@ and tech-net@

Don't acquire global locks for IPsec if NET_MPSAFE

Note that the change is just to make testing easy and IPsec isn't MP-safe yet.

Use pool to allocate tdb_crypto

For ESP and AH, we need to allocate an extra variable space in addition
to struct tdb_crypto. The fixed size of pool items may be larger than
an actual requisite size of a buffer, but still the performance
improvement by replacing malloc with pool wins.

Dedup error paths (NFC)

Fix a debug message

Hold a reference to an SP during opencrypto processing

An SP has a list of isr (ipsecrequest) that represents a sequence
of IPsec encryption/authentication processing. One isr corresponds
to one opencrypto processing. The lifetime of an isr follows its SP.

We pass an isr to a callback function of opencrypto to continue
to a next encryption/authentication processing. However nobody
guaranteed that the isr wasn't freed, i.e., its SP wasn't destroyed.

In order to avoid such unexpected destruction of isr, hold a reference
to its SP during opencrypto processing.

Don't bother the case of crp->crp_buf == NULL in callbacks

Don't release sav if calling crypto_dispatch again

branches: 1.62.2;
Calculate ah_max_authsize on initialization as well as esp_max_ivlen

Prepare to stop using isr->sav

isr is a shared resource and using isr->sav as a temporal storage
for each packet processing is racy. And also having a reference from
isr to sav makes the lifetime of sav non-deterministic; such a reference
is removed when a packet is processed and isr->sav is overwritten by
new one. Let's have a sav locally for each packet processing instead of
using shared isr->sav.

However this change doesn't stop using isr->sav yet because there are
some users of isr->sav. isr->sav will be removed after the users find
a way to not use isr->sav.

Pass sav directly to opencrypto callback

In a callback, use a passed sav as-is by default and look up a sav
only if the passed sav is dead.

Fix header size calculation of ah in the case sav is NULL

This fix was also needed for esp.

Use explicit_memset to surely zero-clear key_auth and key_enc

Rename key_alloc* functions (NFC)

We shouldn't use the term "alloc" for functions that just look up
data and actually don't allocate memory.

Remove codes for PACKET_TAG_IPSEC_IN_CRYPTO_DONE

It seems that PACKET_TAG_IPSEC_IN_CRYPTO_DONE is for network adapters
that have IPsec accelerators; a driver sets the mtag to a packet
when its device has already encrypted the packet.

Unfortunately no driver implements such offload features for long
years and seems unlikely to implement them soon. (Note that neither
FreeBSD nor Linux doesn't have such drivers.) Let's remove related
(unused) codes and simplify the IPsec code.

Apply C99-style struct initialization to xformsw

branches: 1.54.2;
Make ipsec_address() and ipsec_logsastr() mpsafe.

branches: 1.53.2;
Retire ipsec_osdep.h

We don't need to care other OSes (FreeBSD) anymore.

Some macros are alive in ipsec_private.h.

Convert IPSEC_ASSERT to KASSERT or KASSERTMSG

IPSEC_ASSERT just discarded specified message...

Remove __FreeBSD__ and __NetBSD__ switches

No functional changes (except for a debug printf).

Note that there remain some __FreeBSD__ for sysctl knobs which counerparts
to NetBSD don't exist. And ipsec_osdep.h isn't touched yet; tidying it up
requires actual code changes.

cosmetic fixes:
- __func__ in printfs
- no space after sizeof
- eliminate useless casts
- u_intX_t -> uintX_t

- fix old refactoring which zeroed the wrong part of the buffer.
- simplify.

change into __func__

Redo the statistics through an indirection array and put the definitions
of the arrays in pfkeyv2.h so that they are next to the index definitions.
Remove "bogus" comment about compressing the statistics which is now fixed.

Fix that ah_algorithm_lookup and esp_algorithm_lookup don't handle some algorithms

Unrelated upper limit values, AH_ALG_MAX and ESP_ALG_MAX, prevented some
algorithms from being looked up.

Prepare netipsec for rump-ification

- Include "opt_*.h" only if _KERNEL_OPT is defined
- Allow encapinit to be called twice (by ifinit and ipe4_attach)
- ifinit didn't call encapinit if IPSEC is enabled (ipe4_attach called
it instead), however, on a rump kernel ipe4_attach may not be called
even if IPSEC is enabled. So we need to allow ifinit to call it anyway
- Setup sysctls in ipsec_attach explicitly instead of using SYSCTL_SETUP
- Call ip6flow_invalidate_all in key_spdadd only if in6_present
- It's possible that a rump kernel loads the ipsec library but not
the inet6 library

branches: 1.44.2; 1.44.4;
Tidy up opt_ipsec.h inclusions

Some inclusions of opt_ipsec.h were for IPSEC_NAT_T and are now unnecessary.
Add inclusions to some C files for IPSEC_DEBUG.

KNF

branches: 1.42.4; 1.42.6; 1.42.8; 1.42.12;
- apply some __diagused
- remove unused variables
- move some variables inside their relevant use #ifdef

Fix sense of consttime_memequal and update all callers.

Now it returns true (nonzero) to mean equal and false (zero) to mean
inequal, as the name suggests.

As promised on tech-userlevel back in June:

https://mail-index.netbsd.org/tech-userlevel/2013/06/24/msg007843.html

branches: 1.40.2;
Replace consttime_bcmp/explicit_bzero by consttime_memequal/explicit_memset.

consttime_memequal is the same as the old consttime_bcmp.
explicit_memset is to memset as explicit_bzero was to bcmp.

Passes amd64 release and i386/ALL, but I'm sure I missed some spots,
so please let me know.

PR/47886: Dr. Wolfgang Stukenbrock: IPSEC_NAT_T enabled kernels may access
outdated pointers and pass ESP data to UPD-sockets.
While here, simplify the code and remove the IPSEC_NAT_T option; always
compile nat-traversal in so that it does not bitrot.

branches: 1.38.2;
Add "consttime_bcmp" and "explicit_bzero" functions for both kernel
abd userland, as proposed on tech-security, with explicit_bzero using
a volatile function pointer as suggested by Alan Barrett.
Both do what the name says. For userland, both are prefixed by "__"
to keep them out of the user namespace.
Change some memset/memcmp uses to the new functions where it makes
sense -- these are just some examples, more to come.

branches: 1.37.2; 1.37.6; 1.37.8;
remove some DPRINTFs which are not just diagnostics but cause noise
even on regular operation

Make sure the mbufs in the input path (only the parts which we are going
to modify in the AH case) are writable/non-shared.
This addresses PR kern/33162 by Jeff Rizzo, and replaces the insufficient
patch from that time by a radical solution.
(The PR's problem had been worked around by rev.1.3 of xennetback_xenbus.c,
so it needs a network driver modification to reproduce it.)
Being here, clarify a bit of ipcomp -- uncompression is done in-place,
the header must be removed explicitly.

fix pointer/offset mistakes in handling of IPv4 options

add patch from Arnaud Degroote to handle IPv6 extended options with
(FAST_)IPSEC, tested lightly with a DSTOPTS header consisting
of PAD1

branches: 1.33.4; 1.33.8;
copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

As a first step towards more fine-grained locking, don't require
crypto_{new.free}session() to be called with the "crypto_mtx"
spinlock held.
This doesn't change much for now because these functions acquire
the said mutex first on entry now, but at least it keeps the nasty
locks local to the opencrypto core.

more "const"

sprinkle some "const", documenting that the SA is not supposed to
change during an xform operation