PR/56836: Andrew Cagney: IPv6 ESN tunneling IPcomp has corrupt header

Always always send / expect CPI in IPcomp header

Fixes kern/56836 where an IPsec interop combining compression and
ESP|AH would fail.

Since fast ipsec, the outgoing IPcomp header has contained the
compression algorithm instead of the CPI. Adding the
SADB_X_EXT_RAWCPI flag worked around this but ...

The IPcomp's SADB was unconditionally hashed using the compression
algorithm instead of the CPI. This meant that an incoming packet with
a valid CPI could never match its SADB.

opencrypto: crypto_dispatch never fails now. Make it return void.

Same with crypto_kdispatch.

opencrypto: Rip out EAGAIN logic when unregistering crypto drivers.

I'm pretty sure this never worked reliably based on code inspection,
and it's unlikely to have ever been tested because it only applies
when unregistering a driver -- but we have no crypto drivers for
removable devices, so it would only apply if we went out of our way
to trigger detach with drvctl.

Instead, just make the operation fail with ENODEV, and remove all the
callback logic to resubmit the request on EAGAIN. (Maybe this should
be ENXIO, but crypto_kdispatch already does ENODEV.)

opencrypto: Make crypto_freesession return void.

No callers use the return value. It is not sensible to allow this to
fail.

netipsec: Nothing uses xf_zeroize return value. Nix it.

opencrypto: Make crp_callback, krp_callback return void.

Nothing uses the return values inside opencrypto, so let's stop
making users return them.

Fix ipsecif(4) IPV6_MINMTU does not work correctly.

make DPRINTF use varyadic cpp macros, and merge with IPSECLOG.

Merge the [pgoyette-compat] branch

branches: 1.66.2;
Remove unused calls to nat_t_ports_get.

Remove unused 'mp' argument from all the xf_output functions. Also clean
up xform.h a bit.

Remove double include, opencrypto/xform.h is already included in
netipsec/xform.h.

Remove IPSEC_SPLASSERT_SOFTNET, it has always been a no-op.

Remove extra long file paths from the headers.

Add a KASSERT (which is not triggerable since ipsec_common_input already
ensures 8 bytes are present), add an XXX (about the fact that it is
better to use m_copydata, because it is faster and less error-prone), and
improve two m_copybacks (remove useless casts).

Fix the computation. Normally that's harmless since ip6_output recomputes
ip6_plen.

branches: 1.59.2;
Add [ah/esp/ipcomp]_enable sysctls, and remove the FreeBSD #ifdefs.
Discussed with ozaki-r@.

Remove some more FreeBSD sysctl declarations that already have NetBSD
counterparts. Discussed with ozaki-r@.

Style and simplify.

Don't relook up an SP/SA in opencrpyto callbacks

We don't need to do so because we have a reference to it. And also
relooking-up one there may return an sp/sav that has different
parameters from an original one.

Dedup common codes in error paths (NFCI)

Fix mbuf leaks on error paths

Pointed out by maxv@

Constify isr at many places (NFC)

Use pool_cache(9) instead of pool(9) for tdb_crypto objects

The change improves network throughput especially on multi-core systems.

MP-ify SAD (savlist)

localcount(9) is used to protect savlist of sah. The basic design is
similar to MP-ifications of SPD and SAD sahlist. Please read the
locking notes of SAD for more details.

Introduce KEY_SA_UNREF and replace KEY_FREESAV with it where sav will never be actually freed in the future

KEY_SA_UNREF is still key_freesav so no functional change for now.

This change reduces diff of further changes.

Make IPsec SPD MP-safe

We use localcount(9), not psref(9), to make the sptree and secpolicy (SP)
entries MP-safe because SPs need to be referenced over opencrypto
processing that executes a callback in a different context.

SPs on sockets aren't managed by the sptree and can be destroyed in softint.
localcount_drain cannot be used in softint so we delay the destruction of
such SPs to a thread context. To do so, a list to manage such SPs is added
(key_socksplist) and key_timehandler_spd deletes dead SPs in the list.

For more details please read the locking notes in key.c.

Proposed on tech-kern@ and tech-net@

Don't acquire global locks for IPsec if NET_MPSAFE

Note that the change is just to make testing easy and IPsec isn't MP-safe yet.

Use pool to allocate tdb_crypto

For ESP and AH, we need to allocate an extra variable space in addition
to struct tdb_crypto. The fixed size of pool items may be larger than
an actual requisite size of a buffer, but still the performance
improvement by replacing malloc with pool wins.

Hold a reference to an SP during opencrypto processing

An SP has a list of isr (ipsecrequest) that represents a sequence
of IPsec encryption/authentication processing. One isr corresponds
to one opencrypto processing. The lifetime of an isr follows its SP.

We pass an isr to a callback function of opencrypto to continue
to a next encryption/authentication processing. However nobody
guaranteed that the isr wasn't freed, i.e., its SP wasn't destroyed.

In order to avoid such unexpected destruction of isr, hold a reference
to its SP during opencrypto processing.

Don't bother the case of crp->crp_buf == NULL in callbacks

Don't release sav if calling crypto_dispatch again

Prepare to stop using isr->sav

isr is a shared resource and using isr->sav as a temporal storage
for each packet processing is racy. And also having a reference from
isr to sav makes the lifetime of sav non-deterministic; such a reference
is removed when a packet is processed and isr->sav is overwritten by
new one. Let's have a sav locally for each packet processing instead of
using shared isr->sav.

However this change doesn't stop using isr->sav yet because there are
some users of isr->sav. isr->sav will be removed after the users find
a way to not use isr->sav.

Pass sav directly to opencrypto callback

In a callback, use a passed sav as-is by default and look up a sav
only if the passed sav is dead.

Rename key_alloc* functions (NFC)

We shouldn't use the term "alloc" for functions that just look up
data and actually don't allocate memory.

Remove codes for PACKET_TAG_IPSEC_IN_CRYPTO_DONE

It seems that PACKET_TAG_IPSEC_IN_CRYPTO_DONE is for network adapters
that have IPsec accelerators; a driver sets the mtag to a packet
when its device has already encrypted the packet.

Unfortunately no driver implements such offload features for long
years and seems unlikely to implement them soon. (Note that neither
FreeBSD nor Linux doesn't have such drivers.) Let's remove related
(unused) codes and simplify the IPsec code.

Apply C99-style struct initialization to xformsw

branches: 1.38.2;
Make ipsec_address() and ipsec_logsastr() mpsafe.

branches: 1.37.2;
Retire ipsec_osdep.h

We don't need to care other OSes (FreeBSD) anymore.

Some macros are alive in ipsec_private.h.

Convert IPSEC_ASSERT to KASSERT or KASSERTMSG

IPSEC_ASSERT just discarded specified message...

Remove __FreeBSD__ and __NetBSD__ switches

No functional changes (except for a debug printf).

Note that there remain some __FreeBSD__ for sysctl knobs which counerparts
to NetBSD don't exist. And ipsec_osdep.h isn't touched yet; tidying it up
requires actual code changes.

cosmetic fixes:
- __func__ in printfs
- no space after sizeof
- eliminate useless casts
- u_intX_t -> uintX_t

Redo the statistics through an indirection array and put the definitions
of the arrays in pfkeyv2.h so that they are next to the index definitions.
Remove "bogus" comment about compressing the statistics which is now fixed.

Prepare netipsec for rump-ification

- Include "opt_*.h" only if _KERNEL_OPT is defined
- Allow encapinit to be called twice (by ifinit and ipe4_attach)
- ifinit didn't call encapinit if IPSEC is enabled (ipe4_attach called
it instead), however, on a rump kernel ipe4_attach may not be called
even if IPSEC is enabled. So we need to allow ifinit to call it anyway
- Setup sysctls in ipsec_attach explicitly instead of using SYSCTL_SETUP
- Call ip6flow_invalidate_all in key_spdadd only if in6_present
- It's possible that a rump kernel loads the ipsec library but not
the inet6 library

branches: 1.31.6; 1.31.10; 1.31.14;
- apply some __diagused
- remove unused variables
- move some variables inside their relevant use #ifdef

branches: 1.30.2;
PR/47886: Dr. Wolfgang Stukenbrock: IPSEC_NAT_T enabled kernels may access
outdated pointers and pass ESP data to UPD-sockets.
While here, simplify the code and remove the IPSEC_NAT_T option; always
compile nat-traversal in so that it does not bitrot.

branches: 1.29.6;
Make sure the mbufs in the input path (only the parts which we are going
to modify in the AH case) are writable/non-shared.
This addresses PR kern/33162 by Jeff Rizzo, and replaces the insufficient
patch from that time by a radical solution.
(The PR's problem had been worked around by rev.1.3 of xennetback_xenbus.c,
so it needs a network driver modification to reproduce it.)
Being here, clarify a bit of ipcomp -- uncompression is done in-place,
the header must be removed explicitly.

branches: 1.28.4; 1.28.8;
As a first step towards more fine-grained locking, don't require
crypto_{new.free}session() to be called with the "crypto_mtx"
spinlock held.
This doesn't change much for now because these functions acquire
the said mutex first on entry now, but at least it keeps the nasty
locks local to the opencrypto core.

fix C&P botch in diagnostic printfs

mitigation for CVE-2011-1547

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

more "const"

sprinkle some "const", documenting that the SA is not supposed to
change during an xform operation

change locking order, to make sure the cpu is at splsoftnet()
before the softnet_lock (adaptive) mutex is acquired, from
Wolfgang Stukenbrock, should fix a recursive lock panic

-in opencrypto callbacks (which run in a kernel thread), pull softnet_lock
everywhere splsoftnet() was used before, to fix MP concurrency problems
-pull KERNEL_LOCK where ip(6)_output() is called, as this is what
the network stack (unfortunately) expects, in particular to avoid
races for packets in the interface send queues
From Wolfgang Stukenbrock per PR kern/44418, with the application
of KERNEL_LOCK to what I think are the essential points, tested
on a dual-core i386.

branches: 1.20.2; 1.20.4;
Fix ipcomp input counter

Reported Wolfgang Stukenbrock in pr/43250.

branches: 1.19.2; 1.19.4;
bzero -> memset

branches: 1.18.2; 1.18.10; 1.18.12; 1.18.16; 1.18.18; 1.18.22;
Make IPSEC and FAST_IPSEC stats per-cpu. Use <net/net_stats.h> and
netstat_sysctl().

branches: 1.17.6; 1.17.8;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

Add some statistics for case where compression is not useful
(when len(compressed packet) > len(initial packet))

branches: 1.15.6; 1.15.12;
Fix my previous stupid caddr_t fix.

branches: 1.14.6; 1.14.8;
Add support for options IPSEC_NAT_T (RFC 3947 and 3948) for fast_ipsec(4).

No objection on tech-net@

branches: 1.13.2; 1.13.4;
Remove useless cast
Use NULL instead of (void*) 0

Fix fallout from caddr_t changes

Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

Oops, I forgot to commit some bits last time

fast_ipsec and ipcomp works better now.

branches: 1.9.2;
Commit my SoC work
Add ipv6 support for fast_ipsec
Note that currently, packet with extensions headers are not correctly
supported
Change the ipcomp logic

branches: 1.8.2; 1.8.4;
__unused removal on arguments; approved by core.

more __unused

branches: 1.6.20; 1.6.22;
merge ktrace-lwp.

branches: 1.5.4;
nuke trailing whitespace

branches: 1.4.4; 1.4.10; 1.4.12;
Reversion of "netkey merge", part 2 (replacement of removed files in the
repository by christos was part 1). netipsec should now be back as it
was on 2003-09-11, with some very minor changes:

1) Some residual platform-dependent code was moved from ipsec.h to
ipsec_osdep.h; without this, IPSEC_ASSERT() was multiply defined. ipsec.h
now includes ipsec_osdep.h

2) itojun's renaming of netipsec/files.ipsec to netipsec/files.netipsec has
been left in place (it's arguable which name is less confusing but the
rename is pretty harmless).

3) Some #endif TOKEN has been replaced by #endif /* TOKEN */; #endif TOKEN
is invalid and GCC 3 won't compile it.

An i386 kernel with "options FAST_IPSEC" and "options OPENCRYPTO" now
gets through "make depend" but fails to build with errors in ip_input.c.
But it's better than it was (thank heaven for small favors).

merge netipsec/key* into netkey/key*. no need for both.
change confusing filename

opt_inet6.h is FreeBSD-specific, so wrap it with #ifdef __FreeBSD__/#endif.

Initial import of Sam Leffler's `Fast-IPsec' from FreeBSD 4.
Fast-IPsec is a rework of the OpenBSD and KAME IPsec code, using the
OpenCryptoFramework (and thus hardware crypto accelerators) and
numerous detailed performance improvements.

This import is (aside from SPL-level names) the FreeBSD source,
imported ``as-is'' as a historical snapshot, for future maintenance
and comparison against the FreeBSD source. For now, several minor
kernel-API differences are hidden by macros a shim file, ipsec_osdep.h,
which (aside from SPL names) can be targeted at either NetBSD or FreeBSD.

opencrypto: crypto_dispatch never fails now. Make it return void.

Same with crypto_kdispatch.

opencrypto: Rip out EAGAIN logic when unregistering crypto drivers.

I'm pretty sure this never worked reliably based on code inspection,
and it's unlikely to have ever been tested because it only applies
when unregistering a driver -- but we have no crypto drivers for
removable devices, so it would only apply if we went out of our way
to trigger detach with drvctl.

Instead, just make the operation fail with ENODEV, and remove all the
callback logic to resubmit the request on EAGAIN. (Maybe this should
be ENXIO, but crypto_kdispatch already does ENODEV.)

opencrypto: Make crypto_freesession return void.

No callers use the return value. It is not sensible to allow this to
fail.

netipsec: Nothing uses xf_zeroize return value. Nix it.

opencrypto: Make crp_callback, krp_callback return void.

Nothing uses the return values inside opencrypto, so let's stop
making users return them.

Fix ipsecif(4) IPV6_MINMTU does not work correctly.

make DPRINTF use varyadic cpp macros, and merge with IPSECLOG.

Merge the [pgoyette-compat] branch

branches: 1.66.2;
Remove unused calls to nat_t_ports_get.

Remove unused 'mp' argument from all the xf_output functions. Also clean
up xform.h a bit.

Remove double include, opencrypto/xform.h is already included in
netipsec/xform.h.

Remove IPSEC_SPLASSERT_SOFTNET, it has always been a no-op.

Remove extra long file paths from the headers.

Add a KASSERT (which is not triggerable since ipsec_common_input already
ensures 8 bytes are present), add an XXX (about the fact that it is
better to use m_copydata, because it is faster and less error-prone), and
improve two m_copybacks (remove useless casts).

Fix the computation. Normally that's harmless since ip6_output recomputes
ip6_plen.

branches: 1.59.2;
Add [ah/esp/ipcomp]_enable sysctls, and remove the FreeBSD #ifdefs.
Discussed with ozaki-r@.

Remove some more FreeBSD sysctl declarations that already have NetBSD
counterparts. Discussed with ozaki-r@.

Style and simplify.

Don't relook up an SP/SA in opencrpyto callbacks

We don't need to do so because we have a reference to it. And also
relooking-up one there may return an sp/sav that has different
parameters from an original one.

Dedup common codes in error paths (NFCI)

Fix mbuf leaks on error paths

Pointed out by maxv@

Constify isr at many places (NFC)

Use pool_cache(9) instead of pool(9) for tdb_crypto objects

The change improves network throughput especially on multi-core systems.

MP-ify SAD (savlist)

localcount(9) is used to protect savlist of sah. The basic design is
similar to MP-ifications of SPD and SAD sahlist. Please read the
locking notes of SAD for more details.

Introduce KEY_SA_UNREF and replace KEY_FREESAV with it where sav will never be actually freed in the future

KEY_SA_UNREF is still key_freesav so no functional change for now.

This change reduces diff of further changes.

Make IPsec SPD MP-safe

We use localcount(9), not psref(9), to make the sptree and secpolicy (SP)
entries MP-safe because SPs need to be referenced over opencrypto
processing that executes a callback in a different context.

SPs on sockets aren't managed by the sptree and can be destroyed in softint.
localcount_drain cannot be used in softint so we delay the destruction of
such SPs to a thread context. To do so, a list to manage such SPs is added
(key_socksplist) and key_timehandler_spd deletes dead SPs in the list.

For more details please read the locking notes in key.c.

Proposed on tech-kern@ and tech-net@

Don't acquire global locks for IPsec if NET_MPSAFE

Note that the change is just to make testing easy and IPsec isn't MP-safe yet.

Use pool to allocate tdb_crypto

For ESP and AH, we need to allocate an extra variable space in addition
to struct tdb_crypto. The fixed size of pool items may be larger than
an actual requisite size of a buffer, but still the performance
improvement by replacing malloc with pool wins.

Hold a reference to an SP during opencrypto processing

An SP has a list of isr (ipsecrequest) that represents a sequence
of IPsec encryption/authentication processing. One isr corresponds
to one opencrypto processing. The lifetime of an isr follows its SP.

We pass an isr to a callback function of opencrypto to continue
to a next encryption/authentication processing. However nobody
guaranteed that the isr wasn't freed, i.e., its SP wasn't destroyed.

In order to avoid such unexpected destruction of isr, hold a reference
to its SP during opencrypto processing.

Don't bother the case of crp->crp_buf == NULL in callbacks

Don't release sav if calling crypto_dispatch again

Prepare to stop using isr->sav

isr is a shared resource and using isr->sav as a temporal storage
for each packet processing is racy. And also having a reference from
isr to sav makes the lifetime of sav non-deterministic; such a reference
is removed when a packet is processed and isr->sav is overwritten by
new one. Let's have a sav locally for each packet processing instead of
using shared isr->sav.

However this change doesn't stop using isr->sav yet because there are
some users of isr->sav. isr->sav will be removed after the users find
a way to not use isr->sav.

Pass sav directly to opencrypto callback

In a callback, use a passed sav as-is by default and look up a sav
only if the passed sav is dead.

Rename key_alloc* functions (NFC)

We shouldn't use the term "alloc" for functions that just look up
data and actually don't allocate memory.

Remove codes for PACKET_TAG_IPSEC_IN_CRYPTO_DONE

It seems that PACKET_TAG_IPSEC_IN_CRYPTO_DONE is for network adapters
that have IPsec accelerators; a driver sets the mtag to a packet
when its device has already encrypted the packet.

Unfortunately no driver implements such offload features for long
years and seems unlikely to implement them soon. (Note that neither
FreeBSD nor Linux doesn't have such drivers.) Let's remove related
(unused) codes and simplify the IPsec code.

Apply C99-style struct initialization to xformsw

branches: 1.38.2;
Make ipsec_address() and ipsec_logsastr() mpsafe.

branches: 1.37.2;
Retire ipsec_osdep.h

We don't need to care other OSes (FreeBSD) anymore.

Some macros are alive in ipsec_private.h.

Convert IPSEC_ASSERT to KASSERT or KASSERTMSG

IPSEC_ASSERT just discarded specified message...

Remove __FreeBSD__ and __NetBSD__ switches

No functional changes (except for a debug printf).

Note that there remain some __FreeBSD__ for sysctl knobs which counerparts
to NetBSD don't exist. And ipsec_osdep.h isn't touched yet; tidying it up
requires actual code changes.

cosmetic fixes:
- __func__ in printfs
- no space after sizeof
- eliminate useless casts
- u_intX_t -> uintX_t

Redo the statistics through an indirection array and put the definitions
of the arrays in pfkeyv2.h so that they are next to the index definitions.
Remove "bogus" comment about compressing the statistics which is now fixed.

Prepare netipsec for rump-ification

- Include "opt_*.h" only if _KERNEL_OPT is defined
- Allow encapinit to be called twice (by ifinit and ipe4_attach)
- ifinit didn't call encapinit if IPSEC is enabled (ipe4_attach called
it instead), however, on a rump kernel ipe4_attach may not be called
even if IPSEC is enabled. So we need to allow ifinit to call it anyway
- Setup sysctls in ipsec_attach explicitly instead of using SYSCTL_SETUP
- Call ip6flow_invalidate_all in key_spdadd only if in6_present
- It's possible that a rump kernel loads the ipsec library but not
the inet6 library

branches: 1.31.6; 1.31.10; 1.31.14;
- apply some __diagused
- remove unused variables
- move some variables inside their relevant use #ifdef

branches: 1.30.2;
PR/47886: Dr. Wolfgang Stukenbrock: IPSEC_NAT_T enabled kernels may access
outdated pointers and pass ESP data to UPD-sockets.
While here, simplify the code and remove the IPSEC_NAT_T option; always
compile nat-traversal in so that it does not bitrot.

branches: 1.29.6;
Make sure the mbufs in the input path (only the parts which we are going
to modify in the AH case) are writable/non-shared.
This addresses PR kern/33162 by Jeff Rizzo, and replaces the insufficient
patch from that time by a radical solution.
(The PR's problem had been worked around by rev.1.3 of xennetback_xenbus.c,
so it needs a network driver modification to reproduce it.)
Being here, clarify a bit of ipcomp -- uncompression is done in-place,
the header must be removed explicitly.

branches: 1.28.4; 1.28.8;
As a first step towards more fine-grained locking, don't require
crypto_{new.free}session() to be called with the "crypto_mtx"
spinlock held.
This doesn't change much for now because these functions acquire
the said mutex first on entry now, but at least it keeps the nasty
locks local to the opencrypto core.

fix C&P botch in diagnostic printfs

mitigation for CVE-2011-1547

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

more "const"

sprinkle some "const", documenting that the SA is not supposed to
change during an xform operation

change locking order, to make sure the cpu is at splsoftnet()
before the softnet_lock (adaptive) mutex is acquired, from
Wolfgang Stukenbrock, should fix a recursive lock panic

-in opencrypto callbacks (which run in a kernel thread), pull softnet_lock
everywhere splsoftnet() was used before, to fix MP concurrency problems
-pull KERNEL_LOCK where ip(6)_output() is called, as this is what
the network stack (unfortunately) expects, in particular to avoid
races for packets in the interface send queues
From Wolfgang Stukenbrock per PR kern/44418, with the application
of KERNEL_LOCK to what I think are the essential points, tested
on a dual-core i386.

branches: 1.20.2; 1.20.4;
Fix ipcomp input counter

Reported Wolfgang Stukenbrock in pr/43250.

branches: 1.19.2; 1.19.4;
bzero -> memset

branches: 1.18.2; 1.18.10; 1.18.12; 1.18.16; 1.18.18; 1.18.22;
Make IPSEC and FAST_IPSEC stats per-cpu. Use <net/net_stats.h> and
netstat_sysctl().

branches: 1.17.6; 1.17.8;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

Add some statistics for case where compression is not useful
(when len(compressed packet) > len(initial packet))

branches: 1.15.6; 1.15.12;
Fix my previous stupid caddr_t fix.

branches: 1.14.6; 1.14.8;
Add support for options IPSEC_NAT_T (RFC 3947 and 3948) for fast_ipsec(4).

No objection on tech-net@

branches: 1.13.2; 1.13.4;
Remove useless cast
Use NULL instead of (void*) 0

Fix fallout from caddr_t changes

Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

Oops, I forgot to commit some bits last time

fast_ipsec and ipcomp works better now.

branches: 1.9.2;
Commit my SoC work
Add ipv6 support for fast_ipsec
Note that currently, packet with extensions headers are not correctly
supported
Change the ipcomp logic

branches: 1.8.2; 1.8.4;
__unused removal on arguments; approved by core.

more __unused

branches: 1.6.20; 1.6.22;
merge ktrace-lwp.

branches: 1.5.4;
nuke trailing whitespace

branches: 1.4.4; 1.4.10; 1.4.12;
Reversion of "netkey merge", part 2 (replacement of removed files in the
repository by christos was part 1). netipsec should now be back as it
was on 2003-09-11, with some very minor changes:

1) Some residual platform-dependent code was moved from ipsec.h to
ipsec_osdep.h; without this, IPSEC_ASSERT() was multiply defined. ipsec.h
now includes ipsec_osdep.h

2) itojun's renaming of netipsec/files.ipsec to netipsec/files.netipsec has
been left in place (it's arguable which name is less confusing but the
rename is pretty harmless).

3) Some #endif TOKEN has been replaced by #endif /* TOKEN */; #endif TOKEN
is invalid and GCC 3 won't compile it.

An i386 kernel with "options FAST_IPSEC" and "options OPENCRYPTO" now
gets through "make depend" but fails to build with errors in ip_input.c.
But it's better than it was (thank heaven for small favors).

merge netipsec/key* into netkey/key*. no need for both.
change confusing filename

opt_inet6.h is FreeBSD-specific, so wrap it with #ifdef __FreeBSD__/#endif.

Initial import of Sam Leffler's `Fast-IPsec' from FreeBSD 4.
Fast-IPsec is a rework of the OpenBSD and KAME IPsec code, using the
OpenCryptoFramework (and thus hardware crypto accelerators) and
numerous detailed performance improvements.

This import is (aside from SPL-level names) the FreeBSD source,
imported ``as-is'' as a historical snapshot, for future maintenance
and comparison against the FreeBSD source. For now, several minor
kernel-API differences are hidden by macros a shim file, ipsec_osdep.h,
which (aside from SPL names) can be targeted at either NetBSD or FreeBSD.

Fix ipsecif(4) IPV6_MINMTU does not work correctly.

make DPRINTF use varyadic cpp macros, and merge with IPSECLOG.

Merge the [pgoyette-compat] branch

branches: 1.66.2;
Remove unused calls to nat_t_ports_get.

Remove unused 'mp' argument from all the xf_output functions. Also clean
up xform.h a bit.

Remove double include, opencrypto/xform.h is already included in
netipsec/xform.h.

Remove IPSEC_SPLASSERT_SOFTNET, it has always been a no-op.

Remove extra long file paths from the headers.

Add a KASSERT (which is not triggerable since ipsec_common_input already
ensures 8 bytes are present), add an XXX (about the fact that it is
better to use m_copydata, because it is faster and less error-prone), and
improve two m_copybacks (remove useless casts).

Fix the computation. Normally that's harmless since ip6_output recomputes
ip6_plen.

branches: 1.59.2;
Add [ah/esp/ipcomp]_enable sysctls, and remove the FreeBSD #ifdefs.
Discussed with ozaki-r@.

Remove some more FreeBSD sysctl declarations that already have NetBSD
counterparts. Discussed with ozaki-r@.

Style and simplify.

Don't relook up an SP/SA in opencrpyto callbacks

We don't need to do so because we have a reference to it. And also
relooking-up one there may return an sp/sav that has different
parameters from an original one.

Dedup common codes in error paths (NFCI)

Fix mbuf leaks on error paths

Pointed out by maxv@

Constify isr at many places (NFC)

Use pool_cache(9) instead of pool(9) for tdb_crypto objects

The change improves network throughput especially on multi-core systems.

MP-ify SAD (savlist)

localcount(9) is used to protect savlist of sah. The basic design is
similar to MP-ifications of SPD and SAD sahlist. Please read the
locking notes of SAD for more details.

Introduce KEY_SA_UNREF and replace KEY_FREESAV with it where sav will never be actually freed in the future

KEY_SA_UNREF is still key_freesav so no functional change for now.

This change reduces diff of further changes.

Make IPsec SPD MP-safe

We use localcount(9), not psref(9), to make the sptree and secpolicy (SP)
entries MP-safe because SPs need to be referenced over opencrypto
processing that executes a callback in a different context.

SPs on sockets aren't managed by the sptree and can be destroyed in softint.
localcount_drain cannot be used in softint so we delay the destruction of
such SPs to a thread context. To do so, a list to manage such SPs is added
(key_socksplist) and key_timehandler_spd deletes dead SPs in the list.

For more details please read the locking notes in key.c.

Proposed on tech-kern@ and tech-net@

Don't acquire global locks for IPsec if NET_MPSAFE

Note that the change is just to make testing easy and IPsec isn't MP-safe yet.

Use pool to allocate tdb_crypto

For ESP and AH, we need to allocate an extra variable space in addition
to struct tdb_crypto. The fixed size of pool items may be larger than
an actual requisite size of a buffer, but still the performance
improvement by replacing malloc with pool wins.

Hold a reference to an SP during opencrypto processing

An SP has a list of isr (ipsecrequest) that represents a sequence
of IPsec encryption/authentication processing. One isr corresponds
to one opencrypto processing. The lifetime of an isr follows its SP.

We pass an isr to a callback function of opencrypto to continue
to a next encryption/authentication processing. However nobody
guaranteed that the isr wasn't freed, i.e., its SP wasn't destroyed.

In order to avoid such unexpected destruction of isr, hold a reference
to its SP during opencrypto processing.

Don't bother the case of crp->crp_buf == NULL in callbacks

Don't release sav if calling crypto_dispatch again

Prepare to stop using isr->sav

isr is a shared resource and using isr->sav as a temporal storage
for each packet processing is racy. And also having a reference from
isr to sav makes the lifetime of sav non-deterministic; such a reference
is removed when a packet is processed and isr->sav is overwritten by
new one. Let's have a sav locally for each packet processing instead of
using shared isr->sav.

However this change doesn't stop using isr->sav yet because there are
some users of isr->sav. isr->sav will be removed after the users find
a way to not use isr->sav.

Pass sav directly to opencrypto callback

In a callback, use a passed sav as-is by default and look up a sav
only if the passed sav is dead.

Rename key_alloc* functions (NFC)

We shouldn't use the term "alloc" for functions that just look up
data and actually don't allocate memory.

Remove codes for PACKET_TAG_IPSEC_IN_CRYPTO_DONE

It seems that PACKET_TAG_IPSEC_IN_CRYPTO_DONE is for network adapters
that have IPsec accelerators; a driver sets the mtag to a packet
when its device has already encrypted the packet.

Unfortunately no driver implements such offload features for long
years and seems unlikely to implement them soon. (Note that neither
FreeBSD nor Linux doesn't have such drivers.) Let's remove related
(unused) codes and simplify the IPsec code.

Apply C99-style struct initialization to xformsw

branches: 1.38.2;
Make ipsec_address() and ipsec_logsastr() mpsafe.

branches: 1.37.2;
Retire ipsec_osdep.h

We don't need to care other OSes (FreeBSD) anymore.

Some macros are alive in ipsec_private.h.

Convert IPSEC_ASSERT to KASSERT or KASSERTMSG

IPSEC_ASSERT just discarded specified message...

Remove __FreeBSD__ and __NetBSD__ switches

No functional changes (except for a debug printf).

Note that there remain some __FreeBSD__ for sysctl knobs which counerparts
to NetBSD don't exist. And ipsec_osdep.h isn't touched yet; tidying it up
requires actual code changes.

cosmetic fixes:
- __func__ in printfs
- no space after sizeof
- eliminate useless casts
- u_intX_t -> uintX_t

Redo the statistics through an indirection array and put the definitions
of the arrays in pfkeyv2.h so that they are next to the index definitions.
Remove "bogus" comment about compressing the statistics which is now fixed.

Prepare netipsec for rump-ification

- Include "opt_*.h" only if _KERNEL_OPT is defined
- Allow encapinit to be called twice (by ifinit and ipe4_attach)
- ifinit didn't call encapinit if IPSEC is enabled (ipe4_attach called
it instead), however, on a rump kernel ipe4_attach may not be called
even if IPSEC is enabled. So we need to allow ifinit to call it anyway
- Setup sysctls in ipsec_attach explicitly instead of using SYSCTL_SETUP
- Call ip6flow_invalidate_all in key_spdadd only if in6_present
- It's possible that a rump kernel loads the ipsec library but not
the inet6 library

branches: 1.31.6; 1.31.10; 1.31.14;
- apply some __diagused
- remove unused variables
- move some variables inside their relevant use #ifdef

branches: 1.30.2;
PR/47886: Dr. Wolfgang Stukenbrock: IPSEC_NAT_T enabled kernels may access
outdated pointers and pass ESP data to UPD-sockets.
While here, simplify the code and remove the IPSEC_NAT_T option; always
compile nat-traversal in so that it does not bitrot.

branches: 1.29.6;
Make sure the mbufs in the input path (only the parts which we are going
to modify in the AH case) are writable/non-shared.
This addresses PR kern/33162 by Jeff Rizzo, and replaces the insufficient
patch from that time by a radical solution.
(The PR's problem had been worked around by rev.1.3 of xennetback_xenbus.c,
so it needs a network driver modification to reproduce it.)
Being here, clarify a bit of ipcomp -- uncompression is done in-place,
the header must be removed explicitly.

branches: 1.28.4; 1.28.8;
As a first step towards more fine-grained locking, don't require
crypto_{new.free}session() to be called with the "crypto_mtx"
spinlock held.
This doesn't change much for now because these functions acquire
the said mutex first on entry now, but at least it keeps the nasty
locks local to the opencrypto core.

fix C&P botch in diagnostic printfs

mitigation for CVE-2011-1547

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

more "const"

sprinkle some "const", documenting that the SA is not supposed to
change during an xform operation

change locking order, to make sure the cpu is at splsoftnet()
before the softnet_lock (adaptive) mutex is acquired, from
Wolfgang Stukenbrock, should fix a recursive lock panic

-in opencrypto callbacks (which run in a kernel thread), pull softnet_lock
everywhere splsoftnet() was used before, to fix MP concurrency problems
-pull KERNEL_LOCK where ip(6)_output() is called, as this is what
the network stack (unfortunately) expects, in particular to avoid
races for packets in the interface send queues
From Wolfgang Stukenbrock per PR kern/44418, with the application
of KERNEL_LOCK to what I think are the essential points, tested
on a dual-core i386.

branches: 1.20.2; 1.20.4;
Fix ipcomp input counter

Reported Wolfgang Stukenbrock in pr/43250.

branches: 1.19.2; 1.19.4;
bzero -> memset

branches: 1.18.2; 1.18.10; 1.18.12; 1.18.16; 1.18.18; 1.18.22;
Make IPSEC and FAST_IPSEC stats per-cpu. Use <net/net_stats.h> and
netstat_sysctl().

branches: 1.17.6; 1.17.8;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

Add some statistics for case where compression is not useful
(when len(compressed packet) > len(initial packet))

branches: 1.15.6; 1.15.12;
Fix my previous stupid caddr_t fix.

branches: 1.14.6; 1.14.8;
Add support for options IPSEC_NAT_T (RFC 3947 and 3948) for fast_ipsec(4).

No objection on tech-net@

branches: 1.13.2; 1.13.4;
Remove useless cast
Use NULL instead of (void*) 0

Fix fallout from caddr_t changes

Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

Oops, I forgot to commit some bits last time

fast_ipsec and ipcomp works better now.

branches: 1.9.2;
Commit my SoC work
Add ipv6 support for fast_ipsec
Note that currently, packet with extensions headers are not correctly
supported
Change the ipcomp logic

branches: 1.8.2; 1.8.4;
__unused removal on arguments; approved by core.

more __unused

branches: 1.6.20; 1.6.22;
merge ktrace-lwp.

branches: 1.5.4;
nuke trailing whitespace

branches: 1.4.4; 1.4.10; 1.4.12;
Reversion of "netkey merge", part 2 (replacement of removed files in the
repository by christos was part 1). netipsec should now be back as it
was on 2003-09-11, with some very minor changes:

1) Some residual platform-dependent code was moved from ipsec.h to
ipsec_osdep.h; without this, IPSEC_ASSERT() was multiply defined. ipsec.h
now includes ipsec_osdep.h

2) itojun's renaming of netipsec/files.ipsec to netipsec/files.netipsec has
been left in place (it's arguable which name is less confusing but the
rename is pretty harmless).

3) Some #endif TOKEN has been replaced by #endif /* TOKEN */; #endif TOKEN
is invalid and GCC 3 won't compile it.

An i386 kernel with "options FAST_IPSEC" and "options OPENCRYPTO" now
gets through "make depend" but fails to build with errors in ip_input.c.
But it's better than it was (thank heaven for small favors).

merge netipsec/key* into netkey/key*. no need for both.
change confusing filename

opt_inet6.h is FreeBSD-specific, so wrap it with #ifdef __FreeBSD__/#endif.

Initial import of Sam Leffler's `Fast-IPsec' from FreeBSD 4.
Fast-IPsec is a rework of the OpenBSD and KAME IPsec code, using the
OpenCryptoFramework (and thus hardware crypto accelerators) and
numerous detailed performance improvements.

This import is (aside from SPL-level names) the FreeBSD source,
imported ``as-is'' as a historical snapshot, for future maintenance
and comparison against the FreeBSD source. For now, several minor
kernel-API differences are hidden by macros a shim file, ipsec_osdep.h,
which (aside from SPL names) can be targeted at either NetBSD or FreeBSD.

make DPRINTF use varyadic cpp macros, and merge with IPSECLOG.

Merge the [pgoyette-compat] branch

branches: 1.66.2;
Remove unused calls to nat_t_ports_get.

Remove unused 'mp' argument from all the xf_output functions. Also clean
up xform.h a bit.

Remove double include, opencrypto/xform.h is already included in
netipsec/xform.h.

Remove IPSEC_SPLASSERT_SOFTNET, it has always been a no-op.

Remove extra long file paths from the headers.

Add a KASSERT (which is not triggerable since ipsec_common_input already
ensures 8 bytes are present), add an XXX (about the fact that it is
better to use m_copydata, because it is faster and less error-prone), and
improve two m_copybacks (remove useless casts).

Fix the computation. Normally that's harmless since ip6_output recomputes
ip6_plen.

branches: 1.59.2;
Add [ah/esp/ipcomp]_enable sysctls, and remove the FreeBSD #ifdefs.
Discussed with ozaki-r@.

Remove some more FreeBSD sysctl declarations that already have NetBSD
counterparts. Discussed with ozaki-r@.

Style and simplify.

Don't relook up an SP/SA in opencrpyto callbacks

We don't need to do so because we have a reference to it. And also
relooking-up one there may return an sp/sav that has different
parameters from an original one.

Dedup common codes in error paths (NFCI)

Fix mbuf leaks on error paths

Pointed out by maxv@

Constify isr at many places (NFC)

Use pool_cache(9) instead of pool(9) for tdb_crypto objects

The change improves network throughput especially on multi-core systems.

MP-ify SAD (savlist)

localcount(9) is used to protect savlist of sah. The basic design is
similar to MP-ifications of SPD and SAD sahlist. Please read the
locking notes of SAD for more details.

Introduce KEY_SA_UNREF and replace KEY_FREESAV with it where sav will never be actually freed in the future

KEY_SA_UNREF is still key_freesav so no functional change for now.

This change reduces diff of further changes.

Make IPsec SPD MP-safe

We use localcount(9), not psref(9), to make the sptree and secpolicy (SP)
entries MP-safe because SPs need to be referenced over opencrypto
processing that executes a callback in a different context.

SPs on sockets aren't managed by the sptree and can be destroyed in softint.
localcount_drain cannot be used in softint so we delay the destruction of
such SPs to a thread context. To do so, a list to manage such SPs is added
(key_socksplist) and key_timehandler_spd deletes dead SPs in the list.

For more details please read the locking notes in key.c.

Proposed on tech-kern@ and tech-net@

Don't acquire global locks for IPsec if NET_MPSAFE

Note that the change is just to make testing easy and IPsec isn't MP-safe yet.

Use pool to allocate tdb_crypto

For ESP and AH, we need to allocate an extra variable space in addition
to struct tdb_crypto. The fixed size of pool items may be larger than
an actual requisite size of a buffer, but still the performance
improvement by replacing malloc with pool wins.

Hold a reference to an SP during opencrypto processing

An SP has a list of isr (ipsecrequest) that represents a sequence
of IPsec encryption/authentication processing. One isr corresponds
to one opencrypto processing. The lifetime of an isr follows its SP.

We pass an isr to a callback function of opencrypto to continue
to a next encryption/authentication processing. However nobody
guaranteed that the isr wasn't freed, i.e., its SP wasn't destroyed.

In order to avoid such unexpected destruction of isr, hold a reference
to its SP during opencrypto processing.

Don't bother the case of crp->crp_buf == NULL in callbacks

Don't release sav if calling crypto_dispatch again

Prepare to stop using isr->sav

isr is a shared resource and using isr->sav as a temporal storage
for each packet processing is racy. And also having a reference from
isr to sav makes the lifetime of sav non-deterministic; such a reference
is removed when a packet is processed and isr->sav is overwritten by
new one. Let's have a sav locally for each packet processing instead of
using shared isr->sav.

However this change doesn't stop using isr->sav yet because there are
some users of isr->sav. isr->sav will be removed after the users find
a way to not use isr->sav.

Pass sav directly to opencrypto callback

In a callback, use a passed sav as-is by default and look up a sav
only if the passed sav is dead.

Rename key_alloc* functions (NFC)

We shouldn't use the term "alloc" for functions that just look up
data and actually don't allocate memory.

Remove codes for PACKET_TAG_IPSEC_IN_CRYPTO_DONE

It seems that PACKET_TAG_IPSEC_IN_CRYPTO_DONE is for network adapters
that have IPsec accelerators; a driver sets the mtag to a packet
when its device has already encrypted the packet.

Unfortunately no driver implements such offload features for long
years and seems unlikely to implement them soon. (Note that neither
FreeBSD nor Linux doesn't have such drivers.) Let's remove related
(unused) codes and simplify the IPsec code.

Apply C99-style struct initialization to xformsw

branches: 1.38.2;
Make ipsec_address() and ipsec_logsastr() mpsafe.

branches: 1.37.2;
Retire ipsec_osdep.h

We don't need to care other OSes (FreeBSD) anymore.

Some macros are alive in ipsec_private.h.

Convert IPSEC_ASSERT to KASSERT or KASSERTMSG

IPSEC_ASSERT just discarded specified message...

Remove __FreeBSD__ and __NetBSD__ switches

No functional changes (except for a debug printf).

Note that there remain some __FreeBSD__ for sysctl knobs which counerparts
to NetBSD don't exist. And ipsec_osdep.h isn't touched yet; tidying it up
requires actual code changes.

cosmetic fixes:
- __func__ in printfs
- no space after sizeof
- eliminate useless casts
- u_intX_t -> uintX_t

Redo the statistics through an indirection array and put the definitions
of the arrays in pfkeyv2.h so that they are next to the index definitions.
Remove "bogus" comment about compressing the statistics which is now fixed.

Prepare netipsec for rump-ification

- Include "opt_*.h" only if _KERNEL_OPT is defined
- Allow encapinit to be called twice (by ifinit and ipe4_attach)
- ifinit didn't call encapinit if IPSEC is enabled (ipe4_attach called
it instead), however, on a rump kernel ipe4_attach may not be called
even if IPSEC is enabled. So we need to allow ifinit to call it anyway
- Setup sysctls in ipsec_attach explicitly instead of using SYSCTL_SETUP
- Call ip6flow_invalidate_all in key_spdadd only if in6_present
- It's possible that a rump kernel loads the ipsec library but not
the inet6 library

branches: 1.31.6; 1.31.10; 1.31.14;
- apply some __diagused
- remove unused variables
- move some variables inside their relevant use #ifdef

branches: 1.30.2;
PR/47886: Dr. Wolfgang Stukenbrock: IPSEC_NAT_T enabled kernels may access
outdated pointers and pass ESP data to UPD-sockets.
While here, simplify the code and remove the IPSEC_NAT_T option; always
compile nat-traversal in so that it does not bitrot.

branches: 1.29.6;
Make sure the mbufs in the input path (only the parts which we are going
to modify in the AH case) are writable/non-shared.
This addresses PR kern/33162 by Jeff Rizzo, and replaces the insufficient
patch from that time by a radical solution.
(The PR's problem had been worked around by rev.1.3 of xennetback_xenbus.c,
so it needs a network driver modification to reproduce it.)
Being here, clarify a bit of ipcomp -- uncompression is done in-place,
the header must be removed explicitly.

branches: 1.28.4; 1.28.8;
As a first step towards more fine-grained locking, don't require
crypto_{new.free}session() to be called with the "crypto_mtx"
spinlock held.
This doesn't change much for now because these functions acquire
the said mutex first on entry now, but at least it keeps the nasty
locks local to the opencrypto core.

fix C&P botch in diagnostic printfs

mitigation for CVE-2011-1547

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

more "const"

sprinkle some "const", documenting that the SA is not supposed to
change during an xform operation

change locking order, to make sure the cpu is at splsoftnet()
before the softnet_lock (adaptive) mutex is acquired, from
Wolfgang Stukenbrock, should fix a recursive lock panic

-in opencrypto callbacks (which run in a kernel thread), pull softnet_lock
everywhere splsoftnet() was used before, to fix MP concurrency problems
-pull KERNEL_LOCK where ip(6)_output() is called, as this is what
the network stack (unfortunately) expects, in particular to avoid
races for packets in the interface send queues
From Wolfgang Stukenbrock per PR kern/44418, with the application
of KERNEL_LOCK to what I think are the essential points, tested
on a dual-core i386.

branches: 1.20.2; 1.20.4;
Fix ipcomp input counter

Reported Wolfgang Stukenbrock in pr/43250.

branches: 1.19.2; 1.19.4;
bzero -> memset

branches: 1.18.2; 1.18.10; 1.18.12; 1.18.16; 1.18.18; 1.18.22;
Make IPSEC and FAST_IPSEC stats per-cpu. Use <net/net_stats.h> and
netstat_sysctl().

branches: 1.17.6; 1.17.8;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

Add some statistics for case where compression is not useful
(when len(compressed packet) > len(initial packet))

branches: 1.15.6; 1.15.12;
Fix my previous stupid caddr_t fix.

branches: 1.14.6; 1.14.8;
Add support for options IPSEC_NAT_T (RFC 3947 and 3948) for fast_ipsec(4).

No objection on tech-net@

branches: 1.13.2; 1.13.4;
Remove useless cast
Use NULL instead of (void*) 0

Fix fallout from caddr_t changes

Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

Oops, I forgot to commit some bits last time

fast_ipsec and ipcomp works better now.

branches: 1.9.2;
Commit my SoC work
Add ipv6 support for fast_ipsec
Note that currently, packet with extensions headers are not correctly
supported
Change the ipcomp logic

branches: 1.8.2; 1.8.4;
__unused removal on arguments; approved by core.

more __unused

branches: 1.6.20; 1.6.22;
merge ktrace-lwp.

branches: 1.5.4;
nuke trailing whitespace

branches: 1.4.4; 1.4.10; 1.4.12;
Reversion of "netkey merge", part 2 (replacement of removed files in the
repository by christos was part 1). netipsec should now be back as it
was on 2003-09-11, with some very minor changes:

1) Some residual platform-dependent code was moved from ipsec.h to
ipsec_osdep.h; without this, IPSEC_ASSERT() was multiply defined. ipsec.h
now includes ipsec_osdep.h

2) itojun's renaming of netipsec/files.ipsec to netipsec/files.netipsec has
been left in place (it's arguable which name is less confusing but the
rename is pretty harmless).

3) Some #endif TOKEN has been replaced by #endif /* TOKEN */; #endif TOKEN
is invalid and GCC 3 won't compile it.

An i386 kernel with "options FAST_IPSEC" and "options OPENCRYPTO" now
gets through "make depend" but fails to build with errors in ip_input.c.
But it's better than it was (thank heaven for small favors).

merge netipsec/key* into netkey/key*. no need for both.
change confusing filename

opt_inet6.h is FreeBSD-specific, so wrap it with #ifdef __FreeBSD__/#endif.

Initial import of Sam Leffler's `Fast-IPsec' from FreeBSD 4.
Fast-IPsec is a rework of the OpenBSD and KAME IPsec code, using the
OpenCryptoFramework (and thus hardware crypto accelerators) and
numerous detailed performance improvements.

This import is (aside from SPL-level names) the FreeBSD source,
imported ``as-is'' as a historical snapshot, for future maintenance
and comparison against the FreeBSD source. For now, several minor
kernel-API differences are hidden by macros a shim file, ipsec_osdep.h,
which (aside from SPL names) can be targeted at either NetBSD or FreeBSD.

Merge the [pgoyette-compat] branch

Remove unused calls to nat_t_ports_get.

Remove unused 'mp' argument from all the xf_output functions. Also clean
up xform.h a bit.

Remove double include, opencrypto/xform.h is already included in
netipsec/xform.h.

Remove IPSEC_SPLASSERT_SOFTNET, it has always been a no-op.

Remove extra long file paths from the headers.

Add a KASSERT (which is not triggerable since ipsec_common_input already
ensures 8 bytes are present), add an XXX (about the fact that it is
better to use m_copydata, because it is faster and less error-prone), and
improve two m_copybacks (remove useless casts).

Fix the computation. Normally that's harmless since ip6_output recomputes
ip6_plen.

branches: 1.59.2;
Add [ah/esp/ipcomp]_enable sysctls, and remove the FreeBSD #ifdefs.
Discussed with ozaki-r@.

Remove some more FreeBSD sysctl declarations that already have NetBSD
counterparts. Discussed with ozaki-r@.

Style and simplify.

Don't relook up an SP/SA in opencrpyto callbacks

We don't need to do so because we have a reference to it. And also
relooking-up one there may return an sp/sav that has different
parameters from an original one.

Dedup common codes in error paths (NFCI)

Fix mbuf leaks on error paths

Pointed out by maxv@

Constify isr at many places (NFC)

Use pool_cache(9) instead of pool(9) for tdb_crypto objects

The change improves network throughput especially on multi-core systems.

MP-ify SAD (savlist)

localcount(9) is used to protect savlist of sah. The basic design is
similar to MP-ifications of SPD and SAD sahlist. Please read the
locking notes of SAD for more details.

Introduce KEY_SA_UNREF and replace KEY_FREESAV with it where sav will never be actually freed in the future

KEY_SA_UNREF is still key_freesav so no functional change for now.

This change reduces diff of further changes.

Make IPsec SPD MP-safe

We use localcount(9), not psref(9), to make the sptree and secpolicy (SP)
entries MP-safe because SPs need to be referenced over opencrypto
processing that executes a callback in a different context.

SPs on sockets aren't managed by the sptree and can be destroyed in softint.
localcount_drain cannot be used in softint so we delay the destruction of
such SPs to a thread context. To do so, a list to manage such SPs is added
(key_socksplist) and key_timehandler_spd deletes dead SPs in the list.

For more details please read the locking notes in key.c.

Proposed on tech-kern@ and tech-net@

Don't acquire global locks for IPsec if NET_MPSAFE

Note that the change is just to make testing easy and IPsec isn't MP-safe yet.

Use pool to allocate tdb_crypto

For ESP and AH, we need to allocate an extra variable space in addition
to struct tdb_crypto. The fixed size of pool items may be larger than
an actual requisite size of a buffer, but still the performance
improvement by replacing malloc with pool wins.

Hold a reference to an SP during opencrypto processing

An SP has a list of isr (ipsecrequest) that represents a sequence
of IPsec encryption/authentication processing. One isr corresponds
to one opencrypto processing. The lifetime of an isr follows its SP.

We pass an isr to a callback function of opencrypto to continue
to a next encryption/authentication processing. However nobody
guaranteed that the isr wasn't freed, i.e., its SP wasn't destroyed.

In order to avoid such unexpected destruction of isr, hold a reference
to its SP during opencrypto processing.

Don't bother the case of crp->crp_buf == NULL in callbacks

Don't release sav if calling crypto_dispatch again

Prepare to stop using isr->sav

isr is a shared resource and using isr->sav as a temporal storage
for each packet processing is racy. And also having a reference from
isr to sav makes the lifetime of sav non-deterministic; such a reference
is removed when a packet is processed and isr->sav is overwritten by
new one. Let's have a sav locally for each packet processing instead of
using shared isr->sav.

However this change doesn't stop using isr->sav yet because there are
some users of isr->sav. isr->sav will be removed after the users find
a way to not use isr->sav.

Pass sav directly to opencrypto callback

In a callback, use a passed sav as-is by default and look up a sav
only if the passed sav is dead.

Rename key_alloc* functions (NFC)

We shouldn't use the term "alloc" for functions that just look up
data and actually don't allocate memory.

Remove codes for PACKET_TAG_IPSEC_IN_CRYPTO_DONE

It seems that PACKET_TAG_IPSEC_IN_CRYPTO_DONE is for network adapters
that have IPsec accelerators; a driver sets the mtag to a packet
when its device has already encrypted the packet.

Unfortunately no driver implements such offload features for long
years and seems unlikely to implement them soon. (Note that neither
FreeBSD nor Linux doesn't have such drivers.) Let's remove related
(unused) codes and simplify the IPsec code.

Apply C99-style struct initialization to xformsw

branches: 1.38.2;
Make ipsec_address() and ipsec_logsastr() mpsafe.

branches: 1.37.2;
Retire ipsec_osdep.h

We don't need to care other OSes (FreeBSD) anymore.

Some macros are alive in ipsec_private.h.

Convert IPSEC_ASSERT to KASSERT or KASSERTMSG

IPSEC_ASSERT just discarded specified message...

Remove __FreeBSD__ and __NetBSD__ switches

No functional changes (except for a debug printf).

Note that there remain some __FreeBSD__ for sysctl knobs which counerparts
to NetBSD don't exist. And ipsec_osdep.h isn't touched yet; tidying it up
requires actual code changes.

cosmetic fixes:
- __func__ in printfs
- no space after sizeof
- eliminate useless casts
- u_intX_t -> uintX_t

Redo the statistics through an indirection array and put the definitions
of the arrays in pfkeyv2.h so that they are next to the index definitions.
Remove "bogus" comment about compressing the statistics which is now fixed.

Prepare netipsec for rump-ification

- Include "opt_*.h" only if _KERNEL_OPT is defined
- Allow encapinit to be called twice (by ifinit and ipe4_attach)
- ifinit didn't call encapinit if IPSEC is enabled (ipe4_attach called
it instead), however, on a rump kernel ipe4_attach may not be called
even if IPSEC is enabled. So we need to allow ifinit to call it anyway
- Setup sysctls in ipsec_attach explicitly instead of using SYSCTL_SETUP
- Call ip6flow_invalidate_all in key_spdadd only if in6_present
- It's possible that a rump kernel loads the ipsec library but not
the inet6 library

branches: 1.31.6; 1.31.10; 1.31.14;
- apply some __diagused
- remove unused variables
- move some variables inside their relevant use #ifdef

branches: 1.30.2;
PR/47886: Dr. Wolfgang Stukenbrock: IPSEC_NAT_T enabled kernels may access
outdated pointers and pass ESP data to UPD-sockets.
While here, simplify the code and remove the IPSEC_NAT_T option; always
compile nat-traversal in so that it does not bitrot.

branches: 1.29.6;
Make sure the mbufs in the input path (only the parts which we are going
to modify in the AH case) are writable/non-shared.
This addresses PR kern/33162 by Jeff Rizzo, and replaces the insufficient
patch from that time by a radical solution.
(The PR's problem had been worked around by rev.1.3 of xennetback_xenbus.c,
so it needs a network driver modification to reproduce it.)
Being here, clarify a bit of ipcomp -- uncompression is done in-place,
the header must be removed explicitly.

branches: 1.28.4; 1.28.8;
As a first step towards more fine-grained locking, don't require
crypto_{new.free}session() to be called with the "crypto_mtx"
spinlock held.
This doesn't change much for now because these functions acquire
the said mutex first on entry now, but at least it keeps the nasty
locks local to the opencrypto core.

fix C&P botch in diagnostic printfs

mitigation for CVE-2011-1547

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

more "const"

sprinkle some "const", documenting that the SA is not supposed to
change during an xform operation

change locking order, to make sure the cpu is at splsoftnet()
before the softnet_lock (adaptive) mutex is acquired, from
Wolfgang Stukenbrock, should fix a recursive lock panic

-in opencrypto callbacks (which run in a kernel thread), pull softnet_lock
everywhere splsoftnet() was used before, to fix MP concurrency problems
-pull KERNEL_LOCK where ip(6)_output() is called, as this is what
the network stack (unfortunately) expects, in particular to avoid
races for packets in the interface send queues
From Wolfgang Stukenbrock per PR kern/44418, with the application
of KERNEL_LOCK to what I think are the essential points, tested
on a dual-core i386.

branches: 1.20.2; 1.20.4;
Fix ipcomp input counter

Reported Wolfgang Stukenbrock in pr/43250.

branches: 1.19.2; 1.19.4;
bzero -> memset

branches: 1.18.2; 1.18.10; 1.18.12; 1.18.16; 1.18.18; 1.18.22;
Make IPSEC and FAST_IPSEC stats per-cpu. Use <net/net_stats.h> and
netstat_sysctl().

branches: 1.17.6; 1.17.8;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

Add some statistics for case where compression is not useful
(when len(compressed packet) > len(initial packet))

branches: 1.15.6; 1.15.12;
Fix my previous stupid caddr_t fix.

branches: 1.14.6; 1.14.8;
Add support for options IPSEC_NAT_T (RFC 3947 and 3948) for fast_ipsec(4).

No objection on tech-net@

branches: 1.13.2; 1.13.4;
Remove useless cast
Use NULL instead of (void*) 0

Fix fallout from caddr_t changes

Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

Oops, I forgot to commit some bits last time

fast_ipsec and ipcomp works better now.

branches: 1.9.2;
Commit my SoC work
Add ipv6 support for fast_ipsec
Note that currently, packet with extensions headers are not correctly
supported
Change the ipcomp logic

branches: 1.8.2; 1.8.4;
__unused removal on arguments; approved by core.

more __unused

branches: 1.6.20; 1.6.22;
merge ktrace-lwp.

branches: 1.5.4;
nuke trailing whitespace

branches: 1.4.4; 1.4.10; 1.4.12;
Reversion of "netkey merge", part 2 (replacement of removed files in the
repository by christos was part 1). netipsec should now be back as it
was on 2003-09-11, with some very minor changes:

1) Some residual platform-dependent code was moved from ipsec.h to
ipsec_osdep.h; without this, IPSEC_ASSERT() was multiply defined. ipsec.h
now includes ipsec_osdep.h

2) itojun's renaming of netipsec/files.ipsec to netipsec/files.netipsec has
been left in place (it's arguable which name is less confusing but the
rename is pretty harmless).

3) Some #endif TOKEN has been replaced by #endif /* TOKEN */; #endif TOKEN
is invalid and GCC 3 won't compile it.

An i386 kernel with "options FAST_IPSEC" and "options OPENCRYPTO" now
gets through "make depend" but fails to build with errors in ip_input.c.
But it's better than it was (thank heaven for small favors).

merge netipsec/key* into netkey/key*. no need for both.
change confusing filename

opt_inet6.h is FreeBSD-specific, so wrap it with #ifdef __FreeBSD__/#endif.

Initial import of Sam Leffler's `Fast-IPsec' from FreeBSD 4.
Fast-IPsec is a rework of the OpenBSD and KAME IPsec code, using the
OpenCryptoFramework (and thus hardware crypto accelerators) and
numerous detailed performance improvements.

This import is (aside from SPL-level names) the FreeBSD source,
imported ``as-is'' as a historical snapshot, for future maintenance
and comparison against the FreeBSD source. For now, several minor
kernel-API differences are hidden by macros a shim file, ipsec_osdep.h,
which (aside from SPL names) can be targeted at either NetBSD or FreeBSD.

Constify isr at many places (NFC)

Use pool_cache(9) instead of pool(9) for tdb_crypto objects

The change improves network throughput especially on multi-core systems.

MP-ify SAD (savlist)

localcount(9) is used to protect savlist of sah. The basic design is
similar to MP-ifications of SPD and SAD sahlist. Please read the
locking notes of SAD for more details.

Introduce KEY_SA_UNREF and replace KEY_FREESAV with it where sav will never be actually freed in the future

KEY_SA_UNREF is still key_freesav so no functional change for now.

This change reduces diff of further changes.

Make IPsec SPD MP-safe

We use localcount(9), not psref(9), to make the sptree and secpolicy (SP)
entries MP-safe because SPs need to be referenced over opencrypto
processing that executes a callback in a different context.

SPs on sockets aren't managed by the sptree and can be destroyed in softint.
localcount_drain cannot be used in softint so we delay the destruction of
such SPs to a thread context. To do so, a list to manage such SPs is added
(key_socksplist) and key_timehandler_spd deletes dead SPs in the list.

For more details please read the locking notes in key.c.

Proposed on tech-kern@ and tech-net@

Don't acquire global locks for IPsec if NET_MPSAFE

Note that the change is just to make testing easy and IPsec isn't MP-safe yet.

Use pool to allocate tdb_crypto

For ESP and AH, we need to allocate an extra variable space in addition
to struct tdb_crypto. The fixed size of pool items may be larger than
an actual requisite size of a buffer, but still the performance
improvement by replacing malloc with pool wins.

Hold a reference to an SP during opencrypto processing

An SP has a list of isr (ipsecrequest) that represents a sequence
of IPsec encryption/authentication processing. One isr corresponds
to one opencrypto processing. The lifetime of an isr follows its SP.

We pass an isr to a callback function of opencrypto to continue
to a next encryption/authentication processing. However nobody
guaranteed that the isr wasn't freed, i.e., its SP wasn't destroyed.

In order to avoid such unexpected destruction of isr, hold a reference
to its SP during opencrypto processing.

Don't bother the case of crp->crp_buf == NULL in callbacks

Don't release sav if calling crypto_dispatch again

Prepare to stop using isr->sav

isr is a shared resource and using isr->sav as a temporal storage
for each packet processing is racy. And also having a reference from
isr to sav makes the lifetime of sav non-deterministic; such a reference
is removed when a packet is processed and isr->sav is overwritten by
new one. Let's have a sav locally for each packet processing instead of
using shared isr->sav.

However this change doesn't stop using isr->sav yet because there are
some users of isr->sav. isr->sav will be removed after the users find
a way to not use isr->sav.

Pass sav directly to opencrypto callback

In a callback, use a passed sav as-is by default and look up a sav
only if the passed sav is dead.

Rename key_alloc* functions (NFC)

We shouldn't use the term "alloc" for functions that just look up
data and actually don't allocate memory.

Remove codes for PACKET_TAG_IPSEC_IN_CRYPTO_DONE

It seems that PACKET_TAG_IPSEC_IN_CRYPTO_DONE is for network adapters
that have IPsec accelerators; a driver sets the mtag to a packet
when its device has already encrypted the packet.

Unfortunately no driver implements such offload features for long
years and seems unlikely to implement them soon. (Note that neither
FreeBSD nor Linux doesn't have such drivers.) Let's remove related
(unused) codes and simplify the IPsec code.

Apply C99-style struct initialization to xformsw

Make ipsec_address() and ipsec_logsastr() mpsafe.

branches: 1.37.2;
Retire ipsec_osdep.h

We don't need to care other OSes (FreeBSD) anymore.

Some macros are alive in ipsec_private.h.

Convert IPSEC_ASSERT to KASSERT or KASSERTMSG

IPSEC_ASSERT just discarded specified message...

Remove __FreeBSD__ and __NetBSD__ switches

No functional changes (except for a debug printf).

Note that there remain some __FreeBSD__ for sysctl knobs which counerparts
to NetBSD don't exist. And ipsec_osdep.h isn't touched yet; tidying it up
requires actual code changes.

cosmetic fixes:
- __func__ in printfs
- no space after sizeof
- eliminate useless casts
- u_intX_t -> uintX_t

Redo the statistics through an indirection array and put the definitions
of the arrays in pfkeyv2.h so that they are next to the index definitions.
Remove "bogus" comment about compressing the statistics which is now fixed.

Prepare netipsec for rump-ification

- Include "opt_*.h" only if _KERNEL_OPT is defined
- Allow encapinit to be called twice (by ifinit and ipe4_attach)
- ifinit didn't call encapinit if IPSEC is enabled (ipe4_attach called
it instead), however, on a rump kernel ipe4_attach may not be called
even if IPSEC is enabled. So we need to allow ifinit to call it anyway
- Setup sysctls in ipsec_attach explicitly instead of using SYSCTL_SETUP
- Call ip6flow_invalidate_all in key_spdadd only if in6_present
- It's possible that a rump kernel loads the ipsec library but not
the inet6 library

branches: 1.31.6; 1.31.10; 1.31.14;
- apply some __diagused
- remove unused variables
- move some variables inside their relevant use #ifdef

branches: 1.30.2;
PR/47886: Dr. Wolfgang Stukenbrock: IPSEC_NAT_T enabled kernels may access
outdated pointers and pass ESP data to UPD-sockets.
While here, simplify the code and remove the IPSEC_NAT_T option; always
compile nat-traversal in so that it does not bitrot.

branches: 1.29.6;
Make sure the mbufs in the input path (only the parts which we are going
to modify in the AH case) are writable/non-shared.
This addresses PR kern/33162 by Jeff Rizzo, and replaces the insufficient
patch from that time by a radical solution.
(The PR's problem had been worked around by rev.1.3 of xennetback_xenbus.c,
so it needs a network driver modification to reproduce it.)
Being here, clarify a bit of ipcomp -- uncompression is done in-place,
the header must be removed explicitly.

branches: 1.28.4; 1.28.8;
As a first step towards more fine-grained locking, don't require
crypto_{new.free}session() to be called with the "crypto_mtx"
spinlock held.
This doesn't change much for now because these functions acquire
the said mutex first on entry now, but at least it keeps the nasty
locks local to the opencrypto core.

fix C&P botch in diagnostic printfs

mitigation for CVE-2011-1547

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

more "const"

sprinkle some "const", documenting that the SA is not supposed to
change during an xform operation

change locking order, to make sure the cpu is at splsoftnet()
before the softnet_lock (adaptive) mutex is acquired, from
Wolfgang Stukenbrock, should fix a recursive lock panic

-in opencrypto callbacks (which run in a kernel thread), pull softnet_lock
everywhere splsoftnet() was used before, to fix MP concurrency problems
-pull KERNEL_LOCK where ip(6)_output() is called, as this is what
the network stack (unfortunately) expects, in particular to avoid
races for packets in the interface send queues
From Wolfgang Stukenbrock per PR kern/44418, with the application
of KERNEL_LOCK to what I think are the essential points, tested
on a dual-core i386.

branches: 1.20.2; 1.20.4;
Fix ipcomp input counter

Reported Wolfgang Stukenbrock in pr/43250.

branches: 1.19.2; 1.19.4;
bzero -> memset

branches: 1.18.2; 1.18.10; 1.18.12; 1.18.16; 1.18.18; 1.18.22;
Make IPSEC and FAST_IPSEC stats per-cpu. Use <net/net_stats.h> and
netstat_sysctl().

branches: 1.17.6; 1.17.8;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.