opencrypto/cryptodev.h: Fix includes.

- Move sys/condvar.h under #ifdef _KERNEL.
- Add some other necessary includes and forward declarations.
- Sort.

opencrypto: crypto_dispatch never fails now. Make it return void.

Same with crypto_kdispatch.

opencrypto: Rip out EAGAIN logic when unregistering crypto drivers.

I'm pretty sure this never worked reliably based on code inspection,
and it's unlikely to have ever been tested because it only applies
when unregistering a driver -- but we have no crypto drivers for
removable devices, so it would only apply if we went out of our way
to trigger detach with drvctl.

Instead, just make the operation fail with ENODEV, and remove all the
callback logic to resubmit the request on EAGAIN. (Maybe this should
be ENXIO, but crypto_kdispatch already does ENODEV.)

opencrypto: Make crypto_freesession return void.

No callers use the return value. It is not sensible to allow this to
fail.

opencrypto: Make freesession callback return void.

No functional change intended: all drivers already return zero
unconditionally.

opencrypto: Make crp_callback, krp_callback return void.

Nothing uses the return values inside opencrypto, so let's stop
making users return them.

opencrypto: Nix CRYPTO_F_DONE.

Nothing uses it any more.

opencrypto: Make sid=0 always invalid, but OK to free.

Previously, crypto_newsession could sometimes return 0 as the
driver-specific part of the session id, and 0 as the hid, for sid=0.
But netipsec assumes that it is always safe to free sid=0 from
zero-initialized memory even if crypto_newsession has never
succeeded. So it was up to every driver in tree to gracefully handle
sid=0, if it happened to get assigned hid=0. And, as long as the
freesession callback was expected to just return an error code when
given a bogus session id, that worked out fine...because nothing ever
used the error code.

That was a terrible fragile system that should never have been
invented. Instead, let's just ensure that valid session ids are
nonzero, and make crypto_freesession with sid=0 be a no-op.

opencrypto: Nix CRYPTO_F_USER, CRYPTO_F_CBIMM, CRYPTO_F_CBIFSYNC.

CRYPTO_F_USER is no longer needed. It was introduced in 2008 by
darran@ in crypto.c 1.30, cryptodev.c 1.45 in an attempt to avoid
double-free between the issuing thread and asynchronous callback.
But the `fix' didn't work. In 2017, knakahara@ fixed it properly in
cryptodev.c 1.87 by distinguishing `the crypto operation has
completed' (CRYPTO_F_DONE) from `the callback is done touching the
crp object' (CRYPTO_F_DQRETQ, now renamed to CRYPTODEV_F_RET).

CRYPTO_F_CBIMM formerly served to invoke the callback synchronously
from the driver's interrupt completion routine, to reduce contention
on what was once a single cryptoret thread. Now, there is a per-CPU
queue and softint for much cheaper processing, so there is less
motivation for this in the first place. So let's remove the
complicated logic. This means the callbacks never run in hard
interrupt context, which means we don't need to worry about recursion
into crypto_dispatch in hard interrupt context.

fix typo in CRK_ALGORITHM_MIN definition to match CRK_ALGORITHM_MAX one.
while here fix few typos in comments.

fix typos in asymmetry, asymmetric(al), symmetrical.

HMAC-SHA-512 has 32 bytes MAC.

branches: 1.39.4;
update locking notes of opencrypto(9)

branches: 1.38.2;
make cryptoret() context softint to balance dequeuing crypto_ret_q with enqueuing it.

Divide crp_devflags from crp_flags to write exclusively.

CRYPTO_F_DQRETQ(new name is CRYPTODEV_F_RET) is used by cryptodev.c only.
It should be divided to other member.

add locking notes.

restructure locks(1/2): make relation between lock and data explicit.

+ crypto_drv_mtx protects
- whole crypto_drivers
+ crypto_drivers[i].cc_lock (new) protects
- crypto_drivers[i] itself
- member of crypto_drivers[i]
+ crypto_q_mtx protects
- crp_q
- crp_kq
+ crypto_ret_q_mtx protects
- crp_ret_q
- crp_ret_kq
- crypto_exit_flag

I will add locking note later.

branches: 1.34.2;
add cryptkop alloc/free KPI instead of manipulating cryptkop_pool directly.

remove obsoleted declarations.

opencrypto: cleanup debug messages.

branches: 1.31.2;
separate crypto_drv_mtx from crypto_mtx.

crypto_mtx is used only for cryptodev.c and ocryptodev.c now.

fix race among crypto_done(), cryptoret(), and {cryptodev_op(), cryptodev_key()}.

crypto_op() waited to be set CRYPTO_F_DONE with crp->crp_cv.
However, there is context switch chances between being set CRYPTO_F_DONE in
crypto_done() and done cv_signal(crp->crp_cv) in cryptodev_cb(), that is,
cryptodev_op() thread can run to cv_destroy(crp->crp_cv) before cryptoret()
thread is waken up. As a result, cryptodev_cb() can call invalid(destroyed)
cv_signal(crp->crp_cv).

Furthermore, below two implementations cause other races.
- waiting CRYPTO_F_DONE with crp->crp_cv
- context witch chances between set CRYPTO_F_DONE and cv_signal(crp->crp_cv)

So, use other flag(CRYPTO_F_DQRETQ) for cryptodev_op() and cryptodev_key(),
and then call cv_signal(crp->crp_cv) immediately after set CRYPTO_F_DQRETQ.

Tested concurrent over 20 processes with software and hardware drivers.

add sysctl to select software/hardware encryption driver. can enable CRYPTO_DEBUG.

branches: 1.28.2; 1.28.4;
KNF. Remove extra spaces. No functional change.

Needs sys/time.h for struct timespec.

Re-work the module init and destroy code to allow it to be unloaded and
then reloaded.

Should fix PR kern/49842

branches: 1.25.12; 1.25.30;
-if an opencrypto(9) session is allocated, the driver is refcounted
and can not disappear -- no need to hold crypto_mtx to check the
driver list
(the whole check is questionable)
-crp->crp_cv (the condition variable) is used by userland cryptodev
exclusively -- move its initialization there, no need to waste
cycles of in-kernel callers
-add a comment which members of "struct cryptop" are used
by opencrypto(9) and which by crypto(4)
(this should be split, no need to waste memory for in-kernel callers)

branches: 1.24.2;
pull in AES-GCM/GMAC support from OpenBSD
This is still somewhat experimental. Tested between 2 similar boxes
so far. There is much potential for performance improvement. For now,
I've changed the gmac code to accept any data alignment, as the "char *"
pointer suggests. As the code is practically used, 32-bit alignment
can be assumed, at the cost of data copies. I don't know whether
bytewise access or copies are worse performance-wise. For efficient
implementations using SSE2 instructions on x86, even stricter
alignment requirements might arise.

copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

add an AES-CTR xform, from OpenBSD

remove redundant declarations

make camellia-cbc known to the opencrypto framework

make the use of SHA2-HMAC by FAST_IPSEC compliant to current standards:
-RFC2104 says that the block size of the hash algorithm must be used
for key/ipad/opad calculations. While formerly all ciphers used a block
length of 64, SHA384 and SHA512 use 128 bytes. So we can't use the
HMAC_BLOCK_LEN constant anymore. Add a new field to "struct auth_hash"
for the per-cipher blocksize.
-Due to this, there can't be a single "CRYPTO_SHA2_HMAC" external name
anymore. Replace this by 3 for the 3 different keysizes.
This was done by Open/FreeBSD before.
-Also fix the number of authenticator bits used tor ESP and AH to
conform to RFC4868, and remove uses of AH_HMAC_HASHLEN which did
assume a fixed authenticator size of 12 bytes.

FAST_IPSEC will not interoperate with KAME IPSEC anymore if sha2 is used,
because the latter doesn't implement these standards. It should
interoperate with at least modern Free/OpenBSD now.
(I've only tested with NetBSD-current/FAST_IPSEC on both ends.)

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

branches: 1.17.4; 1.17.6; 1.17.8;
It is best to explicitly include <sys/condvar.h> if we're going to
use kcondvar_t.

Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

branches: 1.15.4;
Fix a race condition in opencrypto where the crypto request could be
completed by the crypto device, queued on the retq, but freed by the
ioctl lwp. The problem manifests as various panics relating to the
condvar inside the request. The problem can occur whenever the crypto
device completes the request immediately and the ioctl skips the cv_wait().

The problem can be reproduced by enabling cryptosoft and running an openssl
speed test. E.g.
sysctl -w kern.cryptodevallowsoft=-1
openssl speed -engine cryptodev -evp des-ede3-cbc -multi 64

Add a macro for TAILQ_FOREACH_REVERSE_SAFE() to queue.h, since this
was missing and the opencrypto code removes requests from a list while
iterating with TAILQ_FOREACH_REVERSE().

Add missing cv_destroy() calls for the key request cleanup.

Reviewed by Thor Lancelot Simon.

branches: 1.14.6; 1.14.8; 1.14.10;
Remove clause 3 and 4 from TNF licenses

branches: 1.13.2; 1.13.4;
Extend crypto.4 interface:

* Asynchronous operation with result retrieval via select/poll
* Mutliple-request submit/retrieve ioctls
* Mutliple-session create-destroy ioctls

Revise/rewrite crypto.4 manual page. It should now be much easier to write
new applications to this API.

Measured performance for trivial requests: 84,000 very short modular math
operations/sec, 120,000 very short md5 hashes per sec (with a hardware
accellerator of moderate performance but very low latency, whose driver
will be contributed at a later date).

Contributed to TNF by Coyote Point Systems, Inc.

branches: 1.12.6;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

From Darran Hunt at Coyote Point: don't truncate HMAC to 96 bits unless
actually asked to.

Fixed in FreeBSD a while ago, discussed on tech-kern and tech-crypto.

This code never worked on a released version of FreeBSD in the form it's
been in in our tree, and certainly does not work on any version of FreeBSD
now. Run through unifdef -D__NetBSD__ -U__FreeBSD__ yielding a small
reduction of size and a dramatic improvement in readability.

No, this does not yield any meaningful decrease in patchability (unlike
mechanical changes that touch live source lines) -- try it and see.

Add constants for modular arithmetic operations other than exponentiation -- there's hardware out there which can do them.

branches: 1.8.16; 1.8.22; 1.8.28;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.7.26;
- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

branches: 1.6.4; 1.6.10;
nuke trailing whitespace

branches: 1.5.4; 1.5.10; 1.5.12;
Pull in FreeBSD sys/opencrypto/cryptodev.h, revision 1.2.2.5 -> 1.2.2.6.
Sam Leffler's FreeBSD commit message was
``to eliminate context switch when returning results from the
software crypto driver''
but the patch also contains the CRYPTO_SESID*() macros used in newer
ubsec and hifn drivers.

Pull up `done' flag for crypto operations from FreeBSD. FreeBSD deltas:
cryptodev.c: 1.4.2.3 -> 1.4.2.4
cryptodev.h: 1.4.2.4 -> 1.4.2.5

Move the initialization of the crypto framework from the userland
pseudo-device to init_main(), so the framework is ready for
registration requests at autoconfiguration time.

Thanks to Quentin Garnier for confirming the change was required, and
for testing a similar fix.

Cleanup traces of previous standalone m_apply()/m_getptr().

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

opencrypto: crypto_dispatch never fails now. Make it return void.

Same with crypto_kdispatch.

opencrypto: Rip out EAGAIN logic when unregistering crypto drivers.

I'm pretty sure this never worked reliably based on code inspection,
and it's unlikely to have ever been tested because it only applies
when unregistering a driver -- but we have no crypto drivers for
removable devices, so it would only apply if we went out of our way
to trigger detach with drvctl.

Instead, just make the operation fail with ENODEV, and remove all the
callback logic to resubmit the request on EAGAIN. (Maybe this should
be ENXIO, but crypto_kdispatch already does ENODEV.)

opencrypto: Make crypto_freesession return void.

No callers use the return value. It is not sensible to allow this to
fail.

opencrypto: Make freesession callback return void.

No functional change intended: all drivers already return zero
unconditionally.

opencrypto: Make crp_callback, krp_callback return void.

Nothing uses the return values inside opencrypto, so let's stop
making users return them.

opencrypto: Nix CRYPTO_F_DONE.

Nothing uses it any more.

opencrypto: Make sid=0 always invalid, but OK to free.

Previously, crypto_newsession could sometimes return 0 as the
driver-specific part of the session id, and 0 as the hid, for sid=0.
But netipsec assumes that it is always safe to free sid=0 from
zero-initialized memory even if crypto_newsession has never
succeeded. So it was up to every driver in tree to gracefully handle
sid=0, if it happened to get assigned hid=0. And, as long as the
freesession callback was expected to just return an error code when
given a bogus session id, that worked out fine...because nothing ever
used the error code.

That was a terrible fragile system that should never have been
invented. Instead, let's just ensure that valid session ids are
nonzero, and make crypto_freesession with sid=0 be a no-op.

opencrypto: Nix CRYPTO_F_USER, CRYPTO_F_CBIMM, CRYPTO_F_CBIFSYNC.

CRYPTO_F_USER is no longer needed. It was introduced in 2008 by
darran@ in crypto.c 1.30, cryptodev.c 1.45 in an attempt to avoid
double-free between the issuing thread and asynchronous callback.
But the `fix' didn't work. In 2017, knakahara@ fixed it properly in
cryptodev.c 1.87 by distinguishing `the crypto operation has
completed' (CRYPTO_F_DONE) from `the callback is done touching the
crp object' (CRYPTO_F_DQRETQ, now renamed to CRYPTODEV_F_RET).

CRYPTO_F_CBIMM formerly served to invoke the callback synchronously
from the driver's interrupt completion routine, to reduce contention
on what was once a single cryptoret thread. Now, there is a per-CPU
queue and softint for much cheaper processing, so there is less
motivation for this in the first place. So let's remove the
complicated logic. This means the callbacks never run in hard
interrupt context, which means we don't need to worry about recursion
into crypto_dispatch in hard interrupt context.

fix typo in CRK_ALGORITHM_MIN definition to match CRK_ALGORITHM_MAX one.
while here fix few typos in comments.

fix typos in asymmetry, asymmetric(al), symmetrical.

HMAC-SHA-512 has 32 bytes MAC.

branches: 1.39.4;
update locking notes of opencrypto(9)

branches: 1.38.2;
make cryptoret() context softint to balance dequeuing crypto_ret_q with enqueuing it.

Divide crp_devflags from crp_flags to write exclusively.

CRYPTO_F_DQRETQ(new name is CRYPTODEV_F_RET) is used by cryptodev.c only.
It should be divided to other member.

add locking notes.

restructure locks(1/2): make relation between lock and data explicit.

+ crypto_drv_mtx protects
- whole crypto_drivers
+ crypto_drivers[i].cc_lock (new) protects
- crypto_drivers[i] itself
- member of crypto_drivers[i]
+ crypto_q_mtx protects
- crp_q
- crp_kq
+ crypto_ret_q_mtx protects
- crp_ret_q
- crp_ret_kq
- crypto_exit_flag

I will add locking note later.

branches: 1.34.2;
add cryptkop alloc/free KPI instead of manipulating cryptkop_pool directly.

remove obsoleted declarations.

opencrypto: cleanup debug messages.

branches: 1.31.2;
separate crypto_drv_mtx from crypto_mtx.

crypto_mtx is used only for cryptodev.c and ocryptodev.c now.

fix race among crypto_done(), cryptoret(), and {cryptodev_op(), cryptodev_key()}.

crypto_op() waited to be set CRYPTO_F_DONE with crp->crp_cv.
However, there is context switch chances between being set CRYPTO_F_DONE in
crypto_done() and done cv_signal(crp->crp_cv) in cryptodev_cb(), that is,
cryptodev_op() thread can run to cv_destroy(crp->crp_cv) before cryptoret()
thread is waken up. As a result, cryptodev_cb() can call invalid(destroyed)
cv_signal(crp->crp_cv).

Furthermore, below two implementations cause other races.
- waiting CRYPTO_F_DONE with crp->crp_cv
- context witch chances between set CRYPTO_F_DONE and cv_signal(crp->crp_cv)

So, use other flag(CRYPTO_F_DQRETQ) for cryptodev_op() and cryptodev_key(),
and then call cv_signal(crp->crp_cv) immediately after set CRYPTO_F_DQRETQ.

Tested concurrent over 20 processes with software and hardware drivers.

add sysctl to select software/hardware encryption driver. can enable CRYPTO_DEBUG.

branches: 1.28.2; 1.28.4;
KNF. Remove extra spaces. No functional change.

Needs sys/time.h for struct timespec.

Re-work the module init and destroy code to allow it to be unloaded and
then reloaded.

Should fix PR kern/49842

branches: 1.25.12; 1.25.30;
-if an opencrypto(9) session is allocated, the driver is refcounted
and can not disappear -- no need to hold crypto_mtx to check the
driver list
(the whole check is questionable)
-crp->crp_cv (the condition variable) is used by userland cryptodev
exclusively -- move its initialization there, no need to waste
cycles of in-kernel callers
-add a comment which members of "struct cryptop" are used
by opencrypto(9) and which by crypto(4)
(this should be split, no need to waste memory for in-kernel callers)

branches: 1.24.2;
pull in AES-GCM/GMAC support from OpenBSD
This is still somewhat experimental. Tested between 2 similar boxes
so far. There is much potential for performance improvement. For now,
I've changed the gmac code to accept any data alignment, as the "char *"
pointer suggests. As the code is practically used, 32-bit alignment
can be assumed, at the cost of data copies. I don't know whether
bytewise access or copies are worse performance-wise. For efficient
implementations using SSE2 instructions on x86, even stricter
alignment requirements might arise.

copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

add an AES-CTR xform, from OpenBSD

remove redundant declarations

make camellia-cbc known to the opencrypto framework

make the use of SHA2-HMAC by FAST_IPSEC compliant to current standards:
-RFC2104 says that the block size of the hash algorithm must be used
for key/ipad/opad calculations. While formerly all ciphers used a block
length of 64, SHA384 and SHA512 use 128 bytes. So we can't use the
HMAC_BLOCK_LEN constant anymore. Add a new field to "struct auth_hash"
for the per-cipher blocksize.
-Due to this, there can't be a single "CRYPTO_SHA2_HMAC" external name
anymore. Replace this by 3 for the 3 different keysizes.
This was done by Open/FreeBSD before.
-Also fix the number of authenticator bits used tor ESP and AH to
conform to RFC4868, and remove uses of AH_HMAC_HASHLEN which did
assume a fixed authenticator size of 12 bytes.

FAST_IPSEC will not interoperate with KAME IPSEC anymore if sha2 is used,
because the latter doesn't implement these standards. It should
interoperate with at least modern Free/OpenBSD now.
(I've only tested with NetBSD-current/FAST_IPSEC on both ends.)

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

branches: 1.17.4; 1.17.6; 1.17.8;
It is best to explicitly include <sys/condvar.h> if we're going to
use kcondvar_t.

Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

branches: 1.15.4;
Fix a race condition in opencrypto where the crypto request could be
completed by the crypto device, queued on the retq, but freed by the
ioctl lwp. The problem manifests as various panics relating to the
condvar inside the request. The problem can occur whenever the crypto
device completes the request immediately and the ioctl skips the cv_wait().

The problem can be reproduced by enabling cryptosoft and running an openssl
speed test. E.g.
sysctl -w kern.cryptodevallowsoft=-1
openssl speed -engine cryptodev -evp des-ede3-cbc -multi 64

Add a macro for TAILQ_FOREACH_REVERSE_SAFE() to queue.h, since this
was missing and the opencrypto code removes requests from a list while
iterating with TAILQ_FOREACH_REVERSE().

Add missing cv_destroy() calls for the key request cleanup.

Reviewed by Thor Lancelot Simon.

branches: 1.14.6; 1.14.8; 1.14.10;
Remove clause 3 and 4 from TNF licenses

branches: 1.13.2; 1.13.4;
Extend crypto.4 interface:

* Asynchronous operation with result retrieval via select/poll
* Mutliple-request submit/retrieve ioctls
* Mutliple-session create-destroy ioctls

Revise/rewrite crypto.4 manual page. It should now be much easier to write
new applications to this API.

Measured performance for trivial requests: 84,000 very short modular math
operations/sec, 120,000 very short md5 hashes per sec (with a hardware
accellerator of moderate performance but very low latency, whose driver
will be contributed at a later date).

Contributed to TNF by Coyote Point Systems, Inc.

branches: 1.12.6;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

From Darran Hunt at Coyote Point: don't truncate HMAC to 96 bits unless
actually asked to.

Fixed in FreeBSD a while ago, discussed on tech-kern and tech-crypto.

This code never worked on a released version of FreeBSD in the form it's
been in in our tree, and certainly does not work on any version of FreeBSD
now. Run through unifdef -D__NetBSD__ -U__FreeBSD__ yielding a small
reduction of size and a dramatic improvement in readability.

No, this does not yield any meaningful decrease in patchability (unlike
mechanical changes that touch live source lines) -- try it and see.

Add constants for modular arithmetic operations other than exponentiation -- there's hardware out there which can do them.

branches: 1.8.16; 1.8.22; 1.8.28;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.7.26;
- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

branches: 1.6.4; 1.6.10;
nuke trailing whitespace

branches: 1.5.4; 1.5.10; 1.5.12;
Pull in FreeBSD sys/opencrypto/cryptodev.h, revision 1.2.2.5 -> 1.2.2.6.
Sam Leffler's FreeBSD commit message was
``to eliminate context switch when returning results from the
software crypto driver''
but the patch also contains the CRYPTO_SESID*() macros used in newer
ubsec and hifn drivers.

Pull up `done' flag for crypto operations from FreeBSD. FreeBSD deltas:
cryptodev.c: 1.4.2.3 -> 1.4.2.4
cryptodev.h: 1.4.2.4 -> 1.4.2.5

Move the initialization of the crypto framework from the userland
pseudo-device to init_main(), so the framework is ready for
registration requests at autoconfiguration time.

Thanks to Quentin Garnier for confirming the change was required, and
for testing a similar fix.

Cleanup traces of previous standalone m_apply()/m_getptr().

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

fix typo in CRK_ALGORITHM_MIN definition to match CRK_ALGORITHM_MAX one.
while here fix few typos in comments.

fix typos in asymmetry, asymmetric(al), symmetrical.

HMAC-SHA-512 has 32 bytes MAC.

branches: 1.39.4;
update locking notes of opencrypto(9)

branches: 1.38.2;
make cryptoret() context softint to balance dequeuing crypto_ret_q with enqueuing it.

Divide crp_devflags from crp_flags to write exclusively.

CRYPTO_F_DQRETQ(new name is CRYPTODEV_F_RET) is used by cryptodev.c only.
It should be divided to other member.

add locking notes.

restructure locks(1/2): make relation between lock and data explicit.

+ crypto_drv_mtx protects
- whole crypto_drivers
+ crypto_drivers[i].cc_lock (new) protects
- crypto_drivers[i] itself
- member of crypto_drivers[i]
+ crypto_q_mtx protects
- crp_q
- crp_kq
+ crypto_ret_q_mtx protects
- crp_ret_q
- crp_ret_kq
- crypto_exit_flag

I will add locking note later.

branches: 1.34.2;
add cryptkop alloc/free KPI instead of manipulating cryptkop_pool directly.

remove obsoleted declarations.

opencrypto: cleanup debug messages.

branches: 1.31.2;
separate crypto_drv_mtx from crypto_mtx.

crypto_mtx is used only for cryptodev.c and ocryptodev.c now.

fix race among crypto_done(), cryptoret(), and {cryptodev_op(), cryptodev_key()}.

crypto_op() waited to be set CRYPTO_F_DONE with crp->crp_cv.
However, there is context switch chances between being set CRYPTO_F_DONE in
crypto_done() and done cv_signal(crp->crp_cv) in cryptodev_cb(), that is,
cryptodev_op() thread can run to cv_destroy(crp->crp_cv) before cryptoret()
thread is waken up. As a result, cryptodev_cb() can call invalid(destroyed)
cv_signal(crp->crp_cv).

Furthermore, below two implementations cause other races.
- waiting CRYPTO_F_DONE with crp->crp_cv
- context witch chances between set CRYPTO_F_DONE and cv_signal(crp->crp_cv)

So, use other flag(CRYPTO_F_DQRETQ) for cryptodev_op() and cryptodev_key(),
and then call cv_signal(crp->crp_cv) immediately after set CRYPTO_F_DQRETQ.

Tested concurrent over 20 processes with software and hardware drivers.

add sysctl to select software/hardware encryption driver. can enable CRYPTO_DEBUG.

branches: 1.28.2; 1.28.4;
KNF. Remove extra spaces. No functional change.

Needs sys/time.h for struct timespec.

Re-work the module init and destroy code to allow it to be unloaded and
then reloaded.

Should fix PR kern/49842

branches: 1.25.12; 1.25.30;
-if an opencrypto(9) session is allocated, the driver is refcounted
and can not disappear -- no need to hold crypto_mtx to check the
driver list
(the whole check is questionable)
-crp->crp_cv (the condition variable) is used by userland cryptodev
exclusively -- move its initialization there, no need to waste
cycles of in-kernel callers
-add a comment which members of "struct cryptop" are used
by opencrypto(9) and which by crypto(4)
(this should be split, no need to waste memory for in-kernel callers)

branches: 1.24.2;
pull in AES-GCM/GMAC support from OpenBSD
This is still somewhat experimental. Tested between 2 similar boxes
so far. There is much potential for performance improvement. For now,
I've changed the gmac code to accept any data alignment, as the "char *"
pointer suggests. As the code is practically used, 32-bit alignment
can be assumed, at the cost of data copies. I don't know whether
bytewise access or copies are worse performance-wise. For efficient
implementations using SSE2 instructions on x86, even stricter
alignment requirements might arise.

copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

add an AES-CTR xform, from OpenBSD

remove redundant declarations

make camellia-cbc known to the opencrypto framework

make the use of SHA2-HMAC by FAST_IPSEC compliant to current standards:
-RFC2104 says that the block size of the hash algorithm must be used
for key/ipad/opad calculations. While formerly all ciphers used a block
length of 64, SHA384 and SHA512 use 128 bytes. So we can't use the
HMAC_BLOCK_LEN constant anymore. Add a new field to "struct auth_hash"
for the per-cipher blocksize.
-Due to this, there can't be a single "CRYPTO_SHA2_HMAC" external name
anymore. Replace this by 3 for the 3 different keysizes.
This was done by Open/FreeBSD before.
-Also fix the number of authenticator bits used tor ESP and AH to
conform to RFC4868, and remove uses of AH_HMAC_HASHLEN which did
assume a fixed authenticator size of 12 bytes.

FAST_IPSEC will not interoperate with KAME IPSEC anymore if sha2 is used,
because the latter doesn't implement these standards. It should
interoperate with at least modern Free/OpenBSD now.
(I've only tested with NetBSD-current/FAST_IPSEC on both ends.)

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

branches: 1.17.4; 1.17.6; 1.17.8;
It is best to explicitly include <sys/condvar.h> if we're going to
use kcondvar_t.

Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

branches: 1.15.4;
Fix a race condition in opencrypto where the crypto request could be
completed by the crypto device, queued on the retq, but freed by the
ioctl lwp. The problem manifests as various panics relating to the
condvar inside the request. The problem can occur whenever the crypto
device completes the request immediately and the ioctl skips the cv_wait().

The problem can be reproduced by enabling cryptosoft and running an openssl
speed test. E.g.
sysctl -w kern.cryptodevallowsoft=-1
openssl speed -engine cryptodev -evp des-ede3-cbc -multi 64

Add a macro for TAILQ_FOREACH_REVERSE_SAFE() to queue.h, since this
was missing and the opencrypto code removes requests from a list while
iterating with TAILQ_FOREACH_REVERSE().

Add missing cv_destroy() calls for the key request cleanup.

Reviewed by Thor Lancelot Simon.

branches: 1.14.6; 1.14.8; 1.14.10;
Remove clause 3 and 4 from TNF licenses

branches: 1.13.2; 1.13.4;
Extend crypto.4 interface:

* Asynchronous operation with result retrieval via select/poll
* Mutliple-request submit/retrieve ioctls
* Mutliple-session create-destroy ioctls

Revise/rewrite crypto.4 manual page. It should now be much easier to write
new applications to this API.

Measured performance for trivial requests: 84,000 very short modular math
operations/sec, 120,000 very short md5 hashes per sec (with a hardware
accellerator of moderate performance but very low latency, whose driver
will be contributed at a later date).

Contributed to TNF by Coyote Point Systems, Inc.

branches: 1.12.6;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

From Darran Hunt at Coyote Point: don't truncate HMAC to 96 bits unless
actually asked to.

Fixed in FreeBSD a while ago, discussed on tech-kern and tech-crypto.

This code never worked on a released version of FreeBSD in the form it's
been in in our tree, and certainly does not work on any version of FreeBSD
now. Run through unifdef -D__NetBSD__ -U__FreeBSD__ yielding a small
reduction of size and a dramatic improvement in readability.

No, this does not yield any meaningful decrease in patchability (unlike
mechanical changes that touch live source lines) -- try it and see.

Add constants for modular arithmetic operations other than exponentiation -- there's hardware out there which can do them.

branches: 1.8.16; 1.8.22; 1.8.28;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.7.26;
- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

branches: 1.6.4; 1.6.10;
nuke trailing whitespace

branches: 1.5.4; 1.5.10; 1.5.12;
Pull in FreeBSD sys/opencrypto/cryptodev.h, revision 1.2.2.5 -> 1.2.2.6.
Sam Leffler's FreeBSD commit message was
``to eliminate context switch when returning results from the
software crypto driver''
but the patch also contains the CRYPTO_SESID*() macros used in newer
ubsec and hifn drivers.

Pull up `done' flag for crypto operations from FreeBSD. FreeBSD deltas:
cryptodev.c: 1.4.2.3 -> 1.4.2.4
cryptodev.h: 1.4.2.4 -> 1.4.2.5

Move the initialization of the crypto framework from the userland
pseudo-device to init_main(), so the framework is ready for
registration requests at autoconfiguration time.

Thanks to Quentin Garnier for confirming the change was required, and
for testing a similar fix.

Cleanup traces of previous standalone m_apply()/m_getptr().

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

fix typos in asymmetry, asymmetric(al), symmetrical.

HMAC-SHA-512 has 32 bytes MAC.

branches: 1.39.4;
update locking notes of opencrypto(9)

branches: 1.38.2;
make cryptoret() context softint to balance dequeuing crypto_ret_q with enqueuing it.

Divide crp_devflags from crp_flags to write exclusively.

CRYPTO_F_DQRETQ(new name is CRYPTODEV_F_RET) is used by cryptodev.c only.
It should be divided to other member.

add locking notes.

restructure locks(1/2): make relation between lock and data explicit.

+ crypto_drv_mtx protects
- whole crypto_drivers
+ crypto_drivers[i].cc_lock (new) protects
- crypto_drivers[i] itself
- member of crypto_drivers[i]
+ crypto_q_mtx protects
- crp_q
- crp_kq
+ crypto_ret_q_mtx protects
- crp_ret_q
- crp_ret_kq
- crypto_exit_flag

I will add locking note later.

branches: 1.34.2;
add cryptkop alloc/free KPI instead of manipulating cryptkop_pool directly.

remove obsoleted declarations.

opencrypto: cleanup debug messages.

branches: 1.31.2;
separate crypto_drv_mtx from crypto_mtx.

crypto_mtx is used only for cryptodev.c and ocryptodev.c now.

fix race among crypto_done(), cryptoret(), and {cryptodev_op(), cryptodev_key()}.

crypto_op() waited to be set CRYPTO_F_DONE with crp->crp_cv.
However, there is context switch chances between being set CRYPTO_F_DONE in
crypto_done() and done cv_signal(crp->crp_cv) in cryptodev_cb(), that is,
cryptodev_op() thread can run to cv_destroy(crp->crp_cv) before cryptoret()
thread is waken up. As a result, cryptodev_cb() can call invalid(destroyed)
cv_signal(crp->crp_cv).

Furthermore, below two implementations cause other races.
- waiting CRYPTO_F_DONE with crp->crp_cv
- context witch chances between set CRYPTO_F_DONE and cv_signal(crp->crp_cv)

So, use other flag(CRYPTO_F_DQRETQ) for cryptodev_op() and cryptodev_key(),
and then call cv_signal(crp->crp_cv) immediately after set CRYPTO_F_DQRETQ.

Tested concurrent over 20 processes with software and hardware drivers.

add sysctl to select software/hardware encryption driver. can enable CRYPTO_DEBUG.

branches: 1.28.2; 1.28.4;
KNF. Remove extra spaces. No functional change.

Needs sys/time.h for struct timespec.

Re-work the module init and destroy code to allow it to be unloaded and
then reloaded.

Should fix PR kern/49842

branches: 1.25.12; 1.25.30;
-if an opencrypto(9) session is allocated, the driver is refcounted
and can not disappear -- no need to hold crypto_mtx to check the
driver list
(the whole check is questionable)
-crp->crp_cv (the condition variable) is used by userland cryptodev
exclusively -- move its initialization there, no need to waste
cycles of in-kernel callers
-add a comment which members of "struct cryptop" are used
by opencrypto(9) and which by crypto(4)
(this should be split, no need to waste memory for in-kernel callers)

branches: 1.24.2;
pull in AES-GCM/GMAC support from OpenBSD
This is still somewhat experimental. Tested between 2 similar boxes
so far. There is much potential for performance improvement. For now,
I've changed the gmac code to accept any data alignment, as the "char *"
pointer suggests. As the code is practically used, 32-bit alignment
can be assumed, at the cost of data copies. I don't know whether
bytewise access or copies are worse performance-wise. For efficient
implementations using SSE2 instructions on x86, even stricter
alignment requirements might arise.

copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

add an AES-CTR xform, from OpenBSD

remove redundant declarations

make camellia-cbc known to the opencrypto framework

make the use of SHA2-HMAC by FAST_IPSEC compliant to current standards:
-RFC2104 says that the block size of the hash algorithm must be used
for key/ipad/opad calculations. While formerly all ciphers used a block
length of 64, SHA384 and SHA512 use 128 bytes. So we can't use the
HMAC_BLOCK_LEN constant anymore. Add a new field to "struct auth_hash"
for the per-cipher blocksize.
-Due to this, there can't be a single "CRYPTO_SHA2_HMAC" external name
anymore. Replace this by 3 for the 3 different keysizes.
This was done by Open/FreeBSD before.
-Also fix the number of authenticator bits used tor ESP and AH to
conform to RFC4868, and remove uses of AH_HMAC_HASHLEN which did
assume a fixed authenticator size of 12 bytes.

FAST_IPSEC will not interoperate with KAME IPSEC anymore if sha2 is used,
because the latter doesn't implement these standards. It should
interoperate with at least modern Free/OpenBSD now.
(I've only tested with NetBSD-current/FAST_IPSEC on both ends.)

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

branches: 1.17.4; 1.17.6; 1.17.8;
It is best to explicitly include <sys/condvar.h> if we're going to
use kcondvar_t.

Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

branches: 1.15.4;
Fix a race condition in opencrypto where the crypto request could be
completed by the crypto device, queued on the retq, but freed by the
ioctl lwp. The problem manifests as various panics relating to the
condvar inside the request. The problem can occur whenever the crypto
device completes the request immediately and the ioctl skips the cv_wait().

The problem can be reproduced by enabling cryptosoft and running an openssl
speed test. E.g.
sysctl -w kern.cryptodevallowsoft=-1
openssl speed -engine cryptodev -evp des-ede3-cbc -multi 64

Add a macro for TAILQ_FOREACH_REVERSE_SAFE() to queue.h, since this
was missing and the opencrypto code removes requests from a list while
iterating with TAILQ_FOREACH_REVERSE().

Add missing cv_destroy() calls for the key request cleanup.

Reviewed by Thor Lancelot Simon.

branches: 1.14.6; 1.14.8; 1.14.10;
Remove clause 3 and 4 from TNF licenses

branches: 1.13.2; 1.13.4;
Extend crypto.4 interface:

* Asynchronous operation with result retrieval via select/poll
* Mutliple-request submit/retrieve ioctls
* Mutliple-session create-destroy ioctls

Revise/rewrite crypto.4 manual page. It should now be much easier to write
new applications to this API.

Measured performance for trivial requests: 84,000 very short modular math
operations/sec, 120,000 very short md5 hashes per sec (with a hardware
accellerator of moderate performance but very low latency, whose driver
will be contributed at a later date).

Contributed to TNF by Coyote Point Systems, Inc.

branches: 1.12.6;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

From Darran Hunt at Coyote Point: don't truncate HMAC to 96 bits unless
actually asked to.

Fixed in FreeBSD a while ago, discussed on tech-kern and tech-crypto.

This code never worked on a released version of FreeBSD in the form it's
been in in our tree, and certainly does not work on any version of FreeBSD
now. Run through unifdef -D__NetBSD__ -U__FreeBSD__ yielding a small
reduction of size and a dramatic improvement in readability.

No, this does not yield any meaningful decrease in patchability (unlike
mechanical changes that touch live source lines) -- try it and see.

Add constants for modular arithmetic operations other than exponentiation -- there's hardware out there which can do them.

branches: 1.8.16; 1.8.22; 1.8.28;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.7.26;
- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

branches: 1.6.4; 1.6.10;
nuke trailing whitespace

branches: 1.5.4; 1.5.10; 1.5.12;
Pull in FreeBSD sys/opencrypto/cryptodev.h, revision 1.2.2.5 -> 1.2.2.6.
Sam Leffler's FreeBSD commit message was
``to eliminate context switch when returning results from the
software crypto driver''
but the patch also contains the CRYPTO_SESID*() macros used in newer
ubsec and hifn drivers.

Pull up `done' flag for crypto operations from FreeBSD. FreeBSD deltas:
cryptodev.c: 1.4.2.3 -> 1.4.2.4
cryptodev.h: 1.4.2.4 -> 1.4.2.5

Move the initialization of the crypto framework from the userland
pseudo-device to init_main(), so the framework is ready for
registration requests at autoconfiguration time.

Thanks to Quentin Garnier for confirming the change was required, and
for testing a similar fix.

Cleanup traces of previous standalone m_apply()/m_getptr().

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

HMAC-SHA-512 has 32 bytes MAC.

update locking notes of opencrypto(9)

branches: 1.38.2;
make cryptoret() context softint to balance dequeuing crypto_ret_q with enqueuing it.

Divide crp_devflags from crp_flags to write exclusively.

CRYPTO_F_DQRETQ(new name is CRYPTODEV_F_RET) is used by cryptodev.c only.
It should be divided to other member.

add locking notes.

restructure locks(1/2): make relation between lock and data explicit.

+ crypto_drv_mtx protects
- whole crypto_drivers
+ crypto_drivers[i].cc_lock (new) protects
- crypto_drivers[i] itself
- member of crypto_drivers[i]
+ crypto_q_mtx protects
- crp_q
- crp_kq
+ crypto_ret_q_mtx protects
- crp_ret_q
- crp_ret_kq
- crypto_exit_flag

I will add locking note later.

branches: 1.34.2;
add cryptkop alloc/free KPI instead of manipulating cryptkop_pool directly.

remove obsoleted declarations.

opencrypto: cleanup debug messages.

branches: 1.31.2;
separate crypto_drv_mtx from crypto_mtx.

crypto_mtx is used only for cryptodev.c and ocryptodev.c now.

fix race among crypto_done(), cryptoret(), and {cryptodev_op(), cryptodev_key()}.

crypto_op() waited to be set CRYPTO_F_DONE with crp->crp_cv.
However, there is context switch chances between being set CRYPTO_F_DONE in
crypto_done() and done cv_signal(crp->crp_cv) in cryptodev_cb(), that is,
cryptodev_op() thread can run to cv_destroy(crp->crp_cv) before cryptoret()
thread is waken up. As a result, cryptodev_cb() can call invalid(destroyed)
cv_signal(crp->crp_cv).

Furthermore, below two implementations cause other races.
- waiting CRYPTO_F_DONE with crp->crp_cv
- context witch chances between set CRYPTO_F_DONE and cv_signal(crp->crp_cv)

So, use other flag(CRYPTO_F_DQRETQ) for cryptodev_op() and cryptodev_key(),
and then call cv_signal(crp->crp_cv) immediately after set CRYPTO_F_DQRETQ.

Tested concurrent over 20 processes with software and hardware drivers.

add sysctl to select software/hardware encryption driver. can enable CRYPTO_DEBUG.

branches: 1.28.2; 1.28.4;
KNF. Remove extra spaces. No functional change.

Needs sys/time.h for struct timespec.

Re-work the module init and destroy code to allow it to be unloaded and
then reloaded.

Should fix PR kern/49842

branches: 1.25.12; 1.25.30;
-if an opencrypto(9) session is allocated, the driver is refcounted
and can not disappear -- no need to hold crypto_mtx to check the
driver list
(the whole check is questionable)
-crp->crp_cv (the condition variable) is used by userland cryptodev
exclusively -- move its initialization there, no need to waste
cycles of in-kernel callers
-add a comment which members of "struct cryptop" are used
by opencrypto(9) and which by crypto(4)
(this should be split, no need to waste memory for in-kernel callers)

branches: 1.24.2;
pull in AES-GCM/GMAC support from OpenBSD
This is still somewhat experimental. Tested between 2 similar boxes
so far. There is much potential for performance improvement. For now,
I've changed the gmac code to accept any data alignment, as the "char *"
pointer suggests. As the code is practically used, 32-bit alignment
can be assumed, at the cost of data copies. I don't know whether
bytewise access or copies are worse performance-wise. For efficient
implementations using SSE2 instructions on x86, even stricter
alignment requirements might arise.

copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

add an AES-CTR xform, from OpenBSD

remove redundant declarations

make camellia-cbc known to the opencrypto framework

make the use of SHA2-HMAC by FAST_IPSEC compliant to current standards:
-RFC2104 says that the block size of the hash algorithm must be used
for key/ipad/opad calculations. While formerly all ciphers used a block
length of 64, SHA384 and SHA512 use 128 bytes. So we can't use the
HMAC_BLOCK_LEN constant anymore. Add a new field to "struct auth_hash"
for the per-cipher blocksize.
-Due to this, there can't be a single "CRYPTO_SHA2_HMAC" external name
anymore. Replace this by 3 for the 3 different keysizes.
This was done by Open/FreeBSD before.
-Also fix the number of authenticator bits used tor ESP and AH to
conform to RFC4868, and remove uses of AH_HMAC_HASHLEN which did
assume a fixed authenticator size of 12 bytes.

FAST_IPSEC will not interoperate with KAME IPSEC anymore if sha2 is used,
because the latter doesn't implement these standards. It should
interoperate with at least modern Free/OpenBSD now.
(I've only tested with NetBSD-current/FAST_IPSEC on both ends.)

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

branches: 1.17.4; 1.17.6; 1.17.8;
It is best to explicitly include <sys/condvar.h> if we're going to
use kcondvar_t.

Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

branches: 1.15.4;
Fix a race condition in opencrypto where the crypto request could be
completed by the crypto device, queued on the retq, but freed by the
ioctl lwp. The problem manifests as various panics relating to the
condvar inside the request. The problem can occur whenever the crypto
device completes the request immediately and the ioctl skips the cv_wait().

The problem can be reproduced by enabling cryptosoft and running an openssl
speed test. E.g.
sysctl -w kern.cryptodevallowsoft=-1
openssl speed -engine cryptodev -evp des-ede3-cbc -multi 64

Add a macro for TAILQ_FOREACH_REVERSE_SAFE() to queue.h, since this
was missing and the opencrypto code removes requests from a list while
iterating with TAILQ_FOREACH_REVERSE().

Add missing cv_destroy() calls for the key request cleanup.

Reviewed by Thor Lancelot Simon.

branches: 1.14.6; 1.14.8; 1.14.10;
Remove clause 3 and 4 from TNF licenses

branches: 1.13.2; 1.13.4;
Extend crypto.4 interface:

* Asynchronous operation with result retrieval via select/poll
* Mutliple-request submit/retrieve ioctls
* Mutliple-session create-destroy ioctls

Revise/rewrite crypto.4 manual page. It should now be much easier to write
new applications to this API.

Measured performance for trivial requests: 84,000 very short modular math
operations/sec, 120,000 very short md5 hashes per sec (with a hardware
accellerator of moderate performance but very low latency, whose driver
will be contributed at a later date).

Contributed to TNF by Coyote Point Systems, Inc.

branches: 1.12.6;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

From Darran Hunt at Coyote Point: don't truncate HMAC to 96 bits unless
actually asked to.

Fixed in FreeBSD a while ago, discussed on tech-kern and tech-crypto.

This code never worked on a released version of FreeBSD in the form it's
been in in our tree, and certainly does not work on any version of FreeBSD
now. Run through unifdef -D__NetBSD__ -U__FreeBSD__ yielding a small
reduction of size and a dramatic improvement in readability.

No, this does not yield any meaningful decrease in patchability (unlike
mechanical changes that touch live source lines) -- try it and see.

Add constants for modular arithmetic operations other than exponentiation -- there's hardware out there which can do them.

branches: 1.8.16; 1.8.22; 1.8.28;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.7.26;
- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

branches: 1.6.4; 1.6.10;
nuke trailing whitespace

branches: 1.5.4; 1.5.10; 1.5.12;
Pull in FreeBSD sys/opencrypto/cryptodev.h, revision 1.2.2.5 -> 1.2.2.6.
Sam Leffler's FreeBSD commit message was
``to eliminate context switch when returning results from the
software crypto driver''
but the patch also contains the CRYPTO_SESID*() macros used in newer
ubsec and hifn drivers.

Pull up `done' flag for crypto operations from FreeBSD. FreeBSD deltas:
cryptodev.c: 1.4.2.3 -> 1.4.2.4
cryptodev.h: 1.4.2.4 -> 1.4.2.5

Move the initialization of the crypto framework from the userland
pseudo-device to init_main(), so the framework is ready for
registration requests at autoconfiguration time.

Thanks to Quentin Garnier for confirming the change was required, and
for testing a similar fix.

Cleanup traces of previous standalone m_apply()/m_getptr().

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

update locking notes of opencrypto(9)

make cryptoret() context softint to balance dequeuing crypto_ret_q with enqueuing it.

Divide crp_devflags from crp_flags to write exclusively.

CRYPTO_F_DQRETQ(new name is CRYPTODEV_F_RET) is used by cryptodev.c only.
It should be divided to other member.

add locking notes.

restructure locks(1/2): make relation between lock and data explicit.

+ crypto_drv_mtx protects
- whole crypto_drivers
+ crypto_drivers[i].cc_lock (new) protects
- crypto_drivers[i] itself
- member of crypto_drivers[i]
+ crypto_q_mtx protects
- crp_q
- crp_kq
+ crypto_ret_q_mtx protects
- crp_ret_q
- crp_ret_kq
- crypto_exit_flag

I will add locking note later.

branches: 1.34.2;
add cryptkop alloc/free KPI instead of manipulating cryptkop_pool directly.

remove obsoleted declarations.

opencrypto: cleanup debug messages.

branches: 1.31.2;
separate crypto_drv_mtx from crypto_mtx.

crypto_mtx is used only for cryptodev.c and ocryptodev.c now.

fix race among crypto_done(), cryptoret(), and {cryptodev_op(), cryptodev_key()}.

crypto_op() waited to be set CRYPTO_F_DONE with crp->crp_cv.
However, there is context switch chances between being set CRYPTO_F_DONE in
crypto_done() and done cv_signal(crp->crp_cv) in cryptodev_cb(), that is,
cryptodev_op() thread can run to cv_destroy(crp->crp_cv) before cryptoret()
thread is waken up. As a result, cryptodev_cb() can call invalid(destroyed)
cv_signal(crp->crp_cv).

Furthermore, below two implementations cause other races.
- waiting CRYPTO_F_DONE with crp->crp_cv
- context witch chances between set CRYPTO_F_DONE and cv_signal(crp->crp_cv)

So, use other flag(CRYPTO_F_DQRETQ) for cryptodev_op() and cryptodev_key(),
and then call cv_signal(crp->crp_cv) immediately after set CRYPTO_F_DQRETQ.

Tested concurrent over 20 processes with software and hardware drivers.

add sysctl to select software/hardware encryption driver. can enable CRYPTO_DEBUG.

branches: 1.28.2; 1.28.4;
KNF. Remove extra spaces. No functional change.

Needs sys/time.h for struct timespec.

Re-work the module init and destroy code to allow it to be unloaded and
then reloaded.

Should fix PR kern/49842

branches: 1.25.30;
-if an opencrypto(9) session is allocated, the driver is refcounted
and can not disappear -- no need to hold crypto_mtx to check the
driver list
(the whole check is questionable)
-crp->crp_cv (the condition variable) is used by userland cryptodev
exclusively -- move its initialization there, no need to waste
cycles of in-kernel callers
-add a comment which members of "struct cryptop" are used
by opencrypto(9) and which by crypto(4)
(this should be split, no need to waste memory for in-kernel callers)

branches: 1.24.2;
pull in AES-GCM/GMAC support from OpenBSD
This is still somewhat experimental. Tested between 2 similar boxes
so far. There is much potential for performance improvement. For now,
I've changed the gmac code to accept any data alignment, as the "char *"
pointer suggests. As the code is practically used, 32-bit alignment
can be assumed, at the cost of data copies. I don't know whether
bytewise access or copies are worse performance-wise. For efficient
implementations using SSE2 instructions on x86, even stricter
alignment requirements might arise.

copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

add an AES-CTR xform, from OpenBSD

remove redundant declarations

make camellia-cbc known to the opencrypto framework

make the use of SHA2-HMAC by FAST_IPSEC compliant to current standards:
-RFC2104 says that the block size of the hash algorithm must be used
for key/ipad/opad calculations. While formerly all ciphers used a block
length of 64, SHA384 and SHA512 use 128 bytes. So we can't use the
HMAC_BLOCK_LEN constant anymore. Add a new field to "struct auth_hash"
for the per-cipher blocksize.
-Due to this, there can't be a single "CRYPTO_SHA2_HMAC" external name
anymore. Replace this by 3 for the 3 different keysizes.
This was done by Open/FreeBSD before.
-Also fix the number of authenticator bits used tor ESP and AH to
conform to RFC4868, and remove uses of AH_HMAC_HASHLEN which did
assume a fixed authenticator size of 12 bytes.

FAST_IPSEC will not interoperate with KAME IPSEC anymore if sha2 is used,
because the latter doesn't implement these standards. It should
interoperate with at least modern Free/OpenBSD now.
(I've only tested with NetBSD-current/FAST_IPSEC on both ends.)

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

branches: 1.17.4; 1.17.6; 1.17.8;
It is best to explicitly include <sys/condvar.h> if we're going to
use kcondvar_t.

Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

branches: 1.15.4;
Fix a race condition in opencrypto where the crypto request could be
completed by the crypto device, queued on the retq, but freed by the
ioctl lwp. The problem manifests as various panics relating to the
condvar inside the request. The problem can occur whenever the crypto
device completes the request immediately and the ioctl skips the cv_wait().

The problem can be reproduced by enabling cryptosoft and running an openssl
speed test. E.g.
sysctl -w kern.cryptodevallowsoft=-1
openssl speed -engine cryptodev -evp des-ede3-cbc -multi 64

Add a macro for TAILQ_FOREACH_REVERSE_SAFE() to queue.h, since this
was missing and the opencrypto code removes requests from a list while
iterating with TAILQ_FOREACH_REVERSE().

Add missing cv_destroy() calls for the key request cleanup.

Reviewed by Thor Lancelot Simon.

branches: 1.14.6; 1.14.8; 1.14.10;
Remove clause 3 and 4 from TNF licenses

branches: 1.13.2; 1.13.4;
Extend crypto.4 interface:

* Asynchronous operation with result retrieval via select/poll
* Mutliple-request submit/retrieve ioctls
* Mutliple-session create-destroy ioctls

Revise/rewrite crypto.4 manual page. It should now be much easier to write
new applications to this API.

Measured performance for trivial requests: 84,000 very short modular math
operations/sec, 120,000 very short md5 hashes per sec (with a hardware
accellerator of moderate performance but very low latency, whose driver
will be contributed at a later date).

Contributed to TNF by Coyote Point Systems, Inc.

branches: 1.12.6;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

From Darran Hunt at Coyote Point: don't truncate HMAC to 96 bits unless
actually asked to.

Fixed in FreeBSD a while ago, discussed on tech-kern and tech-crypto.

This code never worked on a released version of FreeBSD in the form it's
been in in our tree, and certainly does not work on any version of FreeBSD
now. Run through unifdef -D__NetBSD__ -U__FreeBSD__ yielding a small
reduction of size and a dramatic improvement in readability.

No, this does not yield any meaningful decrease in patchability (unlike
mechanical changes that touch live source lines) -- try it and see.

Add constants for modular arithmetic operations other than exponentiation -- there's hardware out there which can do them.

branches: 1.8.16; 1.8.22; 1.8.28;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.7.26;
- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

branches: 1.6.4; 1.6.10;
nuke trailing whitespace

branches: 1.5.4; 1.5.10; 1.5.12;
Pull in FreeBSD sys/opencrypto/cryptodev.h, revision 1.2.2.5 -> 1.2.2.6.
Sam Leffler's FreeBSD commit message was
``to eliminate context switch when returning results from the
software crypto driver''
but the patch also contains the CRYPTO_SESID*() macros used in newer
ubsec and hifn drivers.

Pull up `done' flag for crypto operations from FreeBSD. FreeBSD deltas:
cryptodev.c: 1.4.2.3 -> 1.4.2.4
cryptodev.h: 1.4.2.4 -> 1.4.2.5

Move the initialization of the crypto framework from the userland
pseudo-device to init_main(), so the framework is ready for
registration requests at autoconfiguration time.

Thanks to Quentin Garnier for confirming the change was required, and
for testing a similar fix.

Cleanup traces of previous standalone m_apply()/m_getptr().

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

make cryptoret() context softint to balance dequeuing crypto_ret_q with enqueuing it.

Divide crp_devflags from crp_flags to write exclusively.

CRYPTO_F_DQRETQ(new name is CRYPTODEV_F_RET) is used by cryptodev.c only.
It should be divided to other member.

add locking notes.

restructure locks(1/2): make relation between lock and data explicit.

+ crypto_drv_mtx protects
- whole crypto_drivers
+ crypto_drivers[i].cc_lock (new) protects
- crypto_drivers[i] itself
- member of crypto_drivers[i]
+ crypto_q_mtx protects
- crp_q
- crp_kq
+ crypto_ret_q_mtx protects
- crp_ret_q
- crp_ret_kq
- crypto_exit_flag

I will add locking note later.

branches: 1.34.2;
add cryptkop alloc/free KPI instead of manipulating cryptkop_pool directly.

remove obsoleted declarations.

opencrypto: cleanup debug messages.

branches: 1.31.2;
separate crypto_drv_mtx from crypto_mtx.

crypto_mtx is used only for cryptodev.c and ocryptodev.c now.

fix race among crypto_done(), cryptoret(), and {cryptodev_op(), cryptodev_key()}.

crypto_op() waited to be set CRYPTO_F_DONE with crp->crp_cv.
However, there is context switch chances between being set CRYPTO_F_DONE in
crypto_done() and done cv_signal(crp->crp_cv) in cryptodev_cb(), that is,
cryptodev_op() thread can run to cv_destroy(crp->crp_cv) before cryptoret()
thread is waken up. As a result, cryptodev_cb() can call invalid(destroyed)
cv_signal(crp->crp_cv).

Furthermore, below two implementations cause other races.
- waiting CRYPTO_F_DONE with crp->crp_cv
- context witch chances between set CRYPTO_F_DONE and cv_signal(crp->crp_cv)

So, use other flag(CRYPTO_F_DQRETQ) for cryptodev_op() and cryptodev_key(),
and then call cv_signal(crp->crp_cv) immediately after set CRYPTO_F_DQRETQ.

Tested concurrent over 20 processes with software and hardware drivers.

add sysctl to select software/hardware encryption driver. can enable CRYPTO_DEBUG.

branches: 1.28.2; 1.28.4;
KNF. Remove extra spaces. No functional change.

Needs sys/time.h for struct timespec.

Re-work the module init and destroy code to allow it to be unloaded and
then reloaded.

Should fix PR kern/49842

branches: 1.25.30;
-if an opencrypto(9) session is allocated, the driver is refcounted
and can not disappear -- no need to hold crypto_mtx to check the
driver list
(the whole check is questionable)
-crp->crp_cv (the condition variable) is used by userland cryptodev
exclusively -- move its initialization there, no need to waste
cycles of in-kernel callers
-add a comment which members of "struct cryptop" are used
by opencrypto(9) and which by crypto(4)
(this should be split, no need to waste memory for in-kernel callers)

branches: 1.24.2;
pull in AES-GCM/GMAC support from OpenBSD
This is still somewhat experimental. Tested between 2 similar boxes
so far. There is much potential for performance improvement. For now,
I've changed the gmac code to accept any data alignment, as the "char *"
pointer suggests. As the code is practically used, 32-bit alignment
can be assumed, at the cost of data copies. I don't know whether
bytewise access or copies are worse performance-wise. For efficient
implementations using SSE2 instructions on x86, even stricter
alignment requirements might arise.

copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

add an AES-CTR xform, from OpenBSD

remove redundant declarations

make camellia-cbc known to the opencrypto framework

make the use of SHA2-HMAC by FAST_IPSEC compliant to current standards:
-RFC2104 says that the block size of the hash algorithm must be used
for key/ipad/opad calculations. While formerly all ciphers used a block
length of 64, SHA384 and SHA512 use 128 bytes. So we can't use the
HMAC_BLOCK_LEN constant anymore. Add a new field to "struct auth_hash"
for the per-cipher blocksize.
-Due to this, there can't be a single "CRYPTO_SHA2_HMAC" external name
anymore. Replace this by 3 for the 3 different keysizes.
This was done by Open/FreeBSD before.
-Also fix the number of authenticator bits used tor ESP and AH to
conform to RFC4868, and remove uses of AH_HMAC_HASHLEN which did
assume a fixed authenticator size of 12 bytes.

FAST_IPSEC will not interoperate with KAME IPSEC anymore if sha2 is used,
because the latter doesn't implement these standards. It should
interoperate with at least modern Free/OpenBSD now.
(I've only tested with NetBSD-current/FAST_IPSEC on both ends.)

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

branches: 1.17.4; 1.17.6; 1.17.8;
It is best to explicitly include <sys/condvar.h> if we're going to
use kcondvar_t.

Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

branches: 1.15.4;
Fix a race condition in opencrypto where the crypto request could be
completed by the crypto device, queued on the retq, but freed by the
ioctl lwp. The problem manifests as various panics relating to the
condvar inside the request. The problem can occur whenever the crypto
device completes the request immediately and the ioctl skips the cv_wait().

The problem can be reproduced by enabling cryptosoft and running an openssl
speed test. E.g.
sysctl -w kern.cryptodevallowsoft=-1
openssl speed -engine cryptodev -evp des-ede3-cbc -multi 64

Add a macro for TAILQ_FOREACH_REVERSE_SAFE() to queue.h, since this
was missing and the opencrypto code removes requests from a list while
iterating with TAILQ_FOREACH_REVERSE().

Add missing cv_destroy() calls for the key request cleanup.

Reviewed by Thor Lancelot Simon.

branches: 1.14.6; 1.14.8; 1.14.10;
Remove clause 3 and 4 from TNF licenses

branches: 1.13.2; 1.13.4;
Extend crypto.4 interface:

* Asynchronous operation with result retrieval via select/poll
* Mutliple-request submit/retrieve ioctls
* Mutliple-session create-destroy ioctls

Revise/rewrite crypto.4 manual page. It should now be much easier to write
new applications to this API.

Measured performance for trivial requests: 84,000 very short modular math
operations/sec, 120,000 very short md5 hashes per sec (with a hardware
accellerator of moderate performance but very low latency, whose driver
will be contributed at a later date).

Contributed to TNF by Coyote Point Systems, Inc.

branches: 1.12.6;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

From Darran Hunt at Coyote Point: don't truncate HMAC to 96 bits unless
actually asked to.

Fixed in FreeBSD a while ago, discussed on tech-kern and tech-crypto.

This code never worked on a released version of FreeBSD in the form it's
been in in our tree, and certainly does not work on any version of FreeBSD
now. Run through unifdef -D__NetBSD__ -U__FreeBSD__ yielding a small
reduction of size and a dramatic improvement in readability.

No, this does not yield any meaningful decrease in patchability (unlike
mechanical changes that touch live source lines) -- try it and see.

Add constants for modular arithmetic operations other than exponentiation -- there's hardware out there which can do them.

branches: 1.8.16; 1.8.22; 1.8.28;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.7.26;
- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

branches: 1.6.4; 1.6.10;
nuke trailing whitespace

branches: 1.5.4; 1.5.10; 1.5.12;
Pull in FreeBSD sys/opencrypto/cryptodev.h, revision 1.2.2.5 -> 1.2.2.6.
Sam Leffler's FreeBSD commit message was
``to eliminate context switch when returning results from the
software crypto driver''
but the patch also contains the CRYPTO_SESID*() macros used in newer
ubsec and hifn drivers.

Pull up `done' flag for crypto operations from FreeBSD. FreeBSD deltas:
cryptodev.c: 1.4.2.3 -> 1.4.2.4
cryptodev.h: 1.4.2.4 -> 1.4.2.5

Move the initialization of the crypto framework from the userland
pseudo-device to init_main(), so the framework is ready for
registration requests at autoconfiguration time.

Thanks to Quentin Garnier for confirming the change was required, and
for testing a similar fix.

Cleanup traces of previous standalone m_apply()/m_getptr().

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

Divide crp_devflags from crp_flags to write exclusively.

CRYPTO_F_DQRETQ(new name is CRYPTODEV_F_RET) is used by cryptodev.c only.
It should be divided to other member.

add locking notes.

restructure locks(1/2): make relation between lock and data explicit.

+ crypto_drv_mtx protects
- whole crypto_drivers
+ crypto_drivers[i].cc_lock (new) protects
- crypto_drivers[i] itself
- member of crypto_drivers[i]
+ crypto_q_mtx protects
- crp_q
- crp_kq
+ crypto_ret_q_mtx protects
- crp_ret_q
- crp_ret_kq
- crypto_exit_flag

I will add locking note later.

add cryptkop alloc/free KPI instead of manipulating cryptkop_pool directly.

remove obsoleted declarations.

opencrypto: cleanup debug messages.

branches: 1.31.2;
separate crypto_drv_mtx from crypto_mtx.

crypto_mtx is used only for cryptodev.c and ocryptodev.c now.

fix race among crypto_done(), cryptoret(), and {cryptodev_op(), cryptodev_key()}.

crypto_op() waited to be set CRYPTO_F_DONE with crp->crp_cv.
However, there is context switch chances between being set CRYPTO_F_DONE in
crypto_done() and done cv_signal(crp->crp_cv) in cryptodev_cb(), that is,
cryptodev_op() thread can run to cv_destroy(crp->crp_cv) before cryptoret()
thread is waken up. As a result, cryptodev_cb() can call invalid(destroyed)
cv_signal(crp->crp_cv).

Furthermore, below two implementations cause other races.
- waiting CRYPTO_F_DONE with crp->crp_cv
- context witch chances between set CRYPTO_F_DONE and cv_signal(crp->crp_cv)

So, use other flag(CRYPTO_F_DQRETQ) for cryptodev_op() and cryptodev_key(),
and then call cv_signal(crp->crp_cv) immediately after set CRYPTO_F_DQRETQ.

Tested concurrent over 20 processes with software and hardware drivers.

add sysctl to select software/hardware encryption driver. can enable CRYPTO_DEBUG.

branches: 1.28.2; 1.28.4;
KNF. Remove extra spaces. No functional change.

Needs sys/time.h for struct timespec.

Re-work the module init and destroy code to allow it to be unloaded and
then reloaded.

Should fix PR kern/49842

branches: 1.25.30;
-if an opencrypto(9) session is allocated, the driver is refcounted
and can not disappear -- no need to hold crypto_mtx to check the
driver list
(the whole check is questionable)
-crp->crp_cv (the condition variable) is used by userland cryptodev
exclusively -- move its initialization there, no need to waste
cycles of in-kernel callers
-add a comment which members of "struct cryptop" are used
by opencrypto(9) and which by crypto(4)
(this should be split, no need to waste memory for in-kernel callers)

branches: 1.24.2;
pull in AES-GCM/GMAC support from OpenBSD
This is still somewhat experimental. Tested between 2 similar boxes
so far. There is much potential for performance improvement. For now,
I've changed the gmac code to accept any data alignment, as the "char *"
pointer suggests. As the code is practically used, 32-bit alignment
can be assumed, at the cost of data copies. I don't know whether
bytewise access or copies are worse performance-wise. For efficient
implementations using SSE2 instructions on x86, even stricter
alignment requirements might arise.

copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

add an AES-CTR xform, from OpenBSD

remove redundant declarations

make camellia-cbc known to the opencrypto framework

make the use of SHA2-HMAC by FAST_IPSEC compliant to current standards:
-RFC2104 says that the block size of the hash algorithm must be used
for key/ipad/opad calculations. While formerly all ciphers used a block
length of 64, SHA384 and SHA512 use 128 bytes. So we can't use the
HMAC_BLOCK_LEN constant anymore. Add a new field to "struct auth_hash"
for the per-cipher blocksize.
-Due to this, there can't be a single "CRYPTO_SHA2_HMAC" external name
anymore. Replace this by 3 for the 3 different keysizes.
This was done by Open/FreeBSD before.
-Also fix the number of authenticator bits used tor ESP and AH to
conform to RFC4868, and remove uses of AH_HMAC_HASHLEN which did
assume a fixed authenticator size of 12 bytes.

FAST_IPSEC will not interoperate with KAME IPSEC anymore if sha2 is used,
because the latter doesn't implement these standards. It should
interoperate with at least modern Free/OpenBSD now.
(I've only tested with NetBSD-current/FAST_IPSEC on both ends.)

small modifications in dealing with the unknown result size of compression/
decompression:
-seperate the IPCOMP specific rule that compression must not grow the
data from general compression semantics: Introduce a special name
CRYPTO_DEFLATE_COMP_NOGROW/comp_algo_deflate_nogrow to describe
the IPCOMP semantics and use it there. (being here, fix the check
so that equal size is considered failure as well as required by
RFC2393)
Customers of CRYPTO_DEFLATE_COMP/comp_algo_deflate now always get
deflated data back, even if they are not smaller than the original.
-allow to pass a "size hint" to the DEFLATE decompression function
which is used for the initial buffer allocation. Due to the changes
done there, additional allocations and extra copies are avoided if the
initial allocation is sufficient. Set the size hint to MCLBYTES (=2k)
in IPCOMP which should be good for many use cases.

branches: 1.17.4; 1.17.6; 1.17.8;
It is best to explicitly include <sys/condvar.h> if we're going to
use kcondvar_t.

Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

branches: 1.15.4;
Fix a race condition in opencrypto where the crypto request could be
completed by the crypto device, queued on the retq, but freed by the
ioctl lwp. The problem manifests as various panics relating to the
condvar inside the request. The problem can occur whenever the crypto
device completes the request immediately and the ioctl skips the cv_wait().

The problem can be reproduced by enabling cryptosoft and running an openssl
speed test. E.g.
sysctl -w kern.cryptodevallowsoft=-1
openssl speed -engine cryptodev -evp des-ede3-cbc -multi 64

Add a macro for TAILQ_FOREACH_REVERSE_SAFE() to queue.h, since this
was missing and the opencrypto code removes requests from a list while
iterating with TAILQ_FOREACH_REVERSE().

Add missing cv_destroy() calls for the key request cleanup.

Reviewed by Thor Lancelot Simon.

branches: 1.14.6; 1.14.8; 1.14.10;
Remove clause 3 and 4 from TNF licenses

branches: 1.13.2; 1.13.4;
Extend crypto.4 interface:

* Asynchronous operation with result retrieval via select/poll
* Mutliple-request submit/retrieve ioctls
* Mutliple-session create-destroy ioctls

Revise/rewrite crypto.4 manual page. It should now be much easier to write
new applications to this API.

Measured performance for trivial requests: 84,000 very short modular math
operations/sec, 120,000 very short md5 hashes per sec (with a hardware
accellerator of moderate performance but very low latency, whose driver
will be contributed at a later date).

Contributed to TNF by Coyote Point Systems, Inc.

branches: 1.12.6;
Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

From Darran Hunt at Coyote Point: don't truncate HMAC to 96 bits unless
actually asked to.

Fixed in FreeBSD a while ago, discussed on tech-kern and tech-crypto.

This code never worked on a released version of FreeBSD in the form it's
been in in our tree, and certainly does not work on any version of FreeBSD
now. Run through unifdef -D__NetBSD__ -U__FreeBSD__ yielding a small
reduction of size and a dramatic improvement in readability.

No, this does not yield any meaningful decrease in patchability (unlike
mechanical changes that touch live source lines) -- try it and see.

Add constants for modular arithmetic operations other than exponentiation -- there's hardware out there which can do them.

branches: 1.8.16; 1.8.22; 1.8.28;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.7.26;
- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

branches: 1.6.4; 1.6.10;
nuke trailing whitespace

branches: 1.5.4; 1.5.10; 1.5.12;
Pull in FreeBSD sys/opencrypto/cryptodev.h, revision 1.2.2.5 -> 1.2.2.6.
Sam Leffler's FreeBSD commit message was
``to eliminate context switch when returning results from the
software crypto driver''
but the patch also contains the CRYPTO_SESID*() macros used in newer
ubsec and hifn drivers.

Pull up `done' flag for crypto operations from FreeBSD. FreeBSD deltas:
cryptodev.c: 1.4.2.3 -> 1.4.2.4
cryptodev.h: 1.4.2.4 -> 1.4.2.5

Move the initialization of the crypto framework from the userland
pseudo-device to init_main(), so the framework is ready for
registration requests at autoconfiguration time.

Thanks to Quentin Garnier for confirming the change was required, and
for testing a similar fix.

Cleanup traces of previous standalone m_apply()/m_getptr().

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

add locking notes.

restructure locks(1/2): make relation between lock and data explicit.

+ crypto_drv_mtx protects
- whole crypto_drivers
+ crypto_drivers[i].cc_lock (new) protects
- crypto_drivers[i] itself
- member of crypto_drivers[i]
+ crypto_q_mtx protects
- crp_q
- crp_kq
+ crypto_ret_q_mtx protects
- crp_ret_q
- crp_ret_kq
- crypto_exit_flag

I will add locking note later.

add cryptkop alloc/free KPI instead of manipulating cryptkop_pool directly.

remove obsoleted declarations.

opencrypto: cleanup debug messages.

branches: 1.31.2;
separate crypto_drv_mtx from crypto_mtx.

crypto_mtx is used only for cryptodev.c and ocryptodev.c now.

fix race among crypto_done(), cryptoret(), and {cryptodev_op(), cryptodev_key()}.

crypto_op() waited to be set CRYPTO_F_DONE with crp->crp_cv.
However, there is context switch chances between being set CRYPTO_F_DONE in
crypto_done() and done cv_signal(crp->crp_cv) in cryptodev_cb(), that is,
cryptodev_op() thread can run to cv_destroy(crp->crp_cv) before cryptoret()
thread is waken up. As a result, cryptodev_cb() can call invalid(destroyed)
cv_signal(crp->crp_cv).

Furthermore, below two implementations cause other races.
- waiting CRYPTO_F_DONE with crp->crp_cv
- context witch chances between set CRYPTO_F_DONE and cv_signal(crp->crp_cv)

So, use other flag(CRYPTO_F_DQRETQ) for cryptodev_op() and cryptodev_key(),
and then call cv_signal(crp->crp_cv) immediately after set CRYPTO_F_DQRETQ.

Tested concurrent over 20 processes with software and hardware drivers.

add sysctl to select software/hardware encryption driver. can enable CRYPTO_DEBUG.

branches: 1.28.2; 1.28.4;
KNF. Remove extra spaces. No functional change.

Needs sys/time.h for struct timespec.

Re-work the module init and destroy code to allow it to be unloaded and
then reloaded.

Should fix PR kern/49842

branches: 1.25.30;
-if an opencrypto(9) session is allocated, the driver is refcounted
and can not disappear -- no need to hold crypto_mtx to check the
driver list
(the whole check is questionable)
-crp->crp_cv (the condition variable) is used by userland cryptodev
exclusively -- move its initialization there, no need to waste
cycles of in-kernel callers
-add a comment which members of "struct cryptop" are used
by opencrypto(9) and which by crypto(4)
(this should be split, no need to waste memory for in-kernel callers)