fix misspellings of 'available' and nearby typos

opencrypto: Prune dead code now that crypto_dispatch never fails.

opencrypto: crypto_dispatch never fails now. Make it return void.

Same with crypto_kdispatch.

opencrypto: Rip out EAGAIN logic when unregistering crypto drivers.

I'm pretty sure this never worked reliably based on code inspection,
and it's unlikely to have ever been tested because it only applies
when unregistering a driver -- but we have no crypto drivers for
removable devices, so it would only apply if we went out of our way
to trigger detach with drvctl.

Instead, just make the operation fail with ENODEV, and remove all the
callback logic to resubmit the request on EAGAIN. (Maybe this should
be ENXIO, but crypto_kdispatch already does ENODEV.)

crypto(4): Nix dead code now that crypto_freesession never fails.

opencrypto: Make crypto_freesession return void.

No callers use the return value. It is not sensible to allow this to
fail.

crypto(4): crypto_freesession should never fail here.

It can only fail if we pass it an invalid sid, which the logic to
maintain the user sessions should not do. So kassert error=0 here.

crypto(4): Refuse crypto operations with nothing in them earlier.

This way we avoid passing 0 to crypto_getreq -- makes it easier to
reason about everything downstream.

opencrypto: Make crp_callback, krp_callback return void.

Nothing uses the return values inside opencrypto, so let's stop
making users return them.

crypto(4): Fix possible use-after-free in race around detach.

This is extremely unlikely because I don't think we have any drivers
for removable crypto decelerators^Waccelerators...but if we were to
sprout one, and someone ran crypto_dispatch concurrently with
crypto_unregister, cryptodev_cb/mcb would enter with crp->crp_etype =
EAGAIN and with CRYPTO_F_DONE set in crp->crp_flags. In this case,
cryptodev_cb/mcb would issue crypto_dispatch but -- since nothing
clears CRYPTO_F_DONE -- it would _also_ consider the request done and
notify the ioctl thread of that.

With this change, we return early if crypto_dispatch succeeds. No
need to consult CRYPTO_F_DONE: if the callback is invoked it's done,
and if we try to redispatch it on EAGAIN but crypto_dispatch fails,
it's done. (Soon we'll get rid of the possibility of crypto_dispatch
failing synchronously, but not just yet.)

XXX This path could really use some testing!

crypto(4): Fix set-but-unused variable warning.

This deliberately ignores the error code returned by crypto_dispatch,
but that error code is fundamentally incoherent and the issue will be
mooted by subsequent changes to make it return void and always pass
the error through the callback, as well as subsequent changes to rip
out the EAGAIN logic anyway.

crypto(4): Don't signal the condvar for multi-operation completion.

The condvar may be destroyed by the time we got here, and nothing
waits on it anyway -- instead the caller is expected to select/poll
for completion in userland.

The bug was already here, but the recent change to eliminate
CRYPTO_F_CBIMM made it happen more often by causing the callback to
_always_ be run asynchronously instead of sometimes being run
synchronously.

opencrypto: Nix CRYPTO_F_USER, CRYPTO_F_CBIMM, CRYPTO_F_CBIFSYNC.

CRYPTO_F_USER is no longer needed. It was introduced in 2008 by
darran@ in crypto.c 1.30, cryptodev.c 1.45 in an attempt to avoid
double-free between the issuing thread and asynchronous callback.
But the `fix' didn't work. In 2017, knakahara@ fixed it properly in
cryptodev.c 1.87 by distinguishing `the crypto operation has
completed' (CRYPTO_F_DONE) from `the callback is done touching the
crp object' (CRYPTO_F_DQRETQ, now renamed to CRYPTODEV_F_RET).

CRYPTO_F_CBIMM formerly served to invoke the callback synchronously
from the driver's interrupt completion routine, to reduce contention
on what was once a single cryptoret thread. Now, there is a per-CPU
queue and softint for much cheaper processing, so there is less
motivation for this in the first place. So let's remove the
complicated logic. This means the callbacks never run in hard
interrupt context, which means we don't need to worry about recursion
into crypto_dispatch in hard interrupt context.

crypto(4): Simplify error test in cryptodev_op.

No functional change intended.

crypto(4): Narrow scope of cryptodev_mtx to cover wait.

No functional change intended -- this only removes an unnecessary
lock/unlock cycle in the error case.

crypto(4): Nix long-dead code and comments.

crypto(4): Use IPL_NONE, not IPL_NET, for /dev/crypto pools.

These are used (pool_get/put) only from thread context, never from
interrupt or even soft interrupt context.

opencrypto(9): Omit needless casts around callbacks.

Just declare the right types to begin with. No functional change
intended.

For device modules that provide both auto-config and /dev/xxx
interfaces, make sure that initialization and destruction
follow the proper sequence. This is triggered by the recent
changes to the devsw stuff; per riastradh@ the required call
sequence is:

devsw_attach()
config_init_component() or config_cf*_attach()
...
config_fini_component() or config_cf*_detach()
devsw_detach()

While here, add a few missing calls to some of the detach
routines.

Testing of these changes has been limited to:
1. compile without build break
2. no related test failures from atf
3. modload/modunload work as well as
before.

No functional device testing done, since I don't have any
of these devices. Let me know of any damage I might cause
here!

XXX Some of the modules affected by this commit are already
XXX broken; see kern/56772. This commit does not break
any additional modules (as far as I know).

Rename enc_xform_rijndael128 -> enc_xform_aes.

Update netipsec dependency.

slightly change and fix the semantics of pool_set*wat(), pool_sethardlimit()
and pool_prime() (and their pool_cache_* counterparts):

- the pool_set*wat() APIs are supposed to specify thresholds for the count of
free items in the pool before pool pages are automatically allocated or freed
during pool_get() / pool_put(), whereas pool_sethardlimit() and pool_prime()
are supposed to specify minimum and maximum numbers of total items
in the pool (both free and allocated). these were somewhat conflated
in the existing code, so separate them as they were intended.

- change pool_prime() to take an absolute number of items to preallocate
rather than an increment over whatever was done before, and wait for
any memory allocations to succeed. since pool_prime() can no longer fail
after this, change its return value to void and adjust all callers.

- pool_setlowat() is documented as not immediately attempting to allocate
any memory, but it was changed some time ago to immediately try to allocate
up to the lowat level, so just fix the manpage to describe the current
behaviour.

- add a pool_cache_prime() to complete the API set.

branches: 1.104.4;
If we get an error from devsw_attach(), don't destroy the error value,
since we need to return it to our caller. While we're here, improve
the value of the debug message by actually printing the error value.

Initialize the session variable to an impossible session to prevent compiler
warnings.

branches: 1.102.2;
crypto(4): accept CRYPTO_SHA2_384_HMAC and CRYPTO_SHA2_512_HMAC.

don't always panic when modunload crypto (int the pool destroy code, because
the pools are busy). XXX: this is still racy; we need to prevent creating
more sessions while destroying.

Rename the MODULE_*_HOOK() macros to MODULE_HOOK_*() as briefly
discussed on irc.

NFCI intended.

Ride the earlier kernel bump - it;s getting crowded.

Merge the [pgoyette-compat] branch

branches: 1.98.2; 1.98.4;
Typos.

add fo_name so we can identify the fileops in a simple way.

check results of pool_prime.

Divide crp_devflags from crp_flags to write exclusively.

CRYPTO_F_DQRETQ(new name is CRYPTODEV_F_RET) is used by cryptodev.c only.
It should be divided to other member.

sanitize in CIOCNCRYPTM and initialize comp_alg in CIOCNGSESSION

sanitize count used for kmem_alloc size.

Hmm, who uses CIOCNGSESSION, CIOCNFSESSION, CIOCNCRYPTM or CIOCNFKEYM?

branches: 1.92.2;
rename crypto_mtx to cryptodev_mtx

It is used by cryptodev.c and ocryptodev.c only.

add cryptkop alloc/free KPI instead of manipulating cryptkop_pool directly.

opencrypto: cleanup debug messages.

branches: 1.89.2;
separate crypto_drv_mtx from crypto_mtx.

crypto_mtx is used only for cryptodev.c and ocryptodev.c now.

the processing said "ghastly hacks" is unnecessary now.

fix race among crypto_done(), cryptoret(), and {cryptodev_op(), cryptodev_key()}.

crypto_op() waited to be set CRYPTO_F_DONE with crp->crp_cv.
However, there is context switch chances between being set CRYPTO_F_DONE in
crypto_done() and done cv_signal(crp->crp_cv) in cryptodev_cb(), that is,
cryptodev_op() thread can run to cv_destroy(crp->crp_cv) before cryptoret()
thread is waken up. As a result, cryptodev_cb() can call invalid(destroyed)
cv_signal(crp->crp_cv).

Furthermore, below two implementations cause other races.
- waiting CRYPTO_F_DONE with crp->crp_cv
- context witch chances between set CRYPTO_F_DONE and cv_signal(crp->crp_cv)

So, use other flag(CRYPTO_F_DQRETQ) for cryptodev_op() and cryptodev_key(),
and then call cv_signal(crp->crp_cv) immediately after set CRYPTO_F_DQRETQ.

Tested concurrent over 20 processes with software and hardware drivers.

fix processes accessing /dev/crypto stall when over three processes run with a hardware encryption driver

The process has stalled at cv_wait(&crp->crp_cv) because cryptodev_cb()
is not called as cryptoret() kthread keep waiting at cv_wait(&cryptoret_cv).
Previous opencrypto implementation assumes the thread from cryptodev.c
does all processing in the same context, so skips enqueueing and sending
cryptoret_cv. However, the context can be switched, e.g. when we use
a hardware encryption driver.

And add debug messages.

branches: 1.85.2; 1.85.4;
KNF. Remove extra spaces. No functional change.

include "ioconf.h" to get the 'void <driver>attach(int count);' prototype.

Trivial printf format changes and typo fix

branches: 1.82.2;
Return ENOSPC instead of ENOMEM when there is no room in the buffer to
store results. ENOMEM in this subsystem means we cannot allocate more
requests or internal buffers for xforms.

Try not to use f_data, use f_fcrypt to get a correctly typed pointer.

At least crypto_mtx needs initialisation here. Spotted during PR/49065
investigation.

Add d_discard to all struct cdevsw instances I could find.

All have been set to "nodiscard"; some should get a real implementation.

branches: 1.78.2;
Change (mostly mechanically) every cdevsw/bdevsw I can find to use
designated initializers.

I have not built every extant kernel so I have probably broken at
least one build; however I've also found and fixed some wrong
cdevsw/bdevsw entries so even if so I think we come out ahead.

Undo previous - it still needs a lot more work.

For now, we'll use the hand-crafted cf* structures and directly
call all the config routines.

Replace home-grown config with standardized calls to
config_{init,fini}_component()

As requested by mrg@, since there is still a small window during which
the in-module ref-counting can fail, completely disable auto-unload.

Implement in-module ref-counting, and do not allow auto-unload if there
are existing references.

Note that manual unloading is not prevented.

OK christos@

XXX Also note that there is still a small window where the ref-count can
XXX be decremented, and then the process/thread preempted. If auto-unload
XXX happens before that thread can return from the module's code, bad
XXX things (tm) could happen.

knf: Blank line even if no variable declarations.

bail out unloading for now

When crypto(4) is built-in, crypto_modcmd() doesn't need to handle all
the auto-config stuff.

While here, ensure that we depend on opencrypto.

Modularize the opencrypto components and link to the build

Fix return value of cryptodev_msessionfin.

branches: 1.68.2; 1.68.12; 1.68.16;
Fix memset usage.

-if an opencrypto(9) session is allocated, the driver is refcounted
and can not disappear -- no need to hold crypto_mtx to check the
driver list
(the whole check is questionable)
-crp->crp_cv (the condition variable) is used by userland cryptodev
exclusively -- move its initialization there, no need to waste
cycles of in-kernel callers
-add a comment which members of "struct cryptop" are used
by opencrypto(9) and which by crypto(4)
(this should be split, no need to waste memory for in-kernel callers)

branches: 1.66.2;
allow testing of GCM/GMAC code from userland

fix building of a linked list if multiple algorithms are requested
in a session -- this just didn't work

catch some corner cases of user input

copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

-remove references to crypto/arc4/arc4.* -- the code isn't used
anywhere afaics
(The confusion comes probably from use of arc4random() at various places,
but this lives in libkern and doesn't share code with the former.)
-g/c non-implementation of arc4 encryption in swcrypto(4)
-remove special casing of ARC4 in crypto(4) -- the point is that it
doesn't use an IV, and this fact is made explicit by the new "ivsize"
property of xforms

If symmetric encryption is done from userland crypto(4) and no IV
is specified, the kernel gets one from the random generator. Make sure it
is copied out to the user, otherwise the result is quite useless.

being here, export camellia-cbc through crypto(4) to allow userland tests

add an AES-CTR xform, from OpenBSD

-in the descriptor for encryption xforms, split the "blocksize" field
into "blocksize" and "IV size"
-add an "reinit" function pointer which, if set, means that the xform
does its IV handling itself and doesn't want the default CBC handling
by the framework (poor name, but left that way to avoid unecessary
differences)
This syncs with Open/FreeBSD, purpose is to allow non-CBC transforms.
Refer to ivsize instead of blocksize where appropriate.
(At this point, blocksize and ivsize are identical.)

split the "crypto_mtx" spinlock into 3: one spinlock each for
the incoming and outgoing request queues (which can be dealt with
by hardware accelerators) and an adaptive lock for "all the rest"
(mostly driver configuration, but also some unrelated stuff in
cryptodev.c which should be revisited)
The latter one seems to be uneeded at many places, but for now I've
done simple replacements only, except minor fixes (where
softint_schedule() was called without the lock held)

As a first step towards more fine-grained locking, don't require
crypto_{new.free}session() to be called with the "crypto_mtx"
spinlock held.
This doesn't change much for now because these functions acquire
the said mutex first on entry now, but at least it keeps the nasty
locks local to the opencrypto core.

make the compatibility code conditional on COMPAT_50

more "const"

branches: 1.53.2; 1.53.4;
Consistently use a single CRYPTO_SESID2HID-like macro.
Improve CRYPTO_DEBUG printing a bit:
print pointers with %p
print unsigned with %u rather than %d
use CRYPTO_SESID2LID instead of just casting to uint32_t

branches: 1.52.2; 1.52.4;
Add missing "break" for CRYPTO_CAST_CBC, and some assorted comment fixes.
openssl(1) checks for CAST (and others) on ~every startup.

If a multithreaded app closes an fd while another thread is blocked in
read/write/accept, then the expectation is that the blocked thread will
exit and the close complete.
Since only one fd is affected, but many fd can refer to the same file,
the close code can only request the fs code unblock with ERESTART.
Fixed for pipes and sockets, ERESTART will only be generated after such
a close - so there should be no change for other programs.
Also rename fo_abort() to fo_restart() (this used to be fo_drain()).
Fixes PR/26567

Rename fo_drain() to fo_abort(), 'drain' is used to mean 'wait for output
do drain' in many places, whereas fo_drain() was called in order to force
blocking read()/write() etc calls to return to userspace so that a close()
call from a different thread can complete.
In the sockets code comment out the broken code in the inner function,
it was being called from compat code.

Fix locking as Andy explained. Also fill in uid and gid like sys_pipe did.

Fix PR/37878 and PR/37550: Provide stat(2) for all devices and don't use
fbadop_stat.

Add fileops::fo_drain(), to be called from fd_close() when there is more
than one active reference to a file descriptor. It should dislodge threads
sleeping while holding a reference to the descriptor. Implemented only for
sockets but should be extended to pipes, fifos, etc.

Fixes the case of a multithreaded process doing something like the
following, which would have hung until the process got a signal.

thr0 accept(fd, ...)
thr1 close(fd)

Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

branches: 1.45.4;
Fix a race condition in opencrypto where the crypto request could be
completed by the crypto device, queued on the retq, but freed by the
ioctl lwp. The problem manifests as various panics relating to the
condvar inside the request. The problem can occur whenever the crypto
device completes the request immediately and the ioctl skips the cv_wait().

The problem can be reproduced by enabling cryptosoft and running an openssl
speed test. E.g.
sysctl -w kern.cryptodevallowsoft=-1
openssl speed -engine cryptodev -evp des-ede3-cbc -multi 64

Add a macro for TAILQ_FOREACH_REVERSE_SAFE() to queue.h, since this
was missing and the opencrypto code removes requests from a list while
iterating with TAILQ_FOREACH_REVERSE().

Add missing cv_destroy() calls for the key request cleanup.

Reviewed by Thor Lancelot Simon.

branches: 1.44.4; 1.44.6; 1.44.8;
Coverity CID 5021: Check pointers before using.

Coverity CID 5027: Remove impossible test.

KNF, whitespace, b* -> mem*. No functional change.

branches: 1.41.2;
Make various bits of debug code compile again.

Remove clause 3 and 4 from TNF licenses

branches: 1.39.2;
As suggested by rmind, do not check return status of KM_SLEEP/PR_WAITOK
allocations. A little hair-raising but it does make the code easier to
read.

branches: 1.38.2;
Protect selrecord/selnotify calls with crypto_mtx; few misc changes.

fix 64-bit b0rkenness.

Extend crypto.4 interface:

* Asynchronous operation with result retrieval via select/poll
* Mutliple-request submit/retrieve ioctls
* Mutliple-session create-destroy ioctls

Revise/rewrite crypto.4 manual page. It should now be much easier to write
new applications to this API.

Measured performance for trivial requests: 84,000 very short modular math
operations/sec, 120,000 very short md5 hashes per sec (with a hardware
accellerator of moderate performance but very low latency, whose driver
will be contributed at a later date).

Contributed to TNF by Coyote Point Systems, Inc.

Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

branches: 1.34.6;
Some locking fixes (double-release mutex in softintr wakeup case, which I
hadn't tested) and an uninitialized field in cse which Darran Hunt
found. Some more debugging printfs.

Turn on MPSAFE for the kthread. We're not sure it's safe for the softint
yet. Gives a little performance kick for swcrypto with many requests on
MP systems.

Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

From Darran Hunt at Coyote Point: don't truncate HMAC to 96 bits unless
actually asked to.

Fixed in FreeBSD a while ago, discussed on tech-kern and tech-crypto.

This code never worked on a released version of FreeBSD in the form it's
been in in our tree, and certainly does not work on any version of FreeBSD
now. Run through unifdef -D__NetBSD__ -U__FreeBSD__ yielding a small
reduction of size and a dramatic improvement in readability.

No, this does not yield any meaningful decrease in patchability (unlike
mechanical changes that touch live source lines) -- try it and see.

Fix accidental checkin inverting the sense of cryptodev_allowsoft, which
is crazy but has always documented.

Make /dev/crypto properly cloning. Leave CRIOGET in place but note that
it is deprecated, no longer required, and will be removed in a future
release of NetBSD.

Dramatically reduce the size of the session structure by removing an
IOV_MAX array of iovecs where only the first was use. Saves an 8k
bzero on each session creation.

Convert fixed-size allocations in cryptodev.c to pools.

Some minor opencrypto fixes, one with a major performance impact for
OpenSSL:

1) Fix extremely misleading text in crypto.4 manual page so it does not
appear to claim that a new cloned file descriptor is required for every
session.

2) Fix severe performance problem (and fd leak!) in openssl cryptodev
engine resulting from misunderstanding probably caused by said manual
page text.

3) Check for session-ID wraparound in kernel cryptodev provider. Also,
start allocating sessions at 1, not 0 -- this will be necessary when
we add ioctls for the creation of multiple sessions at once, so we
can tell which if any creations failed.

Add constants for modular arithmetic operations other than exponentiation -- there's hardware out there which can do them.

branches: 1.26.16; 1.26.22; 1.26.28;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.25.4;
__unused removal on arguments; approved by core.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

branches: 1.23.2; 1.23.4;
fix incomplete initializer

Use the LWP cached credentials where sane.

integrate kauth.

Coverity CID 1083: Avoid possible NULL pointer deref.

don't use MALLOC with a non-constant size; use malloc instead.

branches: 1.18.2; 1.18.4;
sprinkle DPRINTF()...

branches: 1.17.2;
merge yamt-uio_vmspace branch.

- use vmspace rather than proc or lwp where appropriate.
the latter is more natural to specify an address space.
(and less likely to be abused for random purposes.)
- fix a swdmover race.

branches: 1.16.2; 1.16.4; 1.16.6;
merge ktrace-lwp.

- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

branches: 1.14.6;
No change. Forced commit to record commit message for previous revision, viz:

Fix vulnerability to a denial-of-service attack which passes a
length-0 crypto op. Check for zero length and return EINVAL, taken from:

http://cvsweb.FreeBSD.org/src/sys/opencrypto/cryptodev.c.diff?r1=1.25&r2=1.26

Original FreeBSD log mesage:

Modified files:
sys/opencrypto cryptodev.c
Log:
Fix bogus check. It was possible to panic the kernel by giving 0 length.
This is actually a local DoS, as every user can use /dev/crypto if there
is crypto hardware in the system and cryptodev.ko is loaded (or compiled
into the kernel).

Reported by: Mike Tancsa <mike@sentex.net>


thanks to Sam Leffler for passing on a heads-up about this issue.

*** empty log message ***

branches: 1.12.12;
Cloning cleanup:
1. make fileops const
2. add 2 new negative errno's to `officially' support the cloning hack:
- EDUPFD (used to overload ENODEV)
- EMOVEFD (used to overload ENXIO)
3. Created an fdclone() function to encapsulate the operations needed for
EMOVEFD, and made all cloners use it.
4. Centralize the local noop/badop fileops functions to:
fnullop_fcntl, fnullop_poll, fnullop_kqfilter, fbadop_stat

There's no need to pass a proc value when using UIO_SYSSPACE with
vn_rdwr(9) and uiomove(9).

OK'd by Jason Thorpe

branches: 1.10.4;
Wrap noisy pointless message about denied userspace requests with
`#ifdef CRYPTO_DEBUG', per Jason Thorpe's suggestion.

Clean up userlevel access to software kernel transforms, in preparation
for using /dev/crypto for OpenSSL:

1. Add comments explaining crypto_devallowsoft, explaining the
OpenBSD-style three-way logic actully implemented in crypto_newsession().

2. Pass crypto_devallowsoft as the final argument to crypto_newsession(),
instead of a constant 0 value.

3. Set the default value of crypto_devallowsoft to 1, to allow
/dev/crypto access only for hardware-supported transforms.

Items 1-3 may be revised to match the FreeBSD two-way logic, if the
consensus is that there's no point to forcing software transforms.
But as a first step, let the description match what the code actually does.

GC unused variables usercrypto, userasmcrypto, cryptodevallowsoft from
cryptodev.c, in favour of variables crypto_usercrypto, crypto_userasmcrypto,
crypto_devallowsoft, which are used as well as defined in crypto.c.

Remove '#ifdef notdef' around userspace ioctl() requests for
pure (non-HMAC) MD5 and SHA1.

Remove a bunch of unnecessary includes.

It's bad form to use the <opencrypto/rmd160.h> header file while
using the crypto/ripemd160/rmd160.c implementation. Remove the
opencrypto-local copies of these files entirely.

on netbsd, major # for /dev/crypto depends on arch

Pull up `done' flag for crypto operations from FreeBSD. FreeBSD deltas:
cryptodev.c: 1.4.2.3 -> 1.4.2.4
cryptodev.h: 1.4.2.4 -> 1.4.2.5

Garbage-collect references to OpenBSD-only <dev/rndvar.h>.

Remove vestiges of OpenBSD <sys/md5k.h> header.

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

opencrypto: Prune dead code now that crypto_dispatch never fails.

opencrypto: crypto_dispatch never fails now. Make it return void.

Same with crypto_kdispatch.

opencrypto: Rip out EAGAIN logic when unregistering crypto drivers.

I'm pretty sure this never worked reliably based on code inspection,
and it's unlikely to have ever been tested because it only applies
when unregistering a driver -- but we have no crypto drivers for
removable devices, so it would only apply if we went out of our way
to trigger detach with drvctl.

Instead, just make the operation fail with ENODEV, and remove all the
callback logic to resubmit the request on EAGAIN. (Maybe this should
be ENXIO, but crypto_kdispatch already does ENODEV.)

crypto(4): Nix dead code now that crypto_freesession never fails.

opencrypto: Make crypto_freesession return void.

No callers use the return value. It is not sensible to allow this to
fail.

crypto(4): crypto_freesession should never fail here.

It can only fail if we pass it an invalid sid, which the logic to
maintain the user sessions should not do. So kassert error=0 here.

crypto(4): Refuse crypto operations with nothing in them earlier.

This way we avoid passing 0 to crypto_getreq -- makes it easier to
reason about everything downstream.

opencrypto: Make crp_callback, krp_callback return void.

Nothing uses the return values inside opencrypto, so let's stop
making users return them.

crypto(4): Fix possible use-after-free in race around detach.

This is extremely unlikely because I don't think we have any drivers
for removable crypto decelerators^Waccelerators...but if we were to
sprout one, and someone ran crypto_dispatch concurrently with
crypto_unregister, cryptodev_cb/mcb would enter with crp->crp_etype =
EAGAIN and with CRYPTO_F_DONE set in crp->crp_flags. In this case,
cryptodev_cb/mcb would issue crypto_dispatch but -- since nothing
clears CRYPTO_F_DONE -- it would _also_ consider the request done and
notify the ioctl thread of that.

With this change, we return early if crypto_dispatch succeeds. No
need to consult CRYPTO_F_DONE: if the callback is invoked it's done,
and if we try to redispatch it on EAGAIN but crypto_dispatch fails,
it's done. (Soon we'll get rid of the possibility of crypto_dispatch
failing synchronously, but not just yet.)

XXX This path could really use some testing!

crypto(4): Fix set-but-unused variable warning.

This deliberately ignores the error code returned by crypto_dispatch,
but that error code is fundamentally incoherent and the issue will be
mooted by subsequent changes to make it return void and always pass
the error through the callback, as well as subsequent changes to rip
out the EAGAIN logic anyway.

crypto(4): Don't signal the condvar for multi-operation completion.

The condvar may be destroyed by the time we got here, and nothing
waits on it anyway -- instead the caller is expected to select/poll
for completion in userland.

The bug was already here, but the recent change to eliminate
CRYPTO_F_CBIMM made it happen more often by causing the callback to
_always_ be run asynchronously instead of sometimes being run
synchronously.

opencrypto: Nix CRYPTO_F_USER, CRYPTO_F_CBIMM, CRYPTO_F_CBIFSYNC.

CRYPTO_F_USER is no longer needed. It was introduced in 2008 by
darran@ in crypto.c 1.30, cryptodev.c 1.45 in an attempt to avoid
double-free between the issuing thread and asynchronous callback.
But the `fix' didn't work. In 2017, knakahara@ fixed it properly in
cryptodev.c 1.87 by distinguishing `the crypto operation has
completed' (CRYPTO_F_DONE) from `the callback is done touching the
crp object' (CRYPTO_F_DQRETQ, now renamed to CRYPTODEV_F_RET).

CRYPTO_F_CBIMM formerly served to invoke the callback synchronously
from the driver's interrupt completion routine, to reduce contention
on what was once a single cryptoret thread. Now, there is a per-CPU
queue and softint for much cheaper processing, so there is less
motivation for this in the first place. So let's remove the
complicated logic. This means the callbacks never run in hard
interrupt context, which means we don't need to worry about recursion
into crypto_dispatch in hard interrupt context.

crypto(4): Simplify error test in cryptodev_op.

No functional change intended.

crypto(4): Narrow scope of cryptodev_mtx to cover wait.

No functional change intended -- this only removes an unnecessary
lock/unlock cycle in the error case.

crypto(4): Nix long-dead code and comments.

crypto(4): Use IPL_NONE, not IPL_NET, for /dev/crypto pools.

These are used (pool_get/put) only from thread context, never from
interrupt or even soft interrupt context.

opencrypto(9): Omit needless casts around callbacks.

Just declare the right types to begin with. No functional change
intended.

For device modules that provide both auto-config and /dev/xxx
interfaces, make sure that initialization and destruction
follow the proper sequence. This is triggered by the recent
changes to the devsw stuff; per riastradh@ the required call
sequence is:

devsw_attach()
config_init_component() or config_cf*_attach()
...
config_fini_component() or config_cf*_detach()
devsw_detach()

While here, add a few missing calls to some of the detach
routines.

Testing of these changes has been limited to:
1. compile without build break
2. no related test failures from atf
3. modload/modunload work as well as
before.

No functional device testing done, since I don't have any
of these devices. Let me know of any damage I might cause
here!

XXX Some of the modules affected by this commit are already
XXX broken; see kern/56772. This commit does not break
any additional modules (as far as I know).

Rename enc_xform_rijndael128 -> enc_xform_aes.

Update netipsec dependency.

slightly change and fix the semantics of pool_set*wat(), pool_sethardlimit()
and pool_prime() (and their pool_cache_* counterparts):

- the pool_set*wat() APIs are supposed to specify thresholds for the count of
free items in the pool before pool pages are automatically allocated or freed
during pool_get() / pool_put(), whereas pool_sethardlimit() and pool_prime()
are supposed to specify minimum and maximum numbers of total items
in the pool (both free and allocated). these were somewhat conflated
in the existing code, so separate them as they were intended.

- change pool_prime() to take an absolute number of items to preallocate
rather than an increment over whatever was done before, and wait for
any memory allocations to succeed. since pool_prime() can no longer fail
after this, change its return value to void and adjust all callers.

- pool_setlowat() is documented as not immediately attempting to allocate
any memory, but it was changed some time ago to immediately try to allocate
up to the lowat level, so just fix the manpage to describe the current
behaviour.

- add a pool_cache_prime() to complete the API set.

branches: 1.104.4;
If we get an error from devsw_attach(), don't destroy the error value,
since we need to return it to our caller. While we're here, improve
the value of the debug message by actually printing the error value.

Initialize the session variable to an impossible session to prevent compiler
warnings.

branches: 1.102.2;
crypto(4): accept CRYPTO_SHA2_384_HMAC and CRYPTO_SHA2_512_HMAC.

don't always panic when modunload crypto (int the pool destroy code, because
the pools are busy). XXX: this is still racy; we need to prevent creating
more sessions while destroying.

Rename the MODULE_*_HOOK() macros to MODULE_HOOK_*() as briefly
discussed on irc.

NFCI intended.

Ride the earlier kernel bump - it;s getting crowded.

Merge the [pgoyette-compat] branch

branches: 1.98.2; 1.98.4;
Typos.

add fo_name so we can identify the fileops in a simple way.

check results of pool_prime.

Divide crp_devflags from crp_flags to write exclusively.

CRYPTO_F_DQRETQ(new name is CRYPTODEV_F_RET) is used by cryptodev.c only.
It should be divided to other member.

sanitize in CIOCNCRYPTM and initialize comp_alg in CIOCNGSESSION

sanitize count used for kmem_alloc size.

Hmm, who uses CIOCNGSESSION, CIOCNFSESSION, CIOCNCRYPTM or CIOCNFKEYM?

branches: 1.92.2;
rename crypto_mtx to cryptodev_mtx

It is used by cryptodev.c and ocryptodev.c only.

add cryptkop alloc/free KPI instead of manipulating cryptkop_pool directly.

opencrypto: cleanup debug messages.

branches: 1.89.2;
separate crypto_drv_mtx from crypto_mtx.

crypto_mtx is used only for cryptodev.c and ocryptodev.c now.

the processing said "ghastly hacks" is unnecessary now.

fix race among crypto_done(), cryptoret(), and {cryptodev_op(), cryptodev_key()}.

crypto_op() waited to be set CRYPTO_F_DONE with crp->crp_cv.
However, there is context switch chances between being set CRYPTO_F_DONE in
crypto_done() and done cv_signal(crp->crp_cv) in cryptodev_cb(), that is,
cryptodev_op() thread can run to cv_destroy(crp->crp_cv) before cryptoret()
thread is waken up. As a result, cryptodev_cb() can call invalid(destroyed)
cv_signal(crp->crp_cv).

Furthermore, below two implementations cause other races.
- waiting CRYPTO_F_DONE with crp->crp_cv
- context witch chances between set CRYPTO_F_DONE and cv_signal(crp->crp_cv)

So, use other flag(CRYPTO_F_DQRETQ) for cryptodev_op() and cryptodev_key(),
and then call cv_signal(crp->crp_cv) immediately after set CRYPTO_F_DQRETQ.

Tested concurrent over 20 processes with software and hardware drivers.

fix processes accessing /dev/crypto stall when over three processes run with a hardware encryption driver

The process has stalled at cv_wait(&crp->crp_cv) because cryptodev_cb()
is not called as cryptoret() kthread keep waiting at cv_wait(&cryptoret_cv).
Previous opencrypto implementation assumes the thread from cryptodev.c
does all processing in the same context, so skips enqueueing and sending
cryptoret_cv. However, the context can be switched, e.g. when we use
a hardware encryption driver.

And add debug messages.

branches: 1.85.2; 1.85.4;
KNF. Remove extra spaces. No functional change.

include "ioconf.h" to get the 'void <driver>attach(int count);' prototype.

Trivial printf format changes and typo fix

branches: 1.82.2;
Return ENOSPC instead of ENOMEM when there is no room in the buffer to
store results. ENOMEM in this subsystem means we cannot allocate more
requests or internal buffers for xforms.

Try not to use f_data, use f_fcrypt to get a correctly typed pointer.

At least crypto_mtx needs initialisation here. Spotted during PR/49065
investigation.

Add d_discard to all struct cdevsw instances I could find.

All have been set to "nodiscard"; some should get a real implementation.

branches: 1.78.2;
Change (mostly mechanically) every cdevsw/bdevsw I can find to use
designated initializers.

I have not built every extant kernel so I have probably broken at
least one build; however I've also found and fixed some wrong
cdevsw/bdevsw entries so even if so I think we come out ahead.

Undo previous - it still needs a lot more work.

For now, we'll use the hand-crafted cf* structures and directly
call all the config routines.

Replace home-grown config with standardized calls to
config_{init,fini}_component()

As requested by mrg@, since there is still a small window during which
the in-module ref-counting can fail, completely disable auto-unload.

Implement in-module ref-counting, and do not allow auto-unload if there
are existing references.

Note that manual unloading is not prevented.

OK christos@

XXX Also note that there is still a small window where the ref-count can
XXX be decremented, and then the process/thread preempted. If auto-unload
XXX happens before that thread can return from the module's code, bad
XXX things (tm) could happen.

knf: Blank line even if no variable declarations.

bail out unloading for now

When crypto(4) is built-in, crypto_modcmd() doesn't need to handle all
the auto-config stuff.

While here, ensure that we depend on opencrypto.

Modularize the opencrypto components and link to the build

Fix return value of cryptodev_msessionfin.

branches: 1.68.2; 1.68.12; 1.68.16;
Fix memset usage.

-if an opencrypto(9) session is allocated, the driver is refcounted
and can not disappear -- no need to hold crypto_mtx to check the
driver list
(the whole check is questionable)
-crp->crp_cv (the condition variable) is used by userland cryptodev
exclusively -- move its initialization there, no need to waste
cycles of in-kernel callers
-add a comment which members of "struct cryptop" are used
by opencrypto(9) and which by crypto(4)
(this should be split, no need to waste memory for in-kernel callers)

branches: 1.66.2;
allow testing of GCM/GMAC code from userland

fix building of a linked list if multiple algorithms are requested
in a session -- this just didn't work

catch some corner cases of user input

copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

-remove references to crypto/arc4/arc4.* -- the code isn't used
anywhere afaics
(The confusion comes probably from use of arc4random() at various places,
but this lives in libkern and doesn't share code with the former.)
-g/c non-implementation of arc4 encryption in swcrypto(4)
-remove special casing of ARC4 in crypto(4) -- the point is that it
doesn't use an IV, and this fact is made explicit by the new "ivsize"
property of xforms

If symmetric encryption is done from userland crypto(4) and no IV
is specified, the kernel gets one from the random generator. Make sure it
is copied out to the user, otherwise the result is quite useless.

being here, export camellia-cbc through crypto(4) to allow userland tests

add an AES-CTR xform, from OpenBSD

-in the descriptor for encryption xforms, split the "blocksize" field
into "blocksize" and "IV size"
-add an "reinit" function pointer which, if set, means that the xform
does its IV handling itself and doesn't want the default CBC handling
by the framework (poor name, but left that way to avoid unecessary
differences)
This syncs with Open/FreeBSD, purpose is to allow non-CBC transforms.
Refer to ivsize instead of blocksize where appropriate.
(At this point, blocksize and ivsize are identical.)

split the "crypto_mtx" spinlock into 3: one spinlock each for
the incoming and outgoing request queues (which can be dealt with
by hardware accelerators) and an adaptive lock for "all the rest"
(mostly driver configuration, but also some unrelated stuff in
cryptodev.c which should be revisited)
The latter one seems to be uneeded at many places, but for now I've
done simple replacements only, except minor fixes (where
softint_schedule() was called without the lock held)

As a first step towards more fine-grained locking, don't require
crypto_{new.free}session() to be called with the "crypto_mtx"
spinlock held.
This doesn't change much for now because these functions acquire
the said mutex first on entry now, but at least it keeps the nasty
locks local to the opencrypto core.

make the compatibility code conditional on COMPAT_50

more "const"

branches: 1.53.2; 1.53.4;
Consistently use a single CRYPTO_SESID2HID-like macro.
Improve CRYPTO_DEBUG printing a bit:
print pointers with %p
print unsigned with %u rather than %d
use CRYPTO_SESID2LID instead of just casting to uint32_t

branches: 1.52.2; 1.52.4;
Add missing "break" for CRYPTO_CAST_CBC, and some assorted comment fixes.
openssl(1) checks for CAST (and others) on ~every startup.

If a multithreaded app closes an fd while another thread is blocked in
read/write/accept, then the expectation is that the blocked thread will
exit and the close complete.
Since only one fd is affected, but many fd can refer to the same file,
the close code can only request the fs code unblock with ERESTART.
Fixed for pipes and sockets, ERESTART will only be generated after such
a close - so there should be no change for other programs.
Also rename fo_abort() to fo_restart() (this used to be fo_drain()).
Fixes PR/26567

Rename fo_drain() to fo_abort(), 'drain' is used to mean 'wait for output
do drain' in many places, whereas fo_drain() was called in order to force
blocking read()/write() etc calls to return to userspace so that a close()
call from a different thread can complete.
In the sockets code comment out the broken code in the inner function,
it was being called from compat code.

Fix locking as Andy explained. Also fill in uid and gid like sys_pipe did.

Fix PR/37878 and PR/37550: Provide stat(2) for all devices and don't use
fbadop_stat.

Add fileops::fo_drain(), to be called from fd_close() when there is more
than one active reference to a file descriptor. It should dislodge threads
sleeping while holding a reference to the descriptor. Implemented only for
sockets but should be extended to pipes, fifos, etc.

Fixes the case of a multithreaded process doing something like the
following, which would have hung until the process got a signal.

thr0 accept(fd, ...)
thr1 close(fd)

Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

branches: 1.45.4;
Fix a race condition in opencrypto where the crypto request could be
completed by the crypto device, queued on the retq, but freed by the
ioctl lwp. The problem manifests as various panics relating to the
condvar inside the request. The problem can occur whenever the crypto
device completes the request immediately and the ioctl skips the cv_wait().

The problem can be reproduced by enabling cryptosoft and running an openssl
speed test. E.g.
sysctl -w kern.cryptodevallowsoft=-1
openssl speed -engine cryptodev -evp des-ede3-cbc -multi 64

Add a macro for TAILQ_FOREACH_REVERSE_SAFE() to queue.h, since this
was missing and the opencrypto code removes requests from a list while
iterating with TAILQ_FOREACH_REVERSE().

Add missing cv_destroy() calls for the key request cleanup.

Reviewed by Thor Lancelot Simon.

branches: 1.44.4; 1.44.6; 1.44.8;
Coverity CID 5021: Check pointers before using.

Coverity CID 5027: Remove impossible test.

KNF, whitespace, b* -> mem*. No functional change.

branches: 1.41.2;
Make various bits of debug code compile again.

Remove clause 3 and 4 from TNF licenses

branches: 1.39.2;
As suggested by rmind, do not check return status of KM_SLEEP/PR_WAITOK
allocations. A little hair-raising but it does make the code easier to
read.

branches: 1.38.2;
Protect selrecord/selnotify calls with crypto_mtx; few misc changes.

fix 64-bit b0rkenness.

Extend crypto.4 interface:

* Asynchronous operation with result retrieval via select/poll
* Mutliple-request submit/retrieve ioctls
* Mutliple-session create-destroy ioctls

Revise/rewrite crypto.4 manual page. It should now be much easier to write
new applications to this API.

Measured performance for trivial requests: 84,000 very short modular math
operations/sec, 120,000 very short md5 hashes per sec (with a hardware
accellerator of moderate performance but very low latency, whose driver
will be contributed at a later date).

Contributed to TNF by Coyote Point Systems, Inc.

Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

branches: 1.34.6;
Some locking fixes (double-release mutex in softintr wakeup case, which I
hadn't tested) and an uninitialized field in cse which Darran Hunt
found. Some more debugging printfs.

Turn on MPSAFE for the kthread. We're not sure it's safe for the softint
yet. Gives a little performance kick for swcrypto with many requests on
MP systems.

Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

From Darran Hunt at Coyote Point: don't truncate HMAC to 96 bits unless
actually asked to.

Fixed in FreeBSD a while ago, discussed on tech-kern and tech-crypto.

This code never worked on a released version of FreeBSD in the form it's
been in in our tree, and certainly does not work on any version of FreeBSD
now. Run through unifdef -D__NetBSD__ -U__FreeBSD__ yielding a small
reduction of size and a dramatic improvement in readability.

No, this does not yield any meaningful decrease in patchability (unlike
mechanical changes that touch live source lines) -- try it and see.

Fix accidental checkin inverting the sense of cryptodev_allowsoft, which
is crazy but has always documented.

Make /dev/crypto properly cloning. Leave CRIOGET in place but note that
it is deprecated, no longer required, and will be removed in a future
release of NetBSD.

Dramatically reduce the size of the session structure by removing an
IOV_MAX array of iovecs where only the first was use. Saves an 8k
bzero on each session creation.

Convert fixed-size allocations in cryptodev.c to pools.

Some minor opencrypto fixes, one with a major performance impact for
OpenSSL:

1) Fix extremely misleading text in crypto.4 manual page so it does not
appear to claim that a new cloned file descriptor is required for every
session.

2) Fix severe performance problem (and fd leak!) in openssl cryptodev
engine resulting from misunderstanding probably caused by said manual
page text.

3) Check for session-ID wraparound in kernel cryptodev provider. Also,
start allocating sessions at 1, not 0 -- this will be necessary when
we add ioctls for the creation of multiple sessions at once, so we
can tell which if any creations failed.

Add constants for modular arithmetic operations other than exponentiation -- there's hardware out there which can do them.

branches: 1.26.16; 1.26.22; 1.26.28;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.25.4;
__unused removal on arguments; approved by core.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

branches: 1.23.2; 1.23.4;
fix incomplete initializer

Use the LWP cached credentials where sane.

integrate kauth.

Coverity CID 1083: Avoid possible NULL pointer deref.

don't use MALLOC with a non-constant size; use malloc instead.

branches: 1.18.2; 1.18.4;
sprinkle DPRINTF()...

branches: 1.17.2;
merge yamt-uio_vmspace branch.

- use vmspace rather than proc or lwp where appropriate.
the latter is more natural to specify an address space.
(and less likely to be abused for random purposes.)
- fix a swdmover race.

branches: 1.16.2; 1.16.4; 1.16.6;
merge ktrace-lwp.

- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

branches: 1.14.6;
No change. Forced commit to record commit message for previous revision, viz:

Fix vulnerability to a denial-of-service attack which passes a
length-0 crypto op. Check for zero length and return EINVAL, taken from:

http://cvsweb.FreeBSD.org/src/sys/opencrypto/cryptodev.c.diff?r1=1.25&r2=1.26

Original FreeBSD log mesage:

Modified files:
sys/opencrypto cryptodev.c
Log:
Fix bogus check. It was possible to panic the kernel by giving 0 length.
This is actually a local DoS, as every user can use /dev/crypto if there
is crypto hardware in the system and cryptodev.ko is loaded (or compiled
into the kernel).

Reported by: Mike Tancsa <mike@sentex.net>


thanks to Sam Leffler for passing on a heads-up about this issue.

*** empty log message ***

branches: 1.12.12;
Cloning cleanup:
1. make fileops const
2. add 2 new negative errno's to `officially' support the cloning hack:
- EDUPFD (used to overload ENODEV)
- EMOVEFD (used to overload ENXIO)
3. Created an fdclone() function to encapsulate the operations needed for
EMOVEFD, and made all cloners use it.
4. Centralize the local noop/badop fileops functions to:
fnullop_fcntl, fnullop_poll, fnullop_kqfilter, fbadop_stat

There's no need to pass a proc value when using UIO_SYSSPACE with
vn_rdwr(9) and uiomove(9).

OK'd by Jason Thorpe

branches: 1.10.4;
Wrap noisy pointless message about denied userspace requests with
`#ifdef CRYPTO_DEBUG', per Jason Thorpe's suggestion.

Clean up userlevel access to software kernel transforms, in preparation
for using /dev/crypto for OpenSSL:

1. Add comments explaining crypto_devallowsoft, explaining the
OpenBSD-style three-way logic actully implemented in crypto_newsession().

2. Pass crypto_devallowsoft as the final argument to crypto_newsession(),
instead of a constant 0 value.

3. Set the default value of crypto_devallowsoft to 1, to allow
/dev/crypto access only for hardware-supported transforms.

Items 1-3 may be revised to match the FreeBSD two-way logic, if the
consensus is that there's no point to forcing software transforms.
But as a first step, let the description match what the code actually does.

GC unused variables usercrypto, userasmcrypto, cryptodevallowsoft from
cryptodev.c, in favour of variables crypto_usercrypto, crypto_userasmcrypto,
crypto_devallowsoft, which are used as well as defined in crypto.c.

Remove '#ifdef notdef' around userspace ioctl() requests for
pure (non-HMAC) MD5 and SHA1.

Remove a bunch of unnecessary includes.

It's bad form to use the <opencrypto/rmd160.h> header file while
using the crypto/ripemd160/rmd160.c implementation. Remove the
opencrypto-local copies of these files entirely.

on netbsd, major # for /dev/crypto depends on arch

Pull up `done' flag for crypto operations from FreeBSD. FreeBSD deltas:
cryptodev.c: 1.4.2.3 -> 1.4.2.4
cryptodev.h: 1.4.2.4 -> 1.4.2.5

Garbage-collect references to OpenBSD-only <dev/rndvar.h>.

Remove vestiges of OpenBSD <sys/md5k.h> header.

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

For device modules that provide both auto-config and /dev/xxx
interfaces, make sure that initialization and destruction
follow the proper sequence. This is triggered by the recent
changes to the devsw stuff; per riastradh@ the required call
sequence is:

devsw_attach()
config_init_component() or config_cf*_attach()
...
config_fini_component() or config_cf*_detach()
devsw_detach()

While here, add a few missing calls to some of the detach
routines.

Testing of these changes has been limited to:
1. compile without build break
2. no related test failures from atf
3. modload/modunload work as well as
before.

No functional device testing done, since I don't have any
of these devices. Let me know of any damage I might cause
here!

XXX Some of the modules affected by this commit are already
XXX broken; see kern/56772. This commit does not break
any additional modules (as far as I know).

Rename enc_xform_rijndael128 -> enc_xform_aes.

Update netipsec dependency.

slightly change and fix the semantics of pool_set*wat(), pool_sethardlimit()
and pool_prime() (and their pool_cache_* counterparts):

- the pool_set*wat() APIs are supposed to specify thresholds for the count of
free items in the pool before pool pages are automatically allocated or freed
during pool_get() / pool_put(), whereas pool_sethardlimit() and pool_prime()
are supposed to specify minimum and maximum numbers of total items
in the pool (both free and allocated). these were somewhat conflated
in the existing code, so separate them as they were intended.

- change pool_prime() to take an absolute number of items to preallocate
rather than an increment over whatever was done before, and wait for
any memory allocations to succeed. since pool_prime() can no longer fail
after this, change its return value to void and adjust all callers.

- pool_setlowat() is documented as not immediately attempting to allocate
any memory, but it was changed some time ago to immediately try to allocate
up to the lowat level, so just fix the manpage to describe the current
behaviour.

- add a pool_cache_prime() to complete the API set.

branches: 1.104.4;
If we get an error from devsw_attach(), don't destroy the error value,
since we need to return it to our caller. While we're here, improve
the value of the debug message by actually printing the error value.

Initialize the session variable to an impossible session to prevent compiler
warnings.

branches: 1.102.2;
crypto(4): accept CRYPTO_SHA2_384_HMAC and CRYPTO_SHA2_512_HMAC.

don't always panic when modunload crypto (int the pool destroy code, because
the pools are busy). XXX: this is still racy; we need to prevent creating
more sessions while destroying.

Rename the MODULE_*_HOOK() macros to MODULE_HOOK_*() as briefly
discussed on irc.

NFCI intended.

Ride the earlier kernel bump - it;s getting crowded.

Merge the [pgoyette-compat] branch

branches: 1.98.2; 1.98.4;
Typos.

add fo_name so we can identify the fileops in a simple way.

check results of pool_prime.

Divide crp_devflags from crp_flags to write exclusively.

CRYPTO_F_DQRETQ(new name is CRYPTODEV_F_RET) is used by cryptodev.c only.
It should be divided to other member.

sanitize in CIOCNCRYPTM and initialize comp_alg in CIOCNGSESSION

sanitize count used for kmem_alloc size.

Hmm, who uses CIOCNGSESSION, CIOCNFSESSION, CIOCNCRYPTM or CIOCNFKEYM?

branches: 1.92.2;
rename crypto_mtx to cryptodev_mtx

It is used by cryptodev.c and ocryptodev.c only.

add cryptkop alloc/free KPI instead of manipulating cryptkop_pool directly.

opencrypto: cleanup debug messages.

branches: 1.89.2;
separate crypto_drv_mtx from crypto_mtx.

crypto_mtx is used only for cryptodev.c and ocryptodev.c now.

the processing said "ghastly hacks" is unnecessary now.

fix race among crypto_done(), cryptoret(), and {cryptodev_op(), cryptodev_key()}.

crypto_op() waited to be set CRYPTO_F_DONE with crp->crp_cv.
However, there is context switch chances between being set CRYPTO_F_DONE in
crypto_done() and done cv_signal(crp->crp_cv) in cryptodev_cb(), that is,
cryptodev_op() thread can run to cv_destroy(crp->crp_cv) before cryptoret()
thread is waken up. As a result, cryptodev_cb() can call invalid(destroyed)
cv_signal(crp->crp_cv).

Furthermore, below two implementations cause other races.
- waiting CRYPTO_F_DONE with crp->crp_cv
- context witch chances between set CRYPTO_F_DONE and cv_signal(crp->crp_cv)

So, use other flag(CRYPTO_F_DQRETQ) for cryptodev_op() and cryptodev_key(),
and then call cv_signal(crp->crp_cv) immediately after set CRYPTO_F_DQRETQ.

Tested concurrent over 20 processes with software and hardware drivers.

fix processes accessing /dev/crypto stall when over three processes run with a hardware encryption driver

The process has stalled at cv_wait(&crp->crp_cv) because cryptodev_cb()
is not called as cryptoret() kthread keep waiting at cv_wait(&cryptoret_cv).
Previous opencrypto implementation assumes the thread from cryptodev.c
does all processing in the same context, so skips enqueueing and sending
cryptoret_cv. However, the context can be switched, e.g. when we use
a hardware encryption driver.

And add debug messages.

branches: 1.85.2; 1.85.4;
KNF. Remove extra spaces. No functional change.

include "ioconf.h" to get the 'void <driver>attach(int count);' prototype.

Trivial printf format changes and typo fix

branches: 1.82.2;
Return ENOSPC instead of ENOMEM when there is no room in the buffer to
store results. ENOMEM in this subsystem means we cannot allocate more
requests or internal buffers for xforms.

Try not to use f_data, use f_fcrypt to get a correctly typed pointer.

At least crypto_mtx needs initialisation here. Spotted during PR/49065
investigation.

Add d_discard to all struct cdevsw instances I could find.

All have been set to "nodiscard"; some should get a real implementation.

branches: 1.78.2;
Change (mostly mechanically) every cdevsw/bdevsw I can find to use
designated initializers.

I have not built every extant kernel so I have probably broken at
least one build; however I've also found and fixed some wrong
cdevsw/bdevsw entries so even if so I think we come out ahead.

Undo previous - it still needs a lot more work.

For now, we'll use the hand-crafted cf* structures and directly
call all the config routines.

Replace home-grown config with standardized calls to
config_{init,fini}_component()

As requested by mrg@, since there is still a small window during which
the in-module ref-counting can fail, completely disable auto-unload.

Implement in-module ref-counting, and do not allow auto-unload if there
are existing references.

Note that manual unloading is not prevented.

OK christos@

XXX Also note that there is still a small window where the ref-count can
XXX be decremented, and then the process/thread preempted. If auto-unload
XXX happens before that thread can return from the module's code, bad
XXX things (tm) could happen.

knf: Blank line even if no variable declarations.

bail out unloading for now

When crypto(4) is built-in, crypto_modcmd() doesn't need to handle all
the auto-config stuff.

While here, ensure that we depend on opencrypto.

Modularize the opencrypto components and link to the build

Fix return value of cryptodev_msessionfin.

branches: 1.68.2; 1.68.12; 1.68.16;
Fix memset usage.

-if an opencrypto(9) session is allocated, the driver is refcounted
and can not disappear -- no need to hold crypto_mtx to check the
driver list
(the whole check is questionable)
-crp->crp_cv (the condition variable) is used by userland cryptodev
exclusively -- move its initialization there, no need to waste
cycles of in-kernel callers
-add a comment which members of "struct cryptop" are used
by opencrypto(9) and which by crypto(4)
(this should be split, no need to waste memory for in-kernel callers)

branches: 1.66.2;
allow testing of GCM/GMAC code from userland

fix building of a linked list if multiple algorithms are requested
in a session -- this just didn't work

catch some corner cases of user input

copy AES-XCBC-MAC support from KAME IPSEC to FAST_IPSEC
For this to fit, an API change in cryptosoft was adopted from OpenBSD
(addition of a "Setkey" method to hashes) which was done for GCM/GMAC
support there, so it might be useful in the future anyway.
tested against KAME IPSEC
AFAICT, FAST_IPSEC now supports as much as KAME.

-remove references to crypto/arc4/arc4.* -- the code isn't used
anywhere afaics
(The confusion comes probably from use of arc4random() at various places,
but this lives in libkern and doesn't share code with the former.)
-g/c non-implementation of arc4 encryption in swcrypto(4)
-remove special casing of ARC4 in crypto(4) -- the point is that it
doesn't use an IV, and this fact is made explicit by the new "ivsize"
property of xforms

If symmetric encryption is done from userland crypto(4) and no IV
is specified, the kernel gets one from the random generator. Make sure it
is copied out to the user, otherwise the result is quite useless.

being here, export camellia-cbc through crypto(4) to allow userland tests

add an AES-CTR xform, from OpenBSD

-in the descriptor for encryption xforms, split the "blocksize" field
into "blocksize" and "IV size"
-add an "reinit" function pointer which, if set, means that the xform
does its IV handling itself and doesn't want the default CBC handling
by the framework (poor name, but left that way to avoid unecessary
differences)
This syncs with Open/FreeBSD, purpose is to allow non-CBC transforms.
Refer to ivsize instead of blocksize where appropriate.
(At this point, blocksize and ivsize are identical.)

split the "crypto_mtx" spinlock into 3: one spinlock each for
the incoming and outgoing request queues (which can be dealt with
by hardware accelerators) and an adaptive lock for "all the rest"
(mostly driver configuration, but also some unrelated stuff in
cryptodev.c which should be revisited)
The latter one seems to be uneeded at many places, but for now I've
done simple replacements only, except minor fixes (where
softint_schedule() was called without the lock held)

As a first step towards more fine-grained locking, don't require
crypto_{new.free}session() to be called with the "crypto_mtx"
spinlock held.
This doesn't change much for now because these functions acquire
the said mutex first on entry now, but at least it keeps the nasty
locks local to the opencrypto core.

make the compatibility code conditional on COMPAT_50

more "const"

branches: 1.53.2; 1.53.4;
Consistently use a single CRYPTO_SESID2HID-like macro.
Improve CRYPTO_DEBUG printing a bit:
print pointers with %p
print unsigned with %u rather than %d
use CRYPTO_SESID2LID instead of just casting to uint32_t

branches: 1.52.2; 1.52.4;
Add missing "break" for CRYPTO_CAST_CBC, and some assorted comment fixes.
openssl(1) checks for CAST (and others) on ~every startup.

If a multithreaded app closes an fd while another thread is blocked in
read/write/accept, then the expectation is that the blocked thread will
exit and the close complete.
Since only one fd is affected, but many fd can refer to the same file,
the close code can only request the fs code unblock with ERESTART.
Fixed for pipes and sockets, ERESTART will only be generated after such
a close - so there should be no change for other programs.
Also rename fo_abort() to fo_restart() (this used to be fo_drain()).
Fixes PR/26567

Rename fo_drain() to fo_abort(), 'drain' is used to mean 'wait for output
do drain' in many places, whereas fo_drain() was called in order to force
blocking read()/write() etc calls to return to userspace so that a close()
call from a different thread can complete.
In the sockets code comment out the broken code in the inner function,
it was being called from compat code.

Fix locking as Andy explained. Also fill in uid and gid like sys_pipe did.

Fix PR/37878 and PR/37550: Provide stat(2) for all devices and don't use
fbadop_stat.

Add fileops::fo_drain(), to be called from fd_close() when there is more
than one active reference to a file descriptor. It should dislodge threads
sleeping while holding a reference to the descriptor. Implemented only for
sockets but should be extended to pipes, fifos, etc.

Fixes the case of a multithreaded process doing something like the
following, which would have hung until the process got a signal.

thr0 accept(fd, ...)
thr1 close(fd)

Fixes PR kern/41069 and PR kern/41070.

Extends the Opencrypto API to allow the destination buffer size to be
specified when its not the same size as the input buffer (i.e. for
operations like compress and decompress).
The crypto_op and crypt_n_op structures gain a u_int dst_len field.
The session_op structure gains a comp_alg field to specify a compression
algorithm.
Moved four ioctls to new ids; CIOCGSESSION, CIOCNGSESSION, CIOCCRYPT,
and CIOCNCRYPTM.
Added four backward compatible ioctls; OCIOCGSESSION, OCIOCNGSESSION,
OCIOCCRYPT, and OCIOCNCRYPTM.

Backward compatibility is maintained in ocryptodev.h and ocryptodev.c which
implement the original ioctls and set dst_len and comp_alg to 0.

Adds user-space access to compression features.

Adds software gzip support (CRYPTO_GZIP_COMP).

Adds the fast version of crc32 from zlib to libkern. This should be generally
useful and provide a place to start normalizing the various crc32 routines
in the kernel. The crc32 routine is used in this patch to support GZIP.

With input and support from tls@NetBSD.org.

branches: 1.45.4;
Fix a race condition in opencrypto where the crypto request could be
completed by the crypto device, queued on the retq, but freed by the
ioctl lwp. The problem manifests as various panics relating to the
condvar inside the request. The problem can occur whenever the crypto
device completes the request immediately and the ioctl skips the cv_wait().

The problem can be reproduced by enabling cryptosoft and running an openssl
speed test. E.g.
sysctl -w kern.cryptodevallowsoft=-1
openssl speed -engine cryptodev -evp des-ede3-cbc -multi 64

Add a macro for TAILQ_FOREACH_REVERSE_SAFE() to queue.h, since this
was missing and the opencrypto code removes requests from a list while
iterating with TAILQ_FOREACH_REVERSE().

Add missing cv_destroy() calls for the key request cleanup.

Reviewed by Thor Lancelot Simon.

branches: 1.44.4; 1.44.6; 1.44.8;
Coverity CID 5021: Check pointers before using.

Coverity CID 5027: Remove impossible test.

KNF, whitespace, b* -> mem*. No functional change.

branches: 1.41.2;
Make various bits of debug code compile again.

Remove clause 3 and 4 from TNF licenses

branches: 1.39.2;
As suggested by rmind, do not check return status of KM_SLEEP/PR_WAITOK
allocations. A little hair-raising but it does make the code easier to
read.

branches: 1.38.2;
Protect selrecord/selnotify calls with crypto_mtx; few misc changes.

fix 64-bit b0rkenness.

Extend crypto.4 interface:

* Asynchronous operation with result retrieval via select/poll
* Mutliple-request submit/retrieve ioctls
* Mutliple-session create-destroy ioctls

Revise/rewrite crypto.4 manual page. It should now be much easier to write
new applications to this API.

Measured performance for trivial requests: 84,000 very short modular math
operations/sec, 120,000 very short md5 hashes per sec (with a hardware
accellerator of moderate performance but very low latency, whose driver
will be contributed at a later date).

Contributed to TNF by Coyote Point Systems, Inc.

Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

branches: 1.34.6;
Some locking fixes (double-release mutex in softintr wakeup case, which I
hadn't tested) and an uninitialized field in cse which Darran Hunt
found. Some more debugging printfs.

Turn on MPSAFE for the kthread. We're not sure it's safe for the softint
yet. Gives a little performance kick for swcrypto with many requests on
MP systems.

Rework opencrypto to use a spin mutex (crypto_mtx) instead of "splcrypto"
(actually splnet) and condvars instead of tsleep/wakeup. Fix a few
miscellaneous problems and add some debugging printfs while there.

Restore set of CRYPTO_F_DONE in crypto_done() which was lost at some
point after this code came from FreeBSD -- it made it impossible to wait
properly for a condition.

Add flags analogous to the "crp" flags to the key operation's krp struct.
Add a new flag, CRYPTO_F_ONRETQ which tells us a request finished before
the kthread had a chance to dequeue it and call its callback -- this was
letting requests stick on the queues before even though done and copied
out.

Callers of crypto_newsession() or crypto_freesession() must now take the
mutex. Change netipsec to do so. Dispatch takes the mutex itself as
needed.

This was tested fairly extensively with the cryptosoft backend and lightly
with a new hardware driver. It has not been tested with FAST_IPSEC; I am
unable to ascertain whether FAST_IPSEC currently works at all in our tree.

pjd@FreeBSD.ORG, ad@NetBSD.ORG, and darran@snark.us pointed me in the
right direction several times in the course of this. Remaining bugs
are mine alone.

From Darran Hunt at Coyote Point: don't truncate HMAC to 96 bits unless
actually asked to.

Fixed in FreeBSD a while ago, discussed on tech-kern and tech-crypto.

This code never worked on a released version of FreeBSD in the form it's
been in in our tree, and certainly does not work on any version of FreeBSD
now. Run through unifdef -D__NetBSD__ -U__FreeBSD__ yielding a small
reduction of size and a dramatic improvement in readability.

No, this does not yield any meaningful decrease in patchability (unlike
mechanical changes that touch live source lines) -- try it and see.

Fix accidental checkin inverting the sense of cryptodev_allowsoft, which
is crazy but has always documented.

Make /dev/crypto properly cloning. Leave CRIOGET in place but note that
it is deprecated, no longer required, and will be removed in a future
release of NetBSD.

Dramatically reduce the size of the session structure by removing an
IOV_MAX array of iovecs where only the first was use. Saves an 8k
bzero on each session creation.

Convert fixed-size allocations in cryptodev.c to pools.

Some minor opencrypto fixes, one with a major performance impact for
OpenSSL:

1) Fix extremely misleading text in crypto.4 manual page so it does not
appear to claim that a new cloned file descriptor is required for every
session.

2) Fix severe performance problem (and fd leak!) in openssl cryptodev
engine resulting from misunderstanding probably caused by said manual
page text.

3) Check for session-ID wraparound in kernel cryptodev provider. Also,
start allocating sessions at 1, not 0 -- this will be necessary when
we add ioctls for the creation of multiple sessions at once, so we
can tell which if any creations failed.

Add constants for modular arithmetic operations other than exponentiation -- there's hardware out there which can do them.

branches: 1.26.16; 1.26.22; 1.26.28;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.25.4;
__unused removal on arguments; approved by core.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

branches: 1.23.2; 1.23.4;
fix incomplete initializer

Use the LWP cached credentials where sane.

integrate kauth.

Coverity CID 1083: Avoid possible NULL pointer deref.

don't use MALLOC with a non-constant size; use malloc instead.

branches: 1.18.2; 1.18.4;
sprinkle DPRINTF()...

branches: 1.17.2;
merge yamt-uio_vmspace branch.

- use vmspace rather than proc or lwp where appropriate.
the latter is more natural to specify an address space.
(and less likely to be abused for random purposes.)
- fix a swdmover race.

branches: 1.16.2; 1.16.4; 1.16.6;
merge ktrace-lwp.

- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

branches: 1.14.6;
No change. Forced commit to record commit message for previous revision, viz:

Fix vulnerability to a denial-of-service attack which passes a
length-0 crypto op. Check for zero length and return EINVAL, taken from:

http://cvsweb.FreeBSD.org/src/sys/opencrypto/cryptodev.c.diff?r1=1.25&r2=1.26

Original FreeBSD log mesage:

Modified files:
sys/opencrypto cryptodev.c
Log:
Fix bogus check. It was possible to panic the kernel by giving 0 length.
This is actually a local DoS, as every user can use /dev/crypto if there
is crypto hardware in the system and cryptodev.ko is loaded (or compiled
into the kernel).

Reported by: Mike Tancsa <mike@sentex.net>


thanks to Sam Leffler for passing on a heads-up about this issue.

*** empty log message ***

branches: 1.12.12;
Cloning cleanup:
1. make fileops const
2. add 2 new negative errno's to `officially' support the cloning hack:
- EDUPFD (used to overload ENODEV)
- EMOVEFD (used to overload ENXIO)
3. Created an fdclone() function to encapsulate the operations needed for
EMOVEFD, and made all cloners use it.
4. Centralize the local noop/badop fileops functions to:
fnullop_fcntl, fnullop_poll, fnullop_kqfilter, fbadop_stat

There's no need to pass a proc value when using UIO_SYSSPACE with
vn_rdwr(9) and uiomove(9).

OK'd by Jason Thorpe

branches: 1.10.4;
Wrap noisy pointless message about denied userspace requests with
`#ifdef CRYPTO_DEBUG', per Jason Thorpe's suggestion.

Clean up userlevel access to software kernel transforms, in preparation
for using /dev/crypto for OpenSSL:

1. Add comments explaining crypto_devallowsoft, explaining the
OpenBSD-style three-way logic actully implemented in crypto_newsession().

2. Pass crypto_devallowsoft as the final argument to crypto_newsession(),
instead of a constant 0 value.

3. Set the default value of crypto_devallowsoft to 1, to allow
/dev/crypto access only for hardware-supported transforms.

Items 1-3 may be revised to match the FreeBSD two-way logic, if the
consensus is that there's no point to forcing software transforms.
But as a first step, let the description match what the code actually does.

GC unused variables usercrypto, userasmcrypto, cryptodevallowsoft from
cryptodev.c, in favour of variables crypto_usercrypto, crypto_userasmcrypto,
crypto_devallowsoft, which are used as well as defined in crypto.c.

Remove '#ifdef notdef' around userspace ioctl() requests for
pure (non-HMAC) MD5 and SHA1.

Remove a bunch of unnecessary includes.

It's bad form to use the <opencrypto/rmd160.h> header file while
using the crypto/ripemd160/rmd160.c implementation. Remove the
opencrypto-local copies of these files entirely.

on netbsd, major # for /dev/crypto depends on arch

Pull up `done' flag for crypto operations from FreeBSD. FreeBSD deltas:
cryptodev.c: 1.4.2.3 -> 1.4.2.4
cryptodev.h: 1.4.2.4 -> 1.4.2.5

Garbage-collect references to OpenBSD-only <dev/rndvar.h>.

Remove vestiges of OpenBSD <sys/md5k.h> header.

Commit initial NetBSD port of the OpenCrypto Framework (OCF). This
code is derived from Sam Leffler's FreeBSD port of OCF, which is in
turn a port of Angelos Keromytis's OpenBSD work.
Credit to Sam and Angelos, any blame for the NetBSD port to me.

Rename enc_xform_rijndael128 -> enc_xform_aes.

Update netipsec dependency.

slightly change and fix the semantics of pool_set*wat(), pool_sethardlimit()
and pool_prime() (and their pool_cache_* counterparts):

- the pool_set*wat() APIs are supposed to specify thresholds for the count of
free items in the pool before pool pages are automatically allocated or freed
during pool_get() / pool_put(), whereas pool_sethardlimit() and pool_prime()
are supposed to specify minimum and maximum numbers of total items
in the pool (both free and allocated). these were somewhat conflated
in the existing code, so separate them as they were intended.

- change pool_prime() to take an absolute number of items to preallocate
rather than an increment over whatever was done before, and wait for
any memory allocations to succeed. since pool_prime() can no longer fail
after this, change its return value to void and adjust all callers.

- pool_setlowat() is documented as not immediately attempting to allocate
any memory, but it was changed some time ago to immediately try to allocate
up to the lowat level, so just fix the manpage to describe the current
behaviour.

- add a pool_cache_prime() to complete the API set.

branches: 1.104.4;
If we get an error from devsw_attach(), don't destroy the error value,
since we need to return it to our caller. While we're here, improve
the value of the debug message by actually printing the error value.

Initialize the session variable to an impossible session to prevent compiler
warnings.

branches: 1.102.2;
crypto(4): accept CRYPTO_SHA2_384_HMAC and CRYPTO_SHA2_512_HMAC.

don't always panic when modunload crypto (int the pool destroy code, because
the pools are busy). XXX: this is still racy; we need to prevent creating
more sessions while destroying.

Rename the MODULE_*_HOOK() macros to MODULE_HOOK_*() as briefly
discussed on irc.

NFCI intended.

Ride the earlier kernel bump - it;s getting crowded.

Merge the [pgoyette-compat] branch

branches: 1.98.2; 1.98.4;
Typos.

add fo_name so we can identify the fileops in a simple way.

check results of pool_prime.

Divide crp_devflags from crp_flags to write exclusively.

CRYPTO_F_DQRETQ(new name is CRYPTODEV_F_RET) is used by cryptodev.c only.
It should be divided to other member.

sanitize in CIOCNCRYPTM and initialize comp_alg in CIOCNGSESSION

sanitize count used for kmem_alloc size.

Hmm, who uses CIOCNGSESSION, CIOCNFSESSION, CIOCNCRYPTM or CIOCNFKEYM?

branches: 1.92.2;
rename crypto_mtx to cryptodev_mtx

It is used by cryptodev.c and ocryptodev.c only.

add cryptkop alloc/free KPI instead of manipulating cryptkop_pool directly.

opencrypto: cleanup debug messages.

branches: 1.89.2;
separate crypto_drv_mtx from crypto_mtx.

crypto_mtx is used only for cryptodev.c and ocryptodev.c now.

the processing said "ghastly hacks" is unnecessary now.

fix race among crypto_done(), cryptoret(), and {cryptodev_op(), cryptodev_key()}.

crypto_op() waited to be set CRYPTO_F_DONE with crp->crp_cv.
However, there is context switch chances between being set CRYPTO_F_DONE in
crypto_done() and done cv_signal(crp->crp_cv) in cryptodev_cb(), that is,
cryptodev_op() thread can run to cv_destroy(crp->crp_cv) before cryptoret()
thread is waken up. As a result, cryptodev_cb() can call invalid(destroyed)
cv_signal(crp->crp_cv).

Furthermore, below two implementations cause other races.
- waiting CRYPTO_F_DONE with crp->crp_cv
- context witch chances between set CRYPTO_F_DONE and cv_signal(crp->crp_cv)

So, use other flag(CRYPTO_F_DQRETQ) for cryptodev_op() and cryptodev_key(),
and then call cv_signal(crp->crp_cv) immediately after set CRYPTO_F_DQRETQ.

Tested concurrent over 20 processes with software and hardware drivers.

fix processes accessing /dev/crypto stall when over three processes run with a hardware encryption driver

The process has stalled at cv_wait(&crp->crp_cv) because cryptodev_cb()
is not called as cryptoret() kthread keep waiting at cv_wait(&cryptoret_cv).
Previous opencrypto implementation assumes the thread from cryptodev.c
does all processing in the same context, so skips enqueueing and sending
cryptoret_cv. However, the context can be switched, e.g. when we use
a hardware encryption driver.

And add debug messages.

branches: 1.85.2; 1.85.4;
KNF. Remove extra spaces. No functional change.

include "ioconf.h" to get the 'void <driver>attach(int count);' prototype.

Trivial printf format changes and typo fix

branches: 1.82.2;
Return ENOSPC instead of ENOMEM when there is no room in the buffer to
store results. ENOMEM in this subsystem means we cannot allocate more
requests or internal buffers for xforms.

Try not to use f_data, use f_fcrypt to get a correctly typed pointer.

At least crypto_mtx needs initialisation here. Spotted during PR/49065
investigation.

Add d_discard to all struct cdevsw instances I could find.

All have been set to "nodiscard"; some should get a real implementation.

branches: 1.78.2;
Change (mostly mechanically) every cdevsw/bdevsw I can find to use
designated initializers.

I have not built every extant kernel so I have probably broken at
least one build; however I've also found and fixed some wrong
cdevsw/bdevsw entries so even if so I think we come out ahead.