mpi3mr: Minor tweak to task queue pausing

Use a while loop with cancel / drain to make sure that all tasks have
completed before proceeding to reset.

Suggested by: jhb
Sponsored by: Netflix

mpi3mr: Assume dma_hiaddr is BUS_SPACE_MAXADDR

No sense having a variable for this. So use BUS_SPACE_MAXADDR and remove
dma_hiaddr from softc.

Suggested by: jhb
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D42808

mpi3mr: Make these bus_dmamap_load calls synchronous

These calls "should" all be synchrounous. There's no bouncing that's
needed for them (at least in the typical case that we have a sane card
that has more bits of dma addresses decoded than we have memory), so
there's no errors possible. Ensure these calls are really synchronous
with BUS_DMA_NOWAIT flags (which should never fail now that the
bus_dmamem_alloc() has succeeded).

Reviewed by: mav, jhb, imp
Differential Revision: https://reviews.freebsd.org/D42606

mpi3mr: Fix MAXPHYS usage

This usage is obsolete. Replace with maximum bus space size. maxphys
will sort itself out at higher levels.

Reviewed by: mav, jhb, imp
Differential Revision: https://reviews.freebsd.org/D42605

mpi3mr: Add firmware version

Publish the firmware version on the card like we do for mps/mpr.

Sponsored by: Netflix
Reviewed by: mav
Differential Revision: https://reviews.freebsd.org/D42588

mpi3mr: Trivial trailing white space reduction

Sponsored by: Netflix

mpi3mr: Honor the dma mask from IOCFacts

The number of signficant bits that are decoded are returned in the flags
field of the IOCFacts structure from the device. Rather than assume the
worst with a pessimal 32-bit maximum, look at this value and pass it
along to all the dma map creation requests.

A lof of those creations are repetitive and could just inherit from the
base tag if we moved to the templated interface. This is called out as
desireable future work not done at this time.

In addition, due to a chicken and an egg problem, we have to allocate
some of the maps with a 32-bit loaddr. These are the ones we need to
read iocfacts. And they are fine to be so restricted: they are little
used after startup, and when they are used, bouncing is fine.

Sponsored by: Netflix
Reviewed by: mav
Differential Revision: https://reviews.freebsd.org/D42559

mpi3mr: Reduce the scope of the reset_mutext

Reduce the scope of reset_mutext to protect the msleep in the watch dog
thread as well as the MPI3MR_FLAGS_SHUTDOWN bit. Use it to protect the
wakeup in mpi3mr_detach so this thread can exit sooner when we're trying
to do an orderly shutdown. Optimize the flow to check the sleep and
other conditions before going to sleep.

It's an open question if this should protect sc->unrecoverable, and if
we should wakeup the watchdog thread when we set it. We might also want
to move too booleans for the three flags that we have now in
mpi3mr_flags. There are a number of U8s that should really be bools and
we might want to also group them together to pack softc better.

Sponsored by: Netflix
Reviewed by: mav
Differential Revision: https://reviews.freebsd.org/D42539

mpi3mr: Remove unused fields in struct mpi3mr_cmd

All of these fields are either unused, or just initialized. Remove
them. This saves about 1MB of memory for the cards that I have which can
do 8k transactions at once.

Sponsored by: Netflix
Reviewed by: mav, jhb
Differential Revision: https://reviews.freebsd.org/D42538

mpi3mr: Don't hold fwevt_lock over call to taskqueue_drain

Holding fwevt_lock when we call taskqueue_drain can lead to deadlock
because it's draining a queue needs fwevt_lock to do work, so that other
thread will try to take out the lock and block, making the thread never
finish and taskqueue_drain never complete. There's a witness
warning/error for this which was exposed when the lock was converted to
a MTX_DEF lock from a MTX_SPIN prior to committing to the FreeBSD tree.

The lock appears to be to protect against additional items being added
to the event list while we're doing a reset. Since the taskqueue is
blocked, items can get added to the list, but won't be processed during
the reset, but there is still a (likely small) race between the
taskqueue_drain and the taskqueue_block calls where an interrupt could
fire on another CPU, resulting in a task being enqueued and started
before the block can take effect. The only way to fix that race is to
turn off interrupt processing during a reset. So we replace a deadlock
with a smaller race.

Sponsored by: Netflix
Reviewed by: sumit.saxena_broadcom.com, mav, jhb
Differential Revision: https://reviews.freebsd.org/D42537

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

mpi3mr: 3rd Generation Tri-Mode NVMe/SAS/SATA MegaRaid / eHBA

This is Broadcom's mpi3mr driver for FreeBSD version 8.6.0.2.0.
The mpi3mr driver supports Broadcom SAS4116-based cards in the 9600
series: 9670W-16i, 9670-24i, 9660-16i, 9620-16i, 9600-24i, 9600-16i,
9600W-16e, 9600-16e, 9600-8i8e.

Initially only available as a module and on amd64/arm64, since that's
how it has been tested to date. Future commits will add it to the kernel
build and may expand the architectures it is supported on.

Co-authored-by: Chandrakanth Patil <chandrakanth.patil@broadcom.com>
Feedback-by: ken (prior versions)
Reviewed-by: imp
RelNotes: yes
Differential-Revision: https://reviews.freebsd.org/D36771
Differential-Revision: https://reviews.freebsd.org/D36772