Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r265054:
Reduce number of opens by REOM RAID during provider taste.

Instead opening/closing provider by each of metadata classes, do it only
once in core code. Since for SCSI disks open/close means sending some
SCSI commands to the device, this change reduces taste time.

MFC r256610:
Add unmapped I/O support to GEOM RAID.

MFC r254275:
Return error when opening read-only volumes (like RAID4/5/...) for writing.
Previously opens succeeded, but actual write operations returned errors.

MFC r253706:
Introduce 3 seconds timeout on `graid stop` command (mostly with -f flag).
Since completion waiting goes in g_event thread, it may cause GEOM deadlock
if consumer on top (for example, ZFS) uses g_event thread for closing.

MFC r249974:
Return "descr" field alike to "Intel RAID1 volume" for GEOM RAID to make
it look better in bsdinstall.

MFC r248068 (by sbruno):
Add legacy support to geom raid to create a /dev/arX device for support
of upgrading older machines using ataraid(4) to newer releases.

This optional parameter is controlled via kern.geom.raid.legacy_aliases
and will create a /dev/ar0 device that will point at /dev/raid/r0 for
example.

Tested on Dell SC 1425 DDF-1 format software raid controllers installing from
stable/7 and upgrading to stable/9 without having to adjust /etc/fstab

MFC r245363:
Improve support for disabled disks. If disabled disk disconnected and then
reconnected back, leave it as disabled. If new disk inserted instead of
disabled, rebuild it and leave as enabled.

MFC r245326:
Add basic support for Intel Rapid Recover Technology (Intel RRT).
It is alike to RAID1, but with dedicating master and recovery disks and
providing manual control over synchronization. It allows to use recovery
disk as snapshot of the master disk from the time of the last sync.

This implementation is not functionaly complete comparing to Windows,
but it is better then silent conversion to RAID1 on first boot.

MFC r242323, r242328:
Add basic BIO_DELETE support to GEOM RAID class for all RAID levels.

If at least one subdisk in the volume supports it, BIO_DELETE requests
will be propagated down. Unfortunatelly, for RAID levels with redundancy
unmapped blocks will be mapped back during first rebuild/resync process.

MFC r242314:
Make GEOM RAID more aggressive in marking volumes as clean on shutdown
and move that action from shutdown_pre_sync to shutdown_post_sync stage
to avoid extra flapping.

ZFS tends to not close devices on shutdown, that doesn't allow GEOM RAID
to shutdown gracefully. To handle that, mark volume as clean just when
shutdown time comes and there are no active writes.

MFC r239175:
Add missing FAILED event to g_raid_subdisk_event2str() to print it properly
in debug messages.

MFC r240465:
Add global and per-module sysctls/tunables to enable/disable metadata taste.
That should help to handle some cases when disk has some RAID metadata that
should be ignored, especially during boot.

MFC r234458, r234603, r234610, r234727, r234816, r234848, r234868,
r234869, r234899, r234940, r234993, r234994, r235071 -c r235076, r235080,
r235096:
- Add support for the DDF metadata format, as defined by the SNIA Common
RAID Disk Data Format Specification v2.0;
- Add support for reading non-degraded RAID4/5/5E/5EE/5R/6/MDF volumes.

Sponsored by: iXsystems, Inc.

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Include sys/sbuf.h directly.

Reviewed by: pjd

Reduce geom_raid log verbosity.

Bunch of small bugfixes and cleanups.

Found with: Clang Static Analyzer

MFgraid/head:
Add new RAID GEOM class, that is going to replace ataraid(4) in supporting
various BIOS-based software RAIDs. Unlike ataraid(4) this implementation
does not depend on legacy ata(4) subsystem and can be used with any disk
drivers, including new CAM-based ones (ahci(4), siis(4), mvs(4), ata(4)
with `options ATA_CAM`). To make code more readable and extensible, this
implementation follows modular design, including core part and two sets
of modules, implementing support for different metadata formats and RAID
levels.

Support for such popular metadata formats is now implemented:
Intel, JMicron, NVIDIA, Promise (also used by AMD/ATI) and SiliconImage.

Such RAID levels are now supported:
RAID0, RAID1, RAID1E, RAID10, SINGLE, CONCAT.

For any all of these RAID levels and metadata formats this class supports
full cycle of volume operations: reading, writing, creation, deletion,
disk removal and insertion, rebuilding, dirty shutdown detection
and resynchronization, bad sector recovery, faulty disks tracking,
hot-spare disks. For Intel and Promise formats there is support multiple
volumes per disk set.

Look graid(8) manual page for additional details.

Co-authored by: imp
Sponsored by: Cisco Systems, Inc. and iXsystems, Inc.