sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

virtio: remove virtio_alloc_virtqueues' flags arg

Summary:
the flags argument is unused.
Its initial design idea has been superceded by the addition of
virtio_setup_intr and related APIs.

Sponsored by: The FreeBSD Foundation

Reviewers: bryanv

Reviewed By: bryanv

Subscribers: cognet, imp

Differential Revision: https://reviews.freebsd.org/D41850

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

vtblk: secondary fix for dumping

The code paths while dumping do not got through busdma. As such,
safeguard against calling bus_dmamap_sync() with a NULL map. The x86
implementation tolerates this but others do not, resulting in a
NULL dereference panic when dumping to a vtblk device on arm64, riscv,
etc.

Fixes: 782105f7c898 ("vtblk: Use busdma")
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D37990

vtblk: Bypass iommu on powerpc

Virtio operates with physical addresses, while busdma is designed to
map these to produce bus addresses. On most supported platforms,
these two are interchangeable; on powerpc platforms, they are not.

When on powerpc, set an IOMMU of NULL, which causes the powerpc busdma
code to bypass the iommu mapping; this leaves us with the physical
buffer addresses which the virtio host expects to see.

Tested by: alfredo
Fixes: 782105f7c898 ("vtblk: Use busdma")
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D37891

vtblk: Fix dumping

Now that vtblk uses busdma, it keeps important information inside its
request structures. The functions used for kernel dumps synthesize
their own request structures rather than using structures initialized
with the necessary bits for busdma.

Add busdma-bypass paths. Since dumping writes contiguous blocks of
physical memory, vtblk doesn't need busdma in that case.

Reported by: glebius
Tested by: glebius
Differential Revision: https://reviews.freebsd.org/D37243

vtblk: Use busdma

We assume that bus addresses from busdma are the same thing as
"physical" addresses in the Virtio specification; this seems to
be true for the platforms on which Virtio is currently supported.

For block devices which are limited to a single data segment per
I/O, we request PAGE_SIZE alignment from busdma; this is necessary
in order to support unaligned I/Os from userland, which may cross
a boundary between two non-physically-contiguous pages. On devices
which support more data segments per I/O, we retain the existing
behaviour of limiting I/Os to (max data segs - 1) * PAGE_SIZE.

Reviewed by: bryanv
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D36667

vtblk: Include pointer to softc in request

No functional change intended.

Reviewed by: bryanv, imp
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D36666

vtblk: Requeue inside vtblk_request_execute

Most virtio_blk requests are launched from vtblk_startio; prior to this
commit, if vtblk_request_execute failed (e.g. due to a lack of space on
the virtio queue) vtblk_startio would requeue the request to be
reattempted later.

Add a flag "vbr_requeue_on_error" to requests and perform the requeuing
from inside vtblk_request_execute instead.

No functional change intended.

Reviewed by: bryanv, imp
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D36665

vtblk: Make vtblk_request_execute return void.

The error, if any, now gets stashed in the request structure. (Step 1
of reworking this driver to use busdma.)

No functional change intended.

Reviewed by: bryanv, imp
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D36664

kerneldump: remove physical argument from d_dumper

The physical address argument is essentially ignored by every dumper
method. In addition, the dump routines don't actually pass a real
address; every call to dump_append() passes a value of zero for
physical.

Reviewed by: markj
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D35173

virtio: Remove unused devclass arguments to DRIVER_MODULE.

virtio: Add KMSAN hooks for network and block devices

This ensures that host-written data is marked as initialized.

Sponsored by: The FreeBSD Foundation

virtio_blk: Fix issuing T_GET_ID before DRIVER_OK status

DRIVER_OK status is set after device_attach() succeeds. For now postpone
disk_create to attach_completed() method.

MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Reviewed by: grehan
Approved by: lwhsu (mentor)
Differential Revision: https://reviews.freebsd.org/D30049

virtio: Reduce boilerplate for device driver module definitions

Rather than have every device register itself for both virtio_pci and
virtio_mmio, provide a VIRTIO_DRIVER_MODULE wrapper to declare both,
merge VIRTIO_SIMPLE_PNPTABLE with VIRTIO_SIMPLE_PNPINFO and make the
latter register for both buses. This also has the benefit of abstracting
away the available transports and their names.

Reviewed by: bryanv
Differential Revision: https://reviews.freebsd.org/D28073

virtio: Handle possible failure of virtio_finalize_features()

Try to standardize how drivers negotiate feature and the
function names

Reviewed by: grehan (mentor)
Differential Revision: https://reviews.freebsd.org/D27930

virtio_blk: Use DISKFLAG_WRITE_PROTECT for RO disks

Reviewed by: grehan (mentor)
Differential Revision: https://reviews.freebsd.org/D27905

virtio_blk: Add modern (V1) support

Rework the header file changes from 2cc8a52 to use our
canonical upstream, Linux.

geom_disk already checks DISKFLAG_CANDELETE for BIO_DELETE
so remove an unnecessary check.

Reviewed by: grehan (mentor)
Differential Revision: https://reviews.freebsd.org/D27902

Make MAXPHYS tunable. Bump MAXPHYS to 1M.

Replace MAXPHYS by runtime variable maxphys. It is initialized from
MAXPHYS by default, but can be also adjusted with the tunable kern.maxphys.

Make b_pages[] array in struct buf flexible. Size b_pages[] for buffer
cache buffers exactly to atop(maxbcachebuf) (currently it is sized to
atop(MAXPHYS)), and b_pages[] for pbufs is sized to atop(maxphys) + 1.
The +1 for pbufs allow several pbuf consumers, among them vmapbuf(),
to use unaligned buffers still sized to maxphys, esp. when such
buffers come from userspace (*). Overall, we save significant amount
of otherwise wasted memory in b_pages[] for buffer cache buffers,
while bumping MAXPHYS to desired high value.

Eliminate all direct uses of the MAXPHYS constant in kernel and driver
sources, except a place which initialize maxphys. Some random (and
arguably weird) uses of MAXPHYS, e.g. in linuxolator, are converted
straight. Some drivers, which use MAXPHYS to size embeded structures,
get private MAXPHYS-like constant; their convertion is out of scope
for this work.

Changes to cam/, dev/ahci, dev/ata, dev/mpr, dev/mpt, dev/mvs,
dev/siis, where either submitted by, or based on changes by mav.

Suggested by: mav (*)
Reviewed by: imp, mav, imp, mckusick, scottl (intermediate versions)
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D27225

virtio: clean up empty lines in .c and .h files

Add VIRTIO_BLK_T_DISCARD support to the virtio-blk driver

If the hypervisor advertises support for the DISCARD command then the
guest can perform TRIM commands, freeing space on the backing store.

If VIRTIO_BLK_F_DISCARD is enabled, advertise DISKFLAG_CANDELETE

Tested with FreeBSD guests on bhyve and KVM

Reviewed by: jhb
Tested by: freqlabs
MFC after: 1 month
Relnotes: yes
Sponsored by: Klara Inc.
Differential Revision: https://reviews.freebsd.org/D21708

Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many)

r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are
still not MPSAFE (or already are but aren’t properly marked).
Use it in preparation for a general review of all nodes.

This is non-functional change that adds annotations to SYSCTL_NODE and
SYSCTL_PROC nodes using one of the soon-to-be-required flags.

Mark all obvious cases as MPSAFE. All entries that haven't been marked
as MPSAFE before are by default marked as NEEDGIANT

Approved by: kib (mentor, blanket)
Commented by: kib, gallatin, melifaro
Differential Revision: https://reviews.freebsd.org/D23718

Ever since the block layer expanded its command syntax beyond just
BIO_READ and BIO_WRITE, we've handled this expanded syntax poorly in
drivers when the driver doesn't support a particular command. Do a
sweep and fix that.

Reported by: imp

virtio(4): Add PNP match metadata for virtio devices

Register MODULE_PNP_INFO for virtio devices using the newbus PNP information
provided by the previous commit. Matching can be quite simple; existing
probe routines only matched on bus (implicit) and device_type. The same
matching criteria are retained exactly, but is now also available to
devmatch(8).

Reviewed by: bryanv, markj; imp (earlier version)
Differential Revision: https://reviews.freebsd.org/D20407

sys/dev: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Create an API to reset a struct bio (g_reset_bio). This is mandatory
for all struct bio you get back from g_{new,alloc}_bio. Temporary
bios that you create on the stack or elsewhere should use this before
first use of the bio, and between uses of the bio. At the moment, it
is nothing more than a wrapper around bzero, but that may change in
the future. The wrapper also removes one place where we encode the
size of struct bio in the KBI.

Do not report stripe size if it is equal to sector size.

MFC after: 1 week

Reenable VIRTIO_BLK_F_TOPOLOGY feature.

MFC after: 2 weeks

Rework vtblk dump handling of in flight requests

Previously, the driver resets the device and abandon the requests that
are caught in flight when the dump was initiated. This was problematic
if the system is resumed after the dump is completed.

While that is probably not the typical action, it is simple to rework
the driver to very likely have the device usable after the dump without
making it more likely for the dump to fail. The in flight requests are
simply queued for completion once the dump is finished.

Requested by: markj
MFC after: 1 month

Cleanup and performance improvement of the virtio_blk driver

- Add support for GEOM direct completion. Depending on the benchmark,
this tends to give a ~30% improvement w.r.t IOPs and BW.
- Remove an invariants check in the strategy routine. This assertion
is caught later on by an existing panic.
- Rename and resort various related functions to make more sense.

MFC after: 1 month

Add Virtio MMIO bus driver.

Sponsored by: DARPA, AFRL

Always append new bios to the tail of the queue, instead of sorting them

MFC after: 1 week

Add unmapped IO support to virtio_blk(4)

Merge virtio changes from projects/virtio

Contains projects/virtio commits:

r245738:
virtio: Minor man page tweaks
r246060:
virtio: Cleanup feature description printing
r246306:
virtio: Remove old debugging flag
r247238:
virtio: Remove PRIx64 macros from format strings
r247239:
virtio: Constify some fields
r247240:
virtio: Minor code simplifications
r249962:
virtio: Update to my freebsd.org email address

MFC after: 1 month

Merge several virtio_blk changes from projects/virtio

The notable changes of this commit are support for disk resizing
and chases updates to the spec regarding write caching.

Contains projects/virtio commits:

r245713:
virtio_blk: Replace __FUNCTION__ with __func__
r245714:
virtio_blk: Use more consistent mutex name
r245715:
virtio_blk: Print device name too if failed to reinit during dump
r245716:
virtio_blk: Remove an unuseful ASSERT
r245723:
virtio_blk: Record the vendor and device information
r245724:
virtio_blk: Add resize support
r245726:
virtio_blk: More verbose ASSERT messages
r245730:
virtio_blk: Tweak resize announcement message
r246061:
virtio_blk: Do not always read entire config
r246062:
virtio_blk: Use topology to set the stripe size/offset
r246307:
virtio_blk: Correct stripe offset calculation
r246063:
virtio_blk: Add support for write cache enable feature
r246303:
virtio_blk: Expand a comment
r252529:
virtio_blk: Improve write cache handling
r252681:
virtio_blk: Remove unneeded curly braces

MFC after: 1 month

Convert VirtIO to use ithreads instead of taskqueues

Contains projects/virtio commits:

r245709:
Each VirtIO device was scheduling its own taskqueue(9) to do the
off-level interrupt handling. ithreads(9) is the more nature way
to do this. The primary motivation for this work to better support
network multiqueue.
r245710:
virtio: Change virtqueue intr handlers to return void
r245711:
virtio_blk: Remove interrupt taskqueue
r245721:
vtnet: Remove interrupt taskqueue
r245722:
virtio_scsi: Remove interrupt taskqueue
r245747:
vtnet: Remove taskqueue fields missed in r245721

MFC after: 1 month

Only set the barrier flag if the feature was negotiated

When the VirtIO barrier feature is not negotiated, the driver
must enforce the proper ordering for BIO_ORDERED BIOs. All the
in-flight BIOs must complete before starting the BIO, and the
ordered BIO must complete before subsequent BIOs can start.

Also fix a few whitespace nits.

Reported by: neel
Approved by: grehan (mentor)
MFC after: 3 days

virtio: Start taskqueues threads after attach cannot fail

If virtio_setup_intr() failed during boot, we would hang in
taskqueue_free() -> taskqueue_terminate() for all the taskq
threads to terminate. This will never happen since the
scheduler is not running by this point.

Reported by: neel, grehan
Approved by: grehan (mentor)

Various VirtIO improvements

PCI:
- Properly handle interrupt fallback from MSIX to MSI to legacy.
The host may not have sufficient resources to support MSIX,
so we must be able to fallback to legacy interrupts.
- Add interface to get the (sub) vendor and device IDs.
- Rename flags to VTPCI_FLAG_* like other VirtIO drivers.
Block:
- No longer allocate vtblk_requests from separate UMA zone.
malloc(9) from M_DEVBUF is sufficient. Assert segment counts
at allocation.
- More verbose error and debug messages.
Network:
- Remove stray write once variable.
Virtqueue:
- Shuffle code around in preparation of converting the mb()s to
the appropriate atomic(9) operations.
- Only walk the descriptor chain when freeing if INVARIANTS is
defined since the result is only KASSERT()ed.

Submitted by: Bryan Venteicher (bryanv@daemoninthecloset.org)

Sync with Bryan Venteicher's virtio git repo:

d04e609bdd1973cc7d2e8b38b7dcfae057b0962d
virtio_blk: Use correct temporary variable in vtblk_poll_request

Obtained from: Bryan Venteicher bryanv at daemoninthecloset dot org

Catch up with Bryan Venteicher's virtio git repo:

a8af6270bd96be6ccd86f70b60fa6512b710e4f0
virtio_blk: Include function name in panic string

cbdb03a694b76c5253d7ae3a59b9995b9afbb67a
virtio_balloon: Do the notify outside of the lock

By the time we return from virtqueue_notify(), the descriptor
will be in the used ring so we shouldn't have to sleep.

10ba392e60692529a5cbc1e9987e4064e0128447
virtio: Use DEVMETHOD_END

80cbcc4d6552cac758be67f0c99c36f23ce62110
virtqueue: Add support for VIRTIO_F_RING_EVENT_IDX

This can be used to reduce the number of guest/host and
host/guest interrupts by delaying the interrupt until a
certain index value is reached.

Actual use by the network driver will come along later.

8fc465969acc0c58477153e4c3530390db436c02
virtqueue: Simplify virtqueue_nused()

Since the values just wrap naturally at UINT16_MAX, we
can just subtract the two values directly, rather than
doing 2's complement math.

a8aa22f25959e2767d006cd621b69050e7ffb0ae
virtio_blk: Remove debugging crud from 75dd732a

There seems to be an issue with Qemu (or FreeBSD VirtIO) that sets
the PCI register space for the device config to bogus values. This
only seems to happen after unloading and reloading the module.

d404800661cb2a9769c033f8a50b2133934501aa
virtio_blk: Use better variable name

75dd732a97743d96e7c63f7ced3c2169696dadd3
virtio_blk: Partially revert 92ba40e65

Just use the virtqueue to determine if any requests are
still inflight.

06661ed66b7a9efaea240f99f414c368f1bbcdc7
virtio_blk: error if allowed too few segments

Should never happen unless the host provides use with a
bogus seg_max value.

4b33e5085bc87a818433d7e664a0a2c8f56a1a89
virtio_blk: Sort function declarations

426b9f5cac892c9c64cc7631966461514f7e08c6
virtio_blk: Cleanup whitespace

617c23e12c61e3c2233d942db713c6b8ff0bd112
virtio_blk: Call disk_err() on error'd completed requests

081a5712d4b2e0abf273be4d26affcf3870263a9
virtio_blk: ASSERT the ready and inflight request queues are empty

a9be2631a4f770a84145c18ee03a3f103bed4ca8
virtio_blk: Simplify check for too many segments

At the cost of a small style violation.

e00ec09da014f2e60cc75542d0ab78898672d521
virtio_blk: Add beginnings of suspend/resume

Still not sure if we need to virtio_stop()/virtio_reinit()
the device before/after a suspend.

Don't start additional IO when marked as suspending.

47c71dc6ce8c238aa59ce8afd4bda5aa294bc884
virtio_blk: Panic when dealt an unhandled BIO cmd

1055544f90fb8c0cc6a2395f5b6104039606aafe
virtio_blk: Add VQ enqueue/dequeue wrappers

Wrapper functions managed the added/removing to the in-flight
list of requests.

Normally biodone() any completed IO when draining the virtqueue.

92ba40e65b3bb5e4acb9300ece711f1ea8f3f7f4
virtio_blk: Add in-flight list of requests

74f6d260e075443544522c0833dc2712dd93f49b
virtio_blk: Rename VTBLK_FLAG_DETACHING to VTBLK_FLAG_DETACH

7aa549050f6fc6551c09c6362ed6b2a0728956ef
virtio_blk: Finish all BIOs through vtblk_finish_bio()

Also properly set bio_resid in the case of errors. Most geom_disk
providers seem to do the same.

9eef6d0e6f7e5dd362f71ba097f2e2e4c3744882
Added function to translate VirtIO status to error code

ef06adc337f31e1129d6d5f26de6d8d1be27bcd2
Reset dumping flag when given unexpected parameters

393b3e390c644193a2e392220dcc6a6c50b212d9
Added missing VTBLK_LOCK() in dump handler

Obtained from: Bryan Venteicher bryanv at daemoninthecloset dot org

Catch up with Bryan Venteicher's virtio Hg repo:

c162516
Remove vtblk_sector_size

c162515
Wrap long license lines

c162514
Remove vtblk_unit

c162513
Wrap long lines in the license.

c162512
Remove verbose messages when link goes up/down.

A similar message is printed elsewhere as a result of
if_link_state_change().

c162511
Explicity compare pointer to NULL

c162510
Allocate the mac filter table at attach time.

c162509
Add real BSD licenses to the header files copied from Linux.

The chases upstream changes made in Linux awhile ago.

c162508
Only notify if we actually dequeued something.

c162507
Change a couple of if () { KASSERT(...) } to just KASSERTs.

In non-debug kernels, the if() { } probably get optomized
away, but I guess this is clearer.

c162506
Remove VIRTIO_BLK_F_TOPOLOGY fields in the config.

TOPOLOGY has since been removed from the spec, and the FreeBSD
didn't really do anything with the fields anyways.

c162505
Move vtblk_enqueue_request() outside the locks when getting the ident.

c162504
Remove soon to be uneeded trylock during dump [1].
http://lists.freebsd.org/pipermail/freebsd-current/2011-November/029226.html

c162503
Remove emtpy line

c162502
Drop frame if cannot allocate a vtnet_tx_header.

If we don't, we set OACTIVE, but if there are no
other frames in flight, vtnet_txeof() will never
be called to unset OACTIVE. The interface would
have to be down/up'ed in order to become usable.

We could be cuter here and only do this if the
virtqueue is emtpy, but its probably not worth
the complication.

c162501
Start mbuf replacement loop at 1 for clarity

Obtained from: Bryan Venteicher bryanv at daemoninthecloset dot org

Import virtio base, PCI front-end, and net/block/balloon drivers.
Tested on Qemu/KVM, VirtualBox, and BHyVe.

Currently built as modules-only on i386/amd64. Man pages not yet hooked
up, pending review.

Submitted by: Bryan Venteicher bryanv at daemoninthecloset dot org
Reviewed by: bz
MFC after: 4 weeks or so