Split KASSERT(A && B) into KASSERT(A); KASSERT(B).

fix spelling mistakes, mainly in comments and log messages.

virtio_read_device_config_le_[24]: Fix for {aarch64,arm}eb

Stop byte-swapping for big-endian aarch64 and arm to fix
corrupted read for, e.g., sc_taglen for vio9p(4).

As described as comments in virtio_pci.c, big-endian aarch64 and
armv7 (armeb) are somewhat special. At the moment, all supported
virtual/real machines are configured as little-endian, and only
CPU cores are switched to big-endian mode during early boot stage.

Most peripheral buses for armeb are LSB as a result, and their
default bus_space(9) functions swap byte-order. Therefore, PIOed
data from memory-mapped devices, as well as pci(4) ones, are
actually LSB, but observed as MSB for armeb.

Therefore, we should not swap byte-order further in
virtio_read_device_config_le_[24] on armeb.

Thanks ozaki-r@ for comments.

XXX Centralize?
See virtio_pci.c and sys/dev/fdt/virtio_mmio_fdt.c also.

virtio(4): change members of struct vring_desc_extra before free a slot

This prevents the following race condition.
1. Thread-A: calls virtio_dequeue_commit() and
puts a slot into free descriptor chain in vq_free_slot()
2. Thread-B: calls virtio_enqueue_prep() and get the slot stored by Thread-A
3. Thread-B: calls virtio_enqueue_reserve() and
changes desc_base and desc_free_idx for the slot
4. Thread-A: changes the same members updated by Thread-B

reported by hannken, thanks.

virtio(4): change variable name, nfc

Fix not to allocate unnecessary descriptor

fixes PR/57358

Use enumeration for state of a child driver instead of flags
and check its detaching by using sc->sc_child in virtio_softc

pointed out by riastradh, thanks.
fixes PR/57357

Added flags to store status of attaching a virtio device

This prevents a panic on reboot after a virtio device had called
virtio_child_attach_failed().

Use descriptor chain for free slots instead of vq_entry list

Descriptors can be chained by themself. And descriptors added to
avail ring or used ring are already chained. But it was not used
for unused descriptors and another linked list structure named
vq_entry was used.
The chain is also used for unused descriptors to make virtio(4)
simpler.

virtio(4): Fix sizing of virtqueue allocation.

vq->vq_avail[0].ring is a zero-length array, and thus sizeof is zero;
likewise vq->vq_used[0].ring.

Use vq->vq_avail[0].ring[0] and vq->vq_used[0].ring[0] to fix this
and restore the previous allocation sizing logic.

XXX We shouldn't use zero-length arrays here -- they are asking for
trouble like this, and C99 has a standard way to express what we're
actually trying to get at it, flexible array members.

PR kern/57304

Reported-by: syzbot+7fb1047f5dfa33b26331@syzkaller.appspotmail.com

Use PRIuBUSSIZE to print bus_size_t variables.

virtio(4): Avoid name collision with global intrhand on sparc64.

Pacifies -Werror=shadow.

No functional change intended.

Setup virtqueues after registering them to virtio_softc

Added check of pointer for allocated memory before release of resource

Added functions to set interrupt handler and index into virtqueue

Set virtqueues in virtio_child_attach_finish

The number of virtqueue maybe change in a part of VirtIO devices
(e.g. vioif(4)). And it is fixed after negotiation of features.
So the configuration is moved into the function.

Subtly adjust criteria for notification of Virtio devices in
VIRTIO_F_RING_EVENT_IDX mode.

This fixes stalls in the vioif(4) transmit path that would happen sooner
or later, depending on interface transmit utilization, and were
particularly noticable with a NetBSD guest transmitting to its x86 KVM
host at multi-gigabit speeds.

Fix dmamap_syncs more; the number of ring elements is vq->vq_num, not
sc->sc_nvqs.

branches: 1.63.2;
Use PRIx64 in a debug print macro to print a uint64_t.

KNF

Fix previous by using roundup (and not howmany).

virtio(4): Use howmany from sys/param.h instead of open-coding it.

howmany will divide by VIRTIO_PAGE_SIZE instead of doing &
~(VIRTIO_PAGE_SIZE - 1), but it's a constant 4096 so this should make
no difference in the compiled output except possibly at -O0.

No functional change intended.

virtio(4): Sprinkle KNF.

No functional change intended.

virtio(4): Print numeric device type, even if unrecognized.

virtio(4): Membar and bus_dmamap_sync audit.

- Don't use membar_* for DMA.

- Sync only the header and payload of the rings separately as needed.
If we bounce, this avoids large memcpy when we only care about the
header.

- Sync uring with PREREAD before triggering anything that will return
data in it.

=> Move uring PREREAD in virtio_enqueue_commit to _before_ updating
vq->vq_avail->idx, which is the pointat which the DMA read is
triggered in the `device' (host).

=> Omit needless membar_consumer in virtio_enqueue_commit -- may not
work with DMA memory, and even if it does, redundant with
bus_dmamap_sync uring PREREAD here.

=> XXX Does the device/host ever return unsolicited entries in the
queue, or only solicited ones? If only solicited ones, the
PREREAD in virtio_init_vq is redundant.

- Sync uring with POSTREAD before we read from it. This way the DMA
read into our buffer has finished before we read from the buffer.

=> Add missing uring POSTREAD in virtio_vq_is_enqueued, between read of
vq->vq_used_idx and return to caller, so that the caller can
safely use virtio_dequeue.

=> Add missing uring POSTREADs in virtio_start_vq_intr:
. between entry from caller and the read of vq->vq_used_idx
. between the read of vq->vq_used_idx and return to caller,
so that the caller can safely use virtio_dequeue, just like
virtio_vq_is_enqueued

=> Move uring POSTREADs in virtio_enqueue_commit to _before_ reading
vq->vq_used->flags or *vq->vq_avail_event, not after.

- After we write to aring, sync it with PREWRITE. This way we finish
writing to our buffer before the DMA write from it.

=> Omit needless PREWRITE in virtio_init_vq -- we do the appropriate
PREWRITE in virtio_enqueue_commit now.

=> Convert membar_producer to bus_dmamap_sync PREWRITE in
virtio_enqueue_commit.

=> Omit incorrect aring POSTWRITE in virtio_enqueue_commit -- no need
because the DMA write may not have completed yet at this point,
and we already do a POSTWRITE in virtio_vq_is_enqueued.

=> Omit needless membar_producer in virtio_postpone_intr -- may not
work with DMA memory, and even if it does, redundant with
bus_dmamap_sync PREWRITE here.

- After xfers to aring have completed, sync it with POSTWRITE.

=> Add missing aring POSTWRITE in virtio_free_vq, in case there are
paths from virtio_enqueue_commit to here that don't go through
virtio_is_enqueued. (If there are no such paths, then maybe we
should KASSERT(vq->vq_queued == 0) in virtio_free_vq.)

virtio(4): Move comment for virtio_vq_intr.

No functional change intended.

fix typos in word "functions" in comments, mainly s/fuctions/functions/.

virtio: cosmetics - use (*pfn)(...) syntax.

Some people prefer to call function pointers without an explicit
dereference (that is purely cosmetic in this position), especially to
create faux c++ s->pfn(...) with function pointers in struct members.
Some prefer explicit dereference (that requires parens around it).

(pfn)(...) without dereference but with parens looks odd to both, so
make it conform to one of the established alternatives.

Same object code is generated.

virtio: stop reinit for safety when a device resetting is failed

virtio: fix missing release of interrupt resources

virtio: reconfigure MSI-X vector on re-initialization

This may fix a problem that some interrupts, for example
link-state change of vioif(4) are not notified.

divide setup routine of virtio interrupts
into establishment and device configuration

In virtio_alloc_vq change BUS_DMA_NOWAIT to BUS_DMA_WAITOK. It's always
called from attach routines.

Trailing whitespace

Better reading of 4.1.3.1, it seems that using 32 bit reads/writes is
mandatory for non-legacy devices.

Second round of cleaning up endian code. No more tailored code to maintain.

Rewrite and streamline virtio device config read/write and explicitly cater
for the Aarch64-eb bus problem with Qemu. This removes lots of bus_space
`magic' and cleans up the code.

Remove the virtio child driver finalisation check KASSERT that, while it
should never trigger, a possible bug in a child driver shouldn't have to panic
the kernel. Instead report the internal error.

Add VirtIO PCI v1.0 attachments and fix the drivers affected.

The vioif, ld, scsi, viornd and viomb devices were adjusted when needed and
tested both in legacy 0.9 and v1.0 attachments trough PCI on amd64, sparc64,
aarch64 and aarch64-eb. ACPI/FDT attachments also tested on
aarch64/aarch64-eb.

Known issues

* viomb on aarch64 works only with ACPI/FDT attachment but not with PCI
attachment. PCI and ACPI/FDT attachment works on aarch64-eb.

* virtio on sparc64 attaches but is it not functioning though not a
regression.

branches: 1.42.2;
Fix word-o of function name in comment

Set handlers implemented in child device of virtio(4) to virtqueue
instead of the commonized function

Register an interrupt handler for virtqueue to that for pci
to reduce function call

Remove VIRTIO_F_PCI_INTR_SOFTINT support for multiqueue

The functionality is no longer used by obsolating VIOIF_SOFTINT_INTR.

in many device attach paths, allocate memory with KM_SLEEP instead of KM_NOSLEEP
and remove code to handle failures that can no longer happen.

Add multiqueue support, virtio(4)

Suppress "not configured" message when no device is present

branches: 1.35.2;
Add cfprint_t virtiobusprint() and use it for virtio_mmio.

Fix queue address setup bug introduced in 1.31.

This use-before-initialization problem resulted in non-zero queues never
being initialized with their virtqueue pointer, and queue zero's pointer
to be clobbered with the virtqueue pointer for the higest-indexed ring.

remove now-irrelevant <dev/pci> includes

fix up virtio module for current structure; add virtio_pci module

Begin to detangle virtio from its PCI attachment

branches: 1.30.2;
Cosmetic - consistently use #define<SPACE>

Reorg the virtio(4) config(9) code to be explicitly pci specific.

We'll later use this for a non-pci virtio(4) usecase.

ok martin

branches: 1.28.2;
remove checks for failure after memory allocation calls that cannot fail:

kmem_alloc() with KM_SLEEP
kmem_zalloc() with KM_SLEEP
percpu_alloc()
pserialize_create()
psref_class_create()

all of these paths include an assertion that the allocation has not failed,
so callers should not assert that again.

Handle config change interrupts to inhibit sending packets while link down

PR kern/52103 by s-yamaguchi@IIJ

fix pasto for nbo_bus_space_write_4() - value needs to be uint32_t

Duh, remove accidently left over debug printfs

Backout previous, no need for these accessor in MD code

Provide an accessor for the pci attach args of a virtio device (if
available - future non-pci attachments may return NULL).
Sparc64 MD code needs this to match the OF node of the virtio device
for boot device recognition.

reorganize the attachment process for virtio child devices, so that
more common code is shared among the drivers, and it's possible for
the drivers to be correctly dynamically loaded; forbid direct access
to struct virtio_softc from the child driver code

use ipl level specified by child when attaching interrupts, instead of always
using IPL_NET

PR kern/52107: for now (while we still use the legacy virtio spec) at least
hack around the endianess issues with that.

branches: 1.19.2;
Fix the "typical usage" example for enqueuing a request - if
bus_dmamap_load(9) fails, we shouldn't call bus_dmamap_unload(9).

Modularize the ld driver and all of its attachments. Ensure that all
parents are capable of rescan (or otherwise provide a means of attaching
children post-initialization).

Prevent a panic during system shutdown when vioif(4) is used

branches: 1.16.2;
pci_intr_type() is required pci_chipset_tag_t argument by other than x86.

pointed out by nonaka@n.o.

KNF. Remove extra spaces. No functional change.

PR/50636: Ryo ONODERA: Fix incorrect kmem_free()

more device names

use dma64 if available so we don't have do bounce around too much.

Support MSI-X in virtio

Currently only vioif(4) uses the feature.

knakahara@ helped to migrate to pci_intr_alloc(9). Thanks!

Name the interrupt handler for intrctl

Remove a garbage blank line

Implement softint-based interrupt handling in if_vioif

Softint-based interrupt handling is considered as a future direction
of the (network) device driver architecture in NetBSD. pq3etsec of
ppc is already implemented based on the architecture (unlike pq3etsec,
this change doesn't include softint-based if_start). In this
architecture, a hardware interrupt handler just schedules a softint
and the softint performs actual interrupt processing. It reduces
processing in hardware interrupt context and allows Layer 2 network
stack (e.g., bridge, vlan and even bpf) run in softint context,
which makes it easy to implement fine-grain locking in the layer.

This is an experimental implementation of the architecture in if_viof.

virtio introduces a new flag VIRTIO_F_PCI_INTR_SOFTINT. If a driver
of virtio sets it to sc_flags of virtio_softc, virtio calls
softint_schedule in virtio_intr instead of directly calling the
interrupt handler of the driver.

When VIOIF_SOFTINT_INTR is on, vioif doesn't use the existing softint
(vioif_rx_softint) that is called from vioif_rx_vq_done. Because
vioif_rx_softint already runs in softint context and another softint
isn't needed. This change actually improves performance in some cases.

The feature is disabled by default and enabled when SOFTINT_INTR is
set somewhere (normally in a kernel configuration).

branches: 1.7.2;
use correct macro to handle subsys ids. Functional there is no
change, as both macros select the same bits.

branches: 1.6.2; 1.6.6;
Introduce VIRTIO_F_PCI_INTR_MPSAFE for virtio

It is set by a child driver, e.g., if_vioif. If set, virtio sets
PCI_INTR_MPSAFE for pci_intr_establish.

branches: 1.5.2;
make pci_intr_string and eisa_intr_string take a buffer and a length
instead of relying in local static storage.

Make sure to check if the driver has a valid intr handler in virtio_detach().

Fixes a panic during shutdown on KVM under ubuntu 13.04 with virtio,
as reported in PR kern/48105 by Richard Hansen.

Should be pulled up to netbsd-6 branches.

branches: 1.3.6; 1.3.8; 1.3.12; 1.3.14; 1.3.16; 1.3.22;
Small typo in symbolic name.

branches: 1.2.2;
Use symbolic PCI_PRODUCTs.

Import of the virtio driver written by MINOURA Makoto <minoura@netbsd.org>
with minor changes to make it compile an run on -current. This driver
speeds up disk and network access in virtual environments like KVM.

Enabled on i386 and amd64. Tested with a CentOS 5.7 x86_64 host.

See http://ozlabs.org/~rusty/virtio-spec/virtio.pdf for the specification.

virtio_read_device_config_le_[24]: Fix for {aarch64,arm}eb

Stop byte-swapping for big-endian aarch64 and arm to fix
corrupted read for, e.g., sc_taglen for vio9p(4).

As described as comments in virtio_pci.c, big-endian aarch64 and
armv7 (armeb) are somewhat special. At the moment, all supported
virtual/real machines are configured as little-endian, and only
CPU cores are switched to big-endian mode during early boot stage.

Most peripheral buses for armeb are LSB as a result, and their
default bus_space(9) functions swap byte-order. Therefore, PIOed
data from memory-mapped devices, as well as pci(4) ones, are
actually LSB, but observed as MSB for armeb.

Therefore, we should not swap byte-order further in
virtio_read_device_config_le_[24] on armeb.

Thanks ozaki-r@ for comments.

XXX Centralize?
See virtio_pci.c and sys/dev/fdt/virtio_mmio_fdt.c also.

virtio(4): change members of struct vring_desc_extra before free a slot

This prevents the following race condition.
1. Thread-A: calls virtio_dequeue_commit() and
puts a slot into free descriptor chain in vq_free_slot()
2. Thread-B: calls virtio_enqueue_prep() and get the slot stored by Thread-A
3. Thread-B: calls virtio_enqueue_reserve() and
changes desc_base and desc_free_idx for the slot
4. Thread-A: changes the same members updated by Thread-B

reported by hannken, thanks.

virtio(4): change variable name, nfc

Fix not to allocate unnecessary descriptor

fixes PR/57358

Use enumeration for state of a child driver instead of flags
and check its detaching by using sc->sc_child in virtio_softc

pointed out by riastradh, thanks.
fixes PR/57357

Added flags to store status of attaching a virtio device

This prevents a panic on reboot after a virtio device had called
virtio_child_attach_failed().

Use descriptor chain for free slots instead of vq_entry list

Descriptors can be chained by themself. And descriptors added to
avail ring or used ring are already chained. But it was not used
for unused descriptors and another linked list structure named
vq_entry was used.
The chain is also used for unused descriptors to make virtio(4)
simpler.

virtio(4): Fix sizing of virtqueue allocation.

vq->vq_avail[0].ring is a zero-length array, and thus sizeof is zero;
likewise vq->vq_used[0].ring.

Use vq->vq_avail[0].ring[0] and vq->vq_used[0].ring[0] to fix this
and restore the previous allocation sizing logic.

XXX We shouldn't use zero-length arrays here -- they are asking for
trouble like this, and C99 has a standard way to express what we're
actually trying to get at it, flexible array members.

PR kern/57304

Reported-by: syzbot+7fb1047f5dfa33b26331@syzkaller.appspotmail.com

Use PRIuBUSSIZE to print bus_size_t variables.

virtio(4): Avoid name collision with global intrhand on sparc64.

Pacifies -Werror=shadow.

No functional change intended.

Setup virtqueues after registering them to virtio_softc

Added check of pointer for allocated memory before release of resource

Added functions to set interrupt handler and index into virtqueue

Set virtqueues in virtio_child_attach_finish

The number of virtqueue maybe change in a part of VirtIO devices
(e.g. vioif(4)). And it is fixed after negotiation of features.
So the configuration is moved into the function.

Subtly adjust criteria for notification of Virtio devices in
VIRTIO_F_RING_EVENT_IDX mode.

This fixes stalls in the vioif(4) transmit path that would happen sooner
or later, depending on interface transmit utilization, and were
particularly noticable with a NetBSD guest transmitting to its x86 KVM
host at multi-gigabit speeds.

Fix dmamap_syncs more; the number of ring elements is vq->vq_num, not
sc->sc_nvqs.

branches: 1.63.2;
Use PRIx64 in a debug print macro to print a uint64_t.

KNF

Fix previous by using roundup (and not howmany).

virtio(4): Use howmany from sys/param.h instead of open-coding it.

howmany will divide by VIRTIO_PAGE_SIZE instead of doing &
~(VIRTIO_PAGE_SIZE - 1), but it's a constant 4096 so this should make
no difference in the compiled output except possibly at -O0.

No functional change intended.

virtio(4): Sprinkle KNF.

No functional change intended.

virtio(4): Print numeric device type, even if unrecognized.

virtio(4): Membar and bus_dmamap_sync audit.

- Don't use membar_* for DMA.

- Sync only the header and payload of the rings separately as needed.
If we bounce, this avoids large memcpy when we only care about the
header.

- Sync uring with PREREAD before triggering anything that will return
data in it.

=> Move uring PREREAD in virtio_enqueue_commit to _before_ updating
vq->vq_avail->idx, which is the pointat which the DMA read is
triggered in the `device' (host).

=> Omit needless membar_consumer in virtio_enqueue_commit -- may not
work with DMA memory, and even if it does, redundant with
bus_dmamap_sync uring PREREAD here.

=> XXX Does the device/host ever return unsolicited entries in the
queue, or only solicited ones? If only solicited ones, the
PREREAD in virtio_init_vq is redundant.

- Sync uring with POSTREAD before we read from it. This way the DMA
read into our buffer has finished before we read from the buffer.

=> Add missing uring POSTREAD in virtio_vq_is_enqueued, between read of
vq->vq_used_idx and return to caller, so that the caller can
safely use virtio_dequeue.

=> Add missing uring POSTREADs in virtio_start_vq_intr:
. between entry from caller and the read of vq->vq_used_idx
. between the read of vq->vq_used_idx and return to caller,
so that the caller can safely use virtio_dequeue, just like
virtio_vq_is_enqueued

=> Move uring POSTREADs in virtio_enqueue_commit to _before_ reading
vq->vq_used->flags or *vq->vq_avail_event, not after.

- After we write to aring, sync it with PREWRITE. This way we finish
writing to our buffer before the DMA write from it.

=> Omit needless PREWRITE in virtio_init_vq -- we do the appropriate
PREWRITE in virtio_enqueue_commit now.

=> Convert membar_producer to bus_dmamap_sync PREWRITE in
virtio_enqueue_commit.

=> Omit incorrect aring POSTWRITE in virtio_enqueue_commit -- no need
because the DMA write may not have completed yet at this point,
and we already do a POSTWRITE in virtio_vq_is_enqueued.

=> Omit needless membar_producer in virtio_postpone_intr -- may not
work with DMA memory, and even if it does, redundant with
bus_dmamap_sync PREWRITE here.

- After xfers to aring have completed, sync it with POSTWRITE.

=> Add missing aring POSTWRITE in virtio_free_vq, in case there are
paths from virtio_enqueue_commit to here that don't go through
virtio_is_enqueued. (If there are no such paths, then maybe we
should KASSERT(vq->vq_queued == 0) in virtio_free_vq.)

virtio(4): Move comment for virtio_vq_intr.

No functional change intended.

fix typos in word "functions" in comments, mainly s/fuctions/functions/.

virtio: cosmetics - use (*pfn)(...) syntax.

Some people prefer to call function pointers without an explicit
dereference (that is purely cosmetic in this position), especially to
create faux c++ s->pfn(...) with function pointers in struct members.
Some prefer explicit dereference (that requires parens around it).

(pfn)(...) without dereference but with parens looks odd to both, so
make it conform to one of the established alternatives.

Same object code is generated.

virtio: stop reinit for safety when a device resetting is failed

virtio: fix missing release of interrupt resources

virtio: reconfigure MSI-X vector on re-initialization

This may fix a problem that some interrupts, for example
link-state change of vioif(4) are not notified.

divide setup routine of virtio interrupts
into establishment and device configuration

In virtio_alloc_vq change BUS_DMA_NOWAIT to BUS_DMA_WAITOK. It's always
called from attach routines.

Trailing whitespace

Better reading of 4.1.3.1, it seems that using 32 bit reads/writes is
mandatory for non-legacy devices.

Second round of cleaning up endian code. No more tailored code to maintain.

Rewrite and streamline virtio device config read/write and explicitly cater
for the Aarch64-eb bus problem with Qemu. This removes lots of bus_space
`magic' and cleans up the code.

Remove the virtio child driver finalisation check KASSERT that, while it
should never trigger, a possible bug in a child driver shouldn't have to panic
the kernel. Instead report the internal error.

Add VirtIO PCI v1.0 attachments and fix the drivers affected.

The vioif, ld, scsi, viornd and viomb devices were adjusted when needed and
tested both in legacy 0.9 and v1.0 attachments trough PCI on amd64, sparc64,
aarch64 and aarch64-eb. ACPI/FDT attachments also tested on
aarch64/aarch64-eb.

Known issues

* viomb on aarch64 works only with ACPI/FDT attachment but not with PCI
attachment. PCI and ACPI/FDT attachment works on aarch64-eb.

* virtio on sparc64 attaches but is it not functioning though not a
regression.

branches: 1.42.2;
Fix word-o of function name in comment

Set handlers implemented in child device of virtio(4) to virtqueue
instead of the commonized function

Register an interrupt handler for virtqueue to that for pci
to reduce function call

Remove VIRTIO_F_PCI_INTR_SOFTINT support for multiqueue

The functionality is no longer used by obsolating VIOIF_SOFTINT_INTR.

in many device attach paths, allocate memory with KM_SLEEP instead of KM_NOSLEEP
and remove code to handle failures that can no longer happen.

Add multiqueue support, virtio(4)

Suppress "not configured" message when no device is present

branches: 1.35.2;
Add cfprint_t virtiobusprint() and use it for virtio_mmio.

Fix queue address setup bug introduced in 1.31.

This use-before-initialization problem resulted in non-zero queues never
being initialized with their virtqueue pointer, and queue zero's pointer
to be clobbered with the virtqueue pointer for the higest-indexed ring.

remove now-irrelevant <dev/pci> includes

fix up virtio module for current structure; add virtio_pci module

Begin to detangle virtio from its PCI attachment

branches: 1.30.2;
Cosmetic - consistently use #define<SPACE>

Reorg the virtio(4) config(9) code to be explicitly pci specific.

We'll later use this for a non-pci virtio(4) usecase.

ok martin

branches: 1.28.2;
remove checks for failure after memory allocation calls that cannot fail:

kmem_alloc() with KM_SLEEP
kmem_zalloc() with KM_SLEEP
percpu_alloc()
pserialize_create()
psref_class_create()

all of these paths include an assertion that the allocation has not failed,
so callers should not assert that again.

Handle config change interrupts to inhibit sending packets while link down

PR kern/52103 by s-yamaguchi@IIJ

fix pasto for nbo_bus_space_write_4() - value needs to be uint32_t

Duh, remove accidently left over debug printfs

Backout previous, no need for these accessor in MD code

Provide an accessor for the pci attach args of a virtio device (if
available - future non-pci attachments may return NULL).
Sparc64 MD code needs this to match the OF node of the virtio device
for boot device recognition.

reorganize the attachment process for virtio child devices, so that
more common code is shared among the drivers, and it's possible for
the drivers to be correctly dynamically loaded; forbid direct access
to struct virtio_softc from the child driver code

use ipl level specified by child when attaching interrupts, instead of always
using IPL_NET

PR kern/52107: for now (while we still use the legacy virtio spec) at least
hack around the endianess issues with that.

branches: 1.19.2;
Fix the "typical usage" example for enqueuing a request - if
bus_dmamap_load(9) fails, we shouldn't call bus_dmamap_unload(9).

Modularize the ld driver and all of its attachments. Ensure that all
parents are capable of rescan (or otherwise provide a means of attaching
children post-initialization).

Prevent a panic during system shutdown when vioif(4) is used

branches: 1.16.2;
pci_intr_type() is required pci_chipset_tag_t argument by other than x86.

pointed out by nonaka@n.o.

KNF. Remove extra spaces. No functional change.

PR/50636: Ryo ONODERA: Fix incorrect kmem_free()

more device names

use dma64 if available so we don't have do bounce around too much.

Support MSI-X in virtio

Currently only vioif(4) uses the feature.

knakahara@ helped to migrate to pci_intr_alloc(9). Thanks!

Name the interrupt handler for intrctl

Remove a garbage blank line

Implement softint-based interrupt handling in if_vioif

Softint-based interrupt handling is considered as a future direction
of the (network) device driver architecture in NetBSD. pq3etsec of
ppc is already implemented based on the architecture (unlike pq3etsec,
this change doesn't include softint-based if_start). In this
architecture, a hardware interrupt handler just schedules a softint
and the softint performs actual interrupt processing. It reduces
processing in hardware interrupt context and allows Layer 2 network
stack (e.g., bridge, vlan and even bpf) run in softint context,
which makes it easy to implement fine-grain locking in the layer.

This is an experimental implementation of the architecture in if_viof.

virtio introduces a new flag VIRTIO_F_PCI_INTR_SOFTINT. If a driver
of virtio sets it to sc_flags of virtio_softc, virtio calls
softint_schedule in virtio_intr instead of directly calling the
interrupt handler of the driver.

When VIOIF_SOFTINT_INTR is on, vioif doesn't use the existing softint
(vioif_rx_softint) that is called from vioif_rx_vq_done. Because
vioif_rx_softint already runs in softint context and another softint
isn't needed. This change actually improves performance in some cases.

The feature is disabled by default and enabled when SOFTINT_INTR is
set somewhere (normally in a kernel configuration).

branches: 1.7.2;
use correct macro to handle subsys ids. Functional there is no
change, as both macros select the same bits.

branches: 1.6.2; 1.6.6;
Introduce VIRTIO_F_PCI_INTR_MPSAFE for virtio

It is set by a child driver, e.g., if_vioif. If set, virtio sets
PCI_INTR_MPSAFE for pci_intr_establish.

branches: 1.5.2;
make pci_intr_string and eisa_intr_string take a buffer and a length
instead of relying in local static storage.

Make sure to check if the driver has a valid intr handler in virtio_detach().

Fixes a panic during shutdown on KVM under ubuntu 13.04 with virtio,
as reported in PR kern/48105 by Richard Hansen.

Should be pulled up to netbsd-6 branches.

branches: 1.3.6; 1.3.8; 1.3.12; 1.3.14; 1.3.16; 1.3.22;
Small typo in symbolic name.

branches: 1.2.2;
Use symbolic PCI_PRODUCTs.

Import of the virtio driver written by MINOURA Makoto <minoura@netbsd.org>
with minor changes to make it compile an run on -current. This driver
speeds up disk and network access in virtual environments like KVM.

Enabled on i386 and amd64. Tested with a CentOS 5.7 x86_64 host.

See http://ozlabs.org/~rusty/virtio-spec/virtio.pdf for the specification.

virtio(4): change members of struct vring_desc_extra before free a slot

This prevents the following race condition.
1. Thread-A: calls virtio_dequeue_commit() and
puts a slot into free descriptor chain in vq_free_slot()
2. Thread-B: calls virtio_enqueue_prep() and get the slot stored by Thread-A
3. Thread-B: calls virtio_enqueue_reserve() and
changes desc_base and desc_free_idx for the slot
4. Thread-A: changes the same members updated by Thread-B

reported by hannken, thanks.

virtio(4): change variable name, nfc

Fix not to allocate unnecessary descriptor

fixes PR/57358

Use enumeration for state of a child driver instead of flags
and check its detaching by using sc->sc_child in virtio_softc

pointed out by riastradh, thanks.
fixes PR/57357

Added flags to store status of attaching a virtio device

This prevents a panic on reboot after a virtio device had called
virtio_child_attach_failed().

Use descriptor chain for free slots instead of vq_entry list

Descriptors can be chained by themself. And descriptors added to
avail ring or used ring are already chained. But it was not used
for unused descriptors and another linked list structure named
vq_entry was used.
The chain is also used for unused descriptors to make virtio(4)
simpler.

virtio(4): Fix sizing of virtqueue allocation.

vq->vq_avail[0].ring is a zero-length array, and thus sizeof is zero;
likewise vq->vq_used[0].ring.

Use vq->vq_avail[0].ring[0] and vq->vq_used[0].ring[0] to fix this
and restore the previous allocation sizing logic.

XXX We shouldn't use zero-length arrays here -- they are asking for
trouble like this, and C99 has a standard way to express what we're
actually trying to get at it, flexible array members.

PR kern/57304

Reported-by: syzbot+7fb1047f5dfa33b26331@syzkaller.appspotmail.com

Use PRIuBUSSIZE to print bus_size_t variables.

virtio(4): Avoid name collision with global intrhand on sparc64.

Pacifies -Werror=shadow.

No functional change intended.

Setup virtqueues after registering them to virtio_softc

Added check of pointer for allocated memory before release of resource

Added functions to set interrupt handler and index into virtqueue

Set virtqueues in virtio_child_attach_finish

The number of virtqueue maybe change in a part of VirtIO devices
(e.g. vioif(4)). And it is fixed after negotiation of features.
So the configuration is moved into the function.

Subtly adjust criteria for notification of Virtio devices in
VIRTIO_F_RING_EVENT_IDX mode.

This fixes stalls in the vioif(4) transmit path that would happen sooner
or later, depending on interface transmit utilization, and were
particularly noticable with a NetBSD guest transmitting to its x86 KVM
host at multi-gigabit speeds.

Fix dmamap_syncs more; the number of ring elements is vq->vq_num, not
sc->sc_nvqs.

branches: 1.63.2;
Use PRIx64 in a debug print macro to print a uint64_t.

KNF

Fix previous by using roundup (and not howmany).

virtio(4): Use howmany from sys/param.h instead of open-coding it.

howmany will divide by VIRTIO_PAGE_SIZE instead of doing &
~(VIRTIO_PAGE_SIZE - 1), but it's a constant 4096 so this should make
no difference in the compiled output except possibly at -O0.

No functional change intended.

virtio(4): Sprinkle KNF.

No functional change intended.

virtio(4): Print numeric device type, even if unrecognized.

virtio(4): Membar and bus_dmamap_sync audit.

- Don't use membar_* for DMA.

- Sync only the header and payload of the rings separately as needed.
If we bounce, this avoids large memcpy when we only care about the
header.

- Sync uring with PREREAD before triggering anything that will return
data in it.

=> Move uring PREREAD in virtio_enqueue_commit to _before_ updating
vq->vq_avail->idx, which is the pointat which the DMA read is
triggered in the `device' (host).

=> Omit needless membar_consumer in virtio_enqueue_commit -- may not
work with DMA memory, and even if it does, redundant with
bus_dmamap_sync uring PREREAD here.

=> XXX Does the device/host ever return unsolicited entries in the
queue, or only solicited ones? If only solicited ones, the
PREREAD in virtio_init_vq is redundant.

- Sync uring with POSTREAD before we read from it. This way the DMA
read into our buffer has finished before we read from the buffer.

=> Add missing uring POSTREAD in virtio_vq_is_enqueued, between read of
vq->vq_used_idx and return to caller, so that the caller can
safely use virtio_dequeue.

=> Add missing uring POSTREADs in virtio_start_vq_intr:
. between entry from caller and the read of vq->vq_used_idx
. between the read of vq->vq_used_idx and return to caller,
so that the caller can safely use virtio_dequeue, just like
virtio_vq_is_enqueued

=> Move uring POSTREADs in virtio_enqueue_commit to _before_ reading
vq->vq_used->flags or *vq->vq_avail_event, not after.

- After we write to aring, sync it with PREWRITE. This way we finish
writing to our buffer before the DMA write from it.

=> Omit needless PREWRITE in virtio_init_vq -- we do the appropriate
PREWRITE in virtio_enqueue_commit now.

=> Convert membar_producer to bus_dmamap_sync PREWRITE in
virtio_enqueue_commit.

=> Omit incorrect aring POSTWRITE in virtio_enqueue_commit -- no need
because the DMA write may not have completed yet at this point,
and we already do a POSTWRITE in virtio_vq_is_enqueued.

=> Omit needless membar_producer in virtio_postpone_intr -- may not
work with DMA memory, and even if it does, redundant with
bus_dmamap_sync PREWRITE here.

- After xfers to aring have completed, sync it with POSTWRITE.

=> Add missing aring POSTWRITE in virtio_free_vq, in case there are
paths from virtio_enqueue_commit to here that don't go through
virtio_is_enqueued. (If there are no such paths, then maybe we
should KASSERT(vq->vq_queued == 0) in virtio_free_vq.)

virtio(4): Move comment for virtio_vq_intr.

No functional change intended.

fix typos in word "functions" in comments, mainly s/fuctions/functions/.

virtio: cosmetics - use (*pfn)(...) syntax.

Some people prefer to call function pointers without an explicit
dereference (that is purely cosmetic in this position), especially to
create faux c++ s->pfn(...) with function pointers in struct members.
Some prefer explicit dereference (that requires parens around it).

(pfn)(...) without dereference but with parens looks odd to both, so
make it conform to one of the established alternatives.

Same object code is generated.

virtio: stop reinit for safety when a device resetting is failed

virtio: fix missing release of interrupt resources

virtio: reconfigure MSI-X vector on re-initialization

This may fix a problem that some interrupts, for example
link-state change of vioif(4) are not notified.

divide setup routine of virtio interrupts
into establishment and device configuration

In virtio_alloc_vq change BUS_DMA_NOWAIT to BUS_DMA_WAITOK. It's always
called from attach routines.

Trailing whitespace

Better reading of 4.1.3.1, it seems that using 32 bit reads/writes is
mandatory for non-legacy devices.

Second round of cleaning up endian code. No more tailored code to maintain.

Rewrite and streamline virtio device config read/write and explicitly cater
for the Aarch64-eb bus problem with Qemu. This removes lots of bus_space
`magic' and cleans up the code.

Remove the virtio child driver finalisation check KASSERT that, while it
should never trigger, a possible bug in a child driver shouldn't have to panic
the kernel. Instead report the internal error.

Add VirtIO PCI v1.0 attachments and fix the drivers affected.

The vioif, ld, scsi, viornd and viomb devices were adjusted when needed and
tested both in legacy 0.9 and v1.0 attachments trough PCI on amd64, sparc64,
aarch64 and aarch64-eb. ACPI/FDT attachments also tested on
aarch64/aarch64-eb.

Known issues

* viomb on aarch64 works only with ACPI/FDT attachment but not with PCI
attachment. PCI and ACPI/FDT attachment works on aarch64-eb.

* virtio on sparc64 attaches but is it not functioning though not a
regression.

branches: 1.42.2;
Fix word-o of function name in comment

Set handlers implemented in child device of virtio(4) to virtqueue
instead of the commonized function

Register an interrupt handler for virtqueue to that for pci
to reduce function call

Remove VIRTIO_F_PCI_INTR_SOFTINT support for multiqueue

The functionality is no longer used by obsolating VIOIF_SOFTINT_INTR.

in many device attach paths, allocate memory with KM_SLEEP instead of KM_NOSLEEP
and remove code to handle failures that can no longer happen.

Add multiqueue support, virtio(4)

Suppress "not configured" message when no device is present

branches: 1.35.2;
Add cfprint_t virtiobusprint() and use it for virtio_mmio.

Fix queue address setup bug introduced in 1.31.

This use-before-initialization problem resulted in non-zero queues never
being initialized with their virtqueue pointer, and queue zero's pointer
to be clobbered with the virtqueue pointer for the higest-indexed ring.

remove now-irrelevant <dev/pci> includes

fix up virtio module for current structure; add virtio_pci module

Begin to detangle virtio from its PCI attachment

branches: 1.30.2;
Cosmetic - consistently use #define<SPACE>

Reorg the virtio(4) config(9) code to be explicitly pci specific.

We'll later use this for a non-pci virtio(4) usecase.

ok martin

branches: 1.28.2;
remove checks for failure after memory allocation calls that cannot fail:

kmem_alloc() with KM_SLEEP
kmem_zalloc() with KM_SLEEP
percpu_alloc()
pserialize_create()
psref_class_create()

all of these paths include an assertion that the allocation has not failed,
so callers should not assert that again.

Handle config change interrupts to inhibit sending packets while link down

PR kern/52103 by s-yamaguchi@IIJ

fix pasto for nbo_bus_space_write_4() - value needs to be uint32_t

Duh, remove accidently left over debug printfs

Backout previous, no need for these accessor in MD code

Provide an accessor for the pci attach args of a virtio device (if
available - future non-pci attachments may return NULL).
Sparc64 MD code needs this to match the OF node of the virtio device
for boot device recognition.

reorganize the attachment process for virtio child devices, so that
more common code is shared among the drivers, and it's possible for
the drivers to be correctly dynamically loaded; forbid direct access
to struct virtio_softc from the child driver code

use ipl level specified by child when attaching interrupts, instead of always
using IPL_NET

PR kern/52107: for now (while we still use the legacy virtio spec) at least
hack around the endianess issues with that.

branches: 1.19.2;
Fix the "typical usage" example for enqueuing a request - if
bus_dmamap_load(9) fails, we shouldn't call bus_dmamap_unload(9).

Modularize the ld driver and all of its attachments. Ensure that all
parents are capable of rescan (or otherwise provide a means of attaching
children post-initialization).

Prevent a panic during system shutdown when vioif(4) is used

branches: 1.16.2;
pci_intr_type() is required pci_chipset_tag_t argument by other than x86.

pointed out by nonaka@n.o.

KNF. Remove extra spaces. No functional change.

PR/50636: Ryo ONODERA: Fix incorrect kmem_free()

more device names

use dma64 if available so we don't have do bounce around too much.

Support MSI-X in virtio

Currently only vioif(4) uses the feature.

knakahara@ helped to migrate to pci_intr_alloc(9). Thanks!

Name the interrupt handler for intrctl

Remove a garbage blank line

Implement softint-based interrupt handling in if_vioif

Softint-based interrupt handling is considered as a future direction
of the (network) device driver architecture in NetBSD. pq3etsec of
ppc is already implemented based on the architecture (unlike pq3etsec,
this change doesn't include softint-based if_start). In this
architecture, a hardware interrupt handler just schedules a softint
and the softint performs actual interrupt processing. It reduces
processing in hardware interrupt context and allows Layer 2 network
stack (e.g., bridge, vlan and even bpf) run in softint context,
which makes it easy to implement fine-grain locking in the layer.

This is an experimental implementation of the architecture in if_viof.

virtio introduces a new flag VIRTIO_F_PCI_INTR_SOFTINT. If a driver
of virtio sets it to sc_flags of virtio_softc, virtio calls
softint_schedule in virtio_intr instead of directly calling the
interrupt handler of the driver.

When VIOIF_SOFTINT_INTR is on, vioif doesn't use the existing softint
(vioif_rx_softint) that is called from vioif_rx_vq_done. Because
vioif_rx_softint already runs in softint context and another softint
isn't needed. This change actually improves performance in some cases.

The feature is disabled by default and enabled when SOFTINT_INTR is
set somewhere (normally in a kernel configuration).

branches: 1.7.2;
use correct macro to handle subsys ids. Functional there is no
change, as both macros select the same bits.

branches: 1.6.2; 1.6.6;
Introduce VIRTIO_F_PCI_INTR_MPSAFE for virtio

It is set by a child driver, e.g., if_vioif. If set, virtio sets
PCI_INTR_MPSAFE for pci_intr_establish.

branches: 1.5.2;
make pci_intr_string and eisa_intr_string take a buffer and a length
instead of relying in local static storage.

Make sure to check if the driver has a valid intr handler in virtio_detach().

Fixes a panic during shutdown on KVM under ubuntu 13.04 with virtio,
as reported in PR kern/48105 by Richard Hansen.

Should be pulled up to netbsd-6 branches.

branches: 1.3.6; 1.3.8; 1.3.12; 1.3.14; 1.3.16; 1.3.22;
Small typo in symbolic name.

branches: 1.2.2;
Use symbolic PCI_PRODUCTs.

Import of the virtio driver written by MINOURA Makoto <minoura@netbsd.org>
with minor changes to make it compile an run on -current. This driver
speeds up disk and network access in virtual environments like KVM.

Enabled on i386 and amd64. Tested with a CentOS 5.7 x86_64 host.

See http://ozlabs.org/~rusty/virtio-spec/virtio.pdf for the specification.

virtio(4): change variable name, nfc

Fix not to allocate unnecessary descriptor

fixes PR/57358

Use enumeration for state of a child driver instead of flags
and check its detaching by using sc->sc_child in virtio_softc

pointed out by riastradh, thanks.
fixes PR/57357

Added flags to store status of attaching a virtio device

This prevents a panic on reboot after a virtio device had called
virtio_child_attach_failed().

Use descriptor chain for free slots instead of vq_entry list

Descriptors can be chained by themself. And descriptors added to
avail ring or used ring are already chained. But it was not used
for unused descriptors and another linked list structure named
vq_entry was used.
The chain is also used for unused descriptors to make virtio(4)
simpler.

virtio(4): Fix sizing of virtqueue allocation.

vq->vq_avail[0].ring is a zero-length array, and thus sizeof is zero;
likewise vq->vq_used[0].ring.

Use vq->vq_avail[0].ring[0] and vq->vq_used[0].ring[0] to fix this
and restore the previous allocation sizing logic.

XXX We shouldn't use zero-length arrays here -- they are asking for
trouble like this, and C99 has a standard way to express what we're
actually trying to get at it, flexible array members.

PR kern/57304

Reported-by: syzbot+7fb1047f5dfa33b26331@syzkaller.appspotmail.com

Use PRIuBUSSIZE to print bus_size_t variables.

virtio(4): Avoid name collision with global intrhand on sparc64.

Pacifies -Werror=shadow.

No functional change intended.

Setup virtqueues after registering them to virtio_softc

Added check of pointer for allocated memory before release of resource

Added functions to set interrupt handler and index into virtqueue

Set virtqueues in virtio_child_attach_finish

The number of virtqueue maybe change in a part of VirtIO devices
(e.g. vioif(4)). And it is fixed after negotiation of features.
So the configuration is moved into the function.

Subtly adjust criteria for notification of Virtio devices in
VIRTIO_F_RING_EVENT_IDX mode.

This fixes stalls in the vioif(4) transmit path that would happen sooner
or later, depending on interface transmit utilization, and were
particularly noticable with a NetBSD guest transmitting to its x86 KVM
host at multi-gigabit speeds.

Fix dmamap_syncs more; the number of ring elements is vq->vq_num, not
sc->sc_nvqs.

branches: 1.63.2;
Use PRIx64 in a debug print macro to print a uint64_t.

KNF

Fix previous by using roundup (and not howmany).

virtio(4): Use howmany from sys/param.h instead of open-coding it.

howmany will divide by VIRTIO_PAGE_SIZE instead of doing &
~(VIRTIO_PAGE_SIZE - 1), but it's a constant 4096 so this should make
no difference in the compiled output except possibly at -O0.

No functional change intended.

virtio(4): Sprinkle KNF.

No functional change intended.

virtio(4): Print numeric device type, even if unrecognized.

virtio(4): Membar and bus_dmamap_sync audit.

- Don't use membar_* for DMA.

- Sync only the header and payload of the rings separately as needed.
If we bounce, this avoids large memcpy when we only care about the
header.

- Sync uring with PREREAD before triggering anything that will return
data in it.

=> Move uring PREREAD in virtio_enqueue_commit to _before_ updating
vq->vq_avail->idx, which is the pointat which the DMA read is
triggered in the `device' (host).

=> Omit needless membar_consumer in virtio_enqueue_commit -- may not
work with DMA memory, and even if it does, redundant with
bus_dmamap_sync uring PREREAD here.

=> XXX Does the device/host ever return unsolicited entries in the
queue, or only solicited ones? If only solicited ones, the
PREREAD in virtio_init_vq is redundant.

- Sync uring with POSTREAD before we read from it. This way the DMA
read into our buffer has finished before we read from the buffer.

=> Add missing uring POSTREAD in virtio_vq_is_enqueued, between read of
vq->vq_used_idx and return to caller, so that the caller can
safely use virtio_dequeue.

=> Add missing uring POSTREADs in virtio_start_vq_intr:
. between entry from caller and the read of vq->vq_used_idx
. between the read of vq->vq_used_idx and return to caller,
so that the caller can safely use virtio_dequeue, just like
virtio_vq_is_enqueued

=> Move uring POSTREADs in virtio_enqueue_commit to _before_ reading
vq->vq_used->flags or *vq->vq_avail_event, not after.

- After we write to aring, sync it with PREWRITE. This way we finish
writing to our buffer before the DMA write from it.

=> Omit needless PREWRITE in virtio_init_vq -- we do the appropriate
PREWRITE in virtio_enqueue_commit now.

=> Convert membar_producer to bus_dmamap_sync PREWRITE in
virtio_enqueue_commit.

=> Omit incorrect aring POSTWRITE in virtio_enqueue_commit -- no need
because the DMA write may not have completed yet at this point,
and we already do a POSTWRITE in virtio_vq_is_enqueued.

=> Omit needless membar_producer in virtio_postpone_intr -- may not
work with DMA memory, and even if it does, redundant with
bus_dmamap_sync PREWRITE here.

- After xfers to aring have completed, sync it with POSTWRITE.

=> Add missing aring POSTWRITE in virtio_free_vq, in case there are
paths from virtio_enqueue_commit to here that don't go through
virtio_is_enqueued. (If there are no such paths, then maybe we
should KASSERT(vq->vq_queued == 0) in virtio_free_vq.)

virtio(4): Move comment for virtio_vq_intr.

No functional change intended.

fix typos in word "functions" in comments, mainly s/fuctions/functions/.

virtio: cosmetics - use (*pfn)(...) syntax.

Some people prefer to call function pointers without an explicit
dereference (that is purely cosmetic in this position), especially to
create faux c++ s->pfn(...) with function pointers in struct members.
Some prefer explicit dereference (that requires parens around it).

(pfn)(...) without dereference but with parens looks odd to both, so
make it conform to one of the established alternatives.

Same object code is generated.

virtio: stop reinit for safety when a device resetting is failed

virtio: fix missing release of interrupt resources

virtio: reconfigure MSI-X vector on re-initialization

This may fix a problem that some interrupts, for example
link-state change of vioif(4) are not notified.

divide setup routine of virtio interrupts
into establishment and device configuration

In virtio_alloc_vq change BUS_DMA_NOWAIT to BUS_DMA_WAITOK. It's always
called from attach routines.

Trailing whitespace

Better reading of 4.1.3.1, it seems that using 32 bit reads/writes is
mandatory for non-legacy devices.

Second round of cleaning up endian code. No more tailored code to maintain.

Rewrite and streamline virtio device config read/write and explicitly cater
for the Aarch64-eb bus problem with Qemu. This removes lots of bus_space
`magic' and cleans up the code.

Remove the virtio child driver finalisation check KASSERT that, while it
should never trigger, a possible bug in a child driver shouldn't have to panic
the kernel. Instead report the internal error.

Add VirtIO PCI v1.0 attachments and fix the drivers affected.

The vioif, ld, scsi, viornd and viomb devices were adjusted when needed and
tested both in legacy 0.9 and v1.0 attachments trough PCI on amd64, sparc64,
aarch64 and aarch64-eb. ACPI/FDT attachments also tested on
aarch64/aarch64-eb.

Known issues

* viomb on aarch64 works only with ACPI/FDT attachment but not with PCI
attachment. PCI and ACPI/FDT attachment works on aarch64-eb.

* virtio on sparc64 attaches but is it not functioning though not a
regression.

branches: 1.42.2;
Fix word-o of function name in comment

Set handlers implemented in child device of virtio(4) to virtqueue
instead of the commonized function

Register an interrupt handler for virtqueue to that for pci
to reduce function call

Remove VIRTIO_F_PCI_INTR_SOFTINT support for multiqueue

The functionality is no longer used by obsolating VIOIF_SOFTINT_INTR.

in many device attach paths, allocate memory with KM_SLEEP instead of KM_NOSLEEP
and remove code to handle failures that can no longer happen.

Add multiqueue support, virtio(4)

Suppress "not configured" message when no device is present

branches: 1.35.2;
Add cfprint_t virtiobusprint() and use it for virtio_mmio.

Fix queue address setup bug introduced in 1.31.

This use-before-initialization problem resulted in non-zero queues never
being initialized with their virtqueue pointer, and queue zero's pointer
to be clobbered with the virtqueue pointer for the higest-indexed ring.

remove now-irrelevant <dev/pci> includes

fix up virtio module for current structure; add virtio_pci module

Begin to detangle virtio from its PCI attachment

branches: 1.30.2;
Cosmetic - consistently use #define<SPACE>

Reorg the virtio(4) config(9) code to be explicitly pci specific.

We'll later use this for a non-pci virtio(4) usecase.

ok martin

branches: 1.28.2;
remove checks for failure after memory allocation calls that cannot fail:

kmem_alloc() with KM_SLEEP
kmem_zalloc() with KM_SLEEP
percpu_alloc()
pserialize_create()
psref_class_create()

all of these paths include an assertion that the allocation has not failed,
so callers should not assert that again.

Handle config change interrupts to inhibit sending packets while link down

PR kern/52103 by s-yamaguchi@IIJ

fix pasto for nbo_bus_space_write_4() - value needs to be uint32_t

Duh, remove accidently left over debug printfs

Backout previous, no need for these accessor in MD code

Provide an accessor for the pci attach args of a virtio device (if
available - future non-pci attachments may return NULL).
Sparc64 MD code needs this to match the OF node of the virtio device
for boot device recognition.

reorganize the attachment process for virtio child devices, so that
more common code is shared among the drivers, and it's possible for
the drivers to be correctly dynamically loaded; forbid direct access
to struct virtio_softc from the child driver code

use ipl level specified by child when attaching interrupts, instead of always
using IPL_NET

PR kern/52107: for now (while we still use the legacy virtio spec) at least
hack around the endianess issues with that.

branches: 1.19.2;
Fix the "typical usage" example for enqueuing a request - if
bus_dmamap_load(9) fails, we shouldn't call bus_dmamap_unload(9).

Modularize the ld driver and all of its attachments. Ensure that all
parents are capable of rescan (or otherwise provide a means of attaching
children post-initialization).

Prevent a panic during system shutdown when vioif(4) is used

branches: 1.16.2;
pci_intr_type() is required pci_chipset_tag_t argument by other than x86.

pointed out by nonaka@n.o.

KNF. Remove extra spaces. No functional change.

PR/50636: Ryo ONODERA: Fix incorrect kmem_free()

more device names

use dma64 if available so we don't have do bounce around too much.

Support MSI-X in virtio

Currently only vioif(4) uses the feature.

knakahara@ helped to migrate to pci_intr_alloc(9). Thanks!

Name the interrupt handler for intrctl

Remove a garbage blank line

Implement softint-based interrupt handling in if_vioif

Softint-based interrupt handling is considered as a future direction
of the (network) device driver architecture in NetBSD. pq3etsec of
ppc is already implemented based on the architecture (unlike pq3etsec,
this change doesn't include softint-based if_start). In this
architecture, a hardware interrupt handler just schedules a softint
and the softint performs actual interrupt processing. It reduces
processing in hardware interrupt context and allows Layer 2 network
stack (e.g., bridge, vlan and even bpf) run in softint context,
which makes it easy to implement fine-grain locking in the layer.

This is an experimental implementation of the architecture in if_viof.

virtio introduces a new flag VIRTIO_F_PCI_INTR_SOFTINT. If a driver
of virtio sets it to sc_flags of virtio_softc, virtio calls
softint_schedule in virtio_intr instead of directly calling the
interrupt handler of the driver.

When VIOIF_SOFTINT_INTR is on, vioif doesn't use the existing softint
(vioif_rx_softint) that is called from vioif_rx_vq_done. Because
vioif_rx_softint already runs in softint context and another softint
isn't needed. This change actually improves performance in some cases.

The feature is disabled by default and enabled when SOFTINT_INTR is
set somewhere (normally in a kernel configuration).

branches: 1.7.2;
use correct macro to handle subsys ids. Functional there is no
change, as both macros select the same bits.

branches: 1.6.2; 1.6.6;
Introduce VIRTIO_F_PCI_INTR_MPSAFE for virtio

It is set by a child driver, e.g., if_vioif. If set, virtio sets
PCI_INTR_MPSAFE for pci_intr_establish.

branches: 1.5.2;
make pci_intr_string and eisa_intr_string take a buffer and a length
instead of relying in local static storage.

Make sure to check if the driver has a valid intr handler in virtio_detach().

Fixes a panic during shutdown on KVM under ubuntu 13.04 with virtio,
as reported in PR kern/48105 by Richard Hansen.

Should be pulled up to netbsd-6 branches.

branches: 1.3.6; 1.3.8; 1.3.12; 1.3.14; 1.3.16; 1.3.22;
Small typo in symbolic name.

branches: 1.2.2;
Use symbolic PCI_PRODUCTs.

Import of the virtio driver written by MINOURA Makoto <minoura@netbsd.org>
with minor changes to make it compile an run on -current. This driver
speeds up disk and network access in virtual environments like KVM.

Enabled on i386 and amd64. Tested with a CentOS 5.7 x86_64 host.

See http://ozlabs.org/~rusty/virtio-spec/virtio.pdf for the specification.

Subtly adjust criteria for notification of Virtio devices in
VIRTIO_F_RING_EVENT_IDX mode.

This fixes stalls in the vioif(4) transmit path that would happen sooner
or later, depending on interface transmit utilization, and were
particularly noticable with a NetBSD guest transmitting to its x86 KVM
host at multi-gigabit speeds.

Fix dmamap_syncs more; the number of ring elements is vq->vq_num, not
sc->sc_nvqs.

Use PRIx64 in a debug print macro to print a uint64_t.

KNF

Fix previous by using roundup (and not howmany).

virtio(4): Use howmany from sys/param.h instead of open-coding it.

howmany will divide by VIRTIO_PAGE_SIZE instead of doing &
~(VIRTIO_PAGE_SIZE - 1), but it's a constant 4096 so this should make
no difference in the compiled output except possibly at -O0.

No functional change intended.

virtio(4): Sprinkle KNF.

No functional change intended.

virtio(4): Print numeric device type, even if unrecognized.

virtio(4): Membar and bus_dmamap_sync audit.

- Don't use membar_* for DMA.

- Sync only the header and payload of the rings separately as needed.
If we bounce, this avoids large memcpy when we only care about the
header.

- Sync uring with PREREAD before triggering anything that will return
data in it.

=> Move uring PREREAD in virtio_enqueue_commit to _before_ updating
vq->vq_avail->idx, which is the pointat which the DMA read is
triggered in the `device' (host).

=> Omit needless membar_consumer in virtio_enqueue_commit -- may not
work with DMA memory, and even if it does, redundant with
bus_dmamap_sync uring PREREAD here.

=> XXX Does the device/host ever return unsolicited entries in the
queue, or only solicited ones? If only solicited ones, the
PREREAD in virtio_init_vq is redundant.

- Sync uring with POSTREAD before we read from it. This way the DMA
read into our buffer has finished before we read from the buffer.

=> Add missing uring POSTREAD in virtio_vq_is_enqueued, between read of
vq->vq_used_idx and return to caller, so that the caller can
safely use virtio_dequeue.

=> Add missing uring POSTREADs in virtio_start_vq_intr:
. between entry from caller and the read of vq->vq_used_idx
. between the read of vq->vq_used_idx and return to caller,
so that the caller can safely use virtio_dequeue, just like
virtio_vq_is_enqueued

=> Move uring POSTREADs in virtio_enqueue_commit to _before_ reading
vq->vq_used->flags or *vq->vq_avail_event, not after.

- After we write to aring, sync it with PREWRITE. This way we finish
writing to our buffer before the DMA write from it.

=> Omit needless PREWRITE in virtio_init_vq -- we do the appropriate
PREWRITE in virtio_enqueue_commit now.

=> Convert membar_producer to bus_dmamap_sync PREWRITE in
virtio_enqueue_commit.

=> Omit incorrect aring POSTWRITE in virtio_enqueue_commit -- no need
because the DMA write may not have completed yet at this point,
and we already do a POSTWRITE in virtio_vq_is_enqueued.

=> Omit needless membar_producer in virtio_postpone_intr -- may not
work with DMA memory, and even if it does, redundant with
bus_dmamap_sync PREWRITE here.

- After xfers to aring have completed, sync it with POSTWRITE.

=> Add missing aring POSTWRITE in virtio_free_vq, in case there are
paths from virtio_enqueue_commit to here that don't go through
virtio_is_enqueued. (If there are no such paths, then maybe we
should KASSERT(vq->vq_queued == 0) in virtio_free_vq.)

virtio(4): Move comment for virtio_vq_intr.

No functional change intended.

fix typos in word "functions" in comments, mainly s/fuctions/functions/.

virtio: cosmetics - use (*pfn)(...) syntax.

Some people prefer to call function pointers without an explicit
dereference (that is purely cosmetic in this position), especially to
create faux c++ s->pfn(...) with function pointers in struct members.
Some prefer explicit dereference (that requires parens around it).

(pfn)(...) without dereference but with parens looks odd to both, so
make it conform to one of the established alternatives.

Same object code is generated.

virtio: stop reinit for safety when a device resetting is failed

virtio: fix missing release of interrupt resources

virtio: reconfigure MSI-X vector on re-initialization

This may fix a problem that some interrupts, for example
link-state change of vioif(4) are not notified.

divide setup routine of virtio interrupts
into establishment and device configuration

In virtio_alloc_vq change BUS_DMA_NOWAIT to BUS_DMA_WAITOK. It's always
called from attach routines.

Trailing whitespace

Better reading of 4.1.3.1, it seems that using 32 bit reads/writes is
mandatory for non-legacy devices.

Second round of cleaning up endian code. No more tailored code to maintain.

Rewrite and streamline virtio device config read/write and explicitly cater
for the Aarch64-eb bus problem with Qemu. This removes lots of bus_space
`magic' and cleans up the code.

Remove the virtio child driver finalisation check KASSERT that, while it
should never trigger, a possible bug in a child driver shouldn't have to panic
the kernel. Instead report the internal error.

Add VirtIO PCI v1.0 attachments and fix the drivers affected.

The vioif, ld, scsi, viornd and viomb devices were adjusted when needed and
tested both in legacy 0.9 and v1.0 attachments trough PCI on amd64, sparc64,
aarch64 and aarch64-eb. ACPI/FDT attachments also tested on
aarch64/aarch64-eb.

Known issues

* viomb on aarch64 works only with ACPI/FDT attachment but not with PCI
attachment. PCI and ACPI/FDT attachment works on aarch64-eb.

* virtio on sparc64 attaches but is it not functioning though not a
regression.

branches: 1.42.2;
Fix word-o of function name in comment

Set handlers implemented in child device of virtio(4) to virtqueue
instead of the commonized function

Register an interrupt handler for virtqueue to that for pci
to reduce function call

Remove VIRTIO_F_PCI_INTR_SOFTINT support for multiqueue

The functionality is no longer used by obsolating VIOIF_SOFTINT_INTR.

in many device attach paths, allocate memory with KM_SLEEP instead of KM_NOSLEEP
and remove code to handle failures that can no longer happen.

Add multiqueue support, virtio(4)

Suppress "not configured" message when no device is present

branches: 1.35.2;
Add cfprint_t virtiobusprint() and use it for virtio_mmio.

Fix queue address setup bug introduced in 1.31.

This use-before-initialization problem resulted in non-zero queues never
being initialized with their virtqueue pointer, and queue zero's pointer
to be clobbered with the virtqueue pointer for the higest-indexed ring.

remove now-irrelevant <dev/pci> includes

fix up virtio module for current structure; add virtio_pci module

Begin to detangle virtio from its PCI attachment

branches: 1.30.2;
Cosmetic - consistently use #define<SPACE>

Reorg the virtio(4) config(9) code to be explicitly pci specific.

We'll later use this for a non-pci virtio(4) usecase.

ok martin

branches: 1.28.2;
remove checks for failure after memory allocation calls that cannot fail:

kmem_alloc() with KM_SLEEP
kmem_zalloc() with KM_SLEEP
percpu_alloc()
pserialize_create()
psref_class_create()

all of these paths include an assertion that the allocation has not failed,
so callers should not assert that again.

Handle config change interrupts to inhibit sending packets while link down

PR kern/52103 by s-yamaguchi@IIJ

fix pasto for nbo_bus_space_write_4() - value needs to be uint32_t

Duh, remove accidently left over debug printfs

Backout previous, no need for these accessor in MD code

Provide an accessor for the pci attach args of a virtio device (if
available - future non-pci attachments may return NULL).
Sparc64 MD code needs this to match the OF node of the virtio device
for boot device recognition.

reorganize the attachment process for virtio child devices, so that
more common code is shared among the drivers, and it's possible for
the drivers to be correctly dynamically loaded; forbid direct access
to struct virtio_softc from the child driver code

use ipl level specified by child when attaching interrupts, instead of always
using IPL_NET

PR kern/52107: for now (while we still use the legacy virtio spec) at least
hack around the endianess issues with that.

branches: 1.19.2;
Fix the "typical usage" example for enqueuing a request - if
bus_dmamap_load(9) fails, we shouldn't call bus_dmamap_unload(9).

Modularize the ld driver and all of its attachments. Ensure that all
parents are capable of rescan (or otherwise provide a means of attaching
children post-initialization).

Prevent a panic during system shutdown when vioif(4) is used

branches: 1.16.2;
pci_intr_type() is required pci_chipset_tag_t argument by other than x86.

pointed out by nonaka@n.o.

KNF. Remove extra spaces. No functional change.

PR/50636: Ryo ONODERA: Fix incorrect kmem_free()

more device names

use dma64 if available so we don't have do bounce around too much.

Support MSI-X in virtio

Currently only vioif(4) uses the feature.

knakahara@ helped to migrate to pci_intr_alloc(9). Thanks!

Name the interrupt handler for intrctl

Remove a garbage blank line

Implement softint-based interrupt handling in if_vioif

Softint-based interrupt handling is considered as a future direction
of the (network) device driver architecture in NetBSD. pq3etsec of
ppc is already implemented based on the architecture (unlike pq3etsec,
this change doesn't include softint-based if_start). In this
architecture, a hardware interrupt handler just schedules a softint
and the softint performs actual interrupt processing. It reduces
processing in hardware interrupt context and allows Layer 2 network
stack (e.g., bridge, vlan and even bpf) run in softint context,
which makes it easy to implement fine-grain locking in the layer.

This is an experimental implementation of the architecture in if_viof.

virtio introduces a new flag VIRTIO_F_PCI_INTR_SOFTINT. If a driver
of virtio sets it to sc_flags of virtio_softc, virtio calls
softint_schedule in virtio_intr instead of directly calling the
interrupt handler of the driver.

When VIOIF_SOFTINT_INTR is on, vioif doesn't use the existing softint
(vioif_rx_softint) that is called from vioif_rx_vq_done. Because
vioif_rx_softint already runs in softint context and another softint
isn't needed. This change actually improves performance in some cases.

The feature is disabled by default and enabled when SOFTINT_INTR is
set somewhere (normally in a kernel configuration).

branches: 1.7.2;
use correct macro to handle subsys ids. Functional there is no
change, as both macros select the same bits.

branches: 1.6.2; 1.6.6;
Introduce VIRTIO_F_PCI_INTR_MPSAFE for virtio

It is set by a child driver, e.g., if_vioif. If set, virtio sets
PCI_INTR_MPSAFE for pci_intr_establish.

branches: 1.5.2;
make pci_intr_string and eisa_intr_string take a buffer and a length
instead of relying in local static storage.

Make sure to check if the driver has a valid intr handler in virtio_detach().

Fixes a panic during shutdown on KVM under ubuntu 13.04 with virtio,
as reported in PR kern/48105 by Richard Hansen.

Should be pulled up to netbsd-6 branches.

branches: 1.3.6; 1.3.8; 1.3.12; 1.3.14; 1.3.16; 1.3.22;
Small typo in symbolic name.

branches: 1.2.2;
Use symbolic PCI_PRODUCTs.

Import of the virtio driver written by MINOURA Makoto <minoura@netbsd.org>
with minor changes to make it compile an run on -current. This driver
speeds up disk and network access in virtual environments like KVM.

Enabled on i386 and amd64. Tested with a CentOS 5.7 x86_64 host.

See http://ozlabs.org/~rusty/virtio-spec/virtio.pdf for the specification.

Fix dmamap_syncs more; the number of ring elements is vq->vq_num, not
sc->sc_nvqs.

Use PRIx64 in a debug print macro to print a uint64_t.

KNF

Fix previous by using roundup (and not howmany).

virtio(4): Use howmany from sys/param.h instead of open-coding it.

howmany will divide by VIRTIO_PAGE_SIZE instead of doing &
~(VIRTIO_PAGE_SIZE - 1), but it's a constant 4096 so this should make
no difference in the compiled output except possibly at -O0.

No functional change intended.

virtio(4): Sprinkle KNF.

No functional change intended.

virtio(4): Print numeric device type, even if unrecognized.

virtio(4): Membar and bus_dmamap_sync audit.

- Don't use membar_* for DMA.

- Sync only the header and payload of the rings separately as needed.
If we bounce, this avoids large memcpy when we only care about the
header.

- Sync uring with PREREAD before triggering anything that will return
data in it.

=> Move uring PREREAD in virtio_enqueue_commit to _before_ updating
vq->vq_avail->idx, which is the pointat which the DMA read is
triggered in the `device' (host).

=> Omit needless membar_consumer in virtio_enqueue_commit -- may not
work with DMA memory, and even if it does, redundant with
bus_dmamap_sync uring PREREAD here.

=> XXX Does the device/host ever return unsolicited entries in the
queue, or only solicited ones? If only solicited ones, the
PREREAD in virtio_init_vq is redundant.

- Sync uring with POSTREAD before we read from it. This way the DMA
read into our buffer has finished before we read from the buffer.

=> Add missing uring POSTREAD in virtio_vq_is_enqueued, between read of
vq->vq_used_idx and return to caller, so that the caller can
safely use virtio_dequeue.

=> Add missing uring POSTREADs in virtio_start_vq_intr:
. between entry from caller and the read of vq->vq_used_idx
. between the read of vq->vq_used_idx and return to caller,
so that the caller can safely use virtio_dequeue, just like
virtio_vq_is_enqueued

=> Move uring POSTREADs in virtio_enqueue_commit to _before_ reading
vq->vq_used->flags or *vq->vq_avail_event, not after.

- After we write to aring, sync it with PREWRITE. This way we finish
writing to our buffer before the DMA write from it.

=> Omit needless PREWRITE in virtio_init_vq -- we do the appropriate
PREWRITE in virtio_enqueue_commit now.

=> Convert membar_producer to bus_dmamap_sync PREWRITE in
virtio_enqueue_commit.

=> Omit incorrect aring POSTWRITE in virtio_enqueue_commit -- no need
because the DMA write may not have completed yet at this point,
and we already do a POSTWRITE in virtio_vq_is_enqueued.

=> Omit needless membar_producer in virtio_postpone_intr -- may not
work with DMA memory, and even if it does, redundant with
bus_dmamap_sync PREWRITE here.

- After xfers to aring have completed, sync it with POSTWRITE.

=> Add missing aring POSTWRITE in virtio_free_vq, in case there are
paths from virtio_enqueue_commit to here that don't go through
virtio_is_enqueued. (If there are no such paths, then maybe we
should KASSERT(vq->vq_queued == 0) in virtio_free_vq.)

virtio(4): Move comment for virtio_vq_intr.

No functional change intended.

fix typos in word "functions" in comments, mainly s/fuctions/functions/.

virtio: cosmetics - use (*pfn)(...) syntax.

Some people prefer to call function pointers without an explicit
dereference (that is purely cosmetic in this position), especially to
create faux c++ s->pfn(...) with function pointers in struct members.
Some prefer explicit dereference (that requires parens around it).

(pfn)(...) without dereference but with parens looks odd to both, so
make it conform to one of the established alternatives.

Same object code is generated.

virtio: stop reinit for safety when a device resetting is failed

virtio: fix missing release of interrupt resources

virtio: reconfigure MSI-X vector on re-initialization

This may fix a problem that some interrupts, for example
link-state change of vioif(4) are not notified.

divide setup routine of virtio interrupts
into establishment and device configuration

In virtio_alloc_vq change BUS_DMA_NOWAIT to BUS_DMA_WAITOK. It's always
called from attach routines.

Trailing whitespace

Better reading of 4.1.3.1, it seems that using 32 bit reads/writes is
mandatory for non-legacy devices.

Second round of cleaning up endian code. No more tailored code to maintain.

Rewrite and streamline virtio device config read/write and explicitly cater
for the Aarch64-eb bus problem with Qemu. This removes lots of bus_space
`magic' and cleans up the code.

Remove the virtio child driver finalisation check KASSERT that, while it
should never trigger, a possible bug in a child driver shouldn't have to panic
the kernel. Instead report the internal error.

Add VirtIO PCI v1.0 attachments and fix the drivers affected.

The vioif, ld, scsi, viornd and viomb devices were adjusted when needed and
tested both in legacy 0.9 and v1.0 attachments trough PCI on amd64, sparc64,
aarch64 and aarch64-eb. ACPI/FDT attachments also tested on
aarch64/aarch64-eb.

Known issues

* viomb on aarch64 works only with ACPI/FDT attachment but not with PCI
attachment. PCI and ACPI/FDT attachment works on aarch64-eb.

* virtio on sparc64 attaches but is it not functioning though not a
regression.

branches: 1.42.2;
Fix word-o of function name in comment