MFC r326645 (by imp): Define xpt_path_inq.

This provides a nice wrarpper around the XPT_PATH_INQ ccb creation and
calling.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Convert sys/cam to use make_dev_s().

Reviewed by: hps, jhb
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks
Differential revision: https://reviews.freebsd.org/D4746

Attach pass driver to LUNs is OFFLINE state.

Previously such LUNs were silently ignored. But while they indeed unable
to process most of SCSI commands, some, like RTPG, they still can.

MFC after: 1 month

Merge CAM locking changes from the projects/camlock branch to radically
reduce lock congestion and improve SMP scalability of the SCSI/ATA stack,
preparing the ground for the coming next GEOM direct dispatch support.

Replace big per-SIM locks with bunch of smaller ones:
- per-LUN locks to protect device and peripheral drivers state;
- per-target locks to protect list of LUNs on target;
- per-bus locks to protect reference counting;
- per-send queue locks to protect queue of CCBs to be sent;
- per-done queue locks to protect queue of completed CCBs;
- remaining per-SIM locks now protect only HBA driver internals.

While holding LUN lock it is allowed (while not recommended for performance
reasons) to take SIM lock. The opposite acquisition order is forbidden.
All the other locks are leaf locks, that can be taken anywhere, but should
not be cascaded. Many functions, such as: xpt_action(), xpt_done(),
xpt_async(), xpt_create_path(), etc. are no longer require (but allow) SIM
lock to be held.

To keep compatibility and solve cases where SIM lock can't be dropped, all
xpt_async() calls in addition to xpt_done() calls are queued to completion
threads for async processing in clean environment without SIM lock held.

Instead of single CAM SWI thread, used for commands completion processing
before, use multiple (depending on number of CPUs) threads. Load balanced
between them using "hash" of the device B:T:L address.

HBA drivers that can drop SIM lock during completion processing and have
sufficient number of completion threads to efficiently scale to multiple
CPUs can use new function xpt_done_direct() to avoid extra context switch.
Make ahci(4) driver to use this mechanism depending on hardware setup.

Sponsored by: iXsystems, Inc.
MFC after: 2 months

Unify periph invalidation and destruction reporting.
Print message containing device model and serial number on invalidation.

Requested by: glebius
MFC after: 1 week

Fxi a bunch of typos.

PR: misc/174625
Submitted by: Jeremy Chadwick <jdc@koitsu.org>

Remove two more 'periph == NULL' checks missed in r241404.
This condition can never be true as functions are called from single place
and the checks just pollute the code and confuse Clang Static Analyzer.

Tune and add some missing CAM_DEBUG() points for better consistency.

Work around a race condition in devfs by changing the way closes
are handled in most CAM peripheral drivers that are not handled by
GEOM's disk class.

The usual character driver open and close semantics are that the
driver gets N open calls, but only one close, when the last caller
closes the device.

CAM peripheral drivers expect that behavior to be honored to the
letter, and the CAM peripheral driver code (specifically
cam_periph_release_locked_busses()) panics if it is done incorrectly.

Since devfs has to drop its locks while it calls a driver's close
routine, and it does not have a way to delay or prevent open calls
while it is calling the close routine, there is a race.

The sequence of events, simplified a bit, is:

- devfs acquires a lock
- devfs checks the reference count, and if it is 1, continues to close.
- devfs releases the lock

- 2nd process open call on the device happens here

- devfs calls the driver's close routine

- devfs acquires a lock
- devfs decrements the reference count
- devfs releases the lock

- 2nd process close call on the device happens here

At the second close, we get a panic in
cam_periph_release_locked_busses(), complaining that peripheral
has been released when the reference count is already 0. This is
because we have gotten two closes in a row, which should not
happen.

The fix is to add the D_TRACKCLOSE flag to the driver's cdevsw, so
that we get a close() call for each open(). That does happen
reliably, so we can make sure that our reference counts are
correct.

Note that the sa(4) and pt(4) drivers only allow one context
through the open routine. So these drivers aren't exposed to the
same race condition.

scsi_ch.c,
scsi_enc.c,
scsi_enc_internal.h,
scsi_pass.c,
scsi_sg.c:
For these drivers, change the open() routine to
increment the reference count for every open, and
just decrement the reference count in the close.

Call cam_periph_release_locked() in some scenarios
to avoid additional lock and unlock calls.

scsi_pt.c: Call cam_periph_release_locked() in some scenarios
to avoid additional lock and unlock calls.

MFC after: 3 days

Make CAM report devices with ATA/SATA transport to devstat(9) as IDE.

Replace most of priority numbers with defines. No logical changes.

Separate the parallel scsi knowledge out of the core of the XPT, and
modularize it so that new transports can be created.

Add a transport for SATA

Add a periph+protocol layer for ATA

Add a driver for AHCI-compliant hardware.

Add a maxio field to CAM so that drivers can advertise their max
I/O capability. Modify various drivers so that they are insulated
from the value of MAXPHYS.

The new ATA/SATA code supports AHCI-compliant hardware, and will override
the classic ATA driver if it is loaded as a module at boot time or compiled
into the kernel. The stack now support NCQ (tagged queueing) for increased
performance on modern SATA drives. It also supports port multipliers.

ATA drives are accessed via 'ada' device nodes. ATAPI drives are
accessed via 'cd' device nodes. They can all be enumerated and manipulated
via camcontrol, just like SCSI drives. SCSI commands are not translated to
their ATA equivalents; ATA native commands are used throughout the entire
stack, including camcontrol. See the camcontrol manpage for further
details. Testing this code may require that you update your fstab, and
possibly modify your BIOS to enable AHCI functionality, if available.

This code is very experimental at the moment. The userland ABI/API has
changed, so applications will need to be recompiled. It may change
further in the near future. The 'ada' device name may also change as
more infrastructure is completed in this project. The goal is to
eventually put all CAM busses and devices until newbus, allowing for
interesting topology and management options.

Few functional changes will be seen with existing SCSI/SAS/FC drivers,
though the userland ABI has still changed. In the future, transports
specific modules for SAS and FC may appear in order to better support
the topologies and capabilities of these technologies.

The modularization of CAM and the addition of the ATA/SATA modules is
meant to break CAM out of the mold of being specific to SCSI, letting it
grow to be a framework for arbitrary transports and protocols. It also
allows drivers to be written to support discrete hardware without
jeopardizing the stability of non-related hardware. While only an AHCI
driver is provided now, a Silicon Image driver is also in the works.
Drivers for ICH1-4, ICH5-6, PIIX, classic IDE, and any other hardware
is possible and encouraged. Help with new transports is also encouraged.

Submitted by: scottl, mav
Approved by: re

Don't call destroy_dev(9) with a mutex held. While here, shuffle
things around so the periph destructors look alike. Based on a patch
by Jaakko Heinonen.

Submitted by: Jaakko Heinonen
Reviewed by: scottl
Approved by: rwatson (mentor)
Sponsored by: FreeBSD Foundation

Add a helper function for registering async callbacks. Besides
eliminating a lot of duplicated code, this also fixes a locking edge case.

Avoid problems with make_dev.

Remove Giant from CAM. Drivers (SIMs) now register a mutex that CAM will
use to synchornize and protect all data objects that are used for that
SIM. Drivers that are not yet MPSAFE register Giant and operate as
usual. RIght now, no drivers are MPSAFE, though a few will be changed
in the coming week as this work settles down.

The driver API has changed, so all CAM drivers will need to be recompiled.
The userland API has not changed, so tools like camcontrol do not need to
be recompiled.

Add an xpt_print function to reduce most of the xpt_print_path/printf
pairs. Convert the core code to use it.

Reviewed by: scsi (various folks weighing in with good advice)
MFC after: 2 weeks

Start each of the license/copyright comments with /*-

Do the dreaded s/dev_t/struct cdev */
Bump __FreeBSD_version accordingly.

Device megapatch 4/6:

Introduce d_version field in struct cdevsw, this must always be
initialized to D_VERSION.

Flip sense of D_NOGIANT flag to D_NEEDGIANT, this involves removing
four D_NOGIANT flags and adding 145 D_NEEDGIANT flags.

Device megapatch 1/6:

Free approx 86 major numbers with a mostly automatically generated patch.

A number of strategic drivers have been left behind by caution, and a few
because they still (ab)use their major number.

Use __FBSDID().

Add /* FALLTHROUGH */
Move /* FALLTHROUGH */ to correct location.
Remove unused variable(s).

Found by: FlexeLint

Use bioq_flush() to drain a bio queue with a specific error code.
Retain the mistake of not updating the devstat API for now.

Spell bioq_disksort() consistently with the remaining bioq_*().

#include <geom/geom_disk.h> where this is more appropriate.

Call devstat_start_transaction_bio() instead of devstat_start_transaction()

Allocate the devstat structure with devstat_new_entry().

Gigacommit to improve device-driver source compatibility between
branches:

Initialize struct cdevsw using C99 sparse initializtion and remove
all initializations to default values.

This patch is automatically generated and has been tested by compiling
LINT with all the fields in struct cdevsw in reverse order on alpha,
sparc64 and i386.

Approved by: re(scottl)

Remove usage of cam_extend.c, replace with dev->si_drv1

PR: kern/39809
Approved by: gibbs

KSE Milestone 2
Note ALL MODULES MUST BE RECOMPILED
make the kernel aware that there are smaller units of scheduling than the
process. (but only allow one thread per process at this time).
This is functionally equivalent to teh previousl -current except
that there is a thread associated with each process.

Sorry john! (your next MFC will be a doosie!)

Reviewed by: peter@freebsd.org, dillon@freebsd.org

X-MFC after: ha ha ha ha

Check the void * argument in the AC_FOUND_DEV case against NULL. Whether
correctly or not, this sometimes is propagated up via XPT.

Exploit recent improvements in the disk minilayer to simplify error
handling a bit.

Dogmatic lingupurists can celebrate that a number of gotos got removed.

Reviewed by: mjacob, ken

Actually biofinish(struct bio *, struct devstat *, int error) is more general
than the bioerror().

Most of this patch is generated by scripts.

Fix up unit attention and selection timeout handling in various peripheral
drivers.

- change daprevent() to set CAM_RETRY_SELTO and SF_RETRY_UA when it calls
cam_periph_runccb().
- change the pt(4) driver to ignore unit attentions
- change the targ(4) driver to retry selection timeouts
- clean up a few formatting glitches in the targ(4) driver

Reviewed by: gibbs

Rewrite of the CAM error recovery code.

Some of the major changes include:

- The SCSI error handling portion of cam_periph_error() has
been broken out into a number of subfunctions to better
modularize the code that handles the hierarchy of SCSI errors.
As a result, the code is now much easier to read.

- String handling and error printing has been significantly
revamped. We now use sbufs to do string formatting instead
of using printfs (for the kernel) and snprintf/strncat (for
userland) as before.

There is a new catchall error printing routine,
cam_error_print() and its string-based counterpart,
cam_error_string() that allow the kernel and userland
applications to pass in a CCB and have errors printed out
properly, whether or not they're SCSI errors. Among other
things, this helped eliminate a fair amount of duplicate code
in camcontrol.

We now print out more information than before, including
the CAM status and SCSI status and the error recovery action
taken to remedy the problem.

- sbufs are now available in userland, via libsbuf. This
change was necessary since most of the error printing code
is shared between libcam and the kernel.

- A new transfer settings interface is included in this checkin.
This code is #ifdef'ed out, and is primarily intended to aid
discussion with HBA driver authors on the final form the
interface should take. There is example code in the ahc(4)
driver that implements the HBA driver side of the new
interface. The new transfer settings code won't be enabled
until we're ready to switch all HBA drivers over to the new
interface.

src/Makefile.inc1,
lib/Makefile: Add libsbuf. It must be built before libcam,
since libcam uses sbuf routines.

libcam/Makefile: libcam now depends on libsbuf.

libsbuf/Makefile: Add a makefile for libsbuf. This pulls in the
sbuf sources from sys/kern.

bsd.libnames.mk: Add LIBSBUF.

camcontrol/Makefile: Add -lsbuf. Since camcontrol is statically
linked, we can't depend on the dynamic linker
to pull in libsbuf.

camcontrol.c: Use cam_error_print() instead of checking for
CAM_SCSI_STATUS_ERROR on every failed CCB.

sbuf.9: Change the prototypes for sbuf_cat() and
sbuf_cpy() so that the source string is now a
const char *. This is more in line wth the
standard system string functions, and helps
eliminate warnings when dealing with a const
source buffer.

Fix a typo.

cam.c: Add description strings for the various CAM
error status values, as well as routines to
look up those strings.

Add new cam_error_string() and
cam_error_print() routines for userland and
the kernel.

cam.h: Add a new CAM flag, CAM_RETRY_SELTO.

Add enumerated types for the various options
available with cam_error_print() and
cam_error_string().

cam_ccb.h: Add new transfer negotiation structures/types.

Change inq_len in the ccb_getdev structure to
be "reserved". This field has never been
filled in, and will be removed when we next
bump the CAM version.

cam_debug.h: Fix typo.

cam_periph.c: Modularize cam_periph_error(). The SCSI error
handling part of cam_periph_error() is now
in camperiphscsistatuserror() and
camperiphscsisenseerror().

In cam_periph_lock(), increase the reference
count on the periph while we wait for our lock
attempt to succeed so that the periph won't go
away while we're sleeping.

cam_xpt.c: Add new transfer negotiation code. (ifdefed
out)

Add a new function, xpt_path_string(). This
is a string/sbuf analog to xpt_print_path().

scsi_all.c: Revamp string handing and error printing code.
We now use sbufs for much of the string
formatting code. More of that code is shared
between userland the kernel.

scsi_all.h: Get rid of SS_TURSTART, it wasn't terribly
useful in the first place.

Add a new error action, SS_REQSENSE. (Send a
request sense and then retry the command.)
This is useful when the controller hasn't
performed autosense for some reason.

Change the default actions around a bit.

scsi_cd.c,
scsi_da.c,
scsi_pt.c,
scsi_ses.c: SF_RETRY_SELTO -> CAM_RETRY_SELTO. Selection
timeouts shouldn't be covered by a sense flag.

scsi_pass.[ch]: SF_RETRY_SELTO -> CAM_RETRY_SELTO.

Get rid of the last vestiges of a read/write
interface.

libkern/bsearch.c,
sys/libkern.h,
conf/files: Add bsearch.c, which is needed for some of the
new table lookup routines.

aic7xxx_freebsd.c: Define AHC_NEW_TRAN_SETTINGS if
CAM_NEW_TRAN_CODE is defined.

sbuf.h,
subr_sbuf.c: Add the appropriate #ifdefs so sbufs can
compile and run in userland.

Change sbuf_printf() to use vsnprintf()
instead of kvprintf(), which is only available
in the kernel.

Change the source string for sbuf_cpy() and
sbuf_cat() to be a const char *.

Add __BEGIN_DECLS and __END_DECLS around
function prototypes since they're now exported
to userland.

kdump/mkioctls: Include stdio.h before cam.h since cam.h now
includes a function with a FILE * argument.

Submitted by: gibbs (mostly)
Reviewed by: jdp, marcel (libsbuf makefile changes)
Reviewed by: des (sbuf changes)
Reviewed by: ken

Send the remains (such as I have located) of "block major numbers" to
the bit-bucket.

Change the peripheral driver list from a linker set to module driven
driver registration. This should allow things like da, sa, cd etc to be
in seperate KLD's to the cam core and make them preloadable.

Mechanical change to use <sys/queue.h> macro API instead of
fondling implementation details.

Created with: sed(1)
Reviewed by: md5(1)

Back out the previous change to the queue(3) interface.
It was not discussed and should probably not happen.

Requested by: msmith and others

Change the way that the queue(3) structures are declared; don't assume that
the type argument to *_HEAD and *_ENTRY is a struct.

Suggested by: phk
Reviewed by: phk
Approved by: mdodd

Separate the struct bio related stuff out of <sys/buf.h> into
<sys/bio.h>.

<sys/bio.h> is now a prerequisite for <sys/buf.h> but it shall
not be made a nested include according to bdes teachings on the
subject of nested includes.

Diskdrivers and similar stuff below specfs::strategy() should no
longer need to include <sys/buf.> unless they need caching of data.

Still a few bogus uses of struct buf to track down.

Repocopy by: peter

Complete the bio/buf divorce for all code below devfs::strategy

Exceptions:
Vinum untouched. This means that it cannot be compiled.
Greg Lehey is on the case.

CCD not converted yet, casts to struct buf (still safe)

atapi-cd casts to struct buf to examine B_PHYS

Move B_ERROR flag to b_ioflags and call it BIO_ERROR.

(Much of this done by script)

Move B_ORDERED flag to b_ioflags and call it BIO_ORDERED.

Move b_pblkno and b_iodone_chain to struct bio while we transition, they
will be obsoleted once bio structs chain/stack.

Add bio_queue field for struct bio aware disksort.

Address a lot of stylistic issues brought up by bde.

Remove B_READ, B_WRITE and B_FREEBUF and replace them with a new
field in struct buf: b_iocmd. The b_iocmd is enforced to have
exactly one bit set.

B_WRITE was bogusly defined as zero giving rise to obvious coding
mistakes.

Also eliminate the redundant struct buf flag B_CALL, it can just
as efficiently be done by comparing b_iodone to NULL.

Should you get a panic or drop into the debugger, complaining about
"b_iocmd", don't continue. It is likely to write on your disk
where it should have been reading.

This change is a step in the direction towards a stackable BIO capability.

A lot of this patch were machine generated (Thanks to style(9) compliance!)

Vinum users: Greg has not had time to test this yet, be careful.

Do the minor changes needed because of change to ccb_getdev structure.
JKH Trading Stamps applied.

Reviewed by: gibbs@freebsd.org, ken@freebsd.org

Convert the xpt, pass, pt and target drivers to use the new
make_dev()/destroy_dev() interface.

Introduce the disk mini-layer and devstat_end_transaction_buf() in cam/scsi.

Somewhat reviewed by: ken

Remove five now unused fields from struct cdevsw. They should never
have been there in the first place. A GENERIC kernel shrinks almost 1k.

Add a slightly different safetybelt under nostop for tty drivers.

Add some missing FreeBSD tags

$Id$ -> $FreeBSD$

Fix short timeout problems with the pt(4) driver:

- increase the default timeout from 10 seconds to 60 seconds
- add a new kernel option, SCSI_PT_DEFAULT_TIMEOUT, that lets users specify
the default timeout for the pt driver to use
- add two new ioctls, one to get the timeout for a given pt device, the
other to set the timeout for a given pt device. The idea is that
userland applications using the device can set the timeout to suit their
purposes. The ioctls are defined in a new header file, sys/ptio.h

PR: 10266
Reviewed by: gibbs, joerg

Welcome devtoname(), to most likely be used when printing information
about a dev_t.

printf("%x", dev) now becomes printf("%s", devtoname(dev)) because
printing actual information about the device is much more useful then
printing a pointer to an address that would never help the developer debug.

Submitted by: phk, bde

Simplify cdevsw registration.

The cdevsw_add() function now finds the major number(s) in the
struct cdevsw passed to it. cdevsw_add_generic() is no longer
needed, cdevsw_add() does the same thing.

cdevsw_add() will print an message if the d_maj field looks bogus.

Remove nblkdev and nchrdev variables. Most places they were used
bogusly. Instead check a dev_t for validity by seeing if devsw()
or bdevsw() returns NULL.

Move bdevsw() and devsw() functions to kern/kern_conf.c

Bump __FreeBSD_version to 400006

This commit removes:
72 bogus makedev() calls
26 bogus SYSINIT functions

if_xe.c bogusly accessed cdevsw[], author/maintainer please fix.

I4b and vinum not changed. Patches emailed to authors. LINT
probably broken until they catch up.

This commit should be a extensive NO-OP:

Reformat and initialize correctly all "struct cdevsw".

Initialize the d_maj and d_bmaj fields.

The d_reset field was not removed, although it is never used.

I used a program to do most of this, so all the files now use the
same consistent format. Please keep it that way.

Vinum and i4b not modified, patches emailed to respective authors.

Add a default async handler funstion to cam_periph.c to remove duplicated
code in all initiator type peripheral drivers.

scsi_target.c:
Release ATIO structures that wind up in the 'unkown command queue'
for consumption by our userland counterpart, back to the controller
when the exception for that command is cleared.

Add a facility in the CAM error handling code to retry selection timeouts.
If the client requests that the error recovery code retry a selection
timeout, it will be retried after half a second. The delay is to give the
device time to recover.

For most of these drivers, I only added selection timeout retries where
they were also retrying unit attention type errors. The sa(4) driver calls
saerror() in a number of places, but most of them don't request retrying
unit attentions.

Also, bump the default minimum CD changer timeout from 2 to 5 seconds and
the maximum timeout from 10 to 15 seconds. Some Pioneer changers seem to
have trouble with the shorter timeout.

Reviewed by: gibbs

Introduce two functions: physread() and physwrite() and use these directly
in *devsw[] rather than the 46 local copies of the same functions.

(grog will do the same for vinum when he has time)

Add a prioritization field to the devstat_add_entry() call so that
peripheral drivers can determine where in the devstat(9) list they are
inserted.

This requires recompilation of libdevstat, systat, vmstat, rpc.rstatd, and
any ports that depend on the devstat code, since the size of the devstat
structure has changed. The devstat version number has been incremented as
well to reflect the change.

This sorts devices in the devstat list in "more interesting" to "less
interesting" order. So, for instance, da devices are now more important
than floppy drives, and so will appear before floppy drives in the default
output from systat, iostat, vmstat, etc.

The order of devices is, for now, kept in a central table in devicestat.h.
If individual drivers were able to make a meaningful decision on what
priority they should be at attach time, we could consider splitting the
priority information out into the various drivers. For now, though, they
have no way of knowing that, so it's easier to put them in an easy to find
table.

Also, move the checkversion() call in vmstat(8) to a more logical place.

Thanks to Bruce and David O'Brien for suggestions, for reviewing this, and
for putting up with the long time it has taken me to commit it. Bruce did
object somewhat to the central priority table (he would rather the
priorities be distributed in each driver), so his objection is duly noted
here.

Reviewed by: bde, obrien

Fix a few problems that Bruce noticed about a month ago, and fix oup one
other problem.

- Hold onto splsoftcam() in the peripheral driver open routines until we
have locked the periph. This eliminates a race condition.

- Disallow opening the pass driver when securelevel > 1.

- If a user tries to open the pass driver with O_NONBLOCK set, return
EINVAL instead of ENODEV. (noticed by gibbs)

Fix a problem with the way we handled device invalidation when attaching
to a device failed.

In theory, the same steps that happen when we get an AC_LOST_DEVICE async
notification should have been taken when a driver fails to attach. In
practice, that wasn't the case.

This only affected the da, cd and ch drivers, but the fix affects all
peripheral drivers.

There were several possible problems:
- In the da driver, we didn't remove the peripheral's softc from the da
driver's linked list of softcs. Once the peripheral and softc got
removed, we'd get a kernel panic the next time the timeout routine
called dasendorderedtag().
- In the da, cd and possibly ch drivers, we didn't remove the
peripheral's devstat structure from the devstat queue. Once the
peripheral and softc were removed, this could cause a panic if anyone
tried to access device statistics. (one component of the linked list
wouldn't exist anymore)
- In the cd driver, we didn't take the peripheral off the changer run
queue if it was scheduled to run. In practice, it's highly unlikely,
and maybe impossible that the peripheral would have been on the
changer run queue at that stage of the probe process.

The fix is:
- Add a new peripheral callback function (the "oninvalidate" function)
that is called the first time cam_periph_invalidate() is called for a
peripheral.

- Create new foooninvalidate() routines for each peripheral driver. This
routine is always called at splsoftcam(), and contains all the stuff
that used to be in the AC_LOST_DEVICE case of the async callback
handler.

- Move the devstat cleanup call to the destructor/cleanup routines, since
some of the drivers do I/O in their close routines.

- Make sure that when we're flushing the buffer queue, we traverse it at
splbio().

- Add a check for the invalid flag in the pt driver's open routine.

Reviewed by: gibbs

Clean up some unused variables.

Reviewed by: ken
Submitted by: phk

SCSI Peripheral drivers for CAM:

da - Direct Access Devices (disks, optical devices, SS disks)
cd - CDROM (or devices that can act like them, WORM, CD-RW, etc)
ch - Medium Changer devices.
sa - Sequential Access Devices (tape drives)
pass - Application pass-thru driver
targ - Target Mode "Processor Target" Emulator
pt - Processor Target Devices (scanners, cpus, etc.)

Submitted by: The CAM Team