CAM: Replace random sbuf_printf() with cheaper cat/putc.

cam: Minor opt_cam.h cleanup

sys/cam/cam.h includes opt_cam.h, so none of the clients need to do
this. cam.h does all the right dancing to conditionally include
opt_cam.h only when it makes sense. It generally only matters when
cam_debug.h is included (it must be included before that). Many of the
stray opt_cam.h includes were after cam_debug.h which would be a problem
were it not included in cam/cam.h. The other users of CAM options that
aren't debug all already include cam/cam.h.

Also trim unneeded sys/cdefs.h files from the files touched.

Sponsored by: Netflix

cam: Remove extra break

Sponsored by: Netflix

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

cam: Remove non-sbuf announce/denounce proto and xport ops

Reviewed by: mav, imp
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D41264

cam/scsi: Migrate to modern uintXX_t from u_intXX_t

As per https://lists.freebsd.org/archives/freebsd-scsi/2023-July/000257.html
move to the modern uintXX_t.

MFC After: 3 days
Sponsored by: Netflix

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

Fix bd82711aff98 serial number trailing spaces removal.

For devices returning 16 byte serial numbers with 8 leading spaces
this falsely removed everything due to looking at wrong offset.

cam: Remove trailing spaces from serial numbers too

The SanDisk SD8SB8U1 and likely others pad their serial number with
spaces on the end rather than the start (at least when connected to a
SAS3008). This makes them difficult to wire unit numbers to with the
serial because you have to specify the trailing spaces. Instead, strip
out the trailing spaces.

We already strip leading spaces both here. In addition, when glabel
creates the devfs device nodes, leading and trailing spaces are removed
already (so there will be no change there either).

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D32684

cam: revert second half of 75b5caa08ef

This turns debugging printf() into a KASSERT().

Reviewed By: imp
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D31523

cam(4): Fix quick unplug/replug for SCSI.

If some device is plugged back in after unplug before the probe periph
destroyed, it will just restart the probe process. But I've found that
PROBE_INQUIRY_CKSUM flag not cleared between the iterations may cause
AC_FOUND_DEVICE not reported on the second iteration, and because of
AC_LOST_DEVICE reported during the first iteration, the device end up
configured, but without any periphs attached.

We've found that enabled serial console and 102-disk JBOD cause enough
probe delays to easily trigger the issue for half of the disks. This
change fixes it reliably on my tests.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

cam(4): Mark all sysctls as CTLFLAG_MPSAFE.

This code does not use Giant lock for very long time.

MFC after: 2 weeks

cam: drop unused 'saved_ccb' field from softcs

No functional changes. Do not MFC this, it changes kernel ABI.

Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D30698

cam: turn KASSERTs into printfs for now

It looks like I've missed a couple of places where we don't clear
stack-allocated CCBs. Don't panic when that happens, just print
a warning.

This is a temporary measure until I get those cases fixed.

Reviewed By: markj
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D30296

cam: add missing zeroing of a stack-allocated CCB.

This could cause a panic at boot.

Reported By: Shawn Webb <shawn.webb AT hardenedbsd.org>
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.

cam: allocate CCBs from UMA for SCSI and ATA IO

This patch makes it possible for CAM to use small CCBs allocated
from an periph-specific UMA zone instead of the usual, huge ones.
The end result is that CCBs issued via da(4) take 544B (size of
ccb_scsiio) instead of the usual 2kB (size of 'union ccb', ~1.5kB,
rounded up by malloc(9)). For ATA it's 272B. We waste less
memory, we avoid zeroing the unused 1kB, and it should be easier
to allocate those CCBs in low memory conditions. It should also
be possible to use uma_zone_reserve(9) to improve behaviour
in low memory conditions even further.

Note that this does not change the size, or the layout, of CCBs
as such. CCBs get allocated in various different ways, in particular
on the stack, and I don't want to redo all that. Instead, this
provides an opt-in mechanism for the periph to declare "my start()
callback is fine with receiving a CCB allocated from this UMA zone".
In other words, most of the code works exactly as it used to; the
change only happens to IOs issued by xpt_run_allockq(), which
is - conveniently - pretty much all that matters for performance.

The reason for doing it this way is that it's pretty small, localized
change, and can be implemented gradually and iteratively: take a
periph, make sure its start() callback only casts the CCBs it takes
to a particular type of CCB, for example ccb_scsiio, and that it only
casts CCBs returned by cam_periph_getccb() to that type, then add UMA
zone for that size, and declare it safe to XPT.

This is disabled by default. Set 'kern.cam.ada.enable_uma_ccbs=1'
and 'kern.cam.da.enable_uma_ccbs=1' tunables to enable it. Testing
is welcome; I will flip the default to enable in two weeks from now.

Reviewed By: imp
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D28674

cam: fix integer overflow during inquiry

From my understanding this could happen with iSCSI LUNs with
unusually long names. The bug would make CAM fail to retrieve
the full inquiry data. Instead of bumping the size of the local
variable, just use a macro.

Reviewed By: imp, mav
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
X-NetApp-PR: #50
Differential Revision: https://reviews.freebsd.org/D29991

cam: make sure to clear even more CCBs allocated on the stack

This is my second pass, this time over all of CAM except
for the SCSI target bits. There should be no functional
changes.

Reviewed By: imp
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D29549

cam: make sure to clear CCBs allocated on the stack

This is required for small CCBs support, where we need to track
whether the CCB was allocated from an UMA zone or not. There are
no (intended) functional changes with the current source.

Reviewed By: imp
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D29484

cam: clean up empty lines in .c and .h files

cam: ANSIfy 0-argument function definitions

No functional change.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D24854

Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many)

r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are
still not MPSAFE (or already are but aren’t properly marked).
Use it in preparation for a general review of all nodes.

This is non-functional change that adds annotations to SYSCTL_NODE and
SYSCTL_PROC nodes using one of the soon-to-be-required flags.

Mark all obvious cases as MPSAFE. All entries that haven't been marked
as MPSAFE before are by default marked as NEEDGIANT

Approved by: kib (mentor, blanket)
Commented by: kib, gallatin, melifaro
Differential Revision: https://reviews.freebsd.org/D23718

Take proper lock in ses_setphyspath_callback().

XPT_DEV_ADVINFO call should be protected by the lock of the specific
device it is addressed to, not the lock of SES device. In some weird
case, probably with hardware violating standards, it sometimes caused
NULL dereference due to race.

To protect from it further, add lock assertion to *_dev_advinfo().

MFC after: 1 week
Sponsored by: iXsystems, Inc.

Always check cam_periph_error() status for ERESTART.

Even if we do not expect retries, we better be sure, since otherwise it
may result in use after free kernel panic. I've noticed that it retries
SCSI_STATUS_BUSY even with SF_NO_RECOVERY | SF_NO_RETRY.

MFC after: 1 week
Sponsored by: iXsystems, Inc.

Minor white space changes.

Remove trailing white space that's crept into this file.

CID 1009492: Logically dead code in sys/cam/scsi/scsi_xpt.c

In `probedone()`, for the `PROBE_REPORT_LUNS` case, all paths that
fall to the bottom of the case set `lp` to `NULL`, so the test for a
non-NULL value of `lp` and call to `free()` if true is dead code as
the test can never be true. Fix by eliminating the whole if
statement. To guard against a possible future change that accidentally
violates this assumption, use a `KASSERT()` to catch if `lp` is
non-NULL.

Reviewed by: cem
MFC after: 1 week
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D19109

remove unneeded inclusion of sys/interrupt.h from several files

It's likely that the header was needed in the past for swi(9).
But now that code does not use swi(9) or any other interfaces defined
in sys/interrupt.h.

MFC after: 1 week

Return a C errno for cam_periph_acquire().

There's no compelling reason to return a cam_status type for this
function and doing so only creates confusion with normal C
coding practices. It's technically an API change, but the periph API
isn't widely used. No efffective change to operation.

Reviewed by: imp, mav, ken
Sponsored by: Netflix
Differential Revision: D14063

Define xpt_path_inq.

This provides a nice wrarpper around the XPT_PATH_INQ ccb creation and
calling.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D13387

Remove unused 4th argument to match the standard error routines.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D13386

sys/cam: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Add assertion in probedone() that we're holding the device lock.

Submitted by: ken
Reviewed by: asomers
MFC after: 3 weeks
Sponsored by: Spectra Logic Corp

Fix potential NULL pointer dereference of device physical path

In scsi_dev_advinfo(), if the physical path is being stored and there is a
malloc failure (malloc(9) is called with M_NOWAIT), we could wind up in a
situation where the device's physpath_len is set to the length the user
provided, but the physpath itself is NULL.

If another context then comes in to fetch the physical path value, we would
wind up trying to memcpy a NULL pointer into the caller's buffer.

So, set the physpath_len to 0 when we free the physpath on entry into the
store case for the physical path. Reset the length to a non-zero value only
after we've successfully malloced a buffer to hold it.

Submitted by: ken
Reviewed by: asomers
MFC after: 3 weeks
Sponsored by: Spectra Logic Corp

Introduce Genesys GL3224 quirks

The Genesys chip is failing when issueing READ_CAP(16) command.
Force a quirk to disable it and use READ_CAP(10) instead.

Also, depending on used firmware, GL3224 can be recognized
either as 'storage device' or 'mass storage class' -
enable both variants in scsi_quirk_table.

Submitted by: Wojciech Macek <wma@semihalf.com>
Konrad Adamczyk <ka@semihalf.com>
Obtained from: Semihalf
Sponsored by: Stormshield
Reviewed by: mav
Differential revision: https://reviews.freebsd.org/D10902

Add infrastructure to the ATA and SCSI transports that supports
using a driver-supplied sbuf for printing device discovery
announcements. This helps ensure that messages to the console
will be properly serialized (through sbuf_putbuf) and not be
truncated and interleaved with other messages. The
infrastructure mirrors the existing xpt_announce_periph()
entry point and is opt-in for now. No content or formatting
changes are visible to the operator other than the new coherency.

While here, eliminate the stack usage of the temporary
announcement buffer in some of the drivers. It's moved to the
softc for now, but future work will eliminate it entirely by
making the code flow more linear. Future work will also address
locking so that the sbufs can be dynamically sized.

The scsi_da, scs_cd, scsi_ses, and ata_da drivers are converted
at this point, other drivers can be converted at a later date.
A tunable+sysctl, kern.cam.announce_nosbuf, exists for testing
purposes but will be removed later.

TODO:
Eliminate all of the code duplication and temporary buffers. The
old printf-based methods will be retired, and xpt_announce_periph()
will just be a wrapper that uses a dynamically sized sbuf. This
requires that the register and deregister paths be made malloc-safe,
which they aren't currently.

Sponsored by: Netflix

Improve CAM_CDB_POINTER support.

MFC after: 2 weeks

Fix "camcontrol rescan" with SATA drives behind a SAS controller

A bug in CAM's serial number hash logic resulted in SATA drives behind a SAS
controller getting removed and readded anytime the drive was rescanned for
any reason.

PR: 212914
Submitted by: kadesai
Reported by: kadesai
Reviewed by: asomers, ken
MFC after: 4 weeks
Sponsored by: Spectra Logic Corp

Move protocol specific stuff into a linker set object that's
per-protocol. This reduces the number scsi symbols references by
cam_xpt significantly, and eliminates all ata / nvme symbols. There's
still some NVME / ATA specific code for dealing with XPT_NVME_IO and
XPT_ATA_IO respectively, and a bunch of scsi-specific code, but this
is progress.

Differential Revision: https://reviews.freebsd.org/D7289

Switch to linker sets to find the xport callback object. This
eliminates the need to special case everything in cam_xpt for new
transports. It is now a failure to not have a transport object when
registering the bus as well. You can still, however, create a
transport that's unspecified (XPT_)

Differential Revision: https://reviews.freebsd.org/D7289

Strip leading spaces off of a SCSI disk's serial number

sys/cam/scsi/scsi_xpt.c
Strip leading spaces off of a SCSI disk's reported serial number
when populating the CAM serial number. This affects the output of
"diskinfo -v" and the names of /dev/diskid/DISK-* device nodes,
among other things.

SPC5r05 says that the Product Serial Number field from the Unit
Serial Number VPD page is right-aligned. So any leading spaces are
not part of the actual serial number. Most devices don't left-pad
their serial numbers, but some do. In particular, the SN VPD page
that an LSI HBA emulates for a SATA drive contains enough
left-padding to fill a 20-byte field.

UPDATING
Add a note to UPDATING, because some users may have to update
/etc/fstab or geom labels.

Reviewed by: ken, mav
MFC after: Never
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D6516

cam/scsi: Remove mostly unused scsi_quirk_table_size.

It was only being used only once and it can be directly replaced
instead.

sys/cam: spelling fixes in comments.

No functional change.

sys: use our nitems() macro when param.h is available.

This should cover all the remaining cases in the kernel.

Discussed in: freebsd-current

Fix the SCSI Extended INQUIRY probe case when an error is returned
and a retry is scheduled.

Instead of leaving the device queue frozen, unfreeze the device queue so
that the retry can happen.

Sponsored by: Spectra Logic
MFC after: 3 days

Attach pass driver to LUNs is OFFLINE state.

Previously such LUNs were silently ignored. But while they indeed unable
to process most of SCSI commands, some, like RTPG, they still can.

MFC after: 1 month

Revert r282227. It is clearly incorrect as it frees an object that is still
referenced. I think that there does exist an unlikely edge case for a
memory leak, but only if a driver is incorrectly written and specifies no
valid range of targets to scan. That can be fixed in a follow-up commit.

Obtained from: Netflix, Inc.

Fix memory leak in scsi_scan_bus()

CID: 1007770
PR: 199671

Add support for probing the SCSI VPD Extended Inquiry page (0x86).

This VPD page is effectively an extension of the standard Inquiry
data page, and includes lots of additional bits.

This commit includes support for probing the page in the SCSI probe code,
and an additional request type for the XPT_DEV_ADVINFO CCB. CTL already
supports the Extended Inquiry page.

Support for querying this page in the sa(4) driver will come later.

sys/cam/scsi/scsi_xpt.c:
Probe the Extended Inquiry page, if the device supports it, and
return it in response to a XPT_DEV_ADVINFO CCB if it is requested.

sys/cam/scsi/cam_ccb.h:
Define a new advanced information CCB data type, CDAI_TYPE_EXT_INQ.

sys/cam/cam_xpt.c:
Free the extended inquiry data in a device when the device goes
away.

sys/cam/cam_xpt_internal.h:
Add an extended inquiry data pointer and length to struct cam_ed.

sys/sys/param.h
Bump __FreeBSD_version for the addition of the new
CDAI_TYPE_EXT_INQ advanced information type.

Sponsored by: Spectra Logic
MFC after: 1 week

Fix a bug in the CAM SCSI probe code that caused changes in inquiry
data to go undetected.

The probe code does an MD5 checksum of the inquiry data (and page
0x80 serial number if available) before doing a reprobe of an
existing device, and then compares a checksum after the probe to
see whether the device has changed.

This check was broken in January, 2000 by change 56146 when the extended
inquiry probe code was added.

In the extended inquiry probe case, it was calculating the checksum
a second time. The second time it included the updated inquiry
data from the short inquiry probe (first 36 bytes). So it wouldn't
catch cases where the vendor, product, revision, etc. changed.

This change will have the effect that when a device's inquiry data is
updated and a rescan is issued, it will disappear and then reappear.
This is the appropriate action, because if the inquiry data or serial
number changes, it is either a different device or the device
configuration may have changed significantly. (e.g. with updated
firmware.)

scsi_xpt.c: Don't calculate the initial MD5 checksum on
standard inquiry data and the page 0x80 serial
number if we have already calculated it.

MFC after: 1 week
Sponsored by: Spectra Logic

Remove residual xpt_release_device() call left after r272406 cleanup.

Excessive release here could trigger use-after-free condition and kernel
panic on LUN 0 disconnect.

MFC after: 1 week

Use REPORT LUNS command for SPC-2 devices with LUN 0 disconnected.

SPC-2 tells REPORT LUNS shall be supported by devices supporting LUNs other
then LUN 0. If we see LUN 0 disconnected, guess there may be others, and
so REPORT LUNS shall be supported.

MFC after: 1 month

Make disconnected LUN 0 don't remain in half-configured state if there are
no LUNs on SPC-3 target after we tried REPORT LUNS.

Restore CAM_QUIRK_NOLUNS check, lost in previous commit.

MFC after: 1 month

Rework the logic of sequential SCSI LUN scanner.

Previous logic was not differentiating disconnected LUNs and absent targets.
That made it to stop scan if LUN 0 was not found for any reason. That made
problematic, for example, using iSCSI targets declaring SPC-2 compliance and
having no LUN 0 configured.

The new logic continues sequential LUN scan if:
-- we have more configured LUNs that need recheck;
-- this LUN is connected and its SCSI version allows more LUNs;
-- this LUN is disconnected, its SCSI version allows more LUNs and we
guess they may be connected (we haven't scanned first 8 LUNs yet or
kern.cam.cam_srch_hi sysctl is set to scan more).

Reported by: trasz
MFC after: 1 month

Pull in r267961 and r267973 again. Fix for issues reported will follow.

Revert r267961, r267973:

These changes prevent sysctl(8) from returning proper output,
such as:

1) no output from sysctl(8)
2) erroneously returning ENOMEM with tools like truss(1)
or uname(1)
truss: can not get etype: Cannot allocate memory

Extend the meaning of the CTLFLAG_TUN flag to automatically check if
there is an environment variable which shall initialize the SYSCTL
during early boot. This works for all SYSCTL types both statically and
dynamically created ones, except for the SYSCTL NODE type and SYSCTLs
which belong to VNETs. A new flag, CTLFLAG_NOFETCH, has been added to
be used in the case a tunable sysctl has a custom initialisation
function allowing the sysctl to still be marked as a tunable. The
kernel SYSCTL API is mostly the same, with a few exceptions for some
special operations like iterating childrens of a static/extern SYSCTL
node. This operation should probably be made into a factored out
common macro, hence some device drivers use this. The reason for
changing the SYSCTL API was the need for a SYSCTL parent OID pointer
and not only the SYSCTL parent OID list pointer in order to quickly
generate the sysctl path. The motivation behind this patch is to avoid
parameter loading cludges inside the OFED driver subsystem. Instead of
adding special code to the OFED driver subsystem to post-load tunables
into dynamically created sysctls, we generalize this in the kernel.

Other changes:
- Corrected a possibly incorrect sysctl name from "hw.cbb.intr_mask"
to "hw.pcic.intr_mask".
- Removed redundant TUNABLE statements throughout the kernel.
- Some minor code rewrites in connection to removing not needed
TUNABLE statements.
- Added a missing SYSCTL_DECL().
- Wrapped two very long lines.
- Avoid malloc()/free() inside sysctl string handling, in case it is
called to initialize a sysctl from a tunable, hence malloc()/free() is
not ready when sysctls from the sysctl dataset are registered.
- Bumped FreeBSD version to indicate SYSCTL API change.

MFC after: 2 weeks
Sponsored by: Mellanox Technologies

Fix for r260541: do not drop periph reference when request is restarted.
CAM_DEV_QFREEZE flag is still there and it will freeze device again.

Take additional reference on SCSI probe periph to cover its freeze count.

Otherwise periph may be invalidated and freed before single-stepping freeze
is dropped, causing use after free panic.

Widen lun_id_t to 64 bits. This is a follow-on to r257345 to let the kernel
support all valid SAM-5 LUN IDs. CAM_VERSION is bumped, as the CAM ABI
(though not API) is changed. No behavior is changed relative to r257345
except that LUNs with non-zero high 32 bits will no longer be ignored
during device enumeration for SIMs that have set PIM_EXTLUNS.

Reviewed by: scottl

Implement extended LUN support. If PIM_EXTLUNS is set by a SIM, encode
the upper 32-bits of the LUN, if possible, into the target_lun field as
passed directly from the REPORT LUNs response. This allows extended LUN
support to work for all LUNs with zeros in the lower 32-bits, which covers
most addressing modes without breaking KBI. Behavior for drivers not
setting PIM_EXTLUNS is unchanged. No user-facing interfaces are modified.

Extended LUNs are stored with swizzled 16-bit word order so that, for
devices implementing LUN addressing (like SCSI-2), the numerical
representation of the LUN is identical with and without PIM_EXTLUNS. Thus
setting PIM_EXTLUNS keeps most behavior, and user-facing LUN IDs, unchanged.
This follows the strategy used in Solaris. A macro (CAM_EXTLUN_BYTE_SWIZZLE)
is provided to transform a lun_id_t into a uint64_t ordered for the wire.

This is the second part of work for full 64-bit extended LUN support and is
designed to a bridge for stable/10 to the final 64-bit LUN code. The
third and final part will involve widening lun_id_t to 64 bits and will
not be MFCed. This third part will break the KBI but will keep the KPI
unchanged so that all drivers that will care about this can be updated now
and not require code changes between HEAD and stable/10.

Reviewed by: scottl
MFC after: 2 weeks

Merge CAM locking changes from the projects/camlock branch to radically
reduce lock congestion and improve SMP scalability of the SCSI/ATA stack,
preparing the ground for the coming next GEOM direct dispatch support.

Replace big per-SIM locks with bunch of smaller ones:
- per-LUN locks to protect device and peripheral drivers state;
- per-target locks to protect list of LUNs on target;
- per-bus locks to protect reference counting;
- per-send queue locks to protect queue of CCBs to be sent;
- per-done queue locks to protect queue of completed CCBs;
- remaining per-SIM locks now protect only HBA driver internals.

While holding LUN lock it is allowed (while not recommended for performance
reasons) to take SIM lock. The opposite acquisition order is forbidden.
All the other locks are leaf locks, that can be taken anywhere, but should
not be cascaded. Many functions, such as: xpt_action(), xpt_done(),
xpt_async(), xpt_create_path(), etc. are no longer require (but allow) SIM
lock to be held.

To keep compatibility and solve cases where SIM lock can't be dropped, all
xpt_async() calls in addition to xpt_done() calls are queued to completion
threads for async processing in clean environment without SIM lock held.

Instead of single CAM SWI thread, used for commands completion processing
before, use multiple (depending on number of CPUs) threads. Load balanced
between them using "hash" of the device B:T:L address.

HBA drivers that can drop SIM lock during completion processing and have
sufficient number of completion threads to efficiently scale to multiple
CPUs can use new function xpt_done_direct() to avoid extra context switch.
Make ahci(4) driver to use this mechanism depending on hardware setup.

Sponsored by: iXsystems, Inc.
MFC after: 2 months

Make some improvements to r253322 to really rescan target, not a bus.
Add there and in two more places checks for NULL on xpt_alloc_ccb_nowait().

Introduce accessors for the ccb status word. Convert one (of many more)
modules to use it, will convert the others once the appropriate shed
color is selected by consensus.

Obtained from: Netflix
MFC after: 3 days

Add some cam_freeze_devq()'s missed at r249466.
This makes number of freezes match the number of releases.

Reported by: dim

Refactored scsi_xpt use of device_has_vpd to generic scsi_vpd_supported_page
so its available for use in generic scsi code.

This is a pre-requirement for using VPD queries to determine available SCSI
delete methods within scsi_da.

Reviewed by: mav
Approved by: pjd (mentor)
MFC after: 2 weeks

MFprojects/camlock r248982:
Stop abusing xpt_periph in random plases that really have no periph related
to CCB, for example, bus scanning. NULL value is fine in such cases and it
is correctly logged in debug messages as "noperiph". If at some point we
need some real XPT periphs (alike to pmpX now), quite likely they will be
per-bus, and not a single global instance as xpt_periph now.

MFprojects/camlock r248890, r248897, r248898, r248900, r248903, r248905,
r248917, r248918, r248978, r249001, r249014, r249030:

Remove multilevel freezing mechanism, implemented to handle specifics of
the ATA/SATA error recovery, when post-reset recovery commands should be
allocated when queues are already full of payload requests. Instead of
removing frozen CCBs with specified range of priorities from the queue
to provide free openings, use simple hack, allowing explicit CCBs over-
allocation for requests with priority higher (numerically lower) then
CAM_PRIORITY_OOB threshold.

Simplify CCB allocation logic by removing SIM-level allocation queue.
After that SIM-level queue manages only CCBs execution, while allocation
logic is localized within each single device.

Suggested by: gibbs

MFprojects/camlock:
r249017:
Some cosmetic things:
- Unify device to target insertion inside xpt_alloc_device() instead of
duplicating it three times.
- Remove extra checks for empty lists of devices and targets on release
since zero refcount check also implies it.
- Reformat code to reduce indentation.

r249103:
- Add lock assertions to every point where reference counters are modified.
- When reference counters are reaching zero, add assertions that there are
no children items left.
- Add a bit more locking to the xptpdperiphtraverse().

Don't duplicate path/ccb allocation code, use existing functions.

Remove 'periph == NULL' check from bunch of periph drivers.
This condition can never be true as functions are called from single place
and the checks just pollute the code and confuse Clang Static Analyzer.

One more major cam_periph_error() rewrite to improve error handling and
reporting. It includes:
- removing of error messages controlled by bootverbose, replacing them
with more universal and informative debugging on CAM_DEBUG_INFO level,
that is now built into the kernel by default;
- more close following to the arguments submitted by caller, such as
SF_PRINT_ALWAYS, SF_QUIET_IR and SF_NO_PRINT; consumer knows better which
errors are usual/expected at this point and which are really informative;
- adding two new flags SF_NO_RECOVERY and SF_NO_RETRY to allow caller
specify how much assistance it needs at this point; previously consumers
controlled that by not calling cam_periph_error() at all, but that made
behavior inconsistent and debugging complicated;
- tuning debug messages and taken actions order to make debugging output
more readable and cause-effect relationships visible;
- making camperiphdone() (common device recovery completion handler) to
also use cam_periph_error() in most cases, instead of own dumb code;
- removing manual sense fetching code from cam_periph_error(); I was told
by number of people that it is SIM obligation to fetch sense data, so this
code is useless and only significantly complicates recovery logic;
- making ada, da and pass driver to use cam_periph_error() with new limited
recovery options to handle error recovery and debugging in common way;
as one of results, CAM_REQUEUE_REQ and other retrying statuses are now
working fine with pass driver, that caused many problems before.
- reverting r186891 by raj@ to avoid burning few seconds in tight DELAY()
loops on device probe, while device simply loads media; I think that problem
may already be fixed in other way, and even if it is not, solution must be
different.

Sponsored by: iXsystems, Inc.
MFC after: 2 weeks

Tune and add some more CAM_DEBUG() points for the probe sequences.

Add support for newer garmin devices

PR: kern/163932
Submitted by: Bartosz Fabianowski <freebsd@chillt.de>
Approved by: cperciva
MFC after: 1 week

Plug request and references leak caused by race between invalidated
ond probe periph destruction and new incoming probe request.

This at least caused problems with SATA Port Multipliers hot-plug.

MFC after: 2 weeks

Limit the ST3146855LW U320 drive to 55 tags to avoid command timeouts
under load.

Submitted by: Gelson Borsoi
MFC after: 1 day

Add CAM infrastructure to allow reporting when a drive's long read capacity
data changes.

cam_ccb.h: Add a new advanced information type, CDAI_TYPE_RCAPLONG,
for long read capacity data.

cam_xpt_internal.h:
Add a read capacity data pointer and length to struct cam_ed.

cam_xpt.c: Free the read capacity buffer when a device goes away.
While we're here, make sure we don't leak memory for other
malloced fields in struct cam_ed.

scsi_all.c: Update the scsi_read_capacity_16() to take a uint8_t * and
a length instead of just a pointer to the parameter data
structure. This will hopefully make this function somewhat
immune to future changes in the parameter data.

scsi_all.h: Add some extra bit definitions to struct
scsi_read_capacity_data_long, and bump up the structure
size to the full size specified by SBC-3.

Change the prototype for scsi_read_capacity_16().

scsi_da.c: Register changes in read capacity data with the transport
layer. This allows the transport layer to send out an
async notification to interested parties. Update the
dasetgeom() API.

Use scsi_extract_sense_len() instead of
scsi_extract_sense().

scsi_xpt.c: Add support for the new CDAI_TYPE_RCAPLONG advanced
information type.

Make sure we set the physpath pointer to NULL after freeing
it. This allows blindly freeing it in the struct cam_ed
destructor.

sys/param.h: Bump __FreeBSD_version from 1000005 to 1000006 to make it
easier for third party drivers to determine that the read
capacity data async notification is available.

camcontrol.c,
mptutil/mpt_cam.c:
Update these for the new scsi_read_capacity_16() argument
structure.

Sponsored by: Spectra Logic

Do not use the sometimes-reserved work 'bool' for a variable name.

MFC after: 2 weeks
Sponsored by: Isilon Systems, LLC

Return CAM_REQ_INVALID if the SCSI XPT receives an unsupported operation
via the XPT_DEV_ADVINFO CCB.

Reviewed by: ken

Lay groundwork in CAM for recording and reporting physical path and
other device attributes stored in the CAM Existing Device Table (EDT).
This includes some infrastructure requried by the enclosure services
driver to export physical path information.

Make the CAM device advanced info interface accept store requests.

sys/cam/scsi/scsi_all.c:
sys/cam/scsi/scsi_all.h:
- Replace scsi_get_sas_addr() with a scsi_get_devid() which takes
a callback that decides whether to accept a particular descriptor.
Provide callbacks for NAA IEEE Registered addresses and for SAS
addresses, replacing the old function. This is needed because
the old function doesn't work for an enclosure address for a SAS
device, which is not flagged as a SAS address, but is NAA IEEE
Registered. It may be worthwhile merging this interface with the
devid match interface.
- Add a few more defines for some device ID fields.

sbin/camcontrol/camcontrol.c:
- Update for the CCB_DEV_ADVINFO interface change.

cam/cam_xpt_internal.h:
- Add the new fields for the physical path string to the CAM EDT.
cam/cam_ccb.h:
- Rename CCB_GDEV_ADVINFO to simply CCB_DEV_ADVINFO, and the ccb
structure to ccb_dev_advinfo.
- Add a flag that changes this CCB's action to store, rather than
the default, retrieve.
- Add a new buffer type, CDAI_TYPE_PHYS_PATH, for the new CAM EDT
physpath field.
- Remove the never-implemented transport & proto flags.
cam/cam_xpt.c:
cam/cam_xpt.h:
- Add xpt_getattr(), which provides a wrapper for fetching a device's
attribute using the GEOM strings as key. This method currently
supports "GEOM::ident" and "GEOM::physpath".

Submitted by: will
Reviewed by : gibbs

Extend the XPT_DEV_MATCH api to allow a device search by device ID.
As far as the API is concerned, device ID is a binary blob to be
interpreted by the transport layer. The SCSI implementation assumes
it is an array of VPD device ID descriptors.

sys/cam/cam_ccb.h:
Create a new structure, device_id_match_pattern, and
update the XPT_DEV_MATCH datastructures and flags so
that this pattern type can be used.

sys/cam/cam_xpt.c:
- A single pattern matching on both inquiry data and device
ID is invalid. Report any violators.
- Pass device ID match requests through to the new routine
scsi_devid_match(). The direct call of a SCSI routine is
a layering violation, but no worse than the one a few
lines up that checks inquiry data. Defer cleaning this
up until our future, larger, rototilling of CAM.
- Zero out cam_ed and cam_et nodes on allocation. Prior to
this change, device_id_len and device_id were not inialized,
preventing proper detection of the presence of this
information.

sys/cam/scsi/scsi_all.c:
sys/cam/scsi/scsi_all.h:
Add the scsi_match_devid() routine.

Add a helper function for extracting peripherial driver names

sys/cam/cam_periph.c:
sys/cam/cam_periph.h:
Add the cam_periph_list() method which fills an sbuf
with a comma delimited list of the peripheral instances
associated with a given CAM path.

Add a helper functions for SCSI commands used by the SES driver.

sys/cam/scsi/scsi_all.c:
sys/cam/scsi/scsi_all.h:
Add structure definitions and csio filling functions for
the receive diagnostic results and send diagnostic commands.

Misc CAM XPT cleanups.

sys/cam/cam_xpt.c:
Broadcast AC_FOUND_DEVICE and AC_PATH_REGISTERED
events at the time async event handlers are attached
even when registering just for events on a partitular
SIM. Previously, you had to register for these
events on all SIMs in the system in order to get
the initial broadcast even though subsequent device
and path arrivals would be delivered.

sys/cam/cam_xpt.c:
Remove SIM mutex held asserts from path accessors.
CAM paths are reference counted and it is this
reference count, not the sim mutex, that garantees
they are stable.

Sponsored by: Spectra Logic Corporation

Add Serial Management Protocol (SMP) passthrough support to CAM.

This includes support in the kernel, camcontrol(8), libcam and the mps(4)
driver for SMP passthrough.

The CAM SCSI probe code has been modified to fetch Inquiry VPD page 0x00
to determine supported pages, and will now fetch page 0x83 in addition to
page 0x80 if supported.

Add two new CAM CCBs, XPT_SMP_IO, and XPT_GDEV_ADVINFO. The SMP CCB is
intended for SMP requests and responses. The ADVINFO is currently used to
fetch cached VPD page 0x83 data from the transport layer, but is intended
to be extensible to fetch other types of device-specific data.

SMP-only devices are not currently represented in the CAM topology, and so
the current semantics are that the SIM will route SMP CCBs to either the
addressed device, if it contains an SMP target, or its parent, if it
contains an SMP target. (This is noted in cam_ccb.h, since it will change
later once we have the ability to have SMP-only devices in CAM's topology.)

smp_all.c,
smp_all.h: New helper routines for SMP. This includes
SMP request building routines, response parsing
routines, error decoding routines, and structure
definitions for a number of SMP commands.

libcam/Makefile: Add smp_all.c to libcam, so that SMP functionality
is available to userland applications.

camcontrol.8,
camcontrol.c: Add smp passthrough support to camcontrol. Several
new subcommands are now available:

'smpcmd' functions much like 'cmd', except that it
allows the user to send generic SMP commands.

'smprg' sends the SMP report general command, and
displays the decoded output. It will automatically
fetch extended output if it is available.

'smppc' sends the SMP phy control command, with any
number of potential options. Among other things,
this allows the user to reset a phy on a SAS
expander, or disable a phy on an expander.

'smpmaninfo' sends the SMP report manufacturer
information and displays the decoded output.

'smpphylist' displays a list of phys on an
expander, and the CAM devices attached to those
phys, if any.

cam.h,
cam.c: Add a status value for SMP errors
(CAM_SMP_STATUS_ERROR).

Add a missing description for CAM_SCSI_IT_NEXUS_LOST.

Add support for SMP commands to cam_error_string().

cam_ccb.h: Rename the CAM_DIR_RESV flag to CAM_DIR_BOTH. SMP
commands are by nature bi-directional, and we may
need to support bi-directional SCSI commands later.

Add the XPT_SMP_IO CCB. Since SMP commands are
bi-directional, there are pointers for both the
request and response.

Add a fill routine for SMP CCBs.

Add the XPT_GDEV_ADVINFO CCB. This is currently
used to fetch cached page 0x83 data from the
transport later, but is extensible to fetch many
other types of data.

cam_periph.c: Add support in cam_periph_mapmem() for XPT_SMP_IO
and XPT_GDEV_ADVINFO CCBs.

cam_xpt.c: Add support for executing XPT_SMP_IO CCBs.

cam_xpt_internal.h: Add fields for VPD pages 0x00 and 0x83 in struct
cam_ed.

scsi_all.c: Add scsi_get_sas_addr(), a function that parses
VPD page 0x83 data and pulls out a SAS address.

scsi_all.h: Add VPD page 0x00 and 0x83 structures, and a
prototype for scsi_get_sas_addr().

scsi_pass.c: Add support for mapping buffers in XPT_SMP_IO and
XPT_GDEV_ADVINFO CCBs.

scsi_xpt.c: In the SCSI probe code, first ask the device for
VPD page 0x00. If any VPD pages are supported,
that page is required to be implemented. Based on
the response, we may probe for the serial number
(page 0x80) or device id (page 0x83).

Add support for the XPT_GDEV_ADVINFO CCB.

sys/conf/files: Add smp_all.c.

mps.c: Add support for passing in a uio in mps_map_command(),
so we can map a S/G list at once.

Add support for SMP passthrough commands in
mps_data_cb(). SMP is a special case, because the
first buffer in the S/G list is outbound and the
second buffer is inbound.

Add support for warning the user if the busdma code
comes back with more buffers than will work for the
command. This will, for example, help the user
determine why an SMP command failed if busdma comes
back with three buffers.

mps_pci.c: Add sys/uio.h.

mps_sas.c: Add the SAS address and the parent handle to the
list of fields we pull from device page 0 and cache
in struct mpssas_target. These are needed for SMP
passthrough.

Add support for the XPT_SMP_IO CCB. For now, this
CCB is routed to the addressed device if it supports
SMP, or to its parent if it does not and the parent
does. This is necessary because CAM does not
currently support SMP-only nodes in the topology.

Make SMP passthrough support conditional on
__FreeBSD_version >= 900026. This will make it
easier to MFC this change to the driver without
MFCing the CAM changes as well.

mps_user.c: Un-staticize mpi_init_sge() so we can use it for
the SMP passthrough code.

mpsvar.h: Add a uio and iovecs into struct mps_command for
SMP passthrough commands.

Add a cm_max_segs field to struct mps_command so
that we can warn the user if busdma comes back with
too many segments.

Clear the cm_reply when a command gets freed. If
it is not cleared, reply frames will eventually get
freed into the pool multiple times and corrupt the
pool. (This fix is from scottl.)

Add a prototype for mpi_init_sge().

sys/param.h: Bump __FreeBSD_version to 900026 for the for the
inclusion of the XPT_GDEV_ADVINFO and XPT_SMP_IO
CAM CCBs.

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Implement the usage of Report Luns as part of SCSI probing for SCP3 or
better devices. This can be disabled on a per-device basis using quirks as
well.

This also handles the case where there is actually no connected LUN 0
(which can definitely be the case for storage arrays).

Reviewed by: scsi@
MFC after: 1 month

Add a new primitive, XPT_SCAN_TGT, to cover the range between scanning a
whole bus (XPT_SCAN_BUS) and a single lun on that bus (XPT_SCAN_LUN).

It's less resource comsumptive than scanning a whole bus when the
caller knows only one target has changes.

Reviewed by: scsi@
Sponsored by: Panasas
MFC after: 1 month

MFC: r208348

Remove unnecessary headers which were inherited from cam_xpt.c without
reason.

Reviewed by: mav

Remove unnecessary headers which were inherited from cam_xpt.c without
reason.

Reviewed by: mav
MFC after: 3 days

MFC r204220, r204353:
Virtualize transport part of periph announcement.

Virtualize transport part of periph announcement.

MFC r203108:
Large set of CAM improvements:
- Unify bus reset/probe sequence. Whenever bus attached at boot or later,
CAM will automatically reset and scan it. It allows to remove duplicate
code from many drivers.
- Any bus, attached before CAM completed it's boot-time initialization,
will equally join to the process, delaying boot if needed.
- New kern.cam.boot_delay loader tunable should help controllers that
are still unable to register their buses in time (such as slow USB/
PCCard/ CardBus devices), by adding one more event to wait on boot.
- To allow synchronization between different CAM levels, concept of
requests priorities was extended. Priorities now split between several
"run levels". Device can be freezed at specified level, allowing higher
priority requests to pass. For example, no payload requests allowed,
until PMP driver enable port. ATA XPT negotiate transfer parameters,
periph driver configure caching and so on.
- Frozen requests are no more counted by request allocation scheduler.
It fixes deadlocks, when frozen low priority payload requests occupying
slots, required by higher levels to manage theit execution.
- Two last changes were holding proper ATA reinitialization and error
recovery implementation. Now it is done: SATA controllers and Port
Multipliers now implement automatic hot-plug and should correctly
recover from timeouts and bus resets.
- Improve SCSI error recovery for devices on buses without automatic sense
reporting, such as ATAPI or USB. For example, it allows CAM to wait, while
CD drive loads disk, instead of immediately return error status.
- Decapitalize diagnostic messages and make them more readable and sensible.
- Teach PMP driver to limit maximum speed on fan-out ports.
- Make boot wait for PMP scan completes, and make rescan more reliable.
- Fix pass driver, to return CCB to user level in case of error.
- Increase number of retries in cd driver, as device may return several UAs.

MFp4: Large set of CAM inprovements.

- Unify bus reset/probe sequence. Whenever bus attached at boot or later,
CAM will automatically reset and scan it. It allows to remove duplicate
code from many drivers.
- Any bus, attached before CAM completed it's boot-time initialization,
will equally join to the process, delaying boot if needed.
- New kern.cam.boot_delay loader tunable should help controllers that
are still unable to register their buses in time (such as slow USB/
PCCard/ CardBus devices), by adding one more event to wait on boot.
- To allow synchronization between different CAM levels, concept of
requests priorities was extended. Priorities now split between several
"run levels". Device can be freezed at specified level, allowing higher
priority requests to pass. For example, no payload requests allowed,
until PMP driver enable port. ATA XPT negotiate transfer parameters,
periph driver configure caching and so on.
- Frozen requests are no more counted by request allocation scheduler.
It fixes deadlocks, when frozen low priority payload requests occupying
slots, required by higher levels to manage theit execution.
- Two last changes were holding proper ATA reinitialization and error
recovery implementation. Now it is done: SATA controllers and Port
Multipliers now implement automatic hot-plug and should correctly
recover from timeouts and bus resets.
- Improve SCSI error recovery for devices on buses without automatic sense
reporting, such as ATAPI or USB. For example, it allows CAM to wait, while
CD drive loads disk, instead of immediately return error status.
- Decapitalize diagnostic messages and make them more readable and sensible.
- Teach PMP driver to limit maximum speed on fan-out ports.
- Make boot wait for PMP scan completes, and make rescan more reliable.
- Fix pass driver, to return CCB to user level in case of error.
- Increase number of retries in cd driver, as device may return several UAs.

MFC r199178:
- Move tagged queueing control from ADA to ATA XPT. It allows to control
device command queue length correctly. First step to support < 32 tags.
- Limit queue for non-tagged devices by 2 slots for ahci(4) and siis(4).
- Implement quirk matching for ATA devices.
- Move xpt_schedule_dev_sendq() from header to source file.
- Move delayed queue shrinking to the more expected place - element freeing.
- Remove some SCSIsms in ATA.

MFC r198748, r198782:
Fix reference counting bug, when device unreferenced before
invalidated. To do it, do not handle validity flag as another
reference, but explicitly modify reference count each time flag is
modified.
The async callback could free the device. If it is a broadcast async,
it doesn't hold device reference, so take our own reference.

MFC r198708:
- Reduce code duplication in ATA XPT and PMP driver.
- Move PIO size setting from ada driver to ATA XPT. It is XPT business
to negotiate transfer details. ada driver is now stateless.
- Report PIO size to SIM. It is required for correct PATA SIM operation.
- Tune PMP scan timings. It workarounds some problems with SiI.
- If reset hapens during PMP initialization - restart it.
- Introduce early-initialized periph drivers, which are used during initial
scan process. Use it for xpt, probe, aprobe and pmp. It gives pmp chance
to finish scan before mountroot and numerate devices in right order.

MFC r198382, r198385:
Replace most of priority numbers with defines. No logical changes.

MFp4:
- Move tagged queueing control from ADA to ATA XPT. It allows to control
device command queue length correctly. First step to support < 32 tags.
- Limit queue for non-tagged devices by 2 slots for ahci(4) and siis(4).
- Implement quirk matching for ATA devices.
- Move xpt_schedule_dev_sendq() from header to source file.
- Move delayed queue shrinking to the more expected place - element freeing.
- Remove some SCSIsms in ATA.

MFp4:
Fix reference counting bug, when device unreferenced before then
invalidated. To do it, do not handle validity flag as another
reference, but explicitly modify reference count each time flag is
modified.

Discovered by: thompsa

MFp4:
- Reduce code duplication in ATA XPT and PMP driver.
- Move PIO size setting from ada driver to ATA XPT. It is XPT business
to negotiate transfer details. ada driver is now stateless.
- Report PIO size to SIM. It is required for correct PATA SIM operation.
- Tune PMP scan timings. It workarounds some problems with SiI.
- If reset hapens during PMP initialization - restart it.
- Introduce early-initialized periph drivers, which are used during initial
scan process. Use it for xpt, probe, aprobe and pmp. It gives pmp chance
to finish scan before mountroot and numerate devices in right order.

Replace most of priority numbers with defines. No logical changes.

Change xpt_scan_bus to scsi_scan_bus and xpt_scan_lun to scsi_scan_lun
in comments and printfs to match new function names after refacoring.

Approved by: re

Fix leaks in probestart, probedone, and scsi_scan_bus. Also free
page_list using the matching malloc type for the allocation.

Approved by: re
Reviewed by: scottl [1]
MFC after: 1 week

[1] Original patch was against xpt_cam.c, prior to the cam refactoring.

Separate the parallel scsi knowledge out of the core of the XPT, and
modularize it so that new transports can be created.

Add a transport for SATA

Add a periph+protocol layer for ATA

Add a driver for AHCI-compliant hardware.

Add a maxio field to CAM so that drivers can advertise their max
I/O capability. Modify various drivers so that they are insulated
from the value of MAXPHYS.

The new ATA/SATA code supports AHCI-compliant hardware, and will override
the classic ATA driver if it is loaded as a module at boot time or compiled
into the kernel. The stack now support NCQ (tagged queueing) for increased
performance on modern SATA drives. It also supports port multipliers.

ATA drives are accessed via 'ada' device nodes. ATAPI drives are
accessed via 'cd' device nodes. They can all be enumerated and manipulated
via camcontrol, just like SCSI drives. SCSI commands are not translated to
their ATA equivalents; ATA native commands are used throughout the entire
stack, including camcontrol. See the camcontrol manpage for further
details. Testing this code may require that you update your fstab, and
possibly modify your BIOS to enable AHCI functionality, if available.

This code is very experimental at the moment. The userland ABI/API has
changed, so applications will need to be recompiled. It may change
further in the near future. The 'ada' device name may also change as
more infrastructure is completed in this project. The goal is to
eventually put all CAM busses and devices until newbus, allowing for
interesting topology and management options.

Few functional changes will be seen with existing SCSI/SAS/FC drivers,
though the userland ABI has still changed. In the future, transports
specific modules for SAS and FC may appear in order to better support
the topologies and capabilities of these technologies.

The modularization of CAM and the addition of the ATA/SATA modules is
meant to break CAM out of the mold of being specific to SCSI, letting it
grow to be a framework for arbitrary transports and protocols. It also
allows drivers to be written to support discrete hardware without
jeopardizing the stability of non-related hardware. While only an AHCI
driver is provided now, a Silicon Image driver is also in the works.
Drivers for ICH1-4, ICH5-6, PIIX, classic IDE, and any other hardware
is possible and encouraged. Help with new transports is also encouraged.

Submitted by: scottl, mav
Approved by: re