MFC r360564: Cleanup LUN addition/removal.

- Make ctl_add_lun() synchronous. Asynchronous addition was used by
Copan's proprietary code long ago and never for upstream FreeBSD.
- Move LUN enable/disable calls from backends to CTL core.
- Serialize LUN modification and partially removal to avoid double frees.
- Slightly unify backends code.

Redo r360682, now with only a minimal fix for misleading indentation:

MFC r333465 (partial, by lwhsu):

Fix build for platforms using GCC:

[omitted] - Remove unused or dead store variable
[omitted] - Remove unused function ctl_copyin_alloc
- Add missing curly brackets, this seems a regression in r287720

Reviewed by: jhibbits
Differential Revision: https://reviews.freebsd.org/D15383

Revert r360682, as it does not compile on stable/11 (will commit a
minimal indentation fix instead):

Un-MFC r333465 (by lwhsu):

Fix build for platforms using GCC:

- Remove unused or dead store variable
- Remove unused function ctl_copyin_alloc
- Add missing curly brackets, this seems a regression in r287720

Reviewed by: jhibbits
Differential Revision: https://reviews.freebsd.org/D15383

MFC r333465 (by lwhsu):

Fix build for platforms using GCC:

- Remove unused or dead store variable
- Remove unused function ctl_copyin_alloc
- Add missing curly brackets, this seems a regression in r287720

Reviewed by: jhibbits
Differential Revision: https://reviews.freebsd.org/D15383

MFC r327239 (by manu): ctl: Correct comment in ctl_worker_thread

The incoming queue is handled before the RtR one.
No functional change.

MFC r344636: Refactor command ordering/blocking mechanism in CTL.

Replace long per-LUN queue of blocked commands, scanned on each command
completion and sometimes even twice, causing up to O(n^^2) processing cost,
by much shorter per-command blocked queues, scanned only when respective
command completes, and check only commands before the previous blocker,
reducing cost to O(n).

While there, unblock aborted commands to make them "complete" ASAP to be
removed from the OOA queue and so not waste time ordering other commands
against them. Aborted commands that were not sent to execution yet should
have no visible side effects, so this is safe and easy optimization now,
comparing to commands already in processing, which are a still pain.

Together those two optimizations should fix quite pathological case, when
due to backend slowness CTL accumulated many thousands of blocked requests,
partially aborted by initiator and so supposedly not even existing, but
still wasting CTL CPU time.

MFC r344586: Scrap some debug printf's, unused for years.

MFC r344743: Reduce CTL threads priority to about PUSER.

Since in most configurations CTL serves as network service, we found
that this change improves local system interactivity under heavy load.
Priority of main threads is set slightly higher then worker taskqueues
to make them quickly sort incoming requests not creating bottlenecks,
while plenty of worker taskqueues should be less sensitive to latency.

MFC r344489: Free some space in struct ctl_io_hdr for better use.

- Collapse original_sc and serializing_sc fields into one, since they
are never used simultanously, we have only one local I/O and one remote.

- Move remote_sglist and local_sglist fields into CTL_PRIV_BACKEND,
since they are used only on Originating SC in XFER mode, where requests
don't ever reach backends, so we can reuse backend's private storage.

MFC r325517, r325554

r325517:
ctl: Make max_luns and max_ports tunable variables instead of hardcoded
defines.

Reviewed by: trasz (earlier version), bapt (earlier version), bcr (manpages)
Sponsored by: Gandi.net
Differential Revision: https://reviews.freebsd.org/D12836

r325554:
ctl(4): Insert a new line after a sentence-ending full stop.

Reported by: bjk
Sponsored by: Gandi.net
X-MFC-With: r325517

MFC r324123: Add sysctl/tunable for maximal request time.

MFC r317370: Change ctl_free_lun() locking.

This fixes potential callout_drain() sleep under non-sleepable lock.

PR: 218167

MFC r316653: Fix few minor issues found by Clang Analyzer.

MFC r312622:

Add SCSI descriptors for USB Mass Storage.

Sponsored by: The FreeBSD Foundation

MFC r314338: Polish handling of different reset flavours.

The biggest change is that ctl_remove_initiator() now generates I_T NEXUS
LOSS event, cleaning part of LUs state related to the initiator.

MFC r314496: Add check missed in r314257.

MFC r314387: Make ctl_queue_sense() not sleep.

It may be called in non-sleepable frontend context.

MFC r314257: Add reporting SAS protocol, in case we ever have one.

MFC r314255: Reenable CTL_WITH_CA, optimizing it for lower memory usage.

This code was disabled due to its high memory usage. But now we need this
functionality for cfumass(4) frontend, since USB MS BBB transport does not
support autosense.

MFC 313895:

------------------------------------------------------------------------
r313895 | ken | 2017-02-17 13:15:27 -0700 (Fri, 17 Feb 2017) | 9 lines

Make ctl(4) build with CTL_IO_DELAY defined.

sys/cam/ctl/ctl.c:
In ctl_datamove(), inside CTL_IO_DELAY, add a lun variable and fill
it in before trying to dereference it.

Sponsored by: Spectra Logic

------------------------------------------------------------------------

MFC r312603: Add initial support for CTL module unloading.

It is only a first step and not perfect, but better then nothing.
The main blocker is CAM target frontend, that can not be unloaded,
since CAM does not have mechanism to unregister periph driver now.

MFC r312348: Remove writing 'residual' field of struct ctl_scsiio.

This field has no practical use and never readed. Initiators already
receive respective residual size from frontends. Removed field had
different semantics, which looks useless, and was never passed through
by any frontend.

While there, fix kern_data_resid field support in case of HA, missed in
r312291.

MFC r312291, r312669:
Make CTL frontends report kern_data_resid for under-/overruns.

It seems like kern_data_resid was never really implemented. This change
finally does it. Now frontends update this field while transferring data,
while CTL/backends getting it can more flexibly handle the result.
At this point behavior should not change significantly, still reporting
errors on write overrun, but that may be changed later, if we decide so.

CAM target frontend still does not properly handle overruns due to CAM API
limitations. We may need to add some fields to struct ccb_accept_tio to
pass information about initiator requested transfer size(s).

MFC r311873: Fix malloc(M_WAITOK) under mutex, introduced at r311787.

MFC r311804: Rewrite CTL statistics in more simple and scalable way.

Instead of collecting statistics for each combination of ports and logical
units, that consumed ~45KB per LU with present number of ports, collect
separate statistics for every port and every logical unit separately, that
consume only 176 bytes per each single LU/port. This reduces struct
ctl_lun size down to just 6KB.

Also new IOCTL API/ABI does not hardcode number of LUs/ports, and should
allow handling of very large quantities.

Old API is still enabled in stable branches for compatibility reasons.

MFC r311787: Allocate memory for prevent flags only for removable LUs.

This array takes 64KB of RAM now, that was more then half of struct ctl_lun
size. If at some point we support more ports, this may need another tune.

MFC r311680: Make CTL_GETSTATS ioctl return partial data if buffer is small.

MFC r310778, r310782: Improve use of I/O's private area.

- Since I/Os are allocates from per-port pools, make allocations store
pointer to CTL softc there, and use it where needed instead of global.
- Created bunch of helper macros to access LUN, port and CTL softc.

MFC r310646: Do not update "saved" mode page on every MODE SELECT.

We do not have non-volatile memory to really save those values, so we
neither report nor support this capability. Also saved mode pages are
not replicated between HA peers now.

MFC r310644: Fix/synchronize field types in struct ctl_modepage_header.

MFC r310640, r310643:
Add support for revert to defaults (RTD) bit in MODE SELECT.

MFC r310635: Decouple limits on number of LUNs per port and LUs per CTL.

Those two values are not directly related, so make them independent.
This does not change any limits immediately, but makes number of LUNs
per port controllable via tunable/sysctl kern.cam.ctl.lun_map_size.
After this change increasing CTL_MAX_LUNS should be pretty cheap,
and even making it tunable should be easy.

MFC r310575: Fix improperly used nexus.targ_lun.

MFC r310555: Some random code cleaning.

- Reduce indentation.
- Remove extra braces.
- Add few missing savety checks.

MFC r310539: Remove CTL_MAX_LUNS from places where it is not required.

MFC r310524: Improve length handling when writing sense data.

- Allow maximal sense size limitation via Control Extension mode page.
- When sense size limited, include descriptors atomically: whole or none.
- Set new SDAT_OVFL bit if some descriptors don't fit the limit.
- Report real written sense length instead of static maximal 252 bytes.

MFC r310390: Add support for REPD bit in RSTMF command.

We have no real timeout values to report there, but its better then error.

r310389:
Fix REPORT SUPPORTED OPERATION CODES for READ/WRITE BUFFER commands.

Specifications require MODE parameter of those commands to be treated
the same as SERVICE ACTION parameter of other commands.

MFC r310373:
Add support for REPORTING OPTIONS == 3 in REPORT SUPPORTED OPERATION CODES.

MFC r310366: Add support for SITUA bit in Logical Block Provisioning mode page.

VMware tries to enable this bit to avoid multiple threshold notifications
in case of multiple initiators connected to the same LUN. Unfortunately
their code sends MODE SELECT(6) request with parameter length hardcoded
for the page without any thresholds. Since we have four threshold and our
page is bigger, this attempt fails, that is correct in my understanding.
So all we can do about this now is to report proper error code and hope
VMware fix their code one day.

MFC r310356: Add support for locally assigned RFC 4122 UUID LUN identifiers.

MFC r310339: Bump specifications support to SAM-6/SPC-5.

MFC r310285:
When reporting "Logical block address out of range" error, report the LBA
in sense data INFORMATION field.

MFC r310272: Add new bits into Extended Inquiry VPD page.

MFC r310266: Add support for NUAR bit in Control mode page.

MFC r310265: Add set of macros to simplify code access to mode pages fields.

MFC r310259:
Following SPC-5, make REQUEST SENSE report "Logical unit not supported"
in returned parameter data for not accessible LUNs.

MFC r310257: Improve support for informational exceptions.

While CTL still has no real events to report in this way (like SMART),
it is possible to trigger false event by manually setting TEST bit in
Informational Exceptions Control mode page, that can be useful for
initiator testing. This code supports all flavours of IE reporting:
UNIT ATTENTION, RECOVERED ERROR and NO SENSE sense keys, REQUEST SENSE
command and Informational Exceptions log page.

MFC r308250:

Check for lengths being <= 0. Note that this interface can only
be accessed by root. It uses unsigned ints instead of size_t
to preserve the ABI.

PR: 207627

MFC r307374: Add LU option to control reported provisioning type.

MFC r307350: Add LUN options to limit UNMAP and WRITE SAME sizes.

CTL itself has no limits on on UNMAP and WRITE SAME sizes. But depending
on backends large requests may take too much time. To avoid that new
configuration options allow to hint initiator maximal sizes it should not
exceed.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

sys/cam: spelling fixes in comments.

No functional change.

cam: unsign some types to match their definitions and avoid overflows.

numpatterns is u_int.

ctl:
CTL_NUM_MODE_PAGES comes from sizeof().
In struct:ctl_scsiio, kern_sg_entries is uint32_t.

MFC after: 2 weeks

Plug memory leak in ctl(4) when ctl_copyin_args() is called with a non-
null terminated ASCII string.

PR: 207626
Submitted by: cturt@hardenedbsd.org
MFC after: 2 days

Remove stray semicolons from the iSCSI code.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Convert sys/cam to use make_dev_s().

Reviewed by: hps, jhb
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks
Differential revision: https://reviews.freebsd.org/D4746

Set DS flag, required for LPB log page by spec.

MFC after: 1 week

Fix panic when trying to sort unsupported command in OOA queue.

Handle unsupported commands as not conflicting/blocking.

Add NULL check to make Coverity happy.

Modify target port groups logic in CTL.

- Introduce "ha_shared" port option, which being set to "on" moves the
port into separate port group, shared between HA nodes. This allows to
better handle cases when iSCSI portals are bound to CARP address that can
dynamically move between nodes. Some initiators (at least VMware) don't
detect that after iSCSI reconnect they've attached to different SCSI port
from different port group, that totally breakes ALUA status parsing.
In theory, I believe, it should be enough to have different iSCSI portal
group tags on different nodes to make initiators detect this condition,
but it seems like VMware ignores those values, and even full LUN retaste
forced by UA does not help.
- Make CTL report up to three port groups: 1 -- non-HA mode or ports
with "ha_shared" option set, 2 -- HA node 1, 3 -- HA node 2.
- Report Transitioning state for all port groups when HA interlink is
connected, but neither of nodes is primary for the LUN.

MFC after: 2 weeks

Add two more KASSERTs.

Don't try to replicate mode pages not present on this device.

MFC after: 3 days

Give CTL support for PIM_EXTLUNS when talking to CAM.

CTL itself still lives in flat LUN space, but it can generate extended
numbers if CAM SIM reports such capability.

Make some panic strings mode informative.

Implement SPC-3 exceptions to SPC-2 RESERVE and RELEASE behavior.

Unify PR variable names to reduce confusion.

Use proper STAILQ_* macros where possible.

Really implement PREVENT ALLOW MEDIUM REMOVAL command.

Report that we can read all flavours of DVD. Why not?

Add CD/DVD Capabilities and Mechanical Status Page.

This page is obsolete since MMC-4, but still used by some software.

Umplement media load/eject support for removable devices.

In case of block backend eject really closes the backing store, while
load tries to open it back. Failed store open is reported as no media.

Add to CTL initial support for CDROMs and removable devices.

Relnotes: yes

Remove concept of control device.

Remove few more unused variables.

Remove some duplicate, legacy, dead and questionable code.

Properly lock LUN in ctl_failover_lun().

Constify ctl_serialize_table.

Remove some dead code found by Clang analyzer.

Switch I/O time accounting from system time to uptime.

While there, make num_dmas accounted independently of CTL_TIME_IO.

Collect DMA statistics on secondary HA node.

Make HA handle datamove and done in a universal way, using port methods.

Now from primary node point of view requests transferred from secondary
node should look almost normal and always have valid port.

Remove some control_softc references.

Allow WRITE SAME with NDOB bit set but without UNMAP.

This combination was originally forbidden, but allowed at spc4r3.

Add new report types to REPORT LUNS command.

This is only for completeness, since we have nothing new to report there.

Update WRITE ATOMIC(16) support to sbc4r8 draft.

This is only a cosmetic change. We still don't support atomic boundary
field in the CDB, but at least now we do it formally.

Add support for READ BUFFER(16) command.

Synchronize mode pages between HA peers.

We allow to modify only few fields in mode pages now, but still it is
not good if they unexpectedly change during failover. Also this fixes
reporting of "Mode parameters changed" UAs on secondary node.

Make HA peers announce their parameters on connect.

HA protocol requires strict version, parameters and configuration match.
Differences there may cause full set of problems up to kernel panic.
To avoid that, validate peer parameters on connect, and abort connection
immediately if some mismatch detected.

Add support for Control extension mode page.

Always execute REPORT LUNS and REQUEST SENSE commands locally.

REPORT LUNS command is more related to target rather then specific LUN.
This node may be primary for LUNs for some reason unknown to another,
and command forwarded to another node won't be able to report them.

REQUEST SENSE is related to LUN, but in our implementation it reports
only UAs and CAs, that are stored locally rather then on primary node.

Split two command flags with different meaning.

This is only a cosmetical change.

Fix memory corruption when >128K transferred through HA link.

While there, do some cleaning.

Mark I/Os with DMA flag while moving data through the HA link.

Relax serseq option operation for reads.

Previously, with serseq enabled, next command was unblocked only after
previous completed. With this change, for read operations, next command
is unblocked as soon as last media read completed. This is important
for frontends that actually wait for data move completion (like camtgt),
or when data are moved through the HA link, or especially when both.

Kill HA link and shutdown the threads on shutdown.

Replicate initiators WWPNs and names between HA peers.

Replicate port->init_devid to HA peer.

When reporting TPT UA, report which of thresholds was reached.

Report proper medium error code for VERIFY commands.

Close potential race between datamove and HA failover.

Implement data/status aggregation for secondary HA node.

For short read requests this reduces latency by 30%, reporting command
completion after two interlink roundtrips instead of full three.

Remove CTL_PRIV_LBA_LEN from HA messages.

Previously it was used for statistics, but now just a 16 extra bytes.

Implement QUERY TASK, QUERY TASK SET and QUERY ASYNC EVENT.

Now we support most of SAM-5 task management.

Improve read-only support.

Reannounce port to HA peer if LUN map changed after online.

Report INQUIRY DATA HAS CHANGED for related LUNs on port on-/offline.

Report CTL_UA_LUN_CHANGE on LUN map change.

Make TARGET RESET respect LUN mapping.

Add HA support for CTL_TASK_I_T_NEXUS_RESET.

Fix false CTL_UA_RES_RELEASE on secondary HA node.

Some HA polishing.

Make ctld restart on primary HA node less invasive for secondary.

Reimplement CTL High Availability.

CTL HA functionality was originally implemented by Copan many years ago,
but large part of the sources was never published. This change includes
clean room implementation of the missing code and fixes for many bugs.

This code supports dual-node HA with ALUA in four modes:
- Active/Unavailable without interlink between nodes;
- Active/Standby with second node handling only basic LUN discovery and
reservation, synchronizing with the first node through the interlink;
- Active/Active with both nodes processing commands and accessing the
backing storage, synchronizing with the first node through the interlink;
- Active/Active with second node working as proxy, transfering all
commands to the first node for execution through the interlink.

Unlike original Copan's implementation, depending on specific hardware,
this code uses simple custom TCP-based protocol for interlink. It has
no authentication, so it should never be enabled on public interfaces.

The code may still need some polishing, but generally it is functional.

Relnotes: yes
Sponsored by: iXsystems, Inc.

Remove unused target and initiator IDs.

Disable CTL_IO_DELAY feature.

It is too developer-oriented to be enabled by default.

Move setting of media parameters inside open routines.

This is preparation for possibility to open/close media several times
per LUN life cycle. While there, rename variables to reduce confusion.
As additional bonus this allows to open read-only media, such as ZFS
snapshots.

Small UA cleanup.

Fix copy-paste bug introduced in r275458.

MFC after: 3 days

Make most of port methods optional and remove bunch of dummies.

Move "ioctl" CAM frontend into separate file.

It has nothing to share with too huge ctl.c other then device descriptor,
but even that may be counted as design error that may be fixed later.
At some point we may even want to have several ioctl ports.

Drop "internal" CTL frontend.

Its idea was to be a simple initiator and execute several commands from
kernel level, but FreeBSD never had consumer for that functionality,
while its implementation polluted many unrelated places..

Remove verbose CTL messages.

Reporting SCSI errors to console is often useless, pollutes logs and may
affect performance. For debugging there is kern.cam.ctl.debug sysctl

MFC after: 1 week

Add more ifdefs to fix build with GCC after r286406.

Wrap some unused functions with notyet, it is necessary to be able to
build the modules/ctl directly.
Remove a dead MALLOC_DEFINE.

Differential Revision: D3329
Reviewed by: mav
Sponsored by: gandi.net

Pass SYNCHRONIZE CACHE command parameters to backends.

At this point IMMED flag is translated to MNT_NOWAIT flag of VOP_FSYNC(),
hoping that file system implements that (ZFS seems doesn't).

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Fix REPORT LUNS command output for the case when same LUN mapped to same
port several times. While it is unusual configuration, it is not illegal.

MFC after: 1 week

Bring per-port LUN enable/disable code up to date:
- remove last remnants of never implemented multiple targets support;
- implement missing support for LUN mapping in this area.

Due to existing locking constraints LUN mapping code is practically
unlocked at this point. Hopefully it is not racy enough to live until
somebody get idea how to call sleeping fronend methods under lock also
taken by the same frontend in non-sleepable context. :(

Remove some dead and duplicate LUN enabling code.

Fix wrong function name in error message.

MFC after: 3 days

Remove some #if 0'ed code that apparently confuses cscope.

Requested by: Peter Xu <xzpeter@gmail.com>
MFC after: 3 days

Do not dereference NULL clearing UA that was not set in r277917.

MFC after: 3 days

Add support for General Statistics and Performance log page.

CTL already collects most of statistics reported there, so why not.

MFC after: 2 weeks

Do not abort already aborted tasks.

This fixes abort of new tasks with the same tags as previously aborted,
but still remaining on the queue.

MFC after: 1 week

CTL LUN mapping rewrite.

Replace iSCSI-specific LUN mapping mechanism with new one, working for any
ports. By default all ports are created without LUN mapping, exposing all
CTL LUNs as before. But, if needed, LUN mapping can be manually set on
per-port basis via ctladm. For its iSCSI ports ctld does it via ioctl(2).
The next step will be to teach ctld to work with FibreChannel ports also.

Respecting additional flexibility of the new mechanism, ctl.conf now allows
alternative syntax for LUN definition. LUNs can now be defined in global
context, and then referenced from targets by unique name, as needed. It
allows same LUN to be exposed several times via multiple targets.

While there, increase limit for LUNs per target in ctld from 256 to 1024.
Some initiators do not support LUNs above 255, but that is not our problem.

Discussed with: trasz
MFC after: 2 weeks
Relnotes: yes
Sponsored by: iXsystems, Inc.

Improve SCSI Extended Inquiry VPD page (0x86) support.

sys/cam/scsi/scsi_all.h:
In struct scsi_extended_inquiry_data:
- Increase the length field to 2 bytes, as it is 2 bytes in SPC-4.
- Add bit definitions for the various Activiate Microcode actions.
- Add the Sequential Access Logical Block Protection support bit,
since we need that in the sa(4) driver. (For modifications
that will come later.)
- Add definitions for the various Multi I_T Nexus Microcode
Download modes.

sys/cam/ctl/ctl.c:
As of SPC-4, a single report of "REPORTED LUNS DATA HAS CHANGED"
is to be given per I_T nexus. Once it is reported, the unit
attention condition should be cleared for all LUNS attached to
an I_T nexus.

Previously that only happened when a REPORT LUNS command was
processed.

This behavior may be different (according to SAM-5) when the
UA_INTLCK_CTRL bits are non-zero in the control mode page but
CTL does not currently support that.

So, in view of the spec, whenever we report a LUN inventory
change unit attention, clear it on all LUNs for that
particular I_T nexus.

Add a new function, ctl_clear_ua() that will clear a unit
attention on all LUNs for the given I_T nexus.

One field in the extended inquiry data that we could potentially
report at some point is the maximum supported sense data length.
To do that, we would the SIM to report (via path inquiry
perhaps) how much sense data it is able to send.

Add comments to explain some of the bits that are set in the
Extended Inquiry VPD page.

Add a few comments to make it more clear which functions handle
various VPD pages.

Sponsored by: Spectra Logic
MFC after: 1 week

Don't count requests with status sent as overlapping.

While those requests are still in target OOA queue, for initiator they are
already completed, so tags can be reused.

MFC after: 1 week

Hide block device VPD pages for non-block devices.

MFC after: 2 weeks

Report initiator id in portlist XML in more formalized way.

MFC after: 3 days

Replace ctl_min() macro with MIN().

MFC after: 1 week

Constify some static data.

MFC after: 2 weeks

Reduce number of places where global control_softc is used.

At some point we may want to have several CTL instances, and that is not
really impossible.

MFC after: 2 weeks

Pass real optimal transfer size supported by backend.

For files and ZVOLs that is 1MB now, not 128K.

MFC after: 1 week

Add configuration options to override physical and UNMAP blocks geometry.

While in most cases CTL should correctly fetch those values from backing
storages, there are some initiators (like MS SQL), that may not like large
physical block sizes, even if they are true. For such cases allow override
fetched values with supported ones (like 4K).

MFC after: 1 week

Count consecutive read requests as blocking in CTL for files and ZVOLs.

Technically read requests can be executed in any order or simultaneously
since they are not changing any data. But ZFS prefetcher goes crasy when
it receives consecutive requests from different threads. Since prefetcher
works on level of separate blocks, instead of two consecutive 128K requests
it may receive 32 8K requests in mixed order.

This patch is more workaround then a real fix, and it does not fix all of
prefetcher problems, but it improves sequential read speed by 3-4x times
in some configurations. On the other side it may hurt performance if
some backing store has no prefetch, that is why it is disabled by default
for raw devices.

MFC after: 2 weeks

In addition to r275481 allow threshold notifications work without UNMAP.

While without UNMAP support there is not much initiator can do about it,
the administrator still better be notified about the storage overflow.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Swap resource count scopes for used/available space.

Used count should be reported as per-LUN, while available should not.

MFC after: 1 week

Add GET LBA STATUS command support to CTL.

It is implemented for LUNs backed by ZVOLs in "dev" mode and files.
GEOM has no such API, so for LUNs backed by raw devices all LBAs will
be reported as mapped/unknown.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Unify function names after r275458.

MFC after: 1 month

Do not pre-allocate UNIT ATTENTIONs storage for every possible initiator.

Abusing ability of major UAs cover minor ones we may not account UAs for
inactive ports. Allocate UAs storage for port and start accounting only
after some initiator from that port fetched its first POWER ON OCCURRED.

This reduces per-LUN CTL memory usage from >1MB to less then 100K.

MFC after: 1 month

Remove some unused code.

Do not pre-allocate reservation keys memory for every possible initiator.

In configurations with many ports, like iSCSI, each LUN is typically
accessed only by limited subset of ports. Allocating that memory on
demand allows to reduce CTL memory usage from 5.3MB/LUN to 1.3MB/LUN.

MFC after: 1 month

Plug memory leaks on UNMAP and XCOPY with invalid parameters.

MFC after: 1 week

Convert persis_offset from global variable to softc field.

Reduce code duplication by creating ctl_set_res_ua() helper.

Removed unused variable and unify some names.

Coalesce last data move and command status for read commands.

Make CTL core and block backend set success status before initiating last
data move for read commands. Make CAM target and iSCSI frontends detect
such condition and send command status together with data. New I/O flag
allows to skip duplicate status sending on later fe_done() call.

For Fibre Channel this change saves one of three interrupts per read command,
increasing performance from 126K to 160K IOPS. For iSCSI this change saves
one of three PDUs per read command, increasing performance from 1M to 1.2M
IOPS.

MFC after: 1 month
Sponsored by: iXsystems, Inc.

Use ctl_set_success() instead of direct inlining.

MFC after: 1 week

Replace home-grown CTL IO allocator with UMA.

Old allocator created significant lock congestion protecting its lists
of preallocated I/Os, while UMA provides much better SMP scalability.
The downside of UMA is lack of reliable preallocation, that could guarantee
successful allocation in non-sleepable environments. But careful code
review shown, that only CAM target frontend really has that requirement.
Fix that making that frontend preallocate and statically bind CTL I/O for
every ATIO/INOT it preallocates any way. That allows to avoid allocations
in hot I/O path. Other frontends either may sleep in allocation context
or can properly handle allocation errors.

On 40-core server with 6 ZVOL-backed LUNs and 7 iSCSI client connections
this change increases peak performance from ~700K to >1M IOPS! Yay! :)

MFC after: 1 month
Sponsored by: iXsystems, Inc.

Remove bunch of unused lun variables.

MFC after: 1 week

Reduce race between LUN destruction and request arrival.

MFC after: 1 week

Log errors for absent LUNs too.

MFC after: 1 week

Partially reconstruct Active/Standby clusting.

In this mode one head is in Active state, supporting all commands, while
another is in Standby state, supporting only minimal LUN discovery subset.

It is still incomplete since Standby state requires reservation support,
which is impossible to do right without having interlink between heads.
But it allows to run some basic experiments.

Handle PREEMPT AND ABORT service action equal to PREEMPT.

With command serialization used in CTL, there are no other commands to abort
when PREEMPT AND ABORT gets to run, so it is practically equal to PREEMPT.

MFC after: 1 week

Synchronize medium rotation rate in legacy Rigid Disk Drive Geometry mode
page with modern Block Device Characteristics VPD page.

MFC after: 1 week

Add to CTL support for logical block provisioning threshold notifications.

For ZVOL-backed LUNs this allows to inform initiators if storage's used or
available spaces get above/below the configured thresholds.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Remove comment obsoleted by r273730.

MFC after: 1 week

Reduce code duplication around Write Exclusive persistent reservation.

While there, allow some more commands to pass persistent reservation.

MFC after: 1 week

Allocate buffer for READ BUFFER/WRITE BUFFER commands on demand.

These commands are rare, but consume additional 256KB RAM per LUN.

MFC after: 1 week

Fix support for LUN flat space addressing.

MFC after: 1 week

Fix printing non-terminated strings in devlist XML.

MFC after: 1 week

Add "rpm" and "formfactor" LUN options to match istgt functionality.

MFC after: 1 week

Add support for 12/16-byte EUI and 16-byte NAA IDs.

MFC after: 1 week

Make VPD 80h (Serial Number) transfer length match serial number length.

MFC after: 1 week

Implement more functional CTL debug logging.

Setting bits in kern.cam.ctl.debug allows to log errors, commands and some
commands data respectively.

MFC after: 1 week

Remove couple Copan's vendor-specific mode pages.

Those pages are highly system-/hardware-specific, the code is incomplete,
and so they hardly can be useful for anybody else.

Some groundwork for later Informational Exceptions support.

This includes support for:
- Read-Write Error Recovery mode page;
- Informational Exceptions Control mode page;
- Logical Block Provisioning mode page;
- LOG SENSE command.

No real Informational Exceptions features yet. This is only a placeholder.

Sponsored by: iXsystems, Inc.

Don't confuse frontend with zero length data moves, just return immediately.

MFC after: 1 week

Add support for READ DEFECT DATA (10/12) commands.

SPC-4 r2 allows to return empty defect list if the list is not supported.
We don't reallu support defect data lists, but this suppresses some errors.

MFC after: 1 week

Improve and document `ctladm portlist` subcommand.

Make this subcommand less FC-specific, reporting target and port addresses
in more generic way. Also make it report list of connected initiators in
unified way, working for both FC and iSCSI, and potentially others.

MFC after: 1 week

Make ctld start even if some LUNs are unable to open backing storage.

Such LUNs will be visible to initiators, but return "not ready" status
on media access commands. If backing storage become available later,
`ctladm modify ...` or `service ctld reload` can trigger its reopen.

Store persistent reservation keys as uint64_t instead of uint8_t[8].

This allows to simplify the code and save 512KB of RAM per LUN (8%)
by removing no longer needed "registered" keys flags.

Implement software (mode page) and hardware (config) write protection.

Add support for WRITE ATOMIC (16) command and report SBC-4 compliance.

Atomic writes are only supported for ZVOLs in "dev" mode. In other cases
atomicity can not be guarantied and so the command is blocked.

Fix length of Extended INQUIRY Data VPD page.

MFC after: 3 days

Do not transfer unneeded training zero bytes in INQUIRY response.

It is an addition to r269631.

Fix page length reported for Block Limits VPD page.

When reporting some major UNIT ATTENTION condition, like POWER ON OCCURRED
or I_T NEXUS LOSS, clear all minor UAs for the LUN, redundant in this case.

All SAM specifications tell that target MAY do it, but libiscsi initiator
seems require it to be done, terminating connection with error if some more
UAs happen to be reported during iSCSI connection.

MFC after: 3 days

Deny ANCHOR flag set without UNMAP flag set in WRITE SAME commands.

Don't try to continue aborted commands if status was not set.

Fix UNMAP stuck if the last block descriptor in the list is empty.

MFC after: 3 days

Simplify legacy reservation handling. Drop it on I_T nexus loss.

Report proper errors codes for unsupported SERVICE ACTION values.

Polish INQUIRY command fields validation.

Fix inverted expression to report block size in mode page block descriptor.

Allow more commands to pass persistent reservation according to SPC-4 r37.

Add support for "no Data-Out Buffer" (NDOB) flag of WRITE SAME (16) command.

Always report that we support REPORT TARGET PORT GROUPS command.

Without clustering support we any way have only one group of permanently
active ports, but that gives us one more supported VMWare feature. ;)

Solaris' Comstar also reports it even when only one port is present.

Implement control over command reordering via options and control mode page.

It allows to bypass range checks between UNMAP and READ/WRITE commands,
which may introduce additional delays while waiting for UNMAP parameters.
READ and WRITE commands are always processed in safe order since their
range checks are almost free.

Add "readcache" and "writecache" LUN options to control default behavior.

Default values are "on". Disabling requires backend to support IO_DIRECT
and IO_SYNC flags respectively, or some alternatives.

Implement range checks between UNMAP and READ/WRITE commands.

Before this change UNMAP completely blocked other I/Os while running.
Now it blocks only colliding ones, slowing down others only due to ZFS
locks collisions.

Sponsored by: iXsystems, Inc.

Add support for Extended INQUIRY Data (0x86) VPD page.

Remove uninitialized and unused variable, reported by Coverity.

CID: 1230015

Fix array overrun, reported by Coverity.

CID: 1229970

Fix couple off-by-one range check errors, reported by Coverity.

CID: 1007837

Fix memory leak on error, reported by Coverity.

CID: 1007773

Fix minor buffer overflow reported by Coverity.

CID: 1006781

Report that DPO and FUA bits are supported after r271311.

Add support for Mode Page Policy (0x87) VPD page.

Improve cache control support, including DPO/FUA flags and the mode page.

At this moment it works only for files and ZVOLs in device mode since BIOs
have no respective respective cache control flags (DPO/FUA).

MFC after: 1 month
Sponsored by: iXsystems, Inc.

Reduce reported additional INQUIRY data length.

sizeof(struct scsi_inquiry_data) of 256 bytes combined with off-by-one
error in the changed code gave total INQUIRY data length above 255 bytes,
that was maximal INQUIRY length in SPC-2. While SPC-3 increased the
maximal length to 64K, at least sg3_utils are still confused by that.

MFC after: 1 week

Fix several issues and inconsistencies in UNMAP capabilities reporting.

This makes Windows 2012 to start using UNMAP on our disks.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Add support for Windows dialect of EXTENDED COPY command, aka Microsoft ODX.

This allows to avoid extra network traffic when copying files on NTFS iSCSI
disks within one storage host by drag'n'dropping them in Windows Explorer
of Windows 8/2012. It should also accelerate Hyper-V VM operations, etc.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Fix several cases of NULL dereference when INQUIRY sent to absent LUN.

MFC after: 3 days

Increase maximal number of SCSI ports in CTL from 32 to 128.

After I gave each iSCSI target its own port, the old limit appeared to be
not so big. This change almost proportionally increases per-LUN memory
use, but it is still three times better then it was before r268807.

MFC after: 2 weeks

Reduce per-LUN memory usage from 18MB to 1.8MB.

CTL never had use for CA support code since SPI has gone, and there is no
even frontends supporting that. But it still was reserving 256 bytes of
memory per LUN per every possible initiator on every possible port.

Wrap unused code with ifdef's in case somebody even need it.

MFC after: 2 weeks

Add support for VMWare dialect of EXTENDED COPY command, aka VAAI Clone.

This allows to clone VMs and move them between LUNs inside one storage
host without generating extra network traffic to the initiator and back,
and without being limited by network bandwidth.

LUNs participating in copy operation should have UNIQUE NAA or EUI IDs set.
For LUNs without these IDs VMWare will use traditional copy operations.

Beware: the above LUN IDs explicitly set to values non-unique from the VM
cluster point of view may cause data corruption if wrong LUN is addressed!

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Add LUN options to specify 64-bit EUI and NAA identifiers.

Remove status setting from datamove() path. Leave that to other places.

Enable TAS feature: notify initiator if its command was aborted by other.

That should make operation more kind to multi-initiator environment.
Without this, other initiators may find out that something bad happened
to their commands only via command timeout.

Fix typo in r267873.

Return task management requests to queued execution, but differently.

Testing shown that both original queued design with separate task queue,
and recent direct execution design had significant flaw: If abort request
arrives just after the victim, the last one may not be in the ooa_queue
yet, and so invisible for the task management function.

Unlike original queued implementation, use same queue for all SCSI and
TASK requests from the same initiator. That avoids races between them:
task functions are always executed in proper time, relatively to other
requests.

Fix task management functions status: task not found is not an error,
while not implemented function is.

Add support for READ FULL STATUS action of PERSISTENT RESERVE IN command.

Teach ctl_add_initiator() to dynamically allocate IIDs from pool.

If port passed negative IID value, the function will try to allocate IID
from the pool of unused, based on passed wwpn or name arguments. It does
all its best to make IID unique and persistent across reconnects.

This makes persistent reservation properly work for iSCSI. Previously,
in case of reconnects, reservation could be unexpectedly lost, or even
migrate between intiators.

Implement ABORT TASK SET and I_T NEXUS RESET task management functions.

Use the last one to terminate active commands on iSCSI session termination.
Previous code was aborting only commands doing some data moves.

Make gcc happy, init idlen2.

Close race in r268291 between port destruction, delayed by sessions
teardown, and new port creation during `service ctld restart`.

Close it by returning iSCSI port internal state, that allows to identify
dying ports, which should not be counted as existing, from really alive.

Add support for SCSI Ports (88h) VPD page.

Make REPORT TARGET PORT GROUPS command report realistic data instead of
hardcoded garbage.

Move lun_map() method from command nexus to port.

Previous implementation made impossible to do some things, such as calling
it for ports other then one through which command arrived.

Burry devid port method, which was a gross hack.

Instead make ports provide wanted port and target IDs, and LUNs provide
wanted LUN IDs. After that core Device ID VPD code only had to link all
of them together and add relative port and port group numbers.

LUN ID for iSCSI LUNs no longer created by CTL, but by ctld, and passed
to CTL as "scsiname" LUN option. This makes LUNs to report the same set
of IDs, independently from the port through which it is accessed, as
required by SCSI specifications.

Create separate CTL port for every iSCSI target (and maybe portal group).

Having single port for all iSCSI connections makes problematic implementing
some more advanced SCSI functionality in CTL, that require proper ports
enumeration and identification.

This change extends CTL iSCSI API, making ctld daemon to control list of
iSCSI ports in CTL. When new target is defined in config fine, ctld will
create respective port in CTL. When target is removed -- port will be
also removed after all active commands through that port properly aborted.
This change require ctld to be rebuilt to match the kernel.

As a minor side effect, this allows to have iSCSI targets without LUNs.
While that may look odd and not very useful, that is not incorrect.

Improve CTL_BEARG_* flags support, including optional values copyout.

Implement and use ctl_frontend_find().

Introduce new IOCTL CTL_PORT_LIST reporting in more flexible XML format.

Leave old CTL_GET_PORT_LIST in place so far. Garbage-collect it later.

Improve readability of XML generated by CTL_LUN_LIST.

Make options KPI more generic to allow it to be used for ports too,
not only for LUNs.

Separate concepts of frontend and port.

Before iSCSI implementation CTL had no knowledge about frontend drivers,
it had only frontends, which really were ports (alike to LUNs, if comparing
to backends). But iSCSI added there ioctl() method, which does not belong
to frontend as a port, but belongs to a frontend driver.

Remove targ_enable()/targ_disable() frontend methods.

Those methods were never implemented, and I believe that their concept is
wrong, since single frontend (SCSI port) can not handle several targets.

Use separate memory type M_CTLIO for I/Os.

CTL allocate large amount of RAM. This change give some more stats.

MFC after: 2 weeks

Add support for REPORT TIMESTAMP command.

MFC after: 2 weeks

Add more formal and strict command parsing and validation.

For every supported command define CDB length and mask of bits that are
allowed to be set. This allows to remove bunch of checks through the code
and still make the validation more strict. To properly do it for commands
supporting multiple service actions, formalize their parsing by adding
subtables for each of such commands.

As visible effect, this change allows to add support for REPORT SUPPORTED
OPERATION CODES command, reporting to client all the data about supported
SCSI commands, except timeouts.

MFC after: 2 weeks

Pull in r267961 and r267973 again. Fix for issues reported will follow.

Remove odd practice of inverting error codes.

-EPERM is equal to ERESTART, returning which from ioctl() handler causes
infinite syscall restart.

MFC after: 2 weeks

Revert r267961, r267973:

These changes prevent sysctl(8) from returning proper output,
such as:

1) no output from sysctl(8)
2) erroneously returning ENOMEM with tools like truss(1)
or uname(1)
truss: can not get etype: Cannot allocate memory

Extend the meaning of the CTLFLAG_TUN flag to automatically check if
there is an environment variable which shall initialize the SYSCTL
during early boot. This works for all SYSCTL types both statically and
dynamically created ones, except for the SYSCTL NODE type and SYSCTLs
which belong to VNETs. A new flag, CTLFLAG_NOFETCH, has been added to
be used in the case a tunable sysctl has a custom initialisation
function allowing the sysctl to still be marked as a tunable. The
kernel SYSCTL API is mostly the same, with a few exceptions for some
special operations like iterating childrens of a static/extern SYSCTL
node. This operation should probably be made into a factored out
common macro, hence some device drivers use this. The reason for
changing the SYSCTL API was the need for a SYSCTL parent OID pointer
and not only the SYSCTL parent OID list pointer in order to quickly
generate the sysctl path. The motivation behind this patch is to avoid
parameter loading cludges inside the OFED driver subsystem. Instead of
adding special code to the OFED driver subsystem to post-load tunables
into dynamically created sysctls, we generalize this in the kernel.

Other changes:
- Corrected a possibly incorrect sysctl name from "hw.cbb.intr_mask"
to "hw.pcic.intr_mask".
- Removed redundant TUNABLE statements throughout the kernel.
- Some minor code rewrites in connection to removing not needed
TUNABLE statements.
- Added a missing SYSCTL_DECL().
- Wrapped two very long lines.
- Avoid malloc()/free() inside sysctl string handling, in case it is
called to initialize a sysctl from a tunable, hence malloc()/free() is
not ready when sysctls from the sysctl dataset are registered.
- Bumped FreeBSD version to indicate SYSCTL API change.

MFC after: 2 weeks
Sponsored by: Mellanox Technologies

Simplify statistics calculation.

Instead of trying to guess size of disk I/O operations (it just won't work
that way for newly added commands, and is equal to data move size for old
ones), account data move traffic. If disk I/Os are that interesting, then
backends have to account and provide that information.

Block backend already exports the information about disk I/Os via devstat,
so having it here too is excessive.

MFC after: 2 weeks

Allow MODE SENSE commands through Write Exclusive persistent reservation,
as required by SPC-4.

Report that fact in persistent reservation capabilities.

MFC after: 2 weeks

Add READ BUFFER and improve WRITE BUFFER SCSI commands support.

This gives some use to 512KB per-LUN buffers, allocated for Copan-specific
processor code and not used. It allows, for example, to test transport
performance and/or correctness without accessing the media, as supported
by Linux version of sg3_utils.

MFC after: 2 weeks

Introduce fine-grained CTL locking to improve SMP scalability.

Split global ctl_lock, historically protecting most of CTL context:
- remaining ctl_lock now protects lists of fronends and backends;
- per-LUN lun_lock(s) protect LUN-specific information;
- per-thread queue_lock(s) protect request queues.
This allows to radically reduce congestion on ctl_lock.

Create multiple worker threads, depending on number of CPUs, and assign
each LUN to one of them. This allows to spread load between multiple CPUs,
still avoiging congestion on queues and LUNs locks.

On 40-core server, exporting 5 LUNs, each backed by gstripe of SATA SSDs,
accessed via 6 iSCSI connections, this change improves peak request rate
from 250K to 680K IOPS.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Execute task management request directly in ctl_queue() context.

From one side it allows to remove CTL_FLAG_TASK_PENDING flag, handling of
which significantly complicates fine-grained locking. From the other side
it reduces task management requests latency even below then that flag could.
As downside, it denies task management code to sleep, but that is not needed
any way now.

Discussed with: ken

Add some more CTL_FLAG_ABORT check points.

This should allow to abort commands doing mostly disk I/O, such as VERIFY
or WRITE SAME. Before this change CTL_FLAG_ABORT was only checked around
data moves, which for these commands may not happen for a very long time.

MFC after: 2 weeks

Increase CTL_DEVID_LEN from 16 to 64 bytes.

SPC-4 recommends T10 vendor ID based LUN ID was created by concatenating
product name and serial number (and istgt follows that). But product name
is 16 bytes long by itself, so 16 bytes total length is clearly not enough
to fit both.

To keep compatibility with existing configurations, pad short device IDs
to old length of 16, same as before.

This change probably breaks CTL user-level ABI, so control tools should
be rebuilt after this change.

MFC after: 2 weeks

Add support for VERIFY(10/12/16) and COMPARE AND WRITE SCSI commands.

Make data_submit backends method support not only read and write requests,
but also two new ones: verify and compare. Verify just checks readability
of the data in specified location without transferring them outside.
Compare reads the specified data and compares them to received data,
returning error if they are different.

VERIFY(10/12/16) commands request either verify or compare from backend,
depending on BYTCHK CDB field. COMPARE AND WRITE command executed in two
stages: first it requests compare, and then, if succeesed, requests write.
Atomicity of operation is guarantied by CTL request ordering code.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Remove memcpy() from ctl_private[] accesses.

That union is aligned enough to access data directly.

Move kern_total_len setting from backend to core code.

Respect "vendor" option in all places.

MFC after: 2 weeks

Add "vendor", "product" and "revision" options to control inquiry data.

MFC after: 2 weeks

Remove non-functional remnants of control LUN -- 18MB of RAM for nothing.

Remove limits on size of READ/WRITE operations.

Instead of allocating up to 16MB or RAM at once to handle whole I/O,
allocate up to 1MB at a time, but do multiple ctl_datamove() and storage
I/Os if needed.

Join CTL worker threads into one process for convenience.
Report their idle state as "-".

Add support for SCSI UNMAP commands to CTL.

This patch adds support for three new SCSI commands: UNMAP, WRITE SAME(10)
and WRITE SAME(16). WRITE SAME commands support both normal write mode
and UNMAP flag. To properly report UNMAP capabilities this patch also adds
support for reporting two new VPD pages: Block limits and Logical Block
Provisioning.

UNMAP support can be enabled per-LUN by adding "-o unmap=on" to `ctladm
create` command line or "option unmap on" to lun sections of /etc/ctl.conf.

At this moment UNMAP supported for ramdisks and device-backed block LUNs.
It was tested to work great with ZFS ZVOLs. For file-backed LUNs UNMAP
support is unfortunately missing due to absence of respective VFS KPI.

Reviewed by: ken
MFC after: 1 month
Sponsored by: iXsystems, Inc

Wakeup only one thread of added in r263978i at a time.

This slightly reduces lock congestion between threads.

Submitted by: trasz

Report stripe size and offset of the backing device in READ CAPACITY (16)
as physical sector size and offset.

MFC after: 2 weeks

Hide CTL messages about SCSI error responses. Too many users take
them for actual target errors. They can be enabled back by setting
kern.cam.ctl.verbose=1, or booting with bootverbose.

Sponsored by: The FreeBSD Foundation

Make it possible to have multiple CTL worker threads. Leave the default
of 1 for now.

Sponsored by: The FreeBSD Foundation

Fix missing unlock in persistent reservations code, which resulted in panics
with Hyper-V Failover Cluster.

Reviewed by: ken@
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation

Introduce seperate mutex lock to protect protect CTL I/O pools, slightly
reducing global CTL lock scope and congestion.

While there, simplify CTL I/O pools KPI, hiding implementation details.

Bring in the new iSCSI target and initiator.

Reviewed by: ken (parts)
Approved by: re (delphij)
Sponsored by: FreeBSD Foundation

CTL changes required for iSCSI target, most notably LUN remapping
and a mechanism to allow CTL frontends for retrieving LUN options.

Reviewed by: ken (earlier version)

Turn comments about locking into actual lock assertions.

Reviewed by: ken
Tested by: ken
MFC after: 1 month

Remove dead code.

Remove ctl(4) from GENERIC. Also remove 'options CTL_DISABLE'
and kern.cam.ctl.disable tunable; those were introduced as a workaround
to make it possible to boot GENERIC on low memory machines.

With ctl(4) being built as a module and automatically loaded by ctladm(8),
this makes CTL work out of the box.

Reviewed by: ken
Sponsored by: FreeBSD Foundation

Fix a memory leak that showed up when we delete LUNs. The memory used for
the LUN was never freed.

ctl.c: Adjust ctl_alloc_lun() to make sure we don't clear the
CTL_LUN_MALLOCED flag.

Reported by: Sreenivasa Honnur <shonnur@chelsio.com>
Sponsored by: Spectra Logic
MFC after: 3 days

Since the CTL version in FreeBSD does not support High Availability,
ctl_is_single should always be set to 1. Make it so. Previously
it was always 0, because ctl_isc_start() never got to run.

Suggested by: ken

Fix locking problem in ctl_maintenance_in() - one cannot use M_WAITOK or call
ctl_done() with mutex held.

Reviewed by: ken
Sponsored by: FreeBSD Foundation

Fix comment formatting.

Make it possible to build CTL as a module.

Reviewed by: ken
Sponsored by: FreeBSD Foundation

Re-enable CTL in GENERIC on i386 and amd64, but turn on the CTL disable
tunable by default.

This will allow GENERIC configurations to boot on small memory boxes, but
not require end users who want to use CTL to recompile their kernel. They
can simply set kern.cam.ctl.disable=0 in loader.conf.

The eventual solution to the memory usage problem is to change the way
CTL allocates memory to be more configurable, but this should fix things
for small memory situations in the mean time.

UPDATING: Explain the change in the CTL configuration, and
how users can enable CTL if they would like to use
it.

sys/conf/options: Add a new option, CTL_DISABLE, that prevents CTL
from initializing.

ctl.c: If CTL_DISABLE is turned on, don't initialize.

i386/conf/GENERIC,
amd64/conf/GENERIC: Re-enable device ctl, and add the CTL_DISABLE
option.

Make CTL work a little better with loading and unloading drivers.

Previously CTL would leave individual LUNs enabled in the target
driver, whether or not the port as a whole was enabled. It would
also leave the wildcard LUN enabled indefinitely.

This change means that CTL will enable and disable any active LUNs,
as well as the wildcard LUN, when enabling and disabling a port.

Also, fix a bug that could crop up due to an uninitialized CCB
type.

ctl.c: Before calling ctl_frontend_online(), run through
the LUN list and enable all active LUNs.

After calling ctl_frontend_offline(), run through
the LUN list and disble all active LUNs.

scsi_ctl.c: Before bringing a port online, allocate the
wildcard peripheral for that bus. And after taking
a port offline, invalidate the wildcard peripheral
for that bus.

Make sure that we hold the SIM lock around all
calls to xpt_action() and other transport layer
interfaces that require it.

Use CAM_SIM_{LOCK|UNLOCK} consistently to acquire
and release the SIM lock.

Update a number of outdated comments. Some of
these should have been fixed long ago.

Actually do LUN disbables now. The newer drivers
in the tree work correctly for this as far as I
know.

Initialize the CCB type to CTLFE_CCB_DEFAULT to
avoid a panic due to uninitialized memory.

Submitted by: Chuck Tuffli (partially)
MFC after: 1 week

Fix the CTL OOA queue dumping code so that it does not hold a mutex
while doing a copyout. That can cause a panic, because copyout
can trigger VM faults, and we can't handle VM faults while holding
a mutex.

The solution here is to malloc a separate buffer to hold the OOA
queue entries, so that we don't risk a VM fault while filling up
the buffer and we don't have to drop the lock. The other solution
would be to wire the user's memory while filling their buffer with
copyout, but that would have been a little more complex.

Also fix a debugging parenthesis issue in ctl_abort_task() pointed
out by Chuck Tuffli.

Sponsored by: Spectra Logic Corporation
MFC after: 1 week

Use M_ZERO instead of explicit memsets and bzeros.

remove duplicate semicolons where possible.

Approved by: cperciva
MFC after: 1 week

Remove useless NULL checks after M_WAITOK allocations.

Fix panic in CTL caused by trying to free invalid pointers passed
by the userland process via the IOCTL interface.

Reviewed by: ken@

Add a loader tunable, kern.cam.ctl.disable, that will disable
loading CTL. This may be useful in very low memory installations.

MFC after: 3 days

Change the SCSI INQUIRY peripheral qualifier that CTL reports for LUNs
that don't exist.

Anecdotal evidence indicates that it is better to return 011b (bad LUN)
than 001b (LUN offline). However, this change also gives the user a
sysctl/tunable, kern.cam.ctl.inquiry_pq_no_lun, to override the change
and return to the previous behavior. (The previous behavior was to
return 001b, or LUN offline.)

ctl.c: Change the default inquiry peripheral qualifier to 011b,
and add a sysctl and tunable to allow the user to change
it back to 001b if needed.

Don't insert a Copan copyright statement in the inquiry
data. The copyright statements on the files are
sufficient.

ctl_private.h: Add sysctl variable context to the CTL softc.

ctl_cmd_table.c,
ctl_frontend_internal.c,
ctl_frontend.c,
ctl_backend.c,
ctl_error.c: Include sys/sysctl.h.

MFC after: 3 days

Add LUN resizing to CTL. Also make it possible to explicitly set
size when creating file-backed or device-backed LUN.

Reviewed by: ken (earlier version)
Sponsored by: The FreeBSD Foundation

Quiet some clang warnings when compiling CTL.

ctl_error.c,
ctl_error.h: Take out the ctl_sense_format enumeration, and use
scsi_sense_data_type instead.

Remove ctl_get_sense_format() and switch ctl_build_ua()
over to using scsi_sense_data_type.

ctl_backend_ramdisk.c,
ctl_backend_block.c:
Use C99 structure initializers instead of GNU initializers.

ctl.c: Switch over to using the SCSI sense format enumeration
instead of the CTL-specific enumeration.

Submitted by: dim (partially)
MFC after: 1 month

Add the CAM Target Layer (CTL).

CTL is a disk and processor device emulation subsystem originally written
for Copan Systems under Linux starting in 2003. It has been shipping in
Copan (now SGI) products since 2005.

It was ported to FreeBSD in 2008, and thanks to an agreement between SGI
(who acquired Copan's assets in 2010) and Spectra Logic in 2010, CTL is
available under a BSD-style license. The intent behind the agreement was
that Spectra would work to get CTL into the FreeBSD tree.

Some CTL features:

- Disk and processor device emulation.
- Tagged queueing
- SCSI task attribute support (ordered, head of queue, simple tags)
- SCSI implicit command ordering support. (e.g. if a read follows a mode
select, the read will be blocked until the mode select completes.)
- Full task management support (abort, LUN reset, target reset, etc.)
- Support for multiple ports
- Support for multiple simultaneous initiators
- Support for multiple simultaneous backing stores
- Persistent reservation support
- Mode sense/select support
- Error injection support
- High Availability support (1)
- All I/O handled in-kernel, no userland context switch overhead.

(1) HA Support is just an API stub, and needs much more to be fully
functional.

ctl.c: The core of CTL. Command handlers and processing,
character driver, and HA support are here.

ctl.h: Basic function declarations and data structures.

ctl_backend.c,
ctl_backend.h: The basic CTL backend API.

ctl_backend_block.c,
ctl_backend_block.h: The block and file backend. This allows for using
a disk or a file as the backing store for a LUN.
Multiple threads are started to do I/O to the
backing device, primarily because the VFS API
requires that to get any concurrency.

ctl_backend_ramdisk.c: A "fake" ramdisk backend. It only allocates a
small amount of memory to act as a source and sink
for reads and writes from an initiator. Therefore
it cannot be used for any real data, but it can be
used to test for throughput. It can also be used
to test initiators' support for extremely large LUNs.

ctl_cmd_table.c: This is a table with all 256 possible SCSI opcodes,
and command handler functions defined for supported
opcodes.

ctl_debug.h: Debugging support.

ctl_error.c,
ctl_error.h: CTL-specific wrappers around the CAM sense building
functions.

ctl_frontend.c,
ctl_frontend.h: These files define the basic CTL frontend port API.

ctl_frontend_cam_sim.c: This is a CTL frontend port that is also a CAM SIM.
This frontend allows for using CTL without any
target-capable hardware. So any LUNs you create in
CTL are visible in CAM via this port.

ctl_frontend_internal.c,
ctl_frontend_internal.h:
This is a frontend port written for Copan to do
some system-specific tasks that required sending
commands into CTL from inside the kernel. This
isn't entirely relevant to FreeBSD in general,
but can perhaps be repurposed.

ctl_ha.h: This is a stubbed-out High Availability API. Much
more is needed for full HA support. See the
comments in the header and the description of what
is needed in the README.ctl.txt file for more
details.

ctl_io.h: This defines most of the core CTL I/O structures.
union ctl_io is conceptually very similar to CAM's
union ccb.

ctl_ioctl.h: This defines all ioctls available through the CTL
character device, and the data structures needed
for those ioctls.

ctl_mem_pool.c,
ctl_mem_pool.h: Generic memory pool implementation used by the
internal frontend.

ctl_private.h: Private data structres (e.g. CTL softc) and
function prototypes. This also includes the SCSI
vendor and product names used by CTL.

ctl_scsi_all.c,
ctl_scsi_all.h: CTL wrappers around CAM sense printing functions.

ctl_ser_table.c: Command serialization table. This defines what
happens when one type of command is followed by
another type of command.

ctl_util.c,
ctl_util.h: CTL utility functions, primarily designed to be
used from userland. See ctladm for the primary
consumer of these functions. These include CDB
building functions.

scsi_ctl.c: CAM target peripheral driver and CTL frontend port.
This is the path into CTL for commands from
target-capable hardware/SIMs.

README.ctl.txt: CTL code features, roadmap, to-do list.

usr.sbin/Makefile: Add ctladm.

ctladm/Makefile,
ctladm/ctladm.8,
ctladm/ctladm.c,
ctladm/ctladm.h,
ctladm/util.c: ctladm(8) is the CTL management utility.
It fills a role similar to camcontrol(8).
It allow configuring LUNs, issuing commands,
injecting errors and various other control
functions.

usr.bin/Makefile: Add ctlstat.

ctlstat/Makefile
ctlstat/ctlstat.8,
ctlstat/ctlstat.c: ctlstat(8) fills a role similar to iostat(8).
It reports I/O statistics for CTL.

sys/conf/files: Add CTL files.

sys/conf/NOTES: Add device ctl.

sys/cam/scsi_all.h: To conform to more recent specs, the inquiry CDB
length field is now 2 bytes long.

Add several mode page definitions for CTL.

sys/cam/scsi_all.c: Handle the new 2 byte inquiry length.

sys/dev/ciss/ciss.c,
sys/dev/ata/atapi-cam.c,
sys/cam/scsi/scsi_targ_bh.c,
scsi_target/scsi_cmds.c,
mlxcontrol/interface.c: Update for 2 byte inquiry length field.

scsi_da.h: Add versions of the format and rigid disk pages
that are in a more reasonable format for CTL.

amd64/conf/GENERIC,
i386/conf/GENERIC,
ia64/conf/GENERIC,
sparc64/conf/GENERIC: Add device ctl.

i386/conf/PAE: The CTL frontend SIM at least does not compile
cleanly on PAE.

Sponsored by: Copan Systems, SGI and Spectra Logic
MFC after: 1 month