MFC r360610: Add session locking in cfiscsi_ioctl_handoff().

While there, remove ifdef around cs_target check in cfiscsi_ioctl_list().
I am not sure why this ifdef was added, but without this check code will
crash below on NULL dereference.

MFC r331013:

Fix iSCSI target crash on session reinstation.

The crash scenario goes like this: there's a thread waiting on "reinstate";
because it doesn't update the timeout counter it gets terminated by the
callout; at this point the maintenance thread starts the termination routine.
The first thread finishes waiting, proceeds to icl_conn_handoff(), and drops
the refcount, which allows the maintenance thread to free its resources. At
this point another thread receives a PDU. Boom.

PR: 222898, 219866
Sponsored by: playkey.net

MFC r324261:

Fix iSCSI target panics on concurrent session teardown and display
(eg removing a target and doing "ctladm islist -v" at the same time).

MFC r314659,r314676:

r314659:

usr.sbin: normalize paths using SRCTOP-relative paths or :H when possible

This simplifies make logic/output

r314676:

Fix build after r314656

Some of the changes I introduced to use .ALLSRC were correct in spirit,
but incorrect in reality -- in particular, ../Makefile.inc hadn't been
pulled in via bsd.init.mk (via bsd.lib.mk, bsd.prog.mk), so the value
of .ALLSRC (evaluated immediately) was empty. .include bsd.init.mk
explicitly so we can be certain that the values used as dependencies in
the targets are defined when the target recipe has been evaluated.

Reminder: thou shalt separate out separate functional changes before
committing them.

(YUGE) Pointyhat to: ngie
In collaboration with: bdrewery

MFC r316653: Fix few minor issues found by Clang Analyzer.

MFC r311283:

Don't release the cfiscsi session refcount too early. It wasn't
observed to fix any actual error, but it's the right thing to do
from the correctness point of view.

MFC r313854, r313963: Change the way MaxCmdSN is used.

Before this change MaxCmdSN was reported as CmdSN + delta, that made it
limit number of requests in transmission from the initiator to target,
that was pretty useless. After this change MaxCmdSN limits number of
requests queued to CTL, i.e. maximal queue depth for the initiator.
The default limit is 256 outstanding requests per initiator at a time.

This code uses existing cs_outstanding_ctl_pdus counter to track queue
depth. It's semantics doen't perfectly match, but close enough to not
add another counter. Just don't set the maxtags below 2.

MFC r312603: Add initial support for CTL module unloading.

It is only a first step and not perfect, but better then nothing.
The main blocker is CAM target frontend, that can not be unloaded,
since CAM does not have mechanism to unregister periph driver now.

MFC r312291, r312669:
Make CTL frontends report kern_data_resid for under-/overruns.

It seems like kern_data_resid was never really implemented. This change
finally does it. Now frontends update this field while transferring data,
while CTL/backends getting it can more flexibly handle the result.
At this point behavior should not change significantly, still reporting
errors on write overrun, but that may be changed later, if we decide so.

CAM target frontend still does not properly handle overruns due to CAM API
limitations. We may need to add some fields to struct ccb_accept_tio to
pass information about initiator requested transfer size(s).

MFC r312343: Improve error message on duplicate iSCSI port.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Report negotiated MaxBurstLength and FirstBurstLength in "iscsictl -v"
and "ctladm islist -v" outputs.

MFC after: 1 month

Add mechanism for choosing iSER-capable ICL modules.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Add icl_conn_connect() ICL method, required for iSER.

Obtained from: Mellanox Technologies (earlier version)
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Set bhsdi_target_transfer_tag to reserved value, which is 0xffffffff.

This should be a purely cosmetic change.

Give CTL support for PIM_EXTLUNS when talking to CAM.

CTL itself still lives in flat LUN space, but it can generate extended
numbers if CAM SIM reports such capability.

Remove some dead code found by Clang analyzer.

Remove some control_softc references.

Log iSCSI session reinstatements.

False session reinstatements can be result of misconfiguration, when
several initiators use the same initiator name and ISID.

Mark with DMA flag I/Os waiting for iSCSI write data after R2T.

Reads and immediate writes are not blocking, so don't bother.

Implement QUERY TASK, QUERY TASK SET and QUERY ASYNC EVENT.

Now we support most of SAM-5 task management.

Map CLEAR TASK SET and I_T NEXUS RESET for iSCSI.

The last should not be called without iSCSIProtocolLevel negotiation.

Implement iSCSI TARGET COLD RESET task management function.

Implement it as CTL_TASK_TARGET_RESET plus termination of all sessions.

Reimplement CTL High Availability.

CTL HA functionality was originally implemented by Copan many years ago,
but large part of the sources was never published. This change includes
clean room implementation of the missing code and fixes for many bugs.

This code supports dual-node HA with ALUA in four modes:
- Active/Unavailable without interlink between nodes;
- Active/Standby with second node handling only basic LUN discovery and
reservation, synchronizing with the first node through the interlink;
- Active/Active with both nodes processing commands and accessing the
backing storage, synchronizing with the first node through the interlink;
- Active/Active with second node working as proxy, transfering all
commands to the first node for execution through the interlink.

Unlike original Copan's implementation, depending on specific hardware,
this code uses simple custom TCP-based protocol for interlink. It has
no authentication, so it should never be enabled on public interfaces.

The code may still need some polishing, but generally it is functional.

Relnotes: yes
Sponsored by: iXsystems, Inc.

Remove unused target and initiator IDs.

Make most of port methods optional and remove bunch of dummies.

Drop "internal" CTL frontend.

Its idea was to be a simple initiator and execute several commands from
kernel level, but FreeBSD never had consumer for that functionality,
while its implementation polluted many unrelated places..

Bring per-port LUN enable/disable code up to date:
- remove last remnants of never implemented multiple targets support;
- implement missing support for LUN mapping in this area.

Due to existing locking constraints LUN mapping code is practically
unlocked at this point. Hopefully it is not racy enough to live until
somebody get idea how to call sleeping fronend methods under lock also
taken by the same frontend in non-sleepable context. :(

Fix ordering of "*logout" and "*terminate"; no functional changes.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Extend ICL to add receive offload methods. For software ICL backend
they are no-ops.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Make it possible to set (via ctl.conf(5)) and query (via ctladm islist -v)
target iSCSI offload. Add mechanism to query maximum receive data segment
size supported by chosen hardware offload module, and use it in ctld(8)
to determine the value to advertise to the other side.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Bring some more order into iSCSI portal group tags support.

While ctld(8) still does not allow multiple portal groups per target
to be configured, kernel should now be able to handle it.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

CTL LUN mapping rewrite.

Replace iSCSI-specific LUN mapping mechanism with new one, working for any
ports. By default all ports are created without LUN mapping, exposing all
CTL LUNs as before. But, if needed, LUN mapping can be manually set on
per-port basis via ctladm. For its iSCSI ports ctld does it via ioctl(2).
The next step will be to teach ctld to work with FibreChannel ports also.

Respecting additional flexibility of the new mechanism, ctl.conf now allows
alternative syntax for LUN definition. LUNs can now be defined in global
context, and then referenced from targets by unique name, as needed. It
allows same LUN to be exposed several times via multiple targets.

While there, increase limit for LUNs per target in ctld from 256 to 1024.
Some initiators do not support LUNs above 255, but that is not our problem.

Discussed with: trasz
MFC after: 2 weeks
Relnotes: yes
Sponsored by: iXsystems, Inc.

Add kobj interface between ICL and the rest of the iSCSI stack.
Review note - icl.c was moved to icl_soft.c.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Make sequence numbers checks more strict.

While we don't support MCS, hole in received sequence numbers may mean
only PDU loss. While we don't support lost PDU recovery, terminate the
connection to avoid stuck commands.

While there, improve handling of sequence numbers wrap after 2^32 PDUs.

MFC after: 2 weeks

Coalesce last data move and command status for read commands.

Make CTL core and block backend set success status before initiating last
data move for read commands. Make CAM target and iSCSI frontends detect
such condition and send command status together with data. New I/O flag
allows to skip duplicate status sending on later fe_done() call.

For Fibre Channel this change saves one of three interrupts per read command,
increasing performance from 126K to 160K IOPS. For iSCSI this change saves
one of three PDUs per read command, increasing performance from 1M to 1.2M
IOPS.

MFC after: 1 month
Sponsored by: iXsystems, Inc.

Replace home-grown CTL IO allocator with UMA.

Old allocator created significant lock congestion protecting its lists
of preallocated I/Os, while UMA provides much better SMP scalability.
The downside of UMA is lack of reliable preallocation, that could guarantee
successful allocation in non-sleepable environments. But careful code
review shown, that only CAM target frontend really has that requirement.
Fix that making that frontend preallocate and statically bind CTL I/O for
every ATIO/INOT it preallocates any way. That allows to avoid allocations
in hot I/O path. Other frontends either may sleep in allocation context
or can properly handle allocation errors.

On 40-core server with 6 ZVOL-backed LUNs and 7 iSCSI client connections
this change increases peak performance from ~700K to >1M IOPS! Yay! :)

MFC after: 1 month
Sponsored by: iXsystems, Inc.

Read cs_outstanding_ctl_pdus before incrementing it for NEXUS RESET task.

This removes extra log noise on idle connection termination.

MFC after: 1 week

Make iSCSI frontend less chatty while waiting for tasks termination.

MFC after: 1 week

Make cfiscsi_offline() synchronous, waiting for connections termination
before return. This should make ctld restart more clean and predictable.

MFC after: 2 weeks

Close race between cfiscsi_offline() and new connection arrival.

Incoming connection should be either rejected or accepted and terminated.

Partially reconstruct Active/Standby clusting.

In this mode one head is in Active state, supporting all commands, while
another is in Standby state, supporting only minimal LUN discovery subset.

It is still incomplete since Standby state requires reservation support,
which is impossible to do right without having interlink between heads.
But it allows to run some basic experiments.

Fix typo.

MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation

s/icl_pdu_new_bhs/icl_pdu_new/; no functional changes, just a little
nicer code.

Sponsored by: The FreeBSD Foundation

Change the default log level for iSCSI target from 3 to 1. It should
have been 1 from the beginning; not sure how it ended up at 3.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Give physical and virtual ports numbers some more meaning.

Make iSCSI connection close somewhat less aggressive.

It allows to push out some final data from the send queue to the socket
before its close. In particular, it increases chances for logout response
to be delivered to the initiator.

Add support for MaxBurstLength and Expected Data transfer Length parameters.

Before this change target could send R2T request for write transfer of any
size, that could violate iSCSI RFC, which allows initiator to limit maximum
R2T size by negotiating MaxBurstLength connection parameter.

Also report an error in case of write underflow, when initiator provides
less data than initiator expects. Previously in such case our target
sent R2T request for non-existing data, violating the RFC, and confusing
some initiators. SCSI specs don't explicitly define how write underflows
should be handled and there are different oppinions, but reporting error
is hopefully better then violating iSCSI RFC with unpredictable results.

MFC after: 2 weeks

Fix read overrun handling, broken by using wrong variable.

MFC after: 3 days

Make sure we handle less than zero timeouts in iSCSI initiator and target
in a reasonable way.

Sponsored by: The FreeBSD Foundation

Make it possible to disable NOP-In PDUs by the iSCSI initiator by setting
kern.cam.ctl.iscsi.ping_timeout to 0. This fixes interoperability with
some initiators that don't properly support NOP-Ins, namely iPXE/gPXE.

Submitted by: Chen Wen <pokkys@gmail.com>
MFC after: 1 week
Sponsored by: The FreeBSD Foundation

Use proper include paths in kernel iSCSI code.

MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation

Fix ctl(4) kldload failure that manifested like this:

link_elf_obj: symbol icl_pdu_new_bhs undefined

PR: 192031
Submitted by: Nils Beyer (earlier version)
MFC after: 3 days
Sponsored by: FreeBSD Foundation

Add support for VMWare dialect of EXTENDED COPY command, aka VAAI Clone.

This allows to clone VMs and move them between LUNs inside one storage
host without generating extra network traffic to the initiator and back,
and without being limited by network bandwidth.

LUNs participating in copy operation should have UNIQUE NAA or EUI IDs set.
For LUNs without these IDs VMWare will use traditional copy operations.

Beware: the above LUN IDs explicitly set to values non-unique from the VM
cluster point of view may cause data corruption if wrong LUN is addressed!

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Enable TAS feature: notify initiator if its command was aborted by other.

That should make operation more kind to multi-initiator environment.
Without this, other initiators may find out that something bad happened
to their commands only via command timeout.

Do not return statuses for aborted iSCSI commands.

Fix "use after free" on port creation error in r268291.

Teach ctl_add_initiator() to dynamically allocate IIDs from pool.

If port passed negative IID value, the function will try to allocate IID
from the pool of unused, based on passed wwpn or name arguments. It does
all its best to make IID unique and persistent across reconnects.

This makes persistent reservation properly work for iSCSI. Previously,
in case of reconnects, reservation could be unexpectedly lost, or even
migrate between intiators.

Correction to r268356: collide only sessions to the same target.

When new connection comes in, check whether we already have session from
the same intiator (Name+ISID). If so -- terminate the old session and let
the new one take its place, as required by iSCSI RFC.

Implement ABORT TASK SET and I_T NEXUS RESET task management functions.

Use the last one to terminate active commands on iSCSI session termination.
Previous code was aborting only commands doing some data moves.

Close race in r268291 between port destruction, delayed by sessions
teardown, and new port creation during `service ctld restart`.

Close it by returning iSCSI port internal state, that allows to identify
dying ports, which should not be counted as existing, from really alive.

Move lun_map() method from command nexus to port.

Previous implementation made impossible to do some things, such as calling
it for ports other then one through which command arrived.

Pass through iSCSI session ISID from LOGIN request to the CTL frontend.

ISID is an important part of initiator transport ID for iSCSI. It is not
used now, but should be to properly implement persistent reservation.

Burry devid port method, which was a gross hack.

Instead make ports provide wanted port and target IDs, and LUNs provide
wanted LUN IDs. After that core Device ID VPD code only had to link all
of them together and add relative port and port group numbers.

LUN ID for iSCSI LUNs no longer created by CTL, but by ctld, and passed
to CTL as "scsiname" LUN option. This makes LUNs to report the same set
of IDs, independently from the port through which it is accessed, as
required by SCSI specifications.

Create separate CTL port for every iSCSI target (and maybe portal group).

Having single port for all iSCSI connections makes problematic implementing
some more advanced SCSI functionality in CTL, that require proper ports
enumeration and identification.

This change extends CTL iSCSI API, making ctld daemon to control list of
iSCSI ports in CTL. When new target is defined in config fine, ctld will
create respective port in CTL. When target is removed -- port will be
also removed after all active commands through that port properly aborted.
This change require ctld to be rebuilt to match the kernel.

As a minor side effect, this allows to have iSCSI targets without LUNs.
While that may look odd and not very useful, that is not incorrect.

Make options KPI more generic to allow it to be used for ports too,
not only for LUNs.

Separate concepts of frontend and port.

Before iSCSI implementation CTL had no knowledge about frontend drivers,
it had only frontends, which really were ports (alike to LUNs, if comparing
to backends). But iSCSI added there ioctl() method, which does not belong
to frontend as a port, but belongs to a frontend driver.

Remove targ_enable()/targ_disable() frontend methods.

Those methods were never implemented, and I believe that their concept is
wrong, since single frontend (SCSI port) can not handle several targets.

Pull in r267961 and r267973 again. Fix for issues reported will follow.

Revert r267961, r267973:

These changes prevent sysctl(8) from returning proper output,
such as:

1) no output from sysctl(8)
2) erroneously returning ENOMEM with tools like truss(1)
or uname(1)
truss: can not get etype: Cannot allocate memory

Extend the meaning of the CTLFLAG_TUN flag to automatically check if
there is an environment variable which shall initialize the SYSCTL
during early boot. This works for all SYSCTL types both statically and
dynamically created ones, except for the SYSCTL NODE type and SYSCTLs
which belong to VNETs. A new flag, CTLFLAG_NOFETCH, has been added to
be used in the case a tunable sysctl has a custom initialisation
function allowing the sysctl to still be marked as a tunable. The
kernel SYSCTL API is mostly the same, with a few exceptions for some
special operations like iterating childrens of a static/extern SYSCTL
node. This operation should probably be made into a factored out
common macro, hence some device drivers use this. The reason for
changing the SYSCTL API was the need for a SYSCTL parent OID pointer
and not only the SYSCTL parent OID list pointer in order to quickly
generate the sysctl path. The motivation behind this patch is to avoid
parameter loading cludges inside the OFED driver subsystem. Instead of
adding special code to the OFED driver subsystem to post-load tunables
into dynamically created sysctls, we generalize this in the kernel.

Other changes:
- Corrected a possibly incorrect sysctl name from "hw.cbb.intr_mask"
to "hw.pcic.intr_mask".
- Removed redundant TUNABLE statements throughout the kernel.
- Some minor code rewrites in connection to removing not needed
TUNABLE statements.
- Added a missing SYSCTL_DECL().
- Wrapped two very long lines.
- Avoid malloc()/free() inside sysctl string handling, in case it is
called to initialize a sysctl from a tunable, hence malloc()/free() is
not ready when sysctls from the sysctl dataset are registered.
- Bumped FreeBSD version to indicate SYSCTL API change.

MFC after: 2 weeks
Sponsored by: Mellanox Technologies

Fix typo in r267481.

MFC after: 3 days

Add READ BUFFER and improve WRITE BUFFER SCSI commands support.

This gives some use to 512KB per-LUN buffers, allocated for Copan-specific
processor code and not used. It allows, for example, to test transport
performance and/or correctness without accessing the media, as supported
by Linux version of sg3_utils.

MFC after: 2 weeks

Allow to use iSCSI immediate data by several ctl_datamove() calls.

While for FreeBSD client that is only a minor optimization, VMWare client
doesn't support additional data requests after all data being sent once as
immediate.

MFC after: 1 week
Sponsored by: iXsystems, Inc.

Increase CTL_DEVID_LEN from 16 to 64 bytes.

SPC-4 recommends T10 vendor ID based LUN ID was created by concatenating
product name and serial number (and istgt follows that). But product name
is 16 bytes long by itself, so 16 bytes total length is clearly not enough
to fit both.

To keep compatibility with existing configurations, pad short device IDs
to old length of 16, same as before.

This change probably breaks CTL user-level ABI, so control tools should
be rebuilt after this change.

MFC after: 2 weeks

Add iSCSI Target Name ID descriptor to VPD 83h.

It shall/should be there according to SPC-4, and istgt also provides it.

MFC after: 2 weeks

Rework session termination in iSCSI target to actually wait
for any outstanding commands to be properly aborted by CTL.
Without it, in some cases (such as files backing the LUNs
stored on failing disk drives), terminating a busy session
would result in panic.

Reviewed by: mav@ (earlier version)
Sponsored by: The FreeBSD Foundation

Make cs_terminating a bool; no functional changes.

Sponsored by: The FreeBSD Foundation

Add comment explaining a potential problem with just added LUN ID.

Reminded by: mav@
Sponsored by: The FreeBSD Foundation

Add LUN-associated name to VPD, to make Hyper-V Failover Cluster happy.

Sponsored by: The FreeBSD Foundation

Format Portal Group Tag same as istgt does -- %4.4x instead of %x.

SPC-4 spec tells it should be "two or more hexadecimal digits".
RFC3720 tells it is 16-bit value.

MFC after: 2 weeks

Respect "vendor" option in all places.

MFC after: 2 weeks

Implement small KPI to access LUN options instead doing it by hands.

MFC after: 2 weeks

Modify CTL iSCSI frontend to properly handle situations where datamove
routine is called multiple times per SCSI task.

Sponsored by: The FreeBSD Foundation

Properly pass the initiator address when running in proxy mode.

Sponsored by: The FreeBSD Foundation

Make it possible to interrupt login when running in proxy mode.

Sponsored by: The FreeBSD Foundation

Properly identify target portal when running in proxy mode. While here,
remove CTL_ISCSI_CLOSE, it wasn't used or implemented anyway.

Sponsored by: The FreeBSD Foundation

Add some stuff to make it easier to figure out for the system administrator
whether the ICL_KERNEL_PROXY stuff got compiled in correctly.

Sponsored by: The FreeBSD Foundation

Make it possible for the iSCSI target side to operate in both normal
and ICL_KERNEL_PROXY mode, and fix some bit rot so the latter actually
works again.

Sponsored by: The FreeBSD Foundation

Fix three refcounter leaks and lock recursion they covered.

MFC after: 1 week

All the iSCSI sysctls are also tunables; advertise that.

Sponsored by: The FreeBSD Foundation

Use atomic ops instead of mutexes where appropriate.

Submitted by: mav@
Sponsored by: The FreeBSD Foundation

Instead of "icltx" and "iclrx", use thread names with prefix from upper
layer, so that one can see which side of the stack the threads are for.

Sponsored by: The FreeBSD Foundation

Get rid of ICL lock; use upper-layer (initiator or target) lock instead.
This avoids extra locking in icl_pdu_queue(); the upper layer needs to call
it while holding its own lock anyway, to avoid sending PDUs out of order.

Sponsored by: The FreeBSD Foundation

Rework cfiscsi_datamove_in() to obey expected data transfer length
received from the initiator.

Sponsored by: The FreeBSD Foundation

Target Transfer Tag is opaque; no need to htonl(3) it.

Sponsored by: The FreeBSD Foundation

Use a less unusual syntax in debug printfs.

Sponsored by: The FreeBSD Foundation

Update kernel inclusions of capability.h to use capsicum.h instead; some
further refinement is required as some device drivers intended to be
portable over FreeBSD versions rely on __FreeBSD_version to decide whether
to include capability.h.

MFC after: 3 weeks

Make reset handling in iSCSI target RFC-compliant. This fixes some rare
hangs with Open-iSCSI (Linux).

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Remove no longer useful debugging output and a stale comment.

Approved by: re (gjb)
Sponsored by: FreeBSD Foundation

Make the error handling more consistant. Shouldn't make any functional
difference.

Approved by: re (gjb)
Sponsored by: FreeBSD Foundation

Tidy up, cache return value of a function, and add an assertion;
shouldn't make any functional difference.

Approved by: re (gjb)
Sponsored by: FreeBSD Foundation

Fix NOP-In/NOP-Out payload handling. Previous way didn't work at all; fortunately
nothing seems to actually use this feature, but it's required by standard.

Approved by: re (glebius)
Sponsored by: FreeBSD Foundation

Properly fix out of memory handling in the iSCSI target.

Approved by: re (glebius)
Sponsored by: FreeBSD Foundation

Split cfiscsi_datamove() in two; no functional changes.

Approved by: re (glebius)
Sponsored by: FreeBSD Foundation

Don't leak memory when removing an unconnected session, and remove useless
UMA_ZONE_NOFREE that caused another leak when unloading the module.

Approved by: re (glebius)
Sponsored by: FreeBSD Foundation

Properly ignore PDUs with CmdSN outside of allowed range.

Approved by: re (glebius)
Sponsored by: FreeBSD Foundation

Fix a few instances of M_WAITOK in threads marked as prohibited from sleep,
missed in r255824.

Approved by: re (kib)
Sponsored by: FreeBSD Foundation

Don't use M_WAITOK when running from context where sleeping is prohibited,
such as callout or a geom thread.

Approved by: re (marius)
Sponsored by: FreeBSD Foundation

Bring in the new iSCSI target and initiator.

Reviewed by: ken (parts)
Approved by: re (delphij)
Sponsored by: FreeBSD Foundation