MFC 361400: Do not try to fill socket send buffer to the last byte.

Setting so_snd.sb_lowat to at least 1/8 of the socket buffer size allows
send thread more actively use PDUs coalescing, that dramatically reduces
TCP lock congestion and number of context switches, when the socket is
full and PDUs are small.

MFC r326730:

Move the DIAGNOSTIC check for lost iSCSI PDUs from icl_conn_close()
to icl_conn_free(). It's perfectly valid for the counter to be non-zero
in the former.

Sponsored by: playkey.net

MFC r328341:

Add SPDX tags to iscsi(4).

MFC r314555: Inline some trivial wrapper functions.

MFC r313779: Fix handling of negative sbspace() return values.

I found that at least with Chelsio NICs TOE sockets quite often report
negative sbspace() values. Using unsigned variable to store it resulted
in attempts to aggregate too much data in one sosend() call, that caused
errors and following connection termination.

MFC r313739: Directly call m_gethdr() instead of m_getm2() for BHS.

All this code is based on assumption that data will be stored in one piece,
and since buffer size if known and fixed, it is easier to hardcode it.

MFC r313731: Do not rely on data alignment after m_pullup().

In general case m_pullup() does not really guarantee any data alignment.
Instead of depenting on side effects caused by data being always copied
out of mbuf cluster (which is probably a bug by itself), always allocate
aligned BHS buffer and read data there directly from socket.

While there, reuse new icl_conn_receive_buf() function to read digests.
The code could probably be even more optimized to aggregate those reads,
but until that done, this is still easier then the way it was before.

MFC r313738: Temporary attach AHS to BHS to calculate header digest.

MFC r313707: Remove M_PKTHDR from m_getm2() in icl_pdu_append_data().

ip_data_mbuf is always appended to ip_bhs_mbuf, so it does not need own
packet header. This change first avoids allocation/initialization of the
header, and then avoids dropping one when it later gets to socket buffer.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Add mechanism for choosing iSER-capable ICL modules.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Provide a way for ICL modules to declare they support PIM_UNMAPPED.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Call the ICL module's handoff method even when using ICL proxy.
The upcoming iSER code uses this.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Add icl_conn_connect() ICL method, required for iSER.

Obtained from: Mellanox Technologies (earlier version)
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Extend the ICL interface to include the PDU pointer in the task_setup
method. This is required for upcoming iSER support.

Obtained from: Mellanox Technologies (earlier version)
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

dev/iscsi: minor spelling fixes.

No functional change.

Reviewed by: trasz

Close some potential races around socket start/close.

There are some reports about panics on ic->ic_socket NULL derefence.
This kind of races is the only way I can imagine it to happen.

MFC after: 2 weeks

Remove icl_conn_connected(); was unused.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Extend ICL to add receive offload methods. For software ICL backend
they are no-ops.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Make output of "iscsictl -v" and "ctladm islist -v" a little prettier
by capitalizing "None".

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Tidy up; no functional changes.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Make it possible to set (via iscsi.conf(5)) and query (via iscsictl -v)
initiator iSCSI offload. Pass maximum data segment size supported by
chosen offload module to iscsid(8), and make iscsid(8) not try to negotiate
anything larger than that.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Use proper module name in MODULE_VERSION().

Sponsored by: The FreeBSD Foundation

Add kobj interface between ICL and the rest of the iSCSI stack.
Review note - icl.c was moved to icl_soft.c.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

For both iSCSI initiator and target increase socket buffer sizes before
establishing connection.

This is a workaround for Chelsio TOE driver, that does not update socket
buffer size in hardware after connection established, and unless that is
done beforehand, kernel code will stuck, attempting to send/receive full
PDU at once.

MFC after: 1 week

In preparation of merging projects/sendfile, transform bare access to
sb_cc member of struct sockbuf to a couple of inline functions:

sbavail() and sbused()

Right now they are equal, but once notion of "not ready socket buffer data",
will be checked in, they are going to be different.

Sponsored by: Netflix
Sponsored by: Nginx, Inc.

s/icl_pdu_new_bhs/icl_pdu_new/; no functional changes, just a little
nicer code.

Sponsored by: The FreeBSD Foundation

s/icl_pdu_new/icl_pdu_new_empty/g; no functional changes.

Sponsored by: The FreeBSD Foundation

Make iSCSI connection close somewhat less aggressive.

It allows to push out some final data from the send queue to the socket
before its close. In particular, it increases chances for logout response
to be delivered to the initiator.

Remove one second wait for threads exit from icl_conn_close().

Switch it from polling with pause() to using cv_wait()/cv_signal().

Use proper include paths in kernel iSCSI code.

MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation

Make the iSCSI stack use __FBSDID() properly.

MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation

Fix potential double free that could happen after connection error.

MFC after: 3 days

Pull in r267961 and r267973 again. Fix for issues reported will follow.

Revert r267961, r267973:

These changes prevent sysctl(8) from returning proper output,
such as:

1) no output from sysctl(8)
2) erroneously returning ENOMEM with tools like truss(1)
or uname(1)
truss: can not get etype: Cannot allocate memory

Extend the meaning of the CTLFLAG_TUN flag to automatically check if
there is an environment variable which shall initialize the SYSCTL
during early boot. This works for all SYSCTL types both statically and
dynamically created ones, except for the SYSCTL NODE type and SYSCTLs
which belong to VNETs. A new flag, CTLFLAG_NOFETCH, has been added to
be used in the case a tunable sysctl has a custom initialisation
function allowing the sysctl to still be marked as a tunable. The
kernel SYSCTL API is mostly the same, with a few exceptions for some
special operations like iterating childrens of a static/extern SYSCTL
node. This operation should probably be made into a factored out
common macro, hence some device drivers use this. The reason for
changing the SYSCTL API was the need for a SYSCTL parent OID pointer
and not only the SYSCTL parent OID list pointer in order to quickly
generate the sysctl path. The motivation behind this patch is to avoid
parameter loading cludges inside the OFED driver subsystem. Instead of
adding special code to the OFED driver subsystem to post-load tunables
into dynamically created sysctls, we generalize this in the kernel.

Other changes:
- Corrected a possibly incorrect sysctl name from "hw.cbb.intr_mask"
to "hw.pcic.intr_mask".
- Removed redundant TUNABLE statements throughout the kernel.
- Some minor code rewrites in connection to removing not needed
TUNABLE statements.
- Added a missing SYSCTL_DECL().
- Wrapped two very long lines.
- Avoid malloc()/free() inside sysctl string handling, in case it is
called to initialize a sysctl from a tunable, hence malloc()/free() is
not ready when sysctls from the sysctl dataset are registered.
- Bumped FreeBSD version to indicate SYSCTL API change.

MFC after: 2 weeks
Sponsored by: Mellanox Technologies

Close the race in older code, that caused connection stuck after r264348.

Reviewed by: trasz
MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Fix typo.

Sponsored by: The FreeBSD Foundation

Remove unused val argument value from SYSCTL_INT() calls.

Improve use of socket buffer upcalls.

Use soreadable()/sowriteable() in socket upcalls to avoid extra wakeups
until we have enough data to read or space to write.

Increase partial receive len from 1K to 128K to not wake up on every
received packet.

This significantly reduces locks congestion and CPU usage and improves
throughput for large I/Os on NICs without TSO and LRO.

Reviewed by: trasz
Sponsored by: iXsystems, Inc.

Remove hack to pass STAILQ to a function and do it properly instead.

Sponsored by: The FreeBSD Foundation

Rework the iSCSI PDU transmit code to avoid lock contention and coalesce
PDUs before sending.

Sponsored by: The FreeBSD Foundation

All the iSCSI sysctls are also tunables; advertise that.

Sponsored by: The FreeBSD Foundation

We don't need TAILQ for iSCSI PDUs; STAILQ is enough.

Sponsored by: The FreeBSD Foundation

Fix build, broken by r264025.

Sponsored by: The FreeBSD Foundation

Enable a KASSERT.

Sponsored by: The FreeBSD Foundation

Get rid of the "autoscaling", instead just set socket buffer sizes
in the usual way. The only thing the old code did was making things
less predictable.

Sponsored by: The FreeBSD Foundation

Instead of "icltx" and "iclrx", use thread names with prefix from upper
layer, so that one can see which side of the stack the threads are for.

Sponsored by: The FreeBSD Foundation

Get rid of ICL lock; use upper-layer (initiator or target) lock instead.
This avoids extra locking in icl_pdu_queue(); the upper layer needs to call
it while holding its own lock anyway, to avoid sending PDUs out of order.

Sponsored by: The FreeBSD Foundation

Move the ic_outstanding_count under #ifdef DIAGNOSTIC.

Sponsored by: The FreeBSD Foundation

Use a less unusual syntax in debug printfs.

Sponsored by: The FreeBSD Foundation

Update kernel inclusions of capability.h to use capsicum.h instead; some
further refinement is required as some device drivers intended to be
portable over FreeBSD versions rely on __FreeBSD_version to decide whether
to include capability.h.

MFC after: 3 weeks

Fix a rare "truncated checksums" problem, which manifested like this:

WARNING: icl_pdu_check_data_digest: data digest check failed; got 0xf23b,
should be 0xdb7f23b

Tested by: Darcy Birkbeck
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation

Fix extremely slow operation with data digests enabled. This was caused
by receive code waiting for data digest even when the data segment was
empty. It didn't actually read it, but it waited until those four bytes
become available in the socket buffer, i.e. until any other PDU (such as NOP)
came in.

PR: kern/185240
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation

Don't spin with mutex hold when there is not enough room in the send socket
buffer. While here, make the code flow somewhat nicer.

Thanks to mav@ for tracking it down.

Tested by: mav
MFC after: 3 days
Sponsored by: FreeBSD Foundation

Fail connection upon receiving too large data segment.

Approved by: re (glebius)
Sponsored by: FreeBSD Foundation

Don't leak memory when removing an unconnected session, and remove useless
UMA_ZONE_NOFREE that caused another leak when unloading the module.

Approved by: re (glebius)
Sponsored by: FreeBSD Foundation

Bring in the new iSCSI target and initiator.

Reviewed by: ken (parts)
Approved by: re (delphij)
Sponsored by: FreeBSD Foundation