MFC r345008:
------------------------------------------------------------------------
r345008 | ken | 2019-03-11 10:21:14 -0400 (Mon, 11 Mar 2019) | 59 lines

Fix CRN resets in the isp(4) driver in certain situations.

The Command Reference Number (CRN) is part of the FC-Tape features
that we enable when talking to tape drives. It starts at 1, and
goes to 255 and wraps around to 1. There are a number of reset
type conditions that result in the CRN getting reset to 1. These
are detailed in section 4.10 (table 8) of the FCP-4r02b specification.

One of the conditions is when a PRLI (Process Login) is sent by
the initiator, and the Establish Image Pair bit is set in Word 0
of the PRLI.

Previously, the isp(4) driver core sent a notification via
isp_async() that the target had changed or stayed in place, but
there was no indication of whether a PRLI was sent and whether the
Establish Image Pair bit was set.

The result of this was that in some situations, notably
switching back and forth between a direct connection and a switch
connection to a tape drive, the isp(4) driver would fail to reset
the CRN in situations that require it according to the spec. When
the CRN isn't reset in a situation that requires it, the tape drive
then rejects every subsequent command that is sent to the drive.
It is assuming that the commands are being sent out of order.

So, modify the isp(4) driver to include Word 0 of the PRLI command
when it sends isp_async() notifications of target changes. Look at
the Establish Image Pair bit, and reset the CRN if that bit is set.

With this change, I am able to switch a tape drive back and forth
between a direct connection and a switch connection, and the isp(4)
driver resets the CRN when it should.

sys/dev/isp_stds.h:
Add bit definitions for PRLI Word 0.

sys/dev/ispmbox.h:
Add PRLI Word 0 to the port database type, isp_pdb_t.

sys/dev/ispvar.h
Add PRLI Word 0 to fcportdb_t.

sys/dev/isp.c:
Populate the new prli_word0 parameter in the port database.

In isp_pdb_add_update(), add a check to see if the
Establish Image Pair bit is set in PRLI Word 0. If it is,
then that is an additional reason to create a change
notification.

sys/dev/isp_freebsd.c:
In isp_async(), if the device changed or stayed, look at
PRLI Word 0 to see if the Establish Image Pair bit is set.
If it is, reset the CRN if we haven't already.

Sponsored by: Spectra Logic
Differential Revision: https://reviews.freebsd.org/D19472

------------------------------------------------------------------------

MFC r330121:
Add support for Enhanced Gen 5 (16Gb) and Gen 6 (32Gb) QLogic FC HBAs.

Sponsored by: iXsystems, Inc.

MFC r320604, r320865:
Switch fabric scans from GID_FT to GID_PT+GFF_ID/GFT_ID.

Instead of using GID_FT SNS request to get list of registered FCP ports,
use GID_PT to get list of all Nx_Ports, and then use GFF_ID and/or GFT_ID
requests to find whether they are FCP and target capable.

The problem with old approach is that GID_FT does not report ports without
FC-4 type registered. In particular it was impossible to boot OS from
FreeBSD FC target using QLogic FC BIOS, since one does not register FC-4
type even on new cards and so ignored by old code as incompatible.

As a side bonus this allows initiator to skip pointless logins to other
initiators by fetching that information from SNS instead.

In case some switches do not implement GFF_ID/GFT_ID correctly, add sysctls
to disable that functionality. I handled broken GFF_ID of my Brocade 200E,
but there may be other switches with different bugs.

Linux also uses GID_PT, but GFF_ID is disabled by default there, and GFT_ID
is not supported.

Sponsored by: iXsystems, Inc.

MFC r317740:

Correct loop mode CRN resets to adhere to FCP-4 section 4.10

Prior to this change, the CRN (Command Reference Number) is reset on any
firmware LIP, LOOP DOWN, or LOOP RESET event in violation of FCP-4 which
specifies that the CRN should only be reset in response to a LIP Reset
(LIPyx) primitive. FCP-4 also indicates PLOGI/LOGO and PRLI/PRLO ELS
actions as conditions for resetting the CRN for the associated initiator
port.

These violations manifest themselves when the HBA is removed from the
loop, or a target device is removed (especially during an outstanding
command) without power cycling. If the HBA and and the target device
determine upon re-establishing the loop that no PLOGI or PRLI is
required, and the target does not issue a LIPxy to the initiator, the
CRN for the target will have been improperly reset by the isp driver. As
a result, the target port will silently ignore all FCP commands issued
during the device probe (which will time out) preventing the device from
attaching.

This change corrects thie CRN reset behavior in response to loop state
changes, also introduces CRN resets for the above mentioned ELS actions
as encountered through async PDB change events.

This change also adds cleanup of outstanding commands in isp_loop_dead()
that was previously missing.

sys/dev/isp/isp.c
Add the last login state to debug output when syncing the pdb

sys/dev/isp/isp_freebsd.c
Replace binary statement setting aborted ccb status in
isp_watchdog() with the XS_SETERR macro used elsewhere

In isp_loop_dead(), abort or complete pending commands as done
in isp_watchdog()

In isp_async(), segregate the ISPASYNC_LOOP_RESET action from
ISPASYNC_LIP, ISPASYNC_LOOP_DOWN, and ISPASYNC_LOOP_UP
fallthroughs, and only reset the CRN in the RESET case. Also add
checks to handle false LOOP RESET actions that do not have a
proper associated LIP primitive, and log the primitive in the
debug messages

In isp_async(), remove the goto from ISP_ASYNC_DEV_STAYED, and
only reset the CRN in the DEV_CHANGED action

In isp_async(), when processing an ISPASYNC_CHANGE_PDB status,
reset CRN(s) for the associated nphdl (or all ports) if the
change reason is some form of ELS login/logout. Also remove
assignment to fc since it is not used in the scope

sys/dev/isp/ispmbox.h
Add macro definition for the global N-Port handle, and correct a
macro typo 'PDB24XX_AE_PRLI_DONJE'

sys/dev/isp/ispvar.h
Add macros FCP_AL_DA_ALL, FCP_AL_PA, and FCP_IS_DEST_ALPD for
more legible code when determining if an AL_PD port matches the
portid for a given struct fcparam* by value or by virtue of the
AL_PD port being 0xFF

Submitted by: Reid Linnemann
Sponsored by: Spectra Logic

MFC r315908: Unify initiator and target DMA setup and command sending.

The code is so alike that it is pointless to keep it separate.

MFC r315708: Cleanup response queue processing.

MFC r315681: Improve command timeout handling.

Let firmware do its best first, and if it can't, try software recovery.
I would remove software timeout handler completely, but found bunch of
complains on command timeout on sparc64 mailing list few years ago, so
better be safe in case of interrupt loss.

MFC r315587, r315652: Remove some dead/useless code.

MFC r315579, r315670: Add initial support for multiple MSI-X vectors.

For 24xx and above use 2 vectors (default and response queue).
For 26xx and above use 3 vectors (default, response and ATIO queues).
Due to global lock interrupt hardlers never run simultaneously now, but
at least this allows to save one regitster read per interrupt.

MFC r315545: Remove hackish code delaying ATIOs to unknown virtual port.

Since we support RQSTYPE_RPT_ID_ACQ, that functionality is only useful
in loop mode, which probably doesn't worth having this hack in 2017.

MFC r315307: Refactor interrupt handling.

Instead of single isp_intr() function doing all possible magic, introduce
four different functions to handle mailbox operation completions, async
events, response and ATIO queues. The goal is to isolate different code
paths to make code more readable, and to make easier support for multiple
interrupt vectors. Even oldest hardware in many cases can identify what
code path it should run on interrupt. Contemporary hardware can assign
them to different interrupt vectors.

MFC r315279: Remove some dead/broken code paths around async handling

MFC r315273: Remove tangled isp_mbox_continue() mechanism.

It was implemented to reduce context switches when uploading firmware to
card's RAM. But this mechanism is not used last 10 years since all mbox
operations are now polled, and it was never used for cards produced in
last 15 years. Newer cards can use DMA to upload firmware.

MFC r315234: Improvements around attach, reset and detach.

This change fixes DMA resource leak on driver unload. Also it removes
DMA resources allocation for hardcoded number of requests before fetching
the real number from firmware. Also it prepares ground for more flexible
IRQs allocation according to firmware capabilities.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Add proper reporting for early task management errors.

This covers unknown requests and requests to unknown virtual ports.
Previously it "worked" only because of timeout handling on initiator.

Completely remove broken now autologin port flag.

Firmware automatically logs in only to local loop ports, and those ports
can be easily identified without extra flag by zero domain and area IDs.

MFC after: 1 week

sys/dev: minor spelling fixes.

Most affect comments, very few have user-visible effects.

Extract virtual port address from RQSTYPE_RPT_ID_ACQ.

This should close the race between request arriving on new target mode
virtual port and its scanner thread finally fetch its address for request
routing.

Filter Port Database Changed notifications.

For some reason firmware sends Port Database Changed notifications in case
of explicit login requests from the driver when target port is unavailabe.
Those notifications don't give driver any new information, but only cause
infinite scan loop.

Respect NVRAM topology settings on 24xx and above chips.

Allocate separate DMA area for synchronous IOCB execution.

Usually IOCBs should be put on queue for asynchronous processing and should
not require additional DMA memory. But there are some cases like aborts and
resets that for external reasons has to be synchronous. Give those cases
separate 2*64 byte DMA area to decouple them from other DMA scratch area
users, using it for asynchronous requests.

Allocate separate scratch space for scanner purposes.

This space does not require DMA syncing. It reduces lock scope of the DMA
scratch space. It allows whole DMA scratch space to be used to I/O, so now
we can fetch up to ~1000 ports from SNS.

Due to the last fact, increase maximal number of ports from 256 to 1024.

Make virtual ports control asynchronous.

Before this change virtual ports control IOCBs were executed synchronously
via Execute IOCB mailbox command. It required exclusive use of scratch
space of driver and mailbox registers of the hardware. Because of that
shared resources use this code could not really sleep, having to spin for
completion, blocking any other operation.

This change introduces new asynchronous design, sending the IOCBs directly
on request queue and gracefully waiting for their return on response queue.
Returned IOCBs are identified with unified handle space from r292725.

Unify handles allocation for initiator and target IOCBs.

I am not sure why this was split long ago, but I see no reason for it.
At this point this unification just slightly reduces memory usage, but
as next step I plan to reuse shared handle space for other IOCB types.

Add initial support for 16Gbps FC QLogic chips.

I still don't know how to read NVRAM there, so WWNs and other parameters
are incorrect, but other then that driver seems like attaching normally.

One more round of port scanner rewrite.

- Make scan aborted by event restart immediately and infinitely.
- Improve handling of some loop events from firmware.
- Remove loop down timer, adding its functionality to scanner thread.
- Some more unification and simplification.

Rip off target mode support for parallel SCSI QLogic adapters.

Hacks to enable target mode there complicated code, while didn't really
work. And for outdated hardware fixing it is not really interesting.

Initiator mode tested with Qlogic 1080 adapter is still working fine.

Fix target mode with fabric for pre-24xx chips.

For those chips we are not receiving login events, adding initiators
based on ATIO requests. But there is no port ID in that structure, so
in fabric mode we have to explicitly fetch it from firmware to be able
to do normal scan after that.

Another round of port scanner rewrite.

This change simplifies and unifies port adding/updating for loop and
fabric scanners. It also fixes problems with scanning restarts due to
concurrent port databases changes. It also fixes many cosmetic issues.

Unify and cleanup FC ports scan.

Reimplement next port handle generation.

For some reason port handles should be allocated from HBA-global space,
while old code was not very specific, mixing per-HBA and per-VP logic.

Skip reserved IP Broadcast handle from using.

Some polishing and unification in ISR code.

Export bunch of state variables as sysctls.

Unify port database use for target and initiator roles.

Aside from cleaner and more consistent code, this allows ports to be both
target and initiator same time, and easily switch from any role to any.

Sponsored by: iXsystems, Inc.

Remove extra level of target ID indirection (isp_dev_map).

FreeBSD never had limitation on number of target IDs, and there is no
any other requirement to allocate them densely. Since slots of port
database already populated just sequentially, there is no much need
for another indirection to allocate sequentially too.

Drop discovered targets when initiator role is disabled.

Close a race in the isp(4) driver that caused devices to disappear
and not automatically come back if they were gone for a short
period of time.

The isp(4) driver has a 30 second gone device timer that gets
activated whenever a device goes away. If the device comes back
before the timer expires, we don't send a notification to CAM that
it has gone away. If, however, there is a command sent to the
device while it is gone and before it comes back, the isp(4) driver
sends the command back with CAM_SEL_TIMEOUT status.

CAM responds to the CAM_SEL_TIMEOUT status by removing the device.
In the case where a device comes back within the 30 second gone
device timer window, though, we weren't telling CAM the device
came back.

So, fix this by tracking whether we have told CAM the device is
gone, and if we have, send a rescan if it comes back within the 30
second window.

ispvar.h:
In the fcportdb_t structure, add a new bitfield,
reported_gone. This gets set whenever we return a command
with CAM_SEL_TIMEOUT status on a Fibre Channel device.

isp_freebsd.c:
In isp_done(), if we're sending CAM_SEL_TIMEOUT for for a
command sent to a FC device, set the reported_gone bit.

In isp_async(), in the ISPASYNC_DEV_STAYED case, rescan the
device in question if it is mapped to a target ID and has
been reported gone.

In isp_make_here(), take a port database entry argument,
and clear the reported_gone bit when we send a rescan to
CAM.

In isp_make_gone(), take a port database entry as an
argument, and set the reported_gone bit when we send an
async event telling CAM consumers that the device is gone.

Sponsored by: Spectra Logic
MFC after: 1 week

Optimize isp(4) to reduce CPU usage, especially in target mode:
- Remove two excessive and slow register reads from isp_intr(). Instead
of rereading value every time, assume that registers contain what we have
written there.
- Avoid sequential search through 4096 array elements when looking for
command tag. Use hash of lists to store active tags separately from free
ones and so greatly speedup the searches.

Reviewed by: mjacob

-----------
MISC CHANGES

Add a new async event- ISP_TARGET_NOTIFY_ACK, that will guarantee
eventual delivery of a NOTIFY ACK. This is tons better than just
ignoring the return from isp_notify_ack and hoping for the best.

Clean up the lower level lun enable code to be a bit more sensible.

Fix a botch in isp_endcmd which was messing up the sense data.

Fix notify ack for SRR to use a sensible error code in the case
of a reject.

Clean up and make clear what kind of firmware we've loaded and
what capabilities it has.
-----------
FULL (252 byte) SENSE DATA

In CTIOs for the ISP, there's only a limimted amount of space
to load SENSE DATA for associated CHECK CONDITIONS (24 or 26
bytes). This makes it difficult to send full SENSE DATA that can
be up to 252 bytes.

Implement MODE 2 responses which have us build the FCP Response
in system memory which the ISP will put onto the wire directly.

On the initiator side, the same problem occurs in that a command
status response only has a limited amount of space for SENSE DATA.
This data is supplemented by status continuation responses that
the ISP pushes onto the response queue after the status response.
We now pull them all together so that full sense data can be
returned to the periph driver.

This is supported on 23XX, 24XX and 25XX cards.

This is also preparation for doing >16 byte CDBs.

-----------
FC TAPE

Implement full FC-TAPE on both initiator and target mode side. This
capability is driven by firmware loaded, board type, board NVRAM
settings, or hint configuration options to enable or disable. This
is supported for 23XX, 24XX and 25XX cards.

On the initiator side, we pretty much just have to generate a command
reference number for each command we send out. This is FCP-4 compliant
in that we do this per ITL nexus to generate the allowed 1 thru 255
CRN.

In order to support the target side of FC-TAPE, we now pay attention
to more of the PRLI word 3 parameters which will tell us whether
an initiator wants confirmed responses. While we're at it, we'll
pay attention to the initiator view too and report it.

On sending back CTIOs, we will notice whether the initiator wants
confirmed responses and we'll set up flags to do so.

If a response or data frame is lost the initiator sends us an SRR
(Sequence Retransmit Request) ELS which shows up as an SRR notify
and all outstanding CTIOs are nuked with SRR Received status. The
SRR notify contains the offset that the initiator wants us to restart
the data transfer from or to retransmit the response frame.

If the ISP driver still has the CCB around for which the data segment
or response applies, it will retransmit.

However, we typically don't know about a lost data frame until we
send the FCP Response and the initiator totes up counters for data
moved and notices missing segments. In this case we've already
completed the data CCBs already and sent themn back up to the periph
driver. Because there's no really clean mechanism yet in CAM to
handle this, a hack has been put into place to complete the CTIO
CCB with the CAM_MESSAGE_RECV status which will have a MODIFY DATA
POINTER extended message in it. The internal ISP target groks this
and ctl(8) will be modified to deal with this as well.

At any rate, the data is retransmitted and an an FCP response is
sent. The whole point here is to successfully complete a command
so that you don't have to depend on ULP (SCSI) to have to recover,
which in the case of tape is not really possible (hence the name
FC-TAPE).

Sponsored by: Spectralogic
MFC after: 1 month

Clean up multi-id mode so it's driven by the f/w loaded,
not by some hint setting. Do more preparations for FC-Tape.
Clean up resource counting for 24XX or later chipsets so
we find out after EXEC_FIRMWARE what is actually supported.
Set target mode exchange count based upon whether or not
we are supporting simultaneous target/initiator mode. Clean
up some old (pre-24XX) xfwoption and zfwoption issues.

Sponsored by: Spectralogic
MFC after: 3 days

Prepare for FC-Tape support. This involved doing a lot of little cleanups
and crosschecks against firmware documentation. We now check and report
FC firmware attributes and at least are now prepared for the upper 48 bits
of f/w attributes (which are probably for the 8100 or later cards). This
involed changing how inbits and outbits are calculated for varios commands,
hopefully clearer and cleaner. This also caused me to clean up the actual
mailbox register usage. Finally, we are now unconditionally using a CRN
for initiator mode.

A longstanding issue with the 2400/2500 is that they do *not* support
a "Prefer PTP followed by loop", which explains why enabling that
caused the f/w to crash.

A slightly more invasive change is to let the firmware load entirely
drive whether multi_id support is enabled or not.

Sponsored by: Spectralogic
MFC after: 1 week

Clean up and complete the incomplete deferred enable code.
Make the default role NONE if target mode is selected. This
allows ctl(8) to switch to/from target mode via knob settings.
If we default to role 'none', this causes a reset of the
24XX f/w which then causes initiators to wake up and notice
when we come online.

Reviewed by: kdm
MFC after: 2 weeks
Sponsored by: Spectralogic

Most of these changes to isp are to allow for isp.ko unloading.
We also revive loop down freezes. We also externaliz within isp
isp_prt_endcmd so something outside the core module can print
something about a command completing. Also some work in progress to
assist in handling timed out commands better.

Partially Sponsored by: Panasas
Approved by: re (kib)
MFC after: 1 month

Sync FreeBSD ISP with mercurial tree. Minor changes having to do with
a macro for minima.

- Use the correct DMA tag/map pair for synchronize the FC scratch area.
- Allocate coherent DMA memory for the request/response queue area and
and the FC scratch area.

These changes allow isp(4) to work properly on sparc64 with usage of the
IOMMU streaming buffers enabled.

Approved by: mjacob
MFC after: 2 weeks

Whap. Hook up some wires that were forgotten a few months ago and restore
the zombie device timeout code and the loop down time code and the fabric
hysteresis code.
MFC after: 1 week
Sponsored By: Panasas

Clean up some printing stuff so that we can have a bit finer control
on debug output. Add a new platform function requirement to allow
for printing based upon the ITL nexus instead of the isp unit plus
channel, target and lun. This allows some printouts and error messages
from the core code to appear in the same format as the platform's
subsystem (in FreeBSD's case, CAM path).

MFC after: 1 week

Redo how commands handles are created and managed and implement sequence
numbers and handle types in rational way. This will better protect from
(unwittingly) dealing with stale handles/commands.

Fix the watchdog timeout code to better protect itself from mistakes.

If we run an abort on a putatively timed out command, the command
may in fact get completed, so check to make sure the command we're
timing it out is still around. If the abort succeeds, btw, the command
should get returned via a different path.

Add 8Gb support (isp_2500). Fix a fair number of configuration and
firmware loading bugs.

Target mode support has received some serious attention to make it
more usable and stable.

Some backward compatible additions to CAM have been made that make
target mode async events easier to deal with have also been put
into place.

Further refinement and better support for NP-IV (N-port Virtualization)
is now in place.

Code for release prior to RELENG_7 has been stripped away for code clarity.

Sponsored by: Copan Systems

Reviewed by: scottl, ken, jung-uk kim
Approved by: re

Recover from some major omissions/problems with the 24XX port.
First, we were never correctly checking for a 24XX Status Type 0
response- that cased us to fall through to evaluate status for
commands as if this were a 2100/2200/2300 Status Type 0 response.
This is *close*, but not quite the same. This has been reported
to be apparent with some wierd lun configuration problems with
some arrays. It became glaringly apparent on sparc64 where none
of the correct byte swap things were done.

Fixing this omission then caused a whole universe shifting debug
cycle of endian issues for the 2400. The manual for 24XX f/w turns
out to be wrong about the endianness of a couple of entities. The
lun and cdb fields for the type 7 request are *not* unconditionally
big endian- they happen to be opposite of whatever the endian of
the current machine type is. Same with the sense data for the
24XX type 0 response.

While we're at it investigate and resolve some NVRAM endian
issues.

Approved by: re (ken)
MFC after: 3 days

MFP4: a) Some constification from NetBSD (gcc 4.1.2)
b) Split default param fetching/setting into scsi and fibre functions
and retry the fibre fetch more than once.

MFC after: 1 week

Fix some stupid copyright mistakes that have been there for quite some time.

MFP4: Move default setting to the end of isp_reset instead of the
front of isp_init so we can read NVRAM even if we're role ISP_NONE.
Prepare for reintroduction of channels (for FC) for N-Port
Virtualization.

Fix a botch in handle assignment that caused us to nuke one device
when a new one arrives and end up with two devices with the same
identity in the virtual target mapping table.

Make ISPCTL_PLOGX find a handle to log into the management server
with- not hope for the best. Change some things which were gated
off of 24XX to be gated off of 2K login support. Convert some
isp_prt calls to xpt_print calls.

Make the SAN login/logout stuff more common between different chipsets
and provied an isp_control entry point so that the outer layers can
do PLOGI/LOGO explicitly. Add MS IOCB support. This completes the cycle
for base support for SMI-S.

Push things closer to path failover by implementing loop down and
gone device timers and zombie state entries. There are tunables
that can be used to select a number of parameters.

loop_down_limit - how long to wait for loop to come back up before
declaring
all devices dead (default 300 seconds)

gone_device_time- how long to wait for a device that has appeared
to leave the loop or fabric to reappear (default 30 seconds)

Internal tunables include (which should be externalized):

quick_boot_time- how long to wait when booting for loop to come up

change_is_bad- whether or not to accept devices with the same
WWNN/WWPN that reappear at a different PortID as being the 'same'
device.

Keen students of some of the subtle issues here will ask how
one can keep devices from being re-accepted at all (the answer
is to set a gone_device_time to zero- that effectively would
be the same thing).

Add 4Gb (24XX) support and lay the foundation for a lot of new stuff.

The macro IS_23XX should not mistakenly include 24XX cards.

Do various fixes to support firmware loading for the 2322
(and by extension, the 2422).

One peculiar thing I've found with the 2322 is that if you
don't force it to do Hard LoopID acquisition, the firmware
crashes. This took a while to figure out.

While we're at it, fix various bugs having to do with NVRAM
reading and option setting with respect to pieces of NVRAM.

Fix longstanding bug where exec throttle is 16 bits- not 8.

a) clean up some declaration stuff (i.e., make more modern with respect
to getting rid u_int for uint and so on).

b) Turn back on 64 bit DAC support. Cheeze it a bit in that we have two
DMA callback functions- one when we have bus_addr_t > 4 bits in width and
the other which should be normal. Even Cheezier in that we turn off setting
up DMA maps to be BUS_SPACE_MAXADDR if we're in ISP_TARGET_MODE. More work
on this in a week or so.

c) Tested under amd64 and 1MB DFLTPHYS, sparc64, i386 (PAE, but insufficient
memory to really test > 4GB). LINT check under amd64.

MFC after: 1 month

Remove use of inlines and use the functions as a library.

Larger code space, possibly performance hit, but more portable.
Certainly less questionable use of inlining.

Suggested by: des

First of several commits as this driver is dusted off and maybe brought
up to date. Principle changes for this reelase is to support 2K Port Login
firmware. This allows us to support the 2322 (and 2422 4Gb) cards which only
come with the 2K Port Login firmware. The 2322 should now work- but we don't
have firmware sets for it in ispfw (as the change to load 2K Port Login f/w
hasn't been made- that f/w is so big it has to be loaded in more than one
chunk).

Other changes are the beginnings of cleaning up some long standing target
mode issues. The next changes here will incorporate a lot of bug fixes
from others.

Finally, some copyright cleanup and attempts to make the parts of the
driver that are FreeBSD specific start conforming more to FreeBSD style.

MFC after: 1 month

Refactor isp_prt declaration so that platform
requirements can stay in platform files.

Prefer <sys/cdefs.h>'s __printflike() macro to the recently added
__GNUCLIKE_ATTRIBUTE_PRINTF.

Approved by: mjacob

netchild's mega-patch to isolate compiler dependencies into a central
place.

This moves the dependency on GCC's and other compiler's features into
the central sys/cdefs.h file, while the individual source files can
then refer to #ifdef __COMPILER_FEATURE_FOO where they by now used to
refer to #if __GNUC__ > 3.1415 && __BARC__ <= 42.

By now, GCC and ICC (the Intel compiler) have been actively tested on
IA32 platforms by netchild. Extension to other compilers is supposed
to be possible, of course.

Submitted by: netchild
Reviewed by: various developers on arch@, some time ago

Roll minor number.

MFC after: 2 weeks

Start each of the license/copyright comments with /*-, minor shuffle of lines

Store the target handles in a separate list from normal commands. Add a
CTIO fast post routine to handle CTIO completions.

Submitted by: mjacob

These are changes to allow to use the Intel C/C++ compiler (lang/icc)
to build the kernel. It doesn't affect the operation if gcc.

Most of the changes are just adding __INTEL_COMPILER to #ifdef's, as
icc v8 may define __GNUC__ some parts may look strange but are
necessary.

Additional changes:
- in_cksum.[ch]:
* use a generic C version instead of the assembly version in the !gcc
case (ASM code breaks with the optimizations icc does)
-> no bad checksums with an icc compiled kernel
Help from: andre, grehan, das
Stolen from: alpha version via ppc version
The entire checksum code should IMHO be replaced with the DragonFly
version (because it isn't guaranteed future revisions of gcc will
include similar optimizations) as in:
---snip---
Revision Changes Path
1.12 +1 -0 src/sys/conf/files.i386
1.4 +142 -558 src/sys/i386/i386/in_cksum.c
1.5 +33 -69 src/sys/i386/include/in_cksum.h
1.5 +2 -0 src/sys/netinet/igmp.c
1.6 +0 -1 src/sys/netinet/in.h
1.6 +2 -0 src/sys/netinet/ip_icmp.c

1.4 +3 -4 src/contrib/ipfilter/ip_compat.h
1.3 +1 -2 src/sbin/natd/icmp.c
1.4 +0 -1 src/sbin/natd/natd.c
1.48 +1 -0 src/sys/conf/files
1.2 +0 -1 src/sys/conf/files.amd64
1.13 +0 -1 src/sys/conf/files.i386
1.5 +0 -1 src/sys/conf/files.pc98
1.7 +1 -1 src/sys/contrib/ipfilter/netinet/fil.c
1.10 +2 -3 src/sys/contrib/ipfilter/netinet/ip_compat.h
1.10 +1 -1 src/sys/contrib/ipfilter/netinet/ip_fil.c
1.7 +1 -1 src/sys/dev/netif/txp/if_txp.c
1.7 +1 -1 src/sys/net/ip_mroute/ip_mroute.c
1.7 +1 -2 src/sys/net/ipfw/ip_fw2.c
1.6 +1 -2 src/sys/netinet/igmp.c
1.4 +158 -116 src/sys/netinet/in_cksum.c
1.6 +1 -1 src/sys/netinet/ip_gre.c
1.7 +1 -2 src/sys/netinet/ip_icmp.c
1.10 +1 -1 src/sys/netinet/ip_input.c
1.10 +1 -2 src/sys/netinet/ip_output.c
1.13 +1 -2 src/sys/netinet/tcp_input.c
1.9 +1 -2 src/sys/netinet/tcp_output.c
1.10 +1 -1 src/sys/netinet/tcp_subr.c
1.10 +1 -1 src/sys/netinet/tcp_syncache.c
1.9 +1 -2 src/sys/netinet/udp_usrreq.c

1.5 +1 -2 src/sys/netinet6/ipsec.c
1.5 +1 -2 src/sys/netproto/ipsec/ipsec.c
1.5 +1 -1 src/sys/netproto/ipsec/ipsec_input.c
1.4 +1 -2 src/sys/netproto/ipsec/ipsec_output.c

and finally remove
sys/i386/i386 in_cksum.c
sys/i386/include in_cksum.h
---snip---
- endian.h:
* DTRT in C++ mode
- quad.h:
* we don't use gcc v1 anymore, remove support for it
Suggested by: bde (long ago)
- assym.h:
* avoid zero-length arrays (remove dependency on a gcc specific
feature)
This change changes the contents of the object file, but as it's
only used to generate some values for a header, and the generator
knows how to handle this, there's no impact in the gcc case.
Explained by: bde
Submitted by: Marius Strobl <marius@alchemy.franken.de>
- aicasm.c:
* minor change to teach it about the way icc spells "-nostdinc"
Not approved by: gibbs (no reply to my mail)
- bump __FreeBSD_version (lang/icc needs to know about the changes)

Incarnations of this patch survive gcc compiles since a loooong time,
I use it on my desktop. An icc compiled kernel works since Nov. 2003
(exceptions: snd_* if used as modules), it survives a build of the
entire ports collection with icc.

Parts of this commit contains suggestions or submissions from
Marius Strobl <marius@alchemy.franken.de>.

Reviewed by: -arch
Submitted by: netchild

Change role defines so that they better match class 3 service parameters.
Add ISPCTL_GET_PDB isp_control operation.

MFC after: 1 week

NetBSD'er with time on their hands (dma->DMA)

This should enable 10160 support. As best as I can tell, the same
f/w as 12160 is used, and otherwise, this is just a single channel
variant of the 10160.

MFC after: 0 days

Remove ISP_DMA_ADDR_T definition.

Instead, based upon whether ISP_DAC_SUPPORTED is defined, typedef
isp_dma_addr_t appropriately.

If ISP_DAC_SUPPORTRED is defined, the DMA_WD2/DMA_WD3 macros do something
useful, else they define to '0'.

Remove STRNCAT (==>strncat) usage. Apparently I never read the man
page correctly and it wasn't doing what I thought it was.

Noticed by: Brooks Davis <brooks@one-eyed-alien.net>

Add ISPASYNC_FW_DUMPED async event.

add ISP_FW_NEWER_THAN macro- makes the code easier to read.

Roll minor version. Add ISPASYNC_FW_RESTARTED async event. Add
DEFAULT_FRAMESIZE && DEFAULT_EXEC_THROTTLE references.

MFC after: 1 week

Fix bus dma segment count to be based off of MAXPHYS, not BUS_SPACE_MAXSIZE.
Grumble. I've seen better documented architectures out of Redmond.

Redo fabric evaluation to not use GET ALL NEXT (GA_NXT). Switches seem
to be trying to wriggle out of supporting this well. Instead, use
GID_FT to get a list of Port IDs and then use GPN_ID/GNN_ID to find the
port and node wwn. This should make working on fabrics a bit cleaner and
more stable.

This also caused some cleanup of SNS subcommand canonicalization so that
we can actually check for FS_ACC and FS_RJT, and if we get an FS_RJT,
print out the reason and explanation codes.

We'll keep the old GA_NXT method around if people want to uncomment a
controlling definition in ispvar.h.

This also had us clean up ISPASYNC_FABRICDEV to use a local lportdb argument
and to have the caller explicitly say that a device is at the end of the
fabric list.

MFC after: 1 week

Redo stuff for sparc64- primarily fix bus dma implementation. The endian
stuff was right, but the busdma stuff was massively not right.

Didn't really test on ia64 or i386- don't have the former h/w and my
FreeBSD-current disk is unwell right now. Hope that this is okay.

MFC after: 1 week

Limit fabric search to a default 256 entries. This will all go away
soon because it's just getting harder and harder to find switches
that correctly implement the GET ALL NEXT subcommands for the SNS
protocol.

Latch up result out pointer and set a busy flag when we're looking
at the response queue. This allows for a cleaner way to make sure
we don't get multiple CPUs trying to read the same response queue
entries.

Change how isp_handle_other_response returns values (clarity).

Make PORT UNAVAILABLE the same as PORT LOGOUT (force a LIP).

Do some formatting changes.

MFC after: 0 days

Add in support firmware crash dumps. Change CFG options to split
WWN into WWNN and WWPN.

MFC after: 1 week

+ A variety of 23XX changes:
disable MWI on 2300

based on function code, set an 'isp_port' for the 2312- it's a
separate instance, but the NVRAM is shared, and the second port's
NVRAM is at offset 256.

+ Enable RIO operation for LVD SCSI cards. This makes a *big* difference
as even under reasonable load we get batched completions of about 30
commands at a time on, say, an ISP1080.

+ Do 'continuation' mailbox commands- this allows us to specify a work
area within the softc and 'continue' repeated mailbox commands. This is
more or less on an ad hoc basis and is currently only used for firmware
loading (which f/w now loads substantially faster becuase the calling
thread is only woken when all the f/w words are loaded- not for each
one of the 40000 f/w words that gets loaded).

+ If we're about to return from isp_intr with a 'bogus interrupt' indication,
and we're not a 23XX card, check to see whether the semaphore register is
currently *2* (not *1* as it should be) and whether there's an async completion
sitting in outgoing mailbox0. This seems to capture cases of lost fast posting
and RIO interrupts that the 12160 && 1080 have been known to pump out under
extreme load (extreme, as in > 250 active commands).

+ FC_SCRATCH_ACQUIRE/FC_SCRATCH_RELEASE macros.

+ Endian correct swizzle/unswizzle of an ATIO2 that has a WWPN in it.

MFC after: 1 week

Implement REDUCED INTERRUPT OPERATION usage form FC cards- this allows the
firmware to delay completion of commands so that it can attempt to batch
a bunch of completions at once- either returning 16 bit handles in mailbox
registers, or in a resposne queue entry that has a whole wad of 16 bit handles.

Distinguish between 2300 and 2312 chipsets- if only because the revisions
on the chips have different meanings.

Add more instrumentation plus ISP_GET_STATS and ISP_CLR_STATS ioctls.
Run up the maximum number of response queue entities we'll look at
per interrupt.

If we haven't set HBA role yet, always return success from isp_fc_runstate.

MFC after: 2 weeks

Major restructuring for swizzling to the request queue and unswizzling from
the response queue. Instead of the ad hoc ISP_SWIZZLE_REQUEST, we now have
a complete set of inline functions in isp_inline.h. Each platform is
responsible for providing just one of a set of ISP_IOX_{GET,PUT}{8,16,32}
macros.

The reason this needs to be done is that we need to have a single set of
functions that will work correctly on multiple architectures for both little
and big endian machines. It also needs to work correctly in the case that
we have the request or response queues in memory that has to be treated
specially (e.g., have ddi_dma_sync called on it for Solaris after we update
it or before we read from it). It also has to handle the SBus cards (for
platforms that have them) which, while on a Big Endian machine, do *not*
require *most* of the request/response queue entry fields to be swizzled
or unswizzled.

One thing that falls out of this is that we no longer build requests in the
request queue itself. Instead, we build the request locally (e.g., on the
stack) and then as part of the swizzling operation, copy it to the request
queue entry we've allocated. I thought long and hard about whether this was
too expensive a change to make as it in a lot of cases requires an extra
copy. On balance, the flexbility is worth it. With any luck, the entry that
we build locally stays in a processor writeback cache (after all, it's only
64 bytes) so that the cost of actually flushing it to the memory area that is
the shared queue with the PCI device is not all that expensive. We may examine
this again and try to get clever in the future to try and avoid copies.

Another change that falls out of this is that MEMORYBARRIER should be taken
a lot more seriously. The macro ISP_ADD_REQUEST does a MEMORYBARRIER on the
entry being added. But there had been many other places this had been missing.
It's now very important that it be done.

Additional changes:

Fix a longstanding buglet of sorts. When we get an entry via isp_getrqentry,
the iptr value that gets returned is the value we intend to eventually plug
into the ISP registers as the entry *one past* the last one we've written-
*not* the current entry we're updating. All along we've been calling sync
functions on the wrong index value. Argh. The 'fix' here is to rename all
'iptr' variables as 'nxti' to remember that this is the 'next' pointer-
not the current pointer.

Devote a single bit to mboxbsy- and set aside bits for output mbox registers
that we need to pick up- we can have at least one command which does not
have any defined output registers (MBOX_EXECUTE_FIRMWARE).

MFC after: 2 weeks

Implement a call to get the actual link data rate (if 23XX) so we can
set whether it's a 2Gps or 1Gps link.

MFC after: 1 week

Add some more firmware revision macros. Add firmware attributes field
to fcparam structure.
MFC after: 4 weeks

Add 2 Gigabit Fibre Channel support (2300 && 2312 cards). This required
some reworking (and consequent cleanup) of the interrupt service code.

Also begin to start a cleanup of target mode support that will (eventually)
not require more inforamtion routed with the ATIO to come back with the
CTIO other than tag.

MFC after: 4 weeks

Fix a spelling error in a comment.

Redo how we manage SCSI device settings- have a 3rd flags (nvram) that records
either what's in NVRAM or what the safe defaults would be if we lack NVRAM.
Then we rename cur_XXXX to actv_XXXX (these are the currently active settings)
and the dev_XXX settings to goal_XXXX (these are the settings which we want
cur_XXXX to converge to).

Roll core minor.

Firmware crashes handled in platform specific code (isp_async call).

Fix longstanding silly buglet that left a hole in the debug log defines.

Fix botch for state levels. Role minor release. Start adding code for a
'force logout' path.

MFC after: 4 weeks

Spring MegaChange #1.

----

Make a device for each ISP- really usable only with devfs and add an ioctl
entry point (this can be used to (re)set debug levels, reset the HBA,
rescan the fabric, issue lips, etc).

----

Add in a kernel thread for Fibre Channel cards. The purpose of this
thread is to be woken up to clean up after Fibre Channel events
block things. Basically, any FC event that casts doubt on the
location or identify of FC devices blocks the queues. When, and
if, we get the PORT DATABASE CHANGED or NAME SERVER DATABASE CHANGED
async event, we activate the kthread which will then, in full thread
context, re-evaluate the local loop and/or the fabric. When it's
satisfied that things are stable, it can then release the blocked
queues and let commands flow again.

The prior mechanism was a lazy evaluation. That is, the next command
to come down the pipe after change events would pay the full price
for re-evaluation. And if this was done off of a softcall, it really
could hang up the system.

These changes brings the FreeBSD port more in line with the Solaris,
Linux and NetBSD ports. It also, more importantly, gets us being
more proactive about topology changes which could then be reflected
upwards to CAM so that the periph driver can be informed sooner
rather than later when things arrive or depart.

---

Add in the (correct) usage of locking macros- we now have lock transition
macros which allow us to transition from holding the CAM lock (Giant)
and grabbing the softc lock and vice versa. Switch over to having this
HBA do real locking. Some folks claim this won't be a win. They're right.
But you have to start somewhere, and this will begin to teach us how
to DTRT for HBAs, etc.

--

Start putting in prototype 2300 support. Add back in LIP
and Loop Reset as async events that each platform will handle.
Add in another int_bogus instrumentation point.

Do some more substantial target mode cleanups.

MFC after: 8 weeks

In order to save ourselves grief with the SUNPRO compiler under
Solaris (which, for reasons unknown to me, chokes on u_int16_t
as a typedef of unsigned short if used in a transitional (mixed K&R
and ANSI) way), we'll go the extra mile and fully ANSIfy things.

More 32 to 16 bit handle stuff. Roll core minor version.

Fix a longstanding bug- we had the sense of what bit 14
for the ICB firmware options meant- *I* had taken it to
mean that if you set it, Node Name would be ignored and
derived from Port Name. Actually, it meant the opposite.
As a consequence- change ICBOPT_USE_PORTNAME to the
define ICBOPT_BOTH_WWNS- makes more sense.

Fix wrong input bitmap for MBOX_DUMP_RAM command. Call
ISP_DUMPREGS if we get a f/w crash. Add ISPCTL_RUN_MBOXCMD
control command (so outer layers can run a mailbox command
directly) and add a ISPASYNC_UNHANDLED_RESPONSE hook so
outer layers can understand response queue entries we
might not know about.

Eliminate ISP2100_FABRIC- we always allow for fabric now. Add an
isp_iid_set/isp_iid for fibre channel- this is because we now
fake a port database entry for ourselves. Add the additional loop
states between LOOP_PDB_RCVD and LOOP_READY.

Change and comment on a wad of Fibre Channel isp_control functions.
Change and comment on some of the ISPASYNC Fibre Channel events.

Add was_fabric_dev/fabric_dev tags to our local FC database structure
(so we can see rapidly whether something was a fabric device but is
now gone).

Add a tag which says what role this adapter should take. It can take
on the value of None, Target, Initiator or Both. None is useful for
warm failover purposes. Remove the ISP_CFG_NOINIT silliness since
a role of "None" does this.

Add a isp_lastmbxcmd tag to store the opcode for the last mailbox
command used.

ISPASYNC_PDB_CHANGED -> ISPASYNC_LOGGED_INOUT.

Add in Bill Sommerfeld's -Wformat stuff. Add a ISP_CFG_NOINIT option
to keep from completing initialization when isp_init is called.

Add interrupt instrumentation. Change ISP_CFG_NPORT config option to
a set of options that allows specific loop, loop-only, nport, nport-only
topology settings. Define a required macro for all platforms (USEC_SLEEP).

Change some default macro usages/definitions/requirements.

some copyright cleanups

various fixes

Rewrite for version 2.0. Some structural changes, but also
a substantial amount of commenting about what each platform
specific definitions are supposed to be.

Remove obsolete isp_dogactive tag.

Add 8 bits of volatile mailbox busy mask- this will be the bitmask of
output mailbox values we want to get back out of the chip once a mailbox
command is done. Add storage for the maximum number of output mailbox
registers to the softc.

Roll minor version number.

Roll core minor version. Set ISP_MAX_LUNS to be off of new isp_maxluns
tag in softc.

Roll core minor version. Change our 'fabdev' tag to 'loggedin'.

Roll minor version. Increase size (and add defines for) topology storage.

Add 12160 (Ultra3) defines. Add config option flag for forcing point-to-point
instead of FC-AL (2200 only).

Approved: jkh@freebsd.org

include public target mode functions

Roll minor revision number and add a more finalized list
of target mode related enums.

Add Dual LVD bus (1280) support

Fix some includes for when we (eventually) get target mode working again.
Role the core version minor number. Change the arguments to the dma
setup function to use a u_int16_t for the output request loop pointer
(truly amazing that this hasn't blown up in anyones face so far). Do
some shuffling around of some items.

Roll core version number. Do some stylistic changes. Ensure that
the result queue length is never less than 64. Move (ick) temp port
database used for post-LIP merging off the kernel stack and put it
into the softc. Remove some target mode stuff which will come back
later in a different file. Change how the list of outstanding commands
are stored (now allocated at mailbox setup time to be just enough for
the max for a specific HBA which can vary). Keep a rotating seed of
the last index for this in the softc. Increase the count of active
commands from 10 to 16 bits.

$Id$ -> $FreeBSD$

Clarify lun limits for FC && SCSI.

add 2200 f/w; fix botched define

Roll revision levels. Move DEFAULT_LOOPID definition to platform files.
Change some fcp parameter structures such that we can get the portid
(24 bit value), get both node and port WWN, know whether we're on a fabric
or not, note whether we've ever seen the loop up, and note the current
state of the loop.

Replace the isp_pdb_t structure in fcparams with a reduced cost structure
that maintains a static relationship to 'Target', but can have the actual
loop ID used change (in case, post LIP, we discover things have moved
around). This also retains portid and node/port WWNs. This array gets
larger if we have fabric support compiled in.

Note special loop IDs that are invariate for this device- FL_PORT_ID
(0x7e) which tells us if there's a fabric controller present, FC_PORT_ID
and FC_SNS_ID (fabric controller port and fabric SNS server port). We don't
use the latter two for anything. IDs above FC_SNS_ID up through 255 are
available for mapping fabric devices to 'target' ids.

Add in a config define to set FC full duplex mode. Add in a define to
recognize the Qlogic 2200 boards. Add comments about ISPCTL commands.
Add and change some ISPASYNC enumes.

Bruce pointed out I was being silly with volatile.
Submitted by: bde@freebsd.org

Roll core minor number. Re-layout a lot of SCSI fields to accomodate multiple
bus adapters.

Change f/w revision to major,minor,micro version. Add in ISP1040C definition.

Add in 1080 LVD support and some basis also for the 1240. The port database
printout is now enabled.

A wad of changes- prepping for 1080/1240 support (which caused a massive
thwank in register layout goop). A different mboxcmd approach. Some PDB change
infrastructure. Some better management of loopdown/loopup events (keep them
distinct from resource starvation for simq freeze/unfreeze actions).

Roll internal release tag. Roll core version minor. Fix broken DPARM_DEFAULT
define. Add a new config flag param (ISP_CFG_NONVRAM) whose intent it is
to cause NVRAM to be ignored. Add ISPASYNC_LOOP_DOWN and ISPASYNC_LOOP_UP
isp_async enums.

Amazingly enough, I did all my scsi_sa work recently without realizing
that I had a broken isp card whose (unchangeable- it's an old old old
isp1020) NVRAM has sync mode enabled, but disconnect/reconnect disabled-
the ISP_CFG_NONVRAM is definitely warranted when you want to bloody well
ignore the NVRAM and set something sensible.

Implement and use Fast Posting for both parallel && fibre. Redo a bit of
the startup code. Implement a call to outer framework function so that
asynchronous events can be handled (e.g., speed negotiation, target mode).

Roll internal release tags.

up isp_fifo_threshold NVRAM storage to 3 bits (for future 128 bit)

clarify headers;move uninit to outer layer;remove watchdog

roll core version minor and wire a non-i386 default Loop ID to 113

per bde (who is right about this) that an inlined fucntion with const
char * strings being returned defined in a header file included several
places but only used in one module, is, uh, silly.

Update QLogic ISP support for CAM. Add preliminary target mode support.

Submitted by: Matthew Jacob <mjacob@feral.com>

Fixed pedantic semantics errors (bitfields not of type int, signed int
or unsigned int (this doesn't change the struct layout, size or
alignment in any of the files changed in this commit, at least for
gcc on i386's. Using bitfields of type u_char may affect size and
alignment but not packing)).

Support compiling with `gcc -ansi'. Just use __inline instead of inline.
[__]inline is only used to bloat the code here. It gives a separate copy
of all the strings for each time this header is included...

Fixed misuse of __P(()).

Add support for the Qlogic ISP SCSI && FC/AL Adapters