MFC r317740:

Correct loop mode CRN resets to adhere to FCP-4 section 4.10

Prior to this change, the CRN (Command Reference Number) is reset on any
firmware LIP, LOOP DOWN, or LOOP RESET event in violation of FCP-4 which
specifies that the CRN should only be reset in response to a LIP Reset
(LIPyx) primitive. FCP-4 also indicates PLOGI/LOGO and PRLI/PRLO ELS
actions as conditions for resetting the CRN for the associated initiator
port.

These violations manifest themselves when the HBA is removed from the
loop, or a target device is removed (especially during an outstanding
command) without power cycling. If the HBA and and the target device
determine upon re-establishing the loop that no PLOGI or PRLI is
required, and the target does not issue a LIPxy to the initiator, the
CRN for the target will have been improperly reset by the isp driver. As
a result, the target port will silently ignore all FCP commands issued
during the device probe (which will time out) preventing the device from
attaching.

This change corrects thie CRN reset behavior in response to loop state
changes, also introduces CRN resets for the above mentioned ELS actions
as encountered through async PDB change events.

This change also adds cleanup of outstanding commands in isp_loop_dead()
that was previously missing.

sys/dev/isp/isp.c
Add the last login state to debug output when syncing the pdb

sys/dev/isp/isp_freebsd.c
Replace binary statement setting aborted ccb status in
isp_watchdog() with the XS_SETERR macro used elsewhere

In isp_loop_dead(), abort or complete pending commands as done
in isp_watchdog()

In isp_async(), segregate the ISPASYNC_LOOP_RESET action from
ISPASYNC_LIP, ISPASYNC_LOOP_DOWN, and ISPASYNC_LOOP_UP
fallthroughs, and only reset the CRN in the RESET case. Also add
checks to handle false LOOP RESET actions that do not have a
proper associated LIP primitive, and log the primitive in the
debug messages

In isp_async(), remove the goto from ISP_ASYNC_DEV_STAYED, and
only reset the CRN in the DEV_CHANGED action

In isp_async(), when processing an ISPASYNC_CHANGE_PDB status,
reset CRN(s) for the associated nphdl (or all ports) if the
change reason is some form of ELS login/logout. Also remove
assignment to fc since it is not used in the scope

sys/dev/isp/ispmbox.h
Add macro definition for the global N-Port handle, and correct a
macro typo 'PDB24XX_AE_PRLI_DONJE'

sys/dev/isp/ispvar.h
Add macros FCP_AL_DA_ALL, FCP_AL_PA, and FCP_IS_DEST_ALPD for
more legible code when determining if an AL_PD port matches the
portid for a given struct fcparam* by value or by virtue of the
AL_PD port being 0xFF

Submitted by: Reid Linnemann
Sponsored by: Spectra Logic

MFC r315908: Unify initiator and target DMA setup and command sending.

The code is so alike that it is pointless to keep it separate.

MFC r315708: Cleanup response queue processing.

MFC r315681: Improve command timeout handling.

Let firmware do its best first, and if it can't, try software recovery.
I would remove software timeout handler completely, but found bunch of
complains on command timeout on sparc64 mailing list few years ago, so
better be safe in case of interrupt loss.

MFC r315587, r315652: Remove some dead/useless code.

MFC r315579, r315670: Add initial support for multiple MSI-X vectors.

For 24xx and above use 2 vectors (default and response queue).
For 26xx and above use 3 vectors (default, response and ATIO queues).
Due to global lock interrupt hardlers never run simultaneously now, but
at least this allows to save one regitster read per interrupt.

MFC r315545: Remove hackish code delaying ATIOs to unknown virtual port.

Since we support RQSTYPE_RPT_ID_ACQ, that functionality is only useful
in loop mode, which probably doesn't worth having this hack in 2017.

MFC r315307: Refactor interrupt handling.

Instead of single isp_intr() function doing all possible magic, introduce
four different functions to handle mailbox operation completions, async
events, response and ATIO queues. The goal is to isolate different code
paths to make code more readable, and to make easier support for multiple
interrupt vectors. Even oldest hardware in many cases can identify what
code path it should run on interrupt. Contemporary hardware can assign
them to different interrupt vectors.

MFC r315279: Remove some dead/broken code paths around async handling

MFC r315273: Remove tangled isp_mbox_continue() mechanism.

It was implemented to reduce context switches when uploading firmware to
card's RAM. But this mechanism is not used last 10 years since all mbox
operations are now polled, and it was never used for cards produced in
last 15 years. Newer cards can use DMA to upload firmware.

MFC r315234: Improvements around attach, reset and detach.

This change fixes DMA resource leak on driver unload. Also it removes
DMA resources allocation for hardcoded number of requests before fetching
the real number from firmware. Also it prepares ground for more flexible
IRQs allocation according to firmware capabilities.

MFC r300218: Add proper reporting for early task management errors.

This covers unknown requests and requests to unknown virtual ports.
Previously it "worked" only because of timeout handling on initiator.

MFC r300052: Completely remove broken now autologin port flag.

Firmware automatically logs in only to local loop ports, and those ports
can be easily identified without extra flag by zero domain and area IDs.

MFC r297991: Extract virtual port address from RQSTYPE_RPT_ID_ACQ.

This should close the race between request arriving on new target mode
virtual port and its scanner thread finally fetch its address for request
routing.

MFC r297915: Filter Port Database Changed notifications.

For some reason firmware sends Port Database Changed notifications in case
of explicit login requests from the driver when target port is unavailabe.
Those notifications don't give driver any new information, but only cause
infinite scan loop.

MFC r297912: Respect NVRAM topology settings on 24xx and above chips.

MFC r297858: Allocate separate DMA area for synchronous IOCB execution.

Usually IOCBs should be put on queue for asynchronous processing and should
not require additional DMA memory. But there are some cases like aborts and
resets that for external reasons has to be synchronous. Give those cases
separate 2*64 byte DMA area to decouple them from other DMA scratch area
users, using it for asynchronous requests.

MFC r292765: Allocate separate scratch space for scanner purposes.

This space does not require DMA syncing. It reduces lock scope of the DMA
scratch space. It allows whole DMA scratch space to be used to I/O, so now
we can fetch up to ~1000 ports from SNS.

Due to the last fact, increase maximal number of ports from 256 to 1024.

MFC r292739: Make virtual ports control asynchronous.

Before this change virtual ports control IOCBs were executed synchronously
via Execute IOCB mailbox command. It required exclusive use of scratch
space of driver and mailbox registers of the hardware. Because of that
shared resources use this code could not really sleep, having to spin for
completion, blocking any other operation.

This change introduces new asynchronous design, sending the IOCBs directly
on request queue and gracefully waiting for their return on response queue.
Returned IOCBs are identified with unified handle space from r292725.

MFC r292725: Unify handles allocation for initiator and target IOCBs.

I am not sure why this was split long ago, but I see no reason for it.
At this point this unification just slightly reduces memory usage, but
as next step I plan to reuse shared handle space for other IOCB types.

MFC r291654, r291727, r291821, r291872, r292034, r292041, r292249, r292042:
Add initial support for 16Gbps FC QLogic chips.

MFC r291365, r291369: One more round of port scanner rewrite.

- Make scan aborted by event restart immediately and infinitely.
- Improve handling of some loop events from firmware.
- Remove loop down timer, adding its functionality to scanner thread.
- Some more unification and simplification.

MFC r291188: Rip off target mode support for parallel SCSI QLogic adapters.

Hacks to enable target mode there complicated code, while didn't really
work. And for outdated hardware fixing it is not really interesting.

Initiator mode tested with Qlogic 1080 adapter is still working fine.

MFC r291144: Fix target mode with fabric for pre-24xx chips.

For those chips we are not receiving login events, adding initiators
based on ATIO requests. But there is no port ID in that structure, so
in fabric mode we have to explicitly fetch it from firmware to be able
to do normal scan after that.

MFC r291080: Another round of port scanner rewrite.

This change simplifies and unifies port adding/updating for loop and
fabric scanners. It also fixes problems with scanning restarts due to
concurrent port databases changes. It also fixes many cosmetic issues.

MFC r290993, r290994: Unify and cleanup FC ports scan.

MFC r290054: Reimplement next port handle generation.

For some reason port handles should be allocated from HBA-global space,
while old code was not very specific, mixing per-HBA and per-VP logic.

MFC r289890: Skip reserved IP Broadcast handle from using.

MFC r289812, r289852: Some polishing and unification in ISR code.

MFC r289219: Export bunch of state variables as sysctls.

MFC r285459: Unify port database use for target and initiator roles.

Aside from cleaner and more consistent code, this allows ports to be both
target and initiator same time, and easily switch from any role to any.

MFC r285154: Remove extra level of target ID indirection (isp_dev_map).

FreeBSD never had limitation on number of target IDs, and there is no
any other requirement to allocate them densely. Since slots of port
database already populated just sequentially, there is no much need
for another indirection to allocate sequentially too.

MFC r285146: Drop discovered targets when initiator role is disabled.

MFC isp(4) driver changes:

r276839, r276842, r277513, r277514, r277515

------------------------------------------------------------------------
r276839 | ken | 2015-01-08 10:41:28 -0700 (Thu, 08 Jan 2015) | 49 lines

Fix Fibre Channel Command Reference Number handling in the isp(4) driver.

The Command Reference Number is used for precise delivery of
commands, and is part of the FC-Tape functionality set. (This is
only enabled for devices that support precise delivery of commands.)
It is an 8-bit unsigned number that increments from 1 to 255. The
commands sent by the initiator must be processed by the target in
CRN order if the CRN is non-zero.

There are certain scenarios where the Command Reference Number
sequence needs to be reset. When the target is power cycled, for
instance, the initiator needs to reset the CRN to 1. The initiator
will know this because it will see a LIP (when directly connected)
or get a logout/login event (when connected to a switch).

The isp(4) driver was not resetting the CRN when a target
went away and came back. When it saw the target again after a
power cycle, it would continue the CRN sequence where it left off.
The target would ignore the command because the CRN sequence is
supposed to be reset to 1 after a power cycle or other similar
event.

The symptom that the user would see is that there would be lots of
aborted INQUIRY commands after a tape library was power cycled, and
the library would fail to probe. The INQUIRY commands were being
ignored by the tape drive due to the CRN issue mentioned above.

isp_freebsd.c:
Add a new function, isp_fcp_reset_crn(). This will reset
all of the CRNs for a given port, or the CRNs for all LUNs
on a target.

Reset the CRNs for all targets on a port when we get a LIP,
loop reset, or loop down event.

Reset the CRN for a particular target when it arrives, is changed
or departs. This is less precise behavior than the
clearing behavior specified in the FCP-4 spec (which says
that it should be reset for PRLI, PRLO, PLOGI and LOGO),
but this is the level of information we have here. If this
is insufficient, then we will need to add more precise
notification from the lower level isp(4) code.

isp_freebsd.h:
Add a prototype for isp_fcp_reset_crn().

Sponsored by: Spectra Logic
MFC after: 1 week

------------------------------------------------------------------------
r276842 | ken | 2015-01-08 10:51:12 -0700 (Thu, 08 Jan 2015) | 44 lines

Close a race in the isp(4) driver that caused devices to disappear
and not automatically come back if they were gone for a short
period of time.

The isp(4) driver has a 30 second gone device timer that gets
activated whenever a device goes away. If the device comes back
before the timer expires, we don't send a notification to CAM that
it has gone away. If, however, there is a command sent to the
device while it is gone and before it comes back, the isp(4) driver
sends the command back with CAM_SEL_TIMEOUT status.

CAM responds to the CAM_SEL_TIMEOUT status by removing the device.
In the case where a device comes back within the 30 second gone
device timer window, though, we weren't telling CAM the device
came back.

So, fix this by tracking whether we have told CAM the device is
gone, and if we have, send a rescan if it comes back within the 30
second window.

ispvar.h:
In the fcportdb_t structure, add a new bitfield,
reported_gone. This gets set whenever we return a command
with CAM_SEL_TIMEOUT status on a Fibre Channel device.

isp_freebsd.c:
In isp_done(), if we're sending CAM_SEL_TIMEOUT for for a
command sent to a FC device, set the reported_gone bit.

In isp_async(), in the ISPASYNC_DEV_STAYED case, rescan the
device in question if it is mapped to a target ID and has
been reported gone.

In isp_make_here(), take a port database entry argument,
and clear the reported_gone bit when we send a rescan to
CAM.

In isp_make_gone(), take a port database entry as an
argument, and set the reported_gone bit when we send an
async event telling CAM consumers that the device is gone.

Sponsored by: Spectra Logic
MFC after: 1 week

------------------------------------------------------------------------
r277514 | will | 2015-01-21 13:27:11 -0700 (Wed, 21 Jan 2015) | 18 lines

Force commit to record the correct log for r277513.

If the user sends an XPT_RESET_DEV CCB, make sure to reset the
Fibre Channel Command Reference Number if we're running on a FC
controller.

We send a SCSI Target Reset when we get this CCB, and as a result
need to reset the CRN to 1 on the next command.

isp_freebsd.c:
In the XPT_RESET_DEV implementation in isp_action(), reset
the CRN if we're on a FC controller.

Submitted by: ken
MFC after: 1 week
Sponsored by: Spectra Logic
MFSpectraBSD: 1112787 on 2015/01/15

------------------------------------------------------------------------
r277515 | will | 2015-01-21 13:32:36 -0700 (Wed, 21 Jan 2015) | 25 lines

Fix SCSI status byte reporting on 4Gb and 8Gb Qlogic boards.

The newer boards don't have the response field that indicates
whether the SCSI status byte is present. You have to just look to
see whether it is non-zero.

The code was looking to see whether the sense length was valid
before propagating the SCSI status byte (and sense information) up
the stack. With a status like Reservation Conflict, there is no
sense information, only the SCSI status byte. So it wasn't getting
correctly returned.

isp.c:
In isp_intr(), if we are on a 2400 or 2500 type board and
get a response, look at the actual contents of the
SCSI status value and set the RQSF_GOT_STATUS flag
accordingly so that return any SCSI status value we get. The
RQSF_GOT_SENSE flag will get set later on if there is
actual sense information returned.

Submitted by: ken
MFC after: 1 week
Sponsored by: Spectra Logic
MFSpectraBSD: 1112791 on 2015/01/15

------------------------------------------------------------------------

Sponsored by: Spectra Logic

MFC r256705:
Optimize isp(4) to reduce CPU usage, especially in target mode:
- Remove two excessive and slow register reads from isp_intr(). Instead
of rereading value every time, assume that registers contain what we have
written there.
- Avoid sequential search through 4096 array elements when looking for
command tag. Use hash of lists to store active tags separately from free
ones and so greatly speedup the searches.

MFC r285459: Unify port database use for target and initiator roles.

Aside from cleaner and more consistent code, this allows ports to be both
target and initiator same time, and easily switch from any role to any.

MFC r285154: Remove extra level of target ID indirection (isp_dev_map).

FreeBSD never had limitation on number of target IDs, and there is no
any other requirement to allocate them densely. Since slots of port
database already populated just sequentially, there is no much need
for another indirection to allocate sequentially too.

MFC r285146: Drop discovered targets when initiator role is disabled.

MFC isp(4) driver changes:

r276839, r276842, r277513, r277514, r277515

------------------------------------------------------------------------
r276839 | ken | 2015-01-08 10:41:28 -0700 (Thu, 08 Jan 2015) | 49 lines

Fix Fibre Channel Command Reference Number handling in the isp(4) driver.

The Command Reference Number is used for precise delivery of
commands, and is part of the FC-Tape functionality set. (This is
only enabled for devices that support precise delivery of commands.)
It is an 8-bit unsigned number that increments from 1 to 255. The
commands sent by the initiator must be processed by the target in
CRN order if the CRN is non-zero.

There are certain scenarios where the Command Reference Number
sequence needs to be reset. When the target is power cycled, for
instance, the initiator needs to reset the CRN to 1. The initiator
will know this because it will see a LIP (when directly connected)
or get a logout/login event (when connected to a switch).

The isp(4) driver was not resetting the CRN when a target
went away and came back. When it saw the target again after a
power cycle, it would continue the CRN sequence where it left off.
The target would ignore the command because the CRN sequence is
supposed to be reset to 1 after a power cycle or other similar
event.

The symptom that the user would see is that there would be lots of
aborted INQUIRY commands after a tape library was power cycled, and
the library would fail to probe. The INQUIRY commands were being
ignored by the tape drive due to the CRN issue mentioned above.

isp_freebsd.c:
Add a new function, isp_fcp_reset_crn(). This will reset
all of the CRNs for a given port, or the CRNs for all LUNs
on a target.

Reset the CRNs for all targets on a port when we get a LIP,
loop reset, or loop down event.

Reset the CRN for a particular target when it arrives, is changed
or departs. This is less precise behavior than the
clearing behavior specified in the FCP-4 spec (which says
that it should be reset for PRLI, PRLO, PLOGI and LOGO),
but this is the level of information we have here. If this
is insufficient, then we will need to add more precise
notification from the lower level isp(4) code.

isp_freebsd.h:
Add a prototype for isp_fcp_reset_crn().

Sponsored by: Spectra Logic
MFC after: 1 week

------------------------------------------------------------------------
r276842 | ken | 2015-01-08 10:51:12 -0700 (Thu, 08 Jan 2015) | 44 lines

Close a race in the isp(4) driver that caused devices to disappear
and not automatically come back if they were gone for a short
period of time.

The isp(4) driver has a 30 second gone device timer that gets
activated whenever a device goes away. If the device comes back
before the timer expires, we don't send a notification to CAM that
it has gone away. If, however, there is a command sent to the
device while it is gone and before it comes back, the isp(4) driver
sends the command back with CAM_SEL_TIMEOUT status.

CAM responds to the CAM_SEL_TIMEOUT status by removing the device.
In the case where a device comes back within the 30 second gone
device timer window, though, we weren't telling CAM the device
came back.

So, fix this by tracking whether we have told CAM the device is
gone, and if we have, send a rescan if it comes back within the 30
second window.

ispvar.h:
In the fcportdb_t structure, add a new bitfield,
reported_gone. This gets set whenever we return a command
with CAM_SEL_TIMEOUT status on a Fibre Channel device.

isp_freebsd.c:
In isp_done(), if we're sending CAM_SEL_TIMEOUT for for a
command sent to a FC device, set the reported_gone bit.

In isp_async(), in the ISPASYNC_DEV_STAYED case, rescan the
device in question if it is mapped to a target ID and has
been reported gone.

In isp_make_here(), take a port database entry argument,
and clear the reported_gone bit when we send a rescan to
CAM.

In isp_make_gone(), take a port database entry as an
argument, and set the reported_gone bit when we send an
async event telling CAM consumers that the device is gone.

Sponsored by: Spectra Logic
MFC after: 1 week

------------------------------------------------------------------------
r277514 | will | 2015-01-21 13:27:11 -0700 (Wed, 21 Jan 2015) | 18 lines

Force commit to record the correct log for r277513.

If the user sends an XPT_RESET_DEV CCB, make sure to reset the
Fibre Channel Command Reference Number if we're running on a FC
controller.

We send a SCSI Target Reset when we get this CCB, and as a result
need to reset the CRN to 1 on the next command.

isp_freebsd.c:
In the XPT_RESET_DEV implementation in isp_action(), reset
the CRN if we're on a FC controller.

Submitted by: ken
MFC after: 1 week
Sponsored by: Spectra Logic
MFSpectraBSD: 1112787 on 2015/01/15

------------------------------------------------------------------------
r277515 | will | 2015-01-21 13:32:36 -0700 (Wed, 21 Jan 2015) | 25 lines

Fix SCSI status byte reporting on 4Gb and 8Gb Qlogic boards.

The newer boards don't have the response field that indicates
whether the SCSI status byte is present. You have to just look to
see whether it is non-zero.

The code was looking to see whether the sense length was valid
before propagating the SCSI status byte (and sense information) up
the stack. With a status like Reservation Conflict, there is no
sense information, only the SCSI status byte. So it wasn't getting
correctly returned.

isp.c:
In isp_intr(), if we are on a 2400 or 2500 type board and
get a response, look at the actual contents of the
SCSI status value and set the RQSF_GOT_STATUS flag
accordingly so that return any SCSI status value we get. The
RQSF_GOT_SENSE flag will get set later on if there is
actual sense information returned.

Submitted by: ken
MFC after: 1 week
Sponsored by: Spectra Logic
MFSpectraBSD: 1112791 on 2015/01/15

------------------------------------------------------------------------

Sponsored by: Spectra Logic

MFC r256705:
Optimize isp(4) to reduce CPU usage, especially in target mode:
- Remove two excessive and slow register reads from isp_intr(). Instead
of rereading value every time, assume that registers contain what we have
written there.
- Avoid sequential search through 4096 array elements when looking for
command tag. Use hash of lists to store active tags separately from free
ones and so greatly speedup the searches.