184631Smjacob/* $FreeBSD: releng/11.0/sys/dev/isp/DriverManual.txt 298955 2016-05-03 03:41:25Z pfg $ */ 284631Smjacob 384631Smjacob Driver Theory of Operation Manual 484631Smjacob 584631Smjacob1. Introduction 684631Smjacob 784631SmjacobThis is a short text document that will describe the background, goals 884631Smjacobfor, and current theory of operation for the joint Fibre Channel/SCSI 984631SmjacobHBA driver for QLogic hardware. 1084631Smjacob 1184631SmjacobBecause this driver is an ongoing project, do not expect this manual 1284631Smjacobto remain entirely up to date. Like a lot of software engineering, the 1384631Smjacobultimate documentation is the driver source. However, this manual should 1484631Smjacobserve as a solid basis for attempting to understand where the driver 1584631Smjacobstarted and what is trying to be accomplished with the current source. 1684631Smjacob 1784631SmjacobThe reader is expected to understand the basics of SCSI and Fibre Channel 1884631Smjacoband to be familiar with the range of platforms that Solaris, Linux and 1984631Smjacobthe variant "BSD" Open Source systems are available on. A glossary and 2084631Smjacoba few references will be placed at the end of the document. 2184631Smjacob 2284631SmjacobThere will be references to functions and structures within the body of 2384631Smjacobthis document. These can be easily found within the source using editor 2484631Smjacobtags or grep. There will be few code examples here as the code already 2584631Smjacobexists where the reader can easily find it. 2684631Smjacob 2784631Smjacob2. A Brief History for this Driver 2884631Smjacob 2984631SmjacobThis driver originally started as part of work funded by NASA Ames 3084631SmjacobResearch Center's Numerical Aerodynamic Simulation center ("NAS" for 3184631Smjacobshort) for the QLogic PCI 1020 and 1040 SCSI Host Adapters as part of my 3284631Smjacobwork at porting the NetBSD Operating System to the Alpha architectures 3384631Smjacob(specifically the AlphaServer 8200 and 8400 platforms). In short, it 3484631Smjacobstarted just as simple single SCSI HBA driver for just the purpose of 3584631Smjacobrunning off a SCSI disk. This work took place starting in January, 1997. 3684631Smjacob 3784631SmjacobBecause the first implementation was for NetBSD, which runs on a very 3884631Smjacoblarge number of platforms, and because NetBSD supported both systems with 3984631SmjacobSBus cards (e.g., Sun SPARC systems) as well as systems with PCI cards, 4084631Smjacoband because the QLogic SCSI cards came in both SBus and PCI versions, the 4184631Smjacobinitial implementation followed the very thoughtful NetBSD design tenet 4284631Smjacobof splitting drivers into what are called MI (for Machine Independent) 4384631Smjacoband MD (Machine Dependent) portions. The original design therefore was 4484631Smjacobfrom the premise that the driver would drive both SBus and PCI card 4584631Smjacobvariants. These busses are similar but have quite different constraints, 4684631Smjacoband while the QLogic SBus and PCI cards are very similar, there are some 4784631Smjacobsignificant differences. 4884631Smjacob 4984631SmjacobAfter this initial goal had been met, there began to be some talk about 5084631Smjacoblooking into implementing Fibre Channel mass storage at NAS. At this time 5184631Smjacobthe QLogic 2100 FC/AL HBA was about to become available. After looking at 5284631Smjacobthe way it was designed I concluded that it was so darned close to being 5384631Smjacobjust like the SCSI HBAs that it would be insane to *not* leverage off of 5484631Smjacobthe existing driver. So, we ended up with a driver for NetBSD that drove 5584631SmjacobPCI and SBus SCSI cards, and now also drove the QLogic 2100 FC-AL HBA. 5684631Smjacob 5784631SmjacobAfter this, ports to non-NetBSD platforms became interesting as well. 5884631SmjacobThis took the driver out of the interest with NAS and into interested 5984631Smjacobsupport from a number of other places. Since the original NetBSD 6084631Smjacobdevelopment, the driver has been ported to FreeBSD, OpenBSD, Linux, 6184631SmjacobSolaris, and two proprietary systems. Following from the original MI/MD 6284631Smjacobdesign of NetBSD, a rather successful attempt has been made to keep the 6384631SmjacobOperating System Platform differences segregated and to a minimum. 6484631Smjacob 6584631SmjacobAlong the way, support for the 2200 as well as full fabric and target 6684631Smjacobmode support has been added, and 2300 support as well as an FC-IP stack 6784631Smjacobare planned. 6884631Smjacob 6984631Smjacob3. Driver Design Goals 7084631Smjacob 7184631SmjacobThe driver has not started out as one normally would do such an effort. 72298955SpfgNormally you design via top-down methodologies and set an initial goal 7384631Smjacoband meet it. This driver has had a design goal that changes from almost 7484631Smjacobthe very first. This has been an extremely peculiar, if not risque, 7584631Smjacobexperience. As a consequence, this section of this document contains 7684631Smjacoba bit of "reconstruction after the fact" in that the design goals are 7784631Smjacobas I perceive them to be now- not necessarily what they started as. 7884631Smjacob 7984631SmjacobThe primary design goal now is to have a driver that can run both the 8084631SmjacobSCSI and Fibre Channel SCSI prototocols on multiple OS platforms with 8184631Smjacobas little OS platform support code as possible. 8284631Smjacob 8384631SmjacobThe intended support targets for SCSI HBAs is to support the single and 8484631Smjacobdual channel PCI Ultra2 and PCI Ultra3 cards as well as the older PCI 8584631SmjacobUltra single channel cards and SBus cards. 8684631Smjacob 8784631SmjacobThe intended support targets for Fibre Channel HBAs is the 2100, 2200 8884631Smjacoband 2300 PCI cards. 8984631Smjacob 9084631SmjacobFibre Channel support should include complete fabric and public loop 9184631Smjacobas well as private loop and private loop, direct-attach topologies. 9284631SmjacobFC-IP support is also a goal. 9384631Smjacob 9484631SmjacobFor both SCSI and Fibre Channel, simultaneous target/initiator mode support 9584631Smjacobis a goal. 9684631Smjacob 9784631SmjacobPure, raw, performance is not a primary goal of this design. This design, 9884631Smjacobbecause it has a tremendous amount of code common across multiple 9984631Smjacobplatforms, will undoubtedly never be able to beat the performance of a 10084631Smjacobdriver that is specifically designed for a single platform and a single 10184631Smjacobcard. However, it is a good strong secondary goal to make the performance 10284631Smjacobpenalties in this design as small as possible. 10384631Smjacob 10484631SmjacobAnother primary aim, which almost need not be stated, is that the 10584631Smjacobimplementation of platform differences must not clutter up the common 10684631Smjacobcode with platform specific defines. Instead, some reasonable layering 10784631Smjacobsemantics are defined such that platform specifics can be kept in the 10884631Smjacobplatform specific code. 10984631Smjacob 11084631Smjacob4. QLogic Hardware Architecture 11184631Smjacob 11284631SmjacobIn order to make the design of this driver more intelligible, some 11384631Smjacobdescription of the Qlogic hardware architecture is in order. This will 11484631Smjacobnot be an exhaustive description of how this card works, but will 11584631Smjacobnote enough of the important features so that the driver design is 11684631Smjacobhopefully clearer. 11784631Smjacob 11884631Smjacob4.1 Basic QLogic hardware 11984631Smjacob 12084631SmjacobThe QLogic HBA cards all contain a tiny 16-bit RISC-like processor and 12184631Smjacobvarying sizes of SRAM. Each card contains a Bus Interface Unit (BIU) 12284631Smjacobas appropriate for the host bus (SBus or PCI). The BIUs allow access 12384631Smjacobto a set of dual-ranked 16 bit incoming and outgoing mailbox registers 12484631Smjacobas well as access to control registers that control the RISC or access 12584631Smjacobother portions of the card (e.g., Flash BIOS). The term 'dual-ranked' 12684631Smjacobmeans that at the same host visible address if you write a mailbox 12784631Smjacobregister, that is a write to an (incoming, to the HBA) mailbox register, 12884631Smjacobwhile a read to the same address reads another (outgoing, to the HBA) 12984631Smjacobmailbox register with completely different data. Each HBA also then has 130298955Spfgcore and auxiliary logic which either is used to interface to a SCSI bus 13184631Smjacob(or to external bus drivers that connect to a SCSI bus), or to connect 13284631Smjacobto a Fibre Channel bus. 13384631Smjacob 13484631Smjacob4.2 Basic Control Interface 13584631Smjacob 13684631SmjacobThere are two principle I/O control mechanisms by which the driver 13784631Smjacobcommunicates with and controls the QLogic HBA. The first mechanism is to 13884631Smjacobuse the incoming mailbox registers to interrupt and issue commands to 13984631Smjacobthe RISC processor (with results usually, but not always, ending up in 14084631Smjacobthe ougtoing mailbox registers). The second mechanism is to establish, 14184631Smjacobvia mailbox commands, circular request and response queues in system 14284631Smjacobmemory that are then shared between the QLogic and the driver. The 14384631Smjacobrequest queue is used to queue requests (e.g., I/O requests) for the 14484631SmjacobQLogic HBA's RISC engine to copy into the HBA memory and process. The 14584631Smjacobresult queue is used by the QLogic HBA's RISC engine to place results of 14684631Smjacobrequests read from the request queue, as well as to place notification 14784631Smjacobof asynchronous events (e.g., incoming commands in target mode). 14884631Smjacob 14984631SmjacobTo give a bit more precise scale to the preceding description, the QLogic 15084631SmjacobHBA has 8 dual-ranked 16 bit mailbox registers, mostly for out-of-band 15184631Smjacobcontrol purposes. The QLogic HBA then utilizes a circular request queue 15284631Smjacobof 64 byte fixed size Queue Entries to receive normal initiator mode 15384631SmjacobI/O commands (or continue target mode requests). The request queue may 15484631Smjacobbe up to 256 elements for the QLogic 1020 and 1040 chipsets, but may 15584631Smjacobbe quite larger for the QLogic 12X0/12160 SCSI and QLogic 2X00 Fibre 15684631SmjacobChannel chipsets. 15784631Smjacob 15884631SmjacobIn addition to synchronously initiated usage of mailbox commands by 15984631Smjacobthe host system, the QLogic may also deliver asynchronous notifications 16084631Smjacobsolely in outgoing mailbox registers. These asynchronous notifications in 16184631Smjacobmailboxes may be things like notification of SCSI Bus resets, or that the 16284631SmjacobFabric Name server has sent a change notification, or even that a specific 16384631SmjacobI/O command completed without error (this is called 'Fast Posting' 16484631Smjacoband saves the QLogic HBA from having to write a response queue entry). 16584631Smjacob 16684631SmjacobThe QLogic HBA is an interrupting card, and when servicing an interrupt 16784631Smjacobyou really only have to check for either a mailbox interrupt or an 168226436Seadlerinterrupt notification that the response queue has an entry to 16984631Smjacobbe dequeued. 17084631Smjacob 17184631Smjacob4.3 Fibre Channel SCSI out of SCSI 17284631Smjacob 17384631SmjacobQLogic took the approach in introducing the 2X00 cards to just treat 17484631SmjacobFC-AL as a 'fat' SCSI bus (a SCSI bus with more than 15 targets). All 17584631Smjacobof the things that you really need to do with Fibre Channel with respect 17684631Smjacobto providing FC-4 services on top of a Class 3 connection are performed 17784631Smjacobby the RISC engine on the QLogic card itself. This means that from 17884631Smjacoban HBA driver point of view, very little needs to change that would 17984631Smjacobdistinguish addressing a Fibre Channel disk from addressing a plain 18084631Smjacobold SCSI disk. 18184631Smjacob 18284631SmjacobHowever, in the details it's not *quite* that simple. For example, in 18384631Smjacoborder to manage Fabric Connections, the HBA driver has to do explicit 18484631Smjacobbinding of entities it's queried from the name server to specific 'target' 18584631Smjacobids (targets, in this case, being a virtual entity). 18684631Smjacob 18784631SmjacobStill- the HBA firmware does really nearly all of the tedious management 18884631Smjacobof Fibre Channel login state. The corollary to this sometimes is the 18984631Smjacoblack of ability to say why a particular login connection to a Fibre 19084631SmjacobChannel disk is not working well. 19184631Smjacob 19284631SmjacobThere are clear limits with the QLogic card in managing fabric devices. 19384631SmjacobThe QLogic manages local loop devices (LoopID or Target 0..126) itself, 19484631Smjacobbut for the management of fabric devices, it has an absolute limit of 19584631Smjacob253 simultaneous connections (256 entries less 3 reserved entries). 19684631Smjacob 19784631Smjacob5. Driver Architecture 19884631Smjacob 19984631Smjacob5.1 Driver Assumptions 20084631Smjacob 20184631SmjacobThe first basic assumption for this driver is that the requirements for 20284631Smjacoba SCSI HBA driver for any system is that of a 2 or 3 layer model where 20384631Smjacobthere are SCSI target device drivers (drivers which drive SCSI disks, 20484631SmjacobSCSI tapes, and so on), possibly a middle services layer, and a bottom 20584631Smjacoblayer that manages the transport of SCSI CDB's out a SCSI bus (or across 20684631SmjacobFibre Channel) to a SCSI device. It's assumed that each SCSI command is 20784631Smjacoba separate structure (or pointer to a structure) that contains the SCSI 20884631SmjacobCDB and a place to store SCSI Status and SCSI Sense Data. 20984631Smjacob 21084631SmjacobThis turns out to be a pretty good assumption. All of the Open Source 21184631Smjacobsystems (*BSD and Linux) and most of the proprietary systems have this 21284631Smjacobkind of structure. This has been the way to manage SCSI subsystems for 21384631Smjacobat least ten years. 21484631Smjacob 21584631SmjacobThere are some additional basic assumptions that this driver makes- primarily 21684631Smjacobin the arena of basic simple services like memory zeroing, memory copying, 21784631Smjacobdelay, sleep, microtime functions. It doesn't assume much more than this. 21884631Smjacob 21984631Smjacob5.2 Overall Driver Architecture 22084631Smjacob 22184631SmjacobThe driver is split into a core (machine independent) module and platform 22284631Smjacoband bus specific outer modules (machine dependent). 22384631Smjacob 22484631SmjacobThe core code (in the files isp.c, isp_inline.h, ispvar.h, ispreg.h and 22584631Smjacobispmbox.h) handles: 22684631Smjacob 22784631Smjacob + Chipset recognition and reset and firmware download (isp_reset) 22884631Smjacob + Board Initialization (isp_init) 22984631Smjacob + First level interrupt handling (response retrieval) (isp_intr) 23084631Smjacob + A SCSI command queueing entry point (isp_start) 23184631Smjacob + A set of control services accessed either via local requirements within 23284631Smjacob the core module or via an externally visible control entry point 23384631Smjacob (isp_control). 23484631Smjacob 23584631SmjacobThe platform/bus specific modules (and definitions) depend on each 23684631Smjacobplatform, and they provide both definitions and functions for the core 23784631Smjacobmodule's use. Generally a platform module set is split into a bus 23884631Smjacobdependent module (where configuration is begun from and bus specific 23984631Smjacobsupport functions reside) and relatively thin platform specific layer 24084631Smjacobwhich serves as the interconnect with the rest of this platform's SCSI 24184631Smjacobsubsystem. 24284631Smjacob 24384631SmjacobFor ease of bus specific access issues, a centralized soft state 24484631Smjacobstructure is maintained for each HBA instance (struct ispsoftc). This 24584631Smjacobsoft state structure contains a machine/bus dependent vector (mdvec) 24684631Smjacobfor functions that read and write hardware registers, set up DMA for the 24784631Smjacobrequest/response queues and fibre channel scratch area, set up and tear 24884631Smjacobdown DMA mappings for a SCSI command, provide a pointer to firmware to 24984631Smjacobload, and other minor things. 25084631Smjacob 25184631SmjacobThe machine dependent outer module must provide functional entry points 25284631Smjacobfor the core module: 25384631Smjacob 25484631Smjacob + A SCSI command completion handoff point (isp_done) 25584631Smjacob + An asynchronous event handler (isp_async) 25684631Smjacob + A logging/printing function (isp_prt) 25784631Smjacob 25884631SmjacobThe machine dependent outer module code must also provide a set of 25984631Smjacobabstracting definitions which is what the core module utilizes heavily 26084631Smjacobto do its job. These are discussed in detail in the comments in the 26184631Smjacobfile ispvar.h, but to give a sense of the range of what is required, 26284631Smjacoblet's illustrate two basic classes of these defines. 26384631Smjacob 26484631SmjacobThe first class are "structure definition/access" class. An 26584631Smjacobexample of these would be: 26684631Smjacob 26784631Smjacob XS_T Platform SCSI transaction type (i.e., command for HBA) 26884631Smjacob .. 26984631Smjacob XS_TGT(xs) gets the target from an XS_T 27084631Smjacob .. 27184631Smjacob XS_TAG_TYPE(xs) which type of tag to use 27284631Smjacob .. 27384631Smjacob 27484631SmjacobThe second class are 'functional' class definitions. Some examples of 27584631Smjacobthis class are: 27684631Smjacob 27784631Smjacob MEMZERO(dst, src) platform zeroing function 27884631Smjacob .. 27984631Smjacob MBOX_WAIT_COMPLETE(struct ispsoftc *) wait for mailbox cmd to be done 28084631Smjacob 28184631SmjacobNote that the former is likely to be simple replacement with bzero or 28284631Smjacobmemset on most systems, while the latter could be quite complex. 28384631Smjacob 28484631SmjacobThis soft state structure also contains different parameter information 28584631Smjacobbased upon whether this is a SCSI HBA or a Fibre Channel HBA (which is 28684631Smjacobfilled in by the code module). 28784631Smjacob 28884631SmjacobIn order to clear up what is undoubtedly a seeming confusion of 28984631Smjacobinterconnects, a description of the typical flow of code that performs 29084631Smjacobboards initialization and command transactions may help. 29184631Smjacob 29284631Smjacob5.3 Initialization Code Flow 29384631Smjacob 29484631SmjacobTypically a bus specific module for a platform (e.g., one that wants 29584631Smjacobto configure a PCI card) is entered via that platform's configuration 29684631Smjacobmethods. If this module recognizes a card and can utilize or construct the 29784631Smjacobspace for the HBA instance softc, it does so, and initializes the machine 29884631Smjacobdependent vector as well as any other platform specific information that 29984631Smjacobcan be hidden in or associated with this structure. 30084631Smjacob 30184631SmjacobConfiguration at this point usually involves mapping in board registers 30284631Smjacoband registering an interrupt. It's quite possible that the core module's 30384631Smjacobisp_intr function is adequate to be the interrupt entry point, but often 30484631Smjacobit's more useful have a bus specific wrapper module that calls isp_intr. 30584631Smjacob 30684631SmjacobAfter mapping and interrupt registry is done, isp_reset is called. 30784631SmjacobPart of the isp_reset call may cause callbacks out to the bus dependent 30884631Smjacobmodule to perform allocation and/or mapping of Request and Response 30984631Smjacobqueues (as well as a Fibre Channel scratch area if this is a Fibre 31084631SmjacobChannel HBA). The reason this is considered 'bus dependent' is that 31184631Smjacobonly the bus dependent module may have the information that says how 31284631Smjacobone could perform I/O mapping and dependent (e.g., on a Solaris system) 313298955Spfgon the Request and Response queues. Another callback can enable the *use* 31484631Smjacobof interrupts should this platform be able to finish configuration in 31584631Smjacobinterrupt driven mode. 31684631Smjacob 31784631SmjacobIf isp_reset is successful at resetting the QLogic chipset and downloading 31884631Smjacobnew firmware (if available) and setting it running, isp_init is called. If 31984631Smjacobisp_init is successful in doing initial board setups (including reading 32084631SmjacobNVRAM from the QLogic card), then this bus specicic module will call the 32184631Smjacobplatform dependent module that takes the appropriate steps to 'register' 32284631Smjacobthis HBA with this platform's SCSI subsystem. Examining either the 32384631SmjacobOpenBSD or the NetBSD isp_pci.c or isp_sbus.c files may assist the reader 32484631Smjacobhere in clarifying some of this. 32584631Smjacob 32684631Smjacob5.4 Initiator Mode Command Code Flow 32784631Smjacob 328298955SpfgA successful execution of isp_init will lead to the driver 'registering' 32984631Smjacobitself with this platform's SCSI subsystem. One assumed action for this 330227458Seadleris the registry of a function that the SCSI subsystem for this platform 33184631Smjacobwill call when it has a SCSI command to run. 33284631Smjacob 33384631SmjacobThe platform specific module function that receives this will do whatever 33484631Smjacobit needs to to prepare this command for execution in the core module. This 33584631Smjacobsounds vague, but it's also very flexible. In principle, this could be 33684631Smjacoba complete marshalling/demarshalling of this platform's SCSI command 33784631Smjacobstructure (should it be impossible to represent in an XS_T). In addition, 33884631Smjacobthis function can also block commands from running (if, e.g., Fibre 33984631SmjacobChannel loop state would preclude successful starting of the command). 34084631Smjacob 34184631SmjacobWhen it's ready to do so, the function isp_start is called with this 34284631Smjacobcommand. This core module tries to allocate request queue space for 34384631Smjacobthis command. It also calls through the machine dependent vector 34484631Smjacobfunction to make sure any DMA mapping for this command is done. 34584631Smjacob 34684631SmjacobNow, DMA mapping here is possibly a misnomer, as more than just 34784631SmjacobDMA mapping can be done in this bus dependent function. This is 34884631Smjacobalso the place where any endian byte-swizzling will be done. At any 34984631Smjacobrate, this function is called last because the process of establishing 35084631SmjacobDMA addresses for any command may in fact consume more Request Queue 35184631Smjacobentries than there are currently available. If the mapping and other 35284631Smjacobfunctions are successful, the QLogic mailbox inbox pointer register 35384631Smjacobis updated to indicate to the QLogic that it has a new request to 35484631Smjacobread. 35584631Smjacob 35684631SmjacobIf this function is unsuccessful, policy as to what to do at this point is 35784631Smjacobleft to the machine dependent platform function which called isp_start. In 35884631Smjacobsome platforms, temporary resource shortages can be handled by the main 35984631SmjacobSCSI subsystem. In other platforms, the machine dependent code has to 36084631Smjacobhandle this. 36184631Smjacob 36284631SmjacobIn order to keep track of commands that are in progress, the soft state 36384631Smjacobstructure contains an array of 'handles' that are associated with each 36484631Smjacobactive command. When you send a command to the QLogic firmware, a portion 36584631Smjacobof the Request Queue entry can contain a non-zero handle identifier so 36684631Smjacobthat at a later point in time in reading either a Response Queue entry 36784631Smjacobor from a Fast Posting mailbox completion interrupt, you can take this 36884631Smjacobhandle to find the command you were waiting on. It should be noted that 36984631Smjacobthis is probably one of the most dangerous areas of this driver. Corrupted 37084631Smjacobhandles will lead to system panics. 37184631Smjacob 37284631SmjacobAt some later point in time an interrupt will occur. Eventually, 37384631Smjacobisp_intr will be called. This core module will determine what the cause 37484631Smjacobof the interrupt is, and if it is for a completing command. That is, 37584631Smjacobit'll determine the handle and fetch the pointer to the command out of 37684631Smjacobstorage within the soft state structure. Skipping over a lot of details, 37784631Smjacobthe machine dependent code supplied function isp_done is called with the 37884631Smjacobpointer to the completing command. This would then be the glue layer that 37984631Smjacobinforms the SCSI subsystem for this platform that a command is complete. 38084631Smjacob 38184631Smjacob5.5 Asynchronous Events 38284631Smjacob 38384631SmjacobInterrupts occur for events other than commands (mailbox or request queue 38484631Smjacobstarted commands) completing. These are called Asynchronous Mailbox 38584631Smjacobinterrupts. When some external event causes the SCSI bus to be reset, 38684631Smjacobor when a Fibre Channel loop changes state (e.g., a LIP is observed), 38784631Smjacobthis generates such an asynchronous event. 38884631Smjacob 38984631SmjacobEach platform module has to provide an isp_async entry point that will 39084631Smjacobhandle a set of these. This isp_async entry point also handles things 39184631Smjacobwhich aren't properly async events but are simply natural outgrowths 39284631Smjacobof code flow for another core function (see discussion on fabric device 39384631Smjacobmanagement below). 39484631Smjacob 39584631Smjacob5.6 Target Mode Code Flow 39684631Smjacob 39784631SmjacobThis section could use a lot of expansion, but this covers the basics. 39884631Smjacob 39984631SmjacobThe QLogic cards, when operating in target mode, follow a code flow that is 40084631Smjacobessentially the inverse of that for intiator mode describe above. In this 40184631Smjacobscenario, an interrupt occurs, and present on the Response Queue is a 40284631Smjacobqueue entry element defining a new command arriving from an initiator. 40384631Smjacob 40484631SmjacobThis is passed to possibly external target mode handler. This driver 40584631Smjacobprovides some handling for this in a core module, but also leaves 40684631Smjacobthings open enough that a completely different target mode handler 40784631Smjacobmay accept this incoming queue entry. 40884631Smjacob 40984631SmjacobThe external target mode handler then turns around forms up a response 41084631Smjacobto this 'response' that just arrived which is then placed on the Request 41184631SmjacobQueue and handled very much like an initiator mode command (i.e., calling 41284631Smjacobthe bus dependent DMA mapping function). If this entry completes the 41384631Smjacobcommand, no more need occur. But often this handles only part of the 41484631Smjacobrequested command, so the QLogic firmware will rewrite the response 41584631Smjacobto the initial 'response' again onto the Response Queue, whereupon the 41684631Smjacobtarget mode handler will respond to that, and so on until the command 41784631Smjacobis completely handled. 41884631Smjacob 41984631SmjacobBecause almost no platform provides basic SCSI Subsystem target mode 42084631Smjacobsupport, this design has been left extremely open ended, and as such 42184631Smjacobit's a bit hard to describe in more detail than this. 42284631Smjacob 42384631Smjacob5.7 Locking Assumptions 42484631Smjacob 42584631SmjacobThe observant reader by now is likely to have asked the question, "but what 42684631Smjacobabout locking? Or interrupt masking" by now. 42784631Smjacob 42884631SmjacobThe basic assumption about this is that the core module does not know 42984631Smjacobanything directly about locking or interrupt masking. It may assume that 43084631Smjacobupon entry (e.g., via isp_start, isp_control, isp_intr) that appropriate 43184631Smjacoblocking and interrupt masking has been done. 43284631Smjacob 43384631SmjacobThe platform dependent code may also therefore assume that if it is 43484631Smjacobcalled (e.g., isp_done or isp_async) that any locking or masking that 43584631Smjacobwas in place upon the entry to the core module is still there. It is up 43684631Smjacobto the platform dependent code to worry about avoiding any lock nesting 43784631Smjacobissues. As an example of this, the Linux implementation simply queues 43884631Smjacobup commands completed via the callout to isp_done, which it then pushes 43984631Smjacobout to the SCSI subsystem after a return from it's calling isp_intr is 44084631Smjacobexecuted (and locks dropped appropriately, as well as avoidance of deep 44184631Smjacobinterrupt stacks). 44284631Smjacob 44384631SmjacobRecent changes in the design have now eased what had been an original 44484631Smjacobrequirement that the while in the core module no locks or interrupt 44584631Smjacobmasking could be dropped. It's now up to each platform to figure out how 44684631Smjacobto implement this. This is principally used in the execution of mailbox 44784631Smjacobcommands (which are principally used for Loop and Fabric management via 44884631Smjacobthe isp_control function). 44984631Smjacob 45084631Smjacob5.8 SCSI Specifics 45184631Smjacob 45284631SmjacobThe driver core or platform dependent architecture issues that are specific 45384631Smjacobto SCSI are few. There is a basic assumption that the QLogic firmware 45484631Smjacobsupported Automatic Request sense will work- there is no particular provision 45584631Smjacobfor disabling it's usage on a per-command basis. 45684631Smjacob 45784631Smjacob5.9 Fibre Channel Specifics 45884631Smjacob 45984631SmjacobFibre Channel presents an interesting challenge here. The QLogic firmware 46084631Smjacobarchitecture for dealing with Fibre Channel as just a 'fat' SCSI bus 46184631Smjacobis fine on the face of it, but there are some subtle and not so subtle 46284631Smjacobproblems here. 46384631Smjacob 46484631Smjacob5.9.1 Firmware State 46584631Smjacob 46684631SmjacobPart of the initialization (isp_init) for Fibre Channel HBAs involves 46784631Smjacobsending a command (Initialize Control Block) that establishes Node 46884631Smjacoband Port WWNs as well as topology preferences. After this occurs, 46984631Smjacobthe QLogic firmware tries to traverese through serveral states: 47084631Smjacob 47184631Smjacob FW_CONFIG_WAIT 47284631Smjacob FW_WAIT_AL_PA 47384631Smjacob FW_WAIT_LOGIN 47484631Smjacob FW_READY 47584631Smjacob FW_LOSS_OF_SYNC 47684631Smjacob FW_ERROR 47784631Smjacob FW_REINIT 47884631Smjacob FW_NON_PART 47984631Smjacob 48084631SmjacobIt starts with FW_CONFIG_WAIT, attempts to get an AL_PA (if on an FC-AL 48184631Smjacobloop instead of being connected as an N-port), waits to log into all 48284631SmjacobFC-AL loop entities and then hopefully transitions to FW_READY state. 48384631Smjacob 48484631SmjacobClearly, no command should be attempted prior to FW_READY state is 48584631Smjacobachieved. The core internal function isp_fclink_test (reachable via 48684631Smjacobisp_control with the ISPCTL_FCLINK_TEST function code). This function 48784631Smjacobalso determines connection topology (i.e., whether we're attached to a 48884631Smjacobfabric or not). 48984631Smjacob 49084631Smjacob5.9.2. Loop State Transitions- From Nil to Ready 49184631Smjacob 49284631SmjacobOnce the firmware has transitioned to a ready state, then the state of the 49384631Smjacobconnection to either arbitrated loop or to a fabric has to be ascertained, 49484631Smjacoband the identity of all loop members (and fabric members validated). 49584631Smjacob 49684631SmjacobThis can be very complicated, and it isn't made easy in that the QLogic 49784631Smjacobfirmware manages PLOGI and PRLI to devices that are on a local loop, but 49884631Smjacobit is the driver that must manage PLOGI/PRLI with devices on the fabric. 49984631Smjacob 50084631SmjacobIn order to manage this state an eight level staging of current "Loop" 50184631Smjacob(where "Loop" is taken to mean FC-AL or N- or F-port connections) states 50284631Smjacobin the following ascending order: 50384631Smjacob 50484631Smjacob LOOP_NIL 50584631Smjacob LOOP_LIP_RCVD 50684631Smjacob LOOP_PDB_RCVD 50784631Smjacob LOOP_SCANNING_FABRIC 50884631Smjacob LOOP_FSCAN_DONE 50984631Smjacob LOOP_SCANNING_LOOP 51084631Smjacob LOOP_LSCAN_DONE 51184631Smjacob LOOP_SYNCING_PDB 51284631Smjacob LOOP_READY 51384631Smjacob 51484631SmjacobWhen the core code initializes the QLogic firmware, it sets the loop 51584631Smjacobstate to LOOP_NIL. The first 'LIP Received' asynchronous event sets state 51684631Smjacobto LOOP_LIP_RCVD. This should be followed by a "Port Database Changed" 51784631Smjacobasynchronous event which will set the state to LOOP_PDB_RCVD. Each of 51884631Smjacobthese states, when entered, causes an isp_async event call to the 51984631Smjacobmachine dependent layers with the ISPASYNC_CHANGE_NOTIFY code. 52084631Smjacob 52184631SmjacobAfter the state of LOOP_PDB_RCVD is reached, the internal core function 52284631Smjacobisp_scan_fabric (reachable via isp_control(..ISPCTL_SCAN_FABRIC)) will, 52384631Smjacobif the connection is to a fabric, use Simple Name Server mailbox mediated 52484631Smjacobcommands to dump the entire fabric contents. For each new entity, an 52584631Smjacobisp_async event will be generated that says a Fabric device has arrived 52684631Smjacob(ISPASYNC_FABRIC_DEV). The function that isp_async must perform in this 52784631Smjacobstep is to insert possibly remove devices that it wants to have the 52884631SmjacobQLogic firmware log into (at LOOP_SYNCING_PDB state level)). 52984631Smjacob 53084631SmjacobAfter this has occurred, the state LOOP_FSCAN_DONE is set, and then the 53184631Smjacobinternal function isp_scan_loop (isp_control(...ISPCTL_SCAN_LOOP)) can 53284631Smjacobbe called which will then scan for any local (FC-AL) entries by asking 53384631Smjacobfor each possible local loop id the QLogic firmware for a Port Database 53484631Smjacobentry. It's at this level some entries cached locally are purged 53584631Smjacobor shifting loopids are managed (see section 5.9.4). 53684631Smjacob 53784631SmjacobThe final step after this is to call the internal function isp_pdb_sync 53884631Smjacob(isp_control(..ISPCTL_PDB_SYNC)). The purpose of this function is to 53984631Smjacobthen perform the PLOGI/PRLI functions for fabric devices. The next state 54084631Smjacobentered after this is LOOP_READY, which means that the driver is ready 54184631Smjacobto process commands to send to Fibre Channel devices. 54284631Smjacob 54384631Smjacob5.9.3 Fibre Channel variants of Initiator Mode Code Flow 54484631Smjacob 54584631SmjacobThe code flow in isp_start for Fibre Channel devices is the same as it is 54684631Smjacobfor SCSI devices, but with a notable exception. 54784631Smjacob 54884631SmjacobMaintained within the fibre channel specific portion of the driver soft 54984631Smjacobstate structure is a distillation of the existing population of both 55084631Smjacoblocal loop and fabric devices. Because Loop IDs can shift on a local 55184631Smjacobloop but we wish to retain a 'constant' Target ID (see 5.9.4), this 55284631Smjacobis indexed directly via the Target ID for the command (XS_TGT(xs)). 55384631Smjacob 55484631SmjacobIf there is a valid entry for this Target ID, the command is started 55584631Smjacob(with the stored 'Loop ID'). If not the command is completed with 55684631Smjacobthe error that is just like a SCSI Selection Timeout error. 55784631Smjacob 55884631SmjacobThis code is currently somewhat in transition. Some platforms to 55984631Smjacobdo firmware and loop state management (as described above) at this 56084631Smjacobpoint. Other platforms manage this from the machine dependent layers. The 56184631Smjacobimportant function to watch in this respect is isp_fc_runstate (in 56284631Smjacobisp_inline.h). 56384631Smjacob 56484631Smjacob5.9.4 "Target" in Fibre Channel is a fixed virtual construct 56584631Smjacob 56684631SmjacobVery few systems can cope with the notion that "Target" for a disk 56784631Smjacobdevice can change while you're using it. But one of the properties of 56884631Smjacobfor arbitrated loop is that the physical bus address for a loop member 56984631Smjacob(the AL_PA) can change depending on when and how things are inserted in 57084631Smjacobthe loop. 57184631Smjacob 57284631SmjacobTo illustrate this, let's take an example. Let's say you start with a 57384631Smjacobloop that has 5 disks in it. At boot time, the system will likely find 57484631Smjacobthem and see them in this order: 57584631Smjacob 57684631Smjacobdisk# Loop ID Target ID 57784631Smjacobdisk0 0 0 57884631Smjacobdisk1 1 1 57984631Smjacobdisk2 2 2 58084631Smjacobdisk3 3 3 58184631Smjacobdisk4 4 4 58284631Smjacob 58384631SmjacobThe driver uses 'Loop ID' when it forms requests to send a comamnd to 58484631Smjacobeach disk. However, it reports to NetBSD that things exist as 'Target 58584631SmjacobID'. As you can see here, there is perfect correspondence between disk, 58684631SmjacobLoop ID and Target ID. 58784631Smjacob 58884631SmjacobLet's say you add a new disk between disk2 and disk3 while things are 58984631Smjacobrunning. You don't really often see this, but you *could* see this where 59084631Smjacobthe loop has to renegotiate, and you end up with: 59184631Smjacob 59284631Smjacobdisk# Loop ID Target ID 59384631Smjacobdisk0 0 0 59484631Smjacobdisk1 1 1 59584631Smjacobdisk2 2 2 59684631SmjacobdiskN 3 ? 59784631Smjacobdisk3 4 ? 59884631Smjacobdisk4 5 ? 59984631Smjacob 60084631SmjacobClearly, you don't want disk3 and disk4's "Target ID" to change while you're 60184631Smjacobrunning since currently mounted filesystems will get trashed. 60284631Smjacob 60384631SmjacobWhat the driver is supposed to do (this is the function of isp_scan_loop), 60484631Smjacobis regenerate things such that the following then occurs: 60584631Smjacob 60684631Smjacobdisk# Loop ID Target ID 60784631Smjacobdisk0 0 0 60884631Smjacobdisk1 1 1 60984631Smjacobdisk2 2 2 61084631SmjacobdiskN 3 5 61184631Smjacobdisk3 4 3 61284631Smjacobdisk4 5 4 61384631Smjacob 61484631SmjacobSo, "Target" is a virtual entity that is maintained while you're running. 61584631Smjacob 61684631Smjacob6. Glossary 61784631Smjacob 61884631SmjacobHBA - Host Bus Adapter 61984631Smjacob 62084631SmjacobSCSI - Small Computer 62184631Smjacob 62284631Smjacob7. References 62384631Smjacob 62484631SmjacobVarious URLs of interest: 62584631Smjacob 62684631Smjacobhttp://www.netbsd.org - NetBSD's Web Page 62784631Smjacobhttp://www.openbsd.org - OpenBSD's Web Page 62884631Smjacobhttp://www.freebsd.org - FreeBSD's Web Page 62984631Smjacob 63084631Smjacobhttp://www.t10.org - ANSI SCSI Commitee's Web Page 63184631Smjacob (SCSI Specs) 63284631Smjacobhttp://www.t11.org - NCITS Device Interface Web Page 63384631Smjacob (Fibre Channel Specs) 63484631Smjacob 635