184631Smjacob/* $FreeBSD: releng/10.2/sys/dev/isp/Hardware.txt 108533 2003-01-01 18:49:04Z schweikh $ */ 284631Smjacob 384631Smjacob Hardware that is Known To or Should Work with This Driver 484631Smjacob 584631Smjacob 684631Smjacob0. Intro 784631Smjacob 884631Smjacob This is not an endorsement for hardware vendors (there will be 984631Smjacob no "where to buy" URLs here with a couple of exception). This 1084631Smjacob is simply a list of things I know work, or should work, plus 1184631Smjacob maybe a couple of notes as to what you should do to make it 1284631Smjacob work. Corrections accepted. Even better would be to send me 1384631Smjacob hardware to I can test it. 1484631Smjacob 1584631Smjacob I'll put a rough range of costs in US$ that I know about. No doubt 1684631Smjacob it'll differ from your expectations. 1784631Smjacob 1884631Smjacob1. HBAs 1984631Smjacob 2084631SmjacobQlogic 2100, 2102 2184631Smjacob 2200, 2202, 2204 2284631Smjacob 2384631Smjacob There are various suffices that indicate copper or optical 2484631Smjacob connectors, or 33 vs. 66MHz PCI bus operation. None of these 2584631Smjacob have a software impact. 2684631Smjacob 2784631Smjacob Approx cost: 1K$ for a 2200 2884631Smjacob 2984631SmjacobQlogic 2300, 2312 3084631Smjacob 3184631Smjacob These are the new 2-Gigabit cards. Optical only. 3284631Smjacob 3384631Smjacob Approx cost: ?????? 3484631Smjacob 3584631Smjacob 3684631SmjacobAntares P-0033, P-0034, P-0036 3784631Smjacob 3884631Smjacob There many other vendors that use the Qlogic 2X00 chipset. Some older 3984631Smjacob 2100 boards (not on this list) have a bug in the ROM that causes a 4084631Smjacob failure to download newer firmware that is larger than 0x7fff words. 4184631Smjacob 4284631Smjacob Approx cost: 850$ for a P-0036 4384631Smjacob 4484631Smjacob 4584631Smjacob 4684631Smjacob In general, the 2200 class chip is to be preferred. 4784631Smjacob 4884631Smjacob 4984631Smjacob2. Hubs 5084631Smjacob 5184631SmjacobVixel 1000 5284631SmjacobVixel 2000 5384631Smjacob Of the two, the 1000 (7 ports, vs. 12 ports) has had fewer problems- 5484631Smjacob it's an old workhorse. 5584631Smjacob 5684631Smjacob 5784631Smjacob Approx cost: 1.5K$ for Vixel 1000, 2.5K$ for 2000 5884631Smjacob 5984631SmjacobGadzoox Cappellix 3000 6084631Smjacob Don't forget to use telnet to configure the Cappellix ports 6184631Smjacob to the role you're using them for- otherwise things don't 6284631Smjacob work well at all. 6384631Smjacob 6484631Smjacob (cost: I have no idea... certainly less than a switch) 6584631Smjacob 6684631Smjacob3. Switches 6784631Smjacob 6884631SmjacobBrocade Silkworm II 6984631SmjacobBrocade 2400 7084631Smjacob(other brocades should be fine) 7184631Smjacob 7284631Smjacob Especially with revision 2 or higher f/w, this is now best 7384631Smjacob of breed for fabrics or segmented loop (which Brocade 7484631Smjacob calls "QuickLoop"). 7584631Smjacob 7684631Smjacob For the Silkworm II, set operating mode to "Tachyon" (mode 3). 7784631Smjacob 7884631Smjacob The web interace isn't good- but telnet is what I prefer anyhow. 7984631Smjacob 8084631Smjacob You can't connect a Silkworm II and the other Brocades together 8184631Smjacob as E-ports to make a large fabric (at least with the f/w *I* 8284631Smjacob had for the Silkworm II). 8384631Smjacob 8484631Smjacob Approx cost of a Brocade 2400 with no GBICs is about 8K$ when 8584631Smjacob I recently checked the US Government SEWP price list- no doubt 8684631Smjacob it'll be a bit more for others. I'd assume around 10K$. 8784631Smjacob 8884631SmjacobANCOR SA-8 8984631Smjacob 9084631Smjacob This also is a fine switch, but you have to use a browser 9184631Smjacob with working java to manage it- which is a bit of a pain. 9284631Smjacob This also supports fabric and segmented loop. 9384631Smjacob 9484631Smjacob These switches don't form E-ports with each other for a larger 9584631Smjacob fabric. 9684631Smjacob 9784631Smjacob (cost: no idea) 9884631Smjacob 9984631SmjacobMcData (model unknown) 10084631Smjacob 10184631Smjacob I tried one exactly once for 30 minutes. Seemed to work once 10284631Smjacob I added the "register FC4 types" command to the driver. 10384631Smjacob 10484631Smjacob (cost: very very expensive, 40K$ plus) 10584631Smjacob 10684631Smjacob4. Cables/GBICs 10784631Smjacob 10884631Smjacob Multimode optical is adequate for Fibre Channel- the same cable is 10984631Smjacob used for Gigabit Ethernet. 11084631Smjacob 11184631Smjacob Copper DB-9 and Copper HSS-DC connectors are also fine. Copper && 11284631Smjacob Optical both are rated to 1.026Gbit- copper is naturally shorter 11384631Smjacob (the longest I've used is a 15meter cable but it's supposed to go 11484631Smjacob longer). 11584631Smjacob 11684631Smjacob The reason to use copper instead of optical is that if step on one of 11784631Smjacob the really fat DB-9 cables you can get, it'll survive. Optical usually 11884631Smjacob dies quickly if you step on it. 11984631Smjacob 12084631Smjacob Approx cost: I don't know what optical is- you can expect to pay maybe 12184631Smjacob a 100$ for a 3m copper cable. 12284631Smjacob 12384631SmjacobGBICs- 12484631Smjacob 12584631Smjacob I use Finisar copper and IBM Opticals. 12684631Smjacob 12784631Smjacob Approx Cost: Copper GBICs are 70$ each. Opticals are twice that or more. 12884631Smjacob 12984631Smjacob 13084631SmjacobVendor: (this is the one exception I'll make because it turns out to be 13184631Smjacob an incredible pain to find FC copper cabling and GBICs- the source I 13284631Smjacob use for GBICs and copper cables is http://www.scsi-cables.com) 13384631Smjacob 13484631Smjacob 13584631SmjacobOther: 13684631Smjacob There now is apparently a source for little connector boards 13784631Smjacob to connect to bare drives: http://www.cinonic.com. 13884631Smjacob 13984631Smjacob 14084631Smjacob5. Storage JBODs/RAID 14184631Smjacob 14284631SmjacobJMR 4-Bay 14384631Smjacob 14484631Smjacob Rinky-tink, but a solid 4 bay loop only entry model. 14584631Smjacob 14684631Smjacob I paid 1000$ for mine- overprice, IMO. 14784631Smjacob 14884631SmjacobJMR Fortra 14984631Smjacob 150108533Sschweikh I rather like this box. The blue LEDs are a very nice touch- you 15184631Smjacob can see them very clearly from 50 feet away. 15284631Smjacob 15384631Smjacob I paid 2000$ for one used. 15484631Smjacob 15584631SmjacobSun A5X00 15684631Smjacob 15784631Smjacob Very expensive (in my opinion) but well crafted. Has two SES 15884631Smjacob instances, so you can use the ses driver (and the example 15984631Smjacob code in /usr/share/examples) for power/thermal/slot monitoring. 16084631Smjacob 16184631Smjacob Approx Cost: The last I saw for a price list item on this was 22K$ 162108533Sschweikh for an unpopulated (no disk drive) A5X00. 16384631Smjacob 16484631Smjacob 16584631SmjacobDataDirect E1000 RAID 16684631Smjacob 16784631Smjacob Don't connect both SCSI and FC interfaces at the same time- a SCSI 16884631Smjacob reset will cause the DataDirect to think you want to use the SCSI 16984631Smjacob interface and a LIP on the FC interface will cause it to think you 17084631Smjacob want to use the FC interface. Use only one connector at a time so 17184631Smjacob both you and the DataDirect are sure about what you want. 17284631Smjacob 17384631Smjacob Cost: I have no idea. 17484631Smjacob 17584631SmjacobVeritas ServPoint 17684631Smjacob 17784631Smjacob This is a software storage virtualization engine that 178108533Sschweikh runs on Sparc/Solaris in target mode for frontend 17984631Smjacob and with other FC or SCSI as the backend storage. FreeBSD 18084631Smjacob has been used extensively to test it. 18184631Smjacob 18284631Smjacob 18384631Smjacob Cost: I have no idea. 18484631Smjacob 18584631Smjacob6. Disk Drives 18684631Smjacob 18784631Smjacob I have used lots of different Seagate and a few IBM drives and 18884631Smjacob typically have had few problems with them. These are the bare 18984631Smjacob drives with 40-pin SCA connectors in back. They go into the JBODs 19084631Smjacob you assemble. 19184631Smjacob 19284631Smjacob Seagate does make, but I can no longer find, a little paddleboard 19384631Smjacob single drive connector that goes from DB-9 FC to the 40-pin SCA 19484631Smjacob connector- primarily for you to try and evaluate a single FC drive. 19584631Smjacob 19684631Smjacob All FC-AL disk drives are dual ported (i.e., have separte 'A' and 19784631Smjacob 'B' ports- which are completely separate loops). This seems to work 19884631Smjacob reasonably enough, but I haven't tested it much. It really depends 19984631Smjacob on the JBOD you put them to carry this dual port to the outside 20084631Smjacob world. The JMR boxes have it. The Sun A5X00 you have to pay for 20184631Smjacob an extra IB card to carry it out. 20284631Smjacob 20384631Smjacob Approx Cost: You'll find that FC drives are the same cost if not 20484631Smjacob slightly cheaper than the equivalent Ultra3 SCSI drives. 20584631Smjacob 20684631Smjacob7. Recommended Configurations 20784631Smjacob 20884631SmjacobThese are recommendations that are biased toward the cautious side. They 20984631Smjacobdo not represent formal engineering commitments- just suggestions as to 21084631Smjacobwhat I would expect to work. 21184631Smjacob 21284631SmjacobA. The simpletst form of a connection topology I can suggest for 21384631Smjacoba small SAN (i.e., replacement for SCSI JBOD/RAID): 21484631Smjacob 21584631SmjacobHOST 21684631Smjacob2xxx <----------> Single Unit of Storage (JBOD, RAID) 21784631Smjacob 21884631SmjacobThis is called a PL_DA (Private Loop, Direct Attach) topology. 21984631Smjacob 22084631SmjacobB. The next most simple form of a connection topology I can suggest for 22184631Smjacoba medium local SAN (where you do not plan to do dynamic insertion 22284631Smjacoband removal of devices while I/Os are active): 22384631Smjacob 22484631SmjacobHOST 22584631Smjacob2xxx <----------> +-------- 22684631Smjacob | Vixel | 22784631Smjacob | 1000 | 22884631Smjacob | +<---> Storage 22984631Smjacob | | 23084631Smjacob | +<---> Storage 23184631Smjacob | | 23284631Smjacob | +<---> Storage 23384631Smjacob -------- 23484631Smjacob 23584631SmjacobThis is a Private Loop topology. Remember that this can get very unstable 23684631Smjacobif you make it too long. A good practice is to try it in a staged fashion. 23784631Smjacob 23884631SmjacobIt is possible with some units to "daisy chain", e.g.: 23984631Smjacob 24084631SmjacobHOST 24184631Smjacob2xxx <----------> (JBOD, RAID) <--------> (JBOD, RAID) 24284631Smjacob 24384631SmjacobIn practice I have had poor results with these configurations. They *should* 24484631Smjacobwork fine, but for both the JMR and the Sun A5X00 I tend to get LIP storms 24584631Smjacoband so the second unit just isn't seen and the loop isn't stable. 24684631Smjacob 24784631SmjacobNow, this could simply be my lack of clean, newer, h/w (or, in general, 24884631Smjacoba lack of h/w), but I would recommend the use of a hub if you want to 24984631Smjacobstay with Private Loop and have more than one FC target. 25084631Smjacob 25184631SmjacobYou should also note this can begin to be the basis for a shared SAN 25284631Smjacobsolution. For example, the above configuration can be extended to be: 25384631Smjacob 25484631SmjacobHOST 25584631Smjacob2xxx <----------> +-------- 25684631Smjacob | Vixel | 25784631Smjacob | 1000 | 25884631Smjacob | +<---> Storage 25984631Smjacob | | 26084631Smjacob | +<---> Storage 26184631Smjacob | | 26284631Smjacob | +<---> Storage 26384631SmjacobHOST | | 26484631Smjacob2xxx <----------> +-------- 26584631Smjacob 26684631SmjacobHowever, note that there is nothing to mediate locking of devices, and 26784631Smjacobit is also conceivable that the reboot of one host can, by causing 26884631Smjacoba LIP storm, cause problems with the I/Os from the other host. 26984631Smjacob(in other words, this topology hasn't really been made safe yet for 27084631Smjacobthis driver). 27184631Smjacob 27284631SmjacobD. You can repeat the topology in #B with a switch that is set to be 27384631Smjacobin segmented loop mode. This avoids LIPs propagating where you don't 27484631Smjacobwant them to- and this makes for a much more reliable, if more expensive, 27584631SmjacobSAN. 27684631Smjacob 27784631SmjacobE. The next level of complexity is a Switched Fabric. The following topology 27884631Smjacobis good when you start to begin to get to want more performance. Private 27984631Smjacoband Public Arbitrated Loop, while 100MB/s, is a shared medium. Direct 28084631Smjacobconnections to a switch can run full-duplex at full speed. 28184631Smjacob 28284631SmjacobHOST 28384631Smjacob2xxx <----------> +--------- 28484631Smjacob | Brocade| 28584631Smjacob | 2400 | 28684631Smjacob | +<---> Storage 28784631Smjacob | | 28884631Smjacob | +<---> Storage 28984631Smjacob | | 29084631Smjacob | +<---> Storage 29184631SmjacobHOST | | 29284631Smjacob2xxx <----------> +--------- 29384631Smjacob 29484631Smjacob 29584631SmjacobI would call this the best configuration available now. It can expand 29684631Smjacobsubstantially if you cascade switches. 29784631Smjacob 29884631SmjacobThere is a hard limit of about 253 devices for each Qlogic HBA- and the 29984631Smjacobfabric login policy is simplistic (log them in as you find them). If 30084631Smjacobsomebody actually runs into a configuration that's larger, let me know 30184631Smjacoband I'll work on some tools that would allow you some policy choices 30284631Smjacobas to which would be interesting devices to actually connect to. 30384631Smjacob 30484631Smjacob 305