1222588SjfvFreeBSD Driver for Intel(R) Ethernet 10 Gigabit PCI Express Server Adapters 2222588Sjfv============================================================================ 3190873Sjfv/*$FreeBSD$*/ 4179055Sjfv 5251964SjfvJun 18, 2013 6179055Sjfv 7190873Sjfv 8179055SjfvContents 9179055Sjfv======== 10179055Sjfv 11179055Sjfv- Overview 12179055Sjfv- Supported Adapters 13179055Sjfv- Building and Installation 14222588Sjfv- Additional Configurations and Tuning 15179055Sjfv- Known Limitations 16179055Sjfv 17179055Sjfv 18179055SjfvOverview 19179055Sjfv======== 20179055Sjfv 21251964SjfvThis file describes the FreeBSD* driver for the 22251964SjfvIntel(R) Ethernet 10 Gigabit Family of Adapters. 23179055Sjfv 24179055SjfvFor questions related to hardware requirements, refer to the documentation 25179055Sjfvsupplied with your Intel 10GbE adapter. All hardware requirements listed 26179055Sjfvapply to use with FreeBSD. 27179055Sjfv 28179055Sjfv 29179055SjfvSupported Adapters 30179055Sjfv================== 31179055Sjfv 32222588SjfvThe driver in this release is compatible with 82598 and 82599-based Intel 33222588SjfvNetwork Connections. 34179055Sjfv 35222588SjfvSFP+ Devices with Pluggable Optics 36222588Sjfv---------------------------------- 37179055Sjfv 38222588Sjfv82599-BASED ADAPTERS 39179055Sjfv 40222588SjfvNOTE: If your 82599-based Intel(R) Ethernet Network Adapter came with Intel 41222588Sjfvoptics, or is an Intel(R) Ethernet Server Adapter X520-2, then it only supports 42222588SjfvIntel optics and/or the direct attach cables listed below. 43179055Sjfv 44222588SjfvWhen 82599-based SFP+ devices are connected back to back, they should be set to 45251964Sjfvthe same Speed setting. Results may vary if you mix speed settings. 46222588Sjfv 47222588SjfvSupplier Type Part Numbers 48179055Sjfv 49222588SjfvSR Modules 50222588SjfvIntel DUAL RATE 1G/10G SFP+ SR (bailed) FTLX8571D3BCV-IT 51222588SjfvIntel DUAL RATE 1G/10G SFP+ SR (bailed) AFBR-703SDZ-IN2 52222588SjfvIntel DUAL RATE 1G/10G SFP+ SR (bailed) AFBR-703SDDZ-IN1 53222588SjfvLR Modules 54222588SjfvIntel DUAL RATE 1G/10G SFP+ LR (bailed) FTLX1471D3BCV-IT 55222588SjfvIntel DUAL RATE 1G/10G SFP+ LR (bailed) AFCT-701SDZ-IN2 56222588SjfvIntel DUAL RATE 1G/10G SFP+ LR (bailed) AFCT-701SDDZ-IN1 57179055Sjfv 58222588SjfvThe following is a list of 3rd party SFP+ modules and direct attach cables that 59222588Sjfvhave received some testing. Not all modules are applicable to all devices. 60179055Sjfv 61222588SjfvSupplier Type Part Numbers 62179055Sjfv 63222588SjfvFinisar SFP+ SR bailed, 10g single rate FTLX8571D3BCL 64222588SjfvAvago SFP+ SR bailed, 10g single rate AFBR-700SDZ 65222588SjfvFinisar SFP+ LR bailed, 10g single rate FTLX8571D3BCV-IT 66222588Sjfv 67222588SjfvFinisar DUAL RATE 1G/10G SFP+ SR (No Bail) FTLX8571D3QCV-IT 68222588SjfvAvago DUAL RATE 1G/10G SFP+ SR (No Bail) AFBR-703SDZ-IN1 69222588SjfvFinisar DUAL RATE 1G/10G SFP+ LR (No Bail) FTLX1471D3QCV-IT 70222588SjfvAvago DUAL RATE 1G/10G SFP+ LR (No Bail) AFCT-701SDZ-IN1 71222588SjfvFinistar 1000BASE-T SFP FCLF8522P2BTL 72222588SjfvAvago 1000BASE-T SFP ABCU-5710RZ 73251964Sjfv 74251964SjfvNOTE: As of driver version 2.5.13 it is possible to allow the operation 75251964Sjfvof unsupported modules by setting the static variable 'allow_unsupported_sfp' 76251964Sjfvto TRUE and rebuilding the driver. If problems occur please assure that they 77251964Sjfvcan be reproduced with fully supported optics first. 78251964Sjfv 79222588Sjfv82599-based adapters support all passive and active limiting direct attach 80222588Sjfvcables that comply with SFF-8431 v4.1 and SFF-8472 v10.4 specifications. 81179055Sjfv 82222588SjfvLaser turns off for SFP+ when ifconfig down 83222588Sjfv-------------------------------------------------------- 84222588Sjfv"ifconfig down" turns off the laser for 82599-based SFP+ fiber adapters. 85222588Sjfv"ifconfig up" turns on the later. 86179055Sjfv 87222588Sjfv82598-BASED ADAPTERS 88179055Sjfv 89222588SjfvNOTES for 82598-Based Adapters: 90222588Sjfv- Intel(R) Ethernet Network Adapters that support removable optical modules 91222588Sjfv only support their original module type (i.e., the Intel(R) 10 Gigabit SR 92222588Sjfv Dual Port Express Module only supports SR optical modules). If you plug 93222588Sjfv in a different type of module, the driver will not load. 94222588Sjfv- Hot Swapping/hot plugging optical modules is not supported. 95222588Sjfv- Only single speed, 10 gigabit modules are supported. 96222588Sjfv- LAN on Motherboard (LOMs) may support DA, SR, or LR modules. Other module 97222588Sjfv types are not supported. Please see your system documentation for details. 98179055Sjfv 99222588SjfvThe following is a list of 3rd party SFP+ modules and direct attach cables that have 100222588Sjfvreceived some testing. Not all modules are applicable to all devices. 101179055Sjfv 102222588SjfvSupplier Type Part Numbers 103179055Sjfv 104222588SjfvFinisar SFP+ SR bailed, 10g single rate FTLX8571D3BCL 105222588SjfvAvago SFP+ SR bailed, 10g single rate AFBR-700SDZ 106222588SjfvFinisar SFP+ LR bailed, 10g single rate FTLX1471D3BCL 107222588Sjfv 108222588Sjfv82598-based adapters support all passive direct attach cables that comply 109222588Sjfvwith SFF-8431 v4.1 and SFF-8472 v10.4 specifications. Active direct attach 110222588Sjfvcables are not supported. 111179055Sjfv 112222588SjfvThird party optic modules and cables referred to above are listed only for the 113222588Sjfvpurpose of highlighting third party specifications and potential compatibility, 114222588Sjfvand are not recommendations or endorsements or sponsorship of any third party's 115222588Sjfvproduct by Intel. Intel is not endorsing or promoting products made by any 116222588Sjfvthird party and the third party reference is provided only to share information 117222588Sjfvregarding certain optic modules and cables with the above specifications. There 118222588Sjfvmay be other manufacturers or suppliers, producing or supplying optic modules 119222588Sjfvand cables with similar or matching descriptions. Customers must use their own 120222588Sjfvdiscretion and diligence to purchase optic modules and cables from any third 121222588Sjfvparty of their choice. Customer are solely responsible for assessing the 122222588Sjfvsuitability of the product and/or devices and for the selection of the vendor 123222588Sjfvfor purchasing any product. INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL 124222588SjfvDISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF 125222588SjfvSUCH THIRD PARTY PRODUCTS OR SELECTION OF VENDOR BY CUSTOMERS. 126179055Sjfv 127179055SjfvConfiguration and Tuning 128222588Sjfv======================== 129179055Sjfv 130179055SjfvThe driver supports Transmit/Receive Checksum Offload and Jumbo Frames on 131179055Sjfvall 10 Gigabit adapters. 132179055Sjfv 133179055Sjfv Jumbo Frames 134179055Sjfv ------------ 135179055Sjfv To enable Jumbo Frames, use the ifconfig utility to increase the MTU 136179055Sjfv beyond 1500 bytes. 137179055Sjfv 138179055Sjfv NOTES: 139179055Sjfv 140179055Sjfv - The Jumbo Frames setting on the switch must be set to at least 141179055Sjfv 22 bytes larger than that of the adapter. 142179055Sjfv 143179055Sjfv - There are known performance issues with this driver when running 144179055Sjfv UDP traffic with Jumbo Frames. 145179055Sjfv 146179055Sjfv The Jumbo Frames MTU range for Intel Adapters is 1500 to 16114. The default 147179055Sjfv MTU range is 1500. To modify the setting, enter the following: 148179055Sjfv 149222588Sjfv ifconfig ix<interface_num> <hostname or IP address> mtu 9000 150179055Sjfv 151179055Sjfv To confirm an interface's MTU value, use the ifconfig command. To confirm 152179055Sjfv the MTU used between two specific devices, use: 153179055Sjfv 154179055Sjfv route get <destination_IP_address> 155179055Sjfv 156179055Sjfv VLANs 157179055Sjfv ----- 158179055Sjfv To create a new VLAN pseudo-interface: 159179055Sjfv 160179055Sjfv ifconfig <vlan_name> create 161179055Sjfv 162179055Sjfv To associate the VLAN pseudo-interface with a physical interface and 163179055Sjfv assign a VLAN ID, IP address, and netmask: 164179055Sjfv 165179055Sjfv ifconfig <vlan_name> <ip_address> netmask <subnet_mask> vlan 166179055Sjfv <vlan_id> vlandev <physical_interface> 167179055Sjfv 168179055Sjfv Example: 169179055Sjfv 170179055Sjfv ifconfig vlan10 10.0.0.1 netmask 255.255.255.0 vlan 10 vlandev ixgbe0 171179055Sjfv 172179055Sjfv In this example, all packets will be marked on egress with 802.1Q VLAN 173179055Sjfv tags, specifying a VLAN ID of 10. 174179055Sjfv 175179055Sjfv To remove a VLAN pseudo-interface: 176179055Sjfv 177179055Sjfv ifconfig <vlan_name> destroy 178179055Sjfv 179179055Sjfv 180179055Sjfv Checksum Offload 181179055Sjfv ---------------- 182179055Sjfv 183179055Sjfv Checksum offloading supports both TCP and UDP packets and is 184179055Sjfv supported for both transmit and receive. 185179055Sjfv 186179055Sjfv Checksum offloading can be enabled or disabled using ifconfig. 187179055Sjfv Both transmit and receive offloading will be either enabled or 188179055Sjfv disabled together. You cannot enable/disable one without the other. 189179055Sjfv 190179055Sjfv To enable checksum offloading: 191179055Sjfv 192179055Sjfv ifconfig <interface_num> rxcsum 193179055Sjfv 194179055Sjfv To disable checksum offloading: 195179055Sjfv 196179055Sjfv ifconfig <interface_num> -rxcsum 197179055Sjfv 198179055Sjfv To confirm the current setting: 199179055Sjfv 200179055Sjfv ifconfig <interface_num> 201179055Sjfv 202179055Sjfv 203179055Sjfv TSO 204179055Sjfv --- 205179055Sjfv 206222588Sjfv TSO is enabled by default. 207222588Sjfv 208179055Sjfv To disable: 209179055Sjfv 210179055Sjfv ifconfig <interface_num> -tso 211179055Sjfv 212179055Sjfv To re-enable: 213179055Sjfv 214179055Sjfv ifconfig <interface_num> tso 215179055Sjfv 216179055Sjfv LRO 217222588Sjfv --- 218179055Sjfv 219222588Sjfv Large Receive Offload is available in the driver; it is on by default. 220222588Sjfv It can be disabled by using: 221222588Sjfv ifconfig <interface_num> -lro 222222588Sjfv To enable: 223222588Sjfv ifconfig <interface_num> lro 224179055Sjfv 225179055Sjfv 226179055Sjfv Important system configuration changes: 227179055Sjfv --------------------------------------- 228179055Sjfv 229222588Sjfv When there is a choice run on a 64bit OS rather than 32, it makes a 230222588Sjfv significant difference in improvement. 231222588Sjfv 232251964Sjfv The interface can generate a high number of interrupts. To avoid running 233222588Sjfv into the limit set by the kernel, adjust hw.intr_storm_threshold 234222588Sjfv setting using sysctl: 235179055Sjfv 236222588Sjfv sysctl hw.intr_storm_threshold=9000 (the default is 1000) 237179055Sjfv 238222588Sjfv For this change to take effect on boot, edit /etc/sysctl.conf and add the 239222588Sjfv line: 240222588Sjfv hw.intr_storm_threshold=9000 241222588Sjfv 242222588Sjfv If you still see Interrupt Storm detected messages, increase the limit to a 243251964Sjfv higher number, or the detection can be disabled by setting it to 0. 244222588Sjfv 245251964Sjfv The default number of descriptors is 2048, increasing or descreasing 246251964Sjfv may improve performance in some workloads, but change carefully. 247179055Sjfv 248179055Sjfv 249179055SjfvKnown Limitations 250179055Sjfv================= 251222588Sjfv 252222588SjfvFor known hardware and troubleshooting issues, refer to the following website. 253222588Sjfv 254222588Sjfv http://support.intel.com/support/go/network/adapter/home.htm 255222588Sjfv 256222588SjfvEither select the link for your adapter or perform a search for the adapter 257222588Sjfvnumber. The adapter's page lists many issues. For a complete list of hardware 258222588Sjfvissues download your adapter's user guide and read the Release Notes. 259222588Sjfv 260222588Sjfv UDP stress test with 10GbE driver 261222588Sjfv --------------------------------- 262179055Sjfv Under small packets UDP stress test with 10GbE driver, the FreeBSD system 263179055Sjfv will drop UDP packets due to the fullness of socket buffers. You may want 264179055Sjfv to change the driver's Flow Control variables to the minimum value for 265179055Sjfv controlling packet reception. 266179055Sjfv 267222588Sjfv Attempting to configure larger MTUs with a large numbers of processors may 268222588Sjfv generate the error message "ix0:could not setup receive structures" 269222588Sjfv -------------------------------------------------------------------------- 270222588Sjfv When using the ixgbe driver with RSS autoconfigured based on the number of 271222588Sjfv cores (the default setting) and that number is larger than 4, increase the 272222588Sjfv memory resources allocated for the mbuf pool as follows: 273179055Sjfv 274222588Sjfv Add to the sysctl.conf file for the system: 275222588Sjfv 276222588Sjfv kern.ipc.nmbclusters=262144 277222588Sjfv kern.ipc.nmbjumbop=262144 278222588Sjfv 279222588Sjfv Lower than expected performance on dual port 10GbE devices 280222588Sjfv ---------------------------------------------------------- 281222588Sjfv Some PCI-E x8 slots are actually configured as x4 slots. These slots have 282222588Sjfv insufficient bandwidth for full 10Gbe line rate with dual port 10GbE devices. 283251964Sjfv The driver will detect this situation and will write the following message in 284222588Sjfv the system log: "PCI-Express bandwidth available for this card is not 285222588Sjfv sufficient for optimal performance. For optimal performance a x8 PCI-Express 286222588Sjfv slot is required." 287222588Sjfv 288222588Sjfv If this error occurs, moving your adapter to a true x8 slot will resolve the 289222588Sjfv issue. 290222588Sjfv 291222588Sjfv 292222588Sjfv 293179055SjfvSupport 294179055Sjfv======= 295179055Sjfv 296179055SjfvFor general information and support, go to the Intel support website at: 297179055Sjfv 298222588Sjfv www.intel.com/support/ 299179055Sjfv 300179055SjfvIf an issue is identified with the released source code on the supported 301179055Sjfvkernel with a supported adapter, email the specific information related to 302222588Sjfvthe issue to freebsd@intel.com 303179055Sjfv 304179055Sjfv 305179055Sjfv 306179055SjfvLicense 307179055Sjfv======= 308179055Sjfv 309179055SjfvThis software program is released under the terms of a license agreement 310179055Sjfvbetween you ('Licensee') and Intel. Do not use or load this software or any 311179055Sjfvassociated materials (collectively, the 'Software') until you have carefully 312179055Sjfvread the full terms and conditions of the LICENSE located in this software 313179055Sjfvpackage. By loading or using the Software, you agree to the terms of this 314179055SjfvAgreement. If you do not agree with the terms of this Agreement, do not 315179055Sjfvinstall or use the Software. 316179055Sjfv 317179055Sjfv* Other names and brands may be claimed as the property of others. 318179055Sjfv 319179055Sjfv 320