1270772Sjfv ixl FreeBSD* Base Driver and ixlv VF Driver for the 2270772Sjfv Intel XL710 Ethernet Controller Family 3269201Sjfv 4269201Sjfv/*$FreeBSD: releng/11.0/sys/dev/ixl/README 270772 2014-08-28 21:45:07Z jfv $*/ 5270346Sjfv================================================================ 6269201Sjfv 7270772SjfvAugust 26, 2014 8269201Sjfv 9269201Sjfv 10269201SjfvContents 11269201Sjfv======== 12269201Sjfv 13269201Sjfv- Overview 14269201Sjfv- Supported Adapters 15270772Sjfv- The VF Driver 16269201Sjfv- Building and Installation 17269201Sjfv- Additional Configurations 18269201Sjfv- Known Limitations 19269201Sjfv 20269201Sjfv 21269201SjfvOverview 22269201Sjfv======== 23269201Sjfv 24270772SjfvThis file describes the IXL FreeBSD* Base driver and the IXLV VF Driver 25270772Sjfvfor the XL710 Ethernet Family of Adapters. The Driver has been developed 26270772Sjfvfor use with FreeBSD 10.0 or later, but should be compatible with any 27270772Sjfvsupported release. 28269201Sjfv 29270772SjfvFor questions related to hardware requirements, refer to the documentation 30270772Sjfvsupplied with your Intel XL710 adapter. All hardware requirements listed 31270772Sjfvapply for use with FreeBSD. 32269201Sjfv 33269201Sjfv 34269201SjfvSupported Adapters 35269201Sjfv================== 36269201Sjfv 37270772SjfvThe drivers in this release are compatible with XL710 and X710-based 38270772SjfvIntel Ethernet Network Connections. 39269201Sjfv 40269201Sjfv 41269201SjfvSFP+ Devices with Pluggable Optics 42269201Sjfv---------------------------------- 43269201Sjfv 44269201SjfvSR Modules 45269201Sjfv---------- 46269201Sjfv Intel DUAL RATE 1G/10G SFP+ SR (bailed) FTLX8571D3BCV-IT 47269201Sjfv Intel DUAL RATE 1G/10G SFP+ SR (bailed) AFBR-703SDZ-IN2 48269201Sjfv 49269201SjfvLR Modules 50269201Sjfv---------- 51269201Sjfv Intel DUAL RATE 1G/10G SFP+ LR (bailed) FTLX1471D3BCV-IT 52269201Sjfv Intel DUAL RATE 1G/10G SFP+ LR (bailed) AFCT-701SDZ-IN2 53269201Sjfv 54269201SjfvQSFP+ Modules 55269201Sjfv------------- 56269201Sjfv Intel TRIPLE RATE 1G/10G/40G QSFP+ SR (bailed) E40GQSFPSR 57269201Sjfv Intel TRIPLE RATE 1G/10G/40G QSFP+ LR (bailed) E40GQSFPLR 58269201Sjfv QSFP+ 1G speed is not supported on XL710 based devices. 59269201Sjfv 60270772SjfvX710/XL710 Based SFP+ adapters support all passive and active limiting direct 61270772Sjfvattach cables that comply with SFF-8431 v4.1 and SFF-8472 v10.4 specifications. 62269201Sjfv 63270772SjfvThe VF Driver 64270772Sjfv================== 65270772SjfvThe VF driver is normally used in a virtualized environment where a host 66270772Sjfvdriver manages SRIOV, and provides a VF device to the guest. With this 67270772Sjfvfirst release the only host environment tested was using Linux QEMU/KVM. 68270772SjfvSupport is planned for Xen and VMWare hosts at a later time. 69269201Sjfv 70270772SjfvIn the FreeBSD guest the IXLV driver would be loaded and will function 71270772Sjfvusing the VF device assigned to it. 72270772Sjfv 73270772SjfvThe VF driver provides most of the same functionality as the CORE driver, 74270772Sjfvbut is actually a slave to the Host, access to many controls are actually 75270772Sjfvaccomplished by a request to the Host via what is called the "Admin queue". 76270772SjfvThese are startup and initialization events however, once in operation 77270772Sjfvthe device is self-contained and should achieve near native performance. 78270772Sjfv 79270772SjfvSome notable limitations of the VF environment: for security reasons 80270772Sjfvthe driver is never permitted to be promiscuous, therefore a tcpdump 81270772Sjfvwill not behave the same with the interface. Second, media info is not 82270772Sjfvavailable from the PF, so it will always appear as auto. 83270772Sjfv 84270772SjfvTarball Building and Installation 85269201Sjfv========================= 86269201Sjfv 87270772SjfvNOTE: You must have kernel sources installed to compile the driver tarball. 88269201Sjfv 89270772SjfvThese instructions assume a standalone driver tarball, building the driver 90270772Sjfvalready in the kernel source is simply a matter of adding the device entry 91270772Sjfvto the kernel config file, or building in the ixl or ixlv module directory. 92270772Sjfv 93269201SjfvIn the instructions below, x.x.x is the driver version 94270772Sjfvas indicated in the name of the driver tarball. The example is 95270772Sjfvfor ixl, the same procedure applies for ixlv. 96269201Sjfv 97270772Sjfv1. Move the base driver tar file to the directory of your choice. 98270772Sjfv For example, use /home/username/ixl or /usr/local/src/ixl. 99269201Sjfv 100269201Sjfv2. Untar/unzip the archive: 101270346Sjfv tar xfz ixl-x.x.x.tar.gz 102269201Sjfv 103269201Sjfv3. To install man page: 104270346Sjfv cd ixl-x.x.x 105270346Sjfv gzip -c ixl.4 > /usr/share/man/man4/ixl.4.gz 106269201Sjfv 107269201Sjfv4. To load the driver onto a running system: 108270346Sjfv cd ixl-x.x.x/src 109269201Sjfv make load 110269201Sjfv 111269201Sjfv5. To assign an IP address to the interface, enter the following: 112269201Sjfv ifconfig ixl<interface_num> <IP_address> 113269201Sjfv 114270772Sjfv6. Verify that the interface works. Enter the following, where <IP_address> 115270772Sjfv is the IP address for another machine on the same subnet as the interface 116270772Sjfv that is being tested: 117269201Sjfv 118269201Sjfv ping <IP_address> 119269201Sjfv 120269201Sjfv7. If you want the driver to load automatically when the system is booted: 121269201Sjfv 122270346Sjfv cd ixl-x.x.x/src 123269201Sjfv make 124269201Sjfv make install 125269201Sjfv 126269201Sjfv Edit /boot/loader.conf, and add the following line: 127270346Sjfv if_ixl_load="YES" 128269201Sjfv 129269201Sjfv Edit /etc/rc.conf, and create the appropriate 130269201Sjfv ifconfig_ixl<interface_num> entry: 131269201Sjfv 132269201Sjfv ifconfig_ixl<interface_num>="<ifconfig_settings>" 133269201Sjfv 134269201Sjfv Example usage: 135269201Sjfv 136269201Sjfv ifconfig_ixl0="inet 192.168.10.1 netmask 255.255.255.0" 137269201Sjfv 138269201Sjfv NOTE: For assistance, see the ifconfig man page. 139269201Sjfv 140269201Sjfv 141269201Sjfv 142269201SjfvConfiguration and Tuning 143269201Sjfv========================= 144269201Sjfv 145270772SjfvBoth drivers supports Transmit/Receive Checksum Offload for IPv4 and IPv6, 146269201SjfvTSO forIPv4 and IPv6, LRO, and Jumbo Frames on all 40 Gigabit adapters. 147269201Sjfv 148269201Sjfv Jumbo Frames 149269201Sjfv ------------ 150269201Sjfv To enable Jumbo Frames, use the ifconfig utility to increase 151269201Sjfv the MTU beyond 1500 bytes. 152269201Sjfv 153269201Sjfv - The Jumbo Frames setting on the switch must be set to at least 154269201Sjfv 22 byteslarger than that of the adapter. 155269201Sjfv 156269201Sjfv - The maximum MTU setting for Jumbo Frames is 9706. This value 157269201Sjfv coincides with the maximum jumbo frames size of 9728. 158269201Sjfv To modify the setting, enter the following: 159269201Sjfv 160269201Sjfv ifconfig ixl<interface_num> <hostname or IP address> mtu 9000 161269201Sjfv 162269201Sjfv - To confirm an interface's MTU value, use the ifconfig command. 163269201Sjfv To confirm the MTU used between two specific devices, use: 164269201Sjfv 165269201Sjfv route get <destination_IP_address> 166269201Sjfv 167269201Sjfv VLANs 168269201Sjfv ----- 169269201Sjfv To create a new VLAN pseudo-interface: 170269201Sjfv 171269201Sjfv ifconfig <vlan_name> create 172269201Sjfv 173269201Sjfv To associate the VLAN pseudo-interface with a physical interface 174269201Sjfv and assign a VLAN ID, IP address, and netmask: 175269201Sjfv 176269201Sjfv ifconfig <vlan_name> <ip_address> netmask <subnet_mask> vlan 177269201Sjfv <vlan_id> vlandev <physical_interface> 178269201Sjfv 179269201Sjfv Example: 180269201Sjfv 181269201Sjfv ifconfig vlan10 10.0.0.1 netmask 255.255.255.0 vlan 10 vlandev ixl0 182269201Sjfv 183269201Sjfv In this example, all packets will be marked on egress with 184269201Sjfv 802.1Q VLAN tags, specifying a VLAN ID of 10. 185269201Sjfv 186269201Sjfv To remove a VLAN pseudo-interface: 187269201Sjfv 188269201Sjfv ifconfig <vlan_name> destroy 189269201Sjfv 190269201Sjfv 191269201Sjfv Checksum Offload 192269201Sjfv ---------------- 193269201Sjfv 194269201Sjfv Checksum offloading supports IPv4 and IPv6 with TCP and UDP packets 195269201Sjfv and is supported for both transmit and receive. Checksum offloading 196269201Sjfv for transmit and recieve is enabled by default for both IPv4 and IPv6. 197269201Sjfv 198269201Sjfv Checksum offloading can be enabled or disabled using ifconfig. 199269201Sjfv Transmit and receive offloading for IPv4 and Ipv6 are enabled 200269201Sjfv and disabled seperately. 201269201Sjfv 202269201Sjfv NOTE: TSO requires Tx checksum, so when Tx checksum 203269201Sjfv is disabled, TSO will also be disabled. 204269201Sjfv 205269201Sjfv To enable Tx checksum offloading for ipv4: 206269201Sjfv 207269201Sjfv ifconfig ixl<interface_num> txcsum4 208269201Sjfv 209269201Sjfv To disable Tx checksum offloading for ipv4: 210269201Sjfv 211269201Sjfv ifconfig ixl<interface_num> -txcsum4 212269201Sjfv (NOTE: This will disable TSO4) 213269201Sjfv 214269201Sjfv To enable Rx checksum offloading for ipv6: 215269201Sjfv 216269201Sjfv ifconfig ixl<interface_num> rxcsum6 217269201Sjfv 218269201Sjfv To disable Rx checksum offloading for ipv6: 219269201Sjfv 220269201Sjfv ifconfig ixl<interface_num> -rxcsum6 221269201Sjfv (NOTE: This will disable TSO6) 222269201Sjfv 223269201Sjfv 224269201Sjfv To confirm the current settings: 225269201Sjfv 226269201Sjfv ifconfig ixl<interface_num> 227269201Sjfv 228269201Sjfv 229269201Sjfv TSO 230269201Sjfv --- 231269201Sjfv 232269201Sjfv TSO supports both IPv4 and IPv6 and is enabled by default. TSO can 233269201Sjfv be disabled and enabled using the ifconfig utility. 234269201Sjfv 235269201Sjfv NOTE: TSO requires Tx checksum, so when Tx checksum is 236269201Sjfv disabled, TSO will also be disabled. 237269201Sjfv 238269201Sjfv To disable TSO IPv4: 239269201Sjfv 240269201Sjfv ifconfig ixl<interface_num> -tso4 241269201Sjfv 242269201Sjfv To enable TSO IPv4: 243269201Sjfv 244269201Sjfv ifconfig ixl<interface_num> tso4 245269201Sjfv 246269201Sjfv To disable TSO IPv6: 247269201Sjfv 248269201Sjfv ifconfig ixl<interface_num> -tso6 249269201Sjfv 250269201Sjfv To enable TSO IPv6: 251269201Sjfv 252269201Sjfv ifconfig ixl<interface_num> tso6 253269201Sjfv 254269201Sjfv To disable BOTH TSO IPv4 and IPv6: 255269201Sjfv 256269201Sjfv ifconfig ixl<interface_num> -tso 257269201Sjfv 258269201Sjfv To enable BOTH TSO IPv4 and IPv6: 259269201Sjfv 260269201Sjfv ifconfig ixl<interface_num> tso 261269201Sjfv 262269201Sjfv 263269201Sjfv LRO 264269201Sjfv --- 265269201Sjfv 266269201Sjfv Large Receive Offload is enabled by default. It can be enabled 267269201Sjfv or disabled by using the ifconfig utility. 268269201Sjfv 269269201Sjfv NOTE: LRO should be disabled when forwarding packets. 270269201Sjfv 271269201Sjfv To disable LRO: 272269201Sjfv 273269201Sjfv ifconfig ixl<interface_num> -lro 274269201Sjfv 275269201Sjfv To enable LRO: 276269201Sjfv 277269201Sjfv ifconfig ixl<interface_num> lro 278269201Sjfv 279269201Sjfv 280270772SjfvFlow Control (IXL only) 281269201Sjfv------------ 282269201SjfvFlow control is disabled by default. To change flow control settings use sysctl. 283269201Sjfv 284269201SjfvTo enable flow control to Rx pause frames: 285269201Sjfv 286269201Sjfv sysctl dev.ixl.<interface_num>.fc=1 287269201Sjfv 288269201SjfvTo enable flow control to Tx pause frames: 289269201Sjfv 290269201Sjfv sysctl dev.ixl.<interface_num>.fc=2 291269201Sjfv 292269201SjfvTo enable flow control to Rx and Tx pause frames: 293269201Sjfv 294269201Sjfv sysctl dev.ixl.<interface_num>.fc=3 295269201Sjfv 296269201SjfvTo disable flow control: 297269201Sjfv 298269201Sjfv sysctl dev.ixl.<interface_num>.fc=0 299269201Sjfv 300269201Sjfv 301269201SjfvNOTE: You must have a flow control capable link partner. 302269201Sjfv 303270772SjfvNOTE: The VF driver does not have access to flow control, it must be 304270772Sjfv managed from the host side. 305269201Sjfv 306269201Sjfv 307269201Sjfv Important system configuration changes: 308269201Sjfv ======================================= 309269201Sjfv 310269201Sjfv-Change the file /etc/sysctl.conf, and add the line: 311269201Sjfv 312269201Sjfv hw.intr_storm_threshold: 0 (the default is 1000) 313269201Sjfv 314269201Sjfv-Best throughput results are seen with a large MTU; use 9706 if possible. 315269201Sjfv 316270772Sjfv-The default number of descriptors per ring is 1024, increasing this may 317270772Sjfvimprove performance depending on the use case. 318269201Sjfv 319270772Sjfv-The VF driver uses a relatively large buf ring, this was found to eliminate 320270772Sjfv UDP transmit errors, it is a tuneable, and if no UDP traffic is used it can 321270772Sjfv be reduced. It is memory used per queue. 322269201Sjfv 323270772Sjfv 324269201SjfvKnown Limitations 325269201Sjfv================= 326269201Sjfv 327269201SjfvNetwork Memory Buffer allocation 328269201Sjfv-------------------------------- 329270772Sjfv FreeBSD may have a low number of network memory buffers (mbufs) by default. 330270772SjfvIf your mbuf value is too low, it may cause the driver to fail to initialize 331270772Sjfvand/or cause the system to become unresponsive. You can check to see if the 332270772Sjfvsystem is mbuf-starved by running 'netstat -m'. Increase the number of mbufs 333270772Sjfvby editing the lines below in /etc/sysctl.conf: 334269201Sjfv 335269201Sjfv kern.ipc.nmbclusters 336269201Sjfv kern.ipc.nmbjumbop 337269201Sjfv kern.ipc.nmbjumbo9 338269201Sjfv kern.ipc.nmbjumbo16 339269201Sjfv kern.ipc.nmbufs 340269201Sjfv 341270772SjfvThe amount of memory that you allocate is system specific, and may 342270772Sjfvrequire some trial and error. 343269201Sjfv 344270772SjfvAlso, increasing the follwing in /etc/sysctl.conf could help increase 345270772Sjfvnetwork performance: 346269201Sjfv 347269201Sjfv kern.ipc.maxsockbuf 348269201Sjfv net.inet.tcp.sendspace 349269201Sjfv net.inet.tcp.recvspace 350269201Sjfv net.inet.udp.maxdgram 351269201Sjfv net.inet.udp.recvspace 352269201Sjfv 353269201Sjfv 354269201SjfvUDP Stress Test Dropped Packet Issue 355269201Sjfv------------------------------------ 356270772SjfvUnder small packet UDP stress test with the ixl driver, the FreeBSD system 357270772Sjfvmay drop UDP packets due to the fullness of socket buffers. You may want to 358270772Sjfvchange the driver's Flow Control variables to the minimum value for 359270772Sjfvcontrolling packet reception. 360269201Sjfv 361269201Sjfv 362269201SjfvDisable LRO when routing/bridging 363269201Sjfv--------------------------------- 364269201SjfvLRO must be turned off when forwarding traffic. 365269201Sjfv 366269201Sjfv 367269201SjfvLower than expected performance 368269201Sjfv------------------------------- 369270772SjfvSome PCIe x8 slots are actually configured as x4 slots. These slots have 370270772Sjfvinsufficient bandwidth for full line rate with dual port and quad port 371270772Sjfvdevices. 372269201Sjfv 373270772SjfvIn addition, if you put a PCIe Generation 3-capable adapter into a PCIe 374270772SjfvGeneration 2 slot, you cannot get full bandwidth. The driver detects this 375270772Sjfvsituation and writes the following message in the system log: 376269201Sjfv 377270772Sjfv "PCI-Express bandwidth available for this card is not sufficient for 378270772Sjfv optimal performance. For optimal performance a x8 PCI-Express slot 379270772Sjfv is required." 380269201Sjfv 381270772SjfvIf this error occurs, moving your adapter to a true PCIe Generation 3 x8 382270772Sjfvslot will resolve the issue. 383269201Sjfv 384270772Sjfv 385269201SjfvSupport 386269201Sjfv======= 387269201Sjfv 388269201SjfvFor general information and support, go to the Intel support website at: 389269201Sjfv 390269201Sjfv http://support.intel.com 391269201Sjfv 392270772SjfvIf an issue is identified with the released source code on the supported kernel 393270772Sjfvwith a supported adapter, email the specific information related to the issue 394270772Sjfvto freebsdnic@mailbox.intel.com. 395269201Sjfv 396269201Sjfv 397269201SjfvLicense 398269201Sjfv======= 399269201Sjfv 400270772SjfvThis software program is released under the terms of a license agreement 401270772Sjfvbetween you ('Licensee') and Intel. Do not use or load this software or any 402270772Sjfvassociated materials (collectively, the 'Software') until you have carefully 403270772Sjfvread the full terms and conditions of the LICENSE located in this software 404270772Sjfvpackage. By loadingor using the Software, you agree to the terms of this 405270772SjfvAgreement. If you do not agree with the terms of this Agreement, do not 406270772Sjfvinstall or use the Software. 407269201Sjfv 408269201Sjfv* Other names and brands may be claimed as the property of others. 409269201Sjfv 410269201Sjfv 411