1 ixl FreeBSD* Base Driver and ixlv VF Driver for the 2 Intel XL710 Ethernet Controller Family 3 4/*$FreeBSD: releng/11.0/sys/dev/ixl/README 270772 2014-08-28 21:45:07Z jfv $*/ 5================================================================ 6 7August 26, 2014 8 9 10Contents 11======== 12 13- Overview 14- Supported Adapters 15- The VF Driver 16- Building and Installation 17- Additional Configurations 18- Known Limitations 19 20 21Overview 22======== 23 24This file describes the IXL FreeBSD* Base driver and the IXLV VF Driver 25for the XL710 Ethernet Family of Adapters. The Driver has been developed 26for use with FreeBSD 10.0 or later, but should be compatible with any 27supported release. 28 29For questions related to hardware requirements, refer to the documentation 30supplied with your Intel XL710 adapter. All hardware requirements listed 31apply for use with FreeBSD. 32 33 34Supported Adapters 35================== 36 37The drivers in this release are compatible with XL710 and X710-based 38Intel Ethernet Network Connections. 39 40 41SFP+ Devices with Pluggable Optics 42---------------------------------- 43 44SR Modules 45---------- 46 Intel DUAL RATE 1G/10G SFP+ SR (bailed) FTLX8571D3BCV-IT 47 Intel DUAL RATE 1G/10G SFP+ SR (bailed) AFBR-703SDZ-IN2 48 49LR Modules 50---------- 51 Intel DUAL RATE 1G/10G SFP+ LR (bailed) FTLX1471D3BCV-IT 52 Intel DUAL RATE 1G/10G SFP+ LR (bailed) AFCT-701SDZ-IN2 53 54QSFP+ Modules 55------------- 56 Intel TRIPLE RATE 1G/10G/40G QSFP+ SR (bailed) E40GQSFPSR 57 Intel TRIPLE RATE 1G/10G/40G QSFP+ LR (bailed) E40GQSFPLR 58 QSFP+ 1G speed is not supported on XL710 based devices. 59 60X710/XL710 Based SFP+ adapters support all passive and active limiting direct 61attach cables that comply with SFF-8431 v4.1 and SFF-8472 v10.4 specifications. 62 63The VF Driver 64================== 65The VF driver is normally used in a virtualized environment where a host 66driver manages SRIOV, and provides a VF device to the guest. With this 67first release the only host environment tested was using Linux QEMU/KVM. 68Support is planned for Xen and VMWare hosts at a later time. 69 70In the FreeBSD guest the IXLV driver would be loaded and will function 71using the VF device assigned to it. 72 73The VF driver provides most of the same functionality as the CORE driver, 74but is actually a slave to the Host, access to many controls are actually 75accomplished by a request to the Host via what is called the "Admin queue". 76These are startup and initialization events however, once in operation 77the device is self-contained and should achieve near native performance. 78 79Some notable limitations of the VF environment: for security reasons 80the driver is never permitted to be promiscuous, therefore a tcpdump 81will not behave the same with the interface. Second, media info is not 82available from the PF, so it will always appear as auto. 83 84Tarball Building and Installation 85========================= 86 87NOTE: You must have kernel sources installed to compile the driver tarball. 88 89These instructions assume a standalone driver tarball, building the driver 90already in the kernel source is simply a matter of adding the device entry 91to the kernel config file, or building in the ixl or ixlv module directory. 92 93In the instructions below, x.x.x is the driver version 94as indicated in the name of the driver tarball. The example is 95for ixl, the same procedure applies for ixlv. 96 971. Move the base driver tar file to the directory of your choice. 98 For example, use /home/username/ixl or /usr/local/src/ixl. 99 1002. Untar/unzip the archive: 101 tar xfz ixl-x.x.x.tar.gz 102 1033. To install man page: 104 cd ixl-x.x.x 105 gzip -c ixl.4 > /usr/share/man/man4/ixl.4.gz 106 1074. To load the driver onto a running system: 108 cd ixl-x.x.x/src 109 make load 110 1115. To assign an IP address to the interface, enter the following: 112 ifconfig ixl<interface_num> <IP_address> 113 1146. Verify that the interface works. Enter the following, where <IP_address> 115 is the IP address for another machine on the same subnet as the interface 116 that is being tested: 117 118 ping <IP_address> 119 1207. If you want the driver to load automatically when the system is booted: 121 122 cd ixl-x.x.x/src 123 make 124 make install 125 126 Edit /boot/loader.conf, and add the following line: 127 if_ixl_load="YES" 128 129 Edit /etc/rc.conf, and create the appropriate 130 ifconfig_ixl<interface_num> entry: 131 132 ifconfig_ixl<interface_num>="<ifconfig_settings>" 133 134 Example usage: 135 136 ifconfig_ixl0="inet 192.168.10.1 netmask 255.255.255.0" 137 138 NOTE: For assistance, see the ifconfig man page. 139 140 141 142Configuration and Tuning 143========================= 144 145Both drivers supports Transmit/Receive Checksum Offload for IPv4 and IPv6, 146TSO forIPv4 and IPv6, LRO, and Jumbo Frames on all 40 Gigabit adapters. 147 148 Jumbo Frames 149 ------------ 150 To enable Jumbo Frames, use the ifconfig utility to increase 151 the MTU beyond 1500 bytes. 152 153 - The Jumbo Frames setting on the switch must be set to at least 154 22 byteslarger than that of the adapter. 155 156 - The maximum MTU setting for Jumbo Frames is 9706. This value 157 coincides with the maximum jumbo frames size of 9728. 158 To modify the setting, enter the following: 159 160 ifconfig ixl<interface_num> <hostname or IP address> mtu 9000 161 162 - To confirm an interface's MTU value, use the ifconfig command. 163 To confirm the MTU used between two specific devices, use: 164 165 route get <destination_IP_address> 166 167 VLANs 168 ----- 169 To create a new VLAN pseudo-interface: 170 171 ifconfig <vlan_name> create 172 173 To associate the VLAN pseudo-interface with a physical interface 174 and assign a VLAN ID, IP address, and netmask: 175 176 ifconfig <vlan_name> <ip_address> netmask <subnet_mask> vlan 177 <vlan_id> vlandev <physical_interface> 178 179 Example: 180 181 ifconfig vlan10 10.0.0.1 netmask 255.255.255.0 vlan 10 vlandev ixl0 182 183 In this example, all packets will be marked on egress with 184 802.1Q VLAN tags, specifying a VLAN ID of 10. 185 186 To remove a VLAN pseudo-interface: 187 188 ifconfig <vlan_name> destroy 189 190 191 Checksum Offload 192 ---------------- 193 194 Checksum offloading supports IPv4 and IPv6 with TCP and UDP packets 195 and is supported for both transmit and receive. Checksum offloading 196 for transmit and recieve is enabled by default for both IPv4 and IPv6. 197 198 Checksum offloading can be enabled or disabled using ifconfig. 199 Transmit and receive offloading for IPv4 and Ipv6 are enabled 200 and disabled seperately. 201 202 NOTE: TSO requires Tx checksum, so when Tx checksum 203 is disabled, TSO will also be disabled. 204 205 To enable Tx checksum offloading for ipv4: 206 207 ifconfig ixl<interface_num> txcsum4 208 209 To disable Tx checksum offloading for ipv4: 210 211 ifconfig ixl<interface_num> -txcsum4 212 (NOTE: This will disable TSO4) 213 214 To enable Rx checksum offloading for ipv6: 215 216 ifconfig ixl<interface_num> rxcsum6 217 218 To disable Rx checksum offloading for ipv6: 219 220 ifconfig ixl<interface_num> -rxcsum6 221 (NOTE: This will disable TSO6) 222 223 224 To confirm the current settings: 225 226 ifconfig ixl<interface_num> 227 228 229 TSO 230 --- 231 232 TSO supports both IPv4 and IPv6 and is enabled by default. TSO can 233 be disabled and enabled using the ifconfig utility. 234 235 NOTE: TSO requires Tx checksum, so when Tx checksum is 236 disabled, TSO will also be disabled. 237 238 To disable TSO IPv4: 239 240 ifconfig ixl<interface_num> -tso4 241 242 To enable TSO IPv4: 243 244 ifconfig ixl<interface_num> tso4 245 246 To disable TSO IPv6: 247 248 ifconfig ixl<interface_num> -tso6 249 250 To enable TSO IPv6: 251 252 ifconfig ixl<interface_num> tso6 253 254 To disable BOTH TSO IPv4 and IPv6: 255 256 ifconfig ixl<interface_num> -tso 257 258 To enable BOTH TSO IPv4 and IPv6: 259 260 ifconfig ixl<interface_num> tso 261 262 263 LRO 264 --- 265 266 Large Receive Offload is enabled by default. It can be enabled 267 or disabled by using the ifconfig utility. 268 269 NOTE: LRO should be disabled when forwarding packets. 270 271 To disable LRO: 272 273 ifconfig ixl<interface_num> -lro 274 275 To enable LRO: 276 277 ifconfig ixl<interface_num> lro 278 279 280Flow Control (IXL only) 281------------ 282Flow control is disabled by default. To change flow control settings use sysctl. 283 284To enable flow control to Rx pause frames: 285 286 sysctl dev.ixl.<interface_num>.fc=1 287 288To enable flow control to Tx pause frames: 289 290 sysctl dev.ixl.<interface_num>.fc=2 291 292To enable flow control to Rx and Tx pause frames: 293 294 sysctl dev.ixl.<interface_num>.fc=3 295 296To disable flow control: 297 298 sysctl dev.ixl.<interface_num>.fc=0 299 300 301NOTE: You must have a flow control capable link partner. 302 303NOTE: The VF driver does not have access to flow control, it must be 304 managed from the host side. 305 306 307 Important system configuration changes: 308 ======================================= 309 310-Change the file /etc/sysctl.conf, and add the line: 311 312 hw.intr_storm_threshold: 0 (the default is 1000) 313 314-Best throughput results are seen with a large MTU; use 9706 if possible. 315 316-The default number of descriptors per ring is 1024, increasing this may 317improve performance depending on the use case. 318 319-The VF driver uses a relatively large buf ring, this was found to eliminate 320 UDP transmit errors, it is a tuneable, and if no UDP traffic is used it can 321 be reduced. It is memory used per queue. 322 323 324Known Limitations 325================= 326 327Network Memory Buffer allocation 328-------------------------------- 329 FreeBSD may have a low number of network memory buffers (mbufs) by default. 330If your mbuf value is too low, it may cause the driver to fail to initialize 331and/or cause the system to become unresponsive. You can check to see if the 332system is mbuf-starved by running 'netstat -m'. Increase the number of mbufs 333by editing the lines below in /etc/sysctl.conf: 334 335 kern.ipc.nmbclusters 336 kern.ipc.nmbjumbop 337 kern.ipc.nmbjumbo9 338 kern.ipc.nmbjumbo16 339 kern.ipc.nmbufs 340 341The amount of memory that you allocate is system specific, and may 342require some trial and error. 343 344Also, increasing the follwing in /etc/sysctl.conf could help increase 345network performance: 346 347 kern.ipc.maxsockbuf 348 net.inet.tcp.sendspace 349 net.inet.tcp.recvspace 350 net.inet.udp.maxdgram 351 net.inet.udp.recvspace 352 353 354UDP Stress Test Dropped Packet Issue 355------------------------------------ 356Under small packet UDP stress test with the ixl driver, the FreeBSD system 357may drop UDP packets due to the fullness of socket buffers. You may want to 358change the driver's Flow Control variables to the minimum value for 359controlling packet reception. 360 361 362Disable LRO when routing/bridging 363--------------------------------- 364LRO must be turned off when forwarding traffic. 365 366 367Lower than expected performance 368------------------------------- 369Some PCIe x8 slots are actually configured as x4 slots. These slots have 370insufficient bandwidth for full line rate with dual port and quad port 371devices. 372 373In addition, if you put a PCIe Generation 3-capable adapter into a PCIe 374Generation 2 slot, you cannot get full bandwidth. The driver detects this 375situation and writes the following message in the system log: 376 377 "PCI-Express bandwidth available for this card is not sufficient for 378 optimal performance. For optimal performance a x8 PCI-Express slot 379 is required." 380 381If this error occurs, moving your adapter to a true PCIe Generation 3 x8 382slot will resolve the issue. 383 384 385Support 386======= 387 388For general information and support, go to the Intel support website at: 389 390 http://support.intel.com 391 392If an issue is identified with the released source code on the supported kernel 393with a supported adapter, email the specific information related to the issue 394to freebsdnic@mailbox.intel.com. 395 396 397License 398======= 399 400This software program is released under the terms of a license agreement 401between you ('Licensee') and Intel. Do not use or load this software or any 402associated materials (collectively, the 'Software') until you have carefully 403read the full terms and conditions of the LICENSE located in this software 404package. By loadingor using the Software, you agree to the terms of this 405Agreement. If you do not agree with the terms of this Agreement, do not 406install or use the Software. 407 408* Other names and brands may be claimed as the property of others. 409 410 411