1 ixl FreeBSD* Base Driver and ixlv VF Driver for the
2 Intel XL710 Ethernet Controller Family
3
4/*$FreeBSD$*/
5================================================================
6
7August 26, 2014
8
9
10Contents
11========
12
13- Overview
14- Supported Adapters
15- The VF Driver
16- Building and Installation
17- Additional Configurations
18- Known Limitations
19
20
21Overview
22========
23
24This file describes the IXL FreeBSD* Base driver and the IXLV VF Driver
25for the XL710 Ethernet Family of Adapters. The Driver has been developed
26for use with FreeBSD 10.0 or later, but should be compatible with any
27supported release.
28
29For questions related to hardware requirements, refer to the documentation
30supplied with your Intel XL710 adapter. All hardware requirements listed
31apply for use with FreeBSD.
32
33
34Supported Adapters
35==================
36
37The drivers in this release are compatible with XL710 and X710-based
38Intel Ethernet Network Connections.
39
40
41SFP+ Devices with Pluggable Optics
42----------------------------------
43
44SR Modules
45----------
46 Intel DUAL RATE 1G/10G SFP+ SR (bailed) FTLX8571D3BCV-IT
47 Intel DUAL RATE 1G/10G SFP+ SR (bailed) AFBR-703SDZ-IN2
48
49LR Modules
50----------
51 Intel DUAL RATE 1G/10G SFP+ LR (bailed) FTLX1471D3BCV-IT
52 Intel DUAL RATE 1G/10G SFP+ LR (bailed) AFCT-701SDZ-IN2
53
54QSFP+ Modules
55-------------
56 Intel TRIPLE RATE 1G/10G/40G QSFP+ SR (bailed) E40GQSFPSR
57 Intel TRIPLE RATE 1G/10G/40G QSFP+ LR (bailed) E40GQSFPLR
58 QSFP+ 1G speed is not supported on XL710 based devices.
59
60X710/XL710 Based SFP+ adapters support all passive and active limiting direct
61attach cables that comply with SFF-8431 v4.1 and SFF-8472 v10.4 specifications.
62
63The VF Driver
64==================
65The VF driver is normally used in a virtualized environment where a host
66driver manages SRIOV, and provides a VF device to the guest. With this
67first release the only host environment tested was using Linux QEMU/KVM.
68Support is planned for Xen and VMWare hosts at a later time.
69
70In the FreeBSD guest the IXLV driver would be loaded and will function
71using the VF device assigned to it.
72
73The VF driver provides most of the same functionality as the CORE driver,
74but is actually a slave to the Host, access to many controls are actually
75accomplished by a request to the Host via what is called the "Admin queue".
76These are startup and initialization events however, once in operation
77the device is self-contained and should achieve near native performance.
78
79Some notable limitations of the VF environment: for security reasons
80the driver is never permitted to be promiscuous, therefore a tcpdump
81will not behave the same with the interface. Second, media info is not
82available from the PF, so it will always appear as auto.
83
84Tarball Building and Installation
85=========================
86
87NOTE: You must have kernel sources installed to compile the driver tarball.
88
89These instructions assume a standalone driver tarball, building the driver
90already in the kernel source is simply a matter of adding the device entry
91to the kernel config file, or building in the ixl or ixlv module directory.
92
93In the instructions below, x.x.x is the driver version
94as indicated in the name of the driver tarball. The example is
95for ixl, the same procedure applies for ixlv.
96
971. Move the base driver tar file to the directory of your choice.
98 For example, use /home/username/ixl or /usr/local/src/ixl.
99
1002. Untar/unzip the archive:
101 tar xfz ixl-x.x.x.tar.gz
102
1033. To install man page:
104 cd ixl-x.x.x
105 gzip -c ixl.4 > /usr/share/man/man4/ixl.4.gz
106
1074. To load the driver onto a running system:
108 cd ixl-x.x.x/src
109 make load
110
1115. To assign an IP address to the interface, enter the following:
112 ifconfig ixl<interface_num> <IP_address>
113
1146. Verify that the interface works. Enter the following, where <IP_address>
115 is the IP address for another machine on the same subnet as the interface
116 that is being tested:
117
118 ping <IP_address>
119
1207. If you want the driver to load automatically when the system is booted:
121
122 cd ixl-x.x.x/src
123 make
124 make install
125
126 Edit /boot/loader.conf, and add the following line:
127 if_ixl_load="YES"
128
129 Edit /etc/rc.conf, and create the appropriate
130 ifconfig_ixl<interface_num> entry:
131
132 ifconfig_ixl<interface_num>="<ifconfig_settings>"
133
134 Example usage:
135
136 ifconfig_ixl0="inet 192.168.10.1 netmask 255.255.255.0"
137
138 NOTE: For assistance, see the ifconfig man page.
139
140
141
142Configuration and Tuning
143=========================
144
145Both drivers supports Transmit/Receive Checksum Offload for IPv4 and IPv6,
146TSO forIPv4 and IPv6, LRO, and Jumbo Frames on all 40 Gigabit adapters.
147
148 Jumbo Frames
149 ------------
150 To enable Jumbo Frames, use the ifconfig utility to increase
151 the MTU beyond 1500 bytes.
152
153 - The Jumbo Frames setting on the switch must be set to at least
154 22 byteslarger than that of the adapter.
155
156 - The maximum MTU setting for Jumbo Frames is 9706. This value
157 coincides with the maximum jumbo frames size of 9728.
158 To modify the setting, enter the following:
159
160 ifconfig ixl<interface_num> <hostname or IP address> mtu 9000
161
162 - To confirm an interface's MTU value, use the ifconfig command.
163 To confirm the MTU used between two specific devices, use:
164
165 route get <destination_IP_address>
166
167 VLANs
168 -----
169 To create a new VLAN pseudo-interface:
170
171 ifconfig <vlan_name> create
172
173 To associate the VLAN pseudo-interface with a physical interface
174 and assign a VLAN ID, IP address, and netmask:
175
176 ifconfig <vlan_name> <ip_address> netmask <subnet_mask> vlan
177 <vlan_id> vlandev <physical_interface>
178
179 Example:
180
181 ifconfig vlan10 10.0.0.1 netmask 255.255.255.0 vlan 10 vlandev ixl0
182
183 In this example, all packets will be marked on egress with
184 802.1Q VLAN tags, specifying a VLAN ID of 10.
185
186 To remove a VLAN pseudo-interface:
187
188 ifconfig <vlan_name> destroy
189
190
191 Checksum Offload
192 ----------------
193
194 Checksum offloading supports IPv4 and IPv6 with TCP and UDP packets
195 and is supported for both transmit and receive. Checksum offloading
196 for transmit and recieve is enabled by default for both IPv4 and IPv6.
197
198 Checksum offloading can be enabled or disabled using ifconfig.
199 Transmit and receive offloading for IPv4 and Ipv6 are enabled
200 and disabled seperately.
201
202 NOTE: TSO requires Tx checksum, so when Tx checksum
203 is disabled, TSO will also be disabled.
204
205 To enable Tx checksum offloading for ipv4:
206
207 ifconfig ixl<interface_num> txcsum4
208
209 To disable Tx checksum offloading for ipv4:
210
211 ifconfig ixl<interface_num> -txcsum4
212 (NOTE: This will disable TSO4)
213
214 To enable Rx checksum offloading for ipv6:
215
216 ifconfig ixl<interface_num> rxcsum6
217
218 To disable Rx checksum offloading for ipv6:
219
220 ifconfig ixl<interface_num> -rxcsum6
221 (NOTE: This will disable TSO6)
222
223
224 To confirm the current settings:
225
226 ifconfig ixl<interface_num>
227
228
229 TSO
230 ---
231
232 TSO supports both IPv4 and IPv6 and is enabled by default. TSO can
233 be disabled and enabled using the ifconfig utility.
234
235 NOTE: TSO requires Tx checksum, so when Tx checksum is
236 disabled, TSO will also be disabled.
237
238 To disable TSO IPv4:
239
240 ifconfig ixl<interface_num> -tso4
241
242 To enable TSO IPv4:
243
244 ifconfig ixl<interface_num> tso4
245
246 To disable TSO IPv6:
247
248 ifconfig ixl<interface_num> -tso6
249
250 To enable TSO IPv6:
251
252 ifconfig ixl<interface_num> tso6
253
254 To disable BOTH TSO IPv4 and IPv6:
255
256 ifconfig ixl<interface_num> -tso
257
258 To enable BOTH TSO IPv4 and IPv6:
259
260 ifconfig ixl<interface_num> tso
261
262
263 LRO
264 ---
265
266 Large Receive Offload is enabled by default. It can be enabled
267 or disabled by using the ifconfig utility.
268
269 NOTE: LRO should be disabled when forwarding packets.
270
271 To disable LRO:
272
273 ifconfig ixl<interface_num> -lro
274
275 To enable LRO:
276
277 ifconfig ixl<interface_num> lro
278
279
280Flow Control (IXL only)
281------------
282Flow control is disabled by default. To change flow control settings use sysctl.
283
284To enable flow control to Rx pause frames:
285
286 sysctl dev.ixl.<interface_num>.fc=1
287
288To enable flow control to Tx pause frames:
289
290 sysctl dev.ixl.<interface_num>.fc=2
291
292To enable flow control to Rx and Tx pause frames:
293
294 sysctl dev.ixl.<interface_num>.fc=3
295
296To disable flow control:
297
298 sysctl dev.ixl.<interface_num>.fc=0
299
300
301NOTE: You must have a flow control capable link partner.
302
303NOTE: The VF driver does not have access to flow control, it must be
304 managed from the host side.
305
306
307 Important system configuration changes:
308 =======================================
309
310-Change the file /etc/sysctl.conf, and add the line:
311
312 hw.intr_storm_threshold: 0 (the default is 1000)
313
314-Best throughput results are seen with a large MTU; use 9706 if possible.
315
316-The default number of descriptors per ring is 1024, increasing this may
317improve performance depending on the use case.
318
319-The VF driver uses a relatively large buf ring, this was found to eliminate
320 UDP transmit errors, it is a tuneable, and if no UDP traffic is used it can
321 be reduced. It is memory used per queue.
322
323
324Known Limitations
325=================
326
327Network Memory Buffer allocation
328--------------------------------
329 FreeBSD may have a low number of network memory buffers (mbufs) by default.
330If your mbuf value is too low, it may cause the driver to fail to initialize
331and/or cause the system to become unresponsive. You can check to see if the
332system is mbuf-starved by running 'netstat -m'. Increase the number of mbufs
333by editing the lines below in /etc/sysctl.conf:
334
335 kern.ipc.nmbclusters
336 kern.ipc.nmbjumbop
337 kern.ipc.nmbjumbo9
338 kern.ipc.nmbjumbo16
339 kern.ipc.nmbufs
340
341The amount of memory that you allocate is system specific, and may
342require some trial and error.
343
344Also, increasing the follwing in /etc/sysctl.conf could help increase
345network performance:
346
347 kern.ipc.maxsockbuf
348 net.inet.tcp.sendspace
349 net.inet.tcp.recvspace
350 net.inet.udp.maxdgram
351 net.inet.udp.recvspace
352
353
354UDP Stress Test Dropped Packet Issue
355------------------------------------
356Under small packet UDP stress test with the ixl driver, the FreeBSD system
357may drop UDP packets due to the fullness of socket buffers. You may want to
358change the driver's Flow Control variables to the minimum value for
359controlling packet reception.
360
361
362Disable LRO when routing/bridging
363---------------------------------
364LRO must be turned off when forwarding traffic.
365
366
367Lower than expected performance
368-------------------------------
369Some PCIe x8 slots are actually configured as x4 slots. These slots have
370insufficient bandwidth for full line rate with dual port and quad port
371devices.
372
373In addition, if you put a PCIe Generation 3-capable adapter into a PCIe
374Generation 2 slot, you cannot get full bandwidth. The driver detects this
375situation and writes the following message in the system log:
376
377 "PCI-Express bandwidth available for this card is not sufficient for
378 optimal performance. For optimal performance a x8 PCI-Express slot
379 is required."
380
381If this error occurs, moving your adapter to a true PCIe Generation 3 x8
382slot will resolve the issue.
383
384
385Support
386=======
387
388For general information and support, go to the Intel support website at:
389
390 http://support.intel.com
391
392If an issue is identified with the released source code on the supported kernel
393with a supported adapter, email the specific information related to the issue
394to freebsdnic@mailbox.intel.com.
395
396
397License
398=======
399
400This software program is released under the terms of a license agreement
401between you ('Licensee') and Intel. Do not use or load this software or any
402associated materials (collectively, the 'Software') until you have carefully
403read the full terms and conditions of the LICENSE located in this software
404package. By loadingor using the Software, you agree to the terms of this
405Agreement. If you do not agree with the terms of this Agreement, do not
406install or use the Software.
407
408* Other names and brands may be claimed as the property of others.
409
410
411