1$FreeBSD: releng/11.0/sys/dev/vxge/README 221167 2011-04-28 14:33:15Z gnn $ 2''_Readme for FreeBSD X3100 Series 10GbE PCIe I/O Virtualized Server Adapter Drivers_''' 3 4=== Introduction === 5FreeBSD Driver for X3100 10GbE Server/Storage adapters 6* Drivers support all X3100 10GbE adapters with FreeBSD version 7.x, 8.x and 9.x 7* Supports both i386 and amd64 architectures 8* Features: Jumbo frames (up to 9600), 9 LRO (Large Receive Offload), 10 TSO (TCP segmentation offload), 11 RTH (Receive Traffic Hash). 12 Also, Extended Message Signaled Interrupts (MSI-X). 13 14 15''Features'' 16 17 a. Jumbo frames: 18 X3110 and X3120 supports MTU up to 9600 bytes, modifiable using ifconfig command. 19 20 b. LRO (Large Receive Offload): 21 LRO can be enabled/disabled before loading driver. 22 Set lro_enable in vxge.conf to 1 before loading driver. 23 24 c. TSO (TCP Segmentation Offload) 25 TSO can be enabled/disabled before loading driver. 26 Set tso_enable in vxge.conf to 1 before loading driver. 27 28 d. RTH (Receive Traffic Hash) 29 Receive side steering for better scaling. 30 Set rth_enable in vxge.conf to 1 before loading driver. 31 32 e. MSI-X 33 Can be enabled on platforms which support it, resulting in noticeable 34 performance improvement. 35 36 f. Multi-VPaths 37 Up to 17 hardware based transmit and receive data channels, with 38 multiple steering options. 39 40 41''X3100 & Driver configuration: vxge.conf'' 42 43The vxge.conf contains following attributes. 44 45 46''msix_enable'' 47 48Enable MSI (Message Signaled Interrupts) feature in driver. 490 - INTA 501 - MSI-X 51Default: 1 52 53 54''rth_enable'' 55 56Enables Receive side steering for better scaling (RTH - Receive Traffic Hash) 57Range: 0 - 1 58Default: 1 59 60 61''lro_enable'' 62 63Enables LRO (Large Receive Offload) feature in driver. 64Range: 0 - 1 65Default: 1 66 67 68''tso_enable'' 69 70Enables TSO (TCP Segmentaton Offload) feature in driver. 71Range: 0 - 1 72Default: 1 73 74 75''no_of_vpath'' 76 77Specifies maximum VPATH(s) configured for each device function. 78Valid range: 1-17 79Default: Optimized by driver 80 81 82''func_mode'' 83 84Change PCI function mode 85 0 - SF1_VP17 (1 function with 17 VPATHs) 86 1 - MF8_VP2 (8 functions with 2 VPATHs per function) 87 8 - MF2_VP8 (2 functions, 8 Paths/Function) 88 9 - MF4_VP4 (4 Functions, 4 Paths/Function) 89 11 - MF8P_VP2 (8 functions with 2 VPATHS per function required for DirectIO) 90 Default: -1 91 92 93''port_mode'' 94 95Change the default dual port mode 96 2 - Active Passive 97 3 - Single Port 98 4 - Dual Port 99 100 101''l2_switch'' 102 103Turn on/off the inter function traffic through l2 switch 104 0 - Disallow inter function traffic 105 1 - Allow inter function traffic 106 Default: -1 107 108 109''bandwidth_0 - bandwidth_7'' 110 111 Desired max receive/transmit bandwidth,in Mbps for function 0 to function 7 112 Minimum value is 100 Mbps, for 1 Gbps specify a value of 1024. 113 114 115''priority_0 - priority_7'' 116 117 Desired receive/transmit priority for function 0 to function 7 118 119 120''intr_coalesce'' 121 122 Adaptive interrupt coalescing 123 0 - Disable 124 1 - Enable 125 126 127''Low Latency'' 128 129 0 - Disable 130 1 - Enable 131 132=== Installation Instructions === 133 134''Identifying the Adapter'' 135 136 The X3100 adapter is identified by the board ID number on the adapter. 137 138 Look for a label that has a barcode and a number, for example, 139 SXT0425072. The factory-burned MAC address (hardware address) 140 shows up on the board above the serial number, 141 (similar to 000CFC000449 -- 00:0C:FC:00:04:49). 142 143 144''Kernel Driver Source Package'' 145 146This package contains kernel_update.sh script which is to be used to copy driver sources to kernel path. 147It creates vxge folder with source code in /usr/src/sys/dev and Makefile in /usr/src/sys/modules. 148 149Loadable parameters can be changed by putting below lines in /boot/device.hints and set values as desired. 150 hint.vxge.0.msix_enable="1" 151 hint.vxge.0.rth_enable="1" 152 hint.vxge.0.lro_enable="1" 153 hint.vxge.0.tso_enable="1" 154 hint.vxge.0.tx_steering="1" 155 hint.vxge.0.no_of_vpath="-1" 156 hint.vxge.0.func_mode="-1" 157 hint.vxge.0.port_mode="-1" 158 hint.vxge.0.fw_upgrade="1" 159 hint.vxge.0.bandwidth_0="-1" 160 hint.vxge.0.bandwidth_1="-1" 161 hint.vxge.0.bandwidth_2="-1" 162 hint.vxge.0.bandwidth_3="-1" 163 hint.vxge.0.bandwidth_4="-1" 164 hint.vxge.0.bandwidth_5="-1" 165 hint.vxge.0.bandwidth_6="-1" 166 hint.vxge.0.bandwidth_7="-1" 167 hint.vxge.0.priority_0="-1" 168 hint.vxge.0.priority_1="-1" 169 hint.vxge.0.priority_2="-1" 170 hint.vxge.0.priority_3="-1" 171 hint.vxge.0.priority_4="-1" 172 hint.vxge.0.priority_5="-1" 173 hint.vxge.0.priority_6="-1" 174 hint.vxge.0.priority_7="-1" 175 hint.vxge.0.intr_coalesce="0" 176 hint.vxge.0.low_latency="0" 177 178 e.g., set hint.vxge.0.msix_enable to 0 to load driver in INTA mode. 179 Then reboot the system to add loadable parameters to kenv. 180 181 182''Standalone Driver Source Package'' 183 184vxge (FreeBSD package) 185 This directory contains FreeBSD driver sources for X3100 device(s), 186 Makefile, and X3100 Hardware Abstraction headers and sources 187 (vxgehal and include folders) 188 189vxgehal 190 This directory contains the X3100 HAL sources. 191 The driver uses HAL to perform operations on the X3100 hardware. 192 193include 194 The include subdirectory contains HAL header files. 195 196Source code for vxge-manage tool (used to get statistics, pciconfig and 197register dump) are included in the freebsd directory. 198 199 200''Building the driver'' 201 202 The driver is distributed in the source form. Driver and installation 203 utility executables need to be built for the target platform. 204 205 In the directory containing Makefile for building Exar driver for FreeBSD, 206 #make clean 207 #make 208 209 Please unload previously installed Exar drivers before proceeding with following steps. 210 #make uninstall 211 212 213''Loading the driver'' 214 215 Use "kldload" to load driver module vxge.ko. 216 #kldload ./vxge.ko 217 218 Run "kldstat" and find an entry for vxge kernel module to ensure driver installation 219 was successful. 220 #kldstat | grep vxge 221 3 1 0xc22cc000 26000 vxge.ko 222 223 224''Enabling interface and assigning IP address'' 225 226 #ifconfig <INTERFACE> <IP_ADDRESS> up 227 <INTERFACE> will be similar to vxge0, vxge1 etc. and can be 228 found by executing "ifconfig -a". 229 Neterion adapters typically have MAC addresses starting with 230 "00:0C:FC" or "00:11:25". 231 232 Example: 233 #ifconfig vxge0 10.2.2.40 up 234 Enables vxge0 interface and assigns to it the IP address 10.2.2.40. 235 236 vxge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 237 options=53b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4,LRO> 238 ether 00:0c:fc:00:da:47 239 inet6 fe80::20c:fcff:fe00:da47%vxge0 prefixlen 64 scopeid 0x3 240 inet 10.2.2.40 netmask 0xff000000 broadcast 17.255.255.255 241 media: Ethernet autoselect (10Gbase-SR <full-duplex>) 242 status: active 243 244 245''Disabling the interface'' 246 247 #ifconfig <INTERFACE> down 248 Example: 249 #ifconfig vxge0 down 250 251 252''Unloading the Driver'' 253 #kldunload vxge.ko 254 255 256=== Performance Suggestions === 257 258Sysctl Tuning Parameters 259 #sysctl net.inet.tcp.sendspace=786432 260 #sysctl net.inet.tcp.recvspace=786432 261 #sysctl net.inet.tcp.recvbuf_max=16777216 262 #sysctl net.inet.tcp.sendbuf_max=16777216 263 #sysctl net.inet.tcp.blackhole=1 264 #sysctl net.inet.tcp.rfc1323=1 265 #sysctl net.inet.tcp.path_mtu_discovery=1 266 #sysctl net.inet.tcp.inflight.enable=0 267 #sysctl net.inet.ip.maxfragsperpacket=2147483647 268 #sysctl kern.ipc.maxsockbuf=8388608 269 #sysctl kern.ipc.nmbclusters=2147483647 270 #sysctl kern.ipc.nmbjumbop=262144 271 #sysctl kern.ipc.maxsockets=81920 272 #sysctl hw.intr_storm_threshold=9000 273 274 275''Usage & Troubleshooting'' 276 277For general information and support, please visit Neterion support website at 278http://www.neterion.com/support/support.html 279 280Make sure that the operating system identifies the X3100 adapter. Note that 281Neterion vendor ID is 0x17D5 and X3110 and X3120 adapters can be fixed to both PCIe slots. 282 283The rest of this section details troubleshooting tips and information. Some of 284them are general and some are more specific. For online Troubleshooting tips 285and faqs, please visit 286http://trac.neterion.com/cgi-bin/trac.cgi/wiki/TitleIndex?anonymous 287 288 289''Loading the driver and initializing the device'' 290 291The first time FreeBSD identifies the device it stores the corresponding 292device/vendor IDs in an enumerated tree of PCI devices. Note that Neterion 293vendor id is 0x17d5. 294 295After cold reboot FreeBSD finds the device and tries to load the corresponding 296driver. If it fails, try to switch the card and/or cable. And, in parallel, send 297us the "tail" of the "/var/log/messages". We also need traces, register dump and 298statistics(Use vxge-manage tool). 299 300 301''Collect information'' 302 303If you're reporting a problem to Neterion, please describe: 304host(s) and adapter(s), switch (if used), software version. 305 306 307''ARP'' 308 309If this is a basic connectivity issue (e.g., cannot connect, cannot ping), 310make sure first that ARP works 311Do you see ARPs coming through a switch (in case switch is used)? 312Do you see frame drops at the switch when pinging? 313Do you see frame counts increasing via statistics? 314 315 316''Have you tried'' 317 318 a. A previous driver release; 319 b. A different adapter in the same PCI slot; 320 c. A different PCI slot; 321 d. Back-to-back setup so that the switch is excluded from the equation. 322 e. To replace the cables? 323 f. To use a different PCI slot? 324 325 326''Start clean'' 327 328 a. Have you tried to reboot the switch? Cold-reboot the host? 329 b. Make sure that the latest released driver gets loaded after the host 330 reboot, and that the 10GE interface shows up via ifconfig. 331 332 333''LEDs'' 334 335Do you see LED going green after everything is connected and drivers loaded? 336How/when does the color changes? 337 338 339''ifconfig'' 340 341Run "ifconfig -a" at the command prompt and check whether the output Looks as 342expected. Include the output in your problem report. 343 344Note for instance that "all-foxes" i.e, FF:FF:FF:FF:FF:FF MAC address could 345explain a general connectivity issue, if that's what you see. 346 347 348''Log'' 349 350Please attach the generated log, with traces enabled. 351 352Note that the driver's logging facility is configurable at compile-time. Errors 353and traces can be compiled out on a per-component basis. The components are: HAL 354fifos and rings, device, etc., see VXGE_COMPONENT_HAL_??? in the Makefile. 355 356Here's how you enable all except data path traces: 357CFLAGS_VXGE = -DVXGE_DEBUG_MODULE_MASK=0xffffffbf \ 358-DVXGE_DEBUG_ERR_MASK=0xffffffbf 359Recompile with traces and include the log in the report. 360 361 362=== Utilities === 363 364''Statistics'' 365 366 To print hardware and software statistics for interface instance 0 367 (i.e., vxge0), run: 368 #vxge-manage vxgeX stats common 369 #vxge-manage vxgeX stats mrpcim 370 #vxge-manage vxgeX stats driver 371 #vxge-manage vxgeX pciconfig 372 #vxge-manage vxgeX hwinfo 373 #vxge-manage vxgeX bw_pri_get 374 #vxge-manage vxgeX bw_pri_get vf_id 375 #vxge-manage vxgeX port_mode_get 376 377The vxge-manage tool generates log file in the working directory. Once done, ping a 378few times, and collect the statistics again (Ping both from this and the remote 379machines). 380 381Many counters could be of interest. For example, "rx_vld_frms" counts all 382valid incoming Ethernet frames seen by the adapter. Information could be derived 383from the fact that (for instance) counter stay constant during ping, if that is 384what happening. For detailed description of the X3100 counters, please refer 385to the "X3100 User Guide". 386 387Please include the statistics into your problem report. 388 389 390''X3100 registers'' 391 392 Use vxge-manage to dump all X3100 BAR0 space registers. Include this register dump 393 into your problem report. 394 #vxge-manage vxgeX regs 395 396 397''PCI configuration space'' 398 399Use vxge-manage to retrieve PCI vendor, device, etc. Include the PCI configutation 400space in your problem report. 401 #vxge-manage vxgeX pciconfig 402 403 404''Hardware Info'' 405 406To retrieve hardware info of device, e.g, serial / part number and function mode etc. 407use vxge-manage 408 #vxge-manage vxgeX hwinfo 409 410 411''Bandwidth and Priority'' 412 413Use vxge-manage to display Bandwidth and Priority information. 414 #vxge-manage vxgeX bw_pri_get 415 or 416 #vxge-manage vxgeX bw_pri_get vf_id 417 418vxge-manage can also be used to set bandwidth and priority for individual VF. 419 #vxge-manage vxgeX bw_pri_set vf_id bandwidth 420 or 421 #vxge-manage vxgeX bw_pri_set vf_id bandwidth priority 422 423 Example: 424 #vxge-manage vxge0 bw_pri_set 0 1500 425 or 426 #vxge-manage vxge0 bw_pri_set 0 1500 1 427 428''Port mode'' 429 430Use vxge-manage to display Port mode setting 431 #vxge-manage vxgeX port_mode_get 432 433vxge-manage can also be used to set Port mode. 434 #vxge-manage vxgeX port_mode_set port_mode_value 435 436 Example: 437 #vxge-manage vxge0 port_mode_set 2 438 439=== Known Issues === 440 441 442=== Available Downloads === 443 444 For latest available drivers or further support please contact your network 445 adapter provider or neterionsupport@exar.com. 446 447 448=================================================================== 449 Exar Corp., Proprietary 450 COPYRIGHT (c) 2002-2011 Exar corp., ALL RIGHTS RESERVED 451