1221167Sgnn$FreeBSD: releng/11.0/sys/dev/vxge/README 221167 2011-04-28 14:33:15Z gnn $ 2221167Sgnn''_Readme for FreeBSD X3100 Series 10GbE PCIe I/O Virtualized Server Adapter Drivers_''' 3221167Sgnn 4221167Sgnn=== Introduction === 5221167SgnnFreeBSD Driver for X3100 10GbE Server/Storage adapters 6221167Sgnn* Drivers support all X3100 10GbE adapters with FreeBSD version 7.x, 8.x and 9.x 7221167Sgnn* Supports both i386 and amd64 architectures 8221167Sgnn* Features: Jumbo frames (up to 9600), 9221167Sgnn LRO (Large Receive Offload), 10221167Sgnn TSO (TCP segmentation offload), 11221167Sgnn RTH (Receive Traffic Hash). 12221167Sgnn Also, Extended Message Signaled Interrupts (MSI-X). 13221167Sgnn 14221167Sgnn 15221167Sgnn''Features'' 16221167Sgnn 17221167Sgnn a. Jumbo frames: 18221167Sgnn X3110 and X3120 supports MTU up to 9600 bytes, modifiable using ifconfig command. 19221167Sgnn 20221167Sgnn b. LRO (Large Receive Offload): 21221167Sgnn LRO can be enabled/disabled before loading driver. 22221167Sgnn Set lro_enable in vxge.conf to 1 before loading driver. 23221167Sgnn 24221167Sgnn c. TSO (TCP Segmentation Offload) 25221167Sgnn TSO can be enabled/disabled before loading driver. 26221167Sgnn Set tso_enable in vxge.conf to 1 before loading driver. 27221167Sgnn 28221167Sgnn d. RTH (Receive Traffic Hash) 29221167Sgnn Receive side steering for better scaling. 30221167Sgnn Set rth_enable in vxge.conf to 1 before loading driver. 31221167Sgnn 32221167Sgnn e. MSI-X 33221167Sgnn Can be enabled on platforms which support it, resulting in noticeable 34221167Sgnn performance improvement. 35221167Sgnn 36221167Sgnn f. Multi-VPaths 37221167Sgnn Up to 17 hardware based transmit and receive data channels, with 38221167Sgnn multiple steering options. 39221167Sgnn 40221167Sgnn 41221167Sgnn''X3100 & Driver configuration: vxge.conf'' 42221167Sgnn 43221167SgnnThe vxge.conf contains following attributes. 44221167Sgnn 45221167Sgnn 46221167Sgnn''msix_enable'' 47221167Sgnn 48221167SgnnEnable MSI (Message Signaled Interrupts) feature in driver. 49221167Sgnn0 - INTA 50221167Sgnn1 - MSI-X 51221167SgnnDefault: 1 52221167Sgnn 53221167Sgnn 54221167Sgnn''rth_enable'' 55221167Sgnn 56221167SgnnEnables Receive side steering for better scaling (RTH - Receive Traffic Hash) 57221167SgnnRange: 0 - 1 58221167SgnnDefault: 1 59221167Sgnn 60221167Sgnn 61221167Sgnn''lro_enable'' 62221167Sgnn 63221167SgnnEnables LRO (Large Receive Offload) feature in driver. 64221167SgnnRange: 0 - 1 65221167SgnnDefault: 1 66221167Sgnn 67221167Sgnn 68221167Sgnn''tso_enable'' 69221167Sgnn 70221167SgnnEnables TSO (TCP Segmentaton Offload) feature in driver. 71221167SgnnRange: 0 - 1 72221167SgnnDefault: 1 73221167Sgnn 74221167Sgnn 75221167Sgnn''no_of_vpath'' 76221167Sgnn 77221167SgnnSpecifies maximum VPATH(s) configured for each device function. 78221167SgnnValid range: 1-17 79221167SgnnDefault: Optimized by driver 80221167Sgnn 81221167Sgnn 82221167Sgnn''func_mode'' 83221167Sgnn 84221167SgnnChange PCI function mode 85221167Sgnn 0 - SF1_VP17 (1 function with 17 VPATHs) 86221167Sgnn 1 - MF8_VP2 (8 functions with 2 VPATHs per function) 87221167Sgnn 8 - MF2_VP8 (2 functions, 8 Paths/Function) 88221167Sgnn 9 - MF4_VP4 (4 Functions, 4 Paths/Function) 89221167Sgnn 11 - MF8P_VP2 (8 functions with 2 VPATHS per function required for DirectIO) 90221167Sgnn Default: -1 91221167Sgnn 92221167Sgnn 93221167Sgnn''port_mode'' 94221167Sgnn 95221167SgnnChange the default dual port mode 96221167Sgnn 2 - Active Passive 97221167Sgnn 3 - Single Port 98221167Sgnn 4 - Dual Port 99221167Sgnn 100221167Sgnn 101221167Sgnn''l2_switch'' 102221167Sgnn 103221167SgnnTurn on/off the inter function traffic through l2 switch 104221167Sgnn 0 - Disallow inter function traffic 105221167Sgnn 1 - Allow inter function traffic 106221167Sgnn Default: -1 107221167Sgnn 108221167Sgnn 109221167Sgnn''bandwidth_0 - bandwidth_7'' 110221167Sgnn 111221167Sgnn Desired max receive/transmit bandwidth,in Mbps for function 0 to function 7 112221167Sgnn Minimum value is 100 Mbps, for 1 Gbps specify a value of 1024. 113221167Sgnn 114221167Sgnn 115221167Sgnn''priority_0 - priority_7'' 116221167Sgnn 117221167Sgnn Desired receive/transmit priority for function 0 to function 7 118221167Sgnn 119221167Sgnn 120221167Sgnn''intr_coalesce'' 121221167Sgnn 122221167Sgnn Adaptive interrupt coalescing 123221167Sgnn 0 - Disable 124221167Sgnn 1 - Enable 125221167Sgnn 126221167Sgnn 127221167Sgnn''Low Latency'' 128221167Sgnn 129221167Sgnn 0 - Disable 130221167Sgnn 1 - Enable 131221167Sgnn 132221167Sgnn=== Installation Instructions === 133221167Sgnn 134221167Sgnn''Identifying the Adapter'' 135221167Sgnn 136221167Sgnn The X3100 adapter is identified by the board ID number on the adapter. 137221167Sgnn 138221167Sgnn Look for a label that has a barcode and a number, for example, 139221167Sgnn SXT0425072. The factory-burned MAC address (hardware address) 140221167Sgnn shows up on the board above the serial number, 141221167Sgnn (similar to 000CFC000449 -- 00:0C:FC:00:04:49). 142221167Sgnn 143221167Sgnn 144221167Sgnn''Kernel Driver Source Package'' 145221167Sgnn 146221167SgnnThis package contains kernel_update.sh script which is to be used to copy driver sources to kernel path. 147221167SgnnIt creates vxge folder with source code in /usr/src/sys/dev and Makefile in /usr/src/sys/modules. 148221167Sgnn 149221167SgnnLoadable parameters can be changed by putting below lines in /boot/device.hints and set values as desired. 150221167Sgnn hint.vxge.0.msix_enable="1" 151221167Sgnn hint.vxge.0.rth_enable="1" 152221167Sgnn hint.vxge.0.lro_enable="1" 153221167Sgnn hint.vxge.0.tso_enable="1" 154221167Sgnn hint.vxge.0.tx_steering="1" 155221167Sgnn hint.vxge.0.no_of_vpath="-1" 156221167Sgnn hint.vxge.0.func_mode="-1" 157221167Sgnn hint.vxge.0.port_mode="-1" 158221167Sgnn hint.vxge.0.fw_upgrade="1" 159221167Sgnn hint.vxge.0.bandwidth_0="-1" 160221167Sgnn hint.vxge.0.bandwidth_1="-1" 161221167Sgnn hint.vxge.0.bandwidth_2="-1" 162221167Sgnn hint.vxge.0.bandwidth_3="-1" 163221167Sgnn hint.vxge.0.bandwidth_4="-1" 164221167Sgnn hint.vxge.0.bandwidth_5="-1" 165221167Sgnn hint.vxge.0.bandwidth_6="-1" 166221167Sgnn hint.vxge.0.bandwidth_7="-1" 167221167Sgnn hint.vxge.0.priority_0="-1" 168221167Sgnn hint.vxge.0.priority_1="-1" 169221167Sgnn hint.vxge.0.priority_2="-1" 170221167Sgnn hint.vxge.0.priority_3="-1" 171221167Sgnn hint.vxge.0.priority_4="-1" 172221167Sgnn hint.vxge.0.priority_5="-1" 173221167Sgnn hint.vxge.0.priority_6="-1" 174221167Sgnn hint.vxge.0.priority_7="-1" 175221167Sgnn hint.vxge.0.intr_coalesce="0" 176221167Sgnn hint.vxge.0.low_latency="0" 177221167Sgnn 178221167Sgnn e.g., set hint.vxge.0.msix_enable to 0 to load driver in INTA mode. 179221167Sgnn Then reboot the system to add loadable parameters to kenv. 180221167Sgnn 181221167Sgnn 182221167Sgnn''Standalone Driver Source Package'' 183221167Sgnn 184221167Sgnnvxge (FreeBSD package) 185221167Sgnn This directory contains FreeBSD driver sources for X3100 device(s), 186221167Sgnn Makefile, and X3100 Hardware Abstraction headers and sources 187221167Sgnn (vxgehal and include folders) 188221167Sgnn 189221167Sgnnvxgehal 190221167Sgnn This directory contains the X3100 HAL sources. 191221167Sgnn The driver uses HAL to perform operations on the X3100 hardware. 192221167Sgnn 193221167Sgnninclude 194221167Sgnn The include subdirectory contains HAL header files. 195221167Sgnn 196221167SgnnSource code for vxge-manage tool (used to get statistics, pciconfig and 197221167Sgnnregister dump) are included in the freebsd directory. 198221167Sgnn 199221167Sgnn 200221167Sgnn''Building the driver'' 201221167Sgnn 202221167Sgnn The driver is distributed in the source form. Driver and installation 203221167Sgnn utility executables need to be built for the target platform. 204221167Sgnn 205221167Sgnn In the directory containing Makefile for building Exar driver for FreeBSD, 206221167Sgnn #make clean 207221167Sgnn #make 208221167Sgnn 209221167Sgnn Please unload previously installed Exar drivers before proceeding with following steps. 210221167Sgnn #make uninstall 211221167Sgnn 212221167Sgnn 213221167Sgnn''Loading the driver'' 214221167Sgnn 215221167Sgnn Use "kldload" to load driver module vxge.ko. 216221167Sgnn #kldload ./vxge.ko 217221167Sgnn 218221167Sgnn Run "kldstat" and find an entry for vxge kernel module to ensure driver installation 219221167Sgnn was successful. 220221167Sgnn #kldstat | grep vxge 221221167Sgnn 3 1 0xc22cc000 26000 vxge.ko 222221167Sgnn 223221167Sgnn 224221167Sgnn''Enabling interface and assigning IP address'' 225221167Sgnn 226221167Sgnn #ifconfig <INTERFACE> <IP_ADDRESS> up 227221167Sgnn <INTERFACE> will be similar to vxge0, vxge1 etc. and can be 228221167Sgnn found by executing "ifconfig -a". 229221167Sgnn Neterion adapters typically have MAC addresses starting with 230221167Sgnn "00:0C:FC" or "00:11:25". 231221167Sgnn 232221167Sgnn Example: 233221167Sgnn #ifconfig vxge0 10.2.2.40 up 234221167Sgnn Enables vxge0 interface and assigns to it the IP address 10.2.2.40. 235221167Sgnn 236221167Sgnn vxge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 237221167Sgnn options=53b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4,LRO> 238221167Sgnn ether 00:0c:fc:00:da:47 239221167Sgnn inet6 fe80::20c:fcff:fe00:da47%vxge0 prefixlen 64 scopeid 0x3 240221167Sgnn inet 10.2.2.40 netmask 0xff000000 broadcast 17.255.255.255 241221167Sgnn media: Ethernet autoselect (10Gbase-SR <full-duplex>) 242221167Sgnn status: active 243221167Sgnn 244221167Sgnn 245221167Sgnn''Disabling the interface'' 246221167Sgnn 247221167Sgnn #ifconfig <INTERFACE> down 248221167Sgnn Example: 249221167Sgnn #ifconfig vxge0 down 250221167Sgnn 251221167Sgnn 252221167Sgnn''Unloading the Driver'' 253221167Sgnn #kldunload vxge.ko 254221167Sgnn 255221167Sgnn 256221167Sgnn=== Performance Suggestions === 257221167Sgnn 258221167SgnnSysctl Tuning Parameters 259221167Sgnn #sysctl net.inet.tcp.sendspace=786432 260221167Sgnn #sysctl net.inet.tcp.recvspace=786432 261221167Sgnn #sysctl net.inet.tcp.recvbuf_max=16777216 262221167Sgnn #sysctl net.inet.tcp.sendbuf_max=16777216 263221167Sgnn #sysctl net.inet.tcp.blackhole=1 264221167Sgnn #sysctl net.inet.tcp.rfc1323=1 265221167Sgnn #sysctl net.inet.tcp.path_mtu_discovery=1 266221167Sgnn #sysctl net.inet.tcp.inflight.enable=0 267221167Sgnn #sysctl net.inet.ip.maxfragsperpacket=2147483647 268221167Sgnn #sysctl kern.ipc.maxsockbuf=8388608 269221167Sgnn #sysctl kern.ipc.nmbclusters=2147483647 270221167Sgnn #sysctl kern.ipc.nmbjumbop=262144 271221167Sgnn #sysctl kern.ipc.maxsockets=81920 272221167Sgnn #sysctl hw.intr_storm_threshold=9000 273221167Sgnn 274221167Sgnn 275221167Sgnn''Usage & Troubleshooting'' 276221167Sgnn 277221167SgnnFor general information and support, please visit Neterion support website at 278221167Sgnnhttp://www.neterion.com/support/support.html 279221167Sgnn 280221167SgnnMake sure that the operating system identifies the X3100 adapter. Note that 281221167SgnnNeterion vendor ID is 0x17D5 and X3110 and X3120 adapters can be fixed to both PCIe slots. 282221167Sgnn 283221167SgnnThe rest of this section details troubleshooting tips and information. Some of 284221167Sgnnthem are general and some are more specific. For online Troubleshooting tips 285221167Sgnnand faqs, please visit 286221167Sgnnhttp://trac.neterion.com/cgi-bin/trac.cgi/wiki/TitleIndex?anonymous 287221167Sgnn 288221167Sgnn 289221167Sgnn''Loading the driver and initializing the device'' 290221167Sgnn 291221167SgnnThe first time FreeBSD identifies the device it stores the corresponding 292221167Sgnndevice/vendor IDs in an enumerated tree of PCI devices. Note that Neterion 293221167Sgnnvendor id is 0x17d5. 294221167Sgnn 295221167SgnnAfter cold reboot FreeBSD finds the device and tries to load the corresponding 296221167Sgnndriver. If it fails, try to switch the card and/or cable. And, in parallel, send 297221167Sgnnus the "tail" of the "/var/log/messages". We also need traces, register dump and 298221167Sgnnstatistics(Use vxge-manage tool). 299221167Sgnn 300221167Sgnn 301221167Sgnn''Collect information'' 302221167Sgnn 303221167SgnnIf you're reporting a problem to Neterion, please describe: 304221167Sgnnhost(s) and adapter(s), switch (if used), software version. 305221167Sgnn 306221167Sgnn 307221167Sgnn''ARP'' 308221167Sgnn 309221167SgnnIf this is a basic connectivity issue (e.g., cannot connect, cannot ping), 310221167Sgnnmake sure first that ARP works 311221167SgnnDo you see ARPs coming through a switch (in case switch is used)? 312221167SgnnDo you see frame drops at the switch when pinging? 313221167SgnnDo you see frame counts increasing via statistics? 314221167Sgnn 315221167Sgnn 316221167Sgnn''Have you tried'' 317221167Sgnn 318221167Sgnn a. A previous driver release; 319221167Sgnn b. A different adapter in the same PCI slot; 320221167Sgnn c. A different PCI slot; 321221167Sgnn d. Back-to-back setup so that the switch is excluded from the equation. 322221167Sgnn e. To replace the cables? 323221167Sgnn f. To use a different PCI slot? 324221167Sgnn 325221167Sgnn 326221167Sgnn''Start clean'' 327221167Sgnn 328221167Sgnn a. Have you tried to reboot the switch? Cold-reboot the host? 329221167Sgnn b. Make sure that the latest released driver gets loaded after the host 330221167Sgnn reboot, and that the 10GE interface shows up via ifconfig. 331221167Sgnn 332221167Sgnn 333221167Sgnn''LEDs'' 334221167Sgnn 335221167SgnnDo you see LED going green after everything is connected and drivers loaded? 336221167SgnnHow/when does the color changes? 337221167Sgnn 338221167Sgnn 339221167Sgnn''ifconfig'' 340221167Sgnn 341221167SgnnRun "ifconfig -a" at the command prompt and check whether the output Looks as 342221167Sgnnexpected. Include the output in your problem report. 343221167Sgnn 344221167SgnnNote for instance that "all-foxes" i.e, FF:FF:FF:FF:FF:FF MAC address could 345221167Sgnnexplain a general connectivity issue, if that's what you see. 346221167Sgnn 347221167Sgnn 348221167Sgnn''Log'' 349221167Sgnn 350221167SgnnPlease attach the generated log, with traces enabled. 351221167Sgnn 352221167SgnnNote that the driver's logging facility is configurable at compile-time. Errors 353221167Sgnnand traces can be compiled out on a per-component basis. The components are: HAL 354221167Sgnnfifos and rings, device, etc., see VXGE_COMPONENT_HAL_??? in the Makefile. 355221167Sgnn 356221167SgnnHere's how you enable all except data path traces: 357221167SgnnCFLAGS_VXGE = -DVXGE_DEBUG_MODULE_MASK=0xffffffbf \ 358221167Sgnn-DVXGE_DEBUG_ERR_MASK=0xffffffbf 359221167SgnnRecompile with traces and include the log in the report. 360221167Sgnn 361221167Sgnn 362221167Sgnn=== Utilities === 363221167Sgnn 364221167Sgnn''Statistics'' 365221167Sgnn 366221167Sgnn To print hardware and software statistics for interface instance 0 367221167Sgnn (i.e., vxge0), run: 368221167Sgnn #vxge-manage vxgeX stats common 369221167Sgnn #vxge-manage vxgeX stats mrpcim 370221167Sgnn #vxge-manage vxgeX stats driver 371221167Sgnn #vxge-manage vxgeX pciconfig 372221167Sgnn #vxge-manage vxgeX hwinfo 373221167Sgnn #vxge-manage vxgeX bw_pri_get 374221167Sgnn #vxge-manage vxgeX bw_pri_get vf_id 375221167Sgnn #vxge-manage vxgeX port_mode_get 376221167Sgnn 377221167SgnnThe vxge-manage tool generates log file in the working directory. Once done, ping a 378221167Sgnnfew times, and collect the statistics again (Ping both from this and the remote 379221167Sgnnmachines). 380221167Sgnn 381221167SgnnMany counters could be of interest. For example, "rx_vld_frms" counts all 382221167Sgnnvalid incoming Ethernet frames seen by the adapter. Information could be derived 383221167Sgnnfrom the fact that (for instance) counter stay constant during ping, if that is 384221167Sgnnwhat happening. For detailed description of the X3100 counters, please refer 385221167Sgnnto the "X3100 User Guide". 386221167Sgnn 387221167SgnnPlease include the statistics into your problem report. 388221167Sgnn 389221167Sgnn 390221167Sgnn''X3100 registers'' 391221167Sgnn 392221167Sgnn Use vxge-manage to dump all X3100 BAR0 space registers. Include this register dump 393221167Sgnn into your problem report. 394221167Sgnn #vxge-manage vxgeX regs 395221167Sgnn 396221167Sgnn 397221167Sgnn''PCI configuration space'' 398221167Sgnn 399221167SgnnUse vxge-manage to retrieve PCI vendor, device, etc. Include the PCI configutation 400221167Sgnnspace in your problem report. 401221167Sgnn #vxge-manage vxgeX pciconfig 402221167Sgnn 403221167Sgnn 404221167Sgnn''Hardware Info'' 405221167Sgnn 406221167SgnnTo retrieve hardware info of device, e.g, serial / part number and function mode etc. 407221167Sgnnuse vxge-manage 408221167Sgnn #vxge-manage vxgeX hwinfo 409221167Sgnn 410221167Sgnn 411221167Sgnn''Bandwidth and Priority'' 412221167Sgnn 413221167SgnnUse vxge-manage to display Bandwidth and Priority information. 414221167Sgnn #vxge-manage vxgeX bw_pri_get 415221167Sgnn or 416221167Sgnn #vxge-manage vxgeX bw_pri_get vf_id 417221167Sgnn 418221167Sgnnvxge-manage can also be used to set bandwidth and priority for individual VF. 419221167Sgnn #vxge-manage vxgeX bw_pri_set vf_id bandwidth 420221167Sgnn or 421221167Sgnn #vxge-manage vxgeX bw_pri_set vf_id bandwidth priority 422221167Sgnn 423221167Sgnn Example: 424221167Sgnn #vxge-manage vxge0 bw_pri_set 0 1500 425221167Sgnn or 426221167Sgnn #vxge-manage vxge0 bw_pri_set 0 1500 1 427221167Sgnn 428221167Sgnn''Port mode'' 429221167Sgnn 430221167SgnnUse vxge-manage to display Port mode setting 431221167Sgnn #vxge-manage vxgeX port_mode_get 432221167Sgnn 433221167Sgnnvxge-manage can also be used to set Port mode. 434221167Sgnn #vxge-manage vxgeX port_mode_set port_mode_value 435221167Sgnn 436221167Sgnn Example: 437221167Sgnn #vxge-manage vxge0 port_mode_set 2 438221167Sgnn 439221167Sgnn=== Known Issues === 440221167Sgnn 441221167Sgnn 442221167Sgnn=== Available Downloads === 443221167Sgnn 444221167Sgnn For latest available drivers or further support please contact your network 445221167Sgnn adapter provider or neterionsupport@exar.com. 446221167Sgnn 447221167Sgnn 448221167Sgnn=================================================================== 449221167Sgnn Exar Corp., Proprietary 450221167Sgnn COPYRIGHT (c) 2002-2011 Exar corp., ALL RIGHTS RESERVED 451