1Diagnostic Tools 2shaharf@voltaire.com, halr@voltaire.com 3 4General: 5 6Model of operation: All utilities use direct MAD access to perform their 7operations. Operations that require QP 0 mads only, may use direct routed 8mads, and therefore may work even in unconfigured subnets. Almost all 9utilities can operate without accessing the SM, unless GUID to lid translation 10is required. 11 12Dependencies: Most utilities depend on libibmad and libibumad. 13 All utilities depend on the ib_umad kernel module. 14 15Multiple port/Multiple CA support: when no IB device or port is specified 16 (see the "local umad parameters" below), the libibumad library 17 selects the port to use by the following criteria: 18 1. the first port that is ACTIVE. 19 2. if not found, the first port that is UP (physical link up). 20 21 If a port and/or CA name is specified, the libibumad library 22 attempts to fulfill the user request, and will fail if it is not 23 possible. 24 For example: 25 ibaddr # use the 'best port' 26 ibaddr -C mthca1 # pick the best port from mthca1 only. 27 ibaddr -P 2 # use the second (active/up) port from the 28 first available IB device. 29 ibaddr -C mthca0 -P 2 # use the specified port only. 30 31Common options & flags: 32 Most diagnostics take the following flags. The exact list of supported 33 flags per utility can be found in the usage message and can be shown 34 using util_name -h syntax. 35 36 # Debugging flags 37 -d raise the IB debugging level. May be used 38 several times (-ddd or -d -d -d). 39 -e show umad send receive errors (timeouts and others) 40 -h show the usage message 41 -v increase the application verbosity level. 42 May be used several times (-vv or -v -v -v) 43 -V show the internal version info. 44 45 # Addressing flags 46 -D use directed path address arguments. The path 47 is a comma separated list of out ports. 48 Examples: 49 "0" # self port 50 "0,1,2,1,4" # out via port 1, then 2, ... 51 -G use GUID address arguments. In most cases, it is the Port GUID. 52 Examples: 53 "0x08f1040023" 54 -s <smlid> use 'smlid' as the target lid for SA queries. 55 56 # Local umad parameters: 57 -C <ca_name> use the specified ca_name. 58 -P <ca_port> use the specified ca_port. 59 -t <timeout_ms> override the default timeout for the solicited mads. 60 61CLI notation: all utilities use the POSIX style notation, 62 meaning that all options (flags) must precede all arguments 63 (parameters). 64 65 66Utilities descriptions: 67 681. ibstatus 69 70Description: 71ibstatus is a script which displays basic information obtained from the local 72IB driver. Output includes LID, SMLID, port state, link width active, and port 73physical state. 74 75Syntax: 76ibstatus [-h] [devname[:port]]... 77 78Examples: 79 ibstatus # display status of all IB ports 80 ibstatus mthca1 # status of mthca1 ports 81 ibstatus mthca1:1 mthca0:2 # show status of specified ports 82 83See also: 84 ibstat 85 862. ibstat 87 88Description: 89Similar to the ibstatus utility but implemented as a binary and not a script. 90It has options to list CAs and/or ports. 91 92Syntax: 93ibstat [-d(ebug) -l(ist_of_cas) -p(ort_list) -s(hort)] <ca_name> [portnum] 94 95Examples: 96 ibstat # display status of all IB ports 97 ibstat mthca1 # status of mthca1 ports 98 ibstat mthca1 2 # show status of specified ports 99 ibstat -p mthca0 # list the port guids of mthca0 100 ibstat -l # list all CA names 101 102See also: 103 ibstatus 104 1053. ibroute 106 107Description: 108ibroute uses SMPs to display the forwarding tables (unicast 109(LinearForwardingTable or LFT) or multicast (MulticastForwardingTable or MFT)) 110for the specified switch LID and the optional lid (mlid) range. 111The default range is all valid entries in the range 1...FDBTop. 112 113Syntax: 114ibroute [options] <switch_addr> [<startlid> [<endlid>]]] 115 116Non standard flags: 117 -a show all lids in range, even invalid entries. 118 -n do not try to resolve destinations. 119 -M show multicast forwarding tables. In this case the range 120 parameters are specifying mlid range. 121 122Examples: 123 ibroute 2 # dump all valid entries of switch lid 2 124 ibroute 2 15 # dump entries in the range 15...FDBTop. 125 ibroute -a 2 10 20 # dump all entries in the range 10..20 126 ibroute -n 2 # simple format 127 ibroute -M 2 # show multicast tables 128 129See also: 130 ibtracert 131 1324. ibtracert 133 134Description: 135ibtracert uses SMPs to trace the path from a source GID/LID to a 136destination GID/LID. Each hop along the path is displayed until the destination 137is reached or a hop does not respond. By using the -m option, multicast path 138tracing can be performed between source and destination nodes. 139 140Syntax: 141ibtracert [options] <src-addr> <dest-addr> 142 143Non standard flags: 144 -n simple format; don't show additional information. 145 -m <mlid> show the multicast trace of the specified mlid. 146 147Examples: 148 ibtracert 2 23 # show trace between lid 2 and 23 149 ibtracert -m 0xc000 3 5 # show multicast trace between lid 3 and 5 150 for mcast lid 0xc000. 151 1525. smpquery 153 154Description: 155smpquery allows a basic subset of standard SMP queries including the following: 156node info, node description, switch info, port info. Fields are displayed in 157human readable format. 158 159Syntax: 160smpquery [options] <op> <dest_addr> [op_params] 161 162Current supported operations and their parameters: 163 nodeinfo <addr> 164 nodedesc <addr> 165 portinfo <addr> [<portnum>] # default port is zero 166 switchinfo <addr> 167 pkeys <addr> [<portnum>] 168 sl2vl <addr> [<portnum>] 169 vlarb <addr> [<portnum>] 170 171Examples: 172 smpquery nodeinfo 2 # show nodeinfo for lid 2 173 smpquery portinfo 2 5 # show portinfo for lid 2 port 5 174 1756. smpdump 176 177Description: 178smpdump is a general purpose SMP utility which gets SM attributes from a 179specified SMA. The result is dumped in hex by default. 180 181Syntax: 182smpdump [options] <dest_addr> <attr> [mod] 183 184Non standard flags: 185 -s show output as string 186 187Examples: 188 smpdump -D 0,1,2 0x15 2 # port info, port 2 189 smpdump 3 0x15 2 # port info, lid 3 port 2 190 1917. ibaddr 192 193Description: 194ibaddr can be used to show the lid and GID addresses of the specified port, 195or the local port by default. 196Note: this utility can be used as simple address resolver. 197 198Syntax: 199ibaddr [options] [<dest_addr>] 200 201Examples: 202 ibaddr # show local address 203 ibaddr 2 # show address of the specified port lid 204 ibaddr -G 0x8f1040023 # show address of the specified port guid 205 2068. sminfo 207 208Description: 209sminfo issue and dumps the output of a sminfo query in human readable format. 210The target SM is the one listed in the local port info, or the SM specified 211by the optional SM lid or by the SM direct routed path. 212Note: using sminfo for any purposes other then simple query may be very 213dangerous, and may result in a malfunction of the target SM. 214 215Syntax: 216sminfo [options] <sm_lid|sm_dr_path> [sminfo_modifier] 217 218Non standard flags: 219 -s <state> # use the specified state in sminfo mad 220 -p <priority> # use the specified priority in sminfo mad 221 -a <activity> # use the specified activity in sminfo mad 222 223Examples: 224 sminfo # show sminfo of SM listed in local portinfo 225 sminfo 2 # query SM on port lid 2 226 2279. perfquery 228 229Description: 230perfquery uses PerfMgt GMPs to obtain the PortCounters (basic performance 231and error counters) from the PMA at the node specified. Optionally 232show aggregated counters for all ports of node. Also, optionally, reset 233after read, or only reset counters. 234 235Syntax: 236perfquery [options] [<lid|guid> [[port] [reset_mask]]] 237 238Non standard flags: 239 -a show aggregated counters for all ports of the destination lid. 240 -r reset counters after read. 241 -R only reset counters. 242 243Examples: 244 perfquery # read local port's performance counters 245 perfquery 32 1 # read performance counters from lid 32, port 1 246 perfquery -a 32 # read node aggregated performance counters 247 perfquery -r 32 1 # read performance counters and reset 248 perfquery -R 32 1 # reset performance counters of port 1 only 249 perfquery -R -a 32 # reset performance counters of all ports 250 perfquery -R 32 2 0xf000 # reset only non-error counters of port 2 251 25210. ibping 253 254Description: 255ibping uses vendor mads to validate connectivity between IB nodes. 256On exit, (IP) ping like output is show. ibping is run as client/server. 257Default is to run as client. Note also that a default ping server is 258implemented within the kernel. 259 260Syntax: 261ibping [options] <dest lid|guid> 262 263Non standard flags: 264 -c <count> stop after count packets 265 -f flood destination: send packets back to back w/o delay 266 -o <oui> use specified OUI number to multiplex vendor mads 267 -S start in server mode (do not return) 268 26911. ibnetdiscover 270 271Description: 272ibnetdiscover performs IB subnet discovery and outputs a human readable 273topology file. GUIDs, node types, and port numbers are displayed 274as well as port LIDs and NodeDescriptions. All nodes (and links) are displayed 275(full topology). Optionally this utility can be used to list the current 276connected nodes. The output is printed to the standard output unless a 277topology file is specified. 278 279Syntax: 280ibnetdiscover [options] [<topology-filename>] 281 282Non standard flags: 283 -l List of connected nodes 284 -H List of connected HCAs 285 -S List of connected switches 286 -g Grouping 287 28812. ibhosts 289 290Description: 291ibhosts either walks the IB subnet topology or uses an already saved topology 292file and extracts the CA nodes. 293 294Syntax: 295ibhosts [-h] [<topology-file>] 296 297Dependencies: 298ibnetdiscover, ibnetdiscover format 299 30013. ibswitches 301 302Description: 303ibswitches either walks the IB subnet topology or uses an already saved 304topology file and extracts the IB switches. 305 306Syntax: 307ibswitches [-h] [<topology-file>] 308 309Dependencies: 310ibnetdiscover, ibnetdiscover format 311 31214. ibchecknet 313 314Description: 315ibchecknet uses a full topology file that was created by ibnetdiscover, 316scans the network to validate the connectivity and reports errors 317(from port counters). 318 319Syntax: 320ibchecknet [-h] [<topology-file>] 321 322Dependencies: 323ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckport, ibcheckerrs 324 32515. ibcheckport 326 327Description: 328Check connectivity and do some simple sanity checks for the specified port. 329Port address is lid unless -G option is used to specify a GUID address. 330 331Syntax: 332ibcheckport [-h] [-G] <lid|guid> <port_number> 333 334Example: 335 ibcheckport 2 3 # check lid 2 port 3 336 337Dependencies: 338smpquery, smpquery output format, ibaddr 339 34016. ibchecknode 341 342Description: 343Check connectivity and do some simple sanity checks for the specified node. 344Port address is lid unless -G option is used to specify a GUID address. 345 346Syntax: 347ibchecknode [-h] [-G] <lid|guid> 348 349Example: 350 ibchecknode 2 # check node via lid 2 351 352Dependencies: 353smpquery, smpquery output format, ibaddr 354 355Usage: 356 35717. ibcheckerrs 358 359Description: 360Check specified port (or node) and report errors that surpassed their predefined 361threshold. Port address is lid unless -G option is used to specify a GUID 362address. The predefined thresholds can be dumped using the -s option, and a 363user defined threshold_file (using the same format as the dump) can be 364specified using the -t <file> option. 365 366Syntax: 367ibcheckerrs [-h] [-G] [-t <threshold_file>] [-s(how_thresholds)] <lid|guid> [<port>] 368 369Examples: 370 ibcheckerrs 2 # check aggregated node counter for lid 2 371 ibcheckerrs 2 4 # check port counters for lid 2 port 4 372 ibcheckerrs -t xxx 2 # check node using xxx threshold file 373 374Dependencies: 375perfquery, perfquery output format, ibaddr 376 37718. ibportstate 378 379Description: 380ibportstate allows the port state and port physical state of an IB port 381to be queried or a switch port to be disabled or enabled. 382 383Syntax: 384ibportstate [-d(ebug) -e(rr_show) -v(erbose) -D(irect) -G(uid) -s smlid 385-V(ersion) -C ca_name -P ca_port -t timeout_ms] <dest dr_path|lid|guid> 386<portnum> [<op>] 387 supported ops: enable, disable, query 388 389Examples: 390 ibportstate 3 1 disable # by lid 391 ibportstate -G 0x2C9000100D051 1 enable # by guid 392 ibportstate -D 0 1 # by direct route 393 39419. ibcheckwidth 395 396Description: 397ibcheckwidth uses a full topology file that was created by ibnetdiscover, 398scans the network to validate the active link widths and reports any 1x 399links. 400 401Syntax: 402ibcheckwidth [-h] [<topology-file>] 403 404Dependencies: 405ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckportwidth 406 40720. ibcheckportwidth 408 409Description: 410Check connectivity and check the specified port for 1x link width. 411Port address is lid unless -G option is used to specify a GUID address. 412 413Syntax: 414ibcheckportwidth [-h] [-G] <lid|guid> <port> 415 416Example: 417 ibcheckportwidth 2 3 # check lid 2 port 3 418 419Dependencies: 420smpquery, smpquery output format, ibaddr 421 42221. ibcheckstate 423 424Description: 425ibcheckstate uses a full topology file that was created by ibnetdiscover, 426scans the network to validate the port state and port physical state, 427and reports any ports which have a port state other than Active or 428a port physical state other than LinkUp. 429 430Syntax: 431ibcheckstate [-h] [<topology-file>] 432 433Dependencies: 434ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckportstate 435 43622. ibcheckportstate 437 438Description: 439Check connectivity and check the specified port for proper port state 440(Active) and port physical state (LinkUp). 441Port address is lid unless -G option is used to specify a GUID address. 442 443yntax: 444ibcheckportstate [-h] [-G] <lid|guid> <port_number> 445 446Example: 447 ibcheckportstate 2 3 # check lid 2 port 3 448 449Dependencies: 450smpquery, smpquery output format, ibaddr 451 45223. ibcheckerrors 453 454ibcheckerrors uses a full topology file that was created by ibnetdiscover, 455scans the network to validate the connectivity and reports errors 456(from port counters). 457 458Syntax: 459ibnetcheckerrors [-h] [<topology-file>] 460 461Dependencies: 462ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckport, ibcheckerrs 463 46424. ibdiscover.pl 465 466ibdiscover.pl uses a topology file create by ibnetdiscover and a discover.map 467file which the network administrator creates which indicates the nodes 468to be expected and a ibdiscover.topo file which is the expected connectivity 469and produces a new connectivity file (discover.topo.new) and outputs 470the changes to stdout. The network administrator can choose to replace 471the "old" topo file with the new one or certain changes in. 472 473The syntax of the ibdiscover.map file is: 474<nodeGUID>|port|"Text for node"|<NodeDescription from ibnetdiscover format> 475e.g. 4768f10400410015|8|"ISR 6000"|# SW-6IB4 Voltaire port 0 lid 5 4778f10403960558|2|"HCA 1"|# MT23108 InfiniHost Mellanox Technologies 478 479The syntax of the old and new topo files (ibdiscover.topo and 480ibdiscover.topo.new) are: 481<LocalPort>|<LocalNodeGUID>|<RemotePort>|<RemoteNodeGUID> 482e.g. 48310|5442ba00003080|1|8f10400410015 484 485These topo files are produced by the ibdiscover.pl tool. 486 487Syntax: 488ibnetdiscover | ibdiscover.pl 489 490Dependencies: 491ibnetdiscover, ibnetdiscover format 492 49325. ibnodes 494 495Description: 496ibnodes either walks the IB subnet topology or uses an already saved topology 497file and extracts the IB nodes (CAs and switches). 498 499Syntax: 500ibnodes [<topology-file>] 501 502Dependencies: 503ibnetdiscover, ibnetdiscover format 504 50526. ibclearerrors 506 507Description: 508ibclearerrors clears the PMA error counters in PortCounters by either walking 509the IB subnet topology or using an already saved topology file. 510 511Syntax: 512ibclearerrors [-h] [<topology-file>] 513 514Dependencies: 515ibnetdiscover, ibnetdiscover format, perfquery 516 51727. ibclearcounters 518 519Description: 520ibclearcounters clears the PMA port counters by either walking 521the IB subnet topology or using an already saved topology file. 522 523Syntax: 524ibclearcounters [-h] [<topology-file>] 525 526Dependencies: 527ibnetdiscover, ibnetdiscover format, perfquery 528 52928. saquery 530 531Description: 532Issue some SA queries. 533 534Syntax: 535Usage: saquery [-h -d -P -N -L -G -s -g][<name>] 536 Queries node records by default 537 -d enable debugging 538 -P get PathRecord info 539 -N get NodeRecord info 540 -L Return just the Lid of the name specified 541 -G Return just the Guid of the name specified 542 -s Return the PortInfoRecords with isSM capability mask bit on 543 -g get multicast group info 544 545Dependencies: 546OpenSM libvendor, OpenSM libopensm, libibumad 547 54829. ibsysstat 549 550Description: 551ibsysstat uses vendor mads to validate connectivity between IB nodes 552and obtain other information about the IB node. ibsysstat is run as 553client/server. Default is to run as client. 554 555Syntax: 556ibsysstat [options] <dest lid|guid> [<op>] 557 558Non standard flags: 559 Current supported operations: 560 ping - verify connectivity to server (default) 561 host - obtain host information from server 562 cpu - obtain cpu information from server 563 -o <oui> use specified OUI number to multiplex vendor mads 564 -S start in server mode (do not return) 565 566