1Diagnostic Tools
2shaharf@voltaire.com, halr@voltaire.com
3
4General:
5
6Model of operation: All utilities use direct MAD access to perform their
7operations. Operations that require QP 0 mads only, may use direct routed
8mads, and therefore may work even in unconfigured subnets. Almost all
9utilities can operate without accessing the SM, unless GUID to lid translation
10is required.
11
12Dependencies: Most utilities depend on libibmad and libibumad.
13	All utilities depend on the ib_umad kernel module.
14
15Multiple port/Multiple CA support: when no IB device or port is specified
16	(see the "local umad parameters" below), the libibumad library
17	selects the port to use by the following criteria:
18	1. the first port that is ACTIVE.
19	2. if not found, the first port that is UP (physical link up).
20
21	If a port and/or CA name is specified, the libibumad library
22	attempts to fulfill the user request, and will fail if it is not
23	possible.
24	For example:
25	ibaddr 			# use the 'best port'
26	ibaddr -C mthca1	# pick the best port from mthca1 only.
27	ibaddr -P 2		# use the second (active/up) port from the
28				  first available IB device.
29	ibaddr -C mthca0 -P 2	# use the specified port only.
30
31Common options & flags:
32	Most diagnostics take the following flags. The exact list of supported
33	flags per utility can be found in the usage message and can be shown
34	using util_name -h syntax.
35
36	# Debugging flags
37	-d	raise the IB debugging level. May be used
38		several times (-ddd or -d -d -d).
39	-e	show umad send receive errors (timeouts and others)
40	-h	show the usage message
41	-v	increase the application verbosity level.
42		May be used several times (-vv or -v -v -v)
43	-V	show the internal version info.
44
45	# Addressing flags
46	-D	use directed path address arguments. The path
47		is a comma separated list of out ports.
48		Examples:
49		"0" 		# self port
50		"0,1,2,1,4"	# out via port 1, then 2, ...
51	-G	use GUID address arguments. In most cases, it is the Port GUID.
52		Examples:
53		"0x08f1040023"
54	-s <smlid>	use 'smlid' as the target lid for SA queries.
55
56	# Local umad parameters:
57	-C <ca_name>	use the specified ca_name.
58	-P <ca_port>	use the specified ca_port.
59	-t <timeout_ms>	override the default timeout for the solicited mads.
60
61CLI notation: all utilities use the POSIX style notation,
62	meaning that all options (flags) must precede all arguments
63	(parameters).
64
65
66Utilities descriptions:
67
681. ibstatus
69
70Description:
71ibstatus is a script which displays basic information obtained from the local
72IB driver. Output includes LID, SMLID, port state, link width active, and port
73physical state.
74
75Syntax:
76ibstatus [-h] [devname[:port]]...
77
78Examples:
79	ibstatus		# display status of all IB ports
80	ibstatus mthca1 	# status of mthca1 ports
81	ibstatus mthca1:1 mthca0:2	# show status of specified ports
82
83See also:
84	ibstat
85
862. ibstat
87
88Description:
89Similar to the ibstatus utility but implemented as a binary and not a script.
90It has options to list CAs and/or ports.
91
92Syntax:
93ibstat [-d(ebug) -l(ist_of_cas) -p(ort_list) -s(hort)] <ca_name> [portnum]
94
95Examples:
96	ibstat			# display status of all IB ports
97	ibstat mthca1 		# status of mthca1 ports
98	ibstat mthca1 2		# show status of specified ports
99	ibstat -p mthca0	# list the port guids of mthca0
100	ibstat -l		# list all CA names
101
102See also:
103	ibstatus
104
1053. ibroute
106
107Description:
108ibroute uses SMPs to display the forwarding tables (unicast
109(LinearForwardingTable or LFT) or multicast (MulticastForwardingTable or MFT))
110for the specified switch LID and the optional lid (mlid) range.
111The default range is all valid entries in the range 1...FDBTop.
112
113Syntax:
114ibroute [options] <switch_addr> [<startlid> [<endlid>]]]
115
116Non standard flags:
117	-a	show all lids in range, even invalid entries.
118	-n	do not try to resolve destinations.
119	-M 	show multicast forwarding tables. In this case the range
120		parameters are specifying mlid range.
121
122Examples:
123	ibroute 2		# dump all valid entries of switch lid 2
124	ibroute 2 15		# dump entries in the range 15...FDBTop.
125	ibroute -a 2 10 20	# dump all entries in the range 10..20
126	ibroute -n 2		# simple format
127	ibroute -M 2		# show multicast tables
128
129See also:
130	ibtracert
131
1324. ibtracert
133
134Description:
135ibtracert uses SMPs to trace the path from a source GID/LID to a
136destination GID/LID. Each hop along the path is displayed until the destination
137is reached or a hop does not respond. By using the -m option, multicast path
138tracing can be performed between source and destination nodes.
139
140Syntax:
141ibtracert [options] <src-addr> <dest-addr>
142
143Non standard flags:
144	-n		simple format; don't show additional information.
145	-m <mlid>	show the multicast trace of the specified mlid.
146
147Examples:
148	ibtracert 2 23		# show trace between lid 2 and 23
149	ibtracert -m 0xc000 3 5 # show multicast trace between lid 3 and 5
150				   for mcast lid 0xc000.
151
1525. smpquery
153
154Description:
155smpquery allows a basic subset of standard SMP queries including the following:
156node info, node description, switch info, port info. Fields are displayed in
157human readable format.
158
159Syntax:
160smpquery [options] <op> <dest_addr> [op_params]
161
162Current supported operations and their parameters:
163	nodeinfo <addr>
164	nodedesc <addr>
165	portinfo <addr> [<portnum>]	# default port is zero
166	switchinfo <addr>
167        pkeys <addr> [<portnum>]
168        sl2vl <addr> [<portnum>]
169        vlarb <addr> [<portnum>]
170
171Examples:
172	smpquery nodeinfo 2		# show nodeinfo for lid 2
173	smpquery portinfo 2 5		# show portinfo for lid 2 port 5
174
1756. smpdump
176
177Description:
178smpdump is a general purpose SMP utility which gets SM attributes from a
179specified SMA. The result is dumped in hex by default.
180
181Syntax:
182smpdump [options] <dest_addr> <attr> [mod]
183
184Non standard flags:
185	-s	show output as string
186
187Examples:
188	smpdump -D 0,1,2 0x15 2		# port info, port 2
189	smpdump 3 0x15 2		# port info, lid 3 port 2
190
1917. ibaddr
192
193Description:
194ibaddr can be used to show the lid and GID addresses of the specified port,
195or the local port by default.
196Note: this utility can be used as simple address resolver.
197
198Syntax:
199ibaddr [options] [<dest_addr>]
200
201Examples:
202	ibaddr			# show local address
203	ibaddr 2		# show address of the specified port lid
204	ibaddr -G 0x8f1040023	# show address of the specified port guid
205
2068. sminfo
207
208Description:
209sminfo issue and dumps the output of a sminfo query in human readable format.
210The target SM is the one listed in the local port info, or the SM specified
211by the optional SM lid or by the SM direct routed path.
212Note: using sminfo for any purposes other then simple query may be very
213dangerous, and may result in a malfunction of the target SM.
214
215Syntax:
216sminfo [options] <sm_lid|sm_dr_path> [sminfo_modifier]
217
218Non standard flags:
219	-s <state>	# use the specified state in sminfo mad
220	-p <priority>	# use the specified priority in sminfo mad
221	-a <activity>	# use the specified activity in sminfo mad
222
223Examples:
224	sminfo			# show sminfo of SM listed in local portinfo
225	sminfo 2		# query SM on port lid 2
226
2279. perfquery
228
229Description:
230perfquery uses PerfMgt GMPs to obtain the PortCounters (basic performance
231and error counters) from the PMA at the node specified. Optionally
232show aggregated counters for all ports of node. Also, optionally, reset
233after read, or only reset counters.
234
235Syntax:
236perfquery [options]  [<lid|guid> [[port] [reset_mask]]]
237
238Non standard flags:
239	-a	show aggregated counters for all ports of the destination lid.
240	-r	reset counters after read.
241	-R	only reset counters.
242
243Examples:
244	perfquery               # read local port's performance counters
245	perfquery 32 1          # read performance counters from lid 32, port 1
246	perfquery -a 32         # read node aggregated performance counters
247	perfquery -r 32 1	# read performance counters and reset
248	perfquery -R 32 1	# reset performance counters of port 1 only
249	perfquery -R -a 32	# reset performance counters of all ports
250	perfquery -R 32 2 0xf000  # reset only non-error counters of port 2
251
25210. ibping
253
254Description:
255ibping uses vendor mads to validate connectivity between IB nodes.
256On exit, (IP) ping like output is show. ibping is run as client/server.
257Default is to run as client. Note also that a default ping server is
258implemented within the kernel.
259
260Syntax:
261ibping [options] <dest lid|guid>
262
263Non standard flags:
264	-c <count>	stop after count packets
265	-f		flood destination: send packets back to back w/o delay
266	-o <oui>	use specified OUI number to multiplex vendor mads
267	-S		start in server mode (do not return)
268
26911. ibnetdiscover
270
271Description:
272ibnetdiscover performs IB subnet discovery and outputs a human readable
273topology file. GUIDs, node types, and port numbers are displayed
274as well as port LIDs and NodeDescriptions. All nodes (and links) are displayed
275(full topology). Optionally this utility can be used to list the current
276connected nodes. The output is printed to the standard output unless a
277topology file is specified.
278
279Syntax:
280ibnetdiscover [options] [<topology-filename>]
281
282Non standard flags:
283	-l	List of connected nodes
284	-H	List of connected HCAs
285	-S	List of connected switches
286	-g	Grouping
287
28812. ibhosts
289
290Description:
291ibhosts either walks the IB subnet topology or uses an already saved topology
292file and extracts the CA nodes.
293
294Syntax:
295ibhosts [-h] [<topology-file>]
296
297Dependencies:
298ibnetdiscover, ibnetdiscover format
299
30013. ibswitches
301
302Description:
303ibswitches either walks the IB subnet topology or uses an already saved
304topology file and extracts the IB switches.
305
306Syntax:
307ibswitches [-h] [<topology-file>]
308
309Dependencies:
310ibnetdiscover, ibnetdiscover format
311
31214. ibchecknet
313
314Description:
315ibchecknet uses a full topology file that was created by ibnetdiscover,
316scans the network to validate the connectivity and reports errors
317(from port counters).
318
319Syntax:
320ibchecknet [-h] [<topology-file>]
321
322Dependencies:
323ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckport, ibcheckerrs
324
32515. ibcheckport
326
327Description:
328Check connectivity and do some simple sanity checks for the specified port.
329Port address is lid unless -G option is used to specify a GUID address.
330
331Syntax:
332ibcheckport [-h] [-G] <lid|guid> <port_number>
333
334Example:
335	ibcheckport 2 3		# check lid 2 port 3
336
337Dependencies:
338smpquery, smpquery output format, ibaddr
339
34016. ibchecknode
341
342Description:
343Check connectivity and do some simple sanity checks for the specified node.
344Port address is lid unless -G option is used to specify a GUID address.
345
346Syntax:
347ibchecknode [-h] [-G] <lid|guid>
348
349Example:
350	ibchecknode 2		# check node via lid 2
351
352Dependencies:
353smpquery, smpquery output format, ibaddr
354
355Usage:
356
35717. ibcheckerrs
358
359Description:
360Check specified port (or node) and report errors that surpassed their predefined
361threshold. Port address is lid unless -G option is used to specify a GUID
362address. The predefined thresholds can be dumped using the -s option, and a
363user defined threshold_file (using the same format as the dump) can be
364specified using the -t <file> option.
365
366Syntax:
367ibcheckerrs [-h] [-G] [-t <threshold_file>] [-s(how_thresholds)] <lid|guid> [<port>]
368
369Examples:
370	ibcheckerrs 2		# check aggregated node counter for lid 2
371	ibcheckerrs 2	4	# check port counters for lid 2 port 4
372	ibcheckerrs -t xxx 2	# check node using xxx threshold file
373
374Dependencies:
375perfquery, perfquery output format, ibaddr
376
37718. ibportstate
378
379Description:
380ibportstate allows the port state and port physical state of an IB port
381to be queried or a switch port to be disabled or enabled.
382
383Syntax:
384ibportstate [-d(ebug) -e(rr_show) -v(erbose) -D(irect) -G(uid) -s smlid
385-V(ersion) -C ca_name -P ca_port -t timeout_ms] <dest dr_path|lid|guid>
386<portnum> [<op>]
387	supported ops: enable, disable, query
388
389Examples:
390	ibportstate 3 1 disable                 # by lid
391	ibportstate -G 0x2C9000100D051 1 enable # by guid
392	ibportstate -D 0 1                      # by direct route
393
39419. ibcheckwidth
395
396Description:
397ibcheckwidth uses a full topology file that was created by ibnetdiscover,
398scans the network to validate the active link widths and reports any 1x
399links.
400
401Syntax:
402ibcheckwidth [-h] [<topology-file>]
403
404Dependencies:
405ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckportwidth
406
40720. ibcheckportwidth
408
409Description:
410Check connectivity and check the specified port for 1x link width.
411Port address is lid unless -G option is used to specify a GUID address.
412
413Syntax:
414ibcheckportwidth [-h] [-G] <lid|guid> <port>
415
416Example:
417        ibcheckportwidth 2 3         # check lid 2 port 3
418
419Dependencies:
420smpquery, smpquery output format, ibaddr
421
42221. ibcheckstate
423
424Description:
425ibcheckstate uses a full topology file that was created by ibnetdiscover,
426scans the network to validate the port state and port physical state,
427and reports any ports which have a port state other than Active or
428a port physical state other than LinkUp.
429
430Syntax:
431ibcheckstate [-h] [<topology-file>]
432
433Dependencies:
434ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckportstate
435
43622. ibcheckportstate
437
438Description:
439Check connectivity and check the specified port for proper port state
440(Active) and port physical state (LinkUp).
441Port address is lid unless -G option is used to specify a GUID address.
442
443yntax:
444ibcheckportstate [-h] [-G] <lid|guid> <port_number>
445
446Example:
447        ibcheckportstate 2 3         # check lid 2 port 3
448
449Dependencies:
450smpquery, smpquery output format, ibaddr
451
45223. ibcheckerrors
453
454ibcheckerrors uses a full topology file that was created by ibnetdiscover,
455scans the network to validate the connectivity and reports errors
456(from port counters).
457
458Syntax:
459ibnetcheckerrors [-h] [<topology-file>]
460
461Dependencies:
462ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckport, ibcheckerrs
463
46424. ibdiscover.pl
465
466ibdiscover.pl uses a topology file create by ibnetdiscover and a discover.map
467file which the network administrator creates which indicates the nodes
468to be expected and a ibdiscover.topo file which is the expected connectivity
469and produces a new connectivity file (discover.topo.new) and outputs
470the changes to stdout. The network administrator can choose to replace
471the "old" topo file with the new one or certain changes in.
472
473The syntax of the ibdiscover.map file is:
474<nodeGUID>|port|"Text for node"|<NodeDescription from ibnetdiscover format>
475e.g.
4768f10400410015|8|"ISR 6000"|# SW-6IB4 Voltaire port 0 lid 5
4778f10403960558|2|"HCA 1"|# MT23108 InfiniHost Mellanox Technologies
478
479The syntax of the old and new topo files (ibdiscover.topo and
480ibdiscover.topo.new) are:
481<LocalPort>|<LocalNodeGUID>|<RemotePort>|<RemoteNodeGUID>
482e.g.
48310|5442ba00003080|1|8f10400410015
484
485These topo files are produced by the ibdiscover.pl tool.
486
487Syntax:
488ibnetdiscover | ibdiscover.pl
489
490Dependencies:
491ibnetdiscover, ibnetdiscover format
492
49325. ibnodes
494
495Description:
496ibnodes either walks the IB subnet topology or uses an already saved topology
497file and extracts the IB nodes (CAs and switches).
498
499Syntax:
500ibnodes [<topology-file>]
501
502Dependencies:
503ibnetdiscover, ibnetdiscover format
504
50526. ibclearerrors
506
507Description:
508ibclearerrors clears the PMA error counters in PortCounters by either walking
509the IB subnet topology or using an already saved topology file.
510
511Syntax:
512ibclearerrors [-h] [<topology-file>]
513
514Dependencies:
515ibnetdiscover, ibnetdiscover format, perfquery
516
51727. ibclearcounters
518
519Description:
520ibclearcounters clears the PMA port counters by either walking
521the IB subnet topology or using an already saved topology file.
522
523Syntax:
524ibclearcounters [-h] [<topology-file>]
525
526Dependencies:
527ibnetdiscover, ibnetdiscover format, perfquery
528
52928. saquery
530
531Description:
532Issue some SA queries.
533
534Syntax:
535Usage: saquery [-h -d -P -N -L -G -s -g][<name>]
536   Queries node records by default
537   -d enable debugging
538   -P get PathRecord info
539   -N get NodeRecord info
540   -L Return just the Lid of the name specified
541   -G Return just the Guid of the name specified
542   -s Return the PortInfoRecords with isSM capability mask bit on
543   -g get multicast group info
544
545Dependencies:
546OpenSM libvendor, OpenSM libopensm, libibumad
547
54829. ibsysstat
549
550Description:
551ibsysstat  uses  vendor  mads to validate connectivity between IB nodes
552and obtain other information about the IB node.  ibsysstat  is  run  as
553client/server. Default is to run as client.
554
555Syntax:
556ibsysstat [options] <dest lid|guid> [<op>]
557
558Non standard flags:
559       Current supported operations:
560               ping - verify connectivity to server (default)
561               host - obtain host information from server
562               cpu  - obtain cpu information from server
563        -o <oui>        use specified OUI number to multiplex vendor mads
564        -S              start in server mode (do not return)
565
566