xref: /freebsd-9.3-release/contrib/ofed/management/infiniband-diags/README

Diagnostic Tools
shaharf@voltaire.com, halr@voltaire.com

General:

Model of operation: All utilities use direct MAD access to perform their
operations. Operations that require QP 0 mads only, may use direct routed
mads, and therefore may work even in unconfigured subnets. Almost all
utilities can operate without accessing the SM, unless GUID to lid translation
is required.

Dependencies: Most utilities depend on libibmad and libibumad.
	All utilities depend on the ib_umad kernel module.

Multiple port/Multiple CA support: when no IB device or port is specified
	(see the "local umad parameters" below), the libibumad library
	selects the port to use by the following criteria:
	1. the first port that is ACTIVE.
	2. if not found, the first port that is UP (physical link up).

	If a port and/or CA name is specified, the libibumad library
	attempts to fulfill the user request, and will fail if it is not
	possible.
	For example:
	ibaddr 			# use the 'best port'
	ibaddr -C mthca1	# pick the best port from mthca1 only.
	ibaddr -P 2		# use the second (active/up) port from the
				  first available IB device.
	ibaddr -C mthca0 -P 2	# use the specified port only.

Common options & flags:
	Most diagnostics take the following flags. The exact list of supported
	flags per utility can be found in the usage message and can be shown
	using util_name -h syntax.

	# Debugging flags
	-d	raise the IB debugging level. May be used
		several times (-ddd or -d -d -d).
	-e	show umad send receive errors (timeouts and others)
	-h	show the usage message
	-v	increase the application verbosity level.
		May be used several times (-vv or -v -v -v)
	-V	show the internal version info.

	# Addressing flags
	-D	use directed path address arguments. The path
		is a comma separated list of out ports.
		Examples:
		"0" 		# self port
		"0,1,2,1,4"	# out via port 1, then 2, ...
	-G	use GUID address arguments. In most cases, it is the Port GUID.
		Examples:
		"0x08f1040023"
	-s <smlid>	use 'smlid' as the target lid for SA queries.

	# Local umad parameters:
	-C <ca_name>	use the specified ca_name.
	-P <ca_port>	use the specified ca_port.
	-t <timeout_ms>	override the default timeout for the solicited mads.

CLI notation: all utilities use the POSIX style notation,
	meaning that all options (flags) must precede all arguments
	(parameters).


Utilities descriptions:

1. ibstatus

Description:
ibstatus is a script which displays basic information obtained from the local
IB driver. Output includes LID, SMLID, port state, link width active, and port
physical state.

Syntax:
ibstatus [-h] [devname[:port]]...

Examples:
	ibstatus		# display status of all IB ports
	ibstatus mthca1 	# status of mthca1 ports
	ibstatus mthca1:1 mthca0:2	# show status of specified ports

See also:
	ibstat

2. ibstat

Description:
Similar to the ibstatus utility but implemented as a binary and not a script.
It has options to list CAs and/or ports.

Syntax:
ibstat [-d(ebug) -l(ist_of_cas) -p(ort_list) -s(hort)] <ca_name> [portnum]

Examples:
	ibstat			# display status of all IB ports
	ibstat mthca1 		# status of mthca1 ports
	ibstat mthca1 2		# show status of specified ports
	ibstat -p mthca0	# list the port guids of mthca0
	ibstat -l		# list all CA names

See also:
	ibstatus

3. ibroute

Description:
ibroute uses SMPs to display the forwarding tables (unicast
(LinearForwardingTable or LFT) or multicast (MulticastForwardingTable or MFT))
for the specified switch LID and the optional lid (mlid) range.
The default range is all valid entries in the range 1...FDBTop.

Syntax:
ibroute [options] <switch_addr> [<startlid> [<endlid>]]]

Non standard flags:
	-a	show all lids in range, even invalid entries.
	-n	do not try to resolve destinations.
	-M 	show multicast forwarding tables. In this case the range
		parameters are specifying mlid range.

Examples:
	ibroute 2		# dump all valid entries of switch lid 2
	ibroute 2 15		# dump entries in the range 15...FDBTop.
	ibroute -a 2 10 20	# dump all entries in the range 10..20
	ibroute -n 2		# simple format
	ibroute -M 2		# show multicast tables

See also:
	ibtracert

4. ibtracert

Description:
ibtracert uses SMPs to trace the path from a source GID/LID to a
destination GID/LID. Each hop along the path is displayed until the destination
is reached or a hop does not respond. By using the -m option, multicast path
tracing can be performed between source and destination nodes.

Syntax:
ibtracert [options] <src-addr> <dest-addr>

Non standard flags:
	-n		simple format; don't show additional information.
	-m <mlid>	show the multicast trace of the specified mlid.

Examples:
	ibtracert 2 23		# show trace between lid 2 and 23
	ibtracert -m 0xc000 3 5 # show multicast trace between lid 3 and 5
				   for mcast lid 0xc000.

5. smpquery

Description:
smpquery allows a basic subset of standard SMP queries including the following:
node info, node description, switch info, port info. Fields are displayed in
human readable format.

Syntax:
smpquery [options] <op> <dest_addr> [op_params]

Current supported operations and their parameters:
	nodeinfo <addr>
	nodedesc <addr>
	portinfo <addr> [<portnum>]	# default port is zero
	switchinfo <addr>
        pkeys <addr> [<portnum>]
        sl2vl <addr> [<portnum>]
        vlarb <addr> [<portnum>]

Examples:
	smpquery nodeinfo 2		# show nodeinfo for lid 2
	smpquery portinfo 2 5		# show portinfo for lid 2 port 5

6. smpdump

Description:
smpdump is a general purpose SMP utility which gets SM attributes from a
specified SMA. The result is dumped in hex by default.

Syntax:
smpdump [options] <dest_addr> <attr> [mod]

Non standard flags:
	-s	show output as string

Examples:
	smpdump -D 0,1,2 0x15 2		# port info, port 2
	smpdump 3 0x15 2		# port info, lid 3 port 2

7. ibaddr

Description:
ibaddr can be used to show the lid and GID addresses of the specified port,
or the local port by default.
Note: this utility can be used as simple address resolver.

Syntax:
ibaddr [options] [<dest_addr>]

Examples:
	ibaddr			# show local address
	ibaddr 2		# show address of the specified port lid
	ibaddr -G 0x8f1040023	# show address of the specified port guid

8. sminfo

Description:
sminfo issue and dumps the output of a sminfo query in human readable format.
The target SM is the one listed in the local port info, or the SM specified
by the optional SM lid or by the SM direct routed path.
Note: using sminfo for any purposes other then simple query may be very
dangerous, and may result in a malfunction of the target SM.

Syntax:
sminfo [options] <sm_lid|sm_dr_path> [sminfo_modifier]

Non standard flags:
	-s <state>	# use the specified state in sminfo mad
	-p <priority>	# use the specified priority in sminfo mad
	-a <activity>	# use the specified activity in sminfo mad

Examples:
	sminfo			# show sminfo of SM listed in local portinfo
	sminfo 2		# query SM on port lid 2

9. perfquery

Description:
perfquery uses PerfMgt GMPs to obtain the PortCounters (basic performance
and error counters) from the PMA at the node specified. Optionally reset all
or

Syntax:
perfquery [options]  [<lid|guid> [[port] [reset_mask]]]

Non standard flags:
	-a	show aggregated counters for all ports of the destination lid.
	-r	reset counters after read.
	-R	only reset counters.

Examples:
	perfquery               # read local port's performance counters
	perfquery 32 1          # read performance counters from lid 32, port 1
	perfquery -a 32         # read node aggregated performance counters
	perfquery -r 32 1	# read performance counters and reset
	perfquery -R 32 1	# reset performance counters of port 1 only
	perfquery -R -a 32	# reset performance counters of all ports
	perfquery -R 32 2 0xf000  # reset only non-error counters of port 2

10. ibping

Description:
ibping uses vendor mads to validate connectivity between IB nodes.
On exit, (IP) ping like output is show. ibping is run as client/server.
Default is to run as client. Note also that a default ping server is
implemented within the kernel.

Syntax:
ibping [options] <dest lid|guid>

Non standard flags:
	-c <count>	stop after count packets
	-f		flood destination: send packets back to back w/o delay
	-o <oui>	use specified OUI number to multiplex vendor mads
	-S		start in server mode (do not return)

11. ibnetdiscover

Description:
ibnetdiscover performs IB subnet discovery and outputs a human readable
topology file. GUIDs, node types, and port numbers are displayed
as well as port LIDs and NodeDescriptions. All nodes (and links) are displayed
(full topology). Optionally this utility can be used to list the current
connected nodes. The output is printed to the standard output unless a
topology file is specified.

Syntax:
ibnetdiscover [options] [<topology-filename>]

Non standard flags:
	-l	List of connected nodes
	-H	List of connected HCAs
	-S	List of connected switches
	-g	Grouping

12. ibhosts

Description:
ibhosts either walks the IB subnet topology or uses an already saved topology
file and extracts the CA nodes.

Syntax:
ibhosts [-h] [<topology-file>]

Dependencies:
ibnetdiscover, ibnetdiscover format

13. ibswitches

Description:
ibswitches either walks the IB subnet topology or uses an already saved
topology file and extracts the IB switches.

Syntax:
ibswitches [-h] [<topology-file>]

Dependencies:
ibnetdiscover, ibnetdiscover format

14. ibchecknet

Description:
ibchecknet uses a full topology file that was created by ibnetdiscover,
scans the network to validate the connectivity and reports errors
(from port counters).

Syntax:
ibchecknet [-h] [<topology-file>]

Dependencies:
ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckport, ibcheckerrs

15. ibcheckport

Description:
Check connectivity and do some simple sanity checks for the specified port.
Port address is lid unless -G option is used to specify a GUID address.

Syntax:
ibcheckport [-h] [-G] <lid|guid> <port_number>

Example:
	ibcheckport 2 3		# check lid 2 port 3

Dependencies:
smpquery, smpquery output format, ibaddr

16. ibchecknode

Description:
Check connectivity and do some simple sanity checks for the specified node.
Port address is lid unless -G option is used to specify a GUID address.

Syntax:
ibchecknode [-h] [-G] <lid|guid>

Example:
	ibchecknode 2		# check node via lid 2

Dependencies:
smpquery, smpquery output format, ibaddr

Usage:

17. ibcheckerrs

Description:
Check specified port (or node) and report errors that surpassed their predefined
threshold. Port address is lid unless -G option is used to specify a GUID
address. The predefined thresholds can be dumped using the -s option, and a
user defined threshold_file (using the same format as the dump) can be
specified using the -t <file> option.

Syntax:
ibcheckerrs [-h] [-G] [-t <threshold_file>] [-s(how_thresholds)] <lid|guid> [<port>]

Examples:
	ibcheckerrs 2		# check aggregated node counter for lid 2
	ibcheckerrs 2	4	# check port counters for lid 2 port 4
	ibcheckerrs -t xxx 2	# check node using xxx threshold file

Dependencies:
perfquery, perfquery output format, ibaddr

18. ibportstate

Description:
ibportstate allows the port state and port physical state of an IB port
to be queried or a switch port to be disabled or enabled.

Syntax:
ibportstate [-d(ebug) -e(rr_show) -v(erbose) -D(irect) -G(uid) -s smlid
-V(ersion) -C ca_name -P ca_port -t timeout_ms] <dest dr_path|lid|guid>
<portnum> [<op>]
	supported ops: enable, disable, query

Examples:
	ibportstate 3 1 disable                 # by lid
	ibportstate -G 0x2C9000100D051 1 enable # by guid
	ibportstate -D 0 1                      # by direct route

19. ibcheckwidth

Description:
ibcheckwidth uses a full topology file that was created by ibnetdiscover,
scans the network to validate the active link widths and reports any 1x
links.

Syntax:
ibcheckwidth [-h] [<topology-file>]

Dependencies:
ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckportwidth

20. ibcheckportwidth

Description:
Check connectivity and check the specified port for 1x link width.
Port address is lid unless -G option is used to specify a GUID address.

Syntax:
ibcheckportwidth [-h] [-G] <lid|guid> <port>

Example:
        ibcheckportwidth 2 3         # check lid 2 port 3

Dependencies:
smpquery, smpquery output format, ibaddr

21. ibcheckstate

Description:
ibcheckstate uses a full topology file that was created by ibnetdiscover,
scans the network to validate the port state and port physical state,
and reports any ports which have a port state other than Active or
a port physical state other than LinkUp.

Syntax:
ibcheckstate [-h] [<topology-file>]

Dependencies:
ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckportstate

22. ibcheckportstate

Description:
Check connectivity and check the specified port for proper port state
(Active) and port physical state (LinkUp).
Port address is lid unless -G option is used to specify a GUID address.

yntax:
ibcheckportstate [-h] [-G] <lid|guid> <port_number>

Example:
        ibcheckportstate 2 3         # check lid 2 port 3

Dependencies:
smpquery, smpquery output format, ibaddr

23. ibcheckerrors

ibcheckerrors uses a full topology file that was created by ibnetdiscover,
scans the network to validate the connectivity and reports errors
(from port counters).

Syntax:
ibnetcheckerrors [-h] [<topology-file>]

Dependencies:
ibnetdiscover, ibnetdiscover format, ibchecknode, ibcheckport, ibcheckerrs

24. ibdiscover.pl

ibdiscover.pl uses a topology file create by ibnetdiscover and a discover.map
file which the network administrator creates which indicates the nodes
to be expected and a ibdiscover.topo file which is the expected connectivity
and produces a new connectivity file (discover.topo.new) and outputs
the changes to stdout. The network administrator can choose to replace
the "old" topo file with the new one or certain changes in.

The syntax of the ibdiscover.map file is:
<nodeGUID>|port|"Text for node"|<NodeDescription from ibnetdiscover format>
e.g.
8f10400410015|8|"ISR 6000"|# SW-6IB4 Voltaire port 0 lid 5
8f10403960558|2|"HCA 1"|# MT23108 InfiniHost Mellanox Technologies

The syntax of the old and new topo files (ibdiscover.topo and
ibdiscover.topo.new) are:
<LocalPort>|<LocalNodeGUID>|<RemotePort>|<RemoteNodeGUID>
e.g.
10|5442ba00003080|1|8f10400410015

These topo files are produced by the ibdiscover.pl tool.

Syntax:
ibnetdiscover | ibdiscover.pl

Dependencies:
ibnetdiscover, ibnetdiscover format

25. ibnodes

Description:
ibnodes either walks the IB subnet topology or uses an already saved topology
file and extracts the IB nodes (CAs and switches).

Syntax:
ibnodes [<topology-file>]

Dependencies:
ibnetdiscover, ibnetdiscover format

26. ibclearerrors

Description:
ibclearerrors clears the PMA error counters in PortCounters by either walking
the IB subnet topology or using an already saved topology file.

Syntax:
ibclearerrors [-h] [<topology-file>]

Dependencies:
ibnetdiscover, ibnetdiscover format, perfquery

27. ibclearcounters

Description:
ibclearcounters clears the PMA port counters by either walking
the IB subnet topology or using an already saved topology file.

Syntax:
ibclearcounters [-h] [<topology-file>]

Dependencies:
ibnetdiscover, ibnetdiscover format, perfquery

28. saquery

Description:
Issue some SA queries.

Syntax:
Usage: saquery [-h -d -P -N -L -G -s -g][<name>]
   Queries node records by default
   -d enable debugging
   -P get PathRecord info
   -N get NodeRecord info
   -L Return just the Lid of the name specified
   -G Return just the Guid of the name specified
   -s Return the PortInfoRecords with isSM capability mask bit on
   -g get multicast group info

Dependencies:
OpenSM libvendor, OpenSM libopensm, libibumad

29. ibsysstat

Description:
ibsysstat  uses  vendor  mads to validate connectivity between IB nodes
and obtain other information about the IB node.  ibsysstat  is  run  as
client/server. Default is to run as client.

Syntax:
ibsysstat [options] <dest lid|guid> [<op>]

Non standard flags:
       Current supported operations:
               ping - verify connectivity to server (default)
               host - obtain host information from server
               cpu  - obtain cpu information from server
        -o <oui>        use specified OUI number to multiplex vendor mads
        -S              start in server mode (do not return)