1Introduction 2============ 3 4libibverbs is a library that allows programs to use RDMA "verbs" for 5direct access to RDMA (currently InfiniBand and iWARP) hardware from 6userspace. For more information on RDMA verbs, see the InfiniBand 7Architecture Specification vol. 1, especially chapter 11, and the RDMA 8Consortium's RDMA Protocol Verbs Specification. 9 10Using libibverbs 11================ 12 13Device nodes 14------------ 15 16The verbs library expects special character device files named 17/dev/infiniband/uverbsN to be created. When you load the kernel 18modules, including both the low-level driver for your IB hardware as 19well as the ib_uverbs module, you should see one or more uverbsN 20entries in /sys/class/infiniband_verbs in addition to the 21/dev/infiniband/uverbsN character device files. 22 23To create the appropriate character device files automatically with 24udev, a rule like 25 26 KERNEL="uverbs*", NAME="infiniband/%k" 27 28can be used. This will create device nodes named 29 30 /dev/infiniband/uverbs0 31 32and so on. Since the RDMA userspace verbs should be safe for use by 33non-privileged users, you may want to add an appropriate MODE or GROUP 34to your udev rule. 35 36Permissions 37----------- 38 39To use IB verbs from userspace, a process must be able to access the 40appropriate /dev/infiniband/uverbsN special device file. You can 41check the permissions on this file with the command 42 43 ls -l /dev/infiniband/uverbs* 44 45Make sure that the permissions on these files are such that the 46user/group that your verbs program runs as can access the device file. 47 48To use IB verbs from userspace, a process must also have permission to 49tell the kernel to lock sufficient memory for all of your registered 50memory regions as well as the memory used internally by IB resources 51such as queue pairs (QPs) and completion queues (CQs). To check your 52resource limits, use the command 53 54 ulimit -l 55 56(or "limit memorylocked" for csh-like shells). 57 58If you see a small number such as 32 (the units are KB) then you will 59need to increase this limit. This is usually done for ordinary users 60via the file /etc/security/limits.conf. More configuration may be 61necessary if you are logging in via OpenSSH and your sshd is 62configured to use privilege separation. 63 64Valgrind support 65---------------- 66 67When running applications that use libibverbs under the Valgrind 68memory-checking debugger, Valgrind will falsely report "read from 69uninitialized" for memory that was initialized by the kernel drivers. 70Specifically, Valgrind cannot see when kernel drivers write to 71userspace memory, so when the process reads from that memory, Valgrind 72incorrectly assumes that the memory contents are uninitialized, and 73therefore raises a warning. 74 75libibverbs can be built with specific support for the Valgrind 76memory-checking debugger by specifying the --with-valgrind command 77line argument to configure. This flag enables code in libibverbs to 78tell Valgrind "this memory may look uninitialized, but it's really 79OK," which therefore suppresses the incorrect "read from 80uninitialized" warnings. This code adds trivial overhead to the 81critical performance path, so it is disabled by default. The intent 82is that production users can use a "normal" build of libibverbs and 83developers can use the "valgrind debug" build by simply switching 84their LD_LIBRARY_PATH environment variables. 85 86Libibverbs needs some header files from Valgrind in order to compile 87this support; it is important to use the header files from the same 88version of Valgrind that will be used at run time. You may need to 89specify the directory where Valgrind's header files are installed as 90an argument to --with-valgrind. For example 91 92 ./configure --with-valgrind=/opt/valgrind 93 94will make the libibverbs build look for valgrind headers in 95/opt/valgrind/include 96 97Reporting bugs 98============== 99 100Bugs should be reported to the OpenFabrics mailing list 101<general@lists.openfabrics.org>. In your bug report, please include: 102 103 * Information about your system: 104 - Linux distribution and version 105 - Linux kernel and version 106 - InfiniBand/iWARP hardware and firmware version 107 - ... any other relevant information 108 109 * How to reproduce the bug. Command line arguments for a libibverbs 110 example program or source code that other developers can 111 compile and run is most convenient. 112 113 * If the bug is a crash, the exact output printed out when the crash 114 occurred, including any kernel messages produced. 115 116 * If a verbs call is mysteriously returning an error or failing, the 117 output of "strace -ewrite -ewrite=all <command>". 118 119Submitting patches 120================== 121 122Patches should also be submitted to the OpenFabrics mailing list 123<general@lists.openfabrics.org>. Please use unified diff form (the -u 124option to GNU diff), and include a good description of what your patch 125does and why it should be applied. If your patch fixes a bug, please 126make sure to describe the bug and how your fix works. 127 128Please include a change to the ChangeLog file (in standard GNU 129changelog format) as part of your patch. 130 131Make sure that your contribution can be licensed under the same 132license as the original code you are patching, and that you have all 133necessary permissions to release your work. 134 135TODO 136==== 137 1381.1 series 139---------- 140 141The libibverbs API and ABI are frozen for all releases in the 1.1 142series. Methods were added to struct ibv_context to implement the 143following features, so it should be possible to add them in a future 144release in the 1.1 series: 145 146 * Memory window (MW) support. 147 148 * Implement the reregister memory region (MR) verb. We will add an 149 extension to the IB spec to allow the application to indicate that 150 the region is only being extended, and that operations in progress 151 should _not_ fail (contrary to the IB spec, which states that 152 reregister must be implemented so that it behaves equivalently to a 153 deregister followed by a register). 154 155Other possibilities 156------------------- 157 158There are no plans to implement the following features, which would be 159needed for completeness but don't seem particularly useful. However, 160if there is demand from application developers or an implementation is 161contributed, then the feature may be added. 162 163 * Implement the query address handle (AH) verb. 164 * Implement the query memory region (MR) verb. 165