1tcpp -- Parallel TCP Exercise Tool 2 3This is a new tool, and is rife with bugs. However, it appears to create 4even more problems for device drivers and the kernel, so that's OK. 5 6This tool generates large numbers of TCP connections and stuffs lots of data 7into them. One binary encapsulates both a client and a server. Each of the 8client and the server generates a certain number of worker processes, each of 9which in turn uses its own TCP port. The number of server processes must be 10>= the number of client processes, or some of the ports required by the 11client won't have a listener. The client then proceeds to make connections 12and send data to the server. Each worker multiplexes many connections at 13once, up to a maximum parallelism limit. The client can use one or many IP 14addresses, in order to make more 4-tuples available for testing, and will 15automatically spread the load of new connections across available source 16addresses. 17 18You will need to retune your TCP stack for high volume, see Configuration 19Notes below. 20 21The server has very little to configure, use the following command line 22flags: 23 24 -s Select server mode 25 -p <numprocs> Number of workers, should be >= client -p arg 26 -r <baseport> Non-default base TCP port, should match client 27 -T Print CPU usage every ten seconds 28 -m <maxconnectionsperproc> Maximum simultaneous connections/proc, should 29 be >= client setting. 30 31Typical use: 32 33 ./tcpp -s -p 4 -m 1000000 34 35This selects server mode, four workers, and at most 1 million TCP connections 36per worker at a time. 37 38The client has more to configure, with the following flags: 39 40 -c <remoteIP> Select client mode, and specific dest IP 41 -C Print connections/second instead of GBps 42 -P Pin each worker to a CPU 43 -M <localIPcount> Number of sequential local IPs to use; req. -l 44 -T Include CPU use summary in stats at end of run 45 -b <bytespertcp> Data bytes per connection 46 -l <localIPbase> Starting local IP address to bind 47 -m <maxtcpsatonce> Max simultaneous conn/worker (see server -m) 48 -p <numprocs> Number of workers, should be <= server -p 49 -r <baseport> Non-default base TCP port, should match server 50 -t <tcpsperproc> How many connections to use per worker 51 52Typical use: 53 54 ./tcpp -c 192.168.100.201 -p 4 -t 100000 -m 10000 -b 100000 \ 55 -l 192.168.100.101 -M 4 56 57This creates four workers, each of which will (over its lifetime) set up and 58use 100,000 TCP connections carrying 100K of data, up to 10,000 simultaneous 59connection at any given moment. tcpp will use four source IP addresses, 60starting with 192.168.100.101, and all connections will be to the single 61destination IP of 192.168.100.201. 62 63Having (p) <= the number of cores is advisable. When multiple IPs are used 64on the client, they will be sequential starting with the localIPbase set with 65-l. 66 67Known Issues 68------------ 69 70The bandwidth estimate doesn't handle failures well. It also has serious 71rounding errors and probably conceptual problems. 72 73It's not clear that kevent() is "fair" to multiple connections. 74 75Rather than passing the length for each connection, we might want to pass 76it once with a control connection up front. On the other hand, the server 77is quite dumb right now, so we could take advantage of this to do size 78mixes. 79 80Configuration Notes 81------------------- 82 83In my testing, I use loader.conf entries of: 84 85kern.ipc.maxsockets=1000000 86net.inet.tcp.maxtcptw=3000000 87kern.ipc.somaxconn=49152 88kern.ipc.nmbjumbo16=262144 89kern.ipc.nmbjumbo9=262144 90kern.ipc.nmbjumbop=262144 91kern.ipc.nmbclusters=262144 92net.inet.tcp.syncache.cachelimit=65536 93net.inet.tcp.syncache.bucketlimit=512 94 95# May be useful if you can't use multiple IP addresses 96net.inet.ip.portrange.first=100 97 98# For running !multiq, do this before loading the driver: 99kenv hw.cxgb.singleq="1" 100 101kldload if_cxgb 102 103# Consider turning off TSO and/or adjusting the MTU for some scenarios: 104ifconfig cxgb0 -tso 105ifconfig cxgb0 mtu 1500 106 107 108$FreeBSD: releng/10.2/tools/tools/netrate/tcpp/README 209428 2010-06-22 10:45:35Z rwatson $ 109