1<?xml version="1.0" encoding="UTF-8"?> 2<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN" 3 "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []> 4 5<book id="conntrack-tools-how-to"> 6 <bookinfo> 7 <title>The conntrack-tools user manual</title> 8 9 <authorgroup> 10 <author> 11 <firstname>Pablo</firstname> 12 <surname>Neira Ayuso</surname> 13 <affiliation> 14 <address> 15 <email>pablo@netfilter.org</email> 16 </address> 17 </affiliation> 18 </author> 19 </authorgroup> 20 21 <copyright> 22 <year>2008-2012</year> 23 <holder>Pablo Neira Ayuso</holder> 24 </copyright> 25 26 <legalnotice> 27 <para> 28 Permission is granted to copy, distribute and/or modify this document 29 under the terms of the GNU Free Documentation License, Version 1.2 30 or any later version published by the Free Software Foundation; 31 with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. 32 A copy of the license is included in the section entitled "GNU 33 Free Documentation License". 34 </para> 35 </legalnotice> 36 37 <releaseinfo> 38 This document details how to install and configure the 39 <ulink url="http://conntrack-tools.netfilter.org">conntrack-tools</ulink> 40 >= 1.4.0. This document will evolve in the future to cover new features 41 and changes.</releaseinfo> 42 43 </bookinfo> 44 45 <toc></toc> 46 47 <chapter id="introduction"><title>Introduction</title> 48 49 <para>This document should be a kick-off point to install and configure the 50 <ulink url="http://conntrack-tools.netfilter.org">conntrack-tools</ulink>. 51 If you find any error or imprecision in this document, please send an email 52 to the author, it will be appreciated.</para> 53 54 <para>In this document, the author assumes that the reader is familiar with firewalling concepts and iptables in general. If this is not your case, I suggest you to read the iptables documentation before going ahead. Moreover, the reader must also understand the difference between <emphasis>stateful</emphasis> and <emphasis>stateless</emphasis> firewalls. If this is not your case, I strongly suggest you to read the article <ulink url="http://people.netfilter.org/pablo/docs/login.pdf">Netfilter's Connection Tracking System</ulink> published in <emphasis>:login; the USENIX magazine</emphasis>. That document contains a general description that should help to clarify the concepts.</para> 55 56<para>If you do not fulfill the previous requirements, this documentation is likely to be a source of frustration. Probably, you wonder why I'm insisting on these prerequisites too much, the fact is that if your iptables rule-set is <emphasis>stateless</emphasis>, it is very likely that the <emphasis>conntrack-tools</emphasis> will not be of any help for you. You have been warned!</para> 57 58 </chapter> 59 <chapter id="what"><title>What are the conntrack-tools?</title> 60 61 <para>The conntrack-tools are a set of free software tools for GNU/Linux that allow system administrators interact, from user-space, with the in-kernel <ulink url="http://people.netfilter.org/pablo/docs/login.pdf">Connection Tracking System</ulink>, which is the module that enables stateful packet inspection for iptables. Probably, you did not hear about this module so far. However, if any of the rules of your rule-set use the <emphasis>state</emphasis> or <emphasis>ctstate</emphasis> iptables matches, you are indeed using it. 62 63 </para> 64 65<para>The <ulink url="http://conntrack-tools.netfilter.org">conntrack-tools</ulink> package contains two programs:</para> 66 67 <itemizedlist> 68 <listitem> 69 <para><emphasis>conntrack</emphasis> is command line interface conntrack provides a more flexible interface to the connnection tracking system than /proc/net/ip_conntrack. With conntrack, you can show, delete and update the existing state entries; and you can also listen to flow events.</para> 70 </listitem> 71 <listitem> 72 <para><emphasis>conntrackd</emphasis> is the user-space connection tracking daemon. This daemon can be used to deploy fault-tolerant GNU/Linux firewalls but you can also use it to collect flow-based statistics of the firewall use.</para> 73 </listitem> 74 </itemizedlist> 75 76 <para>Although the name of both tools is very similar - and you can blame me for that, I'm not a marketing guy - they are used for very different tasks.</para> 77 78 </chapter> 79 80 <chapter id="requirements"><title>Requirements</title> 81 82 <para>You have to install the following software in order to get the <emphasis>conntrack-tools</emphasis> working. Make sure that you have installed them correctly before going ahead:</para> 83 84 <itemizedlist> 85 <listitem> 86 <para><ulink url="http://www.kernel.org">Linux kernel</ulink> version >= 2.6.18 that, at least, has support for:</para> 87 <itemizedlist> 88 <listitem> 89 <para>Connection Tracking System.</para> 90 <itemizedlist> 91 <listitem> 92 <para>CONFIG_NF_CONNTRACK=m</para> 93 </listitem> 94 <listitem> 95 <para>CONFIG_NF_CONNTRACK_IPV4=m</para> 96 </listitem> 97 <listitem> 98 <para>CONFIG_NF_CONNTRACK_IPV6=m (if your setup supports IPv6)</para> 99 </listitem> 100 </itemizedlist> 101 </listitem> 102 <listitem> 103 <para>nfnetlink: the generic messaging interface for Netfilter.</para> 104 <itemizedlist> 105 <listitem> 106 <para>CONFIG_NETFILTER_NETLINK=m</para> 107 </listitem> 108 </itemizedlist> 109 </listitem> 110 <listitem> 111 <para>nf_conntrack_netlink: the messaging interface for the Connection Tracking System.</para> 112 <itemizedlist> 113 <listitem> 114 <para>CONFIG_NF_CT_NETLINK=m</para> 115 </listitem> 116 </itemizedlist> 117 </listitem> 118 <listitem> 119 <para>connection tracking event notification API: the flow-based event notification interface.</para> 120 <itemizedlist> 121 <listitem> 122 <para>CONFIG_NF_CONNTRACK_EVENTS=y</para> 123 </listitem> 124 </itemizedlist> 125 </listitem> 126 </itemizedlist> 127 <note><title>Verifying kernel support</title> 128 <para> 129 Make sure you have loaded <emphasis>nf_conntrack</emphasis>, <emphasis>nf_conntrack_ipv4</emphasis> (if your setup also supports IPv6, <emphasis>nf_conntrack_ipv6</emphasis>) and <emphasis>nf_conntrack_netlink</emphasis>. 130 </para> 131 </note> 132 </listitem> 133 <listitem> 134 <para>libnfnetlink: the netfilter netlink library use the official release available in <ulink url="http://www.netfilter.org">netfilter.org</ulink></para> 135 </listitem> 136 <listitem> 137 <para>libnetfilter_conntrack: the netfilter netlink library use the official release available in <ulink url="http://www.netfilter.org">netfilter.org</ulink></para> 138 </listitem> 139 </itemizedlist> 140 </chapter> 141 142 <chapter id="Installation"><title>Installation</title> 143 144 <para>To compile and install the <emphasis>conntrack-tools</emphasis> run the following commands:</para> 145 <programlisting> 146 (non-root)$ tar xvjf conntrack-tools-x.x.x.tar.bz2 147 (non-root)$ cd conntrack-tools-x.x.x 148 (non-root)$ ./configure --prefix=/usr 149 (non-root)$ make 150 (root) # make install</programlisting> 151 152<note><title>Fedora Users</title> 153 <para>If you are installing the libraries in /usr/local/, do not forget to do the following things:</para> 154 <itemizedlist> 155 <listitem><para>PKG_CONFIG_PATH=/usr/local/lib/pkgconfig; export PKG_CONFIG_PATH</para></listitem> 156 <listitem><para>Add `/usr/local/lib' to your /etc/ld.so.conf file and run `ldconfig'</para></listitem> 157 </itemizedlist> 158 <para>Check `ldd' for trouble-shooting, read <ulink url="http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html">this</ulink> for more information on how libraries work.</para> 159</note> 160 161<note><title>Verifying kernel support</title> 162 <para>To check that the modules are enabled in the kernel, run <emphasis>`conntrack -E'</emphasis> and generate traffic, you should see flow events reporting new connections and updates. 163 </para> 164</note> 165 166 </chapter> 167 168 <chapter id="conntrack"><title>Using conntrack: the command line interface</title> 169 170 <para>The <emphasis>/proc/net/ip_conntrack</emphasis> interface is very limited as it only allows you to display the existing flows, their state and other information:</para> 171 172 <programlisting> 173 # cat /proc/net/ip_conntrack 174 tcp 6 431982 ESTABLISHED src=192.168.2.100 dst=123.59.27.117 sport=34846 dport=993 packets=169 bytes=14322 src=123.59.27.117 dst=192.168.2.100 sport=993 dport=34846 packets=113 bytes=34787 [ASSURED] mark=0 secmark=0 use=1 175 tcp 6 431698 ESTABLISHED src=192.168.2.100 dst=123.59.27.117 sport=34849 dport=993 packets=244 bytes=18723 src=123.59.27.117 dst=192.168.2.100 sport=993 dport=34849 packets=203 bytes=144731 [ASSURED] mark=0 secmark=0 use=1 176 </programlisting> 177 178<para>The command line tool <emphasis>conntrack</emphasis> can be used to display the same information:</para> 179 <programlisting> 180 # conntrack -L 181 tcp 6 431982 ESTABLISHED src=192.168.2.100 dst=123.59.27.117 sport=34846 dport=993 packets=169 bytes=14322 src=123.59.27.117 dst=192.168.2.100 sport=993 dport=34846 packets=113 bytes=34787 [ASSURED] mark=0 secmark=0 use=1 182 tcp 6 431698 ESTABLISHED src=192.168.2.100 dst=123.59.27.117 sport=34849 dport=993 packets=244 bytes=18723 src=123.59.27.117 dst=192.168.2.100 sport=993 dport=34849 packets=203 bytes=144731 [ASSURED] mark=0 secmark=0 use=1 183conntrack v0.9.7 (conntrack-tools): 2 flow entries have been shown. 184 </programlisting> 185 186<para>You can natively filter the output without using <emphasis>grep</emphasis>:</para> 187<programlisting> 188 # conntrack -L -p tcp --dport 34856 189 tcp 6 431982 ESTABLISHED src=192.168.2.100 dst=123.59.27.117 sport=34846 dport=993 packets=169 bytes=14322 src=123.59.27.117 dst=192.168.2.100 sport=993 dport=34846 packets=113 bytes=34787 [ASSURED] mark=0 secmark=0 use=1 190conntrack v0.9.7 (conntrack-tools): 1 flow entries have been shown. 191 </programlisting> 192 193<para>Update the mark based on a selection, this allows you to change the mark of an entry without using the CONNMARK target:</para> 194<programlisting> 195 # conntrack -U -p tcp --dport 3486 --mark 10 196 tcp 6 431982 ESTABLISHED src=192.168.2.100 dst=123.59.27.117 sport=34846 dport=993 packets=169 bytes=14322 src=123.59.27.117 dst=192.168.2.100 sport=993 dport=34846 packets=113 bytes=34787 [ASSURED] mark=1 secmark=0 use=1 197conntrack v0.9.7 (conntrack-tools): 1 flow entries has been updated. 198 </programlisting> 199 200<para>Delete one entry, this can be used to block traffic if:</para> 201<itemizedlist> 202 <listitem><para>You have a stateful rule-set that blocks traffic in INVALID state.</para></listitem> 203 <listitem><para>You have set <emphasis>/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_loose</emphasis> or <emphasis>/proc/sys/net/netfilter/nf_conntrack_tcp_loose</emphasis>, depending on your kernel version, to zero.</para></listitem> 204</itemizedlist> 205 206<programlisting> 207 # conntrack -D -p tcp --dport 3486 208 tcp 6 431982 ESTABLISHED src=192.168.2.100 dst=123.59.27.117 sport=34846 dport=993 packets=169 bytes=14322 src=123.59.27.117 dst=192.168.2.100 sport=993 dport=34846 packets=113 bytes=34787 [ASSURED] mark=1 secmark=0 use=1 209conntrack v0.9.7 (conntrack-tools): 1 flow entries has been deleted. 210 </programlisting> 211 212<para>Display the connection tracking events:</para> 213<programlisting> 214 # conntrack -E 215 [NEW] udp 17 30 src=192.168.2.100 dst=192.168.2.1 sport=57767 dport=53 [UNREPLIED] src=192.168.2.1 dst=192.168.2.100 sport=53 dport=57767 216 [UPDATE] udp 17 29 src=192.168.2.100 dst=192.168.2.1 sport=57767 dport=53 src=192.168.2.1 dst=192.168.2.100 sport=53 dport=57767 217 [NEW] tcp 6 120 SYN_SENT src=192.168.2.100 dst=66.102.9.104 sport=33379 dport=80 [UNREPLIED] src=66.102.9.104 dst=192.168.2.100 sport=80 dport=33379 218 [UPDATE] tcp 6 60 SYN_RECV src=192.168.2.100 dst=66.102.9.104 sport=33379 dport=80 src=66.102.9.104 dst=192.168.2.100 sport=80 dport=33379 219 [UPDATE] tcp 6 432000 ESTABLISHED src=192.168.2.100 dst=66.102.9.104 sport=33379 dport=80 src=66.102.9.104 dst=192.168.2.100 sport=80 dport=33379 [ASSURED] 220</programlisting> 221 222<para>You can also display the existing flows in XML format, filter the output based on the NAT handling applied, etc.</para> 223 224</chapter> 225 226 <chapter id="settingup"><title>Setting up conntrackd: the daemon</title> 227 228 <para>The daemon <emphasis>conntrackd</emphasis> supports two working modes:</para> 229 230 <itemizedlist> 231 <listitem> 232 <para><emphasis>State table synchronization</emphasis>: the daemon can be used to synchronize the connection tracking state table between several firewall replicas. This can be used to deploy fault-tolerant stateful firewalls. This is the main feature of the daemon.</para> 233 </listitem> 234 <listitem> 235 <para><emphasis>Flow-based statistics collection</emphasis>: the daemon can be used to collect flow-based statistics. This feature is similar to what <ulink url="http://www.netfilter.org/projects/ulogd/">ulogd-2.x</ulink> provides.</para> 236 </listitem> 237 </itemizedlist> 238 239 <sect1 id="sync"><title>State table synchronization</title> 240 241 <sect2 id="sync-requirements"><title>Requirements</title> 242 243 <para>In order to get <emphasis>conntrackd</emphasis> working in synchronization mode, you have to fulfill the following requirements:</para> 244 245 <orderedlist> 246 <listitem> 247 <para>A <emphasis>high availability manager</emphasis> like <ulink url="http://www.keepalived.org">keepalived</ulink> that manages the virtual IPs of the 248 firewall cluster, detects errors, and decide when to migrate the virtual IPs 249 from one firewall replica to another. Without it, <emphasis>conntrackd</emphasis> will not work appropriately.</para> 250 251 <para>The state synchronization setup requires a working installation of <ulink url="http://www.keepalived.org">keepalived</ulink>, preferibly a recent version. Check if your distribution comes with a recent packaged version. Otherwise, you may compile it from the sources. 252 </para> 253 254 <para> 255 There is a very simple example file in the <emphasis>conntrackd</emphasis> 256 sources to setup a simple HA cluster with keepalived (see the file 257 keepalived.conf under the doc/sync/ directory). This file can be used to 258 set up a simple VRRP cluster composed of two machines that hold the virtual 259 IPs 192.168.0.100 on eth0 and 192.168.1.100 on eth1.</para> 260 261 <para>If you are not familiar with <emphasis>keepalived</emphasis>, please 262 read the official documentation available at the keepalived website 263 (<ulink url="http://www.keepalived.org">http://www.keepalived.org</ulink>).</para> 264 265<para>If you use a different high availability manager, make sure it works correctly before going ahead.</para> 266 267 </listitem> 268 269 <listitem> 270 <para>A dedicated link. The dedicated link between the firewalls is used 271 to transmit and receive the state information. The use of a dedicated link 272 is mandatory for security reasons as someone may pick the state information 273 that is transfered between the firewalls.</para> 274 </listitem> 275 276 <listitem> 277 <para>A well-formed stateful rule-set. Otherwise you are likely to experience 278 problems during the fail-over. An example of a well-formed stateful iptables 279 rule-set is available in the <ulink url="http://conntrack-tools.netfilter.org/testcase.html">conntrack-tools website</ulink>.</para> 280 </listitem> 281 282 <listitem> 283 <para>If your Linux kernel is < 2.6.22, you have to disable TCP window 284 tracking: 285 <programlisting> 286 # echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal 287 </programlisting> 288 </para> 289 </listitem> 290 291 </orderedlist> 292 293 </sect2> 294 295 <sect2 id="sync-configure"><title>Configuring the daemon</title> 296 297 <para>The daemon <emphasis>conntrackd</emphasis> in synchronization mode 298 supports up to three replication approaches:</para> 299 300 <itemizedlist> 301 <listitem> 302 <para><emphasis>notrack</emphasis>: this approach is the most simple as 303 it is based on a best effort replication protocol, ie. unreliable 304 protocol. This protocol sends and receives the state information 305 without performing any specific checking. 306 </para> 307 </listitem> 308 <listitem> 309 <para><emphasis>ft-fw</emphasis>: this approach is based on a reliable 310 protocol that performs message tracking. Thus, the protocol can recover 311 from message loss, re-ordering and corruption.</para> 312 </listitem> 313 <listitem> 314 <para><emphasis>alarm</emphasis>: this approach is spamming. It is based 315 on a alarm-based protocol that periodically re-sends the flow state to 316 the backup firewall replicas. This protocol consumes a lot of bandwidth 317 but it resolves synchronization problems fast.</para> 318 </listitem> 319 </itemizedlist> 320 321 <para>The three existing approaches are soft real-time asynchronous 322 replication protocols that are aimed to have negligible impact in terms 323 of latency and bandwidth throughput in the stateful firewall filtering.</para> 324 325 <para>To configure <emphasis>conntrackd</emphasis> in any of the existing 326 synchronization modes, you have to copy the example configuration file to 327 the directory /etc/conntrackd/ on every firewall replica. Note that 328 <emphasis>_type_</emphasis> is the synchronization type selected.</para> 329 330<programlisting> 331 (conntrack-tools-x.x.x)# cp doc/_type_/conntrackd.conf /etc/conntrackd/conntrackd.conf 332</programlisting> 333 334<para> 335 Do not forget to edit the files before going ahead. There are several 336 parameters that you have to tune to adapt the example configuration file 337 to your setup. 338</para> 339 340<note><title>Configuration file location</title> 341 <para>If you don't want to put the config file under /etc/conntrackd/, just tell conntrackd where to find it passing the option -C.</para> 342</note> 343 344</sect2> 345 346<sect2 id="sync-pb"><title>Active-Backup setup</title> 347 348 <note><title>Stateful firewall architectures</title> 349 <para>A good reading to extend the information about firewall architectures is <ulink url="http://1984.lsi.us.es/~pablo/docs/intcomp09.pdf">Demystifying cluster-based fault-tolerant firewalls</ulink> published in IEEE Internet Computing magazine. 350 </para> 351 </note> 352 353 <para>In the Active-Backup setup, one of the stateful firewall replicas 354 filters traffic and the other acts as backup. If you use this approach, 355 you have to copy the script <emphasis>primary-backup.sh</emphasis> to: 356 </para> 357 358<programlisting> 359 (conntrack-tools-x.x.x)# cp doc/sync/primary-backup.sh /etc/conntrackd/ 360</programlisting> 361 362 <para>The HA manager invokes this script when a transition happens, ie. If 363 a stateful firewall replica:</para> 364 365 <itemizedlist> 366 <listitem> 367 <para>becomes active to recover the filtering.</para> 368 </listitem> 369 <listitem> 370 <para>becomes backup.</para> 371 </listitem> 372 <listitem> 373 <para>hits failure (this is available if the HA manager has a failure state, which is true for <ulink url="http://www.keepalived.org">keepalived</ulink>.</para> 374 </listitem> 375 </itemizedlist> 376 377 <para>The script is simple, and it contains the different actions that 378 <emphasis>conntrackd</emphasis> performs to recover the filtering or 379 purge obsolete entries from the state table, among others. The script is 380 commented, you can have a look at it if you need further information.</para> 381 382</sect2> 383 384<sect2 id="sync-aa"><title>Active-Active setup</title> 385 386 <para>The Active-Active setup consists of having more than one stateful 387 firewall replicas actively filtering traffic. Thus, we reduce the resource 388 waste that implies to have a backup firewall which does nothing.</para> 389 390 <para>We can classify the type of Active-Active setups in several 391 families:</para> 392 393 <itemizedlist> 394 <listitem> 395 <para><emphasis>Symmetric path routing</emphasis>: The stateful firewall 396 replicas share the workload in terms of flows, ie. the packets that are 397 part of a flow are always filtered by the same firewall.</para> 398 </listitem> 399 <listitem> 400 <para><emphasis>Asymmetric multi-path routing</emphasis>: The packets that 401 are part of a flow can be filtered by whatever stateful firewall in the 402 cluster. Thus, every flow-states have to be propagated to all the firewalls 403 in the cluster as we do not know which one would be the next to filter a 404 packet. This setup goes against the design of stateful firewalls as we 405 define the filtering policy based on flows, not in packets anymore. 406 </para> 407 </listitem> 408 </itemizedlist> 409 410 <para>As for 0.9.8, the design of <emphasis>conntrackd</emphasis> allows you 411 to deploy an symmetric Active-Active setup based on a static approach. 412 For example, assume that you have two virtual IPs, vIP1 and vIP2, and two 413 firewall replicas, FW1 and FW2. You can give the virtual vIP1 to the 414 firewall FW1 and the vIP2 to the FW2. 415 </para> 416 417 <para>Unfortunately, you will have to wait for the support for the 418 Active-Active setup based on dynamic approach, ie. a workload sharing setup 419 without directors that allow the stateful firewall share the filtering.</para> 420 421 <para>On the other hand, the asymmetric scenario may work if your setup 422 fulfills several strong assumptions. However, in the opinion of the author 423 of this work, the asymmetric setup goes against the design of stateful 424 firewalls and <emphasis>conntrackd</emphasis>. Therefore, you have two 425 choices here: you can deploy an Active-Backup setup or go back to your 426 old stateless rule-set (in that case, the conntrack-tools will not be 427 of any help anymore, of course).</para> 428 429</sect2> 430 431<sect2 id="sync-launch"><title>Launching conntrackd</title> 432 433 <para> 434 Once you have configured <emphasis>conntrackd</emphasis>, you can run in 435 <emphasis>console mode</emphasis> which is an interactive mode, in that case 436 type 'conntrackd' as root.</para> 437 438 <programlisting>(root)# conntrackd</programlisting> 439 440 <para>If you want to run <emphasis>conntrackd</emphasis> in <emphasis>daemon 441 mode</emphasis>, then type:</para> 442 443 <programlisting>(root)# conntrackd -d</programlisting> 444 445 <para>You can verify that conntrackd is running by checking the log messages 446 via <emphasis>ps</emphasis>. Moreover, if <emphasis>conntrackd</emphasis> is 447 running fine, you can dump the current status of the daemon:</para> 448 449 <programlisting> 450 # conntrackd -s 451 cache internal: 452 current active connections: 4 453 connections created: 4 failed: 0 454 connections updated: 0 failed: 0 455 connections destroyed: 0 failed: 0 456 457 cache external: 458 current active connections: 0 459 connections created: 0 failed: 0 460 connections updated: 0 failed: 0 461 connections destroyed: 0 failed: 0 462 463 traffic processed: 464 0 Bytes 0 Pckts 465 466 multicast traffic: 467 352 Bytes sent 0 Bytes recv 468 22 Pckts sent 0 Pckts recv 469 0 Error send 0 Error recv 470 471 multicast sequence tracking: 472 0 Pckts mfrm 0 Pckts lost 473 </programlisting> 474 475 <para>This command displays the number of entries in the internal and 476 external cache:</para> 477 478 <itemizedlist> 479 <listitem> 480 <para>The internal cache contains the states that this firewall replica is filtering, ie. this is a cache of the kernel state table. 481 </para> 482 </listitem> 483 <listitem> 484 <para>The external cache contains the states that the other firewall replica is filtering. 485 </para> 486 </listitem> 487 </itemizedlist> 488 489 <para>You can dump the internal cache with the following command:</para> 490 491 <programlisting> 492 # conntrackd -i 493 tcp 6 ESTABLISHED src=192.168.2.100 dst=139.174.175.20 sport=58491 dport=993 src=139.174.175.20 dst=192.168.2.100 sport=993 dport=58491 [ASSURED] mark=0 secmark=0 [active since 536s] 494 tcp 6 ESTABLISHED src=192.168.2.100 dst=123.59.27.117 sport=38211 dport=993 src=123.59.27.117 dst=192.168.2.100 sport=993 dport=38211 [ASSURED] mark=0 secmark=0 [active since 536s] 495 tcp 6 ESTABLISHED src=192.168.2.100 dst=123.59.27.117 sport=38209 dport=993 src=123.59.27.117 dst=192.168.2.100 sport=993 dport=38209 [ASSURED] mark=0 secmark=0 [active since 536s] 496 tcp 6 TIME_WAIT src=192.168.2.100 dst=74.125.45.166 sport=42593 dport=80 src=74.125.45.166 dst=192.168.2.100 sport=80 dport=42593 [ASSURED] [active since 165s] 497 tcp 6 ESTABLISHED src=192.168.2.100 dst=139.174.175.20 sport=37962 dport=993 src=139.174.175.20 dst=192.168.2.100 sport=993 dport=37962 [ASSURED] mark=0 secmark=0 [active since 536s] 498 </programlisting> 499 500 <para>You can dump the external cache with the following command:</para> 501 502 <programlisting># conntrackd -e</programlisting> 503 504 <para>If the replication works fine, <emphasis>conntrackd -s</emphasis> 505 displays the active's internal cache should display the same number of 506 entries than the backup's external cache and vice-versa.</para> 507 508 <para>To verify that the recovery works fine, if you trigger a fail-over, 509 the log files should display the following information:</para> 510 511 <programlisting> 512 [Thu Sep 18 18:03:02 2008] (pid=9759) [notice] committing external cache 513 [Thu Sep 18 18:03:02 2008] (pid=9759) [notice] Committed 1545 new entries</programlisting> 514 515 <para>This means that the state entries have been injected into the kernel correctly.</para> 516 517</sect2> 518 519<sect2 id="sync-options"><title>Other configuration options</title> 520 521 <para>The daemon allows several configuration options that you may want to 522 enable. This section contains some information about them.</para> 523 524<sect3 id="sync-disable-external"><title>Disabling external cache</title> 525 526 <para>It is possible to disable the external cache. Thus, 527 <emphasis>conntrackd</emphasis> directly injects the flow-states into the 528 in-kernel Connection Tracking System of the backup firewall. You can do it 529 by enabling the <emphasis>DisableExternalCache</emphasis> option in the 530 <emphasis>conntrackd.conf</emphasis> configuration file: 531 </para> 532 533 <programlisting> 534Sync { 535 Mode FTFW { 536 [...] 537 DisableExternalCache Off 538 } 539} 540 </programlisting> 541 542 <para>You can also use this option with the NOTRACK and ALARM modes. This 543 increases CPU consumption in the backup firewall but now you do not need 544 to commit the flow-states during the master failures since they are already 545 in the in-kernel Connection Tracking table. Moreover, you save memory in 546 the backup firewall since you do not need to store the foreign flow-states 547 anymore. 548 </para> 549 550</sect3> 551 552<sect3 id="sync-disable-internal"><title>Disabling internal cache</title> 553 554 <para>You can also disable the internal cache by means of the 555 <emphasis>DisableInternalCache</emphasis> option in the 556 <emphasis>conntrackd.conf</emphasis> configuration file: 557 </para> 558 559 <programlisting> 560Sync { 561 Mode NOTRACK { 562 [...] 563 DisableInternalCache Off 564 } 565} 566 </programlisting> 567 568 <para>However, this option is only available for the NOTRACK mode. This 569 mode provides unreliable flow-state synchronization between firewalls. 570 Thus, if flow-states are lost during the synchronization, the protocol 571 provides no way to recover them.</para> 572 573</sect3> 574 575<sect3 id="sync-transport-protocol"> 576<title>Using UDP, TCP or multicast for flow-state synchronization</title> 577 578 <para>You can use up to three different transport layer protocols to 579 synchronize flow-state changes between the firewalls: UDP, TCP and 580 Multicast. UDP and multicast are unreliable but together with the FT-FW 581 mode provide partial reliable flow-state synchronization. 582 </para> 583 584 <para>The preferred choice is FT-FW over UDP, or multicast alternatively. 585 TCP introduces latency in the flow-state synchronization due to the 586 congestion control. Under flow-state message are lost, the FIFO delivery 587 becomes also a problem since the backup firewall quickly gets out of 588 sync. For that reason, its use is discouraged. Note that using TCP only 589 makes sense with the NOTRACK mode. 590 </para> 591 592</sect3> 593 594<sect3 id="sync-redundant-link"><title>Redundant dedicated links</title> 595 596 <para>You can set redundant dedicated links without using bonding, you have 597 to configure as many redundant links as you want in the configuration file. 598 In case of failure of the master dedicated link, conntrackd failovers to one 599 of the backups. An example of this configuration is the following: 600 </para> 601 602 <programlisting> 603Sync { 604 Mode FTFW { 605 [...] 606 } 607 # default master dedicated link 608 UDP Default { 609 IPv4_address 192.168.2.1 610 IPv4_Destination_Address 192.168.2.2 611 Port 3780 612 Interface eth3 613 SndSocketBuffer 24985600 614 RcvSocketBuffer 24985600 615 Checksum on 616 } 617 # backup dedicated link 618 UDP { 619 IPv4_address 192.168.1.3 620 IPv4_Destination_Address 192.168.1.4 621 Port 3780 622 Interface eth2 623 SndSocketBuffer 24985600 624 RcvSocketBuffer 24985600 625 Checksum on 626 } 627 [...] 628} 629 </programlisting> 630 631</sect3> 632 633<sect3 id="sync-iptables-filtering"> 634<title>Filtering Connection tracking events with iptables</title> 635 636 <para>Since Linux kernel >= 2.6.34, iptables provides the 637 <emphasis>CT</emphasis> iptables target that allows to reduce the 638 amount of Connection Tracking events that are delivered to user-space. 639 However, you will have to use a Linux kernel >= 2.6.38 to profit 640 from this feature, since several aspects of the event filtering were 641 broken.</para> 642 643 <para>The following example shows how to only generate the 644 <emphasis>assured</emphasis> and <emphasis>destroy</emphasis> 645 events:</para> 646 647 <programlisting> 648 # iptables -I PREROUTING -t raw -j CT --ctevents assured,destroy 649 </programlisting> 650 651 <note><title>Assured flows</title> 652 <para>One flow is assured if the firewall has seen traffic for it in 653 both directions.</para> 654 </note> 655 656 <para>Reducing the amount of events generated helps to reduce CPU 657 consumption in the active firewall.</para> 658 659</sect3> 660 661<sect3 id="sync-expect"><title>Synchronization of expectations</title> 662 663 <note><title>Check your Linux kernel version first</title> 664 <para> 665 The synchronization of expectations require a Linux kernel >= 3.5 666 to work appropriately. 667 </para> 668 </note> 669 670 <para>The connection tracking system provides helpers that allows you to 671 filter multi-flow application protocols like FTP, H.323 and SIP among many 672 others. These protocols usually split the control and data traffic in 673 different flows. Moreover, the control flow usually announces layer 3 and 674 4 information to let the other peer know where the data flows will be 675 open. This sort of protocols require that the firewall inspects the 676 content of the packet, otherwise filtering by layer 3 and 4 selectors 677 like addresses and ports become a real nightmare. Netfilter already 678 provides the so-called <emphasis>helpers</emphasis> that track this 679 protocol aspects to allow deploying appropriate filtering. These 680 helpers create <emphasis>expectation</emphasis> entries that 681 represent expected traffic that will arrive to the firewall according 682 to the inspected packets.</para> 683 684 <para>In case that you have enabled tracking of these protocols, you 685 may want to enable the state-synchronization of expectation as well. 686 Thus, established flows for this specific protocols will not suffer 687 any disruption.</para> 688 689 <para>To enable the expectation support in the configuration file, you 690 have to use the following option:</para> 691 692 <programlisting> 693Sync { 694 ... 695 Options { 696 ExpectationSync { 697 ftp 698 sip 699 ras # for H.323 700 q.931 # for H.323 701 h.245 # for H.323 702 } 703 } 704}</programlisting> 705 706 <para>The example above enables the synchronization of the expectations 707 for the FTP, SIP and H.323 helpers.</para> 708 709 <para>In my testbed, there are two firewalls in a primary-backup 710 configuration running keepalived. They use a couple of floating cluster 711 IP address (192.168.0.100 and 192.168.1.100) that are used by the client. 712 These firewalls protect one FTP server (192.168.1.2) that will be accessed 713 by one client.</para> 714 715 <para>In ASCII art, it looks like this:</para> 716 717 <programlisting> 718 192.168.0.100 192.168.1.100 719 eth1 eth2 720 fw-1 721 / \ FTP 722 client ------ ------ server 723 192.168.0.2 \ / 192.168.1.2 724 fw-2 725 </programlisting> 726 727 <para>This is the rule-set for the firewalls:</para> 728 729 <programlisting> 730 -A FORWARD -m state --state RELATED -j ACCEPT 731 -A FORWARD -i eth2 -m state --state ESTABLISHED -j ACCEPT 732 -A FORWARD -i eth1 -p tcp -m tcp --dport 21 --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW -j ACCEPT 733 -A FORWARD -i eth1 -p tcp -m state --state ESTABLISHED -j ACCEPT 734 -A FORWARD -m state --state INVALID -j LOG --log-prefix "invalid: "</programlisting> 735 736 <para>Before going ahead, make sure <emphasis>nf_conntrack_ftp</emphasis> is 737 loaded.</para> 738 739 <para>The following steps detail how to check that the expectation support 740 works fine with FTP traffic:</para> 741 742 <orderedlist> 743 <listitem> 744 <para>Switch to the client. Start one FTP control connection to one 745 server that is protected by the firewalls, enter passive mode:</para> 746 747 <programlisting> 748 (term-1) user@client$ nc 192.168.1.2 21 749 220 dummy FTP server 750 USER anonymous 751 331 Please specify the password. 752 PASS nothing 753 230 Login successful. 754 PASV 755 227 Entering Passive Mode (192,168,1,2,163,11).</programlisting> 756 757 <para>This means that port 163*256+11=41739 will be used for the data 758 traffic. I suggest you to read <ulink url="http://www.freefire.org/articles/ftpexample.php">djb's FTP protocol description</ulink> in case that you 759 don't understand how this calculation is done.</para> 760 </listitem> 761 762 <listitem> 763 <para> Switch to fw-1 (primary) to check that the expectation is in the 764 internal cache.</para> 765 766 <programlisting> 767 root@fw1# conntrackd -i exp 768 proto=6 src=192.168.0.2 dst=192.168.1.2 sport=0 dport=41739 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.0.2 master-dst=192.168.1.2 sport=36390 dport=21 helper=ftp [active since 5s] 769 </programlisting> 770 </listitem> 771 772 <listitem> 773 <para> Switch to fw-2 (backup) to check that the expectation has been 774 successfully replicated.</para> 775 776 <programlisting> 777 root@fw2# conntrackd -e exp 778 proto=6 src=192.168.0.2 dst=192.168.1.2 sport=0 dport=41739 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.0.2 master-dst=192.168.1.2 sport=36390 dport=21 [active since 8s] 779 </programlisting> 780 </listitem> 781 782 <listitem> 783 <para>Make the primary firewall fw-1 fail. Now fw-2 becomes primary.</para> 784 </listitem> 785 786 <listitem> 787 <para>Switch to fw-2 (primary) to commit the external cache into the 788 kernel. The logs should display that the commit was successful:</para> 789 790 <programlisting> 791 root@fw2# tail -100f /var/log/conntrackd.log 792 [Wed Dec 7 22:16:31 2011] (pid=19195) [notice] committing external cache: expectations 793 [Wed Dec 7 22:16:31 2011] (pid=19195) [notice] Committed 1 new entries 794 [Wed Dec 7 22:16:31 2011] (pid=19195) [notice] commit has taken 0.000366 seconds</programlisting> 795 </listitem> 796 797 <listitem> 798 <para> Switch to the client. Open a new terminal and connect to the port that 799 has been announced by the server:</para> 800 801 <programlisting> 802 (term-2) user@client$ nc -vvv 192.168.1.2 41739 803 (UNKNOWN) [192.168.1.2] 41739 (?) open</programlisting> 804 </listitem> 805 806 <listitem> 807 <para>Switch to term-1 and ask for the file listing:</para> 808 809 <programlisting> 810 [...] 811 227 Entering Passive Mode (192,168,1,2,163,11). 812 LIST</programlisting> 813 </listitem> 814 815 <listitem> 816 <para>Switch to term-2, it should display the listing. That means 817 everything has worked fine.</para> 818 </listitem> 819 820 </orderedlist> 821 822 <para>You may want to try disabling the expectation support and 823 repeating the steps to check that <emphasis>it does not work</emphasis> 824 without the state-synchronization.</para> 825 826</sect3> 827 828</sect2> 829 830</sect1> 831 832<sect1 id="helpers"><title>User-space helpers</title> 833 834 <note><title>Check your Linux kernel version first</title> 835 <para> 836 The user-space helper infrastructure requires a Linux kernel >= 3.6 837 to work appropriately. 838 </para> 839 </note> 840 841<para>Connection tracking helpers allows you to filter multi-flow protocols 842that usually separate control and data traffic into different flows. 843These protocols usually violate network layering by including layer 3/4 844details, eg. IP address and TCP/UDP ports, in their application protocol 845(which resides in layer 7). This is problematic for gateways since they 846operate at packet-level, ie. layers 3/4, and therefore they miss this 847important information to filter these protocols appropriately.</para> 848 849<para>Helpers inspect packet content (at layer 7) and create the so-called 850expectations. These expectations are added to one internal table 851that resides in the gateway. For each new packet arriving to the 852gateway, the gateway first looks up for matching expectations. If 853there is any, then this flow is accepted since it's been expected. 854Note this lookup only occurs for the first packet that is part of one 855newly established flow, not for all packets.</para> 856 857<para>Since 1.4.0, conntrackd provides the infrastructure to develop 858helpers in user-space. The main features of the user-space infrastructure 859for helpers are:</para> 860 861<itemizedlist> 862 863<listitem><para>Rapid connection tracking helper development, as developing code 864in user-space is usually faster.</para></listitem> 865 866<listitem><para>Reliability: A buggy helper does not crash the kernel. If the helper 867fails, ie. the conntrackd crashes, Moreover, we can monitor the helper process 868and restart it in case of problems.</para></listitem> 869 870<listitem><para>Security: Avoid complex string matching and mangling in 871kernel-space running in privileged mode. Going further, we can even think 872about running user-space helper as a non-root process.</para></listitem> 873 874<listitem><para>It allows the development of very specific helpers for 875proprietary protocols that are not standard. This is the case of the SQL*net 876helper. Implementing this in kernel-space may be problematic, since 877this may not be accepted for ainline inclusion in the Linux kernel. 878As an alternative, we can still distribute this support as separate 879patches. However, my personal experience is that, given that the 880kernel API/ABI is not stable, changes in the interface lead to the 881breakage of the patch. This highly increase the overhead in the 882maintainance.</para></listitem> 883 884</itemizedlist> 885 886<para>Currently, the infrastructure supports the following user-space helpers: 887</para> 888 889<itemizedlist> 890<listitem><para>Oracle*TNS, to support its special <emphasis>Redirect</emphasis> message.</para></listitem> 891<listitem><para>NFSv3, mind that version 4 does not require this helper.</para></listitem> 892<listitem><para>FTP (this helper is also available in kernel-space).</para></listitem> 893</itemizedlist> 894 895<para>The following steps describe how to enable the RPC portmapper helper for NFSv3 (this is similar for other helpers):</para> 896 897<orderedlist> 898<listitem><para>Register user-space helper: 899 900<programlisting> 901nfct helper add rpc inet udp 902nfct helper add rpc inet tcp 903</programlisting> 904 905This registers the portmapper helper for both UDP and TCP (NFSv3 traffic goes both over TCP and UDP). 906</para></listitem> 907 908<listitem><para>Add iptables rule using the CT target: 909 910<programlisting> 911# iptables -I OUTPUT -t raw -p udp --dport 111 -j CT --helper rpc 912# iptables -I OUTPUT -t raw -p tcp --dport 111 -j CT --helper rpc 913</programlisting> 914 915With this, packets matching port TCP/UDP/111 are passed to user-space for 916inspection. If there is no instance of conntrackd configured to support 917user-space helpers, no inspection happens and packets are not sent to 918user-space.</para></listitem> 919 920<listitem><para>Add configuration to conntrackd.conf: 921 922<programlisting> 923Helper { 924 Type rpc inet udp { 925 QueueNum 1 926 QueueLen 10240 927 Policy rpc { 928 ExpectMax 1 929 ExpectTimeout 300 930 } 931 } 932 Type rpc inet tcp { 933 QueueNum 2 934 QueueLen 10240 935 Policy rpc { 936 ExpectMax 1 937 ExpectTimeout 300 938 } 939 } 940} 941</programlisting> 942 943This configures conntrackd to use NFQUEUE queue numbers 1 and 2 to send traffic 944for inspection to user-space</para> 945 946 <note><title>If you have some custom libnetfilter_queue application</title> 947 <para> 948 Make sure your queue numbers do not collide with those used in your 949 conntrackd.conf file. 950 </para> 951 </note> 952 953</listitem> 954 955</orderedlist> 956 957<para>Now you can test this (assuming you have some working NFSv3 setup) with: 958 959<programlisting> 960mount -t nfs -onfsvers=3 mynfs.server.info:/srv/cvs /mnt/ 961</programlisting> 962 963</para> 964 965<para>You should see new expectations being added via: 966 967<programlisting> 968# conntrack -E expect 969 [NEW] 300 proto=17 src=1.2.3.4 dst=1.2.3.4 sport=0 dport=54834 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=1.2.3.4 master-dst=1.2.3.4 sport=58190 dport=111 PERMANENT class=0 helper=rpc 970 [NEW] 300 proto=6 src=1.2.3.4 dst=1.2.3.4 sport=0 dport=2049 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=1.2.3.4 master-dst=1.2.3.4 sport=55450 dport=111 PERMANENT class=0 helper=rpc 971 [NEW] 300 proto=17 src=1.2.3.4 dst=1.2.3.4 sport=0 dport=58031 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=1.2.3.4 master-dst=1.2.3.4 sport=56309 dport=111 PERMANENT class=0 helper=rpc 972</programlisting> 973</para> 974 975</sect1> 976 977<sect1 id="sync-trouble"><title>Troubleshooting</title> 978 979 <para>Problems with <emphasis>conntrackd</emphasis>? The following list 980 of questions should help for troubleshooting:</para> 981 982 <qandaset> 983 984 <qandaentry> 985 <question> 986 <para> 987 I see <emphasis>packets lost</emphasis> in <emphasis>conntrackd -s</emphasis> 988 </para> 989 </question> 990 <answer> 991 <para> 992 You can rise the value of <emphasis>McastRcvSocketBuffer</emphasis> and <emphasis>McastRcvSocketBuffer</emphasis>, if the problem is due to buffer overruns in the multicast sender or the receiver, the problem should disapear. 993 </para> 994 </answer> 995 </qandaentry> 996 997 <qandaentry> 998 <question> 999 <para> 1000 The log messages report that the <emphasis>maximum netlink socket buffer has been reached</emphasis>. 1001 </para> 1002 </question> 1003 <answer> 1004 <para> 1005 You can increase the values of <emphasis>SocketBufferSize</emphasis> and <emphasis>SocketBufferSizeMaxGrown</emphasis>. 1006 </para> 1007 </answer> 1008 </qandaentry> 1009 1010 <qandaentry> 1011 <question> 1012 <para> 1013 I see <emphasis>can't open multicast server</emphasis> in the log messages 1014 </para> 1015 </question> 1016 <answer> 1017 <para> 1018 Make sure that the <emphasis>IPv4_interface</emphasis> clause has the IP of the dedicated link. 1019 </para> 1020 </answer> 1021 </qandaentry> 1022 1023 <qandaentry> 1024 <question> 1025 <para> 1026 Can I use <ulink url="http://www.backhand.org/wackamole/">wackamole</ulink>, heartattack or any other HA manager? 1027 </para> 1028 </question> 1029 <answer> 1030 <para> 1031 Absolutely, you can. But before reporting issues, make sure that your HA manager is not the source of the problems. 1032 </para> 1033 </answer> 1034 </qandaentry> 1035 1036 <qandaentry> 1037 <question> 1038 <para> 1039 Does conntrackd support TCP flow-recovery with window tracking enabled? 1040 </para> 1041 </question> 1042 <answer> 1043 <para> 1044 Yes, but you require a Linux kernel >= 2.6.36 and the conntrack-tools >= 0.9.15. To enable it, check the TCPWindowTracking clause in the example configuration files. 1045 </para> 1046 </answer> 1047 </qandaentry> 1048 1049 <qandaentry> 1050 <question> 1051 <para> 1052 Does conntrackd support the H.323 and SIP connection tracking helpers? 1053 </para> 1054 </question> 1055 <answer> 1056 <para> 1057 Yes, conntrackd includes expectation support since version 1.2.0. 1058 </para> 1059 </answer> 1060 </qandaentry> 1061 1062 <qandaentry> 1063 <question> 1064 <para> 1065 Is there any way to set up a more verbose mode in the log message for debugging? 1066 </para> 1067 </question> 1068 <answer> 1069 <para> 1070 No, but conntrackd provides lots of information that you can look up in 1071 runtime via -s option.</para> 1072 1073 <para>You can check network statistics to find anomalies:</para> 1074 <programlisting> 1075# conntrackd -s network 1076 network statistics: 1077 recv: 1078 Malformed messages: 0 1079 Wrong protocol version: 0 1080 Malformed header: 0 1081 Malformed payload: 0 1082 Bad message type: 0 1083 Truncated message: 0 1084 Bad message size: 0 1085 send: 1086 Malformed messages: 0 1087 1088sequence tracking statistics: 1089 recv: 1090 Packets lost: 42726 1091 Packets before: 0 1092 1093UDP traffic (active device=eth3): 1094 564232 Bytes sent 1979844 Bytes recv 1095 2844 Pckts sent 8029 Pckts recv 1096 0 Error send 0 Error recv 1097 </programlisting> 1098 1099 <para>You can check cache statistics:</para> 1100 <programlisting> 1101# conntrackd -s cache 1102cache:internal active objects: 0 1103 active/total entries: 0/ 0 1104 creation OK/failed: 11068/ 0 1105 no memory available: 0 1106 no space left in cache: 0 1107 update OK/failed: 4128/ 0 1108 entry not found: 0 1109 deletion created/failed: 11068/ 0 1110 entry not found: 0 1111 1112cache:external active objects: 0 1113 active/total entries: 0/ 0 1114 creation OK/failed: 10521/ 0 1115 no memory available: 0 1116 no space left in cache: 0 1117 update OK/failed: 8832/ 0 1118 entry not found: 0 1119 deletion created/failed: 10521/ 0 1120 entry not found: 0 1121 </programlisting> 1122 1123 <para>You can check runtime miscelaneous statistics:</para> 1124 <programlisting> 1125# conntrackd -s runtime 1126daemon uptime: 14 min 1127 1128netlink stats: 1129 events received: 24736 1130 events filtered: 0 1131 events unknown type: 0 1132 catch event failed: 0 1133 dump unknown type: 0 1134 netlink overrun: 0 1135 flush kernel table: 1 1136 resync with kernel table: 0 1137 current buffer size (in bytes): 8000000 1138 1139runtime stats: 1140 child process failed: 0 1141 child process segfault: 0 1142 child process termsig: 0 1143 select failed: 0 1144 wait failed: 0 1145 local read failed: 0 1146 local unknown request: 0 1147 </programlisting> 1148 1149 <para>You can check dedicated link statistics:</para> 1150 <programlisting> 1151# conntrackd -s link 1152UDP traffic device=eth3 status=RUNNING role=ACTIVE: 1153 566848 Bytes sent 1982612 Bytes recv 1154 3018 Pckts sent 8203 Pckts recv 1155 0 Error send 0 Error recv 1156 </programlisting> 1157 1158 <para>You can check network queue statistics:</para> 1159 <programlisting> 1160# conntrackd -s queue 1161allocated queue nodes: 1 1162 1163queue txqueue: 1164current elements: 0 1165maximum elements: 2147483647 1166not enough space errors: 0 1167 1168queue errorq: 1169current elements: 0 1170maximum elements: 128 1171not enough space errors: 0 1172 1173queue rsqueue: 1174current elements: 1 1175maximum elements: 131072 1176not enough space errors: 0 1177 </programlisting> 1178 </answer> 1179 </qandaentry> 1180 1181 </qandaset> 1182 1183</sect1> 1184 1185</chapter> 1186 1187</book> 1188