1<?xml version="1.0" encoding="UTF-8" standalone="no"?> 2<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 3<html xmlns="http://www.w3.org/1999/xhtml"> 4 <head> 5 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> 6 <title>Chapter��3.��The DB Replication Framework</title> 7 <link rel="stylesheet" href="gettingStarted.css" type="text/css" /> 8 <meta name="generator" content="DocBook XSL Stylesheets V1.62.4" /> 9 <link rel="home" href="index.html" title="Getting Started with Replicated Berkeley DB Applications" /> 10 <link rel="up" href="index.html" title="Getting Started with Replicated Berkeley DB Applications" /> 11 <link rel="previous" href="simpleprogramlisting.html" title="Program Listing" /> 12 <link rel="next" href="repmgr_init_example_c.html" title="Adding the Replication Framework to SimpleTxn " /> 13 </head> 14 <body> 15 <div class="navheader"> 16 <table width="100%" summary="Navigation header"> 17 <tr> 18 <th colspan="3" align="center">Chapter��3.��The DB Replication Framework</th> 19 </tr> 20 <tr> 21 <td width="20%" align="left"><a accesskey="p" href="simpleprogramlisting.html">Prev</a>��</td> 22 <th width="60%" align="center">��</th> 23 <td width="20%" align="right">��<a accesskey="n" href="repmgr_init_example_c.html">Next</a></td> 24 </tr> 25 </table> 26 <hr /> 27 </div> 28 <div class="chapter" lang="en" xml:lang="en"> 29 <div class="titlepage"> 30 <div> 31 <div> 32 <h2 class="title"><a id="repapp"></a>Chapter��3.��The DB Replication Framework</h2> 33 </div> 34 </div> 35 <div></div> 36 </div> 37 <div class="toc"> 38 <p> 39 <b>Table of Contents</b> 40 </p> 41 <dl> 42 <dt> 43 <span class="sect1"> 44 <a href="repapp.html#rep_init_code"> 45 Starting and Stopping Replication 46 </a> 47 </span> 48 </dt> 49 <dd> 50 <dl> 51 <dt> 52 <span class="sect2"> 53 <a href="repapp.html#election_flags">Managing Election Policies</a> 54 </span> 55 </dt> 56 <dt> 57 <span class="sect2"> 58 <a href="repapp.html#thread_count">Selecting the Number of Threads</a> 59 </span> 60 </dt> 61 </dl> 62 </dd> 63 <dt> 64 <span class="sect1"> 65 <a href="repmgr_init_example_c.html">Adding the Replication Framework to 66 67 SimpleTxn 68 </a> 69 </span> 70 </dt> 71 <dt> 72 <span class="sect1"> 73 <a href="fwrkpermmessage.html">Permanent Message Handling</a> 74 </span> 75 </dt> 76 <dd> 77 <dl> 78 <dt> 79 <span class="sect2"> 80 <a href="fwrkpermmessage.html#fmwrkpermpolicy">Identifying Permanent Message Policies</a> 81 </span> 82 </dt> 83 <dt> 84 <span class="sect2"> 85 <a href="fwrkpermmessage.html#fmwrkpermtimeout">Setting the Permanent Message Timeout</a> 86 </span> 87 </dt> 88 <dt> 89 <span class="sect2"> 90 <a href="fwrkpermmessage.html#perm2fmwrkexample">Adding a Permanent Message Policy to 91 92 RepMgr 93 94 </a> 95 </span> 96 </dt> 97 </dl> 98 </dd> 99 <dt> 100 <span class="sect1"> 101 <a href="electiontimes.html">Managing Election Times</a> 102 </span> 103 </dt> 104 <dd> 105 <dl> 106 <dt> 107 <span class="sect2"> 108 <a href="electiontimes.html#electiontimeout">Managing Election Timeouts</a> 109 </span> 110 </dt> 111 <dt> 112 <span class="sect2"> 113 <a href="electiontimes.html#electretrytime">Managing Election Retry Times</a> 114 </span> 115 </dt> 116 </dl> 117 </dd> 118 <dt> 119 <span class="sect1"> 120 <a href="fmwrkconnectretry.html">Managing Connection Retries</a> 121 </span> 122 </dt> 123 <dt> 124 <span class="sect1"> 125 <a href="heartbeats.html">Managing Heartbeats</a> 126 </span> 127 </dt> 128 </dl> 129 </div> 130 <p> 131 The easiest way to add replication to your transactional 132 application is to use the replication framework. The replication framework provides a comprehensive 133 communications layer that enables replication. For a brief listing 134 of the replication framework's feature set, see 135 <a href="apioverview.html#repframeworkoverview">Replication Framework Overview</a>. 136 </p> 137 <p> 138 To use the replication framework, you make use of special methods off the 139 140 <span><tt class="classname">DbEnv</tt> class.</span> 141 That is: 142 </p> 143 <div class="orderedlist"> 144 <ol type="1"> 145 <li> 146 <p> 147 Create an environment handle as normal. 148 </p> 149 </li> 150 <li> 151 <p> 152 Configure your environment handle as 153 needed (e.g. set the error file and 154 error prefix values, if desired). 155 </p> 156 </li> 157 <li> 158 <p> 159 Use the replication framework replication methods to 160 configure the replication framework. Using these 161 methods causes DB to know that you 162 are using the replication framework. 163 </p> 164 <p> 165 Configuring the replication framework 166 entails setting its replication 167 priority, setting the TCP/IP address 168 that this replication environment will use for 169 incoming replication messages, identify 170 TCP/IP addresses of other replication 171 environments, setting the number of 172 replication environments in the 173 replication group, and so forth. These actions are 174 discussed throughout the remainder of 175 this chapter. 176 </p> 177 </li> 178 <li> 179 <p> 180 Open your environment handle. When you 181 do this, be sure to specify 182 183 <span><tt class="literal">DB_INIT_REP</tt> and 184 <tt class="literal">DB_THREAD</tt> to your 185 open flags. (This is in addition to the 186 flags that you normally use for a 187 single-threaded transactional 188 application). The first of these causes 189 replication to be initialized for the 190 application. The second causes your 191 environment handle to be free-threaded 192 (thread safe). Both flags are required 193 for replication framework usage. 194 </span> 195 196 197 </p> 198 </li> 199 <li> 200 <p> 201 Start replication by calling 202 203 <span><tt class="methodname">DbEnv::repmgr_start()</tt>.</span> 204 205 </p> 206 </li> 207 <li> 208 <p> 209 Open your databases as needed. Masters 210 must open their databases for read 211 and write activity. Replicas can open 212 their databases for read-only activity, but 213 doing so means they must re-open the 214 databases if the replica ever becomes a 215 master. Either way, replicas should never attempt to 216 write to the database(s) directly. 217 </p> 218 </li> 219 </ol> 220 </div> 221 <div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"> 222 <h3 class="title">Note</h3> 223 <p> 224 The replication framework allows you to only use one 225 environment handle per process. 226 </p> 227 </div> 228 <p> 229 When you are ready to shut down your application: 230 </p> 231 <div class="orderedlist"> 232 <ol type="1"> 233 <li> 234 <p> 235 Close your databases 236 </p> 237 </li> 238 <li> 239 <p> 240 Close your environment. This causes 241 replication to stop as well. 242 </p> 243 </li> 244 </ol> 245 </div> 246 <div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"> 247 <h3 class="title">Note</h3> 248 <p> 249 Before you can use the replication framework, you may have to 250 enable it in your DB library. This is 251 <span class="emphasis"><em>not</em></span> a requirement for 252 Microsoft Windows systems, or Unix systems that 253 use pthread mutexes by default. Other systems, 254 notably BSD and BSD-derived systems (such as 255 Mac OS X), must enable the replication framework when you 256 configure the DB build. 257 </p> 258 <p> 259 You do this by <span class="emphasis"><em>not</em></span> 260 disabling replication and by configuring the 261 library with POSIX threads support. In other 262 words, replication must be turned on in the 263 build (it is by default), and POSIX thread 264 support must be enabled if it is not already by 265 default. To do this, use the 266 <tt class="literal">--enable-pthread_api</tt> switch 267 on the configure script. 268 </p> 269 <p> 270 For example: 271 </p> 272 <pre class="programlisting">../dist/configure --enable-pthread-api</pre> 273 </div> 274 <div class="sect1" lang="en" xml:lang="en"> 275 <div class="titlepage"> 276 <div> 277 <div> 278 <h2 class="title" style="clear: both"><a id="rep_init_code"></a> 279 Starting and Stopping Replication 280 </h2> 281 </div> 282 </div> 283 <div></div> 284 </div> 285 <p> 286 As described above, you introduce replication to an 287 application by starting with a transactional 288 application, performing some basic replication 289 configuration, and then starting replication using 290 291 <span><tt class="methodname">DbEnv::repmgr_start()</tt>.</span> 292 293 </p> 294 <p> 295 You stop replication by closing your environment 296 cleanly, as is normal for an DB application. 297 </p> 298 <p> 299 For example, the following code fragment initializes, then 300 stops and starts replication. Note that other replication 301 activities are omitted for brevity. 302 </p> 303 <pre class="programlisting">#include <db_cxx.h> 304 305/* Use a 10mb cache */ 306#define CACHESIZE (10 * 1024 * 1024) 307 308... 309 310 DbEnv *dbenv; /* Environment handle. */ 311 const char *progname; /* Program name. */ 312 const char *envHome; /* Environment home directory. */ 313 const char *listen_host; /* A TCP/IP hostname. */ 314 const char *other_host; /* A TCP/IP hostname. */ 315 u_int16 listen_port; /* A TCP/IP port. */ 316 u_int16 other_port; /* A TCP/IP port. */ 317 318 /* Initialize variables */ 319 dbenv = NULL; 320 progname = "example_replication"; 321 envHome = "ENVIRONMENT_HOME"; 322 listen_host = "mymachine.sleepycat.com"; 323 listen_port = 5001; 324 other_host = "anothermachine.sleepycat.com"; 325 other_port = 4555; 326 327 try { 328 /* Create the environment handle */ 329 dbenv = new DbEnv(0); 330 331 /* 332 * Configure the environment handle. Here we configure 333 * asynchronous transactional commits for performance reasons. 334 */ 335 dbenv->set_errfile(stderr); 336 dbenv->set_errpfx(progname); 337 (void)dbenv->set_cachesize(0, CACHESIZE, 0); 338 (void)dbenv->set_flags(DB_TXN_NOSYNC, 1); 339 340 /* 341 * Configure the local address. This is the local hostname and 342 * port that this replication participant will use to receive 343 * incoming replication messages. Note that this can be performed 344 * only once for the application. It is required. 345 */ 346 dbenv->repmgr_set_local_site(listen_host, listen_port, 0); 347 348 /* 349 * Set this application's priority. This is used for elections. 350 * 351 * Set this number to a positive integer, or 0 if you do not want 352 * this site to be able to become a master. 353 */ 354 dbenv->rep_set_priority(100); 355 356 /* 357 * Add a site to the list of replication environments known to 358 * this application. 359 */ 360 dbenv->repmgr_add_remote_site(dbenv, other_host, other_port, 361 NULL, 0); 362 363 /* 364 * Identify the number of sites in the replication group. This is 365 * necessary so that elections and permanent message handling can 366 * be performed correctly. 367 */ 368 dbenv->repmgr_set_nsites(dbenv, 2); 369 370 /* Open the environment handle. Note that we add DB_THREAD and 371 * DB_INIT_REP to the list of flags. These are required. 372 */ 373 dbenv->open(home, DB_CREATE | DB_RECOVER | 374 DB_INIT_LOCK | DB_INIT_LOG | 375 DB_INIT_MPOOL | DB_INIT_TXN | 376 DB_THREAD | DB_INIT_REP, 377 0); 378 379 /* Start the replication framework such that it uses 3 threads. */ 380 dbenv->repmgr_start(3, DB_REP_ELECTION); 381 382 /* Sleep to give ourselves time to find a master */ 383 sleep(5); 384 385 /* 386 ********************************************************** 387 *** All other application code goes here, including ***** 388 *** database opens ***** 389 ********************************************************** 390 */ 391 392 393 } catch (DbException &de) { 394 /* Error handling goes here */ 395 } 396 397 /* Close out the application here. 398 try { 399 /* 400 * Make sure all your database handles are closed 401 * (omitted from this example). 402 */ 403 404 /* Close the environment */ 405 if (dbenv != NULL) 406 (void)dbenv->close(dbenv, 0); 407 408 } catch (DbException &de) { 409 /* Error handling goes here */ 410 } 411 412 /* All done */ </pre> 413 <div class="sect2" lang="en" xml:lang="en"> 414 <div class="titlepage"> 415 <div> 416 <div> 417 <h3 class="title"><a id="election_flags"></a>Managing Election Policies</h3> 418 </div> 419 </div> 420 <div></div> 421 </div> 422 <p> 423 Before continuing, it is worth taking a look at the 424 425 <span> 426 startup election flags accepted by 427 428 <span><tt class="methodname">DbEnv::repgmr_start()</tt>.</span> 429 These flags control how your replication application will 430 behave when it first starts up. 431 </span> 432 433 434 </p> 435 <p> 436 In the previous example, we specified 437 <tt class="literal">DB_REP_ELECTION</tt> 438 439 when we started replication. This causes the 440 application to try to find a master upon startup. If it 441 cannot, it calls for an election. In the event an 442 election is held, the environment receiving the most number of 443 votes will become the master. 444 </p> 445 <p> 446 There's some important points to make here: 447 </p> 448 <div class="itemizedlist"> 449 <ul type="disc"> 450 <li> 451 <p> 452 This 453 <span>flag</span> 454 455 only requires that other 456 environments in the replication group 457 participate in the vote. There is no 458 requirement that 459 <span class="emphasis"><em>all</em></span> such 460 environments participate. In other 461 words, if an environment 462 starts up, it can call for an 463 election, and select a master, even 464 if all other environment have not yet 465 joined the replication group. 466 </p> 467 </li> 468 <li> 469 <p> 470 It only requires a simple majority of 471 participating environments to elect a master. The number of 472 environments used to calculate the simple 473 majority is based on the value set for 474 475 476 <span><tt class="methodname">DbEnv::rep_set_nsites()</tt>.</span> 477 478 479 480 This is always true of elections held using the replication framework. 481 </p> 482 </li> 483 <li> 484 <p> 485 As always, the environment participating in the election with the most 486 up-to-date log files is selected as 487 master. If an environment with better log files 488 has not yet joined the replication 489 group, it may not become the master. 490 </p> 491 </li> 492 </ul> 493 </div> 494 <p> 495 Any one of these points may be enough to cause a 496 less-than-optimum environment to be selected as master. 497 Therefore, to give you a better degree of control over 498 which environment becomes a master at application startup, 499 the replication framework offers the following start-up 500 <span>flags:</span> 501 502 </p> 503 <div class="informaltable"> 504 <table border="1" width="80%"> 505 <colgroup> 506 <col /> 507 <col /> 508 </colgroup> 509 <thead> 510 <tr> 511 <th>Flag</th> 512 <th>Description</th> 513 </tr> 514 </thead> 515 <tbody> 516 <tr> 517 <td> 518 <tt class="literal">DB_REP_MASTER</tt> 519 </td> 520 <td> 521 <p> 522 The application starts up and declares itself to be a master 523 without calling for an election. It is an error for more 524 than one environment to start up using this flag, or for 525 an environment 526 to use this flag when a master already exists. 527 </p> 528 <p> 529 Note that no replication group should 530 <span class="emphasis"><em>ever</em></span> operate with more than 531 one master. 532 </p> 533 <p> 534 In the event that a environment attempts to become a 535 master when a master already exists, the 536 replication code will resolve the problem by 537 holding an election. Note, however, that there 538 is always a possibility of data loss in the face 539 of duplicate masters, because once a master is 540 selected, the environment that loses the election will 541 have to roll back any transactions committed 542 until it is in sync with the "real" master. 543 </p> 544 </td> 545 </tr> 546 <tr> 547 <td> 548 <tt class="literal">DB_REP_CLIENT</tt> 549 </td> 550 <td> 551 <p> 552 The application starts up and declares 553 itself to be a replica without calling for 554 an election. Note that the application 555 can still become a master if a subsequent 556 application starts up, calls for an 557 election, and this application is elected 558 master. 559 </p> 560 </td> 561 </tr> 562 <tr> 563 <td> 564 <tt class="literal">DB_REP_ELECTION</tt> 565 </td> 566 <td> 567 <p> 568 As described above, the application starts up, 569 looks for a master, and if one is not found calls 570 for an election. 571 </p> 572 </td> 573 </tr> 574 <tr> 575 <td> 576 <tt class="literal">DB_REP_FULL_ELECTION</tt> 577 </td> 578 <td> 579 <p> 580 Identical to 581 <tt class="literal">DB_REP_ELECTION</tt> 582 583 except that the election requires all 584 known members of the replication group to 585 participate. If a given environment has not yet 586 started but it is included in the 587 replication group count (using 588 589 <span><tt class="methodname">DbEnv::rep_set_nsites()</tt>)</span> 590 591 then a master can not be elected. 592 </p> 593 </td> 594 </tr> 595 </tbody> 596 </table> 597 </div> 598 </div> 599 <div class="sect2" lang="en" xml:lang="en"> 600 <div class="titlepage"> 601 <div> 602 <div> 603 <h3 class="title"><a id="thread_count"></a>Selecting the Number of Threads</h3> 604 </div> 605 </div> 606 <div></div> 607 </div> 608 <p> 609 Under the hood, the replication framework is threaded and you can 610 control the number of threads used to process messages received from 611 other replicas. The threads that the replication framework uses are: 612 </p> 613 <div class="itemizedlist"> 614 <ul type="disc"> 615 <li> 616 <p> 617 Incoming message thread. This thread 618 receives messages from the site's 619 socket and passes those messages to 620 message processing threads (see below) 621 for handling. 622 </p> 623 </li> 624 <li> 625 <p> 626 Outgoing message thread. Outgoing 627 are performed in whatever thread 628 performed a write to the database(s). 629 That is, the thread that called, for 630 example, 631 632 <tt class="methodname">Db::put()</tt> 633 634 is the thread that writes replication messages 635 about that fact to the socket. 636 </p> 637 <p> 638 Note that if this write activity would 639 cause the thread to be blocked due to 640 some condition on the socket, the replication framework 641 will hand the outgoing message to the 642 incoming message thread, and it will 643 then write the message to the socket. 644 This prevents your database write 645 threads from blocking due to abnormal 646 network I/O conditions. 647 </p> 648 </li> 649 <li> 650 <p> 651 Message processing threads are 652 responsible for parsing and then 653 responding to incoming replication 654 messages. Typically, a response will 655 include write activity to your 656 database(s), so these threads can be 657 busy performing disk I/O. 658 </p> 659 </li> 660 </ul> 661 </div> 662 <p> 663 Of these threads, the only ones that you have any 664 configuration control over are the message processing 665 threads. In this case, you can determine how many 666 of these threads you want to run. 667 </p> 668 <p> 669 It is always a bit of an art to decide on a thread count, 670 but the short answer is you probably do not need more 671 than three threads here, and it is likely that one will 672 suffice. That said, the best thing to do is set your 673 thread count to a fairly low number and then increase 674 it if it appears that your application will benefit 675 from the additional threads. 676 </p> 677 </div> 678 </div> 679 </div> 680 <div class="navfooter"> 681 <hr /> 682 <table width="100%" summary="Navigation footer"> 683 <tr> 684 <td width="40%" align="left"><a accesskey="p" href="simpleprogramlisting.html">Prev</a>��</td> 685 <td width="20%" align="center"> 686 <a accesskey="u" href="index.html">Up</a> 687 </td> 688 <td width="40%" align="right">��<a accesskey="n" href="repmgr_init_example_c.html">Next</a></td> 689 </tr> 690 <tr> 691 <td width="40%" align="left" valign="top">Program Listing��</td> 692 <td width="20%" align="center"> 693 <a accesskey="h" href="index.html">Home</a> 694 </td> 695 <td width="40%" align="right" valign="top">��Adding the Replication Framework to 696 697 SimpleTxn 698 </td> 699 </tr> 700 </table> 701 </div> 702 </body> 703</html> 704