1<?xml version="1.0" encoding="UTF-8" standalone="no"?> 2<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 3<html xmlns="http://www.w3.org/1999/xhtml"> 4 <head> 5 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> 6 <title>Chapter��3.��The DB Replication Framework</title> 7 <link rel="stylesheet" href="gettingStarted.css" type="text/css" /> 8 <meta name="generator" content="DocBook XSL Stylesheets V1.62.4" /> 9 <link rel="home" href="index.html" title="Getting Started with Replicated Berkeley DB Applications" /> 10 <link rel="up" href="index.html" title="Getting Started with Replicated Berkeley DB Applications" /> 11 <link rel="previous" href="simpleprogramlisting.html" title="Program Listing" /> 12 <link rel="next" href="repmgr_init_example_c.html" title="Adding the Replication Framework to simple_txn " /> 13 </head> 14 <body> 15 <div class="navheader"> 16 <table width="100%" summary="Navigation header"> 17 <tr> 18 <th colspan="3" align="center">Chapter��3.��The DB Replication Framework</th> 19 </tr> 20 <tr> 21 <td width="20%" align="left"><a accesskey="p" href="simpleprogramlisting.html">Prev</a>��</td> 22 <th width="60%" align="center">��</th> 23 <td width="20%" align="right">��<a accesskey="n" href="repmgr_init_example_c.html">Next</a></td> 24 </tr> 25 </table> 26 <hr /> 27 </div> 28 <div class="chapter" lang="en" xml:lang="en"> 29 <div class="titlepage"> 30 <div> 31 <div> 32 <h2 class="title"><a id="repapp"></a>Chapter��3.��The DB Replication Framework</h2> 33 </div> 34 </div> 35 <div></div> 36 </div> 37 <div class="toc"> 38 <p> 39 <b>Table of Contents</b> 40 </p> 41 <dl> 42 <dt> 43 <span class="sect1"> 44 <a href="repapp.html#rep_init_code"> 45 Starting and Stopping Replication 46 </a> 47 </span> 48 </dt> 49 <dd> 50 <dl> 51 <dt> 52 <span class="sect2"> 53 <a href="repapp.html#election_flags">Managing Election Policies</a> 54 </span> 55 </dt> 56 <dt> 57 <span class="sect2"> 58 <a href="repapp.html#thread_count">Selecting the Number of Threads</a> 59 </span> 60 </dt> 61 </dl> 62 </dd> 63 <dt> 64 <span class="sect1"> 65 <a href="repmgr_init_example_c.html">Adding the Replication Framework to 66 simple_txn 67 68 </a> 69 </span> 70 </dt> 71 <dt> 72 <span class="sect1"> 73 <a href="fwrkpermmessage.html">Permanent Message Handling</a> 74 </span> 75 </dt> 76 <dd> 77 <dl> 78 <dt> 79 <span class="sect2"> 80 <a href="fwrkpermmessage.html#fmwrkpermpolicy">Identifying Permanent Message Policies</a> 81 </span> 82 </dt> 83 <dt> 84 <span class="sect2"> 85 <a href="fwrkpermmessage.html#fmwrkpermtimeout">Setting the Permanent Message Timeout</a> 86 </span> 87 </dt> 88 <dt> 89 <span class="sect2"> 90 <a href="fwrkpermmessage.html#perm2fmwrkexample">Adding a Permanent Message Policy to 91 rep_mgr 92 93 94 </a> 95 </span> 96 </dt> 97 </dl> 98 </dd> 99 <dt> 100 <span class="sect1"> 101 <a href="electiontimes.html">Managing Election Times</a> 102 </span> 103 </dt> 104 <dd> 105 <dl> 106 <dt> 107 <span class="sect2"> 108 <a href="electiontimes.html#electiontimeout">Managing Election Timeouts</a> 109 </span> 110 </dt> 111 <dt> 112 <span class="sect2"> 113 <a href="electiontimes.html#electretrytime">Managing Election Retry Times</a> 114 </span> 115 </dt> 116 </dl> 117 </dd> 118 <dt> 119 <span class="sect1"> 120 <a href="fmwrkconnectretry.html">Managing Connection Retries</a> 121 </span> 122 </dt> 123 <dt> 124 <span class="sect1"> 125 <a href="heartbeats.html">Managing Heartbeats</a> 126 </span> 127 </dt> 128 </dl> 129 </div> 130 <p> 131 The easiest way to add replication to your transactional 132 application is to use the replication framework. The replication framework provides a comprehensive 133 communications layer that enables replication. For a brief listing 134 of the replication framework's feature set, see 135 <a href="apioverview.html#repframeworkoverview">Replication Framework Overview</a>. 136 </p> 137 <p> 138 To use the replication framework, you make use of special methods off the 139 <span><tt class="classname">DB_ENV</tt> class.</span> 140 141 That is: 142 </p> 143 <div class="orderedlist"> 144 <ol type="1"> 145 <li> 146 <p> 147 Create an environment handle as normal. 148 </p> 149 </li> 150 <li> 151 <p> 152 Configure your environment handle as 153 needed (e.g. set the error file and 154 error prefix values, if desired). 155 </p> 156 </li> 157 <li> 158 <p> 159 Use the replication framework replication methods to 160 configure the replication framework. Using these 161 methods causes DB to know that you 162 are using the replication framework. 163 </p> 164 <p> 165 Configuring the replication framework 166 entails setting its replication 167 priority, setting the TCP/IP address 168 that this replication environment will use for 169 incoming replication messages, identify 170 TCP/IP addresses of other replication 171 environments, setting the number of 172 replication environments in the 173 replication group, and so forth. These actions are 174 discussed throughout the remainder of 175 this chapter. 176 </p> 177 </li> 178 <li> 179 <p> 180 Open your environment handle. When you 181 do this, be sure to specify 182 183 <span><tt class="literal">DB_INIT_REP</tt> and 184 <tt class="literal">DB_THREAD</tt> to your 185 open flags. (This is in addition to the 186 flags that you normally use for a 187 single-threaded transactional 188 application). The first of these causes 189 replication to be initialized for the 190 application. The second causes your 191 environment handle to be free-threaded 192 (thread safe). Both flags are required 193 for replication framework usage. 194 </span> 195 196 197 </p> 198 </li> 199 <li> 200 <p> 201 Start replication by calling 202 <span><tt class="methodname">DB_ENV->repmgr_start()</tt>.</span> 203 204 205 </p> 206 </li> 207 <li> 208 <p> 209 Open your databases as needed. Masters 210 must open their databases for read 211 and write activity. Replicas can open 212 their databases for read-only activity, but 213 doing so means they must re-open the 214 databases if the replica ever becomes a 215 master. Either way, replicas should never attempt to 216 write to the database(s) directly. 217 </p> 218 </li> 219 </ol> 220 </div> 221 <div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"> 222 <h3 class="title">Note</h3> 223 <p> 224 The replication framework allows you to only use one 225 environment handle per process. 226 </p> 227 </div> 228 <p> 229 When you are ready to shut down your application: 230 </p> 231 <div class="orderedlist"> 232 <ol type="1"> 233 <li> 234 <p> 235 Close your databases 236 </p> 237 </li> 238 <li> 239 <p> 240 Close your environment. This causes 241 replication to stop as well. 242 </p> 243 </li> 244 </ol> 245 </div> 246 <div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"> 247 <h3 class="title">Note</h3> 248 <p> 249 Before you can use the replication framework, you may have to 250 enable it in your DB library. This is 251 <span class="emphasis"><em>not</em></span> a requirement for 252 Microsoft Windows systems, or Unix systems that 253 use pthread mutexes by default. Other systems, 254 notably BSD and BSD-derived systems (such as 255 Mac OS X), must enable the replication framework when you 256 configure the DB build. 257 </p> 258 <p> 259 You do this by <span class="emphasis"><em>not</em></span> 260 disabling replication and by configuring the 261 library with POSIX threads support. In other 262 words, replication must be turned on in the 263 build (it is by default), and POSIX thread 264 support must be enabled if it is not already by 265 default. To do this, use the 266 <tt class="literal">--enable-pthread_api</tt> switch 267 on the configure script. 268 </p> 269 <p> 270 For example: 271 </p> 272 <pre class="programlisting">../dist/configure --enable-pthread-api</pre> 273 </div> 274 <div class="sect1" lang="en" xml:lang="en"> 275 <div class="titlepage"> 276 <div> 277 <div> 278 <h2 class="title" style="clear: both"><a id="rep_init_code"></a> 279 Starting and Stopping Replication 280 </h2> 281 </div> 282 </div> 283 <div></div> 284 </div> 285 <p> 286 As described above, you introduce replication to an 287 application by starting with a transactional 288 application, performing some basic replication 289 configuration, and then starting replication using 290 <span><tt class="methodname">DB_ENV->repmgr_start()</tt>.</span> 291 292 293 </p> 294 <p> 295 You stop replication by closing your environment 296 cleanly, as is normal for an DB application. 297 </p> 298 <p> 299 For example, the following code fragment initializes, then 300 stops and starts replication. Note that other replication 301 activities are omitted for brevity. 302 </p> 303 <pre class="programlisting">#include <db.h> 304 305/* Use a 10mb cache */ 306#define CACHESIZE (10 * 1024 * 1024) 307 308... 309 310 DB_ENV *dbenv; /* Environment handle. */ 311 const char *progname; /* Program name. */ 312 const char *envHome; /* Environment home directory. */ 313 const char *listen_host; /* A TCP/IP hostname. */ 314 const char *other_host; /* A TCP/IP hostname. */ 315 int ret; /* Error return code. */ 316 u_int16 listen_port; /* A TCP/IP port. */ 317 u_int16 other_port; /* A TCP/IP port. */ 318 319 /* Initialize variables */ 320 dbenv = NULL; 321 progname = "example_replication"; 322 envHome = "ENVIRONMENT_HOME"; 323 listen_host = "mymachine.sleepycat.com"; 324 listen_port = 5001; 325 other_host = "anothermachine.sleepycat.com"; 326 other_port = 4555; 327 ret = 0; 328 329 /* Create the environment handle */ 330 if ((ret = db_env_create(&dbenv, 0)) != 0 ) { 331 fprintf(stderr, "Error creating environment handle: %s\n", 332 db_strerror(ret)); 333 goto err; 334 } 335 336 /* 337 * Configure the environment handle. Here we configure asynchronous 338 * transactional commits for performance reasons. 339 */ 340 dbenv->set_errfile(dbenv, stderr); 341 dbenv->set_errpfx(dbenv, progname); 342 (void)dbenv->set_cachesize(dbenv, 0, CACHESIZE, 0); 343 (void)dbenv->set_flags(dbenv, DB_TXN_NOSYNC, 1); 344 345 346 /* 347 * Configure the local address. This is the local hostname and port 348 * that this replication participant will use to receive incoming 349 * replication messages. Note that this can be performed only once for 350 * the application. It is required. 351 */ 352 if ((ret = dbenv->repmgr_set_local_site(dbenv, listen_host, 353 listen_port, 0)) != 0) { 354 fprintf(stderr, "Could not set local address (%d).\n", ret); 355 goto err; 356 } 357 358 /* 359 * Set this application's priority. This is used for elections. 360 * 361 * Set this number to a positive integer, or 0 if you do not want this 362 * site to be able to become a master. 363 */ 364 dbenv->rep_set_priority(dbenv, 100); 365 366 /* 367 * Add a site to the list of replication environments known to this 368 * application. 369 */ 370 if (dbenv->repmgr_add_remote_site(dbenv, other_host, other_port, 371 NULL, 0) != 0) { 372 fprintf(stderr, "Could not add site %s.\n", other_host); 373 goto err; 374 } 375 376 /* 377 * Identify the number of sites in the replication group. This is 378 * necessary so that elections and permanent message handling can be 379 * performed correctly. 380 */ 381 if (dbenv->repmgr_set_nsites(dbenv, 2) != 0) { 382 fprintf(stderr, "Could not set the number of sites.\n"; 383 goto err; 384 } 385 386 /* Open the environment handle. Note that we add DB_THREAD and 387 * DB_INIT_REP to the list of flags. These are required. 388 */ 389 if ((ret = dbenv->open(dbenv, home, DB_CREATE | DB_RECOVER | 390 DB_INIT_LOCK | DB_INIT_LOG | 391 DB_INIT_MPOOL | DB_INIT_TXN | 392 DB_THREAD | DB_INIT_REP, 393 0)) != 0) { 394 goto err; 395 } 396 397 /* Start the replication framework such that it uses 3 threads. */ 398 if ((ret = dbenv->repmgr_start(dbenv, 3, DB_REP_ELECTION)) != 0) 399 goto err; 400 401 /* Sleep to give ourselves time to find a master */ 402 sleep(5); 403 404 /* 405 ********************************************************** 406 *** All other application code goes here, including ***** 407 *** database opens ***** 408 ********************************************************** 409 */ 410 411err: /* 412 * Make sure all your database handles are closed 413 * (omitted from this example). 414 */ 415 416 /* Close the environment */ 417 if (dbenv != NULL) 418 (void)dbenv->close(dbenv, 0); 419 420 /* All done */ 421 return (ret); </pre> 422 <div class="sect2" lang="en" xml:lang="en"> 423 <div class="titlepage"> 424 <div> 425 <div> 426 <h3 class="title"><a id="election_flags"></a>Managing Election Policies</h3> 427 </div> 428 </div> 429 <div></div> 430 </div> 431 <p> 432 Before continuing, it is worth taking a look at the 433 434 <span> 435 startup election flags accepted by 436 <span><tt class="methodname">DB_ENV->repgmr_start()</tt>.</span> 437 438 These flags control how your replication application will 439 behave when it first starts up. 440 </span> 441 442 443 </p> 444 <p> 445 In the previous example, we specified 446 <tt class="literal">DB_REP_ELECTION</tt> 447 448 when we started replication. This causes the 449 application to try to find a master upon startup. If it 450 cannot, it calls for an election. In the event an 451 election is held, the environment receiving the most number of 452 votes will become the master. 453 </p> 454 <p> 455 There's some important points to make here: 456 </p> 457 <div class="itemizedlist"> 458 <ul type="disc"> 459 <li> 460 <p> 461 This 462 <span>flag</span> 463 464 only requires that other 465 environments in the replication group 466 participate in the vote. There is no 467 requirement that 468 <span class="emphasis"><em>all</em></span> such 469 environments participate. In other 470 words, if an environment 471 starts up, it can call for an 472 election, and select a master, even 473 if all other environment have not yet 474 joined the replication group. 475 </p> 476 </li> 477 <li> 478 <p> 479 It only requires a simple majority of 480 participating environments to elect a master. The number of 481 environments used to calculate the simple 482 majority is based on the value set for 483 484 <span><tt class="methodname">DB_ENV->rep_set_nsites()</tt>.</span> 485 486 487 488 489 This is always true of elections held using the replication framework. 490 </p> 491 </li> 492 <li> 493 <p> 494 As always, the environment participating in the election with the most 495 up-to-date log files is selected as 496 master. If an environment with better log files 497 has not yet joined the replication 498 group, it may not become the master. 499 </p> 500 </li> 501 </ul> 502 </div> 503 <p> 504 Any one of these points may be enough to cause a 505 less-than-optimum environment to be selected as master. 506 Therefore, to give you a better degree of control over 507 which environment becomes a master at application startup, 508 the replication framework offers the following start-up 509 <span>flags:</span> 510 511 </p> 512 <div class="informaltable"> 513 <table border="1" width="80%"> 514 <colgroup> 515 <col /> 516 <col /> 517 </colgroup> 518 <thead> 519 <tr> 520 <th>Flag</th> 521 <th>Description</th> 522 </tr> 523 </thead> 524 <tbody> 525 <tr> 526 <td> 527 <tt class="literal">DB_REP_MASTER</tt> 528 </td> 529 <td> 530 <p> 531 The application starts up and declares itself to be a master 532 without calling for an election. It is an error for more 533 than one environment to start up using this flag, or for 534 an environment 535 to use this flag when a master already exists. 536 </p> 537 <p> 538 Note that no replication group should 539 <span class="emphasis"><em>ever</em></span> operate with more than 540 one master. 541 </p> 542 <p> 543 In the event that a environment attempts to become a 544 master when a master already exists, the 545 replication code will resolve the problem by 546 holding an election. Note, however, that there 547 is always a possibility of data loss in the face 548 of duplicate masters, because once a master is 549 selected, the environment that loses the election will 550 have to roll back any transactions committed 551 until it is in sync with the "real" master. 552 </p> 553 </td> 554 </tr> 555 <tr> 556 <td> 557 <tt class="literal">DB_REP_CLIENT</tt> 558 </td> 559 <td> 560 <p> 561 The application starts up and declares 562 itself to be a replica without calling for 563 an election. Note that the application 564 can still become a master if a subsequent 565 application starts up, calls for an 566 election, and this application is elected 567 master. 568 </p> 569 </td> 570 </tr> 571 <tr> 572 <td> 573 <tt class="literal">DB_REP_ELECTION</tt> 574 </td> 575 <td> 576 <p> 577 As described above, the application starts up, 578 looks for a master, and if one is not found calls 579 for an election. 580 </p> 581 </td> 582 </tr> 583 <tr> 584 <td> 585 <tt class="literal">DB_REP_FULL_ELECTION</tt> 586 </td> 587 <td> 588 <p> 589 Identical to 590 <tt class="literal">DB_REP_ELECTION</tt> 591 592 except that the election requires all 593 known members of the replication group to 594 participate. If a given environment has not yet 595 started but it is included in the 596 replication group count (using 597 <span><tt class="methodname">DB_ENV->rep_set_nsites()</tt>)</span> 598 599 600 then a master can not be elected. 601 </p> 602 </td> 603 </tr> 604 </tbody> 605 </table> 606 </div> 607 </div> 608 <div class="sect2" lang="en" xml:lang="en"> 609 <div class="titlepage"> 610 <div> 611 <div> 612 <h3 class="title"><a id="thread_count"></a>Selecting the Number of Threads</h3> 613 </div> 614 </div> 615 <div></div> 616 </div> 617 <p> 618 Under the hood, the replication framework is threaded and you can 619 control the number of threads used to process messages received from 620 other replicas. The threads that the replication framework uses are: 621 </p> 622 <div class="itemizedlist"> 623 <ul type="disc"> 624 <li> 625 <p> 626 Incoming message thread. This thread 627 receives messages from the site's 628 socket and passes those messages to 629 message processing threads (see below) 630 for handling. 631 </p> 632 </li> 633 <li> 634 <p> 635 Outgoing message thread. Outgoing 636 are performed in whatever thread 637 performed a write to the database(s). 638 That is, the thread that called, for 639 example, 640 <tt class="methodname">DB->put()</tt> 641 642 643 is the thread that writes replication messages 644 about that fact to the socket. 645 </p> 646 <p> 647 Note that if this write activity would 648 cause the thread to be blocked due to 649 some condition on the socket, the replication framework 650 will hand the outgoing message to the 651 incoming message thread, and it will 652 then write the message to the socket. 653 This prevents your database write 654 threads from blocking due to abnormal 655 network I/O conditions. 656 </p> 657 </li> 658 <li> 659 <p> 660 Message processing threads are 661 responsible for parsing and then 662 responding to incoming replication 663 messages. Typically, a response will 664 include write activity to your 665 database(s), so these threads can be 666 busy performing disk I/O. 667 </p> 668 </li> 669 </ul> 670 </div> 671 <p> 672 Of these threads, the only ones that you have any 673 configuration control over are the message processing 674 threads. In this case, you can determine how many 675 of these threads you want to run. 676 </p> 677 <p> 678 It is always a bit of an art to decide on a thread count, 679 but the short answer is you probably do not need more 680 than three threads here, and it is likely that one will 681 suffice. That said, the best thing to do is set your 682 thread count to a fairly low number and then increase 683 it if it appears that your application will benefit 684 from the additional threads. 685 </p> 686 </div> 687 </div> 688 </div> 689 <div class="navfooter"> 690 <hr /> 691 <table width="100%" summary="Navigation footer"> 692 <tr> 693 <td width="40%" align="left"><a accesskey="p" href="simpleprogramlisting.html">Prev</a>��</td> 694 <td width="20%" align="center"> 695 <a accesskey="u" href="index.html">Up</a> 696 </td> 697 <td width="40%" align="right">��<a accesskey="n" href="repmgr_init_example_c.html">Next</a></td> 698 </tr> 699 <tr> 700 <td width="40%" align="left" valign="top">Program Listing��</td> 701 <td width="20%" align="center"> 702 <a accesskey="h" href="index.html">Home</a> 703 </td> 704 <td width="40%" align="right" valign="top">��Adding the Replication Framework to 705 simple_txn 706 707 </td> 708 </tr> 709 </table> 710 </div> 711 </body> 712</html> 713