1<?xml version="1.0" encoding="UTF-8" standalone="no"?> 2<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 3<html xmlns="http://www.w3.org/1999/xhtml"> 4 <head> 5 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> 6 <title>Example Processing Loop</title> 7 <link rel="stylesheet" href="gettingStarted.css" type="text/css" /> 8 <meta name="generator" content="DocBook XSL Stylesheets V1.62.4" /> 9 <link rel="home" href="index.html" title="Getting Started with Replicated Berkeley DB Applications" /> 10 <link rel="up" href="fwrkmasterreplica.html" title="Chapter 4. Replica versus Master Processes" /> 11 <link rel="previous" href="processingloop.html" title="Processing Loop" /> 12 <link rel="next" href="addfeatures.html" title="Chapter 5. Additional Features" /> 13 </head> 14 <body> 15 <div class="navheader"> 16 <table width="100%" summary="Navigation header"> 17 <tr> 18 <th colspan="3" align="center">Example Processing Loop</th> 19 </tr> 20 <tr> 21 <td width="20%" align="left"><a accesskey="p" href="processingloop.html">Prev</a> </td> 22 <th width="60%" align="center">Chapter 4. Replica versus Master Processes</th> 23 <td width="20%" align="right"> <a accesskey="n" href="addfeatures.html">Next</a></td> 24 </tr> 25 </table> 26 <hr /> 27 </div> 28 <div class="sect1" lang="en" xml:lang="en"> 29 <div class="titlepage"> 30 <div> 31 <div> 32 <h2 class="title" style="clear: both"><a id="exampledoloop"></a>Example Processing Loop</h2> 33 </div> 34 </div> 35 <div></div> 36 </div> 37 <p> 38 In this section we take the example 39 processing loop that we presented in the 40 previous section and we flesh it out to 41 provide a more complete example. We do this 42 by updating the 43 <tt class="function">doloop()</tt> 44 function that our original transaction 45 application used 46 <span>(see <a href="simpleprogramlisting.html#doloop_c">Function: doloop()</a>)</span> 47 48 49 to fully support our replicated application. 50 </p> 51 <p> 52 In the following example code, code that we 53 add to the original example is presented in 54 <b class="userinput"><tt>bold</tt></b>. 55 </p> 56 <p> 57 To begin, we include a new header file into 58 our application so that we can check for the 59 <tt class="literal">ENOENT</tt> return value later 60 in our processing loop. We also define our 61 <tt class="literal">APP_DATA</tt> 62 structure, and we define a 63 <tt class="literal">sleeptime</tt> value. 64 65 <span> 66 Finally, we 67 add a new forward declaration for our event 68 callback. 69 </span> 70 71 72 </p> 73 <pre class="programlisting">/* 74 * File: rep_mgr.c 75 */ 76 77#include <stdlib.h> 78#include <string.h> 79<b class="userinput"><tt>#include <errno.h></tt></b> 80#ifndef _WIN32 81#include <unistd.h> 82#endif 83 84#include <db.h> 85 86#ifdef _WIN32 87extern int getopt(int, char * const *, const char *); 88#endif 89 90 91#define CACHESIZE (10 * 1024 * 1024) 92#define DATABASE "quote.db" 93<b class="userinput"><tt>#define SLEEPTIME 3</tt></b> 94 95const char *progname = "rep_mgr"; 96 97<b class="userinput"><tt>typedef struct { 98 int is_master; 99} APP_DATA;</tt></b> 100 101int create_env(const char *, DB_ENV **); 102int env_init(DB_ENV *, const char *); 103int doloop (DB_ENV *); 104int print_stocks(DBC *); 105<b class="userinput"><tt>void *event_callback(DB_ENV *, u_int32_t, void *);</tt></b> </pre> 106 <p> 107 In our <tt class="function">main()</tt> function, most of what we 108 have to add to it is some new variable declarations and 109 initializations: 110</p> 111 <pre class="programlisting">int 112main(int argc, char *argv[]) 113{ 114 extern char *optarg; 115 DB_ENV *dbenv; 116 const char *home; 117 char ch, *host, *portstr; 118 int ret, local_is_set, totalsites; 119 u_int32_t port; 120 <b class="userinput"><tt>/* Used to track whether this is a replica or a master */ 121 APP_DATA my_app_data; 122 123 my_app_data.is_master = 0; /* Assume that we start as a replica */</tt></b> 124 dbenv = NULL; 125 126 ret = local_is_set = totalsites = 0; 127 home = NULL;</pre> 128 <p> 129 The rest of our <tt class="function">main()</tt> function 130 is unchanged, except that we make our 131 <tt class="literal">APP_DATA</tt> structure available through our 132 environment handle's <tt class="methodname">app_private</tt> 133 field: 134 </p> 135 <pre class="programlisting"> 136 if ((ret = create_env(progname, &dbenv)) != 0) 137 goto err; 138 139 <b class="userinput"><tt>/* Make APP_DATA available through the environment handle */ 140 dbenv->app_private = &my_app_data;</tt></b> 141 142 /* Default priority is 100 */ 143 dbenv->rep_set_priority(dbenv, 100); 144 145 /* Collect the command line options */ 146 while ((ch = getopt(argc, argv, "h:m:o:p:")) != EOF) 147 switch (ch) { 148 case 'h': 149 home = optarg; 150 break; 151 /* Set the host and port used by this environment */ 152 case 'm': 153 host = strtok(optarg, ":"); 154 if ((portstr = strtok(NULL, ":")) == NULL) { 155 fprintf(stderr, "Bad host specification.\n"); 156 goto err; 157 } 158 port = (unsigned short)atoi(portstr); 159 if (dbenv->repmgr_set_local_site(dbenv, host, port, 0) != 0) { 160 fprintf(stderr, 161 "Could not set local address %s.\n", host); 162 goto err; 163 } 164 local_is_set = 1; 165 break; 166 /* Identify another site in the replication group */ 167 case 'o': 168 host = strtok(optarg, ":"); 169 if ((portstr = strtok(NULL, ":")) == NULL) { 170 fprintf(stderr, "Bad host specification.\n"); 171 goto err; 172 } 173 port = (unsigned short)atoi(portstr); 174 if (dbenv->repmgr_add_remote_site(dbenv, host, port, 0) != 0) { 175 fprintf(stderr, 176 "Could not add site %s.\n", host); 177 goto err; 178 } 179 break; 180 /* Set the number of sites in this replication group */ 181 case 'n': 182 totalsites = atoi(optarg); 183 if ((ret = dbenv->rep_set_nsites(dbenv, totalsites)) != 0) 184 dbenv->err(dbenv, ret, "set_nsites"); 185 break; 186 /* Set this replica's election priority */ 187 case 'p': 188 dbenv->rep_set_priority(dbenv, atoi(optarg)); 189 break; 190 case '?': 191 default: 192 usage(); 193 } 194 195 /* Error check command line. */ 196 if (home == NULL || !local_is_set || !totalsites) 197 usage(); 198 199 if ((ret = env_init(dbenv, home)) != 0) 200 goto err; 201 202 if ((ret = dbenv->repmgr_start(dbenv, 3, DB_REP_ELECTION)) != 0) 203 goto err; 204 205 /* Sleep to give ourselves time to find a master. */ 206 sleep(5); 207 208 if ((ret = doloop(dbenv)) != 0) { 209 dbenv->err(dbenv, ret, "Application failed"); 210 goto err; 211 } 212 213err: if (dbenv != NULL) 214 (void)dbenv->close(dbenv, 0); 215 216 return (ret); 217} </pre> 218 <p> 219 Having updated our <tt class="function">main()</tt>, we must also 220 update our <tt class="function">create_env()</tt> function to 221 register our <tt class="function">event_callback</tt> callback. 222 Notice that our <tt class="function">env_init()</tt> function, which is 223 responsible for actually opening our environment handle, is 224 unchanged: 225 </p> 226 <pre class="programlisting">int 227create_env(char *progname, DB_ENV **dbenvp) 228{ 229 DB_ENV *dbenv; 230 int ret; 231 232 if ((ret = db_env_create(&dbenv, 0)) != 0) { 233 fprintf(stderr, "can't create env handle: %s\n", 234 db_strerror(ret)); 235 return (ret); 236 } 237 238 dbenv->set_errfile(dbenv, stderr); 239 dbenv->set_errpfx(dbenv, progname); 240 <b class="userinput"><tt>(void)dbenv->set_event_notify(dbenv, event_callback);</tt></b> 241 242 *dbenvp = dbenv; 243 return (0); 244} 245 246int 247env_init(DB_ENV *dbenv, const char *home) 248{ 249 u_int32_t flags; 250 int ret; 251 252 (void)dbenv->set_cachesize(dbenv, 0, CACHESIZE, 0); 253 (void)dbenv->set_flags(dbenv, DB_TXN_NOSYNC, 1); 254 255 flags = DB_CREATE | 256 DB_INIT_LOCK | 257 DB_INIT_LOG | 258 DB_INIT_MPOOL | 259 DB_INIT_TXN | 260 DB_INIT_REP | 261 DB_THREAD | 262 DB_RECOVER; 263 if ((ret = dbenv->open(dbenv, home, flags, 0)) != 0) 264 dbenv->err(dbenv, ret, "can't open environment"); 265 return (ret); 266}</pre> 267 <p> 268 That done, we need to implement our 269 <tt class="function">event_callback()</tt> callback. Note that what we use 270 here is no different from the callback that we described in 271 the previous section. However, for the sake of completeness we 272 provide the implementation here again. 273</p> 274 <pre class="programlisting"> 275 <b class="userinput"> 276 <tt>/* 277 * A callback used to determine whether the local environment is a replica 278 * or a master. This is called by the replication framework 279 * when the local replication environment changes state. 280 */ 281void * 282event_callback(DB_ENV *dbenv, u_int32_t which, void *info) 283{ 284 APP_DATA *app = dbenv->app_private; 285 286 info = NULL; /* Currently unused. */ 287 288 switch (which) { 289 case DB_EVENT_REP_MASTER: 290 app->is_master = 1; 291 break; 292 293 case DB_EVENT_REP_CLIENT: 294 app->is_master = 0; 295 break; 296 297 case DB_EVENT_REP_STARTUPDONE: /* fallthrough */ 298 case DB_EVENT_REP_NEWMASTER: 299 /* Ignore. */ 300 break; 301 302 default: 303 dbenv->errx(dbenv, "ignoring event %d", which); 304 } 305}</tt> 306 </b> 307 </pre> 308 <p> 309 That done, we need to update our 310 <tt class="function">doloop()</tt> 311 <span>function. This is the place where we most 312 heavily modify our application.</span> 313 314 315</p> 316 <p> 317 We begin by introducing <tt class="literal">APP_DATA</tt> to the 318 function: 319</p> 320 <pre class="programlisting">/* 321 * Provides the main data processing function for our application. 322 * This function provides a command line prompt to which the user 323 * can provide a ticker string and a stock price. Once a value is 324 * entered to the application, the application writes the value to 325 * the database and then displays the entire database. 326 */ 327#define BUFSIZE 1024 328int 329doloop(DB_ENV *dbenv) 330{ 331 DB *dbp; 332 <b class="userinput"><tt>APP_DATA *app_data;</tt></b> 333 DBT key, data; 334 char buf[BUFSIZE], *rbuf; 335 int ret; 336 u_int32_t flags; 337 338 dbp = NULL; 339 ret = 0; 340 memset(&key, 0, sizeof(key)); 341 memset(&data, 0, sizeof(data)); 342 <b class="userinput"><tt>app_data = dbenv->app_private;</tt></b></pre> 343 <p> 344 Next we begin to modify our main loop. 345 To start, upon entering the loop we create the database handle and configure it as 346 normal. But we also have to decide what flags we will use for 347 the open. Again, it depends on whether we are a replica or a 348 master. 349 </p> 350 <pre class="programlisting"> for (;;) { 351 if (dbp == NULL) { 352 if ((ret = db_create(&dbp, dbenv, 0)) != 0) 353 return (ret); 354 355 /* Set page size small so page allocation is cheap. */ 356 if ((ret = dbp->set_pagesize(dbp, 512)) != 0) 357 goto err; 358 359 <b class="userinput"><tt>flags = DB_AUTO_COMMIT; 360 if (app_data->is_master) 361 flags |= DB_CREATE;</tt></b> </pre> 362 <p> 363 When we open the database, we modify our error handling to 364 account for the case where the database does not yet exist. This can 365 happen if our code is running as a replica and the replication framework has not 366 yet had a chance to create the databases for us. Recall that replicas never 367 write to their own databases directly, and so they cannot 368 create databases on their own. 369</p> 370 <p> 371 If we detect that the database does not yet exist, we simply 372 close the database handle, sleep for a short period of time 373 and then continue processing. This gives the replication framework a chance to 374 create the database so that our replica can continue 375 operations. 376</p> 377 <pre class="programlisting"> if ((ret = dbp->open(dbp, 378 NULL, DATABASE, NULL, DB_BTREE, flags, 0)) != 0) { 379 <b class="userinput"><tt>if (ret == ENOENT) { 380 printf( 381 "No stock database yet available.\n"); 382 if ((ret = dbp->close(dbp, 0)) != 0) { 383 dbenv->err(dbenv, ret, 384 "DB->close"); 385 goto err; 386 } 387 dbp = NULL; 388 sleep(SLEEPTIME); 389 continue; 390 }</tt></b> 391 dbenv->err(dbenv, ret, "DB->open"); 392 goto err; 393 } 394 } </pre> 395 <p> 396 Next we modify our prompt, so that if the local process is running 397 as a replica, we can tell from the shell that the prompt is for a 398 read-only process. 399 </p> 400 <pre class="programlisting"> printf("QUOTESERVER<b class="userinput"><tt>%s</tt></b>> ", 401 <b class="userinput"><tt>app_data->is_master ? "" : " (read-only)");</tt></b> 402 fflush(stdout); </pre> 403 <p> 404 When we collect data from the prompt, there is a case that says 405 if no data is entered then show the entire stocks database. 406 This display is performed by our 407 <tt class="function">print_stocks()</tt> 408 <span>function</span> 409 410 (which has not 411 required a modification since we first introduced it in 412 <a href="simpleprogramlisting.html#printstocks_c"> 413 <span>Function: print_stocks()</span> 414 415 416 </a>). 417 </p> 418 <p> 419 When we call 420 <span><tt class="function">print_stocks()</tt>, </span> 421 422 we check for a dead replication handle. Dead 423 replication handles happen whenever a replication election 424 results in a previously committed transaction becoming 425 invalid. This is an error scenario caused by a new master having a 426 slightly older version of the data than the original 427 master and so all replicas must modify their database(s) to 428 reflect that of the new master. In this situation, some 429 number of previously committed transactions may have to be 430 unrolled. From the replica's perspective, the database 431 handles should all be closed and then opened again. 432 </p> 433 <pre class="programlisting"> if (fgets(buf, sizeof(buf), stdin) == NULL) 434 break; 435 if (strtok(&buf[0], " \t\n") == NULL) { 436 switch ((ret = print_stocks(dbp))) { 437 case 0: 438 continue; 439 <b class="userinput"><tt>case DB_REP_HANDLE_DEAD: 440 (void)dbp->close(dbp, DB_NOSYNC); 441 dbp = NULL; 442 dbenv->errx(dbenv, "Got a dead replication handle"); 443 continue; </tt></b> 444 default: 445 dbp->err(dbp, ret, "Error traversing data"); 446 goto err; 447 } 448 } 449 rbuf = strtok(NULL, " \t\n"); 450 if (rbuf == NULL || rbuf[0] == '\0') { 451 if (strncmp(buf, "exit", 4) == 0 || 452 strncmp(buf, "quit", 4) == 0) 453 break; 454 dbenv->errx(dbenv, "Format: TICKER VALUE"); 455 continue; 456 }</pre> 457 <p> 458 That done, we need to add a little error checking to our 459 command prompt to make sure the user is not attempting to 460 modify the database at a replica. Remember, replicas must never 461 modify their local databases on their own. This guards against 462 that happening due to user input at the prompt. 463 </p> 464 <pre class="programlisting"> <b class="userinput"><tt>if (!app_data->is_master) { 465 dbenv->errx(dbenv, "Can't update at client"); 466 continue; 467 }</tt></b> 468 key.data = buf; 469 key.size = (u_int32_t)strlen(buf); 470 471 data.data = rbuf; 472 data.size = (u_int32_t)strlen(rbuf); 473 474 if ((ret = dbp->put(dbp, 475 NULL, &key, &data, 0)) != 0) { 476 dbp->err(dbp, ret, "DB->put"); 477 goto err; 478 } 479 } 480 481err: if (dbp != NULL) 482 (void)dbp->close(dbp, DB_NOSYNC); 483 484 return (ret); 485} </pre> 486 <p> 487 With that completed, we are all done updating our application 488 for replication. 489 490 The only remaining 491 <span>function, <tt class="function">print_stocks()</tt>,</span> 492 493 494 495 is unmodified from when we 496 originally introduced it. For details on that function, see 497 <a href="simpleprogramlisting.html#printstocks_c"> 498 <span>Function: print_stocks()</span> 499 500 501 </a>. 502</p> 503 <div class="sect2" lang="en" xml:lang="en"> 504 <div class="titlepage"> 505 <div> 506 <div> 507 <h3 class="title"><a id="runningit"></a>Running It</h3> 508 </div> 509 </div> 510 <div></div> 511 </div> 512 <p> 513 To run our replicated application, we need to make 514 sure each participating environment has its own unique 515 home directory. We can do this by running 516 each site on a separate networked machine, but that 517 is not strictly necessary; multiple instances of this 518 code can run on the same machine provided the 519 environment home restriction is observed. 520 </p> 521 <p> 522 To run a process, make sure the environment home 523 exists and then start the process using the 524 <tt class="literal">-h</tt> option to specify that 525 directory. You must also use the <tt class="literal">-m</tt> 526 option to identify the local host and port that this 527 process will use to listen for replication messages, and 528 the <tt class="literal">-o</tt> option to identify the other 529 processes in the replication group. Finally, use the 530 <tt class="literal">-p</tt> option to specify a priority. 531 The process that you designate to have the highest priority will become 532 the master. 533 </p> 534 <pre class="programlisting">> mkdir env1 535> /rep_mgr -h env1 -n 2 -m localhost:8080 -o localhost:8081 -p 10 536No stock database yet available. 537No stock database yet available. </pre> 538 <p> 539 Now, start another process. This time, change the environment 540 home to something else, use the <tt class="literal">-m</tt> to at 541 least change the port number the process is listening on, and 542 use the <tt class="literal">-o</tt> option to identify the host and 543 port of the other replication process: 544</p> 545 <pre class="programlisting">> mkdir env2 546> /rep_mgr -h env2 -n 2 -m localhost:8081 -o localhost:8080 -p 20</pre> 547 <p> 548 After a short pause, the second process should display the master 549 prompt: 550</p> 551 <pre class="programlisting"> 552QUOTESERVER > </pre> 553 <p> 554 And the first process should 555 display the read-only prompt: 556</p> 557 <pre class="programlisting"> 558QUOTESERVER (read-only)> </pre> 559 <p> 560 Now go to the master process and give it a couple of stocks and stock 561 prices: 562</p> 563 <pre class="programlisting">QUOTESERVER> FAKECO 9.87 564QUOTESERVER> NOINC .23 565QUOTESERVER> </pre> 566 <p> 567 Then, go to the replica and hit <b class="userinput"><tt>return</tt></b> at the prompt to 568 see the new values: 569</p> 570 <pre class="programlisting">QUOTESERVER (read-only)> 571 Symbol Price 572 ====== ===== 573 FAKECO 9.87 574 NOINC .23 575QUOTESERVER (read-only)> </pre> 576 <p> 577 Doing the same at the master results in the same thing: 578</p> 579 <pre class="programlisting">QUOTESERVER> 580 Symbol Price 581 ====== ===== 582 FAKECO 9.87 583 NOINC .23 584QUOTESERVER> </pre> 585 <p> 586 You can change a stock by simply entering the stock value and 587 new price at the master's prompt: 588</p> 589 <pre class="programlisting">QUOTESERVER> FAKECO 10.01 590QUOTESERVER> </pre> 591 <p> 592 Then, go to either the master or the replica to see the updated 593 database: 594</p> 595 <pre class="programlisting">QUOTESERVER> 596 Symbol Price 597 ====== ===== 598 FAKECO 10.01 599 NOINC .23 600QUOTESERVER> </pre> 601 <p> 602 And on the replica: 603</p> 604 <pre class="programlisting">QUOTESERVER (read-only)> 605 Symbol Price 606 ====== ===== 607 FAKECO 10.01 608 NOINC .23 609QUOTESERVER (read-only)> </pre> 610 <p> 611 Finally, to quit the applications, simply type 612 <tt class="literal">quit</tt> at both prompts: 613</p> 614 <pre class="programlisting">QUOTESERVER (read-only)> quit 615> </pre> 616 <p> 617 And on the master as well: 618</p> 619 <pre class="programlisting">QUOTESERVER> quit 620> </pre> 621 </div> 622 </div> 623 <div class="navfooter"> 624 <hr /> 625 <table width="100%" summary="Navigation footer"> 626 <tr> 627 <td width="40%" align="left"><a accesskey="p" href="processingloop.html">Prev</a> </td> 628 <td width="20%" align="center"> 629 <a accesskey="u" href="fwrkmasterreplica.html">Up</a> 630 </td> 631 <td width="40%" align="right"> <a accesskey="n" href="addfeatures.html">Next</a></td> 632 </tr> 633 <tr> 634 <td width="40%" align="left" valign="top">Processing Loop </td> 635 <td width="20%" align="center"> 636 <a accesskey="h" href="index.html">Home</a> 637 </td> 638 <td width="40%" align="right" valign="top"> Chapter 5. Additional Features</td> 639 </tr> 640 </table> 641 </div> 642 </body> 643</html> 644