1<?xml version="1.0" encoding="UTF-8" standalone="no"?> 2<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 3<html xmlns="http://www.w3.org/1999/xhtml"> 4 <head> 5 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> 6 <title>In-Memory Transaction Example</title> 7 <link rel="stylesheet" href="gettingStarted.css" type="text/css" /> 8 <meta name="generator" content="DocBook XSL Stylesheets V1.62.4" /> 9 <link rel="home" href="index.html" title="Getting Started with Berkeley DB Transaction Processing" /> 10 <link rel="up" href="wrapup.html" title="Chapter 6. Summary and Examples" /> 11 <link rel="previous" href="txnexample_c.html" title="Transaction Example" /> 12 </head> 13 <body> 14 <div class="navheader"> 15 <table width="100%" summary="Navigation header"> 16 <tr> 17 <th colspan="3" align="center">In-Memory Transaction Example</th> 18 </tr> 19 <tr> 20 <td width="20%" align="left"><a accesskey="p" href="txnexample_c.html">Prev</a> </td> 21 <th width="60%" align="center">Chapter 6. Summary and Examples</th> 22 <td width="20%" align="right"> </td> 23 </tr> 24 </table> 25 <hr /> 26 </div> 27 <div class="sect1" lang="en" xml:lang="en"> 28 <div class="titlepage"> 29 <div> 30 <div> 31 <h2 class="title" style="clear: both"><a id="inmem_txnexample_c"></a>In-Memory Transaction Example</h2> 32 </div> 33 </div> 34 <div></div> 35 </div> 36 <p> 37 DB is sometimes used for applications that simply need to cache 38 data retrieved from some other location (such as a remote database 39 server). DB is also often used in embedded systems. 40 </p> 41 <p> 42 In both cases, applications may want to use transactions for 43 atomicity, consistency, and isolation guarantees, but they may also want 44 to forgo the durability guarantee entirely. In doing so, they can keep 45 their DB environment and databases entirely in-memory so 46 as to avoid the performance impact of unneeded disk I/O. 47 </p> 48 <p> 49 To do this: 50 </p> 51 <div class="itemizedlist"> 52 <ul type="disc"> 53 <li> 54 <p> 55 Refrain from specifying a home directory when you open your 56 environment. The exception to this is if you are using the 57 <tt class="literal">DB_CONFIG</tt> configuration file — in 58 that case you must identify the environment's home 59 directory so that the configuration file can be found. 60 </p> 61 </li> 62 <li> 63 <p> 64 Configure your environment to back your regions from 65 system memory instead of the filesystem. 66 </p> 67 </li> 68 <li> 69 <p> 70 Configure your logging subsystem such that log files are kept 71 entirely in-memory. 72 </p> 73 </li> 74 <li> 75 <p> 76 Increase the size of your in-memory log buffer so that it 77 is large enough to hold the largest set of concurrent write operations. 78 </p> 79 </li> 80 <li> 81 <p> 82 Increase the size of your in-memory cache so that it can 83 hold your entire data set. You do not want your cache to 84 page to disk. 85 </p> 86 </li> 87 <li> 88 <p> 89 Do not specify a file name when you open your database(s). 90 </p> 91 </li> 92 </ul> 93 </div> 94 <p> 95 As an example, this section takes the transaction example provided 96 in <a href="txnexample_c.html">Transaction Example</a> 97 and it updates that example so that the environment, database, log 98 files, and regions are all kept entirely in-memory. 99 </p> 100 <p> 101 For illustration purposes, we also modify this example so that 102 uncommitted reads are no longer used to enable the 103 <tt class="function">count_records()</tt> 104 105 function. Instead, we simply provide a transaction handle to 106 <tt class="function">count_records()</tt> 107 108 so as to avoid the self-deadlock. Be aware that using a transaction handle here rather than 109 uncommitted reads will work just as well as if we had continued to use uncommitted reads. However, 110 the usage of the transaction handle here will 111 probably cause more deadlocks than using read-uncommitted does, because more locking is being performed in 112 this case. 113 </p> 114 <p> 115 To begin, we simplify the beginning of our example a bit. Because 116 we no longer need an environment home directory, we can remove all 117 the code that we used to determine path delimiters 118 and include the <tt class="function">getopt</tt> function. We can also 119 remove our <tt class="function">usage()</tt> function because we no 120 longer require any command line arguments. 121 </p> 122 <pre class="programlisting">/* File: txn_guide_inmemory.c */ 123 124/* We assume an ANSI-compatible compiler */ 125#include <stdio.h> 126#include <stdlib.h> 127#include <string.h> 128#include <pthread.h> 129#include <db.h> 130 131/* Run 5 writers threads at a time. */ 132#define NUMWRITERS 5 133 134/* 135 * Printing of pthread_t is implementation-specific, so we 136 * create our own thread IDs for reporting purposes. 137 */ 138int global_thread_num; 139pthread_mutex_t thread_num_lock; 140 141/* Forward declarations */ 142int count_records(DB *, DB_TXN *); 143int open_db(DB **, const char *, const char *, DB_ENV *, u_int32_t); 144int writer_thread(void *); </pre> 145 <p> 146 Next, in our <tt class="function">main()</tt>, we also eliminate some 147 variables that this example no longer needs. In particular, we are able to remove 148 the 149 <tt class="literal">db_home_dir</tt> 150 151 and 152 <tt class="literal">file_name</tt> 153 154 variables. We also remove all our <tt class="function">getopt</tt> code. 155</p> 156 <pre class="programlisting">int 157main(void) 158{ 159 /* Initialize our handles */ 160 DB *dbp = NULL; 161 DB_ENV *envp = NULL; 162 163 pthread_t writer_threads[NUMWRITERS]; 164 int i, ret, ret_t; 165 u_int32_t env_flags; 166 167 /* Application name */ 168 const char *prog_name = "txn_guide_inmemory"; </pre> 169 <p> 170 Next we create our environment as always. However, we add 171 <tt class="literal">DB_PRIVATE</tt> to our environment open flags. This 172 flag causes our environment to back regions using our 173 application's heap memory rather than by using the filesystem. 174 This is the first important step to keeping our DB data 175 entirely in-memory. 176 </p> 177 <p> 178 We also remove the <tt class="literal">DB_RECOVER</tt> flag from the environment open flags. Because our databases, 179 logs, and regions are maintained in-memory, there will never be anything to recover. 180 </p> 181 <p> 182 Note that we show the additional code here in 183 <b class="userinput"><tt>bold.</tt></b> 184 </p> 185 <pre class="programlisting"> /* Create the environment */ 186 ret = db_env_create(&envp, 0); 187 if (ret != 0) { 188 fprintf(stderr, "Error creating environment handle: %s\n", 189 db_strerror(ret)); 190 goto err; 191 } 192 193 env_flags = 194 DB_CREATE | /* Create the environment if it does not exist */ 195 DB_INIT_LOCK | /* Initialize the locking subsystem */ 196 DB_INIT_LOG | /* Initialize the logging subsystem */ 197 DB_INIT_TXN | /* Initialize the transactional subsystem. This 198 * also turns on logging. */ 199 DB_INIT_MPOOL | /* Initialize the memory pool (in-memory cache) */ 200 <b class="userinput"><tt>DB_PRIVATE | /* Region files are not backed by the filesystem. 201 * Instead, they are backed by heap memory. */</tt></b> 202 DB_THREAD; /* Cause the environment to be free-threaded */ </pre> 203 <p> 204 Now we configure our environment to keep the log files in memory, 205 increase the log buffer size to 10 MB, and increase our in-memory 206 cache to 10 MB. These values should be more than enough for our 207 application's workload. 208 </p> 209 <pre class="programlisting"> 210 <b class="userinput"> 211 <tt> /* Specify in-memory logging */ 212 ret = envp->log_set_config(envp, DB_LOG_IN_MEMORY, 1); 213 if (ret != 0) { 214 fprintf(stderr, "Error setting log subsystem to in-memory: %s\n", 215 db_strerror(ret)); 216 goto err; 217 } 218 219 /* 220 * Specify the size of the in-memory log buffer. 221 */ 222 ret = envp->set_lg_bsize(envp, 10 * 1024 * 1024); 223 if (ret != 0) { 224 fprintf(stderr, "Error increasing the log buffer size: %s\n", 225 db_strerror(ret)); 226 goto err; 227 } 228 229 /* 230 * Specify the size of the in-memory cache. 231 */ 232 ret = envp->set_cachesize(envp, 0, 233 10 * 1024 * 1024, 1); 234 if (ret != 0) { 235 fprintf(stderr, "Error increasing the cache size: %s\n", 236 db_strerror(ret)); 237 goto err; 238 }</tt> 239 </b> 240 </pre> 241 <p> 242 Next, we open the environment and setup our lock detection. This is 243 identical to how the example previously worked, except that we do not 244 provide a location for the environment's home directory. 245 </p> 246 <pre class="programlisting"> /* 247 * Indicate that we want db to perform lock detection internally. 248 * Also indicate that the transaction with the fewest number of 249 * write locks will receive the deadlock notification in 250 * the event of a deadlock. 251 */ 252 ret = envp->set_lk_detect(envp, DB_LOCK_MINWRITE); 253 if (ret != 0) { 254 fprintf(stderr, "Error setting lock detect: %s\n", 255 db_strerror(ret)); 256 goto err; 257 } 258 259 /* Now actually open the environment */ 260 ret = envp->open(envp, <b class="userinput"><tt>NULL</tt></b>, env_flags, 0); 261 if (ret != 0) { 262 fprintf(stderr, "Error opening environment: %s\n", 263 db_strerror(ret)); 264 goto err; 265 } </pre> 266 <p> 267 When we call 268 <span><tt class="function">open_db()</tt>,</span> 269 270 which is what we use 271 to open our database, we no not provide a database filename for the 272 third parameter. When the filename is <tt class="literal">NULL</tt>, the database is not 273 backed by the filesystem. 274 </p> 275 <pre class="programlisting"> /* 276 * If we had utility threads (for running checkpoints or 277 * deadlock detection, for example) we would spawn those 278 * here. However, for a simple example such as this, 279 * that is not required. 280 */ 281 282 /* Open the database */ 283 ret = open_db(&dbp, prog_name, <b class="userinput"><tt>NULL</tt></b>, 284 envp, DB_DUPSORT); 285 if (ret != 0) 286 goto err; </pre> 287 <p> 288 After that, our <tt class="function">main()</tt> function is unchanged, 289 except that when we 290 <span>close the database,</span> 291 292 we change the error message string so as to not reference the database filename. 293 </p> 294 <pre class="programlisting"> /* Initialize a pthread mutex. Used to help provide thread ids. */ 295 (void)pthread_mutex_init(&thread_num_lock, NULL); 296 297 /* Start the writer threads. */ 298 for (i = 0; i < NUMWRITERS; i++) 299 (void)pthread_create( 300 &writer_threads[i], NULL, (void *)writer_thread, (void *)dbp); 301 302 /* Join the writers */ 303 for (i = 0; i < NUMWRITERS; i++) 304 (void)pthread_join(writer_threads[i], NULL); 305 306err: 307 /* Close our database handle, if it was opened. */ 308 if (dbp != NULL) { 309 ret_t = dbp->close(dbp, 0); 310 if (ret_t != 0) { 311 <b class="userinput"><tt>fprintf(stderr, "%s database close failed.\n", 312 db_strerror(ret_t));</tt></b> 313 ret = ret_t; 314 } 315 } 316 317 /* Close our environment, if it was opened. */ 318 if (envp != NULL) { 319 ret_t = envp->close(envp, 0); 320 if (ret_t != 0) { 321 fprintf(stderr, "environment close failed: %s\n", 322 db_strerror(ret_t)); 323 ret = ret_t; 324 } 325 } 326 327 /* Final status message and return. */ 328 printf("I'm all done.\n"); 329 return (ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE); 330} </pre> 331 <p> 332 That completes <tt class="function">main()</tt>. The bulk of our 333 <tt class="function">writer_thread()</tt> function implementation is 334 unchanged from the initial transaction example, except that we no 335 longer check for <tt class="literal">DB_KEYEXISTS</tt> in our 336 <tt class="methodname">DB->put()</tt> return code. Because we are 337 configuring for a completely in-memory database, there is no 338 possibility that we can run this code against an existing database. 339 Therefore, there is no way that <tt class="literal">DB_KEYEXISTS</tt> 340 will be returned by <tt class="methodname">DB->put()</tt>. 341 </p> 342 <pre class="programlisting">/* 343 * A function that performs a series of writes to a 344 * Berkeley DB database. The information written 345 * to the database is largely nonsensical, but the 346 * mechanism of transactional commit/abort and 347 * deadlock detection is illustrated here. 348 */ 349int 350writer_thread(void *args) 351{ 352 DBT key, value; 353 DB_TXN *txn; 354 int i, j, payload, ret, thread_num; 355 int retry_count, max_retries = 20; /* Max retry on a deadlock */ 356 char *key_strings[] = {"key 1", "key 2", "key 3", "key 4", 357 "key 5", "key 6", "key 7", "key 8", 358 "key 9", "key 10"}; 359 360 DB *dbp = (DB *)args; 361 DB_ENV *envp = dbp->get_env(dbp); 362 363 /* Get the thread number */ 364 (void)pthread_mutex_lock(&thread_num_lock); 365 global_thread_num++; 366 thread_num = global_thread_num; 367 (void)pthread_mutex_unlock(&thread_num_lock); 368 369 /* Initialize the random number generator */ 370 srand((u_int)pthread_self()); 371 372 /* Write 50 times and then quit */ 373 for (i = 0; i < 50; i++) { 374 retry_count = 0; /* Used for deadlock retries */ 375 376retry: 377 ret = envp->txn_begin(envp, NULL, &txn, 0); 378 if (ret != 0) { 379 envp->err(envp, ret, "txn_begin failed"); 380 return (EXIT_FAILURE); 381 } 382 for (j = 0; j < 10; j++) { 383 /* Set up our key and values DBTs */ 384 memset(&key, 0, sizeof(DBT)); 385 key.data = key_strings[j]; 386 key.size = (strlen(key_strings[j]) + 1) * sizeof(char); 387 388 memset(&value, 0, sizeof(DBT)); 389 payload = rand() + i; 390 value.data = &payload; 391 value.size = sizeof(int); 392 393 /* Perform the database put. */ 394 switch (ret = dbp->put(dbp, txn, &key, &value, 0)) { 395 case 0: 396 break; 397 398 /* 399 * Here's where we perform deadlock detection. If 400 * DB_LOCK_DEADLOCK is returned by the put operation, 401 * then this thread has been chosen to break a deadlock. 402 * It must abort its operation, and optionally retry the 403 * put. 404 */ 405 case DB_LOCK_DEADLOCK: 406 /* 407 * First that we MUST do is abort the 408 * transaction. 409 */ 410 (void)txn->abort(txn); 411 /* 412 * Now we decide if we want to retry the operation. 413 * If we have retried less than max_retries, 414 * increment the retry count and goto retry. 415 */ 416 if (retry_count < max_retries) { 417 printf("Writer %i: Got DB_LOCK_DEADLOCK.\n", 418 thread_num); 419 printf("Writer %i: Retrying write operation.\n", 420 thread_num); 421 retry_count++; 422 goto retry; 423 } 424 /* 425 * Otherwise, just give up. 426 */ 427 printf("Writer %i: ", thread_num); 428 printf("Got DB_LOCK_DEADLOCK and out of retries.\n"); 429 printf("Writer %i: Giving up.\n", thread_num); 430 return (EXIT_FAILURE); 431 /* 432 * If a generic error occurs, we simply abort the 433 * transaction and exit the thread completely. 434 */ 435 default: 436 envp->err(envp, ret, "db put failed"); 437 ret = txn->abort(txn); 438 if (ret != 0) 439 envp->err(envp, ret, "txn abort failed"); 440 return (EXIT_FAILURE); 441 } /** End case statement **/ 442 443 } /** End for loop **/ </pre> 444 <p> 445 The only other change to <tt class="function">writer_thread()</tt> 446 is that we pass <tt class="function">count_records()</tt> a 447 transaction handle, rather than configuring our entire 448 application for uncommitted reads. Both mechanisms work well-enough 449 for preventing a self-deadlock. However, the individual count 450 in this example will tend to be lower than the counts seen in 451 the previous transaction example, because 452 <tt class="function">count_records()</tt> can no longer see records 453 created but not yet committed by other threads. 454 </p> 455 <pre class="programlisting"> /* 456 * print the number of records found in the database. 457 * See count_records() for usage information. 458 */ 459 printf("Thread %i. Record count: %i\n", thread_num, 460 count_records(dbp, <b class="userinput"><tt>txn</tt></b>)); 461 462 /* 463 * If all goes well, we can commit the transaction and 464 * loop to the next transaction. 465 */ 466 ret = txn->commit(txn, 0); 467 if (ret != 0) { 468 envp->err(envp, ret, "txn commit failed"); 469 return (EXIT_FAILURE); 470 } 471 } 472 return (EXIT_SUCCESS); 473} </pre> 474 <p> 475 Next we update 476 <span><tt class="function">count_records()</tt>.</span> 477 478 The only difference 479 here is that we no longer specify <tt class="literal">DB_READ_UNCOMMITTED</tt> when 480 we open our cursor. Note that even this minor change is not required. 481 If we do not configure our database to support uncommitted reads, 482 <tt class="literal">DB_READ_UNCOMMITTED</tt> on the cursor open will be silently 483 ignored. However, we remove the flag anyway from the cursor open so as to 484 avoid confusion. 485</p> 486 <pre class="programlisting">int 487count_records(DB *dbp, DB_TXN *txn) 488{ 489 DBT key, value; 490 DBC *cursorp; 491 int count, ret; 492 493 cursorp = NULL; 494 count = 0; 495 496 /* Get the cursor */ 497 ret = dbp->cursor(dbp, txn, &cursorp, <b class="userinput"><tt>0</tt></b>); 498 if (ret != 0) { 499 dbp->err(dbp, ret, "count_records: cursor open failed."); 500 goto cursor_err; 501 } 502 503 /* Get the key DBT used for the database read */ 504 memset(&key, 0, sizeof(DBT)); 505 memset(&value, 0, sizeof(DBT)); 506 do { 507 ret = cursorp->get(cursorp, &key, &value, DB_NEXT); 508 switch (ret) { 509 case 0: 510 count++; 511 break; 512 case DB_NOTFOUND: 513 break; 514 default: 515 dbp->err(envp, ret, 516 "Count records unspecified error"); 517 goto cursor_err; 518 } 519 } while (ret == 0); 520 521cursor_err: 522 if (cursorp != NULL) { 523 ret = cursorp->close(cursorp); 524 if (ret != 0) { 525 dbp->err(dbp, ret, 526 "count_records: cursor close failed."); 527 } 528 } 529 530 return (count); 531}</pre> 532 <p> 533 Finally, we update 534 <span><tt class="function">open_db()</tt>.</span> 535 536 This involves 537 removing <tt class="literal">DB_READ_UNCOMMITTED</tt> from the 538 open flags. 539 540 <span>We are also careful to change our database open error 541 message to no longer use the <tt class="literal">file_name</tt> 542 variable because that value will always be <tt class="literal">NULL</tt> for this example.</span> 543 </p> 544 <pre class="programlisting">/* Open a Berkeley DB database */ 545int 546open_db(DB **dbpp, const char *progname, const char *file_name, 547 DB_ENV *envp, u_int32_t extra_flags) 548{ 549 int ret; 550 u_int32_t open_flags; 551 DB *dbp; 552 553 /* Initialize the DB handle */ 554 ret = db_create(&dbp, envp, 0); 555 if (ret != 0) { 556 fprintf(stderr, "%s: %s\n", progname, 557 db_strerror(ret)); 558 return (EXIT_FAILURE); 559 } 560 561 /* Point to the memory malloc'd by db_create() */ 562 *dbpp = dbp; 563 564 if (extra_flags != 0) { 565 ret = dbp->set_flags(dbp, extra_flags); 566 if (ret != 0) { 567 dbp->err(dbp, ret, 568 "open_db: Attempt to set extra flags failed."); 569 return (EXIT_FAILURE); 570 } 571 } 572 573 /* Now open the database */ 574 <b class="userinput"><tt>open_flags = DB_CREATE | /* Allow database creation */ 575 DB_THREAD | 576 DB_AUTO_COMMIT; /* Allow auto commit */</tt></b> 577 578 ret = dbp->open(dbp, /* Pointer to the database */ 579 NULL, /* Txn pointer */ 580 file_name, /* File name */ 581 NULL, /* Logical db name */ 582 DB_BTREE, /* Database type (using btree) */ 583 open_flags, /* Open flags */ 584 0); /* File mode. Using defaults */ 585 if (ret != 0) { 586 <b class="userinput"><tt>dbp->err(dbp, ret, "Database open failed"); 587 return (EXIT_FAILURE);</tt></b> 588 } 589 return (EXIT_SUCCESS); 590} </pre> 591 <p> 592 This completes our in-memory transactional example. If you would like to 593 experiment with this code, you can find the example in the following 594 location in your DB distribution: 595</p> 596 <pre class="programlisting"><span class="emphasis"><em>DB_INSTALL</em></span>/examples_c/txn_guide</pre> 597 </div> 598 <div class="navfooter"> 599 <hr /> 600 <table width="100%" summary="Navigation footer"> 601 <tr> 602 <td width="40%" align="left"><a accesskey="p" href="txnexample_c.html">Prev</a> </td> 603 <td width="20%" align="center"> 604 <a accesskey="u" href="wrapup.html">Up</a> 605 </td> 606 <td width="40%" align="right"> </td> 607 </tr> 608 <tr> 609 <td width="40%" align="left" valign="top">Transaction Example </td> 610 <td width="20%" align="center"> 611 <a accesskey="h" href="index.html">Home</a> 612 </td> 613 <td width="40%" align="right" valign="top"> </td> 614 </tr> 615 </table> 616 </div> 617 </body> 618</html> 619