1<?xml version="1.0" encoding="UTF-8" standalone="no"?> 2<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 3<html xmlns="http://www.w3.org/1999/xhtml"> 4 <head> 5 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> 6 <title>In-Memory Transaction Example</title> 7 <link rel="stylesheet" href="gettingStarted.css" type="text/css" /> 8 <meta name="generator" content="DocBook XSL Stylesheets V1.73.2" /> 9 <link rel="start" href="index.html" title="Getting Started with Berkeley DB Transaction Processing" /> 10 <link rel="up" href="wrapup.html" title="Chapter��6.��Summary and Examples" /> 11 <link rel="prev" href="txnexample_c.html" title="Transaction Example" /> 12 </head> 13 <body> 14 <div class="navheader"> 15 <table width="100%" summary="Navigation header"> 16 <tr> 17 <th colspan="3" align="center">In-Memory Transaction Example</th> 18 </tr> 19 <tr> 20 <td width="20%" align="left"><a accesskey="p" href="txnexample_c.html">Prev</a>��</td> 21 <th width="60%" align="center">Chapter��6.��Summary and Examples</th> 22 <td width="20%" align="right">��</td> 23 </tr> 24 </table> 25 <hr /> 26 </div> 27 <div class="sect1" lang="en" xml:lang="en"> 28 <div class="titlepage"> 29 <div> 30 <div> 31 <h2 class="title" style="clear: both"><a id="inmem_txnexample_c"></a>In-Memory Transaction Example</h2> 32 </div> 33 </div> 34 </div> 35 <p> 36 DB is sometimes used for applications that simply need to cache 37 data retrieved from some other location (such as a remote database 38 server). DB is also often used in embedded systems. 39 </p> 40 <p> 41 In both cases, applications may want to use transactions for 42 atomicity, consistency, and isolation guarantees, but they may also want 43 to forgo the durability guarantee entirely. In doing so, they can keep 44 their DB environment and databases entirely in-memory so 45 as to avoid the performance impact of unneeded disk I/O. 46 </p> 47 <p> 48 To do this: 49 </p> 50 <div class="itemizedlist"> 51 <ul type="disc"> 52 <li> 53 <p> 54 Refrain from specifying a home directory when you open your 55 environment. The exception to this is if you are using the 56 <code class="literal">DB_CONFIG</code> configuration file ��� in 57 that case you must identify the environment's home 58 directory so that the configuration file can be found. 59 </p> 60 </li> 61 <li> 62 <p> 63 Configure your environment to back your regions from 64 system memory instead of the filesystem. 65 </p> 66 </li> 67 <li> 68 <p> 69 Configure your logging subsystem such that log files are kept 70 entirely in-memory. 71 </p> 72 </li> 73 <li> 74 <p> 75 Increase the size of your in-memory log buffer so that it 76 is large enough to hold the largest set of concurrent write operations. 77 </p> 78 </li> 79 <li> 80 <p> 81 Increase the size of your in-memory cache so that it can 82 hold your entire data set. You do not want your cache to 83 page to disk. 84 </p> 85 </li> 86 <li> 87 <p> 88 Do not specify a file name when you open your database(s). 89 </p> 90 </li> 91 </ul> 92 </div> 93 <p> 94 As an example, this section takes the transaction example provided 95 in <a class="xref" href="txnexample_c.html" title="Transaction Example">Transaction Example</a> 96 and it updates that example so that the environment, database, log 97 files, and regions are all kept entirely in-memory. 98 </p> 99 <p> 100 For illustration purposes, we also modify this example so that 101 uncommitted reads are no longer used to enable the 102 <code class="function">count_records()</code> 103 104 function. Instead, we simply provide a transaction handle to 105 <code class="function">count_records()</code> 106 107 so as to avoid the self-deadlock. Be aware that using a transaction handle here rather than 108 uncommitted reads will work just as well as if we had continued to use uncommitted reads. However, 109 the usage of the transaction handle here will 110 probably cause more deadlocks than using read-uncommitted does, because more locking is being performed in 111 this case. 112 </p> 113 <p> 114 To begin, we simplify the beginning of our example a bit. Because 115 we no longer need an environment home directory, we can remove all 116 the code that we used to determine path delimiters 117 and include the <code class="function">getopt</code> function. We can also 118 remove our <code class="function">usage()</code> function because we no 119 longer require any command line arguments. 120 </p> 121 <pre class="programlisting">/* File: txn_guide_inmemory.c */ 122 123/* We assume an ANSI-compatible compiler */ 124#include <stdio.h> 125#include <stdlib.h> 126#include <string.h> 127#include <pthread.h> 128#include <db.h> 129 130/* Run 5 writers threads at a time. */ 131#define NUMWRITERS 5 132 133/* 134 * Printing of pthread_t is implementation-specific, so we 135 * create our own thread IDs for reporting purposes. 136 */ 137int global_thread_num; 138pthread_mutex_t thread_num_lock; 139 140/* Forward declarations */ 141int count_records(DB *, DB_TXN *); 142int open_db(DB **, const char *, const char *, DB_ENV *, u_int32_t); 143int writer_thread(void *); </pre> 144 <p> 145 Next, in our <code class="function">main()</code>, we also eliminate some 146 variables that this example no longer needs. In particular, we are able to remove 147 the 148 <code class="literal">db_home_dir</code> 149 150 and 151 <code class="literal">file_name</code> 152 153 variables. We also remove all our <code class="function">getopt</code> code. 154</p> 155 <pre class="programlisting">int 156main(void) 157{ 158 /* Initialize our handles */ 159 DB *dbp = NULL; 160 DB_ENV *envp = NULL; 161 162 pthread_t writer_threads[NUMWRITERS]; 163 int i, ret, ret_t; 164 u_int32_t env_flags; 165 166 /* Application name */ 167 const char *prog_name = "txn_guide_inmemory"; </pre> 168 <p> 169 Next we create our environment as always. However, we add 170 <code class="literal">DB_PRIVATE</code> to our environment open flags. This 171 flag causes our environment to back regions using our 172 application's heap memory rather than by using the filesystem. 173 This is the first important step to keeping our DB data 174 entirely in-memory. 175 </p> 176 <p> 177 We also remove the <code class="literal">DB_RECOVER</code> flag from the environment open flags. Because our databases, 178 logs, and regions are maintained in-memory, there will never be anything to recover. 179 </p> 180 <p> 181 Note that we show the additional code here in 182 <strong class="userinput"><code>bold.</code></strong> 183 </p> 184 <pre class="programlisting"> /* Create the environment */ 185 ret = db_env_create(&envp, 0); 186 if (ret != 0) { 187 fprintf(stderr, "Error creating environment handle: %s\n", 188 db_strerror(ret)); 189 goto err; 190 } 191 192 env_flags = 193 DB_CREATE | /* Create the environment if it does not exist */ 194 DB_INIT_LOCK | /* Initialize the locking subsystem */ 195 DB_INIT_LOG | /* Initialize the logging subsystem */ 196 DB_INIT_TXN | /* Initialize the transactional subsystem. This 197 * also turns on logging. */ 198 DB_INIT_MPOOL | /* Initialize the memory pool (in-memory cache) */ 199 <strong class="userinput"><code>DB_PRIVATE | /* Region files are not backed by the filesystem. 200 * Instead, they are backed by heap memory. */</code></strong> 201 DB_THREAD; /* Cause the environment to be free-threaded */ </pre> 202 <p> 203 Now we configure our environment to keep the log files in memory, 204 increase the log buffer size to 10 MB, and increase our in-memory 205 cache to 10 MB. These values should be more than enough for our 206 application's workload. 207 </p> 208 <pre class="programlisting"> 209 <strong class="userinput"> 210 <code> /* Specify in-memory logging */ 211 ret = envp->log_set_config(envp, DB_LOG_IN_MEMORY, 1); 212 if (ret != 0) { 213 fprintf(stderr, "Error setting log subsystem to in-memory: %s\n", 214 db_strerror(ret)); 215 goto err; 216 } 217 218 /* 219 * Specify the size of the in-memory log buffer. 220 */ 221 ret = envp->set_lg_bsize(envp, 10 * 1024 * 1024); 222 if (ret != 0) { 223 fprintf(stderr, "Error increasing the log buffer size: %s\n", 224 db_strerror(ret)); 225 goto err; 226 } 227 228 /* 229 * Specify the size of the in-memory cache. 230 */ 231 ret = envp->set_cachesize(envp, 0, 232 10 * 1024 * 1024, 1); 233 if (ret != 0) { 234 fprintf(stderr, "Error increasing the cache size: %s\n", 235 db_strerror(ret)); 236 goto err; 237 }</code> 238 </strong> 239 </pre> 240 <p> 241 Next, we open the environment and setup our lock detection. This is 242 identical to how the example previously worked, except that we do not 243 provide a location for the environment's home directory. 244 </p> 245 <pre class="programlisting"> /* 246 * Indicate that we want db to perform lock detection internally. 247 * Also indicate that the transaction with the fewest number of 248 * write locks will receive the deadlock notification in 249 * the event of a deadlock. 250 */ 251 ret = envp->set_lk_detect(envp, DB_LOCK_MINWRITE); 252 if (ret != 0) { 253 fprintf(stderr, "Error setting lock detect: %s\n", 254 db_strerror(ret)); 255 goto err; 256 } 257 258 /* Now actually open the environment */ 259 ret = envp->open(envp, <strong class="userinput"><code>NULL</code></strong>, env_flags, 0); 260 if (ret != 0) { 261 fprintf(stderr, "Error opening environment: %s\n", 262 db_strerror(ret)); 263 goto err; 264 } </pre> 265 <p> 266 When we call 267 <span><code class="function">open_db()</code>,</span> 268 269 which is what we use 270 to open our database, we no not provide a database filename for the 271 third parameter. When the filename is <code class="literal">NULL</code>, the database is not 272 backed by the filesystem. 273 </p> 274 <pre class="programlisting"> /* 275 * If we had utility threads (for running checkpoints or 276 * deadlock detection, for example) we would spawn those 277 * here. However, for a simple example such as this, 278 * that is not required. 279 */ 280 281 /* Open the database */ 282 ret = open_db(&dbp, prog_name, <strong class="userinput"><code>NULL</code></strong>, 283 envp, DB_DUPSORT); 284 if (ret != 0) 285 goto err; </pre> 286 <p> 287 After that, our <code class="function">main()</code> function is unchanged, 288 except that when we 289 <span>close the database,</span> 290 291 we change the error message string so as to not reference the database filename. 292 </p> 293 <pre class="programlisting"> /* Initialize a pthread mutex. Used to help provide thread ids. */ 294 (void)pthread_mutex_init(&thread_num_lock, NULL); 295 296 /* Start the writer threads. */ 297 for (i = 0; i < NUMWRITERS; i++) 298 (void)pthread_create( 299 &writer_threads[i], NULL, (void *)writer_thread, (void *)dbp); 300 301 /* Join the writers */ 302 for (i = 0; i < NUMWRITERS; i++) 303 (void)pthread_join(writer_threads[i], NULL); 304 305err: 306 /* Close our database handle, if it was opened. */ 307 if (dbp != NULL) { 308 ret_t = dbp->close(dbp, 0); 309 if (ret_t != 0) { 310 <strong class="userinput"><code>fprintf(stderr, "%s database close failed.\n", 311 db_strerror(ret_t));</code></strong> 312 ret = ret_t; 313 } 314 } 315 316 /* Close our environment, if it was opened. */ 317 if (envp != NULL) { 318 ret_t = envp->close(envp, 0); 319 if (ret_t != 0) { 320 fprintf(stderr, "environment close failed: %s\n", 321 db_strerror(ret_t)); 322 ret = ret_t; 323 } 324 } 325 326 /* Final status message and return. */ 327 printf("I'm all done.\n"); 328 return (ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE); 329} </pre> 330 <p> 331 That completes <code class="function">main()</code>. The bulk of our 332 <code class="function">writer_thread()</code> function implementation is 333 unchanged from the initial transaction example, except that we no 334 longer check for <code class="literal">DB_KEYEXISTS</code> in our 335 <code class="methodname">DB->put()</code> return code. Because we are 336 configuring for a completely in-memory database, there is no 337 possibility that we can run this code against an existing database. 338 Therefore, there is no way that <code class="literal">DB_KEYEXISTS</code> 339 will be returned by <code class="methodname">DB->put()</code>. 340 </p> 341 <pre class="programlisting">/* 342 * A function that performs a series of writes to a 343 * Berkeley DB database. The information written 344 * to the database is largely nonsensical, but the 345 * mechanism of transactional commit/abort and 346 * deadlock detection is illustrated here. 347 */ 348int 349writer_thread(void *args) 350{ 351 DBT key, value; 352 DB_TXN *txn; 353 int i, j, payload, ret, thread_num; 354 int retry_count, max_retries = 20; /* Max retry on a deadlock */ 355 char *key_strings[] = {"key 1", "key 2", "key 3", "key 4", 356 "key 5", "key 6", "key 7", "key 8", 357 "key 9", "key 10"}; 358 359 DB *dbp = (DB *)args; 360 DB_ENV *envp = dbp->get_env(dbp); 361 362 /* Get the thread number */ 363 (void)pthread_mutex_lock(&thread_num_lock); 364 global_thread_num++; 365 thread_num = global_thread_num; 366 (void)pthread_mutex_unlock(&thread_num_lock); 367 368 /* Initialize the random number generator */ 369 srand((u_int)pthread_self()); 370 371 /* Write 50 times and then quit */ 372 for (i = 0; i < 50; i++) { 373 retry_count = 0; /* Used for deadlock retries */ 374 375retry: 376 ret = envp->txn_begin(envp, NULL, &txn, 0); 377 if (ret != 0) { 378 envp->err(envp, ret, "txn_begin failed"); 379 return (EXIT_FAILURE); 380 } 381 for (j = 0; j < 10; j++) { 382 /* Set up our key and values DBTs */ 383 memset(&key, 0, sizeof(DBT)); 384 key.data = key_strings[j]; 385 key.size = (strlen(key_strings[j]) + 1) * sizeof(char); 386 387 memset(&value, 0, sizeof(DBT)); 388 payload = rand() + i; 389 value.data = &payload; 390 value.size = sizeof(int); 391 392 /* Perform the database put. */ 393 switch (ret = dbp->put(dbp, txn, &key, &value, 0)) { 394 case 0: 395 break; 396 397 /* 398 * Here's where we perform deadlock detection. If 399 * DB_LOCK_DEADLOCK is returned by the put operation, 400 * then this thread has been chosen to break a deadlock. 401 * It must abort its operation, and optionally retry the 402 * put. 403 */ 404 case DB_LOCK_DEADLOCK: 405 /* 406 * First that we MUST do is abort the 407 * transaction. 408 */ 409 (void)txn->abort(txn); 410 /* 411 * Now we decide if we want to retry the operation. 412 * If we have retried less than max_retries, 413 * increment the retry count and goto retry. 414 */ 415 if (retry_count < max_retries) { 416 printf("Writer %i: Got DB_LOCK_DEADLOCK.\n", 417 thread_num); 418 printf("Writer %i: Retrying write operation.\n", 419 thread_num); 420 retry_count++; 421 goto retry; 422 } 423 /* 424 * Otherwise, just give up. 425 */ 426 printf("Writer %i: ", thread_num); 427 printf("Got DB_LOCK_DEADLOCK and out of retries.\n"); 428 printf("Writer %i: Giving up.\n", thread_num); 429 return (EXIT_FAILURE); 430 /* 431 * If a generic error occurs, we simply abort the 432 * transaction and exit the thread completely. 433 */ 434 default: 435 envp->err(envp, ret, "db put failed"); 436 ret = txn->abort(txn); 437 if (ret != 0) 438 envp->err(envp, ret, "txn abort failed"); 439 return (EXIT_FAILURE); 440 } /** End case statement **/ 441 442 } /** End for loop **/ </pre> 443 <p> 444 The only other change to <code class="function">writer_thread()</code> 445 is that we pass <code class="function">count_records()</code> a 446 transaction handle, rather than configuring our entire 447 application for uncommitted reads. Both mechanisms work well-enough 448 for preventing a self-deadlock. However, the individual count 449 in this example will tend to be lower than the counts seen in 450 the previous transaction example, because 451 <code class="function">count_records()</code> can no longer see records 452 created but not yet committed by other threads. 453 </p> 454 <pre class="programlisting"> /* 455 * print the number of records found in the database. 456 * See count_records() for usage information. 457 */ 458 printf("Thread %i. Record count: %i\n", thread_num, 459 count_records(dbp, <strong class="userinput"><code>txn</code></strong>)); 460 461 /* 462 * If all goes well, we can commit the transaction and 463 * loop to the next transaction. 464 */ 465 ret = txn->commit(txn, 0); 466 if (ret != 0) { 467 envp->err(envp, ret, "txn commit failed"); 468 return (EXIT_FAILURE); 469 } 470 } 471 return (EXIT_SUCCESS); 472} </pre> 473 <p> 474 Next we update 475 <span><code class="function">count_records()</code>.</span> 476 477 The only difference 478 here is that we no longer specify <code class="literal">DB_READ_UNCOMMITTED</code> when 479 we open our cursor. Note that even this minor change is not required. 480 If we do not configure our database to support uncommitted reads, 481 <code class="literal">DB_READ_UNCOMMITTED</code> on the cursor open will be silently 482 ignored. However, we remove the flag anyway from the cursor open so as to 483 avoid confusion. 484</p> 485 <pre class="programlisting">int 486count_records(DB *dbp, DB_TXN *txn) 487{ 488 DBT key, value; 489 DBC *cursorp; 490 int count, ret; 491 492 cursorp = NULL; 493 count = 0; 494 495 /* Get the cursor */ 496 ret = dbp->cursor(dbp, txn, &cursorp, <strong class="userinput"><code>0</code></strong>); 497 if (ret != 0) { 498 dbp->err(dbp, ret, "count_records: cursor open failed."); 499 goto cursor_err; 500 } 501 502 /* Get the key DBT used for the database read */ 503 memset(&key, 0, sizeof(DBT)); 504 memset(&value, 0, sizeof(DBT)); 505 do { 506 ret = cursorp->get(cursorp, &key, &value, DB_NEXT); 507 switch (ret) { 508 case 0: 509 count++; 510 break; 511 case DB_NOTFOUND: 512 break; 513 default: 514 dbp->err(envp, ret, 515 "Count records unspecified error"); 516 goto cursor_err; 517 } 518 } while (ret == 0); 519 520cursor_err: 521 if (cursorp != NULL) { 522 ret = cursorp->close(cursorp); 523 if (ret != 0) { 524 dbp->err(dbp, ret, 525 "count_records: cursor close failed."); 526 } 527 } 528 529 return (count); 530}</pre> 531 <p> 532 Finally, we update 533 <span><code class="function">open_db()</code>.</span> 534 535 This involves 536 removing <code class="literal">DB_READ_UNCOMMITTED</code> from the 537 open flags. 538 539 <span>We are also careful to change our database open error 540 message to no longer use the <code class="literal">file_name</code> 541 variable because that value will always be <code class="literal">NULL</code> for this example.</span> 542 </p> 543 <pre class="programlisting">/* Open a Berkeley DB database */ 544int 545open_db(DB **dbpp, const char *progname, const char *file_name, 546 DB_ENV *envp, u_int32_t extra_flags) 547{ 548 int ret; 549 u_int32_t open_flags; 550 DB *dbp; 551 552 /* Initialize the DB handle */ 553 ret = db_create(&dbp, envp, 0); 554 if (ret != 0) { 555 fprintf(stderr, "%s: %s\n", progname, 556 db_strerror(ret)); 557 return (EXIT_FAILURE); 558 } 559 560 /* Point to the memory malloc'd by db_create() */ 561 *dbpp = dbp; 562 563 if (extra_flags != 0) { 564 ret = dbp->set_flags(dbp, extra_flags); 565 if (ret != 0) { 566 dbp->err(dbp, ret, 567 "open_db: Attempt to set extra flags failed."); 568 return (EXIT_FAILURE); 569 } 570 } 571 572 /* Now open the database */ 573 <strong class="userinput"><code>open_flags = DB_CREATE | /* Allow database creation */ 574 DB_THREAD | 575 DB_AUTO_COMMIT; /* Allow auto commit */</code></strong> 576 577 ret = dbp->open(dbp, /* Pointer to the database */ 578 NULL, /* Txn pointer */ 579 file_name, /* File name */ 580 NULL, /* Logical db name */ 581 DB_BTREE, /* Database type (using btree) */ 582 open_flags, /* Open flags */ 583 0); /* File mode. Using defaults */ 584 if (ret != 0) { 585 <strong class="userinput"><code>dbp->err(dbp, ret, "Database open failed"); 586 return (EXIT_FAILURE);</code></strong> 587 } 588 return (EXIT_SUCCESS); 589} </pre> 590 <p> 591 This completes our in-memory transactional example. If you would like to 592 experiment with this code, you can find the example in the following 593 location in your DB distribution: 594</p> 595 <pre class="programlisting"><span class="emphasis"><em>DB_INSTALL</em></span>/examples_c/txn_guide</pre> 596 </div> 597 <div class="navfooter"> 598 <hr /> 599 <table width="100%" summary="Navigation footer"> 600 <tr> 601 <td width="40%" align="left"><a accesskey="p" href="txnexample_c.html">Prev</a>��</td> 602 <td width="20%" align="center"> 603 <a accesskey="u" href="wrapup.html">Up</a> 604 </td> 605 <td width="40%" align="right">��</td> 606 </tr> 607 <tr> 608 <td width="40%" align="left" valign="top">Transaction Example��</td> 609 <td width="20%" align="center"> 610 <a accesskey="h" href="index.html">Home</a> 611 </td> 612 <td width="40%" align="right" valign="top">��</td> 613 </tr> 614 </table> 615 </div> 616 </body> 617</html> 618