1<?xml version="1.0" encoding="UTF-8" standalone="no"?> 2<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 3<html xmlns="http://www.w3.org/1999/xhtml"> 4 <head> 5 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> 6 <title>In-Memory Transaction Example</title> 7 <link rel="stylesheet" href="gettingStarted.css" type="text/css" /> 8 <meta name="generator" content="DocBook XSL Stylesheets V1.62.4" /> 9 <link rel="home" href="index.html" title="Getting Started with Berkeley DB Transaction Processing" /> 10 <link rel="up" href="wrapup.html" title="Chapter 6. Summary and Examples" /> 11 <link rel="previous" href="txnexample_c.html" title="Transaction Example" /> 12 </head> 13 <body> 14 <div class="navheader"> 15 <table width="100%" summary="Navigation header"> 16 <tr> 17 <th colspan="3" align="center">In-Memory Transaction Example</th> 18 </tr> 19 <tr> 20 <td width="20%" align="left"><a accesskey="p" href="txnexample_c.html">Prev</a> </td> 21 <th width="60%" align="center">Chapter 6. Summary and Examples</th> 22 <td width="20%" align="right"> </td> 23 </tr> 24 </table> 25 <hr /> 26 </div> 27 <div class="sect1" lang="en" xml:lang="en"> 28 <div class="titlepage"> 29 <div> 30 <div> 31 <h2 class="title" style="clear: both"><a id="inmem_txnexample_c"></a>In-Memory Transaction Example</h2> 32 </div> 33 </div> 34 <div></div> 35 </div> 36 <p> 37 DB is sometimes used for applications that simply need to cache 38 data retrieved from some other location (such as a remote database 39 server). DB is also often used in embedded systems. 40 </p> 41 <p> 42 In both cases, applications may want to use transactions for 43 atomicity, consistency, and isolation guarantees, but they may also want 44 to forgo the durability guarantee entirely. In doing so, they can keep 45 their DB environment and databases entirely in-memory so 46 as to avoid the performance impact of unneeded disk I/O. 47 </p> 48 <p> 49 To do this: 50 </p> 51 <div class="itemizedlist"> 52 <ul type="disc"> 53 <li> 54 <p> 55 Refrain from specifying a home directory when you open your 56 environment. The exception to this is if you are using the 57 <tt class="literal">DB_CONFIG</tt> configuration file — in 58 that case you must identify the environment's home 59 directory so that the configuration file can be found. 60 </p> 61 </li> 62 <li> 63 <p> 64 Configure your environment to back your regions from 65 system memory instead of the filesystem. 66 </p> 67 </li> 68 <li> 69 <p> 70 Configure your logging subsystem such that log files are kept 71 entirely in-memory. 72 </p> 73 </li> 74 <li> 75 <p> 76 Increase the size of your in-memory log buffer so that it 77 is large enough to hold the largest set of concurrent write operations. 78 </p> 79 </li> 80 <li> 81 <p> 82 Increase the size of your in-memory cache so that it can 83 hold your entire data set. You do not want your cache to 84 page to disk. 85 </p> 86 </li> 87 <li> 88 <p> 89 Do not specify a file name when you open your database(s). 90 </p> 91 </li> 92 </ul> 93 </div> 94 <p> 95 As an example, this section takes the transaction example provided 96 in <a href="txnexample_c.html">Transaction Example</a> 97 and it updates that example so that the environment, database, log 98 files, and regions are all kept entirely in-memory. 99 </p> 100 <p> 101 For illustration purposes, we also modify this example so that 102 uncommitted reads are no longer used to enable the 103 104 <tt class="function">countRecords()</tt> 105 function. Instead, we simply provide a transaction handle to 106 107 <tt class="function">countRecords()</tt> 108 so as to avoid the self-deadlock. Be aware that using a transaction handle here rather than 109 uncommitted reads will work just as well as if we had continued to use uncommitted reads. However, 110 the usage of the transaction handle here will 111 probably cause more deadlocks than using read-uncommitted does, because more locking is being performed in 112 this case. 113 </p> 114 <p> 115 To begin, we simplify the beginning of our example a bit. Because 116 we no longer need an environment home directory, we can remove all 117 the code that we used to determine path delimiters 118 and include the <tt class="function">getopt</tt> function. We can also 119 remove our <tt class="function">usage()</tt> function because we no 120 longer require any command line arguments. 121 </p> 122 <pre class="programlisting">// File TxnGuideInMemory.cpp 123 124// We assume an ANSI-compatible compiler 125#include <db_cxx.h> 126#include <pthread.h> 127#include <iostream> 128 129// Run 5 writers threads at a time. 130#define NUMWRITERS 5 131 132// Printing of pthread_t is implementation-specific, so we 133// create our own thread IDs for reporting purposes. 134int global_thread_num; 135pthread_mutex_t thread_num_lock; 136 137// Forward declarations 138int countRecords(Db *, DbTxn *); 139int openDb(Db **, const char *, const char *, DbEnv *, u_int32_t); 140int usage(void); 141void *writerThread(void *); </pre> 142 <p> 143 Next, in our <tt class="function">main()</tt>, we also eliminate some 144 variables that this example no longer needs. In particular, we are able to remove 145 the 146 147 <tt class="literal">dbHomeDir</tt> 148 and 149 150 <tt class="literal">fileName</tt> 151 variables. We also remove all our <tt class="function">getopt</tt> code. 152</p> 153 <pre class="programlisting">int 154main(void) 155{ 156 // Initialize our handles 157 Db *dbp = NULL; 158 DbEnv *envp = NULL; 159 160 pthread_t writerThreads[NUMWRITERS]; 161 int i; 162 u_int32_t envFlags; 163 164 // Application name 165 const char *progName = "TxnGuideInMemory"; </pre> 166 <p> 167 Next we create our environment as always. However, we add 168 <tt class="literal">DB_PRIVATE</tt> to our environment open flags. This 169 flag causes our environment to back regions using our 170 application's heap memory rather than by using the filesystem. 171 This is the first important step to keeping our DB data 172 entirely in-memory. 173 </p> 174 <p> 175 We also remove the <tt class="literal">DB_RECOVER</tt> flag from the environment open flags. Because our databases, 176 logs, and regions are maintained in-memory, there will never be anything to recover. 177 </p> 178 <p> 179 Note that we show the additional code here in 180 <b class="userinput"><tt>bold.</tt></b> 181 </p> 182 <pre class="programlisting"> // Env open flags 183 envFlags = 184 DB_CREATE | // Create the environment if it does not exist 185 DB_INIT_LOCK | // Initialize the locking subsystem 186 DB_INIT_LOG | // Initialize the logging subsystem 187 DB_INIT_TXN | // Initialize the transactional subsystem. This 188 // also turns on logging. 189 DB_INIT_MPOOL | // Initialize the memory pool (in-memory cache) 190 <b class="userinput"><tt>DB_PRIVATE | // Region files are not backed by the filesystem. 191 // Instead, they are backed by heap memory.</tt></b> 192 DB_THREAD; // Cause the environment to be free-threaded 193 194 try { 195 // Create the environment 196 envp = new DbEnv(0); </pre> 197 <p> 198 Now we configure our environment to keep the log files in memory, 199 increase the log buffer size to 10 MB, and increase our in-memory 200 cache to 10 MB. These values should be more than enough for our 201 application's workload. 202 </p> 203 <pre class="programlisting"> 204 <b class="userinput"> 205 <tt> // Specify in-memory logging 206 envp->log_set_config(DB_LOG_IN_MEMORY, 1); 207 208 // Specify the size of the in-memory log buffer. 209 envp->set_lg_bsize(10 * 1024 * 1024); 210 211 // Specify the size of the in-memory cache 212 envp->set_cachesize(0, 10 * 1024 * 1024, 1); </tt> 213 </b> 214 </pre> 215 <p> 216 Next, we open the environment and setup our lock detection. This is 217 identical to how the example previously worked, except that we do not 218 provide a location for the environment's home directory. 219 </p> 220 <pre class="programlisting"> // Indicate that we want db to internally perform deadlock 221 // detection. Also indicate that the transaction with 222 // the fewest number of write locks will receive the 223 // deadlock notification in the event of a deadlock. 224 envp->set_lk_detect(DB_LOCK_MINWRITE); 225 226 // Open the environment 227 envp->open(<b class="userinput"><tt>NULL</tt></b>, envFlags, 0); </pre> 228 <p> 229 When we call 230 231 <span><tt class="function">openDb()</tt>,</span> 232 which is what we use 233 to open our database, we no not provide a database filename for the 234 third parameter. When the filename is <tt class="literal">NULL</tt>, the database is not 235 backed by the filesystem. 236 </p> 237 <pre class="programlisting"> // If we had utility threads (for running checkpoints or 238 // deadlock detection, for example) we would spawn those 239 // here. However, for a simple example such as this, 240 // that is not required. 241 242 // Open the database 243 openDb(&dbp, progName, <b class="userinput"><tt>NULL</tt></b>, 244 envp, DB_DUPSORT); 245 </pre> 246 <p> 247 After that, our <tt class="function">main()</tt> function is unchanged, 248 except that when we 249 250 <span>check for exceptions on the database open,</span> 251 we change the error message string so as to not reference the database filename. 252 </p> 253 <pre class="programlisting"> // Initialize a pthread mutex. Used to help provide thread ids. 254 (void)pthread_mutex_init(&thread_num_lock, NULL); 255 256 // Start the writer threads. 257 for (i = 0; i < NUMWRITERS; i++) 258 (void)pthread_create( 259 &writerThreads[i], NULL, 260 writerThread, 261 (void *)dbp); 262 263 // Join the writers 264 for (i = 0; i < NUMWRITERS; i++) 265 (void)pthread_join(writerThreads[i], NULL); 266 267 } catch(DbException &e) { 268 <b class="userinput"><tt>std::cerr << "Error opening database environment: "</tt></b> 269 << std::endl; 270 std::cerr << e.what() << std::endl; 271 return (EXIT_FAILURE); 272 } 273 274 try { 275 // Close our database handle if it was opened. 276 if (dbp != NULL) 277 dbp->close(0); 278 279 // Close our environment if it was opened. 280 if (envp != NULL) 281 envp->close(0); 282 } catch(DbException &e) { 283 std::cerr << "Error closing database and environment." 284 << std::endl; 285 std::cerr << e.what() << std::endl; 286 return (EXIT_FAILURE); 287 } 288 289 // Final status message and return. 290 291 std::cout << "I'm all done." << std::endl; 292 return (EXIT_SUCCESS); 293} </pre> 294 <p> 295 That completes <tt class="function">main()</tt>. The bulk of our 296 <tt class="function">writerThread()</tt> function implementation is 297 unchanged from the initial transaction example, except that we now pass 298 <tt class="function">countRecords</tt> a transaction handle, rather than configuring our 299 application to perform uncommitted reads. Both mechanisms work well-enough 300 for preventing a self-deadlock. However, the individual count 301 in this example will tend to be lower than the counts seen in 302 the previous transaction example, because 303 <tt class="function">countRecords()</tt> can no longer see records 304 created but not yet committed by other threads. 305 </p> 306 <pre class="programlisting">// A function that performs a series of writes to a 307// Berkeley DB database. The information written 308// to the database is largely nonsensical, but the 309// mechanism of transactional commit/abort and 310// deadlock detection is illustrated here. 311void * 312writerThread(void *args) 313{ 314 Db *dbp = (Db *)args; 315 DbEnv *envp = dbp->get_env(dbp); 316 317 int j, thread_num; 318 int max_retries = 20; // Max retry on a deadlock 319 char *key_strings[] = {"key 1", "key 2", "key 3", "key 4", 320 "key 5", "key 6", "key 7", "key 8", 321 "key 9", "key 10"}; 322 323 // Get the thread number 324 (void)pthread_mutex_lock(&thread_num_lock); 325 global_thread_num++; 326 thread_num = global_thread_num; 327 (void)pthread_mutex_unlock(&thread_num_lock); 328 329 // Initialize the random number generator 330 srand((u_int)pthread_self()); 331 332 // Perform 50 transactions 333 for (int i=0; i<50; i++) { 334 DbTxn *txn; 335 bool retry = true; 336 int retry_count = 0; 337 // while loop is used for deadlock retries 338 while (retry) { 339 // try block used for deadlock detection and 340 // general db exception handling 341 try { 342 343 // Begin our transaction. We group multiple writes in 344 // this thread under a single transaction so as to 345 // (1) show that you can atomically perform multiple 346 // writes at a time, and (2) to increase the chances 347 // of a deadlock occurring so that we can observe our 348 // deadlock detection at work. 349 350 // Normally we would want to avoid the potential for 351 // deadlocks, so for this workload the correct thing 352 // would be to perform our puts with auto commit. But 353 // that would excessively simplify our example, so we 354 // do the "wrong" thing here instead. 355 txn = NULL; 356 envp->txn_begin(NULL, &txn, 0); 357 // Perform the database write for this transaction. 358 for (j = 0; j < 10; j++) { 359 Dbt key, value; 360 key.set_data(key_strings[j]); 361 key.set_size((strlen(key_strings[j]) + 1) * 362 sizeof(char)); 363 364 int payload = rand() + i; 365 value.set_data(&payload); 366 value.set_size(sizeof(int)); 367 368 // Perform the database put 369 dbp->put(txn, &key, &value, 0); 370 } 371 372 // countRecords runs a cursor over the entire database. 373 // We do this to illustrate issues of deadlocking 374 std::cout << thread_num << " : Found " 375 << countRecords(dbp, <b class="userinput"><tt>txn</tt></b>) 376 << " records in the database." << std::endl; 377 378 std::cout << thread_num << " : committing txn : " << i 379 << std::endl; 380 381 // commit 382 try { 383 txn->commit(0); 384 retry = false; 385 txn = NULL; 386 } catch (DbException &e) { 387 std::cout << "Error on txn commit: " 388 << e.what() << std::endl; 389 } 390 } catch (DbDeadlockException &de) { 391 // First thing we MUST do is abort the transaction. 392 if (txn != NULL) 393 (void)txn->abort(); 394 395 // Now we decide if we want to retry the operation. 396 // If we have retried less than max_retries, 397 // increment the retry count and goto retry. 398 if (retry_count < max_retries) { 399 std::cout << "############### Writer " << thread_num 400 << ": Got DB_LOCK_DEADLOCK.\n" 401 << "Retrying write operation." 402 << std::endl; 403 retry_count++; 404 retry = true; 405 } else { 406 // Otherwise, just give up. 407 std::cerr << "Writer " << thread_num 408 << ": Got DeadLockException and out of " 409 << "retries. Giving up." << std::endl; 410 retry = false; 411 } 412 } catch (DbException &e) { 413 std::cerr << "db put failed" << std::endl; 414 std::cerr << e.what() << std::endl; 415 if (txn != NULL) 416 txn->abort(); 417 retry = false; 418 } catch (std::exception &ee) { 419 std::cerr << "Unknown exception: " << ee.what() << std::endl; 420 return (0); 421 } 422 } 423 } 424 return (0); 425} </pre> 426 <p> 427 Next we update 428 429 <span><tt class="function">countRecords()</tt>.</span> 430 The only difference 431 here is that we no longer specify <tt class="literal">DB_READ_UNCOMMITTED</tt> when 432 we open our cursor. Note that even this minor change is not required. 433 If we do not configure our database to support uncommitted reads, 434 <tt class="literal">DB_READ_UNCOMMITTED</tt> on the cursor open will be silently 435 ignored. However, we remove the flag anyway from the cursor open so as to 436 avoid confusion. 437</p> 438 <pre class="programlisting">int 439countRecords(Db *dbp, DbTxn *txn) 440{ 441 442 Dbc *cursorp = NULL; 443 int count = 0; 444 445 try { 446 // Get the cursor 447 dbp->cursor(txn, &cursorp, <b class="userinput"><tt>0</tt></b>); 448 449 Dbt key, value; 450 while (cursorp->get(&key, &value, DB_NEXT) == 0) { 451 count++; 452 } 453 } catch (DbDeadlockException &de) { 454 std::cerr << "countRecords: got deadlock" << std::endl; 455 cursorp->close(); 456 throw de; 457 } catch (DbException &e) { 458 std::cerr << "countRecords error:" << std::endl; 459 std::cerr << e.what() << std::endl; 460 } 461 462 if (cursorp != NULL) { 463 try { 464 cursorp->close(); 465 } catch (DbException &e) { 466 std::cerr << "countRecords: cursor close failed:" << std::endl; 467 std::cerr << e.what() << std::endl; 468 } 469 } 470 471 return (count); 472} </pre> 473 <p> 474 Finally, we update 475 476 <span><tt class="function">openDb()</tt>.</span> 477 This involves 478 removing <tt class="literal">DB_READ_UNCOMMITTED</tt> from the 479 open flags. 480 481 482 </p> 483 <pre class="programlisting">// Open a Berkeley DB database 484int 485openDb(Db **dbpp, const char *progname, const char *fileName, 486 DbEnv *envp, u_int32_t extraFlags) 487{ 488 int ret; 489 u_int32_t openFlags; 490 491 try { 492 Db *dbp = new Db(envp, 0); 493 494 // Point to the new'd Db 495 *dbpp = dbp; 496 497 if (extraFlags != 0) 498 ret = dbp->set_flags(extraFlags); 499 500 // Now open the database 501 <b class="userinput"><tt>openFlags = DB_CREATE | // Allow database creation 502 DB_THREAD | 503 DB_AUTO_COMMIT; // Allow auto commit</tt></b> 504 505 dbp->open(NULL, // Txn pointer 506 fileName, // File name 507 NULL, // Logical db name 508 DB_BTREE, // Database type (using btree) 509 openFlags, // Open flags 510 0); // File mode. Using defaults 511 } catch (DbException &e) { 512 std::cerr << progname << ": openDb: db open failed:" << std::endl; 513 std::cerr << e.what() << std::endl; 514 return (EXIT_FAILURE); 515 } 516 517 return (EXIT_SUCCESS); 518} </pre> 519 <p> 520 This completes our in-memory transactional example. If you would like to 521 experiment with this code, you can find the example in the following 522 location in your DB distribution: 523</p> 524 <pre class="programlisting"><span class="emphasis"><em>DB_INSTALL</em></span>/examples_cxx/txn_guide</pre> 525 </div> 526 <div class="navfooter"> 527 <hr /> 528 <table width="100%" summary="Navigation footer"> 529 <tr> 530 <td width="40%" align="left"><a accesskey="p" href="txnexample_c.html">Prev</a> </td> 531 <td width="20%" align="center"> 532 <a accesskey="u" href="wrapup.html">Up</a> 533 </td> 534 <td width="40%" align="right"> </td> 535 </tr> 536 <tr> 537 <td width="40%" align="left" valign="top">Transaction Example </td> 538 <td width="20%" align="center"> 539 <a accesskey="h" href="index.html">Home</a> 540 </td> 541 <td width="40%" align="right" valign="top"> </td> 542 </tr> 543 </table> 544 </div> 545 </body> 546</html> 547