1<?xml version="1.0" encoding="UTF-8" standalone="no"?> 2<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 3<html xmlns="http://www.w3.org/1999/xhtml"> 4 <head> 5 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> 6 <title>In-Memory Transaction Example</title> 7 <link rel="stylesheet" href="gettingStarted.css" type="text/css" /> 8 <meta name="generator" content="DocBook XSL Stylesheets V1.73.2" /> 9 <link rel="start" href="index.html" title="Getting Started with Berkeley DB Transaction Processing" /> 10 <link rel="up" href="wrapup.html" title="Chapter 6. Summary and Examples" /> 11 <link rel="prev" href="txnexample_c.html" title="Transaction Example" /> 12 </head> 13 <body> 14 <div class="navheader"> 15 <table width="100%" summary="Navigation header"> 16 <tr> 17 <th colspan="3" align="center">In-Memory Transaction Example</th> 18 </tr> 19 <tr> 20 <td width="20%" align="left"><a accesskey="p" href="txnexample_c.html">Prev</a> </td> 21 <th width="60%" align="center">Chapter 6. Summary and Examples</th> 22 <td width="20%" align="right"> </td> 23 </tr> 24 </table> 25 <hr /> 26 </div> 27 <div class="sect1" lang="en" xml:lang="en"> 28 <div class="titlepage"> 29 <div> 30 <div> 31 <h2 class="title" style="clear: both"><a id="inmem_txnexample_c"></a>In-Memory Transaction Example</h2> 32 </div> 33 </div> 34 </div> 35 <p> 36 DB is sometimes used for applications that simply need to cache 37 data retrieved from some other location (such as a remote database 38 server). DB is also often used in embedded systems. 39 </p> 40 <p> 41 In both cases, applications may want to use transactions for 42 atomicity, consistency, and isolation guarantees, but they may also want 43 to forgo the durability guarantee entirely. In doing so, they can keep 44 their DB environment and databases entirely in-memory so 45 as to avoid the performance impact of unneeded disk I/O. 46 </p> 47 <p> 48 To do this: 49 </p> 50 <div class="itemizedlist"> 51 <ul type="disc"> 52 <li> 53 <p> 54 Refrain from specifying a home directory when you open your 55 environment. The exception to this is if you are using the 56 <code class="literal">DB_CONFIG</code> configuration file — in 57 that case you must identify the environment's home 58 directory so that the configuration file can be found. 59 </p> 60 </li> 61 <li> 62 <p> 63 Configure your environment to back your regions from 64 system memory instead of the filesystem. 65 </p> 66 </li> 67 <li> 68 <p> 69 Configure your logging subsystem such that log files are kept 70 entirely in-memory. 71 </p> 72 </li> 73 <li> 74 <p> 75 Increase the size of your in-memory log buffer so that it 76 is large enough to hold the largest set of concurrent write operations. 77 </p> 78 </li> 79 <li> 80 <p> 81 Increase the size of your in-memory cache so that it can 82 hold your entire data set. You do not want your cache to 83 page to disk. 84 </p> 85 </li> 86 <li> 87 <p> 88 Do not specify a file name when you open your database(s). 89 </p> 90 </li> 91 </ul> 92 </div> 93 <p> 94 As an example, this section takes the transaction example provided 95 in <a class="xref" href="txnexample_c.html" title="Transaction Example">Transaction Example</a> 96 and it updates that example so that the environment, database, log 97 files, and regions are all kept entirely in-memory. 98 </p> 99 <p> 100 For illustration purposes, we also modify this example so that 101 uncommitted reads are no longer used to enable the 102 103 <code class="function">countRecords()</code> 104 function. Instead, we simply provide a transaction handle to 105 106 <code class="function">countRecords()</code> 107 so as to avoid the self-deadlock. Be aware that using a transaction handle here rather than 108 uncommitted reads will work just as well as if we had continued to use uncommitted reads. However, 109 the usage of the transaction handle here will 110 probably cause more deadlocks than using read-uncommitted does, because more locking is being performed in 111 this case. 112 </p> 113 <p> 114 To begin, we simplify the beginning of our example a bit. Because 115 we no longer need an environment home directory, we can remove all 116 the code that we used to determine path delimiters 117 and include the <code class="function">getopt</code> function. We can also 118 remove our <code class="function">usage()</code> function because we no 119 longer require any command line arguments. 120 </p> 121 <pre class="programlisting">// File TxnGuideInMemory.cpp 122 123// We assume an ANSI-compatible compiler 124#include <db_cxx.h> 125#include <pthread.h> 126#include <iostream> 127 128// Run 5 writers threads at a time. 129#define NUMWRITERS 5 130 131// Printing of pthread_t is implementation-specific, so we 132// create our own thread IDs for reporting purposes. 133int global_thread_num; 134pthread_mutex_t thread_num_lock; 135 136// Forward declarations 137int countRecords(Db *, DbTxn *); 138int openDb(Db **, const char *, const char *, DbEnv *, u_int32_t); 139int usage(void); 140void *writerThread(void *); </pre> 141 <p> 142 Next, in our <code class="function">main()</code>, we also eliminate some 143 variables that this example no longer needs. In particular, we are able to remove 144 the 145 146 <code class="literal">dbHomeDir</code> 147 and 148 149 <code class="literal">fileName</code> 150 variables. We also remove all our <code class="function">getopt</code> code. 151</p> 152 <pre class="programlisting">int 153main(void) 154{ 155 // Initialize our handles 156 Db *dbp = NULL; 157 DbEnv *envp = NULL; 158 159 pthread_t writerThreads[NUMWRITERS]; 160 int i; 161 u_int32_t envFlags; 162 163 // Application name 164 const char *progName = "TxnGuideInMemory"; </pre> 165 <p> 166 Next we create our environment as always. However, we add 167 <code class="literal">DB_PRIVATE</code> to our environment open flags. This 168 flag causes our environment to back regions using our 169 application's heap memory rather than by using the filesystem. 170 This is the first important step to keeping our DB data 171 entirely in-memory. 172 </p> 173 <p> 174 We also remove the <code class="literal">DB_RECOVER</code> flag from the environment open flags. Because our databases, 175 logs, and regions are maintained in-memory, there will never be anything to recover. 176 </p> 177 <p> 178 Note that we show the additional code here in 179 <strong class="userinput"><code>bold.</code></strong> 180 </p> 181 <pre class="programlisting"> // Env open flags 182 envFlags = 183 DB_CREATE | // Create the environment if it does not exist 184 DB_INIT_LOCK | // Initialize the locking subsystem 185 DB_INIT_LOG | // Initialize the logging subsystem 186 DB_INIT_TXN | // Initialize the transactional subsystem. This 187 // also turns on logging. 188 DB_INIT_MPOOL | // Initialize the memory pool (in-memory cache) 189 <strong class="userinput"><code>DB_PRIVATE | // Region files are not backed by the filesystem. 190 // Instead, they are backed by heap memory.</code></strong> 191 DB_THREAD; // Cause the environment to be free-threaded 192 193 try { 194 // Create the environment 195 envp = new DbEnv(0); </pre> 196 <p> 197 Now we configure our environment to keep the log files in memory, 198 increase the log buffer size to 10 MB, and increase our in-memory 199 cache to 10 MB. These values should be more than enough for our 200 application's workload. 201 </p> 202 <pre class="programlisting"> 203 <strong class="userinput"> 204 <code> // Specify in-memory logging 205 envp->log_set_config(DB_LOG_IN_MEMORY, 1); 206 207 // Specify the size of the in-memory log buffer. 208 envp->set_lg_bsize(10 * 1024 * 1024); 209 210 // Specify the size of the in-memory cache 211 envp->set_cachesize(0, 10 * 1024 * 1024, 1); </code> 212 </strong> 213 </pre> 214 <p> 215 Next, we open the environment and setup our lock detection. This is 216 identical to how the example previously worked, except that we do not 217 provide a location for the environment's home directory. 218 </p> 219 <pre class="programlisting"> // Indicate that we want db to internally perform deadlock 220 // detection. Also indicate that the transaction with 221 // the fewest number of write locks will receive the 222 // deadlock notification in the event of a deadlock. 223 envp->set_lk_detect(DB_LOCK_MINWRITE); 224 225 // Open the environment 226 envp->open(<strong class="userinput"><code>NULL</code></strong>, envFlags, 0); </pre> 227 <p> 228 When we call 229 230 <span><code class="function">openDb()</code>,</span> 231 which is what we use 232 to open our database, we no not provide a database filename for the 233 third parameter. When the filename is <code class="literal">NULL</code>, the database is not 234 backed by the filesystem. 235 </p> 236 <pre class="programlisting"> // If we had utility threads (for running checkpoints or 237 // deadlock detection, for example) we would spawn those 238 // here. However, for a simple example such as this, 239 // that is not required. 240 241 // Open the database 242 openDb(&dbp, progName, <strong class="userinput"><code>NULL</code></strong>, 243 envp, DB_DUPSORT); 244 </pre> 245 <p> 246 After that, our <code class="function">main()</code> function is unchanged, 247 except that when we 248 249 <span>check for exceptions on the database open,</span> 250 we change the error message string so as to not reference the database filename. 251 </p> 252 <pre class="programlisting"> // Initialize a pthread mutex. Used to help provide thread ids. 253 (void)pthread_mutex_init(&thread_num_lock, NULL); 254 255 // Start the writer threads. 256 for (i = 0; i < NUMWRITERS; i++) 257 (void)pthread_create( 258 &writerThreads[i], NULL, 259 writerThread, 260 (void *)dbp); 261 262 // Join the writers 263 for (i = 0; i < NUMWRITERS; i++) 264 (void)pthread_join(writerThreads[i], NULL); 265 266 } catch(DbException &e) { 267 <strong class="userinput"><code>std::cerr << "Error opening database environment: "</code></strong> 268 << std::endl; 269 std::cerr << e.what() << std::endl; 270 return (EXIT_FAILURE); 271 } 272 273 try { 274 // Close our database handle if it was opened. 275 if (dbp != NULL) 276 dbp->close(0); 277 278 // Close our environment if it was opened. 279 if (envp != NULL) 280 envp->close(0); 281 } catch(DbException &e) { 282 std::cerr << "Error closing database and environment." 283 << std::endl; 284 std::cerr << e.what() << std::endl; 285 return (EXIT_FAILURE); 286 } 287 288 // Final status message and return. 289 290 std::cout << "I'm all done." << std::endl; 291 return (EXIT_SUCCESS); 292} </pre> 293 <p> 294 That completes <code class="function">main()</code>. The bulk of our 295 <code class="function">writerThread()</code> function implementation is 296 unchanged from the initial transaction example, except that we now pass 297 <code class="function">countRecords</code> a transaction handle, rather than configuring our 298 application to perform uncommitted reads. Both mechanisms work well-enough 299 for preventing a self-deadlock. However, the individual count 300 in this example will tend to be lower than the counts seen in 301 the previous transaction example, because 302 <code class="function">countRecords()</code> can no longer see records 303 created but not yet committed by other threads. 304 </p> 305 <pre class="programlisting">// A function that performs a series of writes to a 306// Berkeley DB database. The information written 307// to the database is largely nonsensical, but the 308// mechanism of transactional commit/abort and 309// deadlock detection is illustrated here. 310void * 311writerThread(void *args) 312{ 313 Db *dbp = (Db *)args; 314 DbEnv *envp = dbp->get_env(dbp); 315 316 int j, thread_num; 317 int max_retries = 20; // Max retry on a deadlock 318 char *key_strings[] = {"key 1", "key 2", "key 3", "key 4", 319 "key 5", "key 6", "key 7", "key 8", 320 "key 9", "key 10"}; 321 322 // Get the thread number 323 (void)pthread_mutex_lock(&thread_num_lock); 324 global_thread_num++; 325 thread_num = global_thread_num; 326 (void)pthread_mutex_unlock(&thread_num_lock); 327 328 // Initialize the random number generator 329 srand((u_int)pthread_self()); 330 331 // Perform 50 transactions 332 for (int i=0; i<50; i++) { 333 DbTxn *txn; 334 bool retry = true; 335 int retry_count = 0; 336 // while loop is used for deadlock retries 337 while (retry) { 338 // try block used for deadlock detection and 339 // general db exception handling 340 try { 341 342 // Begin our transaction. We group multiple writes in 343 // this thread under a single transaction so as to 344 // (1) show that you can atomically perform multiple 345 // writes at a time, and (2) to increase the chances 346 // of a deadlock occurring so that we can observe our 347 // deadlock detection at work. 348 349 // Normally we would want to avoid the potential for 350 // deadlocks, so for this workload the correct thing 351 // would be to perform our puts with auto commit. But 352 // that would excessively simplify our example, so we 353 // do the "wrong" thing here instead. 354 txn = NULL; 355 envp->txn_begin(NULL, &txn, 0); 356 // Perform the database write for this transaction. 357 for (j = 0; j < 10; j++) { 358 Dbt key, value; 359 key.set_data(key_strings[j]); 360 key.set_size((strlen(key_strings[j]) + 1) * 361 sizeof(char)); 362 363 int payload = rand() + i; 364 value.set_data(&payload); 365 value.set_size(sizeof(int)); 366 367 // Perform the database put 368 dbp->put(txn, &key, &value, 0); 369 } 370 371 // countRecords runs a cursor over the entire database. 372 // We do this to illustrate issues of deadlocking 373 std::cout << thread_num << " : Found " 374 << countRecords(dbp, <strong class="userinput"><code>txn</code></strong>) 375 << " records in the database." << std::endl; 376 377 std::cout << thread_num << " : committing txn : " << i 378 << std::endl; 379 380 // commit 381 try { 382 txn->commit(0); 383 retry = false; 384 txn = NULL; 385 } catch (DbException &e) { 386 std::cout << "Error on txn commit: " 387 << e.what() << std::endl; 388 } 389 } catch (DbDeadlockException &de) { 390 // First thing we MUST do is abort the transaction. 391 if (txn != NULL) 392 (void)txn->abort(); 393 394 // Now we decide if we want to retry the operation. 395 // If we have retried less than max_retries, 396 // increment the retry count and goto retry. 397 if (retry_count < max_retries) { 398 std::cout << "############### Writer " << thread_num 399 << ": Got DB_LOCK_DEADLOCK.\n" 400 << "Retrying write operation." 401 << std::endl; 402 retry_count++; 403 retry = true; 404 } else { 405 // Otherwise, just give up. 406 std::cerr << "Writer " << thread_num 407 << ": Got DeadLockException and out of " 408 << "retries. Giving up." << std::endl; 409 retry = false; 410 } 411 } catch (DbException &e) { 412 std::cerr << "db put failed" << std::endl; 413 std::cerr << e.what() << std::endl; 414 if (txn != NULL) 415 txn->abort(); 416 retry = false; 417 } catch (std::exception &ee) { 418 std::cerr << "Unknown exception: " << ee.what() << std::endl; 419 return (0); 420 } 421 } 422 } 423 return (0); 424} </pre> 425 <p> 426 Next we update 427 428 <span><code class="function">countRecords()</code>.</span> 429 The only difference 430 here is that we no longer specify <code class="literal">DB_READ_UNCOMMITTED</code> when 431 we open our cursor. Note that even this minor change is not required. 432 If we do not configure our database to support uncommitted reads, 433 <code class="literal">DB_READ_UNCOMMITTED</code> on the cursor open will be silently 434 ignored. However, we remove the flag anyway from the cursor open so as to 435 avoid confusion. 436</p> 437 <pre class="programlisting">int 438countRecords(Db *dbp, DbTxn *txn) 439{ 440 441 Dbc *cursorp = NULL; 442 int count = 0; 443 444 try { 445 // Get the cursor 446 dbp->cursor(txn, &cursorp, <strong class="userinput"><code>0</code></strong>); 447 448 Dbt key, value; 449 while (cursorp->get(&key, &value, DB_NEXT) == 0) { 450 count++; 451 } 452 } catch (DbDeadlockException &de) { 453 std::cerr << "countRecords: got deadlock" << std::endl; 454 cursorp->close(); 455 throw de; 456 } catch (DbException &e) { 457 std::cerr << "countRecords error:" << std::endl; 458 std::cerr << e.what() << std::endl; 459 } 460 461 if (cursorp != NULL) { 462 try { 463 cursorp->close(); 464 } catch (DbException &e) { 465 std::cerr << "countRecords: cursor close failed:" << std::endl; 466 std::cerr << e.what() << std::endl; 467 } 468 } 469 470 return (count); 471} </pre> 472 <p> 473 Finally, we update 474 475 <span><code class="function">openDb()</code>.</span> 476 This involves 477 removing <code class="literal">DB_READ_UNCOMMITTED</code> from the 478 open flags. 479 480 481 </p> 482 <pre class="programlisting">// Open a Berkeley DB database 483int 484openDb(Db **dbpp, const char *progname, const char *fileName, 485 DbEnv *envp, u_int32_t extraFlags) 486{ 487 int ret; 488 u_int32_t openFlags; 489 490 try { 491 Db *dbp = new Db(envp, 0); 492 493 // Point to the new'd Db 494 *dbpp = dbp; 495 496 if (extraFlags != 0) 497 ret = dbp->set_flags(extraFlags); 498 499 // Now open the database 500 <strong class="userinput"><code>openFlags = DB_CREATE | // Allow database creation 501 DB_THREAD | 502 DB_AUTO_COMMIT; // Allow auto commit</code></strong> 503 504 dbp->open(NULL, // Txn pointer 505 fileName, // File name 506 NULL, // Logical db name 507 DB_BTREE, // Database type (using btree) 508 openFlags, // Open flags 509 0); // File mode. Using defaults 510 } catch (DbException &e) { 511 std::cerr << progname << ": openDb: db open failed:" << std::endl; 512 std::cerr << e.what() << std::endl; 513 return (EXIT_FAILURE); 514 } 515 516 return (EXIT_SUCCESS); 517} </pre> 518 <p> 519 This completes our in-memory transactional example. If you would like to 520 experiment with this code, you can find the example in the following 521 location in your DB distribution: 522</p> 523 <pre class="programlisting"><span class="emphasis"><em>DB_INSTALL</em></span>/examples_cxx/txn_guide</pre> 524 </div> 525 <div class="navfooter"> 526 <hr /> 527 <table width="100%" summary="Navigation footer"> 528 <tr> 529 <td width="40%" align="left"><a accesskey="p" href="txnexample_c.html">Prev</a> </td> 530 <td width="20%" align="center"> 531 <a accesskey="u" href="wrapup.html">Up</a> 532 </td> 533 <td width="40%" align="right"> </td> 534 </tr> 535 <tr> 536 <td width="40%" align="left" valign="top">Transaction Example </td> 537 <td width="20%" align="center"> 538 <a accesskey="h" href="index.html">Home</a> 539 </td> 540 <td width="40%" align="right" valign="top"> </td> 541 </tr> 542 </table> 543 </div> 544 </body> 545</html> 546