1<?xml version="1.0" encoding="UTF-8" standalone="no"?> 2<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 3<html xmlns="http://www.w3.org/1999/xhtml"> 4 <head> 5 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> 6 <title>Writing In-Memory Berkeley DB Applications</title> 7 <link rel="stylesheet" href="gettingStarted.css" type="text/css" /> 8 <meta name="generator" content="DocBook XSL Stylesheets V1.62.4" /> 9 <link rel="home" href="index.html" title="Writing In-Memory Berkeley DB Applications" /> 10 </head> 11 <body> 12 <div class="navheader"> 13 <table width="100%" summary="Navigation header"> 14 <tr> 15 <th colspan="3" align="center">Writing In-Memory Berkeley DB Applications</th> 16 </tr> 17 </table> 18 <hr /> 19 </div> 20 <div class="article" lang="en" xml:lang="en"> 21 <div class="titlepage"> 22 <div> 23 <div> 24 <h1 class="title"><a id="id387251"></a>Writing In-Memory Berkeley DB Applications</h1> 25 </div> 26 <div> 27 <div class="legalnotice"> 28 <p class="legalnotice-title"> 29 <b>Legal Notice</b> 30 </p> 31 <p> 32 This documentation is distributed under an open source license. 33 You may review the terms of this license at: 34 <a href="http://www.oracle.com/technology/software/products/berkeley-db/htdocs/oslicense.html" target="_top">http://www.oracle.com/technology/software/products/berkeley-db/htdocs/oslicense.html</a> 35 36 37 </p> 38 <p> 39 Oracle, Berkeley DB, 40 41 42 and 43 Sleepycat are trademarks or registered trademarks of 44 Oracle. All rights to these marks are reserved. 45 No third-party use is permitted without the 46 express prior written consent of Oracle. 47 </p> 48 <p> 49 To obtain a copy of this document's original source code, please 50 submit a request to the Oracle Technology Network forum at: 51 <a href="http://forums.oracle.com/forums/forum.jspa?forumID=271" target="_top">http://forums.oracle.com/forums/forum.jspa?forumID=271</a> 52 53 54 </p> 55 </div> 56 </div> 57 <div> 58 <p class="pubdate">4/12/2010</p> 59 </div> 60 </div> 61 <div></div> 62 <hr /> 63 </div> 64 <div class="toc"> 65 <p> 66 <b>Table of Contents</b> 67 </p> 68 <dl> 69 <dt> 70 <span class="sect1"> 71 <a href="index.html#intro">Introduction</a> 72 </span> 73 </dt> 74 <dt> 75 <span class="sect1"> 76 <a href="index.html#resources">Resources to be Managed</a> 77 </span> 78 </dt> 79 <dt> 80 <span class="sect1"> 81 <a href="index.html#strategies">Strategies</a> 82 </span> 83 </dt> 84 <dt> 85 <span class="sect1"> 86 <a href="index.html#dbfiles">Keeping the Database in Memory</a> 87 </span> 88 </dt> 89 <dt> 90 <span class="sect1"> 91 <a href="index.html#env">Keeping Environments in Memory</a> 92 </span> 93 </dt> 94 <dt> 95 <span class="sect1"> 96 <a href="index.html#cachesize">Sizing the Cache</a> 97 </span> 98 </dt> 99 <dd> 100 <dl> 101 <dt> 102 <span class="sect2"> 103 <a href="index.html#dbcachesize-db">Specifying a Cache Size using the 104 Database Handle</a> 105 </span> 106 </dt> 107 <dt> 108 <span class="sect2"> 109 <a href="index.html#dbcachesize-env">Specifying a Cache Size using the 110 Environment Handle</a> 111 </span> 112 </dt> 113 </dl> 114 </dd> 115 <dt> 116 <span class="sect1"> 117 <a href="index.html#mpool-nofile">Keeping Temporary Overflow Pages in Memory</a> 118 </span> 119 </dt> 120 <dt> 121 <span class="sect1"> 122 <a href="index.html#logs">Keeping Logs in Memory</a> 123 </span> 124 </dt> 125 <dt> 126 <span class="sect1"> 127 <a href="index.html#in-mem-rep">In-Memory Replicated Applications</a> 128 </span> 129 </dt> 130 <dt> 131 <span class="sect1"> 132 <a href="index.html#example_in-mem">Example In-Memory Application</a> 133 </span> 134 </dt> 135 </dl> 136 </div> 137 <div class="sect1" lang="en" xml:lang="en"> 138 <div class="titlepage"> 139 <div> 140 <div> 141 <h2 class="title" style="clear: both"><a id="intro"></a>Introduction</h2> 142 </div> 143 </div> 144 <div></div> 145 </div> 146 <p> 147 This document describes how to write a DB application 148 that keeps its data entirely in memory. That is, the 149 application writes no data to disk. For this reason, 150 in-memory only applications typically discard all 151 data durability guarantees. 152 </p> 153 <div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"> 154 <h3 class="title">Note</h3> 155 <p> 156 This document assume familiarity with the 157 <i class="citetitle">Getting Started with Berkeley DB</i> guide. If 158 you are using environments or transactions, then 159 you should also have an understanding of the 160 concepts in <i class="citetitle">Berkeley DB Getting Started with Transaction Processing</i> 161 guide. 162 </p> 163 </div> 164 <p> 165 There are several reasons why you might want to write an 166 in-memory only DB application. For platforms on which a 167 disk drive is available to back your data, an in-memory 168 application might be desirable from a performance 169 perspective. In this case, the data that your application 170 manages might be generated during run-time and so is of no 171 interest across application startups. 172 </p> 173 <p> 174 Other platforms are disk-less. In this case, an in-memory only 175 configuration is the only possible choice. Note that this 176 document's primary focus is disk-less systems for which an 177 on-disk filesystem is not available. 178 </p> 179 </div> 180 <div class="sect1" lang="en" xml:lang="en"> 181 <div class="titlepage"> 182 <div> 183 <div> 184 <h2 class="title" style="clear: both"><a id="resources"></a>Resources to be Managed</h2> 185 </div> 186 </div> 187 <div></div> 188 </div> 189 <p> 190 Before continuing, it is worthwhile to briefly 191 describe the DB resources that must be managed 192 if you are going to configure an in-memory only 193 DB application. These are resources that are by 194 default persisted on disk, or backed by a 195 filesystem on disk. Some configuration is therefore 196 required to keep these resources in-memory only. 197 </p> 198 <p> 199 Note that you can configure only some of these 200 resources to be held in-memory, and allow others to 201 be backed by disk. This might be desireable for 202 some applications that wish to improve application 203 performance by, for example, eliminating disk I/O 204 for some, but not all, of these resources. However, 205 for the purpose of this document, we assume you 206 want to configure all of these resources to be held 207 in memory. 208 </p> 209 <p> 210 Managing these resources for an in-memory 211 application is described in detail later in this 212 article. 213 </p> 214 <div class="itemizedlist"> 215 <ul type="disc"> 216 <li> 217 <p> 218 Database files 219 </p> 220 <p> 221 Normally, DB stores your 222 database data within on-disk files. 223 For an entirely in-memory 224 application, you are required to turn 225 off this behavior. 226 </p> 227 </li> 228 <li> 229 <p> 230 Environment region files 231 </p> 232 <p> 233 DB environments manage region 234 files for a variety of purposes. 235 Normally these are backed by the 236 filesystem, but by using the 237 appropriate configuration option 238 you can cause region files to 239 reside in memory only. 240 </p> 241 </li> 242 <li> 243 <p> 244 Database cache 245 </p> 246 <p> 247 The DB cache must be configured 248 large enough to hold all your data 249 in memory. If you do not size your 250 cache large enough, then DB 251 will attempt to write pages to 252 disk. In a disk-less system, this 253 will result in an abnormal 254 termination of your program. 255 </p> 256 </li> 257 <li> 258 <p> 259 Logs 260 </p> 261 <p> 262 DB logs describe the write activity 263 that has occurred in your application. 264 They are used for a number of purposes, 265 such as recovery operations for 266 applications that are seeking data 267 durability guarantees. 268 </p> 269 <p> 270 For in-memory applications that do not 271 care about durability guarantees, logs 272 are still required if you want 273 transactional benefits other than 274 durability (such as isolation 275 and atomicity). This is because DB's 276 transactional subsystem requires logs, 277 even if you want to discard all data 278 durability guarantees. 279 </p> 280 <p> 281 If this describes your application, you 282 must enable logs but configure them to 283 reside only within memory. 284 </p> 285 </li> 286 <li> 287 <p> 288 Temporary overflow pages 289 </p> 290 <p> 291 You must disallow backing temporary 292 database files with the 293 filesystem. This is mostly a 294 configuration option, but it is 295 also dependent upon sizing your 296 cache correctly. 297 </p> 298 </li> 299 </ul> 300 </div> 301 <p> 302 In addition to these, if you are writing a replicated application (see 303 <i class="citetitle">Berkeley DB Getting Started with Replicated Applications</i> for an introduction to writing replicated 304 applications), there is internal replication information that is normally kept 305 on-disk. You can cause this information to be kept in-memory if you are 306 willing to accept some limitations in how your replicated application operates. 307 See <a href="index.html#in-mem-rep">In-Memory Replicated Applications</a> for more information. 308 </p> 309 </div> 310 <div class="sect1" lang="en" xml:lang="en"> 311 <div class="titlepage"> 312 <div> 313 <div> 314 <h2 class="title" style="clear: both"><a id="strategies"></a>Strategies</h2> 315 </div> 316 </div> 317 <div></div> 318 </div> 319 <p> 320 DB is an extremely flexible product that can be 321 adapted to suit almost any data management 322 requirements. This means that you can configure 323 DB to operate entirely within memory, but still 324 retain some data durability guarantees or even 325 throw away all durability guarantees. 326 </p> 327 <p> 328 Data durability guarantees describe how persistent 329 your data is. That is, once you have made a change 330 to the data stored in your database, how much of a 331 guarantee do you require that that modification 332 will persist (not be lost)? There are a great many 333 options here. For the absolute best durability guarantee, you 334 should fully transaction-protect your data and 335 allow your data to be written to disk upon each 336 transaction commit. Of course, this guarantee is 337 not available for disk-less systems. 338 </p> 339 <p> 340 At the opposite end of the spectrum, you can throw 341 away all your data once your application is done 342 with it (for example, at application shutdown). 343 This is a good option if you are using DB only 344 as a kind of caching mechanism. In this case, you 345 obviously must either generate your data entirely 346 during runtime, or obtain it from some remote 347 location during application startup. 348 </p> 349 <p> 350 There are also durability options that exist somewhere in 351 between these two extremes. For example, disk-less 352 systems are sometimes backed by some kind of flash 353 memory (e.g. compact flash cards). These 354 platforms may want to limit the number of writes 355 applied to the backing media because it is capable 356 of accepting only a limited number of writes before it must 357 be replaced. For this reason, you might want to 358 limit data writes to the flash media only during 359 specific moments during your application's 360 runtime; for example, only at application shutdown. 361 </p> 362 <p> 363 Another way to obtain a data durability guarantee 364 for in-memory configurations is to use DB 365 replication to commit data to the network. To support this 366 strategy, the disk-less system is required to 367 be the master server. The use of replication in this 368 way actually results 369 in a fairly high durability guarantee for your 370 data while providing the benefit of avoiding disk 371 I/O on transaction commit. 372 </p> 373 <p> 374 The point here is to be aware that a great many 375 options are available to you when writing an 376 in-memory only application. That said, the focus of 377 this document is strictly disk-less systems; that 378 is, systems that provide no means by which 379 data can be written to persistent media. 380 </p> 381 <p> 382 For an introductory description of 383 DB replication, please see the 384 <i class="citetitle">Berkeley DB Getting Started with Replicated Applications</i> guide. 385 </p> 386 </div> 387 <div class="sect1" lang="en" xml:lang="en"> 388 <div class="titlepage"> 389 <div> 390 <div> 391 <h2 class="title" style="clear: both"><a id="dbfiles"></a>Keeping the Database in Memory</h2> 392 </div> 393 </div> 394 <div></div> 395 </div> 396 <p> 397 Normally DB databases are backed by the 398 filesystem. For in-memory applications, such as 399 is required on disk-less systems, you can cause your 400 database to only reside in-memory. That is, 401 their contents are stored entirely within DB's 402 cache. 403 </p> 404 <p> 405 There are two requirements for keeping your 406 database(s) in-memory. 407 The first is to size your cache such that it is big 408 enough to hold all your data in-memory. If your 409 cache fills up, then DB will return 410 <tt class="literal">ENOMEM</tt> on the next operation 411 that requests additional pages in the cache. As with all errors while 412 updating a database, the current transaction must be aborted. If the update 413 was being done without a transaction, then the application must close its 414 environment and database handles, reopen them, and then refresh the database 415 from some backup data source. 416 </p> 417 <p> 418 For information on setting the cache size, see 419 <a href="index.html#cachesize">Sizing the Cache</a>. 420 </p> 421 <p> 422 Beyond cache sizing, you also must tell DB not 423 to back your database with an on-disk file. You do 424 this by NOT providing a database file name 425 when you open the database. Note that the database 426 file name is different from the database name; you 427 can name your in-memory databases even if you 428 are not storing them in an on-disk file. 429 </p> 430 <p> 431 For example: 432 </p> 433 <pre class="programlisting">#include "db.h" 434 435... 436 437 int ret, ret_c; 438 const char *db_name = "in_mem_db1"; 439 u_int32_t db_flags; /* For open flags */ 440 DB *dbp; /* Database handle */ 441 442... 443 444 /* Initialize the DB handle */ 445 ret = db_create(&dbp, NULL, 0); 446 if (ret != 0) { 447 fprintf(stderr, "Error creating database handle: %s\n", 448 db_strerror(ret)); 449 goto err; 450 } 451 452 db_flags = DB_CREATE; /* If it doesn't exist, create it */ 453 454 /* 455 * Open the database. Note that the file name is NULL. 456 * This forces the database to be stored in the cache only. 457 * Also note that the database has a name, even though its 458 * file name is NULL. 459 */ 460 ret = dbp->open(dbp, /* Pointer to the database */ 461 NULL, /* Txn pointer */ 462 <b class="userinput"><tt>NULL,</tt></b> /* File name is not specified on purpose */ 463 db_name, /* Logical db name. */ 464 DB_BTREE, /* Database type (using btree) */ 465 db_flags, /* Open flags */ 466 0); /* File mode. Using defaults */ 467 if (ret != 0) { 468 dbp->err(dbp, ret, "Database open failed"); 469 goto err; 470 } 471 472err: 473 /* Close the database */ 474 if (dbp != NULL) { 475 ret_c = dbp->close(dbp, 0); 476 if (ret_c != 0) { 477 fprintf(stderr, "%s database close failed.\n", 478 db_strerror(ret_c)); 479 ret = ret_c 480 } 481 } </pre> 482 </div> 483 <div class="sect1" lang="en" xml:lang="en"> 484 <div class="titlepage"> 485 <div> 486 <div> 487 <h2 class="title" style="clear: both"><a id="env"></a>Keeping Environments in Memory</h2> 488 </div> 489 </div> 490 <div></div> 491 </div> 492 <p> 493 Like databases, DB environments are usually backed by the filesystem. In 494 fact, a big part of what environments do is identify the location on disk 495 where resources (such as log and database files) are kept. 496 </p> 497 <p> 498 However, environments are also used for managing resources, such as 499 obtaining new transactions, so they are useful even when building an 500 in-memory application. Therefore, if you are going to use an environment for 501 your in-memory DB application, you must configure it such that it does 502 not want to use the filesystem. There are two things you need to do here. 503 </p> 504 <p> 505 First, when you open your environment, do NOT identify a home directory. To 506 accomplish this, you must: 507 </p> 508 <div class="itemizedlist"> 509 <ul type="disc"> 510 <li> 511 <p> 512 NOT provide a value for the <tt class="literal">db_home</tt> 513 parameter on the 514 <tt class="methodname">DB_ENV->open()</tt> 515 method. 516 </p> 517 </li> 518 <li> 519 <p> 520 NOT have a DB_HOME environment variable set. 521 </p> 522 </li> 523 <li> 524 <p> 525 NOT call any of the methods that affect file naming 526 (<tt class="methodname">DB_ENV->set_data_dir()</tt>, 527 <tt class="methodname">DB_ENV->set_lg_dir()</tt>, or 528 <tt class="methodname">DB_ENV->set_tmp_dir()</tt>). 529 </p> 530 </li> 531 </ul> 532 </div> 533 <p> 534 Beyond this, you must also ensure that regions are 535 backed by heap memory 536 instead of by the filesystem or system shared memory. You do this when you 537 open your environment by specifying the <tt class="literal">DB_PRIVATE</tt> flag. 538 Note that the use of <tt class="literal">DB_PRIVATE</tt> means that other processes 539 cannot share the environment. Consequently, your in-memory only application 540 must be a single-process, although it can be multi-threaded. 541 </p> 542 <p> 543 For example: 544 </p> 545 <pre class="programlisting">#include "db.h" 546 547... 548 549 int ret, ret_c; 550 u_int32_t env_flags; /* For open flags */ 551 DB_ENV *envp; /* Environment handle */ 552 553... 554 555 /* Initialize the ENV handle */ 556 ret = db_env_create(&envp, 0); 557 if (ret != 0) { 558 fprintf(stderr, "Error creating environment handle: %s\n", 559 db_strerror(ret)); 560 goto err; 561 } 562 563 564 /* 565 * Environment flags. These are for a non-threaded 566 * in-memory application. 567 */ 568 env_flags = 569 DB_CREATE | /* Create the environment if it does not exist */ 570 DB_INIT_LOCK | /* Initialize the locking subsystem */ 571 DB_INIT_LOG | /* Initialize the logging subsystem */ 572 DB_INIT_TXN | /* Initialize the transactional subsystem. This 573 * also turns on logging. */ 574 DB_INIT_MPOOL | /* Initialize the memory pool (in-memory cache) */ 575 <b class="userinput"><tt>DB_PRIVATE | /* Region files are not backed by the filesystem. 576 * Instead, they are backed by heap memory. */</tt></b> 577 578 /* 579 * Now open the environment. Notice that we do not provide a location 580 * for the environment's home directory. This is required for an 581 * in-memory only application. 582 */ 583 ret = envp->open(envp, <b class="userinput"><tt>NULL</tt></b>, env_flags, 0); 584 if (ret != 0) { 585 fprintf(stderr, "Error opening environment: %s\n", 586 db_strerror(ret)); 587 goto err; 588 } 589 590err: 591 /* Close the environment */ 592 if (envp != NULL) { 593 ret_c = envp->close(envp, 0); 594 if (ret_c != 0) { 595 fprintf(stderr, "environment close failed: %s\n", 596 db_strerror(ret_c)); 597 ret = ret_c 598 } 599 }</pre> 600 </div> 601 <div class="sect1" lang="en" xml:lang="en"> 602 <div class="titlepage"> 603 <div> 604 <div> 605 <h2 class="title" style="clear: both"><a id="cachesize"></a>Sizing the Cache</h2> 606 </div> 607 </div> 608 <div></div> 609 </div> 610 <p> 611 One of the most important considerations for an 612 in-memory application is to ensure that your 613 database cache is large enough. In a normal 614 application that is not in-memory, the cache 615 provides a mechanism by which frequently-used 616 data can be accessed without resorting to disk I/O. 617 For an in-memory application, the cache is the only 618 location your data can exist so it is critical that 619 you make the cache large enough for your data set. 620 </p> 621 <p> 622 You specify the size of your cache at application 623 startup. Obviously you should not specify a size 624 that is larger than available memory. Note that the 625 size you specify for your cache is actually a 626 <span class="emphasis"><em>maximum</em></span> size; DB will only 627 use memory as required so if you specify a cache 628 size of 1 GB but your data set is only ever 10 MB 629 in size, then 10 MB is what DB will use. 630 </p> 631 <p> 632 Note that if you specify a cache size less than 633 500 MB, then the cache size is automatically 634 increased by 25% to account for internal overhead 635 purposes. 636 </p> 637 <p> 638 There are two ways to specify a cache size, 639 depending on whether you are using a database 640 environment. 641 </p> 642 <div class="sect2" lang="en" xml:lang="en"> 643 <div class="titlepage"> 644 <div> 645 <div> 646 <h3 class="title"><a id="dbcachesize-db"></a>Specifying a Cache Size using the 647 Database Handle</h3> 648 </div> 649 </div> 650 <div></div> 651 </div> 652 <p> 653 To select a cache size using the database 654 handle, use the 655 <tt class="methodname">DB->set_cachesize()</tt> 656 method. Note that you cannot use this 657 method after the database has been opened. 658 </p> 659 <p> 660 Also, if you are using a database environment, 661 it is an error to use this method. See 662 the next section for details on 663 selecting your cache size. 664 </p> 665 <p> 666 The following code fragment creates a 667 database handle, sets the cache size to 668 10 MB and then opens the database: 669 </p> 670 <pre class="programlisting">#include "db.h" 671 672... 673 674 int ret, ret_c; 675 const char *db_name = "in_mem_db1"; 676 u_int32_t db_flags; /* For open flags */ 677 DB *dbp; /* Database handle */ 678 679... 680 681 /* Initialize the DB handle */ 682 ret = db_create(&dbp, NULL, 0); 683 if (ret != 0) { 684 fprintf(stderr, "Error creating database handle: %s\n", 685 db_strerror(ret)); 686 goto err; 687 } 688 689 <b class="userinput"><tt>/*************************************************************/ 690 /*************************************************************/ 691 /************* Set the cache size here **********************/ 692 /*************************************************************/ 693 /*************************************************************/ 694 695 ret = dbp->set_cachesize(dbp, 696 0, /* 0 gigabytes */ 697 10 * 1024 * 1024, /* 10 megabytes */ 698 1); /* Create 1 cache. All memory will 699 * be allocated contiguously. */ 700 if (ret != 0) { 701 dbp->err(dbp, ret, "Database open failed"); 702 goto err; 703 }</tt></b> 704 705 706 db_flags = DB_CREATE; /* If it doesn't exist, create it */ 707 ret = dbp->open(dbp, /* Pointer to the database */ 708 NULL, /* Txn pointer */ 709 NULL, /* File name is not specified on purpose */ 710 db_name, /* Logical db name */ 711 DB_BTREE, /* Database type (using btree) */ 712 db_flags, /* Open flags */ 713 0); /* File mode. Using defaults */ 714 if (ret != 0) { 715 dbp->err(dbp, ret, "Database open failed"); 716 goto err; 717 } 718 719err: 720 /* Close the database */ 721 if (dbp != NULL) { 722 ret_c = dbp->close(dbp, 0); 723 if (ret_c != 0) { 724 fprintf(stderr, "%s database close failed.\n", 725 db_strerror(ret_c)); 726 ret = ret_c 727 } 728 } </pre> 729 </div> 730 <div class="sect2" lang="en" xml:lang="en"> 731 <div class="titlepage"> 732 <div> 733 <div> 734 <h3 class="title"><a id="dbcachesize-env"></a>Specifying a Cache Size using the 735 Environment Handle</h3> 736 </div> 737 </div> 738 <div></div> 739 </div> 740 <p> 741 To select a cache size using the 742 environment handle, use the 743 <tt class="methodname">ENV->set_cachesize()</tt> 744 method. Note that you cannot use this 745 method after the environment has been opened. 746 </p> 747 <p> 748 The following code fragment creates an 749 environment handle, sets the cache size to 750 10 MB and then opens the environment: Once 751 opened, you can use the environment when 752 you open your database(s). This means all 753 your databases will use the same 754 cache. 755 </p> 756 <pre class="programlisting">#include "db.h" 757 758... 759 760 int ret, ret_c; 761 u_int32_t env_flags; /* For open flags */ 762 DB_ENV *envp; /* Environment handle */ 763 764... 765 766 /* Initialize the ENV handle */ 767 ret = db_env_create(&envp, 0); 768 if (ret != 0) { 769 fprintf(stderr, "Error creating environment handle: %s\n", 770 db_strerror(ret)); 771 goto err; 772 } 773 774 <b class="userinput"><tt>/*************************************************************/ 775 /*************************************************************/ 776 /************* Set the cache size here **********************/ 777 /*************************************************************/ 778 /*************************************************************/ 779 780 ret = 781 envp->set_cachesize(envp, 782 0, /* 0 gigabytes */ 783 10 * 1024 * 1024, /* 10 megabytes */ 784 1); /* Create 1 cache. All memory will 785 * be allocated contiguously. */ 786 if (ret != 0) { 787 envp->err(envp, ret, "Environment open failed"); 788 goto err; 789 }</tt></b> 790 791 /* 792 * Environment flags. These are for a non-threaded 793 * in-memory application. 794 */ 795 env_flags = 796 DB_CREATE | /* Create the environment if it does not exist */ 797 DB_INIT_LOCK | /* Initialize the locking subsystem */ 798 DB_INIT_LOG | /* Initialize the logging subsystem */ 799 DB_INIT_TXN | /* Initialize the transactional subsystem. This 800 * also turns on logging. */ 801 DB_INIT_MPOOL | /* Initialize the memory pool (in-memory cache) */ 802 DB_PRIVATE | /* Region files are not backed by the filesystem. 803 * Instead, they are backed by heap memory. */ 804 805 /* 806 * Now open the environment. Notice that we do not provide a location 807 * for the environment's home directory. This is required for an 808 * in-memory only application. 809 */ 810 ret = envp->open(envp, NULL, env_flags, 0); 811 if (ret != 0) { 812 fprintf(stderr, "Error opening environment: %s\n", 813 db_strerror(ret)); 814 goto err; 815 } 816 817err: 818 /* Close the environment */ 819 if (envp != NULL) { 820 ret_c = envp->close(envp, 0); 821 if (ret_c != 0) { 822 fprintf(stderr, "environment close failed: %s\n", 823 db_strerror(ret_c)); 824 ret = ret_c 825 } 826 } </pre> 827 </div> 828 </div> 829 <div class="sect1" lang="en" xml:lang="en"> 830 <div class="titlepage"> 831 <div> 832 <div> 833 <h2 class="title" style="clear: both"><a id="mpool-nofile"></a>Keeping Temporary Overflow Pages in Memory</h2> 834 </div> 835 </div> 836 <div></div> 837 </div> 838 <p> 839 Normally, when a database is opened, a temporary 840 file is opened on-disk to back the database. This 841 file is used if the database grows so large that it 842 fills the entire cache. At that time, database 843 pages that do not fit into the in-memory cache file 844 are written temporarily to this file. 845 </p> 846 <p> 847 For disk-less systems, you should configure your 848 databases so that this temporary file is not 849 created. When you do this, any attempt to create 850 new database pages once the cache is full will 851 fail. 852 </p> 853 <p> 854 You configure this option on a per-database handle 855 basis. That means you must configure this for every 856 in-memory database that your application uses. 857 </p> 858 <p> 859 To set this option, obtain the 860 <tt class="literal">DB_MPOOLFILE</tt> field from you 861 <tt class="literal">DB</tt> and then configure 862 <tt class="literal">DB_MPOOL_NOFILE</tt> using the 863 <tt class="methodname">DB_MPOOLFILE->set_flags()</tt> 864 method. 865 </p> 866 <p> 867 For example: 868 </p> 869 <pre class="programlisting">#include "db.h" 870 871... 872 873 int ret, ret_c; 874 u_int32_t env_flags; /* For open flags */ 875 DB_ENV *envp; /* Environment handle */ 876 877... 878 879 /* 880 * Configure the cache file. This can be done 881 * at any point in the application's life once the 882 * DB handle has been created. 883 */ 884 mpf = dbp->get_mpf(dbp); 885 ret = mpf->set_flags(mpf, DB_MPOOL_NOFILE, 1); 886 887 if (ret != 0) { 888 fprintf(stderr, 889 "Attempt failed to configure for no backing of temp files: %s\n", 890 db_strerror(ret)); 891 goto err; 892 } </pre> 893 </div> 894 <div class="sect1" lang="en" xml:lang="en"> 895 <div class="titlepage"> 896 <div> 897 <div> 898 <h2 class="title" style="clear: both"><a id="logs"></a>Keeping Logs in Memory</h2> 899 </div> 900 </div> 901 <div></div> 902 </div> 903 <p> 904 DB logs describe the write activity that has occurred in 905 your application. For a purely in-memory application, logs 906 should be used only if you wish to transaction-protect your 907 database writes as logs are required by the DB 908 transactional subsystem. 909 </p> 910 <p> 911 Note that transactions provide a number of guarantees. One 912 of these is not interesting to a purely in-memory 913 application (data durability). However, other transaction 914 guarantees such as isolation and atomicity might be of 915 interest to your application. 916 </p> 917 <p> 918 If this is the case for your application, then you must 919 configure your logs to be kept entirely in-memory. You 920 do this by setting a configuration option that prevents 921 DB from writing log data to disk. 922 Do this by setting the <tt class="literal">DB_LOG_IN_MEMORY</tt> flag using the 923 <tt class="methodname">DB_ENV->log_set_config()</tt> method. 924 </p> 925 <p> 926 In addition, you must configure your log buffer size so that it is capable 927 of holding all log information that can accumulate during your longest 928 running transaction. That is, make sure the in-memory log buffer is large 929 enough that no transaction will ever span the entire buffer. Also, avoid a 930 state where the in-memory buffer is full and no space can be freed because a 931 transaction that started the first log "file" is still active. 932 </p> 933 <p> 934 How much log buffer space is required is a function of the number of 935 transactions you have running concurrently, how long they last, and how much 936 write activity occurs within them. When in-memory logging is configured, the 937 default log buffer space is 1 MB. 938 </p> 939 <p> 940 You set your log buffer space using the 941 <tt class="methodname">DB_ENV->set_lg_bsize()</tt>. 942 </p> 943 <p> 944 For example, the following code fragment configure in-memory log usage, and 945 it configures the log buffer size to 10 MB: 946 </p> 947 <pre class="programlisting">#include "db.h" 948 949... 950 951 int ret, ret_c; 952 u_int32_t env_flags; /* For open flags */ 953 DB_ENV *envp; /* Environment handle */ 954 955... 956 957 /* Initialize the ENV handle */ 958 ret = db_env_create(&envp, 0); 959 if (ret != 0) { 960 fprintf(stderr, "Error creating environment handle: %s\n", 961 db_strerror(ret)); 962 goto err; 963 } 964 965 /* 966 * Environment flags. These are for a non-threaded 967 * in-memory application. 968 */ 969 env_flags = 970 DB_CREATE | /* Create the environment if it does not exist */ 971 DB_INIT_LOCK | /* Initialize the locking subsystem */ 972 DB_INIT_LOG | /* Initialize the logging subsystem */ 973 DB_INIT_TXN | /* Initialize the transactional subsystem. This 974 * also turns on logging. */ 975 DB_INIT_MPOOL | /* Initialize the memory pool (in-memory cache) */ 976 DB_PRIVATE | /* Region files are not backed by the filesystem. 977 * Instead, they are backed by heap memory. */ 978 979 980 <b class="userinput"><tt>/* Specify in-memory logging */ 981 ret = envp->log_set_config(envp, DB_LOG_IN_MEMORY, 1); 982 if (ret != 0) { 983 fprintf(stderr, "Error setting log subsystem to in-memory: %s\n", 984 db_strerror(ret)); 985 goto err; 986 } 987 988 /* 989 * Specify the size of the in-memory log buffer. 990 */ 991 ret = envp->set_lg_bsize(envp, 10 * 1024 * 1024); 992 if (ret != 0) { 993 fprintf(stderr, "Error increasing the log buffer size: %s\n", 994 db_strerror(ret)); 995 goto err; 996 }</tt></b> 997 998 /* 999 * Now open the environment. Notice that we do not provide a location 1000 * for the environment's home directory. This is required for an 1001 * in-memory only application. 1002 */ 1003 ret = envp->open(envp, NULL, env_flags, 0); 1004 if (ret != 0) { 1005 fprintf(stderr, "Error opening environment: %s\n", 1006 db_strerror(ret)); 1007 goto err; 1008 } 1009 1010err: 1011 /* Close the environment */ 1012 if (envp != NULL) { 1013 ret_c = envp->close(envp, 0); 1014 if (ret_c != 0) { 1015 fprintf(stderr, "environment close failed: %s\n", 1016 db_strerror(ret_c)); 1017 ret = ret_c 1018 } 1019 } </pre> 1020 </div> 1021 <div class="sect1" lang="en" xml:lang="en"> 1022 <div class="titlepage"> 1023 <div> 1024 <div> 1025 <h2 class="title" style="clear: both"><a id="in-mem-rep"></a>In-Memory Replicated Applications</h2> 1026 </div> 1027 </div> 1028 <div></div> 1029 </div> 1030 <p> 1031 If you are unfamiliar with writing DB replicated applications, or if you are 1032 simply uninterested in this topic, you can skip this section. 1033 </p> 1034 <p> 1035 There are many internal resources used by the DB replication subsystem which are 1036 by default backed by disk. These internal resources help the DB subsystem 1037 ensure election accuracy. While a complete description of these resources is beyond 1038 the scope of this article, you should know that you can cause all of these resources 1039 to be held entirely in-memory. But you do so with some small chance of operational 1040 errors in your replicated application. 1041 </p> 1042 <p> 1043 If you cause a replicated application to keep its internal replication resources 1044 in-memory, you run a small risk that elections will fail or be unable to complete. 1045 However, calling additional elections should eventually yield a winner. 1046 </p> 1047 <p> 1048 In addition, there is a slight possibility that the wrong site might win an 1049 election, which could result in the loss of data. This can happen if you have a site 1050 that is repeatedly crashing and trying to come back up. A site like this might be 1051 repeatedly sending out election information, and the repeated messages might confuse 1052 other sites. For replication applications that are not in-memory, these extra 1053 messages would be ignored by other sites because they would also contain some state 1054 information that allows other sites to know which election messages are relevant. 1055 But strictly in-memory replicated applications cannot maintain this state 1056 information, and so some other sites might become confused. The result might be that 1057 the wrong site could be elected master due to the inconsistent information that is 1058 available to them. 1059 </p> 1060 <div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"> 1061 <h3 class="title">Note</h3> 1062 <p> 1063 This is very much a corner case that you probably will never see in your 1064 production systems, especially if your sites are all stable and well-behaved. 1065 </p> 1066 </div> 1067 <p> 1068 If an election is won by the wrong site (site A), then some other site (site B) probably has 1069 more recent log files than the winner does. But since the wrong site A won the 1070 election, site B will sync with the new master. This will cause site B (and 1071 therefore, your entire replication group) to lose any log files it contains that are 1072 more recent than the files contained by site A. 1073 </p> 1074 <p> 1075 Also, when running an in-memory replicated application, it is often-times desireable for 1076 the master to run in-memory and the clients to commit data to disk. This helps to 1077 improve write through-put on the master by avoiding disk I/O. Instead, the master 1078 simply "commits to the network," and then eliminates its message ackowledgement 1079 expectations so that it is no longer waiting for clients to respond to the commit. 1080 </p> 1081 <p> 1082 If you are running a master that is configured to run with internal 1083 replication resources in-memory, you should never allow that site to 1084 appoint itself master again immediately after crashing or rebooting. 1085 Doing so results in a slightly higher risk of your client sites 1086 crashing. To determine your next master, you should either hold an 1087 election or appoint a different site to be master. 1088 </p> 1089 <p> 1090 In order to cause a replication site to run entire in-memory, do all of the things 1091 described previously in this document to place all other DB resources in-memory. 1092 Then, when configuring replication, specify <tt class="literal">DB_REP_CONF_INMEM</tt> to 1093 the <tt class="methodname">DB_ENV->rep_set_config()</tt> method. 1094 </p> 1095 </div> 1096 <div class="sect1" lang="en" xml:lang="en"> 1097 <div class="titlepage"> 1098 <div> 1099 <div> 1100 <h2 class="title" style="clear: both"><a id="example_in-mem"></a>Example In-Memory Application</h2> 1101 </div> 1102 </div> 1103 <div></div> 1104 </div> 1105 <p> 1106 The following brief example illustrates how to open 1107 an application that is entirely in-memory. The 1108 application opens an environment and a single 1109 database, and does this in a way that the database 1110 is transaction-protected. 1111 </p> 1112 <p> 1113 Transactions can be 1114 desirable for an in-memory application even though 1115 you discard your durability guarantees, because of 1116 the other things that transactions offer such as 1117 atomicity and isolation. 1118 </p> 1119 <p> 1120 Notice that the example does nothing other than 1121 open and close the environment and database. DB 1122 database reads and writes work identically between 1123 in-memory-only and durable applications (that is, 1124 applications that write database application to 1125 durable storage). Consequently, there is no point 1126 in illustrating those actions here. 1127 </p> 1128 <pre class="programlisting">/* We assume an ANSI-compatible compiler */ 1129#include <stdio.h> 1130#include <stdlib.h> 1131#include <string.h> 1132#include <db.h> 1133 1134 1135int 1136main(void) 1137{ 1138 /* Initialize our handles */ 1139 DB *dbp = NULL; 1140 DB_ENV *envp = NULL; 1141 DB_MPOOLFILE *mpf = NULL; 1142 1143 int ret, ret_t; 1144 const char *db_name = "in_mem_db1"; 1145 u_int32_t open_flags; 1146 1147 /* Create the environment */ 1148 ret = db_env_create(&envp, 0); 1149 if (ret != 0) { 1150 fprintf(stderr, "Error creating environment handle: %s\n", 1151 db_strerror(ret)); 1152 goto err; 1153 } 1154 1155 open_flags = 1156 DB_CREATE | /* Create the environment if it does not exist */ 1157 DB_INIT_LOCK | /* Initialize the locking subsystem */ 1158 DB_INIT_LOG | /* Initialize the logging subsystem */ 1159 DB_INIT_MPOOL | /* Initialize the memory pool (in-memory cache) */ 1160 DB_INIT_TXN | 1161 DB_PRIVATE; /* Region files are not backed by the filesystem. 1162 * Instead, they are backed by heap memory. */ 1163 1164 /* Specify in-memory logging */ 1165 ret = envp->log_set_config(envp, DB_LOG_IN_MEMORY, 1); 1166 if (ret != 0) { 1167 fprintf(stderr, "Error setting log subsystem to in-memory: %s\n", 1168 db_strerror(ret)); 1169 goto err; 1170 } 1171 /* 1172 * Specify the size of the in-memory log buffer. 1173 */ 1174 ret = envp->set_lg_bsize(envp, 10 * 1024 * 1024); 1175 if (ret != 0) { 1176 fprintf(stderr, "Error increasing the log buffer size: %s\n", 1177 db_strerror(ret)); 1178 goto err; 1179 } 1180 1181 /* 1182 * Specify the size of the in-memory cache. 1183 */ 1184 ret = envp->set_cachesize(envp, 0, 10 * 1024 * 1024, 1); 1185 if (ret != 0) { 1186 fprintf(stderr, "Error increasing the cache size: %s\n", 1187 db_strerror(ret)); 1188 goto err; 1189 } 1190 1191 /* 1192 * Now actually open the environment. Notice that the environment home 1193 * directory is NULL. This is required for an in-memory only 1194 * application. 1195 */ 1196 ret = envp->open(envp, NULL, open_flags, 0); 1197 if (ret != 0) { 1198 fprintf(stderr, "Error opening environment: %s\n", 1199 db_strerror(ret)); 1200 goto err; 1201 } 1202 1203 1204 /* Initialize the DB handle */ 1205 ret = db_create(&dbp, envp, 0); 1206 if (ret != 0) { 1207 envp->err(envp, ret, 1208 "Attempt to create db handle failed."); 1209 goto err; 1210 } 1211 1212 1213 /* 1214 * Set the database open flags. Autocommit is used because we are 1215 * transactional. 1216 */ 1217 open_flags = DB_CREATE | DB_AUTO_COMMIT; 1218 ret = dbp->open(dbp, /* Pointer to the database */ 1219 NULL, /* Txn pointer */ 1220 NULL, /* File name -- Must be NULL for inmemory! */ 1221 db_name, /* Logical db name */ 1222 DB_BTREE, /* Database type (using btree) */ 1223 open_flags, /* Open flags */ 1224 0); /* File mode. Using defaults */ 1225 1226 if (ret != 0) { 1227 envp->err(envp, ret, 1228 "Attempt to open db failed."); 1229 goto err; 1230 } 1231 1232 /* Configure the cache file */ 1233 mpf = dbp->get_mpf(dbp); 1234 ret = mpf->set_flags(mpf, DB_MPOOL_NOFILE, 1); 1235 1236 if (ret != 0) { 1237 envp->err(envp, ret, 1238 "Attempt failed to configure for no backing of temp files."); 1239 goto err; 1240 } 1241 1242err: 1243 /* Close our database handle, if it was opened. */ 1244 if (dbp != NULL) { 1245 ret_t = dbp->close(dbp, 0); 1246 if (ret_t != 0) { 1247 fprintf(stderr, "%s database close failed.\n", 1248 db_strerror(ret_t)); 1249 ret = ret_t; 1250 } 1251 } 1252 1253 /* Close our environment, if it was opened. */ 1254 if (envp != NULL) { 1255 ret_t = envp->close(envp, 0); 1256 if (ret_t != 0) { 1257 fprintf(stderr, "environment close failed: %s\n", 1258 db_strerror(ret_t)); 1259 ret = ret_t; 1260 } 1261 } 1262 1263 /* Final status message and return. */ 1264 printf("I'm all done.\n"); 1265 return (ret == 0 ? EXIT_SUCCESS : EXIT_FAILURE); 1266} </pre> 1267 </div> 1268 </div> 1269 <div class="navfooter"> 1270 <hr /> 1271 </div> 1272 </body> 1273</html> 1274