1<!-- $Id$ --> 2<html> 3<body> 4Utilities for managing class evolution of persistent objects. 5 6<h1>Class Evolution</h1> 7 8<p>For persistent data that is not short lived, changes to persistent classes 9are almost inevitable. Some changes are compatible with existing types, and 10data conversion for these changes is performed automatically and transparently. 11Other changes are not compatible with existing types. Mutations can be used to 12explicitly manage many types of incompatible changes.</p> 13 14<p>Not all incompatible class changes can be handled via mutations. For 15example, complex refactoring may require a transformation that manipulates 16multiple entity instances at once. Such changes are not possible with 17mutations but can be made by performing a <a href="#storeConversion">store 18conversion</a>.</p> 19 20<p>The different categories of type changes are described below.</p> 21 22<h2>Key Field Changes</h2> 23 24<p>Unlike entity data, key data is not versioned. Therefore, the physical key 25format for an index is fixed once the index has been opened, and the changes 26allowed for key fields are very limited. The only changes allowed for key 27fields are:</p> 28<ul> 29<li>The name of a key field may be changed, as long as this change is 30accompanied by a {@link com.sleepycat.persist.evolve.Renamer} mutation.</li> 31<li>A primitive type may be changed to its corresponding primitive wrapper 32type. This is a compatible change.</li> 33<li>For primary key fields and fields of a composite key class, a primitive 34wrapper type may be changed to its corresponding primitive type. This is 35allowed because these key fields with reference types may never have null 36values. This is a compatible change.</li> 37</ul> 38 39<p>Any other changes to a key field are incompatible and may be made only by 40performing a <a href="#storeConversion">store conversion</a>.</p> 41 42<p>Key ordering, including the behavior of a custom {@link 43java.lang.Comparable}, is also fixed, since keys are stored in order in the 44index. The specifications for key ordering may not be changed, and the 45developer is responsible for not changing the behavior of a {@code Comparable} 46key class. <strong>WARNING:</strong>: Changing the behavior of a {@code 47Comparable} key class is likely to make the index unusable.</p> 48 49<h2>Compatible Type Changes</h2> 50 51<p>Entity data, unlike key data, is versioned. Therefore, some changes can be 52made compatibly and other changes can be handled via mutations. Compatible 53changes are defined below. To make a compatible class change, a mutation is 54not required; however, the class version must be assigned a new (greater) 55integer value.</p> 56 57<p>Changes to a class hierarchy are compatible in some cases. A new class may 58be inserted in the hierarchy. A class may be deleted from the hierarchy as 59long as one of the following is true: 1) it contains no persistent fields, 2) 60any persistent fields are deleted with field Deleter mutations, or 3) the class 61is deleted with a class Deleter mutation. Classes in an existing hierarchy may 62not be reordered compatibly, and fields may not moved from one class to another 63compatibly; for such changes a class Converter mutation is required.</p> 64 65<p>Changes to field types in entity class definitions are compatible when they 66conform to the Java Language Specification definitions for <a 67href="http://java.sun.com/docs/books/jls/third_edition/html/conversions.html#5.1.2">Widening 68Primitive Conversions</a> and <a 69href="http://java.sun.com/docs/books/jls/third_edition/html/conversions.html#5.1.5">Widening 70Reference Conversions</a>. For example, a smaller integer 71type may be changed to a larger integer type, and a reference type may be 72changed to one of its supertypes. Automatic widening conversions are performed 73as described in the Java Language Specification.</p> 74 75<p>Primitive types may also be compatibly changed to their corresponding 76primitive wrapper types, or to the wrapper type for a widened primitive type. 77However, changing from a primitive wrapper type to a primitive type is not a 78compatible change since existing null values could not be represented.</p> 79 80<p>Integer primitive types (byte, short, char, int, long) and their primitive 81wrapper types may be compatibly changed to the BigInteger type.</p> 82 83<p>Enum values may be added compatibly, but may not be deleted or renamed. As 84long as new values are declared after existing values, the default sort order 85for enum key fields will match the declaration order, i.e, the default sort 86order will match the enum ordinal order. If a new value is inserted (declared 87before an existing value), it will be sorted after all existing values but 88before newly added values. However, these ordering rules are only guaranteed 89for enums containing up to 631 values and only if existing values are not 90reordered. If more than 631 values are declared or the declarations of 91existing values are reordered, then the default sort order will be arbitrary 92and will not match the declaration (ordinal) order.</p> 93 94<p>In addition, adding fields to a class is a compatible change. When a 95persistent instance of a class is read that does not contain the new field, the 96new field is initialized by the default constructor.</p> 97 98<p>All other changes to instance fields are considered incompatible. 99Incompatible changes may be handled via mutations, as described next.</p> 100 101<p>Note that whenever a class is changed, either compatibly or incompatibly, a 102new (higher) class version number must be assigned. See {@link 103com.sleepycat.persist.model.Entity#version} and {@link 104com.sleepycat.persist.model.Persistent#version} for information on assigning 105class version numbers.</p> 106 107<h2>Mutations</h2> 108 109<p>There are three types of mutations: {@link 110com.sleepycat.persist.evolve.Renamer}, {@link 111com.sleepycat.persist.evolve.Deleter} and {@link 112com.sleepycat.persist.evolve.Converter}.</p> 113 114<p>A class or field can be renamed using a {@link 115com.sleepycat.persist.evolve.Renamer}. Renaming is not expensive, since it 116does not involve conversion of instance data.</p> 117 118<p>A class or field can be deleted using a {@link 119com.sleepycat.persist.evolve.Deleter}.</p> 120<ul> 121<li>Deleting an entity class causes removal of the primary and secondary 122indices for the store, on other words, removal of all store entities for that 123class and its subclasses. Removal is performed when the store is opened. A 124{@link com.sleepycat.persist.evolve.Deleter} should be used for an entity class 125in all of the following circumstances: 126 <ul> 127 <li>When removing the entity class itself.</li> 128 <li>When removing {@link com.sleepycat.persist.model.Entity} from the class 129 to make it non-persistent.</li> 130 <li>When removing {@link com.sleepycat.persist.model.Entity} from the class 131 and adding {@link com.sleepycat.persist.model.Persistent}, to use it as an 132 embedded persistent class but not an entity class. The version of the class 133 must be incremented in this case.</li> 134 </ul> 135</li> 136 137<li>Deleting a non-entity class does not itself cause deletion of instance 138data, but is needed to inform DPL that the deleted class will not be used. 139Instances of the deleted class must be handled (discarded or converted to 140another class) by {@link com.sleepycat.persist.evolve.Deleter} or {@link 141com.sleepycat.persist.evolve.Converter} mutations for the field or enclosing 142class that contain embedded instances of the deleted class. A {@link 143com.sleepycat.persist.evolve.Deleter} should be used for a non-entity class in 144all of the following circumstances: 145 <ul> 146 <li>When removing the persistent class itself.</li> 147 <li>When removing {@link com.sleepycat.persist.model.Persistent} from the 148 class to make it non-persistent.</li> 149 <li>When removing {@link com.sleepycat.persist.model.Persistent} from the 150 class and adding {@link com.sleepycat.persist.model.Entity}, to use it as an 151 entity class but not an embedded persistent class. The version of the class 152 must be incremented in this case.</li> 153 </ul> 154</li> 155 156<li>Deleting a field causes automatic conversion of the instances containing 157that field, in order to discard the field values.</li> 158</ul> 159 160<p>Other incompatible changes are handled by creating a {@link 161com.sleepycat.persist.evolve.Converter} mutation and implementing a {@link 162com.sleepycat.persist.evolve.Conversion#convert Conversion.convert} method that 163manipulates the raw objects and/or simple values directly. The {@code convert} 164method is passed an object of the old incompatible type and it returns an 165object of a current type.</p> 166 167<p>Conversions can be specified in two ways: for specific fields or for all 168instances of a class. A different {@link 169com.sleepycat.persist.evolve.Converter} constructor is used in each case. 170Field-specific conversions are used instead of class conversions when both are 171applicable.</p> 172 173<p>Note that a class conversion may be not specified for an enum class. A 174field conversion, or a class conversion for the class declaring the field, may 175be used.</p> 176 177<p>Note that each mutation is applied to a specific class version number. The 178class version must be explicitly specified in a mutation for two reasons:</p> 179<ol> 180<li>This provides safety in the face of multiple unconverted versions of a 181given type. Without a version, a single conversion method would have to handle 182multiple input types, and would have to distinguish between them by examining 183the data or type information.</li> 184<li>This allows arbitrary changes to be made. For example, a series of name 185changes may reuse a given name for more than one version. To identify the 186specific type being converted or renamed, a version number is needed.</li> 187</ol> 188<p>See {@link com.sleepycat.persist.model.Entity#version} and {@link 189com.sleepycat.persist.model.Persistent#version} for information on assigning 190class version numbers.</p> 191 192<p>Mutations are therefore responsible for converting each existing 193incompatible class version to the current version as defined by a current class 194definition. For example, consider that class-version A-1 is initially changed 195to A-2 and a mutation is added for converting A-1 to A-2. If later changes in 196version A-3 occur before converting all A-1 instances to version A-2, the 197converter for A-1 will have to be changed. Instead of converting from A-1 to 198A-2 it will need to convert from A-1 to A-3. In addition, a mutation 199converting A-2 to A-3 will be needed.</p> 200 201<p>When a {@link com.sleepycat.persist.evolve.Converter} mutation applies to a 202given object, other mutations that may apply to that object are not 203automatically performed. It is the responsibility of the {@link 204com.sleepycat.persist.evolve.Converter} to return an object that conforms to 205the current class definition, including renaming fields and classes. If the 206input object has nested objects or superclasses that also need conversion, the 207converter must perform these nested conversions before returning the final 208converted object. This rule avoids the complexity and potential errors that 209could result if a converter mutation were automatically combined with other 210mutations in an arbitrary manner.</p> 211 212<p>The {@link com.sleepycat.persist.EntityStore#evolve EntityStore.evolve} 213method may optionally be used to ensure that all instances of an old class 214version are converted to the current version.</p> 215 216<h2>Other Metadata Changes</h2> 217 218<p>When a class that happens to be an entity class is renamed, it remains an 219entity class. When a field that happens to be a primary or 220secondary key field is renamed, its metadata remains intact as well.</p> 221 222<p>When the {@link com.sleepycat.persist.model.SecondaryKey} annotation is 223added to an <em>existing</em> field, a new index is created automatically. The 224new index will be populated by reading the entire primary index when the 225primary index is opened.</p> 226 227<p>When the {@link com.sleepycat.persist.model.SecondaryKey} annotation is 228included with a <em>new</em> field, a new index is created automatically. The 229new field is required to be a reference type (not a primitive) and must be 230initialized to null (the default behavior) in the default constructor. 231Entities will be indexed by the field when they are stored with a non-null key 232value.</p> 233 234<p>When a field with the {@link com.sleepycat.persist.model.SecondaryKey} 235annotation is deleted, or when the {@link 236com.sleepycat.persist.model.SecondaryKey} annotation is removed from a field 237without deleting it, the secondary index is removed (dropped). Removal occurs 238when the store is opened.</p> 239 240<p>The {@link com.sleepycat.persist.model.SecondaryKey#relate 241SecondaryKey.relate} property may NOT be changed. All other properties of a 242{@link com.sleepycat.persist.model.SecondaryKey} may be changed, although 243avoiding changes that cause foreign key integrity errors is the responsibility 244of the application developer. For example, if the {@link 245com.sleepycat.persist.model.SecondaryKey#relatedEntity} property is added but 246not all existing secondary keys reference existing primary keys for the related 247entity, foreign key integrity errors may occur.</p> 248 249<p>The {@link com.sleepycat.persist.model.PrimaryKey} annotation may NOT be 250removed from a field in an entity class.</p> 251 252<p>The {@link com.sleepycat.persist.model.PrimaryKey#sequence} property may be 253added, removed, or changed to a different name.</p> 254 255<p>The {@link com.sleepycat.persist.model.Persistent#proxyFor} property may NOT 256be added, removed, or changed to a different class.</p> 257 258<h2>Warnings on Testing and Backups</h2> 259 260<p>The application developer is responsible for verifying that class evolution 261works properly before deploying with a changed set of persistent classes. The 262DPL will report errors when old class definitions cannot be evolved, for 263example, when a mutation is missing. To test that no such errors will occur, 264application test cases must include instances of all persistent classes.</p> 265 266<p>Converter mutations require special testing. Since the application 267conversion method is allowed to return instances of any type, the DPL cannot 268check that the proper type is returned until the data is accessed. To avoid 269data access errors, application test cases must cover converter mutations for 270all potential input and output types.</p> 271 272<p>When secondary keys are dropped or entity classes are deleted, the 273underlying databases are deleted and cannot be recovered from the store. This 274takes place when the store is opened. It is strongly recommended that a backup 275of the entire store is made before opening the store and causing class 276evolution to proceed.</p> 277 278<h2><a name="storeConversion">Store Conversion<a/></h2> 279 280<p>When mutations are not sufficient for handling class changes, a full store 281conversion may be performed. This is necessary for two particular types of 282class changes:</p> 283<ul> 284<li>A change to a physical key format, for example, a change from type 285{@code int} to type {@code long}.</li> 286<li>A conversion that involves multiple entities at once, for example, 287combining two separate entity classes into a new single entity class.</li> 288</ul> 289 290<p>To perform a full store conversion, a program is written that performs the 291following steps to copy the data from the old store to a new converted 292store:</p> 293<ol> 294<li>The old store is opened as a {@link com.sleepycat.persist.raw.RawStore} and 295the new store is opened as an {@link com.sleepycat.persist.EntityStore}.</li> 296<li>All entities are read from the old store. Entities are read using a {@link 297com.sleepycat.persist.raw.RawStore} to allow access to entities for which no 298compatible class exists.</li> 299<li>The {@link com.sleepycat.persist.raw.RawObject} entities are then converted 300to the format desired. Raw objects can be arbitrarily manipulated as needed. 301The updated raw objects must conform to the new evolved class definitions.</li> 302<li>The updated raw entities are converted to live objects by calling the 303{@link com.sleepycat.persist.model.EntityModel#convertRawObject 304EntityModel.convertRawObject} method of the new store. This method converts 305raw objects obtained from a different store, as long as they conform to the new 306evolved class definitions.</li> 307<li>The new live objects are written to the new {@link 308com.sleepycat.persist.EntityStore} using a {@link 309com.sleepycat.persist.PrimaryIndex} as usual.</li> 310</ol> 311 312<p>To perform such a conversion, two separate stores must be open at once. 313Both stores may be in the same {@link com.sleepycat.db.Environment}, if 314desired, by giving them different store names. But since all data is being 315rewritten, there are performance advantages to creating the new store in a new 316fresh environment: the data will be compacted as it is written, and the old 317store can be removed very quickly by deleting the old environment directory 318after the conversion is complete.</p> 319 320</body> 321</html> 322