README.ENGINE revision 109998
168651Skris ENGINE 268651Skris ====== 368651Skris 4109998Smarkm With OpenSSL 0.9.6, a new component was added to support alternative 5109998Smarkm cryptography implementations, most commonly for interfacing with external 6109998Smarkm crypto devices (eg. accelerator cards). This component is called ENGINE, 7109998Smarkm and its presence in OpenSSL 0.9.6 (and subsequent bug-fix releases) 8109998Smarkm caused a little confusion as 0.9.6** releases were rolled in two 9109998Smarkm versions, a "standard" and an "engine" version. In development for 0.9.7, 10109998Smarkm the ENGINE code has been merged into the main branch and will be present 11109998Smarkm in the standard releases from 0.9.7 forwards. 1268651Skris 13109998Smarkm There are currently built-in ENGINE implementations for the following 14109998Smarkm crypto devices: 1568651Skris 1668651Skris o CryptoSwift 1768651Skris o Compaq Atalla 1868651Skris o nCipher CHIL 19109998Smarkm o Nuron 20109998Smarkm o Broadcom uBSec 2168651Skris 22109998Smarkm In addition, dynamic binding to external ENGINE implementations is now 23109998Smarkm provided by a special ENGINE called "dynamic". See the "DYNAMIC ENGINE" 24109998Smarkm section below for details. 2568651Skris 26109998Smarkm At this stage, a number of things are still needed and are being worked on: 2768651Skris 28109998Smarkm 1 Integration of EVP support. 29109998Smarkm 2 Configuration support. 30109998Smarkm 3 Documentation! 31109998Smarkm 32109998Smarkm1 With respect to EVP, this relates to support for ciphers and digests in 33109998Smarkm the ENGINE model so that alternative implementations of existing 34109998Smarkm algorithms/modes (or previously unimplemented ones) can be provided by 35109998Smarkm ENGINE implementations. 36109998Smarkm 37109998Smarkm2 Configuration support currently exists in the ENGINE API itself, in the 38109998Smarkm form of "control commands". These allow an application to expose to the 39109998Smarkm user/admin the set of commands and parameter types a given ENGINE 40109998Smarkm implementation supports, and for an application to directly feed string 41109998Smarkm based input to those ENGINEs, in the form of name-value pairs. This is an 42109998Smarkm extensible way for ENGINEs to define their own "configuration" mechanisms 43109998Smarkm that are specific to a given ENGINE (eg. for a particular hardware 44109998Smarkm device) but that should be consistent across *all* OpenSSL-based 45109998Smarkm applications when they use that ENGINE. Work is in progress (or at least 46109998Smarkm in planning) for supporting these control commands from the CONF (or 47109998Smarkm NCONF) code so that applications using OpenSSL's existing configuration 48109998Smarkm file format can have ENGINE settings specified in much the same way. 49109998Smarkm Presently however, applications must use the ENGINE API itself to provide 50109998Smarkm such functionality. To see first hand the types of commands available 51109998Smarkm with the various compiled-in ENGINEs (see further down for dynamic 52109998Smarkm ENGINEs), use the "engine" openssl utility with full verbosity, ie; 53109998Smarkm openssl engine -vvvv 54109998Smarkm 55109998Smarkm3 Documentation? Volunteers welcome! The source code is reasonably well 56109998Smarkm self-documenting, but some summaries and usage instructions are needed - 57109998Smarkm moreover, they are needed in the same POD format the existing OpenSSL 58109998Smarkm documentation is provided in. Any complete or incomplete contributions 59109998Smarkm would help make this happen. 60109998Smarkm 61109998Smarkm STABILITY & BUG-REPORTS 62109998Smarkm ======================= 63109998Smarkm 6468651Skris What already exists is fairly stable as far as it has been tested, but 65109998Smarkm the test base has been a bit small most of the time. For the most part, 66109998Smarkm the vendors of the devices these ENGINEs support have contributed to the 67109998Smarkm development and/or testing of the implementations, and *usually* (with no 68109998Smarkm guarantees) have experience in using the ENGINE support to drive their 69109998Smarkm devices from common OpenSSL-based applications. Bugs and/or inexplicable 70109998Smarkm behaviour in using a specific ENGINE implementation should be sent to the 71109998Smarkm author of that implementation (if it is mentioned in the corresponding C 72109998Smarkm file), and in the case of implementations for commercial hardware 73109998Smarkm devices, also through whatever vendor support channels are available. If 74109998Smarkm none of this is possible, or the problem seems to be something about the 75109998Smarkm ENGINE API itself (ie. not necessarily specific to a particular ENGINE 76109998Smarkm implementation) then you should mail complete details to the relevant 77109998Smarkm OpenSSL mailing list. For a definition of "complete details", refer to 78109998Smarkm the OpenSSL "README" file. As for which list to send it to; 7968651Skris 80109998Smarkm openssl-users: if you are *using* the ENGINE abstraction, either in an 81109998Smarkm pre-compiled application or in your own application code. 8268651Skris 83109998Smarkm openssl-dev: if you are discussing problems with OpenSSL source code. 8468651Skris 85109998Smarkm USAGE 8668651Skris ===== 8768651Skris 88109998Smarkm The default "openssl" ENGINE is always chosen when performing crypto 89109998Smarkm operations unless you specify otherwise. You must actively tell the 90109998Smarkm openssl utility commands to use anything else through a new command line 91109998Smarkm switch called "-engine". Also, if you want to use the ENGINE support in 92109998Smarkm your own code to do something similar, you must likewise explicitly 93109998Smarkm select the ENGINE implementation you want. 9468651Skris 95109998Smarkm Depending on the type of hardware, system, and configuration, "settings" 96109998Smarkm may need to be applied to an ENGINE for it to function as expected/hoped. 97109998Smarkm The recommended way of doing this is for the application to support 98109998Smarkm ENGINE "control commands" so that each ENGINE implementation can provide 99109998Smarkm whatever configuration primitives it might require and the application 100109998Smarkm can allow the user/admin (and thus the hardware vendor's support desk 101109998Smarkm also) to provide any such input directly to the ENGINE implementation. 102109998Smarkm This way, applications do not need to know anything specific to any 103109998Smarkm device, they only need to provide the means to carry such user/admin 104109998Smarkm input through to the ENGINE in question. Ie. this connects *you* (and 105109998Smarkm your helpdesk) to the specific ENGINE implementation (and device), and 106109998Smarkm allows application authors to not get buried in hassle supporting 107109998Smarkm arbitrary devices they know (and care) nothing about. 10868651Skris 109109998Smarkm A new "openssl" utility, "openssl engine", has been added in that allows 110109998Smarkm for testing and examination of ENGINE implementations. Basic usage 111109998Smarkm instructions are available by specifying the "-?" command line switch. 11268651Skris 113109998Smarkm DYNAMIC ENGINES 114109998Smarkm =============== 11568651Skris 116109998Smarkm The new "dynamic" ENGINE provides a low-overhead way to support ENGINE 117109998Smarkm implementations that aren't pre-compiled and linked into OpenSSL-based 118109998Smarkm applications. This could be because existing compiled-in implementations 119109998Smarkm have known problems and you wish to use a newer version with an existing 120109998Smarkm application. It could equally be because the application (or OpenSSL 121109998Smarkm library) you are using simply doesn't have support for the ENGINE you 122109998Smarkm wish to use, and the ENGINE provider (eg. hardware vendor) is providing 123109998Smarkm you with a self-contained implementation in the form of a shared-library. 124109998Smarkm The other use-case for "dynamic" is with applications that wish to 125109998Smarkm maintain the smallest foot-print possible and so do not link in various 126109998Smarkm ENGINE implementations from OpenSSL, but instead leaves you to provide 127109998Smarkm them, if you want them, in the form of "dynamic"-loadable 128109998Smarkm shared-libraries. It should be possible for hardware vendors to provide 129109998Smarkm their own shared-libraries to support arbitrary hardware to work with 130109998Smarkm applications based on OpenSSL 0.9.7 or later. If you're using an 131109998Smarkm application based on 0.9.7 (or later) and the support you desire is only 132109998Smarkm announced for versions later than the one you need, ask the vendor to 133109998Smarkm backport their ENGINE to the version you need. 134109998Smarkm 135109998Smarkm How does "dynamic" work? 136109998Smarkm ------------------------ 137109998Smarkm The dynamic ENGINE has a special flag in its implementation such that 138109998Smarkm every time application code asks for the 'dynamic' ENGINE, it in fact 139109998Smarkm gets its own copy of it. As such, multi-threaded code (or code that 140109998Smarkm multiplexes multiple uses of 'dynamic' in a single application in any 141109998Smarkm way at all) does not get confused by 'dynamic' being used to do many 142109998Smarkm independent things. Other ENGINEs typically don't do this so there is 143109998Smarkm only ever 1 ENGINE structure of its type (and reference counts are used 144109998Smarkm to keep order). The dynamic ENGINE itself provides absolutely no 145109998Smarkm cryptographic functionality, and any attempt to "initialise" the ENGINE 146109998Smarkm automatically fails. All it does provide are a few "control commands" 147109998Smarkm that can be used to control how it will load an external ENGINE 148109998Smarkm implementation from a shared-library. To see these control commands, 149109998Smarkm use the command-line; 150109998Smarkm 151109998Smarkm openssl engine -vvvv dynamic 152109998Smarkm 153109998Smarkm The "SO_PATH" control command should be used to identify the 154109998Smarkm shared-library that contains the ENGINE implementation, and "NO_VCHECK" 155109998Smarkm might possibly be useful if there is a minor version conflict and you 156109998Smarkm (or a vendor helpdesk) is convinced you can safely ignore it. 157109998Smarkm "ID" is probably only needed if a shared-library implements 158109998Smarkm multiple ENGINEs, but if you know the engine id you expect to be using, 159109998Smarkm it doesn't hurt to specify it (and this provides a sanity check if 160109998Smarkm nothing else). "LIST_ADD" is only required if you actually wish the 161109998Smarkm loaded ENGINE to be discoverable by application code later on using the 162109998Smarkm ENGINE's "id". For most applications, this isn't necessary - but some 163109998Smarkm application authors may have nifty reasons for using it. The "LOAD" 164109998Smarkm command is the only one that takes no parameters and is the command 165109998Smarkm that uses the settings from any previous commands to actually *load* 166109998Smarkm the shared-library ENGINE implementation. If this command succeeds, the 167109998Smarkm (copy of the) 'dynamic' ENGINE will magically morph into the ENGINE 168109998Smarkm that has been loaded from the shared-library. As such, any control 169109998Smarkm commands supported by the loaded ENGINE could then be executed as per 170109998Smarkm normal. Eg. if ENGINE "foo" is implemented in the shared-library 171109998Smarkm "libfoo.so" and it supports some special control command "CMD_FOO", the 172109998Smarkm following code would load and use it (NB: obviously this code has no 173109998Smarkm error checking); 174109998Smarkm 175109998Smarkm ENGINE *e = ENGINE_by_id("dynamic"); 176109998Smarkm ENGINE_ctrl_cmd_string(e, "SO_PATH", "/lib/libfoo.so", 0); 177109998Smarkm ENGINE_ctrl_cmd_string(e, "ID", "foo", 0); 178109998Smarkm ENGINE_ctrl_cmd_string(e, "LOAD", NULL, 0); 179109998Smarkm ENGINE_ctrl_cmd_string(e, "CMD_FOO", "some input data", 0); 180109998Smarkm 181109998Smarkm For testing, the "openssl engine" utility can be useful for this sort 182109998Smarkm of thing. For example the above code excerpt would achieve much the 183109998Smarkm same result as; 184109998Smarkm 185109998Smarkm openssl engine dynamic \ 186109998Smarkm -pre SO_PATH:/lib/libfoo.so \ 187109998Smarkm -pre ID:foo \ 188109998Smarkm -pre LOAD \ 189109998Smarkm -pre "CMD_FOO:some input data" 190109998Smarkm 191109998Smarkm Or to simply see the list of commands supported by the "foo" ENGINE; 192109998Smarkm 193109998Smarkm openssl engine -vvvv dynamic \ 194109998Smarkm -pre SO_PATH:/lib/libfoo.so \ 195109998Smarkm -pre ID:foo \ 196109998Smarkm -pre LOAD 197109998Smarkm 198109998Smarkm Applications that support the ENGINE API and more specifically, the 199109998Smarkm "control commands" mechanism, will provide some way for you to pass 200109998Smarkm such commands through to ENGINEs. As such, you would select "dynamic" 201109998Smarkm as the ENGINE to use, and the parameters/commands you pass would 202109998Smarkm control the *actual* ENGINE used. Each command is actually a name-value 203109998Smarkm pair and the value can sometimes be omitted (eg. the "LOAD" command). 204109998Smarkm Whilst the syntax demonstrated in "openssl engine" uses a colon to 205109998Smarkm separate the command name from the value, applications may provide 206109998Smarkm their own syntax for making that separation (eg. a win32 registry 207109998Smarkm key-value pair may be used by some applications). The reason for the 208109998Smarkm "-pre" syntax in the "openssl engine" utility is that some commands 209109998Smarkm might be issued to an ENGINE *after* it has been initialised for use. 210109998Smarkm Eg. if an ENGINE implementation requires a smart-card to be inserted 211109998Smarkm during initialisation (or a PIN to be typed, or whatever), there may be 212109998Smarkm a control command you can issue afterwards to "forget" the smart-card 213109998Smarkm so that additional initialisation is no longer possible. In 214109998Smarkm applications such as web-servers, where potentially volatile code may 215109998Smarkm run on the same host system, this may provide some arguable security 216109998Smarkm value. In such a case, the command would be passed to the ENGINE after 217109998Smarkm it has been initialised for use, and so the "-post" switch would be 218109998Smarkm used instead. Applications may provide a different syntax for 219109998Smarkm supporting this distinction, and some may simply not provide it at all 220109998Smarkm ("-pre" is almost always what you're after, in reality). 221109998Smarkm 222109998Smarkm How do I build a "dynamic" ENGINE? 223109998Smarkm ---------------------------------- 224109998Smarkm This question is trickier - currently OpenSSL bundles various ENGINE 225109998Smarkm implementations that are statically built in, and any application that 226109998Smarkm calls the "ENGINE_load_builtin_engines()" function will automatically 227109998Smarkm have all such ENGINEs available (and occupying memory). Applications 228109998Smarkm that don't call that function have no ENGINEs available like that and 229109998Smarkm would have to use "dynamic" to load any such ENGINE - but on the other 230109998Smarkm hand such applications would only have the memory footprint of any 231109998Smarkm ENGINEs explicitly loaded using user/admin provided control commands. 232109998Smarkm The main advantage of not statically linking ENGINEs and only using 233109998Smarkm "dynamic" for hardware support is that any installation using no 234109998Smarkm "external" ENGINE suffers no unnecessary memory footprint from unused 235109998Smarkm ENGINEs. Likewise, installations that do require an ENGINE incur the 236109998Smarkm overheads from only *that* ENGINE once it has been loaded. 237109998Smarkm 238109998Smarkm Sounds good? Maybe, but currently building an ENGINE implementation as 239109998Smarkm a shared-library that can be loaded by "dynamic" isn't automated in 240109998Smarkm OpenSSL's build process. It can be done manually quite easily however. 241109998Smarkm Such a shared-library can either be built with any OpenSSL code it 242109998Smarkm needs statically linked in, or it can link dynamically against OpenSSL 243109998Smarkm if OpenSSL itself is built as a shared library. The instructions are 244109998Smarkm the same in each case, but in the former (statically linked any 245109998Smarkm dependencies on OpenSSL) you must ensure OpenSSL is built with 246109998Smarkm position-independent code ("PIC"). The default OpenSSL compilation may 247109998Smarkm already specify the relevant flags to do this, but you should consult 248109998Smarkm with your compiler documentation if you are in any doubt. 249109998Smarkm 250109998Smarkm This example will show building the "atalla" ENGINE in the 251109998Smarkm crypto/engine/ directory as a shared-library for use via the "dynamic" 252109998Smarkm ENGINE. 253109998Smarkm 1) "cd" to the crypto/engine/ directory of a pre-compiled OpenSSL 254109998Smarkm source tree. 255109998Smarkm 2) Recompile at least one source file so you can see all the compiler 256109998Smarkm flags (and syntax) being used to build normally. Eg; 257109998Smarkm touch hw_atalla.c ; make 258109998Smarkm will rebuild "hw_atalla.o" using all such flags. 259109998Smarkm 3) Manually enter the same compilation line to compile the 260109998Smarkm "hw_atalla.c" file but with the following two changes; 261109998Smarkm (a) add "-DENGINE_DYNAMIC_SUPPORT" to the command line switches, 262109998Smarkm (b) change the output file from "hw_atalla.o" to something new, 263109998Smarkm eg. "tmp_atalla.o" 264109998Smarkm 4) Link "tmp_atalla.o" into a shared-library using the top-level 265109998Smarkm OpenSSL libraries to resolve any dependencies. The syntax for doing 266109998Smarkm this depends heavily on your system/compiler and is a nightmare 267109998Smarkm known well to anyone who has worked with shared-library portability 268109998Smarkm before. 'gcc' on Linux, for example, would use the following syntax; 269109998Smarkm gcc -shared -o dyn_atalla.so tmp_atalla.o -L../.. -lcrypto 270109998Smarkm 5) Test your shared library using "openssl engine" as explained in the 271109998Smarkm previous section. Eg. from the top-level directory, you might try; 272109998Smarkm apps/openssl engine -vvvv dynamic \ 273109998Smarkm -pre SO_PATH:./crypto/engine/dyn_atalla.so -pre LOAD 274109998Smarkm If the shared-library loads successfully, you will see both "-pre" 275109998Smarkm commands marked as "SUCCESS" and the list of control commands 276109998Smarkm displayed (because of "-vvvv") will be the control commands for the 277109998Smarkm *atalla* ENGINE (ie. *not* the 'dynamic' ENGINE). You can also add 278109998Smarkm the "-t" switch to the utility if you want it to try and initialise 279109998Smarkm the atalla ENGINE for use to test any possible hardware/driver 280109998Smarkm issues. 281109998Smarkm 28268651Skris PROBLEMS 28368651Skris ======== 28468651Skris 285109998Smarkm It seems like the ENGINE part doesn't work too well with CryptoSwift on Win32. 286109998Smarkm A quick test done right before the release showed that trying "openssl speed 287109998Smarkm -engine cswift" generated errors. If the DSO gets enabled, an attempt is made 288109998Smarkm to write at memory address 0x00000002. 28968651Skris 290