facets.html revision 1.1.1.2
1<?xml version="1.0" encoding="UTF-8" standalone="no"?> 2<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>Facets</title><meta name="generator" content="DocBook XSL-NS Stylesheets V1.78.1" /><meta name="keywords" content="ISO C++, library" /><meta name="keywords" content="ISO C++, runtime, library" /><link rel="home" href="../index.html" title="The GNU C++ Library" /><link rel="up" href="localization.html" title="Chapter��8.�� Localization" /><link rel="prev" href="localization.html" title="Chapter��8.�� Localization" /><link rel="next" href="containers.html" title="Chapter��9.�� Containers" /></head><body><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Facets</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="localization.html">Prev</a>��</td><th width="60%" align="center">Chapter��8.�� 3 Localization 4 5</th><td width="20%" align="right">��<a accesskey="n" href="containers.html">Next</a></td></tr></table><hr /></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="std.localization.facet"></a>Facets</h2></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="std.localization.facet.ctype"></a>ctype</h3></div></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.ctype.impl"></a>Implementation</h4></div></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="idm269999753024"></a>Specializations</h5></div></div></div><p> 6For the required specialization codecvt<wchar_t, char, mbstate_t> , 7conversions are made between the internal character set (always UCS4 8on GNU/Linux) and whatever the currently selected locale for the 9LC_CTYPE category implements. 10</p><p> 11The two required specializations are implemented as follows: 12</p><p> 13<code class="code"> 14ctype<char> 15</code> 16</p><p> 17This is simple specialization. Implementing this was a piece of cake. 18</p><p> 19<code class="code"> 20ctype<wchar_t> 21</code> 22</p><p> 23This specialization, by specifying all the template parameters, pretty 24much ties the hands of implementors. As such, the implementation is 25straightforward, involving mcsrtombs for the conversions between char 26to wchar_t and wcsrtombs for conversions between wchar_t and char. 27</p><p> 28Neither of these two required specializations deals with Unicode 29characters. 30</p></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.ctype.future"></a>Future</h4></div></div></div><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p> 31 How to deal with the global locale issue? 32 </p></li><li class="listitem"><p> 33 How to deal with different types than char, wchar_t? </p></li><li class="listitem"><p> 34 Overlap between codecvt/ctype: narrow/widen 35 </p></li><li class="listitem"><p> 36 Mask typedef in codecvt_base, argument types in codecvt. what 37 is know about this type? 38 </p></li><li class="listitem"><p> 39 Why mask* argument in codecvt? 40 </p></li><li class="listitem"><p> 41 Can this be made (more) generic? is there a simple way to 42 straighten out the configure-time mess that is a by-product of 43 this class? 44 </p></li><li class="listitem"><p> 45 Get the ctype<wchar_t>::mask stuff under control. Need to 46 make some kind of static table, and not do lookup every time 47 somebody hits the do_is... functions. Too bad we can't just 48 redefine mask for ctype<wchar_t> 49 </p></li><li class="listitem"><p> 50 Rename abstract base class. See if just smash-overriding is a 51 better approach. Clarify, add sanity to naming. 52 </p></li></ul></div></div><div class="bibliography"><div class="titlepage"><div><div><h4 class="title"><a id="facet.ctype.biblio"></a>Bibliography</h4></div></div></div><div class="biblioentry"><a id="idm269999738032"></a><p><span class="citetitle"><em class="citetitle"> 53 The GNU C Library 54 </em>. </span><span class="author"><span class="firstname">Roland</span> <span class="surname">McGrath</span>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright �� 2007 FSF. </span><span class="pagenums">Chapters 6 Character Set Handling and 7 Locales and Internationalization. </span></p></div><div class="biblioentry"><a id="idm269999733280"></a><p><span class="citetitle"><em class="citetitle"> 55 Correspondence 56 </em>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright �� 2002 . </span></p></div><div class="biblioentry"><a id="idm269999730192"></a><p><span class="citetitle"><em class="citetitle"> 57 ISO/IEC 14882:1998 Programming languages - C++ 58 </em>. </span><span class="copyright">Copyright �� 1998 ISO. </span></p></div><div class="biblioentry"><a id="idm269999727904"></a><p><span class="citetitle"><em class="citetitle"> 59 ISO/IEC 9899:1999 Programming languages - C 60 </em>. </span><span class="copyright">Copyright �� 1999 ISO. </span></p></div><div class="biblioentry"><a id="idm269999725632"></a><p><span class="title"><em> 61 <a class="link" href="http://www.unix.org/version3/ieee_std.html" target="_top"> 62 The Open Group Base Specifications, Issue 6 (IEEE Std. 1003.1-2004) 63 </a> 64 </em>. </span><span class="copyright">Copyright �� 1999 65 The Open Group/The Institute of Electrical and Electronics Engineers, Inc.. </span></p></div><div class="biblioentry"><a id="idm269999722384"></a><p><span class="citetitle"><em class="citetitle"> 66 The C++ Programming Language, Special Edition 67 </em>. </span><span class="author"><span class="firstname">Bjarne</span> <span class="surname">Stroustrup</span>. </span><span class="copyright">Copyright �� 2000 Addison Wesley, Inc.. </span><span class="pagenums">Appendix D. </span><span class="publisher"><span class="publishername"> 68 Addison Wesley 69 . </span></span></p></div><div class="biblioentry"><a id="idm269999717760"></a><p><span class="citetitle"><em class="citetitle"> 70 Standard C++ IOStreams and Locales 71 </em>. </span><span class="subtitle"> 72 Advanced Programmer's Guide and Reference 73 . </span><span class="author"><span class="firstname">Angelika</span> <span class="surname">Langer</span>. </span><span class="author"><span class="firstname">Klaus</span> <span class="surname">Kreft</span>. </span><span class="copyright">Copyright �� 2000 Addison Wesley Longman, Inc.. </span><span class="publisher"><span class="publishername"> 74 Addison Wesley Longman 75 . </span></span></p></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="std.localization.facet.codecvt"></a>codecvt</h3></div></div></div><p> 76The standard class codecvt attempts to address conversions between 77different character encoding schemes. In particular, the standard 78attempts to detail conversions between the implementation-defined wide 79characters (hereafter referred to as wchar_t) and the standard type 80char that is so beloved in classic <span class="quote">���<span class="quote">C</span>���</span> (which can now be 81referred to as narrow characters.) This document attempts to describe 82how the GNU libstdc++ implementation deals with the conversion between 83wide and narrow characters, and also presents a framework for dealing 84with the huge number of other encodings that iconv can convert, 85including Unicode and UTF8. Design issues and requirements are 86addressed, and examples of correct usage for both the required 87specializations for wide and narrow characters and the 88implementation-provided extended functionality are given. 89</p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.req"></a>Requirements</h4></div></div></div><p> 90Around page 425 of the C++ Standard, this charming heading comes into view: 91</p><div class="blockquote"><blockquote class="blockquote"><p> 9222.2.1.5 - Template class codecvt 93</p></blockquote></div><p> 94The text around the codecvt definition gives some clues: 95</p><div class="blockquote"><blockquote class="blockquote"><p> 96<span class="emphasis"><em> 97-1- The class codecvt<internT,externT,stateT> is for use when 98converting from one codeset to another, such as from wide characters 99to multibyte characters, between wide character encodings such as 100Unicode and EUC. 101</em></span> 102</p></blockquote></div><p> 103Hmm. So, in some unspecified way, Unicode encodings and 104translations between other character sets should be handled by this 105class. 106</p><div class="blockquote"><blockquote class="blockquote"><p> 107<span class="emphasis"><em> 108-2- The stateT argument selects the pair of codesets being mapped between. 109</em></span> 110</p></blockquote></div><p> 111Ah ha! Another clue... 112</p><div class="blockquote"><blockquote class="blockquote"><p> 113<span class="emphasis"><em> 114-3- The instantiations required in the Table ?? 115(lib.locale.category), namely codecvt<wchar_t,char,mbstate_t> and 116codecvt<char,char,mbstate_t>, convert the implementation-defined 117native character set. codecvt<char,char,mbstate_t> implements a 118degenerate conversion; it does not convert at 119all. codecvt<wchar_t,char,mbstate_t> converts between the native 120character sets for tiny and wide characters. Instantiations on 121mbstate_t perform conversion between encodings known to the library 122implementor. Other encodings can be converted by specializing on a 123user-defined stateT type. The stateT object can contain any state that 124is useful to communicate to or from the specialized do_convert member. 125</em></span> 126</p></blockquote></div><p> 127At this point, a couple points become clear: 128</p><p> 129One: The standard clearly implies that attempts to add non-required 130(yet useful and widely used) conversions need to do so through the 131third template parameter, stateT.</p><p> 132Two: The required conversions, by specifying mbstate_t as the third 133template parameter, imply an implementation strategy that is mostly 134(or wholly) based on the underlying C library, and the functions 135mcsrtombs and wcsrtombs in particular.</p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.design"></a>Design</h4></div></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="codecvt.design.wchar_t_size"></a><span class="type">wchar_t</span> Size</h5></div></div></div><p> 136 The simple implementation detail of wchar_t's size seems to 137 repeatedly confound people. Many systems use a two byte, 138 unsigned integral type to represent wide characters, and use an 139 internal encoding of Unicode or UCS2. (See AIX, Microsoft NT, 140 Java, others.) Other systems, use a four byte, unsigned integral 141 type to represent wide characters, and use an internal encoding 142 of UCS4. (GNU/Linux systems using glibc, in particular.) The C 143 programming language (and thus C++) does not specify a specific 144 size for the type wchar_t. 145 </p><p> 146 Thus, portable C++ code cannot assume a byte size (or endianness) either. 147 </p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="codecvt.design.unicode"></a>Support for Unicode</h5></div></div></div><p> 148 Probably the most frequently asked question about code conversion 149 is: "So dudes, what's the deal with Unicode strings?" 150 The dude part is optional, but apparently the usefulness of 151 Unicode strings is pretty widely appreciated. Sadly, this specific 152 encoding (And other useful encodings like UTF8, UCS4, ISO 8859-10, 153 etc etc etc) are not mentioned in the C++ standard. 154 </p><p> 155 A couple of comments: 156 </p><p> 157 The thought that all one needs to convert between two arbitrary 158 codesets is two types and some kind of state argument is 159 unfortunate. In particular, encodings may be stateless. The naming 160 of the third parameter as stateT is unfortunate, as what is really 161 needed is some kind of generalized type that accounts for the 162 issues that abstract encodings will need. The minimum information 163 that is required includes: 164 </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p> 165 Identifiers for each of the codesets involved in the 166 conversion. For example, using the iconv family of functions 167 from the Single Unix Specification (what used to be called 168 X/Open) hosted on the GNU/Linux operating system allows 169 bi-directional mapping between far more than the following 170 tantalizing possibilities: 171 </p><p> 172 (An edited list taken from <code class="code">`iconv --list`</code> on a 173 Red Hat 6.2/Intel system: 174 </p><div class="blockquote"><blockquote class="blockquote"><pre class="programlisting"> 1758859_1, 8859_9, 10646-1:1993, 10646-1:1993/UCS4, ARABIC, ARABIC7, 176ASCII, EUC-CN, EUC-JP, EUC-KR, EUC-TW, GREEK-CCIcode, GREEK, GREEK7-OLD, 177GREEK7, GREEK8, HEBREW, ISO-8859-1, ISO-8859-2, ISO-8859-3, 178ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, 179ISO-8859-9, ISO-8859-10, ISO-8859-11, ISO-8859-13, ISO-8859-14, 180ISO-8859-15, ISO-10646, ISO-10646/UCS2, ISO-10646/UCS4, 181ISO-10646/UTF-8, ISO-10646/UTF8, SHIFT-JIS, SHIFT_JIS, UCS-2, UCS-4, 182UCS2, UCS4, UNICODE, UNICODEBIG, UNICODELIcodeLE, US-ASCII, US, UTF-8, 183UTF-16, UTF8, UTF16). 184</pre></blockquote></div><p> 185For iconv-based implementations, string literals for each of the 186encodings (i.e. "UCS-2" and "UTF-8") are necessary, 187although for other, 188non-iconv implementations a table of enumerated values or some other 189mechanism may be required. 190</p></li><li class="listitem"><p> 191 Maximum length of the identifying string literal. 192</p></li><li class="listitem"><p> 193 Some encodings require explicit endian-ness. As such, some kind 194 of endian marker or other byte-order marker will be necessary. See 195 "Footnotes for C/C++ developers" in Haible for more information on 196 UCS-2/Unicode endian issues. (Summary: big endian seems most likely, 197 however implementations, most notably Microsoft, vary.) 198</p></li><li class="listitem"><p> 199 Types representing the conversion state, for conversions involving 200 the machinery in the "C" library, or the conversion descriptor, for 201 conversions using iconv (such as the type iconv_t.) Note that the 202 conversion descriptor encodes more information than a simple encoding 203 state type. 204</p></li><li class="listitem"><p> 205 Conversion descriptors for both directions of encoding. (i.e., both 206 UCS-2 to UTF-8 and UTF-8 to UCS-2.) 207</p></li><li class="listitem"><p> 208 Something to indicate if the conversion requested if valid. 209</p></li><li class="listitem"><p> 210 Something to represent if the conversion descriptors are valid. 211</p></li><li class="listitem"><p> 212 Some way to enforce strict type checking on the internal and 213 external types. As part of this, the size of the internal and 214 external types will need to be known. 215</p></li></ul></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="codecvt.design.issues"></a>Other Issues</h5></div></div></div><p> 216In addition, multi-threaded and multi-locale environments also impact 217the design and requirements for code conversions. In particular, they 218affect the required specialization codecvt<wchar_t, char, mbstate_t> 219when implemented using standard "C" functions. 220</p><p> 221Three problems arise, one big, one of medium importance, and one small. 222</p><p> 223First, the small: mcsrtombs and wcsrtombs may not be multithread-safe 224on all systems required by the GNU tools. For GNU/Linux and glibc, 225this is not an issue. 226</p><p> 227Of medium concern, in the grand scope of things, is that the functions 228used to implement this specialization work on null-terminated 229strings. Buffers, especially file buffers, may not be null-terminated, 230thus giving conversions that end prematurely or are otherwise 231incorrect. Yikes! 232</p><p> 233The last, and fundamental problem, is the assumption of a global 234locale for all the "C" functions referenced above. For something like 235C++ iostreams (where codecvt is explicitly used) the notion of 236multiple locales is fundamental. In practice, most users may not run 237into this limitation. However, as a quality of implementation issue, 238the GNU C++ library would like to offer a solution that allows 239multiple locales and or simultaneous usage with computationally 240correct results. In short, libstdc++ is trying to offer, as an 241option, a high-quality implementation, damn the additional complexity! 242</p><p> 243For the required specialization codecvt<wchar_t, char, mbstate_t> , 244conversions are made between the internal character set (always UCS4 245on GNU/Linux) and whatever the currently selected locale for the 246LC_CTYPE category implements. 247</p></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.impl"></a>Implementation</h4></div></div></div><p> 248The two required specializations are implemented as follows: 249</p><p> 250<code class="code"> 251codecvt<char, char, mbstate_t> 252</code> 253</p><p> 254This is a degenerate (i.e., does nothing) specialization. Implementing 255this was a piece of cake. 256</p><p> 257<code class="code"> 258codecvt<char, wchar_t, mbstate_t> 259</code> 260</p><p> 261This specialization, by specifying all the template parameters, pretty 262much ties the hands of implementors. As such, the implementation is 263straightforward, involving mcsrtombs for the conversions between char 264to wchar_t and wcsrtombs for conversions between wchar_t and char. 265</p><p> 266Neither of these two required specializations deals with Unicode 267characters. As such, libstdc++ implements a partial specialization 268of the codecvt class with and iconv wrapper class, encoding_state as the 269third template parameter. 270</p><p> 271This implementation should be standards conformant. First of all, the 272standard explicitly points out that instantiations on the third 273template parameter, stateT, are the proper way to implement 274non-required conversions. Second of all, the standard says (in Chapter 27517) that partial specializations of required classes are a-ok. Third 276of all, the requirements for the stateT type elsewhere in the standard 277(see 21.1.2 traits typedefs) only indicate that this type be copy 278constructible. 279</p><p> 280As such, the type encoding_state is defined as a non-templatized, POD 281type to be used as the third type of a codecvt instantiation. This 282type is just a wrapper class for iconv, and provides an easy interface 283to iconv functionality. 284</p><p> 285There are two constructors for encoding_state: 286</p><p> 287<code class="code"> 288encoding_state() : __in_desc(0), __out_desc(0) 289</code> 290</p><p> 291This default constructor sets the internal encoding to some default 292(currently UCS4) and the external encoding to whatever is returned by 293nl_langinfo(CODESET). 294</p><p> 295<code class="code"> 296encoding_state(const char* __int, const char* __ext) 297</code> 298</p><p> 299This constructor takes as parameters string literals that indicate the 300desired internal and external encoding. There are no defaults for 301either argument. 302</p><p> 303One of the issues with iconv is that the string literals identifying 304conversions are not standardized. Because of this, the thought of 305mandating and or enforcing some set of pre-determined valid 306identifiers seems iffy: thus, a more practical (and non-migraine 307inducing) strategy was implemented: end-users can specify any string 308(subject to a pre-determined length qualifier, currently 32 bytes) for 309encodings. It is up to the user to make sure that these strings are 310valid on the target system. 311</p><p> 312<code class="code"> 313void 314_M_init() 315</code> 316</p><p> 317Strangely enough, this member function attempts to open conversion 318descriptors for a given encoding_state object. If the conversion 319descriptors are not valid, the conversion descriptors returned will 320not be valid and the resulting calls to the codecvt conversion 321functions will return error. 322</p><p> 323<code class="code"> 324bool 325_M_good() 326</code> 327</p><p> 328Provides a way to see if the given encoding_state object has been 329properly initialized. If the string literals describing the desired 330internal and external encoding are not valid, initialization will 331fail, and this will return false. If the internal and external 332encodings are valid, but iconv_open could not allocate conversion 333descriptors, this will also return false. Otherwise, the object is 334ready to convert and will return true. 335</p><p> 336<code class="code"> 337encoding_state(const encoding_state&) 338</code> 339</p><p> 340As iconv allocates memory and sets up conversion descriptors, the copy 341constructor can only copy the member data pertaining to the internal 342and external code conversions, and not the conversion descriptors 343themselves. 344</p><p> 345Definitions for all the required codecvt member functions are provided 346for this specialization, and usage of codecvt<internal character type, 347external character type, encoding_state> is consistent with other 348codecvt usage. 349</p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.use"></a>Use</h4></div></div></div><p>A conversions involving string literal.</p><pre class="programlisting"> 350 typedef codecvt_base::result result; 351 typedef unsigned short unicode_t; 352 typedef unicode_t int_type; 353 typedef char ext_type; 354 typedef encoding_state state_type; 355 typedef codecvt<int_type, ext_type, state_type> unicode_codecvt; 356 357 const ext_type* e_lit = "black pearl jasmine tea"; 358 int size = strlen(e_lit); 359 int_type i_lit_base[24] = 360 { 25088, 27648, 24832, 25344, 27392, 8192, 28672, 25856, 24832, 29184, 361 27648, 8192, 27136, 24832, 29440, 27904, 26880, 28160, 25856, 8192, 29696, 362 25856, 24832, 2560 363 }; 364 const int_type* i_lit = i_lit_base; 365 const ext_type* efrom_next; 366 const int_type* ifrom_next; 367 ext_type* e_arr = new ext_type[size + 1]; 368 ext_type* eto_next; 369 int_type* i_arr = new int_type[size + 1]; 370 int_type* ito_next; 371 372 // construct a locale object with the specialized facet. 373 locale loc(locale::classic(), new unicode_codecvt); 374 // sanity check the constructed locale has the specialized facet. 375 VERIFY( has_facet<unicode_codecvt>(loc) ); 376 const unicode_codecvt& cvt = use_facet<unicode_codecvt>(loc); 377 // convert between const char* and unicode strings 378 unicode_codecvt::state_type state01("UNICODE", "ISO_8859-1"); 379 initialize_state(state01); 380 result r1 = cvt.in(state01, e_lit, e_lit + size, efrom_next, 381 i_arr, i_arr + size, ito_next); 382 VERIFY( r1 == codecvt_base::ok ); 383 VERIFY( !int_traits::compare(i_arr, i_lit, size) ); 384 VERIFY( efrom_next == e_lit + size ); 385 VERIFY( ito_next == i_arr + size ); 386</pre></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.future"></a>Future</h4></div></div></div><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p> 387 a. things that are sketchy, or remain unimplemented: 388 do_encoding, max_length and length member functions 389 are only weakly implemented. I have no idea how to do 390 this correctly, and in a generic manner. Nathan? 391</p></li><li class="listitem"><p> 392 b. conversions involving std::string 393 </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: circle; "><li class="listitem"><p> 394 how should operators != and == work for string of 395 different/same encoding? 396 </p></li><li class="listitem"><p> 397 what is equal? A byte by byte comparison or an 398 encoding then byte comparison? 399 </p></li><li class="listitem"><p> 400 conversions between narrow, wide, and unicode strings 401 </p></li></ul></div></li><li class="listitem"><p> 402 c. conversions involving std::filebuf and std::ostream 403</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: circle; "><li class="listitem"><p> 404 how to initialize the state object in a 405 standards-conformant manner? 406 </p></li><li class="listitem"><p> 407 how to synchronize the "C" and "C++" 408 conversion information? 409 </p></li><li class="listitem"><p> 410 wchar_t/char internal buffers and conversions between 411 internal/external buffers? 412 </p></li></ul></div></li></ul></div></div><div class="bibliography"><div class="titlepage"><div><div><h4 class="title"><a id="facet.codecvt.biblio"></a>Bibliography</h4></div></div></div><div class="biblioentry"><a id="idm269999640080"></a><p><span class="citetitle"><em class="citetitle"> 413 The GNU C Library 414 </em>. </span><span class="author"><span class="firstname">Roland</span> <span class="surname">McGrath</span>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright �� 2007 FSF. </span><span class="pagenums"> 415 Chapters 6 Character Set Handling and 7 Locales and Internationalization 416 . </span></p></div><div class="biblioentry"><a id="idm269999635328"></a><p><span class="citetitle"><em class="citetitle"> 417 Correspondence 418 </em>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright �� 2002 . </span></p></div><div class="biblioentry"><a id="idm269999632240"></a><p><span class="citetitle"><em class="citetitle"> 419 ISO/IEC 14882:1998 Programming languages - C++ 420 </em>. </span><span class="copyright">Copyright �� 1998 ISO. </span></p></div><div class="biblioentry"><a id="idm269999629952"></a><p><span class="citetitle"><em class="citetitle"> 421 ISO/IEC 9899:1999 Programming languages - C 422 </em>. </span><span class="copyright">Copyright �� 1999 ISO. </span></p></div><div class="biblioentry"><a id="idm269999627680"></a><p><span class="title"><em> 423 <a class="link" href="http://www.opengroup.org/austin/" target="_top"> 424 System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008) 425 </a> 426 </em>. </span><span class="copyright">Copyright �� 2008 427 The Open Group/The Institute of Electrical and Electronics 428 Engineers, Inc. 429 . </span></p></div><div class="biblioentry"><a id="idm269999624448"></a><p><span class="citetitle"><em class="citetitle"> 430 The C++ Programming Language, Special Edition 431 </em>. </span><span class="author"><span class="firstname">Bjarne</span> <span class="surname">Stroustrup</span>. </span><span class="copyright">Copyright �� 2000 Addison Wesley, Inc.. </span><span class="pagenums">Appendix D. </span><span class="publisher"><span class="publishername"> 432 Addison Wesley 433 . </span></span></p></div><div class="biblioentry"><a id="idm269999619824"></a><p><span class="citetitle"><em class="citetitle"> 434 Standard C++ IOStreams and Locales 435 </em>. </span><span class="subtitle"> 436 Advanced Programmer's Guide and Reference 437 . </span><span class="author"><span class="firstname">Angelika</span> <span class="surname">Langer</span>. </span><span class="author"><span class="firstname">Klaus</span> <span class="surname">Kreft</span>. </span><span class="copyright">Copyright �� 2000 Addison Wesley Longman, Inc.. </span><span class="publisher"><span class="publishername"> 438 Addison Wesley Longman 439 . </span></span></p></div><div class="biblioentry"><a id="idm269999614160"></a><p><span class="title"><em> 440 <a class="link" href="http://www.lysator.liu.se/c/na1.html" target="_top"> 441 A brief description of Normative Addendum 1 442 </a> 443 </em>. </span><span class="author"><span class="firstname">Clive</span> <span class="surname">Feather</span>. </span><span class="pagenums">Extended Character Sets. </span></p></div><div class="biblioentry"><a id="idm269999610912"></a><p><span class="title"><em> 444 <a class="link" href="http://tldp.org/HOWTO/Unicode-HOWTO.html" target="_top"> 445 The Unicode HOWTO 446 </a> 447 </em>. </span><span class="author"><span class="firstname">Bruno</span> <span class="surname">Haible</span>. </span></p></div><div class="biblioentry"><a id="idm269999608128"></a><p><span class="title"><em> 448 <a class="link" href="http://www.cl.cam.ac.uk/~mgk25/unicode.html" target="_top"> 449 UTF-8 and Unicode FAQ for Unix/Linux 450 </a> 451 </em>. </span><span class="author"><span class="firstname">Markus</span> <span class="surname">Khun</span>. </span></p></div></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a id="manual.localization.facet.messages"></a>messages</h3></div></div></div><p> 452The std::messages facet implements message retrieval functionality 453equivalent to Java's java.text.MessageFormat .using either GNU gettext 454or IEEE 1003.1-200 functions. 455</p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.req"></a>Requirements</h4></div></div></div><p> 456The std::messages facet is probably the most vaguely defined facet in 457the standard library. It's assumed that this facility was built into 458the standard library in order to convert string literals from one 459locale to the other. For instance, converting the "C" locale's 460<code class="code">const char* c = "please"</code> to a German-localized <code class="code">"bitte"</code> 461during program execution. 462</p><div class="blockquote"><blockquote class="blockquote"><p> 46322.2.7.1 - Template class messages [lib.locale.messages] 464</p></blockquote></div><p> 465This class has three public member functions, which directly 466correspond to three protected virtual member functions. 467</p><p> 468The public member functions are: 469</p><p> 470<code class="code">catalog open(const string&, const locale&) const</code> 471</p><p> 472<code class="code">string_type get(catalog, int, int, const string_type&) const</code> 473</p><p> 474<code class="code">void close(catalog) const</code> 475</p><p> 476While the virtual functions are: 477</p><p> 478<code class="code">catalog do_open(const string&, const locale&) const</code> 479</p><div class="blockquote"><blockquote class="blockquote"><p> 480<span class="emphasis"><em> 481-1- Returns: A value that may be passed to get() to retrieve a 482message, from the message catalog identified by the string name 483according to an implementation-defined mapping. The result can be used 484until it is passed to close(). Returns a value less than 0 if no such 485catalog can be opened. 486</em></span> 487</p></blockquote></div><p> 488<code class="code">string_type do_get(catalog, int, int, const string_type&) const</code> 489</p><div class="blockquote"><blockquote class="blockquote"><p> 490<span class="emphasis"><em> 491-3- Requires: A catalog cat obtained from open() and not yet closed. 492-4- Returns: A message identified by arguments set, msgid, and dfault, 493according to an implementation-defined mapping. If no such message can 494be found, returns dfault. 495</em></span> 496</p></blockquote></div><p> 497<code class="code">void do_close(catalog) const</code> 498</p><div class="blockquote"><blockquote class="blockquote"><p> 499<span class="emphasis"><em> 500-5- Requires: A catalog cat obtained from open() and not yet closed. 501-6- Effects: Releases unspecified resources associated with cat. 502-7- Notes: The limit on such resources, if any, is implementation-defined. 503</em></span> 504</p></blockquote></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.design"></a>Design</h4></div></div></div><p> 505A couple of notes on the standard. 506</p><p> 507First, why is <code class="code">messages_base::catalog</code> specified as a typedef 508to int? This makes sense for implementations that use 509<code class="code">catopen</code> and define <code class="code">nl_catd</code> as int, but not for 510others. Fortunately, it's not heavily used and so only a minor irritant. 511This has been reported as a possible defect in the standard (LWG 2028). 512</p><p> 513Second, by making the member functions <code class="code">const</code>, it is 514impossible to save state in them. Thus, storing away information used 515in the 'open' member function for use in 'get' is impossible. This is 516unfortunate. 517</p><p> 518The 'open' member function in particular seems to be oddly 519designed. The signature seems quite peculiar. Why specify a <code class="code">const 520string& </code> argument, for instance, instead of just <code class="code">const 521char*</code>? Or, why specify a <code class="code">const locale&</code> argument that is 522to be used in the 'get' member function? How, exactly, is this locale 523argument useful? What was the intent? It might make sense if a locale 524argument was associated with a given default message string in the 525'open' member function, for instance. Quite murky and unclear, on 526reflection. 527</p><p> 528Lastly, it seems odd that messages, which explicitly require code 529conversion, don't use the codecvt facet. Because the messages facet 530has only one template parameter, it is assumed that ctype, and not 531codecvt, is to be used to convert between character sets. 532</p><p> 533It is implicitly assumed that the locale for the default message 534string in 'get' is in the "C" locale. Thus, all source code is assumed 535to be written in English, so translations are always from "en_US" to 536other, explicitly named locales. 537</p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.impl"></a>Implementation</h4></div></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="messages.impl.models"></a>Models</h5></div></div></div><p> 538 This is a relatively simple class, on the face of it. The standard 539 specifies very little in concrete terms, so generic 540 implementations that are conforming yet do very little are the 541 norm. Adding functionality that would be useful to programmers and 542 comparable to Java's java.text.MessageFormat takes a bit of work, 543 and is highly dependent on the capabilities of the underlying 544 operating system. 545 </p><p> 546 Three different mechanisms have been provided, selectable via 547 configure flags: 548 </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p> 549 generic 550 </p><p> 551 This model does very little, and is what is used by default. 552 </p></li><li class="listitem"><p> 553 gnu 554 </p><p> 555 The gnu model is complete and fully tested. It's based on the 556 GNU gettext package, which is part of glibc. It uses the 557 functions <code class="code">textdomain, bindtextdomain, gettext</code> to 558 implement full functionality. Creating message catalogs is a 559 relatively straight-forward process and is lightly documented 560 below, and fully documented in gettext's distributed 561 documentation. 562 </p></li><li class="listitem"><p> 563 ieee_1003.1-200x 564 </p><p> 565 This is a complete, though untested, implementation based on 566 the IEEE standard. The functions <code class="code">catopen, catgets, 567 catclose</code> are used to retrieve locale-specific messages 568 given the appropriate message catalogs that have been 569 constructed for their use. Note, the script <code class="code"> 570 po2msg.sed</code> that is part of the gettext distribution can 571 convert gettext catalogs into catalogs that 572 <code class="code">catopen</code> can use. 573 </p></li></ul></div><p> 574A new, standards-conformant non-virtual member function signature was 575added for 'open' so that a directory could be specified with a given 576message catalog. This simplifies calling conventions for the gnu 577model. 578</p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a id="messages.impl.gnu"></a>The GNU Model</h5></div></div></div><p> 579 The messages facet, because it is retrieving and converting 580 between characters sets, depends on the ctype and perhaps the 581 codecvt facet in a given locale. In addition, underlying "C" 582 library locale support is necessary for more than just the 583 <code class="code">LC_MESSAGES</code> mask: <code class="code">LC_CTYPE</code> is also 584 necessary. To avoid any unpleasantness, all bits of the "C" mask 585 (i.e. <code class="code">LC_ALL</code>) are set before retrieving messages. 586 </p><p> 587 Making the message catalogs can be initially tricky, but become 588 quite simple with practice. For complete info, see the gettext 589 documentation. Here's an idea of what is required: 590 </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p> 591 Make a source file with the required string literals that need 592 to be translated. See <code class="code">intl/string_literals.cc</code> for 593 an example. 594 </p></li><li class="listitem"><p> 595 Make initial catalog (see "4 Making the PO Template File" from 596 the gettext docs).</p><p> 597 <code class="code"> xgettext --c++ --debug string_literals.cc -o libstdc++.pot </code> 598 </p></li><li class="listitem"><p>Make language and country-specific locale catalogs.</p><p> 599 <code class="code">cp libstdc++.pot fr_FR.po</code> 600 </p><p> 601 <code class="code">cp libstdc++.pot de_DE.po</code> 602 </p></li><li class="listitem"><p> 603 Edit localized catalogs in emacs so that strings are 604 translated. 605 </p><p> 606 <code class="code">emacs fr_FR.po</code> 607 </p></li><li class="listitem"><p>Make the binary mo files.</p><p> 608 <code class="code">msgfmt fr_FR.po -o fr_FR.mo</code> 609 </p><p> 610 <code class="code">msgfmt de_DE.po -o de_DE.mo</code> 611 </p></li><li class="listitem"><p>Copy the binary files into the correct directory structure.</p><p> 612 <code class="code">cp fr_FR.mo (dir)/fr_FR/LC_MESSAGES/libstdc++.mo</code> 613 </p><p> 614 <code class="code">cp de_DE.mo (dir)/de_DE/LC_MESSAGES/libstdc++.mo</code> 615 </p></li><li class="listitem"><p>Use the new message catalogs.</p><p> 616 <code class="code">locale loc_de("de_DE");</code> 617 </p><p> 618 <code class="code"> 619 use_facet<messages<char> >(loc_de).open("libstdc++", locale(), dir); 620 </code> 621 </p></li></ul></div></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.use"></a>Use</h4></div></div></div><p> 622 A simple example using the GNU model of message conversion. 623 </p><pre class="programlisting"> 624#include <iostream> 625#include <locale> 626using namespace std; 627 628void test01() 629{ 630 typedef messages<char>::catalog catalog; 631 const char* dir = 632 "/mnt/egcs/build/i686-pc-linux-gnu/libstdc++/po/share/locale"; 633 const locale loc_de("de_DE"); 634 const messages<char>& mssg_de = use_facet<messages<char> >(loc_de); 635 636 catalog cat_de = mssg_de.open("libstdc++", loc_de, dir); 637 string s01 = mssg_de.get(cat_de, 0, 0, "please"); 638 string s02 = mssg_de.get(cat_de, 0, 0, "thank you"); 639 cout << "please in german:" << s01 << '\n'; 640 cout << "thank you in german:" << s02 << '\n'; 641 mssg_de.close(cat_de); 642} 643</pre></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.future"></a>Future</h4></div></div></div><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p> 644 Things that are sketchy, or remain unimplemented: 645 </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: circle; "><li class="listitem"><p> 646 _M_convert_from_char, _M_convert_to_char are in flux, 647 depending on how the library ends up doing character set 648 conversions. It might not be possible to do a real character 649 set based conversion, due to the fact that the template 650 parameter for messages is not enough to instantiate the 651 codecvt facet (1 supplied, need at least 2 but would prefer 652 3). 653 </p></li><li class="listitem"><p> 654 There are issues with gettext needing the global locale set 655 to extract a message. This dependence on the global locale 656 makes the current "gnu" model non MT-safe. Future versions 657 of glibc, i.e. glibc 2.3.x will fix this, and the C++ library 658 bits are already in place. 659 </p></li></ul></div></li><li class="listitem"><p> 660 Development versions of the GNU "C" library, glibc 2.3 will allow 661 a more efficient, MT implementation of std::messages, and will 662 allow the removal of the _M_name_messages data member. If this is 663 done, it will change the library ABI. The C++ parts to support 664 glibc 2.3 have already been coded, but are not in use: once this 665 version of the "C" library is released, the marked parts of the 666 messages implementation can be switched over to the new "C" 667 library functionality. 668 </p></li><li class="listitem"><p> 669 At some point in the near future, std::numpunct will probably use 670 std::messages facilities to implement truename/falsename 671 correctly. This is currently not done, but entries in 672 libstdc++.pot have already been made for "true" and "false" string 673 literals, so all that remains is the std::numpunct coding and the 674 configure/make hassles to make the installed library search its 675 own catalog. Currently the libstdc++.mo catalog is only searched 676 for the testsuite cases involving messages members. 677 </p></li><li class="listitem"><p> The following member functions:</p><p> 678 <code class="code"> 679 catalog 680 open(const basic_string<char>& __s, const locale& __loc) const 681 </code> 682 </p><p> 683 <code class="code"> 684 catalog 685 open(const basic_string<char>&, const locale&, const char*) const; 686 </code> 687 </p><p> 688 Don't actually return a "value less than 0 if no such catalog 689 can be opened" as required by the standard in the "gnu" 690 model. As of this writing, it is unknown how to query to see 691 if a specified message catalog exists using the gettext 692 package. 693 </p></li></ul></div></div><div class="bibliography"><div class="titlepage"><div><div><h4 class="title"><a id="facet.messages.biblio"></a>Bibliography</h4></div></div></div><div class="biblioentry"><a id="idm269999527584"></a><p><span class="citetitle"><em class="citetitle"> 694 The GNU C Library 695 </em>. </span><span class="author"><span class="firstname">Roland</span> <span class="surname">McGrath</span>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright �� 2007 FSF. </span><span class="pagenums">Chapters 6 Character Set Handling, and 7 Locales and Internationalization 696 . </span></p></div><div class="biblioentry"><a id="idm269999522832"></a><p><span class="citetitle"><em class="citetitle"> 697 Correspondence 698 </em>. </span><span class="author"><span class="firstname">Ulrich</span> <span class="surname">Drepper</span>. </span><span class="copyright">Copyright �� 2002 . </span></p></div><div class="biblioentry"><a id="idm269999519744"></a><p><span class="citetitle"><em class="citetitle"> 699 ISO/IEC 14882:1998 Programming languages - C++ 700 </em>. </span><span class="copyright">Copyright �� 1998 ISO. </span></p></div><div class="biblioentry"><a id="idm269999517456"></a><p><span class="citetitle"><em class="citetitle"> 701 ISO/IEC 9899:1999 Programming languages - C 702 </em>. </span><span class="copyright">Copyright �� 1999 ISO. </span></p></div><div class="biblioentry"><a id="idm269999515184"></a><p><span class="title"><em> 703 <a class="link" href="http://www.opengroup.org/austin/" target="_top"> 704 System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008) 705 </a> 706 </em>. </span><span class="copyright">Copyright �� 2008 707 The Open Group/The Institute of Electrical and Electronics 708 Engineers, Inc. 709 . </span></p></div><div class="biblioentry"><a id="idm269999511952"></a><p><span class="citetitle"><em class="citetitle"> 710 The C++ Programming Language, Special Edition 711 </em>. </span><span class="author"><span class="firstname">Bjarne</span> <span class="surname">Stroustrup</span>. </span><span class="copyright">Copyright �� 2000 Addison Wesley, Inc.. </span><span class="pagenums">Appendix D. </span><span class="publisher"><span class="publishername"> 712 Addison Wesley 713 . </span></span></p></div><div class="biblioentry"><a id="idm269999507328"></a><p><span class="citetitle"><em class="citetitle"> 714 Standard C++ IOStreams and Locales 715 </em>. </span><span class="subtitle"> 716 Advanced Programmer's Guide and Reference 717 . </span><span class="author"><span class="firstname">Angelika</span> <span class="surname">Langer</span>. </span><span class="author"><span class="firstname">Klaus</span> <span class="surname">Kreft</span>. </span><span class="copyright">Copyright �� 2000 Addison Wesley Longman, Inc.. </span><span class="publisher"><span class="publishername"> 718 Addison Wesley Longman 719 . </span></span></p></div><div class="biblioentry"><a id="idm269999501664"></a><p><span class="title"><em> 720 <a class="link" href="http://www.oracle.com/technetwork/java/api/index.html" target="_top"> 721 API Specifications, Java Platform 722 </a> 723 </em>. </span><span class="pagenums">java.util.Properties, java.text.MessageFormat, 724java.util.Locale, java.util.ResourceBundle 725 . </span></p></div><div class="biblioentry"><a id="idm269999499328"></a><p><span class="title"><em> 726 <a class="link" href="https://www.gnu.org/software/gettext/" target="_top"> 727 GNU gettext tools, version 0.10.38, Native Language Support 728 Library and Tools. 729 </a> 730 </em>. </span></p></div></div></div></div><div class="navfooter"><hr /><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="localization.html">Prev</a>��</td><td width="20%" align="center"><a accesskey="u" href="localization.html">Up</a></td><td width="40%" align="right">��<a accesskey="n" href="containers.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">Chapter��8.�� 731 Localization 732 733��</td><td width="20%" align="center"><a accesskey="h" href="../index.html">Home</a></td><td width="40%" align="right" valign="top">��Chapter��9.�� 734 Containers 735 736</td></tr></table></div></body></html>