1<section xmlns="http://docbook.org/ns/docbook" version="5.0" 2 xml:id="std.localization.facet.codecvt" xreflabel="codecvt"> 3<?dbhtml filename="codecvt.html"?> 4 5<info><title>codecvt</title> 6 <keywordset> 7 <keyword>ISO C++</keyword> 8 <keyword>codecvt</keyword> 9 </keywordset> 10</info> 11 12 13 14<para> 15The standard class codecvt attempts to address conversions between 16different character encoding schemes. In particular, the standard 17attempts to detail conversions between the implementation-defined wide 18characters (hereafter referred to as <type>wchar_t</type>) and the standard 19type <type>char</type> that is so beloved in classic <quote>C</quote> 20(which can now be referred to as narrow characters.) This document attempts 21to describe how the GNU libstdc++ implementation deals with the conversion 22between wide and narrow characters, and also presents a framework for dealing 23with the huge number of other encodings that iconv can convert, 24including Unicode and UTF8. Design issues and requirements are 25addressed, and examples of correct usage for both the required 26specializations for wide and narrow characters and the 27implementation-provided extended functionality are given. 28</para> 29 30<section xml:id="facet.codecvt.req"><info><title>Requirements</title></info> 31 32 33<para> 34Around page 425 of the C++ Standard, this charming heading comes into view: 35</para> 36 37<blockquote> 38<para> 3922.2.1.5 - Template class codecvt 40</para> 41</blockquote> 42 43<para> 44The text around the codecvt definition gives some clues: 45</para> 46 47<blockquote> 48<para> 49<emphasis> 50-1- The class <code>codecvt<internT,externT,stateT></code> is for use 51when converting from one codeset to another, such as from wide characters 52to multibyte characters, between wide character encodings such as 53Unicode and EUC. 54</emphasis> 55</para> 56</blockquote> 57 58<para> 59Hmm. So, in some unspecified way, Unicode encodings and 60translations between other character sets should be handled by this 61class. 62</para> 63 64<blockquote> 65<para> 66<emphasis> 67-2- The <type>stateT</type> argument selects the pair of codesets being mapped between. 68</emphasis> 69</para> 70</blockquote> 71 72<para> 73Ah ha! Another clue... 74</para> 75 76<blockquote> 77<para> 78<emphasis> 79-3- The instantiations required in the Table 51 (lib.locale.category), namely 80<classname>codecvt<wchar_t,char,mbstate_t></classname> and 81<classname>codecvt<char,char,mbstate_t></classname>, convert the 82implementation-defined native character set. 83<classname>codecvt<char,char,mbstate_t></classname> implements a 84degenerate conversion; it does not convert at all. 85<classname>codecvt<wchar_t,char,mbstate_t></classname> converts between 86the native character sets for tiny and wide characters. Instantiations on 87<type>mbstate_t</type> perform conversion between encodings known to the library 88implementor. Other encodings can be converted by specializing on a 89user-defined <type>stateT</type> type. The <type>stateT</type> object can 90contain any state that is useful to communicate to or from the specialized 91<function>do_convert</function> member. 92</emphasis> 93</para> 94</blockquote> 95 96<para> 97At this point, a couple points become clear: 98</para> 99 100<para> 101One: The standard clearly implies that attempts to add non-required 102(yet useful and widely used) conversions need to do so through the 103third template parameter, <type>stateT</type>.</para> 104 105<para> 106Two: The required conversions, by specifying <type>mbstate_t</type> as the 107third template parameter, imply an implementation strategy that is mostly 108(or wholly) based on the underlying C library, and the functions 109<function>mcsrtombs</function> and <function>wcsrtombs</function> in 110particular.</para> 111</section> 112 113<section xml:id="facet.codecvt.design"><info><title>Design</title></info> 114 115 116<section xml:id="codecvt.design.wchar_t_size"><info><title><type>wchar_t</type> Size</title></info> 117 118 119 <para> 120 The simple implementation detail of <type>wchar_t</type>'s size seems to 121 repeatedly confound people. Many systems use a two byte, 122 unsigned integral type to represent wide characters, and use an 123 internal encoding of Unicode or UCS2. (See AIX, Microsoft NT, 124 Java, others.) Other systems, use a four byte, unsigned integral 125 type to represent wide characters, and use an internal encoding 126 of UCS4. (GNU/Linux systems using glibc, in particular.) The C 127 programming language (and thus C++) does not specify a specific 128 size for the type <type>wchar_t</type>. 129 </para> 130 131 <para> 132 Thus, portable C++ code cannot assume a byte size (or endianness) either. 133 </para> 134 </section> 135 136<section xml:id="codecvt.design.unicode"><info><title>Support for Unicode</title></info> 137 138 <para> 139 Probably the most frequently asked question about code conversion 140 is: "So dudes, what's the deal with Unicode strings?" 141 The dude part is optional, but apparently the usefulness of 142 Unicode strings is pretty widely appreciated. The Unicode character 143 set (and useful encodings like UTF-8, UCS-4, ISO 8859-10, 144 etc etc etc) were not mentioned in the first C++ standard. (The 2011 145 standard added support for string literals with different encodings 146 and some library facilities for converting between encodings, but the 147 notes below have not been updated to reflect that.) 148 </para> 149 150 <para> 151 A couple of comments: 152 </para> 153 154 <para> 155 The thought that all one needs to convert between two arbitrary 156 codesets is two types and some kind of state argument is 157 unfortunate. In particular, encodings may be stateless. The naming 158 of the third parameter as <type>stateT</type> is unfortunate, as what is 159 really needed is some kind of generalized type that accounts for the 160 issues that abstract encodings will need. The minimum information 161 that is required includes: 162 </para> 163 164 <itemizedlist> 165 <listitem> 166 <para> 167 Identifiers for each of the codesets involved in the 168 conversion. For example, using the iconv family of functions 169 from the Single Unix Specification (what used to be called 170 X/Open) hosted on the GNU/Linux operating system allows 171 bi-directional mapping between far more than the following 172 tantalizing possibilities: 173 </para> 174 175 <para> 176 (An edited list taken from <code>`iconv --list`</code> on a 177 Red Hat 6.2/Intel system: 178 </para> 179 180<blockquote> 181<programlisting> 1828859_1, 8859_9, 10646-1:1993, 10646-1:1993/UCS4, ARABIC, ARABIC7, 183ASCII, EUC-CN, EUC-JP, EUC-KR, EUC-TW, GREEK-CCIcode, GREEK, GREEK7-OLD, 184GREEK7, GREEK8, HEBREW, ISO-8859-1, ISO-8859-2, ISO-8859-3, 185ISO-8859-4, ISO-8859-5, ISO-8859-6, ISO-8859-7, ISO-8859-8, 186ISO-8859-9, ISO-8859-10, ISO-8859-11, ISO-8859-13, ISO-8859-14, 187ISO-8859-15, ISO-10646, ISO-10646/UCS2, ISO-10646/UCS4, 188ISO-10646/UTF-8, ISO-10646/UTF8, SHIFT-JIS, SHIFT_JIS, UCS-2, UCS-4, 189UCS2, UCS4, UNICODE, UNICODEBIG, UNICODELIcodeLE, US-ASCII, US, UTF-8, 190UTF-16, UTF8, UTF16). 191</programlisting> 192</blockquote> 193 194<para> 195For iconv-based implementations, string literals for each of the 196encodings (i.e. "UCS-2" and "UTF-8") are necessary, 197although for other, 198non-iconv implementations a table of enumerated values or some other 199mechanism may be required. 200</para> 201</listitem> 202 203<listitem><para> 204 Maximum length of the identifying string literal. 205</para></listitem> 206 207<listitem><para> 208 Some encodings require explicit endian-ness. As such, some kind 209 of endian marker or other byte-order marker will be necessary. See 210 "Footnotes for C/C++ developers" in Haible for more information on 211 UCS-2/Unicode endian issues. (Summary: big endian seems most likely, 212 however implementations, most notably Microsoft, vary.) 213</para></listitem> 214 215<listitem><para> 216 Types representing the conversion state, for conversions involving 217 the machinery in the "C" library, or the conversion descriptor, for 218 conversions using iconv (such as the type iconv_t.) Note that the 219 conversion descriptor encodes more information than a simple encoding 220 state type. 221</para></listitem> 222 223<listitem><para> 224 Conversion descriptors for both directions of encoding. (i.e., both 225 UCS-2 to UTF-8 and UTF-8 to UCS-2.) 226</para></listitem> 227 228<listitem><para> 229 Something to indicate if the conversion requested if valid. 230</para></listitem> 231 232<listitem><para> 233 Something to represent if the conversion descriptors are valid. 234</para></listitem> 235 236<listitem><para> 237 Some way to enforce strict type checking on the internal and 238 external types. As part of this, the size of the internal and 239 external types will need to be known. 240</para></listitem> 241</itemizedlist> 242</section> 243 244<section xml:id="codecvt.design.issues"><info><title>Other Issues</title></info> 245 246<para> 247In addition, multi-threaded and multi-locale environments also impact 248the design and requirements for code conversions. In particular, they 249affect the required specialization 250<classname>codecvt<wchar_t, char, mbstate_t></classname> 251when implemented using standard "C" functions. 252</para> 253 254<para> 255Three problems arise, one big, one of medium importance, and one small. 256</para> 257 258<para> 259First, the small: <function>mcsrtombs</function> and 260<function>wcsrtombs</function> may not be multithread-safe 261on all systems required by the GNU tools. For GNU/Linux and glibc, 262this is not an issue. 263</para> 264 265<para> 266Of medium concern, in the grand scope of things, is that the functions 267used to implement this specialization work on null-terminated 268strings. Buffers, especially file buffers, may not be null-terminated, 269thus giving conversions that end prematurely or are otherwise 270incorrect. Yikes! 271</para> 272 273<para> 274The last, and fundamental problem, is the assumption of a global 275locale for all the "C" functions referenced above. For something like 276C++ iostreams (where codecvt is explicitly used) the notion of 277multiple locales is fundamental. In practice, most users may not run 278into this limitation. However, as a quality of implementation issue, 279the GNU C++ library would like to offer a solution that allows 280multiple locales and or simultaneous usage with computationally 281correct results. In short, libstdc++ is trying to offer, as an 282option, a high-quality implementation, damn the additional complexity! 283</para> 284 285<para> 286For the required specialization 287<classname>codecvt<wchar_t, char, mbstate_t></classname>, 288conversions are made between the internal character set (always UCS4 289on GNU/Linux) and whatever the currently selected locale for the 290LC_CTYPE category implements. 291</para> 292 293</section> 294 295</section> 296 297<section xml:id="facet.codecvt.impl"><info><title>Implementation</title></info> 298 299 300<para> 301The two required specializations are implemented as follows: 302</para> 303 304<para> 305<code> 306codecvt<char, char, mbstate_t> 307</code> 308</para> 309<para> 310This is a degenerate (i.e., does nothing) specialization. Implementing 311this was a piece of cake. 312</para> 313 314<para> 315<code> 316codecvt<char, wchar_t, mbstate_t> 317</code> 318</para> 319 320<para> 321This specialization, by specifying all the template parameters, pretty 322much ties the hands of implementors. As such, the implementation is 323straightforward, involving <function>mcsrtombs</function> for the conversions 324between <type>char</type> to <type>wchar_t</type> and 325<function>wcsrtombs</function> for conversions between <type>wchar_t</type> 326and <type>char</type>. 327</para> 328 329<para> 330Neither of these two required specializations deals with Unicode 331characters. As such, libstdc++ implements a partial specialization 332of the <type>codecvt</type> class with an iconv wrapper class, 333<classname>encoding_state</classname> as the third template parameter. 334</para> 335 336<para> 337This implementation should be standards conformant. First of all, the 338standard explicitly points out that instantiations on the third 339template parameter, <type>stateT</type>, are the proper way to implement 340non-required conversions. Second of all, the standard says (in Chapter 34117) that partial specializations of required classes are A-OK. Third 342of all, the requirements for the <type>stateT</type> type elsewhere in the 343standard (see 21.1.2 traits typedefs) only indicate that this type be copy 344constructible. 345</para> 346 347<para> 348As such, the type <type>encoding_state</type> is defined as a non-templatized, 349POD type to be used as the third type of a <type>codecvt</type> instantiation. 350This type is just a wrapper class for iconv, and provides an easy interface 351to iconv functionality. 352</para> 353 354<para> 355There are two constructors for <type>encoding_state</type>: 356</para> 357 358<para> 359<code> 360encoding_state() : __in_desc(0), __out_desc(0) 361</code> 362</para> 363<para> 364This default constructor sets the internal encoding to some default 365(currently UCS4) and the external encoding to whatever is returned by 366<code>nl_langinfo(CODESET)</code>. 367</para> 368 369<para> 370<code> 371encoding_state(const char* __int, const char* __ext) 372</code> 373</para> 374 375<para> 376This constructor takes as parameters string literals that indicate the 377desired internal and external encoding. There are no defaults for 378either argument. 379</para> 380 381<para> 382One of the issues with iconv is that the string literals identifying 383conversions are not standardized. Because of this, the thought of 384mandating and/or enforcing some set of pre-determined valid 385identifiers seems iffy: thus, a more practical (and non-migraine 386inducing) strategy was implemented: end-users can specify any string 387(subject to a pre-determined length qualifier, currently 32 bytes) for 388encodings. It is up to the user to make sure that these strings are 389valid on the target system. 390</para> 391 392<para> 393<code> 394void 395_M_init() 396</code> 397</para> 398<para> 399Strangely enough, this member function attempts to open conversion 400descriptors for a given encoding_state object. If the conversion 401descriptors are not valid, the conversion descriptors returned will 402not be valid and the resulting calls to the codecvt conversion 403functions will return error. 404</para> 405 406<para> 407<code> 408bool 409_M_good() 410</code> 411</para> 412 413<para> 414Provides a way to see if the given <type>encoding_state</type> object has been 415properly initialized. If the string literals describing the desired 416internal and external encoding are not valid, initialization will 417fail, and this will return false. If the internal and external 418encodings are valid, but <function>iconv_open</function> could not allocate 419conversion descriptors, this will also return false. Otherwise, the object is 420ready to convert and will return true. 421</para> 422 423<para> 424<code> 425encoding_state(const encoding_state&) 426</code> 427</para> 428 429<para> 430As iconv allocates memory and sets up conversion descriptors, the copy 431constructor can only copy the member data pertaining to the internal 432and external code conversions, and not the conversion descriptors 433themselves. 434</para> 435 436<para> 437Definitions for all the required codecvt member functions are provided 438for this specialization, and usage of <code>codecvt<<replaceable>internal 439character type</replaceable>, <replaceable>external character type</replaceable>, <replaceable>encoding_state</replaceable>></code> is consistent with other 440codecvt usage. 441</para> 442 443</section> 444 445<section xml:id="facet.codecvt.use"><info><title>Use</title></info> 446 447<para>A conversion involving a string literal.</para> 448 449<programlisting> 450 typedef codecvt_base::result result; 451 typedef unsigned short unicode_t; 452 typedef unicode_t int_type; 453 typedef char ext_type; 454 typedef encoding_state state_type; 455 typedef codecvt<int_type, ext_type, state_type> unicode_codecvt; 456 457 const ext_type* e_lit = "black pearl jasmine tea"; 458 int size = strlen(e_lit); 459 int_type i_lit_base[24] = 460 { 25088, 27648, 24832, 25344, 27392, 8192, 28672, 25856, 24832, 29184, 461 27648, 8192, 27136, 24832, 29440, 27904, 26880, 28160, 25856, 8192, 29696, 462 25856, 24832, 2560 463 }; 464 const int_type* i_lit = i_lit_base; 465 const ext_type* efrom_next; 466 const int_type* ifrom_next; 467 ext_type* e_arr = new ext_type[size + 1]; 468 ext_type* eto_next; 469 int_type* i_arr = new int_type[size + 1]; 470 int_type* ito_next; 471 472 // construct a locale object with the specialized facet. 473 locale loc(locale::classic(), new unicode_codecvt); 474 // sanity check the constructed locale has the specialized facet. 475 VERIFY( has_facet<unicode_codecvt>(loc) ); 476 const unicode_codecvt& cvt = use_facet<unicode_codecvt>(loc); 477 // convert between const char* and unicode strings 478 unicode_codecvt::state_type state01("UNICODE", "ISO_8859-1"); 479 initialize_state(state01); 480 result r1 = cvt.in(state01, e_lit, e_lit + size, efrom_next, 481 i_arr, i_arr + size, ito_next); 482 VERIFY( r1 == codecvt_base::ok ); 483 VERIFY( !int_traits::compare(i_arr, i_lit, size) ); 484 VERIFY( efrom_next == e_lit + size ); 485 VERIFY( ito_next == i_arr + size ); 486</programlisting> 487 488</section> 489 490<section xml:id="facet.codecvt.future"><info><title>Future</title></info> 491 492<itemizedlist> 493<listitem> 494 <para> 495 a. things that are sketchy, or remain unimplemented: 496 do_encoding, max_length and length member functions 497 are only weakly implemented. I have no idea how to do 498 this correctly, and in a generic manner. Nathan? 499</para> 500</listitem> 501 502<listitem> 503 <para> 504 b. conversions involving <type>std::string</type> 505 </para> 506 <itemizedlist> 507 <listitem><para> 508 how should operators != and == work for string of 509 different/same encoding? 510 </para></listitem> 511 512 <listitem><para> 513 what is equal? A byte by byte comparison or an 514 encoding then byte comparison? 515 </para></listitem> 516 517 <listitem><para> 518 conversions between narrow, wide, and unicode strings 519 </para></listitem> 520 </itemizedlist> 521</listitem> 522<listitem><para> 523 c. conversions involving std::filebuf and std::ostream 524</para> 525 <itemizedlist> 526 <listitem><para> 527 how to initialize the state object in a 528 standards-conformant manner? 529 </para></listitem> 530 531 <listitem><para> 532 how to synchronize the "C" and "C++" 533 conversion information? 534 </para></listitem> 535 536 <listitem><para> 537 wchar_t/char internal buffers and conversions between 538 internal/external buffers? 539 </para></listitem> 540 </itemizedlist> 541</listitem> 542</itemizedlist> 543</section> 544 545 546<bibliography xml:id="facet.codecvt.biblio"><info><title>Bibliography</title></info> 547 548 549 <biblioentry> 550 <citetitle> 551 The GNU C Library 552 </citetitle> 553 <author><personname><surname>McGrath</surname><firstname>Roland</firstname></personname></author> 554 <author><personname><surname>Drepper</surname><firstname>Ulrich</firstname></personname></author> 555 <copyright> 556 <year>2007</year> 557 <holder>FSF</holder> 558 </copyright> 559 <pagenums> 560 Chapters 6 Character Set Handling and 7 Locales and Internationalization 561 </pagenums> 562 </biblioentry> 563 564 <biblioentry> 565 <citetitle> 566 Correspondence 567 </citetitle> 568 <author><personname><surname>Drepper</surname><firstname>Ulrich</firstname></personname></author> 569 <copyright> 570 <year>2002</year> 571 <holder/> 572 </copyright> 573 </biblioentry> 574 575 <biblioentry> 576 <citetitle> 577 ISO/IEC 14882:1998 Programming languages - C++ 578 </citetitle> 579 <copyright> 580 <year>1998</year> 581 <holder>ISO</holder> 582 </copyright> 583 </biblioentry> 584 585 <biblioentry> 586 <citetitle> 587 ISO/IEC 9899:1999 Programming languages - C 588 </citetitle> 589 <copyright> 590 <year>1999</year> 591 <holder>ISO</holder> 592 </copyright> 593 </biblioentry> 594 595 <biblioentry> 596 <title> 597 <link xmlns:xlink="http://www.w3.org/1999/xlink" 598 xlink:href="https://pubs.opengroup.org/onlinepubs/9699919799/"> 599 System Interface Definitions, Issue 7 (IEEE Std. 1003.1-2008) 600 </link> 601 </title> 602 603 <copyright> 604 <year>2008</year> 605 <holder> 606 The Open Group/The Institute of Electrical and Electronics 607 Engineers, Inc. 608 </holder> 609 </copyright> 610 </biblioentry> 611 612 <biblioentry> 613 <citetitle> 614 The C++ Programming Language, Special Edition 615 </citetitle> 616 <author><personname><surname>Stroustrup</surname><firstname>Bjarne</firstname></personname></author> 617 <copyright> 618 <year>2000</year> 619 <holder>Addison Wesley, Inc.</holder> 620 </copyright> 621 <pagenums>Appendix D</pagenums> 622 <publisher> 623 <publishername> 624 Addison Wesley 625 </publishername> 626 </publisher> 627 </biblioentry> 628 629 630 <biblioentry> 631 <citetitle> 632 Standard C++ IOStreams and Locales 633 </citetitle> 634 <subtitle> 635 Advanced Programmer's Guide and Reference 636 </subtitle> 637 <author><personname><surname>Langer</surname><firstname>Angelika</firstname></personname></author> 638 <author><personname><surname>Kreft</surname><firstname>Klaus</firstname></personname></author> 639 <copyright> 640 <year>2000</year> 641 <holder>Addison Wesley Longman, Inc.</holder> 642 </copyright> 643 <publisher> 644 <publishername> 645 Addison Wesley Longman 646 </publishername> 647 </publisher> 648 </biblioentry> 649 650 <biblioentry> 651 <title> 652 <link xmlns:xlink="http://www.w3.org/1999/xlink" 653 xlink:href="http://www.lysator.liu.se/c/na1.html"> 654 A brief description of Normative Addendum 1 655 </link> 656 </title> 657 658 <author><personname><surname>Feather</surname><firstname>Clive</firstname></personname></author> 659 <pagenums>Extended Character Sets</pagenums> 660 </biblioentry> 661 662 <biblioentry> 663 <title> 664 <link xmlns:xlink="http://www.w3.org/1999/xlink" 665 xlink:href="http://tldp.org/HOWTO/Unicode-HOWTO.html"> 666 The Unicode HOWTO 667 </link> 668 </title> 669 670 <author><personname><surname>Haible</surname><firstname>Bruno</firstname></personname></author> 671 </biblioentry> 672 673 <biblioentry> 674 <title> 675 <link xmlns:xlink="http://www.w3.org/1999/xlink" 676 xlink:href="https://www.cl.cam.ac.uk/~mgk25/unicode.html"> 677 UTF-8 and Unicode FAQ for Unix/Linux 678 </link> 679 </title> 680 681 682 <author><personname><surname>Khun</surname><firstname>Markus</firstname></personname></author> 683 </biblioentry> 684 685</bibliography> 686 687</section> 688