1=head1 NAME 2 3perluniintro - Perl Unicode introduction 4 5=head1 DESCRIPTION 6 7This document gives a general idea of Unicode and how to use Unicode 8in Perl. See L</Further Resources> for references to more in-depth 9treatments of Unicode. 10 11=head2 Unicode 12 13Unicode is a character set standard which plans to codify all of the 14writing systems of the world, plus many other symbols. 15 16Unicode and ISO/IEC 10646 are coordinated standards that unify 17almost all other modern character set standards, 18covering more than 80 writing systems and hundreds of languages, 19including all commercially-important modern languages. All characters 20in the largest Chinese, Japanese, and Korean dictionaries are also 21encoded. The standards will eventually cover almost all characters in 22more than 250 writing systems and thousands of languages. 23Unicode 1.0 was released in October 1991, and 6.0 in October 2010. 24 25A Unicode I<character> is an abstract entity. It is not bound to any 26particular integer width, especially not to the C language C<char>. 27Unicode is language-neutral and display-neutral: it does not encode the 28language of the text, and it does not generally define fonts or other graphical 29layout details. Unicode operates on characters and on text built from 30those characters. 31 32Unicode defines characters like C<LATIN CAPITAL LETTER A> or C<GREEK 33SMALL LETTER ALPHA> and unique numbers for the characters, in this 34case 0x0041 and 0x03B1, respectively. These unique numbers are called 35I<code points>. A code point is essentially the position of the 36character within the set of all possible Unicode characters, and thus in 37Perl, the term I<ordinal> is often used interchangeably with it. 38 39The Unicode standard prefers using hexadecimal notation for the code 40points. If numbers like C<0x0041> are unfamiliar to you, take a peek 41at a later section, L</"Hexadecimal Notation">. The Unicode standard 42uses the notation C<U+0041 LATIN CAPITAL LETTER A>, to give the 43hexadecimal code point and the normative name of the character. 44 45Unicode also defines various I<properties> for the characters, like 46"uppercase" or "lowercase", "decimal digit", or "punctuation"; 47these properties are independent of the names of the characters. 48Furthermore, various operations on the characters like uppercasing, 49lowercasing, and collating (sorting) are defined. 50 51A Unicode I<logical> "character" can actually consist of more than one internal 52I<actual> "character" or code point. For Western languages, this is adequately 53modelled by a I<base character> (like C<LATIN CAPITAL LETTER A>) followed 54by one or more I<modifiers> (like C<COMBINING ACUTE ACCENT>). This sequence of 55base character and modifiers is called a I<combining character 56sequence>. Some non-western languages require more complicated 57models, so Unicode created the I<grapheme cluster> concept, which was 58later further refined into the I<extended grapheme cluster>. For 59example, a Korean Hangul syllable is considered a single logical 60character, but most often consists of three actual 61Unicode characters: a leading consonant followed by an interior vowel followed 62by a trailing consonant. 63 64Whether to call these extended grapheme clusters "characters" depends on your 65point of view. If you are a programmer, you probably would tend towards seeing 66each element in the sequences as one unit, or "character". However from 67the user's point of view, the whole sequence could be seen as one 68"character" since that's probably what it looks like in the context of the 69user's language. In this document, we take the programmer's point of 70view: one "character" is one Unicode code point. 71 72For some combinations of base character and modifiers, there are 73I<precomposed> characters. There is a single character equivalent, for 74example, for the sequence C<LATIN CAPITAL LETTER A> followed by 75C<COMBINING ACUTE ACCENT>. It is called C<LATIN CAPITAL LETTER A WITH 76ACUTE>. These precomposed characters are, however, only available for 77some combinations, and are mainly meant to support round-trip 78conversions between Unicode and legacy standards (like ISO 8859). Using 79sequences, as Unicode does, allows for needing fewer basic building blocks 80(code points) to express many more potential grapheme clusters. To 81support conversion between equivalent forms, various I<normalization 82forms> are also defined. Thus, C<LATIN CAPITAL LETTER A WITH ACUTE> is 83in I<Normalization Form Composed>, (abbreviated NFC), and the sequence 84C<LATIN CAPITAL LETTER A> followed by C<COMBINING ACUTE ACCENT> 85represents the same character in I<Normalization Form Decomposed> (NFD). 86 87Because of backward compatibility with legacy encodings, the "a unique 88number for every character" idea breaks down a bit: instead, there is 89"at least one number for every character". The same character could 90be represented differently in several legacy encodings. The 91converse is not true: some code points do not have an assigned 92character. Firstly, there are unallocated code points within 93otherwise used blocks. Secondly, there are special Unicode control 94characters that do not represent true characters. 95 96When Unicode was first conceived, it was thought that all the world's 97characters could be represented using a 16-bit word; that is a maximum of 98C<0x10000> (or 65,536) characters would be needed, from C<0x0000> to 99C<0xFFFF>. This soon proved to be wrong, and since Unicode 2.0 (July 1001996), Unicode has been defined all the way up to 21 bits (C<0x10FFFF>), 101and Unicode 3.1 (March 2001) defined the first characters above C<0xFFFF>. 102The first C<0x10000> characters are called the I<Plane 0>, or the 103I<Basic Multilingual Plane> (BMP). With Unicode 3.1, 17 (yes, 104seventeen) planes in all were defined--but they are nowhere near full of 105defined characters, yet. 106 107When a new language is being encoded, Unicode generally will choose a 108C<block> of consecutive unallocated code points for its characters. So 109far, the number of code points in these blocks has always been evenly 110divisible by 16. Extras in a block, not currently needed, are left 111unallocated, for future growth. But there have been occasions when 112a later release needed more code points than the available extras, and a 113new block had to allocated somewhere else, not contiguous to the initial 114one, to handle the overflow. Thus, it became apparent early on that 115"block" wasn't an adequate organizing principle, and so the C<Script> 116property was created. (Later an improved script property was added as 117well, the C<Script_Extensions> property.) Those code points that are in 118overflow blocks can still 119have the same script as the original ones. The script concept fits more 120closely with natural language: there is C<Latin> script, C<Greek> 121script, and so on; and there are several artificial scripts, like 122C<Common> for characters that are used in multiple scripts, such as 123mathematical symbols. Scripts usually span varied parts of several 124blocks. For more information about scripts, see L<perlunicode/Scripts>. 125The division into blocks exists, but it is almost completely 126accidental--an artifact of how the characters have been and still are 127allocated. (Note that this paragraph has oversimplified things for the 128sake of this being an introduction. Unicode doesn't really encode 129languages, but the writing systems for them--their scripts; and one 130script can be used by many languages. Unicode also encodes things that 131aren't really about languages, such as symbols like C<BAGGAGE CLAIM>.) 132 133The Unicode code points are just abstract numbers. To input and 134output these abstract numbers, the numbers must be I<encoded> or 135I<serialised> somehow. Unicode defines several I<character encoding 136forms>, of which I<UTF-8> is the most popular. UTF-8 is a 137variable length encoding that encodes Unicode characters as 1 to 4 138bytes. Other encodings 139include UTF-16 and UTF-32 and their big- and little-endian variants 140(UTF-8 is byte-order independent). The ISO/IEC 10646 defines the UCS-2 141and UCS-4 encoding forms. 142 143For more information about encodings--for instance, to learn what 144I<surrogates> and I<byte order marks> (BOMs) are--see L<perlunicode>. 145 146=head2 Perl's Unicode Support 147 148Starting from Perl v5.6.0, Perl has had the capacity to handle Unicode 149natively. Perl v5.8.0, however, is the first recommended release for 150serious Unicode work. The maintenance release 5.6.1 fixed many of the 151problems of the initial Unicode implementation, but for example 152regular expressions still do not work with Unicode in 5.6.1. 153Perl v5.14.0 is the first release where Unicode support is 154(almost) seamlessly integratable without some gotchas. (There are a few 155exceptions. Firstly, some differences in L<quotemeta|perlfunc/quotemeta> 156were fixed starting in Perl 5.16.0. Secondly, some differences in 157L<the range operator|perlop/Range Operators> were fixed starting in 158Perl 5.26.0. Thirdly, some differences in L<split|perlfunc/split> were fixed 159started in Perl 5.28.0.) 160 161To enable this 162seamless support, you should C<use feature 'unicode_strings'> (which is 163automatically selected if you C<use v5.12> or higher). See L<feature>. 164(5.14 also fixes a number of bugs and departures from the Unicode 165standard.) 166 167Before Perl v5.8.0, the use of C<use utf8> was used to declare 168that operations in the current block or file would be Unicode-aware. 169This model was found to be wrong, or at least clumsy: the "Unicodeness" 170is now carried with the data, instead of being attached to the 171operations. 172Starting with Perl v5.8.0, only one case remains where an explicit C<use 173utf8> is needed: if your Perl script itself is encoded in UTF-8, you can 174use UTF-8 in your identifier names, and in string and regular expression 175literals, by saying C<use utf8>. This is not the default because 176scripts with legacy 8-bit data in them would break. See L<utf8>. 177 178=head2 Perl's Unicode Model 179 180Perl supports both pre-5.6 strings of eight-bit native bytes, and 181strings of Unicode characters. The general principle is that Perl tries 182to keep its data as eight-bit bytes for as long as possible, but as soon 183as Unicodeness cannot be avoided, the data is transparently upgraded 184to Unicode. Prior to Perl v5.14.0, the upgrade was not completely 185transparent (see L<perlunicode/The "Unicode Bug">), and for backwards 186compatibility, full transparency is not gained unless C<use feature 187'unicode_strings'> (see L<feature>) or C<use v5.12> (or higher) is 188selected. 189 190Internally, Perl currently uses either whatever the native eight-bit 191character set of the platform (for example Latin-1) is, defaulting to 192UTF-8, to encode Unicode strings. Specifically, if all code points in 193the string are C<0xFF> or less, Perl uses the native eight-bit 194character set. Otherwise, it uses UTF-8. 195 196A user of Perl does not normally need to know nor care how Perl 197happens to encode its internal strings, but it becomes relevant when 198outputting Unicode strings to a stream without a PerlIO layer (one with 199the "default" encoding). In such a case, the raw bytes used internally 200(the native character set or UTF-8, as appropriate for each string) 201will be used, and a "Wide character" warning will be issued if those 202strings contain a character beyond 0x00FF. 203 204For example, 205 206 perl -e 'print "\x{DF}\n", "\x{0100}\x{DF}\n"' 207 208produces a fairly useless mixture of native bytes and UTF-8, as well 209as a warning: 210 211 Wide character in print at ... 212 213To output UTF-8, use the C<:encoding> or C<:utf8> output layer. Prepending 214 215 binmode(STDOUT, ":utf8"); 216 217to this sample program ensures that the output is completely UTF-8, 218and removes the program's warning. 219 220You can enable automatic UTF-8-ification of your standard file 221handles, default C<open()> layer, and C<@ARGV> by using either 222the C<-C> command line switch or the C<PERL_UNICODE> environment 223variable, see L<perlrun|perlrun/-C [numberE<sol>list]> for the 224documentation of the C<-C> switch. 225 226Note that this means that Perl expects other software to work the same 227way: 228if Perl has been led to believe that STDIN should be UTF-8, but then 229STDIN coming in from another command is not UTF-8, Perl will likely 230complain about the malformed UTF-8. 231 232All features that combine Unicode and I/O also require using the new 233PerlIO feature. Almost all Perl 5.8 platforms do use PerlIO, though: 234you can see whether yours is by running "perl -V" and looking for 235C<useperlio=define>. 236 237=head2 Unicode and EBCDIC 238 239Perl 5.8.0 added support for Unicode on EBCDIC platforms. This support 240was allowed to lapse in later releases, but was revived in 5.22. 241Unicode support is somewhat more complex to implement since additional 242conversions are needed. See L<perlebcdic> for more information. 243 244On EBCDIC platforms, the internal Unicode encoding form is UTF-EBCDIC 245instead of UTF-8. The difference is that as UTF-8 is "ASCII-safe" in 246that ASCII characters encode to UTF-8 as-is, while UTF-EBCDIC is 247"EBCDIC-safe", in that all the basic characters (which includes all 248those that have ASCII equivalents (like C<"A">, C<"0">, C<"%">, I<etc.>) 249are the same in both EBCDIC and UTF-EBCDIC. Often, documentation 250will use the term "UTF-8" to mean UTF-EBCDIC as well. This is the case 251in this document. 252 253=head2 Creating Unicode 254 255This section applies fully to Perls starting with v5.22. Various 256caveats for earlier releases are in the L</Earlier releases caveats> 257subsection below. 258 259To create Unicode characters in literals, 260use the C<\N{...}> notation in double-quoted strings: 261 262 my $smiley_from_name = "\N{WHITE SMILING FACE}"; 263 my $smiley_from_code_point = "\N{U+263a}"; 264 265Similarly, they can be used in regular expression literals 266 267 $smiley =~ /\N{WHITE SMILING FACE}/; 268 $smiley =~ /\N{U+263a}/; 269 270or, starting in v5.32: 271 272 $smiley =~ /\p{Name=WHITE SMILING FACE}/; 273 $smiley =~ /\p{Name=whitesmilingface}/; 274 275At run-time you can use: 276 277 use charnames (); 278 my $hebrew_alef_from_name 279 = charnames::string_vianame("HEBREW LETTER ALEF"); 280 my $hebrew_alef_from_code_point = charnames::string_vianame("U+05D0"); 281 282Naturally, C<ord()> will do the reverse: it turns a character into 283a code point. 284 285There are other runtime options as well. You can use C<pack()>: 286 287 my $hebrew_alef_from_code_point = pack("U", 0x05d0); 288 289Or you can use C<chr()>, though it is less convenient in the general 290case: 291 292 $hebrew_alef_from_code_point = chr(utf8::unicode_to_native(0x05d0)); 293 utf8::upgrade($hebrew_alef_from_code_point); 294 295The C<utf8::unicode_to_native()> and C<utf8::upgrade()> aren't needed if 296the argument is above 0xFF, so the above could have been written as 297 298 $hebrew_alef_from_code_point = chr(0x05d0); 299 300since 0x5d0 is above 255. 301 302C<\x{}> and C<\o{}> can also be used to specify code points at compile 303time in double-quotish strings, but, for backward compatibility with 304older Perls, the same rules apply as with C<chr()> for code points less 305than 256. 306 307C<utf8::unicode_to_native()> is used so that the Perl code is portable 308to EBCDIC platforms. You can omit it if you're I<really> sure no one 309will ever want to use your code on a non-ASCII platform. Starting in 310Perl v5.22, calls to it on ASCII platforms are optimized out, so there's 311no performance penalty at all in adding it. Or you can simply use the 312other constructs that don't require it. 313 314See L</"Further Resources"> for how to find all these names and numeric 315codes. 316 317=head3 Earlier releases caveats 318 319On EBCDIC platforms, prior to v5.22, using C<\N{U+...}> doesn't work 320properly. 321 322Prior to v5.16, using C<\N{...}> with a character name (as opposed to a 323C<U+...> code point) required a S<C<use charnames :full>>. 324 325Prior to v5.14, there were some bugs in C<\N{...}> with a character name 326(as opposed to a C<U+...> code point). 327 328C<charnames::string_vianame()> was introduced in v5.14. Prior to that, 329C<charnames::vianame()> should work, but only if the argument is of the 330form C<"U+...">. Your best bet there for runtime Unicode by character 331name is probably: 332 333 use charnames (); 334 my $hebrew_alef_from_name 335 = pack("U", charnames::vianame("HEBREW LETTER ALEF")); 336 337=head2 Handling Unicode 338 339Handling Unicode is for the most part transparent: just use the 340strings as usual. Functions like C<index()>, C<length()>, and 341C<substr()> will work on the Unicode characters; regular expressions 342will work on the Unicode characters (see L<perlunicode> and L<perlretut>). 343 344Note that Perl considers grapheme clusters to be separate characters, so for 345example 346 347 print length("\N{LATIN CAPITAL LETTER A}\N{COMBINING ACUTE ACCENT}"), 348 "\n"; 349 350will print 2, not 1. The only exception is that regular expressions 351have C<\X> for matching an extended grapheme cluster. (Thus C<\X> in a 352regular expression would match the entire sequence of both the example 353characters.) 354 355Life is not quite so transparent, however, when working with legacy 356encodings, I/O, and certain special cases: 357 358=head2 Legacy Encodings 359 360When you combine legacy data and Unicode, the legacy data needs 361to be upgraded to Unicode. Normally the legacy data is assumed to be 362ISO 8859-1 (or EBCDIC, if applicable). 363 364The C<Encode> module knows about many encodings and has interfaces 365for doing conversions between those encodings: 366 367 use Encode 'decode'; 368 $data = decode("iso-8859-3", $data); # convert from legacy 369 370=head2 Unicode I/O 371 372Normally, writing out Unicode data 373 374 print FH $some_string_with_unicode, "\n"; 375 376produces raw bytes that Perl happens to use to internally encode the 377Unicode string. Perl's internal encoding depends on the system as 378well as what characters happen to be in the string at the time. If 379any of the characters are at code points C<0x100> or above, you will get 380a warning. To ensure that the output is explicitly rendered in the 381encoding you desire--and to avoid the warning--open the stream with 382the desired encoding. Some examples: 383 384 open FH, ">:utf8", "file"; 385 386 open FH, ">:encoding(ucs2)", "file"; 387 open FH, ">:encoding(UTF-8)", "file"; 388 open FH, ">:encoding(shift_jis)", "file"; 389 390and on already open streams, use C<binmode()>: 391 392 binmode(STDOUT, ":utf8"); 393 394 binmode(STDOUT, ":encoding(ucs2)"); 395 binmode(STDOUT, ":encoding(UTF-8)"); 396 binmode(STDOUT, ":encoding(shift_jis)"); 397 398The matching of encoding names is loose: case does not matter, and 399many encodings have several aliases. Note that the C<:utf8> layer 400must always be specified exactly like that; it is I<not> subject to 401the loose matching of encoding names. Also note that currently C<:utf8> is unsafe for 402input, because it accepts the data without validating that it is indeed valid 403UTF-8; you should instead use C<:encoding(UTF-8)> (with or without a 404hyphen). 405 406See L<PerlIO> for the C<:utf8> layer, L<PerlIO::encoding> and 407L<Encode::PerlIO> for the C<:encoding()> layer, and 408L<Encode::Supported> for many encodings supported by the C<Encode> 409module. 410 411Reading in a file that you know happens to be encoded in one of the 412Unicode or legacy encodings does not magically turn the data into 413Unicode in Perl's eyes. To do that, specify the appropriate 414layer when opening files 415 416 open(my $fh,'<:encoding(UTF-8)', 'anything'); 417 my $line_of_unicode = <$fh>; 418 419 open(my $fh,'<:encoding(Big5)', 'anything'); 420 my $line_of_unicode = <$fh>; 421 422The I/O layers can also be specified more flexibly with 423the C<open> pragma. See L<open>, or look at the following example. 424 425 use open ':encoding(UTF-8)'; # input/output default encoding will be 426 # UTF-8 427 open X, ">file"; 428 print X chr(0x100), "\n"; 429 close X; 430 open Y, "<file"; 431 printf "%#x\n", ord(<Y>); # this should print 0x100 432 close Y; 433 434With the C<open> pragma you can use the C<:locale> layer 435 436 BEGIN { $ENV{LC_ALL} = $ENV{LANG} = 'ru_RU.KOI8-R' } 437 # the :locale will probe the locale environment variables like 438 # LC_ALL 439 use open OUT => ':locale'; # russki parusski 440 open(O, ">koi8"); 441 print O chr(0x430); # Unicode CYRILLIC SMALL LETTER A = KOI8-R 0xc1 442 close O; 443 open(I, "<koi8"); 444 printf "%#x\n", ord(<I>), "\n"; # this should print 0xc1 445 close I; 446 447These methods install a transparent filter on the I/O stream that 448converts data from the specified encoding when it is read in from the 449stream. The result is always Unicode. 450 451The L<open> pragma affects all the C<open()> calls after the pragma by 452setting default layers. If you want to affect only certain 453streams, use explicit layers directly in the C<open()> call. 454 455You can switch encodings on an already opened stream by using 456C<binmode()>; see L<perlfunc/binmode>. 457 458The C<:locale> does not currently work with 459C<open()> and C<binmode()>, only with the C<open> pragma. The 460C<:utf8> and C<:encoding(...)> methods do work with all of C<open()>, 461C<binmode()>, and the C<open> pragma. 462 463Similarly, you may use these I/O layers on output streams to 464automatically convert Unicode to the specified encoding when it is 465written to the stream. For example, the following snippet copies the 466contents of the file "text.jis" (encoded as ISO-2022-JP, aka JIS) to 467the file "text.utf8", encoded as UTF-8: 468 469 open(my $nihongo, '<:encoding(iso-2022-jp)', 'text.jis'); 470 open(my $unicode, '>:utf8', 'text.utf8'); 471 while (<$nihongo>) { print $unicode $_ } 472 473The naming of encodings, both by the C<open()> and by the C<open> 474pragma allows for flexible names: C<koi8-r> and C<KOI8R> will both be 475understood. 476 477Common encodings recognized by ISO, MIME, IANA, and various other 478standardisation organisations are recognised; for a more detailed 479list see L<Encode::Supported>. 480 481C<read()> reads characters and returns the number of characters. 482C<seek()> and C<tell()> operate on byte counts, as does C<sysseek()>. 483 484C<sysread()> and C<syswrite()> should not be used on file handles with 485character encoding layers, they behave badly, and that behaviour has 486been deprecated since perl 5.24. 487 488Notice that because of the default behaviour of not doing any 489conversion upon input if there is no default layer, 490it is easy to mistakenly write code that keeps on expanding a file 491by repeatedly encoding the data: 492 493 # BAD CODE WARNING 494 open F, "file"; 495 local $/; ## read in the whole file of 8-bit characters 496 $t = <F>; 497 close F; 498 open F, ">:encoding(UTF-8)", "file"; 499 print F $t; ## convert to UTF-8 on output 500 close F; 501 502If you run this code twice, the contents of the F<file> will be twice 503UTF-8 encoded. A C<use open ':encoding(UTF-8)'> would have avoided the 504bug, or explicitly opening also the F<file> for input as UTF-8. 505 506B<NOTE>: the C<:utf8> and C<:encoding> features work only if your 507Perl has been built with L<PerlIO>, which is the default 508on most systems. 509 510=head2 Displaying Unicode As Text 511 512Sometimes you might want to display Perl scalars containing Unicode as 513simple ASCII (or EBCDIC) text. The following subroutine converts 514its argument so that Unicode characters with code points greater than 515255 are displayed as C<\x{...}>, control characters (like C<\n>) are 516displayed as C<\x..>, and the rest of the characters as themselves: 517 518 sub nice_string { 519 join("", 520 map { $_ > 255 # if wide character... 521 ? sprintf("\\x{%04X}", $_) # \x{...} 522 : chr($_) =~ /[[:cntrl:]]/ # else if control character... 523 ? sprintf("\\x%02X", $_) # \x.. 524 : quotemeta(chr($_)) # else quoted or as themselves 525 } unpack("W*", $_[0])); # unpack Unicode characters 526 } 527 528For example, 529 530 nice_string("foo\x{100}bar\n") 531 532returns the string 533 534 'foo\x{0100}bar\x0A' 535 536which is ready to be printed. 537 538(C<\\x{}> is used here instead of C<\\N{}>, since it's most likely that 539you want to see what the native values are.) 540 541=head2 Special Cases 542 543=over 4 544 545=item * 546 547Starting in Perl 5.28, it is illegal for bit operators, like C<~>, to 548operate on strings containing code points above 255. 549 550=item * 551 552The vec() function may produce surprising results if 553used on strings containing characters with ordinal values above 554255. In such a case, the results are consistent with the internal 555encoding of the characters, but not with much else. So don't do 556that, and starting in Perl 5.28, a deprecation message is issued if you 557do so, becoming illegal in Perl 5.32. 558 559=item * 560 561Peeking At Perl's Internal Encoding 562 563Normal users of Perl should never care how Perl encodes any particular 564Unicode string (because the normal ways to get at the contents of a 565string with Unicode--via input and output--should always be via 566explicitly-defined I/O layers). But if you must, there are two 567ways of looking behind the scenes. 568 569One way of peeking inside the internal encoding of Unicode characters 570is to use C<unpack("C*", ...> to get the bytes of whatever the string 571encoding happens to be, or C<unpack("U0..", ...)> to get the bytes of the 572UTF-8 encoding: 573 574 # this prints c4 80 for the UTF-8 bytes 0xc4 0x80 575 print join(" ", unpack("U0(H2)*", pack("U", 0x100))), "\n"; 576 577Yet another way would be to use the Devel::Peek module: 578 579 perl -MDevel::Peek -e 'Dump(chr(0x100))' 580 581That shows the C<UTF8> flag in FLAGS and both the UTF-8 bytes 582and Unicode characters in C<PV>. See also later in this document 583the discussion about the C<utf8::is_utf8()> function. 584 585=back 586 587=head2 Advanced Topics 588 589=over 4 590 591=item * 592 593String Equivalence 594 595The question of string equivalence turns somewhat complicated 596in Unicode: what do you mean by "equal"? 597 598(Is C<LATIN CAPITAL LETTER A WITH ACUTE> equal to 599C<LATIN CAPITAL LETTER A>?) 600 601The short answer is that by default Perl compares equivalence (C<eq>, 602C<ne>) based only on code points of the characters. In the above 603case, the answer is no (because 0x00C1 != 0x0041). But sometimes, any 604CAPITAL LETTER A's should be considered equal, or even A's of any case. 605 606The long answer is that you need to consider character normalization 607and casing issues: see L<Unicode::Normalize>, Unicode Technical Report #15, 608L<Unicode Normalization Forms|https://www.unicode.org/reports/tr15> and 609sections on case mapping in the L<Unicode Standard|https://www.unicode.org>. 610 611As of Perl 5.8.0, the "Full" case-folding of I<Case 612Mappings/SpecialCasing> is implemented, but bugs remain in C<qr//i> with them, 613mostly fixed by 5.14, and essentially entirely by 5.18. 614 615=item * 616 617String Collation 618 619People like to see their strings nicely sorted--or as Unicode 620parlance goes, collated. But again, what do you mean by collate? 621 622(Does C<LATIN CAPITAL LETTER A WITH ACUTE> come before or after 623C<LATIN CAPITAL LETTER A WITH GRAVE>?) 624 625The short answer is that by default, Perl compares strings (C<lt>, 626C<le>, C<cmp>, C<ge>, C<gt>) based only on the code points of the 627characters. In the above case, the answer is "after", since 628C<0x00C1> > C<0x00C0>. 629 630The long answer is that "it depends", and a good answer cannot be 631given without knowing (at the very least) the language context. 632See L<Unicode::Collate>, and I<Unicode Collation Algorithm> 633L<https://www.unicode.org/reports/tr10/> 634 635=back 636 637=head2 Miscellaneous 638 639=over 4 640 641=item * 642 643Character Ranges and Classes 644 645Character ranges in regular expression bracketed character classes ( e.g., 646C</[a-z]/>) and in the C<tr///> (also known as C<y///>) operator are not 647magically Unicode-aware. What this means is that C<[A-Za-z]> will not 648magically start to mean "all alphabetic letters" (not that it does mean that 649even for 8-bit characters; for those, if you are using locales (L<perllocale>), 650use C</[[:alpha:]]/>; and if not, use the 8-bit-aware property C<\p{alpha}>). 651 652All the properties that begin with C<\p> (and its inverse C<\P>) are actually 653character classes that are Unicode-aware. There are dozens of them, see 654L<perluniprops>. 655 656Starting in v5.22, you can use Unicode code points as the end points of 657regular expression pattern character ranges, and the range will include 658all Unicode code points that lie between those end points, inclusive. 659 660 qr/ [ \N{U+03} - \N{U+20} ] /xx 661 662includes the code points 663C<\N{U+03}>, C<\N{U+04}>, ..., C<\N{U+20}>. 664 665This also works for ranges in C<tr///> starting in Perl v5.24. 666 667=item * 668 669String-To-Number Conversions 670 671Unicode does define several other decimal--and numeric--characters 672besides the familiar 0 to 9, such as the Arabic and Indic digits. 673Perl does not support string-to-number conversion for digits other 674than ASCII C<0> to C<9> (and ASCII C<a> to C<f> for hexadecimal). 675To get safe conversions from any Unicode string, use 676L<Unicode::UCD/num()>. 677 678=back 679 680=head2 Questions With Answers 681 682=over 4 683 684=item * 685 686Will My Old Scripts Break? 687 688Very probably not. Unless you are generating Unicode characters 689somehow, old behaviour should be preserved. About the only behaviour 690that has changed and which could start generating Unicode is the old 691behaviour of C<chr()> where supplying an argument more than 255 692produced a character modulo 255. C<chr(300)>, for example, was equal 693to C<chr(45)> or "-" (in ASCII), now it is LATIN CAPITAL LETTER I WITH 694BREVE. 695 696=item * 697 698How Do I Make My Scripts Work With Unicode? 699 700Very little work should be needed since nothing changes until you 701generate Unicode data. The most important thing is getting input as 702Unicode; for that, see the earlier I/O discussion. 703To get full seamless Unicode support, add 704C<use feature 'unicode_strings'> (or C<use v5.12> or higher) to your 705script. 706 707=item * 708 709How Do I Know Whether My String Is In Unicode? 710 711You shouldn't have to care. But you may if your Perl is before 5.14.0 712or you haven't specified C<use feature 'unicode_strings'> or C<use 7135.012> (or higher) because otherwise the rules for the code points 714in the range 128 to 255 are different depending on 715whether the string they are contained within is in Unicode or not. 716(See L<perlunicode/When Unicode Does Not Happen>.) 717 718To determine if a string is in Unicode, use: 719 720 print utf8::is_utf8($string) ? 1 : 0, "\n"; 721 722But note that this doesn't mean that any of the characters in the 723string are necessary UTF-8 encoded, or that any of the characters have 724code points greater than 0xFF (255) or even 0x80 (128), or that the 725string has any characters at all. All the C<is_utf8()> does is to 726return the value of the internal "utf8ness" flag attached to the 727C<$string>. If the flag is off, the bytes in the scalar are interpreted 728as a single byte encoding. If the flag is on, the bytes in the scalar 729are interpreted as the (variable-length, potentially multi-byte) UTF-8 encoded 730code points of the characters. Bytes added to a UTF-8 encoded string are 731automatically upgraded to UTF-8. If mixed non-UTF-8 and UTF-8 scalars 732are merged (double-quoted interpolation, explicit concatenation, or 733printf/sprintf parameter substitution), the result will be UTF-8 encoded 734as if copies of the byte strings were upgraded to UTF-8: for example, 735 736 $a = "ab\x80c"; 737 $b = "\x{100}"; 738 print "$a = $b\n"; 739 740the output string will be UTF-8-encoded C<ab\x80c = \x{100}\n>, but 741C<$a> will stay byte-encoded. 742 743Sometimes you might really need to know the byte length of a string 744instead of the character length. For that use the C<bytes> pragma 745and the C<length()> function: 746 747 my $unicode = chr(0x100); 748 print length($unicode), "\n"; # will print 1 749 use bytes; 750 print length($unicode), "\n"; # will print 2 751 # (the 0xC4 0x80 of the UTF-8) 752 no bytes; 753 754=item * 755 756How Do I Find Out What Encoding a File Has? 757 758You might try L<Encode::Guess>, but it has a number of limitations. 759 760=item * 761 762How Do I Detect Data That's Not Valid In a Particular Encoding? 763 764Use the C<Encode> package to try converting it. 765For example, 766 767 use Encode 'decode'; 768 769 if (eval { decode('UTF-8', $string, Encode::FB_CROAK); 1 }) { 770 # $string is valid UTF-8 771 } else { 772 # $string is not valid UTF-8 773 } 774 775Or use C<unpack> to try decoding it: 776 777 use warnings; 778 @chars = unpack("C0U*", $string_of_bytes_that_I_think_is_utf8); 779 780If invalid, a C<Malformed UTF-8 character> warning is produced. The "C0" means 781"process the string character per character". Without that, the 782C<unpack("U*", ...)> would work in C<U0> mode (the default if the format 783string starts with C<U>) and it would return the bytes making up the UTF-8 784encoding of the target string, something that will always work. 785 786=item * 787 788How Do I Convert Binary Data Into a Particular Encoding, Or Vice Versa? 789 790This probably isn't as useful as you might think. 791Normally, you shouldn't need to. 792 793In one sense, what you are asking doesn't make much sense: encodings 794are for characters, and binary data are not "characters", so converting 795"data" into some encoding isn't meaningful unless you know in what 796character set and encoding the binary data is in, in which case it's 797not just binary data, now is it? 798 799If you have a raw sequence of bytes that you know should be 800interpreted via a particular encoding, you can use C<Encode>: 801 802 use Encode 'from_to'; 803 from_to($data, "iso-8859-1", "UTF-8"); # from latin-1 to UTF-8 804 805The call to C<from_to()> changes the bytes in C<$data>, but nothing 806material about the nature of the string has changed as far as Perl is 807concerned. Both before and after the call, the string C<$data> 808contains just a bunch of 8-bit bytes. As far as Perl is concerned, 809the encoding of the string remains as "system-native 8-bit bytes". 810 811You might relate this to a fictional 'Translate' module: 812 813 use Translate; 814 my $phrase = "Yes"; 815 Translate::from_to($phrase, 'english', 'deutsch'); 816 ## phrase now contains "Ja" 817 818The contents of the string changes, but not the nature of the string. 819Perl doesn't know any more after the call than before that the 820contents of the string indicates the affirmative. 821 822Back to converting data. If you have (or want) data in your system's 823native 8-bit encoding (e.g. Latin-1, EBCDIC, etc.), you can use 824pack/unpack to convert to/from Unicode. 825 826 $native_string = pack("W*", unpack("U*", $Unicode_string)); 827 $Unicode_string = pack("U*", unpack("W*", $native_string)); 828 829If you have a sequence of bytes you B<know> is valid UTF-8, 830but Perl doesn't know it yet, you can make Perl a believer, too: 831 832 $Unicode = $bytes; 833 utf8::decode($Unicode); 834 835or: 836 837 $Unicode = pack("U0a*", $bytes); 838 839You can find the bytes that make up a UTF-8 sequence with 840 841 @bytes = unpack("C*", $Unicode_string) 842 843and you can create well-formed Unicode with 844 845 $Unicode_string = pack("U*", 0xff, ...) 846 847=item * 848 849How Do I Display Unicode? How Do I Input Unicode? 850 851See L<http://www.alanwood.net/unicode/> and 852L<http://www.cl.cam.ac.uk/~mgk25/unicode.html> 853 854=item * 855 856How Does Unicode Work With Traditional Locales? 857 858If your locale is a UTF-8 locale, starting in Perl v5.26, Perl works 859well for all categories; before this, starting with Perl v5.20, it works 860for all categories but C<LC_COLLATE>, which deals with 861sorting and the C<cmp> operator. But note that the standard 862C<L<Unicode::Collate>> and C<L<Unicode::Collate::Locale>> modules offer 863much more powerful solutions to collation issues, and work on earlier 864releases. 865 866For other locales, starting in Perl 5.16, you can specify 867 868 use locale ':not_characters'; 869 870to get Perl to work well with them. The catch is that you 871have to translate from the locale character set to/from Unicode 872yourself. See L</Unicode IE<sol>O> above for how to 873 874 use open ':locale'; 875 876to accomplish this, but full details are in L<perllocale/Unicode and 877UTF-8>, including gotchas that happen if you don't specify 878C<:not_characters>. 879 880=back 881 882=head2 Hexadecimal Notation 883 884The Unicode standard prefers using hexadecimal notation because 885that more clearly shows the division of Unicode into blocks of 256 characters. 886Hexadecimal is also simply shorter than decimal. You can use decimal 887notation, too, but learning to use hexadecimal just makes life easier 888with the Unicode standard. The C<U+HHHH> notation uses hexadecimal, 889for example. 890 891The C<0x> prefix means a hexadecimal number, the digits are 0-9 I<and> 892a-f (or A-F, case doesn't matter). Each hexadecimal digit represents 893four bits, or half a byte. C<print 0x..., "\n"> will show a 894hexadecimal number in decimal, and C<printf "%x\n", $decimal> will 895show a decimal number in hexadecimal. If you have just the 896"hex digits" of a hexadecimal number, you can use the C<hex()> function. 897 898 print 0x0009, "\n"; # 9 899 print 0x000a, "\n"; # 10 900 print 0x000f, "\n"; # 15 901 print 0x0010, "\n"; # 16 902 print 0x0011, "\n"; # 17 903 print 0x0100, "\n"; # 256 904 905 print 0x0041, "\n"; # 65 906 907 printf "%x\n", 65; # 41 908 printf "%#x\n", 65; # 0x41 909 910 print hex("41"), "\n"; # 65 911 912=head2 Further Resources 913 914=over 4 915 916=item * 917 918Unicode Consortium 919 920L<https://www.unicode.org/> 921 922=item * 923 924Unicode FAQ 925 926L<https://www.unicode.org/faq/> 927 928=item * 929 930Unicode Glossary 931 932L<https://www.unicode.org/glossary/> 933 934=item * 935 936Unicode Recommended Reading List 937 938The Unicode Consortium has a list of articles and books, some of which 939give a much more in depth treatment of Unicode: 940L<http://unicode.org/resources/readinglist.html> 941 942=item * 943 944Unicode Useful Resources 945 946L<https://www.unicode.org/unicode/onlinedat/resources.html> 947 948=item * 949 950Unicode and Multilingual Support in HTML, Fonts, Web Browsers and Other Applications 951 952L<http://www.alanwood.net/unicode/> 953 954=item * 955 956UTF-8 and Unicode FAQ for Unix/Linux 957 958L<http://www.cl.cam.ac.uk/~mgk25/unicode.html> 959 960=item * 961 962Legacy Character Sets 963 964L<http://www.czyborra.com/> 965L<http://www.eki.ee/letter/> 966 967=item * 968 969You can explore various information from the Unicode data files using 970the C<Unicode::UCD> module. 971 972=back 973 974=head1 UNICODE IN OLDER PERLS 975 976If you cannot upgrade your Perl to 5.8.0 or later, you can still 977do some Unicode processing by using the modules C<Unicode::String>, 978C<Unicode::Map8>, and C<Unicode::Map>, available from CPAN. 979If you have the GNU recode installed, you can also use the 980Perl front-end C<Convert::Recode> for character conversions. 981 982The following are fast conversions from ISO 8859-1 (Latin-1) bytes 983to UTF-8 bytes and back, the code works even with older Perl 5 versions. 984 985 # ISO 8859-1 to UTF-8 986 s/([\x80-\xFF])/chr(0xC0|ord($1)>>6).chr(0x80|ord($1)&0x3F)/eg; 987 988 # UTF-8 to ISO 8859-1 989 s/([\xC2\xC3])([\x80-\xBF])/chr(ord($1)<<6&0xC0|ord($2)&0x3F)/eg; 990 991=head1 SEE ALSO 992 993L<perlunitut>, L<perlunicode>, L<Encode>, L<open>, L<utf8>, L<bytes>, 994L<perlretut>, L<perlrun>, L<Unicode::Collate>, L<Unicode::Normalize>, 995L<Unicode::UCD> 996 997=head1 ACKNOWLEDGMENTS 998 999Thanks to the kind readers of the perl5-porters@perl.org, 1000perl-unicode@perl.org, linux-utf8@nl.linux.org, and unicore@unicode.org 1001mailing lists for their valuable feedback. 1002 1003=head1 AUTHOR, COPYRIGHT, AND LICENSE 1004 1005Copyright 2001-2011 Jarkko Hietaniemi E<lt>jhi@iki.fiE<gt>. 1006Now maintained by Perl 5 Porters. 1007 1008This document may be distributed under the same terms as Perl itself. 1009