1<html> 2<head> 3<title>pcrepartial specification</title> 4</head> 5<body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB"> 6<h1>pcrepartial man page</h1> 7<p> 8Return to the <a href="index.html">PCRE index page</a>. 9</p> 10<p> 11This page is part of the PCRE HTML documentation. It was generated automatically 12from the original man page. If there is any nonsense in it, please consult the 13man page, in case the conversion went wrong. 14<br> 15<ul> 16<li><a name="TOC1" href="#SEC1">PARTIAL MATCHING IN PCRE</a> 17<li><a name="TOC2" href="#SEC2">PARTIAL MATCHING USING pcre_exec() OR pcre16_exec()</a> 18<li><a name="TOC3" href="#SEC3">PARTIAL MATCHING USING pcre_dfa_exec() OR pcre16_dfa_exec()</a> 19<li><a name="TOC4" href="#SEC4">PARTIAL MATCHING AND WORD BOUNDARIES</a> 20<li><a name="TOC5" href="#SEC5">FORMERLY RESTRICTED PATTERNS</a> 21<li><a name="TOC6" href="#SEC6">EXAMPLE OF PARTIAL MATCHING USING PCRETEST</a> 22<li><a name="TOC7" href="#SEC7">MULTI-SEGMENT MATCHING WITH pcre_dfa_exec() OR pcre16_dfa_exec()</a> 23<li><a name="TOC8" href="#SEC8">MULTI-SEGMENT MATCHING WITH pcre_exec() OR pcre16_exec()</a> 24<li><a name="TOC9" href="#SEC9">ISSUES WITH MULTI-SEGMENT MATCHING</a> 25<li><a name="TOC10" href="#SEC10">AUTHOR</a> 26<li><a name="TOC11" href="#SEC11">REVISION</a> 27</ul> 28<br><a name="SEC1" href="#TOC1">PARTIAL MATCHING IN PCRE</a><br> 29<P> 30In normal use of PCRE, if the subject string that is passed to a matching 31function matches as far as it goes, but is too short to match the entire 32pattern, PCRE_ERROR_NOMATCH is returned. There are circumstances where it might 33be helpful to distinguish this case from other cases in which there is no 34match. 35</P> 36<P> 37Consider, for example, an application where a human is required to type in data 38for a field with specific formatting requirements. An example might be a date 39in the form <i>ddmmmyy</i>, defined by this pattern: 40<pre> 41 ^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$ 42</pre> 43If the application sees the user's keystrokes one by one, and can check that 44what has been typed so far is potentially valid, it is able to raise an error 45as soon as a mistake is made, by beeping and not reflecting the character that 46has been typed, for example. This immediate feedback is likely to be a better 47user interface than a check that is delayed until the entire string has been 48entered. Partial matching can also be useful when the subject string is very 49long and is not all available at once. 50</P> 51<P> 52PCRE supports partial matching by means of the PCRE_PARTIAL_SOFT and 53PCRE_PARTIAL_HARD options, which can be set when calling any of the matching 54functions. For backwards compatibility, PCRE_PARTIAL is a synonym for 55PCRE_PARTIAL_SOFT. The essential difference between the two options is whether 56or not a partial match is preferred to an alternative complete match, though 57the details differ between the two types of matching function. If both options 58are set, PCRE_PARTIAL_HARD takes precedence. 59</P> 60<P> 61If you want to use partial matching with just-in-time optimized code, you must 62call <b>pcre_study()</b> or <b>pcre16_study()</b> with one or both of these 63options: 64<pre> 65 PCRE_STUDY_JIT_PARTIAL_SOFT_COMPILE 66 PCRE_STUDY_JIT_PARTIAL_HARD_COMPILE 67</pre> 68PCRE_STUDY_JIT_COMPILE should also be set if you are going to run non-partial 69matches on the same pattern. If the appropriate JIT study mode has not been set 70for a match, the interpretive matching code is used. 71</P> 72<P> 73Setting a partial matching option disables two of PCRE's standard 74optimizations. PCRE remembers the last literal data unit in a pattern, and 75abandons matching immediately if it is not present in the subject string. This 76optimization cannot be used for a subject string that might match only 77partially. If the pattern was studied, PCRE knows the minimum length of a 78matching string, and does not bother to run the matching function on shorter 79strings. This optimization is also disabled for partial matching. 80</P> 81<br><a name="SEC2" href="#TOC1">PARTIAL MATCHING USING pcre_exec() OR pcre16_exec()</a><br> 82<P> 83A partial match occurs during a call to <b>pcre_exec()</b> or 84<b>pcre16_exec()</b> when the end of the subject string is reached successfully, 85but matching cannot continue because more characters are needed. However, at 86least one character in the subject must have been inspected. This character 87need not form part of the final matched string; lookbehind assertions and the 88\K escape sequence provide ways of inspecting characters before the start of a 89matched substring. The requirement for inspecting at least one character exists 90because an empty string can always be matched; without such a restriction there 91would always be a partial match of an empty string at the end of the subject. 92</P> 93<P> 94If there are at least two slots in the offsets vector when a partial match is 95returned, the first slot is set to the offset of the earliest character that 96was inspected. For convenience, the second offset points to the end of the 97subject so that a substring can easily be identified. 98</P> 99<P> 100For the majority of patterns, the first offset identifies the start of the 101partially matched string. However, for patterns that contain lookbehind 102assertions, or \K, or begin with \b or \B, earlier characters have been 103inspected while carrying out the match. For example: 104<pre> 105 /(?<=abc)123/ 106</pre> 107This pattern matches "123", but only if it is preceded by "abc". If the subject 108string is "xyzabc12", the offsets after a partial match are for the substring 109"abc12", because all these characters are needed if another match is tried 110with extra characters added to the subject. 111</P> 112<P> 113What happens when a partial match is identified depends on which of the two 114partial matching options are set. 115</P> 116<br><b> 117PCRE_PARTIAL_SOFT WITH pcre_exec() OR pcre16_exec() 118</b><br> 119<P> 120If PCRE_PARTIAL_SOFT is set when <b>pcre_exec()</b> or <b>pcre16_exec()</b> 121identifies a partial match, the partial match is remembered, but matching 122continues as normal, and other alternatives in the pattern are tried. If no 123complete match can be found, PCRE_ERROR_PARTIAL is returned instead of 124PCRE_ERROR_NOMATCH. 125</P> 126<P> 127This option is "soft" because it prefers a complete match over a partial match. 128All the various matching items in a pattern behave as if the subject string is 129potentially complete. For example, \z, \Z, and $ match at the end of the 130subject, as normal, and for \b and \B the end of the subject is treated as a 131non-alphanumeric. 132</P> 133<P> 134If there is more than one partial match, the first one that was found provides 135the data that is returned. Consider this pattern: 136<pre> 137 /123\w+X|dogY/ 138</pre> 139If this is matched against the subject string "abc123dog", both 140alternatives fail to match, but the end of the subject is reached during 141matching, so PCRE_ERROR_PARTIAL is returned. The offsets are set to 3 and 9, 142identifying "123dog" as the first partial match that was found. (In this 143example, there are two partial matches, because "dog" on its own partially 144matches the second alternative.) 145</P> 146<br><b> 147PCRE_PARTIAL_HARD WITH pcre_exec() OR pcre16_exec() 148</b><br> 149<P> 150If PCRE_PARTIAL_HARD is set for <b>pcre_exec()</b> or <b>pcre16_exec()</b>, 151PCRE_ERROR_PARTIAL is returned as soon as a partial match is found, without 152continuing to search for possible complete matches. This option is "hard" 153because it prefers an earlier partial match over a later complete match. For 154this reason, the assumption is made that the end of the supplied subject string 155may not be the true end of the available data, and so, if \z, \Z, \b, \B, 156or $ are encountered at the end of the subject, the result is 157PCRE_ERROR_PARTIAL, provided that at least one character in the subject has 158been inspected. 159</P> 160<P> 161Setting PCRE_PARTIAL_HARD also affects the way UTF-8 and UTF-16 162subject strings are checked for validity. Normally, an invalid sequence 163causes the error PCRE_ERROR_BADUTF8 or PCRE_ERROR_BADUTF16. However, in the 164special case of a truncated character at the end of the subject, 165PCRE_ERROR_SHORTUTF8 or PCRE_ERROR_SHORTUTF16 is returned when 166PCRE_PARTIAL_HARD is set. 167</P> 168<br><b> 169Comparing hard and soft partial matching 170</b><br> 171<P> 172The difference between the two partial matching options can be illustrated by a 173pattern such as: 174<pre> 175 /dog(sbody)?/ 176</pre> 177This matches either "dog" or "dogsbody", greedily (that is, it prefers the 178longer string if possible). If it is matched against the string "dog" with 179PCRE_PARTIAL_SOFT, it yields a complete match for "dog". However, if 180PCRE_PARTIAL_HARD is set, the result is PCRE_ERROR_PARTIAL. On the other hand, 181if the pattern is made ungreedy the result is different: 182<pre> 183 /dog(sbody)??/ 184</pre> 185In this case the result is always a complete match because that is found first, 186and matching never continues after finding a complete match. It might be easier 187to follow this explanation by thinking of the two patterns like this: 188<pre> 189 /dog(sbody)?/ is the same as /dogsbody|dog/ 190 /dog(sbody)??/ is the same as /dog|dogsbody/ 191</pre> 192The second pattern will never match "dogsbody", because it will always find the 193shorter match first. 194</P> 195<br><a name="SEC3" href="#TOC1">PARTIAL MATCHING USING pcre_dfa_exec() OR pcre16_dfa_exec()</a><br> 196<P> 197The DFA functions move along the subject string character by character, without 198backtracking, searching for all possible matches simultaneously. If the end of 199the subject is reached before the end of the pattern, there is the possibility 200of a partial match, again provided that at least one character has been 201inspected. 202</P> 203<P> 204When PCRE_PARTIAL_SOFT is set, PCRE_ERROR_PARTIAL is returned only if there 205have been no complete matches. Otherwise, the complete matches are returned. 206However, if PCRE_PARTIAL_HARD is set, a partial match takes precedence over any 207complete matches. The portion of the string that was inspected when the longest 208partial match was found is set as the first matching string, provided there are 209at least two slots in the offsets vector. 210</P> 211<P> 212Because the DFA functions always search for all possible matches, and there is 213no difference between greedy and ungreedy repetition, their behaviour is 214different from the standard functions when PCRE_PARTIAL_HARD is set. Consider 215the string "dog" matched against the ungreedy pattern shown above: 216<pre> 217 /dog(sbody)??/ 218</pre> 219Whereas the standard functions stop as soon as they find the complete match for 220"dog", the DFA functions also find the partial match for "dogsbody", and so 221return that when PCRE_PARTIAL_HARD is set. 222</P> 223<br><a name="SEC4" href="#TOC1">PARTIAL MATCHING AND WORD BOUNDARIES</a><br> 224<P> 225If a pattern ends with one of sequences \b or \B, which test for word 226boundaries, partial matching with PCRE_PARTIAL_SOFT can give counter-intuitive 227results. Consider this pattern: 228<pre> 229 /\bcat\b/ 230</pre> 231This matches "cat", provided there is a word boundary at either end. If the 232subject string is "the cat", the comparison of the final "t" with a following 233character cannot take place, so a partial match is found. However, normal 234matching carries on, and \b matches at the end of the subject when the last 235character is a letter, so a complete match is found. The result, therefore, is 236<i>not</i> PCRE_ERROR_PARTIAL. Using PCRE_PARTIAL_HARD in this case does yield 237PCRE_ERROR_PARTIAL, because then the partial match takes precedence. 238</P> 239<br><a name="SEC5" href="#TOC1">FORMERLY RESTRICTED PATTERNS</a><br> 240<P> 241For releases of PCRE prior to 8.00, because of the way certain internal 242optimizations were implemented in the <b>pcre_exec()</b> function, the 243PCRE_PARTIAL option (predecessor of PCRE_PARTIAL_SOFT) could not be used with 244all patterns. From release 8.00 onwards, the restrictions no longer apply, and 245partial matching with can be requested for any pattern. 246</P> 247<P> 248Items that were formerly restricted were repeated single characters and 249repeated metasequences. If PCRE_PARTIAL was set for a pattern that did not 250conform to the restrictions, <b>pcre_exec()</b> returned the error code 251PCRE_ERROR_BADPARTIAL (-13). This error code is no longer in use. The 252PCRE_INFO_OKPARTIAL call to <b>pcre_fullinfo()</b> to find out if a compiled 253pattern can be used for partial matching now always returns 1. 254</P> 255<br><a name="SEC6" href="#TOC1">EXAMPLE OF PARTIAL MATCHING USING PCRETEST</a><br> 256<P> 257If the escape sequence \P is present in a <b>pcretest</b> data line, the 258PCRE_PARTIAL_SOFT option is used for the match. Here is a run of <b>pcretest</b> 259that uses the date example quoted above: 260<pre> 261 re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ 262 data> 25jun04\P 263 0: 25jun04 264 1: jun 265 data> 25dec3\P 266 Partial match: 23dec3 267 data> 3ju\P 268 Partial match: 3ju 269 data> 3juj\P 270 No match 271 data> j\P 272 No match 273</pre> 274The first data string is matched completely, so <b>pcretest</b> shows the 275matched substrings. The remaining four strings do not match the complete 276pattern, but the first two are partial matches. Similar output is obtained 277if DFA matching is used. 278</P> 279<P> 280If the escape sequence \P is present more than once in a <b>pcretest</b> data 281line, the PCRE_PARTIAL_HARD option is set for the match. 282</P> 283<br><a name="SEC7" href="#TOC1">MULTI-SEGMENT MATCHING WITH pcre_dfa_exec() OR pcre16_dfa_exec()</a><br> 284<P> 285When a partial match has been found using a DFA matching function, it is 286possible to continue the match by providing additional subject data and calling 287the function again with the same compiled regular expression, this time setting 288the PCRE_DFA_RESTART option. You must pass the same working space as before, 289because this is where details of the previous partial match are stored. Here is 290an example using <b>pcretest</b>, using the \R escape sequence to set the 291PCRE_DFA_RESTART option (\D specifies the use of the DFA matching function): 292<pre> 293 re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ 294 data> 23ja\P\D 295 Partial match: 23ja 296 data> n05\R\D 297 0: n05 298</pre> 299The first call has "23ja" as the subject, and requests partial matching; the 300second call has "n05" as the subject for the continued (restarted) match. 301Notice that when the match is complete, only the last part is shown; PCRE does 302not retain the previously partially-matched string. It is up to the calling 303program to do that if it needs to. 304</P> 305<P> 306You can set the PCRE_PARTIAL_SOFT or PCRE_PARTIAL_HARD options with 307PCRE_DFA_RESTART to continue partial matching over multiple segments. This 308facility can be used to pass very long subject strings to the DFA matching 309functions. 310</P> 311<br><a name="SEC8" href="#TOC1">MULTI-SEGMENT MATCHING WITH pcre_exec() OR pcre16_exec()</a><br> 312<P> 313From release 8.00, the standard matching functions can also be used to do 314multi-segment matching. Unlike the DFA functions, it is not possible to 315restart the previous match with a new segment of data. Instead, new data must 316be added to the previous subject string, and the entire match re-run, starting 317from the point where the partial match occurred. Earlier data can be discarded. 318</P> 319<P> 320It is best to use PCRE_PARTIAL_HARD in this situation, because it does not 321treat the end of a segment as the end of the subject when matching \z, \Z, 322\b, \B, and $. Consider an unanchored pattern that matches dates: 323<pre> 324 re> /\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d/ 325 data> The date is 23ja\P\P 326 Partial match: 23ja 327</pre> 328At this stage, an application could discard the text preceding "23ja", add on 329text from the next segment, and call the matching function again. Unlike the 330DFA matching functions, the entire matching string must always be available, 331and the complete matching process occurs for each call, so more memory and more 332processing time is needed. 333</P> 334<P> 335<b>Note:</b> If the pattern contains lookbehind assertions, or \K, or starts 336with \b or \B, the string that is returned for a partial match includes 337characters that precede the partially matched string itself, because these must 338be retained when adding on more characters for a subsequent matching attempt. 339However, in some cases you may need to retain even earlier characters, as 340discussed in the next section. 341</P> 342<br><a name="SEC9" href="#TOC1">ISSUES WITH MULTI-SEGMENT MATCHING</a><br> 343<P> 344Certain types of pattern may give problems with multi-segment matching, 345whichever matching function is used. 346</P> 347<P> 3481. If the pattern contains a test for the beginning of a line, you need to pass 349the PCRE_NOTBOL option when the subject string for any call does start at the 350beginning of a line. There is also a PCRE_NOTEOL option, but in practice when 351doing multi-segment matching you should be using PCRE_PARTIAL_HARD, which 352includes the effect of PCRE_NOTEOL. 353</P> 354<P> 3552. Lookbehind assertions that have already been obeyed are catered for in the 356offsets that are returned for a partial match. However a lookbehind assertion 357later in the pattern could require even earlier characters to be inspected. You 358can handle this case by using the PCRE_INFO_MAXLOOKBEHIND option of the 359<b>pcre_fullinfo()</b> or <b>pcre16_fullinfo()</b> functions to obtain the length 360of the largest lookbehind in the pattern. This length is given in characters, 361not bytes. If you always retain at least that many characters before the 362partially matched string, all should be well. (Of course, near the start of the 363subject, fewer characters may be present; in that case all characters should be 364retained.) 365</P> 366<P> 3673. Because a partial match must always contain at least one character, what 368might be considered a partial match of an empty string actually gives a "no 369match" result. For example: 370<pre> 371 re> /c(?<=abc)x/ 372 data> ab\P 373 No match 374</pre> 375If the next segment begins "cx", a match should be found, but this will only 376happen if characters from the previous segment are retained. For this reason, a 377"no match" result should be interpreted as "partial match of an empty string" 378when the pattern contains lookbehinds. 379</P> 380<P> 3814. Matching a subject string that is split into multiple segments may not 382always produce exactly the same result as matching over one single long string, 383especially when PCRE_PARTIAL_SOFT is used. The section "Partial Matching and 384Word Boundaries" above describes an issue that arises if the pattern ends with 385\b or \B. Another kind of difference may occur when there are multiple 386matching possibilities, because (for PCRE_PARTIAL_SOFT) a partial match result 387is given only when there are no completed matches. This means that as soon as 388the shortest match has been found, continuation to a new subject segment is no 389longer possible. Consider again this <b>pcretest</b> example: 390<pre> 391 re> /dog(sbody)?/ 392 data> dogsb\P 393 0: dog 394 data> do\P\D 395 Partial match: do 396 data> gsb\R\P\D 397 0: g 398 data> dogsbody\D 399 0: dogsbody 400 1: dog 401</pre> 402The first data line passes the string "dogsb" to a standard matching function, 403setting the PCRE_PARTIAL_SOFT option. Although the string is a partial match 404for "dogsbody", the result is not PCRE_ERROR_PARTIAL, because the shorter 405string "dog" is a complete match. Similarly, when the subject is presented to 406a DFA matching function in several parts ("do" and "gsb" being the first two) 407the match stops when "dog" has been found, and it is not possible to continue. 408On the other hand, if "dogsbody" is presented as a single string, a DFA 409matching function finds both matches. 410</P> 411<P> 412Because of these problems, it is best to use PCRE_PARTIAL_HARD when matching 413multi-segment data. The example above then behaves differently: 414<pre> 415 re> /dog(sbody)?/ 416 data> dogsb\P\P 417 Partial match: dogsb 418 data> do\P\D 419 Partial match: do 420 data> gsb\R\P\P\D 421 Partial match: gsb 422</pre> 4235. Patterns that contain alternatives at the top level which do not all start 424with the same pattern item may not work as expected when PCRE_DFA_RESTART is 425used. For example, consider this pattern: 426<pre> 427 1234|3789 428</pre> 429If the first part of the subject is "ABC123", a partial match of the first 430alternative is found at offset 3. There is no partial match for the second 431alternative, because such a match does not start at the same point in the 432subject string. Attempting to continue with the string "7890" does not yield a 433match because only those alternatives that match at one point in the subject 434are remembered. The problem arises because the start of the second alternative 435matches within the first alternative. There is no problem with anchored 436patterns or patterns such as: 437<pre> 438 1234|ABCD 439</pre> 440where no string can be a partial match for both alternatives. This is not a 441problem if a standard matching function is used, because the entire match has 442to be rerun each time: 443<pre> 444 re> /1234|3789/ 445 data> ABC123\P\P 446 Partial match: 123 447 data> 1237890 448 0: 3789 449</pre> 450Of course, instead of using PCRE_DFA_RESTART, the same technique of re-running 451the entire match can also be used with the DFA matching functions. Another 452possibility is to work with two buffers. If a partial match at offset <i>n</i> 453in the first buffer is followed by "no match" when PCRE_DFA_RESTART is used on 454the second buffer, you can then try a new match starting at offset <i>n+1</i> in 455the first buffer. 456</P> 457<br><a name="SEC10" href="#TOC1">AUTHOR</a><br> 458<P> 459Philip Hazel 460<br> 461University Computing Service 462<br> 463Cambridge CB2 3QH, England. 464<br> 465</P> 466<br><a name="SEC11" href="#TOC1">REVISION</a><br> 467<P> 468Last updated: 24 February 2012 469<br> 470Copyright © 1997-2012 University of Cambridge. 471<br> 472<p> 473Return to the <a href="index.html">PCRE index page</a>. 474</p> 475