1228072SbaptChanges between release 2.5.4 (11Sep96) and release 2.5.3: 2228072Sbapt 3228072Sbapt - Fixed a bug introduced in 2.5.3 that blew it when a call 4228072Sbapt to input() occurred at the end of an input file. 5228072Sbapt 6228072Sbapt - Fixed scanner skeleton so the example in the man page of 7228072Sbapt scanning strings using exclusive start conditions works. 8228072Sbapt 9228072Sbapt - Minor Makefile tweaks. 10228072Sbapt 11228072Sbapt 12228072SbaptChanges between release 2.5.3 (29May96) and release 2.5.2: 13228072Sbapt 14228072Sbapt - Some serious bugs in yymore() have been fixed. In particular, 15228072Sbapt when using AT&T-lex-compatibility or %array, you can intermix 16228072Sbapt calls to input(), unput(), and yymore(). (This still doesn't 17228072Sbapt work for %pointer, and isn't likely to in the future.) 18228072Sbapt 19228072Sbapt - A bug in handling NUL's in the input stream of scanners using 20228072Sbapt REJECT has been fixed. 21228072Sbapt 22228072Sbapt - The default main() in libfl.a now repeatedly calls yylex() until 23228072Sbapt it returns 0, rather than just calling it once. 24228072Sbapt 25228072Sbapt - Minor tweak for Windows NT Makefile, MISC/NT/Makefile. 26228072Sbapt 27228072Sbapt 28228072SbaptChanges between release 2.5.2 (25Apr95) and release 2.5.1: 29228072Sbapt 30228072Sbapt - The --prefix configuration option now works. 31228072Sbapt 32228072Sbapt - A bug that completely broke the "-Cf" table compression 33228072Sbapt option has been fixed. 34228072Sbapt 35228072Sbapt - A major headache involving "const" declarators and Solaris 36228072Sbapt systems has been fixed. 37228072Sbapt 38228072Sbapt - An octal escape sequence in a flex regular expression must 39228072Sbapt now contain only the digits 0-7. 40228072Sbapt 41228072Sbapt - You can now use "--" on the flex command line to mark the 42228072Sbapt end of flex options. 43228072Sbapt 44228072Sbapt - You can now specify the filename '-' as a synonym for stdin. 45228072Sbapt 46228072Sbapt - By default, the scanners generated by flex no longer 47228072Sbapt statically initialize yyin and yyout to stdin and stdout. 48228072Sbapt This change is necessary because in some ANSI environments, 49228072Sbapt stdin and stdout are not compile-time constant. You can 50228072Sbapt force the initialization using "%option stdinit" in the first 51228072Sbapt section of your flex input. 52228072Sbapt 53228072Sbapt - "%option nounput" now correctly omits the unput() routine 54228072Sbapt from the output. 55228072Sbapt 56228072Sbapt - "make clean" now removes config.log, config.cache, and the 57228072Sbapt flex binary. The fact that it removes the flex binary means 58228072Sbapt you should take care if making changes to scan.l, to make 59228072Sbapt sure you don't wind up in a bootstrap problem. 60228072Sbapt 61228072Sbapt - In general, the Makefile has been reworked somewhat (thanks 62228072Sbapt to Francois Pinard) for added flexibility - more changes will 63228072Sbapt follow in subsequent releases. 64228072Sbapt 65228072Sbapt - The .texi and .info files in MISC/texinfo/ have been updated, 66228072Sbapt thanks also to Francois Pinard. 67228072Sbapt 68228072Sbapt - The FlexLexer::yylex(istream* new_in, ostream* new_out) method 69228072Sbapt now does not have a default for the first argument, to disambiguate 70228072Sbapt it from FlexLexer::yylex(). 71228072Sbapt 72228072Sbapt - A bug in destructing a FlexLexer object before doing any scanning 73228072Sbapt with it has been fixed. 74228072Sbapt 75228072Sbapt - A problem with including FlexLexer.h multiple times has been fixed. 76228072Sbapt 77228072Sbapt - The alloca() chud necessary to accommodate bison has grown 78228072Sbapt even uglier, but hopefully more correct. 79228072Sbapt 80228072Sbapt - A portability tweak has been added to accommodate compilers that 81228072Sbapt use char* generic pointers. 82228072Sbapt 83228072Sbapt - EBCDIC contact information in the file MISC/EBCDIC has been updated. 84228072Sbapt 85228072Sbapt - An OS/2 Makefile and config.h for flex 2.5 is now available in 86228072Sbapt MISC/OS2/, contributed by Kai Uwe Rommel. 87228072Sbapt 88228072Sbapt - The descrip.mms file for building flex under VMS has been updated, 89228072Sbapt thanks to Pat Rankin. 90228072Sbapt 91228072Sbapt - The notes on building flex for the Amiga have been updated for 92228072Sbapt flex 2.5, contributed by Andreas Scherer. 93228072Sbapt 94228072Sbapt 95228072SbaptChanges between release 2.5.1 (28Mar95) and release 2.4.7: 96228072Sbapt 97228072Sbapt - A new concept of "start condition" scope has been introduced. 98228072Sbapt A start condition scope is begun with: 99228072Sbapt 100228072Sbapt <SCs>{ 101228072Sbapt 102228072Sbapt where SCs is a list of one or more start conditions. Inside 103228072Sbapt the start condition scope, every rule automatically has the 104228072Sbapt prefix <SCs> applied to it, until a '}' which matches the 105228072Sbapt initial '{'. So, for example: 106228072Sbapt 107228072Sbapt <ESC>{ 108228072Sbapt "\\n" return '\n'; 109228072Sbapt "\\r" return '\r'; 110228072Sbapt "\\f" return '\f'; 111228072Sbapt "\\0" return '\0'; 112228072Sbapt } 113228072Sbapt 114228072Sbapt is equivalent to: 115228072Sbapt 116228072Sbapt <ESC>"\\n" return '\n'; 117228072Sbapt <ESC>"\\r" return '\r'; 118228072Sbapt <ESC>"\\f" return '\f'; 119228072Sbapt <ESC>"\\0" return '\0'; 120228072Sbapt 121228072Sbapt As indicated in this example, rules inside start condition scopes 122228072Sbapt (and any rule, actually, other than the first) can be indented, 123228072Sbapt to better show the extent of the scope. 124228072Sbapt 125228072Sbapt Start condition scopes may be nested. 126228072Sbapt 127228072Sbapt - The new %option directive can be used in the first section of 128228072Sbapt a flex scanner to control scanner-generation options. Most 129228072Sbapt options are given simply as names, optionally preceded by the 130228072Sbapt word "no" (with no intervening whitespace) to negate their 131228072Sbapt meaning. Some are equivalent to flex flags, so putting them 132228072Sbapt in your scanner source is equivalent to always specifying 133228072Sbapt the flag (%option's take precedence over flags): 134228072Sbapt 135228072Sbapt 7bit -7 option 136228072Sbapt 8bit -8 option 137228072Sbapt align -Ca option 138228072Sbapt backup -b option 139228072Sbapt batch -B option 140228072Sbapt c++ -+ option 141228072Sbapt caseful opposite of -i option (caseful is the default); 142228072Sbapt case-sensitive same as above 143228072Sbapt caseless -i option; 144228072Sbapt case-insensitive same as above 145228072Sbapt debug -d option 146228072Sbapt default opposite of -s option 147228072Sbapt ecs -Ce option 148228072Sbapt fast -F option 149228072Sbapt full -f option 150228072Sbapt interactive -I option 151228072Sbapt lex-compat -l option 152228072Sbapt meta-ecs -Cm option 153228072Sbapt perf-report -p option 154228072Sbapt read -Cr option 155228072Sbapt stdout -t option 156228072Sbapt verbose -v option 157228072Sbapt warn opposite of -w option (so use "%option nowarn" for -w) 158228072Sbapt 159228072Sbapt array equivalent to "%array" 160228072Sbapt pointer equivalent to "%pointer" (default) 161228072Sbapt 162228072Sbapt Some provide new features: 163228072Sbapt 164228072Sbapt always-interactive generate a scanner which always 165228072Sbapt considers its input "interactive" (no call to isatty() 166228072Sbapt will be made when the scanner runs) 167228072Sbapt main supply a main program for the scanner, which 168228072Sbapt simply calls yylex(). Implies %option noyywrap. 169228072Sbapt never-interactive generate a scanner which never 170228072Sbapt considers its input "interactive" (no call to isatty() 171228072Sbapt will be made when the scanner runs) 172228072Sbapt stack if set, enable start condition stacks (see below) 173228072Sbapt stdinit if unset ("%option nostdinit"), initialize yyin 174228072Sbapt and yyout statically to nil FILE* pointers, instead 175228072Sbapt of stdin and stdout 176228072Sbapt yylineno if set, keep track of the current line 177228072Sbapt number in global yylineno (this option is expensive 178228072Sbapt in terms of performance). The line number is available 179228072Sbapt to C++ scanning objects via the new member function 180228072Sbapt lineno(). 181228072Sbapt yywrap if unset ("%option noyywrap"), scanner does not 182228072Sbapt call yywrap() upon EOF but simply assumes there 183228072Sbapt are no more files to scan 184228072Sbapt 185228072Sbapt Flex scans your rule actions to determine whether you use the 186228072Sbapt REJECT or yymore features (this is not new). Two %options can be 187228072Sbapt used to override its decision, either by setting them to indicate 188228072Sbapt the feature is indeed used, or unsetting them to indicate it 189228072Sbapt actually is not used: 190228072Sbapt 191228072Sbapt reject 192228072Sbapt yymore 193228072Sbapt 194228072Sbapt Three %option's take string-delimited values, offset with '=': 195228072Sbapt 196228072Sbapt outfile="<name>" equivalent to -o<name> 197228072Sbapt prefix="<name>" equivalent to -P<name> 198228072Sbapt yyclass="<name>" set the name of the C++ scanning class 199228072Sbapt (see below) 200228072Sbapt 201228072Sbapt A number of %option's are available for lint purists who 202228072Sbapt want to suppress the appearance of unneeded routines in 203228072Sbapt the generated scanner. Each of the following, if unset, 204228072Sbapt results in the corresponding routine not appearing in the 205228072Sbapt generated scanner: 206228072Sbapt 207228072Sbapt input, unput 208228072Sbapt yy_push_state, yy_pop_state, yy_top_state 209228072Sbapt yy_scan_buffer, yy_scan_bytes, yy_scan_string 210228072Sbapt 211228072Sbapt You can specify multiple options with a single %option directive, 212228072Sbapt and multiple directives in the first section of your flex input file. 213228072Sbapt 214228072Sbapt - The new function: 215228072Sbapt 216228072Sbapt YY_BUFFER_STATE yy_scan_string( const char *str ) 217228072Sbapt 218228072Sbapt returns a YY_BUFFER_STATE (which also becomes the current input 219228072Sbapt buffer) for scanning the given string, which occurs starting 220228072Sbapt with the next call to yylex(). The string must be NUL-terminated. 221228072Sbapt A related function: 222228072Sbapt 223228072Sbapt YY_BUFFER_STATE yy_scan_bytes( const char *bytes, int len ) 224228072Sbapt 225228072Sbapt creates a buffer for scanning "len" bytes (including possibly NUL's) 226228072Sbapt starting at location "bytes". 227228072Sbapt 228228072Sbapt Note that both of these functions create and scan a *copy* of 229228072Sbapt the string/bytes. (This may be desirable, since yylex() modifies 230228072Sbapt the contents of the buffer it is scanning.) You can avoid the 231228072Sbapt copy by using: 232228072Sbapt 233228072Sbapt YY_BUFFER_STATE yy_scan_buffer( char *base, yy_size_t size ) 234228072Sbapt 235228072Sbapt which scans in place the buffer starting at "base", consisting 236228072Sbapt of "size" bytes, the last two bytes of which *must* be 237228072Sbapt YY_END_OF_BUFFER_CHAR (these bytes are not scanned; thus, scanning 238228072Sbapt consists of base[0] through base[size-2], inclusive). If you 239228072Sbapt fail to set up "base" in this manner, yy_scan_buffer returns a 240228072Sbapt nil pointer instead of creating a new input buffer. 241228072Sbapt 242228072Sbapt The type yy_size_t is an integral type to which you can cast 243228072Sbapt an integer expression reflecting the size of the buffer. 244228072Sbapt 245228072Sbapt - Three new routines are available for manipulating stacks of 246228072Sbapt start conditions: 247228072Sbapt 248228072Sbapt void yy_push_state( int new_state ) 249228072Sbapt 250228072Sbapt pushes the current start condition onto the top of the stack 251228072Sbapt and BEGIN's "new_state" (recall that start condition names are 252228072Sbapt also integers). 253228072Sbapt 254228072Sbapt void yy_pop_state() 255228072Sbapt 256228072Sbapt pops the top of the stack and BEGIN's to it, and 257228072Sbapt 258228072Sbapt int yy_top_state() 259228072Sbapt 260228072Sbapt returns the top of the stack without altering the stack's 261228072Sbapt contents. 262228072Sbapt 263228072Sbapt The start condition stack grows dynamically and so has no built-in 264228072Sbapt size limitation. If memory is exhausted, program execution 265228072Sbapt is aborted. 266228072Sbapt 267228072Sbapt To use start condition stacks, your scanner must include 268228072Sbapt a "%option stack" directive. 269228072Sbapt 270228072Sbapt - flex now supports POSIX character class expressions. These 271228072Sbapt are expressions enclosed inside "[:" and ":]" delimiters (which 272228072Sbapt themselves must appear between the '[' and ']' of a character 273228072Sbapt class; other elements may occur inside the character class, too). 274228072Sbapt The expressions flex recognizes are: 275228072Sbapt 276228072Sbapt [:alnum:] [:alpha:] [:blank:] [:cntrl:] [:digit:] [:graph:] 277228072Sbapt [:lower:] [:print:] [:punct:] [:space:] [:upper:] [:xdigit:] 278228072Sbapt 279228072Sbapt These expressions all designate a set of characters equivalent to 280228072Sbapt the corresponding isXXX function (for example, [:alnum:] designates 281228072Sbapt those characters for which isalnum() returns true - i.e., any 282228072Sbapt alphabetic or numeric). Some systems don't provide isblank(), 283228072Sbapt so flex defines [:blank:] as a blank or a tab. 284228072Sbapt 285228072Sbapt For example, the following character classes are all equivalent: 286228072Sbapt 287228072Sbapt [[:alnum:]] 288228072Sbapt [[:alpha:][:digit:] 289228072Sbapt [[:alpha:]0-9] 290228072Sbapt [a-zA-Z0-9] 291228072Sbapt 292228072Sbapt If your scanner is case-insensitive (-i flag), then [:upper:] 293228072Sbapt and [:lower:] are equivalent to [:alpha:]. 294228072Sbapt 295228072Sbapt - The promised rewrite of the C++ FlexLexer class has not yet 296228072Sbapt been done. Support for FlexLexer is limited at the moment to 297228072Sbapt fixing show-stopper bugs, so, for example, the new functions 298228072Sbapt yy_scan_string() & friends are not available to FlexLexer 299228072Sbapt objects. 300228072Sbapt 301228072Sbapt - The new macro 302228072Sbapt 303228072Sbapt yy_set_interactive(is_interactive) 304228072Sbapt 305228072Sbapt can be used to control whether the current buffer is considered 306228072Sbapt "interactive". An interactive buffer is processed more slowly, 307228072Sbapt but must be used when the scanner's input source is indeed 308228072Sbapt interactive to avoid problems due to waiting to fill buffers 309228072Sbapt (see the discussion of the -I flag in flex.1). A non-zero value 310228072Sbapt in the macro invocation marks the buffer as interactive, a zero 311228072Sbapt value as non-interactive. Note that use of this macro overrides 312228072Sbapt "%option always-interactive" or "%option never-interactive". 313228072Sbapt 314228072Sbapt yy_set_interactive() must be invoked prior to beginning to 315228072Sbapt scan the buffer. 316228072Sbapt 317228072Sbapt - The new macro 318228072Sbapt 319228072Sbapt yy_set_bol(at_bol) 320228072Sbapt 321228072Sbapt can be used to control whether the current buffer's scanning 322228072Sbapt context for the next token match is done as though at the 323228072Sbapt beginning of a line (non-zero macro argument; makes '^' anchored 324228072Sbapt rules active) or not at the beginning of a line (zero argument, 325228072Sbapt '^' rules inactive). 326228072Sbapt 327228072Sbapt - Related to this change, the mechanism for determining when a scan is 328228072Sbapt starting at the beginning of a line has changed. It used to be 329228072Sbapt that '^' was active iff the character prior to that at which the 330228072Sbapt scan started was a newline. The mechanism now is that '^' is 331228072Sbapt active iff the last token ended in a newline (or the last call to 332228072Sbapt input() returned a newline). For most users, the difference in 333228072Sbapt mechanisms is negligible. Where it will make a difference, 334228072Sbapt however, is if unput() or yyless() is used to alter the input 335228072Sbapt stream. When in doubt, use yy_set_bol(). 336228072Sbapt 337228072Sbapt - The new beginning-of-line mechanism involved changing some fairly 338228072Sbapt twisted code, so it may have introduced bugs - beware ... 339228072Sbapt 340228072Sbapt - The macro YY_AT_BOL() returns true if the next token scanned from 341228072Sbapt the current buffer will have '^' rules active, false otherwise. 342228072Sbapt 343228072Sbapt - The new function 344228072Sbapt 345228072Sbapt void yy_flush_buffer( struct yy_buffer_state* b ) 346228072Sbapt 347228072Sbapt flushes the contents of the current buffer (i.e., next time 348228072Sbapt the scanner attempts to match a token using b as the current 349228072Sbapt buffer, it will begin by invoking YY_INPUT to fill the buffer). 350228072Sbapt This routine is also available to C++ scanners (unlike some 351228072Sbapt of the other new routines). 352228072Sbapt 353228072Sbapt The related macro 354228072Sbapt 355228072Sbapt YY_FLUSH_BUFFER 356228072Sbapt 357228072Sbapt flushes the contents of the current buffer. 358228072Sbapt 359228072Sbapt - A new "-ooutput" option writes the generated scanner to "output". 360228072Sbapt If used with -t, the scanner is still written to stdout, but 361228072Sbapt its internal #line directives (see previous item) use "output". 362228072Sbapt 363228072Sbapt - Flex now generates #line directives relating the code it 364228072Sbapt produces to the output file; this means that error messages 365228072Sbapt in the flex-generated code should be correctly pinpointed. 366228072Sbapt 367228072Sbapt - When generating #line directives, filenames with embedded '\'s 368228072Sbapt have those characters escaped (i.e., turned into '\\'). This 369228072Sbapt feature helps with reporting filenames for some MS-DOS and OS/2 370228072Sbapt systems. 371228072Sbapt 372228072Sbapt - The FlexLexer class includes two new public member functions: 373228072Sbapt 374228072Sbapt virtual void switch_streams( istream* new_in = 0, 375228072Sbapt ostream* new_out = 0 ) 376228072Sbapt 377228072Sbapt reassigns yyin to new_in (if non-nil) and yyout to new_out 378228072Sbapt (ditto), deleting the previous input buffer if yyin is 379228072Sbapt reassigned. It is used by: 380228072Sbapt 381228072Sbapt int yylex( istream* new_in = 0, ostream* new_out = 0 ) 382228072Sbapt 383228072Sbapt which first calls switch_streams() and then returns the value 384228072Sbapt of calling yylex(). 385228072Sbapt 386228072Sbapt - C++ scanners now have yy_flex_debug as a member variable of 387228072Sbapt FlexLexer rather than a global, and member functions for testing 388228072Sbapt and setting it. 389228072Sbapt 390228072Sbapt - When generating a C++ scanning class, you can now use 391228072Sbapt 392228072Sbapt %option yyclass="foo" 393228072Sbapt 394228072Sbapt to inform flex that you have derived "foo" as a subclass of 395228072Sbapt yyFlexLexer, so flex will place your actions in the member 396228072Sbapt function foo::yylex() instead of yyFlexLexer::yylex(). It also 397228072Sbapt generates a yyFlexLexer::yylex() member function that generates a 398228072Sbapt run-time error if called (by invoking yyFlexLexer::LexerError()). 399228072Sbapt This feature is necessary if your subclass "foo" introduces some 400228072Sbapt additional member functions or variables that you need to access 401228072Sbapt from yylex(). 402228072Sbapt 403228072Sbapt - Current texinfo files in MISC/texinfo, contributed by Francois 404228072Sbapt Pinard. 405228072Sbapt 406228072Sbapt - You can now change the name "flex" to something else (e.g., "lex") 407228072Sbapt by redefining $(FLEX) in the Makefile. 408228072Sbapt 409228072Sbapt - Two bugs (one serious) that could cause "bigcheck" to fail have 410228072Sbapt been fixed. 411228072Sbapt 412228072Sbapt - A number of portability/configuration changes have been made 413228072Sbapt for easier portability. 414228072Sbapt 415228072Sbapt - You can use "YYSTATE" in your scanner as an alias for YY_START 416228072Sbapt (for AT&T lex compatibility). 417228072Sbapt 418228072Sbapt - input() now maintains yylineno. 419228072Sbapt 420228072Sbapt - input() no longer trashes yytext. 421228072Sbapt 422228072Sbapt - interactive scanners now read characters in YY_INPUT up to a 423228072Sbapt newline, a large performance gain. 424228072Sbapt 425228072Sbapt - C++ scanner objects now work with the -P option. You include 426228072Sbapt <FlexLexer.h> once per scanner - see comments in <FlexLexer.h> 427228072Sbapt (or flex.1) for details. 428228072Sbapt 429228072Sbapt - C++ FlexLexer objects now use the "cerr" stream to report -d output 430228072Sbapt instead of stdio. 431228072Sbapt 432228072Sbapt - The -c flag now has its full glorious POSIX interpretation (do 433228072Sbapt nothing), rather than being interpreted as an old-style -C flag. 434228072Sbapt 435228072Sbapt - Scanners generated by flex now include two #define's giving 436228072Sbapt the major and minor version numbers (YY_FLEX_MAJOR_VERSION, 437228072Sbapt YY_FLEX_MINOR_VERSION). These can then be tested to see 438228072Sbapt whether certain flex features are available. 439228072Sbapt 440228072Sbapt - Scanners generated using -l lex compatibility now have the symbol 441228072Sbapt YY_FLEX_LEX_COMPAT #define'd. 442228072Sbapt 443228072Sbapt - When initializing (i.e., yy_init is non-zero on entry to yylex()), 444228072Sbapt generated scanners now set yy_init to zero before executing 445228072Sbapt YY_USER_INIT. This means that you can set yy_init back to a 446228072Sbapt non-zero value in YY_USER_INIT if you need the scanner to be 447228072Sbapt reinitialized on the next call. 448228072Sbapt 449228072Sbapt - You can now use "#line" directives in the first section of your 450228072Sbapt scanner specification. 451228072Sbapt 452228072Sbapt - When generating full-table scanners (-Cf), flex now puts braces 453228072Sbapt around each row of the 2-d array initialization, to silence warnings 454228072Sbapt on over-zealous compilers. 455228072Sbapt 456228072Sbapt - Improved support for MS-DOS. The flex sources have been successfully 457228072Sbapt built, unmodified, for Borland 4.02 (all that's required is a 458228072Sbapt Borland Makefile and config.h file, which are supplied in 459228072Sbapt MISC/Borland - contributed by Terrence O Kane). 460228072Sbapt 461228072Sbapt - Improved support for Macintosh using Think C - the sources should 462228072Sbapt build for this platform "out of the box". Contributed by Scott 463228072Sbapt Hofmann. 464228072Sbapt 465228072Sbapt - Improved support for VMS, in MISC/VMS/, contributed by Pat Rankin. 466228072Sbapt 467228072Sbapt - Support for the Amiga, in MISC/Amiga/, contributed by Andreas 468228072Sbapt Scherer. Note that the contributed files were developed for 469228072Sbapt flex 2.4 and have not been tested with flex 2.5. 470228072Sbapt 471228072Sbapt - Some notes on support for the NeXT, in MISC/NeXT, contributed 472228072Sbapt by Raf Schietekat. 473228072Sbapt 474228072Sbapt - The MISC/ directory now includes a preformatted version of flex.1 475228072Sbapt in flex.man, and pre-yacc'd versions of parse.y in parse.{c,h}. 476228072Sbapt 477228072Sbapt - The flex.1 and flexdoc.1 manual pages have been merged. There 478228072Sbapt is now just one document, flex.1, which includes an overview 479228072Sbapt at the beginning to help you find the section you need. 480228072Sbapt 481228072Sbapt - Documentation now clarifies that start conditions persist across 482228072Sbapt switches to new input files or different input buffers. If you 483228072Sbapt want to e.g., return to INITIAL, you must explicitly do so. 484228072Sbapt 485228072Sbapt - The "Performance Considerations" section of the manual has been 486228072Sbapt updated. 487228072Sbapt 488228072Sbapt - Documented the "yy_act" variable, which when YY_USER_ACTION is 489228072Sbapt invoked holds the number of the matched rule, and added an 490228072Sbapt example of using yy_act to profile how often each rule is matched. 491228072Sbapt 492228072Sbapt - Added YY_NUM_RULES, a definition that gives the total number 493228072Sbapt of rules in the file, including the default rule (even if you 494228072Sbapt use -s). 495228072Sbapt 496228072Sbapt - Documentation now clarifies that you can pass a nil FILE* pointer 497228072Sbapt to yy_create_buffer() or yyrestart() if you've arrange YY_INPUT 498228072Sbapt to not need yyin. 499228072Sbapt 500228072Sbapt - Documentation now clarifies that YY_BUFFER_STATE is a pointer to 501228072Sbapt an opaque "struct yy_buffer_state". 502228072Sbapt 503228072Sbapt - Documentation now stresses that you gain the benefits of removing 504228072Sbapt backing-up states only if you remove *all* of them. 505228072Sbapt 506228072Sbapt - Documentation now points out that traditional lex allows you 507228072Sbapt to put the action on a separate line from the rule pattern if 508228072Sbapt the pattern has trailing whitespace (ugh!), but flex doesn't 509228072Sbapt support this. 510228072Sbapt 511228072Sbapt - A broken example in documentation of the difference between 512228072Sbapt inclusive and exclusive start conditions is now fixed. 513228072Sbapt 514228072Sbapt - Usage (-h) report now goes to stdout. 515228072Sbapt 516228072Sbapt - Version (-V) info now goes to stdout. 517228072Sbapt 518228072Sbapt - More #ifdef chud has been added to the parser in attempt to 519228072Sbapt deal with bison's use of alloca(). 520228072Sbapt 521228072Sbapt - "make clean" no longer deletes emacs backup files (*~). 522228072Sbapt 523228072Sbapt - Some memory leaks have been fixed. 524228072Sbapt 525228072Sbapt - A bug was fixed in which dynamically-expanded buffers were 526228072Sbapt reallocated a couple of bytes too small. 527228072Sbapt 528228072Sbapt - A bug was fixed which could cause flex to read and write beyond 529228072Sbapt the end of the input buffer. 530228072Sbapt 531228072Sbapt - -S will not be going away. 532228072Sbapt 533228072Sbapt 534228072SbaptChanges between release 2.4.7 (03Aug94) and release 2.4.6: 535228072Sbapt 536228072Sbapt - Fixed serious bug in reading multiple files. 537228072Sbapt 538228072Sbapt - Fixed bug in scanning NUL's. 539228072Sbapt 540228072Sbapt - Fixed bug in input() returning 8-bit characters. 541228072Sbapt 542228072Sbapt - Fixed bug in matching text with embedded NUL's when 543228072Sbapt using %array or lex compatibility. 544228072Sbapt 545228072Sbapt - Fixed multiple invocations of YY_USER_ACTION when using '|' 546228072Sbapt continuation action. 547228072Sbapt 548228072Sbapt - Minor prototyping fixes. 549228072Sbapt 550228072SbaptChanges between release 2.4.6 (04Jan94) and release 2.4.5: 551228072Sbapt 552228072Sbapt - Linking with -lfl no longer required if your program includes 553228072Sbapt its own yywrap() and main() functions. (This change will cause 554228072Sbapt problems if you have a non-ANSI compiler on a system for which 555228072Sbapt sizeof(int) != sizeof(void*) or sizeof(int) != sizeof(size_t).) 556228072Sbapt 557228072Sbapt - The use of 'extern "C++"' in FlexLexer.h has been modified to 558228072Sbapt get around an incompatibility with g++'s header files. 559228072Sbapt 560228072SbaptChanges between release 2.4.5 (11Dec93) and release 2.4.4: 561228072Sbapt 562228072Sbapt - Fixed bug breaking C++ scanners that use REJECT or variable 563228072Sbapt trailing context. 564228072Sbapt 565228072Sbapt - Fixed serious input problem for interactive scanners on 566228072Sbapt systems for which char is unsigned. 567228072Sbapt 568228072Sbapt - Fixed bug in incorrectly treating '$' operator as variable 569228072Sbapt trailing context. 570228072Sbapt 571228072Sbapt - Fixed bug in -CF table representation that could lead to 572228072Sbapt corrupt tables. 573228072Sbapt 574228072Sbapt - Fixed fairly benign memory leak. 575228072Sbapt 576228072Sbapt - Added `extern "C++"' wrapper to FlexLexer.h header. This 577228072Sbapt should overcome the g++ 2.5.X problems mentioned in the 578228072Sbapt NEWS for release 2.4.3. 579228072Sbapt 580228072Sbapt - Changed #include of FlexLexer.h to use <> instead of "". 581228072Sbapt 582228072Sbapt - Added feature to control whether the scanner attempts to 583228072Sbapt refill the input buffer once it's exhausted. This feature 584228072Sbapt will be documented in the 2.5 release. 585228072Sbapt 586228072Sbapt 587228072SbaptChanges between release 2.4.4 (07Dec93) and release 2.4.3: 588228072Sbapt 589228072Sbapt - Fixed two serious bugs in scanning 8-bit characters. 590228072Sbapt 591228072Sbapt - Fixed bug in YY_USER_ACTION that caused it to be executed 592228072Sbapt inappropriately (on the scanner's own internal actions, and 593228072Sbapt with incorrect yytext/yyleng values). 594228072Sbapt 595228072Sbapt - Fixed bug in pointing yyin at a new file and resuming scanning. 596228072Sbapt 597228072Sbapt - Portability fix regarding min/max/abs macros conflicting with 598228072Sbapt function definitions in standard header files. 599228072Sbapt 600228072Sbapt - Added a virtual LexerError() method to the C++ yyFlexLexer class 601228072Sbapt for reporting error messages instead of always using cerr. 602228072Sbapt 603228072Sbapt - Added warning in flexdoc that the C++ scanning class is presently 604228072Sbapt experimental and subject to considerable change between major 605228072Sbapt releases. 606228072Sbapt 607228072Sbapt 608228072SbaptChanges between release 2.4.3 (03Dec93) and release 2.4.2: 609228072Sbapt 610228072Sbapt - Fixed bug causing fatal scanner messages to fail to print. 611228072Sbapt 612228072Sbapt - Fixed things so FlexLexer.h can be included in other C++ 613228072Sbapt sources. One side-effect of this change is that -+ and -CF 614228072Sbapt are now incompatible. 615228072Sbapt 616250874Sjkim - libfl.a now supplies private versions of the <string.h>/ 617228072Sbapt <strings.h> string routines needed by flex and the scanners 618228072Sbapt it generates, to enhance portability to some BSD systems. 619228072Sbapt 620228072Sbapt - More robust solution to 2.4.2's flexfatal() bug fix. 621228072Sbapt 622228072Sbapt - Added ranlib of installed libfl.a. 623228072Sbapt 624228072Sbapt - Some lint tweaks. 625228072Sbapt 626228072Sbapt - NOTE: problems have been encountered attempting to build flex 627228072Sbapt C++ scanners using g++ version 2.5.X. The problem is due to an 628228072Sbapt unfortunate heuristic in g++ 2.5.X that attempts to discern between 629228072Sbapt C and C++ headers. Because FlexLexer.h is installed (by default) 630228072Sbapt in /usr/local/include and not /usr/local/lib/g++-include, g++ 2.5.X 631228072Sbapt decides that it's a C header :-(. So if you have problems, install 632228072Sbapt the header in /usr/local/lib/g++-include instead. 633228072Sbapt 634228072Sbapt 635228072SbaptChanges between release 2.4.2 (01Dec93) and release 2.4.1: 636228072Sbapt 637228072Sbapt - Fixed bug in libfl.a referring to non-existent "flexfatal" function. 638228072Sbapt 639228072Sbapt - Modified to produce both compress'd and gzip'd tar files for 640228072Sbapt distributions (you probably don't care about this change!). 641228072Sbapt 642228072Sbapt 643228072SbaptChanges between release 2.4.1 (30Nov93) and release 2.3.8: 644228072Sbapt 645228072Sbapt - The new '-+' flag instructs flex to generate a C++ scanner class 646228072Sbapt (thanks to Kent Williams). flex writes an implementation of the 647228072Sbapt class defined in FlexLexer.h to lex.yy.cc. You may include 648228072Sbapt multiple scanner classes in your program using the -P flag. Note 649228072Sbapt that the scanner class also provides a mechanism for creating 650228072Sbapt reentrant scanners. The scanner class uses C++ streams for I/O 651228072Sbapt instead of FILE*'s (thanks to Tom Epperly). If the flex executable's 652228072Sbapt name ends in '+' then the '-+' flag is automatically on, so creating 653228072Sbapt a symlink or copy of "flex" to "flex++" results in a version of 654228072Sbapt flex that can be used exclusively for C++ scanners. 655228072Sbapt 656228072Sbapt Note that without the '-+' flag, flex-generated scanners can still 657228072Sbapt be compiled using C++ compilers, though they use FILE*'s for I/O 658228072Sbapt instead of streams. 659228072Sbapt 660228072Sbapt See the "GENERATING C++ SCANNERS" section of flexdoc for details. 661228072Sbapt 662228072Sbapt - The new '-l' flag turns on maximum AT&T lex compatibility. In 663228072Sbapt particular, -l includes support for "yylineno" and makes yytext 664228072Sbapt be an array instead of a pointer. It does not, however, do away 665228072Sbapt with all incompatibilities. See the "INCOMPATIBILITIES WITH LEX 666228072Sbapt AND POSIX" section of flexdoc for details. 667228072Sbapt 668228072Sbapt - The new '-P' option specifies a prefix to use other than "yy" 669228072Sbapt for the scanner's globally-visible variables, and for the 670228072Sbapt "lex.yy.c" filename. Using -P you can link together multiple 671228072Sbapt flex scanners in the same executable. 672228072Sbapt 673228072Sbapt - The distribution includes a "texinfo" version of flexdoc.1, 674228072Sbapt contributed by Roland Pesch (thanks also to Marq Kole, who 675228072Sbapt contributed another version). It has not been brought up to 676228072Sbapt date, but reflects version 2.3. See MISC/flex.texinfo. 677228072Sbapt 678228072Sbapt The flex distribution will soon include G.T. Nicol's flex 679228072Sbapt manual; he is presently bringing it up-to-date for version 2.4. 680228072Sbapt 681228072Sbapt - yywrap() is now a function, and you now *must* link flex scanners 682228072Sbapt with libfl.a. 683228072Sbapt 684228072Sbapt - Site-configuration is now done via an autoconf-generated 685228072Sbapt "configure" script contributed by Francois Pinard. 686228072Sbapt 687228072Sbapt - Scanners now use fread() (or getc(), if interactive) and not 688228072Sbapt read() for input. A new "table compression" option, -Cr, 689228072Sbapt overrides this change and causes the scanner to use read() 690228072Sbapt (because read() is a bit faster than fread()). -f and -F 691228072Sbapt are now equivalent to -Cfr and -CFr; i.e., they imply the 692228072Sbapt -Cr option. 693228072Sbapt 694228072Sbapt - In the blessed name of POSIX compliance, flex supports "%array" 695228072Sbapt and "%pointer" directives in the definitions (first) section of 696228072Sbapt the scanner specification. The former specifies that yytext 697228072Sbapt should be an array (of size YYLMAX), the latter, that it should 698228072Sbapt be a pointer. The array version of yytext is universally slower 699228072Sbapt than the pointer version, but has the advantage that its contents 700228072Sbapt remain unmodified across calls to input() and unput() (the pointer 701228072Sbapt version of yytext is, still, trashed by such calls). 702228072Sbapt 703228072Sbapt "%array" cannot be used with the '-+' C++ scanner class option. 704228072Sbapt 705228072Sbapt - The new '-Ca' option directs flex to trade off memory for 706228072Sbapt natural alignment when generating a scanner's tables. In 707228072Sbapt particular, table entries that would otherwise be "short" 708228072Sbapt become "long". 709228072Sbapt 710228072Sbapt - The new '-h' option produces a summary of the flex flags. 711228072Sbapt 712228072Sbapt - The new '-V' option reports the flex version number and exits. 713228072Sbapt 714228072Sbapt - The new scanner macro YY_START returns an integer value 715228072Sbapt corresponding to the current start condition. You can return 716228072Sbapt to that start condition by passing the value to a subsequent 717228072Sbapt "BEGIN" action. You also can implement "start condition stacks" 718228072Sbapt by storing the values in an integer stack. 719228072Sbapt 720228072Sbapt - You can now redefine macros such as YY_INPUT by just #define'ing 721228072Sbapt them to some other value in the first section of the flex input; 722228072Sbapt no need to first #undef them. 723228072Sbapt 724228072Sbapt - flex now generates warnings for rules that can't be matched. 725228072Sbapt These warnings can be turned off using the new '-w' flag. If 726228072Sbapt your scanner uses REJECT then you will not get these warnings. 727228072Sbapt 728228072Sbapt - If you specify the '-s' flag but the default rule can be matched, 729228072Sbapt flex now generates a warning. 730228072Sbapt 731228072Sbapt - "yyleng" is now a global, and may be modified by the user (though 732228072Sbapt doing so and then using yymore() will yield weird results). 733228072Sbapt 734228072Sbapt - Name definitions in the first section of a scanner specification 735228072Sbapt can now include a leading '^' or trailing '$' operator. In this 736228072Sbapt case, the definition is *not* pushed back inside of parentheses. 737228072Sbapt 738228072Sbapt - Scanners with compressed tables are now "interactive" (-I option) 739228072Sbapt by default. You can suppress this attribute (which makes them 740228072Sbapt run slightly slower) using the new '-B' flag. 741228072Sbapt 742228072Sbapt - Flex now generates 8-bit scanners by default, unless you use the 743228072Sbapt -Cf or -CF compression options (-Cfe and -CFe result in 8-bit 744228072Sbapt scanners). You can force it to generate a 7-bit scanner using 745228072Sbapt the new '-7' flag. You can build flex to generate 8-bit scanners 746228072Sbapt for -Cf and -CF, too, by adding -DDEFAULT_CSIZE=256 to CFLAGS 747228072Sbapt in the Makefile. 748228072Sbapt 749228072Sbapt - You no longer need to call the scanner routine yyrestart() to 750228072Sbapt inform the scanner that you have switched to a new file after 751228072Sbapt having seen an EOF on the current input file. Instead, just 752228072Sbapt point yyin at the new file and continue scanning. 753228072Sbapt 754228072Sbapt - You no longer need to invoke YY_NEW_FILE in an <<EOF>> action 755228072Sbapt to indicate you wish to continue scanning. Simply point yyin 756228072Sbapt at a new file. 757228072Sbapt 758228072Sbapt - A leading '#' no longer introduces a comment in a flex input. 759228072Sbapt 760228072Sbapt - flex no longer considers formfeed ('\f') a whitespace character. 761228072Sbapt 762228072Sbapt - %t, I'm happy to report, has been nuked. 763228072Sbapt 764228072Sbapt - The '-p' option may be given twice ('-pp') to instruct flex to 765228072Sbapt report minor performance problems as well as major ones. 766228072Sbapt 767228072Sbapt - The '-v' verbose output no longer includes start/finish time 768228072Sbapt information. 769228072Sbapt 770228072Sbapt - Newlines in flex inputs can optionally include leading or 771228072Sbapt trailing carriage-returns ('\r'), in support of several PC/Mac 772228072Sbapt run-time libraries that automatically include these. 773228072Sbapt 774228072Sbapt - A start condition of the form "<*>" makes the following rule 775228072Sbapt active in every start condition, whether exclusive or inclusive. 776228072Sbapt 777228072Sbapt - The following items have been corrected in the flex documentation: 778228072Sbapt 779228072Sbapt - '-C' table compression options *are* cumulative. 780228072Sbapt 781228072Sbapt - You may modify yytext but not lengthen it by appending 782228072Sbapt characters to the end. Modifying its final character 783228072Sbapt will affect '^' anchoring for the next rule matched 784228072Sbapt if the character is changed to or from a newline. 785228072Sbapt 786228072Sbapt - The term "backtracking" has been renamed "backing up", 787228072Sbapt since it is a one-time repositioning and not a repeated 788228072Sbapt search. What used to be the "lex.backtrack" file is now 789228072Sbapt "lex.backup". 790228072Sbapt 791228072Sbapt - Unindented "/* ... */" comments are allowed in the first 792228072Sbapt flex input section, but not in the second. 793228072Sbapt 794228072Sbapt - yyless() can only be used in the flex input source, not 795228072Sbapt externally. 796228072Sbapt 797228072Sbapt - You can use "yyrestart(yyin)" to throw away the 798228072Sbapt current contents of the input buffer. 799228072Sbapt 800228072Sbapt - To write high-speed scanners, attempt to match as much 801228072Sbapt text as possible with each rule. See MISC/fastwc/README 802228072Sbapt for more information. 803228072Sbapt 804228072Sbapt - Using the beginning-of-line operator ('^') is fairly 805228072Sbapt cheap. Using unput() is expensive. Using yyless() is 806228072Sbapt cheap. 807228072Sbapt 808228072Sbapt - An example of scanning strings with embedded escape 809228072Sbapt sequences has been added. 810228072Sbapt 811228072Sbapt - The example of backing-up in flexdoc was erroneous; it 812228072Sbapt has been corrected. 813228072Sbapt 814228072Sbapt - A flex scanner's internal buffer now dynamically grows if needed 815228072Sbapt to match large tokens. Note that growing the buffer presently 816228072Sbapt requires rescanning the (large) token, so consuming a lot of 817228072Sbapt text this way is a slow process. Also note that presently the 818228072Sbapt buffer does *not* grow if you unput() more text than can fit 819228072Sbapt into the buffer. 820228072Sbapt 821228072Sbapt - The MISC/ directory has been reorganized; see MISC/README for 822228072Sbapt details. 823228072Sbapt 824228072Sbapt - yyless() can now be used in the third (user action) section 825228072Sbapt of a scanner specification, thanks to Ceriel Jacobs. yyless() 826228072Sbapt remains a macro and cannot be used outside of the scanner source. 827228072Sbapt 828228072Sbapt - The skeleton file is no longer opened at run-time, but instead 829228072Sbapt compiled into a large string array (thanks to John Gilmore and 830228072Sbapt friends at Cygnus). You can still use the -S flag to point flex 831228072Sbapt at a different skeleton file. 832228072Sbapt 833228072Sbapt - flex no longer uses a temporary file to store the scanner's 834228072Sbapt actions. 835228072Sbapt 836228072Sbapt - A number of changes have been made to decrease porting headaches. 837228072Sbapt In particular, flex no longer uses memset() or ctime(), and 838228072Sbapt provides a single simple mechanism for dealing with C compilers 839228072Sbapt that still define malloc() as returning char* instead of void*. 840228072Sbapt 841228072Sbapt - Flex now detects if the scanner specification requires the -8 flag 842228072Sbapt but the flag was not given or on by default. 843228072Sbapt 844228072Sbapt - A number of table-expansion fencepost bugs have been fixed, 845228072Sbapt making flex more robust for generating large scanners. 846228072Sbapt 847228072Sbapt - flex more consistently identifies the location of errors in 848228072Sbapt its input. 849228072Sbapt 850228072Sbapt - YY_USER_ACTION is now invoked only for "real" actions, not for 851228072Sbapt internal actions used by the scanner for things like filling 852228072Sbapt the buffer or handling EOF. 853228072Sbapt 854228072Sbapt - The rule "[^]]" now matches any character other than a ']'; 855228072Sbapt formerly it matched any character at all followed by a ']'. 856228072Sbapt This change was made for compatibility with AT&T lex. 857228072Sbapt 858228072Sbapt - A large number of miscellaneous bugs have been found and fixed 859228072Sbapt thanks to Gerhard Wilhelms. 860228072Sbapt 861228072Sbapt - The source code has been heavily reformatted, making patches 862228072Sbapt relative to previous flex releases no longer accurate. 863228072Sbapt 864228072Sbapt 865228072SbaptChanges between 2.3 Patch #8 (21Feb93) and 2.3 Patch #7: 866228072Sbapt 867228072Sbapt - Fixed bugs in dynamic memory allocation leading to grievous 868228072Sbapt fencepost problems when generating large scanners. 869228072Sbapt - Fixed bug causing infinite loops on character classes with 8-bit 870228072Sbapt characters in them. 871228072Sbapt - Fixed bug in matching repetitions with a lower bound of 0. 872228072Sbapt - Fixed bug in scanning NUL characters using an "interactive" scanner. 873228072Sbapt - Fixed bug in using yymore() at the end of a file. 874228072Sbapt - Fixed bug in misrecognizing rules with variable trailing context. 875228072Sbapt - Fixed bug compiling flex on Suns using gcc 2. 876228072Sbapt - Fixed bug in not recognizing that input files with the character 877228072Sbapt ASCII 128 in them require the -8 flag. 878228072Sbapt - Fixed bug that could cause an infinite loop writing out 879228072Sbapt error messages. 880228072Sbapt - Fixed bug in not recognizing old-style lex % declarations if 881228072Sbapt followed by a tab instead of a space. 882228072Sbapt - Fixed potential crash when flex terminated early (usually due 883228072Sbapt to a bad flag) and the -v flag had been given. 884228072Sbapt - Added some missing declarations of void functions. 885228072Sbapt - Changed to only use '\a' for __STDC__ compilers. 886228072Sbapt - Updated mailing addresses. 887228072Sbapt 888228072Sbapt 889228072SbaptChanges between 2.3 Patch #7 (28Mar91) and 2.3 Patch #6: 890228072Sbapt 891228072Sbapt - Fixed out-of-bounds array access that caused bad tables 892228072Sbapt to be produced on machines where the bad reference happened 893228072Sbapt to yield a 1. This caused problems installing or running 894228072Sbapt flex on some Suns, in particular. 895228072Sbapt 896228072Sbapt 897228072SbaptChanges between 2.3 Patch #6 (29Aug90) and 2.3 Patch #5: 898228072Sbapt 899228072Sbapt - Fixed a serious bug in yymore() which basically made it 900228072Sbapt completely broken. Thanks goes to Jean Christophe of 901228072Sbapt the Nethack development team for finding the problem 902228072Sbapt and passing along the fix. 903228072Sbapt 904228072Sbapt 905228072SbaptChanges between 2.3 Patch #5 (16Aug90) and 2.3 Patch #4: 906228072Sbapt 907228072Sbapt - An up-to-date version of initscan.c so "make test" will 908228072Sbapt work after applying the previous patches 909228072Sbapt 910228072Sbapt 911228072SbaptChanges between 2.3 Patch #4 (14Aug90) and 2.3 Patch #3: 912228072Sbapt 913228072Sbapt - Fixed bug in hexadecimal escapes which allowed only digits, 914228072Sbapt not letters, in escapes 915228072Sbapt - Fixed bug in previous "Changes" file! 916228072Sbapt 917228072Sbapt 918228072SbaptChanges between 2.3 Patch #3 (03Aug90) and 2.3 Patch #2: 919228072Sbapt 920228072Sbapt - Correction to patch #2 for gcc compilation; thanks goes to 921228072Sbapt Paul Eggert for catching this. 922228072Sbapt 923228072Sbapt 924228072SbaptChanges between 2.3 Patch #2 (02Aug90) and original 2.3 release: 925228072Sbapt 926228072Sbapt - Fixed (hopefully) headaches involving declaring malloc() 927228072Sbapt and free() for gcc, which defines __STDC__ but (often) doesn't 928228072Sbapt come with the standard include files such as <stdlib.h>. 929228072Sbapt Reordered #ifdef maze in the scanner skeleton in the hope of 930228072Sbapt getting the declarations right for cfront and g++, too. 931228072Sbapt 932228072Sbapt - Note that this patch supercedes patch #1 for release 2.3, 933228072Sbapt which was never announced but was available briefly for 934228072Sbapt anonymous ftp. 935228072Sbapt 936228072Sbapt 937228072SbaptChanges between 2.3 (full) release of 28Jun90 and 2.2 (alpha) release: 938228072Sbapt 939228072SbaptUser-visible: 940228072Sbapt 941228072Sbapt - A lone <<EOF>> rule (that is, one which is not qualified with 942228072Sbapt a list of start conditions) now specifies the EOF action for 943228072Sbapt *all* start conditions which haven't already had <<EOF>> actions 944228072Sbapt given. To specify an end-of-file action for just the initial 945228072Sbapt state, use <INITIAL><<EOF>>. 946228072Sbapt 947250874Sjkim - -d debug output is now contingent on the global yy_flex_debug 948228072Sbapt being set to a non-zero value, which it is by default. 949228072Sbapt 950228072Sbapt - A new macro, YY_USER_INIT, is provided for the user to specify 951228072Sbapt initialization action to be taken on the first call to the 952228072Sbapt scanner. This action is done before the scanner does its 953228072Sbapt own initialization. 954228072Sbapt 955228072Sbapt - yy_new_buffer() has been added as an alias for yy_create_buffer() 956228072Sbapt 957228072Sbapt - Comments beginning with '#' and extending to the end of the line 958228072Sbapt now work, but have been deprecated (in anticipation of making 959228072Sbapt flex recognize #line directives). 960228072Sbapt 961228072Sbapt - The funky restrictions on when semi-colons could follow the 962228072Sbapt YY_NEW_FILE and yyless macros have been removed. They now 963228072Sbapt behave identically to functions. 964228072Sbapt 965228072Sbapt - A bug in the sample redefinition of YY_INPUT in the documentation 966228072Sbapt has been corrected. 967228072Sbapt 968228072Sbapt - A bug in the sample simple tokener in the documentation has 969228072Sbapt been corrected. 970228072Sbapt 971228072Sbapt - The documentation on the incompatibilities between flex and 972228072Sbapt lex has been reordered so that the discussion of yylineno 973228072Sbapt and input() come first, as it's anticipated that these will 974228072Sbapt be the most common source of headaches. 975228072Sbapt 976228072Sbapt 977228072SbaptThings which didn't used to be documented but now are: 978228072Sbapt 979228072Sbapt - flex interprets "^foo|bar" differently from lex. flex interprets 980228072Sbapt it as "match either a 'foo' or a 'bar', providing it comes at the 981228072Sbapt beginning of a line", whereas lex interprets it as "match either 982228072Sbapt a 'foo' at the beginning of a line, or a 'bar' anywhere". 983228072Sbapt 984228072Sbapt - flex initializes the global "yyin" on the first call to the 985228072Sbapt scanner, while lex initializes it at compile-time. 986228072Sbapt 987228072Sbapt - yy_switch_to_buffer() can be used in the yywrap() macro/routine. 988228072Sbapt 989228072Sbapt - flex scanners do not use stdio for their input, and hence when 990250874Sjkim writing an interactive scanner one must explicitly call fflush() 991228072Sbapt after writing out a prompt. 992228072Sbapt 993228072Sbapt - flex scanner can be made reentrant (after a fashion) by using 994228072Sbapt "yyrestart( yyin );". This is useful for interactive scanners 995228072Sbapt which have interrupt handlers that long-jump out of the scanner. 996228072Sbapt 997228072Sbapt - a defense of why yylineno is not supported is included, along 998228072Sbapt with a suggestion on how to convert scanners which rely on it. 999228072Sbapt 1000228072Sbapt 1001228072SbaptOther changes: 1002228072Sbapt 1003228072Sbapt - Prototypes and proper declarations of void routines have 1004228072Sbapt been added to the flex source code, courtesy of Kevin B. Kenny. 1005228072Sbapt 1006228072Sbapt - Routines dealing with memory allocation now use void* pointers 1007228072Sbapt instead of char* - see Makefile for porting implications. 1008228072Sbapt 1009228072Sbapt - Error-checking is now done when flex closes a file. 1010228072Sbapt 1011228072Sbapt - Various lint tweaks were added to reduce the number of gripes. 1012228072Sbapt 1013228072Sbapt - Makefile has been further parameterized to aid in porting. 1014228072Sbapt 1015228072Sbapt - Support for SCO Unix added. 1016228072Sbapt 1017228072Sbapt - Flex now sports the latest & greatest UC copyright notice 1018228072Sbapt (which is only slightly different from the previous one). 1019228072Sbapt 1020228072Sbapt - A note has been added to flexdoc.1 mentioning work in progress 1021228072Sbapt on modifying flex to generate straight C code rather than a 1022228072Sbapt table-driven automaton, with an email address of whom to contact 1023228072Sbapt if you are working along similar lines. 1024228072Sbapt 1025228072Sbapt 1026228072SbaptChanges between 2.2 Patch #3 (30Mar90) and 2.2 Patch #2: 1027228072Sbapt 1028228072Sbapt - fixed bug which caused -I scanners to bomb 1029228072Sbapt 1030228072Sbapt 1031228072SbaptChanges between 2.2 Patch #2 (27Mar90) and 2.2 Patch #1: 1032228072Sbapt 1033228072Sbapt - fixed bug writing past end of input buffer in yyunput() 1034228072Sbapt - fixed bug detecting NUL's at the end of a buffer 1035228072Sbapt 1036228072Sbapt 1037228072SbaptChanges between 2.2 Patch #1 (23Mar90) and 2.2 (alpha) release: 1038228072Sbapt 1039228072Sbapt - Makefile fixes: definition of MAKE variable for systems 1040228072Sbapt which don't have it; installation of flexdoc.1 along with 1041228072Sbapt flex.1; fixed two bugs which could cause "bigtest" to fail. 1042228072Sbapt 1043228072Sbapt - flex.skel fix for compiling with g++. 1044228072Sbapt 1045228072Sbapt - README and flexdoc.1 no longer list an out-of-date BITNET address 1046228072Sbapt for contacting me. 1047228072Sbapt 1048228072Sbapt - minor typos and formatting changes to flex.1 and flexdoc.1. 1049228072Sbapt 1050228072Sbapt 1051228072SbaptChanges between 2.2 (alpha) release of March '90 and previous release: 1052228072Sbapt 1053228072SbaptUser-visible: 1054228072Sbapt 1055228072Sbapt - Full user documentation now available. 1056228072Sbapt 1057228072Sbapt - Support for 8-bit scanners. 1058228072Sbapt 1059228072Sbapt - Scanners now accept NUL's. 1060228072Sbapt 1061228072Sbapt - A facility has been added for dealing with multiple 1062228072Sbapt input buffers. 1063228072Sbapt 1064228072Sbapt - Two manual entries now. One which fully describes flex 1065228072Sbapt (rather than just its differences from lex), and the 1066228072Sbapt other for quick(er) reference. 1067228072Sbapt 1068228072Sbapt - A number of changes to bring flex closer into compliance 1069228072Sbapt with the latest POSIX lex draft: 1070228072Sbapt 1071228072Sbapt %t support 1072228072Sbapt flex now accepts multiple input files and concatenates 1073228072Sbapt them together to form its input 1074228072Sbapt previous -c (compress) flag renamed -C 1075228072Sbapt do-nothing -c and -n flags added 1076228072Sbapt Any indented code or code within %{}'s in section 2 is 1077228072Sbapt now copied to the output 1078228072Sbapt 1079228072Sbapt - yyleng is now a bona fide global integer. 1080228072Sbapt 1081228072Sbapt - -d debug information now gives the line number of the 1082228072Sbapt matched rule instead of which number rule it was from 1083228072Sbapt the beginning of the file. 1084228072Sbapt 1085228072Sbapt - -v output now includes a summary of the flags used to generate 1086228072Sbapt the scanner. 1087228072Sbapt 1088228072Sbapt - unput() and yyrestart() are now globally callable. 1089228072Sbapt 1090228072Sbapt - yyrestart() no longer closes the previous value of yyin. 1091228072Sbapt 1092228072Sbapt - C++ support; generated scanners can be compiled with C++ compiler. 1093228072Sbapt 1094228072Sbapt - Primitive -lfl library added, containing default main() 1095228072Sbapt which calls yylex(). A number of routines currently living 1096228072Sbapt in the scanner skeleton will probably migrate to here 1097228072Sbapt in the future (in particular, yywrap() will probably cease 1098228072Sbapt to be a macro and instead be a function in the -lfl library). 1099228072Sbapt 1100228072Sbapt - Hexadecimal (\x) escape sequences added. 1101228072Sbapt 1102228072Sbapt - Support for MS-DOS, VMS, and Turbo-C integrated. 1103228072Sbapt 1104228072Sbapt - The %used/%unused operators have been deprecated. They 1105228072Sbapt may go away soon. 1106228072Sbapt 1107228072Sbapt 1108228072SbaptOther changes: 1109228072Sbapt 1110228072Sbapt - Makefile enhanced for easier testing and installation. 1111228072Sbapt - The parser has been tweaked to detect some erroneous 1112228072Sbapt constructions which previously were missed. 1113228072Sbapt - Scanner input buffer overflow is now detected. 1114228072Sbapt - Bugs with missing "const" declarations fixed. 1115228072Sbapt - Out-of-date Minix/Atari patches provided. 1116228072Sbapt - Scanners no longer require printf() unless FLEX_DEBUG is being used. 1117228072Sbapt - A subtle input() bug has been fixed. 1118228072Sbapt - Line numbers for "continued action" rules (those following 1119228072Sbapt the special '|' action) are now correct. 1120228072Sbapt - unput() bug fixed; had been causing problems porting flex to VMS. 1121228072Sbapt - yymore() handling rewritten to fix bug with interaction 1122228072Sbapt between yymore() and trailing context. 1123228072Sbapt - EOF in actions now generates an error message. 1124228072Sbapt - Bug involving -CFe and generating equivalence classes fixed. 1125228072Sbapt - Bug which made -CF be treated as -Cf fixed. 1126228072Sbapt - Support for SysV tmpnam() added. 1127228072Sbapt - Unused #define's for scanner no longer generated. 1128228072Sbapt - Error messages which are associated with a particular input 1129228072Sbapt line are now all identified with their input line in standard 1130228072Sbapt format. 1131228072Sbapt - % directives which are valid to lex but not to flex are 1132228072Sbapt now ignored instead of generating warnings. 1133228072Sbapt - -DSYS_V flag can now also be specified -DUSG for System V 1134228072Sbapt compilation. 1135228072Sbapt 1136228072Sbapt 1137228072SbaptChanges between 2.1 beta-test release of June '89 and previous release: 1138228072Sbapt 1139228072SbaptUser-visible: 1140228072Sbapt 1141228072Sbapt - -p flag generates a performance report to stderr. The report 1142228072Sbapt consists of comments regarding features of the scanner rules 1143228072Sbapt which result in slower scanners. 1144228072Sbapt 1145228072Sbapt - -b flag generates backtracking information to lex.backtrack. 1146228072Sbapt This is a list of scanner states which require backtracking 1147228072Sbapt and the characters on which they do so. By adding rules 1148228072Sbapt one can remove backtracking states. If all backtracking states 1149228072Sbapt are eliminated, the generated scanner will run faster. 1150228072Sbapt Backtracking is not yet documented in the manual entry. 1151228072Sbapt 1152228072Sbapt - Variable trailing context now works, i.e., one can have 1153228072Sbapt rules like "(foo)*/[ \t]*bletch". Some trailing context 1154228072Sbapt patterns still cannot be properly matched and generate 1155228072Sbapt error messages. These are patterns where the ending of the 1156228072Sbapt first part of the rule matches the beginning of the second 1157228072Sbapt part, such as "zx*/xy*", where the 'x*' matches the 'x' at 1158228072Sbapt the beginning of the trailing context. Lex won't get these 1159228072Sbapt patterns right either. 1160228072Sbapt 1161228072Sbapt - Faster scanners. 1162228072Sbapt 1163228072Sbapt - End-of-file rules. The special rule "<<EOF>>" indicates 1164228072Sbapt actions which are to be taken when an end-of-file is 1165228072Sbapt encountered and yywrap() returns non-zero (i.e., indicates 1166228072Sbapt no further files to process). See manual entry for example. 1167228072Sbapt 1168228072Sbapt - The -r (reject used) flag is gone. flex now scans the input 1169228072Sbapt for occurrences of the string "REJECT" to determine if the 1170228072Sbapt action is needed. It tries to be intelligent about this but 1171228072Sbapt can be fooled. One can force the presence or absence of 1172228072Sbapt REJECT by adding a line in the first section of the form 1173228072Sbapt "%used REJECT" or "%unused REJECT". 1174228072Sbapt 1175228072Sbapt - yymore() has been implemented. Similarly to REJECT, flex 1176228072Sbapt detects the use of yymore(), which can be overridden using 1177228072Sbapt "%used" or "%unused". 1178228072Sbapt 1179228072Sbapt - Patterns like "x{0,3}" now work (i.e., with lower-limit == 0). 1180228072Sbapt 1181228072Sbapt - Removed '\^x' for ctrl-x misfeature. 1182228072Sbapt 1183228072Sbapt - Added '\a' and '\v' escape sequences. 1184228072Sbapt 1185228072Sbapt - \<digits> now works for octal escape sequences; previously 1186228072Sbapt \0<digits> was required. 1187228072Sbapt 1188228072Sbapt - Better error reporting; line numbers are associated with rules. 1189228072Sbapt 1190228072Sbapt - yyleng is a macro; it cannot be accessed outside of the 1191228072Sbapt scanner source file. 1192228072Sbapt 1193228072Sbapt - yytext and yyleng should not be modified within a flex action. 1194228072Sbapt 1195228072Sbapt - Generated scanners #define the name FLEX_SCANNER. 1196228072Sbapt 1197228072Sbapt - Rules are internally separated by YY_BREAK in lex.yy.c rather 1198228072Sbapt than break, to allow redefinition. 1199228072Sbapt 1200228072Sbapt - The macro YY_USER_ACTION can be redefined to provide an action 1201228072Sbapt which is always executed prior to the matched rule's action. 1202228072Sbapt 1203228072Sbapt - yyrestart() is a new action which can be used to restart 1204228072Sbapt the scanner after it has seen an end-of-file (a "real" one, 1205228072Sbapt that is, one for which yywrap() returned non-zero). It takes 1206228072Sbapt a FILE* argument indicating a new file to scan and sets 1207228072Sbapt things up so that a subsequent call to yylex() will start 1208228072Sbapt scanning that file. 1209228072Sbapt 1210228072Sbapt - Internal scanner names all preceded by "yy_" 1211228072Sbapt 1212228072Sbapt - lex.yy.c is deleted if errors are encountered during processing. 1213228072Sbapt 1214228072Sbapt - Comments may be put in the first section of the input by preceding 1215228072Sbapt them with '#'. 1216228072Sbapt 1217228072Sbapt 1218228072Sbapt 1219228072SbaptOther changes: 1220228072Sbapt 1221228072Sbapt - Some portability-related bugs fixed, in particular for machines 1222228072Sbapt with unsigned characters or sizeof( int* ) != sizeof( int ). 1223228072Sbapt Also, tweaks for VMS and Microsoft C (MS-DOS), and identifiers all 1224228072Sbapt trimmed to be 31 or fewer characters. Shortened file names 1225228072Sbapt for dinosaur OS's. Checks for allocating > 64K memory 1226228072Sbapt on 16 bit'ers. Amiga tweaks. Compiles using gcc on a Sun-3. 1227228072Sbapt - Compressed and fast scanner skeletons merged. 1228228072Sbapt - Skeleton header files done away with. 1229228072Sbapt - Generated scanner uses prototypes and "const" for __STDC__. 1230228072Sbapt - -DSV flag is now -DSYS_V for System V compilation. 1231228072Sbapt - Removed all references to FTL language. 1232228072Sbapt - Software now covered by BSD Copyright. 1233228072Sbapt - flex will replace lex in subsequent BSD releases. 1234