remove prototypes with no matching function and externs with no var
partly checked by millert@

Add -E flag (make warnings fatal), following the behavior of GNU m4 1.4.9+
Help and direction millert@ espie@ anton@ deraadt@

ok espie@

adjust to ohash being in libutil now, and to the interface changes.
fix potential integer overflows in memory allocation (mostly for pedagogical
purposes, these are unlikely to overflow in practice)
move the rest of lst.lib stuff into its own directory.

check for integer overflows in custom allocs, okay jca@

new m4 -g stuff:
- expr(`4**3')
- include(`hey I am not there') keeps going.

work with Baptiste Daroussin, who had the idea but didn't nail all details
right.

okay otto@, miod@

zap dead stuff
from Michael W. Bombardieri

Remove stray spaces. No code change.

Support the -P option from gnu m4, which prefixes builtins with the
string m4_. Feedback from jmc@, Tobias Ulmer, Matthew Dempsky - thanks!
ok espie@

use unsigned char for buffers, allow differentiating between EOF and y".

okay otto@

not yet, committed too much

argument parsing should only skip spaces outside of parenthesis.
Inside matching parenthesis, keep spaces as is (use chrsave instead of
pbstr, since there's no way it can be a further macro expansion).
Fixes a long-standing issue with autoconf ( --option -> --option),
matches other m4 than gnum4

okay millert@, fries@

add limited support for format builtin in gnu-m4 mode, because I'm fed
up of patching it away in various autoconf derivatives.

okay miod@

report function m4errx, that shows app-specific data (filename/linenumber)
before the error message. Use it to simplify code.

okay miod@

use stdint.h where appropriate. okay millert@

finally make our m4 SusV3-compliant.

- changecom and changequote have a simple definition (that matches gnu-m4,
coincidentally, so we no longer need two distinct modes for these)

- off-by-one bug in -s, this finally works.

- reorder main parser loop, so that we can use alphabetic constructs in
quotes/comments.

- rename putback to pushback, this matches comments, and makes more sense.

- more uniform (and updated) description of changequote/changecom.

- new, systematic regression tests of comments/quotes.

- framework to test -s: one perl script to reconstitute `full' files with
all line numbers, so that we can verify the output without needing a
complete match.

okay otto@, fries@

fix a few minor compatibility issues.
- let eval() handle a base and number of digits, like it's advertized to.
- in gnu-mode, undivert can take file names as arguments.
- in gnu-mode, map can handle reversed charsets.

Suggestions and okay otto@, mostly prompted by looking at the regress
tests in newer gnu-m4.

let m4wrap handle multiple wraps, both in normal and gnu-mode.
based on Noah Misch's bug report.
okay otto, jmc.

Modify xalloc so that it also takes err(3)-like arguments.
Write an xrealloc wrapper that works the same way, and use it as well.

People who feel like it may want to add more explicit error messages to
all the places m4 can fail allocating memory...

okay tedu@

Fold trace status into the single hash table that's left.
Inline some macros/functions for speed.
So, this achieves the goal of one single lookup for macro/trace status,
which does speed up m4 in partial tracing situations somewhat.
This does also speed up m4 in large pushdef situations, since it no longer
has to lookup large chains of macros.

okay millert@

add a flag for each macro name that records built-in status.
Fold built-in lookup into normal lookup.
okay millert@

replace old hash structure with open hashing.
make the stack structure of macro definitions explicit.
okay millert@

Make the trace status of a macro an actual argument that gets pushed
in the frame for the macro expansion.

(This will allow one single lookup to grab the macro definition and
the trace status)

okay millert@

clean up internal lookup interface:
define an interface with explicit define/pushdef/popdef... and use it.
That way, most details of the hashtable are no longer visible.

okay millert@

- store builtin name as definition for builtin macros.
this removes the need for code->name conversion, in exchange for
systematically testing the definition type, since we can no longer
rely on the defn being NULL.

- commonnalize the builtin-detection code, so that we can use it for pushdef
as well with define, so that pushdef handles builtins correctly as well.

okay fries@, millert@.

switch from linked list to hash table for traced macros.
speeds up recent autoconf somewhat, since it traces a large set
of individual macro.

(more rework of m4 internal interfaces to unify lookup tables in order)

okay fries@

Remove the advertising clause in the UCB license which Berkeley
rescinded 22 July 1999. Proofed by myself and Theo.

Implement -s.
Triggered by recent FreeBSD changes.

- emits #line directives at every file change (like FreeBSD)
- maintains a synch_lineno variable to verify when the output gets out
of synch with the input, so that it can emit #line to re-synch as well
(unlike FreeBSD)

To do: either handle \end-of-line, or recognize when a macro expansion
is in progress, so that line synch don't perturb cpp on multi-line
expansions.

With this, we should have a fully POSIX-compliant m4.

ok miod@

Part one of userland __P removal. Done with a simple regexp with some minor hand editing to make comments line up correctly. Another pass is forthcoming that handles the cases that could not be done automatically.

Prototypes shouldn't mention argument name.

traceon/traceoff built-ins.

inline some very common putback/chrsave. Worth roughly 10%

Tracing facilities. Same style as gnu-m4, because it's mostly used for
autoconf right now.

instrumentation for tracing mode: macro expansion

let defn(builtin) work enough so that
define(`newmacro', defn(builtin))
will work, as it should.

One single point for all macros/builtin expansion.

Handle unlimited diverts for m4 -g.

Problem with autoconf noticed by d@

Implement esyscmd

Make the m4 machine stack dynamically sized.

Fix strspace automatic extension.
The assumption that simply updating the current pointer works is false,
there are cases where previous entries on the stack would absorp vast
amounts of string space, and overload the non-updated entries.

To fix it, we use a shadow copy of the stack, which only records which
entries are pointers within strspace, so that a resize can adjust all
those pointers at once.

Reviewed by millert@

Last needed m4 changes to bootstrap autoconf without gnu-m4.

Add a few builtins for greater compatibility with gnu-m4, and extended
functionality.
* regular expressions,
* line-number reporting
* `meta'-macros, builtin and indir.

Reviewed by pjanzen@, tested by fries@ and a few others.

There is no need to waste short for holding EOF in the pushback buffer.
Now that the input_file structure is sufficiently fleshed out, just stop
EOF at the putback level, and make sure files at EOF STAY at EOF.

Use a proper abstract interface for file accesses, so that we can record
the file name and line number.

This yields more meaningful error messages, and the possibility for yet
more.

Don't recognize built-ins in contexts where they don't make sense.
Namely, it doesn't help to try and expand include if it's not followed
by parenthesis and a filename.

This should make applications like sendmail m4 scripts more sturdy for
unquoted machine names that happen to collide with built-ins.

The only drawback is that our m4 may now do intelligent things with scripts
that don't work on other systems.

Clean up comment/quote recognition a little bit:
- use strlcpy to make clear that the strings are line terminated,
- remove the number of magic constants,
- use assert() for preconditions,
- use puts instead of looping over array of chars...

Optimization: cache the hashed value to avoid negative comparisons.
With 2^32 possible hash values, this means that collisions no longer
incur supplementary string compares, which was most of the reason for
STREQ in the first place...

Turn warnings on,
Add missing prototypes,
Make local functions static,
Sort extern.h by file,
Constify all char * that can be,
Copy temp file name so that eval does not modify its arguments.

Style: kill register and indent properly.
Let indx match netbsd flavor, to simplify diffs.
Show how many quotes were not closed.
Increase stack slightly, now that we're no longer bound by argspace.

Implement -I option and M4PATH environment variable

Remove STRSPACE limitations.
A bit wasteful, but not too intrusive.

Also remove pushback buffer limitations, as this would be mostly useless
otherwise.

Incidentally, pushback buffer overflow detection in pbstr was wrong.

Get rid of mktemp in the handling of diversion.

Simply put, mkstemp/unlink/rewind has the proper semantics under Unix,
and so we don't have to keep track about temp file names and remove them.

synch

Misc minor changes:
- use err.h and kill oops,
- use __progname and kill basename,
- let indx use strstr
- proper EOS decl

cgd: add a typedef which describes elements in the push-back buffer. Make
that typedef 'short'. 'char' (which was previously used) because char
may be unsigned and ((char)EOF) != EOF if that is the case. That was
causing the (char)EOF (0xff) pushed back in main to be interepreted as
a character, and, in some cases, to be written to the output. 'short'
was used rather than 'signed char' because if the latter is used,
0xff characters in the input would confuse m4. (No point in introducing
(more?) 8-bit lossage.)

rcsid

from netbsd:
Handle multichar comment and quote delimiters (up to 5 characters, per the
manual page). Takes care of PR#485.

branches: 1.1.1;
Initial revision

Add -E flag (make warnings fatal), following the behavior of GNU m4 1.4.9+
Help and direction millert@ espie@ anton@ deraadt@

ok espie@

adjust to ohash being in libutil now, and to the interface changes.
fix potential integer overflows in memory allocation (mostly for pedagogical
purposes, these are unlikely to overflow in practice)
move the rest of lst.lib stuff into its own directory.

check for integer overflows in custom allocs, okay jca@

new m4 -g stuff:
- expr(`4**3')
- include(`hey I am not there') keeps going.

work with Baptiste Daroussin, who had the idea but didn't nail all details
right.

okay otto@, miod@

zap dead stuff
from Michael W. Bombardieri

Remove stray spaces. No code change.

Support the -P option from gnu m4, which prefixes builtins with the
string m4_. Feedback from jmc@, Tobias Ulmer, Matthew Dempsky - thanks!
ok espie@

use unsigned char for buffers, allow differentiating between EOF and y".

okay otto@

not yet, committed too much

argument parsing should only skip spaces outside of parenthesis.
Inside matching parenthesis, keep spaces as is (use chrsave instead of
pbstr, since there's no way it can be a further macro expansion).
Fixes a long-standing issue with autoconf ( --option -> --option),
matches other m4 than gnum4

okay millert@, fries@

add limited support for format builtin in gnu-m4 mode, because I'm fed
up of patching it away in various autoconf derivatives.

okay miod@

report function m4errx, that shows app-specific data (filename/linenumber)
before the error message. Use it to simplify code.

okay miod@

use stdint.h where appropriate. okay millert@

finally make our m4 SusV3-compliant.

- changecom and changequote have a simple definition (that matches gnu-m4,
coincidentally, so we no longer need two distinct modes for these)

- off-by-one bug in -s, this finally works.

- reorder main parser loop, so that we can use alphabetic constructs in
quotes/comments.

- rename putback to pushback, this matches comments, and makes more sense.

- more uniform (and updated) description of changequote/changecom.

- new, systematic regression tests of comments/quotes.

- framework to test -s: one perl script to reconstitute `full' files with
all line numbers, so that we can verify the output without needing a
complete match.

okay otto@, fries@

fix a few minor compatibility issues.
- let eval() handle a base and number of digits, like it's advertized to.
- in gnu-mode, undivert can take file names as arguments.
- in gnu-mode, map can handle reversed charsets.

Suggestions and okay otto@, mostly prompted by looking at the regress
tests in newer gnu-m4.

let m4wrap handle multiple wraps, both in normal and gnu-mode.
based on Noah Misch's bug report.
okay otto, jmc.

Modify xalloc so that it also takes err(3)-like arguments.
Write an xrealloc wrapper that works the same way, and use it as well.

People who feel like it may want to add more explicit error messages to
all the places m4 can fail allocating memory...

okay tedu@

Fold trace status into the single hash table that's left.
Inline some macros/functions for speed.
So, this achieves the goal of one single lookup for macro/trace status,
which does speed up m4 in partial tracing situations somewhat.
This does also speed up m4 in large pushdef situations, since it no longer
has to lookup large chains of macros.

okay millert@

add a flag for each macro name that records built-in status.
Fold built-in lookup into normal lookup.
okay millert@

replace old hash structure with open hashing.
make the stack structure of macro definitions explicit.
okay millert@

Make the trace status of a macro an actual argument that gets pushed
in the frame for the macro expansion.

(This will allow one single lookup to grab the macro definition and
the trace status)

okay millert@

clean up internal lookup interface:
define an interface with explicit define/pushdef/popdef... and use it.
That way, most details of the hashtable are no longer visible.

okay millert@

- store builtin name as definition for builtin macros.
this removes the need for code->name conversion, in exchange for
systematically testing the definition type, since we can no longer
rely on the defn being NULL.

- commonnalize the builtin-detection code, so that we can use it for pushdef
as well with define, so that pushdef handles builtins correctly as well.

okay fries@, millert@.

switch from linked list to hash table for traced macros.
speeds up recent autoconf somewhat, since it traces a large set
of individual macro.

(more rework of m4 internal interfaces to unify lookup tables in order)

okay fries@

Remove the advertising clause in the UCB license which Berkeley
rescinded 22 July 1999. Proofed by myself and Theo.

Implement -s.
Triggered by recent FreeBSD changes.

- emits #line directives at every file change (like FreeBSD)
- maintains a synch_lineno variable to verify when the output gets out
of synch with the input, so that it can emit #line to re-synch as well
(unlike FreeBSD)

To do: either handle \end-of-line, or recognize when a macro expansion
is in progress, so that line synch don't perturb cpp on multi-line
expansions.

With this, we should have a fully POSIX-compliant m4.

ok miod@

Part one of userland __P removal. Done with a simple regexp with some minor hand editing to make comments line up correctly. Another pass is forthcoming that handles the cases that could not be done automatically.

Prototypes shouldn't mention argument name.

traceon/traceoff built-ins.

inline some very common putback/chrsave. Worth roughly 10%

Tracing facilities. Same style as gnu-m4, because it's mostly used for
autoconf right now.

instrumentation for tracing mode: macro expansion

let defn(builtin) work enough so that
define(`newmacro', defn(builtin))
will work, as it should.

One single point for all macros/builtin expansion.

Handle unlimited diverts for m4 -g.

Problem with autoconf noticed by d@

Implement esyscmd

Make the m4 machine stack dynamically sized.

Fix strspace automatic extension.
The assumption that simply updating the current pointer works is false,
there are cases where previous entries on the stack would absorp vast
amounts of string space, and overload the non-updated entries.

To fix it, we use a shadow copy of the stack, which only records which
entries are pointers within strspace, so that a resize can adjust all
those pointers at once.

Reviewed by millert@

Last needed m4 changes to bootstrap autoconf without gnu-m4.

Add a few builtins for greater compatibility with gnu-m4, and extended
functionality.
* regular expressions,
* line-number reporting
* `meta'-macros, builtin and indir.

Reviewed by pjanzen@, tested by fries@ and a few others.

There is no need to waste short for holding EOF in the pushback buffer.
Now that the input_file structure is sufficiently fleshed out, just stop
EOF at the putback level, and make sure files at EOF STAY at EOF.

Use a proper abstract interface for file accesses, so that we can record
the file name and line number.

This yields more meaningful error messages, and the possibility for yet
more.

Don't recognize built-ins in contexts where they don't make sense.
Namely, it doesn't help to try and expand include if it's not followed
by parenthesis and a filename.

This should make applications like sendmail m4 scripts more sturdy for
unquoted machine names that happen to collide with built-ins.

The only drawback is that our m4 may now do intelligent things with scripts
that don't work on other systems.

Clean up comment/quote recognition a little bit:
- use strlcpy to make clear that the strings are line terminated,
- remove the number of magic constants,
- use assert() for preconditions,
- use puts instead of looping over array of chars...

Optimization: cache the hashed value to avoid negative comparisons.
With 2^32 possible hash values, this means that collisions no longer
incur supplementary string compares, which was most of the reason for
STREQ in the first place...

Turn warnings on,
Add missing prototypes,
Make local functions static,
Sort extern.h by file,
Constify all char * that can be,
Copy temp file name so that eval does not modify its arguments.

Style: kill register and indent properly.
Let indx match netbsd flavor, to simplify diffs.
Show how many quotes were not closed.
Increase stack slightly, now that we're no longer bound by argspace.

Implement -I option and M4PATH environment variable

Remove STRSPACE limitations.
A bit wasteful, but not too intrusive.

Also remove pushback buffer limitations, as this would be mostly useless
otherwise.

Incidentally, pushback buffer overflow detection in pbstr was wrong.

Get rid of mktemp in the handling of diversion.

Simply put, mkstemp/unlink/rewind has the proper semantics under Unix,
and so we don't have to keep track about temp file names and remove them.

synch

Misc minor changes:
- use err.h and kill oops,
- use __progname and kill basename,
- let indx use strstr
- proper EOS decl

cgd: add a typedef which describes elements in the push-back buffer. Make
that typedef 'short'. 'char' (which was previously used) because char
may be unsigned and ((char)EOF) != EOF if that is the case. That was
causing the (char)EOF (0xff) pushed back in main to be interepreted as
a character, and, in some cases, to be written to the output. 'short'
was used rather than 'signed char' because if the latter is used,
0xff characters in the input would confuse m4. (No point in introducing
(more?) 8-bit lossage.)

rcsid

from netbsd:
Handle multichar comment and quote delimiters (up to 5 characters, per the
manual page). Takes care of PR#485.

branches: 1.1.1;
Initial revision