PR bin/48875 (is related, and ameliorated, but not exactly "fixed")

Import a whole set of tree evaluation enhancements from FreeBSD.

With these, before forking, the shell predicts (often) when all it will
have to do after forking (in the parent) is wait for the child and then
exit with the status from the child, and in such a case simply does not
fork, but rather allows the child to take over the parent's role.

This turns out to handle the particular test case from PR bin/48875 in
such a way that it works as hoped, rather than as it did (the delay there
was caused by an extra copy of the shell hanging around waiting for the
background child to complete ... and keeping the command substitution
stdout open, so the "real" parent had to wait in case more output appeared).

As part of doing this, redirection processing for compound commands gets
moved out of evalsubshell() and into a new evalredir(), which allows us
to properly handle errors occurring while performing those redirects,
and not mishandle (as in simply forget) fd's which had been moved out
of the way temporarily.

evaltree() has its degree of recursion reduced by making it loop to
handle the subsequent operation: that is instead of (for any binop
like ';' '&&' (etc)) where it used to
evaltree(node->left);
evaltree(node->right);
return;
it now does (kind of)
next = node;
while ((node = next) != NULL) {
next = NULL;

if (node is a binary op) {
evaltree(node->left);
if appropriate /* if && test for success, etc */
next = node->right;
continue;
}
/* similar for loops, etc */
}
which can be a good saving, as while the left side (now) tends to be
(usually) a simple (or simpleish) command, the right side can be many
commands (in a command sequence like a; b; c; d; ... the node at the
top of the tree will now have "a" as its left node, and the tree for
b; c; d; ... as its right node - until now everything was evaluated
recursively so it made no difference, and the tree was constructed
the other way).

if/while/... statements are done similarly, recurse to evaluate the
condition, then if the (or one of the) body parts is to be evaluated,
set next to that, and loop (previously it recursed).

There is more to do in this area (particularly in the way that case
statements are processed - we can avoid recursion there as well) but
that can wait for another day.

While doing all of this we keep much better track of when the shell is
just going to exit once the current tree is evaluated (with a new
predicate at_eof() to tell us that we have, for sure, reached the end
of the input stream, that is, this shell will, for certain, not be reading
more command input) and use that info to avoid unneeded forks. For that
we also need another new predicate (have_traps()) to determine of there
are any caught traps which might occur - if there are, we need to remain
to (potentially) handle them, so these optimisations will not occur (to
make the issue in PR 48875 appear again, run the same code, but with a
trap set to execute some code when a signal (or EXIT) occurs - note that
the trap must be set in the appropriate level of sub-shell to have this
effect, any caught traps are cleared in a subshell whenever one is created).

There is still work to be done to handle traps properly, whatever
weirdness they do (some of which is related to some of this.)

These changes do not need man page updates, but 48875 does - an update
to sh.1 will be forthcoming once it is decided what it should say...

Once again, all the heavy lifting for this set of changes comes directly
(with thanks) from the FreeBSD shell.

XXX pullup-8 (but not very soon)

branches: 1.20.4;

Implement PS1, PS2 and PS4 expansions (variable expansions, arithmetic
expansions, and if enabled by the promptcmds option, command substitutions.)

An initial attempt at implementing LINENO to meet the specs.

Aside from one problem (not too hard to fix if it was ever needed) this version
does about as well as most other shell implementations when expanding
$((LINENO)) and better for ${LINENO} as it retains the "LINENO hack" for the
latter, and that is very accurate.

Unfortunately that means that ${LINENO} and $((LINENO)) do not always produce
the same value when used on the same line (a defect that other shells do not
share - aside from the FreeBSD sh as it is today, where only the LINENO hack
exists and so (like for us before this commit) $((LINENO)) is always either
0, or at least whatever value was last set, perhaps by
LINENO=${LINENO}
which does actually work ... for that one line...)

This could be corrected by simply removing the LINENO hack (look for the string
LINENO in parser.c) in which case ${LINENO} and $((LINENO)) would give the
same (not perfectly accurate) values, as do most other shells.

POSIX requires that LINENO be set before each command, and this implementation
does that fairly literally - except that we only bother before the commands
which actually expand words (for, case and simple commands). Unfortunately
this forgot that expansions also occur in redirects, and the other compound
commands can also have redirects, so if a redirect on one of the other compound
commands wants to use the value of $((LINENO)) as a part of a generated file
name, then it will get an incorrect value. This is the "one problem" above.
(Because the LINENO hack is still enabled, using ${LINENO} works.)

This could be fixed, but as this version of the LINENO implementation is just
for reference purposes (it will be superseded within minutes by a better one)
I won't bother. However should anyone else decide that this is a better choice
(it is probably a smaller implementation, in terms of code & data space then
the replacement, but also I would expect, slower, and definitely less accurate)
this defect is something to bear in mind, and fix.

This version retains the *BSD historical practice that line numbers in functions
(all functions) count from 1 from the start of the function, and elsewhere,
start from 1 from where the shell started reading the input file/stream in
question. In an "eval" expression the line number starts at the line of the
"eval" (and then increases if the input is a multi-line string).

Note: this version is not documented (beyond as much as LINENO was before)
hence this slightly longer than usual commit message.

branches: 1.18.2;

Deal with \newline line continuations more correctly.
They can occur anywhere (*anywhere*) not only where it
happens to be convenient to the parser...

This fix from FreeBSD (thanks again folks).

To make this work, pushstring()'s signature needed to change to allow a
const char * as its string arg, which meant sprinkling some const other
places for a brighter appearance (and handling fallout).

All this because I wanted to see what number would come from

echo $\
{\
L\
I\
N\
E\
N\
O\
}

and was surprised at the result! That works now...

The bug would also affect stuff like

true &\
& false

and all kinds of other uses where the \newline occurred in the
"wrong" place.

An ATF test for sh syntax is coming... (sometime.)

Fix idiot typos in previous (this is not the advertised :next commit")
Same typo - two different places. Ugh!

NFC: Change prototype of pushstring() to give a real type for the 3rd
arg (struct alias *) rather than using void * and then casting it
when used. For callers, the arg either is a struct alias *, or is NULL,
so nothing to adjust there.

NB: This change untested by itself, it was going to be a part of the next
change (coming in a few minutes) but is logically unrelated, so ...

branches: 1.15.88;
Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22249, verified by myself.

Fixes from David Laight:
- ansification
- format of output of jobs command (etc)
- job identiers %+, %- etc
- $? and $(...)
- correct quoting of output of set, export -p and readonly -p
- differentiation between nornal and 'posix special' builtins
- correct behaviour (posix) for errors on builtins and special builtins
- builtin printf and kill
- set -o debug (if compiled with DEBUG)
- cd src obj (as ksh - too useful to do without)
- unset -e name, remove non-readonly variable from export list.
(so I could unset -e PS1 before running the test shell...)

VFork()ing shell: From elric@netbsd.org:
Plus my changes:
- walking process group fix in foregrounding a job.
- reset of process group in parent shell if interrupted before the wait.
- move INTON lower in the dowait so that the job structure is
consistent.
- error check all setpgid(), tcsetpgrp() calls.
- eliminate unneeded strpgid() call.
- check that we don't belong in the process group before we try to
set it.

branches: 1.12.6;
Back out previous vfork changes.

Now we use vfork(2) instead of fork(2) when we can.

compile with WARNS = 2

PR/2808: Remove trailing whitespace (from FreeBSD)

branches: 1.8.6;
Merge in my changes from vangogh, and fix the x=`false`; echo $? == 0
bug.

convert to new RCS id conventions.

Add RCS ids.

sync with 4.4lite

Add RCS identifiers.

changed "Id" to "Header" for rcsids

added rcs ids to all files

branches: 1.1.1;
Initial revision

Implement PS1, PS2 and PS4 expansions (variable expansions, arithmetic
expansions, and if enabled by the promptcmds option, command substitutions.)

An initial attempt at implementing LINENO to meet the specs.

Aside from one problem (not too hard to fix if it was ever needed) this version
does about as well as most other shell implementations when expanding
$((LINENO)) and better for ${LINENO} as it retains the "LINENO hack" for the
latter, and that is very accurate.

Unfortunately that means that ${LINENO} and $((LINENO)) do not always produce
the same value when used on the same line (a defect that other shells do not
share - aside from the FreeBSD sh as it is today, where only the LINENO hack
exists and so (like for us before this commit) $((LINENO)) is always either
0, or at least whatever value was last set, perhaps by
LINENO=${LINENO}
which does actually work ... for that one line...)

This could be corrected by simply removing the LINENO hack (look for the string
LINENO in parser.c) in which case ${LINENO} and $((LINENO)) would give the
same (not perfectly accurate) values, as do most other shells.

POSIX requires that LINENO be set before each command, and this implementation
does that fairly literally - except that we only bother before the commands
which actually expand words (for, case and simple commands). Unfortunately
this forgot that expansions also occur in redirects, and the other compound
commands can also have redirects, so if a redirect on one of the other compound
commands wants to use the value of $((LINENO)) as a part of a generated file
name, then it will get an incorrect value. This is the "one problem" above.
(Because the LINENO hack is still enabled, using ${LINENO} works.)

This could be fixed, but as this version of the LINENO implementation is just
for reference purposes (it will be superseded within minutes by a better one)
I won't bother. However should anyone else decide that this is a better choice
(it is probably a smaller implementation, in terms of code & data space then
the replacement, but also I would expect, slower, and definitely less accurate)
this defect is something to bear in mind, and fix.

This version retains the *BSD historical practice that line numbers in functions
(all functions) count from 1 from the start of the function, and elsewhere,
start from 1 from where the shell started reading the input file/stream in
question. In an "eval" expression the line number starts at the line of the
"eval" (and then increases if the input is a multi-line string).

Note: this version is not documented (beyond as much as LINENO was before)
hence this slightly longer than usual commit message.

Deal with \newline line continuations more correctly.
They can occur anywhere (*anywhere*) not only where it
happens to be convenient to the parser...

This fix from FreeBSD (thanks again folks).

To make this work, pushstring()'s signature needed to change to allow a
const char * as its string arg, which meant sprinkling some const other
places for a brighter appearance (and handling fallout).

All this because I wanted to see what number would come from

echo $\
{\
L\
I\
N\
E\
N\
O\
}

and was surprised at the result! That works now...

The bug would also affect stuff like

true &\
& false

and all kinds of other uses where the \newline occurred in the
"wrong" place.

An ATF test for sh syntax is coming... (sometime.)

Fix idiot typos in previous (this is not the advertised :next commit")
Same typo - two different places. Ugh!

NFC: Change prototype of pushstring() to give a real type for the 3rd
arg (struct alias *) rather than using void * and then casting it
when used. For callers, the arg either is a struct alias *, or is NULL,
so nothing to adjust there.

NB: This change untested by itself, it was going to be a part of the next
change (coming in a few minutes) but is logically unrelated, so ...

branches: 1.15.88;
Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22249, verified by myself.

Fixes from David Laight:
- ansification
- format of output of jobs command (etc)
- job identiers %+, %- etc
- $? and $(...)
- correct quoting of output of set, export -p and readonly -p
- differentiation between nornal and 'posix special' builtins
- correct behaviour (posix) for errors on builtins and special builtins
- builtin printf and kill
- set -o debug (if compiled with DEBUG)
- cd src obj (as ksh - too useful to do without)
- unset -e name, remove non-readonly variable from export list.
(so I could unset -e PS1 before running the test shell...)

VFork()ing shell: From elric@netbsd.org:
Plus my changes:
- walking process group fix in foregrounding a job.
- reset of process group in parent shell if interrupted before the wait.
- move INTON lower in the dowait so that the job structure is
consistent.
- error check all setpgid(), tcsetpgrp() calls.
- eliminate unneeded strpgid() call.
- check that we don't belong in the process group before we try to
set it.

branches: 1.12.6;
Back out previous vfork changes.

Now we use vfork(2) instead of fork(2) when we can.

compile with WARNS = 2

PR/2808: Remove trailing whitespace (from FreeBSD)

branches: 1.8.6;
Merge in my changes from vangogh, and fix the x=`false`; echo $? == 0
bug.

convert to new RCS id conventions.

Add RCS ids.

sync with 4.4lite

Add RCS identifiers.

changed "Id" to "Header" for rcsids

added rcs ids to all files

branches: 1.1.1;
Initial revision

An initial attempt at implementing LINENO to meet the specs.

Aside from one problem (not too hard to fix if it was ever needed) this version
does about as well as most other shell implementations when expanding
$((LINENO)) and better for ${LINENO} as it retains the "LINENO hack" for the
latter, and that is very accurate.

Unfortunately that means that ${LINENO} and $((LINENO)) do not always produce
the same value when used on the same line (a defect that other shells do not
share - aside from the FreeBSD sh as it is today, where only the LINENO hack
exists and so (like for us before this commit) $((LINENO)) is always either
0, or at least whatever value was last set, perhaps by
LINENO=${LINENO}
which does actually work ... for that one line...)

This could be corrected by simply removing the LINENO hack (look for the string
LINENO in parser.c) in which case ${LINENO} and $((LINENO)) would give the
same (not perfectly accurate) values, as do most other shells.

POSIX requires that LINENO be set before each command, and this implementation
does that fairly literally - except that we only bother before the commands
which actually expand words (for, case and simple commands). Unfortunately
this forgot that expansions also occur in redirects, and the other compound
commands can also have redirects, so if a redirect on one of the other compound
commands wants to use the value of $((LINENO)) as a part of a generated file
name, then it will get an incorrect value. This is the "one problem" above.
(Because the LINENO hack is still enabled, using ${LINENO} works.)

This could be fixed, but as this version of the LINENO implementation is just
for reference purposes (it will be superseded within minutes by a better one)
I won't bother. However should anyone else decide that this is a better choice
(it is probably a smaller implementation, in terms of code & data space then
the replacement, but also I would expect, slower, and definitely less accurate)
this defect is something to bear in mind, and fix.

This version retains the *BSD historical practice that line numbers in functions
(all functions) count from 1 from the start of the function, and elsewhere,
start from 1 from where the shell started reading the input file/stream in
question. In an "eval" expression the line number starts at the line of the
"eval" (and then increases if the input is a multi-line string).

Note: this version is not documented (beyond as much as LINENO was before)
hence this slightly longer than usual commit message.

Deal with \newline line continuations more correctly.
They can occur anywhere (*anywhere*) not only where it
happens to be convenient to the parser...

This fix from FreeBSD (thanks again folks).

To make this work, pushstring()'s signature needed to change to allow a
const char * as its string arg, which meant sprinkling some const other
places for a brighter appearance (and handling fallout).

All this because I wanted to see what number would come from

echo $\
{\
L\
I\
N\
E\
N\
O\
}

and was surprised at the result! That works now...

The bug would also affect stuff like

true &\
& false

and all kinds of other uses where the \newline occurred in the
"wrong" place.

An ATF test for sh syntax is coming... (sometime.)

Fix idiot typos in previous (this is not the advertised :next commit")
Same typo - two different places. Ugh!

NFC: Change prototype of pushstring() to give a real type for the 3rd
arg (struct alias *) rather than using void * and then casting it
when used. For callers, the arg either is a struct alias *, or is NULL,
so nothing to adjust there.

NB: This change untested by itself, it was going to be a part of the next
change (coming in a few minutes) but is logically unrelated, so ...

branches: 1.15.88;
Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22249, verified by myself.

Fixes from David Laight:
- ansification
- format of output of jobs command (etc)
- job identiers %+, %- etc
- $? and $(...)
- correct quoting of output of set, export -p and readonly -p
- differentiation between nornal and 'posix special' builtins
- correct behaviour (posix) for errors on builtins and special builtins
- builtin printf and kill
- set -o debug (if compiled with DEBUG)
- cd src obj (as ksh - too useful to do without)
- unset -e name, remove non-readonly variable from export list.
(so I could unset -e PS1 before running the test shell...)

VFork()ing shell: From elric@netbsd.org:
Plus my changes:
- walking process group fix in foregrounding a job.
- reset of process group in parent shell if interrupted before the wait.
- move INTON lower in the dowait so that the job structure is
consistent.
- error check all setpgid(), tcsetpgrp() calls.
- eliminate unneeded strpgid() call.
- check that we don't belong in the process group before we try to
set it.

branches: 1.12.6;
Back out previous vfork changes.

Now we use vfork(2) instead of fork(2) when we can.

compile with WARNS = 2

PR/2808: Remove trailing whitespace (from FreeBSD)

branches: 1.8.6;
Merge in my changes from vangogh, and fix the x=`false`; echo $? == 0
bug.

convert to new RCS id conventions.

Add RCS ids.

sync with 4.4lite

Add RCS identifiers.

changed "Id" to "Header" for rcsids

added rcs ids to all files

branches: 1.1.1;
Initial revision

Deal with \newline line continuations more correctly.
They can occur anywhere (*anywhere*) not only where it
happens to be convenient to the parser...

This fix from FreeBSD (thanks again folks).

To make this work, pushstring()'s signature needed to change to allow a
const char * as its string arg, which meant sprinkling some const other
places for a brighter appearance (and handling fallout).

All this because I wanted to see what number would come from

echo $\
{\
L\
I\
N\
E\
N\
O\
}

and was surprised at the result! That works now...

The bug would also affect stuff like

true &\
& false

and all kinds of other uses where the \newline occurred in the
"wrong" place.

An ATF test for sh syntax is coming... (sometime.)

Fix idiot typos in previous (this is not the advertised :next commit")
Same typo - two different places. Ugh!

NFC: Change prototype of pushstring() to give a real type for the 3rd
arg (struct alias *) rather than using void * and then casting it
when used. For callers, the arg either is a struct alias *, or is NULL,
so nothing to adjust there.

NB: This change untested by itself, it was going to be a part of the next
change (coming in a few minutes) but is logically unrelated, so ...

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22249, verified by myself.

Fixes from David Laight:
- ansification
- format of output of jobs command (etc)
- job identiers %+, %- etc
- $? and $(...)
- correct quoting of output of set, export -p and readonly -p
- differentiation between nornal and 'posix special' builtins
- correct behaviour (posix) for errors on builtins and special builtins
- builtin printf and kill
- set -o debug (if compiled with DEBUG)
- cd src obj (as ksh - too useful to do without)
- unset -e name, remove non-readonly variable from export list.
(so I could unset -e PS1 before running the test shell...)

VFork()ing shell: From elric@netbsd.org:
Plus my changes:
- walking process group fix in foregrounding a job.
- reset of process group in parent shell if interrupted before the wait.
- move INTON lower in the dowait so that the job structure is
consistent.
- error check all setpgid(), tcsetpgrp() calls.
- eliminate unneeded strpgid() call.
- check that we don't belong in the process group before we try to
set it.

branches: 1.12.6;
Back out previous vfork changes.

Now we use vfork(2) instead of fork(2) when we can.

compile with WARNS = 2

PR/2808: Remove trailing whitespace (from FreeBSD)

branches: 1.8.6;
Merge in my changes from vangogh, and fix the x=`false`; echo $? == 0
bug.

convert to new RCS id conventions.

Add RCS ids.

sync with 4.4lite

Add RCS identifiers.

changed "Id" to "Header" for rcsids

added rcs ids to all files

branches: 1.1.1;
Initial revision

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22249, verified by myself.

Fixes from David Laight:
- ansification
- format of output of jobs command (etc)
- job identiers %+, %- etc
- $? and $(...)
- correct quoting of output of set, export -p and readonly -p
- differentiation between nornal and 'posix special' builtins
- correct behaviour (posix) for errors on builtins and special builtins
- builtin printf and kill
- set -o debug (if compiled with DEBUG)
- cd src obj (as ksh - too useful to do without)
- unset -e name, remove non-readonly variable from export list.
(so I could unset -e PS1 before running the test shell...)

VFork()ing shell: From elric@netbsd.org:
Plus my changes:
- walking process group fix in foregrounding a job.
- reset of process group in parent shell if interrupted before the wait.
- move INTON lower in the dowait so that the job structure is
consistent.
- error check all setpgid(), tcsetpgrp() calls.
- eliminate unneeded strpgid() call.
- check that we don't belong in the process group before we try to
set it.

branches: 1.12.6;
Back out previous vfork changes.

Now we use vfork(2) instead of fork(2) when we can.

compile with WARNS = 2

PR/2808: Remove trailing whitespace (from FreeBSD)

branches: 1.8.6;
Merge in my changes from vangogh, and fix the x=`false`; echo $? == 0
bug.

convert to new RCS id conventions.

Add RCS ids.

sync with 4.4lite

Add RCS identifiers.

changed "Id" to "Header" for rcsids

added rcs ids to all files

branches: 1.1.1;
Initial revision

44lite code

initial import of 386bsd-0.1 sources

Update /bin/sh from trunk per request of Christos Zoulas. Fixes
many bugs.

Doing the vfork work on ash on a branch to try to shake out the
problems before I expose everyone to them. This checkin represents
a merge of the prior work, which I backed out a while ago, to the
HEAD only and does not incorporate any additional bugfixes. The
additional bugfixes and code-cleanup will occur in later checkins.

For reference the patches that were used are:
cvs diff -kk -r1.51 -r1.55 eval.c | patch
cvs diff -kk -r1.27 -r1.28 exec.c | patch
cvs diff -kk -r1.15 -r1.16 exec.h | patch
cvs diff -kk -r1.32 -r1.33 input.c | patch
cvs diff -kk -r1.10 -r1.11 input.h | patch
cvs diff -kk -r1.32 -r1.35 jobs.c | patch
cvs diff -kk -r1.9 -r1.11 jobs.h | patch
cvs diff -kk -r1.36 -r1.37 main.c | patch
cvs diff -kk -r1.20 -r1.21 redir.c | patch
cvs diff -kk -r1.10 -r1.11 redir.h | patch
cvs diff -kk -r1.10 -r1.12 shell.h | patch
cvs diff -kk -r1.22 -r1.23 trap.c | patch
cvs diff -kk -r1.12 -r1.13 trap.h | patch
cvs diff -kk -r1.23 -r1.24 var.c | patch
cvs diff -kk -r1.16 -r1.17 var.h | patch

All other changes were simply the resolution of the resulting
conflicts, which occured only in the merge of jobs.c.

Begins to address PR: bin/5475