Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r263777: sh: Fix possible memory leaks and double frees with unexpected
SIGINT.

MFC r229201: sh: Make patmatch() non-recursive.

MFC r229220: sh: Make various functions static.

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

sh: Add support for named character classes in bracket expressions.

Example:
case x in [[:alpha:]]) echo yes ;; esac

sh: Fix duplicate prototypes for builtins.

Have mkbuiltins write the prototypes for the *cmd functions to builtins.h
instead of builtins.c and include builtins.h in more .c files instead of
duplicating prototypes for *cmd functions in other headers.

sh: Save/restore changed variables in optimized command substitution.

In optimized command substitution, save and restore any variables changed by
expansions (${var=value} and $((var=assigned))), instead of trying to
determine if an expansion may cause such changes.

If $! is referenced in optimized command substitution, do not cause jobs to
be remembered longer.

This fixes $(jobs $!) again, simplifies the man page and shortens the code.

sh: Fix locale-dependent ranges in bracket expressions.

When I added UTF-8 support in r221646, the LC_COLLATE-based ordering broke
because of sign extension of char.

Because of libc restrictions, this does not work for UTF-8. For UTF-8
locales, ranges always use character code order.

sh: Do parameter expansion before printing PS4 (set -x).

The function name expandstr() and the general idea of doing this kind of
expansion by treating the text as a here document without end marker is from
dash.

All variants of parameter expansion and arithmetic expansion also work (the
latter is not required by POSIX but it does not take extra code and many
other shells also allow it).

Command substitution is prevented because I think it causes too much code to
be re-entered (for example creating an unbounded recursion of trace lines).

Unfortunately, our LINENO is somewhat crude, otherwise PS4='$LINENO+ ' would
be quite useful.

sh: Fix unquoted $@/$* if IFS=''.

If IFS is null, unquoted $@/$* should still expand to separate words.
This differs from quoted $@ (which does not depend on IFS) in that pathname
generation is performed and empty words are removed.

sh: Add UTF-8 support to pattern matching.

?, [...] patterns match codepoints instead of bytes. They do not match
invalid sequences. [...] patterns must not contain invalid sequences
otherwise they will not match anything. This is so that ${var#?} removes the
first codepoint, not the first byte, without putting UTF-8 knowledge into
the ${var#pattern} code. However, * continues to match any string and an
invalid sequence matches an identical invalid sequence. (This differs from
fnmatch(3).)

sh: Add UTF-8 support to ${#var}.

If the current locale uses UTF-8, ${#var} counts codepoints (more precisely,
bytes b with (b & 0xc0) != 0x80).

Fix typos - remove duplicate "the".

PR: bin/154928
Submitted by: Eitan Adler <lists at eitanadler.com>
MFC after: 3 days

sh: Remove comment mentioning herefd, which is gone.

sh: Don't do optimized command substitution if expansions have side effects.

Before considering to execute a command substitution in the same process,
check if any of the expansions may have a side effect; if so, execute it in
a new process just like happens if it is not a single simple command.

Although the check happens at run time, it is a static check that does not
depend on current state. It is triggered by:
- expanding $! (which may cause the job to be remembered)
- ${var=value} default value assignment
- assignment operators in arithmetic
- parameter substitutions in arithmetic except ${#param}, $$, $# and $?
- command substitutions in arithmetic

This means that $((v+1)) does not prevent optimized command substitution,
whereas $(($v+1)) does, because $v might expand to something containing
assignment operators.

Scripts should not depend on these exact details for correctness. It is also
imaginable to have the shell fork if and when a side effect is encountered
or to create a new temporary namespace for variables.

Due to the $! change, the construct $(jobs $!) no longer works. The value of
$! should be stored in a variable outside command substitution first.

sh: Allow arbitrary large numbers in CHECKSTRSPACE.
Reduce "stack string" API somewhat and simplify code.
Add a check for integer overflow of the "stack string" length (probably
incomplete).

Remove dead code.

c is assigned 0 and *loc is pointing to NULL, so c!=0 cannot be true,
and dereferencing loc would be a bad idea anyway.

Coverity Prevent: CID 5113
Reviewed by: jilles

sh: Fix corruption of command substitutions with special chars after newline

The CTLESC byte to protect a special character was output before instead of
after a newline directly preceding the special character.

The special handling of newlines is because command substitutions discard
all trailing newlines.

sh: Remove the herefd hack.

The herefd hack wrote out partial here documents while expanding them. It
seems unnecessary complication given that other expansions just allocate
memory. It causes bugs because the stack is also used for intermediate
results such as arithmetic expressions. Such places should disable herefd
for the duration but not all of them do, and I prefer removing the need for
disabling herefd to disabling it everywhere needed.

Here documents larger than 1024 bytes will use a bit more CPU time and
memory.

Additionally this allows a later change to expand here documents in the
current shell environment. (This is faster for small here documents but also
changes behaviour.)

Obtained from: dash

sh: Replace some macros and repeated code in expand.c with functions.

No functional change is intended, but the binary is about 1K smaller on
i386.

sh: Code size optimizations to "stack string" memory allocation:
* Prefer one CHECKSTRSPACE with multiple USTPUTC to multiple STPUTC.
* Add STPUTS macro (based on function) and use it instead of loops that add
nul-terminated strings to the stack string.

No functional change is intended, but code size is about 1K less on i386.

sh: Code size optimizations to buffered output.

This is mainly less use of the outc macro.

No functional change is intended, but code size is about 2K less on i386.

sh: Fix some issues with CTL* bytes and ${var#pat}.

subevalvar() incorrectly assumed that CTLESC bytes were present iff the
expansion was quoted. However, they are present iff various processing such
as word splitting is to be done later on.

Example:
v=@$e@$e@$e@
y="${v##*"$e"}"
echo "$y"
failed if $e contained the magic CTLESC byte.

Exp-run done by: pav (with some other sh(1) changes)

sh: Do IFS splitting on word in ${v+word} and ${v-word}.

The code is inspired by NetBSD sh somewhat, but different because we
preserve the old Almquist/Bourne/Korn ability to have an unquoted part in a
quoted ${v+word}. For example, "${v-"*"}" expands to $v as a single field if
v is set, but generates filenames otherwise.

Note that this is the only place where we split text literally from the
script (the similar ${v=word} assigns to v and then expands $v). The parser
must now add additional markers to allow the expansion code to know whether
arbitrary characters in substitutions are quoted.

Example:
for i in ${$+a b c}; do echo $i; done

Exp-run done by: pav (with some other sh(1) changes)

In the spirit of r90111, depend on c89 and remove the "STATIC" macro
and its usage.

Make DEBUG traces 64-bit clean:
- Use %t to print ptrdiff_t values.
- Cast a ptrdiff_t value explicitly to int for a field width specifier.

While here, sort includes.

Submitted by: Garrett Cooper

Consistently use "STATIC" for all functions in order to be able to set
breakpoints with in a debugger. And use naked "static" for variables.

Noticed by: bde

sh: Improve comments in expand.c.

sh: Remove remnants of '!!' to negate pattern.

This Almquist extension was disabled long ago.

In pathname generation, components starting with '!!' were treated as
containing wildcards, causing unnecessary readdir (which could fail, causing
pathname generation to fail while it should not).

sh: Fix heap-based buffer overflow in pathname generation.

The buffer for generated pathnames could be too small in some cases. It
happened to be always at least PATH_MAX long, so there was never an overflow
if the resulting pathnames would be usable.

This bug may be abused if a script subjects input from an untrusted source
to pathname generation, which a bad idea anyhow. Most shell scripts do not
work on untrusted data. secteam@ says no advisory is necessary.

PR: bin/148733
Reported by: Changming Sun snnn119 at gmail com
MFC after: 10 days

sh: Forget about terminated background processes sooner.

Unless $! has been referenced for a particular job or $! still contains that
job's pid, forget about it after it has terminated. If $! has been
referenced, remember the job until the wait builtin has reported its
completion (either with the pid as parameter or without parameters).

In interactive mode, jobs are forgotten after termination has been reported,
which happens before primary prompts and through the jobs builtin. Even
then, though, remember a job if $! has been referenced.

This is similar to what is suggested by POSIX and should fix most memory
leaks (which also tend to cause sh to use more CPU time) with long running
scripts that start background jobs.

Caveats:
* Repeatedly referencing $! without ever doing 'wait', like
while :; do foo & echo started foo: $!; sleep 60; done
will still use a lot of memory and CPU time in the long run.
* The jobs and jobid builtins do not cause a job to be remembered for longer
like expanding $! does.

PR: bin/55346

sh: Fix pathname expansion with quoted slashes like *\/.

These are git commits 36f0fa8fcbc8c7b2b194addd29100fb40e73e4e9 and
d6d06ff5c2ea0fa44becc5ef4340e5f2f15073e4 in dash.

Because this is the first code I'm importing from dash to expand.c, add the
Herbert Xu copyright notice which is in dash's expand.c.

When pathname expanding *\/, the CTLESC representing the quoted state was
erroneously taken as part of the * pathname component. This CTLESC was then
seen by the pattern matching code as escaping the '\0' terminating the
string.

The code is slightly different because dash converts the CTLESC characters
to backslashes and removes all the other CTL* characters to allow
substituting glob(3).

The effect of the bug was also slightly different from dash (where nothing
matched at all). Because a CTLESC can escape a '\0' in some way, whether
files were included despite the bug depended on memory that should not be
read. In particular, on many machines /*\/ expanded to a strict subset of
what /*/ expanded to.

Example:
echo /*"/null"

This should print /dev/null, not /*/null.

PR: bin/146378
Obtained from: dash

sh: Use stalloc for arith variable names.

This is simpler than the custom memory tracker I added earlier, and is also
needed by the dash arith code I plan to import.

sh: Do tilde expansion in substitutions.

This applies to word in ${v-word}, ${v+word}, ${v=word}, ${v?word} (which
inherits quoting from the outside) and in ${v%word}, ${v%%word}, ${v#word},
${v##word} (which does not inherit any quoting).

In all cases tilde expansion is only attempted at the start of word, even if
word contains spaces. This agrees with POSIX and other shells.

This is the last part of the patch tested in the exp-run.

Exp-run done by: erwin (with some other sh(1) changes)

sh: Allow quoting pattern match characters in ${v%pat} and ${v#pat}.

Note that this depends on r206145 for allowing pattern match characters to
have their special meaning inside a double-quoted expansion like "${v%pat}".

PR: bin/117748
Exp-run done by: erwin (with some other sh(1) changes)

sh: Fix some bugs with backquoted builtins:
- correctly handle error output in $(builtin 2>&1), clarify out1/out2 vs
output/errout in the code
- treat all builtins as regular builtins so errors do not abort the shell
and variable assignments do not persist
- respect the caller's INTOFF

Some bugs still exist:
- expansion errors may still abort the shell
- some side effects of expansions and builtins persist

sh: Various warning fixes (from WARNS=6 NO_WERROR=1):
- const
- initializations to silence -Wuninitialized (it was safe anyway)
- remove nested extern declarations
- rename "index" locals to "idx"

sh: Do not consider a tilde-prefix with expansions in it.

That is, do not do tilde expansion if any of the CTL* bytes (\201-\210), not
only CTLESC and CTLQUOTEMARK, are encountered. Such an expansion would look
up a user name with sh's internal representation.

The parser does not currently distinguish between backslashed and
unbackslashed \201-\210, so tilde expansion of user names with these bytes
in them is not so easy to fix.

sh: Constify various strings.

Most of this is adding const keywords, but setvar() in var.c had to be
changed somewhat more.

sh: Exempt $@ and $* from set -u

This seems more useful and will likely be in the next POSIX standard.

Also document more precisely in the man page what set -u does (note that
$@, $* and $! are the only special parameters that can ever be unset, all
the others are always set, although they may be empty).

Fix some weirdnesses in the NetBSD IFS code,
in particular "$@"$ifschar if the final positional parameter is empty.
With the NetBSD code, adding the $ifschar removes a parameter.

PR: standards/79067
Approved by: ed (mentor) (implicit)

Improve IFS expansion using code from NetBSD.

We now pass the ifs.sh testsuite.

PR: standards/79067
Approved by: ed (mentor) (implicit)
Obtained from: NetBSD

Expand $LINENO to the current line number. This is required by SUSv3's "User
Portability Utilities" option.

Often configure scripts generated by the autotools test if $LINENO works and
refuse to use /bin/sh if not.

Package test run by: pav

Sigh, when reapplying the patch to HEAD, I somehow forgot to commit this file.

Reported by: Jaakko Heinonen

Fix expanding of quoted positional parameters in case patterns.

Obtained from: NetBSD (expand.c 1.58 and 1.59)
Submitted by: Paul Jarc
PR: 56147

When parsing an invalid parameter expansion (eg. ${} or ${foo@bar}) do not
issue a syntax error immediately but save the information that it is erroneous
for later when the parameter expansion is actually done. This means eg. "false
&& ${}" will not generate an error which seems to be required by POSIX.
Include the invalid parameter expansion in the error message (sometimes
abbreviated with ... because recovering it would require a lot of code).

PR: 105078
Submitted by: emaste

Remove some white space at EOL.

Various small code cleanups resulting from a code reviewing
and linting procedure:

1. Remove useless sub-expression:

- if (*start || (!ifsspc && start > string && (nulonly || 1))) {
+ if (*start || (!ifsspc && start > string)) {

The sub-expression "(nulonly || 1)" always evaluates to true and
according to CVS logs seems to be just a left-over from some
debugging and introduced by accident. Removing the sub-expression
doesn't change semantics and a code inspection showed that the
variable "nulonly" is also not necessary here in any way (and the
expression would require fixing instead of removing).

2. Remove dead code:

- if (backslash && c == '\\') {
- if (read(STDIN_FILENO, &c, 1) != 1) {
- status = 1;
- break;
- }
- STPUTC(c, p);
- } else if (ap[1] != NULL && strchr(ifs, c) != NULL) {
+ if (ap[1] != NULL && strchr(ifs, c) != NULL) {

Inspection of the control and data flow showed that variable
"backslash" is always false (0) when the "if"-expression is
evaluated, hence the whole block is effectively dead code.
Additionally, the skipping of characters after a backslash is already
performed correctly a few lines above, so this code is also not
needed at all. According to the CVS logs and the ASH 0.2 sources,
this code existed in this way already since its early days.

3. Cleanup Style:

- ! trap[signo][0] == '\0' &&
+ ! (trap[signo][0] == '\0') &&

The expression wants to ensure the trap is not assigned the empty
string. But the "!" operator has higher precedence than "==", so the
comparison should be put into parenthesis to form the intended way of
expression. Nevertheless the code was effectively not really broken
as both particular NUL comparisons are semantically equal, of course.
But the parenthesized version is a lot more intuitive.

4. Remove shadowing variable declaration:

- char *q;

The declaration of symbol "q" hides another identical declaration of
"q" in the same context. As the other "q" is already reused multiple
times and also can be reused again without negative side-effects,
just remove the shadowing declaration.

5. Just small cosmetics:

- if (ifsset() != 0)
+ if (ifsset())

The ifsset() macro is already coded by returning the boolean result
of a comparison operator, so no need to compare this boolean result
again against a numerical value. This also aligns the macros usage to
the remaining existing code.

Reviewed by: stefanf@

Fix a bug when shell expansion is done against dangling symlinks, by
converting the stat() call to a lstat() call, which will cover the
situation. One can exercise this bug by referring a dangling link with
something like */the-link.

Approved by: re (scottl)
Submitted by: Simon 'corecode' Schubert [corecode fs ei tum de]
Obtained from: NetBSD via DragonFlyBSD (NetBSD rev. 1.51 and DragonFly
rev. 1.6)
MFC After: 3 days

Remove clause 3 from the UCB licenses.

OK'ed by: imp, core

Remove collate_range_cmp() stabilization, it conflicts with ranges

Changes following CScout analysis:

- Removed dead declarations
- Made objects that should have been declared as static, static.

The changes use STATIC instead of static, following the existing
convention in the rest of the code.

Approved by: schweikh (mentor)
MFC after: 2 weeks

Instead of eating trailing newlines after inserting them into the
output buffer, don't insert them at all. This prevents a buffer
*underrun* when the substitution consists completely of newlines
(e.g. `echo`) and the byte before the source buffer to which p
points is a '\n', in which case more characters would be removed
from the output buffer than were inserted.

This fixes certain port builds on sparc64.

Approved by: re (scottl)
Reviewed by: des, tjr

Fixed (soon might be fatal) -Wformat warnings.

Do not strip CTL* escapes from redirection filenames in exptilde(),
expari(), expbackq() and evalvar(). Similar to revision 1.39.
Patch from Tor Egge.

PR: 45349
MFC after: 2 weeks

Add the "wordexp" shell built-in command which will be used to implement
the POSIX wordexp() function.

Do not strip CTL* escapes from redirection filenames in argstr(); they
are later stripped with rmescapes() in expandarg(). If the filename has
already been unescaped, doing it again in rmescapes() can walk off the
end of the string, leading to memory corruption and eventually SIGSEGV.

Noticed by: kris

Replace various spellings with FALLTHROUGH which is lint()able

Consistently use FBSDID

Implement the -u (-o nounset) option, which gives an error message if
an unset variable is expanded.

Obtained from: NetBSD (bjh21, christos)

o __P has been reoved
o Old-style K&R declarations have been converted to new C89 style
o register has been removed
o prototype for main() has been removed (gcc3 makes it an error)
o int main(int argc, char *argv[]) is the preferred main definition.
o Attempt to not break style(9) conformance for declarations more than
they already are.
o Change
int
foo() {
...
to
int
foo(void)
{
...

Don't check uninitialized memory for having the shell control character
value CTLARI since this might break expansion of arithmetic expressions.

Don't access memory below start of stackblock.

Problem analyzed by hunt@iprg.nokia.com, slightly different patch applied.

PR: 24443
Submitted by: hunt@iprg.nokia.com

BASESYNTAX, DQSYNTAX, SQSYNTAX and ARISYNTAX handles negative indexes.
Allow those to be used to properly quote characters in the shell
control character range.

Fix environment passung to eval'ed commands.

PR: bin/6577
Submitted by: Anatoly Vorobey <mellon@pobox.com>
Approved by: silence amoung other sh experts

First round of 8-bit fixes.

Fix "subscript has type `char'" warnings by casting to int, as
discussed on -arch.

$Id$ -> $FreeBSD$

Various spelling/formatting changes.

Submitted by: Philippe Charnier <charnier@xp11.frmug.org>

Next approach to make loops in interactive interruptable.

PR: bin/9173

During variable expansion, the internal representation of the expression
might be relocated. Handle this case.
PR: 7059

When a variable expansion is enclosed in double quotes, the internal
representation of the expression is quoted. Take care of this when
doing pattern matching in conjunction with trimming.

#!/bin/sh
c=d:e; echo "${c%:e}"

PR: NetBSD PR#7231
Noticed by: Havard Eidnes <Havard.Eidnes@runit.sintef.no>

Be more consistent with handling of quote mark control character.
Don't output double-quotes inside variable expansion/arithmetic
expansion region in here-documents. When leaving the arithmetic
expansion syntax mode, adjust the dblquote flag according to
previous syntax, in order to avoid splitting of quoted variables.

Better handling of word splitting. Don't record the same region
multiple times when performing nested variable expansion, and
preserve some quoting information in order to avoid removing
apparently empty expansion result.

Add rcsid. Spelling.

Back out previous fix - this bug's got diplomatic immunity as a registered
political issue.

>Number: 3780
>Category: bin
>Synopsis: WEXITSTATUS() may return nagative value, which causes sh to generate bad $?

PR: 3780
Submitted by: sanewo@ba2.so-net.or.jp

Now [^abc] means the same as [!abc] like bash and *csh already does

Use the __unused attribute where warranted.

Sync with NetBSD's revision 1.29 of this file.

Obtained from: NetBSD

Revert $FreeBSD$ to $Id$

Fix a expansion bug that caused the result of `echo $((1 << 30))`
to get truncated.

Submitted by: bde

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

This doesn't change any behavior, but may be a slight
optimization. (num-- > 0) --> (--num >= 0).

Obtained from: NetBSD

Merge in NetBSD mods and -Wall cleaning.

Obtained from: NetBSD, me

1) define STATIC as static and not empty
2) replace collate_range_cmp call with its code

Ok, lets try this again, shall we? It was definatly my mistake, not
Steve's.. :-]

ack! back these out so I can see what I did wrong. It looks like a
patch-by-hand botch, but it sig-11's during make world.

Fix for PR#1248, sh doesn't expand past ${9}

Submitted by: Steve Price <sprice@hiwaay.net>

Merge of 4.4-Lite2 sh source, plus some gcc -Wall cleaning. This is a
merge of parallel duplicate work by Steve Price and myself. :-]

There are some changes to the build that are my fault... mkinit.c was
trying (poorly) to duplicate some of the work that make(1) is designed to
do. The Makefile hackery is my fault too, the depend list was incomplete
because of some explicit OBJS+= entries, so mkdep wasn't picking up their
source file #includes.

This closes a pile of /bin/sh PR's, but not all of them..

Submitted by: Steve Price <steve@bonsai.hiwaay.net>, peter

Convert to newly added collate compare function

Localize it

Remove trailing whitespace.

Reviewed by: phk

Fix the deletion of trailing newlines with backquote expansion.
Reviewed by:
Submitted by:
Obtained from:

Added $Id$

This commit was generated by cvs2svn to compensate for changes in r1556,
which included commits to RCS files with non-trunk default branches.

BSD 4.4 Lite bin Sources