bin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

bin: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

sh: introduce a function to iterate over all hashed commands

While aliases and built-ins are opportunistically stored in cmdtable, each
function will be added to it immediately on definition.

Factor out the hashing function, write the iterator function and make it use
the hashing function.

Add the cmdname pointer to struct cmdentry so that the command name can be
exposed that way.

Differential Revision: https://reviews.freebsd.org/D40619

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

sh: Add -o verify to use O_VERIFY when sourcing scripts

Add -o verify to sh to make it use O_VERIFY when
sourcing scripts and reading profiles.

Useful in conjunction with mac_veriexec to help protect at
least some parts of the boot sequence, e.g., /etc/rc*.

Differential revision: https://reviews.freebsd.org/D30464
Reviewed by: jilles, sjg
Obtained from: Stormshield

sh: Write absolute path in command -vV and type

POSIX is pretty clear that command -v, command -V and type shall write
absolute pathnames. Therefore, we need to prepend the current directory's
name to relative pathnames.

This can happen either when PATH contains a relative pathname or when the
operand contains a slash but is not an absolute pathname.

sh: Allow more scripts without #!

Austin Group bugs #1226 and #1250 changed the requirements for shell scripts
without #! (POSIX does not specify #!; this is about the shell execution
when execve(2) returns an [ENOEXEC] error).

POSIX says we shall allow execution if the initial part intended to be
parsed by the shell consists of characters and does not contain the NUL
character. This allows concatenating a shell script (ending with exec or
exit) and a binary payload.

In order to reject common binary files such as PNG images, check that there
is a lowercase letter or expansion before the last newline before the NUL
character, in addition to the check for the newline character suggested by
POSIX.

sh: Unify EXERROR and EXEXEC

The difference between EXERROR and EXEXEC was that EXEXEC passed along
exitstatus and EXERROR set exitstatus to 2 in the handling code.

By changing the places that raised EXERROR to set exitstatus to 2, the
handling of EXERROR and EXEXEC becomes the same.

sh: Use exitstatus instead of exerrno to pass EXEXEC status

No functional change is intended.

sh: Don't treat % specially in CDPATH

sh: Update TOUR and comments for some code changes, some of them old.

Also, improve some terminology in TOUR and comments.

Renumber copyright clause 4

Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is too pedantic, so give up on that point.

Submitted by: Jan Schaumann <jschauma@stevens.edu>
Pull Request: https://github.com/freebsd/freebsd/pull/96

sh: Fix copying uninitialized field 'special'.

This just copied uninitialized data and did not depend on it later, so it
should not be dangerous.

Found by: clang static analyzer

sh: Reduce size of builtins table.

sh: Fix some arithmetic undefined behaviour.

Fix shifts of possibly negative numbers found with ubsan and avoid signed
integer overflow when hashing an extremely long command name.

MFC after: 1 week

sh: Remove prefix() function. Use strncmp() instead.

sh: Simplify find_command() slightly.

sh: Fix possible memory leaks and double frees with unexpected SIGINT.

sh: Prefer memcpy() to strcpy() in most cases. Remove the scopy macro.

sh: Properly quote alias output from command -v.

An alias should be printed by command -v as a command line; therefore, make
the alias definition suitable for re-input to the shell.

sh: Recognize "--" as end of options in type builtin.

This implementation makes minimal changes: command names starting with "-"
(other than "--") can still be queried normally.

sh: Remove mkinit.

Replace the RESET blocks with regular functions and a reset() function that
calls them all.

This code generation tool is unusual and does not appear to provide much
benefit. I do not think isolating the knowledge about which modules need to
be reset is worth an almost 500-line build tool and wider scope for
variables used by the reset functions.

Also, relying on reset functions is often wrong: the cleanup should be done
in exception handlers so that no stale state remains after 'command eval'
and the like.

sh: Fix various compiler warnings.

It now passes WARNS=7 with clang on i386.

GCC 4.2.1 does not understand setjmp() properly so will always trigger
-Wuninitialized. I will not add the volatile keywords to suppress this.

sh: Simplify cd-hash interaction.

Instead of rechecking relative paths for all hashed utilities after a cd,
track if any utility in cmdtable depends on a relative path in PATH.
If there is such a utility, cd clears the entire table.

As a result, the '*' in hash no longer happens.

sh: Change cmdtype in tblentry from short to signed char.

If this is a smaller type than int anyway, we can make it the smallest
possible.

sh: Use C99 flexible array instead of accessing array beyond bounds.

Although sufficient memory is available for a longer string in cmdname,
this is undefined behaviour anyway.

Side effect: for alignment reasons, an additional byte of memory is
allocated per hashed command.

sh: Expand assignment-like words specially for export/readonly/local.

Examples:
export x=~
now expands the tilde
local y=$1
is now safe, even if $1 contains IFS characters or metacharacters.

For a word to "look like an assignment", it must start with a name followed
by an equals sign, none of which may be quoted.

The special treatment applies when the first word (potentially after
"command") is "export", "readonly" or "local". There may be quoting
characters but no expansions. If "local" is overridden with a function there
is no special treatment ("export" and "readonly" cannot be overridden with a
function).

If things like
local arr=(1 2 3)
are ever allowed in the future, they cannot call a "local" function. This
would either be a run-time error or it would call the builtin.

This matches Austin Group bug #351, planned for the next issue of POSIX.1.

PR: bin/166771

sh: Make 'hash' return 1 if at least one utility is not found.

Reported by: lme

Add prototypes, ANSIfy functions definitions to reduce WARNS=6 output.

sh: Make various functions static.

sh: Remove clearcmdentry()'s now unused argument.

sh: Forget all cached command locations on any PATH change.

POSIX requires this and it is simpler than the previous code that remembered
command locations when appending directories to PATH.

In particular,
PATH=$PATH
is no longer a no-op but discards all cached command locations.

sh: Do not try to execute binary files as scripts.

If execve() returns an [ENOEXEC] error, check if the file is binary before
trying to execute it using sh. A file is considered binary if at least one
of the first 256 bytes is '\0'.

In particular, trying to execute ELF binaries for the wrong architecture now
fails with an "Exec format error" message instead of syntax errors and
potentially strange results.

sh: Remove special code for shell scripts without magic number.

These are called "shell procedures" in the source.

If execve() failed with [ENOEXEC], the shell would reinitialize itself
and execute the program as a script. This requires a fair amount of code
which is not frequently used (most scripts have a #! magic number).
Therefore just execute a new instance of sh (_PATH_BSHELL) to run the
script.

sh: Return only 126 or 127 for execve() failures.

Do not return 2 for errors other than [EACCES] or [ENOENT].

sh: Remove special %builtin PATH entry.
All builtins are now always found before a PATH search.

Most ash derivatives have an undocumented feature where the presence of an
entry "%builtin" in $PATH will cause builtins to be checked at that point of
the PATH search, rather than before looking at any directories as documented
in the man page (very old versions do document this feature).

I am removing this feature from sh, as it complicates the code, may violate
expectations (for example, /usr/bin/alias is very close to a forkbomb with
PATH=/usr/bin:%builtin, only /usr/bin/builtin not being another link saves
it) and appears to be unused (all the %builtin google code search finds is
in some sort of ash source code).

Note that aliases and functions took and take precedence above builtins.
Because aliases work on a lexical level they can only ever be overridden on
a lexical level (quoting or preceding 'builtin' or 'command'). Allowing
override of functions via PATH does not really fit in the model of sh and it
would work differently from %builtin if implemented.

Note: POSIX says special builtins are found before functions. We comply to
this because we do not allow functions with the same name as a special
builtin.

Silence from: freebsd-hackers@ (message sent 20101225)
Discussed with: dougb

sh: Allow arbitrary large numbers in CHECKSTRSPACE.
Reduce "stack string" API somewhat and simplify code.
Add a check for integer overflow of the "stack string" length (probably
incomplete).

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

In the spirit of r90111, depend on c89 and remove the "STATIC" macro
and its usage.

Consistently use "STATIC" for all functions in order to be able to set
breakpoints with in a debugger. And use naked "static" for variables.

Noticed by: bde

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

sh: Improve the command builtin:
* avoid unnecessary fork
* allow executing builtins via command
* executing a special builtin via command removes its special properties

Obtained from: NetBSD (parts)

sh: Fix some bugs with backquoted builtins:
- correctly handle error output in $(builtin 2>&1), clarify out1/out2 vs
output/errout in the code
- treat all builtins as regular builtins so errors do not abort the shell
and variable assignments do not persist
- respect the caller's INTOFF

Some bugs still exist:
- expansion errors may still abort the shell
- some side effects of expansions and builtins persist

sh: Use PATH= assignment in type.

Example:
PATH=/var/empty; PATH=/bin type ls

sh: Allow command -pv and command -pV (lookup using _PATH_STDPATH).

sh: Various warning fixes (from WARNS=6 NO_WERROR=1):
- const
- initializations to silence -Wuninitialized (it was safe anyway)
- remove nested extern declarations
- rename "index" locals to "idx"

sh: Constify various strings.

Most of this is adding const keywords, but setvar() in var.c had to be
changed somewhat more.

MFC r196483,r196634:
sh: Fix crash when undefining or redefining a currently executing function

Add a reference count to function definitions.
Memory may leak if a SIGINT arrives in interactive mode at exactly the wrong
time, this will be fixed later by changing SIGINT handling.

PR: bin/137640
Approved by: re (kib)

sh: Send the "xyz: not found" message to redirected fd 2.
This also fixes that trying to execute a non-regular file with a command
name without '/' returns 127 instead of 126.
The fix is rather simplistic: treat CMDUNKNOWN as if the command were found
as an external program. The resulting fork is a bit wasteful but executing
unknown commands should not be very frequent.

PR: bin/137659

sh: Fix crash with empty functions (f() { }) introduced in r196483

Empty pairs of braces are represented by a NULL node pointer, just like
empty lines at the top level.

Support for empty pairs of braces may be removed later. They make the code
more complex, have inconsistent behaviour (may or may not change $?), are
not specified by POSIX and are not allowed by some other shells like bash,
dash and ksh93.

Reported by: kan

sh: Fix crash when undefining or redefining a currently executing function.

Add a reference count to function definitions.
Memory may leak if multiple SIGINTs arrive in interactive mode,
this will be fixed later by changing SIGINT handling.

PR: bin/137640

Designate special builtins as such in command -V and type.
Also document various properties of special builtins that we implement.

Approved by: ed (mentor) (implicit)

align coding style with style(9) to avoid misunderstandings

Report error messages of the builtins 'type' and 'command -V' to stderr instead
of stdout.

Noticed by: Zajcev Evgeny

- Import the HEAD csup code which is the basis for the cvsmode work.

Use eaccess() instead of access() for the type builtin, like we do for the
test builtin.

Submitted by: Martin Kammerhofer

Return an error status (127) from the builtins 'type' and 'command' (with
either -v or -V) if a file with a slash in the name doesn't exist (if there is
no slash we already did that).

Additionally, suppress the error message for command -v for files with a slash.

PR: 107674
Submitted by: Martin Kammerhofer

Implement some of the differences between special built-ins and other builtins
demanded by POSIX.
- A redirection error is only fatal (meaning the execution of a shell script is
terminated) for special built-ins. Previously it was fatal for all shell
builtins, causing problems like the one reported in PR 88845.
- Variable assignments remain in effect for special built-ins.
- Option or operand errors are only fatal for special built-ins.
This change also makes errors from 'fc' non-fatal (I could not find any reasons
for this behaviour).

Somewhat independently from the above down-grade the error handling in the
shift built-in if the operand is bigger than $# from an error() call (which is
now fatal) to a return 1. I'm not sure if this should be considered a POSIX
"operand error", however this change is needed for now as we trigger that error
while building libncurses. Comparing with other shells, zsh does the same as
our sh before this change (write a diagnostic, return 1), bash behaves as our
sh after this commit (no diagnostic, return 1) and ksh93 and NetBSD's sh treat
it as a fatal error.

Add the POSIX options -v and -V to the 'command' builtin. Both describe the
type of their argument, if it is a shell function, an alias, a builtin, etc.
-V is more verbose than -v.

PR: 77259, 84539

Use prototypes in the MKINIT lines collected by mkinit.

Attempting to unset an undefined variable or function should not be
considered an error according to the Open Group Base Specification.

PR: standards/45738
Submitted by: Matthias Andree <matthias.andree@web.de>
MFC after: 3 days

Remove clause 3 from the UCB licenses.

OK'ed by: imp, core

Changes following CScout analysis:

- Removed dead declarations
- Made objects that should have been declared as static, static.

The changes use STATIC instead of static, following the existing
convention in the rest of the code.

Approved by: schweikh (mentor)
MFC after: 2 weeks

Restore "not found" error message when searching for (or executing)
a program fails because the file or a path component does not exist.

Suggested by: bde

Convert the remaining callers of errmsg() to use strerror(), and remove
errmsg() and its table of error messages.

Remove broken and incomplete support for old releases of System V,
don't support system that implement getcwd(3) with a pipe to /bin/pwd.

Consistently use FBSDID

Don't list shell builtins when the hash command is used (SUSv3)

Stop a null pointer dereference in the builtin hash function.

PR: 36141
Approved by: cracauer
MFC after: 1 week

o __P has been reoved
o Old-style K&R declarations have been converted to new C89 style
o register has been removed
o prototype for main() has been removed (gcc3 makes it an error)
o int main(int argc, char *argv[]) is the preferred main definition.
o Attempt to not break style(9) conformance for declarations more than
they already are.
o Change
int
foo() {
...
to
int
foo(void)
{
...

Fix type builtin for absolute paths and relative paths with directory
names in them.

Also use a colon in the answer of `type` everytime the questioned item
is not usable.

PR: bin/20567

Fix command hash handling on
PATH=... command

Noted by and fix works for Marcel Moolenaar <marcel@scc.nl>

$Id$ -> $FreeBSD$

Add rcsid. Spelling.

Use the __unused attribute where warranted.

Add a type builtin and nuke register keyword usage.

Obtained from: NetBSD

Revert $FreeBSD$ to $Id$

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

Merge in NetBSD mods and -Wall cleaning.

Obtained from: NetBSD, me

Fix for PR#1287. This makes sh behave sensibly in case statements in the
face of aliases. Note, bash doesn't do aliases while running scripts, but
"real" ksh does..

Also:
Reduce redundant .Nm macros in (unused) bltin/echo.1
nuke error2, it's hardly used.
More -Wall cleanups
dont do certain history operations if NO_HISTORY defined
handle quad_t's from resource limits

Submitted by: Steve Price <sprice@hiwaay.net> (minor tweaks by me)

Merge of 4.4-Lite2 sh source, plus some gcc -Wall cleaning. This is a
merge of parallel duplicate work by Steve Price and myself. :-]

There are some changes to the build that are my fault... mkinit.c was
trying (poorly) to duplicate some of the work that make(1) is designed to
do. The Makefile hackery is my fault too, the depend list was incomplete
because of some explicit OBJS+= entries, so mkdep wasn't picking up their
source file #includes.

This closes a pile of /bin/sh PR's, but not all of them..

Submitted by: Steve Price <steve@bonsai.hiwaay.net>, peter

Import the 4.4BSD-Lite2 /bin/sh sources

Requested by: joerg

(Note, this is mostly going to be conflicts, which is expected. Our entire
sh source has a mainline, so this should not change anything except for
a few new files appearing. I dont think they are a problem)

Remove trailing whitespace.

Reviewed by: phk

Added $Id$

Got rid of stupid warning message.

BSD 4.4 Lite bin Sources