make: replace magic numbers with POSIX FILENO constants

No binary change.

make: fix memory leak for command strings

make: fix some more memory leaks

make: all command line overrides go in .MAKEOVERRIDES

Not all variables that start with '.' are internals,
and unless they are explicitly flagged as internal should go
into .MAKEOVERRIDES

Update varname-dot-makeoverrides to check this.

Also avoid using SCOPE_CMDLINE when ReadOnly will do.

make: clean up, test .NOPATH

Trim down the comments in the archive module, as they mainly repeated
the code. Trim down the binary code size in the archive module, as it
is rarely used.

In Var_Parse, delay two variable assignments until they are actually
needed.

make: in parallel mode, print the directory in which a job failed

When multiple targets run in parallel, the "stopped in" line may be
several lines away from the "Failed target" line, making them hard to
correlate.

make: provide more context information for parse/evaluate errors

make: record exit status in GNode

SetErrorVars can now set .ERROR_EXIT which allows
a .ERROR target to ignore the case of .ERROR_EXIT == 6
which means failure happened elsewhere.

Reviewed by:

make: export target scope values

Pass target scope to Var_ReexportVars so that a target process
will see the correct values in its env.
We must then mark any Global scope variable as unexported
so targets without local value get the Global one.

make: clean up comments, constify shell name

make: more consistent error messages

Move %s: progname from Job_CheckCommands to Fatal
to avoid is being repeated when Job_CheckCommands is passed Error.

This means some errors from var also report progname (and level)
which is useful.

Reviewed by: rillig

make: miscellaneous cleanups

make: clean up comments

No binary change, except for line numbers in assertions.

make: clean up comments

No binary change, except for line numbers in assertions.

make: clean up names of local variables

No binary change.

make: inline macros for variable names

The variable name '.MAKEOVERRIDES' was already used in the non-macro
form.

No binary change.

make: clean up calls to Var_Subst

None of the calls to Var_Subst used the return value, and the return
value was always VPR_OK.

No functional change.

Accept whitespace between command specifiers @+- like gmake does. New binutils
does this.

make: change return type of unlink_file back to int

As unlink_file is a wrapper around unlink, use the same encoding for the
possible return values as in the wrapped function. This consistency is
more important than expressing all possible return values in the return
type 'bool'.

https://mail-index.netbsd.org/tech-toolchain/2022/10/06/msg004155.html

No functional change.

make: untangle output handling in jobs mode, remove redundant braces

No functional change.

make: fix handling of null bytes in the output in jobs mode

The test job-output-null failed occasionally, depending on the exact
timing of the child's write and make's read.

make: rename Compat_Run to Compat_MakeAll

No functional change.

make: fix echoing of command with '-' in silent target in jobs mode

Since job.c 1.83 from 2003-12-20, the command had been echoed even if
the target had the attribute '.SILENT'.

In sh-flags.exp, each removed 'echo' command is below a target name
matching the pattern 'opt-?j????-tgt-??s-cmd-?i?', which means that the
target was marked as silent, either through a global '.SILENT'
declaration or the command line option '-s' or the attribute '.SILENT'
on the target.

Reported by Alan Barrett in PR#45356.

make: use unsigned int for line numbers everywhere

Previously, some line numbers were stored as signed int while others
were stored as size_t. Since line numbers are never negative, use an
unsigned type. Since the maximum file size for makefiles is 1 GB (see
loadfile), unsigned int is large enough even on 64-bit platforms.

Using a single data types reduces the number of type conversions. Using
unsigned int improves compatibility with C90 (printf %u instead of %zu),
which is needed by bmake, which is derived from usr.bin/make.

No functional change.

Make the GNode lineno unsigned to fix lint warning in var.c calling
PrintLocation()

make: add missing newline after "cannot continue" message

It was wrong of Parse_File to output an unfinished line and hope for
some other code to finish it. As demonstrated in the test, PrintOnError
did not do that in the case of additional debug output.

To keep the overall behavior as close as possible to before, the other
callers of PrintOnError now have to pass the newline themselves. Passing
strings that start with newlines but don't end with them looked
suspicious anyway.

make: remove redundant braces

No binary change, except for assertion line numbers.

make: rename and inline Targ_Precious

No functional change.

make: reduce code for initializing error handling in shell

No functional change.

make: remove unnecessary words from command line options

Several years ago, the command line options were individual global
variables. The global variable could therefore not be named 'silent'
since that would have conflicted with local variables of the same name.
After moving the global variable to the namespace 'struct CmdOpts',
there is no conflict anymore.

There doesn't seem to be any risk of naming collisions for the names
'touch' and 'query'.

No functional change.

make: rename eunlink to unlink_file

The name eunlink suggested a relation with the similarly named functions
emalloc or esnprintf, but that was misleading. Instead, unlink_file
works like unlink, except that it refuses to remove an empty directory.

No functional change.

make: format comments according to /usr/share/misc/style

Assisted by indent(1), with manual corrections due to its many remaining
bugs.

No functional change.

make: use consistent indentation for statements and continuations

No binary change, except for line numbers in assertions in suff.c.

make: change return type of Compat_RunCommand from int to bool

The documentation was wrong before since status was not restricted to
only 0 or 1.

No functional change.

make: convert GNodeFlags from enum into bit-fields

Now that Enum_ToString is implemented for each type separately, it's
easy to convert them to bit-fields. This gets rid of the magic numbers
12 for CYCLE and 13 for DONECYCLE that left a suspicious gap in the
numbers. This gap was not needed since the code didn't make use of the
relative ordering of the enum constants.

The effects of this conversion are fewer capital letters in the code,
smaller scope for the GNode flags, and clearer code especially when
setting a flag back to false.

One strange thing is that GCC 10.3.0 doesn't optimize GNodeFlags_IsNone
to an single bitmasking instruction, at least on x86_64. Instead it
generates a testb instruction for each of the flags, even loading bit 8
separately from the others. Clang 12.0.1 knows this optimization
though and generates the obvious sequence of movzwl, testl, jz.

No functional change.

make: eliminate common subexpressions in DebugFailedJob

No functional change.

make: fix memory leak in jobs mode with -de (since today)

make: with the option -de in jobs mode, output expanded commands

This helps tracking down the actual cause of build failures in build
systems like NetBSD's build.sh that use highly abstracted commands that
are mainly defined in terms of variables.

Do not ignore write failures.

We should not ignore failure to write to cmdFILE,
meta files and filemon.

Reviewed by: rillig

make: fix error handling in jobs mode

By using the same error handling code as in the branch for non-empty
commands, the behavior is the same again as before 2021-01-29.

make: extract commands-ok check from JobWriteShellCommands

This piece of code did not match the function name and thus could not
reasonably be expected in that function.

In job.c 1.399 from 2021-01-29 I missed exactly this little detail when
I added code to skip the apparently unnecessary creation of empty shell
files. The code I added only handled the happy case, not the case where
the target could not be made.

That code path then differed, leading to a much more verbose error
message than before.

before:
don't know how to make ../missing/no-such.o. Stop

after:
don't know how to make ../missing/no-such.o. Stop
...
`../missing/no-such.o' was not built (made BEINGMADE, ...)!
`muck' was not built (made DEFERRED, type OP_DEPENDS|...)!
`muck' has .ORDER dependency against build-all (made DEFERRED, ...)

Thanks to sjg for finding and reproducing this unintended change of
behavior.

make: clean up comment for JobStart

The parameter 'flags' was renamed in job.c 1.354 from 2020-12-10 without
adjusting the documentation.

The parameter 'previous' was removed in job.c 1.108 from 2006-03-12,
also without adjusting the documentation of JobStart.

No functional change.

make: use consistent variable names, types and braces in DebugFailedJob

No functional change.

make: remove stray space in -de output in jobs mode

In compat mode, having a space in this place makes sense to align the
target name with the command. In jobs mode, since each command is
listed in a separate line, there is no need for the double space.

Print -de error information when running multiple jobs

Problem and patch description from https://reviews.freebsd.org/D29647:

When running `make -de` (without any -j flag) bmake prints which command
failed. However, when using the -j flag the -de flag is ignored. This can
make it rather difficult to determine which command failed in an very
parallel build (especially when combined with the -s flag to avoid
ridiculously large logfiles). For single-threaded builds we can combine
-s with -de to get the failed command but this does not work with -jN
(even with -j1). This patch prints the failed shell script with -de in the
multiple jobs mode as well.

From Alexander Richardson @ FreeBSD

make: use distinct wording for writing to the shell commands file

The word 'write' now means to write to the file that holds the shell
commands to be run later.

The word 'print' is now used exclusively for handling the output of the
child commands and printing them to make's stdout.

No functional change.

make: align name of jobs_table_dump to the other functions

make: remove type name for the abort status in job handling

make: rename PrintOutput to PrintFilteredOutput to avoid confusion

In the default configuration, the function PrintOutput did nothing.
Only if the shell has defined an output filter, something happens at
all.

make: document two previously unknown bugs in job mode

make: rename a few functions to be more descriptive

No functional change.

make: rename ambiguous functions

These two functions have counterparts that include the word 'Do' in
their name, which is confusing.

No functional change.

make: backport to C90

In the past few months I had accidentally used C99 features in the make
code. According to tools/README, tools that are used in the build
system should restrict themselves to C90.

This allows make to build with GCC's options "-pedantic
-Wno-system-headers -Dinline= -Wno-error=cast-qual".

I didn't notice anyone actively complaining though, I just wanted to see
how much work this backporting would be. The identifier __func__ is
still used, as in other tools.

No functional change.

make: use C99 bool type instead of defining its own

No functional change.

Job_TempFile: unlink while signals blocked

Avoid strdup in mkTempFile

Require caller to pass a buffer and size if they
want the tempfile not unlinked.

Add Job_TempFile to handle blocking signals around
call to mkTempFile, so that meta_open_filemon can use it
in jobs mode.

make: remove unnecessary assignment in Job_Start, clean up comments

make: in the Var_ functions, move the scope to the front

This change provides for a more natural reading order in the code.
Placing the scope first makes it immediately clear in which context the
remaining parameters are interpreted.

No functional change.

make: rename some VAR constants to SCOPE

The word "context" does not fit perfectly to the variables that are
associate with a GNode, as the context is usually something from the
outside and the variables are more like properties inherent to the
GNode.

The term "global context" fits even less. Since the thing where
variables are looked up is commonly named a scope, use that term
instead.

This commit only renames the global variables VAR_GLOBAL, VAR_INTERNAL
and VAR_CMDLINE, plus a few very closely related comments. These are:

GNode.vars (because of line breaks)
GNode_Free (dito)
varname-make_print_var_on_error.mk
varname-make_print_var_on_error-jobs.mk

The debug message in Var_Stats is left as-is since there is no unit test
for it yet.

The other renamings (variable names "context", "ctxt", as well as
further comments) will be done in a follow-up commit.

make: replace Global_AppendExpand with Global_Append

All callers with a variable name that is guaranteed to not contain a
dollar sign have been converted to call Global_Append instead of the
previous Global_AppendExpand. After that, Global_AppendExpand was
unused, therefore it was effectively just renamed.

make: replace Global_SetExpand with Global_Set for constant names

make: use shortcut functions Global_SetExpand and Global_AppendExpand

There are many places where global variables are set or appended to. To
reduce clutter and code size, encode the VAR_GLOBAL in the function
name.

The word Expand in the function names says that the variable name is
expanded. In most of the cases, this is not necessary, but there are no
corresponding functions Global_Set or Global_Append yet.

Encoding the information whether the name is expanded or not in the
function name will make inconsistencies obvious in future manual code
reviews. Letting the compiler check this by using different types for
unexpanded and expanded variable names is probably not worth the effort.
There are still a few bugs to be fixed, such as in SetVar, which expands
the variable name twice in a row.

make: indent preprocessor directives consistently

As seen in share/misc/style.

make: always use vfork, never fork

Before compat.c 1.217, job.c 1.390 and main.c 1.504 from 2020-12-27, the
exported make variables were exported from each freshly forked child
process. There was no practical difference though between exporting the
variables from the parent process or the child process since these two
processes share the same address space, except that the forked process
is very limited in what it may actually do. This limitation was
violated on a regular basis.

When an exported variable referred to a variable that used the :sh
variable modifier, this led to a fork from within vfork, which is not
allowed. Since 2020-12-27, exporting the variables is done from the
main process, which prevents this situation from ever occurring.

Since that day, there is no need anymore to distinguish between vfork
and fork, which removes any need for the macro.

make: remove unused Lst_Destroy

The code in job.c that seemed to use it is inside an '#if 0' block.

make: clean up Job_CatchOutput

The switch statement was unnecessarily long since every branch needed to
be terminated with a break or with a comment explaining that it didn't
fall through to the next case label.

make: rename fd-related variables in job.c

In job.c 1.330 from 2020-11-28 I had renamed them because of naming
collisions. At at time I apparently didn't know exactly what these
variables were used for since their new names were incorrect and
misleading. Try better names this time.

make: clean up job.c

There used to be a time when Job_Init only ran the commands of the
special .BEGIN target, but that's several years ago.

The 'becomes' for echoTmpl was confusing since it referred to a time
where echoTmpl shared the same memory location as errOn, which was
really confusing since echoTmpl, as the name says, is a printf template
while errOn is a plain string. All this confusion, just to save 4 or 8
bytes of memory, was not worth it.

Since shellArgv is only used for freeing it and no code is supposed to
ever interpret that memory, rename the variable.

Use sig_atomic_t for caught_sigchld

Reduce unnecessary calls to waitpid

Set a flag when we catch SIGCHLD and
don't call waitpid if it is not set.
Clear it when we call waitpid.

make(1): clean up JobWriteShellCommands

make(1): clean up comments in JobSaveCommands and JobStart

make(1): explain seemingly redundant condition in jobs mode

make(1): remove unnecessary fflush for empty shell files

If nothing is done to any file, there is no need to flush stdout. Move
the call to each of the remaining branches, except for -t mode, where
Job_Touch already takes care of everything.

make(1): rename JobOpenTmpFile to JobWriteShellCommands

The old name didn't reflect that the function not only opens the
temporary file for writing but also actually writes the shell commands
to that file.

make(1): do not create empty shell files in jobs mode

In a chain of dependencies such as "all: build; build: do-build", the
intermediate targets do not have any commands. In jobs mode,
nevertheless, an empty file was created and fed to the shell. This was
unnecessary. See jobs-empty-commands.mk.

The case of the special command line "...", which was suggested on
current-users, is not optimized since it doesn't occur in practice.

Suggested by Mateusz Guzik on current-users:
https://mail-index.netbsd.org/current-users/2021/01/26/msg040215.html

make(1): remove do-not-format markers from comments

These markers had been used inconsistently. Furthermore the source code
had not been formatted automatically before 2020 at all, otherwise there
wouldn't have been any trailing whitespace left.

make(1): make a few more bool expressions more precise

The previous version of lint(1) from a few hours ago didn't catch all
occurrences. And even the current one doesn't catch everything.
Function arguments and return types still need some work. The "return
quietly" from shouldDieQuietly still implicitly converts from int to
_Bool.

No functional change.

make(1): consistently use boolean expressions in conditions

Most of the make code already followed the style of explicitly writing
(ptr != NULL) instead of the shorter (ptr) in conditions.

The remaining 50 instances have been found by an experimental,
unpublished check in lint(1) that treats bool expressions as
incompatible to any other scalar type, just as in Java, C#, Pascal and
several other languages.

The only unsafe operation on Boolean that is left over is (flags &
FLAG), for an enum implementing a bit set. If Boolean is an ordinary
integer type (the default), some high bits may get lost. But if Boolean
is the same as _Bool (by compiling with -DUSE_C99_BOOLEAN), C99 6.3.1.2
defines that a conversion from any scalar to the type _Bool acts as a
comparison to 0, which cannot lose any bits.

make(1): fix lint warnings

Ensure PrintOnError always reports 'stopped in' on first call.

We may still suppress the rest of the noise if shouldDieQuietly
says to, but the 'stopped in' output is too important to lose.
Avoid repeating it though in the same process.

For the case of aborting due to failure detected elsewhere,
exit 6 so we have a clue.

PR: 55578
Reviewed by:

Set wantToken when none available.

PR: 53285
Reviewed by: gson

make(1): add a few remarks to JobOutput

That function is not used in practice. Still, there are a lot of subtle
details that can get wrong in that code.

make(1): format multi-line comments

make(1): re-export variables from the actual make process

Since make uses vfork if available, re-exporting the variables happens
in the address space of the main process anyway, so there is no point in
mentioning anything about "our client process" anywhere.

make(1): omit linear search for command in Compat_RunCommand

make(1): format function definitions consistently

make(1): add str_basename to reduce duplicate code

The function basename from POSIX has a few unfortunate properties, it is
allowed to return a pointer to static memory. This is too unreliable,
therefore this trivial own implementation.

make(1): replace *line with line[0]

Since a line is not an iterator and since the expression *line typically
means "the current element", not "the first character", replacing *line
with line[0] more directly expresses the idea of accessing the first
character of a string.

make(1): rename Var_ExportVars to Var_ReexportVars

make(1): clean up comments around JobPrintSpecials

make(1): rename ParseRunOptions to ParseCommandFlags

make(1): extract ShellWriter_ErrOn from JobPrintCommand

It had been conceptually wrong to modify cmdFlags.echo just to suppress
echoing while enabling error checking.

Now the code in JobPrintCommand speaks for itself and no longer needs
any comments. The few lines at the end have the sole purpose of
restoring the default state (echo + errChk) in the shell file.

make(1): fix typo in comment of JobPrintCommand

make(1): make Job_Touch simpler

make(1): split JobFinishDone into smaller functions

make(1): extract JobFinishDone from JobFinish

make(1): clean up comments in JobFinish

make(1): inline ShellWriter_PrintCmd

The parameter name escCmd was wrong for the call in JobPrintCommand.

make(1): rename Shell.exitFlag to errFlag

Now its name matches the user-visible .SHELL parameter again, reducing a
tiny bit of that mapping chaos.

make(1): move Job.xtraced to ShellWriter

This flag was placed wrong in the Job since it is only necessary as long
as the shell commands are written to the shell file.

Resetting it in JobStart and JobExec was completely misguided since that
is far away from writing the shell commands; this should have been done
in JobPrintCommands instead.

The status of this flag doesn't need to be printed in debugging mode
since it is controlled by a single command line option (-dx) and does
not interact with all the other switches.

make(1): in jobs mode, extract echo control into separate functions

This removes the clutter from the code that calls these functions. That
code is still complicated enough since it modifies variables it
shouldn't.

make(1): make printing of shell commands independent from the job

make(1): in jobs mode, extract writing of shell commands

Right now, the test sh-flags.mk demonstrates many variants to configure
echoing of the shell commands (-s, .SILENT, '@'), error handling (-i,
.IGNORE, '-') and whether the commands are run (-n, -N, .MAKE,
.RECURSIVE, '+').

Even more variants are possible by configuring the shell to have error
control. None of the built-in shell definitions has error control, so
it is unlikely that anybody uses them, but who knows.

Being able to configure these details at 3 levels is good, but what
makes all this really hard to understand is that some of these switches
interact in non-obvious ways. For example, in jobs mode, a single
command can change job->ignerr (in JobPrintSpecialsEchoCtl), which will
affect all further commands of that job.

The goal of this refactoring is to make the code easier to understand by
making the switches on the job level constant and by moving all
modifications to them to the ShellWriter.

make(1): remove const from function parameters

These have been left-overs from refactoring, when these pieces were
extracted to separate functions.

make(1): inline Targ_Ignore and Targ_Silent

Each of these functions was only used 2 times, and each of these calls
used a different part of the whole expression.

make(1): mark strange code that calls exit(signo)

In other places, the exit status of make is carefully chosen to be 0
(success), 1 (did not make everything), 2 (other error). Using a signal
number is not guaranteed to be consistent among operating systems and is
therefore a weird choice.

make(1): document why runIgnImpl is used in jobs mode

make(1): rename shell flags in struct Shell

The previous name 'echo' was too unspecific, given that the other fields
in the same struct are called echoOff, echoOn, echoTmpl.

make(1): clean up comments for struct Shell

make(1): name Shell fields consistently

They are all templates with a single %s placeholder, so embed this
unambiguously in the variable name. The previous variable names didn't
provide any clue that echoCmd, execIgnore and errExit had a lot in
common.

make(1): clean up JobPrintCommand

The local variable 'cmd' had been used for both the unexpanded and the
expanded command. This in turn had required the parameter to be
non-const.

make(1): eliminate global variable numCommands

It had only been used in a very small scope. The elimination of
Lst_ForEach made it easy to turn this global variable into a local
variable.

make(1): extract JobOpenTmpFile from JobStart

make(1): rename RunFlags to CommandFlags

Flags of similar names are also available in CmdOpts (global command
line options) and Job. The new name CommandFlags emphasizes that these
flags apply to the smallest scope possible, which is a single command,
as opposed to the whole GNode/Job.

make(1): split Job.errOffOrExecIgnore into separate fields

Like errOnOrEcho, these two fields were completely unrelated. One of
them expected a %s placeholder, the other didn't.

make(1): split Job.errOnOrEcho into separate fields

They have never been related. Furthermore, Job.errOn is really tricky
to reach at all. To do that, one has to define a custom shell and set
hasErrCtl=true. The manual page has an example for that, but it's
probably not use in practice, just like the possibility to use the C
shell for all commands.

make(1): negate Job.silent to echo

This gets rid of a few explicit negations and an implicit negation in
the variable name.

make(1): unpack struct JobFlags

The job flags are only used individually.

make(1): rename commandShell to shell

The word 'command' was not necessary for understanding the variable.

make(1): split JobFlags into separate fields

Having all these flags in a single bitmask makes it harder to see where
exactly they can possibly be used since their state could also be
modified using the unsuspicious job->flags = 0. Using individual names
just leaves the single memset, and that is only used during
initialization.

make(1): in JobStart, negate 'noExec' to 'run'

make(1): fix theoretical type mismatch for Job_Touch

make(1): in jobs mode, invert local variable noSpecials to run

make(1): negate RunFlags.silent to become RunFlags.echo

This reduces the number of negations in the code.

make(1): move flags errOff into RunFlags.ignerr

make(1): move flag runAlways into RunFlags

make(1): move flag shutUp into struct RunFlags

Running a command is controlled by several flags. Instead of passing
them around individually, it's easier to have them grouped.

make(1): split JobPrintSpecials into manageable pieces

make(1): extract JobPrintSpecials from JobPrintCommand

make(1): extract InitShellNameAndPath from Shell_Init

This gets rid of the ugly "else #endif".

make(1): clean up debug logging

make(1): replace signal handling macros with local functions

make(1): indent job.c with tabs instead of spaces

make(1): remove duplicate code for job output

make(1): replace macro MESSAGE with local function

The first parameter of the macro was always stdout, and there was no
apparent reason to pass anything else there.

Let the compiler decide whether to inline this or not, it's not
time-critical.

make(1): remove comment decoration

make(1): remove redundant assignments

make(1): remove unused parameter from JobOutput

make(1): use consistent variable names for list nodes

make(1): use space instead of tab for preprocessor directives

make(1): reduce memory allocation for GNode.parents and GNode.children

make(1): remove pointer indirection from GNode.commands

Just to save a few memory allocations. No noticeable effect on the
performance though.

make(1): rename conflicting global variables called 'error'

When compiling make in all-in-one mode, these variable names conflict.
They could have been merged into a single variable, but that would have
required to make it a global variable for the other modules as well.
The parse module has a similar variable called 'fatals'. All these can
possibly be merged into a single variable, but not now.

make(1): rename global variables in job.c

When compiling make in all-in-one mode, the variable name nfds conflicts
with the local variable name nfds in meta_compat_parent.

The variable name jobfds was misleading. It has nothing to do with file
descriptors, it's just an array of jobs.

make(1): fix error handling for dependency of .END in -k mode

Fix one bug, find 4 new ones. All these bugs have been around since
2005-05-08, when dependencies on the .BEGIN, .END and .INTERRUPT nodes
were implemented. Before that, checking gn->made == ERROR was
appropriate, but adding the dependencies made ABORTED a new possible
error value from Compat_Make.

make(1): use comparisons in boolean expressions

The generated code stays exactly the same.

make(1): align end-of-line comments with tabs

make(1): use postfix increment where possible

make(1): clean up JobExec, JobStart, JobDoOutput

The comments about temporary files for shell output are no longer
relevant since the output is always handled via pipes.

make(1): remove indentation in JobOutput

make(1): rename DEFAULT to defaultNode

In C, uppercase names are typically used for constants, and this is not
a constant.

make(1): use stderr for error message about failed touch

Error messages belong on stderr, not stdout. Since at least 1993,
stdout had been used for no good reason. Plus, in all these years,
nobody ever tested this code path since otherwise the missing newline at
the end of the error message would have been obvious.

This is a good candidate for the "oldest bug in make" trophy.

make(1): clean up Job_Touch

Several more outdated comments were removed. To compensate for that,
several new comments mark places where errors are not properly
propagated.

make(1): rename JobClose to JobClosePipes, remove confusing comments

The comments referred to w_status, which had been there in 1993, but not
anymore. Several other details were outdated as well.

make(1): fix grammar: setup -> set up

make(1): replace macro DBPRINTF with JobPrintln

This macro had been bad in several ways. Its name started with DB, an
unnecessary abbreviation for DEBUG. Many places that used this macro
used it with the same format string, "%s\n". This format string can
better be expressed in a function name, JobPrintln.

Only in a few places was the macro used with different format strings.
The one for "set -x" was even forced to arbitrarily separate the
argument from the format string in order to match the expected macro
parameters.

A better choice would have been to use the standard form "%s\n", "set
-x" from the beginning. Anyway, that call is calling JobPrintln as well
now.

The remaining templates are user-specified, and if anyone should ever
define a completely custom shell with echo handling and error handling
enabled, they will easily crash make when the templates don't contain
exactly one %s conversion. That's the responsibility of the user, and
it has always been.

make(1): rename JobState to JobStatus

The status is only part of the whole state of a job.

make(1): use strict typing in conditions of the form !var

make(1): change return type of Dir_MTime to void

Only some callers actually needed the updated time, and because of the
many branches, it was difficult to see that the return value was indeed
gn->mtime all the time.

make(1): remove unused JOB_IGNDOTS

It had been used in 1993 at least, don't know when it was finally
removed.

make(1): format source code in JobStart and JobDoOutput

make(1): fix indentation in JobExec

make(1): clean up debug output for running jobs

There is no remote anymore, therefore mentioning locally is redundant.
It was a bad idea anyway to build format strings that work with parts of
words.

make(1): fix type of Job.suspended

make(1): clean up JobStart

Initialize the fields in declaration order.
Initialize Job.flags in a single assignment.

make(1): fix indentation in JobPrintCommand

make(1): add pp_skip_hspace to skip horizontal whitespace during parsing

make(1): clean up code stylistically

* Replace character literal 0 with '\0'.
* Replace pointer literal 0 with NULL.
* Remove redundant parentheses.
* Parentheses in multi-line conditions are not redundant at the
beginning of a line.
* Replace a few !ptr with ptr == NULL.
* Replace a few ptr with ptr != NULL.
* Replace (expr & mask) == 0 with !(expr & mask).
* Remove redundant braces for blocks in cases where the generated code
stays the same. (Assertions further down in the code would get
different line numbers.)
* Rename parameters in CondParser_String to reflect the data flow.
* Replace #ifdef notdef with #if 0.

The generated code stays exactly the same, at least with GCC 5.5.0 on
NetBSD 8.0 amd64 using the default configuration.

make(1): rename Var_Set_with_flags to Var_SetWithFlags

Now that the function is exported from the var module, it should stick
to the naming conventions for public functions.

make(1): rename dieQuietly to shouldDieQuietly

It was too confusing to have a function named die that doesn't actually
die. Plus, the return type int didn't give any clue about what the
function actually returns.

make(1): remove redundant parentheses from sizeof operator

The parentheses are only needed if the argument is a type, not an
expression.

make(1): fix indentation of source code

make(1): rename JobMatchShell to FindShellByName

The word 'match' was too unspecific.

make(1): negate NoExecute to GNode_ShouldExecute

make(1): extract EscapeShellDblQuot from JobPrintCommand

make(1): extract ParseRunOptions from JobPrintCommand

make(1): do not look up local variables like .TARGET anywhere else

Nobody defines a global variable named .TARGET since that would have
many unpredictable effects, applying to all targets at once.

Nobody defines an environment variable named .TARGET since that's
against the naming conventions for environment variables and would have
the same effect.

Because of this, there is no point looking up the variables that are
local to a GNode anywhere else. This means they cannot come from the
environment and thus their value doesn't need to be freed after use,
which makes the code simpler.

The newly added accessor functions in make.h refer to external
functions, but since that header is not used anywhere outside of
usr.bin/make, it doesn't matter. Between 2020-08-25 and 2020-10-30,
that header had been referenced by usr.bin/xinstall.

make(1): change char * to void * in Var_Value

The only purpose of the parameter freeIt is to free the memory
associated with the return value. To do this, no pointer arithmetic is
needed. Therefore, change to a void pointer, to catch accidental use of
that pointer.

make(1): fix indentation in source code

make(1): rename VAR_CMD to VAR_CMDLINE

Since make has to do with both the command line and child commands, the
former name was confusing.

make(1): remove "Results: none" from the documentation of void functions

make(1): use proper enum constants for aborting, instead of 0

make(1): remove unnecessary double negation in Job_ParseShell

make(1): group the command line options and arguments

By having a single struct that holds all command line options and
arguments, it is easy to see in the code when such a command line
argument is modified. It also cleans up the namespace since the command
line options don't follow a common naming style. Having them in a
struct also means that there is a single place for putting the
documentation, not two as before.

The struct also suggests to extract the initialization code out of main,
which is still too large, having more than 400 lines of code and
covering far too many topics.

make(1): fix comments in job.c

make(1): fix typos in job.c comments

make(1): add GNode_Path to access the path of a GNode

make(1): refactor JobDeleteTarget

Splitting the many conditions into separate pieces makes the code more
understandable.

make(1): inline macro STATIC

It was a leftover from the remote code, which was removed on 2006-03-11.

make(1): convert result of JobStart from macros to enum

Right now the result is not used by any code.

make(1): convert abort reason macros to enum

make(1): invert condition in Job_CheckCommands, reduce indentation

make(1): reduce indentation in Job_CheckCommands

make(1): split and invert complex condition in Job_CheckCommands

make(1): inline error message

This saves a variable, and since bmake needs to adhere to C90, this
variable declaration prevents refactoring the code.

make(1): clean up API comment in job.c

make(1): negate OP_NOP and rename it to GNode_IsTarget

make(1): rename Shell fields

Some of them should have been in an anonymous union, or should just kept
as separate fields. There is only a small, fixed number of shells,
therefore it doesn't make sense to squeeze two completely separate
meanings into a single field just because that field happens to have a
fitting data type.

make(1): describe the Shell fields by topic

Describing each field in isolation does not help to understand how these
fields work together.

make(1): add field names in declaration of DEFSHELL_CUSTOM

make(1): remove unused null shell

make(1): decorate shell declarations with field names

Without these names, the declarations were hard to read, even though
they were already grouped into lines. This grouping was inconsistent
though and the many string literals were hard to tell apart.

make(1): convert JobState and JobFlags to enum types

Both GCC and Clang complained when JobFindPid had its parameter "status"
as an int. Strangely both compilers complained about a comparison
between unsigned and signed int, even though enums are defined to be
int, not unsigned.

make(1): move handling of the "..." command to JobPrintCommands

Over there, the current list node is known and thus doesn't need to be
searched again.

make(1): remove void pointers from JobPrintCommand

make(1): remove redundant type casts

This mainly affects the void pointers in callback functions for lists.
These had been necessary once when the parameter type was still
ClientData instead of void pointer.

make(1): clean up headers, no functional change

make(1): move struct Shell from job.h to job.c

This type is only needed in job.c.

make(1): rename Lst_Init to Lst_New

For the other types such as HashTable and Buffer, the Init function does
not allocate the memory for the structure itself, it only fills it.

make(1): replace execError with execDie

All calls to this function were followed by _exit(1).

make(1): normalize initialization and cleanup of the modules

make(1): remove macro FILENO

This macro was obviously wrong since it would have converted file
numbers above 127 to very large numbers, close to 2^32.

The fact that it didn't blow up at all is that this macro was only ever
given the file descriptor 4 as an argument, which can well be
represented as a char, be it signed or unsigned. And this is how the
story goes:

In Job_Init, two jobs are started. The server job allocates file
descriptors 15 and above. The childExitJob is created next, and the
pipe that it creates is [3, 4]. Using F_DUPFD, fd 3 is mapped to fd 5,
and fd 3 is closed. After that, fd 4 is mapped to fd 3 (which had just
been closed), and fd 4 is closed.

After this initialization, file descriptors 0, 1, 2, 3 and 5 are taken.
This leaves a gap at file descriptor 4, and this gap is filled whenever
cmdFILE is created.

Because of this particular order of events, the macro is not necessary.
If it should ever become necessary on platforms like the old SunOS, the
parameter minfd for JobCreatePipe should be increased to 5, which would
leave the file descriptors 3 and 4 to be used by stdio streams.

On 1995-11-03, when the macro was added, SunOS must have been in its
very early development. In Solaris 8 (released in January 2000),
FILE._file is already an unsigned char, therefore the seeming need for
this macro must have been due to an older version, probably from the 2.x
series of SunOS.

make(1): prepare job.c, main.c, parse.c, suff.c for WARNS=6

In job.c, GCC 5 complains about the macro FILENO that it has type
unsigned int, which is then passed as the argument of dup2, which
expects a simple int. Maybe this workaround from 1995 is not necessary
anymore, or maybe it is but can be restricted to the few affected
platforms.

This leaves meta.c and the filemon files to be converted. I didn't do
them since they are not well covered by the unit tests.

make(1): revert previous commit

It had accidentally reverted all the work from the past few days.

make(1): fix double-free bug in -DCLEANUP mode (since 2020-10-02)

The bug had been introduced with dir.c 1.155 on 2020-10-02 22:20:25. In
that commit, openDirectories was replaced with a combination of a list
with a hash table, for more efficient lookup by name.

Upon cleanup, OpenDirs_Done is called, which in turn called
Dir_ClearPath. Dir_ClearPath takes full ownership of the given list and
empties it. This was no problem before since afterwards the list was
empty and calling Lst_Free just frees the remaining list pointer.

With OpenDirs, this list was combined with a hash table, and the hash
table contains the list nodes, assuming that the OpenDirs functions have
full ownership of both the list and the hash table. This assumption was
generally correct, except for the one moment during cleanup where full
ownership of the list was passed to Dir_ClearPath, while the hash table
still contained pointers to the (now freed) list nodes. This by itself
was not a problem since the hash table would be freed afterwards. But
as part of Dir_ClearPath, OpenDirs_Remove was called, which looked up
the freed directory by name and now found the freed list node, trying to
free it again. Boom.

Fixed by replacing the call to Dir_ClearPath with code that only frees
the directories, without giving up control over the list.

make(1): remove dead code from JobOutput

make(1): use consistent pattern for parsing whitespace

The pp and cpp in the function names stand for "parsing position" and
"const parsing position".

make(1): fix shell definition for csh

The bug has been there since at least 1993-03-21. This is strong
evidence that nobody has ever used the csh in parallel mode.

make(1): inline Str_FindSubstring in JobOutput

make(1): remove redundant function prototypes

make(1): replace Lst_Open with direct iteration

make(1): replace += 1 with ++ and -= 1 with --

Just for visual consistency. The generated code stays exactly the same.

make(1): make debug logging simpler

This avoids referring to the debug_file variable in many places where
this implementation detail is not necessary.

make(1): make debugging code shorter

make(1): split Job.jobPipe into 2 separate fields

Just because these file descriptors have to be in an array when they are
created is not reason enough to keep this array and a few access macros
in the Job struct. It's better to have separate fields, as they can be
documented independently.

make(1): in JobCreatePipe, use descriptive field names

make(1): fix indentation in JobCreatePipe

make(1): normalize whitespace in source code

There is no more space tab. Either only tabs or only spaces or tabs
followed by spaces, but not spaces followed by tabs.

make(1): reduce line noise in Job_ParseShell

The generated code is exactly the same.

make(1): fix indentation in Job_ParseShell

make(1): clean up comments in job.c and make.c

make(1): revert migration from Lst_ForEachUntil to Lst_ForEach

There is a crucial difference between these functions, in that
Lst_ForEachUntil can cope with a few concurrent modifications while
iterating over the list. This is something that Lst_ForEach doesn't do.

This difference led to a crash very early in NetBSD's build.sh.

make(1): inline and remove LstNode_Prev and LstNode_Next

These functions made the code larger than necessary. The prev and next
fields are published intentionally since navigating in a doubly-linked
list is simple to do and there is no need to wrap this in a layer of
function calls, not even syntactically. (On the execution level, the
function calls had been inlined anyway.)

make(1): replace a few Lst_ForEachUntil with simpler Lst_ForEach

make(1): clean up API for finding and creating GNodes

The previous API had complicated rules for the cases in which the single
function returned NULL or what it did. The flags for that function were
confusing since passing TARG_NOHASH would create a new node even though
TARG_CREATE was not included in that bit mask.

Splitting the function into 3 separate functions avoids this confusion.
It also reveals several places where the complicated API led to
unreachable code. Such code has been removed.

make(1): document details on why JobRun runs .END in compat mode

make(1): rename Lst_ForEach to Lst_ForEachUntil

Since the callback function returns a terminating condition, this is not
really a foreach loop.

Many of the calls to Lst_ForEachUntil don't make use of the terminating
condition, and several don't modify the list structurally, which means
they don't need this complicated implementation.

In a follow-up commit, Lst_ForEach will be added back with a much
simpler implementation that iterates over the list naively, without a
terminating condition and without taking the iteration state from
Lst_Open/Lst_Next/Lst_Close into account. The migration to this simpler
implementation will be done step by step since each callback function
needs to be examined closely.

make(1): merge duplicate code for delayed commands

make(1): fix assertion failure in -j mode with .END node

There had been two separate global variables for the .END node, and in
parallel mode, only the one in jobs.c was initialized.

The code in JobRun heads over to Compat_Make without calling Compat_Run
first, which left the variable ENDNode uninitialized.

make(1): prepare Var_Subst for proper error handling

Returning a VarParseResult instead of a string makes it possible to let
the error bubble up, until it reaches the main expression.

make(1): add specific typedefs for lists

These typedefs are only intended to help human readers, they do not
provide any type-safety. They also make the pointers explicit, which
had been hidden before by the typedef for Lst and LstNode. Typing a few
'*' is less work than finding out which of the many types are pointers
and which aren't.

In meta.c, the variable "ln" served two completely different purposes,
which have been split again. Register allocation is the job of the
compiler, not of the human source code reader.

make(1): clean up RCSID blocks

These blocks mostly consisted of redundant structure, following the same
#ifndef pattern over and over, with only minimal variation.

It's easier to maintain if the common structure is only written once and
encapsulated in a macro.

To avoid "defined but unused" warnings from GCC in the case where
MAKE_NATIVE is not defined, I had to add volatile. Adding
MAKE_ATTR_UNUSED alone would not preserve the rcsid variable in the
resulting binary.

make(1): fix name and prototype of Compat_RunCommand

By convention, exported identifiers are written with underscore.

The prototype of an exported function must not use void * just because
it is used in Lst_ForEach. This is an implementation detail and must
remain so.

make(1): add wrappers around ctype.h functions

This avoids casting the argument to unsigned char, and to cast the
result of toupper/tolower back to char.

make(1): replace *a->b with a->b[0]

This allows the code to be read strictly from left to right. In most
places this style was already used.

make(1): clean up comments in job.c

make(1): replace brk_string with Str_Words

The API is much simpler, and there is less detail that is exposed by
default and fewer punctuation to type on the caller's side. To see that
there is some memory to be freed, one would have to look into the
struct. Having part of the return value as the actual return value and
the rest in output parameters was unnecessarily asymmetrical.

make(1): remove ineffective malloc_options variable

According to jemalloc(3), the variable must be called _malloc_options,
with a leading underscore, to have an effect.

Renaming the variable indeed enables the option. There's not much point
having this variable around though, since it neither detects a trivial
double-free nor freeing an invalid pointer in the following code
snippet:

char *asdf = bmake_malloc(10);
fprintf(stderr, "%c\n", *asdf);
free(asdf + 8);
free(asdf);
free(asdf);
exit(1);

Instead, it just crashes with a segmentation fault.

make(1): rename Lst_Memeber to Lst_FindDatum

The new name nicely aligns with Lst_Find and Lst_FindFrom.

make(1): remove ReturnStatus, SUCCESS and FAILURE

These are used in so few places now that it is easier to use a simple
Boolean for them.

make(1): rename LstNode functions to match their type

make(1): clean up indentation, comments and variable scope in job.c

make(1): remove unused variable not_parallel

The variable malloc_options seems unused as well, but the manual page
for jemalloc(3) mentions it.

make(1): remove trailing 'S' from names of Lst functions

The migration from null-passing Lst functions to argument-checking Lst
functions is completed.

There were 2 surprises: The targets list may be NULL, and in Dir_AddDir,
the path may be NULL. The latter case is especially surprising since
that function turns into an almost-nop in that case. This is another
case where probably 2 independent functions have been squeezed into a
single function. This may be improved in a follow-up commit.

All other lists were fine. They were always defined and thus didn't
need much work.

make(1): migrate Lst_IsEmpty to Lst_IsEmptyS

make(1): migrate Lst_Succ to Lst_SuccS

make(1): migrate Lst_ForEach to Lst_ForEachS

Most lists are always valid. Only the "targets" variable may be null in
some cases, probably.

make(1): migrate Lst_ForEachFrom to Lst_ForEachFromS

make(1): make brk_string return size_t for the number of words

make(1): clean up headers

Remove redundant headers that are already included by "make.h".
Make <assert.h> available to all compilation units that use "make.h".

Add .SHELL as read-only variable

The .SHELL variable represents the shellPath used to run
scripts.

Reviewed by: rillig, christos

make(1): require argument of Lst_Member to be non-null

Since the lists don't contain null pointers, it doesn't make sense to
search for a null pointer. All calls but one already had obviously
non-null arguments. The one remaining call using targ->suff has been
guarded for now.

The code for Lst_Member became much simpler than before. Partly because
the old code had an extra condition for circular lists, which are not
used by make.

make(1): convert remaining Lst_AtEnd to the stricter Lst_Append

The general-purpose list library that is included in make allows to call
Lst_AtEnd for invalid lists, silently ignoring this programming error.
This is a flexibility that make doesn't need.

Another unneeded "feature" is that list items can theoretically be null
pointers. This doesn't make sense as well and is therefore not needed
by make.

These programming errors are now caught early by assertions.

make(1): replace "(void)Lst_AtEnd" with stricter "Lst_AppendS"

This change ensures that there is actually something added to the list.
Lst_AtEnd had silently skipped the addition if the list was invalid
(null pointer), which was not intended in these cases. The "(void)" is
assumed to mean "I know that this cannot fail", while it could also mean
"I don't care whether something actually happened".

Running "./build.sh -j6 tools" still succeeds after this change,
therefore chances are very low that this change breaks anything. If
there is any change, it's an obvious assertion failure. There is no
silent change in behavior though.

make(1): fix constness of shellArgv

This is needed to build with -DCLEANUP, which apparently nobody did for
the last 17 years, or at least since -Werror was introduced.

make(1): remove form feeds in the code

make(1): remove unused code for circular lists

The list library had probably been imported from a general-purpose
library that also supported circular lists. These are not used by make
though.

After replacing Lst_Init(FALSE) with Lst_Init(), only a single call to
Lst_Init remained with a non-constant argument, and that was in
Lst_Concat, which was to be expected.

make(1): replace str_concat with str_concat2 and str_concat3

The new functions have a simpler interface, and str_concat3 is even more
general-purpose, since the middle string is no longer required to be
exactly of length 1.

make(1): use consistent indentation in source code

Tabs for multiples of 8, then spaces.

The usage string has been kept as-is since the spaces there are
indentional and do influence the output.

make(1): avoid calls to free(3) in the common case of a NULL pointer

make(1): remove dead code from Var_Subst

The first parameter from Var_Subst had been a literal NULL in all cases.
These have been fixed using this command:

sed -i 's|Var_Subst(NULL, |Var_Subst(|' *.c

The one remaining case was not found because the "NULL," was followed by
a line break instead of a space.

The removed code probably wouldn't have worked as expected anyway.
Expanding a single variable to a literal string would have led to
unexpected behavior for cases like ${VAR:M${pattern}}, in case pattern
would contain an unescaped ':' itself.

make(1): rename Varf_Flags to VarEvalFlags

In var.c there are lots of different flag types. To make any accidental
mixture obvious, each flag group gets its own prefix.

The only flag group that is visible outside of var.c is concerned with
evaluating variables, therefore the "e", which replaces the former "f"
that probably just meant "flag".

make(1): remove trailing whitespace

make(1): remove redundant parentheses around return values

make(1): remove useless parameter from Var_Set

The enum corresponding to this int parameter is only defined in var.c,
which makes it impractical for the outside to set this parameter to
anything but 0.

On x86_64, this reduces the size of the resulting executable by 5 kB.

Avoid unnecessary noise when sub-make or sibling dies

When analyzing a build log, the first 'stopped' output
from make, is the end of interesting output.

Normally when a build fails deep down in a parallel build
the log ends with many blockes of error output from make,
with all but the fist being unhelpful.

We add a function dieQuietly() which will return true
if we should supress the error output from make.
If the failing node was a sub-make, we want to die quietly.

Also when we read an abort token we call dieQuietly telling we
want to die quietly.

This behavior is suppressed by -dj or
setting .MAKE.DIE_QUIETLY=no

Reviewed by: christos

Fix meta_compat mode.

We need to call meta_job_output() from meta_compat_parent()
Also, the need for extra fds for meta mode only applies when
using filemon_ktrace.c.
Since we hope to see more interfaces like that, test for
USE_FILEMON_DEV being not defined.

Reviewed by: riastradh

Reimplement make(1) meta mode without filemon(4).

branches: 1.195.2;
Just skip polling job token pipe.

The sigchld pipe ensures no busy wait.

PR: 53285
Reviewed by:

PR/53285: Andreas Gustafsson: Build times tripled with make/job.c 1.193
Revert previous:
2018.05.12.15.14.49/bracket.db:build_wall_time=4896.09
2018.05.12.18.17.04/bracket.db:build_wall_time=16268.98

Skip setting wantToken.

polling the job token pipe adds a lot of overhead
for little gain.
For now, just leave wantToken=0

And avoid busy waiting when no tokens are available and
no jobs are running.

Reviewed by: christos

branches: 1.192.2;
Typos.

Make compat.c handle SIGINT etc more like job.c

If there is a running child, pass the signal on, and
wait for it to exit before we self-terminate.

Reviewed by: christos

Plug memory leak in JobPrintCommand.

CID 978366

Check fcntl return values out of paranoia.

CID 975277
CID 975278

branches: 1.188.2;
Add a .DELETE_ON_ERROR: magic target that causes *failed* targets as
well as *interrupted* targets to be deleted. The name and behavior of
the variable matches gmake.

Also fix a glitch in newline output on error in compat mode that I
discovered while doing it.

Closes PR 51376.

branches: 1.187.2;
Propagate errors from filemon.

If we encounter errors producing a .meta file,
we should not consider the target completed successfully.

Collapse the 3 boolean parameter to 1 flags parameter. No functional change.

remove free NULL checks (Tilman Sauerbeck)

provide missing FD_CLOEXEC for the havenots.

Use FD_CLOEXEC (Tilman Sauerbeck)

Preserve $$ in := assignments..

FOO=\$$CRAP
BAR:=${FOO}

all:
echo ${FOO}
echo ${BAR}

Add Boolean wantit to Var_Parse and Var_Subst

wantit will be FALSE when we are just consuming to discard
in which case we skip "expensive" things like Cmd_Exec.

Reviewed by: christos

Explain what max is and handle i reaching it again.

Don't use memcpy for overlapping ranges. Simplify, memmove is good
enough to skip empty ranges.

jobs mode handles '-' very differently to compat mode, but this is not
reflected in make.1
Also fix handling to match the behavior described in the comment
of JobPrintCommand (only matters if shell sets hasErrCtl).

A better long term solution is needed since the current behavior
is sub-optimal wrt '-' and different from all other makes.

Reviewed by: christos

PR/48367: David A. Holland: Mark possible submake nodes so that we can
avoid closing the job pipe on exec for them in order to make recursive
makes work in parallel.

branches: 1.176.2;
Move the call to Job_SetPrefix() to Job_Init() so that
makefiles have had a chance to set .MAKE.JOB.PREFIX

Allow an empty .MAKE.JOB.PREFIX to supress output of --- job --- tokens

If commandShell hasErrCtl is true, set shellErrFlag for use by
CompatRunCommand() so that behavior in jobs and compat mode
remains consistent.

JobCreatePipe: do fcntl() *after* any fiddling of fd's
to avoid leaking descriptors.
Job_ServerStart: set closed on exec for jp_0 and jp_1.

Add a .STALE special target that gets invoked when dependency files contain
stail entries.

Keep track of the location where a dependency is defined, so we can report
about it.

PR/45042: Thomas Cort: HAVE_foo for setrlimit(2) and setpgid(2)

fix broken logic:
- poll can return EINTR, it is not restartable like read/write
- check poll return
- it does not make sense to check readyfd() is nready < 0
- check read return
- always call Job_CatchChildren, it is harmless
- short circuit scanning of the poll array if we got all the ready descriptors

tools build is non-c-99

avoid the long shadow of i

Don't wait forever for output from a child job.

Check read and write errors to avoid warnings from linux.
XXX: Should we print an error and exit instead?

Remove Check_Cwd - it is no longer needed.

branches: 1.163.2;
An X!= command, can cause Shell_Init() to be called before Job_ParseShell().
Job_ParseShell() should call Shell_Init() again if needed
to ensure shellPath is correct.

Replace __dead, __unused and the various printf format attributes
with versions prefixed by MAKE_ATTR_* to avoid modifying the
implementation namespace. Make sure they are available in all places
using nonints.h to fix bootstrap on Linux.

Remove recheck hackery that caused extra stats, and explicitly ask for
recheck when needed. Before it used to be the case that we could only
use the cached entry once. Once the cached entry was used, we removed
it from the cache. Now it is kept forever.

branches: 1.160.2;
Use __dead consistently. If it doesn't exist, define it away.

In meta mode, we create .meta files for most targets.
These capture all the interesting data - useful for debugging.
In such cases there is no need to replicate commands in the build log.
Rather than run the entire build .SILENT, allow meta mode to set that flag
per target iff a .meta file is created.
Normal behavior is retained for targets where no .meta file is created,
ensuring that no build data is lost.

- remove gcc-4.5 warnings
- simplify job printing code
- document non-literal format strings

Don't attempt to touch special targets; make -t would create .END in every
directory.

We must now call meta_job_finish() after meta_job_error()
so that .ERROR_META_FILE is captured.

Add meta.c which implements "meta" mode for make.
In this mode, a .meta file is created for each target, capturing
the expanded commands used, any command output, and if filemon(9)
is available, a record of system calls which are of interest.
Not enabled unless USE_META=yes is set when building make.
Also, if FILEMON_H exists, meta.c will be compiled to use filemon(9).

In jobs mode, if we cannot make a target we die without any further
clue. Call PrintOnError() before DieHorribly().

wait[pid]() is called in several places.
If we encounter an error and run the .ERROR target, we may
reap a pid which jobs is waiting for. Ensure that we
cleanup so that make isn't left waiting for an already
deceased child.

PR/43643: David A. Holland: dash prefix broken with make -j
As the comment says in JobPrintJob set JOB_IGNERR with - commands. Fix
whitespace so that the output is consistent, although there are two separate
execution logic implementations (command.c, job.c)

JobFinish: call PrintOnError if we detected an error we are not ignoring.
This gives the .ERROR target a chance to run at the first sign of trouble.

We have required sigaction() for quite a while.
Use bmake_signal() - a wrapper around sigaction()
rather than signal() to ensure that signals are handled
consistently.

On darwin at least, vfork() fails in child of vfork().
It probably shouldn't work anyway, so avoid this.
We use the macro vFork() - a function seems to cause problems
and is unnecessary overhead.

Just because $TMPDIR is set does not mean it is valid.
Add a central function for creating temp files so we have one place to
audit.

Reviewed by: dh

Add:

.error "message"
.warning "message"
based on FreeBSD implementation.
add .info while were at it.

.ERROR: a target to run on error.
We pass the failing GNode to PrintOnError so it can set
.ERROR_TARGET.

.MAKE.MAKEFILE_PREFERENCE
As a means to control make's list of prefered makefile names.
(Default: "makefile Makefile")

.MAKE.DEPENDFILE
Names the file to read dependencies from
(Default ".depend")

.MAKE.MODE
Processed after all makefiles are read.
Can put make into "compat" mode (more to come).

Fix:

compat.c: Error code should not be sent to debug_file.
Make_DoAllVar: use DONE_ALLSRC to avoid processing a node multiple times.
ReadMakefile: we can simply use doing_depend to control setting MAKEFILE.

JobExec: child of vfork must not empty the sigset_t that parent will
restore. Use a separate mask.

Honour the TMPDIR environment variable instead of always using /tmp
as a place to store temporary files.

branches: 1.144.2;
Change 'ClientData' to 'void *' so that relevant parameters can
be made 'const void *'.

if brk_string fail, so do we.

Use NULL instead of -1 cast to the relavant type (usually via NIL).
This was a suggestion from christos - so blame him if there is a deep
reason for using -1 :-)

branches: 1.141.2;
If Parse_Error is called after curFile is invalid, set it to a dummy
to avoid seg-fault. Also the NULL filename will cause ParseVErrorInternal
to skip trying to report file and line number info.

If CondToken is called outside the context of a .if etc, variables in
the expression being parsed will already be expanded, so allow for
an unqouted number to jump us back into the compare logic.
This allows ${${SomeNumber:U42} > 0:?True:False} to work anywhere.

Job_Finish should process postCommands if it has commands or children.

Don't use emalloc and friends directly, but call them consistently
bmake_malloc and friends. Implement them via macros for the native case
and provide fallback implementations otherwise. Avoid polluting the
namespace by not defining enomem globally. Don't bother to provide
strdup and strndup, they were only used for the estrdup and estrndup
comapt code.

This addresses the presence of emalloc in system libraries on A/UX and
resulted strange issues as reported by Timothy E. Larson.

back all changes out until I fix it properly.

Improve dh's fix:
- make sure i is initialized
- don't subtract 1 in comparison, add 1 to the other side

Avoid SIGSEGV caused by signed/unsigned mismatch. (hi xtos)

- use pid_t/size_t as appropriate instead of int.
- use %ld to print pids.
- fix a bit of lint.
- WARNS=4

Implement -dl (aka LOUD) to override '@' at the start of script lines.
Based on supplied patch.

PR: 37202

If make can't find a rule to create a file it needs to satisfy a rule
that came from .depend then just output a message and contine.

Make it easier for the build environment (such as a configure script)
to override the default shell:

* Rename _BASENAME_DEFSHELL to DEFSHELL_CUSTOM, because it's no longer
always a basename, it might be a full path;
* Rename DEFSHELL to DEFSHELL_INDEX, because it's actually an index into
an array;
* Provide symbolic names for the possible values if DEFSHELL_INDEX;
* Document how the build environment may set DEFSHELL_INDEX or
DEFSHELL_CUSTOM to choose the default shell;
* Move the fallback definition of DEFSHELL_INDEX from config.h to job.c,
because it's used only in job.c.

Discussed with sjg.

Revert to revision 1.128. Now that the Var* functions no longer try to
write into const strings, it's safe to pass a truly constant string to
Var_Subst.

The 'job_prefix' string can be static but not const 'hi xtos'.

improve on previous

Var_Subst() seems to write into its input string! So we do need a copy.

Remove tmp[] buffer and strncpy() call.

If _BASENAME_DEFSHELL is defined, it might be a full path,
in which case we should not prepend _PATH_DEFSHELLDIR.

Add the ability to .export variables to the environment.

Add the ability to tweak the token output before targets in job mode.
Eg.
.MAKE.JOB.PREFIX=${.newline}---${.MAKE:T}[${.MAKE.PID}]
would produce
---make[1234] target ---

branches: 1.124.2; 1.124.8;
Since 'ClientData' is 'void *', nuke almost all the (ClientData) casts.

Output all debug trace output through 'debug_file' defaulting to 'stdout'.
(Almost all the debug output went there, but some went to stderr.)
Split the parsing of -d (debug flags) out into its own routine.
Allow the output filename to be changed by specifying -dF<file> to create
a log file, or -dF+<file> to append to it. <file> may be stdout or stderr.
Also change so that -d-<flags> acts on <flags> locally but doesn't copy
them to MAKEFLAGS so they aren't inherited by child makes.
I'm not 100% happy with the command line syntax for the above, so they are
currently undocumented.

There is no need to resize the pollfd array, it can only ever have 2+maxJobs
entries, so allocate at startup.
Use an 'int jobPipe[2]' within the job structure, and create pipes directly
into it. Common up the code that creates all the pipes - making them all
non-block on the read side in the process.
Call Job_CatchChildren() directly from Job_CatchOutput() so that it only
gets called when a child actually exits.
NB: Something causes a 'pregnant pause' if (for example) you call 'nbmake obj'
in src/tools. Introduced between netbsd 3 and 4.

Allow make(1) to be built with something other than "sh" as the
shell. To use this, build with -D_PATH_DEFSHELLDIR=\"/path/to/dir\"
-D_BASENAME_DEFSHELL=\"shell\".

* Change the order of entries in shells[]. Now DEFSHELL defaults to 0,
and shells[0] describes the default shell. This will be "sh" in the
usual case.

* If _BASENAME_DEFSHELL is defined, insert an additional entry above
"sh" in the shells[] array, making this new entry shells[0]. The
new entry is assumed to refer to an sh-compatible shell with a
non-standard name. (Tested using _PATH_DEFSHELLDIR="/usr/pkg/bin" and
_BASENAME_DEFSHELL="bash".)

* In the shells[] entry for "sh", test defined(MAKE_NATIVE) &&
defined(__NetBSD__) to decide whether we can use the "q" flag.

OK sjg, christos

Rip out the code for the undocumented -P (don't use pipes for command
output) option. I'm sure it is baggage from the past.

Rip out the code for not 'USE_PGRP', I can't imagine it being useful!
My guess is that it is compat code for before the pgrp stuff was added.

Instead of trying to report all the 'job suspended' and 'job resumed'
messages whan make itself is suspended (ie by ^Z) before make actually
suspends, supress the messages during this sequence.
This means we don't care that they would be output after the suspend
and we can stop attempting to reap child status from withing the signal
handler (which doesn't work for recursive parallel makes).
The code simplification means that we can remove much of the code that
blocked signals - since the signal handlers (expect that for ^C and friends)
now do almost no work.

Reap any dead children that we have at startup.
If there are any undead ones set a flag so we don't report the 'Child (pid)
not in table' message when they die - it is impossible to (portably) find
the childrens pids.
This happens when make is run as 'make -f- ... <<EOF' and the shell uses
a child of (what will be) make to write the data into a pipe.

Complete revamp of the way make handles job control signals.
- Send each type of signal to its own handler.
- Only call JobFinish when a process exits, in particular don't 'fake up'
'exitstatus' for jobs being continued, nor call it for suspends.
- When a job is stopped, use an entire variable to remember the fact, so
we know we need to send a SIGCONT. Don't change any other state.
- In order to report '*** [job3] Suspended' before we suspend ourselves we
have to call waitpid() from the signal handler - where we don't want to
process job termination events. Save the exit status and process later.
The code now handles:
- jobs that suspend themselves
- jobs exiting while suspended
- jobs that don't actually suspend at all
Hoewever it still does printfs() from the signal handler, and I haven't yet
stopped it thrashing the signal mask.

Fix build on 64 bit systems.
Remove now-unecessary temporary local.

Replace the two lists 'job_list' (previously 'jobs') and 'stoppedJobs'
with a table that is malloced with 'maxJobs' entries.
Add a 'job_state' field to the Job type that exactly follows which of
the old lists the job was on (or not).
Change all the code that scanned the lists to scan the array.
No logic changes in this commit.
(Soon we'll no longer need to lock out signals for the changes to job
statuses that are done from signal handlers now that there is no linked list.)

Rename the variable 'jobs' to 'job_list' in preparation for making it an array.

Fixed the bug reported in PR 33866, which is that the :Q operator does not
handle newlines correctly. Ok'ed by christos.

There is no need to count jobs and job tokens.
If we don't create the job pipe, use the '-j n' option to limit the number
of tokens we will remove from the pipe.

nJobs and jobTokensRunning are both counting the same thing.
Kill nJobs and just use the count of tokens in use.
Don't eat job tokens when we are ignoring errors.

JOB_FIRST is always set (since we only execute the shell once for every
target). Nuke it.

Remove the 'never non-NULL' 'previous' argument to JobStart.
Looks like it was intended as a minor (and pointless) optimisation to
remove a free() malloc() pair.
Make he comment about the stoppedJobs list more correct.

It isn't possibly to execute any part of job.c that references compatMake
when it is 'true'. So delete the unexecutable code.

Remove most of the code seemed to be there to support 'remote jobs'.
It isn't clear that it ever worked, if it did it has almost certainly
bitrotted in the last 12 years. I'm not even sure all the required
components were present.
I suspect it was written to attempt to use a 'farm' of diskless sun3s.
In any case the apparant random assignment fo jobs to other systems doesn't
actually seem like a good idea!
Things like 'distcc' han be used to help slow systems run native builds.
Removing this code also simplifies make, and should let me speed up some of
its processing - without worrying about bitrotting it further.

Avoid calling JobRestartJobs() from the SIGCONT signal handler, instead
push a byte through the (now badly named) exit_pipe and call JobRestartJobs()
from the main code path when poll() wakes up.
Part of a plan to remove JobSigLock() and the zillions of system calls
it does.

Actually SIG_DFL the correct signal (not SIGINT every time) just prior to
exec'ing a program in the child.

Make .NOTPARALLEL stop the job engine issuing more than one token (instead
of trying to de-jobify the make.
You can now put .NOTPARALLEL in a submake of a recursive make (where it is
using a job-token pipe from the outer make and have it only run a single job.
You can also specify .NOTPARRALEL in the root makefile of a large recursive
make and have the submakes run multiple commands.
Add some diagnostics printfs (enabled with -dp) to the parser.

Check the 'job_pipe' for the error token 'E' before every job, including
the first one for each make. This significantly speeds up the detection
of errors in other branches of the make (ie those running in a different
make process). The cost of reading and writing a byte from the pipe
should be insignificant.
Defer replacing job tokens until we've decided there is an error.
If we detect an error in another branch of the make, then call Fatal(),
setting 'aborting' and failing to return a token leads to infinite loops.
Now parallel makes actually stop with the failing command on the screen.

When a parallel make job completes, always put the job token back into the
job_pipe and collect another one for the next job.
If we are aborting, remove all the 'normal' job tokens and add an 'error' one.
If we get an 'error' token, remove any other tokens, re-insert the error
token and exit (with error 'cos that is easier).
Add the current pid to some of the DEBUG(JOB) traces.
Combined effect is that parallel makes actually stop some fairly shortly
after an error, rather than running on long enough to fill the scrollback.

F_DUPFD the job_pipe fds onto higher numbers (>15) to make it less likely
that they will get closed in a complex recursive parallel make.

If -s (silent) is specified, don't output the target name before target
output in parallel makes.
After all with -s you wouldn't know the command for a non-parallel make.
Makes (sic) the output of parallel NetBSD build fathomable.

From Max Okumoto:
- Remove casts to NULL.
- Remove space between cast and object.

More KNF cleanups from Max Okumoto

Whitespace KNF cleanup from Max Okumoto

Don't apply .DEFAULT commands to special targets (.BEGIN, .END, .INTERRUPT).

Add new debugging option '-dn' which will leave the scripts fed into make on
-j jobs in /tmp for debugging purposes. Add a note to the man page that
this could cause problems if run a lot (due to the number of files created)

If the line being passed to the shell is a blank line and we're not in compat
mode, use the ignErr template for the command as shell doesn't like an empty
construct of the form { } || <something>. Fixes build breakage on cats
distrib where a command ends up expanding to nothing.

Don't complain that we cannot make .END.

Now that dependencies in .BEGIN, .END, and .INTERRUPT work, allow them.

Mark the interrupt job as special so that when we hit ^C with -j and we
have a .INTERRUPT target we don't print "token botch".

PR/29203, PR/29204: Max Okumoto: KNF changes to make [no functional changes]

PR/22559: Valeriy E. Ushakov: make -j can execute commands in wrong directory
(not in the obj dir).
Fixed by detecting when we cd, and cd'ing back.

Change to use __unused instead and provide a compat definition in make.h if
not already defined from cdefs.h

Add some checks for gcc around a few function declarations and note the
unused variables. Also fix a few other warnings that PR#22118 shows when
trying to compile bmake on non-NetBSD hosts

Remove use of sh -e when running in compat mode.
Its not posix compliant and serves very little purpose.
With this change compat and jobs modes are consistent wrt how
they treat each line of a script.

Add support for the '+' command line prefix as required by posix.
Lines prefixed with '+' are executed even when -n is given.
[Actually posix says they should also be done for -q and -t]

PR:
Reviewed by: jmc

Simplify build, no functional changes.

Instead of adding MAKE_BOOTSTRAP for hosted environments, i.e., when
you want things simple, instead add MAKE_NATIVE to get those hugely
important features like __RCSID().

It's now possible to build make on some hosts with: cc *.c */*.c

branches: 1.83.2;
Fixes for PR#18573 (make -j not stopping correctly on error).
Using -e in sh/ksh to stop on error doesn't work with grouped commands. At
least for any SUSE compliant sh(1). Instead, extend the Shell struct and add
errOut which provides a template to use to check error status from commands.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22365, verified by myself.

Allow .SHELL: to control the shell used by compat mode too.
Add a shell spec for ksh - a nice portable posix shell.
Document .SHELL:

backout. make(1) either uses strcpy() to malloc'ed region (enough size),
or function signature prohibits us from using strlcpy().

strlcpy (fixed)

strlcpy

Pass WARNS=3

PR/1523: Chris Demetriou: don't count a library as built, if there is no way
to build it.

When JobExec() was called from JobRestartJobs() via JobRestart(), it
executed the make job with sigprocmask(SIG_BLOCK) in effect for a
number of signals including SIGCHLD. This caused recursive submakes
of the make process in question not to receive SIGCHLDs when their
jobs exited. This was a second, independent cause of bin/18895 in
addition to the race condition already fixed. Fixed by unblocking all
signals before executing the job.

Call fcntl() on the right file descriptors.

Fixed race condition that would cause make -j to pause for five
seconds if a SIGCHLD arrived while make was not blocked in poll(),
by making the SIGCHLD handler write to a pipe included in the poll.
Avoided the need to implement a duplicate fix for the USE_SELECT case
by emulating poll() in terms of select() when USE_SELECT is defined.
Fixes bin/18895.

Remove !__STDC__ stuff, de-__P(), ANSIfy, and de-register.

PR/16361: J. Mallett: make(1)'s job.c does not properly initialise postCommands

Fix some of the syntax errors (in unused code) and layout glitches.

Block signals during Lst_Open/Lst_Close list traverals and in JobExec
while setting up a child process. This prevents the major sources of
list inconsistencies that may result from doing list manipulations
from a signal handler.

XXX - all Lst_*() operations are at risk; look for ways to avoid touching
so much global state from the signal handlers.

Fix signal-handling bogons in JobPassSig():
- don't block the signal we're about to send to ourselves to
take the default action for
- restore the signal handler for the current signal
- remove duplicate sigprocmask() call

As a result, restoring the SIGTSTP handler in JobContinueSig() is no longer
necessary.

JobInterrupt(): don't try to unlink JOIN or PHONY targets.

Remove bogons from JobMatchShell()

Job_ParseShell(): fix keyword counting error.

JobExec(): don't use Punt() in the child; it can't possibly DTRT, and will also
mess up the parents variables.

Instead, use execError() for all error paths in the child code.

Job_AbortAll(): add missing Lst_Close().

JobInterrupt(): add missing and fix a misplaced Lst_Close().

Add the job's name to the `*** Error/Suspended/Continuing' informational
output. Useful when doing, e.g. a `make -j42'

We don't want token while we're aborting. Otherwise, make spins wasting cpu.

Fix for previous: JobRestart() return value check was reversed.

JobRestartJobs() should cease scheduling stopped jobs if the maximum
number of running jobs limit has been reached, otherwise an infinite
loop is the result.

Give JobStart() a return value to report this condition.

JobStart(): if there no commands to execute for a node, set its status
to MADE before calling Make_Update(). This is necessary to complete
all rules associated with sources on an .ORDER dependency.

JobPrintCommand: avoid stale pointers on a node's command list.
The debugging value of the code that did this is limited, so drop it
altogether.

When attempting to chdir to .CURDIR because Check_Cwd_Cmd says we need to
make sure that the dir actually exists. In compat mode the corresponding
chdir simply fails and all is well - the issue only arises when playing
tricky games with the value of .CURDIR.

Reviewed: christos

precede, not preceed.

Print out errno when vfork fails.

add .USEBEFORE Attribute

Add 4th arg (flags) to Var_Set so that VarLoopExpand can tell it not
to export interator variables when using context VAR_CMD.

Reviewed: christos