buildman: Use oldconfig when adjusting the config

We cannot be sure that the new config is consistent, particularly when
changing a major item like CONFIG_CMDLINE. Use 'make oldconfig' to
check that and avoid any such problems.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Include symbols in the read-only data section

When symbols switch between the inited data section and the read-only
data section their visbility changes, at present, with the -B option.

This is confusing, since adding 'const' to a variable declaration can
make it look like a significant improvement in bloat. But in fact
nothing has changed.

Add 'r' to the list of symbols types that are recorded, to correct this
problem. Add a constant to make it easier to find this code next time.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reported-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Start the clock when the build starts

The Kconfig and maintainer processing can take a while, sometimes 5
seconds or more. This skews the timing printed by buildmand when the build
completes. Start the clock when the threads start to avoid this problem.

Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Tom Rini <trini@konsulko.com>

buildman: Export _get_output_dir() to avoid warnings

Make this a public memory since it is used outside the class.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in builderthread.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in builder.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test for --boards

Add a simple functional test for the --boards option. Fix the example in
the docs while we are here. Also improve the docs for Builder.count so it
is clearer what it contains.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move remaining builder properties to constructor

Do these all in the constructor, so it is consistent.

Move the stray builder comment into the correct place.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move library functions into a library directory

The patman directory has a number of modules which are used by other tools
in U-Boot. This makes it hard to package the tools using pypi since the
common files must be copied along with the tool that uses them.

To address this, move these files into a new u_boot_pylib library. This
can be packaged separately and listed as a dependency of each tool.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a flag for reproducible builds

This is quite a useful thing to use when building since it avoids small
size changes between commits. Add a -r flag for it.

Also undefine CONFIG_LOCALVERSION_AUTO since this appends the git hash
to the version string, causing every build to be slightly different.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support disabling LTO

This cuts down build performance considerably and is not always needed,
when checking for build errors, etc.

Add a flag to disable it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add --allow-missing flag to allow missing blobs

Add a new flag to buildman so that we will in turn pass
BINMAN_ALLOW_MISSING=1 to 'make'. Make use of this flag in CI.

Allow the settings file to control this.

Cc: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow lines without a symbol

The 'nm' tool can produce lines without a symbol, for example:

00000004 t

Silently skip these and anything else without three fields. Drop the
warning since there is nothing the user can do about it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reported-by: Tom Rini <trini@konsulko.com>

buildman: Drop use of 'boards' in control

Use brds instead so that we can reserve 'boards' for a module name.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support running from an IDE

Add a flag to allow buildman to behave properly for use from an IDE. This
shows error/warning output on stderr and drops all summary and progress
information.

This should normally only be used when building a single board.

Fix up a confusing comment for GetResultSummary() while we are here, since
we want to use the Outcome object to access the unprocessed error lines
from the build.

Signed-off-by: Simon Glass <sjg@chromium.org>

nds32: Remove the architecture

As removal of nds32 has been ack'd for the Linux kernel, remove support
here as well.

Cc: Rick Chen <rick@andestech.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Rick Chen <rick@andestech.com>

buildman: Correct pylint errors

Fix pylint errors that can be fixed and mask those that seem to be
incorrect.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in terminal.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Color() method to build()

This method has the same name as its class which is confusing. It is also
annoying when searching the code.

It builds a string with a colour, so rename it to build().

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Print() to Tprint()

Rename this function so that when we convert it to snake case it will not
conflict with the built-in print() function.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in gitutil.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in command.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow adjusting board config on the fly

Add a -a option to specify changes to the config before the build
commences. For example

buildman -a ~CONFIG_CMDLINE

disables CONFIG_CMDLINE before doing the build.

This makes it easier to try things out as well as to write tests without
creating a new board or manually manging the .config file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Detect Kconfig loops

Hex and int Kconfig options are supposed to have defaults. This is so we
can configure U-Boot without having to enter particular values for the
items that don't have specific values in the board's defconfig file.

If this rule is not followed, then introducing a new Kconfig can produce
a loop like this:

Break things (BREAK_ME) [] (NEW)
Error in reading or end of file.

Break things (BREAK_ME) [] (NEW)
Error in reading or end of file.

The continues forever since buildman passes /dev/null to 'conf', and
the build system just tries again. Eventually there is so much output that
buildman runs out of memory.

We can detect this situation by looking for a symbol (like 'BREAK_ME')
which has no default (the '[]' above) and is marked as new. If this
appears multiple times in the output, we know something is wrong.

Add a filter function for the output which detects this situation. Allow
it to return True to terminate the process. Implement this termination in
cros_subprocess.

With this we get a nice message:

buildman --board sandbox -T0
Building current source for 1 boards (0 threads, 32 jobs per thread)
sandbox: w+ sandbox
+.config:66:warning: symbol value '' invalid for BREAK_ME
+
+Error in reading or end of file.
+make[3]: *** [scripts/kconfig/Makefile:75: syncconfig] Terminated
+make[2]: *** [Makefile:569: syncconfig] Terminated
+make: *** [Makefile:177: sub-make] Terminated
+(** did you define an int/hex Kconfig with no default? **)

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Handle exceptions in threads gracefully

There have been at least a few cases where an exception has occurred in a
thread and resulted in buildman hanging: running out of disk space and
getting a unicode error.

Handle these by collecting a list of exceptions, printing them out and
reporting failure if any are found. Add a test for this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support single-threaded operation

At present even if only a single thread is in use, buildman still uses
threading.

For some debugging it is helpful to do everything in the main process.
Allow -T0 to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: 'Thread' object has no attribute 'isAlive'

The isAlive() method was deprecated in Python 3.8 and has been removed in
Python 3.9. See https://bugs.python.org/issue37804. Use is_alive() instead.

Since Python 2.6 is_alive() has been a synonym for isAlive(). So there
should be no problems for users using elder Python 3 versions.

Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Use git worktrees instead of git clones when possible

This patch makes buildman create linked working trees instead of clones
of the source repository, but keeps updating the older clones of the
repository that might already exist. These worktrees share "everything
except working directory specific files such as HEAD, index, etc." with
the source repository. See the git-worktree(1) manual page for more
information.

If git-worktree isn't available, silently falls back to cloning the
repository.

Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

buildman: Show the build rate at the end

It is interesting to note the number of builds completed per second to
track machine performance and build speed. Add a 'rate' value at the end
of the build to show this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Remove _of_#_ from results directory paths

Currently, the following scenario will rebuild the first commit even
though it is not really necessary - the commit sha or the position in the
patchset did not change:

$ git am <local-patch-0001>
$ tools/buildman/buildman -P -E -W -b master mx6
<do some more development work>
$ git am <local-patch-0002>
$ tools/buildman/buildman -P -E -W -b master mx6 <- will rebuild the first
commit as well, even
though nothing has
changed about it.

This is due to the fact that previous results directories get removed
when the number of commits change. By removing the _of_#_ part of the
directory path, the commits will be rebuilt only if the commit sha or the
position in the patchset changes. Also, update the testcase to reflect this
change.

Signed-off-by: Ovidiu Panait <ovidiu.panait@windriver.com>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Write output files when using -w

At present buildman does not write its own output files (err, done, the
environment) when using -w. However this is useful for when the build is
run with -s to check it.

In fact ProduceResultSummary() reads the result from those files rather
than using the 'result' info directly. So ProcessResult() does not work
with -w at present. It does not print any output.

Fix this by writing output files even when -w is used.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore migration warnings

These are becoming more common now. They cause boards to show warnings
which can be mistaking for compiler warnings.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore device-tree warnings

Unfortunately the plague of device-tree warnings has not lifted. These
warnings infiltrate almost every build, adding noise and confusion.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make -I the default

At present buildman defaults to running 'mrproper' on every thread before
it starts building commits for each board. This can add a delay of about 5
seconds to the start of the process, since the tools and other invariants
must be rebuilt.

In particular, a build without '-b', to build current source, runs much
slower without -I, since any existing build is removed, thus losing the
possibility of an incremental build.

Partly this behaviour was to avoid strange build-system problems caused by
running 'make defconfig' for one board and then one with a different
architecture. But these problems were fixed quite a while ago.

The -I option (which disabled mrproper) was introduced four years ago and
does not seem to cause any problems with builds.

So make -I the default and deprecate the option. To allow use of
'mrproper', add a new -m flag.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show a summary of the build result

When buildman finishes it leaves the last summary line visible, which
shows the number of successful builds, builds with warnings and builds
with errors.

It is useful also to see how many builds were done in total along with
the time taken. Show these on a separate line before buildman finishes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Limit the length of progress messages

If a progress message is longer than the terminal line it will scroll the
terminal. Limit the messages to the terminal width.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show the number of builds remaining

It is nice to see the actual number of builds remaining to complete. Add
this in the progress message, using a different colour.

Drop the unnecessary 'name' variable while we are here.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Drop unused output code

The commit counter is a hangover from when buildman processed each board
for a commit. Now buildman processes each commit for a board, so this
output is never triggered.

Delete it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show a message when fetching a repo

Fetching updated versions of a repo can take time. At present buildman
gives no indication that it is doing this.

Add a message to explain the delay.

Tidy up a few other messages while we are here.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Drop the line-clearing code in Builder

The new feature in terminal can be used by buildman. Update the Builder
class accordingly.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use spaces in the board list

At present the board names shown with -l are separated by commas. This
makes it hard to double-click to select a particular board. Also it is not
possible to select all boards and paste them as arguments to a subsequent
buildman run, since buildman requires spaces to separate the list on the
command line, not commas.

Change the output format to use spaces instead of commas.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show the list of boards in magenta

It is quite hard to see the list of board for each error line since the
colour is the same as the actual error line. Show the board list in
magenta so that it is easier to distinguish them.

There is no point in checking the colour of the overall line, since there
are now multiple colours. So drop those tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use an object to hold error lines

At present the string for each error line is created in _CalcErrorDelta()
and used to create the summary output. This is inflexible since all the
information (error/warning character, error line, list of boards with that
error line) is munged together in a string.

Create an object to hold this information and only convert it to a string
when printing the actual output.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use yellow consistently for warning lines

At present warnings are shown in yellow in the summary (-s) but magenta in
the detail listing (-e). Use yellow in both.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Refactor error-line output int a function

Reduce the amount of repeated code by creating an _OutputErrLines()
function to hold this code.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow building within a subdir of the current dir

This is useful in some situations, in particular with -w and when building
in-tree. Now that we are more careful about what we remove in
_PrepareOutputSpace(), it should be safe to relax this restriction.

Update the progress information also so it is clear what buildman is
doing. Remove files can take a long time.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Be more selective about which directories to remove

At present buildman removes any directory it doesn't intend to write
output into. This is overly expansive since if the output directory
happens to be somewhere with existing files, they may be removed. Using
an existing directory for buildman is not a good practice, but since the
result might be catastrophic, it is best to guard against it.

A previous commit[1] fixed this by refusing to write to a subdirectory
of the current directory, assumed to have U-Boot source code. But we can
do better by only removing directories that look like the ones buildman
creates.

Update the code to do this and add a test.

Signed-off-by: Simon Glass <sjg@chromium.org>

[1] 409fc029c40 tools: buildman: Don't use the working dir as build dir

buildman: Update help for -d

This help is a bit ambiguous. It only does anything if asked to show size
changes with -S. Update the help and the function comments.

Signed-off-by: Simon Glass <sjg@chromium.org>

bulidman: Add support for a simple build

It is useful to run a simple build and put all the output in a single
directory. Add a -w option to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: buildman: fix typo

Signed-off-by: Flavio Suligoi <f.suligoi@asem.it>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Ignore blank lines during size checks

Today when parsing the .sizes files we get a warning about an invalid
line in the file as it's blank. Solve this by checking that we have a
non-blank line prior to processing.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Deal more nicely with invalid build-status file

The 'done' files created by buildman may end up being empty if buildman
runs out of disk space while writing them. At present buildman dies with
an exception when using -s to check the build status. Fix this.

Seriesl-cc: trini

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show boards with warning with w+

At present we should boards with warnings in the same way as those with
errors. This is not ideal. Add a new 'warn' state and show these listed
in yellow to match the actual warning lines printing with -e.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Rename the good, better, worse variables

At present we don't distinguish between errors and warnings when printing
the architecture summary. Rename the variables to better describe their
purpose.

'Worse' at present means we got an error, so use that as the name.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Detect dtc warnings

At present messages from the device-tree compiler like this:

arch/arm/dts/socfpga_arria10_socdk_sdmmc.dtb: Warning
(avoid_unnecessary_addr_size): /clocks: unnecessary
#address-cells/#size-cells without "ranges" or child "reg" property

are detected as errors since they don't match the gcc warning regex. Add a
new one for dtc to fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid hanging when the config changes

Something has changed in the last several month such that when buildman
builds U-Boot incrementally and a new CONFIG option has been added to the
Kconfig, the build hanges waiting for input:

Test new config (NEW_CONFIG) [N/y/?] (NEW)

Since binamn does not connect the build's stdin to anything this waits on
stdin to the build thread, which never comes. Eventually I suspect all the
threads end up in this state and the build does not progress.

Fix this by passing /dev/null as input to the build. That way, if there is
a new CONFIG, the build will stop (and fail):

Test new config (NEW_CONFIG) [N/y/?] (NEW)

Error in reading or end of file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add support for environment delta in summary

When summarising the builds, add the -U option to emit delta lines for
the default environment built into U-Boot at each commit.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: add option -E for treating compiler warnings as errors

Add a new option '-E' for treating all compiler warnings as errors.
Eventually this will pass 'KCFLAGS=-Werror' to Kbuild.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>

buildman: Fix bloat option when 'new' only drops functions

In the case where a new build only decreases sizes and does not increase
any size we still want to report what functions have been dropped when
doing a bloat comparison.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Handle commit subjects containing unicode

One of these has crept in in this commit:

40a808f1 ARCv2: SLC: Make sure busy bit is set properly on SLC flushing

Adjust buildman to handle it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Clean up odd characters on the terminal

At present buildman leaves behind a few characters during its progress
updates, which looks odd. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Squash useless output from -K

When using #define CONFIG_SOME_OPTION, the value it set to '1'. When using
defconfig (i.e. CONFIG_SOME_OPTION=y) the value is set to 'y'. This results
in differences showing up with -K. These differences are seldom useful.

Adjust buildman to suppress these differences by default.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to just create the config

Normally buildman does a full build of a board. This includes creating the
u-boot.cfg file which contains all the configuration options. Buildman uses
this file with the -K option, to show differences in effective configuration
for each commit.

Doing a full build of U-Boot just to create the u-boot.cfg file is wasteful.
Add a -D option which causes buildman to only create the configuration. This
is enough to support use of -K and can be done much more quickly (typically
5-10 times faster).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Don't show a stacktrace on Ctrl-C

When Ctrl-C is pressed, just exited quietly. There is no sense in displaying
a stack trace since buildman will always be in the same place: waiting for
threads to complete building all the jobs on the queue.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Drop the 'active' flag in the builder

This serves no real purpose, since when we are not active, we exit. Drop it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow builds to terminate cleanly

It is annoying that buildman does not respond cleanly to Ctrl-C or SIGINT,
particularly on machines with lots of CPUS. Unfortunately queue.join()
blocks the main thread and does not allow it to see the signal. Use a
separate thread instead,

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Print a message indicating the build is starting

Make it clear when buildman actually starts building. This happens when it
has prepared the threads, working directory and output directories.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Print a message when removing old directories

When buildman starts, it prepares its output directory by removing any old
build directories which will not be used this time. This can happen if a
previous build left directories around for commit hashes which are no-longer
part of the branch.

This can take quite a while, so print a message to indicate what is going
on.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Tidy up the 'cloning' message

On a machine with a lot of CPUs this prints a lot of useless lines of the
form:

Cloning repo for thread <n>

Adjust the output so that these all appear on one line, and disappear when
the cloning is complete.

Note: This cloning is actually unnecessary and very wasteful on disk space
(about 3.5GB each time). It would be better to create symlinks.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: allow more incremental building

One use-case for buildman is to continually run it interactively after
each small step in a large refactoring operation. This gives more
immediate feedback than making a number of commits and then going back and
testing them. For this to work well, buildman needs to be extremely fast.
At present, a couple issues prevent it being as fast as it could be:

1) Each time buildman runs "make %_defconfig", it runs "make mrproper"
first. This throws away all previous build results, requiring a
from-scratch build. Optionally avoiding this would speed up the build, at
the cost of potentially causing or missing some build issues.

2) A build tree is created per thread rather than per board. When a thread
switches between building different boards, this often causes many files
to be rebuilt due to changing config options. Using a separate build tree
for each board would avoid this. This does put more strain on the system's
disk cache, but it is worth it on my system at least.

This commit adds two command-line options to implement the changes
described above; -I ("--incremental") turns of "make mrproper" and -P
("--per-board-out-dir") creats a build directory per board rather than per
thread.

Tested:

./tools/buildman/buildman.py tegra
./tools/buildman/buildman.py -I -P tegra
./tools/buildman/buildman.py -b tegra_dev tegra
./tools/buildman/buildman.py -b tegra_dev -I -P tegra

... each once after deleting the buildman result/work directory, and once
"incrementally" after a previous identical invocation.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Tom Rini <trini@konsulko.com>
Acked-by: Simon Glass <sjg@chromium.org> # v1
Tested-by: Simon Glass <sjg@chromium.org> # v1
Acked-by: Simon Glass <sjg@chromium.org>

buildman: Improve the config comparison feature

At present buildman can compare configurations between commits but the
feature is less useful than it could be. There is no summary by architecture
and changes are not reported on a per-board basis.

Correct these deficiencies so that it is possible to see exactly what is
changing for any number of boards.

Note that 'buildman -b <branch> -C' is recommended for any build where you
will be comparing configuration. Without -C the correct configuration will
not be reported since changes will often not be picked up.

Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow comparison of build configuration

It is useful to be able to see CONFIG changes made by commits. Add this
feature to buildman using the -K flag so that all CONFIG changes are
reported.

The CONFIG options exist in a number of files. Each is reported
individually as well as a summary that covers all files. The output
shows three parts: green for additions, red for removals and yellow for
changes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show 'make' command line when -V is used

When a verbose build it selected, show the make command before the output of
that command.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a space before the list of boards

Tweak the output slightly so we don't get things like:

- board1 board2+ board3 board4

There should be a space before the '+'.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to write the full build output

Normally buildman runs with 'make -s' meaning that only errors and warnings
appear in the log file. Add a -V option to run make in verbose mode, and
with V=1, causing a full build log to be created.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to use the full tool chain path

In some cases there may be multiple toolchains with the same name in the
path. Provide an option to use the full path in the CROSS_COMPILE
environment variable.

Note: Wolfgang mentioned that this is dangerous since in some cases there
may be other tools on the path that are needed. So this is set up as an
option, not the default. I will need test confirmation (i.e. that this
commit fixes a real problem) before merging it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Steve Rae <srae@broadcom.com>

buildman: Add an option to flatten output directory trees

When building current source for a single board, buildman puts the output
in <output_dir>/current/current/<board>. Add an option to make it use
<output_dir>/<board> instead. This removes the unnecessary directories
in that case, controlled by the --no-subdirs/-N option.

Suggested-by: Tom Rini <trini@ti.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Don't prune output space for 'current source' build

This is not needed since we always do a full (non-incremental) build. Also
it might be dangerous since it will try to delete everything below the
base directory.

Fix this potentially nasty bug.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix repeating board list with -l

Ensure that we don't print duplicate board names when -l is used.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reported-by: Albert Aribaud <albert.u.boot@aribaud.net>

buildman: Create parent directories as necessary

When creating build directories also create parents as necessary. This
fixes a failure when building a hierarchical branch (i.e. foo/bar).

Signed-off-by: Thierry Reding <treding@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Tom Rini <trini@ti.com>

buildman: Send builder output through a function for testing

To allow us to verify the builder's console output, send it through a
function which can collect it when running in test mode.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Separate out display of warnings and errors

Some boards unfortunately build with warnings and it is useful to be able
to easily distinguish the warnings from the errors.

Use a simple pattern match to categorise gcc output into warnings and
errors, and display each separately. New warnings are shown in magenta (with
a w+ prefix) and fixed warnings are shown in yellow with a w- prefix.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to show which boards caused which errors

Add a -l option to display a list of offending boards against each
error/warning line. The information will be shown in brackets as below:

02: wip
sandbox: + sandbox
arm: + seaboard
+(sandbox) arch/sandbox/cpu/cpu.c: In function 'timer_get_us':
+(sandbox) arch/sandbox/cpu/cpu.c:40:9: warning: unused variable 'i' [-Wunused-variable]
+(seaboard) board/nvidia/seaboard/seaboard.c: In function 'pin_mux_mmc':
+(seaboard) board/nvidia/seaboard/seaboard.c:36:9: warning: unused variable 'fred' [-Wunused-variable]
+(seaboard) int fred;
+(seaboard) ^

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Set the return code to indicate build result

When buildman finds errors/warnings when building, set the return code to
indicate this.

Suggested-by: York Sun <yorksun@freescale.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

Makefile: remove generated boards.cfg within make distclean

Signed-off-by: Roger Meier <roger@bufferoverflow.ch>
Reviewed-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Tested-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Cc: Tom Rini <trini@ti.com>
Cc: Wolfgang Denk <wd@denx.de>
Cc: Simon Glass <sjg@chromium.org>

buildman: Add a message indicating there are no errors

If buildman finds no problems it prints nothing. This can be a bit confusing,
so add a message that all is well.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Remove unused non-incremental build method code

The non-incremental build method is no longer used, so remove it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add verbose option to display errors as they happen

Normally buildman operates in two passes - one to do the build and another
to summarise the errors. Add a verbose option (-v) to display build problems
as they happen. With -e also given, this will display errors too.

When building the current source tree (rather than a list of commits in a
branch), both -v and -e are enabled automatically.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Refactor output options

We need the output options to be available in several places. It's a pain
to pass them into each function. Make them properties of the builder and
add a single function to set them up. At the same time, add a function which
produces summary output using these options.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move BuilderThread code to its own file

The builder.py file is getting too long, so split out some code.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow building of current source tree

Originally buildman had some support for building the current source tree.
However this was dropped before it was submitted, as part of the effort to
make it faster when building entire branches.

Reinstate this support. If no -b option is given, buildman will build the
current source tree.

Reported-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: adjust for Kconfig

Use "make <board>_defconfig" instead of "make <board>_config".

Invoke tools/genboardscfg.py to generate boards.cfg when it is missing.

Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: make sure to invoke GNU Make

Since the command name 'make' may not be GNU Make on some platforms
such as FreeBSD, buildman should call scripts/show-gnu-make to get
the command name for GNU MAKE (and error out if it is not found).

Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Jeroen Hofstee <jeroen@myspectrum.nl>

buildman: Support in-tree builds

At present buildman always builds out-of-tree, that is it uses a separate
output directory from the source directory. Normally this is what you want,
but it is important that in-tree builds work also. Some Makefile changes may
break this.

Add a -i option to tell buildman to use in-tree builds, so that it is easy
to test this feature.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add -C option to force a reconfigure for each commit

Normally buildman wil try to configure U-Boot for a particular board on the
first commit that it builds in a series. Subsequent commits are built
without reconfiguring which normally works. Where it doesn't, buildman
automatically reconfigures and retries.

To fully emulate the way MAKEALL works, we should have an option to disable
this optimisation.

Add a -C option to cause buildman to always reconfigure on each commit.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid retrying a build if it definitely failed

After a build fails buildman will reconfigure and try again, if it did not
reconfigure before the build. However it doesn't actually keep track of
whether it did reconfigure on the previous attempt.

Fix that logic to avoid a pointless rebuild. This speeds things up quite a
bit for failing builds. Previously they would always be built twice.

Change-Id: Ib37f21320baa7c60bed98f4042c0b7ed7c0dc85e
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add -F flag to retry failed builds

Generally a build failure with a particular commit cannot be fixed except
by changing that commit. Changing the commit will automatically cause
buildman to retry when you run it again: buildman sees that the commit
hash is different and that it has no previous build result for the new
commit hash.

However sometimes the build failure is due to a toolchain issue or some
other environment problem. In that case, retrying failed builds may yield
a different result.

Add a flag to retry failed builds. This differs from the force rebuild
flag (-f) in that it will not rebuild commits which are already marked as
succeeded.

Series-to: u-boot

Change-Id: Iac4306df499d65ff0888b1c60f06fc162a6faad8

buildman: Allow make flags to be specified for each board

There are a few make options such as BUILD_TAG which can be provided when
building U-Boot. Provide a way for buildman to pass these flags to make
also.

The flags should be in a [make-flags] section and arranged by target name
(the 'target' column in boards.cfg. See the README for more details.

Signed-off-by: Simon Glass <sjg@chromium.org>

Add GPL-2.0+ SPDX-License-Identifier to source files

Signed-off-by: Wolfgang Denk <wd@denx.de>
[trini: Fixup common/cmd_io.c]
Signed-off-by: Tom Rini <trini@ti.com>

buildman - U-Boot multi-threaded builder and summary tool

This tool handles building U-Boot to check that you have not broken it
with your patch series. It can build each individual commit and report
which boards fail on which commits, and which errors come up. It also
shows differences in image sizes due to particular commits.

Buildman aims to make full use of multi-processor machines.

Documentation and caveats are in tools/buildman/README.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Include symbols in the read-only data section

When symbols switch between the inited data section and the read-only
data section their visbility changes, at present, with the -B option.

This is confusing, since adding 'const' to a variable declaration can
make it look like a significant improvement in bloat. But in fact
nothing has changed.

Add 'r' to the list of symbols types that are recorded, to correct this
problem. Add a constant to make it easier to find this code next time.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reported-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Start the clock when the build starts

The Kconfig and maintainer processing can take a while, sometimes 5
seconds or more. This skews the timing printed by buildmand when the build
completes. Start the clock when the threads start to avoid this problem.

Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Tom Rini <trini@konsulko.com>

buildman: Export _get_output_dir() to avoid warnings

Make this a public memory since it is used outside the class.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in builderthread.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in builder.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test for --boards

Add a simple functional test for the --boards option. Fix the example in
the docs while we are here. Also improve the docs for Builder.count so it
is clearer what it contains.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move remaining builder properties to constructor

Do these all in the constructor, so it is consistent.

Move the stray builder comment into the correct place.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move library functions into a library directory

The patman directory has a number of modules which are used by other tools
in U-Boot. This makes it hard to package the tools using pypi since the
common files must be copied along with the tool that uses them.

To address this, move these files into a new u_boot_pylib library. This
can be packaged separately and listed as a dependency of each tool.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a flag for reproducible builds

This is quite a useful thing to use when building since it avoids small
size changes between commits. Add a -r flag for it.

Also undefine CONFIG_LOCALVERSION_AUTO since this appends the git hash
to the version string, causing every build to be slightly different.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support disabling LTO

This cuts down build performance considerably and is not always needed,
when checking for build errors, etc.

Add a flag to disable it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add --allow-missing flag to allow missing blobs

Add a new flag to buildman so that we will in turn pass
BINMAN_ALLOW_MISSING=1 to 'make'. Make use of this flag in CI.

Allow the settings file to control this.

Cc: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow lines without a symbol

The 'nm' tool can produce lines without a symbol, for example:

00000004 t

Silently skip these and anything else without three fields. Drop the
warning since there is nothing the user can do about it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reported-by: Tom Rini <trini@konsulko.com>

buildman: Drop use of 'boards' in control

Use brds instead so that we can reserve 'boards' for a module name.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support running from an IDE

Add a flag to allow buildman to behave properly for use from an IDE. This
shows error/warning output on stderr and drops all summary and progress
information.

This should normally only be used when building a single board.

Fix up a confusing comment for GetResultSummary() while we are here, since
we want to use the Outcome object to access the unprocessed error lines
from the build.

Signed-off-by: Simon Glass <sjg@chromium.org>

nds32: Remove the architecture

As removal of nds32 has been ack'd for the Linux kernel, remove support
here as well.

Cc: Rick Chen <rick@andestech.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Rick Chen <rick@andestech.com>

buildman: Correct pylint errors

Fix pylint errors that can be fixed and mask those that seem to be
incorrect.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in terminal.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Color() method to build()

This method has the same name as its class which is confusing. It is also
annoying when searching the code.

It builds a string with a colour, so rename it to build().

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Print() to Tprint()

Rename this function so that when we convert it to snake case it will not
conflict with the built-in print() function.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in gitutil.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in command.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow adjusting board config on the fly

Add a -a option to specify changes to the config before the build
commences. For example

buildman -a ~CONFIG_CMDLINE

disables CONFIG_CMDLINE before doing the build.

This makes it easier to try things out as well as to write tests without
creating a new board or manually manging the .config file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Detect Kconfig loops

Hex and int Kconfig options are supposed to have defaults. This is so we
can configure U-Boot without having to enter particular values for the
items that don't have specific values in the board's defconfig file.

If this rule is not followed, then introducing a new Kconfig can produce
a loop like this:

Break things (BREAK_ME) [] (NEW)
Error in reading or end of file.

Break things (BREAK_ME) [] (NEW)
Error in reading or end of file.

The continues forever since buildman passes /dev/null to 'conf', and
the build system just tries again. Eventually there is so much output that
buildman runs out of memory.

We can detect this situation by looking for a symbol (like 'BREAK_ME')
which has no default (the '[]' above) and is marked as new. If this
appears multiple times in the output, we know something is wrong.

Add a filter function for the output which detects this situation. Allow
it to return True to terminate the process. Implement this termination in
cros_subprocess.

With this we get a nice message:

buildman --board sandbox -T0
Building current source for 1 boards (0 threads, 32 jobs per thread)
sandbox: w+ sandbox
+.config:66:warning: symbol value '' invalid for BREAK_ME
+
+Error in reading or end of file.
+make[3]: *** [scripts/kconfig/Makefile:75: syncconfig] Terminated
+make[2]: *** [Makefile:569: syncconfig] Terminated
+make: *** [Makefile:177: sub-make] Terminated
+(** did you define an int/hex Kconfig with no default? **)

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Handle exceptions in threads gracefully

There have been at least a few cases where an exception has occurred in a
thread and resulted in buildman hanging: running out of disk space and
getting a unicode error.

Handle these by collecting a list of exceptions, printing them out and
reporting failure if any are found. Add a test for this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support single-threaded operation

At present even if only a single thread is in use, buildman still uses
threading.

For some debugging it is helpful to do everything in the main process.
Allow -T0 to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: 'Thread' object has no attribute 'isAlive'

The isAlive() method was deprecated in Python 3.8 and has been removed in
Python 3.9. See https://bugs.python.org/issue37804. Use is_alive() instead.

Since Python 2.6 is_alive() has been a synonym for isAlive(). So there
should be no problems for users using elder Python 3 versions.

Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Use git worktrees instead of git clones when possible

This patch makes buildman create linked working trees instead of clones
of the source repository, but keeps updating the older clones of the
repository that might already exist. These worktrees share "everything
except working directory specific files such as HEAD, index, etc." with
the source repository. See the git-worktree(1) manual page for more
information.

If git-worktree isn't available, silently falls back to cloning the
repository.

Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

buildman: Show the build rate at the end

It is interesting to note the number of builds completed per second to
track machine performance and build speed. Add a 'rate' value at the end
of the build to show this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Remove _of_#_ from results directory paths

Currently, the following scenario will rebuild the first commit even
though it is not really necessary - the commit sha or the position in the
patchset did not change:

$ git am <local-patch-0001>
$ tools/buildman/buildman -P -E -W -b master mx6
<do some more development work>
$ git am <local-patch-0002>
$ tools/buildman/buildman -P -E -W -b master mx6 <- will rebuild the first
commit as well, even
though nothing has
changed about it.

This is due to the fact that previous results directories get removed
when the number of commits change. By removing the _of_#_ part of the
directory path, the commits will be rebuilt only if the commit sha or the
position in the patchset changes. Also, update the testcase to reflect this
change.

Signed-off-by: Ovidiu Panait <ovidiu.panait@windriver.com>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Write output files when using -w

At present buildman does not write its own output files (err, done, the
environment) when using -w. However this is useful for when the build is
run with -s to check it.

In fact ProduceResultSummary() reads the result from those files rather
than using the 'result' info directly. So ProcessResult() does not work
with -w at present. It does not print any output.

Fix this by writing output files even when -w is used.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore migration warnings

These are becoming more common now. They cause boards to show warnings
which can be mistaking for compiler warnings.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore device-tree warnings

Unfortunately the plague of device-tree warnings has not lifted. These
warnings infiltrate almost every build, adding noise and confusion.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make -I the default

At present buildman defaults to running 'mrproper' on every thread before
it starts building commits for each board. This can add a delay of about 5
seconds to the start of the process, since the tools and other invariants
must be rebuilt.

In particular, a build without '-b', to build current source, runs much
slower without -I, since any existing build is removed, thus losing the
possibility of an incremental build.

Partly this behaviour was to avoid strange build-system problems caused by
running 'make defconfig' for one board and then one with a different
architecture. But these problems were fixed quite a while ago.

The -I option (which disabled mrproper) was introduced four years ago and
does not seem to cause any problems with builds.

So make -I the default and deprecate the option. To allow use of
'mrproper', add a new -m flag.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show a summary of the build result

When buildman finishes it leaves the last summary line visible, which
shows the number of successful builds, builds with warnings and builds
with errors.

It is useful also to see how many builds were done in total along with
the time taken. Show these on a separate line before buildman finishes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Limit the length of progress messages

If a progress message is longer than the terminal line it will scroll the
terminal. Limit the messages to the terminal width.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show the number of builds remaining

It is nice to see the actual number of builds remaining to complete. Add
this in the progress message, using a different colour.

Drop the unnecessary 'name' variable while we are here.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Drop unused output code

The commit counter is a hangover from when buildman processed each board
for a commit. Now buildman processes each commit for a board, so this
output is never triggered.

Delete it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show a message when fetching a repo

Fetching updated versions of a repo can take time. At present buildman
gives no indication that it is doing this.

Add a message to explain the delay.

Tidy up a few other messages while we are here.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Drop the line-clearing code in Builder

The new feature in terminal can be used by buildman. Update the Builder
class accordingly.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use spaces in the board list

At present the board names shown with -l are separated by commas. This
makes it hard to double-click to select a particular board. Also it is not
possible to select all boards and paste them as arguments to a subsequent
buildman run, since buildman requires spaces to separate the list on the
command line, not commas.

Change the output format to use spaces instead of commas.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show the list of boards in magenta

It is quite hard to see the list of board for each error line since the
colour is the same as the actual error line. Show the board list in
magenta so that it is easier to distinguish them.

There is no point in checking the colour of the overall line, since there
are now multiple colours. So drop those tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use an object to hold error lines

At present the string for each error line is created in _CalcErrorDelta()
and used to create the summary output. This is inflexible since all the
information (error/warning character, error line, list of boards with that
error line) is munged together in a string.

Create an object to hold this information and only convert it to a string
when printing the actual output.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use yellow consistently for warning lines

At present warnings are shown in yellow in the summary (-s) but magenta in
the detail listing (-e). Use yellow in both.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Refactor error-line output int a function

Reduce the amount of repeated code by creating an _OutputErrLines()
function to hold this code.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow building within a subdir of the current dir

This is useful in some situations, in particular with -w and when building
in-tree. Now that we are more careful about what we remove in
_PrepareOutputSpace(), it should be safe to relax this restriction.

Update the progress information also so it is clear what buildman is
doing. Remove files can take a long time.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Be more selective about which directories to remove

At present buildman removes any directory it doesn't intend to write
output into. This is overly expansive since if the output directory
happens to be somewhere with existing files, they may be removed. Using
an existing directory for buildman is not a good practice, but since the
result might be catastrophic, it is best to guard against it.

A previous commit[1] fixed this by refusing to write to a subdirectory
of the current directory, assumed to have U-Boot source code. But we can
do better by only removing directories that look like the ones buildman
creates.

Update the code to do this and add a test.

Signed-off-by: Simon Glass <sjg@chromium.org>

[1] 409fc029c40 tools: buildman: Don't use the working dir as build dir

buildman: Update help for -d

This help is a bit ambiguous. It only does anything if asked to show size
changes with -S. Update the help and the function comments.

Signed-off-by: Simon Glass <sjg@chromium.org>

bulidman: Add support for a simple build

It is useful to run a simple build and put all the output in a single
directory. Add a -w option to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: buildman: fix typo

Signed-off-by: Flavio Suligoi <f.suligoi@asem.it>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Ignore blank lines during size checks

Today when parsing the .sizes files we get a warning about an invalid
line in the file as it's blank. Solve this by checking that we have a
non-blank line prior to processing.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Deal more nicely with invalid build-status file

The 'done' files created by buildman may end up being empty if buildman
runs out of disk space while writing them. At present buildman dies with
an exception when using -s to check the build status. Fix this.

Seriesl-cc: trini

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show boards with warning with w+

At present we should boards with warnings in the same way as those with
errors. This is not ideal. Add a new 'warn' state and show these listed
in yellow to match the actual warning lines printing with -e.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Rename the good, better, worse variables

At present we don't distinguish between errors and warnings when printing
the architecture summary. Rename the variables to better describe their
purpose.

'Worse' at present means we got an error, so use that as the name.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Detect dtc warnings

At present messages from the device-tree compiler like this:

arch/arm/dts/socfpga_arria10_socdk_sdmmc.dtb: Warning
(avoid_unnecessary_addr_size): /clocks: unnecessary
#address-cells/#size-cells without "ranges" or child "reg" property

are detected as errors since they don't match the gcc warning regex. Add a
new one for dtc to fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid hanging when the config changes

Something has changed in the last several month such that when buildman
builds U-Boot incrementally and a new CONFIG option has been added to the
Kconfig, the build hanges waiting for input:

Test new config (NEW_CONFIG) [N/y/?] (NEW)

Since binamn does not connect the build's stdin to anything this waits on
stdin to the build thread, which never comes. Eventually I suspect all the
threads end up in this state and the build does not progress.

Fix this by passing /dev/null as input to the build. That way, if there is
a new CONFIG, the build will stop (and fail):

Test new config (NEW_CONFIG) [N/y/?] (NEW)

Error in reading or end of file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add support for environment delta in summary

When summarising the builds, add the -U option to emit delta lines for
the default environment built into U-Boot at each commit.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: add option -E for treating compiler warnings as errors

Add a new option '-E' for treating all compiler warnings as errors.
Eventually this will pass 'KCFLAGS=-Werror' to Kbuild.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>

buildman: Fix bloat option when 'new' only drops functions

In the case where a new build only decreases sizes and does not increase
any size we still want to report what functions have been dropped when
doing a bloat comparison.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Handle commit subjects containing unicode

One of these has crept in in this commit:

40a808f1 ARCv2: SLC: Make sure busy bit is set properly on SLC flushing

Adjust buildman to handle it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Clean up odd characters on the terminal

At present buildman leaves behind a few characters during its progress
updates, which looks odd. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Squash useless output from -K

When using #define CONFIG_SOME_OPTION, the value it set to '1'. When using
defconfig (i.e. CONFIG_SOME_OPTION=y) the value is set to 'y'. This results
in differences showing up with -K. These differences are seldom useful.

Adjust buildman to suppress these differences by default.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to just create the config

Normally buildman does a full build of a board. This includes creating the
u-boot.cfg file which contains all the configuration options. Buildman uses
this file with the -K option, to show differences in effective configuration
for each commit.

Doing a full build of U-Boot just to create the u-boot.cfg file is wasteful.
Add a -D option which causes buildman to only create the configuration. This
is enough to support use of -K and can be done much more quickly (typically
5-10 times faster).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Don't show a stacktrace on Ctrl-C

When Ctrl-C is pressed, just exited quietly. There is no sense in displaying
a stack trace since buildman will always be in the same place: waiting for
threads to complete building all the jobs on the queue.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Drop the 'active' flag in the builder

This serves no real purpose, since when we are not active, we exit. Drop it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow builds to terminate cleanly

It is annoying that buildman does not respond cleanly to Ctrl-C or SIGINT,
particularly on machines with lots of CPUS. Unfortunately queue.join()
blocks the main thread and does not allow it to see the signal. Use a
separate thread instead,

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Print a message indicating the build is starting

Make it clear when buildman actually starts building. This happens when it
has prepared the threads, working directory and output directories.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Print a message when removing old directories

When buildman starts, it prepares its output directory by removing any old
build directories which will not be used this time. This can happen if a
previous build left directories around for commit hashes which are no-longer
part of the branch.

This can take quite a while, so print a message to indicate what is going
on.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Tidy up the 'cloning' message

On a machine with a lot of CPUs this prints a lot of useless lines of the
form:

Cloning repo for thread <n>

Adjust the output so that these all appear on one line, and disappear when
the cloning is complete.

Note: This cloning is actually unnecessary and very wasteful on disk space
(about 3.5GB each time). It would be better to create symlinks.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: allow more incremental building

One use-case for buildman is to continually run it interactively after
each small step in a large refactoring operation. This gives more
immediate feedback than making a number of commits and then going back and
testing them. For this to work well, buildman needs to be extremely fast.
At present, a couple issues prevent it being as fast as it could be:

1) Each time buildman runs "make %_defconfig", it runs "make mrproper"
first. This throws away all previous build results, requiring a
from-scratch build. Optionally avoiding this would speed up the build, at
the cost of potentially causing or missing some build issues.

2) A build tree is created per thread rather than per board. When a thread
switches between building different boards, this often causes many files
to be rebuilt due to changing config options. Using a separate build tree
for each board would avoid this. This does put more strain on the system's
disk cache, but it is worth it on my system at least.

This commit adds two command-line options to implement the changes
described above; -I ("--incremental") turns of "make mrproper" and -P
("--per-board-out-dir") creats a build directory per board rather than per
thread.

Tested:

./tools/buildman/buildman.py tegra
./tools/buildman/buildman.py -I -P tegra
./tools/buildman/buildman.py -b tegra_dev tegra
./tools/buildman/buildman.py -b tegra_dev -I -P tegra

... each once after deleting the buildman result/work directory, and once
"incrementally" after a previous identical invocation.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Tom Rini <trini@konsulko.com>
Acked-by: Simon Glass <sjg@chromium.org> # v1
Tested-by: Simon Glass <sjg@chromium.org> # v1
Acked-by: Simon Glass <sjg@chromium.org>

buildman: Improve the config comparison feature

At present buildman can compare configurations between commits but the
feature is less useful than it could be. There is no summary by architecture
and changes are not reported on a per-board basis.

Correct these deficiencies so that it is possible to see exactly what is
changing for any number of boards.

Note that 'buildman -b <branch> -C' is recommended for any build where you
will be comparing configuration. Without -C the correct configuration will
not be reported since changes will often not be picked up.

Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow comparison of build configuration

It is useful to be able to see CONFIG changes made by commits. Add this
feature to buildman using the -K flag so that all CONFIG changes are
reported.

The CONFIG options exist in a number of files. Each is reported
individually as well as a summary that covers all files. The output
shows three parts: green for additions, red for removals and yellow for
changes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show 'make' command line when -V is used

When a verbose build it selected, show the make command before the output of
that command.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a space before the list of boards

Tweak the output slightly so we don't get things like:

- board1 board2+ board3 board4

There should be a space before the '+'.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to write the full build output

Normally buildman runs with 'make -s' meaning that only errors and warnings
appear in the log file. Add a -V option to run make in verbose mode, and
with V=1, causing a full build log to be created.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to use the full tool chain path

In some cases there may be multiple toolchains with the same name in the
path. Provide an option to use the full path in the CROSS_COMPILE
environment variable.

Note: Wolfgang mentioned that this is dangerous since in some cases there
may be other tools on the path that are needed. So this is set up as an
option, not the default. I will need test confirmation (i.e. that this
commit fixes a real problem) before merging it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Steve Rae <srae@broadcom.com>

buildman: Add an option to flatten output directory trees

When building current source for a single board, buildman puts the output
in <output_dir>/current/current/<board>. Add an option to make it use
<output_dir>/<board> instead. This removes the unnecessary directories
in that case, controlled by the --no-subdirs/-N option.

Suggested-by: Tom Rini <trini@ti.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Don't prune output space for 'current source' build

This is not needed since we always do a full (non-incremental) build. Also
it might be dangerous since it will try to delete everything below the
base directory.

Fix this potentially nasty bug.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix repeating board list with -l

Ensure that we don't print duplicate board names when -l is used.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reported-by: Albert Aribaud <albert.u.boot@aribaud.net>

buildman: Create parent directories as necessary

When creating build directories also create parents as necessary. This
fixes a failure when building a hierarchical branch (i.e. foo/bar).

Signed-off-by: Thierry Reding <treding@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Tom Rini <trini@ti.com>

buildman: Send builder output through a function for testing

To allow us to verify the builder's console output, send it through a
function which can collect it when running in test mode.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Separate out display of warnings and errors

Some boards unfortunately build with warnings and it is useful to be able
to easily distinguish the warnings from the errors.

Use a simple pattern match to categorise gcc output into warnings and
errors, and display each separately. New warnings are shown in magenta (with
a w+ prefix) and fixed warnings are shown in yellow with a w- prefix.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to show which boards caused which errors

Add a -l option to display a list of offending boards against each
error/warning line. The information will be shown in brackets as below:

02: wip
sandbox: + sandbox
arm: + seaboard
+(sandbox) arch/sandbox/cpu/cpu.c: In function 'timer_get_us':
+(sandbox) arch/sandbox/cpu/cpu.c:40:9: warning: unused variable 'i' [-Wunused-variable]
+(seaboard) board/nvidia/seaboard/seaboard.c: In function 'pin_mux_mmc':
+(seaboard) board/nvidia/seaboard/seaboard.c:36:9: warning: unused variable 'fred' [-Wunused-variable]
+(seaboard) int fred;
+(seaboard) ^

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Set the return code to indicate build result

When buildman finds errors/warnings when building, set the return code to
indicate this.

Suggested-by: York Sun <yorksun@freescale.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

Makefile: remove generated boards.cfg within make distclean

Signed-off-by: Roger Meier <roger@bufferoverflow.ch>
Reviewed-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Tested-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Cc: Tom Rini <trini@ti.com>
Cc: Wolfgang Denk <wd@denx.de>
Cc: Simon Glass <sjg@chromium.org>

buildman: Add a message indicating there are no errors

If buildman finds no problems it prints nothing. This can be a bit confusing,
so add a message that all is well.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Remove unused non-incremental build method code

The non-incremental build method is no longer used, so remove it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add verbose option to display errors as they happen

Normally buildman operates in two passes - one to do the build and another
to summarise the errors. Add a verbose option (-v) to display build problems
as they happen. With -e also given, this will display errors too.

When building the current source tree (rather than a list of commits in a
branch), both -v and -e are enabled automatically.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Refactor output options

We need the output options to be available in several places. It's a pain
to pass them into each function. Make them properties of the builder and
add a single function to set them up. At the same time, add a function which
produces summary output using these options.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move BuilderThread code to its own file

The builder.py file is getting too long, so split out some code.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow building of current source tree

Originally buildman had some support for building the current source tree.
However this was dropped before it was submitted, as part of the effort to
make it faster when building entire branches.

Reinstate this support. If no -b option is given, buildman will build the
current source tree.

Reported-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: adjust for Kconfig

Use "make <board>_defconfig" instead of "make <board>_config".

Invoke tools/genboardscfg.py to generate boards.cfg when it is missing.

Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: make sure to invoke GNU Make

Since the command name 'make' may not be GNU Make on some platforms
such as FreeBSD, buildman should call scripts/show-gnu-make to get
the command name for GNU MAKE (and error out if it is not found).

Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Jeroen Hofstee <jeroen@myspectrum.nl>

buildman: Support in-tree builds

At present buildman always builds out-of-tree, that is it uses a separate
output directory from the source directory. Normally this is what you want,
but it is important that in-tree builds work also. Some Makefile changes may
break this.

Add a -i option to tell buildman to use in-tree builds, so that it is easy
to test this feature.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add -C option to force a reconfigure for each commit

Normally buildman wil try to configure U-Boot for a particular board on the
first commit that it builds in a series. Subsequent commits are built
without reconfiguring which normally works. Where it doesn't, buildman
automatically reconfigures and retries.

To fully emulate the way MAKEALL works, we should have an option to disable
this optimisation.

Add a -C option to cause buildman to always reconfigure on each commit.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid retrying a build if it definitely failed

After a build fails buildman will reconfigure and try again, if it did not
reconfigure before the build. However it doesn't actually keep track of
whether it did reconfigure on the previous attempt.

Fix that logic to avoid a pointless rebuild. This speeds things up quite a
bit for failing builds. Previously they would always be built twice.

Change-Id: Ib37f21320baa7c60bed98f4042c0b7ed7c0dc85e
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add -F flag to retry failed builds

Generally a build failure with a particular commit cannot be fixed except
by changing that commit. Changing the commit will automatically cause
buildman to retry when you run it again: buildman sees that the commit
hash is different and that it has no previous build result for the new
commit hash.

However sometimes the build failure is due to a toolchain issue or some
other environment problem. In that case, retrying failed builds may yield
a different result.

Add a flag to retry failed builds. This differs from the force rebuild
flag (-f) in that it will not rebuild commits which are already marked as
succeeded.

Series-to: u-boot

Change-Id: Iac4306df499d65ff0888b1c60f06fc162a6faad8

buildman: Allow make flags to be specified for each board

There are a few make options such as BUILD_TAG which can be provided when
building U-Boot. Provide a way for buildman to pass these flags to make
also.

The flags should be in a [make-flags] section and arranged by target name
(the 'target' column in boards.cfg. See the README for more details.

Signed-off-by: Simon Glass <sjg@chromium.org>

Add GPL-2.0+ SPDX-License-Identifier to source files

Signed-off-by: Wolfgang Denk <wd@denx.de>
[trini: Fixup common/cmd_io.c]
Signed-off-by: Tom Rini <trini@ti.com>

buildman - U-Boot multi-threaded builder and summary tool

This tool handles building U-Boot to check that you have not broken it
with your patch series. It can build each individual commit and report
which boards fail on which commits, and which errors come up. It also
shows differences in image sizes due to particular commits.

Buildman aims to make full use of multi-processor machines.

Documentation and caveats are in tools/buildman/README.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Start the clock when the build starts

The Kconfig and maintainer processing can take a while, sometimes 5
seconds or more. This skews the timing printed by buildmand when the build
completes. Start the clock when the threads start to avoid this problem.

Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Tom Rini <trini@konsulko.com>

buildman: Export _get_output_dir() to avoid warnings

Make this a public memory since it is used outside the class.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in builderthread.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in builder.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test for --boards

Add a simple functional test for the --boards option. Fix the example in
the docs while we are here. Also improve the docs for Builder.count so it
is clearer what it contains.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move remaining builder properties to constructor

Do these all in the constructor, so it is consistent.

Move the stray builder comment into the correct place.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move library functions into a library directory

The patman directory has a number of modules which are used by other tools
in U-Boot. This makes it hard to package the tools using pypi since the
common files must be copied along with the tool that uses them.

To address this, move these files into a new u_boot_pylib library. This
can be packaged separately and listed as a dependency of each tool.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a flag for reproducible builds

This is quite a useful thing to use when building since it avoids small
size changes between commits. Add a -r flag for it.

Also undefine CONFIG_LOCALVERSION_AUTO since this appends the git hash
to the version string, causing every build to be slightly different.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support disabling LTO

This cuts down build performance considerably and is not always needed,
when checking for build errors, etc.

Add a flag to disable it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add --allow-missing flag to allow missing blobs

Add a new flag to buildman so that we will in turn pass
BINMAN_ALLOW_MISSING=1 to 'make'. Make use of this flag in CI.

Allow the settings file to control this.

Cc: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow lines without a symbol

The 'nm' tool can produce lines without a symbol, for example:

00000004 t

Silently skip these and anything else without three fields. Drop the
warning since there is nothing the user can do about it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reported-by: Tom Rini <trini@konsulko.com>

buildman: Drop use of 'boards' in control

Use brds instead so that we can reserve 'boards' for a module name.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support running from an IDE

Add a flag to allow buildman to behave properly for use from an IDE. This
shows error/warning output on stderr and drops all summary and progress
information.

This should normally only be used when building a single board.

Fix up a confusing comment for GetResultSummary() while we are here, since
we want to use the Outcome object to access the unprocessed error lines
from the build.

Signed-off-by: Simon Glass <sjg@chromium.org>

nds32: Remove the architecture

As removal of nds32 has been ack'd for the Linux kernel, remove support
here as well.

Cc: Rick Chen <rick@andestech.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Rick Chen <rick@andestech.com>

buildman: Correct pylint errors

Fix pylint errors that can be fixed and mask those that seem to be
incorrect.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in terminal.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Color() method to build()

This method has the same name as its class which is confusing. It is also
annoying when searching the code.

It builds a string with a colour, so rename it to build().

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Print() to Tprint()

Rename this function so that when we convert it to snake case it will not
conflict with the built-in print() function.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in gitutil.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in command.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow adjusting board config on the fly

Add a -a option to specify changes to the config before the build
commences. For example

buildman -a ~CONFIG_CMDLINE

disables CONFIG_CMDLINE before doing the build.

This makes it easier to try things out as well as to write tests without
creating a new board or manually manging the .config file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Detect Kconfig loops

Hex and int Kconfig options are supposed to have defaults. This is so we
can configure U-Boot without having to enter particular values for the
items that don't have specific values in the board's defconfig file.

If this rule is not followed, then introducing a new Kconfig can produce
a loop like this:

Break things (BREAK_ME) [] (NEW)
Error in reading or end of file.

Break things (BREAK_ME) [] (NEW)
Error in reading or end of file.

The continues forever since buildman passes /dev/null to 'conf', and
the build system just tries again. Eventually there is so much output that
buildman runs out of memory.

We can detect this situation by looking for a symbol (like 'BREAK_ME')
which has no default (the '[]' above) and is marked as new. If this
appears multiple times in the output, we know something is wrong.

Add a filter function for the output which detects this situation. Allow
it to return True to terminate the process. Implement this termination in
cros_subprocess.

With this we get a nice message:

buildman --board sandbox -T0
Building current source for 1 boards (0 threads, 32 jobs per thread)
sandbox: w+ sandbox
+.config:66:warning: symbol value '' invalid for BREAK_ME
+
+Error in reading or end of file.
+make[3]: *** [scripts/kconfig/Makefile:75: syncconfig] Terminated
+make[2]: *** [Makefile:569: syncconfig] Terminated
+make: *** [Makefile:177: sub-make] Terminated
+(** did you define an int/hex Kconfig with no default? **)

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Handle exceptions in threads gracefully

There have been at least a few cases where an exception has occurred in a
thread and resulted in buildman hanging: running out of disk space and
getting a unicode error.

Handle these by collecting a list of exceptions, printing them out and
reporting failure if any are found. Add a test for this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support single-threaded operation

At present even if only a single thread is in use, buildman still uses
threading.

For some debugging it is helpful to do everything in the main process.
Allow -T0 to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: 'Thread' object has no attribute 'isAlive'

The isAlive() method was deprecated in Python 3.8 and has been removed in
Python 3.9. See https://bugs.python.org/issue37804. Use is_alive() instead.

Since Python 2.6 is_alive() has been a synonym for isAlive(). So there
should be no problems for users using elder Python 3 versions.

Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Use git worktrees instead of git clones when possible

This patch makes buildman create linked working trees instead of clones
of the source repository, but keeps updating the older clones of the
repository that might already exist. These worktrees share "everything
except working directory specific files such as HEAD, index, etc." with
the source repository. See the git-worktree(1) manual page for more
information.

If git-worktree isn't available, silently falls back to cloning the
repository.

Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

buildman: Show the build rate at the end

It is interesting to note the number of builds completed per second to
track machine performance and build speed. Add a 'rate' value at the end
of the build to show this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Remove _of_#_ from results directory paths

Currently, the following scenario will rebuild the first commit even
though it is not really necessary - the commit sha or the position in the
patchset did not change:

$ git am <local-patch-0001>
$ tools/buildman/buildman -P -E -W -b master mx6
<do some more development work>
$ git am <local-patch-0002>
$ tools/buildman/buildman -P -E -W -b master mx6 <- will rebuild the first
commit as well, even
though nothing has
changed about it.

This is due to the fact that previous results directories get removed
when the number of commits change. By removing the _of_#_ part of the
directory path, the commits will be rebuilt only if the commit sha or the
position in the patchset changes. Also, update the testcase to reflect this
change.

Signed-off-by: Ovidiu Panait <ovidiu.panait@windriver.com>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Write output files when using -w

At present buildman does not write its own output files (err, done, the
environment) when using -w. However this is useful for when the build is
run with -s to check it.

In fact ProduceResultSummary() reads the result from those files rather
than using the 'result' info directly. So ProcessResult() does not work
with -w at present. It does not print any output.

Fix this by writing output files even when -w is used.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore migration warnings

These are becoming more common now. They cause boards to show warnings
which can be mistaking for compiler warnings.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore device-tree warnings

Unfortunately the plague of device-tree warnings has not lifted. These
warnings infiltrate almost every build, adding noise and confusion.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make -I the default

At present buildman defaults to running 'mrproper' on every thread before
it starts building commits for each board. This can add a delay of about 5
seconds to the start of the process, since the tools and other invariants
must be rebuilt.

In particular, a build without '-b', to build current source, runs much
slower without -I, since any existing build is removed, thus losing the
possibility of an incremental build.

Partly this behaviour was to avoid strange build-system problems caused by
running 'make defconfig' for one board and then one with a different
architecture. But these problems were fixed quite a while ago.

The -I option (which disabled mrproper) was introduced four years ago and
does not seem to cause any problems with builds.

So make -I the default and deprecate the option. To allow use of
'mrproper', add a new -m flag.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show a summary of the build result

When buildman finishes it leaves the last summary line visible, which
shows the number of successful builds, builds with warnings and builds
with errors.

It is useful also to see how many builds were done in total along with
the time taken. Show these on a separate line before buildman finishes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Limit the length of progress messages

If a progress message is longer than the terminal line it will scroll the
terminal. Limit the messages to the terminal width.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show the number of builds remaining

It is nice to see the actual number of builds remaining to complete. Add
this in the progress message, using a different colour.

Drop the unnecessary 'name' variable while we are here.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Drop unused output code

The commit counter is a hangover from when buildman processed each board
for a commit. Now buildman processes each commit for a board, so this
output is never triggered.

Delete it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show a message when fetching a repo

Fetching updated versions of a repo can take time. At present buildman
gives no indication that it is doing this.

Add a message to explain the delay.

Tidy up a few other messages while we are here.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Drop the line-clearing code in Builder

The new feature in terminal can be used by buildman. Update the Builder
class accordingly.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use spaces in the board list

At present the board names shown with -l are separated by commas. This
makes it hard to double-click to select a particular board. Also it is not
possible to select all boards and paste them as arguments to a subsequent
buildman run, since buildman requires spaces to separate the list on the
command line, not commas.

Change the output format to use spaces instead of commas.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show the list of boards in magenta

It is quite hard to see the list of board for each error line since the
colour is the same as the actual error line. Show the board list in
magenta so that it is easier to distinguish them.

There is no point in checking the colour of the overall line, since there
are now multiple colours. So drop those tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use an object to hold error lines

At present the string for each error line is created in _CalcErrorDelta()
and used to create the summary output. This is inflexible since all the
information (error/warning character, error line, list of boards with that
error line) is munged together in a string.

Create an object to hold this information and only convert it to a string
when printing the actual output.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use yellow consistently for warning lines

At present warnings are shown in yellow in the summary (-s) but magenta in
the detail listing (-e). Use yellow in both.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Refactor error-line output int a function

Reduce the amount of repeated code by creating an _OutputErrLines()
function to hold this code.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow building within a subdir of the current dir

This is useful in some situations, in particular with -w and when building
in-tree. Now that we are more careful about what we remove in
_PrepareOutputSpace(), it should be safe to relax this restriction.

Update the progress information also so it is clear what buildman is
doing. Remove files can take a long time.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Be more selective about which directories to remove

At present buildman removes any directory it doesn't intend to write
output into. This is overly expansive since if the output directory
happens to be somewhere with existing files, they may be removed. Using
an existing directory for buildman is not a good practice, but since the
result might be catastrophic, it is best to guard against it.

A previous commit[1] fixed this by refusing to write to a subdirectory
of the current directory, assumed to have U-Boot source code. But we can
do better by only removing directories that look like the ones buildman
creates.

Update the code to do this and add a test.

Signed-off-by: Simon Glass <sjg@chromium.org>

[1] 409fc029c40 tools: buildman: Don't use the working dir as build dir

buildman: Update help for -d

This help is a bit ambiguous. It only does anything if asked to show size
changes with -S. Update the help and the function comments.

Signed-off-by: Simon Glass <sjg@chromium.org>

bulidman: Add support for a simple build

It is useful to run a simple build and put all the output in a single
directory. Add a -w option to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: buildman: fix typo

Signed-off-by: Flavio Suligoi <f.suligoi@asem.it>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Ignore blank lines during size checks

Today when parsing the .sizes files we get a warning about an invalid
line in the file as it's blank. Solve this by checking that we have a
non-blank line prior to processing.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Deal more nicely with invalid build-status file

The 'done' files created by buildman may end up being empty if buildman
runs out of disk space while writing them. At present buildman dies with
an exception when using -s to check the build status. Fix this.

Seriesl-cc: trini

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show boards with warning with w+

At present we should boards with warnings in the same way as those with
errors. This is not ideal. Add a new 'warn' state and show these listed
in yellow to match the actual warning lines printing with -e.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Rename the good, better, worse variables

At present we don't distinguish between errors and warnings when printing
the architecture summary. Rename the variables to better describe their
purpose.

'Worse' at present means we got an error, so use that as the name.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Detect dtc warnings

At present messages from the device-tree compiler like this:

arch/arm/dts/socfpga_arria10_socdk_sdmmc.dtb: Warning
(avoid_unnecessary_addr_size): /clocks: unnecessary
#address-cells/#size-cells without "ranges" or child "reg" property

are detected as errors since they don't match the gcc warning regex. Add a
new one for dtc to fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid hanging when the config changes

Something has changed in the last several month such that when buildman
builds U-Boot incrementally and a new CONFIG option has been added to the
Kconfig, the build hanges waiting for input:

Test new config (NEW_CONFIG) [N/y/?] (NEW)

Since binamn does not connect the build's stdin to anything this waits on
stdin to the build thread, which never comes. Eventually I suspect all the
threads end up in this state and the build does not progress.

Fix this by passing /dev/null as input to the build. That way, if there is
a new CONFIG, the build will stop (and fail):

Test new config (NEW_CONFIG) [N/y/?] (NEW)

Error in reading or end of file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add support for environment delta in summary

When summarising the builds, add the -U option to emit delta lines for
the default environment built into U-Boot at each commit.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: add option -E for treating compiler warnings as errors

Add a new option '-E' for treating all compiler warnings as errors.
Eventually this will pass 'KCFLAGS=-Werror' to Kbuild.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>

buildman: Fix bloat option when 'new' only drops functions

In the case where a new build only decreases sizes and does not increase
any size we still want to report what functions have been dropped when
doing a bloat comparison.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Handle commit subjects containing unicode

One of these has crept in in this commit:

40a808f1 ARCv2: SLC: Make sure busy bit is set properly on SLC flushing

Adjust buildman to handle it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Clean up odd characters on the terminal

At present buildman leaves behind a few characters during its progress
updates, which looks odd. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Squash useless output from -K

When using #define CONFIG_SOME_OPTION, the value it set to '1'. When using
defconfig (i.e. CONFIG_SOME_OPTION=y) the value is set to 'y'. This results
in differences showing up with -K. These differences are seldom useful.

Adjust buildman to suppress these differences by default.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to just create the config

Normally buildman does a full build of a board. This includes creating the
u-boot.cfg file which contains all the configuration options. Buildman uses
this file with the -K option, to show differences in effective configuration
for each commit.

Doing a full build of U-Boot just to create the u-boot.cfg file is wasteful.
Add a -D option which causes buildman to only create the configuration. This
is enough to support use of -K and can be done much more quickly (typically
5-10 times faster).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Don't show a stacktrace on Ctrl-C

When Ctrl-C is pressed, just exited quietly. There is no sense in displaying
a stack trace since buildman will always be in the same place: waiting for
threads to complete building all the jobs on the queue.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Drop the 'active' flag in the builder

This serves no real purpose, since when we are not active, we exit. Drop it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow builds to terminate cleanly

It is annoying that buildman does not respond cleanly to Ctrl-C or SIGINT,
particularly on machines with lots of CPUS. Unfortunately queue.join()
blocks the main thread and does not allow it to see the signal. Use a
separate thread instead,

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Print a message indicating the build is starting

Make it clear when buildman actually starts building. This happens when it
has prepared the threads, working directory and output directories.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Print a message when removing old directories

When buildman starts, it prepares its output directory by removing any old
build directories which will not be used this time. This can happen if a
previous build left directories around for commit hashes which are no-longer
part of the branch.

This can take quite a while, so print a message to indicate what is going
on.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Tidy up the 'cloning' message

On a machine with a lot of CPUs this prints a lot of useless lines of the
form:

Cloning repo for thread <n>

Adjust the output so that these all appear on one line, and disappear when
the cloning is complete.

Note: This cloning is actually unnecessary and very wasteful on disk space
(about 3.5GB each time). It would be better to create symlinks.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: allow more incremental building

One use-case for buildman is to continually run it interactively after
each small step in a large refactoring operation. This gives more
immediate feedback than making a number of commits and then going back and
testing them. For this to work well, buildman needs to be extremely fast.
At present, a couple issues prevent it being as fast as it could be:

1) Each time buildman runs "make %_defconfig", it runs "make mrproper"
first. This throws away all previous build results, requiring a
from-scratch build. Optionally avoiding this would speed up the build, at
the cost of potentially causing or missing some build issues.

2) A build tree is created per thread rather than per board. When a thread
switches between building different boards, this often causes many files
to be rebuilt due to changing config options. Using a separate build tree
for each board would avoid this. This does put more strain on the system's
disk cache, but it is worth it on my system at least.

This commit adds two command-line options to implement the changes
described above; -I ("--incremental") turns of "make mrproper" and -P
("--per-board-out-dir") creats a build directory per board rather than per
thread.

Tested:

./tools/buildman/buildman.py tegra
./tools/buildman/buildman.py -I -P tegra
./tools/buildman/buildman.py -b tegra_dev tegra
./tools/buildman/buildman.py -b tegra_dev -I -P tegra

... each once after deleting the buildman result/work directory, and once
"incrementally" after a previous identical invocation.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Tom Rini <trini@konsulko.com>
Acked-by: Simon Glass <sjg@chromium.org> # v1
Tested-by: Simon Glass <sjg@chromium.org> # v1
Acked-by: Simon Glass <sjg@chromium.org>

buildman: Improve the config comparison feature

At present buildman can compare configurations between commits but the
feature is less useful than it could be. There is no summary by architecture
and changes are not reported on a per-board basis.

Correct these deficiencies so that it is possible to see exactly what is
changing for any number of boards.

Note that 'buildman -b <branch> -C' is recommended for any build where you
will be comparing configuration. Without -C the correct configuration will
not be reported since changes will often not be picked up.

Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow comparison of build configuration

It is useful to be able to see CONFIG changes made by commits. Add this
feature to buildman using the -K flag so that all CONFIG changes are
reported.

The CONFIG options exist in a number of files. Each is reported
individually as well as a summary that covers all files. The output
shows three parts: green for additions, red for removals and yellow for
changes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show 'make' command line when -V is used

When a verbose build it selected, show the make command before the output of
that command.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a space before the list of boards

Tweak the output slightly so we don't get things like:

- board1 board2+ board3 board4

There should be a space before the '+'.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to write the full build output

Normally buildman runs with 'make -s' meaning that only errors and warnings
appear in the log file. Add a -V option to run make in verbose mode, and
with V=1, causing a full build log to be created.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to use the full tool chain path

In some cases there may be multiple toolchains with the same name in the
path. Provide an option to use the full path in the CROSS_COMPILE
environment variable.

Note: Wolfgang mentioned that this is dangerous since in some cases there
may be other tools on the path that are needed. So this is set up as an
option, not the default. I will need test confirmation (i.e. that this
commit fixes a real problem) before merging it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Steve Rae <srae@broadcom.com>

buildman: Add an option to flatten output directory trees

When building current source for a single board, buildman puts the output
in <output_dir>/current/current/<board>. Add an option to make it use
<output_dir>/<board> instead. This removes the unnecessary directories
in that case, controlled by the --no-subdirs/-N option.

Suggested-by: Tom Rini <trini@ti.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Don't prune output space for 'current source' build

This is not needed since we always do a full (non-incremental) build. Also
it might be dangerous since it will try to delete everything below the
base directory.

Fix this potentially nasty bug.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix repeating board list with -l

Ensure that we don't print duplicate board names when -l is used.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reported-by: Albert Aribaud <albert.u.boot@aribaud.net>

buildman: Create parent directories as necessary

When creating build directories also create parents as necessary. This
fixes a failure when building a hierarchical branch (i.e. foo/bar).

Signed-off-by: Thierry Reding <treding@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Tom Rini <trini@ti.com>

buildman: Send builder output through a function for testing

To allow us to verify the builder's console output, send it through a
function which can collect it when running in test mode.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Separate out display of warnings and errors

Some boards unfortunately build with warnings and it is useful to be able
to easily distinguish the warnings from the errors.

Use a simple pattern match to categorise gcc output into warnings and
errors, and display each separately. New warnings are shown in magenta (with
a w+ prefix) and fixed warnings are shown in yellow with a w- prefix.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to show which boards caused which errors

Add a -l option to display a list of offending boards against each
error/warning line. The information will be shown in brackets as below:

02: wip
sandbox: + sandbox
arm: + seaboard
+(sandbox) arch/sandbox/cpu/cpu.c: In function 'timer_get_us':
+(sandbox) arch/sandbox/cpu/cpu.c:40:9: warning: unused variable 'i' [-Wunused-variable]
+(seaboard) board/nvidia/seaboard/seaboard.c: In function 'pin_mux_mmc':
+(seaboard) board/nvidia/seaboard/seaboard.c:36:9: warning: unused variable 'fred' [-Wunused-variable]
+(seaboard) int fred;
+(seaboard) ^

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Set the return code to indicate build result

When buildman finds errors/warnings when building, set the return code to
indicate this.

Suggested-by: York Sun <yorksun@freescale.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

Makefile: remove generated boards.cfg within make distclean

Signed-off-by: Roger Meier <roger@bufferoverflow.ch>
Reviewed-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Tested-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Cc: Tom Rini <trini@ti.com>
Cc: Wolfgang Denk <wd@denx.de>
Cc: Simon Glass <sjg@chromium.org>

buildman: Add a message indicating there are no errors

If buildman finds no problems it prints nothing. This can be a bit confusing,
so add a message that all is well.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Remove unused non-incremental build method code

The non-incremental build method is no longer used, so remove it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add verbose option to display errors as they happen

Normally buildman operates in two passes - one to do the build and another
to summarise the errors. Add a verbose option (-v) to display build problems
as they happen. With -e also given, this will display errors too.

When building the current source tree (rather than a list of commits in a
branch), both -v and -e are enabled automatically.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Refactor output options

We need the output options to be available in several places. It's a pain
to pass them into each function. Make them properties of the builder and
add a single function to set them up. At the same time, add a function which
produces summary output using these options.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move BuilderThread code to its own file

The builder.py file is getting too long, so split out some code.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow building of current source tree

Originally buildman had some support for building the current source tree.
However this was dropped before it was submitted, as part of the effort to
make it faster when building entire branches.

Reinstate this support. If no -b option is given, buildman will build the
current source tree.

Reported-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: adjust for Kconfig

Use "make <board>_defconfig" instead of "make <board>_config".

Invoke tools/genboardscfg.py to generate boards.cfg when it is missing.

Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: make sure to invoke GNU Make

Since the command name 'make' may not be GNU Make on some platforms
such as FreeBSD, buildman should call scripts/show-gnu-make to get
the command name for GNU MAKE (and error out if it is not found).

Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Jeroen Hofstee <jeroen@myspectrum.nl>

buildman: Support in-tree builds

At present buildman always builds out-of-tree, that is it uses a separate
output directory from the source directory. Normally this is what you want,
but it is important that in-tree builds work also. Some Makefile changes may
break this.

Add a -i option to tell buildman to use in-tree builds, so that it is easy
to test this feature.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add -C option to force a reconfigure for each commit

Normally buildman wil try to configure U-Boot for a particular board on the
first commit that it builds in a series. Subsequent commits are built
without reconfiguring which normally works. Where it doesn't, buildman
automatically reconfigures and retries.

To fully emulate the way MAKEALL works, we should have an option to disable
this optimisation.

Add a -C option to cause buildman to always reconfigure on each commit.

Signed-off-by: Simon Glass <sjg@chromium.org>