tools: typo arguemnts

%s/arguemnts/arguemnts/

Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>

buildman: Export _get_output_dir() to avoid warnings

Make this a public memory since it is used outside the class.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in builder.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in bsettings.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in control.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move library functions into a library directory

The patman directory has a number of modules which are used by other tools
in U-Boot. This makes it hard to package the tools using pypi since the
common files must be copied along with the tool that uses them.

To address this, move these files into a new u_boot_pylib library. This
can be packaged separately and listed as a dependency of each tool.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Split out Boards into its own file

Use a separate file for the Boards class so that its name matches the
module name.

Fix up the function names to match the pylint style and fix some other
warnings.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in board.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix use of 'boards' in test

We want to create a module called 'boards' so avoid use of this variable
name in this module. Change the global to be capitalised, as required by
Python style.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: buildman: Fix doc path in warning message

Fix documentation path in deprecated warning message about device
driver.

Signed-off-by: Johannes Krottmayer <krjdev@gmail.com>
Reviewed-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>

patman: Convert camel case in terminal.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Color() method to build()

This method has the same name as its class which is confusing. It is also
annoying when searching the code.

It builds a string with a colour, so rename it to build().

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add helper functions for updating .config files

At present the only straightforward way to write tests that need a
slightly different configuration is to create a new board with its own
configuration. This is cumbersome.

It would be useful if buildman could adjust the configuration of a build
on the fly. In preparation for this, add a utility library which can
modify a .config file according to various parameters passed to it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support single-threaded operation

At present even if only a single thread is in use, buildman still uses
threading.

For some debugging it is helpful to do everything in the main process.
Allow -T0 to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: Drop unnecessary use of __file__

There are few places where the path of the current modules is calculated
but not used. Drop them.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct the testOutputDir() unit test

This current fails with an error. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: 7664b03ffc5 ("buildman: Remove _of_#_ from results directory paths")

buildman: Remove _of_#_ from results directory paths

Currently, the following scenario will rebuild the first commit even
though it is not really necessary - the commit sha or the position in the
patchset did not change:

$ git am <local-patch-0001>
$ tools/buildman/buildman -P -E -W -b master mx6
<do some more development work>
$ git am <local-patch-0002>
$ tools/buildman/buildman -P -E -W -b master mx6 <- will rebuild the first
commit as well, even
though nothing has
changed about it.

This is due to the fact that previous results directories get removed
when the number of commits change. By removing the _of_#_ part of the
directory path, the commits will be rebuilt only if the commit sha or the
position in the patchset changes. Also, update the testcase to reflect this
change.

Signed-off-by: Ovidiu Panait <ovidiu.panait@windriver.com>

patman: Tidy up sys.path changes

Now that we are using absolute paths we can remove some of the sys.path
mangling that appears in the tools.

We only need to add the path to 'tools/' so that everything can find
modules relative to that directory.

The special paths for finding pylibfdt remain.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix test for new 9.2 kernel

The naming is slightly different on kernel.org now. Update the regex so
that the test still passes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore migration warnings

These are becoming more common now. They cause boards to show warnings
which can be mistaking for compiler warnings.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore device-tree warnings

Unfortunately the plague of device-tree warnings has not lifted. These
warnings infiltrate almost every build, adding noise and confusion.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show a summary of the build result

When buildman finishes it leaves the last summary line visible, which
shows the number of successful builds, builds with warnings and builds
with errors.

It is useful also to see how many builds were done in total along with
the time taken. Show these on a separate line before buildman finishes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use spaces in the board list

At present the board names shown with -l are separated by commas. This
makes it hard to double-click to select a particular board. Also it is not
possible to select all boards and paste them as arguments to a subsequent
buildman run, since buildman requires spaces to separate the list on the
command line, not commas.

Change the output format to use spaces instead of commas.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show the list of boards in magenta

It is quite hard to see the list of board for each error line since the
colour is the same as the actual error line. Show the board list in
magenta so that it is easier to distinguish them.

There is no point in checking the colour of the overall line, since there
are now multiple colours. So drop those tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use yellow consistently for warning lines

At present warnings are shown in yellow in the summary (-s) but magenta in
the detail listing (-e). Use yellow in both.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Test the output with --list-error-boards

Add a test to cover this flag, which adds the name of each board to each
error/warning line.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test helper for creating a line prefix

The split/join code is repeated in a lot of places. Add a function to
handle this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Split out testOutput() into separate functions

We want to add a few more tests similar to testOutput(). Split its logic
into a function which runs buildman to get the output and another which
checks the output. This will make it easier to reuse the code.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Create temp directory in test setup

Rather than having a few tests handle this themselves, create the
temporary directory in the setUp() method and remove it in tearDown().
This will make it easier to add more tests.

Only testOutput and testGit() actually need it, but it doesn't add to the
test time noticeably to do this for all tests in this file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use an iterator to check test output

Rather than using the absolute array index, use an interator to work
through the expected output lines. This is easier to follow.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add test coverage for error/warning colour

Buildman should output the right colours for each error/warning line. Some
of these checks are missing. Add them.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Be more selective about which directories to remove

At present buildman removes any directory it doesn't intend to write
output into. This is overly expansive since if the output directory
happens to be somewhere with existing files, they may be removed. Using
an existing directory for buildman is not a good practice, but since the
result might be catastrophic, it is best to guard against it.

A previous commit[1] fixed this by refusing to write to a subdirectory
of the current directory, assumed to have U-Boot source code. But we can
do better by only removing directories that look like the ones buildman
creates.

Update the code to do this and add a test.

Signed-off-by: Simon Glass <sjg@chromium.org>

[1] 409fc029c40 tools: buildman: Don't use the working dir as build dir

buildman: Add options to get the arch and toolchain info

Sometimes it is useful for external tools to use buildman to provide the
toolchain information. Add an -a option which shows the value to use for
the ARCH environment variable, and -A which does the same for
CROSS_COMPILE

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: buildman: Remove useless mkdir() in Make() in test.py

In the 'Make' function, the codes tries to create a directory
if current stage is 'build'. But the directory isn't used at
all anywhere.

Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Show boards with warning with w+

At present we should boards with warnings in the same way as those with
errors. This is not ideal. Add a new 'warn' state and show these listed
in yellow to match the actual warning lines printing with -e.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Detect dtc warnings

At present messages from the device-tree compiler like this:

arch/arm/dts/socfpga_arria10_socdk_sdmmc.dtb: Warning
(avoid_unnecessary_addr_size): /clocks: unnecessary
#address-cells/#size-cells without "ranges" or child "reg" property

are detected as errors since they don't match the gcc warning regex. Add a
new one for dtc to fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a --boards option to specify particular boards to build

At present 'buildman sandbox' will build all 5 boards for the sandbox
architecture rather than the single board 'sandbox'. The only current way
to exclude sandbox_spl, sandbox_noblk, etc. is to use -x which is a bit
clumbsy.

Add a --boards option to allow individual build targets to be specified.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: dtoc: Suppress unwanted output from test

There are a few test cases which print output. Suppress this so that tests
can run silently in the normal case.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make the toolchain test more forgiving

The filenames of the toolchains on kernel.org changes every now and then.
Fix it for the current change, and make the test use a regex so that it
has a better chance of passing with future changes too.

Signed-off-by: Simon Glass <sjg@chromium.org>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: Fix up tests

The tests were broken by two separate commits which adjusted the output
when boards are listed. Fix this by adding back a PowerPC board and
putting the name of each board in the test.

Fixes: b9f7d881 (powerpc, 5xx: remove some "5xx" remains)
Fixes: 8d7523c5 (buildman: Allow showing the list of boards with -n)

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow skipping of tests which use the network

Accessing the network slows down the test and limits the environment in
which it can be run. Add an option to disable network tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

powerpc, 5xx: remove some "5xx" remains

we removed 5xx support. So delete some forgotten remains.

Signed-off-by: Heiko Schocher <hs@denx.de>

buildman: Print a message indicating the build is starting

Make it clear when buildman actually starts building. This happens when it
has prepared the threads, working directory and output directories.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add gcc 4.9.0 with Microblaze toolchain

Also read gcc 4.9.0 at kernel.org which also have Microblaze toolchain.

Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

Fixed unit test failure by updating the test:
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a space before the list of boards

Tweak the output slightly so we don't get things like:

- board1 board2+ board3 board4

There should be a space before the '+'.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add the option to download toolchains from kernel.org

The site at https://www.kernel.org/pub/tools/crosstool/ is a convenient
repository of toolchains which can be used for U-Boot. Add a feature to
download and install a toolchain for a selected architecture automatically.

It isn't clear how long this site will stay in the current place and
format, but we should be able to rely on bug reports if it changes.

Suggested-by: Marek Vašut <marex@denx.de>
Suggested-by: Fabio Estevam <festevam@gmail.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow architecture to alias to multiple toolchains

Some archs have need than one alias, so support a list of alises in the
..buildman file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Don't use the local settings when running tests

We should create a test setting file when running testes, not use whatever
happens to be on the local machine.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to flatten output directory trees

When building current source for a single board, buildman puts the output
in <output_dir>/current/current/<board>. Add an option to make it use
<output_dir>/<board> instead. This removes the unnecessary directories
in that case, controlled by the --no-subdirs/-N option.

Suggested-by: Tom Rini <trini@ti.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add tests that check the correct output directory is used

Add a few tests of the output directory logic.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Expand output test to cover directory prefixes

Now that buildman supports removing the build directory prefix from output,
add a test for it. Also ensure that output directories are removed when the
test completes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Enhance basic test to check summary output

Adjust the basic test so that it checks all console output. This will help
to ensure that the builder is behaving correctly with printing summary
information.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Introduce an 'and' operator for board selection

Currently buildman allows a list of boards to build to be specified on the
command line. The list can include specific board names, architecture, SOC
and so on.

At present the list of boards is dealt with in an 'OR' fashion, and there
is no way to specify something like 'arm & freescale', meaning boards with
ARM architecture but only those made by Freescale. This would exclude the
PowerPC boards made by Freescale.

Support an '&' operator on the command line to permit this. Ensure that
arguments can be specified in a single string to permit easy shell quoting.

Suggested-by: York Sun <yorksun@freescale.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
Acked-by: York Sun <yorksun@freescale.com>

buildman: Remove unused non-incremental build method code

The non-incremental build method is no longer used, so remove it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add verbose option to display errors as they happen

Normally buildman operates in two passes - one to do the build and another
to summarise the errors. Add a verbose option (-v) to display build problems
as they happen. With -e also given, this will display errors too.

When building the current source tree (rather than a list of commits in a
branch), both -v and -e are enabled automatically.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Refactor output options

We need the output options to be available in several places. It's a pain
to pass them into each function. Make them properties of the builder and
add a single function to set them up. At the same time, add a function which
produces summary output using these options.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Adjust tests for new boards.cfg format

Commit 27af930e9a5c91365ca639ada580b338eabe4989 changed the boards.cfg format
but missed to change the parsing in buildman. A follow-on commit
03c1bb242548e4e9d267d784861ccd69a1887aa0 fixed this but missed fixing the
tests.

This patch updates the tests to fit the new Board constructor.

./tools/buildman/buildman -t
<unittest.result.TestResult run=1 errors=0 failures=0>

Signed-off-by: Simon Glass <sjg@chromium.org>

Add GPL-2.0+ SPDX-License-Identifier to source files

Signed-off-by: Wolfgang Denk <wd@denx.de>
[trini: Fixup common/cmd_io.c]
Signed-off-by: Tom Rini <trini@ti.com>

buildman - U-Boot multi-threaded builder and summary tool

This tool handles building U-Boot to check that you have not broken it
with your patch series. It can build each individual commit and report
which boards fail on which commits, and which errors come up. It also
shows differences in image sizes due to particular commits.

Buildman aims to make full use of multi-processor machines.

Documentation and caveats are in tools/buildman/README.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Export _get_output_dir() to avoid warnings

Make this a public memory since it is used outside the class.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in builder.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in bsettings.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in control.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move library functions into a library directory

The patman directory has a number of modules which are used by other tools
in U-Boot. This makes it hard to package the tools using pypi since the
common files must be copied along with the tool that uses them.

To address this, move these files into a new u_boot_pylib library. This
can be packaged separately and listed as a dependency of each tool.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Split out Boards into its own file

Use a separate file for the Boards class so that its name matches the
module name.

Fix up the function names to match the pylint style and fix some other
warnings.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in board.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix use of 'boards' in test

We want to create a module called 'boards' so avoid use of this variable
name in this module. Change the global to be capitalised, as required by
Python style.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: buildman: Fix doc path in warning message

Fix documentation path in deprecated warning message about device
driver.

Signed-off-by: Johannes Krottmayer <krjdev@gmail.com>
Reviewed-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>

patman: Convert camel case in terminal.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Color() method to build()

This method has the same name as its class which is confusing. It is also
annoying when searching the code.

It builds a string with a colour, so rename it to build().

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add helper functions for updating .config files

At present the only straightforward way to write tests that need a
slightly different configuration is to create a new board with its own
configuration. This is cumbersome.

It would be useful if buildman could adjust the configuration of a build
on the fly. In preparation for this, add a utility library which can
modify a .config file according to various parameters passed to it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support single-threaded operation

At present even if only a single thread is in use, buildman still uses
threading.

For some debugging it is helpful to do everything in the main process.
Allow -T0 to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: Drop unnecessary use of __file__

There are few places where the path of the current modules is calculated
but not used. Drop them.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct the testOutputDir() unit test

This current fails with an error. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: 7664b03ffc5 ("buildman: Remove _of_#_ from results directory paths")

buildman: Remove _of_#_ from results directory paths

Currently, the following scenario will rebuild the first commit even
though it is not really necessary - the commit sha or the position in the
patchset did not change:

$ git am <local-patch-0001>
$ tools/buildman/buildman -P -E -W -b master mx6
<do some more development work>
$ git am <local-patch-0002>
$ tools/buildman/buildman -P -E -W -b master mx6 <- will rebuild the first
commit as well, even
though nothing has
changed about it.

This is due to the fact that previous results directories get removed
when the number of commits change. By removing the _of_#_ part of the
directory path, the commits will be rebuilt only if the commit sha or the
position in the patchset changes. Also, update the testcase to reflect this
change.

Signed-off-by: Ovidiu Panait <ovidiu.panait@windriver.com>

patman: Tidy up sys.path changes

Now that we are using absolute paths we can remove some of the sys.path
mangling that appears in the tools.

We only need to add the path to 'tools/' so that everything can find
modules relative to that directory.

The special paths for finding pylibfdt remain.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix test for new 9.2 kernel

The naming is slightly different on kernel.org now. Update the regex so
that the test still passes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore migration warnings

These are becoming more common now. They cause boards to show warnings
which can be mistaking for compiler warnings.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore device-tree warnings

Unfortunately the plague of device-tree warnings has not lifted. These
warnings infiltrate almost every build, adding noise and confusion.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show a summary of the build result

When buildman finishes it leaves the last summary line visible, which
shows the number of successful builds, builds with warnings and builds
with errors.

It is useful also to see how many builds were done in total along with
the time taken. Show these on a separate line before buildman finishes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use spaces in the board list

At present the board names shown with -l are separated by commas. This
makes it hard to double-click to select a particular board. Also it is not
possible to select all boards and paste them as arguments to a subsequent
buildman run, since buildman requires spaces to separate the list on the
command line, not commas.

Change the output format to use spaces instead of commas.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show the list of boards in magenta

It is quite hard to see the list of board for each error line since the
colour is the same as the actual error line. Show the board list in
magenta so that it is easier to distinguish them.

There is no point in checking the colour of the overall line, since there
are now multiple colours. So drop those tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use yellow consistently for warning lines

At present warnings are shown in yellow in the summary (-s) but magenta in
the detail listing (-e). Use yellow in both.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Test the output with --list-error-boards

Add a test to cover this flag, which adds the name of each board to each
error/warning line.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test helper for creating a line prefix

The split/join code is repeated in a lot of places. Add a function to
handle this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Split out testOutput() into separate functions

We want to add a few more tests similar to testOutput(). Split its logic
into a function which runs buildman to get the output and another which
checks the output. This will make it easier to reuse the code.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Create temp directory in test setup

Rather than having a few tests handle this themselves, create the
temporary directory in the setUp() method and remove it in tearDown().
This will make it easier to add more tests.

Only testOutput and testGit() actually need it, but it doesn't add to the
test time noticeably to do this for all tests in this file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use an iterator to check test output

Rather than using the absolute array index, use an interator to work
through the expected output lines. This is easier to follow.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add test coverage for error/warning colour

Buildman should output the right colours for each error/warning line. Some
of these checks are missing. Add them.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Be more selective about which directories to remove

At present buildman removes any directory it doesn't intend to write
output into. This is overly expansive since if the output directory
happens to be somewhere with existing files, they may be removed. Using
an existing directory for buildman is not a good practice, but since the
result might be catastrophic, it is best to guard against it.

A previous commit[1] fixed this by refusing to write to a subdirectory
of the current directory, assumed to have U-Boot source code. But we can
do better by only removing directories that look like the ones buildman
creates.

Update the code to do this and add a test.

Signed-off-by: Simon Glass <sjg@chromium.org>

[1] 409fc029c40 tools: buildman: Don't use the working dir as build dir

buildman: Add options to get the arch and toolchain info

Sometimes it is useful for external tools to use buildman to provide the
toolchain information. Add an -a option which shows the value to use for
the ARCH environment variable, and -A which does the same for
CROSS_COMPILE

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: buildman: Remove useless mkdir() in Make() in test.py

In the 'Make' function, the codes tries to create a directory
if current stage is 'build'. But the directory isn't used at
all anywhere.

Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Show boards with warning with w+

At present we should boards with warnings in the same way as those with
errors. This is not ideal. Add a new 'warn' state and show these listed
in yellow to match the actual warning lines printing with -e.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Detect dtc warnings

At present messages from the device-tree compiler like this:

arch/arm/dts/socfpga_arria10_socdk_sdmmc.dtb: Warning
(avoid_unnecessary_addr_size): /clocks: unnecessary
#address-cells/#size-cells without "ranges" or child "reg" property

are detected as errors since they don't match the gcc warning regex. Add a
new one for dtc to fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a --boards option to specify particular boards to build

At present 'buildman sandbox' will build all 5 boards for the sandbox
architecture rather than the single board 'sandbox'. The only current way
to exclude sandbox_spl, sandbox_noblk, etc. is to use -x which is a bit
clumbsy.

Add a --boards option to allow individual build targets to be specified.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: dtoc: Suppress unwanted output from test

There are a few test cases which print output. Suppress this so that tests
can run silently in the normal case.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make the toolchain test more forgiving

The filenames of the toolchains on kernel.org changes every now and then.
Fix it for the current change, and make the test use a regex so that it
has a better chance of passing with future changes too.

Signed-off-by: Simon Glass <sjg@chromium.org>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: Fix up tests

The tests were broken by two separate commits which adjusted the output
when boards are listed. Fix this by adding back a PowerPC board and
putting the name of each board in the test.

Fixes: b9f7d881 (powerpc, 5xx: remove some "5xx" remains)
Fixes: 8d7523c5 (buildman: Allow showing the list of boards with -n)

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow skipping of tests which use the network

Accessing the network slows down the test and limits the environment in
which it can be run. Add an option to disable network tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

powerpc, 5xx: remove some "5xx" remains

we removed 5xx support. So delete some forgotten remains.

Signed-off-by: Heiko Schocher <hs@denx.de>

buildman: Print a message indicating the build is starting

Make it clear when buildman actually starts building. This happens when it
has prepared the threads, working directory and output directories.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add gcc 4.9.0 with Microblaze toolchain

Also read gcc 4.9.0 at kernel.org which also have Microblaze toolchain.

Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

Fixed unit test failure by updating the test:
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a space before the list of boards

Tweak the output slightly so we don't get things like:

- board1 board2+ board3 board4

There should be a space before the '+'.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add the option to download toolchains from kernel.org

The site at https://www.kernel.org/pub/tools/crosstool/ is a convenient
repository of toolchains which can be used for U-Boot. Add a feature to
download and install a toolchain for a selected architecture automatically.

It isn't clear how long this site will stay in the current place and
format, but we should be able to rely on bug reports if it changes.

Suggested-by: Marek Vašut <marex@denx.de>
Suggested-by: Fabio Estevam <festevam@gmail.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow architecture to alias to multiple toolchains

Some archs have need than one alias, so support a list of alises in the
..buildman file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Don't use the local settings when running tests

We should create a test setting file when running testes, not use whatever
happens to be on the local machine.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to flatten output directory trees

When building current source for a single board, buildman puts the output
in <output_dir>/current/current/<board>. Add an option to make it use
<output_dir>/<board> instead. This removes the unnecessary directories
in that case, controlled by the --no-subdirs/-N option.

Suggested-by: Tom Rini <trini@ti.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add tests that check the correct output directory is used

Add a few tests of the output directory logic.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Expand output test to cover directory prefixes

Now that buildman supports removing the build directory prefix from output,
add a test for it. Also ensure that output directories are removed when the
test completes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Enhance basic test to check summary output

Adjust the basic test so that it checks all console output. This will help
to ensure that the builder is behaving correctly with printing summary
information.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Introduce an 'and' operator for board selection

Currently buildman allows a list of boards to build to be specified on the
command line. The list can include specific board names, architecture, SOC
and so on.

At present the list of boards is dealt with in an 'OR' fashion, and there
is no way to specify something like 'arm & freescale', meaning boards with
ARM architecture but only those made by Freescale. This would exclude the
PowerPC boards made by Freescale.

Support an '&' operator on the command line to permit this. Ensure that
arguments can be specified in a single string to permit easy shell quoting.

Suggested-by: York Sun <yorksun@freescale.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
Acked-by: York Sun <yorksun@freescale.com>

buildman: Remove unused non-incremental build method code

The non-incremental build method is no longer used, so remove it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add verbose option to display errors as they happen

Normally buildman operates in two passes - one to do the build and another
to summarise the errors. Add a verbose option (-v) to display build problems
as they happen. With -e also given, this will display errors too.

When building the current source tree (rather than a list of commits in a
branch), both -v and -e are enabled automatically.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Refactor output options

We need the output options to be available in several places. It's a pain
to pass them into each function. Make them properties of the builder and
add a single function to set them up. At the same time, add a function which
produces summary output using these options.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Adjust tests for new boards.cfg format

Commit 27af930e9a5c91365ca639ada580b338eabe4989 changed the boards.cfg format
but missed to change the parsing in buildman. A follow-on commit
03c1bb242548e4e9d267d784861ccd69a1887aa0 fixed this but missed fixing the
tests.

This patch updates the tests to fit the new Board constructor.

./tools/buildman/buildman -t
<unittest.result.TestResult run=1 errors=0 failures=0>

Signed-off-by: Simon Glass <sjg@chromium.org>

Add GPL-2.0+ SPDX-License-Identifier to source files

Signed-off-by: Wolfgang Denk <wd@denx.de>
[trini: Fixup common/cmd_io.c]
Signed-off-by: Tom Rini <trini@ti.com>

buildman - U-Boot multi-threaded builder and summary tool

This tool handles building U-Boot to check that you have not broken it
with your patch series. It can build each individual commit and report
which boards fail on which commits, and which errors come up. It also
shows differences in image sizes due to particular commits.

Buildman aims to make full use of multi-processor machines.

Documentation and caveats are in tools/buildman/README.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move library functions into a library directory

The patman directory has a number of modules which are used by other tools
in U-Boot. This makes it hard to package the tools using pypi since the
common files must be copied along with the tool that uses them.

To address this, move these files into a new u_boot_pylib library. This
can be packaged separately and listed as a dependency of each tool.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Split out Boards into its own file

Use a separate file for the Boards class so that its name matches the
module name.

Fix up the function names to match the pylint style and fix some other
warnings.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in board.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix use of 'boards' in test

We want to create a module called 'boards' so avoid use of this variable
name in this module. Change the global to be capitalised, as required by
Python style.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: buildman: Fix doc path in warning message

Fix documentation path in deprecated warning message about device
driver.

Signed-off-by: Johannes Krottmayer <krjdev@gmail.com>
Reviewed-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>

patman: Convert camel case in terminal.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Color() method to build()

This method has the same name as its class which is confusing. It is also
annoying when searching the code.

It builds a string with a colour, so rename it to build().

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add helper functions for updating .config files

At present the only straightforward way to write tests that need a
slightly different configuration is to create a new board with its own
configuration. This is cumbersome.

It would be useful if buildman could adjust the configuration of a build
on the fly. In preparation for this, add a utility library which can
modify a .config file according to various parameters passed to it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support single-threaded operation

At present even if only a single thread is in use, buildman still uses
threading.

For some debugging it is helpful to do everything in the main process.
Allow -T0 to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: Drop unnecessary use of __file__

There are few places where the path of the current modules is calculated
but not used. Drop them.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct the testOutputDir() unit test

This current fails with an error. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: 7664b03ffc5 ("buildman: Remove _of_#_ from results directory paths")

buildman: Remove _of_#_ from results directory paths

Currently, the following scenario will rebuild the first commit even
though it is not really necessary - the commit sha or the position in the
patchset did not change:

$ git am <local-patch-0001>
$ tools/buildman/buildman -P -E -W -b master mx6
<do some more development work>
$ git am <local-patch-0002>
$ tools/buildman/buildman -P -E -W -b master mx6 <- will rebuild the first
commit as well, even
though nothing has
changed about it.

This is due to the fact that previous results directories get removed
when the number of commits change. By removing the _of_#_ part of the
directory path, the commits will be rebuilt only if the commit sha or the
position in the patchset changes. Also, update the testcase to reflect this
change.

Signed-off-by: Ovidiu Panait <ovidiu.panait@windriver.com>

patman: Tidy up sys.path changes

Now that we are using absolute paths we can remove some of the sys.path
mangling that appears in the tools.

We only need to add the path to 'tools/' so that everything can find
modules relative to that directory.

The special paths for finding pylibfdt remain.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix test for new 9.2 kernel

The naming is slightly different on kernel.org now. Update the regex so
that the test still passes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore migration warnings

These are becoming more common now. They cause boards to show warnings
which can be mistaking for compiler warnings.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore device-tree warnings

Unfortunately the plague of device-tree warnings has not lifted. These
warnings infiltrate almost every build, adding noise and confusion.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show a summary of the build result

When buildman finishes it leaves the last summary line visible, which
shows the number of successful builds, builds with warnings and builds
with errors.

It is useful also to see how many builds were done in total along with
the time taken. Show these on a separate line before buildman finishes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use spaces in the board list

At present the board names shown with -l are separated by commas. This
makes it hard to double-click to select a particular board. Also it is not
possible to select all boards and paste them as arguments to a subsequent
buildman run, since buildman requires spaces to separate the list on the
command line, not commas.

Change the output format to use spaces instead of commas.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show the list of boards in magenta

It is quite hard to see the list of board for each error line since the
colour is the same as the actual error line. Show the board list in
magenta so that it is easier to distinguish them.

There is no point in checking the colour of the overall line, since there
are now multiple colours. So drop those tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use yellow consistently for warning lines

At present warnings are shown in yellow in the summary (-s) but magenta in
the detail listing (-e). Use yellow in both.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Test the output with --list-error-boards

Add a test to cover this flag, which adds the name of each board to each
error/warning line.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test helper for creating a line prefix

The split/join code is repeated in a lot of places. Add a function to
handle this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Split out testOutput() into separate functions

We want to add a few more tests similar to testOutput(). Split its logic
into a function which runs buildman to get the output and another which
checks the output. This will make it easier to reuse the code.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Create temp directory in test setup

Rather than having a few tests handle this themselves, create the
temporary directory in the setUp() method and remove it in tearDown().
This will make it easier to add more tests.

Only testOutput and testGit() actually need it, but it doesn't add to the
test time noticeably to do this for all tests in this file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use an iterator to check test output

Rather than using the absolute array index, use an interator to work
through the expected output lines. This is easier to follow.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add test coverage for error/warning colour

Buildman should output the right colours for each error/warning line. Some
of these checks are missing. Add them.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Be more selective about which directories to remove

At present buildman removes any directory it doesn't intend to write
output into. This is overly expansive since if the output directory
happens to be somewhere with existing files, they may be removed. Using
an existing directory for buildman is not a good practice, but since the
result might be catastrophic, it is best to guard against it.

A previous commit[1] fixed this by refusing to write to a subdirectory
of the current directory, assumed to have U-Boot source code. But we can
do better by only removing directories that look like the ones buildman
creates.

Update the code to do this and add a test.

Signed-off-by: Simon Glass <sjg@chromium.org>

[1] 409fc029c40 tools: buildman: Don't use the working dir as build dir

buildman: Add options to get the arch and toolchain info

Sometimes it is useful for external tools to use buildman to provide the
toolchain information. Add an -a option which shows the value to use for
the ARCH environment variable, and -A which does the same for
CROSS_COMPILE

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: buildman: Remove useless mkdir() in Make() in test.py

In the 'Make' function, the codes tries to create a directory
if current stage is 'build'. But the directory isn't used at
all anywhere.

Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Show boards with warning with w+

At present we should boards with warnings in the same way as those with
errors. This is not ideal. Add a new 'warn' state and show these listed
in yellow to match the actual warning lines printing with -e.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Detect dtc warnings

At present messages from the device-tree compiler like this:

arch/arm/dts/socfpga_arria10_socdk_sdmmc.dtb: Warning
(avoid_unnecessary_addr_size): /clocks: unnecessary
#address-cells/#size-cells without "ranges" or child "reg" property

are detected as errors since they don't match the gcc warning regex. Add a
new one for dtc to fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a --boards option to specify particular boards to build

At present 'buildman sandbox' will build all 5 boards for the sandbox
architecture rather than the single board 'sandbox'. The only current way
to exclude sandbox_spl, sandbox_noblk, etc. is to use -x which is a bit
clumbsy.

Add a --boards option to allow individual build targets to be specified.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: dtoc: Suppress unwanted output from test

There are a few test cases which print output. Suppress this so that tests
can run silently in the normal case.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make the toolchain test more forgiving

The filenames of the toolchains on kernel.org changes every now and then.
Fix it for the current change, and make the test use a regex so that it
has a better chance of passing with future changes too.

Signed-off-by: Simon Glass <sjg@chromium.org>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: Fix up tests

The tests were broken by two separate commits which adjusted the output
when boards are listed. Fix this by adding back a PowerPC board and
putting the name of each board in the test.

Fixes: b9f7d881 (powerpc, 5xx: remove some "5xx" remains)
Fixes: 8d7523c5 (buildman: Allow showing the list of boards with -n)

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow skipping of tests which use the network

Accessing the network slows down the test and limits the environment in
which it can be run. Add an option to disable network tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

powerpc, 5xx: remove some "5xx" remains

we removed 5xx support. So delete some forgotten remains.

Signed-off-by: Heiko Schocher <hs@denx.de>

buildman: Print a message indicating the build is starting

Make it clear when buildman actually starts building. This happens when it
has prepared the threads, working directory and output directories.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add gcc 4.9.0 with Microblaze toolchain

Also read gcc 4.9.0 at kernel.org which also have Microblaze toolchain.

Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

Fixed unit test failure by updating the test:
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a space before the list of boards

Tweak the output slightly so we don't get things like:

- board1 board2+ board3 board4

There should be a space before the '+'.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add the option to download toolchains from kernel.org

The site at https://www.kernel.org/pub/tools/crosstool/ is a convenient
repository of toolchains which can be used for U-Boot. Add a feature to
download and install a toolchain for a selected architecture automatically.

It isn't clear how long this site will stay in the current place and
format, but we should be able to rely on bug reports if it changes.

Suggested-by: Marek Vašut <marex@denx.de>
Suggested-by: Fabio Estevam <festevam@gmail.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow architecture to alias to multiple toolchains

Some archs have need than one alias, so support a list of alises in the
..buildman file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Don't use the local settings when running tests

We should create a test setting file when running testes, not use whatever
happens to be on the local machine.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to flatten output directory trees

When building current source for a single board, buildman puts the output
in <output_dir>/current/current/<board>. Add an option to make it use
<output_dir>/<board> instead. This removes the unnecessary directories
in that case, controlled by the --no-subdirs/-N option.

Suggested-by: Tom Rini <trini@ti.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add tests that check the correct output directory is used

Add a few tests of the output directory logic.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Expand output test to cover directory prefixes

Now that buildman supports removing the build directory prefix from output,
add a test for it. Also ensure that output directories are removed when the
test completes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Enhance basic test to check summary output

Adjust the basic test so that it checks all console output. This will help
to ensure that the builder is behaving correctly with printing summary
information.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Introduce an 'and' operator for board selection

Currently buildman allows a list of boards to build to be specified on the
command line. The list can include specific board names, architecture, SOC
and so on.

At present the list of boards is dealt with in an 'OR' fashion, and there
is no way to specify something like 'arm & freescale', meaning boards with
ARM architecture but only those made by Freescale. This would exclude the
PowerPC boards made by Freescale.

Support an '&' operator on the command line to permit this. Ensure that
arguments can be specified in a single string to permit easy shell quoting.

Suggested-by: York Sun <yorksun@freescale.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
Acked-by: York Sun <yorksun@freescale.com>

buildman: Remove unused non-incremental build method code

The non-incremental build method is no longer used, so remove it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add verbose option to display errors as they happen

Normally buildman operates in two passes - one to do the build and another
to summarise the errors. Add a verbose option (-v) to display build problems
as they happen. With -e also given, this will display errors too.

When building the current source tree (rather than a list of commits in a
branch), both -v and -e are enabled automatically.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Refactor output options

We need the output options to be available in several places. It's a pain
to pass them into each function. Make them properties of the builder and
add a single function to set them up. At the same time, add a function which
produces summary output using these options.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Adjust tests for new boards.cfg format

Commit 27af930e9a5c91365ca639ada580b338eabe4989 changed the boards.cfg format
but missed to change the parsing in buildman. A follow-on commit
03c1bb242548e4e9d267d784861ccd69a1887aa0 fixed this but missed fixing the
tests.

This patch updates the tests to fit the new Board constructor.

./tools/buildman/buildman -t
<unittest.result.TestResult run=1 errors=0 failures=0>

Signed-off-by: Simon Glass <sjg@chromium.org>

Add GPL-2.0+ SPDX-License-Identifier to source files

Signed-off-by: Wolfgang Denk <wd@denx.de>
[trini: Fixup common/cmd_io.c]
Signed-off-by: Tom Rini <trini@ti.com>

buildman - U-Boot multi-threaded builder and summary tool

This tool handles building U-Boot to check that you have not broken it
with your patch series. It can build each individual commit and report
which boards fail on which commits, and which errors come up. It also
shows differences in image sizes due to particular commits.

Buildman aims to make full use of multi-processor machines.

Documentation and caveats are in tools/buildman/README.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Split out Boards into its own file

Use a separate file for the Boards class so that its name matches the
module name.

Fix up the function names to match the pylint style and fix some other
warnings.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in board.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix use of 'boards' in test

We want to create a module called 'boards' so avoid use of this variable
name in this module. Change the global to be capitalised, as required by
Python style.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: buildman: Fix doc path in warning message

Fix documentation path in deprecated warning message about device
driver.

Signed-off-by: Johannes Krottmayer <krjdev@gmail.com>
Reviewed-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>

patman: Convert camel case in terminal.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Color() method to build()

This method has the same name as its class which is confusing. It is also
annoying when searching the code.

It builds a string with a colour, so rename it to build().

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add helper functions for updating .config files

At present the only straightforward way to write tests that need a
slightly different configuration is to create a new board with its own
configuration. This is cumbersome.

It would be useful if buildman could adjust the configuration of a build
on the fly. In preparation for this, add a utility library which can
modify a .config file according to various parameters passed to it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support single-threaded operation

At present even if only a single thread is in use, buildman still uses
threading.

For some debugging it is helpful to do everything in the main process.
Allow -T0 to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: Drop unnecessary use of __file__

There are few places where the path of the current modules is calculated
but not used. Drop them.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct the testOutputDir() unit test

This current fails with an error. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: 7664b03ffc5 ("buildman: Remove _of_#_ from results directory paths")

buildman: Remove _of_#_ from results directory paths

Currently, the following scenario will rebuild the first commit even
though it is not really necessary - the commit sha or the position in the
patchset did not change:

$ git am <local-patch-0001>
$ tools/buildman/buildman -P -E -W -b master mx6
<do some more development work>
$ git am <local-patch-0002>
$ tools/buildman/buildman -P -E -W -b master mx6 <- will rebuild the first
commit as well, even
though nothing has
changed about it.

This is due to the fact that previous results directories get removed
when the number of commits change. By removing the _of_#_ part of the
directory path, the commits will be rebuilt only if the commit sha or the
position in the patchset changes. Also, update the testcase to reflect this
change.

Signed-off-by: Ovidiu Panait <ovidiu.panait@windriver.com>

patman: Tidy up sys.path changes

Now that we are using absolute paths we can remove some of the sys.path
mangling that appears in the tools.

We only need to add the path to 'tools/' so that everything can find
modules relative to that directory.

The special paths for finding pylibfdt remain.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix test for new 9.2 kernel

The naming is slightly different on kernel.org now. Update the regex so
that the test still passes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore migration warnings

These are becoming more common now. They cause boards to show warnings
which can be mistaking for compiler warnings.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore device-tree warnings

Unfortunately the plague of device-tree warnings has not lifted. These
warnings infiltrate almost every build, adding noise and confusion.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show a summary of the build result

When buildman finishes it leaves the last summary line visible, which
shows the number of successful builds, builds with warnings and builds
with errors.

It is useful also to see how many builds were done in total along with
the time taken. Show these on a separate line before buildman finishes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use spaces in the board list

At present the board names shown with -l are separated by commas. This
makes it hard to double-click to select a particular board. Also it is not
possible to select all boards and paste them as arguments to a subsequent
buildman run, since buildman requires spaces to separate the list on the
command line, not commas.

Change the output format to use spaces instead of commas.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show the list of boards in magenta

It is quite hard to see the list of board for each error line since the
colour is the same as the actual error line. Show the board list in
magenta so that it is easier to distinguish them.

There is no point in checking the colour of the overall line, since there
are now multiple colours. So drop those tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use yellow consistently for warning lines

At present warnings are shown in yellow in the summary (-s) but magenta in
the detail listing (-e). Use yellow in both.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Test the output with --list-error-boards

Add a test to cover this flag, which adds the name of each board to each
error/warning line.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test helper for creating a line prefix

The split/join code is repeated in a lot of places. Add a function to
handle this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Split out testOutput() into separate functions

We want to add a few more tests similar to testOutput(). Split its logic
into a function which runs buildman to get the output and another which
checks the output. This will make it easier to reuse the code.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Create temp directory in test setup

Rather than having a few tests handle this themselves, create the
temporary directory in the setUp() method and remove it in tearDown().
This will make it easier to add more tests.

Only testOutput and testGit() actually need it, but it doesn't add to the
test time noticeably to do this for all tests in this file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use an iterator to check test output

Rather than using the absolute array index, use an interator to work
through the expected output lines. This is easier to follow.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add test coverage for error/warning colour

Buildman should output the right colours for each error/warning line. Some
of these checks are missing. Add them.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Be more selective about which directories to remove

At present buildman removes any directory it doesn't intend to write
output into. This is overly expansive since if the output directory
happens to be somewhere with existing files, they may be removed. Using
an existing directory for buildman is not a good practice, but since the
result might be catastrophic, it is best to guard against it.

A previous commit[1] fixed this by refusing to write to a subdirectory
of the current directory, assumed to have U-Boot source code. But we can
do better by only removing directories that look like the ones buildman
creates.

Update the code to do this and add a test.

Signed-off-by: Simon Glass <sjg@chromium.org>

[1] 409fc029c40 tools: buildman: Don't use the working dir as build dir

buildman: Add options to get the arch and toolchain info

Sometimes it is useful for external tools to use buildman to provide the
toolchain information. Add an -a option which shows the value to use for
the ARCH environment variable, and -A which does the same for
CROSS_COMPILE

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: buildman: Remove useless mkdir() in Make() in test.py

In the 'Make' function, the codes tries to create a directory
if current stage is 'build'. But the directory isn't used at
all anywhere.

Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Show boards with warning with w+

At present we should boards with warnings in the same way as those with
errors. This is not ideal. Add a new 'warn' state and show these listed
in yellow to match the actual warning lines printing with -e.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Detect dtc warnings

At present messages from the device-tree compiler like this:

arch/arm/dts/socfpga_arria10_socdk_sdmmc.dtb: Warning
(avoid_unnecessary_addr_size): /clocks: unnecessary
#address-cells/#size-cells without "ranges" or child "reg" property

are detected as errors since they don't match the gcc warning regex. Add a
new one for dtc to fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a --boards option to specify particular boards to build

At present 'buildman sandbox' will build all 5 boards for the sandbox
architecture rather than the single board 'sandbox'. The only current way
to exclude sandbox_spl, sandbox_noblk, etc. is to use -x which is a bit
clumbsy.

Add a --boards option to allow individual build targets to be specified.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: dtoc: Suppress unwanted output from test

There are a few test cases which print output. Suppress this so that tests
can run silently in the normal case.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make the toolchain test more forgiving

The filenames of the toolchains on kernel.org changes every now and then.
Fix it for the current change, and make the test use a regex so that it
has a better chance of passing with future changes too.

Signed-off-by: Simon Glass <sjg@chromium.org>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: Fix up tests

The tests were broken by two separate commits which adjusted the output
when boards are listed. Fix this by adding back a PowerPC board and
putting the name of each board in the test.

Fixes: b9f7d881 (powerpc, 5xx: remove some "5xx" remains)
Fixes: 8d7523c5 (buildman: Allow showing the list of boards with -n)

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow skipping of tests which use the network

Accessing the network slows down the test and limits the environment in
which it can be run. Add an option to disable network tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

powerpc, 5xx: remove some "5xx" remains

we removed 5xx support. So delete some forgotten remains.

Signed-off-by: Heiko Schocher <hs@denx.de>

buildman: Print a message indicating the build is starting

Make it clear when buildman actually starts building. This happens when it
has prepared the threads, working directory and output directories.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add gcc 4.9.0 with Microblaze toolchain

Also read gcc 4.9.0 at kernel.org which also have Microblaze toolchain.

Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

Fixed unit test failure by updating the test:
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a space before the list of boards

Tweak the output slightly so we don't get things like:

- board1 board2+ board3 board4

There should be a space before the '+'.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add the option to download toolchains from kernel.org

The site at https://www.kernel.org/pub/tools/crosstool/ is a convenient
repository of toolchains which can be used for U-Boot. Add a feature to
download and install a toolchain for a selected architecture automatically.

It isn't clear how long this site will stay in the current place and
format, but we should be able to rely on bug reports if it changes.

Suggested-by: Marek Vašut <marex@denx.de>
Suggested-by: Fabio Estevam <festevam@gmail.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow architecture to alias to multiple toolchains

Some archs have need than one alias, so support a list of alises in the
..buildman file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Don't use the local settings when running tests

We should create a test setting file when running testes, not use whatever
happens to be on the local machine.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to flatten output directory trees

When building current source for a single board, buildman puts the output
in <output_dir>/current/current/<board>. Add an option to make it use
<output_dir>/<board> instead. This removes the unnecessary directories
in that case, controlled by the --no-subdirs/-N option.

Suggested-by: Tom Rini <trini@ti.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add tests that check the correct output directory is used

Add a few tests of the output directory logic.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Expand output test to cover directory prefixes

Now that buildman supports removing the build directory prefix from output,
add a test for it. Also ensure that output directories are removed when the
test completes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Enhance basic test to check summary output

Adjust the basic test so that it checks all console output. This will help
to ensure that the builder is behaving correctly with printing summary
information.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Introduce an 'and' operator for board selection

Currently buildman allows a list of boards to build to be specified on the
command line. The list can include specific board names, architecture, SOC
and so on.

At present the list of boards is dealt with in an 'OR' fashion, and there
is no way to specify something like 'arm & freescale', meaning boards with
ARM architecture but only those made by Freescale. This would exclude the
PowerPC boards made by Freescale.

Support an '&' operator on the command line to permit this. Ensure that
arguments can be specified in a single string to permit easy shell quoting.

Suggested-by: York Sun <yorksun@freescale.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
Acked-by: York Sun <yorksun@freescale.com>

buildman: Remove unused non-incremental build method code

The non-incremental build method is no longer used, so remove it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add verbose option to display errors as they happen

Normally buildman operates in two passes - one to do the build and another
to summarise the errors. Add a verbose option (-v) to display build problems
as they happen. With -e also given, this will display errors too.

When building the current source tree (rather than a list of commits in a
branch), both -v and -e are enabled automatically.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Refactor output options

We need the output options to be available in several places. It's a pain
to pass them into each function. Make them properties of the builder and
add a single function to set them up. At the same time, add a function which
produces summary output using these options.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Adjust tests for new boards.cfg format

Commit 27af930e9a5c91365ca639ada580b338eabe4989 changed the boards.cfg format
but missed to change the parsing in buildman. A follow-on commit
03c1bb242548e4e9d267d784861ccd69a1887aa0 fixed this but missed fixing the
tests.

This patch updates the tests to fit the new Board constructor.

./tools/buildman/buildman -t
<unittest.result.TestResult run=1 errors=0 failures=0>

Signed-off-by: Simon Glass <sjg@chromium.org>

Add GPL-2.0+ SPDX-License-Identifier to source files

Signed-off-by: Wolfgang Denk <wd@denx.de>
[trini: Fixup common/cmd_io.c]
Signed-off-by: Tom Rini <trini@ti.com>

buildman - U-Boot multi-threaded builder and summary tool

This tool handles building U-Boot to check that you have not broken it
with your patch series. It can build each individual commit and report
which boards fail on which commits, and which errors come up. It also
shows differences in image sizes due to particular commits.

Buildman aims to make full use of multi-processor machines.

Documentation and caveats are in tools/buildman/README.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: buildman: Fix doc path in warning message

Fix documentation path in deprecated warning message about device
driver.

Signed-off-by: Johannes Krottmayer <krjdev@gmail.com>
Reviewed-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>

patman: Convert camel case in terminal.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Color() method to build()

This method has the same name as its class which is confusing. It is also
annoying when searching the code.

It builds a string with a colour, so rename it to build().

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add helper functions for updating .config files

At present the only straightforward way to write tests that need a
slightly different configuration is to create a new board with its own
configuration. This is cumbersome.

It would be useful if buildman could adjust the configuration of a build
on the fly. In preparation for this, add a utility library which can
modify a .config file according to various parameters passed to it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support single-threaded operation

At present even if only a single thread is in use, buildman still uses
threading.

For some debugging it is helpful to do everything in the main process.
Allow -T0 to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: Drop unnecessary use of __file__

There are few places where the path of the current modules is calculated
but not used. Drop them.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct the testOutputDir() unit test

This current fails with an error. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: 7664b03ffc5 ("buildman: Remove _of_#_ from results directory paths")

buildman: Remove _of_#_ from results directory paths

Currently, the following scenario will rebuild the first commit even
though it is not really necessary - the commit sha or the position in the
patchset did not change:

$ git am <local-patch-0001>
$ tools/buildman/buildman -P -E -W -b master mx6
<do some more development work>
$ git am <local-patch-0002>
$ tools/buildman/buildman -P -E -W -b master mx6 <- will rebuild the first
commit as well, even
though nothing has
changed about it.

This is due to the fact that previous results directories get removed
when the number of commits change. By removing the _of_#_ part of the
directory path, the commits will be rebuilt only if the commit sha or the
position in the patchset changes. Also, update the testcase to reflect this
change.

Signed-off-by: Ovidiu Panait <ovidiu.panait@windriver.com>

patman: Tidy up sys.path changes

Now that we are using absolute paths we can remove some of the sys.path
mangling that appears in the tools.

We only need to add the path to 'tools/' so that everything can find
modules relative to that directory.

The special paths for finding pylibfdt remain.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix test for new 9.2 kernel

The naming is slightly different on kernel.org now. Update the regex so
that the test still passes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore migration warnings

These are becoming more common now. They cause boards to show warnings
which can be mistaking for compiler warnings.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore device-tree warnings

Unfortunately the plague of device-tree warnings has not lifted. These
warnings infiltrate almost every build, adding noise and confusion.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show a summary of the build result

When buildman finishes it leaves the last summary line visible, which
shows the number of successful builds, builds with warnings and builds
with errors.

It is useful also to see how many builds were done in total along with
the time taken. Show these on a separate line before buildman finishes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use spaces in the board list

At present the board names shown with -l are separated by commas. This
makes it hard to double-click to select a particular board. Also it is not
possible to select all boards and paste them as arguments to a subsequent
buildman run, since buildman requires spaces to separate the list on the
command line, not commas.

Change the output format to use spaces instead of commas.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show the list of boards in magenta

It is quite hard to see the list of board for each error line since the
colour is the same as the actual error line. Show the board list in
magenta so that it is easier to distinguish them.

There is no point in checking the colour of the overall line, since there
are now multiple colours. So drop those tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use yellow consistently for warning lines

At present warnings are shown in yellow in the summary (-s) but magenta in
the detail listing (-e). Use yellow in both.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Test the output with --list-error-boards

Add a test to cover this flag, which adds the name of each board to each
error/warning line.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test helper for creating a line prefix

The split/join code is repeated in a lot of places. Add a function to
handle this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Split out testOutput() into separate functions

We want to add a few more tests similar to testOutput(). Split its logic
into a function which runs buildman to get the output and another which
checks the output. This will make it easier to reuse the code.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Create temp directory in test setup

Rather than having a few tests handle this themselves, create the
temporary directory in the setUp() method and remove it in tearDown().
This will make it easier to add more tests.

Only testOutput and testGit() actually need it, but it doesn't add to the
test time noticeably to do this for all tests in this file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use an iterator to check test output

Rather than using the absolute array index, use an interator to work
through the expected output lines. This is easier to follow.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add test coverage for error/warning colour

Buildman should output the right colours for each error/warning line. Some
of these checks are missing. Add them.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Be more selective about which directories to remove

At present buildman removes any directory it doesn't intend to write
output into. This is overly expansive since if the output directory
happens to be somewhere with existing files, they may be removed. Using
an existing directory for buildman is not a good practice, but since the
result might be catastrophic, it is best to guard against it.

A previous commit[1] fixed this by refusing to write to a subdirectory
of the current directory, assumed to have U-Boot source code. But we can
do better by only removing directories that look like the ones buildman
creates.

Update the code to do this and add a test.

Signed-off-by: Simon Glass <sjg@chromium.org>

[1] 409fc029c40 tools: buildman: Don't use the working dir as build dir

buildman: Add options to get the arch and toolchain info

Sometimes it is useful for external tools to use buildman to provide the
toolchain information. Add an -a option which shows the value to use for
the ARCH environment variable, and -A which does the same for
CROSS_COMPILE

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: buildman: Remove useless mkdir() in Make() in test.py

In the 'Make' function, the codes tries to create a directory
if current stage is 'build'. But the directory isn't used at
all anywhere.

Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Show boards with warning with w+

At present we should boards with warnings in the same way as those with
errors. This is not ideal. Add a new 'warn' state and show these listed
in yellow to match the actual warning lines printing with -e.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Detect dtc warnings

At present messages from the device-tree compiler like this:

arch/arm/dts/socfpga_arria10_socdk_sdmmc.dtb: Warning
(avoid_unnecessary_addr_size): /clocks: unnecessary
#address-cells/#size-cells without "ranges" or child "reg" property

are detected as errors since they don't match the gcc warning regex. Add a
new one for dtc to fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a --boards option to specify particular boards to build

At present 'buildman sandbox' will build all 5 boards for the sandbox
architecture rather than the single board 'sandbox'. The only current way
to exclude sandbox_spl, sandbox_noblk, etc. is to use -x which is a bit
clumbsy.

Add a --boards option to allow individual build targets to be specified.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: dtoc: Suppress unwanted output from test

There are a few test cases which print output. Suppress this so that tests
can run silently in the normal case.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make the toolchain test more forgiving

The filenames of the toolchains on kernel.org changes every now and then.
Fix it for the current change, and make the test use a regex so that it
has a better chance of passing with future changes too.

Signed-off-by: Simon Glass <sjg@chromium.org>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: Fix up tests

The tests were broken by two separate commits which adjusted the output
when boards are listed. Fix this by adding back a PowerPC board and
putting the name of each board in the test.

Fixes: b9f7d881 (powerpc, 5xx: remove some "5xx" remains)
Fixes: 8d7523c5 (buildman: Allow showing the list of boards with -n)

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow skipping of tests which use the network

Accessing the network slows down the test and limits the environment in
which it can be run. Add an option to disable network tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

powerpc, 5xx: remove some "5xx" remains

we removed 5xx support. So delete some forgotten remains.

Signed-off-by: Heiko Schocher <hs@denx.de>

buildman: Print a message indicating the build is starting

Make it clear when buildman actually starts building. This happens when it
has prepared the threads, working directory and output directories.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add gcc 4.9.0 with Microblaze toolchain

Also read gcc 4.9.0 at kernel.org which also have Microblaze toolchain.

Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

Fixed unit test failure by updating the test:
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a space before the list of boards

Tweak the output slightly so we don't get things like:

- board1 board2+ board3 board4

There should be a space before the '+'.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add the option to download toolchains from kernel.org

The site at https://www.kernel.org/pub/tools/crosstool/ is a convenient
repository of toolchains which can be used for U-Boot. Add a feature to
download and install a toolchain for a selected architecture automatically.

It isn't clear how long this site will stay in the current place and
format, but we should be able to rely on bug reports if it changes.

Suggested-by: Marek Vašut <marex@denx.de>
Suggested-by: Fabio Estevam <festevam@gmail.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow architecture to alias to multiple toolchains

Some archs have need than one alias, so support a list of alises in the
..buildman file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Don't use the local settings when running tests

We should create a test setting file when running testes, not use whatever
happens to be on the local machine.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to flatten output directory trees

When building current source for a single board, buildman puts the output
in <output_dir>/current/current/<board>. Add an option to make it use
<output_dir>/<board> instead. This removes the unnecessary directories
in that case, controlled by the --no-subdirs/-N option.

Suggested-by: Tom Rini <trini@ti.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add tests that check the correct output directory is used

Add a few tests of the output directory logic.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Expand output test to cover directory prefixes

Now that buildman supports removing the build directory prefix from output,
add a test for it. Also ensure that output directories are removed when the
test completes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Enhance basic test to check summary output

Adjust the basic test so that it checks all console output. This will help
to ensure that the builder is behaving correctly with printing summary
information.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Introduce an 'and' operator for board selection

Currently buildman allows a list of boards to build to be specified on the
command line. The list can include specific board names, architecture, SOC
and so on.

At present the list of boards is dealt with in an 'OR' fashion, and there
is no way to specify something like 'arm & freescale', meaning boards with
ARM architecture but only those made by Freescale. This would exclude the
PowerPC boards made by Freescale.

Support an '&' operator on the command line to permit this. Ensure that
arguments can be specified in a single string to permit easy shell quoting.

Suggested-by: York Sun <yorksun@freescale.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
Acked-by: York Sun <yorksun@freescale.com>

buildman: Remove unused non-incremental build method code

The non-incremental build method is no longer used, so remove it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add verbose option to display errors as they happen

Normally buildman operates in two passes - one to do the build and another
to summarise the errors. Add a verbose option (-v) to display build problems
as they happen. With -e also given, this will display errors too.

When building the current source tree (rather than a list of commits in a
branch), both -v and -e are enabled automatically.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Refactor output options

We need the output options to be available in several places. It's a pain
to pass them into each function. Make them properties of the builder and
add a single function to set them up. At the same time, add a function which
produces summary output using these options.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Adjust tests for new boards.cfg format

Commit 27af930e9a5c91365ca639ada580b338eabe4989 changed the boards.cfg format
but missed to change the parsing in buildman. A follow-on commit
03c1bb242548e4e9d267d784861ccd69a1887aa0 fixed this but missed fixing the
tests.

This patch updates the tests to fit the new Board constructor.

./tools/buildman/buildman -t
<unittest.result.TestResult run=1 errors=0 failures=0>

Signed-off-by: Simon Glass <sjg@chromium.org>

Add GPL-2.0+ SPDX-License-Identifier to source files

Signed-off-by: Wolfgang Denk <wd@denx.de>
[trini: Fixup common/cmd_io.c]
Signed-off-by: Tom Rini <trini@ti.com>

buildman - U-Boot multi-threaded builder and summary tool

This tool handles building U-Boot to check that you have not broken it
with your patch series. It can build each individual commit and report
which boards fail on which commits, and which errors come up. It also
shows differences in image sizes due to particular commits.

Buildman aims to make full use of multi-processor machines.

Documentation and caveats are in tools/buildman/README.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in terminal.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Color() method to build()

This method has the same name as its class which is confusing. It is also
annoying when searching the code.

It builds a string with a colour, so rename it to build().

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add helper functions for updating .config files

At present the only straightforward way to write tests that need a
slightly different configuration is to create a new board with its own
configuration. This is cumbersome.

It would be useful if buildman could adjust the configuration of a build
on the fly. In preparation for this, add a utility library which can
modify a .config file according to various parameters passed to it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support single-threaded operation

At present even if only a single thread is in use, buildman still uses
threading.

For some debugging it is helpful to do everything in the main process.
Allow -T0 to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>

tools: Drop unnecessary use of __file__

There are few places where the path of the current modules is calculated
but not used. Drop them.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct the testOutputDir() unit test

This current fails with an error. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: 7664b03ffc5 ("buildman: Remove _of_#_ from results directory paths")

buildman: Remove _of_#_ from results directory paths

Currently, the following scenario will rebuild the first commit even
though it is not really necessary - the commit sha or the position in the
patchset did not change:

$ git am <local-patch-0001>
$ tools/buildman/buildman -P -E -W -b master mx6
<do some more development work>
$ git am <local-patch-0002>
$ tools/buildman/buildman -P -E -W -b master mx6 <- will rebuild the first
commit as well, even
though nothing has
changed about it.

This is due to the fact that previous results directories get removed
when the number of commits change. By removing the _of_#_ part of the
directory path, the commits will be rebuilt only if the commit sha or the
position in the patchset changes. Also, update the testcase to reflect this
change.

Signed-off-by: Ovidiu Panait <ovidiu.panait@windriver.com>

patman: Tidy up sys.path changes

Now that we are using absolute paths we can remove some of the sys.path
mangling that appears in the tools.

We only need to add the path to 'tools/' so that everything can find
modules relative to that directory.

The special paths for finding pylibfdt remain.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix test for new 9.2 kernel

The naming is slightly different on kernel.org now. Update the regex so
that the test still passes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore migration warnings

These are becoming more common now. They cause boards to show warnings
which can be mistaking for compiler warnings.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to ignore device-tree warnings

Unfortunately the plague of device-tree warnings has not lifted. These
warnings infiltrate almost every build, adding noise and confusion.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show a summary of the build result

When buildman finishes it leaves the last summary line visible, which
shows the number of successful builds, builds with warnings and builds
with errors.

It is useful also to see how many builds were done in total along with
the time taken. Show these on a separate line before buildman finishes.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use spaces in the board list

At present the board names shown with -l are separated by commas. This
makes it hard to double-click to select a particular board. Also it is not
possible to select all boards and paste them as arguments to a subsequent
buildman run, since buildman requires spaces to separate the list on the
command line, not commas.

Change the output format to use spaces instead of commas.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Show the list of boards in magenta

It is quite hard to see the list of board for each error line since the
colour is the same as the actual error line. Show the board list in
magenta so that it is easier to distinguish them.

There is no point in checking the colour of the overall line, since there
are now multiple colours. So drop those tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use yellow consistently for warning lines

At present warnings are shown in yellow in the summary (-s) but magenta in
the detail listing (-e). Use yellow in both.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Test the output with --list-error-boards

Add a test to cover this flag, which adds the name of each board to each
error/warning line.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test helper for creating a line prefix

The split/join code is repeated in a lot of places. Add a function to
handle this.

Signed-off-by: Simon Glass <sjg@chromium.org>