CI: Update to gcc-13.2.0

The latest kernel.org toolchains for gcc are now 13.2.0, so upgrade to
that.

Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: Convert camel case in bsettings.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

CI: Update to gcc-13.1.0

As this is the current version of the public cross toolchains we use,
upgrade to this now.

Suggested-by: Alexey Brodkin <Alexey.Brodkin@synopsys.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
Acked-by: Alexey Brodkin <abrodkin@synopsys.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

Merge branch 'next'

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: Correct overwriting of settings file

The toolchain test causes the settings file to be overwritten, which is
annoying for local development. Fix it by passing None as the filename.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix CROSS_COMPILE output for sandbox

The previous attempt at fixing this broke the normal usage of the -A
flag.

At present, 'buildman -A sandbox' adds the path containing the
toolchain. We can assume that this is in the path and we don't want to
set CROSS_COMPILE=/bin/

Change this to align with what MakeEnvironment() does, but only for
sandbox boards.

Signed-off-by: Simon Glass <sjg@chromium.org>

Revert "buildman: Correct CROSS_COMPILE output for sandbox"

This reverts commit bd0a548ad4a155fec29473d4cc8e135832926973.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct CROSS_COMPILE output for sandbox

At present, 'buildman -A sandbox' adds the path containing the
toolchain at present. We can assume that this is in the path and
we don't want to set CROSS_COMPILE=/bin/ so change this to align
with what MakeEnvironment() does.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move library functions into a library directory

The patman directory has a number of modules which are used by other tools
in U-Boot. This makes it hard to package the tools using pypi since the
common files must be copied along with the tool that uses them.

To address this, move these files into a new u_boot_pylib library. This
can be packaged separately and listed as a dependency of each tool.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: invalid reference to README

The readme file for buildman is called buildman.rst.

Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Fetch 12.2.0 toolchains by default

Update the toolchain list to be first 12.2.0 and second 11.1.0 and
that's it.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Color() method to build()

This method has the same name as its class which is confusing. It is also
annoying when searching the code.

It builds a string with a colour, so rename it to build().

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in command.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Tidy up the download function a little

Reverse the order of the return tuple, so that the filename is first.
This seems more obvious than putting the temporary directory first.

Correct a bug that leaves a space on the final line.

Allow the caller to control the name of the temporary directory.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move the download function to tools

This function is handy for binman as well. Move it into the shared 'tools'
module.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add gcc-11.1.0 to the directory list

While CI has been using gcc-11.1.0 for a long time, we have not updated
buildman to match. Correct this omission.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: Use bytes for the environment

At present we sometimes see problems in gitlab where the environment has
0x80 characters or sequences which are not valid UTF-8.

Avoid this by using bytes for the environment, both internal to buildman
and when writing out the 'env' file. Add a test to make sure this works
as expected.

Reported-by: Marek Vasut <marex@denx.de>
Fixes: e5fc79ea718 ("buildman: Write the environment out to an 'env' file")
Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support fetching gcc 9.2.0

This adds support to fetch gcc 9.2.0 toolchains.

Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Enable buildman on aarch64 hosts

At kernel.org aarch64 toolchains are published in folder
arm64. Fix the URL for that case, so that we can fetch
toolchains on aarch64 machines.

Signed-off-by: Matthias Brugger <mbrugger@suse.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Add options to get the arch and toolchain info

Sometimes it is useful for external tools to use buildman to provide the
toolchain information. Add an -a option which shows the value to use for
the ARCH environment variable, and -A which does the same for
CROSS_COMPILE

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Improve [make-flags] section parser to allow quoted strings

The parser responsible for the '[make-flags]' section in
the '.buildman' settings file is currently not able to
handle quoted strings, as given in the sample bellow:

[make-flags]
qemu_arm=HOSTCC="cc -isystem /add/include" HOSTLDFLAGS="-L/add/lib"

This patch replaces the simple string splitter based on the <space>
delimiter with a regex tokenizer that preserves spaces inside double
quoted strings.

Signed-off-by: Cristian Ciocaltea <cristian.ciocaltea@gmail.com>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix tabs in GetWrapper()

This function has tabs instead of spaces. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add support for building with clang

Add a -O option which allows building with clang.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman/toolchain.py: handle inconsistent tarball names

Unfortunately, for some releases the kernel.org toolchain tarball names adhere
to the following pattern:

<hostarch>-gcc-<ver>-nolib-<targetarch>-<type>.tar.xz

e.g.:
x86_64-gcc-8.1.0-nolibc-aarch64-linux.tar.xz

while others use the following pattern:

<hostarch>-gcc-<ver>-nolib_<targetarch>-<type>.tar.xz

e.g.:

x86_64-gcc-7.3.0-nolibc_aarch64-linux.tar.xz

Notice that the first pattern has dashes throughout, while the second has
dashes throughout except just before the target architecture which has an
underscore.

The "dash throughout" versions from kernel.org are:

8.1.0, 6.4.0, 5.5.0, 4.9.4, 4.8.5, 4.6.1

while the "dash and underscore" versions from kernel.org are:

7.3.0, 4.9.0, 4.8.0, 4.7.3, 4.6.3, 4.6.2, 4.5.1, 4.2.4

This tweak allows the code to handle both versions. Note that this tweak also
causes the architecture parsing to get confused and find the following two
bogus architectures, "2.0" and "64", which are explicitly checked for, and
removed.

Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Change single quotes to double quotes:
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman/toolchain.py: fix toolchain directory

The hexagon toolchain (4.6.1) from kernel.org, for example, was packaged in
a way that is different from most toolchains. The first entry when unpacking
most toolchain tarballs is:

gcc-<version>-nolib/<targetarch>-<system>

e.g.:

gcc-8.1.0-nolibc/aarch64-linux/

The first entry of the hexagon toolchain, however, is:

gcc-4.6.1-nolibc/

This causes the buildman logic in toolchain.py::ScanPath() to not be able to
find the "*gcc" executable since it looks in gcc-4.6.1-nolib/{.|bin|usr/bin}
instead of gcc-4.6.1/hexagon-linux/{.|bin|usr/bin}. Therefore when buildman
tries to download a set of toolchains that includes hexagon, the script fails.

This update takes the second line of the tarball unpacking (which works for
all the toolchains I've tested from kernel.org) and parses it to take the
first two elements, separated by '/'. It makes this logic a bit more robust.

Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: support newer gcc versions from kernel.org

Add support for gcc versions 7.3.0, 6.4.0 and 4.9.4.

Also use a regex for matching the tarball names. Some gcc versions
use '-ARCH-' instead of '_ARCH-'.

As part of this, we switch TravisCI to also using these toolchains for
all platforms.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
Signed-off-by: Tom Rini <trini@konsulko.com>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: disable localized and unicode output of all build tools

Build tools like Make, gcc or binutils support localized output
or unicode encoded output dependent on the default system locale.
This is not useful for buildman, where we want reproducible
warning or error messages or where the output of binutils is
further processed.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>

buildman: Allow 'gnueabihf' toolchains for ARM

Many toolchains for ARM use the 'gnueabihf' suffix rather than just
'gnueabi', so allow these to be used, but with a lower priority than
'gnueabi' ones.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

tools: buildman: Add compiler wrapper

Now we can use compiler wrapper such as ccache or distcc for buildman.

Signed-off-by: York Sun <york.sun@nxp.com>
CC: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: Avoid overwriting existing toolchain entries

The current code for setting up the toolchain config always writes the new
paths to an item called 'toolchain'. This means that it will overwrite any
existing toolchain item with the same name. In practice, this means that:

buildman --fetch-arch all

will fetch all toolchains, but only the path of the final one will be added
to the config. This normally works out OK, since most toolchains are the
same version (e.g. gcc 4.9) and will be found on the same path. But it is
not correct and toolchains for archs which don't use the same version will
not function as expected.

Adjust the code to use a complete glob of the toolchain path.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Fix a typo in TestSettingsHasPath()

The function comment should say 'buildman'. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Improve the toolchain progress/error output

Use colour to make it easier to see what is going on. Also print a message
before downloading a new toolchain. Mention --fetch-arch in the message that
is shown when there are no available toolchains, since this is the quickest
way to resolve the problem.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Allow the toolchain error to be suppressed

When there are no toolchains a warning is printed. But in some cases this is
confusing, such as when the user is fetching new toolchains.

Adjust the function to supress the warning in this case.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a way to specific a full toolchain prefix

At present buildman allows you to specify the directory containing the
toolchain, but not the actual toolchain prefix. If there are multiple
toolchains in a single directory, this can be inconvenient.

Add a new 'toolchain-prefix' setting to the settings file, which allows
the full prefix (or path to the C compiler) to be specified.

Update the documentation to match.

Suggested-by: Stephen Warren <swarren@wwwdotorg.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>

buildman: Allow the toolchain architecture to be specified

At present the architecture is deduced from the toolchain filename. Allow it
to be specified by the caller.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com

buildman: Allow the toolchain priority to be specified

At present the priority of a toolchain is calculated from its filename based
on hard-coded rules. Allow it to be specified by the caller. We will use
this in a later patch. Also display the priority and provide a message when
it is overriden by another toolchain of higher priority.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

buildman: Fix up a few code inconsistencies in toolchain.py

Normally we use a single quote for strings unless there is a reason not to
(such as an embedded single quote). Fix a few counter-examples in this file.
Also add a missing function-argument comment.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

buildman: Add gcc 4.9.0 with Microblaze toolchain

Also read gcc 4.9.0 at kernel.org which also have Microblaze toolchain.

Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

Fixed unit test failure by updating the test:
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct toolchain download feature

Commit d908898 updated the ScanPath() function but not its documentation
and not all its callers.

This breaks the toolchain check after it is downloaded. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Acked-by: Heiko Schocher <hs@denx.de>

buildman: allow multiple toolchains in a single path

When buildman scans a toolchain path, it stops at the
first toolchain found. However, a single path can contains
several toolchains, each with its own prefix.

This patch lets buildman scan all toolchains in the path.

Signed-off-by: Albert ARIBAUD <albert.u.boot@aribaud.net>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: Add the option to download toolchains from kernel.org

The site at https://www.kernel.org/pub/tools/crosstool/ is a convenient
repository of toolchains which can be used for U-Boot. Add a feature to
download and install a toolchain for a selected architecture automatically.

It isn't clear how long this site will stay in the current place and
format, but we should be able to rely on bug reports if it changes.

Suggested-by: Marek Vašut <marex@denx.de>
Suggested-by: Fabio Estevam <festevam@gmail.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow architecture to alias to multiple toolchains

Some archs have need than one alias, so support a list of alises in the
..buildman file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to use the full tool chain path

In some cases there may be multiple toolchains with the same name in the
path. Provide an option to use the full path in the CROSS_COMPILE
environment variable.

Note: Wolfgang mentioned that this is dangerous since in some cases there
may be other tools on the path that are needed. So this is set up as an
option, not the default. I will need test confirmation (i.e. that this
commit fixes a real problem) before merging it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Steve Rae <srae@broadcom.com>

buildman: Put the toolchain path first instead of last in PATH

If:

1. Toolchains A and B have the same filename
2. Toolchain A is in the PATH
3. Toolchain B is given in ~/.buildman and buildman uses it to build

then buildman will add toolchain B to the end of its path but will not
necessarily use it since U-Boot will find toolchain A first in the PATH.

Try to fix this by putting the toolchain first in the path instead of
last.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Try to avoid hard-coded string parsing

The assumption that the compiler name will always end in gcc is incorrect
for clang and apparently on BSD.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a functional test

Buildman currently lacks testing in many areas, including its use of git,
make and many command-line flags.

Add a functional test which covers some of these areas. So far it does
a fake 'build' of all boards for the current source tree.

This version reads the real ~/.buildman and boards.cfg files. Future work
will improve this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow make-flags variables to include '-' and '_'

These characters are commonly used in variables, so permit them. Also
document the permitted characters.

Reported-by: Tom Rini <trini@ti.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: fix toolchain priority_list

'-elf' appears twice in the toolchain priority_list.
The second one is rudundant.

Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Cc: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: fix to display warning message for missing [toolchain] section

Toolchains.__init__ is expected to display a warning message
when the [toolchain] section is missing from ~/.buildman file.
But it never works.
In that case, instead, buildmain fails with an error message
which is difficult to understand:

Traceback (most recent call last):
File "tools/buildman/buildman", line 126, in <module>
control.DoBuildman(options, args)
File "/home/foo/u-boot/tools/buildman/control.py", line 78, in DoBuildman
toolchains = toolchain.Toolchains()
File "/home/foo/u-boot/tools/buildman/toolchain.py", line 106, in __init__
config_fname)
NameError: global name 'config_fname' is not defined

Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Cc: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: don't fail --list-toolchains when toolchains fail

When a toolchain invocation fails, an exception is thrown but not caught
which then aborts the entire toolchain detection process. To solve this,
request that exceptions not be thrown, since the toolchain init code
already error-checks the command result. This solves e.g.:

- found '/usr/bin/winegcc'
Traceback (most recent call last):
...
Exception: Error running '/usr/bin/winegcc --version'

Change-Id: I579c72ab3b021e38b14132893c3375ea257c74f0
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Simon Glass <sjg@chromium.org>
(formatted to 80cols)

buildman: Allow make flags to be specified for each board

There are a few make options such as BUILD_TAG which can be provided when
building U-Boot. Provide a way for buildman to pass these flags to make
also.

The flags should be in a [make-flags] section and arranged by target name
(the 'target' column in boards.cfg. See the README for more details.

Signed-off-by: Simon Glass <sjg@chromium.org>

Add GPL-2.0+ SPDX-License-Identifier to source files

Signed-off-by: Wolfgang Denk <wd@denx.de>
[trini: Fixup common/cmd_io.c]
Signed-off-by: Tom Rini <trini@ti.com>

buildman - U-Boot multi-threaded builder and summary tool

This tool handles building U-Boot to check that you have not broken it
with your patch series. It can build each individual commit and report
which boards fail on which commits, and which errors come up. It also
shows differences in image sizes due to particular commits.

Buildman aims to make full use of multi-processor machines.

Documentation and caveats are in tools/buildman/README.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in bsettings.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

CI: Update to gcc-13.1.0

As this is the current version of the public cross toolchains we use,
upgrade to this now.

Suggested-by: Alexey Brodkin <Alexey.Brodkin@synopsys.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
Acked-by: Alexey Brodkin <abrodkin@synopsys.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

Merge branch 'next'

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: Correct overwriting of settings file

The toolchain test causes the settings file to be overwritten, which is
annoying for local development. Fix it by passing None as the filename.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix CROSS_COMPILE output for sandbox

The previous attempt at fixing this broke the normal usage of the -A
flag.

At present, 'buildman -A sandbox' adds the path containing the
toolchain. We can assume that this is in the path and we don't want to
set CROSS_COMPILE=/bin/

Change this to align with what MakeEnvironment() does, but only for
sandbox boards.

Signed-off-by: Simon Glass <sjg@chromium.org>

Revert "buildman: Correct CROSS_COMPILE output for sandbox"

This reverts commit bd0a548ad4a155fec29473d4cc8e135832926973.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct CROSS_COMPILE output for sandbox

At present, 'buildman -A sandbox' adds the path containing the
toolchain at present. We can assume that this is in the path and
we don't want to set CROSS_COMPILE=/bin/ so change this to align
with what MakeEnvironment() does.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move library functions into a library directory

The patman directory has a number of modules which are used by other tools
in U-Boot. This makes it hard to package the tools using pypi since the
common files must be copied along with the tool that uses them.

To address this, move these files into a new u_boot_pylib library. This
can be packaged separately and listed as a dependency of each tool.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: invalid reference to README

The readme file for buildman is called buildman.rst.

Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Fetch 12.2.0 toolchains by default

Update the toolchain list to be first 12.2.0 and second 11.1.0 and
that's it.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Color() method to build()

This method has the same name as its class which is confusing. It is also
annoying when searching the code.

It builds a string with a colour, so rename it to build().

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in command.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Tidy up the download function a little

Reverse the order of the return tuple, so that the filename is first.
This seems more obvious than putting the temporary directory first.

Correct a bug that leaves a space on the final line.

Allow the caller to control the name of the temporary directory.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move the download function to tools

This function is handy for binman as well. Move it into the shared 'tools'
module.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add gcc-11.1.0 to the directory list

While CI has been using gcc-11.1.0 for a long time, we have not updated
buildman to match. Correct this omission.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: Use bytes for the environment

At present we sometimes see problems in gitlab where the environment has
0x80 characters or sequences which are not valid UTF-8.

Avoid this by using bytes for the environment, both internal to buildman
and when writing out the 'env' file. Add a test to make sure this works
as expected.

Reported-by: Marek Vasut <marex@denx.de>
Fixes: e5fc79ea718 ("buildman: Write the environment out to an 'env' file")
Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support fetching gcc 9.2.0

This adds support to fetch gcc 9.2.0 toolchains.

Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Enable buildman on aarch64 hosts

At kernel.org aarch64 toolchains are published in folder
arm64. Fix the URL for that case, so that we can fetch
toolchains on aarch64 machines.

Signed-off-by: Matthias Brugger <mbrugger@suse.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Add options to get the arch and toolchain info

Sometimes it is useful for external tools to use buildman to provide the
toolchain information. Add an -a option which shows the value to use for
the ARCH environment variable, and -A which does the same for
CROSS_COMPILE

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Improve [make-flags] section parser to allow quoted strings

The parser responsible for the '[make-flags]' section in
the '.buildman' settings file is currently not able to
handle quoted strings, as given in the sample bellow:

[make-flags]
qemu_arm=HOSTCC="cc -isystem /add/include" HOSTLDFLAGS="-L/add/lib"

This patch replaces the simple string splitter based on the <space>
delimiter with a regex tokenizer that preserves spaces inside double
quoted strings.

Signed-off-by: Cristian Ciocaltea <cristian.ciocaltea@gmail.com>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix tabs in GetWrapper()

This function has tabs instead of spaces. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add support for building with clang

Add a -O option which allows building with clang.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman/toolchain.py: handle inconsistent tarball names

Unfortunately, for some releases the kernel.org toolchain tarball names adhere
to the following pattern:

<hostarch>-gcc-<ver>-nolib-<targetarch>-<type>.tar.xz

e.g.:
x86_64-gcc-8.1.0-nolibc-aarch64-linux.tar.xz

while others use the following pattern:

<hostarch>-gcc-<ver>-nolib_<targetarch>-<type>.tar.xz

e.g.:

x86_64-gcc-7.3.0-nolibc_aarch64-linux.tar.xz

Notice that the first pattern has dashes throughout, while the second has
dashes throughout except just before the target architecture which has an
underscore.

The "dash throughout" versions from kernel.org are:

8.1.0, 6.4.0, 5.5.0, 4.9.4, 4.8.5, 4.6.1

while the "dash and underscore" versions from kernel.org are:

7.3.0, 4.9.0, 4.8.0, 4.7.3, 4.6.3, 4.6.2, 4.5.1, 4.2.4

This tweak allows the code to handle both versions. Note that this tweak also
causes the architecture parsing to get confused and find the following two
bogus architectures, "2.0" and "64", which are explicitly checked for, and
removed.

Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Change single quotes to double quotes:
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman/toolchain.py: fix toolchain directory

The hexagon toolchain (4.6.1) from kernel.org, for example, was packaged in
a way that is different from most toolchains. The first entry when unpacking
most toolchain tarballs is:

gcc-<version>-nolib/<targetarch>-<system>

e.g.:

gcc-8.1.0-nolibc/aarch64-linux/

The first entry of the hexagon toolchain, however, is:

gcc-4.6.1-nolibc/

This causes the buildman logic in toolchain.py::ScanPath() to not be able to
find the "*gcc" executable since it looks in gcc-4.6.1-nolib/{.|bin|usr/bin}
instead of gcc-4.6.1/hexagon-linux/{.|bin|usr/bin}. Therefore when buildman
tries to download a set of toolchains that includes hexagon, the script fails.

This update takes the second line of the tarball unpacking (which works for
all the toolchains I've tested from kernel.org) and parses it to take the
first two elements, separated by '/'. It makes this logic a bit more robust.

Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: support newer gcc versions from kernel.org

Add support for gcc versions 7.3.0, 6.4.0 and 4.9.4.

Also use a regex for matching the tarball names. Some gcc versions
use '-ARCH-' instead of '_ARCH-'.

As part of this, we switch TravisCI to also using these toolchains for
all platforms.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
Signed-off-by: Tom Rini <trini@konsulko.com>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: disable localized and unicode output of all build tools

Build tools like Make, gcc or binutils support localized output
or unicode encoded output dependent on the default system locale.
This is not useful for buildman, where we want reproducible
warning or error messages or where the output of binutils is
further processed.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>

buildman: Allow 'gnueabihf' toolchains for ARM

Many toolchains for ARM use the 'gnueabihf' suffix rather than just
'gnueabi', so allow these to be used, but with a lower priority than
'gnueabi' ones.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

tools: buildman: Add compiler wrapper

Now we can use compiler wrapper such as ccache or distcc for buildman.

Signed-off-by: York Sun <york.sun@nxp.com>
CC: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: Avoid overwriting existing toolchain entries

The current code for setting up the toolchain config always writes the new
paths to an item called 'toolchain'. This means that it will overwrite any
existing toolchain item with the same name. In practice, this means that:

buildman --fetch-arch all

will fetch all toolchains, but only the path of the final one will be added
to the config. This normally works out OK, since most toolchains are the
same version (e.g. gcc 4.9) and will be found on the same path. But it is
not correct and toolchains for archs which don't use the same version will
not function as expected.

Adjust the code to use a complete glob of the toolchain path.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Fix a typo in TestSettingsHasPath()

The function comment should say 'buildman'. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Improve the toolchain progress/error output

Use colour to make it easier to see what is going on. Also print a message
before downloading a new toolchain. Mention --fetch-arch in the message that
is shown when there are no available toolchains, since this is the quickest
way to resolve the problem.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Allow the toolchain error to be suppressed

When there are no toolchains a warning is printed. But in some cases this is
confusing, such as when the user is fetching new toolchains.

Adjust the function to supress the warning in this case.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a way to specific a full toolchain prefix

At present buildman allows you to specify the directory containing the
toolchain, but not the actual toolchain prefix. If there are multiple
toolchains in a single directory, this can be inconvenient.

Add a new 'toolchain-prefix' setting to the settings file, which allows
the full prefix (or path to the C compiler) to be specified.

Update the documentation to match.

Suggested-by: Stephen Warren <swarren@wwwdotorg.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>

buildman: Allow the toolchain architecture to be specified

At present the architecture is deduced from the toolchain filename. Allow it
to be specified by the caller.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com

buildman: Allow the toolchain priority to be specified

At present the priority of a toolchain is calculated from its filename based
on hard-coded rules. Allow it to be specified by the caller. We will use
this in a later patch. Also display the priority and provide a message when
it is overriden by another toolchain of higher priority.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

buildman: Fix up a few code inconsistencies in toolchain.py

Normally we use a single quote for strings unless there is a reason not to
(such as an embedded single quote). Fix a few counter-examples in this file.
Also add a missing function-argument comment.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

buildman: Add gcc 4.9.0 with Microblaze toolchain

Also read gcc 4.9.0 at kernel.org which also have Microblaze toolchain.

Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

Fixed unit test failure by updating the test:
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct toolchain download feature

Commit d908898 updated the ScanPath() function but not its documentation
and not all its callers.

This breaks the toolchain check after it is downloaded. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Acked-by: Heiko Schocher <hs@denx.de>

buildman: allow multiple toolchains in a single path

When buildman scans a toolchain path, it stops at the
first toolchain found. However, a single path can contains
several toolchains, each with its own prefix.

This patch lets buildman scan all toolchains in the path.

Signed-off-by: Albert ARIBAUD <albert.u.boot@aribaud.net>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: Add the option to download toolchains from kernel.org

The site at https://www.kernel.org/pub/tools/crosstool/ is a convenient
repository of toolchains which can be used for U-Boot. Add a feature to
download and install a toolchain for a selected architecture automatically.

It isn't clear how long this site will stay in the current place and
format, but we should be able to rely on bug reports if it changes.

Suggested-by: Marek Vašut <marex@denx.de>
Suggested-by: Fabio Estevam <festevam@gmail.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow architecture to alias to multiple toolchains

Some archs have need than one alias, so support a list of alises in the
..buildman file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to use the full tool chain path

In some cases there may be multiple toolchains with the same name in the
path. Provide an option to use the full path in the CROSS_COMPILE
environment variable.

Note: Wolfgang mentioned that this is dangerous since in some cases there
may be other tools on the path that are needed. So this is set up as an
option, not the default. I will need test confirmation (i.e. that this
commit fixes a real problem) before merging it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Steve Rae <srae@broadcom.com>

buildman: Put the toolchain path first instead of last in PATH

If:

1. Toolchains A and B have the same filename
2. Toolchain A is in the PATH
3. Toolchain B is given in ~/.buildman and buildman uses it to build

then buildman will add toolchain B to the end of its path but will not
necessarily use it since U-Boot will find toolchain A first in the PATH.

Try to fix this by putting the toolchain first in the path instead of
last.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Try to avoid hard-coded string parsing

The assumption that the compiler name will always end in gcc is incorrect
for clang and apparently on BSD.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a functional test

Buildman currently lacks testing in many areas, including its use of git,
make and many command-line flags.

Add a functional test which covers some of these areas. So far it does
a fake 'build' of all boards for the current source tree.

This version reads the real ~/.buildman and boards.cfg files. Future work
will improve this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow make-flags variables to include '-' and '_'

These characters are commonly used in variables, so permit them. Also
document the permitted characters.

Reported-by: Tom Rini <trini@ti.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: fix toolchain priority_list

'-elf' appears twice in the toolchain priority_list.
The second one is rudundant.

Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Cc: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: fix to display warning message for missing [toolchain] section

Toolchains.__init__ is expected to display a warning message
when the [toolchain] section is missing from ~/.buildman file.
But it never works.
In that case, instead, buildmain fails with an error message
which is difficult to understand:

Traceback (most recent call last):
File "tools/buildman/buildman", line 126, in <module>
control.DoBuildman(options, args)
File "/home/foo/u-boot/tools/buildman/control.py", line 78, in DoBuildman
toolchains = toolchain.Toolchains()
File "/home/foo/u-boot/tools/buildman/toolchain.py", line 106, in __init__
config_fname)
NameError: global name 'config_fname' is not defined

Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Cc: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: don't fail --list-toolchains when toolchains fail

When a toolchain invocation fails, an exception is thrown but not caught
which then aborts the entire toolchain detection process. To solve this,
request that exceptions not be thrown, since the toolchain init code
already error-checks the command result. This solves e.g.:

- found '/usr/bin/winegcc'
Traceback (most recent call last):
...
Exception: Error running '/usr/bin/winegcc --version'

Change-Id: I579c72ab3b021e38b14132893c3375ea257c74f0
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Simon Glass <sjg@chromium.org>
(formatted to 80cols)

buildman: Allow make flags to be specified for each board

There are a few make options such as BUILD_TAG which can be provided when
building U-Boot. Provide a way for buildman to pass these flags to make
also.

The flags should be in a [make-flags] section and arranged by target name
(the 'target' column in boards.cfg. See the README for more details.

Signed-off-by: Simon Glass <sjg@chromium.org>

Add GPL-2.0+ SPDX-License-Identifier to source files

Signed-off-by: Wolfgang Denk <wd@denx.de>
[trini: Fixup common/cmd_io.c]
Signed-off-by: Tom Rini <trini@ti.com>

buildman - U-Boot multi-threaded builder and summary tool

This tool handles building U-Boot to check that you have not broken it
with your patch series. It can build each individual commit and report
which boards fail on which commits, and which errors come up. It also
shows differences in image sizes due to particular commits.

Buildman aims to make full use of multi-processor machines.

Documentation and caveats are in tools/buildman/README.

Signed-off-by: Simon Glass <sjg@chromium.org>

CI: Update to gcc-13.1.0

As this is the current version of the public cross toolchains we use,
upgrade to this now.

Suggested-by: Alexey Brodkin <Alexey.Brodkin@synopsys.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
Acked-by: Alexey Brodkin <abrodkin@synopsys.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

Merge branch 'next'

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: Correct overwriting of settings file

The toolchain test causes the settings file to be overwritten, which is
annoying for local development. Fix it by passing None as the filename.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix CROSS_COMPILE output for sandbox

The previous attempt at fixing this broke the normal usage of the -A
flag.

At present, 'buildman -A sandbox' adds the path containing the
toolchain. We can assume that this is in the path and we don't want to
set CROSS_COMPILE=/bin/

Change this to align with what MakeEnvironment() does, but only for
sandbox boards.

Signed-off-by: Simon Glass <sjg@chromium.org>

Revert "buildman: Correct CROSS_COMPILE output for sandbox"

This reverts commit bd0a548ad4a155fec29473d4cc8e135832926973.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct CROSS_COMPILE output for sandbox

At present, 'buildman -A sandbox' adds the path containing the
toolchain at present. We can assume that this is in the path and
we don't want to set CROSS_COMPILE=/bin/ so change this to align
with what MakeEnvironment() does.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move library functions into a library directory

The patman directory has a number of modules which are used by other tools
in U-Boot. This makes it hard to package the tools using pypi since the
common files must be copied along with the tool that uses them.

To address this, move these files into a new u_boot_pylib library. This
can be packaged separately and listed as a dependency of each tool.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: invalid reference to README

The readme file for buildman is called buildman.rst.

Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Fetch 12.2.0 toolchains by default

Update the toolchain list to be first 12.2.0 and second 11.1.0 and
that's it.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Color() method to build()

This method has the same name as its class which is confusing. It is also
annoying when searching the code.

It builds a string with a colour, so rename it to build().

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in command.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Tidy up the download function a little

Reverse the order of the return tuple, so that the filename is first.
This seems more obvious than putting the temporary directory first.

Correct a bug that leaves a space on the final line.

Allow the caller to control the name of the temporary directory.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move the download function to tools

This function is handy for binman as well. Move it into the shared 'tools'
module.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add gcc-11.1.0 to the directory list

While CI has been using gcc-11.1.0 for a long time, we have not updated
buildman to match. Correct this omission.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: Use bytes for the environment

At present we sometimes see problems in gitlab where the environment has
0x80 characters or sequences which are not valid UTF-8.

Avoid this by using bytes for the environment, both internal to buildman
and when writing out the 'env' file. Add a test to make sure this works
as expected.

Reported-by: Marek Vasut <marex@denx.de>
Fixes: e5fc79ea718 ("buildman: Write the environment out to an 'env' file")
Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support fetching gcc 9.2.0

This adds support to fetch gcc 9.2.0 toolchains.

Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Enable buildman on aarch64 hosts

At kernel.org aarch64 toolchains are published in folder
arm64. Fix the URL for that case, so that we can fetch
toolchains on aarch64 machines.

Signed-off-by: Matthias Brugger <mbrugger@suse.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Add options to get the arch and toolchain info

Sometimes it is useful for external tools to use buildman to provide the
toolchain information. Add an -a option which shows the value to use for
the ARCH environment variable, and -A which does the same for
CROSS_COMPILE

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Improve [make-flags] section parser to allow quoted strings

The parser responsible for the '[make-flags]' section in
the '.buildman' settings file is currently not able to
handle quoted strings, as given in the sample bellow:

[make-flags]
qemu_arm=HOSTCC="cc -isystem /add/include" HOSTLDFLAGS="-L/add/lib"

This patch replaces the simple string splitter based on the <space>
delimiter with a regex tokenizer that preserves spaces inside double
quoted strings.

Signed-off-by: Cristian Ciocaltea <cristian.ciocaltea@gmail.com>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix tabs in GetWrapper()

This function has tabs instead of spaces. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add support for building with clang

Add a -O option which allows building with clang.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman/toolchain.py: handle inconsistent tarball names

Unfortunately, for some releases the kernel.org toolchain tarball names adhere
to the following pattern:

<hostarch>-gcc-<ver>-nolib-<targetarch>-<type>.tar.xz

e.g.:
x86_64-gcc-8.1.0-nolibc-aarch64-linux.tar.xz

while others use the following pattern:

<hostarch>-gcc-<ver>-nolib_<targetarch>-<type>.tar.xz

e.g.:

x86_64-gcc-7.3.0-nolibc_aarch64-linux.tar.xz

Notice that the first pattern has dashes throughout, while the second has
dashes throughout except just before the target architecture which has an
underscore.

The "dash throughout" versions from kernel.org are:

8.1.0, 6.4.0, 5.5.0, 4.9.4, 4.8.5, 4.6.1

while the "dash and underscore" versions from kernel.org are:

7.3.0, 4.9.0, 4.8.0, 4.7.3, 4.6.3, 4.6.2, 4.5.1, 4.2.4

This tweak allows the code to handle both versions. Note that this tweak also
causes the architecture parsing to get confused and find the following two
bogus architectures, "2.0" and "64", which are explicitly checked for, and
removed.

Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Change single quotes to double quotes:
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman/toolchain.py: fix toolchain directory

The hexagon toolchain (4.6.1) from kernel.org, for example, was packaged in
a way that is different from most toolchains. The first entry when unpacking
most toolchain tarballs is:

gcc-<version>-nolib/<targetarch>-<system>

e.g.:

gcc-8.1.0-nolibc/aarch64-linux/

The first entry of the hexagon toolchain, however, is:

gcc-4.6.1-nolibc/

This causes the buildman logic in toolchain.py::ScanPath() to not be able to
find the "*gcc" executable since it looks in gcc-4.6.1-nolib/{.|bin|usr/bin}
instead of gcc-4.6.1/hexagon-linux/{.|bin|usr/bin}. Therefore when buildman
tries to download a set of toolchains that includes hexagon, the script fails.

This update takes the second line of the tarball unpacking (which works for
all the toolchains I've tested from kernel.org) and parses it to take the
first two elements, separated by '/'. It makes this logic a bit more robust.

Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: support newer gcc versions from kernel.org

Add support for gcc versions 7.3.0, 6.4.0 and 4.9.4.

Also use a regex for matching the tarball names. Some gcc versions
use '-ARCH-' instead of '_ARCH-'.

As part of this, we switch TravisCI to also using these toolchains for
all platforms.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
Signed-off-by: Tom Rini <trini@konsulko.com>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: disable localized and unicode output of all build tools

Build tools like Make, gcc or binutils support localized output
or unicode encoded output dependent on the default system locale.
This is not useful for buildman, where we want reproducible
warning or error messages or where the output of binutils is
further processed.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>

buildman: Allow 'gnueabihf' toolchains for ARM

Many toolchains for ARM use the 'gnueabihf' suffix rather than just
'gnueabi', so allow these to be used, but with a lower priority than
'gnueabi' ones.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

tools: buildman: Add compiler wrapper

Now we can use compiler wrapper such as ccache or distcc for buildman.

Signed-off-by: York Sun <york.sun@nxp.com>
CC: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: Avoid overwriting existing toolchain entries

The current code for setting up the toolchain config always writes the new
paths to an item called 'toolchain'. This means that it will overwrite any
existing toolchain item with the same name. In practice, this means that:

buildman --fetch-arch all

will fetch all toolchains, but only the path of the final one will be added
to the config. This normally works out OK, since most toolchains are the
same version (e.g. gcc 4.9) and will be found on the same path. But it is
not correct and toolchains for archs which don't use the same version will
not function as expected.

Adjust the code to use a complete glob of the toolchain path.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Fix a typo in TestSettingsHasPath()

The function comment should say 'buildman'. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Improve the toolchain progress/error output

Use colour to make it easier to see what is going on. Also print a message
before downloading a new toolchain. Mention --fetch-arch in the message that
is shown when there are no available toolchains, since this is the quickest
way to resolve the problem.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Allow the toolchain error to be suppressed

When there are no toolchains a warning is printed. But in some cases this is
confusing, such as when the user is fetching new toolchains.

Adjust the function to supress the warning in this case.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a way to specific a full toolchain prefix

At present buildman allows you to specify the directory containing the
toolchain, but not the actual toolchain prefix. If there are multiple
toolchains in a single directory, this can be inconvenient.

Add a new 'toolchain-prefix' setting to the settings file, which allows
the full prefix (or path to the C compiler) to be specified.

Update the documentation to match.

Suggested-by: Stephen Warren <swarren@wwwdotorg.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>

buildman: Allow the toolchain architecture to be specified

At present the architecture is deduced from the toolchain filename. Allow it
to be specified by the caller.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com

buildman: Allow the toolchain priority to be specified

At present the priority of a toolchain is calculated from its filename based
on hard-coded rules. Allow it to be specified by the caller. We will use
this in a later patch. Also display the priority and provide a message when
it is overriden by another toolchain of higher priority.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

buildman: Fix up a few code inconsistencies in toolchain.py

Normally we use a single quote for strings unless there is a reason not to
(such as an embedded single quote). Fix a few counter-examples in this file.
Also add a missing function-argument comment.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

buildman: Add gcc 4.9.0 with Microblaze toolchain

Also read gcc 4.9.0 at kernel.org which also have Microblaze toolchain.

Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

Fixed unit test failure by updating the test:
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct toolchain download feature

Commit d908898 updated the ScanPath() function but not its documentation
and not all its callers.

This breaks the toolchain check after it is downloaded. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Acked-by: Heiko Schocher <hs@denx.de>

buildman: allow multiple toolchains in a single path

When buildman scans a toolchain path, it stops at the
first toolchain found. However, a single path can contains
several toolchains, each with its own prefix.

This patch lets buildman scan all toolchains in the path.

Signed-off-by: Albert ARIBAUD <albert.u.boot@aribaud.net>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: Add the option to download toolchains from kernel.org

The site at https://www.kernel.org/pub/tools/crosstool/ is a convenient
repository of toolchains which can be used for U-Boot. Add a feature to
download and install a toolchain for a selected architecture automatically.

It isn't clear how long this site will stay in the current place and
format, but we should be able to rely on bug reports if it changes.

Suggested-by: Marek Vašut <marex@denx.de>
Suggested-by: Fabio Estevam <festevam@gmail.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow architecture to alias to multiple toolchains

Some archs have need than one alias, so support a list of alises in the
..buildman file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to use the full tool chain path

In some cases there may be multiple toolchains with the same name in the
path. Provide an option to use the full path in the CROSS_COMPILE
environment variable.

Note: Wolfgang mentioned that this is dangerous since in some cases there
may be other tools on the path that are needed. So this is set up as an
option, not the default. I will need test confirmation (i.e. that this
commit fixes a real problem) before merging it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Steve Rae <srae@broadcom.com>

buildman: Put the toolchain path first instead of last in PATH

If:

1. Toolchains A and B have the same filename
2. Toolchain A is in the PATH
3. Toolchain B is given in ~/.buildman and buildman uses it to build

then buildman will add toolchain B to the end of its path but will not
necessarily use it since U-Boot will find toolchain A first in the PATH.

Try to fix this by putting the toolchain first in the path instead of
last.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Try to avoid hard-coded string parsing

The assumption that the compiler name will always end in gcc is incorrect
for clang and apparently on BSD.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a functional test

Buildman currently lacks testing in many areas, including its use of git,
make and many command-line flags.

Add a functional test which covers some of these areas. So far it does
a fake 'build' of all boards for the current source tree.

This version reads the real ~/.buildman and boards.cfg files. Future work
will improve this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow make-flags variables to include '-' and '_'

These characters are commonly used in variables, so permit them. Also
document the permitted characters.

Reported-by: Tom Rini <trini@ti.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: fix toolchain priority_list

'-elf' appears twice in the toolchain priority_list.
The second one is rudundant.

Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Cc: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: fix to display warning message for missing [toolchain] section

Toolchains.__init__ is expected to display a warning message
when the [toolchain] section is missing from ~/.buildman file.
But it never works.
In that case, instead, buildmain fails with an error message
which is difficult to understand:

Traceback (most recent call last):
File "tools/buildman/buildman", line 126, in <module>
control.DoBuildman(options, args)
File "/home/foo/u-boot/tools/buildman/control.py", line 78, in DoBuildman
toolchains = toolchain.Toolchains()
File "/home/foo/u-boot/tools/buildman/toolchain.py", line 106, in __init__
config_fname)
NameError: global name 'config_fname' is not defined

Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Cc: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: don't fail --list-toolchains when toolchains fail

When a toolchain invocation fails, an exception is thrown but not caught
which then aborts the entire toolchain detection process. To solve this,
request that exceptions not be thrown, since the toolchain init code
already error-checks the command result. This solves e.g.:

- found '/usr/bin/winegcc'
Traceback (most recent call last):
...
Exception: Error running '/usr/bin/winegcc --version'

Change-Id: I579c72ab3b021e38b14132893c3375ea257c74f0
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Simon Glass <sjg@chromium.org>
(formatted to 80cols)

buildman: Allow make flags to be specified for each board

There are a few make options such as BUILD_TAG which can be provided when
building U-Boot. Provide a way for buildman to pass these flags to make
also.

The flags should be in a [make-flags] section and arranged by target name
(the 'target' column in boards.cfg. See the README for more details.

Signed-off-by: Simon Glass <sjg@chromium.org>

Add GPL-2.0+ SPDX-License-Identifier to source files

Signed-off-by: Wolfgang Denk <wd@denx.de>
[trini: Fixup common/cmd_io.c]
Signed-off-by: Tom Rini <trini@ti.com>

buildman - U-Boot multi-threaded builder and summary tool

This tool handles building U-Boot to check that you have not broken it
with your patch series. It can build each individual commit and report
which boards fail on which commits, and which errors come up. It also
shows differences in image sizes due to particular commits.

Buildman aims to make full use of multi-processor machines.

Documentation and caveats are in tools/buildman/README.

Signed-off-by: Simon Glass <sjg@chromium.org>

Merge branch 'next'

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: Correct overwriting of settings file

The toolchain test causes the settings file to be overwritten, which is
annoying for local development. Fix it by passing None as the filename.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix CROSS_COMPILE output for sandbox

The previous attempt at fixing this broke the normal usage of the -A
flag.

At present, 'buildman -A sandbox' adds the path containing the
toolchain. We can assume that this is in the path and we don't want to
set CROSS_COMPILE=/bin/

Change this to align with what MakeEnvironment() does, but only for
sandbox boards.

Signed-off-by: Simon Glass <sjg@chromium.org>

Revert "buildman: Correct CROSS_COMPILE output for sandbox"

This reverts commit bd0a548ad4a155fec29473d4cc8e135832926973.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct CROSS_COMPILE output for sandbox

At present, 'buildman -A sandbox' adds the path containing the
toolchain at present. We can assume that this is in the path and
we don't want to set CROSS_COMPILE=/bin/ so change this to align
with what MakeEnvironment() does.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move library functions into a library directory

The patman directory has a number of modules which are used by other tools
in U-Boot. This makes it hard to package the tools using pypi since the
common files must be copied along with the tool that uses them.

To address this, move these files into a new u_boot_pylib library. This
can be packaged separately and listed as a dependency of each tool.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: invalid reference to README

The readme file for buildman is called buildman.rst.

Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Fetch 12.2.0 toolchains by default

Update the toolchain list to be first 12.2.0 and second 11.1.0 and
that's it.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Color() method to build()

This method has the same name as its class which is confusing. It is also
annoying when searching the code.

It builds a string with a colour, so rename it to build().

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in command.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Tidy up the download function a little

Reverse the order of the return tuple, so that the filename is first.
This seems more obvious than putting the temporary directory first.

Correct a bug that leaves a space on the final line.

Allow the caller to control the name of the temporary directory.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move the download function to tools

This function is handy for binman as well. Move it into the shared 'tools'
module.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add gcc-11.1.0 to the directory list

While CI has been using gcc-11.1.0 for a long time, we have not updated
buildman to match. Correct this omission.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: Use bytes for the environment

At present we sometimes see problems in gitlab where the environment has
0x80 characters or sequences which are not valid UTF-8.

Avoid this by using bytes for the environment, both internal to buildman
and when writing out the 'env' file. Add a test to make sure this works
as expected.

Reported-by: Marek Vasut <marex@denx.de>
Fixes: e5fc79ea718 ("buildman: Write the environment out to an 'env' file")
Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support fetching gcc 9.2.0

This adds support to fetch gcc 9.2.0 toolchains.

Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Enable buildman on aarch64 hosts

At kernel.org aarch64 toolchains are published in folder
arm64. Fix the URL for that case, so that we can fetch
toolchains on aarch64 machines.

Signed-off-by: Matthias Brugger <mbrugger@suse.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Add options to get the arch and toolchain info

Sometimes it is useful for external tools to use buildman to provide the
toolchain information. Add an -a option which shows the value to use for
the ARCH environment variable, and -A which does the same for
CROSS_COMPILE

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Improve [make-flags] section parser to allow quoted strings

The parser responsible for the '[make-flags]' section in
the '.buildman' settings file is currently not able to
handle quoted strings, as given in the sample bellow:

[make-flags]
qemu_arm=HOSTCC="cc -isystem /add/include" HOSTLDFLAGS="-L/add/lib"

This patch replaces the simple string splitter based on the <space>
delimiter with a regex tokenizer that preserves spaces inside double
quoted strings.

Signed-off-by: Cristian Ciocaltea <cristian.ciocaltea@gmail.com>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix tabs in GetWrapper()

This function has tabs instead of spaces. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add support for building with clang

Add a -O option which allows building with clang.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman/toolchain.py: handle inconsistent tarball names

Unfortunately, for some releases the kernel.org toolchain tarball names adhere
to the following pattern:

<hostarch>-gcc-<ver>-nolib-<targetarch>-<type>.tar.xz

e.g.:
x86_64-gcc-8.1.0-nolibc-aarch64-linux.tar.xz

while others use the following pattern:

<hostarch>-gcc-<ver>-nolib_<targetarch>-<type>.tar.xz

e.g.:

x86_64-gcc-7.3.0-nolibc_aarch64-linux.tar.xz

Notice that the first pattern has dashes throughout, while the second has
dashes throughout except just before the target architecture which has an
underscore.

The "dash throughout" versions from kernel.org are:

8.1.0, 6.4.0, 5.5.0, 4.9.4, 4.8.5, 4.6.1

while the "dash and underscore" versions from kernel.org are:

7.3.0, 4.9.0, 4.8.0, 4.7.3, 4.6.3, 4.6.2, 4.5.1, 4.2.4

This tweak allows the code to handle both versions. Note that this tweak also
causes the architecture parsing to get confused and find the following two
bogus architectures, "2.0" and "64", which are explicitly checked for, and
removed.

Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Change single quotes to double quotes:
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman/toolchain.py: fix toolchain directory

The hexagon toolchain (4.6.1) from kernel.org, for example, was packaged in
a way that is different from most toolchains. The first entry when unpacking
most toolchain tarballs is:

gcc-<version>-nolib/<targetarch>-<system>

e.g.:

gcc-8.1.0-nolibc/aarch64-linux/

The first entry of the hexagon toolchain, however, is:

gcc-4.6.1-nolibc/

This causes the buildman logic in toolchain.py::ScanPath() to not be able to
find the "*gcc" executable since it looks in gcc-4.6.1-nolib/{.|bin|usr/bin}
instead of gcc-4.6.1/hexagon-linux/{.|bin|usr/bin}. Therefore when buildman
tries to download a set of toolchains that includes hexagon, the script fails.

This update takes the second line of the tarball unpacking (which works for
all the toolchains I've tested from kernel.org) and parses it to take the
first two elements, separated by '/'. It makes this logic a bit more robust.

Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: support newer gcc versions from kernel.org

Add support for gcc versions 7.3.0, 6.4.0 and 4.9.4.

Also use a regex for matching the tarball names. Some gcc versions
use '-ARCH-' instead of '_ARCH-'.

As part of this, we switch TravisCI to also using these toolchains for
all platforms.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
Signed-off-by: Tom Rini <trini@konsulko.com>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: disable localized and unicode output of all build tools

Build tools like Make, gcc or binutils support localized output
or unicode encoded output dependent on the default system locale.
This is not useful for buildman, where we want reproducible
warning or error messages or where the output of binutils is
further processed.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>

buildman: Allow 'gnueabihf' toolchains for ARM

Many toolchains for ARM use the 'gnueabihf' suffix rather than just
'gnueabi', so allow these to be used, but with a lower priority than
'gnueabi' ones.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

tools: buildman: Add compiler wrapper

Now we can use compiler wrapper such as ccache or distcc for buildman.

Signed-off-by: York Sun <york.sun@nxp.com>
CC: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: Avoid overwriting existing toolchain entries

The current code for setting up the toolchain config always writes the new
paths to an item called 'toolchain'. This means that it will overwrite any
existing toolchain item with the same name. In practice, this means that:

buildman --fetch-arch all

will fetch all toolchains, but only the path of the final one will be added
to the config. This normally works out OK, since most toolchains are the
same version (e.g. gcc 4.9) and will be found on the same path. But it is
not correct and toolchains for archs which don't use the same version will
not function as expected.

Adjust the code to use a complete glob of the toolchain path.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Fix a typo in TestSettingsHasPath()

The function comment should say 'buildman'. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Improve the toolchain progress/error output

Use colour to make it easier to see what is going on. Also print a message
before downloading a new toolchain. Mention --fetch-arch in the message that
is shown when there are no available toolchains, since this is the quickest
way to resolve the problem.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Allow the toolchain error to be suppressed

When there are no toolchains a warning is printed. But in some cases this is
confusing, such as when the user is fetching new toolchains.

Adjust the function to supress the warning in this case.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a way to specific a full toolchain prefix

At present buildman allows you to specify the directory containing the
toolchain, but not the actual toolchain prefix. If there are multiple
toolchains in a single directory, this can be inconvenient.

Add a new 'toolchain-prefix' setting to the settings file, which allows
the full prefix (or path to the C compiler) to be specified.

Update the documentation to match.

Suggested-by: Stephen Warren <swarren@wwwdotorg.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>

buildman: Allow the toolchain architecture to be specified

At present the architecture is deduced from the toolchain filename. Allow it
to be specified by the caller.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com

buildman: Allow the toolchain priority to be specified

At present the priority of a toolchain is calculated from its filename based
on hard-coded rules. Allow it to be specified by the caller. We will use
this in a later patch. Also display the priority and provide a message when
it is overriden by another toolchain of higher priority.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

buildman: Fix up a few code inconsistencies in toolchain.py

Normally we use a single quote for strings unless there is a reason not to
(such as an embedded single quote). Fix a few counter-examples in this file.
Also add a missing function-argument comment.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

buildman: Add gcc 4.9.0 with Microblaze toolchain

Also read gcc 4.9.0 at kernel.org which also have Microblaze toolchain.

Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

Fixed unit test failure by updating the test:
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct toolchain download feature

Commit d908898 updated the ScanPath() function but not its documentation
and not all its callers.

This breaks the toolchain check after it is downloaded. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Acked-by: Heiko Schocher <hs@denx.de>

buildman: allow multiple toolchains in a single path

When buildman scans a toolchain path, it stops at the
first toolchain found. However, a single path can contains
several toolchains, each with its own prefix.

This patch lets buildman scan all toolchains in the path.

Signed-off-by: Albert ARIBAUD <albert.u.boot@aribaud.net>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: Add the option to download toolchains from kernel.org

The site at https://www.kernel.org/pub/tools/crosstool/ is a convenient
repository of toolchains which can be used for U-Boot. Add a feature to
download and install a toolchain for a selected architecture automatically.

It isn't clear how long this site will stay in the current place and
format, but we should be able to rely on bug reports if it changes.

Suggested-by: Marek Vašut <marex@denx.de>
Suggested-by: Fabio Estevam <festevam@gmail.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow architecture to alias to multiple toolchains

Some archs have need than one alias, so support a list of alises in the
..buildman file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to use the full tool chain path

In some cases there may be multiple toolchains with the same name in the
path. Provide an option to use the full path in the CROSS_COMPILE
environment variable.

Note: Wolfgang mentioned that this is dangerous since in some cases there
may be other tools on the path that are needed. So this is set up as an
option, not the default. I will need test confirmation (i.e. that this
commit fixes a real problem) before merging it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Steve Rae <srae@broadcom.com>

buildman: Put the toolchain path first instead of last in PATH

If:

1. Toolchains A and B have the same filename
2. Toolchain A is in the PATH
3. Toolchain B is given in ~/.buildman and buildman uses it to build

then buildman will add toolchain B to the end of its path but will not
necessarily use it since U-Boot will find toolchain A first in the PATH.

Try to fix this by putting the toolchain first in the path instead of
last.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Try to avoid hard-coded string parsing

The assumption that the compiler name will always end in gcc is incorrect
for clang and apparently on BSD.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a functional test

Buildman currently lacks testing in many areas, including its use of git,
make and many command-line flags.

Add a functional test which covers some of these areas. So far it does
a fake 'build' of all boards for the current source tree.

This version reads the real ~/.buildman and boards.cfg files. Future work
will improve this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow make-flags variables to include '-' and '_'

These characters are commonly used in variables, so permit them. Also
document the permitted characters.

Reported-by: Tom Rini <trini@ti.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: fix toolchain priority_list

'-elf' appears twice in the toolchain priority_list.
The second one is rudundant.

Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Cc: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: fix to display warning message for missing [toolchain] section

Toolchains.__init__ is expected to display a warning message
when the [toolchain] section is missing from ~/.buildman file.
But it never works.
In that case, instead, buildmain fails with an error message
which is difficult to understand:

Traceback (most recent call last):
File "tools/buildman/buildman", line 126, in <module>
control.DoBuildman(options, args)
File "/home/foo/u-boot/tools/buildman/control.py", line 78, in DoBuildman
toolchains = toolchain.Toolchains()
File "/home/foo/u-boot/tools/buildman/toolchain.py", line 106, in __init__
config_fname)
NameError: global name 'config_fname' is not defined

Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Cc: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: don't fail --list-toolchains when toolchains fail

When a toolchain invocation fails, an exception is thrown but not caught
which then aborts the entire toolchain detection process. To solve this,
request that exceptions not be thrown, since the toolchain init code
already error-checks the command result. This solves e.g.:

- found '/usr/bin/winegcc'
Traceback (most recent call last):
...
Exception: Error running '/usr/bin/winegcc --version'

Change-Id: I579c72ab3b021e38b14132893c3375ea257c74f0
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Simon Glass <sjg@chromium.org>
(formatted to 80cols)

buildman: Allow make flags to be specified for each board

There are a few make options such as BUILD_TAG which can be provided when
building U-Boot. Provide a way for buildman to pass these flags to make
also.

The flags should be in a [make-flags] section and arranged by target name
(the 'target' column in boards.cfg. See the README for more details.

Signed-off-by: Simon Glass <sjg@chromium.org>

Add GPL-2.0+ SPDX-License-Identifier to source files

Signed-off-by: Wolfgang Denk <wd@denx.de>
[trini: Fixup common/cmd_io.c]
Signed-off-by: Tom Rini <trini@ti.com>

buildman - U-Boot multi-threaded builder and summary tool

This tool handles building U-Boot to check that you have not broken it
with your patch series. It can build each individual commit and report
which boards fail on which commits, and which errors come up. It also
shows differences in image sizes due to particular commits.

Buildman aims to make full use of multi-processor machines.

Documentation and caveats are in tools/buildman/README.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: invalid reference to README

The readme file for buildman is called buildman.rst.

Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Fetch 12.2.0 toolchains by default

Update the toolchain list to be first 12.2.0 and second 11.1.0 and
that's it.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Color() method to build()

This method has the same name as its class which is confusing. It is also
annoying when searching the code.

It builds a string with a colour, so rename it to build().

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in command.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Tidy up the download function a little

Reverse the order of the return tuple, so that the filename is first.
This seems more obvious than putting the temporary directory first.

Correct a bug that leaves a space on the final line.

Allow the caller to control the name of the temporary directory.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move the download function to tools

This function is handy for binman as well. Move it into the shared 'tools'
module.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add gcc-11.1.0 to the directory list

While CI has been using gcc-11.1.0 for a long time, we have not updated
buildman to match. Correct this omission.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: Use bytes for the environment

At present we sometimes see problems in gitlab where the environment has
0x80 characters or sequences which are not valid UTF-8.

Avoid this by using bytes for the environment, both internal to buildman
and when writing out the 'env' file. Add a test to make sure this works
as expected.

Reported-by: Marek Vasut <marex@denx.de>
Fixes: e5fc79ea718 ("buildman: Write the environment out to an 'env' file")
Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support fetching gcc 9.2.0

This adds support to fetch gcc 9.2.0 toolchains.

Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Enable buildman on aarch64 hosts

At kernel.org aarch64 toolchains are published in folder
arm64. Fix the URL for that case, so that we can fetch
toolchains on aarch64 machines.

Signed-off-by: Matthias Brugger <mbrugger@suse.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Add options to get the arch and toolchain info

Sometimes it is useful for external tools to use buildman to provide the
toolchain information. Add an -a option which shows the value to use for
the ARCH environment variable, and -A which does the same for
CROSS_COMPILE

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Improve [make-flags] section parser to allow quoted strings

The parser responsible for the '[make-flags]' section in
the '.buildman' settings file is currently not able to
handle quoted strings, as given in the sample bellow:

[make-flags]
qemu_arm=HOSTCC="cc -isystem /add/include" HOSTLDFLAGS="-L/add/lib"

This patch replaces the simple string splitter based on the <space>
delimiter with a regex tokenizer that preserves spaces inside double
quoted strings.

Signed-off-by: Cristian Ciocaltea <cristian.ciocaltea@gmail.com>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix tabs in GetWrapper()

This function has tabs instead of spaces. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add support for building with clang

Add a -O option which allows building with clang.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman/toolchain.py: handle inconsistent tarball names

Unfortunately, for some releases the kernel.org toolchain tarball names adhere
to the following pattern:

<hostarch>-gcc-<ver>-nolib-<targetarch>-<type>.tar.xz

e.g.:
x86_64-gcc-8.1.0-nolibc-aarch64-linux.tar.xz

while others use the following pattern:

<hostarch>-gcc-<ver>-nolib_<targetarch>-<type>.tar.xz

e.g.:

x86_64-gcc-7.3.0-nolibc_aarch64-linux.tar.xz

Notice that the first pattern has dashes throughout, while the second has
dashes throughout except just before the target architecture which has an
underscore.

The "dash throughout" versions from kernel.org are:

8.1.0, 6.4.0, 5.5.0, 4.9.4, 4.8.5, 4.6.1

while the "dash and underscore" versions from kernel.org are:

7.3.0, 4.9.0, 4.8.0, 4.7.3, 4.6.3, 4.6.2, 4.5.1, 4.2.4

This tweak allows the code to handle both versions. Note that this tweak also
causes the architecture parsing to get confused and find the following two
bogus architectures, "2.0" and "64", which are explicitly checked for, and
removed.

Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Change single quotes to double quotes:
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman/toolchain.py: fix toolchain directory

The hexagon toolchain (4.6.1) from kernel.org, for example, was packaged in
a way that is different from most toolchains. The first entry when unpacking
most toolchain tarballs is:

gcc-<version>-nolib/<targetarch>-<system>

e.g.:

gcc-8.1.0-nolibc/aarch64-linux/

The first entry of the hexagon toolchain, however, is:

gcc-4.6.1-nolibc/

This causes the buildman logic in toolchain.py::ScanPath() to not be able to
find the "*gcc" executable since it looks in gcc-4.6.1-nolib/{.|bin|usr/bin}
instead of gcc-4.6.1/hexagon-linux/{.|bin|usr/bin}. Therefore when buildman
tries to download a set of toolchains that includes hexagon, the script fails.

This update takes the second line of the tarball unpacking (which works for
all the toolchains I've tested from kernel.org) and parses it to take the
first two elements, separated by '/'. It makes this logic a bit more robust.

Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: support newer gcc versions from kernel.org

Add support for gcc versions 7.3.0, 6.4.0 and 4.9.4.

Also use a regex for matching the tarball names. Some gcc versions
use '-ARCH-' instead of '_ARCH-'.

As part of this, we switch TravisCI to also using these toolchains for
all platforms.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
Signed-off-by: Tom Rini <trini@konsulko.com>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: disable localized and unicode output of all build tools

Build tools like Make, gcc or binutils support localized output
or unicode encoded output dependent on the default system locale.
This is not useful for buildman, where we want reproducible
warning or error messages or where the output of binutils is
further processed.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>

buildman: Allow 'gnueabihf' toolchains for ARM

Many toolchains for ARM use the 'gnueabihf' suffix rather than just
'gnueabi', so allow these to be used, but with a lower priority than
'gnueabi' ones.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

tools: buildman: Add compiler wrapper

Now we can use compiler wrapper such as ccache or distcc for buildman.

Signed-off-by: York Sun <york.sun@nxp.com>
CC: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: Avoid overwriting existing toolchain entries

The current code for setting up the toolchain config always writes the new
paths to an item called 'toolchain'. This means that it will overwrite any
existing toolchain item with the same name. In practice, this means that:

buildman --fetch-arch all

will fetch all toolchains, but only the path of the final one will be added
to the config. This normally works out OK, since most toolchains are the
same version (e.g. gcc 4.9) and will be found on the same path. But it is
not correct and toolchains for archs which don't use the same version will
not function as expected.

Adjust the code to use a complete glob of the toolchain path.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Fix a typo in TestSettingsHasPath()

The function comment should say 'buildman'. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Improve the toolchain progress/error output

Use colour to make it easier to see what is going on. Also print a message
before downloading a new toolchain. Mention --fetch-arch in the message that
is shown when there are no available toolchains, since this is the quickest
way to resolve the problem.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Allow the toolchain error to be suppressed

When there are no toolchains a warning is printed. But in some cases this is
confusing, such as when the user is fetching new toolchains.

Adjust the function to supress the warning in this case.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a way to specific a full toolchain prefix

At present buildman allows you to specify the directory containing the
toolchain, but not the actual toolchain prefix. If there are multiple
toolchains in a single directory, this can be inconvenient.

Add a new 'toolchain-prefix' setting to the settings file, which allows
the full prefix (or path to the C compiler) to be specified.

Update the documentation to match.

Suggested-by: Stephen Warren <swarren@wwwdotorg.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Stephen Warren <swarren@nvidia.com>

buildman: Allow the toolchain architecture to be specified

At present the architecture is deduced from the toolchain filename. Allow it
to be specified by the caller.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com

buildman: Allow the toolchain priority to be specified

At present the priority of a toolchain is calculated from its filename based
on hard-coded rules. Allow it to be specified by the caller. We will use
this in a later patch. Also display the priority and provide a message when
it is overriden by another toolchain of higher priority.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

buildman: Fix up a few code inconsistencies in toolchain.py

Normally we use a single quote for strings unless there is a reason not to
(such as an embedded single quote). Fix a few counter-examples in this file.
Also add a missing function-argument comment.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

buildman: Add gcc 4.9.0 with Microblaze toolchain

Also read gcc 4.9.0 at kernel.org which also have Microblaze toolchain.

Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

Fixed unit test failure by updating the test:
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct toolchain download feature

Commit d908898 updated the ScanPath() function but not its documentation
and not all its callers.

This breaks the toolchain check after it is downloaded. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Acked-by: Heiko Schocher <hs@denx.de>

buildman: allow multiple toolchains in a single path

When buildman scans a toolchain path, it stops at the
first toolchain found. However, a single path can contains
several toolchains, each with its own prefix.

This patch lets buildman scan all toolchains in the path.

Signed-off-by: Albert ARIBAUD <albert.u.boot@aribaud.net>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: Add the option to download toolchains from kernel.org

The site at https://www.kernel.org/pub/tools/crosstool/ is a convenient
repository of toolchains which can be used for U-Boot. Add a feature to
download and install a toolchain for a selected architecture automatically.

It isn't clear how long this site will stay in the current place and
format, but we should be able to rely on bug reports if it changes.

Suggested-by: Marek Vašut <marex@denx.de>
Suggested-by: Fabio Estevam <festevam@gmail.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow architecture to alias to multiple toolchains

Some archs have need than one alias, so support a list of alises in the
..buildman file.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add an option to use the full tool chain path

In some cases there may be multiple toolchains with the same name in the
path. Provide an option to use the full path in the CROSS_COMPILE
environment variable.

Note: Wolfgang mentioned that this is dangerous since in some cases there
may be other tools on the path that are needed. So this is set up as an
option, not the default. I will need test confirmation (i.e. that this
commit fixes a real problem) before merging it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Steve Rae <srae@broadcom.com>

buildman: Put the toolchain path first instead of last in PATH

If:

1. Toolchains A and B have the same filename
2. Toolchain A is in the PATH
3. Toolchain B is given in ~/.buildman and buildman uses it to build

then buildman will add toolchain B to the end of its path but will not
necessarily use it since U-Boot will find toolchain A first in the PATH.

Try to fix this by putting the toolchain first in the path instead of
last.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Try to avoid hard-coded string parsing

The assumption that the compiler name will always end in gcc is incorrect
for clang and apparently on BSD.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a functional test

Buildman currently lacks testing in many areas, including its use of git,
make and many command-line flags.

Add a functional test which covers some of these areas. So far it does
a fake 'build' of all boards for the current source tree.

This version reads the real ~/.buildman and boards.cfg files. Future work
will improve this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow make-flags variables to include '-' and '_'

These characters are commonly used in variables, so permit them. Also
document the permitted characters.

Reported-by: Tom Rini <trini@ti.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: fix toolchain priority_list

'-elf' appears twice in the toolchain priority_list.
The second one is rudundant.

Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Cc: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: fix to display warning message for missing [toolchain] section

Toolchains.__init__ is expected to display a warning message
when the [toolchain] section is missing from ~/.buildman file.
But it never works.
In that case, instead, buildmain fails with an error message
which is difficult to understand:

Traceback (most recent call last):
File "tools/buildman/buildman", line 126, in <module>
control.DoBuildman(options, args)
File "/home/foo/u-boot/tools/buildman/control.py", line 78, in DoBuildman
toolchains = toolchain.Toolchains()
File "/home/foo/u-boot/tools/buildman/toolchain.py", line 106, in __init__
config_fname)
NameError: global name 'config_fname' is not defined

Signed-off-by: Masahiro Yamada <yamada.m@jp.panasonic.com>
Cc: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org>

buildman: don't fail --list-toolchains when toolchains fail

When a toolchain invocation fails, an exception is thrown but not caught
which then aborts the entire toolchain detection process. To solve this,
request that exceptions not be thrown, since the toolchain init code
already error-checks the command result. This solves e.g.:

- found '/usr/bin/winegcc'
Traceback (most recent call last):
...
Exception: Error running '/usr/bin/winegcc --version'

Change-Id: I579c72ab3b021e38b14132893c3375ea257c74f0
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Simon Glass <sjg@chromium.org>
(formatted to 80cols)

buildman: Allow make flags to be specified for each board

There are a few make options such as BUILD_TAG which can be provided when
building U-Boot. Provide a way for buildman to pass these flags to make
also.

The flags should be in a [make-flags] section and arranged by target name
(the 'target' column in boards.cfg. See the README for more details.

Signed-off-by: Simon Glass <sjg@chromium.org>

Add GPL-2.0+ SPDX-License-Identifier to source files

Signed-off-by: Wolfgang Denk <wd@denx.de>
[trini: Fixup common/cmd_io.c]
Signed-off-by: Tom Rini <trini@ti.com>

buildman - U-Boot multi-threaded builder and summary tool

This tool handles building U-Boot to check that you have not broken it
with your patch series. It can build each individual commit and report
which boards fail on which commits, and which errors come up. It also
shows differences in image sizes due to particular commits.

Buildman aims to make full use of multi-processor machines.

Documentation and caveats are in tools/buildman/README.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fetch 12.2.0 toolchains by default

Update the toolchain list to be first 12.2.0 and second 11.1.0 and
that's it.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Rename Color() method to build()

This method has the same name as its class which is confusing. It is also
annoying when searching the code.

It builds a string with a colour, so rename it to build().

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in command.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Tidy up the download function a little

Reverse the order of the return tuple, so that the filename is first.
This seems more obvious than putting the temporary directory first.

Correct a bug that leaves a space on the final line.

Allow the caller to control the name of the temporary directory.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move the download function to tools

This function is handy for binman as well. Move it into the shared 'tools'
module.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add gcc-11.1.0 to the directory list

While CI has been using gcc-11.1.0 for a long time, we have not updated
buildman to match. Correct this omission.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: Use bytes for the environment

At present we sometimes see problems in gitlab where the environment has
0x80 characters or sequences which are not valid UTF-8.

Avoid this by using bytes for the environment, both internal to buildman
and when writing out the 'env' file. Add a test to make sure this works
as expected.

Reported-by: Marek Vasut <marex@denx.de>
Fixes: e5fc79ea718 ("buildman: Write the environment out to an 'env' file")
Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support fetching gcc 9.2.0

This adds support to fetch gcc 9.2.0 toolchains.

Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Tom Rini <trini@konsulko.com>

buildman: Enable buildman on aarch64 hosts

At kernel.org aarch64 toolchains are published in folder
arm64. Fix the URL for that case, so that we can fetch
toolchains on aarch64 machines.

Signed-off-by: Matthias Brugger <mbrugger@suse.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: Add options to get the arch and toolchain info

Sometimes it is useful for external tools to use buildman to provide the
toolchain information. Add an -a option which shows the value to use for
the ARCH environment variable, and -A which does the same for
CROSS_COMPILE

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Improve [make-flags] section parser to allow quoted strings

The parser responsible for the '[make-flags]' section in
the '.buildman' settings file is currently not able to
handle quoted strings, as given in the sample bellow:

[make-flags]
qemu_arm=HOSTCC="cc -isystem /add/include" HOSTLDFLAGS="-L/add/lib"

This patch replaces the simple string splitter based on the <space>
delimiter with a regex tokenizer that preserves spaces inside double
quoted strings.

Signed-off-by: Cristian Ciocaltea <cristian.ciocaltea@gmail.com>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix tabs in GetWrapper()

This function has tabs instead of spaces. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add support for building with clang

Add a -O option which allows building with clang.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman/toolchain.py: handle inconsistent tarball names

Unfortunately, for some releases the kernel.org toolchain tarball names adhere
to the following pattern:

<hostarch>-gcc-<ver>-nolib-<targetarch>-<type>.tar.xz

e.g.:
x86_64-gcc-8.1.0-nolibc-aarch64-linux.tar.xz

while others use the following pattern:

<hostarch>-gcc-<ver>-nolib_<targetarch>-<type>.tar.xz

e.g.:

x86_64-gcc-7.3.0-nolibc_aarch64-linux.tar.xz

Notice that the first pattern has dashes throughout, while the second has
dashes throughout except just before the target architecture which has an
underscore.

The "dash throughout" versions from kernel.org are:

8.1.0, 6.4.0, 5.5.0, 4.9.4, 4.8.5, 4.6.1

while the "dash and underscore" versions from kernel.org are:

7.3.0, 4.9.0, 4.8.0, 4.7.3, 4.6.3, 4.6.2, 4.5.1, 4.2.4

This tweak allows the code to handle both versions. Note that this tweak also
causes the architecture parsing to get confused and find the following two
bogus architectures, "2.0" and "64", which are explicitly checked for, and
removed.

Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Change single quotes to double quotes:
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman/toolchain.py: fix toolchain directory

The hexagon toolchain (4.6.1) from kernel.org, for example, was packaged in
a way that is different from most toolchains. The first entry when unpacking
most toolchain tarballs is:

gcc-<version>-nolib/<targetarch>-<system>

e.g.:

gcc-8.1.0-nolibc/aarch64-linux/

The first entry of the hexagon toolchain, however, is:

gcc-4.6.1-nolibc/

This causes the buildman logic in toolchain.py::ScanPath() to not be able to
find the "*gcc" executable since it looks in gcc-4.6.1-nolib/{.|bin|usr/bin}
instead of gcc-4.6.1/hexagon-linux/{.|bin|usr/bin}. Therefore when buildman
tries to download a set of toolchains that includes hexagon, the script fails.

This update takes the second line of the tarball unpacking (which works for
all the toolchains I've tested from kernel.org) and parses it to take the
first two elements, separated by '/'. It makes this logic a bit more robust.

Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

buildman: support newer gcc versions from kernel.org

Add support for gcc versions 7.3.0, 6.4.0 and 4.9.4.

Also use a regex for matching the tarball names. Some gcc versions
use '-ARCH-' instead of '_ARCH-'.

As part of this, we switch TravisCI to also using these toolchains for
all platforms.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
Signed-off-by: Tom Rini <trini@konsulko.com>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

buildman: disable localized and unicode output of all build tools

Build tools like Make, gcc or binutils support localized output
or unicode encoded output dependent on the default system locale.
This is not useful for buildman, where we want reproducible
warning or error messages or where the output of binutils is
further processed.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>

buildman: Allow 'gnueabihf' toolchains for ARM

Many toolchains for ARM use the 'gnueabihf' suffix rather than just
'gnueabi', so allow these to be used, but with a lower priority than
'gnueabi' ones.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>