buildman: Use oldconfig when adjusting the config

We cannot be sure that the new config is consistent, particularly when
changing a major item like CONFIG_CMDLINE. Use 'make oldconfig' to
check that and avoid any such problems.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Drop warning about orphaned defconfigs

Some boards use a MAINTAINERS entry to specify common files without
referencing any defconfigs. This is allowed and should not result in a
warning.

Drop the warning in this case.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Specify the output directory in tests

The default output directory is generally '../' in tests so we end up
trying to create '../.bm-work'. This does not work with azure, so update
these tests to use the temporary directory instead.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a way to print the architecture for a board

This is useful for some tools and is easily available for buildman. Add
a new --print-arch option.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Tidy up reporting of a toolchain error

Provide the text of the exception when something goes wrong.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in builder.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in bsettings.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert to argparse

Use argparse to parse the arguments, since OptionParser is deprecated now.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test for --boards

Add a simple functional test for the --boards option. Fix the example in
the docs while we are here. Also improve the docs for Builder.count so it
is clearer what it contains.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in cmdline.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add tests for excluding things

Add some tests for the -x flag.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test for the -A option

This lacks a test at present. Add one.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix most pylint warnings in control

Tidy up the easier-to-fix pylint warnings in module 'control'.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in control.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Provide an argument to the -R option

Allow writing the file to a selected location, since otherwise this is
controlled by the buildman configuration, so cannot be determined by the
caller.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: https://source.denx.de/u-boot/u-boot/-/issues/17

buildman: Add an option to check maintainers

Rather than using the -R option to get this report as a side effect, add
a dedicated option for it.

Disable CI for now as there are some missing maintainers, unfortunately.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct logic for missing maintainers

An orphaned board should produce a warning, as should a missing name for
the maintainer (when '-' is provided). Add these cases.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Sort the maintainer warnings

Sort the warnings into alphabetical order, for easier reading. Also make
sure that the buildman test files are ignored.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Just display a single line for missing maintainers

At present we get multiple lines of output when a board has no MAINTAINERS
entry:

WARNING: no status info for 'bananapi-m2-pro'
WARNING: no maintainers for 'bananapi-m2-pro'

Suppress the 'status' one since it is implied by the other.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct operation of MAINTAINERS N:

This doesn't work as intended. Instead it scans every defconfig file
in the source tree.

Fix it and add a test.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Detect boards with no CONFIG_TARGET defined

We generally expected exactly one of these. Add a check for it.

Note: This warning is not displayed by default. An option will be added
to enable it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Detect boards with multiple CONFIG_TARGETs defined

The TARGET_xxx options are special in that they refer to a single target.
Exactly one should be enabled for each target, corresponding to a
defconfig file.

Detect configs which result in two TARGET_xxx options being set. For
example, at present, TARGET_POLEG and TARET_POLEG_EVB are enabled for the
same board.

Note: This warning is not displayed by default. An option will be added
to enable it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Warn about dangling maintainer entries

Other than the top-level MAINTAINERS file, all maintainer entries should
actually reference a target. Add a warning to detect those that do not.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a partial test for ensure_board_list()

Create a new function which has the non-UI parts of ensure_board_list().
Add some tests for everything except the N: tag.

While we are here, fix the confusing usage of fname inside a loops that
also uses fname.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test for Boards.output_is_new()

Add a test for this code, adjusting the timestamp on various files to
check each use case.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test for Boards.scan_defconfigs()

Add a test for this code. It requires some defconfig files and a test
Kconfig to work with, so copy these into the temporary directory at the
start.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Rename the ARM boards

Use names consistent with their target names.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move library functions into a library directory

The patman directory has a number of modules which are used by other tools
in U-Boot. This makes it hard to package the tools using pypi since the
common files must be copied along with the tool that uses them.

To address this, move these files into a new u_boot_pylib library. This
can be packaged separately and listed as a dependency of each tool.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a flag for reproducible builds

This is quite a useful thing to use when building since it avoids small
size changes between commits. Add a -r flag for it.

Also undefine CONFIG_LOCALVERSION_AUTO since this appends the git hash
to the version string, causing every build to be slightly different.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support disabling LTO

This cuts down build performance considerably and is not always needed,
when checking for build errors, etc.

Add a flag to disable it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Write out the build command used

It is sometimes useful to see the exact 'make' command used by buildman
for a commit. Add an output file for this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add --allow-missing flag to allow missing blobs

Add a new flag to buildman so that we will in turn pass
BINMAN_ALLOW_MISSING=1 to 'make'. Make use of this flag in CI.

Allow the settings file to control this.

Cc: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Reinstate removal of temp output dir in tests

This was dropped my mistake. Reinstate it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: d829f1217c6 ("bulidman: Add support for a simple build")

buildman: Convert documentation to rST

Convert the buildman documentation to rST format and include it in the
'build' section.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Quentin Schulz <foss+uboot@0leil.net>

buildman: Split out Boards into its own file

Use a separate file for the Boards class so that its name matches the
module name.

Fix up the function names to match the pylint style and fix some other
warnings.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in board.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Drop use of 'boards' in control

Use brds instead so that we can reserve 'boards' for a module name.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix use of 'boards' in func_test

We want to create a module called 'boards' so avoid use of this variable
name in this module. Change the global to be capitalised, as required by
Python style.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct pylint errors

Fix pylint errors that can be fixed and mask those that seem to be
incorrect.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in terminal.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in gitutil.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in command.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Provide a hint on how to debug thread crashes

If a thread crashes it is helpful to try the operation again with
threading disabled. Add a hint about that.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add helper functions for updating .config files

At present the only straightforward way to write tests that need a
slightly different configuration is to create a new board with its own
configuration. This is cumbersome.

It would be useful if buildman could adjust the configuration of a build
on the fly. In preparation for this, add a utility library which can
modify a .config file according to various parameters passed to it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use bytes for the environment

At present we sometimes see problems in gitlab where the environment has
0x80 characters or sequences which are not valid UTF-8.

Avoid this by using bytes for the environment, both internal to buildman
and when writing out the 'env' file. Add a test to make sure this works
as expected.

Reported-by: Marek Vasut <marex@denx.de>
Fixes: e5fc79ea718 ("buildman: Write the environment out to an 'env' file")
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Handle exceptions in threads gracefully

There have been at least a few cases where an exception has occurred in a
thread and resulted in buildman hanging: running out of disk space and
getting a unicode error.

Handle these by collecting a list of exceptions, printing them out and
reporting failure if any are found. Add a test for this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Tidy up a few comments

Add some function comments which are missing, or missing arguments.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use git worktrees instead of git clones when possible

This patch makes buildman create linked working trees instead of clones
of the source repository, but keeps updating the older clones of the
repository that might already exist. These worktrees share "everything
except working directory specific files such as HEAD, index, etc." with
the source repository. See the git-worktree(1) manual page for more
information.

If git-worktree isn't available, silently falls back to cloning the
repository.

Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Write output files when using -w

At present buildman does not write its own output files (err, done, the
environment) when using -w. However this is useful for when the build is
run with -s to check it.

In fact ProduceResultSummary() reads the result from those files rather
than using the 'result' info directly. So ProcessResult() does not work
with -w at present. It does not print any output.

Fix this by writing output files even when -w is used.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use out-env for environment output

At present the environment used by U-Boot is written to the 'env'
directory. This is fine when the output directory is not the same as the
source directory, but when it is (as with -w) it conflicts with the source
directory of the same name.

Rename 'env' to 'out-env' to fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make sure that -o is given with -w

It is a bad idea to use the default output directory ('..') with -w since
it does a build in that directory and writes various files these.

Require that -o is given to avoid this problem.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Change the exit codes

The current exit codes of 128 and 129 are useful in that they do not
conflict with those returned by tools, but they are not actually valid.
It seems better to pick some codes which work with 'bit bisect run'.

Update them to 100 (for errors) and 101 (for warnings).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make -I the default

At present buildman defaults to running 'mrproper' on every thread before
it starts building commits for each board. This can add a delay of about 5
seconds to the start of the process, since the tools and other invariants
must be rebuilt.

In particular, a build without '-b', to build current source, runs much
slower without -I, since any existing build is removed, thus losing the
possibility of an incremental build.

Partly this behaviour was to avoid strange build-system problems caused by
running 'make defconfig' for one board and then one with a different
architecture. But these problems were fixed quite a while ago.

The -I option (which disabled mrproper) was introduced four years ago and
does not seem to cause any problems with builds.

So make -I the default and deprecate the option. To allow use of
'mrproper', add a new -m flag.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow building within a subdir of the current dir

This is useful in some situations, in particular with -w and when building
in-tree. Now that we are more careful about what we remove in
_PrepareOutputSpace(), it should be safe to relax this restriction.

Update the progress information also so it is clear what buildman is
doing. Remove files can take a long time.

Signed-off-by: Simon Glass <sjg@chromium.org>

bulidman: Add support for a simple build

It is useful to run a simple build and put all the output in a single
directory. Add a -w option to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Perform tests in a temporary directory

We may not always be able to write to the default output directory so
have a temporary directory for our output be created.

Cc: Simon Glass <sjg@chromium.org>
Reviewed-by: Stephen Warren <swarren@wwwdotorg.org>
Suggested-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Tom Rini <trini@konsulko.com>

Remove CONFIG_USE_STDINT

You do not need to use the typedefs provided by compiler.

Our compilers are either IPL32 or LP64. Hence, U-Boot can/should
always use int-ll64.h typedefs like Linux kernel, whatever the
typedefs the compiler internally uses.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

buildman: Extract environment as part of each build

As we're building the boards, extract the default U-Boot environment to
uboot.env so we can interrogate it later.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

tools: buildman: Don't use the working dir as build dir

When the U-Boot base directory happens to have the same name as the branch
that buildman is directed to use via the '-b' option and no output
directory is specified with '-o', buildman happily starts removing the
whole U-Boot sources eventually only stopped with the error message:

OSError: [Errno 20] Not a directory: '../<branch-name>/boards.cfg

Add a check to avoid this and also deal with the case where '-o' points
to the source directory, or any subdirectory of it.

Finally, tidy up the confusing logic for removing the old tree when using
-b. This is only done when building a branch.

Signed-off-by: Lothar Waßmann <LW@KARO-electronics.de>
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Lothar Waßmann <LW@KARO-electronics.de>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

tools: Update python "help" tests to cope with "more" oddities

In some cases when "more" is told to page a given file it will prepend
the output with:
::::::::::::::
/PATH/TO/THE/FILE
::::::::::::::

And when this happens the output will not match the expected length.
Further, if we use a different pager we will instead fail the coverage
tests as we will not have 100% coverage. Update the help test to remove
the string in question.

Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>

powerpc, 5xx: remove some "5xx" remains

we removed 5xx support. So delete some forgotten remains.

Signed-off-by: Heiko Schocher <hs@denx.de>

buildman: Fix the 'help' test to use the correct path

When buildman is run via a symlink, this test fails. Fix it to work the same
way as buildman itself.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow branch names which conflict with directories

At present if you try to use buildman with the branch 'test' it will
complain that it is unsure whether you mean the branch or the directory.
This is a feature of the 'git log' command that buildman uses. Fix it
by resolving the ambiguity.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

buildman: Ignore conflicting tags

Tags like Series-version are normally expected to appear once, and with a
unique value. But buildman doesn't actually look at these tags. So ignore
conflicts.

This allows bulidman to build a branch containing multiple patman series.

Reported-by: Steve Rae <srae@broadcom.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Permit branch names with an embedded '/'

At present buildman naively uses the branch name as part of its directory
path, which causes problems if the name has an embedded '/'.

Replace these with '_' to fix the problem.

Reported-by: Steve Rae <srae@broadcom.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add additional functional tests

This adds coverage of core features of the builder, including the
command-line options which affect building.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow tests to have their own boards

Rather than reading boards.cfg, which may take time to generate and is not
necessarily suitable for running tests, create our own list of boards.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid looking at config file or toolchains in tests

These files may not exist in the environment, or may not be suitable for
testing. Provide our own config file and our own toolchains when running
tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a functional test

Buildman currently lacks testing in many areas, including its use of git,
make and many command-line flags.

Add a functional test which covers some of these areas. So far it does
a fake 'build' of all boards for the current source tree.

This version reads the real ~/.buildman and boards.cfg files. Future work
will improve this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Drop warning about orphaned defconfigs

Some boards use a MAINTAINERS entry to specify common files without
referencing any defconfigs. This is allowed and should not result in a
warning.

Drop the warning in this case.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Specify the output directory in tests

The default output directory is generally '../' in tests so we end up
trying to create '../.bm-work'. This does not work with azure, so update
these tests to use the temporary directory instead.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a way to print the architecture for a board

This is useful for some tools and is easily available for buildman. Add
a new --print-arch option.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Tidy up reporting of a toolchain error

Provide the text of the exception when something goes wrong.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in builder.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in bsettings.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert to argparse

Use argparse to parse the arguments, since OptionParser is deprecated now.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test for --boards

Add a simple functional test for the --boards option. Fix the example in
the docs while we are here. Also improve the docs for Builder.count so it
is clearer what it contains.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in cmdline.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add tests for excluding things

Add some tests for the -x flag.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test for the -A option

This lacks a test at present. Add one.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix most pylint warnings in control

Tidy up the easier-to-fix pylint warnings in module 'control'.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in control.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Provide an argument to the -R option

Allow writing the file to a selected location, since otherwise this is
controlled by the buildman configuration, so cannot be determined by the
caller.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: https://source.denx.de/u-boot/u-boot/-/issues/17

buildman: Add an option to check maintainers

Rather than using the -R option to get this report as a side effect, add
a dedicated option for it.

Disable CI for now as there are some missing maintainers, unfortunately.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct logic for missing maintainers

An orphaned board should produce a warning, as should a missing name for
the maintainer (when '-' is provided). Add these cases.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Sort the maintainer warnings

Sort the warnings into alphabetical order, for easier reading. Also make
sure that the buildman test files are ignored.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Just display a single line for missing maintainers

At present we get multiple lines of output when a board has no MAINTAINERS
entry:

WARNING: no status info for 'bananapi-m2-pro'
WARNING: no maintainers for 'bananapi-m2-pro'

Suppress the 'status' one since it is implied by the other.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct operation of MAINTAINERS N:

This doesn't work as intended. Instead it scans every defconfig file
in the source tree.

Fix it and add a test.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Detect boards with no CONFIG_TARGET defined

We generally expected exactly one of these. Add a check for it.

Note: This warning is not displayed by default. An option will be added
to enable it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Detect boards with multiple CONFIG_TARGETs defined

The TARGET_xxx options are special in that they refer to a single target.
Exactly one should be enabled for each target, corresponding to a
defconfig file.

Detect configs which result in two TARGET_xxx options being set. For
example, at present, TARGET_POLEG and TARET_POLEG_EVB are enabled for the
same board.

Note: This warning is not displayed by default. An option will be added
to enable it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Warn about dangling maintainer entries

Other than the top-level MAINTAINERS file, all maintainer entries should
actually reference a target. Add a warning to detect those that do not.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a partial test for ensure_board_list()

Create a new function which has the non-UI parts of ensure_board_list().
Add some tests for everything except the N: tag.

While we are here, fix the confusing usage of fname inside a loops that
also uses fname.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test for Boards.output_is_new()

Add a test for this code, adjusting the timestamp on various files to
check each use case.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test for Boards.scan_defconfigs()

Add a test for this code. It requires some defconfig files and a test
Kconfig to work with, so copy these into the temporary directory at the
start.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Rename the ARM boards

Use names consistent with their target names.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move library functions into a library directory

The patman directory has a number of modules which are used by other tools
in U-Boot. This makes it hard to package the tools using pypi since the
common files must be copied along with the tool that uses them.

To address this, move these files into a new u_boot_pylib library. This
can be packaged separately and listed as a dependency of each tool.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a flag for reproducible builds

This is quite a useful thing to use when building since it avoids small
size changes between commits. Add a -r flag for it.

Also undefine CONFIG_LOCALVERSION_AUTO since this appends the git hash
to the version string, causing every build to be slightly different.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support disabling LTO

This cuts down build performance considerably and is not always needed,
when checking for build errors, etc.

Add a flag to disable it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Write out the build command used

It is sometimes useful to see the exact 'make' command used by buildman
for a commit. Add an output file for this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add --allow-missing flag to allow missing blobs

Add a new flag to buildman so that we will in turn pass
BINMAN_ALLOW_MISSING=1 to 'make'. Make use of this flag in CI.

Allow the settings file to control this.

Cc: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Reinstate removal of temp output dir in tests

This was dropped my mistake. Reinstate it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: d829f1217c6 ("bulidman: Add support for a simple build")

buildman: Convert documentation to rST

Convert the buildman documentation to rST format and include it in the
'build' section.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Quentin Schulz <foss+uboot@0leil.net>

buildman: Split out Boards into its own file

Use a separate file for the Boards class so that its name matches the
module name.

Fix up the function names to match the pylint style and fix some other
warnings.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in board.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Drop use of 'boards' in control

Use brds instead so that we can reserve 'boards' for a module name.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix use of 'boards' in func_test

We want to create a module called 'boards' so avoid use of this variable
name in this module. Change the global to be capitalised, as required by
Python style.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct pylint errors

Fix pylint errors that can be fixed and mask those that seem to be
incorrect.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in terminal.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in gitutil.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in command.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Provide a hint on how to debug thread crashes

If a thread crashes it is helpful to try the operation again with
threading disabled. Add a hint about that.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add helper functions for updating .config files

At present the only straightforward way to write tests that need a
slightly different configuration is to create a new board with its own
configuration. This is cumbersome.

It would be useful if buildman could adjust the configuration of a build
on the fly. In preparation for this, add a utility library which can
modify a .config file according to various parameters passed to it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use bytes for the environment

At present we sometimes see problems in gitlab where the environment has
0x80 characters or sequences which are not valid UTF-8.

Avoid this by using bytes for the environment, both internal to buildman
and when writing out the 'env' file. Add a test to make sure this works
as expected.

Reported-by: Marek Vasut <marex@denx.de>
Fixes: e5fc79ea718 ("buildman: Write the environment out to an 'env' file")
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Handle exceptions in threads gracefully

There have been at least a few cases where an exception has occurred in a
thread and resulted in buildman hanging: running out of disk space and
getting a unicode error.

Handle these by collecting a list of exceptions, printing them out and
reporting failure if any are found. Add a test for this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Tidy up a few comments

Add some function comments which are missing, or missing arguments.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use git worktrees instead of git clones when possible

This patch makes buildman create linked working trees instead of clones
of the source repository, but keeps updating the older clones of the
repository that might already exist. These worktrees share "everything
except working directory specific files such as HEAD, index, etc." with
the source repository. See the git-worktree(1) manual page for more
information.

If git-worktree isn't available, silently falls back to cloning the
repository.

Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Write output files when using -w

At present buildman does not write its own output files (err, done, the
environment) when using -w. However this is useful for when the build is
run with -s to check it.

In fact ProduceResultSummary() reads the result from those files rather
than using the 'result' info directly. So ProcessResult() does not work
with -w at present. It does not print any output.

Fix this by writing output files even when -w is used.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use out-env for environment output

At present the environment used by U-Boot is written to the 'env'
directory. This is fine when the output directory is not the same as the
source directory, but when it is (as with -w) it conflicts with the source
directory of the same name.

Rename 'env' to 'out-env' to fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make sure that -o is given with -w

It is a bad idea to use the default output directory ('..') with -w since
it does a build in that directory and writes various files these.

Require that -o is given to avoid this problem.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Change the exit codes

The current exit codes of 128 and 129 are useful in that they do not
conflict with those returned by tools, but they are not actually valid.
It seems better to pick some codes which work with 'bit bisect run'.

Update them to 100 (for errors) and 101 (for warnings).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make -I the default

At present buildman defaults to running 'mrproper' on every thread before
it starts building commits for each board. This can add a delay of about 5
seconds to the start of the process, since the tools and other invariants
must be rebuilt.

In particular, a build without '-b', to build current source, runs much
slower without -I, since any existing build is removed, thus losing the
possibility of an incremental build.

Partly this behaviour was to avoid strange build-system problems caused by
running 'make defconfig' for one board and then one with a different
architecture. But these problems were fixed quite a while ago.

The -I option (which disabled mrproper) was introduced four years ago and
does not seem to cause any problems with builds.

So make -I the default and deprecate the option. To allow use of
'mrproper', add a new -m flag.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow building within a subdir of the current dir

This is useful in some situations, in particular with -w and when building
in-tree. Now that we are more careful about what we remove in
_PrepareOutputSpace(), it should be safe to relax this restriction.

Update the progress information also so it is clear what buildman is
doing. Remove files can take a long time.

Signed-off-by: Simon Glass <sjg@chromium.org>

bulidman: Add support for a simple build

It is useful to run a simple build and put all the output in a single
directory. Add a -w option to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Perform tests in a temporary directory

We may not always be able to write to the default output directory so
have a temporary directory for our output be created.

Cc: Simon Glass <sjg@chromium.org>
Reviewed-by: Stephen Warren <swarren@wwwdotorg.org>
Suggested-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Tom Rini <trini@konsulko.com>

Remove CONFIG_USE_STDINT

You do not need to use the typedefs provided by compiler.

Our compilers are either IPL32 or LP64. Hence, U-Boot can/should
always use int-ll64.h typedefs like Linux kernel, whatever the
typedefs the compiler internally uses.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

buildman: Extract environment as part of each build

As we're building the boards, extract the default U-Boot environment to
uboot.env so we can interrogate it later.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

tools: buildman: Don't use the working dir as build dir

When the U-Boot base directory happens to have the same name as the branch
that buildman is directed to use via the '-b' option and no output
directory is specified with '-o', buildman happily starts removing the
whole U-Boot sources eventually only stopped with the error message:

OSError: [Errno 20] Not a directory: '../<branch-name>/boards.cfg

Add a check to avoid this and also deal with the case where '-o' points
to the source directory, or any subdirectory of it.

Finally, tidy up the confusing logic for removing the old tree when using
-b. This is only done when building a branch.

Signed-off-by: Lothar Waßmann <LW@KARO-electronics.de>
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Lothar Waßmann <LW@KARO-electronics.de>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

tools: Update python "help" tests to cope with "more" oddities

In some cases when "more" is told to page a given file it will prepend
the output with:
::::::::::::::
/PATH/TO/THE/FILE
::::::::::::::

And when this happens the output will not match the expected length.
Further, if we use a different pager we will instead fail the coverage
tests as we will not have 100% coverage. Update the help test to remove
the string in question.

Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>

powerpc, 5xx: remove some "5xx" remains

we removed 5xx support. So delete some forgotten remains.

Signed-off-by: Heiko Schocher <hs@denx.de>

buildman: Fix the 'help' test to use the correct path

When buildman is run via a symlink, this test fails. Fix it to work the same
way as buildman itself.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow branch names which conflict with directories

At present if you try to use buildman with the branch 'test' it will
complain that it is unsure whether you mean the branch or the directory.
This is a feature of the 'git log' command that buildman uses. Fix it
by resolving the ambiguity.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

buildman: Ignore conflicting tags

Tags like Series-version are normally expected to appear once, and with a
unique value. But buildman doesn't actually look at these tags. So ignore
conflicts.

This allows bulidman to build a branch containing multiple patman series.

Reported-by: Steve Rae <srae@broadcom.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Permit branch names with an embedded '/'

At present buildman naively uses the branch name as part of its directory
path, which causes problems if the name has an embedded '/'.

Replace these with '_' to fix the problem.

Reported-by: Steve Rae <srae@broadcom.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add additional functional tests

This adds coverage of core features of the builder, including the
command-line options which affect building.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow tests to have their own boards

Rather than reading boards.cfg, which may take time to generate and is not
necessarily suitable for running tests, create our own list of boards.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid looking at config file or toolchains in tests

These files may not exist in the environment, or may not be suitable for
testing. Provide our own config file and our own toolchains when running
tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a functional test

Buildman currently lacks testing in many areas, including its use of git,
make and many command-line flags.

Add a functional test which covers some of these areas. So far it does
a fake 'build' of all boards for the current source tree.

This version reads the real ~/.buildman and boards.cfg files. Future work
will improve this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Specify the output directory in tests

The default output directory is generally '../' in tests so we end up
trying to create '../.bm-work'. This does not work with azure, so update
these tests to use the temporary directory instead.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a way to print the architecture for a board

This is useful for some tools and is easily available for buildman. Add
a new --print-arch option.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Tidy up reporting of a toolchain error

Provide the text of the exception when something goes wrong.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in builder.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in bsettings.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert to argparse

Use argparse to parse the arguments, since OptionParser is deprecated now.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test for --boards

Add a simple functional test for the --boards option. Fix the example in
the docs while we are here. Also improve the docs for Builder.count so it
is clearer what it contains.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in cmdline.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add tests for excluding things

Add some tests for the -x flag.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test for the -A option

This lacks a test at present. Add one.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix most pylint warnings in control

Tidy up the easier-to-fix pylint warnings in module 'control'.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in control.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Provide an argument to the -R option

Allow writing the file to a selected location, since otherwise this is
controlled by the buildman configuration, so cannot be determined by the
caller.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: https://source.denx.de/u-boot/u-boot/-/issues/17

buildman: Add an option to check maintainers

Rather than using the -R option to get this report as a side effect, add
a dedicated option for it.

Disable CI for now as there are some missing maintainers, unfortunately.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct logic for missing maintainers

An orphaned board should produce a warning, as should a missing name for
the maintainer (when '-' is provided). Add these cases.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Sort the maintainer warnings

Sort the warnings into alphabetical order, for easier reading. Also make
sure that the buildman test files are ignored.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Just display a single line for missing maintainers

At present we get multiple lines of output when a board has no MAINTAINERS
entry:

WARNING: no status info for 'bananapi-m2-pro'
WARNING: no maintainers for 'bananapi-m2-pro'

Suppress the 'status' one since it is implied by the other.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct operation of MAINTAINERS N:

This doesn't work as intended. Instead it scans every defconfig file
in the source tree.

Fix it and add a test.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Detect boards with no CONFIG_TARGET defined

We generally expected exactly one of these. Add a check for it.

Note: This warning is not displayed by default. An option will be added
to enable it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Detect boards with multiple CONFIG_TARGETs defined

The TARGET_xxx options are special in that they refer to a single target.
Exactly one should be enabled for each target, corresponding to a
defconfig file.

Detect configs which result in two TARGET_xxx options being set. For
example, at present, TARGET_POLEG and TARET_POLEG_EVB are enabled for the
same board.

Note: This warning is not displayed by default. An option will be added
to enable it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Warn about dangling maintainer entries

Other than the top-level MAINTAINERS file, all maintainer entries should
actually reference a target. Add a warning to detect those that do not.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a partial test for ensure_board_list()

Create a new function which has the non-UI parts of ensure_board_list().
Add some tests for everything except the N: tag.

While we are here, fix the confusing usage of fname inside a loops that
also uses fname.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test for Boards.output_is_new()

Add a test for this code, adjusting the timestamp on various files to
check each use case.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a test for Boards.scan_defconfigs()

Add a test for this code. It requires some defconfig files and a test
Kconfig to work with, so copy these into the temporary directory at the
start.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Rename the ARM boards

Use names consistent with their target names.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move library functions into a library directory

The patman directory has a number of modules which are used by other tools
in U-Boot. This makes it hard to package the tools using pypi since the
common files must be copied along with the tool that uses them.

To address this, move these files into a new u_boot_pylib library. This
can be packaged separately and listed as a dependency of each tool.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a flag for reproducible builds

This is quite a useful thing to use when building since it avoids small
size changes between commits. Add a -r flag for it.

Also undefine CONFIG_LOCALVERSION_AUTO since this appends the git hash
to the version string, causing every build to be slightly different.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support disabling LTO

This cuts down build performance considerably and is not always needed,
when checking for build errors, etc.

Add a flag to disable it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Write out the build command used

It is sometimes useful to see the exact 'make' command used by buildman
for a commit. Add an output file for this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add --allow-missing flag to allow missing blobs

Add a new flag to buildman so that we will in turn pass
BINMAN_ALLOW_MISSING=1 to 'make'. Make use of this flag in CI.

Allow the settings file to control this.

Cc: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Reinstate removal of temp output dir in tests

This was dropped my mistake. Reinstate it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: d829f1217c6 ("bulidman: Add support for a simple build")

buildman: Convert documentation to rST

Convert the buildman documentation to rST format and include it in the
'build' section.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Quentin Schulz <foss+uboot@0leil.net>

buildman: Split out Boards into its own file

Use a separate file for the Boards class so that its name matches the
module name.

Fix up the function names to match the pylint style and fix some other
warnings.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in board.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Drop use of 'boards' in control

Use brds instead so that we can reserve 'boards' for a module name.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix use of 'boards' in func_test

We want to create a module called 'boards' so avoid use of this variable
name in this module. Change the global to be capitalised, as required by
Python style.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct pylint errors

Fix pylint errors that can be fixed and mask those that seem to be
incorrect.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in terminal.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in gitutil.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in command.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Provide a hint on how to debug thread crashes

If a thread crashes it is helpful to try the operation again with
threading disabled. Add a hint about that.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add helper functions for updating .config files

At present the only straightforward way to write tests that need a
slightly different configuration is to create a new board with its own
configuration. This is cumbersome.

It would be useful if buildman could adjust the configuration of a build
on the fly. In preparation for this, add a utility library which can
modify a .config file according to various parameters passed to it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use bytes for the environment

At present we sometimes see problems in gitlab where the environment has
0x80 characters or sequences which are not valid UTF-8.

Avoid this by using bytes for the environment, both internal to buildman
and when writing out the 'env' file. Add a test to make sure this works
as expected.

Reported-by: Marek Vasut <marex@denx.de>
Fixes: e5fc79ea718 ("buildman: Write the environment out to an 'env' file")
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Handle exceptions in threads gracefully

There have been at least a few cases where an exception has occurred in a
thread and resulted in buildman hanging: running out of disk space and
getting a unicode error.

Handle these by collecting a list of exceptions, printing them out and
reporting failure if any are found. Add a test for this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Tidy up a few comments

Add some function comments which are missing, or missing arguments.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use git worktrees instead of git clones when possible

This patch makes buildman create linked working trees instead of clones
of the source repository, but keeps updating the older clones of the
repository that might already exist. These worktrees share "everything
except working directory specific files such as HEAD, index, etc." with
the source repository. See the git-worktree(1) manual page for more
information.

If git-worktree isn't available, silently falls back to cloning the
repository.

Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Write output files when using -w

At present buildman does not write its own output files (err, done, the
environment) when using -w. However this is useful for when the build is
run with -s to check it.

In fact ProduceResultSummary() reads the result from those files rather
than using the 'result' info directly. So ProcessResult() does not work
with -w at present. It does not print any output.

Fix this by writing output files even when -w is used.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use out-env for environment output

At present the environment used by U-Boot is written to the 'env'
directory. This is fine when the output directory is not the same as the
source directory, but when it is (as with -w) it conflicts with the source
directory of the same name.

Rename 'env' to 'out-env' to fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make sure that -o is given with -w

It is a bad idea to use the default output directory ('..') with -w since
it does a build in that directory and writes various files these.

Require that -o is given to avoid this problem.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Change the exit codes

The current exit codes of 128 and 129 are useful in that they do not
conflict with those returned by tools, but they are not actually valid.
It seems better to pick some codes which work with 'bit bisect run'.

Update them to 100 (for errors) and 101 (for warnings).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make -I the default

At present buildman defaults to running 'mrproper' on every thread before
it starts building commits for each board. This can add a delay of about 5
seconds to the start of the process, since the tools and other invariants
must be rebuilt.

In particular, a build without '-b', to build current source, runs much
slower without -I, since any existing build is removed, thus losing the
possibility of an incremental build.

Partly this behaviour was to avoid strange build-system problems caused by
running 'make defconfig' for one board and then one with a different
architecture. But these problems were fixed quite a while ago.

The -I option (which disabled mrproper) was introduced four years ago and
does not seem to cause any problems with builds.

So make -I the default and deprecate the option. To allow use of
'mrproper', add a new -m flag.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow building within a subdir of the current dir

This is useful in some situations, in particular with -w and when building
in-tree. Now that we are more careful about what we remove in
_PrepareOutputSpace(), it should be safe to relax this restriction.

Update the progress information also so it is clear what buildman is
doing. Remove files can take a long time.

Signed-off-by: Simon Glass <sjg@chromium.org>

bulidman: Add support for a simple build

It is useful to run a simple build and put all the output in a single
directory. Add a -w option to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Perform tests in a temporary directory

We may not always be able to write to the default output directory so
have a temporary directory for our output be created.

Cc: Simon Glass <sjg@chromium.org>
Reviewed-by: Stephen Warren <swarren@wwwdotorg.org>
Suggested-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Tom Rini <trini@konsulko.com>

Remove CONFIG_USE_STDINT

You do not need to use the typedefs provided by compiler.

Our compilers are either IPL32 or LP64. Hence, U-Boot can/should
always use int-ll64.h typedefs like Linux kernel, whatever the
typedefs the compiler internally uses.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

buildman: Extract environment as part of each build

As we're building the boards, extract the default U-Boot environment to
uboot.env so we can interrogate it later.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

tools: buildman: Don't use the working dir as build dir

When the U-Boot base directory happens to have the same name as the branch
that buildman is directed to use via the '-b' option and no output
directory is specified with '-o', buildman happily starts removing the
whole U-Boot sources eventually only stopped with the error message:

OSError: [Errno 20] Not a directory: '../<branch-name>/boards.cfg

Add a check to avoid this and also deal with the case where '-o' points
to the source directory, or any subdirectory of it.

Finally, tidy up the confusing logic for removing the old tree when using
-b. This is only done when building a branch.

Signed-off-by: Lothar Waßmann <LW@KARO-electronics.de>
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Lothar Waßmann <LW@KARO-electronics.de>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

tools: Update python "help" tests to cope with "more" oddities

In some cases when "more" is told to page a given file it will prepend
the output with:
::::::::::::::
/PATH/TO/THE/FILE
::::::::::::::

And when this happens the output will not match the expected length.
Further, if we use a different pager we will instead fail the coverage
tests as we will not have 100% coverage. Update the help test to remove
the string in question.

Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>

powerpc, 5xx: remove some "5xx" remains

we removed 5xx support. So delete some forgotten remains.

Signed-off-by: Heiko Schocher <hs@denx.de>

buildman: Fix the 'help' test to use the correct path

When buildman is run via a symlink, this test fails. Fix it to work the same
way as buildman itself.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow branch names which conflict with directories

At present if you try to use buildman with the branch 'test' it will
complain that it is unsure whether you mean the branch or the directory.
This is a feature of the 'git log' command that buildman uses. Fix it
by resolving the ambiguity.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

buildman: Ignore conflicting tags

Tags like Series-version are normally expected to appear once, and with a
unique value. But buildman doesn't actually look at these tags. So ignore
conflicts.

This allows bulidman to build a branch containing multiple patman series.

Reported-by: Steve Rae <srae@broadcom.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Permit branch names with an embedded '/'

At present buildman naively uses the branch name as part of its directory
path, which causes problems if the name has an embedded '/'.

Replace these with '_' to fix the problem.

Reported-by: Steve Rae <srae@broadcom.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add additional functional tests

This adds coverage of core features of the builder, including the
command-line options which affect building.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow tests to have their own boards

Rather than reading boards.cfg, which may take time to generate and is not
necessarily suitable for running tests, create our own list of boards.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid looking at config file or toolchains in tests

These files may not exist in the environment, or may not be suitable for
testing. Provide our own config file and our own toolchains when running
tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a functional test

Buildman currently lacks testing in many areas, including its use of git,
make and many command-line flags.

Add a functional test which covers some of these areas. So far it does
a fake 'build' of all boards for the current source tree.

This version reads the real ~/.buildman and boards.cfg files. Future work
will improve this.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Move library functions into a library directory

The patman directory has a number of modules which are used by other tools
in U-Boot. This makes it hard to package the tools using pypi since the
common files must be copied along with the tool that uses them.

To address this, move these files into a new u_boot_pylib library. This
can be packaged separately and listed as a dependency of each tool.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a flag for reproducible builds

This is quite a useful thing to use when building since it avoids small
size changes between commits. Add a -r flag for it.

Also undefine CONFIG_LOCALVERSION_AUTO since this appends the git hash
to the version string, causing every build to be slightly different.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Support disabling LTO

This cuts down build performance considerably and is not always needed,
when checking for build errors, etc.

Add a flag to disable it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Write out the build command used

It is sometimes useful to see the exact 'make' command used by buildman
for a commit. Add an output file for this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add --allow-missing flag to allow missing blobs

Add a new flag to buildman so that we will in turn pass
BINMAN_ALLOW_MISSING=1 to 'make'. Make use of this flag in CI.

Allow the settings file to control this.

Cc: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Reinstate removal of temp output dir in tests

This was dropped my mistake. Reinstate it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: d829f1217c6 ("bulidman: Add support for a simple build")

buildman: Convert documentation to rST

Convert the buildman documentation to rST format and include it in the
'build' section.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Quentin Schulz <foss+uboot@0leil.net>

buildman: Split out Boards into its own file

Use a separate file for the Boards class so that its name matches the
module name.

Fix up the function names to match the pylint style and fix some other
warnings.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in board.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Drop use of 'boards' in control

Use brds instead so that we can reserve 'boards' for a module name.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix use of 'boards' in func_test

We want to create a module called 'boards' so avoid use of this variable
name in this module. Change the global to be capitalised, as required by
Python style.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct pylint errors

Fix pylint errors that can be fixed and mask those that seem to be
incorrect.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in terminal.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in gitutil.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in command.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Provide a hint on how to debug thread crashes

If a thread crashes it is helpful to try the operation again with
threading disabled. Add a hint about that.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add helper functions for updating .config files

At present the only straightforward way to write tests that need a
slightly different configuration is to create a new board with its own
configuration. This is cumbersome.

It would be useful if buildman could adjust the configuration of a build
on the fly. In preparation for this, add a utility library which can
modify a .config file according to various parameters passed to it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use bytes for the environment

At present we sometimes see problems in gitlab where the environment has
0x80 characters or sequences which are not valid UTF-8.

Avoid this by using bytes for the environment, both internal to buildman
and when writing out the 'env' file. Add a test to make sure this works
as expected.

Reported-by: Marek Vasut <marex@denx.de>
Fixes: e5fc79ea718 ("buildman: Write the environment out to an 'env' file")
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Handle exceptions in threads gracefully

There have been at least a few cases where an exception has occurred in a
thread and resulted in buildman hanging: running out of disk space and
getting a unicode error.

Handle these by collecting a list of exceptions, printing them out and
reporting failure if any are found. Add a test for this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Tidy up a few comments

Add some function comments which are missing, or missing arguments.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use git worktrees instead of git clones when possible

This patch makes buildman create linked working trees instead of clones
of the source repository, but keeps updating the older clones of the
repository that might already exist. These worktrees share "everything
except working directory specific files such as HEAD, index, etc." with
the source repository. See the git-worktree(1) manual page for more
information.

If git-worktree isn't available, silently falls back to cloning the
repository.

Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Write output files when using -w

At present buildman does not write its own output files (err, done, the
environment) when using -w. However this is useful for when the build is
run with -s to check it.

In fact ProduceResultSummary() reads the result from those files rather
than using the 'result' info directly. So ProcessResult() does not work
with -w at present. It does not print any output.

Fix this by writing output files even when -w is used.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use out-env for environment output

At present the environment used by U-Boot is written to the 'env'
directory. This is fine when the output directory is not the same as the
source directory, but when it is (as with -w) it conflicts with the source
directory of the same name.

Rename 'env' to 'out-env' to fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make sure that -o is given with -w

It is a bad idea to use the default output directory ('..') with -w since
it does a build in that directory and writes various files these.

Require that -o is given to avoid this problem.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Change the exit codes

The current exit codes of 128 and 129 are useful in that they do not
conflict with those returned by tools, but they are not actually valid.
It seems better to pick some codes which work with 'bit bisect run'.

Update them to 100 (for errors) and 101 (for warnings).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make -I the default

At present buildman defaults to running 'mrproper' on every thread before
it starts building commits for each board. This can add a delay of about 5
seconds to the start of the process, since the tools and other invariants
must be rebuilt.

In particular, a build without '-b', to build current source, runs much
slower without -I, since any existing build is removed, thus losing the
possibility of an incremental build.

Partly this behaviour was to avoid strange build-system problems caused by
running 'make defconfig' for one board and then one with a different
architecture. But these problems were fixed quite a while ago.

The -I option (which disabled mrproper) was introduced four years ago and
does not seem to cause any problems with builds.

So make -I the default and deprecate the option. To allow use of
'mrproper', add a new -m flag.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow building within a subdir of the current dir

This is useful in some situations, in particular with -w and when building
in-tree. Now that we are more careful about what we remove in
_PrepareOutputSpace(), it should be safe to relax this restriction.

Update the progress information also so it is clear what buildman is
doing. Remove files can take a long time.

Signed-off-by: Simon Glass <sjg@chromium.org>

bulidman: Add support for a simple build

It is useful to run a simple build and put all the output in a single
directory. Add a -w option to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Perform tests in a temporary directory

We may not always be able to write to the default output directory so
have a temporary directory for our output be created.

Cc: Simon Glass <sjg@chromium.org>
Reviewed-by: Stephen Warren <swarren@wwwdotorg.org>
Suggested-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Tom Rini <trini@konsulko.com>

Remove CONFIG_USE_STDINT

You do not need to use the typedefs provided by compiler.

Our compilers are either IPL32 or LP64. Hence, U-Boot can/should
always use int-ll64.h typedefs like Linux kernel, whatever the
typedefs the compiler internally uses.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

buildman: Extract environment as part of each build

As we're building the boards, extract the default U-Boot environment to
uboot.env so we can interrogate it later.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

tools: buildman: Don't use the working dir as build dir

When the U-Boot base directory happens to have the same name as the branch
that buildman is directed to use via the '-b' option and no output
directory is specified with '-o', buildman happily starts removing the
whole U-Boot sources eventually only stopped with the error message:

OSError: [Errno 20] Not a directory: '../<branch-name>/boards.cfg

Add a check to avoid this and also deal with the case where '-o' points
to the source directory, or any subdirectory of it.

Finally, tidy up the confusing logic for removing the old tree when using
-b. This is only done when building a branch.

Signed-off-by: Lothar Waßmann <LW@KARO-electronics.de>
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Lothar Waßmann <LW@KARO-electronics.de>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

tools: Update python "help" tests to cope with "more" oddities

In some cases when "more" is told to page a given file it will prepend
the output with:
::::::::::::::
/PATH/TO/THE/FILE
::::::::::::::

And when this happens the output will not match the expected length.
Further, if we use a different pager we will instead fail the coverage
tests as we will not have 100% coverage. Update the help test to remove
the string in question.

Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>

powerpc, 5xx: remove some "5xx" remains

we removed 5xx support. So delete some forgotten remains.

Signed-off-by: Heiko Schocher <hs@denx.de>

buildman: Fix the 'help' test to use the correct path

When buildman is run via a symlink, this test fails. Fix it to work the same
way as buildman itself.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow branch names which conflict with directories

At present if you try to use buildman with the branch 'test' it will
complain that it is unsure whether you mean the branch or the directory.
This is a feature of the 'git log' command that buildman uses. Fix it
by resolving the ambiguity.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

buildman: Ignore conflicting tags

Tags like Series-version are normally expected to appear once, and with a
unique value. But buildman doesn't actually look at these tags. So ignore
conflicts.

This allows bulidman to build a branch containing multiple patman series.

Reported-by: Steve Rae <srae@broadcom.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Permit branch names with an embedded '/'

At present buildman naively uses the branch name as part of its directory
path, which causes problems if the name has an embedded '/'.

Replace these with '_' to fix the problem.

Reported-by: Steve Rae <srae@broadcom.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add additional functional tests

This adds coverage of core features of the builder, including the
command-line options which affect building.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow tests to have their own boards

Rather than reading boards.cfg, which may take time to generate and is not
necessarily suitable for running tests, create our own list of boards.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid looking at config file or toolchains in tests

These files may not exist in the environment, or may not be suitable for
testing. Provide our own config file and our own toolchains when running
tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a functional test

Buildman currently lacks testing in many areas, including its use of git,
make and many command-line flags.

Add a functional test which covers some of these areas. So far it does
a fake 'build' of all boards for the current source tree.

This version reads the real ~/.buildman and boards.cfg files. Future work
will improve this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add --allow-missing flag to allow missing blobs

Add a new flag to buildman so that we will in turn pass
BINMAN_ALLOW_MISSING=1 to 'make'. Make use of this flag in CI.

Allow the settings file to control this.

Cc: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Reinstate removal of temp output dir in tests

This was dropped my mistake. Reinstate it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: d829f1217c6 ("bulidman: Add support for a simple build")

buildman: Convert documentation to rST

Convert the buildman documentation to rST format and include it in the
'build' section.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Quentin Schulz <foss+uboot@0leil.net>

buildman: Split out Boards into its own file

Use a separate file for the Boards class so that its name matches the
module name.

Fix up the function names to match the pylint style and fix some other
warnings.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in board.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Drop use of 'boards' in control

Use brds instead so that we can reserve 'boards' for a module name.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix use of 'boards' in func_test

We want to create a module called 'boards' so avoid use of this variable
name in this module. Change the global to be capitalised, as required by
Python style.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct pylint errors

Fix pylint errors that can be fixed and mask those that seem to be
incorrect.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in terminal.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in gitutil.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in command.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Provide a hint on how to debug thread crashes

If a thread crashes it is helpful to try the operation again with
threading disabled. Add a hint about that.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add helper functions for updating .config files

At present the only straightforward way to write tests that need a
slightly different configuration is to create a new board with its own
configuration. This is cumbersome.

It would be useful if buildman could adjust the configuration of a build
on the fly. In preparation for this, add a utility library which can
modify a .config file according to various parameters passed to it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use bytes for the environment

At present we sometimes see problems in gitlab where the environment has
0x80 characters or sequences which are not valid UTF-8.

Avoid this by using bytes for the environment, both internal to buildman
and when writing out the 'env' file. Add a test to make sure this works
as expected.

Reported-by: Marek Vasut <marex@denx.de>
Fixes: e5fc79ea718 ("buildman: Write the environment out to an 'env' file")
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Handle exceptions in threads gracefully

There have been at least a few cases where an exception has occurred in a
thread and resulted in buildman hanging: running out of disk space and
getting a unicode error.

Handle these by collecting a list of exceptions, printing them out and
reporting failure if any are found. Add a test for this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Tidy up a few comments

Add some function comments which are missing, or missing arguments.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use git worktrees instead of git clones when possible

This patch makes buildman create linked working trees instead of clones
of the source repository, but keeps updating the older clones of the
repository that might already exist. These worktrees share "everything
except working directory specific files such as HEAD, index, etc." with
the source repository. See the git-worktree(1) manual page for more
information.

If git-worktree isn't available, silently falls back to cloning the
repository.

Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>

patman: Move to absolute imports

At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Move to absolute imports

At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Write output files when using -w

At present buildman does not write its own output files (err, done, the
environment) when using -w. However this is useful for when the build is
run with -s to check it.

In fact ProduceResultSummary() reads the result from those files rather
than using the 'result' info directly. So ProcessResult() does not work
with -w at present. It does not print any output.

Fix this by writing output files even when -w is used.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use out-env for environment output

At present the environment used by U-Boot is written to the 'env'
directory. This is fine when the output directory is not the same as the
source directory, but when it is (as with -w) it conflicts with the source
directory of the same name.

Rename 'env' to 'out-env' to fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make sure that -o is given with -w

It is a bad idea to use the default output directory ('..') with -w since
it does a build in that directory and writes various files these.

Require that -o is given to avoid this problem.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Change the exit codes

The current exit codes of 128 and 129 are useful in that they do not
conflict with those returned by tools, but they are not actually valid.
It seems better to pick some codes which work with 'bit bisect run'.

Update them to 100 (for errors) and 101 (for warnings).

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Make -I the default

At present buildman defaults to running 'mrproper' on every thread before
it starts building commits for each board. This can add a delay of about 5
seconds to the start of the process, since the tools and other invariants
must be rebuilt.

In particular, a build without '-b', to build current source, runs much
slower without -I, since any existing build is removed, thus losing the
possibility of an incremental build.

Partly this behaviour was to avoid strange build-system problems caused by
running 'make defconfig' for one board and then one with a different
architecture. But these problems were fixed quite a while ago.

The -I option (which disabled mrproper) was introduced four years ago and
does not seem to cause any problems with builds.

So make -I the default and deprecate the option. To allow use of
'mrproper', add a new -m flag.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow building within a subdir of the current dir

This is useful in some situations, in particular with -w and when building
in-tree. Now that we are more careful about what we remove in
_PrepareOutputSpace(), it should be safe to relax this restriction.

Update the progress information also so it is clear what buildman is
doing. Remove files can take a long time.

Signed-off-by: Simon Glass <sjg@chromium.org>

bulidman: Add support for a simple build

It is useful to run a simple build and put all the output in a single
directory. Add a -w option to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert to Python 3

Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Perform tests in a temporary directory

We may not always be able to write to the default output directory so
have a temporary directory for our output be created.

Cc: Simon Glass <sjg@chromium.org>
Reviewed-by: Stephen Warren <swarren@wwwdotorg.org>
Suggested-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Tom Rini <trini@konsulko.com>

Remove CONFIG_USE_STDINT

You do not need to use the typedefs provided by compiler.

Our compilers are either IPL32 or LP64. Hence, U-Boot can/should
always use int-ll64.h typedefs like Linux kernel, whatever the
typedefs the compiler internally uses.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

buildman: Extract environment as part of each build

As we're building the boards, extract the default U-Boot environment to
uboot.env so we can interrogate it later.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>

tools: buildman: Don't use the working dir as build dir

When the U-Boot base directory happens to have the same name as the branch
that buildman is directed to use via the '-b' option and no output
directory is specified with '-o', buildman happily starts removing the
whole U-Boot sources eventually only stopped with the error message:

OSError: [Errno 20] Not a directory: '../<branch-name>/boards.cfg

Add a check to avoid this and also deal with the case where '-o' points
to the source directory, or any subdirectory of it.

Finally, tidy up the confusing logic for removing the old tree when using
-b. This is only done when building a branch.

Signed-off-by: Lothar Waßmann <LW@KARO-electronics.de>
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Lothar Waßmann <LW@KARO-electronics.de>

SPDX: Convert all of our single license tags to Linux Kernel style

When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>

tools: Update python "help" tests to cope with "more" oddities

In some cases when "more" is told to page a given file it will prepend
the output with:
::::::::::::::
/PATH/TO/THE/FILE
::::::::::::::

And when this happens the output will not match the expected length.
Further, if we use a different pager we will instead fail the coverage
tests as we will not have 100% coverage. Update the help test to remove
the string in question.

Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>

powerpc, 5xx: remove some "5xx" remains

we removed 5xx support. So delete some forgotten remains.

Signed-off-by: Heiko Schocher <hs@denx.de>

buildman: Fix the 'help' test to use the correct path

When buildman is run via a symlink, this test fails. Fix it to work the same
way as buildman itself.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow branch names which conflict with directories

At present if you try to use buildman with the branch 'test' it will
complain that it is unsure whether you mean the branch or the directory.
This is a feature of the 'git log' command that buildman uses. Fix it
by resolving the ambiguity.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Joe Hershberger <joe.hershberger@ni.com>

buildman: Ignore conflicting tags

Tags like Series-version are normally expected to appear once, and with a
unique value. But buildman doesn't actually look at these tags. So ignore
conflicts.

This allows bulidman to build a branch containing multiple patman series.

Reported-by: Steve Rae <srae@broadcom.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Permit branch names with an embedded '/'

At present buildman naively uses the branch name as part of its directory
path, which causes problems if the name has an embedded '/'.

Replace these with '_' to fix the problem.

Reported-by: Steve Rae <srae@broadcom.com>
Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add additional functional tests

This adds coverage of core features of the builder, including the
command-line options which affect building.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Allow tests to have their own boards

Rather than reading boards.cfg, which may take time to generate and is not
necessarily suitable for running tests, create our own list of boards.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid looking at config file or toolchains in tests

These files may not exist in the environment, or may not be suitable for
testing. Provide our own config file and our own toolchains when running
tests.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add a functional test

Buildman currently lacks testing in many areas, including its use of git,
make and many command-line flags.

Add a functional test which covers some of these areas. So far it does
a fake 'build' of all boards for the current source tree.

This version reads the real ~/.buildman and boards.cfg files. Future work
will improve this.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Split out Boards into its own file

Use a separate file for the Boards class so that its name matches the
module name.

Fix up the function names to match the pylint style and fix some other
warnings.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Convert camel case in board.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Drop use of 'boards' in control

Use brds instead so that we can reserve 'boards' for a module name.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Fix use of 'boards' in func_test

We want to create a module called 'boards' so avoid use of this variable
name in this module. Change the global to be capitalised, as required by
Python style.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Avoid using board as a variable

We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Correct pylint errors

Fix pylint errors that can be fixed and mask those that seem to be
incorrect.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in terminal.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in gitutil.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in command.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

patman: Convert camel case in tools.py

Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Provide a hint on how to debug thread crashes

If a thread crashes it is helpful to try the operation again with
threading disabled. Add a hint about that.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Add helper functions for updating .config files

At present the only straightforward way to write tests that need a
slightly different configuration is to create a new board with its own
configuration. This is cumbersome.

It would be useful if buildman could adjust the configuration of a build
on the fly. In preparation for this, add a utility library which can
modify a .config file according to various parameters passed to it.

Signed-off-by: Simon Glass <sjg@chromium.org>

buildman: Use bytes for the environment

At present we sometimes see problems in gitlab where the environment has
0x80 characters or sequences which are not valid UTF-8.

Avoid this by using bytes for the environment, both internal to buildman
and when writing out the 'env' file. Add a test to make sure this works
as expected.

Reported-by: Marek Vasut <marex@denx.de>
Fixes: e5fc79ea718 ("buildman: Write the environment out to an 'env' file")
Signed-off-by: Simon Glass <sjg@chromium.org>