Merge llvm-project main llvmorg-17-init-19304-gd0b54bb50e51

This updates llvm, clang, compiler-rt, libc++, libunwind, lld, lldb and
openmp to llvm-project main llvmorg-17-init-19304-gd0b54bb50e51, the
last commit before the upstream release/17.x branch was created.

PR: 273753
MFC after: 1 month

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

Fix bootstrapping to actually build lldb-tblgen for later use

Because MK_LLDB=no is in BSARGS, the bootstrap-tools recursive make does
not add lldb-tblgen to _clang_tblgen, causing it to not be built. This
means that the build currently always uses the host's lldb-tblgen
(which, whilst currently it appears to work, could in future break if
TableGen backends are added or altered) and, if it doesn't exist (either
because the current FreeBSD system was built with it disabled, or you're
building on macOS/Linux), fails. Linux and macOS cross-builds used to
work simply because LLDB was previously in BROKEN_OPTIONS when building
on non-FreeBSD.

Instead, move MK_LLDB=no from BSARGS to XMAKE. This ensures that the
lib/clang build in cross-tools continues to not build LLDB parts for the
bootstrap toolchain (both to save time/space on FreeBSD, and because our
vendored LLDB does not include the macOS and Linux host files so those
would fail to build).

The DIRDEPS target is updated to move MK_LLDB=no from the BSARGS block
that mirrors Makefile.inc1 to the line that disables additional
toolchain components. The DIRDEPS build likely suffers from the same
issue currently, but having never used it and not being familiar with
how it works I am leaving that as-is. If it does suffer from the same
issue it should be easily reproducible by renaming /usr/bin/lldb-tblgen
or moving it to a directory not in PATH.

Fixes: 31ba4ce8898f ("Allow bootstrapping llvm-tblgen on macOS and Linux")
MFC after: 1 week
Reviewed by: dim, emaste, imp
Differential Revision: https://reviews.freebsd.org/D31531

Prefer MK_SSP=no to SSP_CFLAGS=

It is more idiomatic. CFLAGS is only augmented with $SSP_CFLAGS when
$MK_SSP != "no".

Reviewed by: imp
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D31401

mk: format some option lists

Alphabetize and give each option its own line, ahead of making another
change to these lists. This makes future diffs easier to read.

Reviewed by: imp, emaste
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D31399

Rename NO_WERROR -> MK_WERROR=no

As suggested in D27598. This also supports MK_WERROR.clang=no and
MK_WERROR.gcc=no to support the existing NO_WERROR.<compiler> uses.

Reviewed By: brooks
Differential Revision: https://reviews.freebsd.org/D27601

Stop passing MK_WARNS=no for bootstrap tools

I got annoyed by the number of warnings that the CheriBSD build was
emitting. It turns out that we are emitting lots of warnings during
bootstrap because bootstrap tools are built with the default compiler
flags and ignore the warnings flags that are set in bsd.sys.mk.

Looking at git blame, it appears that MK_WARNS=no has been passed since
rS112869, replacing the -DNO_WERROR option that was added in rS87775.

This commit changes MK_WARNS=no back to -DNO_WERROR. We need to pass
-DNO_WERROR, since the system compiler might have new warnings that we
don't know about yet, and we shouldn't fail the build in that case.

Reviewed By: imp, brooks
Differential Revision: https://reviews.freebsd.org/D27598

Add WITH_CLANG_FORMAT option

clang-format is enabled conditional on either WITH_CLANG_EXTRAS or
WITH_CLANG_FORMAT. Some sources in libclang are build conditional on
either rule, and obviously the clang-format binary itself depends on the
rule.

clang-format could still use a manual page.

Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D25427

remove GCC 4.2.1 build infrastructure

As described in Warner's email message[1] to the FreeBSD-arch mailing
list we have reached GCC 4.2.1's retirement date. At this time all
supported architectures either use in-tree Clang, or rely on external
toolchain (i.e., a contemporary GCC version from ports).

GCC 4.2.1 was released July 18, 2007 and was imported into FreeBSD later
that year, in r171825. GCC has served us well, but version 4.2.1 is
obsolete and not used by default on any architecture in FreeBSD. It
does not support modern C and does not support arm64 or RISC-V.

Thanks to everyone responsible for maintaining, updating, and testing
GCC in the FreeBSD base system over the years.

So long, and thanks for all the fish.

[1] https://lists.freebsd.org/pipermail/freebsd-arch/2020-January/019823.html

PR: 228919
Reviewed by: brooks, imp
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23124

Fix nested MAKEOBJDIRPREFIX breaking various release/buildworld/toolchain targets.

This problem was caused by r325329 and r325350.

For the release(7) targets, some will run mm-mtree.sh which itself runs make
with a MAKEOBJDIRPREFIX. The execution of that script leaks OBJROOT,
MAKEOBJDIR, and MAKELEVEL=1 in the environment. This causes the mm-mtree makes
to not do some basic setup of OBJROOT and only use this special
MAKEOBJDIRPREFIX case which fails to empty out MAKEOBJDIRPREFIX for further
nested makes, such as a tree walk. If that tree walk sets OBJROOT/OBJTOP such
as r325329 is doing, then the wrong OBJDIRs end up being used due to the
unemptied MAKEOBJDIRPREFIX being preferred over the proper MAKEOBJDIR.

Pointyhat to: bdrewery
Sponsored by: Dell EMC Isilon

Follow-up r325329: Store all WORLDTMP objects back in the same directory.

This still keeps the reduced MAKEOBJDIRPREFIX (SRCTOP) redundancy
removed in the OBJDIR, but now keeps all early phase objects
in the same directory rather than split per phase.

The problem of splitting per phase is that later phases want to link in
libraries from earlier phases and base their location on ${OBJTOP}.

Pointyhat to: bdrewery
Reported by: mjoras, Mark Millard
Sponsored by: Dell EMC Isilon

Reduce MAKEOBJDIRPREFIX path spam by specifying a direct objdir to use.

Sponsored by: Dell EMC Isilon

Use more clear SRCTOP here.

Sponsored by: Dell EMC Isilon

buildworld: Pass which world phase the build is in down to submakes.

This is useful for having directories behave differently depending
on the phase - such as enabling SUBDIR_PARALLEL or disabling
redundant building of library directories already done by
earlier 'make _libraries'.

Sponsored by: Dell EMC Isilon

DIRDEPS_BUILD: Fix 'make bootstrap-tools' not using the proper tblgen binaries.

This was an incomplete item from r291561. The host {clang,llvm}-tblgen
binaries were used, rather than the ones built into the host stagedir by
normal Makefile.depend dependencies on tblgen.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

DIRDEPS_BUILD: Fix bootstrap-tools not handling CCACHE_DIR properly.

CCACHE_DIR needs to be resolved to its full path before processing
legacy.meta or meta mode may see the change to the stats file.

Sponsored by: EMC / Isilon Storage Division

META_MODE: Simplify the META_COOKIE handling to use .USE/.USEBEFORE.

Extend it to other cases of meta mode cookies so they get the proper rm
cookie behavior when a .meta file detects it needs to rebuild and fails.

Sponsored by: EMC / Isilon Storage Division

META MODE: Rework [bootstrapped] tools PATH support.

- Support more of the toolchain from TOOLSDIR.
- This also improves 'make bootstrap-tools' to pass, for example,
AS=/usr/bin/as to Makefile.inc1, which will tell cross-tools to use
external toolchain support and avoid building things we won't be using
in the build.
- Always set the PATH to contain the staged TOOLSDIR directories when
not building the bootstrap targets.

The previous version was only setting this at MAKE.LEVEL==0 and if the
TOOLSDIR existed. Both of these prevented using staged tools that were
built during the build though as DIRDEPS with .host dependencies, such
as the fix for needing usr.bin/localedef.host in r291311.
This is not a common tool so we must build and use it during the build,
and need to be prepared to change PATH as soon as it appears.

This should also fix the issue of host dependencies disappearing from
Makefile.depend and then reappearing due to the start of the fresh build not
having the directory yet, resulting in the tools that were built not actually
being used.
- Only use LEGACY_TOOLS while building in Makefile.inc1. After r291317
and r291546 there is no need to add LEGACY_TOOLS into the PATH for
the pseudo/targets/toolchain build.
- Because the pseudo/targets/toolchain will now build its own
[clang-]tblgen, the special logic in clang.build.mk is no longer needed.
- LEGACY_TOOLS is no longer used outside of targets/pseudo/bootstrap-tools
so is no longer passed into the environment in its build.

Sponsored by: EMC / Isilon Storage Division

META MODE: For bootstrap-tools, don't let Makefile.inc1 build any of the
toolchain.

This change prevents building of [clang-]tblgen, which is unneeded due
to already being a host dependency where needed for
targets/pseudo/toolchain.

Sponsored by: EMC / Isilon Storage Division

META MODE: Fix bootstrap-tools after r291349.

Now that OBJTOP is assumed to be proper, it needs to be overridden for
the bootstrap-tools phase to ensure it is not kept as the top-level
OBJTOP. Otherwise the libraries were not found during the
bootstrap-tools phase.

Sponsored by: EMC / Isilon Storage Division

META MODE: Fix 'make bootstrap-tools'.

The main problem was bitrot after elftoolchain being swapped in for the
GNU toolchain.

This also reworks how the list of 'host allowed' libraries is determined
to only allow INTERNALLIBs, which is needed for libelftc to come in.

For usr.bin/readelf use the same hack, as libelf and libdward, to bring in
the needed sys/ headers for host builds. This has not yet been a problem due
to readelf not being built as a host tool in buildworld. This is possible
in the meta build though when building the toolchain.

Sponsored by: EMC / Isilon Storage Division

META MODE: Need to include the LEGACY_TOOLS directories into PATH.

This is just as Makefile.inc1 does it for these phases. Otherwise some
of the build tools are not found and used properly, such as 'make-roken'
in the kerberos5 build on older releases.

Sponsored by: EMC / Isilon Storage Division

META MODE: Avoid creating TESTS directories for bootstrap.

Sponsored by: EMC / Isilon Storage Division

META MODE: Don't override DESTDIR to the default when building the toolchain.

For this case, the normal META MODE staging logic should be used. The
BSARGS having DESTDIR= is due to Makefile.inc1 overriding DESTDIR to
something when appropriate. For the toolchain stage META MODE is in charge of
it, not Makefile.inc1.

At least include/, if staging it to the host stagedir, was impacted from
this. Staging of include.host is not yet done.

Sponsored by: EMC / Isilon Storage Division

META MODE: Prefer INSTALL=tools/install.sh to lessen the need for xinstall.host.

This both avoids some dependencies on xinstall.host and allows
bootstrapping on older releases to work due to lack of at least 'install -l'
support.

Sponsored by: EMC / Isilon Storage Division

META MODE: Don't set schg flags in the stagedir for distrib-dirs.

This recently started setting /var/empty to schg after r290628.

Sponsored by: EMC / Isilon Storage Division

Don't create cookie until the command is all finished, it or the commands
after may fail.

Sponsored by: EMC / Isilon Storage Division

META_MODE: Make it easier to keep the bootstrap-tools args in sync.

Sponsored by: EMC / Isilon Storage Division

Merge sync of head

Rename pkgs to more generic targets (that's what they are)

Reviewed by: marcel

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

Fix bootstrapping to actually build lldb-tblgen for later use

Because MK_LLDB=no is in BSARGS, the bootstrap-tools recursive make does
not add lldb-tblgen to _clang_tblgen, causing it to not be built. This
means that the build currently always uses the host's lldb-tblgen
(which, whilst currently it appears to work, could in future break if
TableGen backends are added or altered) and, if it doesn't exist (either
because the current FreeBSD system was built with it disabled, or you're
building on macOS/Linux), fails. Linux and macOS cross-builds used to
work simply because LLDB was previously in BROKEN_OPTIONS when building
on non-FreeBSD.

Instead, move MK_LLDB=no from BSARGS to XMAKE. This ensures that the
lib/clang build in cross-tools continues to not build LLDB parts for the
bootstrap toolchain (both to save time/space on FreeBSD, and because our
vendored LLDB does not include the macOS and Linux host files so those
would fail to build).

The DIRDEPS target is updated to move MK_LLDB=no from the BSARGS block
that mirrors Makefile.inc1 to the line that disables additional
toolchain components. The DIRDEPS build likely suffers from the same
issue currently, but having never used it and not being familiar with
how it works I am leaving that as-is. If it does suffer from the same
issue it should be easily reproducible by renaming /usr/bin/lldb-tblgen
or moving it to a directory not in PATH.

Fixes: 31ba4ce8898f ("Allow bootstrapping llvm-tblgen on macOS and Linux")
MFC after: 1 week
Reviewed by: dim, emaste, imp
Differential Revision: https://reviews.freebsd.org/D31531

Prefer MK_SSP=no to SSP_CFLAGS=

It is more idiomatic. CFLAGS is only augmented with $SSP_CFLAGS when
$MK_SSP != "no".

Reviewed by: imp
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D31401

mk: format some option lists

Alphabetize and give each option its own line, ahead of making another
change to these lists. This makes future diffs easier to read.

Reviewed by: imp, emaste
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D31399

Rename NO_WERROR -> MK_WERROR=no

As suggested in D27598. This also supports MK_WERROR.clang=no and
MK_WERROR.gcc=no to support the existing NO_WERROR.<compiler> uses.

Reviewed By: brooks
Differential Revision: https://reviews.freebsd.org/D27601

Stop passing MK_WARNS=no for bootstrap tools

I got annoyed by the number of warnings that the CheriBSD build was
emitting. It turns out that we are emitting lots of warnings during
bootstrap because bootstrap tools are built with the default compiler
flags and ignore the warnings flags that are set in bsd.sys.mk.

Looking at git blame, it appears that MK_WARNS=no has been passed since
rS112869, replacing the -DNO_WERROR option that was added in rS87775.

This commit changes MK_WARNS=no back to -DNO_WERROR. We need to pass
-DNO_WERROR, since the system compiler might have new warnings that we
don't know about yet, and we shouldn't fail the build in that case.

Reviewed By: imp, brooks
Differential Revision: https://reviews.freebsd.org/D27598

Add WITH_CLANG_FORMAT option

clang-format is enabled conditional on either WITH_CLANG_EXTRAS or
WITH_CLANG_FORMAT. Some sources in libclang are build conditional on
either rule, and obviously the clang-format binary itself depends on the
rule.

clang-format could still use a manual page.

Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D25427

remove GCC 4.2.1 build infrastructure

As described in Warner's email message[1] to the FreeBSD-arch mailing
list we have reached GCC 4.2.1's retirement date. At this time all
supported architectures either use in-tree Clang, or rely on external
toolchain (i.e., a contemporary GCC version from ports).

GCC 4.2.1 was released July 18, 2007 and was imported into FreeBSD later
that year, in r171825. GCC has served us well, but version 4.2.1 is
obsolete and not used by default on any architecture in FreeBSD. It
does not support modern C and does not support arm64 or RISC-V.

Thanks to everyone responsible for maintaining, updating, and testing
GCC in the FreeBSD base system over the years.

So long, and thanks for all the fish.

[1] https://lists.freebsd.org/pipermail/freebsd-arch/2020-January/019823.html

PR: 228919
Reviewed by: brooks, imp
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23124

Fix nested MAKEOBJDIRPREFIX breaking various release/buildworld/toolchain targets.

This problem was caused by r325329 and r325350.

For the release(7) targets, some will run mm-mtree.sh which itself runs make
with a MAKEOBJDIRPREFIX. The execution of that script leaks OBJROOT,
MAKEOBJDIR, and MAKELEVEL=1 in the environment. This causes the mm-mtree makes
to not do some basic setup of OBJROOT and only use this special
MAKEOBJDIRPREFIX case which fails to empty out MAKEOBJDIRPREFIX for further
nested makes, such as a tree walk. If that tree walk sets OBJROOT/OBJTOP such
as r325329 is doing, then the wrong OBJDIRs end up being used due to the
unemptied MAKEOBJDIRPREFIX being preferred over the proper MAKEOBJDIR.

Pointyhat to: bdrewery
Sponsored by: Dell EMC Isilon

Follow-up r325329: Store all WORLDTMP objects back in the same directory.

This still keeps the reduced MAKEOBJDIRPREFIX (SRCTOP) redundancy
removed in the OBJDIR, but now keeps all early phase objects
in the same directory rather than split per phase.

The problem of splitting per phase is that later phases want to link in
libraries from earlier phases and base their location on ${OBJTOP}.

Pointyhat to: bdrewery
Reported by: mjoras, Mark Millard
Sponsored by: Dell EMC Isilon

Reduce MAKEOBJDIRPREFIX path spam by specifying a direct objdir to use.

Sponsored by: Dell EMC Isilon

Use more clear SRCTOP here.

Sponsored by: Dell EMC Isilon

buildworld: Pass which world phase the build is in down to submakes.

This is useful for having directories behave differently depending
on the phase - such as enabling SUBDIR_PARALLEL or disabling
redundant building of library directories already done by
earlier 'make _libraries'.

Sponsored by: Dell EMC Isilon

DIRDEPS_BUILD: Fix 'make bootstrap-tools' not using the proper tblgen binaries.

This was an incomplete item from r291561. The host {clang,llvm}-tblgen
binaries were used, rather than the ones built into the host stagedir by
normal Makefile.depend dependencies on tblgen.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

DIRDEPS_BUILD: Fix bootstrap-tools not handling CCACHE_DIR properly.

CCACHE_DIR needs to be resolved to its full path before processing
legacy.meta or meta mode may see the change to the stats file.

Sponsored by: EMC / Isilon Storage Division

META_MODE: Simplify the META_COOKIE handling to use .USE/.USEBEFORE.

Extend it to other cases of meta mode cookies so they get the proper rm
cookie behavior when a .meta file detects it needs to rebuild and fails.

Sponsored by: EMC / Isilon Storage Division

META MODE: Rework [bootstrapped] tools PATH support.

- Support more of the toolchain from TOOLSDIR.
- This also improves 'make bootstrap-tools' to pass, for example,
AS=/usr/bin/as to Makefile.inc1, which will tell cross-tools to use
external toolchain support and avoid building things we won't be using
in the build.
- Always set the PATH to contain the staged TOOLSDIR directories when
not building the bootstrap targets.

The previous version was only setting this at MAKE.LEVEL==0 and if the
TOOLSDIR existed. Both of these prevented using staged tools that were
built during the build though as DIRDEPS with .host dependencies, such
as the fix for needing usr.bin/localedef.host in r291311.
This is not a common tool so we must build and use it during the build,
and need to be prepared to change PATH as soon as it appears.

This should also fix the issue of host dependencies disappearing from
Makefile.depend and then reappearing due to the start of the fresh build not
having the directory yet, resulting in the tools that were built not actually
being used.
- Only use LEGACY_TOOLS while building in Makefile.inc1. After r291317
and r291546 there is no need to add LEGACY_TOOLS into the PATH for
the pseudo/targets/toolchain build.
- Because the pseudo/targets/toolchain will now build its own
[clang-]tblgen, the special logic in clang.build.mk is no longer needed.
- LEGACY_TOOLS is no longer used outside of targets/pseudo/bootstrap-tools
so is no longer passed into the environment in its build.

Sponsored by: EMC / Isilon Storage Division

META MODE: For bootstrap-tools, don't let Makefile.inc1 build any of the
toolchain.

This change prevents building of [clang-]tblgen, which is unneeded due
to already being a host dependency where needed for
targets/pseudo/toolchain.

Sponsored by: EMC / Isilon Storage Division

META MODE: Fix bootstrap-tools after r291349.

Now that OBJTOP is assumed to be proper, it needs to be overridden for
the bootstrap-tools phase to ensure it is not kept as the top-level
OBJTOP. Otherwise the libraries were not found during the
bootstrap-tools phase.

Sponsored by: EMC / Isilon Storage Division

META MODE: Fix 'make bootstrap-tools'.

The main problem was bitrot after elftoolchain being swapped in for the
GNU toolchain.

This also reworks how the list of 'host allowed' libraries is determined
to only allow INTERNALLIBs, which is needed for libelftc to come in.

For usr.bin/readelf use the same hack, as libelf and libdward, to bring in
the needed sys/ headers for host builds. This has not yet been a problem due
to readelf not being built as a host tool in buildworld. This is possible
in the meta build though when building the toolchain.

Sponsored by: EMC / Isilon Storage Division

META MODE: Need to include the LEGACY_TOOLS directories into PATH.

This is just as Makefile.inc1 does it for these phases. Otherwise some
of the build tools are not found and used properly, such as 'make-roken'
in the kerberos5 build on older releases.

Sponsored by: EMC / Isilon Storage Division

META MODE: Avoid creating TESTS directories for bootstrap.

Sponsored by: EMC / Isilon Storage Division

META MODE: Don't override DESTDIR to the default when building the toolchain.

For this case, the normal META MODE staging logic should be used. The
BSARGS having DESTDIR= is due to Makefile.inc1 overriding DESTDIR to
something when appropriate. For the toolchain stage META MODE is in charge of
it, not Makefile.inc1.

At least include/, if staging it to the host stagedir, was impacted from
this. Staging of include.host is not yet done.

Sponsored by: EMC / Isilon Storage Division

META MODE: Prefer INSTALL=tools/install.sh to lessen the need for xinstall.host.

This both avoids some dependencies on xinstall.host and allows
bootstrapping on older releases to work due to lack of at least 'install -l'
support.

Sponsored by: EMC / Isilon Storage Division

META MODE: Don't set schg flags in the stagedir for distrib-dirs.

This recently started setting /var/empty to schg after r290628.

Sponsored by: EMC / Isilon Storage Division

Don't create cookie until the command is all finished, it or the commands
after may fail.

Sponsored by: EMC / Isilon Storage Division

META_MODE: Make it easier to keep the bootstrap-tools args in sync.

Sponsored by: EMC / Isilon Storage Division

Rename pkgs to more generic targets (that's what they are)

Reviewed by: marcel