[build] USE_LINKER_GC -> USER_USE_LINKER_GC, KERNEL_USE_LINKER_GC

Replace the USE_LINKER_GC=false command-line make switch with two
separate switches affecting userland and kernel distinctly.

Test: CQ
Change-Id: I671953c55e00c152198aea8162ee01abdc3088a1

[build] provide ENABLE_INSTALL_SAMPLES build parameter

When true, modules marked as sample or ddk-sample will be included
in the zircon.zbi. Otherwise they will not be included. They will
always be built.

Previously there was no way to override USER_MANIFEST_GROUPS,
and if you did (in engine.mk) change it to the empty string,
the --groups= that got passed to zbi would filter out *everything*.

Tested: manually

Change-Id: Ie975666c7817e020908f1681fab13f987e8783a7

[build] Get rid of .mod.o files

The .mod.o files don't add any value except in shortening the linker
command line. Hopefully we don't really need them that short to avoid
problems on any real host.

Avoiding them makes it easy to have ALLMODULE_OBJS actually list everything
going into the kernel, which is useful for running analysis steps.

Test: CQ
Change-Id: I19d78d250d4df76b0262e48f2e98b0a1b7904800

[build][zbi] Exclude modules from BOOTFS by group

This adds a negation feature to the --groups switch to zbi: if the
group list is preceded by a `!` then it excludes the listed groups
and allows all others (and unmarked entries). Incidentally it also
fixes group matching (really, parsing the option value), which was
completely broken since it wasn't being tested in the build before.

The build now uses this to control the groups that go into the
zircon.zbi BOOTFS image via the USER_MANIFEST_GROUPS makefile
variable. The default setting is `!sample,ddk-sample` to include
everything except modules marked `sample` or `ddk-sample`.

Bug: ZX-2380 #done
Test: CQ and manual inspection with zbi -tv
Change-Id: I41d651774f607acaf97c5fc47c88b2ea8dddd923

[build][scripts] Build complete ZBI, boot it in QEMU

The primary outputs of the build are now kernel.zbi and zircon.zbi.
The latter is a complete ZBI for standalone booting.
The run-zircon script now uses zircon.zbi by default.
The deprecated -x switch makes it use the old method with direct
kernel loading rather than a trampoline.

The old names zircon.bin and bootdata.bin are made as links, and the
shimmed qemu-zircon.bin is still made. These names are deprecated and
will be removed when all the consumers of the build have been updated.

Test: CQ
Change-Id: I37d8661f8f5766e3947d0a6f71b13178a82af3c4

[host][zbi] Clean up command line syntax again

The --target switch is now --type and is more general.
The --target=boot case is replaced by --files (-F).

Item extraction now works like file extraction, using
synthesized file names for items. The -- PATTERN...
syntax can be used to match synthesized file names, and
thus match by item index or type.

Change-Id: Idace6255edefba04a6ed2e7750b100ec64811cbd

[host] zbi will replace mkbootfs

Rewrite the mkbootfs host tool from scratch in C++14, under the new
name zbi (Zircon Boot Image), which will become the new name for the
BOOTDATA format soon. The old tool is left intact during a
transition period.

The new implementation is marginally shorter but with more comments
and structure. It opens each file only once and uses mmap heavily.
It uses getopt_long for argument parsing. It uses the cksum and lz4
code as proper hostlibs rather than source files.

The --complete switch makes the tool check that the result is a
complete bootable image, which has a kernel and a /boot BOOTFS (and
can optionally have additional items).

The --ramdisk switch is replaced with --target=ramdisk: all
following input file arguments will be treated as raw images.
Similarly, --target=zbi treats following input file arguments
as incomplete ZBI files and never as manifest text files.

The --board, --vid, --pid, --header, and --header-align switches are
not supported by zbi. Those things are now handled entirely by boot
loaders. Boot shims for legacy boot loaders are standalone images
that can just be prepended to a complete ZBI image.

Change-Id: I4903b7d0c5824c496301a52c05a85a2e36b3cebd

[scripts] Use prebuilt objdump for errors on kaslr fixup

Use the objdump from zircon build environment so it supports the right
architecture when an error is generated.

SEC-31 #comment ensure we can debug relocation issues on arm64

Change-Id: I3ff733ae7ec02b1c79afc73fe3557fc273ec1325

[kernel] Use --emit-relocs to add self-fixups

SEC-31 #comment This applies relocations to the kernel.

Change-Id: I07b40308b768a56c889eeb9a9a89c0e897924617

[prebuilt] Revamp prebuilts

scripts/download-prebuilt replaces scripts/download-toolchain and
scripts/download-sysroot, which are both now obsolete stubs that
warn and exec download-prebuilt. The new script uses CIPD when the
tool is available. When not using CIPD, it uses a single file for
all the versions (ZIP file SHA1 checksums) rather than the old
arrangement of many files in many subdirectories.

The new script has --list and --verify modes to describe the state
of the installed prebuilts vs the current versions described in the
file. The makefile check for stale prebuilts is now done via
`scripts/download-prebuilt --verify`.

scripts/update-prebuilt-versions automatically maintains the
prebuilt/zircon.versions file that drives the (non-CIPD) downloads
and the sanity checks. prebuilt/zircon.ensure is maintained by hand
to deploy new versions, and then the script must be run to update
prebuilt/zircon.versions before commit.

ZX-1860 #done
TO-937 #done

Change-Id: Iaf02d2c5f3bdd81008573cd2939b35f1a450398c

[build] Remove the USER_AUTORUN tombstone

Change-Id: Ic7bb1ec9d531c31f65729db5267e561448664773

[build][arm64] Unified build for all arm64 targets

Zircon now has a single unified arm64 build.
Instead of multiple build outputs for different arm targets,
there is now a single build-arm64/ containing a common
arm64 kernel and userspace.

Within build-arm64/ we also build custom <board>-bootdata.bin
and <board>-kernel-bootdata.bin files for each arm64 board.
These differ only by their kernel MDI and platform ID records.

To build the unified arm64 build, type "make arm64" or run the
scripts/build-zircon-arm64 script.

The following scripts were updated to use the new unified arm64 build:

scripts/build-bootable-odroidc2-sdcard.sh
scripts/fastboot-flash
scripts/flash-hikey
scripts/flash-vim
scripts/flash-vim2
scripts/run-zircon-arm64

Change-Id: Ie999f3b4adc938ece29af4fa0d6e41f655972189

[devmgr] remove redundant use of /boot/autorun

The zircon.autorun.{boot,system} kernel commandline options supplant
the use of /{boot,system}/autorun for running commands.

Change-Id: I66c44c66ac0c88b62b7de17156f0b1a71c2fb050

[build] EXTRA_USER_MANIFEST_LINES allows adding files to bootfs.manifest

Add a deprecation warning for USER_AUTORUN

INTK-133

Change-Id: Ib9fbaee0fa85ebca0800d49fd646e84c32c247e7

[kernel][counters] Use C arrays to reserve kcounters_arena space

This trades linker script magic to explicitly allocate space for
kcounters_arena for simpler linker script magic relying on
-fdata-sections to let C code allocate the space.

Change-Id: I323b2c0275137a03d4279848775b695b92a3d647

[zircon][kernel] kernel counters

Implementation based on the requirement doc
by travisg. This is only the core kernel part.

The implementation uses linker tricks to set
as much as possible static memory so they can
be used early and be declared almost anywhere.

The counters are stored in per-cpu arrays so
that there is no need to use atomics to keep
the increments consistent and there is no
false cache line sharing.

The counters are exposed via a temporary kernel
command 'k counters'

This change also adds 11 counters, here is sample
output.

$ k counters all
[00006.351] 01044.01047> 11 counters available:
[00006.351] 01044.01047> [0] kernel.dispatcher.cookie.reset <0>
[00006.351] 01044.01047> [1] kernel.dispatcher.cookie.set <0>
[00006.351] 01044.01047> [2] kernel.dispatcher.create
[00006.351] 01044.01047> [0:451][1:226][2:285][3:561] = 1523
[00006.351] 01044.01047> [3] kernel.dispatcher.destroy
[00006.351] 01044.01047> [0:223][1:174][2:161][3:354] = 912
[00006.351] 01044.01047> [4] kernel.dispatcher.observer.add
[00006.351] 01044.01047> [5] kernel.dispatcher.observer.cancel.byhandle
[00006.351] 01044.01047> [0:474][1:327][2:334][3:622] = 1757
[00006.351] 01044.01047> [6] kernel.dispatcher.observer.cancel.bykey <0>
[00006.351] 01044.01047> [7] kernel.thread.create
[00006.351] 01044.01047> [0:15][1:4][2:5][3:17] = 41
[00006.351] 01044.01047> [8] kernel.thread.join
[00006.351] 01044.01047> [1:1][2:4] = 5
[00006.351] 01044.01047> [9] kernel.thread.resume
[00006.351] 01044.01047> [0:12][1:4][2:5][3:17] = 38
[00006.351] 01044.01047> [10] kernel.thread.suspend <0>

Change-Id: I5eaabc3b7646dcaca72604953f0fb5338b3a776d

[kernel][link] remove the AT() physical offset of the kernel link

Both x86 and arm64 builds do not load raw elf files any more, so there
is no need to set the physical load address of the kernel in the linker
script.

Change-Id: I0941aae49f3c83ad1b89f71d6bc29e125dd6ad25

[kernel] remove MEMBASE and some other unused variables

Change-Id: I6e644de42f84b3a25e6e6e9697c32fabcedaa0d5

[build] Support user-only and kernel-only builds

The new user-arm64 and user-x86-64 projects can be used to build
the sysroot and/or user bootfs without building any kernel (or
choosing a target platform).

e.g. `make PROJECT=user-x86-64 user-only` will build the sysroot,
the bootfs.manifest file and everything in it (but not the user
`bootdata.bin` image itself, since Fuchsia builds don't use it).

e.g. `make PROJECT=zircon-qemu-arm64 kernel-only` will build the
kernel images and a `kernel-bootdata.bin` file that should be
combined with a bootfs image to go with this kernel.

To keep the build process uniform, `kernel-bootdata.bin` is
generated even when it's empty (which is currently always the
case for x86), hence `mkbootfs` acquires the `--empty` switch.

Change-Id: I1090b299e2a7f30b4e6afb53e785faf480dc9ea7

[build] Replace GNU strip with llvm-objcopy

Change-Id: I32a86512bc3d266723c31ae06dcb50dc990b7e89

[kernel] Don't massage kernel.ld to insert constants

Instead, generate a linker script that just defines
the constants as symbols.

Change-Id: Ice7570668fd53f6005185283c0241a83bba03b2a

[kernel] Consolidate kernel linker script

There is much more that's the same than that's different between
the architectures. Maintaining a single linker script that works
for both is easier than keeping two in synch.

Change-Id: I0655919e3f7dc5f8788d47bbe733f444c87ec97b

[mkbootfs] Support writing BOOTDATA_PLATFORM_ID by itself

This makes mkbootfs treat BOOTDATA_PLATFORM_ID more like other
bootdata items rather than a special snowflake. The build now
writes a platform-id.bin separately and then combines this into
bootdata.bin, so that it's easy for other tools to do their own
combination of platform-id.bin with their own bootfs manifests.

Change-Id: I80fa873e29a10af063fb12a5a3d0c79f59892932

[build][arm64] Set platform ID information in the bootdata

Change-Id: I9c8edcb5d6c2f7edac69f8a4cb914f1bd21bd9fe

[build] Don't use extra files for BOOTFS_DEBUG_MODULES

There is no need to make a ".debug" copy of the unstripped ELF file.

Change-Id: Iedf4827e54fefb1d624c14dd03a44cc2066e344e

[build] Use link instead of objcopy to generate debug info file

We're already using a straight copy rather than actual split-debug for
the sysroot/debug-info/ copies. This patch changes the
BOOTFS_DEBUG_MODULES configuration to avoid relying on this particular
objcopy feature in our build.

Change-Id: Id3fb6d35191b0e28eb9c4969cfa3fd76f2fe4354

[make] add a meta rule for the kernel to let you easily just build it

Change-Id: Ic1509a8c6fc407236523ab46acb223a4c6a9d2a6

[build] Support for LTO and ThinLTO in kernel

ThinLTO build is currently broken due to linker issues.

Change-Id: I4648b77c8eb8d4333fc69967b4f68af012febf14

[build] Use input ld script rather than relocatable object for modules

This allows using the same codepath for both LTO and non-LTO builds.

Change-Id: Id402408c9647994ae43167d3f11dfa608cda05a4

[make] properly honor QUIET to squelch build spew when using ENABLE_BUILD_LISTFILES

Change-Id: Ib624959c1f3d847c29915e18b9f0a4d92bb99330

[make] add QUIET=1 option to the build to squelch all output

Change-Id: I57742bb6aa0ad1f0dc516c917dfecd65f6f65f43

[build] flatten the build

Previously we treated kernel/, system/, and third_party/ as
overlays on a shared namespace. This required the concept
of "canonical" module names, and a lot of complexity to ensure
that things didn't collide and the build worked.

This change gets rid of that, no longer passes -I to make,
so that include directives from our *.mk files do not magically
wildcard across various paths, etc.

The most user-visible change is that everywhere where a module
name is specified (MODULE_DEPS, MODULE_LIBS, etc), full module
names like kernel/lib/io or system/ulib/mxio must be used instead
of previously-allowed "short" names like lib/io and ulib/mxio.

The build output still has a similar shape, but the first segment
of the module path (kernel/, system/, or third_party/) is no
longer elided under $(BUILDDIR)

Change-Id: I525aba1da1c86eb7a86007bddc669f7eeebfedd5