[fs][minfs] Allow filesystems to implement custom interfaces

As a simple test case, propagate the metrics collection API
to minfs, and test these fields (which previously could
only be dumped to stdout).

ZX-2710 #done
Test: Expanded minfs tests

Change-Id: I6b45693731d912be3fab7d95f0bc1ff7d52fb593

[bitmap] Make bitmap a static library and use -fvisibility=hidden

ZX-1895 #comment

Test: cq
Change-Id: Ia4a3ad32e78ac3cbe12bc40c01d55a5e1835d126

[fdio][fidl] Replace fuchsia-io hand-rolled client bindings

ZX-1919 #comment In Progress

Test: CQ (fs tests, fdio tests, blobfs tests, etc)

Change-Id: I31588bb62f5788fb9372fdc7042f42a3cae7c71c

[minfs] Add aggregator metrics

These metrics are, if active, logged at unmount time.

ZX-1942 #done

Change-Id: Idc1d5f999bb23caa8f71ba1083fcc1321e37571a

[make] Link against static libraries using --start-group/--end-group

Before this change, ordering of libraries in MODULE_STATIC_LIBS is
significant in the GCC build: targets (such as minfs) have to order
the libraries according to dependencies between the libraries.
However, that apparently wasn't the case in the Clang build (which
uses lld).

Fix this by using --start-group/--end-group, which tells the linker to
resolve dependencies between libraries in any order. This makes the
GCC build behave like the Clang build in this respect.

The entries in minfs's MODULE_STATIC_LIBS can now be sorted.

Change-Id: If784b2f3e781721629f9c1fb3a6d57a95cde1354

[async] Move async-loop implementation to its own directory.

People keep getting confused by the relationship between async-loop
and the rest of the library. So let's fix it.

This change includes compatibility includes for loop.h to be removed
after upper layers are fixed to use the new paths.

Change-Id: I95c57be533fbd91468275bb2470890a4f10101eb

[async] Separate fbl dependency from async_t

The C++ FIDL support library in Garnet wants to be able to use async_t
without linking against libfbl. Previously, the C declarations needed
create an async_wait_t were in the same header as code that depended on
fbl, which meant C++ clients of async_wait_t were forced to depend on
fbl.

After this change, the C++ wrappers for libasync interfaces are
separated from the core C declarations.

Change-Id: Ic172911a9dc775d5b5e540a33ce87d2dc218a3d9

[minfs][blobstore][fvm] Relocate host-side tools to /host

Change-Id: I21e3f17a7dd926dc5038af85edc0e2e016d220bd

[minfs] Split host/target MinFS applications

As we take steps to separate the host/target interface
to MinFS (and reduce ifdefs), making a separate application
for the host-side MinFS is a necessary first step.

ZX-1502 #done

Change-Id: I53d65f6522039b20d869df733d5225023db8d613

[fs][minfs] Add basic support for tracing MinFS

Conditionally define "TRACE_DURATION" (and other
macros which will be used by filesystem tracing)
within ulib/fs, to avoid "ifdef __Fuchsia__" bloat.

ulib/fs's tracing will continue to be modified
in future patches.

ZX-1300 #done

Change-Id: I1e25b5f2feb68e58b41887d8958eafa6ef5b1582

[minfs] Convert MinFS to a library

ZX-1314 #comment In Progress

Change-Id: I0a7672d8321e0be07dd4c66232a0b4836db777a3

[minfs] Enable asynchronous writeback for MinFS

MinFS spawns a background thread which manages all writes
to disk. Rather than blocking, waiting for a write to complete
in an I/O operation (like write, truncate, create, etc) the
modified blocks are copied into an in-memory buffer, which the
background thread dumps to disk.

ZX-1285 #done

Change-Id: Iccdb70da02db450551091d46c897de770f7559ce

[minfs] Allow host-side tool to check/modify sparse/fvm file

Change-Id: I164b2ab2f35866b2299e01418f0b31f2dd748ba9

[minfs] Add support for host-side fsck

Change-Id: Ie50df5d28f1cc8f6fbd235c02ee8cf273980d541

[build][ulib/hash] move hash/rand functions to public headers

Instead of maintaining a library for these,

hash.h -> zircon/misc/fnv1hash.h and zircon/misc/xorshiftrand.h

Change-Id: Ia4c1a547b10f6444acd89f72c3ed340a4210f30b

[fvm] Add host-side tool

Example usage:
truncate --size=1g blobfs.bin;
./build-zircon-pc-x86-64/tools/blobstore blobfs.bin create
./build-zircon-pc-x86-64/tools/fvm sparse.bin sparse --blobstore blobfs.bin

Change-Id: Ifc44ea30d0cf68e3914d4e8591fa8c65998ba9ce

[fs] Ensure that O_PATH and O_DIRECTORY are defined in all clients of libfs

Change-Id: I9ed92fb1c0d79e9dcb2d43984495509ce673134b

[fs] Explicitly define O_PATH for host tools

Change-Id: I7c3669bdf35765724d07c56729460ddf6c3aa9ad

[minfs] Add host-side tests

US-332 #done

Change-Id: I26475e93227c2950cfcda707025df5b94179e9bc

[fs] Move Vnode method definitions out-of-line.

This matches C++ best practice since virtual functions don't get
inlined.

Moved the functions to a new file so they match the header structure.

Change-Id: Ia4b68df92c4b79a315077df2d355a00c7c31733d

[minfs] Minimize host-side tooling

- Remove bitrotting tests
- Remove "rename" and "unlink" from host-side tools.
These functions should not be necessary for generating
golden images, and they increase the complexity of the
VFS layer (the Vfs::Walk function should be private).

Change-Id: I96599828e6e39db15cce95286481671fa49b2611

[ulib][hash] New directory

Make minfs's misc.h hash functions generally available

Change-Id: I70e933a185222dcfa2b55ac73f170712b31f459c

[zx] Magenta -> Zircon

The Great Renaming is here!

Change-Id: I3229bdeb2a3d0e40fb4db6fec8ca7d971fbffb94

[fs][minfs] Replace the vfs-dispatcher with libasync

Change-Id: Icfc3566510ae0e5f29ac2f0615681c00c831ece9

[zircon][mxtl->fbl] Rename mxtl to fbl

Change-Id: Ie21b6498e1bfb0a7fa0315e40b9e5c3ee78646be

[build][groups] annotate 'misc' and 'test' modules

This allows us to generate a bootdata.bin which excludes them,
reducing it from 4.52MB to 2.65MB

The default bootdata.bin still includes all modules.

Change-Id: If13bfcacba9386a691e6b7c380eed53d67bb44eb

[mxtl] Compile mxtl as a hostlib.

Also update clients to link to it that way instead of pulling in
alloc_checker.cpp directly.

Change-Id: I7c98b4bd1a5e6a0d2173400a44b87a840ede4726

[mxtl][mxalloc] Redirect all clients of mxalloc to mxtl.

Change-Id: Ib6d88937212ab23612f59add8bd028862308ead5

[fs] Improve vfs_lock_ encapsulation

- Convert thread annotations to no-op on builds with
"DISABLE_THREAD_ANNOTATIONS" defined; this allows static thread
analysis for operations which act on filesystems that
are compiled for both host and target (such as MinFS), but
which use a single-threaded model for host-side tools.
- Add thread annotations to VFS operations
- Continue refactoring required to eventually make "vfs_lock_"
internal to the VFS class. "Mount + Mkdir", "Locked" functions,
and iostate token access are now all internal to the Vfs class.

Change-Id: I5d083f38726c493602cd24139303ab377ef655e4

[minfs] Update fsck to check link counts

fsck:
- Check that iNodes' reported link count is correct
- Check that directory iNodes do not have more than one parent

mkfs:
- Change initial link count of root to 2

Change-Id: Id16e81a4bb11c5e6d8a477b31a9ca6071e8335c4

[minfs] Convert MinFS, bitmap to use libmx handles

Change-Id: Id0a90f523496d04d6012cf6f04ff04c757224c23

[minfs] Remove reliance of fsck on bcache

We're using page-cache structures (VMOs) to
store files on mounted filesystems; we may as
well do the same for fsck. This is necessary
to remove portions of the bcache.

Change-Id: I0ace36d260ce1d0970472d3a28ed0d3387a8a78d

[minfs] Use the FIFO-based Block I/O protocol

Additionally, delay operations (wherever possible) by
putting them into "block transactions", which allow
enqueueing (and merging) operations which are sent to the
underlying block device.

Change-Id: I9355d343974f6adab57a6da769d576483069eda7

[build][host] Use C++14 for host tools

Also teach host tools to use libc++ rather than the host library when
using the prebuilt clang host toolchain.

This requires changing fidl to use the std make_unique, as otherwise
the calls to make_unique are ambiguous.

Change-Id: Iedccf38cd2d3f751fa82a0ea30151085f4ef3436

[mxcpp][mxalloc] Split mxcpp into mxcpp and mxalloc

mxcpp used to provide three things.

(1) is a very minimal C++ runtime environment. This means essentially
the definition of __cxa_pure_virtual, which is a function the compiler
expects to have ambient. This functionality is necessary for code
written in environments which cannot link the standard library (the
kernel and a handful of initial userspace processes).

(2) is the definitions of the standard new and delete
overloads. Again, these are needed for code that wants to call them
but to not link the standard library, but this time only in userspace
(the kernel does not want certain behavior stemming from the standard
overloads). The behavior is slightly different than the standard ones
in failure cases, but not in a way particularly interesting to the
rest of this email.

(3) is the userspace version of our special AllocChecker new
overloads. This mechanism exists for kernelspace, where allocation
failure is real and needs to be handled. This mechanism works
perfectly well in userspace, in that it compiles, even though the
underlying allocator is unlikely to ever return nullptr. It's
therefore essentially harmless in userspace. A handful of small
libraries are used in both userspace and in the kernel, and also use
this mechanism. Note that this feature is essentially copy-pasted
between the kernel and userspace.

This all works perfectly well for the kernel.

Some userspace code also uses these mechanisms, as mentioned. As long
as the standard library isn't linked, there are again no issues.

But as soon as you want both (3) and std::vector, which is a
reasonable thing to want!, there are issues because (1) and (3) come
in the same static library. This causes link failures due to duplicate
__cxa_pure_virtual symbols. This issue is exacerbated by different
behavior in Linux, OS X, and target linkage, in terms of confusion and
it-works-on-my-machine sort of issues.

This patch splits mxcpp into: mxcpp, containing (1) __cxa_pure_virtual
and (2) only if in userspace, the standard new and delete overloads;
and mxalloc, containing (3) AllocChecker. mxcpp is mutually exclusive
of the standard library. mxalloc can be linked against both code using
the standard library, and against code using mxcpp.

This patch also deduplicates the AllocChecker between userspace and
kernel, and moves the InlineArray class into the kernel (its only
callsites are in the kernel).

Change-Id: Id42d5d019ee977d44d6cb89ce7ab6884f7a56cd1

[ulib] Make mxtl, mxcpp, and mx libraries static-only

mxtl is actually a header-only library, so don't build a
shared object for it.

The mxtl and mx libraries are already built only as static
libraries in the Fuchsia GN build. Their code is very small.
Make the Magenta make build follow suit.

Change-Id: Ia79e5a996a981a8b046370d4ece9597f8e5bd1c2

[fs] Simplify requirements for linking with ulib/fs

During the transition from C to C++ (and the refactoring of
a ulib/fs out of our many filesystems), a couple of ugly
artifacts emerged. As a hacky choice to reach code deduplication
with minimal friction, some symbols were declared in
ulib/fs headers (and expected) but not defined; the definition
of these symbols were left to the linking libraries.

This patch removes the last traces of these symbols:
- vfs_lock (definition moved to ulib/fs), and
- vfs_handler

Additionally, "vfs_handler_generic" (the common, shared
implementation used by most filesystems) has been renamed to
"vfs_handler" for clarity.

MG-724 #done

Change-Id: I5b3dbec1be0b2e53fde1ca29b3d9573b7353826b

[build] flatten the build

Previously we treated kernel/, system/, and third_party/ as
overlays on a shared namespace. This required the concept
of "canonical" module names, and a lot of complexity to ensure
that things didn't collide and the build worked.

This change gets rid of that, no longer passes -I to make,
so that include directives from our *.mk files do not magically
wildcard across various paths, etc.

The most user-visible change is that everywhere where a module
name is specified (MODULE_DEPS, MODULE_LIBS, etc), full module
names like kernel/lib/io or system/ulib/mxio must be used instead
of previously-allowed "short" names like lib/io and ulib/mxio.

The build output still has a similar shape, but the first segment
of the module path (kernel/, system/, or third_party/) is no
longer elided under $(BUILDDIR)

Change-Id: I525aba1da1c86eb7a86007bddc669f7eeebfedd5

[build] use '.', not '-' to add "sub-module" postfixes

Since '-' is a common feature of directory names in Magenta, it's
not possible to easily figure out if ulib/foo-bar refers to a module
named foo-bar or a module named foo with a sub-module named bar.

I want to enforce that modules *must* exist in a matching directory
(as the build expects) and that sub-modules *not* have a matching
directory (so there's never a case where we could have a collision in
the build-*/... space between a module and a sub-module.

Using a character not currently used in source directory names will
allow me to forbid its use in source directory names and ensure that
there is no ambiguity.

Change-Id: Ibf3600b33c70e6920c6c52ffc1c7139f2658034a

[fs] Convert Vnodes to C++

- vnode_t struct split into:
- fs::Vnode abstract base class, with default "ERR_NOT_SUPPORTED"
versions of operations.
- MinFS: subclass minfs::VnodeMinfs
- Blobstore: subclass blobstore::VnodeBlob
- MemFS: subclass memfs::VnodeMemfs, which provides some common
functionality to all types of MemFS vnodes, but which is itself
subclassed into "VnodeFile", "VnodeDir", "VnodeVmo", and
"VnodeDevice".

Change-Id: I114acd067dec7849f1591eddedf020102ddf7e67

[build][musl] refer to musl libc as ulib/c

This makes the build rules and the intermediate
output files easier to understand, without having
to modify the path where musl libc lives.

The biggest observeable difference is the build
intermediate files:

$ ls build-magenta-pc-x86-64/ulib/c
config-module.h libc.abi.h libc.abi.stamp libc.so libc.so.id libc.so.strip
libc.a libc.abi.o libc.a.opts libc.so.abi libc.so.opts third_party

which previously were at build-magenta-pc-x86/ulib/musl
and named libmusl.*

and the replacing of ulib/musl with ulib/c in the library
dependencies of all the rules.mk files (the vast bulk of
this change)

Change-Id: If9c5d522099d2c9ac5bdb666b8e67a2eb3d316ee

[build][minfs] build minfs host tool as a hostapp module

Change-Id: Ibbb92a5821584bfa28f49e80d9016f4301b8ada9

[minfs] Use ulib/bitmap instead of ad-hoc bitmap

Change-Id: Ie0ef231c058df65c26729312b900e74c72626384

[minfs][fs] Convert MinFS to C++

This patch contains the minimal set of changes to compile MinFS as a C++
program -- it does not intend to use "best-style" C++ yet; those changes
will appear in subsequent patches.

Change-Id: Iebdf381f3aece27b863a30a496b05520352757d8

[minfs] modify minfs to use common fs vfs library

Change-Id: I227578453076e636742003f58a678e7ce966b3f3

[magenta] migrate from libddk to magenta/device headers

Change-Id: I91a8df001043635fa595d6fed1a6d6f092fc6b3e

[system] align license and license banners with Fuchsia

This adjusts the Magenta userspace code to match the prevailing
license in Fuchsia (three-clause BSD), using the same banner
and license text.

Change-Id: I6b6d79435c5dbaa32027888874849e4f47aaa7be

[minfs] a minimal unix-ish filesystem and fs-server framework (1/n)

- minfs.c minfs-ops.c minfs-check.c
MinFS provides (or it will when done) a minimal unix-ish filesystem:
8k blocks, 256b inodes, 16 direct blocks, 32 indirect blocks, max
file size of 512MB.

- bcache.c bitmap.c vfs.c
Non-minfs-specific code for a block cache, allocation bitmaps,
and general vfs interface glue.

- main.c rpc.c
Driver program / server process (can mkfs, fsck, mount, run-tests).
On Magenta this speaks mxio/remoteio and can install itself as
a filesystem server.

- wrap.c
A tiny posix io wrapper around the vfs interface, allowing test code
to call open/close/read/write/unlink/stat/etc and have them work
on either real fds or fake fds mapped to the vfs.

- test.c
Some code to exercise the filesystem.

- working: open(), close(), mkdir(), close()
- todo: read(), write(), readdir(), unlink()
- later: rename(), locking/multithreaded, ordered updates, lazy writes

Change-Id: I5762ec8753acb8c9f44c903f05ec9723990d8f57