[devmgr] Use C++ only for internal headers

Test: cq

Change-Id: I9f8f0b0f7eed557c9234d061e7bb60f8c08d6a73

[fdio] Move headers into lib/fdio/...

Change-Id: Ie8d74e716da913bf6e2672c4acf8cd67b4962b7f

[memfs] Split memfs into C / C++ libraries

For clients which utilize memfs to create
temporary filesystems, export a C ABI shared
library.

Internally, this C library uses a C++ library,
which is exported as source.

Change-Id: I73b95a2a2b8c4870d1b07f5f2f2525f4b0cd4e9f

[core][bootfs] fold ulib/bootfs into system/core

This is only needed by devmgr and fshost, is tiny, and hopefully will
stay that way.

Also removed some stale library dependencies in devmgr, trying to get
back to a relatively small set of library dependencies for it.

Change-Id: I03e8b205f822c2391a53972a2f83e347d40ab946

[fdio][namespaces] remove overlay functionality

Under the new rules you cannot bind a path below a path
that has a remote bind point. So, for example:

create(ns) OK
bind(ns, "/dev/foo", 1) OK
bind(ns, "/dev/bar", 2) OK
bind(ns, "/dev", 3) FAIL
bind(ns, "/tmp", 4) OK
bind(ns, "/tmp/tmp", 5) FAIL

create(ns2) OK
bind(ns2, "/", 6) OK
bind(ns2, "/tmp", 7) FAIL

And when you open something via the namespace you either get a
pure virtual directory (like "/" or "/dev" in the above example)
that you can readdir() and open_at() but cannot otherwise interact
with, or you get a clone of the bound directory ("/dev/foo, "/dev/bar",
or "/tmp" in the example) which you can pass to another process,
do all the usual directory operations on, etc.

Change-Id: Ifc88718c532a7cf24fbbb7bdd728978cc8382304

[devmgr] Wire up 'unmount on poweroff' code

ZX-1815 #done

Change-Id: Ibe1ee4dbda58f4c65bd7f7cc0a000a23833ac9f7

[devmgr] Fix vfs_create_root_handle not to conflate status and handle

Change-Id: Ibbf74def1dbf5b91858987b80e84a51a6d0d3b86

[memfs][devmgr] Refactor MemFS into a library

Relocates the MemFS implementation from devmgr and splits it into
different files, depending on which Vnode type is being implemented.

Additionally, removes the unused "ioctl_vfs_mount_bootfs_vmo",
which tightly coupled the MemFS implementation with the devmgr.

Additional changes for ulib/memfs would be useful (splitting
the header, adding tests, adding documentation, etc), but
those changes are not included in this patch so the relocation
can remain simplified.

Change-Id: I66413215a4000565865c6394a09180922d82e16a

[fshost] Add mechanism to make /system readonly

We'll flip the bit on making /system readonly in a separate CL.

Change-Id: If04e6a6da3289a1bd0591ad2d237450a95a90da1

[fshost] Always mount /system remotely

Previously, /system would be a remote mount if /system was backed by minfs but
would just be a subdirectory if /system was backed by memfs. Now we use a
remote mount for memfs as well.

Change-Id: If1659fec0ad584dec414795238152952ebbbed96

[fs] Split open into 'open' and 'validateFlags'

This will allow flag validation to occur for cases
where the underlying Vnode is not actually opened,
as will be the case for O_PATH.

Change-Id: I75357e7efe42108eef6353f68794aaebc2abcc67

[fs] Make readdir return actual size as output parameter

One of many commits trying to reduce the overloaded use of
"zx_status_t".

ZX-1189 #done

Change-Id: I8b393b2d5440abfc7646223cab9f9c94115538a4

[fs] Use fbl::StringPiece for libfs paths

ZX-1145 #done

Change-Id: I43e7b8c45e8d53d5c7cb90273b5eb57b2f9b4464

[fs] Update Write API to allow for atomic append

Add 'Append' method to Vnode methods.

ZX-1150 #done

Change-Id: I2d1d593d923e3723a56eb9be7f3790e610bd13ec

[fs] Make GetHandles return hcount from arg, not return value

ZX-1153 #done

Change-Id: I4813a1076f50c434d24981cab77e5e979431282d

[libfs] Allow Open() to redirect to a different vnode.

This is useful for implementing pseudo-files and could also come in
handy for certain forms of dynamic resolution such as on-demand
instantiation of vnodes.

Change-Id: I9fedefc3e78ba7a8a8179a6bee98fed6cf5fb252

[fs] Remove Vnode flags, simplify remote waiting logic

Change-Id: I62421fbd61cf0629b96d5ec3ffd9186e3a762aa8

[devmgr][fshost] ability to compile fs glue into fshost

Not enabled by default since we can't quite boot in this mode
yet (need a little more plumbing and chicken-and-egg-juggling),
but this takes us one step closer to a separate filesystem
host process.

Change-Id: Ic667e4979b7aed53d040702ad294b6dad7db2a36

[vfs] Rename WatchDirV2 to WatchDir

Now that the migration is complete, we can use the nice name.

Change-Id: I41cb40173da4be6f2614ce54b2f32bf4eee56c8c

[vfs] Remove unused WatchDir method

All callers of the old deprecated WatchDir protocol have migrated to V2.

Change-Id: Ia0f729aac3ef00bc8b202d27f0e439198ad65208

[fs] Make the vfs_lock private

- Create a "MemVfs" class in vfs-memory, which is
capable of redirecting functions to locked versions.
This prevents the need for the vfs_lock to be publically
exposed from the fs::Vfs class.
- Create a locked "Readdir" function in fs::Vfs, which
encapsulates locking against path modification operations.

Change-Id: Iedc422c857f6ed876f25cc459dc63db7b568d4df

[fs] Pass actual bytes through out parameter (read, write, ioctl)

ZX-1141 #done

Change-Id: Ie3b42cdf3394b52336b85fdbfb05eb5222280684

[fs] Separate Vnode, Vfs include files

Additionally, fix some headers, remove "ifdef cplusplus"
macros in ulib/fs (it isn't functional for C filesystems),
make the Readdir cookie typed, and make "MountNode" private
to the Vfs class.

MG-1117 #in-progress

Change-Id: Iba2169875b5dd73619f3e8c919fefb998a94a08f

[zx] Magenta -> Zircon

The Great Renaming is here!

Change-Id: I3229bdeb2a3d0e40fb4db6fec8ca7d971fbffb94

[devmgr][memfs] enable pipeline-style connections to the root fs

Change-Id: I5eebb05ef28b77823826d868b50870f918f99395

[fs] VFS-related cleanups

- RemoteContainer, WatchContainer moved to their own headers.
(Vnode be moved in a later patch).
- V_FLAG --> VFS_FLAG
- "vfs-XYZ.cpp" renamed to "XYZ.cpp" (headers too)
- WatcherBuffer (implementation detail) moved out of header.

MG-1117 #comment in-progress

Change-Id: I39471b142564d3d8bcf90a05949073d0fa1b32d3

[zircon][mxtl->fbl] Rename mxtl to fbl

Change-Id: Ie21b6498e1bfb0a7fa0315e40b9e5c3ee78646be

[devmgr] Add ino values to Memfs vnodes using an atomic counter

Change-Id: Ic22ab9ff513bbdfcabf959079ecdb47957c948c0

[fs][memfs] Fix mtime in MemFS, improve mtime testing

MG-1086 #done

Change-Id: Id543b316f491703f7f0d9ed742d40654a9970701

[fs] Utilize managed handle classes

Additionally, create "mountchannel", which
unmounts a handle which has been passed as an
argument to be mounted.

Change-Id: I119371dce4358ae3027d3733419ff332d5d1227e

[fs] Improve vfs_lock_ encapsulation

- Convert thread annotations to no-op on builds with
"DISABLE_THREAD_ANNOTATIONS" defined; this allows static thread
analysis for operations which act on filesystems that
are compiled for both host and target (such as MinFS), but
which use a single-threaded model for host-side tools.
- Add thread annotations to VFS operations
- Continue refactoring required to eventually make "vfs_lock_"
internal to the VFS class. "Mount + Mkdir", "Locked" functions,
and iostate token access are now all internal to the Vfs class.

Change-Id: I5d083f38726c493602cd24139303ab377ef655e4

[fs] Make VFS non-static object holding filesystem-global state

Change-Id: I60735f33a751e98d1b2324886b59d3144ef49276

[libfs] Move more common code into libfs and use mx::channel.

Move the check for whether a Vnode really is a directory into libfs
and renamed fs::Vfs::ServeFilesystem to ServeDirectory to more clearly
indicate this responsibility.

Changed the vfs dispatcher contract to use mx::channel to clarify
where handle ownership transfers occur.

Change-Id: I4238518dd6a0d3ac3cf64ea991ed9df9a3eb0e52

[libfs] Pass fs::Dispatcher as an argument to Serve().

This change removes GetDispatcher() from Vnode and instead requires it
to be passed to Serve. This cleans up the code a bit and eliminates a
fair amount of boilerplate downstream.

Change-Id: I0f6297f26642f5d55e9b65912441d7e905bd3729

[fs][devmgr] Add memfs implementation of mmap

Additionally, created small test suite for mmap
on general-purpose filesystems.

Change-Id: I57b6a543dc0a845fb382dab01a1214fe5bc783fe

[fs] Add support for vmo-supplied file creation

The provided VMO handle must be the LAST handle
to the VMO to prevent resize attacks.

Currently, only supported by memfs.

Change-Id: I2a7766f8bd579a8cd9a42227953962284c8c9441

[memfs] Replace override with final where appropriate in Vnode class

Change-Id: I8db7c286dc7fc2fa8dc1bb29359a0fc2dec9b1fc

[fs] Generalize Vnode notifications

Previously, the WatcherContainer was only capable of receiving
'add' notifications. This patch allows the container to receive
(and transmit to clients) generic events from the watch_dir_v2
protocol.

MG-834 #comment In Progress

Change-Id: If88636a61cec94255fe3000fad4597a9c0c144ec

[fs] Add VFS support for v2 of the fs watcher

Change-Id: Id32f3eb458b1d721d43f615ed18c8ed84004d355

[devmgr] migrate to new error codes

Change-Id: I74ca599e1c0eea0a8f4ef97ab12d392b0142bb8a

[memfs][vmofs] Use vmo_clone to create client VMOs

Previously, we gave clients access to the VMO for the entire file system. Now
we give them a clone of just the portion of the VMO that contains the file they
are accessing.

Change-Id: Ie4f589eb2a938fdc37a2ac8aa82f2ff6faf2b596

[devmgr][vmofs][blobstore] Prevent opening readonly files as writable

An open using O_WRONLY or O_RDWR should fail if the object is
known to always be immutable.

Based off of https://fuchsia-review.googlesource.com/c/31662, by
Roland McGrath.

MG-804 #done

Change-Id: Id17634824da3815af01761788691023b28ad4185

[fs] Decoupling filesystems from dispatchers

- Make the VFS layer only interact with the abstract
"Dispatcher" interface.
- Subclass this Dispatcher with the following:
- MxioDispatcher, which wraps the C dispatcher
- VfsDispatcher, which wraps the multithreaded dispatcher

Additionally, clean up the VfsDispatcher (making it use ulib/mx,
unique_ptrs, and clang thread-safety annotations).

Change-Id: I07b3a4e990404a98cf337740b16791c84ac4d232

[devmgr] remove devfs bits from shared memfs code

Change-Id: Ib7e7a5f9de875a1a11a82f49be14aa8ae0b273bf

[devmgr] devcoord now uses internal devfs implementation

This allows tighter coupling with the underlying device_t tree
and lifecycle, and will allow further reduction in duplicate fields
between devnode_t's and device_t's in later changes.

memfs will no longer need to support the quirks of devfs. The only
remaining API is devfs_mount(mx_handle_t) which installs the remote
handle in /dev during startup.

Change-Id: Ib3cc0eeac2c8d38bf3e9e4f03efac3a333b3f194

Revert "[devmgr] devcoord now uses internal devfs implementation"

This breaks /dev/socket, sadly, so roll it back for the moment.

This reverts commit ddca75b2027cb09b434fec5f38531ecdf3c51e0f.

Change-Id: I7c81a481794899b9d07fd28778e7418c01f1c2bf

[devmgr] devcoord now uses internal devfs implementation

This allows tighter coupling with the underlying device_t tree
and lifecycle, and will allow further reduction in duplicate fields
between devnode_t's and device_t's in later changes.

memfs will no longer need to support the quirks of devfs. The only
remaining API is devfs_mount(mx_handle_t) which installs the remote
handle in /dev during startup.

Change-Id: I4cd9ba5b5e2a9dd3d9967fa1bc99ad21f813e658

[fs] Simplify Vnode Closing

- VFS close didn't do anything; get rid of it.
- Most filesystems return NO_ERROR on close; make
that the default.

Change-Id: Ie53cacf07c934b3be5e482daac72b75203843ece

[fs] Add WatcherContainer for simple VFS directory watching

Additionally, allow the watcher to trigger on general
"entry added" activity within a directory (create, rename, link).

Change-Id: I369faf984442025816ac645ffae46087682e0f9d

[fs] Make Vnodes refcounted

Rather than using the old "RefAcquire" and "RefRelease"
functions, manually incrementing and decrementing, this
patch updates Vnodes to use "mxtl::RefPtr".

MG-719 #done

Change-Id: Idb998a36c52d44aaf35b6a03f4c46077eaa7709f

[fs] Add RemoteContainer, making the 'remote_' field optional

Previously, fs::Vnode has a 'remote_' field, which could
be used to mount remote devices, services, and filesystems on
top of parent nodes. For MemFS, MinFS, and other traditional
filesystems, this provided an easy mechanism to get the
ability to mount sub-filesystems for free. However, for
other filesystems (service FS, blobstore), mounting is not
a desirable functionality, and the "remote_" field is wasted.

This patch creates the "RemoteContainer" class, which can be
used by Vnodes that want to host remote filesystems.
Additionally, this patch updates MinFS and MemFS directories
to use this RemoteContainer.

Change-Id: Ic56e84a82f01503119f16e39c70967b14cb9cbc5

[bootfs] Make bootfs / systemfs read-only

Change-Id: I25b46d5f992a754f4ca4b215339252097f37d911

[mxio][fs] Update RIO to use reply handle as communication channel

RIO previously opened new connections by doing the following:
- Create a client-side reply channel
- Send half of the client-side reply channel to a remote server,
where it may be forwarded across mount points to remote devices.
- When the request reaches the final server endpoint, a new
channel is created, where one end of the channel is used by
the server (to dispatch) and the other is sent back to the
client (for fd-like control) over the reply channel.
Any additional handles are also sent.
- When this transaction is complete, both the client and server
close their reply channels.

This approach is problematic for a few reasons:
- It's wasteful. The client and server ultimately want to each
have a handle to either end of a single channel, but this transaction
creates two channels in the process.
- It's not async-friendly. It's impossible for RIO clients
to asynchronously issue a request for a connection and
dynamically react to a channel handle, expecting that it may
become valid in the future (possibly enqueuing future requests on it
asynchronously as well), and this limitation is due to the fact
that the "real" communciation channel is received from the
requesting reply channel.

This patch modified the Remote IO protocol to use the reply handle
as the *real* communication channel. Preventing the unnecessary
channel creation, and allowing for future expansion into pipelined
open requests.

Change-Id: I45b093c3ec14613a7b6b92c05b49e59afe9022a9

[fs] Unify device and remote handle opening

Change-Id: I1c43a31a0d0cfc1bae227dee5213042578dc8279

[devmgr] Refactor & Clean devmgr Vnode constructors

- Avoid 'combined' create function. This makes passing arguments
and initializing vnodes with different demands difficult.
- Avoid leaking 'combined' create function to vfs-boot.
- Delete "create_from_buffer" functionality.

Change-Id: I6722b3076dc384338a4e689a4acfcd98400ca4bd

[devmgr] Improve MemFS locking

- Convert "_foo" functions to "foo" where appropriate,
and to "foo_locked" when a lock is required.
- Converted mtx_locks to mxtl::AutoLocks
- Added Clang thread-safety annotations

Change-Id: I21f04fff787d0d6e2aa676342d09dcd223f85e8a

[devmgr] Convert Vnode Watcher to C++

- Relocate Vnode Watcher from "vfs-devmgr" to "vfs-rpc".
- Convert watch_list to C++ doubly linked list of
unique_ptr watchers
- Move "watch_list_" to VnodeDirectories exclusively
(Vnode files / vmos don't need to waste space for a
watch list).
- Add comment to "ioctl_device_watch_dir" declaration
explaining lack of string termination.

Change-Id: Ic34f15c4b459c6b77f949961e419c285819bf64e

[devmgr] Fix device removal regression

Reinstate "dn_list_" for device hardlinks within C++
implementation of dnodes.

MG-661 #done

Change-Id: I39a55c8edc334dcd00b6da072c8393670bf1115a

[memfs] Convert dnode to class, make them refcounted

- Converted dnode struct to C++ class. All data members
are now private, and access to them is restricted.
- Dnodes have names, which are accessible via "unique_ptr".
Updated implementation of "rename" to quickly steal
this name, rather than allocating a new dnode.
- Removed unused "vn_entry". Removed unused "dn_list_".
- Made "dn_entry" implicit by converting Dnode into a doubly linked
listable. This may be updated in the future to allow faster lookup
times.
- Converted "VnodeDevices" to subclass from "VnodeDir"s, since
devices ARE directories.
- Reimplemented readdir to avoid leaking pointers
to dnode in cookie.

Change-Id: I08fd3b23a609af4138edf84facad800521c5db33

[fs] Convert Mount Node list to C++

Change-Id: I2d6436b6ca43c968b0baa1a08e393e03fba940fc

[fs] Test and defend against 'bad' mounting

This patch tests for
- Mounting on files
- Mounting on unlinked directories
- Double mounting

Additionally, this patch protects all filesystems against
these types of malicious mounts. If the "AttachRemote"
virtual function is not overwritten, a filesystem will
by default be unable to mount sub-filesystems.

Change-Id: Ic53e79dcb824c73b7842da2afc78f2136bc26187

[fs] Refactor Ioctls, eliminate 'local ioctl' hack

Reduce the Ioctl call path from:
FS Handler --> Vfs::Ioctl --> do_local_ioctl --> Vnode::Ioctl
to
FS Handler --> Vfs::Ioctl --> Vnode::Ioctl

Change-Id: I6e51c16a6294ea8e35aee9ed5bd08b8e4e882688

[fs][minfs][memfs] Add support for unlinking open directories

- Update tests to allow these "empty, unlinked" directories.
- Update test suite to use "rmdir" instead of "unlink".
Exclusively for clarity; they functionally identical at the
moment.

MG-629 #done

Change-Id: I37024b873471639a83562fe9dfb68e18e4e1d3ee

[fs][minfs][memfs] Report/Fix/Test hard link counts

- Hard link count now propagated back through stat.
- Hard link counts tested in a variety of situations:
- Directories should be created with two hardlinks,
one for "." and one for their parent direntry.
- Adding a child directory should increase the parent
link count (due to the child's new ".." entry).
- Files should be created with one hardlink, for
their parent direntry.
- Test hard link count under a variety of conditions:
- Adding directories
- Unlinking
- Linking
- Renaming

Change-Id: Iea72b497b165ec4c471363840d3736fda1b75ffa

[fs] Convert Vnodes to C++

- vnode_t struct split into:
- fs::Vnode abstract base class, with default "ERR_NOT_SUPPORTED"
versions of operations.
- MinFS: subclass minfs::VnodeMinfs
- Blobstore: subclass blobstore::VnodeBlob
- MemFS: subclass memfs::VnodeMemfs, which provides some common
functionality to all types of MemFS vnodes, but which is itself
subclassed into "VnodeFile", "VnodeDir", "VnodeVmo", and
"VnodeDevice".

Change-Id: I114acd067dec7849f1591eddedf020102ddf7e67

[memfs][bootfs] do not close shared VMOs on release() of vnode

Fixes US-152

Change-Id: I2df977b2b396f86eb0f7206b80ae887088c0f458

[devmgr] update utimes/setattr for memfs

. implement setattr so *utimes* doesn't core dump
. add create_ and modify_ times for memfs, so time
functions are sensible
. cleanup: implement placeholder _none functions for ioctl
and setattr for types that don't have any operations
. cleanup: s/mem_/memfs_/g [mem_ and memfs_ prefixes meant
something in the old world

Change-Id: I35d88149ae67db155ce70ee1aff27e8bfa1abad6

[devmgr][fs] Mount volumes at location based on type GUID

Change-Id: Ief829a6aa15b52e31cacff35807a7e8ed8eb7396

[devmgr] Make MemFS use VMOs instead of alloc'd blocks

Change-Id: I393605f0cf1084d50d6f7ea3f3d7873125da5739

[devmgr][minfs][fs] Unified RPC and Mounting layers

- Allows MinFS and MemFS to share remoteio-related code, especially
related to dispatching mxrio messages to VFS operations.
- Allows MinFS to mount sub-filesystems.
- Removes iostate tracking, which is currently in the form of an unused
linked list. This can easily be re-added later, but it (in present
form) prevented clean RPC unification.
- Unified iostate. Removed unused 'device' field of MemFS iostate;
devhost already has it's own distinct iostate, so this field was
unused.

Change-Id: I0fc8a7a3b3a7a738637d54828cefb18ed6e9c865

[devmgr][minfs][mxio] Enforce that trailing '/' ops act on directories

- Added arguments to 'unlink' and 'rename', specifying additional
requirements on "*must_be_dir"
- Added correspoding error cases to MinFS and MemFS
- Added fs test which identifies these error cases for files
- While I was there, changed some camelCase to use_underscores

Change-Id: Ia481d35fffac100fda88a2b93a75ef57346679c3

[devmgr] unify memfs minfs vfs.[ch] helpers in devmgr

next patch will add the same vfs.[ch] to minfs, then
subsequent patch(es) will move them into a common library

made common trace functions for debug in trace.h
created vfs-devmgr.c to hold devmgr-specific vfs code
created memfs-private.h to declaere devmgr-specific vnode & ifcs
reordered include files as necessary to get local-> system-> std

Change-Id: Iaf58822ac0ea6f05758d1210e33581975bf9f91d