[fs][minfs] Allow filesystems to implement custom interfaces

As a simple test case, propagate the metrics collection API
to minfs, and test these fields (which previously could
only be dumped to stdout).

ZX-2710 #done
Test: Expanded minfs tests

Change-Id: I6b45693731d912be3fab7d95f0bc1ff7d52fb593

[fs][fidl] Replaced filesystem ioctls with FIDL.

This patch allows "ioctl" to be removed from the
C++ VFS Vnode interface.

IOCTL_VFS_MOUNT_FS -> Mount()
IOCTL_VFS_UNMOUNT_FS -> Unmount()
IOCTL_VFS_UNMOUNT_NODE -> UnmounteNode()
IOCTL_VFS_QUERY_FS -> QueryFilesystem()
IOCTL_VFS_MOUNT_MKDIR_FS -> MountAndCreate()
IOCTL_VFS_GET_DEVICE_PATH -> GetDevicePath()
IOCTL_VFS_WATCH_DIR -> WatchDir()

Test: fs-management / fs / blobfs / fvm tests, df manually

Change-Id: Iefe8ec251ca6b8f4e4578ceb61bae6dff7a59e3d

[fdio] Remove redundant ordinal definitions

Test: No behavior change
Change-Id: Id0d174229deda2eecdb3ba5ccdfc028053f20d37

[fdio] Don't repeat definition of fidl_message_header_t

We can just embed the actual fidl_message_header_t struct rather than
repeating its definition.

Test: No behavior change.
Change-Id: I666c1fa8a6da1c02142add1150dc05dc6a3cf016

[fs] Deprecate GetToken ioctl

Test: CQ

Change-Id: I1e328f6ae7af0c624fbedce7035fdff74c830fec

[fs] Implement Describe for C++ file systems

This CL implements the Describe method on Node.

Test: memfs-test.test_fidl_basic
Change-Id: I07be854beb04a7d42fe3c08a1a9aac85c361641c

[io][fidl] Update io.fidl to comply with the FIDL style guide

As documented:
https://fuchsia.googlesource.com/docs/+/master/development/api/fidl.md

- Constants must be named in ALL_CAPS_SNAKE_CASE.
- Enum members must be named in ALL_CAPS_SNAKE_CASE.

Test: CQ

Change-Id: I643e0aec420c3c61753fad15a03a79f6f4fa9d52

[fidl][io] Removed unimplemented FIDL interfaces.

Until we decide on a mechanism for FIDL interface discovery,
remove this bind method. It is better for the method to not
exist than for no one to implement it.

Additionally, remove GetVmoAt, since it is currently not
implemented.

Test: CQ

Change-Id: I804b3908c39e4c66b01693f8075ad0d0822ca53f

[fd] Rename zxrio_object_info_t to zxrio_node_info_t

The new name matches the name in the FIDL protocol.

Test: No behavior change.
Change-Id: I742b72c2adbbaaf6aa16063edad7126a31d74bae

[fidl] Flatten interface Object and Node into Node

Test: fs-tests

Change-Id: I1cc0aaf4e89caaa821b75361010cea5f2d77c644

[fs][fdio] Ensure O_APPEND is not lost on clone

Additionally, ensure that O_APPEND may be both
toggled on and off using fcntl. For cases where
fcntl is called with "F_SETFL" and the flags "0",
it's not possible to tell if this is turning off
a remote "O_APPEND" state without actually
sending the RPC.

Test: /boot/test/fs/fs-test, new append tests added.
ZX-2519 #done

Change-Id: I86ab0ad61554820409a1eb52aa83b10e590d8d3e

[fdio][fidl] Replace fuchsia-io hand-rolled client bindings

ZX-1919 #comment In Progress

Test: CQ (fs tests, fdio tests, blobfs tests, etc)

Change-Id: I31588bb62f5788fb9372fdc7042f42a3cae7c71c

[fdio][fidl][devmgr][fs] Deprecate Remote IO, enable FIDL2

Test: runtests (fs tests, fdio tests)
ZX-1919 #done

Change-Id: I459eb79546a3b952633faad2bb4914d9a3c68e05

[fidl] Use autogenerated fuchsia-io header

Remove hand-rolled verison.

Test: CQ

ZX-1919 #comment In Progress

Change-Id: I8816a9616067e4d437385db4d552d385e33eb975

[fs][fidl] Fix computation of attributes.storage_size

From the vnattr docs:

blksize: Block size for filesystem I/O (not always = VNATTR_BLKSIZE)
blkcount: Number of VNATTR_BLKSIZE byte blocks allocated

Test: fs-test (in particular, TestFullOperations)

Change-Id: I2e5a3aea6a51ae99db7eec9e1625edff4dac1e26

[fs][fidl] Fix lseek offset return bug, add tests.

Minor bug fixed in 'fidl_seek' implementation,
multiple tests added to prevent additional regression.

Test: /boot/test/fs/fs-test

Change-Id: Icdd5d79ffeee9e20c48b7b2373fbb1ac820bacb1

[fs] Ensure that cloned vnodes are not opened with O_PATH

ZX-2346 #done
Test: Added test_opath_fdcount to prevent regressions

Change-Id: I7a71dbd0f210b35a4d8f33c8f7938cc01c22d58c

[async] Change async_t -> async_dispatcher_t

Change name of dispatcher type from async_t to async_dispatcher_t
Keep typedef of async_t for compatibility

Change async_{get,set}_default() to async_{get,set}_default_dispatcher()
Again, keep compatible apis until other layers have been modified to use
new api.

Bug:ZX-2337
Test:CQ
Change-Id: If0b5d061fd114d60e23078dbb5f2a6ed9ed6f6cc

[fs][fidl] Implement Directory::GetToken (server-side)

ZX-2315 #done
Test: fs-tests, although no client currently exists in Zircon

Change-Id: Iba08e88d7cbe92b2470a1934ed4b395bda17c4f7

[fidl][rio] Ensure FIDL_HANDLE_PRESENT is correctly set/checked for ObjectInfo

Test: runtests (fs tests, memfs tests, driver access works), with and
without ZXRIO_FIDL defined.

Change-Id: Ie5fa400575c3fd1cbd5eb5b88280c7e99f3d45d2

[fdio] Move headers into lib/fdio/...

Change-Id: Ie8d74e716da913bf6e2672c4acf8cd67b4962b7f

[fdio] Regenerate I/O bindings to use fuchsia_io prefix

Change-Id: I444cda2448ca2a560abb7a38513f40005ac2fdce

[fs] Add a managed subclass of Vfs capable of terminating synchronously

Change-Id: I56bf34ed1c25dee556c8d0ad0e12282a9e4752ef

[public][ulib][dev] Add __FALLTHROUGH macro for warning suppression

GCC 7 enables -Wimplicit-fallthrough with -Wextra, so we'll get it.
Clang has it available, so start using it and clean up code accordingly.

The GCC feature will actually recognize some formulaic comments to
suppress the warning. But there's also an explicit syntax, and
C++17 added a standard one. So use that instead, via the new macro
__FALLTHROUGH in <zircon/compiler.h>.

TC-36 #comment Clean up code in preparation for more -Wimplicit-fallthrough

Change-Id: I28309c91c32db1313d52f4255836c5ea687e141f

[fidl][io] Regenerate, update usage of io bindings

Change-Id: I9990f4c98a2c57ee8d3e953093b7be7e2d4c71e4

[fs][fdio] Explicitly observe VFS terminating status

ManagedVfs teardown involves the following:
1) On the async dispatch thread, identiify
that the vfs is terminating.
2) Signal all open connections.
3) When all connections are terminated, invoke
a ShutdownCallback.

Unfortunately if a connection (also executing
on the async dispatch thread) receives a port
packet that their channel is "readable", it may
be dequeued without identifying that the channel
is also "signalled for teardown".

Expose the "terminating" status of the VFS layer,
and explicitly check that instead of observing the
signal. This prevents a flaky failure condition
in memfs test.

Change-Id: Iedc17ae0cce759eec81bdd5e2d5a14a92613f2fa

[minfs][blobfs] Re-implement filesystem teardown

Rather than simply "sync + exit process", implement
a fully featured filesystem teardown procedure, capable
of:
- Tearing down all active connections.
- Syncing underlying media using a non-blocking dispatch loop.
- Use closures to signal both mounters and unmounters on
filesystem destruction.

As a consequence of the controlled filesystem teardown
procedure, make the "filesystem" objects in minfs and blobfs
no longer refcounted, to clarify that "Shutdown" is the
single point of destruction after mounting.

ZX-1577 #done
ZX-1923 #done
ZX-1959 #done
ZX-1576 #comment In Progress

Change-Id: I3f8d71af76e391c41c665434baf86a245a7cd223

[fs] Re-implement ManagedVfs for async callback safety.

Multiple race conditions could occur while using ManagedVfs:
- Tearing down connections racing with asynchronous callbacks.
- Destroying the ManagedVfs racing with tearing down connections (which
may be serviced by async callbacks, without accounting).
- Destroying the ManagedVfs racing with torn down/newly registered
connections.

To resolve these issues, provide an explicit "ShutDown" API which will
invoke a closure on behalf of the caller when all Connections (and their
corresponding Vnodes) have terminated. The ShutDown implementation
signals all local channels to terminate, which eventually results in all
Connections tearing down.

Simulate these race conditions and test their solutions using a custom
Vnode.

ZX-1576 #comment In Progress

Change-Id: Id3850e3fec55e1417e79ae9417ec1c5d1c818fe8

[async] Tidy up the libasync API futher.

Deleted WaitWithTimeout since it is no longer used.
Removed flags and reserved storage.
Removed return code from wait handler.
Removed *_or_report_error variants after further consideration.
Deleted the non-RAII variants of the C++ wrappers; these wrappers
might as well be safe to use. Clients desiring minimum overhead
can use the C API instead if they prefer.
Made the C++ wrappers more systematic; each one has a fbl::Function
and a Method based variant.
Added a C++ wrapper for Tasks which takes a no-arg closure.
Improved consistency across all async operations, including docs.
Ported all programs to the new API as required.
Beefed up the tests.

Change-Id: I97beca26cec6d94fa7efd282c8d2620cdbcb84e5

[zx] Move public zx headers to match naming convention

The zx library provides headers intended for general use, so the
include path should be spelled <lib/zx/foo.h> as per
https://fuchsia.googlesource.com/docs/+/master/development/languages/c-cpp/naming.md.
This moves the headers into the proper location and updates includes to
match.

Change-Id: I6fc9b2a491b5f8b0d270c4dfc77fa4be5739071b

[fs] Add "FLAG_EXACT" to mmap, re-enable MAP_SHARED for blobfs

- MAP_PRIVATE implies a COW clone of the underlying VMO must
be used.
- MAP_SHARED specifies that updates to the underlying file
should be visibile to the filesystem and vice versa.

These two modes are not capable of expressing a third modality:
acquisition of an 'exact', non-cloned VMO. Although MAP_SHARED
is superficially somewhat similar to this requirement, MAP_SHARED,
according to the posix specifications, only dictates the behavior
of updates, not the implementation. As a consequence, for read-only
filesystems, "MAP_SHARED" and "MAP_PRIVATE" can both be expressed
by requesting a clone of the underlying file VMO.

This patch expands the FDIO_MMAP flags to include "FLAG_EXACT",
allowing callers to express a desire for this 'exact match' which
is not expressed with MAP_SHARED.

PKG-31 #done

Change-Id: I564f91761ea7a8a17634de66eb3b004a2b0c9047

[fidl] Rename fidl2 things to just fidl

Change-Id: I17780c327272c554b4c80a7c5fc64df78d0a5601

[fs] Ensure Vnode::Close is called on unclean teardown

The contract of ulib/fs is that every call to "Open"
will be paired with an accompanying call to "Close",
in the event of either:

1) A client sends a close message explicitly, making
their channel defunct (and no longer observed), or
2) A client closes their channel, implicitly closing
their connection to the filesystem.

Although the Connection class in ulib/fs would properly
cancel waits and tear down the connection object,
it previoulsy missed sending a close in the second
condition, since it invoked "CallHandler" while the
channel was not yet ZX_HANDLE_INVALID.

This patch ensures that the close handler is invoked
in this condition.

ZX-1853 #done

Change-Id: I291192cae32e8250f7cef932cbf1a4de4b8b2d2d

[fidl2] Recompile io.fidl2

The C FIDL2 parser has changed in the past month;
this patch adjusts the Zircon codebase to deal
with those changes.

Regenerated using:
./build-x64/tools/fidl2 \
--c-header system/ulib/fdio/include/fdio/io.fidl2.h \
--files system/ulib/fdio/include/fdio/io.fidl2

Change-Id: Ia7671f201dbd89b44149ce744d5c247f206da89b

[fs][fdio] Simplify the 'Mmap' operation, convert to 'GetVmo'

This operation used to have some weird semantics: an input
length parameter, which was largely ignored, and an output
"offset" parameter. The reason for these arguments was for
servicing vmofiles, which are handled client-side anyway.

Rename the 'Mmap' operation to 'GetVmo' (since that's what
most filesystem servers are implementing anyway) and avoid
passing unused arguments.

Change-Id: I3fe9de194c3ae9d14a1f7dd790adba7fec1ab0e0

[rio][fidl] Convert RemoteIO operations to FIDL2

This patch updates all Zircon servers to respond to
either RIO or FIDL2 requests. This will allow
compatibility with clients speaking either protocol.

Although this patch allows clients and servers to communicate
with FIDL2 instead of RIO, it is a slight modification
from strict FIDL2 standards: Most servers don't implement
the exact FIDL2 Object / Node / File / Directory protocols,
they just implement a subset of all operations. Updating
these servers to exactly implement these interfaces
will occur in subsequent patches.

ZXRIO_FIDL toggle can be used to request RIO or FIDL2 client
requests.

TO-496 #done
ZX-1358 #comment In Progress

Change-Id: Ia52f469e553be977deb3d0f280ba3efaf36ed893

[rio][fidl] Align RIO Description with FIDL2

This change eases the FIDL2 conversion by ensuring
that regardless of which protocol a client/server may
be speaking, objects will at least openable regardless.

ZX-1697 #done

Change-Id: I3197e9dbe00d2469618d55968e76696210a15c39

[crypto][fdio][zxcrypt] Improve debug output

This CL improves fdio/debug's xprintf with location info.

Change-Id: Idd05c0439258ed3fe160086bcce05575eddbc38b

[fs-management] Add timeout for failed mount tests

Zircon filesystem mounting is a multi-stage process:
1) Handles to the root Vnode and block device are created
2) A filesystem process is mounted, which begins
loading necessary structures to serve the root Vnode.
3) The root Vnode handle is passed to a mount point, where
it is attached.

The "fs-management" test suite attempts testing multiple cases
where step (3) fails. Due to the fifo-based block protocol (which
requires exclusive access to the underlying block device), if a
filesystem is not properly torn down, the NEXT mounted filesystem
attempting to use the underlying block device may discover that it is
already in use.

The following presents a sequence of events where this is possible:

Test A: Start, Steps (1), (2), (3 -- fails, as expected)
Test B: Start
Test A: Start (but don't finish) unmounting filesystem
Test B: Steps (1), (2 -- fails, unexpectedly, since block device is in
use)

This patch adds a small timeout to the locations where, after failing
to mount, a filesystem is unmounted. This gives the underlying
filesystem a moment to clean itself up before continuing.

An alternative, adding timeout code to the "failed mounting" pathway
on the filesystem server is cleaner from a testing perspective, but
wasteful from a practical perspective: there is no reason a failed
mount should block the parent (mount point) server for any amount
of time.

ZX-1629 #done

Change-Id: Ie07459cd96f3dad859a3cadc3083f401c2d54f39

[fs] Convert Sync (flush) operation to be asynchronous

Since the "Sync" operation may block for disk to complete
a potentially expensive operation, it is ironically a perfect
candidate for becoming an asynchronous operation.

This patch updates the VFS interface for sync to use closures,
allowing filesystems to defer responding to the requesting
process.

ZX-1308 #done

Change-Id: Ibc00c4963be79ff80504a7dafcffce6d81ccc66f

[fdio] Use ZXRIO_ON_OPEN unsolicited requests for open

- The zxrio_describe_t object replaces the zxrio_object_t
with a FIDL2 compatible format.
- Flip the usage of ZX_FS_FLAG_PIPELINE to be implied by
default. "ZX_FS_FLAG_DESCRIBE" may be used to request a
description object.
- ZXRIO_DESCRIBE can be trivially implemented using the
description object -- it currently isn't, because there
would be no clients through RIO.

ZX-1360 #comment In Progress

Change-Id: I1c7cad185af468b26c770dfd092382814a436c04

[fdio] Remove unnecessary void* cast of rio callback

Change-Id: I0149c24b3b9f0457ed4e2de4ced0faeb9c5a94c6

[fs] Re-open vnodes which are ZXRIO_CLONE-d

For filesystems which track "external fd count",
this needs to be updated, because there will also
be an increased number of eventual closes.

ZX-1494 #done

Change-Id: Ic531ab018468ac72166f2e753679c860267fb1df

[fs] Use the Fuchsia-defined open flags

Re-lands 2aff2cba029cf0f7e98e6d19f6dc7a663c318227

This change uses the Fuchsia-defined flags on the wire
as well as on filesystem servers. Clients interacting
with filesystems through the POSIX compatibility layer
can continue using POSIX open flags; they will be translated
by the RIO layers client-side.

ZX-1359 #comment In Progress

Change-Id: I4972c024547a0daa2146831e49e34d3ef905d4b4

[fs] Add simple tracing support to ulib/fs

ZX-1302 #comment In Progress

Change-Id: I6611a7a63fa6e6d068bfe9d31b0b4a4de152eac9

[fs] Fix racy usage of remote handle

ZX-1161 #done

Change-Id: I520cc89f79c81524e025b93dfa4f6b7d3c2ffaa1

Revert "[fs] Use the Fuchsia-defined open flags"

This reverts commit 2aff2cba029cf0f7e98e6d19f6dc7a663c318227.

Reason for revert: Fuchsia uses upstream Rust, which doesn't pull these constants from our C library, but their own. I'll need to wait a few days for us to update upstream Rust before we can make progress here.

Original change's description:
> [fs] Use the Fuchsia-defined open flags
>
> This change uses the Fuchsia-defined flags on the wire
> as well as on filesystem servers. Clients interacting
> with filesystems through the POSIX compatibility layer
> can continue using POSIX open flags; they will be translated
> by the RIO layers client-side.
>
> ZX-1359 #comment In Progress
>
> Change-Id: I27dfdea2024183d5d59daa1ed7d8886d89d503d0

TBR=kulakowski@google.com,smklein@google.com,swetland@google.com,mcgrathr@google.com,planders@google.com

Change-Id: If93c2da36f92ea28ed013a61c01c6ec521da26cc
No-Presubmit: true
No-Tree-Checks: true
No-Try: true

[fs] Use the Fuchsia-defined open flags

This change uses the Fuchsia-defined flags on the wire
as well as on filesystem servers. Clients interacting
with filesystems through the POSIX compatibility layer
can continue using POSIX open flags; they will be translated
by the RIO layers client-side.

ZX-1359 #comment In Progress

Change-Id: I27dfdea2024183d5d59daa1ed7d8886d89d503d0

[fs] Implement and test O_PATH

Although libfs will ensure invalid operations won't reach
the underlying filesystem, it is up to the implementation
of Vnode::Open to parse the O_PATH flag and construct
less state, if desired.

ZX-1151 #comment In Progress

Change-Id: I11f617b326f92b7ca03c3ec95d24db1e25ab2e59

[fs] Make readdir return actual size as output parameter

One of many commits trying to reduce the overloaded use of
"zx_status_t".

ZX-1189 #done

Change-Id: I8b393b2d5440abfc7646223cab9f9c94115538a4

[fs] Stop passing remote Vnode handle in zx_status_t return

Change-Id: I9ea0a6ffb08a04b21f53c2922ee27d85e6273dfc

[fs] Use fbl::StringPiece for libfs paths

ZX-1145 #done

Change-Id: I43e7b8c45e8d53d5c7cb90273b5eb57b2f9b4464

[fs] Update Write API to allow for atomic append

Add 'Append' method to Vnode methods.

ZX-1150 #done

Change-Id: I2d1d593d923e3723a56eb9be7f3790e610bd13ec

[fs] Make GetHandles return hcount from arg, not return value

ZX-1153 #done

Change-Id: I4813a1076f50c434d24981cab77e5e979431282d

[fs] Remove Vnode flags, simplify remote waiting logic

Change-Id: I62421fbd61cf0629b96d5ec3ffd9186e3a762aa8

[fs] Move VFS RPC code into Connection object.

The RPC functions become members of the Connection object itself
such that the Connection encapsulates both the state and the behavior.

As a result of this change, we no longer need separate AsyncDispatcher
or AsyncHandler objects since all of the work is being done by the
Connection.

Also moves the readable/writable predicates to a common location
and fixed them to use O_ACCMODE, which will make it easier to add
support for O_PATH later.

Change-Id: Ib2940aae1ff03d36d3310d6887ce725069373030

[fs] Convert iostate_t to a Connection class

Change-Id: I4fc8935e719c6919d5ae36d6bfddb7a1283f86af