[fs][minfs] Allow filesystems to implement custom interfaces

As a simple test case, propagate the metrics collection API
to minfs, and test these fields (which previously could
only be dumped to stdout).

ZX-2710 #done
Test: Expanded minfs tests

Change-Id: I6b45693731d912be3fab7d95f0bc1ff7d52fb593

[minfs] Return FVM allocation info through Minfs API

Additionally...
- Fix the FVM Query API (the partition manager and
individual partitions should return the same info).
- Export a constant from Minfs identifying the minimal
number of slices allocated to a MinFS partition.
- Expand the minfs "query info" tests, deriving expected
values rather than relying on magic numbers.

Test: /boot/test/fs/fs-test (test-minfs expanded)
ZX-2165 #comment In Progress

Change-Id: Ic72d595a97d06573d979df3999821ddddbfb1659

[fs][fidl] Replaced filesystem ioctls with FIDL.

This patch allows "ioctl" to be removed from the
C++ VFS Vnode interface.

IOCTL_VFS_MOUNT_FS -> Mount()
IOCTL_VFS_UNMOUNT_FS -> Unmount()
IOCTL_VFS_UNMOUNT_NODE -> UnmounteNode()
IOCTL_VFS_QUERY_FS -> QueryFilesystem()
IOCTL_VFS_MOUNT_MKDIR_FS -> MountAndCreate()
IOCTL_VFS_GET_DEVICE_PATH -> GetDevicePath()
IOCTL_VFS_WATCH_DIR -> WatchDir()

Test: fs-management / fs / blobfs / fvm tests, df manually

Change-Id: Iefe8ec251ca6b8f4e4578ceb61bae6dff7a59e3d

[fdio] Simplify process args and protocol names

Now that we can recover the type of a socket from the kernel, we only
need one process arg id for passing pipes, stream, and datagram sockets.

Also, replace the duplicated FDIO_PROTOCOL_* definitions with their FIDL
equivalent.

Test: No behavior change

Change-Id: I8b4bd830b9ef836827be6060cd0c4e0c096a7bbc

[fd] Rename zxrio_object_info_t to zxrio_node_info_t

The new name matches the name in the FIDL protocol.

Test: No behavior change.
Change-Id: I742b72c2adbbaaf6aa16063edad7126a31d74bae

[fdio][fidl] Use unaligned dirents for readdir, add ino

Test: fs-tests
ZX-2502 #done

Change-Id: I93e932a629f25c7da9f9a1a06e022d907ada47ed

[fs][minfs][blobfs] Simplify BlockTxn.

This patch acts as a pre-requisite for latter patches
which require transactions that combine operations, such
as "BLOCKIO_WRITE" and "BLOCKIO_FLUSH".

- Remove templatization of BlockTxn. This will allow
the implementation to be moved into a ".cpp" file
in a future patch. ReadTxn and WriteTxn still exist
as simple type-safe wrappers around an untemplated
BlockTxn.
- Rename "Flush" to "Transact"; "Flush" is an overloaded
term in the prescence of "BLOCKIO_FLUSH".
- Define an explicit TransactionHandler pure virtual
interface, to explicitly define the usage of a BlockTxn,
rather than relying on implicit templatization rules.

Test: runtests, fs-tests

Change-Id: Ic18687d31b44876ea9c06b49256ab25bd56f75db

[zircon][kernel] Use proper time types and overflow-safe arithmetic

This change is preparation for converting zx_time_t and zx_duration_t
from unsigned to signed types (ZX-2100).

1. Fix several places where the wrong time type was used (e.g.
zx_time_t instead of zx_duration_t). Because these are simple typedefs
it's easy to use the wrong one. While it generally doesn't cause a
problem, using the wrong type can be misleading and lead to bugs like
passing a relative time when an absolute time is expected.

2. Update kernel/ and some other parts of zircon to use the new
overflow-safe arithmetic functions from zircon/time.h.

Performance: Ran "/pkgfs/packages/zircon_benchmarks/0/bin/app -p" on
NUC before and after this change. Differences were withing the noise,
some slightly faster, some slightly slower.

Test: CQ

ZX-2100 #comment cleanup before conversion

Change-Id: I75d074e5850ade5385decd9bbf330497bf7610e9

[block-client] Add RAII C++ wrapper around block client

Test: fs, ramdisk, blobfs, fvm
ZX-2123 #done

Change-Id: I1f0d9eca48939c5c874fe5a6a9721e309e4eb338

[minfs] Pre-reserve inodes/blocks to mitigate mid-op failures.

The most common failure for a Minfs operation (in particular for
writes) occurs when we attempt but are unable to allocate a new
block or inode. To ensure that this kind of error cannot happen
partway through an operation and leave the in-memory metadata in
an inconsistent state, the maximum number of blocks required for
an operation will be "reserved" prior to any state-changing calls.

In many cases the reserved elements will be an over-estimated, in
which case they will be unreserved at the end of the operation.

Additionally, in cases where a directory entry must be appended in
conjunction with other updates, ensure that the directory has
space for the new entry before making any other changes.

Test: TestFullOperations in utest/fs/test-minfs

Change-Id: I16bad01344723dc6b1dddfd2a04c919846032597

[minfs] Avoid shrinking the VMO corresponding to a file

Instead, decommit the underlying memory when a logical
file shrinks, but keep the VMO a larger size.

Test: /boot/test/fs/fs-test -f minfs

Change-Id: Ib3617cd4d4c0ea6d3a1aaa5fef0a4fee386e4a72

[fzl] Move fs::MappedVmo to fzl::MappedVmo

This is a useful utility outside of the scope of just filesystems.

Change-Id: Id9cfa3d40e7d45dd6dd94242c1a85204f1139bfa
Tested: Run `runtests`

[block] Implement v2 of Block Protocol

- "Transactions" renamed to "Groups". The term
transaction remains, and is used to corresponds to a
single FIFO request.
- "Groups" pre-allocated by block server.
Clients can use groups between [0, MAX_TXN_GROUP_COUNT)
without needing to allocate. Groups signify structures
that can wait to transmit responses for a collection of
transactions. These transactions may be requested via
"BLOCKIO_TXN_GROUP"; the block-client library does this
automatically.
- "One-shot" transactions, which are not associated with
a group, are also implemented. All "one-shot" transactions
receive a FIFO response.

ZX-1584 #done
ZX-1580 #comment In Progress

TEST=ramdisk, fvm, zxcrypt, block, fs tests, running iochk manually

Change-Id: I7caf342804baf32543adfef3fec76f7d77d89b3f

[minfs] Ensure GetHandles is marked final

Also, only defined on Fuchsia; it doesn't make sense on host.

Test: fs-tests

Change-Id: I0d2b25acb8a3009f5109c97f5a81ad8257891129

[minfs][memfs] Ensure the appropriate FDIO_PROTOCOL type is set

Previously, these filesystems returned the default protocol type:
FDIO_PROTOCOL_SERVICE. This functioned correctly for the Zircon
C library, but was insufficient for more strict implementations
of the FIDL protocol (which refused to use "file / directory"
methods upon seeing the SERVICE type).

Test: fs-tests
ZX-2305 #done

Change-Id: I8eb438d5491f0d72c81064415669b5e58ff36bea

[minfs] Refactor allocator, inode manager, and superblock

The MinFS allocators should be opaque entities, only
leaking details of on-disk format during initialization.

This patch reduces the reliance on callbacks into the Minfs
superclass, instead providing ownership of the fields
which need to be updated to the underlying allocator.

Test: /boot/test/fs/fs-test -f minfs

Change-Id: I50090c17a2e2b7487aa072860af1286f3ab53e0b

[fvm][zxcrypt] Refactor libraries to avoid fdio dependency

ZX-2266 #done

Test: Refactor; relying on exitant library/driver tests.

Change-Id: Ia54acac0fb8056c91b26c097a8efae22dd0c5b3a

[minfs] Improve encapsulation of Inode Table

This restructuring will ideally make it easier to
experiment with different allocator / inode table
formats (such as B-trees, extents, space maps, etc).

Change-Id: I640c4c73b841ad40895f7ad5246a4e694a6c0c8d

[fdio] Move headers into lib/fdio/...

Change-Id: Ie8d74e716da913bf6e2672c4acf8cd67b4962b7f

[minfs] Improve encapsulation of Minfs 'info' field

This ensures that mutability of the global filesystem
counters is constrained to the Minfs class.

Change-Id: I2f6cdcefb458fb85c64e1a4694efae92cf777829

[minfs] Improve encapsulation of Vnode lookup table locking

Change-Id: I785bca62ebd39678f48cee772ca12c41ecb82ba9

[public][ulib][dev] Add __FALLTHROUGH macro for warning suppression

GCC 7 enables -Wimplicit-fallthrough with -Wextra, so we'll get it.
Clang has it available, so start using it and clean up code accordingly.

The GCC feature will actually recognize some formulaic comments to
suppress the warning. But there's also an explicit syntax, and
C++17 added a standard one. So use that instead, via the new macro
__FALLTHROUGH in <zircon/compiler.h>.

TC-36 #comment Clean up code in preparation for more -Wimplicit-fallthrough

Change-Id: I28309c91c32db1313d52f4255836c5ea687e141f

[minfs][blobfs] Re-implement filesystem teardown

Rather than simply "sync + exit process", implement
a fully featured filesystem teardown procedure, capable
of:
- Tearing down all active connections.
- Syncing underlying media using a non-blocking dispatch loop.
- Use closures to signal both mounters and unmounters on
filesystem destruction.

As a consequence of the controlled filesystem teardown
procedure, make the "filesystem" objects in minfs and blobfs
no longer refcounted, to clarify that "Shutdown" is the
single point of destruction after mounting.

ZX-1577 #done
ZX-1923 #done
ZX-1959 #done
ZX-1576 #comment In Progress

Change-Id: I3f8d71af76e391c41c665434baf86a245a7cd223

[blobfs] [minfs] [host] Combine common functionality between tools

De-dupe code between the minfs and blobfs host side tools as much as
possible. Additionally, add minfs support for adding manifests on create.

ZX-2033 #done

Change-Id: I0ece73d37927e6589ccd7d1ce235fc484668a221

[minfs] Writeback reorganization.

1. Pass WritebackWork to methods instead of WriteTxn

2. Add a Minfs method to create a WritebackWork

3. Add methods to handle updating the inode and block bitmaps

Change-Id: I22ba501e0a69351ac1b490215664a2a99e424daf

[blobfs] Update blobfs metrics aggregation for consistency with minfs

ZX-1587 #done

Change-Id: I8c1ff4968ee1f514bb17c93738cf32de21028a69

[minfs] Add aggregator metrics

These metrics are, if active, logged at unmount time.

ZX-1942 #done

Change-Id: Idc1d5f999bb23caa8f71ba1083fcc1321e37571a

[minfs] Improve error handling, begin logging rationale for IO errors

ZX-1830 #comment In progress

Change-Id: I263f0af430fbfee935057c9e67ac36fbf50845d5

[syscall][vmo] removal of zx_vmo_{read,write}() actual parameter 9/n

Remove actual from vmo::read/write
Use new vmo::read/write throughout Zircon

ZX-1802

Change-Id: Ia1a353e7f2a9f84ccf28b307f1c47147e782ae5b

[syscall][vmo] removal of zx_vmo_{read,write}() actual parameter 6/n

Introduce _old clones in vmo.h, in preparation for rebasing higher
layers on top of them, then modifying read,write.
Rebase Zircon usage on top of _old clones.

ZX-1802

Change-Id: Icd77378d8999ecad68d1bf0e86c9aab218332ea2

[blobfs][minfs] Prevent lookup function from double-acquiring lock

An race condition (most reproducible in blobfs, with the null blob test,
on QEMU + ARM64 without KVM, but possible with any blob) can occur with
the following steps:

1) A Vnode is created, and has data which is being written
back to storage asynchronously.
2) All client-side connections to the Vnode are closed, resulting
in a final reference to the RefPtr from the async writeback
thread.
3) A client-side call to lookup the Vnode is issued. This lookup
call successfully acquires a reference to the Vnode, while holding
a lock on the "Vnode lookup table".
4) The writeback thread releases the reference to the vnode.
5) The lookup call (still holding the lock!) also releases a reference
to the Vnode -- this time, it's the last reference, so the Vnode's
fbl_recycle function is called. This function attempts to lock and
remove the Vnode from "Vnode lookup table", but it is UNSAFE to do
so: the lock is already held from earlier in the lookup call.

This double-lock causes blobfs to hang.

Fix: Avoid releasing references to Vnodes while holding the
|hash_lock_|.

Although this bug can only trigger with blobfs' implementation of
lookup & async writeback (since the minfs lookup implementation
may only release the last reference of unlinked vnodes, which
don't cause a lock to be acquired in the destructor), this
patch also refactors the minfs lookup function to be more resistant
to this class of failure conditions.

ZX-1842 #done

Change-Id: I3b680ed1cc059bdc5c9b47ca1b573d981af7e10a

[fs][minfs][memfs] Add support for renaming to self

ZX-1729 #done

Change-Id: I6f780d6403b5e0bc97c8f3d33659acd1f81d4ea8

[minfs] Reorganize indirect block operations

Create a single set of 'BlockOp's, which will be called by both
BlocksShrink and BlockGet (previously GetBno). BlockOps unify
as much common code as possible between the separate operations.

Change-Id: I1c44699e1e78c9fe8076f54abd375acde3e5069b

[minfs] Avoid using closure after it has gone out of scope

Instead of capturing a closure by reference and returning
from VnodeMinfs::Sync (which ends the SyncCallback closure
lifetime) move it into the closure explicitly.

ZX-1628 #comment In Progress

Change-Id: I646d08e4facead165ca2ab4b1ad72f7f942203ed

[fs] Convert Sync (flush) operation to be asynchronous

Since the "Sync" operation may block for disk to complete
a potentially expensive operation, it is ironically a perfect
candidate for becoming an asynchronous operation.

This patch updates the VFS interface for sync to use closures,
allowing filesystems to defer responding to the requesting
process.

ZX-1308 #done

Change-Id: Ibc00c4963be79ff80504a7dafcffce6d81ccc66f

[minfs] Privatize VnodeMinfs interface

Change-Id: I1f976e1602f5c9215a1bf915013132bcd9698c5b

[minfs] Convert minfs.h into private interface

Change-Id: I3a0fe542871172b591a0f178eb1d855f12742558

[minfs] Artificially cap MinFS max file size at 4GB

MinFS can encounter overflows on disks which attempt
to allocate larger files. Properly storing sizes in
fields larger than 32 bits is the end goal, but in
the meantime, overflows should not occur.

ZX-1523 #comment In Progress

Change-Id: I7c87d9acf9397b569a0d8852486154e1aa5bd05c

[storage] Have ioctl_vfs_query_fs return a unique fs id.

This is a "random" instance id to be used for statfs calls.

Minfs and Blobstore now create a unique instance identifier.

ZX-1319
ZX-1391

Change-Id: Ifb77695bf6559f3e1fa3cd4362d026af1bd13926

[storage] Have ioctl_vfs_query_fs return a file-system identifier.

This is a short id that is meant to be usable for an statfs call.

Minfs and Blobstore now return the low order bytes of the magic fs id,
which mostly starts spelling the FS name.

ZX-1319

Change-Id: I2d243c6d3ea42c8a3b0324f2d58c46e328a65ebb

[minfs] Improve messages from unexpected errors

Change-Id: I109ed92150e69e7a19a982349cc45b53704f90cb

[fs] Use the Fuchsia-defined open flags

Re-lands 2aff2cba029cf0f7e98e6d19f6dc7a663c318227

This change uses the Fuchsia-defined flags on the wire
as well as on filesystem servers. Clients interacting
with filesystems through the POSIX compatibility layer
can continue using POSIX open flags; they will be translated
by the RIO layers client-side.

ZX-1359 #comment In Progress

Change-Id: I4972c024547a0daa2146831e49e34d3ef905d4b4

[fs] Clean usage of fs/trace.h, remove extern 'trace_bits'

- Use "xprintf" (conditionally compiled printfs), allowing
for a debugging printf style which matches the rest of Zircon.
- Remove the trace categories from fs/trace.h (which were
used rarely, and often inconsistently).

Change-Id: Ifd6c91c7dc91bb7d0ba31a42affd5ec13c150761

[fs][minfs] Add basic support for tracing MinFS

Conditionally define "TRACE_DURATION" (and other
macros which will be used by filesystem tracing)
within ulib/fs, to avoid "ifdef __Fuchsia__" bloat.

ulib/fs's tracing will continue to be modified
in future patches.

ZX-1300 #done

Change-Id: I1e25b5f2feb68e58b41887d8958eafa6ef5b1582

[minfs] Refactor MinFS headers to be more fine-grained

Separate the following:
- On-Disk Structures
- Bcache (mechanism to interact with disk)
- MinFS Writeback
- MinFS Vnodes
- MinFS fsck

Into separate headers.

Change-Id: I920fe68d5c38ce53ed8e59ecee5003e6c3336577