fusefs: make the tests more robust to changes to maxphys

Remove assumptions in two test cases that maxphys won't be huge.

Reported by: kib
MFC after: 2 weeks
Sponsored by: Axcient

tests/fusefs: fix all tests that depend on kern.maxphys

The tests try to read kern.maxphys sysctl into int value, while
unsigned long is required. Not sure when this was broken, seems like
since cd8537910406e.

Reviewed by: asomers
Differential Revision: https://reviews.freebsd.org/D45053

fusefs: prefer new/delete over malloc/free

MFC after: 2 weeks
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D43464

Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

fusefs tests: handle -Wdeprecated* issues with GoogleTest 1.14.0

`INSTANTIATE_TEST_CASE_P` has been replaced with `INSTANTIATE_TEST_SUITE_P`.
Replace all uses of the former macro with the latter macro.

While here, address the fact that the latter macro doesn't permit some
of the constructions that the former macro did, e.g., empty parameters,
etc.

MFC after: 2 weeks
Reviewed by: asomers
Differential Revision: https://reviews.freebsd.org/D41398

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

fusefs: fix a buffer overflow in the tests

The actual overflow occured in the ReadAhead.readahead test.
Surprisingly it has never segfaulted or resulted in any bad behavior.

MFC after: 1 week
Sponsored by: Axcient
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D38718

fusefs: truncate write if it would exceed RLIMIT_FSIZE

PR: 164793
MFC after: 2 weeks
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D36703

fusefs: correctly handle servers that report too much data written

During a FUSE_WRITE, the kernel requests the server to write a certain
amount of data, and the server responds with the amount that it actually
did write. It is obviously an error for the server to write more than
it was provided, and we always treated it as such, but there were two
problems:

* If the server responded with a huge amount, greater than INT_MAX, it
would trigger an integer overflow which would cause a panic.

* When extending the file, we wrongly set the file's size before
validing the amount written.

PR: 263263
Reported by: Robert Morris <rtm@lcs.mit.edu>
MFC after: 2 weeks
Sponsored by: Axcient
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D34955

fusefs: Fix a bug during VOP_STRATEGY when the server changes file size

If the FUSE server tells the kernel that a file's size has changed, then
the kernel must invalidate any portion of that file in cache. But the
kernel can't do that during VOP_STRATEGY, because the file's buffers are
already locked. Instead, proceed with the write.

PR: 256937
Reported by: Agata <chogata@moosefs.pro>
Tested by: Agata <chogata@moosefs.pro>
MFC after: 2 weeks
Reviewed by: pfg
Differential Revision: https://reviews.freebsd.org/D32332

fusefs: reenable the WriteCluster.cluster_write_err test

The underlying panic was just fixed by
revision 27006229f7a40a18a61a0e8fd270bc583326b690

PR: 238565
MFC after: 1 week
MFC with: 27006229f7a40a18a61a0e8fd270bc583326b690

fusefs: implement FUSE_COPY_FILE_RANGE.

This updates the FUSE protocol to 7.28, though most of the new features
are optional and are not yet implemented.

MFC after: 2 weeks
Relnotes: yes
Reviewed by: cem
Differential Revision: https://reviews.freebsd.org/D27818

fusefs: fix an expectation in one of the tests

An order-of-operations problem caused an expectation intended for
FUSE_READ to instead match FUSE_ACCESS. Surprisingly, only one test
case was affected.

MFC after: 2 weeks
Reviewed by: cem
Differential Revision: https://reviews.freebsd.org/D27818

fusefs: fix the tests for a wider range of maxphys

maxphys is now a tunable, ever since r368124. The default value is also
larger than it used to be. That broke several fusefs tests that made
assumptions about maxphys.

* WriteCluster.clustering used the MAXPHYS compile-time constant.

* WriteBackAsync.direct_io_partially_overlaps_cached_block implicitly
depended on the default value of maxphys. Fix it by making the
dependency explicit.

* Write.write_large implicitly assumed that maxphys would be no more
than twice maxbcachebuf. Fix it by explicitly setting m_max_write.

* WriteCluster.clustering and several others failed because the MockFS
module did not work for max_write > 128KB (which most tests would set
when maxphys > 256KB). Limit max_write accordingly. This is the same
as fusefs-libs's behavior.

* Bmap's tests were originally written for MAXPHYS=128KB. With larger
values, the simulated file size was too small.

PR: 252096
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D27769

fusefs tests: quell Coverity "Argument cannot be negative" warnings

Must abort tests early if open(2) fails.

Reported by: Coverity
Coverity CID: 1432810 and many others
Reviewed by: kevans
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D26635

fusefs: fix mmap'd writes in direct_io mode

If a FUSE server returns FOPEN_DIRECT_IO in response to FUSE_OPEN, that
instructs the kernel to bypass the page cache for that file. This feature
is also known by libfuse's name: "direct_io".

However, when accessing a file via mmap, there is no possible way to bypass
the cache completely. This change fixes a deadlock that would happen when
an mmap'd write tried to invalidate a portion of the cache, wrongly assuming
that a write couldn't possibly come from cache if direct_io were set.

Arguably, we could instead disable mmap for files with FOPEN_DIRECT_IO set.
But allowing it is less likely to cause user complaints, and is more in
keeping with the spirit of open(2), where O_DIRECT instructs the kernel to
"reduce", not "eliminate" cache effects.

PR: 247276
Reported by: trapexit@spawn.link
Reviewed by: cem
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D26485

fusefs: fix some memory leaks in the tests.

Oddly, most of these were not detected by Coverity.

Reported by: Coverity (one of them, anyway)
Coverity CID: 1404490
MFC after: 2 weeks

fusefs: coverity cleanup in the tests

Address the following defects reported by Coverity:

* Structurally dead code (CID 1404366): set m_quit before FAIL, not after

* Unchecked return value of sysctlbyname (CID 1404321)

* Unchecked return value of stat(2) (CID 1404471)

* Unchecked return value of open(2) (CID 1404402, 1404529)

* Unchecked return value of dup(2) (CID 1404478)

* Buffer overflows. These are all false positives caused by the fact that
Coverity thinks I'm using a buffer to store strings, when in fact I'm
really just using it to store a byte array that happens to be initialized
with a string. I'm changing the type from char to uint8_t in the hopes
that it will placate Coverity. (CID 1404338, 1404350, 1404367, 1404376,
1404379, 1404381, 1404388, 1404403, 1404425, 1404433, 1404434, 1404474,
1404480, 1404484, 1404503, 1404505)

* False positive file descriptor leak. I'm going to try to fix this with
Coverity modeling, but I'll also change an EXPECT to ASSERT so we don't
perform meaningless assertions after the failure. (CID 1404320, 1404324,
1404440, 1404445).

* Unannotated file descriptor leak. This will be followed up by a Coverity
modeling change. (CID 1404326, 1404334, 1404336, 1404357, 1404361,
1404372, 1404391, 1404395, 1404409, 1404430, 1404448, 1404451, 1404455,
1404457, 1404458, 1404460)

* Uninitialized variables in C++ constructors (CID 1404327, 1404346). In the
case of m_maxphys, this actually led to part of the FUSE_INIT's response
being set to stack garbage during the WriteCluster::clustering test.

* Uninitialized sun_len field in struct sockaddr_un (CID 1404330, 1404371,
1404429).

Reported by: Coverity
Reviewed by: emaste
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D21457

fusefs: add SVN Keywords to the test files

Reported by: SVN pre-commit hooks
MFC after: 15 days
MFC-With: r350665
Sponsored by: The FreeBSD Foundation

fusefs: skip some tests when unsafe aio is disabled

MFC after: 16 days
MFC-With: r350665
Sponsored by: The FreeBSD Foundation

fusefs: fix panic when writing with O_DIRECT and using writeback cache

When a fusefs file system is mounted using the writeback cache, the cache
may still be bypassed by opening a file with O_DIRECT. When writing with
O_DIRECT, the cache must be invalidated for the affected portion of the
file. Fix some panics caused by inadvertently invalidating too much.

Sponsored by: The FreeBSD Foundation

fusefs: fix warnings in the tests reported by GCC

Sponsored by: The FreeBSD Foundation

fusefs: annotate deliberate file descriptor leaks in the tests

closing a file descriptor causes FUSE activity that is superfluous to the
purpose of most tests, but would nonetheless require matching expectations.
Rather than do that, most tests deliberately leak file descriptors instead.
This commit moves the leakage from each test into two trivial functions:
leak and leakdir. Hopefully Coverity will only complain about those
functions and not all of their callers.

Sponsored by: The FreeBSD Foundation

fusefs: implement protocol 7.23's FUSE_WRITEBACK_CACHE option

As of protocol 7.23, fuse file systems can specify their cache behavior on a
per-mountpoint basis. If they set FUSE_WRITEBACK_CACHE in
fuse_init_out.flags, then they'll get the writeback cache. If not, then
they'll get the writethrough cache. If they set FOPEN_DIRECT_IO in every
FUSE_OPEN response, then they'll get no cache at all.

The old vfs.fusefs.data_cache_mode sysctl is ignored for servers that use
protocol 7.23 or later. However, it's retained for older servers,
especially for those running in jails that lack access to the new protocol.

This commit also fixes two other minor test bugs:
* WriteCluster:SetUp was using an uninitialized variable.
* Read.direct_io_pread wasn't verifying that the cache was actually
bypassed.

Sponsored by: The FreeBSD Foundation

fusefs: implement the "time_gran" feature.

If a server supports a timestamp granularity other than 1ns, it can tell the
client this as of protocol 7.23. The client will use that granularity when
updating its cached timestamps during write. This way the timestamps won't
appear to change following flush.

Sponsored by: The FreeBSD Foundation

fusefs: set ctime during FUSE_SETATTR following a write

As of r349396 the kernel will internally update the mtime and ctime of files
on write. It will also flush the mtime should a SETATTR happen before the
data cache gets flushed. Now it will flush the ctime too, if the server is
using protocol 7.23 or higher.

This is the only case in which the kernel will explicitly set a file's
ctime, since neither utimensat(2) nor any other user interfaces allow it.

Sponsored by: The FreeBSD Foundation

fusefs: automatically update mtime and ctime on write

Writing should implicitly update a file's mtime and ctime. For fuse, the
server is supposed to do that. But the client needs to do it too, because
the FUSE_WRITE response does not include time attributes, and it's not
desirable to issue a GETATTR after every WRITE. When using the writeback
cache, there's another hitch: the kernel should ignore the mtime and ctime
fields in any GETATTR response for files with a dirty write cache.

Sponsored by: The FreeBSD Foundation

fusefs: fix the tests for non-default values of MAXPHYS

Sponsored by: The FreeBSD Foundation

fusefs: fix the tests for nondefault values of vfs.maxbcachebuf

Sponsored by: The FreeBSD Foundation

fusefs: writes should update the file size, even when data_cache_mode=0

Writes that extend a file should update the file's size. r344185 restricted
that behavior for fusefs to only happen when the data cache was enabled.
That probably made sense at the time because the attribute cache wasn't
fully baked yet. Now that it is, we should always update the cached file
size during write.

Sponsored by: The FreeBSD Foundation

fusefs: rewrite vop_getpages and vop_putpages

Use the standard facilities for getpages and putpages instead of bespoke
implementations that don't work well with the writeback cache. This has
several corollaries:

* Change the way we handle short reads _again_. vfs_bio_getpages doesn't
provide any way to handle unexpected short reads. Plus, I found some more
lock-order problems. So now when the short read is detected we'll just
clear the vnode's attribute cache, forcing the file size to be requeried
the next time it's needed. VOP_GETPAGES doesn't have any way to indicate
a short read to the "caller", so we just bzero the rest of the page
whenever a short read happens.

* Change the way we decide when to set the FUSE_WRITE_CACHE bit. We now set
it for clustered writes even when the writeback cache is not in use.

Sponsored by: The FreeBSD Foundation

fusefs: correctly handle short reads

A fuse server may return a short read for three reasons:

* The file is opened with FOPEN_DIRECT_IO. In this case, the short read
should be returned directly to userland. We already handled this case
correctly.

* The file was truncated server-side, and the read hit EOF. In this case,
the kernel should update the file size. Fixed in the case of VOP_READ.
Fixing this for VOP_GETPAGES is TODO.

* The file is opened in writeback mode, there are dirty buffers past what
the server thinks is the file's EOF, and the read hit what the server
thinks is the file's EOF. In this case, the client is trying to read a
hole, and should zero-fill it. We already handled this case, and I added
a test for it.

Sponsored by: The FreeBSD Foundation

fusefs: multiple fixes related to the write cache

* Don't always write the last page synchronously. That's not actually
required. It was probably just masking another bug that I fixed later,
possibly in r349021.

* Enable the NotifyWriteback tests now that Writeback cache is working.

* Add a test to ensure that the write cache isn't flushed synchronously when
in writeback mode.

Sponsored by: The FreeBSD Foundation

fusefs: skip the Write.mmap test when mmap is not available

fusefs doesn't not allow mmap when data caching is disabled.

Sponsored by: The FreeBSD Foundation

fusefs: fix the "write-through" of write-through cacheing

Our fusefs(5) module supports three cache modes: uncached, write-through,
and write-back. However, the write-through mode (which is the default) has
never actually worked as its name suggests. Rather, it's always been more
like "write-around". It wrote directly, bypassing the cache. The cache
would only be populated by a subsequent read of the same data.

This commit fixes that problem. Now the write-through mode works as one
would expect: write(2) immediately adds data to the cache and then blocks
while the daemon processes the write operation.

A side effect of this change is that non-cache-block-aligned writes will now
incur a read-modify-write cycle of the cache block. The old behavior
(bypassing write cache entirely) can still be achieved by opening a file
with O_DIRECT.

PR: 237588
Sponsored by: The FreeBSD Foundation

fusefs: enable write clustering

Enable write clustering in fusefs whenever cache mode is set to writeback
and the "async" mount option is used. With default values for MAXPHYS,
DFLTPHYS, and the fuse max_write mount parameter, that means sequential
writes will now be written 128KB at a time instead of 64KB.

Also, add a regression test for PR 238565, a panic during unmount that
probably affects UFS, ext2, and msdosfs as well as fusefs.

PR: 238565
Sponsored by: The FreeBSD Foundation

fusefs: WIP fixing writeback cacheing

The current "writeback" cache mode, selected by the
vfs.fusefs.data_cache_mode sysctl, doesn't do writeback cacheing at all. It
merely goes through the motions of using buf(9), but then writes every
buffer synchronously. This commit:

* Enables delayed writes when the sysctl is set to writeback cacheing
* Fixes a cache-coherency problem when extending a file whose last page has
just been written.
* Removes the "sync" mount option, which had been set unconditionally.
* Adjusts some SDT probes
* Adds several new tests that mimic what fsx does but with more control and
without a real file system. As I discover failures with fsx, I add
regression tests to this file.
* Adds a test that ensures we can append to a file without reading any data
from it.

This change is still incomplete. Clustered writing is not yet supported,
and there are frequent "panic: vm_fault_hold: fault on nofault entry" panics
that I need to fix.

Sponsored by: The FreeBSD Foundation

fusefs: respect RLIMIT_FSIZE

Sponsored by: The FreeBSD Foundation

fusefs: set the flags fields of fuse_write_in and fuse_read_in

These fields are supposed to contain the file descriptor flags as supplied
to open(2) or set by fcntl(2). The feature is kindof useless on FreeBSD
since we don't supply all of these flags to fuse (because of the weak
relationship between struct file and struct vnode). But we should at least
set the access mode flags (O_RDONLY, etc).

This is the last fusefs change needed to get full protocol 7.9 support.
There are still a few options we don't support for good reason (mandatory
file locking is dumb, flock support is broken in the protocol until 7.17,
etc), but there's nothing else to do at this protocol level.

Sponsored by: The FreeBSD Foundation

fusefs: set FUSE_WRITE_CACHE when writing from cache

This bit tells the server that we're not sure which uid, gid, and/or pid
originated the write. I don't know of a single file system that cares, but
it's part of the protocol.

Sponsored by: The FreeBSD Foundation

fusefs: make the tests more cplusplusy

* Prefer std::unique_ptr to raw pointers
* Prefer pass-by-reference to pass-by-pointer
* Prefer static_cast to C-style cast, unless it's too much typing

Reported by: ngie
Sponsored by: The FreeBSD Foundation

fusefs: Upgrade FUSE protocol to version 7.9.

This commit upgrades the FUSE API to protocol 7.9 and adds unit tests for
backwards compatibility with servers built for version 7.8. It doesn't
implement any of 7.9's new features yet.

Sponsored by: The FreeBSD Foundation

fusefs: commit missing file from r347547

Sponsored by: The FreeBSD Foundation

fusefs: enable the Write.mmap test

This test had been disabled because it was designed to check protocol
7.9-specific functionality. Enable it without the 7.9-specific bit.

Sponsored by: The FreeBSD Foundation

fusefs: fix cache invalidation error from r346162

An off-by-one error led to the last page of a write not being removed from
its object, even though that page's buffer was marked as invalid.

PR: 235774
Sponsored by: The FreeBSD Foundation

fusefs: evict invalidated cache contents during write-through

fusefs's default cache mode is "writethrough", although it currently works
more like "write-around"; writes bypass the cache completely. Since writes
bypass the cache, they were leaving stale previously-read data in the cache.
This commit invalidates that stale data. It also adds a new global
v_inval_buf_range method, like vtruncbuf but for a range of a file.

PR: 235774
Reported by: cem
Sponsored by: The FreeBSD Foundation

fusefs: cache file attributes

FUSE_LOOKUP, FUSE_GETATTR, FUSE_SETATTR, FUSE_MKDIR, FUSE_LINK,
FUSE_SYMLINK, FUSE_MKNOD, and FUSE_CREATE all return file attributes with a
cache validity period. fusefs will now cache the attributes, if the server
returns a non-zero cache validity period.

This change does _not_ implement finite attr cache timeouts. That will
follow as part of PR 235773.

PR: 235775
Reported by: cem
Sponsored by: The FreeBSD Foundation

fusefs: correctly handle short writes

If a FUSE daemon returns FOPEN_DIRECT_IO when a file is opened, then it's
allowed to write less data than was requested during a FUSE_WRITE operation
on that file handle. fusefs should simply return a short write to userland.

The old code attempted to resend the unsent data. Not only was that
incorrect behavior, but it did it in an ineffective way, by attempting to
"rewind" the uio and uiomove the unsent data again.

This commit correctly handles short writes by returning directly to
userland if FOPEN_DIRECT_IO was set. If it wasn't set (making the short
write technically a protocol violation), then we resend the unsent data.
But instead of rewinding the uio, just resend the data that's already in the
kernel.

That necessitated a few changes to fuse_ipc.c to reduce the amount of bzero
activity. fusefs may be marginally faster as a result.

PR: 236381
Sponsored by: The FreeBSD Foundation

fusefs: send FUSE_FLUSH during VOP_CLOSE

The FUSE protocol says that FUSE_FLUSH should be send every time a file
descriptor is closed. That's not quite possible in FreeBSD because multiple
file descriptors can share a single struct file, and closef doesn't call
fo_close until the last close. However, we can still send FUSE_FLUSH on
every VOP_CLOSE, which is probably good enough.

There are two purposes for FUSE_FLUSH. One is to allow file systems to
return EIO if they have an error when writing data that's cached
server-side. The other is to release POSIX file locks (which fusefs(5) does
not yet support).

PR: 236405, 236327
Sponsored by: The FreeBSD Foundation

fusefs: don't force direct io for files opened O_WRONLY

Previously fusefs would treat any file opened O_WRONLY as though the
FOPEN_DIRECT_IO flag were set, in an attempt to avoid issuing reads as part
of a RMW write operation on a cached part of the file. However, the FUSE
protocol explicitly allows reads of write-only files for precisely that
reason.

Sponsored by: The FreeBSD Foundation

fusefs: adapt the tests to the fuse => fusefs rename

Sponsored by: The FreeBSD Foundation