MFC r345359, r345384:
Don't attempt to measure TSC skew when running as a VM guest.

PR: 218452

MFC r338724:
Fix an nvpair leak in vdev_geom_read_config().

PR: 230704

Revert r335546 as temporary pool name feature has not been merged

MFC r333630: Fix 'zpool create -t <tempname>'

Creating a pool with a temporary name fails when we also specify custom
dataset properties: this is because we mistakenly call
zfs_set_prop_nvlist() on the "real" pool name which, as expected,
cannot be found because the SPA is present in the namespace with the
temporary name.

MFC r333243: opensolaris system_taskq does not need to run at maximum priority

MFC r333234: zfs_ioctl: avoid out-of-bound read

admbugs: 796
Submitted by: Domagoj Stolfa <ds815@cam.ac.uk>
Reported by: Ilja Van Sprundel <ivansprundel@ioactive.com>

MFC r332426: allow ZFS pool to have temporary name for duration of current import

The change adds -t <name> option to zpool create and -t option to zpool
import in its form with an old name and a new name. This allows to
import (or create) a pool under a name that's different from its real,
permanent name without affecting that name. This is useful when working
with VM images or images of other physical systems if they happen to
have a ZFS pool with the same name as the host system.

Sponsored by: Panzura (porting)

MFC r330974: MFV r330973: 9164 assert: newds == os->os_dsl_dataset

PR: 225877

MFC r330057: add ZFS_ENTER protection to .zfs/snapshot vnode operations that need it

MFC r322245,r329717: MFV r322242: 8373 TXG_WAIT in ZIL commit path

MFC r322245: MFV r322242: 8373 TXG_WAIT in ZIL commit path
MFC r329717: MFV r329715: 8997 ztest assertion failure in zil_lwb_write_issue

MFC r329265, r329384

r329265:
Implement .vop_pathconf and .vop_getacl for the .zfs ctldir

zfsctl_common_pathconf will report all the same variables that regular ZFS
volumes report. zfsctl_common_getacl will report an ACL equivalent to 555,
except that you can't read xattrs or edit attributes.

Fixes a bug where "ls .zfs" will occasionally print something like:
ls: .zfs/.: Operation not supported

PR: 225793
Reviewed by: avg
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D14365

r329384:
Handle generic pathconf attributes in the .zfs ctldir

MFC instructions: change the value of _PC_LINK_MAX to INT_MAX

Reported by: jhb
X-MFC-With: 329265
Sponsored by: Spectra Logic Corp

MFC r329714: MFV r329713: 8731 ASSERT3U(nui64s, <=, UINT16_MAX) fails for large blocks

MFC r324940:

Fix the error message when creating a zpool on a too-small device

Don't check for SPA_MINDEVSIZE in vdev_geom_attach when opening by path.
It's redundant with the check in vdev_open, and failing to attach here
results in the wrong error message being printed. However, still check for
it in some other situations:

* When opening by guids, so we don't get bogged down reading from slow
devices like floppy drives.
* In vdev_geom_read_pool_label for the same reason, because we iterate over
all providers.
* If the caller requests that we verify the guid, because then we'll have to
read from the device before vdev_open verifies the size.

PR: 222227
Reported by: Marie Helene Kvello-Aune <marieheleneka@gmail.com>
Reviewed by: avg, mav
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D12531

MFC r326401:

Fix assertion when ZFS fails to open certain devices

"panic: vdev_geom_close_locked: cp->private is NULL"
This panic will result if ZFS fails to open a device due to either of the
following reasons:

1) The device's sector size is greater than 8KB.
2) ZFS wants to open the device RW, but it can't be opened for writing.

The solution is to change the initialization order to ensure that the
assertion will be satisfied.

PR: 221066
Reported by: David NewHamlet <wheelcomplex@gmail.com>
Reviewed by: avg
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D13278

MFC r329314: MFV r329313: 8857 zio_remove_child() panic due to already destroyed parent zio

PR: 223803

MFC r329711: MFV r329710: 8966 use after end of the lifetime of a local variable

PR: 225162

MFC r329556,r329820 remove an assert in zfsctl_snapdir_lookup to match r323578

MFC r329016: remove a duplicate assignment

MFC r328881: zfs: move a utility function, ioflags, closer to its consumers

MFC r328217: zfs: no need to check that size of zfs_cmd_t is not greater than IOCPARM_MAX

MFC r327725: zfs_mount: restore a bit of ifdef-out illumos code

MFC r327167:

Remove obsolete register keyword from opensolaris's sysmacros.h. When
compiling zfsd with recent clang, it leads to a warning about the
register storage class being incompatible with C++17.

MFC r326070: zfs_write: fix problem with writes appearing to succeed when over quota

The problem happens when the writes have offsets and sizes aligned with
a filesystem's recordsize (maximum block size). In this scenario
dmu_tx_assign() would fail because of being over the quota, but the uio
would already be modified in the code path where we copy data from the
uio into a borrowed ARC buffer. That makes an appearance of a partial
write, so zfs_write() would return success and the uio would be modified
consistently with writing a single block.

That bug can result in a data loss because the writes over the quota
would appear to succeed while the actual data is being discarded.

This commit fixes the bug by ensuring that the uio is not changed until
after all error checks are done. To achieve that the code now uses
uiocopy() + uioskip() as in the original illumos design. We can do that
now that uiocopy() has been updated in r326067 to use
vn_io_fault_uiomove().

MFC r326067: make illumos uiocopy use vn_io_fault_uiomove

MFC r323813:

MFV r323789: 8473 scrub does not detect errors on active spares

illumos/illumos-gate@554675eee75dd2d7398d960aa5c81083ceb8505a
https://github.com/illumos/illumos-gate/commit/554675eee75dd2d7398d960aa5c81083ceb8505a

https://www.illumos.org/issues/8473
Scrubbing is supposed to detect and repair all errors in the pool. However,
it wrongly ignores active spare devices. The problem can easily be
reproduced in OpenZFS at git rev 0ef125d with these commands:

truncate -s 64m /tmp/a /tmp/b /tmp/c
sudo zpool create testpool mirror /tmp/a /tmp/b spare /tmp/c
sudo zpool replace testpool /tmp/a /tmp/c
/bin/dd if=/dev/zero bs=1024k count=63 oseek=1 conv=notrunc of=/tmp/c
sync
sudo zpool scrub testpool
zpool status testpool # Will show 0 errors, which is wrong
sudo zpool offline testpool /tmp/a
sudo zpool scrub testpool
zpool status testpool # Will show errors on /tmp/c,
# which should've already been fixed

FreeBSD head is partially affected: the first scrub will detect some errors, but the second scrub will detect more.

Reviewed by: Andy Stormont <astormont@racktopsystems.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Richard Lowe <richlowe@richlowe.net>

Sponsored by: Spectra Logic Corp

MFC r323194:

Fix remounting ZFS filesystem with "zfs mount"

"zfs mount -o" passes a list of mount options directly to nmount(2) after
sanity checking them. In particular, zfs(8) will refuse to mount an already
existing file system unless "remount" is specified in the option list.
However, the "remount" option only exists in Illumos. FreeBSD's equivalent is
"update".

PR: 221985
Reviewed by: avg
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D12233

MFC r322546:

Fix some ZFS debugging messages

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c
Be more careful about the use of provider names vs vdev names in
ZFS_LOG statements.

Sponsored by: Spectra Logic Corp

MFC r325610: MFV r325609: 7531 Assign correct flags to prefetched buffers

MFC r325228: vdev_geom_close: close errored consumer even if vdev_reopening is set

MFC r325606: MFV r325605: 8713 Buffer overflow in dsl_dataset_name()

MFC r324757: remove spa_sync_on assert from spa_async_thread_vd

MFC r324309: remove heuristic error detection from ddi_strto*()

MFC r323578,r323769: dounmount: do not release the mount point's reference
on the covered vnode

As long as mnt_ref is not zero there can be a consumer that might try
to access mnt_vnodecovered. For this reason the covered vnode must not
be freed until mnt_ref goes to zero.
So, move the release of the covered vnode to vfs_mount_destroy.

MFC r323483: zfsctl_snapdir_lookup should be able to handle an uncovered vnode

MFC r323481: zfsvfs_hold: assert that the busied filesystem can not be unmounted

MFC r323918: MFV r323917: 8648 Fix range locking in ZIL commit codepath

This fixes a problem introduced in r320496, MFC of r308782.

MFC r323522: slightly simplify zfs_vptocnp

MFC r323480: zfs_get_vfs: reference a requested filesystem instead of vfs_busy-ing it

Sponsored by: Panzura

MFC r323479,r323491: zfs: tighten debug versions of ZTOV and VTOZ

Sponsored by: Panzura

MFC r323482: zfs_ctldir: remove obsolete / bogus ARGSUSED lint directives

None of the tagged functions had unused parameters.

MFC r322241: MFV r322240: 8491 uberblock on-disk padding to reserve space for smoothly merging zpool checkpoint & MMP in ZFS

illumos/illumos-gate@79c2b812ee2010ebf20fdd92dc5f06b59000a94c
https://github.com/illumos/illumos-gate/commit/79c2b812ee2010ebf20fdd92dc5f06b59000a94c

https://www.illumos.org/issues/8491
The zpool checkpoint feature in DxOS added a new field in the uberblock.
The Multi-Modifier Protection Pull Request from ZoL adds two new fields in the
uberblock (Reference: https://github.com/zfsonlinux/zfs/pull/6279).
As these two changes come from two different sources and once upstreamed and
deployed will introduce an incompatibility with each other we want
to upstream a change that will reserve the padding for both of them so
integration goes smoothly and everyone gets both features.

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Olaf Faaland <faaland1@llnl.gov>
Approved by: Gordon Ross <gwr@nexenta.com>
Author: Serapheim Dimitropoulos <serapheim@delphix.com>

MFC r322228: MFV r322227: 8377 Panic in bookmark deletion

illumos/illumos-gate@42418f9e73f0d007aa87675ecc206c26fc8e073e
https://github.com/illumos/illumos-gate/commit/42418f9e73f0d007aa87675ecc206c26fc8e073e

https://www.illumos.org/issues/8377
The problem is that when dsl_bookmark_destroy_check() is executed from open
context (the pre-check), it fills in dbda_success based on the existence of the
bookmark.
But the bookmark (or containing filesystem as in this case) can be destroyed
before we get to syncing context. When we re-run dsl_bookmark_destroy_check()
in syncing
context, it will not add the deleted bookmark to dbda_success, intending for
dsl_bookmark_destroy_sync() to not process it. But because the bookmark is
still in dbda_success
from the open-context call, we do try to destroy it.
The fix is that dsl_bookmark_destroy_check() should not modify dbda_success
when called from open context.

Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Matthew Ahrens <mahrens@delphix.com>

MFC r321471: spa_import_rootpool should be able to handle an imported root pool

That is required to support reboot -r with a new root filesystem being
on an already imported pool.

PR: 210721

MFC r320352: zfs: port vdev_file part of illumos change 3306

3306 zdb should be able to issue reads in parallel
illumos/illumos-gate/31d7e8fa33fae995f558673adb22641b5aa8b6e1
https://www.illumos.org/issues/3306

The upstream change was made before we started to import upstream commits
individually. It was imported into the illumos vendor area as r242733.
That commit was MFV-ed in r260138, but as the commit message says
vdev_file.c was left intact.

This commit actually implements the parallel I/O for vdev_file using a
taskqueue with multiple thread. This implementation does not depend on
the illumos or FreeBSD bio interface at all, but uses zio_t to pass
around all the relevent data. So, the code looks a bit different from
the upstream.

This commit also incorporates ZoL commit
zfsonlinux/zfs/bc25c9325b0e5ced897b9820dad239539d561ec9 that fixed
https://github.com/zfsonlinux/zfs/issues/2270
We need to use a dedicated taskqueue for exactly the same reason as ZoL
as we do not implement TASKQ_DYNAMIC.

MFC r323002: zfs: do not advertise unsupported hash algorithms

illumos 4185 ("add new cryptographic checksums to ZFS: SHA-512, Skein,
Edon-R") was intentionally merged only partially in r289422, without
adding support for skein, sha512 and edonr on FreeBSD.

Support for skein and sha512 was added later on (in head), but none of
these are supported in stable/10. Prior to this commit zfs(8) correctly
rejected these algorithms, but with an error message that claimed
support:

fk@r500 ~ $zfs set checksum=edonr tank
cannot set property for 'tank': 'checksum' must be one of 'on | off |
fletcher2 | fletcher4 | sha256 | sha512 | skein | edonr'

(This commit removes sha512 and skein in addition to edonr from the
merge of head's r323002.)

PR: 204055
Submitted by: Fabian Keil
Approved by: re (kib)
Obtained from: ElectroBSD

MFC r321218: zfs: Fix a typo in the delay_min_dirty_percent sysctl description

The description is FreeBSD-specific and was added in r266497
to fix PR189865.

PR: 220825
Submitted by: Fabian Keil
Obtained from: ElectroBSD

MFC r308782:
After some ZIL changes 6 years ago zil_slog_limit got partially broken
due to zl_itx_list_sz not updated when async itx'es upgraded to sync.
Actually because of other changes about that time zl_itx_list_sz is not
really required to implement the functionality, so this patch removes
some unneeded broken code and variables.

Original idea of zil_slog_limit was to reduce chance of SLOG abuse by
single heavy logger, that increased latency for other (more latency critical)
loggers, by pushing heavy log out into the main pool instead of SLOG. Beside
huge latency increase for heavy writers, this implementation caused double
write of all data, since the log records were explicitly prepared for SLOG.
Since we now have I/O scheduler, I've found it can be much more efficient
to reduce priority of heavy logger SLOG writes from ZIO_PRIORITY_SYNC_WRITE
to ZIO_PRIORITY_ASYNC_WRITE, while still leave them on SLOG.

Existing ZIL implementation had problem with space efficiency when it
has to write large chunks of data into log blocks of limited size. In some
cases efficiency stopped to almost as low as 50%. In case of ZIL stored on
spinning rust, that also reduced log write speed in half, since head had to
uselessly fly over allocated but not written areas. This change improves
the situation by offloading problematic operations from z*_log_write() to
zil_lwb_commit(), which knows real situation of log blocks allocation and
can split large requests into pieces much more efficiently. Also as side
effect it removes one of two data copy operations done by ZIL code WR_COPIED
case.

While there, untangle and unify code of z*_log_write() functions.
Also zfs_log_write() alike to zvol_log_write() can now handle writes crossing
block boundary, that may also improve efficiency if ZPL is made to do that.

MFC r318943 (avg):

MFV r318942: 8166 zpool scrub thinks it repaired offline device

https://www.illumos.org/issues/8166
If we do a scrub while a leaf device is offline (via "zpool offline"),
we will inadvertently clear the DTL (dirty time log) of the offline
device, even though it is still damaged. When the device comes back
online, we will incompletely resilver it, thinking that the scrub
repaired blocks written before the scrub was started. The incomplete
resilver can lead to data loss if there is a subsequent failure of a
different leaf device.
The fix is to never clear the DTL of offline devices. Note that if a
device is onlined while a scrub is in progress, the scrub will be
restarted.
The problem can be worked around by running "zpool scrub" after
"zpool online".
See also https://github.com/zfsonlinux/zfs/issues/5806

PR: 219537
Sponsored by: The FreeBSD Foundation

MFC r318832: MFV r316923: 8026 retire zfs_throttle_delay and zfs_throttle_resolution

MFC r318830: MFV r316921: 8027 tighten up dsl_pool_dirty_delta

MFC r319096: zfs_lookup: fix bogus arguments to lookup of "snapshot" directory

MFC r318189:

vdev_geom may associate multiple vdevs per g_consumer

vdev_geom.c currently uses the g_consumer's private field to point to a
vdev_t. That way, a GEOM event can cause a change to a ZFS vdev. For
example, when you remove a disk, the vdev's status will change to REMOVED.
However, vdev_geom will sometimes attach multiple vdevs to the same GEOM
consumer. If this happens, then geom events will only be propagated to one
of the vdevs.

Fix this by storing a linked list of vdevs in g_consumer's private field.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c

* g_consumer.private now stores a linked list of vdev pointers associated
with the consumer instead of just a single vdev pointer.

* Change vdev_geom_set_physpath's signature to more closely match
vdev_geom_set_rotation_rate

* Don't bother calling g_access in vdev_geom_set_physpath. It's guaranteed
that we've already accessed the consumer by the time we get here.

* Don't call vdev_geom_set_physpath in vdev_geom_attach. Instead, call it
in vdev_geom_open, after we know that the open has succeeded.

PR: 218634
Reviewed by: gibbs
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D10391

MFC r316760:

Fix vdev_geom_attach_by_guids for partitioned disks

When opening a vdev whose path is unknown, vdev_geom must find a geom
provider with a label whose guids match the desired vdev. However, due to
partitioning, it is possible that two non-synonomous providers will share
some labels. For example, if the first partition starts at the beginning of
the drive, then ada0 and ada0p1 will share the first label. More troubling,
if the last partition runs to the end of the drive, then ada0p3 and ada0
will share the last label. If vdev_geom opens ada0 when it should've opened
ada0p3, then the pool won't be readable. If it opens ada0 when it should've
opened ada0p1, then it will corrupt some other partition when it writes the
3rd and 4th labels.

The easiest way to reproduce this problem is to install a mirrored root pool
with the default partition layout, then swap the positions of the two boot
drives and reboot. Whether the bug manifests depends on the order in which
geom lists its providers, which is arbitrary.

Fix this situation by modifying the search algorithm to prefer geom
providers that have all four labels intact. If no such provider exists, then
open whichever provider has the most.

Reviewed by: mav
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D10365

MFC r316854: rename vfs.zfs.debug_flags to vfs.zfs.debugflags

Since this is a stable branch vfs.zfs.debug_flags sysctl is also kept.
The corresponing tunable could never work.

MFC r315852: zfs: add zio_buf_alloc_nowait and use it in vdev_queue_aggregate

MFC r315853: zfs_putpages: use TXG_WAIT

MFC r315076: zfs: provide a special vptocnp method for the .zfs vnode

MFC r314048,r314194: reimplement zfsctl (.zfs) support

MFC r314913: MFV r314911: 7867 ARC space accounting leak

MFC r314912: MFV r314910: 7843 get_clones_stat() is suboptimal for lots of clones

MFC r314549: Execute last ZIO of log commit synchronously.

For short transactions overhead of context switch can be too large.
Skipping it gives significant latency reduction. For large ones,
including multiple ZIOs, latency is less critical, while throughput
there may become limited by checksumming speed of single CPU core.
To get best of both cases, execute last ZIO directly from calling
thread context to save latency, while all others (if there are any)
enqueue to taskqueues in traditional way.

MFC r314548: Completely skip cache flushing for not supporting log devices.

MFC r314274: l2arc: fix write size calculation broken by Compressed ARC commit

MFC r313841, r313850:
Prevent CPU migration when checking the DTrace nofault flag on x86.

MFC 313879

MVF: 313876

7504 kmem_reap hangs spa_sync and administrative tasks

illumos/illumos-gate@405a5a0f5c3ab36cb76559467d1a62ba648bd809
https://github.com/illumos/illumos-gate/commit/405a5a0f5c3ab36cb76559467d1a62ba648bd80

https://www.illumos.org/issues/7504

We see long spa_sync(). We are waiting to hold dp_config_rwlock for writer. Some
other thread holds dp_config_rwlock for reader, then calls arc_get_data_buf(),
which finds that arc_is_overflowing()==B_TRUE. So it waits (while holding
dp_config_rwlock for reader) for arc_reclaim_thread to signal arc_reclaim_waiters_cv.
Before signaling, arc_reclaim_thread does arc_kmem_reap_now(), which takes ~seconds.

Author: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Approved by: Dan McDonald <danmcd@omniti.com>

MFC r314058: zfs: lower priority of zio_write_issue threads by four

Obtained from: Panzura
Sponsored by: Panzura

MFC r314572:

Fix null pointer dereference in zfs_freebsd_setacl().

Prevents unprivileged users from panicking the kernel by calling
__acl_delete_*() on files or directories inside a ZFS mount.

MFC r314273: zfs: call spa_deadman on a taskqueue thread

MFC r283291: don't use CALLOUT_MPSAFE with callout_init()

The main purpose of this MFC is to reduce conflicts for other merges.
Parts of the original change have already "trickled down" via individual MFCs.

MFC r314059: zfs: move zio_taskq_basedc under SYSDC

MFC r292782: Replace sys/crypto/sha2/sha2.c with lib/libmd/sha512c.c

cperciva's libmd implementation is 5-30% faster
The same was done for SHA256 previously in r263218

Approved by: secteam

MFC r313687: remove l2_padding_needed statistic from zfs arc

MFC r313686: check remaining space in zfs implementations of vptocnp

PR: 216939

MFC r258903,r264487,r271699,r288415:

r258903 (by markj):

Enable some previously-disabled DTrace tests for umod, ufunc and usym. They
expect the installed ksh binary to be named "ksh", which is not the case
when it's installed on FreeBSD via the shells/ksh93 port. Allow for it to be
"ksh93" as well so that the tests can actually pass.

r264487 (by markj):

Replace a few Solarisisms with their corresponding FreeBSDisms to make a few
printf tests pass.

r271699 (by markj):

Implement a workaround to allow this test program to be compiled with clang.
It seems that if a pragma is used to define a weak alias for a local
function, the pragma must appear after the function is defined.

PR: 193056

r288415 (by markj):

MFV r288408:
6266 harden dtrace_difo_chunksize() with respect to malicious DIF

illumos/illumos-gate@395c7a3dcfc66b8b671dc4b3c4a2f0ca26449922

Author: Bryan Cantrill <bryan@joyent.com>

MFC r310647:
Remove an obsolete pragma from dtrace.h.

MFC r309250: MFV r309249: 3821 Race in rollback, zil close, and zil flush

MFC r309099: MFV r308990: 7181 race between zfs_mount and zfs_ioc_rollback

MFC r309098: MFV r308988: 7199, 7200 dsl_dataset_rollback_sync may try
to free already free blocks

MFC r309097: MFV r308987: 7180 potential race between
zfs_suspend_fs+zfs_resume_fs and zfs_ioc_rename

MFC 309714: Fix spa_alloc_tree sorting by offset in r305331.

Original commit "7090 zfs should improve allocation order" declares alloc
queue sorted by time and offset. But in practice io_offset is always zero,
so sorting happened only by time, while order of writes with equal time was
completely random. On Illumos this did not affected much thanks to using
high resolution timestamps. On FreeBSD due to using much faster but low
resolution timestamps it caused bad data placement on disks, affecting
further read performance.

This change switches zio_timestamp_compare() from comparing uninitialized
io_offset to really populated io_bookmark values. I haven't decided yet
what to do with timestampts, but on simple tests this change gives the
same peformance results by just making code to work as declared.

MFC r308887,309090: fix unsafe modification of zfs_vnodeops when
DIAGNOSTIC is enabled

MFC r308089: zfsbootcfg: a simple tool to set next boot (one time)
options for zfsboot

There is a branch-specific change in sbin/zfsbootcfg/Makefile because of
LIBADD vs LDADD/DPADD.

Revert r308753: some unrelated changes were included into the commit

MFC r308040,308479: nap time between pats is forced to be at most half
of the timeout

Note that in this branch the default nap period is 1 second unlike the
head where the period is 10 seconds.

MFC r308173:
Fix ZIL records ordering when ZVOL opened both with and without FSYNC.

Before this an earlier writes to a ZVOL opened without FSYNC could get to
ZIL after later writes to the same ZVOL opened with FSYNC. Fix this by
replicating functionality of ZPL (zv_sync_cnt equivalent to z_sync_cnt),
marking all log records sync if anybody opened the ZVOL with FSYNC.

MFC r308169:
Pass to zvol_log_truncate() same sync values as to zvol_log_write().

Surplus marking of TX_TRUNCATE records as sync could result in putting them
into ZIL before previous writes if ones were async.

MFC r308055: Add vdev_reopening support to vdev_geom.

It allows to avoid extra GEOM providers flapping without significant need.
Since GEOM got resize support, we don't need to reopen provider to get new
size. If provider was orphaned and no longer valid, ZFS should already
know that, and in such case reopen should be done in full as expected.

MFC r308051: Matching GUIDs, handle possible race on vdev detach.

In case of vdev detach, causing top level mirror vdev destruction, leaf
vdev changes its GUID to one of the destroyed mirror, that creates race
condition when GUID in vdev label may not match one in the pool config.

This change replicates logic nuance of vdev_validate() by adding special
exception, matching the vdev GUID against the top level vdev GUID.
Since this exception is not completely reliable (may give false positives
if we fail to erase label on detached vdev), use it only as last resort.

Quick way to reproduce this scenario now is detach vdev from a pool with
enabled autoextend. During vdev detach autoextend logic tries to reopen
remaining vdev, that always fails now since in-memory configuration is
already updated, while on-disk labels are not yet.

MFC r308049: Improve few debugging log messages.

MFC r307318: MFV r307314:
6988 spa_sync() spends half its time in dmu_objset_do_userquota_updates

Using a benchmark which creates 2 million files in one TXG, I observe
that the thread running spa_sync() is on CPU almost the entire time we
are syncing, and therefore can be a performance bottleneck. About 50% of
the time in spa_sync() is in dmu_objset_do_userquota_updates().

The problem is that dmu_objset_do_userquota_updates() calls
zap_increment_int(DMU_USERUSED_OBJECT) once for every file that was
modified (or created). In this benchmark, all the files are owned by the
same user/group, so all 2 million calls to zap_increment_int() are
modifying the same entry in the zap. The same issue exists for the
DMU_GROUPUSED_OBJECT.

We should keep an in-memory map from user to space delta while we are
syncing, and when we finish, iterate over the in-memory map and modify
the ZAP once per entry. This reduces the number of calls to
zap_increment_int() from "number of objects modified" to "number of
owners/groups of modified files".

This reduced the time spent in spa_sync() in the file create benchmark
by ~33%, from 11 seconds to 7 seconds.

Closes #107

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Steve Gonczi <steve.gonczi@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Jinshan Xiong <jinshan.xiong@intel.com>
Author: Matthew Ahrens <mahrens@delphix.com>

openzfs/openzfs@5fc46359c569369d87728ca09f8705cdff6cc8e2

MFC r307857: Fix panic after ZVOL renamed to name invalid for DEVFS.

MFC r307994: 3746 ZRLs are racy

PR: 204037

MFC r306456: Add #ifdef _KERNEL around send_holes_without_birth_time sysctl.

MFC r306425: MFV r306423:
7402 Create tunable to ignore hole_birth feature

Until we can resolve the numerous hole_birth bugs that have cropped up
recently, and come up with a way going forwards to protect users from
corruption, we should disable the hole_birth feature. Using a tunable
allows those who are confident that their data is correct to continue to
take advantage of the feature.

Closes #188

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Author: Paul Dagnelie <pcd@delphix.com>

MFC r306424: MFV r306422:
7254 ztest failed assertion in ztest_dataset_dirobj_verify: dirobjs + 1 == usedo
bjs

dsl_dataset_space is looking at the ds_bp's fill count while
dmu_objset_write_ready() is concurrently modifying it. This fix adds an
rrwlock to protect the ds_bp.

Closes #180

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Steve Gonczi <steve.gonczi@delphix.com>
Author: Paul Dagnelie <pcd@delphix.com>

MFC r300881, r302058 (by asomers):
Avoid issuing spa config updates for physical path when not necessary

ZFS's configuration needs to be updated whenever the physical path for a
device changes, but not when a new device is introduced. This is because new
devices necessarily cause config updates, but only if they are actually
accepted into the pool.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c
Split vdev_geom_set_physpath out of vdev_geom_attrchanged. When
setting the vdev's physical path, only request a config update if
the physical path has changed. Don't request it when opening a
device for the first time, because the config sync will happen
anyway upstack.

sys/geom/geom_dev.c
Split g_dev_set_physpath and g_dev_set_media out of
g_dev_attrchanged

MFC r300059 (by asomers): Speed up vdev_geom_open_by_guids

Speedup is hard to measure because the only time vdev_geom_open_by_guids
gets called on many drives at the same time is during boot. But with
vdev_geom_open hacked to always call vdev_geom_open_by_guids, operations
like "zpool create" speed up by 65%.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c

* Read all of a vdev's labels in parallel instead of sequentially.
* In vdev_geom_read_config, don't read the entire label, including
the uberblock. That's a waste of RAM. Just read the vdev config
nvlist. Reduces the IO and RAM involved with tasting from 1MB to
448KB.

MFC r298814 (by asomers): Fix a use-after-free when "zpool import" fails

clear vd->vdev_tsd in vdev_geom_close_locked instead of vdev_geom_detach.
In the latter function, it would fail to happen in certain circumstances
where cp->private was unset. Ideally, the latter should never happen, but
it can happen when vdev open fails, or where spares are involved.

MFC r298786 (by asomers):
Refactor vdev_geom_attach and friends to reduce code duplication

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c
Move checks for provider's sectorsize and mediasize into a single
location in vdev_geom_attach. Remove the zfs::vdev::taste class;
it's ok to use the regular vdev class for tasting. Consolidate guid
checks into a single location in vdev_attach_ok. Consolidate some
error handling code from vdev_geom_attach into vdev_geom_detach,
closing a resource leak of geom consumers in the process.

MFC r294329 (by asomers): Disallow zvol-backed ZFS pools

Using zvols as backing devices for ZFS pools is fraught with panics and
deadlocks. For example, attempting to online a missing device in the
presence of a zvol can cause a panic when vdev_geom tastes the zvol. Better
to completely disable vdev_geom from ever opening a zvol. The solution
relies on setting a thread-local variable during vdev_geom_open, and
returning EOPNOTSUPP during zvol_open if that thread-local variable is set.

Remove the check for MUTEX_HELD(&zfsdev_state_lock) in zvol_open. Its intent
was to prevent a recursive mutex acquisition panic. However, the new check
for the thread-local variable also fixes that problem.

Also, fix a panic in vdev_geom_taste_orphan. For an unknown reason, this
function was set to panic. But it can occur that a device disappears during
tasting, and it causes no problems to ignore this departure.

MFC r306801: implement zfs_vptocnp() using z_parent property

MFC r307218:
Fix a race in vm_page_busy_sleep(9).

MFC r305563: MFV r305562: 7259 DS_FIELD_LARGE_BLOCKS is unused

The DS_FIELD_LARGE_BLOCKS macro has been unused since the integration of
this patch:

commit ca0cc3918a1789fa839194af2a9245f801a06b1a
Author: Matthew Ahrens <mahrens@delphix.com>
Date: Fri Jul 24 09:53:55 2015 -0700

5959 clean up per-dataset feature count code
Reviewed by: Toomas Soome <tsoome@me.com>
Reviewed by: George Wilson <george@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Approved by: Richard Lowe <richlowe@richlowe.net>

This patch simply removes this macro from dsl_dataset.h.

Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Reviewed by: Igor Kozhukhov <ikozhukhov@gmail.com>
Author: Matthew Ahrens <mahrens@delphix.com>

MFC r305561: MFV r305560:
7278 tuning zfs_arc_max does not impact arc_c_min

When changing zfs_arc_max (e.g. as zdb does), it may be set to less
than the default arc_c_min. arc_c_min should decrease to not be more than
arc_c_max, but it doesn't; therefore tuning of arc_c_max is ineffective.

Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: Paul Dagnelie <paul.dagnelie@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Igor Kozhukhov <ikozhukhov@gmail.com>
Author: Matthew Ahrens <mahrens@delphix.com>

openzfs/openzfs@608764beadaf4bb71c5d8fe1818e8392ac66a61b

MFC r305456 (by avg): fix zfs pool creation accidentally broken by r305331

The upstream change introduced a new load state, SPA_LOAD_CREATE,
and vdev_geom code needs to be aware of it.

MFC r305342: Missed FreeBSD-specific piece of r305338.

MFC r305340: MFC r305337:
7004 dmu_tx_hold_zap() does dnode_hold() 7x on same object

Using a benchmark which has 32 threads creating 2 million files in the
same directory, on a machine with 16 CPU cores, I observed poor
performance. I noticed that dmu_tx_hold_zap() was using about 30% of
all CPU, and doing dnode_hold() 7 times on the same object (the ZAP
object that is being held).

dmu_tx_hold_zap() keeps a hold on the dnode_t the entire time it is
running, in dmu_tx_hold_t:txh_dnode, so it would be nice to use the
dnode_t that we already have in hand, rather than repeatedly calling
dnode_hold(). To do this, we need to pass the dnode_t down through
all the intermediate calls that dmu_tx_hold_zap() makes, making these
routines take the dnode_t* rather than an objset_t* and a uint64_t
object number. In particular, the following routines will need to have
analogous *_by_dnode() variants created:

dmu_buf_hold_noread()
dmu_buf_hold()
zap_lookup()
zap_lookup_norm()
zap_count_write()
zap_lockdir()
zap_count_write()

This can improve performance on the benchmark described above by 100%,
from 30,000 file creations per second to 60,000. (This improvement is on
top of that provided by working around the object allocation issue. Peak
performance of ~90,000 creations per second was observed with 8 CPUs;
adding CPUs past that decreased performance due to lock contention.) The
CPU used by dmu_tx_hold_zap() was reduced by 88%, from 340 CPU-seconds
to 40 CPU-seconds.

Sponsored by: Intel Corp.

Closes #109

Reviewed by: Steve Gonczi <steve.gonczi@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Author: Matthew Ahrens <mahrens@delphix.com>

openzfs/openzfs@d3e523d489a169ab36f9ec1b2a111a60a5563a9f

MFC r305339: MFV r305336: 7247 zfs receive of deduplicated stream fails

This resolves two 'zfs recv' issues. First, when receiving into an
existing filesystem, a snapshot created during the receive process is
not added to the guid->dataset map for the stream, resulting in failed
lookups for deduped streams when a WRITE_BYREF record refers to a
snapshot received earlier in the stream. Second, the newly created
snapshot was also not set properly, referencing the snapshot before the
new receiving dataset rather than the existing filesystem.

Closes #159

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Author: Chris Williamson <chris.williamson@delphix.com>

openzfs/openzfs@b09697c8c18be68abfe538de9809938239402ae8

MFC r305338: MFV r305335: 7003 zap_lockdir() should tag hold

zap_lockdir() / zap_unlockdir() should take a "void *tag" argument which
tags the hold on the zap. This will help diagnose programming errors
which misuse the hold on the ZAP.

Sponsored by: Intel Corp.

Closes #108

Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: Steve Gonczi <steve.gonczi@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Author: Matthew Ahrens <mahrens@delphix.com>

openzfs/openzfs@0780b3eab5a2c13e04328b39ecd2a6d0d3c4f7cb

MFC r305334: MFV r304157:
7230 add assertions to dmu_send_impl() to verify that stream includes BEGIN and
END records

illumos/illumos-gate@12b90ee2d3b10689fc45f4930d2392f5fe1d9cfa
https://github.com/illumos/illumos-gate/commit/12b90ee2d3b10689fc45f4930d2392f5f
e1d9cfa

https://www.illumos.org/issues/7230
A test failure occurred where a send stream had only a BEGIN record. This
should not be possible if the send returns without error. Prevented this from
happening in the future by adding an assertion to dmu_send_impl() to verify
that if the function returns 0 (success) both a BEGIN and END record are
present. Did this by adding flags to dmu_sendarg_t (indicating whether BEGIN o
r
END records sent), having dump_record() set flags appropriately, adding VERIFY
statement to dmu_send_impl().

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Igor Kozhukhov <ikozhukhov@gmail.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Matt Krantz <matt.krantz@delphix.com>

MFC r305333: MFV r304156: 7235 remove unused func dsl_dataset_set_blkptr

illumos/illumos-gate@bd56f80007857b960e0981ed0797ad8ec844a96b
https://github.com/illumos/illumos-gate/commit/bd56f80007857b960e0981ed0797ad8ec
844a96b

https://www.illumos.org/issues/7235
The function dsl_dataset_set_blkptr() is unused. We should remove it.

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Igor Kozhukhov <ikozhukhov@gmail.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Matthew Ahrens <mahrens@delphix.com>

MFC r305331: MFV r304155:
7090 zfs should improve allocation order and throttle allocations

illumos/illumos-gate@0f7643c7376dd69a08acbfc9d1d7d548b10c846a
https://github.com/illumos/illumos-gate/commit/0f7643c7376dd69a08acbfc9d1d7d548b
10c846a

https://www.illumos.org/issues/7090
When write I/Os are issued, they are issued in block order but the ZIO pipelin
e
will drive them asynchronously through the allocation stage which can result i
n
blocks being allocated out-of-order. It would be nice to preserve as much of
the logical order as possible.
In addition, the allocations are equally scattered across all top-level VDEVs
but not all top-level VDEVs are created equally. The pipeline should be able t
o
detect devices that are more capable of handling allocations and should
allocate more blocks to those devices. This allows for dynamic allocation
distribution when devices are imbalanced as fuller devices will tend to be
slower than empty devices.
The change includes a new pool-wide allocation queue which would throttle and
order allocations in the ZIO pipeline. The queue would be ordered by issued
time and offset and would provide an initial amount of allocation of work to
each top-level vdev. The allocation logic utilizes a reservation system to
reserve allocations that will be performed by the allocator. Once an allocatio
n
is successfully completed it's scheduled on a given top-level vdev. Each top-
level vdev maintains a maximum number of allocations that it can handle
(mg_alloc_queue_depth). The pool-wide reserved allocations (top-levels *
mg_alloc_queue_depth) are distributed across the top-level vdevs metaslab
groups and round robin across all eligible metaslab groups to distribute the
work. As top-levels complete their work, they receive additional work from the
pool-wide allocation queue until the allocation queue is emptied.

Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <paul.dagnelie@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Sebastien Roy <sebastien.roy@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: George Wilson <george.wilson@delphix.com>

MFC r305325: MFV r303078:
7086 ztest attempts dva_get_dsize_sync on an embedded blockpointer

illumos/illumos-gate@926549256b71acd595f69b236779ff6b78fa08ef
https://github.com/illumos/illumos-gate/commit/926549256b71acd595f69b236779ff6b7
8fa08ef

https://www.illumos.org/issues/7086
In dbuf_dirty(), we need to grab the dn_struct_rwlock before looking at the
db_blkptr, to prevent it from being changed by syncing context.
Otherwise we may see that ztest got a segfault from this stack:
libzpool.so.1`dva_get_dsize_sync+0x98(872f000, b32b240, fed7811b, 0, b4cda20,
0)
libzpool.so.1`bp_get_dsize+0x60(872f000, b32b240, 0, 97cb780, 9d4c1a8, 0)
libzpool.so.1`dbuf_dirty+0x9b3(ce0a100, 97cb780, 9, fecd2530)
libzpool.so.1`dmu_buf_will_dirty+0xc3(ce0a100, 97cb780, ea293d6c, 1)
libzpool.so.1`zap_lockdir+0x1a0(8aaa3c0, 1, 0, 97cb780, 1, 1)
libzpool.so.1`zap_remove_norm+0x30(8aaa3c0, 1, 0, 8728b10, 0, 97cb780)
libzpool.so.1`zap_remove+0x29(8aaa3c0, 1, 0, 8728b10, 97cb780, a)
ztest_replay_remove+0x225(ea294588, 8728ae8, 0, 38010000, 0, 0)
ztest_remove+0x9f(ea294588, ea293f50, 4, 3)
ztest_object_init+0x78(ea294588, ea293f50, 4e0, 1)
ztest_dmu_object_alloc_free+0x71(ea294588, 13)
ztest_dmu_objset_create_destroy+0x224(80cef08, 13, 0, 805d36c, 9017ad44, 0)
ztest_execute+0x89(a, 807c720, 13, 0)
ztest_thread+0xea(13, 0, 0, 0)
libc.so.1`_thrp_setup+0x88(f0983240)
libc.so.1`_lwp_start(f0983240, 0, 0, 0, 0, 0)
Looking into it a bit, we see that this is an embedded blockpointer, so
BP_GET_NDVAS should have returned 0:
b32b240::blkptr
EMBEDDED [L0 ZAP_OTHER] et=0 LZ4 size=200L/4aP birth=80L
Instead, it looks like another thread is modifying this blockpointer:
b32b240::ugrep | ::whatis
f47a0e0c is in [ stack tid=0x19f ]
ebd6ec40 is in [ stack tid=0x226 ]
ea293bd0 is in [ stack tid=0x244 ]
ea293be4 is in [ stack tid=0x244 ]

Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Matthew Ahrens <mahrens@delphix.com>

MFC r305324: MFV r303077:
7072 zfs fails to expand if lun added when os is in shutdown state

illumos/illumos-gate@c39a2aae1e2c439d156021edfc20910dad7f9891
https://github.com/illumos/illumos-gate/commit/c39a2aae1e2c439d156021edfc20910da
d7f9891

https://www.illumos.org/issues/7072
upstream:
38733 zfs fails to expand if lun added when os is in shutdown state
DLPX-36910 spares and caches should not display expandable space
DLPX-39262 vdev_disk_open spam zfs_dbgmsg buffer

Reviewed by: Igor Kozhukhov <ikozhukhov@gmail.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: George Wilson <george.wilson@delphix.com>

MFC r305323: MFV r302991: 6950 ARC should cache compressed data

illumos/illumos-gate@dcbf3bd6a1f1360fc1afcee9e22c6dcff7844bf2
https://github.com/illumos/illumos-gate/commit/dcbf3bd6a1f1360fc1afcee9e22c6dcff
7844bf2

https://www.illumos.org/issues/6950
When reading compressed data from disk, the ARC should keep the compressed
block cached and only decompress it when consumers access the block. The
uncompressed data should be short-lived allowing the ARC to cache a much large
r
amount of data. The DMU would also maintain a smaller cache of uncompressed
blocks to minimize the impact of decompressing frequently accessed blocks.

Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Don Brady <don.brady@intel.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: George Wilson <george.wilson@delphix.com>

MFC r306665: zfs: fix a wrong assertion for extended attributes

PR: 213112

MFC r305224: MFV r304158:
7136 ESC_VDEV_REMOVE_AUX ought to always include vdev information

7115 6922 generates ESC_ZFS_VDEV_REMOVE_AUX a bit too often

illumos/illumos-gate@b72b6bb10ad55121a1b352c6f68ebdc8e20c9086
https://github.com/illumos/illumos-gate/commit/b72b6bb10ad55121a1b352c6f68ebdc8e
20c9086

https://www.illumos.org/issues/7136
6922 added ESC_ZFS_VDEV_REMOVE_AUX and ESC_ZFS_VDEV_REMOVE_DEV sysevents
whenever an aux device gets removed from a pool. However, those sysevents will
be created without the vdev_guid and vdev_path fields. It would be better to
always populate those fields.

https://www.illumos.org/issues/7115
The addition of spa_event_notify in vdev removal code (see #6922) causes event
s
to be generated even if the spare failed to be removed with EBUSY.

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Josef 'Jeff' Sipek <jeffpc@josefsipek.net>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Alan Somers <asomers@gmail.com>

MFC r305222: MFV r302993: 7104 increase indirect block size

illumos/illumos-gate@4b5c8e93cab28d3c65ba9d407fd8f46e3be1db1c
https://github.com/illumos/illumos-gate/commit/4b5c8e93cab28d3c65ba9d407fd8f46e3
be1db1c

https://www.illumos.org/issues/7104
The current default indirect block size is 16KB. We can improve
performance by increasing it to 128KB. This is especially helpful for
any workload that needs to read most of the metadata, e.g.
scrub/resilver, file deletion, filesystem deletion, and zfs send.
We also need to fix a few space estimation errors to make the tests
pass.

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Matthew Ahrens <mahrens@delphix.com>

MFC r305221: MFV r302992:
7071 lzc_snapshot does not fill in errlist on ENOENT

illumos/illumos-gate@25f7d993adbfb3452ac4625b3791670746d35ae3
https://github.com/illumos/illumos-gate/commit/25f7d993adbfb3452ac4625b379167074
6d35ae3

https://www.illumos.org/issues/7071
upstream
DLPX-40482 lzc_snapshot does not fill in errlist on ENOENT

Reviewed by: Igor Kozhukhov <ikozhukhov@gmail.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Matthew Ahrens <mahrens@delphix.com>

MFC r305211: MFV r302662: 6447 handful of nvpair cleanups

illumos/illumos-gate@759e89be359f2af635e4122d147df56bce948773
https://github.com/illumos/illumos-gate/commit/759e89be359f2af635e4122d147df56bc
e948773

https://www.illumos.org/issues/6447
I got a patch from someone who uses nvpair code outside of illumos. It fixes a
couple of gcc warnings/bugs for him.
1. silence uninitialized use warnings
2. add parentheses around assignment used as truth value
3. fix printf format specifier (ll is for integers only)
4. strstr, strspn, strcspn, and strcmp are declared in string.h, not
strings.h.
5. avoid scanning integer into boolean variable

Reviewed by: Josef 'Jeff' Sipek <jeffpc@josefsipek.net>
Reviewed by: Andy Stormont <astormont@racktopsystems.com>
Reviewed by: Garrett D'Amore <garrett@damore.org>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Steve Dougherty <sdougherty@barracuda.com>

MFC r305210: MFV r302661:
7082 bptree_iterate() passes wrong args to zfs_dbgmsg()

illumos/illumos-gate@10e67aa0db0823d5464aafdd681f3c966155c68e
https://github.com/illumos/illumos-gate/commit/10e67aa0db0823d5464aafdd681f3c966
155c68e

https://www.illumos.org/issues/7082
upstream
DLPX-40542 bptree_iterate() passes wrong args to zfs_dbgmsg()

Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Igor Kozhukhov <ikozhukhov@gmail.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Matthew Ahrens <mahrens@delphix.com>

MFC r305209: MFV r302660: 6314 buffer overflow in dsl_dataset_name

illumos/illumos-gate@9adfa60d484ce2435f5af77cc99dcd4e692b6660
https://github.com/illumos/illumos-gate/commit/9adfa60d484ce2435f5af77cc99dcd4e6
92b6660

https://www.illumos.org/issues/6314
Callers of dsl_dataset_name pass a buffer of size ZFS_MAXNAMELEN, but
dsl_dataset_name copies the datasets' name PLUS the snapshot name to it,
resulting in a max of 2 * ZFS_MAXNAMELEN + '@'.

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Igor Kozhukhov <ikozhukhov@gmail.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Matthew Ahrens <mahrens@delphix.com>

MFC r305207: MFV r302659: 6931 lib/libzfs: cleanup gcc warnings

illumos/illumos-gate@88f61dee20b358671b1b643e9d1dbf220a1d69be
https://github.com/illumos/illumos-gate/commit/88f61dee20b358671b1b643e9d1dbf220a1d69be

https://www.illumos.org/issues/6931
need cleanup:
CERRWARN += -_gcc=-Wno-switch
CERRWARN += -_gcc=-Wno-parentheses
CERRWARN += -_gcc=-Wno-unused-function

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Igor Kozhukhov <ikozhukhov@gmail.com>

MFC r305200: MFV r302651:
7054 dmu_tx_hold_t should use refcount_t to track space

illumos/illumos-gate@0c779ad424a92a84d1e07d47cab7f8009189202b
https://github.com/illumos/illumos-gate/commit/0c779ad424a92a84d1e07d47cab7f8009
189202b

https://www.illumos.org/issues/7054
upstream:
ee0003de7d3e598499be7ac3fe6b61efcc47cb7f
DLPX-40399 dmu_tx_hold_t should use refcount_t to track space

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Igor Kozhukhov <ikozhukhov@gmail.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Matthew Ahrens <mahrens@delphix.com>

MFC r305199: MFV r302648: 7019 zfsdev_ioctl skips secpolicy when FKIOCTL is set

Note that the bulk of the upstream change is not applicable to FreeBSD
and the affected files are not even in the vendor area.

illumos/illumos-gate@45b1747515a17db45e8971501ee84a26bdff37b2
https://github.com/illumos/illumos-gate/commit/45b1747515a17db45e8971501ee84a26bdff37b2

https://www.illumos.org/issues/7019
Currently zfsdev_ioctl, when confronted by a request with the FKIOCTL flag set,
skips all processing of secpolicy functions. This means that ZFS is not doing
any kind of verification of the credentials or access rights of the caller and
assuming that (as it is an in-kernel client) all such checks have already been
done.
This turns out to be quite a dangerous assumption, especially with respect to
sdev. In general I don't think it's particularly reasonable to offload this
enforcement of access rights onto other kernel subsystems when ZFS has some
particular local semantics in this area (delegated datasets etc) and does not
provide any kind of API to allow other subsystems to avoid code duplication
when doing it. ZFS should apply its normal access policy to requests from
within the kernel, and callers should take care to give it the correct
credentials and call it from the correct context in order to get the results
they need.
You can observe the currently unfortunate consequences of this bug in any non-
global zone that has access to /dev/zvol or any subset of it via sdev profiles.
In particular, a zone used to contain a KVM or similar which has a single zvol
passed through to it using a <device match= block in its zone XML.
Even though sdev makes something of an attempt to control for whether the
caller should have access to nodes in /dev/zvol, it doesn't do this correctly,
or really at all in the lookup call path. So, if we have a zone that's been
given access to any part of /dev/zvol, it can simply look up the full path to
any other zvol on the entire system, and the node will appear and be able to be
used.

Reviewed by: Robert Mustacchi <rm@joyent.com>
Reviewed by: Richard Lowe <richlowe@richlowe.net>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Alex Wilson <alex.wilson@joyent.com>

MFC r305198: MFV r302647:
6922 Emit ESC_ZFS_VDEV_REMOVE_AUX after removing an aux device

illumos/illumos-gate@63364b0ee2604783e7a55f8425888867768eafa4
https://github.com/illumos/illumos-gate/commit/63364b0ee2604783e7a55f84258888677
68eafa4

https://www.illumos.org/issues/6922
ZFS does not do a config_sync after removing an aux (spare, log, or cache)
device. AFAICT this isn't being done because it is slow and was deemed
unnecessary. However, it should be such a rare operation that speed doesn't
matter, and not doing it results in two problems:
1) It is theoretically possible to remove an aux device from one pool and
attach it to another, then lose power. When power is restored, both pools woul
d
think that they own the aux device.
2) Removal of the aux device doesn't send any useful sysevents to userland.

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Alan Somers <asomers@gmail.com>

MFC r305197: MFV r302646:
6980 6902 causes zfs send to break due to 32-bit/64-bit struct mismatch

illumos/illumos-gate@ea4a67f462de0a39a9adea8197bcdef849de5371
https://github.com/illumos/illumos-gate/commit/ea4a67f462de0a39a9adea8197bcdef84
9de5371

https://www.illumos.org/issues/6980
doing zfs send -i snap1 snap2 >testfile results in
internal error: Invalid argument
Abort (core dumped)

Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Matthew Ahrens <mahrens@delphix.com>

MFC r305195: MFV r302643:
6902 speed up listing of snapshots if requesting name only and sorting by name

This was our change from the beginning, so just reduce the upstream diff.

MFC r305193: MFV r302642:
6876 Stack corruption after importing a pool with a too-long name

illumos/illumos-gate@c971037baa5d64dfecf6d87ed602fc3116ebec41
https://github.com/illumos/illumos-gate/commit/c971037baa5d64dfecf6d87ed602fc3116ebec41

https://www.illumos.org/issues/6876
Calling dsl_dataset_name on a dataset with a 256 byte buffer is asking for
trouble. We should check every dataset on import, using a 1024 byte buffer and
checking each time to see if the dataset's new name is longer than 256 bytes.

Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: Paul Dagnelie <pcd@delphix.com>

MFC r306292: fix vnode lock assertion for extended attributes directory

MFC r305123: Fix kernel panic when inheriting properties without default.

There are two writable hidden properties "iscsioptions" and "stmf_sbd_lu",
that have no default string value. Attempt to unset them or replicate
caused kernel panic. This simple bandaid seems fixes the problem nicely.

MFC r305679: Switch random_get_pseudo_bytes() shim to arc4rand().

Our shim for Solaris random_get_bytes() uses read_random(), that looks
reasonable, since it guaranties reliably seeded random data. On the other
side Solaris random_get_pseudo_bytes() does not provide this guarantie,
and its original Solaris implementation is equivalent to our arc4rand(),
using software crypto without stressing slower hardware RNG.

MFstable/11 r305271:

MFC r303576:

Conditionalize code which defines sysctls per _KERNEL #ifdef guard

This resolves several issues when compiling libzpool (userspace library), i.e.
-Wimplicit-function-declaration and -Wmissing-declarations issues.

Tested with: clang 3.8.1, gcc 4.2.1, gcc 5.3.0

MFC r303763,303791,303869: zfs: honour and make use of vfs vnode locking protocol

PR: 209158

MFC r302838: 6513 partially filled holes lose birth time

MFC r302837: 6844 dnode_next_offset can detect fictional holes

MFC r302836: 6874 rollback and receive need to reset ZPL state to what's on disk

MFC r302840: 6878 Add scrub completion info to "zpool history"

MFC r302839: 6940 Cannot unlink directories when over quota

MFC r302772: re-apply r299908: zfsctl_snapdir_lookup: clear VV_ROOT of
snapshot's root

MFC r302507:
Avoid truncating the return value of DTrace predicates.

MFC r300134: move zfsctl_freebsd_root_lookup right next to
zfsctl_root_lookup

MFC r300130: zfsctl_freebsd_root_lookup: gfs_vop_lookup may return a
doomed vnode

MFC r299951: do not destroy 'snapdir' when it becomes inactive

MFC r299949: try to recycle "snap" vnodes as soon as possible

MFC r299947: fix locking in zfsctl_root_lookup

MFC r299946: gfs_lookup_dot() does not have to acquire any locks

MFC r299945: avoid deadlock between zfsctl_snapdir_lookup and
zfsctl_snapshot_reclaim

MFC r299940: fix a vnode reference leak caused by illumos compat traverse()

MFC r299914: zfsctl_ops_snapshot: remove methods should never be called

MFC r301273: zfs_root: fix a potential root vnode reference leak

MFC r299908,300131,301275: zfs: set VROOT / VV_ROOT consistently and in
a single place

MFC r299902,299938: mount_snapshot: consolidate all error handling

MFC r299906,301870: add zfs_vptocnp with special handling for snapshots
under .zfs

Note that the changed is adjusted for the lack of LK_VNHELD in this
branch.

MFC r300145: add vop_print methods to vnode operatios of various zfsctl
node types

MFC r301873: l2arc: reset b_tmp_cdata to NULL in the case of unset b_daddr

MFC r302123: fix deadlock-prone code in getzfsvfs()

MFC r299900: zfsctl: fix several problems with reference counts

PR: 207464

MFC r298105: zfs: enable vn_io_fault support

MFC r300133: zfsctl_common_fid: remove redundant assignment

MFC r300132: zfsctl: tighten assertion and remove unused definition

MFC r302265, r302382

Allow ZFS ARC min / max to be tuned at runtime

Relnotes: YES
Sponsored by: Multiplay

MFC r300870,r300884:

r300870:

Unbreak the zfs(4) build

vm/vm_pageout.h grew a dependency on the bool typedef in r300865

arc.c didn't include sys/types.h, which included the definition for the typedef

Other items (ofed, drm2) might need to be chased for this commit.

Pointyhat to: alc

r300884:

Fix up r300870

The sys/types.h fix I proposed was only tested with zfs(4), not with
libzpool, which is where the build failure actually existed

Remove vm/vm_pageout.h from arc.c and zfs_vnops.c because they're both
unneeded

In collaboration with: kib

MFC r299657:

Include arpa/inet.h to get the htonl(3) definition

MFC r300024: zfs_ioc_rename: fix a reversed condition

PR: 209093

MFC r297848: l2arc: make sure that all writes honor ashift of a cache device

Note: no MFC stable/9 because it has become quite out of date with head,
so the merge would be quite labourious and, thus, risky.

MFC r298106: zfs_rezget: z_vnode can not be NULL if zp is valid

MFC r298473,298787: add invpcid, fix sahf/lahf in dtrace disassembler

MFC r298472: MFV r298471: 6052 decouple lzc_create() from the
implementation details

MFC r298072

Don't corrupt ZFS label's physpath attribute when booting while a disk is
missing

Prior to this change, vdev_geom_open_by_path would call vdev_geom_attach
prior to verifying the device's GUIDs. vdev_geom_attach calls
vdev_geom_attrchange to set the physpath in the vdev object. The result is
that if the disk could not be found, then the labels for other disks in the
same TLD would overwrite the missing disk's physpath with the physpath of
whichever disk currently has the same devname as the missing one used to
have.

MFC r297986, r298017 to vdev_geom.c

r297986 | asomers | 2016-04-14 13:20:31 -0600 (Thu, 14 Apr 2016) | 6 lines

Update a debugging message in vdev_geom_open_by_guids for consistency with
similar messages elsewhere in the file.

r298017 | asomers | 2016-04-14 17:14:41 -0600 (Thu, 14 Apr 2016) | 8 lines

Add more debugging statements in vdev_geom.c

Log a debugging message whenever geom functions fail in vdev_geom_attach.
Printing these messages is controlled by vfs.zfs.debug

MFC r297832: MFV r297831: 6322 ZFS indirect block predictive prefetch

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Author: Alexander Motin <mav@FreeBSD.org>

Improve speculative prefetch of indirect blocks.

Scalability of many operations on wide ZFS pool can be limited by
requirement to prefetch indirect blocks first. Recently added
asynchronous indirect block read partially helped, but did not
solve the problem completely. This patch extends existing prefetcher
functionality to explicitly work with indirect blocks.

Before this change prefetcher issued reads for up to 8MB of data in
advance. With this change it also issues indirect block reads
for up to 64MB of data in advance, so that when it will be time to
actually read those data, it can be done immediately. Alike effect
can be achieved by just increasing maximal data prefetch distance,
but at higher memory cost.

Also this change introduces indirect block prefetch for rewrite
operations, that was never done before. Previously ARC miss for
Indirect blocks regularly blocked rewrites, converting perfectly
aligned asynchronous operations into synchronous read-write pairs,
significantly reducing maximal rewrite speed.

While being there this issue was also fixed:
- prefetch was done always, even if caching for the dataset was
completely disabled.

Testing on FreeBSD with zvol on top of 6x striped 2x mirrored pool
of 12 assorted HDDs shown me such performance numbers:
------- BEFORE --------
Write 491363677 bytes/sec
Read 312430631 bytes/sec
Rewrite 97680464 bytes/sec
-------- AFTER --------
Write 493524146 bytes/sec
Read 438598079 bytes/sec
Rewrite 277506044 bytes/sec

Closes #65
Closes #80

openzfs/openzfs@792fd28ac04f78cc5e43ead2d72a96f244ea84e8

MFC r297509: MFV r297506: 6738 zfs send stream padding needs documentation

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Eli Rosenthal <eli.rosenthal@delphix.com>

illumos/illumos-gate@c20404ff77119516354b0d112d28b7ea0dadd303

MFC r297507: MFV r297504:
6681 zfs list burning lots of time in dodefault() via dsl_prop_*

Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Approved by: Matthew Ahrens <mahrens@delphix.com>
Author: Alex Wilson <alex.wilson@joyent.com>

illumos/illumos-gate@d09e4475f635b6f66ee68d8c17a32bba7be17c96

MFC r297763: MFV r297760: 6418 zpool should have a label clearing command

Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Author: Will Andrews <will@firepipe.net>

Closes #83
Closes #32

openzfs/openzfs@9663688425131744221ea99f9e66b9ed964492ae

FreeBSD already had `zpool labelclear` functionality, so this is mostly
just a diff reduction.

MFC 297868

Fix rare double free in vdev_geom_attrchanged

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c
Don't drop the g_topology_lock before freeing old_physpath. That
opens up a race where one thread can call vdev_geom_attrchanged,
set old_physpath, drop the g_topology_lock, then block trying to
acquire the SCL_STATE lock. Then another thread can come into
vdev_geom_attrchanged, set old_physpath to the same value, and
proceed to free it. When the first thread resumes, it will free
the same location.

It turns out that the SCL_STATE lock isn't needed. It was
originally added by gibbs to protect vd->vdev_physpath while
updating the same. However, the update process subsequently was
switched to an atomic operation (a pointer swap). Now, there is
no need for the SCL_STATE lock, and hence no need to drop the
g_topology_lock.

MFC r298589:
Allow DOF sections with excessively long probe function components.

PR: 207735

MFC r297812: zio: align use of "no dump" flag between use_uma and !use_uma cases

MFC r296479:
Fix fasttrap tracepoint locking.

MFC r297771:
Initialize SDT probes during SI_SUB_DTRACE_PROVIDER.

MFC r297513: remove emulation of VFS_HOLD and VFS_RELE from opensolaris compat

MFC r297709: zio write issue threads should have lower (numerically
greater) priority

MFC r297672: Alike to r293708 relax pool check in vdev_geom_open_by_path().

This made impossible spare disk open by known path, which kind of worked
only because the same fix was applied to vdev_geom_attach_by_guids() in
r293708.

MFC r297421: Plug open count leak on zvol rename.

MFC r297420: Switch from using make_dev_p() to make_dev_s() to close races.

MFC r297337: Pass through error code from make_dev_p().

ENAMETOOLONG is much more informative in logs then ENXIO.

MFC r297232: Unify ignoring EEXIST from zvol_create_minor().

This fixes creation of zvol devices for snapshots during zfs receive,
that previously failed with "ZFS WARNING: Unable to create ZVOL" message.
This solution is not perfect, but IMHO better then it was before.

MFC r277504 (by will): Remove commented log messages.

MFC r277450 (by will):
Use the "zfs_gfs" tag for GFS vnodes to make them easier to identify.

MFC r270382 (by delphij): MFV r270197:

Illumos issue:
5066 remove support for non-ANSI compilation
5068 Remove SCCSID() macro from <macros.h>

MFC r277629 (by will):
When creating or updating a node, use vfs_timestamp() for "now" instead
of gethrestime(), to allow the administrator to decide the appropriate
timestamp precision instead of always using nanosecond precision.

MFC r296615: Make ZFS ignore stripe sizes above SPA_MAXASHIFT (8KB).

If device has stripe size bigger then maximal sector size supported by
ZFS, there is nothing can be done to avoid read-modify-write cycles.
Taking that stripe size into account will only reduce space efficiency
and pointlessly bother user with warnings that can not be fixed.

MFC r296613: Make ZFS more picky to GEOM stripe sizes and offsets.

Use of misaligned or non-power-of-2 stripes is not really useful for ZFS,
since increased ashift won't help to avoid read-modify-write cycles, and
only reduce pool space efficiency and compression rates.

MFC r296530: MFV r296529:
6672 arc_reclaim_thread() should use gethrtime() instead of ddi_get_lbolt()
6673 want a macro to convert seconds to nanoseconds and vice-versa

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Josef 'Jeff' Sipek <jeffpc@josefsipek.net>
Reviewed by: Robert Mustacchi <rm@joyent.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Eli Rosenthal <eli.rosenthal@delphix.com>

illumos/illumos-gate@a8f6344fa0921599e1f4511e41b5f9a25c38c0f9

MFC r296528: MFV r296527: 6659 nvlist_free(NULL) is a no-op

Reviewed by: Toomas Soome <tsoome@me.com>
Reviewed by: Marcel Telka <marcel@telka.sk>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Josef 'Jeff' Sipek <jeffpc@josefsipek.net>

illumos/illumos-gate@aab83bb83be7342f6cfccaed8d5fe0b2f404855d

MFC r296523: MFV r296522:
6541 Pool feature-flag check defeated if "verify" is included in the dedup
property value

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Richard Laager <rlaager@wiktel.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: ilovezfs <ilovezfs@icloud.com>

illumos/illumos-gate@971640e6aa954c91b0706543741aa4570299f4d7

MFC r296521: MFV r296520:
6562 Refquota on receive doesn't account for overage

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Reviewed by: Toomas Soome <tsoome@me.com>
Approved by: Gordon Ross <gwr@nexenta.com>
Author: Dan McDonald <danmcd@omniti.com>

illumos/illumos-gate@5f7a8e6d750cb070a3347f045201c6206caee6aa

MFC r296519: MFV r296518: 5027 zfs large block support (add copyright)

Author: Matthew Ahrens <matt@mahrens.org>

illumos/illumos-gate@c3d26abc9ee97b4f60233556aadeb57e0bd30bb9

MFC r296516: MFV r296515:
6536 zfs send: want a way to disable setting of DRR_FLAG_FREERECORDS

Reviewed by: Anil Vijarnia <avijarnia@racktopsystems.com>
Reviewed by: Kim Shrier <kshrier@racktopsystems.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Andrew Stormont <astormont@racktopsystems.com>

illumos/illumos-gate@880094b6062aebeec8eda6a8651757611c83b13e

MFC r296514: MFV r296513:
6450 scrub/resilver unnecessarily traverses snapshots created after the
scrub started

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: Matthew Ahrens <mahrens@delphix.com>

illumos/illumos-gate@38d61036746e2273cc18f6698392e1e29f87d1bf

MFC r296512: MFV r296511: 6537 Panic on zpool scrub with DEBUG kernel

Reviewed by: Steve Gonczi <gonczi@comcast.net>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Reviewed by: Igor Kozhukhov <ikozhukhov@gmail.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Matthew Ahrens <mahrens@delphix.com>
Author: Gary Mills <gary_mills@fastmail.fm>

illumos/illumos-gate@8c04a1fa3f7d569d48fe9b5342d0bd4c533179b9

MFC r296510, r296563, r296567: MFV r296505:
6531 Provide mechanism to artificially limit disk performance

Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Prakash Surya <prakash.surya@delphix.com>

illumos/illumos-gate@97e81309571898df9fdd94aab1216dfcf23e060b

MFC r296021 (by smh): Removed unused label and fix mutex_exit order

Remove unused done label from zfs_setacl fixing PVS-Studio V729.

Fix mutex_exit order to mirror the mutex_enter order.

MFC r295125: MFV r294821:
6529 Properly handle updates of variably-sized SA entries.

Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Ned Bass <bass6@llnl.gov>
Reviewed by: Tim Chase <tim@chase2k.com>
Approved by: Gordon Ross <gwr@nexenta.com>
Author: Andriy Gapon <avg@icyb.net.ua>

illumos/illumos-gate@e7e978b1f75353cb29673af9b35453c20c2827bf

During the update process in sa_modify_attrs(), the sizes of existing
variably-sized SA entries are obtained from sa_lengths[]. The case where
a variably-sized SA was being replaced neglected to increment the index
into sa_lengths[], so subsequent variable-length SAs would be rewritten
with the wrong length. This patch adds the missing increment operation
so all variably-sized SA entries are stored with their correct lengths.

Another problem was that index into attr_desc[] was increased even when
an attribute was removed. If that attribute was not the last attribute,
then the last attribute was lost.

MFC r294820: MFV r294819: 6495 Fix mutex leak in dmu_objset_find_dp

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Albert Lee <trisk@omniti.com>
Author: Steven Hartland <steven.hartland@multiplay.co.uk>

illumos/illumos-gate@2bad22584defe4667f99737e3158d336e4dcca11

MFC r294817: MFV r294816:
4986 receiving replication stream fails if any snapshot exceeds refquota

Reviewed by: John Kennedy <john.kennedy@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Gordon Ross <gordon.ross@nexenta.com>
Author: Dan McDonald <danmcd@omniti.com>

illumos/illumos-gate@5878fad70d76d8711f6608c1f80b0447601261c6

MFC r294815: MFV r294814: 6393 zfs receive a full send as a clone

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Paul Dagnelie <pcd@delphix.com>

illumos/illumos-gate@68ecb2ec930c4b0f00acaf8e0abb2b19c4b8b76f

This allows to do a full (non-incremental send) and receive it as a clone
of an existing dataset. It can leverage nopwrite to share blocks with the
origin. This can be used to change the relationship of datasets on the
target. For example, maybe on the source you have:

A ---- B ---- C

And you have sent to the target a full of B, and the incremental B->C:

B ---- C

You later realize that you want to have A on the target. You will have to
do a full send of A, but nopwrite can save you space on the target if you
receive it as a clone of B, assuming that A and B have some blocks inxi
common:

B ---- C
\
A

MFC r294813: MFV r294812:
6434 sa_find_sizes() may compute wrong SA header size

Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Andriy Gapon <avg@freebsd.org>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: James Pan <jiaming.pan@yahoo.com>

illumos/illumos-gate@3502ed6e7cb3f3d2e781960ab8fe465fdc884834

MFC r294811: MFV r294810: 6414 vdev_config_sync could be simpler

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Will Andrews <will@firepipe.net>

illumos/illumos-gate@eb5bb58421f46cee79155a55688e6c675e7dd361

MFC r294809: MFV r294808:
6421 Add missing multilist_destroy calls to arc_fini

Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Jorgen Lundman <lundman@lundman.net>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Prakash Surya <prakash.surya@delphix.com>

illumos/illumos-gate@57deb2328260c447bf1db25fe74e0eece102733e

MFC r294807: MFV r294806:
6388 Failure of userland copy should return EFAULT

Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Richard Yao <ryao@gentoo.org>

illumos/illumos-gate@c71c00bbe8a9cdc7e3f4048b751f48e80441d506

MFC r294805: MFV r294804:
6386 Fix function call with uninitialized value in vdev_inuse

Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Richard Yao <ryao@gentoo.org>

illumos/illumos-gate@5bdd995ddb777f538bfbcc5e2d5ff1bed07ae56e

MFC r294803: MFV r294802: 6334 Cannot unlink files when over quota

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Toomas Soome <tsoome@me.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Simon Klinkert <simon.klinkert@gmail.com>

illumos/illumos-gate@6575bca01367958c7237253d88e5fa9ef0b1650a

MFC r294801: MFV r294800: 6385 Fix unlocking order in zfs_zget

Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: Andriy Gapon <avg@freebsd.org>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Richard Yao <ryao@gentoo.org>

illumos/illumos-gate@eaef6a96de3f6afbbccc69bd7a0aed4463689d0a

MFC r294799: MFV r294798:
6292 exporting a pool while an async destroy is running can leave entries
in the deferred tree

Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Andriy Gapon <avg@FreeBSD.org>
Reviewed by: Fabian Keil <fk@fabiankeil.de>
Approved by: Gordon Ross <gordon.ross@nexenta.com>

illumos/illumos-gate@a443cc80c742af740aa82130db840f02b4389365

MFC r294797: MFV r294796:
6319 assertion failed in zio_ddt_write: bp->blk_birth == txg

Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Dan McDonald <danmcd@omniti.com>

illumos/illumos-gate@b39b744be78c6327db43c1f69d11c2f5909f73cb

This is revert of 5693.

MFC r294794: MFV r294793:
6367 spa_config_tryenter incorrectly handles the multiple-lock case

Reviewed by: Alek Pinchuk <alek.pinchuk@nexenta.com>
Reviewed by: Josef 'Jeff' Sipek <josef.sipek@nexenta.com>
Reviewed by: Prashanth Sreenivasa <prashksp@gmail.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Reviewed by: Steven Hartland <steven.hartland@multiplay.co.uk>
Approved by: Matthew Ahrens <mahrens@delphix.com>

illumos/illumos-gate@e495b6e6735b803e422025a630352ef9bba788c5

MFC r294625 (by trasz):
Fix ru_oublocks accounting for ZFS. There are two code paths that can be
called from zfs_write() - one of them, through dmu_write(), was handled
correctly; the other wasn't.

Differential Revision: https://reviews.freebsd.org/D4923

MFC r293677 (by asomers): Record physical path information in ZFS Vdevs

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c:
If available, record the physical path of a vdev in ZFS meta-data.
Do this both when opening the vdev, and when receiving an attribute
change notification from GEOM.

Make vdev_geom_close() synchronous instead of deferring its work to
a GEOM event handler. There is no benefit to deferring the work and
this prevents a future open call from referencing a consumer that is
scheduled for destruction. The close followed by an immediate open
will occur during a vdev reprobe triggered by any type of I/O error.

Consolidate vdev_geom_close() and vdev_geom_detach() into
vdev_geom_close() and vdev_geom_close_locked(). This also moves the
cross linking operations between vdev and GEOM consumer into a
single place (linking in vdev_geom_attach() and unlinking in
vdev_geom_close_locked()).

Differential Revision: https://reviews.freebsd.org/D4524

MFC r290266 (by avg):
zfs: allow the lookup of extended attributes of an unlinked file

That's required for extattr_get_fd(2) and the like to work properly.

PR: 203201

MFC r274627 (by avg):
Revert r269093 which introduced physical zio alignment transform

Size of physical ZIOs must never be implicitly adjusted, it's
a responsibility of a caller to make sure that such a ZIO has proper offset
and size.

Discussed with: delphij, gibbs

MFV r258597 (by pjd):
When append-only, immutable or read-only flag is set don't allow for
hard links creation. This matches UFS behaviour.

Reported by: Oleg Ginzburg <olevole@olevole.ru>

MFC r262990: MFV r262983:

4638 Panic in ZFS via rfs3_setattr()/rfs3_write(): dirtying snapshot!

illumos/illumos-gate@2144b121c08e0eb676cc6ca4662ebbc9f9c22fe3

MFC r272359 (by will):
zfsvfs_create(): Refuse to mount datasets whose names are too long.

This is checked for in the zfs_snapshot_004_neg STF/ATF test (currently
still in projects/zfsd rather than head).

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:
- zfsvfs_create(): Check whether the objset name fits into
statfs.f_mntfromname, and return ENAMETOOLONG if not. Although
the filesystem can be unmounted via the umount(8) command, any
interface that relies on iterating on statfs (e.g. libzfs) will
fail to find the filesystem by its objset name, and thus assume
it's not mounted. This causes "zfs unmount", "zfs destroy",
etc. to fail on these filesystems, whether or not -f is passed.

MFSpectraBSD: 974872 on 2013/08/09

MFC r277503 (by will): Ignore sync requests from the system syncher,
i.e. VFS_SYNC(waitfor=MNT_LAZY).

ZFS already commits outstanding data every zfs_txg_timeout seconds, so these
syncs are unnecessarily intrusive.

MFSpectraBSD: 1105759 on 2014/12/11

MFC r277501 (by will): Eliminate an #ifdef illumos for zfs_ioc_rename().

Since allow_mounted is a FreeBSD-specific change, default to B_TRUE, then
locally check for the magic bit. Unconditionally check allow_mounted below.
Convert the setting of allow_mounted to an explicit boolean.

MFSpectraBSD: 672578 (in part) on 2013/07/19

MFC r286223 (by smh): Fix KSTACK_PAGES check in ZFS module

The check introduced by r285946 failed to add the dependency on
opt_kstack_pages.h which meant the default value for the platform instead
of the customised options KSTACK_PAGES=X was being tested.

Also wrap in #ifdef __FreeBSD__ for portability.

MFC r274304 (by delphij): MFV r274272 and diff reduction with upstream.

Illumos issue:
5244 zio pipeline callers should explicitly invoke next stage

MFC r277300 (by smh): Mechanically convert cddl sun #ifdef's to illumos

Since the upstream for cddl code is now illumos not sun, mechanically
convert all sun #ifdef's to illumos #ifdef's which have been used in all
newer code for some time.

Also do a manual pass to correct the use if #ifdef comments as per style(9)
as well as few uses of #if defined(__FreeBSD__) vs #ifndef illumos.

MFC r271785:
Reorder sysctls for spa.c global tunables; add sysctl for ccw_retry_interval.

MFC r269222: Reschedule the 'deadman' callout after handling, this makes our
code behave more like it is on Solaris.

Differential Revision: https://phabric.freebsd.org/D457

MFC r271788 (by will):
Enable ZFS debug flags to be modified via vfs.zfs.debug_flags.

This is primarily only of interest to ZFS developers, but it makes it
easier to get additional debugging.

MFSpectraBSD: 517074 on 2011/12/15 (by will), 662343 on 2013/03/20 (by gibbs)

MFC r277492 (by will): Add vfs.zfs.reference_tracking_enable sysctl/tunable.

This is primarily for developer/debugging use; it enables built-in tagged
tracking of refcounts inside ZFS. It can only be enabled from the loader,
since it modifies how in-core state is managed. Default remains disabled.

MFC r271781i (by will):
bpobj_iterate_impl(): Close a refcount leak iterating on a sublist.

If bpobj_space() returned non-zero here, the sublist would have been
left open, along with the bonus buffer hold it requires. This call
does not invoke any calls to bpobj_close() itself.

This bug doesn't have any known vector, but was found on inspection.

MFC after: 1 week
Sponsored by: Spectra Logic
Affects: All ZFS versions starting 21 May 2010 (illumos cde58dbc)
MFSpectraBSD: r1050998 on 2014/03/26

MFC r264670: MFV r264667:

4752 fan out read zio taskqs

illumos/illumos-gate@1b497ab83e8f1c58bba5da59c649207a442a4720

MFC r296919: Fix small memory leak on attempt to access deleted snapshot.

MFC r296610:

ZFS send fails to transmit some holes

PR: 207714
Approved by: re (gjb)
Sponsored by: Multiplay

MFC r292066, r292069, r293708, r294027, and r294358, mostly to vdev_geom.c

r292066 | asomers | 2015-12-10 14:46:21 -0700 (Thu, 10 Dec 2015) | 25 lines

During vdev_geom_open, require that the vdev guids match the device's label
except during split, add, or create operations. This fixes a bug where the
wrong disk could be returned, and higher layers of ZFS would immediately
eject it again.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c:
o When opening by GUID, require both the pool and vdev GUIDs to
match. While it is highly unlikely for two vdevs to have the same
vdev GUIDs, the ZFS storage pool allocator only guarantees they
are unique within a pool.

o Modify the open behavior to:
- If we are opening a vdev that hasn't previously been opened,
open by path without checking GUIDs.
- Otherwise, open by path and verify GUIDs.
- If that fails, search all geom providers for a device with
matching GUIDs.
- If that fails, return ENOENT.

r292069 | asomers | 2015-12-10 17:04:13 -0700 (Thu, 10 Dec 2015) | 6 lines

Change an important error message from ZFS_LOG to printf

r293708 | asomers | 2016-01-11 15:15:46 -0700 (Mon, 11 Jan 2016) | 16 lines

Fix importing l2arc device by guid

After r292066, vdev_geom verifies both the vdev and pool guids of device
labels during open. However, spare and l2arc devices don't have pool guids,
so opening them by guid will fail (opening by path, when the pathname is
known, still succeeds). This change allows a vdev to be opened by guid if
the label contains no pool_guid, which is the case for inactive spares and
l2arc devices.

r294027 | asomers | 2016-01-14 11:19:05 -0700 (Thu, 14 Jan 2016) | 14 lines

Fix race condition involving ZFS remove events

When a ZFS drive disappears, ZFS sends a resource.fs.zfs.removed event to
userland. A userland program like zfsd(8) can use that event, for example to
activate a hotspare. The current code contains a race condition: vdev_geom
will sent the sysevent _before_ spa.c would update the vdev's status,
causing userland processes to see pool state that does not reflect the
device removal. This change moves the sysevent to spa.c, closing the race.

r294358 | asomers | 2016-01-19 16:16:24 -0700 (Tue, 19 Jan 2016) | 10 lines

Quell harmless CID about unchecked return value in nvlist_get_guids.

The return value doesn't need to be checked, because nvlist_get_guid's
callers check the returned values of the guids.

MFC r293271:

Fix const conversion warning in lz4_decompress

Sponsored by: Multiplay

MFC 289769,289822,290143,290144:
Rename remaining linux32 symbols from linux_* to linux32_*.

289769:
Rename remaining linux32 symbols such as linux_sysent[] and
linux_syscallnames[] from linux_* to linux32_* to avoid conflicts with
linux64.ko. While here, add support for linux64 binaries to systrace.
- Update NOPROTO entries in amd64/linux/syscalls.master to match the
main table to fix systrace build.
- Add a special case for union l_semun arguments to the systrace
generation.
- The systrace_linux32 module now only builds the systrace_linux32.ko.
module on amd64.
- Add a new systrace_linux module that builds on both i386 and amd64.
For i386 it builds the existing systrace_linux.ko. For amd64 it
builds a systrace_linux.ko for 64-bit binaries.

289822:
Fix build for the KTR-enabled kernels.

290143:
Fix build with DEBUG defined.

290144:
Update for LINUX32 rename. The assembler didn't complain about undefined
symbols but just used 0 after the rename.

MFC r294102:

MFV r294101: 6527 Possible access beyond end of string in zpool comment

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Approved by: Gordon Ross <gwr@nexenta.com>

illumos/illumos-gate@2bd7a8d078223b122d65fea49bb8641f858b1409

This fixes erroneous double increments of the 'check' variable in a loop
in spa_prop_validate(). I ran into this in the clang380-import branch,
where clang 3.8.0 warns about it. (It is already fixed there.)

MFC: r287337
Apply the noline attribute to vdev_queue_max_async_writes

This makes it possible to analyze the performance of the new ZFS
write throttle with dtrace

PR: 200316
Sponsored by: FreeBSD Mastery: Advanced ZFS

MFC r291545: make the number of fasttrap probes and the size of the trace points hash table
tunable via sysctl or kernel tunables.

MFC r279437,r284107:

r279437 (by rstone):

Allow Illumos code to co-exist with nv(9)

r284107 (by avg):

compat nvpair.h: make sure that the names are mangled only for kernel

Currently there is no good reason to mangle the userland API.
The change was introduced in eac1d566b46edef765754203bef22c75c1699966,
r279437. Also see https://reviews.freebsd.org/D1881.

I am still convinced that nv should not have introduced intentionally
conflicting API.

MFC nv(3) and part of nv(9) to stable/10

This includes the following revisions from head:

r258065,r258594,r259430,r260222,r261407,r261408,r263479,r264021,r266351,
r269603,r271026,r271027,r271028,r271241,r271578,r271579,r271847,r272102,
r272843,r273752,r277920,r277921,r277925,r277926,r277927,r279421,r279422,
r279423,r279424,r279425,r279426,r279427,r279428,r279429,r279430,r279431,
r279432,r279434,r279435,r279436,r279438,r279439,r279440,r279760,r282122,
r282254,r282257,r282304,r282312,r285339,r288340

This change reverts stable/10@r282122 and stable/10@r288340, and re-MFCs the
series again (r282122, r285339, and r288340).

More changes are pending to nv(9)/pci(4) after further review/work. Please see
the Phabricator review for more details (both https://reviews.freebsd.org/D4232
and https://reviews.freebsd.org/D4249 ).

- Tested with:
-- Booting VMware Fusion 8.1.0 running on a Haswell Apple Macbook Pro
-- Booting a Haswell machine with zfs and running some stress workloads with
VirtualBox guests
-- make tinderbox
-- kyua test -k /usr/tests/lib/libnv

Differential Revision: https://reviews.freebsd.org/D4249 (part of a larger diff)
Relnotes: yes
Reviewed by: oshogbo (implicit), sbruno (implicit)
Submitted by: Kevin Bowling <kevin.bowling@kev009.com>
Sponsored by: EMC / Isilon Storage Division

MFC r289195:

Integrate the tests from lib/libarchive, usr.bin/cpio, and usr.bin/tar in to
the FreeBSD test suite

functional_test.sh was ported from bin/sh/tests/functional_test.sh, as a
small wrapper around libarchive_test, bsdcpio_test, and bsdtar_test provided
by upstream.

A handful of testcases in lib/libarchive/tests have been disabled as they
were failing when run with kyua test (see BROKEN_TESTS in
lib/libarchive/tests/Makefile)

As a sidenote: this removes the check/test targets from the Makefiles as they
don't match the pattern used in the rest of the FreeBSD test suite.

Sponsored by: EMC / Isilon Storage Division

Conflicts:
lib/libarchive/test
usr.bin/cpio/test

MFC r290191 (by avg):
l2arc: do not call trim_map_free() for blocks with zero b_asize

b_asize can be zero if the block is compressed into an empty block
(ZIO_COMPRESS_EMPTY) and the trim code asserts that meaningless
zero-sized trimming is not attempted.
The logic for calling trim_map_free() is extracted into a new function
l2arc_trim() to minimize code duplication.

PR: 203473
Reported by: Willem Jan Withagen <wjw@digiware.nl>
Tested by: Willem Jan Withagen <wjw@digiware.nl>

MFC r289562: 6328 Fix cstyle errors in zfs codebase

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed by: Jorgen Lundman <lundman@lundman.net>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Paul Dagnelie <pcd@delphix.com>

illumos/illumos-gate@9a686fbc186e8e2a64e9a5094d44c7d6fa0ea167

MFC r289527:
5561 support root pools on EFI/GPT partitioned disks
5125 update zpool/libzfs to manage bootable whole disk pools (EFI/GPT labeled disks)

Reviewed by: Jean McCormack <jean.mccormack@nexenta.com>
Reviewed by: Josef 'Jeff' Sipek <josef.sipek@nexenta.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Hans Rosenfeld <hans.rosenfeld@nexenta.com>

illumos/illumos-gate@1a902ef8628b0dffd6df5442354ab59bb8530962

This is NOP changes for FreeBSD.

MFC r289422:
4185 add new cryptographic checksums to ZFS: SHA-512, Skein, Edon-R

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Saso Kiselkov <saso.kiselkov@nexenta.com>
Reviewed by: Richard Lowe <richlowe@richlowe.net>
Approved by: Garrett D'Amore <garrett@damore.org>
Author: Matthew Ahrens <mahrens@delphix.com>

illumos/illumos-gate@45818ee124adeaaf947698996b4f4c722afc6d1f

This is only a partial merge of respective ZFS infrastructure changes.
At this moment FreeBSD kernel has no those crypto algorithms, so the
parts of the code to enable them are commented out. When they are
implemented, it will be trivial to plug them in.

MFC r289362, r289445: 2605 want to resume interrupted zfs send

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed by: Xin Li <delphij@freebsd.org>
Reviewed by: Arne Jansen <sensille@gmx.net>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Matthew Ahrens <mahrens@delphix.com>

illumos/illumos-gate@9c3fd1216fa7fb02cfbc78a2518a686d54b48ab8

For more info, see:
- slides http://www.slideshare.net/MatthewAhrens/openzfs-send-and-receive
- video https://www.youtube.com/watch?v=iY44jPMvxog
- manpage changes (for zfs resume -s and zfs send -t)
- upcoming talk at the OpenZFS Developer Summit

The TL;DR is:
Use "zfs receive -s" to save the partially received state on failure.
On failure, get the receive token with "zfs get receive_resume_token <fs>"
Resume the send with "zfs send -t <token_value>"

Relnotes: yes

MFC r289309: 6267 dn_bonus evicted too early

Reviewed by: Richard Yao <ryao@gentoo.org>
Reviewed by: Xin LI <delphij@freebsd.org>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: Justin T. Gibbs <gibbs@FreeBSD.org>

illumos/illumos-gate@d2058105c61ec61df3a2dd3f839fed8c3fe7bfd6

MFC r289307: 6295 metaslab_condense's dbgmsg should include vdev id

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Andriy Gapon <avg@freebsd.org>
Reviewed by: Xin Li <delphij@freebsd.org>
Reviewed by: Justin Gibbs <gibbs@scsiguy.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: Joe Stein <joe.stein@delphix.com>

illumos/illumos-gate@daec38ecb4fb5e73e4ca9e99be84f6b8c50c02fa

MFC r289305: 6293 ztest failure: error == 28 (0xc == 0x1c) in ztest_tx_assign()

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: Matthew Ahrens <mahrens@delphix.com>

illumos/illumos-gate@8fe00bfb8790ad51653f67b01d5ac14256cbb404

MFC r289299: 6286 ZFS internal error when set large block on bootfs

Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Andriy Gapon <avg@FreeBSD.org>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Matthew Ahrens <mahrens@delphix.com>

illumos/illumos-gate@6de9bb5603e65b16816b7ab29e39bac820e2da2b

MFC r289297: 6288 dmu_buf_will_dirty could be faster

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Justin Gibbs <gibbs@scsiguy.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Matthew Ahrens <mahrens@delphix.com>

illumos/illumos-gate@0f2e7d03b8f588387cb8dd8dd500cbe5ff4484e0

MFC r289295: 5219 l2arc_write_buffers() may write beyond target_sz

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Saso Kiselkov <skiselkov@gmail.com>
Reviewed by: George Wilson <george@delphix.com>
Reviewed by: Steven Hartland <steven.hartland@multiplay.co.uk>
Reviewed by: Justin Gibbs <gibbs@FreeBSD.org>
Approved by: Matthew Ahrens <mahrens@delphix.com>
Author: Andriy Gapon <avg@freebsd.org>

illumos/illumos-gate@d7d9a6d919f92d74ea0510a53f8441396048e800

MFC r289192: 6281 prefetching should apply to 1MB reads

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Alexander Motin <mav@freebsd.org>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Reviewed by: Justin Gibbs <gibbs@scsiguy.com>
Reviewed by: Xin Li <delphij@freebsd.org>
Approved by: Gordon Ross <gordon.ross@nexenta.com>
Author: George Wilson <george.wilson@delphix.com>

illumos/illumos-gate@632802744ef6d17e06d6980a95f631615c3b060f

MFC r289191, r289194: 6251 add tunable to disable free_bpobj processing

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Simon Klinkert <simon.klinkert@gmail.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed by: Albert Lee <trisk@omniti.com>
Reviewed by: Xin Li <delphij@freebsd.org>
Approved by: Garrett D'Amore <garrett@damore.org>
Author: George Wilson <george.wilson@delphix.com>

illumos/illumos-gate@139510fb6efa97dbe5f5479594b308d940cab8d1

MFC r289190: 6250 zvol_dump_init() can hold txg open

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Albert Lee <trisk@omniti.com>
Reviewed by: Xin Li <delphij@freebsd.org>
Approved by: Garrett D'Amore <garrett@damore.org>
Author: George Wilson <george.wilson@delphix.com>

illumos/illumos-gate@b10bba72460aeaa53119c76ff5e647fd5585bece

MFC r287745: 5997 FRU field not set during pool creation and never
updated

ZFS already supports storing the vdev FRU in a vdev property. There
is code in libzfs to work with this property, and there is code in
the zfs-retire FMA module that looks for that information. But there
is no code actually setting or updating the FRU.

To address this, ZFS is changed to send a handful of new events
whenever a vdev is added, attached, cleared, or onlined, as well
as when a pool is created or imported.

Note that syseventd is not currently available on FreeBSD and thus
some work is needed to actually support the new ZFS events (e.g. in
zfsd) to actually use this capability, this changeset is mostly a
diff reduction from upstream.

illumos/illumos-gate@1437283407f89cab03860accf49408f94559bc34

Illumos issues:

5997 FRU field not set during pool creation and never updated
https://www.illumos.org/issues/5997

MFC r290401 & r290466

Provide information about bad DVA

Sponsored by: Multiplay

MFC r290399:

Allow zfs_recover to be changed at runtime

Sponsored by: Multiplay

MFC r276450

Correct zpool list displaying invalid EXPANDSZ for unavailable pool vdevs

Sponsored by: Multiplay

MFC r288363: std: it is important that func name is never an empty string

MFC r288340: define aok in libnvpair which is linked to all zfs
libraries that need aok

MFC r288362: sdt: start checking version field when parsing probe definitions

MFC r288365: sdt: static-ize couple of variables

MFC r288364: sdt module does not seem to actually use any symbol from
opensolaris module

MFC r288361: dtrace_getarg: remove stray return statement on amd64, powerpc

MFC r288204: MFV r288063:

make dataset property de-registration operation O(1)

MFC r288579: Restore original array_rd_sz semantics.

Before r278702 prefetch was blocked for I/Os > 1MB, after -- >= 1MB.
1MB I/Os are used for bulk operations in CTL (XCOPY, VERIFY), and disabling
prefetch for them reduced the performance.

MFC r288064 (by avg): 6220 memleak in l2arc on debug build

illumos/illumos-gate/commit/c546f36aa898d913ff77674fb5ff97f15b2e08b4
https://www.illumos.org/issues/6220
5408 introduced a memleak in l2arc, namely the member b_thawed gets leaked
when an arc_hdr is realloced from full to l2only.

Reviewed by: Saso Kiselkov <saso.kiselkov@nexenta.com>
Reviewed by: Simon Klinkert <simon.klinkert@gmail.com>
Reviewed by: George Wilson <george@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Arne Jansen <sensille@gmx.net>

MFC r287744 (by delphij): Reduce diff against upstream.

MFC r287706 (by delphij):
6214 zpools going south

In r286570 (MFV of r277426) an unprotected write to b_flags to
set the compression mode was introduced. This would open a race
window where data is partially decompressed, modified, checksummed
and written to the pool, resulting in pool corruption due to the
partial decompression.

Prevent this by reintroducing b_compress

illumos/illumos-gate@d4cd038c92c36fd0ae35945831a8fc2975b5272c

Illumos issues:

6214 zpools going south
https://www.illumos.org/issues/6214

MFV r287703, r287705 (by delphij):
6091 avl_add doesn't assert on non-debug builds

Use assfail() from libuutil instead of ASSERT() in userland
AVL avl_add.

illumos/illumos-gate@faa2b6be2fc102adf9ed584fc1a667b4ddf50d78

Illumos issues:

6091 avl_add doesn't assert on non-debug builds
https://www.illumos.org/issues/6091

MFC r287702: 5987 zfs prefetch code needs work

Rewrite the ZFS prefetch code to detect only forward, sequential
streams.

The following kstats have been added:

kstat.zfs.misc.arcstats.sync_wait_for_async

How many sync reads have waited for async read
to complete. (less is better)

kstat.zfs.misc.arcstats.demand_hit_predictive_prefetch

How many demand read didn't have to wait for I/O
because of predictive prefetch. (more is better)

zfetch kstats have been similified to hits, misses, and max_streams,
with max_streams representing times when we were not able to create
new stream because we already have the maximum number of sequences
for a file.

The sysctl variable/loader tunable vfs.zfs.zfetch.block_cap have been
replaced by vfs.zfs.zfetch.max_distance, which controls maximum bytes
to prefetch per stream.

illumos/illumos-gate@cf6106c8a0d6598b045811f9650d66e07eb332af

Illumos ZFS issues:

5987 zfs prefetch code needs work
https://www.illumos.org/issues/5987

MFC r287283 (by delphij):
Fix a buffer overrun which may lead to data corruption, introduced in
r286951 by reinstating changes in r274628.

In l2arc_compress_buf(), we allocate a buffer to stash away the compressed
data in 'cdata', allocated of l2hdr->b_asize bytes.

We then ask zio_compress_data() to compress the buffer, b_l1hdr.b_tmp_cdata,
which is of l2hdr->b_asize bytes, and have the compressed size (or original
size, if compress didn't gain enough) stored in csize.

To pad the buffer to fit the optimal write size, we round up the compressed
size to L2 device's vdev_ashift.

Illumos code rounds up the size by at most SPA_MINBLOCKSIZE. Because we
know csize <= b_asize, and b_asize is integer multiple of SPA_MINBLOCKSIZE,
we are guaranteed that the rounded up csize would be <= b_asize. However,
this is not necessarily true when we round up to 1 << vdev_ashift, because
it could be larger than SPA_MINBLOCKSIZE.

So, in the worst case scenario, we are overwriting at most

(1 << vdev_ashift - SPA_MINBLOCKSIZE)

bytes of memory next to the compressed data buffer.

Andriy's original change in r274628 reorganized the code a little bit,
by moving the padding to after we determined that the compression was
beneficial. At which point, we would check rounded size against the
allocated buffer size, and the buffer overrun would not be possible.

MFC r287280 (by delphij):
In r286705 (Illumos 5960/a2cdcdd), a separate thread is created with curproc
as parent. In the case of a send or receive, the curproc would be the
userland application that issues the ioctl. This would trigger an assertion
failure introduced in Solaris compatibility shims in r196458 when kernel is
compiled with INVARIANTS.

Fix this by using p0 (proc0 or kernel) as the parent thread when creating
the kernel threads.

MFC r287103 (by avg): 5692 expose the number of hole blocks in a file

FreeBSD porting notes:
- only kernel-side changes are merged
- the new ioctl is not actually implemented yet
- thus, the goal is to synchronize DMU code

illumos/illumos-gate@2bcf0248e992f292c7b814458bcdce2f004925d6

https://www.illumos.org/issues/5692
we would like to expose the number of hole (sparse) blocks in a file.
this can be useful to for example if you want to fill in the holes with
some data; knowing the number of holes in advances allows you to report
progress on hole filling. We could use SEEK_HOLE to do that but it would
be O(n) where n is the number of holes present in the file.

Author: Max Grossman <max.grossman@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Boris Protopopov <bprotopopov@hotmail.com>
Approved by: Richard Lowe <richlowe@richlowe.net>

MFC r286983 (by avg):
fix a mismerge in r286539 (MFV 286538: 5562 ZFS sa_handle's violate...)

MFC r286951: Restore part of r274628, reverted at r286776.

MFC r286776: Remove some random accumulated diff from Illumos.

MFC r286774: 2618 arc.c mistypes in the comments

Reviewed by: Jason King <jason.brian.king@gmail.com>
Reviewed by: Josef Sipek <jeffpc@josefsipek.net>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: Bart Coddens <bart.coddens@gmail.com>

illumos/illumos-gate@fc98fea58e89224f6f13d7fae246d6cb5dfa35ea

MFC r286770: Fix r286766 build with debug.

MFC r286767: Fix minor mismerge sometimes earlier.

MFC r286766: 5817 change type of arcs_size from uint64_t to refcount_t

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <paul.dagnelie@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Reviewed by: Richard Elling <richard.elling@richardelling.com>
Approved by: Garrett D'Amore <garrett@damore.org>
Author: Prakash Surya <prakash.surya@delphix.com>

illumos/illumos-gate@2fd872a734cf486007a8dba532cec52bfb4d40e5

As a way to make it more difficult to introduce bugs into the ARC, and to
make it easier to diagnose issues when bugs do creep in, it would be
beneficial to change the type of the arc_state_t's arcs_size field to be
a refcount_t instead of a uint64_t. This would allow us to make stricter
checks when incrementing and decrementing the value with debugging enabled,
but still fallback to simple, fast atomic operations when debugging is
disabled.

MFC r286764: 6033 arc_adjust() should search MFU lists for oldest buffer
when adjusting MFU size.

illumos/illumos-gate@31c46cf23cd1cf4d66390a983dc5072d7d299ba2

https://www.illumos.org/issues/6033
When we're looking for the list containing oldest buffer we never
actually look at the MFU lists even when we try to evict from MFU.
looks like a copy paste error, the fix is here:

Reviewed by: Saso Kiselkov <saso.kiselkov@nexenta.com>
Reviewed by: Xin Li <delphij@delphij.net>
Reviewed by: Prakash Surya <me@prakashsurya.com>
Approved by: Matthew Ahrens <mahrens@delphix.com>
Author: Alek Pinchuk <alek@nexenta.com>
Obtained from: illumos

MFC r286763: 5497 lock contention on arcs_mtx

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Richard Elling <richard.elling@richardelling.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Prakash Surya <prakash.surya@delphix.com>

illumos/illumos-gate@244781f10dcd82684fd8163c016540667842f203

This patch attempts to reduce lock contention on the current arc_state_t
mutexes. These mutexes are used liberally to protect the number of LRU
lists within the ARC (e.g. ARC_mru, ARC_mfu, etc). The granularity at
which these locks are acquired has been shown to greatly affect the
performance of highly concurrent, cached workloads.

MFC r286762: Revert part of r205231, introducing multiple ARC state locks.

This local implementation will be replaced by one from Illumos to reduce
code divergence and make further merges easier.

Fix r288549 build on stable.

For some reason this (odd) code builds on head, but not on stable.

MFC r286712: 6096 ZFS_SMB_ACL_RENAME needs to cleanup better

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Gordon Ross <gordon.w.ross@gmail.com>
Reviewed by: George Wilson <gwilson@zfsmail.com>
Approved by: Robert Mustacchi <rm@joyent.com>

illumos/illumos-gate@8f5190a540d64d2debee6664bbc740e4c38f5b7f

MFC r286710:
6093 zfsctl_shares_lookup should only VN_RELE() on zfs_zget() success

Reviewed by: Gordon Ross <gwr@nexenta.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Dan McDonald <danmcd@omniti.com>

illumos/illumos-gate@0f92170f1ec2737ee5a0e51b5f74093904811452

MFC r286708: 5959 clean up per-dataset feature count code

Reviewed by: Toomas Soome <tsoome@me.com>
Reviewed by: George Wilson <george@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: Matthew Ahrens <mahrens@delphix.com>

illumos/illumos-gate@ca0cc3918a1789fa839194af2a9245f801a06b1a

A ZFS feature flags (large blocks) tracks its refcounts as the number of
datasets that have ever used the feature. Several features of this type
are planned to be added (new checksum functions). This code should be made
common infrastructure rather than duplicating the code for each feature.

MFC r286705: 5960 zfs recv should prefetch indirect blocks
5925 zfs receive -o origin=

Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Author: Paul Dagnelie <pcd@delphix.com>

While running 'zfs recv' we noticed that every 128th 8K block required a
read. We were seeing that restore_write() was calling dmu_tx_hold_write()
and the indirect block was not cached. We should prefetch upcoming indirect
blocks to avoid having to go to disk and blocking the restore_write().

Allow an incremental send stream to be received as a clone, even if the
stream does not mark it as a clone.

MFC r286689: 5981 Deadlock in dmu_objset_find_dp

illumos/illumos-gate@1d3f896f5469c69c1339890ec3d68e9feddb0343

https://www.illumos.org/issues/5981
When dmu_objset_find_dp gets called with a read lock held, it fans out
the work to the task queue. Each task in turn acquires its own read
lock before calling the callback. If during this process anyone tries
to a acquire a write lock, it will stall all read lock requests.Thus
the tasks will never finish, the read lock of the caller will never
get freed and the write lock never acquired. deadlock.

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Arne Jansen <jansen@webgods.de>

MFC r286686: 5269 zpool import slow

illumos/illumos-gate@12380e1e701fda28c9e9f32d01cafb54af279eb5

https://www.illumos.org/issues/5269
When importing a pool (at boot or with zpool import) with many
filesystem, the process can take minutes. It doesn't matter whether
the pool has been exported cleanly or uncleanly. The problem is that
each dataset has its own log chain. On import, all datasets have to be
checked if there are logs to replay. The idea is to speed up this
process by paralellizing it.

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george@delphix.com>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Arne Jansen <jansen@webgods.de>

MFC r286683: 5765 add support for estimating send stream size with
lzc_send_space when source is a bookmark

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Steven Hartland <killing@multiplay.co.uk>
Reviewed by: Bayard Bell <buffer.g.overflow@gmail.com>
Approved by: Albert Lee <trisk@nexenta.com>
Author: Max Grossman <max.grossman@delphix.com>

illumos/illumos-gate@643da460c8ca583e39ce053081754e24087f84c8

MFC r286677: 5695 dmu_sync'ed holes do not retain birth time

illumos/illumos-gate@70163ac57e58ace1c5c94dfbe85dca5a974eff36

https://www.illumos.org/issues/5695
In dmu_sync_ready(), a hole block pointer will have it's logical size
explicitly set as it's necessary for replay purposes. To "undo" this,
dmu_sync_done() will zero out any hole that it finds. This becomes a
problem when using the "hole_birth" feature, as this will also wipe out
any birth time that might have happened to be set on the hole.
...
As a fix, the logic to zero out a hole is only applied to old style
holes with a birth time of zero. Holes created with the "hole_birth"
feature enabled will have a non-zero birth time, and will be skipped
(thus preserving the ltime, type, and level information as well).
In addition, zdb was updated to also print the ltime, type, and level
information for these new style holes. Previously, only the logical
birth time would be printed.

Author: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Bayard Bell <buffer.g.overflow@gmail.com>
Approved by: Dan McDonald <danmcd@omniti.com>

MFC r286655: Fix set of sign extension bugs in r286625.

MFC r286647: Fix assertion panic caused by combination of r286598 and TRIM.

MFC r286628: Fix r286625 build on i386.

MFC r286626: Fix minor mismerge in r286574.

MFC r286625:
5376 arc_kmem_reap_now() should not result in clearing arc_no_grow

Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Steven Hartland <killing@multiplay.co.uk>
Reviewed by: Richard Elling <richard.elling@richardelling.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Matthew Ahrens <mahrens@delphix.com>

illumos/illumos-gate@2ec99e3e987d8aa273f1e9ba2b983557d058198c

MFC r286623: Remove extra lock, that IMO only creates potential problems now.

MFC r286605: 5812 assertion failed in zrl_tryenter(): zr_owner==NULL

Reviewed by: George Wilson <george@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Reviewed by: Will Andrews <will@freebsd.org>
Approved by: Gordon Ross <gwr@nexenta.com>
Author: Matthew Ahrens <mahrens@delphix.com>

illumos/illumos-gate@8df173054ca442cd8845a7364c3edad9d6822351

MFC r286603: 5810 zdb should print details of bpobj

Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Reviewed by: George Wilson <george@delphix.com>
Reviewed by: Will Andrews <will@freebsd.org>
Reviewed by: Simon Klinkert <simon.klinkert@gmail.com>
Approved by: Gordon Ross <gwr@nexenta.com>
Author: Matthew Ahrens <mahrens@delphix.com>

illumos/illumos-gate@732885fca09e11183dd0ea69aaaab5588fb7dbff

MFC r286600: 5808 spa_check_logs is not necessary on readonly pools

Reviewed by: George Wilson <george@delphix.com>
Reviewed by: Paul Dagnelie <paul.dagnelie@delphix.com>
Reviewed by: Simon Klinkert <simon.klinkert@gmail.com>
Reviewed by: Will Andrews <will@freebsd.org>
Approved by: Gordon Ross <gwr@nexenta.com>
Author: Matthew Ahrens <mahrens@delphix.com>

illumos/illumos-gate@23367a2f2caec1ccb4d918bdd0f2fc2c9cadcd06

MFC r286598: 5701 zpool list reports incorrect "alloc" value for cache devices

MFC r286593: Local addition and mismerge fix for r286579.

MFC r286589: 5820 verify failed in zio_done(): BP_EQUAL(bp, io_bp_orig)

Reviewed by: Alex Reece <alex@delphix.com>
Reviewed by: George Wilson <george@delphix.com>
Reviewed by: Steven Hartland <killing@multiplay.co.uk>
Approved by: Garrett D'Amore <garrett@damore.org>
Author: Matthew Ahrens <mahrens@delphix.com>

illumod/illumos-gate@34e8acef009195effafdcf6417aec385e241796e

MFC r286587: 5746 more checksumming in zfs send

Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Bayard Bell <buffer.g.overflow@gmail.com>
Approved by: Albert Lee <trisk@omniti.com>
Author: Matthew Ahrens <mahrens@delphix.com>

illumos/illumos-gate@98110f08fa182032082d98be2ddb9391fcd62bf1

MFC r286579: 5313 Allow I/Os to be aggregated across ZIO priority classes
Reviewed by: Andriy Gapon <avg@FreeBSD.org>
Reviewed by: Will Andrews <willa@SpectraLogic.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Justin T. Gibbs <justing@spectralogic.com>

illumos/illumos-gate@fe319232d24f4ae183730a5a24a09423d8ab4429

MFC r286578: Fix r286574 build in user-space.

MFC r286576: Fix r286570 build with debug.

MFC r286575: 5056 ZFS deadlock on db_mtx and dn_holds

Reviewed by: Will Andrews <willa@spectralogic.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Justin Gibbs <justing@spectralogic.com>

illumos/illumos-gate@bc9014e6a81272073b9854d9f65dd59e18d18c35

MFC r286574: 5445 Add more visibility via arcstats; specifically
arc_state_t stats and differentiate between "data" and "metadata"

Reviewed by: Basil Crow <basil.crow@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Bayard Bell <bayard.bell@nexenta.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Prakash Surya <prakash.surya@delphix.com>

illumos/illumos-gate@4076b1bf41cfd9f968a33ed54a7ae76d9e996fe8

MFC r286570: 5408 managing ZFS cache devices requires lots of RAM
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Don Brady <dev.fs.zfs@gmail.com>
Reviewed by: Josef 'Jeff' Sipek <josef.sipek@nexenta.com>
Approved by: Garrett D'Amore <garrett@damore.org>
Author: Chris Williamson <Chris.Williamson@delphix.com>

illumos/illumos-gate@89c86e32293a30cdd7af530c38b2073fee01411c

Currently, every buffer cached in the L2ARC is accompanied by a 240-byte
header in memory, leading to very high memory consumption when using very
large cache devices. These changes significantly reduce this overhead.

Currently:

L1-only header = 176 bytes
L1 + L2 or L2-only header = 176 bytes + 32 byte checksum + 32 byte l2hdr
= 240 bytes

Memory-optimized:

L1-only header = 176 bytes
L1 + L2 header = 176 bytes + 32 byte checksum = 208 bytes
L2-only header = 96 bytes + 32 byte checksum = 128 bytes

So overall:

Trunk Optimized
+-----------------+
L1-only | 176 B | 176 B | (same)
+-----------------+
L1 & L2 | 240 B | 208 B | (saved 32 bytes)
+-----------------+
L2-only | 240 B | 128 B | (saved 116 bytes)
+-----------------+

For an average blocksize of 8KB, this means that for the L2ARC, the ratio
of metadata to data has gone down from about 2.92% to 1.56%. For a
'storage optimized' EC2 instance with 1600GB of SSD and 60GB of RAM, this
means that we expect a completely full L2ARC to use (1600 GB * 0.0156) /
60GB = 41% of the available memory, down from 78%.

Relnotes: yes