MFC r363447:
MFOpenZFS: Fix zpool history unbounded memory usage

PR: 247557

MFC r362396
ZFS: Allow setting checksum=skein on boot pools

PR: 245889
Reported by: delphij
Sponsored by: Klara Inc.
Event: July 2020 Bugathon

MFC r360080:

zfs-program.8: fix orphan .Xr

Reported by: phk
Reviewed by: avg
Differential Revision: https://reviews.freebsd.org/D24488

MFC -fno-common fixes: r359389, r359394, r359397-r359399, r359403-r359404,
r359406, r359413-r359416, r359425, r359427, r359432-r359433, r359443,
r359675-r359677

Note: this is not necessarily a complete fix to get these programs to build
with -fno-common applied.

r359389:
config(8): fixes for -fno-common

Move this handful of definitions into main.c, properly declare these as
extern in config.h. This fixes the config(8) build with -fno-common.

Unexplained in my previous commit to gas, -fno-common will become the
default in GCC10 and LLVM11, so it's worth addressing these in advance.

r359394:
MFV r359393: tcsh: import 6974bc35a5cd

This removes an extra variable definition that causes the -fno-common build
to fail, which will be a new default in GCC10/LLVM11.

r359397:
zfs: fix -fno-common issues

A similar (or identical?) fix has already landed in OpenZFS.

-fno-common will become the default in GCC10/LLVM11.

r359398:
sh: remove duplicate el definition

el is declared extern in myhistedit.h and defined in histedit.c. Remove the
duplicate definition in input.c to appease the -fno-common build.

-fno-common will become the default in GCC10/LLVM11.

r359399:
telnet: remove some duplicate definitions, mark terminaltype extern

Most of these were already properly declared and defined elsewhere, this is
effectively just a minor cleanup that fixes the -fno-common build.

-fno-common will become the default in GCC10/LLVM11.

r359403:
Revert 359399: telnet -fno-common bits

There was a large misfire from my local diff that I need to investigate, and
this version committed did not build.

r359404:
Re-apply r359399: telnet -fno-common fix

line and auth_level's redefinitions are just extraneous

telnetd will #define extern and then include ext.h to allocate storage for
all of these extern'd vars; however, two of them are actually defined in
libtelnet instead. Instead of doing an #ifdef extern dance around those
function pointers, just add an EXTERN macro to make it easier to
differentiate by sight which ones will get allocated in globals.c and which
ones are defined elsewhere.

r359406:
telnet: kill off remaining duplicate definition

r359413:
ipfilter: remove duplicate definition of 'thishost'

thishost is already defined in lib/initparse.c; no need for this one. This
fixes the ipfilter build with -fno-common.

-fno-common will become the default in GCC10/LLVM11.

r359414:
iscontrol: move definition of vflag/iscsidev to iscontrol.c

Mark the declaration extern as these are used elsewhere; this fixes the
build with -fno-common.

r359415:
userboot: mark host_fsops as extern

This is already defined elsewhere; mark this declaration extern to the fix
the -fno-common build.

r359416:
systat: remove redundant definition of kd

kd is already properly declared in extern.h and defined in main.c, rendering
this definition useless. This fixes the -fno-common build.

r359425:
locate: fix -fno-common build

Just a single variable declaration to extern and define elsewhere here,
myctype.

-fno-common will become a default in GCC10/LLVM11.

r359427:
fsck_ffs/fsdb: fix -fno-common build

This one is also a small list:

- 3x duplicate definition (ufs2_zino, returntosingle, nflag)
- 5x 'needs extern', 3/5 of which are referenced in fsdb

-fno-common will become the default in GCC10/LLVM11.

r359432:
gdb: compile with -fcommon explicitly

As described in the comment, gdb relies on some of the linker magic that
happens with -fcommon. I suspect the life expectancy of gdb-in-base is low
enough that this isn't worth spending much time addressing, especially given
the vintage. Hit it with the -fcommon hammer so that it continues to just
work.

r359433:
bmake: fix -fno-common build

debug was declared extern, but debug_file was not; correct this and define
debug_file in main.c (as debug is) to fix the -fno-common build.

-fno-common will become the default with GCC10/LLVM11.

r359443:
MFV r359442: bmake: import -fno-common fix build back from upstream

sjg@ committed the local patch previously committed upstream; pull it in to
vendor/ to ease any potential stress of future imports.

r359675:
kqueue tests: fix -fno-common build

vnode_fd and kqfd are both shared among multiple CU; define them exactly
once.

In the case of vnode_fd, it was simply the declaration that needed
correction.

-fno-common will become the default in GCC10/LLVM11.

r359676:
ntpd: fix build with -fno-common

Only a small nit here: psl should be declared extern and defined exactly
once.

-fno-common will become the default in GCC10/LLVM11.

r359677:
yp*: fix -fno-common build

This is mostly two problems spread out far and wide:
- ypldap_process should be declared properly
- debug is defined differently in many programs

For the latter, just extern it and define it everywhere that actually needs
it. This mostly works out nicely for ^/libexec/ypxfr, which can remove the
assignment at the beginning of main in favor of defining it properly.

-fno-common will become the default in GCC10/LLVM11.

MFC r359112: MFOpenZFS: make zil max block size tunable

We've observed that on some highly fragmented pools, most metaslab
allocations are small (~2-8KB), but there are some large, 128K
allocations. The large allocations are for ZIL blocks. If there is a
lot of fragmentation, the large allocations can be hard to satisfy.

The most common impact of this is that we need to check (and thus load)
lots of metaslabs from the ZIL allocation code path, causing sync writes
to wait for metaslabs to load, which can take a second or more. In the
worst case, we may not be able to satisfy the allocation, in which case
the ZIL will resort to txg_wait_synced() to ensure the change is on
disk.

To provide a workaround for this, this change adds a tunable that can
reduce the size of ZIL blocks.

External-issue: DLPX-61719
Reviewed-by: George Wilson <george.wilson@delphix.com>
Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Matthew Ahrens <mahrens@delphix.com>
Closes #8865
openzfs/zfs@b8738257c2607c73c731ce8e0fd73282b266d6ef

MFC r354062: zfs.8: fix a typo in the date

MFC r353607: MFV r353606: 10067 Miscellaneous man page typos

MFC r353624: MFV r353623: 10473 zfs(1M) missing cross-reference to zfs-program(1M)

MFC r353037: ZFS: add bookmark renaming

MFC r353649: fix section number in zfs-program.8

MFC r352447,r352449,r352507: MFZoL: Add -vnP support to 'zfs send' for bookmarks

MFC r350897: MFV r350896: 6585 sha512, skein, and edonr have an
unenforced dependency on extensible dataset

MFC r350893: Allow ZVOL bookmarks to be listed recursively

PR: 197821

MFC r344360,r344361: zpool.8: document -D flag for zpool status

Also, sort zpool status flags in the same order as in illumos manual.

MFC r347953:

MFV/ZoL: `zfs userspace` ignored all unresolved UIDs after the first

zfsonlinux/zfs@88cfff182432e4d1c24c877f33b47ee6cf109eee

zfs_main: fix `zfs userspace` squashing unresolved entries

The `zfs userspace` squashes all entries with unresolved numeric
values into a single output entry due to the comparsion always
made by the string name which is empty in case of unresolved IDs.

Fix this by falling to a numerical comparison when either one
of string values is not found. This then compares any numerical
values after all with a name resolved.

Signed-off-by: Pavel Boldin <boldin.pavel@gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>

Reported by: clusteradm
Obtained from: ZFS-on-Linux

Approved by: re (gjb)

MFC r347240: Fix dataset name comparison in zfs_compare().

The code never returned match comparing two datasets (not snapshots).
As result, uu_avl_find(), called from zfs_callback(), never succeeded,
allowing to add same dataset into the list multiple times, for example:

# zfs get name pers pers pers@z pers@z
NAME PROPERTY VALUE SOURCE
pers name pers -
pers name pers -
pers@z name pers@z -

With the patch:

# zfs get name pers pers pers@z pers@z
NAME PROPERTY VALUE SOURCE
pers name pers -
pers@z name pers@z -

MFC r344569, r344618, r344621 (by bapt):

r344569:
Implement parallel mounting for ZFS filesystem

It was first implemented on Illumos and then ported to ZoL.
This patch is a port to FreeBSD of the ZoL version.
This patch also includes a fix for a race condition that was amended

With such patch Delphix has seen a huge decrease in latency of the mount phase
(https://github.com/openzfs/openzfs/commit/a3f0e2b569 for details).
With that current change Gandi has measured improvments that are on par with
those reported by Delphix.

Zol commits incorporated:
https://github.com/zfsonlinux/zfs/commit/a10d50f999511d304f910852c7825c70c9c9e303
https://github.com/zfsonlinux/zfs/commit/e63ac16d25fbe991a356489c86d4077567dfea21

Reviewed by: avg, sef
Approved by: avg, sef
Obtained from: ZoL
Relnotes: yes
Sponsored by: Gandi.net
Differential Revision: https://reviews.freebsd.org/D19098

r344618:
Fix regression introduced in r344569

Reported by: cy
Tested by: cy
Submitted by: Fatih Acar <fatih@gandi.net>

r344621:
Fix a regression introduced in r344569

Import a fix from illumos (thanks Toomas Soomas for pointing at it)

See https://www.illumos.org/issues/10205 for more details
Illumos commit: https://github.com/illumos/illumos-gate/commit/247b7da039fd88350c50e3d7fef15bdab6bef215

Submitted by: jack@gandi.net
Reported by: cy
Reviewed by: tsoome, cy, bapt
Obtained from: Illumos

MFC r344601 (by sef): Set process title during zfs send.

This adds a '-V' option to 'zfs send', which sets the process title once a
second to the progress information.

This code has been in FreeNAS for a long time now; this is just upstreaming
it here. It was originially written by delphij.

MFC r342541: MFV r342532: 5882 Temporary pool names

Note that this commit brings only formatting changes that were done
during the final review of the illumos change, because FreeBSD got the
main changes before illumos.

MFC r337215: MFV 337214:
9621 Make createtxg and guid properties public

illumos/illumos-gate@e8d4a73c868afb740396041be80ed2b141065e76

Reviewed by: Andy Stormont <astormont@racktopsystems.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Matt Ahrens <matt@delphix.com>
Reviewed by: Yuri Pankov <yuripv@yuripv.net>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Josh Paetzel <josh@tcbug.org>

MFC r337183:
MFV r337182: 9330 stack overflow when creating a deeply nested dataset

Datasets that are deeply nested (~100 levels) are impractical. We just put
a limit of 50 levels to newly created datasets. Existing datasets should
work without a problem.

illumos/illumos-gate@5ac95da7d61660aa299c287a39277cb0372be959

Reviewed by: John Kennedy <john.kennedy@delphix.com>
Reviewed by: Matt Ahrens <matt@delphix.com>
Approved by: Garrett D'Amore <garrett@damore.org>
Author: Serapheim Dimitropoulos <serapheim.dimitro@delphix.com>

MFC r337179: 9523 Large alloc in zdb can cause trouble

16MB alloc in zdb_embedded_block() can cause cores in certain situations
(clang, gcc55).

OsX commit: https://github.com/openzfsonosx/zfs/commit/ced236a5da6e72ea7bf6d2919fe14e17cffe10f1
FreeBSD commit: https://svnweb.freebsd.org/base?view=revision&revision=326150
illumos/illumos-gate@03a4c2f4bfaca30115963b76445279b36468a614

Reviewed by: Igor Kozhukhov <igor@dilos.org>
Reviewed by: Andriy Gapon <avg@FreeBSD.org>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Dan McDonald <danmcd@joyent.com>
Author: Jorgen Lundman <lundman@lundman.net>

This is an update for r326150 (by avg), where this change comes from.

MFC r337163: MFV r337161: 9512 zfs remap poolname@snapname coredumps

Only filesystems and volumes are valid "zfs remap" parameters: when passed
a snapshot name zfs_remap_indirects() does not handle the EINVAL returned
from libzfs_core, which results in failing an assertion and consequently
crashing.

illumos/illumos-gate@0b2e8253986c5c761129b58cfdac46d204903de1

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: John Wren Kennedy <john.kennedy@delphix.com>
Reviewed by: Sara Hartse <sara.hartse@delphix.com>
Approved by: Matt Ahrens <mahrens@delphix.com>
Author: loli10K <ezomori.nozomu@gmail.com>

MFC r337063: MFV r316926:
7955 libshare needs to initialize only those datasets being modified by the consumer

illumos/illumos-gate@8a981c3356b194b3b5c0ae9276a9cc31cd2f93a3
https://github.com/illumos/illumos-gate/commit/8a981c3356b194b3b5c0ae9276a9cc31cd2f93a3

https://www.illumos.org/issues/7955
Libshare currently initializes all available filesystems when doing any
libshare operation. This requires iterating through all the filesystem
multiple times, which is a huge performance problem for sharing and
unsharing operations.

Reviewed by: Steve Gonczi <steve.gonczi@delphix.com>
Reviewed by: Sebastien Roy <sebastien.roy@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: Yuri Pankov <yuri.pankov@gmail.com>
Approved by: Gordon Ross <gordon.w.ross@gmail.com>
Author: Daniel Hoffman <dj.hoffman@delphix.com>

For FreeBSD this is practically a NOP, just a diff reduction.

MFC r337017: MFV r337014:
9421 zdb should detect and print out the number of "leaked" objects
9422 zfs diff and zdb should explicitly mark objects that are on the deleted queue

illumos/illumos-gate@20b5dafb425396adaebd0267d29e1026fc4dc413

Reviewed by: Matt Ahrens <matt@delphix.com>
Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Approved by: Matt Ahrens <mahrens@delphix.com>
Author: Paul Dagnelie <pcd@delphix.com>

MFC r337007: MFV r336991, r337001:
9102 zfs should be able to initialize storage devices

The first access to a disk block can incur a performance penalty on some
platforms (e.g. AWS's EBS, VMware VMDKs). Therefore it is recommended that
volumes be "thick provisioned", where supported by the platform (VMware).
Thick provisioning is time consuming and often is ignored. If the thick
provision step is omitted, customers will see suboptimal performance until
we have written to all parts of the LUN. ZFS should be able to initialize
any unused storage to remove any first-write penalty that exists.

illumos/illumos-gate@094e47e980b0796b94b1b8f51f462a64d246e516

Reviewed by: John Wren Kennedy <john.kennedy@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: George Wilson <george.wilson@delphix.com>

MFC r336956: MFV r336955: 9236 nuke spa_dbgmsg

We should use zfs_dbgmsg instead of spa_dbgmsg. Or at least,
metaslab_condense() should call zfs_dbgmsg because it's important and rare
enough to always log. It's possible that the message in zio_dva_allocate()
would be too high-frequency for zfs_dbgmsg.

illumos/illumos-gate@21f7c81cc1156e9202ce3412d3ecaa697c3b2222

Reviewed by: Serapheim Dimitropoulos <serapheim.dimitro@delphix.com>
Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: Matthew Ahrens <mahrens@delphix.com>

MFC r336951: MFV r336950: 9290 device removal reduces redundancy of mirrors

Mirrors are supposed to provide redundancy in the face of whole-disk failure
and silent damage (e.g. some data on disk is not right, but ZFS hasn't
detected the whole device as being broken). However, the current device
removal implementation bypasses some of the mirror's redundancy.

illumos/illumos-gate@3a4b1be953ee5601bab748afa07c26ed4996cde6

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Prashanth Sreenivasa <pks@delphix.com>
Reviewed by: Sara Hartse <sara.hartse@delphix.com>
Reviewed by: Serapheim Dimitropoulos <serapheim@delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Tim Chase <tim@chase2k.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: Matthew Ahrens <mahrens@delphix.com>

MFC r336947: MFV r336946: 9238 ZFS Spacemap Encoding V2

The current space map encoding has the following disadvantages:
[1] Assuming 512 sector size each entry can represent at most 16MB for a segment.
This makes the encoding very inefficient for large regions of space.
[2] As vdev-wide space maps have started to be used by new features (i.e.
device removal, zpool checkpoint) we've started imposing limits in the
vdevs that can be used with them based on the maximum addressable offset
(currently 64PB for a top-level vdev).

The new remains backwards compatible with the old one. The introduced
two-word entry format, besides extending the limits imposed by the single-entry
layout, also includes a vdev field and some extra padding after its prefix.

The extra padding after the prefix should is reserved for future usage (e.g.
new prefixes for future encodings or new fields for flags). The new vdev field
not only makes the space maps more self-descriptive, but also opens the doors
for pool-wide space maps.

One final important note is that the number of bits used for vdevs is reduced
to 24 bits for blkptrs. That was decided as we don't know of any setups that
use more than 16M vdevs for the time being and
we wanted to fit the vdev field in the space map. In addition that gives us
some extra bits in dva_t.

illumos/illumos-gate@17f11284b49b98353b5119463254074fd9bc0a28

Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <gwilson@zfsmail.com>
Approved by: Gordon Ross <gwr@nexenta.com>
Author: Serapheim Dimitropoulos <serapheim@delphix.com>

MFC r336945: MFV r336944: 9286 want refreservation=auto

When a ZFS volume is created with zfs create -V (but without -s), the
refreservation property is set to a value that is volsize plus the maximum
size of metadata. If refreservation is ever set to another value, it is
impossible to set it back to the automatically determined value. There are
other cases where refreservation may be wrong. These include receiving a
volume that was sent without properties and zfs clone.

We need:

zfs set refreservation=auto <volume>
zfs clone -o refreservation=auto <volume>

Each one would use the same function used by zfs create -V to determine the
proper value for refreservation.

illumos/illumos-gate@1c10ae76c0cb31326c320e7cef1d3f24a1f47125

Reviewed by: Allan Jude <allanjude@freebsd.org>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: John Kennedy <john.kennedy@delphix.com>
Reviewed by: Andy Stormont <astormont@racktopsystems.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: Mike Gerdts <mike.gerdts@joyent.com>

MFC r334844, r336180, r336458

r334844

This originated from ZFS On Linux, as
https://github.com/zfsonlinux/zfs/commit/d4a72f23863382bdf6d0ae33196f5b5decbc48fd

During scans (scrubs or resilvers), it sorts the blocks in each transaction
group by block offset; the result can be a significant improvement. (On my
test system just now, which I put some effort to introduce fragmentation into
the pool since I set it up yesterday, a scrub went from 1h2m to 33.5m with the
changes.) I've seen similar rations on production systems.

r336180

Fix up some missed and mis-merges from the sequential scan code
(r334844). Most of the changes involve moving some code around to
reduce conflicts with future merges. One of the missing changes
included a notification on scrub cancellation.

r336458

Fix a couple of typos in r334844 noticed by Richard Kojedzinszky

Approved by: mav
Sponsored by: iXsystems, Inc

MFC r333081 (by eadler):
zpool(8): correct list of default properties in 'list'.

The default provides output in the following form:
```
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP
HEALTH ALTROOT
```

this corrects the man page.

Also submitted upstream as
https://github.com/openzfs/openzfs/pull/632/files (with slightly
different changes needed)

MFC r337926:
Add partial documentation for dtrace(1)'s -x configuration options.

MFC r323620:

Fix the raise tests.

MFC r332426: allow ZFS pool to have temporary name for duration of current import

The change adds -t <name> option to zpool create and -t option to zpool
import in its form with an old name and a new name. This allows to
import (or create) a pool under a name that's different from its real,
permanent name without affecting that name. This is useful when working
with VM images or images of other physical systems if they happen to
have a ZFS pool with the same name as the host system.

Sponsored by: Panzura (porting)

MFC r332641: MFV man pages update from r329502:
7614 zfs device evacuation/removal.

MFC r331713: MFV r331712:
9280 Assertion failure while running removal_with_ganging test with 4K devices

illumos/illumos-gate@243952c7eeef020886e3e2e3df99a513df40584a

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: John Kennedy <john.kennedy@delphix.com>
Approved by: Garrett D'Amore <garrett@damore.org>
Author: Matt Ahrens <Matt.Ahrens@delphix.com>

MFC r331707: MFV r331706:
9235 rename zpool_rewind_policy_t to zpool_load_policy_t

illumos/illumos-gate@5dafeea3ebd2dd77affc802bcb90f63faf01589f

We want to be able to pass various settings during import/open of a pool,
which are not only related to rewind. Instead of adding a new policy and
duplicate a bunch of code, we should just rename rewind_policy to a more
generic term like load_policy.

For instance, we'd like to set spa->spa_import_flags from the nvlist,
rather from a flags parameter passed to spa_import as in some cases we want
those flags not only for the import case, but also for the open case. One
such flag could be ZFS_IMPORT_MISSING_LOG (as used in zdb) which would
allow zfs to open a pool when logs are missing.

Reviewed by: Matt Ahrens <matt@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Pavel Zakharov <pavel.zakharov@delphix.com>

MFC r331701: MFV r331695, 331700: 9166 zfs storage pool checkpoint

illumos/illumos-gate@8671400134a11c848244896ca51a7db4d0f69da4

The idea of Storage Pool Checkpoint (aka zpool checkpoint) deals with
exactly that. It can be thought of as a “pool-wide snapshot” (or a
variation of extreme rewind that doesn’t corrupt your data). It remembers
the entire state of the pool at the point that it was taken and the user
can revert back to it later or discard it. Its generic use case is an
administrator that is about to perform a set of destructive actions to ZFS
as part of a critical procedure. She takes a checkpoint of the pool before
performing the actions, then rewinds back to it if one of them fails or puts
the pool into an unexpected state. Otherwise, she discards it. With the
assumption that no one else is making modifications to ZFS, she basically
wraps all these actions into a “high-level transaction”.

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: John Kennedy <john.kennedy@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: Serapheim Dimitropoulos <serapheim.dimitro@delphix.com>

MFC r331699: Partial MFV r329753:
8809 libzpool should leverage work done in libfakekernel

illumos/illumos-gate@f06dce2c1f0f3af78581e7574f65bfba843ddb6e

Reviewed by: Sebastien Roy <sebastien.roy@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Gordon Ross <gordon.w.ross@gmail.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: Andrew Stormont <astormont@racktopsystems.com>

We do not have libfakekernel, but need to reduce code divergence.

MFC r331408: MFV r331407: 9213 zfs: sytem typo

illumos/illumos-gate@edc8ef7d921c96b23969898aeb766cb24960bda7

Reviewed by: C Fraire <cfraire@me.com>
Reviewed by: Andy Fiddaman <omnios@citrus-it.co.uk>
Approved by: Joshua M. Clulow <josh@sysmgr.org>
Author: Toomas Soome <tsoome@me.com>

MFC r329798: MFV r329793, r329795:
9075 Improve ZFS pool import/load process and corrupted pool recovery

illumos/illumos-gate@6f7938128a2c5e23f4b970ea101137eadd1470a1

Some work has been done lately to improve the debugability of the ZFS pool
load (and import) process. This includes:

https://www.illumos.org/issues/7638: Refactor spa_load_impl into several functions
https://www.illumos.org/issues/8961: SPA load/import should tell us why it failed
https://www.illumos.org/issues/7277: zdb should be able to print zfs_dbgmsg's

To iterate on top of that, there's a few changes that were made to make the
import process more resilient and crash free. One of the first tasks during the
pool load process is to parse a config provided from userland that describes
what devices the pool is composed of. A vdev tree is generated from that config,
and then all the vdevs are opened.

The Meta Object Set (MOS) of the pool is accessed, and several metadata objects
that are necessary to load the pool are read. The exact configuration of the
pool is also stored inside the MOS. Since the configuration provided from
userland is external and might not accurately describe the vdev tree
of the pool at the txg that is being loaded, it cannot be relied upon to safely
operate the pool. For that reason, the configuration in the MOS is read early
on. In the past, the two configurations were compared together and if there was
a mismatch then the load process was aborted and an error was returned.

The latter was a good way to ensure a pool does not get corrupted, however it
made the pool load process needlessly fragile in cases where the vdev
configuration changed or the userland configuration was outdated. Since the MOS
is stored in 3 copies, the configuration provided by userland doesn't have to be
perfect in order to read its contents. Hence, a new approach has been adopted:
The pool is first opened with the untrusted userland configuration just so that
the real configuration can be read from the MOS. The trusted MOS configuration
is then used to generate a new vdev tree and the pool is re-opened.

When the pool is opened with an untrusted configuration, writes are disabled
to avoid accidentally damaging it. During reads, some sanity checks are
performed on block pointers to see if each DVA points to a known vdev;
when the configuration is untrusted, instead of panicking the system if those
checks fail we simply avoid issuing reads to the invalid DVAs.

This new two-step pool load process now allows rewinding pools accross
vdev tree changes such as device replacement, addition, etc. Loading a pool
from an external config file in a clustering environment also becomes much
safer now since the pool will import even if the config is outdated and didn't,
for instance, register a recent device addition.

With this code in place, it became relatively easy to implement a
long-sought-after feature: the ability to import a pool with missing top level
(i.e. non-redundant) devices. Note that since this almost guarantees some loss
Of data, this feature is for now restricted to a read-only import.

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Andrew Stormont <andyjstormont@gmail.com>
Approved by: Hans Rosenfeld <rosenfeld@grumpf.hope-2000.org>
Author: Pavel Zakharov <pavel.zakharov@delphix.com>

MFC r329769: MFV r329766: 8962 zdb should work on non-idle pools

illumos/illumos-gate@e144c4e6c90e7d4dccaad6db660ee42b6e7ba04f

Currently `zdb` consistently fails to examine non-idle pools as it fails
during the `spa_load()` process. The main problem seems to be that
`spa_load_verify()` fails as can be seen below:

$ sudo zdb -d -G dcenter
zdb: can't open 'dcenter': I/O error

ZFS_DBGMSG(zdb):
spa_open_common: opening dcenter
spa_load(dcenter): LOADING
disk vdev '/dev/dsk/c4t11d0s0': best uberblock found for spa dcenter. txg 40824950
spa_load(dcenter): using uberblock with txg=40824950
spa_load(dcenter): UNLOADING
spa_load(dcenter): RELOADING
spa_load(dcenter): LOADING
disk vdev '/dev/dsk/c3t10d0s0': best uberblock found for spa dcenter. txg 40824952
spa_load(dcenter): using uberblock with txg=40824952
spa_load(dcenter): FAILED: spa_load_verify failed [error=5]
spa_load(dcenter): UNLOADING

This change makes `spa_load_verify()` a dryrun when ran from `zdb`. This is
done by creating a global flag in zfs and then setting it in `zdb`.

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Andy Stormont <astormont@racktopsystems.com>
Approved by: Dan McDonald <danmcd@joyent.com>
Author: Pavel Zakharov <pavel.zakharov@delphix.com>

MFC r329732: MFV r329502: 7614 zfs device evacuation/removal

illumos/illumos-gate@5cabbc6b49070407fb9610cfe73d4c0e0dea3e77

https://www.illumos.org/issues/7614:
This project allows top-level vdevs to be removed from the storage pool with
“zpool remove”, reducing the total amount of storage in the pool. This
operation copies all allocated regions of the device to be removed onto other
devices, recording the mapping from old to new location. After the removal is
complete, read and free operations to the removed (now “indirect”) vdev must
be remapped and performed at the new location on disk. The indirect mapping
table is kept in memory whenever the pool is loaded, so there is minimal
performance overhead when doing operations on the indirect vdev.

The size of the in-memory mapping table will be reduced when its entries
become “obsolete” because they are no longer used by any block pointers in
the pool. An entry becomes obsolete when all the blocks that use it are
freed. An entry can also become obsolete when all the snapshots that
reference it are deleted, and the block pointers that reference it have been
“remapped” in all filesystems/zvols (and clones). Whenever an indirect block
is written, all the block pointers in it will be “remapped” to their new
(concrete) locations if possible. This process can be accelerated by using
the “zfs remap” command to proactively rewrite all indirect blocks that
reference indirect (removed) vdevs.

Note that when a device is removed, we do not verify the checksum of the data
that is copied. This makes the process much faster, but if it were used on
redundant vdevs (i.e. mirror or raidz vdevs), it would be possible to copy
the wrong data, when we have the correct data on e.g. the other side of the
mirror. Therefore, mirror and raidz devices can not be removed.

Reviewed by: Alex Reece <alex@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: John Kennedy <john.kennedy@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Richard Laager <rlaager@wiktel.com>
Reviewed by: Tim Chase <tim@chase2k.com>
Approved by: Garrett D'Amore <garrett@damore.org>
Author: Prashanth Sreenivasa <pks@delphix.com>

MFC r331135:
Use __syscall(2) rather than syscall(2) in syscall/tst.args.c.

MFC r329681: MFV r318941: 7446 zpool create should support efi system partition

illumos/illumos-gate@7855d95b30fd903e3918bad5a29b777e765db821
https://github.com/illumos/illumos-gate/commit/7855d95b30fd903e3918bad5a29b777e765db821

https://www.illumos.org/issues/7446
Since we support whole-disk configuration for boot pool, we also will need
whole disk support with UEFI boot and for this, zpool create should create efi-
system partition.
I have borrowed the idea from oracle solaris, and introducing zpool create -
B switch to provide an way to specify that boot partition should be created.
However, there is still an question, how big should the system partition be.
For time being, I have set default size 256MB (thats minimum size for FAT32
with 4k blocks). To support custom size, the set on creation "bootsize"
property is created and so the custom size can be set as: zpool create B -
o bootsize=34MB rpool c0t0d0
After pool is created, the "bootsize" property is read only. When -B switch is
not used, the bootsize defaults to 0 and is shown in zpool get output with
value ''. Older zfs/zpool implementations are ignoring this property.
https://www.illumos.org/rb/r/219/

Reviewed by: Andrew Stormont <andyjstormont@gmail.com>
Reviewed by: Yuri Pankov <yuri.pankov@gmail.com>
Approved by: Dan McDonald <danmcd@kebe.com>
Author: Toomas Soome <tsoome@me.com>

This commit makes no sense for FreeBSD, that is why I blocked the option,
but it should be good to stay closer to upstream.

MFC r329658: MFV r316872: 7502 ztest should run zdb with -G (debug mode)

illumos/illumos-gate@c3c65d17f7b6689bbd6568a1a1fcc0c4a3bac127
https://github.com/illumos/illumos-gate/commit/c3c65d17f7b6689bbd6568a1a1fcc0c4a3bac127

https://www.illumos.org/issues/7502
Right now ztest executes zdb without -G, so when it has errors, the messages
are often not very helpful:
Executing zdb -bccsv -d -U /rpool/tmp/zpool.cache ztest
zdb: can't open 'ztest': Operation not supported
ztest: '/usr/sbin/amd64/zdb -bccsv -d -U /rpool/tmp/zpool.cache ztest' exit
code 1
With -G, we'd have:
/usr/sbin/amd64/zdb -bccsv -d -U /rpool/tmp/zpool.cache -G ztest
zdb: can't open 'ztest': Operation not supported

ZFS_DBGMSG(zdb):
spa_open_common: opening ztest
spa_load(ztest): LOADING
spa_load(ztest): FAILED: unable to parse config [error=48]
spa_load(ztest): UNLOADING
Which indicates where the error came from

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Approved by: Gordon Ross <gordon.w.ross@gmail.com>
Author: Pavel Zakharov <pavel.zakharov@delphix.com>

MFC r329657 (by asomers): Fix memory leaks in zdb introduced by r329508

Reported by: Coverity
CID: 1386185

MFC r329508: MFV r324198: 8081 Compiler warnings in zdb

illumos/illumos-gate@3f7978d02b206a6ebc5652c91aa9f42da6fbe00c
https://github.com/illumos/illumos-gate/commit/3f7978d02b206a6ebc5652c91aa9f42da6fbe00c

https://www.illumos.org/issues/8081
zdb(8) is full of minor problems that generate compiler warnings. On FreeBSD,
which uses -WError, the only way to build it is to disable all compiler
warnings. This makes it much harder to detect newly introduced bugs. We should
cleanup all the warnings.

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: Alan Somers <asomers@gmail.com>

MFC r313264:

Avoid using Sun compiler-specific flags.

MFC r329067:

Fix "zpool add" crash when a replacing vdev has a spare child

Fix an assertion in zpool that causes a crash when running any "zpool add"
command on a spare that contains a replacing vdev with a spare child.

This likely affects Illumos, too.

PR: 225546
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D14138

MFC r328256:
MFV r328255: 8972 zfs holds: In scripted mode, do not pad columns with spaces

illumos/illumos-gate@e9b7d6e7f7a6477679a35b73eb3934b096b3dd39

https://www.illumos.org/issues/8972:
'zfs holds -H' does not properly output content in scripted mode. It uses a
tab instead of two spaces, but it still pads column widths with spaces when
it should not.

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Dan McDonald <danmcd@joyent.com>
Author: Allan Jude <allanjude@freebsd.org>

MFC r328252: MFV r328251: 8652 Tautological comparisons with ZPROP_INVAL

illumos/illumos-gate@4ae5f5f06c6c2d1db8167480f7d9e3b5378ba2f2

https://www.illumos.org/issues/8652:
Clang and GCC prefer to use unsigned ints to store enums. With Clang, that
causes tautological comparison warnings when comparing a zfs_prop_t or
zpool_prop_t variable to the macro ZPROP_INVAL. It's likely that error
handling code is being silently removed as a result.

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Igor Kozhukhov <igor@dilos.org>
Approved by: Gordon Ross <gwr@nexenta.com>
Author: Alan Somers <asomers@gmail.com>

MFC r328234: MFV r328233:
8898 creating fs with checksum=skein on the boot pools fails ungracefully

illumos/illumos-gate@9fa2266d9a78b8366e1cd2d5f050e8b5e37d558c

https://www.illumos.org/issues/8898:
# zfs create -o checksum=skein rpool/test
internal error: Result too large
Abort (core dumped)

Not a big deal per se, but should be handled correctly.

Reviewed by: Toomas Soome <tsoome@me.com>
Reviewed by: Andy Stormont <astormont@racktopsystems.com>
Approved by: Dan McDonald <danmcd@joyent.com>
Author: Yuri Pankov <yuri.pankov@nexenta.com>

PR: 222199

MFC r328224: MFV r328220: 8677 Open-Context Channel Programs

illumos/illumos-gate@a3b2868063897ff0083dea538f55f9873eec981f

https://www.illumos.org/issues/8677
We want to be able to run channel programs outside of synching context.
This would greatly improve performance of channel program that just gather
information, as we won't have to wait for synching context anymore.

This feature should introduce the following:
- A new command line flag in "zfs program" to specify our intention to
run in open context.
- A new flag/option within the channel program ioctl which selects the
context.
- Appropriate error handling whenever we try a channel program in
open-context that contains zfs.sync* expressions.
- Documentation for the new feature in the manual pages.

Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: Chris Williamson <chris.williamson@delphix.com>
Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Serapheim Dimitropoulos <serapheim@delphix.com>

MFC r319671:
SHA-512 and Skein have been supported by the boot loader for some time.

MFC r327888, r327972, r327973:
Add "jid" and "jailname" variables to DTrace.

MFC r326150: zdb: use a heap allocation instead of a huge array on stack

MFC r326061, r326063:
DTrace test fixups.

MFC r325035: MFV r325013,r325034: 640 number_to_scaled_string is duplicated in several commands

FreeBSD note: of all libcmdutils functionality ZFS (and other illumos
contrib code) currently uses only nicenum() function (which is similar
to humanize_number but has some formatting differences). For this
reason I decided to not port the whole library. As a result, nicenum.c
from libcmdutils is compiled into libzfs and libzpool. This is a bit
ugly, but works. If one day we are forced to create libillumos, then
the file should be moved to that library.

MFC r324197: MFV r323913: 8600 ZFS channel programs - snapshot

MFC r324196: MFV r323912: 8592 ZFS channel programs - rollback

MFC r324170: MFV r323794: 8605 zfs channel programs: zfs.exists undocumented and non-working

MFC r324163: MFV r323530,r323533,r323534: 7431 ZFS Channel Programs, and followups

Also MFC-ed are the following fixes:
- r324164 Add several new files to the files enabled by ZFS kernel option
- r324178 unbreak kernel builds on sparc64 and powerpc
- r324194 fix incorrect use of getzfsvfs_impl in r324163
- r324292 really unbreak kernel builds on sparc64 and powerpc64

MFC r324350: zdb.8: replace with the slighly modified upstream version

MFC r324349: MFV r322235: 8067 zdb should be able to dump literal embedded block pointer

MFC r324344: MFV r316864: 6392 zdb: introduce -V for verbatim import

MFC r324343: MFV r316862: 6410 teach zdb to perform object lookups by path

MFC r324011, r324016: MFV r323535: 8585 improve batching done in zil_commit()

FreeBSD notes:
- this MFV reverts FreeBSD commit r314549 to make the merge easier
- at present our emulation of cv_timedwait_hires is rather poor,
so I elected to use cv_timedwait_sbt directly
Please see the differential revision for details.
Unfortunately, I did not get any positive reviews, so there could be
bugs in the FreeBSD-specific piece of the merge.
Hence, the long MFC timeout.

illumos/illumos-gate@1271e4b10dfaaed576c08a812f466f6e81370e5e
https://github.com/illumos/illumos-gate/commit/1271e4b10dfaaed576c08a812f466f6e81370e5e

https://www.illumos.org/issues/8585
The current implementation of zil_commit() can introduce significant
latency, beyond what is inherent due to the latency of the underlying
storage. The additional latency comes from two main problems:
1. When there's outstanding ZIL blocks being written (i.e. there's
already a "writer thread" in progress), then any new calls to
zil_commit() will block waiting for the currently oustanding ZIL
blocks to complete. The blocks written for each "writer thread" is
coined a "batch", and there can only ever be a single "batch" being
written at a time. When a batch is being written, any new ZIL
transactions will have to wait for the next batch to be written,
which won't occur until the current batch finishes.
As a result, the underlying storage may not be used as efficiently
as possible. While "new" threads enter zil_commit() and are blocked
waiting for the next batch, it's possible that the underlying
storage isn't fully utilized by the current batch of ZIL blocks. In
that case, it'd be better to allow these new threads to generate
(and issue) a new ZIL block, such that it could be serviced by the
underlying storage concurrently with the other ZIL blocks that are
being serviced.
2. Any call to zil_commit() must wait for all ZIL blocks in its "batch"
to complete, prior to zil_commit() returning. The size of any given
batch is proportional to the number of ZIL transaction in the queue
at the time that the batch starts processing the queue; which
doesn't occur until the previous batch completes. Thus, if there's a
lot of transactions in the queue, the batch could be composed of
many ZIL blocks, and each call to zil_commit() will have to wait for
all of these writes to complete (even if the thread calling
zil_commit() only cared about one of the transactions in the batch).

Reviewed by: Brad Lewis <brad.lewis@delphix.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Dan McDonald <danmcd@joyent.com>
Author: Prakash Surya <prakash.surya@delphix.com>

MFC r324223:

MFV r319743: 8108 zdb -l fails to read labels 2 and 3

illumos/illumos-gate@22c8b9583d07895c16549075a53668d7bc988cf3
https://github.com/illumos/illumos-gate/commit/22c8b9583d07895c16549075a53668d7bc988cf3

https://www.illumos.org/issues/8108

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Toomas Soome <tsoome@me.com>
Reviewed by: Igor Kozhukhov <igor@dilos.org>
Reviewed by: Andrew Stormont <andyjstormont@gmail.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Yuri Pankov <yuri.pankov@nexenta.com>

MFC r324222:

MFV r316863: 3871 fix issues introduced by 3604

illumos/illumos-gate@de05b58863498b10283637eb9ac85e92fe85150e
https://github.com/illumos/illumos-gate/commit/de05b58863498b10283637eb9ac85e92fe85150e

https://www.illumos.org/issues/3871
GCC 4.5.3 on Gentoo Linux did not like a few of the changes made in the issue
3604 patch. It printed an error and a couple of warnings:
../../cmd/zdb/zdb.c: In function 'dump_bpobj':
../../cmd/zdb/zdb.c:1257:3: error: 'for' loop initial declarations are only
allowed in C99 mode
../../cmd/zdb/zdb.c:1257:3: note: use option -std=c99 or -std=gnu99 to compile
your code
../../cmd/zdb/zdb.c: In function 'dump_deadlist':
../../cmd/zdb/zdb.c:1323:8: warning: too many arguments for format
../../cmd/zdb/zdb.c:1323:8: warning: too many arguments for format

Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Serapheim Dimitropoulos <serapheim@delphix.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Richard Yao <ryao@gentoo.org>

MFC r324221:

MFV r316861: 6866 zdb -l should return non-zero if it fails to find any label

illumos/illumos-gate@64723e361134b2a2c45341334174f9d34002f8d0
https://github.com/illumos/illumos-gate/commit/64723e361134b2a2c45341334174f9d34002f8d0

https://www.illumos.org/issues/6866
Need this for #6865.
To be generally more scripting-friendly, overload this issue with adding '-q'
option which should skip printing any label information.

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: John Kennedy <john.kennedy@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Yuri Pankov <yuri.pankov@nexenta.com>

MFC r324220:

MFV r316858 7280 Allow changing global libzpool variables in zdb

7280 Allow changing global libzpool variables in zdb and ztest through command line

illumos/illumos-gate@0e60744c982adecd0a1f146f5637475d07ab1069
https://github.com/illumos/illumos-gate/commit/0e60744c982adecd0a1f146f5637475d07ab1069

https://www.illumos.org/issues/7280
zdb is very handy for diagnosing problems with a pool in a safe and
quick way. When a pool is in a bad shape, we often want to disable some
fail-safes, or adjust some tunables in order to open them. In the
kernel, this is done by changing public variables in mdb. The goal of
this feature is to add the same capability to zdb and ztest, so that
they can change libzpool tuneables from the command line.

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Pavel Zakharov <pavel.zakharov@delphix.com>

MFC r323791: MFV r323790: 8567 Inconsistent return value in zpool_read_label

MFC r323355: MFV r323107: 8414 Implemented zpool scrub pause/resume

illumos/illumos-gate@1702cce751c5cb7ead878d0205a6c90b027e3de8
https://github.com/illumos/illumos-gate/commit/1702cce751c5cb7ead878d0205a6c90b027e3de8

FreeBSD note: rather than merging the zpool.8 update I copied the zpool
scrub section from the illumos zpool.1m to FreeBSD zpool.8 almost
verbatim. Now that the illumos page uses the mdoc format, it was an
easier option. Perhaps the change is not in perfect compliance with the
FreeBSD style, but I think that it is acceptible.

https://www.illumos.org/issues/8414
This issue tracks the port of scrub pause from ZoL: https://github.com/zfsonlinux/zfs/pull/6167
Currently, there is no way to pause a scrub. Pausing may be useful when
the pool is busy with other I/O to preserve bandwidth.

Description

This patch adds the ability to pause and resume scrubbing. This is achieved
by maintaining a persistent on-disk scrub state. While the state is 'paused'
we do not scrub any more blocks. We do however perform regular scan
housekeeping such as freeing async destroyed and deadlist blocks while paused.

Motivation and Context

Scrub pausing can be an I/O intensive operation and people have been asking
for the ability to pause a scrub for a while. This allows one to preserve scrub
progress while freeing up bandwidth for other I/O.

Reviewed by: George Melikov <mail@gmelikov.ru>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Brad Lewis <brad.lewis@delphix.com>
Reviewed by: Serapheim Dimitropoulos <serapheim@delphix.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Approved by: Dan McDonald <danmcd@joyent.com>
Author: Alek Pinchuk <apinchuk@datto.com>

MFC: 323253, 323499

Add D definitions for the named values in socket.h (gnn@)

Fix DTrace test tst_inet_ntop_d: remove definitions are already in libdtrace (lwhsu@)

MFC r322226: MFV r322223: 8378 crash due to bp in-memory modification of nopwrite block

illumos/illumos-gate@b7edcb940884114e61382937505433c4c38c0278
https://github.com/illumos/illumos-gate/commit/b7edcb940884114e61382937505433c4c38c0278

https://www.illumos.org/issues/8378
The problem is that zfs_get_data() supplies a stale zgd_bp to dmu_sync(), which
we then nopwrite against.
zfs_get_data() doesn't hold any DMU-related locks, so after it copies db_blkptr
to zgd_bp, dbuf_write_ready()
could change db_blkptr, and dbuf_write_done() could remove the dirty record.
dmu_sync() then sees the stale
BP and that the dbuf it not dirty, so it is eligible for nop-writing.
The fix is for dmu_sync() to copy db_blkptr to zgd_bp after acquiring the
db_mtx. We could still see a stale
db_blkptr, but if it is stale then the dirty record will still exist and thus
we won't attempt to nopwrite.

Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Matthew Ahrens <mahrens@delphix.com>

MFC r319746,r319747,r319769: 8269 dtrace stddev aggregation is normalized incorrectly

illumos/illumos-gate@79809f9cf402f130667349b2d4007ecd65d63c6f
https://github.com/illumos/illumos-gate/commit/79809f9cf402f130667349b2d4007ecd65d63c6f

https://www.illumos.org/issues/8269
It seems that currently normalization of stddev aggregation is done
incorrectly.
We divide both the sum of values and the sum of their squares by the
normalization factor. But we should divide the sum of squares by the
normalization factor squared to scale the original values properly.

FreeBSD note: the actual change was committed in r316853, this commit
adds the test files and record merge information.

Reviewed by: Bryan Cantrill <bryan@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Andriy Gapon <avg@FreeBSD.org>

Sponsored by: Panzura

MFC r321963:
Rework and simplify the ksyms implementation.

MFC r322773-r322775:
Fix an off-by-two in the llquantize() action parameter validation.

MFC r321387:

Don't use incorrect hardcoded path to ksh -- use /usr/bin/env
to find ksh instead

MFC r321843, r321843:
Let lockstat use ksyms(4)'s mmap interface.

MFC r320156, r320185, r320186, r320262, r320452, r321111:
MFV r318946: 8021 ARC buf data scatter-ization

illumos/illumos-gate@770499e185d15678ccb0be57ebc626ad18d93383
https://github.com/illumos/illumos-gate/commit/770499e185d15678ccb0be57ebc626ad1
8d93383

https://www.illumos.org/issues/8021
The ARC buf data project (known simply as "ABD" since its genesis in the ZoL
community) changes the way the ARC allocates `b_pdata` memory from using linea
r
`void *` buffers to using scatter/gather lists of fixed-size 1KB chunks. This
improves ZFS's performance by helping to defragment the address space occupied
by the ARC, in particular for cases where compressed ARC is enabled. It could
also ease future work to allocate pages directly from `segkpm` for minimal-
overhead memory allocations, bypassing the `kmem` subsystem.
This is essentially the same change as the one which recently landed in ZFS on
Linux, although they made some platform-specific changes while adapting this
work to their codebase:
1. Implemented the equivalent of the `segkpm` suggestion for future work
mentioned above to bypass issues that they've had with the Linux kernel memory
allocator.
2. Changed the internal representation of the ABD's scatter/gather list so it
could be used to pass I/O directly into Linux block device drivers. (This
feature is not available in the illumos block device interface yet.)

FreeBSD notes:
- the actual (default) chunk size is 4KB (despite the text above saying 1KB)
- we can try to reimplement ABDs, so that they are not permanently
mapped into the KVA unless explicitly requested, especially on
platforms with scarce KVA
- we can try to use unmapped I/O and avoid intermediate allocation of a
linear, virtual memory mapped buffer
- we can try to avoid extra data copying by referring to chunks / pages
in the original ABD

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: John Kennedy <john.kennedy@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Prashanth Sreenivasa <pks@delphix.com>
Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: Chris Williamson <chris.williamson@delphix.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: Dan Kimmel <dan.kimmel@delphix.com>

MFC r319672 (by allanjude):
New sentences start on new lines, fix two violations

MFC r318812: MFV r316860: 7545 zdb should disable reference tracking

illumos/illumos-gate@4dd77f9e38ef05b39db128ff7608d926fd3218c6
https://github.com/illumos/illumos-gate/commit/4dd77f9e38ef05b39db128ff7608d926fd3218c6

https://www.illumos.org/issues/7545
When evicting from the ARC, we manipulate some refcount_t's, e.g. arcs_size.
When using zdb to examine a large amount of data (e.g. zdb -bb on a large pool
with small blocks), the ARC may have a large number of entries. If reference
tracking is enabled, there will be ~1 reference for each block in the ARC. When
evicting, we decrement the refcount and have to search all the references to
find the one that we are removing, which is very slow.
Since zdb is typically used to find problems with the on-disk format, and not
with the code it is running, we should disable reference tracking in zdb.

Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: Steve Gonczi <steve.gonczi@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Matthew Ahrens <mahrens@delphix.com>

MFC r317414: MFV 316894

7252 7628 compressed zfs send / receive

illumos/illumos-gate@5602294fda888d923d57a78bafdaf48ae6223dea
https://github.com/illumos/illumos-gate/commit/5602294fda888d923d57a78bafdaf48ae6223dea

https://www.illumos.org/issues/7252
This feature includes code to allow a system with compressed ARC enabled to
send data in its compressed form straight out of the ARC, and receive data in
its compressed form directly into the ARC.

https://www.illumos.org/issues/7628
We should have longer, more readable versions of the ZFS send / recv options.

7628 create long versions of ZFS send / receive options

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: John Kennedy <john.kennedy@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: Sebastien Roy <sebastien.roy@delphix.com>
Reviewed by: David Quigley <dpquigl@davequigley.com>
Reviewed by: Thomas Caputi <tcaputi@datto.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Dan Kimmel <dan.kimmel@delphix.com>

MFC r317267: MFV 316891

7386 zfs get does not work properly with bookmarks

illumos/illumos-gate@edb901aab9c738b5eb15aa55933e82b0f2f9d9a2
https://github.com/illumos/illumos-gate/commit/edb901aab9c738b5eb15aa55933e82b0f2f9d9a2

https://www.illumos.org/issues/7386
The zfs get command does not work with the bookmark parameter while it works
properly with both filesystem and snapshot:
# zfs get -t all -r creation rpool/test
NAME PROPERTY VALUE SOURCE
rpool/test creation Fri Sep 16 15:00 2016 -
rpool/test@snap creation Fri Sep 16 15:00 2016 -
rpool/test#bkmark creation Fri Sep 16 15:00 2016 -
# zfs get -t all -r creation rpool/test@snap
NAME PROPERTY VALUE SOURCE
rpool/test@snap creation Fri Sep 16 15:00 2016 -
# zfs get -t all -r creation rpool/test#bkmark
cannot open 'rpool/test#bkmark': invalid dataset name
#
The zfs get command should be modified to work properly with bookmarks too.

Reviewed by: Simon Klinkert <simon.klinkert@gmail.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Approved by: Matthew Ahrens <mahrens@delphix.com>
Author: Marcel Telka <marcel@telka.sk>

MFC r315896: MFV r315290, r315291: 7303 dynamic metaslab selection

illumos/illumos-gate@8363e80ae72609660f6090766ca8c2c18aa53f0c
https://github.com/illumos/illumos-gate/commit/8363e80ae72609660f6090766ca8c2c18

https://www.illumos.org/issues/7303

This change introduces a new weighting algorithm to improve metaslab selection
.
The new weighting algorithm relies on the SPACEMAP_HISTOGRAM feature. As a res
ult,
the metaslab weight now encodes the type of weighting algorithm used
(size-based vs segment-based).

This also introduce a new allocation tracing facility and two new dcmds to hel
p
debug allocation problems. Each zio now contains a zio_alloc_list_t structure
that is populated as the zio goes through the allocations stage. Here's an
example of how to use the tracing facility:

> c5ec000::print zio_t io_alloc_list | ::walk list | ::metaslab_trace
MSID DVA ASIZE WEIGHT RESULT VDEV
- 0 400 0 NOT_ALLOCATABLE ztest.0a
- 0 400 0 NOT_ALLOCATABLE ztest.0a
- 0 400 0 ENOSPC ztest.0a
- 0 200 0 NOT_ALLOCATABLE ztest.0a
- 0 200 0 NOT_ALLOCATABLE ztest.0a
- 0 200 0 ENOSPC ztest.0a
1 0 400 1 x 8M 17b1a00 ztest.0a

> 1ff2400::print zio_t io_alloc_list | ::walk list | ::metaslab_trace
MSID DVA ASIZE WEIGHT RESULT VDEV
- 0 200 0 NOT_ALLOCATABLE mirror-2
- 0 200 0 NOT_ALLOCATABLE mirror-0
1 0 200 1 x 4M 112ae00 mirror-1
- 1 200 0 NOT_ALLOCATABLE mirror-2
- 1 200 0 NOT_ALLOCATABLE mirror-0
1 1 200 1 x 4M 112b000 mirror-1
- 2 200 0 NOT_ALLOCATABLE mirror-2

If the metaslab is using segment-based weighting then the WEIGHT column will
display the number of segments available in the bucket where the allocation
attempt was made.

Author: George Wilson <george.wilson@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Reviewed by: Chris Siden <christopher.siden@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <paul.dagnelie@delphix.com>
Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Don Brady <don.brady@intel.com>
Approved by: Richard Lowe <richlowe@richlowe.net>

MFC r316856:

MFV 316855

7900 zdb shouldn't print the path of a znode at verbosity < 5

Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Alan Somers <asomers@freebsd.org>

illumos/illumos-gate@e548d2fa41d1baa06662ed9abbb8bcec86e27dd9
https://www.illumos.org/issues/7900

Sponsored by: Spectra Logic Corp

MFC r308782:
After some ZIL changes 6 years ago zil_slog_limit got partially broken
due to zl_itx_list_sz not updated when async itx'es upgraded to sync.
Actually because of other changes about that time zl_itx_list_sz is not
really required to implement the functionality, so this patch removes
some unneeded broken code and variables.

Original idea of zil_slog_limit was to reduce chance of SLOG abuse by
single heavy logger, that increased latency for other (more latency critical)
loggers, by pushing heavy log out into the main pool instead of SLOG. Beside
huge latency increase for heavy writers, this implementation caused double
write of all data, since the log records were explicitly prepared for SLOG.
Since we now have I/O scheduler, I've found it can be much more efficient
to reduce priority of heavy logger SLOG writes from ZIO_PRIORITY_SYNC_WRITE
to ZIO_PRIORITY_ASYNC_WRITE, while still leave them on SLOG.

Existing ZIL implementation had problem with space efficiency when it
has to write large chunks of data into log blocks of limited size. In some
cases efficiency stopped to almost as low as 50%. In case of ZIL stored on
spinning rust, that also reduced log write speed in half, since head had to
uselessly fly over allocated but not written areas. This change improves
the situation by offloading problematic operations from z*_log_write() to
zil_lwb_commit(), which knows real situation of log blocks allocation and
can split large requests into pieces much more efficiently. Also as side
effect it removes one of two data copy operations done by ZIL code WR_COPIED
case.

While there, untangle and unify code of z*_log_write() functions.
Also zfs_log_write() alike to zvol_log_write() can now handle writes crossing
block boundary, that may also improve efficiency if ZPL is made to do that.

Sponsored by: iXsystems, Inc.

MFC r307400, r307401:
DTrace test fixes.

MFC r309698, r309699, r309700:
DTrace test fixes.

MFC r305332: MFV r304159: 7277 zdb should be able to print zfs_dbgmsg's

illumos/illumos-gate@29bdd2f916366ece37c4748bca6b3d61f57a223b
https://github.com/illumos/illumos-gate/commit/29bdd2f916366ece37c4748bca6b3d61f
57a223b

https://www.illumos.org/issues/7277
ztest always prints the debug messages (zfs_dbgmsg()) by calling
zfs_dbgmsg_print(). We should add a flag to zdb to make it do this as well
before exiting.

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Igor Kozhukhov <ikozhukhov@gmail.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Pavel Zakharov <pavel.zakharov@delphix.com>

MFC r305328: MFV r303081: 7163 ztest failures due to excess error injection

illumos/illumos-gate@f34284d835bc555f987c1310df46c034c3101155
https://github.com/illumos/illumos-gate/commit/f34284d835bc555f987c1310df46c034c
3101155

https://www.illumos.org/issues/7163
Running zloop from zfs-precommit hit this assertion:
*panicstr/s
0xfffffd7fd7419370: assertion failed for thread 0xfffffd7fe29ed240,
thread-id 577: parent != NULL, file ../../../uts/common/fs/zfs/dbuf.c, line
1827
$c
libc.so.1`_lwp_kill+0xa()
libc.so.1`_assfail+0x182(fffffd7ffb1c29fa, fffffd7ffb1cc028, 723)
libc.so.1`assfail+0x19(fffffd7ffb1c29fa, fffffd7ffb1cc028, 723)
libzpool.so.1`dbuf_dirty+0xc69(10e3bc10, 3601700)
libzpool.so.1`dbuf_dirty+0x61e(10c73640, 3601700)
libzpool.so.1`dbuf_dirty+0x61e(10e28280, 3601700)
libzpool.so.1`dmu_buf_will_fill+0x64(10e28280, 3601700)
libzpool.so.1`dmu_write+0x1b6(2c7e640, d, 400000002e000000, 200, 3717b40,
3601700)
ztest_replay_write+0x568(4950d0, 3717a80, 0)
ztest_write+0x125(4950d0, d, 400000002e000000, 200, 413f000)
ztest_io+0x1bb(4950d0, d, 400000002e000000)
ztest_dmu_write_parallel+0xaa(4950d0, 6)
ztest_execute+0x83(1, 420c98, 6)
ztest_thread+0xf4(6)
libc.so.1`_thrp_setup+0x8a(fffffd7fe29ed240)
libc.so.1`_lwp_start()
This is another manifestation of ECKSUM in ztest:
The lowest level ancestor that’s in memory is the L8 (topmost). The L7
ancestor is blkid 0x10:
::dbufs -O 0x2c7e640 -o d -l 7 |::dbuf
addr object lvl blkid holds os
600be50 d 7 4 1 ztest/ds_6
719d880 d 7 0 4 ztest/ds_6

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Matthew Ahrens <mahrens@delphix.com>

MFC r305327: MFV r303080: 6451 ztest fails due to checksum errors

illumos/illumos-gate@f9eb9fdf196b6ed476e4ffc69cecd8b0da3cb7e7
https://github.com/illumos/illumos-gate/commit/f9eb9fdf196b6ed476e4ffc69cecd8b0d
a3cb7e7

https://www.illumos.org/issues/6451
Sometimes ztest fails because zdb detects checksum errors. e.g.:
Traversing all blocks to verify checksums and verify nothing leaked ...
zdb_blkptr_cb: Got error 50 reading <71, 47, 0, 8000160> DVA0=<0:1cc2000:
180000> [L0 other uint64[]] sha256 uncompressed LE contiguou
s unique single size=100000L/100000P birth=271L/271P fill=1
cksum=c5a3e27d1ed0f894:843bca3a5473c4bf:f76a19b6830a2e4:91292591613a12bf --
skipping
zdb_blkptr_cb: Got error 50 reading <71, 47, 0, 800000180> DVA0=<0:ce16800:
180000> [L0 other uint64[]] sha256 uncompressed LE contigu
ous unique single size=100000L/100000P birth=840L/840P fill=1
cksum=5d018f3d061e17f3:6d1584784587bf63:2805a74a0ce37369:ba68a214806c7e75
-- skipping
zdb_blkptr_cb: Got error 50 reading <71, 47, 0, 1000000360> DVA0=<0:10d37400:
180000> [L0 other uint64[]] sha256 uncompressed LE conti
guous unique single size=100000L/100000P birth=904L/904P fill=1
cksum=fa1e11d4138bd14b:86c9488c444473e3:f31e43c72e72e46b:e3446472d1174d
ba -- skipping
zdb_blkptr_cb: Got error 50 reading <71, 47, 0, 400000002c0> DVA0=<0:127ef400:
180000> [L0 other uint64[]] sha256 uncompressed LE cont
iguous dedup single size=100000L/100000P birth=549L/549P fill=1
cksum=30e14955ebf13522:66dc2ff8067e6810:4607e750abb9d3b3:6582b8af909fcb
58 -- skipping
zdb_blkptr_cb: Got error 50 reading <657, 5, 0, 1c0> DVA0=<0:1a180400:180000>
[L0 other uint64[]] fletcher4 uncompressed LE contiguou
s unique single size=100000L/100000P birth=1091L/1091P fill=1 cksum=a6cf1e50:
29b3bd01c57e5:36779b914035db9a:db61cdcf6bec56f0 -- skippin
g
The problem is that ztest_fault_inject() can inject multiple faults into the
same block. It is designed such that it can inject errors on all leafs of a
RAID-Z or mirror, but for a given range of offsets, it will only inject errors

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Jorgen Lundman <lundman@lundman.net>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Matthew Ahrens <mahrens@delphix.com>

MFC r305326: MFV r303079:
7147 ztest: ztest_ddt_repair fails with ztest_pattern_match assertion

illumos/illumos-gate@aab80726335c76a7cae32c7300890248d73a51e3
https://github.com/illumos/illumos-gate/commit/aab80726335c76a7cae32c7300890248d
73a51e3

https://www.illumos.org/issues/7147
Here's the dbuf we're currently reading:
966f200::dbuf
addr object lvl blkid holds os
966f200 4 0 0 1 ztest/ds_3
966f200::print dmu_buf_t db_data
db_data = 0x9ae0400
0x9ae0400/10J
0x9ae0400: c1c7ced932020d c1c7ced932020d c1c7ced932020d c1c7ced932020d
c1c7ced932020d c1c7ced932020d c1c7ced932020d c1c7ced932020d
c1c7ced932020d c1c7ced932020d
The pattern we're expecting is actually this: a34ae10b5f2db2. If we attempt to
read the block on disk we find that it has matches what ztest_ddt_repair()
would have written:
~c1c7ced932020d=J
ff3e383126cdfdf2
966f200::print dmu_buf_impl_t db_blkptr | ::blkptr
DVA0=<0:71d3c00:800>
[L0 UINT64_OTHER] SHA256 OFF LE contiguous dedup single
size=400L/400P birth=55L/55P fill=1
cksum=18486450d3ce8c6d:75a72f4bbf117b0f:2d3a226314eb5650:2eb0fd68648b1af0
1. zdb -U /rpool/tmp/zpool.cache -R ztest 0:71d3c00:800 | head
Found vdev type: mirror
0:71d3c00:800
0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
000000: ff3e383126cdfdf2 ff3e383126cdfdf2 ...&18>....&18>.
000010: ff3e383126cdfdf2 ff3e383126cdfdf2 ...&18>....&18>.
000020: ff3e383126cdfdf2 ff3e383126cdfdf2 ...&18>....&18>.
000030: ff3e383126cdfdf2 ff3e383126cdfdf2 ...&18>....&18>.
000040: ff3e383126cdfdf2 ff3e383126cdfdf2 ...&18>....&18>.
000050: ff3e383126cdfdf2 ff3e383126cdfdf2 ...&18>....&18>.

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: George Wilson <george.wilson@delphix.com>

MFC r305323: MFV r302991: 6950 ARC should cache compressed data

illumos/illumos-gate@dcbf3bd6a1f1360fc1afcee9e22c6dcff7844bf2
https://github.com/illumos/illumos-gate/commit/dcbf3bd6a1f1360fc1afcee9e22c6dcff7844bf2

https://www.illumos.org/issues/6950
When reading compressed data from disk, the ARC should keep the compressed
block cached and only decompress it when consumers access the block. The
uncompressed data should be short-lived allowing the ARC to cache a much larger
amount of data. The DMU would also maintain a smaller cache of uncompressed
blocks to minimize the impact of decompressing frequently accessed blocks.

Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Don Brady <don.brady@intel.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: George Wilson <george.wilson@delphix.com>

MFC r305209: MFV r302660: 6314 buffer overflow in dsl_dataset_name

illumos/illumos-gate@9adfa60d484ce2435f5af77cc99dcd4e692b6660
https://github.com/illumos/illumos-gate/commit/9adfa60d484ce2435f5af77cc99dcd4e6
92b6660

https://www.illumos.org/issues/6314
Callers of dsl_dataset_name pass a buffer of size ZFS_MAXNAMELEN, but
dsl_dataset_name copies the datasets' name PLUS the snapshot name to it,
resulting in a max of 2 * ZFS_MAXNAMELEN + '@'.

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Igor Kozhukhov <ikozhukhov@gmail.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Matthew Ahrens <mahrens@delphix.com>

MFC r305206: MFV r302658:
6872 zfs libraries should not allow uninitialized variables

illumos/illumos-gate@f83b46baf98d276f5f84fa84c8b461f412ac1f5e
https://github.com/illumos/illumos-gate/commit/f83b46baf98d276f5f84fa84c8b461f41
2ac1f5e

https://www.illumos.org/issues/6872
We compile the zfs libraries with -Wno-uninitialized. We should remove
this. Change makefiles, fix new warnings, fix pbchk errors.

Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Paul Dagnelie <pcd@delphix.com>

MFC r305205: MFV r302657:
4521 zfstest is trying to execute evil "zfs unmount -a"

illumos/illumos-gate@8808ac5dae118369991f158b6ab736cb2691ecde
https://github.com/illumos/illumos-gate/commit/8808ac5dae118369991f158b6ab736cb2
691ecde

https://www.illumos.org/issues/4521
zfstest is trying to execute evil "zfs unmount -a", which fails (fortunately,
as it would otherwise leave me with my ~ missing):
03:44:11.86 cannot unmount '/export/home/yuri': Device busy cannot unmount '/
export/home': Device busy
03:44:11.86 ERROR: /usr/sbin/zfs unmount -a exited 1
This affects, at least, zfs_mount_009_neg and zfs_mount_all_001_pos, both
failing on that step. The pool containing the /export/home hierarchy is
included in KEEP variable, but it doesn't seem to affect anything here.

Reviewed by: Andriy Gapon <avg@FreeBSD.org>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: John Kennedy <john.kennedy@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Yuri Pankov <yuri.pankov@nexenta.com>

MFC r305195: MFV r302643:
6902 speed up listing of snapshots if requesting name only and sorting by name

This was our change from the beginning, so just reduce the upstream diff.

MFC r304431:
Add a SIGINFO handler for dtrace(1).

MFC r303086: 7164 zdb should be able to open the root dataset

MFC r303084: 6391 Override default SPA config location via environment

MFC r302787

zpool(8): update Hot Spares section to mention zfsd(8)

Approved by: re (gjb)

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Fix missing space in mandoc syntax

Reported by: rpokala

Update zfs(8) and zpool-features(7) man pages with new hashing algorithms

Sponsored by: ScaleEngine Inc.

Print error messages to stderr

Implement support for boot-time DTrace.

This allows one to enable DTrace probes relatively early during boot,
during SI_SUB_DTRACE_ANON, before dtrace(1) can invoked. The desired
enabling is created using dtrace -A, which writes a /boot/dtrace.dof
file and uses nextboot(8) to ensure that DTrace kernel modules are loaded
and that the DOF file describing the enabling is loaded by loader(8)
during the subsequent boot. The trace output can then be fetched with
dtrace -a.

With this commit, boot-time DTrace is only functional on i386 and amd64: on
other architectures, the high-resolution timer frequency is initialized
during SI_SUB_CLOCKS and is thus not available when the anonymous
tracing state is initialized. On x86, the TSC is used and is thus available
earlier.

MFC after: 1 month
Relnotes: yes

MFV r297760: 6418 zpool should have a label clearing command

Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Author: Will Andrews <will@firepipe.net>

Closes #83
Closes #32

openzfs/openzfs@9663688425131744221ea99f9e66b9ed964492ae

FreeBSD already had `zpool labelclear` functionality, so this is mostly
just a diff reduction.

MFC after: 1 month

MFV r296536: 6551 cmd/zpool: cleanup gcc warnings

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Andy Stormont <astormont@racktopsystems.com>
Approved by: Robert Mustacchi <rm@joyent.com>

illumos/illumos-gate@b327cd3f3b4dab4f29e7140159b1e01ed2ceef2a

MFV r296534: 6550 cmd/zfs: cleanup gcc warnings

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Andy Stormont <astormont@racktopsystems.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Igor Kozhukhov <ikozhukhov@gmail.com>

illumos/illumos-gate@c16bcc4577f389573eff411c7b7e040294078c3b

MFV r296532: 6637 replacing "dontclose" with "should_close"

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: David Schwartz <dschwartz783@gmail.com>

illumos/illumos-gate@d189620258b3c9b0e2f7e2104840be2eee7c68e5

MFV r296527: 6659 nvlist_free(NULL) is a no-op

Reviewed by: Toomas Soome <tsoome@me.com>
Reviewed by: Marcel Telka <marcel@telka.sk>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Josef 'Jeff' Sipek <jeffpc@josefsipek.net>

illumos/illumos-gate@aab83bb83be7342f6cfccaed8d5fe0b2f404855d

MFV r296518: 5027 zfs large block support (add copyright)

Author: Matthew Ahrens <matt@mahrens.org>

illumos/illumos-gate@c3d26abc9ee97b4f60233556aadeb57e0bd30bb9

MFV r296505: 6531 Provide mechanism to artificially limit disk performance

Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Prakash Surya <prakash.surya@delphix.com>

illumos/illumos-gate@97e81309571898df9fdd94aab1216dfcf23e060b

Upgrade our copies of clang, llvm, lldb and compiler-rt to 3.8.0
release.

Please note that from 3.5.0 onwards, clang, llvm and lldb require C++11
support to build; see UPDATING for more information.

Release notes for llvm and clang will soon be available here:
<http://llvm.org/releases/3.8.0/docs/ReleaseNotes.html>
<http://llvm.org/releases/3.8.0/tools/clang/docs/ReleaseNotes.html>

Thanks to Ed Maste, Roman Divacky, Davide Italiano and Antoine Brodin
for their help.

Relnotes: yes

Fix "invalid type '(null)'" usage messages in zfs(8) and zpool(8).

Currently, zfs(8) and zpool(8) print "invalid type '(null)'" or similar
messages, if you pass in invalid types, sources or column names for "zfs
get", "zfs list" and "zpool get". This is because the commands use
getsubopt(3), and in case of failure, they print 'value', which is NULL
when sub options don't match.

They should print 'suboptarg' instead, which is the documented way to
get at the non-matching sub option value.

Reviewed by: smh
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D5365

MFV r294814: 6393 zfs receive a full send as a clone

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Paul Dagnelie <pcd@delphix.com>

illumos/illumos-gate@68ecb2ec930c4b0f00acaf8e0abb2b19c4b8b76f

This allows to do a full (non-incremental send) and receive it as a clone
of an existing dataset. It can leverage nopwrite to share blocks with the
origin. This can be used to change the relationship of datasets on the
target. For example, maybe on the source you have:

A ---- B ---- C

And you have sent to the target a full of B, and the incremental B->C:

B ---- C

You later realize that you want to have A on the target. You will have to
do a full send of A, but nopwrite can save you space on the target if you
receive it as a clone of B, assuming that A and B have some blocks inxi
common:

B ---- C
\
A

Report an error if zdb cannot initialize zfs

If the zfs module is not present and not loadable, report an error
to the user instead of crashing

Reviewed by: mahrens
Sponsored by: Gandi.net
Differential Revision: https://reviews.freebsd.org/D4691

Add a trailing newline to the expected output for tst.walltimestamp.ksh.

MFC after: 1 week

MFV r289003:
6271 dtrace caused excessive fork time

Author: Bryan Cantrill <bryan@joyent.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Reviewed by: Richard Lowe <richlowe@richlowe.net>
Approved by: Gordon Ross <gwr@nexenta.com>

illumos/illumos-gate@7bd3c1d12d0c764e1517c3aca62c634409356764

Fix the build for non-amd64.

Fix zfs(8) set options

Fix zfs(8) not formatting due to wrong macro (Oc) in the syntax for the new
zfs set multiple dataset properties option.

PR: 204631
Submitted by: Thomas Eberhardt
Sponsored by: Multiplay

Improve collation string and locales support

Merge collation support from Illumos and DragonflyBSD.

Locales are now generated with the new localedef(1) tool from CLDR POSIX files.
The generated files are now identified as "BSD 1.0" format.

The libc now only read "BSD 1.0" locales definitions, all other version will be
set to "C"
The localedef(1) tool has been imported from Illumos and modified to use tree(3)
instead of the CDDL avl(3)
A set of tool created by edwin@ and extended by marino@ for dragonfly has been
added to be able to generate locales and the Makefiles from the vanilla CLDR
unicode databases + a universal UTF-8 charmap (by marino@)
Update the locales to unicode v27
Given our regex(3) does not support multibyte (yet) it has been forced to always
use locale C
Remove now unused colldef(1) and mklocale(1)
Finish implementing the numeric BSD extension for ctypes
The number of supported locales has grown from 175 to 250 locales. Among the new
locales: 6 Arabic locales (AE EG JO MA QA SA), Different variations of spanish
locales.
Added new 3 components locales for mn_Cyrl_MN, sr_Cyrl_RS sr_Latn_RS,
zh_Hans_CN, zh_Hant_HK and zh_Hant_TW. Some aliases has been for 2 components
version when possible.

Thanks: Garrett D'Amore (Illumos) who made sure all his work was done under
BSD license!, Edwin Groothuis (edwin@) for the work he made on tools to be able
to generate locales definition usable in freebsd sources out of vanilla CLDR
definitions, John Marino (DragonflyBSD) who first merge the Illumos work into
Dragonfly and spent hours tracking down bugs.

Merge mpsutil(8) branch

mpsutil(8)/mprutil(8) are new utilities for managing LSI Fusion-MPT
2/3 controllers (mps(4) and mpr(4))

For now only informational commands have been implemented.

This utility has been written by scottl@ [1] and polished by myself[2]

Submitted by: scottl
Discussed with: scottl
Relnotes: yes
Sponsored by: Netflix [1]
Sponsored by: Gandi.net [2]

Allow 'zfs holds -r' to recurse over a file system or volume to find holds

Previously, the parameters of 'zfs holds' could only be snapshots

Add -d <depth> flag to limit depth of recursion
Add -p flag to print literal values, rather than interpreted values
Add -H flag to suppress header output and use tabs rather than whitespace

Reviewed by: mahrens, smh, dteske
Approved by: bapt (mentor)
MFC after: 3 weeks
Relnotes: yes
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D3994

Record MFV r289689.

The change itself was directly committed earlier at r284309.

MFV r289561: 6328 Fix cstyle errors in zfs codebase

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed by: Jorgen Lundman <lundman@lundman.net>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Paul Dagnelie <pcd@delphix.com>

illumos/illumos-gate@9a686fbc186e8e2a64e9a5094d44c7d6fa0ea167

MFV r289535: 5767 fix several problems with zfs test suite

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Approved by: Gordon Ross <gwr@nexenta.com>
Author: John Wren Kennedy <john.kennedy@delphix.com>

illumos/illumos-gate@52244c0958bdf281ca42932b449f644b4decfdc2

In fact, only unrelated part of that commit is applicable:
8. zpool list -v doesn't print spares
It also doesn't correctly identify log devices.

MFC r289498: 6298 zfs_create_008_neg and zpool_create_023_neg need to be
updated for large block support.

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: John Kennedy <john.kennedy@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Joe Stein <joe.stein@delphix.com>

illumos/illumos-gate@e9316f7696401f3e5e263a5939031cb8d5641a88

MFV r247180: Update vendor/illumos/dist and vendor-sys/illumos/dist
to illumos-gate 13967:92bec6d87f59

Illumos ZFS issues:
3557 dumpvp_size is not updated correctly when a dump zvol's size is
changed
3558 setting the volsize on a dump device does not return back ENOSPC
3559 setting a volsize larger than the space available sometimes succeeds

MFV r289493: 5745 zfs set allows only one dataset property to be set at a time

Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: George Wilson <george@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Bayard Bell <buffer.g.overflow@gmail.com>
Reviewed by: Richard PALO <richard@NetBSD.org>
Reviewed by: Steven Hartland <killing@multiplay.co.uk>
Approved by: Rich Lowe <richlowe@richlowe.net>
Author: Chris Williamson <chris.williamson@delphix.com>

illumos/illumos-gate@30925561c223021e91d15899cbe75f80e54d8889

MFV r289312: 2605 want to resume interrupted zfs send

Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed by: Xin Li <delphij@freebsd.org>
Reviewed by: Arne Jansen <sensille@gmx.net>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Matthew Ahrens <mahrens@delphix.com>

illumos/illumos-gate@9c3fd1216fa7fb02cfbc78a2518a686d54b48ab8

For more info, see:
- slides http://www.slideshare.net/MatthewAhrens/openzfs-send-and-receive
- video https://www.youtube.com/watch?v=iY44jPMvxog
- manpage changes (for zfs resume -s and zfs send -t)
- upcoming talk at the OpenZFS Developer Summit

The TL;DR is:
Use "zfs receive -s" to save the partially received state on failure.
On failure, get the receive token with "zfs get receive_resume_token <fs>"
Resume the send with "zfs send -t <token_value>"

Relnotes: yes

MFV r289311: 5764 "zfs send -nv" directs output to stderr

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <paul.dagnelie@delphix.com>
Reviewed by: Basil Crow <basil.crow@delphix.com>
Reviewed by: Steven Hartland <killing@multiplay.co.uk>
Reviewed by: Bayard Bell <buffer.g.overflow@gmail.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Author: Manoj Joseph <manoj.joseph@delphix.com>

illumos/illumos-gate@dc5f28a3c341db7c241bba77ddc109c141072f27

Upgrade our copies of clang, llvm, lldb, compiler-rt and libc++ to 3.7.0
release.

Please note that from 3.5.0 onwards, clang, llvm and lldb require C++11
support to build; see UPDATING for more information.

Release notes for llvm and clang can be found here:
<http://llvm.org/releases/3.7.0/docs/ReleaseNotes.html>
<http://llvm.org/releases/3.7.0/tools/clang/docs/ReleaseNotes.html>

Thanks to Ed Maste, Andrew Turner and Antoine Brodin for their help.

Exp-run: antoine
Relnotes: yes

Have lockstat(1) trace locks by name rather than by address.

Previously, lockstat(1) would use a lock's address as its identifier when
consuming data describing lock contention and hold events. After collecting
the requested data, it would use ksyms(4) to resolve lock addresses to
names. Of course, this doesn't work too well for locks contained in
dynamically-allocated memory. This change modifies lockstat(1) to trace the
lock names obtained from the base struct lock_object instead, leading to
output that is generally much more useful.

This change also removes the -c option, which is used to coalesce data for
locks in an array. It's not possible to support this option without also
tracing lock addresses, and since lock arrays in which the lock names are
distinct are not very common in FreeBSD, it's simpler to just remove the
option.

Reviewed by: avg (earlier revision)
Differential Revision: https://reviews.freebsd.org/D3661

MFV r288408:
6266 harden dtrace_difo_chunksize() with respect to malicious DIF

illumos/illumos-gate@395c7a3dcfc66b8b671dc4b3c4a2f0ca26449922

Reviewed by: Alex Wilson <alex.wilson@joyent.com>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Approved by: Garrett D'Amore <garrett@damore.org>
Author: Bryan Cantrill <bryan@joyent.com>

MFC after: 1 week

Stop hard-coding a 32-bit data model for USDT tests, and just use the native
model. This was causing many of the tests to fail on amd64 since USDT
support for 32-bit programs is currently non-functional.

MFC after: 1 week

remove unused sgsmsg utility (originally imported from opensolaris)

MFC after: 25 days

MFV r286224: 5695 dmu_sync'ed holes do not retain birth time
(userland portion that was not merged in r286677)

Update zdb to also print ltime, type, and level information
for these new style holes. Previously, only the logical birth
time would be printed.

MFV r277429:

Document -S option when zfs inherit fails on quota and
in manual pages.

Illumos ZFS issues:

5410 Document -S option to zfs inherit
https://illumos.org/issues/5410

5412 Mention -S option when zfs inherit fails on quota
https://illumos.org/issues/5412

illumos/illumos-gate@5ff8cfa92ec8ea0f8593ad21aa2a04829b0ef5ea

MFC after: 2 weeks

Fix build (r287703). Lesson learned: no matter how a change looks like an
innocent one, always do a build test first.

Pointy hat to: delphij

Plug a memory leak.

MFC after: 2 weeks

MFV 286707: 5959 clean up per-dataset feature count code

Reviewed by: Toomas Soome <tsoome@me.com>
Reviewed by: George Wilson <george@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Author: Matthew Ahrens <mahrens@delphix.com>

illumos/illumos-gate@ca0cc3918a1789fa839194af2a9245f801a06b1a

A ZFS feature flags (large blocks) tracks its refcounts as the number of
datasets that have ever used the feature. Several features of this type
are planned to be added (new checksum functions). This code should be made
common infrastructure rather than duplicating the code for each feature.

MFV r286704: 5960 zfs recv should prefetch indirect blocks
5925 zfs receive -o origin=

Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Author: Paul Dagnelie <pcd@delphix.com>

While running 'zfs recv' we noticed that every 128th 8K block required a
read. We were seeing that restore_write() was calling dmu_tx_hold_write()
and the indirect block was not cached. We should prefetch upcoming indirect
blocks to avoid having to go to disk and blocking the restore_write().

Allow an incremental send stream to be received as a clone, even if the
stream does not mark it as a clone.

Record mergeinfo.

MFV 286602: 5810 zdb should print details of bpobj

Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Reviewed by: George Wilson <george@delphix.com>
Reviewed by: Will Andrews <will@freebsd.org>
Reviewed by: Simon Klinkert <simon.klinkert@gmail.com>
Approved by: Gordon Ross <gwr@nexenta.com>
Author: Matthew Ahrens <mahrens@delphix.com>

illumos/illumos-gate@732885fca09e11183dd0ea69aaaab5588fb7dbff

MFV 286586: 5746 more checksumming in zfs send

Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Bayard Bell <buffer.g.overflow@gmail.com>
Approved by: Albert Lee <trisk@omniti.com>
Author: Matthew Ahrens <mahrens@delphix.com>

illumos/illumos-gate@98110f08fa182032082d98be2ddb9391fcd62bf1

MFV 286546:
5661 ZFS: "compression = on" should use lz4 if feature is enabled

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Josef 'Jeff' Sipek <jeffpc@josefsipek.net>
Reviewed by: Xin LI <delphij@freebsd.org>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Justin T. Gibbs <justing@spectralogic.com>

illumos/illumos-gate@db1741f555ec79def5e9846e6bfd132248514ffe

Don't hardcode the module or function component of lockstat probes.

MFC after: 1 week

- Remove hardcoded paths for the perl executable.
- Rather than assuming that a process is listening on 127.0.0.1:22, use
nc(1) to find an available port and bind to it for the duration of the
test.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Fix a rendering issue in the zfs(8) manual.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Consistently use a reader/writer flag for lockstat probes in rwlock(9) and
sx(9), rather than using the probe function name to determine whether a
given lock is a read lock or a write lock. Update lockstat(1) accordingly.

zfs clone should not mount the clone if canmount == noauto

Creation of a new filesystem does not imply an intent to mount it.

Since canmount property is not inherited and its default value is 'on',
the only scenario where this matters is zfs clone -o canmount=noauto.
zfs create -o canmount=noauto already does not mount the new filesystem.

Also see:
https://www.illumos.org/issues/5984
https://reviews.csiden.org/r/228/
https://github.com/FransUrbo/zfs/commit/dd0e0e69f5b1c83bf2895ac00a0b83af77473175
https://github.com/zfsonlinux/zfs/issues/2241

Reviewed by: mahrens
MFC after: 8 days
Sponsored by: ClusterHQ

MFV r284030: 5818 zfs {ref}compressratio is incorrect with 4k sector size

illumos/illumos-gate@81cd5c555f505484180a62ca5a2fbb00d70c57d6

Author: Matthew Ahrens <mahrens@delphix.com>
MFC after: 17 days

several lockstat improvements

0. For spin events report time spent spinning, not a loop count.
While loop count is much easier and cheaper to obtain it is hard
to reason about the reported numbers, espcially for adaptive locks
where both spinning and sleeping can happen.
So, it's better to compare apples and apples.

1. Teach lockstat about FreeBSD rw locks.
This is done in part by changing the corresponding probes
and in part by changing what probes lockstat should expect.

2. Teach lockstat that rw locks are adaptive and can spin on FreeBSD.

3. Report lock acquisition events for successful rw try-lock operations.

4. Teach lockstat about FreeBSD sx locks.
Reporting of events for those locks completely mirrors
rw locks.

5. Report spin and block events before acquisition event.
This is behavior documented for the upstream, so it makes sense to stick
to it. Note that because of FreeBSD adaptive lock implementations
both the spin and block events may be reported for the same acquisition
while the upstream reports only one of them.

Differential Revision: https://reviews.freebsd.org/D2727
Reviewed by: markj
MFC after: 17 days
Relnotes: yes
Sponsored by: ClusterHQ

At revision r247852 accidentally was removed from print the variable obj.
While here, declare some global variables as static to silence clang
warnings.

Differential Revision: D2722
Reviewed by: delphij, mahrens

lockstat(1): document the -V option.

MFC after: 3 days

Add manual pages for the io, ip, proc, sched, tcp and udp DTrace providers.
The format of these pages is somewhat experimental, so they may be subject
to further tweaking.

Differential Revision: https://reviews.freebsd.org/D2170
Reviewed by: bcr, rpaulo
MFC after: 2 weeks

Fix ping(8) and ping6(8) usage in a couple of ip provider tests, and
update expected test output to reflect differences in default TTL and
payload length.

MFC after: 1 week

Fix ping(8) usage in funcs/tst.system.d so that the test actually completes.

MFC after: 1 week

tcp provider tests: sleep for a bit after closing the socket so that
libdtrace has a chance to capture the traced data.

MFC after: 1 week

Upgrade our copy of clang, llvm and lldb to 3.6.0 release.

Please note that from 3.5.0 onwards, clang/llvm/lldb require C++11
support to build; see UPDATING for more information.

Release notes for llvm and clang can be found here:
<http://llvm.org/releases/3.6.0/docs/ReleaseNotes.html>
<http://llvm.org/releases/3.6.0/tools/clang/docs/ReleaseNotes.html>

Thanks to Ed Maste for the lldb part of this upgrade.

Exp-run: antoine

Use the -shared option to create a shared library.

MFC after: 1 week

Remove a leading tab that causes a make(1) error when running the test.

MFC after: 1 week

Only kill sleep processes that were forked from the test script.

MFC after: 1 week

Stop hard-coding an incorrect path to rm(1).

MFC after: 1 week

Set altroot if the user have specified it via -o altroot.

MFC after: 2 weeks

Tweak the fds test program so that it actually compiles. Also use 0 instead
of -1 for the bogus ioctl command so that dmesg doesn't get spammed with
sign extension warnings when the test program runs.

MFC after: 1 week

Avoid hard-coding a path to ksh in scripts that are executed directly by
the test harness. This is a problem in many of the *.ksh test scripts as
well, but those scripts are executed using a shell whose path is specified
in dtest.pl, so there's no need to modify them.

MFC after: 1 week

Use a MIME type of text/plain for a couple of shell scripts with a file
extension of ".exe".

MFC after: 1 week

Fix mktemp(1) usage.

MFC after: 1 week

Use syscall::exit instead of the nonexistent syscall::rexit.

MFC after: 1 week

Remove hard-coded invocations of gcc; use cc(1) instead.

MFC after: 1 week

Remove a makefile that isn't present upstream.

MFC after: 1 week

Fix a number of DTrace scripting tests:
* Avoid hard-coding program paths.
* Use -x when searching for oneself in ps(1) output.
* Use the correct keyword (egid instead of pgid) in tst.egid.ksh.

MFC after: 1 week

Fix some proc provider tests:
* Avoid hard-coding program paths, except when it's necessary in order to
override the use of a shell builtin.
* Translate struct proc through psinfo_t so that we can access process
arguments via the pr_psargs field of psinfo_t.
* Replace uses of pstop and prun with kill(1).

MFC after: 1 week

pause() isn't a system call on FreeBSD; use sigsuspend(2) instead.

MFC after: 1 week

The DTrace harness (dtest.pl) executes DTrace scripts with the -C flag,
which causes dtrace(1) to run the C preprocessor on input scripts before
executing them. Suppress some warnings emitted by the preprocessor which are
confusing the DTrace lexer tests.

MFC after: 1 week

Include required headers in DTrace test programs.

MFC after: 1 week

Replace Solaris identifiers and types with their FreeBSD equivalents.

MFC after: 1 week

Use the correct field when reading the PID out of a struct proc.

MFC after: 1 week

When searching for an accompanying test program, look in the directory
containing the test script rather than the current directory. This is needed
in order to run the DTrace tests under Kyua.

MFC after: 1 week

Mechanically convert cddl sun #ifdef's to illumos

Since the upstream for cddl code is now illumos not sun, mechanically
convert all sun #ifdef's to illumos #ifdef's which have been used in all
newer code for some time.

Also do a manual pass to correct the use if #ifdef comments as per style(9)
as well as few uses of #if defined(__FreeBSD__) vs #ifndef illumos.

MFC after: 1 month
Sponsored by: Multiplay

Upgrade our copy of clang, llvm and lldb to 3.5.0 release.

Please note that this version now requires C++11 support to build; see
UPDATING for more information.

Release notes for llvm and clang can be found here:
<http://llvm.org/releases/3.5.0/docs/ReleaseNotes.html>
<http://llvm.org/releases/3.5.0/tools/clang/docs/ReleaseNotes.html>

Thanks to Ed Maste, Roman Divacky, Andrew Turner, Justin Hibbits and
Antoine Brodin for their invaluable help with this import.

Approved by: portmgr (antoine)
MFC after: 1 month

mdoc: remove EOL whitespace.

Enhancements to zpool upgrade processing

Introduce a seperate phase to list all unavailable pools when listing
pools to upgrade. This avoids confusing output when displaying older
and disabled feature pools. These existing phases now silently skip
unavailable pools.

Introduce cb_unavail to upgrade_cbdata_t which enables the final
output for zpool list to correctly detail if all pools or only all
available pools where up-to-date on version / features.

Correct the type of upgrade_cbdata_t.cb_first from int -> boolean_t.

Change the pool iteration when upgrading named pools to include
unavailable pools and update upgrade_one so it doesn't try to upgrade
unavailable pools but warns about them. This allows the correct error
to be displayed as well as upgrades with available and unavailable
pools intermixed to partially complete.

Also correct some missing trailing \n's from output in upgrade_one.

MFC after: 1 month
X-MFC-With: r276194

Prevent zpool upgrade failing due to unavailable pools

Prior to this fix "zpool upgrade" and "zpool upgrade -a" would fail due to
an assert when operating on unavailable pools.

We now print a warning to stderr but allow the processing of other pools
to procesed.

MFC after: 1 month

MFV r275783:

Convert ARC flags to use enum. Previously, public flags are defined in
arc.h and private flags are defined in arc.c which can lead to confusion
and programming errors.

Consistently use 'hdr' (when referencing arc_buf_hdr_t) instead of 'buf'
or 'ab' because arc_buf_t are often named 'buf' as well.

Illumos issue:
5369 arc flags should be an enum
5370 consistent arc_buf_hdr_t naming scheme

MFC after: 2 weeks

MFV r275551:

Remove "dbuf phys" db->db_data pointer aliases.

Use function accessors that cast db->db_data to the appropriate
"phys" type, removing the need for clients of the dmu buf user
API to keep properly typed pointer aliases to db->db_data in order
to conveniently access their data.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_leaf.c:
In zap_leaf() and zap_leaf_byteswap, now that the pointer alias
field l_phys has been removed, use the db_data field in an on
stack dmu_buf_t to point to the leaf's phys data.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:
Remove the db_user_data_ptr_ptr field from dbuf and all logic
to maintain it.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dbuf.h:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dmu.h:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dir.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:
Modify the DMU buf user API to remove the ability to specify
a db_data aliasing pointer (db_user_data_ptr_ptr).

cddl/contrib/opensolaris/cmd/zdb/zdb.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_diff.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_tx.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_bookmark.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_deadlist.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_deleg.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_destroy.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dir.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_prop.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_synctask.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_userhold.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa_history.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_leaf.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dataset.h:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dir.h:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zap_impl.h:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zap_leaf.h:
Create and use the new "phys data" accessor functions
dsl_dir_phys(), dsl_dataset_phys(), zap_m_phys(),
zap_f_phys(), and zap_leaf_phys().

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dataset.h:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/dsl_dir.h:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zap_impl.h:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zap_leaf.h:
Remove now unused "phys pointer" aliases to db->db_data
from clients of the DMU buf user API.

Illumos issue:
5314 Remove "dbuf phys" db->db_data pointer aliases in ZFS

MFC after: 2 weeks

MFV r275547:

Port Illumos 'zfs allow' examples update. While I'm there also fix
a typo.

Illumos issue:
4181 zfs(1m): 'zfs allow' examples in the man page are outdated

MFC after: 2 weeks

MFV r275537:

Illumos issue:
5316 allow smbadm join to use RPC

(Due to our lack of smbsrv this is mostly no-op on
FreeBSD)

MFC after: 2 weeks

Fix an mdoc(7) macro that is not an option in the provided
description.

Bump Dd.

As CDDL License dictates, update the Copyright accordingly.

Sponsored by: The FreeBSD Foundation

MFV r274273:

ZFS large block support.

Please note that booting from datasets that have recordsize greater
than 128KB is not supported (but it's Okay to enable the feature on
the pool). This *may* remain unchanged because of memory constraint.

Limited safety belt is provided for mounted root filesystem but use
caution is advised.

Illumos issue:
5027 zfs large block support

MFC after: 1 month

MFV r274271:

Improve zdb -b performance:

- Reduce gethrtime() call to 1/100th of blkptr's;
- Skip manipulating the size-ordered tree;
- Issue more (10, previously 3) async reads;
- Use lighter weight testing in traverse_visitbp();

Illumos issue:
5243 zdb -b could be much faster

MFC after: 2 weeks

Remove a build artifact of the USDT tests.

MFC after: 3 days

Document the CPU+Pri_Class column rather than CPU+PRI, as the latter isn't
used or implemented on FreeBSD.

Reported by: kmacy
Reviewed by: kmacy, rpaulo
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

MFV r272802:

- Limit ARC for zdb at 256MB. zdb do not typically revisit data
in the ARC.
- Increase default max_inflight from 200 to 1000 (can be overriden
by -I) so we can queue more I/Os when doing scrubbing.
- Print status while loading meataslabs for leak detection.

Illumos issues:

5169 zdb should limit its ARC size
5170 zdb -c should create more scrub i/os by default
5171 zdb should print status while loading metaslabs for leak detection

MFC after: 2 weeks

MFV r272588:

Handle old format deadlist.

Illumos issue:
5178 zdb -vvvvv on old-format pool fails in dump_deadlist()

MFC after: 2 weeks

MFV r272585:

Split the godfather zio into CPU number's to reduce lock
contention.

Illumos issue:
5176 lock contention on godfather zio

MFC after: 2 weeks

MFV r272493:

Show individual disk capacity when doing zpool list -v.

Illumos issue:
5147 zpool list -v should show individual disk capacity

MFC after: 1 week

Hook up support for userland CTF support in DTrace. This required some
modifications to libproc to support fetching the CTF info for a given file.

With this change, dtrace(1) is able to resolve type info for function and
USDT probe arguments, and function return values. In particular, the args[n]
syntax should now work for referencing arguments of userland probes,
provided that the requisite CTF info is available.

The uctf tests pass if the test programs are compiled with CTF info. The
current infrastructure around the DTrace test suite doesn't support this
yet.

Differential Revision: https://reviews.freebsd.org/D891
MFC after: 1 month
Relnotes: yes
Sponsored by: EMC / Isilon Storage Division

Have dtrace(1) handle SIGPIPE by cleaning up and exiting. Additionally,
install signal handlers when running in list mode (-l), and acknowledge
interrupts by cleaning up and exiting. This ensures that a command like

$ dtrace -l -P 'pid$target' -p <target PID> | less

won't cause the ptrace(2)d target process to be killed if less(1) exits
before all dtrace output is consumed.

Reported by: Anton Yuzhaninov <citrin+bsd@citrin.ru>
Differential Revision: https://reviews.freebsd.org/D880
Reviewed by: rpaulo
MFC after: 1 month
Sponsored by: EMC / Isilon Storage Division

Fix the description of the -h option.

Reported by: swills
MFC after: 3 days

Output boot code warning when zpool upgrade -a is used to add features.

In the case where new features where enabled by a zpool upgrade -a the
boot code warning wasn't output.

Submitted by: Jan Kokemueller
MFC after: 3 days

Fix an assert to tolerate spare parents with more than 2 children.

This can occur if a spare is being spared, which would yield three
children: the original pool drive, the previous spare, and the spare
that is replacing it.

MFC after: 1 week
Sponsored by: Spectra Logic
Affects: All ZFS versions starting 7 Jun 2006 (illumos 94de1d4c)
MFSpectraBSD: r668345 on 2013/06/04 17:10:43

Implement a workaround to allow this test program to be compiled with clang.
It seems that if a pragma is used to define a weak alias for a local
function, the pragma must appear after the function is defined.

PR: 193056
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Fix a number of typos and programming errors in the userland CTF tests. It
seems that they would only pass by chance on illumos; on FreeBSD, they still
fail since userland CTF is not yet supported.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Fix elfdump(1) usage in the userland CTF tests.

PR: 193110
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

MFV r271225:

Iterate through all the children instead of returning error when we hit
the first error. This makes the error message give more information
rather than just the first device that causes problem.

Illumos issue:
5118 When verifying or creating a storage pool, error messages only
show one device

MFC after: 2 weeks

Fix typo.

Submitted by: Dmitry Morozovsky <marck rinet ru>
MFC after: 3 days

Include two headers to provide prototype for modfind(2) and kldload(2).

MFC after: 2 weeks

MFV r269426:

Double test device size for ztest(1).

Illumos issue:
5039 ztest should default to larger device sizes
Author: Matthew Ahrens <mahrens@delphix.com>

MFC after: 2 weeks

MFV r269010:

Import Illumos changes to address the following Illumos issues:
4976 zfs should only avoid writing to a failing non-redundant
top-level vdev
4978 ztest fails in get_metaslab_refcount()
4979 extend free space histogram to device and pool
4980 metaslabs should have a fragmentation metric
4981 remove fragmented ops vector from block allocator
4982 space_map object should proactively upgrade when feature
is enabled
4984 device selection should use fragmentation metric

MFC after: 2 weeks

Diff reduction against Illumos.

MFC after: 2 weeks

MFV r268848:

Instead of asserting all zio's be properly aligned, only assert
on the logical ones.

Cap uberblocks at 8k, otherwise with ashift=17, there would be
only one uberblock.

This fixes a problem that zdb would trip assert on pools with
ashift >= 0xe (8k).

While there, also change the code so it only attempt to condense
space map unless the uncondensed size consumes greater than
zfs_metaslab_condense_block_threshold blocks.

Illumos issue:
4958 zdb trips assert on pools with ashift >= 0xe

MFC after: 2 weeks

MFV r268714:

Improve extreme rewind import.

When doing an "extreme rewind" import ("zpool import -XF"), we attempt
to verify all data in the pool, essentially scrubbing the entire pool.
The problem is that spa_load_verify_cb() issues an unbounded number of
concurrent scrub i/os. This can lead to all of memory being used for
these zio's, wedging the system. Like normal scrub, we need to put a
cap on the number of outstanding i/os, and have the traverse thread
block when we reach this cap.

For this purpose the cap can be very large (10,000) to optimize the
elevator algorithm. Three kernel tunables have been added:

vfs.zfs.spa_load_verify_maxinflight
vfs.zfs.spa_load_verify_metadata
vfs.zfs.spa_load_verify_data

The latter two tunables controls whether metadata and/or user data
when doing extreme rewind.

Make 'zpool import -T' imply scrub.

Make zpool import -T <txg> accept hexadecimal values for the txg when
prefixed with 0x.

Skip txg's for which there is no uberblock when doing extreme rewind.

Skip reading all user data twice by skipping prefetches when doing
extreme rewinds as we do not access via the ARC.

Illumos issues:
4970 need controls on i/o issued by zpool import -XF
4971 zpool import -T should accept hex values
4972 zpool import -T implies extreme rewind, and thus a scrub
4973 spa_load_retry retries the same txg
4974 spa_load_verify() reads all data twice

MFC after: 2 weeks

Bump mdoc date after r268621.

X-MFC-With: r268621

Don't report non-native block-size pools under zpool status -x

zpool status -x is used to identify pools that are exhibiting
errors or are otherwise unavailable, therefore non-native
block-size pools shouldn't be reported.

Also update man page to clarify other additional conditions
which won't cause a pool to be displayed under zpool status -x.

Sponsored by: Multiplay

MFV r268455:

Use reserved space for ZFS administrative commands.

We reserve 1/2^spa_slop_shift = 1/32 or 3.125% of pool space (or 32MB at
least) for system use. Most ZPL operations, e.g. write(2), creat(2), will
fail with ENOSPC if we fall below this.

Certain operations, e.g. file removal and most administrative actions,
still permitted until half of the slop space is used. This would allow
users to use these operations to free up space in the pool when pool is
close to full but half of slop space is still free.

A very restricted set of operations that frees up space or change quota
are always permitted, regardless of the amount of free space.

MFC after: 2 weeks

MFV r268454:

Refresh zpool list for each interval in order to produce fresh
output.

Illumos issue: 4966 zpool list iterator does not update output

MFC after: 2 weeks

MFV r268121:

4924 LZ4 Compression for metadata

illumos/illumos-gate@b8289d24d866c1af02d7007348f7f057693c15d3

MFC after: 2 weeks

MFV r268119:

4914 zfs on-disk bookmark structure should be named *_phys_t

illumos/illumos-gate@7802d7bf98dec568dadf72286893b1fe5abd8602

MFC after: 2 weeks

MFV r267570:

4756 metaslab_group_preload() could deadlock

illumos/illumos-gate@30beaff42d8240ebf5386e8b7a14e3d137a1631f

MFC after: 2 weeks

MFV r267568:

4891 want zdb option to dump all metadata

illumos/illumos-gate@df15e419cb7359ba56ddddab9045e438d89e7cbc

MFC after: 2 weeks

MFV r267566:

4390 i/o errors when deleting filesystem/zvol can lead to space map corruption

MFC after: 2 weeks

MFV r267565:

4757 ZFS embedded-data block pointers ("zero block compression")
4913 zfs release should not be subject to space checks

MFC after: 2 weeks

MFV illumos

4471 DTrace count() with histogram
4472 DTrace full width distribution histograms
4473 DTrace frequency trails

MFC after: 2 weeks

MFV illumos

4474 DTrace Userland CTF Support
4475 DTrace userland Keyword
4476 DTrace tests should be better citizens
4479 pid provider types
4480 dof emulation is missing checks

MFC after: 2 weeks

Record MFV of r266988.

MFC after: 2 weeks

MFV illumos

4477 DTrace should speak JSON

MFC after: 2 weeks

MFV illumos r266986:

2915 DTrace in a zone should see "cpu", "curpsinfo", et al
2916 DTrace in a zone should be able to access fds[]
2917 DTrace in a zone should have limited provider access

MFC after: 2 weeks

Revert r267898.

Bring the following change from the illumos-joyent repository:

commit 78e24ab6803bbe11ba37642624e1498ede5b239d
Author: Bryan Cantrill <bryan@joyent.com>
Date: Thu Oct 31 01:20:54 2013

OS-1688 DTrace count() with histogram
OS-2360 DTrace full width distribution histograms
OS-2361 DTrace frequency trails

MFC after: 2 weeks

mdoc: remove superfluous paragraph macros.

MFV r249332 (illumos-gate 14005:55fc53126003)

Illumos ZFS issues:
3654 zdb should print number of ganged blocks

MFC after: 2 weeks

MFV r266988:

Merge from r258379 missed the tests.

4248 dtrace(1M) should never create DOF with empty probes section
4249 Only probes from the first DTrace object file will be included

Illumos Revision: 54a20ab41aadcb81c53e72fc65886e964e9add59

MFC after: 5 days

MFV r266766:

Add a new zfs property, "redundant_metadata" which can have values "all" or
"most". The default will be "all", which is the current behavior. When set
to all, ZFS stores an extra copy of all metadata. If a single on-disk block
is corrupt, at worst a single block of user data (which is recordsize bytes
long) can be lost.

Setting to "most" will cause us to only store 1 copy of level-1 indirect
blocks of user data files. This can improve performance of random writes,
because less metadata has to be written. In practice, at worst about
100 blocks (of recordsize bytes each) of user data can be lost if a single
on-disk block is corrupt.

The exact behavior of which metadata blocks are stored redundantly may change
in future releases.

Illumos issue: 3835 zfs need not store 2 copies of all metadata

MFC after: 2 weeks

Fix tst.ZeroModuleProbes.d.ksh, which was incorrectly modified in r178534.
Since "BEGIN" is not the name of a module, the test would just hang.

MFC after: 3 days

Comment out some pointless device open/close around reading device IDs.

FreeBSD ZFS port unlike OpenSolaris does not use device IDs, and does not
implement respective devid_*() fuctions. It is pointless to open devices
just to close them back immediately.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Eliminated optarg global being used outside of the function which called getopt

MFC after: 2 weeks

MFV r264829:

3897 zfs filesystem and snapshot limits

MFC after: 2 weeks

MFV r264666:

4374 dn_free_ranges should use range_tree_t

illumos/illumos-gate@bf16b11e8deb633dd6c4296d46e92399d1582df4

MFC after: 2 weeks

Replace a few Solarisisms with their corresponding FreeBSDisms to make a few
printf tests pass.

Take into account when zpool history block grows exceeding 128KB in zpool(8)
and zdb(8) by growing the buffer on demand with a cap of 1GB (specified in
spa_history_create_obj()).

PR: bin/186574
Submitted by: Andrew Childs <lorne cons org nz> (with changes)
MFC after: 2 weeks

Add property and sysctl to control how ZVOLs are exposed to OS.

New ZFS property volmode and sysctl vfs.zfs.vol.mode allow switching ZVOL
between three modes:
geom -- existing fully functional behavior (default);
dev -- exposing volumes only as raw disk device file in devfs;
none -- not exposing volumes outside ZFS.

The "dev" mode is less functional (can't be partitioned, mounted, etc),
but it is faster, and in some scenarios with untrusted consumers safer.
It can be useful for NAS, VM block storages, etc.
The "none" mode may be convenient for backup servers, etc. that don't
need direct data access.

Due to the way ZVOL is integrated with main ZFS code, those property
and sysctl are checked only during pool import and volume creation.

MFC after: 1 month
Sponsored by: iXsystems, Inc.

MFV r263887:

3993 zpool(1M) and zfs(1M) should support -p for "list" and "get"
4700 "zpool get" doesn't support -H or -o options

MFC after: 2 weeks

MFV 263436-263438:

3947 zpool(1M) references nonexistent zfs-features(5)
4540 zpool(1M) man page doesn't describe "readonly" property
3948 zfs sync=default is not accepted
4611 zfs(1M) still mentions 'send -r' in synopsis
4415 zpool(1M) man page missing "import -m" description
4570 Document dedupditto pool property
4572 Dedup-related documentation additions for zpool and zdb.
1371 Add -D option description to zpool(1M) manpage
4571 Add documentation for -T and interval to "zpool list"

MFC after: 2 weeks

Remove unused option -r from zpool.

Submitted by: Richard Yao <ryao gentoo org>
MFC after: 2 weeks

4478 dtrace_dof_maxsize is far too small

illumos/illumos-gate@d339a29bb4765c4b6883a935cf69b669cd05bca0

PR: 187027
MFC after: 1 week

1452 DTrace buffer autoscaling should be less violent

illumos/illumos-gate@6fb4854bed54ce82bd8610896b64ddebcd4af706

This fixes the tst.resize1.d and tst.resize2.d DTrace tests, which have
been failing since r261122 since they were causing dtrace(1) to attempt to
allocate and use large amounts of memory, and get killed by the OOM killer
as a result.

MFC after: 1 month

Fix formatting.

"Manpages should start a new sentence on a new line. This makes it easier
for translators to track changes." -jhb

Approved by: jhb
MFC after: 3 days
Sponsored by: SupraNet Communications, Inc

zfs.8: fix garbled options in a sample zfs send -R command line

MFC after: 5 days

zpool.8: fix typo in option description of labelclear command

MFC after: 5 days

Add caveat to zpool manpage indicating that we do not automatically activate
hot spares. This should be MFC'd to all STABLE branches.

Upon the availability of zfsd, the zpool manpage on relevant branches should
be updated to remove this caveat and document hot spare's reliance on zfsd.

Approved by: avg
MFC after: 1 week
Sponsored by: SupraNet Communications

zdb -R: do not treat numeric parameters to a flag as more flags

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
MFC after: 1 week

zinject must use ioctl(2) compatibility wrapper

MFC after: 8 days
Sponsored by: HybridCluster

MFV r260154 + 260182:

4369 implement zfs bookmarks
4368 zfs send filesystems from readonly pools

Illumos/illumos-gate@78f171005391b928aaf1642b3206c534ed644332

MFC after: 2 weeks

MFV r260152:

4208 Typo in zfs_main.c: "posxiuser"

illumos/illumos-gate@f38cb554a534c6df738be3f4d23327e69888e634

Note: this is a stripped down version of Illumos change.

MFC after: 2 weeks

MFV r259170:

4370 avoid transmitting holes during zfs send

4371 DMU code clean up

illumos/illumos-gate@43466aae47bfcd2ad9bf501faec8e75c08095e4f

NOTE: Make sure the boot code is updated if a zpool upgrade is
done on boot zpool.

MFC after: 2 weeks

MFV r258972:

4373 add block contents print to zstreamdump

illumos/illumos-gate@994fb6b8a9d07a8021d77d79f46e30637bca3ad3

MFC after: 2 weeks

MFV r242733:

3306 zdb should be able to issue reads in parallel
3321 'zpool reopen' command should be documented in the man page
and help message

illumos/illumos-gate@31d7e8fa33fae995f558673adb22641b5aa8b6e1

FreeBSD porting notes: the kernel part of this changeset depends
on Solaris buf(9S) interfaces and are not really applicable for
our use. vdev_disk.c is patched as-is to reduce diverge from
upstream, but vdev_file.c is left intact.

MFC after: 2 weeks

MFV r258384:

2583 Add -p (parsable) option to zfs list

illumos/illumos-gate@43d68d68c1ce08fb35026bebfb141af422e7082e

MFC after: 2 weeks

Fix incorrect markup introduced in r259813.

Pointy hat to: delphij
X-MFC-after: r259813

MFV r258374:

4171 clean up spa_feature_*() interfaces

4172 implement extensible_dataset feature for use by other zpool
features

illumos/illumos-gate@2acef22db7808606888f8f92715629ff3ba555b9

MFC after: 2 weeks

MFV r258373:

4168 ztest assertion failure in dbuf_undirty

4169 verbatim import causes zdb to segfa
4170 zhack leaves pool in ACTIVE state

illumos/illumos-gate@7fdd916c474ea52896c671bbe7b56ba34a1ca132

MFC after: 2 weeks

Don't panic when we get ZPOOL_STATUS_NON_NATIVE_ASHIFT
while listing importable pools.

MFC after: 3 days

mdoc: remove EOL whitespace.

Enable some previously-disabled DTrace tests for umod, ufunc and usym. They
expect the installed ksh binary to be named "ksh", which is not the case
when it's installed on FreeBSD via the shells/ksh93 port. Allow for it to be
"ksh93" as well so that the tests can actually pass.

MFV r258371,r258372: 4101 metaslab_debug should allow for fine-grained control

4101 metaslab_debug should allow for fine-grained control
4102 space_maps should store more information about themselves
4103 space map object blocksize should be increased
4104 ::spa_space no longer works
4105 removing a mirrored log device results in a leaked object
4106 asynchronously load metaslab

illumos/illumos-gate@0713e232b7712cd27d99e1e935ebb8d5de61c57d

Note that some tunables have been removed and some new tunables have
been added. Of particular note, FreeBSD-only knob
vfs.zfs.space_map_last_hope is removed as it was a nop for some time now
(after one of the previous merges from upstream).

MFC after: 11 days
Sponsored by: HybridCluster [merge]

MFV r255255: 4045 zfs write throttle & i/o scheduler performance work

illumos/illumos-gate@69962b5647e4a8b9b14998733b765925381b727e

Please note the following changes:
- zio_ioctl has lost its priority parameter and now TRIM is executed
with 'now' priority
- some knobs are gone and some new knobs are added; not all of them are
exposed as tunables / sysctls yet

MFC after: 10 days
Sponsored by: HybridCluster [merge]

Use 'int' to store the return value of getopt(), rather than char.

On some architectures (powerpc), char is unsigned by default, which means
comparisons against -1 always fail, so the programs get stuck in an
infinite loop.

MFC after: 1 week

If the initial attempt to open /dev/ksyms fails, kldload the ksyms module
and retry.

Convert the lockstat(1) man page to mdoc and make sure that it gets
installed. Additionally, remove Solaris-specific sections and references,
and replace example outputs with output from lockstat on FreeBSD, since
lockstat's output contains stack traces.

This change also removes some examples that don't seem to work properly on
FreeBSD. The examples should be re-added when lockstat is fixed.

Reported by: avg
MFC after: 1 week

Added support for the 'zfs list -t snap' and 'zfs snap' aliases which are
available under Oracle Solaris 11.

This includes an update to the ZFS(8) man page to reflect all the
available alias (snap, umount, and recv).

Initial changes obtained from ZFS On Linux + fixes for man page and cmd
help:
https://github.com/zfsonlinux/zfs/commit/10b75496bb0cb7a7b8146c263164adc37f1d176a
https://github.com/zfsonlinux/zfs/commit/cf81b00a73fe47fdb21586ac1cc179b734540973

Obtained from: https://github.com/zfsonlinux/zfs
MFC after: 2 weeks
Sponsored by: Multiplay

Convert the dtrace(1) man page to mdoc and fix up some aspects of it that
don't make sense on FreeBSD. In particular,

- remove the ATTRIBUTES section,
- remove references to the Solaris Dynamic Tracing Guide, except in the
SEE ALSO section,
- update the description of the -A option for FreeBSD's implementation,
- remove references to Solaris-specific programs and configuration files,
and replace them with FreeBSD equivalents where possible.

The content has not changed aside from this.

Approved by: re (joel)
MFC after: 1 week

Fix implicit declaration of jail_getid()

Approved by: re

MFV r254750:

Add support of Illumos dumps on zvol over RAID-Z.

Note that this only adds the features. FreeBSD would
still need more work to support dumping on zvols.

Illumos ZFS issues:
2932 support crash dumps to raidz, etc. pools

MFC after: 1 month
Approved by: re (ZFS blanket)

MFV r254751:

Don't treat the parameter as a number (pool GUID) when there is
error converting it from string, instead, treat it as the pool
name.

Illumos ZFS issues:
1765 assert triggered in libzfs_import.c trying to import pool
name beginning with a number

Enhance the ZFS vdev layer to maintain both a logical and a physical
minimum allocation size for devices. Use this information to
automatically increase ZFS's minimum allocation size for new top-level
vdevs to a value that more closely matches the optimum device
allocation size.

Use GEOM's stripesize attribute, if set, as the physical sector
size of the GEOM.

Calculate the minimum blocksize of each metaslab class. Use the
calculated value instead of SPA_MINBLOCKSIZE (512b) when determining
the likelyhood of compression yeilding a reduction in physical space
usage.

Report devices with sub-optimal block size configuration in "zpool
status". Also properly fail attempts to attach devices with a
logical block size greater than 8kB, since this will cause corruption
to ZFS's label area.

Sponsored by: Spectra Logic Corporaion
MFC after: 2 weeks

Background
==========
Many modern devices use physical allocation units that are much
larger than the minimum logical allocation size accessible by
external commands. Two prevalent examples of this are 512e disk
drives (512b logical sector, 4K physical sector) and flash devices
(512b logical sector, 4K or larger allocation block size, and 128k
or larger erase block size). Operations that modify less than the
physical sector size result in a costly read-modify-write or garbage
collection sequence on these devices.

Simply exporting the true physical sector of the device to ZFS would
yield optimal performance, but has two serious drawbacks:

1) Existing pools created with devices that have different logical
and physical block sizes, but were configured to use the logical
block size (e.g. because the OS version used for pool construction
reported the logical block size instead of the physical block
size) will suddenly find that the vdev allocation size has
increased. This can be easily tolerated for active members of
the array, but ZFS would prevent replacement of a vdev with
another identical device because it now appears that the smaller
allocation size required by the pool is not supported by the new
device.

2) The device's physical block size may be too large to be supported
by ZFS. The optimal allocation size for the vdev may be quite
large. For example, a RAID controller may export a vdev that
requires read-modify-write cycles unless accessed using 64k
aligned/sized requests. ZFS currently has an 8k minimum block
size limit.

Reporting both the logical and physical allocation sizes for vdevs
solves these problems. A device may be used so long as the logical
block size is compatible with the configuration. By comparing the
logical and physical block sizes, new configurations can be optimized
and administrators can be notified of any existing pools that are
sub-optimal.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h:
Add the SPA_ASHIFT constant. ZFS currently has a hard upper
limit of 13 (8k) for ashift and this constant is used to
both document and enforce this limit.

sys/cddl/contrib/opensolaris/uts/common/sys/fs/zfs.h:
Add the VDEV_AUX_ASHIFT_TOO_BIG error code.

Add fields for exporting the configured, logical, and
physical ashift to the vdev_stat_t structure.

Add VDEV_STAT_VALID() macro which can be used to verify the
presence of required vdev_stat_t fields in nvlist data.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:
Provide a SYSCTL_PROC handler for "max_auto_ashift". Since
the limit is only referenced long after boot when a create
operation occurs, there's no compelling need for it to be
a boot time configurable tunable. This also allows the
validation code for the max_auto_ashift value to be contained
within the sysctl handler.

Populate the new fields in the vdev_stat_t structure.

Fail vdev opens if the vdev reports an ashift larger than
SPA_MAXASHIFT.

Propogate vdev_logical_ashift and vdev_physical_ashift between
child and parent vdevs as is done for vdev_ashift.

In vdev_open(), restore code that fails opens for devices
where vdev_ashift grows. This can only happen now if the
device's logical ashift grows, which means it really isn't
safe to use the device.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/vdev_impl.h:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_file.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_missing.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_raidz.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_root.c:
Update the vdev_open() API so that both logical (what was
just ashift before) and physical ashift are reported.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/vdev_impl.h:
Add two new fields, vdev_physical_ashift and vdev_logical_ashift,
to vdev_t.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa_config.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:
Add vdev_ashift_optimize(). Call it anytime a new top-level
vdev is allocated.

cddl/contrib/opensolaris/cmd/zpool/zpool_main.c:
Add text for the VDEV_AUX_ASHIFT_TOO_BIG error.

For each sub-optimally configured leaf vdev, report configured
and native block sizes.

cddl/contrib/opensolaris/cmd/zpool/zpool_main.c:
cddl/contrib/opensolaris/lib/libzfs/common/libzfs.h:
cddl/contrib/opensolaris/lib/libzfs/common/libzfs_status.c:
Introduce a new zpool status: ZPOOL_STATUS_NON_NATIVE_ASHIFT.
This status is reported on healthy pools containing vdevs
configured to use a block size smaller than their reported
physical block size.

cddl/contrib/opensolaris/lib/libzfs/common/libzfs_status.c:
Update find_vdev_problem() and supporting functions to
provide the full vdev_stat_t structure to problem checking
routines, and to allow decent into replacing vdevs.

Add a vdev_non_native_ashift() validator which is used on
the full vdev tree to check for ZPOOL_STATUS_NON_NATIVE_ASHIFT.

cddl/contrib/opensolaris/lib/libzpool/common/kernel.c:
cddl/contrib/opensolaris/lib/libzpool/common/sys/zfs_context.h:
Enhance sysctl userland stubs now that a SYSCTL_PROC handler
is used in vdev.c.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/metaslab_impl.h:
When the group membership of a metaslab class changes (i.e.
when a vdev is added or removed from a pool), walk the group
list to determine the smallest block size currently available
and record this in the metaslab class.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/metaslab.h:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:
Add the metaslab_class_get_minblocksize() accessor.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio_compress.h:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio_compress.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:
In zio_compress_data(), take the minimum blocksize as an
input parameter instead of assuming SPA_MINBLOCKSIZE.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:
In l2arc_compress_buf(), pass SPA_MINBLOCKSIZE as the minimum
blocksize of the device. The l2arc code performs has it's own
code for deciding if compression is worth while, so this
effectively disables zio_compress_data() from second guessing
the original decision.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:
In zio_write_bp_init(), use the minimum blocksize of the
normal metaslab class when compressing data.

MFV r254079:

Illumos ZFS issues:
3957 ztest should update the cachefile before killing itself
3958 multiple scans can lead to partial resilvering
3959 ddt entries are not always resilvered
3960 dsl_scan can skip over dedup-ed blocks if
physical birth != logical birth
3961 freed gang blocks are not resilvered and can cause pool to suspend
3962 ztest should print out zfs debug buffer before exiting

MFV r254071:

Fix a regression introduced by fix for Illumos bug #3834. Quote from
Matthew Ahrens on the Illumos issue:

ztest fails this assertion because ztest_dmu_read_write() does
dmu_tx_hold_free(tx, bigobj, bigoff, bigsize);
and then
dmu_object_set_checksum(os, bigobj,
(enum zio_checksum)ztest_random_dsl_prop(ZFS_PROP_CHECKSUM), tx);

If the region to free is past the end of the file, the DMU assumes that there
will be nothing to do for this object. However, ztest does set_checksum(),
which must modify the dnode. The fix is for ztest to also call

dmu_tx_hold_bonus(tx, bigobj);

so we can account for the dirty data associated with setting the checksum

Illumos ZFS issues:
3955 ztest failure: assertion refcount_count(&tx->tx_space_written)
+ delta <= tx->tx_space_towrite

MFV r254070:

Merge vendor bugfix for ZFS test suite that triggers false positives.

Illumos ZFS issues:
3949 ztest fault injection should avoid resilvering devices
3950 ztest: deadman fires when we're doing a scan
3951 ztest hang when running dedup test
3952 ztest: ztest_reguid test and ztest_fault_inject don't place nice together

Fix implicit declaration of warnx().

MFV r253781 + r253871:

Illumos ZFS issues:
3894 zfs should not allow snapshot of inconsistent dataset

MFC after: 2 weeks

DTrace: re-apply r249426 now that the underlying issues have been solved.

Merge change from illumos:

3519 DTrace fails to resolve const types from fbt
3520 dtrace internal error -- token type 316 is not a valid D
compilation token
3521 clean up dtrace unit tests

Illumos Revision: e98f46c

Reference:
https://www.illumos.org/issues/3519
https://www.illumos.org/issues/3520
https://www.illumos.org/issues/3521

Tested by: Fabian Keil
Obtained from: Illumos
MFC after: 1 month

DTrace: re-merge remainder of r249367 (original from Illumos).

Bring back some important fixes from Illumos:

3022 DTrace: keys should not affect the sort order when sorting by value
3023 it should be possible to dereference dynamic variables
3024 D integer narrowing needs some work

We particularly avoid the LD_NOLAZYLOAD changes that Illumos made
as those don't apply to FreeBSD and were causing problems in
interactive mode.

Illumos Revision: 13758:23432da34147

Reference:

https://www.illumos.org/issues/3022
https://www.illumos.org/issues/3023
https://www.illumos.org/issues/3024

MFC after: 1 month
Tested by: markj

Use kern_ioctl() rather than ioctl() for testing the FBT provider, since the
latter doesn't exist in FreeBSD. All the tests under fbtprovider pass now.

Manually merge part of vendor import r238583 from Illumos.

Illumos changeset: 13680:2bd022a765e2
Illumos ZFS issue:

2671 zpool import should not fail if vdev ashift has increased

MFC after: 3 days

Fix misleading or remove irrelevant illumos messages and manpage references
in the zfs command.

PR: bin/178996
Submitted by: Peter Schaefer <peter.schaefer@wilhelmheinrichs.de>
MFC after: 3 days

Fixed ZFS zpool freeze (debug command) not processing due to invalid ioctl call syntax.

MFC after: 1 week

MFV r251644:

Poor ZFS send / receive performance due to snapshot
hold / release processing (by smh@)

Illumos ZFS issues:
3740 Poor ZFS send / receive performance due to snapshot
hold / release processing

MFC after: 2 weeks