Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Merge ZFS feature flags support and related bugfixes:
236884, 237001, 237119, 237458, 237972, 238113, 238391, 238422, 238926,
238950, 238951, 239389, 239394, 239620, 239774, 239953, 239958, 239967,
239968, 240063, 240133, 240153, 240303, 240345, 240415, 240955, 241655,
243014, 243505, 243506

MFC r236884:
Introduce "feature flags" for ZFS pools (bump SPA version to 5000).
Add first feature "com.delphix:async_destroy" (asynchronous destroy
of ZFS datasets).
Implement features support in ZFS boot code.

Illumos revisions merged:
13700:2889e2596bd6
13701:1949b688d5fb
2619 asynchronous destruction of ZFS file systems
2747 SPA versioning with zfs feature flags

References:
https://www.illumos.org/issues/2619
https://www.illumos.org/issues/2747

MFC r237001:
Fix ZFS boot with pre-features pools (version <= 28) broken in r236884

MFC r237119 [1]:
Do not remount ZFS dataset if changing canmount property to "on" and
dataset is already mounted.

MFC r237458:
Import Illumos revision 13736:9f1d48e1681f
2901 ZFS receive fails for exabyte sparse files

References:
https://www.illumos.org/issues/2901

MFC r237972:
Expose scrub and resilver tunables.
This allows the user to tune the priority trade-off between scrub/resilver
and other ZFS I/O.

MFC r238113 (pjd):
vdev_io_done stage is not used for ioctls.

MFC r238391:
Change behavior introduced in r237119 to vendor solution

References:
https://www.illumos.org/issues/2883

MFC r238422:
Merge illumos commit 13749:df4cd82e2b60

1796 "ZFS HOLD" should not be used when doing "ZFS SEND" froma read-only pool
2871 support for __ZFS_POOL_RESTRICT used by ZFS test suite
2903 zfs destroy -d does not work
2957 zfs destroy -R/r sometimes fails when removing defer-destroyed snapshot

References:
https://www.illumos.org/issues/1796
https://www.illumos.org/issues/2871
https://www.illumos.org/issues/2903
https://www.illumos.org/issues/2957

MFC r238926:
Partial MFV (illumos-gate 13753:2aba784c276b)
2762 zpool command should have better support for feature flags

References:
https://www.illumos.org/issues/2762

MFC r238950:
Fix reporting of root pool upgrade notice.

MFC r238951:
Fix wrong indent according to style(9)

MFC r239389:
Backport fix for vendor issue #3085
3085 zfs diff panics, then panics in a loop on booting

References:
https://www.illumos.org/issues/3085

MFC r239394:
Update zfs(8) manpage with illumos version of "zfs diff"

Illumos issue:
2399 zfs manual page does not document use of "zfs diff"

References:
https://www.illumos.org/issues/2399

MFC r239620 [2]:
Merge recent vendor changes:
3086 unnecessarily setting DS_FLAG_INCONSISTENT on async destroyed datasets
3090 vdev_reopen() during reguid causes vdev to be treated as corrupt
3102 vdev_uberblock_load() and vdev_validate() may read the wrong label

Referenes:
https://www.illumos.org/issues/3086
https://www.illumos.org/issues/3090
https://www.illumos.org/issues/3102

MFC r239774:
Merge recent vendor changes:
3100 zvol rename fails with EBUSY when dirty
3104 eliminate empty bpobjs
3120 zinject hangs in zfsdev_ioctl() due to uninitialized zc

References:
https://www.illumos.org/issues/3100
https://www.illumos.org/issues/3104
https://www.illumos.org/issues/3120

MFC r239953 (joel):
Mdoc fixes.

MFC r239958 (joel):
Minor mdoc fixes.

MFC r239967 (joel):
Mdoc fixes.

MFC r239968 (joel):
Remove trailing whitespace.

MFC r240063 (gjb):
Add myself to copyright sections, per CDDL license.

MFC r240133:
Merge recent vendor changes and sync code:
1862 incremental zfs receive fails for sparse file > 8PB
3112 ztest does not honor ZFS_DEBUG
3122 zfs destroy filesystem should prefetch blocks
3129 'zpool reopen' restarts resilvers
3130 ztest failure: Assertion failed:
0 == dmu_objset_destroy(name, B_FALSE) (0x0 == 0x10)

References:
https://www.illumos.org/issues/1862
https://www.illumos.org/issues/3112
https://www.illumos.org/issues/3122
https://www.illumos.org/issues/3129
https://www.illumos.org/issues/3130

MFC r240153 (gjb) [3]:
Typo fix and minor word swap.

MFC r240303:
Add assfail() and assfail3() to the opensolaris module.
Remove obsoleted intermediate cddl/compat/opensolaris/sys/debug.h.

MFC r240345 (avg):
zfs: fix sa_modify_attrs handling of variable-sized attributes

- skip length_idx index for a replaced variable-sized attribute
- skip length_idx index for a removed variable-sized attribute
- also re-arranged code to make sure that length_idx is always
incremented for variable-sized attributes
- additionally add an assertion that the number of actually produced
attributes is the same as the expected number of resulting
attributes

MFC r240415:
Merge recent zfs vendor changes, sync code and adjust userland DEBUG.

Illumos issued covered:
1884 Empty "used" field for zfs *space commands
3006 VERIFY[S,U,P] and ASSERT[S,U,P] frequently check if first argument
is zero
3028 zfs {group,user}space -n prints (null) instead of numeric GID/UID
3048 zfs {user,group}space [-s|-S] is broken
3049 zfs {user,group}space -t doesn't really filter the results
3060 zfs {user,group}space -H output isn't tab-delimited
3061 zfs {user,group}space -o doesn't use specified fields order
3064 usr/src/cmd/zpool/zpool_main.c misspells "successful"
3093 zfs {user,group}space's -i is noop
3098 zfs userspace/groupspace fail without saying why when run as non-root

References:
https://www.illumos.org/issues/ + [issue_id]

MFC r240955 (partial):
Merge recent vendor changes in ZFS.

Illumos issued covered:
3139 zdb dies when it tries to determine path of unlinked file
3189 kernel panic in ZFS test suite during hotspare_onoffline_004_neg
3208 moving zpool cross-endian results in incorrect user/group accounting

References:
https://www.illumos.org/issues/ + [issue_id]

MFC r241655:
Add missing initialization for do_prefix.
Corrects porting error in r238391

Vendor issue and changeset reference:
2883 changing "canmount" property to "on" should not always remount dataset
https://www.illumos.org/issues/2883
Changeset 13743:95aba6e49b9f

MFC r243014:
Move zpool-features manual page from section 5 to section 7
and fix references

Reported by: pluknet

MFC r243505:
Illumos 13886:e3261d03efbf

3349 zpool upgrade -V bumps the on disk version number, but leaves
the in core version

References:
https://www.illumos.org/issues/3349

MFC r243506:
zfs sha256 checksum is missing in zfs.8 manpage

PR: kern/167905 [1], kern/170912 [2], kern/170914 [2], doc/171356 [3]

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Finally... Import the latest open-source ZFS version - (SPA) 28.

Few new things available from now on:

- Data deduplication.
- Triple parity RAIDZ (RAIDZ3).
- zfs diff.
- zpool split.
- Snapshot holds.
- zpool import -F. Allows to rewind corrupted pool to earlier
transaction group.
- Possibility to import pool in read-only mode.

MFC after: 1 month

Forced commit to note that these files were repo copied.

MFp4:

@118370 Correct typo.

@118371 Integrate changes from vendor.

@118491 Show backtrace on unexpected code paths.

@118494 Integrate changes from vendor.

@118504 Fix sendfile(2). I had two ways of fixing it:
1. Fixing sendfile(2) itself to use VOP_GETPAGES() instead of
hacking around with vn_rdwr(UIO_NOCOPY), which was suggested
by ups.
2. Modify ZFS behaviour to handle this special case.

Although 1 is more correct, I've choosen 2, because hack from 1
have a side-effect of beeing faster - it reads ahead MAXBSIZE
bytes instead of reading page by page. This is not easy to implement
with VOP_GETPAGES(), at least not for me in this very moment.

Reported by: Andrey V. Elsukov <bu7cher@yandex.ru>

@118525 Reorganize the code to reduce diff.

@118526 This code path is expected. It is simply when file is opened with
O_FSYNC flag.

Reported by: kris
Reported by: Michal Suszko <dry@dry.pl>

Please welcome ZFS - The last word in file systems.

ZFS file system was ported from OpenSolaris operating system. The code in under
CDDL license.

I'd like to thank all SUN developers that created this great piece of software.

Supported by: Wheel LTD (http://www.wheel.pl/)
Supported by: The FreeBSD Foundation (http://www.freebsdfoundation.org/)
Supported by: Sentex (http://www.sentex.net/)