#
296373 |
|
04-Mar-2016 |
marius |
- Copy stable/10@296371 to releng/10.3 in preparation for 10.3-RC1 builds. - Update newvers.sh to reflect RC1. - Update __FreeBSD_version to reflect 10.3. - Update default pkg(8) configuration to use the quarterly branch.
Approved by: re (implicit) |
#
290766 |
|
13-Nov-2015 |
mav |
MFC r290191 (by avg): l2arc: do not call trim_map_free() for blocks with zero b_asize
b_asize can be zero if the block is compressed into an empty block (ZIO_COMPRESS_EMPTY) and the trim code asserts that meaningless zero-sized trimming is not attempted. The logic for calling trim_map_free() is extracted into a new function l2arc_trim() to minimize code duplication.
PR: 203473 Reported by: Willem Jan Withagen <wjw@digiware.nl> Tested by: Willem Jan Withagen <wjw@digiware.nl>
|
#
290757 |
|
13-Nov-2015 |
mav |
MFC r289422: 4185 add new cryptographic checksums to ZFS: SHA-512, Skein, Edon-R
Reviewed by: George Wilson <george.wilson@delphix.com> Reviewed by: Prakash Surya <prakash.surya@delphix.com> Reviewed by: Saso Kiselkov <saso.kiselkov@nexenta.com> Reviewed by: Richard Lowe <richlowe@richlowe.net> Approved by: Garrett D'Amore <garrett@damore.org> Author: Matthew Ahrens <mahrens@delphix.com>
illumos/illumos-gate@45818ee124adeaaf947698996b4f4c722afc6d1f
This is only a partial merge of respective ZFS infrastructure changes. At this moment FreeBSD kernel has no those crypto algorithms, so the parts of the code to enable them are commented out. When they are implemented, it will be trivial to plug them in.
|
#
290752 |
|
13-Nov-2015 |
mav |
MFC r289305: 6293 ztest failure: error == 28 (0xc == 0x1c) in ztest_tx_assign()
Reviewed by: George Wilson <george.wilson@delphix.com> Reviewed by: Prakash Surya <prakash.surya@delphix.com> Reviewed by: Richard Elling <Richard.Elling@RichardElling.com> Approved by: Richard Lowe <richlowe@richlowe.net> Author: Matthew Ahrens <mahrens@delphix.com>
illumos/illumos-gate@8fe00bfb8790ad51653f67b01d5ac14256cbb404
|
#
290749 |
|
13-Nov-2015 |
mav |
MFC r289295: 5219 l2arc_write_buffers() may write beyond target_sz
Reviewed by: Matthew Ahrens <mahrens@delphix.com> Reviewed by: Saso Kiselkov <skiselkov@gmail.com> Reviewed by: George Wilson <george@delphix.com> Reviewed by: Steven Hartland <steven.hartland@multiplay.co.uk> Reviewed by: Justin Gibbs <gibbs@FreeBSD.org> Approved by: Matthew Ahrens <mahrens@delphix.com> Author: Andriy Gapon <avg@freebsd.org>
illumos/illumos-gate@d7d9a6d919f92d74ea0510a53f8441396048e800
|
#
288599 |
|
03-Oct-2015 |
mav |
MFC r288064 (by avg): 6220 memleak in l2arc on debug build
illumos/illumos-gate/commit/c546f36aa898d913ff77674fb5ff97f15b2e08b4 https://www.illumos.org/issues/6220 5408 introduced a memleak in l2arc, namely the member b_thawed gets leaked when an arc_hdr is realloced from full to l2only.
Reviewed by: Saso Kiselkov <saso.kiselkov@nexenta.com> Reviewed by: Simon Klinkert <simon.klinkert@gmail.com> Reviewed by: George Wilson <george@delphix.com> Approved by: Robert Mustacchi <rm@joyent.com> Author: Arne Jansen <sensille@gmx.net>
|
#
288596 |
|
03-Oct-2015 |
mav |
MFC r287706 (by delphij): 6214 zpools going south
In r286570 (MFV of r277426) an unprotected write to b_flags to set the compression mode was introduced. This would open a race window where data is partially decompressed, modified, checksummed and written to the pool, resulting in pool corruption due to the partial decompression.
Prevent this by reintroducing b_compress
illumos/illumos-gate@d4cd038c92c36fd0ae35945831a8fc2975b5272c
Illumos issues:
6214 zpools going south https://www.illumos.org/issues/6214
|
#
288594 |
|
03-Oct-2015 |
mav |
MFC r287702: 5987 zfs prefetch code needs work
Rewrite the ZFS prefetch code to detect only forward, sequential streams.
The following kstats have been added:
kstat.zfs.misc.arcstats.sync_wait_for_async
How many sync reads have waited for async read to complete. (less is better)
kstat.zfs.misc.arcstats.demand_hit_predictive_prefetch
How many demand read didn't have to wait for I/O because of predictive prefetch. (more is better)
zfetch kstats have been similified to hits, misses, and max_streams, with max_streams representing times when we were not able to create new stream because we already have the maximum number of sequences for a file.
The sysctl variable/loader tunable vfs.zfs.zfetch.block_cap have been replaced by vfs.zfs.zfetch.max_distance, which controls maximum bytes to prefetch per stream.
illumos/illumos-gate@cf6106c8a0d6598b045811f9650d66e07eb332af
Illumos ZFS issues:
5987 zfs prefetch code needs work https://www.illumos.org/issues/5987
|
#
288592 |
|
03-Oct-2015 |
mav |
MFC r287283 (by delphij): Fix a buffer overrun which may lead to data corruption, introduced in r286951 by reinstating changes in r274628.
In l2arc_compress_buf(), we allocate a buffer to stash away the compressed data in 'cdata', allocated of l2hdr->b_asize bytes.
We then ask zio_compress_data() to compress the buffer, b_l1hdr.b_tmp_cdata, which is of l2hdr->b_asize bytes, and have the compressed size (or original size, if compress didn't gain enough) stored in csize.
To pad the buffer to fit the optimal write size, we round up the compressed size to L2 device's vdev_ashift.
Illumos code rounds up the size by at most SPA_MINBLOCKSIZE. Because we know csize <= b_asize, and b_asize is integer multiple of SPA_MINBLOCKSIZE, we are guaranteed that the rounded up csize would be <= b_asize. However, this is not necessarily true when we round up to 1 << vdev_ashift, because it could be larger than SPA_MINBLOCKSIZE.
So, in the worst case scenario, we are overwriting at most
(1 << vdev_ashift - SPA_MINBLOCKSIZE)
bytes of memory next to the compressed data buffer.
Andriy's original change in r274628 reorganized the code a little bit, by moving the padding to after we determined that the compression was beneficial. At which point, we would check rounded size against the allocated buffer size, and the buffer overrun would not be possible.
|
#
288588 |
|
03-Oct-2015 |
mav |
MFC r286951: Restore part of r274628, reverted at r286776.
|
#
288587 |
|
03-Oct-2015 |
mav |
MFC r286776: Remove some random accumulated diff from Illumos.
|
#
288586 |
|
03-Oct-2015 |
mav |
MFC r286774: 2618 arc.c mistypes in the comments
Reviewed by: Jason King <jason.brian.king@gmail.com> Reviewed by: Josef Sipek <jeffpc@josefsipek.net> Approved by: Richard Lowe <richlowe@richlowe.net> Author: Bart Coddens <bart.coddens@gmail.com>
illumos/illumos-gate@fc98fea58e89224f6f13d7fae246d6cb5dfa35ea
|
#
288585 |
|
03-Oct-2015 |
mav |
MFC r286770: Fix r286766 build with debug.
|
#
288584 |
|
03-Oct-2015 |
mav |
MFC r286767: Fix minor mismerge sometimes earlier.
|
#
288583 |
|
03-Oct-2015 |
mav |
MFC r286766: 5817 change type of arcs_size from uint64_t to refcount_t
Reviewed by: Matthew Ahrens <mahrens@delphix.com> Reviewed by: Paul Dagnelie <paul.dagnelie@delphix.com> Reviewed by: Adam Leventhal <ahl@delphix.com> Reviewed by: Alex Reece <alex@delphix.com> Reviewed by: Richard Elling <richard.elling@richardelling.com> Approved by: Garrett D'Amore <garrett@damore.org> Author: Prakash Surya <prakash.surya@delphix.com>
illumos/illumos-gate@2fd872a734cf486007a8dba532cec52bfb4d40e5
As a way to make it more difficult to introduce bugs into the ARC, and to make it easier to diagnose issues when bugs do creep in, it would be beneficial to change the type of the arc_state_t's arcs_size field to be a refcount_t instead of a uint64_t. This would allow us to make stricter checks when incrementing and decrementing the value with debugging enabled, but still fallback to simple, fast atomic operations when debugging is disabled.
|
#
288582 |
|
03-Oct-2015 |
mav |
MFC r286764: 6033 arc_adjust() should search MFU lists for oldest buffer when adjusting MFU size.
illumos/illumos-gate@31c46cf23cd1cf4d66390a983dc5072d7d299ba2
https://www.illumos.org/issues/6033 When we're looking for the list containing oldest buffer we never actually look at the MFU lists even when we try to evict from MFU. looks like a copy paste error, the fix is here:
Reviewed by: Saso Kiselkov <saso.kiselkov@nexenta.com> Reviewed by: Xin Li <delphij@delphij.net> Reviewed by: Prakash Surya <me@prakashsurya.com> Approved by: Matthew Ahrens <mahrens@delphix.com> Author: Alek Pinchuk <alek@nexenta.com> Obtained from: illumos
|
#
288581 |
|
03-Oct-2015 |
mav |
MFC r286763: 5497 lock contention on arcs_mtx
Reviewed by: George Wilson <george.wilson@delphix.com> Reviewed by: Matthew Ahrens <mahrens@delphix.com> Reviewed by: Richard Elling <richard.elling@richardelling.com> Approved by: Dan McDonald <danmcd@omniti.com> Author: Prakash Surya <prakash.surya@delphix.com>
illumos/illumos-gate@244781f10dcd82684fd8163c016540667842f203
This patch attempts to reduce lock contention on the current arc_state_t mutexes. These mutexes are used liberally to protect the number of LRU lists within the ARC (e.g. ARC_mru, ARC_mfu, etc). The granularity at which these locks are acquired has been shown to greatly affect the performance of highly concurrent, cached workloads.
|
#
288580 |
|
03-Oct-2015 |
mav |
MFC r286762: Revert part of r205231, introducing multiple ARC state locks.
This local implementation will be replaced by one from Illumos to reduce code divergence and make further merges easier.
|
#
288566 |
|
03-Oct-2015 |
mav |
MFC r286655: Fix set of sign extension bugs in r286625.
|
#
288565 |
|
03-Oct-2015 |
mav |
MFC r286647: Fix assertion panic caused by combination of r286598 and TRIM.
|
#
288564 |
|
03-Oct-2015 |
mav |
MFC r286628: Fix r286625 build on i386.
|
#
288563 |
|
03-Oct-2015 |
mav |
MFC r286626: Fix minor mismerge in r286574.
|
#
288562 |
|
03-Oct-2015 |
mav |
MFC r286625: 5376 arc_kmem_reap_now() should not result in clearing arc_no_grow
Reviewed by: Christopher Siden <christopher.siden@delphix.com> Reviewed by: George Wilson <george.wilson@delphix.com> Reviewed by: Steven Hartland <killing@multiplay.co.uk> Reviewed by: Richard Elling <richard.elling@richardelling.com> Approved by: Dan McDonald <danmcd@omniti.com> Author: Matthew Ahrens <mahrens@delphix.com>
illumos/illumos-gate@2ec99e3e987d8aa273f1e9ba2b983557d058198c
|
#
288561 |
|
03-Oct-2015 |
mav |
MFC r286623: Remove extra lock, that IMO only creates potential problems now.
|
#
288557 |
|
03-Oct-2015 |
mav |
MFC r286598: 5701 zpool list reports incorrect "alloc" value for cache devices
|
#
288550 |
|
03-Oct-2015 |
mav |
MFC r286576: Fix r286570 build with debug.
|
#
288548 |
|
03-Oct-2015 |
mav |
MFC r286574: 5445 Add more visibility via arcstats; specifically arc_state_t stats and differentiate between "data" and "metadata"
Reviewed by: Basil Crow <basil.crow@delphix.com> Reviewed by: George Wilson <george.wilson@delphix.com> Reviewed by: Matthew Ahrens <mahrens@delphix.com> Reviewed by: Bayard Bell <bayard.bell@nexenta.com> Approved by: Robert Mustacchi <rm@joyent.com> Author: Prakash Surya <prakash.surya@delphix.com>
illumos/illumos-gate@4076b1bf41cfd9f968a33ed54a7ae76d9e996fe8
|
#
288547 |
|
03-Oct-2015 |
mav |
MFC r286570: 5408 managing ZFS cache devices requires lots of RAM Reviewed by: Christopher Siden <christopher.siden@delphix.com> Reviewed by: George Wilson <george.wilson@delphix.com> Reviewed by: Matthew Ahrens <mahrens@delphix.com> Reviewed by: Don Brady <dev.fs.zfs@gmail.com> Reviewed by: Josef 'Jeff' Sipek <josef.sipek@nexenta.com> Approved by: Garrett D'Amore <garrett@damore.org> Author: Chris Williamson <Chris.Williamson@delphix.com>
illumos/illumos-gate@89c86e32293a30cdd7af530c38b2073fee01411c
Currently, every buffer cached in the L2ARC is accompanied by a 240-byte header in memory, leading to very high memory consumption when using very large cache devices. These changes significantly reduce this overhead.
Currently:
L1-only header = 176 bytes L1 + L2 or L2-only header = 176 bytes + 32 byte checksum + 32 byte l2hdr = 240 bytes
Memory-optimized:
L1-only header = 176 bytes L1 + L2 header = 176 bytes + 32 byte checksum = 208 bytes L2-only header = 96 bytes + 32 byte checksum = 128 bytes
So overall:
Trunk Optimized +-----------------+ L1-only | 176 B | 176 B | (same) +-----------------+ L1 & L2 | 240 B | 208 B | (saved 32 bytes) +-----------------+ L2-only | 240 B | 128 B | (saved 116 bytes) +-----------------+
For an average blocksize of 8KB, this means that for the L2ARC, the ratio of metadata to data has gone down from about 2.92% to 1.56%. For a 'storage optimized' EC2 instance with 1600GB of SSD and 60GB of RAM, this means that we expect a completely full L2ARC to use (1600 GB * 0.0156) / 60GB = 41% of the available memory, down from 78%.
Relnotes: yes
|
#
288536 |
|
03-Oct-2015 |
mav |
MFC r281109: Add DTrace probe to the new ARC reclaim cause added in r281026.
|
#
288519 |
|
02-Oct-2015 |
mav |
MFC r277826 (by delphij): Diff reduction with upstream. The actual change was merged in r272483 already.
|
#
288518 |
|
02-Oct-2015 |
mav |
MFC r277452 (by will): Fix arc__shrink DTrace probe's to_free argument.
Remove the unnecessary #ifdef _KERNEL, which did not differ in the true or false cases. Actually set the value of to_free before using it.
|
#
288517 |
|
02-Oct-2015 |
mav |
MFC r275780 (by delphij): Add a loader tunable, vfs.zfs.arc_meta_min, which controls how much metadata ZFS should keep in ARC at minimum.
In arc_evict(), when doing recycle, take more factors into account by applying the following policy:
1. If no evictable data, evict metadata; 2. If no evictable metadata, evict data; 3. If we hit arc_meta_limit, evict metadata; 4. If we haven't hit arc_meta_min, evict data; 5* (Illumos only, not present in new FreeBSD code, yet) evict the oldest cached element from data and metadata. (FreeBSD) evict the data type specified by caller, which is the existing behavior.
Note that because of our splitted locks (implemented in r205231 to improve scalability by reducing lock contention), implementing the fifth Illumos behavior will not be cheap, so for now just implement the 1-4 and fall back to current behavior for 5.
Illumos issue: 5368 ARC should cache more metadata
|
#
287665 |
|
11-Sep-2015 |
avg |
MFC r287099: account for ashift when gathering buffers to be written to l2arc device
The change differs from that in head because of other changes that have not been MFC-ed yet.
|
#
287656 |
|
11-Sep-2015 |
avg |
MFC r284513: l2arc: pass correct size to trim requests
|
#
285717 |
|
20-Jul-2015 |
jpaetzel |
MFC 278040:
Prevent inlining txg_quiesce
This allows dtrace to monitor the calls to txg_quiesce which can be really helpful.
Also standardize __noinline order for arc_kmem_reap_now.
Sponsored by: Multiplay
Approved by: re
|
#
282361 |
|
03-May-2015 |
mav |
MFC r281026, r281108, r281109: Make ZFS ARC track both KVA usage and fragmentation.
Even on Illumos, with its much larger KVA, ZFS ARC steps back if KVA usage reaches certain threshold (3/4 on i386 or 16/17 otherwise). FreeBSD has even less KVA, but had no such limit on archs with direct map as amd64. As result, on machines with a lot of RAM, during load with very small user- space memory pressure, such as `zfs send`, it was possible to reach state, when there is enough both physical RAM and KVA (I've seen up to 25-30%), but no continuous KVA range to allocate even single 128KB I/O request.
Address this situation from two sides: - restore KVA usage limitations in a way the most close to Illumos; - introduce new requirement for KVA fragmentation, specifying that we should have at least one sequential KVA range of zfs_max_recordsize bytes.
Experiments show that first limitation done alone is not sufficient. On machine with 64GB of RAM it is sometimes needed to drop up to half of ARC size to get at leats one 1MB KVA chunk. Statically limiting ARC to half of KVA/RAM is too strict, so second limitation makes it to work in cycles: accumulate trash up to certain critical mass, do massive spring-cleaning, and then start littering again.
|
#
281104 |
|
05-Apr-2015 |
mav |
MFC r280822: Some cosmetic polishing. No functional change.
|
#
277586 |
|
23-Jan-2015 |
delphij |
MFC r275811: MFV r275783:
Convert ARC flags to use enum. Previously, public flags are defined in arc.h and private flags are defined in arc.c which can lead to confusion and programming errors.
Consistently use 'hdr' (when referencing arc_buf_hdr_t) instead of 'buf' or 'ab' because arc_buf_t are often named 'buf' as well.
Illumos issue: 5369 arc flags should be an enum 5370 consistent arc_buf_hdr_t naming scheme
|
#
277583 |
|
23-Jan-2015 |
delphij |
MFC r275748: MFV r247174:
Expose arc_meta_limit, et al via kstats.
Note that as a result, vfs.zfs.arc_meta_used is removed. The existing vfs.zfs.arc_meta_limit sysctl/tunable is retained with a SYSCTL_PROC wrapper.
Illumos ZFS issues: 3561 arc_meta_limit should be exposed via kstats
Relnotes: yes
|
#
275609 |
|
08-Dec-2014 |
avg |
MFC r274628: l2arc: restore correct rounding up of asize of compressed data
|
#
275492 |
|
05-Dec-2014 |
delphij |
MFC r274172 (avg)
fix l2arc compression buffers leak
We have observed that arc_release() can be called concurrently with a l2arc in-flight write. Also, we have observed that arc_hdr_destroy() can be called from arc_write_done() for a zio with ZIO_FLAG_IO_REWRITE flag in similar circumstances.
Previously the l2arc headers would be freed while leaking their associated compression buffers. Now the buffers are placed on l2arc_free_on_write list for delayed freeing. This is similar to what was already done to arc buffers that were supposed to be freed concurrently with in-flight writes of those buffers.
In addition to fixing the discovered leaks this change also adds some protective code to assert that a compression buffer associated with a l2arc header is never leaked.
A new kstat l2_cdata_free_on_write is added. It keeps a count of delayed compression buffer frees which previously would have been leaks.
Tested by: Vitalij Satanivskij <satan@ukr.net> et al Requested by: many Sponsored by: HybridCluster / ClusterHQ
This is a 10.1-RELEASE errata candidate.
|
#
274625 |
|
17-Nov-2014 |
avg |
MFC r272708: l2arc_write_buffers: reduce headroom value
|
#
273984 |
|
02-Nov-2014 |
delphij |
MFC r273026:
Add a tunable for arc_shrink_shift (vfs.zfs.arc_shrink_shift) that controls how much fraction, 1/2^arc_shrink_shift, should be reclaimed when there is memory pressure.
Submitted by: Richard Kojedzinszky <krichy at tvnetwork.hu>
|
#
273194 |
|
16-Oct-2014 |
delphij |
MFC r272527:
Don't make nested definition for range_seg_cache.
Reported by: ian
|
#
273193 |
|
16-Oct-2014 |
delphij |
MFC r272506: MFV r272495:
In arc_kmem_reap_now(), reap range_seg_cache too to reclaim memory in response of memory pressure.
Illumos issue: 5163 arc should reap range_seg_cache
|
#
273191 |
|
16-Oct-2014 |
delphij |
MFV r273060:
Use write_psize instead of write_asize when doing vdev_space_update. Without this change the accounting of L2ARC usage would be wrong and give 16EB free space because the number became negative and overflows.
Obtained from: FreeNAS (issue #6239)
|
#
272879 |
|
10-Oct-2014 |
smh |
MFC r271754: Remove unused ZFS ARC functions
Sponsored by: Multiplay
|
#
272875 |
|
10-Oct-2014 |
smh |
MFC r270759: Refactor ZFS ARC reclaim logic to be more VM cooperative
MFC r270861: Ensure that ZFS ARC free memory checks include cached pages
MFC r272483: Refactor ZFS ARC reclaim checks and limits
Sponsored by: Multiplay
|
#
269846 |
|
12-Aug-2014 |
delphij |
MFC r269230: MFV r269224:
Increase default ARC buf_hash_table size. When typical block size is small, the hash table could be too small, which would lead to long hash chains and limit performance for cached reads.
A new loader tunable, vfs.zfs.arc_average_blocksize, have been added which allows users to override the default assumption of average (typical) block size. Old default was 65536 (64 KiB) and new default is 8192 (8 KiB).
Illumos issue: 5034 ARC's buf_hash_table is too small
|
#
269732 |
|
08-Aug-2014 |
delphij |
MFC r269086:
As of r268075, the responsibility of rounding up buffer to optimal size have been transferred from zio_compress_data to its caller. Therefore, passing the 'minblocksize' down will be a no-op.
Eliminate the parameter to reduce diff against upstream.
|
#
269417 |
|
02-Aug-2014 |
delphij |
MFC r268858: MFV r268850:
Change the interaction between the DMU and ARC so that when the DMU is shutting down an objset, we do not evict the data from the ARC. Instead we simply coordinate the destruction of the DMU's data with the ARC.
The only case where we actually need to explicitly evict from the ARC is when dbuf_rele_and_unlock() determines that the administrator has requested that it not be kept in memory, via the primarycache/secondarycache properties. In this case, we evict the data from the ARC by its blkptr_t, the same way as when a block is freed we explicitly evict it from the ARC.
Illumos issue: 4631 zvol_get_stats triggering too many reads
|
#
268657 |
|
15-Jul-2014 |
delphij |
MFC r268123: MFV r268119:
4914 zfs on-disk bookmark structure should be named *_phys_t
|
#
268654 |
|
15-Jul-2014 |
delphij |
MFC r268085: MFV r267569:
4897 Space accounting mismatch in L2ARC/zpool
|
#
268649 |
|
15-Jul-2014 |
delphij |
MFC r268075: MFV r267565:
4757 ZFS embedded-data block pointers ("zero block compression") 4913 zfs release should not be subject to space checks
|
#
263397 |
|
19-Mar-2014 |
delphij |
MFC r260150: MFV r259170:
4370 avoid transmitting holes during zfs send
4371 DMU code clean up
illumos/illumos-gate@43466aae47bfcd2ad9bf501faec8e75c08095e4f
NOTE: Make sure the boot code is updated if a zpool upgrade is done on boot zpool.
|
#
262115 |
|
17-Feb-2014 |
avg |
MFC r260835: MFV r260834: Fix memory leak of compressed buffers in l2arc_write_done
|
#
260763 |
|
16-Jan-2014 |
avg |
MFC r258632,258704: MFV r255255: 4045 zfs write throttle & i/o scheduler performance work
Sponsored by: HybridCluster [merge]
|
#
258566 |
|
25-Nov-2013 |
avg |
MFV r258378: 4089 NULL pointer dereference in arc_read()
illumos/illumos-gate@57815f6b95a743697e148327725b7f568e75e6ea
Tested by: adrian Approved by: re (gjb)
|
#
258565 |
|
25-Nov-2013 |
avg |
MFV r258377: 4088 use after free in arc_release()
illumos/illumos-gate@ccc22e130479b5bd7c0002267fee1e0602d3f772
Approved by: re (gjb)
|
#
257058 |
|
24-Oct-2013 |
smh |
MFC r256889:
Use the vdev's ashift to calculate the supported min block size passed to zio_compress_data(..) when compressing l2arc buffers.
This eliminates L2ARC I/O errors, which resulted in very poor performance on vdev's configured with block size greater than 512b due to compression assuming a smaller min block size than the vdev supports.
Approved by: re (glebius)
|
#
256281 |
|
10-Oct-2013 |
gjb |
Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.
Approved by: re (implicit) Sponsored by: The FreeBSD Foundation
|
#
255753 |
|
21-Sep-2013 |
gibbs |
Optimize the block size used on ZFS cache devices as is already done for data and log devices.
Reported by: Dmitryy Makarov Submitted by: smh Reviewed by: gibbs Approved by: re (delphij) MFC after: 2 weeks
|
#
254591 |
|
21-Aug-2013 |
gibbs |
Enhance the ZFS vdev layer to maintain both a logical and a physical minimum allocation size for devices. Use this information to automatically increase ZFS's minimum allocation size for new top-level vdevs to a value that more closely matches the optimum device allocation size.
Use GEOM's stripesize attribute, if set, as the physical sector size of the GEOM.
Calculate the minimum blocksize of each metaslab class. Use the calculated value instead of SPA_MINBLOCKSIZE (512b) when determining the likelyhood of compression yeilding a reduction in physical space usage.
Report devices with sub-optimal block size configuration in "zpool status". Also properly fail attempts to attach devices with a logical block size greater than 8kB, since this will cause corruption to ZFS's label area.
Sponsored by: Spectra Logic Corporaion MFC after: 2 weeks
Background ========== Many modern devices use physical allocation units that are much larger than the minimum logical allocation size accessible by external commands. Two prevalent examples of this are 512e disk drives (512b logical sector, 4K physical sector) and flash devices (512b logical sector, 4K or larger allocation block size, and 128k or larger erase block size). Operations that modify less than the physical sector size result in a costly read-modify-write or garbage collection sequence on these devices.
Simply exporting the true physical sector of the device to ZFS would yield optimal performance, but has two serious drawbacks:
1) Existing pools created with devices that have different logical and physical block sizes, but were configured to use the logical block size (e.g. because the OS version used for pool construction reported the logical block size instead of the physical block size) will suddenly find that the vdev allocation size has increased. This can be easily tolerated for active members of the array, but ZFS would prevent replacement of a vdev with another identical device because it now appears that the smaller allocation size required by the pool is not supported by the new device.
2) The device's physical block size may be too large to be supported by ZFS. The optimal allocation size for the vdev may be quite large. For example, a RAID controller may export a vdev that requires read-modify-write cycles unless accessed using 64k aligned/sized requests. ZFS currently has an 8k minimum block size limit.
Reporting both the logical and physical allocation sizes for vdevs solves these problems. A device may be used so long as the logical block size is compatible with the configuration. By comparing the logical and physical block sizes, new configurations can be optimized and administrators can be notified of any existing pools that are sub-optimal.
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h: Add the SPA_ASHIFT constant. ZFS currently has a hard upper limit of 13 (8k) for ashift and this constant is used to both document and enforce this limit.
sys/cddl/contrib/opensolaris/uts/common/sys/fs/zfs.h: Add the VDEV_AUX_ASHIFT_TOO_BIG error code.
Add fields for exporting the configured, logical, and physical ashift to the vdev_stat_t structure.
Add VDEV_STAT_VALID() macro which can be used to verify the presence of required vdev_stat_t fields in nvlist data.
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c: Provide a SYSCTL_PROC handler for "max_auto_ashift". Since the limit is only referenced long after boot when a create operation occurs, there's no compelling need for it to be a boot time configurable tunable. This also allows the validation code for the max_auto_ashift value to be contained within the sysctl handler.
Populate the new fields in the vdev_stat_t structure.
Fail vdev opens if the vdev reports an ashift larger than SPA_MAXASHIFT.
Propogate vdev_logical_ashift and vdev_physical_ashift between child and parent vdevs as is done for vdev_ashift.
In vdev_open(), restore code that fails opens for devices where vdev_ashift grows. This can only happen now if the device's logical ashift grows, which means it really isn't safe to use the device.
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/vdev_impl.h: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_file.c: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_missing.c: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_raidz.c: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_root.c: Update the vdev_open() API so that both logical (what was just ashift before) and physical ashift are reported.
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/vdev_impl.h: Add two new fields, vdev_physical_ashift and vdev_logical_ashift, to vdev_t.
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa_config.c: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c: Add vdev_ashift_optimize(). Call it anytime a new top-level vdev is allocated.
cddl/contrib/opensolaris/cmd/zpool/zpool_main.c: Add text for the VDEV_AUX_ASHIFT_TOO_BIG error.
For each sub-optimally configured leaf vdev, report configured and native block sizes.
cddl/contrib/opensolaris/cmd/zpool/zpool_main.c: cddl/contrib/opensolaris/lib/libzfs/common/libzfs.h: cddl/contrib/opensolaris/lib/libzfs/common/libzfs_status.c: Introduce a new zpool status: ZPOOL_STATUS_NON_NATIVE_ASHIFT. This status is reported on healthy pools containing vdevs configured to use a block size smaller than their reported physical block size.
cddl/contrib/opensolaris/lib/libzfs/common/libzfs_status.c: Update find_vdev_problem() and supporting functions to provide the full vdev_stat_t structure to problem checking routines, and to allow decent into replacing vdevs.
Add a vdev_non_native_ashift() validator which is used on the full vdev tree to check for ZPOOL_STATUS_NON_NATIVE_ASHIFT.
cddl/contrib/opensolaris/lib/libzpool/common/kernel.c: cddl/contrib/opensolaris/lib/libzpool/common/sys/zfs_context.h: Enhance sysctl userland stubs now that a SYSCTL_PROC handler is used in vdev.c.
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/metaslab_impl.h: When the group membership of a metaslab class changes (i.e. when a vdev is added or removed from a pool), walk the group list to determine the smallest block size currently available and record this in the metaslab class.
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/metaslab.h: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c: Add the metaslab_class_get_minblocksize() accessor.
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio_compress.h: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio_compress.c: sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c: In zio_compress_data(), take the minimum blocksize as an input parameter instead of assuming SPA_MINBLOCKSIZE.
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c: In l2arc_compress_buf(), pass SPA_MINBLOCKSIZE as the minimum blocksize of the device. The l2arc code performs has it's own code for deciding if compression is worth while, so this effectively disables zio_compress_data() from second guessing the original decision.
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c: In zio_write_bp_init(), use the minimum blocksize of the normal metaslab class when compressing data.
|
#
251631 |
|
11-Jun-2013 |
delphij |
MFV r251620:
ZFS comments need cleaner, more consistent style
Illumos ZFS issues: 3741 zfs comments need cleaner, more consistent style
MFC after: 2 weeks
|
#
251629 |
|
11-Jun-2013 |
delphij |
MFV r251619:
ZFS needs better comments.
Illumos ZFS issues: 3741 zfs needs better comments
MFC after: 2 weeks
|
#
251520 |
|
08-Jun-2013 |
delphij |
MFV r251519:
* Illumos ZFS issue #3805 arc shouldn't cache freed blocks
Quote from the Illumos issue:
ZFS should proactively evict freed blocks from the cache.
Even though these freed blocks will never be used again, and thus will eventually be evicted, this causes us to use memory inefficiently for 2 reasons:
1. A block that is freed has no chance of being accessed again, but will be kept in memory preferentially to a block that was accessed before it (and is thus older) but has not been freed and thus has at least some chance of being accessed again.
2. We partition the ARC into several buckets: user data that has been accessed only once (MRU) metadata that has been accessed only once (MRU) user data that has been accessed more than once (MFU) metadata that has been accessed more than once (MFU)
The user data vs metadata split is somewhat arbitrary, and the primary control on how much memory is used to cache data vs metadata is to simply try to keep the proportion the same as it has been in the past (each bucket "evicts against" itself). The secondary control is to evict data before evicting metadata.
Because of this bucketing, we may end up with one bucket mostly containing freed blocks that are very old, while another bucket has more recently accessed, still-allocated blocks. Data in the useful bucket (with still-allocated blocks) may be evicted in preference to data in the useless bucket (with old, freed blocks).
On dcenter, we saw that the MFU metadata bucket was 230MB, while the MFU data bucket was 27GB and the MRU metadata bucket was 256GB. However, the vast majority of data in the MRU metadata bucket (256GB) was freed blocks, and thus useless. Meanwhile, the MFU metadata bucket (230MB) was constantly evicting useful blocks that will be soon needed.
The problem of cache segmentation is a larger problem that needs more investigation. However, if we stop caching freed blocks, it should reduce the impact of this more fundamental issue.
MFC after: 2 weeks
|
#
251478 |
|
06-Jun-2013 |
delphij |
MFV r251474:
* Illumos zfs issue #3137 L2ARC compression
Whether or not to compress buffers entering the L2ARC is controlled by "compression" setting on the dataset, when compression is not "off", L2ARC compression is enabled.
The compress method is always LZ4 for L2ARC when enabled because it works best for the scenario.
MFC after: 2 weeks
|
#
249195 |
|
06-Apr-2013 |
mm |
MFV r248217: Merge change from vendor to reduce diff only. ZFS dtrace probes are not supported on FreeBSD yet.
Illumos ZFS issues: 3598 want to dtrace when errors are generated in zfs
MFC after: 3 weeks
|
#
248574 |
|
21-Mar-2013 |
smh |
Improve TXG handling in the TRIM module. This patch adds some improvements to the way the trim module considers TXGs:
- Free ZIOs are registered with the TXG from the ZIO itself, not the current SPA syncing TXG (which may be out of date); - L2ARC are registered with a zero TXG number, as L2ARC has no concept of TXGs; - The TXG limit for issuing TRIMs is now computed from the last synced TXG, not the currently syncing TXG. Indeed, under extremely unlikely race conditions, there is a risk we could trim blocks which have been freed in a TXG that has not finished syncing, resulting in potential data corruption in case of a crash.
Reviewed by: pjd (mentor) Approved by: pjd (mentor) Obtained from: https://github.com/dechamps/zfs/commit/5b46ad40d9081d75505d6f3bf04ac652445df366 MFC after: 2 weeks
|
#
248572 |
|
21-Mar-2013 |
smh |
Add TRIM support for L2ARC.
This adds TRIM support to cache vdevs. When ARC buffers are removed from the L2ARC in arc_hdr_destroy(), arc_release() or l2arc_evict(), the size previously occupied by the buffer gets scheduled for TRIMming. As always, actual TRIMs are only issued to the L2ARC after txg_trim_limit.
Reviewed by: pjd (mentor) Approved by: pjd (mentor) Obtained from: https://github.com/dechamps/zfs/commit/31aae373994fd112256607edba7de2359da3e9dc MFC after: 2 weeks
|
#
248571 |
|
21-Mar-2013 |
mm |
Merge libzfs_core branch: includes MFV 238590, 238592, 247580
MFV 238590, 238592: In the first zfs ioctl restructuring phase, the libzfs_core library was introduced. It is a new thin library that wraps around kernel ioctl's. The idea is to provide a forward-compatible way of dealing with new features. Arguments are passed in nvlists and not random zfs_cmd fields, new-style ioctls are logged to pool history using a new method of history logging.
http://blog.delphix.com/matt/2012/01/17/the-future-of-libzfs/
MFV 247580 [1]: To address issues of several deadlocks and race conditions the locking code around dsl_dataset was rewritten and the interface to synctasks was changed.
User-Visible Changes: "zfs snapshot" can create more arbitrary snapshots at once (atomically) "zfs destroy" destroys multiple snapshots at once "zfs recv" has improved performance
Backward Compatibility: I have extended the compatibility layer to support full backward compatibility by remapping or rewriting the responsible ioctl arguments. Old utilities are fully supported by the new kernel module.
Forward Compatibility: New utilities work with old kernels with the following restrictions: - creating, destroying, holding and releasing of multiple snapshots at once is not supported, this includes recursive (-r) commands
Illumos ZFS issues: 2882 implement libzfs_core 2900 "zfs snapshot" should be able to create multiple, arbitrary snapshots at once 3464 zfs synctask code needs restructuring
References: https://www.illumos.org/issues/2882 https://www.illumos.org/issues/2900 https://www.illumos.org/issues/3464 [1]
MFC after: 1 month Sponsored by: Hybrid Logic Inc. [1]
|
#
247187 |
|
23-Feb-2013 |
mm |
MFV r246653: Import vendor change to avoid "unitialized variable" warnings.
Illumos ZFS issues: 3522 zfs module should not allow uninitialized variables
References: https://www.illumos.org/issues/3522
|
#
246666 |
|
11-Feb-2013 |
mm |
MFV r246392: Import vendor ZFS bugfix fixing a possible deadlock in arc_read().
Illumos ZFS issues: 3498 panic in arc_read(): !refcount_is_zero(&pbuf->b_hdr->b_refcnt)
References: https://www.illumos.org/issues/3498
MFC after: 2 weeks
|
#
243524 |
|
25-Nov-2012 |
mm |
MFV r243013 and r243267:
Import the zio nop-write improvement from Illumos. To reduce I/O, nop-write omits overwriting data if the checksum (cryptographically secure) of new data matches the checksum of existing data. It also saves space if snapshots are in use.
It currently works only on datasets with enabled compression, disabled deduplication and sha256 checksums.
IllumOS 13887:196932ec9e6a and 13888:7204b3392a58 3236 zio nop-write
References: https://www.illumos.org/issues/3236
MFC after: 2 weeks
|
#
242845 |
|
10-Nov-2012 |
delphij |
MFV r242729 (mm):
Illumos r13840:97fd5cdf328a:
3145 single-copy arc 3212 ztest: race condition between vdev_online() and spa_vdev_remove()
Illumos r13849:3468a95b27cd:
3258 ztest's use of file descriptors is unstable
|
#
241773 |
|
20-Oct-2012 |
avg |
zfs: wait in arc_lowmem only if curproc == pageproc
... otherwise the current thread might be holding ARC locks and thus run into a deadlock. This happens, for example, when a thread does memory allocation in the ARC code and runs into KVA shortage. Also, it really makes the most sense to wait in pageproc, so that the results of ARC reclamation are seen before the page cache is acted upon. In other cases where vm_lowmem is invoked, e.g. on KVA space shortage, the callers perform multiple attempts (up to 8) and wait for rather long intervals between them (up to 4 seconds), so ARC reclaim results should become visible even without explicit waiting on the ARC thread.
Note that this is not a critical issue for typical ZFS usages where KVA space should already be large enough. On amd64 systems setting KVA size to twice the physical memory size is known to mitigate KVA fragmentation issues in practice.
Side note: perhaps vm_lowmem 'how' parameter should be used to differentiate between causes of the event.
Reported by: Nikolay Denev <ndenev@gmail.com> MFC after: 19 days
|
#
240415 |
|
12-Sep-2012 |
mm |
Merge recent zfs vendor changes, sync code and adjust userland DEBUG.
Illumos issued covered: 1884 Empty "used" field for zfs *space commands 3006 VERIFY[S,U,P] and ASSERT[S,U,P] frequently check if first argument is zero 3028 zfs {group,user}space -n prints (null) instead of numeric GID/UID 3048 zfs {user,group}space [-s|-S] is broken 3049 zfs {user,group}space -t doesn't really filter the results 3060 zfs {user,group}space -H output isn't tab-delimited 3061 zfs {user,group}space -o doesn't use specified fields order 3064 usr/src/cmd/zpool/zpool_main.c misspells "successful" 3093 zfs {user,group}space's -i is noop 3098 zfs userspace/groupspace fail without saying why when run as non-root
References: https://www.illumos.org/issues/ + [issue_id]
Obtained from: illumos (vendor/illumos, vendor/illumos-sys) MFC after: 2 weeks
|
#
240133 |
|
05-Sep-2012 |
mm |
Merge recent vendor changes and sync code: 1862 incremental zfs receive fails for sparse file > 8PB 3112 ztest does not honor ZFS_DEBUG 3122 zfs destroy filesystem should prefetch blocks 3129 'zpool reopen' restarts resilvers 3130 ztest failure: Assertion failed: 0 == dmu_objset_destroy(name, B_FALSE) (0x0 == 0x10)
References: https://www.illumos.org/issues/1862 https://www.illumos.org/issues/3112 https://www.illumos.org/issues/3122 https://www.illumos.org/issues/3129 https://www.illumos.org/issues/3130
Obtained from: illumos (vendor/illumos, vendor/illumos-sys) MFC after: 2 weeks
|
#
236884 |
|
11-Jun-2012 |
mm |
Introduce "feature flags" for ZFS pools (bump SPA version to 5000). Add first feature "com.delphix:async_destroy" (asynchronous destroy of ZFS datasets). Implement features support in ZFS boot code.
Illumos revisions merged: 13700:2889e2596bd6 13701:1949b688d5fb 2619 asynchronous destruction of ZFS file systems 2747 SPA versioning with zfs feature flags
References: https://www.illumos.org/issues/2619 https://www.illumos.org/issues/2747
Obtained from: illumos (issue #2619, #2747) MFC after: 1 month
|
#
229663 |
|
05-Jan-2012 |
pjd |
- Allow to change vfs.zfs.arc_meta_limit at runtime. - Change vfs.zfs.arc_meta_used from CTLFLAG_RDTUN to CTLFLAG_RD, as it is not a tunable.
MFC after: 3 days
|
#
228392 |
|
10-Dec-2011 |
pjd |
Move ru_inblock increment into arc_read_nolock() so we don't account for cached reads.
Discussed with: gibbs No objections from: avg Tested by: Marcus Reid <marcus@blazingdot.com> MFC after: 1 week
|
#
228103 |
|
28-Nov-2011 |
mm |
Merge new ZFS features from illumos:
1644 add ZFS "clones" property https://www.illumos.org/issues/1644
1645 add ZFS "written" and "written@..." properties https://www.illumos.org/issues/1645
1646 "zfs send" should estimate size of stream https://www.illumos.org/issues/1646
1647 "zfs destroy" should determine space reclaimed by destroying multiple snapshots https://www.illumos.org/issues/1647
1693 persistent 'comment' field for a zpool https://www.illumos.org/issues/1693
1708 adjust size of zpool history data https://www.illumos.org/issues/1708
1748 desire support for reguid in zfs https://www.illumos.org/issues/1748
Obtained from: illumos (changesets 13514, 13524, 13525) MFC after: 1 month
|
#
219089 |
|
27-Feb-2011 |
pjd |
Finally... Import the latest open-source ZFS version - (SPA) 28.
Few new things available from now on:
- Data deduplication. - Triple parity RAIDZ (RAIDZ3). - zfs diff. - zpool split. - Snapshot holds. - zpool import -F. Allows to rewind corrupted pool to earlier transaction group. - Possibility to import pool in read-only mode.
MFC after: 1 month
|
#
217367 |
|
13-Jan-2011 |
mdf |
Re-commit the zfs sysctl(9) type-safety changes.
Thanks to dim and pjd for the pointer to zfs_context.h for building userland.
|
#
217332 |
|
12-Jan-2011 |
mdf |
Revert cddl changes for sysctl(9) until I understand why this isn't building on universe.
|
#
217319 |
|
12-Jan-2011 |
mdf |
sysctl(9) cleanup checkpoint: amd64 GENERIC builds cleanly.
Commit the zfs piece.
|
#
212783 |
|
17-Sep-2010 |
avg |
zfs arc_reclaim_needed: fix typo in mismerge in r212780
PR: kern/146410, kern/138790 MFC after: 3 weeks X-MFC with: r212780
|
#
212781 |
|
17-Sep-2010 |
avg |
zfs arc_reclaim_needed: remove redundant checks for arc_c_max and arc_c_max
Those checks are not present in upstream code and they are enforced in actual calculations of delta by which ARC size can be grown or should be reduced.
MFC after: 3 weeks
|
#
212780 |
|
17-Sep-2010 |
avg |
zfs arc_reclaim_needed: more reasonable threshold for available pages
vm_paging_target() is not a trigger of any kind for pageademon, but rather a "soft" target for it when it's already triggered. Thus, trying to keep 2048 pages above that level at the expense of ARC was simply driving ARC size into the ground even with normal memory loads. Instead, use a threshold at which a pagedaemon scan is triggered, so that ARC reclaiming helps with pagedaemon's task, but the latter still recycles active and inactive pages.
PR: kern/146410, kern/138790 MFC after: 3 weeks
|
#
211762 |
|
24-Aug-2010 |
avg |
zfs arc_reclaim_thread: no need to call arc_reclaim_needed when resetting needfree
needfree is checked at the very start of arc_reclaim_needed. This change makes code easier to follow and maintain in face of potential changed in arc_reclaim_needed.
Also, put the whole sub-block under _KERNEL because needfree can be set only in kernel code.
To do: rename needfree to something else to aovid confusion with OpenSolaris global variable of the same name which is used in the same code, but has different meaning (page deficit).
Note: I have an impression that locking around accesses to this variable as well as mutual notifications between arc_reclaim_thread and arc_lowmem are not proper.
MFC after: 1 week
|
#
210427 |
|
23-Jul-2010 |
avg |
zfs arc_memory_throttle: available memory is free + cache
OpenSolaris freemem has the same meaning as our v_free_count + v_cache_count.
Obtained from: Artem Belevich <fbsdlist@src.cx>, Peter Jeremy <peterjeremy@acm.org> Discussed with: pjd MFC after: 2 weeks
|
#
209962 |
|
13-Jul-2010 |
mm |
Merge ZFS version 15 and almost all OpenSolaris bugfixes referenced in Solaris 10 updates 141445-09 and 142901-14.
Detailed information: (OpenSolaris revisions and Bug IDs, Solaris 10 patch numbers)
7844:effed23820ae 6755435 zfs_open() and zfs_close() needs to use ZFS_ENTER/ZFS_VERIFY_ZP (141445-01)
7897:e520d8258820 6748436 inconsistent zpool.cache in boot_archive could panic a zfs root filesystem upon boot-up (141445-01)
7965:b795da521357 6740164 zpool attach can create an illegal root pool (141909-02)
8084:b811cc60d650 6769612 zpool_import() will continue to write to cachefile even if altroot is set (N/A)
8121:7fd09d4ebd9c 6757430 want an option for zdb to disable space map loading and leak tracking (141445-01)
8129:e4f45a0bfbb0 6542860 ASSERT: reason != VDEV_LABEL_REMOVE||vdev_inuse(vd, crtxg, reason, 0) (141445-01)
8188:fd00c0a81e80 6761100 want zdb option to select older uberblocks (141445-01)
8190:6eeea43ced42 6774886 zfs_setattr() won't allow ndmp to restore SUNWattr_rw (141445-01)
8225:59a9961c2aeb 6737463 panic while trying to write out config file if root pool import fails (141445-01)
8227:f7d7be9b1f56 6765294 Refactor replay (141445-01)
8228:51e9ca9ee3a5 6572357 libzfs should do more to avoid mnttab lookups (141909-01) 6572376 zfs_iter_filesystems and zfs_iter_snapshots get objset stats twice (141909-01)
8241:5a60f16123ba 6328632 zpool offline is a bit too conservative (141445-01) 6739487 ASSERT: txg <= spa_final_txg due to scrub/export race (141445-01) 6767129 ASSERT: cvd->vdev_isspare, in spa_vdev_detach() (141445-01) 6747698 checksum failures after offline -t / export / import / scrub (141445-01) 6745863 ZFS writes to disk after it has been offlined (141445-01) 6722540 50% slowdown on scrub/resilver with certain vdev configurations (141445-01) 6759999 resilver logic rewrites ditto blocks on both source and destination (141445-01) 6758107 I/O should never suspend during spa_load() (141445-01) 6776548 codereview(1) runs off the page when faced with multi-line comments (N/A) 6761406 AMD errata 91 workaround doesn't work on 64-bit systems (141445-01)
8242:e46e4b2f0a03 6770866 GRUB/ZFS should require physical path or devid, but not both (141445-01)
8269:03a7e9050cfd 6674216 "zfs share" doesn't work, but "zfs set sharenfs=on" does (141445-01) 6621164 $SRC/cmd/zfs/zfs_main.c seems to have a syntax error in the translation note (141445-01) 6635482 i18n problems in libzfs_dataset.c and zfs_main.c (141445-01) 6595194 "zfs get" VALUE column is as wide as NAME (141445-01) 6722991 vdev_disk.c: error checking for ddi_pathname_to_dev_t() must test for NODEV (141445-01) 6396518 ASSERT strings shouldn't be pre-processed (141445-01)
8274:846b39508aff 6713916 scrub/resilver needlessly decompress data (141445-01)
8343:655db2375fed 6739553 libzfs_status msgid table is out of sync (141445-01) 6784104 libzfs unfairly rejects numerical values greater than 2^63 (141445-01) 6784108 zfs_realloc() should not free original memory on failure (141445-01)
8525:e0e0e525d0f8 6788830 set large value to reservation cause core dump (141445-01) 6791064 want sysevents for ZFS scrub (141445-01) 6791066 need to be able to set cachefile on faulted pools (141445-01) 6791071 zpool_do_import() should not enable datasets on faulted pools (141445-01) 6792134 getting multiple properties on a faulted pool leads to confusion (141445-01)
8547:bcc7b46e5ff7 6792884 Vista clients cannot access .zfs (141445-01)
8632:36ef517870a3 6798384 It can take a village to raise a zio (141445-01)
8636:7e4ce9158df3 6551866 deadlock between zfs_write(), zfs_freesp(), and zfs_putapage() (141909-01) 6504953 zfs_getpage() misunderstands VOP_GETPAGE() interface (141909-01) 6702206 ZFS read/writer lock contention throttles sendfile() benchmark (141445-01) 6780491 Zone on a ZFS filesystem has poor fork/exec performance (141445-01) 6747596 assertion failed: DVA_EQUAL(BP_IDENTITY(&zio->io_bp_orig), BP_IDENTITY(zio->io_bp))); (141445-01)
8692:692d4668b40d 6801507 ZFS read aggregation should not mind the gap (141445-01)
8697:e62d2612c14d 6633095 creating a filesystem with many properties set is slow (141445-01)
8768:dfecfdbb27ed 6775697 oracle crashes when overwriting after hitting quota on zfs (141909-01)
8811:f8deccf701cf 6790687 libzfs mnttab caching ignores external changes (141445-01) 6791101 memory leak from libzfs_mnttab_init (141445-01)
8845:91af0d9c0790 6800942 smb_session_create() incorrectly stores IP addresses (N/A) 6582163 Access Control List (ACL) for shares (141445-01) 6804954 smb_search - shortname field should be space padded following the NULL terminator (N/A) 6800184 Panic at smb_oplock_conflict+0x35() (N/A)
8876:59d2e67b4b65 6803822 Reboot after replacement of system disk in a ZFS mirror drops to grub> prompt (141445-01)
8924:5af812f84759 6789318 coredump when issue zdb -uuuu poolname/ (141445-01) 6790345 zdb -dddd -e poolname coredump (141445-01) 6797109 zdb: 'zdb -dddddd pool_name/fs_name inode' coredump if the file with inode was deleted (141445-01) 6797118 zdb: 'zdb -dddddd poolname inum' coredump if I miss the fs name (141445-01) 6803343 shareiscsi=on failed, iscsitgtd failed request to share (141445-01)
9030:243fd360d81f 6815893 hang mounting a dataset after booting into a new boot environment (141445-01)
9056:826e1858a846 6809691 'zpool create -f' no longer overwrites ufs infomation (141445-01)
9179:d8fbd96b79b3 6790064 zfs needs to determine uid and gid earlier in create process (141445-01)
9214:8d350e5d04aa 6604992 forced unmount + being in .zfs/snapshot/<snap1> = not happy (141909-01) 6810367 assertion failed: dvp->v_flag & VROOT, file: ../../common/fs/gfs.c, line: 426 (141909-01)
9229:e3f8b41e5db4 6807765 ztest_dsl_dataset_promote_busy needs to clean up after ENOSPC (141445-01)
9230:e4561e3eb1ef 6821169 offlining a device results in checksum errors (141445-01) 6821170 ZFS should not increment error stats for unavailable devices (141445-01) 6824006 need to increase issue and interrupt taskqs threads in zfs (141445-01)
9234:bffdc4fc05c4 6792139 recovering from a suspended pool needs some work (141445-01) 6794830 reboot command hangs on a failed zfs pool (141445-01)
9246:67c03c93c071 6824062 System panicked in zfs_mount due to NULL pointer dereference when running btts and svvs tests (141909-01)
9276:a8a7fc849933 6816124 System crash running zpool destroy on broken zpool (141445-03)
9355:09928982c591 6818183 zfs snapshot -r is slow due to set_snap_props() doing txg_wait_synced() for each new snapshot (141445-03)
9391:413d0661ef33 6710376 log device can show incorrect status when other parts of pool are degraded (141445-03)
9396:f41cf682d0d3 (part already merged) 6501037 want user/group quotas on ZFS (141445-03) 6827260 assertion failed in arc_read(): hdr == pbuf->b_hdr (141445-03) 6815592 panic: No such hold X on refcount Y from zfs_znode_move (141445-03) 6759986 zfs list shows temporary %clone when doing online zfs recv (141445-03)
9404:319573cd93f8 6774713 zfs ignores canmount=noauto when sharenfs property != off (141445-03)
9412:4aefd8704ce0 6717022 ZFS DMU needs zero-copy support (141445-03)
9425:e7ffacaec3a8 6799895 spa_add_spares() needs to be protected by config lock (141445-03) 6826466 want to post sysevents on hot spare activation (141445-03) 6826468 spa 'allowfaulted' needs some work (141445-03) 6826469 kernel support for storing vdev FRU information (141445-03) 6826470 skip posting checksum errors from DTL regions of leaf vdevs (141445-03) 6826471 I/O errors after device remove probe can confuse FMA (141445-03) 6826472 spares should enjoy some of the benefits of cache devices (141445-03)
9443:2a96d8478e95 6833711 gang leaders shouldn't have to be logical (141445-03)
9463:d0bd231c7518 6764124 want zdb to be able to checksum metadata blocks only (141445-03)
9465:8372081b8019 6830237 zfs panic in zfs_groupmember() (141445-03)
9466:1fdfd1fed9c4 6833162 phantom log device in zpool status (141445-03)
9469:4f68f041ddcd 6824968 add ZFS userquota support to rquotad (141445-03)
9470:6d827468d7b5 6834217 godfather I/O should reexecute (141445-03)
9480:fcff33da767f 6596237 Stop looking and start ganging (141909-02)
9493:9933d599bc93 6623978 lwb->lwb_buf != NULL, file ../../../uts/common/fs/zfs/zil.c, line 787, function zil_lwb_commit (141445-06)
9512:64cafcbcc337 6801810 Commit of aligned streaming rewrites to ZIL device causes unwanted disk reads (N/A)
9515:d3b739d9d043 6586537 async zio taskqs can block out userland commands (142901-09)
9554:787363635b6a 6836768 zfs_userspace() callback has no way to indicate failure (N/A)
9574:1eb6a6ab2c57 6838062 zfs panics when an error is encountered in space_map_load() (141909-02)
9583:b0696cd037cc 6794136 Panic BAD TRAP: type=e when importing degraded zraid pool. (141909-03)
9630:e25a03f552e0 6776104 "zfs import" deadlock between spa_unload() and spa_async_thread() (141445-06)
9653:a70048a304d1 6664765 Unable to remove files when using fat-zap and quota exceeded on ZFS filesystem (141445-06)
9688:127be1845343 6841321 zfs userspace / zfs get userused@ doesn't work on mounted snapshot (N/A) 6843069 zfs get userused@S-1-... doesn't work (N/A)
9873:8ddc892eca6e 6847229 assertion failed: refcount_count(&tx->tx_space_written) + delta <= tx->tx_space_towrite in dmu_tx.c (141445-06)
9904:d260bd3fd47c 6838344 kernel heap corruption detected on zil while stress testing (141445-06)
9951:a4895b3dd543 6844900 zfs_ioc_userspace_upgrade leaks (N/A)
10040:38b25aeeaf7a 6857012 zfs panics on zpool import (141445-06)
10000:241a51d8720c 6848242 zdb -e no longer works as expected (N/A)
10100:4a6965f6bef8 6856634 snv_117 not booting: zfs_parse_bootfs: error2 (141445-07)
10160:a45b03783d44 6861983 zfs should use new name <-> SID interfaces (N/A) 6862984 userquota commands can hang (141445-06)
10299:80845694147f 6696858 zfs receive of incremental replication stream can dereference NULL pointer and crash (N/A)
10302:a9e3d1987706 6696858 zfs receive of incremental replication stream can dereference NULL pointer and crash (fix lint) (N/A)
10575:2a8816c5173b (partial merge) 6882227 spa_async_remove() shouldn't do a full clear (142901-14)
10800:469478b180d9 6880764 fsync on zfs is broken if writes are greater than 32kb on a hard crash and no log attached (142901-09) 6793430 zdb -ivvvv assertion failure: bp->blk_cksum.zc_word[2] == dmu_objset_id(zilog->zl_os) (N/A)
10801:e0bf032e8673 (partial merge) 6822816 assertion failed: zap_remove_int(ds_next_clones_obj) returns ENOENT (142901-09)
10810:b6b161a6ae4a 6892298 buf->b_hdr->b_state != arc_anon, file: ../../common/fs/zfs/arc.c, line: 2849 (142901-09)
10890:499786962772 6807339 spurious checksum errors when replacing a vdev (142901-13)
11249:6c30f7dfc97b 6906110 bad trap panic in zil_replay_log_record (142901-13) 6906946 zfs replay isn't handling uid/gid correctly (142901-13)
11454:6e69bacc1a5a 6898245 suspended zpool should not cause rest of the zfs/zpool commands to hang (142901-10)
11546:42ea6be8961b (partial merge) 6833999 3-way deadlock in dsl_dataset_hold_ref() and dsl_sync_task_group_sync() (142901-09)
Discussed with: pjd Approved by: delphij (mentor) Obtained from: OpenSolaris (multiple Bug IDs) MFC after: 2 months
|
#
209275 |
|
17-Jun-2010 |
mm |
Import latest ARC change from OpenSolaris: - large ghost eviction causes high write latency - arc_adjust might adjust MRU unnecessarily - arc_adapt can lead to wild arc_p adjustment
OpenSolaris onnv-revision: 12636:13b5d698941e
Submitted by: avg Approved by: pjd, delphij (mentor) Obtained from: OpenSolaris (Bug ID 6950219, 6953403, 6951024) MFC after: 1 month
|
#
209101 |
|
12-Jun-2010 |
mm |
Fix arc_read_done may try to byteswap undefined data (sparc related)
OpenSolaris onnv-revision: 10839:cf83b553a2ab
Obtained from: OpenSolaris (Bug ID 6836714) Approved by: pjd, delphij (mentor) MFC after: 3 days
|
#
208474 |
|
23-May-2010 |
mm |
Remove kstat.zfs.arcstats.l2_write_bytes_written
The arcstats.l2_write_bytes_written kstat counter introduced in r205231 was duplicite with vendor's arcstats.l2_write_bytes counter imported in r208373 (OpenSolaris revision 8582:df9361868dbe)
Approved by: pjd, delphij (mentor) MFC after: 3 days
|
#
208454 |
|
23-May-2010 |
pjd |
Remove ZIO_USE_UMA from arc.c as well.
MFC after: 3 days
|
#
208373 |
|
21-May-2010 |
mm |
Update L2ARC code and fix several bugs.
- improve ARC memory consumption (Bug ID 6488341) - ARC/L2ARC metadata accounting (Bug ID 6748019) - L2ARC turbo warmup (Bud ID 6748023) - kstats for ARC content (Bug ID 6748023) - kstats for evicted bytes from ARC by L2ARC state (Bud ID 6871680) - fix panic on i386 systems (Bug ID 6821260)
OpenSolaris onnv revisions: 8582:df9361868dbe, 8628:97dcded6e556, 9215:7c4584f76b47, 9274:a10f8bd993c1, 10357:29060492b29d
OpenSolaris Bug IDs: 6748019, 6748023, 6748030, 6488341, 6798268, 6821260, 6790261, 6871680
Approved by: pjd, delphij (mentor) Obtained from: OpenSlaris (multiple bug IDs) MFC after: 3 days
|
#
206796 |
|
18-Apr-2010 |
pjd |
Style fixes.
|
#
206795 |
|
18-Apr-2010 |
pjd |
Add missing list and lock destruction.
|
#
206794 |
|
18-Apr-2010 |
pjd |
Extend locks scope to match OpenSolaris.
|
#
206793 |
|
18-Apr-2010 |
pjd |
Remove racy assertion.
Obtained from: OpenSolaris
|
#
206792 |
|
18-Apr-2010 |
pjd |
Set ARC_L2_WRITING on L2ARC header creation.
Obtained from: OpenSolaris
|
#
205264 |
|
17-Mar-2010 |
kmacy |
- cache line align arcs_lock array (h/t Marius Nuennerich) - fix ARCS_LOCK_PAD to use architecture defined CACHE_LINE_SIZE - cache line align buf_hash_table ht_locks array
MFC after: 7 days
|
#
205253 |
|
17-Mar-2010 |
kmacy |
use CACHE_LINE_SIZE instead of hardcoding 128 for lock pad
pointed out by Marius Nuennerich and jhb@
|
#
205231 |
|
16-Mar-2010 |
kmacy |
- reduce contention by breaking up ARC state locks in to 16 for data and 16 for metadata - export L2ARC tunables as sysctls - add several kstats to track L2ARC state more precisely - avoid holding a contended lock when atomically incrementing a contended counter (no lock protection needed for atomics)
|
#
205133 |
|
13-Mar-2010 |
kmacy |
fix compilation under ZIO_USE_UMA
|
#
204804 |
|
06-Mar-2010 |
pjd |
Remove racy assertion.
Reported by: Attila Nagy <bra@fsn.hu> Obtained from: OpenSolaris, Bug ID 6827260 MFC after: 1 week
|
#
197816 |
|
06-Oct-2009 |
kmacy |
Prevent paging pressure from draining arc too much - always drain arc if above arc_c_max - never drain arc if arc is below arc_c_max
MFC after: 3 days
|
#
196941 |
|
07-Sep-2009 |
trasz |
Prevent the line from wrapping.
|
#
196863 |
|
05-Sep-2009 |
trasz |
Improve wording.
Discussed with: pjd, cperciva, rink, wkoszek and des, in order of appearance.
|
#
194043 |
|
11-Jun-2009 |
kmacy |
pjd has requested that I keep the tunable as zfs_prefetch_disable to minimize gratuitous differences with Opensolaris' ZFS
Sorry for the churn
|
#
193953 |
|
10-Jun-2009 |
kmacy |
use default policy for enabling prefetching unless the TUNABLE is set
|
#
193878 |
|
10-Jun-2009 |
kmacy |
As far as I can tell systems that have less than 4GB are more often hurt by prefetched than helped. On i386 systems and systems with less than 4GB, prefetch is now disabled by default. I've added a prefetch enable tunable, to enable prefetching for those systems. The prefetch disable tunable will continue to unconditionally disable prefetching.
|
#
192360 |
|
19-May-2009 |
kmacy |
- back out direct map hack - it is no longer needed
|
#
192209 |
|
16-May-2009 |
kmacy |
only use direct map if system has more than 2GB
|
#
192207 |
|
16-May-2009 |
kmacy |
apply band-aid to x86_64 systems with more physical memory than kmem by allocating from the direct map
|
#
191903 |
|
07-May-2009 |
kmacy |
avoid LOR and gratuitous extra lock acquisitions by moving user_evict list buffers to a temporary list
|
#
191902 |
|
07-May-2009 |
kmacy |
Allow the VM to provide backpressure on the ARC cache as it does on Solaris.
|
#
185029 |
|
17-Nov-2008 |
pjd |
Update ZFS from version 6 to 13 and bring some FreeBSD-specific changes.
This bring huge amount of changes, I'll enumerate only user-visible changes:
- Delegated Administration
Allows regular users to perform ZFS operations, like file system creation, snapshot creation, etc.
- L2ARC
Level 2 cache for ZFS - allows to use additional disks for cache. Huge performance improvements mostly for random read of mostly static content.
- slog
Allow to use additional disks for ZFS Intent Log to speed up operations like fsync(2).
- vfs.zfs.super_owner
Allows regular users to perform privileged operations on files stored on ZFS file systems owned by him. Very careful with this one.
- chflags(2)
Not all the flags are supported. This still needs work.
- ZFSBoot
Support to boot off of ZFS pool. Not finished, AFAIK.
Submitted by: dfr
- Snapshot properties
- New failure modes
Before if write requested failed, system paniced. Now one can select from one of three failure modes: - panic - panic on write error - wait - wait for disk to reappear - continue - serve read requests if possible, block write requests
- Refquota, refreservation properties
Just quota and reservation properties, but don't count space consumed by children file systems, clones and snapshots.
- Sparse volumes
ZVOLs that don't reserve space in the pool.
- External attributes
Compatible with extattr(2).
- NFSv4-ACLs
Not sure about the status, might not be complete yet.
Submitted by: trasz
- Creation-time properties
- Regression tests for zpool(8) command.
Obtained from: OpenSolaris
|
#
175633 |
|
24-Jan-2008 |
pjd |
- Reduce how much ZFS caches by default. This is another change to mitigate 'kmem_map too small panics'. - Print two warnings if there is not enough memory and not enough address space. - Improve comment.
|
#
174049 |
|
28-Nov-2007 |
jb |
* Check endianness the FreeBSD way.
* Use LBOLT rather than lbolt to avoid a clash with a FreeBSD global variable.
|
#
173419 |
|
07-Nov-2007 |
pjd |
Warn if kmem_map size is set to less than 512MB. Previous warning was a bit pointless, because default is set to something around 300MB and also insufficient.
MFC after: 3 days
|
#
172130 |
|
10-Sep-2007 |
pjd |
After dfr@ vnode leak fix, we can allow ARC to consume more memory.
Tested by: kris Approved by: re (bmah)
|
#
172030 |
|
01-Sep-2007 |
pjd |
Use CTLFLAG_RDTUN for tunable sysctls.
Approved by: re (bmah)
|
#
168987 |
|
23-Apr-2007 |
bmah |
Mostly-cosmetic fixes in low-memory warning messages:
o Fix linewrap issues.
o Fix two typos (s/Recomended/Recommended/ and s/tunning/tuning/)
o Remove a couple of extra instances of the word "of".
o Update names of kmem_size variables.
Approved by: pjd
|
#
168696 |
|
13-Apr-2007 |
pjd |
Fix overflow, which was causing endless loops when 32bit machine had more than 2GB of RAM. This was because our physmem is long and 'physmem*PAGESIZE' can be negative for more than 2GB of memory.
Reported by: Andrey V. Elsukov <bu7cher@yandex.ru>
It is not yet tested by Andrey, so there can be other problems, but this was definiately a bug, so I'm committing a fix now.
|
#
168582 |
|
10-Apr-2007 |
pjd |
MFp4: Hide under '#ifdef _KERNEL' only what's really needed.
|
#
168566 |
|
10-Apr-2007 |
pjd |
Try to stabilize ZFS with regard to memory consumption: - Allow to shrink ARC down to 16MB (instead of 64MB). - Set arc_max to 1/2 of kmem_map by default. - Start freeing things earlier when low memory situation is detected. - Serialize execution of arc_lowmem().
I decided to setup minimum ZFS memory requirements to 512MB of RAM and 256MB of kmem_map size. If there is less RAM or kmem_map, a warning will be printed. World is cruel, be no better. In other words: modern file system requires modern hardware:)
From ZFS administration guide:
"Currently the minimum amount of memory recommended to install a Solaris system is 512 Mbytes. However, for good ZFS performance, at least one Gbyte or more of memory is recommended."
|
#
168559 |
|
09-Apr-2007 |
pjd |
Remove unused #define.
|
#
168481 |
|
07-Apr-2007 |
pjd |
Fix libzpool compilation.
Reported by: des
|
#
168473 |
|
07-Apr-2007 |
pjd |
Allow to tune maximum and minimum memory used by ARC.
|
#
168460 |
|
07-Apr-2007 |
pjd |
Add missing mutex_init() which was causing assertion panic when on clone destruction.
Reported by: kris
|
#
168404 |
|
06-Apr-2007 |
pjd |
Please welcome ZFS - The last word in file systems.
ZFS file system was ported from OpenSolaris operating system. The code in under CDDL license.
I'd like to thank all SUN developers that created this great piece of software.
Supported by: Wheel LTD (http://www.wheel.pl/) Supported by: The FreeBSD Foundation (http://www.freebsdfoundation.org/) Supported by: Sentex (http://www.sentex.net/)
|