Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFH (r250056,r250057): add -Z option to zero unused blocks.

Merge checked by: Glen Barber (gjb@)

MFC of 247212:
When running with the -d option, instrument fsck_ffs to track the number,
data type, and running time of its I/O operations.

No functional changes.

MFC of 247234:
Catch up with internal API changes for initbarea() and getdatablk()
of fsck_ffs introduced with r247212.

Submitted by: David Wolfskill <david@catwhisker.org>

MFC of 248625:
Speed up fsck by caching the cylinder group maps in pass1 so
that they do not need to be read again in pass5. As this nearly
doubles the memory requirement for fsck, the cache is thrown away
if other memory needs in fsck would otherwise fail. Thus, the
memory footprint of fsck remains unchanged in memory constrained
environments.

This work was inspired by a paper presented at Usenix's FAST '13:
www.usenix.org/conference/fast13/ffsck-fast-file-system-checker

Details of this implementation appears in the April 2013 of ;login:
www.usenix.org/publications/login/april-2013-volume-38-number-2.
A copy of the April 2013 ;login: paper can also be downloaded
from: www.mckusick.com/publications/faster_fsck.pdf.

Reviewed by: kib
Tested by: Peter Holm

MFC of 248639:
Fix the build after addition of cylinder group cacheing (r248625)

Reported by: Glen Barber (gjb@)
Pointy hat to: Kirk McKusick (mckusick@)

MFC of 248673:
Minor formatting fix for printf() to fix clang builds.

Submitted by: db
Reviewed by: gjb

MFC of 248680:
Resolve clang compile errors on amd64/i386 for certain by casting.

compile tested with clang on i386, amd64
compile tested with gcc on i386, amd64, sparc64

Submitted by: delphij

MFC of 248691:
Note that output is in seconds, not msec.
KNF indentation.
No functional change.
No change to printf strings.
No change to casting of printf arguments.

Reported by: Bruce Evans

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Break out the pass 5 inode and block map updating into a separate function
so that the function can be used by the journaling soft updates recovery.

Add an -E option to mirror newfs's. The idea is that if you have a system
that was built before ffs grew support for TRIM, your filesystem will have
plenty of free blocks that the flash chip doesn't know are free, so it
can't take advantage of them for wear leveling. Once you've upgraded your
kernel, you enable TRIM on the filesystem (tunefs -t enable), then run
fsck_ffs -E on it before mounting it.

I tested this patch by half-filling an mdconfig'ed filesystem image,
running fsck_ffs -E on it, then verifying that the contents were not
damaged by comparing them to a pristine copy using rsync's checksum
functionality. There is no reliable way to test it on real hardware.

Many thanks to mckusick@, who provided the tricky parts of this patch and
reviewed the final version.

Reviewed by: mckusick@
MFC after: 3 weeks

- Merge soft-updates journaling from projects/suj/head into head. This
brings in support for an optional intent log which eliminates the need
for background fsck on unclean shutdown.

Sponsored by: iXsystems, Yahoo!, and Juniper.
With help from: McKusick and Peter Holm

Background fsck applies twice some summary totals changes. The next
background fsck on the same file system might then print negative
numbers for reclaimed directories/files/fragments.

Address the issue in a limited degree, by using old summary data for
cg when bgfsck is performed.

Submitted by: tegge
MFC after: 1 week

check_maps() in /usr/src/sbin/fsck_ffs/pass5.c seems to be limited to file
systems less than 1 TB, due to using 32-bits integers for file system block
numbers. This also causes incorrect error reporting for foreground fsck.

Convert it to use ufs2_daddr_t for block numbers.

PR: kern/127951
Submitted by: tegge
MFC after: 1 week

Implements gjournal support. If file system has gjournal support enabled
and -p flag was given perform fast file system checking (bascially only
garbage collecting of orphaned objects).

Rename bread() to blread() and bwrite() to blwrite() as we now link to
the libufs library, which also implement functions with that names.

Sponsored by: home.pl

Make background fsck based summary adjustments actually work by
initializing the sysctl mibs data before actually using them.

The original patchset (which is the actual version that is running
on my testboxes) have checked whether all of these sysctls and
refuses to do background fsck if we don't have them. Kirk has
pointed out that refusing running fsck on old kernels is pointless,
as old kernels will recompute the summary at mount time, so I
have removed these checks.

Unfortunatelly, as the checks will initialize the mib values of
those sysctl's, and which are vital for the runtime summary
adjustment to work, we can not simply remove the check, which
will lead to problem when running background fsck over a dirty
volume. Add these checks in a different way: give a warning rather
than refusing to work, and complain if the functionality is not
available when adjustments are necessary.

Noticed by: A power failure at my lab
Pointy hat: me
MFC After: 3 days

The recomputation of file system summary at mount time can be a
very slow process, especially for large file systems that is just
recovered from a crash.

Since the summary is already re-sync'ed every 30 second, we will
not lag behind too much after a crash. With this consideration
in mind, it is more reasonable to transfer the responsibility to
background fsck, to reduce the delay after a crash.

Add a new sysctl variable, vfs.ffs.compute_summary_at_mount, to
control this behavior. When set to nonzero, we will get the
"old" behavior, that the summary is computed immediately at mount
time.

Add five new sysctl variables to adjust ndir, nbfree, nifree,
nffree and numclusters respectively. Teach fsck_ffs about these
API, however, intentionally not to check the existence, since
kernels without these sysctls must have recomputed the summary
and hence no adjustments are necessary.

This change has eliminated the usual tens of minutes of delay of
mounting large dirty volumes.

Reviewed by: mckusick
MFC After: 1 week

Eliminate linked list used to track inodes with an initial link
count of zero and instead encode this information in the inode state.
Pass 4 performed a linear search of this list for each inode in
the file system, which performs poorly if the list is long.

Reviewed by: sam & keramida (an earlier version of the patch), mckusick
MFC after: 1 month

Remove advertising clause from University of California Regent's license,
per letter dated July 22, 1999.

Approved by: core, imp

Remove a stray \n from a setproctitle.

Submitted by: Niki Denev <nike_d@cytexbg.com>

In the case of a background fsck, periodically update the process title
with a progress update.

Use __FBSDID() to quiet GCC 3.3 warnings.

MFp4: Make ifs consistant with rest of file.

Forced commit to note that the last PR mentionned should have been 40697
and that it was reviewed by mckusick.

Fsck needs to check each CG's rotor values to ensure thay are not -ve.
It seems a common corruption to have them -ve (I've seen it several times)
and if fsck doesn't fix it, it leads to a kernel pagefault.

Reviewd by: kirk
Submitted by: Eric Jacobs <eaja@erols.com> and me independently.
MFC in: 2 days
PR: bin/40967
Approved by: re

Use the standardized CHAR_BIT constant instead of NBBY in userland.

Replace various spelling with FALLTHROUGH which is lint()able

Remove some additional paranoia which Kirk forgot to remove from his
UFS2 commit.

These bits in essence made any instance of "softupdates expected
corrution", (ie blocks marked allocated but not referenced by an
inode etc) result in a exit value for fsck_ffs of 2.

2 is part of the magic and appearantly undocumented protocol between
fsck_FOO and fsck and means "dump into single user mode ASAP.

Sponsored by: DARPA & NAI Labs.

Warning cleanup.

Format changes by peter

This commit adds basic support for the UFS2 filesystem. The UFS2
filesystem expands the inode to 256 bytes to make space for 64-bit
block pointers. It also adds a file-creation time field, an ability
to use jumbo blocks per inode to allow extent like pointer density,
and space for extended attributes (up to twice the filesystem block
size worth of attributes, e.g., on a 16K filesystem, there is space
for 32K of attributes). UFS2 fully supports and runs existing UFS1
filesystems. New filesystems built using newfs can be built in either
UFS1 or UFS2 format using the -O option. In this commit UFS1 is
the default format, so if you want to build UFS2 format filesystems,
you must specify -O 2. This default will be changed to UFS2 when
UFS2 proves itself to be stable. In this commit the boot code for
reading UFS2 filesystems is not compiled (see /sys/boot/common/ufsread.c)
as there is insufficient space in the boot block. Once the size of the
boot block is increased, this code can be defined.

Things to note: the definition of SBSIZE has changed to SBLOCKSIZE.
The header file <ufs/ufs/dinode.h> must be included before
<ufs/ffs/fs.h> so as to get the definitions of ufs2_daddr_t and
ufs_lbn_t.

Still TODO:
Verify that the first level bootstraps work for all the architectures.
Convert the utility ffsinfo to understand UFS2 and test growfs.
Add support for the extended attribute storage. Update soft updates
to ensure integrity of extended attribute storage. Switch the
current extended attribute interfaces to use the extended attribute
storage. Add the extent like functionality (framework is there,
but is currently never used).

Sponsored by: DARPA & NAI Labs.
Reviewed by: Poul-Henning Kamp <phk@freebsd.org>

UFS2 preparation commit:
Remove support for converting old FFS formats to newer.

Submitted by: mckusick
Sponspored by: DARPA & NAI Labs.

Minor bug fixes uncovered while working on UFS2.
They should have no effect on existing systems.

o __P removed
o ansi function prototypes
o unifdef -D__STDC__
o __dead2 on usage prototype
o remove now-bogus main prototype

Remove 'register' keyword.
It does not help modern compilers, and some may take some hit from it.
(I also found several functions that listed *every* of its 10 local vars with
"register" -- just how many free registers do people think machines have?)

Fix a large number of -Wall, -Wformat and -W compiler warnings.
These were mainly missing casts or wrong format strings in printf
statements, but there were also missing includes, unused variables,
functions and arguments.

The choice of `long' vs `int' still seems almost random in a lot
of places though.

Minor background cleanups:
1) Set the FS_NEEDSFSCK flag when unexpected problems are encountered.
2) Clear the FS_NEEDSFSCK flag after a successful foreground cleanup.
3) Refuse to run in background when the FS_NEEDSFSCK flag is set.
4) Avoid taking and removing a snapshot when the filesystem is already clean.
5) Properly implement the force cleaning (-f) flag when in preen mode.

Note that you need to have revision 1.21 (date: 2001/04/14 05:26:28) of
fs.h installed in <ufs/ffs/fs.h> defining FS_NEEDSFSCK for this to compile.

Use fs->fs_ipg rather than cg->cg_niblk as the latter is only
16-bits and may overflow.

Obtained from: Bruce Evans <bde@zeta.org.au>

Additions to run checks on live filesystems. This change will not
affect current systems until fsck is modified to use these new
facilities. To try out this change, set the fsck passno to zero
in /etc/fstab to cause the filesystem to be mounted without running
fsck, then run `fsck_ffs -p -B <filesystem>' after the system has
been brought up multiuser to run a background cleanup on <filesystem>.
Note that the <filesystem> in question must have soft updates enabled.

Add a simple SIGINFO handler to fsck_ffs. Shortly after receipt of
a SIGINFO (normally via Ctrl-T), a line will be output indicating
the current phase number and progress information relevant to the
current phase.

Approved by: mckusick

Teach fsck about snapshot files. These changes should have no
effect on operation of fsck on filesystems without snapshots.
If you get compilation errors, be sure that you have copies of
/usr/include/sys/mount.h (1.94), /usr/include/sys/stat.h (1.21),
and /usr/include/ufs/ffs/fs.h (1.16) as of July 4, 2000 or later.

$Id$ -> $FreeBSD$

Cosmetic and documentation changes brought from earlier FreeBSD versions.
(e.g. RCS Id:)

Reviewed by: Don Lewis <Don.Lewis@tsc.tdk.com>
Submitted by: Kirk McKusick <mckusick@McKusick.COM>
Obtained from: Mckusick, BSDI and a host of others

This exactly matches Kirks sources imported under the
Tag MCKUSICK2. These are as supplied by kirk with one small
change needed to compile under freeBSD.

Some FreeBSD patches will be added back, though many have been
added to Kirk's sources already.

Use explicitly sized types when laying out the cylinder groups. This
bug was the cause of the 'freeing free frag' panics that people have been
seeing with FreeBSD/alpha. I have a similar patch to newfs but I've not
finished testing it.

There seem to be two messages that were added with soft-updates
support, which need a final "\n". I only observed one line of
mangled output, but I think there is another one which suffers
from the same problem, and thus I provide a patch that covers
both.

PR: 7483
Reviewed by: phk
Submitted by: Stefan Esser <se@FreeBSD.org>

Fixed printf format errors.

Correct use of .Nm. Add rcsid. Remove unused #includes. Use err(3).

Reviewed by: Kirk Mckusick (mckusick@mckusick.com)
Submitted by: luoqi Chen
fix a type in fsck.
(also add a comment that got picked up by mistake but is worth adding)

Reviewed by: dyson@freebsd.org (john Dyson), dg@root.com (david greenman)
Submitted by: Kirk McKusick (mcKusick@mckusick.com)
Obtained from: WHistle development tree

Finished (?) merging with Lite2: cleaned up #include mess.

Merge from Lite2. Note that Lite2 has it's own filesystem clean check
skipping code that overrides ours sooner. One should be eliminated,
but for now it works.

Submitted by: Archie@whistle.com

clear the fmod flag if it's set.

Remove trailing whitespace.

Submitted by: phk, added to by bde

Fix all the warnings from `gcc -Wall'.

Slight change of wording on clean flag not set message to appease some
complaints.

Better support for clean bit: prompt the user to fix it if it's wrong
when not preening, and indicate if it was fixed when preening.

BSD 4.4 Lite sbin Sources

Note: XNSrouted and routed NOT imported here, they shall be imported with
usr.sbin.