Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFH (r250056,r250057): add -Z option to zero unused blocks.

Merge checked by: Glen Barber (gjb@)

MFC of 247212:
When running with the -d option, instrument fsck_ffs to track the number,
data type, and running time of its I/O operations.

No functional changes.

MFC of 247234:
Catch up with internal API changes for initbarea() and getdatablk()
of fsck_ffs introduced with r247212.

Submitted by: David Wolfskill <david@catwhisker.org>

MFC of 248625:
Speed up fsck by caching the cylinder group maps in pass1 so
that they do not need to be read again in pass5. As this nearly
doubles the memory requirement for fsck, the cache is thrown away
if other memory needs in fsck would otherwise fail. Thus, the
memory footprint of fsck remains unchanged in memory constrained
environments.

This work was inspired by a paper presented at Usenix's FAST '13:
www.usenix.org/conference/fast13/ffsck-fast-file-system-checker

Details of this implementation appears in the April 2013 of ;login:
www.usenix.org/publications/login/april-2013-volume-38-number-2.
A copy of the April 2013 ;login: paper can also be downloaded
from: www.mckusick.com/publications/faster_fsck.pdf.

Reviewed by: kib
Tested by: Peter Holm

MFC of 248639:
Fix the build after addition of cylinder group cacheing (r248625)

Reported by: Glen Barber (gjb@)
Pointy hat to: Kirk McKusick (mckusick@)

MFC of 248673:
Minor formatting fix for printf() to fix clang builds.

Submitted by: db
Reviewed by: gjb

MFC of 248680:
Resolve clang compile errors on amd64/i386 for certain by casting.

compile tested with clang on i386, amd64
compile tested with gcc on i386, amd64, sparc64

Submitted by: delphij

MFC of 248691:
Note that output is in seconds, not msec.
KNF indentation.
No functional change.
No change to printf strings.
No change to casting of printf arguments.

Reported by: Bruce Evans

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Forgot this nit in r221107.

Approved by: re (kib)

Add an -E option to mirror newfs's. The idea is that if you have a system
that was built before ffs grew support for TRIM, your filesystem will have
plenty of free blocks that the flash chip doesn't know are free, so it
can't take advantage of them for wear leveling. Once you've upgraded your
kernel, you enable TRIM on the filesystem (tunefs -t enable), then run
fsck_ffs -E on it before mounting it.

I tested this patch by half-filling an mdconfig'ed filesystem image,
running fsck_ffs -E on it, then verifying that the contents were not
damaged by comparing them to a pristine copy using rsync's checksum
functionality. There is no reliable way to test it on real hardware.

Many thanks to mckusick@, who provided the tricky parts of this patch and
reviewed the final version.

Reviewed by: mckusick@
MFC after: 3 weeks

Mechanical whitespace cleanup.

MFC after: 3 weeks

The dump, fsck_ffs, fsdb, fsirand, newfs, makefs, and quot utilities
include sys/time.h instead of time.h. This include is incorrect as
per the manpages for the APIs and the POSIX definitions. This commit
replaces sys/time.h where necessary with time.h.

The commit also includes some minor style(9) header fixup in newfs.

This commit is part of a larger effort by Garrett Cooper started in
//depot/user/gcooper/posix-conformance-work/ -- to make FreeBSD more
POSIX compliant.

Submitted by: Garrett Cooper yanegomi at gmail dot com

One question mark per question; everything else is just exaggerating.
reply() will output a '?', when printing the question along with [yn],
so no need to have another here.

Improve fsck robustness for SU+J cases:

- Use err/errx only when the case is really fatal. For other
cases, fall back to full fsck instead of quiting fsck.
- Plug a memory leak.
- Avoid divide by zero when printing summary.
- Output "FILE SYSTEM IS MARKED CLEAN" when a successful
journal recovering is done.
- When -f is specified, do full fsck instead of journal recovery.

- Merge soft-updates journaling from projects/suj/head into head. This
brings in support for an optional intent log which eliminates the need
for background fsck on unclean shutdown.

Sponsored by: iXsystems, Yahoo!, and Juniper.
With help from: McKusick and Peter Holm

Add some error messages suggested in PR bin/138043. The code to
correct the problem was added in r176575 by delphij on 2008-02-25.

PR: 138043
Reported by: Heikki Suonsivu

Update the actions previously attempted by the -D option to make them
robust. With these changes fsck is now able to detect and reliably
rebuild corrupted cylinder group maps. The -D option is no longer
necessary as it has been replaced by a prompt asking whether the
corrupted cylinder group should be rebuilt and doing so when requested.
These actions are only offered and taken when running fsck in manual
mode. Corrupted cylinder groups found during preen mode cause the fsck
to fail.

Add the -r option to free up excess unused inodes. Decreasing the
number of preallocated inodes reduces the running time of future
runs of fsck and frees up space that can allocated to files. The -r
option is ignored when running in preen mode.

Reviewed by: Xin LI <delphij@>
Sponsored by: Rsync.net

Add the '-C' "check clean" flag. If the FS is marked clean, skip file
system checking. However, if the file system is not clean, perform a
full fsck.

Reviewed by: delphij
Obtained from: Juniper Networks

Rename option 'C' to 'D' (damaged) in order to avoid a conflict with upcoming
Juniper 'C' (clean) flag.

Requested by: obrien
MFC after: 1 week

Garbage collect 'fflags'.

Instead of passing MNT_UPDATE, MNT_SNAPSHOT, MNT_RELOAD from
userspace to kernel via nmount(), pass in the strings
"update", "snapshot", "reload".

We want to move away from passing MNT_ flags from userspace -> kernel
via nmount(), and instead favor passing the string options.

- Display '-C' option in usage().

Approved by: kib

Add a new flag, '-C' which enables a special mode that is intended for
catastrophic recovery. Currently, this mode only validates whether a
cylindergroup has good signature data, and prompts the user to decide
whether to clear it as a whole.

This mode is useful when there is data damage on a disk and you are
working on copy of the original disk, as fsck_ffs(8) tends to abnormally
exit in such case, as a last resort to recover data from the disk.

Add comment about specifying "ro" mount option when
doing an update mount on a read-only file system.

Requested by: yar

For a mounted file system which is read-only, when
doing the MNT_RELOAD, pass in "ro" and "update"
string mount options to nmount() instead of MNT_RDONLY and MNT_UPDATE flags.

Due to the complexity of the mount parsing code especially
with respect to the root file system, passing in MNT_RDONLY and MNT_UPDATE
flags would do weird things and would cause fsck to convert the root
file system from a read-only mount to read-write.

To test:
- boot into single user mode
- show mounted file systems with: mount
- root file system should be mounted read-only
- fsck /
- show mounted file systems with: mount
- root file system should still be mounted read-only

PR: 120319
MFC after: 1 month
Reported by: yar

Remove hacks to filter out MNT_ROOTFS, since we now
do that internally inside nmount() in revision 1.267 of vfs_mount.c.

Convert fsck_ffs to nmount(). This seems to solve
an intermittent problem where MNT_RELOAD fails
for the root file system.

Reported and tested by: phk
Approved by: re (bmah)

Fix fscking gjournaled root file system: root file system is already mounted
read-only, so we can't simply exit right after calling gjournal_check(),
instead we need to ask about super block reload.

Submitted by: Niki Denev <niki@totalterror.net>
PR: misc/113889
Approved by: re (kensmith)

Implements gjournal support. If file system has gjournal support enabled
and -p flag was given perform fast file system checking (bascially only
garbage collecting of orphaned objects).

Rename bread() to blread() and bwrite() to blwrite() as we now link to
the libufs library, which also implement functions with that names.

Sponsored by: home.pl

Sync program's usage() with manpage's SYNOPSIS.

Do not pass random bits as mount arguments.

Eliminate linked list used to track inodes with an initial link
count of zero and instead encode this information in the inode state.
Pass 4 performed a linear search of this list for each inode in
the file system, which performs poorly if the list is long.

Reviewed by: sam & keramida (an earlier version of the patch), mckusick
MFC after: 1 month

Remove advertising clause from University of California Regent's license,
per letter dated July 22, 1999.

Approved by: core, imp

In the case of a background fsck, periodically update the process title
with a progress update.

Create a .snap directory mode 770 group operator in the root of each
filesystem that is checked in background. Create the snapshot in this
directory rather than in the root. There are two benefits:

1) For terabyte-sized filesystems, the snapshot may require many
minutes to build. Although the filesystem will not be suspended
during most of the snapshot build, the snapshot file itself is
locked during the entire snapshot build period. Thus, if it is
accessed during the period that it is being built, the process
trying to access it will block holding its containing directory
locked. If the snapshot is in the root, the root will lock and
the system will come to a halt until the snapshot finishes. By
putting the snapshot in a subdirectory, it is out of the likely
path of any process traversing through the root and hence much
less likely to cause a lock race to the root.

2) The dump program is usually run by a non-root user running with
operator group privilege. Such a user is typically not permitted
to create files in the root of a filesystem. By having a directory
in group operator with group write access available, such a user
will be able to create a snapshot there. Having the dump program
create its snapshot in a subdirectory below the root will benefit
from point (1) as well.

Sponsored by: DARPA & NAI Labs.

Spell "file system" correctly.

Use __FBSDID() to quiet GCC 3.3 warnings.

s/filesystem/file system/g as discussed on -developers

Fix a bunch of format string warnings which broke
the sparc64 build.

Tested on: sparc64, i386

Warning cleanup.

Format changes by peter

This commit adds basic support for the UFS2 filesystem. The UFS2
filesystem expands the inode to 256 bytes to make space for 64-bit
block pointers. It also adds a file-creation time field, an ability
to use jumbo blocks per inode to allow extent like pointer density,
and space for extended attributes (up to twice the filesystem block
size worth of attributes, e.g., on a 16K filesystem, there is space
for 32K of attributes). UFS2 fully supports and runs existing UFS1
filesystems. New filesystems built using newfs can be built in either
UFS1 or UFS2 format using the -O option. In this commit UFS1 is
the default format, so if you want to build UFS2 format filesystems,
you must specify -O 2. This default will be changed to UFS2 when
UFS2 proves itself to be stable. In this commit the boot code for
reading UFS2 filesystems is not compiled (see /sys/boot/common/ufsread.c)
as there is insufficient space in the boot block. Once the size of the
boot block is increased, this code can be defined.

Things to note: the definition of SBSIZE has changed to SBLOCKSIZE.
The header file <ufs/ufs/dinode.h> must be included before
<ufs/ffs/fs.h> so as to get the definitions of ufs2_daddr_t and
ufs_lbn_t.

Still TODO:
Verify that the first level bootstraps work for all the architectures.
Convert the utility ffsinfo to understand UFS2 and test growfs.
Add support for the extended attribute storage. Update soft updates
to ensure integrity of extended attribute storage. Switch the
current extended attribute interfaces to use the extended attribute
storage. Add the extent like functionality (framework is there,
but is currently never used).

Sponsored by: DARPA & NAI Labs.
Reviewed by: Poul-Henning Kamp <phk@freebsd.org>

more file system > filesystem

Usage style sweep: spell "usage" with a small 'u'.
Also change one case of blatant __progname abuse (several more remain)
This commit does not touch anything in src/{contrib,crypto,gnu}/.

Replace __progname with the documented, more acceptable and functionally identical getprogname(3).

o __P removed
o ansi function prototypes
o unifdef -D__STDC__
o __dead2 on usage prototype
o remove now-bogus main prototype

Fix a large number of -Wall, -Wformat and -W compiler warnings.
These were mainly missing casts or wrong format strings in printf
statements, but there were also missing includes, unused variables,
functions and arguments.

The choice of `long' vs `int' still seems almost random in a lot
of places though.

Update usage message with new options.

Submitted by: Ruslan Ermilov <ru@FreeBSD.org>

Add support for the -F flag which determines whether a specified
filesystem needs foreground checking (usually at boot time) or
can defer to background checking (after the system is up and running).
See the manual page, fsck_ffs(8), for details on the -F and -B options.
These options are primarily intended for use by the fsck front end.

All output is directed to stdout so that the output is coherent
when redirected to a file or a pipe. Unify the code with the fsck
front end that allows either a device or a mount point to be
specified as the argument to be checked.

Minor background cleanups:
1) Set the FS_NEEDSFSCK flag when unexpected problems are encountered.
2) Clear the FS_NEEDSFSCK flag after a successful foreground cleanup.
3) Refuse to run in background when the FS_NEEDSFSCK flag is set.
4) Avoid taking and removing a snapshot when the filesystem is already clean.
5) Properly implement the force cleaning (-f) flag when in preen mode.

Note that you need to have revision 1.21 (date: 2001/04/14 05:26:28) of
fs.h installed in <ufs/ffs/fs.h> defining FS_NEEDSFSCK for this to compile.

Additions to run checks on live filesystems. This change will not
affect current systems until fsck is modified to use these new
facilities. To try out this change, set the fsck passno to zero
in /etc/fstab to cause the filesystem to be mounted without running
fsck, then run `fsck_ffs -p -B <filesystem>' after the system has
been brought up multiuser to run a background cleanup on <filesystem>.
Note that the <filesystem> in question must have soft updates enabled.

Add a simple SIGINFO handler to fsck_ffs. Shortly after receipt of
a SIGINFO (normally via Ctrl-T), a line will be output indicating
the current phase number and progress information relevant to the
current phase.

Approved by: mckusick

Reviewed by: rwatson, bp
Approved by: rwatson
Obtained from: NetBSD-current source tree

The beginnings of the fsck wrappers stuff from NetBSD. This particular commit
brings a newly repo-copied sbin/fsck_ffs/ (from sbin/fsck/) into fsck wrappers
mode.

A quick overview (the code reflects this):

* Documentation changed to reflect fsck_ffs instead of fsck
* Simply acts on a single filesystem, doesn't try to do any multiple filesystem
magic - this is done by the fsck wrappers now

And then specific to fsck_ffs:

* link to /sbin/fsck_4.2bsd and /sbin/fsck_ufs. This is because right now
the filesystem is of type ufs not ffs, and that during autodetection the
labeltype rather than the VFS type is used - this is because when doing
an autodetection of filesystem type in the fsck wrapper program, it does
not have any link between label type (4.2bsd, vinum, etc) and VFS string.

Note that this shouldn't break a build since the required buildworld Makefile
magic and import of the fsck wrapper code into src/sbin/fsck/ will happen
in a seperate commit.

Fix the use of an uninitialized variable in the previous commit.

Also, in addition to the previous log message, the last change had a fix
for the case where where f.mntfromname is a relative path like da0a.

Submitted by: bde

- Style/bde changes.
- Don't use realpath as stat does the right thing.
- Only check ufs filesystems in getmntpt.
- Dont' bother checking that the ufs-mounted-on
device is a special file. It *must* be a special
file, or ufs wouldn't have mounted it.

Submitted by: Paul Saab <ps@yahoo-inc.com>

Make fsck(8) do a MNT_RELOAD after cleaning for all read-only mounted
filesystems, not just for the root fs.

Reviewed by: mckusick
Submitted by: Paul Saab <ps@yahoo-inc.com>

$Id$ -> $FreeBSD$

Cosmetic and documentation changes brought from earlier FreeBSD versions.
(e.g. RCS Id:)

Reviewed by: Don Lewis <Don.Lewis@tsc.tdk.com>
Submitted by: Kirk McKusick <mckusick@McKusick.COM>
Obtained from: Mckusick, BSDI and a host of others

This exactly matches Kirks sources imported under the
Tag MCKUSICK2. These are as supplied by kirk with one small
change needed to compile under freeBSD.

Some FreeBSD patches will be added back, though many have been
added to Kirk's sources already.

For large filesystems you can run past default resource limits causing
fsck to exit unhappily. Fix this by doing a getrlimit/setrlimit for
RLIMIT_DATA. I made the same fix in NetBSD.

Reviewed by: dg@root.com

Correct use of .Nm. Add rcsid. Remove unused #includes. Use err(3).

Reviewed by: dyson@freebsd.org (john Dyson), dg@root.com (david greenman)
Submitted by: Kirk McKusick (mcKusick@mckusick.com)
Obtained from: WHistle development tree

Fixed style bugs in the printing of statistics after preening. Use
floating point better in the percentage calculation there to avoid
overflow when there are more than about 20 million fragments. Start
using floating point in the other percentage calculation to avoid
overflow when there are more than about 2 million fragments.

Fixed printf format strings.

Converted sccsid to rcsid.

Don't attempt to print the statistics for a "clean" preened filesystem
when there isn't even a filesystem. Attempting to print them tended
to cause SIGSEGV or SIGFPE depending on how far setup() got before it
returned 0. This was broken in the previous revision by removing a
return statement that the previous case depended on falling into.

PR: 4840 (fixed by this commit)
PR: 2537 (possibly fixed by Lite2 merge and later changes. setup()
does more checking now)

Kill the Lite2 early "filesystem clean abort" check and go back to
something closer to how we used to do it. The Lite2 way is to check the
"fsclean" flag in the superblock and stop there if so (during preen).
We now do the various superblock sanity checks that we used to do before
since it's cheap. We now get the filesystem state summary again instead
of "FILESYSTEM CLEAN; CHECKING SKIPPED" (or whatever).

Finished (?) merging with Lite2: cleaned up #include mess.

Updated getopt() usage.

Merge from Lite2. Note that Lite2 has it's own filesystem clean check
skipping code that overrides ours sooner. One should be eliminated,
but for now it works.

Fix the case where fsck would not see sparse directories and the kernel would
panic. If such a thing is fixed fsck needs a rerun (and bugs the user to do
so).

Reviewed by: Kirk McKusick

Remove unneeded ctype.h

Remove trailing whitespace.

Submitted by: phk, added to by bde

Fix all the warnings from `gcc -Wall'.

Better support for clean bit: prompt the user to fix it if it's wrong
when not preening, and indicate if it was fixed when preening.

Added filesystem clean bit support. This only affects fsck during a
preen (-p), and in that case the filesystem is skipped if it is clean.
A new flag "-f" for 'force' has been added which basically gives back
the old behavior of checking all the filesystems all the time. This
very closely models the behavior of SunOS and Ultrix.

BSD 4.4 Lite sbin Sources

Note: XNSrouted and routed NOT imported here, they shall be imported with
usr.sbin.