Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Merge checked by: Glen Barber (gjb@)

MFC of 247212:
When running with the -d option, instrument fsck_ffs to track the number,
data type, and running time of its I/O operations.

No functional changes.

MFC of 247234:
Catch up with internal API changes for initbarea() and getdatablk()
of fsck_ffs introduced with r247212.

Submitted by: David Wolfskill <david@catwhisker.org>

MFC of 248625:
Speed up fsck by caching the cylinder group maps in pass1 so
that they do not need to be read again in pass5. As this nearly
doubles the memory requirement for fsck, the cache is thrown away
if other memory needs in fsck would otherwise fail. Thus, the
memory footprint of fsck remains unchanged in memory constrained
environments.

This work was inspired by a paper presented at Usenix's FAST '13:
www.usenix.org/conference/fast13/ffsck-fast-file-system-checker

Details of this implementation appears in the April 2013 of ;login:
www.usenix.org/publications/login/april-2013-volume-38-number-2.
A copy of the April 2013 ;login: paper can also be downloaded
from: www.mckusick.com/publications/faster_fsck.pdf.

Reviewed by: kib
Tested by: Peter Holm

MFC of 248639:
Fix the build after addition of cylinder group cacheing (r248625)

Reported by: Glen Barber (gjb@)
Pointy hat to: Kirk McKusick (mckusick@)

MFC of 248673:
Minor formatting fix for printf() to fix clang builds.

Submitted by: db
Reviewed by: gjb

MFC of 248680:
Resolve clang compile errors on amd64/i386 for certain by casting.

compile tested with clang on i386, amd64
compile tested with gcc on i386, amd64, sparc64

Submitted by: delphij

MFC of 248691:
Note that output is in seconds, not msec.
KNF indentation.
No functional change.
No change to printf strings.
No change to casting of printf arguments.

Reported by: Bruce Evans

MFC r243017:

- blk_equals() is too strict. If the journal entry defines more frags
than we're claiming it should still be considered an exact match. This
would previously leak frags that had been extended.
- If there is a sequence number problem in the journal print the sequence
numbers we've seen so far for debugging.
- Clean up the block mask related debuging printfs. Some are redundant.

MFC r240463:
- Fix a typo in debug message.

MFC r238984:
fsck_ffs shall accept the configured journal size, and not refuse to
operate on it if journal size is greater then SUJ_MAX. The later
constant is only to select maximal journal size when user did not
specified size explicitely.

MFC r236976:
For incompleted block allocations or frees, the inode block count usage
must be recalculated. The blk_check pass of suj checker explicitely marks
inodes which owned such blocks as needing block count adjustment. But
ino_adjblks() is only called by cg_trunc pass, which is performed before
blk_check. As result, the block use count for such inodes is left wrong.
This causes full fsck run after journaled run to still find inconsistencies
like 'INCORRECT BLOCK COUNT I=14557 (328 should be 0)' in phase 1.

Fix this issue by running additional adj_blk pass after blk_check, which
updates the field.

MFC r228751:
Change the type of real_dev_bsize variable from long to u_int.
The DIOCGSECTORSIZE takes u_int * as an argument, using long *
causes failures on big-endian targets.

PR: sparc64/163460

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

- Handle the JOP_SYNC case as appropriate.

Reported by: pho

Implement fully asynchronous partial truncation with softupdates journaling
to resolve errors which can cause corruption on recovery with the old
synchronous mechanism.

- Append partial truncation freework structures to indirdeps while
truncation is proceeding. These prevent new block pointers from
becoming valid until truncation completes and serialize truncations.
- On completion of a partial truncate journal work waits for zeroed
pointers to hit indirects.
- softdep_journal_freeblocks() handles last frag allocation and last
block zeroing.
- vtruncbuf/ffs_page_remove moved into softdep_*_freeblocks() so it
is only implemented in one place.
- Block allocation failure handling moved up one level so it does not
proceed with buf locks held. This permits us to do more extensive
reclaims when filesystem space is exhausted.
- softdep_sync_metadata() is broken into two parts, the first executes
once at the start of ffs_syncvnode() and flushes truncations and
inode dependencies. The second is called on each locked buf. This
eliminates excessive looping and rollbacks.
- Improve the mechanism in process_worklist_item() that handles
acquiring vnode locks for handle_workitem_remove() so that it works
more generally and does not loop excessively over the same worklist
items on each call.
- Don't corrupt directories by zeroing the tail in fsck. This is only
done for regular files.
- Push a fsync complete record for files that need it so the checker
knows a truncation in the journal is no longer valid.

Discussed with: mckusick, kib (ffs_pages_remove and ffs_truncate parts)
Tested by: pho

Mechanical whitespace cleanup.

MFC after: 3 weeks

In checker, read journal by sectors.

Due to UFS insistence to pretend that device sector size is 512 bytes,
sector size is obtained from ioctl(DIOCGSECTORSIZE) for real devices,
and from the label otherwise. The file images without label have to
be made with 512 sector size.

In collaboration with: pho
Reviewed by: jeff
Tested by: bz, pho

The dump, fsck_ffs, fsdb, fsirand, newfs, makefs, and quot utilities
include sys/time.h instead of time.h. This include is incorrect as
per the manpages for the APIs and the POSIX definitions. This commit
replaces sys/time.h where necessary with time.h.

The commit also includes some minor style(9) header fixup in newfs.

This commit is part of a larger effort by Garrett Cooper started in
//depot/user/gcooper/posix-conformance-work/ -- to make FreeBSD more
POSIX compliant.

Submitted by: Garrett Cooper yanegomi at gmail dot com

- Permit zero length directories as a handled inconsistency. This allows
directory truncation to proceed before the link has been cleared. This
is accomplished by detecting a directory with no . or .. links and
clearing the named directory entry in the parent.
- Add a new function ino_remref() which handles the details of removing
a reference to an inode as a result of a lost directory. There were
some minor errors in various subcases of this routine.

Improve fsck robustness for SU+J cases:

- Use err/errx only when the case is really fatal. For other
cases, fall back to full fsck instead of quiting fsck.
- Plug a memory leak.
- Avoid divide by zero when printing summary.
- Output "FILE SYSTEM IS MARKED CLEAN" when a successful
journal recovering is done.
- When -f is specified, do full fsck instead of journal recovery.

suj.c seems to contain two versions of the code.
Remove the one that doesn't compile.

- Merge soft-updates journaling from projects/suj/head into head. This
brings in support for an optional intent log which eliminates the need
for background fsck on unclean shutdown.

Sponsored by: iXsystems, Yahoo!, and Juniper.
With help from: McKusick and Peter Holm