sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

sbin: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

Ensure I/O buffers in libufs(3) are 128-byte aligned.

Various disk controllers require their buffers to be aligned to a
cache-line size (128 bytes). For buffers allocated in structures,
ensure that they are 128-byte aligned. Use aligned_malloc to allocate
memory to ensure that the returned memory is 128-byte aligned.

While we are here, we replace the dynamically allocated inode buffer
with a buffer allocated in the uufsd structure just as the superblock
and cylinder group buffers do.

This can be removed if/when the kernel is fixed. Because this problem
has existed on one I/O subsystem or another since the 1990's, we
are probably stuck with dealing with it forever.

The problem most recent showed up in Azure, see:
https://reviews.freebsd.org/D41728
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=267654
Before these fixes were applied, it was confirmed that the changes
in this commit also fixed the issue in Azure.

Reviewed-by: Warner Losh, kib
Tested-by: Souradeep Chakrabarti of Microsoft (earlier version)
PR: 267654
Differential Revision: https://reviews.freebsd.org/D41724

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

Cleanups to fsck_ffs(8).

When checking an inode ensure that it does not have a negative size.
Stop scaning a directory when an unallocated block is found.
Fully clear an inode when it is first allocated.
Ensure that an inode is marked dirty whenever it is updated and that
it has a correct check hash when it is released.

MFC-after: 1 week
Sponsored-by: The FreeBSD Foundation

Fix a bug in fsck_ffs(8) triggered by corrupted filesystems.

The last valid inode in the filesystem is maxino - 1, not maxino.
Thus validity checks should ino < maxino, not ino <= maxino.

Reported-by: Robert Morris
PR: 271312
MFC-after: 1 week
Sponsored-by: The FreeBSD Foundation

Fix off-by-one error in fsck_ffs(8) chkrange() block-number check.

On an amd64-CURRENT machine with an i-node that refers to a block
number that is one too large will cause a core dump, due to writing
beyond the end of blockmap[] and corrupting the next heap block,
which happens to contain a struct inoinfo in inphash[]. Note that
valgrind catches the blockmap[] access.

Reported by: Robert Morris
PR: 271289
MFC after: 1 week
Sponsored by: The FreeBSD Foundation

fsck_ffs: fix the previous change that skipped pass 5 in some cases

The previous change involved calling check_cgmagic() twice in a row
for the same CG in order to differentiate when the CG was already ok vs.
when the CG was rebuilt, but that doesn't work because the second call
(which was supposed to rebuild the CG) returns 0 (indicating that
the CG was not rebuilt) due to the prevfailcg check causing an early
failure return. Fix this by moving the rebuild part of check_cgmagic()
out into a separate function which is called by pass1() when it wants to
rebuild a CG.

Fixes: da86e7a20dc4a4b17e8d9e7630ed9b675cf71702
Reported by: pho
Discussed with: mckusick
Sponsored by: Netflix

Skip Pass 5 in fsck_ffs(8) when corrupt cylinder groups remain unfixed.

Pass 1 of fsck_ffs checks the integrity of all the cylinder groups.
If any are found to have been corrupted it offers to rebuild them.
Pass 5 then makes a second pass over the cylinder groups to validate
their block and inode maps. Pass 5 assumes that the cylinder groups
are not corrupted and can segment fault if they are corrupted. Rather
than rerunning the corruption checks a second time in pass 5, this
fix keeps track whether any corrupt cylinder groups were found but not
fixed in pass 1 either due to running with the -n flag or by explicitly
answering `no' when asked whether to fix a corrupted cylinder group.
If any corrupted cylinder groups remain after pass 1, fsck_ffs will
decline to run pass 5. Instead it marks the filesystem as unclean
so that fsck_ffs will need to be run again before the filesystem can
be mounted.

This patch cleans up and documents the return value from check_cgmagic().
It also renames the variable / parameter "rebuildcg" to "rebuiltcg".
This parameter describes whether the cylinder group has been rebuilt
rather than whether it should be rebuilt.

Reported by: Chuck Silvers
Reviewed by: Chuck Silvers
MFC after: 1 week

Bug fixes for fsck_ffs(8).

Increment a reference count when returning a zero'ed out buffer
after a failed read.

Zero out a structure before using it.

Only dirty a buffer that has been modified.

Submitted by: Chuck Silvers
Sponsored by: Netflix
MFC after: 1 week

Improvement in UFS/FFS directory placement when doing mkdir(2).

The algorithm for laying out new directories was devised in the 1980s
and markedly improved the performance of the filesystem. In those days
large disks had at most 100 cylinder groups and often as few as 10-20.
Modern multi-terrabyte disks have thousands of cylinder groups. The
original algorithm does not handle these large sizes well. This change
attempts to expand the scope of the original algorithm to work well
with these much larger disks while still retaining the properties
of the original algorithm for small disks.

The filesystem implementation is divided into policy routines and
implementation routines. The policy routines can be changed in any
way desired without risk of corrupting the filesystem. The policy
requests are handled by the implementation layer. If the policy
asks for an available resource, it is granted. But if it asks for
an already in-use resource, then the implementation will provide
an available one nearby the request. Thus it is impossible for a
policy to double allocate. This change is limited to the policy
implementation.

This change updates the ffs_dirpref() routine which is responsible
for selecting the cylinder group into which a new directory should
be placed. If we are near the root of the filesystem we aim to
spread them out as much as possible. As we descend deeper from the
root we cluster them closer together around their parent as we
expect them to be more closely interactive. Higher-level directories
like usr/src/sys and usr/src/bin should be separated while the
directories in these areas are more likely to be accessed together
so should be closer. And directories within commands or kernel
subsystems should be closer still.

We pick a range of cylinder groups around the cylinder group of the
directory in which we are being created. The size of the range for
our search is based on our depth from the root of our filesystem.
We then probe that range based on how many directories are already
present. The first new directory is at 1/2 (middle) of the range;
the second is in the first 1/4 of the range, then at 3/4, 1/8, 3/8,
5/8, 7/8, 1/16, 3/16, 5/16, etc.

It is desirable to store the depth of a directory in its on-disk
inode so that it is available when we need it. We add a new field
di_dirdepth to track the depth of each directory. Because there are
few spare fields left in the inode, we choose to share an existing
field in the inode rather than having one of our own. Specifically
we create a union with the di_freelink field. The di_freelink field
is used to track inodes that have been unlinked but remain referenced.
It is not needed until a rmdir(2) operation has been done on a
directory. At that point, the directory has no contents and even
if it is kept active as a current directory is no longer able to
have any new directories or files created in it. Thus the use of
di_dirdepth and di_freelink will never coincide.

Reported by: Timo Voelker
Reviewed by: kib
Tested by: Peter Holm
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39246

Rewrite function definitions with identifier lists.

A few functions snuck in with K&R style definitions.

Also add some missing memory frees.

MFC after: 1 week

Correct several bugs in fsck_ffs(8) triggered by corrupted filesystems.

If a directory entry has an illegal inode number (less than zero
or greater than the last inode in the filesystem) the entry is removed.
If a directory '.' or '..' entry had an illegal inode number they
were being removed. Since fsck_ffs knows what the correct value is
for these two entries fix them rather deleting them.

Add much more extensive cylinder group checks and use them to be
more careful about rebuilding a cylinder group.

Check for out-of-range block numbers before trying to free them.

When a directory is deleted also remove its cache entry created
in pass1 so that later passes do not try to operate on a deleted
directory.

Check for ctime(3) returning NULL before trying to use its return.

When freeing a directory inode, do not try to interpret it as a
directory.

Reserve space in the inostatlist to have room to allocate a
lost+found directory.

If an invalid block number is found past the end of an inode simply
remove it rather than clearing and removing the inode.

Modernize the inoinfo structure to use queue(3) LIST rather than a
handrolled linked list implementation.

Reported by: Bob Prohaska, John-Mark Gurney, and Mark Millard
Tested by: Peter Holm
Reviewed by: Peter Holm
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D38668

Fix printfs for fsck_ffs(8) i386 build.

Reported by: jenkins
Sponsored by: The FreeBSD Foundation

Fix types for fsck_ffs(8) i386 build.

Reported by: jenkins
Reported by: Cy Schubert
Sponsored by: The FreeBSD Foundation

Add support for managing UFS/FFS snapshots to fsck_ffs(8).

The kernel handles the managment of UFS/FFS snapshots. Since UFS/FFS
updates filesystem data (rather than always writing changes to new
locations like ZFS), the kernel must check every filesystem write
to see if the block being written is part of a snapshot. If it is
part of a snapshot, then the kernel must make a copy of the old
block value into a newly allocated block for the snapshot before
allowing the write to be done. Similarly, if a block is being freed,
the kernel must check to see if it is part of a snapshot and let
the snapshot claim the block rather than freeing it for future use.
When a snapshot is freed, its blocks need to be offered to older
snapshots and freed only if no older snapshots wish to claim them.

When snapshots were added to UFS/FFS they were integrated into soft
updates and just a small part of the management of snapshots needed
to be added to fsck_ffs(8) as soft updates minimized the set of
snapshot changes that might need correction. When journaling was
added to soft updates a much more complete knowledge of snapshots
needed to be added to fsck_ffs(8) for it to be able to properly
handle the filesystem changes that a journal rollback needs to do
(specifically the freeing and allocation of blocks). Since this
functionality was unavailable, the use of snapshots was disabled
when running with journaled soft updates.

This set of changes imports the kernel code for the management of
snapshots to fsck_ffs(8). With this code in place it will become
possible to enable snapshots when running with journalled soft
updates. The most immediate benefit will be the ability to use
snapshots to take consistent filesystem dumps on live filesystems.
Future work will be done to update fsck_ffs(8) to be able to use
snapshots to run in background on live filesystems running with
journaled soft updates.

Reviewed by: kib
Tested by: Peter Holm
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36491

Fix printf formating.

Fix for f4fc389.

Reported by: Jenkins
Sponsored by: The FreeBSD Foundation

Properly handle the replacement of a partially allocated root directory.

If the root directory exists but has a bad block number Pass1 will
accept it and setup an inoinfo structure for it. When Pass2 runs
and cannot read the root inode's content because of a bad (or
duplicate) block number, it removes the bad root inode and replaces
it. As part of creating the replacement root inode, it creates an
inoinfo entry for it. But Pass2 did delete the inoinfo entry that
Pass1 had set up for the root inode so ended up with two inoinfo
structures for it. The final step of Pass2 checks that all the ".."
entries are correct adding them if they are missing which resulted
in a second ".." entry being added to the root directory which
definitely did not go over well in the kernel name cache!

Reported by: Peter Holm
Sponsored by: The FreeBSD Foundation

Correct calculation of inode location in getnextino cache.

Fix for 345bfec.

Reported by: Peter Holm
Sponsored by: The FreeBSD Foundation

Correct calculation of inode location in getnextino cache.

Fix for 345bfec.

Reported by: Peter Holm
Sponsored by: The FreeBSD Foundation

Provide cache coherency between getnextinode() and ginode()

The fsck_ffs(8) utility has two subsystems for reading and writing
inodes. The getnextinode() interface is used in Pass 1 (and Pass
1b if needed) to sequentially walk through all the inodes in the
filesystem. The ginode() interface is used to read and write
individual inodes. Pass 1 uses a mix of both interfaces. This
change ensures that ginode() returns a pointer to the inode in the
cache maintained by getnextinode() when that interface holds the
requested inode so that all modifications to the inode are made in
a single place and are all written to the disk together.

Reported by: Peter Holm
Tested by: Peter Holm
Sponsored by: The FreeBSD Foundation

Correctness cleanups in fsck_ffs(8).

Allocation or I/O failures in fsck_ffs(8) could cause segment
faults because of missing checks or not-yet-initialized data
structures. Correct these issues.

Reported by: Peter Holm
Sponsored by: The FreeBSD Foundation

Properly specify the level of indirect block being looked up.
The value is used only for diagnostic purposes so no functional
change should result.

Avoid lost buffers in fsck_ffs.

The ino_blkatoff() and indir_blkatoff() functions failed to release
the buffers holding second and third level indirect blocks. This
commit ensures that these buffers are now properly released.

MFC after: 1 week
Sponsored by: Netflix

Fix fsck_ufs segfault when it needs to rerun.

The segfault was being hit in the rerun of Pass 1 in ginode() when
trying to get an inode that needs to be repaired. When the first run
of fsck_ffs finishes it clears the inode cache, but ginode() was
failing to check properly and tried to access the deallocated cache entry.

Reported by: Peter Holm
Reviewed by: Chuck Silvers
Tested by: Peter Holm and Chuck Silvers
MFC after: 3 days
Sponsored by: Netflix

Make fsck_ffs more persistent in creating a lost+found directory.

When fsck_ffs is running in interactive mode and finds unlinked files,
it offers to either unlink them or place them in a lost+found directory.
If the lost+found directory option is requested and no lost+found
directory exists, fsck_ffs offers to create one. When creating one,
it must allocate an inode and a filesystem block. It attempts to
allocate them from the first cylinder group. If the first cylinder
group has a bad check hash, it gives up.

This change expands the search into later cylinder groups when the
first one fails with a bad check hash.

Reported by: Chuck Silvers
Tested by: Chuck Silvers
MFC after: 1 week
Sponsored by: Netflix

Fix fsck_ffs -R finds unfixed duplicate block errors when rerunning.

This fixes a long-standing but very obscure bug in fsck_ffs when
it is run with the -R (rerun after unexpected errors). It only
occurs if fsck_ffs finds duplicate blocks and they are all contained
in inodes that reside in the first block of inodes (typically among
the first 128 inodes).

Rather than use the usual ginode() interface to walk through the
inodes in pass1, there is a special optimized `getnextinode()'
routine for walking through all the inodes. It has its own private
buffer for reading the inode blocks. If pass 1 finds duplicate
blocks it runs pass 1b to find all the inodes that contain these
duplicate blocks. Pass 1b also uses the `getnextinode()' to search
for the inodes with duplicate blocks. Pass 1b stops when all the
duplicate blocks have been found. If all the duplicate blocks are
found in the first block of inodes, then the getnextinode cache
holds this block of bad inodes. The subsequent cleanup of the inodes
in passes 2-5 is done using ginode() which uses the regular fsck_ffs
cache.

When fsck_ffs restarts, pass1() calls setinodebuf() to point at the
first block of inodes. When it calls getnextinode() to get inode
2, getnextino() sees that its private cache already has the first
set of inodes loaded and starts using them. They are of course the
trashed inodes left over from the previous run of pass1b().

The fix is to always invalidate the getnextinode cache when calling
setinodebuf().

Reported by: Chuck Silvers
Tested by: Chuck Silvers
MFC after: 3 days
Sponsored by: Netflix

Fix fsck_ffs incorrectly reporting "CANNOT READ BLK: NNNN" errors.

A long-standing bug in Pass 1 of fsck_ffs in which it is reading in
blocks of inodes to check their block pointers. It failed to round
up the size of the read to a disk block size. When disks would
accept 512-byte aligned reads, the bug rarely manifested itself.
But many recent disks will no longer accept 512-byte aligned reads
but require 4096-byte aligned reads, so the failure to properly
round-up read sizes to multiples of 4096 bytes makes the error
much more likely to occur.

Reported by: Peter Holm and others
Tested by: Peter Holm and Rozhuk Ivan
MFC after: 3 days
Sponsored by: Netflix

Rewrite the disk I/O management system in fsck_ffs(8). Other than
making fsck_ffs(8) run faster, there should be no functional change.

The original fsck_ffs(8) had its own disk I/O management system.
When gjournal(8) was added to FreeBSD 7, code was added to fsck_ffs(8)
to do the necessary gjournal rollback. Rather than use the existing
fsck_ffs(8) disk I/O system, it wrote its own from scratch. Similarly
when journalled soft updates were added in FreeBSD 9, code was added
to fsck_ffs(8) to do the necessary journal rollback. And once again,
rather than using either of the existing fsck_ffs(8) disk I/O
systems, it wrote its own from scratch. Lastly the fsdb(8) utility
uses the fsck_ffs(8) disk I/O management system. In preparation for
making the changes necessary to enable snapshots to be taken when
using journalled soft updates, it was necessary to have a single
disk I/O system used by all the various subsystems in fsck_ffs(8).

This commit merges the functionality required by all the different
subsystems into a single disk I/O system that supports all of their
needs. In so doing it picks up optimizations from each of them
with the results that each of the subsystems does fewer reads and
writes than it did with its own customized I/O system. It also
greatly simplifies making changes to fsck_ffs(8) since everything
goes through a single place. For example the ginode() function
fetches an inode from the disk. When inode check hashes were added,
they previously had to be checked in the code implementing inode
fetch in each of the three different disk I/O systems. Now they
need only be checked in ginode().

Tested by: Peter Holm
Sponsored by: Netflix

Rename pass4check() to freeblock() and move from pass4.c to inode.c.
The new name more accurately describes what it does and the file move
puts it with other similar functions. Done in preparation for future
cleanups. No functional differences intended.

Sponsored by: Netflix
Historic Footnote: my last FreeBSD svn commit

Followup to -r344552 in which fsck_ffs checks for a size past the
last allocated block of the file and if that is found, shortens the
file to reference the last allocated block thus avoiding having it
reference a hole at its end.

This update corrects an error where fsck_ffs miscalculated the last
logical block of the file when the file contained a large hole.

Reported by: Jamie Landeg-Jones
Tested by: Peter Holm
MFC after: 2 weeks
Sponsored by: Netflix

Properly calculate the last used logical block of a file when checking
inodes that reference directories. While here tighten the check for
comparing the last logical block with the end of the file.

Reported by: Peter Holm
Tested by: Peter Holm
Sponsored by: Netflix

Ensure that inode updates are properly flushed out during the first
pass of fsck_ffs. Some changes, such as check-hash corrections were
being lost.

Reported by: Michael Tuexen (tuexen@)
Tested by: Michael Tuexen (tuexen@)
MFC after: 3 days

Fsck would find, report, and offer to fix inode check-hash failures.
If requested to fix the inode check-hash it would confirm having done
it, but then fail to make the fix. The same code is used in fsdb which,
unlike fsck, would actually fix the inode check-hash.

The discrepancy occurred because fsck has two ways to fetch inodes.
The inode by number function ginode() and the streaming inode
function getnextinode() used during pass1. Fsdb uses the ginode()
function which correctly does the fix, while fsck first encounters
the bad inode check-hash in pass1 where it is using the getnextinode()
function that failed to make the correction. This patch corrects
the getnextinode() function so that fsck now correctly fixes inodes
with incorrect inode check-hashs.

Reported by: Gary Jennejohn <gljennjohn@gmail.com>
Sponsored by: Netflix

Continuing efforts to provide hardening of FFS. This change adds a
check hash to the filesystem inodes. Access attempts to files
associated with an inode with an invalid check hash will fail with
EINVAL (Invalid argument). Access is reestablished after an fsck
is run to find and validate the inodes with invalid check-hashes.
This check avoids a class of filesystem panics related to corrupted
inodes. The hash is done using crc32c.

Note this check-hash is for the inode itself and not any of its
indirect blocks. Check-hash validation may be extended to also
cover indirect block pointers, but that will be a separate (and
more costly) feature.

Check hashes are added only to UFS2 and not to UFS1 as UFS1 is
primarily used in embedded systems with small memories and low-powered
processors which need as light-weight a filesystem as possible.

Reviewed by: kib
Tested by: Peter Holm
Sponsored by: Netflix

Ensure that cylinder-group check-hashes are properly updated when first
creating them and when correcting them when they are found to be corrupted.

Reported by: Don Lewis (truckman@)
Sponsored by: Netflix

In preparation for adding inode check-hashes, clean up and
document the libufs interface for fetching and storing inodes.
The undocumented getino / putino interface has been replaced
with a new getinode / putinode interface.

Convert the utilities that had been using the undocumented
interface to use the new documented interface.

No functional change (as for now the libufs library does not
do inode check-hashes).

Reviewed by: kib
Tested by: Peter Holm
Sponsored by: Netflix

In preparation for adding inode check-hashes, change the fsck_ffs
inodirty() function to have a pointer to the inode being dirtied.
No functional change (as for now the parameter is ununsed).

Sponsored by: Netflix

Revert r313780 (UFS_ prefix)

Prefix UFS symbols with UFS_ to reduce namespace pollution

Followup to r313780. Also prefix ext2's and nandfs's versions with
EXT2_ and NANDFS_.

Reported by: kib
Reviewed by: kib, mckusick
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D9623

Fix a read past the end of a buffer in fsck.

To minimize the time spent scanning all of the directories in pass 2
(Check Pathnames), fsck uses a search order based on the location
of their first block. Zero length directories have no first block,
so the array being used to hold the block numbers of directory
inodes was of zero length. Thus a lookup was done past the end of
the array getting at best a random value and at worst a segment
fault. For zero length directories, this change allocates a one
element block array and initializes it to zero. The effect is that
all zero length directories are handled first in pass 2.

Reviewed by: brooks
Differential Revision: https://reviews.freebsd.org/D14163

Rename cgget => cglookup to clear name space for new libufs function cgget.
No functional change.

General further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 3-Clause license.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.

fsck_ffs: Unsign some variables and make use of reallocarray(3).

Instead of casting listmax and numdirs to unsigned values just define
them as unsigned and avoid the casts. Use reallocarray(3).

While here, fs_ncg is already unsigned so the cast is unnecessary.

Reviewed by: mckusick
MFC after: 2 weeks

Renumber copyright clause 4

Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is too pedantic, so give up on that point.

Submitted by: Jan Schaumann <jschauma@stevens.edu>
Pull Request: https://github.com/freebsd/freebsd/pull/96

prefix UFS symbols with UFS_ to reduce namespace pollution

Specifically:
ROOTINO -> UFS_ROOTINO
WINO -> UFS_WINO
NXADDR -> UFS_NXADDR
NDADDR -> UFS_NDADDR
NIADDR -> UFS_NIADDR
MAXSYMLINKLEN_UFS[12] -> UFS[12]_MAXSYMLINKLEN (for consistency)

Also prefix ext2's and nandfs's NDADDR and NIADDR with EXT2_ and NANDFS_

Reviewed by: kib, mckusick
Obtained from: NetBSD
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D9536

Use MIN/MAX macros from sys/param.h.

MFC after: 2 weeks.

sbin: ake use of our rounddown() macro when sys/param.h is available.

No functional change.

fsck_ffs for pointers replace 0 with NULL.

Found with devel/coccinelle.

Reviewed by: mckusick

Revert 248634 and 248643 (e.g., restoring 248625 and 248639).

Build verified by: Glen Barber (gjb@)

Revert svn r248625

Clang errors around printf could be trivially fixed, but the breakage in
sbin/fsdb were to significant for this type of change.

Submitter of this changeset has been notified and hopefully this can be
restored soon.

Speed up fsck by caching the cylinder group maps in pass1 so
that they do not need to be read again in pass5. As this nearly
doubles the memory requirement for fsck, the cache is thrown away
if other memory needs in fsck would otherwise fail. Thus, the
memory footprint of fsck remains unchanged in memory constrained
environments.

This work was inspired by a paper presented at Usenix's FAST '13:
www.usenix.org/conference/fast13/ffsck-fast-file-system-checker

Details of this implementation appears in the April 2013 of ;login:
www.usenix.org/publications/login/april-2013-volume-38-number-2.
A copy of the April 2013 ;login: paper can also be downloaded
from: www.mckusick.com/publications/faster_fsck.pdf.

Reviewed by: kib
Tested by: Peter Holm
MFC after: 4 weeks

When running with the -d option, instrument fsck_ffs to track the number,
data type, and running time of its I/O operations.

No functional changes.

Fix fsck_ffs build with a 64-bit ino_t.

Original code by: Gleb Kurtsou

Mechanical whitespace cleanup.

MFC after: 3 weeks

The dump, fsck_ffs, fsdb, fsirand, newfs, makefs, and quot utilities
include sys/time.h instead of time.h. This include is incorrect as
per the manpages for the APIs and the POSIX definitions. This commit
replaces sys/time.h where necessary with time.h.

The commit also includes some minor style(9) header fixup in newfs.

This commit is part of a larger effort by Garrett Cooper started in
//depot/user/gcooper/posix-conformance-work/ -- to make FreeBSD more
POSIX compliant.

Submitted by: Garrett Cooper yanegomi at gmail dot com

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

Update the actions previously attempted by the -D option to make them
robust. With these changes fsck is now able to detect and reliably
rebuild corrupted cylinder group maps. The -D option is no longer
necessary as it has been replaced by a prompt asking whether the
corrupted cylinder group should be rebuilt and doing so when requested.
These actions are only offered and taken when running fsck in manual
mode. Corrupted cylinder groups found during preen mode cause the fsck
to fail.

Add the -r option to free up excess unused inodes. Decreasing the
number of preallocated inodes reduces the running time of future
runs of fsck and frees up space that can allocated to files. The -r
option is ignored when running in preen mode.

Reviewed by: Xin LI <delphij@>
Sponsored by: Rsync.net

- Import the HEAD csup code which is the basis for the cvsmode work.

Add a new flag, '-C' which enables a special mode that is intended for
catastrophic recovery. Currently, this mode only validates whether a
cylindergroup has good signature data, and prompts the user to decide
whether to clear it as a whole.

This mode is useful when there is data damage on a disk and you are
working on copy of the original disk, as fsck_ffs(8) tends to abnormally
exit in such case, as a last resort to recover data from the disk.

Implements gjournal support. If file system has gjournal support enabled
and -p flag was given perform fast file system checking (bascially only
garbage collecting of orphaned objects).

Rename bread() to blread() and bwrite() to blwrite() as we now link to
the libufs library, which also implement functions with that names.

Sponsored by: home.pl

Eliminate linked list used to track inodes with an initial link
count of zero and instead encode this information in the inode state.
Pass 4 performed a linear search of this list for each inode in
the file system, which performs poorly if the list is long.

Reviewed by: sam & keramida (an earlier version of the patch), mckusick
MFC after: 1 month

Create DIP_SET() and IBLK_SET() macros to fix lvalue warnings.

Inspired by: kan

Remove advertising clause from University of California Regent's license,
per letter dated July 22, 1999.

Approved by: core, imp

Use __FBSDID() to quiet GCC 3.3 warnings.

Fix a bunch of format string warnings which broke
the sparc64 build.

Tested on: sparc64, i386

Warning cleanup.

Format changes by peter

This commit adds basic support for the UFS2 filesystem. The UFS2
filesystem expands the inode to 256 bytes to make space for 64-bit
block pointers. It also adds a file-creation time field, an ability
to use jumbo blocks per inode to allow extent like pointer density,
and space for extended attributes (up to twice the filesystem block
size worth of attributes, e.g., on a 16K filesystem, there is space
for 32K of attributes). UFS2 fully supports and runs existing UFS1
filesystems. New filesystems built using newfs can be built in either
UFS1 or UFS2 format using the -O option. In this commit UFS1 is
the default format, so if you want to build UFS2 format filesystems,
you must specify -O 2. This default will be changed to UFS2 when
UFS2 proves itself to be stable. In this commit the boot code for
reading UFS2 filesystems is not compiled (see /sys/boot/common/ufsread.c)
as there is insufficient space in the boot block. Once the size of the
boot block is increased, this code can be defined.

Things to note: the definition of SBSIZE has changed to SBLOCKSIZE.
The header file <ufs/ufs/dinode.h> must be included before
<ufs/ffs/fs.h> so as to get the definitions of ufs2_daddr_t and
ufs_lbn_t.

Still TODO:
Verify that the first level bootstraps work for all the architectures.
Convert the utility ffsinfo to understand UFS2 and test growfs.
Add support for the extended attribute storage. Update soft updates
to ensure integrity of extended attribute storage. Switch the
current extended attribute interfaces to use the extended attribute
storage. Add the extent like functionality (framework is there,
but is currently never used).

Sponsored by: DARPA & NAI Labs.
Reviewed by: Poul-Henning Kamp <phk@freebsd.org>

UFS2 preparation commit:
Remove support for converting old FFS formats to newer.

Submitted by: mckusick
Sponspored by: DARPA & NAI Labs.

o __P removed
o ansi function prototypes
o unifdef -D__STDC__
o __dead2 on usage prototype
o remove now-bogus main prototype

Remove 'register' keyword.
It does not help modern compilers, and some may take some hit from it.
(I also found several functions that listed *every* of its 10 local vars with
"register" -- just how many free registers do people think machines have?)

declare locally used globals as static.

Fix a large number of -Wall, -Wformat and -W compiler warnings.
These were mainly missing casts or wrong format strings in printf
statements, but there were also missing includes, unused variables,
functions and arguments.

The choice of `long' vs `int' still seems almost random in a lot
of places though.

Silence non-constant format string warnings by marking functions
as __printflike()/__printf0like(), adding const, or adding missing "%s"
format strings, as appropriate.

MFC after: 2 weeks

Just notify us once when encountering a partially allocated inode.

Additions to run checks on live filesystems. This change will not
affect current systems until fsck is modified to use these new
facilities. To try out this change, set the fsck passno to zero
in /etc/fstab to cause the filesystem to be mounted without running
fsck, then run `fsck_ffs -p -B <filesystem>' after the system has
been brought up multiuser to run a background cleanup on <filesystem>.
Note that the <filesystem> in question must have soft updates enabled.

Make a tighter test for valid inode numbers in getnextinode().

Teach fsck about snapshot files. These changes should have no
effect on operation of fsck on filesystems without snapshots.
If you get compilation errors, be sure that you have copies of
/usr/include/sys/mount.h (1.94), /usr/include/sys/stat.h (1.21),
and /usr/include/ufs/ffs/fs.h (1.16) as of July 4, 2000 or later.

Yesterday I had to fix a badly broken disk, and found that fsck kept dying:

DIR I=64512 CONNECTED. PARENT WAS I=4032
fsck: cannot find inode 995904

fsdb found the inodes with no problem:

fsdb (inum: 64512)> inode 995904
current inode: directory
I=995904 MODE=40777 SIZE=512
MTIME=Feb 14 15:27:07 2000 [0 nsec]
CTIME=Feb 14 15:27:07 2000 [0 nsec]
ATIME=Feb 24 10:31:58 2000 [0 nsec]
OWNER=nobody GRP=nobody LINKCNT=4 FLAGS=0 BLKCNT=2 GEN=38a41386
Direct blocks: 8094568 0 0 0 0 0 0 0 0 0 0 0
Indirect blocks: 0 0 0

The problem turns out to be a program logic error in fsck. It stores
directory inodes internally in hash lists, using the number of
directories to form the hash key:

inpp = &inphead[inumber % numdirs];

Elsewhere, however, it increments numdirs when it finds unattached
directories. I've made the following fix, which solved the problem in
the case in hand.

Submitted by: Greg Lehey <grog@lemis.com>
Reviewed by: Matthew Dillon <dillon@apollo.backplane.com>
Approved by: Kirk McKusick <mckusick@mckusick.com>

$Id$ -> $FreeBSD$

Cosmetic and documentation changes brought from earlier FreeBSD versions.
(e.g. RCS Id:)

Reviewed by: Don Lewis <Don.Lewis@tsc.tdk.com>
Submitted by: Kirk McKusick <mckusick@McKusick.COM>
Obtained from: Mckusick, BSDI and a host of others

This exactly matches Kirks sources imported under the
Tag MCKUSICK2. These are as supplied by kirk with one small
change needed to compile under freeBSD.

Some FreeBSD patches will be added back, though many have been
added to Kirk's sources already.

Avoid trying to malloc > (1<<32) bytes of memory due to an arithmetic
underflow on the alpha.

Fixed printf format errors.

Correct use of .Nm. Add rcsid. Remove unused #includes. Use err(3).

Reviewed by: dyson@freebsd.org (john Dyson), dg@root.com (david greenman)
Submitted by: Kirk McKusick (mcKusick@mckusick.com)
Obtained from: WHistle development tree

Fixed overflow in chkrange(). Some out of bounds block numbers,
e.g. -1, were not detected. Use a bulletproof check that doesn't
depend on special properties of the args or the limit.

PR: 3528

patch up some "int *" vs. "time_t *" (long) mismatches. They could be
nasty if sizeof(int) != sizeof(long).

Finished (?) merging with Lite2: cleaned up #include mess and fixed a
style bug.

Removed a redundant declaration.

Merge from Lite2. Note that Lite2 has it's own filesystem clean check
skipping code that overrides ours sooner. One should be eliminated,
but for now it works.

Fix the case where fsck would not see sparse directories and the kernel would
panic. If such a thing is fixed fsck needs a rerun (and bugs the user to do
so).

Reviewed by: Kirk McKusick

ts_sec -> tv_sec
ts_nsec -> tv_nsec

Remove trailing whitespace.

Submitted by: Philippe Charnier <charnier@lirmm.fr>, distilled by bde

Fix a couple more bogus types that aren't reported by `gcc -Wall'.

Submitted by: phk, added to by bde

Fix all the warnings from `gcc -Wall'.

>The fix for the missing ".." in the root directory is enclosed below.

Submitted by: Kirk McKusick

Fixed so that it can grok old style "fastlinks".

BSD 4.4 Lite sbin Sources

Note: XNSrouted and routed NOT imported here, they shall be imported with
usr.sbin.