sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

sbin: Remove ancient SCCS tags.

Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.

Sponsored by: Netflix

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

newfs: nextnum should be a u_int32_t, not an int

The function that uses nextnum expects to return a u_int32_t, not a mere
int, so let's make nextnum a u_int32_t instead.

Note: retained current u_int32_t style, since the rest of the file uses
it.

Reviewed by: imp, mckusick
Pull Request: https://github.com/freebsd/freebsd-src/pull/734

ffs: restore backward compatibility of newfs and makefs with older binaries

The previous change to CGSIZE had the unintended side-effect of allowing
newfs and makefs to create file systems that would fail validation when
examined by older commands and kernels, by allowing newfs/makefs to pack
slightly more blocks into a CG than those older binaries think is valid.
Fix this by having newfs/makefs artificially restrict the number of blocks
in a CG to the slightly smaller value that those older binaries will accept.
The validation code will continue to accept the slightly larger value
that the current newfs/makefs (before this change) could create.

Fixes: 0a6e34e950cd5889122a199c34519b67569be9cc
Reviewed by: mckusick
MFC after: 3 days
Sponsored by: Netflix

Fix size differences between architectures of the UFS/FFS CGSIZE macro value.

The cylinder group header structure ended with `u_int8_t cg_space[1]'
representing the beginning of the inode bitmap array. Some architectures
like the i386 rounded this up to a 4-byte boundry while other
architectures like the amd64 rounded it up to an 8-byte boundry.
Thus sizeof(struct cg) was four bytes bigger on an amd64 machine
than on an i386 machine. If a filesystem created on an i386 machine
was moved to an amd64 machine, the size of the cylinder group
calculated by the CGSIZE macro would appear to grow by four bytes.
Filesystems whose cylinder groups were exactly equal to the block
size on an i386 machine would appear to have a cylinder group that
was four bytes too big when moved to an amd64 machine. Note that
although the structure appears to be too big, it in fact is fine.
It is just the calaculation of its size that is in error.

The fix is to remove the cg_space element from the cylinder-group
structure so that the calculated size of the structure is the same
size on all architectures.

Reported by: Tijl Coosemans
Tested by: Tijl Coosemans and Peter Holm
MFC after: 1 week
Sponsored by: The FreeBSD Foundation

newfs: fix up 32-bit compile

Sponsored by: Rubicon Communications, LLC ("Netgate")

Additional validity checking in newfs(8).

A check in the superblock validity code verifies that the computed
size of the filesystem cylinder groups (CGSIZE macro) does not
exceed the filesystem block size (fs_bsize).

A report was received that a filesystem had been flagged as failing
this check. We were unable to determine how the reported filesystem
could have been created. This commit adds a check at the end of the
newfs(8) command to verify that the the cylinder group size is valid.
If an oversize cylinder group is found newfs(8) prints a diagnostic
output and rebuilds the filesystem to make it compiliant.

MFC after: 1 week

Improvement in UFS/FFS directory placement when doing mkdir(2).

The algorithm for laying out new directories was devised in the 1980s
and markedly improved the performance of the filesystem. In those days
large disks had at most 100 cylinder groups and often as few as 10-20.
Modern multi-terrabyte disks have thousands of cylinder groups. The
original algorithm does not handle these large sizes well. This change
attempts to expand the scope of the original algorithm to work well
with these much larger disks while still retaining the properties
of the original algorithm for small disks.

The filesystem implementation is divided into policy routines and
implementation routines. The policy routines can be changed in any
way desired without risk of corrupting the filesystem. The policy
requests are handled by the implementation layer. If the policy
asks for an available resource, it is granted. But if it asks for
an already in-use resource, then the implementation will provide
an available one nearby the request. Thus it is impossible for a
policy to double allocate. This change is limited to the policy
implementation.

This change updates the ffs_dirpref() routine which is responsible
for selecting the cylinder group into which a new directory should
be placed. If we are near the root of the filesystem we aim to
spread them out as much as possible. As we descend deeper from the
root we cluster them closer together around their parent as we
expect them to be more closely interactive. Higher-level directories
like usr/src/sys and usr/src/bin should be separated while the
directories in these areas are more likely to be accessed together
so should be closer. And directories within commands or kernel
subsystems should be closer still.

We pick a range of cylinder groups around the cylinder group of the
directory in which we are being created. The size of the range for
our search is based on our depth from the root of our filesystem.
We then probe that range based on how many directories are already
present. The first new directory is at 1/2 (middle) of the range;
the second is in the first 1/4 of the range, then at 3/4, 1/8, 3/8,
5/8, 7/8, 1/16, 3/16, 5/16, etc.

It is desirable to store the depth of a directory in its on-disk
inode so that it is available when we need it. We add a new field
di_dirdepth to track the depth of each directory. Because there are
few spare fields left in the inode, we choose to share an existing
field in the inode rather than having one of our own. Specifically
we create a union with the di_freelink field. The di_freelink field
is used to track inodes that have been unlinked but remain referenced.
It is not needed until a rmdir(2) operation has been done on a
directory. At that point, the directory has no contents and even
if it is kept active as a current directory is no longer able to
have any new directories or files created in it. Thus the use of
di_dirdepth and di_freelink will never coincide.

Reported by: Timo Voelker
Reviewed by: kib
Tested by: Peter Holm
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39246

sys/param.h: Add _WANT_P_OSREL

Use it instead of defining IN_RTLD by base sources that want P_OSREL_
defines in userspace, but are not rtld.
This allows to remove abuse of IN_RTLD from userspace.

Reviewed by: dchagin, markj, imp
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D38585

Delete UFS2 backup superblock recovery info when building a UFS1 filesystem.

Only the UFS2 filesystem has support for storing information needed
to find alternate superblocks. If that information is inadvertently
left in place when building a UFS1 filesystem, fsck_ffs may stumble
across it and attempt to use it to recover the UFS1 filesystem
which can only end poorly.

newfs(8): Fix a bug in initialization of sblock.fs_maxbsize .

Fixes: 1c85e6a35d93195e896b030d9a55f7ac4ccee2c3 (SVN r98542)
Pull Request: https://github.com/freebsd/freebsd-src/pull/587
MFC after: 1 week
Sponsored by: NVIDIA Networking

In the newfs(8) utility, use the more appropriate sbwrite() and cgwrite()
libufs interfaces rather than sbput() and cgput().

No functional change.

MFC after: 7 days
Sponsored by: Netflix

Move the pointers stored in the superblock into a separate
fs_summary_info structure. This change was originally done
by the CheriBSD project as they need larger pointers that
do not fit in the existing superblock.

This cleanup of the superblock eases the task of the commit
that immediately follows this one.

Suggested by: brooks
Reviewed by: kib
PR: 246983
Sponsored by: Netflix

Continuing efforts to provide hardening of FFS. This change adds a
check hash to the filesystem inodes. Access attempts to files
associated with an inode with an invalid check hash will fail with
EINVAL (Invalid argument). Access is reestablished after an fsck
is run to find and validate the inodes with invalid check-hashes.
This check avoids a class of filesystem panics related to corrupted
inodes. The hash is done using crc32c.

Note this check-hash is for the inode itself and not any of its
indirect blocks. Check-hash validation may be extended to also
cover indirect block pointers, but that will be a separate (and
more costly) feature.

Check hashes are added only to UFS2 and not to UFS1 as UFS1 is
primarily used in embedded systems with small memories and low-powered
processors which need as light-weight a filesystem as possible.

Reviewed by: kib
Tested by: Peter Holm
Sponsored by: Netflix

In preparation for adding inode check-hashes, clean up and
document the libufs interface for fetching and storing inodes.
The undocumented getino / putino interface has been replaced
with a new getinode / putinode interface.

Convert the utilities that had been using the undocumented
interface to use the new documented interface.

No functional change (as for now the libufs library does not
do inode check-hashes).

Reviewed by: kib
Tested by: Peter Holm
Sponsored by: Netflix

Continuing efforts to provide hardening of FFS, this change adds a
check hash to the superblock. If a check hash fails when an attempt
is made to mount a filesystem, the mount fails with EINVAL (Invalid
argument). This avoids a class of filesystem panics related to
corrupted superblocks. The hash is done using crc32c.

Check hases are added only to UFS2 and not to UFS1 as UFS1 is primarily
used in embedded systems with small memories and low-powered processors
which need as light-weight a filesystem as possible.

Reviewed by: kib
Tested by: Peter Holm
Sponsored by: Netflix

Revert r313780 (UFS_ prefix)

Prefix UFS symbols with UFS_ to reduce namespace pollution

Followup to r313780. Also prefix ext2's and nandfs's versions with
EXT2_ and NANDFS_.

Reported by: kib
Reviewed by: kib, mckusick
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D9623

The goal of this change is to prevent accidental foot shooting by
folks running filesystems created on check-hash enabled kernels
(which I will call "new") on a non-check-hash enabled kernels (which
I will call "old). The idea here is to detect when a filesystem is
run on an old kernel and flag the filesystem so that when it gets
moved back to a new kernel, it will not start getting a slew of
check-hash errors.

Back when the UFS version 2 filesystem was created, it added a file
flag FS_INDEXDIRS that was to be set on any filesystem that kept
some sort of on-disk indexing for directories. The idea was precisely
to solve the issue we have today. Specifically that a newer kernel
that supported indexing would be able to tell that the filesystem
had been run on an older non-indexing kernel and that the indexes
should not be used until they had been rebuilt. Since we have never
implemented on-disk directory indicies, the FS_INDEXDIRS flag is
cleared every time any UFS version 2 filesystem ever created is
mounted for writing.

This commit repurposes the FS_INDEXDIRS flag as the FS_METACKHASH
flag. Thus, the FS_METACKHASH is definitively known to have always
been cleared. The FS_INDEXDIRS flag has been moved to a new block
of flags that will always be cleared starting with this commit
(until they get used to implement some future feature which needs
to detect that the filesystem was mounted on a kernel that predates
the new feature).

If a filesystem with check-hashes enabled is mounted on an old
kernel the FS_METACKHASH flag is cleared. When that filesystem is
mounted on a new kernel it will see that the FS_METACKHASH has been
cleared and clears all of the fs_metackhash flags. To get them
re-enabled the user must run fsck (in interactive mode without the
-y flag) which will ask for each supported check hash whether it
should be rebuilt and enabled. When fsck is run in its default preen
mode, it will just ignore the check hashes so they will remain
disabled.

The kernel has always disabled any check hash functions that it
does not support, so as more types of check hashes are added, we
will get a non-surprising result. Specifically if filesystems get
moved to kernels supporting fewer of the check hashes, those that
are not supported will be disabled. If the filesystem is moved back
to a kernel with more of the check-hashes available and fsck is run
interactively to rebuild them, then their checking will resume.
Otherwise just the smaller subset will be checked.

A side effect of this commit is that filesystems running with
cylinder-group check hashes will stop having them checked until
fsck is run to re-enable them (since none of them currently have
the FS_METACKHASH flag set). So, if you want check hashes enabled
on your filesystems after booting a kernel with these changes, you
need to run fsck to enable them. Any newly created filesystems will
have check hashes enabled. If in doubt as to whether you have check
hashes emabled, run dumpfs and look at the list of enabled flags
at the end of the superblock details.

Check and report error returns from sbput(3) calls.
Convert to using cgput(3) for writing cylinder groups.
Check and report error returns from cgput(3).

Submitted by: Bruce Evans <bde@freebsd.org>

Refactoring of reading and writing of the UFS/FFS superblock.
Specifically reading is done if ffs_sbget() and writing is done
in ffs_sbput(). These functions are exported to libufs via the
sbget() and sbput() functions which then used in the various
filesystem utilities. This work is in preparation for adding
subperblock check hashes.

No functional change intended.

Reviewed by: kib

General further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 3-Clause license.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.

Rename P_OSREL_CK_CLYGRP to P_OSREL_CK_CYLGRP

Only try to enable CK_CLYGRP if we're running on kernel newer than
1200046, the first version that supports this feature. If we set it,
then use an old kernel, we'll break the 'contract' of having
checksummed cylinder groups this flag signifies. To avoid creating
something with an inconsistent state, don't turn the flag on in these
cases. The first full fsck with a new kernel will turn this on.

Spnsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D13114

Continuing efforts to provide hardening of FFS, this change adds a
check hash to cylinder groups. If a check hash fails when a cylinder
group is read, no further allocations are attempted in that cylinder
group until it has been fixed by fsck. This avoids a class of
filesystem panics related to corrupted cylinder group maps. The
hash is done using crc32c.

Check hases are added only to UFS2 and not to UFS1 as UFS1 is primarily
used in embedded systems with small memories and low-powered processors
which need as light-weight a filesystem as possible.

Specifics of the changes:

sys/sys/buf.h:
Add BX_FSPRIV to reserve a set of eight b_xflags that may be used
by individual filesystems for their own purpose. Their specific
definitions are found in the header files for each filesystem
that uses them. Also add fields to struct buf as noted below.

sys/kern/vfs_bio.c:
It is only necessary to compute a check hash for a cylinder
group when it is actually read from disk. When calling bread,
you do not know whether the buffer was found in the cache or
read. So a new flag (GB_CKHASH) and a pointer to a function to
perform the hash has been added to breadn_flags to say that the
function should be called to calculate a hash if the data has
been read. The check hash is placed in b_ckhash and the B_CKHASH
flag is set to indicate that a read was done and a check hash
calculated. Though a rather elaborate mechanism, it should
also work for check hashing other metadata in the future. A
kernel internal API change was to change breada into a static
fucntion and add flags and a function pointer to a check-hash
function.

sys/ufs/ffs/fs.h:
Add flags for types of check hashes; stored in a new word in the
superblock. Define corresponding BX_ flags for the different types
of check hashes. Add a check hash word in the cylinder group.

sys/ufs/ffs/ffs_alloc.c:
In ffs_getcg do the dance with breadn_flags to get a check hash and
if one is provided, check it.

sys/ufs/ffs/ffs_vfsops.c:
Copy across the BX_FFSTYPES flags in background writes.
Update the check hash when writing out buffers that need them.

sys/ufs/ffs/ffs_snapshot.c:
Recompute check hash when updating snapshot cylinder groups.

sys/libkern/crc32.c:
lib/libufs/Makefile:
lib/libufs/libufs.h:
lib/libufs/cgroup.c:
Include libkern/crc32.c in libufs and use it to compute check
hashes when updating cylinder groups.

Four utilities are affected:

sbin/newfs/mkfs.c:
Add the check hashes when building the cylinder groups.

sbin/fsck_ffs/fsck.h:
sbin/fsck_ffs/fsutil.c:
Verify and update check hashes when checking and writing cylinder groups.

sbin/fsck_ffs/pass5.c:
Offer to add check hashes to existing filesystems.
Precompute check hashes when rebuilding cylinder group
(although this will be done when it is written in fsutil.c
it is necessary to do it early before comparing with the old
cylinder group)

sbin/dumpfs/dumpfs.c
Print out the new check hash flag(s)

sbin/fsdb/Makefile:
Needs to add libufs now used by pass5.c imported from fsck_ffs.

Reviewed by: kib
Tested by: Peter Holm (pho)

The new fsck recovery information to enable it to find backup
superblocks created in revision 322297 only works on disks
with sector sizes up to 4K. This update allows the recovery
information to be created by newfs and used by fsck on disks
with sector sizes up to 64K. Note that FFS currently limits
filesystem to be mounted from disks with up to 8K sectors.
Expanding this limitation will be the subject of another
commit.

Reported by: Peter Holm
Reviewed with: kib

Since the switch to GPT disk labels, fsck for UFS/FFS has been
unable to automatically find alternate superblocks. This checkin
places the information needed to find alternate superblocks to the
end of the area reserved for the boot block.

Filesystems created with a newfs of this vintage or later will
create the recovery information. If you have a filesystem created
prior to this change and wish to have a recovery block created for
your filesystem, you can do so by running fsck in forground mode
(i.e., do not use the -p or -y options). As it starts, fsck will
ask ``SAVE DATA TO FIND ALTERNATE SUPERBLOCKS'' to which you should
answer yes.

Discussed with: kib, imp
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D11589

Renumber copyright clause 4

Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is too pedantic, so give up on that point.

Submitted by: Jan Schaumann <jschauma@stevens.edu>
Pull Request: https://github.com/freebsd/freebsd/pull/96

prefix UFS symbols with UFS_ to reduce namespace pollution

Specifically:
ROOTINO -> UFS_ROOTINO
WINO -> UFS_WINO
NXADDR -> UFS_NXADDR
NDADDR -> UFS_NDADDR
NIADDR -> UFS_NIADDR
MAXSYMLINKLEN_UFS[12] -> UFS[12]_MAXSYMLINKLEN (for consistency)

Also prefix ext2's and nandfs's NDADDR and NIADDR with EXT2_ and NANDFS_

Reviewed by: kib, mckusick
Obtained from: NetBSD
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D9536

Use MIN()/MAX() macros from sys/param.h.

Reviewed by: trasz
MFC after: 2 weeks.
Differential Revision: https://reviews.freebsd.org/D6118

Refer newfs and growfs users to fsck_ffs instead of
fsck, the latter does not accept the referred to "-b" flag.

This change was accidently committed directly to 9-STABLE in
r237505.

PR: 82720
Submitted by: David D.W. Downey
MFC after: 1 week

Don't call arc4random_stir() explicitly. To quote arc4random(3)
manual page:

There is no need to call arc4random_stir() before using
arc4random() functions family, since they automatically
initialize themselves.

No objection: des
MFC after: 2 weeks

The purpose of this change to the FFS layout policy is to reduce the
running time for a full fsck. It also reduces the random access time
for large files and speeds the traversal time for directory tree walks.

The key idea is to reserve a small area in each cylinder group
immediately following the inode blocks for the use of metadata,
specifically indirect blocks and directory contents. The new policy
is to preferentially place metadata in the metadata area and
everything else in the blocks that follow the metadata area.

The size of this area can be set when creating a filesystem using
newfs(8) or changed in an existing filesystem using tunefs(8).
Both utilities use the `-k held-for-metadata-blocks' option to
specify the amount of space to be held for metadata blocks in each
cylinder group. By default, newfs(8) sets this area to half of
minfree (typically 4% of the data area).

This work was inspired by a paper presented at Usenix's FAST '13:
www.usenix.org/conference/fast13/ffsck-fast-file-system-checker

Details of this implementation appears in the April 2013 of ;login:
www.usenix.org/publications/login/april-2013-volume-38-number-2.
A copy of the April 2013 ;login: paper can also be downloaded
from: www.mckusick.com/publications/faster_fsck.pdf.

Reviewed by: kib
Tested by: Peter Holm
MFC after: 4 weeks

Fix problem with geom_label(4) not recognizing UFS labels on filesystems
extended using growfs(8). The problem here is that geom_label checks if
the filesystem size recorded in UFS superblock is equal to the provider
(i.e. device) size. This check cannot be removed due to backward
compatibility. On the other hand, in most cases growfs(8) cannot set
fs_size in the superblock to match the provider size, because, differently
from newfs(8), it cannot recompute cylinder group sizes.

To fix this problem, add another superblock field, fs_providersize, used
only for this purpose. The geom_label(4) will attach if either fs_size
(filesystem created with newfs(8)) or fs_providersize (filesystem expanded
using growfs(8)) matches the device size.

PR: kern/165962
Reviewed by: mckusick
Sponsored by: FreeBSD Foundation

Fix sbin/ build with a 64-bit ino_t.

Original code by: Gleb Kurtsou

Fix warning when compiling with gcc46:
error: variable 'c' set but not used

Approved by: dim
MFC after: 3 days

Add missing static keywords for global variables to tools in sbin/.

These tools declare global variables without using the static keyword,
even though their use is limited to a single C-file, or without placing
an extern declaration of them in the proper header file.

Stop trying to zero UFS1 superblocks if we fall off the end of the disk.

This avoids a potentially many-hours-long loop of failed writes if newfs
finds a partially-overwritten superblock (or, for that matter, random
garbage which happens to have superblock magic bytes); on one occasion I
found newfs trying to zero 800 million superblocks on a 50 MB disk.

Reviewed by: mckusick
MFC after: 1 week

The dump, fsck_ffs, fsdb, fsirand, newfs, makefs, and quot utilities
include sys/time.h instead of time.h. This include is incorrect as
per the manpages for the APIs and the POSIX definitions. This commit
replaces sys/time.h where necessary with time.h.

The commit also includes some minor style(9) header fixup in newfs.

This commit is part of a larger effort by Garrett Cooper started in
//depot/user/gcooper/posix-conformance-work/ -- to make FreeBSD more
POSIX compliant.

Submitted by: Garrett Cooper yanegomi at gmail dot com

Add support for FS_TRIM to user-mode UFS utilities.

Reviewed by: mckusick, pjd, pho
Tested by: pho
MFC after: 1 month

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Reported problem:
Large (60GB) filesystems created using "newfs -U -O 1 -b 65536 -f 8192"
show incorrect results from "df" for free and used space when mounted
immediately after creation. fsck on the new filesystem (before ever
mounting it once) gives a "SUMMARY INFORMATION BAD" error in phase 5.

This error hasn't occurred in any runs of fsck immediately after
"newfs -U -b 65536 -f 8192" (leaving out the "-O 1" option).

Solution:
The default UFS1 superblock is located at offset 8K in the filesystem
partition; the default UFS2 superblock is located at offset 64K in
the filesystem partition. For UFS1 filesystems with a blocksize of
64K, the first alternate superblock resides at 64K which is the the
location used for the default UFS2 superblock. By default, the
system first checks for a valid superblock at the default location
for a UFS2 filoesystem. For a UFS1 filesystem with a blocksize of
64K, there is a valid UFS1 superblock at this location. Thus, even
though it is expected to be a backup superblock, the system will
use it as its default superblock. So, we have to ensure that all the
statistcs on usage are correct in this first alternate superblock
as it is the superblock that will actually be used.

While tracking down this problem, another limitation of UFS1 became
evident. For UFS1, the number of inodes per cylinder group is stored
in an int16_t. Thus the maximum number of inodes per cylinder group
is limited to 2^15 - 1. This limit can easily be exceeded for block
sizes of 32K and above. Thus when building UFS1 filesystems, newfs
must limit the number of inodes per cylinder group to 2^15 - 1.

Reported by: Guy Helmer<ghelmer@palisadesys.com>
Followup by: Bruce Cran <brucec@freebsd.org>
PR: 107692
MFC after: 4 weeks

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

o bdeficize expand_number_int() function;

o revert most of the recent changes (int -> int64_t conversion) by using
this functon for parsing all options.

Cast these to intmax_t before printing to fix build bustage. Better
solutions welcome.

MFC of 203763, 203764, 203768, 203769, 203770, 203782, and 203784.

These fixes correct a problem in the file system that treats large
inode numbers as negative rather than unsigned. For a default
(16K block) file system, this bug began to show up at a file system
size above about 16Tb.

These fixes also update newfs to ensure that it will never create a
filesystem with more than 2^32 inodes.

They also update libufs, tunefs, and growfs so that they properly
handle inode numbers as unsigned.

Reported by: Scott Burns, John Kilburg, and Bruce Evans
Followup by: Jeff Roberson
PR: 133980

One last pass to get all the unsigned comparisons correct.

Ensure that newfs will never create a filesystem with more than 2^32
inodes by cutting back on the number of inodes per cylinder group if
necessary to stay under the limit. For a default (16K block) file
system, this limit begins to take effect for file systems above 32Tb.

This fix is in addition to -r203763 which corrected a problem in the
kernel that treated large inode numbers as negative rather than unsigned.
For a default (16K block) file system, this bug began to show up at a
file system size above about 16Tb.

Reported by: Scott Burns, John Kilburg, Bruce Evans
Followup by: Jeff Roberson
PR: 133980
MFC after: 2 weeks

Fix typo: s/partion/partition/

Submitted by: Marc Balmer <marc@msys.ch>
MFC after: 3 days

Don't add a bwrite() symbol, it breaks the build when building newfs
statically.
Instead, bring in a stripped down version of sbwrite(), and add the offset
to every bwrite() calls.

Enable operation of newfs on plain files, which is useful when you
want to prepare disk images for emulators (though 'makefs' in port
can do something similar).

This relies on:
+ minor changes to pass the consistency checks even when working on a file;

+ an additional option, '-p partition' , to specify the disk partition to
initialize;

+ some changes on the I/O routines to deal with partition offsets.

The latter was a bit tricky to implement, see the details in newfs.h:
in newfs, I/O is done through libufs which assumes that the file
descriptor refers to the whole partition. Introducing support for
the offset in libufs would require a non-backward compatible change
in the library, to be dealt with a version bump or with symbol
versioning.

I felt both approaches to be overkill for this specific application,
especially because there might be other changes to libufs that might
become necessary in the near future.

So I used the following trick:
- read access is always done by calling bread() directly, so we just add
the offset in the (few) places that call bread();
- write access is done through bwrite() and sbwrite(), which in turn
calls bwrite(). To avoid rewriting sbwrite(), we supply our own version
of bwrite() here, which takes precedence over the version in libufs.

MFC after: 4 weeks

- Import the HEAD csup code which is the basis for the cvsmode work.

Use calloc().

Report erase interval (correctly) in sectors.

Rename the undocumented -E option to -X.

Implement -E option which will erase the filesystem sectors before
making the new filesystem. Reserved space in front of the superblock
(bootcode) is not erased.

NB: Erasing can take as long time as writing every sector sequentially.

This is relevant for all flash based disks which use wearlevelling.

Add -J flag to both newfs(8) and tunefs(8) which allows to enable gjournal
support.
I left -j flag for UFS journal implementation which we may gain at some
point.

Sponsored by: home.pl

Explicitly say which gid do we use as a fallback, when operator
is not found.

Suggested by: kensmith

Don't treat failure to find the operator GID as a fatal error; this
made it impossible to use newfs (and mdmfs) when /etc/group is
missing and /etc is read-only.

When creating a new FFS file system, the block size will indirectly
affect the largest file size that is allowed by the file system.
On the other hand, when creating a snapshot, the snapshot file will
appear as it is as big as the file system itself. Hence we will not
be able to create a file system on large file systems with small
block sizes.

Add a warning about this, and gives some hints to correct the issue.

Reviewed by: mckusick
MFC After: 1 week

Add an option to suppress the creation of the .snap directory in
the new filesystem. This is intended for memory and vnode filesystems
that will never be fsck'ed or dumped.

Obtained from: St. Bernard Software RAPID
MFC after: 2 weeks

Generalize the UFS bad magic value used to determine when a filesystem
has only been partly initialized via newfs(8) so that it applies to both
UFS1 and UFS2.

Submitted by: "Xin LI" delphij at frontfree dot net
MFC: maybe?

Remove advertising clause from University of California Regent's license,
per letter dated July 22, 1999.

Approved by: core, imp

Add a "-l" flag to newfs, which sets the FS_MULTILABEL flag. This
permits users of newfs to set the multilabel flag on UFS1 and UFS2
file systems from inception without using tunefs.

Obtained from: TrustedBSD Project
Sponsored by: DARPA, McAfee Research

Fix whitespace error in previous commit.

Approved by: RE@ (Robert Watson)

Don't use UFS2_BAD_MAGIC on UFS (v1) filesystems; it is Not Ready
for Prime Time there.

Submitted by: Xin LI <delphij@frontfree.net>
Approved by: RE@ (John, Scott)

Add the -E command line option to force error conditions for testing.

Sponsord by: St. Bernard Software

Write the UFS2 superblock with a 'BAD' magic number at the beginning
of newfs, to signify the newfs operation has not yet completed. Re-
write the superblock with the correct magic number once all of the
cylinder groups have been created to show the operation has finished.

Sponsored by: St. Bernard Software

Create a .snap directory mode 770 group operator in the root of
a new filesystem. Dump and fsck will create snapshots in this
directory rather than in the root for two reasons:

1) For terabyte-sized filesystems, the snapshot may require many
minutes to build. Although the filesystem will not be suspended
during most of the snapshot build, the snapshot file itself is
locked during the entire snapshot build period. Thus, if it is
accessed during the period that it is being built, the process
trying to access it will block holding its containing directory
locked. If the snapshot is in the root, the root will lock and
the system will come to a halt until the snapshot finishes. By
putting the snapshot in a subdirectory, it is out of the likely
path of any process traversing through the root and hence much
less likely to cause a lock race to the root.

2) The dump program is usually run by a non-root user running with
operator group privilege. Such a user is typically not permitted
to create files in the root of a filesystem. By having a directory
in group operator with group write access available, such a user
will be able to create a snapshot there. Having the dump program
create its snapshot in a subdirectory below the root will benefit
from point (1) as well.

Sponsored by: DARPA & NAI Labs.

Exit with a non-zero status upon a block allocation failure.
The old way of just returning could result in a file system
extremely likely to panic the kernel. The warning printed
wouldn't help much since tools invoking newfs(8), e.g., mdmfs(8),
couldn't detect the error.

PR: bin/55078
MFC after: 1 week

When newfs'ing a partition with UFS2 that had previously been newfs'ed
with UFS1, the UFS1 superblocks were not deleted. This allowed any
RELENG_4 (or other non-UFS2-aware) fsck to think it knew how to "fix"
the file system, resulting in severe data scrambling.

This patch is a more advanced version than the one originally submitted.
Lukas improved it based on feedback from Kirk, and testing by me. It
blanks all UFS1 superblocks (if any) during a UFS2 newfs, thereby causing
fsck's that are not UFS2 aware to generate the "SEARCH FOR ALTERNATE
SUPER-BLOCK FAILED" message, and exit without damaging the fs.

PR: bin/51619
Submitted by: Lukas Ertl <l.ertl@univie.ac.at>
Reviewed by: kirk
Approved by: re (scottl)

Put back the error checking in wtfs() that was lost when newfs was
changed to use libufs in revision 1.71. Without this, any write
failures in newfs were silently ignored.

Note that this will display a meaningless errno string in the case
of a short write as opposed to a write error, since bwrite()'s
return value does not allow the caller to determine if errno is
valid.

Reported by: Lukas Ertl <l.ertl@univie.ac.at>
Reviewed by: jmallett
Approved by: re (bmah)

Use __FBSDID() to quiet GCC 3.3 warnings.

Fix the -R flag so that it provides sequential "random" numbers
so that the regression test will succeed.

Sponsored by: DARPA & NAI Labs.

Replace use of random() with arc4random() to provide less guessable
values for the initial inode generation numbers in newfs and for
newly allocated inode generation numbers in the kernel.

Submitted by: Theo de Raadt <deraadt@cvs.openbsd.org>
Sponsored by: DARPA & NAI Labs.

Correct lines incorrectly added to the copyright message. Add missing period.

Submitted by: Bruce Evans <bde@zeta.org.au>
Sponsored by: DARPA & NAI Labs.

Convert newfs to libufs (really). Solves one real issue with previous
version of such. Differences in filesystems generated were found to be
from 1) sbwrite with the "all" parameter 2) removal of writecache. The
sbwrite call was made to perform as the original version, and otherwise
this was checked against a version of newfs with the write cache removed.

Bring in support for volume labels to the filesystem utilities.

Reviewed by: mckusick

Back out conversion to libufs, for now. It seems to cause problems.

Reported by: phk

Convert newfs to use libufs. I've tested this on md filesystems, as has
keramida, and all seems well.

Correctly calculate the initial number of fragments in a filesystem
so that fsck does not complain with `SUMMARY BLK COUNT(S) WRONG IN
SUPERBLK' the first time it is run on a new filesystem.

Reported by: Poul-Henning Kamp <phk@freebsd.org>
Sponsored by: DARPA & NAI Labs.

Add some more checks to newfs so that it will not build filesystems
that the kernel will refuse to mount. Specifically it now enforces
the MAXBSIZE blocksize limit. This update also fixes a problem where
newfs could segment fault if the selected fragment size was too large.

PR: bin/30959
Submitted by: Ceri Davies <setantae@submonkey.net>
Sponsored by: DARPA & NAI Labs.

Create a new 32-bit fs_flags word in the superblock. Add code to move
the old 8-bit fs_old_flags to the new location the first time that the
filesystem is mounted by a new kernel. One of the unused flags in
fs_old_flags is used to indicate that the flags have been moved.
Leave the fs_old_flags word intact so that it will work properly if
used on an old kernel.

Change the fs_sblockloc superblock location field to be in units
of bytes instead of in units of filesystem fragments. The old units
did not work properly when the fragment size exceeeded the superblock
size (8192). Update old fs_sblockloc values at the same time that
the flags are moved.

Suggested by: BOUWSMA Barry <freebsd-misuser@netscum.dyndns.dk>
Sponsored by: DARPA & NAI Labs.

Properly calculate the initial number of fragments in a large filesystem.

Sponsored by: DARPA & NAI Labs.

Bound the size of the superblock to SBLOCKSIZE.

Submitted by: BOUWSMA Beery <freebsd-misuser@netscum.dyndns.dk>
Sponsored by: DARPA & NAI Labs.

Use the standardized CHAR_BIT constant instead of NBBY in userland.

s/filesystem/file system/g as discussed on -developers

di_createtime -> di_birthtime.

Submitted by: Udo Schweigert <Udo.Schweigert@siemens.com>

Fixed 4 printf format errors that were fatal on alphas. %qd is not even
suitable for printing quad_t's since it is equivalent to %lld but quad_t
is unsigned long on alphas. quad_t shouldn't be used anyway.

Get rid of paranoia that zeros the boot block area as this has
bad effect on existing bootstraps.

Submitted by: Jake Burkholder <jake@locore.ca>
Sponsored by: DARPA & NAI Labs.

This commit adds basic support for the UFS2 filesystem. The UFS2
filesystem expands the inode to 256 bytes to make space for 64-bit
block pointers. It also adds a file-creation time field, an ability
to use jumbo blocks per inode to allow extent like pointer density,
and space for extended attributes (up to twice the filesystem block
size worth of attributes, e.g., on a 16K filesystem, there is space
for 32K of attributes). UFS2 fully supports and runs existing UFS1
filesystems. New filesystems built using newfs can be built in either
UFS1 or UFS2 format using the -O option. In this commit UFS1 is
the default format, so if you want to build UFS2 format filesystems,
you must specify -O 2. This default will be changed to UFS2 when
UFS2 proves itself to be stable. In this commit the boot code for
reading UFS2 filesystems is not compiled (see /sys/boot/common/ufsread.c)
as there is insufficient space in the boot block. Once the size of the
boot block is increased, this code can be defined.

Things to note: the definition of SBSIZE has changed to SBLOCKSIZE.
The header file <ufs/ufs/dinode.h> must be included before
<ufs/ffs/fs.h> so as to get the definitions of ufs2_daddr_t and
ufs_lbn_t.

Still TODO:
Verify that the first level bootstraps work for all the architectures.
Convert the utility ffsinfo to understand UFS2 and test growfs.
Add support for the extended attribute storage. Update soft updates
to ensure integrity of extended attribute storage. Switch the
current extended attribute interfaces to use the extended attribute
storage. Add the extent like functionality (framework is there,
but is currently never used).

Sponsored by: DARPA & NAI Labs.
Reviewed by: Poul-Henning Kamp <phk@freebsd.org>

more file system > filesystem

Continue the cleanup preparations for UFS2 (& GEOM):

Use only one filedescriptor. Open in R/O or R/W based in the '-N' option.
Make the filedescriptor a global variable instead of passing it around
as semi-global variable(s).

Remove the undocumented ability to specify type without '-T' option.

Replace fatal() with straight err(3)/errx(3). Save calls to strerror()
where applicable. Loose the progname variable.

Get the sense of the cpgflag test correct so we only issue warnings if
people specify cpg and can't get that. It can be argued that this
should be an error.

Remove the check to see if the disk is mounted: Open for writing
would fail if it were mounted.

Attempt to get the sectorsize and mediasize with the generic disk
ioctls, fall back to disklabel and /etc/disktab as we can.

Notice that on-disk labels still take precedence over /etc/disktab,
this is probably wrong, but not as wrong as the entire concept of
/etc/disktab is.

Sponsored by: DARPA & NAI Labs.

bbsize and sbsize cannot ever be trusted from the disklabel, in
particular as there may not be one. Remove #if 0'ed code which might
mislead people to think otherwise.

unifdef -ULOSTDIR, fsck can make lost+found on the fly.

Sponsored by: DARPA & NAI Labs

Fixed some English errors in previous commit.

Fixed some style bugs in the removal of __P(()). Whitespace before
"__P((" was not removed.

Add more DWIM/autoadjustment and less evil style(9) banned exit(2) codes.
Add some missing statics.

Sponsored by: DARPA & NAI Labs.

Swing the axe and remove some archaic features from newfs which modern
diskdrives do neither need nor want:

-O create a 4.3BSD format filesystem
-d rotational delay between contiguous blocks
-k sector 0 skew, per track
-l hardware sector interleave
-n number of distinguished rotational positions
-p spare sectors per track
-r revolutions/minute
-t tracks/cylinder
-x spare sectors per cylinder

No change in the produced filesystem image unless one or more of
these options were used.

Approved by: mckusick

Add the undocumented -R option to disable randomness for regression-testing.

Add a couple of simple regression tests accessible with "make test", they
depend on the md(4) driver.

FYI I have also tried running the test against a week old newfs and it
passed.

Further cleanups.

Replace a number of similar `for' loops with a new `ilog2()' function
that computes the base-2 log of a power of 2.

Complete the ANSIfication of newfs by converting function declarations
to C89 style.

The FSIRAND code is always compiled in, and it is unlikely that
anyone needs a newfs without it. Remove the #ifdef's from around
the code and the -DFSIRAND from the Makefile. Also remove redundant
declarations of random() and srandomdev().

Remove the ancient STANDALONE code.

Approved by: phk

Remove yet more vestiges of mount_mfs.

Fixed some style bugs (mainly ones not fixed or made worse by rev.1.41).
Old code obfuscates long (but single-line) messages by printing them in
pieces using %s. Rev.1.41 obfuscated some new long messages using ISO
string concatenation. This commit only fixes the new obfuscations.

Remove __P() and register.
Set WARNS=2

This is the beginning of a pre-UFS2 cleanup of newfs.

Sponsored by: DARPA, NAI Labs

style(9) cleanup.

Submitted by: j mckitrick <jcm@freebsd-uk.eu.org>
Reviewed by: phk, /sbin/md5

Handle snprintf() returning < 0 (not just -1)

MFC after: 2 weeks

Handle snprintf() returning -1.

MFC after: 2 weeks

A more complete removal of MFS related code.

XXX: This program badly needs a style(9) + BDECFLAGS treatment.

sprintf() -> snprintf()

Partially submitted by: "Andrew R. Reiter" <arr@watson.org>
Obtained from: OpenBSD

Directory layout preference improvements from Grigoriy Orlov <gluk@ptci.ru>.
His description of the problem and solution follow. My own tests show
speedups on typical filesystem intensive workloads of 5% to 12% which
is very impressive considering the small amount of code change involved.

------

One day I noticed that some file operations run much faster on
small file systems then on big ones. I've looked at the ffs
algorithms, thought about them, and redesigned the dirpref algorithm.

First I want to describe the results of my tests. These results are old
and I have improved the algorithm after these tests were done. Nevertheless
they show how big the perfomance speedup may be. I have done two file/directory
intensive tests on a two OpenBSD systems with old and new dirpref algorithm.
The first test is "tar -xzf ports.tar.gz", the second is "rm -rf ports".
The ports.tar.gz file is the ports collection from the OpenBSD 2.8 release.
It contains 6596 directories and 13868 files. The test systems are:

1. Celeron-450, 128Mb, two IDE drives, the system at wd0, file system for
test is at wd1. Size of test file system is 8 Gb, number of cg=991,
size of cg is 8m, block size = 8k, fragment size = 1k OpenBSD-current
from Dec 2000 with BUFCACHEPERCENT=35

2. PIII-600, 128Mb, two IBM DTLA-307045 IDE drives at i815e, the system
at wd0, file system for test is at wd1. Size of test file system is 40 Gb,
number of cg=5324, size of cg is 8m, block size = 8k, fragment size = 1k
OpenBSD-current from Dec 2000 with BUFCACHEPERCENT=50

You can get more info about the test systems and methods at:
http://www.ptci.ru/gluk/dirpref/old/dirpref.html

Test Results

tar -xzf ports.tar.gz rm -rf ports
mode old dirpref new dirpref speedup old dirprefnew dirpref speedup
First system
normal 667 472 1.41 477 331 1.44
async 285 144 1.98 130 14 9.29
sync 768 616 1.25 477 334 1.43
softdep 413 252 1.64 241 38 6.34
Second system
normal 329 81 4.06 263.5 93.5 2.81
async 302 25.7 11.75 112 2.26 49.56
sync 281 57.0 4.93 263 90.5 2.9
softdep 341 40.6 8.4 284 4.76 59.66

"old dirpref" and "new dirpref" columns give a test time in seconds.
speedup - speed increasement in times, ie. old dirpref / new dirpref.

------

Algorithm description

The old dirpref algorithm is described in comments:

/*
* Find a cylinder to place a directory.
*
* The policy implemented by this algorithm is to select from
* among those cylinder groups with above the average number of
* free inodes, the one with the smallest number of directories.
*/

A new directory is allocated in a different cylinder groups than its
parent directory resulting in a directory tree that is spreaded across
all the cylinder groups. This spreading out results in a non-optimal
access to the directories and files. When we have a small filesystem
it is not a problem but when the filesystem is big then perfomance
degradation becomes very apparent.

What I mean by a big file system ?

1. A big filesystem is a filesystem which occupy 20-30 or more percent
of total drive space, i.e. first and last cylinder are physically
located relatively far from each other.
2. It has a relatively large number of cylinder groups, for example
more cylinder groups than 50% of the buffers in the buffer cache.

The first results in long access times, while the second results in
many buffers being used by metadata operations. Such operations use
cylinder group blocks and on-disk inode blocks. The cylinder group
block (fs->fs_cblkno) contains struct cg, inode and block bit maps.
It is 2k in size for the default filesystem parameters. If new and
parent directories are located in different cylinder groups then the
system performs more input/output operations and uses more buffers.
On filesystems with many cylinder groups, lots of cache buffers are
used for metadata operations.

My solution for this problem is very simple. I allocate many directories
in one cylinder group. I also do some things, so that the new allocation
method does not cause excessive fragmentation and all directory inodes
will not be located at a location far from its file's inodes and data.
The algorithm is:
/*
* Find a cylinder group to place a directory.
*
* The policy implemented by this algorithm is to allocate a
* directory inode in the same cylinder group as its parent
* directory, but also to reserve space for its files inodes
* and data. Restrict the number of directories which may be
* allocated one after another in the same cylinder group
* without intervening allocation of files.
*
* If we allocate a first level directory then force allocation
* in another cylinder group.
*/

My early versions of dirpref give me a good results for a wide range of
file operations and different filesystem capacities except one case:
those applications that create their entire directory structure first
and only later fill this structure with files.

My solution for such and similar cases is to limit a number of
directories which may be created one after another in the same cylinder
group without intervening file creations. For this purpose, I allocate
an array of counters at mount time. This array is linked to the superblock
fs->fs_contigdirs[cg]. Each time a directory is created the counter
increases and each time a file is created the counter decreases. A 60Gb
filesystem with 8mb/cg requires 10kb of memory for the counters array.

The maxcontigdirs is a maximum number of directories which may be created
without an intervening file creation. I found in my tests that the best
performance occurs when I restrict the number of directories in one cylinder
group such that all its files may be located in the same cylinder group.
There may be some deterioration in performance if all the file inodes
are in the same cylinder group as its containing directory, but their
data partially resides in a different cylinder group. The maxcontigdirs
value is calculated to try to prevent this condition. Since there is
no way to know how many files and directories will be allocated later
I added two optimization parameters in superblock/tunefs. They are:

int32_t fs_avgfilesize; /* expected average file size */
int32_t fs_avgfpdir; /* expected # of files per directory */

These parameters have reasonable defaults but may be tweeked for special
uses of a filesystem. They are only necessary in rare cases like better
tuning a filesystem being used to store a squid cache.

I have been using this algorithm for about 3 months. I have done
a lot of testing on filesystems with different capacities, average
filesize, average number of files per directory, and so on. I think
this algorithm has no negative impact on filesystem perfomance. It
works better than the default one in all cases. The new dirpref
will greatly improve untarring/removing/coping of big directories,
decrease load on cvs servers and much more. The new dirpref doesn't
speedup a compilation process, but also doesn't slow it down.

Obtained from: Grigoriy Orlov <gluk@ptci.ru>

Fixed style bugs in previous commit.

Allow enabling soft updates (with -U) on a new filesystem.

[I first added this functionality, and thought to check prior art. Seeing
OpenBSD had already done this, I changed my addition to reduce the diffs
between the two and went with their option letter.]
Obtained from: OpenBSD

The ffs superblock includes a 128-byte region for use by temporary
in-core pointers to summary information. An array in this region
(fs_csp) could overflow on filesystems with a very large number of
cylinder groups (~16000 on i386 with 8k blocks). When this happens,
other fields in the superblock get corrupted, and fsck refuses to
check the filesystem.

Solve this problem by replacing the fs_csp array in 'struct fs'
with a single pointer, and add padding to keep the length of the
128-byte region fixed. Update the kernel and userland utilities
to use just this single pointer.

With this change, the kernel no longer makes use of the superblock
fields 'fs_csshift' and 'fs_csmask'. Add a comment to newfs/mkfs.c
to indicate that these fields must be calculated for compatibility
with older kernels.

Reviewed by: mckusick

Cast block number to off_t to avoid possible overflow bugs.

Pointed out by: Bruce Evans <bde@zeta.org.au>

The write combining code in revision 1.30 needs a few additional
touch ups. The cache needs to be flushed against block
reads, and a final flush at process termination to force the
backup superblocks to disk.

I believe this will allow 'make release' to complete.

Submitted by: Tor.Egge@fast.no

Implement simple write combining for newfs - this is particularly useful
for large scsi disks with WCE = 0. This yields around a 7 times speedup
on elapsed newfs time on test disks here. 64k clusters seems to be the
sweet spot for scsi disks using our present drivers.

$Id$ -> $FreeBSD$

Don't print a "," after the last superblock.

Submitted by: adrian

Fix bug in mount_mfs whereby mount_mfs would sometimes return before
the mount is completely active, causing the next few commands attempting
to manipulate data on the mount to fail. mount_mfs's parent now tries
to wait for the mount point st_dev to change before returning, indicating
that the mount has gone active.

Use explicitly sized types when formatting cylinder groups.

Forgot to remove a ';' in my previous commit.

Add prototypes. Check malloc() return value. Use err(). Remove unused #includes
Do not \n nor dot terminate syslog()/err() messages. -Wall.

Fixed printf format errors.

Fixed overflow in the calculation of the number of inodes per group
for filesystems with almost the maximum number of sectors. The maxiumum
is 2^31, but overflow is common for that size, and overflow normally
occurred here at size (2^31 - 4096).

Don't create superblocks with size larger than SBSIZE (8192). The
size was rounded up to a multiple of the fragment size, but this
gave invalid file systems when the fragment size was > SBSIZE (fsck
aborts early on them). Now a fragment size of 32768 seems to work
(too-simple tests with fsck and iozone worked).

Some tweaks to get this to cope with ELF where the address space starts
higher up in memory (0x0800000 upwards) rather than near zero (0x1000
for our qmagic a.out format). The method that mount_mfs uses to allocate
the memory within data size rlimits for the ram disk is entirely too much
of a kludge for my liking. I mean, if it's run as root, surely it makes
sense to just raise the resource limits to infinity or something, and if
it's a non-root user mount (do these work? with mfs?) it could just fail
if it's outside limits.

Removed "hack to prevent overflow of a 32bit block number". Lite2 has a
better hack in ffs_vfsops.c. The hack here restricted the maximum file
size to 2^39 bytes (512GB). fs_bsize * 2^31 - 1 (16TB for the default
blocksize of 8K) would have been better. There is no good way to remove
this limit on old BSD4.4 file systems.

Remove srandomdev fallback

Fix the mount_mfs case from the last cleanup. The code was (ab)using
it's internal malloc() implementation to try and avoid overstepping it's
resource limits (yuk!). Remain using libc's malloc(), but check the
resource limits right before trying to malloc the ramdisk space and leave
some spare memory for libc. In Andrey's words, the internal malloc
was "true evil".. Among it's sins is it's ability to allocate less memory
than asked for and still return success. stdio would just love that. :-)

Reviewed by: ache

Cleanup STANDALONE stuff
Not replace malloc() family for non-standalone variant
Pay attention on malloc() family return code now
Use srandomdev() now for RNG initialization

Add generation number randomization. Newly created filesystems wil now
automatically have random generation numbers. The kenel way of handling those
also changed. Further it is advised to run fsirand on all your nfs exported
filesystems. the code is mostly copied from OpenBSD, with the randomization
chanegd to use /dev/urandom
Reviewed by: Garrett
Obtained from: OpenBSD

Merge from Lite2:
- use new getvfsbyname() and mount(2) interface (mount_mfs)
- use new fs include files
- updated inode / cg layout calculations (?)

Sweep through the tree fixing mmap() usage:

- Use MAP_FAILED instead of the constant -1 to indicate
failure (required by POSIX).
- Removed flag arguments of '0' (required by POSIX).
- Fixed code which expected an error return of 0.
- Fixed code which thought any address with the high bit set
was an error.
- Check for failure where no checks were present.

Discussed with: bde

This update adds the support for != 512 byte sector SCSI devices to
the sd & od drivers. There is also slight changes to fdisk & newfs
in order to comply with different sectorsizes.
Currently sectors of size 512, 1024 & 2048 are supported, the only
restriction beeing in fdisk, which hunts for the sectorsize of
the device.
This is based on patches to od.c and the other system files by
John Gumb & Barry Scott, minor changes and the sd.c patches by
me.
There also exist some patches for the msdos filesys code, but I
havn't been able to test those (yet).

John Gumb (john@talisker.demon.co.uk)
Barry Scott (barry@scottb.demon.co.uk)

Second try: attempt to import Lite2's newfs.

ts_sec -> tv_sec
ts_nsec -> tv_nsec

A better algorithm to place the numbers on the lines.

Submitted by: satoshi

Make the numbers for the "superblock backups" fit nicely on the screen,
even for larger partitions. Until now, partition sizes > 500 MB messed
up the screen.

Remove trailing whitespace.

Flush stdout when writing out each superblock backup.

Restrict fs_maxfilesize to 2^40; this is part of a bug fix from Kirk
McKusick to work around problems in FFS related to the blkno of a 64bit
offset not fitting into an int.

Submitted by: Marshall Kirk McKusick

Added '-F file' option of mount_mfs. This allows me to make floppy images
without waiting for my floppy-drive all the time :-) Might have other
interesting uses too.

Backed out part of the last change that prevents the rpos table from
being output if <= 1 rpos; there is a bug in the kernel which doesn't
quite get along with this. Changed default #rpos to 1, and fixed up
manual page. Converted nrpos to 1 if user specifies 0.

1) If nrpos <= 1, don't output rpos table (and set fs_cpc to 0) - disabling
the use of the rotational position table.
2) Allow specification of 0 rotational positions (disables function).
3) Make rotdelay=0 and nrpos=0 by default.

The purpose of the above is to optimize for modern SCSI (and IDE) drives
that do read-ahead/write-behind.

Set fs_clean.

BSD 4.4 Lite sbin Sources

Note: XNSrouted and routed NOT imported here, they shall be imported with
usr.sbin.