Avoid common symbols for fsck_lfs.

Remove duplicate assignments.
Fixes PR bin/51512 by Jose Luis Rodriguez Garcia

branches: 1.73.4;
Update inode member i_flag --> i_state to keep up with kernel changes

branches: 1.72.6;
PR/51478: Jose Luis Rodriguez Garcia: Fix leak mem fsck_lfs/lfs.c

branches: 1.71.2;
Comment out unused logic. PR 50973

Need <stdbool.h> for true/false.

Enable lfs64 in fsck_lfs.
(and in everything else that uses this code)

Track down and fix two missing uses of SEGSUM_FINFOBASE().

Drop an explicit sign-extension in fsck that shouldn't be needed any
more.

Add lfs_checkword type for reading checksum data out of structures.
This is always uint32_t, but having a name for it both makes things
clearer and avoids confusion about whether it should be 32 or 64 bit.

Note: deployed in only one place (that was erroneously tagged
ondisk32) so far.

Fix hardwired 32-bit stuff in fsck:
- compute the maximum file size using LFS_BLKPTRSIZE()
- use the new IINFO in pass 6 instead of uint32_t pointers
- use accessors to read and write indirect blocks

Apply IINFO in the other userland tools.

Use daddr_t, not ulfs_daddr_t, as the latter's 32 bits wide.
Don't use either for on-disk items.
Declare external data in header files.
Part 3 of 3.

Use daddr_t, not ulfs_daddr_t, as the latter's 32 bits wide.
Don't use either for on-disk items.
Part 2 of 3.

Use daddr_t, not ulfs_daddr_t, as the latter's 32 bits wide.
Don't use either for on-disk items.
Part 1 of 3.

The ifile's inode number is constant. (it is always 1)

Therefore, storing the value in the superblock and reading it out
again is silly and offers the opportunity for it to become corrupted.
So, don't do that (most of the code already didn't) and use the
existing constant instead. Initialize new 32-bit superblocks with
the value for the sake of old userland programs, but don't keep the
value in the 64-bit superblock at all.

(approved by Margo Seltzer)

Add byteswapping to the dinode accessors.

This prevents regressions in the ulfs code when switching to the new
accessors. Note that while adding byteswapping to the other accessors
is straightforward, I haven't done it yet; and that also is not enough
to make LFS_EI work, because there are places lying around that bypass
the accessors for one reason and another and all of them need to be
updated. That is going to have to wait for a later day as LFS_EI is
not on the critical path right now.

Use the lfs dinode accessors in place of the ufs-derived ones.
(Mostly.)

The ufs-derived ones are fake structure member macros, which are gross
and not very safe. Also, it seems that a lot of places in the lfs code
were using the ffsv1 branch of them unconditionally, and this way it's
guaranteed all those places have been updated.

Found while doing this: for non-devices, have getattr produce NODEV
in the rdev field instead of leaking the address of the first direct
block.

Part two of dinodes; use the same union everywhere.
(previously the ufs-derived code had things set up slightly different)

Remove a bunch of associated mess.

Hack up dinode usage to be 64 vs. 32 as needed. Part 1.

(This part changes the native lfs code; the ufs-derived code already
has 64 vs. 32 logic, but as aspects of it are unsafe, and don't
entirely interoperate cleanly with the lfs 64/32 stuff, pass 2 will be
rehashing that.)

Provide 32-bit and 64-bit versions of FINFO.

This also entailed sorting out part of struct segment, as that
contains a pointer into the current FINFO data.

Make 32-bit and 64-bit versions of SEGSUM.
Also fix some of the FINFO handling as it's closely entangled.

Add IFILE32 and IFILE64 structures for the on-disk ifile entries.
Add and use accessors. There are also a bunch of places that cast and
I hope I've found them all...

Add a (draft) 64-bit superblock. Make things build again.

Add pieces of support for using both superblock types where
convenient, and specifically to the superblock accessors, but don't
actually enable it anywhere.

First substantive step on PR 50000.

Use accessor functions for the version field of the lfs superblock.
I thought at first maybe the cases that test the version should be
rolled into the accessors, but on the whole I think the conclusion on
that is no.

Second batch of 64 -> 32 truncations in lfs, along with more minor
tidyups and corrections in passing.

Fix assorted 64 -> 32 truncations in lfs. Also, some minor tidyups and
corrections in passing.

Add a new lfs header file: lfs_accessors.h.

This contains all the accessor functions and macros out of lfs.h.
Add an include of lfs_accessors.h after all uses of lfs.h... except
for code that wants to define its own struct lfs-alike that the
accessors are supposed to play along with. For these, set STRUCT_LFS
and include lfs_accessors.h after the necessary structure has been
defined, so that lfs_accessors.h can emit functions in terms of it.

More lfs superblock accessors.
(This changes the rest of the code over; all the accessors were
already added.)

The difference between this commit and the previous one is arbitrary,
but the previous one passed the regression tests on its own so I'm
keeping it separate to help with any bisections that might be needed
in the future.

Switch to accessor functions for elements of the LFS on-disk
superblock. This will allow switching between 32/64 bit forms on the
fly; it will also allow handling LFS_EI reasonably tidily. (That
currently doesn't work on the superblock.)

It also gets rid of cpp abuse in the form of fake structure member
macros.

Also, instead of doing sleep/wakeup on &lfs_avail and &lfs_nextseg
inside the on-disk superblock, add extra elements to the in-memory
struct lfs for this. (XXX: these should be changed to condvars, but
not right now)

XXX: this migrates a structure needed by the lfs code in libsa (struct
salfs) into lfs.h, where it doesn't belong, but for the time being
this is necessary in order to allow the accessors (and the various
lfs macros and other goop that relies on them) to compile.

Use VFS_PROTOS() for lfs.
Rename conflicting struct lfs field "lfs_start" to "lfs_s0addr".

No functional change.

- Fix fallout for recent bread() change (removing cred arg).

Revert previous; it doesn't work because all the lfs tools
promiscuously .PATH in each other's source files, and I haven't the
time or patience to deal with it tonight.

Remove pointless function indirection through panic_func(). It was
initialized to one thing, and then set to another right at the top of
main and never changed again.

branches: 1.41.2;
fix unused variable warnings.

Prefix most of the cpp macros with lfs_ and LFS_ to avoid conflicts with ffs.
This was done so that boot blocks that want to compile both FFS and LFS in
the same file work.

Tidy up the LFS userland build hacks.
Don't use -I${NETBSDSRCDIR}/sys; don't include files other than the
exported LFS headers, which are lfs.h, lfs_inode.h, and (for now)
lfs_extern.h.

Cleanups and hacks to make lfs userland stuff build:
- lfs_cksum.c doesn't actually need ulfs_inode.h any more.
- neither does lfs_itimes.c.
- add hacks to fsck_lfs to make it compile.
- add hacks to newfs_lfs to make it compile.
- fix warning in ulfs_quota.c when quotas are fully disabled
(as I guess is happening with the rumpity version)

XXX: This commit adds -I${NETBSDSRCDIR}/sys to the Makefiles for
XXX: fsck_lfs, newfs_lfs, and lfs_cleanerd. This needs to be cleaned
XXX: up ASAP; but I consider this less problematic in the short term
XXX: than spewing ulfs_*.h into /usr/include.

ufs -> ulfs for fsck_lfs.

Stuff UFS_ in front of a few of ufs's symbols to reduce namespace
pollution. Specifically:
ROOTINO -> UFS_ROOTINO
WINO -> UFS_WINO
NXADDR -> UFS_NXADDR
NDADDR -> UFS_NDADDR
NIADDR -> UFS_NIADDR
MAXSYMLINKLEN -> UFS_MAXSYMLINKLEN
MAXSYMLINKLEN_UFS[12] -> UFS[12]_MAXSYMLINKLEN (for consistency)

Sort out ext2fs's misuse of NDADDR and NIADDR; fortunately, these have
the same values in ext2fs and ffs.

No functional change intended.

branches: 1.35.2; 1.35.8;
Don't need to initialize (one of) the elements now in i_crap; nothing in
here touches that material.

avoid code that looks like this:
x = x = y;
since GCC 4.5 complains about it having sequence point issues.

branches: 1.33.4;
dev_bsize wasn't initialized. The actual value doesn't matter since
the same value is used to compute byte offsets into the special file
but a value of zero causes a division by zero.

Three changes in a single commit.

- drop the notion of frags (LFS fragments) vs fsb (FFS fragments)
The code uses a complicated unity function that just makes the
code difficult to understand.

- support larger sector sizes. Fix disk address computations
to use DEV_BSIZE in the kernel as required by device drivers
and to use sector sizes in userland.

- Fix several locking bugs in lfs_bio.c and lfs_subr.c.

Define syscalls of lfs userspace tools (cleaner, mainly) through
a struct called kernelops, which contains standard system calls
for the normal case and rump system calls for the rump case.

Make it possible to run the lfs cleaner in a library fashion (taking
the quick route with the implementation).

PR kern/26878 FFSv2 + softdep = livelock (no free ram)
PR kern/16942 panic with softdep and quotas
PR kern/19565 panic: softdep_write_inodeblock: indirect pointer #1 mismatch
PR kern/26274 softdep panic: allocdirect_merge: ...
PR kern/26374 Long delay before non-root users can write to softdep partitions
PR kern/28621 1.6.x "vp != NULL" panic in ffs_softdep.c:4653 while unmounting a softdep (+quota) filesystem
PR kern/29513 FFS+Softdep panic with unfsck-able file-corruption
PR kern/31544 The ffs softdep code appears to fail to write dirty bits to disk
PR kern/31981 stopping scsi disk can cause panic (softdep)
PR kern/32116 kernel panic in softdep (assertion failure)
PR kern/32532 softdep_trackbufs deadlock
PR kern/37191 softdep: locking against myself
PR kern/40474 Kernel panic after remounting raid root with softdep

Retire softdep, pass 2. As discussed and later formally announced on the
mailing lists.

branches: 1.29.4;
Make sure all cached buffers with valid, not yet written data have been
run through copy-on-write. Call fscow_run() with valid data where possible.

The LP_UFSCOW hack is no longer needed to protect ffs_copyonwrite() against
endless recursion.

- Add a flag B_MODIFY to bread(), breada() and breadn(). If set the caller
intends to modify the buffer returned.

- Always run copy-on-write on buffers returned from ffs_balloc().

- Add new function ffs_getblk() that gets a buffer, assigns a new blkno,
may clear the buffer and runs copy-on-write. Process possible errors
from getblk() or fscow_run(). Part of PR kern/38664.

Welcome to 4.99.63

Reviewed by: YAMAMOTO Takashi <yamt@netbsd.org>

branches: 1.28.2;
Remove clause 3 and 4 from TNF licenses

branches: 1.27.8; 1.27.10;
Give brelse() a second argument so that it matches the kernel.
fsck_lfs now compiles again.

branches: 1.26.8;
Fix malloc/realloc/calloc issues: always check and exit, use EEXIT instead
of 8.

Several fixes to improve the reliability of the roll-forward agent.
Also, note "properly orphaned" files as distinct from corrupted files.

Various improvements to fsck_lfs, to wit:

* Add lfs_balloc capability to the lfs library.
* Extend the Ifile if we run out of free inodes when creating lost+found.
* Don't roll forward if we have allocated a lost+found, to avoid
conflicts when adding new files in roll-forward.
* Make some messages slightly more verbose (e.g. include inode number,
and use pwarn() instead of printf() so the messages include the device
name when preening).
* Change superblock detection/avoidance to use the offset table in the
primary superblock, rather than looking at the contents.
* Be more verbose about various operations when passed the -d flag,
especially roll-forward.
* Be more careful about dirops during roll forward, since the cleaner can
sometimes write blocks from dirop vnodes. Detect and avoid this problem.
* Always check the free list, even if given -i; if we're going to write
it we have to check it first.
* Mark inodes dirty when blocks are found during roll forward, so the
inodes are written with the new block locations.
* Update size of inodes if blocks beyond EOF are found during roll
forward.
* Fix segment accounting for blocks and inodes found during roll
forward.
* Report statistics on roll forward: how many new/deleted/moved files
and how many updated blocks (or "nothing new").
* Don't care if the device being checked is really a device, if we have
been passed the -f flag (to facilitate automated testing).
* When writing to the disk, use the current time in the segment headers
rathern than time 0.
* When passed the -i flag, locate the partial segment containing the
Ifile inode and use that to calculate lfs_offset, lfs_curseg,
lfs_nextseg. (Again for automated testing.)

fsck_lfs does not us fsutil.c; perhaps it should? revert to perror for now.

s/perror/perr

Remove the free list ordering/disordering code, since the kernel now keeps
the list in order (ordering it on mount).

Regularize error messages: these are now all in ALL CAPS, with all hex
numbers (not reported in caps) prefixed by 0x. (The non-fsck-specific
messages are an exception to this all-caps rule.)

Check for allocation failures in malloc, calloc, realloc, asprintf, and
vasprintf and try to handle them.

Put back removed initializer. gcc on sh3 still doesn't get it correct...

avoid the need for a bogus initializer.

rename lfs.h to lfs_user.h so that it does not conflict.

Use the correct method to create a new inode, when we allocate lost+found.

Correct uninitialized variable issues in pass6.c and dir.c (PR#30411 and
PR#30394, respectively).

Initialize metalbn in ufs_getlbns to appease -Wuninitialized.
Marked with XXXGCC for dreamcast (found while compiling for it).

Reviewed by lukem.

appease gcc -Wuninitialized

Check some error conditions that would otherwise cause fsck_lfs to dump core.
Pointed out by Pavel Cahyna in a follow-on to PR #29151.

Check parts of pass 5 even if only rolling forward. We can't check the true
segment holdings against the blocks held by the inodes, but we can still
check the cleanerinfo data against the segment table.

When creating an int hash value from pointer, go via intptr_t.
Fixes PR 29953.

Be more efficient with the hash tables for the buffer and vnode caches.

Note that roll-forward can add more inodes to the filesystem; don't overflow
the tables but reallocate them.

"#define lfs_devvp lfs_unlockvp" for readability, since that's what we
use it for in fsck_lfs/newfs_lfs.

branches: 1.8.2;
Various minor LFS improvements:

* Extend the lfs library from fsck_lfs(8) so that it can be used with a
not-yet-existent LFS. Make newfs_lfs(8) use this library, so it can
create LFSs whose Ifile is larger than one segment.
* Make newfs_lfs(8) use strsuftoi64() for its arguments, a la newfs(8).
* Make fsck_lfs(8) respect the "file system is clean" flag.
* Don't let fsck_lfs(8) think it has dirty blocks when invoked with the
-n flag.

branches: 1.7.4;
Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22308, verified by myself.

in lfs_raw_vget(),
- don't leave references into free'ed memory region. (fix SIGBUSes)
- plug a memory leak.

- don't assume that malloc'ed memory is zero-filled.
- LIST_INIT {dirty,clean} block lists.

only read superblock (ie. struct dlfs) from disk and
initialize rest of struct lfs by hand.
(this shouldn't cause a real problem since if superblock is valid,
LFS_SBPAD-sizeof(struct dlfs) bytes after it is always zero-filled, though)

PR/22123 (Izumi Tsutsui)

Fix format string. (no % in PRIx..).

Add support for UFS2. UFS2 is an enhanced FFS, adding support for
64 bit block pointers, extended attribute storage, and a few
other things.

This commit does not yet include the code to manipulate the extended
storage (for e.g. ACLs), this will be done later.

Originally written by Kirk McKusick and Network Associates Laboratories for
FreeBSD.

Add working writing ability to fsck_lfs, including roll-forward, based on
a partial-segment writer ported from the kernel.

Remove duplicate assignments.
Fixes PR bin/51512 by Jose Luis Rodriguez Garcia

branches: 1.73.4;
Update inode member i_flag --> i_state to keep up with kernel changes

branches: 1.72.6;
PR/51478: Jose Luis Rodriguez Garcia: Fix leak mem fsck_lfs/lfs.c

branches: 1.71.2;
Comment out unused logic. PR 50973

Need <stdbool.h> for true/false.

Enable lfs64 in fsck_lfs.
(and in everything else that uses this code)

Track down and fix two missing uses of SEGSUM_FINFOBASE().

Drop an explicit sign-extension in fsck that shouldn't be needed any
more.

Add lfs_checkword type for reading checksum data out of structures.
This is always uint32_t, but having a name for it both makes things
clearer and avoids confusion about whether it should be 32 or 64 bit.

Note: deployed in only one place (that was erroneously tagged
ondisk32) so far.

Fix hardwired 32-bit stuff in fsck:
- compute the maximum file size using LFS_BLKPTRSIZE()
- use the new IINFO in pass 6 instead of uint32_t pointers
- use accessors to read and write indirect blocks

Apply IINFO in the other userland tools.

Use daddr_t, not ulfs_daddr_t, as the latter's 32 bits wide.
Don't use either for on-disk items.
Declare external data in header files.
Part 3 of 3.

Use daddr_t, not ulfs_daddr_t, as the latter's 32 bits wide.
Don't use either for on-disk items.
Part 2 of 3.

Use daddr_t, not ulfs_daddr_t, as the latter's 32 bits wide.
Don't use either for on-disk items.
Part 1 of 3.

The ifile's inode number is constant. (it is always 1)

Therefore, storing the value in the superblock and reading it out
again is silly and offers the opportunity for it to become corrupted.
So, don't do that (most of the code already didn't) and use the
existing constant instead. Initialize new 32-bit superblocks with
the value for the sake of old userland programs, but don't keep the
value in the 64-bit superblock at all.

(approved by Margo Seltzer)

Add byteswapping to the dinode accessors.

This prevents regressions in the ulfs code when switching to the new
accessors. Note that while adding byteswapping to the other accessors
is straightforward, I haven't done it yet; and that also is not enough
to make LFS_EI work, because there are places lying around that bypass
the accessors for one reason and another and all of them need to be
updated. That is going to have to wait for a later day as LFS_EI is
not on the critical path right now.

Use the lfs dinode accessors in place of the ufs-derived ones.
(Mostly.)

The ufs-derived ones are fake structure member macros, which are gross
and not very safe. Also, it seems that a lot of places in the lfs code
were using the ffsv1 branch of them unconditionally, and this way it's
guaranteed all those places have been updated.

Found while doing this: for non-devices, have getattr produce NODEV
in the rdev field instead of leaking the address of the first direct
block.

Part two of dinodes; use the same union everywhere.
(previously the ufs-derived code had things set up slightly different)

Remove a bunch of associated mess.

Hack up dinode usage to be 64 vs. 32 as needed. Part 1.

(This part changes the native lfs code; the ufs-derived code already
has 64 vs. 32 logic, but as aspects of it are unsafe, and don't
entirely interoperate cleanly with the lfs 64/32 stuff, pass 2 will be
rehashing that.)

Provide 32-bit and 64-bit versions of FINFO.

This also entailed sorting out part of struct segment, as that
contains a pointer into the current FINFO data.

Make 32-bit and 64-bit versions of SEGSUM.
Also fix some of the FINFO handling as it's closely entangled.

Add IFILE32 and IFILE64 structures for the on-disk ifile entries.
Add and use accessors. There are also a bunch of places that cast and
I hope I've found them all...

Add a (draft) 64-bit superblock. Make things build again.

Add pieces of support for using both superblock types where
convenient, and specifically to the superblock accessors, but don't
actually enable it anywhere.

First substantive step on PR 50000.

Use accessor functions for the version field of the lfs superblock.
I thought at first maybe the cases that test the version should be
rolled into the accessors, but on the whole I think the conclusion on
that is no.

Second batch of 64 -> 32 truncations in lfs, along with more minor
tidyups and corrections in passing.

Fix assorted 64 -> 32 truncations in lfs. Also, some minor tidyups and
corrections in passing.

Add a new lfs header file: lfs_accessors.h.

This contains all the accessor functions and macros out of lfs.h.
Add an include of lfs_accessors.h after all uses of lfs.h... except
for code that wants to define its own struct lfs-alike that the
accessors are supposed to play along with. For these, set STRUCT_LFS
and include lfs_accessors.h after the necessary structure has been
defined, so that lfs_accessors.h can emit functions in terms of it.

More lfs superblock accessors.
(This changes the rest of the code over; all the accessors were
already added.)

The difference between this commit and the previous one is arbitrary,
but the previous one passed the regression tests on its own so I'm
keeping it separate to help with any bisections that might be needed
in the future.

Switch to accessor functions for elements of the LFS on-disk
superblock. This will allow switching between 32/64 bit forms on the
fly; it will also allow handling LFS_EI reasonably tidily. (That
currently doesn't work on the superblock.)

It also gets rid of cpp abuse in the form of fake structure member
macros.

Also, instead of doing sleep/wakeup on &lfs_avail and &lfs_nextseg
inside the on-disk superblock, add extra elements to the in-memory
struct lfs for this. (XXX: these should be changed to condvars, but
not right now)

XXX: this migrates a structure needed by the lfs code in libsa (struct
salfs) into lfs.h, where it doesn't belong, but for the time being
this is necessary in order to allow the accessors (and the various
lfs macros and other goop that relies on them) to compile.

Use VFS_PROTOS() for lfs.
Rename conflicting struct lfs field "lfs_start" to "lfs_s0addr".

No functional change.

- Fix fallout for recent bread() change (removing cred arg).

Revert previous; it doesn't work because all the lfs tools
promiscuously .PATH in each other's source files, and I haven't the
time or patience to deal with it tonight.

Remove pointless function indirection through panic_func(). It was
initialized to one thing, and then set to another right at the top of
main and never changed again.

branches: 1.41.2;
fix unused variable warnings.

Prefix most of the cpp macros with lfs_ and LFS_ to avoid conflicts with ffs.
This was done so that boot blocks that want to compile both FFS and LFS in
the same file work.

Tidy up the LFS userland build hacks.
Don't use -I${NETBSDSRCDIR}/sys; don't include files other than the
exported LFS headers, which are lfs.h, lfs_inode.h, and (for now)
lfs_extern.h.

Cleanups and hacks to make lfs userland stuff build:
- lfs_cksum.c doesn't actually need ulfs_inode.h any more.
- neither does lfs_itimes.c.
- add hacks to fsck_lfs to make it compile.
- add hacks to newfs_lfs to make it compile.
- fix warning in ulfs_quota.c when quotas are fully disabled
(as I guess is happening with the rumpity version)

XXX: This commit adds -I${NETBSDSRCDIR}/sys to the Makefiles for
XXX: fsck_lfs, newfs_lfs, and lfs_cleanerd. This needs to be cleaned
XXX: up ASAP; but I consider this less problematic in the short term
XXX: than spewing ulfs_*.h into /usr/include.

ufs -> ulfs for fsck_lfs.

Stuff UFS_ in front of a few of ufs's symbols to reduce namespace
pollution. Specifically:
ROOTINO -> UFS_ROOTINO
WINO -> UFS_WINO
NXADDR -> UFS_NXADDR
NDADDR -> UFS_NDADDR
NIADDR -> UFS_NIADDR
MAXSYMLINKLEN -> UFS_MAXSYMLINKLEN
MAXSYMLINKLEN_UFS[12] -> UFS[12]_MAXSYMLINKLEN (for consistency)

Sort out ext2fs's misuse of NDADDR and NIADDR; fortunately, these have
the same values in ext2fs and ffs.

No functional change intended.

branches: 1.35.2; 1.35.8;
Don't need to initialize (one of) the elements now in i_crap; nothing in
here touches that material.

avoid code that looks like this:
x = x = y;
since GCC 4.5 complains about it having sequence point issues.

branches: 1.33.4;
dev_bsize wasn't initialized. The actual value doesn't matter since
the same value is used to compute byte offsets into the special file
but a value of zero causes a division by zero.

Three changes in a single commit.

- drop the notion of frags (LFS fragments) vs fsb (FFS fragments)
The code uses a complicated unity function that just makes the
code difficult to understand.

- support larger sector sizes. Fix disk address computations
to use DEV_BSIZE in the kernel as required by device drivers
and to use sector sizes in userland.

- Fix several locking bugs in lfs_bio.c and lfs_subr.c.

Define syscalls of lfs userspace tools (cleaner, mainly) through
a struct called kernelops, which contains standard system calls
for the normal case and rump system calls for the rump case.

Make it possible to run the lfs cleaner in a library fashion (taking
the quick route with the implementation).

PR kern/26878 FFSv2 + softdep = livelock (no free ram)
PR kern/16942 panic with softdep and quotas
PR kern/19565 panic: softdep_write_inodeblock: indirect pointer #1 mismatch
PR kern/26274 softdep panic: allocdirect_merge: ...
PR kern/26374 Long delay before non-root users can write to softdep partitions
PR kern/28621 1.6.x "vp != NULL" panic in ffs_softdep.c:4653 while unmounting a softdep (+quota) filesystem
PR kern/29513 FFS+Softdep panic with unfsck-able file-corruption
PR kern/31544 The ffs softdep code appears to fail to write dirty bits to disk
PR kern/31981 stopping scsi disk can cause panic (softdep)
PR kern/32116 kernel panic in softdep (assertion failure)
PR kern/32532 softdep_trackbufs deadlock
PR kern/37191 softdep: locking against myself
PR kern/40474 Kernel panic after remounting raid root with softdep

Retire softdep, pass 2. As discussed and later formally announced on the
mailing lists.

branches: 1.29.4;
Make sure all cached buffers with valid, not yet written data have been
run through copy-on-write. Call fscow_run() with valid data where possible.

The LP_UFSCOW hack is no longer needed to protect ffs_copyonwrite() against
endless recursion.

- Add a flag B_MODIFY to bread(), breada() and breadn(). If set the caller
intends to modify the buffer returned.

- Always run copy-on-write on buffers returned from ffs_balloc().

- Add new function ffs_getblk() that gets a buffer, assigns a new blkno,
may clear the buffer and runs copy-on-write. Process possible errors
from getblk() or fscow_run(). Part of PR kern/38664.

Welcome to 4.99.63

Reviewed by: YAMAMOTO Takashi <yamt@netbsd.org>

branches: 1.28.2;
Remove clause 3 and 4 from TNF licenses

branches: 1.27.8; 1.27.10;
Give brelse() a second argument so that it matches the kernel.
fsck_lfs now compiles again.

branches: 1.26.8;
Fix malloc/realloc/calloc issues: always check and exit, use EEXIT instead
of 8.

Several fixes to improve the reliability of the roll-forward agent.
Also, note "properly orphaned" files as distinct from corrupted files.

Various improvements to fsck_lfs, to wit:

* Add lfs_balloc capability to the lfs library.
* Extend the Ifile if we run out of free inodes when creating lost+found.
* Don't roll forward if we have allocated a lost+found, to avoid
conflicts when adding new files in roll-forward.
* Make some messages slightly more verbose (e.g. include inode number,
and use pwarn() instead of printf() so the messages include the device
name when preening).
* Change superblock detection/avoidance to use the offset table in the
primary superblock, rather than looking at the contents.
* Be more verbose about various operations when passed the -d flag,
especially roll-forward.
* Be more careful about dirops during roll forward, since the cleaner can
sometimes write blocks from dirop vnodes. Detect and avoid this problem.
* Always check the free list, even if given -i; if we're going to write
it we have to check it first.
* Mark inodes dirty when blocks are found during roll forward, so the
inodes are written with the new block locations.
* Update size of inodes if blocks beyond EOF are found during roll
forward.
* Fix segment accounting for blocks and inodes found during roll
forward.
* Report statistics on roll forward: how many new/deleted/moved files
and how many updated blocks (or "nothing new").
* Don't care if the device being checked is really a device, if we have
been passed the -f flag (to facilitate automated testing).
* When writing to the disk, use the current time in the segment headers
rathern than time 0.
* When passed the -i flag, locate the partial segment containing the
Ifile inode and use that to calculate lfs_offset, lfs_curseg,
lfs_nextseg. (Again for automated testing.)

fsck_lfs does not us fsutil.c; perhaps it should? revert to perror for now.

s/perror/perr

Remove the free list ordering/disordering code, since the kernel now keeps
the list in order (ordering it on mount).

Regularize error messages: these are now all in ALL CAPS, with all hex
numbers (not reported in caps) prefixed by 0x. (The non-fsck-specific
messages are an exception to this all-caps rule.)

Check for allocation failures in malloc, calloc, realloc, asprintf, and
vasprintf and try to handle them.

Put back removed initializer. gcc on sh3 still doesn't get it correct...

avoid the need for a bogus initializer.

rename lfs.h to lfs_user.h so that it does not conflict.

Use the correct method to create a new inode, when we allocate lost+found.

Correct uninitialized variable issues in pass6.c and dir.c (PR#30411 and
PR#30394, respectively).

Initialize metalbn in ufs_getlbns to appease -Wuninitialized.
Marked with XXXGCC for dreamcast (found while compiling for it).

Reviewed by lukem.

appease gcc -Wuninitialized

Check some error conditions that would otherwise cause fsck_lfs to dump core.
Pointed out by Pavel Cahyna in a follow-on to PR #29151.

Check parts of pass 5 even if only rolling forward. We can't check the true
segment holdings against the blocks held by the inodes, but we can still
check the cleanerinfo data against the segment table.

When creating an int hash value from pointer, go via intptr_t.
Fixes PR 29953.

Be more efficient with the hash tables for the buffer and vnode caches.

Note that roll-forward can add more inodes to the filesystem; don't overflow
the tables but reallocate them.

"#define lfs_devvp lfs_unlockvp" for readability, since that's what we
use it for in fsck_lfs/newfs_lfs.

branches: 1.8.2;
Various minor LFS improvements:

* Extend the lfs library from fsck_lfs(8) so that it can be used with a
not-yet-existent LFS. Make newfs_lfs(8) use this library, so it can
create LFSs whose Ifile is larger than one segment.
* Make newfs_lfs(8) use strsuftoi64() for its arguments, a la newfs(8).
* Make fsck_lfs(8) respect the "file system is clean" flag.
* Don't let fsck_lfs(8) think it has dirty blocks when invoked with the
-n flag.

branches: 1.7.4;
Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22308, verified by myself.

in lfs_raw_vget(),
- don't leave references into free'ed memory region. (fix SIGBUSes)
- plug a memory leak.

- don't assume that malloc'ed memory is zero-filled.
- LIST_INIT {dirty,clean} block lists.

only read superblock (ie. struct dlfs) from disk and
initialize rest of struct lfs by hand.
(this shouldn't cause a real problem since if superblock is valid,
LFS_SBPAD-sizeof(struct dlfs) bytes after it is always zero-filled, though)

PR/22123 (Izumi Tsutsui)

Fix format string. (no % in PRIx..).

Add support for UFS2. UFS2 is an enhanced FFS, adding support for
64 bit block pointers, extended attribute storage, and a few
other things.

This commit does not yet include the code to manipulate the extended
storage (for e.g. ACLs), this will be done later.

Originally written by Kirk McKusick and Network Associates Laboratories for
FreeBSD.

Add working writing ability to fsck_lfs, including roll-forward, based on
a partial-segment writer ported from the kernel.

Update inode member i_flag --> i_state to keep up with kernel changes

PR/51478: Jose Luis Rodriguez Garcia: Fix leak mem fsck_lfs/lfs.c

branches: 1.71.2;
Comment out unused logic. PR 50973

Need <stdbool.h> for true/false.

Enable lfs64 in fsck_lfs.
(and in everything else that uses this code)

Track down and fix two missing uses of SEGSUM_FINFOBASE().

Drop an explicit sign-extension in fsck that shouldn't be needed any
more.

Add lfs_checkword type for reading checksum data out of structures.
This is always uint32_t, but having a name for it both makes things
clearer and avoids confusion about whether it should be 32 or 64 bit.

Note: deployed in only one place (that was erroneously tagged
ondisk32) so far.

Fix hardwired 32-bit stuff in fsck:
- compute the maximum file size using LFS_BLKPTRSIZE()
- use the new IINFO in pass 6 instead of uint32_t pointers
- use accessors to read and write indirect blocks

Apply IINFO in the other userland tools.

Use daddr_t, not ulfs_daddr_t, as the latter's 32 bits wide.
Don't use either for on-disk items.
Declare external data in header files.
Part 3 of 3.

Use daddr_t, not ulfs_daddr_t, as the latter's 32 bits wide.
Don't use either for on-disk items.
Part 2 of 3.

Use daddr_t, not ulfs_daddr_t, as the latter's 32 bits wide.
Don't use either for on-disk items.
Part 1 of 3.

The ifile's inode number is constant. (it is always 1)

Therefore, storing the value in the superblock and reading it out
again is silly and offers the opportunity for it to become corrupted.
So, don't do that (most of the code already didn't) and use the
existing constant instead. Initialize new 32-bit superblocks with
the value for the sake of old userland programs, but don't keep the
value in the 64-bit superblock at all.

(approved by Margo Seltzer)

Add byteswapping to the dinode accessors.

This prevents regressions in the ulfs code when switching to the new
accessors. Note that while adding byteswapping to the other accessors
is straightforward, I haven't done it yet; and that also is not enough
to make LFS_EI work, because there are places lying around that bypass
the accessors for one reason and another and all of them need to be
updated. That is going to have to wait for a later day as LFS_EI is
not on the critical path right now.

Use the lfs dinode accessors in place of the ufs-derived ones.
(Mostly.)

The ufs-derived ones are fake structure member macros, which are gross
and not very safe. Also, it seems that a lot of places in the lfs code
were using the ffsv1 branch of them unconditionally, and this way it's
guaranteed all those places have been updated.

Found while doing this: for non-devices, have getattr produce NODEV
in the rdev field instead of leaking the address of the first direct
block.

Part two of dinodes; use the same union everywhere.
(previously the ufs-derived code had things set up slightly different)

Remove a bunch of associated mess.

Hack up dinode usage to be 64 vs. 32 as needed. Part 1.

(This part changes the native lfs code; the ufs-derived code already
has 64 vs. 32 logic, but as aspects of it are unsafe, and don't
entirely interoperate cleanly with the lfs 64/32 stuff, pass 2 will be
rehashing that.)

Provide 32-bit and 64-bit versions of FINFO.

This also entailed sorting out part of struct segment, as that
contains a pointer into the current FINFO data.

Make 32-bit and 64-bit versions of SEGSUM.
Also fix some of the FINFO handling as it's closely entangled.

Add IFILE32 and IFILE64 structures for the on-disk ifile entries.
Add and use accessors. There are also a bunch of places that cast and
I hope I've found them all...

Add a (draft) 64-bit superblock. Make things build again.

Add pieces of support for using both superblock types where
convenient, and specifically to the superblock accessors, but don't
actually enable it anywhere.

First substantive step on PR 50000.

Use accessor functions for the version field of the lfs superblock.
I thought at first maybe the cases that test the version should be
rolled into the accessors, but on the whole I think the conclusion on
that is no.

Second batch of 64 -> 32 truncations in lfs, along with more minor
tidyups and corrections in passing.

Fix assorted 64 -> 32 truncations in lfs. Also, some minor tidyups and
corrections in passing.

Add a new lfs header file: lfs_accessors.h.

This contains all the accessor functions and macros out of lfs.h.
Add an include of lfs_accessors.h after all uses of lfs.h... except
for code that wants to define its own struct lfs-alike that the
accessors are supposed to play along with. For these, set STRUCT_LFS
and include lfs_accessors.h after the necessary structure has been
defined, so that lfs_accessors.h can emit functions in terms of it.

More lfs superblock accessors.
(This changes the rest of the code over; all the accessors were
already added.)

The difference between this commit and the previous one is arbitrary,
but the previous one passed the regression tests on its own so I'm
keeping it separate to help with any bisections that might be needed
in the future.

Switch to accessor functions for elements of the LFS on-disk
superblock. This will allow switching between 32/64 bit forms on the
fly; it will also allow handling LFS_EI reasonably tidily. (That
currently doesn't work on the superblock.)

It also gets rid of cpp abuse in the form of fake structure member
macros.

Also, instead of doing sleep/wakeup on &lfs_avail and &lfs_nextseg
inside the on-disk superblock, add extra elements to the in-memory
struct lfs for this. (XXX: these should be changed to condvars, but
not right now)

XXX: this migrates a structure needed by the lfs code in libsa (struct
salfs) into lfs.h, where it doesn't belong, but for the time being
this is necessary in order to allow the accessors (and the various
lfs macros and other goop that relies on them) to compile.

Use VFS_PROTOS() for lfs.
Rename conflicting struct lfs field "lfs_start" to "lfs_s0addr".

No functional change.

- Fix fallout for recent bread() change (removing cred arg).

Revert previous; it doesn't work because all the lfs tools
promiscuously .PATH in each other's source files, and I haven't the
time or patience to deal with it tonight.

Remove pointless function indirection through panic_func(). It was
initialized to one thing, and then set to another right at the top of
main and never changed again.

branches: 1.41.2;
fix unused variable warnings.

Prefix most of the cpp macros with lfs_ and LFS_ to avoid conflicts with ffs.
This was done so that boot blocks that want to compile both FFS and LFS in
the same file work.

Tidy up the LFS userland build hacks.
Don't use -I${NETBSDSRCDIR}/sys; don't include files other than the
exported LFS headers, which are lfs.h, lfs_inode.h, and (for now)
lfs_extern.h.

Cleanups and hacks to make lfs userland stuff build:
- lfs_cksum.c doesn't actually need ulfs_inode.h any more.
- neither does lfs_itimes.c.
- add hacks to fsck_lfs to make it compile.
- add hacks to newfs_lfs to make it compile.
- fix warning in ulfs_quota.c when quotas are fully disabled
(as I guess is happening with the rumpity version)

XXX: This commit adds -I${NETBSDSRCDIR}/sys to the Makefiles for
XXX: fsck_lfs, newfs_lfs, and lfs_cleanerd. This needs to be cleaned
XXX: up ASAP; but I consider this less problematic in the short term
XXX: than spewing ulfs_*.h into /usr/include.

ufs -> ulfs for fsck_lfs.

Stuff UFS_ in front of a few of ufs's symbols to reduce namespace
pollution. Specifically:
ROOTINO -> UFS_ROOTINO
WINO -> UFS_WINO
NXADDR -> UFS_NXADDR
NDADDR -> UFS_NDADDR
NIADDR -> UFS_NIADDR
MAXSYMLINKLEN -> UFS_MAXSYMLINKLEN
MAXSYMLINKLEN_UFS[12] -> UFS[12]_MAXSYMLINKLEN (for consistency)

Sort out ext2fs's misuse of NDADDR and NIADDR; fortunately, these have
the same values in ext2fs and ffs.

No functional change intended.

branches: 1.35.2; 1.35.8;
Don't need to initialize (one of) the elements now in i_crap; nothing in
here touches that material.

avoid code that looks like this:
x = x = y;
since GCC 4.5 complains about it having sequence point issues.

branches: 1.33.4;
dev_bsize wasn't initialized. The actual value doesn't matter since
the same value is used to compute byte offsets into the special file
but a value of zero causes a division by zero.

Three changes in a single commit.

- drop the notion of frags (LFS fragments) vs fsb (FFS fragments)
The code uses a complicated unity function that just makes the
code difficult to understand.

- support larger sector sizes. Fix disk address computations
to use DEV_BSIZE in the kernel as required by device drivers
and to use sector sizes in userland.

- Fix several locking bugs in lfs_bio.c and lfs_subr.c.

Define syscalls of lfs userspace tools (cleaner, mainly) through
a struct called kernelops, which contains standard system calls
for the normal case and rump system calls for the rump case.

Make it possible to run the lfs cleaner in a library fashion (taking
the quick route with the implementation).

PR kern/26878 FFSv2 + softdep = livelock (no free ram)
PR kern/16942 panic with softdep and quotas
PR kern/19565 panic: softdep_write_inodeblock: indirect pointer #1 mismatch
PR kern/26274 softdep panic: allocdirect_merge: ...
PR kern/26374 Long delay before non-root users can write to softdep partitions
PR kern/28621 1.6.x "vp != NULL" panic in ffs_softdep.c:4653 while unmounting a softdep (+quota) filesystem
PR kern/29513 FFS+Softdep panic with unfsck-able file-corruption
PR kern/31544 The ffs softdep code appears to fail to write dirty bits to disk
PR kern/31981 stopping scsi disk can cause panic (softdep)
PR kern/32116 kernel panic in softdep (assertion failure)
PR kern/32532 softdep_trackbufs deadlock
PR kern/37191 softdep: locking against myself
PR kern/40474 Kernel panic after remounting raid root with softdep

Retire softdep, pass 2. As discussed and later formally announced on the
mailing lists.

branches: 1.29.4;
Make sure all cached buffers with valid, not yet written data have been
run through copy-on-write. Call fscow_run() with valid data where possible.

The LP_UFSCOW hack is no longer needed to protect ffs_copyonwrite() against
endless recursion.

- Add a flag B_MODIFY to bread(), breada() and breadn(). If set the caller
intends to modify the buffer returned.

- Always run copy-on-write on buffers returned from ffs_balloc().

- Add new function ffs_getblk() that gets a buffer, assigns a new blkno,
may clear the buffer and runs copy-on-write. Process possible errors
from getblk() or fscow_run(). Part of PR kern/38664.

Welcome to 4.99.63

Reviewed by: YAMAMOTO Takashi <yamt@netbsd.org>

branches: 1.28.2;
Remove clause 3 and 4 from TNF licenses

branches: 1.27.8; 1.27.10;
Give brelse() a second argument so that it matches the kernel.
fsck_lfs now compiles again.

branches: 1.26.8;
Fix malloc/realloc/calloc issues: always check and exit, use EEXIT instead
of 8.

Several fixes to improve the reliability of the roll-forward agent.
Also, note "properly orphaned" files as distinct from corrupted files.

Various improvements to fsck_lfs, to wit:

* Add lfs_balloc capability to the lfs library.
* Extend the Ifile if we run out of free inodes when creating lost+found.
* Don't roll forward if we have allocated a lost+found, to avoid
conflicts when adding new files in roll-forward.
* Make some messages slightly more verbose (e.g. include inode number,
and use pwarn() instead of printf() so the messages include the device
name when preening).
* Change superblock detection/avoidance to use the offset table in the
primary superblock, rather than looking at the contents.
* Be more verbose about various operations when passed the -d flag,
especially roll-forward.
* Be more careful about dirops during roll forward, since the cleaner can
sometimes write blocks from dirop vnodes. Detect and avoid this problem.
* Always check the free list, even if given -i; if we're going to write
it we have to check it first.
* Mark inodes dirty when blocks are found during roll forward, so the
inodes are written with the new block locations.
* Update size of inodes if blocks beyond EOF are found during roll
forward.
* Fix segment accounting for blocks and inodes found during roll
forward.
* Report statistics on roll forward: how many new/deleted/moved files
and how many updated blocks (or "nothing new").
* Don't care if the device being checked is really a device, if we have
been passed the -f flag (to facilitate automated testing).
* When writing to the disk, use the current time in the segment headers
rathern than time 0.
* When passed the -i flag, locate the partial segment containing the
Ifile inode and use that to calculate lfs_offset, lfs_curseg,
lfs_nextseg. (Again for automated testing.)

fsck_lfs does not us fsutil.c; perhaps it should? revert to perror for now.

s/perror/perr

Remove the free list ordering/disordering code, since the kernel now keeps
the list in order (ordering it on mount).

Regularize error messages: these are now all in ALL CAPS, with all hex
numbers (not reported in caps) prefixed by 0x. (The non-fsck-specific
messages are an exception to this all-caps rule.)

Check for allocation failures in malloc, calloc, realloc, asprintf, and
vasprintf and try to handle them.

Put back removed initializer. gcc on sh3 still doesn't get it correct...

avoid the need for a bogus initializer.

rename lfs.h to lfs_user.h so that it does not conflict.

Use the correct method to create a new inode, when we allocate lost+found.

Correct uninitialized variable issues in pass6.c and dir.c (PR#30411 and
PR#30394, respectively).

Initialize metalbn in ufs_getlbns to appease -Wuninitialized.
Marked with XXXGCC for dreamcast (found while compiling for it).

Reviewed by lukem.

appease gcc -Wuninitialized

Check some error conditions that would otherwise cause fsck_lfs to dump core.
Pointed out by Pavel Cahyna in a follow-on to PR #29151.

Check parts of pass 5 even if only rolling forward. We can't check the true
segment holdings against the blocks held by the inodes, but we can still
check the cleanerinfo data against the segment table.

When creating an int hash value from pointer, go via intptr_t.
Fixes PR 29953.

Be more efficient with the hash tables for the buffer and vnode caches.

Note that roll-forward can add more inodes to the filesystem; don't overflow
the tables but reallocate them.

"#define lfs_devvp lfs_unlockvp" for readability, since that's what we
use it for in fsck_lfs/newfs_lfs.

branches: 1.8.2;
Various minor LFS improvements:

* Extend the lfs library from fsck_lfs(8) so that it can be used with a
not-yet-existent LFS. Make newfs_lfs(8) use this library, so it can
create LFSs whose Ifile is larger than one segment.
* Make newfs_lfs(8) use strsuftoi64() for its arguments, a la newfs(8).
* Make fsck_lfs(8) respect the "file system is clean" flag.
* Don't let fsck_lfs(8) think it has dirty blocks when invoked with the
-n flag.

branches: 1.7.4;
Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22308, verified by myself.

in lfs_raw_vget(),
- don't leave references into free'ed memory region. (fix SIGBUSes)
- plug a memory leak.

- don't assume that malloc'ed memory is zero-filled.
- LIST_INIT {dirty,clean} block lists.

only read superblock (ie. struct dlfs) from disk and
initialize rest of struct lfs by hand.
(this shouldn't cause a real problem since if superblock is valid,
LFS_SBPAD-sizeof(struct dlfs) bytes after it is always zero-filled, though)

PR/22123 (Izumi Tsutsui)

Fix format string. (no % in PRIx..).

Add support for UFS2. UFS2 is an enhanced FFS, adding support for
64 bit block pointers, extended attribute storage, and a few
other things.

This commit does not yet include the code to manipulate the extended
storage (for e.g. ACLs), this will be done later.

Originally written by Kirk McKusick and Network Associates Laboratories for
FreeBSD.

Add working writing ability to fsck_lfs, including roll-forward, based on
a partial-segment writer ported from the kernel.

PR/51478: Jose Luis Rodriguez Garcia: Fix leak mem fsck_lfs/lfs.c

branches: 1.71.2;
Comment out unused logic. PR 50973

Need <stdbool.h> for true/false.

Enable lfs64 in fsck_lfs.
(and in everything else that uses this code)

Track down and fix two missing uses of SEGSUM_FINFOBASE().

Drop an explicit sign-extension in fsck that shouldn't be needed any
more.

Add lfs_checkword type for reading checksum data out of structures.
This is always uint32_t, but having a name for it both makes things
clearer and avoids confusion about whether it should be 32 or 64 bit.

Note: deployed in only one place (that was erroneously tagged
ondisk32) so far.

Fix hardwired 32-bit stuff in fsck:
- compute the maximum file size using LFS_BLKPTRSIZE()
- use the new IINFO in pass 6 instead of uint32_t pointers
- use accessors to read and write indirect blocks

Apply IINFO in the other userland tools.

Use daddr_t, not ulfs_daddr_t, as the latter's 32 bits wide.
Don't use either for on-disk items.
Declare external data in header files.
Part 3 of 3.

Use daddr_t, not ulfs_daddr_t, as the latter's 32 bits wide.
Don't use either for on-disk items.
Part 2 of 3.

Use daddr_t, not ulfs_daddr_t, as the latter's 32 bits wide.
Don't use either for on-disk items.
Part 1 of 3.

The ifile's inode number is constant. (it is always 1)

Therefore, storing the value in the superblock and reading it out
again is silly and offers the opportunity for it to become corrupted.
So, don't do that (most of the code already didn't) and use the
existing constant instead. Initialize new 32-bit superblocks with
the value for the sake of old userland programs, but don't keep the
value in the 64-bit superblock at all.

(approved by Margo Seltzer)

Add byteswapping to the dinode accessors.

This prevents regressions in the ulfs code when switching to the new
accessors. Note that while adding byteswapping to the other accessors
is straightforward, I haven't done it yet; and that also is not enough
to make LFS_EI work, because there are places lying around that bypass
the accessors for one reason and another and all of them need to be
updated. That is going to have to wait for a later day as LFS_EI is
not on the critical path right now.

Use the lfs dinode accessors in place of the ufs-derived ones.
(Mostly.)

The ufs-derived ones are fake structure member macros, which are gross
and not very safe. Also, it seems that a lot of places in the lfs code
were using the ffsv1 branch of them unconditionally, and this way it's
guaranteed all those places have been updated.

Found while doing this: for non-devices, have getattr produce NODEV
in the rdev field instead of leaking the address of the first direct
block.

Part two of dinodes; use the same union everywhere.
(previously the ufs-derived code had things set up slightly different)

Remove a bunch of associated mess.

Hack up dinode usage to be 64 vs. 32 as needed. Part 1.

(This part changes the native lfs code; the ufs-derived code already
has 64 vs. 32 logic, but as aspects of it are unsafe, and don't
entirely interoperate cleanly with the lfs 64/32 stuff, pass 2 will be
rehashing that.)

Provide 32-bit and 64-bit versions of FINFO.

This also entailed sorting out part of struct segment, as that
contains a pointer into the current FINFO data.

Make 32-bit and 64-bit versions of SEGSUM.
Also fix some of the FINFO handling as it's closely entangled.

Add IFILE32 and IFILE64 structures for the on-disk ifile entries.
Add and use accessors. There are also a bunch of places that cast and
I hope I've found them all...

Add a (draft) 64-bit superblock. Make things build again.

Add pieces of support for using both superblock types where
convenient, and specifically to the superblock accessors, but don't
actually enable it anywhere.

First substantive step on PR 50000.

Use accessor functions for the version field of the lfs superblock.
I thought at first maybe the cases that test the version should be
rolled into the accessors, but on the whole I think the conclusion on
that is no.

Second batch of 64 -> 32 truncations in lfs, along with more minor
tidyups and corrections in passing.

Fix assorted 64 -> 32 truncations in lfs. Also, some minor tidyups and
corrections in passing.

Add a new lfs header file: lfs_accessors.h.

This contains all the accessor functions and macros out of lfs.h.
Add an include of lfs_accessors.h after all uses of lfs.h... except
for code that wants to define its own struct lfs-alike that the
accessors are supposed to play along with. For these, set STRUCT_LFS
and include lfs_accessors.h after the necessary structure has been
defined, so that lfs_accessors.h can emit functions in terms of it.

More lfs superblock accessors.
(This changes the rest of the code over; all the accessors were
already added.)

The difference between this commit and the previous one is arbitrary,
but the previous one passed the regression tests on its own so I'm
keeping it separate to help with any bisections that might be needed
in the future.

Switch to accessor functions for elements of the LFS on-disk
superblock. This will allow switching between 32/64 bit forms on the
fly; it will also allow handling LFS_EI reasonably tidily. (That
currently doesn't work on the superblock.)

It also gets rid of cpp abuse in the form of fake structure member
macros.

Also, instead of doing sleep/wakeup on &lfs_avail and &lfs_nextseg
inside the on-disk superblock, add extra elements to the in-memory
struct lfs for this. (XXX: these should be changed to condvars, but
not right now)

XXX: this migrates a structure needed by the lfs code in libsa (struct
salfs) into lfs.h, where it doesn't belong, but for the time being
this is necessary in order to allow the accessors (and the various
lfs macros and other goop that relies on them) to compile.

Use VFS_PROTOS() for lfs.
Rename conflicting struct lfs field "lfs_start" to "lfs_s0addr".

No functional change.

- Fix fallout for recent bread() change (removing cred arg).

Revert previous; it doesn't work because all the lfs tools
promiscuously .PATH in each other's source files, and I haven't the
time or patience to deal with it tonight.

Remove pointless function indirection through panic_func(). It was
initialized to one thing, and then set to another right at the top of
main and never changed again.

branches: 1.41.2;
fix unused variable warnings.

Prefix most of the cpp macros with lfs_ and LFS_ to avoid conflicts with ffs.
This was done so that boot blocks that want to compile both FFS and LFS in
the same file work.

Tidy up the LFS userland build hacks.
Don't use -I${NETBSDSRCDIR}/sys; don't include files other than the
exported LFS headers, which are lfs.h, lfs_inode.h, and (for now)
lfs_extern.h.

Cleanups and hacks to make lfs userland stuff build:
- lfs_cksum.c doesn't actually need ulfs_inode.h any more.
- neither does lfs_itimes.c.
- add hacks to fsck_lfs to make it compile.
- add hacks to newfs_lfs to make it compile.
- fix warning in ulfs_quota.c when quotas are fully disabled
(as I guess is happening with the rumpity version)

XXX: This commit adds -I${NETBSDSRCDIR}/sys to the Makefiles for
XXX: fsck_lfs, newfs_lfs, and lfs_cleanerd. This needs to be cleaned
XXX: up ASAP; but I consider this less problematic in the short term
XXX: than spewing ulfs_*.h into /usr/include.

ufs -> ulfs for fsck_lfs.

Stuff UFS_ in front of a few of ufs's symbols to reduce namespace
pollution. Specifically:
ROOTINO -> UFS_ROOTINO
WINO -> UFS_WINO
NXADDR -> UFS_NXADDR
NDADDR -> UFS_NDADDR
NIADDR -> UFS_NIADDR
MAXSYMLINKLEN -> UFS_MAXSYMLINKLEN
MAXSYMLINKLEN_UFS[12] -> UFS[12]_MAXSYMLINKLEN (for consistency)

Sort out ext2fs's misuse of NDADDR and NIADDR; fortunately, these have
the same values in ext2fs and ffs.

No functional change intended.

branches: 1.35.2; 1.35.8;
Don't need to initialize (one of) the elements now in i_crap; nothing in
here touches that material.

avoid code that looks like this:
x = x = y;
since GCC 4.5 complains about it having sequence point issues.

branches: 1.33.4;
dev_bsize wasn't initialized. The actual value doesn't matter since
the same value is used to compute byte offsets into the special file
but a value of zero causes a division by zero.

Three changes in a single commit.

- drop the notion of frags (LFS fragments) vs fsb (FFS fragments)
The code uses a complicated unity function that just makes the
code difficult to understand.

- support larger sector sizes. Fix disk address computations
to use DEV_BSIZE in the kernel as required by device drivers
and to use sector sizes in userland.

- Fix several locking bugs in lfs_bio.c and lfs_subr.c.

Define syscalls of lfs userspace tools (cleaner, mainly) through
a struct called kernelops, which contains standard system calls
for the normal case and rump system calls for the rump case.

Make it possible to run the lfs cleaner in a library fashion (taking
the quick route with the implementation).

PR kern/26878 FFSv2 + softdep = livelock (no free ram)
PR kern/16942 panic with softdep and quotas
PR kern/19565 panic: softdep_write_inodeblock: indirect pointer #1 mismatch
PR kern/26274 softdep panic: allocdirect_merge: ...
PR kern/26374 Long delay before non-root users can write to softdep partitions
PR kern/28621 1.6.x "vp != NULL" panic in ffs_softdep.c:4653 while unmounting a softdep (+quota) filesystem
PR kern/29513 FFS+Softdep panic with unfsck-able file-corruption
PR kern/31544 The ffs softdep code appears to fail to write dirty bits to disk
PR kern/31981 stopping scsi disk can cause panic (softdep)
PR kern/32116 kernel panic in softdep (assertion failure)
PR kern/32532 softdep_trackbufs deadlock
PR kern/37191 softdep: locking against myself
PR kern/40474 Kernel panic after remounting raid root with softdep

Retire softdep, pass 2. As discussed and later formally announced on the
mailing lists.

branches: 1.29.4;
Make sure all cached buffers with valid, not yet written data have been
run through copy-on-write. Call fscow_run() with valid data where possible.

The LP_UFSCOW hack is no longer needed to protect ffs_copyonwrite() against
endless recursion.

- Add a flag B_MODIFY to bread(), breada() and breadn(). If set the caller
intends to modify the buffer returned.

- Always run copy-on-write on buffers returned from ffs_balloc().

- Add new function ffs_getblk() that gets a buffer, assigns a new blkno,
may clear the buffer and runs copy-on-write. Process possible errors
from getblk() or fscow_run(). Part of PR kern/38664.

Welcome to 4.99.63

Reviewed by: YAMAMOTO Takashi <yamt@netbsd.org>

branches: 1.28.2;
Remove clause 3 and 4 from TNF licenses

branches: 1.27.8; 1.27.10;
Give brelse() a second argument so that it matches the kernel.
fsck_lfs now compiles again.

branches: 1.26.8;
Fix malloc/realloc/calloc issues: always check and exit, use EEXIT instead
of 8.

Several fixes to improve the reliability of the roll-forward agent.
Also, note "properly orphaned" files as distinct from corrupted files.

Various improvements to fsck_lfs, to wit:

* Add lfs_balloc capability to the lfs library.
* Extend the Ifile if we run out of free inodes when creating lost+found.
* Don't roll forward if we have allocated a lost+found, to avoid
conflicts when adding new files in roll-forward.
* Make some messages slightly more verbose (e.g. include inode number,
and use pwarn() instead of printf() so the messages include the device
name when preening).
* Change superblock detection/avoidance to use the offset table in the
primary superblock, rather than looking at the contents.
* Be more verbose about various operations when passed the -d flag,
especially roll-forward.
* Be more careful about dirops during roll forward, since the cleaner can
sometimes write blocks from dirop vnodes. Detect and avoid this problem.
* Always check the free list, even if given -i; if we're going to write
it we have to check it first.
* Mark inodes dirty when blocks are found during roll forward, so the
inodes are written with the new block locations.
* Update size of inodes if blocks beyond EOF are found during roll
forward.
* Fix segment accounting for blocks and inodes found during roll
forward.
* Report statistics on roll forward: how many new/deleted/moved files
and how many updated blocks (or "nothing new").
* Don't care if the device being checked is really a device, if we have
been passed the -f flag (to facilitate automated testing).
* When writing to the disk, use the current time in the segment headers
rathern than time 0.
* When passed the -i flag, locate the partial segment containing the
Ifile inode and use that to calculate lfs_offset, lfs_curseg,
lfs_nextseg. (Again for automated testing.)

fsck_lfs does not us fsutil.c; perhaps it should? revert to perror for now.

s/perror/perr

Remove the free list ordering/disordering code, since the kernel now keeps
the list in order (ordering it on mount).

Regularize error messages: these are now all in ALL CAPS, with all hex
numbers (not reported in caps) prefixed by 0x. (The non-fsck-specific
messages are an exception to this all-caps rule.)

Check for allocation failures in malloc, calloc, realloc, asprintf, and
vasprintf and try to handle them.

Put back removed initializer. gcc on sh3 still doesn't get it correct...

avoid the need for a bogus initializer.

rename lfs.h to lfs_user.h so that it does not conflict.

Use the correct method to create a new inode, when we allocate lost+found.

Correct uninitialized variable issues in pass6.c and dir.c (PR#30411 and
PR#30394, respectively).

Initialize metalbn in ufs_getlbns to appease -Wuninitialized.
Marked with XXXGCC for dreamcast (found while compiling for it).

Reviewed by lukem.

appease gcc -Wuninitialized

Check some error conditions that would otherwise cause fsck_lfs to dump core.
Pointed out by Pavel Cahyna in a follow-on to PR #29151.

Check parts of pass 5 even if only rolling forward. We can't check the true
segment holdings against the blocks held by the inodes, but we can still
check the cleanerinfo data against the segment table.

When creating an int hash value from pointer, go via intptr_t.
Fixes PR 29953.

Be more efficient with the hash tables for the buffer and vnode caches.

Note that roll-forward can add more inodes to the filesystem; don't overflow
the tables but reallocate them.

"#define lfs_devvp lfs_unlockvp" for readability, since that's what we
use it for in fsck_lfs/newfs_lfs.

branches: 1.8.2;
Various minor LFS improvements:

* Extend the lfs library from fsck_lfs(8) so that it can be used with a
not-yet-existent LFS. Make newfs_lfs(8) use this library, so it can
create LFSs whose Ifile is larger than one segment.
* Make newfs_lfs(8) use strsuftoi64() for its arguments, a la newfs(8).
* Make fsck_lfs(8) respect the "file system is clean" flag.
* Don't let fsck_lfs(8) think it has dirty blocks when invoked with the
-n flag.

branches: 1.7.4;
Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22308, verified by myself.

in lfs_raw_vget(),
- don't leave references into free'ed memory region. (fix SIGBUSes)
- plug a memory leak.

- don't assume that malloc'ed memory is zero-filled.
- LIST_INIT {dirty,clean} block lists.

only read superblock (ie. struct dlfs) from disk and
initialize rest of struct lfs by hand.
(this shouldn't cause a real problem since if superblock is valid,
LFS_SBPAD-sizeof(struct dlfs) bytes after it is always zero-filled, though)

PR/22123 (Izumi Tsutsui)

Fix format string. (no % in PRIx..).

Add support for UFS2. UFS2 is an enhanced FFS, adding support for
64 bit block pointers, extended attribute storage, and a few
other things.

This commit does not yet include the code to manipulate the extended
storage (for e.g. ACLs), this will be done later.

Originally written by Kirk McKusick and Network Associates Laboratories for
FreeBSD.

Add working writing ability to fsck_lfs, including roll-forward, based on
a partial-segment writer ported from the kernel.