ext2: use st_blocks from the disk inode structure

* this lets cp from coreutils use lseek to find the data chunks in a
sparse file.
* fix endian conversion macro used for num_blocks_high

Change-Id: I221d6316002b1c491ae987aeef3f25e8721b5ab9
Reviewed-on: https://review.haiku-os.org/c/haiku/+/5218
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>

ext2: fix metadata_csum for inodes

the checksum assumes the full inode block.

Change-Id: I60ef5d0b769750107b7e104709edff95beee8085
Reviewed-on: https://review.haiku-os.org/c/haiku/+/4118
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

file_systems/ext2: Fix -WFormat=

Change-Id: I10d3ab0343c4226dc19e0eade0e91eed6f27eb07
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3291
Reviewed-by: Jérôme Duval <jerome.duval@gmail.com>

ext2: implements metadata_csum and 64bit features.

* Some PVS-Studio warnings are removed.
* fixes hardlink count in source and destination folders on rename.
* tested with fstorture and fsx (one job).
* add new definitions.
* import crc32c code from BSD.
* add some consistency checks at mount.
* DirectoryIterator::_AllocateBestEntryInBlock(), the direntry is checked for
consistency before using it, which should avoid crashing on bad data.
* DirectoryIterator::_SplitIndexedBlock(): the dotdot entry length should
include the dot and dotdot entries length.

Change-Id: I0f80d73b65b1ae6ddb2e746a6f85ef806f23dbb0
Reviewed-on: https://review.haiku-os.org/c/haiku/+/1735
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

file_systems: Remove now-redundant bounds checks before file_cache_read().

Change-Id: Iafb7d188c7e7cb4406d924eb3354a7ede40c6641
Reviewed-on: https://review.haiku-os.org/c/1421
Reviewed-by: waddlesplash <waddlesplash@gmail.com>
Reviewed-by: Michael Lotz <mmlr@mlotz.ch>

vfs/{b|btr|package|b}fs/ext2/exfat: common access check.

* Added VFS helper function check_access_permissions() that combines
several partially correct versions to the one true version (tm).
* All but BFS (since recently) missed the S_IXOTH for root on directories,
and all but packagefs missed proper group handling.

ext2: Fix enabling/disabling the file cache

* Inode:
- Rename {Enable,Disable}FileCache() to {Create,Delete}FileCache()
and IsFileCacheDisabled() to HasFileCache(), since that is what they
actually do. DeleteFileCache() now also sets the attributes to NULL,
which makes fCached superfluous.
- Introduce {Enable,Disable}FileCache(), which actually enable/disable
the file cache. Use those methods for handling O_NOCACHE.
* ext2_free_cookie(): Reenable the file cache in case of O_NOCACHE.

Fixes crash when O_NOCACHE was used, since the file cache was deleted
without clearing the attribute and Inode::ReadAt() would use the deleted
object afterward.

ext2: access the parent variable once checked it's non null

* cleanup
* add some inode flags we don't use

ext2: access the parent variable once checked it's non null

* cleanup
* add some inode flags we don't use

ext2: fixes 64 bit warnings

* added flex group, dirnlink and gdtchecksum features for ext4.
* I reused crc_table.cpp from the UDF filesystem and switched it to have the reversed algorithm,
then generated the table in CRCTable.cpp
* added a binary search for extent tree leaves.
* fixed a check in InodeAllocator::New().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40129 a95241bf-73f2-0310-859d-f6bbb57e9c96

resource leaks CID 5711, 5712, 5713, 5714


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39902 a95241bf-73f2-0310-859d-f6bbb57e9c96

* added fsblock_t and fileblock_t definitions, used them when needed.
* validate fBlockShift in superblock.
* Volume::AllocateBlocks() now uses an absolute fsblock instead of a blockgroup related fsblock.
* AllocationBlockGroup now provides absolute fsblock values.
* added support for extents feature: ExtentStream class is the equivalent for DataStream class for extent operations.
The extent tree implementation is very basic, should work for normal growing/shrinking
operations, but not for sparse files. When enlarging a file and extent tree is full,
the root is moved in a new block and a new level is added on top. Extents can usually
be extended when adjacent blocks are allocated. Shrinking happens by removing leafs
one after another.
* removed empty IndexedDirectoryIterator.*


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39886 a95241bf-73f2-0310-859d-f6bbb57e9c96

moved FindBlock() from Inode to DataStream


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39820 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Applied patch by Rohit Yadav that fixes #6750 for ext2. Thanks a lot!
* Fixed warning in Inode::WriteAt(), but left warnings in kernel_interface.cpp
in there; I guess that's work in progress (although having the io hook would
be nice).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39379 a95241bf-73f2-0310-859d-f6bbb57e9c96

* more cleanup
* more fixes in BitmapBlock::FindNextMarked() and BitmapBlock::FindPreviousMarked()


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39242 a95241bf-73f2-0310-859d-f6bbb57e9c96

* switched more uint32 to off_t for 64bit support
* added error messages in InodeAllocator, Inode
* if BlockAllocator can't initialize, don't fail completely but switch to readonly
* fixed a bug in FindNextMarked() for bitmaps with a length non multiple of 32
* Inode::FindBlock() now returns an optional block_run length, useful for get_file_map()
* added flag for Inode for extents


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39234 a95241bf-73f2-0310-859d-f6bbb57e9c96

* added some support for 64bit feature: extended struct ext2_block_group, block number types changed from uint32 to off_t
* added error traces, asserts
* BitmapBlock::CheckUnmarked() and CheckMarked() computed a wrong remainingBits and mask


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39203 a95241bf-73f2-0310-859d-f6bbb57e9c96

added some definitions for readonly features
we already support the extra isize feature, so it's now added
removed unused Inode::AttributeBlockReadAt()
group_descriptor_size was previously a reserved field in the super block


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39125 a95241bf-73f2-0310-859d-f6bbb57e9c96

added large files feature support


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39120 a95241bf-73f2-0310-859d-f6bbb57e9c96

ext3 inodes whose size is at least 256 bytes can also contain attributes (like smalldata for bfs).
We now use the Attribute class from bfs (instead of AttributeIterator) to manage small data and block attributes, though it's still readonly.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39036 a95241bf-73f2-0310-859d-f6bbb57e9c96

* added support for extra inode attributes in volumes with an inodesize of 256 or more:
this brings nanoseconds for access, change and modification times, and also brings creation time.
* switched off some debug output
* HTreeEntryIterator: fCount can equal fLimit.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@38964 a95241bf-73f2-0310-859d-f6bbb57e9c96

* added support for huge_file feature, untested (need a >2TB file)
* actually checks for readonly features if mounting read-write
* don't use a journal when mounting readonly
* DataStream::_BlocksNeeded(): in case of double or triple indirects,
compute the additional blocks needed using the difference between the
old and new indirects blocks. This was resulting in a bad NumBlocks on an inode
* BitmapBlock: mark methods missed an iteration when the startingBit was zero.
Some blocks were then allocated two times (at most 32 for each allocation run).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@38950 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch from Janito Ferreira Filho (aka jjvf): this is the result of his GSoC assignment "Implement ext2/3 Read and Write Support for Haiku".
* Tested and checked most features and fs operations, while passing successfully the Linux fsck. Though the implementation still needs more testing and is to be used with
caution, it's better in my mind to have the code committed now given the size of the patch.
* Code style isn't extensively checked but is mostly OK. Code review is welcome.
Some notes from Janito:
* Sparse files aren't supported and hard links aren't supported. Write attributes methods aren't activated nor tested.
* Journaling needs more testing to make sure it behaves in a compatible way to Ext3, and support for the different modes hasn't been implemented (due to the block
and file cache incompatibility). Correct revoke management is also lacking, as is proper management of the superblock state and copies and block group copies.
* The code is partly based and inspired by the BFS implementation. Author information might need to be fixed.

I'd like to congratulate and thank Janito for his hard work to bring the implementation to the current state. I hope he'll keep on maintaining it and become a regular
contributor/committer.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@38573 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch from Janito Ferreira Filho with fixes by myself: Ext3 Indexed Directory Lookup (as part of GSOC).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37295 a95241bf-73f2-0310-859d-f6bbb57e9c96

- cleanup,
- fix copyright,
- disable tracing,
- add support for reading attributes. The current naming is not final as I'm actually experimenting with it.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36152 a95241bf-73f2-0310-859d-f6bbb57e9c96

Partial implementation for attribute support. Currently listattr successfully lists attributes on files.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36148 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Also the indirect array indices can point to a sparse point if they are 0;
this fixes problems with large files with sparse ranges (for example, Haiku
images).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@32131 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Changed to use Volume::InodeSize() over sizeof(ext2_inode) as described
in ticket #2707. Thanks Adrian!


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@28536 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Seems like GCC4 does not like declaring structures inside unnamed unions;
that looks like a compiler bug to me, though.
* Pulled struct data_stream out of the union (now called ext2_data_stream),
based on a patch by Maurice, thanks!


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26214 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Fixed wrong group block offset computation: the
ext2_super_block::first_data_block must be used; the superblock offset
doesn't matter if the block size is greater than 2KB.
* Fixed block access beyond the 4GB limit - we need to cast the block to
a 64 bit type in ext2_get_file_map() before shifting it.
* Tested with various larger ext3 disks, and everything seems to work fine.
* Added some optional debug output.
* Minor cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26213 a95241bf-73f2-0310-859d-f6bbb57e9c96

gcc4 build fix


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26204 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Fixed a pretty dumb copy&paste error that prevented (/double/triple) indirect
blocks from working.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26189 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added support for files larger than 4GB - not yet tested, though.
* The stat::st_nlink value is now correctly reported.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26188 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Implemented a read-only ext2 file system.
* It's not yet complete, doesn't support some ext2 stuff (like files over 4 GB),
and might have some other bugs (I only tested it with a single 20 MB ext2
image).
* To have a read/write ext2 file system, it would probably make more sense to
port GNU sources (like ext2fs lib), and use that. But a small read-only ext2
file sytem doesn't hurt, I think, and I don't know if ext2fs lib would be
feasible for kernel use (porting the file system from Linux directly would
also be an alternative, but probably more work).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26187 a95241bf-73f2-0310-859d-f6bbb57e9c96

vfs/{b|btr|package|b}fs/ext2/exfat: common access check.

* Added VFS helper function check_access_permissions() that combines
several partially correct versions to the one true version (tm).
* All but BFS (since recently) missed the S_IXOTH for root on directories,
and all but packagefs missed proper group handling.

ext2: Fix enabling/disabling the file cache

* Inode:
- Rename {Enable,Disable}FileCache() to {Create,Delete}FileCache()
and IsFileCacheDisabled() to HasFileCache(), since that is what they
actually do. DeleteFileCache() now also sets the attributes to NULL,
which makes fCached superfluous.
- Introduce {Enable,Disable}FileCache(), which actually enable/disable
the file cache. Use those methods for handling O_NOCACHE.
* ext2_free_cookie(): Reenable the file cache in case of O_NOCACHE.

Fixes crash when O_NOCACHE was used, since the file cache was deleted
without clearing the attribute and Inode::ReadAt() would use the deleted
object afterward.

ext2: access the parent variable once checked it's non null

* cleanup
* add some inode flags we don't use

ext2: access the parent variable once checked it's non null

* cleanup
* add some inode flags we don't use

ext2: fixes 64 bit warnings

* added flex group, dirnlink and gdtchecksum features for ext4.
* I reused crc_table.cpp from the UDF filesystem and switched it to have the reversed algorithm,
then generated the table in CRCTable.cpp
* added a binary search for extent tree leaves.
* fixed a check in InodeAllocator::New().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40129 a95241bf-73f2-0310-859d-f6bbb57e9c96

resource leaks CID 5711, 5712, 5713, 5714


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39902 a95241bf-73f2-0310-859d-f6bbb57e9c96

* added fsblock_t and fileblock_t definitions, used them when needed.
* validate fBlockShift in superblock.
* Volume::AllocateBlocks() now uses an absolute fsblock instead of a blockgroup related fsblock.
* AllocationBlockGroup now provides absolute fsblock values.
* added support for extents feature: ExtentStream class is the equivalent for DataStream class for extent operations.
The extent tree implementation is very basic, should work for normal growing/shrinking
operations, but not for sparse files. When enlarging a file and extent tree is full,
the root is moved in a new block and a new level is added on top. Extents can usually
be extended when adjacent blocks are allocated. Shrinking happens by removing leafs
one after another.
* removed empty IndexedDirectoryIterator.*


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39886 a95241bf-73f2-0310-859d-f6bbb57e9c96

moved FindBlock() from Inode to DataStream


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39820 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Applied patch by Rohit Yadav that fixes #6750 for ext2. Thanks a lot!
* Fixed warning in Inode::WriteAt(), but left warnings in kernel_interface.cpp
in there; I guess that's work in progress (although having the io hook would
be nice).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39379 a95241bf-73f2-0310-859d-f6bbb57e9c96

* more cleanup
* more fixes in BitmapBlock::FindNextMarked() and BitmapBlock::FindPreviousMarked()


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39242 a95241bf-73f2-0310-859d-f6bbb57e9c96

* switched more uint32 to off_t for 64bit support
* added error messages in InodeAllocator, Inode
* if BlockAllocator can't initialize, don't fail completely but switch to readonly
* fixed a bug in FindNextMarked() for bitmaps with a length non multiple of 32
* Inode::FindBlock() now returns an optional block_run length, useful for get_file_map()
* added flag for Inode for extents


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39234 a95241bf-73f2-0310-859d-f6bbb57e9c96

* added some support for 64bit feature: extended struct ext2_block_group, block number types changed from uint32 to off_t
* added error traces, asserts
* BitmapBlock::CheckUnmarked() and CheckMarked() computed a wrong remainingBits and mask


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39203 a95241bf-73f2-0310-859d-f6bbb57e9c96

added some definitions for readonly features
we already support the extra isize feature, so it's now added
removed unused Inode::AttributeBlockReadAt()
group_descriptor_size was previously a reserved field in the super block


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39125 a95241bf-73f2-0310-859d-f6bbb57e9c96

added large files feature support


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39120 a95241bf-73f2-0310-859d-f6bbb57e9c96

ext3 inodes whose size is at least 256 bytes can also contain attributes (like smalldata for bfs).
We now use the Attribute class from bfs (instead of AttributeIterator) to manage small data and block attributes, though it's still readonly.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@39036 a95241bf-73f2-0310-859d-f6bbb57e9c96

* added support for extra inode attributes in volumes with an inodesize of 256 or more:
this brings nanoseconds for access, change and modification times, and also brings creation time.
* switched off some debug output
* HTreeEntryIterator: fCount can equal fLimit.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@38964 a95241bf-73f2-0310-859d-f6bbb57e9c96

* added support for huge_file feature, untested (need a >2TB file)
* actually checks for readonly features if mounting read-write
* don't use a journal when mounting readonly
* DataStream::_BlocksNeeded(): in case of double or triple indirects,
compute the additional blocks needed using the difference between the
old and new indirects blocks. This was resulting in a bad NumBlocks on an inode
* BitmapBlock: mark methods missed an iteration when the startingBit was zero.
Some blocks were then allocated two times (at most 32 for each allocation run).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@38950 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch from Janito Ferreira Filho (aka jjvf): this is the result of his GSoC assignment "Implement ext2/3 Read and Write Support for Haiku".
* Tested and checked most features and fs operations, while passing successfully the Linux fsck. Though the implementation still needs more testing and is to be used with
caution, it's better in my mind to have the code committed now given the size of the patch.
* Code style isn't extensively checked but is mostly OK. Code review is welcome.
Some notes from Janito:
* Sparse files aren't supported and hard links aren't supported. Write attributes methods aren't activated nor tested.
* Journaling needs more testing to make sure it behaves in a compatible way to Ext3, and support for the different modes hasn't been implemented (due to the block
and file cache incompatibility). Correct revoke management is also lacking, as is proper management of the superblock state and copies and block group copies.
* The code is partly based and inspired by the BFS implementation. Author information might need to be fixed.

I'd like to congratulate and thank Janito for his hard work to bring the implementation to the current state. I hope he'll keep on maintaining it and become a regular
contributor/committer.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@38573 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch from Janito Ferreira Filho with fixes by myself: Ext3 Indexed Directory Lookup (as part of GSOC).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37295 a95241bf-73f2-0310-859d-f6bbb57e9c96

- cleanup,
- fix copyright,
- disable tracing,
- add support for reading attributes. The current naming is not final as I'm actually experimenting with it.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36152 a95241bf-73f2-0310-859d-f6bbb57e9c96

Partial implementation for attribute support. Currently listattr successfully lists attributes on files.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36148 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Also the indirect array indices can point to a sparse point if they are 0;
this fixes problems with large files with sparse ranges (for example, Haiku
images).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@32131 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Changed to use Volume::InodeSize() over sizeof(ext2_inode) as described
in ticket #2707. Thanks Adrian!


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@28536 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Seems like GCC4 does not like declaring structures inside unnamed unions;
that looks like a compiler bug to me, though.
* Pulled struct data_stream out of the union (now called ext2_data_stream),
based on a patch by Maurice, thanks!


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26214 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Fixed wrong group block offset computation: the
ext2_super_block::first_data_block must be used; the superblock offset
doesn't matter if the block size is greater than 2KB.
* Fixed block access beyond the 4GB limit - we need to cast the block to
a 64 bit type in ext2_get_file_map() before shifting it.
* Tested with various larger ext3 disks, and everything seems to work fine.
* Added some optional debug output.
* Minor cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26213 a95241bf-73f2-0310-859d-f6bbb57e9c96

gcc4 build fix


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26204 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Fixed a pretty dumb copy&paste error that prevented (/double/triple) indirect
blocks from working.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26189 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added support for files larger than 4GB - not yet tested, though.
* The stat::st_nlink value is now correctly reported.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26188 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Implemented a read-only ext2 file system.
* It's not yet complete, doesn't support some ext2 stuff (like files over 4 GB),
and might have some other bugs (I only tested it with a single 20 MB ext2
image).
* To have a read/write ext2 file system, it would probably make more sense to
port GNU sources (like ext2fs lib), and use that. But a small read-only ext2
file sytem doesn't hurt, I think, and I don't know if ext2fs lib would be
feasible for kernel use (porting the file system from Linux directly would
also be an alternative, but probably more work).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26187 a95241bf-73f2-0310-859d-f6bbb57e9c96