kernel/io_request: Add assertion in NotifyFinished() that fStatus is set.

IORequest: Refactor IOOperation transferred-bytes and status accounting.

Until the introduction of the nvme_disk driver, these classes were
mostly only used directly by the IO scheduler, and then a few direct
usages of IOOperation itself in the individual disk drivers; so
API confusions were easily missed.

When writing the nvme_disk driver's IORequest support, however, it
became readily apparent that there were some pretty bad confusions
around transferred-bytes accounting in IOOperation. This commit
attempts to resolve all of those.

There are two basic changes here:

1. Move transferred-bytes accounting into IOOperation::SetStatus.

The "TransferredBytes" field of IOOperation is against the *original*
range, not the actual operation's range (which will be wider, due to
bouncing, etc.), and furthermore only applies to the actual content
of the request (and not e.g. to a read half of a bounced write.)

These two facts meant that determining what value to pass to
SetTransferredBytes was not trivial, and was easy to get wrong.
I recall messing that up when working on nvme_disk multiple times
before reading the API carefully.

2. Do not pass redundant values to IORequest::OperationFinished.

All of the values here can be derived (albeit indirectly) from the
IOOperation, and all consumers of this API basically did just that.
Rather than make them do it, make the IORequest take care of
computing all of those values itself.

Change-Id: Ic9ae29e1100319e5b7647647c4db7e5aad4d125e

IORequest: Add an assertion in SetTransferredBytes.

kernel: Add mechanism in IOBuffer & IORequest to clamp the last iovec.

This is needed by CreateSubRequest, which creates a sub-request
using some subset of the passed IO vectors. In the case that the
last vector will not be fully used, we need to clamp its size
in the sub-request to the remaining length.

do_iterative_fd_io, used by vfs_read_pages (which is in turn
used by the file cache) used this to split up requests
into their constituent block-run requests. So, previously,
the invalid IO requests created by this could, under one possible
interpretation, overwrite valid file data and cause disk corruption.

It is slightly unfortunate that generic_size_t has no unsigned
equivalent, so we are left with 0 as the magic number here,
instead of -1. However, the passed "length" remains unchanged,
so any callers that pass the wrong value for lastVecSize
will be trapped by the assert added in the previous commit

Fixes #15912 (the assert added in the previous commit),
and potentially disk corruption caused by this.

kernel: Add assert in IOBuffer::SetVecs() that the passed length is the actual length.

These are, at present, triggered on rare invocations of vfs_read_pages.

See #15912.

device_manager: IORequest: use user_memcpy() to access a user buffer.

* should fix #14164, but the case can't be reproduced easily it seems.

Change-Id: I636db93cc1efe2784b2530dced6a900fbfe3dce8

IORequest: NULL check does not belong in method body.

Found by Clang's -Wtautological-undefined-compare.

Kernel IOBuffer: Fixed potential memory leak.

* IOBuffer::FreeVirtualVecCookie() did not put the last physical page
back. This was only an issue in case GetNextVirtualVec() wasn't
iterated through the end, for example in case of an error.
* Fixed the condition when to put back a physical page; the current
solution will also work with the generic page mapper implementation.
* This fixes the low hanging fruits of Ingo's comment in #5777.

IORequest::Init(): Assert offset >= 0

Just to make sure we catch sparse file vectors that shouldn't be passed
here.

IORequest: Add ClearData()

IORequest::_CopyData(): Small optimization

If we're running in the context of the user team the I/O buffer belongs
to, we can use _CopySimple() as well.

Fixed compilation of VFS/IORequest trace messages.

Changed {,u}int64 to be long rather than long long on x86_64.

More 64-bit compilation/safety fixes.

Merged the signals branch into trunk, with these changes:
* The team and thread kernel structures have been renamed to Team and Thread
respectively and moved into the new BKernel namespace.
* Several (kernel add-on) sources have been converted from C to C++ since
private kernel headers are included that are no longer C compatible.

Changes after merging:
* Fixed gcc 2 build (warnings mainly in the scary firewire bus manager).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40196 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fixed build with debug output enabled.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37246 a95241bf-73f2-0310-859d-f6bbb57e9c96

IORequest::_Copy*(): Resolved TODO: Don't cast the generic_addr_t to void*
anymore as that truncates physical addresses when PAE is enabled.
Now, if a 4 GB physical address limit is forced in DMAResource, the system
continues to work fine when the physical memory > 4 GB is used. Otherwise it
hangs or crashes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37134 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fixed more address types related issues. Mostly printf() or comparison
warnings, but also some oversights from earlier changes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37000 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Introduced type generic_io_vec, which is similar to iovec, but uses types
that are wide enough for both virtual and physical addresses.
* DMABuffer, IORequest, IOScheduler,... and code using them: Use
generic_io_vec and generic_{addr,size}_t where necessary.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36997 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Changed physical_entry::{address,size} to phys_{addr,size}_t and changed
map_physical_memory()'s physicalAddress parameter type from void* to
phys_addr_t. This breaks source compatibility, but -- as long as
phys_{addr,size}_t remain 32 bit wide -- keeps binary compatibility with
BeOS.
* Adjusted all code using the affected interfaces (Oh what fun!). Added a few
TODOs in places where the wrong types (e.g. void* for physical addresses
are used). Looks like quite a few drivers aren't 64 bit safe and others
will break with PAE.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36960 a95241bf-73f2-0310-859d-f6bbb57e9c96

IORequest::NotifyFinished(): Assert that the request hasn't been finished
before.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36594 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added several TODOs for problems of the IOBuffer virtual vecs code.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35509 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Introduced {malloc,memalign,free}_etc() which take an additional "flags"
argument. They replace the previous special-purpose allocation functions
(malloc_nogrow(), vip_io_request_malloc()).
* Moved the I/O VIP heap to heap.cpp accordingly.
* Added quite a bit of passing around of allocation flags in the VM,
particularly in the VM*AddressSpace classes.
* Fixed IOBuffer::GetNextVirtualVec(): It was ignoring the VIP flag and always
allocated on the normal heap.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35316 a95241bf-73f2-0310-859d-f6bbb57e9c96

* arch_debug_get_stack_trace():
- Replaced the "userOnly" parameter by a "flags" parameter, that allows to
specify kernel and userland stack traces individually.
- x86, m68k: Don't always skip the first frame as that prevents the caller
from being able to record its own address.
* capture_tracing_stack_trace(): Replaced the "userOnly" parameter by
"kernelOnly", since one is probably always interested in the kernel stack
trace, but might not want the userland stack trace.
* Added stack trace support for VM cache kernel tracing.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34742 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fixed a stupid race condition between IORequest finishing and
IORequest::Wait(). Wait() immediately returned when IsFinished() returned
true, but this is the case as soon as the last IOOperation has finished. The
I/O scheduler is not done with the request at this point, though, since it
will still be sitting in at least one of three doubly linked lists. Since the
usual procedure to issue synchronous I/O requests is to create an IORequest
on the stack, pass it to the I/O scheduler, and Wait() on it, Wait()
returning early might cause the IORequest object to be destroyed while it is
still in use, leading to invalid memory access in the I/O scheduler,
corruption of its list structures, as well as later corruption of the issuing
thread's stack.
Related tickets:
* #4431: The request issuing thread returned and already deleted the area the
request was writing to before NotifyFinished() was called.
* #3048, #4883: Caused by the on stack IORequest being overwritten with other
data while being handled by the I/O scheduler thread.
* #4517: Hard to say, but I've seen a such a problem too, after a thread
scheduling related change. An explanation would be a list structure
corruption in the I/O scheduler causing an infinite loop with disabled
interrupts.
* #2845, #3428, #3429: The block notifier/writer is I/O heavy and as such
quite likely to run into the stack corruption issue.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34655 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved the VM headers into subdirectory vm/.
* Renamed vm_cache.h/vm_address_space.h to VMCache.h/VMAddressSpace.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34449 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved VMAddressSpace definition to vm_address_space.h.
* "Classified" VMAddressSpace, i.e. turned the vm_address_space_*() functions
into methods, made all attributes (but "areas") private, and added
accessors.
* Also turned the vm.cpp functions vm_area_lookup() and
remove_area_from_address_space() into VMAddressSpace methods. The rest of
the area management functionality will follow soon.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34447 a95241bf-73f2-0310-859d-f6bbb57e9c96

IOBuffer::SetVecs(): fUser was initialized incorrectly, if the vecs referred
to physical memory whose address would accidentally satisfy the
IS_USER_ADDRESS() check.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34136 a95241bf-73f2-0310-859d-f6bbb57e9c96

Removed obsolete uncommented function stubs.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34122 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Implement per-CPU heaps. They only get enabled in case there's enough memory.
* Allow an allocator to be created on the heap to allow for non-locked
allocators to be created.
* Some cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@33721 a95241bf-73f2-0310-859d-f6bbb57e9c96

Provide a way to directly request virtual vecs from an IOBuffer. If the buffer
is virtual already it just returns the vecs directly, if it is physical it takes
over the task of virtualizing the vecs either using vm_map_physical_memory_vecs,
if there are multiple vecs or more than one page, or falls back to page wise
mapping if mapping fails or is not needed. In the best case, scattered physical
pages are mapped into one linear virtual buffer so that subsystems operating on
virtual memory only get a single vector and can then burst read/write.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@33524 a95241bf-73f2-0310-859d-f6bbb57e9c96

Give an IORequest user the possibility to suppress child finish notifications.
This allows for synchronous uses where subrequests are forked off and waited on.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@32936 a95241bf-73f2-0310-859d-f6bbb57e9c96

When deleting all subrequests the pending children count should probably be
reset as well.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@32935 a95241bf-73f2-0310-859d-f6bbb57e9c96

bonefish + axeld:
* Reverted r31809 as it introduced a race condition; if the I/O request had been
notified, it could already been deleted at that point.
* Instead, we need to notify the request in each file system/driver that uses
it. Added new notify_io_request() function that does that exactly.
* Added a TODO comment to the userlandfs where the request notification needs
a bit more thought.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@31903 a95241bf-73f2-0310-859d-f6bbb57e9c96

Removed unintentional thought mumbling left-over that would fortunately cause
no harm.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@30002 a95241bf-73f2-0310-859d-f6bbb57e9c96

IOOperation::Finish():
* Fixed the read with bounce buffer case. When skipping a partial bounce
buffer before the part we're interested in, we forgot to update "offset".
* Added some more comments for readability.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@29999 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Renamed src/system/kernel/device_manager/io_requests.{h,cpp} to
IORequest.{h,cpp}.
* Introduced public <io_requests.h> header. Currently it only declares the
single function BFS uses.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@29446 a95241bf-73f2-0310-859d-f6bbb57e9c96

IORequest::Init(): Assert offset >= 0

Just to make sure we catch sparse file vectors that shouldn't be passed
here.

IORequest: Add ClearData()

IORequest::_CopyData(): Small optimization

If we're running in the context of the user team the I/O buffer belongs
to, we can use _CopySimple() as well.

Fixed compilation of VFS/IORequest trace messages.

Changed {,u}int64 to be long rather than long long on x86_64.

More 64-bit compilation/safety fixes.

Merged the signals branch into trunk, with these changes:
* The team and thread kernel structures have been renamed to Team and Thread
respectively and moved into the new BKernel namespace.
* Several (kernel add-on) sources have been converted from C to C++ since
private kernel headers are included that are no longer C compatible.

Changes after merging:
* Fixed gcc 2 build (warnings mainly in the scary firewire bus manager).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40196 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fixed build with debug output enabled.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37246 a95241bf-73f2-0310-859d-f6bbb57e9c96

IORequest::_Copy*(): Resolved TODO: Don't cast the generic_addr_t to void*
anymore as that truncates physical addresses when PAE is enabled.
Now, if a 4 GB physical address limit is forced in DMAResource, the system
continues to work fine when the physical memory > 4 GB is used. Otherwise it
hangs or crashes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37134 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fixed more address types related issues. Mostly printf() or comparison
warnings, but also some oversights from earlier changes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37000 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Introduced type generic_io_vec, which is similar to iovec, but uses types
that are wide enough for both virtual and physical addresses.
* DMABuffer, IORequest, IOScheduler,... and code using them: Use
generic_io_vec and generic_{addr,size}_t where necessary.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36997 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Changed physical_entry::{address,size} to phys_{addr,size}_t and changed
map_physical_memory()'s physicalAddress parameter type from void* to
phys_addr_t. This breaks source compatibility, but -- as long as
phys_{addr,size}_t remain 32 bit wide -- keeps binary compatibility with
BeOS.
* Adjusted all code using the affected interfaces (Oh what fun!). Added a few
TODOs in places where the wrong types (e.g. void* for physical addresses
are used). Looks like quite a few drivers aren't 64 bit safe and others
will break with PAE.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36960 a95241bf-73f2-0310-859d-f6bbb57e9c96

IORequest::NotifyFinished(): Assert that the request hasn't been finished
before.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36594 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added several TODOs for problems of the IOBuffer virtual vecs code.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35509 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Introduced {malloc,memalign,free}_etc() which take an additional "flags"
argument. They replace the previous special-purpose allocation functions
(malloc_nogrow(), vip_io_request_malloc()).
* Moved the I/O VIP heap to heap.cpp accordingly.
* Added quite a bit of passing around of allocation flags in the VM,
particularly in the VM*AddressSpace classes.
* Fixed IOBuffer::GetNextVirtualVec(): It was ignoring the VIP flag and always
allocated on the normal heap.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35316 a95241bf-73f2-0310-859d-f6bbb57e9c96

* arch_debug_get_stack_trace():
- Replaced the "userOnly" parameter by a "flags" parameter, that allows to
specify kernel and userland stack traces individually.
- x86, m68k: Don't always skip the first frame as that prevents the caller
from being able to record its own address.
* capture_tracing_stack_trace(): Replaced the "userOnly" parameter by
"kernelOnly", since one is probably always interested in the kernel stack
trace, but might not want the userland stack trace.
* Added stack trace support for VM cache kernel tracing.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34742 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fixed a stupid race condition between IORequest finishing and
IORequest::Wait(). Wait() immediately returned when IsFinished() returned
true, but this is the case as soon as the last IOOperation has finished. The
I/O scheduler is not done with the request at this point, though, since it
will still be sitting in at least one of three doubly linked lists. Since the
usual procedure to issue synchronous I/O requests is to create an IORequest
on the stack, pass it to the I/O scheduler, and Wait() on it, Wait()
returning early might cause the IORequest object to be destroyed while it is
still in use, leading to invalid memory access in the I/O scheduler,
corruption of its list structures, as well as later corruption of the issuing
thread's stack.
Related tickets:
* #4431: The request issuing thread returned and already deleted the area the
request was writing to before NotifyFinished() was called.
* #3048, #4883: Caused by the on stack IORequest being overwritten with other
data while being handled by the I/O scheduler thread.
* #4517: Hard to say, but I've seen a such a problem too, after a thread
scheduling related change. An explanation would be a list structure
corruption in the I/O scheduler causing an infinite loop with disabled
interrupts.
* #2845, #3428, #3429: The block notifier/writer is I/O heavy and as such
quite likely to run into the stack corruption issue.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34655 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved the VM headers into subdirectory vm/.
* Renamed vm_cache.h/vm_address_space.h to VMCache.h/VMAddressSpace.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34449 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved VMAddressSpace definition to vm_address_space.h.
* "Classified" VMAddressSpace, i.e. turned the vm_address_space_*() functions
into methods, made all attributes (but "areas") private, and added
accessors.
* Also turned the vm.cpp functions vm_area_lookup() and
remove_area_from_address_space() into VMAddressSpace methods. The rest of
the area management functionality will follow soon.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34447 a95241bf-73f2-0310-859d-f6bbb57e9c96

IOBuffer::SetVecs(): fUser was initialized incorrectly, if the vecs referred
to physical memory whose address would accidentally satisfy the
IS_USER_ADDRESS() check.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34136 a95241bf-73f2-0310-859d-f6bbb57e9c96

Removed obsolete uncommented function stubs.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34122 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Implement per-CPU heaps. They only get enabled in case there's enough memory.
* Allow an allocator to be created on the heap to allow for non-locked
allocators to be created.
* Some cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@33721 a95241bf-73f2-0310-859d-f6bbb57e9c96

Provide a way to directly request virtual vecs from an IOBuffer. If the buffer
is virtual already it just returns the vecs directly, if it is physical it takes
over the task of virtualizing the vecs either using vm_map_physical_memory_vecs,
if there are multiple vecs or more than one page, or falls back to page wise
mapping if mapping fails or is not needed. In the best case, scattered physical
pages are mapped into one linear virtual buffer so that subsystems operating on
virtual memory only get a single vector and can then burst read/write.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@33524 a95241bf-73f2-0310-859d-f6bbb57e9c96

Give an IORequest user the possibility to suppress child finish notifications.
This allows for synchronous uses where subrequests are forked off and waited on.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@32936 a95241bf-73f2-0310-859d-f6bbb57e9c96

When deleting all subrequests the pending children count should probably be
reset as well.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@32935 a95241bf-73f2-0310-859d-f6bbb57e9c96

bonefish + axeld:
* Reverted r31809 as it introduced a race condition; if the I/O request had been
notified, it could already been deleted at that point.
* Instead, we need to notify the request in each file system/driver that uses
it. Added new notify_io_request() function that does that exactly.
* Added a TODO comment to the userlandfs where the request notification needs
a bit more thought.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@31903 a95241bf-73f2-0310-859d-f6bbb57e9c96

Removed unintentional thought mumbling left-over that would fortunately cause
no harm.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@30002 a95241bf-73f2-0310-859d-f6bbb57e9c96

IOOperation::Finish():
* Fixed the read with bounce buffer case. When skipping a partial bounce
buffer before the part we're interested in, we forgot to update "offset".
* Added some more comments for readability.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@29999 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Renamed src/system/kernel/device_manager/io_requests.{h,cpp} to
IORequest.{h,cpp}.
* Introduced public <io_requests.h> header. Currently it only declares the
single function BFS uses.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@29446 a95241bf-73f2-0310-859d-f6bbb57e9c96