refactor: Swap %Ld for %lld in all format usages

* %Ld is an undocumented alias for %lld in glibc.
* muslc doesn't implement it for this reason.
* While we will likely never drop %Ld support,
lets clean house and set a better example.

Change-Id: Id46dad3104abae483e80cc5c05d1464d3ecd8030
Reviewed-on: https://review.haiku-os.org/c/haiku/+/6636
Reviewed-by: Adrien Destugues <pulkomandy@pulkomandy.tk>
Reviewed-by: Alex von Gluck IV <kallisti5@unixzen.com>
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>

VMCache: Use THREAD_BLOCK_TYPE_OTHER_OBJECT.

This would have assisted in debugging #18390.

kernel/vm: Specifiy the page as the object being waited on in VMCache.

I got a strange whole-system hang in file_cache_resize into this
function. Possibly the page was spuriously busy, perhaps it wasn't,
but not having function arguments on x86_64 in stack traces, I was
unable to deduce what page was actually being waited on.

In case it happens again, this should allow it to be debugged further
by placing the address of the vm_page structure in the thread wait
informations. (A string is not very useful here anyway.)

kernel/vm: Cast temporary to uint32.

It is defined as a uint32:1, which apparently becomes an "int" on
newer GCC versions, which thus triggers a -Werror=format. So,
convert it explicitly in order to prevent the error.

Add more missing headers following previous commits.

These also are only a problem on riscv64; I guess thread.h
must not be included in some arch header there.

kernel/vm: Add discard_address_range that discards pages.

Pages in the given range are unmapped and freed without getting written
back anywhere. It can be used whenever a caller does not care about the
data in the given range anymore and wants to reduce page pressure.

Change-Id: I8bcce68fab278efef710d3714677e1d463504a56
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2843
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

kernel/vm: Fix build with swap support disabled.

The rename of the system_info members was missed in
d02aaee17e007631fcfa91a012ec7b6386927012 (part of the scheduler branch
merge of hrev46690). The unguarded object_cache was introduced even
earlier as part of hrev43133.

VMCache: Factor out a _FreePageRange method.

The code in the Resize and Rebase methods was identical except for the
iterator.

Change-Id: I9f6b3c2c09af0c26778215bd627fed030c4d46f1
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2835
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

kernel/vm: Implement swap adoption for cut_area middle case.

Rename MovePageRange to Adopt and group it with Resize/Rebase as it
covers the third, middle cut case.

Implement VMAnonymousCache::Adopt() to actually adopt swap pages. This
has to recreate swap blocks instead of taking them over from the source
cache as the cut offset or base offset between the caches may not be
swap block aligned. This means that adoption may fail due to memory
shortage in allocating the swap blocks.

For the middle cut case it is therefore now possible to have the adopt
fail in which case the previous cache restore logic is applied. Since
the readoption of the pages from the second cache can fail for the same
reason, there is a slight chance that we can't restore and lose pages.
For now, just panic in such a case and add a TODO to free memory and
retry.

Change-Id: I9a661f00c8f03bbbea2fe6dee90371c68d7951e6
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2588
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

Resize caches in all cases when cutting areas

* Adds VMCache::MovePageRange() and VMCache::Rebase() to facilitate
this.

Applied on top of hrev45098 and rebased with the hrev45564 page_num_t to
off_t change included.

Change-Id: Ie61bf43696783e3376fb4144ddced3781aa092ba
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2581
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

Remove useless includes of khash.h

* These files were already converted to BOpenHashTable.
* For #9552.

Fix various 64 bit related warnings

Mostly printf() format strings and signed-unsigned comparisons.
Fixes the x86_64 build.

kernel: Add thread_unblock() and use it where possible

Print page and cache using initial commands in panic message.

VMCache: Fix broken trace statements

Fixed tracing printf formats in VM code.

Changed {,u}int64 to be long rather than long long on x86_64.

More 64-bit compilation/safety fixes.

mmlr (distracted) + bonefish:
* Turn VMCache::consumers C list into a DoublyLinkedList.
* Use object caches for the different VMCache types and the VMCacheRefs.
The purpose is to reduce slab area fragmentation.
* Requires the introduction of a pure virtual VMCache::DeleteObject()
method, implemented in the derived classes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43133 a95241bf-73f2-0310-859d-f6bbb57e9c96

Tiny style cleanup. No functional change.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43044 a95241bf-73f2-0310-859d-f6bbb57e9c96

Merged signals-merge branch into trunk with the following changes:
* Reorganized the kernel locking related to threads and teams.
* We now discriminate correctly between process and thread signals. Signal
handlers have been moved to teams. Fixes #5679.
* Implemented real-time signal support, including signal queuing, SA_SIGINFO
support, sigqueue(), sigwaitinfo(), sigtimedwait(), waitid(), and the addition
of the real-time signal range. Closes #1935 and #2695.
* Gave SIGBUS a separate signal number. Fixes #6704.
* Implemented <time.h> clock and timer support, and fixed/completed alarm() and
[set]itimer(). Closes #5682.
* Implemented support for thread cancellation. Closes #5686.
* Moved send_signal() from <signal.h> to <OS.h>. Fixes #7554.
* Lots over smaller more or less related changes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@42116 a95241bf-73f2-0310-859d-f6bbb57e9c96

Merged the signals branch into trunk, with these changes:
* The team and thread kernel structures have been renamed to Team and Thread
respectively and moved into the new BKernel namespace.
* Several (kernel add-on) sources have been converted from C to C++ since
private kernel headers are included that are no longer C compatible.

Changes after merging:
* Fixed gcc 2 build (warnings mainly in the scary firewire bus manager).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40196 a95241bf-73f2-0310-859d-f6bbb57e9c96

* VMCache::Unlock(): Renamed local variable consumerLocked to avoid shadowing
the parameter (CID 5329).
* _MergeWithOnlyConsumer(): Removed the somewhat weird consumerLocked
parameter. The caller can unlock itself, if desired. Improves Unlock()
readability.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40084 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved the vm_page initialization from vm_page.cpp:vm_page_init() to the new
vm_page::Init().
* Made vm_page::wired_count private and added accessor methods.
* Added VMCache::fWiredPagesCount (the number of wired pages the cache
contains) and accessor methods.
* Made more use of vm_page::IsMapped().
* vm_copy_on_write_area(): Added vm_page_reservation* parameter that can be
used to request a special handling for wired pages. If given the wired pages
are replaced by copies and the original pages are moved to the upper cache.
* vm_copy_area():
- We don't need to do any wired ranges handling, if the source area is a
B_SHARED_AREA, since we don't touch the area's mappings in this case.
- We no longer wait for wired ranges of the concerned areas to disappear.
Instead we use the new vm_copy_on_write_area() feature and just let it
copy the wired pages. This fixes #6288, an issue introduced with the use
of user mutexes in libroot: When executing multiple concurrent fork()s all
but the first one would wait on the fork mutex, which (being a user mutex)
would wire a page that the vm_copy_area() of the first fork() would wait
for.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37460 a95241bf-73f2-0310-859d-f6bbb57e9c96

Removed never read VMCache::scan_skip.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37195 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Renamed cache_type_to_string() to vm_cache_type_to_string() and made in
kernel private.
* Moved dumping code from dump_cache() to new VMCache::Dump().
* Override VMCache::Dump() in VMVnodeCache to also print the vnode.
* Removed no longer needed VMCache::GetLock().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37138 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fixed more address types related issues. Mostly printf() or comparison
warnings, but also some oversights from earlier changes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37000 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Introduced type generic_io_vec, which is similar to iovec, but uses types
that are wide enough for both virtual and physical addresses.
* DMABuffer, IORequest, IOScheduler,... and code using them: Use
generic_io_vec and generic_{addr,size}_t where necessary.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36997 a95241bf-73f2-0310-859d-f6bbb57e9c96

* VMCache::Resize(): Corrected TODO comment.
* Changed the semantics of VMCache::HasPage(). It was interpreted
inconsistently by the derived classes. Now it returns whether the backing
store can provide the page (via Read()). The default implementation returns
false. VNodeCache::HasPage() only returns true, if the given offset is
within the cache (i.e. file) bounds. This prevents vm_soft_fault() from
adding clean pages to vnode caches on faults beyond the file bounds.
Probably fixes #5473 -- at least mmap_resize_test behaves correctly, now.
* Removed redundant HasPage() and Fault() overrides in VMCache derived
classes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36374 a95241bf-73f2-0310-859d-f6bbb57e9c96

Made VMCache::Resize() virtual and let VMAnonymousCache override it to free
swap space when the cache shrinks. Currently the implementation stil leaks
swap space of busy pages.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36373 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added TODO.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36236 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added VMCache::DebugHasPage() and DebugLookupPage() for use in the kernel
debugger.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36228 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Made vm_page::state private and added accessor methods.
* Added kernel tracing for page state transitions.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35538 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Removed useless return parameter from vm_remove_all_page_mappings().
* Added vm_clear_page_mapping_accessed_flags() and
vm_remove_all_page_mappings_if_unaccessed(), which combine the functionality
of vm_test_map_activation(), vm_clear_map_flags(), and
vm_remove_all_page_mappings(), thus saving lots of calls to translation map
methods. The backend is the new method
VMTranslationMap::ClearAccessedAndModified().
* Started to make use of the cached page queue and changed the meaning of the
other non-free queues slightly:
- Active queue: Contains mapped pages that have been used recently.
- Inactive queue: Contains mapped pages that have not been used recently. Also
contains unmapped temporary pages.
- Modified queue: Contains unmapped modified pages.
- Cached queue: Contains unmapped unmodified pages (LRU sorted).
Unless we're actually low on memory and actively do paging, modified and
cached queues only contain non-temporary pages. Cached pages are considered
quasi free. They still belong to a cache, but since they are unmodified and
unmapped, they can be freed immediately. And this is what
vm_page_[try_]reserve_pages() do now when there are no more actually free
pages at hand. Essentially this means that pages storing cached file data,
unless mmap()ped, no longer are considered used and don't contribute to page
pressure. Paging will not happen as long there are enough free + cached pages
available.
* Reimplemented the page daemon. It no longer scans all pages, but instead works
the page queues. As long as the free pages situation is harmless, it only
iterates through the active queue and deactivates pages that have not been
used recently. When paging occurs it additionally scans the inactive queue and
frees pages that have not been used recently.
* Changed the page reservation/allocation interface:
vm_page_[try_]reserve_pages(), vm_page_unreserve_pages(), and
vm_page_allocate_page() now take a vm_page_reservation structure pointer.
The reservation functions initialize the structure -- currently consisting
only of a count member for the number of still reserved pages.
vm_page_allocate_page() decrements the count and vm_page_unreserve_pages()
unreserves the remaining pages (if any). Advantages are that reservation/
unreservation mismatches cannot occur anymore, that vm_page_allocate_page()
can verify that the caller has indeed a reserved page left, and that there's
no unnecessary pressure on the free page pool anymore. The only disadvantage
is that the vm_page_reservation object needs to be passed around a bit.
* Reworked the page reservation implementation:
- Got rid of sSystemReservedPages and sPageDeficit. Instead
sUnreservedFreePages now actually contains the number of free pages that
have not yet been reserved (it cannot become negative anymore) and the new
sUnsatisfiedPageReservations contains the number of pages that are still
needed for reservation.
- Threads waiting for reservations do now add themselves to a waiter queue,
which is ordered by descending priority (VM priority and thread priority).
High priority waiters are served first when pages become available.
Fixes #5328.
* cache_prefetch_vnode(): Would reserve one less page than allocated later, if
the size wasn't page aligned.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35393 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Removed the page state PAGE_STATE_BUSY and instead introduced a vm_page::busy
flag. The obvious advantage is that one can still see what state a page is in
and even move it between states while being marked busy.
* Removed the vm_page::is_dummy flag. Instead we mark marker pages busy, which
in all cases has the same effect. Introduced a vm_page_is_dummy() that can
still check whether a given page is a dummy page.
* vm_page_unreserve_pages(): Before adding to the system reserve make sure
sUnreservedFreePages is non-negative. Otherwise we'd make nonexisting pages
available for allocation. steal_pages() still has the same problem and it
can't be solved that easily.
* map_page(): No longer changes the page state/mark the page unbusy. That's the
caller's responsibility.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35331 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Introduced {malloc,memalign,free}_etc() which take an additional "flags"
argument. They replace the previous special-purpose allocation functions
(malloc_nogrow(), vip_io_request_malloc()).
* Moved the I/O VIP heap to heap.cpp accordingly.
* Added quite a bit of passing around of allocation flags in the VM,
particularly in the VM*AddressSpace classes.
* Fixed IOBuffer::GetNextVirtualVec(): It was ignoring the VIP flag and always
allocated on the normal heap.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35316 a95241bf-73f2-0310-859d-f6bbb57e9c96

* The system now holds back a small reserve of committable memory and pages. The
memory and page reservation functions have a new "priority" parameter that
indicates how deep the function may tap into that reserve. The currently
existing priority levels are "user", "system", and "VIP". The idea is that
user programs should never be able to cause a state that gets the kernel into
trouble due to heavy battling for memory. The "VIP" level (not really used
yet) is intended for allocations that are required to free memory eventually
(in the page writer). More levels are thinkable in the future, like "user real
time" or "user system server".
* Added "priority" parameters to several VMCache methods.
* Replaced the map_backing_store() "unmapAddressRange" parameter by a "flags"
parameter.
* Added area creation flag CREATE_AREA_PRIORITY_VIP and slab allocator flag
CACHE_PRIORITY_VIP indicating the importance of the request.
* Changed most code to pass the right priorities/flags.

These changes already significantly improve the behavior in low memory
situations. I've tested a bit with 64 MB (virtual) RAM and, while not
particularly fast and responsive, the system remains at least usable under high
memory pressure.
As a side effect the slab allocator can now be used as general memory allocator.
Not done by default yet, though.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35295 a95241bf-73f2-0310-859d-f6bbb57e9c96

VMCache::Init(): Initialize the debug_* members before allocating the
VMCacheRef object, since that can fail, in which case the subsequently called
Delete() would use uninitialized pointers.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35279 a95241bf-73f2-0310-859d-f6bbb57e9c96

vm_page no longer points directly to its containing cache, but rather to a
VMCacheRef object which points to the cache. This allows to optimize
VMCache::MoveAllPages(), since it no longer needs to iterate over all pages
to adjust their cache pointer. It can simple swap the cache refs of the two
caches instead.

Reduces the total -j8 Haiku image build time only marginally. The kernel time
drops almost 10%, though.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35155 a95241bf-73f2-0310-859d-f6bbb57e9c96

* VMCache: Added a UserData attribute which can be used by the lock holder.
* Added "bool consumerLocked" parameter to VMCache::Unlock() and
ReleaseRefAndUnlock(). Since Unlock() may cause the cache to be merged with
a consumer cache, the flag is needed to prevent a deadlock in case the
caller still holds a lock to the consumer. Hasn't been a problem yet, since
that situation never occurred.
* VMCacheChainLocker: Reversed unlocking order to bottom-up. The other
direction could cause a deadlock in case caches would be merged, since the
locking order would be reversed. The way VMCacheChainLocker was used this
didn't happen, though.
* fault_get_page(): While copying a page from a lower cache to the top cache,
we do now unlock all caches but the top one, so we don't unnecessarily
kill concurrency.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35153 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added TODO.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34964 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added new debug feature (DEBUG_PAGE_ACCESS) to detect invalid concurrent
access to a vm_page. It is basically an atomically accessed thread ID field
in the vm_page structure, which is explicitly set by macros marking the
critical sections. As a first positive effect I had to review quite a bit of
code and found several issues.
* Added several TODOs and comments. Some harmless ones, but also a few
troublesome ones in vm.cpp regarding page unmapping.
* file_cache: PrecacheIO::Prepare()/read_into_cache: Removed superfluous
vm_page_allocate_page() return value checks. It cannot fail anymore.
* Removed the heavily contended "pages" lock. We use different policies now:
- sModifiedTemporaryPages is accessed atomically.
- sPageDeficitLock and sFreePageCondition are protected by a new mutex.
- The page queues have individual locks (mutexes).
- Renamed set_page_state_nolock() to set_page_state(). Unless the caller says
otherwise, it does now lock the affected pages queues itself. Also changed
the return value to void -- we panic() anyway.
* set_page_state(): Add free/clear pages to the beginning of their respective
queues as this is more cache-friendly.
* Pages with the states PAGE_STATE_WIRED or PAGE_STATE_UNUSED are no longer
in any queue. They were in the "active" queue, but there's no good reason
to have them there. In case we decide to let the page daemon work the queues
(like FreeBSD) they would just be in the way.
* Pulled the common part of vm_page_allocate_page_run[_no_base]() into a helper
function. Also fixed a bug I introduced previously: The functions must not
vm_page_unreserve_pages() on success, since they remove the pages from the
free/clear queue without decrementing sUnreservedFreePages.
* vm_page_set_state(): Changed return type to void. The function cannot really
fail and no-one was checking it anyway.
* vm_page_free(), vm_page_set_state(): Added assertion: The page must not be
free/clear before. This is implied by the policy that no-one is allowed to
access free/clear pages without holding the respective queue's lock, which is
not the case at this point. This found the bug fixed in r34912.
* vm_page_requeue(): Added general assertions. panic() when requeuing of
free/clear pages is requested. Same reason as above.
* vm_clone_area(), B_FULL_LOCK case: Don't map busy pages. The implementation is
still not correct, though.

My usual -j8 Haiku build test runs another 10% faster, now. The total kernel
time drops about 18%. As hoped the new locks have only a fraction of the old
"pages" lock contention. Other locks lead the "most wanted list" now.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34933 a95241bf-73f2-0310-859d-f6bbb57e9c96

RemovePage() + InsertPage() == MovePage()


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34913 a95241bf-73f2-0310-859d-f6bbb57e9c96

Inlined several VMCache methods.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34837 a95241bf-73f2-0310-859d-f6bbb57e9c96

* agp_gart(): Use vm_page_[un]reserve_pages().
* Removed unused vm_page_allocate_pages().
* Removed now unused (always true) "reserved" parameter from
vm_page_allocate_page().
* Removed unused (always false) "stealActive" parameter from steal_page().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34836 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Removed DEBUG_PAGE_CACHE_TRANSITIONS debugging.
* Added VMCache::MovePage() and MoveAllPages() to move pages between caches.
* VMAnonymousCache:
- _MergeSwapPages(): Avoid doing anything, if neither cache has swapped out
pages.
- _MergeSwapPages() does now also remove source cache pages that are
shadowed by consumer swap pages. This allows us to call _MergeSwapPages()
before _MergePagesSmallerSource(), save the swap page shadowing check
there and get rid of the vm_page::merge_swap flag. This is an
optimization based on the assumption that usually none or only few pages
are swapped out, so we save a lot of checks.
- Implemented _MergePagesSmallerConsumer() as an alternative to
_MergePagesSmallerSource(). The former is used when the source cache has
more pages than the consumer cache. It iterates over the consumer cache's
pages, moves them to the source and finally moves all pages back to the
consumer. The final move is relatively cheap (though unfortunately we
still have to update all pages' vm_page::cache field), so that overall we
save iterations of the main loop with the more expensive checks.

The optimizations particularly improve the common fork()+exec*() situations.
fork() uses CoW, which is implemented by putting two new empty caches between
the to be copied area and its cache. exec*() destroys one copy of the area,
its cache and thus causes merging of the other new cache with the old cache.
Since this usually happens in a very short time, the old cache does still
contain many pages and the new cache only few. Previously the many pages were
all checked and moved individually. Now we do that for the few pages instead.

A very extreme example of this situation is the Haiku image build. jam has a
huge heap (> 200 MB) and it fork()s+exec*()s for every action to be executed.
Since during the cache merging the cache is locked, any write access to a
heap page causes jam to block until the cache merging is done. Formerly that
took so long that it killed a lot of parallelism in multi-job builds. That
could be observed particularly well when lots of small actions where executed
(like the Link, XRes, Mimeset, SetType, SetVersion combos when building
executables/libraries/add-ons). Those look dramatically better now.
The overall speed improvement for a -j8 image build on my machine is only
about 15%, though.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34784 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added method VMCache::TransferAreas() moving areas from one cache to
another. The code originates from vm_copy_on_write_area(). We now generate
the VM cache tracing entries, though.
* count_writable_areas() -> VMCache::CountWritableAreas()
* Added debugger command "cache_stack" which is enabled when VM cache tracing
is enabled. It prints the source caches of a given cache or area at the
time of a specified tracing entry.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34751 a95241bf-73f2-0310-859d-f6bbb57e9c96

* arch_debug_get_stack_trace():
- Replaced the "userOnly" parameter by a "flags" parameter, that allows to
specify kernel and userland stack traces individually.
- x86, m68k: Don't always skip the first frame as that prevents the caller
from being able to record its own address.
* capture_tracing_stack_trace(): Replaced the "userOnly" parameter by
"kernelOnly", since one is probably always interested in the kernel stack
trace, but might not want the userland stack trace.
* Added stack trace support for VM cache kernel tracing.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34742 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added a simple mechanism to wait for events to VMCache. WaitForPageEvents()
waits for certain events on a given page, NotifyPageEvents() wakes up
waiting threads respectively.
* Used the new feature instead of condition variables for waiting on busy
pages. We save publishing and unpublishing of a condition variable whenever
a page is marked busy. There's only something to do, if there's at least
one thread waiting in the list of the respective cache. The general
assumption is that this is only rarely the case and even if it happens,
there should be only very few threads.
* Added an apparently missing notification in cache_io(). At least I didn't
see the reason for it not being there.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34537 a95241bf-73f2-0310-859d-f6bbb57e9c96

Created VMArea.{h,cpp} and moved VMArea and the global area hash table (new
class VMAreaHash) there.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34450 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved the VM headers into subdirectory vm/.
* Renamed vm_cache.h/vm_address_space.h to VMCache.h/VMAddressSpace.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34449 a95241bf-73f2-0310-859d-f6bbb57e9c96

Remove useless includes of khash.h

* These files were already converted to BOpenHashTable.
* For #9552.

Fix various 64 bit related warnings

Mostly printf() format strings and signed-unsigned comparisons.
Fixes the x86_64 build.

kernel: Add thread_unblock() and use it where possible

Print page and cache using initial commands in panic message.

VMCache: Fix broken trace statements

Fixed tracing printf formats in VM code.

Changed {,u}int64 to be long rather than long long on x86_64.

More 64-bit compilation/safety fixes.

mmlr (distracted) + bonefish:
* Turn VMCache::consumers C list into a DoublyLinkedList.
* Use object caches for the different VMCache types and the VMCacheRefs.
The purpose is to reduce slab area fragmentation.
* Requires the introduction of a pure virtual VMCache::DeleteObject()
method, implemented in the derived classes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43133 a95241bf-73f2-0310-859d-f6bbb57e9c96

Tiny style cleanup. No functional change.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43044 a95241bf-73f2-0310-859d-f6bbb57e9c96

Merged signals-merge branch into trunk with the following changes:
* Reorganized the kernel locking related to threads and teams.
* We now discriminate correctly between process and thread signals. Signal
handlers have been moved to teams. Fixes #5679.
* Implemented real-time signal support, including signal queuing, SA_SIGINFO
support, sigqueue(), sigwaitinfo(), sigtimedwait(), waitid(), and the addition
of the real-time signal range. Closes #1935 and #2695.
* Gave SIGBUS a separate signal number. Fixes #6704.
* Implemented <time.h> clock and timer support, and fixed/completed alarm() and
[set]itimer(). Closes #5682.
* Implemented support for thread cancellation. Closes #5686.
* Moved send_signal() from <signal.h> to <OS.h>. Fixes #7554.
* Lots over smaller more or less related changes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@42116 a95241bf-73f2-0310-859d-f6bbb57e9c96

Merged the signals branch into trunk, with these changes:
* The team and thread kernel structures have been renamed to Team and Thread
respectively and moved into the new BKernel namespace.
* Several (kernel add-on) sources have been converted from C to C++ since
private kernel headers are included that are no longer C compatible.

Changes after merging:
* Fixed gcc 2 build (warnings mainly in the scary firewire bus manager).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40196 a95241bf-73f2-0310-859d-f6bbb57e9c96

* VMCache::Unlock(): Renamed local variable consumerLocked to avoid shadowing
the parameter (CID 5329).
* _MergeWithOnlyConsumer(): Removed the somewhat weird consumerLocked
parameter. The caller can unlock itself, if desired. Improves Unlock()
readability.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40084 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved the vm_page initialization from vm_page.cpp:vm_page_init() to the new
vm_page::Init().
* Made vm_page::wired_count private and added accessor methods.
* Added VMCache::fWiredPagesCount (the number of wired pages the cache
contains) and accessor methods.
* Made more use of vm_page::IsMapped().
* vm_copy_on_write_area(): Added vm_page_reservation* parameter that can be
used to request a special handling for wired pages. If given the wired pages
are replaced by copies and the original pages are moved to the upper cache.
* vm_copy_area():
- We don't need to do any wired ranges handling, if the source area is a
B_SHARED_AREA, since we don't touch the area's mappings in this case.
- We no longer wait for wired ranges of the concerned areas to disappear.
Instead we use the new vm_copy_on_write_area() feature and just let it
copy the wired pages. This fixes #6288, an issue introduced with the use
of user mutexes in libroot: When executing multiple concurrent fork()s all
but the first one would wait on the fork mutex, which (being a user mutex)
would wire a page that the vm_copy_area() of the first fork() would wait
for.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37460 a95241bf-73f2-0310-859d-f6bbb57e9c96

Removed never read VMCache::scan_skip.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37195 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Renamed cache_type_to_string() to vm_cache_type_to_string() and made in
kernel private.
* Moved dumping code from dump_cache() to new VMCache::Dump().
* Override VMCache::Dump() in VMVnodeCache to also print the vnode.
* Removed no longer needed VMCache::GetLock().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37138 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fixed more address types related issues. Mostly printf() or comparison
warnings, but also some oversights from earlier changes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37000 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Introduced type generic_io_vec, which is similar to iovec, but uses types
that are wide enough for both virtual and physical addresses.
* DMABuffer, IORequest, IOScheduler,... and code using them: Use
generic_io_vec and generic_{addr,size}_t where necessary.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36997 a95241bf-73f2-0310-859d-f6bbb57e9c96

* VMCache::Resize(): Corrected TODO comment.
* Changed the semantics of VMCache::HasPage(). It was interpreted
inconsistently by the derived classes. Now it returns whether the backing
store can provide the page (via Read()). The default implementation returns
false. VNodeCache::HasPage() only returns true, if the given offset is
within the cache (i.e. file) bounds. This prevents vm_soft_fault() from
adding clean pages to vnode caches on faults beyond the file bounds.
Probably fixes #5473 -- at least mmap_resize_test behaves correctly, now.
* Removed redundant HasPage() and Fault() overrides in VMCache derived
classes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36374 a95241bf-73f2-0310-859d-f6bbb57e9c96

Made VMCache::Resize() virtual and let VMAnonymousCache override it to free
swap space when the cache shrinks. Currently the implementation stil leaks
swap space of busy pages.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36373 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added TODO.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36236 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added VMCache::DebugHasPage() and DebugLookupPage() for use in the kernel
debugger.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36228 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Made vm_page::state private and added accessor methods.
* Added kernel tracing for page state transitions.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35538 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Removed useless return parameter from vm_remove_all_page_mappings().
* Added vm_clear_page_mapping_accessed_flags() and
vm_remove_all_page_mappings_if_unaccessed(), which combine the functionality
of vm_test_map_activation(), vm_clear_map_flags(), and
vm_remove_all_page_mappings(), thus saving lots of calls to translation map
methods. The backend is the new method
VMTranslationMap::ClearAccessedAndModified().
* Started to make use of the cached page queue and changed the meaning of the
other non-free queues slightly:
- Active queue: Contains mapped pages that have been used recently.
- Inactive queue: Contains mapped pages that have not been used recently. Also
contains unmapped temporary pages.
- Modified queue: Contains unmapped modified pages.
- Cached queue: Contains unmapped unmodified pages (LRU sorted).
Unless we're actually low on memory and actively do paging, modified and
cached queues only contain non-temporary pages. Cached pages are considered
quasi free. They still belong to a cache, but since they are unmodified and
unmapped, they can be freed immediately. And this is what
vm_page_[try_]reserve_pages() do now when there are no more actually free
pages at hand. Essentially this means that pages storing cached file data,
unless mmap()ped, no longer are considered used and don't contribute to page
pressure. Paging will not happen as long there are enough free + cached pages
available.
* Reimplemented the page daemon. It no longer scans all pages, but instead works
the page queues. As long as the free pages situation is harmless, it only
iterates through the active queue and deactivates pages that have not been
used recently. When paging occurs it additionally scans the inactive queue and
frees pages that have not been used recently.
* Changed the page reservation/allocation interface:
vm_page_[try_]reserve_pages(), vm_page_unreserve_pages(), and
vm_page_allocate_page() now take a vm_page_reservation structure pointer.
The reservation functions initialize the structure -- currently consisting
only of a count member for the number of still reserved pages.
vm_page_allocate_page() decrements the count and vm_page_unreserve_pages()
unreserves the remaining pages (if any). Advantages are that reservation/
unreservation mismatches cannot occur anymore, that vm_page_allocate_page()
can verify that the caller has indeed a reserved page left, and that there's
no unnecessary pressure on the free page pool anymore. The only disadvantage
is that the vm_page_reservation object needs to be passed around a bit.
* Reworked the page reservation implementation:
- Got rid of sSystemReservedPages and sPageDeficit. Instead
sUnreservedFreePages now actually contains the number of free pages that
have not yet been reserved (it cannot become negative anymore) and the new
sUnsatisfiedPageReservations contains the number of pages that are still
needed for reservation.
- Threads waiting for reservations do now add themselves to a waiter queue,
which is ordered by descending priority (VM priority and thread priority).
High priority waiters are served first when pages become available.
Fixes #5328.
* cache_prefetch_vnode(): Would reserve one less page than allocated later, if
the size wasn't page aligned.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35393 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Removed the page state PAGE_STATE_BUSY and instead introduced a vm_page::busy
flag. The obvious advantage is that one can still see what state a page is in
and even move it between states while being marked busy.
* Removed the vm_page::is_dummy flag. Instead we mark marker pages busy, which
in all cases has the same effect. Introduced a vm_page_is_dummy() that can
still check whether a given page is a dummy page.
* vm_page_unreserve_pages(): Before adding to the system reserve make sure
sUnreservedFreePages is non-negative. Otherwise we'd make nonexisting pages
available for allocation. steal_pages() still has the same problem and it
can't be solved that easily.
* map_page(): No longer changes the page state/mark the page unbusy. That's the
caller's responsibility.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35331 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Introduced {malloc,memalign,free}_etc() which take an additional "flags"
argument. They replace the previous special-purpose allocation functions
(malloc_nogrow(), vip_io_request_malloc()).
* Moved the I/O VIP heap to heap.cpp accordingly.
* Added quite a bit of passing around of allocation flags in the VM,
particularly in the VM*AddressSpace classes.
* Fixed IOBuffer::GetNextVirtualVec(): It was ignoring the VIP flag and always
allocated on the normal heap.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35316 a95241bf-73f2-0310-859d-f6bbb57e9c96

* The system now holds back a small reserve of committable memory and pages. The
memory and page reservation functions have a new "priority" parameter that
indicates how deep the function may tap into that reserve. The currently
existing priority levels are "user", "system", and "VIP". The idea is that
user programs should never be able to cause a state that gets the kernel into
trouble due to heavy battling for memory. The "VIP" level (not really used
yet) is intended for allocations that are required to free memory eventually
(in the page writer). More levels are thinkable in the future, like "user real
time" or "user system server".
* Added "priority" parameters to several VMCache methods.
* Replaced the map_backing_store() "unmapAddressRange" parameter by a "flags"
parameter.
* Added area creation flag CREATE_AREA_PRIORITY_VIP and slab allocator flag
CACHE_PRIORITY_VIP indicating the importance of the request.
* Changed most code to pass the right priorities/flags.

These changes already significantly improve the behavior in low memory
situations. I've tested a bit with 64 MB (virtual) RAM and, while not
particularly fast and responsive, the system remains at least usable under high
memory pressure.
As a side effect the slab allocator can now be used as general memory allocator.
Not done by default yet, though.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35295 a95241bf-73f2-0310-859d-f6bbb57e9c96

VMCache::Init(): Initialize the debug_* members before allocating the
VMCacheRef object, since that can fail, in which case the subsequently called
Delete() would use uninitialized pointers.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35279 a95241bf-73f2-0310-859d-f6bbb57e9c96

vm_page no longer points directly to its containing cache, but rather to a
VMCacheRef object which points to the cache. This allows to optimize
VMCache::MoveAllPages(), since it no longer needs to iterate over all pages
to adjust their cache pointer. It can simple swap the cache refs of the two
caches instead.

Reduces the total -j8 Haiku image build time only marginally. The kernel time
drops almost 10%, though.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35155 a95241bf-73f2-0310-859d-f6bbb57e9c96

* VMCache: Added a UserData attribute which can be used by the lock holder.
* Added "bool consumerLocked" parameter to VMCache::Unlock() and
ReleaseRefAndUnlock(). Since Unlock() may cause the cache to be merged with
a consumer cache, the flag is needed to prevent a deadlock in case the
caller still holds a lock to the consumer. Hasn't been a problem yet, since
that situation never occurred.
* VMCacheChainLocker: Reversed unlocking order to bottom-up. The other
direction could cause a deadlock in case caches would be merged, since the
locking order would be reversed. The way VMCacheChainLocker was used this
didn't happen, though.
* fault_get_page(): While copying a page from a lower cache to the top cache,
we do now unlock all caches but the top one, so we don't unnecessarily
kill concurrency.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35153 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added TODO.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34964 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added new debug feature (DEBUG_PAGE_ACCESS) to detect invalid concurrent
access to a vm_page. It is basically an atomically accessed thread ID field
in the vm_page structure, which is explicitly set by macros marking the
critical sections. As a first positive effect I had to review quite a bit of
code and found several issues.
* Added several TODOs and comments. Some harmless ones, but also a few
troublesome ones in vm.cpp regarding page unmapping.
* file_cache: PrecacheIO::Prepare()/read_into_cache: Removed superfluous
vm_page_allocate_page() return value checks. It cannot fail anymore.
* Removed the heavily contended "pages" lock. We use different policies now:
- sModifiedTemporaryPages is accessed atomically.
- sPageDeficitLock and sFreePageCondition are protected by a new mutex.
- The page queues have individual locks (mutexes).
- Renamed set_page_state_nolock() to set_page_state(). Unless the caller says
otherwise, it does now lock the affected pages queues itself. Also changed
the return value to void -- we panic() anyway.
* set_page_state(): Add free/clear pages to the beginning of their respective
queues as this is more cache-friendly.
* Pages with the states PAGE_STATE_WIRED or PAGE_STATE_UNUSED are no longer
in any queue. They were in the "active" queue, but there's no good reason
to have them there. In case we decide to let the page daemon work the queues
(like FreeBSD) they would just be in the way.
* Pulled the common part of vm_page_allocate_page_run[_no_base]() into a helper
function. Also fixed a bug I introduced previously: The functions must not
vm_page_unreserve_pages() on success, since they remove the pages from the
free/clear queue without decrementing sUnreservedFreePages.
* vm_page_set_state(): Changed return type to void. The function cannot really
fail and no-one was checking it anyway.
* vm_page_free(), vm_page_set_state(): Added assertion: The page must not be
free/clear before. This is implied by the policy that no-one is allowed to
access free/clear pages without holding the respective queue's lock, which is
not the case at this point. This found the bug fixed in r34912.
* vm_page_requeue(): Added general assertions. panic() when requeuing of
free/clear pages is requested. Same reason as above.
* vm_clone_area(), B_FULL_LOCK case: Don't map busy pages. The implementation is
still not correct, though.

My usual -j8 Haiku build test runs another 10% faster, now. The total kernel
time drops about 18%. As hoped the new locks have only a fraction of the old
"pages" lock contention. Other locks lead the "most wanted list" now.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34933 a95241bf-73f2-0310-859d-f6bbb57e9c96

RemovePage() + InsertPage() == MovePage()


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34913 a95241bf-73f2-0310-859d-f6bbb57e9c96

Inlined several VMCache methods.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34837 a95241bf-73f2-0310-859d-f6bbb57e9c96

* agp_gart(): Use vm_page_[un]reserve_pages().
* Removed unused vm_page_allocate_pages().
* Removed now unused (always true) "reserved" parameter from
vm_page_allocate_page().
* Removed unused (always false) "stealActive" parameter from steal_page().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34836 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Removed DEBUG_PAGE_CACHE_TRANSITIONS debugging.
* Added VMCache::MovePage() and MoveAllPages() to move pages between caches.
* VMAnonymousCache:
- _MergeSwapPages(): Avoid doing anything, if neither cache has swapped out
pages.
- _MergeSwapPages() does now also remove source cache pages that are
shadowed by consumer swap pages. This allows us to call _MergeSwapPages()
before _MergePagesSmallerSource(), save the swap page shadowing check
there and get rid of the vm_page::merge_swap flag. This is an
optimization based on the assumption that usually none or only few pages
are swapped out, so we save a lot of checks.
- Implemented _MergePagesSmallerConsumer() as an alternative to
_MergePagesSmallerSource(). The former is used when the source cache has
more pages than the consumer cache. It iterates over the consumer cache's
pages, moves them to the source and finally moves all pages back to the
consumer. The final move is relatively cheap (though unfortunately we
still have to update all pages' vm_page::cache field), so that overall we
save iterations of the main loop with the more expensive checks.

The optimizations particularly improve the common fork()+exec*() situations.
fork() uses CoW, which is implemented by putting two new empty caches between
the to be copied area and its cache. exec*() destroys one copy of the area,
its cache and thus causes merging of the other new cache with the old cache.
Since this usually happens in a very short time, the old cache does still
contain many pages and the new cache only few. Previously the many pages were
all checked and moved individually. Now we do that for the few pages instead.

A very extreme example of this situation is the Haiku image build. jam has a
huge heap (> 200 MB) and it fork()s+exec*()s for every action to be executed.
Since during the cache merging the cache is locked, any write access to a
heap page causes jam to block until the cache merging is done. Formerly that
took so long that it killed a lot of parallelism in multi-job builds. That
could be observed particularly well when lots of small actions where executed
(like the Link, XRes, Mimeset, SetType, SetVersion combos when building
executables/libraries/add-ons). Those look dramatically better now.
The overall speed improvement for a -j8 image build on my machine is only
about 15%, though.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34784 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added method VMCache::TransferAreas() moving areas from one cache to
another. The code originates from vm_copy_on_write_area(). We now generate
the VM cache tracing entries, though.
* count_writable_areas() -> VMCache::CountWritableAreas()
* Added debugger command "cache_stack" which is enabled when VM cache tracing
is enabled. It prints the source caches of a given cache or area at the
time of a specified tracing entry.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34751 a95241bf-73f2-0310-859d-f6bbb57e9c96

* arch_debug_get_stack_trace():
- Replaced the "userOnly" parameter by a "flags" parameter, that allows to
specify kernel and userland stack traces individually.
- x86, m68k: Don't always skip the first frame as that prevents the caller
from being able to record its own address.
* capture_tracing_stack_trace(): Replaced the "userOnly" parameter by
"kernelOnly", since one is probably always interested in the kernel stack
trace, but might not want the userland stack trace.
* Added stack trace support for VM cache kernel tracing.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34742 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added a simple mechanism to wait for events to VMCache. WaitForPageEvents()
waits for certain events on a given page, NotifyPageEvents() wakes up
waiting threads respectively.
* Used the new feature instead of condition variables for waiting on busy
pages. We save publishing and unpublishing of a condition variable whenever
a page is marked busy. There's only something to do, if there's at least
one thread waiting in the list of the respective cache. The general
assumption is that this is only rarely the case and even if it happens,
there should be only very few threads.
* Added an apparently missing notification in cache_io(). At least I didn't
see the reason for it not being there.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34537 a95241bf-73f2-0310-859d-f6bbb57e9c96

Created VMArea.{h,cpp} and moved VMArea and the global area hash table (new
class VMAreaHash) there.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34450 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved the VM headers into subdirectory vm/.
* Renamed vm_cache.h/vm_address_space.h to VMCache.h/VMAddressSpace.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34449 a95241bf-73f2-0310-859d-f6bbb57e9c96