kernel: follow-up to hrev56619: use new(std::nothrow)

to handle allocation failures.

Change-Id: I15d84b24dcea17741382b1d5285acf6219a39811
Reviewed-on: https://review.haiku-os.org/c/haiku/+/5868
Reviewed-by: waddlesplash <waddlesplash@gmail.com>
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>

VMAnonymousCache: Use new/delete instead of malloc/free.

kernel/vm: Completely replace mlock() implementation.

The old implementation used the real lock_memory(). This is problematic
and does not work for a large number of reasons:

1) Various parts of the kernel assume memory is locked only very
temporarily, and will often wait on locked memory to become unlocked.
The transient nature of locks is further demonstrated by the fact that
lock_memory acquires references to structures, like the address space,
which are only released by unlock_memory

2) The VM has a hard assumption that all lock_memory calls will be
exactly balanced, and maintains internal "WiredRange" structures
on areas, etc. corresponding to the original lock_memory calls.
Maintaining separate data structures as this code did is a recipe
for even more problems when the structures are manipulated separately,
leading to confusing or incorrect behavior on unlocks.

3) Areas with locked memory cannot be deleted, nor can the pages which are
locked be removed from the areas/caches. This of course is most notable
when destroying teams which locked memory, but the problem also occurs
when just using delete_area, resize_area, mmap/munmap, etc.

Because of (2) and especially (3), adding support for mlock()-like semantics
to the existing memory locking system is just not a good option. A further
reason is that our lock_memory is much stricter than mlock(), which only
demands the pages in question must remain resident in RAM and cannot be
swapped out (or, it seems, otherwise written back to disk.)

Thus, this commit completely removes the old implementation (which
was seriously broken and did not actually automatically unlock memory
on team exit or area destruction at all, etc.) and instead adds a new
feature to VMAnonymousCache to block certain pages from being written out.
The syscall then just invokes this to do its work.

Fixes #17674. Related to #13651.

Change-Id: Id2745c51796bcf9a74ba5325fe686a95623cd521
Reviewed-on: https://review.haiku-os.org/c/haiku/+/5147
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

Add more missing headers following previous commits.

These also are only a problem on riscv64; I guess thread.h
must not be included in some arch header there.

Cleanup: Various comment and whitespace fixes.

Change-Id: I37c3e3346813efc595df651421b7e8ff4fbf3339
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2845
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

kernel/vm: Add discard_address_range that discards pages.

Pages in the given range are unmapped and freed without getting written
back anywhere. It can be used whenever a caller does not care about the
data in the given range anymore and wants to reduce page pressure.

Change-Id: I8bcce68fab278efef710d3714677e1d463504a56
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2843
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

kernel/vm: Change free swap reporting to actually free pages.

Previously system_info->free_swap_pages was using swap_available_pages
which has reservations removed. Tools like ActivityMonitor would
therefore show mere reservations as swap use which is misleading at
best.

Switch to use the sum of the free slots of all swap files instead.
This doesn't add overhead as the swap file list was already locked and
traversed for max_swap_pages before (via swap_total_swap_pages()).

Fixes #16248.

Change-Id: I3ebf223ec108bf342d4f32d68405170e72528899
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2917
Reviewed-by: Axel Dörfler <axeld@pinc-software.de>
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

kernel/vm: Fix build with swap support disabled.

The rename of the system_info members was missed in
d02aaee17e007631fcfa91a012ec7b6386927012 (part of the scheduler branch
merge of hrev46690). The unguarded object_cache was introduced even
earlier as part of hrev43133.

VMAnonymousCache: Reuse _FreeSwapPageRange in destructor.

The use of individual _SwapBlockGetAddress() and _SwapBlockFree() calls
would lock and unlock the swap hash for each page.

Using _FreeSwapPageRange() also allows to skip entire blocks when they
are not present or get empty early.

Change-Id: Ia76735e514cf8967d282e099cf5409fe1b104297
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2590
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

kernel/vm: Implement swap adoption for cut_area middle case.

Rename MovePageRange to Adopt and group it with Resize/Rebase as it
covers the third, middle cut case.

Implement VMAnonymousCache::Adopt() to actually adopt swap pages. This
has to recreate swap blocks instead of taking them over from the source
cache as the cut offset or base offset between the caches may not be
swap block aligned. This means that adoption may fail due to memory
shortage in allocating the swap blocks.

For the middle cut case it is therefore now possible to have the adopt
fail in which case the previous cache restore logic is applied. Since
the readoption of the pages from the second cache can fail for the same
reason, there is a slight chance that we can't restore and lose pages.
For now, just panic in such a case and add a TODO to free memory and
retry.

Change-Id: I9a661f00c8f03bbbea2fe6dee90371c68d7951e6
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2588
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

VMAnonymousCache: Fix use after free when freeing partial block.

When only some initial slots of a block were filled, the block would
become unused early and get freed. The iteration for the remaining slots
would then operate on the stale swapBlock as the pointer was not reset.

As we already know that the remaining slots can't be in use, directly
skip to the next swap block to avoid needless hash lookups.

Change-Id: Ib25377beb092aaf3533de1786b5f4c1099464599
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2584
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

VMAnonymousCache: Fix off by one in page freeing on resize.

The pageIndex is incremented on loop, so rounding up to the next swap
block would skip the first entry of the next block which would leak its
swap slot.

Change-Id: Ief3d29e711d323756034ea5ba6e300c489198aff
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2583
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

VMAnonymousCache: Factor out page freeing from Resize/Rebase.

Except for the offsets the code was identical. Also simplify the
conditions with early returns.

Change-Id: Ia7c44578ab06e571f6bf992db6c6d3493fd93230
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2582
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

Resize caches in all cases when cutting areas

* Adds VMCache::MovePageRange() and VMCache::Rebase() to facilitate
this.

Applied on top of hrev45098 and rebased with the hrev45564 page_num_t to
off_t change included.

Change-Id: Ie61bf43696783e3376fb4144ddced3781aa092ba
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2581
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

kernel: vm: add unit tests for get_mount_point

Change-Id: I1eb7540ffadb26acf05b695af2e7508c71ac7274
Reviewed-on: https://review.haiku-os.org/460
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

kernel: vm: remove panic

* That's embarrassing.

kernel: vm: fix off-by-one errors in get_mount_point

* Fix for 03df8bfcf2b58bf114cf876eccdd5242448926ce.
* Fix misleading indentation.
* Fixes #14225.

kernel: vm: reduce stack usage in swap_init_post_modules().

* avoid a struct copy in PartitionScorer.
* reduce stack usage in get_mount_point().

Change-Id: I60a3161ba39e9a50eaae972b7ff5b4a26d6292fa

Fix users of PAGE_SIZE

- Replaced by B_PAGE_SIZE where possible
- Enabled the _XOPEN_SOURCE feature define otherwise

VMAnonymousCache::_MergePagesSmallerConsumer(): Add ASSERT

kernel, libroot: Add more memory info in system_info

system_info now contains all information previously available only
through __get_system_info_etc(B_MEMORY_INFO, ...).

Fixed resource leak and possible strings corruption

Signed-off-by: Ingo Weinhold <ingo_weinhold@gmx.de>

VMAnonymousCache.cpp: use off_t instead of page_num_t

* use off_t instead of page_num_t to fit in swap_hash_key struct.
* this fixes narrowing conversion and signedness warnings on GCC 4.7.

Revert "VMAnonymousCache.cpp: changed page_index type to page_num_t"

This reverts commit f7176b0ee50d5367762d904a943a693b0a8e3e2f. Citing Ingo:
"off_t is the correct type to use for addressing pages in a cache/file,
which page_num_t should only be used for physical pages." I'll see how to
fix the GCC 4.7 warnings differently :)

VMAnonymousCache.cpp: changed page_index type to page_num_t

* consistently use page_num_t for page numbers and off_t for offsets and sizes.

Add support for pthread_attr_get/setguardsize()

* Added the aforementioned functions.
* create_area_etc() now takes a guard size parameter.
* The thread_info::stack_base/end range now refers to the usable range
only.

Kernel VM: Utilize swap_auto option

* Refine the swap logic a little
* Refine header copyright to preferred format
I had hamishm's verbal permission to change his entry

Kernel VM: A few changes as per Axel in #7742

* Avoid floating point numbers in the kernel
* Warning would always show if custom swap file in use
* Don't change a custom swap file size if low space occurs
* Ram > 1GB? Don't double the memory for the automatic size

Kernel VM: Add compatibility logic

* If old-format swap config file found, parse it properly

Kernel VM: Improve swap file selection

* Heavily based on Hamish Morrison's GCI work with some
modified logic and cleanup. #3723
* Adds automatic swap as well as user specified swap
* Limits:
Automatic: (ram * 2) up to 25% of the disk
User: user specified up to 90% of the disk
* Supports changing the swap disk location

Kernel VM: Style cleanup; No functional change

Fixed tracing printf formats in VM code.

Changed {,u}int64 to be long rather than long long on x86_64.

More 64-bit compilation/safety fixes.

VM: Memory leak fix in case of bad driver settings file

CID 5891.

mmlr (distracted) + bonefish:
* Turn VMCache::consumers C list into a DoublyLinkedList.
* Use object caches for the different VMCache types and the VMCacheRefs.
The purpose is to reduce slab area fragmentation.
* Requires the introduction of a pure virtual VMCache::DeleteObject()
method, implemented in the derived classes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43133 a95241bf-73f2-0310-859d-f6bbb57e9c96

Use MovePage() instead of RemovePage() + InsertPage().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37459 a95241bf-73f2-0310-859d-f6bbb57e9c96

Replaced the swap_addr_t and SWAP_SLOT_NONE in RadixBitmap.{h,cpp} by
radix_slot_t and RADIX_SLOT_NONE.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36998 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Introduced type generic_io_vec, which is similar to iovec, but uses types
that are wide enough for both virtual and physical addresses.
* DMABuffer, IORequest, IOScheduler,... and code using them: Use
generic_io_vec and generic_{addr,size}_t where necessary.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36997 a95241bf-73f2-0310-859d-f6bbb57e9c96

VMAnonymous[NoSwap]Cache, overcommitting mode:
* Commit(): Unreserve memory when asked to shrink the commitment.
* Fault(): The whole logic is flawed, since this is always called by
vm_soft_fault(), even, if the page is finally mapped from a lower cache.
Now we do at least limit our commitment to (page_count + 1) * B_PAGE_SIZE
instead of always reserving memory for another page.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36551 a95241bf-73f2-0310-859d-f6bbb57e9c96

Made VMCache::Resize() virtual and let VMAnonymousCache override it to free
swap space when the cache shrinks. Currently the implementation stil leaks
swap space of busy pages.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36373 a95241bf-73f2-0310-859d-f6bbb57e9c96

Missed that one in r36228: DebugHasPage() implementation.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36229 a95241bf-73f2-0310-859d-f6bbb57e9c96

VMAnonymousCache::_MergeSwapPages(): Missing check which could lead to a NULL
pointer access. Should fix #5453.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35575 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Introduced {malloc,memalign,free}_etc() which take an additional "flags"
argument. They replace the previous special-purpose allocation functions
(malloc_nogrow(), vip_io_request_malloc()).
* Moved the I/O VIP heap to heap.cpp accordingly.
* Added quite a bit of passing around of allocation flags in the VM,
particularly in the VM*AddressSpace classes.
* Fixed IOBuffer::GetNextVirtualVec(): It was ignoring the VIP flag and always
allocated on the normal heap.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35316 a95241bf-73f2-0310-859d-f6bbb57e9c96

* The system now holds back a small reserve of committable memory and pages. The
memory and page reservation functions have a new "priority" parameter that
indicates how deep the function may tap into that reserve. The currently
existing priority levels are "user", "system", and "VIP". The idea is that
user programs should never be able to cause a state that gets the kernel into
trouble due to heavy battling for memory. The "VIP" level (not really used
yet) is intended for allocations that are required to free memory eventually
(in the page writer). More levels are thinkable in the future, like "user real
time" or "user system server".
* Added "priority" parameters to several VMCache methods.
* Replaced the map_backing_store() "unmapAddressRange" parameter by a "flags"
parameter.
* Added area creation flag CREATE_AREA_PRIORITY_VIP and slab allocator flag
CACHE_PRIORITY_VIP indicating the importance of the request.
* Changed most code to pass the right priorities/flags.

These changes already significantly improve the behavior in low memory
situations. I've tested a bit with 64 MB (virtual) RAM and, while not
particularly fast and responsive, the system remains at least usable under high
memory pressure.
As a side effect the slab allocator can now be used as general memory allocator.
Not done by default yet, though.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35295 a95241bf-73f2-0310-859d-f6bbb57e9c96

Replaced CACHE_DONT_SLEEP by two new flags CACHE_DONT_WAIT_FOR_MEMORY and
CACHE_DONT_LOCK_KERNEL_SPACE. If the former is given, the slab memory manager
does not wait when reserving memory or pages. The latter prevents area
operations. The new flags add a bit of flexibility. E.g. when allocating page
mapping objects for userland areas CACHE_DONT_WAIT_FOR_MEMORY is sufficient,
i.e. the allocation will succeed as long as pages are available.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35246 a95241bf-73f2-0310-859d-f6bbb57e9c96

slab allocator:
* Implemented a more elaborated raw memory allocation backend (MemoryManager).
We allocate 8 MB areas whose pages we allocate and map when needed. An area is
divided into equally-sized chunks which form the basic units of allocation. We
have areas with three possible chunk sizes (small, medium, large), which is
basically what the ObjectCache implementations were using anyway.
* Added "uint32 flags" parameter to several of the slab allocator's object
cache and object depot functions. E.g. object_depot_store() potentially wants
to allocate memory for a magazine. But also in pure freeing functions it
might eventually become useful to have those flags, since they could end up
deleting an area, which might not be allowable in all situations. We should
introduce specific flags to indicate that.
* Reworked the block allocator. Since the MemoryManager allocates block-aligned
areas, maintains a hash table for lookup, and maps chunks to object caches,
we can quickly find out which object cache a to be freed allocation belongs
to and thus don't need the boundary tags anymore.
* Reworked the slab boot strap process. We allocate from the initial area only
when really necessary, i.e. when the object cache for the respective
allocation size has not been created yet. A single page is thus sufficient.

other:
* vm_allocate_early(): Added boolean "blockAlign" parameter. If true, the
semantics is the same as for B_ANY_KERNEL_BLOCK_ADDRESS.
* Use an object cache for page mappings. This significantly reduces the
contention on the heap bin locks.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35232 a95241bf-73f2-0310-859d-f6bbb57e9c96

vm_page no longer points directly to its containing cache, but rather to a
VMCacheRef object which points to the cache. This allows to optimize
VMCache::MoveAllPages(), since it no longer needs to iterate over all pages
to adjust their cache pointer. It can simple swap the cache refs of the two
caches instead.

Reduces the total -j8 Haiku image build time only marginally. The kernel time
drops almost 10%, though.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35155 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added new debug feature (DEBUG_PAGE_ACCESS) to detect invalid concurrent
access to a vm_page. It is basically an atomically accessed thread ID field
in the vm_page structure, which is explicitly set by macros marking the
critical sections. As a first positive effect I had to review quite a bit of
code and found several issues.
* Added several TODOs and comments. Some harmless ones, but also a few
troublesome ones in vm.cpp regarding page unmapping.
* file_cache: PrecacheIO::Prepare()/read_into_cache: Removed superfluous
vm_page_allocate_page() return value checks. It cannot fail anymore.
* Removed the heavily contended "pages" lock. We use different policies now:
- sModifiedTemporaryPages is accessed atomically.
- sPageDeficitLock and sFreePageCondition are protected by a new mutex.
- The page queues have individual locks (mutexes).
- Renamed set_page_state_nolock() to set_page_state(). Unless the caller says
otherwise, it does now lock the affected pages queues itself. Also changed
the return value to void -- we panic() anyway.
* set_page_state(): Add free/clear pages to the beginning of their respective
queues as this is more cache-friendly.
* Pages with the states PAGE_STATE_WIRED or PAGE_STATE_UNUSED are no longer
in any queue. They were in the "active" queue, but there's no good reason
to have them there. In case we decide to let the page daemon work the queues
(like FreeBSD) they would just be in the way.
* Pulled the common part of vm_page_allocate_page_run[_no_base]() into a helper
function. Also fixed a bug I introduced previously: The functions must not
vm_page_unreserve_pages() on success, since they remove the pages from the
free/clear queue without decrementing sUnreservedFreePages.
* vm_page_set_state(): Changed return type to void. The function cannot really
fail and no-one was checking it anyway.
* vm_page_free(), vm_page_set_state(): Added assertion: The page must not be
free/clear before. This is implied by the policy that no-one is allowed to
access free/clear pages without holding the respective queue's lock, which is
not the case at this point. This found the bug fixed in r34912.
* vm_page_requeue(): Added general assertions. panic() when requeuing of
free/clear pages is requested. Same reason as above.
* vm_clone_area(), B_FULL_LOCK case: Don't map busy pages. The implementation is
still not correct, though.

My usual -j8 Haiku build test runs another 10% faster, now. The total kernel
time drops about 18%. As hoped the new locks have only a fraction of the old
"pages" lock contention. Other locks lead the "most wanted list" now.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34933 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Removed DEBUG_PAGE_CACHE_TRANSITIONS debugging.
* Added VMCache::MovePage() and MoveAllPages() to move pages between caches.
* VMAnonymousCache:
- _MergeSwapPages(): Avoid doing anything, if neither cache has swapped out
pages.
- _MergeSwapPages() does now also remove source cache pages that are
shadowed by consumer swap pages. This allows us to call _MergeSwapPages()
before _MergePagesSmallerSource(), save the swap page shadowing check
there and get rid of the vm_page::merge_swap flag. This is an
optimization based on the assumption that usually none or only few pages
are swapped out, so we save a lot of checks.
- Implemented _MergePagesSmallerConsumer() as an alternative to
_MergePagesSmallerSource(). The former is used when the source cache has
more pages than the consumer cache. It iterates over the consumer cache's
pages, moves them to the source and finally moves all pages back to the
consumer. The final move is relatively cheap (though unfortunately we
still have to update all pages' vm_page::cache field), so that overall we
save iterations of the main loop with the more expensive checks.

The optimizations particularly improve the common fork()+exec*() situations.
fork() uses CoW, which is implemented by putting two new empty caches between
the to be copied area and its cache. exec*() destroys one copy of the area,
its cache and thus causes merging of the other new cache with the old cache.
Since this usually happens in a very short time, the old cache does still
contain many pages and the new cache only few. Previously the many pages were
all checked and moved individually. Now we do that for the few pages instead.

A very extreme example of this situation is the Haiku image build. jam has a
huge heap (> 200 MB) and it fork()s+exec*()s for every action to be executed.
Since during the cache merging the cache is locked, any write access to a
heap page causes jam to block until the cache merging is done. Formerly that
took so long that it killed a lot of parallelism in multi-job builds. That
could be observed particularly well when lots of small actions where executed
(like the Link, XRes, Mimeset, SetType, SetVersion combos when building
executables/libraries/add-ons). Those look dramatically better now.
The overall speed improvement for a -j8 image build on my machine is only
about 15%, though.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34784 a95241bf-73f2-0310-859d-f6bbb57e9c96

Moved merging swap pages from Merge() to a separate method.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34779 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Some style cleanup.
* Pulled the code moving the pages out of Merge() into a separate method.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34778 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved the VM headers into subdirectory vm/.
* Renamed vm_cache.h/vm_address_space.h to VMCache.h/VMAddressSpace.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34449 a95241bf-73f2-0310-859d-f6bbb57e9c96

Renamed vm_address_space to VMAddressSpace.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34422 a95241bf-73f2-0310-859d-f6bbb57e9c96

Make the swap hash lock into a rw_lock to reduce lock contention a bit.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@33622 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Rework page writing to combine page writes where possible. For now the pages
are required to be physically contiguos, which should be reworked to put them
into seperate iovecs. Still this manages to combine a great deal of page
writes into larger bursts already. Reduces the amount of IO requests being
scheduled (and greatly benefits media where page wise writes are slow when
they are accessed through a non-IOScheduler path, i.e. USB mass storage until
that is properly implemented).
* Abstracted per page page writing tasks into a PageWriteWrapper class.
* Abstracted per transfer page writing tasks into PageWriteTransfer class which
formerly was the PageWriterCallback.
* Use both classes from the PageWriterRun and from
vm_page_write_modified_page_range to remove code duplication.
* Adjusted synchronous VMAnonymousCache::Write() to cope correctly with larger
iovecs and more than one iovec. It assumed that there was exactly one page per
vector previously.
* Introduced MaxPagesPerWrite() and MaxPagesPerAsyncWrite() to VMCache to allow
a cache to specify restricitions. VMAnonymousCache does restrict the max pages
to 1 for WriteAsync right now as I didn't feel like reworking that one to cope
with non single page writes just yet.
* Pulled out PageWriteTransfer methods for better readability.
* Some typo fixes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@33507 a95241bf-73f2-0310-859d-f6bbb57e9c96

headers/private/kernel/util/OpenHashTable.h, Hugo's version, is a bit nicer than
Tracker's OpenHashTable.h which it should eventually replace. We've renamed the
class to BOpenHashTable and changed the interface slightly so that HashTableLink
became superfluous.
Adapted all the code that used it. Since the OpenHashTables no longer clash,
this should fix the GCC4 build.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@31791 a95241bf-73f2-0310-859d-f6bbb57e9c96

* vfs_boot.cpp now also exports gReadOnlyBootDevice which is true when the
boot device is actually read-only (even if it's using the write overlay).
* Do not create a swap file on a read-only device - this would really be a
stupid use of the write overlay (just saw this happening on an older
machine).
* Made swap_file_{add|delete}() take a const char* path - there was no reason
this was writable, and this also avoids casting away the const when adding
the default swap file.
* Minor cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@30975 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Reworked vm_soft_fault() and friends:
- While walking down the cache chain, we keep all upper caches locked.
- When we have to unlock -- when waiting for a busy page or reading a page in
-- we unlock completely, including the address space, and restart
vm_soft_fault().
- Folded fault_get_page() and fault_find_page() into one.
This simplifies and improves things considerably:
- We no longer need dummy pages.
- We no longer need vm_area::no_cache_change.
- #2710 is fixed, since we no longer hold the address space lock while
waiting.
* vm_soft_fault(): When we have found our page, we first check whether a page
is already mapped at the address. If it is already our page, we just change
its protection. If not, we unmap it first. Fixes race conditions when multiple
threads fault at the same address at the same time.
* fault_get_page(): When copying a read-only page from a lower cache, no longer
mark it active, since at least for the fault area it is shadowed from then on.
* vm_set_area_protection(): Fixed potential overflow for in the
vm_translation_map::protect() call.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@30911 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Introduces VMCache::CanWritePage() returning whether the given cache can
theoretically write the given page.
* page writer: Fixed the incorrect check whether a temporary page can be
written by using the new CanWritePage().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@30281 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Renamed src/system/kernel/device_manager/io_requests.{h,cpp} to
IORequest.{h,cpp}.
* Introduced public <io_requests.h> header. Currently it only declares the
single function BFS uses.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@29446 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fixing warnings under GCC4 in preparation to enable -Werror there as well:
* Replaced the use of offsetof() for structs that aren't PODs. Add a
offset_of_member() macro to util/khash.h because that's what it's used for
in our cases.
* Change the signature of add_debugger_command()/remove_debugger_command() on
GCC > 2 to avoid the depricated conversion from string constants to char *.
* Adding some "suggested" parenthesis. I know that not everyone likes that, but
it pointed out at least one bug that is fixed here as well.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@29113 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved more debug macros to kernel_debug_config.h.
* Turned the checks for all those macros to "#if"s instead of "#ifdef"s.
* Introduced macro KDEBUG_LEVEL which serves as a master setting.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@28248 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added "flags" parameter to VMCache::Read().
* Use the new VMCache::Read() flags parameter to directly read into the
physical page in the page fault handler instead of mapping it first.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27966 a95241bf-73f2-0310-859d-f6bbb57e9c96

Yay, infinite loop when debug output is disabled.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27878 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Fixed several instances of conversions from page to byte counts. We
need to cast explicitly before the multiplication/shift, since the
former is 32 bit and the latter 64 bit. The worst instance was in
swap_file_add(), where the page count was int32, so that swap file
sizes between 2 and 4 GB resulted in a negative available swap space
size. Fixes bug #2721.
* Fixed and added optional debug output.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27877 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Implemented a (private for now) get_system_info_etc() call, that can retrieve
various system information.
* Implemented retrieving some VM stats via this call.
* The VM now maintains a page fault counter, and sets system_info::page_faults
accordingly.
* Added a (pretty simple) "vmstat" command line app.
* Minor cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27597 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch by Zhao Shuai:
* Imported radix bitmap tree implementation from FreeBSD and adjusted it
for Haiku.
* Make use of the radix tree in the swap support implementation instead
of using simple bitmaps. This will allow for faster swap slot
allocations. ATM Haiku doesn't benefit that much, since we always
allocate single pages, but that will change eventually.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27355 a95241bf-73f2-0310-859d-f6bbb57e9c96

Adding/removing swap files:
* swap_file_add() open()s the swap file now with O_NOCACHE and
swap_file_delete() closes it. This squashes a TODO (the file cache
wasn't kept disabled for the swap file before).
* swap_file_add() only adds swap files that can actually be used (i.e.
non 0-sized ones).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27232 a95241bf-73f2-0310-859d-f6bbb57e9c96

Don't always commit memory in VMAnonymousCache::Fault() for
overcommitting caches. If the page in question was just not mapped or
swapped out, we would increase the committment unnecessarily
(potentially even beyond the size of the cache).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27229 a95241bf-73f2-0310-859d-f6bbb57e9c96

Changes by Zhao Shuai and myself:
* The VMAnonymousCache destructor was unreserving too much physical
memory.
* Addressed TODO in _SwapBlockBuild(): When the swap block couldn't be
allocated we wait and loop until it can.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27202 a95241bf-73f2-0310-859d-f6bbb57e9c96

* vm_page: Swapped members usage_count and wired_count. We lost 4 bytes
due to alignment padding before.
* Reorganized merging of caches a bit. Renamed MergeStore() to Merge()
and moved some more functionality into it. The method also moves the
pages from source to consumer, now. This is necessary, since
VMAnonymousCache needs to consider both physical pages and swap pages
at the same time. Before we first moved the physical pages and the
swap pages later, which was broken for two reasons: (1) A swap page in
the consumer cache shadows a physical page of the source cache, which
we ignored. (2) A source cache's physical page that also had a swap
page would lose the latter in the process when moved to the consumer
cache, i.e. if the page was not marked modified, it could be stolen
and its data would be lost.

These changes improve the situation when building Haiku with 256 MB RAM
in that jam doesn't crash anymore, but in my test the system became
totally unusable after about an hour or 7000 targets (GUI froze). For
some reason it didn't manage to free pages anymore although swapping
heavily.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27179 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added kernel tracing for the swap support.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27178 a95241bf-73f2-0310-859d-f6bbb57e9c96

Squashed a TODO: The global swap hash table is no longer resized
synchronously; we use the resource resizer instead. This avoids another
potential deadlock.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27123 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch by Zhao Shuai with some changes by myself: Some optimization of
_SwapBlockBuild() and _SwapBlockFree().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27107 a95241bf-73f2-0310-859d-f6bbb57e9c96

* The callback object created in WriteAsync() was never deleted. Thus
eventually the VIP heap would be exhausted.
* WriteAsync() didn't call the provided callback when an error occurred
before invoking vfs_asynchronous_write_pages(). The page writer would
get stuck in those cases.
* The object cache used for the swap blocks does now use the
asynchronous resizing feature to avoid deadlocks.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27101 a95241bf-73f2-0310-859d-f6bbb57e9c96

Implemented (a simple) WriteAsync(), i.e. swap pages are now written
asynchronously, too.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27077 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added "swap" debugger command printing some info on the swap space.
* Fixed lock leak in Write().
* Fixed bug in _Commit(): swap_space_reserve() was fed with the wrong
value (could even be negative).
* swap_free_page_swap_space(): Removed incrementing of sAvailSwapSpace.
The function is only supposed to deallocate swap space, not to
unreserve it.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27071 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch by Zhao Shuai with small changes by myself:
* Some renaming: A location in a swap file where a page can be stored is
now called "slot" instead of "page" or "swap page".
* swap_slot_alloc(): Update the hint more correctly after allocating
slots at the hint.
* swap_space_reserve(): When less than the requested space could be
reserved, it always returned 0 and leaked the remaining pages.
* swap_file_delete(): sSwapFileListLock wasn't unlocked in error cases.
Use MutexLocker now.
* swap_free_page_swap_space(): sAvailSwapSpace wasn't updated.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27057 a95241bf-73f2-0310-859d-f6bbb57e9c96

* VMCache::Write(): Added "uint32 flags" argument which is supposed to
be passed on to the IORequest. Most relevantly physical pages can now
be written directly by passing B_PHYSICAL_IO_REQUEST.
* Added VMCache::WriteAsync() which is supposed to write pages
asynchronously. The base class version version falls back to the
synchronous Write(). Only VMVnodeCache implements WriteAsync() ATM,
VMAnonymousCache (swap support) still has to be adjusted accordingly.
* write_page() doesn't need to map the page anymore as it can write the
physical page directly.
* Modified the page writer to write pages asynchronously. This shouldn't
have any noticeable effect yet. It will though as soon as the I/O
scheduler reorders I/O operations.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27056 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch by Zhao Shuai with minor changes by myself:
* Moved the static functions to the beginning of the file.
* Renamed several variables for more clarity, particularly
offset/cacheOffset to index/pageIndex in case where the unit is pages.
* _SwapBlock{Build,Free}() can deal with a range of pages now, which
should be more efficient.
* Additional checks/panics in swap_page_alloc().
* swap_file_add(): Wrong byte count was used when allocating and
clearing the bitmap.
* swap_page_alloc(): Fixed off-by-one check. The last page of the swap
file could not be used.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26975 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch by Zhao Shuai with some changes by myself:
* Added functions swap_free_page_swap_space(), swap_available_pages(),
and swap_total_swap_pages(). They will be used by the page daemon
code.
* Free allocated swap space in the VMAnonymousCache destructor.
* Write(): First free swap space assigned to the pages to be written
(was leaked before) and update fAllocatedSwapSize upfront. Both is now
done with the cache locked, as it should be.
* Fixes several instance where the cache offset in bytes was used
instead of in pages.
* Print the correct error when _kern_write_stat() fails.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26926 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Now uses _kern_write_stat() directly in order to resize the swap file
without filling it with zeros (ie. make use of B_STAT_SIZE_INSECURE).
* Added TODO that closing the file descriptor in swap_init_post_modules()
(that should probably renamed to swap_init_post_boot_device()) is not really
a good idea.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26797 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch by Zhao Shuai with changes by myself:
* Keep track of the stack space actually allocated for the cache and let
Write() fail when we've already allocated as much as reserved.
* Added second phase of swap initialization (swap_init_post_modules())
which reads the virtual memory driver settings and creates/resizes a
swap file. ATM truncate() is used to resize the swap file, but that is
a bit slow. We should probably introduce a VFS function to use BFS's
fast method.

This should make swap support work somewhat, but since swap space is
never freed ATM this would be a relatively short pleasure. Still
disabled by default.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26783 a95241bf-73f2-0310-859d-f6bbb57e9c96

vfs_{read,write}_pages() have an additional parameter now.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26693 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch by Zhao Shuai with changes by myself:
* Init swap support in main().
* Added "bool swappable" parameter to
VMCacheFactory::CreateAnonymousCache(). A cache supporting swapping
is created when true. Adjusted invocations accordingly.
* The page writer does now write non-locked swappable pages (when
memory is low).
* Fixed header guard of VMAnonymousNoSwapCache.h.
* Swap support is compiled conditionally, controlled by the
ENABLE_SWAP_SUPPORT in src/system/kernel/vm/VMAnonymousCache.h. It is
disabled ATM. Since no swap files are added, it wouldn't have much
effect anyway.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26625 a95241bf-73f2-0310-859d-f6bbb57e9c96

* bfs_fsync() was the only place which could cause the
fs_vnode_ops::write_pages() to be called with fsReenter = true. Since
this is no longer the case, the argument has become superfluous. For
read_pages() it always was. Removed the argument from the functions
and all functions that propagated it.
* Some whitespace at the end of lines was removed.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26579 a95241bf-73f2-0310-859d-f6bbb57e9c96

Typo spotted by Axel.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26573 a95241bf-73f2-0310-859d-f6bbb57e9c96

Merged branch haiku/branches/developer/bonefish/vm into trunk. This
introduces the following relevant changes:
* VMCache:
- Renamed vm_cache to VMCache, merged it with vm_store and made it a
C++ class with virtual methods (replacing the store operations).
Turned the different store implementations into subclasses.
- Introduced MergeStore() callback, changed semantics of Commit().
- Changed locking and referencing semantics. A reference can only be
acquired/released with the cache locked. An unreferenced cache is
deleted and a mergeable cache merged when it is unlocked. This
removes the "busy" state of a cache and simplifies the page fault
code.
* Added VMAnonymousCache, which will implement swap support (work by
Zhao Shuai). It is not integrated and used yet, though.
* Enabled the mutex/recursive lock holder asserts.
* Fixed DoublyLinkedList::Swap().
* Generalized the low memory handler to a low resource handler. And made
semaphores and reserved memory handled resources. Made
vm_try_resource_memory() optionally wait (with timeout), and used that
feature to reserve memory for areas.
...


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26572 a95241bf-73f2-0310-859d-f6bbb57e9c96

VMAnonymousCache::_MergePagesSmallerConsumer(): Add ASSERT

kernel, libroot: Add more memory info in system_info

system_info now contains all information previously available only
through __get_system_info_etc(B_MEMORY_INFO, ...).

Fixed resource leak and possible strings corruption

Signed-off-by: Ingo Weinhold <ingo_weinhold@gmx.de>

VMAnonymousCache.cpp: use off_t instead of page_num_t

* use off_t instead of page_num_t to fit in swap_hash_key struct.
* this fixes narrowing conversion and signedness warnings on GCC 4.7.

Revert "VMAnonymousCache.cpp: changed page_index type to page_num_t"

This reverts commit f7176b0ee50d5367762d904a943a693b0a8e3e2f. Citing Ingo:
"off_t is the correct type to use for addressing pages in a cache/file,
which page_num_t should only be used for physical pages." I'll see how to
fix the GCC 4.7 warnings differently :)

VMAnonymousCache.cpp: changed page_index type to page_num_t

* consistently use page_num_t for page numbers and off_t for offsets and sizes.

Add support for pthread_attr_get/setguardsize()

* Added the aforementioned functions.
* create_area_etc() now takes a guard size parameter.
* The thread_info::stack_base/end range now refers to the usable range
only.

Kernel VM: Utilize swap_auto option

* Refine the swap logic a little
* Refine header copyright to preferred format
I had hamishm's verbal permission to change his entry

Kernel VM: A few changes as per Axel in #7742

* Avoid floating point numbers in the kernel
* Warning would always show if custom swap file in use
* Don't change a custom swap file size if low space occurs
* Ram > 1GB? Don't double the memory for the automatic size

Kernel VM: Add compatibility logic

* If old-format swap config file found, parse it properly

Kernel VM: Improve swap file selection

* Heavily based on Hamish Morrison's GCI work with some
modified logic and cleanup. #3723
* Adds automatic swap as well as user specified swap
* Limits:
Automatic: (ram * 2) up to 25% of the disk
User: user specified up to 90% of the disk
* Supports changing the swap disk location

Kernel VM: Style cleanup; No functional change

Fixed tracing printf formats in VM code.

Changed {,u}int64 to be long rather than long long on x86_64.

More 64-bit compilation/safety fixes.

VM: Memory leak fix in case of bad driver settings file

CID 5891.

mmlr (distracted) + bonefish:
* Turn VMCache::consumers C list into a DoublyLinkedList.
* Use object caches for the different VMCache types and the VMCacheRefs.
The purpose is to reduce slab area fragmentation.
* Requires the introduction of a pure virtual VMCache::DeleteObject()
method, implemented in the derived classes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@43133 a95241bf-73f2-0310-859d-f6bbb57e9c96

Use MovePage() instead of RemovePage() + InsertPage().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@37459 a95241bf-73f2-0310-859d-f6bbb57e9c96

Replaced the swap_addr_t and SWAP_SLOT_NONE in RadixBitmap.{h,cpp} by
radix_slot_t and RADIX_SLOT_NONE.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36998 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Introduced type generic_io_vec, which is similar to iovec, but uses types
that are wide enough for both virtual and physical addresses.
* DMABuffer, IORequest, IOScheduler,... and code using them: Use
generic_io_vec and generic_{addr,size}_t where necessary.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36997 a95241bf-73f2-0310-859d-f6bbb57e9c96

VMAnonymous[NoSwap]Cache, overcommitting mode:
* Commit(): Unreserve memory when asked to shrink the commitment.
* Fault(): The whole logic is flawed, since this is always called by
vm_soft_fault(), even, if the page is finally mapped from a lower cache.
Now we do at least limit our commitment to (page_count + 1) * B_PAGE_SIZE
instead of always reserving memory for another page.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36551 a95241bf-73f2-0310-859d-f6bbb57e9c96

Made VMCache::Resize() virtual and let VMAnonymousCache override it to free
swap space when the cache shrinks. Currently the implementation stil leaks
swap space of busy pages.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36373 a95241bf-73f2-0310-859d-f6bbb57e9c96

Missed that one in r36228: DebugHasPage() implementation.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@36229 a95241bf-73f2-0310-859d-f6bbb57e9c96

VMAnonymousCache::_MergeSwapPages(): Missing check which could lead to a NULL
pointer access. Should fix #5453.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35575 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Introduced {malloc,memalign,free}_etc() which take an additional "flags"
argument. They replace the previous special-purpose allocation functions
(malloc_nogrow(), vip_io_request_malloc()).
* Moved the I/O VIP heap to heap.cpp accordingly.
* Added quite a bit of passing around of allocation flags in the VM,
particularly in the VM*AddressSpace classes.
* Fixed IOBuffer::GetNextVirtualVec(): It was ignoring the VIP flag and always
allocated on the normal heap.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35316 a95241bf-73f2-0310-859d-f6bbb57e9c96

* The system now holds back a small reserve of committable memory and pages. The
memory and page reservation functions have a new "priority" parameter that
indicates how deep the function may tap into that reserve. The currently
existing priority levels are "user", "system", and "VIP". The idea is that
user programs should never be able to cause a state that gets the kernel into
trouble due to heavy battling for memory. The "VIP" level (not really used
yet) is intended for allocations that are required to free memory eventually
(in the page writer). More levels are thinkable in the future, like "user real
time" or "user system server".
* Added "priority" parameters to several VMCache methods.
* Replaced the map_backing_store() "unmapAddressRange" parameter by a "flags"
parameter.
* Added area creation flag CREATE_AREA_PRIORITY_VIP and slab allocator flag
CACHE_PRIORITY_VIP indicating the importance of the request.
* Changed most code to pass the right priorities/flags.

These changes already significantly improve the behavior in low memory
situations. I've tested a bit with 64 MB (virtual) RAM and, while not
particularly fast and responsive, the system remains at least usable under high
memory pressure.
As a side effect the slab allocator can now be used as general memory allocator.
Not done by default yet, though.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35295 a95241bf-73f2-0310-859d-f6bbb57e9c96

Replaced CACHE_DONT_SLEEP by two new flags CACHE_DONT_WAIT_FOR_MEMORY and
CACHE_DONT_LOCK_KERNEL_SPACE. If the former is given, the slab memory manager
does not wait when reserving memory or pages. The latter prevents area
operations. The new flags add a bit of flexibility. E.g. when allocating page
mapping objects for userland areas CACHE_DONT_WAIT_FOR_MEMORY is sufficient,
i.e. the allocation will succeed as long as pages are available.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35246 a95241bf-73f2-0310-859d-f6bbb57e9c96

slab allocator:
* Implemented a more elaborated raw memory allocation backend (MemoryManager).
We allocate 8 MB areas whose pages we allocate and map when needed. An area is
divided into equally-sized chunks which form the basic units of allocation. We
have areas with three possible chunk sizes (small, medium, large), which is
basically what the ObjectCache implementations were using anyway.
* Added "uint32 flags" parameter to several of the slab allocator's object
cache and object depot functions. E.g. object_depot_store() potentially wants
to allocate memory for a magazine. But also in pure freeing functions it
might eventually become useful to have those flags, since they could end up
deleting an area, which might not be allowable in all situations. We should
introduce specific flags to indicate that.
* Reworked the block allocator. Since the MemoryManager allocates block-aligned
areas, maintains a hash table for lookup, and maps chunks to object caches,
we can quickly find out which object cache a to be freed allocation belongs
to and thus don't need the boundary tags anymore.
* Reworked the slab boot strap process. We allocate from the initial area only
when really necessary, i.e. when the object cache for the respective
allocation size has not been created yet. A single page is thus sufficient.

other:
* vm_allocate_early(): Added boolean "blockAlign" parameter. If true, the
semantics is the same as for B_ANY_KERNEL_BLOCK_ADDRESS.
* Use an object cache for page mappings. This significantly reduces the
contention on the heap bin locks.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35232 a95241bf-73f2-0310-859d-f6bbb57e9c96

vm_page no longer points directly to its containing cache, but rather to a
VMCacheRef object which points to the cache. This allows to optimize
VMCache::MoveAllPages(), since it no longer needs to iterate over all pages
to adjust their cache pointer. It can simple swap the cache refs of the two
caches instead.

Reduces the total -j8 Haiku image build time only marginally. The kernel time
drops almost 10%, though.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@35155 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added new debug feature (DEBUG_PAGE_ACCESS) to detect invalid concurrent
access to a vm_page. It is basically an atomically accessed thread ID field
in the vm_page structure, which is explicitly set by macros marking the
critical sections. As a first positive effect I had to review quite a bit of
code and found several issues.
* Added several TODOs and comments. Some harmless ones, but also a few
troublesome ones in vm.cpp regarding page unmapping.
* file_cache: PrecacheIO::Prepare()/read_into_cache: Removed superfluous
vm_page_allocate_page() return value checks. It cannot fail anymore.
* Removed the heavily contended "pages" lock. We use different policies now:
- sModifiedTemporaryPages is accessed atomically.
- sPageDeficitLock and sFreePageCondition are protected by a new mutex.
- The page queues have individual locks (mutexes).
- Renamed set_page_state_nolock() to set_page_state(). Unless the caller says
otherwise, it does now lock the affected pages queues itself. Also changed
the return value to void -- we panic() anyway.
* set_page_state(): Add free/clear pages to the beginning of their respective
queues as this is more cache-friendly.
* Pages with the states PAGE_STATE_WIRED or PAGE_STATE_UNUSED are no longer
in any queue. They were in the "active" queue, but there's no good reason
to have them there. In case we decide to let the page daemon work the queues
(like FreeBSD) they would just be in the way.
* Pulled the common part of vm_page_allocate_page_run[_no_base]() into a helper
function. Also fixed a bug I introduced previously: The functions must not
vm_page_unreserve_pages() on success, since they remove the pages from the
free/clear queue without decrementing sUnreservedFreePages.
* vm_page_set_state(): Changed return type to void. The function cannot really
fail and no-one was checking it anyway.
* vm_page_free(), vm_page_set_state(): Added assertion: The page must not be
free/clear before. This is implied by the policy that no-one is allowed to
access free/clear pages without holding the respective queue's lock, which is
not the case at this point. This found the bug fixed in r34912.
* vm_page_requeue(): Added general assertions. panic() when requeuing of
free/clear pages is requested. Same reason as above.
* vm_clone_area(), B_FULL_LOCK case: Don't map busy pages. The implementation is
still not correct, though.

My usual -j8 Haiku build test runs another 10% faster, now. The total kernel
time drops about 18%. As hoped the new locks have only a fraction of the old
"pages" lock contention. Other locks lead the "most wanted list" now.



git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34933 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Removed DEBUG_PAGE_CACHE_TRANSITIONS debugging.
* Added VMCache::MovePage() and MoveAllPages() to move pages between caches.
* VMAnonymousCache:
- _MergeSwapPages(): Avoid doing anything, if neither cache has swapped out
pages.
- _MergeSwapPages() does now also remove source cache pages that are
shadowed by consumer swap pages. This allows us to call _MergeSwapPages()
before _MergePagesSmallerSource(), save the swap page shadowing check
there and get rid of the vm_page::merge_swap flag. This is an
optimization based on the assumption that usually none or only few pages
are swapped out, so we save a lot of checks.
- Implemented _MergePagesSmallerConsumer() as an alternative to
_MergePagesSmallerSource(). The former is used when the source cache has
more pages than the consumer cache. It iterates over the consumer cache's
pages, moves them to the source and finally moves all pages back to the
consumer. The final move is relatively cheap (though unfortunately we
still have to update all pages' vm_page::cache field), so that overall we
save iterations of the main loop with the more expensive checks.

The optimizations particularly improve the common fork()+exec*() situations.
fork() uses CoW, which is implemented by putting two new empty caches between
the to be copied area and its cache. exec*() destroys one copy of the area,
its cache and thus causes merging of the other new cache with the old cache.
Since this usually happens in a very short time, the old cache does still
contain many pages and the new cache only few. Previously the many pages were
all checked and moved individually. Now we do that for the few pages instead.

A very extreme example of this situation is the Haiku image build. jam has a
huge heap (> 200 MB) and it fork()s+exec*()s for every action to be executed.
Since during the cache merging the cache is locked, any write access to a
heap page causes jam to block until the cache merging is done. Formerly that
took so long that it killed a lot of parallelism in multi-job builds. That
could be observed particularly well when lots of small actions where executed
(like the Link, XRes, Mimeset, SetType, SetVersion combos when building
executables/libraries/add-ons). Those look dramatically better now.
The overall speed improvement for a -j8 image build on my machine is only
about 15%, though.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34784 a95241bf-73f2-0310-859d-f6bbb57e9c96

Moved merging swap pages from Merge() to a separate method.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34779 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Some style cleanup.
* Pulled the code moving the pages out of Merge() into a separate method.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34778 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved the VM headers into subdirectory vm/.
* Renamed vm_cache.h/vm_address_space.h to VMCache.h/VMAddressSpace.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34449 a95241bf-73f2-0310-859d-f6bbb57e9c96

Renamed vm_address_space to VMAddressSpace.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@34422 a95241bf-73f2-0310-859d-f6bbb57e9c96

Make the swap hash lock into a rw_lock to reduce lock contention a bit.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@33622 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Rework page writing to combine page writes where possible. For now the pages
are required to be physically contiguos, which should be reworked to put them
into seperate iovecs. Still this manages to combine a great deal of page
writes into larger bursts already. Reduces the amount of IO requests being
scheduled (and greatly benefits media where page wise writes are slow when
they are accessed through a non-IOScheduler path, i.e. USB mass storage until
that is properly implemented).
* Abstracted per page page writing tasks into a PageWriteWrapper class.
* Abstracted per transfer page writing tasks into PageWriteTransfer class which
formerly was the PageWriterCallback.
* Use both classes from the PageWriterRun and from
vm_page_write_modified_page_range to remove code duplication.
* Adjusted synchronous VMAnonymousCache::Write() to cope correctly with larger
iovecs and more than one iovec. It assumed that there was exactly one page per
vector previously.
* Introduced MaxPagesPerWrite() and MaxPagesPerAsyncWrite() to VMCache to allow
a cache to specify restricitions. VMAnonymousCache does restrict the max pages
to 1 for WriteAsync right now as I didn't feel like reworking that one to cope
with non single page writes just yet.
* Pulled out PageWriteTransfer methods for better readability.
* Some typo fixes.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@33507 a95241bf-73f2-0310-859d-f6bbb57e9c96

headers/private/kernel/util/OpenHashTable.h, Hugo's version, is a bit nicer than
Tracker's OpenHashTable.h which it should eventually replace. We've renamed the
class to BOpenHashTable and changed the interface slightly so that HashTableLink
became superfluous.
Adapted all the code that used it. Since the OpenHashTables no longer clash,
this should fix the GCC4 build.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@31791 a95241bf-73f2-0310-859d-f6bbb57e9c96

* vfs_boot.cpp now also exports gReadOnlyBootDevice which is true when the
boot device is actually read-only (even if it's using the write overlay).
* Do not create a swap file on a read-only device - this would really be a
stupid use of the write overlay (just saw this happening on an older
machine).
* Made swap_file_{add|delete}() take a const char* path - there was no reason
this was writable, and this also avoids casting away the const when adding
the default swap file.
* Minor cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@30975 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Reworked vm_soft_fault() and friends:
- While walking down the cache chain, we keep all upper caches locked.
- When we have to unlock -- when waiting for a busy page or reading a page in
-- we unlock completely, including the address space, and restart
vm_soft_fault().
- Folded fault_get_page() and fault_find_page() into one.
This simplifies and improves things considerably:
- We no longer need dummy pages.
- We no longer need vm_area::no_cache_change.
- #2710 is fixed, since we no longer hold the address space lock while
waiting.
* vm_soft_fault(): When we have found our page, we first check whether a page
is already mapped at the address. If it is already our page, we just change
its protection. If not, we unmap it first. Fixes race conditions when multiple
threads fault at the same address at the same time.
* fault_get_page(): When copying a read-only page from a lower cache, no longer
mark it active, since at least for the fault area it is shadowed from then on.
* vm_set_area_protection(): Fixed potential overflow for in the
vm_translation_map::protect() call.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@30911 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Introduces VMCache::CanWritePage() returning whether the given cache can
theoretically write the given page.
* page writer: Fixed the incorrect check whether a temporary page can be
written by using the new CanWritePage().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@30281 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Renamed src/system/kernel/device_manager/io_requests.{h,cpp} to
IORequest.{h,cpp}.
* Introduced public <io_requests.h> header. Currently it only declares the
single function BFS uses.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@29446 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fixing warnings under GCC4 in preparation to enable -Werror there as well:
* Replaced the use of offsetof() for structs that aren't PODs. Add a
offset_of_member() macro to util/khash.h because that's what it's used for
in our cases.
* Change the signature of add_debugger_command()/remove_debugger_command() on
GCC > 2 to avoid the depricated conversion from string constants to char *.
* Adding some "suggested" parenthesis. I know that not everyone likes that, but
it pointed out at least one bug that is fixed here as well.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@29113 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Moved more debug macros to kernel_debug_config.h.
* Turned the checks for all those macros to "#if"s instead of "#ifdef"s.
* Introduced macro KDEBUG_LEVEL which serves as a master setting.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@28248 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added "flags" parameter to VMCache::Read().
* Use the new VMCache::Read() flags parameter to directly read into the
physical page in the page fault handler instead of mapping it first.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27966 a95241bf-73f2-0310-859d-f6bbb57e9c96

Yay, infinite loop when debug output is disabled.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27878 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Fixed several instances of conversions from page to byte counts. We
need to cast explicitly before the multiplication/shift, since the
former is 32 bit and the latter 64 bit. The worst instance was in
swap_file_add(), where the page count was int32, so that swap file
sizes between 2 and 4 GB resulted in a negative available swap space
size. Fixes bug #2721.
* Fixed and added optional debug output.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27877 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Implemented a (private for now) get_system_info_etc() call, that can retrieve
various system information.
* Implemented retrieving some VM stats via this call.
* The VM now maintains a page fault counter, and sets system_info::page_faults
accordingly.
* Added a (pretty simple) "vmstat" command line app.
* Minor cleanup.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27597 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch by Zhao Shuai:
* Imported radix bitmap tree implementation from FreeBSD and adjusted it
for Haiku.
* Make use of the radix tree in the swap support implementation instead
of using simple bitmaps. This will allow for faster swap slot
allocations. ATM Haiku doesn't benefit that much, since we always
allocate single pages, but that will change eventually.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27355 a95241bf-73f2-0310-859d-f6bbb57e9c96

Adding/removing swap files:
* swap_file_add() open()s the swap file now with O_NOCACHE and
swap_file_delete() closes it. This squashes a TODO (the file cache
wasn't kept disabled for the swap file before).
* swap_file_add() only adds swap files that can actually be used (i.e.
non 0-sized ones).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27232 a95241bf-73f2-0310-859d-f6bbb57e9c96

Don't always commit memory in VMAnonymousCache::Fault() for
overcommitting caches. If the page in question was just not mapped or
swapped out, we would increase the committment unnecessarily
(potentially even beyond the size of the cache).


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27229 a95241bf-73f2-0310-859d-f6bbb57e9c96

Changes by Zhao Shuai and myself:
* The VMAnonymousCache destructor was unreserving too much physical
memory.
* Addressed TODO in _SwapBlockBuild(): When the swap block couldn't be
allocated we wait and loop until it can.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27202 a95241bf-73f2-0310-859d-f6bbb57e9c96

* vm_page: Swapped members usage_count and wired_count. We lost 4 bytes
due to alignment padding before.
* Reorganized merging of caches a bit. Renamed MergeStore() to Merge()
and moved some more functionality into it. The method also moves the
pages from source to consumer, now. This is necessary, since
VMAnonymousCache needs to consider both physical pages and swap pages
at the same time. Before we first moved the physical pages and the
swap pages later, which was broken for two reasons: (1) A swap page in
the consumer cache shadows a physical page of the source cache, which
we ignored. (2) A source cache's physical page that also had a swap
page would lose the latter in the process when moved to the consumer
cache, i.e. if the page was not marked modified, it could be stolen
and its data would be lost.

These changes improve the situation when building Haiku with 256 MB RAM
in that jam doesn't crash anymore, but in my test the system became
totally unusable after about an hour or 7000 targets (GUI froze). For
some reason it didn't manage to free pages anymore although swapping
heavily.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27179 a95241bf-73f2-0310-859d-f6bbb57e9c96

Added kernel tracing for the swap support.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27178 a95241bf-73f2-0310-859d-f6bbb57e9c96

Squashed a TODO: The global swap hash table is no longer resized
synchronously; we use the resource resizer instead. This avoids another
potential deadlock.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27123 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch by Zhao Shuai with some changes by myself: Some optimization of
_SwapBlockBuild() and _SwapBlockFree().


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27107 a95241bf-73f2-0310-859d-f6bbb57e9c96

* The callback object created in WriteAsync() was never deleted. Thus
eventually the VIP heap would be exhausted.
* WriteAsync() didn't call the provided callback when an error occurred
before invoking vfs_asynchronous_write_pages(). The page writer would
get stuck in those cases.
* The object cache used for the swap blocks does now use the
asynchronous resizing feature to avoid deadlocks.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27101 a95241bf-73f2-0310-859d-f6bbb57e9c96

Implemented (a simple) WriteAsync(), i.e. swap pages are now written
asynchronously, too.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27077 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Added "swap" debugger command printing some info on the swap space.
* Fixed lock leak in Write().
* Fixed bug in _Commit(): swap_space_reserve() was fed with the wrong
value (could even be negative).
* swap_free_page_swap_space(): Removed incrementing of sAvailSwapSpace.
The function is only supposed to deallocate swap space, not to
unreserve it.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27071 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch by Zhao Shuai with small changes by myself:
* Some renaming: A location in a swap file where a page can be stored is
now called "slot" instead of "page" or "swap page".
* swap_slot_alloc(): Update the hint more correctly after allocating
slots at the hint.
* swap_space_reserve(): When less than the requested space could be
reserved, it always returned 0 and leaked the remaining pages.
* swap_file_delete(): sSwapFileListLock wasn't unlocked in error cases.
Use MutexLocker now.
* swap_free_page_swap_space(): sAvailSwapSpace wasn't updated.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27057 a95241bf-73f2-0310-859d-f6bbb57e9c96

* VMCache::Write(): Added "uint32 flags" argument which is supposed to
be passed on to the IORequest. Most relevantly physical pages can now
be written directly by passing B_PHYSICAL_IO_REQUEST.
* Added VMCache::WriteAsync() which is supposed to write pages
asynchronously. The base class version version falls back to the
synchronous Write(). Only VMVnodeCache implements WriteAsync() ATM,
VMAnonymousCache (swap support) still has to be adjusted accordingly.
* write_page() doesn't need to map the page anymore as it can write the
physical page directly.
* Modified the page writer to write pages asynchronously. This shouldn't
have any noticeable effect yet. It will though as soon as the I/O
scheduler reorders I/O operations.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@27056 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch by Zhao Shuai with minor changes by myself:
* Moved the static functions to the beginning of the file.
* Renamed several variables for more clarity, particularly
offset/cacheOffset to index/pageIndex in case where the unit is pages.
* _SwapBlock{Build,Free}() can deal with a range of pages now, which
should be more efficient.
* Additional checks/panics in swap_page_alloc().
* swap_file_add(): Wrong byte count was used when allocating and
clearing the bitmap.
* swap_page_alloc(): Fixed off-by-one check. The last page of the swap
file could not be used.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26975 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch by Zhao Shuai with some changes by myself:
* Added functions swap_free_page_swap_space(), swap_available_pages(),
and swap_total_swap_pages(). They will be used by the page daemon
code.
* Free allocated swap space in the VMAnonymousCache destructor.
* Write(): First free swap space assigned to the pages to be written
(was leaked before) and update fAllocatedSwapSize upfront. Both is now
done with the cache locked, as it should be.
* Fixes several instance where the cache offset in bytes was used
instead of in pages.
* Print the correct error when _kern_write_stat() fails.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26926 a95241bf-73f2-0310-859d-f6bbb57e9c96

* Now uses _kern_write_stat() directly in order to resize the swap file
without filling it with zeros (ie. make use of B_STAT_SIZE_INSECURE).
* Added TODO that closing the file descriptor in swap_init_post_modules()
(that should probably renamed to swap_init_post_boot_device()) is not really
a good idea.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26797 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch by Zhao Shuai with changes by myself:
* Keep track of the stack space actually allocated for the cache and let
Write() fail when we've already allocated as much as reserved.
* Added second phase of swap initialization (swap_init_post_modules())
which reads the virtual memory driver settings and creates/resizes a
swap file. ATM truncate() is used to resize the swap file, but that is
a bit slow. We should probably introduce a VFS function to use BFS's
fast method.

This should make swap support work somewhat, but since swap space is
never freed ATM this would be a relatively short pleasure. Still
disabled by default.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26783 a95241bf-73f2-0310-859d-f6bbb57e9c96

vfs_{read,write}_pages() have an additional parameter now.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26693 a95241bf-73f2-0310-859d-f6bbb57e9c96

Patch by Zhao Shuai with changes by myself:
* Init swap support in main().
* Added "bool swappable" parameter to
VMCacheFactory::CreateAnonymousCache(). A cache supporting swapping
is created when true. Adjusted invocations accordingly.
* The page writer does now write non-locked swappable pages (when
memory is low).
* Fixed header guard of VMAnonymousNoSwapCache.h.
* Swap support is compiled conditionally, controlled by the
ENABLE_SWAP_SUPPORT in src/system/kernel/vm/VMAnonymousCache.h. It is
disabled ATM. Since no swap files are added, it wouldn't have much
effect anyway.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26625 a95241bf-73f2-0310-859d-f6bbb57e9c96

* bfs_fsync() was the only place which could cause the
fs_vnode_ops::write_pages() to be called with fsReenter = true. Since
this is no longer the case, the argument has become superfluous. For
read_pages() it always was. Removed the argument from the functions
and all functions that propagated it.
* Some whitespace at the end of lines was removed.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26579 a95241bf-73f2-0310-859d-f6bbb57e9c96

Typo spotted by Axel.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26573 a95241bf-73f2-0310-859d-f6bbb57e9c96

Merged branch haiku/branches/developer/bonefish/vm into trunk. This
introduces the following relevant changes:
* VMCache:
- Renamed vm_cache to VMCache, merged it with vm_store and made it a
C++ class with virtual methods (replacing the store operations).
Turned the different store implementations into subclasses.
- Introduced MergeStore() callback, changed semantics of Commit().
- Changed locking and referencing semantics. A reference can only be
acquired/released with the cache locked. An unreferenced cache is
deleted and a mergeable cache merged when it is unlocked. This
removes the "busy" state of a cache and simplifies the page fault
code.
* Added VMAnonymousCache, which will implement swap support (work by
Zhao Shuai). It is not integrated and used yet, though.
* Enabled the mutex/recursive lock holder asserts.
* Fixed DoublyLinkedList::Swap().
* Generalized the low memory handler to a low resource handler. And made
semaphores and reserved memory handled resources. Made
vm_try_resource_memory() optionally wait (with timeout), and used that
feature to reserve memory for areas.
...


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@26572 a95241bf-73f2-0310-859d-f6bbb57e9c96