vm_fault: Shoot down multiply mapped COW source page mappings

Reviewed by: kib, rlibby
Discussed with: alc
Approved by: so
Security: CVE-2021-29626
Security: FreeBSD-SA-21:08.vm

Git Hash: 71a0b26df14a18b720faaa924bd4e18fcb9638d5
Git Author: markj@FreeBSD.org

MFC r345324:
vm_fault_copy_entry: accept invalid source pages.

MFC r338999:
Correct vm_fault_copy_entry() handling of backing file truncation
after the file mapping was wired.

MFC r338998:
In vm_fault_copy_entry(), we should not assert that entry is charged
if the dst_object is not of swap type.

MFC r338997:
In vm_fault_copy_entry(), collect the code to initialize a newly
allocated dst_object in a single place.

MFC r334389:
Typo.

PR: 228533

MFC r333091:
Eliminate some vm object relocks in vm fault.

Approved by: re (marius)

MFC r331557:
Allow to specify for vm_fault_quick_hold_pages() that nofault mode
should be honored.

Revert r330897:

This was intended to be a non-functional change. It wasn't. The commit
message was thus wrong. In addition it broke arm, and merged crypto
related code.

Revert with prejudice.

This revert skips files touched in r316370 since that commit was since
MFCed. This revert also skips files that require $FreeBSD$ property
changes.

Thank you to those who helped me get out of this mess including but not
limited to gonzo, kevans, rgrimes.

Requested by: gjb (re)

Partial merge of the SPDX changes

These changes are incomplete but are making it difficult
to determine what other changes can/should be merged.

No objections from: pfg

MFC r329254:
Ensure memory consistency on COW.

MFC r329252:
Do not call pmap_enter() with invalid protection mode.

MFC r321386,321393
Utilize pmap_enter(..., psind=1) in vm_fault_soft_fast() on amd64. (The
Differential Revision discusses the benefits of this change.)

Add a function, vm_reserv_to_superpage(), that returns the superpage
containing the specified base page.

MFC r322913:
Replace global swhash in swap pager with per-object trie to track swap
blocks assigned to the object pages.

MFC r322970 (by alc):
Do not call vm_pager_page_unswapped() on the fast fault path.

MFC r322971 (by alc):
Update a couple vm_object lock assertions in the swap pager.

MFC r323224:
In swp_pager_meta_build(), handle a race with other thread allocating
swapblk for our index while we dropped the object lock.

MFC r323226:
Do not leak empty swblk.

Add MAP_GUARD and use it for stack grow area protection.
Bump __FreeBSD_version.

MFC r320317:
Implement address space guards.

MFC r320338:
Remove stale part of the comment.

MFC r320339:
Correctly handle small MAP_STACK requests.

MFC r320344:
For now, allow mprotect(2) over the guards to succeed regardless of
the requested protection.

MFC r320430:
Treat the addr argument for mmap(2) request without MAP_FIXED flag as
a hint.

MFC r320560 (by alc):
Modify vm_map_growstack() to protect itself from the possibility of the
gap entry in the vm map being smaller than the sysctl-derived stack guard
size.

MFC r308474, r308691, r309203, r309365, r309703, r309898, r310720,
r308489, r308706:
Add PQ_LAUNDRY and remove PG_CACHED pages.

MFC r315281:
Use atop() instead of OFF_TO_IDX() for convertion of addresses or
addresses offsets, as intended.

MFC r315580 (by alc):
Simplify the logic for clipping the range returned by the pager to fit
within the map entry.
Use atop() rather than OFF_TO_IDX() on addresses.

MFC r315552:
Fix off-by-one in the vm_fault_populate() code.

MFC r314195:
Properly handle possible underflow in vm_fault_prefault().

MFC r309710:
Add a new populate() pager method and extend device pager ops vector
with cdev_pg_populate() to provide device drivers access to it.

MFC r310849:
Fix two similar bugs in the populate vm_fault() code.

MFC r309709:
Move map_generation snapshot value into struct faultstate.

MFC r308733:
Move the fast fault path into the separate function.

MFC r308174, r308261
Move and revise a comment about the relation between the object's paging-
in-progress count and the vnode. Prior to r188331, we always acquired
the vnode lock before incrementing the object's paging-in-progress count.
Now, we increment it before attempting to acquire the vnode lock with
LK_NOWAIT, but we never sleep acquiring the vnode lock while we have the
count incremented.

In vm_fault()'s loop over the shadow chain, move a comment describing our
invariants to a better place. Also, add two comments concerning the
relationship between the map and vnode locks.

MFC r308114:
Change remained internal uses of boolean_t to bool in vm/vm_fault.c.

MFC r308109:
Remove vnode_locked label and goto.

MFC r308096, r308098, r308112
With one exception, "hardfault" is used like a "bool". Change that
exception and make it a "bool".

The "lookup_still_valid" field is used as a "bool". Make it one.

Convert vm_fault_hold()'s Boolean variables that are only used
internally to "bool". Add a comment describing why the one
remaining "boolean_t" was not converted.

Merge and sort vm_fault_hold()'s "int" variable definitions.

MFC r308094:
Add unlock_vp() helper.

MFC r308095 (by markj):
Add one more use of unlock_vp().

MFC r307501:
If vm_fault_hold(9) finds that fs.m is wired, do not free it after a
pager error, leave the page to the wire owner.

MFC r307236:
Plug a vnode lock leak in vm_fault_hold().

MFC r303101
Add a comment describing the 'fast path' that was introduced in r270011.

Approved by: re (gjb)

MFC r302980
Break up vm_fault()'s implementation of the read-ahead and delete-behind
optimizations into two distinct pieces. The first piece consists of the
code that should only be performed once per page fault and requires the
map to be locked. The second piece consists of the code that should be
performed each time a pager is called on an object in the shadow chain.
(This second piece expects the map to be unlocked.)

Previously, the entire implementation could be executed multiple times.
Moreover, the second and subsequent executions would occur with the map
unlocked. Usually, the ensuing unsynchronized accesses to the map were
harmless because the map was not changing. Nonetheless, it was possible
for a use-after-free error to occur, where vm_fault() wrote to a freed
map entry. This change corrects that problem.

Approved by: re (gjb)