LinuxKPI: Add want_init_on_free to linux/mm.h

want_init_on_free returns if heap memory zeroing on free is enabled.
FreeBSD does not zeroes heap memory on free().

Sponsored by: Serenity Cyber Security
Reviewed by: emaste
MFC after: 1 week

LinuxKPI: Add vm_flags_(clear|set) functions

Sponsored by: Serenity Cyber Security, LLC
Reviewed by: emaste
MFC after: 1 week

LinuxKPI: Add VM_ACCESS_FLAGS define to linux/mm.h

VM_ACCESS_FLAGS is a basic access permission flags.

Sponsored by: Serenity CyberSecurity, LLC
Reviewed by: emaste
MFC after: 1 week

LinuxKPI: Import vanilla linux/overflow.h

It is dual-licensed (GPLv2 & MIT) and self-contained header file.
No need to reimplement it.

Sponsored by: Serenity CyberSecurity, LLC
Reviewed by: emaste
MFC after: 1 week

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

LinuxKPI: reduce usage of struct vm_page and vm_page_t

We currently define (Linux) page to (FreeBSD) vm_page.
Cleanup some of the direct struct vm_page and vm_page_t declarations
and usages in the Linux KPI and make them 'struct page' or
'struct page *' to prepare for more upcoming work.

This should be a NOP.

Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D41255

linuxkpi: Define `totalram_pages()`

It returns `physmem`.

`__FreeBSD_version` is bumped to 1400080. We need that to remove a copy
of this function in the i915 DRM driver, maintained outside of base.

Reviewed by: manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D38531

LinuxKPI: mm.h: implement virt_to_head_page()

Implement virt_to_head_page().

Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D38240

linuxkpi: Define `might_alloc()`

... as a no-op.

Reviewed by: manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D38146

linuxkpi: Add `freeram` and `freehigh` to `struct sysinfo`

The struct layout is modified with this commit because new fields are
added in the middle, keeping original Linux order.

Reviewed by: manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D37932

linuxkpi: Add `pin_user_pages*()` functions

They were defined in the i915 DRM driver. I move the code in linuxkpi so
it can benefit other drivers.

Reviewed by: manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D38078

linuxkpi: Define `is_cow_mapping(flags)` as false

Reviewed by: manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D38158

linuxkpi: Add `io_mapping_map_user()` and `remap_pfn_range()`

The code comes from the i915 DRM driver.

In Linux commits b739f125e4ebd73d10ed30a856574e13649119ed and
b12d691ea5e01db42ccf3b4207e57cb3ce7cfe91 (Linux 5.13), the i915 DRM
driver dropped specific implementations to use Linux generic functions.
Therefore I moved the FreeBSD code from that i915 driver to linuxkpi.

However, these commits were later reverted (also in Linux 5.13) so the
i915 driver doesn't use these functions. But perhaps it will help in the
future.

To sum up, the code comes from the i915 DRM driver but it doesn't use it
(i.e. it continues to use its internal implementation).

Reviewed by: manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D38088

linuxkpi: Introduce `vma_set_file()`

This code was moved from the i915 driver in Linux 5.11.

Reviewed by: manu
Approved by: manu
Differential Revision: https://reviews.freebsd.org/D36957

linuxkpi: Include <linux/shinker.h> in <linux/mm.h>

This is done like this in Linux.

Reviewed by: emaste manu
Approved by: emaste manu
Differential Revision: https://reviews.freebsd.org/D36956

linuxkpi: Add trylock_page and unlock_page

Simple wrapper around vm_page_trylock and vm_page_unlock.

Reviewed by: hselasky
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D36019

linuxkpi: Add mmap_lock.h

This contain mmap_read_lock, mmap_read_unlock and mmap_write_lock_killable
which are abstraction around down_read, up_read and down_write_killable.
Note that in Linux 5.8 mmap_sem was renamed to mmap_lock.
We might want to do the same at some point but some drivers still uses
the old mmap locking API.

Reviewed by: bz
MFC after: 2 weeks
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D34297

LinuxKPI: Constantly use _LINUXKPI_ prefix in include guards

MFC after: 1 week
Reviewed by: bz, emaste, hselasky, manu
Differential Revision: https://reviews.freebsd.org/D33562

linuxkpi: add PAGE_ALIGNED macro

Needed for the virtio-gpu driver.

Reviewed By: #linuxkpi, manu, bz, hselasky
Differential Revision: https://reviews.freebsd.org/D32366

LinuxKPI: Remove vma argument from fault method of vm_operations_struct

It is removed from Linux since 4.11.
In FreeBSD it results in several #ifdefs in drm-kmod.

Reviewed by: emaste, hselasky, manu
Differential revision: https://reviews.freebsd.org/D32169

LinuxKPI: Import linux_page.c and some dependent code from drm-kmod

No functional changes intended

Reviewed by: hselasky, manu, markj
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D32167

LinuxKPI: Update pte_fn_t definition to match Linux 5.3

Reviewed by: emaste, hselasky, manu
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D32166

LinuxKPI: Cast offset_in_page() parameter to unsigned long

to reduce number of patches in drm-kmod

Reviewed by: hselasky
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D32093

LinuxKPI: Factor out vmf_insert_pfn_prot() routine

from GEM and TTM page fault handlers and move it in to base system. This
code is tightly integrated with LKPI mmap support to belong to drm-kmod.

As this routine requires associated vm_object to be locked, it got
additional _locked suffix.

Reviewed by: hselasky, markj
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D32068

LinuxKPI: add fault_flag_allow_retry_first

Used by drm 5.7.

Reviewed by: bz, hselasky, nc
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D30673

Implement read-only VM_SHARED flag in the LinuxKPI.

For use by mmap(2) callbacks.

MFC after: 1 week
Sponsored by: Mellanox Technologies // NVIDIA Networking

Remove set_page_dirty_lock().

Its use of the page lock is incorrect, and it is not used by the DRM
modules.

Reviewed by: hselasky
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D23002

Change synchonization rules for vm_page reference counting.

There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.

Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.

The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.

The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.

__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.

Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486

Implement si_meminfo() in the LinuxKPI.

Submitted by: Johannes Lundberg <johalun0@gmail.com>
MFC after: 1 week
Sponsored by: Limelight Networks
Sponsored by: Mellanox Technologies

Implement zap_vma_ptes() for managed device objects.

Reviewed by: markj
Discussed with: hselasky
Tested by: zeising
MFC after: 1 week
Sponsored by: Mellanox Technologies
Differential revision: https://reviews.freebsd.org/D18606

Add "access" function pointer to the "vm_operations_struct" structure
in the LinuxKPI. While at it document when to use the "virtual_address" or
the "address" field in the "vm_fault" structure.

Submitted by: Johannes Lundberg <johalun0@gmail.com>
MFC after: 1 week
Sponsored by: Mellanox Technologies
Sponsored by: Limelight Networks

Make the vm_fault structure in the LinuxKPI compatible with
newer versions of the Linux kernel. No functional change.

MFC after: 1 week
Submitted by: Johannes Lundberg <johalun0@gmail.com>
Sponsored by: Mellanox Technologies
Sponsored by: Limelight Networks

linuxkpi: Do not leak pages on put.

When the owner of the wire reference releases the last reference, it
might be that the page was already attempted to be freed (but free
cannot be performed at that time due to wire). Check that the page
was removed from the object as the indicator of the free attempt and
finish the free operation if so.

Reported and tested by: Slava Shwartsman
Reviewed by: hselasky
Sponsored by: Mellanox Technologies
MFC after: 1 week

Only wire pages in the LinuxKPI instead of holding and wiring them.
This prevents the page daemon from regularly scanning the held pages.

Suggested by: kib @
MFC after: 1 week
Sponsored by: Mellanox Technologies

Remove prototypes for unimplemented LinuxKPI functions.

MFC after: 1 week

Fix compilation of LinuxKPI for PowerPC.

Found by: emaste @
MFC after: 1 week
Sponsored by: Mellanox Technologies

Define VM_READ, VM_WRITE and VM_EXEC in the LinuxKPI.

MFC after: 1 week
Sponsored by: Mellanox Technologies

Implement proper support for memory map operations in the LinuxKPI,
like open, close and fault using the character device pager.

Some notes about the implementation:

1) Linux drivers set the vm_ops and vm_private_data fields during a
mmap() call to indicate that the driver wants to use the LinuxKPI VM
operations. Else these operations are not used.

2) The vm_private_data pointer is associated with a VM area structure
and inserted into an internal LinuxKPI list. If the vm_private_data
pointer already exists, the existing VM area structure is used instead
of the allocated one which gets freed.

3) The LinuxKPI's vm_private_data pointer is used as the callback
handle for the FreeBSD VM object. The VM subsystem in FreeBSD has a
similar list to identify equal handles and will only call the
character device pager's close function once.

4) All LinuxKPI VM operations are serialized through the mmap_sem
sempaphore, which is per procedure, which prevents simultaneous access
to the shared VM area structure when receiving page faults.

Obtained from: kmacy @
MFC after: 1 week
Sponsored by: Mellanox Technologies

Implement a series of physical page management related functions in
the LinuxKPI for accessing user-space memory in the kernel.

Add functions to hold and wire physical page(s) based on a given range
of user-space virtual addresses.

Add functions to get and put a reference on, wire, hold, mark
accessed, copy and dirty a physical page.

Add new VM related structures and defines as a preparation step for
advancing the memory map capabilities of the LinuxKPI.

Add function to figure out if a virtual address was allocated using
malloc().

Add function to convert a virtual kernel address into its physical
page pointer.

Obtained from: kmacy @
MFC after: 1 week
Sponsored by: Mellanox Technologies

Implement minimalistic memory mapping structure, struct mm_struct, and
some associated helper functions in the LinuxKPI. Let the existing
linux_alloc_current() function allocate and initialize the new
structure and let linux_free_current() drop the refcount on the memory
mapping structure. When the mm_struct's refcount reaches zero, the
structure is freed.

Obtained from: kmacy @
MFC after: 1 week
Sponsored by: Mellanox Technologies

Finish process of moving the LinuxKPI module into the default kernel build.

- Move all files related to the LinuxKPI into sys/compat/linuxkpi and
its subfolders.
- Update sys/conf/files and some Makefiles to use new file locations.
- Added description of COMPAT_LINUXKPI to sys/conf/NOTES which in turn
adds the LinuxKPI to all LINT builds.
- The LinuxKPI can be added to the kernel by setting the
COMPAT_LINUXKPI option. The OFED kernel option no longer builds the
LinuxKPI into the kernel. This was done to keep the build rules for
the LinuxKPI in sys/conf/files simple.
- Extend the LinuxKPI module to include support for USB by moving the
Linux USB compat from usb.ko to linuxkpi.ko.
- Bump the FreeBSD_version.
- A universe kernel build has been done.

Reviewed by: np @ (cxgb and cxgbe related changes only)
Sponsored by: Mellanox Technologies