MFC r318479:
Don't bother enqueuing a page immediately before freeing it.

MFC r318476, r318478:
Fix up some kern_yield() usages.

MFC r308612,r308614:

r308612:
Allow DRM2 code to be built on platforms without AGP. This patch is taken
from original drm-3.8 code.
r308614:
Allow embeding DRM2 code into kernel. It's usefull for development (for
netboot) and it also helps to boot FreeBSD on some embeded platforms (where
we must boot kernel directly, without standard boot loader).

MFC r307218:
Fix a race in vm_page_busy_sleep(9).

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

sys/dev: minor spelling fixes.

Most affect comments, very few have user-visible effects.

etc: minor spelling fixes.

Mostly comments but also some user-visible strings.

MFC after: 2 weeks

Handle the driver KPI change from r292373. Ensure that managed device
pagers fault routines always return with a result page, be it the
proper and valid result page, or initially passed freshly allocated
placeholder. Do not free the passed in page until we are able to
provide the replacement, and do not assign NULL to *mres.

Reported and tested by: dumbbell
Reviewed by: royger (who also verified that Xen code is safe)
Sponsored by: The FreeBSD Foundation

Introduce a new mechanism for relocating virtual pages to a new physical
address and use this mechanism when:

1. kmem_alloc_{attr,contig}() can't find suitable free pages in the physical
memory allocator's free page lists. This replaces the long-standing
approach of scanning the inactive and inactive queues, converting clean
pages into PG_CACHED pages and laundering dirty pages. In contrast, the
new mechanism does not use PG_CACHED pages nor does it trigger a large
number of I/O operations.

2. on 32-bit MIPS processors, uma_small_alloc() and the pmap can't find
free pages in the physical memory allocator's free page lists that are
covered by the direct map. Tested by: adrian

3. ttm_bo_global_init() and ttm_vm_page_alloc_dma32() can't find suitable
free pages in the physical memory allocator's free page lists.

In the coming months, I expect that this new mechanism will be applied in
other places. For example, balloon drivers should use relocation to
minimize fragmentation of the guest physical address space.

Make vm_phys_alloc_contig() a little smarter (and more efficient in some
cases). Specifically, use vm_phys_segs[] earlier to avoid scanning free
page lists that can't possibly contain suitable pages.

Reviewed by: kib, markj
Glanced at: jhb
Discussed with: jeff
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D4444

A change to KPI of vm_pager_get_pages() and underlying VOP_GETPAGES().

o With new KPI consumers can request contiguous ranges of pages, and
unlike before, all pages will be kept busied on return, like it was
done before with the 'reqpage' only. Now the reqpage goes away. With
new interface it is easier to implement code protected from race
conditions.

Such arrayed requests for now should be preceeded by a call to
vm_pager_haspage() to make sure that request is possible. This
could be improved later, making vm_pager_haspage() obsolete.

Strenghtening the promises on the business of the array of pages
allows us to remove such hacks as swp_pager_free_nrpage() and
vm_pager_free_nonreq().

o New KPI accepts two integer pointers that may optionally point at
values for read ahead and read behind, that a pager may do, if it
can. These pages are completely owned by pager, and not controlled
by the caller.

This shifts the UFS-specific readahead logic from vm_fault.c, which
should be file system agnostic, into vnode_pager.c. It also removes
one VOP_BMAP() request per hard fault.

Discussed with: kib, alc, jeff, scottl
Sponsored by: Nginx, Inc.
Sponsored by: Netflix

drm/ttm: Drain taskqueue if taskqueue_cancel_timeout() returned an error

Before, this was done if `pending` was true. This is not what the
manpage suggests and not what was done elsewhere in the same file.

ttm_vm_page_alloc: use vm_page_alloc for pages without dma32 restriction

This change re-organizes code a little bit to extract common pieces
of ttm_alloc_new_pages() and ttm_get_pages() into dedicated functions.
Also, for requests without address restrictions regular vm_page_alloc()
is used.
Lastly, when vm_page_alloc_contig() fails we call VM_WAIT before calling
vm_pageout_grow_cache() to ensure that there is enough free pages at all.

Reviewed by: kib
MFC after: 15 days

drm: Update the device-independent code to match Linux 3.8.13

This update brings few features:
o Support for the setmaster/dropmaster ioctls. For instance, they
are used to run multiple X servers simultaneously.
o Support for minor devices. The only user-visible change is a new
entry in /dev/dri but it is useless at the moment. This is a
first step to support render nodes [1].

The main benefit is to greatly reduce the diff with Linux (at the
expense of an unreadable commit diff). Hopefully, next upgrades will be
easier.

No updates were made to the drivers, beside adapting them to API
changes.

[1] https://en.wikipedia.org/wiki/Direct_Rendering_Manager#Render_nodes

Tested by: Many people
MFC after: 1 month
Relnotes: yes

If the vm_page_alloc_contig() failed in the ttm page allocators, do
what other callers of vm_page_alloc_contig() do, retry after
vm_pageout_grow_cache().

Sponsored by: The FreeBSD Foundation

Use default memory type for TTM buffer objects that may be cached.

MFC after: 1 week

Port the TTM AGP backend to the FreeBSD agp driver and enable AGP support
in the radeonkms driver.

Note: In PCI mode virtual addresses on the graphics card that map to system
RAM are translated to physical addresses by the graphics card itself. In
AGP mode address translation is done by the AGP chipset so fictitious
addresses appear on the system bus. For the CPU cache management to work
correctly when the CPU accesses this memory it needs to use the same
fictitious addresses (and let the chipset translate them) instead of using
the physical addresses directly.

Reviewed by: kib
MFC after: 1 month

drm: fix usage of vm_phys_fictitious_to_vm_page

vm_phys_fictitious_to_vm_page should not be called directly, even when
operating on a range that has been registered using
vm_phys_fictitious_reg_range. PHYS_TO_VM_PAGE should be used instead
because on arches that use VM_PHYSSEG_DENSE the page might come
directly from vm_page_array.

Reported by: nwhitehorn
Tested by: nwhitehorn, David Mackay <davidm.jx8p@gmail.com>
Sponsored by: Citrix Systems R&D

- Modify vm_page_unwire() and vm_page_enqueue() to directly accept
the queue where to enqueue pages that are going to be unwired.
- Add stronger checks to the enqueue/dequeue for the pagequeues when
adding and removing pages to them.

Of course, for unmanaged pages the queue parameter of vm_page_unwire() will
be ignored, just as the active parameter today.
This makes adding new pagequeues quicker.

This change effectively modifies the KPI. __FreeBSD_version will be,
however, bumped just when the full cache of free pages will be
evicted.

Sponsored by: EMC / Isilon storage division
Reviewed by: alc
Tested by: pho

drm/ttm, drm/radeon: Replace EINTR/ERESTART by ERESTARTSYS...

... for msleep/cv_*wait() return values, where wait_event*() is used
on Linux. ERESTARTSYS is the return code expected by callers when the
operation was interrupted.

For instance, this is the case of radeon_cs_ioctl() (radeon_cs.c): if
an error occurs, and the code isn't ERESTARTSYS (eg. EINTR), it logs an
error.

Note that ERESTARTSYS is defined as ERESTART, but this keeps callers'
code close to Linux.

Submitted by: avg@ (previous version)

ttm_bo_vm_lookup_rb: actually make use of the red-black tree

Previously the code would just iterate over the whole tree as if it were
just a list.

Without this change I would observe X server becoming more and more
jerky over time.

MFC after: 5 days

Fix undefined behavior: (1 << 31) is not defined as 1 is an int and this
shifts into the sign bit. Instead use (1U << 31) which gets the
expected result.

This fix is not ideal as it assumes a 32 bit int, but does fix the issue
for most cases.

A similar change was made in OpenBSD.

Discussed with: -arch, rdivacky
Reviewed by: cperciva

Partially revert r254880. The bitmap operations actually use long type now.

drm: Use the new drm_atomic.h, following the merge of projects/atomic64

Submitted by: jkim@

drm/ttm: Remove unused VM_ALLOC_DMA32 define

drm/ttm: Fix a reversed condition and add missing locks

This allows to run OpenGL applications on at least two test machines
with the Radeon driver.

Approved by: kib@

drm/ttm: Fix style in ttm_bo_release_mmap()

drm/ttm: Fix unmap of buffer object

Add a new ttm_bo_release_mmap() function to unmap pages in a
vm_object_t. Pages are freed when the buffer object is later released.

This function is called in ttm_bo_unmap_virtual_locked(), replacing
Linux' unmap_mapping_range(). In particular this is called when a buffer
object is about to be moved, so that its mapping is invalidated.

However, we don't use this function in ttm_bo_vm_dtor(), because the
vm_object_t is already marked as OBJ_DEAD and the pages will be
unmapped.

Approved by: kib@

ttm: "to_page->valid = VM_PAGE_BITS_ALL" before vm_page_dirty(to_page)

Approved by; kib@

drm/ttm: Improve comment in ttm_bo_vm_ctor() about lack of ref acquisition

Approved by: kib@

drm/ttm: When removing a range of pages from a pool, remove all of them

Submitted by: Mark Kettenis and Jonathan Gray from OpenBSD
Approved by: kib@

drm/ttm: Fix style errors

drm/ttm: Make ttm_bo_wait() call uninterruptible in page fault handler

This fixes a crash where a SIGLALRM, heavily used by X.Org, would
interrupt the wait, causing the page fault to fail and the "Xorg"
process to receive a SIGSEGV.

Approved by: kib@

drm/ttm: Import Linux commit ff7c60c580d9722f820d85c9c58ca55ecc1ee7c4

Author: Daniel Vetter <daniel.vetter@ffwll.ch>
Date: Mon Jan 14 15:08:14 2013 +0100

drm/ttm: fix fence locking in ttm_buffer_object_transfer, 2nd try

This fixes up

commit e8e89622ed361c46bf90ba4828e685a8b603f7e5
Author: Daniel Vetter <daniel.vetter@ffwll.ch>
Date: Tue Dec 18 22:25:11 2012 +0100

drm/ttm: fix fence locking in ttm_buffer_object_transfer

which leaves behind a might_sleep in atomic context, since the
fence_lock spinlock is held over a kmalloc(GFP_KERNEL) call. The fix
is to revert the above commit and only take the lock where we need it,
around the call to ->sync_obj_ref.

v2: Fixup things noticed by Maarten Lankhorst:
- Brown paper bag locking bug.
- No need for kzalloc if we clear the entire thing on the next line.
- check for bo->sync_obj (totally unlikely race, but still someone
else could have snuck in) and clear fbo->sync_obj if it's cleared
already.

Reported-by: Dave Airlie <airlied@gmail.com>
Cc: Jerome Glisse <jglisse@redhat.com>
Cc: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: Dave Airlie <airlied@redhat.com>

Approved by: kib@

drm/ttm: Import Linux commit 014b34409fb2015f63663b6cafdf557fdf289628

Author: Dave Airlie <airlied@gmail.com>
Date: Wed Jan 16 15:58:34 2013 +1000

ttm: on move memory failure don't leave a node dangling

if we have a move notify callback, when moving fails, we call move notify
the opposite way around, however this ends up with *mem containing the mm_node
from the bo, which means we double free it. This is a follow on to the previous
fix.

Reviewed-by: Jerome Glisse <jglisse@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

Approved by: kib@

drm/ttm: Import Linux commit 630541863b29f88c7ab34e647758344e4cd1eafd

Author: Dave Airlie <airlied@gmail.com>
Date: Wed Jan 16 14:25:44 2013 +1000

ttm: don't destroy old mm_node on memcpy failure

When we are using memcpy to move objects around, and we fail to memcpy
due to lack of memory to populate or failure to finish the copy, we don't
want to destroy the mm_node that has been copied into old_copy.

While working on a new kms driver that uses memcpy, if I overallocated bo's
up to the memory limits, and eviction failed, then machine would oops soon
after due to having an active bo with an already freed drm_mm embedded in it,
freeing it a second time didn't end well.

Reviewed-by: Jerome Glisse <jglisse@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

Approved by: kib@

drm/ttm: Import Linux commit cc4c0c4de3c775be22072ec3251f2e581b63d9a0

Author: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Date: Tue Jan 15 14:57:28 2013 +0100

drm/ttm: unexport ttm_bo_wait_unreserved

All legitimate users of this function outside ttm_bo.c are gone, now
it's only an implementation detail.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>

Approved by: kib@

drm/ttm: Import Linux commit f2d476a110bc24fde008698ae9018c99e803e25c

Author: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Date: Tue Jan 15 14:57:10 2013 +0100

drm/ttm: use ttm_bo_reserve_slowpath_nolru in ttm_eu_reserve_buffers, v2

This requires re-use of the seqno, which increases fairness slightly.
Instead of spinning with a new seqno every time we keep the current one,
but still drop all other reservations we hold. Only when we succeed,
we try to get back our other reservations again.

This should increase fairness slightly as well.

Changes since v1:
- Increase val_seq before calling ttm_bo_reserve_slowpath_nolru and
retrying to take all entries to prevent a race.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>

Approved by: kib@

drm/ttm: Import Linux commit 5e45d7dfd74100d622f9cdc70bfd1f9fae1671de

Author: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Date: Tue Jan 15 14:57:05 2013 +0100

drm/ttm: add ttm_bo_reserve_slowpath

Instead of dropping everything, waiting for the bo to be unreserved
and trying over, a better strategy would be to do a blocking wait.

This can be mapped a lot better to a mutex_lock-like call.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>

Approved by: kib@

drm/ttm: Import Linux commit 7a1863084c9d90ce4b67d645bf9b0f1612e68f62

Author: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Date: Tue Jan 15 14:56:48 2013 +0100

drm/ttm: cleanup ttm_eu_reserve_buffers handling

With the lru lock no longer required for protecting reservations we
can just do a ttm_bo_reserve_nolru on -EBUSY, and handle all errors
in a single path.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>

drm/ttm: Import Linux commit 63d0a4195560362e2e00a3ad38fc331d34e1da9b

Author: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Date: Tue Jan 15 14:56:37 2013 +0100

drm/ttm: remove lru_lock around ttm_bo_reserve

There should no longer be assumptions that reserve will always succeed
with the lru lock held, so we can safely break the whole atomic
reserve/lru thing. As a bonus this fixes most lockdep annotations for
reservations.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>

Remove the deprecated VM_ALLOC_RETRY flag for the vm_page_grab(9).
The flag was mandatory since r209792, where vm_page_grab(9) was
changed to only support the alloc retry semantic.

Suggested and reviewed by: alc
Sponsored by: The FreeBSD Foundation

Different consumers of the struct vm_page abuse pageq member to keep
additional information, when the page is guaranteed to not belong to a
paging queue. Usually, this results in a lot of type casts which make
reasoning about the code correctness harder.

Sometimes m->object is used instead of pageq, which could cause real
and confusing bugs if non-NULL m->object is leaked. See r141955 and
r253140 for examples.

Change the pageq member into a union containing explicitly-typed
members. Use them instead of type-punning or abusing m->object in x86
pmaps, uma and vm_page_alloc_contig().

Requested and reviewed by: alc
Sponsored by: The FreeBSD Foundation

On all the architectures, avoid to preallocate the physical memory
for nodes used in vm_radix.
On architectures supporting direct mapping, also avoid to pre-allocate
the KVA for such nodes.

In order to do so make the operations derived from vm_radix_insert()
to fail and handle all the deriving failure of those.

vm_radix-wise introduce a new function called vm_radix_replace(),
which can replace a leaf node, already present, with a new one,
and take into account the possibility, during vm_radix_insert()
allocation, that the operations on the radix trie can recurse.
This means that if operations in vm_radix_insert() recursed
vm_radix_insert() will start from scratch again.

Sponsored by: EMC / Isilon storage division
Reviewed by: alc (older version)
Reviewed by: jeff
Tested by: pho, scottl

The soft and hard busy mechanism rely on the vm object lock to work.
Unify the 2 concept into a real, minimal, sxlock where the shared
acquisition represent the soft busy and the exclusive acquisition
represent the hard busy.
The old VPO_WANTED mechanism becames the hard-path for this new lock
and it becomes per-page rather than per-object.
The vm_object lock becames an interlock for this functionality:
it can be held in both read or write mode.
However, if the vm_object lock is held in read mode while acquiring
or releasing the busy state, the thread owner cannot make any
assumption on the busy state unless it is also busying it.

Also:
- Add a new flag to directly shared busy pages while vm_page_alloc
and vm_page_grab are being executed. This will be very helpful
once these functions happen under a read object lock.
- Move the swapping sleep into its own per-object flag

The KPI is heavilly changed this is why the version is bumped.
It is very likely that some VM ports users will need to change
their own code.

Sponsored by: EMC / Isilon storage division
Discussed with: alc
Reviewed by: jeff, kib
Tested by: gavin, bapt (older version)
Tested by: pho, scottl

Replace kernel virtual address space allocation with vmem. This provides
transparent layering and better fragmentation.

- Normalize functions that allocate memory to use kmem_*
- Those that allocate address space are named kva_*
- Those that operate on maps are named kmap_*
- Implement recursive allocation handling for kmem_arena in vmem.

Reviewed by: alc
Tested by: pho
Sponsored by: EMC / Isilon Storage Division

MFgem r251960: re-check the mgt device object for the requested page
after the object was relocked.

Tested by: dumbbell
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

Remove unneeded page lock around vm_page_insert().

Submitted by: alc

Don't busy the page unless we are likely to release the object lock.

Reviewed by: kib
Sponsored by: EMC / Isilon Storage Division

Do not call malloc(M_WAITOK) while bodev->fence_lock mutex is
held. The ttm_buffer_object_transfer() does not need the mutex locked
at all, except for the call to the driver sync_obj_ref() method.

Reported and tested by: dumbbell
MFC after: 2 weeks

drm/ttm: Fix a typo: s/pTTM]/[TTM]/

drm/ttm: Explain why we don't need to acquire a ref in ttm_bo_vm_ctor()

drm/ttm: Fix TTM buffer object refcount

This fixes memory leaks in the radeonkms driver.

Reviewed by: Konstantin Belousov (kib@)
Tested by: J.R. Oldroyd <jr@opal.com>

Switch the vm_object mutex to be a rwlock. This will enable in the
future further optimizations where the vm_object lock will be held
in read mode most of the time the page cache resident pool of pages
are accessed for reading purposes.

The change is mostly mechanical but few notes are reported:
* The KPI changes as follow:
- VM_OBJECT_LOCK() -> VM_OBJECT_WLOCK()
- VM_OBJECT_TRYLOCK() -> VM_OBJECT_TRYWLOCK()
- VM_OBJECT_UNLOCK() -> VM_OBJECT_WUNLOCK()
- VM_OBJECT_LOCK_ASSERT(MA_OWNED) -> VM_OBJECT_ASSERT_WLOCKED()
(in order to avoid visibility of implementation details)
- The read-mode operations are added:
VM_OBJECT_RLOCK(), VM_OBJECT_TRYRLOCK(), VM_OBJECT_RUNLOCK(),
VM_OBJECT_ASSERT_RLOCKED(), VM_OBJECT_ASSERT_LOCKED()
* The vm/vm_pager.h namespace pollution avoidance (forcing requiring
sys/mutex.h in consumers directly to cater its inlining functions
using VM_OBJECT_LOCK()) imposes that all the vm/vm_pager.h
consumers now must include also sys/rwlock.h.
* zfs requires a quite convoluted fix to include FreeBSD rwlocks into
the compat layer because the name clash between FreeBSD and solaris
versions must be avoided.
At this purpose zfs redefines the vm_object locking functions
directly, isolating the FreeBSD components in specific compat stubs.

The KPI results heavilly broken by this commit. Thirdy part ports must
be updated accordingly (I can think off-hand of VirtualBox, for example).

Sponsored by: EMC / Isilon storage division
Reviewed by: jeff
Reviewed by: pjd (ZFS specific review)
Discussed with: alc
Tested by: pho

drm: Fix a call to free(9) with an incorrect malloc type

While here, the call to free(9) is moved to drm_global.c, near the
initial malloc(9).

Reviewed by: Konstantin Belousov (kib@)

Fix build with gcc, do not use unnamed union.

Reported and tested by: gjb
MFC after: 1 month

Fix build with gcc, remove redundand declarations.

Reported and tested by: gjb
MFC after: 1 month

Import the preliminary port of the TTM.

The early commit is done to facilitate the off-tree work on the
porting of the Radeon driver.

Sponsored by: The FreeBSD Foundation
Debugged and tested by: dumbbell
MFC after: 1 month