Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC vm_page_alloc_contig()

This function will be used by TTM, a memory manager used by the Radeon
KMS driver.

Compared to HEAD, the type of the "boundary" argument to several functions
is left as "unsigned long" (instead of "vm_paddr_t" in HEAD) to prevent an
API change in a stable branch.

The following revisions were merged in this single commit:

MFC r226928:
Eliminate vm_phys_bootstrap_alloc(). It was a failed attempt at
eliminating duplicated code in the various pmap implementations.

Micro-optimize vm_phys_free_pages().

Introduce vm_phys_free_contig(). It is fast routine for freeing an
arbitrary number of physically contiguous pages. In particular, it
doesn't require the number of pages to be a power of two.

Use "u_long" instead of "unsigned long".

Bruce Evans (bde@) has convinced me that the "boundary" parameters
to kmem_alloc_contig(), vm_phys_alloc_contig(), and
vm_reserv_reclaim_contig() should be of type "vm_paddr_t" and not
"u_long". Make this change.

MFC r227012:
Add support for VM_ALLOC_WIRED and VM_ALLOC_ZERO to vm_page_alloc_freelist()
and use these new options in the mips pmap.

Wake up the page daemon in vm_page_alloc_freelist() if the number of free
and cached pages becomes too low.

Tidy up vm_page_alloc_init(). In particular, add a comment about an
important restriction on its use.

Tested by: jchandra@

MFC r227072:
Simplify the implementation of the failure case in kmem_alloc_attr().

MFC r227127:
Wake up the page daemon in vm_page_alloc_freelist() if it couldn't
allocate the requested page because too few pages are cached or free.

Document the VM_ALLOC_COUNT() option to vm_page_alloc() and
vm_page_alloc_freelist().

Make style changes to vm_page_alloc() and vm_page_alloc_freelist(),
such as using a variable name that more closely corresponds to the
comments.

MFC r227568:
Refactor the code that performs physically contiguous memory allocation,
yielding a new public interface, vm_page_alloc_contig(). This new function
addresses some of the limitations of the current interfaces, contigmalloc()
and kmem_alloc_contig(). For example, the physically contiguous memory that
is allocated with those interfaces can only be allocated to the kernel vm
object and must be mapped into the kernel virtual address space. It also
provides functionality that vm_phys_alloc_contig() doesn't, such as wiring
the returned pages. Moreover, unlike that function, it respects the low
water marks on the paging queues and wakes up the page daemon when
necessary. That said, at present, this new function can't be applied to all
types of vm objects. However, that restriction will be eliminated in the
coming weeks.

From a design standpoint, this change also addresses an inconsistency
between vm_phys_alloc_contig() and the other vm_phys_alloc*() functions.
Specifically, vm_phys_alloc_contig() manipulated vm_page fields that other
functions in vm/vm_phys.c didn't. Moreover, vm_phys_alloc_contig() knew
about vnodes and reservations. Now, vm_page_alloc_contig() is responsible
for these things.

Reviewed by: kib
Discussed with: jhb

Merge the second part of the unmapped I/O changes. This enables the
infrastructure in the block layer and UFS filesystem as well as a few
drivers. The list of MFC revisions is long, so I won't quote changelogs.

r248508,248510,248511,248512,248514,248515,248516,248517,248518,
248519,248520,248521,248550,248568,248789,248790,249032,250936

Submitted by: kib
Approved by: kib
Obtained from: Netflix

MFC r248280:
Add pmap function pmap_copy_pages().

MFC r240317
Simplify pmap_unmapdev().

Don't set PTE_W on the page table entry in pmap_kenter{,_attr}() on MIPS.

MFC r233122,r237086,r237228,r237264,r237290,r237404,r237414,r237513,r237551,
r237592,r237604,r237623,r237684,r237733,r237813,r237855,r238124,r238126,
r238163,r238414,r238610,r238889,r238970,r239072,r239137,r240126 (all by alc):

Add fine-grained PV chunk and list locking to the amd64 pmap, enabling
concurrent execution of the following functions on different pmaps:

pmap_change_wiring()
pmap_copy()
pmap_enter()
pmap_enter_object()
pmap_enter_quick()
pmap_page_exists_quick()
pmap_page_is_mapped()
pmap_protect()
pmap_remove()
pmap_remove_pages()

Requested and approved by: alc

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Split the vm_page flags PG_WRITEABLE and PG_REFERENCED into atomic
flags field. Updates to the atomic flags are performed using the atomic
ops on the containing word, do not require any vm lock to be held, and
are non-blocking. The vm_page_aflag_set(9) and vm_page_aflag_clear(9)
functions are provided to modify afalgs.

Document the changes to flags field to only require the page lock.

Introduce vm_page_reference(9) function to provide a stable KPI and
KBI for filesystems like tmpfs and zfs which need to mark a page as
referenced.

Reviewed by: alc, attilio
Tested by: marius, flo (sparc64); andreast (powerpc, powerpc64)
Approved by: re (bz)

- Move the PG_UNMANAGED flag from m->flags to m->oflags, renaming the flag
to VPO_UNMANAGED (and also making the flag protected by the vm object
lock, instead of vm page queue lock).
- Mark the fake pages with both PG_FICTITIOUS (as it is now) and
VPO_UNMANAGED. As a consequence, pmap code now can use use just
VPO_UNMANAGED to decide whether the page is unmanaged.

Reviewed by: alc
Tested by: pho (x86, previous version), marius (sparc64),
marcel (arm, ia64, powerpc), ray (mips)
Sponsored by: The FreeBSD Foundation
Approved by: re (bz)

With retirement of cpumask_t and usage of cpuset_t for representing a
mask of CPUs, pc_other_cpus and pc_cpumask become highly inefficient.

Remove them and replace their usage with custom pc_cpuid magic (as,
atm, pc_cpumask can be easilly represented by (1 << pc_cpuid) and
pc_other_cpus by (all_cpus & ~(1 << pc_cpuid))).

This change is not targeted for MFC because of struct pcpu members
removal and dependency by cpumask_t retirement.

MD review by: marcel, marius, alc
Tested by: pluknet
MD testing by: marcel, marius, gonzo, andreast

When iterating over a paging queue, explicitly check for PG_MARKER, instead
of relying on zeroed memory being interpreted as an empty PV list.

Reviewed by: kib

Add a new option, OBJPR_NOTMAPPED, to vm_object_page_remove(). Passing this
option to vm_object_page_remove() asserts that the specified range of pages
is not mapped, or more precisely that none of these pages have any managed
mappings. Thus, vm_object_page_remove() need not call pmap_remove_all() on
the pages.

This change not only saves time by eliminating pointless calls to
pmap_remove_all(), but it also eliminates an inconsistency in the use of
pmap_remove_all() versus related functions, like pmap_remove_write(). It
eliminates harmless but pointless calls to pmap_remove_all() that were being
performed on PG_UNMANAGED pages.

Update all of the existing assertions on pmap_remove_all() to reflect this
change.

Reviewed by: kib

etire the cpumask_t type and replace it with cpuset_t usage.

This is intended to fix the bug where cpu mask objects are
capped to 32. MAXCPU, then, can now arbitrarely bumped to whatever
value. Anyway, as long as several structures in the kernel are
statically allocated and sized as MAXCPU, it is suggested to keep it
as low as possible for the time being.

Technical notes on this commit itself:
- More functions to handle with cpuset_t objects are introduced.
The most notable are cpusetobj_ffs() (which calculates a ffs(3)
for a cpuset_t object), cpusetobj_strprint() (which prepares a string
representing a cpuset_t object) and cpusetobj_strscan() (which
creates a valid cpuset_t starting from a string representation).
- pc_cpumask and pc_other_cpus are target to be removed soon.
With the moving from cpumask_t to cpuset_t they are now inefficient
and not really useful. Anyway, for the time being, please note that
access to pcpu datas is protected by sched_pin() in order to avoid
migrating the CPU while reading more than one (possible) word
- Please note that size of cpuset_t objects may differ between kernel
and userland. While this is not directly related to the patch itself,
it is good to understand that concept and possibly use the patch
as a reference on how to deal with cpuset_t objects in userland, when
accessing kernland members.
- KTR_CPUMASK is changed and now is represented through a string, to be
set as the example reported in NOTES.

Please additively note that no MAXCPU is bumped in this patch, but
private testing has been done until to MAXCPU=128 on a real 8x8x2(htt)
machine (amd64).

Please note that the FreeBSD version is not yet bumped because of
the upcoming pcpu changes. However, note that this patch is not
targeted for MFC.

People to thank for the time spent on this patch:
- sbruno, pluknet and Nicholas Esborn (nick AT desert DOT net) tested
several revision of the patches and really helped in improving
stability of this work.
- marius fixed several bugs in the sparc64 implementation and reviewed
patches related to ktr.
- jeff and jhb discussed the basic approach followed.
- kib and marcel made targeted review on some specific part of the
patch.
- marius, art, nwhitehorn and andreast reviewed MD specific part of
the patch.
- marius, andreast, gonzo, nwhitehorn and jceel tested MD specific
implementations of the patch.
- Other people have made contributions on other patches that have been
already committed and have been listed separately.

Companies that should be mentioned for having participated at several
degrees:
- Yahoo! for having offered the machines used for testing on big
count of CPUs.
- The FreeBSD Foundation for having sponsored my devsummit attendance,
which has been instrumental.
- Sandvine for having offered offices and infrastructure during
development.

(I really hope I didn't forget anyone, if it happened I apologize in
advance).

Use correct types and fromats for physical address

- Use vm_paddr_t for pa in pmap_steal_memory()
- Use uintmax_t and %jx to ensure that physical address are printed
correctly in cpu_startup() and pmap_bootstrap()

Fix typos - remove duplicate "the".

PR: bin/154928
Submitted by: Eitan Adler <lists at eitanadler.com>
MFC after: 3 days

Make MSGBUF_SIZE kernel option a loader tunable kern.msgbufsize.

Submitted by: perryh pluto.rain.com (previous version)
Reviewed by: jhb
Approved by: kib (mentor)
Tested by: universe

Support for 64 bit PTEs on n32 and n64 compilation.

In n32 and n64, add support for physical address above 4GB by having
64 bit page table entries and physical addresses. Major changes are:
- param.h: update PTE sizes, masks and shift values to support 64 bit PTEs.
- param.h: remove DELAY(), mips_btop(same as atop), mips_ptob (same as
ptoa), and reformat.
- param.h: remove casting to unsigned long in trunc_page and round_page
since this will be used on physical addresses.
- _types.h: have 64 bit __vm_paddr_t for n32.
- pte.h: update TLB LO0/1 access macros to support 64 bit PTE
- pte.h: assembly macros for PTE operations.
- proc.h: md_upte is now 64 bit for n32 and n64.
- exception.S and swtch.S: use the new PTE macros for PTE operations.
- cpufunc.h: TLB_LO0/1 registers are 64bit for n32 and n64.
- xlr_machdep.c: Add memory segments above 4GB to phys_avail[] as they are
supported now.

Reviewed by: jmallett (earlier version)

Cleanup physical address and PTE types on MIPS.

1. Use vm_paddr_t for physical addresses.

There are a few places in the MIPS platform code where vm_offset_t is
used for physical addresses, change these to use vm_paddr_t:
- phys_avail[], physmem_desc[] arrays
- pmap_mapdev(), page_is_managed(), is_cacheable_mem() pmap_map() args
- local variables of various pmap functions

2. Change init_pte_prot() return from int to pt_entry_t, as this can be
64 bit when using 64 bit TLB entries.

3. Update printing of pt_entry_t and of vm_paddr_t to use 'j' format with
uintmax_t. This will be useful later if we plan to use 64bit phsical addr
on 32 bit n32 compilation.

Reviewed by: imp

Fix compilation when DEBUG option is enabled.

- remove unused code in mips/rmi/xlr_pci.c
- remove unused variable in mips/rmi/dev/nlge/if_nlge.c
- fix reference to old function in mips/mips/pmap.c

Reported by: Prabhath Raman (prabhath at netlogicmicro com)

UMA_MD_SMALL_ALLOC for mips.

Implement uma_small_alloc() and uma_small_free() for mips that allocates
pages from direct mapped memory. Uses the same mechanism as the page table
page allocator, so that we allocate from KSEG0 in 32 bit, and from XKPHYS
on 64 bit.

Reviewed by: alc, jmallett

1. Fix off by one errors in calls to MIPS_DIRECT_MAPPABLE, reported by alc@
2. Remove unnecessary #defines from vmparam.h

Submitted by: alc (2)
Reviewed by: alc (1)

Fix issue noted by alc while reviewing r215938:
The current implementation of vm_page_alloc_freelist() does not handle
order > 0 correctly. Remove order parameter to the function and use it
only for order 0 pages.

Submitted by: alc

Enforce that 'pmap_kenter()' is only used to establish cacheable mappings.

Mappings with other cacheability attributes can be established, if needed,
by using 'pmap_kenter_attr()'.

Suggested by: jchandra, imp

Enforce that pmap_mapdev() always returns uncacheable mappings.

Reviewed by: imp, jchandra, jmallett

Apply MIPS pmap clean up patch from alc@ (with minor change to KASSERT):

PMAP_DIAGNOSTIC was eliminated from amd64/i386, and, in fact, the
non-MIPS parts of the kernel, several years ago. Any of the interesting
checks were turned into KASSERT()s. Basically, the motivation was that
lots of people run with INVARIANTS but no one runs with DIAGNOSTIC.

panic strings needn't and shouldn't have a terminating newline.

Finally, there is one functional change. The sched_pin() in
pmap_remove_pages() is an artifact of the way we temporarily map page
table pages on i386. (The mappings are processor private. We don't do
a system-wide shootdown.) It isn't needed by MIPS.

Tested by: jchandra

Submitted by: alc

MIPS n64 support - continued...

1. On n64, use XKPHYS to map page table pages instead of KSEG0. Maintain
just one freepages list on n64.

The changes are mainly to introduce MIPS_PHYS_TO_DIRECT(pa),
MIPS_DIRECT_TO_PHYS(), which will use KSEG0 in 32 bit compilation
and XKPHYS in 64 bit compilation.

2. Change macro based PMAP_LMEM_MAP1(), PMAP_LMEM_MAP2(), PMAP_LMEM_UNMAP()
to inline functions.

3. Introduce MIPS_DIRECT_MAPPABLE(pa), which will further reduce the cases
in which we will need to have a special case for 64 bit compilation.

4. Update CP0 hazard definitions for CPU_RMI - the cpu does not need any
nops

Reviewed by: neel

Make return statements style(9) compliant in MIPS pmap.c

Reported by: neel

Reviewed by: neel (earlier version)

Implement pmap changes suggested by alc@:

1. Move dirty bit emulation code that is duplicted for kernel and user
in trap.c to a function pmap_emulate_modified() in pmap.c.

2. While doing dirty bit emulation, it is not necessary to update the
TLB entry on all CPUs using smp_rendezvous(), we can just update the
TLB entry on the current CPU, and let the other CPUs update their TLB
entry lazily if they get an exception.

Reviewed by: alc, neel

Optimization for pmap_kenter(), call pmap_update_page() only when
necessary. On SMP, pmap_update_page() can be costly as it needs a
a smp_rendezvous().

Reviewed by: alc, neel
Obtained from: jmallett (http://svn.freebsd.org/base/user/jmallett/octeon)

Re-arrange the pmap calls that use smp_rendezvous() on SMP, so that their
per-cpu variants are also available to be called. The per-cpu variants
are needed for some later optimizations.

Also remove unnecessary casts, do some style fixes.

Reviewed by: alc, neel

Fix compilation when DDB disabled. Adds 'ifdef DDB' around
DB_SHOW_COMMAND definitions.

loadandclear() for PTEs are not needed on MIPS. The PTEs are software
managed and we already take pmap lock for PTE operations(see r210922)

Reviewed by: alc

Fix issue reported by alc :

MIPS doesn't really need to use atomic_cmpset_int() in situations like
this because the software dirty bit emulation in trap.c acquires
the pmap lock. Atomics like this appear to be a carryover from i386
where the hardware-managed TLB might concurrently set the modified bit.

Reviewed by: alc

Fix the issue reported by alc:

pmap_page_wired_mappings() counts the number of pv entries for the
specified page that have the pv entry wired flag set to TRUE.
pmap_enter() correctly initializes this flag. However,
pmap_change_wiring() doesn't update the corresponding pv entry flag,
only the PTE. So, the count returned by pmap_page_wired_mappings()
will sometimes be wrong.

In the short term, the best fix would be to eliminate the pv entry
flag and use only the PTE. That flag is wasting non-trivial memory.

Remove pv_wired flag, and use PTE flag to count the wired mappings.

Reviewed by: alc

Add 3 level page tables for MIPS in n64.

- 32 bit compilation will still use old 2 level page tables
- re-arrange pmap code so that adding another level is easier
- pmap code for 3 level page tables for n64
- update TLB handler to traverse 3 levels in n64

Reviewed by: jmallett

Prepare for 3 level page tables for MIPS.

- Move page table second level shift and mask to param.h
- rename SEGOFSET to SEGMASK
- fix values for 64 bit maximum kernel and user addresses.

Redo the page table page allocation on MIPS, as suggested by
alc@.

The UMA zone based allocation is replaced by a scheme that creates
a new free page list for the KSEG0 region, and a new function
in sys/vm that allocates pages from a specific free page list.

This also fixes a race condition introduced by the UMA based page table
page allocation code. Dropping the page queue and pmap locks before
the call to uma_zfree, and re-acquiring them afterwards will introduce
a race condtion(noted by alc@).

The changes are :
- Revert the earlier changes in MIPS pmap.c that added UMA zone for
page table pages.
- Add a new freelist VM_FREELIST_HIGHMEM to MIPS vmparam.h for memory that
is not directly mapped (in 32bit kernel). Normal page allocations will first
try the HIGHMEM freelist and then the default(direct mapped) freelist.
- Add a new function 'vm_page_t vm_page_alloc_freelist(int flind, int
order, int req)' to vm/vm_page.c to allocate a page from a specified
freelist. The MIPS page table pages will be allocated using this function
from the freelist containing direct mapped pages.
- Move the page initialization code from vm_phys_alloc_contig() to a
new function vm_page_alloc_init(), and use this function to initialize
pages in vm_page_alloc_freelist() too.
- Split the function vm_phys_alloc_pages(int pool, int order) to create
vm_phys_alloc_freelist_pages(int flind, int pool, int order), and use
this function from both vm_page_alloc_freelist() and vm_phys_alloc_pages().

Reviewed by: alc

Merge jmallett@'s n64 work into HEAD.

Merge changes for initial n64 support in pmap.c. Use direct mapped (XKPHYS)
access for a lot of operations that earlier needed temporary mapping. Add
support for using XKSEG for kernel mappings.

Reviewed by: imp
Obtained from: jmallett (http://svn.freebsd.org/base/user/jmallett/octeon)

Merge jmallett@'s n64 work into HEAD - changeset 8

Updated PTE/PDE macros from http://svn.freebsd.org/base/user/jmallett/octeon
Introduce pmap_segshift() macro, use pmap_segmap() in place of pmap_pde, and
remove pmap_pde().

Approved by: rrs (mentor)
Obtained from: jmallett@

Merge jmallett@'s n64 work into HEAD - changeset 6

PTE flag cleanup from http://svn.freebsd.org/base/user/jmallett/octeon
- Rename PTE_xx flags to match their MIPS names
- Use the new pte_set/test/clear macros uniformly, instead of a mixture
of mips_pg_xxx(), pmap_pte_x() macros and direct access.
- Remove unused macros and defines from pte.h and pmap.c

Discussed on freebsd-mips@

Approved by: rrs(mentor), jmallett

Merge jmallett@'s n64 work into HEAD - changeset 5

Remove unnecessary locking and sched_pin() call while creating a temporary
mapping.

Changes from http://svn.freebsd.org/base/user/jmallett/octeon

Approved by: rrs (mentor), jmallett

Merge jmallett@'s n64 work into HEAD - changeset 4

Re-write tlb operations in C with a simpler API.
Update callers to use the new API.

Changes from http://svn.freebsd.org/base/user/jmallett/octeon

Approved by: rrs(mentor), jmallett

Relax one of the new assertions in pmap_enter() a little. Specifically,
allow pmap_enter() to be performed on an unmanaged page that doesn't have
VPO_BUSY set. Having VPO_BUSY set really only matters for managed pages.
(See, for example, pmap_remove_write().)

Reduce the scope of the page queues lock and the number of
PG_REFERENCED changes in vm_pageout_object_deactivate_pages().
Simplify this function's inner loop using TAILQ_FOREACH(), and shorten
some of its overly long lines. Update a stale comment.

Assert that PG_REFERENCED may be cleared only if the object containing
the page is locked. Add a comment documenting this.

Assert that a caller to vm_page_requeue() holds the page queues lock,
and assert that the page is on a page queue.

Push down the page queues lock into pmap_ts_referenced() and
pmap_page_exists_quick(). (As of now, there are no longer any pmap
functions that expect to be called with the page queues lock held.)

Neither pmap_ts_referenced() nor pmap_page_exists_quick() should ever
be passed an unmanaged page. Assert this rather than returning "0"
and "FALSE" respectively.

ARM:

Simplify pmap_page_exists_quick() by switching to TAILQ_FOREACH().

Push down the page queues lock inside of pmap_clearbit(), simplifying
pmap_clear_modify(), pmap_clear_reference(), and pmap_remove_write().
Additionally, this allows for avoiding the acquisition of the page
queues lock in some cases.

PowerPC/AIM:

moea*_page_exits_quick() and moea*_page_wired_mappings() will never be
called before pmap initialization is complete. Therefore, the check
for moea_initialized can be eliminated.

Push down the page queues lock inside of moea*_clear_bit(),
simplifying moea*_clear_modify() and moea*_clear_reference().

The last parameter to moea*_clear_bit() is never used. Eliminate it.

PowerPC/BookE:

Simplify mmu_booke_page_exists_quick()'s control flow.

Reviewed by: kib@

Don't set PG_WRITEABLE in init_pte_prot() (and thus pmap_enter()) unless
the page is managed.

Don't set the machine-independent layer's dirty field for the page being
mapped in init_pte_prot(). (The dirty field is only supposed to set when
a mapping is removed or write-protected and the page was managed and
modified.)

Determine whether or not to perform dirty bit emulation based on whether
or not the page is managed, i.e., pageable, not based on whether the page
is being mapped into the kernel address space. Nearly all of the kernel
address space consists of unmanaged pages, so this has neglible impact on
the overhead of dirty bit emulation for the kernel address space. However,
there can also exist unmanaged pages in the user address space. Previously,
dirty bit emulation was unnecessarily performed on these pages.

Tested by: jchandra@

Make vm_contig_grow_cache() extern, and use it when vm_phys_alloc_contig()
fails to allocate MIPS page table pages. The current usage of VM_WAIT in
case of vm_phys_alloc_contig() failure is not correct, because:

"There is no guarantee that any of the available free (or cached) pages
after the VM_WAIT will fall within the range of suitable physical
addresses. Every time this function sleeps and a single page is freed
(or cached) by someone else, this function will be reawakened. With
a little bad luck, you could spin indefinitely."

We also add low and high parameters to vm_contig_grow_cache() and
vm_contig_launder() so that we restrict vm_contig_launder() to the range
of pages we are interested in.

Reported by: alc

Reviewed by: alc
Approved by: rrs (mentor)

Fix a KASSERT() that was broken in r208665.

Reported by: jmallett

Eliminate a stale comment.

Merge portions of r208645 and supporting code from the i386 pmap:
When I pushed down the page queues lock into pmap_is_modified(), I created
an ordering dependence: A pmap operation that clears PG_WRITEABLE and calls
vm_page_dirty() must perform the call first. Otherwise, pmap_is_modified()
could return FALSE without acquiring the page queues lock because the page
is not (currently) writeable, and the caller to pmap_is_modified() might
believe that the page's dirty field is clear because it has not seen the
effect of the vm_page_dirty() call.

When I pushed down the page queues lock into pmap_is_modified(), I
overlooked one place where this ordering dependence is violated:
pmap_enter(). In a rare situation pmap_enter() can be called to replace a
dirty mapping to one page with a mapping to another page. (I say rare
because replacements generally occur as a result of a copy-on-write fault,
and so the old page is not dirty.) This change delays clearing PG_WRITEABLE
until after vm_page_dirty() has been called.

Fixing the ordering dependency also makes it easy to introduce a small
optimization: When pmap_enter() used to replace a mapping to one page with a
mapping to another page, it freed the pv entry for the first mapping and
later called the pv entry allocator for the new mapping. Now, pmap_enter()
attempts to recycle the old pv entry, saving two calls to the pv entry
allocator.

Simplify the inner loop of get_pv_entry(): While iterating over the page's
pv list, there is no point in checking whether or not the pv list is empty,
wait instead until the loop completes.

Fix lock order reversal, unlock page queue and pmap locks before
calling uma_zfree().
Also if needed, acquire page queue lock before modifying pte page
attributes.

Approved by: rrs (mentor)

Get rid of unused variable 'virtual_sys_start'.

Call VM_WAIT in pmap_ptpgzone_allocf() if M_WAITOK is set.
Removed unused variable.

Approved by: rrs (mentor)

Push down page queues lock acquisition in pmap_enter_object() and
pmap_is_referenced(). Eliminate the corresponding page queues lock
acquisitions from vm_map_pmap_enter() and mincore(), respectively. In
mincore(), this allows some additional cases to complete without ever
acquiring the page queues lock.

Assert that the page is managed in pmap_is_referenced().

On powerpc/aim, push down the page queues lock acquisition from
moea*_is_modified() and moea*_is_referenced() into moea*_query_bit().
Again, this will allow some additional cases to complete without ever
acquiring the page queues lock.

Reorder a few statements in vm_page_dontneed() so that a race can't lead
to an old reference persisting. This scenario is described in detail by a
comment.

Correct a spelling error in vm_page_dontneed().

Assert that the object is locked in vm_page_clear_dirty(), and restrict the
page queues lock assertion to just those cases in which the page is
currently writeable.

Add object locking to vnode_pager_generic_putpages(). This was the one
and only place where vm_page_clear_dirty() was being called without the
object being locked.

Eliminate an unnecessary vm_page_lock() around vnode_pager_setsize()'s call
to vm_page_clear_dirty().

Change vnode_pager_generic_putpages() to the modern-style of function
definition. Also, change the name of one of the parameters to follow
virtual memory system naming conventions.

Reviewed by: kib

Fix mips kernel build breakage caused by revision 208504.

Roughly half of a typical pmap_mincore() implementation is machine-
independent code. Move this code into mincore(), and eliminate the
page queues lock from pmap_mincore().

Push down the page queues lock into pmap_clear_modify(),
pmap_clear_reference(), and pmap_is_modified(). Assert that these
functions are never passed an unmanaged page.

Eliminate an inaccurate comment from powerpc/powerpc/mmu_if.m:
Contrary to what the comment says, pmap_mincore() is not simply an
optimization. Without a complete pmap_mincore() implementation,
mincore() cannot return either MINCORE_MODIFIED or MINCORE_REFERENCED
because only the pmap can provide this information.

Eliminate the page queues lock from vfs_setdirty_locked_object(),
vm_pageout_clean(), vm_object_page_collect_flush(), and
vm_object_page_clean(). Generally speaking, these are all accesses
to the page's dirty field, which are synchronized by the containing
vm object's lock.

Reduce the scope of the page queues lock in vm_object_madvise() and
vm_page_dontneed().

Reviewed by: kib (an earlier version)

- Use ptpgzone zone to allocate page table pages irrespective of the amount of
memory on a platform. Tested on the Sibyte with 256MB and 1GB memory
configurations.

- Replace vtophys() with MIPS_KSEG0_TO_PHYS() to convert a page table
page's virtual address to physical. We can safely do this because
page table pages are allocated out of KSEG0.

- Add an assertion to verify that when a page table page is freed it
contains all zeroes. We can now use it after allocation without
zeroing it.

On entry to pmap_enter(), assert that the page is busy. While I'm
here, make the style of assertion used by pmap_enter() consistent
across all architectures.

On entry to pmap_remove_write(), assert that the page is neither
unmanaged nor fictitious, since we cannot remove write access to
either kind of page.

With the push down of the page queues lock, pmap_remove_write() cannot
condition its behavior on the state of the PG_WRITEABLE flag if the
page is busy. Assert that the object containing the page is locked.
This allows us to know that the page will neither become busy nor will
PG_WRITEABLE be set on it while pmap_remove_write() is running.

Correct a long-standing bug in vm_page_cowsetup(). We cannot possibly
do copy-on-write-based zero-copy transmit on unmanaged or fictitious
pages, so don't even try. Previously, the call to pmap_remove_write()
would have failed silently.

This pushes all of JC's patches that I have in place. I
am now able to run 32 cores ok.. but I still will hang
on buildworld with a NFS problem. I suspect I am missing
a patch for the netlogic rge driver.

JC check and see if I am missing anything except your
core-mask changes

Obtained from: JC

Push down the page queues into vm_page_cache(), vm_page_try_to_cache(), and
vm_page_try_to_free(). Consequently, push down the page queues lock into
pmap_enter_quick(), pmap_page_wired_mapped(), pmap_remove_all(), and
pmap_remove_write().

Push down the page queues lock into Xen's pmap_page_is_mapped(). (I
overlooked the Xen pmap in r207702.)

Switch to a per-processor counter for the total number of pages cached.

Eliminate dead code.

Bug in the memory mapping module. The wrong
physaddr was being used in the macro (1 should be
used not 2)...

Obtained from: JC

On Alan's advice, rather than do a wholesale conversion on a single
architecture from page queue lock to a hashed array of page locks
(based on a patch by Jeff Roberson), I've implemented page lock
support in the MI code and have only moved vm_page's hold_count
out from under page queue mutex to page lock. This changes
pmap_extract_and_hold on all pmaps.

Supported by: Bitgravity Inc.

Discussed with: alc, jeffr, and kib

Adapt i386 r207205 to mips: Clearing PV_TABLE_REF and setting the page's
PG_REFERENCED flag in pmap_protect() can't really be justified, so don't
do it.

Resurrect pmap_is_referenced() and use it in mincore(). Essentially,
pmap_ts_referenced() is not always appropriate for checking whether or
not pages have been referenced because it clears any reference bits
that it encounters. For example, in mincore(), clearing the reference
bits has two negative consequences. First, it throws off the activity
count calculations performed by the page daemon. Specifically, a page
on which mincore() has called pmap_ts_referenced() looks less active
to the page daemon than it should. Consequently, the page could be
deactivated prematurely by the page daemon. Arguably, this problem
could be fixed by having mincore() duplicate the activity count
calculation on the page. However, there is a second problem for which
that is not a solution. In order to clear a reference on a 4KB page,
it may be necessary to demote a 2/4MB page mapping. Thus, a mincore()
by one process can have the side effect of demoting a superpage
mapping within another process!

Most MIPS systems have a comparatively-sparse physical memory layout. Switch
to using the sparse physseg layout in the VM system.

Large memory mappings are always CPU local and always done with interrupts
disabled. Be doubly-sure that we don't try to do a TLB shootdown on SMP
systems for those mappings.

Submitted by: C. Jayachandran

o) Add a VM find-space option, VMFS_TLB_ALIGNED_SPACE, which searches the
address space for an address as aligned by the new pmap_align_tlb()
function, which is for constraints imposed by the TLB. [1]
o) Add a kmem_alloc_nofault_space() function, which acts like
kmem_alloc_nofault() but allows the caller to specify which find-space
option to use. [1]
o) Use kmem_alloc_nofault_space() with VMFS_TLB_ALIGNED_SPACE to allocate the
kernel stack address on MIPS. [1]
o) Make pmap_align_tlb() on MIPS align addresses so that they do not start on
an odd boundary within the TLB, so that they are suitable for insertion as
wired entries and do not have to share a TLB entry with another mapping,
assuming they are appropriately-sized.
o) Eliminate md_realstack now that the kstack will be appropriately-aligned on
MIPS.
o) Increase the number of guard pages to 2 so that we retain the proper
alignment of the kstack address.

Reviewed by: [1] alc
X-MFC-after: Making sure alc has not come up with a better interface.

o) Make pcb_onfault a pointer rather than an obscure integer value.
o) Mask off PAGE_MASK bits in pmap_update_page, etc., rather than modifying the
badvaddr in trapframe. Some nearby interfaces already did this.
o) Make PTEs "unsigned int" for now, not "unsigned long" -- we are only ready
for them to be 32-bit on 64-bit platforms.
o) Rather than using pmap_segmap and calculating the offset into the page table
by hand in trap.c, use pmap_pte().
o) Remove unused quad_syscall variable in trap.c.
o) Log things for illegal instructions like we do for bad page faults.
o) Various cast cleanups related to how to print registers.
o) When logging page faults, show the page table information not just for the
program counter, but for the fault address.
o) Modify support.S to use ABI-neutral macros for operating on pointers.
o) Consistently use CALLFRAME_SIZ rather than STAND_FRAME_SIZE, etc.
o) Remove unused insque/remque functions.
o) Remove some coprocessor 0 accessor functions implemented in assembly that
are unused and have inline assembly counterparts.

o) Add NPDEPG, like NPTEPG but for PDEs.
o) Remove NBPG, PGOFSET and PGSHIFT. Use the standard names.
o) Remove some unused macros and move things from param.h to vmparam.h that
belong in the latter. (Actually, all of the kernel segment values, virtual
addresses, etc., belong in one place, but this is a step in the right
direction.)

o) Use inline functions to access coprocessor 0 registers rather than external
ones implemented using assembly.
o) Use TRAPF_USERMODE() consistently rather than USERMODE(). Eliminate
<machine/psl.h> as a result.
o) Use intr_*() rather than *intr(), consistently.
o) Use register_t instead of u_int in some trap code.
o) Merge some more endian-related macros to machine/asm.h from NetBSD.
o) Add PTR_LI macro, which loads an address with the correct sign-extension for
a pointer.
o) Restore interrupts when bailing out due to an excessive IRQ in
nexus_setup_intr().
o) Remove unused functions from psraccess.S.
o) Enter temporary virtual entries for large memory access into the page tables
rather than simply hoping they stay resident in the TLB and we don't need to
do a refill for them.
o) Abstract out large memory mapping setup/teardown using some macros.
o) Do mips_dcache_wbinv_range() when using temporary virtual addresses just
like we do when we can use the direct map.

o) Remove code related to VM_ALLOC_WIRED_TLB_PG_POOL, VM_KERNEL_ALLOC_OFFSET
and floating pages. They are unused and unsupported.

Destroy the pmap 'pm_mutex' in pmap_release() otherwise we will panic
subsequently in pmap_pinit() with the following signature:

panic: lock "pmap" 0xc7878bc8 already initialized

This bug was uncovered by the changes made to vm_map.c in r206140.

This change enables use of physical memory that is beyond the direct
mapped kseg0 region.

The basic idea is to use KVA from the kseg2 region for mapping page
table pages that lie beyond the direct mapped region.

The TLB miss handler can now recursively fault into the TLB invalid
handler if it dereferences a kseg2 page table page address that is not
in the TLB.

Tested by: JC (c.jayachandran@gmail.com)

Make the ddb command "show tlb" SMP friendly.

It now accepts an argument to dump out the tlb of a particular cpu.

o) Consistently use MIPS_KSEGn_TO_PHYS instead of MIPS_{,UN}CACHED_TO_PHYS etc.
Get rid of the macros that spell KSEG0 CACHED and KSEG1 UNCACHED.
o) Get rid of some nearby duplicated and unused macros.

Reviewed by: imp

SMP support for the mips port.

The platform that supports SMP currently is a SWARM with a dual-core Sibyte
processor. The kernel config file to use is SWARM_SMP.

Reviewed by: imp, rrs

Provide access to pcpu structures for SMP kernels.

The basic idea is to use a the same virtual address as a window onto
distinct physical memory locations - one per processor. The physical
address that you access through this mapping depends on which cpu you
are currently executing on. We can now use the same virtual address
on any processor to access its per-cpu area.

The details are:

- The virtual address for 'struct pcpu *pcpup' is obtained by
stealing 2 pages worth of KVA in pmap_bootstrap().

- The mapping from the constant virtual address to a distinct
physical page is done in cpu_pcpu_init() through a wired TLB entry.

- A side-effect of this is that we reserve 2 pages worth of memory
for the pcpu but in reality it needs much less than that. The unused
memory is now used as the boot stack for the BSP and APs.

Remove SMP-specific bits from locore.S. The plan is to use a separate
mpboot.S for AP bootstrap.

Discussed on: freebsd-mips

Approved by: imp (mentor)

Follow Neel's suggestion and switch to using
restoreint() in combination with saving off the
old level. That way we don't blast out the old
level.

For our memory re-mapping trick to work
interrupts must be disabled through the
page_zero's or copys etc. Note that the
temporary mapping used by panic's may
cause us pain since int's may not be disabled.
When we get dumps working we may have to revist
this. Note that with this fix the build got
much much further.. until it hung on disk IO (I
would imagine thats the rge/msgring driver acting
up).

Fix two of the extended memory hacks. The copy pages
routine in one place was setting the valid2 bit to
2 not 1. This meant the PTE was NOT valid and so
you would crash.

In Zero Page there was a incorrect setting of
the valid bit AFTER the actual zero (opps)..

Hopefully this will fix the 0xc0000000 crashes
that I have been seeing (unless of course there are
other problems with these old hacks of mine to get
to memory above 512Meg)

Set the svn:eol-style = native and svn:mime-type = text/plain
properties on all files in this tree.

Submitted by: rpaulo@

Merge from projects/mips to head by hand:

sorry for the huge firehose on this commit, it would be too tedious
to do file by file

r201881 | imp | 2010-01-08 20:08:22 -0700 (Fri, 08 Jan 2010) | 3 lines
Rename mips_pcpu_init to mips_pcpu0_init since it applies only to the
BSP. Provide a missing prototype.

r201880 | neel | 2010-01-08 19:17:14 -0700 (Fri, 08 Jan 2010) | 7 lines
Compute the target of the jump in the 'J' and 'JAL' instructions
correctly. The 256MB segment is formed by taking the top 4 bits
of the address of the instruction in the "branch delay" slot
as opposed to the 'J' or 'JAL' instruction itself.

r201845 | imp | 2010-01-08 15:48:21 -0700 (Fri, 08 Jan 2010) | 2 lines
Centralize initialization of pcpu, and set curthread early...

r201770 | neel | 2010-01-07 22:53:11 -0700 (Thu, 07 Jan 2010) | 4 lines
Add a DDB command "show pcb" to dump out the contents of a thread's PCB.

r201631 | neel | 2010-01-05 23:42:08 -0700 (Tue, 05 Jan 2010) | 5 lines
Remove all CFE-specific code from locore.S. The CFE entrypoint initialization
is now done in platform-specific code.

r201563 | neel | 2010-01-04 23:58:54 -0700 (Mon, 04 Jan 2010) | 6 lines
This change increases the size of the kernel stack for thread0 from
PAGE_SIZE to (2 * PAGE_SIZE). It depends on the memory allocated by
pmap_steal_memory() being aligned to a PAGE_SIZE boundary.

r200656 | imp | 2009-12-17 16:55:49 -0700 (Thu, 17 Dec 2009) | 7 lines
Place holder ptrace mips module. Not entirely sure what's required
here yet, so I've not connected it to the build. I think that we'll
need to move something into the processor specific part of the mips
port by requiring mips_cpu_ptrace or platform_cpu_ptrace be provided
by the ports to get/set processor specific registers, ala SSE
registers on x86.

r200342 | imp | 2009-12-09 18:42:44 -0700 (Wed, 09 Dec 2009) | 4 lines
app_descriptor_addr is unused (I know it is referened still). And
unnecessary since we pass in a3 unmodified to platform_start.
Eliminate it from here and kill one more TARGET_OCTEON in the process.

r199760 | imp | 2009-11-24 10:15:22 -0700 (Tue, 24 Nov 2009) | 2 lines
Add in Cavium's CID. Report what the unknown CID is.

r199755 | imp | 2009-11-24 09:53:58 -0700 (Tue, 24 Nov 2009) | 5 lines
looks like there's more to this patch than just this one file. I'll
leave it to neel@ to get all the relevant pieces into the tree.

r199754 | imp | 2009-11-24 09:32:31 -0700 (Tue, 24 Nov 2009) | 6 lines
Include opt_cputype.h for all .c and .S files referencing TARGET_OCTEON.
Spell ld script name right.
# for the most part, we need to enhance infrastructure to obviate the need
# for such an intrusive option.

r199753 | imp | 2009-11-24 09:30:29 -0700 (Tue, 24 Nov 2009) | 3 lines
Remove a comment that's bogus.
Include opt_cputype.h since TARGET_OCTEON moved there.

r199752 | imp | 2009-11-24 09:29:23 -0700 (Tue, 24 Nov 2009) | 4 lines
Make sure kstack0 is page aligned.
# this may have been from neel@ for the sibyte stuff

r199742 | imp | 2009-11-24 01:35:11 -0700 (Tue, 24 Nov 2009) | 8 lines
Move the hard-wiring of the dcache on octeon outside of the if
statement. When no caches support was added, it looks like
TARGET_OCTEON was bogusly moved inside the if. Also, include
opt_cputype.h to make TARGET_OCTEON actually active.

# now we die in pmap init somewhere... Most likely because 32MB of RAM is
# too tight given the load address we're using.

r199741 | imp | 2009-11-24 01:21:48 -0700 (Tue, 24 Nov 2009) | 2 lines
TARGET_OCTEON reqiures opt_cputype.h.

r199736 | imp | 2009-11-24 00:40:38 -0700 (Tue, 24 Nov 2009) | 2 lines
Prefer ANSI spellings of uintXX_t, etc.

r199598 | imp | 2009-11-20 09:30:35 -0700 (Fri, 20 Nov 2009) | 3 lines
Horrible kludge to make octeon32 work. I think a better way is to
move the generic code into the config files....

r199597 | imp | 2009-11-20 09:27:50 -0700 (Fri, 20 Nov 2009) | 4 lines
cast vaddr to uintptr_t before casting it to a bus_space_handle_t.
# I'm sure this indicates a problem, but I'm not sure what...

r199496 | gonzo | 2009-11-18 15:52:05 -0700 (Wed, 18 Nov 2009) | 5 lines
- Add cpu_init_interrupts function that is supposed to
prepeare stuff required for spinning out interrupts later
- Add API for managing intrcnt/intrnames arrays
- Some minor style(9) fixes

r199246 | neel | 2009-11-13 02:24:09 -0700 (Fri, 13 Nov 2009) | 10 lines
Make pmap_copy_page() L2-cache friendly by doing the copy through the
cacheable window on physical memory (KSEG0). On the Sibyte processor
going through the uncacheable window (KSEG1) bypasses both L1 and L2
caches so we may end up with stale contents in the L2 cache.

This also makes it consistent with the rest of the function that
uses cacheable mappings to copy pages.

Approved by: imp (mentor)

r198842 | gonzo | 2009-11-02 23:42:55 -0700 (Mon, 02 Nov 2009) | 3 lines
- Handle errors when adding children to nexus. This sittuation
might occure when there is dublicate of child's entry in hints

r198669 | rrs | 2009-10-30 02:53:11 -0600 (Fri, 30 Oct 2009) | 5 lines
With this commit our friend RMI will now compile. I have
not tested it and the chances of it running yet are about
ZERO.. but it will now compile. The hard part now begins,
making it run ;-)

r198569 | neel | 2009-10-28 23:18:02 -0600 (Wed, 28 Oct 2009) | 5 lines
Deal with overflow of the COUNT register correctly. The 'cycles_per_hz'
has nothing to do with the rollover.

r198550 | imp | 2009-10-28 11:03:20 -0600 (Wed, 28 Oct 2009) | 3 lines
Remove useless for statement. i isn't used after it.
Remove needless braces.

r198534 | gonzo | 2009-10-27 21:34:05 -0600 (Tue, 27 Oct 2009) | 8 lines
- Fix busdma sync: dcache invalidation operates on cache line aligned
addresses and could modify areas of memory that share the same cache
line at the beginning and at the ending of the buffer. In order to
prevent a data loss we save these chunks in temporary buffer before
invalidation and restore them afer it.
Idea suggested by: cognet

r198531 | gonzo | 2009-10-27 18:01:20 -0600 (Tue, 27 Oct 2009) | 3 lines
- Remove bunch of declared but not defined cach-related variables
- Add mips_picache_linesize and mips_pdcache_linesize variables

r198530 | gonzo | 2009-10-27 17:45:48 -0600 (Tue, 27 Oct 2009) | 3 lines
- Replace stubs with actual cache info
- minor style(9) fix

r198355 | neel | 2009-10-21 22:35:32 -0600 (Wed, 21 Oct 2009) | 11 lines
Remove redundant instructions from tlb.S
The "_MTC0 v0, COP_0_TLB_HI" is actually incorrect because v0 has not been
initialized at that point. It worked correctly because we subsequently
did the right thing and initialized TLB_HI correctly.
The "li v0, MIPS_KSEG0_START" is redundant because we do exactly the same
thing 2 instructions down.

r198354 | neel | 2009-10-21 20:51:31 -0600 (Wed, 21 Oct 2009) | 9 lines
Get rid of the hardcoded constants to define cacheable memory:
SDRAM_ADDR_START, SDRAM_ADDR_END and SDRAM_MEM_SIZE
Instead we now keep a copy of the memory regions enumerated by
platform-specific code and use that to determine whether an address
is cacheable or not.

r198310 | gonzo | 2009-10-20 17:13:08 -0600 (Tue, 20 Oct 2009) | 5 lines
- Commit missing part of "bt" fix: store PC register in pcb_context struct
in cpu_switch and use it in stack_trace function later. pcb_regs contains
state of the process stored by exception handler and therefor is not
valid for sleeping processes.

r198264 | neel | 2009-10-19 22:36:08 -0600 (Mon, 19 Oct 2009) | 5 lines
Fix a bug where we would think that the L1 instruction and data cache are
present even though the line size field in the CP0 Config1 register is 0.

r198208 | imp | 2009-10-18 09:21:48 -0600 (Sun, 18 Oct 2009) | 3 lines
Get the PC from the trap frame, since it isn't saved as part of the
pcb regs.

r198205 | imp | 2009-10-18 08:55:55 -0600 (Sun, 18 Oct 2009) | 3 lines
Use correct signature for MipsEmulateBranch. The other one doesn't
work for 64-bit compiles.

r198182 | gonzo | 2009-10-16 18:22:07 -0600 (Fri, 16 Oct 2009) | 11 lines
- Use PC/RA/SP values as arguments for stacktrace_subr instead of trapframe.
Context info could be obtained from other sources (see below) no only from
td_pcb field
- Do not show a0..a3 values unless they're obtained from the stack. These
are only confirmed values.
- Fix bt command in DDB. Previous implementation used thread's trapframe
structure as a source info for trace unwinding, but this structure
is filled only when exception occurs. Valid register values for sleeping
processes are in pcb_context array. For curthread use pc/sp/ra for current
frame

r198181 | gonzo | 2009-10-16 16:52:18 -0600 (Fri, 16 Oct 2009) | 2 lines
- Get rid of label_t. It came from NetBSD and was used only in one place

r198066 | gonzo | 2009-10-13 19:43:53 -0600 (Tue, 13 Oct 2009) | 5 lines
- Move stack tracing function to db_trace.c
- Axe unused extern MipsXXX declarations
- Move all declarations for functions in exceptions.S/swtch.S
from trap.c to respective headers

r197796 | gonzo | 2009-10-05 17:19:51 -0600 (Mon, 05 Oct 2009) | 2 lines
- Revert part of r197685 because this change leads to wrong data in cache.

r197685 | gonzo | 2009-10-01 14:05:36 -0600 (Thu, 01 Oct 2009) | 2 lines
- Sync caches properly when dealing with sf_buf

r197014 | imp | 2009-09-08 21:57:10 -0600 (Tue, 08 Sep 2009) | 2 lines
Ugly hack to get this to compile. I'm sure there's a better way...

r197013 | imp | 2009-09-08 21:54:55 -0600 (Tue, 08 Sep 2009) | 2 lines
First half of making this 64-bit clean: fix prototypes.

r196988 | gonzo | 2009-09-08 13:15:29 -0600 (Tue, 08 Sep 2009) | 2 lines
- MFC from head@196987

r196313 | imp | 2009-08-17 06:14:40 -0600 (Mon, 17 Aug 2009) | 2 lines
suword64 and csuword64. Needed by ELF64 stuff...

r196266 | imp | 2009-08-15 16:51:11 -0600 (Sat, 15 Aug 2009) | 5 lines
(1) Fix a few 32/64-bit bugs.
(2) Also, always allocate 2 pages for the stack to optimize TLB usage.
Submitted by: neel@ (2)

r196265 | imp | 2009-08-15 16:48:09 -0600 (Sat, 15 Aug 2009) | 2 lines
Various 32/64-bit confusion cleanups.

r196264 | imp | 2009-08-15 16:45:46 -0600 (Sat, 15 Aug 2009) | 6 lines
(1) Some CPUs have a range to map I/O cyces on the pci bus. So allow
them to work by allowding the nexus to assign ports.
(2) Remove some Octeon junk that shouldn't be necessary.
Submitted by: neel@ (#1) for SB1 port.

r196061 | gonzo | 2009-08-04 11:32:55 -0600 (Tue, 04 Aug 2009) | 2 lines
- Use register_t for registers values

r195984 | gonzo | 2009-07-30 17:48:29 -0600 (Thu, 30 Jul 2009) | 4 lines
- Properly unwind stack for functions with __noreturn__ attribute
Submitted by: Neelkanth Natu <neelnatu@yahoo.com>

r195983 | gonzo | 2009-07-30 17:29:59 -0600 (Thu, 30 Jul 2009) | 4 lines
- mark map as coherent if requested by flags
- explicitly set memory allocation method in map flags instead
of duplicating conditions for malloc/contigalloc

r195584 | imp | 2009-07-10 13:09:34 -0600 (Fri, 10 Jul 2009) | 3 lines
Use PTR_* macros for pointers, and not potentially mips64 unsafe
operations.

r195583 | imp | 2009-07-10 13:08:48 -0600 (Fri, 10 Jul 2009) | 2 lines
Use PTR_* macros to deal with pointers.

r195579 | imp | 2009-07-10 13:04:32 -0600 (Fri, 10 Jul 2009) | 2 lines
use ta0-ta3 rather than t4-t7 for n32/n64 goodness.

r195511 | gonzo | 2009-07-09 13:02:17 -0600 (Thu, 09 Jul 2009) | 3 lines
- Ooops, this debug code wasn't supposed to get into
final commit. My appologises.

r195478 | gonzo | 2009-07-08 16:28:36 -0600 (Wed, 08 Jul 2009) | 5 lines
- Port busdma code from FreeBSD/arm. This is more mature version
that takes into account all limitation to DMA memory (boundaries,
alignment) and implements bounce pages.
- Add BUS_DMASYNC_POSTREAD case to bus_dmamap_sync_buf

r195438 | imp | 2009-07-08 00:00:18 -0600 (Wed, 08 Jul 2009) | 2 lines
Turns out this code was right, revert last change.

r195429 | gonzo | 2009-07-07 13:55:09 -0600 (Tue, 07 Jul 2009) | 5 lines
- Move dpcpu initialization to mips_proc0_init. It's
more appropriate place for it. Besides dpcpu_init
requires pmap module to be initialized and calling it
int pmap.c hangs the system

r195399 | imp | 2009-07-06 01:49:24 -0600 (Mon, 06 Jul 2009) | 2 lines
Prefer uintptr_t to int cast here.

r195398 | imp | 2009-07-06 01:48:31 -0600 (Mon, 06 Jul 2009) | 3 lines
Better types for 64-bit compatibility. Use %p and cast to void * and
prefer uintptr_t to other int-type casts.

r195397 | imp | 2009-07-06 01:47:39 -0600 (Mon, 06 Jul 2009) | 2 lines
No need to force mips32 here.

r195396 | imp | 2009-07-06 01:46:13 -0600 (Mon, 06 Jul 2009) | 3 lines
Pass in the uint64 value, rather than a pointer to it. that's what
the function expects...

r195395 | imp | 2009-07-06 01:45:02 -0600 (Mon, 06 Jul 2009) | 3 lines
Use ta0 instead of t4 and ta1 instead of t5. These map to the same
registers on O32 builds, but t4 and t5 don't exist on N32 or N64.

r195394 | imp | 2009-07-06 01:43:50 -0600 (Mon, 06 Jul 2009) | 3 lines
Use better casts for passing the small integer as a pointer here.
Basically, replace int with uintptr_t.

r195393 | imp | 2009-07-06 01:42:54 -0600 (Mon, 06 Jul 2009) | 5 lines
(1) Improvements for SB1. only allow real memory to be accessed.
(2) make compile n64 by using more-proper casts.
Submitted by: Neelkanth Natu (1)

r195373 | imp | 2009-07-05 09:23:54 -0600 (Sun, 05 Jul 2009) | 5 lines
(1) Use PTR_LA rather than bare la for N64 goodness (it is dla there)
(2) SB1 needs COHERENT policy, not cached for the config register
Submitted by: (2) Neelkanth Natu

r195372 | imp | 2009-07-05 09:22:22 -0600 (Sun, 05 Jul 2009) | 3 lines
use "PTR_LA" in preference to a bare la so it translates to dla on
64-bit ABIs.

r195371 | imp | 2009-07-05 09:21:35 -0600 (Sun, 05 Jul 2009) | 6 lines
Now that we define atomic_{load,store}_64 inline in atomic.h, we don't
need to define them here for the !N64 case.
We now define atomic_readandclear_64 in atomic.h, so no need to repeat
it here.

r195364 | imp | 2009-07-05 09:10:07 -0600 (Sun, 05 Jul 2009) | 5 lines
use %p in preference to 0x%08x for printing register_t values. Cast
them to void * first. This neatly solves the "how do I print a
register_t" problem because sizeof(void *) is always the same as
sizeof(register_t), afaik.

r195353 | imp | 2009-07-05 00:46:54 -0600 (Sun, 05 Jul 2009) | 6 lines
Publish PAGE_SHIFT to assembler
# we should likely phase out PGSHIFT
Submitted by: Neelkanth Natu

r195350 | imp | 2009-07-05 00:39:37 -0600 (Sun, 05 Jul 2009) | 7 lines
Switch to ABI agnostic ta0-ta3. Provide defs for this in the right
places. Provide n32/n64 register name defintions. This should have
no effect for the O32 builds that everybody else uses, but should help
make N64 builds possible (lots of other changes are needed for that).
Obtained from: NetBSD (for the regdef.h changes)

r195334 | imp | 2009-07-03 21:22:34 -0600 (Fri, 03 Jul 2009) | 6 lines
Move from using the lame invalid address I chose when trying to get
Octeon going... Turns out that you get tlb shutdowns with this...
Use PGSHIFT instead of PAGE_SHIFT.
Submitted by: Neelkanth Natu

r195147 | gonzo | 2009-06-28 15:01:00 -0600 (Sun, 28 Jun 2009) | 2 lines
- Replace casuword and casuword32 stubs with proper implementation

r195128 | gonzo | 2009-06-27 17:27:41 -0600 (Sat, 27 Jun 2009) | 4 lines
- Add support for handling TLS area address in kernel space.
From the userland point of view get/set operations are
performed using sysarch(2) call.

r195127 | gonzo | 2009-06-27 17:01:35 -0600 (Sat, 27 Jun 2009) | 4 lines
- Make cpu_set_upcall_kse conform MIPS ABI. T9 should be
the same as PC in subroutine entry point
- Preserve interrupt mask

r194938 | gonzo | 2009-06-24 20:15:04 -0600 (Wed, 24 Jun 2009) | 3 lines
- Invalidate cache in pmap_qenter. Fixes corruption of data
that comes through pipe (may be other bugs)

r194505 | gonzo | 2009-06-19 13:02:40 -0600 (Fri, 19 Jun 2009) | 5 lines
- Keep interrupts mask intact by RESTORE_CPU in MipsKernGenException
trap() function re-enables interrupts if exception happened with
interrupts enabled and therefor status register might be modified
by interrupt filters

r194277 | gonzo | 2009-06-15 20:36:21 -0600 (Mon, 15 Jun 2009) | 2 lines
- Remove debug printfs

r194275 | gonzo | 2009-06-15 19:43:33 -0600 (Mon, 15 Jun 2009) | 2 lines
- Handle KSEG0/KSEG1 addresses for /dev/mem as well. netstat requires it

r193491 | gonzo | 2009-06-05 03:21:03 -0600 (Fri, 05 Jun 2009) | 6 lines
- Status register should be set last in RESTORE_CPU in order
to prevent race over k0, k1 registers.
- Update interrupts mask in saved status register for
MipsUserIntr and MipsUserGenException. It might be
modified by intr filter or ithread.

r192864 | gonzo | 2009-05-26 16:40:12 -0600 (Tue, 26 May 2009) | 4 lines
- Replace CPU_NOFPU and SOFTFLOAT options with CPU_FPU. By default
we assume that there is no FPU, because majority of SoC does
not have it.

r192794 | gonzo | 2009-05-26 00:20:50 -0600 (Tue, 26 May 2009) | 5 lines
- Preserve INT_MASK fields in Status register across
context switches. They should be modified only by
interrupt setup/teardown and pre_ithread/post_ithread
functions

r192793 | gonzo | 2009-05-26 00:02:38 -0600 (Tue, 26 May 2009) | 2 lines
- Remove erroneus "break" instruction, it was meant for debug

r192792 | gonzo | 2009-05-26 00:01:17 -0600 (Tue, 26 May 2009) | 2 lines
- Remove now unused NetBSDism intr.h

r192791 | gonzo | 2009-05-25 23:59:05 -0600 (Mon, 25 May 2009) | 7 lines
- Provide proper pre_ithread/post_ithread functions for both
hard and soft interrupts
- Do not handle masked interrupts
- Do not write Cause register because most bytes are read-only and
writing the same byte to RW fields are pointless. And in case of
software interrupt utterly wrong

r192664 | gonzo | 2009-05-23 13:42:23 -0600 (Sat, 23 May 2009) | 4 lines
- cpu_establish_hardintr modifies INT_MASK of Status
register, so we should use disableintr/restoreintr that
modifies only IE bit.

r192655 | gonzo | 2009-05-23 12:00:20 -0600 (Sat, 23 May 2009) | 6 lines
- Remove stale comments
- Replace a1 with k1 to while restoring context. a1 was there by mistake,
interrupts are disabled at this point and it's safe to use k0, k1.
This code never was reached beacasue current Status register handling
prevented interrupta from user mode.

r192496 | gonzo | 2009-05-20 17:07:10 -0600 (Wed, 20 May 2009) | 4 lines
- Invalidate caches for respective areain KSEG0 in order
to prevent further overwriting of KSEG1 data with
writeback.

r192364 | gonzo | 2009-05-18 20:43:21 -0600 (Mon, 18 May 2009) | 6 lines
- Cleanup ticker initialization code. For some MIPS cpu Counter
register increments only every second cycle. The only timing
references for us is Count value. Therefore it's better to convert
frequencies related to it and use them. Besides cleanup this commit
fixes twice more then requested sleep interval problem.

r192176 | gonzo | 2009-05-15 20:34:03 -0600 (Fri, 15 May 2009) | 3 lines
- Add informational title for cache info lines to separate
them from environment variables dump

r192119 | gonzo | 2009-05-14 15:26:07 -0600 (Thu, 14 May 2009) | 3 lines
- Off by one check fix. Check for last address in region
to fit in KSEG1

r191841 | gonzo | 2009-05-05 20:55:43 -0600 (Tue, 05 May 2009) | 5 lines
- Use index ops in order to avoid TLBMiss exceptions when flushing caches
on mapping removal
- Writeback all VA for page that is being copied in pmap_copy_page to
guaranty up-to-date data in SDRAM

r191613 | gonzo | 2009-04-27 20:59:18 -0600 (Mon, 27 Apr 2009) | 4 lines
- When destroying va -> pa mapping writeback all caches or we may endup
with partial page content in SDRAM
- style(9) fix

r191583 | gonzo | 2009-04-27 12:46:57 -0600 (Mon, 27 Apr 2009) | 5 lines
- Use new spacebus
- Be a bit more verbose on failures
- style(9) fixes
- Use default rid value of 0 instead of MIPS_MEM_RID (0x20)

r191577 | gonzo | 2009-04-27 12:29:59 -0600 (Mon, 27 Apr 2009) | 4 lines
- Use naming convention the same as MIPS spec does: eliminate _sel1 sufix
and just use selector number. e.g. mips_rd_config_sel1 -> mips_rd_config1
- Add WatchHi/WatchLo accessors for selctors 1..3 (for debug purposes)

r191453 | gonzo | 2009-04-23 23:28:44 -0600 (Thu, 23 Apr 2009) | 4 lines
Fix cut'n'paste code. cfg3 should get the value of selector 3
Spotted by: thompa@

r191452 | gonzo | 2009-04-23 22:18:16 -0600 (Thu, 23 Apr 2009) | 2 lines
- Print supported CPU capabilities during stratup

r191448 | gonzo | 2009-04-23 21:38:51 -0600 (Thu, 23 Apr 2009) | 2 lines
- Fix whitespace to conform style(9)

r191282 | gonzo | 2009-04-19 16:02:14 -0600 (Sun, 19 Apr 2009) | 3 lines
- Make mips_bus_space_generic be of type bus_space_tag_t instead of
struct bus_space and update all relevant places.

r191084 | gonzo | 2009-04-14 20:28:26 -0600 (Tue, 14 Apr 2009) | 6 lines
Use FreeBSD/arm approach for handling bus space access: space tag is a pointer
to bus_space structure that defines access methods and hence every bus can
define own accessors. Default space is mips_bus_space_generic. It's a simple
interface to physical memory, values are read with regard to host system
byte order.

r191083 | gonzo | 2009-04-14 19:47:52 -0600 (Tue, 14 Apr 2009) | 4 lines
- Cleanout stale #ifdef'ed chunk of code
- Fix whitespaces
- Explicitly undefine NEXUS_DEBUG flag

r191079 | gonzo | 2009-04-14 16:53:22 -0600 (Tue, 14 Apr 2009) | 2 lines
- Revert changes accidentally killed by merge operation

------------------------------------------------------------------------
r187512 | gonzo | 2009-01-20 22:49:30 -0700 (Tue, 20 Jan 2009) | 4 lines
- Check if maddr/msize hints are there before setting hinted
resources to device
- Check for irq hint too

r187418 | gonzo | 2009-01-18 19:37:10 -0700 (Sun, 18 Jan 2009) | 4 lines
- Add trampoline stuff for bootloaders that do not support ELF
- Replace arm'ish KERNPHYSADDR/KERNVIRTADDR with
KERNLOADADDR/TRAMPLOADADDR and clean configs

Unbreak booting of FreeBSD/mips by merging r195429 from projects/mips:
- Move dpcpu initialization to mips_proc0_init. It's
more appropriate place for it. Besides dpcpu_init
requires pmap module to be initialized and calling it
int pmap.c hangs the system

o Introduce vm_sync_icache() for making the I-cache coherent with
the memory or D-cache, depending on the semantics of the platform.
vm_sync_icache() is basically a wrapper around pmap_sync_icache(),
that translates the vm_map_t argumument to pmap_t.
o Introduce pmap_sync_icache() to all PMAP implementation. For powerpc
it replaces the pmap_page_executable() function, added to solve
the I-cache problem in uiomove_fromphys().
o In proc_rwmem() call vm_sync_icache() when writing to a page that
has execute permissions. This assures that when breakpoints are
written, the I-cache will be coherent and the process will actually
hit the breakpoint.
o This also fixes the Book-E PMAP implementation that was missing
necessary locking while trying to deal with the I-cache coherency
in pmap_enter() (read: mmu_booke_enter_locked).

The key property of this change is that the I-cache is made coherent
*after* writes have been done. Doing it in the PMAP layer when adding
or changing a mapping means that the I-cache is made coherent *before*
any writes happen. The difference is key when the I-cache prefetches.

Add a new type of VM object: OBJT_SG. An OBJT_SG object is very similar to
a device pager (OBJT_DEVICE) object in that it uses fictitious pages to
provide aliases to other memory addresses. The primary difference is that
it uses an sglist(9) to determine the physical addresses for a given offset
into the object instead of invoking the d_mmap() method in a device driver.

Reviewed by: alc
Approved by: re (kensmith)
MFC after: 2 weeks

Implement a facility for dynamic per-cpu variables.
- Modules and kernel code alike may use DPCPU_DEFINE(),
DPCPU_GET(), DPCPU_SET(), etc. akin to the statically defined
PCPU_*. Requires only one extra instruction more than PCPU_* and is
virtually the same as __thread for builtin and much faster for shared
objects. DPCPU variables can be initialized when defined.
- Modules are supported by relocating the module's per-cpu linker set
over space reserved in the kernel. Modules may fail to load if there
is insufficient space available.
- Track space available for modules with a one-off extent allocator.
Free may block for memory to allocate space for an extent.

Reviewed by: jhb, rwatson, kan, sam, grehan, marius, marcel, stas

pmap_enter() *must* set PG_WRITEABLE on the given page if it creates a
mapping that permits write access. Otherwise, pmap_remove_write() will not
remove write access from any of the page's mappings.

Give pmap_enter()'s third parameter the same name that it has on amd64 and
i386. Otherwise, my next to last commit (r192628) to this file doesn't
actually compile.

When a page is mapped for write access on a read fault, the PTE should be
configured to trap on a write access unless *all* of the page's dirty bits
are set.

Preset the modified bit in the PTE when pmap_enter() is called during a
write fault or while wiring a mapping that must support write access.

In general, this change should reduce the number of traps that occur for
the purpose of setting the modified bit. More specifically, this change
should prevent traps while holding locks in a sysctl handler. See
kern/kern_sysctl.c revisions 1.168 and 1.195 (svn r192160) for further
details.

Tested by: gonzo

A variety of changes:

Reimplement "kernel_pmap" in the standard way.

Eliminate unused variables. (These are mostly variables that were
discarded by the machine-independent layer after FreeBSD 4.x.)

Properly handle a vm_page_alloc() failure in pmap_init().

Eliminate dead or legacy (FreeBSD 4.x) code.

Eliminate unnecessary page queues locking.

Eliminate some excess white space.

Correct the synchronization of pmap_page_exists_quick().

Tested by: gonzo

MFamd64/i386
Introduce pmap_try_insert_pv_entry(), a function that conditionally
creates a pv entry if the number of entries is below the high water mark
for pv entries.

Introduce pmap_enter_quick_locked() and use it to reimplement
pmap_enter_object(). The old implementation was broken. For example,
it could block while holding a mutex lock.

Change pmap_enter_quick_locked() to fail rather than wait if it is
unable to allocate a page table page. This prevents a race between
pmap_enter_object() and the page daemon. Specifically, an inactive
page that is a successor to the page that was given to
pmap_enter_quick_locked() might become a cache page while
pmap_enter_quick_locked() waits and later pmap_enter_object() maps
the cache page violating the invariant that cache pages are never
mapped. Similarly, change
pmap_enter_quick_locked() to call pmap_try_insert_pv_entry() rather
than pmap_insert_entry(). Generally speaking,
pmap_enter_quick_locked() is used to create speculative mappings. So,
it should not try hard to allocate memory if free memory is scarce.

Tested by: gonzo

make loop clearer that it isn't a mistake...

Rewrite get_pv_entry() to match expectations of the rest of the
kernel. Rather than just kick off the page daemon, we actively retire
more mappings. The inner loop now looks a lot like the inner loop of
pmap_remove_all.

Also, get_pv_entry can't return NULL now, so remove panic if it did.

Reviewed by: alc@

- pmap_track_modified was retired in r178606. Reintroducing it was a mistake.

Spotted by: alc@

MFp4:

- Add debug output
- Fix pmap_zero_page and related places: use uncached segments and invalidate
cache after zeroing memory.
- Do not test for modified bit if it's not neccessary
(merged from mips-juniper p4 branch)
- Some #includes reorganization

opt_msgbuf.h is needed for MSGBUF_SIZE overrides, if any.

Submitted by: alc@

Remove critical_enter() and critical_exit(): pmap_growkernel is called
with kernel_map->system_mtx held so these aren't needed. Add an
assertion to make sure this is the case.

Also, fix a minor style(9) nit.

Reviewed by: alc@

Remove stale comment. splhigh -> critical_enter transition happend
some time ago, and no longer needs special attention.

In FreeBSD 7.0 and beyond, pmap_growkernel() should pass VM_ALLOC_INTERRUPT
to vm_page_alloc() instead of VM_ALLOC_SYSTEM.

Retire pmap_addr_hint(). It is no longer used.

Add a stub for pmap_align_superpage() on machines that don't (yet)
implement pmap-level support for superpages.

Remove an XXX comment. sys/amd64/amd64/pmap.c revision 1.516 explains why
"pmap_initialized" no longer exists.

MFamd64 revision 1.544/i386 revision 1.549
Retire pmap_track_modified(). We no longer need it because we do not
create managed mappings within the clean submap. To prevent regressions,
add assertions blocking the creation of managed mappings within the clean
submap.

Approved by: imp

FreeBSD/mips port. The FreeBSD/mips port targets mips32, mips64,
mips32r2 and mips64r2 (and close relatives) processors. There
presently is support for ADMtek ADM5120, A mips 4Kc in a malta board,
the RB533 routerboard (based on IDT RC32434) and some preliminary
support for sibtye/broadcom designs. Other hardware support will be
forthcomcing.

This port boots multiuser under gxemul emulating the malta board and
also bootstraps on the hardware whose support is forthcoming...

Oleksandr Tymoshenko, Wojciech Koszek, Warner Losh, Olivier Houchard,
Randall Stewert and others that have contributed to the mips2 and/or
mips2-jnpr perforce branches. Juniper contirbuted a generic mips port
late in the life cycle of the misp2 branch. Warner Losh merged the
mips2 and Juniper code bases, and others list above have worked for
the past several months to get to multiuser.

In addition, the mips2 work owe a debt to the trail blazing efforts of
the original mips branch in perforce done by Juli Mallett.