mips: use asm-generic/mmu_context.h for no-op implementations

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
Acked-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
Cc: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
Cc: linux-mips@vger.kernel.org
Signed-off-by: Arnd Bergmann <arnd@arndb.de>

MIPS: MemoryMapID (MMID) Support

Introduce support for using MemoryMapIDs (MMIDs) as an alternative to
Address Space IDs (ASIDs). The major difference between the two is that
MMIDs are global - ie. an MMID uniquely identifies an address space
across all coherent CPUs. In contrast ASIDs are non-global per-CPU IDs,
wherein each address space is allocated a separate ASID for each CPU
upon which it is used. This global namespace allows a new GINVT
instruction be used to globally invalidate TLB entries associated with a
particular MMID across all coherent CPUs in the system, removing the
need for IPIs to invalidate entries with separate ASIDs on each CPU.

The allocation scheme used here is largely borrowed from arm64 (see
arch/arm64/mm/context.c). In essence we maintain a bitmap to track
available MMIDs, and MMIDs in active use at the time of a rollover to a
new MMID version are preserved in the new version. The allocation scheme
requires efficient 64 bit atomics in order to perform reasonably, so
this support depends upon CONFIG_GENERIC_ATOMIC64=n (ie. currently it
will only be included in MIPS64 kernels).

The first, and currently only, available CPU with support for MMIDs is
the MIPS I6500. This CPU supports 16 bit MMIDs, and so for now we cap
our MMIDs to 16 bits wide in order to prevent the bitmap growing to
absurd sizes if any future CPU does implement 32 bit MMIDs as the
architecture manuals suggest is recommended.

When MMIDs are in use we also make use of GINVT instruction which is
available due to the global nature of MMIDs. By executing a sequence of
GINVT & SYNC 0x14 instructions we can avoid the overhead of an IPI to
each remote CPU in many cases. One complication is that GINVT will
invalidate wired entries (in all cases apart from type 0, which targets
the entire TLB). In order to avoid GINVT invalidating any wired TLB
entries we set up, we make sure to create those entries using a reserved
MMID (0) that we never associate with any address space.

Also of note is that KVM will require further work in order to support
MMIDs & GINVT, since KVM is involved in allocating IDs for guests & in
configuring the MMU. That work is not part of this patch, so for now
when MMIDs are in use KVM is disabled.

Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: linux-mips@vger.kernel.org

MIPS: mm: Add set_cpu_context() for ASID assignments

When we gain MMID support we'll be storing MMIDs as atomic64_t values
and accessing them via atomic64_* functions. This necessitates that we
don't use cpu_context() as the left hand side of an assignment, ie. as a
modifiable lvalue. In preparation for this introduce a new
set_cpu_context() function & replace all assignments with cpu_context()
on their left hand side with an equivalent call to set_cpu_context().

To enforce that cpu_context() should not be used for assignments, we
rewrite it as a static inline function.

Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: linux-mips@vger.kernel.org

MIPS: mm: Unify ASID version checks

Introduce a new check_mmu_context() function to check an mm's ASID
version & get a new one if it's outdated, and a
check_switch_mmu_context() function which additionally sets up the new
ASID & page directory. Simplify switch_mm() & various
get_new_mmu_context() callsites in MIPS KVM by making use of the new
functions, which will help reduce the amount of code that requires
modification to gain MMID support.

Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: linux-mips@vger.kernel.org

MIPS: mm: Un-inline get_new_mmu_context

In preparation for adding MMID support to get_new_mmu_context() which
will increase the size of the function somewhat, move it from
asm/mmu_context.h into a C file.

Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: linux-mips@vger.kernel.org

MIPS: mm: Move drop_mmu_context() comment into appropriate block

drop_mmu_context() is preceded by a comment indicating what happens if
the mm provided is currently active on the local CPU. Move that comment
into the block that executes in this case, adjusting slightly to reflect
its new location.

Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: linux-mips@vger.kernel.org

MIPS: mm: Consolidate drop_mmu_context() has-ASID checks

If an mm does not have an ASID on the local CPU then drop_mmu_context()
is always redundant, since there's no context to "drop". Various callers
of drop_mmu_context() check whether the mm has been allocated an ASID
before making the call. Move that check into drop_mmu_context() and
remove it from callers to simplify them.

Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: linux-mips@vger.kernel.org

MIPS: mm: Avoid HTW stop/start when dropping an inactive mm

If drop_mmu_context() is called with an mm that is not currently active
on the local CPU then there's no need for us to stop & start a hardware
page table walker because it can't be fetching entries for the ASID
corresponding to the mm we're operating on.

Move the htw_stop() & htw_start() calls into the block which we run only
if the mm is currently active, in order to avoid the redundant work.

Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: linux-mips@vger.kernel.org

MIPS: mm: Remove redundant get_new_mmu_context() cpu argument

get_new_mmu_context() accepts a cpu argument, but implicitly assumes
that this is always equal to smp_processor_id() by operating on the
local CPU's TLB & icache.

Remove the cpu argument and have get_new_mmu_context() call
smp_processor_id() instead.

Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: linux-mips@vger.kernel.org

MIPS: mm: Remove redundant drop_mmu_context() cpu argument

The drop_mmu_context() function accepts a cpu argument, but it
implicitly expects that this is always equal to smp_processor_id() by
allocating & configuring an ASID on the local CPU when the mm is active
on the CPU indicated by the cpu argument.

All callers do provide the value of smp_processor_id() to the cpu
argument.

Remove the redundant argument and have drop_mmu_context() call
smp_processor_id() itself, making it clearer that the cpu variable
always represents the local CPU.

Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: linux-mips@vger.kernel.org

MIPS: mm: Define activate_mm() using switch_mm()

MIPS has separate definitions of activate_mm() & switch_mm() which are
identical apart from switch_mm() checking that the ASID is valid before
acquiring a new one.

We know that when activate_mm() is called cpu_context(X, mm) will be
zero, and this will never be considered a valid ASID because we never
allow the ASID version number to be zero, instead beginning with version
1 using asid_first_version(). Therefore switch_mm() will always allocate
a new ASID when called for a new task, meaning that it will behave
identically to activate_mm().

Take advantage of this to remove the duplication & define activate_mm()
using switch_mm() just like many other architectures do.

Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: linux-mips@vger.kernel.org

MIPS: Expand MIPS32 ASIDs to 64 bits

ASIDs have always been stored as unsigned longs, ie. 32 bits on MIPS32
kernels. This is problematic because it is feasible for the ASID version
to overflow & wrap around to zero.

We currently attempt to handle this overflow by simply setting the ASID
version to 1, using asid_first_version(), but we make no attempt to
account for the fact that there may be mm_structs with stale ASIDs that
have versions which we now reuse due to the overflow & wrap around.

Encountering this requires that:

1) A struct mm_struct X is active on CPU A using ASID (V,n).

2) That mm is not used on CPU A for the length of time that it takes
for CPU A's asid_cache to overflow & wrap around to the same
version V that the mm had in step 1. During this time tasks using
the mm could either be sleeping or only scheduled on other CPUs.

3) Some other mm Y becomes active on CPU A and is allocated the same
ASID (V,n).

4) mm X now becomes active on CPU A again, and now incorrectly has the
same ASID as mm Y.

Where struct mm_struct ASIDs are represented above in the format
(version, EntryHi.ASID), and on a typical MIPS32 system version will be
24 bits wide & EntryHi.ASID will be 8 bits wide.

The length of time required in step 2 is highly dependent upon the CPU &
workload, but for a hypothetical 2GHz CPU running a workload which
generates a new ASID every 10000 cycles this period is around 248 days.
Due to this long period of time & the fact that tasks need to be
scheduled in just the right (or wrong, depending upon your inclination)
way, this is obviously a difficult bug to encounter but it's entirely
possible as evidenced by reports.

In order to fix this, simply extend ASIDs to 64 bits even on MIPS32
builds. This will extend the period of time required for the
hypothetical system above to encounter the problem from 28 days to
around 3 trillion years, which feels safely outside of the realms of
possibility.

The cost of this is slightly more generated code in some commonly
executed paths, but this is pretty minimal:

| Code Size Gain | Percentage
-----------------------|----------------|-------------
decstation_defconfig | +270 | +0.00%
32r2el_defconfig | +652 | +0.01%
32r6el_defconfig | +1000 | +0.01%

I have been unable to measure any change in performance of the LMbench
lat_ctx or lat_proc tests resulting from the 64b ASIDs on either
32r2el_defconfig+interAptiv or 32r6el_defconfig+I6500 systems.

Signed-off-by: Paul Burton <paul.burton@mips.com>
Suggested-by: James Hogan <jhogan@kernel.org>
References: https://lore.kernel.org/linux-mips/80B78A8B8FEE6145A87579E8435D78C30205D5F3@fzex.ruijie.com.cn/
References: https://lore.kernel.org/linux-mips/1488684260-18867-1-git-send-email-jiwei.sun@windriver.com/
Cc: Jiwei Sun <jiwei.sun@windriver.com>
Cc: Yu Huabing <yhb@ruijie.com.cn>
Cc: stable@vger.kernel.org # 2.6.12+
Cc: linux-mips@vger.kernel.org

MIPS: Consistently declare TLB functions

Since at least the beginning of the git era we've declared our TLB
exception handling functions inconsistently. They're actually functions,
but we declare them as arrays of u32 where each u32 is an encoded
instruction. This has always been the case for arch/mips/mm/tlbex.c, and
has also been true for arch/mips/kernel/traps.c since commit
86a1708a9d54 ("MIPS: Make tlb exception handler definitions and
declarations match.") which aimed for consistency but did so by
consistently making the our C code inconsistent with our assembly.

This is all usually harmless, but when using GCC 7 or newer to build a
kernel targeting microMIPS (ie. CONFIG_CPU_MICROMIPS=y) it becomes
problematic. With microMIPS bit 0 of the program counter indicates the
ISA mode. When bit 0 is zero instructions are decoded using the standard
MIPS32 or MIPS64 ISA. When bit 0 is one instructions are decoded using
microMIPS. This means that function pointers become odd - their least
significant bit is one for microMIPS code. We work around this in cases
where we need to access code using loads & stores with our
msk_isa16_mode() macro which simply clears bit 0 of the value it is
given:

#define msk_isa16_mode(x) ((x) & ~0x1)

For example we do this for our TLB load handler in
build_r4000_tlb_load_handler():

u32 *p = (u32 *)msk_isa16_mode((ulong)handle_tlbl);

We then write code to p, expecting it to be suitably aligned (our LEAF
macro aligns functions on 4 byte boundaries, so (ulong)handle_tlbl will
give a value one greater than a multiple of 4 - ie. the start of a
function on a 4 byte boundary, with the ISA mode bit 0 set).

This worked fine up to GCC 6, but GCC 7 & onwards is smart enough to
presume that handle_tlbl which we declared as an array of u32s must be
aligned sufficiently that bit 0 of its address will never be set, and as
a result optimize out msk_isa16_mode(). This leads to p having an
address with bit 0 set, and when we go on to attempt to store code at
that address we take an address error exception due to the unaligned
memory access.

This leads to an exception prior to the kernel having configured its own
exception handlers, so we jump to whatever handlers the bootloader
configured. In the case of QEMU this results in a silent hang, since it
has no useful general exception vector.

Fix this by consistently declaring our TLB-related functions as
functions. For handle_tlbl(), handle_tlbs() & handle_tlbm() we do this
in asm/tlbex.h & we make use of the existing declaration of
tlbmiss_handler_setup_pgd() in asm/mmu_context.h. Our TLB handler
generation code in arch/mips/mm/tlbex.c is adjusted to deal with these
definitions, in most cases simply by casting the function pointers to
u32 pointers.

This allows us to include asm/mmu_context.h in arch/mips/mm/tlbex.c to
get the definitions of tlbmiss_handler_setup_pgd & pgd_current, removing
some needless duplication. Consistently using msk_isa16_mode() on
function pointers means we no longer need the
tlbmiss_handler_setup_pgd_start symbol so that is removed entirely.

Now that we're declaring our functions as functions GCC stops optimizing
out msk_isa16_mode() & a microMIPS kernel built with either GCC 7.3.0 or
8.1.0 boots successfully.

Signed-off-by: Paul Burton <paul.burton@mips.com>

MIPS: Schedule on CPUs we need to lose FPU for a mode switch

Commit 6b8322576e9d ("MIPS: Force CPUs to lose FP context during mode
switches") ensures that we react to PR_SET_FP_MODE prctl syscalls
quickly by broadcasting an IPI in order to cause CPUs to lose FPU access
when necessary. Whilst it achieves that, unfortunately it causes all
sorts of strange race conditions because:

1) The IPI may arrive at a point where the FPU is in the process of
being enabled, but that process is not yet complete leading to a
state we aren't prepared to handle. For example:

[ 370.215903] do_cpu invoked from kernel context![#1]:
[ 370.221064] CPU: 0 PID: 963 Comm: fp-prctl Not tainted 4.9.0-rc5-00323-g210db32-dirty #226
[ 370.229420] task: a8000000fd672e00 task.stack: a8000000fd630000
[ 370.235399] $ 0 : 0000000000000000 0000000000000001 0000000000000001 a8000000fd630000
[ 370.243882] $ 4 : a8000000fd672e00 0000000000000000 0000000000000453 0000000000000000
[ 370.252317] $ 8 : 0000000000000000 a8000000fd637c28 1000000000000000 0000000000000010
[ 370.260753] $12 : 00000000140084e0 ffffffff80109c00 0000000000000000 0000000000000002
[ 370.269179] $16 : ffffffff8092f080 a8000000fd672e00 ffffffff80107fe8 a8000000fd485000
[ 370.277612] $20 : ffffffff8084d328 ffffffff80940000 0000000000000009 ffffffff80930000
[ 370.286038] $24 : 0000000000000000 900000001612048c
[ 370.294476] $28 : a8000000fd630000 a8000000fd637ac0 ffffffff80937300 ffffffff8010807c
[ 370.302909] Hi : 0000000000000000
[ 370.306595] Lo : 0000000000000200
[ 370.310376] epc : ffffffff80115d38 _save_fp+0x10/0xa0
[ 370.315784] ra : ffffffff8010807c prepare_for_fp_mode_switch+0x94/0x1b0
[ 370.322707] Status: 140084e2 KX SX UX KERNEL EXL
[ 370.327980] Cause : 1080002c (ExcCode 0b)
[ 370.332091] PrId : 0001a428 (MIPS P6600)
[ 370.336179] Modules linked in:
[ 370.339486] Process fp-prctl (pid: 963, threadinfo=a8000000fd630000, task=a8000000fd672e00, tls=00000000756e67d0)
[ 370.349724] Stack : 0000000000000000 a8000000fd557dc0 0000000000000000 ffffffff801ca8e0
[ 370.358161] 0000000000000000 a8000000fd637b9c 0000000000000009 ffffffff80923780
[ 370.366575] ffffffff80850000 ffffffff8011610c 00000000000000b8 ffffffff801a5084
[ 370.374989] ffffffff8084a370 ffffffff8084a388 ffffffff80923780 ffffffff80923828
[ 370.383395] 0000000000010000 ffffffff809237a8 0000000000020000 ffffffff80a40000
[ 370.391817] 000000000000007c 00000000004a0000 00000000756dedd0 ffffffff801a5188
[ 370.400230] a800000002014900 0000000000000001 ffffffff80923780 0000000080923828
[ 370.408644] ffffffff80923780 ffffffff80923780 ffffffff80923828 ffffffff801a521c
[ 370.417066] ffffffff80923780 ffffffff80923828 0000000000010000 ffffffff801a8f84
[ 370.425472] ffffffff80a40000 a8000000fd637c20 ffffffff80a39240 0000000000000001
[ 370.433885] ...
[ 370.436562] Call Trace:
[ 370.439222] [<ffffffff80115d38>] _save_fp+0x10/0xa0
[ 370.444305] [<ffffffff8010807c>] prepare_for_fp_mode_switch+0x94/0x1b0
[ 370.451035] [<ffffffff801ca8e0>] flush_smp_call_function_queue+0xf8/0x230
[ 370.457991] [<ffffffff8011610c>] ipi_call_interrupt+0xc/0x20
[ 370.463814] [<ffffffff801a5084>] __handle_irq_event_percpu+0xc4/0x1a8
[ 370.470404] [<ffffffff801a5188>] handle_irq_event_percpu+0x20/0x68
[ 370.476734] [<ffffffff801a521c>] handle_irq_event+0x4c/0x88
[ 370.482486] [<ffffffff801a8f84>] handle_edge_irq+0x12c/0x210
[ 370.488316] [<ffffffff801a47a0>] generic_handle_irq+0x38/0x48
[ 370.494280] [<ffffffff804a2dbc>] gic_handle_shared_int+0x194/0x268
[ 370.500616] [<ffffffff801a47a0>] generic_handle_irq+0x38/0x48
[ 370.506529] [<ffffffff80107e60>] do_IRQ+0x18/0x28
[ 370.511445] [<ffffffff804a1524>] plat_irq_dispatch+0xc4/0x140
[ 370.517339] [<ffffffff80106230>] ret_from_irq+0x0/0x4
[ 370.522583] [<ffffffff8010fad4>] do_ri+0x4fc/0x7e8
[ 370.527546] [<ffffffff80106220>] ret_from_exception+0x0/0x10

2) The IPI may arrive during kernel use of the FPU, since we generally
only disable preemption around use of the FPU & leave interrupts
enabled. This can lead to us unexpectedly losing access to the FPU
in places where it previously had not been possible. For example:

do_cpu invoked from kernel context![#2]:
CPU: 2 PID: 7338 Comm: fp-prctl Tainted: G D 4.7.0-00424-g49b0c82
#2
task: 838e4000 ti: 88d38000 task.ti: 88d38000
$ 0 : 00000000 00000001 ffffffff 88d3fef8
$ 4 : 838e4000 88d38004 00000000 00000001
$ 8 : 3400fc01 801f8020 808e9100 24000000
$12 : dbffffff 807b69d8 807b0000 00000000
$16 : 00000000 80786150 00400fc4 809c0398
$20 : 809c0338 0040273c 88d3ff28 808e9d30
$24 : 808e9d30 00400fb4
$28 : 88d38000 88d3fe88 00000000 8011a2ac
Hi : 0040273c
Lo : 88d3ff28
epc : 80114178 _restore_fp+0x10/0xa0
ra : 8011a2ac mipsr2_decoder+0xd5c/0x1660
Status: 1400fc03 KERNEL EXL IE
Cause : 1080002c (ExcCode 0b)
PrId : 0001a920 (MIPS I6400)
Modules linked in:
Process fp-prctl (pid: 7338, threadinfo=88d38000, task=838e4000, tls=766527d0)
Stack : 00000000 00000000 00000000 88d3fe98 00000000 00000000 809c0398 809c0338
808e9100 00000000 88d3ff28 00400fc4 00400fc4 0040273c 7fb69e18 004a0000
004a0000 004a0000 7664add0 8010de18 00000000 00000000 88d3fef8 88d3ff28
808e9100 00000000 766527d0 8010e534 000c0000 85755000 8181d580 00000000
00000000 00000000 004a0000 00000000 766527d0 7fb69e18 004a0000 80105c20
...
Call Trace:
[<80114178>] _restore_fp+0x10/0xa0
[<8011a2ac>] mipsr2_decoder+0xd5c/0x1660
[<8010de18>] do_ri+0x90/0x6b8
[<80105c20>] ret_from_exception+0x0/0x10

At first glance a simple fix may seem to be to disable interrupts around
kernel use of the FPU rather than merely preemption, however this would
introduce further overhead outside of the mode switch path & doesn't
solve the third problem:

3) The IPI may arrive whilst the kernel is running code that will lead
to a preempt_disable() call & FPU usage soon. If this happens then
the IPI will be serviced & we'll proceed to enable an FPU whilst the
mode switch is in progress, leading to strange & inconsistent
behaviour.

Further to all of this is a separate but related problem:

4) There are various paths through which we may enable the FPU without
the user having triggered a coprocessor 1 disabled exception. These
paths are those in which we emulate instructions & then enable the
FPU with the expectation that the user might execute an FP
instruction shortly afterwards. However these paths have not
previously checked whether an FP mode switch is underway for the
task, and therefore could enable the FPU whilst such a mode switch
is in progress leading to strange & inconsistent behaviour for user
code.

This patch fixes all of the above by taking a step back & re-examining
our approach to FP mode switches. Up until now we have taken these basic
steps:

a) Prevent any threads that are part of the affected process from being
able to obtain ownership of the FPU.

b) Cause any threads that are part of the affected process and already
have ownership of an FPU to lose it.

c) Set the thread flags for each thread that is part of the affected
process to reflect the new FP mode.

d) Allow threads to obtain ownership of the FPU again.

This approach is however more complex than necessary. All that we really
require is that the mode switch has occurred for all threads that are
part of the affected process before mips_set_process_fp_mode(), and thus
the PR_SET_FP_MODE prctl() syscall, returns. This doesn't require that
we stop threads from owning or using an FPU whilst a mode switch occurs,
only that we force them to relinquish it after the mode switch has
occurred such that they next own an FPU with the correct mode
configured. Our basic steps therefore simplify to:

A) Set the thread flags for each thread that is part of the affected
process to reflect the new FP mode.

B) Cause any threads that are part of the affected process and already
have ownership of an FPU to lose it.

We implement B) by forcing each CPU which might be running a thread
which is part of the affected process to schedule a no-op function,
which causes the affected thread to lose its FPU ownership when it is
descheduled.

The end result is simpler FP mode switching with less overhead in the
FPU enable path (ie. enable_restore_fp_context()) and fewer moving
parts.

Signed-off-by: Paul Burton <paul.burton@mips.com>
Fixes: 9791554b45a2 ("MIPS,prctl: add PR_[GS]ET_FP_MODE prctl options for MIPS")
Fixes: 6b8322576e9d ("MIPS: Force CPUs to lose FP context during mode switches")
Cc: James Hogan <jhogan@kernel.org>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: linux-mips@linux-mips.org
Cc: stable <stable@vger.kernel.org> # v4.0+

sched/headers: Prepare to remove the <linux/mm_types.h> dependency from <linux/sched.h>

Update code that relied on sched.h including various MM types for them.

This will allow us to remove the <linux/mm_types.h> include from <linux/sched.h>.

Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>

KVM: MIPS/TLB: Drop kvm_local_flush_tlb_all()

Now that KVM no longer uses wired entries we can safely use
local_flush_tlb_all() when we need to flush the entire TLB (on the start
of a new ASID cycle). This doesn't flush wired entries, which allows
other code to use them without KVM clobbering them all the time. It also
is more up to date, knowing about the tlbinv architectural feature,
flushing of micro TLB on cores where that is necessary (Loongson I
believe), and knows to stop the HTW while doing so.

Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: "Radim Krčmář" <rkrcmar@redhat.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: linux-mips@linux-mips.org
Cc: kvm@vger.kernel.org

KVM: MIPS/T&E: Activate GVA page tables in guest context

Activate the GVA page tables when in guest context. This will allow the
normal Linux TLB refill handler to fill from it when guest memory is
read, as well as preventing accidental reading from user memory.

Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: "Radim Krčmář" <rkrcmar@redhat.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: linux-mips@linux-mips.org
Cc: kvm@vger.kernel.org

MIPS: Use per-mm page to execute branch delay slot instructions

In some cases the kernel needs to execute an instruction from the delay
slot of an emulated branch instruction. These cases include:

- Emulated floating point branch instructions (bc1[ft]l?) for systems
which don't include an FPU, or upon which the kernel is run with the
"nofpu" parameter.

- MIPSr6 systems running binaries targeting older revisions of the
architecture, which may include branch instructions whose encodings
are no longer valid in MIPSr6.

Executing instructions from such delay slots is done by writing the
instruction to memory followed by a trap, as part of an "emuframe", and
executing it. This avoids the requirement of an emulator for the entire
MIPS instruction set. Prior to this patch such emuframes are written to
the user stack and executed from there.

This patch moves FP branch delay emuframes off of the user stack and
into a per-mm page. Allocating a page per-mm leaves userland with access
to only what it had access to previously, and compared to other
solutions is relatively simple.

When a thread requires a delay slot emulation, it is allocated a frame.
A thread may only have one frame allocated at any one time, since it may
only ever be executing one instruction at any one time. In order to
ensure that we can free up allocated frame later, its index is recorded
in struct thread_struct. In the typical case, after executing the delay
slot instruction we'll execute a break instruction with the BRK_MEMU
code. This traps back to the kernel & leads to a call to do_dsemulret
which frees the allocated frame & moves the user PC back to the
instruction that would have executed following the emulated branch.
In some cases the delay slot instruction may be invalid, such as a
branch, or may trigger an exception. In these cases the BRK_MEMU break
instruction will not be hit. In order to ensure that frames are freed
this patch introduces dsemul_thread_cleanup() and calls it to free any
allocated frame upon thread exit. If the instruction generated an
exception & leads to a signal being delivered to the thread, or indeed
if a signal simply happens to be delivered to the thread whilst it is
executing from the struct emuframe, then we need to take care to exit
the frame appropriately. This is done by either rolling back the user PC
to the branch or advancing it to the continuation PC prior to signal
delivery, using dsemul_thread_rollback(). If this were not done then a
sigreturn would return to the struct emuframe, and if that frame had
meanwhile been used in response to an emulated branch instruction within
the signal handler then we would execute the wrong user code.

Whilst a user could theoretically place something like a compact branch
to self in a delay slot and cause their thread to become stuck in an
infinite loop with the frame never being deallocated, this would:

- Only affect the users single process.

- Be architecturally invalid since there would be a branch in the
delay slot, which is forbidden.

- Be extremely unlikely to happen by mistake, and provide a program
with no more ability to harm the system than a simple infinite loop
would.

If a thread requires a delay slot emulation & no frame is available to
it (ie. the process has enough other threads that all frames are
currently in use) then the thread joins a waitqueue. It will sleep until
a frame is freed by another thread in the process.

Since we now know whether a thread has an allocated frame due to our
tracking of its index, the cookie field of struct emuframe is removed as
we can be more certain whether we have a valid frame. Since a thread may
only ever have a single frame at any given time, the epc field of struct
emuframe is also removed & the PC to continue from is instead stored in
struct thread_struct. Together these changes simplify & shrink struct
emuframe somewhat, allowing twice as many frames to fit into the page
allocated for them.

The primary benefit of this patch is that we are now free to mark the
user stack non-executable where that is possible.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: Maciej Rozycki <maciej.rozycki@imgtec.com>
Cc: Faraz Shahbazker <faraz.shahbazker@imgtec.com>
Cc: Raghu Gandham <raghu.gandham@imgtec.com>
Cc: Matthew Fortune <matthew.fortune@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/13764/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Retrieve ASID masks using function accepting struct cpuinfo_mips

In preparation for supporting variable ASID masks, retrieve ASID masks
using functions in asm/cpu-info.h which accept struct cpuinfo_mips. This
will allow those functions to determine the ASID mask based upon the CPU
in a later patch. This also allows for the r3k & r8k cases to be handled
in Kconfig, which is arguably cleaner than the previous #ifdefs.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Radim Krčmář <rkrcmar@redhat.com>
Cc: linux-mips@linux-mips.org
Cc: kvm@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/13210/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: HTW: Prevent accidental HTW start due to nested htw_{start, stop}

activate_mm() and switch_mm() call get_new_mmu_context() which in turn
can enable the HTW before the entryhi is changed with the new ASID.
Since the latter will enable the HTW in local_flush_tlb_all(),
then there is a small timing window where the HTW is running with the
new ASID but with an old pgd since the TLBMISS_HANDLER_SETUP_PGD
hasn't assigned a new one yet. In order to prevent that, we introduce a
simple htw counter to avoid starting HTW accidentally due to nested
htw_{start,stop}() sequences. Moreover, since various IPI calls can
enforce TLB flushing operations on a different core, such an operation
may interrupt another htw_{stop,start} in progress leading inconsistent
updates of the htw_seq variable. In order to avoid that, we disable the
interrupts whenever we update that variable.

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: <stable@vger.kernel.org> # 3.17+
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9118/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS,prctl: add PR_[GS]ET_FP_MODE prctl options for MIPS

Userland code may be built using an ABI which permits linking to objects
that have more restrictive floating point requirements. For example,
userland code may be built to target the O32 FPXX ABI. Such code may be
linked with other FPXX code, or code built for either one of the more
restrictive FP32 or FP64. When linking with more restrictive code, the
overall requirement of the process becomes that of the more restrictive
code. The kernel has no way to know in advance which mode the process
will need to be executed in, and indeed it may need to change during
execution. The dynamic loader is the only code which will know the
overall required mode, and so it needs to have a means to instruct the
kernel to switch the FP mode of the process.

This patch introduces 2 new options to the prctl syscall which provide
such a capability. The FP mode of the process is represented as a
simple bitmask combining a number of mode bits mirroring those present
in the hardware. Userland can either retrieve the current FP mode of
the process:

mode = prctl(PR_GET_FP_MODE);

or modify the current FP mode of the process:

err = prctl(PR_SET_FP_MODE, new_mode);

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: Matthew Fortune <matthew.fortune@imgtec.com>
Cc: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/8899/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: mm: Use the Hardware Page Table Walker if the core supports it

The Hardware Page Table Walker aims to speed up TLB refill exceptions
by handling them in the hardware level instead of having a software
TLB refill handler. However, a TLB refill exception can still be
thrown in certain cases such as, synchronus exceptions, or address
translation or memory errors during the HTW operation. As a result of
which, HTW must not be considered a complete replacement for the TLB
refill software handler, but rather a fast-path for it.
For HTW to work, the PWBase register must contain the task's page
global directory address so the HTW will kick in on TLB refill
exceptions.

Due to HTW being a separate engine embedded deep in the CPU pipeline,
we need to restart the HTW everytime a PTE changes to avoid HTW
fetching a old entry from the page tables. It's also necessary to
restart the HTW on context switches to prevent it from fetching a
page from the previous process. Finally, since HTW is using the
entryhi register to write the translations to the TLB, it's necessary
to stop the HTW whenever the entryhi changes (eg for tlb probe
perations) and enable it back afterwards.

== Performance ==

The following trivial test was used to measure the performance of the
HTW. Using the same root filesystem, the following command was used
to measure the number of tlb refill handler executions with and
without (using 'nohtw' kernel parameter) HTW support. The kernel was
modified to use a scratch register as a counter for the TLB refill
exceptions.

find /usr -type f -exec ls -lh {} \;

HTW Enabled:
TLB refill exceptions: 12306

HTW Disabled:
TLB refill exceptions: 17805

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-mips@linux-mips.org
Cc: Markos Chandras <markos.chandras@imgtec.com>
Patchwork: https://patchwork.linux-mips.org/patch/7336/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: MT: Remove SMTC support

Nobody is maintaining SMTC anymore and there also seems to be no userbase.
Which is a pity - the SMTC technology primarily developed by Kevin D.
Kissell <kevink@paralogos.com> is an ingenious demonstration for the MT
ASE's power and elegance.

Based on Markos Chandras <Markos.Chandras@imgtec.com> patch
https://patchwork.linux-mips.org/patch/6719/ which while very similar did
no longer apply cleanly when I tried to merge it plus some additional
post-SMTC cleanup - SMTC was a feature as tricky to remove as it was to
merge once upon a time.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: traps: Add CPU PM callback for trap configuration

Implement a CPU power management callback for restoring trap related CPU
configuration after CPU power up from a low power state. The following
state is restored:

- Status register
- HWREna register
- Exception vector configuration registers
- Context/XContext register

Signed-off-by: James Hogan <james.hogan@imgtec.com>
Signed-off-by: Paul Burton <paul.burton@imgtec.com>

MIPS: mm: Use scratch for PGD when !CONFIG_MIPS_PGD_C0_CONTEXT

Allow usage of scratch register for current pgd even when
MIPS_PGD_C0_CONTEXT is not configured. MIPS_PGD_C0_CONTEXT is set
for 64r2 platforms to indicate availability of Xcontext for saving
cpuid, thus freeing Context to be used for saving PGD. This option
was also tied to using a scratch register for storing PGD.

This commit will allow usage of scratch register to store the current
pgd if one can be allocated for the platform, even when
MIPS_PGD_C0_CONTEXT is not set. The cpuid will be kept in the CP0
Context register in this case.

The code to store the current pgd for the TLB miss handler is now
generated in all cases. When scratch register is available, the PGD
is also stored in the scratch register.

Signed-off-by: Jayachandran C <jchandra@broadcom.com>
Cc: linux-mips@linux-mips.org
Cc: Hauke Mehrtens <hauke@hauke-m.de>
Patchwork: https://patchwork.linux-mips.org/patch/5906/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Move definition of SMP processor id register to header file

The definition of the CP0 register used to save the smp processor
id is repicated in many files, move them all to thread_info.h.

Signed-off-by: Jayachandran C <jchandra@broadcom.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5708/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Move generated code to .text for microMIPS

Prepare of a next patch which will call tlbmiss_handler_setup_pgd on
microMIPS. MicroMIPS complains if the called code s not in the .text
section. To fix this we generate code into space reserved in
arch/mips/mm/tlb-funcs.S

While there, move the rest of the generated functions (handle_tlbl,
handle_tlbs, handle_tlbm) to the same file.

Signed-off-by: Jayachandran C <jchandra@broadcom.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5542/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: include: mmu_context.h: Replace VIRTUALIZATION with KVM

The kvm_* symbols are only available if KVM is selected.

Fixes the following linking problem on a randconfig:

arch/mips/built-in.o: In function `local_flush_tlb_mm':
(.text+0x18a94): undefined reference to `kvm_local_flush_tlb_all'
arch/mips/built-in.o: In function `local_flush_tlb_range':
(.text+0x18d0c): undefined reference to `kvm_local_flush_tlb_all'
kernel/built-in.o: In function `__schedule':
core.c:(.sched.text+0x2a00): undefined reference to `kvm_local_flush_tlb_all'
mm/built-in.o: In function `use_mm':
(.text+0x30214): undefined reference to `kvm_local_flush_tlb_all'
fs/built-in.o: In function `flush_old_exec':
(.text+0xf0a0): undefined reference to `kvm_local_flush_tlb_all'
make: *** [vmlinux] Error 1

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Acked-by: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5437/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Revert "MIPS: Allow ASID size to be determined at boot time."

This reverts commit d532f3d26716a39dfd4b88d687bd344fbe77e390.

The original commit has several problems:

1) Doesn't work with 64-bit kernels.

2) Calls TLBMISS_HANDLER_SETUP() before the code is generated.

3) Calls TLBMISS_HANDLER_SETUP() twice in per_cpu_trap_init() when
only one call is needed.

[ralf@linux-mips.org: Also revert the bits of the ASID patch which were
hidden in the KVM merge.]

Signed-off-by: David Daney <david.daney@cavium.com>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Cc: "Steven J. Hill" <Steven.Hill@imgtec.com>
Cc: David Daney <david.daney@cavium.com>
Patchwork: https://patchwork.linux-mips.org/patch/5242/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Allow ASID size to be determined at boot time.

Original patch by Ralf Baechle and removed by Harold Koerfgen
with commit f67e4ffc79905482c3b9b8c8dd65197bac7eb508. This
allows for more generic kernels since the size of the ASID
and corresponding masks can be determined at run-time. This
patch is also required for the new Aptiv cores and has been
tested on Malta and Malta Aptiv platforms.

[ralf@linux-mips.org: Added relevant part of fix
https://patchwork.linux-mips.org/patch/5213/]

Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: If KVM is enabled then use the KVM specific routine to flush the TLBs on a ASID wrap.

Signed-off-by: Sanjay Lal <sanjayl@kymasys.com>
Cc: kvm@vger.kernel.org
Cc: linux-mips@linux-mips.org
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Init new mmu_context for each possible CPU to avoid memory corruption

Currently, init_new_context() only for each online CPU, this may cause
memory corruption when CPU hotplug and fork() happens at the same time.
To avoid this, we make init_new_context() cover each possible CPU.

Scenario:
1, CPU#1 is being offline;
2, On CPU#0, do_fork() call dup_mm() and copy a mm_struct to the child;
3, On CPU#0, dup_mm() call init_new_context(), since CPU#1 is offline
and init_new_context() only covers the online CPUs, child has the
same asid as its parent on CPU#1 (however, child's asid should be 0);
4, CPU#1 is being online;
5, Now, if both parent and child run on CPU#1, memory corruption (e.g.
segfault, bus error, etc.) will occur.

Signed-off-by: Huacai Chen <chenhc@lemote.com>
Acked-by: David Daney <david.daney@cavium.com>
Patchwork: http://patchwork.linux-mips.org/patch/4995/
Acked-by: John Crispin <blogic@openwrt.org>

MIPS: Make declarations and definitions of tlbmiss_handler_setup_pgd match.

tlbmiss_handler_setup_pgd is run-time generated code and it was convenient
to pretend the symbol was an array in the generator but a function for
the users. LTO gcc won't tolerate this kind of lie anymore so solve the
problem through a cast and function pointer instead.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Whitespace cleanup.

Having received another series of whitespace patches I decided to do this
once and for all rather than dealing with this kind of patches trickling
in forever.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: PMC-Sierra Yosemite: Remove support.

Nobody seems to be interested anymore and upstream also never had an
ethernet driver.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Remove get_current_pgd().

It is unused in the tree.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/3557/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Use C0_KScratch (if present) to hold PGD pointer.

Decide at runtime to use either Context or KScratch to hold the PGD
pointer.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/1876/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Preliminary VDSO

This is a preliminary patch to add a vdso to all user processes. Still
missing are ELF headers and .eh_frame information. But it is enough to
allow us to move signal trampolines off of the stack. Note that emulation
of branch delay slots in the FPU emulator still requires the stack.

We allocate a single page (the vdso) and write all possible signal
trampolines into it. The stack is moved down by one page and the vdso is
mapped into this space.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: http://patchwork.linux-mips.org/patch/975/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Put PGD in C0_CONTEXT for 64-bit R2 processors.

Processors that support the mips64r2 ISA can in four instructions
convert a shifted PGD pointer stored in the upper bits of c0_context
into a usable pointer. By doing this we save a memory load and
associated potential cache miss in the TLB exception handlers.

Since the upper bits of c0_context were holding the CPU number, we
move this to the upper bits of c0_xcontext which doesn't have enough
bits to hold the PGD pointer, but has plenty for the CPU number.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Avoid potential hazard on Context register

set_saved_sp reads Context register. Avoid reading stale value from
earlier incomplete write.

Issue found and fixed for head.S by Chris Dearman <chris@mips.com>.

Signed-off-by: Chris Dearman <chris@mips.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Don't write ones to reserved entryhi bits.

We've silently been relying on the hardware chopping off excess, reserved
ASID bits for no better reason that it saving an instruction. Because we
already have:

#define cpu_asid(cpu, mm) (cpu_context((cpu), (mm)) & ASID_MASK)

in <asm/mmu_context.h>.

We can use a cleanup to avoid writing non-zero bits into the reserved
entryhi bits. This avoid triggering some debugging assertion in the
Cavium simulator.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

cpumask: use mm_cpumask() wrapper: mips

Makes code futureproof against the impending change to mm->cpu_vm_mask.

It's also a chance to use the new cpumask_ ops which take a pointer
(the older ones are deprecated, but there's no hurry for arch code).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

MIPS: Build fix - include <linux/smp.h> into all smp_processor_id() users.

Some of the were relying into smp.h being dragged in by another header
which of course is fragile. <asm/cpu-info.h> uses smp_processor_id()
only in macros and including smp.h there leads to an include loop, so
don't change cpu-info.h.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Move headfiles to new location below arch/mips/include

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>