MIPS: tlbex: Use GPR number macros

Use GPR number macros in uasm code generation parts to
reduce code duplication.

No functional change.

Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>

MIPS: Unify define of CP0 registers for uasm code

Definitions of uasm variant of CP0 registers are unified to
mipsregs.h, so they lay together with uasm variant of
the code.

Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>

MIPS: Fix typos

Fix typos, most reported by "codespell arch/mips". Only touches comments,
no code changes.

Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Cc: linux-mips@vger.kernel.org
Reviewed-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>

mips: implement the new page table range API

Rename _PFN_SHIFT to PFN_PTE_SHIFT. Convert a few places
to call set_pte() instead of set_pte_at(). Add set_ptes(),
update_mmu_cache_range(), flush_icache_pages() and flush_dcache_folio().
Change the PG_arch_1 (aka PG_dcache_dirty) flag from being per-page
to per-folio.

Link: https://lkml.kernel.org/r/20230802151406.3735276-18-willy@infradead.org
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Acked-by: Mike Rapoport (IBM) <rppt@kernel.org>
Cc: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

MIPS: mm: Remove special handling for OCTEON CPUs

Macro cpu_has_mips_r2_exec_hazard correctly handles OCTEON CPUs,
so we don't need the extra switch cases for them.

Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>

MIPS: tlbex: Explicitly compare _PAGE_NO_EXEC against 0

When CONFIG_XPA is enabled, Clang warns:

arch/mips/mm/tlbex.c:629:24: error: converting the result of '<<' to a boolean; did you mean '(1 << _PAGE_NO_EXEC_SHIFT) != 0'? [-Werror,-Wint-in-bool-context]
if (cpu_has_rixi && !!_PAGE_NO_EXEC) {
^
arch/mips/include/asm/pgtable-bits.h:174:28: note: expanded from macro '_PAGE_NO_EXEC'
# define _PAGE_NO_EXEC (1 << _PAGE_NO_EXEC_SHIFT)
^
arch/mips/mm/tlbex.c:2568:24: error: converting the result of '<<' to a boolean; did you mean '(1 << _PAGE_NO_EXEC_SHIFT) != 0'? [-Werror,-Wint-in-bool-context]
if (!cpu_has_rixi || !_PAGE_NO_EXEC) {
^
arch/mips/include/asm/pgtable-bits.h:174:28: note: expanded from macro '_PAGE_NO_EXEC'
# define _PAGE_NO_EXEC (1 << _PAGE_NO_EXEC_SHIFT)
^
2 errors generated.

_PAGE_NO_EXEC can be '0' or '1 << _PAGE_NO_EXEC_SHIFT' depending on the
build and runtime configuration, which is what the negation operators
are trying to convey. To silence the warning, explicitly compare against
0 so the result of the '<<' operator is not implicitly converted to a
boolean.

According to its documentation, GCC enables -Wint-in-bool-context with
-Wall but this warning is not visible when building the same
configuration with GCC. It appears GCC only warns when compiling C++,
not C, although the documentation makes no note of this:
https://godbolt.org/z/x39q3brxf

Reported-by: Sudip Mukherjee (Codethink) <sudipm.mukherjee@gmail.com>
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>

MIPS: Remove VR41xx support

No (active) developer owns this hardware, so let's remove Linux support.

Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>

mips: rename PGD_ORDER to PGD_TABLE_ORDER

This is the order of the page table allocation, not the order of a PGD.

While at it remove unused defintion of _PGD_ORDER in asm-offsets.

Link: https://lkml.kernel.org/r/20220703141203.147893-7-rppt@kernel.org
Signed-off-by: Mike Rapoport <rppt@linux.ibm.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Dinh Nguyen <dinguyen@kernel.org>
Cc: Guo Ren <guoren@kernel.org>
Cc: Helge Deller <deller@gmx.de>
Cc: Huacai Chen <chenhuacai@kernel.org>
Cc: James Bottomley <James.Bottomley@HansenPartnership.com>
Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org>
Cc: Max Filippov <jcmvbkbc@gmail.com>
Cc: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
Cc: Xuerui Wang <kernel@xen0n.name>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

mips: drop definitions of PTE_ORDER

This is the order of the page table allocation, not the order of a PTE.
Since its always hardwired to 0, simply drop it.

Link: https://lkml.kernel.org/r/20220703141203.147893-6-rppt@kernel.org
Signed-off-by: Mike Rapoport <rppt@linux.ibm.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Dinh Nguyen <dinguyen@kernel.org>
Cc: Guo Ren <guoren@kernel.org>
Cc: Helge Deller <deller@gmx.de>
Cc: Huacai Chen <chenhuacai@kernel.org>
Cc: James Bottomley <James.Bottomley@HansenPartnership.com>
Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org>
Cc: Max Filippov <jcmvbkbc@gmail.com>
Cc: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
Cc: Xuerui Wang <kernel@xen0n.name>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

MIPS: Sanitise Cavium switch cases in TLB handler synthesizers

It makes no sense to fall through to `break'. Therefore reorder the
switch statements so as to have the Cavium cases first, followed by the
default case, which improves readability and pacifies code analysis
tools. No change in semantics, assembly produced is exactly the same.

Reported-by: kernel test robot <lkp@intel.com>
Signed-off-by: Maciej W. Rozycki <macro@orcam.me.uk>
Fixes: bc431d2153cc ("MIPS: Fix fall-through warnings for Clang")
Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>

MIPS: remove asm/war.h

The major part for workaround handling has already moved to config
options. This change replaces the remaining defines by already
available config options and gets rid of war.h

Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>

MIPS: Remove NETLOGIC support

No (active) developer owns this hardware, so let's remove Linux support.

Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>

MIPS: Fix fall-through warnings for Clang

Fix the following fallthrough warnings:

arch/mips/mm/tlbex.c:1386:2: warning: unannotated fall-through between switch labels [-Wimplicit-fallthrough]
arch/mips/mm/tlbex.c:2173:3: error: unannotated fall-through between switch labels [-Werror,-Wimplicit-fallthrough]

Reported-by: kernel test robot <lkp@intel.com>
Link: https://lore.kernel.org/lkml/60edca25.k00ut905IFBjPyt5%25lkp@intel.com/
Signed-off-by: Gustavo A. R. Silva <gustavoars@kernel.org>

MIPS: clean up CONFIG_MIPS_PGD_C0_CONTEXT handling

+. LOONGSON64 use 0x98xx_xxxx_xxxx_xxxx as xphys cached, instread of
0xa8xx_xxxx_xxxx_xxxx

+. let CONFIG_MIPS_PGD_C0_CONTEXT depend on 64bit

+. cast CAC_BASE into u64 to silence warning on MIPS32

CP0 Context has enough room for wraping pgd into its 41-bit PTEBase field.

+. For XPHYS, the trick is that pgd is 4kB aligned, and the PABITS <= 53,
only save 53 - 12 = 41 bits, aka :

bit[63:59] | 0000 00 | bit[53:12] | 0000 0000 0000

+. for CKSEG0, only save 29 - 12 = 17 bits

when switching pgd, only need to save bit[53:12] or bit[28:12] into
CP0 Context's bit[63:23], see folling asm generated at run time

tlbmiss_handler_setup_pgd:
.set push
.set noreorder

dsra a2, a0, 29
move a3, a0
dins a0, zero, 29, 35
daddiu a2, a2, 4 //for CKSEG0, a2 from 0xfffffffffffffffc
//into 0

movn a0, a3, a2
dsll a0, a0, 11
jr ra
dmtc0 a0, CP0_CONTEXT

.set pop

when using it on page walking

dmfc0 k0, CP0_CONTEXT
dins k0, zero, 0, 23 // zero badv2
ori k0, k0, (CAC_BASE >> 53) // *prefix* with bit[63:59]
drotr k0, k0, 11 // kick it in the right place

Signed-off-by: Huang Pei <huangpei@loongson.cn>
Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>

Revert "MIPS: Remove unused R4300 CPU support"

This reverts commit f9065b54d437c4660e3d974ad9ce5188c068cd76.

We're adding Nintendo 64 support, so the VR4300 is no longer unused.

Signed-off-by: Lauri Kasanen <cand@gmx.com>
Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>

MIPS: Get rid of BCM1250_M3_WAR

BCM1250_M3_WAR is depending on CONFIG_CONFIG_SB1_PASS_2_WORKAROUNDS.
So using this option directly lets and remove define.

Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>

MIPS: Convert R10000_LLSC_WAR info a config option

Use a new config option to enabel R1000_LLSC workaound and remove
define from different war.h files.

Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>

mm: reorder includes after introduction of linux/pgtable.h

The replacement of <asm/pgrable.h> with <linux/pgtable.h> made the include
of the latter in the middle of asm includes. Fix this up with the aid of
the below script and manual adjustments here and there.

import sys
import re

if len(sys.argv) is not 3:
print "USAGE: %s <file> <header>" % (sys.argv[0])
sys.exit(1)

hdr_to_move="#include <linux/%s>" % sys.argv[2]
moved = False
in_hdrs = False

with open(sys.argv[1], "r") as f:
lines = f.readlines()
for _line in lines:
line = _line.rstrip('
')
if line == hdr_to_move:
continue
if line.startswith("#include <linux/"):
in_hdrs = True
elif not moved and in_hdrs:
moved = True
print hdr_to_move
print line

Signed-off-by: Mike Rapoport <rppt@linux.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Cain <bcain@codeaurora.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Chris Zankel <chris@zankel.net>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Greentime Hu <green.hu@gmail.com>
Cc: Greg Ungerer <gerg@linux-m68k.org>
Cc: Guan Xuetao <gxt@pku.edu.cn>
Cc: Guo Ren <guoren@kernel.org>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Helge Deller <deller@gmx.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Ley Foon Tan <ley.foon.tan@intel.com>
Cc: Mark Salter <msalter@redhat.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Matt Turner <mattst88@gmail.com>
Cc: Max Filippov <jcmvbkbc@gmail.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Michal Simek <monstr@monstr.eu>
Cc: Nick Hu <nickhu@andestech.com>
Cc: Paul Walmsley <paul.walmsley@sifive.com>
Cc: Richard Weinberger <richard@nod.at>
Cc: Rich Felker <dalias@libc.org>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Stafford Horne <shorne@gmail.com>
Cc: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Vincent Chen <deanbo422@gmail.com>
Cc: Vineet Gupta <vgupta@synopsys.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yoshinori Sato <ysato@users.sourceforge.jp>
Link: http://lkml.kernel.org/r/20200514170327.31389-4-rppt@kernel.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

mm: introduce include/linux/pgtable.h

The include/linux/pgtable.h is going to be the home of generic page table
manipulation functions.

Start with moving asm-generic/pgtable.h to include/linux/pgtable.h and
make the latter include asm/pgtable.h.

Signed-off-by: Mike Rapoport <rppt@linux.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Cain <bcain@codeaurora.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Chris Zankel <chris@zankel.net>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Greentime Hu <green.hu@gmail.com>
Cc: Greg Ungerer <gerg@linux-m68k.org>
Cc: Guan Xuetao <gxt@pku.edu.cn>
Cc: Guo Ren <guoren@kernel.org>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Helge Deller <deller@gmx.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Ley Foon Tan <ley.foon.tan@intel.com>
Cc: Mark Salter <msalter@redhat.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Matt Turner <mattst88@gmail.com>
Cc: Max Filippov <jcmvbkbc@gmail.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Michal Simek <monstr@monstr.eu>
Cc: Nick Hu <nickhu@andestech.com>
Cc: Paul Walmsley <paul.walmsley@sifive.com>
Cc: Richard Weinberger <richard@nod.at>
Cc: Rich Felker <dalias@libc.org>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Stafford Horne <shorne@gmail.com>
Cc: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Vincent Chen <deanbo422@gmail.com>
Cc: Vineet Gupta <vgupta@synopsys.com>
Cc: Will Deacon <will@kernel.org>
Cc: Yoshinori Sato <ysato@users.sourceforge.jp>
Link: http://lkml.kernel.org/r/20200514170327.31389-3-rppt@kernel.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

MIPS: Use fallthrough for arch/mips

Convert the various /* fallthrough */ comments to the pseudo-keyword
fallthrough;

Done via script:
https://lore.kernel.org/lkml/b56602fcf79f849e733e7b521bb0e17895d390fa.1582230379.git.joe@perches.com/

Signed-off-by: Liangliang Huang <huangll@lemote.com>
Reviewed-by: Huacai Chen <chenhc@lemote.com>
Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>

MIPS/tlbex: Fix LDDIR usage in setup_pw() for Loongson-3

LDDIR/LDPTE is Loongson-3's acceleration for Page Table Walking. If BD
(Base Directory, the 4th page directory) is not enabled, then GDOffset
is biased by BadVAddr[63:62]. So, if GDOffset (aka. BadVAddr[47:36] for
Loongson-3) is big enough, "0b11(BadVAddr[63:62])|BadVAddr[47:36]|...."
can far beyond pg_swapper_dir. This means the pg_swapper_dir may NOT be
accessed by LDDIR correctly, so fix it by set PWDirExt in CP0_PWCtl.

Cc: <stable@vger.kernel.org>
Signed-off-by: Pei Huang <huangpei@loongson.cn>
Signed-off-by: Huacai Chen <chenhc@lemote.com>
Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>

MIPS: Loongson64: Rename CPU TYPES

CPU_LOONGSON2 -> CPU_LOONGSON2EF
CPU_LOONGSON3 -> CPU_LOONGSON64

As newer loongson-2 products (2G/2H/2K1000) can share kernel
implementation with loongson-3 while 2E/2F are less similar with
other LOONGSON64 products.

Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
Signed-off-by: Paul Burton <paulburton@kernel.org>
Cc: linux-mips@vger.kernel.org
Cc: chenhc@lemote.com
Cc: paul.burton@mips.com

MIPS: tlbex: Fix build_restore_pagemask KScratch restore

build_restore_pagemask() will restore the value of register $1/$at when
its restore_scratch argument is non-zero, and aims to do so by filling a
branch delay slot. Commit 0b24cae4d535 ("MIPS: Add missing EHB in mtc0
-> mfc0 sequence.") added an EHB instruction (Execution Hazard Barrier)
prior to restoring $1 from a KScratch register, in order to resolve a
hazard that can result in stale values of the KScratch register being
observed. In particular, P-class CPUs from MIPS with out of order
execution pipelines such as the P5600 & P6600 are affected.

Unfortunately this EHB instruction was inserted in the branch delay slot
causing the MFC0 instruction which performs the restoration to no longer
execute along with the branch. The result is that the $1 register isn't
actually restored, ie. the TLB refill exception handler clobbers it -
which is exactly the problem the EHB is meant to avoid for the P-class
CPUs.

Similarly build_get_pgd_vmalloc() will restore the value of $1/$at when
its mode argument equals refill_scratch, and suffers from the same
problem.

Fix this by in both cases moving the EHB earlier in the emitted code.
There's no reason it needs to immediately precede the MFC0 - it simply
needs to be between the MTC0 & MFC0.

This bug only affects Cavium Octeon systems which use
build_fast_tlb_refill_handler().

Signed-off-by: Paul Burton <paulburton@kernel.org>
Fixes: 0b24cae4d535 ("MIPS: Add missing EHB in mtc0 -> mfc0 sequence.")
Cc: Dmitry Korotin <dkorotin@wavecomp.com>
Cc: stable@vger.kernel.org # v3.15+
Cc: linux-mips@vger.kernel.org
Cc: linux-kernel@vger.kernel.org

MIPS: tlbex: Remove cpu_has_local_ebase

The cpu_has_local_ebase macro is, confusingly, not used to indicate
whether the EBase register is local to a CPU or not. Instead it
indicates whether we want to generate the TLB refill exception vector
each time a CPU is brought online. Doing this makes little sense on any
system, since we always use the same value for EBase & thus we cannot
have different TLB refill exception handlers per CPU.

Regenerating the code is not only pointless but also can be actively
harmful, as commit 8759934e2b6b ("MIPS: Build uasm-generated code only
once to avoid CPU Hotplug problem") described. That commit introduced
cpu_has_local_ebase to disable the handler regeneration for Loongson
machines, but this is by no means a Loongson-specific problem.

Remove cpu_has_local_ebase & simply generate the TLB refill handler once
during boot, just like the rest of the TLB exception handlers.

Signed-off-by: Paul Burton <paul.burton@mips.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Cc: linux-mips@vger.kernel.org

MIPS: tlbex: Simplify r3k check

We already know whether a CPU has r3k style exceptions, including TLB
exceptions, by checking cpu_has_3kex. Remove the list of CPU types in
build_tlb_refill_handler() & check cpu_has_3kex instead.

Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: linux-mips@vger.kernel.org

MIPS: tlbex: Explicitly cast _PAGE_NO_EXEC to a boolean

clang warns:

arch/mips/mm/tlbex.c:634:19: error: use of logical '&&' with constant
operand [-Werror,-Wconstant-logical-operand]
if (cpu_has_rixi && _PAGE_NO_EXEC) {
^ ~~~~~~~~~~~~~
arch/mips/mm/tlbex.c:634:19: note: use '&' for a bitwise operation
if (cpu_has_rixi && _PAGE_NO_EXEC) {
^~
&
arch/mips/mm/tlbex.c:634:19: note: remove constant to silence this
warning
if (cpu_has_rixi && _PAGE_NO_EXEC) {
~^~~~~~~~~~~~~~~~
1 error generated.

Explicitly cast this value to a boolean so that clang understands we
intend for this to be a non-zero value.

Fixes: 00bf1c691d08 ("MIPS: tlbex: Avoid placing software PTE bits in Entry* PFN fields")
Link: https://github.com/ClangBuiltLinux/linux/issues/609
Signed-off-by: Nathan Chancellor <natechancellor@gmail.com>
Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: James Hogan <jhogan@kernel.org>
Cc: Nick Desaulniers <ndesaulniers@google.com>
Cc: linux-mips@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: clang-built-linux@googlegroups.com

MIPS: Remove unused R8000 CPU support

Our R8000 CPU support can only be included if a system selects
CONFIG_SYS_HAS_CPU_R8000. No system does, making all R8000-related CPU
support dead code. Remove it.

Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: linux-mips@vger.kernel.org

MIPS: Remove unused R5432 CPU support

Our R5432 CPU support can only be included if a system selects
CONFIG_SYS_HAS_CPU_R5432. No system does, making all R5432-related CPU
support dead code. Remove it.

Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: linux-mips@vger.kernel.org

MIPS: Remove unused R4300 CPU support

Our R4300 CPU support can only be included if a system selects
CONFIG_SYS_HAS_CPU_R4300. No system does, making all R4300-related CPU
support dead code. Remove it.

Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: linux-mips@vger.kernel.org

MIPS: Rename JZRISC to XBURST

The real name of the CPU present in the JZ line of SoCs from Ingenic is
XBurst, not JZRISC.

Signed-off-by: Paul Cercueil <paul@crapouillou.net>
[paul.burton@mips.com: Leave /proc/cpuinfo string as-is.]
Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: James Hogan <jhogan@kernel.org>
Cc: od@zcrc.me
Cc: linux-mips@vger.kernel.org
Cc: linux-kernel@vger.kernel.org

MIPS: Add missing EHB in mtc0 -> mfc0 sequence.

Add a missing EHB (Execution Hazard Barrier) in mtc0 -> mfc0 sequence.
Without this execution hazard barrier it's possible for the value read
back from the KScratch register to be the value from before the mtc0.

Reproducible on P5600 & P6600.

The hazard is documented in the MIPS Architecture Reference Manual Vol.
III: MIPS32/microMIPS32 Privileged Resource Architecture (MD00088), rev
6.03 table 8.1 which includes:

Producer | Consumer | Hazard
----------|----------|----------------------------
mtc0 | mfc0 | any coprocessor 0 register

Signed-off-by: Dmitry Korotin <dkorotin@wavecomp.com>
[paul.burton@mips.com:
- Commit message tweaks.
- Add Fixes tags.
- Mark for stable back to v3.15 where P5600 support was introduced.]
Signed-off-by: Paul Burton <paul.burton@mips.com>
Fixes: 3d8bfdd03072 ("MIPS: Use C0_KScratch (if present) to hold PGD pointer.")
Fixes: 829dcc0a956a ("MIPS: Add MIPS P5600 probe support")
Cc: linux-mips@vger.kernel.org
Cc: stable@vger.kernel.org # v3.15+

MIPS: Loongson: Introduce and use loongson_llsc_mb()

On the Loongson-2G/2H/3A/3B there is a hardware flaw that ll/sc and
lld/scd is very weak ordering. We should add sync instructions "before
each ll/lld" and "at the branch-target between ll/sc" to workaround.
Otherwise, this flaw will cause deadlock occasionally (e.g. when doing
heavy load test with LTP).

Below is the explaination of CPU designer:

"For Loongson 3 family, when a memory access instruction (load, store,
or prefetch)'s executing occurs between the execution of LL and SC, the
success or failure of SC is not predictable. Although programmer would
not insert memory access instructions between LL and SC, the memory
instructions before LL in program-order, may dynamically executed
between the execution of LL/SC, so a memory fence (SYNC) is needed
before LL/LLD to avoid this situation.

Since Loongson-3A R2 (3A2000), we have improved our hardware design to
handle this case. But we later deduce a rarely circumstance that some
speculatively executed memory instructions due to branch misprediction
between LL/SC still fall into the above case, so a memory fence (SYNC)
at branch-target (if its target is not between LL/SC) is needed for
Loongson 3A1000, 3B1500, 3A2000 and 3A3000.

Our processor is continually evolving and we aim to to remove all these
workaround-SYNCs around LL/SC for new-come processor."

Here is an example:

Both cpu1 and cpu2 simutaneously run atomic_add by 1 on same atomic var,
this bug cause both 'sc' run by two cpus (in atomic_add) succeed at same
time('sc' return 1), and the variable is only *added by 1*, sometimes,
which is wrong and unacceptable(it should be added by 2).

Why disable fix-loongson3-llsc in compiler?
Because compiler fix will cause problems in kernel's __ex_table section.

This patch fix all the cases in kernel, but:

+. the fix at the end of futex_atomic_cmpxchg_inatomic is for branch-target
of 'bne', there other cases which smp_mb__before_llsc() and smp_llsc_mb() fix
the ll and branch-target coincidently such as atomic_sub_if_positive/
cmpxchg/xchg, just like this one.

+. Loongson 3 does support CONFIG_EDAC_ATOMIC_SCRUB, so no need to touch
edac.h

+. local_ops and cmpxchg_local should not be affected by this bug since
only the owner can write.

+. mips_atomic_set for syscall.c is deprecated and rarely used, just let
it go

Signed-off-by: Huacai Chen <chenhc@lemote.com>
Signed-off-by: Huang Pei <huangpei@loongson.cn>
[paul.burton@mips.com:
- Simplify the addition of -mno-fix-loongson3-llsc to cflags, and add
a comment describing why it's there.
- Make loongson_llsc_mb() a no-op when
CONFIG_CPU_LOONGSON3_WORKAROUNDS=n, rather than a compiler memory
barrier.
- Add a comment describing the bug & how loongson_llsc_mb() helps
in asm/barrier.h.]
Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: ambrosehua@gmail.com
Cc: Steven J . Hill <Steven.Hill@cavium.com>
Cc: linux-mips@linux-mips.org
Cc: Fuxin Zhang <zhangfx@lemote.com>
Cc: Zhangjin Wu <wuzhangjin@gmail.com>
Cc: Li Xuefeng <lixuefeng@loongson.cn>
Cc: Xu Chenghua <xuchenghua@loongson.cn>

mips: annotate implicit fall throughs

There is a plan to build the kernel with -Wimplicit-fallthrough and
these places in the code produced warnings. Fix them up.

This patch produces no change in behaviour, but should be reviewed in
case these are actually bugs not intentional fallthoughs.

Signed-off-by: Mathieu Malaterre <malat@debian.org>
Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: Kees Cook <keescook@google.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: James Hogan <jhogan@kernel.org>
Cc: linux-mips@vger.kernel.org
Cc: linux-kernel@vger.kernel.org

MIPS: Consistently declare TLB functions

Since at least the beginning of the git era we've declared our TLB
exception handling functions inconsistently. They're actually functions,
but we declare them as arrays of u32 where each u32 is an encoded
instruction. This has always been the case for arch/mips/mm/tlbex.c, and
has also been true for arch/mips/kernel/traps.c since commit
86a1708a9d54 ("MIPS: Make tlb exception handler definitions and
declarations match.") which aimed for consistency but did so by
consistently making the our C code inconsistent with our assembly.

This is all usually harmless, but when using GCC 7 or newer to build a
kernel targeting microMIPS (ie. CONFIG_CPU_MICROMIPS=y) it becomes
problematic. With microMIPS bit 0 of the program counter indicates the
ISA mode. When bit 0 is zero instructions are decoded using the standard
MIPS32 or MIPS64 ISA. When bit 0 is one instructions are decoded using
microMIPS. This means that function pointers become odd - their least
significant bit is one for microMIPS code. We work around this in cases
where we need to access code using loads & stores with our
msk_isa16_mode() macro which simply clears bit 0 of the value it is
given:

#define msk_isa16_mode(x) ((x) & ~0x1)

For example we do this for our TLB load handler in
build_r4000_tlb_load_handler():

u32 *p = (u32 *)msk_isa16_mode((ulong)handle_tlbl);

We then write code to p, expecting it to be suitably aligned (our LEAF
macro aligns functions on 4 byte boundaries, so (ulong)handle_tlbl will
give a value one greater than a multiple of 4 - ie. the start of a
function on a 4 byte boundary, with the ISA mode bit 0 set).

This worked fine up to GCC 6, but GCC 7 & onwards is smart enough to
presume that handle_tlbl which we declared as an array of u32s must be
aligned sufficiently that bit 0 of its address will never be set, and as
a result optimize out msk_isa16_mode(). This leads to p having an
address with bit 0 set, and when we go on to attempt to store code at
that address we take an address error exception due to the unaligned
memory access.

This leads to an exception prior to the kernel having configured its own
exception handlers, so we jump to whatever handlers the bootloader
configured. In the case of QEMU this results in a silent hang, since it
has no useful general exception vector.

Fix this by consistently declaring our TLB-related functions as
functions. For handle_tlbl(), handle_tlbs() & handle_tlbm() we do this
in asm/tlbex.h & we make use of the existing declaration of
tlbmiss_handler_setup_pgd() in asm/mmu_context.h. Our TLB handler
generation code in arch/mips/mm/tlbex.c is adjusted to deal with these
definitions, in most cases simply by casting the function pointers to
u32 pointers.

This allows us to include asm/mmu_context.h in arch/mips/mm/tlbex.c to
get the definitions of tlbmiss_handler_setup_pgd & pgd_current, removing
some needless duplication. Consistently using msk_isa16_mode() on
function pointers means we no longer need the
tlbmiss_handler_setup_pgd_start symbol so that is removed entirely.

Now that we're declaring our functions as functions GCC stops optimizing
out msk_isa16_mode() & a microMIPS kernel built with either GCC 7.3.0 or
8.1.0 boots successfully.

Signed-off-by: Paul Burton <paul.burton@mips.com>

MIPS: Export tlbmiss_handler_setup_pgd near its definition

We export tlbmiss_handler_setup_pgd in arch/mips/mm/tlbex.c close to a
declaration of it, rather than close to its definition as is standard.

We've supported exporting symbols in assembly code since commit
22823ab419d8 ("EXPORT_SYMBOL() for asm"), so move the export to follow
the function's (stub) definition.

Signed-off-by: Paul Burton <paul.burton@mips.com>

MIPS: Avoid using array as parameter to write_c0_kpgd()

Passing an array (swapper_pg_dir) as the argument to write_c0_kpgd() in
setup_pw() will become problematic if we modify __write_64bit_c0_split()
to cast its val argument to unsigned long long, because for 32-bit
kernel builds the size of a pointer will differ from the size of an
unsigned long long. This would fall foul of gcc's pointer-to-int-cast
diagnostic.

Cast the value to a long, which should be the same width as the pointer
that we ultimately want & will be sign extended if required to the
unsigned long long that __write_64bit_c0_split() ultimately needs.

Signed-off-by: Paul Burton <paul.burton@mips.com>

MIPS: Remove unused R6000 support

The kernel contains a small amount of incomplete code aimed at
supporting old R6000 CPUs. This is:

- Unused, as no machine selects CONFIG_SYS_HAS_CPU_R6000.

- Broken, since there are glaring errors such as r6000_fpu.S moving
the FCSR register to t1, then ignoring it & instead saving t0 into
struct sigcontext...

- A maintenance headache, since it's code that nobody can test which
nevertheless imposes constraints on code which it shares with other
machines.

Remove this incomplete & broken R6000 CPU support in order to clean up
and in preparation for changes which will no longer need to consider
dragging the pretense of R6000 support along with them.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/16236/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Use current_cpu_type() in m4kc_tlbp_war()

Use current_cpu_type() to check for 4Kc processors instead of checking
the PRID directly. This will allow for the 4Kc case to be optimised out
of kernels that can't run on 4KC processors, thanks to __get_cpu_type()
and its unreachable() call.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/16205/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Handle tlbex-tlbp race condition

In systems where there are multiple actors updating the TLB, the
potential exists for a race condition wherein a CPU hits a TLB exception
but by the time it reaches a TLBP instruction the affected TLB entry may
have been replaced. This can happen if, for example, a CPU shares the
TLB between hardware threads (VPs) within a core and one of them
replaces the entry that another has just taken a TLB exception for.

We handle this race in the case of the Hardware Table Walker (HTW) being
the other actor already, but didn't take into account the potential for
multiple threads racing. Include the code for aborting TLB exception
handling in affected multi-threaded systems, those being the I6400 &
I6500 CPUs which share TLB entries between VPs.

In the case of using RiXi without dedicated exceptions we have never
handled this race even for HTW. This patch adds WARN()s to these cases
which ought never to be hit because all CPUs with either HTW or shared
FTLB RAMs also include dedicated RiXi exceptions, but the WARN()s will
ensure this is always the case.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/16203/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Add 48-bit VA space (and 4-level page tables) for 4K pages.

Some users must have 4K pages while needing a 48-bit VA space size.
The cleanest way do do this is to go to a 4-level page table for this
case. Each page table level using order-0 pages adds 9 bits to the
VA size (at 4K pages, so for four levels we get 9 * 4 + 12 == 48-bits.

For the 4K page size case only we add support functions for the PUD
level of the page table tree, also the TLB exception handlers get an
extra level of tree walk.

[david.daney@cavium.com: Forward port to v4.10.]
[david.daney@cavium.com: Forward port to v4.11.]

Signed-off-by: Alex Belits <alex.belits@cavium.com>
Signed-off-by: David Daney <david.daney@cavium.com>
Cc: James Hogan <james.hogan@imgtec.com>
Cc: Alex Belits <alex.belits@cavium.com>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/15312/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Flush wrong invalid FTLB entry for huge page

On VTLB+FTLB platforms (such as Loongson-3A R2), FTLB's pagesize is
usually configured the same as PAGE_SIZE. In such a case, Huge page
entry is not suitable to write in FTLB.

Unfortunately, when a huge page is created, its page table entries
haven't created immediately. Then the TLB refill handler will fetch an
invalid page table entry which has no "HUGE" bit, and this entry may be
written to FTLB. Since it is invalid, TLB load/store handler will then
use tlbwi to write the valid entry at the same place. However, the
valid entry is a huge page entry which isn't suitable for FTLB.

Our solution is to modify build_huge_handler_tail. Flush the invalid
old entry (whether it is in FTLB or VTLB, this is in order to reduce
branches) and use tlbwr to write the valid new entry.

Signed-off-by: Rui Wang <wangr@lemote.com>
Signed-off-by: Huacai Chen <chenhc@lemote.com>
Cc: John Crispin <john@phrozen.org>
Cc: Steven J . Hill <Steven.Hill@caviumnetworks.com>
Cc: Fuxin Zhang <zhangfx@lemote.com>
Cc: Zhangjin Wu <wuzhangjin@gmail.com>
Cc: Huacai Chen <chenhc@lemote.com>
Cc: linux-mips@linux-mips.org
Cc: stable@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/15754/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Export some tlbex internals for KVM to use

Export to TLB exception code generating functions so that KVM can
construct a fast TLB refill handler for guest context without
reinventing the wheel quite so much.

Signed-off-by: James Hogan <james.hogan@imgtec.com>
Acked-by: Ralf Baechle <ralf@linux-mips.org>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: "Radim Krčmář" <rkrcmar@redhat.com>
Cc: linux-mips@linux-mips.org
Cc: kvm@vger.kernel.org

MIPS: Export pgd/pmd symbols for KVM

Export pmd_init(), invalid_pmd_table and tlbmiss_handler_setup_pgd to
GPL kernel modules so that MIPS KVM can use the inline page table
management functions and switch between page tables:

- pmd_init() will be used directly by KVM to initialise newly allocated
pmd tables with invalid lower level table pointers.

- invalid_pmd_table is used by pud_present(), pud_none(), and
pud_clear(), which KVM will use to test and clear pud entries.

- tlbmiss_handler_setup_pgd() will be called by KVM entry code to switch
to the appropriate GVA page tables.

Signed-off-by: James Hogan <james.hogan@imgtec.com>
Acked-by: Ralf Baechle <ralf@linux-mips.org>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: "Radim Krčmář" <rkrcmar@redhat.com>
Cc: linux-mips@linux-mips.org
Cc: kvm@vger.kernel.org

MIPS: tlbex: Clear ISA bit when writing to handle_tlb{l,m,s}

When generating TLB exception handling code we write to memory reserved
at the handle_tlbl, handle_tlbm & handle_tlbs symbols. Up until now the
ISA bit has always been clear simply because the assembly code reserving
the space for those functions places no instructions in them. In
preparation for marking all LEAF functions as containing code,
explicitly clear the ISA bit when calculating the addresses at which to
write TLB exception handling code.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/14507/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

tree-wide: replace config_enabled() with IS_ENABLED()

The use of config_enabled() against config options is ambiguous. In
practical terms, config_enabled() is equivalent to IS_BUILTIN(), but the
author might have used it for the meaning of IS_ENABLED(). Using
IS_ENABLED(), IS_BUILTIN(), IS_MODULE() etc. makes the intention
clearer.

This commit replaces config_enabled() with IS_ENABLED() where possible.
This commit is only touching bool config options.

I noticed two cases where config_enabled() is used against a tristate
option:

- config_enabled(CONFIG_HWMON)
[ drivers/net/wireless/ath/ath10k/thermal.c ]

- config_enabled(CONFIG_BACKLIGHT_CLASS_DEVICE)
[ drivers/gpu/drm/gma500/opregion.c ]

I did not touch them because they should be converted to IS_BUILTIN()
in order to keep the logic, but I was not sure it was the authors'
intention.

Link: http://lkml.kernel.org/r/1465215656-20569-1-git-send-email-yamada.masahiro@socionext.com
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Kees Cook <keescook@chromium.org>
Cc: Stas Sergeev <stsp@list.ru>
Cc: Matt Redfearn <matt.redfearn@imgtec.com>
Cc: Joshua Kinard <kumba@gentoo.org>
Cc: Jiri Slaby <jslaby@suse.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: Markos Chandras <markos.chandras@imgtec.com>
Cc: "Dmitry V. Levin" <ldv@altlinux.org>
Cc: yu-cheng yu <yu-cheng.yu@intel.com>
Cc: James Hogan <james.hogan@imgtec.com>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Johannes Berg <johannes@sipsolutions.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Will Drewry <wad@chromium.org>
Cc: Nikolay Martynov <mar.kolya@gmail.com>
Cc: Huacai Chen <chenhc@lemote.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Cc: Rafal Milecki <zajec5@gmail.com>
Cc: James Cowgill <James.Cowgill@imgtec.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Alex Smith <alex.smith@imgtec.com>
Cc: Adam Buchbinder <adam.buchbinder@gmail.com>
Cc: Qais Yousef <qais.yousef@imgtec.com>
Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: Mikko Rapeli <mikko.rapeli@iki.fi>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Brian Norris <computersforpeace@gmail.com>
Cc: Hidehiro Kawai <hidehiro.kawai.ez@hitachi.com>
Cc: "Luis R. Rodriguez" <mcgrof@do-not-panic.com>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Cc: Roland McGrath <roland@hack.frob.com>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Kalle Valo <kvalo@qca.qualcomm.com>
Cc: Viresh Kumar <viresh.kumar@linaro.org>
Cc: Tony Wu <tung7970@gmail.com>
Cc: Huaitong Han <huaitong.han@intel.com>
Cc: Sumit Semwal <sumit.semwal@linaro.org>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Juergen Gross <jgross@suse.com>
Cc: Jason Cooper <jason@lakedaemon.net>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Andrea Gelmini <andrea.gelmini@gelma.net>
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Rabin Vincent <rabin@rab.in>
Cc: "Maciej W. Rozycki" <macro@imgtec.com>
Cc: David Daney <david.daney@cavium.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

MIPS: tlbex: Avoid duplicated single_insn_swpd

The expression "uasm_in_compat_space_p(swpd) && !uasm_rel_lo(swpd)" is
used twice in build_get_pgd_vmalloc64(), one of which is assigned to the
local variable single_insn_swpd. Update the other use to just use
single_insn_swpd instead to remove the duplication.

Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: David Daney <ddaney@caviumnetworks.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/13779/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Fix 64-bit HTW configuration

The Hardware page Table Walker (HTW) is being misconfigured on 64-bit
kernels. The PWSize.PS (pointer size) bit determines whether pointers
within directories are loaded as 32-bit or 64-bit addresses, but was
never being set to 1 for 64-bit kernels where the unsigned long in pgd_t
is 64-bits wide.

This actually reduces rather than improves performance when the HTW is
enabled on P6600 since the HTW is initiated lots, but walks are all
aborted due I think to bad intermediate pointers.

Since we were already taking the width of the PTEs into account by
setting PWSize.PTEW, which is the left shift applied to the page table
index *in addition to* the native pointer size, we also need to reduce
PTEW by 1 when PS=1. This is done by calculating PTEW based on the
relative size of pte_t compared to pgd_t.

Finally in order for the HTW to be used when PS=1, the appropriate
XK/XS/XU bits corresponding to the different 64-bit segments need to be
set in PWCtl. We enable only XU for now to enable walking for XUSeg.

Supporting walking for XKSeg would be a bit more involved so is left for
a future patch. It would either require the use of a per-CPU top level
base directory if supported by the HTW (a bit like pgd_current but with
a second entry pointing at swapper_pg_dir), or the HTW would prepend bit
63 of the address to the global directory index which doesn't really
match how we split user and kernel page directories.

Fixes: cab25bc7537b ("MIPS: Extend hardware table walking support to MIPS64")
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/13364/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Add 64-bit HTW fields

Add field definitions for some of the 64-bit specific Hardware page
Table Walker (HTW) register fields in PWSize and PWCtl, in preparation
for fixing the 64-bit HTW configuration.

Also print these fields out along with the others in print_htw_config().

Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/13363/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: mm: Panic if an XPA kernel is run without RIXI

XPA kernels hardcode for the presence of RIXI - the PTE format & its
handling presume RI & XI bits. Make this dependence explicit by panicing
if we run on a system that violates it.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Reviewed-by: James Hogan <james.hogan@imgtec.com>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/13125/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: mm: Don't do MTHC0 if XPA not present

Performing an MTHC0 instruction without XPA being present will trigger a
reserved instruction exception, therefore conditionalise the use of this
instruction when building TLB handlers (build_update_entries()), and in
__update_tlb().

This allows an XPA kernel to run on non XPA hardware without that
instruction implemented, just like it can run on XPA capable hardware
without XPA in use (with the noxpa kernel argument) or with XPA not
configured in hardware.

[paul.burton@imgtec.com:
- Rebase atop other TLB work.
- Add "mm" to subject.
- Handle the __kmap_pgprot case.]

Fixes: c5b367835cfc ("MIPS: Add support for XPA.")
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: David Hildenbrand <dahi@linux.vnet.ibm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Jerome Marchand <jmarchan@redhat.com>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/13124/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: mm: Simplify build_update_entries

We can simplify build_update_entries by unifying the code for the 36 bit
physical addressing with MIPS32 case with the general case, by using
pte_off_ variables in all cases & handling the trivial
_PAGE_GLOBAL_SHIFT == 0 case in build_convert_pte_to_entrylo. This
leaves XPA as the only special case.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Reviewed-by: James Hogan <james.hogan@imgtec.com>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/13123/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: mm: Be more explicit about PTE mode bit handling

The XPA case in iPTE_SW or's in software mode bits to the pte_low value
(which is what actually ends up in the high 32 bits of EntryLo...). It
does this presuming that only bits in the upper 16 bits of the 32 bit
pte_low value will be set. Make this assumption explicit with a BUG_ON.

A similar assumption is made for the hardware mode bits, which are or'd
in with a single ori instruction. Make that assumption explicit with a
BUG_ON too.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: James Hogan <james.hogan@imgtec.com>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/13122/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: mm: Pass scratch register through to iPTE_SW

Rather than hardcode a scratch register for the XPA case in iPTE_SW,
pass one through from the work registers allocated by the caller. This
allows for the XPA path to function correctly regardless of the work
registers in use.

Without doing this there are cases (where KScratch registers are
unavailable) in which iPTE_SW will incorrectly clobber $1 despite it
already being in use for the PTE or PTE pointer.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Reviewed-by: James Hogan <james.hogan@imgtec.com>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/13121/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: mm: Don't clobber $1 on XPA TLB refill

For XPA kernels build_update_entries() uses $1 (at) as a scratch
register, but doesn't arrange for it to be preserved, so it will always
be clobbered by the TLB refill exception. Although this register
normally has a very short lifetime that doesn't cross memory accesses,
TLB refills due to instruction fetches (either on a page boundary or
after preemption) could clobber live data, and its easy to reproduce
the clobber with a little bit of assembler code.

Note that the use of a hardware page table walker will partly mask the
problem, as the TLB refill handler will not always be invoked.

This is fixed by avoiding the use of the extra scratch register. The
pte_high parts (going into the lower half of the EntryLo registers) are
loaded and manipulated separately so as to keep the PTE pointer around
for the other halves (instead of storing in the scratch register), and
the pte_low parts (going into the high half of the EntryLo registers)
are masked with 0x00ffffff using an ext instruction (instead of loading
0x00ffffff into the scratch register and AND'ing).

[paul.burton@imgtec.com:
- Rebase atop other TLB work.
- Use ext instead of an sll, srl sequence.
- Use cpu_has_xpa instead of #ifdefs.
- Modify commit subject to include "mm".]

Fixes: c5b367835cfc ("MIPS: Add support for XPA.")
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/13120/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: mm: Fix MIPS32 36b physical addressing (alchemy, netlogic)

There are 2 distinct cases in which a kernel for a MIPS32 CPU
(CONFIG_CPU_MIPS32=y) may use 64 bit physical addresses
(CONFIG_PHYS_ADDR_T_64BIT=y):

- 36 bit physical addressing as used by RMI Alchemy & Netlogic XLP/XLR
CPUs.

- MIPS32r5 eXtended Physical Addressing (XPA).

These 2 cases are distinct in that they require different behaviour from
the kernel - the EntryLo registers have different formats. Until Linux
v4.1 we only supported the first case, with code conditional upon the 2
aforementioned Kconfig variables being set. Commit c5b367835cfc ("MIPS:
Add support for XPA.") added support for the second case, but did so by
modifying the code that existed for the first case rather than treating
the 2 cases as distinct. Since the EntryLo registers have different
formats this breaks the 36 bit Alchemy/XLP/XLR case. Fix this by
splitting the 2 cases, with XPA cases now being conditional upon
CONFIG_XPA and the non-XPA case matching the code as it existed prior to
commit c5b367835cfc ("MIPS: Add support for XPA.").

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Reported-by: Manuel Lauss <manuel.lauss@gmail.com>
Tested-by: Manuel Lauss <manuel.lauss@gmail.com>
Fixes: c5b367835cfc ("MIPS: Add support for XPA.")
Cc: James Hogan <james.hogan@imgtec.com>
Cc: David Daney <david.daney@cavium.com>
Cc: Huacai Chen <chenhc@lemote.com>
Cc: Maciej W. Rozycki <macro@linux-mips.org>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
Cc: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: David Hildenbrand <dahi@linux.vnet.ibm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Alex Smith <alex.smith@imgtec.com>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: stable@vger.kernel.org # v4.1+
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/13119/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: mm: Standardise on _PAGE_NO_READ, drop _PAGE_READ

Ever since support for RI/XI was implemented by commit 6dd9344cfc41
("MIPS: Implement Read Inhibit/eXecute Inhibit") we've had a mixture of
_PAGE_READ & _PAGE_NO_READ bits. Rather than keep both around, switch
away from using _PAGE_READ to determine page presence & instead invert
the use to _PAGE_NO_READ. Wherever we formerly had no definition for
_PAGE_NO_READ, change what was _PAGE_READ to _PAGE_NO_READ. The end
result is that we consistently use _PAGE_NO_READ to determine whether a
page is readable, regardless of whether RI/XI is implemented.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Reviewed-by: James Hogan <james.hogan@imgtec.com>
Cc: David Daney <david.daney@cavium.com>
Cc: Huacai Chen <chenhc@lemote.com>
Cc: Maciej W. Rozycki <macro@linux-mips.org>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Alex Smith <alex.smith@imgtec.com>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/13116/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Fix HTW config on XPA kernel without LPA enabled

The hardware page table walker (HTW) configuration is broken on XPA
kernels where XPA couldn't be enabled (either nohtw or the hardware
doesn't support it). This is because the PWSize.PTEW field (PTE width)
was only set to 8 bytes (an extra shift of 1) in config_htw_params() if
PageGrain.ELPA (enable large physical addressing) is set. On an XPA
kernel though the size of PTEs is fixed at 8 bytes regardless of whether
XPA could actually be enabled.

Fix the initialisation of this field based on sizeof(pte_t) instead.

Fixes: c5b367835cfc ("MIPS: Add support for XPA.")
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Steven J. Hill <sjhill@realitydiluted.com>
Cc: Paul Burton <paul.burton@imgtec.com>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/13113/
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Loongson-3: Fast TLB refill handler

Loongson-3A R2 has pwbase/pwfield/pwsize/pwctl registers in CP0 (this
is very similar to HTW) and lwdir/lwpte/lddir/ldpte instructions which
can be used for fast TLB refill.

[ralf@linux-mips.org: Resolve conflict.]

Signed-off-by: Huacai Chen <chenhc@lemote.com>
Cc: Aurelien Jarno <aurelien@aurel32.net>
Cc: Steven J . Hill <sjhill@realitydiluted.com>
Cc: Fuxin Zhang <zhangfx@lemote.com>
Cc: Zhangjin Wu <wuzhangjin@gmail.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/12754/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Fix misspellings in comments.

Signed-off-by: Adam Buchbinder <adam.buchbinder@gmail.com>
Cc: linux-mips@linux-mips.org
Cc: trivial@kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/12617/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Fix some missing CONFIG_CPU_MIPSR6 #ifdefs

Commit be0c37c985eddc4 (MIPS: Rearrange PTE bits into fixed positions.)
defines fixed PTE bits for MIPS R2. Then, commit d7b631419b3d230a4d383
(MIPS: pgtable-bits: Fix XPA damage to R6 definitions.) adds the MIPS
R6 definitions in the same way as MIPS R2. But some R6 #ifdefs in the
later commit are missing, so in this patch I fix that.

Signed-off-by: Huacai Chen <chenhc@lemote.com>
Cc: Aurelien Jarno <aurelien@aurel32.net>
Cc: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: Fuxin Zhang <zhangfx@lemote.com>
Cc: Zhangjin Wu <wuzhangjin@gmail.com>
Cc: linux-mips@linux-mips.org
Cc: stable@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/12164/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

mips, thp: remove infrastructure for handling splitting PMDs

With new refcounting we don't need to mark PMDs splitting. Let's drop
code to handle this.

pmdp_splitting_flush() is not needed too: on splitting PMD we will do
pmdp_clear_flush() + set_pte_at(). pmdp_clear_flush() will do IPI as
needed for fast_gup.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Sasha Levin <sasha.levin@oracle.com>
Cc: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
Cc: Jerome Marchand <jmarchan@redhat.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Rik van Riel <riel@redhat.com>
Cc: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
Cc: Steve Capper <steve.capper@linaro.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@suse.cz>
Cc: Christoph Lameter <cl@linux.com>
Cc: David Rientjes <rientjes@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

MIPS: Extend hardware table walking support to MIPS64

Extend the existing support for Hardware Table Walking (HTW) to MIPS64
systems by supporting PMDs & setting the pointer size bit in PWSize,
then ceasing to blacklist HTW on MIPS64 systems.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Cc: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: Joshua Kinard <kumba@gentoo.org>
Cc: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Cc: Maciej W. Rozycki <macro@linux-mips.org>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: linux-kernel@vger.kernel.org
Cc: James Hogan <james.hogan@imgtec.com>
Cc: Markos Chandras <markos.chandras@imgtec.com>
Patchwork: https://patchwork.linux-mips.org/patch/11224/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: tlbex: Avoid placing software PTE bits in Entry* PFN fields

Commit 748e787eb6de ("MIPS: Optimize TLB refill for RI/XI
configurations.") stopped explicitly clearing the bits used by software
in PTEs by making use of a rotate instruction that rotates them into the
fill bits of the Entry{Lo,Hi} register. This can only work if there are
actually enough fill bits in the register to cover the software
maintained bits, otherwise we end up writing those bits into the upper
bits of the PFN or PFNX field of the Entry{Lo,Hi} register.

Fix this by detecting the number of fill bits present in the
Entry{Lo,Hi} registers & explicitly clearing the software bits where
necessary.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Cc: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: linux-kernel@vger.kernel.org
Cc: James Hogan <james.hogan@imgtec.com>
Cc: Markos Chandras <markos.chandras@imgtec.com>
Patchwork: https://patchwork.linux-mips.org/patch/11218/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: tlbex: Share MIPS32 32 bit phys & MIPS64 64 bit phys code

The code in build_update_entries for 64 bit physical addresses on a
MIPS64 CPU and 32 bit physical addresses on a MIPS32 CPU is now
identical, with the exception of r4k bug workaround in the latter which
would simply not apply to the former. Remove the duplication and some

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Cc: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Cc: linux-kernel@vger.kernel.org
Cc: James Hogan <james.hogan@imgtec.com>
Cc: Markos Chandras <markos.chandras@imgtec.com>
Patchwork: https://patchwork.linux-mips.org/patch/11216/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: tlbex: Remove some RIXI redundancy

The cpu_has_rixi cases in build_update_entries are now identical to the
non-RIXI cases with the one exception of the r45k_bvahwbug case which is
hardcoded as never happening anyway & presumably was either missed from
the RIXI path or would never happen on a CPU with RIXI support. Remove
the redundant checks & duplication.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Cc: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: linux-kernel@vger.kernel.org
Cc: James Hogan <james.hogan@imgtec.com>
Cc: Markos Chandras <markos.chandras@imgtec.com>
Patchwork: https://patchwork.linux-mips.org/patch/11215/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: tlbex: Stop open-coding build_convert_pte_to_entrylo

Make use of build_convert_pte_to_entrylo in the RIXI cases within
build_update_entries rather than open-coding it 4 times.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Cc: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Cc: linux-kernel@vger.kernel.org
Cc: James Hogan <james.hogan@imgtec.com>
Cc: Markos Chandras <markos.chandras@imgtec.com>
Patchwork: https://patchwork.linux-mips.org/patch/11214/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: tlbex: Avoid unnecessary _PAGE_PRESENT shifts

Commit c5b367835cfc ("MIPS: Add support for XPA.") added generation of a
shift by _PAGE_PRESENT_SHIFT in build_pte_present() and
build_pte_writable(), however except for the XPA case this is always
zero making it unnecessary.

Make the shift conditional upon _PAGE_PRESENT_SHIFT being non-zero to
save an instruction in those cases.

Fixes: c5b367835cfc ("MIPS: Add support for XPA.")
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9889/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: tlbex: Fix broken offsets on r2 without XPA

Commit c5b367835cfc ("MIPS: Add support for XPA.") changed
build_pte_present() and build_pte_writable() to assume a constant offset
of _PAGE_READ and _PAGE_WRITE relative to _PAGE_PRESENT, however this is
no longer true for some MIPS32R2 builds since commit be0c37c985ed
("MIPS: Rearrange PTE bits into fixed positions.") which moved the
_PAGE_READ PTE bit away from the _PAGE_PRESENT bit, with the _PAGE_WRITE
bit falling into its place.

Make use of the _PAGE_READ and _PAGE_WRITE definitions to calculate the
correct mask to apply instead of hard coding 3 (for _PAGE_PRESENT |
_PAGE_READ) or 5 (for _PAGE_PRESENT | _PAGE_WRITE).

Fixes: c5b367835cfc ("MIPS: Add support for XPA.")
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9888/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: tlbex.c: Remove new instance of __cpuinitdata that crept back in

We removed __cpuinit support (leaving no-op stubs) quite some time ago.
However a new instance was added in commit c5b367835cfc7a8ef53b9670a409ff
("MIPS: Add support for XPA.")

Since we want to clobber the stubs soon, get this removed now.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/9894/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

mips/mm/tlbex: remove new instance of __cpuinit that crept back in

We removed __cpuinit support (leaving no-op stubs) quite some time ago.
However a new instance was added in commit c5b367835cfc7a8ef53b9670a409ff
("MIPS: Add support for XPA.")

Since we want to clobber the stubs soon, get this removed now.

Cc: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: linux-mips@linux-mips.org
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>

Revert "MIPS: Avoid pipeline stalls on some MIPS32R2 cores."

For a discussion, see http://patchwork.linux-mips.org/patch/9539/.

This reverts commit 625c0a21700bdb90844d926a1508a17a77e369c9.

MIPS: Add R16000 detection

This allows the kernel to correctly detect an R16000 MIPS CPU on systems that
have those. Otherwise, such systems will detect the CPU as an R14000, due to
similarities in the CPU PRId value.

Signed-off-by: Joshua Kinard <kumba@gentoo.org>
Cc: Linux MIPS List <linux-mips@linux-mips.org>
Patchwork: https://patchwork.linux-mips.org/patch/9092/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Add support for XPA.

Add support for extended physical addressing (XPA) so that
32-bit platforms can access equal to or greater than 40 bits
of physical addresses.

NOTE:
1) XPA and EVA are not the same and cannot be used
simultaneously.
2) If you configure your kernel for XPA, the PTEs
and all address sizes become 64-bit.
3) Your platform MUST have working HIGHMEM support.

Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9355/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Rearrange PTE bits into fixed positions.

This patch rearranges the PTE bits into fixed positions for R2
and later cores. In the past, the TLB handling code did runtime
checking of RI/XI and adjusted the shifts and rotates in order
to fit the largest PFN value into the PTE. The checking now
occurs when building the TLB handler, thus eliminating those
checks. These new arrangements also define the largest possible
PFN value that can fit in the PTE. HUGE page support is only
available for 64-bit cores. Layouts of the PTE bits are now:

64-bit, R1 or earlier: CCC D V G [S H] M A W R P
32-bit, R1 or earler: CCC D V G M A W R P
64-bit, R2 or later: CCC D V G RI/R XI [S H] M A W P
32-bit, R2 or later: CCC D V G RI/R XI M A W P

[ralf@linux-mips.org: Fix another build error *rant* *rant*]

Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9353/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: mm: tlbex: Use cpu_has_mips_r2_exec_hazard for the EHB instruction

MIPS uses the cpu_has_mips_r2_exec_hazard macro to determine whether the
EHB instruction is available or not. This is necessary for MIPS R6
which also supports the EHB instruction.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>

MIPS: Add cases for CPU_QEMU_GENERIC

Add a CPU_QEMU_GENERIC case to various switch statements.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>

MIPS: tlbex: Fix potential HTW race on TLBL/M/S handlers

There is a potential race when probing the TLB in TLBL/M/S exception
handlers for a matching entry. Between the time we hit a TLBL/S/M
exception and the time we get to execute the TLBP instruction, the
HTW may have replaced the TLB entry we are interested in hence the TLB
probe may fail. However, in the existing handlers, we never checked the
status of the TLBP (ie check the result in the C0/Index register). We
fix this by adding such a check when the core implements the HTW. If
we couldn't find a matching entry, we return back and try again.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Reviewed-by: James Hogan <james.hogan@imgtec.com>
Cc: <stable@vger.kernel.org> # v3.17+
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/8599/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Replace MIPS-specific 64BIT_PHYS_ADDR with generic PHYS_ADDR_T_64BIT

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: tlbex: Properly fix HUGE TLB Refill exception handler

In commit 8393c524a25609 (MIPS: tlbex: Fix a missing statement for
HUGETLB), the TLB Refill handler was fixed so that non-OCTEON targets
would work properly with huge pages. The change was incorrect in that
it broke the OCTEON case.

The problem is shown here:

xxx0: df7a0000 ld k0,0(k1)
.
.
.
xxxc0: df610000 ld at,0(k1)
xxxc4: 335a0ff0 andi k0,k0,0xff0
xxxc8: e825ffcd bbit1 at,0x5,0x0
xxxcc: 003ad82d daddu k1,at,k0
.
.
.

In the non-octeon case there is a destructive test for the huge PTE
bit, and then at 0, $k0 is reloaded (that is what the 8393c524a25609
patch added).

In the octeon case, we modify k1 in the branch delay slot, but we
never need k0 again, so the new load is not needed, but since k1 is
modified, if we do the load, we load from a garbage location and then
get a nested TLB Refill, which is seen in userspace as either SIGBUS
or SIGSEGV (depending on the garbage).

The real fix is to only do this reloading if it is needed, and never
where it is harmful.

Signed-off-by: David Daney <david.daney@cavium.com>
Cc: Huacai Chen <chenhc@lemote.com>
Cc: Fuxin Zhang <zhangfx@lemote.com>
Cc: Zhangjin Wu <wuzhangjin@gmail.com>
Cc: stable@vger.kernel.org
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/8151/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Use dedicated exception handler if CPU supports RI/XI exceptions

Use the regular tlb_do_page_fault_0 (no write) handler to handle
the RI and XI exceptions. Also skip the RI/XI validation check
on TLB load handler since it's redundant when the CPU has
unique RI/XI exceptions.

Singed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/7339/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: mm: Use the Hardware Page Table Walker if the core supports it

The Hardware Page Table Walker aims to speed up TLB refill exceptions
by handling them in the hardware level instead of having a software
TLB refill handler. However, a TLB refill exception can still be
thrown in certain cases such as, synchronus exceptions, or address
translation or memory errors during the HTW operation. As a result of
which, HTW must not be considered a complete replacement for the TLB
refill software handler, but rather a fast-path for it.
For HTW to work, the PWBase register must contain the task's page
global directory address so the HTW will kick in on TLB refill
exceptions.

Due to HTW being a separate engine embedded deep in the CPU pipeline,
we need to restart the HTW everytime a PTE changes to avoid HTW
fetching a old entry from the page tables. It's also necessary to
restart the HTW on context switches to prevent it from fetching a
page from the previous process. Finally, since HTW is using the
entryhi register to write the translations to the TLB, it's necessary
to stop the HTW whenever the entryhi changes (eg for tlb probe
perations) and enable it back afterwards.

== Performance ==

The following trivial test was used to measure the performance of the
HTW. Using the same root filesystem, the following command was used
to measure the number of tlb refill handler executions with and
without (using 'nohtw' kernel parameter) HTW support. The kernel was
modified to use a scratch register as a counter for the TLB refill
exceptions.

find /usr -type f -exec ls -lh {} \;

HTW Enabled:
TLB refill exceptions: 12306

HTW Disabled:
TLB refill exceptions: 17805

Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-mips@linux-mips.org
Cc: Markos Chandras <markos.chandras@imgtec.com>
Patchwork: https://patchwork.linux-mips.org/patch/7336/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: bugfix: missed cache flush of TLB refill handler

Commit

Commit 1d40cfcd3442a53e98468cdb3e6d4d9a568d76cf
Author: Ralf Baechle <ralf@linux-mips.org>
Date: Fri Jul 15 15:23:23 2005 +0000

Avoid SMP cacheflushes. This is a minor optimization of startup but
will also avoid smp_call_function from doing stupid things when called
from a CPU that is not yet marked online.

missed an appropriate cache flush of TLB refill handler because that time it was
at fixed location CAC_BASE. After years the refill handler in EBASE vector
is not at that location and can be allocated in some another memory and needs
I-cache sync as other TLB exception vectors.

Besides that, the new function - local_flash_icache_range() was introduced
to avoid SMP cacheflushes.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
Cc: paul.gortmaker@windriver.com
Cc: jchandra@broadcom.com
Cc: linux-kernel@vger.kernel.org
Cc: david.daney@cavium.com
Patchwork: https://patchwork.linux-mips.org/patch/7312/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: tlbex: Fix a missing statement for HUGETLB

In commit 2c8c53e28f1 (MIPS: Optimize TLB handlers for Octeon CPUs)
build_r4000_tlb_refill_handler() is modified. But it doesn't compatible
with the original code in HUGETLB case. Because there is a copy & paste
error and one line of code is missing. It is very easy to produce a bug
with LTP's hugemmap05 test.

Signed-off-by: Huacai Chen <chenhc@lemote.com>
Signed-off-by: Binbin Zhou <zhoubb@lemote.com>
Cc: <stable@vger.kernel.org>
Cc: John Crispin <john@phrozen.org>
Cc: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: linux-mips@linux-mips.org
Cc: Fuxin Zhang <zhangfx@lemote.com>
Cc: Zhangjin Wu <wuzhangjin@gmail.com>
Patchwork: https://patchwork.linux-mips.org/patch/7496/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Add code for new system 'paravirt'

For para-virtualized guests running under KVM or other equivalent
hypervisor.

Signed-off-by: David Daney <david.daney@cavium.com>
Signed-off-by: Andreas Herrmann <andreas.herrmann@caviumnetworks.com>
Cc: linux-mips@linux-mips.org
Cc: James Hogan <james.hogan@imgtec.com>
Cc: kvm@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/7004/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Don't build fast TLB refill handler with 32-bit kernels

The fast handler only supports 64-bit kernels.

Signed-off-by: David Daney <david.daney@cavium.com>
Signed-off-by: Andreas Herrmann <andreas.herrmann@caviumnetworks.com>
Cc: linux-mips@linux-mips.org
Cc: James Hogan <james.hogan@imgtec.com>
Cc: kvm@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/7010/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: mm: Fix broken microMIPS kernel regression.

Commit f4ae17aa0f2122b52f642985b46210a1f2eceb0a [MIPS: mm: Use scratch for
PGD when !CONFIG_MIPS_PGD_C0_CONTEXT] broke microMIPS kernel builds. This
patch refactors that code similar to what was done for the 'clear_page'
and 'copy_page' functions.

Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/6744/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Loongson: Add basic Loongson-3 CPU support

Basic Loongson-3 CPU support include CPU probing and TLB/cache
initializing.

Signed-off-by: Huacai Chen <chenhc@lemote.com>
Signed-off-by: Hongliang Tao <taohl@lemote.com>
Signed-off-by: Hua Yan <yanh@lemote.com>
Tested-by: Alex Smith <alex.smith@imgtec.com>
Reviewed-by: Alex Smith <alex.smith@imgtec.com>
Cc: John Crispin <john@phrozen.org>
Cc: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: Aurelien Jarno <aurelien@aurel32.net>
Cc: linux-mips@linux-mips.org
Cc: Fuxin Zhang <zhangfx@lemote.com>
Cc: Zhangjin Wu <wuzhangjin@gmail.com>
Patchwork: https://patchwork.linux-mips.org/patch/6630
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Use current_cpu_type() instead of c->cputype

If current_cpu_type() is pre-defined in cpu-feature-overrides.h, This
may save about 10k for the compressed kernel image(vmlinuz).

Signed-off-by: Wu Zhangjin <wuzhangjin@gmail.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/1901/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Add support for the M5150 processor

The M5150 core is a 32-bit MIPS RISC which implements the
MIPS Architecture Release-5 in a 5-stage pipeline.
In addition, it includes the MIPS Architecture Virtualization Module
that enables virtualization of operating systems,
which provides a scalable, trusted, and secure execution environment.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/6596/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Add cases for CPU_P5600

Add a CPU_P5600 case to various switch statements, doing the same thing
as for CPU_PROAPTIV.

Signed-off-by: James Hogan <james.hogan@imgtec.com>
Reviewed-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/6408/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Add 1074K CPU support explicitly.

The 1074K is a multiprocessing coherent processing system (CPS) based
on modified 74K cores. This patch makes the 1074K an actual unique
CPU type, instead of a 74K derivative, which it is not.

Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
Reviewed-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/6389/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

mips: delete non-required instances of include <linux/init.h>

None of these files are actually using any __init type directives
and hence don't need to include <linux/init.h>. Most are just a
left over from __devinit and __cpuinit removal, or simply due to
code getting copied from one driver to the next.

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: John Crispin <blogic@openwrt.org>
Patchwork: http://patchwork.linux-mips.org/patch/6320/

MIPS: Add support for the proAptiv cores

The proAptiv Multiprocessing System is a power efficient multi-core
microprocessor for use in system-on-chip (SoC) applications.
The proAptiv Multiprocessing System combines a deep pipeline
with multi-issue out of order execution for improved computational
throughput. The proAptiv Multiprocessing System can contain one to
six MIPS32r3 proAptiv cores, system level coherence
manager with L2 cache, optional coherent I/O port, and optional
floating point unit.

Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Signed-off-by: John Crispin <blogic@openwrt.org>
Patchwork: http://patchwork.linux-mips.org/patch/6134/

MIPS: Loongson: Get rid of Loongson 2 #ifdefery all over arch/mips.

It was ugly.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: mm: Use scratch for PGD when !CONFIG_MIPS_PGD_C0_CONTEXT

Allow usage of scratch register for current pgd even when
MIPS_PGD_C0_CONTEXT is not configured. MIPS_PGD_C0_CONTEXT is set
for 64r2 platforms to indicate availability of Xcontext for saving
cpuid, thus freeing Context to be used for saving PGD. This option
was also tied to using a scratch register for storing PGD.

This commit will allow usage of scratch register to store the current
pgd if one can be allocated for the platform, even when
MIPS_PGD_C0_CONTEXT is not set. The cpuid will be kept in the CP0
Context register in this case.

The code to store the current pgd for the TLB miss handler is now
generated in all cases. When scratch register is available, the PGD
is also stored in the scratch register.

Signed-off-by: Jayachandran C <jchandra@broadcom.com>
Cc: linux-mips@linux-mips.org
Cc: Hauke Mehrtens <hauke@hauke-m.de>
Patchwork: https://patchwork.linux-mips.org/patch/5906/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Move definition of SMP processor id register to header file

The definition of the CP0 register used to save the smp processor
id is repicated in many files, move them all to thread_info.h.

Signed-off-by: Jayachandran C <jchandra@broadcom.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5708/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Optimize current_cpu_type() for better code.

o Move current_cpu_type() to a separate header file
o #ifdefing on supported CPU types lets modern GCC know that certain
code in callers may be discarded ideally turning current_cpu_type() into
a function returning a constant.
o Use current_cpu_type() rather than direct access to struct cpuinfo_mips.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Cc: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5833/

MIPS: Generate OCTEON3 TLB handlers with the same features as OCTEON2.

OCTEON2 need the same code.

Signed-off-by: David Daney <david.daney@cavium.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5637/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: tlbex: fix broken build in v3.11-rc1

Commit 6ba045f9fbdafb48da42aa8576ea7a3980443136 (MIPS: Move generated code
to .text for microMIPS) deleted tlbmiss_handler_setup_pgd_array, but some
references were not converted. Fix that to enable building a MIPS kernel.

Signed-off-by: Aaro Koskinen <aaro.koskinen@iki.fi>
Acked-by: Jayachandran C. <jchandra@broadcom.com>
Acked-by: David Daney <david.daney@cavium.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5589/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: tlbex: Fix typo in r3000 tlb store handler

commit 6ba045f (MIPS: Move generated code to .text for microMIPS)
causes a panic at boot. The handler builder should test against
handle_tlbs_end, not handle_tlbs.

Signed-off-by: Tony Wu <tung7970@gmail.com>
Acked-by: Jayachandran C. <jchandra@broadcom.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5600/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Delete __cpuinit/__CPUINIT usage from MIPS code

commit 3747069b25e419f6b51395f48127e9812abc3596 upstream.

The __cpuinit type of throwaway sections might have made sense
some time ago when RAM was more constrained, but now the savings
do not offset the cost and complications. For example, the fix in
commit 5e427ec2d0 ("x86: Fix bit corruption at CPU resume time")
is a good example of the nasty type of bugs that can be created
with improper use of the various __init prefixes.

After a discussion on LKML[1] it was decided that cpuinit should go
the way of devinit and be phased out. Once all the users are gone,
we can then finally remove the macros themselves from linux/init.h.

Note that some harmless section mismatch warnings may result, since
notify_cpu_starting() and cpu_up() are arch independent (kernel/cpu.c)
and are flagged as __cpuinit -- so if we remove the __cpuinit from
the arch specific callers, we will also get section mismatch warnings.
As an intermediate step, we intend to turn the linux/init.h cpuinit
related content into no-ops as early as possible, since that will get
rid of these warnings. In any case, they are temporary and harmless.

Here, we remove all the MIPS __cpuinit from C code and __CPUINIT
from asm files. MIPS is interesting in this respect, because there
are also uasm users hiding behind their own renamed versions of the
__cpuinit macros.

[1] https://lkml.org/lkml/2013/5/20/589

[ralf@linux-mips.org: Folded in Paul's followup fix.]

Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5494/
Patchwork: https://patchwork.linux-mips.org/patch/5495/
Patchwork: https://patchwork.linux-mips.org/patch/5509/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Fix typos and cleanup comment

Signed-off-by: Tony Wu <tung7970@gmail.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5535/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Move generated code to .text for microMIPS

Prepare of a next patch which will call tlbmiss_handler_setup_pgd on
microMIPS. MicroMIPS complains if the called code s not in the .text
section. To fix this we generate code into space reserved in
arch/mips/mm/tlb-funcs.S

While there, move the rest of the generated functions (handle_tlbl,
handle_tlbs, handle_tlbm) to the same file.

Signed-off-by: Jayachandran C <jchandra@broadcom.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5542/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: tlbex: Fix size of area to be flushed.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Flush TLB handlers directly after writing them

When having enabled MIPS_PGD_C0_CONTEXT, trap_init() might call the
generated tlbmiss_handler_setup_pgd before it was committed to memory,
causing boot failures:

trap_init()
|- per_cpu_trap_init()
| |- TLBMISS_HANDLER_SETUP()
| |- tlbmiss_handler_setup_pgd()
|- flush_tlb_handlers()

To avoid this, move flush_tlb_handlers() into build_tlb_refill_handler()
right after they were generated. We can do this as the cache handling is
initialized just before creating the tlb handlers.

This issue was introduced in 3d8bfdd0307223de678962f1c1907a7cec549136
("MIPS: Use C0_KScratch (if present) to hold PGD pointer.").

Signed-off-by: Jonas Gorski <jogo@openwrt.org>
Cc: linux-mips@linux-mips.org
Cc: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: Jayachandran C <jchandra@broadcom.com>
Cc: David Daney <david.daney@cavium.com>
Patchwork: https://patchwork.linux-mips.org/patch/5539/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Fix TLBR-use hazards for R2 cores in the TLB reload handlers

MIPS R2 documents state that an execution hazard barrier is needed
after a TLBR before reading EntryLo.

Original patch by Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Patchwork: https://patchwork.linux-mips.org/patch/5526/

MIPS: Fixup check for invalid scratch register

The invalid value for scratch register is -1, so update the checks of
the form (scratch_reg > 0) to be (scratch_reg >= 0). This will fix
the case in Netlogic XLP where the scratch_reg can be 0.

Signed-off-by: Jayachandran C <jchandra@broadcom.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5444/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Allow platform specific scratch registers

XLR/XLP COP0 scratch is register 22, sel 0-7. Add a function
c0_kscratch() which returns the scratch register for the platform,
and use the return value while generating TLB handlers.

Setup kscratch_mask to 0xf for XLR/XLP since the config4 register
does not exist. This allows the kernel to allocate scratch registers
0-3 if needed.

Signed-off-by: Jayachandran C <jchandra@broadcom.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5445/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Remove duplicate definition of check_for_high_segbits.

In C, one definition is sufficient.

Signed-off-by: David Daney <david.daney@cavium.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/5304/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Revert "MIPS: Allow ASID size to be determined at boot time."

This reverts commit d532f3d26716a39dfd4b88d687bd344fbe77e390.

The original commit has several problems:

1) Doesn't work with 64-bit kernels.

2) Calls TLBMISS_HANDLER_SETUP() before the code is generated.

3) Calls TLBMISS_HANDLER_SETUP() twice in per_cpu_trap_init() when
only one call is needed.

[ralf@linux-mips.org: Also revert the bits of the ASID patch which were
hidden in the KVM merge.]

Signed-off-by: David Daney <david.daney@cavium.com>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Cc: "Steven J. Hill" <Steven.Hill@imgtec.com>
Cc: David Daney <david.daney@cavium.com>
Patchwork: https://patchwork.linux-mips.org/patch/5242/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Revert "MIPS: microMIPS: Support dynamic ASID sizing."

This reverts commit f6b06d9361a008afb93b97fb3683a6e92d69d0f4.

The next revert depends on this one, so this has to go too.

Signed-off-by: David Daney <david.daney@cavium.com>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Cc: "Steven J. Hill" <Steven.Hill@imgtec.com>
Cc: David Daney <david.daney@cavium.com>
Patchwork: https://patchwork.linux-mips.org/patch/5241/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: microMIPS: Add support for exception handling.

All exceptions must be taken in microMIPS mode, never in classic
MIPS mode or the kernel falls apart. A few NOP instructions are
used to maintain the correct alignment of microMIPS versions of
the exception vectors.

Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>

MIPS: microMIPS: Support dynamic ASID sizing.

Changes for pure microMIPS cores to dynamically determine the ASID
size at boot time.

Includes bug fix https://patchwork.linux-mips.org/patch/5230/

Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
Signed-off-by: Jonas Gorski <jogo@openwrt.org>

MIPS: Allow ASID size to be determined at boot time.

Original patch by Ralf Baechle and removed by Harold Koerfgen
with commit f67e4ffc79905482c3b9b8c8dd65197bac7eb508. This
allows for more generic kernels since the size of the ASID
and corresponding masks can be determined at run-time. This
patch is also required for the new Aptiv cores and has been
tested on Malta and Malta Aptiv platforms.

[ralf@linux-mips.org: Added relevant part of fix
https://patchwork.linux-mips.org/patch/5213/]

Signed-off-by: Steven J. Hill <Steven.Hill@imgtec.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Build uasm-generated code only once to avoid CPU Hotplug problem

This and the next patch resolve memory corruption problems while CPU
hotplug. Without these patches, memory corruption can triggered easily
as below:

On a quad-core MIPS platform, use "spawn" of UnixBench-5.1.3 (http://
code.google.com/p/byte-unixbench/) and a CPU hotplug script like this
(hotplug.sh):
while true; do
echo 0 >/sys/devices/system/cpu/cpu1/online
echo 0 >/sys/devices/system/cpu/cpu2/online
echo 0 >/sys/devices/system/cpu/cpu3/online
sleep 1
echo 1 >/sys/devices/system/cpu/cpu1/online
echo 1 >/sys/devices/system/cpu/cpu2/online
echo 1 >/sys/devices/system/cpu/cpu3/online
sleep 1
done

Run "hotplug.sh" and then run "spawn 10000", spawn will get segfault
after a few minutes.

This patch:
Currently, clear_page()/copy_page() are generated by Micro-assembler
dynamically. But they are unavailable until uasm_resolve_relocs() has
finished because jump labels are illegal before that. Since these
functions are shared by every CPU, we only call build_clear_page()/
build_copy_page() only once at boot time. Without this patch, programs
will get random memory corruption (segmentation fault, bus error, etc.)
while CPU Hotplug (e.g. one CPU is using clear_page() while another is
generating it in cpu_cache_init()).

For similar reasons we modify build_tlb_refill_handler()'s invocation.

V2:
1, Rework the code to make CPU#0 can be online/offline.
2, Introduce cpu_has_local_ebase feature since some types of MIPS CPU
need a per-CPU tlb_refill_handler().

Signed-off-by: Huacai Chen <chenhc@lemote.com>
Signed-off-by: Hongbing Hu <huhb@lemote.com>
Acked-by: David Daney <david.daney@cavium.com>
Patchwork: http://patchwork.linux-mips.org/patch/4994/
Acked-by: John Crispin <blogic@openwrt.org>

MIPS: Make declarations and definitions of tlbmiss_handler_setup_pgd match.

tlbmiss_handler_setup_pgd is run-time generated code and it was convenient
to pretend the symbol was an array in the generator but a function for
the users. LTO gcc won't tolerate this kind of lie anymore so solve the
problem through a cast and function pointer instead.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Add support for the M14KEc core.

Signed-off-by: Steven J. Hill <sjhill@mips.com>
Patchwork: http://patchwork.linux-mips.org/patch/4682/
Signed-off-by: John Crispin <blogic@openwrt.org>

MIPS: Whitespace cleanup.

Having received another series of whitespace patches I decided to do this
once and for all rather than dealing with this kind of patches trickling
in forever.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Revert "MIPS: Optimise TLB handlers for MIPS32/64 R2 cores."

This reverts commit ff401e52100dcdc85e572d1ad376d3307b3fe28e.

This breaks on MIPS64 R2 cores such as Broadcom's.

MIPS: PMC-Sierra Yosemite: Remove support.

Nobody seems to be interested anymore and upstream also never had an
ethernet driver.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Transparent Huge Pages support

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Control huge tlb support via Kconfig symbol MIPS_HUGE_TLB_SUPPORT

We need Huge TLBs for HUGETLB_PAGE, or the soon to follow
TRANSPARENT_HUGEPAGE. collect this information under a single Kconfig
symbol.

Signed-off-by: David Daney <david.daney@cavium.com>

MIPS: tlbex: Better debug output.

Pgtable bits are assigned dynamically depending on processor feature and
statically based on kernel configuration. To make sense out of the
disassembled TLB exception handlers a list of the actual assignments
used for a particular configuration and hardware setup can be very useful.

Output the actual TLB exception handlers in a format that simplifies their
post processsing from dmesg output.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Remove R5000A.

From a software perspective R5000 and R5000A are the same thing which is
why the symbol CPU_R5000A never got used, so finally delete it.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: tlbex: Fix section mismatches

The new functions introduced in commit 02a5417751 (MIPS: tlbex: Deal with
re-definition of label) should be marked __cpuinit, to eliminate a
warning that can pop up when CONFIG_EXPORT_UASM is disabled:

LD arch/mips/mm/built-in.o
WARNING: arch/mips/mm/built-in.o(.text+0x2a4c): Section mismatch in reference from the function uasm_bgezl_hazard() to the function .cpuinit.text:uasm_il_bgezl()
The function uasm_bgezl_hazard() references
the function __cpuinit uasm_il_bgezl().
This is often because uasm_bgezl_hazard lacks a __cpuinit
annotation or the annotation of uasm_il_bgezl is wrong.

WARNING: arch/mips/mm/built-in.o(.text+0x2a68): Section mismatch in reference from the function uasm_bgezl_label() to the function .cpuinit.text:uasm_build_label()
The function uasm_bgezl_label() references
the function __cpuinit uasm_build_label().
This is often because uasm_bgezl_label lacks a __cpuinit
annotation or the annotation of uasm_build_label is wrong.

(This warning might not occur if the function was inlined.)

Signed-off-by: Kevin Cernekee <cernekee@gmail.com>
Patchwork: http://patchwork.linux-mips.org/patch/4517
Signed-off-by: John Crispin <blogic@openwrt.org>

MIPS: R5000: Fix TLB hazard handling.

R5000 and the Nevada CPUs (RM5230, RM5231, RM5260, RM5261, RM5270 and
RM5271) are basically the same CPU core and all are documented to require
two instructions separating a write to c0_pagemask, c0_entryhi, c0_entrylo0,
c0_entrylo1 or c0_index.

So far we were only providing on cycle before / after a TLBR/TLBWI
for R5000 but 3 cycles before and 1 cycles after for the Nevadas.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: tlbex: Deal with re-definition of label

The microassembler used in tlbex.c does not notice if a label is redefined
resulting in relocations against such labels silently missrelocated.
The issues exists since commit add6eb04776db4189ea89f596cbcde31b899be9d
[Synthesize TLB exception handlers at runtime.] in 2.6.10 and went unnoticed
for so long because the relocations for the affected branches got computed
to do something *almost* sensible.

The issue affects R4000, R4400, QED/IDT RM5230, RM5231, RM5260, RM5261,
RM5270 and RM5271 processors.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Optimize TLB refill for RI/XI configurations.

We don't have to do a separate shift to eliminate the software bits,
just rotate them into the fill and they will be ignored.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/4294/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Replace 'kernel_uses_smartmips_rixi' with 'cpu_has_rixi'.

Remove usage of the 'kernel_uses_smartmips_rixi' macro from all files
and use new 'cpu_has_rixi' instead.

Signed-off-by: Steven J. Hill <sjhill@mips.com>
Acked-by: David Daney <david.daney@cavium.com>

MIPS: Optimise TLB handlers for MIPS32/64 R2 cores.

The EXT and INS instructions can be used to decrease code size and
thus speed up TLB handlers on MIPS32R2 and MIPS64R2 cores.

Signed-off-by: Steven J. Hill <sjhill@mips.com>

MIPS: Avoid pipeline stalls on some MIPS32R2 cores.

The architecture specification says that an EHB instruction is
needed to avoid a hazard when writing TLB entries. However, some
cores do not have this hazard, and thus the EHB instruction causes
a costly pipeline stall. Detect these cores and do not use the EHB
instruction.

Signed-off-by: Steven J. Hill <sjhill@mips.com>

MIPS: Add support for the M14Kc core.

[ralf@linux-mips.org: Fixed whitespace damage.]

Signed-off-by: Steven J. Hill <sjhill@mips.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/3773/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Disintegrate asm/system.h for MIPS

Disintegrate asm/system.h for MIPS.

Signed-off-by: David Howells <dhowells@redhat.com>
Acked-by: Ralf Baechle <ralf@linux-mips.org>
cc: linux-mips@linux-mips.org

MIPS: No branches in delay slots for huge pages in handle_tlbl

For the case PM_DEFAULT_MASK == 0, we were placing a branch in the
delay slot of another branch. This leads to undefined behavior.

Signed-off-by: David Daney <david.daney@cavium.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/2775/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: tlbex: Fix build error in R3000 code.

Only some GCC versions such as gcc 4.2 notice that the variable wr in
build_r3000_tlb_modify_handler is used uninitialized. When using one
of those GCCs the build will fail due to -Werror. GCC 4.6 does not
warn about the uninitialized use of wr.

This issue was introduced by 7211f4d7a3dcbe57c5d396c334dca525315dceb2
[MIPS: Close races in TLB modify handlers.]

Reported-by: Ganesan Ramalingam <ganesan18@gmail.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Close races in TLB modify handlers.

Page table entries are made invalid by writing a zero into the the PTE
slot in a page table. This creates a race condition with the TLB
modify handlers when they are updating the PTE.

CPU0 CPU1

Test for _PAGE_PRESENT
. set to not _PAGE_PRESENT (zero)
Set to _PAGE_VALID

So now the page not present value (zero) is suddenly valid and user
space programs have access to physical page zero.

We close the race by putting the test for _PAGE_PRESENT and setting of
_PAGE_VALID into an atomic LL/SC section. This requires more registers
than just K0 and K1 in the handlers, so we need to save some registers
to a save area and then restore them when we are done.

The save area is an array of cacheline aligned structures that should
not suffer cache line bouncing as they are CPU private.

[ralf@linux-mips.org: Fix !defined(CONFIG_MIPS_PGD_C0_CONTEXT) build error.]

Signed-off-by: David Daney <david.daney@cavium.com>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/2577/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Netlogic: Cache, TLB support and feature overrides for XLR

CPU_XLR case added to mm/tlbex.c
CPU_XLR case added to mm/c-r4k.c for PINDEX attribute
Feature overrides for XLR cpu.

Signed-off-by: Jayachandran C <jayachandranc@netlogicmicro.com>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/2333/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: tlbex: Fix GCC 4.6.0 build error

CC arch/mips/mm/tlbex.o
arch/mips/mm/tlbex.c: In function 'build_r4000_tlb_refill_handler':
arch/mips/mm/tlbex.c:1155:22: error: variable 'vmalloc_mode' set but not used [-Werror=unused-but-set-variable]
arch/mips/mm/tlbex.c:1154:28: error: variable 'htlb_info' set but not used [-Werror=unused-but-set-variable]
cc1: all warnings being treated as errors

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Fix common misspellings

Fixes generated by 'codespell' and manually reviewed.

Signed-off-by: Lucas De Marchi <lucas.demarchi@profusion.mobi>

MIPS: Add an unreachable return statement to satisfy buggy GCCs.

It was reported that GCC-4.3.3 (with CodeSourcery extensions) fails
without this.

Reported-by: Jonas Gorski <jonas.gorski@gmail.com>
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/2010/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Optimize TLB handlers for Octeon CPUs

Octeon can use scratch registers in the TLB handlers. Octeon II can
use LDX instructions.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/1904/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Use BBIT instructions in TLB handlers

If the CPU supports BBIT0 and BBIT1, use them in TLB handlers as they
are more efficient than an AND followed by an branch and then
restoring the clobbered register.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/1873/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Use C0_KScratch (if present) to hold PGD pointer.

Decide at runtime to use either Context or KScratch to hold the PGD
pointer.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/1876/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Decouple BMIPS CPU support from bcm47xx/bcm63xx SoC code

BMIPS processor cores are used in 50+ different chipsets spread across
5+ product lines. In many cases the chipsets do not share the same
peripheral register layouts, the same register blocks, the same
interrupt controllers, the same memory maps, or much of anything else.

But, across radically different SoCs that share nothing more than the
same BMIPS CPU, a few things are still mostly constant:

SMP operations
Access to performance counters
DMA cache coherency quirks
Cache and memory bus configuration

So, it makes sense to treat each BMIPS processor type as a generic
"building block," rather than tying it to a specific SoC. This makes it
easier to support a large number of BMIPS-based chipsets without
unnecessary duplication of code, and provides the infrastructure needed
to support BMIPS-proprietary features.

Signed-off-by: Kevin Cernekee <cernekee@gmail.com>
Cc: mbizon@freebox.fr
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Tested-by: Florian Fainelli <ffainelli@freebox.fr>
Patchwork: https://patchwork.linux-mips.org/patch/1706/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org

MIPS: JZ4740: Add base support for Ingenic JZ4740 System-on-a-Chip

Adds a new cpu type for the JZ4740 to the Linux MIPS architecture code.
It also adds the iomem addresses for the different components found on
a JZ4740 SoC.

Signed-off-by: Lars-Peter Clausen <lars@metafoo.de>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/1464/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Check for accesses beyond the end of the PGD.

For some combinations of PAGE_SIZE and vmbits, it is possible to have
userspace access that are beyond what is covered by the PGD, but within
vmbits. Such an access would cause the TLB refill handler to load garbage
values for PMD and PTE potentially giving userspace access to parts of the
physical address space to which it is not entitled.

In the TLB refill hot path, we add a single dsrl instruction so we can
check if any bits outside of the range covered by the PGD are set. In
the vmalloc side we then separate the bad case from the normal vmalloc
case and call tlb_do_page_fault_0 if warranted. This slows us down a
bit, but has the benefit of yielding deterministic behavior.

[Ralf: Fixed build error for 32-bit kernels.]
[Ralf: Folded lmo commit c8c0e22b2aa3982852b44279638ef37f9aa31b7d into this
commit.]

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: http://patchwork.linux-mips.org/patch/1152/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

---

MIPS: Use uasm_i_ds{r,l}l_safe() instead of uasm_i_ds{r,l}l() in tlbex.c

This makes the code somewhat cleaner while reducing the risk of shift
amount overflows when various page table related options are changed.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: http://patchwork.linux-mips.org/patch/1154/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Sibyte: Fix M3 TLB exception handler workaround.

The M3 workaround needs to cmpare the region and VPN2 fields only.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Implement Read Inhibit/eXecute Inhibit

The SmartMIPS ASE specifies how Read Inhibit (RI) and eXecute Inhibit
(XI) bits in the page tables work. The upper two bits of EntryLo{0,1}
are RI and XI when the feature is enabled in the PageGrain register.
SmartMIPS only covers 32-bit systems. Cavium Octeon+ extends this to
64-bit systems by continuing to place the RI and XI bits in the top of
EntryLo even when EntryLo is 64-bits wide.

Because we need to carry the RI and XI bits in the PTE, the layout of
the PTE is changed. There is a two instruction overhead in the TLB
refill hot path to get the EntryLo bits into the proper position.
Also the TLB load exception has to probe the TLB to check if RI or XI
caused the exception.

Also of note is that the layout of the PTE bits is done at compile and
runtime rather than statically. In the 32-bit case this allows for
the same number of PFN bits as before the patch as the _PAGE_HUGE is
not supported in 32-bit kernels (we have _PAGE_NO_EXEC and
_PAGE_NO_READ instead of _PAGE_READ and _PAGE_HUGE).

The patch is tested on Cavium Octeon+, but should also work on 32-bit
systems with the Smart-MIPS ASE.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: http://patchwork.linux-mips.org/patch/952/
Patchwork: http://patchwork.linux-mips.org/patch/956/
Patchwork: http://patchwork.linux-mips.org/patch/962/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Use 64-bit stores to c0_entrylo on 64-bit kernels.

64-bit CPUs have 64-bit c0_entrylo{0,1} registers. We should use the
64-bit dmtc0 instruction to set them. This becomes important if we
want to set the RI and XI bits present in some processors.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
To: linux-mips@linux-mips.org
Patchwork: http://patchwork.linux-mips.org/patch/954/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Move arch/mips/mm/uasm.h to arch/mips/include/asm/uasm.h

Signed-off-by: Florian Fainelli <florian@openwrt.org>
To: linux-mips@linux-mips.org
To: David Daney <ddaney@caviumnetworks.com>
Patchwork: http://patchwork.linux-mips.org/patch/887/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Two-level pagetables for 64-bit kernels with 64KB pages.

For 64-bit kernels with 64KB pages and two level page tables, there are
42 bits worth of virtual address space This is larger than the 40 bits of
virtual address space obtained with the default 4KB Page size and three
levels, so there are no draw backs for using two level tables with this
configuration.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Cc: linux-mips@linux-mips.org
Patchwork: http://patchwork.linux-mips.org/patch/761/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Cleanup forgotten label_module_alloc in tlbex.c

commit c8af165342e83a4eb078c9607d29a7c399d30a53 (lmo) rsp.
e0cc87f59490d7d62a8ab2a76498dc8a2b64927a (kernel.org) left
label_module_alloc unused. Remove it now.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Cc: linux-mips@linux-mips.org
Patchwork: http://patchwork.linux-mips.org/patch/752/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Put PGD in C0_CONTEXT for 64-bit R2 processors.

Processors that support the mips64r2 ISA can in four instructions
convert a shifted PGD pointer stored in the upper bits of c0_context
into a usable pointer. By doing this we save a memory load and
associated potential cache miss in the TLB exception handlers.

Since the upper bits of c0_context were holding the CPU number, we
move this to the upper bits of c0_xcontext which doesn't have enough
bits to hold the PGD pointer, but has plenty for the CPU number.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: BCM63xx: Add Broadcom 63xx CPU definitions.

Todo: Nothing ever detects CPU_BCM6338 but the code tests for it anyway.

Signed-off-by: Maxime Bizon <mbizon@freebox.fr>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Shrink the size of tlb handler

By combining swapper_pg_dir and module_pg_dir, several if conditions
can be eliminated from the tlb exception handler. The reason they
can be combined is that, the effective virtual address of vmalloc
returned is at the bottom, and of module_alloc returned is at the
top. It also fixes the bug in vmalloc(), which happens when its
return address is not covered by the first pgd.

Signed-off-by: Wu Fei <at.wufei@gmail.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Build fix - include <linux/smp.h> into all smp_processor_id() users.

Some of the were relying into smp.h being dragged in by another header
which of course is fragile. <asm/cpu-info.h> uses smp_processor_id()
only in macros and including smp.h there leads to an include loop, so
don't change cpu-info.h.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: TLB support for hugetlbfs.

The TLB handlers need to check for huge pages and give them special
handling. Huge pages consist of two contiguous sub-pages of physical
memory.

* Loading entrylo0 and entrylo1 need to be handled specially.

* The page mask must be set for huge pages and then restored after
writing the TLB entries.

* The PTE for huge pages resides in the PMD, we halt traversal of the
tables there.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Remove unused parameters from iPTE_LW.

The l parameter to iPTE_LW() is unused. Remove it and from some of its
callers as well.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Remove dead case label.

CPU_CAVIUM_OCTEON is mips_r2 which is handled before the switch. This
label in the switch statement is dead code, so we remove it.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Reviewed by: David VomLehn <dvomlehn@cisco.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Allow R2 CPUs to turn off generation of 'ehb' instructions.

Some CPUs do not need ehb instructions after writing CP0 registers.
By allowing ehb generation to be overridden in
cpu-feature-overrides.h, we can save a few instructions in the TLB
handler hot paths.

Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Fold the TLB refill at the vmalloc path if possible.

Try to fold the 64-bit TLB refill handler opportunistically at the
beginning of the vmalloc path so as to avoid splitting execution flow in
half and wasting cycles for a branch required at that point then. Resort
to doing the split if either of the newly created parts would not fit into
its designated slot.

Original-patch-by: Maciej W. Rozycki <macro@linux-mips.org>
Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Replace some magic numbers with symbolic values in tlbex.c

The logic used to split the r4000 refill handler is liberally
sprinkled with magic numbers. We attempt to explain what they are and
normalize them against a new symbolic value (MIPS64_REFILL_INSNS).

CC: David VomLehn <dvomlehn@cisco.com>
Reviewed-by: Paul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Alchemy: MIPS hazard workarounds are not required.

The Alchemy manuals state:

"All pipeline hazards and dependencies are enforced by hardware interlocks
so that any sequence of instructions is guaranteed to execute correctly.
Therefore, it is not necessary to pad legacy MIPS hazards (such as
load delay slots and coprocessor accesses) with NOPs."

Run-tested on Au12x0, without any ill effects.

Signed-off-by: Manuel Lauss <mano@roarinelk.homelinux.net>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Alchemy: unify CPU model constants.

This patch removes the various CPU_AU1??? model constants in favor of
a single CPU_ALCHEMY one.

All currently existing Alchemy models are identical in terms of cpu
core and cache size/organization. The parts of the mips kernel which
need to know the exact CPU revision extract it from the c0_prid register
already; and finally nothing else in-tree depends on those any more.

Should a new variant with slightly different "company options" and/or
"processor revision" bits in c0_prid appear, it will be supported
immediately (minus an exact model string in cpuinfo).

Signed-off-by: Manuel Lauss <mano@roarinelk.homelinux.net>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: NEC VR5500 processor support fixup

Current VR5500 processor support lacks of some functions which are
expected to be configured/synthesized on arch initialization.

Here're some VR5500A spec notes:

* All execution hazards are handled in hardware.

* Once VR5500A stops the operation of the pipeline by WAIT instruction,
it could return from the standby mode only when either a reset, NMI
request, or all enabled interrupts is/are detected. In other words,
if interrupts are disabled by Status.IE=0, it keeps in standby mode
even when interrupts are internally asserted.

Notes on WAIT: The operation of the processor is undefined if WAIT
insn is in the branch delay slot. The operation is also undefined
if WAIT insn is executed when Status.EXL and Status.ERL are set to 1.

* VR5500A core only implements the Load prefetch.

With these changes, it boots fine.

Signed-off-by: Shinya Kuribayashi <shinya.kuribayashi@necel.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

MIPS: Add Cavium OCTEON slot into proper tlb category.

Expand the case statement for build_tlb_write_entry so that it does
the right thing on Cavium CPU variants.

Signed-off-by: Tomaso Paoletti <tpaoletti@caviumnetworks.com>
Signed-off-by: Paul Gortmaker <Paul.Gortmaker@windriver.com>
Signed-off-by: David Daney <ddaney@caviumnetworks.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] Fix WARNING: at kernel/smp.c:290

trap_init issues flush_icache_range(), which uses ipi functions to
get icache flushing done on all cpus. But this is done before interrupts
are enabled and caused WARN_ON messages. This changeset introduces
a new local_flush_icache_range() and uses it before interrupts (and
additional CPUs) are enabled to avoid this problem.

Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] R4700: Fix build_tlb_probe_entry

Treat R4700 like R4600 in build_tlb_probe_entry. Without this fix kernel
will lock up.

Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] Add missing 4KEC TLB refill handler

Early 4KEc were MIPS32r1 and therefore need some love to get a TLB
refill handler.

Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] Fix loads of section missmatches

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] Split the micro-assembler from tlbex.c.

This patch moves the micro-assembler in a separate implementation, as
it is useful for further run-time optimizations. The only change in
behaviour is cutting down printk noise at kernel startup time.

Checkpatch complains about macro parameters which aren't protected by
parentheses. I believe this is a flaw in checkpatch, the paste operator
used in those macros won't work with parenthesised parameters.

Signed-off-by: Thiemo Seufer <ths@networkno.de>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] Alchemy: Au1210/Au1250 CPU support

This patch adds IDs for new Au1200 variants: Au1210 and Au1250.
They are essentially identical to the Au1200 except for the Au1210
which has a different SoC-ID in the PRId register [bits 31:24].
The Au1250 is a "Au1200 V0.2".

Signed-off-by: Manuel Lauss <mano@roarinelk.homelinux.net>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] tlbex.c: cleanup debug code

Signed-off-by: Franck Bui-Huu <fbuihuu@gmail.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] tlbex.c: use __cacheline_aligned instead of __tlb_handler_align

Signed-off-by: Franck Bui-Huu <fbuihuu@gmail.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] tlbex.c: cleanup include files

Signed-off-by: Franck Bui-Huu <fbuihuu@gmail.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] tlbex.c: Cleanup __init usages.

Signed-off-by: Franck Bui-Huu <fbuihuu@gmail.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] R4000/R4400 daddiu erratum workaround

This complements the generic R4000/R4400 errata workaround code and adds
bits for the daddiu problem. In most places it just modifies handwritten
assembly code so that the assembler is allowed to use a temporary register
as daddiu may now be treated as a macro that expands to a sequence of li
and daddu. It is the AT register or, where AT is unavailable or used
explicitly for another purpose, an explicitly-named register is selected,
using the .set at=<reg> feature added recently to gas. This feature is
only used if CONFIG_CPU_DADDI_WORKAROUNDS has been set, so if the
workaround remains disabled, the required version of binutils stays
unchanged.

Similarly, daddiu instructions put in branch delay slots in noreorder
fragments are now taken out of them and the assembler is allowed to
reorder them itself as possible (which it does making the whole idea of
scheduling them into delay slots manually questionable).

Also in the very few places where such a simple conversion was not
possible, a handcoded longer sequence is implemented.

Other than that there are changes to code responsible for building the
TLB fault and page clear/copy handlers to avoid daddiu as appropriate.
These are only effective if the erratum is verified to be present at the
run time.

Finally there is a trivial update to __delay(), because it uses daddiu in
a branch delay slot.

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] tlbex: Cleanup handling of R2 hazards in TLB handlers.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] Revert "[MIPS] tlbex.c: Cleanup __init usage."

This reverts commit aaf76a3245c02faba51c96b9a340c14d6bb0dcc0.

As requested by ranck Bui-Huu <fbuihuu@gmail.com>.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] tlbex.c: Cleanup __init usage.

Signed-off-by: Franck Bui-Huu <fbuihuu@gmail.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] checkfiles: Fix "need space after that ','" errors.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] Allow hardwiring of the CPU type to a single type for optimization.

This saves a few k on systems which only ever ship with a single CPU type.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] tlbex: Size optimize code by declaring a few functions inline.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] Add support for BCM47XX CPUs.

Note that the BCM4710 does not support the wait instruction, this
is not a mistake in the code.

It originally comes from the OpenWrt patches.

Cc: Michael Buesch <mb@bu3sch.de>
Cc: Felix Fietkau <nbd@openwrt.org>
Cc: Florian Schirmer <jolt@tuxbox.org>
Signed-off-by: Aurelien Jarno <aurelien@aurel32.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] Workaround for 4Kc machine check exception

Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] TLB: Fix instruction bitmasks

Signed-Off-By: Thiemo Seufer <ths@networkno.de>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] TLB: Fix instruction bitmasks

Signed-Off-By: Thiemo Seufer <ths@networkno.de>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] define Hit_Invalidate_I to Index_Invalidate_I for loongson2

Signed-off-by: Fuxin Zhang <zhangfx@lemote.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] tlbex: use __maybe_unused

Replace function instances of __attribute__((unused)) with
__maybe_unused.

Acked-by: Ralf Baechle <ralf@linux-mips.org>
Signed-off-by: David Rientjes <rientjes@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] Load modules to CKSEG0 if CONFIG_BUILD_ELF64=n

This is a patch to load 64-bit modules to CKSEG0 so that can be
compiled with -msym32 option. This makes each module ~10% smaller.

* introduce MODULE_START and MODULE_END
* custom module_alloc()
* PGD for modules
* change XTLB refill handler synthesizer
* enable -msym32 for modules again
(revert ca78b1a5c6a6e70e052d3ea253828e49b5d07c8a)

New XTLB refill handler looks like this:

80000080 dmfc0 k0,C0_BADVADDR
80000084 bltz k0,800000e4 # goto l_module_alloc
80000088 lui k1,0x8046 # %high(pgd_current)
8000008c ld k1,24600(k1) # %low(pgd_current)
80000090 dsrl k0,k0,0x1b # l_vmalloc_done:
80000094 andi k0,k0,0x1ff8
80000098 daddu k1,k1,k0
8000009c dmfc0 k0,C0_BADVADDR
800000a0 ld k1,0(k1)
800000a4 dsrl k0,k0,0x12
800000a8 andi k0,k0,0xff8
800000ac daddu k1,k1,k0
800000b0 dmfc0 k0,C0_XCONTEXT
800000b4 ld k1,0(k1)
800000b8 andi k0,k0,0xff0
800000bc daddu k1,k1,k0
800000c0 ld k0,0(k1)
800000c4 ld k1,8(k1)
800000c8 dsrl k0,k0,0x6
800000cc mtc0 k0,C0_ENTRYLO0
800000d0 dsrl k1,k1,0x6
800000d4 mtc0 k1,C0_ENTRYL01
800000d8 nop
800000dc tlbwr
800000e0 eret
800000e4 dsll k1,k0,0x2 # l_module_alloc:
800000e8 bgez k1,80000008 # goto l_vmalloc
800000ec lui k1,0xc000
800000f0 dsubu k0,k0,k1
800000f4 lui k1,0x8046 # %high(module_pg_dir)
800000f8 beq zero,zero,80000000
800000fc nop
80000000 beq zero,zero,80000090 # goto l_vmalloc_done
80000004 daddiu k1,k1,0x4000
80000008 dsll32 k1,k1,0x0 # l_vmalloc:
8000000c dsubu k0,k0,k1
80000010 beq zero,zero,80000090 # goto l_vmalloc_done
80000014 lui k1,0x8046 # %high(swapper_pg_dir)

Signed-off-by: Atsushi Nemoto <anemo@mba.ocn.ne.jp>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] 16K & 64K page size fixes

Derived from Peter Watkins <treestem@gmail.com>'s work.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Attack of "the the"s in arch

The patch below corrects multiple occurances of "the the"
typos across several files, both in source comments and KConfig files.
There is no actual code changed, only text. Note this only affects the /arch
directory, and I believe I could find many more elsewhere. :)

Signed-off-by: Adrian Bunk <bunk@stusta.de>

[MIPS] Print out TLB handler assembly for debugging.

Small update, using pr_debug and pr_info.

Signed-off-by: Thiemo Seufer <ths@networkno.de>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Remove obsolete #include <linux/config.h>

Signed-off-by: Jörn Engel <joern@wohnheim.fh-wedel.de>
Signed-off-by: Adrian Bunk <bunk@stusta.de>

[MIPS] Treat R14000 like R10000.

Signed-off-by: Joshua Kinard <kumba@gentoo.org>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] Fix detection and handling of the 74K processor.

Nothing exciting; Linux just didn't know it yet so this is most adding
a value to a case statement.

Signed-off-by: Chris Dearman <chris@mips.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] MT: Improved multithreading support.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] Fix vectored interrupt support in TLB exception handler generator.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] Remove CONFIG_BUILD_ELF64.

This option is no longer usable with supported compilers. It will be
replaced by usage of -msym32 in a separate patch.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[MIPS] Scatter a bunch of __init over tlbex.c.

Found by make buildcheck.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Add support for SB1A CPU.

Signed-Off-By: Andy Isaacson <adi@broadcom.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

R4600 v2.0 needs a nop before tlbp.

Signed-off-by: Thiemo Seufer <ths@networkno.de>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Handle mtc0 - tlb write hazard for VR5432.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Avoid SMP cacheflushes. This is a minor optimization of startup but
will also avoid smp_call_function from doing stupid things when called
from a CPU that is not yet marked online.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Philips PNX8550 support: MIPS32-like core with 2 Trimedias on it.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Detect the 34K.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Date: Fri Jul 8 20:10:17 2005 +0000

Those literals are long.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Avoid tlbw* hazards for the R4600/R4700/R5000.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Fix the diagnostic dump for the XTLB refill handler.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Fix a diagnostic message.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Optimize R3k TLB Load/Store/Modified handlers, by scheduling
delay slots properly and avoiding an unnecessary jump to a jump.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Fill R3k load delay slots properly.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Only dump instructions actually emitted.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Handle _PAGE_DIRTY correctly for CONFIG_64BIT_PHYS_ADDR on 32bit CPUs.

Signed-off-by: Thiemo Seufer <ths@networkno.de>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Fix 64bit SMP TLB handler and stack frame handling, optimize 32bit SMP
TLB handlers a bit, match definitions in pgtable-{32,64}.h better.

Signed-off-by: Thiemo Seufer <ths@networkno.de>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

R4300 delay slot.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

Base Au1200 2.6 support.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>

[PATCH] mips: fix build warnings

This patch has fixed the following warnings.

arch/mips/kernel/genex.S:250:5: warning: "CONFIG_64BIT" is not defined
arch/mips/math-emu/cp1emu.c:1128:5: warning: "__mips64" is not defined
arch/mips/math-emu/cp1emu.c:1206:5: warning: "__mips64" is not defined
arch/mips/math-emu/cp1emu.c:1270:5: warning: "__mips64" is not defined
arch/mips/math-emu/cp1emu.c:323:5: warning: "__mips64" is not defined
arch/mips/math-emu/cp1emu.c:808:5: warning: "__mips64" is not defined
arch/mips/math-emu/cp1emu.c:953:5: warning: "__mips64" is not defined
arch/mips/mm/tlbex.c:519:5: warning: "CONFIG_64BIT" is not defined
include/asm/reg.h:73:5: warning: "CONFIG_64BIT" is not defined

Signed-off-by: Yoichi Yuasa <yuasa@hh.iij4u.or.jp>
Cc: Ralf Baechle <ralf@linux-mips.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>

[PATCH] mips: clean up 32/64-bit configuration

Start cleaning 32-bit vs. 64-bit configuration.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>

Linux-2.6.12-rc2

Initial git repository build. I'm not bothering with the full history,
even though we have it. We can create a separate "historical" git
archive of that later if we want to, and in the meantime it's about
3.2GB when imported into git - space that would just make the early
git days unnecessarily complicated, when we don't have a lot of good
infrastructure for it.

Let it rip!