bpf, riscv64/cfi: Support kCFI + BPF on riscv64

The riscv BPF JIT doesn't emit proper kCFI prologues for BPF programs
and struct_ops trampolines when CONFIG_CFI_CLANG is enabled.

This causes CFI failures when calling BPF programs and can even crash
the kernel due to invalid memory accesses.

Example crash:

root@rv-selftester:~/bpf# ./test_progs -a dummy_st_ops

Unable to handle kernel paging request at virtual address ffffffff78204ffc
Oops [#1]
Modules linked in: bpf_testmod(OE) [....]
CPU: 3 PID: 356 Comm: test_progs Tainted: P OE 6.8.0-rc1 #1
Hardware name: riscv-virtio,qemu (DT)
epc : bpf_struct_ops_test_run+0x28c/0x5fc
ra : bpf_struct_ops_test_run+0x26c/0x5fc
epc : ffffffff82958010 ra : ffffffff82957ff0 sp : ff200000007abc80
gp : ffffffff868d6218 tp : ff6000008d87b840 t0 : 000000000000000f
t1 : 0000000000000000 t2 : 000000002005793e s0 : ff200000007abcf0
s1 : ff6000008a90fee0 a0 : 0000000000000000 a1 : 0000000000000000
a2 : 0000000000000000 a3 : 0000000000000000 a4 : 0000000000000000
a5 : ffffffff868dba26 a6 : 0000000000000001 a7 : 0000000052464e43
s2 : 00007ffffc0a95f0 s3 : ff6000008a90fe80 s4 : ff60000084c24c00
s5 : ffffffff78205000 s6 : ff60000088750648 s7 : ff20000000035008
s8 : fffffffffffffff4 s9 : ffffffff86200610 s10: 0000000000000000
s11: 0000000000000000 t3 : ffffffff8483dc30 t4 : ffffffff8483dc10
t5 : ffffffff8483dbf0 t6 : ffffffff8483dbd0
status: 0000000200000120 badaddr: ffffffff78204ffc cause: 000000000000000d
[<ffffffff82958010>] bpf_struct_ops_test_run+0x28c/0x5fc
[<ffffffff805083ee>] bpf_prog_test_run+0x170/0x548
[<ffffffff805029c8>] __sys_bpf+0x2d2/0x378
[<ffffffff804ff570>] __riscv_sys_bpf+0x5c/0x120
[<ffffffff8000e8fe>] syscall_handler+0x62/0xe4
[<ffffffff83362df6>] do_trap_ecall_u+0xc6/0x27c
[<ffffffff833822c4>] ret_from_exception+0x0/0x64
Code: b603 0109 b683 0189 b703 0209 8493 0609 157d 8d65 (a303) ffca
---[ end trace 0000000000000000 ]---
Kernel panic - not syncing: Fatal exception
SMP: stopping secondary CPUs

Implement proper kCFI prologues for the BPF programs and callbacks and
drop __nocfi for riscv64. Fix the trampoline generation code to emit kCFI
prologue when a struct_ops trampoline is being prepared.

Signed-off-by: Puranjay Mohan <puranjay12@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Björn Töpel <bjorn@kernel.org>
Link: https://lore.kernel.org/bpf/20240303170207.82201-2-puranjay12@gmail.com

cfi: Flip headers

Normal include order is that linux/foo.h should include asm/foo.h, CFI has it
the wrong way around.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Sami Tolvanen <samitolvanen@google.com>
Link: https://lore.kernel.org/r/20231215092707.231038174@infradead.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>

riscv: Add CFI error handling

With CONFIG_CFI_CLANG, the compiler injects a type preamble immediately
before each function and a check to validate the target function type
before indirect calls:

; type preamble
.word <id>
function:
...
; indirect call check
lw t1, -4(a0)
lui t2, <hi20>
addiw t2, t2, <lo12>
beq t1, t2, .Ltmp0
ebreak
.Ltmp0:
jarl a0

Implement error handling code for the ebreak traps emitted for the
checks. This produces the following oops on a CFI failure (generated
using lkdtm):

[ 21.177245] CFI failure at lkdtm_indirect_call+0x22/0x32 [lkdtm]
(target: lkdtm_increment_int+0x0/0x18 [lkdtm]; expected type: 0x3ad55aca)
[ 21.178483] Kernel BUG [#1]
[ 21.178671] Modules linked in: lkdtm
[ 21.179037] CPU: 1 PID: 104 Comm: sh Not tainted
6.3.0-rc6-00037-g37d5ec6297ab #1
[ 21.179511] Hardware name: riscv-virtio,qemu (DT)
[ 21.179818] epc : lkdtm_indirect_call+0x22/0x32 [lkdtm]
[ 21.180106] ra : lkdtm_CFI_FORWARD_PROTO+0x48/0x7c [lkdtm]
[ 21.180426] epc : ffffffff01387092 ra : ffffffff01386f14 sp : ff20000000453cf0
[ 21.180792] gp : ffffffff81308c38 tp : ff6000000243f080 t0 : ff20000000453b78
[ 21.181157] t1 : 000000003ad55aca t2 : 000000007e0c52a5 s0 : ff20000000453d00
[ 21.181506] s1 : 0000000000000001 a0 : ffffffff0138d170 a1 : ffffffff013870bc
[ 21.181819] a2 : b5fea48dd89aa700 a3 : 0000000000000001 a4 : 0000000000000fff
[ 21.182169] a5 : 0000000000000004 a6 : 00000000000000b7 a7 : 0000000000000000
[ 21.182591] s2 : ff20000000453e78 s3 : ffffffffffffffea s4 : 0000000000000012
[ 21.183001] s5 : ff600000023c7000 s6 : 0000000000000006 s7 : ffffffff013882a0
[ 21.183653] s8 : 0000000000000008 s9 : 0000000000000002 s10: ffffffff0138d878
[ 21.184245] s11: ffffffff0138d878 t3 : 0000000000000003 t4 : 0000000000000000
[ 21.184591] t5 : ffffffff8133df08 t6 : ffffffff8133df07
[ 21.184858] status: 0000000000000120 badaddr: 0000000000000000
cause: 0000000000000003
[ 21.185415] [<ffffffff01387092>] lkdtm_indirect_call+0x22/0x32 [lkdtm]
[ 21.185772] [<ffffffff01386f14>] lkdtm_CFI_FORWARD_PROTO+0x48/0x7c [lkdtm]
[ 21.186093] [<ffffffff01383552>] lkdtm_do_action+0x22/0x34 [lkdtm]
[ 21.186445] [<ffffffff0138350c>] direct_entry+0x128/0x13a [lkdtm]
[ 21.186817] [<ffffffff8033ed8c>] full_proxy_write+0x58/0xb2
[ 21.187352] [<ffffffff801d4fe8>] vfs_write+0x14c/0x33a
[ 21.187644] [<ffffffff801d5328>] ksys_write+0x64/0xd4
[ 21.187832] [<ffffffff801d53a6>] sys_write+0xe/0x1a
[ 21.188171] [<ffffffff80003996>] ret_from_syscall+0x0/0x2
[ 21.188595] Code: 0513 0f65 a303 ffc5 53b7 7e0c 839b 2a53 0363 0073 (9002) 9582
[ 21.189178] ---[ end trace 0000000000000000 ]---
[ 21.189590] Kernel panic - not syncing: Fatal exception

Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Conor Dooley <conor.dooley@microchip.com> # ISA bits
Tested-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Link: https://lore.kernel.org/r/20230710183544.999540-12-samitolvanen@google.com
Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>