LoongArch: BPF: Prevent out-of-bounds memory access

The test_tag test triggers an unhandled page fault:

# ./test_tag
[ 130.640218] CPU 0 Unable to handle kernel paging request at virtual address ffff80001b898004, era == 9000000003137f7c, ra == 9000000003139e70
[ 130.640501] Oops[#3]:
[ 130.640553] CPU: 0 PID: 1326 Comm: test_tag Tainted: G D O 6.7.0-rc4-loong-devel-gb62ab1a397cf #47 61985c1d94084daa2432f771daa45b56b10d8d2a
[ 130.640764] Hardware name: QEMU QEMU Virtual Machine, BIOS unknown 2/2/2022
[ 130.640874] pc 9000000003137f7c ra 9000000003139e70 tp 9000000104cb4000 sp 9000000104cb7a40
[ 130.641001] a0 ffff80001b894000 a1 ffff80001b897ff8 a2 000000006ba210be a3 0000000000000000
[ 130.641128] a4 000000006ba210be a5 00000000000000f1 a6 00000000000000b3 a7 0000000000000000
[ 130.641256] t0 0000000000000000 t1 00000000000007f6 t2 0000000000000000 t3 9000000004091b70
[ 130.641387] t4 000000006ba210be t5 0000000000000004 t6 fffffffffffffff0 t7 90000000040913e0
[ 130.641512] t8 0000000000000005 u0 0000000000000dc0 s9 0000000000000009 s0 9000000104cb7ae0
[ 130.641641] s1 00000000000007f6 s2 0000000000000009 s3 0000000000000095 s4 0000000000000000
[ 130.641771] s5 ffff80001b894000 s6 ffff80001b897fb0 s7 9000000004090c50 s8 0000000000000000
[ 130.641900] ra: 9000000003139e70 build_body+0x1fcc/0x4988
[ 130.642007] ERA: 9000000003137f7c build_body+0xd8/0x4988
[ 130.642112] CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
[ 130.642261] PRMD: 00000004 (PPLV0 +PIE -PWE)
[ 130.642353] EUEN: 00000003 (+FPE +SXE -ASXE -BTE)
[ 130.642458] ECFG: 00071c1c (LIE=2-4,10-12 VS=7)
[ 130.642554] ESTAT: 00010000 [PIL] (IS= ECode=1 EsubCode=0)
[ 130.642658] BADV: ffff80001b898004
[ 130.642719] PRID: 0014c010 (Loongson-64bit, Loongson-3A5000)
[ 130.642815] Modules linked in: [last unloaded: bpf_testmod(O)]
[ 130.642924] Process test_tag (pid: 1326, threadinfo=00000000f7f4015f, task=000000006499f9fd)
[ 130.643062] Stack : 0000000000000000 9000000003380724 0000000000000000 0000000104cb7be8
[ 130.643213] 0000000000000000 25af8d9b6e600558 9000000106250ea0 9000000104cb7ae0
[ 130.643378] 0000000000000000 0000000000000000 9000000104cb7be8 90000000049f6000
[ 130.643538] 0000000000000090 9000000106250ea0 ffff80001b894000 ffff80001b894000
[ 130.643685] 00007ffffb917790 900000000313ca94 0000000000000000 0000000000000000
[ 130.643831] ffff80001b894000 0000000000000ff7 0000000000000000 9000000100468000
[ 130.643983] 0000000000000000 0000000000000000 0000000000000040 25af8d9b6e600558
[ 130.644131] 0000000000000bb7 ffff80001b894048 0000000000000000 0000000000000000
[ 130.644276] 9000000104cb7be8 90000000049f6000 0000000000000090 9000000104cb7bdc
[ 130.644423] ffff80001b894000 0000000000000000 00007ffffb917790 90000000032acfb0
[ 130.644572] ...
[ 130.644629] Call Trace:
[ 130.644641] [<9000000003137f7c>] build_body+0xd8/0x4988
[ 130.644785] [<900000000313ca94>] bpf_int_jit_compile+0x228/0x4ec
[ 130.644891] [<90000000032acfb0>] bpf_prog_select_runtime+0x158/0x1b0
[ 130.645003] [<90000000032b3504>] bpf_prog_load+0x760/0xb44
[ 130.645089] [<90000000032b6744>] __sys_bpf+0xbb8/0x2588
[ 130.645175] [<90000000032b8388>] sys_bpf+0x20/0x2c
[ 130.645259] [<9000000003f6ab38>] do_syscall+0x7c/0x94
[ 130.645369] [<9000000003121c5c>] handle_syscall+0xbc/0x158
[ 130.645507]
[ 130.645539] Code: 380839f6 380831f9 28412bae <24000ca6> 004081ad 0014cb50 004083e8 02bff34c 58008e91
[ 130.645729]
[ 130.646418] ---[ end trace 0000000000000000 ]---

On my machine, which has CONFIG_PAGE_SIZE_16KB=y, the test failed at
loading a BPF prog with 2039 instructions:

prog = (struct bpf_prog *)ffff80001b894000
insn = (struct bpf_insn *)(prog->insnsi)ffff80001b894048
insn + 2039 = (struct bpf_insn *)ffff80001b898000 <- end of the page

In the build_insn() function, we are trying to access next instruction
unconditionally, i.e. `(insn + 1)->imm`. The address lies in the next
page and can be not owned by the current process, thus an page fault is
inevitable and then segfault.

So, let's access next instruction only under `dst = imm64` context.

With this fix, we have:

# ./test_tag
test_tag: OK (40945 tests)

Fixes: bbfddb904df6f82 ("LoongArch: BPF: Avoid declare variables in switch-case")
Tested-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

LoongArch: BPF: Support 64-bit pointers to kfuncs

Like commit 1cf3bfc60f9836f ("bpf: Support 64-bit pointers to kfuncs")
for s390x, add support for 64-bit pointers to kfuncs for LoongArch.
Since the infrastructure is already implemented in BPF core, the only
thing need to be done is to override bpf_jit_supports_far_kfunc_call().

Before this change, several test_verifier tests failed:

# ./test_verifier | grep # | grep FAIL
#119/p calls: invalid kfunc call: ptr_to_mem to struct with non-scalar FAIL
#120/p calls: invalid kfunc call: ptr_to_mem to struct with nesting depth > 4 FAIL
#121/p calls: invalid kfunc call: ptr_to_mem to struct with FAM FAIL
#122/p calls: invalid kfunc call: reg->type != PTR_TO_CTX FAIL
#123/p calls: invalid kfunc call: void * not allowed in func proto without mem size arg FAIL
#124/p calls: trigger reg2btf_ids[reg->type] for reg->type > __BPF_REG_TYPE_MAX FAIL
#125/p calls: invalid kfunc call: reg->off must be zero when passed to release kfunc FAIL
#126/p calls: invalid kfunc call: don't match first member type when passed to release kfunc FAIL
#127/p calls: invalid kfunc call: PTR_TO_BTF_ID with negative offset FAIL
#128/p calls: invalid kfunc call: PTR_TO_BTF_ID with variable offset FAIL
#129/p calls: invalid kfunc call: referenced arg needs refcounted PTR_TO_BTF_ID FAIL
#130/p calls: valid kfunc call: referenced arg needs refcounted PTR_TO_BTF_ID FAIL
#486/p map_kptr: ref: reference state created and released on xchg FAIL

This is because the kfuncs in the loaded module are far away from
__bpf_call_base:

ffff800002009440 t bpf_kfunc_call_test_fail1 [bpf_testmod]
9000000002e128d8 T __bpf_call_base

The offset relative to __bpf_call_base does NOT fit in s32, which breaks
the assumption in BPF core. Enable bpf_jit_supports_far_kfunc_call() lifts
this limit.

Note that to reproduce the above result, tools/testing/selftests/bpf/config
should be applied, and run the test with JIT enabled, unpriv BPF enabled.

With this change, the test_verifier tests now all passed:

# ./test_verifier
...
Summary: 777 PASSED, 0 SKIPPED, 0 FAILED

Tested-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

LoongArch: BPF: Fix unconditional bswap instructions

We can see that "bswap32: Takes an unsigned 32-bit number in either big-
or little-endian format and returns the equivalent number with the same
bit width but opposite endianness" in BPF Instruction Set Specification,
so it should clear the upper 32 bits in "case 32:" for both BPF_ALU and
BPF_ALU64.

[root@linux fedora]# echo 1 > /proc/sys/net/core/bpf_jit_enable
[root@linux fedora]# modprobe test_bpf

Before:
test_bpf: #313 BSWAP 32: 0x0123456789abcdef -> 0xefcdab89 jited:1 ret 1460850314 != -271733879 (0x5712ce8a != 0xefcdab89)FAIL (1 times)
test_bpf: #317 BSWAP 32: 0xfedcba9876543210 -> 0x10325476 jited:1 ret -1460850316 != 271733878 (0xa8ed3174 != 0x10325476)FAIL (1 times)

After:
test_bpf: #313 BSWAP 32: 0x0123456789abcdef -> 0xefcdab89 jited:1 4 PASS
test_bpf: #317 BSWAP 32: 0xfedcba9876543210 -> 0x10325476 jited:1 4 PASS

Fixes: 4ebf9216e7df ("LoongArch: BPF: Support unconditional bswap instructions")
Acked-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

LoongArch: BPF: Fix sign-extension mov instructions

We can see that "Short form of movsx, dst_reg = (s8,s16,s32)src_reg" in
include/linux/filter.h, additionally, for BPF_ALU64 the value of the
destination register is unchanged whereas for BPF_ALU the upper 32 bits
of the destination register are zeroed, so it should clear the upper 32
bits for BPF_ALU.

[root@linux fedora]# echo 1 > /proc/sys/net/core/bpf_jit_enable
[root@linux fedora]# modprobe test_bpf

Before:
test_bpf: #81 ALU_MOVSX | BPF_B jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)
test_bpf: #82 ALU_MOVSX | BPF_H jited:1 ret 2 != 1 (0x2 != 0x1)FAIL (1 times)

After:
test_bpf: #81 ALU_MOVSX | BPF_B jited:1 6 PASS
test_bpf: #82 ALU_MOVSX | BPF_H jited:1 6 PASS

By the way, the bpf selftest case "./test_progs -t verifier_movsx" can
also be fixed with this patch.

Fixes: f48012f16150 ("LoongArch: BPF: Support sign-extension mov instructions")
Acked-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

LoongArch: BPF: Don't sign extend function return value

The `cls_redirect` test triggers a kernel panic like:

# ./test_progs -t cls_redirect
Can't find bpf_testmod.ko kernel module: -2
WARNING! Selftests relying on bpf_testmod.ko will be skipped.
[ 30.938489] CPU 3 Unable to handle kernel paging request at virtual address fffffffffd814de0, era == ffff800002009fb8, ra == ffff800002009f9c
[ 30.939331] Oops[#1]:
[ 30.939513] CPU: 3 PID: 1260 Comm: test_progs Not tainted 6.7.0-rc2-loong-devel-g2f56bb0d2327 #35 a896aca3f4164f09cc346f89f2e09832e07be5f6
[ 30.939732] Hardware name: QEMU QEMU Virtual Machine, BIOS unknown 2/2/2022
[ 30.939901] pc ffff800002009fb8 ra ffff800002009f9c tp 9000000104da4000 sp 9000000104da7ab0
[ 30.940038] a0 fffffffffd814de0 a1 9000000104da7a68 a2 0000000000000000 a3 9000000104da7c10
[ 30.940183] a4 9000000104da7c14 a5 0000000000000002 a6 0000000000000021 a7 00005555904d7f90
[ 30.940321] t0 0000000000000110 t1 0000000000000000 t2 fffffffffd814de0 t3 0004c4b400000000
[ 30.940456] t4 ffffffffffffffff t5 00000000c3f63600 t6 0000000000000000 t7 0000000000000000
[ 30.940590] t8 000000000006d803 u0 0000000000000020 s9 9000000104da7b10 s0 900000010504c200
[ 30.940727] s1 fffffffffd814de0 s2 900000010504c200 s3 9000000104da7c10 s4 9000000104da7ad0
[ 30.940866] s5 0000000000000000 s6 90000000030e65bc s7 9000000104da7b44 s8 90000000044f6fc0
[ 30.941015] ra: ffff800002009f9c bpf_prog_846803e5ae81417f_cls_redirect+0xa0/0x590
[ 30.941535] ERA: ffff800002009fb8 bpf_prog_846803e5ae81417f_cls_redirect+0xbc/0x590
[ 30.941696] CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
[ 30.942224] PRMD: 00000004 (PPLV0 +PIE -PWE)
[ 30.942330] EUEN: 00000003 (+FPE +SXE -ASXE -BTE)
[ 30.942453] ECFG: 00071c1c (LIE=2-4,10-12 VS=7)
[ 30.942612] ESTAT: 00010000 [PIL] (IS= ECode=1 EsubCode=0)
[ 30.942764] BADV: fffffffffd814de0
[ 30.942854] PRID: 0014c010 (Loongson-64bit, Loongson-3A5000)
[ 30.942974] Modules linked in:
[ 30.943078] Process test_progs (pid: 1260, threadinfo=00000000ce303226, task=000000007d10bb76)
[ 30.943306] Stack : 900000010a064000 90000000044f6fc0 9000000104da7b48 0000000000000000
[ 30.943495] 0000000000000000 9000000104da7c14 9000000104da7c10 900000010504c200
[ 30.943626] 0000000000000001 ffff80001b88c000 9000000104da7b70 90000000030e6668
[ 30.943785] 0000000000000000 9000000104da7b58 ffff80001b88c048 9000000003d05000
[ 30.943936] 900000000303ac88 0000000000000000 0000000000000000 9000000104da7b70
[ 30.944091] 0000000000000000 0000000000000001 0000000731eeab00 0000000000000000
[ 30.944245] ffff80001b88c000 0000000000000000 0000000000000000 54b99959429f83b8
[ 30.944402] ffff80001b88c000 90000000044f6fc0 9000000101d70000 ffff80001b88c000
[ 30.944538] 000000000000005a 900000010504c200 900000010a064000 900000010a067000
[ 30.944697] 9000000104da7d88 0000000000000000 9000000003d05000 90000000030e794c
[ 30.944852] ...
[ 30.944924] Call Trace:
[ 30.945120] [<ffff800002009fb8>] bpf_prog_846803e5ae81417f_cls_redirect+0xbc/0x590
[ 30.945650] [<90000000030e6668>] bpf_test_run+0x1ec/0x2f8
[ 30.945958] [<90000000030e794c>] bpf_prog_test_run_skb+0x31c/0x684
[ 30.946065] [<90000000026d4f68>] __sys_bpf+0x678/0x2724
[ 30.946159] [<90000000026d7288>] sys_bpf+0x20/0x2c
[ 30.946253] [<90000000032dd224>] do_syscall+0x7c/0x94
[ 30.946343] [<9000000002541c5c>] handle_syscall+0xbc/0x158
[ 30.946492]
[ 30.946549] Code: 0015030e 5c0009c0 5001d000 <28c00304> 02c00484 29c00304 00150009 2a42d2e4 0280200d
[ 30.946793]
[ 30.946971] ---[ end trace 0000000000000000 ]---
[ 32.093225] Kernel panic - not syncing: Fatal exception in interrupt
[ 32.093526] Kernel relocated by 0x2320000
[ 32.093630] .text @ 0x9000000002520000
[ 32.093725] .data @ 0x9000000003400000
[ 32.093792] .bss @ 0x9000000004413200
[ 34.971998] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

This is because we signed-extend function return values. When subprog
mode is enabled, we have:

cls_redirect()
-> get_global_metrics() returns pcpu ptr 0xfffffefffc00b480

The pointer returned is later signed-extended to 0xfffffffffc00b480 at
`BPF_JMP | BPF_EXIT`. During BPF prog run, this triggers unhandled page
fault and a kernel panic.

Drop the unnecessary signed-extension on return values like other
architectures do.

With this change, we have:

# ./test_progs -t cls_redirect
Can't find bpf_testmod.ko kernel module: -2
WARNING! Selftests relying on bpf_testmod.ko will be skipped.
#51/1 cls_redirect/cls_redirect_inlined:OK
#51/2 cls_redirect/IPv4 TCP accept unknown (no hops, flags: SYN):OK
#51/3 cls_redirect/IPv6 TCP accept unknown (no hops, flags: SYN):OK
#51/4 cls_redirect/IPv4 TCP accept unknown (no hops, flags: ACK):OK
#51/5 cls_redirect/IPv6 TCP accept unknown (no hops, flags: ACK):OK
#51/6 cls_redirect/IPv4 TCP forward unknown (one hop, flags: ACK):OK
#51/7 cls_redirect/IPv6 TCP forward unknown (one hop, flags: ACK):OK
#51/8 cls_redirect/IPv4 TCP accept known (one hop, flags: ACK):OK
#51/9 cls_redirect/IPv6 TCP accept known (one hop, flags: ACK):OK
#51/10 cls_redirect/IPv4 UDP accept unknown (no hops, flags: none):OK
#51/11 cls_redirect/IPv6 UDP accept unknown (no hops, flags: none):OK
#51/12 cls_redirect/IPv4 UDP forward unknown (one hop, flags: none):OK
#51/13 cls_redirect/IPv6 UDP forward unknown (one hop, flags: none):OK
#51/14 cls_redirect/IPv4 UDP accept known (one hop, flags: none):OK
#51/15 cls_redirect/IPv6 UDP accept known (one hop, flags: none):OK
#51/16 cls_redirect/cls_redirect_subprogs:OK
#51/17 cls_redirect/IPv4 TCP accept unknown (no hops, flags: SYN):OK
#51/18 cls_redirect/IPv6 TCP accept unknown (no hops, flags: SYN):OK
#51/19 cls_redirect/IPv4 TCP accept unknown (no hops, flags: ACK):OK
#51/20 cls_redirect/IPv6 TCP accept unknown (no hops, flags: ACK):OK
#51/21 cls_redirect/IPv4 TCP forward unknown (one hop, flags: ACK):OK
#51/22 cls_redirect/IPv6 TCP forward unknown (one hop, flags: ACK):OK
#51/23 cls_redirect/IPv4 TCP accept known (one hop, flags: ACK):OK
#51/24 cls_redirect/IPv6 TCP accept known (one hop, flags: ACK):OK
#51/25 cls_redirect/IPv4 UDP accept unknown (no hops, flags: none):OK
#51/26 cls_redirect/IPv6 UDP accept unknown (no hops, flags: none):OK
#51/27 cls_redirect/IPv4 UDP forward unknown (one hop, flags: none):OK
#51/28 cls_redirect/IPv6 UDP forward unknown (one hop, flags: none):OK
#51/29 cls_redirect/IPv4 UDP accept known (one hop, flags: none):OK
#51/30 cls_redirect/IPv6 UDP accept known (one hop, flags: none):OK
#51/31 cls_redirect/cls_redirect_dynptr:OK
#51/32 cls_redirect/IPv4 TCP accept unknown (no hops, flags: SYN):OK
#51/33 cls_redirect/IPv6 TCP accept unknown (no hops, flags: SYN):OK
#51/34 cls_redirect/IPv4 TCP accept unknown (no hops, flags: ACK):OK
#51/35 cls_redirect/IPv6 TCP accept unknown (no hops, flags: ACK):OK
#51/36 cls_redirect/IPv4 TCP forward unknown (one hop, flags: ACK):OK
#51/37 cls_redirect/IPv6 TCP forward unknown (one hop, flags: ACK):OK
#51/38 cls_redirect/IPv4 TCP accept known (one hop, flags: ACK):OK
#51/39 cls_redirect/IPv6 TCP accept known (one hop, flags: ACK):OK
#51/40 cls_redirect/IPv4 UDP accept unknown (no hops, flags: none):OK
#51/41 cls_redirect/IPv6 UDP accept unknown (no hops, flags: none):OK
#51/42 cls_redirect/IPv4 UDP forward unknown (one hop, flags: none):OK
#51/43 cls_redirect/IPv6 UDP forward unknown (one hop, flags: none):OK
#51/44 cls_redirect/IPv4 UDP accept known (one hop, flags: none):OK
#51/45 cls_redirect/IPv6 UDP accept known (one hop, flags: none):OK
#51 cls_redirect:OK
Summary: 1/45 PASSED, 0 SKIPPED, 0 FAILED

Fixes: 5dc615520c4d ("LoongArch: Add BPF JIT support")
Signed-off-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

LoongArch: BPF: Don't sign extend memory load operand

The `cgrp_local_storage` test triggers a kernel panic like:

# ./test_progs -t cgrp_local_storage
Can't find bpf_testmod.ko kernel module: -2
WARNING! Selftests relying on bpf_testmod.ko will be skipped.
[ 550.930632] CPU 1 Unable to handle kernel paging request at virtual address 0000000000000080, era == ffff80000200be34, ra == ffff80000200be00
[ 550.931781] Oops[#1]:
[ 550.931966] CPU: 1 PID: 1303 Comm: test_progs Not tainted 6.7.0-rc2-loong-devel-g2f56bb0d2327 #35 a896aca3f4164f09cc346f89f2e09832e07be5f6
[ 550.932215] Hardware name: QEMU QEMU Virtual Machine, BIOS unknown 2/2/2022
[ 550.932403] pc ffff80000200be34 ra ffff80000200be00 tp 9000000108350000 sp 9000000108353dc0
[ 550.932545] a0 0000000000000000 a1 0000000000000517 a2 0000000000000118 a3 00007ffffbb15558
[ 550.932682] a4 00007ffffbb15620 a5 90000001004e7700 a6 0000000000000021 a7 0000000000000118
[ 550.932824] t0 ffff80000200bdc0 t1 0000000000000517 t2 0000000000000517 t3 00007ffff1c06ee0
[ 550.932961] t4 0000555578ae04d0 t5 fffffffffffffff8 t6 0000000000000004 t7 0000000000000020
[ 550.933097] t8 0000000000000040 u0 00000000000007b8 s9 9000000108353e00 s0 90000001004e7700
[ 550.933241] s1 9000000004005000 s2 0000000000000001 s3 0000000000000000 s4 0000555555eb2ec8
[ 550.933379] s5 00007ffffbb15bb8 s6 00007ffff1dafd60 s7 000055555663f610 s8 00007ffff1db0050
[ 550.933520] ra: ffff80000200be00 bpf_prog_98f1b9e767be2a84_on_enter+0x40/0x200
[ 550.933911] ERA: ffff80000200be34 bpf_prog_98f1b9e767be2a84_on_enter+0x74/0x200
[ 550.934105] CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
[ 550.934596] PRMD: 00000004 (PPLV0 +PIE -PWE)
[ 550.934712] EUEN: 00000003 (+FPE +SXE -ASXE -BTE)
[ 550.934836] ECFG: 00071c1c (LIE=2-4,10-12 VS=7)
[ 550.934976] ESTAT: 00010000 [PIL] (IS= ECode=1 EsubCode=0)
[ 550.935097] BADV: 0000000000000080
[ 550.935181] PRID: 0014c010 (Loongson-64bit, Loongson-3A5000)
[ 550.935291] Modules linked in:
[ 550.935391] Process test_progs (pid: 1303, threadinfo=000000006c3b1c41, task=0000000061f84a55)
[ 550.935643] Stack : 00007ffffbb15bb8 0000555555eb2ec8 0000000000000000 0000000000000001
[ 550.935844] 9000000004005000 ffff80001b864000 00007ffffbb15450 90000000029aa034
[ 550.935990] 0000000000000000 9000000108353ec0 0000000000000118 d07d9dfb09721a09
[ 550.936175] 0000000000000001 0000000000000000 9000000108353ec0 0000000000000118
[ 550.936314] 9000000101d46ad0 900000000290abf0 000055555663f610 0000000000000000
[ 550.936479] 0000000000000003 9000000108353ec0 00007ffffbb15450 90000000029d7288
[ 550.936635] 00007ffff1dafd60 000055555663f610 0000000000000000 0000000000000003
[ 550.936779] 9000000108353ec0 90000000035dd1f0 00007ffff1dafd58 9000000002841c5c
[ 550.936939] 0000000000000119 0000555555eea5a8 00007ffff1d78780 00007ffffbb153e0
[ 550.937083] ffffffffffffffda 00007ffffbb15518 0000000000000040 00007ffffbb15558
[ 550.937224] ...
[ 550.937299] Call Trace:
[ 550.937521] [<ffff80000200be34>] bpf_prog_98f1b9e767be2a84_on_enter+0x74/0x200
[ 550.937910] [<90000000029aa034>] bpf_trace_run2+0x90/0x154
[ 550.938105] [<900000000290abf0>] syscall_trace_enter.isra.0+0x1cc/0x200
[ 550.938224] [<90000000035dd1f0>] do_syscall+0x48/0x94
[ 550.938319] [<9000000002841c5c>] handle_syscall+0xbc/0x158
[ 550.938477]
[ 550.938607] Code: 580009ae 50016000 262402e4 <28c20085> 14092084 03a00084 16000024 03240084 00150006
[ 550.938851]
[ 550.939021] ---[ end trace 0000000000000000 ]---

Further investigation shows that this panic is triggered by memory
load operations:

ptr = bpf_cgrp_storage_get(&map_a, task->cgroups->dfl_cgrp, 0,
BPF_LOCAL_STORAGE_GET_F_CREATE);

The expression `task->cgroups->dfl_cgrp` involves two memory load.
Since the field offset fits in imm12 or imm14, we use ldd or ldptrd
instructions. But both instructions have the side effect that it will
signed-extended the imm operand. Finally, we got the wrong addresses
and panics is inevitable.

Use a generic ldxd instruction to avoid this kind of issues.

With this change, we have:

# ./test_progs -t cgrp_local_storage
Can't find bpf_testmod.ko kernel module: -2
WARNING! Selftests relying on bpf_testmod.ko will be skipped.
test_cgrp_local_storage:PASS:join_cgroup /cgrp_local_storage 0 nsec
#48/1 cgrp_local_storage/tp_btf:OK
test_attach_cgroup:PASS:skel_open 0 nsec
test_attach_cgroup:PASS:prog_attach 0 nsec
test_attach_cgroup:PASS:prog_attach 0 nsec
libbpf: prog 'update_cookie_tracing': failed to attach: ERROR: strerror_r(-524)=22
test_attach_cgroup:FAIL:prog_attach unexpected error: -524
#48/2 cgrp_local_storage/attach_cgroup:FAIL
test_recursion:PASS:skel_open_and_load 0 nsec
libbpf: prog 'on_lookup': failed to attach: ERROR: strerror_r(-524)=22
libbpf: prog 'on_lookup': failed to auto-attach: -524
test_recursion:FAIL:skel_attach unexpected error: -524 (errno 524)
#48/3 cgrp_local_storage/recursion:FAIL
#48/4 cgrp_local_storage/negative:OK
#48/5 cgrp_local_storage/cgroup_iter_sleepable:OK
test_yes_rcu_lock:PASS:skel_open 0 nsec
test_yes_rcu_lock:PASS:skel_load 0 nsec
libbpf: prog 'yes_rcu_lock': failed to attach: ERROR: strerror_r(-524)=22
libbpf: prog 'yes_rcu_lock': failed to auto-attach: -524
test_yes_rcu_lock:FAIL:skel_attach unexpected error: -524 (errno 524)
#48/6 cgrp_local_storage/yes_rcu_lock:FAIL
#48/7 cgrp_local_storage/no_rcu_lock:OK
#48 cgrp_local_storage:FAIL

All error logs:
test_cgrp_local_storage:PASS:join_cgroup /cgrp_local_storage 0 nsec
test_attach_cgroup:PASS:skel_open 0 nsec
test_attach_cgroup:PASS:prog_attach 0 nsec
test_attach_cgroup:PASS:prog_attach 0 nsec
libbpf: prog 'update_cookie_tracing': failed to attach: ERROR: strerror_r(-524)=22
test_attach_cgroup:FAIL:prog_attach unexpected error: -524
#48/2 cgrp_local_storage/attach_cgroup:FAIL
test_recursion:PASS:skel_open_and_load 0 nsec
libbpf: prog 'on_lookup': failed to attach: ERROR: strerror_r(-524)=22
libbpf: prog 'on_lookup': failed to auto-attach: -524
test_recursion:FAIL:skel_attach unexpected error: -524 (errno 524)
#48/3 cgrp_local_storage/recursion:FAIL
test_yes_rcu_lock:PASS:skel_open 0 nsec
test_yes_rcu_lock:PASS:skel_load 0 nsec
libbpf: prog 'yes_rcu_lock': failed to attach: ERROR: strerror_r(-524)=22
libbpf: prog 'yes_rcu_lock': failed to auto-attach: -524
test_yes_rcu_lock:FAIL:skel_attach unexpected error: -524 (errno 524)
#48/6 cgrp_local_storage/yes_rcu_lock:FAIL
#48 cgrp_local_storage:FAIL
Summary: 0/4 PASSED, 0 SKIPPED, 1 FAILED

No panics any more (The test still failed because lack of BPF trampoline
which I am actively working on).

Fixes: 5dc615520c4d ("LoongArch: Add BPF JIT support")
Signed-off-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

LoongArch: BPF: Support signed mod instructions

Add support for signed mod instructions.

Signed-off-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

LoongArch: BPF: Support signed div instructions

Add support for signed div instructions.

Signed-off-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

LoongArch: BPF: Support 32-bit offset jmp instructions

Add support for 32-bit offset jmp instruction. Currently, we use b
instruction which supports range within ±128MB for such jumps. This
should be large enough for BPF progs.

Signed-off-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

LoongArch: BPF: Support unconditional bswap instructions

Add support for unconditional bswap instruction. Since LoongArch is
always little-endian, just treat unconditional bswap the same as big-
endian conversion.

Signed-off-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

LoongArch: BPF: Support sign-extension mov instructions

Add support for sign-extension mov instructions.

Signed-off-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

LoongArch: BPF: Support sign-extension load instructions

Add support for sign-extension load instructions.

Signed-off-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

LoongArch: BPF: Support mixing bpf2bpf and tailcalls

The current implementation already allow such mixing.
Let's enable it in JIT.

Signed-off-by: Hengqi Chen <hengqi.chen@gmail.com>
Link: https://lore.kernel.org/r/20230218105317.4139666-1-hengqi.chen@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>

LoongArch, bpf: Fix jit to skip speculation barrier opcode

Just skip the opcode(BPF_ST | BPF_NOSPEC) in the BPF JIT instead of
failing to JIT the entire program, given LoongArch currently has no
couterpart of a speculation barrier instruction. To verify the issue,
use the ltp testcase as shown below.

Also, Wang says:

I can confirm there's currently no speculation barrier equivalent
on LonogArch. (Loongson says there are builtin mitigations for
Spectre-V1 and V2 on their chips, and AFAIK efforts to port the
exploits to mips/LoongArch have all failed a few years ago.)

Without this patch:

$ ./bpf_prog02
[...]
bpf_common.c:123: TBROK: Failed verification: ??? (524)
[...]
Summary:
passed 0
failed 0
broken 1
skipped 0
warnings 0

With this patch:

$ ./bpf_prog02
[...]
Summary:
passed 0
failed 0
broken 0
skipped 0
warnings 0

Fixes: 5dc615520c4d ("LoongArch: Add BPF JIT support")
Signed-off-by: George Guo <guodongtai@kylinos.cn>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: WANG Xuerui <git@xen0n.name>
Cc: Tiezhu Yang <yangtiezhu@loongson.cn>
Link: https://lore.kernel.org/bpf/20230328071335.2664966-1-guodongtai@kylinos.cn

LoongArch, bpf: Use 4 instructions for function address in JIT

This patch fixes the following issue of function calls in JIT, like:

[ 29.346981] multi-func JIT bug 105 != 103

The issus can be reproduced by running the "inline simple bpf_loop call"
verifier test.

This is because we are emiting 2-4 instructions for 64-bit immediate moves.
During the first pass of JIT, the placeholder address is zero, emiting two
instructions for it. In the extra pass, the function address is in XKVRANGE,
emiting four instructions for it. This change the instruction index in
JIT context. Let's always use 4 instructions for function address in JIT.
So that the instruction sequences don't change between the first pass and
the extra pass for function calls.

Fixes: 5dc615520c4d ("LoongArch: Add BPF JIT support")
Signed-off-by: Hengqi Chen <hengqi.chen@gmail.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Tested-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Link: https://lore.kernel.org/bpf/20230214152633.2265699-1-hengqi.chen@gmail.com

LoongArch: BPF: Add BPF exception tables

Inspired by commit 800834285361("bpf, arm64: Add BPF exception tables"),
do similar to LoongArch to add BPF exception tables.

When a tracing BPF program attempts to read memory without using the
bpf_probe_read() helper, the verifier marks the load instruction with
the BPF_PROBE_MEM flag. Since the LoongArch JIT does not currently
recognize this flag it falls back to the interpreter.

Add support for BPF_PROBE_MEM, by appending an exception table to the
BPF program. If the load instruction causes a data abort, the fixup
infrastructure finds the exception table and fixes up the fault, by
clearing the destination register and jumping over the faulting
instruction.

To keep the compact exception table entry format, inspect the pc in
fixup_exception(). A more generic solution would add a "handler" field
to the table entry, like on x86, s390 and arm64, etc.

Signed-off-by: Youling Tang <tangyouling@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

LoongArch: BPF: Avoid declare variables in switch-case

Not all compilers support declare variables in switch-case, so move
declarations to the beginning of a function. Otherwise we may get such
build errors:

arch/loongarch/net/bpf_jit.c: In function ‘emit_atomic’:
arch/loongarch/net/bpf_jit.c:362:3: error: a label can only be part of a statement and a declaration is not a statement
u8 r0 = regmap[BPF_REG_0];
^~
arch/loongarch/net/bpf_jit.c: In function ‘build_insn’:
arch/loongarch/net/bpf_jit.c:727:3: error: a label can only be part of a statement and a declaration is not a statement
u8 t7 = -1;
^~
arch/loongarch/net/bpf_jit.c:778:3: error: a label can only be part of a statement and a declaration is not a statement
int ret;
^~~
arch/loongarch/net/bpf_jit.c:779:3: error: expected expression before ‘u64’
u64 func_addr;
^~~
arch/loongarch/net/bpf_jit.c:780:3: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
bool func_addr_fixed;
^~~~
arch/loongarch/net/bpf_jit.c:784:11: error: ‘func_addr’ undeclared (first use in this function); did you mean ‘in_addr’?
&func_addr, &func_addr_fixed);
^~~~~~~~~
in_addr
arch/loongarch/net/bpf_jit.c:784:11: note: each undeclared identifier is reported only once for each function it appears in
arch/loongarch/net/bpf_jit.c:814:3: error: a label can only be part of a statement and a declaration is not a statement
u64 imm64 = (u64)(insn + 1)->imm << 32 | (u32)insn->imm;
^~~

Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>

LoongArch: Add BPF JIT support

BPF programs are normally handled by a BPF interpreter, add BPF JIT
support for LoongArch to allow the kernel to generate native code when
a program is loaded into the kernel. This will significantly speed-up
processing of BPF programs.

Co-developed-by: Youling Tang <tangyouling@loongson.cn>
Signed-off-by: Youling Tang <tangyouling@loongson.cn>
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>