Completing eBPF JIT support for MIPS32

[Johan Almbladh <johan.almbladh@anyfinetworks.com>]

The MIPS64 architecture has had an eBPF JIT for quite some time, but
the kernel still lacks eBPF JIT support for MIPS32. In 2019 there was
an effort extend the MIPS64 JIT to also support 32-bit MIPS ISAs,
716850ab104d ("MIPS: eBPF: Initial eBPF support for MIPS32
architecture."), but the work was never completed. What would be
needed to bring eBPF JIT support to MIPS32?

According to Paul Burton, 36366e367ee9 ("MIPS: BPF: Restore MIPS32 cBPF JIT")

> The eBPF JIT has a number of problems on MIPS32:
>
> - Most notably various code paths still result in emission of MIPS64
>   instructions which will cause reserved instruction exceptions & kernel
>   panics when run on MIPS32 CPUs.
>
> - The eBPF JIT doesn't account for differences between the O32 ABI used
>   by MIPS32 kernels versus the N64 ABI used by MIPS64 kernels. Notably
>   arguments beyond the first 4 are passed on the stack in O32, and this
>   is entirely unhandled when JITing a BPF_CALL instruction. Stack space
>   must be reserved for arguments even if they all fit in registers, and
>   the callee is free to assume that stack space has been reserved for
>   its use - with the eBPF JIT this is not the case, so calling any
>   function can result in clobbering values on the stack & unpredictable
>   behaviour. Function arguments in eBPF are always 64-bit values which
>   is also entirely unhandled - the JIT still uses a single (32-bit)
>   register per argument. As a result all function arguments are always
>   passed incorrectly when JITing a BPF_CALL instruction, leading to
>   kernel crashes or strange behavior.
>
> - The JIT attempts to bail our on use of ALU64 instructions or 64-bit
>   memory access instructions. The code doing this at the start of
>   build_one_insn() incorrectly checks whether BPF_OP() equals BPF_DW,
>   when it should really be checking BPF_SIZE() & only doing so when
>   BPF_CLASS() is one of BPF_{LD,LDX,ST,STX}. This results in false
>   positives that cause more bailouts than intended, and that in turns
>   hides some of the problems described above.
>
> - The kernel's cBPF->eBPF translation makes heavy use of 64-bit eBPF
>   instructions that the MIPS32 eBPF JIT bails out on, leading to most
>   cBPF programs not being JITed at all.

I can see two different ways to proceed from here.

1) Continue the work on the 64-bit eBPF JIT and complete the MIPS32
support there.

2) Use the 32-bit cBPF JIT as a fresh start for a new 32-bit eBPF JIT
implementation.

It depends of course on how much effort it would be to fix the
remaining issues with the current eBPF JIT. On the other hand, since
eBPF is a 64-bit architecture I would expect the bytecode to translate
nicely to a 64-bit target, and that a 32-bit JIT might differ a lot.
For other targets there are often separate 64-bit and 32-bit JIT
implementations. I also think the JIT support for MIPS32 should make
it possible to replace the cBPF JIT. That means it cannot fall back to
use the interpreter as I understand the current (32-bit)
implementation does to a large extent. Perhaps it would then be
cleaner to let the 32-bit version be separate implementation.

Any thoughts and suggestions? What do you think would be the best way forward?

Thanks,
Johan