Re: Running JITed and interpreted programs simultaneously

From: Juraj Vijtiuk <juraj.vijtiuk@sartura.hr>
To: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>,
	Daniel Borkmann <daniel@iogearbox.net>, bpf <bpf@vger.kernel.org>,
	Luka Perkov <luka.perkov@sartura.hr>,
	David Marcinkovic <david.marcinkovic@sartura.hr>
Subject: Re: Running JITed and interpreted programs simultaneously
Date: Tue, 20 Oct 2020 22:56:34 +0200	[thread overview]
Message-ID: <CAOjtDRWSB9_yYMqSy6Z_RGrtpk6uJUkXy4gby0-hqAgafitgTA@mail.gmail.com> (raw)
In-Reply-To: <CAADnVQLLQnyiwnw8jPxgJtb59t78wz8X6JQZhTxUe0gw+yRz7w@mail.gmail.com>

On Tue, Oct 20, 2020 at 12:02 AM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Mon, Oct 19, 2020 at 11:26 AM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > That wasn't happening last time people reported this on ARM32.
> > BPF_XADD was causing load failure, no fail back to interpreter mode.
> >
> > >
> > > Wrt force-interpret vs force-jit BPF_PROG_LOAD flag, I'm more concerned that this
> > > decision will then be pushed to the user who should not have to care about these
> > > internals. And how would generic loaders try to react if force-jit fails? They would
> > > then fallback to force-interpret same way as kernel does?
> >
> > The way I imagined this was if the user wants to force the mode and
> > the kernel doesn't support it (or the program can't be loaded in that
> > mode), then it's a fail-stop, no fall back. And it's strictly an
> > opt-in flag, if nothing is specified then it's current behavior with
> > fallback (which apparently doesn't always work).
>
> That doesn't sound right.
> Fallback to interpreter should always work unless features like
> trampoline are used.
> But that's not the case for arm32. Missing xadd support shouldn't cause
> load failure.

After some retesting, it turns out that everything is working as it is
supposed to. I'm sorry for the confusion this caused.

My colleagues and I originally ran into the XADD issue on a device
that had CONFIG_BPF_JIT_ALWAYS_ON [0].
That resulted in libbpf reporting the following error:
libbpf: load bpf program failed:
ERROR: strerror_r(524)=22

Other than that the log was mostly empty, except for the number of
processed instructions and other similar info.
After the suggestion to try running the program without JIT, we
recompiled the image without JIT_ALWAYS_ON, but wrongly assumed that
/proc/sys/net/core/bpf_jit_enable has to be set to 0 for the program
to work, so we have never tested with bpf_jit_enable set to 1.

We have now tested on a device with JIT_ALWAYS_ON turned off, and the
program works with bpf_jit_enable set to both 1 or 0, while running on
a device with JIT_ALWAYS_ON still causes the same error that we
originally encountered.

Thank you for the help everyone.

[0] https://lore.kernel.org/bpf/CAO__=G6kqajLdP_cWJiAUjXMRdJe2xBy2FJGiM1v4h6YquD3kg@mail.gmail.com/