Re: [PATCH bpf-next] selftests/bpf: change llvm flag -mcpu=probe to -mcpu=v3

From: Daniel Borkmann <daniel@iogearbox.net>
To: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Cc: Yonghong Song <yhs@fb.com>, bpf <bpf@vger.kernel.org>,
	Alexei Starovoitov <ast@fb.com>, Kernel Team <kernel-team@fb.com>
Subject: Re: [PATCH bpf-next] selftests/bpf: change llvm flag -mcpu=probe to -mcpu=v3
Date: Thu, 20 Feb 2020 01:15:30 +0100	[thread overview]
Message-ID: <e53c30a2-5979-6358-025a-bae11a8d01a3@iogearbox.net> (raw)
In-Reply-To: <CAADnVQ+CtuOkpidngFxEXWU_efLOv9_ozj=eSgNo1os1w3b8Sw@mail.gmail.com>

On 2/20/20 12:17 AM, Alexei Starovoitov wrote:
> On Wed, Feb 19, 2020 at 9:06 AM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
>> On Wed, Feb 19, 2020 at 8:56 AM Daniel Borkmann <daniel@iogearbox.net> wrote:
>>> On 2/19/20 1:42 AM, Yonghong Song wrote:
>>>> The latest llvm supports cpu version v3, which is cpu version v1
>>>> plus some additional 64bit jmp insns and 32bit jmp insn support.
>>>>
>>>> In selftests/bpf Makefile, the llvm flag -mcpu=probe did runtime
>>>> probe into the host system. Depending on compilation environments,
>>>> it is possible that runtime probe may fail, e.g., due to
>>>> memlock issue. This will cause generated code with cpu version v1.
>>>
>>> But those are tiny BPF progs that LLVM is probing. If memlock is not
>>> sufficient, should it try to bump the limit with the diff needed and
>>> only if that fails as well then it bails out to v1.
>>
>> with hundred parallel clangs running and all stamping on the same rlimit
>> I don't think bumping that limit can work.

Right, my main worry is that the default memlock limit is usually very
low, so it would be quite ugly to have the probe fail and fall-back to
v1 even though the underlying kernel would be totally fine to support v3
in general. Hard-coding v3 for selftests is okay; perhaps we need to
resurrect the old CAP_IPC_LOCK patch or some different accounting, the
memlock limit has never been working great from a usability pov.

>> Also building on older kernel should still do v3, since build should
>> produce selftest binaries for the same vmlinux as this kernel tree.
>> We hit this issue with github/libbpf CI. The vm used to do the build
>> was too old. So far we cannot build vmlinux out of latest tree,
>> boot into it and only then build selftests inside. It's too complex
>> for CI system.
>> So we build vmlinux and build selftests in that CI's VM, and then boot into it
>> and run selftests.
>> Upgrading VM is an easy fix for now, but the issue will cause the problems
>> later. So imo fixing selftests build to predictable -mcpu=v3 is the
>> most sensible way.

It would definitely be great to test all at some point, meaning, test run
with v1, v2, v3 to ensure there are no regressions e.g. on verifier side
for all of them.

Thanks,
Daniel