Re: Calling kfuncs in modules - BTF mismatch?

From: Alan Maguire <alan.maguire@oracle.com>
To: "Toke Høiland-Jørgensen" <toke@redhat.com>,
	"Lorenzo Bianconi" <lorenzo.bianconi@redhat.com>,
	"Arnaldo Carvalho de Melo" <acme@kernel.org>
Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com>,
	Andrii Nakryiko <andrii@kernel.org>,
	bpf@vger.kernel.org, Jiri Benc <jbenc@redhat.com>,
	Lorenzo Bianconi <lorenzo@kernel.org>
Subject: Re: Calling kfuncs in modules - BTF mismatch?
Date: Tue, 29 Nov 2022 16:21:23 +0000	[thread overview]
Message-ID: <08fa5e85-4d7b-4725-f340-bcb8525036f1@oracle.com> (raw)
In-Reply-To: <875yfiwx1g.fsf@toke.dk>

On 13/11/2022 18:04, Toke Høiland-Jørgensen wrote:
> Lorenzo Bianconi <lorenzo.bianconi@redhat.com> writes:
> 
>>>
>>> Hi everyone
>>>
>>> There seems to be some issue with BTF mismatch when trying to run the
>>> bpf_ct_set_nat_info() kfunc from a module. I was under the impression
>>> that this is supposed to work, so is there some kind of BTF dedup issue
>>> here or something?
>>>
>>> Steps to reproduce:
>>>
>>> 1. Compile kernel with nf_conntrack built-in and run selftests;
>>>    './test_progs -a bpf_nf' works
>>>
>>> 2. Change the kernel config so nf_conntrack is build as a module
>>>
>>> 3. Start the test kernel and manually modprobe nf_conntrack and nf_nat
>>>
>>> 4. Run ./test_progs -a bpf_nf; this now fails with an error like:
>>>
>>> kernel function bpf_ct_set_nat_info args#0 expected pointer to STRUCT nf_conn___init but R1 has a pointer to STRUCT nf_conn___init
>>
>> This week Kumar and I took a look at this issue and we ended up
>> identifying a duplication of nf_conn___init structure. In particular:
>>
>> [~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file
>> net/netfilter/nf_conntrack.ko format raw | grep nf_conn__
>> [110941] STRUCT 'nf_conn___init' size=248 vlen=1
>> [~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file
>> net/netfilter/nf_nat.ko format raw | grep nf_conn__
>> [107488] STRUCT 'nf_conn___init' size=248 vlen=1
>>
>> Is it the root cause of the problem?
> 
> It certainly seems to be related to it, at least. Amending the log
> message to include the BTF object IDs of the two versions shows that the
> register has a reference to nf_conn__init in nf_conntrack.ko, while the kernel
> expects it to point to nf_nat.ko.
> 
> Not sure what's the right fix for this? Should libbpf be smart enough to
> pull the kfunc arg ID from the same BTF ID as the function itself? Or
> should the kernel compare structs and allow things if they're identical?
> Andrii, WDYT?
> 

There were some dedup issues fixed recently in pahole
and libbpf; since dwarves libbpf hasn't been synced with
libbpf recently as far as I can see it won't have the fix 
for [1]; I suspect it may help with dedup-ing here. Would 
probably be worth trying rebuilding dwarves with a libbpf 
with [1] applied and seeing if the dedup issue goes away
before we go any further. If it fixes the issue, would it
be worth updating the libbpf that dwarves uses Arnaldo?
I saw some pretty large improvements in removing
redundant definitions.

Thanks!

Alan

https://lore.kernel.org/bpf/1666622309-22289-1-git-send-email-alan.maguire@oracle.com/

> -Toke
> 