BPF calls to modules?

BPF calls to modules?

[Alex Ghiti]

Hi guys,

First, pardon my ignorance regarding BPF, the following might be silly.

We were wondering here 
https://patchwork.kernel.org/project/linux-riscv/patch/20210615004928.2d27d2ac@xhacker/ 
if BPF programs that now have the capability to call kernel functions 
(https://lwn.net/Articles/856005/) can also call modules function or 
vice-versa?

The underlying important fact is that in riscv, we are limited to 2GB 
offset to call functions and that restricts where we can place modules 
and BPF regions wrt kernel (see Documentation/riscv/vm-layout.rst for 
the current possibly wrong layout).

So should we make sure that modules and BPF lie in the same 2GB region?

Thanks,

Alex

Re: BPF calls to modules?

[Alexei Starovoitov]

On Fri, Jun 18, 2021 at 2:13 AM Alex Ghiti <alex@ghiti.fr> wrote:
>
> Hi guys,
>
> First, pardon my ignorance regarding BPF, the following might be silly.
>
> We were wondering here
> https://patchwork.kernel.org/project/linux-riscv/patch/20210615004928.2d27d2ac@xhacker/
> if BPF programs that now have the capability to call kernel functions
> (https://lwn.net/Articles/856005/) can also call modules function or
> vice-versa?
>
> The underlying important fact is that in riscv, we are limited to 2GB
> offset to call functions and that restricts where we can place modules
> and BPF regions wrt kernel (see Documentation/riscv/vm-layout.rst for
> the current possibly wrong layout).
>
> So should we make sure that modules and BPF lie in the same 2GB region?

Ideally yes. bpf programs can call functions in modules and vice versa.
riscv can keep them in separate regions, but then riscv JIT will get more
complicated and run-time overhead of the function call will increase.

Re: BPF calls to modules?

[Andrii Nakryiko]

On Fri, Jun 18, 2021 at 2:13 AM Alex Ghiti <alex@ghiti.fr> wrote:
>
> Hi guys,
>
> First, pardon my ignorance regarding BPF, the following might be silly.
>
> We were wondering here
> https://patchwork.kernel.org/project/linux-riscv/patch/20210615004928.2d27d2ac@xhacker/
> if BPF programs that now have the capability to call kernel functions
> (https://lwn.net/Articles/856005/) can also call modules function or
> vice-versa?

Not yet, but it was an explicit design consideration and there was
public interest just recently. So I'd say this is going to happen
sooner rather than later.

>
> The underlying important fact is that in riscv, we are limited to 2GB
> offset to call functions and that restricts where we can place modules
> and BPF regions wrt kernel (see Documentation/riscv/vm-layout.rst for
> the current possibly wrong layout).
>
> So should we make sure that modules and BPF lie in the same 2GB region?

Based on the above and what you are explaining about 2GB limits, I'd
say yes?.. Or alternatively those 2GB restrictions might perhaps be
lifted somehow?

>
> Thanks,
>
> Alex

Re: BPF calls to modules?

[Alex Ghiti]

Hi,

Le 18/06/2021 à 19:32, Andrii Nakryiko a écrit :
> On Fri, Jun 18, 2021 at 2:13 AM Alex Ghiti <alex@ghiti.fr> wrote:
>>
>> Hi guys,
>>
>> First, pardon my ignorance regarding BPF, the following might be silly.
>>
>> We were wondering here
>> https://patchwork.kernel.org/project/linux-riscv/patch/20210615004928.2d27d2ac@xhacker/
>> if BPF programs that now have the capability to call kernel functions
>> (https://lwn.net/Articles/856005/) can also call modules function or
>> vice-versa?
> 
> Not yet, but it was an explicit design consideration and there was
> public interest just recently. So I'd say this is going to happen
> sooner rather than later.
> 
>>
>> The underlying important fact is that in riscv, we are limited to 2GB
>> offset to call functions and that restricts where we can place modules
>> and BPF regions wrt kernel (see Documentation/riscv/vm-layout.rst for
>> the current possibly wrong layout).
>>
>> So should we make sure that modules and BPF lie in the same 2GB region?
> 
> Based on the above and what you are explaining about 2GB limits, I'd
> say yes?.. Or alternatively those 2GB restrictions might perhaps be
> lifted somehow?


Actually we have this limit when we have PC-relative branch which is our 
current code model. To better understand what happened, I took a look at 
our JIT implementation and noticed that BPF_CALL are implemented using 
absolute addressing so for this pseudo-instruction, the limit I evoked 
does not apply. How are the kernel (and modules) symbol addresses 
resolved? Is it relative or absolute? Is there then any guarantee that a 
kernel or module call will always emit a BPF_CALL?

Thanks Alexei and Andrii for your answers,

Alex

> 
>>
>> Thanks,
>>
>> Alex

Re: BPF calls to modules?

[Alexei Starovoitov]

On Sun, Jun 20, 2021 at 11:43 PM Alex Ghiti <alex@ghiti.fr> wrote:
>
> Hi,
>
> Le 18/06/2021 à 19:32, Andrii Nakryiko a écrit :
> > On Fri, Jun 18, 2021 at 2:13 AM Alex Ghiti <alex@ghiti.fr> wrote:
> >>
> >> Hi guys,
> >>
> >> First, pardon my ignorance regarding BPF, the following might be silly.
> >>
> >> We were wondering here
> >> https://patchwork.kernel.org/project/linux-riscv/patch/20210615004928.2d27d2ac@xhacker/
> >> if BPF programs that now have the capability to call kernel functions
> >> (https://lwn.net/Articles/856005/) can also call modules function or
> >> vice-versa?
> >
> > Not yet, but it was an explicit design consideration and there was
> > public interest just recently. So I'd say this is going to happen
> > sooner rather than later.
> >
> >>
> >> The underlying important fact is that in riscv, we are limited to 2GB
> >> offset to call functions and that restricts where we can place modules
> >> and BPF regions wrt kernel (see Documentation/riscv/vm-layout.rst for
> >> the current possibly wrong layout).
> >>
> >> So should we make sure that modules and BPF lie in the same 2GB region?
> >
> > Based on the above and what you are explaining about 2GB limits, I'd
> > say yes?.. Or alternatively those 2GB restrictions might perhaps be
> > lifted somehow?
>
>
> Actually we have this limit when we have PC-relative branch which is our
> current code model. To better understand what happened, I took a look at
> our JIT implementation and noticed that BPF_CALL are implemented using
> absolute addressing so for this pseudo-instruction, the limit I evoked
> does not apply. How are the kernel (and modules) symbol addresses
> resolved? Is it relative or absolute? Is there then any guarantee that a
> kernel or module call will always emit a BPF_CALL?

Are those questions for riscv bpf JIT experts?
Like 'relative or absolute' depends on arch.
On x86-64 BPF_CALL is JITed into single x86 call instruction that
has 32-bit immediate which is PC relative.
Every JIT picks what's the best for that particular arch.

Re: BPF calls to modules?

[Alex Ghiti]

Hi Alexei,

Le 22/06/2021 à 02:28, Alexei Starovoitov a écrit :
> On Sun, Jun 20, 2021 at 11:43 PM Alex Ghiti <alex@ghiti.fr> wrote:
>>
>> Hi,
>>
>> Le 18/06/2021 à 19:32, Andrii Nakryiko a écrit :
>>> On Fri, Jun 18, 2021 at 2:13 AM Alex Ghiti <alex@ghiti.fr> wrote:
>>>>
>>>> Hi guys,
>>>>
>>>> First, pardon my ignorance regarding BPF, the following might be silly.
>>>>
>>>> We were wondering here
>>>> https://patchwork.kernel.org/project/linux-riscv/patch/20210615004928.2d27d2ac@xhacker/
>>>> if BPF programs that now have the capability to call kernel functions
>>>> (https://lwn.net/Articles/856005/) can also call modules function or
>>>> vice-versa?
>>>
>>> Not yet, but it was an explicit design consideration and there was
>>> public interest just recently. So I'd say this is going to happen
>>> sooner rather than later.
>>>
>>>>
>>>> The underlying important fact is that in riscv, we are limited to 2GB
>>>> offset to call functions and that restricts where we can place modules
>>>> and BPF regions wrt kernel (see Documentation/riscv/vm-layout.rst for
>>>> the current possibly wrong layout).
>>>>
>>>> So should we make sure that modules and BPF lie in the same 2GB region?
>>>
>>> Based on the above and what you are explaining about 2GB limits, I'd
>>> say yes?.. Or alternatively those 2GB restrictions might perhaps be
>>> lifted somehow?
>>
>>
>> Actually we have this limit when we have PC-relative branch which is our
>> current code model. To better understand what happened, I took a look at
>> our JIT implementation and noticed that BPF_CALL are implemented using
>> absolute addressing so for this pseudo-instruction, the limit I evoked
>> does not apply. How are the kernel (and modules) symbol addresses
>> resolved? Is it relative or absolute? Is there then any guarantee that a
>> kernel or module call will always emit a BPF_CALL?
> 
> Are those questions for riscv bpf JIT experts?

Yes more or less, sorry about that, I added Bjorn in cc in case he wants 
to intervene. But I think my last question is relevant: Is there then 
any guarantee that a kernel or module call will always emit a BPF_CALL? 
Because that would mean that we don't need to place BPF close to modules 
since BPF_CALL are JITed into an absolute branch in riscv.

Sorry to bother,

Thanks you for your time,

Alex

> Like 'relative or absolute' depends on arch.
> On x86-64 BPF_CALL is JITed into single x86 call instruction that
> has 32-bit immediate which is PC relative.
> Every JIT picks what's the best for that particular arch.
> 

Re: BPF calls to modules?

[Jisheng Zhang]

Hi Alex,

On Tue, 22 Jun 2021 09:31:33 +0200
Alex Ghiti <alex@ghiti.fr> wrote:

> Hi Alexei,
> 
> Le 22/06/2021 à 02:28, Alexei Starovoitov a écrit :
> > On Sun, Jun 20, 2021 at 11:43 PM Alex Ghiti <alex@ghiti.fr> wrote:  
> >>
> >> Hi,
> >>
> >> Le 18/06/2021 à 19:32, Andrii Nakryiko a écrit :  
> >>> On Fri, Jun 18, 2021 at 2:13 AM Alex Ghiti <alex@ghiti.fr> wrote:  
> >>>>
> >>>> Hi guys,
> >>>>
> >>>> First, pardon my ignorance regarding BPF, the following might be silly.
> >>>>
> >>>> We were wondering here
> >>>> https://patchwork.kernel.org/project/linux-riscv/patch/20210615004928.2d27d2ac@xhacker/
> >>>> if BPF programs that now have the capability to call kernel functions
> >>>> (https://lwn.net/Articles/856005/) can also call modules function or
> >>>> vice-versa?  
> >>>
> >>> Not yet, but it was an explicit design consideration and there was
> >>> public interest just recently. So I'd say this is going to happen
> >>> sooner rather than later.
> >>>  
> >>>>
> >>>> The underlying important fact is that in riscv, we are limited to 2GB
> >>>> offset to call functions and that restricts where we can place modules
> >>>> and BPF regions wrt kernel (see Documentation/riscv/vm-layout.rst for
> >>>> the current possibly wrong layout).
> >>>>
> >>>> So should we make sure that modules and BPF lie in the same 2GB region?  
> >>>
> >>> Based on the above and what you are explaining about 2GB limits, I'd
> >>> say yes?.. Or alternatively those 2GB restrictions might perhaps be
> >>> lifted somehow?  
> >>
> >>
> >> Actually we have this limit when we have PC-relative branch which is our
> >> current code model. To better understand what happened, I took a look at
> >> our JIT implementation and noticed that BPF_CALL are implemented using
> >> absolute addressing so for this pseudo-instruction, the limit I evoked
> >> does not apply. How are the kernel (and modules) symbol addresses
> >> resolved? Is it relative or absolute? Is there then any guarantee that a
> >> kernel or module call will always emit a BPF_CALL?  
> > 
> > Are those questions for riscv bpf JIT experts?  
> 
> Yes more or less, sorry about that, I added Bjorn in cc in case he wants 
> to intervene. But I think my last question is relevant: Is there then 
> any guarantee that a kernel or module call will always emit a BPF_CALL? 

Just my humble opinion: not always, but some kernel/module functions may
call BPF. The problem is we dunno where the kernel or module functions
sit.

> Because that would mean that we don't need to place BPF close to modules 
> since BPF_CALL are JITed into an absolute branch in riscv.
> 
> Sorry to bother,
> 
> Thanks you for your time,
> 
> Alex
> 
> > Like 'relative or absolute' depends on arch.
> > On x86-64 BPF_CALL is JITed into single x86 call instruction that
> > has 32-bit immediate which is PC relative.
> > Every JIT picks what's the best for that particular arch.
> >   

Re: BPF calls to modules?

[Alexei Starovoitov]

On Tue, Jun 22, 2021 at 12:31 AM Alex Ghiti <alex@ghiti.fr> wrote:
>
> Hi Alexei,
>
> Le 22/06/2021 à 02:28, Alexei Starovoitov a écrit :
> > On Sun, Jun 20, 2021 at 11:43 PM Alex Ghiti <alex@ghiti.fr> wrote:
> >>
> >> Hi,
> >>
> >> Le 18/06/2021 à 19:32, Andrii Nakryiko a écrit :
> >>> On Fri, Jun 18, 2021 at 2:13 AM Alex Ghiti <alex@ghiti.fr> wrote:
> >>>>
> >>>> Hi guys,
> >>>>
> >>>> First, pardon my ignorance regarding BPF, the following might be silly.
> >>>>
> >>>> We were wondering here
> >>>> https://patchwork.kernel.org/project/linux-riscv/patch/20210615004928.2d27d2ac@xhacker/
> >>>> if BPF programs that now have the capability to call kernel functions
> >>>> (https://lwn.net/Articles/856005/) can also call modules function or
> >>>> vice-versa?
> >>>
> >>> Not yet, but it was an explicit design consideration and there was
> >>> public interest just recently. So I'd say this is going to happen
> >>> sooner rather than later.
> >>>
> >>>>
> >>>> The underlying important fact is that in riscv, we are limited to 2GB
> >>>> offset to call functions and that restricts where we can place modules
> >>>> and BPF regions wrt kernel (see Documentation/riscv/vm-layout.rst for
> >>>> the current possibly wrong layout).
> >>>>
> >>>> So should we make sure that modules and BPF lie in the same 2GB region?
> >>>
> >>> Based on the above and what you are explaining about 2GB limits, I'd
> >>> say yes?.. Or alternatively those 2GB restrictions might perhaps be
> >>> lifted somehow?
> >>
> >>
> >> Actually we have this limit when we have PC-relative branch which is our
> >> current code model. To better understand what happened, I took a look at
> >> our JIT implementation and noticed that BPF_CALL are implemented using
> >> absolute addressing so for this pseudo-instruction, the limit I evoked
> >> does not apply. How are the kernel (and modules) symbol addresses
> >> resolved? Is it relative or absolute? Is there then any guarantee that a
> >> kernel or module call will always emit a BPF_CALL?
> >
> > Are those questions for riscv bpf JIT experts?
>
> Yes more or less, sorry about that, I added Bjorn in cc in case he wants
> to intervene. But I think my last question is relevant: Is there then
> any guarantee that a kernel or module call will always emit a BPF_CALL?
> Because that would mean that we don't need to place BPF close to modules
> since BPF_CALL are JITed into an absolute branch in riscv.

I don't understand what you mean with this question.
BPF_CALL is a BPF instruction to call from bpf prog. Not into bpf prog.
When kernel or module calls into JITed bpf prog
they use indirect call insn of the given arch.
In case of bpf dispatcher there is a generated asm code that uses jmp
by register
or retpoline style.
So JITed bpf progs not only 'called' into.
From bpf prog the helpers and kernel funcs are called via BPF_CALL.
And this bpf insn has 32-bit offset requirement across archs. So all callable
functions have to be in the same 4G region. So far that was the case
for all archs.
If riscv is going to separate things by more than 4G it will cause
plenty of headaches
for riscv JIT.

Re: BPF calls to modules?

[Alex Ghiti]

Le 22/06/2021 à 19:25, Alexei Starovoitov a écrit :
> On Tue, Jun 22, 2021 at 12:31 AM Alex Ghiti <alex@ghiti.fr> wrote:
>>
>> Hi Alexei,
>>
>> Le 22/06/2021 à 02:28, Alexei Starovoitov a écrit :
>>> On Sun, Jun 20, 2021 at 11:43 PM Alex Ghiti <alex@ghiti.fr> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Le 18/06/2021 à 19:32, Andrii Nakryiko a écrit :
>>>>> On Fri, Jun 18, 2021 at 2:13 AM Alex Ghiti <alex@ghiti.fr> wrote:
>>>>>>
>>>>>> Hi guys,
>>>>>>
>>>>>> First, pardon my ignorance regarding BPF, the following might be silly.
>>>>>>
>>>>>> We were wondering here
>>>>>> https://patchwork.kernel.org/project/linux-riscv/patch/20210615004928.2d27d2ac@xhacker/
>>>>>> if BPF programs that now have the capability to call kernel functions
>>>>>> (https://lwn.net/Articles/856005/) can also call modules function or
>>>>>> vice-versa?
>>>>>
>>>>> Not yet, but it was an explicit design consideration and there was
>>>>> public interest just recently. So I'd say this is going to happen
>>>>> sooner rather than later.
>>>>>
>>>>>>
>>>>>> The underlying important fact is that in riscv, we are limited to 2GB
>>>>>> offset to call functions and that restricts where we can place modules
>>>>>> and BPF regions wrt kernel (see Documentation/riscv/vm-layout.rst for
>>>>>> the current possibly wrong layout).
>>>>>>
>>>>>> So should we make sure that modules and BPF lie in the same 2GB region?
>>>>>
>>>>> Based on the above and what you are explaining about 2GB limits, I'd
>>>>> say yes?.. Or alternatively those 2GB restrictions might perhaps be
>>>>> lifted somehow?
>>>>
>>>>
>>>> Actually we have this limit when we have PC-relative branch which is our
>>>> current code model. To better understand what happened, I took a look at
>>>> our JIT implementation and noticed that BPF_CALL are implemented using
>>>> absolute addressing so for this pseudo-instruction, the limit I evoked
>>>> does not apply. How are the kernel (and modules) symbol addresses
>>>> resolved? Is it relative or absolute? Is there then any guarantee that a
>>>> kernel or module call will always emit a BPF_CALL?
>>>
>>> Are those questions for riscv bpf JIT experts?
>>
>> Yes more or less, sorry about that, I added Bjorn in cc in case he wants
>> to intervene. But I think my last question is relevant: Is there then
>> any guarantee that a kernel or module call will always emit a BPF_CALL?
>> Because that would mean that we don't need to place BPF close to modules
>> since BPF_CALL are JITed into an absolute branch in riscv.
> 
> I don't understand what you mean with this question.
> BPF_CALL is a BPF instruction to call from bpf prog. Not into bpf prog.
> When kernel or module calls into JITed bpf prog
> they use indirect call insn of the given arch.

So a call to BPF program from kernel/module function is done using an 
indirect call. I have to check how this is actually done in riscv.

> In case of bpf dispatcher there is a generated asm code that uses jmp
> by register
> or retpoline style.
> So JITed bpf progs not only 'called' into.
>  From bpf prog the helpers and kernel funcs are called via BPF_CALL.

And this answers my question (I admit my phrasing may have been a bit 
confusing): a call to a kernel/module function from within a BPF program 
is done using a BPF_CALL.

> And this bpf insn has 32-bit offset requirement across archs. So all callable
> functions have to be in the same 4G region. So far that was the case
> for all archs.
> If riscv is going to separate things by more than 4G it will cause
> plenty of headaches
> for riscv JIT.
> 

No no, don't worry, we already are in the same 4G region, the question 
is do we need to be in the same *2G* region. I have enough material to 
continue my investigation.

Thank you very much for your time,

Alex