Interruption in eBPF program after kernel update

Interruption in eBPF program after kernel update

[Eric Leblond]

Hello,

I've got a simple socket filter program that seems to interrupt when
calling the load_word construct. I'm seeing the first message but never
the second one.

The code is the following 

int SEC("filter") hashfilter(struct __sk_buff *skb) {
    __u32 nhoff = BPF_LL_OFF + ETH_HLEN;
    __u32 ip = 0;
    __u32 *value = NULL;
#if 1
    char fmtrei[] = "word load\n";
    bpf_trace_printk(fmtrei, sizeof(fmtrei));
#endif
    ip = load_word(skb, nhoff + offsetof(struct iphdr, saddr));
#if 1
    char fmti[] = "work done\n";
    bpf_trace_printk(fmti, sizeof(fmti));
#endif

In 4.18 and above this is not working but older kernel were ok.

Any suggestion will be appreciated.

Best regards,
-- 
Eric Leblond <eric@regit.org>

Re: Interruption in eBPF program after kernel update

[Eric Leblond]

Hello,

On Wed, 2019-02-13 at 20:46 +0100, Eric Leblond wrote:
> Hello,
> 
> I've got a simple socket filter program that seems to interrupt when
> calling the load_word construct. I'm seeing the first message but
> never
> the second one.
> 
> The code is the following 
> 
> int SEC("filter") hashfilter(struct __sk_buff *skb) {
>     __u32 nhoff = BPF_LL_OFF + ETH_HLEN;

So it seems the problem was coming from this line. Removing BPF_LL_OFF
fixes the problem for kernel starting 4.18.

Is this a known issue ?

Is there something that would allow a BPF file to work independently of
the kernel version ?

>     __u32 ip = 0;
>     __u32 *value = NULL;
> #if 1
>     char fmtrei[] = "word load\n";
>     bpf_trace_printk(fmtrei, sizeof(fmtrei));
> #endif
>     ip = load_word(skb, nhoff + offsetof(struct iphdr, saddr));
> #if 1
>     char fmti[] = "work done\n";
>     bpf_trace_printk(fmti, sizeof(fmti));
> #endif

> In 4.18 and above this is not working but older kernel were ok.
> 
> Any suggestion will be appreciated.
> 
> Best regards,

BR,
-- 
Eric Leblond <eric@regit.org>

Re: Interruption in eBPF program after kernel update

[Alexei Starovoitov]

On Thu, Feb 14, 2019 at 10:33 AM Eric Leblond <eric@regit.org> wrote:
>
> Hello,
>
> On Wed, 2019-02-13 at 20:46 +0100, Eric Leblond wrote:
> > Hello,
> >
> > I've got a simple socket filter program that seems to interrupt when
> > calling the load_word construct. I'm seeing the first message but
> > never
> > the second one.
> >
> > The code is the following
> >
> > int SEC("filter") hashfilter(struct __sk_buff *skb) {
> >     __u32 nhoff = BPF_LL_OFF + ETH_HLEN;
>
> So it seems the problem was coming from this line. Removing BPF_LL_OFF
> fixes the problem for kernel starting 4.18.
>
> Is this a known issue ?
>
> Is there something that would allow a BPF file to work independently of
> the kernel version ?

Daniel,
can this be related to removal of ld_abs legacy ?

Re: Interruption in eBPF program after kernel update

[Y Song]

On Thu, Feb 14, 2019 at 10:33 AM Eric Leblond <eric@regit.org> wrote:
>
> Hello,
>
> On Wed, 2019-02-13 at 20:46 +0100, Eric Leblond wrote:
> > Hello,
> >
> > I've got a simple socket filter program that seems to interrupt when
> > calling the load_word construct. I'm seeing the first message but
> > never
> > the second one.
> >
> > The code is the following
> >
> > int SEC("filter") hashfilter(struct __sk_buff *skb) {
> >     __u32 nhoff = BPF_LL_OFF + ETH_HLEN;
>
> So it seems the problem was coming from this line. Removing BPF_LL_OFF
> fixes the problem for kernel starting 4.18.

Maybe removing BPF_LL_OFF should work for all kernels?

>
> Is this a known issue ?
>
> Is there something that would allow a BPF file to work independently of
> the kernel version ?
>
> >     __u32 ip = 0;
> >     __u32 *value = NULL;
> > #if 1
> >     char fmtrei[] = "word load\n";
> >     bpf_trace_printk(fmtrei, sizeof(fmtrei));
> > #endif
> >     ip = load_word(skb, nhoff + offsetof(struct iphdr, saddr));
> > #if 1
> >     char fmti[] = "work done\n";
> >     bpf_trace_printk(fmti, sizeof(fmti));
> > #endif
>
> > In 4.18 and above this is not working but older kernel were ok.
> >
> > Any suggestion will be appreciated.
> >
> > Best regards,
>
> BR,
> --
> Eric Leblond <eric@regit.org>
>

Re: Interruption in eBPF program after kernel update

[Eric Leblond]

Hello,

On Thu, 2019-02-14 at 14:39 -0800, Y Song wrote:
> On Thu, Feb 14, 2019 at 10:33 AM Eric Leblond <eric@regit.org> wrote:
> > Hello,
> > 
> > On Wed, 2019-02-13 at 20:46 +0100, Eric Leblond wrote:
> > > Hello,
> > > 
> > > I've got a simple socket filter program that seems to interrupt
> > > when
> > > calling the load_word construct. I'm seeing the first message but
> > > never
> > > the second one.
> > > 
> > > The code is the following
> > > 
> > > int SEC("filter") hashfilter(struct __sk_buff *skb) {
> > >     __u32 nhoff = BPF_LL_OFF + ETH_HLEN;
> > 
> > So it seems the problem was coming from this line. Removing
> > BPF_LL_OFF
> > fixes the problem for kernel starting 4.18.
> 
> Maybe removing BPF_LL_OFF should work for all kernels?

OK, you are right. I really don't understand how it is possible but
same eBPF file without BPF_LL_OFF work on 4.17 at 4.19. Looks like
BPF_LL_OFF was just ignored in 4.17 but caused a problem on 4.19.

Thanks a lot.

BR,

> > Is this a known issue ?
> > 
> > Is there something that would allow a BPF file to work
> > independently of
> > the kernel version ?
> > 
> > >     __u32 ip = 0;
> > >     __u32 *value = NULL;
> > > #if 1
> > >     char fmtrei[] = "word load\n";
> > >     bpf_trace_printk(fmtrei, sizeof(fmtrei));
> > > #endif
> > >     ip = load_word(skb, nhoff + offsetof(struct iphdr, saddr));
> > > #if 1
> > >     char fmti[] = "work done\n";
> > >     bpf_trace_printk(fmti, sizeof(fmti));
> > > #endif
> > > In 4.18 and above this is not working but older kernel were ok.
> > > 
> > > Any suggestion will be appreciated.
> > > 
> > > Best regards,
> > 
> > BR,
> > --
> > Eric Leblond <eric@regit.org>
> > 
-- 
Eric Leblond <eric@regit.org>

Re: Interruption in eBPF program after kernel update

[Daniel Borkmann]

On 02/14/2019 10:37 PM, Alexei Starovoitov wrote:
> On Thu, Feb 14, 2019 at 10:33 AM Eric Leblond <eric@regit.org> wrote:
>>
>> Hello,
>>
>> On Wed, 2019-02-13 at 20:46 +0100, Eric Leblond wrote:
>>> Hello,
>>>
>>> I've got a simple socket filter program that seems to interrupt when
>>> calling the load_word construct. I'm seeing the first message but
>>> never
>>> the second one.
>>>
>>> The code is the following
>>>
>>> int SEC("filter") hashfilter(struct __sk_buff *skb) {
>>>     __u32 nhoff = BPF_LL_OFF + ETH_HLEN;
>>
>> So it seems the problem was coming from this line. Removing BPF_LL_OFF
>> fixes the problem for kernel starting 4.18.
>>
>> Is this a known issue ?
>>
>> Is there something that would allow a BPF file to work independently of
>> the kernel version ?
> 
> Daniel,
> can this be related to removal of ld_abs legacy ?

Hm, strange, I'll take a look.

Thanks,
Daniel

Re: Interruption in eBPF program after kernel update

[Daniel Borkmann]

Hi Eric,

On 02/15/2019 02:07 PM, Daniel Borkmann wrote:
> On 02/14/2019 10:37 PM, Alexei Starovoitov wrote:
>> On Thu, Feb 14, 2019 at 10:33 AM Eric Leblond <eric@regit.org> wrote:
>>>
>>> Hello,
>>>
>>> On Wed, 2019-02-13 at 20:46 +0100, Eric Leblond wrote:
>>>> Hello,
>>>>
>>>> I've got a simple socket filter program that seems to interrupt when
>>>> calling the load_word construct. I'm seeing the first message but
>>>> never
>>>> the second one.
>>>>
>>>> The code is the following
>>>>
>>>> int SEC("filter") hashfilter(struct __sk_buff *skb) {
>>>>     __u32 nhoff = BPF_LL_OFF + ETH_HLEN;
>>>
>>> So it seems the problem was coming from this line. Removing BPF_LL_OFF
>>> fixes the problem for kernel starting 4.18.
>>>
>>> Is this a known issue ?
>>>
>>> Is there something that would allow a BPF file to work independently of
>>> the kernel version ?
>>
>> Daniel,
>> can this be related to removal of ld_abs legacy ?
> 
> Hm, strange, I'll take a look.

Do you have a concrete stand-alone reproducer? Works fine for me on
latest bpf tree at least (internal code gen is same for filter and
tc BPF) ...

# pwd
/root/iproute2/examples/bpf
# cat foo.c
#include <stdio.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include "../../include/bpf_api.h"

__section_cls_entry
int cls_entry(struct __sk_buff *skb)
{
    __u32 nhoff = BPF_LL_OFF + ETH_HLEN;
    __u32 ip = 0;
    __u32 *value = NULL;
#if 1
    char fmtrei[] = "word load\n";
    trace_printk(fmtrei, sizeof(fmtrei));
#endif
    ip = load_word(skb, nhoff + offsetof(struct iphdr, saddr));
#if 1
    char fmti[] = "work done\n";
    trace_printk(fmti, sizeof(fmti));
#endif
    return TC_ACT_OK;
}

BPF_LICENSE("GPL");
# clang -target bpf -O2 -Wall -c foo.c -o foo.o -I ../../include/uapi/
# tc qdisc add dev bond0 clsact
# tc filter add dev bond0 ingress bpf da obj ./foo.o
# tc exec bpf dbg
[...]
          <idle>-0     [005] .Ns. 21102.794996: 0: work done
          <idle>-0     [005] ..s. 21102.796023: 0: word load
          <idle>-0     [005] .Ns. 21102.796045: 0: work done
          <idle>-0     [005] ..s. 21102.807417: 0: word load
          <idle>-0     [005] .Ns. 21102.807438: 0: work done
          <idle>-0     [005] ..s. 21102.808365: 0: word load
          <idle>-0     [005] .Ns. 21102.808387: 0: work done
          <idle>-0     [005] ..s. 21102.819905: 0: word load
          <idle>-0     [005] .Ns. 21102.819923: 0: work done
          <idle>-0     [005] ..s. 21102.820626: 0: word load
          <idle>-0     [005] .Ns. 21102.820646: 0: work done
          <idle>-0     [005] ..s. 21102.832293: 0: word load
          <idle>-0     [005] .Ns. 21102.832314: 0: work done
[...]
# uname -a
Linux test2 5.0.0-rc6+ #1 SMP Mon Feb 18 13:06:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
# llc --version
[...]
  LLVM version 3.8.0
# cat /proc/sys/net/core/bpf_jit_enable
1

Thanks,
Daniel

Re: Interruption in eBPF program after kernel update

[Daniel Borkmann]

Hi Eric,

On 02/18/2019 10:51 PM, Daniel Borkmann wrote:
> On 02/15/2019 02:07 PM, Daniel Borkmann wrote:
>> On 02/14/2019 10:37 PM, Alexei Starovoitov wrote:
>>> On Thu, Feb 14, 2019 at 10:33 AM Eric Leblond <eric@regit.org> wrote:
>>>>
>>>> Hello,
>>>>
>>>> On Wed, 2019-02-13 at 20:46 +0100, Eric Leblond wrote:
>>>>> Hello,
>>>>>
>>>>> I've got a simple socket filter program that seems to interrupt when
>>>>> calling the load_word construct. I'm seeing the first message but
>>>>> never
>>>>> the second one.
>>>>>
>>>>> The code is the following
>>>>>
>>>>> int SEC("filter") hashfilter(struct __sk_buff *skb) {
>>>>>     __u32 nhoff = BPF_LL_OFF + ETH_HLEN;
>>>>
>>>> So it seems the problem was coming from this line. Removing BPF_LL_OFF
>>>> fixes the problem for kernel starting 4.18.
>>>>
>>>> Is this a known issue ?
>>>>
>>>> Is there something that would allow a BPF file to work independently of
>>>> the kernel version ?
>>>
>>> Daniel,
>>> can this be related to removal of ld_abs legacy ?
>>
>> Hm, strange, I'll take a look.
> 
> Do you have a concrete stand-alone reproducer? Works fine for me on
> latest bpf tree at least (internal code gen is same for filter and
> tc BPF) ...
> 
> # pwd
> /root/iproute2/examples/bpf
> # cat foo.c
> #include <stdio.h>
> #include <linux/if_ether.h>
> #include <linux/ip.h>
> #include "../../include/bpf_api.h"
> 
> __section_cls_entry
> int cls_entry(struct __sk_buff *skb)
> {
>     __u32 nhoff = BPF_LL_OFF + ETH_HLEN;
>     __u32 ip = 0;
>     __u32 *value = NULL;
> #if 1
>     char fmtrei[] = "word load\n";
>     trace_printk(fmtrei, sizeof(fmtrei));
> #endif
>     ip = load_word(skb, nhoff + offsetof(struct iphdr, saddr));
> #if 1
>     char fmti[] = "work done\n";
>     trace_printk(fmti, sizeof(fmti));
> #endif
>     return TC_ACT_OK;
> }
> 
> BPF_LICENSE("GPL");
> # clang -target bpf -O2 -Wall -c foo.c -o foo.o -I ../../include/uapi/
> # tc qdisc add dev bond0 clsact
> # tc filter add dev bond0 ingress bpf da obj ./foo.o
> # tc exec bpf dbg
> [...]
>           <idle>-0     [005] .Ns. 21102.794996: 0: work done
>           <idle>-0     [005] ..s. 21102.796023: 0: word load
>           <idle>-0     [005] .Ns. 21102.796045: 0: work done
>           <idle>-0     [005] ..s. 21102.807417: 0: word load
>           <idle>-0     [005] .Ns. 21102.807438: 0: work done
>           <idle>-0     [005] ..s. 21102.808365: 0: word load
>           <idle>-0     [005] .Ns. 21102.808387: 0: work done
>           <idle>-0     [005] ..s. 21102.819905: 0: word load
>           <idle>-0     [005] .Ns. 21102.819923: 0: work done
>           <idle>-0     [005] ..s. 21102.820626: 0: word load
>           <idle>-0     [005] .Ns. 21102.820646: 0: work done
>           <idle>-0     [005] ..s. 21102.832293: 0: word load
>           <idle>-0     [005] .Ns. 21102.832314: 0: work done
> [...]
> # uname -a
> Linux test2 5.0.0-rc6+ #1 SMP Mon Feb 18 13:06:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> # llc --version
> [...]
>   LLVM version 3.8.0
> # cat /proc/sys/net/core/bpf_jit_enable
> 1

Small ping, can you confirm the same on your side? If not, do you
have a reproducer?

Thanks,
Daniel