(no subject)

[Topi Wala]

I have a setup that has 2 namespaces, connected using a linux bridge,
with veth pairs in each of the namespace.

ns1=192.168.1.10/24
ns2=192.168.1.11/24
host-br=192.168.1.1/24

I can ping between host, ns1, ns2 fine.

I'm attaching an xdp filter program
https://github.com/xdp-project/xdp-tools/blob/master/xdp-filter/xdpfilt_dny_ip.c

I'm using bpftool to attach this to ns1-host end. I also attach a
dummy xdp prog (that just returns XDP_PASS) to the end inside the ns1.
I see all ping packets to this destination dropped. Dumping
xdp_stats_map does show counters incremented for XDP_DROP

However, when using bpftool to update the filter_ipv4 map to allow
packets with destination to go through, it doesn't work.

./bpftool map update name filter_ipv4 key 192 168 1 10 value 2 0 0 0 0 0 0 0

I've tried with pinned maps, and different combinations of key/value
as well, to no avail. The lookup just doesn't seem to succeed. Any
suggestions on how I might go about debugging this?

--------

Update:

I did try with bpf_printk to see what was going on, and there seems to
be some really weird issue that happens after the bpf map is updated.

So, to keep things simple, I attached the xdp filter program to my
host bridge interface. Pinging the bridge address from either
namespace drops the packet, AND my printk message is logged and I can
read it from /sys/kernel/debug/tracing/trace_pipe

I insert entries into the map, and then when I try to do the same, not
only does it not work, there is no printk message either. Removing
these entries still does not get the printk message back.

How do I go about debugging this? Are there any known issues with
using maps that are not pinned (I have tried with pinning them, but
didn't debug that setup deeply).

Thanks.
Topi

Re: xdp-filter troubles

[Toke Høiland-Jørgensen]

[ adding a subject - please make sure to include one in the future ]

> I have a setup that has 2 namespaces, connected using a linux bridge,
> with veth pairs in each of the namespace.
>
> ns1=192.168.1.10/24
> ns2=192.168.1.11/24
> host-br=192.168.1.1/24
>
> I can ping between host, ns1, ns2 fine.
>
> I'm attaching an xdp filter program
> https://github.com/xdp-project/xdp-tools/blob/master/xdp-filter/xdpfilt_dny_ip.c
>
> I'm using bpftool to attach this to ns1-host end. I also attach a
> dummy xdp prog (that just returns XDP_PASS) to the end inside the ns1.
> I see all ping packets to this destination dropped. Dumping
> xdp_stats_map does show counters incremented for XDP_DROP
>
> However, when using bpftool to update the filter_ipv4 map to allow
> packets with destination to go through, it doesn't work.
>
> ./bpftool map update name filter_ipv4 key 192 168 1 10 value 2 0 0 0 0 0 0 0
>
> I've tried with pinned maps, and different combinations of key/value
> as well, to no avail. The lookup just doesn't seem to succeed. Any
> suggestions on how I might go about debugging this?

What kernel version are you using? And how are you attaching the program
- from your description I'm guessing you may be using generic XDP? Also,
why are you using bpftool to load the program instead of just using the
xdp-filter binary?

-Toke

Re: xdp-filter troubles

[Topi Wala]

Sorry about the subject. I had to forward after using text (default is
http in gmail).
My kernel version is 5.7.17 (quite new). I am using xdp (ip link show
on the device shows the xdp, not xdpgeneric).
I'm using bpftool, since this is to deploy to a third party, and using
standard linux tools (bpftool, iproute2) is fine, userspace binaries
are not.
Is there anything I miss by not using the xdp-filter binary?

However, I think I have figured out the issue, and I'm quite surprised by it.
It appears that as soon as the xdpfilter program is attached, the arp
entry for that IP is discarded from all the entities connected to that
bridge.
The peer node consequently does an ARP before ping, and since ARP is
not an IP packet, it doesn't show up in the bpf log.
Since the remote node doesn't get an ARP response (arp request is
dropped), it doesn't send the ping packet at all.

Thanks,
Hari



On Mon, Sep 21, 2020 at 2:44 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> [ adding a subject - please make sure to include one in the future ]
>
> > I have a setup that has 2 namespaces, connected using a linux bridge,
> > with veth pairs in each of the namespace.
> >
> > ns1=192.168.1.10/24
> > ns2=192.168.1.11/24
> > host-br=192.168.1.1/24
> >
> > I can ping between host, ns1, ns2 fine.
> >
> > I'm attaching an xdp filter program
> > https://github.com/xdp-project/xdp-tools/blob/master/xdp-filter/xdpfilt_dny_ip.c
> >
> > I'm using bpftool to attach this to ns1-host end. I also attach a
> > dummy xdp prog (that just returns XDP_PASS) to the end inside the ns1.
> > I see all ping packets to this destination dropped. Dumping
> > xdp_stats_map does show counters incremented for XDP_DROP
> >
> > However, when using bpftool to update the filter_ipv4 map to allow
> > packets with destination to go through, it doesn't work.
> >
> > ./bpftool map update name filter_ipv4 key 192 168 1 10 value 2 0 0 0 0 0 0 0
> >
> > I've tried with pinned maps, and different combinations of key/value
> > as well, to no avail. The lookup just doesn't seem to succeed. Any
> > suggestions on how I might go about debugging this?
>
> What kernel version are you using? And how are you attaching the program
> - from your description I'm guessing you may be using generic XDP? Also,
> why are you using bpftool to load the program instead of just using the
> xdp-filter binary?
>
> -Toke
>

Re: xdp-filter troubles

[Toke Høiland-Jørgensen]

Topi Wala <walatopi@gmail.com> writes:

> Sorry about the subject. I had to forward after using text (default is
> http in gmail).
> My kernel version is 5.7.17 (quite new). I am using xdp (ip link show
> on the device shows the xdp, not xdpgeneric).
> I'm using bpftool, since this is to deploy to a third party, and using
> standard linux tools (bpftool, iproute2) is fine, userspace binaries
> are not.

Heh, that's an interesting policy - I'm guessing whoever set it doesn't
really understand BPF... :)

> Is there anything I miss by not using the xdp-filter binary?

Well, a nicer user interface?
But no, technically the BPF side should work just fine, as long as you
populate the maps correctly. Just don't expect any future changes to
xdp-filter to take your... interesting... use case into account :)

> However, I think I have figured out the issue, and I'm quite surprised by it.
> It appears that as soon as the xdpfilter program is attached, the arp
> entry for that IP is discarded from all the entities connected to that
> bridge.
> The peer node consequently does an ARP before ping, and since ARP is
> not an IP packet, it doesn't show up in the bpf log.
> Since the remote node doesn't get an ARP response (arp request is
> dropped), it doesn't send the ping packet at all.

Hmm, installing an XDP program does call the netdev notifiers, so I
guess the iface state can flip, causing the neighbour entries to be
evicted. In general, it's probably safest to consider loading an XDP
program to be a potentially destructive event (some physical devices
will even reconfigure the hardware when this happens). This is only on
first install, though, swapping between already-installed XDP programs
should not have such issues.

You're not the first person to run into the ARP behaviour, BTW. I guess
it may make sense to parse neighbour requests and include them in the IP
filter; feel free to open an issue in the xdp-tools repo for this (or
even better, a pull request!).

-Toke

Re: xdp-filter troubles

[Topi Wala]

You're not the first person to run into the ARP behaviour, BTW. I guess
it may make sense to parse neighbour requests and include them in the IP
filter; feel free to open an issue in the xdp-tools repo for this (or
even better, a pull request!).

Sure, will take a look. There are lots of control packets to be
worried about, especially in the IPv6 world.


On Mon, Sep 21, 2020 at 7:03 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> Topi Wala <walatopi@gmail.com> writes:
>
> > Sorry about the subject. I had to forward after using text (default is
> > http in gmail).
> > My kernel version is 5.7.17 (quite new). I am using xdp (ip link show
> > on the device shows the xdp, not xdpgeneric).
> > I'm using bpftool, since this is to deploy to a third party, and using
> > standard linux tools (bpftool, iproute2) is fine, userspace binaries
> > are not.
>
> Heh, that's an interesting policy - I'm guessing whoever set it doesn't
> really understand BPF... :)
>
> > Is there anything I miss by not using the xdp-filter binary?
>
> Well, a nicer user interface?
> But no, technically the BPF side should work just fine, as long as you
> populate the maps correctly. Just don't expect any future changes to
> xdp-filter to take your... interesting... use case into account :)
>
> > However, I think I have figured out the issue, and I'm quite surprised by it.
> > It appears that as soon as the xdpfilter program is attached, the arp
> > entry for that IP is discarded from all the entities connected to that
> > bridge.
> > The peer node consequently does an ARP before ping, and since ARP is
> > not an IP packet, it doesn't show up in the bpf log.
> > Since the remote node doesn't get an ARP response (arp request is
> > dropped), it doesn't send the ping packet at all.
>
> Hmm, installing an XDP program does call the netdev notifiers, so I
> guess the iface state can flip, causing the neighbour entries to be
> evicted. In general, it's probably safest to consider loading an XDP
> program to be a potentially destructive event (some physical devices
> will even reconfigure the hardware when this happens). This is only on
> first install, though, swapping between already-installed XDP programs
> should not have such issues.
>
> You're not the first person to run into the ARP behaviour, BTW. I guess
> it may make sense to parse neighbour requests and include them in the IP
> filter; feel free to open an issue in the xdp-tools repo for this (or
> even better, a pull request!).
>
> -Toke
>