xsk_fwd.c usage

xsk_fwd.c usage

[Ferenc Fejes]

Hi!

I wanted to try out the xsk_fwd.c [0] sample with Mininet. This sample
is capable of forwarding between network interfaces with AF_XDP. My
topology is very simple:
host1 <-----> switch <-----> host2
sudo mn --switch=lxbr
I started the xsk_fwd to forward between the two veth interface of the switch:
sudo ./xsk_fwd -i s1-eth1 -q0 -i s1-eth2 -q0 -c 0

This is failed with EOPNOTSUPP error first. Then I looked into the
xsk_bind function and soon after I found this (at there [1]):
if (force_copy)
    /* For copy-mode, we are done. */
    return 0;

if (!netdev->netdev_ops->ndo_bpf ||
    !netdev->netdev_ops->ndo_xsk_wakeup) {
    err = -EOPNOTSUPP;
    goto err_unreg_pool;
}

After that just like the code said, I removed the XDP_ZEROCOPY from
the bind flags [2] at the xsk_fwd.c which solved the problem and the
program started successfully, but without any traffic between host1
and host2.

My main question: removing XDP_ZEROCOPY requires any semantical
modification in the forwarding code? For example I should have to
memcopy the xdp buffer manually? Or the kernel do the copy
automatically and I have to look for the problem elsewhere? I have
5.10.5 kernel version which supports shared UMEMs.

[0] https://elixir.bootlin.com/linux/v5.11-rc3/source/samples/bpf/xsk_fwd.c
[1] https://elixir.bootlin.com/linux/v5.10.5/source/net/xdp/xsk_buff_pool.c#L158
[2] https://elixir.bootlin.com/linux/v5.11-rc3/source/samples/bpf/xsk_fwd.c#L732

Thanks for the help!
Ferenc

Re: xsk_fwd.c usage

[Ferenc Fejes]

Magnus Karlsson <magnus.karlsson@gmail.com> ezt írta (időpont: 2021.
jan. 14., Cs, 15:28):
>
>
>
> On Wed, Jan 13, 2021 at 9:32 PM Ferenc Fejes <fejes@inf.elte.hu> wrote:
>>
>> Hi!
>>
>> I wanted to try out the xsk_fwd.c [0] sample with Mininet. This sample
>> is capable of forwarding between network interfaces with AF_XDP. My
>> topology is very simple:
>> host1 <-----> switch <-----> host2
>> sudo mn --switch=lxbr
>> I started the xsk_fwd to forward between the two veth interface of the switch:
>> sudo ./xsk_fwd -i s1-eth1 -q0 -i s1-eth2 -q0 -c 0
>>
>> This is failed with EOPNOTSUPP error first. Then I looked into the
>> xsk_bind function and soon after I found this (at there [1]):
>> if (force_copy)
>>     /* For copy-mode, we are done. */
>>     return 0;
>>
>> if (!netdev->netdev_ops->ndo_bpf ||
>>     !netdev->netdev_ops->ndo_xsk_wakeup) {
>>     err = -EOPNOTSUPP;
>>     goto err_unreg_pool;
>> }
>>
>> After that just like the code said, I removed the XDP_ZEROCOPY from
>> the bind flags [2] at the xsk_fwd.c which solved the problem and the
>> program started successfully, but without any traffic between host1
>> and host2.
>>
>
> The xsk_fwd app was written with zero-copy in mind. We should probably add an "-S" option to it to start it in skb mode that should work with all netdev drivers. You need to change both these lines:
>
>                 .xdp_flags = XDP_FLAGS_DRV_MODE,
>                 .bind_flags = XDP_USE_NEED_WAKEUP | XDP_ZEROCOPY,
>
> XDP_FLAGS_DRV_MODE -> XDP_FLAGS_SKB_MODE
> XDP_ZEROCOPY -> XDP_COPY
>
> Please try this. If not, try simpler programs first and see if they work to get some hints.

Thank you for the quick answer! It turned out both SKB and DRV mode
works very well with veth devices. The XDP_COPY required however.

I finally managed to forward traffic between the namespaces: there is
a MAX_BURST_TX 64 define and I simply tested with ping (not some
traffic generator) and it required 64 sec to catch the first TX
transmission because of that. If I set MAX_BURST_RX and MAX_BURST_TX
to 1, ping works well (but TCP dont for some reason).

I working with DPDK for a while and I would like to do a migration to
AF_XDP because I love the fact that's already in the kernel and I only
have to install libbpf for make it work.

>
>>
>> My main question: removing XDP_ZEROCOPY requires any semantical
>> modification in the forwarding code? For example I should have to
>> memcopy the xdp buffer manually? Or the kernel do the copy
>> automatically and I have to look for the problem elsewhere? I have
>> 5.10.5 kernel version which supports shared UMEMs.
>
>
> From a programmatic perspective, it does not matter if you are using zero-copy or not. It will work the same way in both cases. The performance will be different of course.
>
Thank you for the answer, this is very convenient.
>>
>>
>> [0] https://elixir.bootlin.com/linux/v5.11-rc3/source/samples/bpf/xsk_fwd.c
>> [1] https://elixir.bootlin.com/linux/v5.10.5/source/net/xdp/xsk_buff_pool.c#L158
>> [2] https://elixir.bootlin.com/linux/v5.11-rc3/source/samples/bpf/xsk_fwd.c#L732
>>
>> Thanks for the help!
>> Ferenc