AF_XDP Sockets Not Receiving Packets In XDP DRV Mode

[Christian Deacon <gamemann@gflclan.com>]

Hey everyone,


I believe I made a thread regarding this issue around a year ago or so, 
but I thought it was related to the `virtio_net` driver. This time, 
however, I am testing this on a dedicated server with a dedicated NIC 
(see information at bottom) and have the same issue.


I am trying to create an AF_XDP program and in this program, I currently 
spin up n AF_XDP sockets where n represents the RX queue count. Each 
AF_XDP socket is bound to a specific RX queue. From what I've seen, most 
examples recommend binding the socket to one RX queue and using flow 
steering via `ethtool` to steer all packets destined for the AF_XDP 
socket to the RX queue the AF_XDP socket is bound to. The two problems I 
see here are:


1. To my understanding, this NIC only supports matching on layer 2/3/4 
headers and with my AF_XDP program, I want to redirect packets to the 
AF_XDP sockets depending on the first 4 - 5 bytes of the payload.


2. It would be better for performance to spread these packets out to 
individual RX queues, even if they're destined for the AF_XDP sockets 
(please correct me if I am wrong here).


In the program, I create a separate thread for each AF_XDP socket and 
this thread polls that specific AF_XDP socket.


Here is the program's source code:


https://github.com/gamemann/AF_XDP-Test


When loading the XDP program in DRV mode, the packets destined for the 
AF_XDP sockets never reach the socket. However, when loaded in SKB mode, 
the packets do arrive. The return code of the `bpf_redirect_map()` 
function is the same for each mode (`4` which is successful I believe, 
XDP_REDIRECT). The AF_XDP sockets are also created successfully with 
each mode and I confirmed I receive a file descriptor (>0) from each 
AF_XDP socket.


The AF_XDP sockets do not share the same UMEM as well. Individual UMEMs 
are assigned to each AF_XDP socket.


Here is information regarding the Linux kernel and NIC.


Kernel:


```

cdeacon@xxxxxx:~/AF_XDP-Test$ uname -r
5.12.5-051205-generic
```


Note - I've had this same issue on other Linux kernels from 5.4 to 5.10. 
Though, that was on a VM with the `virtio_net` driver.


NIC information:


```

   *-network
        description: Ethernet interface
        physical id: 1
        bus info: pci@0000:04:00.0
        logical name: yyyyyyyy
        serial: xx:xx:xx:xx:xx
        capabilities: ethernet physical fibre
        configuration: autonegotiation=on broadcast=yes driver=nfp 
driverversion=5.12.5-051205-generic duplex=full firmware=0.0.3.5 0.22 
nic-2.1.16.1 nic ip=xxx.xxx.xxx.xxx link=yes multicast=yes port=fibre
```


NIC queues:


```

cdeacon@xxxxxx:~/AF_XDP-Test$ sudo ethtool -l yyyyyyyy
Channel parameters for yyyyyyyy:
Pre-set maximums:
RX:        16
TX:        16
Other:        2
Combined:    16
Current hardware settings:
RX:        0
TX:        8
Other:        2
Combined:    8
```


NIC settings/features:


```

cdeacon@xxxxxx:~/AF_XDP-Test$ sudo ethtool -k yyyyyyyy
Features for yyyyyyyy:
rx-checksumming: on
tx-checksumming: on
     tx-checksum-ipv4: on
     tx-checksum-ip-generic: off [fixed]
     tx-checksum-ipv6: on
     tx-checksum-fcoe-crc: off [fixed]
     tx-checksum-sctp: off [fixed]
scatter-gather: on
     tx-scatter-gather: on
     tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
     tx-tcp-segmentation: off
     tx-tcp-ecn-segmentation: off [fixed]
     tx-tcp-mangleid-segmentation: off
     tx-tcp6-segmentation: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]
```


 From what I've read below, setting up individual AF_XDP sockets on 
specific RX queues is supported, even with XDP DRV mode.


https://github.com/xdp-project/xdp-tutorial/tree/master/advanced03-AF_XDP#gotcha-by-rx-queue-id-binding


The AF_XDP test program I made is also based off of the AF_XDP lesson in 
the `xdp-tutorial` repository.


One last note I want to add is I've tried adding the AF_XDP socket's FD 
to the XSK map inside the userspace via `bpf_map_update_elem(xskmapfd, 
&rxqueue, &fd, BPF_ANY)` with no luck (also confirmed the map updated 
successfully).


Any help is highly appreciated and if you need any additional 
information, please let me know!


Thank you for your time.