Dropped packets mapping IRQs for adjusted queue counts on i40e

[Zvi Effron <zeffron@riotgames.com>]

Hi-ho, friends!

I'm trying to tune network queues so that there's one queue per core,
as recommended in many, many places. I've noticed that if I adjust the
number of queues from the default AND adjust the IRQ affinities, then
I get some percentage (varying from a small percent to 100%, seemingly
proportional to the number of reduced queues) of packets not making it
through.

For my setup, I have a single 36 core Skylake processor with two dual
port X710 NICs. All traffic coming in one one port of each NIC is
redirected out the other port (traffic arrives on all four ports). 34
cores are isolated for network processing. I adjust the combined
queues from the default of 36 down, and then map the IRQ associated
with each queue to one of the 34 isolated cores. Everything works fine
if I don't map the IRQs.

For a minimum repro case, I reduced my program (reproduced below) to a
blind redirector using a devmap (it doesn't even adjust MACs, which is
not a problem as my DUT is directly connected to a measurement device
in promiscuous mode) reproduced below. I use bpftool to load 4 copies
of the program and pin them, use bpftool to configure the egress
interface in the devmap, and then use ip link to attach the programs
to the interfaces.

I have played around with when I adjust the queue counts and IRQs
(before attaching XDP programs, after, XDP attachment in the middle,
etc.) and it doesn't seem to matter. But with any ordering, if I just
don't remap the IRQs, everything works fine, and if I remap, I lose
packets.

Has anyone encountered anything like this? Does anyone know what might
be causing it? How can I assign a single queue to a single core
without using the default number of queues and without losing packets?

Thanks!
--Zvi

#include <linux/bpf.h>

struct bpf_map_def {
    unsigned int type;
    unsigned int key_size;
    unsigned int value_size;
    unsigned int max_entries;
    unsigned int map_flags;
    struct bpf_map_def* inner_map;
};

struct bpf_map_def __attribute__((section("maps"), used)) device_map = {
    .type = BPF_MAP_TYPE_DEVMAP,
    .key_size = sizeof(__u32),
    .value_size = sizeof(__u32),
    .max_entries = 1,
};

static int (*bpf_redirect_map)(void *map, int key, int flags) = (void
*)BPF_FUNC_redirect_map;

__attribute__((section("xdp/test"), used))
int test(struct xdp_md *context) {
    __u32 key = 0;
    bpf_redirect_map(&device_map, key, 0);
    return XDP_REDIRECT;
}