packet seems disappeared after vxlan_rcv/gro_cells_receive/napi_schedule(&cell->napi)

packet seems disappeared after vxlan_rcv/gro_cells_receive/napi_schedule(&cell->napi)

[Vincent Li]

Hi Experts,

I am doing a tunnel communication  test between Cilium eBPF tunnel and
Linux VXLAN vni key based tunnel device (to simulate BIG-IP VXLAN vni
key based device), https://github.com/cilium/cilium/issues/16462

I came across a problem when packet is handled by
vxlan_rcv->gro_cells_receive->napi_schedule(&cell->napi),  the packet
seems getting dropped somewhere after that. I suspect I might have
done something wrong to setup the VXLAN device.

Here is how I setup the vxlan device  test on my Centos 8 with most
recent mainline git kernel build (5.13.0-rc4+ )

====start of the script====
#!/bin/bash


ip link add vxlan666 type vxlan id 666 dstport 8472 local
10.169.72.236 dev ens192 nolearning l2miss l3miss proxy

ip link set vxlan666 up

ip a add 10.0.4.236/24 dev vxlan666

ip route add 10.0.1.0/24 dev vxlan666  proto kernel  scope link  src 10.0.4.236

arp -i vxlan666 -s 10.0.1.17 6a:29:d2:78:63:7d

bridge fdb append 6a:29:d2:78:63:7d dst 10.169.72.238 dev vxlan666

====end of the script====

then I run tcpdump in the background

#tcpdump -nn -e -i vxlan666 &

and start to ping the IP 10.0.1.17 which is a POD IP in Cilium managed
K8S cluster

#ping -c 1 10.0.1.17


PING 10.0.1.17 (10.0.1.17) 56(84) bytes of data.

19:06:44.452994 d6:04:7c:b2:93:54 > 6a:29:d2:78:63:7d, ethertype IPv4
(0x0800), length 98: 10.0.4.236 > 10.0.1.17: ICMP echo request, id
1522, seq 1, length 64

19:06:44.454515 56:3d:9c:3a:09:78 > b2:1c:3c:57:9e:97, ethertype IPv4
(0x0800), length 98: 10.0.1.17 > 10.0.4.236: ICMP echo reply, id 1522,
seq 1, length 64


^C

--- 10.0.1.17 ping statistics ---

1 packets transmitted, 0 received, 100% packet loss, time 0ms

You can see the tcpdump shows ICMP echo reply, but ping did not get
the ICMP echo reply  and shows 100% packet loss.

I added netdev_info log below and I can see the kernel log:

@@ -35,13 +39,17 @@ int gro_cells_receive(struct gro_cells *gcells,
struct sk_buff *skb)

        }


        __skb_queue_tail(&cell->napi_skbs, skb);
-       if (skb_queue_len(&cell->napi_skbs) == 1)
+       if (skb_queue_len(&cell->napi_skbs) == 1) {
+               netdev_info(skb->dev, "gro_cells_receive: napi_schedule\n");
                napi_schedule(&cell->napi);
+       }
+       netdev_info(skb->dev, "gro_cells_receive: NET_RX_SUCCESS\n");
        res = NET_RX_SUCCESS;

Jun  9 19:06:44 kernel-dev kernel: vxlan666: gro_cells_receive: napi_schedule

Jun  9 19:06:44 kernel-dev kernel: vxlan666: gro_cells_receive: NET_RX_SUCCESS


 I don't know where I have done wrong, any help is appreciated ! :)

Re: packet seems disappeared after vxlan_rcv/gro_cells_receive/napi_schedule(&cell->napi)

[Vincent Li]

Just reply myself for conclusion, I figured out the problem in
https://github.com/cilium/cilium/issues/16462#issuecomment-859680801

On Wed, Jun 9, 2021 at 4:50 PM Vincent Li <vincent.mc.li@gmail.com> wrote:
>
> Hi Experts,
>
> I am doing a tunnel communication  test between Cilium eBPF tunnel and
> Linux VXLAN vni key based tunnel device (to simulate BIG-IP VXLAN vni
> key based device), https://github.com/cilium/cilium/issues/16462
>
> I came across a problem when packet is handled by
> vxlan_rcv->gro_cells_receive->napi_schedule(&cell->napi),  the packet
> seems getting dropped somewhere after that. I suspect I might have
> done something wrong to setup the VXLAN device.
>
> Here is how I setup the vxlan device  test on my Centos 8 with most
> recent mainline git kernel build (5.13.0-rc4+ )
>
> ====start of the script====
> #!/bin/bash
>
>
> ip link add vxlan666 type vxlan id 666 dstport 8472 local
> 10.169.72.236 dev ens192 nolearning l2miss l3miss proxy
>
> ip link set vxlan666 up
>
> ip a add 10.0.4.236/24 dev vxlan666
>
> ip route add 10.0.1.0/24 dev vxlan666  proto kernel  scope link  src 10.0.4.236
>
> arp -i vxlan666 -s 10.0.1.17 6a:29:d2:78:63:7d
>
> bridge fdb append 6a:29:d2:78:63:7d dst 10.169.72.238 dev vxlan666
>
> ====end of the script====
>
> then I run tcpdump in the background
>
> #tcpdump -nn -e -i vxlan666 &
>
> and start to ping the IP 10.0.1.17 which is a POD IP in Cilium managed
> K8S cluster
>
> #ping -c 1 10.0.1.17
>
>
> PING 10.0.1.17 (10.0.1.17) 56(84) bytes of data.
>
> 19:06:44.452994 d6:04:7c:b2:93:54 > 6a:29:d2:78:63:7d, ethertype IPv4
> (0x0800), length 98: 10.0.4.236 > 10.0.1.17: ICMP echo request, id
> 1522, seq 1, length 64
>
> 19:06:44.454515 56:3d:9c:3a:09:78 > b2:1c:3c:57:9e:97, ethertype IPv4
> (0x0800), length 98: 10.0.1.17 > 10.0.4.236: ICMP echo reply, id 1522,
> seq 1, length 64
>
>
> ^C
>
> --- 10.0.1.17 ping statistics ---
>
> 1 packets transmitted, 0 received, 100% packet loss, time 0ms
>
> You can see the tcpdump shows ICMP echo reply, but ping did not get
> the ICMP echo reply  and shows 100% packet loss.
>
> I added netdev_info log below and I can see the kernel log:
>
> @@ -35,13 +39,17 @@ int gro_cells_receive(struct gro_cells *gcells,
> struct sk_buff *skb)
>
>         }
>
>
>         __skb_queue_tail(&cell->napi_skbs, skb);
> -       if (skb_queue_len(&cell->napi_skbs) == 1)
> +       if (skb_queue_len(&cell->napi_skbs) == 1) {
> +               netdev_info(skb->dev, "gro_cells_receive: napi_schedule\n");
>                 napi_schedule(&cell->napi);
> +       }
> +       netdev_info(skb->dev, "gro_cells_receive: NET_RX_SUCCESS\n");
>         res = NET_RX_SUCCESS;
>
> Jun  9 19:06:44 kernel-dev kernel: vxlan666: gro_cells_receive: napi_schedule
>
> Jun  9 19:06:44 kernel-dev kernel: vxlan666: gro_cells_receive: NET_RX_SUCCESS
>
>
>  I don't know where I have done wrong, any help is appreciated ! :)