issues with vxlan RX checksum offload under OVS datapath

issues with vxlan RX checksum offload under OVS datapath

[Or Gerlitz]

Hi Pravin,

While testing the gro udp patches over a setup with openvswitch I noted 
that the RX checksum offload support introduced by Joseph's commit 
0afb166 "vxlan: Add capability of Rx checksum offload for inner packet" 
works fine when you use a setup made of

NIC --> IP stack --> vxlan device --> bridge --> tap

but not when its

NIC --> IP stack --> ovs vxlan port --> OVS DP --> tap

I narrowed it down to the fact the when going the OVS 
pathskb->encapsulation remains true also after the decap is done. 
Basically, this is the original hunk


> @@ -607,7 +607,17 @@ static int vxlan_udp_encap_recv(struct sock *sk, 
> struct sk_buff *skb)
>
>         __skb_tunnel_rx(skb, vxlan->dev);
>         skb_reset_network_header(skb);
> -       skb->ip_summed = CHECKSUM_NONE;
> +
> +       /* If the NIC driver gave us an encapsulated packet with
> +        * CHECKSUM_UNNECESSARY and Rx checksum feature is enabled,
> +        * leave the CHECKSUM_UNNECESSARY, the device checksummed it
> +        * for us. Otherwise force the upper layers to verify it.
> +        */
> +       if (skb->ip_summed != CHECKSUM_UNNECESSARY || 
> !skb->encapsulation ||
> +           !(vxlan->dev->features & NETIF_F_RXCSUM))
> +               skb->ip_summed = CHECKSUM_NONE;
> +
> +       skb->encapsulation = 0;

which was moved by your commits

5cfccc5 vxlan: Add vxlan recv demux.
7ce0475 vxlan: Restructure vxlan receive.

to a code which isn't shared by the vxlan driver and ovs vxlan datapath 
code, and I was thinking lets just move it back to vxlan_udp_encap_recv. 
However, we can't really make the "vxlan->dev->features & 
NETIF_F_RXCSUM" check - since there's no vxlan device for ovs datapath, 
any idea how to overcome this.

Moving this to shared code (while removing the check for 
vxlan->dev->features) made things to work on my setup, but this misses 
one of the original conditions, ideas?

I believe its too late for 3.13 unless there is going to be -rc9, but 
lets first see what's the right fix.

Or.

Re: issues with vxlan RX checksum offload under OVS datapath

[Pravin Shelar]

On Sun, Jan 19, 2014 at 2:05 PM, Or Gerlitz <ogerlitz@mellanox.com> wrote:
> Hi Pravin,
>
> While testing the gro udp patches over a setup with openvswitch I noted that
> the RX checksum offload support introduced by Joseph's commit 0afb166
> "vxlan: Add capability of Rx checksum offload for inner packet" works fine
> when you use a setup made of
>
> NIC --> IP stack --> vxlan device --> bridge --> tap
>
> but not when its
>
> NIC --> IP stack --> ovs vxlan port --> OVS DP --> tap
>
> I narrowed it down to the fact the when going the OVS pathskb->encapsulation
> remains true also after the decap is done. Basically, this is the original
> hunk
>
>
>> @@ -607,7 +607,17 @@ static int vxlan_udp_encap_recv(struct sock *sk,
>> struct sk_buff *skb)
>>
>>         __skb_tunnel_rx(skb, vxlan->dev);
>>         skb_reset_network_header(skb);
>> -       skb->ip_summed = CHECKSUM_NONE;
>> +
>> +       /* If the NIC driver gave us an encapsulated packet with
>> +        * CHECKSUM_UNNECESSARY and Rx checksum feature is enabled,
>> +        * leave the CHECKSUM_UNNECESSARY, the device checksummed it
>> +        * for us. Otherwise force the upper layers to verify it.
>> +        */
>> +       if (skb->ip_summed != CHECKSUM_UNNECESSARY || !skb->encapsulation
>> ||
>> +           !(vxlan->dev->features & NETIF_F_RXCSUM))
>> +               skb->ip_summed = CHECKSUM_NONE;
>> +
>> +       skb->encapsulation = 0;
>
>
> which was moved by your commits
>
> 5cfccc5 vxlan: Add vxlan recv demux.
> 7ce0475 vxlan: Restructure vxlan receive.
>
> to a code which isn't shared by the vxlan driver and ovs vxlan datapath
> code, and I was thinking lets just move it back to vxlan_udp_encap_recv.
> However, we can't really make the "vxlan->dev->features & NETIF_F_RXCSUM"
> check - since there's no vxlan device for ovs datapath, any idea how to
> overcome this.
>
> Moving this to shared code (while removing the check for
> vxlan->dev->features) made things to work on my setup, but this misses one
> of the original conditions, ideas?
>
I kept csum check in vxlan-device recv path for same reason. As of now
there is no efficient way to get ovs-dev features.
May be we can cache device features in struct datapath from datapath-netdev.

> I believe its too late for 3.13 unless there is going to be -rc9, but lets
> first see what's the right fix.
>
> Or.

Re: issues with vxlan RX checksum offload under OVS datapath

[Jesse Gross]

On Tue, Jan 21, 2014 at 9:30 AM, Pravin Shelar <pshelar@nicira.com> wrote:
> On Sun, Jan 19, 2014 at 2:05 PM, Or Gerlitz <ogerlitz@mellanox.com> wrote:
>> Hi Pravin,
>>
>> While testing the gro udp patches over a setup with openvswitch I noted that
>> the RX checksum offload support introduced by Joseph's commit 0afb166
>> "vxlan: Add capability of Rx checksum offload for inner packet" works fine
>> when you use a setup made of
>>
>> NIC --> IP stack --> vxlan device --> bridge --> tap
>>
>> but not when its
>>
>> NIC --> IP stack --> ovs vxlan port --> OVS DP --> tap
>>
>> I narrowed it down to the fact the when going the OVS pathskb->encapsulation
>> remains true also after the decap is done. Basically, this is the original
>> hunk
>>
>>
>>> @@ -607,7 +607,17 @@ static int vxlan_udp_encap_recv(struct sock *sk,
>>> struct sk_buff *skb)
>>>
>>>         __skb_tunnel_rx(skb, vxlan->dev);
>>>         skb_reset_network_header(skb);
>>> -       skb->ip_summed = CHECKSUM_NONE;
>>> +
>>> +       /* If the NIC driver gave us an encapsulated packet with
>>> +        * CHECKSUM_UNNECESSARY and Rx checksum feature is enabled,
>>> +        * leave the CHECKSUM_UNNECESSARY, the device checksummed it
>>> +        * for us. Otherwise force the upper layers to verify it.
>>> +        */
>>> +       if (skb->ip_summed != CHECKSUM_UNNECESSARY || !skb->encapsulation
>>> ||
>>> +           !(vxlan->dev->features & NETIF_F_RXCSUM))
>>> +               skb->ip_summed = CHECKSUM_NONE;
>>> +
>>> +       skb->encapsulation = 0;
>>
>>
>> which was moved by your commits
>>
>> 5cfccc5 vxlan: Add vxlan recv demux.
>> 7ce0475 vxlan: Restructure vxlan receive.
>>
>> to a code which isn't shared by the vxlan driver and ovs vxlan datapath
>> code, and I was thinking lets just move it back to vxlan_udp_encap_recv.
>> However, we can't really make the "vxlan->dev->features & NETIF_F_RXCSUM"
>> check - since there's no vxlan device for ovs datapath, any idea how to
>> overcome this.
>>
>> Moving this to shared code (while removing the check for
>> vxlan->dev->features) made things to work on my setup, but this misses one
>> of the original conditions, ideas?
>>
> I kept csum check in vxlan-device recv path for same reason. As of now
> there is no efficient way to get ovs-dev features.
> May be we can cache device features in struct datapath from datapath-netdev.\

I don't think that getting the features from the internal device makes
sense - a packet could be destined to a VM instead.

I think that controlling RX checksum on the device that physically
received it should be enough to be able to check for broken devices.
It's not clear what the benefit to turning it off at the VXLAN layer
is - if you really don't want it for some reason, you can always throw
away the checksum later.

Re: issues with vxlan RX checksum offload under OVS datapath

[Or Gerlitz]

On Tue, Jan 21, 2014 at 7:30 PM, Pravin Shelar <pshelar@nicira.com> wrote:
> On Sun, Jan 19, 2014 at 2:05 PM, Or Gerlitz <ogerlitz@mellanox.com> wrote:

>> While testing the gro udp patches over a setup with openvswitch I noted that
>> the RX checksum offload support introduced by Joseph's commit 0afb166
>> "vxlan: Add capability of Rx checksum offload for inner packet" works fine
>> when you use a setup made of
>> NIC --> IP stack --> vxlan device --> bridge --> tap
>> but not when its
>> NIC --> IP stack --> ovs vxlan port --> OVS DP --> tap
>> I narrowed it down to the fact the when going the OVS pathskb->encapsulation
>> remains true also after the decap is done. Basically, this is the original hunk
[...]
>>> +       skb->encapsulation = 0;
[...]

>> Moving this to shared code (while removing the check for
>> vxlan->dev->features) made things to work on my setup, but this misses one
>> of the original conditions, ideas?

> I kept csum check in vxlan-device recv path for same reason. As of now
> there is no efficient way to get ovs-dev features.
> May be we can cache device features in struct datapath from datapath-netdev.

To be a bit more precise/concrete here, do we agree that the both paths must do

   skb->encapsulation = 0;

which is done now only by the non-ovs path

Or.

Re: issues with vxlan RX checksum offload under OVS datapath

[Or Gerlitz]

On Tue, Jan 21, 2014, Joseph Gasparakis <joseph.gasparakis@intel.com> wrote:
> On Tue, 21 Jan 2014, Or Gerlitz wrote:

>> To be a bit more precise/concrete here, do we agree that the both paths must >>    skb->encapsulation = 0;
>> which is done now only by the non-ovs path

> Originally skb->encapsulation had (and still has) the meaning of "does
> this skb have outer *and* (valid) inner headers? If so, it is an
> encapsulated packet".
> So based on this skb->encapsulation should be set as soon an (inner)
> packet gets encapsulated and unset when decapsulation takes place, and
> ideally this should happen in the ovs path too.

I would say critically not ideally... e.g when the packet is
destinat-ed to a VM and goes through tap netdevice plugged to OVS --
without this de-assignment weird things happen after the point in time
where the ovs TX path calls dev_queue_xmit() in
net/openvswitch/vport-netdev.c

> Together with skb->encapsulation the inner headers should be correctly set.

That's interesting point, what might break if this isn't done?

Re: issues with vxlan RX checksum offload under OVS datapath

[Joseph Gasparakis]

On Tue, 21 Jan 2014, Or Gerlitz wrote:

> On Tue, Jan 21, 2014 at 7:30 PM, Pravin Shelar <pshelar@nicira.com> wrote:
> > On Sun, Jan 19, 2014 at 2:05 PM, Or Gerlitz <ogerlitz@mellanox.com> wrote:
> 
> >> While testing the gro udp patches over a setup with openvswitch I noted that
> >> the RX checksum offload support introduced by Joseph's commit 0afb166
> >> "vxlan: Add capability of Rx checksum offload for inner packet" works fine
> >> when you use a setup made of
> >> NIC --> IP stack --> vxlan device --> bridge --> tap
> >> but not when its
> >> NIC --> IP stack --> ovs vxlan port --> OVS DP --> tap
> >> I narrowed it down to the fact the when going the OVS pathskb->encapsulation
> >> remains true also after the decap is done. Basically, this is the original hunk
> [...]
> >>> +       skb->encapsulation = 0;
> [...]
> 
> >> Moving this to shared code (while removing the check for
> >> vxlan->dev->features) made things to work on my setup, but this misses one
> >> of the original conditions, ideas?
> 
> > I kept csum check in vxlan-device recv path for same reason. As of now
> > there is no efficient way to get ovs-dev features.
> > May be we can cache device features in struct datapath from datapath-netdev.
> 
> To be a bit more precise/concrete here, do we agree that the both paths must do
> 
>    skb->encapsulation = 0;
> 
> which is done now only by the non-ovs path

Originally skb->encapsulation had (and still has) the meaning of "does 
this skb have outer *and* (valid) inner headers? If so, it is an 
encapsulated packet". 
So based on this skb->encapsulation should be set as soon an (inner) 
packet gets encapsulated and unset when decapsulation takes place, and 
ideally this should happen in the ovs path too. Together with 
skb->encapsulation the inner headers should be correctly set.
 
> 
> Or.
> 

Re: issues with vxlan RX checksum offload under OVS datapath

[Joseph Gasparakis]

On Tue, 21 Jan 2014, Or Gerlitz wrote:

> On Tue, Jan 21, 2014, Joseph Gasparakis <joseph.gasparakis@intel.com> wrote:
> > On Tue, 21 Jan 2014, Or Gerlitz wrote:
> 
> >> To be a bit more precise/concrete here, do we agree that the both paths must >>    skb->encapsulation = 0;
> >> which is done now only by the non-ovs path
> 
> > Originally skb->encapsulation had (and still has) the meaning of "does
> > this skb have outer *and* (valid) inner headers? If so, it is an
> > encapsulated packet".
> > So based on this skb->encapsulation should be set as soon an (inner)
> > packet gets encapsulated and unset when decapsulation takes place, and
> > ideally this should happen in the ovs path too.
> 
> I would say critically not ideally... e.g when the packet is
> destinat-ed to a VM and goes through tap netdevice plugged to OVS --
> without this de-assignment weird things happen after the point in time
> where the ovs TX path calls dev_queue_xmit() in
> net/openvswitch/vport-netdev.c
> 
> > Together with skb->encapsulation the inner headers should be correctly set.
> 
> That's interesting point, what might break if this isn't done?

Well, the inner headers, same as the generic (outer) headers are pointers 
or offsets that are going to be added to the packet data pointer. If for 
any reason we try to access these pointers when they are invalid anything 
can go wrong... The way to indicate if they are valid or not should be 
skb->encapsulation.

>