Invalid GSO - from 4.x (TBA) to 5.5-rc1. Was: Re: 64 bit time regression in recvmmsg()

Invalid GSO - from 4.x (TBA) to 5.5-rc1. Was: Re: 64 bit time regression in recvmmsg()

[Anton Ivanov]

On 06/12/2019 20:06, Arnd Bergmann wrote:
> On Fri, Dec 6, 2019 at 6:50 PM Anton Ivanov
> <anton.ivanov@cambridgegreys.com> wrote:
>> On 29/11/2019 16:34, Anton Ivanov wrote:
>>
>> I apologize, problem elsewhere. I have narrowed it down, it is a host
>> regression at the end, not a recvmmsg/time one.
>>
>> The EINVAL is occasionally returned from the guts of packet_rcv_vnet
>>
>> https://elixir.bootlin.com/linux/latest/ident/packet_rcv_vnet
>>
>> in af_packet. I am going to try to figure out exactly when it happens
>> and why.
>>
>> My sincere apologies,
> 
> No, worries, I'm glad it wasn't me this time ;-)

At some point in late 4.x (I am going to try narrowing down the 
version), gso code introduced a condition which should not occur:

We have

sinfo(skb)->gso_type = 0

while

sinfo(skb)->gso_size = 752.
skb->len = 818
skb->data_len = 0

As a result we get a skb which is GSO, but it has a no GSO type.

This shows up in virtio_net_hdr_from_skb() which detects the condition 
as invalid and returns an -EINVAL

A few interesting things.

1. Size is always 752.

2. I have found it while tracing an -EINVAL when using recvmmsg() on raw 
sockets. No idea if it shows up elsewhere.

3. The environment under test is reading/writing to a raw socket opened 
on a vEth pair configured as follows:


ip link add l-veth0 type veth peer name p-veth0 && ifconfig p-veth0 up
ifconfig l-veth0 192.168.97.1 netmask 255.255.255.252

the UML linux instance used as a reader/writer relies on 
recvmmsg/sendmmesg raw socket with vnet headers enabled.

virtio_net_hdr_from_skb() is called from the af_packet packet_recv_vnet 
function.

This condition simply should not occur. A skb with no data in the frags, 
gso on, gso size less than MTU and no type looks bogus.


> 
>       Arnd
> 
> _______________________________________________
> linux-um mailing list
> linux-um@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-um
> 

-- 
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/

Re: Invalid GSO - from 4.x (TBA) to 5.5-rc1. Was: Re: 64 bit time regression in recvmmsg()

[Anton Ivanov]

On 09/12/2019 09:19, Anton Ivanov wrote:
> 
> 
> On 06/12/2019 20:06, Arnd Bergmann wrote:
>> On Fri, Dec 6, 2019 at 6:50 PM Anton Ivanov
>> <anton.ivanov@cambridgegreys.com> wrote:
>>> On 29/11/2019 16:34, Anton Ivanov wrote:
>>>
>>> I apologize, problem elsewhere. I have narrowed it down, it is a host
>>> regression at the end, not a recvmmsg/time one.
>>>
>>> The EINVAL is occasionally returned from the guts of packet_rcv_vnet
>>>
>>> https://elixir.bootlin.com/linux/latest/ident/packet_rcv_vnet
>>>
>>> in af_packet. I am going to try to figure out exactly when it happens
>>> and why.
>>>
>>> My sincere apologies,
>>
>> No, worries, I'm glad it wasn't me this time ;-)
> 
> At some point in late 4.x (I am going to try narrowing down the 
> version), gso code introduced a condition which should not occur:
> 
> We have
> 
> sinfo(skb)->gso_type = 0
> 
> while
> 
> sinfo(skb)->gso_size = 752.
> skb->len = 818
> skb->data_len = 0
> 
> As a result we get a skb which is GSO, but it has a no GSO type.
> 
> This shows up in virtio_net_hdr_from_skb() which detects the condition 
> as invalid and returns an -EINVAL
> 
> A few interesting things.
> 
> 1. Size is always 752.
> 
> 2. I have found it while tracing an -EINVAL when using recvmmsg() on raw 
> sockets. No idea if it shows up elsewhere.
> 
> 3. The environment under test is reading/writing to a raw socket opened 
> on a vEth pair configured as follows:
> 
> 
> ip link add l-veth0 type veth peer name p-veth0 && ifconfig p-veth0 up
> ifconfig l-veth0 192.168.97.1 netmask 255.255.255.252
> 
> the UML linux instance used as a reader/writer relies on 
> recvmmsg/sendmmesg raw socket with vnet headers enabled.
> 
> virtio_net_hdr_from_skb() is called from the af_packet packet_recv_vnet 
> function.
> 
> This condition simply should not occur. A skb with no data in the frags, 
> gso on, gso size less than MTU and no type looks bogus.

While the gso marking on the frame is bogus, the frame itself actually 
looks good (as seen from the UML used as a reader).

I get ~ 10% throughput increase because recvmmsg does not get broken all 
the time into smaller vector sizes and no frames reported as dropped due 
to checksum or other issues.

I will send a patch with a proposed fix shortly.

We should still find who is producing these frames though.

> 
> 
>>
>>       Arnd
>>
>> _______________________________________________
>> linux-um mailing list
>> linux-um@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-um
>>
> 

-- 
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/