* Invalid GSO - from 4.x (TBA) to 5.5-rc1. Was: Re: 64 bit time regression in recvmmsg() [not found] ` <CAK8P3a1KcRcpUB2PdA17tnHN_dDs2Y02zPtkgM22Z-JdugAkfQ@mail.gmail.com> @ 2019-12-09 9:19 ` Anton Ivanov 2019-12-09 10:05 ` Anton Ivanov 0 siblings, 1 reply; 2+ messages in thread From: Anton Ivanov @ 2019-12-09 9:19 UTC (permalink / raw) To: Arnd Bergmann; +Cc: linux-um, netdev On 06/12/2019 20:06, Arnd Bergmann wrote: > On Fri, Dec 6, 2019 at 6:50 PM Anton Ivanov > <anton.ivanov@cambridgegreys.com> wrote: >> On 29/11/2019 16:34, Anton Ivanov wrote: >> >> I apologize, problem elsewhere. I have narrowed it down, it is a host >> regression at the end, not a recvmmsg/time one. >> >> The EINVAL is occasionally returned from the guts of packet_rcv_vnet >> >> https://elixir.bootlin.com/linux/latest/ident/packet_rcv_vnet >> >> in af_packet. I am going to try to figure out exactly when it happens >> and why. >> >> My sincere apologies, > > No, worries, I'm glad it wasn't me this time ;-) At some point in late 4.x (I am going to try narrowing down the version), gso code introduced a condition which should not occur: We have sinfo(skb)->gso_type = 0 while sinfo(skb)->gso_size = 752. skb->len = 818 skb->data_len = 0 As a result we get a skb which is GSO, but it has a no GSO type. This shows up in virtio_net_hdr_from_skb() which detects the condition as invalid and returns an -EINVAL A few interesting things. 1. Size is always 752. 2. I have found it while tracing an -EINVAL when using recvmmsg() on raw sockets. No idea if it shows up elsewhere. 3. The environment under test is reading/writing to a raw socket opened on a vEth pair configured as follows: ip link add l-veth0 type veth peer name p-veth0 && ifconfig p-veth0 up ifconfig l-veth0 192.168.97.1 netmask 255.255.255.252 the UML linux instance used as a reader/writer relies on recvmmsg/sendmmesg raw socket with vnet headers enabled. virtio_net_hdr_from_skb() is called from the af_packet packet_recv_vnet function. This condition simply should not occur. A skb with no data in the frags, gso on, gso size less than MTU and no type looks bogus. > > Arnd > > _______________________________________________ > linux-um mailing list > linux-um@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-um > -- Anton R. Ivanov Cambridgegreys Limited. Registered in England. Company Number 10273661 https://www.cambridgegreys.com/ ^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: Invalid GSO - from 4.x (TBA) to 5.5-rc1. Was: Re: 64 bit time regression in recvmmsg() 2019-12-09 9:19 ` Invalid GSO - from 4.x (TBA) to 5.5-rc1. Was: Re: 64 bit time regression in recvmmsg() Anton Ivanov @ 2019-12-09 10:05 ` Anton Ivanov 0 siblings, 0 replies; 2+ messages in thread From: Anton Ivanov @ 2019-12-09 10:05 UTC (permalink / raw) To: Arnd Bergmann; +Cc: netdev, linux-um On 09/12/2019 09:19, Anton Ivanov wrote: > > > On 06/12/2019 20:06, Arnd Bergmann wrote: >> On Fri, Dec 6, 2019 at 6:50 PM Anton Ivanov >> <anton.ivanov@cambridgegreys.com> wrote: >>> On 29/11/2019 16:34, Anton Ivanov wrote: >>> >>> I apologize, problem elsewhere. I have narrowed it down, it is a host >>> regression at the end, not a recvmmsg/time one. >>> >>> The EINVAL is occasionally returned from the guts of packet_rcv_vnet >>> >>> https://elixir.bootlin.com/linux/latest/ident/packet_rcv_vnet >>> >>> in af_packet. I am going to try to figure out exactly when it happens >>> and why. >>> >>> My sincere apologies, >> >> No, worries, I'm glad it wasn't me this time ;-) > > At some point in late 4.x (I am going to try narrowing down the > version), gso code introduced a condition which should not occur: > > We have > > sinfo(skb)->gso_type = 0 > > while > > sinfo(skb)->gso_size = 752. > skb->len = 818 > skb->data_len = 0 > > As a result we get a skb which is GSO, but it has a no GSO type. > > This shows up in virtio_net_hdr_from_skb() which detects the condition > as invalid and returns an -EINVAL > > A few interesting things. > > 1. Size is always 752. > > 2. I have found it while tracing an -EINVAL when using recvmmsg() on raw > sockets. No idea if it shows up elsewhere. > > 3. The environment under test is reading/writing to a raw socket opened > on a vEth pair configured as follows: > > > ip link add l-veth0 type veth peer name p-veth0 && ifconfig p-veth0 up > ifconfig l-veth0 192.168.97.1 netmask 255.255.255.252 > > the UML linux instance used as a reader/writer relies on > recvmmsg/sendmmesg raw socket with vnet headers enabled. > > virtio_net_hdr_from_skb() is called from the af_packet packet_recv_vnet > function. > > This condition simply should not occur. A skb with no data in the frags, > gso on, gso size less than MTU and no type looks bogus. While the gso marking on the frame is bogus, the frame itself actually looks good (as seen from the UML used as a reader). I get ~ 10% throughput increase because recvmmsg does not get broken all the time into smaller vector sizes and no frames reported as dropped due to checksum or other issues. I will send a patch with a proposed fix shortly. We should still find who is producing these frames though. > > >> >> Arnd >> >> _______________________________________________ >> linux-um mailing list >> linux-um@lists.infradead.org >> http://lists.infradead.org/mailman/listinfo/linux-um >> > -- Anton R. Ivanov Cambridgegreys Limited. Registered in England. Company Number 10273661 https://www.cambridgegreys.com/ ^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2019-12-09 10:05 UTC | newest] Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <3820d68b-1d97-8f41-d55d-237d1695458c@cambridgegreys.com> [not found] ` <CAMuHMdWuiGC4ay=f6M2H=-PLiffavnFSu8CPXE26euAi6aoY0w@mail.gmail.com> [not found] ` <CAK8P3a1mrFgRyh5Fgv-d8Szd2pq0T6Ac7wL3ogeYcf-Uyrg4ZQ@mail.gmail.com> [not found] ` <a5b9709d-b93b-46e1-ab18-a94ab921ccf7@cambridgegreys.com> [not found] ` <9dc1be66-5c55-8b3d-875b-4e1206914e78@cambridgegreys.com> [not found] ` <CAK8P3a1KcRcpUB2PdA17tnHN_dDs2Y02zPtkgM22Z-JdugAkfQ@mail.gmail.com> 2019-12-09 9:19 ` Invalid GSO - from 4.x (TBA) to 5.5-rc1. Was: Re: 64 bit time regression in recvmmsg() Anton Ivanov 2019-12-09 10:05 ` Anton Ivanov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).