Re: Linux 4.14 - regression: broken tun/tap / bridge network with virtio - bisected

From: Andreas Hartmann <andihartmann@01019freenet.de>
To: Michal Kubecek <mkubecek@suse.cz>
Cc: Jason Wang <jasowang@redhat.com>,
	David Miller <davem@davemloft.net>,
	netdev@vger.kernel.org
Subject: Re: Linux 4.14 - regression: broken tun/tap / bridge network with virtio - bisected
Date: Fri, 8 Dec 2017 11:31:50 +0100	[thread overview]
Message-ID: <b0e85abe-7c69-3f86-7bf0-3c13cfdfc4cb@01019freenet.de> (raw)
In-Reply-To: <20171208084751.tom4auppogz4lanz@unicorn.suse.cz>

On 12/08/2017 at 09:47 AM Michal Kubecek wrote:
> On Fri, Dec 08, 2017 at 08:21:16AM +0100, Andreas Hartmann wrote:
>>
>> Thanks for this hint - I'm not using xdp. Therefore I rechecked my
>> bisect and detected a mistake. The rebisect now leads to
>>
>>
>>
>> [v2,RFC,11/13] net: Remove all references to SKB_GSO_UDP. [1]
>>
>>
>>
>> For the repeated bisect, I switched back to the original qemu 2.6.2
>> (instead of 2.10.1), because problems can be seen reliably with 2.6.2.
>>
>> All my VMs are using virtio_net. BTW: I couldn't see the problems
>> (sometimes, the VM couldn't be stopped at all) if all my VMs are using
>> e1000 as interface instead.
>>
>> This finding now matches pretty much the responsible UDP-package which
>> caused the stall. I already mentioned it here [2].
>>
>> To prove it, I reverted from the patch series "[PATCH v2 RFC 0/13]
>> Remove UDP Fragmentation Offload support" [3]
>>
>> 11/13 [v2,RFC,11/13] net: Remove all references to SKB_GSO_UDP. [4]
>> 12/13 [v2,RFC,12/13] inet: Remove software UFO fragmenting code. [5]
>> 13/13 [v2,RFC,13/13] net: Kill NETIF_F_UFO and SKB_GSO_UDP. [6]
>>
>> and applied it to Linux 4.14.4. It compiled fine and is running fine.
>> The vnet doesn't die anymore. Yet, I can't say if the qemu stop hangs
>> are gone, too.
>>
>> Obviously, there is something broken with the new UDP handling. Could
>> you please analyze this problem? I could test some more patches ... .
> 
> Any chance your VMs were live migrated from pre-4.14 host kernel?

No - the VMs are not live migrated. They are always running on the same
host - either with kernel < 4.14 or with kernel 4.14.x.

> If
> this is the case, you should try commit 0c19f846d582 ("net: accept UFO
> datagrams from tuntap and packet"). 

It doesn't apply to 4.14.4

> Or disabling UFO in the guest should
> work around the issue.

ethtool -K ethX ufo off for each device / bridge in VM.

Yes, this seems to work. I'll wait and see if the non stoppable
qemu-problem on shutdown will remain.

When will there be a fix for 4.14? It is clearly a regression. Is it
possible / a good idea to just remove the complete patch series "Remove
UDP Fragmentation Offload support"?

Thanks,
Andreas