From mboxrd@z Thu Jan  1 00:00:00 1970
From: Andreas Hartmann <andihartmann@01019freenet.de>
Subject: Re: Linux 4.14 - regression: broken tun/tap / bridge network with
 virtio - bisected
Date: Fri, 15 Dec 2017 07:05:07 +0100
Message-ID: <c2044fb8-a2ce-241f-b1ce-054ac70a327d@01019freenet.de>
References: <9615150a-eb78-2f9d-798f-6aa460932aec@01019freenet.de>
 <2e2392b7-84c5-be89-b0e5-5bae3b2fdaed@01019freenet.de>
 <4efbaf24-f419-2c8e-c705-59a5242b0575@01019freenet.de>
 <881560f8-54ec-e946-50cb-b2e80ddb5f97@01019freenet.de>
 <73b7a7b0-4264-2bd0-9e65-69841377f09f@redhat.com>
 <401a0715-fd28-63a3-8dfd-e89835d70db0@01019freenet.de>
 <a259bb30-8cc0-fceb-5481-1295de54b5fb@redhat.com>
 <11c25b88-af9b-a1f7-b5f5-0420c75916d7@01019freenet.de>
 <20171208084751.tom4auppogz4lanz@unicorn.suse.cz>
 <b0e85abe-7c69-3f86-7bf0-3c13cfdfc4cb@01019freenet.de>
 <20171208114025.kjcaratqcveq7zu5@unicorn.suse.cz>
 <CAF=yD-JvdPqZ1fbeSitZ192Xhz0qYK-P_=5ydnu2GiJBU5GLDA@mail.gmail.com>
 <96a16c1f-c026-f506-78c1-dad88471361d@01019freenet.de>
 <bc075f2c-85d0-cf97-60cd-ee6d6efca12e@01019freenet.de>
 <d71df64e-e65f-4db4-6f2e-c002c15fcbe4@01019freenet.de>
 <CAF=yD-JnDSfH-2wMjN5BUVdXeezp+k4ievSuvVCvEjX-jncD7Q@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Cc: Michal Kubecek <mkubecek@suse.cz>,
        Jason Wang <jasowang@redhat.com>,
        David Miller <davem@davemloft.net>,
        Network Development <netdev@vger.kernel.org>
To: Willem de Bruijn <willemdebruijn.kernel@gmail.com>
Return-path: <netdev-owner@vger.kernel.org>
Received: from mx1.mailbox.org ([80.241.60.212]:45370 "EHLO mx1.mailbox.org"
        rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
        id S1751266AbdLOGFT (ORCPT <rfc822;netdev@vger.kernel.org>);
        Fri, 15 Dec 2017 01:05:19 -0500
In-Reply-To: <CAF=yD-JnDSfH-2wMjN5BUVdXeezp+k4ievSuvVCvEjX-jncD7Q@mail.gmail.com>
Content-Language: en-US
Sender: netdev-owner@vger.kernel.org
List-ID: <netdev.vger.kernel.org>

On 12/14/2017 at 11:17 PM Willem de Bruijn wrote:
>>> Well, the patch does not fix hanging VMs, which have been shutdown and
>>> can't be killed any more.
>>> Because of the stack trace
>>>
>>> [<ffffffffc0d0e3c5>] vhost_net_ubuf_put_and_wait+0x35/0x60 [vhost_net]
>>> [<ffffffffc0d0f264>] vhost_net_ioctl+0x304/0x870 [vhost_net]
>>> [<ffffffff9b25460f>] do_vfs_ioctl+0x8f/0x5c0
>>> [<ffffffff9b254bb4>] SyS_ioctl+0x74/0x80
>>> [<ffffffff9b00365b>] do_syscall_64+0x5b/0x100
>>> [<ffffffff9b78e7ab>] entry_SYSCALL64_slow_path+0x25/0x25
>>> [<ffffffffffffffff>] 0xffffffffffffffff
>>>
>>> I was hoping, that the problems could be related - but that seems not to
>>> be true.
>>
>> However, it turned out, that reverting the complete patchset "Remove UDP
>> Fragmentation Offload support" prevent hanging qemu processes.
> 
> That implies a combination of UFO and vhost zerocopy. Disabling
> experimental_zcopytx in vhost_net will probably work around the bug
> then.

I already tested it w/ options vhost_net experimental_zcopytx=0 - but
this didn't "resolve" anything. See
https://www.mail-archive.com/netdev@vger.kernel.org/msg203197.html

Therefore, I think your following thoughts are lapsed unfortunately,
aren't they?

> On the surface the two features are independent. Most of the relevant
> UFO code is reverted with the patch mentioned earlier. Missing from
> that is protocol stack support, but it is unlikely that your host OS is
> generating these UFO packets.
> 
> They are coming from a guest over virtio_net, to which vhost_net then
> applies zerocopy. Then the packet(s) is/are either freed without calling
> uarg->callback() or queued somewhere for a very long time.
> 
> Looking at the diff-of-diffs between my stable patch and your full revert,
> the majority of missing bits beside the procol layer is in device driver
> support. Removing that causes the UFO packets to be segmented at any
> dev_queue_xmit on their path. skb_segment ensures that when it segments
> a large zerocopy packet, all new segments also point to the zerocopy
> callback struct (ubuf_info), as the shared memory pages may not be
> released until all skbs pointing to them are freed.
> 
> That may be wrong with vhost_zerocopy_callback, which does not use
> refcounting. I will look into that. It may be that before the msg_zerocopy
> patchsets large packets were copied before entering segmentation. It is
> safe to enter segmentation for msg_zerocopy skbs, but not legacy zerocopy
> skbs.
> 
> I will also set up two VMs and try to send UFO packets and see whether
> they indeed are freed somewhere in the stack without notifying vhost_net.


Thanks,
Andreas