From: Jason Wang <jasowang@redhat.com>
To: Willem de Bruijn <willemdebruijn.kernel@gmail.com>
Cc: Wei Wang <weiwan@google.com>, David Miller <davem@davemloft.net>,
Network Development <netdev@vger.kernel.org>,
Jakub Kicinski <kuba@kernel.org>,
virtualization@lists.linux-foundation.org
Subject: Re: [PATCH net] virtio-net: suppress bad irq warning for tx napi
Date: Tue, 9 Feb 2021 11:27:05 +0800 [thread overview]
Message-ID: <00de1b0f-f2aa-de54-9c7e-472643400417@redhat.com> (raw)
In-Reply-To: <CAF=yD-J5-60D=JDwvpecjaO6J03SZHoHJyCsR3B1HbP1-jbqng@mail.gmail.com>
On 2021/2/9 上午3:08, Willem de Bruijn wrote:
> On Sun, Feb 7, 2021 at 10:29 PM Jason Wang <jasowang@redhat.com> wrote:
>>
>> On 2021/2/5 上午4:50, Willem de Bruijn wrote:
>>> On Wed, Feb 3, 2021 at 10:06 PM Jason Wang <jasowang@redhat.com> wrote:
>>>> On 2021/2/4 上午2:28, Willem de Bruijn wrote:
>>>>> On Wed, Feb 3, 2021 at 12:33 AM Jason Wang <jasowang@redhat.com> wrote:
>>>>>> On 2021/2/2 下午10:37, Willem de Bruijn wrote:
>>>>>>> On Mon, Feb 1, 2021 at 10:09 PM Jason Wang <jasowang@redhat.com> wrote:
>>>>>>>> On 2021/1/29 上午8:21, Wei Wang wrote:
>>>>>>>>> With the implementation of napi-tx in virtio driver, we clean tx
>>>>>>>>> descriptors from rx napi handler, for the purpose of reducing tx
>>>>>>>>> complete interrupts. But this could introduce a race where tx complete
>>>>>>>>> interrupt has been raised, but the handler found there is no work to do
>>>>>>>>> because we have done the work in the previous rx interrupt handler.
>>>>>>>>> This could lead to the following warning msg:
>>>>>>>>> [ 3588.010778] irq 38: nobody cared (try booting with the
>>>>>>>>> "irqpoll" option)
>>>>>>>>> [ 3588.017938] CPU: 4 PID: 0 Comm: swapper/4 Not tainted
>>>>>>>>> 5.3.0-19-generic #20~18.04.2-Ubuntu
>>>>>>>>> [ 3588.017940] Call Trace:
>>>>>>>>> [ 3588.017942] <IRQ>
>>>>>>>>> [ 3588.017951] dump_stack+0x63/0x85
>>>>>>>>> [ 3588.017953] __report_bad_irq+0x35/0xc0
>>>>>>>>> [ 3588.017955] note_interrupt+0x24b/0x2a0
>>>>>>>>> [ 3588.017956] handle_irq_event_percpu+0x54/0x80
>>>>>>>>> [ 3588.017957] handle_irq_event+0x3b/0x60
>>>>>>>>> [ 3588.017958] handle_edge_irq+0x83/0x1a0
>>>>>>>>> [ 3588.017961] handle_irq+0x20/0x30
>>>>>>>>> [ 3588.017964] do_IRQ+0x50/0xe0
>>>>>>>>> [ 3588.017966] common_interrupt+0xf/0xf
>>>>>>>>> [ 3588.017966] </IRQ>
>>>>>>>>> [ 3588.017989] handlers:
>>>>>>>>> [ 3588.020374] [<000000001b9f1da8>] vring_interrupt
>>>>>>>>> [ 3588.025099] Disabling IRQ #38
>>>>>>>>>
>>>>>>>>> This patch adds a new param to struct vring_virtqueue, and we set it for
>>>>>>>>> tx virtqueues if napi-tx is enabled, to suppress the warning in such
>>>>>>>>> case.
>>>>>>>>>
>>>>>>>>> Fixes: 7b0411ef4aa6 ("virtio-net: clean tx descriptors from rx napi")
>>>>>>>>> Reported-by: Rick Jones <jonesrick@google.com>
>>>>>>>>> Signed-off-by: Wei Wang <weiwan@google.com>
>>>>>>>>> Signed-off-by: Willem de Bruijn <willemb@google.com>
>>>>>>>> Please use get_maintainer.pl to make sure Michael and me were cced.
>>>>>>> Will do. Sorry about that. I suggested just the virtualization list, my bad.
>>>>>>>
>>>>>>>>> ---
>>>>>>>>> drivers/net/virtio_net.c | 19 ++++++++++++++-----
>>>>>>>>> drivers/virtio/virtio_ring.c | 16 ++++++++++++++++
>>>>>>>>> include/linux/virtio.h | 2 ++
>>>>>>>>> 3 files changed, 32 insertions(+), 5 deletions(-)
>>>>>>>>>
>>>>>>>>> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
>>>>>>>>> index 508408fbe78f..e9a3f30864e8 100644
>>>>>>>>> --- a/drivers/net/virtio_net.c
>>>>>>>>> +++ b/drivers/net/virtio_net.c
>>>>>>>>> @@ -1303,13 +1303,22 @@ static void virtnet_napi_tx_enable(struct virtnet_info *vi,
>>>>>>>>> return;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> + /* With napi_tx enabled, free_old_xmit_skbs() could be called from
>>>>>>>>> + * rx napi handler. Set work_steal to suppress bad irq warning for
>>>>>>>>> + * IRQ_NONE case from tx complete interrupt handler.
>>>>>>>>> + */
>>>>>>>>> + virtqueue_set_work_steal(vq, true);
>>>>>>>>> +
>>>>>>>>> return virtnet_napi_enable(vq, napi);
>>>>>>>> Do we need to force the ordering between steal set and napi enable?
>>>>>>> The warning only occurs after one hundred spurious interrupts, so not
>>>>>>> really.
>>>>>> Ok, so it looks like a hint. Then I wonder how much value do we need to
>>>>>> introduce helper like virtqueue_set_work_steal() that allows the caller
>>>>>> to toggle. How about disable the check forever during virtqueue
>>>>>> initialization?
>>>>> Yes, that is even simpler.
>>>>>
>>>>> We still need the helper, as the internal variables of vring_virtqueue
>>>>> are not accessible from virtio-net. An earlier patch added the
>>>>> variable to virtqueue itself, but I think it belongs in
>>>>> vring_virtqueue. And the helper is not a lot of code.
>>>> It's better to do this before the allocating the irq. But it looks not
>>>> easy unless we extend find_vqs().
>>> Can you elaborate why that is better? At virtnet_open the interrupts
>>> are not firing either.
>>
>> I think you meant NAPI actually?
> I meant interrupt: we don't have to worry about the spurious interrupt
> warning when no interrupts will be firing. Until virtnet_open
> completes, the device is down.
Ok.
>
>
>>> I have no preference. Just curious, especially if it complicates the patch.
>>>
>> My understanding is that. It's probably ok for net. But we probably need
>> to document the assumptions to make sure it was not abused in other drivers.
>>
>> Introduce new parameters for find_vqs() can help to eliminate the subtle
>> stuffs but I agree it looks like a overkill.
>>
>> (Btw, I forget the numbers but wonder how much difference if we simple
>> remove the free_old_xmits() from the rx NAPI path?)
> The committed patchset did not record those numbers, but I found them
> in an earlier iteration:
>
> [PATCH net-next 0/3] virtio-net tx napi
> https://lists.openwall.net/netdev/2017/04/02/55
>
> It did seem to significantly reduce compute cycles ("Gcyc") at the
> time. For instance:
>
> TCP_RR Latency (us):
> 1x:
> p50 24 24 21
> p99 27 27 27
> Gcycles 299 432 308
>
> I'm concerned that removing it now may cause a regression report in a
> few months. That is higher risk than the spurious interrupt warning
> that was only reported after years of use.
Right.
So if Michael is fine with this approach, I'm ok with it. But we
probably need to a TODO to invent the interrupt handlers that can be
used for more than one virtqueues. When MSI-X is enabled, the interrupt
handler (vring_interrup()) assumes the interrupt is used by a single
virtqueue.
Thanks
>
next prev parent reply other threads:[~2021-02-09 3:37 UTC|newest]
Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-29 0:21 [PATCH net] virtio-net: suppress bad irq warning for tx napi Wei Wang
2021-02-02 2:24 ` Jakub Kicinski
2021-02-02 3:06 ` Jason Wang
2021-02-02 14:37 ` Willem de Bruijn
2021-02-03 5:33 ` Jason Wang
2021-02-03 18:28 ` Willem de Bruijn
2021-02-04 3:06 ` Jason Wang
2021-02-04 20:50 ` Willem de Bruijn
2021-02-08 3:29 ` Jason Wang
2021-02-08 19:08 ` Willem de Bruijn
2021-02-09 3:27 ` Jason Wang [this message]
2021-02-09 14:58 ` Willem de Bruijn
2021-02-09 18:00 ` Wei Wang
2021-02-10 9:14 ` Michael S. Tsirkin
2021-02-11 0:13 ` Wei Wang
2021-02-18 5:39 ` Jason Wang
2021-02-21 11:33 ` Michael S. Tsirkin
2021-02-02 23:11 ` Michael S. Tsirkin
2021-02-02 23:47 ` Wei Wang
2021-02-02 23:53 ` Willem de Bruijn
2021-02-03 0:06 ` Willem de Bruijn
2021-02-03 10:38 ` Michael S. Tsirkin
2021-02-03 18:24 ` Willem de Bruijn
2021-02-03 23:09 ` Michael S. Tsirkin
2021-02-03 23:52 ` Wei Wang
2021-02-04 20:47 ` Willem de Bruijn
2021-02-05 22:28 ` Wei Wang
2021-04-13 5:15 ` Michael S. Tsirkin
2021-09-29 20:21 ` Wei Wang
2021-09-29 21:53 ` Michael S. Tsirkin
2021-09-29 23:08 ` Wei Wang
2021-09-30 5:40 ` Michael S. Tsirkin
2021-09-30 18:10 ` Dave Taht
2021-04-12 22:08 ` Michael S. Tsirkin
2021-04-12 22:33 ` Michael S. Tsirkin
2021-04-12 23:14 ` David Miller
2021-04-13 4:24 ` Michael S. Tsirkin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=00de1b0f-f2aa-de54-9c7e-472643400417@redhat.com \
--to=jasowang@redhat.com \
--cc=davem@davemloft.net \
--cc=kuba@kernel.org \
--cc=netdev@vger.kernel.org \
--cc=virtualization@lists.linux-foundation.org \
--cc=weiwan@google.com \
--cc=willemdebruijn.kernel@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).