From: Josh Hunt <johunt@akamai.com>
To: Jike Song <albcamus@gmail.com>
Cc: Paolo Abeni <pabeni@redhat.com>,
Jonas Bonn <jonas.bonn@netrounds.com>,
Cong Wang <xiyou.wangcong@gmail.com>,
Michael Zhivich <mzhivich@akamai.com>,
David Miller <davem@davemloft.net>,
John Fastabend <john.fastabend@gmail.com>,
LKML <linux-kernel@vger.kernel.org>,
Linux Kernel Network Developers <netdev@vger.kernel.org>,
kehuan.feng@gmail.com
Subject: Re: Packet gets stuck in NOLOCK pfifo_fast qdisc
Date: Thu, 20 Aug 2020 11:13:27 -0700 [thread overview]
Message-ID: <74921739-d344-38eb-aa19-c078783b6328@akamai.com> (raw)
In-Reply-To: <CANE52Ki8rZGDPLZkxY--RPeEG+0=wFeyCD6KKkeG1WREUwramw@mail.gmail.com>
Hi Jike
On 8/20/20 12:43 AM, Jike Song wrote:
> Hi Josh,
>
>
> We met possibly the same problem when testing nvidia/mellanox's
> GPUDirect RDMA product, we found that changing NET_SCH_DEFAULT to
> DEFAULT_FQ_CODEL mitigated the problem, having no idea why. Maybe you
> can also have a try?
We also did something similar where we've switched over to using the fq
scheduler everywhere for now. We believe the bug is in the nolock code
which only pfifo_fast uses atm, but we've been unable to come up with a
satisfactory solution.
>
> Besides, our testing is pretty complex, do you have a quick test to
> reproduce it?
>
Unfortunately we don't have a simple test case either. Our current
reproducer is complex as well, although it would seem like we should be
able to come up with something where you have maybe 2 threads trying to
send on the same tx queue running pfifo_fast every few hundred
milliseconds and not much else/no other tx traffic on that queue. IIRC
we believe the scenario is when one thread is in the process of
dequeuing a packet while another is enqueuing, the enqueue-er (word? :))
sees the dequeue is in progress and so does not xmit the packet assuming
the dequeue operation will take care of it. However b/c the dequeue is
in the process of completing it doesn't and the newly enqueued packet
stays in the qdisc until another packet is enqueued pushing both out.
Given that we have a workaround with using fq or any other qdisc not
named pfifo_fast this has gotten bumped down in priority for us. I would
like to work on a reproducer at some point, but won't likely be for a
few weeks :(
Josh
next prev parent reply other threads:[~2020-08-20 19:06 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-09 6:46 Packet gets stuck in NOLOCK pfifo_fast qdisc Jonas Bonn
2019-10-09 19:14 ` Paolo Abeni
2019-10-10 6:27 ` Jonas Bonn
2019-10-11 0:39 ` Jonas Bonn
2020-06-23 13:42 ` Michael Zhivich
2020-06-30 19:14 ` Josh Hunt
2020-07-01 7:53 ` Jonas Bonn
2020-07-01 16:05 ` Cong Wang
2020-07-01 19:58 ` Cong Wang
2020-07-01 22:02 ` Josh Hunt
2020-07-02 6:14 ` Jonas Bonn
2020-07-02 9:45 ` Paolo Abeni
2020-07-02 18:08 ` Josh Hunt
2020-07-07 14:18 ` Paolo Abeni
2020-07-08 20:16 ` Cong Wang
2020-07-09 9:20 ` Paolo Abeni
2020-07-08 20:33 ` Zhivich, Michael
2020-08-20 7:43 ` Jike Song
2020-08-20 18:13 ` Josh Hunt [this message]
[not found] ` <20200822032800.16296-1-hdanton@sina.com>
2020-08-25 2:18 ` Fengkehuan Feng
[not found] ` <20200825032312.11776-1-hdanton@sina.com>
2020-08-25 7:14 ` Fengkehuan Feng
[not found] ` <20200825162329.11292-1-hdanton@sina.com>
2020-08-26 2:38 ` Kehuan Feng
[not found] ` <CACS=qqKptAQQGiMoCs1Zgs9S4ZppHhasy1AK4df2NxnCDR+vCw@mail.gmail.com>
[not found] ` <5f46032e.1c69fb81.9880c.7a6cSMTPIN_ADDED_MISSING@mx.google.com>
2020-08-27 6:56 ` Kehuan Feng
[not found] ` <20200827125747.5816-1-hdanton@sina.com>
2020-08-28 1:45 ` Kehuan Feng
2020-09-03 5:01 ` Cong Wang
2020-09-03 8:39 ` Paolo Abeni
2020-09-03 17:43 ` Cong Wang
2020-09-04 5:07 ` John Fastabend
2020-09-10 20:15 ` Cong Wang
2020-09-10 21:07 ` John Fastabend
2020-09-10 21:40 ` Paolo Abeni
2021-04-02 19:25 ` Jiri Kosina
2021-04-02 19:33 ` Josh Hunt
[not found] ` <20210403003537.2032-1-hdanton@sina.com>
2021-04-03 12:23 ` Jiri Kosina
2021-04-06 0:55 ` Yunsheng Lin
2021-04-06 7:06 ` Michal Kubecek
2021-04-06 10:13 ` Juergen Gross
2021-04-06 12:17 ` Yunsheng Lin
2021-04-06 1:49 ` Cong Wang
2021-04-06 2:46 ` Yunsheng Lin
2021-04-06 7:31 ` Michal Kubecek
2021-04-06 12:24 ` Yunsheng Lin
[not found] ` <20200903101957.428-1-hdanton@sina.com>
2020-09-04 3:20 ` Kehuan Feng
2020-09-10 20:19 ` Cong Wang
2020-09-14 2:10 ` Yunsheng Lin
2020-09-17 19:52 ` Cong Wang
2020-09-18 2:06 ` Kehuan Feng
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=74921739-d344-38eb-aa19-c078783b6328@akamai.com \
--to=johunt@akamai.com \
--cc=albcamus@gmail.com \
--cc=davem@davemloft.net \
--cc=john.fastabend@gmail.com \
--cc=jonas.bonn@netrounds.com \
--cc=kehuan.feng@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mzhivich@akamai.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=xiyou.wangcong@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).