From: Omar Kilani <omar.kilani@gmail.com>
To: linux-block@vger.kernel.org
Subject: Re: Deadlock in wbt / rq-qos
Date: Sun, 13 Jun 2021 10:03:38 -0700 [thread overview]
Message-ID: <CA+8F9hjJF8e4U7W9tUgGu7dUR_hz-y_EDuq-McHS-GBiZm0-rQ@mail.gmail.com> (raw)
In-Reply-To: <CA+8F9hggf7jOcGRxvBoa8FYxQs8ZV+XueVAd9BodpQQP_+8Pdw@mail.gmail.com>
Just looking at blk-wbt.c...
Should...
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/block/blk-wbt.c?h=v5.13-rc5&id=482e302a61f1fc62b0e13be20bc7a11a91b5832d#n164
if (!inflight || diff >= rwb->wb_background / 2)
Be:
if (!inflight || diff >= limit / 2)
?
On Sun, Jun 13, 2021 at 8:49 AM Omar Kilani <omar.kilani@gmail.com> wrote:
>
> Hi there,
>
> I appear to have stumbled upon a deadlock in wbt or rq-qos.
>
> My journal of a lot of data points is over here:
>
> https://github.com/openzfs/zfs/issues/12204
>
> I initially deadlocked on RHEL 8.4's 4.18.0-305.3.1.el8_4.x86_64
> kernel, but the code in blk-wbt.c / blk-rq-qos.c is functionally
> identical to 5.13.0-rc5, so I tried that and I'm able to deadlock that
> as well. I believe the same code exists all the way back to 5.0.1.
>
> The Something Weird (tm) about this is that it possibly only happens
> on AMD EPYC CPUs. I just don't have the necessary setup to confirm
> that either way, but it's a hunch because I can't reproduce it on an
> Ice Lake VM (but the Ice Lake VM also has more storage bandwidth so
> that could be the thing, and I can't decrease that storage bandwidth,
> so I can't do a like-for-like test.)
>
> I "instrumented" wbt / rq-qos with a bunch of printk's which you can
> see with this patch:
>
> https://gist.github.com/omarkilani/2ad526c3546b40537b546450c8f685dc
>
> I then ran my repro workload to cause the deadlock, here's the dmesg
> output just before the deadlock and then the backtraces with my printk
> patch applied:
>
> https://gist.githubusercontent.com/omarkilani/ff0a96d872e09b4fb648272d104e0053/raw/d3da3974162f8aa87b7309317af80929fadf250f/dmesg.wbt.deadlock.log
>
> Happy to apply whatever / run whatever to get more data.
>
> Thanks!
>
> Regards,
> Omar
next prev parent reply other threads:[~2021-06-13 17:04 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-13 15:49 Deadlock in wbt / rq-qos Omar Kilani
2021-06-13 17:03 ` Omar Kilani [this message]
2021-06-14 20:26 ` Omar Kilani
2021-06-15 9:22 ` Ming Lei
[not found] ` <CA+8F9hjFDE9b31-qsxsVJf4SV9Ctr-mwOJrsw0kVeC7DdN=5XQ@mail.gmail.com>
2021-06-15 14:07 ` Ming Lei
2021-06-16 15:06 ` Omar Kilani
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CA+8F9hjJF8e4U7W9tUgGu7dUR_hz-y_EDuq-McHS-GBiZm0-rQ@mail.gmail.com \
--to=omar.kilani@gmail.com \
--cc=linux-block@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).