From: Paolo Valente <paolo.valente@linaro.org>
To: Jan Kara <jack@suse.cz>
Cc: linux-block@vger.kernel.org
Subject: Re: False waker detection in BFQ
Date: Mon, 23 Aug 2021 15:58:25 +0200 [thread overview]
Message-ID: <A72B321A-3952-4C00-B7DB-67954B05B99A@linaro.org> (raw)
In-Reply-To: <20210813140111.GG11955@quack2.suse.cz>
> Il giorno 13 ago 2021, alle ore 16:01, Jan Kara <jack@suse.cz> ha scritto:
>
> Hi Paolo!
>
> On Thu 20-05-21 17:05:45, Paolo Valente wrote:
>>> Il giorno 5 mag 2021, alle ore 18:20, Jan Kara <jack@suse.cz> ha scritto:
>>>
>>> Hi Paolo!
>>>
>>> I have two processes doing direct IO writes like:
>>>
>>> dd if=/dev/zero of=/mnt/file$i bs=128k oflag=direct count=4000M
>>>
>>> Now each of these processes belongs to a different cgroup and it has
>>> different bfq.weight. I was looking into why these processes do not split
>>> bandwidth according to BFQ weights. Or actually the bandwidth is split
>>> accordingly initially but eventually degrades into 50/50 split. After some
>>> debugging I've found out that due to luck, one of the processes is decided
>>> to be a waker of the other process and at that point we loose isolation
>>> between the two cgroups. This pretty reliably happens sometime during the
>>> run of these two processes on my test VM. So can we tweak the waker logic
>>> to reduce the chances for false positives? Essentially when there are only
>>> two processes doing heavy IO against the device, the logic in
>>> bfq_check_waker() is such that they are very likely to eventually become
>>> wakers of one another. AFAICT the only condition that needs to get
>>> fulfilled is that they need to submit IO within 4 ms of the completion of
>>> IO of the other process 3 times.
>>>
>>
>> Hi Jan!
>> as I happened to tell you moths ago, I feared some likely cover case
>> to show up eventually. Actually, I was even more pessimistic than how
>> reality proved to be :)
>>
>> I'm sorry for my delay, but I've had to think about this issue for a
>> while. Being too strict would easily run out journald as a waker for
>> processes belonging to a different group.
>>
>> So, what do you think of this proposal: add the extra filter that a
>> waker must belong to the same group of the woken, or, at most, to the
>> root group?
>
> Returning back to this :). I've been debugging other BFQ problems with IO
> priorities not really leading to service differentiation (mostly because
> scheduler tag exhaustion, false waker detection, and how we inject IO for a
> waker) and as a result I have come up with a couple of patches that also
> address this issue as a side effect - I've added an upper time limit
> (128*slice_idle) for the "third cooperation" detection and that mostly got
> rid of these false waker detections.
Great!
> We could fail to detect waker-wakee
> processes if they do not cooperate frequently but then the value of the
> detection is small and the lack of isolation may do more harm than good
> anyway.
>
IIRC, dbench was our best benchmark for checking whether the detection is
(still) effective.
> Currently I'm running wider set of benchmarks for the patches to see
> whether I didn't regress anything else. If not, I'll post the patches to
> the list.
>
Any news?
Thanks,
Paolo
> Honza
> --
> Jan Kara <jack@suse.com>
> SUSE Labs, CR
next prev parent reply other threads:[~2021-08-23 13:58 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-05 16:20 False waker detection in BFQ Jan Kara
2021-05-20 15:05 ` Paolo Valente
2021-05-21 13:10 ` Jan Kara
2021-08-13 14:01 ` Jan Kara
2021-08-23 13:58 ` Paolo Valente [this message]
2021-08-23 16:06 ` Jan Kara
2021-08-25 16:43 ` Jan Kara
2021-08-26 9:45 ` Paolo Valente
2021-08-26 17:51 ` Jan Kara
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=A72B321A-3952-4C00-B7DB-67954B05B99A@linaro.org \
--to=paolo.valente@linaro.org \
--cc=jack@suse.cz \
--cc=linux-block@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).