Re: False waker detection in BFQ

[Jan Kara <jack@suse.cz>]

On Thu 26-08-21 11:45:17, Paolo Valente wrote:
> 
> 
> > Il giorno 25 ago 2021, alle ore 18:43, Jan Kara <jack@suse.cz> ha scritto:
> > 
> > On Mon 23-08-21 18:06:18, Jan Kara wrote:
> >> On Mon 23-08-21 15:58:25, Paolo Valente wrote:
> >>>> Currently I'm running wider set of benchmarks for the patches to see
> >>>> whether I didn't regress anything else. If not, I'll post the patches to
> >>>> the list.
> >>> 
> >>> Any news?
> >> 
> >> It took a while for all those benchmarks to run. Overall results look sane,
> >> I'm just verifying by hand now whether some of the localized regressions
> >> (usually specific to a particular fs+machine config) are due to a measurement
> >> noise or real regressions...
> > 
> > OK, so after some manual analysis I've found out that dbench indeed becomes
> > more noisy with my changes for high numbers of processes. I'm leaving for
> > vacation soon so I will not be probably able to debug it before I leave but
> > let me ask you one thing: The problematic change seems to be mostly a
> > revert of 7cc4ffc55564 ("block, bfq: put reqs of waker and woken in
> > dispatch list") and I'm currently puzzled why it has such an effect. What
> > I've found out is that 7cc4ffc55564 results in IO of higher priority
> > process being injected into the time slice of lower priority process and
> > thus there's always only single busy queue (of the lower priority process)
> > and thus higher priority process queue never gets scheduled. As a result
> > higher priority IO always competes with lower priority IO and there's no
> > service differentiation (we get 50/50 split of throughput between the
> > processes despite different IO priorities).
> 
> I need a little help here.  Since high-priority I/O is immediately
> injected, I wonder why it does not receive all the bandwidth it
> demands.  Maybe, from your analysis, you have an answer.  Perhaps it
> happens because:
> 1) high-priority I/O is FIFO-queued with lower-priority I/O in the
>    dispatch list?

Yes, this is the case.

> >  And this scenario shows that
> > always injecting IO of waker/wakee isn't desirable, especially in a way as
> > done in 7cc4ffc55564 where injected IO isn't accounted within BFQ at all
> > (which easily allows for service degradation unnoticed by BFQ).
> 
> Not sure that accounting would help high-priority I/O in your scenario.

It did help noticeably. Because then both high and low priority bfq queues
become busy so bfq_select_queue() sees both queues and schedules higher
priority queue.

> >  That's why
> > I've basically reverted that commit on the ground that on next dispatch we
> > call bfq_select_queue() which will see waker/wakee has IO to do and can
> > decide to inject the IO anyway. We do more CPU work but the IO pattern
> > should be similar. But apparently I was wrong :)
> 
> For the pattern to be similar, I guess that, when new high-priority
> I/O arrives, this I/O should preempt lower-priority I/O.
> Unfortunately, this is not always the case, depending on other
> parameters.  Waker/wakee I/O is guaranteed to be injected only when the
> in-service queue has no I/O.
> 
> At any rate, probably we can solve this puzzle by just analyzing a
> trace in which you detect a loss of throughput without 7cc4ffc55564.
> The best case would be one with the minimum possible number of
> threads, to get a simpler trace.

Yeah, OK, I'll gather the trace once I return from vacation and look into
it. Thanks for help!

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR