Re: [blk] 8c5035dfbb: fio.read_iops -10.6% regression

[Feng Tang <feng.tang@intel.com>]

[-- Attachment #1: Type: text/plain, Size: 2446 bytes --]

On Sun, Oct 09, 2022 at 05:32:34PM +0800, Yu Kuai wrote:
> Hi,
> 
> 在 2022/10/09 16:43, Ming Lei 写道:
> > On Sat, Oct 08, 2022 at 04:00:10PM +0800, Yu Kuai wrote:
> >> Hi,
> >>
> >> 在 2022/10/08 10:50, kernel test robot 写道:
> >>> Greeting,
> >>>
> >>> FYI, we noticed a -10.6% regression of fio.read_iops due to commit:
> >>
> >> I don't know how this is working but I'm *sure* this commit won't affect
> > 
> > Looks it is wrong to move
> > 
> > 	wbt_set_write_cache(q, test_bit(QUEUE_FLAG_WC, &q->queue_flags));
> > 
> > before rq_qos_add() in wbt_init().
> > 
> > Without adding wbt rq_qos, wbt_set_write_cache is just a nop.
> 
> Yes, I got it now, I'm being foolish here.
> 
> I missed that "rwb->wc" is got by rq_qos in wbt_set_write_cache(), which
> is NULL before rq_qos_add(). By the way, it's interesting that how read
> performance is affected, I still don't know why yet...

Indeed, we are confused too. So we did some further check, and found
it could be related with the less calls of wake_up_all(), due to the
'rwb->wc' value changed. 

I'm not familiar with the block layer and VFS, and just checked the
'blk-wbt.c'. Before commit 8c5035dfbb, the 'rwb->wc' is 0 in 0Day's
test env, while it's 1 after the commit.

Inside wbt_rqw_done(), 'rwb->wc' be used to judge whether to wakeup
other waiters in system, so we add some debug code to check the
wakeup and skip-wakeup counter:

  ----------------------------------------------------------------
  @@ -130,6 +130,8 @@ static void wbt_rqw_done(struct rq_wb *rwb, struct rq_wait *rqw,
   	/*
   	 * Don't wake anyone up if we are above the normal limit.
   	 */
  -	if (inflight && inflight >= limit)
  +	if (inflight && inflight >= limit) {
  +		skip_wakeup++;
   		return;
  +	}
   
   	if (wq_has_sleeper(&rqw->wait)) {
   		int diff = limit - inflight;
   
  -		if (!inflight || diff >= rwb->wb_background / 2)
  +		if (!inflight || diff >= rwb->wb_background / 2) {
  +			wakeup++;
   			wake_up_all(&rqw->wait);
  +		}
  ----------------------------------------------------------------

And after the fio task, the 'skip_wakeup' number is much bigger
after the patch:
  
  before patch:
      422.274394: wbt_rqw_done: wakeup_skip=19408 wakup_all=1944759
  
  after patch:
      433.753345: wbt_rqw_done: wakeup_skip=2090585 wakup_all=13630

Hope this can help the root causing.

Thanks,
Feng