linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Josef Bacik <josef@toxicpanda.com>
To: Valentin Schneider <valentin.schneider@arm.com>
Cc: Thorsten Leemhuis <regressions@leemhuis.info>,
	peterz@infradead.org, vincent.guittot@linaro.org,
	torvalds@linux-foundation.org, linux-kernel@vger.kernel.org,
	linux-btrfs@vger.kernel.org, guro@fb.com, clm@fb.com
Subject: Re: [REGRESSION] 5-10% increase in IO latencies with nohz balance patch
Date: Mon, 3 Jan 2022 11:16:01 -0500	[thread overview]
Message-ID: <YdMhQRq1K8tW+S05@localhost.localdomain> (raw)
In-Reply-To: <87tuf07hdk.mognet@arm.com>

On Wed, Dec 22, 2021 at 04:07:35PM +0000, Valentin Schneider wrote:
> 
> Hi,
> 
> On 22/12/21 13:42, Thorsten Leemhuis wrote:
> > What's the status here? Just wondering, because there hasn't been any
> > activity in this thread since 11 days and the festive season is upon us.
> >
> > Was the discussion moved elsewhere? Or is this still a mystery? And if
> > it is: how bad is it, does it need to be fixed before Linus releases 5.16?
> >
> 
> I got to the end of bisect #3 yesterday, the incriminated commit doesn't
> seem to make much sense but I've just re-tested it and there is a clear
> regression between that commit and its parent (unlike bisect #1 and #2):
> 
> 2127d22509aec3a83dffb2a3c736df7ba747a7ce mm, slub: fix two bugs in slab_debug_trace_open()
> write_clat_ns_p99     195395.92     199638.20      4797.01    2.17%
> write_iops             17305.79      17188.24       250.66   -0.68%
> 
> write_clat_ns_p99     195543.84     199996.70      5122.88    2.28%
> write_iops             17300.61      17241.86       251.56   -0.34%
> 
> write_clat_ns_p99     195543.84     200724.48      5122.88    2.65%
> write_iops             17300.61      17246.63       251.56   -0.31%
> 
> write_clat_ns_p99     195543.84     200445.41      5122.88    2.51%
> write_iops             17300.61      17215.47       251.56   -0.49%
> 
> 6d2aec9e123bb9c49cb5c7fc654f25f81e688e8c mm/mempolicy: do not allow illegal MPOL_F_NUMA_BALANCING | MPOL_LOCAL in mbind() 
> write_clat_ns_p99     195395.92     197942.30      4797.01    1.30%
> write_iops             17305.79      17246.56       250.66   -0.34%
> 
> write_clat_ns_p99     195543.84     196183.92      5122.88    0.33%
> write_iops             17300.61      17310.33       251.56    0.06%
> 
> write_clat_ns_p99     195543.84     196990.71      5122.88    0.74%
> write_iops             17300.61      17346.32       251.56    0.26%
> 
> write_clat_ns_p99     195543.84     196362.24      5122.88    0.42%
> write_iops             17300.61      17315.71       251.56    0.09%
> 
> It's pure debug stuff and AFAICT is a correct fix...
> @Josef, could you test that on your side?

Sorry, holidays and all that.  I see 0 difference between the two commits, and
no regression from baseline.  It'll take me a few days to recover from the
holidays, but I'll put some more effort into actively debugging wtf is going on
here on my side since we're all having trouble pinning down what's going on.
Thanks,

Josef

  reply	other threads:[~2022-01-03 16:16 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-11-29 17:03 Josef Bacik
2021-11-29 18:03 ` Valentin Schneider
2021-11-29 18:15   ` Josef Bacik
2021-11-29 18:31     ` Valentin Schneider
2021-11-29 19:49       ` Josef Bacik
2021-11-30  0:26         ` Valentin Schneider
2021-12-03 12:03           ` Valentin Schneider
2021-12-03 19:00             ` Josef Bacik
2021-12-06  9:48               ` Valentin Schneider
2021-12-09 17:22                 ` Valentin Schneider
2021-12-09 19:16                   ` Josef Bacik
2021-12-22 12:42                     ` Thorsten Leemhuis
2021-12-22 16:07                       ` Valentin Schneider
2022-01-03 16:16                         ` Josef Bacik [this message]
2022-01-13 16:41                           ` Valentin Schneider
2022-01-13 16:57                             ` Roman Gushchin
2022-02-18 11:00                               ` Thorsten Leemhuis
2022-02-18 15:34                                 ` Josef Bacik
2021-11-30  7:16 ` Thorsten Leemhuis

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=YdMhQRq1K8tW+S05@localhost.localdomain \
    --to=josef@toxicpanda.com \
    --cc=clm@fb.com \
    --cc=guro@fb.com \
    --cc=linux-btrfs@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=peterz@infradead.org \
    --cc=regressions@leemhuis.info \
    --cc=torvalds@linux-foundation.org \
    --cc=valentin.schneider@arm.com \
    --cc=vincent.guittot@linaro.org \
    --subject='Re: [REGRESSION] 5-10% increase in IO latencies with nohz balance patch' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).