From: Josef Bacik <firstname.lastname@example.org> To: Thorsten Leemhuis <email@example.com> Cc: Roman Gushchin <firstname.lastname@example.org>, Valentin Schneider <email@example.com>, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com, firstname.lastname@example.org, email@example.com Subject: Re: [REGRESSION] 5-10% increase in IO latencies with nohz balance patch Date: Fri, 18 Feb 2022 10:34:46 -0500 [thread overview] Message-ID: <Yg+8lsJw00qEWEza@localhost.localdomain> (raw) In-Reply-To: <firstname.lastname@example.org> On Fri, Feb 18, 2022 at 12:00:41PM +0100, Thorsten Leemhuis wrote: > Hi, this is your Linux kernel regression tracker speaking. Top-posting > for once, to make this easy accessible to everyone. > > FWIW, this is a gentle reminder that I'm still tracking this regression. > Afaics nothing happened in the last few weeks. > > If the discussion continued somewhere else, please let me know; you can > do this directly or simply tell my regression tracking bot yourself by > sending a reply to this mail with a paragraph containing a regzbot > command like "#regzbot monitor > https://email@example.com/" > > If you think there are valid reasons to drop this regressions from the > tracking, let me know; you can do this directly or simply tell my > regression tracking bot yourself by sending a reply to this mail with a > paragraph containing a regzbot command like "#regzbot invalid: Some > explanation" (without the quotes). > > Anyway: I'm putting it on back burner now to reduce the noise, as this > afaics is less important than other regressions: > > #regzbot backburner: Culprit is hard to track down > #regzbot poke > > You likely get two more mails like this after the next two merge > windows, then I'll drop it if I don't here anything back. > > Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) > > P.S.: As the Linux kernel's regression tracker I'm getting a lot of > reports on my table. I can only look briefly into most of them and lack > knowledge about most of the areas they concern. I thus unfortunately > will sometimes get things wrong or miss something important. I hope > that's not the case here; if you think it is, don't hesitate to tell me > in a public reply, it's in everyone's interest to set the public record > straight. > > Roman and I sat down to mess with this some more and had some weird observations. On our Facebook internal boxes we couldn't reproduce. If we disable all the normal FB specific stuff so the box is "quiet" 5.16 performs better. However these are all single socket machines with stupid high numbers of cores, my local machine is 2 socket 6 cores. On my box it was actually pretty noisy testing in isolation as well. In the end I rigged up fsperf to run 1000 runs and graph each kernel on top of eachother. What came out was really strange. 1. The "good" kernel had a period for the first ~100 runs that were very low, the p50 was ~9000ns, but after those first 100 runs it jumped up and was right ontop of 5.16. This explains why it shows up on my overnight tests, the box literally reboots and runs tests. So there's a "warmup" period for the scheduler, once it's been hammered on enough it matches 5.16 exactly, otherwise its faster at the beginning. 2. The regression essentially disappears looking at the graphs over 1000 runs. The results are so jittery this was the only way we could honestly look at the results and see anything. The only place the "regression" shows up is in the write completion latency p99. There 5.15 ranges between 75000-85000 ns, whereas 5.16 ranges between 80000 and 100000 ns. However again this is only on my machine, and the p50 latencies and the actual bw_bytes is the same. Given this test is relatively bursty anyways and the fact that we can't reproduce it internally, and the fact that 5.16 actually consistently performs better internally has convinced us to drop this, it's simply too noisy to get a handle on to actually call it a problem. #regzbot invalid: test too noisy and the results aren't clear cut. Thanks, Josef
next prev parent reply other threads:[~2022-02-18 15:34 UTC|newest] Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-11-29 17:03 Josef Bacik 2021-11-29 18:03 ` Valentin Schneider 2021-11-29 18:15 ` Josef Bacik 2021-11-29 18:31 ` Valentin Schneider 2021-11-29 19:49 ` Josef Bacik 2021-11-30 0:26 ` Valentin Schneider 2021-12-03 12:03 ` Valentin Schneider 2021-12-03 19:00 ` Josef Bacik 2021-12-06 9:48 ` Valentin Schneider 2021-12-09 17:22 ` Valentin Schneider 2021-12-09 19:16 ` Josef Bacik 2021-12-22 12:42 ` Thorsten Leemhuis 2021-12-22 16:07 ` Valentin Schneider 2022-01-03 16:16 ` Josef Bacik 2022-01-13 16:41 ` Valentin Schneider 2022-01-13 16:57 ` Roman Gushchin 2022-02-18 11:00 ` Thorsten Leemhuis 2022-02-18 15:34 ` Josef Bacik [this message] 2021-11-30 7:16 ` Thorsten Leemhuis
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=Yg+8lsJw00qEWEza@localhost.localdomain \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --subject='Re: [REGRESSION] 5-10% increase in IO latencies with nohz balance patch' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).