From: Jinpu Wang <jinpu.wang@cloud.ionos.com>
To: NeilBrown <neilb@suse.com>
Cc: Neil F Brown <nfbrown@suse.com>,
Alexandr Iarygin <alexandr.iarygin@cloud.ionos.com>,
Guoqing Jiang <guoqing.jiang@cloud.ionos.com>,
Paul Menzel <pmenzel@molgen.mpg.de>,
linux-kernel@vger.kernel.org,
linux-raid <linux-raid@vger.kernel.org>
Subject: Re: Bisected: Kernel 4.14 + has 3 times higher write IO latency than Kernel 4.4 with raid1
Date: Wed, 7 Aug 2019 08:36:51 +0200 [thread overview]
Message-ID: <CAMGffE=cpxumr0QqJsiGGKpmZr+4a0BiCx3n0_twa5KPs=yX1g@mail.gmail.com> (raw)
In-Reply-To: <87blx1kglx.fsf@notabene.neil.brown.name>
On Wed, Aug 7, 2019 at 1:40 AM NeilBrown <neilb@suse.com> wrote:
>
> On Tue, Aug 06 2019, Jinpu Wang wrote:
>
> > On Tue, Aug 6, 2019 at 9:54 AM Jinpu Wang <jinpu.wang@cloud.ionos.com> wrote:
> >>
> >> On Tue, Aug 6, 2019 at 1:46 AM NeilBrown <neilb@suse.com> wrote:
> >> >
> >> > On Mon, Aug 05 2019, Jinpu Wang wrote:
> >> >
> >> > > Hi Neil,
> >> > >
> >> > > For the md higher write IO latency problem, I bisected it to these commits:
> >> > >
> >> > > 4ad23a97 MD: use per-cpu counter for writes_pending
> >> > > 210f7cd percpu-refcount: support synchronous switch to atomic mode.
> >> > >
> >> > > Do you maybe have an idea? How can we fix it?
> >> >
> >> > Hmmm.... not sure.
> >> Hi Neil,
> >>
> >> Thanks for reply, detailed result in line.
>
> Thanks for the extra testing.
> ...
> > [ 105.133299] md md0 in_sync is 0, sb_flags 2, recovery 3, external
> > 0, safemode 0, recovery_cp 524288
> ...
>
> ahh - the resync was still happening. That explains why set_in_sync()
> is being called so often. If you wait for sync to complete (or create
> the array with --assume-clean) you should see more normal behaviour.
I've updated my tests accordingly, thanks for the hint.
>
> This patch should fix it. I think we can do better but it would be more
> complex so no suitable for backports to -stable.
>
> Once you confirm it works, I'll send it upstream with a
> Reported-and-Tested-by from you.
>
> Thanks,
> NeilBrown
Thanks a lot, Neil, my quick test show, yes, it fixed the problem for me.
I will run more tests to be sure, will report back the test result.
Regards,
Jack Wang
>
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 24638ccedce4..624cf1ac43dc 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -8900,6 +8900,7 @@ void md_check_recovery(struct mddev *mddev)
>
> if (mddev_trylock(mddev)) {
> int spares = 0;
> + bool try_set_sync = mddev->safemode != 0;
>
> if (!mddev->external && mddev->safemode == 1)
> mddev->safemode = 0;
> @@ -8945,7 +8946,7 @@ void md_check_recovery(struct mddev *mddev)
> }
> }
>
> - if (!mddev->external && !mddev->in_sync) {
> + if (try_set_sync && !mddev->external && !mddev->in_sync) {
> spin_lock(&mddev->lock);
> set_in_sync(mddev);
> spin_unlock(&mddev->lock);
next prev parent reply other threads:[~2019-08-07 6:37 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-08-02 14:48 Kernel 4.14 + has 100 times higher IO latency than Kernel 4.4 with raid1 Jinpu Wang
2019-08-02 19:52 ` Paul Menzel
2019-08-05 8:34 ` Jinpu Wang
2019-08-05 11:55 ` Bisected: Kernel 4.14 + has 3 times higher write " Jinpu Wang
2019-08-05 23:46 ` NeilBrown
2019-08-06 4:57 ` riccardofarabia
2019-08-06 7:03 ` Unrelated question and threading (was: Bisected: Kernel 4.14 + has 3 times higher write IO latency than Kernel 4.4 with raid1) Paul Menzel
2019-08-06 7:54 ` Bisected: Kernel 4.14 + has 3 times higher write IO latency than Kernel 4.4 with raid1 Jinpu Wang
2019-08-06 11:57 ` Jinpu Wang
2019-08-06 23:40 ` NeilBrown
2019-08-07 6:36 ` Jinpu Wang [this message]
2019-08-07 12:35 ` Jinpu Wang
2019-08-16 8:10 ` Jinpu Wang
2019-08-20 0:27 ` NeilBrown
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAMGffE=cpxumr0QqJsiGGKpmZr+4a0BiCx3n0_twa5KPs=yX1g@mail.gmail.com' \
--to=jinpu.wang@cloud.ionos.com \
--cc=alexandr.iarygin@cloud.ionos.com \
--cc=guoqing.jiang@cloud.ionos.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-raid@vger.kernel.org \
--cc=neilb@suse.com \
--cc=nfbrown@suse.com \
--cc=pmenzel@molgen.mpg.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).