From: NeilBrown <neilb@suse.com>
To: Jinpu Wang <jinpu.wang@cloud.ionos.com>,
linux-raid <linux-raid@vger.kernel.org>
Cc: Alexandr Iarygin <alexandr.iarygin@cloud.ionos.com>,
Guoqing Jiang <guoqing.jiang@cloud.ionos.com>,
Paul Menzel <pmenzel@molgen.mpg.de>,
linux-kernel@vger.kernel.org
Subject: Re: Bisected: Kernel 4.14 + has 3 times higher write IO latency than Kernel 4.4 with raid1
Date: Tue, 06 Aug 2019 09:46:27 +1000 [thread overview]
Message-ID: <87h86vjhv0.fsf@notabene.neil.brown.name> (raw)
In-Reply-To: <CAMGffEkcXcQC+kjwdH0iVSrFDk-o+dp+b3Q1qz4z=R=6D+QqLQ@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1284 bytes --]
On Mon, Aug 05 2019, Jinpu Wang wrote:
> Hi Neil,
>
> For the md higher write IO latency problem, I bisected it to these commits:
>
> 4ad23a97 MD: use per-cpu counter for writes_pending
> 210f7cd percpu-refcount: support synchronous switch to atomic mode.
>
> Do you maybe have an idea? How can we fix it?
Hmmm.... not sure.
My guess is that the set_in_sync() call from md_check_recovery()
is taking a long time, and is being called too often.
Could you try two experiments please.
1/ set /sys/block/md0/md/safe_mode_delay
to 20 or more. It defaults to about 0.2.
2/ comment out the call the set_in_sync() in md_check_recovery().
Then run the least separately after each of these changes.
I the second one makes a difference, I'd like to know how often it gets
called - and why. The test
if ( ! (
(mddev->sb_flags & ~ (1<<MD_SB_CHANGE_PENDING)) ||
test_bit(MD_RECOVERY_NEEDED, &mddev->recovery) ||
test_bit(MD_RECOVERY_DONE, &mddev->recovery) ||
(mddev->external == 0 && mddev->safemode == 1) ||
(mddev->safemode == 2
&& !mddev->in_sync && mddev->recovery_cp == MaxSector)
))
return;
should normally return when doing lots of IO - I'd like to know
which condition causes it to not return.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]
next prev parent reply other threads:[~2019-08-05 23:46 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-08-02 14:48 Kernel 4.14 + has 100 times higher IO latency than Kernel 4.4 with raid1 Jinpu Wang
2019-08-02 19:52 ` Paul Menzel
2019-08-05 8:34 ` Jinpu Wang
2019-08-05 11:55 ` Bisected: Kernel 4.14 + has 3 times higher write " Jinpu Wang
2019-08-05 23:46 ` NeilBrown [this message]
2019-08-06 4:57 ` riccardofarabia
2019-08-06 7:03 ` Unrelated question and threading (was: Bisected: Kernel 4.14 + has 3 times higher write IO latency than Kernel 4.4 with raid1) Paul Menzel
2019-08-06 7:54 ` Bisected: Kernel 4.14 + has 3 times higher write IO latency than Kernel 4.4 with raid1 Jinpu Wang
2019-08-06 11:57 ` Jinpu Wang
2019-08-06 23:40 ` NeilBrown
2019-08-07 6:36 ` Jinpu Wang
2019-08-07 12:35 ` Jinpu Wang
2019-08-16 8:10 ` Jinpu Wang
2019-08-20 0:27 ` NeilBrown
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87h86vjhv0.fsf@notabene.neil.brown.name \
--to=neilb@suse.com \
--cc=alexandr.iarygin@cloud.ionos.com \
--cc=guoqing.jiang@cloud.ionos.com \
--cc=jinpu.wang@cloud.ionos.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-raid@vger.kernel.org \
--cc=pmenzel@molgen.mpg.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).