From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8D67C433FF for ; Tue, 6 Aug 2019 23:40:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id BD646208C3 for ; Tue, 6 Aug 2019 23:40:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726889AbfHFXkj (ORCPT ); Tue, 6 Aug 2019 19:40:39 -0400 Received: from mx2.suse.de ([195.135.220.15]:35848 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726133AbfHFXkj (ORCPT ); Tue, 6 Aug 2019 19:40:39 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id EBFD5AD12; Tue, 6 Aug 2019 23:40:36 +0000 (UTC) From: NeilBrown To: Jinpu Wang , Neil F Brown Date: Wed, 07 Aug 2019 09:40:26 +1000 Cc: Alexandr Iarygin , Guoqing Jiang , Paul Menzel , linux-kernel@vger.kernel.org, linux-raid Subject: Re: Bisected: Kernel 4.14 + has 3 times higher write IO latency than Kernel 4.4 with raid1 In-Reply-To: References: <0a83fde3-1a74-684c-0d70-fb44b9021f96@molgen.mpg.de> <87h86vjhv0.fsf@notabene.neil.brown.name> Message-ID: <87blx1kglx.fsf@notabene.neil.brown.name> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable On Tue, Aug 06 2019, Jinpu Wang wrote: > On Tue, Aug 6, 2019 at 9:54 AM Jinpu Wang wr= ote: >> >> On Tue, Aug 6, 2019 at 1:46 AM NeilBrown wrote: >> > >> > On Mon, Aug 05 2019, Jinpu Wang wrote: >> > >> > > Hi Neil, >> > > >> > > For the md higher write IO latency problem, I bisected it to these c= ommits: >> > > >> > > 4ad23a97 MD: use per-cpu counter for writes_pending >> > > 210f7cd percpu-refcount: support synchronous switch to atomic mode. >> > > >> > > Do you maybe have an idea? How can we fix it? >> > >> > Hmmm.... not sure. >> Hi Neil, >> >> Thanks for reply, detailed result in line. Thanks for the extra testing. ... > [ 105.133299] md md0 in_sync is 0, sb_flags 2, recovery 3, external > 0, safemode 0, recovery_cp 524288 ... ahh - the resync was still happening. That explains why set_in_sync() is being called so often. If you wait for sync to complete (or create the array with --assume-clean) you should see more normal behaviour. This patch should fix it. I think we can do better but it would be more complex so no suitable for backports to -stable. Once you confirm it works, I'll send it upstream with a Reported-and-Tested-by from you. Thanks, NeilBrown diff --git a/drivers/md/md.c b/drivers/md/md.c index 24638ccedce4..624cf1ac43dc 100644 =2D-- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -8900,6 +8900,7 @@ void md_check_recovery(struct mddev *mddev) =20 if (mddev_trylock(mddev)) { int spares =3D 0; + bool try_set_sync =3D mddev->safemode !=3D 0; =20 if (!mddev->external && mddev->safemode =3D=3D 1) mddev->safemode =3D 0; @@ -8945,7 +8946,7 @@ void md_check_recovery(struct mddev *mddev) } } =20 =2D if (!mddev->external && !mddev->in_sync) { + if (try_set_sync && !mddev->external && !mddev->in_sync) { spin_lock(&mddev->lock); set_in_sync(mddev); spin_unlock(&mddev->lock); --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEG8Yp69OQ2HB7X0l6Oeye3VZigbkFAl1KD+oACgkQOeye3VZi gbkOBBAAvAchxMWgyz5FTx4uM4/I228Z2NCgD/+0UHkI6Z46bivBLwpCu3BhqSgX EADIcAm+gfJU569NUFaMBQbTy+m/X3e6GoEaN6hVrilBhNKTOyUSibk6qFTXT9J0 2el59+qsf2fx9MqtwT5ClEMTIIp9P1251snNDMSdIFjJwmEe1tKTselsO2GI4fNJ 0CNBxsUa4dQwDOLx7+7at7dWF26ZRWVWkSp+eaUgtyxc3w2WFroM7xAlIvr6pzwL M/mfLTXNyuSUsB2em/QsXiguKECbtZYDhhQrspYbvS/Z6CZYGalDQGhCv9efeVvQ 7MAGSeo3emoy5fm2JB9z+xqF6KHdG+uK0PNOs7LE22tBiSnPlXW3raGZso2K7arQ H++UvSwkqRTw8GY2vTa0wLJ0cG4I7twXJjxrFQU8Tjk89BUECxryW9TkJHl/AjhU hL6X+itFyKMg7mc2H7B0PqdLz5dkcvnpJbmbs0O3BFNEGGjR+PG+qdAee/fbPmxk p5X2qRocvzUG8AaAlVpYJpaUi++9B5EPxtaGaMEiz4Xa4mEvFzwZ2KXIytJfiy7X mUo/EOGtodMV2L8Rim5hSXTTx29NYh97/PiPFb+GMRwm+txNyopiFi+GyK072GTq 7trxozNthdQLrvEAq9CTa/yyxDHVtEhTL7Mun85HqZIrUZdf15c= =O8+M -----END PGP SIGNATURE----- --=-=-=--