From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.4 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FFEFC4338F for ; Thu, 5 Aug 2021 19:52:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 466AD60720 for ; Thu, 5 Aug 2021 19:52:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239523AbhHETw0 (ORCPT ); Thu, 5 Aug 2021 15:52:26 -0400 Received: from UPDC19PA21.eemsg.mail.mil ([214.24.27.196]:18208 "EHLO UPDC19PA21.eemsg.mail.mil" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231387AbhHETw0 (ORCPT ); Thu, 5 Aug 2021 15:52:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mail.mil; i=@mail.mil; q=dns/txt; s=EEMSG2021v1a; t=1628193131; x=1659729131; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=KqY0LEmrjcN3v6lErD6Ii62+Slw5gLg41OgTo7PpXL4=; b=J2B6+7VSRDUdfwpb4dE4hR9MDgDk1Oh3wyEQlWFfPqrYW43kTVwoUTak tMaIyLZp5n02pREwQpqCFp9d5XZ+26lRW5S3bY3sOUrYjal/8P6na7LQu WSZuuiZFvEF5V2qaWqn0JAdFaNtfahLUP/HHM9VtidG/e7XkmcdrABXoF ZGd3//UIlAK5qqLM6b76YmloloO5cst8LLto6GMWN/jmk5/pJTyQU7J1G 2+7qQkB+ycRteZIIlml1qxmqEU6TiY212qvwKn3m1b4R1J865jKJIvA42 XH3WlooqgXxaB9ambs2t4X+Mw7UYOVesoBLsavt2g7/EZdLS/qavv8wJO w==; X-EEMSG-check-017: 251535518|UPDC19PA21_ESA_OUT03.csd.disa.mil X-IronPort-AV: E=Sophos;i="5.84,296,1620691200"; d="scan'208";a="251535518" Received: from edge-mech02.mail.mil ([214.21.130.230]) by UPDC19PA21.eemsg.mail.mil with ESMTP/TLS/ECDHE-RSA-AES256-SHA384; 05 Aug 2021 19:52:07 +0000 Received: from UMECHPAPB.easf.csd.disa.mil (214.21.130.171) by edge-mech02.mail.mil (214.21.130.230) with Microsoft SMTP Server (TLS) id 14.3.498.0; Thu, 5 Aug 2021 19:52:01 +0000 Received: from UMECHPA7B.easf.csd.disa.mil ([169.254.8.164]) by umechpapb.easf.csd.disa.mil ([214.21.130.171]) with mapi id 14.03.0513.000; Thu, 5 Aug 2021 19:52:01 +0000 From: "Finlayson, James M CIV (USA)" To: "'linux-raid@vger.kernel.org'" CC: 'Gal Ofri' , "Finlayson, James M CIV (USA)" Subject: RE: [Non-DoD Source] Re: Can't get RAID5/RAID6 NVMe randomread IOPS - AMD ROME what am I missing????? Thread-Topic: [Non-DoD Source] Re: Can't get RAID5/RAID6 NVMe randomread IOPS - AMD ROME what am I missing????? Thread-Index: AQHXg5T/4rk7hCCa50OEioMjNG53XatlVF4ggAAKU8CAAACJAA== Date: Thu, 5 Aug 2021 19:52:01 +0000 Message-ID: <5EAED86C53DED2479E3E145969315A2385856B25@UMECHPA7B.easf.csd.disa.mil> References: <5EAED86C53DED2479E3E145969315A2385841062@UMECHPA7B.easf.csd.disa.mil> <5EAED86C53DED2479E3E145969315A2385856AD0@UMECHPA7B.easf.csd.disa.mil> <5EAED86C53DED2479E3E145969315A2385856AF7@UMECHPA7B.easf.csd.disa.mil> In-Reply-To: <5EAED86C53DED2479E3E145969315A2385856AF7@UMECHPA7B.easf.csd.disa.mil> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [214.21.44.12] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org Sorry - again..I sent HTML instead of plain text Resend - mailing list bounce=A0=20 All,=20 Sorry for the delay - both work and life got into the way.=A0=A0 Here is so= me feedback: BLUF upfront with 5.14rc3 kernel that our SA built - md0 a 10+1+1 RAID5 - 5= .332 M IOPS 20.3GiB/s, md1 a 10+1+1 RAID5, 5.892M IOPS 22.5GiB/s=A0 - best = hero numbers I've ever seen on mdraid =A0RAID5 IOPS.=A0=A0 I think the kern= el patch is good.=A0 Prior was =A0socket0 1.263M IOPS 4934MiB/s, socket1 1.= 071M IOSP, 4183MiB/s....=A0=A0 I'm willing to help push this as hard as we = can until we hit a bottleneck outside of our control.=A0=A0=20 I need to verify the RAW IOPS - admittedly this is a different server and I= didn't do any regression testing before the kernel, but my raw were=A0 soc= ket0: 13.2M IOPS and socket1 =A013.5M IOPS.=A0=A0 Prior was socket0 16.0M I= OPS and socket1 13.5M IOPS.=A0 =A0- admittedly there appears to a regressio= n in the socket0 "hero run" but what I don't know that since this is a diff= erent server, I don't know if I have a configuration management issue in my= zealousness to test this patch or whether we have a regression.=A0=A0 I wa= s so excited to have the attention of kernel developers that needed my help= that I borrowed another system, because I didn't want to tear apart my "Fr= ankenstein's monster" 32 partition mdraid LVM mess.=A0=A0 If I can switch k= ernels and reboot before work and life get back in the way, I'll follow=A0 = up.. I think I might have to give myself the action to run this to ground next w= eek on the other server.=A0=A0 Without a doubt the mdraid lock improvement = is worth taking forward.=A0=A0 I either have to find my error or point a fi= nger as my raw hero numbers got worse. =A0=A0I tend to see one socket outru= n another - =A0the way HPE allocates the nvme drives to pcie root complexes= =A0is not how I'd like to do it so the drives are unbalanced on the PCIe r= oot complexes (drives are in 4 different root complexes on socket 0 and 3 o= n socket 1, so one would think socket0 will always be faster for hero runs = =A0(an NPS4 numa mapping is the best way to show it:=A0=20 [root@gremlin04 hornet05]# cat *nps4 #filename=3D/dev/nvme0n1 0 #filename=3D/dev/nvme1n1 0 #filename=3D/dev/nvme2n1 1 #filename=3D/dev/nvme3n1 1 #filename=3D/dev/nvme4n1 2 #filename=3D/dev/nvme5n1 2 #filename=3D/dev/nvme6n1 2 #filename=3D/dev/nvme7n1 2 #filename=3D/dev/nvme8n1 3 #filename=3D/dev/nvme9n1 3 #filename=3D/dev/nvme10n1 3 #filename=3D/dev/nvme11n1 3 #filename=3D/dev/nvme12n1 4 #filename=3D/dev/nvme13n1 4 #filename=3D/dev/nvme14n1 4 #filename=3D/dev/nvme15n1 4 #filename=3D/dev/nvme17n1 5 #filename=3D/dev/nvme18n1 5 #filename=3D/dev/nvme19n1 5 #filename=3D/dev/nvme20n1 5 #filename=3D/dev/nvme21n1 6 #filename=3D/dev/nvme22n1 6 #filename=3D/dev/nvme23n1 6 #filename=3D/dev/nvme24n1 6 fio fiojim.hpdl385.nps1 socket0: (g=3D0): rw=3Drandread, bs=3D(R) 4096B-4096B, (W) 4096B-4096B, (T)= 4096B-4096B, ioengine=3Dlibaio, iodepth=3D128 ... socket1: (g=3D1): rw=3Drandread, bs=3D(R) 4096B-4096B, (W) 4096B-4096B, (T)= 4096B-4096B, ioengine=3Dlibaio, iodepth=3D128 ... socket0-md: (g=3D2): rw=3Drandread, bs=3D(R) 4096B-4096B, (W) 4096B-4096B, = (T) 4096B-4096B, ioengine=3Dlibaio, iodepth=3D128 ... socket1-md: (g=3D3): rw=3Drandread, bs=3D(R) 4096B-4096B, (W) 4096B-4096B, = (T) 4096B-4096B, ioengine=3Dlibaio, iodepth=3D128 ... fio-3.26 Starting 256 processes Jobs: 128 (f=3D128): [_(128),r(128)][1.5%][r=3D42.8GiB/s][r=3D11.2M IOPS][e= ta 10h:40m:00s]=A0=A0=A0=A0=A0=A0=A0=20 socket0: (groupid=3D0, jobs=3D64): err=3D 0: pid=3D522428: Thu Aug=A0 5 19:= 33:05 2021 =A0 read: IOPS=3D13.2M, BW=3D50.2GiB/s (53.9GB/s)(14.7TiB/300005msec) =A0=A0=A0 slat (nsec): min=3D1312, max=3D8308.1k, avg=3D2206.72, stdev=3D15= 05.92 =A0=A0=A0 clat (usec): min=3D14, max=3D42033, avg=3D619.56, stdev=3D671.45 =A0=A0=A0=A0 lat (usec): min=3D19, max=3D42045, avg=3D621.83, stdev=3D671.4= 6 =A0=A0=A0 clat percentiles (usec): =A0=A0=A0=A0 |=A0 1.00th=3D[=A0 113],=A0 5.00th=3D[=A0 149], 10.00th=3D[=A0= 180], 20.00th=3D[=A0 229], =A0=A0=A0=A0 | 30.00th=3D[=A0 273], 40.00th=3D[=A0 310], 50.00th=3D[=A0 351= ], 60.00th=3D[=A0 408], =A0=A0=A0 =A0| 70.00th=3D[=A0 578], 80.00th=3D[=A0 938], 90.00th=3D[ 1467],= 95.00th=3D[ 1909], =A0=A0=A0=A0 | 99.00th=3D[ 3163], 99.50th=3D[ 4178], 99.90th=3D[ 5800], 99.= 95th=3D[ 6390], =A0=A0=A0=A0 | 99.99th=3D[ 8455] =A0=A0 bw (=A0 MiB/s): min=3D28741, max=3D61365, per=3D18.56%, avg=3D51489.= 80, stdev=3D82.09, samples=3D38016 =A0=A0 iops=A0=A0=A0=A0=A0=A0=A0 : min=3D7357916, max=3D15709528, avg=3D131= 81362.22, stdev=3D21013.83, samples=3D38016 =A0 lat (usec)=A0=A0 : 20=3D0.01%, 50=3D0.02%, 100=3D0.42%, 250=3D24.52%, 5= 00=3D42.21% =A0 lat (usec)=A0=A0 : 750=3D7.94%, 1000=3D6.34% =A0 lat (msec)=A0=A0 : 2=3D14.26%, 4=3D3.74%, 10=3D0.54%, 20=3D0.01%, 50=3D= 0.01% =A0 cpu=A0=A0=A0=A0=A0=A0=A0=A0=A0 : usr=3D14.58%, sys=3D47.48%, ctx=3D2919= 12925, majf=3D0, minf=3D10492 =A0 IO depths=A0=A0=A0 : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.1%,= 32=3D0.1%, >=3D64=3D100.0% =A0=A0=A0=A0 submit=A0=A0=A0 : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 3= 2=3D0.0%, 64=3D0.0%, >=3D64=3D0.0% =A0=A0=A0=A0 complete=A0 : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D= 0.0%, 64=3D0.0%, >=3D64=3D0.1% =A0=A0=A0=A0 issued rwts: total=3D3949519687,0,0,0 short=3D0,0,0,0 dropped= =3D0,0,0,0 =A0=A0=A0=A0 latency=A0=A0 : target=3D0, window=3D0, percentile=3D100.00%, = depth=3D128 socket1: (groupid=3D1, jobs=3D64): err=3D 0: pid=3D522492: Thu Aug=A0 5 19:= 33:05 2021 =A0 read: IOPS=3D13.6M, BW=3D51.8GiB/s (55.7GB/s)(15.2TiB/300004msec) =A0=A0=A0 slat (nsec): min=3D1323, max=3D4335.7k, avg=3D2242.27, stdev=3D16= 08.25 =A0=A0=A0 clat (usec): min=3D14, max=3D41341, avg=3D600.15, stdev=3D726.62 =A0=A0=A0=A0 lat (usec): min=3D20, max=3D41358, avg=3D602.46, stdev=3D726.6= 4 =A0=A0=A0 clat percentiles (usec): =A0=A0=A0=A0 |=A0 1.00th=3D[=A0 115],=A0 5.00th=3D[=A0 151], 10.00th=3D[=A0= 184], 20.00th=3D[=A0 231], =A0=A0=A0=A0 | 30.00th=3D[=A0 269], 40.00th=3D[=A0 306], 50.00th=3D[=A0 347= ], 60.00th=3D[=A0 400], =A0=A0=A0=A0 | 70.00th=3D[=A0 506], 80.00th=3D[=A0 799], 90.00th=3D[ 1303],= 95.00th=3D[ 1909], =A0=A0=A0=A0 | 99.00th=3D[ 3589], 99.50th=3D[ 4424], 99.90th=3D[ 7111], 99.= 95th=3D[ 7767], =A0=A0=A0=A0 | 99.99th=3D[10290] =A0=A0 bw (=A0 MiB/s): min=3D28663, max=3D71847, per=3D21.11%, avg=3D53145.= 09, stdev=3D111.29, samples=3D38016 =A0=A0 iops=A0=A0=A0=A0=A0=A0=A0 : min=3D7337860, max=3D18392866, avg=3D136= 05117.00, stdev=3D28491.19, samples=3D38016 =A0 lat (usec)=A0=A0 : 20=3D0.01%, 50=3D0.02%, 100=3D0.36%, 250=3D24.52%, 5= 00=3D44.77% =A0 lat (usec)=A0=A0 : 750=3D8.90%, 1000=3D6.37% =A0 lat (msec)=A0=A0 : 2=3D10.52%, 4=3D3.87%, 10=3D0.66%, 20=3D0.01%, 50=3D= 0.01% =A0 cpu=A0=A0=A0=A0=A0=A0=A0=A0=A0 : usr=3D14.86%, sys=3D49.40%, ctx=3D2826= 34154, majf=3D0, minf=3D10276 =A0 IO depths=A0=A0=A0 : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.1%,= 32=3D0.1%, >=3D64=3D100.0% =A0=A0=A0=A0 submit=A0=A0=A0 : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 3= 2=3D0.0%, 64=3D0.0%, >=3D64=3D0.0% =A0 =A0=A0=A0complete=A0 : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D= 0.0%, 64=3D0.0%, >=3D64=3D0.1% =A0=A0=A0=A0 issued rwts: total=3D4076360454,0,0,0 short=3D0,0,0,0 dropped= =3D0,0,0,0 =A0=A0=A0=A0 latency=A0=A0 : target=3D0, window=3D0, percentile=3D100.00%, = depth=3D128 socket0-md: (groupid=3D2, jobs=3D64): err=3D 0: pid=3D524061: Thu Aug=A0 5 = 19:33:05 2021 =A0 read: IOPS=3D5332k, BW=3D20.3GiB/s (21.8GB/s)(6102GiB/300002msec) =A0=A0=A0 slat (nsec): min=3D1633, max=3D17043k, avg=3D11123.38, stdev=3D86= 94.61 =A0=A0=A0 clat (usec): min=3D186, max=3D18705, avg=3D1524.87, stdev=3D115.2= 9 =A0=A0=A0=A0 lat (usec): min=3D200, max=3D18743, avg=3D1536.08, stdev=3D115= .90 =A0=A0=A0 clat percentiles (usec): =A0=A0=A0=A0 |=A0 1.00th=3D[ 1270],=A0 5.00th=3D[ 1336], 10.00th=3D[ 1369],= 20.00th=3D[ 1418], =A0=A0=A0=A0 | 30.00th=3D[ 1467], 40.00th=3D[ 1500], 50.00th=3D[ 1532], 60.= 00th=3D[ 1549], =A0=A0=A0=A0 | 70.00th=3D[ 1582], 80.00th=3D[ 1631], 90.00th=3D[ 1680], 95.= 00th=3D[ 1713], =A0=A0=A0=A0 | 99.00th=3D[ 1795], 99.50th=3D[ 1811], 99.90th=3D[ 1893], 99.= 95th=3D[ 1926], =A0=A0=A0=A0 | 99.99th=3D[ 2089] =A0=A0 bw (=A0 MiB/s): min=3D19030, max=3D21969, per=3D100.00%, avg=3D20843= .43, stdev=3D 5.35, samples=3D38272 =A0=A0 iops=A0=A0=A0=A0=A0=A0=A0 : min=3D4871687, max=3D5624289, avg=3D5335= 900.01, stdev=3D1370.43, samples=3D38272 =A0 lat (usec)=A0=A0 : 250=3D0.01%, 500=3D0.01%, 750=3D0.01%, 1000=3D0.01% =A0 lat (msec)=A0=A0 : 2=3D99.97%, 4=3D0.02%, 10=3D0.01%, 20=3D0.01% =A0 cpu=A0=A0=A0=A0=A0=A0=A0=A0=A0 : usr=3D5.56%, sys=3D77.91%, ctx=3D8118,= majf=3D0, minf=3D9018 =A0 IO depths=A0=A0=A0 : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.1%,= 32=3D0.1%, >=3D64=3D100.0% =A0=A0=A0=A0 submit=A0=A0=A0 : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 3= 2=3D0.0%, 64=3D0.0%, >=3D64=3D0.0% =A0=A0=A0=A0 complete=A0 : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D= 0.0%, 64=3D0.0%, >=3D64=3D0.1% =A0=A0=A0=A0issued rwts: total=3D1599503201,0,0,0 short=3D0,0,0,0 dropped= =3D0,0,0,0 =A0=A0=A0=A0 latency=A0=A0 : target=3D0, window=3D0, percentile=3D100.00%, = depth=3D128 socket1-md: (groupid=3D3, jobs=3D64): err=3D 0: pid=3D524125: Thu Aug=A0 5 = 19:33:05 2021 =A0 read: IOPS=3D5892k, BW=3D22.5GiB/s (24.1GB/s)(6743GiB/300002msec) =A0=A0=A0 slat (nsec): min=3D1663, max=3D1274.1k, avg=3D9896.09, stdev=3D79= 39.50 =A0=A0=A0 clat (usec): min=3D236, max=3D11102, avg=3D1379.86, stdev=3D148.6= 4 =A0=A0=A0=A0 lat (usec): min=3D239, max=3D11110, avg=3D1389.84, stdev=3D149= .54 =A0=A0=A0 clat percentiles (usec): =A0=A0=A0=A0 |=A0 1.00th=3D[ 1106],=A0 5.00th=3D[ 1172], 10.00th=3D[ 1205],= 20.00th=3D[ 1254], =A0=A0=A0=A0 | 30.00th=3D[ 1287], 40.00th=3D[ 1336], 50.00th=3D[ 1369], 60.= 00th=3D[ 1401], =A0=A0=A0=A0 | 70.00th=3D[ 1434], 80.00th=3D[ 1500], 90.00th=3D[ 1582], 95.= 00th=3D[ 1663], =A0=A0=A0=A0 | 99.00th=3D[ 1811], 99.50th=3D[ 1860], 99.90th=3D[ 1942], 99.= 95th=3D[ 1958], =A0=A0=A0=A0 | 99.99th=3D[ 2040] =A0=A0 bw (=A0 MiB/s): min=3D20982, max=3D24535, per=3D-82.15%, avg=3D23034= .61, stdev=3D15.46, samples=3D38272 =A0=A0 iops=A0=A0=A0=A0=A0=A0=A0 : min=3D5371404, max=3D6281119, avg=3D5896= 843.14, stdev=3D3958.21, samples=3D38272 =A0 lat (usec)=A0=A0 : 250=3D0.01%, 500=3D0.01%, 750=3D0.01%, 1000=3D0.01% =A0 lat (msec)=A0=A0 : 2=3D99.97%, 4=3D0.02%, 10=3D0.01%, 20=3D0.01% =A0 cpu=A0=A0=A0=A0=A0=A0=A0=A0=A0 : usr=3D6.55%, sys=3D74.98%, ctx=3D9833,= majf=3D0, minf=3D8956 =A0 IO depths=A0=A0=A0 : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D0.1%,= 32=3D0.1%, >=3D64=3D100.0% =A0=A0=A0=A0 submit=A0=A0=A0 : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 3= 2=3D0.0%, 64=3D0.0%, >=3D64=3D0.0% =A0=A0=A0=A0 complete=A0 : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D= 0.0%, 64=3D0.0%, >=3D64=3D0.1% =A0=A0=A0=A0 issued rwts: total=3D1767618924,0,0,0 short=3D0,0,0,0 dropped= =3D0,0,0,0 =A0=A0=A0=A0 latency=A0=A0 : target=3D0, window=3D0, percentile=3D100.00%, = depth=3D128 Run status group 0 (all jobs): =A0=A0 READ: bw=3D50.2GiB/s (53.9GB/s), 50.2GiB/s-50.2GiB/s (53.9GB/s-53.9G= B/s), io=3D14.7TiB (16.2TB), run=3D300005-300005msec Run status group 1 (all jobs): =A0=A0 READ: bw=3D51.8GiB/s (55.7GB/s), 51.8GiB/s-51.8GiB/s (55.7GB/s-55.7G= B/s), io=3D15.2TiB (16.7TB), run=3D300004-300004msec Run status group 2 (all jobs): =A0=A0 READ: bw=3D20.3GiB/s (21.8GB/s), 20.3GiB/s-20.3GiB/s (21.8GB/s-21.8G= B/s), io=3D6102GiB (6552GB), run=3D300002-300002msec Run status group 3 (all jobs): =A0=A0 READ: bw=3D22.5GiB/s (24.1GB/s), 22.5GiB/s-22.5GiB/s (24.1GB/s-24.1G= B/s), io=3D6743GiB (7240GB), run=3D300002-300002msec Disk stats (read/write): =A0 nvme0n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.00= % =A0 nvme1n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.00= % =A0 nvme2n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.00= % =A0 nvme3n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.00= % =A0 nvme4n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.00= % =A0 nvme5n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.00= % =A0 nvme6n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.00= % =A0 nvme7n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.00= % =A0 nvme8n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.00= % =A0 nvme9n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.00= % =A0 nvme10n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.0= 0% =A0 nvme11n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.0= 0% =A0 nvme12n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.0= 0% =A0 nvme13n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.0= 0% =A0 nvme14n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.0= 0% =A0 nvme15n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.0= 0% =A0 nvme17n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.0= 0% =A0 nvme18n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.0= 0% =A0 nvme19n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.0= 0% =A0 nvme20n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.0= 0% =A0 nvme21n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.0= 0% =A0 nvme22n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.0= 0% =A0 nvme23n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.0= 0% =A0 nvme24n1: ios=3D0/0, merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.0= 0% =A0 md0: ios=3D1599378656/0, merge=3D0/0, ticks=3D391992721/0, in_queue=3D3= 91992721, util=3D100.00% =A0 md1: ios=3D1767484212/0, merge=3D0/0, ticks=3D427666887/0, in_queue=3D4= 27666887, util=3D100.00% From: Gal Ofri =20 Sent: Wednesday, July 28, 2021 5:43 AM To: Finlayson, James M CIV (USA) ; 'linux-= raid@vger.kernel.org' Subject: [Non-DoD Source] Re: Can't get RAID5/RAID6 NVMe randomread IOPS - = AMD ROME what am I missing????? All active links contained in this email were disabled. Please verify the i= dentity of the sender, and confirm the authenticity of all links contained = within the message prior to copying and pasting the address to a Web browse= r.=20 ________________________________________ A recent commit raised the=A0limit on raid5/6 read iops. It's available in 5.14. See=A0Caution-https://github.com/torvalds/linux/commit/97ae27252f4962d0fcc3= 8ee1d9f913d817a2024e=A0<=A0Caution-https://github.com/torvalds/linux/commit= /97ae27252f4962d0fcc38ee1d9f913d817a2024e=A0>=A0 commit 97ae27252f4962d0fcc38ee1d9f913d817a2024e Author: Gal Ofri Date: =A0 Mon Jun 7 14:07:03 2021 +0300 =A0 =A0 md/raid5: avoid device_lock in read_one_chunk() Please do share if you reach more iops in your env than described in the co= mmit. Cheers, Gal,=A0 Volumez (formerly storing.io)