* heavy IO on nearly idle RAID1
@ 2024-03-17 10:31 Michael Reinelt
2024-03-18 1:33 ` Yu Kuai
0 siblings, 1 reply; 9+ messages in thread
From: Michael Reinelt @ 2024-03-17 10:31 UTC (permalink / raw)
To: linux-raid
Hallo all,
I have a very strange behaviour on my RAID-1 array with Kernel 6.6.13.
Kernel 6.1.76 and earlier do not show this symptoms, but I *think* I
have seen it on any 6.5 Kernel (I am not sure)
the array contains my /home, and consists of an internal NVME drive, a
internal SATA device, and an external USB drive.
even if the array is (neraly) idle, I see very heave I/O on the
external USB drive, which makes the system more or less unusable.
artus:~ # iostat -p sda,sdb,nvme0n1,md0,md1 -y 5
Linux 6.6.13+bpo-amd64 (artus) 2024-03-17 _x86_64_ (12 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0,12 0,00 0,15 0,45 0,00 99,28
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
md0 2,20 0,00 8,80 0,00 0 44 0
md1 0,00 0,00 0,00 0,00 0 0 0
nvme0n1 8,00 0,00 39,10 0,00 0 195 0
nvme0n1p1 0,00 0,00 0,00 0,00 0 0 0
nvme0n1p2 4,40 0,00 29,60 0,00 0 148 0
nvme0n1p3 3,60 0,00 9,50 0,00 0 47 0
nvme0n1p4 0,00 0,00 0,00 0,00 0 0 0
sda 3,80 0,00 9,50 0,00 0 47 0
sda1 3,80 0,00 9,50 0,00 0 47 0
sda2 0,00 0,00 0,00 0,00 0 0 0
sdb 54,20 0,00 26223,10 0,00 0 131115 0
sdb1 54,20 0,00 26223,10 0,00 0 131115 0
sdb2 0,00 0,00 0,00 0,00 0 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
0,20 0,00 0,12 0,37 0,00 99,32
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
md0 0,20 0,00 0,80 0,00 0 4 0
md1 0,00 0,00 0,00 0,00 0 0 0
nvme0n1 1,60 0,00 3,70 0,00 0 18 0
nvme0n1p1 0,00 0,00 0,00 0,00 0 0 0
nvme0n1p2 0,40 0,00 2,40 0,00 0 12 0
nvme0n1p3 1,20 0,00 1,30 0,00 0 6 0
nvme0n1p4 0,00 0,00 0,00 0,00 0 0 0
sda 1,20 0,00 1,30 0,00 0 6 0
sda1 1,20 0,00 1,30 0,00 0 6 0
sda2 0,00 0,00 0,00 0,00 0 0 0
sdb 39,00 0,00 19661,50 0,00 0 98307 0
sdb1 39,00 0,00 19661,50 0,00 0 98307 0
sdb2 0,00 0,00 0,00 0,00 0 0 0
I am sure that there is no rebuild or check at the moment... and I have
no idea WTF is going on here... sometimes the write rate goes up to
>100 MB/sec
when I check with iotop, i can see similar high write rates, but no
process or thread responsible for it.
Some more information:
artus:~ # uname -a
Linux artus 6.6.13+bpo-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.6.13-1~bpo12+1 (2024-02-15) x86_64 GNU/Linux
artus:~ # mdadm --version
mdadm - v4.2 - 2021-12-30 - Debian 4.2-5
artus:~ # cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid1 sda2[2](W) sdb2[4]
1919826944 blocks super 1.2 [2/2] [UU]
bitmap: 0/15 pages [0KB], 65536KB chunk
md0 : active raid1 sda1[5](W) nvme0n1p3[3] sdb1[4](W)
33520640 blocks super 1.2 [3/3] [UUU]
bitmap: 1/1 pages [4KB], 65536KB chunk
unused devices: <none>
note: I see similar behaviour on /dev/md1 which is a quite large device
containing virtual machines
artus:~ # smartctl -H -i -l scterc /dev/sda
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.6.13+bpo-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Samsung based SSDs
Device Model: Samsung SSD 870 EVO 2TB
Serial Number: S6P4NF0W307661D
LU WWN Device Id: 5 002538 f43329aaa
Firmware Version: SVT02B6Q
User Capacity: 2.000.398.934.016 bytes [2,00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
TRIM Command: Available, deterministic, zeroed
Device is: In smartctl database 7.3/5319
ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sun Mar 17 11:06:23 2024 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: Disabled
Write: Disabled
artus:~ # smartctl -d sat,auto -H -i -l scterc /dev/sdb
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.6.13+bpo-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: Samsung
Product: PSSD T7
Revision: 0
Compliance: SPC-4
User Capacity: 2.000.398.934.016 bytes [2,00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: Solid State Device
Logical Unit id: 0x5000000000000001
Serial number: K611523T0SNDT5S
Device type: disk
Local Time is: Sun Mar 17 11:13:59 2024 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Disabled or Not Supported
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
artus:~ # smartctl -H -i -l scterc /dev/nvme0n1
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.6.13+bpo-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: TS512GMTE400S
Serial Number: I216681028
Firmware Version: V0804S3
PCI Vendor/Subsystem ID: 0x1d79
IEEE OUI Identifier: 0x7c3548
Controller ID: 1
NVMe Version: 1.3
Number of Namespaces: 1
Namespace 1 Size/Capacity: 512.110.190.592 [512 GB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 7c3548 52255b6444
Local Time is: Sun Mar 17 11:08:56 2024 CET
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
artus:~ # mdadm --examine /dev/sda1
/dev/sda1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : f6189e48:e5dadfbd:8a7a9239:c6074410
Name : any:home
Creation Time : Fri Nov 15 07:07:21 2019
Raid Level : raid1
Raid Devices : 3
Avail Dev Size : 67041280 sectors (31.97 GiB 34.33 GB)
Array Size : 33520640 KiB (31.97 GiB 34.33 GB)
Data Offset : 67584 sectors
Super Offset : 8 sectors
Unused Space : before=67504 sectors, after=0 sectors
State : clean
Device UUID : 9023e389:b12c69be:abd226c2:fceba209
Internal Bitmap : 8 sectors from superblock
Flags : write-mostly
Update Time : Sun Mar 17 11:15:22 2024
Bad Block Log : 512 entries available at offset 16 sectors
Checksum : 7958e05b - correct
Events : 864895
Device Role : Active device 2
Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
artus:~ # mdadm --examine /dev/sdb1
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : f6189e48:e5dadfbd:8a7a9239:c6074410
Name : any:home
Creation Time : Fri Nov 15 07:07:21 2019
Raid Level : raid1
Raid Devices : 3
Avail Dev Size : 67041280 sectors (31.97 GiB 34.33 GB)
Array Size : 33520640 KiB (31.97 GiB 34.33 GB)
Data Offset : 67584 sectors
Super Offset : 8 sectors
Unused Space : before=67504 sectors, after=0 sectors
State : clean
Device UUID : a5a29aca:ebfaa7b1:c484660d:bb455305
Internal Bitmap : 8 sectors from superblock
Flags : write-mostly
Update Time : Sun Mar 17 11:15:34 2024
Bad Block Log : 512 entries available at offset 16 sectors
Checksum : f43f398c - correct
Events : 864895
Device Role : Active device 1
Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
artus:~ # mdadm --examine /dev/nvme0n1p3
/dev/nvme0n1p3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : f6189e48:e5dadfbd:8a7a9239:c6074410
Name : any:home
Creation Time : Fri Nov 15 07:07:21 2019
Raid Level : raid1
Raid Devices : 3
Avail Dev Size : 67041280 sectors (31.97 GiB 34.33 GB)
Array Size : 33520640 KiB (31.97 GiB 34.33 GB)
Data Offset : 67584 sectors
Super Offset : 8 sectors
Unused Space : before=67504 sectors, after=0 sectors
State : clean
Device UUID : dae5b41c:8eaceaae:d65a1486:819723c1
Internal Bitmap : 8 sectors from superblock
Update Time : Sun Mar 17 11:16:38 2024
Bad Block Log : 512 entries available at offset 16 sectors
Checksum : 781a567b - correct
Events : 864895
Device Role : Active device 0
Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
artus:~ # mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Fri Nov 15 07:07:21 2019
Raid Level : raid1
Array Size : 33520640 (31.97 GiB 34.33 GB)
Used Dev Size : 33520640 (31.97 GiB 34.33 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Sun Mar 17 11:19:35 2024
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Consistency Policy : bitmap
Name : any:home
UUID : f6189e48:e5dadfbd:8a7a9239:c6074410
Events : 864895
Number Major Minor RaidDevice State
3 259 3 0 active sync /dev/nvme0n1p3
4 8 17 1 active sync writemostly /dev/sdb1
5 8 1 2 active sync writemostly /dev/sda1
Any hints?
If you need more information, please let me know.
sunny greetings from Austria, Michael
--
Michael Reinelt <michael@reinelt.co.at>
Ringsiedlung 75
A-8111 Gratwein-Straßengel
+43 676 3079941
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: heavy IO on nearly idle RAID1
2024-03-17 10:31 heavy IO on nearly idle RAID1 Michael Reinelt
@ 2024-03-18 1:33 ` Yu Kuai
2024-03-18 5:57 ` Michael Reinelt
0 siblings, 1 reply; 9+ messages in thread
From: Yu Kuai @ 2024-03-18 1:33 UTC (permalink / raw)
To: Michael Reinelt, linux-raid, yukuai (C)
Hi,
在 2024/03/17 18:31, Michael Reinelt 写道:
> when I check with iotop, i can see similar high write rates, but no
> process or thread responsible for it.
you might need to learn some tools like blktrace or bpftrace to find out
which thread is issuing IO to sdb1.
Thanks,
Kuai
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: heavy IO on nearly idle RAID1
2024-03-18 1:33 ` Yu Kuai
@ 2024-03-18 5:57 ` Michael Reinelt
2024-03-19 14:08 ` Michael Reinelt
0 siblings, 1 reply; 9+ messages in thread
From: Michael Reinelt @ 2024-03-18 5:57 UTC (permalink / raw)
To: Yu Kuai, linux-raid, yukuai (C)
Am Montag, dem 18.03.2024 um 09:33 +0800 schrieb Yu Kuai:
> you might need to learn some tools like blktrace or bpftrace to find out
> which thread is issuing IO to sdb1.
Thnaks for the hint, I'll play around with these tools.
Some other musings: as this is a RAID-1 array, and both sda1 and sdb1 are "identical" (both are
flagged with write-mostly), I *should* see identical write patterns to sda1 and sdb1?
If we look at my iostat output from above:
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
md0 2,20 0,00 8,80 0,00 0 44 0
nvme0n1p3 3,60 0,00 9,50 0,00 0 47 0
sda1 3,80 0,00 9,50 0,00 0 47 0
sdb1 54,20 0,00 26223,10 0,00 0 131115 0
44 kb have been written to md0, the md subsystem converts these to writes to the RAID members (plus
some overhead like bitmaps and stuff)
The 47 kb written to nvme and sda1 is what I'd expect to see. But the 130 MB to sdb1 are wrong...
btw, when I run this tests on kernel 6.1.76, I get identical writes to all RAID members:
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
md0 5,40 0,00 67,20 0,00 0 336 0
nvme0n1p3 3,40 0,00 68,00 0,00 0 340 0
sda1 3,40 0,00 68,00 0,00 0 340 0
sdb1 3,40 0,00 68,00 0,00 0 340 0
Wild guess: the (external) USB device sdb1 is using a huge "transfer size", so when only a few
sectors are written to sda1, megabytes are written to sdb1?
How could I proove this?
thanks, Michael
--
Michael Reinelt <michael@reinelt.co.at>
Ringsiedlung 75
A-8111 Gratwein-Straßengel
+43 676 3079941
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: heavy IO on nearly idle RAID1
2024-03-18 5:57 ` Michael Reinelt
@ 2024-03-19 14:08 ` Michael Reinelt
2024-03-19 17:35 ` Roger Heflin
2024-03-19 18:05 ` Roman Mamedov
0 siblings, 2 replies; 9+ messages in thread
From: Michael Reinelt @ 2024-03-19 14:08 UTC (permalink / raw)
To: linux-raid
I think I found at least a workaround: the strange behaviour disappears immediately, if I disable
UAS, and use usb-storage for the externel USB drive.
options usb-storage quirks=04e8:4001:u
I am sure that UAS has been used with kernel 6.1, too, where it did not cause any issues...
Ideas what is going wrong in kernel 6.6? I'd like to re-enable UAS, because UAS is about 200 MB/sec
faster than usb-storage
regards, Michael
--
Michael Reinelt <michael@reinelt.co.at>
Ringsiedlung 75
A-8111 Gratwein-Straßengel
+43 676 3079941
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: heavy IO on nearly idle RAID1
2024-03-19 14:08 ` Michael Reinelt
@ 2024-03-19 17:35 ` Roger Heflin
2024-03-19 17:43 ` Michael Reinelt
2024-03-19 18:05 ` Roman Mamedov
1 sibling, 1 reply; 9+ messages in thread
From: Roger Heflin @ 2024-03-19 17:35 UTC (permalink / raw)
To: Michael Reinelt; +Cc: linux-raid
It is possible that for UAS the counters are purely wrong and that no
IO is really happening.
I have seen the perf counters be wrong in a number of ways for a
number of devices and often no one notices that the counters are
wrong.
And since the counters being wrong does not really affect it working
or not this is often missed.
If it is a spinning disk you might try listening to it and see if it
is really doing something.
On Tue, Mar 19, 2024 at 9:09 AM Michael Reinelt <michael@reinelt.co.at> wrote:
>
> I think I found at least a workaround: the strange behaviour disappears immediately, if I disable
> UAS, and use usb-storage for the externel USB drive.
>
> options usb-storage quirks=04e8:4001:u
>
> I am sure that UAS has been used with kernel 6.1, too, where it did not cause any issues...
>
> Ideas what is going wrong in kernel 6.6? I'd like to re-enable UAS, because UAS is about 200 MB/sec
> faster than usb-storage
>
>
> regards, Michael
>
> --
> Michael Reinelt <michael@reinelt.co.at>
> Ringsiedlung 75
> A-8111 Gratwein-Straßengel
> +43 676 3079941
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: heavy IO on nearly idle RAID1
2024-03-19 17:35 ` Roger Heflin
@ 2024-03-19 17:43 ` Michael Reinelt
2024-03-19 17:48 ` Paul Menzel
0 siblings, 1 reply; 9+ messages in thread
From: Michael Reinelt @ 2024-03-19 17:43 UTC (permalink / raw)
To: linux-raid
Good Point, thanks!
but I doubt that:
- I see heavy flickering of the USB Drive LED
- the system has a very bad "responsiveness", it kind of "freezes" very often
none of these I see neither with Kernel 6.1 nor with 6.6 / UAS disabled
Michael
--
Michael Reinelt <michael@reinelt.co.at>
Ringsiedlung 75
A-8111 Gratwein-Straßengel
+43 676 3079941
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: heavy IO on nearly idle RAID1
2024-03-19 17:43 ` Michael Reinelt
@ 2024-03-19 17:48 ` Paul Menzel
2024-03-24 11:05 ` Michael Reinelt
0 siblings, 1 reply; 9+ messages in thread
From: Paul Menzel @ 2024-03-19 17:48 UTC (permalink / raw)
To: Michael Reinelt; +Cc: linux-raid
Dear Michael,
I am sorry, that you hit a regression.
Am 19.03.24 um 18:43 schrieb Michael Reinelt:
> Good Point, thanks!
>
> but I doubt that:
>
> - I see heavy flickering of the USB Drive LED
> - the system has a very bad "responsiveness", it kind of "freezes"
> very often
>
> none of these I see neither with Kernel 6.1 nor with 6.6 / UAS
> disabled
As you can reproduce this, and it works with an earlier version, the
fastest way to resolve the issue is unfortunately to bisect the issue to
find the commit causing the regression.
Kind regards,
Paul
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: heavy IO on nearly idle RAID1
2024-03-19 14:08 ` Michael Reinelt
2024-03-19 17:35 ` Roger Heflin
@ 2024-03-19 18:05 ` Roman Mamedov
1 sibling, 0 replies; 9+ messages in thread
From: Roman Mamedov @ 2024-03-19 18:05 UTC (permalink / raw)
To: Michael Reinelt; +Cc: linux-raid
On Tue, 19 Mar 2024 15:08:57 +0100
Michael Reinelt <michael@reinelt.co.at> wrote:
> I think I found at least a workaround: the strange behaviour disappears immediately, if I disable
> UAS, and use usb-storage for the externel USB drive.
>
> options usb-storage quirks=04e8:4001:u
>
> I am sure that UAS has been used with kernel 6.1, too, where it did not cause any issues...
>
> Ideas what is going wrong in kernel 6.6? I'd like to re-enable UAS, because UAS is about 200 MB/sec
> faster than usb-storage
I think it might be related to discard or write zeroes support on 6.6. I had
some issues enabling USB TRIM on kernel 6.6, compared to 6.1.
What do you get for "lsblk -D" on both kernels and both storage drivers on 6.6,
are there any differences?
Aside from that, trying "blktrace" was a good suggestion to figure out the
process writing or even the content of what is being written.
--
With respect,
Roman
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: heavy IO on nearly idle RAID1
2024-03-19 17:48 ` Paul Menzel
@ 2024-03-24 11:05 ` Michael Reinelt
0 siblings, 0 replies; 9+ messages in thread
From: Michael Reinelt @ 2024-03-24 11:05 UTC (permalink / raw)
To: linux-raid
Am Dienstag, dem 19.03.2024 um 18:48 +0100 schrieb Paul Menzel:
> As you can reproduce this, and it works with an earlier version, the
> fastest way to resolve the issue is unfortunately to bisect the issue to
> find the commit causing the regression.
I agree, but bisecting between kernel 6.1.76 and 6.6.13 sounds like a bit of work, doesn't it? :-(
As this happens on my computer that I need for work every day (and night :-), it makes it even more
complicated. I could try to set up another system (hardware available, but I'd have to buy a SSD for
it), but this will take some time...
Am Dienstag, dem 19.03.2024 um 23:05 +0500 schrieb Roman Mamedov:
> I think it might be related to discard or write zeroes support on 6.6. I had
> some issues enabling USB TRIM on kernel 6.6, compared to 6.1.
>
> What do you get for "lsblk -D" on both kernels and both storage drivers on 6.6,
> are there any differences?
I tried 6.1 and 6.6 both with UAS enabled/disabled, and I get identical results:
NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO
sda 0 512B 2G 0
├─sda1 0 512B 2G 0
│ └─md0 0 512B 2G 0
└─sda2 0 512B 2G 0
└─md1 0 512B 2G 0
sdb 0 0B 0B 0
├─sdb1 0 0B 0B 0
│ └─md0 0 512B 2G 0
└─sdb2 0 0B 0B 0
└─md1 0 512B 2G 0
sdc 0 0B 0B 0
nvme0n1 0 512B 2T 0
├─nvme0n1p1 0 512B 2T 0
├─nvme0n1p2 0 512B 2T 0
├─nvme0n1p3 0 512B 2T 0
│ └─md0 0 512B 2G 0
└─nvme0n1p4 0 512B 2T 0
> Aside from that, trying "blktrace" was a good suggestion to figure out the
> process writing or even the content of what is being written.
I tried to understand blktrace, but failed :-) I've never worked with this tool...
Can someone give me advices how to use it, and which results you are interested in?
greetings, Michael
--
Michael Reinelt <michael@reinelt.co.at>
Ringsiedlung 75
A-8111 Gratwein-Straßengel
+43 676 3079941
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2024-03-24 11:05 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-03-17 10:31 heavy IO on nearly idle RAID1 Michael Reinelt
2024-03-18 1:33 ` Yu Kuai
2024-03-18 5:57 ` Michael Reinelt
2024-03-19 14:08 ` Michael Reinelt
2024-03-19 17:35 ` Roger Heflin
2024-03-19 17:43 ` Michael Reinelt
2024-03-19 17:48 ` Paul Menzel
2024-03-24 11:05 ` Michael Reinelt
2024-03-19 18:05 ` Roman Mamedov
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.