* Slow mounting raid1
@ 2017-07-31 18:30 Leonidas Spyropoulos
2017-08-01 1:12 ` Duncan
0 siblings, 1 reply; 6+ messages in thread
From: Leonidas Spyropoulos @ 2017-07-31 18:30 UTC (permalink / raw)
To: linux-btrfs
Hello,
I got a raid1 setup of btrfs on a HDD array of 2 disks. The fstab has
the following mount settings:
# cat /etc/fstab | grep raid1
UUID=c9db91e6-0ba8-4ae6-b471-8fd4ff7ee72d /media/raid1 btrfs rw,relatime,compress=lzo,space_cache 0 0
When I try to mount the array it's consistent about 5 seconds+
# time umount /media/raid1
real 0m0.358s
user 0m0.010s
sys 0m0.010s
# time mount /media/raid1
real 0m5.605s
user 0m0.504s
sys 0m0.071s
I have this setup for sometime now and from the time I made it the mount
time went up (I notice that on boot). When I first build that was
almost instant. In terms of maintenance I regularly run a scrub and
rebalance every now and then.
Running kernel 4.11.12 (with -ck patchs)
Is there something I can do to speed it up (apart buying 2 SSDs :D ). I
feel like I'm missing something as the usage of the raid is not really
frequent - just backup mainly.
Thanks for your time.
--
Leonidas Spyropoulos
A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Slow mounting raid1
2017-07-31 18:30 Slow mounting raid1 Leonidas Spyropoulos
@ 2017-08-01 1:12 ` Duncan
2017-08-01 6:43 ` Leonidas Spyropoulos
0 siblings, 1 reply; 6+ messages in thread
From: Duncan @ 2017-08-01 1:12 UTC (permalink / raw)
To: linux-btrfs
Leonidas Spyropoulos posted on Mon, 31 Jul 2017 19:30:47 +0100 as
excerpted:
> Hello,
>
> I got a raid1 setup of btrfs on a HDD array of 2 disks. The fstab has
> the following mount settings:
> # cat /etc/fstab | grep raid1
> UUID=c9db91e6-0ba8-4ae6-b471-8fd4ff7ee72d /media/raid1 btrfs
> rw,relatime,compress=lzo,space_cache 0 0
If you're doing any snapshotting, you almost certainly want noatime, not
the default relatime. Even without snapshotting and regardless of the
filesystem, tho on btrfs it's a bigger factor due to COW, noatime is a
recommended performance optimization.
The biggest caveat with that is if you're running something that actually
depends on atime. Few if any modern applications depend on atime, with
mutt in some configurations being an older application that still does.
But AFAIK it only does in some configurations...
Tho I haven't the foggiest whether it'd affect your mount times...
(FWIW I have a number of pair-device btrfs raid1s, but I'm all-ssd these
days and mounts seem to be fast enough here.)
> When I try to mount the array it's consistent about 5 seconds+
> # time umount /media/raid1
>
> real 0m0.358s user 0m0.010s sys 0m0.010s # time mount
> /media/raid1
>
> real 0m5.605s user 0m0.504s sys 0m0.071s
>
> I have this setup for sometime now and from the time I made it the mount
> time went up (I notice that on boot). When I first build that was almost
> instant. In terms of maintenance I regularly run a scrub and rebalance
> every now and then.
>
> Running kernel 4.11.12 (with -ck patchs)
>
> Is there something I can do to speed it up (apart buying 2 SSDs :D ). I
> feel like I'm missing something as the usage of the raid is not really
> frequent - just backup mainly.
Is there anything suspect in dmesg during the mount? What does smartctl
say about the health of the devices? (smartctl -AH at least, the selftest
data is unlikely to be useful unless you actually run the selftests.)
To my awareness there's a few things that can affect mount speed, tho as
I said I'm on ssd so I really don't know if 5 seconds for spinning rust
is unusual or not, you'll need the experience of others on that.
1) Attempting to mount filesystems with many devices is of course
slower. But two devices shouldn't be a problem.
2) Sometimes a device might take awhile to "spin up" and initialize
itself. Since you're still on spinning rust, are the devices perhaps
spinning down to save power, and the delay you see is them spinning back
up?
SSDs may have a similar, tho generally shorter and for different reasons,
delay. SSDs often have a capacitor that charges up so they can finish a
write and avoid corrupting themselves in the event of an unexpected power
loss in the middle of a write. A lower end device might allow the device
to appear ready while the capacitor is still charging to avoid long power-
on response times, while higher end devices both tend to have higher
capacity capacitors, and don't signify ready until they are sufficiently
charged to avoid issues in "blink" situations where the power supply
comes back on but isn't immediately steady and might go out again right
away.
If a device takes too long and times out you'll see resets and the like
in dmesg, but that normally starts at ~30 seconds, not the 5 seconds you
mention. Still, doesn't hurt to check.
3) If the space cache is damaged the mount may take longer, but btrfs
will complain so you'll see it in dmesg.
[OT but quoting the signature]
> A: Because it messes up the order in which people normally read text.
> Q: Why is it such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing on usenet and in e-mail?
My sentiments exactly! (Well, except that HTML's even more annoying!) =:^)
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Slow mounting raid1
2017-08-01 1:12 ` Duncan
@ 2017-08-01 6:43 ` Leonidas Spyropoulos
2017-08-01 12:32 ` E V
0 siblings, 1 reply; 6+ messages in thread
From: Leonidas Spyropoulos @ 2017-08-01 6:43 UTC (permalink / raw)
To: Duncan; +Cc: linux-btrfs
Hi Duncan,
Thanks for your answer
On 01/08/17, Duncan wrote:
>
> If you're doing any snapshotting, you almost certainly want noatime, not
> the default relatime. Even without snapshotting and regardless of the
> filesystem, tho on btrfs it's a bigger factor due to COW, noatime is a
> recommended performance optimization.
>
> The biggest caveat with that is if you're running something that actually
> depends on atime. Few if any modern applications depend on atime, with
> mutt in some configurations being an older application that still does.
> But AFAIK it only does in some configurations...
The array has no snapshots and my mutt resides on a diff SSD btrfs so I can
safely try this option.
>
> Is there anything suspect in dmesg during the mount? What does smartctl
> say about the health of the devices? (smartctl -AH at least, the selftest
> data is unlikely to be useful unless you actually run the selftests.)
>
dmesg while mount says:
[19823.896790] BTRFS info (device sde): use lzo compression
[19823.896798] BTRFS info (device sde): disk space caching is enabled
[19823.896800] BTRFS info (device sde): has skinny extents
Smartctl tests are scheduled to run all disks once every day (for short test) and every week for long tests.
smartctl output:
# smartctl -AH /dev/sdd
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.11.12-1-ck] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 143 143 054 Pre-fail Offline - 67
3 Spin_Up_Time 0x0007 124 124 024 Pre-fail Always - 185 (Average 185)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 651
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 110 110 020 Pre-fail Offline - 36
9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 4594
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 353
192 Power-Off_Retract_Count 0x0032 094 094 000 Old_age Always - 7671
193 Load_Cycle_Count 0x0012 094 094 000 Old_age Always - 7671
194 Temperature_Celsius 0x0002 162 162 000 Old_age Always - 37 (Min/Max 17/62)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
# smartctl -AH /dev/sde
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.11.12-1-ck] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 142 142 054 Pre-fail Offline - 69
3 Spin_Up_Time 0x0007 123 123 024 Pre-fail Always - 186 (Average 187)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 709
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 113 113 020 Pre-fail Offline - 35
9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 4678
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 353
192 Power-Off_Retract_Count 0x0032 093 093 000 Old_age Always - 8407
193 Load_Cycle_Count 0x0012 093 093 000 Old_age Always - 8407
194 Temperature_Celsius 0x0002 166 166 000 Old_age Always - 36 (Min/Max 17/64)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
> 1) Attempting to mount filesystems with many devices is of course
> slower. But two devices shouldn't be a problem.
>
> 2) Sometimes a device might take awhile to "spin up" and initialize
> itself. Since you're still on spinning rust, are the devices perhaps
> spinning down to save power, and the delay you see is them spinning back
> up?
>
Good idea but even when I do it like below it's still 6 seconds:
# time umount /media/raid1 && time mount /media/raid1
real 0m0.501s
user 0m0.046s
sys 0m0.011s
real 0m5.540s
user 0m0.943s
sys 0m0.062s
> SSDs may have a similar, tho generally shorter and for different reasons,
> delay. SSDs often have a capacitor that charges up so they can finish a
> write and avoid corrupting themselves in the event of an unexpected power
> loss in the middle of a write. A lower end device might allow the device
> to appear ready while the capacitor is still charging to avoid long power-
> on response times, while higher end devices both tend to have higher
> capacity capacitors, and don't signify ready until they are sufficiently
> charged to avoid issues in "blink" situations where the power supply
> comes back on but isn't immediately steady and might go out again right
> away.
>
> If a device takes too long and times out you'll see resets and the like
> in dmesg, but that normally starts at ~30 seconds, not the 5 seconds you
> mention. Still, doesn't hurt to check.
Nothing on dmesg related as you can see.
>
> 3) If the space cache is damaged the mount may take longer, but btrfs
> will complain so you'll see it in dmesg.
>
That was my idea and that's why I asked in ML.
I'll give 'noatime' a go and see if it's changes anything.
--
Leonidas Spyropoulos
A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Slow mounting raid1
2017-08-01 6:43 ` Leonidas Spyropoulos
@ 2017-08-01 12:32 ` E V
2017-08-01 20:21 ` Leonidas Spyropoulos
0 siblings, 1 reply; 6+ messages in thread
From: E V @ 2017-08-01 12:32 UTC (permalink / raw)
To: Leonidas Spyropoulos, linux-btrfs
On Tue, Aug 1, 2017 at 2:43 AM, Leonidas Spyropoulos
<artafinde@gmail.com> wrote:
> Hi Duncan,
>
> Thanks for your answer
In general I think btrfs takes time proportional to the size of your
metadata to mount. Bigger and/or fragmented metadata leads to longer
mount times. My big backup fs with >300GB of metadata takes over
20minutes to mount, and that's with the space tree which is
significantly faster then space cache v1.
>>
>> If a device takes too long and times out you'll see resets and the like
>> in dmesg, but that normally starts at ~30 seconds, not the 5 seconds you
>> mention. Still, doesn't hurt to check.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Slow mounting raid1
2017-08-01 12:32 ` E V
@ 2017-08-01 20:21 ` Leonidas Spyropoulos
2017-08-01 20:40 ` Timofey Titovets
0 siblings, 1 reply; 6+ messages in thread
From: Leonidas Spyropoulos @ 2017-08-01 20:21 UTC (permalink / raw)
To: linux-btrfs
On 01/08/17, E V wrote:
> In general I think btrfs takes time proportional to the size of your
> metadata to mount. Bigger and/or fragmented metadata leads to longer
> mount times. My big backup fs with >300GB of metadata takes over
> 20minutes to mount, and that's with the space tree which is
> significantly faster then space cache v1.
>
Hmm my raid1 doesn't seem near to full or has a significant Metadata so
I don't I'm on this case:
# btrfs fi show /media/raid1/
Label: 'raid1' uuid: c9db91e6-0ba8-4ae6-b471-8fd4ff7ee72d
Total devices 2 FS bytes used 516.18GiB
devid 1 size 931.51GiB used 518.03GiB path /dev/sdd
devid 2 size 931.51GiB used 518.03GiB path /dev/sde
# btrfs fi df /media/raid1/
Data, RAID1: total=513.00GiB, used=512.21GiB
System, RAID1: total=32.00MiB, used=112.00KiB
Metadata, RAID1: total=5.00GiB, used=3.97GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
I tried the space_cache=v2 just to see if it would do any
difference but nothing changed
# cat /etc/fstab | grep raid1
UUID=c9db91e6-0ba8-4ae6-b471-8fd4ff7ee72d /media/raid1 btrfs rw,noatime,compress=lzo,space_cache=v2 0 0
# time umount /media/raid1 && time mount /media/raid1/
real 0m0.807s
user 0m0.237s
sys 0m0.441s
real 0m5.494s
user 0m0.618s
sys 0m0.116s
I did a couple of rebalances on metadata and data and it improved a bit:
# btrfs balance start -musage=100 /media/raid1/
# btrfs balance start -dusage=10 /media/raid1/
[.. incremental dusage 10 -> 95]
# btrfs balance start -dusage=95 /media/raid1
Down to 3.7 sec
# time umount /media/raid1 && time mount /media/raid1/
real 0m0.807s
user 0m0.237s
sys 0m0.441s
real 0m3.790s
user 0m0.430s
sys 0m0.031s
I think maybe the next step is to disable compression if I want to mount
it faster. Is this normal for BTRFS that performance would degrade after
some time?
Regards,
--
Leonidas Spyropoulos
A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Slow mounting raid1
2017-08-01 20:21 ` Leonidas Spyropoulos
@ 2017-08-01 20:40 ` Timofey Titovets
0 siblings, 0 replies; 6+ messages in thread
From: Timofey Titovets @ 2017-08-01 20:40 UTC (permalink / raw)
To: Leonidas Spyropoulos, linux-btrfs
2017-08-01 23:21 GMT+03:00 Leonidas Spyropoulos <artafinde@gmail.com>:
> On 01/08/17, E V wrote:
>> In general I think btrfs takes time proportional to the size of your
>> metadata to mount. Bigger and/or fragmented metadata leads to longer
>> mount times. My big backup fs with >300GB of metadata takes over
>> 20minutes to mount, and that's with the space tree which is
>> significantly faster then space cache v1.
>>
> Hmm my raid1 doesn't seem near to full or has a significant Metadata so
> I don't I'm on this case:
> # btrfs fi show /media/raid1/
> Label: 'raid1' uuid: c9db91e6-0ba8-4ae6-b471-8fd4ff7ee72d
> Total devices 2 FS bytes used 516.18GiB
> devid 1 size 931.51GiB used 518.03GiB path /dev/sdd
> devid 2 size 931.51GiB used 518.03GiB path /dev/sde
>
> # btrfs fi df /media/raid1/
> Data, RAID1: total=513.00GiB, used=512.21GiB
> System, RAID1: total=32.00MiB, used=112.00KiB
> Metadata, RAID1: total=5.00GiB, used=3.97GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> I tried the space_cache=v2 just to see if it would do any
> difference but nothing changed
> # cat /etc/fstab | grep raid1
> UUID=c9db91e6-0ba8-4ae6-b471-8fd4ff7ee72d /media/raid1 btrfs rw,noatime,compress=lzo,space_cache=v2 0 0
> # time umount /media/raid1 && time mount /media/raid1/
>
> real 0m0.807s
> user 0m0.237s
> sys 0m0.441s
>
> real 0m5.494s
> user 0m0.618s
> sys 0m0.116s
>
> I did a couple of rebalances on metadata and data and it improved a bit:
> # btrfs balance start -musage=100 /media/raid1/
> # btrfs balance start -dusage=10 /media/raid1/
> [.. incremental dusage 10 -> 95]
> # btrfs balance start -dusage=95 /media/raid1
>
> Down to 3.7 sec
> # time umount /media/raid1 && time mount /media/raid1/
>
> real 0m0.807s
> user 0m0.237s
> sys 0m0.441s
>
> real 0m3.790s
> user 0m0.430s
> sys 0m0.031s
>
> I think maybe the next step is to disable compression if I want to mount
> it faster. Is this normal for BTRFS that performance would degrade after
> some time?
>
> Regards,
>
> --
> Leonidas Spyropoulos
>
> A: Because it messes up the order in which people normally read text.
> Q: Why is it such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing on usenet and in e-mail?
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
AFAIK, for space_cache=v2, you need do something like:
btrfs check --clear-space-cache v1 /dev/sdd
mount -o space_cache=v2 /dev/sdd <mount_point>
First mount will be very slow, because that require rebuild of space_cache
Thanks.
--
Have a nice day,
Timofey.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-08-01 20:41 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-07-31 18:30 Slow mounting raid1 Leonidas Spyropoulos
2017-08-01 1:12 ` Duncan
2017-08-01 6:43 ` Leonidas Spyropoulos
2017-08-01 12:32 ` E V
2017-08-01 20:21 ` Leonidas Spyropoulos
2017-08-01 20:40 ` Timofey Titovets
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.