48 seconds to mount a BTRFS hard disk drive seems too long to me

48 seconds to mount a BTRFS hard disk drive seems too long to me

[Juan Simón]

Hard disk: 16TB SEAGATE IRONWOLF PRO 3.5", 7200 RPM 256MB CACHE
Arch Linux
Linux juan-PC 5.15.13-xanmod1-tt-1 #1 SMP Thu, 06 Jan 2022 12:14:06
+0000 x86_64 GNU/Linux
btrfs-progs v5.15.1

$ btrfs fi df /multimedia
Data, single: total=10.89TiB, used=10.72TiB
System, DUP: total=8.00MiB, used=1.58MiB
Metadata, DUP: total=15.00GiB, used=13.19GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

I have formatted it as BTRFS and the mounting options (fstab) are:

/multimedia     btrfs
rw,noatime,autodefrag,compress-force=zstd,nossd,space_cache=v2    0 0

The disk works fine, I have not detected any problems but every time I
reboot the system takes a long time due to the mounting of this drive

$ systemd-analyze blame
48.575s multimedia.mount
....

I find it too long to mount a drive, is this normal, is it because of
one of the mounting options, or because of the size of the hard drive?

Thanks in advance. Regards.

Re: 48 seconds to mount a BTRFS hard disk drive seems too long to me

[Chris Murphy]

On Thu, Jan 6, 2022 at 8:48 AM Juan Simón <decedion@gmail.com> wrote:
>
> Hard disk: 16TB SEAGATE IRONWOLF PRO 3.5", 7200 RPM 256MB CACHE
> Arch Linux
> Linux juan-PC 5.15.13-xanmod1-tt-1 #1 SMP Thu, 06 Jan 2022 12:14:06
> +0000 x86_64 GNU/Linux
> btrfs-progs v5.15.1
>
> $ btrfs fi df /multimedia
> Data, single: total=10.89TiB, used=10.72TiB
> System, DUP: total=8.00MiB, used=1.58MiB
> Metadata, DUP: total=15.00GiB, used=13.19GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> I have formatted it as BTRFS and the mounting options (fstab) are:
>
> /multimedia     btrfs
> rw,noatime,autodefrag,compress-force=zstd,nossd,space_cache=v2    0 0
>
> The disk works fine, I have not detected any problems but every time I
> reboot the system takes a long time due to the mounting of this drive
>
> $ systemd-analyze blame
> 48.575s multimedia.mount
> ....
>
> I find it too long to mount a drive, is this normal, is it because of
> one of the mounting options, or because of the size of the hard drive?

I think it's due to reading all the block group items which are buried
in the extent tree. And on large file systems with a large extent
tree, it results in a lot of random IO reads, so on spinning drives
you get a ton of latency hits for this search.

This thread discusses some of the details as it relates to zoned
devices, which have even longer mount times I guess. But the comments
about block groups and extent tree apply the same to non-zoned.
https://lore.kernel.org/linux-btrfs/CAHQ7scVGPAwEGQOq3Kmn75GJzyzSQ9qrBBZrHFu+4YWQhGE0Lw@mail.gmail.com/

The mount options you're using aren't causing the mount delay. But be
certain you really want autodefrag - it's designed for the desktop,
small databases used by desktops, web browsers, and the like. If you
use it with bigger and/or very busy databases, it just results in a
ton of I/O and can really slow things down rather than speed them up.
You're better off in such a use case with scheduled defragment on a
dir by dir basis, which you can do using the btrfsmaintenance
package's btrfs-defrag.timer and .service unit, which you can
customize. It's maintained by kdave, also maintainer of btrfs-progs.
https://github.com/kdave/btrfsmaintenance


--
Chris Murphy

Re: 48 seconds to mount a BTRFS hard disk drive seems too long to me

[Forza]

---- From: Chris Murphy <lists@colorremedies.com> -- Sent: 2022-01-06 - 20:58 ----

> On Thu, Jan 6, 2022 at 8:48 AM Juan Simón <decedion@gmail.com> wrote:
>>
>> Hard disk: 16TB SEAGATE IRONWOLF PRO 3.5", 7200 RPM 256MB CACHE
>> Arch Linux
>> Linux juan-PC 5.15.13-xanmod1-tt-1 #1 SMP Thu, 06 Jan 2022 12:14:06
>> +0000 x86_64 GNU/Linux
>> btrfs-progs v5.15.1
>>
>> $ btrfs fi df /multimedia
>> Data, single: total=10.89TiB, used=10.72TiB
>> System, DUP: total=8.00MiB, used=1.58MiB
>> Metadata, DUP: total=15.00GiB, used=13.19GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>> I have formatted it as BTRFS and the mounting options (fstab) are:
>>
>> /multimedia     btrfs
>> rw,noatime,autodefrag,compress-force=zstd,nossd,space_cache=v2    0 0
>>
>> The disk works fine, I have not detected any problems but every time I
>> reboot the system takes a long time due to the mounting of this drive
>>
>> $ systemd-analyze blame
>> 48.575s multimedia.mount
>> ....
>>
>> I find it too long to mount a drive, is this normal, is it because of
>> one of the mounting options, or because of the size of the hard drive?
> 
> I think it's due to reading all the block group items which are buried
> in the extent tree. And on large file systems with a large extent
> tree, it results in a lot of random IO reads, so on spinning drives
> you get a ton of latency hits for this search.
> 
> This thread discusses some of the details as it relates to zoned
> devices, which have even longer mount times I guess. But the comments
> about block groups and extent tree apply the same to non-zoned.
> https://lore.kernel.org/linux-btrfs/CAHQ7scVGPAwEGQOq3Kmn75GJzyzSQ9qrBBZrHFu+4YWQhGE0Lw@mail.gmail.com/
> 
> The mount options you're using aren't causing the mount delay. But be
> certain you really want autodefrag - it's designed for the desktop,
> small databases used by desktops, web browsers, and the like. If you
> use it with bigger and/or very busy databases, it just results in a
> ton of I/O and can really slow things down rather than speed them up.
> You're better off in such a use case with scheduled defragment on a
> dir by dir basis, which you can do using the btrfsmaintenance
> package's btrfs-defrag.timer and .service unit, which you can
> customize. It's maintained by kdave, also maintainer of btrfs-progs.
> https://github.com/kdave/btrfsmaintenance
> 

It might be worth trying to defragment the extent tree of each subvolume by pointing btrfs defrag to the subvolume and not use the -r option. I don't think the btrfsmaintenance script does this.

# btrfs fi defragment /mnt/btrfs 
# btrfs fi defragment /mnt/btrfs/subvol_A
# btrfs fi defragment /mnt/btrfs/subvol_B
...

Read-only snapshots can't be defragmented this way. 

Re: 48 seconds to mount a BTRFS hard disk drive seems too long to me

[Zygo Blaxell]

On Thu, Jan 06, 2022 at 04:48:21PM +0100, Juan Simón wrote:
> Hard disk: 16TB SEAGATE IRONWOLF PRO 3.5", 7200 RPM 256MB CACHE
> Arch Linux
> Linux juan-PC 5.15.13-xanmod1-tt-1 #1 SMP Thu, 06 Jan 2022 12:14:06
> +0000 x86_64 GNU/Linux
> btrfs-progs v5.15.1
> 
> $ btrfs fi df /multimedia
> Data, single: total=10.89TiB, used=10.72TiB
> System, DUP: total=8.00MiB, used=1.58MiB
> Metadata, DUP: total=15.00GiB, used=13.19GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> I have formatted it as BTRFS and the mounting options (fstab) are:
> 
> /multimedia     btrfs
> rw,noatime,autodefrag,compress-force=zstd,nossd,space_cache=v2    0 0
> 
> The disk works fine, I have not detected any problems but every time I
> reboot the system takes a long time due to the mounting of this drive
> 
> $ systemd-analyze blame
> 48.575s multimedia.mount
> ....
> 
> I find it too long to mount a drive, is this normal, is it because of
> one of the mounting options, or because of the size of the hard drive?

Worst-case, you'll need about one second of mounting time for every ~180
block groups on a spinning disk, which works out to about 89 seconds on
that drive when it's full.

It's a known issue, but it will require a disk format change to fix, so
it will likely be rolled into a larger change set like extent tree v2.

> Thanks in advance. Regards.

Re: 48 seconds to mount a BTRFS hard disk drive seems too long to me

[David Sterba]

On Thu, Jan 06, 2022 at 12:58:58PM -0700, Chris Murphy wrote:
> On Thu, Jan 6, 2022 at 8:48 AM Juan Simón <decedion@gmail.com> wrote:
> >
> > Hard disk: 16TB SEAGATE IRONWOLF PRO 3.5", 7200 RPM 256MB CACHE
> > Arch Linux
> > Linux juan-PC 5.15.13-xanmod1-tt-1 #1 SMP Thu, 06 Jan 2022 12:14:06
> > +0000 x86_64 GNU/Linux
> > btrfs-progs v5.15.1
> >
> > $ btrfs fi df /multimedia
> > Data, single: total=10.89TiB, used=10.72TiB
> > System, DUP: total=8.00MiB, used=1.58MiB
> > Metadata, DUP: total=15.00GiB, used=13.19GiB
> > GlobalReserve, single: total=512.00MiB, used=0.00B
> >
> > I have formatted it as BTRFS and the mounting options (fstab) are:
> >
> > /multimedia     btrfs
> > rw,noatime,autodefrag,compress-force=zstd,nossd,space_cache=v2    0 0
> >
> > The disk works fine, I have not detected any problems but every time I
> > reboot the system takes a long time due to the mounting of this drive
> >
> > $ systemd-analyze blame
> > 48.575s multimedia.mount
> > ....
> >
> > I find it too long to mount a drive, is this normal, is it because of
> > one of the mounting options, or because of the size of the hard drive?
> 
> I think it's due to reading all the block group items which are buried
> in the extent tree. And on large file systems with a large extent
> tree, it results in a lot of random IO reads, so on spinning drives
> you get a ton of latency hits for this search.
> 
> This thread discusses some of the details as it relates to zoned
> devices, which have even longer mount times I guess. But the comments
> about block groups and extent tree apply the same to non-zoned.
> https://lore.kernel.org/linux-btrfs/CAHQ7scVGPAwEGQOq3Kmn75GJzyzSQ9qrBBZrHFu+4YWQhGE0Lw@mail.gmail.com/

That's unrelated to the problem on zoned drives, the delay there is
caused by time spent on reading the zone information.

Re: 48 seconds to mount a BTRFS hard disk drive seems too long to me

[David Sterba]

On Fri, Jan 07, 2022 at 12:07:06PM -0500, Zygo Blaxell wrote:
> On Thu, Jan 06, 2022 at 04:48:21PM +0100, Juan Simón wrote:
> > Hard disk: 16TB SEAGATE IRONWOLF PRO 3.5", 7200 RPM 256MB CACHE
> > Arch Linux
> > Linux juan-PC 5.15.13-xanmod1-tt-1 #1 SMP Thu, 06 Jan 2022 12:14:06
> > +0000 x86_64 GNU/Linux
> > btrfs-progs v5.15.1
> > 
> > $ btrfs fi df /multimedia
> > Data, single: total=10.89TiB, used=10.72TiB
> > System, DUP: total=8.00MiB, used=1.58MiB
> > Metadata, DUP: total=15.00GiB, used=13.19GiB
> > GlobalReserve, single: total=512.00MiB, used=0.00B
> > 
> > I have formatted it as BTRFS and the mounting options (fstab) are:
> > 
> > /multimedia     btrfs
> > rw,noatime,autodefrag,compress-force=zstd,nossd,space_cache=v2    0 0
> > 
> > The disk works fine, I have not detected any problems but every time I
> > reboot the system takes a long time due to the mounting of this drive
> > 
> > $ systemd-analyze blame
> > 48.575s multimedia.mount
> > ....
> > 
> > I find it too long to mount a drive, is this normal, is it because of
> > one of the mounting options, or because of the size of the hard drive?
> 
> Worst-case, you'll need about one second of mounting time for every ~180
> block groups on a spinning disk, which works out to about 89 seconds on
> that drive when it's full.
> 
> It's a known issue, but it will require a disk format change to fix, so
> it will likely be rolled into a larger change set like extent tree v2.

Yeah, there was even a suggestion to add yet another tree to store the
block group information with better locality, this will be part of the
extent tree v2 update. Otherwise we can try to do readahead, there is
something already done I can't find right now.