Speed up mount time?

Speed up mount time?

[Joakim]

I have a couple of machines (A and B) set up where each machine has a
~430 TB BTRFS subvolume, same data on both. Mounting these volumes
with the following flags: noatime,compress=lzo,space_cache=v2

Initially mount times were quite long, about 10 minutes. But after i
did run a defrag with -c option on machine B the mount time increased
to over 30 minutes. This volume has a little over 100 TB stored.

How come the mount time increased by this?

And is there any way to decrease the mount times? 10 minutes is long
but acceptable, while 30 minutes is way too long.

Advice would be highly appreciated. :)



Linux sm07b 5.4.17-2136.311.6.1.el8uek.x86_64 #2 SMP Thu Sep 22
19:29:28 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux
btrfs-progs v5.15.1
Label: 'Storage'  uuid: 6ecd172e-3ebd-478c-9515-68162a41590d
        Total devices 1 FS bytes used 105.04TiB
        devid    1 size 436.57TiB used 107.87TiB path /dev/sdb

Data, single: total=107.34TiB, used=104.82TiB
System, DUP: total=40.00MiB, used=11.23MiB
Metadata, DUP: total=269.00GiB, used=221.57GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

[   51.764445] BTRFS info (device sdb): flagging fs with big metadata feature
[   51.764450] BTRFS info (device sdb): use lzo compression, level 0
[   51.764454] BTRFS info (device sdb): using free space tree
[   51.764455] BTRFS info (device sdb): has skinny extents

Re: Speed up mount time?

[Qu Wenruo]

On 2022/11/23 20:33, Joakim wrote:
> I have a couple of machines (A and B) set up where each machine has a
> ~430 TB BTRFS subvolume, same data on both. Mounting these volumes
> with the following flags: noatime,compress=lzo,space_cache=v2
> 
> Initially mount times were quite long, about 10 minutes. But after i
> did run a defrag with -c option on machine B the mount time increased
> to over 30 minutes. This volume has a little over 100 TB stored.
> 
> How come the mount time increased by this?

In kernel 6.2 release, we will have block-group-tree feature, which 
should greatly reduce the mount time.

And we will provide btrfs-progs tool to do an offline convert, keeping 
the data and enable the new feature.

Thanks,
Qu
> 
> And is there any way to decrease the mount times? 10 minutes is long
> but acceptable, while 30 minutes is way too long.
> 
> Advice would be highly appreciated. :)
> 
> 
> 
> Linux sm07b 5.4.17-2136.311.6.1.el8uek.x86_64 #2 SMP Thu Sep 22
> 19:29:28 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux
> btrfs-progs v5.15.1
> Label: 'Storage'  uuid: 6ecd172e-3ebd-478c-9515-68162a41590d
>          Total devices 1 FS bytes used 105.04TiB
>          devid    1 size 436.57TiB used 107.87TiB path /dev/sdb
> 
> Data, single: total=107.34TiB, used=104.82TiB
> System, DUP: total=40.00MiB, used=11.23MiB
> Metadata, DUP: total=269.00GiB, used=221.57GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> [   51.764445] BTRFS info (device sdb): flagging fs with big metadata feature
> [   51.764450] BTRFS info (device sdb): use lzo compression, level 0
> [   51.764454] BTRFS info (device sdb): using free space tree
> [   51.764455] BTRFS info (device sdb): has skinny extents

Re: Speed up mount time?

[Roman Mamedov]

On Wed, 23 Nov 2022 13:33:17 +0100
Joakim <ahoj79@gmail.com> wrote:

> Initially mount times were quite long, about 10 minutes. But after i
> did run a defrag with -c option on machine B the mount time increased
> to over 30 minutes. This volume has a little over 100 TB stored.
> 
> How come the mount time increased by this?

With compression each file now has to be split into extents each at most 128K
long, which means there are much more extents to keep track of than before.

> And is there any way to decrease the mount times? 10 minutes is long
> but acceptable, while 30 minutes is way too long.
> 
> Advice would be highly appreciated. :)

It is unfortunate that you use the bare sdb device, if you used LVM, you could
easily add an SSD and deploy LVM caching in writethrough mode (or a RAID1 of
SSDs and use the writeback mode). In my experience LVM caching of HDD based
storage to SSDs improves mount times of Btrfs really by a lot.

There have been a number of attempts to implement Btrfs native storage tiering
of HDDs vs SSDs (storing metadata on the latter), such as [1], but none have
made it so far, and the patches from 2020 probably no longer apply.

[1]https://lwn.net/ml/linux-btrfs/20201029053556.10619-1-wangyugui@e16-tech.com/

-- 
With respect,
Roman

Re: Speed up mount time?

[Forza]

---- From: Joakim <ahoj79@gmail.com> -- Sent: 2022-11-23 - 13:33 ----

> I have a couple of machines (A and B) set up where each machine has a
> ~430 TB BTRFS subvolume, same data on both. Mounting these volumes
> with the following flags: noatime,compress=lzo,space_cache=v2
> 
> Initially mount times were quite long, about 10 minutes. But after i
> did run a defrag with -c option on machine B the mount time increased
> to over 30 minutes. This volume has a little over 100 TB stored.
> 
> How come the mount time increased by this?

You now have now extents to keep track of, which means more metadata to parse at mount. 

> 
> And is there any way to decrease the mount times? 10 minutes is long
> but acceptable, while 30 minutes is way too long.
> 
> Advice would be highly appreciated. :)
> 

You can defrag the subvolume and extent trees by using `btrfs fi defrag /path/to/subvol`. You can do this on each subvol and on the top level volume. This can reduce amounts of seeks, improving performance some. 

> 
> 
> Linux sm07b 5.4.17-2136.311.6.1.el8uek.x86_64 #2 SMP Thu Sep 22
> 19:29:28 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux
> btrfs-progs v5.15.1
> Label: 'Storage'  uuid: 6ecd172e-3ebd-478c-9515-68162a41590d
>         Total devices 1 FS bytes used 105.04TiB
>         devid    1 size 436.57TiB used 107.87TiB path /dev/sdb
> 
> Data, single: total=107.34TiB, used=104.82TiB
> System, DUP: total=40.00MiB, used=11.23MiB
> Metadata, DUP: total=269.00GiB, used=221.57GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> [   51.764445] BTRFS info (device sdb): flagging fs with big metadata feature
> [   51.764450] BTRFS info (device sdb): use lzo compression, level 0
> [   51.764454] BTRFS info (device sdb): using free space tree
> [   51.764455] BTRFS info (device sdb): has skinny extents

Re: Speed up mount time?

[Joakim]

That sounds great! Is there any rough ETA for when that might be released?
Thanks! :)

Re: Speed up mount time?

[Joakim]

Thanks for describing, now it makes sense.
Regarding LVM-caching, I will keep that in mind and perhaps try that
on the next storage cluster I deploy.

Re: Speed up mount time?

[Joakim]

Yes, thanks! Metadata defrag finished an hour ago. But time to mount is still: 31m6.720s.
I guess I'll just have to wait for the new kernel.

Re: Speed up mount time?

[Qu Wenruo]

On 2022/11/25 16:55, Joakim wrote:
> That sounds great! Is there any rough ETA for when that might be released?
> Thanks! :)

As mentioned, if nothing wrong happened, v6.2 would be the target for 
the kernel.

For progs, it should already be implemented in btrfstune from the latest 
btrfs-progs.

But only one-way conversion is supported (aka, regular -> bg tree), no 
way to convert back yet.

If your distro doesn't ship newer kernel/progs, it may take much longer 
though.

Thanks,
Qu

Re: Speed up mount time?

[Joakim]

Alright, thanks for clarification.  
This machine has Oracle Linux, so i guess there will be som waiting.

//Joakim

Re: Speed up mount time?

[Torbjörn Jansson]

On 2022-11-25 10:22, Joakim wrote:
> Alright, thanks for clarification.
> This machine has Oracle Linux, so i guess there will be som waiting.
> 
> //Joakim

Quoting the original mail when replying helps a lot to understand what you are 
actually replying to.

Re: Speed up mount time?

[kenneth topp]

On Fri, 25 Nov 2022, Qu Wenruo wrote:

>
>
> On 2022/11/25 16:55, Joakim wrote:
>>  That sounds great! Is there any rough ETA for when that might be released?
>>  Thanks! :)
>
> As mentioned, if nothing wrong happened, v6.2 would be the target for the 
> kernel.

Am I mistaken this was actually inside v6.1 and not 6.2?

https://lore.kernel.org/lkml/cover.1664798047.git.dsterba@suse.com/

I'm eager to give this a shot if it is, as 6.1 was just tagged.

Ken

>
> For progs, it should already be implemented in btrfstune from the latest 
> btrfs-progs.
>
> But only one-way conversion is supported (aka, regular -> bg tree), no way to 
> convert back yet.
>
> If your distro doesn't ship newer kernel/progs, it may take much longer 
> though.
>
> Thanks,
> Qu
>
>