Question about metadata size

Question about metadata size

[Libor Klepáč]

Hi,
we have filesystem like this

Overall:
    Device size:                  30.00TiB
    Device allocated:             24.93TiB
    Device unallocated:            5.07TiB
    Device missing:                  0.00B
    Used:                         24.92TiB
    Free (estimated):              5.07TiB      (min: 2.54TiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              512.00MiB      (used: 0.00B)

Data,single: Size:24.85TiB, Used:24.84TiB (99.98%)
   /dev/sdc       24.85TiB

Metadata,single: Size:88.00GiB, Used:81.54GiB (92.65%)
   /dev/sdc       88.00GiB

System,DUP: Size:32.00MiB, Used:3.25MiB (10.16%)
   /dev/sdc       64.00MiB

Unallocated:
   /dev/sdc        5.07TiB


Is it normal to have so much metadata? We have only 119 files with size
of 2048 bytes or less.
There is 885 files in total and 17 directories, we don't use snapshots.

Most of the files are multi gigabyte, some of them have around 3TB -
all are snapshots from vmware stored using nakivo.

Working with filesystem - mostly deleting files seems to be very slow -
it took several hours to delete snapshot of one machine, which
consisted of four or five of those 3TB files.

We run beesd on those data, but i think, there was this much metadata
even before we started to do so.

With regards,
Libor

Re: Question about metadata size

[Qu Wenruo]

On 2022/6/27 17:02, Libor Klepáč wrote:
> Hi,
> we have filesystem like this
>
> Overall:
>      Device size:                  30.00TiB
>      Device allocated:             24.93TiB
>      Device unallocated:            5.07TiB
>      Device missing:                  0.00B
>      Used:                         24.92TiB
>      Free (estimated):              5.07TiB      (min: 2.54TiB)
>      Data ratio:                       1.00
>      Metadata ratio:                   1.00
>      Global reserve:              512.00MiB      (used: 0.00B)
>
> Data,single: Size:24.85TiB, Used:24.84TiB (99.98%)
>     /dev/sdc       24.85TiB
>
> Metadata,single: Size:88.00GiB, Used:81.54GiB (92.65%)
>     /dev/sdc       88.00GiB
>
> System,DUP: Size:32.00MiB, Used:3.25MiB (10.16%)
>     /dev/sdc       64.00MiB
>
> Unallocated:
>     /dev/sdc        5.07TiB
>
>
> Is it normal to have so much metadata? We have only 119 files with size
> of 2048 bytes or less.

That would only take around 50KiB so no problem.

> There is 885 files in total and 17 directories, we don't use snapshots.

The other files really depends.

Do you use compression, if so metadata usage will be greately increased.

For non-compressed files, the max file extent size is 128M, while for
compressed files, the max file extent size is only 128K.

This means, for a 3TiB file, if you have compress enabled, it will take
24M file extents, and since each file extent takes at least 53 bytes for
metadata, one such 3TiB file can already take over 1 GiB for metadata.

Thanks,
Qu
>
> Most of the files are multi gigabyte, some of them have around 3TB -
> all are snapshots from vmware stored using nakivo.
>
> Working with filesystem - mostly deleting files seems to be very slow -
> it took several hours to delete snapshot of one machine, which
> consisted of four or five of those 3TB files.
>
> We run beesd on those data, but i think, there was this much metadata
> even before we started to do so.
>
> With regards,
> Libor

Re: Question about metadata size

[Libor Klepáč]

On Po, 2022-06-27 at 18:10 +0800, Qu Wenruo wrote:
> 
> 
> On 2022/6/27 17:02, Libor Klepáč wrote:
> > Hi,
> > we have filesystem like this
> > 
> > Overall:
> >      Device size:                  30.00TiB
> >      Device allocated:             24.93TiB
> >      Device unallocated:            5.07TiB
> >      Device missing:                  0.00B
> >      Used:                         24.92TiB
> >      Free (estimated):              5.07TiB      (min: 2.54TiB)
> >      Data ratio:                       1.00
> >      Metadata ratio:                   1.00
> >      Global reserve:              512.00MiB      (used: 0.00B)
> > 
> > Data,single: Size:24.85TiB, Used:24.84TiB (99.98%)
> >     /dev/sdc       24.85TiB
> > 
> > Metadata,single: Size:88.00GiB, Used:81.54GiB (92.65%)
> >     /dev/sdc       88.00GiB
> > 
> > System,DUP: Size:32.00MiB, Used:3.25MiB (10.16%)
> >     /dev/sdc       64.00MiB
> > 
> > Unallocated:
> >     /dev/sdc        5.07TiB
> > 
> > 
> > Is it normal to have so much metadata? We have only 119 files with
> > size
> > of 2048 bytes or less.
> 
> That would only take around 50KiB so no problem.
> 
> > There is 885 files in total and 17 directories, we don't use
> > snapshots.
> 
> The other files really depends.
> 
> Do you use compression, if so metadata usage will be greately
> increased.


Yes, we use zstd compression - filesystem is mounted with compress-
force=zstd:9

> 
> For non-compressed files, the max file extent size is 128M, while for
> compressed files, the max file extent size is only 128K.
> 
> This means, for a 3TiB file, if you have compress enabled, it will
> take
> 24M file extents, and since each file extent takes at least 53 bytes
> for

That is lot of extents ;)

> metadata, one such 3TiB file can already take over 1 GiB for
> metadata.

I guess there is no way to increase extent size?
We can use internal compression of nakivo, but not without deleting all
stored data and creating empty repository.
Also we wanted to do compression in btrfs, we hoped it will give more
power to beesd to do it's thing (for comparing data)

> 
> Thanks,
> Qu

With regards, Libor

> > 
> > Most of the files are multi gigabyte, some of them have around 3TB
> > -
> > all are snapshots from vmware stored using nakivo.
> > 
> > Working with filesystem - mostly deleting files seems to be very
> > slow -
> > it took several hours to delete snapshot of one machine, which
> > consisted of four or five of those 3TB files.
> > 
> > We run beesd on those data, but i think, there was this much
> > metadata
> > even before we started to do so.
> > 
> > With regards,
> > Libor

Re: Question about metadata size

[Qu Wenruo]

On 2022/6/27 18:23, Libor Klepáč wrote:
> On Po, 2022-06-27 at 18:10 +0800, Qu Wenruo wrote:
>>
>>
>> On 2022/6/27 17:02, Libor Klepáč wrote:
>>> Hi,
>>> we have filesystem like this
>>>
>>> Overall:
>>>       Device size:                  30.00TiB
>>>       Device allocated:             24.93TiB
>>>       Device unallocated:            5.07TiB
>>>       Device missing:                  0.00B
>>>       Used:                         24.92TiB
>>>       Free (estimated):              5.07TiB      (min: 2.54TiB)
>>>       Data ratio:                       1.00
>>>       Metadata ratio:                   1.00
>>>       Global reserve:              512.00MiB      (used: 0.00B)
>>>
>>> Data,single: Size:24.85TiB, Used:24.84TiB (99.98%)
>>>      /dev/sdc       24.85TiB
>>>
>>> Metadata,single: Size:88.00GiB, Used:81.54GiB (92.65%)
>>>      /dev/sdc       88.00GiB
>>>
>>> System,DUP: Size:32.00MiB, Used:3.25MiB (10.16%)
>>>      /dev/sdc       64.00MiB
>>>
>>> Unallocated:
>>>      /dev/sdc        5.07TiB
>>>
>>>
>>> Is it normal to have so much metadata? We have only 119 files with
>>> size
>>> of 2048 bytes or less.
>>
>> That would only take around 50KiB so no problem.
>>
>>> There is 885 files in total and 17 directories, we don't use
>>> snapshots.
>>
>> The other files really depends.
>>
>> Do you use compression, if so metadata usage will be greately
>> increased.
>
>
> Yes, we use zstd compression - filesystem is mounted with compress-
> force=zstd:9
>
>>
>> For non-compressed files, the max file extent size is 128M, while for
>> compressed files, the max file extent size is only 128K.
>>
>> This means, for a 3TiB file, if you have compress enabled, it will
>> take
>> 24M file extents, and since each file extent takes at least 53 bytes
>> for
>
> That is lot of extents ;)
>
>> metadata, one such 3TiB file can already take over 1 GiB for
>> metadata.
>
> I guess there is no way to increase extent size?

Currently it's hard coded. So no way to change that yet.

But please keep in mind that, btrfs compression needs to do trade-off
between writes, and the decompressed size.

E.g. if we can have an 1MiB compressed extent, but if 1020KiB are
overwritten, just one 4KiB is really referred, then to read that 4KiB we
need to decompress all that 1MiB just to read that 4KiB.

So personally speaking, if the main purpose of those large files are
just to archive, not to do frequent write on, then user space
compression would make more sense.

The default btrfs tends to lean to write support.

> We can use internal compression of nakivo, but not without deleting all
> stored data and creating empty repository.
> Also we wanted to do compression in btrfs, we hoped it will give more
> power to beesd to do it's thing (for comparing data)

Then I guess there is not much thing we can help right now, and that
many extents are also slowing down file deletion just as you mentioned.

Thanks,
Qu

>
>>
>> Thanks,
>> Qu
>
> With regards, Libor
>
>>>
>>> Most of the files are multi gigabyte, some of them have around 3TB
>>> -
>>> all are snapshots from vmware stored using nakivo.
>>>
>>> Working with filesystem - mostly deleting files seems to be very
>>> slow -
>>> it took several hours to delete snapshot of one machine, which
>>> consisted of four or five of those 3TB files.
>>>
>>> We run beesd on those data, but i think, there was this much
>>> metadata
>>> even before we started to do so.
>>>
>>> With regards,
>>> Libor

Re: Question about metadata size

[Libor Klepáč]

On Po, 2022-06-27 at 18:54 +0800, Qu Wenruo wrote:
> 
> 
> On 2022/6/27 18:23, Libor Klepáč wrote:
> > On Po, 2022-06-27 at 18:10 +0800, Qu Wenruo wrote:
> > > 
> > > 
> > > On 2022/6/27 17:02, Libor Klepáč wrote:
> > > > Hi,
> > > > we have filesystem like this
> > > > 
> > > > Overall:
> > > >       Device size:                  30.00TiB
> > > >       Device allocated:             24.93TiB
> > > >       Device unallocated:            5.07TiB
> > > >       Device missing:                  0.00B
> > > >       Used:                         24.92TiB
> > > >       Free (estimated):              5.07TiB      (min:
> > > > 2.54TiB)
> > > >       Data ratio:                       1.00
> > > >       Metadata ratio:                   1.00
> > > >       Global reserve:              512.00MiB      (used: 0.00B)
> > > > 
> > > > Data,single: Size:24.85TiB, Used:24.84TiB (99.98%)
> > > >      /dev/sdc       24.85TiB
> > > > 
> > > > Metadata,single: Size:88.00GiB, Used:81.54GiB (92.65%)
> > > >      /dev/sdc       88.00GiB
> > > > 
> > > > System,DUP: Size:32.00MiB, Used:3.25MiB (10.16%)
> > > >      /dev/sdc       64.00MiB
> > > > 
> > > > Unallocated:
> > > >      /dev/sdc        5.07TiB
> > > > 
> > > > 
> > > > Is it normal to have so much metadata? We have only 119 files
> > > > with
> > > > size
> > > > of 2048 bytes or less.
> > > 
> > > That would only take around 50KiB so no problem.
> > > 
> > > > There is 885 files in total and 17 directories, we don't use
> > > > snapshots.
> > > 
> > > The other files really depends.
> > > 
> > > Do you use compression, if so metadata usage will be greately
> > > increased.
> > 
> > 
> > Yes, we use zstd compression - filesystem is mounted with compress-
> > force=zstd:9
> > 
> > > 
> > > For non-compressed files, the max file extent size is 128M, while
> > > for
> > > compressed files, the max file extent size is only 128K.
> > > 
> > > This means, for a 3TiB file, if you have compress enabled, it
> > > will
> > > take
> > > 24M file extents, and since each file extent takes at least 53
> > > bytes
> > > for
> > 
> > That is lot of extents ;)
> > 
> > > metadata, one such 3TiB file can already take over 1 GiB for
> > > metadata.
> > 
> > I guess there is no way to increase extent size?
> 
> Currently it's hard coded. So no way to change that yet.
> 
> But please keep in mind that, btrfs compression needs to do trade-off
> between writes, and the decompressed size.
> 
> E.g. if we can have an 1MiB compressed extent, but if 1020KiB are
> overwritten, just one 4KiB is really referred, then to read that 4KiB
> we
> need to decompress all that 1MiB just to read that 4KiB.

Yes, i get reason for this.
I just never realised the difference in extent size and it's impact on
metadata size/number of extents.

> So personally speaking, if the main purpose of those large files are
> just to archive, not to do frequent write on, then user space
> compression would make more sense.

Ok, these files are writen once and deleted after 14 days (every 14
days, new full backup is created and oldest fullbackup is deleted. Full
backup is dump of whole disk image from vmware), unless needed for some
recovery. Then it's mounted as disk image.

> 
> The default btrfs tends to lean to write support.
> 
> > We can use internal compression of nakivo, but not without deleting
> > all
> > stored data and creating empty repository.
> > Also we wanted to do compression in btrfs, we hoped it will give
> > more
> > power to beesd to do it's thing (for comparing data)
> 
> Then I guess there is not much thing we can help right now, and that
> many extents are also slowing down file deletion just as you
> mentioned.

So i will have to experiment, if user land compression allows us to do
some reasonble deduplication with beesd.
It may maybe speed up beesd, it cannot keep up with data influx, maybe
it's (also) because the number of file extents.
Unfortunately it will mean some serious data juggling in production
environment.

Thanks,
Libor


> 
> Thanks,
> Qu
> 
> > 
> > > 
> > > Thanks,
> > > Qu
> > 
> > With regards, Libor
> > 
> > > > 
> > > > Most of the files are multi gigabyte, some of them have around
> > > > 3TB
> > > > -
> > > > all are snapshots from vmware stored using nakivo.
> > > > 
> > > > Working with filesystem - mostly deleting files seems to be
> > > > very
> > > > slow -
> > > > it took several hours to delete snapshot of one machine, which
> > > > consisted of four or five of those 3TB files.
> > > > 
> > > > We run beesd on those data, but i think, there was this much
> > > > metadata
> > > > even before we started to do so.
> > > > 
> > > > With regards,
> > > > Libor

Re: Question about metadata size

[Qu Wenruo]

On 2022/6/27 20:17, Libor Klepáč wrote:
> On Po, 2022-06-27 at 18:54 +0800, Qu Wenruo wrote:
>>
>>
>> On 2022/6/27 18:23, Libor Klepáč wrote:
>>> On Po, 2022-06-27 at 18:10 +0800, Qu Wenruo wrote:
>>>>
>>>>
>>>> On 2022/6/27 17:02, Libor Klepáč wrote:
>>>>> Hi,
>>>>> we have filesystem like this
>>>>>
>>>>> Overall:
>>>>>        Device size:                  30.00TiB
>>>>>        Device allocated:             24.93TiB
>>>>>        Device unallocated:            5.07TiB
>>>>>        Device missing:                  0.00B
>>>>>        Used:                         24.92TiB
>>>>>        Free (estimated):              5.07TiB      (min:
>>>>> 2.54TiB)
>>>>>        Data ratio:                       1.00
>>>>>        Metadata ratio:                   1.00
>>>>>        Global reserve:              512.00MiB      (used: 0.00B)
>>>>>
>>>>> Data,single: Size:24.85TiB, Used:24.84TiB (99.98%)
>>>>>       /dev/sdc       24.85TiB
>>>>>
>>>>> Metadata,single: Size:88.00GiB, Used:81.54GiB (92.65%)
>>>>>       /dev/sdc       88.00GiB
>>>>>
>>>>> System,DUP: Size:32.00MiB, Used:3.25MiB (10.16%)
>>>>>       /dev/sdc       64.00MiB
>>>>>
>>>>> Unallocated:
>>>>>       /dev/sdc        5.07TiB
>>>>>
>>>>>
>>>>> Is it normal to have so much metadata? We have only 119 files
>>>>> with
>>>>> size
>>>>> of 2048 bytes or less.
>>>>
>>>> That would only take around 50KiB so no problem.
>>>>
>>>>> There is 885 files in total and 17 directories, we don't use
>>>>> snapshots.
>>>>
>>>> The other files really depends.
>>>>
>>>> Do you use compression, if so metadata usage will be greately
>>>> increased.
>>>
>>>
>>> Yes, we use zstd compression - filesystem is mounted with compress-
>>> force=zstd:9
>>>
>>>>
>>>> For non-compressed files, the max file extent size is 128M, while
>>>> for
>>>> compressed files, the max file extent size is only 128K.
>>>>
>>>> This means, for a 3TiB file, if you have compress enabled, it
>>>> will
>>>> take
>>>> 24M file extents, and since each file extent takes at least 53
>>>> bytes
>>>> for
>>>
>>> That is lot of extents ;)
>>>
>>>> metadata, one such 3TiB file can already take over 1 GiB for
>>>> metadata.
>>>
>>> I guess there is no way to increase extent size?
>>
>> Currently it's hard coded. So no way to change that yet.
>>
>> But please keep in mind that, btrfs compression needs to do trade-off
>> between writes, and the decompressed size.
>>
>> E.g. if we can have an 1MiB compressed extent, but if 1020KiB are
>> overwritten, just one 4KiB is really referred, then to read that 4KiB
>> we
>> need to decompress all that 1MiB just to read that 4KiB.
>
> Yes, i get reason for this.
> I just never realised the difference in extent size and it's impact on
> metadata size/number of extents.
>
>> So personally speaking, if the main purpose of those large files are
>> just to archive, not to do frequent write on, then user space
>> compression would make more sense.
>
> Ok, these files are writen once and deleted after 14 days (every 14
> days, new full backup is created and oldest fullbackup is deleted. Full
> backup is dump of whole disk image from vmware), unless needed for some
> recovery. Then it's mounted as disk image.
>
>>
>> The default btrfs tends to lean to write support.
>>
>>> We can use internal compression of nakivo, but not without deleting
>>> all
>>> stored data and creating empty repository.
>>> Also we wanted to do compression in btrfs, we hoped it will give
>>> more
>>> power to beesd to do it's thing (for comparing data)
>>
>> Then I guess there is not much thing we can help right now, and that
>> many extents are also slowing down file deletion just as you
>> mentioned.
>
> So i will have to experiment, if user land compression allows us to do
> some reasonble deduplication with beesd.

As long as the compression algorithm/tool can reproduce the same
compressed data for the same input, then it would be fine.

> It may maybe speed up beesd, it cannot keep up with data influx, maybe
> it's (also) because the number of file extents.
> Unfortunately it will mean some serious data juggling in production
> environment.

I'm wondering can we just remount the fs to remove the compress=zstd
mount option?

Since compress=zstd will only affect new writes and to user-space
compression should be transparent, disabling btrfs compression at any
time point should not cause problems.

Thanks,
Qu

>
> Thanks,
> Libor
>
>
>>
>> Thanks,
>> Qu
>>
>>>
>>>>
>>>> Thanks,
>>>> Qu
>>>
>>> With regards, Libor
>>>
>>>>>
>>>>> Most of the files are multi gigabyte, some of them have around
>>>>> 3TB
>>>>> -
>>>>> all are snapshots from vmware stored using nakivo.
>>>>>
>>>>> Working with filesystem - mostly deleting files seems to be
>>>>> very
>>>>> slow -
>>>>> it took several hours to delete snapshot of one machine, which
>>>>> consisted of four or five of those 3TB files.
>>>>>
>>>>> We run beesd on those data, but i think, there was this much
>>>>> metadata
>>>>> even before we started to do so.
>>>>>
>>>>> With regards,
>>>>> Libor

Re: Question about metadata size

[Libor Klepáč]

On Po, 2022-06-27 at 20:39 +0800, Qu Wenruo wrote:
> 
> 
> On 2022/6/27 20:17, Libor Klepáč wrote:
> > On Po, 2022-06-27 at 18:54 +0800, Qu Wenruo wrote:
> > > 
> > > 
> > > On 2022/6/27 18:23, Libor Klepáč wrote:
> > > > On Po, 2022-06-27 at 18:10 +0800, Qu Wenruo wrote:
> > > > > 
> > > > > 
> > > > > On 2022/6/27 17:02, Libor Klepáč wrote:
> > > > > > Hi,
> > > > > > we have filesystem like this
> > > > > > 
> > > > > > Overall:
> > > > > >        Device size:                  30.00TiB
> > > > > >        Device allocated:             24.93TiB
> > > > > >        Device unallocated:            5.07TiB
> > > > > >        Device missing:                  0.00B
> > > > > >        Used:                         24.92TiB
> > > > > >        Free (estimated):              5.07TiB      (min:
> > > > > > 2.54TiB)
> > > > > >        Data ratio:                       1.00
> > > > > >        Metadata ratio:                   1.00
> > > > > >        Global reserve:              512.00MiB      (used:
> > > > > > 0.00B)
> > > > > > 
> > > > > > Data,single: Size:24.85TiB, Used:24.84TiB (99.98%)
> > > > > >       /dev/sdc       24.85TiB
> > > > > > 
> > > > > > Metadata,single: Size:88.00GiB, Used:81.54GiB (92.65%)
> > > > > >       /dev/sdc       88.00GiB
> > > > > > 
> > > > > > System,DUP: Size:32.00MiB, Used:3.25MiB (10.16%)
> > > > > >       /dev/sdc       64.00MiB
> > > > > > 
> > > > > > Unallocated:
> > > > > >       /dev/sdc        5.07TiB
> > > > > > 
> > > > > > 
> > > > > > Is it normal to have so much metadata? We have only 119
> > > > > > files
> > > > > > with
> > > > > > size
> > > > > > of 2048 bytes or less.
> > > > > 
> > > > > That would only take around 50KiB so no problem.
> > > > > 
> > > > > > There is 885 files in total and 17 directories, we don't
> > > > > > use
> > > > > > snapshots.
> > > > > 
> > > > > The other files really depends.
> > > > > 
> > > > > Do you use compression, if so metadata usage will be greately
> > > > > increased.
> > > > 
> > > > 
> > > > Yes, we use zstd compression - filesystem is mounted with
> > > > compress-
> > > > force=zstd:9
> > > > 
> > > > > 
> > > > > For non-compressed files, the max file extent size is 128M,
> > > > > while
> > > > > for
> > > > > compressed files, the max file extent size is only 128K.
> > > > > 
> > > > > This means, for a 3TiB file, if you have compress enabled, it
> > > > > will
> > > > > take
> > > > > 24M file extents, and since each file extent takes at least
> > > > > 53
> > > > > bytes
> > > > > for
> > > > 
> > > > That is lot of extents ;)
> > > > 
> > > > > metadata, one such 3TiB file can already take over 1 GiB for
> > > > > metadata.
> > > > 
> > > > I guess there is no way to increase extent size?
> > > 
> > > Currently it's hard coded. So no way to change that yet.
> > > 
> > > But please keep in mind that, btrfs compression needs to do
> > > trade-off
> > > between writes, and the decompressed size.
> > > 
> > > E.g. if we can have an 1MiB compressed extent, but if 1020KiB are
> > > overwritten, just one 4KiB is really referred, then to read that
> > > 4KiB
> > > we
> > > need to decompress all that 1MiB just to read that 4KiB.
> > 
> > Yes, i get reason for this.
> > I just never realised the difference in extent size and it's impact
> > on
> > metadata size/number of extents.
> > 
> > > So personally speaking, if the main purpose of those large files
> > > are
> > > just to archive, not to do frequent write on, then user space
> > > compression would make more sense.
> > 
> > Ok, these files are writen once and deleted after 14 days (every 14
> > days, new full backup is created and oldest fullbackup is deleted.
> > Full
> > backup is dump of whole disk image from vmware), unless needed for
> > some
> > recovery. Then it's mounted as disk image.
> > 
> > > 
> > > The default btrfs tends to lean to write support.
> > > 
> > > > We can use internal compression of nakivo, but not without
> > > > deleting
> > > > all
> > > > stored data and creating empty repository.
> > > > Also we wanted to do compression in btrfs, we hoped it will
> > > > give
> > > > more
> > > > power to beesd to do it's thing (for comparing data)
> > > 
> > > Then I guess there is not much thing we can help right now, and
> > > that
> > > many extents are also slowing down file deletion just as you
> > > mentioned.
> > 
> > So i will have to experiment, if user land compression allows us to
> > do
> > some reasonble deduplication with beesd.
> 
> As long as the compression algorithm/tool can reproduce the same
> compressed data for the same input, then it would be fine.

That i don't know. I assume that raw disk dump from vmware should be
the same in case that data were not changed inside virtual machine. 
I just don't know if for example, one bit changes at 1TB mark (out of
all 3TB of data), if the rest of image gets different output from
compression or not.

> 
> > It may maybe speed up beesd, it cannot keep up with data influx,
> > maybe
> > it's (also) because the number of file extents.
> > Unfortunately it will mean some serious data juggling in production
> > environment.
> 
> I'm wondering can we just remount the fs to remove the compress=zstd
> mount option?
> 
> Since compress=zstd will only affect new writes and to user-space
> compression should be transparent, disabling btrfs compression at any
> time point should not cause problems.

Yeah, sure, i can remount btrfs without compression.
But the app has compression (it uses zlib) setting hardcoded in it's
data repository format, so i will have to create empty repository and
migrate data there from active repository (so i will need to squeeze
two copies of data to the same disk)

Thanks,
Libor

> 
> Thanks,
> Qu
> 
> > 
> > Thanks,
> > Libor
> > 
> > 
> > > 
> > > Thanks,
> > > Qu
> > > 
> > > > 
> > > > > 
> > > > > Thanks,
> > > > > Qu
> > > > 
> > > > With regards, Libor
> > > > 
> > > > > > 
> > > > > > Most of the files are multi gigabyte, some of them have
> > > > > > around
> > > > > > 3TB
> > > > > > -
> > > > > > all are snapshots from vmware stored using nakivo.
> > > > > > 
> > > > > > Working with filesystem - mostly deleting files seems to be
> > > > > > very
> > > > > > slow -
> > > > > > it took several hours to delete snapshot of one machine,
> > > > > > which
> > > > > > consisted of four or five of those 3TB files.
> > > > > > 
> > > > > > We run beesd on those data, but i think, there was this
> > > > > > much
> > > > > > metadata
> > > > > > even before we started to do so.
> > > > > > 
> > > > > > With regards,
> > > > > > Libor

Re: Question about metadata size

[Joshua Villwock]

—Joshua Villwock

> On Jun 27, 2022, at 5:47 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> 
> 
> 
>> On 2022/6/27 20:17, Libor Klepáč wrote:
>>> On Po, 2022-06-27 at 18:54 +0800, Qu Wenruo wrote:
>>> 
>>> 
>>> On 2022/6/27 18:23, Libor Klepáč wrote:
>>>> On Po, 2022-06-27 at 18:10 +0800, Qu Wenruo wrote:
>>>>> 
>>>>> 
>>>>> On 2022/6/27 17:02, Libor Klepáč wrote:
>>>>>> Hi,
>>>>>> we have filesystem like this
>>>>>> 
>>>>>> Overall:
>>>>>>       Device size:                  30.00TiB
>>>>>>       Device allocated:             24.93TiB
>>>>>>       Device unallocated:            5.07TiB
>>>>>>       Device missing:                  0.00B
>>>>>>       Used:                         24.92TiB
>>>>>>       Free (estimated):              5.07TiB      (min:
>>>>>> 2.54TiB)
>>>>>>       Data ratio:                       1.00
>>>>>>       Metadata ratio:                   1.00
>>>>>>       Global reserve:              512.00MiB      (used: 0.00B)
>>>>>> 
>>>>>> Data,single: Size:24.85TiB, Used:24.84TiB (99.98%)
>>>>>>      /dev/sdc       24.85TiB
>>>>>> 
>>>>>> Metadata,single: Size:88.00GiB, Used:81.54GiB (92.65%)
>>>>>>      /dev/sdc       88.00GiB
>>>>>> 
>>>>>> System,DUP: Size:32.00MiB, Used:3.25MiB (10.16%)
>>>>>>      /dev/sdc       64.00MiB
>>>>>> 
>>>>>> Unallocated:
>>>>>>      /dev/sdc        5.07TiB
>>>>>> 
>>>>>> 
>>>>>> Is it normal to have so much metadata? We have only 119 files
>>>>>> with
>>>>>> size
>>>>>> of 2048 bytes or less.
>>>>> 
>>>>> That would only take around 50KiB so no problem.
>>>>> 
>>>>>> There is 885 files in total and 17 directories, we don't use
>>>>>> snapshots.
>>>>> 
>>>>> The other files really depends.
>>>>> 
>>>>> Do you use compression, if so metadata usage will be greately
>>>>> increased.
>>>> 
>>>> 
>>>> Yes, we use zstd compression - filesystem is mounted with compress-
>>>> force=zstd:9
>>>> 
>>>>> 
>>>>> For non-compressed files, the max file extent size is 128M, while
>>>>> for
>>>>> compressed files, the max file extent size is only 128K.
>>>>> 
>>>>> This means, for a 3TiB file, if you have compress enabled, it
>>>>> will
>>>>> take
>>>>> 24M file extents, and since each file extent takes at least 53
>>>>> bytes
>>>>> for
>>>> 
>>>> That is lot of extents ;)
>>>> 
>>>>> metadata, one such 3TiB file can already take over 1 GiB for
>>>>> metadata.
>>>> 
>>>> I guess there is no way to increase extent size?
>>> 
>>> Currently it's hard coded. So no way to change that yet.
>>> 
>>> But please keep in mind that, btrfs compression needs to do trade-off
>>> between writes, and the decompressed size.
>>> 
>>> E.g. if we can have an 1MiB compressed extent, but if 1020KiB are
>>> overwritten, just one 4KiB is really referred, then to read that 4KiB
>>> we
>>> need to decompress all that 1MiB just to read that 4KiB.
>> 
>> Yes, i get reason for this.
>> I just never realised the difference in extent size and it's impact on
>> metadata size/number of extents.
>> 
>>> So personally speaking, if the main purpose of those large files are
>>> just to archive, not to do frequent write on, then user space
>>> compression would make more sense.
>> 
>> Ok, these files are writen once and deleted after 14 days (every 14
>> days, new full backup is created and oldest fullbackup is deleted. Full
>> backup is dump of whole disk image from vmware), unless needed for some
>> recovery. Then it's mounted as disk image.
>> 
>>> 
>>> The default btrfs tends to lean to write support.
>>> 
>>>> We can use internal compression of nakivo, but not without deleting
>>>> all
>>>> stored data and creating empty repository.
>>>> Also we wanted to do compression in btrfs, we hoped it will give
>>>> more
>>>> power to beesd to do it's thing (for comparing data)
>>> 
>>> Then I guess there is not much thing we can help right now, and that
>>> many extents are also slowing down file deletion just as you
>>> mentioned.
>> 
>> So i will have to experiment, if user land compression allows us to do
>> some reasonble deduplication with beesd.
> 
> As long as the compression algorithm/tool can reproduce the same
> compressed data for the same input, then it would be fine.

Does Nakivo support compression that plays nice with dedupe?

I know Veeam for example has “dedupe-friendly” as a compression option which makes it output less-compressed data, but ensures your storage appliance can dedupe it.

Not sure if they have something like that, but if so, it would probably be the best solution.

>> It may maybe speed up beesd, it cannot keep up with data influx, maybe
>> it's (also) because the number of file extents.
>> Unfortunately it will mean some serious data juggling in production
>> environment.
> 
> I'm wondering can we just remount the fs to remove the compress=zstd
> mount option?
> 
> Since compress=zstd will only affect new writes and to user-space
> compression should be transparent, disabling btrfs compression at any
> time point should not cause problems.
> 
> Thanks,
> Qu
> 
>> 
>> Thanks,
>> Libor
>> 
>> 
>>> 
>>> Thanks,
>>> Qu
>>> 
>>>> 
>>>>> 
>>>>> Thanks,
>>>>> Qu
>>>> 
>>>> With regards, Libor
>>>> 
>>>>>> 
>>>>>> Most of the files are multi gigabyte, some of them have around
>>>>>> 3TB
>>>>>> -
>>>>>> all are snapshots from vmware stored using nakivo.
>>>>>> 
>>>>>> Working with filesystem - mostly deleting files seems to be
>>>>>> very
>>>>>> slow -
>>>>>> it took several hours to delete snapshot of one machine, which
>>>>>> consisted of four or five of those 3TB files.
>>>>>> 
>>>>>> We run beesd on those data, but i think, there was this much
>>>>>> metadata
>>>>>> even before we started to do so.
>>>>>> 
>>>>>> With regards,
>>>>>> Libor

Re: Question about metadata size

[Libor Klepáč]

On Po, 2022-06-27 at 07:47 -0700, Joshua Villwock wrote:
> 
> 
> —Joshua Villwock
> 
> > On Jun 27, 2022, at 5:47 AM, Qu Wenruo <quwenruo.btrfs@gmx.com>
> > wrote:
> > 
> > 
> > 
> > > On 2022/6/27 20:17, Libor Klepáč wrote:
> > > > On Po, 2022-06-27 at 18:54 +0800, Qu Wenruo wrote:
> > > > 
> > > > 
> > > > On 2022/6/27 18:23, Libor Klepáč wrote:
> > > > > On Po, 2022-06-27 at 18:10 +0800, Qu Wenruo wrote:
> > > > > > 
> > > > > > 
> > > > > > On 2022/6/27 17:02, Libor Klepáč wrote:
> > > > > > > Hi,
> > > > > > > we have filesystem like this
> > > > > > > 
> > > > > > > Overall:
> > > > > > >       Device size:                  30.00TiB
> > > > > > >       Device allocated:             24.93TiB
> > > > > > >       Device unallocated:            5.07TiB
> > > > > > >       Device missing:                  0.00B
> > > > > > >       Used:                         24.92TiB
> > > > > > >       Free (estimated):              5.07TiB      (min:
> > > > > > > 2.54TiB)
> > > > > > >       Data ratio:                       1.00
> > > > > > >       Metadata ratio:                   1.00
> > > > > > >       Global reserve:              512.00MiB      (used:
> > > > > > > 0.00B)
> > > > > > > 
> > > > > > > Data,single: Size:24.85TiB, Used:24.84TiB (99.98%)
> > > > > > >      /dev/sdc       24.85TiB
> > > > > > > 
> > > > > > > Metadata,single: Size:88.00GiB, Used:81.54GiB (92.65%)
> > > > > > >      /dev/sdc       88.00GiB
> > > > > > > 
> > > > > > > System,DUP: Size:32.00MiB, Used:3.25MiB (10.16%)
> > > > > > >      /dev/sdc       64.00MiB
> > > > > > > 
> > > > > > > Unallocated:
> > > > > > >      /dev/sdc        5.07TiB
> > > > > > > 
> > > > > > > 
> > > > > > > Is it normal to have so much metadata? We have only 119
> > > > > > > files
> > > > > > > with
> > > > > > > size
> > > > > > > of 2048 bytes or less.
> > > > > > 
> > > > > > That would only take around 50KiB so no problem.
> > > > > > 
> > > > > > > There is 885 files in total and 17 directories, we don't
> > > > > > > use
> > > > > > > snapshots.
> > > > > > 
> > > > > > The other files really depends.
> > > > > > 
> > > > > > Do you use compression, if so metadata usage will be
> > > > > > greately
> > > > > > increased.
> > > > > 
> > > > > 
> > > > > Yes, we use zstd compression - filesystem is mounted with
> > > > > compress-
> > > > > force=zstd:9
> > > > > 
> > > > > > 
> > > > > > For non-compressed files, the max file extent size is 128M,
> > > > > > while
> > > > > > for
> > > > > > compressed files, the max file extent size is only 128K.
> > > > > > 
> > > > > > This means, for a 3TiB file, if you have compress enabled,
> > > > > > it
> > > > > > will
> > > > > > take
> > > > > > 24M file extents, and since each file extent takes at least
> > > > > > 53
> > > > > > bytes
> > > > > > for
> > > > > 
> > > > > That is lot of extents ;)
> > > > > 
> > > > > > metadata, one such 3TiB file can already take over 1 GiB
> > > > > > for
> > > > > > metadata.
> > > > > 
> > > > > I guess there is no way to increase extent size?
> > > > 
> > > > Currently it's hard coded. So no way to change that yet.
> > > > 
> > > > But please keep in mind that, btrfs compression needs to do
> > > > trade-off
> > > > between writes, and the decompressed size.
> > > > 
> > > > E.g. if we can have an 1MiB compressed extent, but if 1020KiB
> > > > are
> > > > overwritten, just one 4KiB is really referred, then to read
> > > > that 4KiB
> > > > we
> > > > need to decompress all that 1MiB just to read that 4KiB.
> > > 
> > > Yes, i get reason for this.
> > > I just never realised the difference in extent size and it's
> > > impact on
> > > metadata size/number of extents.
> > > 
> > > > So personally speaking, if the main purpose of those large
> > > > files are
> > > > just to archive, not to do frequent write on, then user space
> > > > compression would make more sense.
> > > 
> > > Ok, these files are writen once and deleted after 14 days (every
> > > 14
> > > days, new full backup is created and oldest fullbackup is
> > > deleted. Full
> > > backup is dump of whole disk image from vmware), unless needed
> > > for some
> > > recovery. Then it's mounted as disk image.
> > > 
> > > > 
> > > > The default btrfs tends to lean to write support.
> > > > 
> > > > > We can use internal compression of nakivo, but not without
> > > > > deleting
> > > > > all
> > > > > stored data and creating empty repository.
> > > > > Also we wanted to do compression in btrfs, we hoped it will
> > > > > give
> > > > > more
> > > > > power to beesd to do it's thing (for comparing data)
> > > > 
> > > > Then I guess there is not much thing we can help right now, and
> > > > that
> > > > many extents are also slowing down file deletion just as you
> > > > mentioned.
> > > 
> > > So i will have to experiment, if user land compression allows us
> > > to do
> > > some reasonble deduplication with beesd.
> > 
> > As long as the compression algorithm/tool can reproduce the same
> > compressed data for the same input, then it would be fine.
> 
> Does Nakivo support compression that plays nice with dedupe?
> 
> I know Veeam for example has “dedupe-friendly” as a compression
> option which makes it output less-compressed data, but ensures your
> storage appliance can dedupe it.
> 
> Not sure if they have something like that, but if so, it would
> probably be the best solution.
> 

Not sure, it has option to output to "Dell EMC Data Domain Boost, HPE
StoreOnce Catalyst and NEC HYDRAstor" deduplication appliances, when
you want to use local disk, you can only set compression level.
I will try to ask their support about it

Libor
 

> > > It may maybe speed up beesd, it cannot keep up with data influx,
> > > maybe
> > > it's (also) because the number of file extents.
> > > Unfortunately it will mean some serious data juggling in
> > > production
> > > environment.
> > 
> > I'm wondering can we just remount the fs to remove the
> > compress=zstd
> > mount option?
> > 
> > Since compress=zstd will only affect new writes and to user-space
> > compression should be transparent, disabling btrfs compression at
> > any
> > time point should not cause problems.
> > 
> > Thanks,
> > Qu
> > 
> > > 
> > > Thanks,
> > > Libor
> > > 
> > > 
> > > > 
> > > > Thanks,
> > > > Qu
> > > > 
> > > > > 
> > > > > > 
> > > > > > Thanks,
> > > > > > Qu
> > > > > 
> > > > > With regards, Libor
> > > > > 
> > > > > > > 
> > > > > > > Most of the files are multi gigabyte, some of them have
> > > > > > > around
> > > > > > > 3TB
> > > > > > > -
> > > > > > > all are snapshots from vmware stored using nakivo.
> > > > > > > 
> > > > > > > Working with filesystem - mostly deleting files seems to
> > > > > > > be
> > > > > > > very
> > > > > > > slow -
> > > > > > > it took several hours to delete snapshot of one machine,
> > > > > > > which
> > > > > > > consisted of four or five of those 3TB files.
> > > > > > > 
> > > > > > > We run beesd on those data, but i think, there was this
> > > > > > > much
> > > > > > > metadata
> > > > > > > even before we started to do so.
> > > > > > > 
> > > > > > > With regards,
> > > > > > > Libor

Re: Question about metadata size

[Libor Klepáč]

On Po, 2022-06-27 at 15:25 +0000, Joshua wrote:
> June 27, 2022 8:18 AM, "Libor Klepáč" <libor.klepac@bcom.cz> wrote:
> 
> > On Po, 2022-06-27 at 07:47 -0700, Joshua Villwock wrote:
> > 
> > > snip
> > > 
> > > > 
> > > > So i will have to experiment, if user land compression allows
> > > > us
> > > > to do
> > > > some reasonble deduplication with beesd.
> > > 
> > > As long as the compression algorithm/tool can reproduce the same
> > > compressed data for the same input, then it would be fine.
> > > 
> > > Does Nakivo support compression that plays nice with dedupe?
> > > 
> > > I know Veeam for example has “dedupe-friendly” as a compression
> > > option which makes it output less-compressed data, but ensures
> > > your
> > > storage appliance can dedupe it.
> > > 
> > > Not sure if they have something like that, but if so, it would
> > > probably be the best solution.
> > 
> > Looking at the veeam docs, it seems to be just compression level,
> > ie
> > very fast compression
> > 
> > Libor
> 
> Hmm, you appear to be partially correct.
> 
> Their documentation:
> https://helpcenter.veeam.com/backup/70/bp_vsphere/topic_compression.h
> tml
> Claims that:
> "Dedupe-friendly — this is an optimized compression level for very
> low CPU usage and uses a very simple, fixed dictionary. This method
> can be a good compromise when using deduplicating storage or WAN
> accelerators because, while it will lower the dedupe ratio compared
> to no compression, it will send less data to the deduplicating
> appliance."
> 
> So it didn't do what I was expecting, as it's purpose is what you
> said:  more for taking the dedupe load off the backup server(s).  
> 
> Interesting, thanks for pointing that out.
> 

Thanks for this link, i will use it to ask nakivo support. 
But i don't expect them to have it implemented.

Libor

> > 
> > > > It may maybe speed up beesd, it cannot keep up with data
> > > > influx,
> > > > maybe
> > > > it's (also) because the number of file extents.
> > > > Unfortunately it will mean some serious data juggling in
> > > > production
> > > > environment.
> > > 
> > > I'm wondering can we just remount the fs to remove the
> > > compress=zstd
> > > mount option?
> > > 
> > > Since compress=zstd will only affect new writes and to user-space
> > > compression should be transparent, disabling btrfs compression at
> > > any
> > > time point should not cause problems.
> > > 
> > > Thanks,
> > > Qu
> > > 
> > > > 
> > > > Thanks,
> > > > Libor
> > > > 
> > > > 
> > > > > 
> > > > > Thanks,
> > > > > Qu
> > > > > 
> > > > > > 
> > > > > > > 
> > > > > > > Thanks,
> > > > > > > Qu
> > > > > > 
> > > > > > With regards, Libor
> > > > > > 
> > > > > > > > 
> > > > > > > > Most of the files are multi gigabyte, some of them have
> > > > > > > > around
> > > > > > > > 3TB
> > > > > > > > -
> > > > > > > > all are snapshots from vmware stored using nakivo.
> > > > > > > > 
> > > > > > > > Working with filesystem - mostly deleting files seems
> > > > > > > > to
> > > > > > > > be
> > > > > > > > very
> > > > > > > > slow -
> > > > > > > > it took several hours to delete snapshot of one
> > > > > > > > machine,
> > > > > > > > which
> > > > > > > > consisted of four or five of those 3TB files.
> > > > > > > > 
> > > > > > > > We run beesd on those data, but i think, there was this
> > > > > > > > much
> > > > > > > > metadata
> > > > > > > > even before we started to do so.
> > > > > > > > 
> > > > > > > > With regards,
> > > > > > > > Libor

Re: Question about metadata size

[Chris Murphy]

On Mon, Jun 27, 2022 at 4:23 AM Libor Klepáč <libor.klepac@bcom.cz> wrote:

> Yes, we use zstd compression - filesystem is mounted with compress-
> force=zstd:9

compress-force means even uncompressed extents are 512KiB max, so this
could be the source of the problem

If you use just compress, then uncompressed extents can be 128MiB max.

So it's a set of tradeoffs depending on how compressible the files are.



-- 
Chris Murphy

Re: Question about metadata size

[remi]

As I understand it, (which can be mistaken), the files in question are VM Images, which will have a mix of compressible and uncompressible data.  BTRFS however, will mark the file as uncompressible when it tries to write such, and no compression will then work on the compressible parts.  (IE.. to actually have any useful compression in this case, compress-force is needed.)

On Mon, Jun 27, 2022, at 2:12 PM, Chris Murphy wrote:
> On Mon, Jun 27, 2022 at 4:23 AM Libor Klepáč <libor.klepac@bcom.cz> wrote:
>
>> Yes, we use zstd compression - filesystem is mounted with compress-
>> force=zstd:9
>
> compress-force means even uncompressed extents are 512KiB max, so this
> could be the source of the problem
>
> If you use just compress, then uncompressed extents can be 128MiB max.
>
> So it's a set of tradeoffs depending on how compressible the files are.
>
>
>
> -- 
> Chris Murphy

Re: Question about metadata size

[Libor Klepáč]

Hi,

On Po, 2022-06-27 at 15:25 +0000, Joshua wrote:
> 
> Hmm, you appear to be partially correct.
> 
> Their documentation:
> https://helpcenter.veeam.com/backup/70/bp_vsphere/topic_compression.h
> tml
> Claims that:
> "Dedupe-friendly — this is an optimized compression level for very
> low CPU usage and uses a very simple, fixed dictionary. This method
> can be a good compromise when using deduplicating storage or WAN
> accelerators because, while it will lower the dedupe ratio compared
> to no compression, it will send less data to the deduplicating
> appliance."
> 
> So it didn't do what I was expecting, as it's purpose is what you
> said:  more for taking the dedupe load off the backup server(s).  
> 
> Interesting, thanks for pointing that out.
> 

I just got answer from nakivo support, about compression.
It compresses separately over 4MB blocks of data.
So it should/may be dedupe friendly?

Libor


> > 
> > > > It may maybe speed up beesd, it cannot keep up with data
> > > > influx,
> > > > maybe
> > > > it's (also) because the number of file extents.
> > > > Unfortunately it will mean some serious data juggling in
> > > > production
> > > > environment.
> > > 
> > > I'm wondering can we just remount the fs to remove the
> > > compress=zstd
> > > mount option?
> > > 
> > > Since compress=zstd will only affect new writes and to user-space
> > > compression should be transparent, disabling btrfs compression at
> > > any
> > > time point should not cause problems.
> > > 
> > > Thanks,
> > > Qu
> > > 
> > > > 
> > > > Thanks,
> > > > Libor
> > > > 
> > > > 
> > > > > 
> > > > > Thanks,
> > > > > Qu
> > > > > 
> > > > > > 
> > > > > > > 
> > > > > > > Thanks,
> > > > > > > Qu
> > > > > > 
> > > > > > With regards, Libor
> > > > > > 
> > > > > > > > 
> > > > > > > > Most of the files are multi gigabyte, some of them have
> > > > > > > > around
> > > > > > > > 3TB
> > > > > > > > -
> > > > > > > > all are snapshots from vmware stored using nakivo.
> > > > > > > > 
> > > > > > > > Working with filesystem - mostly deleting files seems
> > > > > > > > to
> > > > > > > > be
> > > > > > > > very
> > > > > > > > slow -
> > > > > > > > it took several hours to delete snapshot of one
> > > > > > > > machine,
> > > > > > > > which
> > > > > > > > consisted of four or five of those 3TB files.
> > > > > > > > 
> > > > > > > > We run beesd on those data, but i think, there was this
> > > > > > > > much
> > > > > > > > metadata
> > > > > > > > even before we started to do so.
> > > > > > > > 
> > > > > > > > With regards,
> > > > > > > > Libor