CEPH to BTRFS over NFS results in no compression

CEPH to BTRFS over NFS results in no compression

[Konstantin V. Gavrilenko]

Hi list, 

I have noticed an interesting and surprising behaviour of my BTRFS with regards to compression of the files and NFS. 

I have BTRFS RAID10 with 8 disks , that is mounted with the " nofail,noatime,space_cache=v2,compress-force=zstd:9,subvol=@cloudstack-secondary" flags and is exported via NFS. 

When I create a snapshot of a disk in Cloudstack from CEPH and save it to a secondary storage to this BTRFS RAID10 over NFS, the file does not compress, despite the compress-force mount option being set on FS 
So in the below example, the file eeceaf0e-9780-408b-a748-1495d517a9b6 was copied over NFS and is not compressed. When I copy the same file directly on a host, it does get compressed pretty well, as per example below. 


root@backup1 : /mnt/backup1/cloudstack-secondary/snapshots/49/5355: compsize eeceaf0e-9780-408b-a748-1495d517a9b6 
Type Perc Disk Usage Uncompressed Referenced 
TOTAL 99% 9.8G 9.8G 9.8G 
none 100% 9.8G 9.8G 9.8G 
zstd 3% 4.0K 128K 124K 

root@backup1 : /mnt/backup1/cloudstack-secondary/snapshots/49/5355: cp eeceaf0e-9780-408b-a748-1495d517a9b6 testfile 
'eeceaf0e-9780-408b-a748-1495d517a9b6' -> 'testfile' 


root@backup1 : /mnt/backup1/cloudstack-secondary/snapshots/49/5355: compsize testfile 
Type Perc Disk Usage Uncompressed Referenced 
TOTAL 38% 3.7G 9.8G 9.8G 
none 100% 1.1G 1.1G 1.1G 
zstd 30% 2.6G 8.6G 8.6G 

root@backup1: /mnt/backup1/cloudstack-secondary/snapshots/49/5355  btrfs filesystem defrag -v -f -czstd ./eeceaf0e-9780-408b-a748-1495d517a9b6 
./eeceaf0e-9780-408b-a748-1495d517a9b6 

root@backup1: /mnt/backup1/cloudstack-secondary/snapshots/49/5355  compsize eeceaf0e-9780-408b-a748-1495d517a9b6 
Type Perc Disk Usage Uncompressed Referenced 
TOTAL 38% 3.7G 9.8G 9.8G 
none 100% 1.1G 1.1G 1.1G 
zstd 30% 2.6G 8.6G 8.6G 



So what I have checked so far what works 
- after the original files is copied over NFS, the copy of the file using #cp gets compressed. 
- after the original files is copied over NFS, the original file can be compressed using #btrfs defrag -czstd option 
- If I copy the original file to some other host, and copy it back via NFS using cp, it does get compressed. 

So the problem seems to appear only when the file is exported from Ceph and copied to NFS. 

Any hints what could be causing such a behaviour? 



Yours sincerely, 
Kos

Re: CEPH to BTRFS over NFS results in no compression

[Lukas Straub]

[-- Attachment #1: Type: text/plain, Size: 1730 bytes --]

On Sun, 2 Jan 2022 11:32:53 +0000 (UTC)
"Konstantin V. Gavrilenko" <k.gavrilenko@arhont.com> wrote:

> Hi list, 
> 
> I have noticed an interesting and surprising behaviour of my BTRFS with regards to compression of the files and NFS. 
> 
> I have BTRFS RAID10 with 8 disks , that is mounted with the " nofail,noatime,space_cache=v2,compress-force=zstd:9,subvol=@cloudstack-secondary" flags and is exported via NFS. 
> 
> When I create a snapshot of a disk in Cloudstack from CEPH and save it to a secondary storage to this BTRFS RAID10 over NFS, the file does not compress, despite the compress-force mount option being set on FS 
> So in the below example, the file eeceaf0e-9780-408b-a748-1495d517a9b6 was copied over NFS and is not compressed. When I copy the same file directly on a host, it does get compressed pretty well, as per example below. 
> 
> [...]
> 
> 
> So what I have checked so far what works 
> - after the original files is copied over NFS, the copy of the file using #cp gets compressed. 
> - after the original files is copied over NFS, the original file can be compressed using #btrfs defrag -czstd option 
> - If I copy the original file to some other host, and copy it back via NFS using cp, it does get compressed. 
> 
> So the problem seems to appear only when the file is exported from Ceph and copied to NFS. 
> 
> Any hints what could be causing such a behaviour? 
> 
> 
> 
> Yours sincerely, 
> Kos
> 

Btrfs doesn't compress direct-io writes. This suggests that whatever you use for "I create a snapshot of a disk in Cloudstack from CEPH and save it to a secondary storage to this BTRFS RAID10 over NFS", it writes with O_DIRECT.

Regards,
Lukas Straub

-- 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: CEPH to BTRFS over NFS results in no compression

[Konstantin V. Gavrilenko]

Thanks for clarification Lukas. I didn't know that. 

I guess the easiest option for me is to recompress snapshots on a daily basis after they get uploaded.

yours,
kos


----- Original Message -----
> From: "Lukas Straub" <lukasstraub2@web.de>
> To: "Konstantin V Gavrilenko" <k.gavrilenko@arhont.com>
> Cc: "linux-btrfs" <linux-btrfs@vger.kernel.org>
> Sent: Sunday, 2 January, 2022 15:08:33
> Subject: Re: CEPH to BTRFS over NFS results in no compression

> On Sun, 2 Jan 2022 11:32:53 +0000 (UTC)
> "Konstantin V. Gavrilenko" <k.gavrilenko@arhont.com> wrote:
> 
>> Hi list,
>> 
>> I have noticed an interesting and surprising behaviour of my BTRFS with regards
>> to compression of the files and NFS.
>> 
>> I have BTRFS RAID10 with 8 disks , that is mounted with the "
>> nofail,noatime,space_cache=v2,compress-force=zstd:9,subvol=@cloudstack-secondary"
>> flags and is exported via NFS.
>> 
>> When I create a snapshot of a disk in Cloudstack from CEPH and save it to a
>> secondary storage to this BTRFS RAID10 over NFS, the file does not compress,
>> despite the compress-force mount option being set on FS
>> So in the below example, the file eeceaf0e-9780-408b-a748-1495d517a9b6 was
>> copied over NFS and is not compressed. When I copy the same file directly on a
>> host, it does get compressed pretty well, as per example below.
>> 
>> [...]
>> 
>> 
>> So what I have checked so far what works
>> - after the original files is copied over NFS, the copy of the file using #cp
>> gets compressed.
>> - after the original files is copied over NFS, the original file can be
>> compressed using #btrfs defrag -czstd option
>> - If I copy the original file to some other host, and copy it back via NFS using
>> cp, it does get compressed.
>> 
>> So the problem seems to appear only when the file is exported from Ceph and
>> copied to NFS.
>> 
>> Any hints what could be causing such a behaviour?
>> 
>> 
>> 
>> Yours sincerely,
>> Kos
>> 
> 
> Btrfs doesn't compress direct-io writes. This suggests that whatever you use for
> "I create a snapshot of a disk in Cloudstack from CEPH and save it to a
> secondary storage to this BTRFS RAID10 over NFS", it writes with O_DIRECT.
> 
> Regards,
> Lukas Straub
> 
> --