How to disable/revoke 'compression'?

How to disable/revoke 'compression'?

[Cloud Admin]

Hi,
I used the mount option 'compression' on some mounted sub volumes. How
can I revoke the compression? Means to delete the option and get all
data uncompressed on this volume.
Is it enough to remount the sub volume without this option? Or is it
necessary to do some addional step (balancing?) to get all stored data
uncompressed. Beside of it, is it possible to find out what the real
and compressed size of a file, for example or the ratio?
Bye
   Frank

Re: How to disable/revoke 'compression'?

[Adam Borowski]

On Sun, Sep 03, 2017 at 07:32:01PM +0200, Cloud Admin wrote:
> Hi,
> I used the mount option 'compression' on some mounted sub volumes. How
> can I revoke the compression? Means to delete the option and get all
> data uncompressed on this volume.
> Is it enough to remount the sub volume without this option? Or is it
> necessary to do some addional step (balancing?) to get all stored data
> uncompressed.

If you set it via mount option, removing the option is enough to disable
compression for _new_ files.  Other ways are chattr +c and btrfs-property,
but if you haven't heard about those you almost surely don't have such
attributes set.

After remounting, you may uncompress existing files.  Balancing won't do
this as it moves extents around without looking inside; defrag on the other
hand rewrites extents thus as a side effect it applies new [non]compression
settings.  Thus: ｢btrfs fi defrag -r /path/to/filesystem｣.

> Beside of it, is it possible to find out what the real and compressed size
> of a file, for example or the ratio?

Currently not.

I've once written a tool which does this, but 1. it's extremely slow, 2.
insane, 3. so insane a certain member of this list would kill me had I
distributed the tool.  Thus, I'd need to rewrite it first...


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢰⠒⠀⣿⡁ Vat kind uf sufficiently advanced technology iz dis!?
⢿⡄⠘⠷⠚⠋⠀                                 -- Genghis Ht'rok'din
⠈⠳⣄⠀⠀⠀⠀ 

Re: How to disable/revoke 'compression'?

[Hans van Kranenburg]

On 09/03/2017 08:06 PM, Adam Borowski wrote:
> On Sun, Sep 03, 2017 at 07:32:01PM +0200, Cloud Admin wrote:
>> Hi,
>> I used the mount option 'compression' on some mounted sub volumes. How
>> can I revoke the compression? Means to delete the option and get all
>> data uncompressed on this volume.
>> Is it enough to remount the sub volume without this option? Or is it
>> necessary to do some addional step (balancing?) to get all stored data
>> uncompressed.
> 
> If you set it via mount option, removing the option is enough to disable
> compression for _new_ files.  Other ways are chattr +c and btrfs-property,
> but if you haven't heard about those you almost surely don't have such
> attributes set.
> 
> After remounting, you may uncompress existing files.  Balancing won't do
> this as it moves extents around without looking inside; defrag on the other
> hand rewrites extents thus as a side effect it applies new [non]compression
> settings.  Thus: ｢btrfs fi defrag -r /path/to/filesystem｣.
> 
>> Beside of it, is it possible to find out what the real and compressed size
>> of a file, for example or the ratio?
> 
> Currently not.
> 
> I've once written a tool which does this, but 1. it's extremely slow, 2.
> insane, 3. so insane a certain member of this list would kill me had I
> distributed the tool.  Thus, I'd need to rewrite it first...

Heh, I wouldn't do that, since I need you to do my debian uploads. :D

But it would certainly help to be a bit less stubborn only wanting to
code in the language that matches your country code. :O

Or maybe I can help a bit, since it sounds like a nice one for the
coding examples in the lib. ;] Days are getting shorter again, so the
amount of indoor coding activity will hopefully increase a bit again soon.

-- 
Hans van Kranenburg

Re: How to disable/revoke 'compression'?

[Qu Wenruo]

On 2017年09月04日 02:06, Adam Borowski wrote:
> On Sun, Sep 03, 2017 at 07:32:01PM +0200, Cloud Admin wrote:
>> Hi,
>> I used the mount option 'compression' on some mounted sub volumes. How
>> can I revoke the compression? Means to delete the option and get all
>> data uncompressed on this volume.
>> Is it enough to remount the sub volume without this option? Or is it
>> necessary to do some addional step (balancing?) to get all stored data
>> uncompressed.
> 
> If you set it via mount option, removing the option is enough to disable
> compression for _new_ files.  Other ways are chattr +c and btrfs-property,
> but if you haven't heard about those you almost surely don't have such
> attributes set.
> 
> After remounting, you may uncompress existing files.  Balancing won't do
> this as it moves extents around without looking inside; defrag on the other
> hand rewrites extents thus as a side effect it applies new [non]compression
> settings.  Thus: ｢btrfs fi defrag -r /path/to/filesystem｣.
> 
>> Beside of it, is it possible to find out what the real and compressed size
>> of a file, for example or the ratio?
> 
> Currently not.
> 
> I've once written a tool which does this, but 1. it's extremely slow, 2.
> insane, 3. so insane a certain member of this list would kill me had I
> distributed the tool.  Thus, I'd need to rewrite it first...

AFAIK the only method to determine the compression ratio is to check the 
EXTENT_DATA key and its corresponding file_extent_item structure.
(Which I assume Adam is doing this way)

In that structure is records its on-disk data size and in-memory data 
size. (All rounded up to sectorsize, which is 4K in most case)
So in theory it's possible to determine the compression ratio.

The only method I can think of (maybe I forgot some methods?) is to use 
offline tool (btrfs-debug-tree) to check that.
FS APIs like fiemap doesn't even support to report on-disk data size so 
we can't use it.


But the problem is more complicated, especially when compressed CoW is 
involved.

For example, there is an extent (A) which represents the data for inode 
258, range [0,128k).
On disk size its just 4K.

And when we write the range [32K, 64K), which get CoWed and compressed, 
resulting a new file extent (B) for inode 258, range [32K, 64K), and on 
disk size is 4K as an example.

Then file extent layout for 258 will be:
[0,32k):  range [0,32K) of uncompressed Extent A
[32k, 64k): range [0,32k) of uncompressed Extent B
[64k, 128k): range [64k, 128K) of uncompressed Extent A.

And on disk extent size is 4K (compressed Extent A) + 4K (compressed 
Extent B) = 8K.

Before the write, the compresstion ratio is 4K/128K = 3.125%
While after write, the compression ratio is 8K/128K = 6.25%

Not to mention that it's possible to have uncompressed file extent.

So it's complicated even we're just using offline tool to determine the 
compression ratio of btrfs compressed file.

Thanks,
Qu

> 
> 
> Meow!
> 

Re: How to disable/revoke 'compression'?

[Adam Borowski]

On Mon, Sep 04, 2017 at 07:55:27AM +0800, Qu Wenruo wrote:
> On 2017年09月04日 02:06, Adam Borowski wrote:
> > I've once written a tool which does this, but 1. it's extremely slow, 2.
> > insane, 3. so insane a certain member of this list would kill me had I
> > distributed the tool.  Thus, I'd need to rewrite it first...
> 
> AFAIK the only method to determine the compression ratio is to check the
> EXTENT_DATA key and its corresponding file_extent_item structure.
> (Which I assume Adam is doing this way)
> 
> In that structure is records its on-disk data size and in-memory data size.
> (All rounded up to sectorsize, which is 4K in most case)
> So in theory it's possible to determine the compression ratio.
> 
> The only method I can think of (maybe I forgot some methods?) is to use
> offline tool (btrfs-debug-tree) to check that.
> FS APIs like fiemap doesn't even support to report on-disk data size so we
> can't use it.

BTRFS_IOC_TREE_SEARCH_V2 returns all we want to know; its only downside is
being root only.

> But the problem is more complicated, especially when compressed CoW is
> involved.
> 
> For example, there is an extent (A) which represents the data for inode 258,
> range [0,128k).
> On disk size its just 4K.
> 
> And when we write the range [32K, 64K), which get CoWed and compressed,
> resulting a new file extent (B) for inode 258, range [32K, 64K), and on disk
> size is 4K as an example.
> 
> Then file extent layout for 258 will be:
> [0,32k):  range [0,32K) of uncompressed Extent A
> [32k, 64k): range [0,32k) of uncompressed Extent B
> [64k, 128k): range [64k, 128K) of uncompressed Extent A.
> 
> And on disk extent size is 4K (compressed Extent A) + 4K (compressed Extent
> B) = 8K.
> 
> Before the write, the compresstion ratio is 4K/128K = 3.125%
> While after write, the compression ratio is 8K/128K = 6.25%

There's no real meaningful way to speak about compression ratio of a partial
extent.  Thus, I decided to, for every extent, take compressed:uncompressed
sizes of the whole extent, no matter whether the file uses only a few bytes
of that extent or references it a thousand times.

> Not to mention that it's possible to have uncompressed file extent.

Yeah, the tool gives a report like:
all   74%  9.2M/  13M
lzo   68%  7.1M/  11M
none 100%  2.1M/ 2.1M
as you typically have a mix of compressible and uncompressible data.


喵!
-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢰⠒⠀⣿⡁ Vat kind uf sufficiently advanced technology iz dis!?
⢿⡄⠘⠷⠚⠋⠀                                 -- Genghis Ht'rok'din
⠈⠳⣄⠀⠀⠀⠀ 

Re: How to disable/revoke 'compression'?

[Qu Wenruo]

On 2017年09月04日 08:14, Adam Borowski wrote:
> On Mon, Sep 04, 2017 at 07:55:27AM +0800, Qu Wenruo wrote:
>> On 2017年09月04日 02:06, Adam Borowski wrote:
>>> I've once written a tool which does this, but 1. it's extremely slow, 2.
>>> insane, 3. so insane a certain member of this list would kill me had I
>>> distributed the tool.  Thus, I'd need to rewrite it first...
>>
>> AFAIK the only method to determine the compression ratio is to check the
>> EXTENT_DATA key and its corresponding file_extent_item structure.
>> (Which I assume Adam is doing this way)
>>
>> In that structure is records its on-disk data size and in-memory data size.
>> (All rounded up to sectorsize, which is 4K in most case)
>> So in theory it's possible to determine the compression ratio.
>>
>> The only method I can think of (maybe I forgot some methods?) is to use
>> offline tool (btrfs-debug-tree) to check that.
>> FS APIs like fiemap doesn't even support to report on-disk data size so we
>> can't use it.
> 
> BTRFS_IOC_TREE_SEARCH_V2 returns all we want to know; its only downside is
> being root only.

Just forgot that.

> 
>> But the problem is more complicated, especially when compressed CoW is
>> involved.
>>
>> For example, there is an extent (A) which represents the data for inode 258,
>> range [0,128k).
>> On disk size its just 4K.
>>
>> And when we write the range [32K, 64K), which get CoWed and compressed,
>> resulting a new file extent (B) for inode 258, range [32K, 64K), and on disk
>> size is 4K as an example.
>>
>> Then file extent layout for 258 will be:
>> [0,32k):  range [0,32K) of uncompressed Extent A
>> [32k, 64k): range [0,32k) of uncompressed Extent B
>> [64k, 128k): range [64k, 128K) of uncompressed Extent A.
>>
>> And on disk extent size is 4K (compressed Extent A) + 4K (compressed Extent
>> B) = 8K.
>>
>> Before the write, the compresstion ratio is 4K/128K = 3.125%
>> While after write, the compression ratio is 8K/128K = 6.25%
> 
> There's no real meaningful way to speak about compression ratio of a partial
> extent.  Thus, I decided to, for every extent, take compressed:uncompressed
> sizes of the whole extent, no matter whether the file uses only a few bytes
> of that extent or references it a thousand times.

Very clever move.

> 
>> Not to mention that it's possible to have uncompressed file extent.
> 
> Yeah, the tool gives a report like:
> all   74%  9.2M/  13M
> lzo   68%  7.1M/  11M
> none 100%  2.1M/ 2.1M
> as you typically have a mix of compressible and uncompressible data.

Looks quite nice!

Thanks,
Qu

> 
> 
> 喵!
> 

Re: How to disable/revoke 'compression'?

[Adam Borowski]

On Sun, Sep 03, 2017 at 08:30:59PM +0200, Hans van Kranenburg wrote:
> On 09/03/2017 08:06 PM, Adam Borowski wrote:
> > On Sun, Sep 03, 2017 at 07:32:01PM +0200, Cloud Admin wrote:
> >> Beside of it, is it possible to find out what the real and compressed size
> >> of a file, for example or the ratio?
> > 
> > Currently not.
> > 
> > I've once written a tool which does this, but 1. it's extremely slow, 2.
> > insane, 3. so insane a certain member of this list would kill me had I
> > distributed the tool.  Thus, I'd need to rewrite it first...
> 
> Heh, I wouldn't do that, since I need you to do my debian uploads. :D
> 
> But it would certainly help to be a bit less stubborn only wanting to
> code in the language that matches your country code. :O

It's even funnier that I prefer the C.UTF-8 locale over pl_PL.UTF-8, and
the C code works several orders of magnitude faster than the Perl+Python
version.

Ok, ok, in this case it's C++ but only because I couldn't be arsed to
implement a <set> the hard way.

> Or maybe I can help a bit, since it sounds like a nice one for the
> coding examples in the lib. ;]

The lib is a great help -- the ioctls are totally undocumented, your code is
a lot easier than RTFKing.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢰⠒⠀⣿⡁ Vat kind uf sufficiently advanced technology iz dis!?
⢿⡄⠘⠷⠚⠋⠀                                 -- Genghis Ht'rok'din
⠈⠳⣄⠀⠀⠀⠀ 

Re: How to disable/revoke 'compression'?

[Austin S. Hemmelgarn]

On 2017-09-03 19:55, Qu Wenruo wrote:
> 
> 
> On 2017年09月04日 02:06, Adam Borowski wrote:
>> On Sun, Sep 03, 2017 at 07:32:01PM +0200, Cloud Admin wrote:
>>> Hi,
>>> I used the mount option 'compression' on some mounted sub volumes. How
>>> can I revoke the compression? Means to delete the option and get all
>>> data uncompressed on this volume.
>>> Is it enough to remount the sub volume without this option? Or is it
>>> necessary to do some addional step (balancing?) to get all stored data
>>> uncompressed.
>>
>> If you set it via mount option, removing the option is enough to disable
>> compression for _new_ files.  Other ways are chattr +c and 
>> btrfs-property,
>> but if you haven't heard about those you almost surely don't have such
>> attributes set.
>>
>> After remounting, you may uncompress existing files.  Balancing won't do
>> this as it moves extents around without looking inside; defrag on the 
>> other
>> hand rewrites extents thus as a side effect it applies new 
>> [non]compression
>> settings.  Thus: ｢btrfs fi defrag -r /path/to/filesystem｣.
>>
>>> Beside of it, is it possible to find out what the real and compressed 
>>> size
>>> of a file, for example or the ratio?
>>
>> Currently not.
>>
>> I've once written a tool which does this, but 1. it's extremely slow, 2.
>> insane, 3. so insane a certain member of this list would kill me had I
>> distributed the tool.  Thus, I'd need to rewrite it first...
> 
> AFAIK the only method to determine the compression ratio is to check the 
> EXTENT_DATA key and its corresponding file_extent_item structure.
> (Which I assume Adam is doing this way)
> 
> In that structure is records its on-disk data size and in-memory data 
> size. (All rounded up to sectorsize, which is 4K in most case)
> So in theory it's possible to determine the compression ratio.
> 
> The only method I can think of (maybe I forgot some methods?) is to use 
> offline tool (btrfs-debug-tree) to check that.
> FS APIs like fiemap doesn't even support to report on-disk data size so 
> we can't use it.
> 
> 
> But the problem is more complicated, especially when compressed CoW is 
> involved.
> 
> For example, there is an extent (A) which represents the data for inode 
> 258, range [0,128k).
> On disk size its just 4K.
> 
> And when we write the range [32K, 64K), which get CoWed and compressed, 
> resulting a new file extent (B) for inode 258, range [32K, 64K), and on 
> disk size is 4K as an example.
> 
> Then file extent layout for 258 will be:
> [0,32k):  range [0,32K) of uncompressed Extent A
> [32k, 64k): range [0,32k) of uncompressed Extent B
> [64k, 128k): range [64k, 128K) of uncompressed Extent A.
> 
> And on disk extent size is 4K (compressed Extent A) + 4K (compressed 
> Extent B) = 8K.
> 
> Before the write, the compresstion ratio is 4K/128K = 3.125%
> While after write, the compression ratio is 8K/128K = 6.25%
> 
> Not to mention that it's possible to have uncompressed file extent.
> 
> So it's complicated even we're just using offline tool to determine the 
> compression ratio of btrfs compressed file.
Out of curiosity, is there any easier method if you just want an 
aggregate ratio for the whole filesystem?  The intuitive option of 
comparing `du -sh` output to how much space is actually used in chunks 
is obviously out because that will count sparse ranges as 'compressed', 
and there should actually be a significant difference in values there 
for an uncompressed filesystem (the chunk usage should be higher).

Re: How to disable/revoke 'compression'?

[Qu Wenruo]

On 2017年09月05日 19:36, Austin S. Hemmelgarn wrote:
> On 2017-09-03 19:55, Qu Wenruo wrote:
>>
>>
>> On 2017年09月04日 02:06, Adam Borowski wrote:
>>> On Sun, Sep 03, 2017 at 07:32:01PM +0200, Cloud Admin wrote:
>>>> Hi,
>>>> I used the mount option 'compression' on some mounted sub volumes. How
>>>> can I revoke the compression? Means to delete the option and get all
>>>> data uncompressed on this volume.
>>>> Is it enough to remount the sub volume without this option? Or is it
>>>> necessary to do some addional step (balancing?) to get all stored data
>>>> uncompressed.
>>>
>>> If you set it via mount option, removing the option is enough to disable
>>> compression for _new_ files.  Other ways are chattr +c and 
>>> btrfs-property,
>>> but if you haven't heard about those you almost surely don't have such
>>> attributes set.
>>>
>>> After remounting, you may uncompress existing files.  Balancing won't do
>>> this as it moves extents around without looking inside; defrag on the 
>>> other
>>> hand rewrites extents thus as a side effect it applies new 
>>> [non]compression
>>> settings.  Thus: ｢btrfs fi defrag -r /path/to/filesystem｣.
>>>
>>>> Beside of it, is it possible to find out what the real and 
>>>> compressed size
>>>> of a file, for example or the ratio?
>>>
>>> Currently not.
>>>
>>> I've once written a tool which does this, but 1. it's extremely slow, 2.
>>> insane, 3. so insane a certain member of this list would kill me had I
>>> distributed the tool.  Thus, I'd need to rewrite it first...
>>
>> AFAIK the only method to determine the compression ratio is to check 
>> the EXTENT_DATA key and its corresponding file_extent_item structure.
>> (Which I assume Adam is doing this way)
>>
>> In that structure is records its on-disk data size and in-memory data 
>> size. (All rounded up to sectorsize, which is 4K in most case)
>> So in theory it's possible to determine the compression ratio.
>>
>> The only method I can think of (maybe I forgot some methods?) is to 
>> use offline tool (btrfs-debug-tree) to check that.
>> FS APIs like fiemap doesn't even support to report on-disk data size 
>> so we can't use it.
>>
>>
>> But the problem is more complicated, especially when compressed CoW is 
>> involved.
>>
>> For example, there is an extent (A) which represents the data for 
>> inode 258, range [0,128k).
>> On disk size its just 4K.
>>
>> And when we write the range [32K, 64K), which get CoWed and 
>> compressed, resulting a new file extent (B) for inode 258, range [32K, 
>> 64K), and on disk size is 4K as an example.
>>
>> Then file extent layout for 258 will be:
>> [0,32k):  range [0,32K) of uncompressed Extent A
>> [32k, 64k): range [0,32k) of uncompressed Extent B
>> [64k, 128k): range [64k, 128K) of uncompressed Extent A.
>>
>> And on disk extent size is 4K (compressed Extent A) + 4K (compressed 
>> Extent B) = 8K.
>>
>> Before the write, the compresstion ratio is 4K/128K = 3.125%
>> While after write, the compression ratio is 8K/128K = 6.25%
>>
>> Not to mention that it's possible to have uncompressed file extent.
>>
>> So it's complicated even we're just using offline tool to determine 
>> the compression ratio of btrfs compressed file.
> Out of curiosity, is there any easier method if you just want an 
> aggregate ratio for the whole filesystem?  The intuitive option of 
> comparing `du -sh` output to how much space is actually used in chunks 
> is obviously out because that will count sparse ranges as 'compressed', 
> and there should actually be a significant difference in values there 
> for an uncompressed filesystem (the chunk usage should be higher).

I can be totally wrong (since I just forgot the quite obvious 
SEARCH_TREE ioctl), but according to btrfs on-disk format, only 
EXTENT_DATA contains the compression ratio (ram size and on-disk size).

So to get ratio for the whole fs, one needs to iterate through the whole 
extent tree, and follows the (any is enough) backref to locate the 
EXTENT_DATA and get the compression ratio.

That's to say, that will be slow anyway.

Thanks,
Qu

> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html