Re: Feature requests: online backup - defrag - change RAID level

["Austin S. Hemmelgarn" <ahferroin7@gmail.com>]

On 2019-09-09 15:26, webmaster@zedlx.com wrote:
> This post is a reply to Remi Gauvin's post, but the email got lost so I 
> can't reply to him directly.
> 
> Remi Gauvin wrote on 2019-09-09 17:24 :
>>
>> On 2019-09-09 11:29 a.m., Graham Cobb wrote:
>>
>>>  and does anyone really care about
>>> defrag any more?).
>>>
>>
>>
>> Err, yes, yes absolutely.
>>
>> I don't have any issues with the current btrfs defrag implementions, but
>> it's *vital* for btrfs. (which works just as the OP requested, as far as
>> I can tell, recursively for a subvolume)
>>
>> Just booting Windows on a BTRFS virtual image, for example, will create
>> almost 20,000 file fragments.  Even on SSD's, you get into problems
>> trying to work with files that are over 200,000 fragments.
>>
>> Another huge problem is rsync --inplace.  which is perfect backup
>> solution to take advantage of BTRFS snapshots, but fragments larges
>> files into tiny pieces (and subsequently creates files that are very
>> slow to read.).. for some reason, autodefrag doesn't catch that one 
>> either.
>>
>> But the wiki could do a beter job of trying to explain that the snapshot
>> duplication of defrag only affects the fragmented portions.  As I
>> understand, it's really only a problem when using defrag to change
>> compression.
> 
> 
> Ok, a few things.
> 
> First, my defrag suggestion doesn't EVER unshare extents. The defrag 
> should never unshare, not even a single extent. Why? Because it violates 
> the expectation that defrag would not decrease free space.
No, it should by default not unshare, but still allow the possibility of 
unsharing extents.  Sometimes completely removing all fragmentation is 
more important than space usage.
> 
> Defrag may break up extents. Defrag may fuse extents. But it shouln't 
> ever unshare extents.
Actually, spitting or merging extents will unshare them in a large 
majority of cases.
> 
> Therefore, I doubt that the current defrag does "just as the OP 
> requested". Nonsense. The current implementation does the unsharing all 
> the time.
> 
> Second, I never used btrfs defrag in my life, despite mananging at least 
> 10 btrfs filesystems. I can't. Because, all my btrfs volumes have lot of 
> subvolumes, so I'm afraid that defrag will unshare much more than I can 
> tolerate. In my subvolumes, over 90% of data is shared. If all 
> subvolumes were to be unshared, the disk usage would likely increase 
> tenfold, and that I cannot afford.
> 
> I agree that btrfs defrag is vital. But currently, it's unusable for 
> many use cases.
> 
> Also, I don't quite understand what the poster means by "the snapshot 
> duplication of defrag only affects the fragmented portions". Possibly it 
> means approximately: if a file wasn't modified in the current (latest) 
> subvolume, it doesn't need to be unshared. But, that would still unshare 
> all the log files, for example, even all files that have been appended, 
> etc... that's quite bad. Even if just one byte was appended to a log 
> file, then defrag will unshare the entire file (I suppose).
> 
What it means is that defrag will only ever touch a file if that file 
has extents that require defragmentation, and will then only touch 
extents that are smaller than the target extent size (32M by default, 
configurable at run-time with the `-t` option for the defrag command) 
and possibly those directly adjacent to such extents (because it might 
merge the small extents into larger neighbors, which will in turn 
rewrite the larger extent too).

This, in turn, leads to a couple of interesting behaviors:

* If you have a subvolume with snapshots , it may or may not break 
reflinks between that subvolume and it's snapshots, but will not break 
any of the reflinks between the snapshots themselves.
* When dealing with append-only files that are significantly larger than 
the target extent size which are defragmented regularly, only extents 
near the end of the file are likely to be unshared by the operation.
* If you fully defragment a subvolume, then snapshot it, then defrag it 
again, the second defrag will not unshare anything unless you were 
writing to the subvolume or snapshot while the second defrag was running.
* There's almost no net benefit to not defragmenting when dealing with 
very large files that mostly see internal rewrites (VM disk images, 
large databases, etc) because every internal rewrite will implicitly 
unshare extents anyway.