Re: BUG: btrfs send: Kernel's memory usage rises until OOM kernel panic after sending ~37GiB

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

[-- Attachment #1.1: Type: text/plain, Size: 2403 bytes --]



On 2019/10/27 下午8:55, Atemu wrote:
>> This depends on how shared one file extent is.
> 
> But shouldn't it catch that and cancel the btrfs send before it panics
> the kernel due to its memory usage?

Backref walk is quite tricky in btrfs, we don't really have a good way
to detect whether it's a good idea or not, until we crash...

But at least, we have some plan to fix it, hopefully sooner than later.

> 
>> If one file extent is shared 10,000 times for one subvolume, and you
>> have 1000 snapshots of that subvolume, it will really go crazy.
> 
>> But as I mentioned, snapshot is another catalyst for such problem.
> 
> I only have two snapshots of the subvolume but some the extents might
> very well be shared many many times.
> 
>> I can't say for 100% sure. We need more info on that.
> 
> Sure.
> 
>> That's trim (or discard), not hole punching.
> 
> I didn't mean discarding the btrfs to the underlying storage, I meant
> mounting the filesystems in the image files sitting inside the btrfs
> through a loop device and running fstrim on them.
> The loop device should punch holes into the underlying image files
> when it receives a discard, right?

That's correctly, that will punch holes for *unused* space.
But still, all 0 extents are still considered used, thus won't really work.

Since deduperemover has already skipped all 0 extents, it should be a
big problem I guess?

Thanks,
Qu
> 
>> Normally hole punching is done by ioctl fpunch(). Not sure if dupremove
>> does that too.
> 
> Duperemove doesn't punch holes afaik it can only ignore the 0 pages
> and not dedup them.>
>> Extent tree dump can provide per-subvolume level view of how shared one
>> extent is.
> 
>> It's really hard to determine, you could try the following command to
>> determine:
>> # btrfs ins dump-tree -t extent --bfs /dev/nvme/btrfs |\
>>   grep "(.*_ITEM.*)" | awk '{print $4" "$5" "$6" size "$10}'
>>
>> Then which key is the most shown one and its size.
>>
>> If a key's objectid (the first value) shows up multiple times, it's a
>> kinda heavily shared extent.
>>
>> Then search that objectid in the full extent tree dump, to find out how
>> it's shared.
> 
> Thanks, I'll try that out when I can unmount the btrfs.
> 
>> You can see it's already complex...
> 
> That's not an issue, I'm fluent in bash ;)
> 
> - Atemu
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]