Re: BUG: btrfs send: Kernel's memory usage rises until OOM kernel panic after sending ~37GiB

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

[-- Attachment #1.1: Type: text/plain, Size: 1478 bytes --]



On 2019/10/27 下午11:19, Atemu wrote:
>> It's really hard to determine, you could try the following command to
>> determine:
>> # btrfs ins dump-tree -t extent --bfs /dev/nvme/btrfs |\
>>   grep "(.*_ITEM.*)" | awk '{print $4" "$5" "$6" size "$10}'
>>
>> Then which key is the most shown one and its size.
>>
>> If a key's objectid (the first value) shows up multiple times, it's a
>> kinda heavily shared extent.
>>
>> Then search that objectid in the full extent tree dump, to find out how
>> it's shared.
> 
> I analyzed it a bit differently but this should be the information we wanted:
> 
> https://gist.github.com/Atemu/206c44cd46474458c083721e49d84a42
> 
> Yeah...

Holy s***...

Almost every line means 30~1000 refs, and there are over 2000 lines.
No wonder it eats up all memory.

> 
> Is there any way to "unshare" these worst cases without having to
> btrfs defragment everything?

Btrfs defrag should do that, but at the cost of hugely increased space
usage.

BTW, have you verified the content of that extent?
Is that all zero? If so, just find a tool to punch all these files and
you should be OK to go.

Or I can't see any reason why a data extent can be shared so many times.

Thanks,
Qu

> 
> I also uploaded the (compressed) extent tree dump if you want to take
> a look yourself (205MB, expires in 7 days):
> 
> https://send.firefox.com/download/a729c57a94fcd89e/#w51BjzRmGnCg2qKNs39UNw
> 
> -Atemu
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]