Re: BUG: btrfs send: Kernel's memory usage rises until OOM kernel panic after sending ~37GiB

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

[-- Attachment #1.1: Type: text/plain, Size: 2169 bytes --]



On 2019/10/28 下午7:30, Filipe Manana wrote:
> On Sun, Oct 27, 2019 at 4:51 PM Atemu <atemu.main@gmail.com> wrote:
>>
>>> It's really hard to determine, you could try the following command to
>>> determine:
>>> # btrfs ins dump-tree -t extent --bfs /dev/nvme/btrfs |\
>>>   grep "(.*_ITEM.*)" | awk '{print $4" "$5" "$6" size "$10}'
>>>
>>> Then which key is the most shown one and its size.
>>>
>>> If a key's objectid (the first value) shows up multiple times, it's a
>>> kinda heavily shared extent.
>>>
>>> Then search that objectid in the full extent tree dump, to find out how
>>> it's shared.
>>
>> I analyzed it a bit differently but this should be the information we wanted:
>>
>> https://gist.github.com/Atemu/206c44cd46474458c083721e49d84a42
> 
> That's quite a lot of extents shared many times.
> That indeed slows backreference walking and therefore send which uses it.
> While the slowdown is known, the memory consumption I wasn't aware of,
> but from your logs, it's not clear
> where it comes exactly from, something to be looked at. There's also a
> significant number of data checksum errors.
> 
> I think in the meanwhile send can just skip backreference walking and
> attempt to clone whenever the number of
> backreferences for an inode exceeds some limit, in which case it would
> fallback to writes instead of cloning.

Long time ago I had a purpose to record sent extents in an rbtree, then
instead of do the full backref walk, go that rbtree walk instead.
That should still be way faster than full backref walk, and still have a
good enough hit rate.
(And of course, if it fails, falls back to regular write)

Thanks,
Qu

> 
> I'll look into it, thanks for the report (and Qu for telling how to
> get the backreference counts).
> 
>>
>> Yeah...
>>
>> Is there any way to "unshare" these worst cases without having to
>> btrfs defragment everything?
>>
>> I also uploaded the (compressed) extent tree dump if you want to take
>> a look yourself (205MB, expires in 7 days):
>>
>> https://send.firefox.com/download/a729c57a94fcd89e/#w51BjzRmGnCg2qKNs39UNw
>>
>> -Atemu
> 
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]