Re: BUG: btrfs send: Kernel's memory usage rises until OOM kernel panic after sending ~37GiB

[Atemu <atemu.main@gmail.com>]

> This depends on how shared one file extent is.

But shouldn't it catch that and cancel the btrfs send before it panics
the kernel due to its memory usage?

> If one file extent is shared 10,000 times for one subvolume, and you
> have 1000 snapshots of that subvolume, it will really go crazy.

> But as I mentioned, snapshot is another catalyst for such problem.

I only have two snapshots of the subvolume but some the extents might
very well be shared many many times.

> I can't say for 100% sure. We need more info on that.

Sure.

> That's trim (or discard), not hole punching.

I didn't mean discarding the btrfs to the underlying storage, I meant
mounting the filesystems in the image files sitting inside the btrfs
through a loop device and running fstrim on them.
The loop device should punch holes into the underlying image files
when it receives a discard, right?

> Normally hole punching is done by ioctl fpunch(). Not sure if dupremove
> does that too.

Duperemove doesn't punch holes afaik it can only ignore the 0 pages
and not dedup them.

> Extent tree dump can provide per-subvolume level view of how shared one
> extent is.

> It's really hard to determine, you could try the following command to
> determine:
> # btrfs ins dump-tree -t extent --bfs /dev/nvme/btrfs |\
>   grep "(.*_ITEM.*)" | awk '{print $4" "$5" "$6" size "$10}'
>
> Then which key is the most shown one and its size.
>
> If a key's objectid (the first value) shows up multiple times, it's a
> kinda heavily shared extent.
>
> Then search that objectid in the full extent tree dump, to find out how
> it's shared.

Thanks, I'll try that out when I can unmount the btrfs.

> You can see it's already complex...

That's not an issue, I'm fluent in bash ;)

- Atemu