From: Atemu <atemu.main@gmail.com>
To: Qu Wenruo <quwenruo.btrfs@gmx.com>
Cc: linux-btrfs@vger.kernel.org
Subject: Re: BUG: btrfs send: Kernel's memory usage rises until OOM kernel panic after sending ~37GiB
Date: Sun, 27 Oct 2019 13:55:28 +0100 [thread overview]
Message-ID: <CAE4GHg=-K3JdhvQpTC8fPGBm1sVLDOUW+UkBCSZJwz27fkW90A@mail.gmail.com> (raw)
In-Reply-To: <b4673e3b-b9b2-e8e5-2783-4b5eac7f656d@gmx.com>
> This depends on how shared one file extent is.
But shouldn't it catch that and cancel the btrfs send before it panics
the kernel due to its memory usage?
> If one file extent is shared 10,000 times for one subvolume, and you
> have 1000 snapshots of that subvolume, it will really go crazy.
> But as I mentioned, snapshot is another catalyst for such problem.
I only have two snapshots of the subvolume but some the extents might
very well be shared many many times.
> I can't say for 100% sure. We need more info on that.
Sure.
> That's trim (or discard), not hole punching.
I didn't mean discarding the btrfs to the underlying storage, I meant
mounting the filesystems in the image files sitting inside the btrfs
through a loop device and running fstrim on them.
The loop device should punch holes into the underlying image files
when it receives a discard, right?
> Normally hole punching is done by ioctl fpunch(). Not sure if dupremove
> does that too.
Duperemove doesn't punch holes afaik it can only ignore the 0 pages
and not dedup them.
> Extent tree dump can provide per-subvolume level view of how shared one
> extent is.
> It's really hard to determine, you could try the following command to
> determine:
> # btrfs ins dump-tree -t extent --bfs /dev/nvme/btrfs |\
> grep "(.*_ITEM.*)" | awk '{print $4" "$5" "$6" size "$10}'
>
> Then which key is the most shown one and its size.
>
> If a key's objectid (the first value) shows up multiple times, it's a
> kinda heavily shared extent.
>
> Then search that objectid in the full extent tree dump, to find out how
> it's shared.
Thanks, I'll try that out when I can unmount the btrfs.
> You can see it's already complex...
That's not an issue, I'm fluent in bash ;)
- Atemu
next prev parent reply other threads:[~2019-10-27 12:55 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-26 17:46 BUG: btrfs send: Kernel's memory usage rises until OOM kernel panic after sending ~37GiB Atemu
2019-10-27 0:50 ` Qu Wenruo
2019-10-27 10:33 ` Atemu
2019-10-27 11:34 ` Qu Wenruo
2019-10-27 12:55 ` Atemu [this message]
2019-10-27 13:43 ` Qu Wenruo
2019-10-27 15:19 ` Atemu
2019-10-27 15:19 ` Atemu
2019-10-27 23:16 ` Qu Wenruo
2019-10-28 12:26 ` Atemu
2019-10-28 11:30 ` Filipe Manana
2019-10-28 12:36 ` Qu Wenruo
2019-10-28 12:43 ` Filipe Manana
2019-10-28 14:58 ` Martin Raiber
2019-10-28 12:44 ` Atemu
2019-10-28 13:01 ` Filipe Manana
2019-10-28 13:44 ` Atemu
2019-10-31 13:55 ` Atemu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAE4GHg=-K3JdhvQpTC8fPGBm1sVLDOUW+UkBCSZJwz27fkW90A@mail.gmail.com' \
--to=atemu.main@gmail.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=quwenruo.btrfs@gmx.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).