All of lore.kernel.org
 help / color / mirror / Atom feed
From: Atemu <atemu.main@gmail.com>
To: Qu Wenruo <quwenruo.btrfs@gmx.com>
Cc: linux-btrfs@vger.kernel.org
Subject: Re: BUG: btrfs send: Kernel's memory usage rises until OOM kernel panic after sending ~37GiB
Date: Sun, 27 Oct 2019 11:33:40 +0100	[thread overview]
Message-ID: <CAE4GHgmW2A-2SUUw8FzgafRhQ2BoViBx2DsLigwBrrbbp=oOsw@mail.gmail.com> (raw)
In-Reply-To: <cb5f9048-919f-0ff9-0765-d5a33e58afa7@gmx.com>

> That's the problem.
>
> Deduped files caused heavy overload for backref walk.
> And send has to do backref walk, and you see the problem...

Interesting!
But should it really be able to make btrfs send use up >15GiB of RAM
and cause a kernel panic because of that? The btrfs doesn't even have
that much metadata on-disk in total.

> I'm very interested how heavily deduped the file is.

So am I, how could I get my hands on that information?

Are that particular file's extents what causes btrfs send's memory
usage to spiral out of control?

> If it's just all 0 pages, hole punching is more effective than dedupe,
> and causes 0 backref overhead.

I did punch holes into the disk images I have stored on it by mounting
and fstrim'ing them and the duperemove command I used has a flag that
ignores all 0 pages (those get compressed down to next to nothing
anyways) but it's likely that I ran duperememove once or twice before
I knew about that flag.

Is there a way to find such extents that could cause the backref walk
to overload?

Thanks,
Atemu

  reply	other threads:[~2019-10-27 10:33 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-26 17:46 BUG: btrfs send: Kernel's memory usage rises until OOM kernel panic after sending ~37GiB Atemu
2019-10-27  0:50 ` Qu Wenruo
2019-10-27 10:33   ` Atemu [this message]
2019-10-27 11:34     ` Qu Wenruo
2019-10-27 12:55       ` Atemu
2019-10-27 13:43         ` Qu Wenruo
2019-10-27 15:19           ` Atemu
2019-10-27 15:19       ` Atemu
2019-10-27 23:16         ` Qu Wenruo
2019-10-28 12:26           ` Atemu
2019-10-28 11:30         ` Filipe Manana
2019-10-28 12:36           ` Qu Wenruo
2019-10-28 12:43             ` Filipe Manana
2019-10-28 14:58               ` Martin Raiber
2019-10-28 12:44           ` Atemu
2019-10-28 13:01             ` Filipe Manana
2019-10-28 13:44               ` Atemu
2019-10-31 13:55                 ` Atemu

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAE4GHgmW2A-2SUUw8FzgafRhQ2BoViBx2DsLigwBrrbbp=oOsw@mail.gmail.com' \
    --to=atemu.main@gmail.com \
    --cc=linux-btrfs@vger.kernel.org \
    --cc=quwenruo.btrfs@gmx.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.