From: Paul Jones <paul@pauljones.id.au>
To: Marc MERLIN <marc@merlins.org>, Qu Wenruo <quwenruo.btrfs@gmx.com>
Cc: Su Yue <suy.fnst@cn.fujitsu.com>,
"linux-btrfs@vger.kernel.org" <linux-btrfs@vger.kernel.org>
Subject: RE: how to best segment a big block device in resizeable btrfs filesystems?
Date: Tue, 3 Jul 2018 09:55:13 +0000 [thread overview]
Message-ID: <MEXPR01MB1208F79F27CD83BBAB2AE3D29E420@MEXPR01MB1208.ausprd01.prod.outlook.com> (raw)
In-Reply-To: <20180703041550.GH5567@merlins.org>
> -----Original Message-----
> From: linux-btrfs-owner@vger.kernel.org <linux-btrfs-
> owner@vger.kernel.org> On Behalf Of Marc MERLIN
> Sent: Tuesday, 3 July 2018 2:16 PM
> To: Qu Wenruo <quwenruo.btrfs@gmx.com>
> Cc: Su Yue <suy.fnst@cn.fujitsu.com>; linux-btrfs@vger.kernel.org
> Subject: Re: how to best segment a big block device in resizeable btrfs
> filesystems?
>
> On Tue, Jul 03, 2018 at 09:37:47AM +0800, Qu Wenruo wrote:
> > > If I do this, I would have
> > > software raid 5 < dmcrypt < bcache < lvm < btrfs That's a lot of
> > > layers, and that's also starting to make me nervous :)
> >
> > If you could keep the number of snapshots to minimal (less than 10)
> > for each btrfs (and the number of send source is less than 5), one big
> > btrfs may work in that case.
>
> Well, we kind of discussed this already. If btrfs falls over if you reach
> 100 snapshots or so, and it sure seems to in my case, I won't be much better
> off.
> Having btrfs check --repair fail because 32GB of RAM is not enough, and it's
> unable to use swap, is a big deal in my case. You also confirmed that btrfs
> check lowmem does not scale to filesystems like mine, so this translates into
> "if regular btrfs check repair can't fit in 32GB, I am completely out of luck if
> anything happens to the filesystem"
Just out of curiosity I had a look at my backup filesystem.
vm-server /media/backup # btrfs fi us /media/backup/
Overall:
Device size: 5.46TiB
Device allocated: 3.42TiB
Device unallocated: 2.04TiB
Device missing: 0.00B
Used: 1.80TiB
Free (estimated): 1.83TiB (min: 1.83TiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Data,RAID1: Size:1.69TiB, Used:906.26GiB
/dev/mapper/a-backup--a 1.69TiB
/dev/mapper/b-backup--b 1.69TiB
Metadata,RAID1: Size:19.00GiB, Used:16.90GiB
/dev/mapper/a-backup--a 19.00GiB
/dev/mapper/b-backup--b 19.00GiB
System,RAID1: Size:64.00MiB, Used:336.00KiB
/dev/mapper/a-backup--a 64.00MiB
/dev/mapper/b-backup--b 64.00MiB
Unallocated:
/dev/mapper/a-backup--a 1.02TiB
/dev/mapper/b-backup--b 1.02TiB
compress=zstd,space_cache=v2
202 snapshots, heavily de-duplicated
551G / 361,000 files in latest snapshot
Btrfs check normal mode took 12 mins and 11.5G ram
Lowmem mode I stopped after 4 hours, max memory usage was around 3.9G
next prev parent reply other threads:[~2018-07-03 9:55 UTC|newest]
Thread overview: 65+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-06-29 4:27 So, does btrfs check lowmem take days? weeks? Marc MERLIN
2018-06-29 5:07 ` Qu Wenruo
2018-06-29 5:28 ` Marc MERLIN
2018-06-29 5:48 ` Qu Wenruo
2018-06-29 6:06 ` Marc MERLIN
2018-06-29 6:29 ` Qu Wenruo
2018-06-29 6:59 ` Marc MERLIN
2018-06-29 7:09 ` Roman Mamedov
2018-06-29 7:22 ` Marc MERLIN
2018-06-29 7:34 ` Roman Mamedov
2018-06-29 8:04 ` Lionel Bouton
2018-06-29 16:24 ` btrfs send/receive vs rsync Marc MERLIN
2018-06-30 8:18 ` Duncan
2018-06-29 7:20 ` So, does btrfs check lowmem take days? weeks? Qu Wenruo
2018-06-29 7:28 ` Marc MERLIN
2018-06-29 17:10 ` Marc MERLIN
2018-06-30 0:04 ` Chris Murphy
2018-06-30 2:44 ` Marc MERLIN
2018-06-30 14:49 ` Qu Wenruo
2018-06-30 21:06 ` Marc MERLIN
2018-06-29 6:02 ` Su Yue
2018-06-29 6:10 ` Marc MERLIN
2018-06-29 6:32 ` Su Yue
2018-06-29 6:43 ` Marc MERLIN
2018-07-01 23:22 ` Marc MERLIN
2018-07-02 2:02 ` Su Yue
2018-07-02 3:22 ` Marc MERLIN
2018-07-02 6:22 ` Su Yue
2018-07-02 14:05 ` Marc MERLIN
2018-07-02 14:42 ` Qu Wenruo
2018-07-02 15:18 ` how to best segment a big block device in resizeable btrfs filesystems? Marc MERLIN
2018-07-02 16:59 ` Austin S. Hemmelgarn
2018-07-02 17:34 ` Marc MERLIN
2018-07-02 18:35 ` Austin S. Hemmelgarn
2018-07-02 19:40 ` Marc MERLIN
2018-07-03 4:25 ` Andrei Borzenkov
2018-07-03 7:15 ` Duncan
2018-07-06 4:28 ` Andrei Borzenkov
2018-07-08 8:05 ` Duncan
2018-07-03 0:51 ` Paul Jones
2018-07-03 4:06 ` Marc MERLIN
2018-07-03 4:26 ` Paul Jones
2018-07-03 5:42 ` Marc MERLIN
2018-07-03 1:37 ` Qu Wenruo
2018-07-03 4:15 ` Marc MERLIN
2018-07-03 9:55 ` Paul Jones [this message]
2018-07-03 11:29 ` Qu Wenruo
2018-07-03 4:23 ` Andrei Borzenkov
2018-07-02 15:19 ` So, does btrfs check lowmem take days? weeks? Marc MERLIN
2018-07-02 17:08 ` Austin S. Hemmelgarn
2018-07-02 17:33 ` Roman Mamedov
2018-07-02 17:39 ` Marc MERLIN
2018-07-03 0:31 ` Chris Murphy
2018-07-03 4:22 ` Marc MERLIN
2018-07-03 8:34 ` Su Yue
2018-07-03 21:34 ` Chris Murphy
2018-07-03 21:40 ` Marc MERLIN
2018-07-04 1:37 ` Su Yue
2018-07-03 8:50 ` Qu Wenruo
2018-07-03 14:38 ` Marc MERLIN
2018-07-03 21:46 ` Chris Murphy
2018-07-03 22:00 ` Marc MERLIN
2018-07-03 22:52 ` Qu Wenruo
2018-06-29 5:35 ` Su Yue
2018-06-29 5:46 ` Marc MERLIN
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=MEXPR01MB1208F79F27CD83BBAB2AE3D29E420@MEXPR01MB1208.ausprd01.prod.outlook.com \
--to=paul@pauljones.id.au \
--cc=linux-btrfs@vger.kernel.org \
--cc=marc@merlins.org \
--cc=quwenruo.btrfs@gmx.com \
--cc=suy.fnst@cn.fujitsu.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.