Creating backup snapshots (8 per filesystem) causes No space left on device?

Creating backup snapshots (8 per filesystem) causes No space left on device?

[Marc MERLIN]

Howdy,

I have a little script that creates hourly/daily/weekly snapshots on a device that
otherwise has plenty of disk space free:
gandalfthegreat:~# df -h | grep cryptroot
/dev/mapper/cryptroot     232G  144G   85G  63% /
/dev/mapper/cryptroot     232G  144G   85G  63% /usr
/dev/mapper/cryptroot     232G  144G   85G  63% /var
/dev/mapper/cryptroot     232G  144G   85G  63% /home
/dev/mapper/cryptroot     232G  144G   85G  63% /tmp
/dev/mapper/cryptroot     232G  144G   85G  63% /mnt/btrfs_pool1

I have kernel 3.3.1.

The FAQ of course talks about the topic:
https://btrfs.wiki.kernel.org/articles/f/a/q/FAQ_1fe9.html

but I can't get the filesystem show command to output anything useful:
gandalfthegreat:~# btrfs filesystem show /dev/mapper/cryptroot 
Btrfs Btrfs v0.19
gandalfthegreat:~# 

and the btrfs df ssems to show that I'm ok:
gandalfthegreat:~# btrfs filesystem df /home
Data: total=169.01GB, used=134.70GB
System, DUP: total=8.00MB, used=28.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=5.88GB, used=4.39GB
Metadata: total=8.00MB, used=0.00
gandalfthegreat:~# 

I read about rebalance but it's a mostly new fliesystem will little churn, and I'm 
not anywhere close to full filesystem yet.

So far, when this happened, I've had to delete a set of older snapshots.
This would make sense if I was close to full, but at 63% I'm nowhere that.

Any idea what's going on and how I can debug further and more specifically
what I should capture next time I get a no free space error in userspace?

Thanks,
Marc


gandalfthegreat:/mnt/btrfs_pool1# l
total 4
dr-xr-xr-x 1 root root 2210 Apr 15 08:00 ./
drwxr-xr-x 1 root root  112 Feb 12 17:38 ../
drwxr-xr-x 1 root root   12 Feb 12 17:57 home/
drwxr-xr-x 1 root root   12 Feb 12 17:57 home_daily_20120412_00:01:01/
drwxr-xr-x 1 root root   12 Feb 12 17:57 home_daily_20120413_00:01:02/
drwxr-xr-x 1 root root   12 Feb 12 17:57 home_daily_20120414_00:01:01/
drwxr-xr-x 1 root root   12 Feb 12 17:57 home_daily_20120415_00:01:01/
drwxr-xr-x 1 root root   12 Feb 12 17:57 home_hourly_20120415_06:00:01/
drwxr-xr-x 1 root root   12 Feb 12 17:57 home_hourly_20120415_07:00:01/
drwxr-xr-x 1 root root   12 Feb 12 17:57 home_hourly_20120415_08:00:01/
drwxr-xr-x 1 root root   12 Feb 12 17:57 home_weekly_20120415_00:02:01/
drwxr-xr-x 1 root root  436 Apr  3 07:26 root/
drwxr-xr-x 1 root root  436 Apr  3 07:26 root_daily_20120412_00:01:01/
drwxr-xr-x 1 root root  436 Apr  3 07:26 root_daily_20120414_00:01:01/
drwxr-xr-x 1 root root  436 Apr  3 07:26 root_daily_20120415_00:01:01/
drwxr-xr-x 1 root root  436 Apr  3 07:26 root_hourly_20120415_06:00:01/
drwxr-xr-x 1 root root  436 Apr  3 07:26 root_hourly_20120415_07:00:01/
drwxr-xr-x 1 root root  436 Apr  3 07:26 root_hourly_20120415_08:00:01/
drwxr-xr-x 1 root root  436 Apr  3 07:26 root_weekly_20120415_00:02:01/
drwxrwxrwt 1 root root 7476 Apr 15 08:05 tmp/
drwxrwxrwt 1 root root 7156 Apr 12 00:01 tmp_daily_20120412_00:01:01/
drwxrwxrwt 1 root root 7130 Apr 13 00:01 tmp_daily_20120413_00:01:02/
drwxrwxrwt 1 root root 7236 Apr 14 00:01 tmp_daily_20120414_00:01:01/
drwxrwxrwt 1 root root 7368 Apr 15 00:01 tmp_daily_20120415_00:01:01/
drwxrwxrwt 1 root root 7368 Apr 15 06:00 tmp_hourly_20120415_06:00:01/
drwxrwxrwt 1 root root 7368 Apr 15 07:00 tmp_hourly_20120415_07:00:01/
drwxrwxrwt 1 root root 7476 Apr 15 08:00 tmp_hourly_20120415_08:00:01/
drwxrwxrwt 1 root root 7368 Apr 15 00:02 tmp_weekly_20120415_00:02:01/
drwxr-xr-x 1 root root  206 Mar 31 11:07 usr/
drwxr-xr-x 1 root root  206 Mar 31 11:07 usr_daily_20120412_00:01:01/
drwxr-xr-x 1 root root  206 Mar 31 11:07 usr_daily_20120413_00:01:02/
drwxr-xr-x 1 root root  206 Mar 31 11:07 usr_daily_20120414_00:01:01/
drwxr-xr-x 1 root root  206 Mar 31 11:07 usr_daily_20120415_00:01:01/
drwxr-xr-x 1 root root  206 Mar 31 11:07 usr_hourly_20120415_06:00:01/
drwxr-xr-x 1 root root  206 Mar 31 11:07 usr_hourly_20120415_07:00:01/
drwxr-xr-x 1 root root  206 Mar 31 11:07 usr_hourly_20120415_08:00:01/
drwxr-xr-x 1 root root  206 Mar 31 11:07 usr_weekly_20120415_00:02:01/
drwxr-xr-x 1 root root  130 Feb 12 23:52 var/
drwxr-xr-x 1 root root  130 Feb 12 23:52 var_daily_20120412_00:01:01/
drwxr-xr-x 1 root root  130 Feb 12 23:52 var_daily_20120413_00:01:02/
drwxr-xr-x 1 root root  130 Feb 12 23:52 var_daily_20120414_00:01:01/
drwxr-xr-x 1 root root  130 Feb 12 23:52 var_daily_20120415_00:01:01/
drwxr-xr-x 1 root root  130 Feb 12 23:52 var_hourly_20120415_06:00:01/
drwxr-xr-x 1 root root  130 Feb 12 23:52 var_hourly_20120415_07:00:01/
drwxr-xr-x 1 root root  130 Feb 12 23:52 var_hourly_20120415_08:00:01/
drwxr-xr-x 1 root root  130 Feb 12 23:52 var_weekly_20120415_00:02:01/

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Creating backup snapshots (8 per filesystem) causes No space left on device?

[cwillu]

> but I can't get the filesystem show command to output anything useful:
> gandalfthegreat:~# btrfs filesystem show /dev/mapper/cryptroot
> Btrfs Btrfs v0.19

You need to run that as root.

> gandalfthegreat:~#
>
> and the btrfs df ssems to show that I'm ok:
> gandalfthegreat:~# btrfs filesystem df /home
> Data: total=169.01GB, used=134.70GB
> System, DUP: total=8.00MB, used=28.00KB
> System: total=4.00MB, used=0.00
> Metadata, DUP: total=5.88GB, used=4.39GB
> Metadata: total=8.00MB, used=0.00
> gandalfthegreat:~#
>
> I read about rebalance but it's a mostly new fliesystem will little churn, and I'm
> not anywhere close to full filesystem yet.

The recommendation is basically a workaround for when the accounting
gets screwed up.  Balancing wouldn't help in the case where the
filesystem is actually full.

Re: Creating backup snapshots (8 per filesystem) causes No space left on device?

[Marc MERLIN]

(replying on list)

On Sun, Apr 15, 2012 at 05:52:05PM +0200, Bart Noordervliet wrote:
> Hi Marc,
> 
> there's a known regression causing early "Out of space"-errors in
> kernel 3.3. A patch for stable has been queued I think, but it's not
> in 3.3.1 yet. So your best bet would be to either downgrade to 3.2 or
> use a 3.4-rc kernel. Otherwise you'd have to apply the patch in
> question yourself. It's been discussed on this list very recently.

I'll watch for 3.3.x updates (I see nothing in 3.3.2 yet), thanks.

Or is it just a matter of reverting this patch?
https://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=5500cdbe14d7435e04f66ff3cfb8ecd8b8e44ebf
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index dc083f5..079e5a1 100644 (file)
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -4108,7 +4108,7 @@ static u64 calc_global_metadata_size(struct btrfs_fs_info *fs_info)
        num_bytes += div64_u64(data_used + meta_used, 50);
 
        if (num_bytes * 3 > meta_used)
-               num_bytes = div64_u64(meta_used, 3);
+               num_bytes = div64_u64(meta_used, 3) * 2;
 
        return ALIGN(num_bytes, fs_info->extent_root->leafsize << 10);
 }


On Sun, Apr 15, 2012 at 10:19:30AM -0600, cwillu wrote:
> > but I can't get the filesystem show command to output anything useful:
> > gandalfthegreat:~# btrfs filesystem show /dev/mapper/cryptroot
> > Btrfs Btrfs v0.19
> 
> You need to run that as root.
 
That was run as root :)  '#'

Thanks for the replies,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: Creating backup snapshots (8 per filesystem) causes No space left on device?

[Marc MERLIN]

On Sun, Apr 15, 2012 at 09:27:27AM -0700, Marc MERLIN wrote:
> I'll watch for 3.3.x updates (I see nothing in 3.3.2 yet), thanks.
> 
> Or is it just a matter of reverting this patch?
> https://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=5500cdbe14d7435e04f66ff3cfb8ecd8b8e44ebf
> diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
> index dc083f5..079e5a1 100644 (file)
> --- a/fs/btrfs/extent-tree.c
> +++ b/fs/btrfs/extent-tree.c
> @@ -4108,7 +4108,7 @@ static u64 calc_global_metadata_size(struct btrfs_fs_info *fs_info)
>         num_bytes += div64_u64(data_used + meta_used, 50);
>  
>         if (num_bytes * 3 > meta_used)
> -               num_bytes = div64_u64(meta_used, 3);
> +               num_bytes = div64_u64(meta_used, 3) * 2;
>  
>         return ALIGN(num_bytes, fs_info->extent_root->leafsize << 10);
>  }

After I knew what to look for, I searched the archives some more and
they only seemed to point to this patch.
I have reverted it, but I'm still seeing the same problem on my laptop.

It sounds like I'll have to downgrade back to 3.2.x unless there is
some other patch to revert that I missed.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/