Btrfs-progs-3.16: fs metadata is both single and dup?

Btrfs-progs-3.16: fs metadata is both single and dup?

[Holger Hoffstätte]

I updated to progs-3.16 and noticed during testing:

root>losetup 
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
/dev/loop0         0      0         0  0 /tmp/img

root>mkfs.btrfs -f /dev/loop0
Btrfs v3.16
See http://btrfs.wiki.kernel.org for more information.

Performing full device TRIM (8.00GiB) ...
Turning ON incompat feature 'extref': increased hardlink limit per file 
to 65536
fs created label (null) on /dev/loop0
	nodesize 16384 leafsize 16384 sectorsize 4096 size 8.00GiB
root>mkdir /tmp/btrfs
root>mount /dev/loop0 /tmp/btrfs 

All fine until here..

root>btrfs filesystem df /tmp/btrfs 
Data, single: total=8.00MiB, used=64.00KiB
System, DUP: total=8.00MiB, used=16.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=409.56MiB, used=112.00KiB
Metadata, single: total=8.00MiB, used=0.00

..wait, what? Let's be clear..

root>btrfs balance start -mconvert=dup /tmp/btrfs 
Done, had to relocate 4 out of 5 chunks

root>btrfs filesystem df /tmp/btrfs            
Data, single: total=8.00MiB, used=64.00KiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=128.00MiB, used=112.00KiB

Now looking as expected.

root>btrfs balance start -mconvert=single -f /tmp/btrfs 
Done, had to relocate 2 out of 3 chunks

root>btrfs filesystem df /tmp/btrfs            
Data, single: total=8.00MiB, used=128.00KiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=256.00MiB, used=112.00KiB

Looking good too (metadata doubled)

So where does the confusing initial display come from? I'm running this 
against a (very patched) 3.14.17, but don't remember ever seeing this 
with btrfs-progs-3.14.2.

Any ideas?

thanks
Holger

Re: Btrfs-progs-3.16: fs metadata is both single and dup?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1828 bytes --]

On Tue, Sep 02, 2014 at 12:05:33PM +0000, Holger Hoffstätte wrote:
> 
> I updated to progs-3.16 and noticed during testing:
> 
> root>losetup 
> NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
> /dev/loop0         0      0         0  0 /tmp/img
> 
> root>mkfs.btrfs -f /dev/loop0
> Btrfs v3.16
> See http://btrfs.wiki.kernel.org for more information.
> 
> Performing full device TRIM (8.00GiB) ...
> Turning ON incompat feature 'extref': increased hardlink limit per file 
> to 65536
> fs created label (null) on /dev/loop0
> 	nodesize 16384 leafsize 16384 sectorsize 4096 size 8.00GiB
> root>mkdir /tmp/btrfs
> root>mount /dev/loop0 /tmp/btrfs 
> 
> All fine until here..
> 
> root>btrfs filesystem df /tmp/btrfs 
> Data, single: total=8.00MiB, used=64.00KiB
> System, DUP: total=8.00MiB, used=16.00KiB
> System, single: total=4.00MiB, used=0.00
> Metadata, DUP: total=409.56MiB, used=112.00KiB
> Metadata, single: total=8.00MiB, used=0.00

   Note that the "single" chunks are empty, and will remain so.

[snip]
> So where does the confusing initial display come from? I'm running this 
> against a (very patched) 3.14.17, but don't remember ever seeing this 
> with btrfs-progs-3.14.2.

   Your memory is faulty, I'm afraid. It's always done that -- at
least since I started using btrfs, several years ago.

   I believe it comes from mkfs creating a trivial basic filesystem
(with the single profiles), and then setting enough flags on it that
the kernel can bootstrap it with the desired chunks in it -- but I may
be wrong about that.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
      --- Normaliser unix c'est comme pasteuriser le Camembert ---       

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: Btrfs-progs-3.16: fs metadata is both single and dup?

[Holger Hoffstätte]

On Tue, 02 Sep 2014 13:13:49 +0100, Hugo Mills wrote:

> [snip]
>> So where does the confusing initial display come from? I'm running this
>> against a (very patched) 3.14.17, but don't remember ever seeing this
>> with btrfs-progs-3.14.2.
> 
>    Your memory is faulty, I'm afraid. It's always done that -- at

After 40 it's all downhill.. :-)

> least since I started using btrfs, several years ago.

Alright then. I was just getting nervous I might have borked something 
either with my kernel or that the new progs somehow get information from 
the kernel that is off/wrong.

All my other filesystems display as expected, but they have been balanced 
previously - so nothing to see there.

Thanks!

Holger

Re: Btrfs-progs-3.16: fs metadata is both single and dup?

[Duncan]

Hugo Mills posted on Tue, 02 Sep 2014 13:13:49 +0100 as excerpted:

> On Tue, Sep 02, 2014 at 12:05:33PM +0000, Holger Hoffstätte wrote:
>> 
>> I updated to progs-3.16 and noticed during testing:

>> root>mkfs.btrfs -f /dev/loop0

>> All fine until here..
>> 
>> root>btrfs filesystem df /tmp/btrfs
>> Data, single: total=8.00MiB, used=64.00KiB
>> System, DUP: total=8.00MiB, used=16.00KiB
>> System, single: total=4.00MiB, used=0.00
>> Metadata, DUP: total=409.56MiB, used=112.00KiB
>> Metadata, single: total=8.00MiB, used=0.00
> 
> Note that the "single" chunks are empty, and will remain so.
> 
>> So where does the confusing initial display come from? [I] don't
>> remember ever seeing this with btrfs-progs-3.14.2.
> 
>    Your memory is faulty, I'm afraid. It's always done that -- at
> least since I started using btrfs, several years ago.
> 
>    I believe it comes from mkfs creating a trivial basic filesystem
> (with the single profiles), and then setting enough flags on it that the
> kernel can bootstrap it with the desired chunks in it -- but I may be
> wrong about that.

Agreed.  It's an artifact of the mkfs.btrfs process and a btrfs fi df on 
a new filesystem always seems to have those extra unused single profile 
lines.

I got so the first thing I'd do on first mount was a balance -- before 
there was anything actually on the filesystem so it was real fast -- to 
get rid of those null entries.

Actually, I had already created a little mkfs.btrfs helper script that 
sets options I normally want, etc, and after doing the mkfs and balance 
drill a few times, I setup the script such that if at the appropriate 
prompt I give it a mountpoint to point balance at, it'll mount the 
filesystem and immediately run a balance, thus automating things and 
making the balance part of the same scripted process that does the 
mkfs.btrfs in the first place.

IOW, those null-entry lines bother me too... enough that even tho I know 
what they are I arranged things so they're automatically and immediately 
eliminated and I don't have to see 'em! =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Btrfs-progs-3.16: fs metadata is both single and dup?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2419 bytes --]

On Wed, Sep 03, 2014 at 04:53:39AM +0000, Duncan wrote:
> Hugo Mills posted on Tue, 02 Sep 2014 13:13:49 +0100 as excerpted:
> 
> > On Tue, Sep 02, 2014 at 12:05:33PM +0000, Holger Hoffstätte wrote:
> >> So where does the confusing initial display come from? [I] don't
> >> remember ever seeing this with btrfs-progs-3.14.2.
> > 
> >    Your memory is faulty, I'm afraid. It's always done that -- at
> > least since I started using btrfs, several years ago.
> > 
> >    I believe it comes from mkfs creating a trivial basic filesystem
> > (with the single profiles), and then setting enough flags on it that the
> > kernel can bootstrap it with the desired chunks in it -- but I may be
> > wrong about that.
> 
> Agreed.  It's an artifact of the mkfs.btrfs process and a btrfs fi df on 
> a new filesystem always seems to have those extra unused single profile 
> lines.
> 
> I got so the first thing I'd do on first mount was a balance -- before 
> there was anything actually on the filesystem so it was real fast -- to 
> get rid of those null entries.

   Interesting. Last time I tried that (balance without any contents),
the balance removed *all* the chunks, and then the FS forgot about
what configuration it should have and reverted to RAID-1/single. I
usually recommend writing at least one 4k+ file to the FS first, if
it's bothering someone so much that they can't let it go.

   Hugo.

> Actually, I had already created a little mkfs.btrfs helper script that 
> sets options I normally want, etc, and after doing the mkfs and balance 
> drill a few times, I setup the script such that if at the appropriate 
> prompt I give it a mountpoint to point balance at, it'll mount the 
> filesystem and immediately run a balance, thus automating things and 
> making the balance part of the same scripted process that does the 
> mkfs.btrfs in the first place.
> 
> IOW, those null-entry lines bother me too... enough that even tho I know 
> what they are I arranged things so they're automatically and immediately 
> eliminated and I don't have to see 'em! =:^)
> 

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
       --- Never underestimate the bandwidth of a Volvo filled ---       
                           with backup tapes.                            

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: Btrfs-progs-3.16: fs metadata is both single and dup?

[Duncan]

Hugo Mills posted on Wed, 03 Sep 2014 08:33:05 +0100 as excerpted:

> On Wed, Sep 03, 2014 at 04:53:39AM +0000, Duncan wrote:
>> Hugo Mills posted on Tue, 02 Sep 2014 13:13:49 +0100 as excerpted:
>> 
>> 
>> [A] btrfs fi df on a new filesystem always seems to have those extra
>> unused single profile lines.
>> 
>> I got so the first thing I'd do on first mount was a balance -- before
>> there was anything actually on the filesystem so it was real fast -- to
>> get rid of those null entries.
> 
>    Interesting. Last time I tried that (balance without any contents),
> the balance removed *all* the chunks, and then the FS forgot about what
> configuration it should have and reverted to RAID-1/single. I usually
> recommend writing at least one 4k+ file to the FS first, if it's
> bothering someone so much that they can't let it go.

Interesting indeed.  From memory, even before I've put anything on the 
filesystem it always seems to have a bit of the first chunk of both data 
and metadata used -- not much but enough that it's obvious in the df 
which mode chunks are the null-chunks, and apparently obvious to the 
balance as well, as it has always left me with at least a first chunk of 
each.

I wonder what the difference might be.  Perhaps it's just the versions of 
kernel and/or userspace I've happened to do all my mkfs.btrfs with?  Or 
maybe it's one of the features (like thin-metadata or noholes) I enable 
by default, or the fact that I use labels for partition ID and tracking, 
so I always fill that in.  Whatever it is, it seems to put a bit of 
something in the filesystem, possibly at first mount, so the actually 
used chunks, one each of data and metadata, aren't entirely empty.

Or maybe I'm remembering wrong and I've just been lucky. <shrug>

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman