Re: Btrfs-progs release 3.17

Re: Btrfs-progs release 3.17

[Petr Janecek]

Hello,

> the version 3.17 of btrfs-progs has been released.

  on a system with 3-disk raid1 and 4 and 5-disk raid10 fs,
"btrfs filesystem show" now stalls for approx. half a minute after the
listing, just before the version information. During that time, it
often prints something like

[...]
        devid    4 size 2.73TiB used 806.01GiB path /dev/sdh
        devid    5 size 3.64TiB used 123.00GiB path /dev/sdp

parent transid verify failed on 2636556500992 wanted 2264 found 2268
parent transid verify failed on 2636556500992 wanted 2264 found 2268
parent transid verify failed on 2636556500992 wanted 2264 found 2268
parent transid verify failed on 2636556500992 wanted 2264 found 2268
Ignoring transid failure
extent buffer leak: start 2636556091392 len 16384
Btrfs v3.17

I've had a balance running that time, but "btrfs fi sh /any/btrfs/mount"
returned always quickly as in previous versions.

  Now, with the balance paused, the stalls continue to appear, but no
messages as above. During the stalls, btrfs is reading the disks at rate
several MB/s, probably seek limited. /proc/<btrfs fi sh pid>/fd shows
that it scans again the devices it has already written about.

  Umounted all the btrfs, checked them -- nothing wrong.

  Resumed the balance -- error messages are (sometimes) back:

[...]
Label: 'BTR3'  uuid: 0bf93650-fd12-4f82-8086-2c7484141e97
        Total devices 5 FS bytes used 1.63TiB
        devid    1 size 2.73TiB used 800.00GiB path /dev/sde
        devid    2 size 2.73TiB used 800.00GiB path /dev/sdf
        devid    3 size 2.73TiB used 800.03GiB path /dev/sdg
        devid    4 size 2.73TiB used 801.00GiB path /dev/sdh
        devid    5 size 3.64TiB used 143.03GiB path /dev/sdp

Check tree block failed, want=2730665181184, have=9727841172931608576
Check tree block failed, want=2730665181184, have=9727841172931608576
Check tree block failed, want=2730665181184, have=8388343671073157408
Check tree block failed, want=2730665181184, have=9727841172931608576
Check tree block failed, want=2730665181184, have=9727841172931608576
read block failed check_tree_block
Label: 'BTR1'  uuid: 2ec4b5a1-a2c4-4687-9015-cf5b519297f3
        Total devices 3 FS bytes used 2.47TiB
        devid    1 size 3.64TiB used 1.79TiB path /dev/sdm
        devid    2 size 3.64TiB used 1.79TiB path /dev/sdn
        devid    3 size 3.64TiB used 1.79TiB path /dev/sdo

Btrfs v3.17

  (or the "parent transid verify failed..." + "extent buffer leak..." as
in the first case as well)

  Kernel version 3.18-rc1


Regards,

Petr

Re: Btrfs-progs release 3.17

[Petr Janecek]

Hello,
      one more thing: I just overwrote part of one disk.
"btrfs filesystem show" could be more helpful diagnosing this:

# btrfs fi sh
Label: 'BTRFSROOT'  uuid: d877125e-9b8d-47ea-b57b-7411292fd26c
        Total devices 1 FS bytes used 2.91GiB
        devid    1 size 29.44GiB used 5.04GiB path /dev/sdi1

Label: 'BTR1'  uuid: 2ec4b5a1-a2c4-4687-9015-cf5b519297f3
        Total devices 3 FS bytes used 2.47TiB
        devid    1 size 3.64TiB used 1.79TiB path /dev/sdn
        devid    2 size 3.64TiB used 1.79TiB path /dev/sdo
        devid    3 size 3.64TiB used 1.79TiB path /dev/sdp

Label: 'BTR0'  uuid: 0ec83db3-4574-4e40-8d57-ebbe9fe246e1
        Total devices 4 FS bytes used 2.41TiB
        devid    1 size 2.73TiB used 1.21TiB path /dev/sda
        devid    2 size 2.73TiB used 1.21TiB path /dev/sdb
        devid    3 size 2.73TiB used 1.21TiB path /dev/sdc
        devid    4 size 2.73TiB used 1.21TiB path /dev/sdd

Label: 'BTR3'  uuid: 89f2016b-984f-4811-99d9-df3b8b5ce318
        Total devices 4 FS bytes used 896.00KiB
        devid    1 size 2.73TiB used 2.02GiB path /dev/sde
        devid    2 size 2.73TiB used 2.00GiB path /dev/sdf
        devid    3 size 2.73TiB used 1.01GiB path /dev/sdg
        devid    4 size 2.73TiB used 1.01GiB path /dev/sdh

warning, device 3 is missing
warning devid 3 not found already
Btrfs v3.17

-- there are three disks with devid 3, can't tell which one it is. The
fs uuid can be displayed with very little effort:

diff --git a/volumes.c b/volumes.c
index 5b007fc..02866bc 100644
--- a/volumes.c
+++ b/volumes.c
@@ -30,6 +30,7 @@
 #include "print-tree.h"
 #include "volumes.h"
 #include "math.h"
+#include "utils.h"
 
 struct stripe {
 	struct btrfs_device *dev;
@@ -1625,9 +1626,11 @@ static int read_one_chunk(struct btrfs_root *root, struct btrfs_key *key,
 		map->stripes[i].dev = btrfs_find_device(root, devid, uuid,
 							NULL);
 		if (!map->stripes[i].dev) {
+			char uuidbuf[BTRFS_UUID_UNPARSED_SIZE];
+			uuid_unparse(root->fs_info->fsid, uuidbuf);
 			map->stripes[i].dev = fill_missing_device(devid);
-			printf("warning, device %llu is missing\n",
-			       (unsigned long long)devid);
+			printf("warning, device %llu from fs %s is missing\n",
+			       (unsigned long long)devid, uuidbuf);
 		}
 
 	}
@@ -1720,8 +1723,10 @@ static int read_one_dev(struct btrfs_root *root,
 
 	device = btrfs_find_device(root, devid, dev_uuid, fs_uuid);
 	if (!device) {
-		printk("warning devid %llu not found already\n",
-			(unsigned long long)devid);
+		char uuidbuf[BTRFS_UUID_UNPARSED_SIZE];
+		uuid_unparse(fs_uuid, uuidbuf);
+		printk("warning devid %llu from fs %s not found already\n",
+			(unsigned long long)devid, uuidbuf);
 		device = kzalloc(sizeof(*device), GFP_NOFS);
 		if (!device)
 			return -ENOMEM;


Regards,

Petr

Re: Btrfs-progs release 3.17

[Anand Jain]

Hi,

  You have mentioned two issues when balance and fi show running
  concurrently
   - stalling and
   - errors.

first of all..
3.17 replaced our own system wide disk scan methods by lblkid scan
methods. lblkid with its feature-rich is slower as reported here.
   https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg37839.html

Next, as of 3.17 it still miss the poor man's optimizations patches
(without seeking for a new kernel interface)

[PATCH 1/1] btrfs-progs: code optimize cmd_scan_dev() use 
btrfs_register_one_device()
[PATCH 1/2] btrfs-progs: introduce btrfs_register_all_device()
[PATCH 2/2] btrfs-progs: optimize btrfs_scan_lblkid() for multiple calls


If you could, pls..
  Now on 3.17 apply above 3 patches and see if you see any better
  performance for the stalling issue.
  can you do same steps on 3.16 and report what you observe


Now about the 2nd issue errors when balance is running.. here is my
initial analysis with some guess work..

 > just before the version information. During that time, it
 > often prints something like

 > parent transid verify failed on 2636556500992 wanted 2264 found 2268
or
 > Check tree block failed, want=2730665181184, have=9727841172931608576

  The default btrfs fi show will scan kernel for mounted FS first and
  then for umounted btrfs disks.

  When we already know about the mounted disks its just unnecessary that
  we would still read the mounted disks and later drop them as seen,
  looks like we need to write a real fix to replace this workaround fix..
    btrfs filesystem show: skip duplicate fsids



-Anand


On 10/22/14 05:28, Petr Janecek wrote:
> Hello,
>
>> the version 3.17 of btrfs-progs has been released.
>
>    on a system with 3-disk raid1 and 4 and 5-disk raid10 fs,
> "btrfs filesystem show" now stalls for approx. half a minute after the
> listing, just before the version information. During that time, it
> often prints something like
>


> [...]
>          devid    4 size 2.73TiB used 806.01GiB path /dev/sdh
>          devid    5 size 3.64TiB used 123.00GiB path /dev/sdp
>
> parent transid verify failed on 2636556500992 wanted 2264 found 2268
> parent transid verify failed on 2636556500992 wanted 2264 found 2268
> parent transid verify failed on 2636556500992 wanted 2264 found 2268
> parent transid verify failed on 2636556500992 wanted 2264 found 2268
> Ignoring transid failure
> extent buffer leak: start 2636556091392 len 16384
> Btrfs v3.17
>
> I've had a balance running that time, but "btrfs fi sh /any/btrfs/mount"
> returned always quickly as in previous versions.
>
>    Now, with the balance paused, the stalls continue to appear, but no
> messages as above. During the stalls, btrfs is reading the disks at rate
> several MB/s, probably seek limited. /proc/<btrfs fi sh pid>/fd shows
> that it scans again the devices it has already written about.

>    Umounted all the btrfs, checked them -- nothing wrong.
>
>    Resumed the balance -- error messages are (sometimes) back:
>
> [...]
> Label: 'BTR3'  uuid: 0bf93650-fd12-4f82-8086-2c7484141e97
>          Total devices 5 FS bytes used 1.63TiB
>          devid    1 size 2.73TiB used 800.00GiB path /dev/sde
>          devid    2 size 2.73TiB used 800.00GiB path /dev/sdf
>          devid    3 size 2.73TiB used 800.03GiB path /dev/sdg
>          devid    4 size 2.73TiB used 801.00GiB path /dev/sdh
>          devid    5 size 3.64TiB used 143.03GiB path /dev/sdp
>
> Check tree block failed, want=2730665181184, have=9727841172931608576
> Check tree block failed, want=2730665181184, have=9727841172931608576
> Check tree block failed, want=2730665181184, have=8388343671073157408
> Check tree block failed, want=2730665181184, have=9727841172931608576
> Check tree block failed, want=2730665181184, have=9727841172931608576
> read block failed check_tree_block
> Label: 'BTR1'  uuid: 2ec4b5a1-a2c4-4687-9015-cf5b519297f3
>          Total devices 3 FS bytes used 2.47TiB
>          devid    1 size 3.64TiB used 1.79TiB path /dev/sdm
>          devid    2 size 3.64TiB used 1.79TiB path /dev/sdn
>          devid    3 size 3.64TiB used 1.79TiB path /dev/sdo
>
> Btrfs v3.17
>
>    (or the "parent transid verify failed..." + "extent buffer leak..." as
> in the first case as well)
>
>    Kernel version 3.18-rc1
>
>
> Regards,
>
> Petr
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Btrfs-progs release 3.17

[Petr Janecek]

Hello,

>  You have mentioned two issues when balance and fi show running
>  concurrently

  my mail was a bit chaotic, but I get the stalls even on idle system.
Today I got

parent transid verify failed on 1559973888000 wanted 1819 found 1821
parent transid verify failed on 1559973888000 wanted 1819 found 1821
parent transid verify failed on 1559973888000 wanted 1819 found 1821
parent transid verify failed on 1559973888000 wanted 1819 found 1821
Ignoring transid failure
leaf parent key incorrect 1559973888000

from 'btrfs fi sh' while I was just copying something, no balance running.

[...]
> [PATCH 1/1] btrfs-progs: code optimize cmd_scan_dev() use
> btrfs_register_one_device()
> [PATCH 1/2] btrfs-progs: introduce btrfs_register_all_device()
> [PATCH 2/2] btrfs-progs: optimize btrfs_scan_lblkid() for multiple calls
> 
> If you could, pls..
>  Now on 3.17 apply above 3 patches and see if you see any better
>  performance for the stalling issue.

  no perceptible change: takes ~40 seconds both before and after
applying. Old version <1 sec.

>  can you do same steps on 3.16 and report what you observe

  So many rejects -- do you have older versions of these patches?


Thanks,

Petr

Re: Btrfs-progs release 3.17

[Anand Jain]

Some of the disks on my system were missing and I was able to hit
this issue.


----------------
Check tree block failed, want=12582912, have=0
read block failed check_tree_block
Couldn't read chunk root
warning devid 2 not found already
Check tree block failed, want=143360, have=0
read block failed check_tree_block
Couldn't read chunk root
warning, device 4 is missing
warning, device 3 is missing
warning, device 2 is missing
warning, device 1 is missing
----------------


Did a bisect and it leads to this following patch.

----------------
commit 915902c5002485fb13d27c4b699a73fb66cc0f09

     btrfs-progs: fix device missing of btrfs fi show with seed devices
----------------

  Also this patch stalls ~2sec in the cmd btrfs fi show, on my system
  with 48 disks.

Also a simple test case hits some warnings...

----------------
  mkfs.btrfs -draid1 -mraid1 /dev/sdb /dev/sdc
  mount /dev/sdb /btrfs && fillfs /btrfs 100 && umount /btrfs
  wipefs -a /dev/sdb
  modprobe -r btrfs && modprobe btrfs
  mount -o degraded /dev/sdb /btrfs
  btrfs fi show
Label: none  uuid: 9844cd05-1c8c-473e-a84b-bac95aab7bc9
         Total devices 2 FS bytes used 1.59MiB
         devid    2 size 967.87MiB used 104.75MiB path /dev/sdc
         *** Some devices missing

warning, device 1 is missing
warning, device 1 is missing
warning devid 1 not found already
----------------





On 10/23/14 14:57, Petr Janecek wrote:
> Hello,
>
>>   You have mentioned two issues when balance and fi show running
>>   concurrently
>
>    my mail was a bit chaotic, but I get the stalls even on idle system.
> Today I got
>
> parent transid verify failed on 1559973888000 wanted 1819 found 1821
> parent transid verify failed on 1559973888000 wanted 1819 found 1821
> parent transid verify failed on 1559973888000 wanted 1819 found 1821
> parent transid verify failed on 1559973888000 wanted 1819 found 1821
> Ignoring transid failure
> leaf parent key incorrect 1559973888000
>
> from 'btrfs fi sh' while I was just copying something, no balance running.
>
> [...]
>> [PATCH 1/1] btrfs-progs: code optimize cmd_scan_dev() use
>> btrfs_register_one_device()
>> [PATCH 1/2] btrfs-progs: introduce btrfs_register_all_device()
>> [PATCH 2/2] btrfs-progs: optimize btrfs_scan_lblkid() for multiple calls
>>
>> If you could, pls..
>>   Now on 3.17 apply above 3 patches and see if you see any better
>>   performance for the stalling issue.
>
>    no perceptible change: takes ~40 seconds both before and after
> applying. Old version <1 sec.
>
>>   can you do same steps on 3.16 and report what you observe
>
>    So many rejects -- do you have older versions of these patches?
>
>
> Thanks,
>
> Petr
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Btrfs-progs release 3.17

[Gui Hecheng]

On Thu, 2014-10-23 at 16:13 +0800, Anand Jain wrote:
> 
> Some of the disks on my system were missing and I was able to hit
> this issue.
> 
> 
> ----------------
> Check tree block failed, want=12582912, have=0
> read block failed check_tree_block
> Couldn't read chunk root
> warning devid 2 not found already
> Check tree block failed, want=143360, have=0
> read block failed check_tree_block
> Couldn't read chunk root
> warning, device 4 is missing
> warning, device 3 is missing
> warning, device 2 is missing
> warning, device 1 is missing
> ----------------
> 
> 
> Did a bisect and it leads to this following patch.
> 
> ----------------
> commit 915902c5002485fb13d27c4b699a73fb66cc0f09
> 
>      btrfs-progs: fix device missing of btrfs fi show with seed devices
> ----------------
> 
>   Also this patch stalls ~2sec in the cmd btrfs fi show, on my system
>   with 48 disks.
> 
> Also a simple test case hits some warnings...
> 
> ----------------
>   mkfs.btrfs -draid1 -mraid1 /dev/sdb /dev/sdc
>   mount /dev/sdb /btrfs && fillfs /btrfs 100 && umount /btrfs
>   wipefs -a /dev/sdb
>   modprobe -r btrfs && modprobe btrfs
>   mount -o degraded /dev/sdb /btrfs
>   btrfs fi show
> Label: none  uuid: 9844cd05-1c8c-473e-a84b-bac95aab7bc9
>          Total devices 2 FS bytes used 1.59MiB
>          devid    2 size 967.87MiB used 104.75MiB path /dev/sdc
>          *** Some devices missing
> 
> warning, device 1 is missing
> warning, device 1 is missing
> warning devid 1 not found already
> ----------------
> 

Hi Anand and Petr,

Oh, it seems that there are btrfs with missing devs that are bringing
troubles to the @open_ctree_... function.
This should be a missing case of the patch above which should only take
effects when seeding devices are present.
I will try my best to follow this case, suggestions are welcome, Thanks!

-Gui

> 
> 
> 
> On 10/23/14 14:57, Petr Janecek wrote:
> > Hello,
> >
> >>   You have mentioned two issues when balance and fi show running
> >>   concurrently
> >
> >    my mail was a bit chaotic, but I get the stalls even on idle system.
> > Today I got
> >
> > parent transid verify failed on 1559973888000 wanted 1819 found 1821
> > parent transid verify failed on 1559973888000 wanted 1819 found 1821
> > parent transid verify failed on 1559973888000 wanted 1819 found 1821
> > parent transid verify failed on 1559973888000 wanted 1819 found 1821
> > Ignoring transid failure
> > leaf parent key incorrect 1559973888000
> >
> > from 'btrfs fi sh' while I was just copying something, no balance running.
> >
> > [...]
> >> [PATCH 1/1] btrfs-progs: code optimize cmd_scan_dev() use
> >> btrfs_register_one_device()
> >> [PATCH 1/2] btrfs-progs: introduce btrfs_register_all_device()
> >> [PATCH 2/2] btrfs-progs: optimize btrfs_scan_lblkid() for multiple calls
> >>
> >> If you could, pls..
> >>   Now on 3.17 apply above 3 patches and see if you see any better
> >>   performance for the stalling issue.
> >
> >    no perceptible change: takes ~40 seconds both before and after
> > applying. Old version <1 sec.
> >
> >>   can you do same steps on 3.16 and report what you observe
> >
> >    So many rejects -- do you have older versions of these patches?
> >
> >
> > Thanks,
> >
> > Petr
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >

Re: Btrfs-progs release 3.17

[Petr Janecek]

Hello Gui,

> Oh, it seems that there are btrfs with missing devs that are bringing
> troubles to the @open_ctree_... function.

  what do you mean by "missing devs"? I have no degraded fs.

  The time "btrfs fi sh" spends scanning disks of a filesystem seems to
be proportional to the amount of data stored on them: on a completely
idle system, of ~20s total time it spends 10s scanning each of /mnt/b
and /mnt/b0, and almost no time on /mnt/b3 (which is the biggest)

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdm        5.5T  2.4T  2.1T  54% /mnt/b
/dev/sda        5.5T  2.5T  3.1T  45% /mnt/b0
/dev/sde        7.3T   90G  5.4T   2% /mnt/b3


Thanks,

Petr

Re: Btrfs-progs release 3.17

[Anand Jain]

  there is no point in re-creating so many btrfs kernel's logic in user
  space. its just unnecessary, when kernel is already doing it. use
  some interface to get info from kernel after device is registered,
  (not necessarily mounted). so progs can be as sleek as possible.
  to me it started as just one more bug now we have fixed so many many.
  It all needs one good interface for kernel which provides anything
  anything from the kernel.



On 10/23/14 16:52, Gui Hecheng wrote:
> On Thu, 2014-10-23 at 16:13 +0800, Anand Jain wrote:
>>
>> Some of the disks on my system were missing and I was able to hit
>> this issue.
>>
>>
>> ----------------
>> Check tree block failed, want=12582912, have=0
>> read block failed check_tree_block
>> Couldn't read chunk root
>> warning devid 2 not found already
>> Check tree block failed, want=143360, have=0
>> read block failed check_tree_block
>> Couldn't read chunk root
>> warning, device 4 is missing
>> warning, device 3 is missing
>> warning, device 2 is missing
>> warning, device 1 is missing
>> ----------------
>>
>>
>> Did a bisect and it leads to this following patch.
>>
>> ----------------
>> commit 915902c5002485fb13d27c4b699a73fb66cc0f09
>>
>>       btrfs-progs: fix device missing of btrfs fi show with seed devices
>> ----------------
>>
>>    Also this patch stalls ~2sec in the cmd btrfs fi show, on my system
>>    with 48 disks.
>>
>> Also a simple test case hits some warnings...
>>
>> ----------------
>>    mkfs.btrfs -draid1 -mraid1 /dev/sdb /dev/sdc
>>    mount /dev/sdb /btrfs && fillfs /btrfs 100 && umount /btrfs
>>    wipefs -a /dev/sdb
>>    modprobe -r btrfs && modprobe btrfs
>>    mount -o degraded /dev/sdb /btrfs
>>    btrfs fi show
>> Label: none  uuid: 9844cd05-1c8c-473e-a84b-bac95aab7bc9
>>           Total devices 2 FS bytes used 1.59MiB
>>           devid    2 size 967.87MiB used 104.75MiB path /dev/sdc
>>           *** Some devices missing
>>
>> warning, device 1 is missing
>> warning, device 1 is missing
>> warning devid 1 not found already
>> ----------------
>>
>
> Hi Anand and Petr,
>
> Oh, it seems that there are btrfs with missing devs that are bringing
> troubles to the @open_ctree_... function.
> This should be a missing case of the patch above which should only take
> effects when seeding devices are present.
> I will try my best to follow this case, suggestions are welcome, Thanks!
>
> -Gui
>
>>
>>
>>
>> On 10/23/14 14:57, Petr Janecek wrote:
>>> Hello,
>>>
>>>>    You have mentioned two issues when balance and fi show running
>>>>    concurrently
>>>
>>>     my mail was a bit chaotic, but I get the stalls even on idle system.
>>> Today I got
>>>
>>> parent transid verify failed on 1559973888000 wanted 1819 found 1821
>>> parent transid verify failed on 1559973888000 wanted 1819 found 1821
>>> parent transid verify failed on 1559973888000 wanted 1819 found 1821
>>> parent transid verify failed on 1559973888000 wanted 1819 found 1821
>>> Ignoring transid failure
>>> leaf parent key incorrect 1559973888000
>>>
>>> from 'btrfs fi sh' while I was just copying something, no balance running.
>>>
>>> [...]
>>>> [PATCH 1/1] btrfs-progs: code optimize cmd_scan_dev() use
>>>> btrfs_register_one_device()
>>>> [PATCH 1/2] btrfs-progs: introduce btrfs_register_all_device()
>>>> [PATCH 2/2] btrfs-progs: optimize btrfs_scan_lblkid() for multiple calls
>>>>
>>>> If you could, pls..
>>>>    Now on 3.17 apply above 3 patches and see if you see any better
>>>>    performance for the stalling issue.
>>>
>>>     no perceptible change: takes ~40 seconds both before and after
>>> applying. Old version <1 sec.
>>>
>>>>    can you do same steps on 3.16 and report what you observe
>>>
>>>     So many rejects -- do you have older versions of these patches?
>>>
>>>
>>> Thanks,
>>>
>>> Petr
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Btrfs-progs release 3.17

[Gui Hecheng]

On Thu, 2014-10-23 at 15:23 +0200, Petr Janecek wrote:
> Hello Gui,
> 
> > Oh, it seems that there are btrfs with missing devs that are bringing
> > troubles to the @open_ctree_... function.
> 
>   what do you mean by "missing devs"? I have no degraded fs.

Ah, sorry, I'm too focused on the problem that Anand's script pointed
out. Ignore this "missing devs".

>   The time "btrfs fi sh" spends scanning disks of a filesystem seems to
> be proportional to the amount of data stored on them: on a completely
> idle system, of ~20s total time it spends 10s scanning each of /mnt/b
> and /mnt/b0, and almost no time on /mnt/b3 (which is the biggest)
> 
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/sdm        5.5T  2.4T  2.1T  54% /mnt/b
> /dev/sda        5.5T  2.5T  3.1T  45% /mnt/b0
> /dev/sde        7.3T   90G  5.4T   2% /mnt/b3

For your original problems:
o error messages:
  The concurrency problem exists as Anand said. As you said, running
balance & cp lead to such messages, so I think there are some
unintentional redundency works over the mounted devices when dealing
with umounted ones. I'll try to 

o stalling:
  This may be due to concurrency problem either. After the first problem
handled, let's see what happens.

Thanks,
Gui
> 
> Thanks,
> 
> Petr
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Btrfs-progs release 3.17

[Gui Hecheng]

On Thu, 2014-10-23 at 21:36 +0800, Anand Jain wrote:
> 
>   there is no point in re-creating so many btrfs kernel's logic in user
>   space. its just unnecessary, when kernel is already doing it. use
>   some interface to get info from kernel after device is registered,
>   (not necessarily mounted). so progs can be as sleek as possible.
>   to me it started as just one more bug now we have fixed so many many.
>   It all needs one good interface for kernel which provides anything
>   anything from the kernel.
> 

Oh, the interface for kernel you described is really interesting.
But how to store the seed/sprout relationships so that we can fetch them
correctly for umounted btrfs?

-Gui 

> 
> On 10/23/14 16:52, Gui Hecheng wrote:
> > On Thu, 2014-10-23 at 16:13 +0800, Anand Jain wrote:
> >>
> >> Some of the disks on my system were missing and I was able to hit
> >> this issue.
> >>
> >>
> >> ----------------
> >> Check tree block failed, want=12582912, have=0
> >> read block failed check_tree_block
> >> Couldn't read chunk root
> >> warning devid 2 not found already
> >> Check tree block failed, want=143360, have=0
> >> read block failed check_tree_block
> >> Couldn't read chunk root
> >> warning, device 4 is missing
> >> warning, device 3 is missing
> >> warning, device 2 is missing
> >> warning, device 1 is missing
> >> ----------------
> >>
> >>
> >> Did a bisect and it leads to this following patch.
> >>
> >> ----------------
> >> commit 915902c5002485fb13d27c4b699a73fb66cc0f09
> >>
> >>       btrfs-progs: fix device missing of btrfs fi show with seed devices
> >> ----------------
> >>
> >>    Also this patch stalls ~2sec in the cmd btrfs fi show, on my system
> >>    with 48 disks.
> >>
> >> Also a simple test case hits some warnings...
> >>
> >> ----------------
> >>    mkfs.btrfs -draid1 -mraid1 /dev/sdb /dev/sdc
> >>    mount /dev/sdb /btrfs && fillfs /btrfs 100 && umount /btrfs
> >>    wipefs -a /dev/sdb
> >>    modprobe -r btrfs && modprobe btrfs
> >>    mount -o degraded /dev/sdb /btrfs
> >>    btrfs fi show
> >> Label: none  uuid: 9844cd05-1c8c-473e-a84b-bac95aab7bc9
> >>           Total devices 2 FS bytes used 1.59MiB
> >>           devid    2 size 967.87MiB used 104.75MiB path /dev/sdc
> >>           *** Some devices missing
> >>
> >> warning, device 1 is missing
> >> warning, device 1 is missing
> >> warning devid 1 not found already
> >> ----------------
> >>
> >
> > Hi Anand and Petr,
> >
> > Oh, it seems that there are btrfs with missing devs that are bringing
> > troubles to the @open_ctree_... function.
> > This should be a missing case of the patch above which should only take
> > effects when seeding devices are present.
> > I will try my best to follow this case, suggestions are welcome, Thanks!
> >
> > -Gui
> >
> >>
> >>
> >>
> >> On 10/23/14 14:57, Petr Janecek wrote:
> >>> Hello,
> >>>
> >>>>    You have mentioned two issues when balance and fi show running
> >>>>    concurrently
> >>>
> >>>     my mail was a bit chaotic, but I get the stalls even on idle system.
> >>> Today I got
> >>>
> >>> parent transid verify failed on 1559973888000 wanted 1819 found 1821
> >>> parent transid verify failed on 1559973888000 wanted 1819 found 1821
> >>> parent transid verify failed on 1559973888000 wanted 1819 found 1821
> >>> parent transid verify failed on 1559973888000 wanted 1819 found 1821
> >>> Ignoring transid failure
> >>> leaf parent key incorrect 1559973888000
> >>>
> >>> from 'btrfs fi sh' while I was just copying something, no balance running.
> >>>
> >>> [...]
> >>>> [PATCH 1/1] btrfs-progs: code optimize cmd_scan_dev() use
> >>>> btrfs_register_one_device()
> >>>> [PATCH 1/2] btrfs-progs: introduce btrfs_register_all_device()
> >>>> [PATCH 2/2] btrfs-progs: optimize btrfs_scan_lblkid() for multiple calls
> >>>>
> >>>> If you could, pls..
> >>>>    Now on 3.17 apply above 3 patches and see if you see any better
> >>>>    performance for the stalling issue.
> >>>
> >>>     no perceptible change: takes ~40 seconds both before and after
> >>> applying. Old version <1 sec.
> >>>
> >>>>    can you do same steps on 3.16 and report what you observe
> >>>
> >>>     So many rejects -- do you have older versions of these patches?
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> Petr
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> >>> the body of a message to majordomo@vger.kernel.org
> >>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >

Re: Btrfs-progs release 3.17

[Anand Jain]

On 28/10/2014 12:03, Gui Hecheng wrote:
> On Thu, 2014-10-23 at 21:36 +0800, Anand Jain wrote:
>>
>>    there is no point in re-creating so many btrfs kernel's logic in user
>>    space. its just unnecessary, when kernel is already doing it. use
>>    some interface to get info from kernel after device is registered,
>>    (not necessarily mounted). so progs can be as sleek as possible.
>>    to me it started as just one more bug now we have fixed so many many.
>>    It all needs one good interface for kernel which provides anything
>>    anything from the kernel.
>>
>
> Oh, the interface for kernel you described is really interesting.
> But how to store the seed/sprout relationships so that we can fetch them
> correctly for umounted btrfs?

  remember -  btrfs-control ioctl READY does not work yet when seed is
  present. Some how we need to fix that in kernel right. ? for which
  we need the seed/sprout relation when devices are unmounted. you may
  get that working/accepted in the kernel first, a simple user space
  interface (such as nacked /proc/fs/btrfs/devlist interface or
  discussing sysfs interface (bit risky though)) is all that is needed
  to get this working.



> -Gui
>
>>
>> On 10/23/14 16:52, Gui Hecheng wrote:
>>> On Thu, 2014-10-23 at 16:13 +0800, Anand Jain wrote:
>>>>
>>>> Some of the disks on my system were missing and I was able to hit
>>>> this issue.
>>>>
>>>>
>>>> ----------------
>>>> Check tree block failed, want=12582912, have=0
>>>> read block failed check_tree_block
>>>> Couldn't read chunk root
>>>> warning devid 2 not found already
>>>> Check tree block failed, want=143360, have=0
>>>> read block failed check_tree_block
>>>> Couldn't read chunk root
>>>> warning, device 4 is missing
>>>> warning, device 3 is missing
>>>> warning, device 2 is missing
>>>> warning, device 1 is missing
>>>> ----------------
>>>>
>>>>
>>>> Did a bisect and it leads to this following patch.
>>>>
>>>> ----------------
>>>> commit 915902c5002485fb13d27c4b699a73fb66cc0f09
>>>>
>>>>        btrfs-progs: fix device missing of btrfs fi show with seed devices
>>>> ----------------
>>>>
>>>>     Also this patch stalls ~2sec in the cmd btrfs fi show, on my system
>>>>     with 48 disks.
>>>>
>>>> Also a simple test case hits some warnings...
>>>>
>>>> ----------------
>>>>     mkfs.btrfs -draid1 -mraid1 /dev/sdb /dev/sdc
>>>>     mount /dev/sdb /btrfs && fillfs /btrfs 100 && umount /btrfs
>>>>     wipefs -a /dev/sdb
>>>>     modprobe -r btrfs && modprobe btrfs
>>>>     mount -o degraded /dev/sdb /btrfs
>>>>     btrfs fi show
>>>> Label: none  uuid: 9844cd05-1c8c-473e-a84b-bac95aab7bc9
>>>>            Total devices 2 FS bytes used 1.59MiB
>>>>            devid    2 size 967.87MiB used 104.75MiB path /dev/sdc
>>>>            *** Some devices missing
>>>>
>>>> warning, device 1 is missing
>>>> warning, device 1 is missing
>>>> warning devid 1 not found already
>>>> ----------------
>>>>
>>>
>>> Hi Anand and Petr,
>>>
>>> Oh, it seems that there are btrfs with missing devs that are bringing
>>> troubles to the @open_ctree_... function.
>>> This should be a missing case of the patch above which should only take
>>> effects when seeding devices are present.
>>> I will try my best to follow this case, suggestions are welcome, Thanks!
>>>
>>> -Gui
>>>
>>>>
>>>>
>>>>
>>>> On 10/23/14 14:57, Petr Janecek wrote:
>>>>> Hello,
>>>>>
>>>>>>     You have mentioned two issues when balance and fi show running
>>>>>>     concurrently
>>>>>
>>>>>      my mail was a bit chaotic, but I get the stalls even on idle system.
>>>>> Today I got
>>>>>
>>>>> parent transid verify failed on 1559973888000 wanted 1819 found 1821
>>>>> parent transid verify failed on 1559973888000 wanted 1819 found 1821
>>>>> parent transid verify failed on 1559973888000 wanted 1819 found 1821
>>>>> parent transid verify failed on 1559973888000 wanted 1819 found 1821
>>>>> Ignoring transid failure
>>>>> leaf parent key incorrect 1559973888000
>>>>>
>>>>> from 'btrfs fi sh' while I was just copying something, no balance running.
>>>>>
>>>>> [...]
>>>>>> [PATCH 1/1] btrfs-progs: code optimize cmd_scan_dev() use
>>>>>> btrfs_register_one_device()
>>>>>> [PATCH 1/2] btrfs-progs: introduce btrfs_register_all_device()
>>>>>> [PATCH 2/2] btrfs-progs: optimize btrfs_scan_lblkid() for multiple calls
>>>>>>
>>>>>> If you could, pls..
>>>>>>     Now on 3.17 apply above 3 patches and see if you see any better
>>>>>>     performance for the stalling issue.
>>>>>
>>>>>      no perceptible change: takes ~40 seconds both before and after
>>>>> applying. Old version <1 sec.
>>>>>
>>>>>>     can you do same steps on 3.16 and report what you observe
>>>>>
>>>>>      So many rejects -- do you have older versions of these patches?
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Petr
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Btrfs-progs release 3.17

[Gui Hecheng]

On Tue, 2014-10-28 at 16:42 +0800, Anand Jain wrote:
> 
> 
> On 28/10/2014 12:03, Gui Hecheng wrote:
> > On Thu, 2014-10-23 at 21:36 +0800, Anand Jain wrote:
> >>
> >>    there is no point in re-creating so many btrfs kernel's logic in user
> >>    space. its just unnecessary, when kernel is already doing it. use
> >>    some interface to get info from kernel after device is registered,
> >>    (not necessarily mounted). so progs can be as sleek as possible.
> >>    to me it started as just one more bug now we have fixed so many many.
> >>    It all needs one good interface for kernel which provides anything
> >>    anything from the kernel.
> >>
> >
> > Oh, the interface for kernel you described is really interesting.
> > But how to store the seed/sprout relationships so that we can fetch them
> > correctly for umounted btrfs?
> 
>   remember -  btrfs-control ioctl READY does not work yet when seed is
>   present. Some how we need to fix that in kernel right. ? for which
>   we need the seed/sprout relation when devices are unmounted. you may
>   get that working/accepted in the kernel first, a simple user space
>   interface (such as nacked /proc/fs/btrfs/devlist interface or
>   discussing sysfs interface (bit risky though)) is all that is needed
>   to get this working.
> 

   As far as I could understand, so we have to make the ioctl READY to
do the seed/sprout detecting relation detecting first.
   And then, all the user space tool has to do is fetching stuff through
a new user interface.

   I think this is really a neat way. So is it on your schedule to make
the ioctl READY 'ready'? If not, I am always ready to serve.

   Since there may be more discussions and comments on the above work,
for now, I just plan to make the current way in 3.17 not so 'slow' and
'noisy'.

Thanks,
Gui

> 
> > -Gui
> >
> >>
> >> On 10/23/14 16:52, Gui Hecheng wrote:
> >>> On Thu, 2014-10-23 at 16:13 +0800, Anand Jain wrote:
> >>>>
> >>>> Some of the disks on my system were missing and I was able to hit
> >>>> this issue.
> >>>>
> >>>>
> >>>> ----------------
> >>>> Check tree block failed, want=12582912, have=0
> >>>> read block failed check_tree_block
> >>>> Couldn't read chunk root
> >>>> warning devid 2 not found already
> >>>> Check tree block failed, want=143360, have=0
> >>>> read block failed check_tree_block
> >>>> Couldn't read chunk root
> >>>> warning, device 4 is missing
> >>>> warning, device 3 is missing
> >>>> warning, device 2 is missing
> >>>> warning, device 1 is missing
> >>>> ----------------
> >>>>
> >>>>
> >>>> Did a bisect and it leads to this following patch.
> >>>>
> >>>> ----------------
> >>>> commit 915902c5002485fb13d27c4b699a73fb66cc0f09
> >>>>
> >>>>        btrfs-progs: fix device missing of btrfs fi show with seed devices
> >>>> ----------------
> >>>>
> >>>>     Also this patch stalls ~2sec in the cmd btrfs fi show, on my system
> >>>>     with 48 disks.
> >>>>
> >>>> Also a simple test case hits some warnings...
> >>>>
> >>>> ----------------
> >>>>     mkfs.btrfs -draid1 -mraid1 /dev/sdb /dev/sdc
> >>>>     mount /dev/sdb /btrfs && fillfs /btrfs 100 && umount /btrfs
> >>>>     wipefs -a /dev/sdb
> >>>>     modprobe -r btrfs && modprobe btrfs
> >>>>     mount -o degraded /dev/sdb /btrfs
> >>>>     btrfs fi show
> >>>> Label: none  uuid: 9844cd05-1c8c-473e-a84b-bac95aab7bc9
> >>>>            Total devices 2 FS bytes used 1.59MiB
> >>>>            devid    2 size 967.87MiB used 104.75MiB path /dev/sdc
> >>>>            *** Some devices missing
> >>>>
> >>>> warning, device 1 is missing
> >>>> warning, device 1 is missing
> >>>> warning devid 1 not found already
> >>>> ----------------
> >>>>
> >>>
> >>> Hi Anand and Petr,
> >>>
> >>> Oh, it seems that there are btrfs with missing devs that are bringing
> >>> troubles to the @open_ctree_... function.
> >>> This should be a missing case of the patch above which should only take
> >>> effects when seeding devices are present.
> >>> I will try my best to follow this case, suggestions are welcome, Thanks!
> >>>
> >>> -Gui
> >>>
> >>>>
> >>>>
> >>>>
> >>>> On 10/23/14 14:57, Petr Janecek wrote:
> >>>>> Hello,
> >>>>>
> >>>>>>     You have mentioned two issues when balance and fi show running
> >>>>>>     concurrently
> >>>>>
> >>>>>      my mail was a bit chaotic, but I get the stalls even on idle system.
> >>>>> Today I got
> >>>>>
> >>>>> parent transid verify failed on 1559973888000 wanted 1819 found 1821
> >>>>> parent transid verify failed on 1559973888000 wanted 1819 found 1821
> >>>>> parent transid verify failed on 1559973888000 wanted 1819 found 1821
> >>>>> parent transid verify failed on 1559973888000 wanted 1819 found 1821
> >>>>> Ignoring transid failure
> >>>>> leaf parent key incorrect 1559973888000
> >>>>>
> >>>>> from 'btrfs fi sh' while I was just copying something, no balance running.
> >>>>>
> >>>>> [...]
> >>>>>> [PATCH 1/1] btrfs-progs: code optimize cmd_scan_dev() use
> >>>>>> btrfs_register_one_device()
> >>>>>> [PATCH 1/2] btrfs-progs: introduce btrfs_register_all_device()
> >>>>>> [PATCH 2/2] btrfs-progs: optimize btrfs_scan_lblkid() for multiple calls
> >>>>>>
> >>>>>> If you could, pls..
> >>>>>>     Now on 3.17 apply above 3 patches and see if you see any better
> >>>>>>     performance for the stalling issue.
> >>>>>
> >>>>>      no perceptible change: takes ~40 seconds both before and after
> >>>>> applying. Old version <1 sec.
> >>>>>
> >>>>>>     can you do same steps on 3.16 and report what you observe
> >>>>>
> >>>>>      So many rejects -- do you have older versions of these patches?
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Petr
> >>>>> --
> >>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> >>>>> the body of a message to majordomo@vger.kernel.org
> >>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>>>
> >>>
> >>>
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> >>> the body of a message to majordomo@vger.kernel.org
> >>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >

Btrfs-progs release 3.17

[David Sterba]

Hi,

the version 3.17 of btrfs-progs has been released.

git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git v3.17
https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/btrfs-progs-v3.17.tar.xz

Among other fixes and updates, there are many fsck improvements, most notably a
fix for the bug introduced in 3.17 regarding inconsistencies after read-only
snapshots (fix https://patchwork.kernel.org/patch/5086521/).

User visible changes:
 * check: --init-csum-tree acutally does something useful, rebuilds the whole
   csum tree
 * /dev scanning for btrfs devices is gone
 * /proc/partitions scanning is gone, blkid is used exclusively
 * new subcommand 'subvolume sync'
 * filesystem df: new options to set unit format
 * convert: allow to copy label from the origin, or specify a new one