Linux-BTRFS Archive on lore.kernel.org
 help / Atom feed
* Cannot 'mount -o degraded /dev/replacement' after a replace
@ 2019-02-09 10:36 Jakob Schöttl
  2019-02-09 12:16 ` Qu Wenruo
                   ` (2 more replies)
  0 siblings, 3 replies; 8+ messages in thread
From: Jakob Schöttl @ 2019-02-09 10:36 UTC (permalink / raw)
  To: linux-btrfs

Hi,

I've setup a RAID1 with two disks (disk1 and disk2) and I'm testing the 
btrfs replace command.

After replacing disk2 with disk3, I can only mount
(a) disk1 or disk3 (if both disk are plugged) and
(b) the original disk1 (degraded, if disk3 is unplugged).

I cannot mount the replacement disk3 if disk1 is unplugged.

 > mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3, 
missing codepage or helper program, or other error.

What I expect is that both disk1 and disk3 are fully valid and working 
after a replace.

Steps to reproduce:

   dd if=/dev/zero of=/vdisk1 bs=1024 count=300000
   losetup /dev/loop1 /vdisk1
   dd if=/dev/zero of=/vdisk2 bs=1024 count=300000
   losetup /dev/loop2 /vdisk2
   dd if=/dev/zero of=/vdisk3 bs=1024 count=300000
   # losetup /dev/loop3 /vdisk3    # don't plug this device yet

Create RAID1 file system:

   mkfs.btrfs -L datavol -m raid1 -d raid1 /dev/loop1 /dev/loop2

Unplug device 2 to simulate a defect:

   losetup -d /dev/loop2

Plug device 3:

   losetup /dev/loop3 /vdisk3

Replace device 2 with device 3:

   mount -o degraded /dev/loop1 /mnt
   btrfs filesystem show   # to get devid of device 2
   btrfs replace start -Br 2 /dev/loop3 /mnt
   btrfs replace status /mnt   # check success
   umount /mnt

Unplug the original device 1 to see if device 3 has really replaced 
device 2:

   losetup -d /dev/loop1
   mount -o degraded /dev/loop3 /mnt

The mount fails with this error:

 > mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3, 
missing codepage or helper program, or other error.

In this situation, btrfs device scan does not change anything and btrfs
filesystem show shows:

 > warning, device 1 is missing
 > warning, device 1 is missing
 > warning, device 1 is missing
 > warning, device 1 is missing
 > bad tree block 198180864, bytenr mismatch, want=198180864, have=0
 > ERROR: cannot read chunk root
 > Label: 'datavol'  uuid: 640e45d3-e741-4a78-a24e-2d8a41c6b8c3
 >     Total devices 2 FS bytes used 128.00KiB
 >     devid    2 size 292.97MiB used 104.00MiB path /dev/loop3
 >     *** Some devices missing

Is this a known problem? Can you reproduce it? Am I doing something wrong?

Regards, Jakob


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Cannot 'mount -o degraded /dev/replacement' after a replace
  2019-02-09 10:36 Cannot 'mount -o degraded /dev/replacement' after a replace Jakob Schöttl
@ 2019-02-09 12:16 ` Qu Wenruo
  2019-02-09 13:45   ` Jakob Schöttl
  2019-02-09 15:36 ` Andrei Borzenkov
  2019-02-10 18:44 ` Chris Murphy
  2 siblings, 1 reply; 8+ messages in thread
From: Qu Wenruo @ 2019-02-09 12:16 UTC (permalink / raw)
  To: Jakob Schöttl, linux-btrfs

[-- Attachment #1.1: Type: text/plain, Size: 2817 bytes --]



On 2019/2/9 下午6:36, Jakob Schöttl wrote:
> Hi,
> 
> I've setup a RAID1 with two disks (disk1 and disk2) and I'm testing the
> btrfs replace command.
> 
> After replacing disk2 with disk3, I can only mount
> (a) disk1 or disk3 (if both disk are plugged) and
> (b) the original disk1 (degraded, if disk3 is unplugged).
> 
> I cannot mount the replacement disk3 if disk1 is unplugged.

Sounds like there is one single chunk on disk1, which caused the problem.

> 
>> mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3,
> missing codepage or helper program, or other error.

dmesg please.

And btrfs-progs version please.

Maybe mkfs is too old to leave SINGLE profile chunks on the original fs.

And you could verify the chunk mapping by executing 'btrfs ins dump-tree
-t chunk <device>' and paste the output.

Thanks,
Qu

> 
> What I expect is that both disk1 and disk3 are fully valid and working
> after a replace.
> 
> Steps to reproduce:
> 
>   dd if=/dev/zero of=/vdisk1 bs=1024 count=300000
>   losetup /dev/loop1 /vdisk1
>   dd if=/dev/zero of=/vdisk2 bs=1024 count=300000
>   losetup /dev/loop2 /vdisk2
>   dd if=/dev/zero of=/vdisk3 bs=1024 count=300000
>   # losetup /dev/loop3 /vdisk3    # don't plug this device yet
> 
> Create RAID1 file system:
> 
>   mkfs.btrfs -L datavol -m raid1 -d raid1 /dev/loop1 /dev/loop2
> 
> Unplug device 2 to simulate a defect:
> 
>   losetup -d /dev/loop2
> 
> Plug device 3:
> 
>   losetup /dev/loop3 /vdisk3
> 
> Replace device 2 with device 3:
> 
>   mount -o degraded /dev/loop1 /mnt
>   btrfs filesystem show   # to get devid of device 2
>   btrfs replace start -Br 2 /dev/loop3 /mnt
>   btrfs replace status /mnt   # check success
>   umount /mnt
> 
> Unplug the original device 1 to see if device 3 has really replaced
> device 2:
> 
>   losetup -d /dev/loop1
>   mount -o degraded /dev/loop3 /mnt
> 
> The mount fails with this error:
> 
>> mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3,
> missing codepage or helper program, or other error.
> 
> In this situation, btrfs device scan does not change anything and btrfs
> filesystem show shows:
> 
>> warning, device 1 is missing
>> warning, device 1 is missing
>> warning, device 1 is missing
>> warning, device 1 is missing
>> bad tree block 198180864, bytenr mismatch, want=198180864, have=0
>> ERROR: cannot read chunk root
>> Label: 'datavol'  uuid: 640e45d3-e741-4a78-a24e-2d8a41c6b8c3
>>     Total devices 2 FS bytes used 128.00KiB
>>     devid    2 size 292.97MiB used 104.00MiB path /dev/loop3
>>     *** Some devices missing
> 
> Is this a known problem? Can you reproduce it? Am I doing something wrong?
> 
> Regards, Jakob
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Cannot 'mount -o degraded /dev/replacement' after a replace
  2019-02-09 12:16 ` Qu Wenruo
@ 2019-02-09 13:45   ` Jakob Schöttl
  2019-02-09 15:32     ` Andrei Borzenkov
  0 siblings, 1 reply; 8+ messages in thread
From: Jakob Schöttl @ 2019-02-09 13:45 UTC (permalink / raw)
  To: Qu Wenruo, linux-btrfs

Thanks Qu,

Am 09.02.19 um 13:16 schrieb Qu Wenruo:
> On 2019/2/9 下午6:36, Jakob Schöttl wrote:
>> Hi,
>>
>> I've setup a RAID1 with two disks (disk1 and disk2) and I'm testing the
>> btrfs replace command.
>>
>> After replacing disk2 with disk3, I can only mount
>> (a) disk1 or disk3 (if both disk are plugged) and
>> (b) the original disk1 (degraded, if disk3 is unplugged).
>>
>> I cannot mount the replacement disk3 if disk1 is unplugged.
> Sounds like there is one single chunk on disk1, which caused the problem.
>
>>> mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3,
>> missing codepage or helper program, or other error.
> dmesg please.
[60456.856883] BTRFS: device label datavol devid 1 transid 5 /dev/loop1
[60456.940785] BTRFS: device label datavol devid 2 transid 5 /dev/loop2
[60525.211389] BTRFS info (device loop1): allowing degraded mounts
[60525.211395] BTRFS info (device loop1): disk space caching is enabled
[60525.211398] BTRFS info (device loop1): has skinny extents
[60525.211401] BTRFS info (device loop1): flagging fs with big metadata 
feature
[60525.213854] BTRFS warning (device loop1): devid 2 uuid 
0b4e0b31-e2b1-40a0-8360-09978f58a2e4 is missing
[60525.214639] BTRFS info (device loop1): checking UUID tree
[60525.386695] BTRFS info (device loop1): dev_replace from <missing 
disk> (devid 2) to /dev/loop3 started
[60525.394403] BTRFS info (device loop1): dev_replace from <missing 
disk> (devid 2) to /dev/loop3 finished
[60533.721841] BTRFS info (device loop3): allowing degraded mounts
[60533.721846] BTRFS info (device loop3): disk space caching is enabled
[60533.721850] BTRFS info (device loop3): has skinny extents
[60533.723703] BTRFS error (device loop3): failed to read chunk root
[60533.773553] BTRFS error (device loop3): open_ctree failed

> And btrfs-progs version please.
$ pacman -Q btrfs-progs
btrfs-progs 4.20.1-2
$ uname -a
Linux jathink 4.20.7-arch1-1-ARCH #1 SMP PREEMPT Wed Feb 6 18:42:40 UTC 
2019 x86_64 GNU/Linux

> Maybe mkfs is too old to leave SINGLE profile chunks on the original fs.
>
> And you could verify the chunk mapping by executing 'btrfs ins dump-tree
> -t chunk <device>' and paste the output.

When only /dev/loop3 is plugged:

# btrfs inspect-internal dump-tree -t chunk /dev/loop3
btrfs-progs v4.20.1
warning, device 1 is missing
warning, device 1 is missing
warning, device 1 is missing
warning, device 1 is missing
bad tree block 198180864, bytenr mismatch, want=198180864, have=0
ERROR: cannot read chunk root
ERROR: unable to open /dev/loop3

When only /dev/loop1 is plugged:

# btrfs inspect-internal dump-tree -t chunk /dev/loop1
btrfs-progs v4.20.1
warning, device 2 is missing
chunk tree
leaf 198180864 items 10 free space 15005 generation 8 owner CHUNK_TREE
leaf 198180864 flags 0x1(WRITTEN) backref revision 1
fs uuid 005a8d59-a561-4371-869e-b0ccc4a4862b
chunk uuid b3e609f1-a7fe-4add-bc51-6231f0bbf320
     item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
         devid 1 total_bytes 307200000 bytes_used 306053120
         io_align 4096 io_width 4096 sector_size 4096 type 0
         generation 0 start_offset 0 dev_group 0
         seek_speed 0 bandwidth 0
         uuid 043443c7-ac91-4085-a5e4-983b59dd0803
         fsid 005a8d59-a561-4371-869e-b0ccc4a4862b
     item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
         devid 2 total_bytes 307200000 bytes_used 109051904
         io_align 4096 io_width 4096 sector_size 4096 type 0
         generation 0 start_offset 0 dev_group 0
         seek_speed 0 bandwidth 0
         uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
         fsid 005a8d59-a561-4371-869e-b0ccc4a4862b
     item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975 
itemsize 112
         length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
         io_align 65536 io_width 65536 sector_size 4096
         num_stripes 2 sub_stripes 0
             stripe 0 devid 2 offset 1048576
             dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
             stripe 1 devid 1 offset 22020096
             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
     item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863 
itemsize 112
         length 33554432 owner 2 stripe_len 65536 type METADATA|RAID1
         io_align 65536 io_width 65536 sector_size 4096
         num_stripes 2 sub_stripes 0
             stripe 0 devid 2 offset 9437184
             dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
             stripe 1 devid 1 offset 30408704
             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
     item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 63963136) itemoff 15751 
itemsize 112
         length 67108864 owner 2 stripe_len 65536 type DATA|RAID1
         io_align 65536 io_width 65536 sector_size 4096
         num_stripes 2 sub_stripes 0
             stripe 0 devid 2 offset 42991616
             dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
             stripe 1 devid 1 offset 63963136
             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
     item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 131072000) itemoff 15671 
itemsize 80
         length 33554432 owner 2 stripe_len 65536 type METADATA
         io_align 65536 io_width 65536 sector_size 4096
         num_stripes 1 sub_stripes 1
             stripe 0 devid 1 offset 131072000
             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
     item 6 key (FIRST_CHUNK_TREE CHUNK_ITEM 164626432) itemoff 15591 
itemsize 80
         length 33554432 owner 2 stripe_len 65536 type SYSTEM
         io_align 65536 io_width 65536 sector_size 4096
         num_stripes 1 sub_stripes 1
             stripe 0 devid 1 offset 164626432
             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
     item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 198180864) itemoff 15479 
itemsize 112
         length 33554432 owner 2 stripe_len 65536 type SYSTEM|DUP
         io_align 65536 io_width 65536 sector_size 4096
         num_stripes 2 sub_stripes 1
             stripe 0 devid 1 offset 198180864
             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
             stripe 1 devid 1 offset 231735296
             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
     item 8 key (FIRST_CHUNK_TREE CHUNK_ITEM 231735296) itemoff 15367 
itemsize 112
         length 20905984 owner 2 stripe_len 65536 type METADATA|DUP
         io_align 65536 io_width 65536 sector_size 4096
         num_stripes 2 sub_stripes 1
             stripe 0 devid 1 offset 265289728
             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
             stripe 1 devid 1 offset 286195712
             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
     item 9 key (FIRST_CHUNK_TREE CHUNK_ITEM 252641280) itemoff 15255 
itemsize 112
         length 10485760 owner 2 stripe_len 65536 type DATA|DUP
         io_align 65536 io_width 65536 sector_size 4096
         num_stripes 2 sub_stripes 1
             stripe 0 devid 1 offset 1048576
             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
             stripe 1 devid 1 offset 11534336
             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Cannot 'mount -o degraded /dev/replacement' after a replace
  2019-02-09 13:45   ` Jakob Schöttl
@ 2019-02-09 15:32     ` Andrei Borzenkov
  2019-02-09 15:57       ` Jakob Schöttl
  0 siblings, 1 reply; 8+ messages in thread
From: Andrei Borzenkov @ 2019-02-09 15:32 UTC (permalink / raw)
  To: Jakob Schöttl, Qu Wenruo, linux-btrfs

09.02.2019 16:45, Jakob Schöttl пишет:
> Thanks Qu,
> 
> Am 09.02.19 um 13:16 schrieb Qu Wenruo:
>> On 2019/2/9 下午6:36, Jakob Schöttl wrote:
>>> Hi,
>>>
>>> I've setup a RAID1 with two disks (disk1 and disk2) and I'm testing the
>>> btrfs replace command.
>>>
>>> After replacing disk2 with disk3, I can only mount
>>> (a) disk1 or disk3 (if both disk are plugged) and
>>> (b) the original disk1 (degraded, if disk3 is unplugged).
>>>
>>> I cannot mount the replacement disk3 if disk1 is unplugged.
>> Sounds like there is one single chunk on disk1, which caused the problem.
>>
>>>> mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3,
>>> missing codepage or helper program, or other error.
>> dmesg please.
> [60456.856883] BTRFS: device label datavol devid 1 transid 5 /dev/loop1
> [60456.940785] BTRFS: device label datavol devid 2 transid 5 /dev/loop2
> [60525.211389] BTRFS info (device loop1): allowing degraded mounts
> [60525.211395] BTRFS info (device loop1): disk space caching is enabled
> [60525.211398] BTRFS info (device loop1): has skinny extents
> [60525.211401] BTRFS info (device loop1): flagging fs with big metadata
> feature
> [60525.213854] BTRFS warning (device loop1): devid 2 uuid
> 0b4e0b31-e2b1-40a0-8360-09978f58a2e4 is missing
> [60525.214639] BTRFS info (device loop1): checking UUID tree
> [60525.386695] BTRFS info (device loop1): dev_replace from <missing
> disk> (devid 2) to /dev/loop3 started
> [60525.394403] BTRFS info (device loop1): dev_replace from <missing
> disk> (devid 2) to /dev/loop3 finished
> [60533.721841] BTRFS info (device loop3): allowing degraded mounts
> [60533.721846] BTRFS info (device loop3): disk space caching is enabled
> [60533.721850] BTRFS info (device loop3): has skinny extents
> [60533.723703] BTRFS error (device loop3): failed to read chunk root
> [60533.773553] BTRFS error (device loop3): open_ctree failed
> 
>> And btrfs-progs version please.
> $ pacman -Q btrfs-progs
> btrfs-progs 4.20.1-2
> $ uname -a
> Linux jathink 4.20.7-arch1-1-ARCH #1 SMP PREEMPT Wed Feb 6 18:42:40 UTC
> 2019 x86_64 GNU/Linux
> 
>> Maybe mkfs is too old to leave SINGLE profile chunks on the original fs.
>>
>> And you could verify the chunk mapping by executing 'btrfs ins dump-tree
>> -t chunk <device>' and paste the output.
> 
> When only /dev/loop3 is plugged:
> 
> # btrfs inspect-internal dump-tree -t chunk /dev/loop3
> btrfs-progs v4.20.1
> warning, device 1 is missing
> warning, device 1 is missing
> warning, device 1 is missing
> warning, device 1 is missing
> bad tree block 198180864, bytenr mismatch, want=198180864, have=0
> ERROR: cannot read chunk root
> ERROR: unable to open /dev/loop3
> 
> When only /dev/loop1 is plugged:
> 
> # btrfs inspect-internal dump-tree -t chunk /dev/loop1
> btrfs-progs v4.20.1
> warning, device 2 is missing
> chunk tree
> leaf 198180864 items 10 free space 15005 generation 8 owner CHUNK_TREE
> leaf 198180864 flags 0x1(WRITTEN) backref revision 1
> fs uuid 005a8d59-a561-4371-869e-b0ccc4a4862b
> chunk uuid b3e609f1-a7fe-4add-bc51-6231f0bbf320
>     item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
>         devid 1 total_bytes 307200000 bytes_used 306053120
>         io_align 4096 io_width 4096 sector_size 4096 type 0
>         generation 0 start_offset 0 dev_group 0
>         seek_speed 0 bandwidth 0
>         uuid 043443c7-ac91-4085-a5e4-983b59dd0803
>         fsid 005a8d59-a561-4371-869e-b0ccc4a4862b
>     item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
>         devid 2 total_bytes 307200000 bytes_used 109051904
>         io_align 4096 io_width 4096 sector_size 4096 type 0
>         generation 0 start_offset 0 dev_group 0
>         seek_speed 0 bandwidth 0
>         uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
>         fsid 005a8d59-a561-4371-869e-b0ccc4a4862b
>     item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975
> itemsize 112
>         length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
>         io_align 65536 io_width 65536 sector_size 4096
>         num_stripes 2 sub_stripes 0
>             stripe 0 devid 2 offset 1048576
>             dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
>             stripe 1 devid 1 offset 22020096
>             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
>     item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863
> itemsize 112
>         length 33554432 owner 2 stripe_len 65536 type METADATA|RAID1
>         io_align 65536 io_width 65536 sector_size 4096
>         num_stripes 2 sub_stripes 0
>             stripe 0 devid 2 offset 9437184
>             dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
>             stripe 1 devid 1 offset 30408704
>             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
>     item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 63963136) itemoff 15751
> itemsize 112
>         length 67108864 owner 2 stripe_len 65536 type DATA|RAID1
>         io_align 65536 io_width 65536 sector_size 4096
>         num_stripes 2 sub_stripes 0
>             stripe 0 devid 2 offset 42991616
>             dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
>             stripe 1 devid 1 offset 63963136
>             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
>     item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 131072000) itemoff 15671
> itemsize 80
>         length 33554432 owner 2 stripe_len 65536 type METADATA
>         io_align 65536 io_width 65536 sector_size 4096
>         num_stripes 1 sub_stripes 1
>             stripe 0 devid 1 offset 131072000
>             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
>     item 6 key (FIRST_CHUNK_TREE CHUNK_ITEM 164626432) itemoff 15591
> itemsize 80
>         length 33554432 owner 2 stripe_len 65536 type SYSTEM
>         io_align 65536 io_width 65536 sector_size 4096
>         num_stripes 1 sub_stripes 1
>             stripe 0 devid 1 offset 164626432
>             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
>     item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 198180864) itemoff 15479
> itemsize 112
>         length 33554432 owner 2 stripe_len 65536 type SYSTEM|DUP
>         io_align 65536 io_width 65536 sector_size 4096
>         num_stripes 2 sub_stripes 1
>             stripe 0 devid 1 offset 198180864
>             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
>             stripe 1 devid 1 offset 231735296
>             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
>     item 8 key (FIRST_CHUNK_TREE CHUNK_ITEM 231735296) itemoff 15367
> itemsize 112
>         length 20905984 owner 2 stripe_len 65536 type METADATA|DUP
>         io_align 65536 io_width 65536 sector_size 4096
>         num_stripes 2 sub_stripes 1
>             stripe 0 devid 1 offset 265289728
>             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
>             stripe 1 devid 1 offset 286195712
>             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
>     item 9 key (FIRST_CHUNK_TREE CHUNK_ITEM 252641280) itemoff 15255
> itemsize 112
>         length 10485760 owner 2 stripe_len 65536 type DATA|DUP
>         io_align 65536 io_width 65536 sector_size 4096
>         num_stripes 2 sub_stripes 1
>             stripe 0 devid 1 offset 1048576
>             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
>             stripe 1 devid 1 offset 11534336
>             dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
> 
> 

Yes, I can trivially reproduce it using commands in original post.

Immediately after mkfs:

10:/home/bor # btrfs inspect-internal dump-tree -t chunk /dev/sdb1
btrfs-progs v4.15
warning, device 2 is missing
chunk tree
leaf 22036480 items 5 free space 15626 generation 5 owner 3
leaf 22036480 flags 0x1(WRITTEN) backref revision 1
fs uuid d33e6bff-5903-449d-b49c-f2d992676e30
chunk uuid ed55b2ad-66c4-4764-acd5-7fa3a9fe619e
	item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
		devid 1 total_bytes 5367640064 bytes_used 1081868288
		io_align 4096 io_width 4096 sector_size 4096 type 0
		generation 0 start_offset 0 dev_group 0
		seek_speed 0 bandwidth 0
		uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
		fsid d33e6bff-5903-449d-b49c-f2d992676e30
	item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
		devid 2 total_bytes 5367640064 bytes_used 1081868288
		io_align 4096 io_width 4096 sector_size 4096 type 0
		generation 0 start_offset 0 dev_group 0
		seek_speed 0 bandwidth 0
		uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
		fsid d33e6bff-5903-449d-b49c-f2d992676e30
	item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975
itemsize 112
		length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 1048576
			dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
			stripe 1 devid 1 offset 22020096
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863
itemsize 112
		length 536739840 owner 2 stripe_len 65536 type METADATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 9437184
			dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
			stripe 1 devid 1 offset 30408704
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 567148544) itemoff 15751
itemsize 112
		length 536739840 owner 2 stripe_len 65536 type DATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 546177024
			dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
			stripe 1 devid 1 offset 567148544
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6


Immediately after degraded mount

btrfs-progs v4.15
warning, device 2 is missing
chunk tree
leaf 22020096 items 8 free space 15311 generation 6 owner 3
leaf 22020096 flags 0x1(WRITTEN) backref revision 1
fs uuid e69b93ef-d852-4afa-8c74-50fa07d33a01
chunk uuid 74317bdd-f011-44a8-9c62-0cce665f1b52
	item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
		devid 1 total_bytes 5368709120 bytes_used 2994733056
		io_align 4096 io_width 4096 sector_size 4096 type 0
		generation 0 start_offset 0 dev_group 0
		seek_speed 0 bandwidth 0
		uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
		fsid e69b93ef-d852-4afa-8c74-50fa07d33a01
	item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
		devid 2 total_bytes 5368709120 bytes_used 2155872256
		io_align 4096 io_width 4096 sector_size 4096 type 0
		generation 0 start_offset 0 dev_group 0
		seek_speed 0 bandwidth 0
		uuid 1989d9cd-9f82-4d3a-a577-c7e583157ed3
		fsid e69b93ef-d852-4afa-8c74-50fa07d33a01
	item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975
itemsize 112
		length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 1048576
			dev_uuid 1989d9cd-9f82-4d3a-a577-c7e583157ed3
			stripe 1 devid 1 offset 22020096
			dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
	item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863
itemsize 112
		length 1073741824 owner 2 stripe_len 65536 type METADATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 9437184
			dev_uuid 1989d9cd-9f82-4d3a-a577-c7e583157ed3
			stripe 1 devid 1 offset 30408704
			dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
	item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 1104150528) itemoff 15751
itemsize 112
		length 1073741824 owner 2 stripe_len 65536 type DATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 1083179008
			dev_uuid 1989d9cd-9f82-4d3a-a577-c7e583157ed3
			stripe 1 devid 1 offset 1104150528
			dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
	item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 2177892352) itemoff 15671
itemsize 80
		length 268435456 owner 2 stripe_len 65536 type METADATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 2177892352
			dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
	item 6 key (FIRST_CHUNK_TREE CHUNK_ITEM 2446327808) itemoff 15591
itemsize 80
		length 33554432 owner 2 stripe_len 65536 type SYSTEM
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 2446327808
			dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
	item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 2479882240) itemoff 15511
itemsize 80
		length 536870912 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 2479882240
			dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8


After device replacement completed

10:/home/bor # btrfs replace status /mnt
Started on  9.Feb 18:07:18, finished on  9.Feb 18:07:19, 0 write errs, 0
uncorr. read errs
10:/home/bor # btrfs inspect-internal dump-tree -t chunk /dev/sdb1
btrfs-progs v4.15
chunk tree
leaf 1942749184 items 11 free space 14900 generation 9 owner 3
leaf 1942749184 flags 0x1(WRITTEN) backref revision 1
fs uuid d33e6bff-5903-449d-b49c-f2d992676e30
chunk uuid ed55b2ad-66c4-4764-acd5-7fa3a9fe619e
	item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
		devid 1 total_bytes 5367640064 bytes_used 2793144320
		io_align 4096 io_width 4096 sector_size 4096 type 0
		generation 0 start_offset 0 dev_group 0
		seek_speed 0 bandwidth 0
		uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
		fsid d33e6bff-5903-449d-b49c-f2d992676e30
	item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
		devid 2 total_bytes 5367640064 bytes_used 1081868288
		io_align 4096 io_width 4096 sector_size 4096 type 0
		generation 0 start_offset 0 dev_group 0
		seek_speed 0 bandwidth 0
		uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
		fsid d33e6bff-5903-449d-b49c-f2d992676e30
	item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975
itemsize 112
		length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 1048576
			dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
			stripe 1 devid 1 offset 22020096
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863
itemsize 112
		length 536739840 owner 2 stripe_len 65536 type METADATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 9437184
			dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
			stripe 1 devid 1 offset 30408704
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 567148544) itemoff 15751
itemsize 112
		length 536739840 owner 2 stripe_len 65536 type DATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 546177024
			dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
			stripe 1 devid 1 offset 567148544
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 1103888384) itemoff 15671
itemsize 80
		length 268435456 owner 2 stripe_len 65536 type METADATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 1103888384
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 6 key (FIRST_CHUNK_TREE CHUNK_ITEM 1372323840) itemoff 15591
itemsize 80
		length 33554432 owner 2 stripe_len 65536 type SYSTEM
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 1372323840
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 1405878272) itemoff 15511
itemsize 80
		length 536870912 owner 2 stripe_len 65536 type DATA
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 1 sub_stripes 1
			stripe 0 devid 1 offset 1405878272
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 8 key (FIRST_CHUNK_TREE CHUNK_ITEM 1942749184) itemoff 15399
itemsize 112
		length 33554432 owner 2 stripe_len 65536 type SYSTEM|DUP
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 1 offset 1942749184
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
			stripe 1 devid 1 offset 1976303616
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 9 key (FIRST_CHUNK_TREE CHUNK_ITEM 1976303616) itemoff 15287
itemsize 112
		length 134217728 owner 2 stripe_len 65536 type METADATA|DUP
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 1 offset 2009858048
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
			stripe 1 devid 1 offset 2144075776
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
	item 10 key (FIRST_CHUNK_TREE CHUNK_ITEM 2110521344) itemoff 15175
itemsize 112
		length 268435456 owner 2 stripe_len 65536 type DATA|DUP
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 1
			stripe 0 devid 1 offset 2278293504
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
			stripe 1 devid 1 offset 2546728960
			dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
10:/home/bor #



Running

btrfs balance start -f -ssoft,convert=raid1 -dsoft,convert=raid1
-msoft,convert=raid1 /mnt

fixes it. I am not sure whether btrfs is expected to do it automatically
as of now.


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Cannot 'mount -o degraded /dev/replacement' after a replace
  2019-02-09 10:36 Cannot 'mount -o degraded /dev/replacement' after a replace Jakob Schöttl
  2019-02-09 12:16 ` Qu Wenruo
@ 2019-02-09 15:36 ` Andrei Borzenkov
  2019-02-10 18:44 ` Chris Murphy
  2 siblings, 0 replies; 8+ messages in thread
From: Andrei Borzenkov @ 2019-02-09 15:36 UTC (permalink / raw)
  To: Jakob Schöttl, linux-btrfs

09.02.2019 13:36, Jakob Schöttl пишет:
>   btrfs filesystem show   # to get devid of device 2

Funny because for me it just says "Some devices missing" without giving
exact device id. May be has been fixed in recent versions but for a long
time it really annoyed me.

OK, when I just unplug device it does not say it - but it also does not
say *which* device is missing, it happily continues to show old device
names at the time of mount, be they actually present or not.

I always wondered how users are supposed to know which device to replace.

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Cannot 'mount -o degraded /dev/replacement' after a replace
  2019-02-09 15:32     ` Andrei Borzenkov
@ 2019-02-09 15:57       ` Jakob Schöttl
  0 siblings, 0 replies; 8+ messages in thread
From: Jakob Schöttl @ 2019-02-09 15:57 UTC (permalink / raw)
  To: Andrei Borzenkov, linux-btrfs; +Cc: Qu Wenruo


Am 09.02.19 um 16:32 schrieb Andrei Borzenkov:
> Running
>
> btrfs balance start -f -ssoft,convert=raid1 -dsoft,convert=raid1
> -msoft,convert=raid1 /mnt
>
> fixes it. I am not sure whether btrfs is expected to do it automatically
> as of now.

This fix seems to work for me, too. Output was:
 > Done, had to relocate 5 out of 8 chunks

Tested without any subvolumes or data.


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Cannot 'mount -o degraded /dev/replacement' after a replace
  2019-02-09 10:36 Cannot 'mount -o degraded /dev/replacement' after a replace Jakob Schöttl
  2019-02-09 12:16 ` Qu Wenruo
  2019-02-09 15:36 ` Andrei Borzenkov
@ 2019-02-10 18:44 ` Chris Murphy
  2019-02-11  1:36   ` Qu Wenruo
  2 siblings, 1 reply; 8+ messages in thread
From: Chris Murphy @ 2019-02-10 18:44 UTC (permalink / raw)
  To: Jakob Schöttl; +Cc: Btrfs BTRFS

On Sat, Feb 9, 2019 at 3:36 AM Jakob Schöttl <jschoett@gmail.com> wrote:
>
> Replace device 2 with device 3:
>
>    mount -o degraded /dev/loop1 /mnt
>    btrfs filesystem show   # to get devid of device 2
>    btrfs replace start -Br 2 /dev/loop3 /mnt
>    btrfs replace status /mnt   # check success
>    umount /mnt

The first command causes the creation of metadata and system chunk
using single profile, on loop1, and even after a replace, both devices
know that there is a single and required chunk on loop1. So subsequent
degraded mount fails because the volume was not fully redundant.

The missing step is between 'btrfs replaced start' completing, and
umount. There needed to be at least a 'btrfs fi us' to check if there
are any single chunks and to convert them with 'btrfs balance start
-dconvert=raid1,soft -mconvert=raid1,soft' where soft is only possible
if there were no data writes while degraded; if there were degraded
writes, a scrub is also necessary to make sure any single mirror raid1
chunk writes to the degraded device are replicated to the replacement.

Yep - kinda ugly. And not intuitive. But that's the present status.


-- 
Chris Murphy

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Cannot 'mount -o degraded /dev/replacement' after a replace
  2019-02-10 18:44 ` Chris Murphy
@ 2019-02-11  1:36   ` Qu Wenruo
  0 siblings, 0 replies; 8+ messages in thread
From: Qu Wenruo @ 2019-02-11  1:36 UTC (permalink / raw)
  To: Chris Murphy, Jakob Schöttl; +Cc: Btrfs BTRFS

[-- Attachment #1.1: Type: text/plain, Size: 1561 bytes --]



On 2019/2/11 上午2:44, Chris Murphy wrote:
> On Sat, Feb 9, 2019 at 3:36 AM Jakob Schöttl <jschoett@gmail.com> wrote:
>>
>> Replace device 2 with device 3:
>>
>>    mount -o degraded /dev/loop1 /mnt
>>    btrfs filesystem show   # to get devid of device 2
>>    btrfs replace start -Br 2 /dev/loop3 /mnt
>>    btrfs replace status /mnt   # check success
>>    umount /mnt
> 
> The first command causes the creation of metadata and system chunk
> using single profile, on loop1, and even after a replace, both devices
> know that there is a single and required chunk on loop1. So subsequent
> degraded mount fails because the volume was not fully redundant.
> 
> The missing step is between 'btrfs replaced start' completing, and
> umount. There needed to be at least a 'btrfs fi us' to check if there
> are any single chunks and to convert them with 'btrfs balance start
> -dconvert=raid1,soft -mconvert=raid1,soft' where soft is only possible
> if there were no data writes while degraded; if there were degraded
> writes, a scrub is also necessary to make sure any single mirror raid1
> chunk writes to the degraded device are replicated to the replacement.
> 
> Yep - kinda ugly. And not intuitive. But that's the present status.

Not exactly.

Currently the problem is, there is extra DUP sys/meta/data chunks created.
So despite the SINGLE chunks, we still can't mount it degraded.

IIRC there is some patch trying to address the SINGLE problem, but the
DUP one seems to be something new.

Thanks,
Qu

> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, back to index

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-02-09 10:36 Cannot 'mount -o degraded /dev/replacement' after a replace Jakob Schöttl
2019-02-09 12:16 ` Qu Wenruo
2019-02-09 13:45   ` Jakob Schöttl
2019-02-09 15:32     ` Andrei Borzenkov
2019-02-09 15:57       ` Jakob Schöttl
2019-02-09 15:36 ` Andrei Borzenkov
2019-02-10 18:44 ` Chris Murphy
2019-02-11  1:36   ` Qu Wenruo

Linux-BTRFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-btrfs/0 linux-btrfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-btrfs linux-btrfs/ https://lore.kernel.org/linux-btrfs \
		linux-btrfs@vger.kernel.org linux-btrfs@archiver.kernel.org
	public-inbox-index linux-btrfs


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-btrfs


AGPL code for this site: git clone https://public-inbox.org/ public-inbox