* Cannot 'mount -o degraded /dev/replacement' after a replace
@ 2019-02-09 10:36 Jakob Schöttl
2019-02-09 12:16 ` Qu Wenruo
` (2 more replies)
0 siblings, 3 replies; 8+ messages in thread
From: Jakob Schöttl @ 2019-02-09 10:36 UTC (permalink / raw)
To: linux-btrfs
Hi,
I've setup a RAID1 with two disks (disk1 and disk2) and I'm testing the
btrfs replace command.
After replacing disk2 with disk3, I can only mount
(a) disk1 or disk3 (if both disk are plugged) and
(b) the original disk1 (degraded, if disk3 is unplugged).
I cannot mount the replacement disk3 if disk1 is unplugged.
> mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3,
missing codepage or helper program, or other error.
What I expect is that both disk1 and disk3 are fully valid and working
after a replace.
Steps to reproduce:
dd if=/dev/zero of=/vdisk1 bs=1024 count=300000
losetup /dev/loop1 /vdisk1
dd if=/dev/zero of=/vdisk2 bs=1024 count=300000
losetup /dev/loop2 /vdisk2
dd if=/dev/zero of=/vdisk3 bs=1024 count=300000
# losetup /dev/loop3 /vdisk3 # don't plug this device yet
Create RAID1 file system:
mkfs.btrfs -L datavol -m raid1 -d raid1 /dev/loop1 /dev/loop2
Unplug device 2 to simulate a defect:
losetup -d /dev/loop2
Plug device 3:
losetup /dev/loop3 /vdisk3
Replace device 2 with device 3:
mount -o degraded /dev/loop1 /mnt
btrfs filesystem show # to get devid of device 2
btrfs replace start -Br 2 /dev/loop3 /mnt
btrfs replace status /mnt # check success
umount /mnt
Unplug the original device 1 to see if device 3 has really replaced
device 2:
losetup -d /dev/loop1
mount -o degraded /dev/loop3 /mnt
The mount fails with this error:
> mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3,
missing codepage or helper program, or other error.
In this situation, btrfs device scan does not change anything and btrfs
filesystem show shows:
> warning, device 1 is missing
> warning, device 1 is missing
> warning, device 1 is missing
> warning, device 1 is missing
> bad tree block 198180864, bytenr mismatch, want=198180864, have=0
> ERROR: cannot read chunk root
> Label: 'datavol' uuid: 640e45d3-e741-4a78-a24e-2d8a41c6b8c3
> Total devices 2 FS bytes used 128.00KiB
> devid 2 size 292.97MiB used 104.00MiB path /dev/loop3
> *** Some devices missing
Is this a known problem? Can you reproduce it? Am I doing something wrong?
Regards, Jakob
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Cannot 'mount -o degraded /dev/replacement' after a replace
2019-02-09 10:36 Cannot 'mount -o degraded /dev/replacement' after a replace Jakob Schöttl
@ 2019-02-09 12:16 ` Qu Wenruo
2019-02-09 13:45 ` Jakob Schöttl
2019-02-09 15:36 ` Andrei Borzenkov
2019-02-10 18:44 ` Chris Murphy
2 siblings, 1 reply; 8+ messages in thread
From: Qu Wenruo @ 2019-02-09 12:16 UTC (permalink / raw)
To: Jakob Schöttl, linux-btrfs
[-- Attachment #1.1: Type: text/plain, Size: 2817 bytes --]
On 2019/2/9 下午6:36, Jakob Schöttl wrote:
> Hi,
>
> I've setup a RAID1 with two disks (disk1 and disk2) and I'm testing the
> btrfs replace command.
>
> After replacing disk2 with disk3, I can only mount
> (a) disk1 or disk3 (if both disk are plugged) and
> (b) the original disk1 (degraded, if disk3 is unplugged).
>
> I cannot mount the replacement disk3 if disk1 is unplugged.
Sounds like there is one single chunk on disk1, which caused the problem.
>
>> mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3,
> missing codepage or helper program, or other error.
dmesg please.
And btrfs-progs version please.
Maybe mkfs is too old to leave SINGLE profile chunks on the original fs.
And you could verify the chunk mapping by executing 'btrfs ins dump-tree
-t chunk <device>' and paste the output.
Thanks,
Qu
>
> What I expect is that both disk1 and disk3 are fully valid and working
> after a replace.
>
> Steps to reproduce:
>
> dd if=/dev/zero of=/vdisk1 bs=1024 count=300000
> losetup /dev/loop1 /vdisk1
> dd if=/dev/zero of=/vdisk2 bs=1024 count=300000
> losetup /dev/loop2 /vdisk2
> dd if=/dev/zero of=/vdisk3 bs=1024 count=300000
> # losetup /dev/loop3 /vdisk3 # don't plug this device yet
>
> Create RAID1 file system:
>
> mkfs.btrfs -L datavol -m raid1 -d raid1 /dev/loop1 /dev/loop2
>
> Unplug device 2 to simulate a defect:
>
> losetup -d /dev/loop2
>
> Plug device 3:
>
> losetup /dev/loop3 /vdisk3
>
> Replace device 2 with device 3:
>
> mount -o degraded /dev/loop1 /mnt
> btrfs filesystem show # to get devid of device 2
> btrfs replace start -Br 2 /dev/loop3 /mnt
> btrfs replace status /mnt # check success
> umount /mnt
>
> Unplug the original device 1 to see if device 3 has really replaced
> device 2:
>
> losetup -d /dev/loop1
> mount -o degraded /dev/loop3 /mnt
>
> The mount fails with this error:
>
>> mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3,
> missing codepage or helper program, or other error.
>
> In this situation, btrfs device scan does not change anything and btrfs
> filesystem show shows:
>
>> warning, device 1 is missing
>> warning, device 1 is missing
>> warning, device 1 is missing
>> warning, device 1 is missing
>> bad tree block 198180864, bytenr mismatch, want=198180864, have=0
>> ERROR: cannot read chunk root
>> Label: 'datavol' uuid: 640e45d3-e741-4a78-a24e-2d8a41c6b8c3
>> Total devices 2 FS bytes used 128.00KiB
>> devid 2 size 292.97MiB used 104.00MiB path /dev/loop3
>> *** Some devices missing
>
> Is this a known problem? Can you reproduce it? Am I doing something wrong?
>
> Regards, Jakob
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Cannot 'mount -o degraded /dev/replacement' after a replace
2019-02-09 12:16 ` Qu Wenruo
@ 2019-02-09 13:45 ` Jakob Schöttl
2019-02-09 15:32 ` Andrei Borzenkov
0 siblings, 1 reply; 8+ messages in thread
From: Jakob Schöttl @ 2019-02-09 13:45 UTC (permalink / raw)
To: Qu Wenruo, linux-btrfs
Thanks Qu,
Am 09.02.19 um 13:16 schrieb Qu Wenruo:
> On 2019/2/9 下午6:36, Jakob Schöttl wrote:
>> Hi,
>>
>> I've setup a RAID1 with two disks (disk1 and disk2) and I'm testing the
>> btrfs replace command.
>>
>> After replacing disk2 with disk3, I can only mount
>> (a) disk1 or disk3 (if both disk are plugged) and
>> (b) the original disk1 (degraded, if disk3 is unplugged).
>>
>> I cannot mount the replacement disk3 if disk1 is unplugged.
> Sounds like there is one single chunk on disk1, which caused the problem.
>
>>> mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3,
>> missing codepage or helper program, or other error.
> dmesg please.
[60456.856883] BTRFS: device label datavol devid 1 transid 5 /dev/loop1
[60456.940785] BTRFS: device label datavol devid 2 transid 5 /dev/loop2
[60525.211389] BTRFS info (device loop1): allowing degraded mounts
[60525.211395] BTRFS info (device loop1): disk space caching is enabled
[60525.211398] BTRFS info (device loop1): has skinny extents
[60525.211401] BTRFS info (device loop1): flagging fs with big metadata
feature
[60525.213854] BTRFS warning (device loop1): devid 2 uuid
0b4e0b31-e2b1-40a0-8360-09978f58a2e4 is missing
[60525.214639] BTRFS info (device loop1): checking UUID tree
[60525.386695] BTRFS info (device loop1): dev_replace from <missing
disk> (devid 2) to /dev/loop3 started
[60525.394403] BTRFS info (device loop1): dev_replace from <missing
disk> (devid 2) to /dev/loop3 finished
[60533.721841] BTRFS info (device loop3): allowing degraded mounts
[60533.721846] BTRFS info (device loop3): disk space caching is enabled
[60533.721850] BTRFS info (device loop3): has skinny extents
[60533.723703] BTRFS error (device loop3): failed to read chunk root
[60533.773553] BTRFS error (device loop3): open_ctree failed
> And btrfs-progs version please.
$ pacman -Q btrfs-progs
btrfs-progs 4.20.1-2
$ uname -a
Linux jathink 4.20.7-arch1-1-ARCH #1 SMP PREEMPT Wed Feb 6 18:42:40 UTC
2019 x86_64 GNU/Linux
> Maybe mkfs is too old to leave SINGLE profile chunks on the original fs.
>
> And you could verify the chunk mapping by executing 'btrfs ins dump-tree
> -t chunk <device>' and paste the output.
When only /dev/loop3 is plugged:
# btrfs inspect-internal dump-tree -t chunk /dev/loop3
btrfs-progs v4.20.1
warning, device 1 is missing
warning, device 1 is missing
warning, device 1 is missing
warning, device 1 is missing
bad tree block 198180864, bytenr mismatch, want=198180864, have=0
ERROR: cannot read chunk root
ERROR: unable to open /dev/loop3
When only /dev/loop1 is plugged:
# btrfs inspect-internal dump-tree -t chunk /dev/loop1
btrfs-progs v4.20.1
warning, device 2 is missing
chunk tree
leaf 198180864 items 10 free space 15005 generation 8 owner CHUNK_TREE
leaf 198180864 flags 0x1(WRITTEN) backref revision 1
fs uuid 005a8d59-a561-4371-869e-b0ccc4a4862b
chunk uuid b3e609f1-a7fe-4add-bc51-6231f0bbf320
item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
devid 1 total_bytes 307200000 bytes_used 306053120
io_align 4096 io_width 4096 sector_size 4096 type 0
generation 0 start_offset 0 dev_group 0
seek_speed 0 bandwidth 0
uuid 043443c7-ac91-4085-a5e4-983b59dd0803
fsid 005a8d59-a561-4371-869e-b0ccc4a4862b
item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
devid 2 total_bytes 307200000 bytes_used 109051904
io_align 4096 io_width 4096 sector_size 4096 type 0
generation 0 start_offset 0 dev_group 0
seek_speed 0 bandwidth 0
uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
fsid 005a8d59-a561-4371-869e-b0ccc4a4862b
item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975
itemsize 112
length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 0
stripe 0 devid 2 offset 1048576
dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
stripe 1 devid 1 offset 22020096
dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863
itemsize 112
length 33554432 owner 2 stripe_len 65536 type METADATA|RAID1
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 0
stripe 0 devid 2 offset 9437184
dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
stripe 1 devid 1 offset 30408704
dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 63963136) itemoff 15751
itemsize 112
length 67108864 owner 2 stripe_len 65536 type DATA|RAID1
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 0
stripe 0 devid 2 offset 42991616
dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
stripe 1 devid 1 offset 63963136
dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 131072000) itemoff 15671
itemsize 80
length 33554432 owner 2 stripe_len 65536 type METADATA
io_align 65536 io_width 65536 sector_size 4096
num_stripes 1 sub_stripes 1
stripe 0 devid 1 offset 131072000
dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
item 6 key (FIRST_CHUNK_TREE CHUNK_ITEM 164626432) itemoff 15591
itemsize 80
length 33554432 owner 2 stripe_len 65536 type SYSTEM
io_align 65536 io_width 65536 sector_size 4096
num_stripes 1 sub_stripes 1
stripe 0 devid 1 offset 164626432
dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 198180864) itemoff 15479
itemsize 112
length 33554432 owner 2 stripe_len 65536 type SYSTEM|DUP
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 1
stripe 0 devid 1 offset 198180864
dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
stripe 1 devid 1 offset 231735296
dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
item 8 key (FIRST_CHUNK_TREE CHUNK_ITEM 231735296) itemoff 15367
itemsize 112
length 20905984 owner 2 stripe_len 65536 type METADATA|DUP
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 1
stripe 0 devid 1 offset 265289728
dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
stripe 1 devid 1 offset 286195712
dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
item 9 key (FIRST_CHUNK_TREE CHUNK_ITEM 252641280) itemoff 15255
itemsize 112
length 10485760 owner 2 stripe_len 65536 type DATA|DUP
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 1
stripe 0 devid 1 offset 1048576
dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
stripe 1 devid 1 offset 11534336
dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Cannot 'mount -o degraded /dev/replacement' after a replace
2019-02-09 13:45 ` Jakob Schöttl
@ 2019-02-09 15:32 ` Andrei Borzenkov
2019-02-09 15:57 ` Jakob Schöttl
0 siblings, 1 reply; 8+ messages in thread
From: Andrei Borzenkov @ 2019-02-09 15:32 UTC (permalink / raw)
To: Jakob Schöttl, Qu Wenruo, linux-btrfs
09.02.2019 16:45, Jakob Schöttl пишет:
> Thanks Qu,
>
> Am 09.02.19 um 13:16 schrieb Qu Wenruo:
>> On 2019/2/9 下午6:36, Jakob Schöttl wrote:
>>> Hi,
>>>
>>> I've setup a RAID1 with two disks (disk1 and disk2) and I'm testing the
>>> btrfs replace command.
>>>
>>> After replacing disk2 with disk3, I can only mount
>>> (a) disk1 or disk3 (if both disk are plugged) and
>>> (b) the original disk1 (degraded, if disk3 is unplugged).
>>>
>>> I cannot mount the replacement disk3 if disk1 is unplugged.
>> Sounds like there is one single chunk on disk1, which caused the problem.
>>
>>>> mount: /mnt: wrong fs type, bad option, bad superblock on /dev/loop3,
>>> missing codepage or helper program, or other error.
>> dmesg please.
> [60456.856883] BTRFS: device label datavol devid 1 transid 5 /dev/loop1
> [60456.940785] BTRFS: device label datavol devid 2 transid 5 /dev/loop2
> [60525.211389] BTRFS info (device loop1): allowing degraded mounts
> [60525.211395] BTRFS info (device loop1): disk space caching is enabled
> [60525.211398] BTRFS info (device loop1): has skinny extents
> [60525.211401] BTRFS info (device loop1): flagging fs with big metadata
> feature
> [60525.213854] BTRFS warning (device loop1): devid 2 uuid
> 0b4e0b31-e2b1-40a0-8360-09978f58a2e4 is missing
> [60525.214639] BTRFS info (device loop1): checking UUID tree
> [60525.386695] BTRFS info (device loop1): dev_replace from <missing
> disk> (devid 2) to /dev/loop3 started
> [60525.394403] BTRFS info (device loop1): dev_replace from <missing
> disk> (devid 2) to /dev/loop3 finished
> [60533.721841] BTRFS info (device loop3): allowing degraded mounts
> [60533.721846] BTRFS info (device loop3): disk space caching is enabled
> [60533.721850] BTRFS info (device loop3): has skinny extents
> [60533.723703] BTRFS error (device loop3): failed to read chunk root
> [60533.773553] BTRFS error (device loop3): open_ctree failed
>
>> And btrfs-progs version please.
> $ pacman -Q btrfs-progs
> btrfs-progs 4.20.1-2
> $ uname -a
> Linux jathink 4.20.7-arch1-1-ARCH #1 SMP PREEMPT Wed Feb 6 18:42:40 UTC
> 2019 x86_64 GNU/Linux
>
>> Maybe mkfs is too old to leave SINGLE profile chunks on the original fs.
>>
>> And you could verify the chunk mapping by executing 'btrfs ins dump-tree
>> -t chunk <device>' and paste the output.
>
> When only /dev/loop3 is plugged:
>
> # btrfs inspect-internal dump-tree -t chunk /dev/loop3
> btrfs-progs v4.20.1
> warning, device 1 is missing
> warning, device 1 is missing
> warning, device 1 is missing
> warning, device 1 is missing
> bad tree block 198180864, bytenr mismatch, want=198180864, have=0
> ERROR: cannot read chunk root
> ERROR: unable to open /dev/loop3
>
> When only /dev/loop1 is plugged:
>
> # btrfs inspect-internal dump-tree -t chunk /dev/loop1
> btrfs-progs v4.20.1
> warning, device 2 is missing
> chunk tree
> leaf 198180864 items 10 free space 15005 generation 8 owner CHUNK_TREE
> leaf 198180864 flags 0x1(WRITTEN) backref revision 1
> fs uuid 005a8d59-a561-4371-869e-b0ccc4a4862b
> chunk uuid b3e609f1-a7fe-4add-bc51-6231f0bbf320
> item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
> devid 1 total_bytes 307200000 bytes_used 306053120
> io_align 4096 io_width 4096 sector_size 4096 type 0
> generation 0 start_offset 0 dev_group 0
> seek_speed 0 bandwidth 0
> uuid 043443c7-ac91-4085-a5e4-983b59dd0803
> fsid 005a8d59-a561-4371-869e-b0ccc4a4862b
> item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
> devid 2 total_bytes 307200000 bytes_used 109051904
> io_align 4096 io_width 4096 sector_size 4096 type 0
> generation 0 start_offset 0 dev_group 0
> seek_speed 0 bandwidth 0
> uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
> fsid 005a8d59-a561-4371-869e-b0ccc4a4862b
> item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975
> itemsize 112
> length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 2 sub_stripes 0
> stripe 0 devid 2 offset 1048576
> dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
> stripe 1 devid 1 offset 22020096
> dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
> item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863
> itemsize 112
> length 33554432 owner 2 stripe_len 65536 type METADATA|RAID1
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 2 sub_stripes 0
> stripe 0 devid 2 offset 9437184
> dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
> stripe 1 devid 1 offset 30408704
> dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
> item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 63963136) itemoff 15751
> itemsize 112
> length 67108864 owner 2 stripe_len 65536 type DATA|RAID1
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 2 sub_stripes 0
> stripe 0 devid 2 offset 42991616
> dev_uuid 0b4e0b31-e2b1-40a0-8360-09978f58a2e4
> stripe 1 devid 1 offset 63963136
> dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
> item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 131072000) itemoff 15671
> itemsize 80
> length 33554432 owner 2 stripe_len 65536 type METADATA
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 1 sub_stripes 1
> stripe 0 devid 1 offset 131072000
> dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
> item 6 key (FIRST_CHUNK_TREE CHUNK_ITEM 164626432) itemoff 15591
> itemsize 80
> length 33554432 owner 2 stripe_len 65536 type SYSTEM
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 1 sub_stripes 1
> stripe 0 devid 1 offset 164626432
> dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
> item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 198180864) itemoff 15479
> itemsize 112
> length 33554432 owner 2 stripe_len 65536 type SYSTEM|DUP
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 2 sub_stripes 1
> stripe 0 devid 1 offset 198180864
> dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
> stripe 1 devid 1 offset 231735296
> dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
> item 8 key (FIRST_CHUNK_TREE CHUNK_ITEM 231735296) itemoff 15367
> itemsize 112
> length 20905984 owner 2 stripe_len 65536 type METADATA|DUP
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 2 sub_stripes 1
> stripe 0 devid 1 offset 265289728
> dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
> stripe 1 devid 1 offset 286195712
> dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
> item 9 key (FIRST_CHUNK_TREE CHUNK_ITEM 252641280) itemoff 15255
> itemsize 112
> length 10485760 owner 2 stripe_len 65536 type DATA|DUP
> io_align 65536 io_width 65536 sector_size 4096
> num_stripes 2 sub_stripes 1
> stripe 0 devid 1 offset 1048576
> dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
> stripe 1 devid 1 offset 11534336
> dev_uuid 043443c7-ac91-4085-a5e4-983b59dd0803
>
>
Yes, I can trivially reproduce it using commands in original post.
Immediately after mkfs:
10:/home/bor # btrfs inspect-internal dump-tree -t chunk /dev/sdb1
btrfs-progs v4.15
warning, device 2 is missing
chunk tree
leaf 22036480 items 5 free space 15626 generation 5 owner 3
leaf 22036480 flags 0x1(WRITTEN) backref revision 1
fs uuid d33e6bff-5903-449d-b49c-f2d992676e30
chunk uuid ed55b2ad-66c4-4764-acd5-7fa3a9fe619e
item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
devid 1 total_bytes 5367640064 bytes_used 1081868288
io_align 4096 io_width 4096 sector_size 4096 type 0
generation 0 start_offset 0 dev_group 0
seek_speed 0 bandwidth 0
uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
fsid d33e6bff-5903-449d-b49c-f2d992676e30
item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
devid 2 total_bytes 5367640064 bytes_used 1081868288
io_align 4096 io_width 4096 sector_size 4096 type 0
generation 0 start_offset 0 dev_group 0
seek_speed 0 bandwidth 0
uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
fsid d33e6bff-5903-449d-b49c-f2d992676e30
item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975
itemsize 112
length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 0
stripe 0 devid 2 offset 1048576
dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
stripe 1 devid 1 offset 22020096
dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863
itemsize 112
length 536739840 owner 2 stripe_len 65536 type METADATA|RAID1
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 0
stripe 0 devid 2 offset 9437184
dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
stripe 1 devid 1 offset 30408704
dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 567148544) itemoff 15751
itemsize 112
length 536739840 owner 2 stripe_len 65536 type DATA|RAID1
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 0
stripe 0 devid 2 offset 546177024
dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
stripe 1 devid 1 offset 567148544
dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
Immediately after degraded mount
btrfs-progs v4.15
warning, device 2 is missing
chunk tree
leaf 22020096 items 8 free space 15311 generation 6 owner 3
leaf 22020096 flags 0x1(WRITTEN) backref revision 1
fs uuid e69b93ef-d852-4afa-8c74-50fa07d33a01
chunk uuid 74317bdd-f011-44a8-9c62-0cce665f1b52
item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
devid 1 total_bytes 5368709120 bytes_used 2994733056
io_align 4096 io_width 4096 sector_size 4096 type 0
generation 0 start_offset 0 dev_group 0
seek_speed 0 bandwidth 0
uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
fsid e69b93ef-d852-4afa-8c74-50fa07d33a01
item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
devid 2 total_bytes 5368709120 bytes_used 2155872256
io_align 4096 io_width 4096 sector_size 4096 type 0
generation 0 start_offset 0 dev_group 0
seek_speed 0 bandwidth 0
uuid 1989d9cd-9f82-4d3a-a577-c7e583157ed3
fsid e69b93ef-d852-4afa-8c74-50fa07d33a01
item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975
itemsize 112
length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 0
stripe 0 devid 2 offset 1048576
dev_uuid 1989d9cd-9f82-4d3a-a577-c7e583157ed3
stripe 1 devid 1 offset 22020096
dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863
itemsize 112
length 1073741824 owner 2 stripe_len 65536 type METADATA|RAID1
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 0
stripe 0 devid 2 offset 9437184
dev_uuid 1989d9cd-9f82-4d3a-a577-c7e583157ed3
stripe 1 devid 1 offset 30408704
dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 1104150528) itemoff 15751
itemsize 112
length 1073741824 owner 2 stripe_len 65536 type DATA|RAID1
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 0
stripe 0 devid 2 offset 1083179008
dev_uuid 1989d9cd-9f82-4d3a-a577-c7e583157ed3
stripe 1 devid 1 offset 1104150528
dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 2177892352) itemoff 15671
itemsize 80
length 268435456 owner 2 stripe_len 65536 type METADATA
io_align 65536 io_width 65536 sector_size 4096
num_stripes 1 sub_stripes 1
stripe 0 devid 1 offset 2177892352
dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
item 6 key (FIRST_CHUNK_TREE CHUNK_ITEM 2446327808) itemoff 15591
itemsize 80
length 33554432 owner 2 stripe_len 65536 type SYSTEM
io_align 65536 io_width 65536 sector_size 4096
num_stripes 1 sub_stripes 1
stripe 0 devid 1 offset 2446327808
dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 2479882240) itemoff 15511
itemsize 80
length 536870912 owner 2 stripe_len 65536 type DATA
io_align 65536 io_width 65536 sector_size 4096
num_stripes 1 sub_stripes 1
stripe 0 devid 1 offset 2479882240
dev_uuid 85365dff-ceb8-46bb-a4c5-78ce422a32f8
After device replacement completed
10:/home/bor # btrfs replace status /mnt
Started on 9.Feb 18:07:18, finished on 9.Feb 18:07:19, 0 write errs, 0
uncorr. read errs
10:/home/bor # btrfs inspect-internal dump-tree -t chunk /dev/sdb1
btrfs-progs v4.15
chunk tree
leaf 1942749184 items 11 free space 14900 generation 9 owner 3
leaf 1942749184 flags 0x1(WRITTEN) backref revision 1
fs uuid d33e6bff-5903-449d-b49c-f2d992676e30
chunk uuid ed55b2ad-66c4-4764-acd5-7fa3a9fe619e
item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
devid 1 total_bytes 5367640064 bytes_used 2793144320
io_align 4096 io_width 4096 sector_size 4096 type 0
generation 0 start_offset 0 dev_group 0
seek_speed 0 bandwidth 0
uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
fsid d33e6bff-5903-449d-b49c-f2d992676e30
item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
devid 2 total_bytes 5367640064 bytes_used 1081868288
io_align 4096 io_width 4096 sector_size 4096 type 0
generation 0 start_offset 0 dev_group 0
seek_speed 0 bandwidth 0
uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
fsid d33e6bff-5903-449d-b49c-f2d992676e30
item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975
itemsize 112
length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 0
stripe 0 devid 2 offset 1048576
dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
stripe 1 devid 1 offset 22020096
dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863
itemsize 112
length 536739840 owner 2 stripe_len 65536 type METADATA|RAID1
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 0
stripe 0 devid 2 offset 9437184
dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
stripe 1 devid 1 offset 30408704
dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 567148544) itemoff 15751
itemsize 112
length 536739840 owner 2 stripe_len 65536 type DATA|RAID1
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 0
stripe 0 devid 2 offset 546177024
dev_uuid 1e91c346-eba9-4a41-aeb2-d7efef64a5ea
stripe 1 devid 1 offset 567148544
dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 1103888384) itemoff 15671
itemsize 80
length 268435456 owner 2 stripe_len 65536 type METADATA
io_align 65536 io_width 65536 sector_size 4096
num_stripes 1 sub_stripes 1
stripe 0 devid 1 offset 1103888384
dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
item 6 key (FIRST_CHUNK_TREE CHUNK_ITEM 1372323840) itemoff 15591
itemsize 80
length 33554432 owner 2 stripe_len 65536 type SYSTEM
io_align 65536 io_width 65536 sector_size 4096
num_stripes 1 sub_stripes 1
stripe 0 devid 1 offset 1372323840
dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 1405878272) itemoff 15511
itemsize 80
length 536870912 owner 2 stripe_len 65536 type DATA
io_align 65536 io_width 65536 sector_size 4096
num_stripes 1 sub_stripes 1
stripe 0 devid 1 offset 1405878272
dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
item 8 key (FIRST_CHUNK_TREE CHUNK_ITEM 1942749184) itemoff 15399
itemsize 112
length 33554432 owner 2 stripe_len 65536 type SYSTEM|DUP
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 1
stripe 0 devid 1 offset 1942749184
dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
stripe 1 devid 1 offset 1976303616
dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
item 9 key (FIRST_CHUNK_TREE CHUNK_ITEM 1976303616) itemoff 15287
itemsize 112
length 134217728 owner 2 stripe_len 65536 type METADATA|DUP
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 1
stripe 0 devid 1 offset 2009858048
dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
stripe 1 devid 1 offset 2144075776
dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
item 10 key (FIRST_CHUNK_TREE CHUNK_ITEM 2110521344) itemoff 15175
itemsize 112
length 268435456 owner 2 stripe_len 65536 type DATA|DUP
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 1
stripe 0 devid 1 offset 2278293504
dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
stripe 1 devid 1 offset 2546728960
dev_uuid 653ea52e-f95f-417b-af6f-feb82ca49cd6
10:/home/bor #
Running
btrfs balance start -f -ssoft,convert=raid1 -dsoft,convert=raid1
-msoft,convert=raid1 /mnt
fixes it. I am not sure whether btrfs is expected to do it automatically
as of now.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Cannot 'mount -o degraded /dev/replacement' after a replace
2019-02-09 10:36 Cannot 'mount -o degraded /dev/replacement' after a replace Jakob Schöttl
2019-02-09 12:16 ` Qu Wenruo
@ 2019-02-09 15:36 ` Andrei Borzenkov
2019-02-10 18:44 ` Chris Murphy
2 siblings, 0 replies; 8+ messages in thread
From: Andrei Borzenkov @ 2019-02-09 15:36 UTC (permalink / raw)
To: Jakob Schöttl, linux-btrfs
09.02.2019 13:36, Jakob Schöttl пишет:
> btrfs filesystem show # to get devid of device 2
Funny because for me it just says "Some devices missing" without giving
exact device id. May be has been fixed in recent versions but for a long
time it really annoyed me.
OK, when I just unplug device it does not say it - but it also does not
say *which* device is missing, it happily continues to show old device
names at the time of mount, be they actually present or not.
I always wondered how users are supposed to know which device to replace.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Cannot 'mount -o degraded /dev/replacement' after a replace
2019-02-09 15:32 ` Andrei Borzenkov
@ 2019-02-09 15:57 ` Jakob Schöttl
0 siblings, 0 replies; 8+ messages in thread
From: Jakob Schöttl @ 2019-02-09 15:57 UTC (permalink / raw)
To: Andrei Borzenkov, linux-btrfs; +Cc: Qu Wenruo
Am 09.02.19 um 16:32 schrieb Andrei Borzenkov:
> Running
>
> btrfs balance start -f -ssoft,convert=raid1 -dsoft,convert=raid1
> -msoft,convert=raid1 /mnt
>
> fixes it. I am not sure whether btrfs is expected to do it automatically
> as of now.
This fix seems to work for me, too. Output was:
> Done, had to relocate 5 out of 8 chunks
Tested without any subvolumes or data.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Cannot 'mount -o degraded /dev/replacement' after a replace
2019-02-09 10:36 Cannot 'mount -o degraded /dev/replacement' after a replace Jakob Schöttl
2019-02-09 12:16 ` Qu Wenruo
2019-02-09 15:36 ` Andrei Borzenkov
@ 2019-02-10 18:44 ` Chris Murphy
2019-02-11 1:36 ` Qu Wenruo
2 siblings, 1 reply; 8+ messages in thread
From: Chris Murphy @ 2019-02-10 18:44 UTC (permalink / raw)
To: Jakob Schöttl; +Cc: Btrfs BTRFS
On Sat, Feb 9, 2019 at 3:36 AM Jakob Schöttl <jschoett@gmail.com> wrote:
>
> Replace device 2 with device 3:
>
> mount -o degraded /dev/loop1 /mnt
> btrfs filesystem show # to get devid of device 2
> btrfs replace start -Br 2 /dev/loop3 /mnt
> btrfs replace status /mnt # check success
> umount /mnt
The first command causes the creation of metadata and system chunk
using single profile, on loop1, and even after a replace, both devices
know that there is a single and required chunk on loop1. So subsequent
degraded mount fails because the volume was not fully redundant.
The missing step is between 'btrfs replaced start' completing, and
umount. There needed to be at least a 'btrfs fi us' to check if there
are any single chunks and to convert them with 'btrfs balance start
-dconvert=raid1,soft -mconvert=raid1,soft' where soft is only possible
if there were no data writes while degraded; if there were degraded
writes, a scrub is also necessary to make sure any single mirror raid1
chunk writes to the degraded device are replicated to the replacement.
Yep - kinda ugly. And not intuitive. But that's the present status.
--
Chris Murphy
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Cannot 'mount -o degraded /dev/replacement' after a replace
2019-02-10 18:44 ` Chris Murphy
@ 2019-02-11 1:36 ` Qu Wenruo
0 siblings, 0 replies; 8+ messages in thread
From: Qu Wenruo @ 2019-02-11 1:36 UTC (permalink / raw)
To: Chris Murphy, Jakob Schöttl; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 1561 bytes --]
On 2019/2/11 上午2:44, Chris Murphy wrote:
> On Sat, Feb 9, 2019 at 3:36 AM Jakob Schöttl <jschoett@gmail.com> wrote:
>>
>> Replace device 2 with device 3:
>>
>> mount -o degraded /dev/loop1 /mnt
>> btrfs filesystem show # to get devid of device 2
>> btrfs replace start -Br 2 /dev/loop3 /mnt
>> btrfs replace status /mnt # check success
>> umount /mnt
>
> The first command causes the creation of metadata and system chunk
> using single profile, on loop1, and even after a replace, both devices
> know that there is a single and required chunk on loop1. So subsequent
> degraded mount fails because the volume was not fully redundant.
>
> The missing step is between 'btrfs replaced start' completing, and
> umount. There needed to be at least a 'btrfs fi us' to check if there
> are any single chunks and to convert them with 'btrfs balance start
> -dconvert=raid1,soft -mconvert=raid1,soft' where soft is only possible
> if there were no data writes while degraded; if there were degraded
> writes, a scrub is also necessary to make sure any single mirror raid1
> chunk writes to the degraded device are replicated to the replacement.
>
> Yep - kinda ugly. And not intuitive. But that's the present status.
Not exactly.
Currently the problem is, there is extra DUP sys/meta/data chunks created.
So despite the SINGLE chunks, we still can't mount it degraded.
IIRC there is some patch trying to address the SINGLE problem, but the
DUP one seems to be something new.
Thanks,
Qu
>
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2019-02-11 1:37 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-02-09 10:36 Cannot 'mount -o degraded /dev/replacement' after a replace Jakob Schöttl
2019-02-09 12:16 ` Qu Wenruo
2019-02-09 13:45 ` Jakob Schöttl
2019-02-09 15:32 ` Andrei Borzenkov
2019-02-09 15:57 ` Jakob Schöttl
2019-02-09 15:36 ` Andrei Borzenkov
2019-02-10 18:44 ` Chris Murphy
2019-02-11 1:36 ` Qu Wenruo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).