raid1 mount as read only

raid1 mount as read only

[Stefan K]

Hello,

I've played a little bit with raid1:
my steps was:
1. create a raid1 with btrfs (add device; balance start -mconvert=raid1 -dconvert=raid1 /)
2.  after finishing, i shutdown the server and remove a device and start it again,
3. it works (i used degraded options in fstab)
4. I shutdown the server, add the 'old' device, start it and run a btrfs balance
5. until here everything is fine
6. I shutdown the server, remove the other harddisk, and start it, 
7. it works
8. I shutdown the server, add the 'old' device, start it and try a btrfs balance
9. but now the FS is mounted as read only

I got the following messages in dmesg:
[    6.401740] BTRFS: device fsid b997e926-ab95-46d0-a9be-da52aa09203d devid 1 transid 3570 /dev/sda1
[    6.403079] BTRFS info (device sda1): allowing degraded mounts
[    6.403084] BTRFS info (device sda1): disk space caching is enabled
[    6.403086] BTRFS info (device sda1): has skinny extents
[    6.405878] BTRFS warning (device sda1): devid 2 uuid 09a7c9f6-9a13-4852-bca6-2d9120f388d4 missing
[    6.409108] BTRFS info (device sda1): detected SSD devices, enabling SSD mode

[    6.652411] BTRFS info (device sda1): allowing degraded mounts
[    6.652414] BTRFS info (device sda1): disk space caching is enabled
[    6.652416] BTRFS warning (device sda1): too many missing devices, writeable remount is not allowed

but all devices are available:
btrfs fi sh
Label: none  uuid: b997e926-ab95-46d0-a9be-da52aa09203d
        Total devices 2 FS bytes used 895.41MiB
        devid    1 size 223.57GiB used 4.06GiB path /dev/sda1
        devid    2 size 223.57GiB used 2.00GiB path /dev/sdb1

mount | grep btrfs
/dev/sda1 on / type btrfs (ro,relatime,degraded,ssd,space_cache,subvolid=5,subvol=/)

If I try to remount it via, I got the following in dmesg:
mount -o remount,rw b997e926-ab95-46d0-a9be-da52aa09203d /
[  320.048563] BTRFS info (device sda1): disk space caching is enabled
[  356.200552] BTRFS: error (device sda1) in write_all_supers:3752: errno=-5 IO failure (errors while submitting device barriers.)
[  356.200628] BTRFS info (device sda1): forced readonly
[  356.200632] BTRFS warning (device sda1): Skipping commit of aborted transaction.
[  356.200633] ------------[ cut here ]------------
[  356.200677] WARNING: CPU: 0 PID: 1914 at /build/linux-EbeuWA/linux-4.9.130/fs/btrfs/transaction.c:1850 cleanup_transaction+0x1f3/0x2e0 [btrfs]
[  356.200678] BTRFS: Transaction aborted (error -5)
[  356.200679] Modules linked in: intel_rapl skx_edac edac_core x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel mgag200 ttm drm_kms_helper intel_uncore sg drm dcdbas mei_me i2c_algo_bit intel_rapl_perf pcspkr joydev iTCO_wdt lpc_ich iTCO_vendor_support mfd_core shpchp mei evdev ipmi_si ipmi_msghandler acpi_power_meter button ip_tables x_tables autofs4 ses enclosure scsi_transport_sas sd_mod hid_generic usbhid hid btrfs crc32c_generic xor raid6_pq crc32c_intel aesni_intel aes_x86_64 glue_helper lrw gf128mul ablk_helper cryptd ahci xhci_pci libahci tg3 xhci_hcd ptp megaraid_sas libata pps_core i2c_i801 usbcore bnxt_en i2c_smbus libphy scsi_mod usb_common
[  356.200764] CPU: 0 PID: 1914 Comm: btrfs-transacti Not tainted 4.9.0-8-amd64 #1 Debian 4.9.130-2
[  356.200766] Hardware name: Dell Inc. PowerEdge R840/08XR9M, BIOS 1.2.4 10/18/2018
[  356.200768]  0000000000000000 ffffffff9ef33d74 ffffb9535d70fd50 0000000000000000
[  356.200774]  ffffffff9ec7a59e ffff8ea977b261f8 ffffb9535d70fda8 ffff8e9971cdd500
[  356.200778]  00000000fffffffb ffff8ea97114c980 0000000000000000 ffffffff9ec7a61f
[  356.200783] Call Trace:
[  356.200796]  [<ffffffff9ef33d74>] ? dump_stack+0x5c/0x78
[  356.200801]  [<ffffffff9ec7a59e>] ? __warn+0xbe/0xe0
[  356.200803]  [<ffffffff9ec7a61f>] ? warn_slowpath_fmt+0x5f/0x80
[  356.200828]  [<ffffffffc04ac423>] ? cleanup_transaction+0x1f3/0x2e0 [btrfs]
[  356.200835]  [<ffffffff9ecbd210>] ? prepare_to_wait_event+0xf0/0xf0
[  356.200858]  [<ffffffffc04ade78>] ? btrfs_commit_transaction+0x298/0xa10 [btrfs]
[  356.200879]  [<ffffffffc04ae686>] ? start_transaction+0x96/0x480 [btrfs]
[  356.200900]  [<ffffffffc04a8a8c>] ? transaction_kthread+0x1dc/0x200 [btrfs]
[  356.200919]  [<ffffffffc04a88b0>] ? btrfs_cleanup_transaction+0x580/0x580 [btrfs]
[  356.200926]  [<ffffffff9ec9a569>] ? kthread+0xd9/0xf0
[  356.200933]  [<ffffffff9f2190a4>] ? __switch_to_asm+0x34/0x70
[  356.200937]  [<ffffffff9ec9a490>] ? kthread_park+0x60/0x60
[  356.200941]  [<ffffffff9f219137>] ? ret_from_fork+0x57/0x70
[  356.200943] ---[ end trace 17ece06f94583f5a ]---
[  356.200946] BTRFS: error (device sda1) in cleanup_transaction:1850: errno=-5 IO failure
[  356.200997] BTRFS info (device sda1): delayed_refs has NO entry


My question is whats happen and how can I fix this?

uname -a
Linux yamazaki-06 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27) x86_64 GNU/Linux

btrfs version
btrfs-progs v4.7.3

Re: raid1 mount as read only

[Chris Murphy]

On Mon, Feb 18, 2019 at 2:01 AM Stefan K <shadow_7@gmx.net> wrote:
>
> Hello,
>
> I've played a little bit with raid1:
> my steps was:
> 1. create a raid1 with btrfs (add device; balance start -mconvert=raid1 -dconvert=raid1 /)
> 2.  after finishing, i shutdown the server and remove a device and start it again,
> 3. it works (i used degraded options in fstab)
> 4. I shutdown the server, add the 'old' device, start it and run a btrfs balance

What exact command? There's a very good chance there are newly created
single profile block groups that must be converted, so it requires
another -mconvert and maybe also a -dconvert, depending on what block
groups have been created during the degraded mount.

> 5. until here everything is fine
> 6. I shutdown the server, remove the other harddisk, and start it,
> 7. it works
> 8. I shutdown the server, add the 'old' device, start it and try a btrfs balance
> 9. but now the FS is mounted as read only
>
> I got the following messages in dmesg:
> [    6.401740] BTRFS: device fsid b997e926-ab95-46d0-a9be-da52aa09203d devid 1 transid 3570 /dev/sda1
> [    6.403079] BTRFS info (device sda1): allowing degraded mounts
> [    6.403084] BTRFS info (device sda1): disk space caching is enabled
> [    6.403086] BTRFS info (device sda1): has skinny extents
> [    6.405878] BTRFS warning (device sda1): devid 2 uuid 09a7c9f6-9a13-4852-bca6-2d9120f388d4 missing
> [    6.409108] BTRFS info (device sda1): detected SSD devices, enabling SSD mode
>
> [    6.652411] BTRFS info (device sda1): allowing degraded mounts
> [    6.652414] BTRFS info (device sda1): disk space caching is enabled
> [    6.652416] BTRFS warning (device sda1): too many missing devices, writeable remount is not allowed


Debian 4.9.130-2 (2018-10-27) x86_64 GNU/Linux

Kernel is too old, it doesn't have the patches that permit mounting rw
degraded more than one time before fully converting everything. You'll
need a newer kernel to permit the mount of this file system, and then
you'll need to make sure there are only raid1 block groups. I know
this is not obvious.

Not related to your problem but btrfs-progs 4.7 is really old too.
It's a common issue on the very stable distros like Debian and RHEL,
but strictly speaking if you have problems with such tools and
kernels, you need to get support from the distro. One upstream mailing
lists, no matter what file system, it's very development oriented with
focus on fixing bugs in the mainline and development kernels.

Anyway it should be safe to get a recent LiveOS. Unfortunately I'm not
certain what kernel this behavior changed in, I know for sure kernel
5.0.0 rc's have it, and I'm almost positive 4.20 has it. I'm not sure
if 4.18 has it.


-- 
Chris Murphy