BTRFS converted from EXT4 becomes read-only after reboot

BTRFS converted from EXT4 becomes read-only after reboot

[Alexandru Guzu]

Hi all,

In a VirtualBox VM, I converted a EXT4 fs to BTRFS that is now running
on Ubuntu 16.04 (Kernel 4.4.0-72). I was able to use the system for
several weeks. I even did kernel updates, compression, deduplication
without issues.

Since today, a little while after booting (usually when I start
opening applications), the FS becomes read-only. IMO, the only thing
that might have changed that could affect this is a kernel upgrade.
However, since the FS becomes RO, I cannot easily do a kernel update
(I guess I'd have to chroot and do it).

The stack trace is the following:

[   88.836057] ------------[ cut here ]------------
[   88.836082] WARNING: CPU: 0 PID: 25 at
/build/linux-wXdoVv/linux-4.4.0/fs/btrfs/inode.c:2931
btrfs_finish_ordered_io+0x63b/0x650 [btrfs]()
[   88.836083] BTRFS: Transaction aborted (error -95)
[   88.836084] Modules linked in: nvram msr zram lz4_compress
vboxsf(OE) joydev crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
aesni_intel snd_intel8x0 aes_x86_64 snd_ac97_codec lrw gf128mul
glue_helper ablk_helper cryptd ac97_bus snd_pcm snd_seq_midi
snd_seq_midi_event snd_rawmidi snd_seq vboxvideo(OE) snd_seq_device
snd_timer ttm input_leds drm_kms_helper serio_raw i2c_piix4 snd drm
fb_sys_fops syscopyarea sysfillrect sysimgblt soundcore 8250_fintek
vboxguest(OE) mac_hid parport_pc ppdev lp parport autofs4 btrfs xor
raid6_pq hid_generic usbhid hid psmouse ahci libahci fjes video
pata_acpi
[   88.836116] CPU: 0 PID: 25 Comm: kworker/u2:1 Tainted: G
OE   4.4.0-72-generic #93-Ubuntu
[   88.836117] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS
VirtualBox 12/01/2006
[   88.836130] Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
[   88.836132]  0000000000000286 00000000618d3a00 ffff88007cabbc78
ffffffff813f82b3
[   88.836134]  ffff88007cabbcc0 ffffffffc01912a8 ffff88007cabbcb0
ffffffff81081302
[   88.836135]  ffff880058bf01b0 ffff8800355f2800 ffff88007b9e64e0
ffff88007c66f098
[   88.836137] Call Trace:
[   88.836142]  [<ffffffff813f82b3>] dump_stack+0x63/0x90
[   88.836145]  [<ffffffff81081302>] warn_slowpath_common+0x82/0xc0
[   88.836147]  [<ffffffff8108139c>] warn_slowpath_fmt+0x5c/0x80
[   88.836159]  [<ffffffffc012373f>] ? unpin_extent_cache+0x8f/0xe0 [btrfs]
[   88.836167]  [<ffffffffc00e0b06>] ? btrfs_free_path+0x26/0x30 [btrfs]
[   88.836178]  [<ffffffffc011554b>] btrfs_finish_ordered_io+0x63b/0x650 [btrfs]
[   88.836188]  [<ffffffffc0115845>] finish_ordered_fn+0x15/0x20 [btrfs]
[   88.836200]  [<ffffffffc013dfda>] btrfs_scrubparity_helper+0xca/0x2f0 [btrfs]
[   88.836202]  [<ffffffff810caee1>] ?
__raw_callee_save___pv_queued_spin_unlock+0x11/0x20
[   88.836214]  [<ffffffffc013e28e>] btrfs_endio_write_helper+0xe/0x10 [btrfs]
[   88.836217]  [<ffffffff8109a555>] process_one_work+0x165/0x480
[   88.836219]  [<ffffffff8109a8bb>] worker_thread+0x4b/0x4c0
[   88.836220]  [<ffffffff8109a870>] ? process_one_work+0x480/0x480
[   88.836222]  [<ffffffff8109a870>] ? process_one_work+0x480/0x480
[   88.836224]  [<ffffffff810a0be8>] kthread+0xd8/0xf0
[   88.836225]  [<ffffffff810a0b10>] ? kthread_create_on_node+0x1e0/0x1e0
[   88.836229]  [<ffffffff8183ca0f>] ret_from_fork+0x3f/0x70
[   88.836230]  [<ffffffff810a0b10>] ? kthread_create_on_node+0x1e0/0x1e0
[   88.836232] ---[ end trace f4b8dbb54f0db139 ]---
[   88.836234] BTRFS: error (device sda1) in
btrfs_finish_ordered_io:2931: errno=-95 unknown
[   88.836236] BTRFS info (device sda1): forced readonly
[   88.836392] pending csums is 118784

If I reboot with the Live CD version, I can run `scrub` and `check`
without any issues. Also the FS stays read-write.

Is this a known issue? Could it be due to the conversion, or can this
happen on any system on any time. This is a test VM that I can
re-make, but I would like to know if this can happen on a production
system.

Regards,
Alex.

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Chris Murphy]

On Wed, May 3, 2017 at 2:28 PM, Alexandru Guzu <alexguzu@gmail.com> wrote:

> In a VirtualBox VM, I converted a EXT4 fs to BTRFS that is now running
> on Ubuntu 16.04 (Kernel 4.4.0-72). I was able to use the system for
> several weeks. I even did kernel updates, compression, deduplication
> without issues.

Which version of btrfs-progs? The convert code was totally rewritten
for btrfs-progs 4.6, and it has had a number of significant fixes
since then. I'd say 4.8.2 is about the minimum I'd recommend. But if
it were me I'd use the current release, 4.10.2.

But that alone may not fix it, I think you need a newer kernel...

>
> The stack trace is the following:
>
> [   88.836057] ------------[ cut here ]------------
> [   88.836082] WARNING: CPU: 0 PID: 25 at
> /build/linux-wXdoVv/linux-4.4.0/fs/btrfs/inode.c:2931
> btrfs_finish_ordered_io+0x63b/0x650 [btrfs]()
> [   88.836083] BTRFS: Transaction aborted (error -95)

Fits this thread, which suggests you're using a btrfs-progs 4.6 or newer.

The call trace breakdown:
http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg57565.html

The evaluation:
http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg57634.html


-- 
Chris Murphy

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Duncan]

Chris Murphy posted on Wed, 03 May 2017 16:18:34 -0600 as excerpted:

> On Wed, May 3, 2017 at 2:28 PM, Alexandru Guzu <alexguzu@gmail.com>
> wrote:
> 
>> In a VirtualBox VM, I converted a EXT4 fs to BTRFS that is now running
>> on Ubuntu 16.04 (Kernel 4.4.0-72). I was able to use the system for
>> several weeks. I even did kernel updates, compression, deduplication
>> without issues.
> 
> Which version of btrfs-progs? The convert code was totally rewritten for
> btrfs-progs 4.6, and it has had a number of significant fixes since
> then. I'd say 4.8.2 is about the minimum I'd recommend. But if it were
> me I'd use the current release, 4.10.2.
> 
> But that alone may not fix it, I think you need a newer kernel...

Well, while the 4.4 LTS kernel series /is/ getting a bit long in the 
tooth by now, it's still the second newest LTS series available, 4.9 
being the newest.

And on-list we've long recommended staying within the latest two series 
in either the LTS or current kernel series, which means the 4.4 series 
should still be reasonably supported.  

So if there's a problem with his 4.4, it means one of three things:

a) We're regressing on our stability efforts and are no longer trying to 
support the second newest LTS kernel series.

I would hope not.  If so, it means we /are/ actually regressing in our 
stability efforts, as AFAIK we've been supporting the two latest LTS 
series since we formally peeled off the experimental label and started 
actually doing stable series backports in the 3.12-ish era.

OTOH, giving up on the penultimate LTS /would/ mean less backport and 
support hassles for anything older than the latest 4.9 LTS series, as we 
could simply tell people the same thing we're telling them about older 
versions now, that btrfs is under fast enough development and unstable 
enough that if they choose to run it, they really should try to stay 
reasonably current, and that they need to upgrade, and we can forget 
about it until/unless they do.

b) A recent patch that should have been marked for stable either wasn't, 
or is still in the stable queue.

This is certainly possible, tho along with most here I track current (or 
development), not LTS stable, and would be unlikely to know the status of 
stable backports.  This being the reality, perhaps we /should/ just give 
up on the second newest LTS.

This one's obviously the most worrying from the point of view of this 
list, assuming we /are/ still trying to support the 4.4 series as the 
penultimate LTS series available.

c) It's the still supported 4.4-LTS, but whatever his 4.4.0-72 converts 
to in upstream version numbers is behind the current release (4.4.66 ATM 
according to kernel.org) in one or more current btrfs stable-series 
patches.

That's something the poster would have to track down with his distro, 
since they obviously chose to use non-transparent versioning that doesn't 
correspond to upstream in that regard, so people not on that distro have 
little way of knowing without going the extra distance to attempt looking 
it up on the distro site (and why should they, the distro obviously chose 
non-transparency), and people /on/ the distro might be able to look it 
up, but probably won't know either unless they actually do, /because/ of 
that non-transparency choice.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Chris Murphy]

On Fri, May 5, 2017 at 8:56 AM, Alexandru Guzu <alexguzu@gmail.com> wrote:
> btrfs-progs is 4.4

That's before the rewrite of convert.


> I upgraded the kernel to 4.8.0-51 and the issue persists.
> However, I noticed that the issue is triggered when I start Firefox. I
> think Firefox starts writing something to some specific files and that
> triggers the issue because I was able to upgrade the kernel without
> getting the disk in RO mode.

When Btrfs becomes aware of its confusion, it'll go readonly to
prevent corrupting the file system. It might already have some
isolated corruption. It doesn't really tell us more than that.

>
> I can try to get a newer version of btrfs-progs, but what should I do
> with it? Should I check/scrub? I don't see how simply having a newer
> version of the btrfs-progs would prevent the kernel from throwing the
> error and putting the disk in read-only mode.

It won't change anything now. The version question is to establish
whether you have old or new style converted ext4. The call traces you
have come up in the list only with new style convert. But you have old
style. That's why I included Qu in the cc, I have no idea what the
problem is and if it's fixable with btrfs check from a recent
btrfs-progs or if it needs an even newer kernel, or if it's an unfixed
bug still.

In the meantime I would do this:

1. Backup what you care about. Decent chance the only way forward is
to create a new file system, so you might as well take advantage of
the fact at least you can mount it read only right now.

2. Boot some live media and use btrfs-image to capture an image of the
file system as it it. The newer progs the better, as they get a bit
more tolerant of file system errors and hopefully you can capture a
complete image. btrfs-image -c9 -t4 -s /dev/sdX filename.bin  \\ This
is metadata only, and will sanitize filenames.

3. Output from both of these
btrfs check <device>
btrfs check --mode=lowmem <device>   ##this will spit out more
problems if it's older than 4.10.2

Do this without --repair.

What seems safe to me, is while booted with live media, mount the
problem Btrfs file system, and find the Firefox cache,
~/.cache/mozilla/firefox/*.default and delete that whole default
directory. Now you can reboot the installed system, and see if the
problem still happens. If it does, you can try to reinstall Firefox.

If that doesn't help, it's a coin toss if it's worth trying btrfs
check --repair, or if you're better off taking a read-only snapshot
and using btrfs send/receive to a new Btrfs filesystem. The send
receive process is easier to use than rsync -a or cp -a, but things
get allocated natively rather than however convert did it.



-- 
Chris Murphy

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Chris Murphy]

Please reply to all. You keep dropping Qu and the Btrfs list off your responses.

On Fri, May 5, 2017 at 2:22 PM, Alexandru Guzu <alexguzu@gmail.com> wrote:
> I booted a Live CD with btrfs-progs 4.10.2 and ran a check on the
> partition, the regular btrfs check did not find any errors, but the
> lowmem one did find errors:
>
> btrfs check --mode lowmem /dev/sda1
> Checking filesystem on /dev/sda1
> UUID: bc8af368-8565-48d8-a16b-e2d25061f6ac
> checking extents
> checking free space cache
> checking fs roots
> ERROR: root 5 EXTENT_DATA[627 12288] csum missing, have: 0, expected: 4096
> ERROR: root 5 EXTENT_DATA[627 20480] csum missing, have: 0, expected: 45056
> ERROR: root 5 EXTENT_DATA[627 73728] csum missing, have: 0, expected: 4096
> ERROR: root 5 INODE[627] nbytes(2928640) not equal to extent_size(2981888)
> ERROR: root 5 EXTENT_DATA[5759 4096] csum missing, have: 0, expected: 4096
> ERROR: root 5 INODE[5759] nbytes(12288) not equal to extent_size(16384)
> ERROR: root 5 EXTENT_DATA[5760 4096] csum missing, have: 0, expected: 12288
> ERROR: root 5 INODE[5760] nbytes(53248) not equal to extent_size(65536)
> ERROR: root 5 EXTENT_DATA[284832 4096] csum missing, have: 0, expected: 24576
> ERROR: root 5 INODE[284832] nbytes(8192) not equal to extent_size(32768)
> ERROR: root 5 EXTENT_DATA[284834 36864] csum missing, have: 0, expected: 253952
> ERROR: root 5 INODE[284834] nbytes(40960) not equal to extent_size(294912)
> ERROR: root 5 EXTENT_DATA[839436 8192] interrupt
> ERROR: errors found in fs roots
> found 4971778048 bytes used, error(s) found
> total csum bytes: 4042156
> total tree bytes: 831029248
> total fs tree bytes: 815988736
> total extent tree bytes: 10010624
> btree space waste bytes: 221535838
> file data blocks allocated: 4330766336
>  referenced 5587816448
>
> I suppose I should attempt a repair?

No, the lowmem mode doesn't support repair yet. I'd expect normal mode
won't repair anything if it doesn't find problems.

>I have no important data on this drive.
> I haven't deleted the Firefox cache yet.
> btrfs scrub did not find any errors

I think you've found a bug in kernel handling, we'll just have to see
what Qu says.

I would go with the deleting of the Firefox cache first, and then if
necessary reinstall just Firefox. Are there other things that trigger
it?


-- 
Chris Murphy

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Sean Greenslade]

On May 3, 2017 4:28:11 PM EDT, Alexandru Guzu <alexguzu@gmail.com> wrote:
>Hi all,
>
>In a VirtualBox VM, I converted a EXT4 fs to BTRFS that is now running
>on Ubuntu 16.04 (Kernel 4.4.0-72). I was able to use the system for
>several weeks. I even did kernel updates, compression, deduplication
>without issues.
>
>Since today, a little while after booting (usually when I start
>opening applications), the FS becomes read-only. IMO, the only thing
>that might have changed that could affect this is a kernel upgrade.
>However, since the FS becomes RO, I cannot easily do a kernel update
>(I guess I'd have to chroot and do it).
>
>The stack trace is the following:
>
>[   88.836057] ------------[ cut here ]------------
>[   88.836082] WARNING: CPU: 0 PID: 25 at
>/build/linux-wXdoVv/linux-4.4.0/fs/btrfs/inode.c:2931
>btrfs_finish_ordered_io+0x63b/0x650 [btrfs]()
>[   88.836083] BTRFS: Transaction aborted (error -95)
>[   88.836084] Modules linked in: nvram msr zram lz4_compress
>vboxsf(OE) joydev crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
>aesni_intel snd_intel8x0 aes_x86_64 snd_ac97_codec lrw gf128mul
>glue_helper ablk_helper cryptd ac97_bus snd_pcm snd_seq_midi
>snd_seq_midi_event snd_rawmidi snd_seq vboxvideo(OE) snd_seq_device
>snd_timer ttm input_leds drm_kms_helper serio_raw i2c_piix4 snd drm
>fb_sys_fops syscopyarea sysfillrect sysimgblt soundcore 8250_fintek
>vboxguest(OE) mac_hid parport_pc ppdev lp parport autofs4 btrfs xor
>raid6_pq hid_generic usbhid hid psmouse ahci libahci fjes video
>pata_acpi
>[   88.836116] CPU: 0 PID: 25 Comm: kworker/u2:1 Tainted: G
>OE   4.4.0-72-generic #93-Ubuntu
>[   88.836117] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS
>VirtualBox 12/01/2006
>[   88.836130] Workqueue: btrfs-endio-write btrfs_endio_write_helper
>[btrfs]
>[   88.836132]  0000000000000286 00000000618d3a00 ffff88007cabbc78
>ffffffff813f82b3
>[   88.836134]  ffff88007cabbcc0 ffffffffc01912a8 ffff88007cabbcb0
>ffffffff81081302
>[   88.836135]  ffff880058bf01b0 ffff8800355f2800 ffff88007b9e64e0
>ffff88007c66f098
>[   88.836137] Call Trace:
>[   88.836142]  [<ffffffff813f82b3>] dump_stack+0x63/0x90
>[   88.836145]  [<ffffffff81081302>] warn_slowpath_common+0x82/0xc0
>[   88.836147]  [<ffffffff8108139c>] warn_slowpath_fmt+0x5c/0x80
>[   88.836159]  [<ffffffffc012373f>] ? unpin_extent_cache+0x8f/0xe0
>[btrfs]
>[   88.836167]  [<ffffffffc00e0b06>] ? btrfs_free_path+0x26/0x30
>[btrfs]
>[   88.836178]  [<ffffffffc011554b>]
>btrfs_finish_ordered_io+0x63b/0x650 [btrfs]
>[   88.836188]  [<ffffffffc0115845>] finish_ordered_fn+0x15/0x20
>[btrfs]
>[   88.836200]  [<ffffffffc013dfda>]
>btrfs_scrubparity_helper+0xca/0x2f0 [btrfs]
>[   88.836202]  [<ffffffff810caee1>] ?
>__raw_callee_save___pv_queued_spin_unlock+0x11/0x20
>[   88.836214]  [<ffffffffc013e28e>] btrfs_endio_write_helper+0xe/0x10
>[btrfs]
>[   88.836217]  [<ffffffff8109a555>] process_one_work+0x165/0x480
>[   88.836219]  [<ffffffff8109a8bb>] worker_thread+0x4b/0x4c0
>[   88.836220]  [<ffffffff8109a870>] ? process_one_work+0x480/0x480
>[   88.836222]  [<ffffffff8109a870>] ? process_one_work+0x480/0x480
>[   88.836224]  [<ffffffff810a0be8>] kthread+0xd8/0xf0
>[   88.836225]  [<ffffffff810a0b10>] ?
>kthread_create_on_node+0x1e0/0x1e0
>[   88.836229]  [<ffffffff8183ca0f>] ret_from_fork+0x3f/0x70
>[   88.836230]  [<ffffffff810a0b10>] ?
>kthread_create_on_node+0x1e0/0x1e0
>[   88.836232] ---[ end trace f4b8dbb54f0db139 ]---
>[   88.836234] BTRFS: error (device sda1) in
>btrfs_finish_ordered_io:2931: errno=-95 unknown
>[   88.836236] BTRFS info (device sda1): forced readonly
>[   88.836392] pending csums is 118784
>
>If I reboot with the Live CD version, I can run `scrub` and `check`
>without any issues. Also the FS stays read-write.
>
>Is this a known issue? Could it be due to the conversion, or can this
>happen on any system on any time. This is a test VM that I can
>re-make, but I would like to know if this can happen on a production
>system.
>
>Regards,
>Alex.
>--
>To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
>in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

Just FYI, I ran into basically this same issue a while back. 
My solution was to copy all the data off it,  re-run mkfs.btrfs,
then copy all the data back. I found that none of the existing
data was damaged, so I think the actual issue is something
left over from the conversion that confuses the commit
logic.

--Sean

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Alexandru Guzu]

Deleting the Firefox cache with the LiveCD resolved the issue where
the FS would become read-only when opening Firefox, however, the
lowmem check still reports errors. There are probably some other files
that would cause the FS to become read-only again, but I'm not sure
how to find them.
I took a snapshot of the Virtual disk before deleting the FF profile,
so I can reproduce the issue if you guys want to track down this
potential bug, otherwise I will delete the VM by the end of this week.

Sean, how would you approach the copy of the data back and forth if
the OS is on it? Would a Send-receive and then back work?

On Sun, May 7, 2017 at 1:32 PM, Sean Greenslade <sean@seangreenslade.com> wrote:
> On May 3, 2017 4:28:11 PM EDT, Alexandru Guzu <alexguzu@gmail.com> wrote:
>>Hi all,
>>
>>In a VirtualBox VM, I converted a EXT4 fs to BTRFS that is now running
>>on Ubuntu 16.04 (Kernel 4.4.0-72). I was able to use the system for
>>several weeks. I even did kernel updates, compression, deduplication
>>without issues.
>>
>>Since today, a little while after booting (usually when I start
>>opening applications), the FS becomes read-only. IMO, the only thing
>>that might have changed that could affect this is a kernel upgrade.
>>However, since the FS becomes RO, I cannot easily do a kernel update
>>(I guess I'd have to chroot and do it).
>>
>>The stack trace is the following:
>>
>>[   88.836057] ------------[ cut here ]------------
>>[   88.836082] WARNING: CPU: 0 PID: 25 at
>>/build/linux-wXdoVv/linux-4.4.0/fs/btrfs/inode.c:2931
>>btrfs_finish_ordered_io+0x63b/0x650 [btrfs]()
>>[   88.836083] BTRFS: Transaction aborted (error -95)
>>[   88.836084] Modules linked in: nvram msr zram lz4_compress
>>vboxsf(OE) joydev crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
>>aesni_intel snd_intel8x0 aes_x86_64 snd_ac97_codec lrw gf128mul
>>glue_helper ablk_helper cryptd ac97_bus snd_pcm snd_seq_midi
>>snd_seq_midi_event snd_rawmidi snd_seq vboxvideo(OE) snd_seq_device
>>snd_timer ttm input_leds drm_kms_helper serio_raw i2c_piix4 snd drm
>>fb_sys_fops syscopyarea sysfillrect sysimgblt soundcore 8250_fintek
>>vboxguest(OE) mac_hid parport_pc ppdev lp parport autofs4 btrfs xor
>>raid6_pq hid_generic usbhid hid psmouse ahci libahci fjes video
>>pata_acpi
>>[   88.836116] CPU: 0 PID: 25 Comm: kworker/u2:1 Tainted: G
>>OE   4.4.0-72-generic #93-Ubuntu
>>[   88.836117] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS
>>VirtualBox 12/01/2006
>>[   88.836130] Workqueue: btrfs-endio-write btrfs_endio_write_helper
>>[btrfs]
>>[   88.836132]  0000000000000286 00000000618d3a00 ffff88007cabbc78
>>ffffffff813f82b3
>>[   88.836134]  ffff88007cabbcc0 ffffffffc01912a8 ffff88007cabbcb0
>>ffffffff81081302
>>[   88.836135]  ffff880058bf01b0 ffff8800355f2800 ffff88007b9e64e0
>>ffff88007c66f098
>>[   88.836137] Call Trace:
>>[   88.836142]  [<ffffffff813f82b3>] dump_stack+0x63/0x90
>>[   88.836145]  [<ffffffff81081302>] warn_slowpath_common+0x82/0xc0
>>[   88.836147]  [<ffffffff8108139c>] warn_slowpath_fmt+0x5c/0x80
>>[   88.836159]  [<ffffffffc012373f>] ? unpin_extent_cache+0x8f/0xe0
>>[btrfs]
>>[   88.836167]  [<ffffffffc00e0b06>] ? btrfs_free_path+0x26/0x30
>>[btrfs]
>>[   88.836178]  [<ffffffffc011554b>]
>>btrfs_finish_ordered_io+0x63b/0x650 [btrfs]
>>[   88.836188]  [<ffffffffc0115845>] finish_ordered_fn+0x15/0x20
>>[btrfs]
>>[   88.836200]  [<ffffffffc013dfda>]
>>btrfs_scrubparity_helper+0xca/0x2f0 [btrfs]
>>[   88.836202]  [<ffffffff810caee1>] ?
>>__raw_callee_save___pv_queued_spin_unlock+0x11/0x20
>>[   88.836214]  [<ffffffffc013e28e>] btrfs_endio_write_helper+0xe/0x10
>>[btrfs]
>>[   88.836217]  [<ffffffff8109a555>] process_one_work+0x165/0x480
>>[   88.836219]  [<ffffffff8109a8bb>] worker_thread+0x4b/0x4c0
>>[   88.836220]  [<ffffffff8109a870>] ? process_one_work+0x480/0x480
>>[   88.836222]  [<ffffffff8109a870>] ? process_one_work+0x480/0x480
>>[   88.836224]  [<ffffffff810a0be8>] kthread+0xd8/0xf0
>>[   88.836225]  [<ffffffff810a0b10>] ?
>>kthread_create_on_node+0x1e0/0x1e0
>>[   88.836229]  [<ffffffff8183ca0f>] ret_from_fork+0x3f/0x70
>>[   88.836230]  [<ffffffff810a0b10>] ?
>>kthread_create_on_node+0x1e0/0x1e0
>>[   88.836232] ---[ end trace f4b8dbb54f0db139 ]---
>>[   88.836234] BTRFS: error (device sda1) in
>>btrfs_finish_ordered_io:2931: errno=-95 unknown
>>[   88.836236] BTRFS info (device sda1): forced readonly
>>[   88.836392] pending csums is 118784
>>
>>If I reboot with the Live CD version, I can run `scrub` and `check`
>>without any issues. Also the FS stays read-write.
>>
>>Is this a known issue? Could it be due to the conversion, or can this
>>happen on any system on any time. This is a test VM that I can
>>re-make, but I would like to know if this can happen on a production
>>system.
>>
>>Regards,
>>Alex.
>>--
>>To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
>>in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> Just FYI, I ran into basically this same issue a while back.
> My solution was to copy all the data off it,  re-run mkfs.btrfs,
> then copy all the data back. I found that none of the existing
> data was damaged, so I think the actual issue is something
> left over from the conversion that confuses the commit
> logic.
>
> --Sean

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Sanidhya Solanki]

On Mon, 8 May 2017 10:16:44 -0400
Alexandru Guzu <alexguzu@gmail.com> wrote:
 
> Sean, how would you approach the copy of the data back and forth if
> the OS is on it? Would a Send-receive and then back work?

You could use a Live-USB and then just dd it to remote or attached storage, if
you want to be absolutely sure the data is not affected.

SS

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Sean Greenslade]

On May 8, 2017 11:28:42 AM EDT, Sanidhya Solanki <lkml.page@gmail.com> wrote:
>On Mon, 8 May 2017 10:16:44 -0400
>Alexandru Guzu <alexguzu@gmail.com> wrote:
> 
>> Sean, how would you approach the copy of the data back and forth if
>> the OS is on it? Would a Send-receive and then back work?
>
>You could use a Live-USB and then just dd it to remote or attached
>storage, if
>you want to be absolutely sure the data is not affected.

I would not suggest either of those. Send / receive might work, but since we don't know the source of the problem, it risks transferring the problem. DD would not solve the problem at all, since we're trying to rebuild the partition, not clone it.

--Sean

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Austin S. Hemmelgarn]

On 2017-05-08 12:22, Sean Greenslade wrote:
> On May 8, 2017 11:28:42 AM EDT, Sanidhya Solanki <lkml.page@gmail.com> wrote:
>> On Mon, 8 May 2017 10:16:44 -0400
>> Alexandru Guzu <alexguzu@gmail.com> wrote:
>>
>>> Sean, how would you approach the copy of the data back and forth if
>>> the OS is on it? Would a Send-receive and then back work?
>>
>> You could use a Live-USB and then just dd it to remote or attached
>> storage, if
>> you want to be absolutely sure the data is not affected.
>
> I would not suggest either of those. Send / receive might work, but since we don't know the source of the problem, it risks transferring the problem. DD would not solve the problem at all, since we're trying to rebuild the partition, not clone it.
Send/receive is not likely to transfer the problem unless it has 
something to do with how things are reflinked.  Receive operates by 
recreating the sent subvolume from userspace using regular commands and 
the clone ioctls, so it won't replicate any low-level structural issues 
in the filesystem unless they directly involve the way extents are being 
shared (or are a side effect of that).  On top of that, if there is an 
issue on the sending side, send itself will probably not send that data, 
so it's actually only marginally more dangerous than using something 
like rsync to copy the data.

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Sean Greenslade]

On Mon, May 08, 2017 at 12:41:11PM -0400, Austin S. Hemmelgarn wrote:
> Send/receive is not likely to transfer the problem unless it has something
> to do with how things are reflinked.  Receive operates by recreating the
> sent subvolume from userspace using regular commands and the clone ioctls,
> so it won't replicate any low-level structural issues in the filesystem
> unless they directly involve the way extents are being shared (or are a side
> effect of that).  On top of that, if there is an issue on the sending side,
> send itself will probably not send that data, so it's actually only
> marginally more dangerous than using something like rsync to copy the data.

True, but my goal was to eliminate as many btrfs variables as I could.
To answer the original question, I used rsync to copy the data and
attributes (something like rsync -aHXp --numeric-ids) from a live CD to
an external hard drive (formatted ext4), then ran mkfs.btrfs on the
original partition, then re-ran the rsync in the opposite direction. It
worked quite well for me, and the problem hasn't resurfaced.

--Sean

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Alexandru Guzu]

Getting a bit off-topic here, but were you able to boot from that fs
with grub after a simple rsync?

On Mon, May 8, 2017 at 1:01 PM, Sean Greenslade <sean@seangreenslade.com> wrote:
> On Mon, May 08, 2017 at 12:41:11PM -0400, Austin S. Hemmelgarn wrote:
>> Send/receive is not likely to transfer the problem unless it has something
>> to do with how things are reflinked.  Receive operates by recreating the
>> sent subvolume from userspace using regular commands and the clone ioctls,
>> so it won't replicate any low-level structural issues in the filesystem
>> unless they directly involve the way extents are being shared (or are a side
>> effect of that).  On top of that, if there is an issue on the sending side,
>> send itself will probably not send that data, so it's actually only
>> marginally more dangerous than using something like rsync to copy the data.
>
> True, but my goal was to eliminate as many btrfs variables as I could.
> To answer the original question, I used rsync to copy the data and
> attributes (something like rsync -aHXp --numeric-ids) from a live CD to
> an external hard drive (formatted ext4), then ran mkfs.btrfs on the
> original partition, then re-ran the rsync in the opposite direction. It
> worked quite well for me, and the problem hasn't resurfaced.
>
> --Sean
>

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Sean Greenslade]

On Mon, May 08, 2017 at 02:22:52PM -0400, Alexandru Guzu wrote:
> Getting a bit off-topic here, but were you able to boot from that fs
> with grub after a simple rsync?

Grub was installed to the MBR, and /boot was a separate partition, so I
didn't have to re-install grub. I did have to edit the root partition
UUID in /etc/fstab and in the kernel command line in grub.cfg, though.

--Sean

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Chris Murphy]

On Mon, May 8, 2017 at 8:16 AM, Alexandru Guzu <alexguzu@gmail.com> wrote:
> Deleting the Firefox cache with the LiveCD resolved the issue where
> the FS would become read-only when opening Firefox, however, the
> lowmem check still reports errors. There are probably some other files
> that would cause the FS to become read-only again, but I'm not sure
> how to find them.
> I took a snapshot of the Virtual disk before deleting the FF profile,
> so I can reproduce the issue if you guys want to track down this
> potential bug, otherwise I will delete the VM by the end of this week.
>
> Sean, how would you approach the copy of the data back and forth if
> the OS is on it? Would a Send-receive and then back work?

Btrfs send receive. Optionally you can send to a file; and then
receive from a file. In that case the file can be on any filesystem.

The one thing not to use is dd.


-- 
Chris Murphy

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Chris Murphy]

On Mon, May 8, 2017 at 12:22 PM, Alexandru Guzu <alexguzu@gmail.com> wrote:
> Getting a bit off-topic here, but were you able to boot from that fs
> with grub after a simple rsync?

I've done it with rsync as well as send/receive and both result in
bootable systems. I use rsync -pogAXtlHrDx because that's what
Fedora's installer is using when installing live systems, regardless
of file system. Of course you still have to fix up some things
afterthefact, like installing fixing fstab to have the correct volume
UUIDs, installing bootloader and making a new grub.cfg, and also
rebuild the initramfs.

-- 
Chris Murphy

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Alexandru Guzu]

My setup is a single disk, single btrfs subvolume (/) without a
separate boot partition.
I resolved the issue like this from a Live CD:

1 - take a (read-only) snapshot of the current problematic root system
and use send-receive to another BTRFS disk
2 - re-format the original partition as btrfs (mkfs.btrfs is enought)
and mount it under /mnt (I added -o compress=lzo)
3 - cp -arv  the files from the other disk subovlume to the newly
formatted partition, /mnt
4 - chroot to the new partition:
    # cd /mnt
    # mount --bind /sys sys/
    # mount --bind /dev dev/
    # mount --bind /proc proc/
    # chroot /mnt /bin/bash
5 - re-install grub:
    # update-grub
    # grub-install
6 - edited the /mnt/etc/fstab entry to match the new blkid /dev/sda1
7 - reboot magically brought me to the original system.
8 - reboot again in LiveCD, and using btrfs-tools 4.10.2 I ran a
lowmem check without errors.

So the bottom line is that send-receive did not carry over the
problems and that cp -arv is good enough to clone the partition, but
you need to manually install the bootloader. Maybe Boot-repair could
have done it, not sure.
Also I was unable to find a way to send-receive a snapshot to the root
subvolume of a different disk. Is this possible?

Thanks for the help and may this solution remain on the internet for
anyone that has a similar issue.

On Mon, May 8, 2017 at 3:02 PM, Sean Greenslade <sean@seangreenslade.com> wrote:
> On Mon, May 08, 2017 at 02:22:52PM -0400, Alexandru Guzu wrote:
>> Getting a bit off-topic here, but were you able to boot from that fs
>> with grub after a simple rsync?
>
> Grub was installed to the MBR, and /boot was a separate partition, so I
> didn't have to re-install grub. I did have to edit the root partition
> UUID in /etc/fstab and in the kernel command line in grub.cfg, though.
>
> --Sean
>

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Chris Murphy]

On Mon, May 8, 2017 at 1:15 PM, Alexandru Guzu <alexguzu@gmail.com> wrote:
> Also I was unable to find a way to send-receive a snapshot to the root
> subvolume of a different disk. Is this possible?


btrfs send /mnt/brick1/root.20170508 | btrfs receive /mnt/brick2

You need to snapshot it again to make it a read-write snapshot, and
then move the contents wherever you want, if you do not want to boot a
subvolume directly; and rather boot from the top level subvolume.

Each subvolume is a separate fs tree, so you need to have the top
level mounted if you want to do an efficient move which is really a cp
--reflink followed by rm, behind the scenes. I find it easy to just
boot subvolumes rather than booting from the top level (subvolume id
5, a.k.a. id 0), using rootflags=subvol=<nameofsubvol>.


-- 
Chris Murphy

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Marc MERLIN]

On Thu, May 04, 2017 at 03:55:28AM +0000, Duncan wrote:
> > But that alone may not fix it, I think you need a newer kernel...
> 
> Well, while the 4.4 LTS kernel series /is/ getting a bit long in the 
> tooth by now, it's still the second newest LTS series available, 4.9 
> being the newest.
> 
> And on-list we've long recommended staying within the latest two series 
> in either the LTS or current kernel series, which means the 4.4 series 
> should still be reasonably supported.  

For what it's worth, I also had an ext4 filesystem created by some
debian testing install, tried to convert it, it worked, I rebooted, then
it worked once and got corrupted and in the end I had to destroy it and
convert by hand.

I've tried an ext4 to btrfs conversion 3 times in the last 3 years, it
never worked properly any of those times, sadly.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Chris Murphy]

On Tue, May 23, 2017 at 11:00 AM, Marc MERLIN <marc@merlins.org> wrote:
> On Thu, May 04, 2017 at 03:55:28AM +0000, Duncan wrote:
>> > But that alone may not fix it, I think you need a newer kernel...
>>
>> Well, while the 4.4 LTS kernel series /is/ getting a bit long in the
>> tooth by now, it's still the second newest LTS series available, 4.9
>> being the newest.
>>
>> And on-list we've long recommended staying within the latest two series
>> in either the LTS or current kernel series, which means the 4.4 series
>> should still be reasonably supported.
>
> For what it's worth, I also had an ext4 filesystem created by some
> debian testing install, tried to convert it, it worked, I rebooted, then
> it worked once and got corrupted and in the end I had to destroy it and
> convert by hand.
>
> I've tried an ext4 to btrfs conversion 3 times in the last 3 years, it
> never worked properly any of those times, sadly.

Since the 4.6 total rewrite? There are also recent bug fixes related
to convert in the changelog, it should be working now and if there are
problems Qu is probably interested in getting them fixed.

-- 
Chris Murphy

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Marc MERLIN]

On Tue, May 23, 2017 at 03:38:01PM -0600, Chris Murphy wrote:
> > I've tried an ext4 to btrfs conversion 3 times in the last 3 years, it
> > never worked properly any of those times, sadly.
> 
> Since the 4.6 total rewrite? There are also recent bug fixes related
> to convert in the changelog, it should be working now and if there are
> problems Qu is probably interested in getting them fixed.

It was a 4.9 kernel from debian.
The conversion looked like it worked, I rebooted ok, and then it got
corrupted quickly after I deleted the subvolumes that had the old ext4 data.
I've since wiped that disk and done a fresh btrfs install on it, because I
had to get some work done :)

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Chris Murphy]

On Tue, May 23, 2017 at 3:49 PM, Marc MERLIN <marc@merlins.org> wrote:
> On Tue, May 23, 2017 at 03:38:01PM -0600, Chris Murphy wrote:
>> > I've tried an ext4 to btrfs conversion 3 times in the last 3 years, it
>> > never worked properly any of those times, sadly.
>>
>> Since the 4.6 total rewrite? There are also recent bug fixes related
>> to convert in the changelog, it should be working now and if there are
>> problems Qu is probably interested in getting them fixed.
>
> It was a 4.9 kernel from debian.

Convert is done by user space code, so btrfs-progs version is relevant.



-- 
Chris Murphy

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1066 bytes --]

On Tue, May 23, 2017 at 02:49:43PM -0700, Marc MERLIN wrote:
> On Tue, May 23, 2017 at 03:38:01PM -0600, Chris Murphy wrote:
> > > I've tried an ext4 to btrfs conversion 3 times in the last 3 years, it
> > > never worked properly any of those times, sadly.
> > 
> > Since the 4.6 total rewrite? There are also recent bug fixes related
> > to convert in the changelog, it should be working now and if there are
> > problems Qu is probably interested in getting them fixed.
> 
> It was a 4.9 kernel from debian.

   It's the userspace tools that make the difference here (and what
Chris was referring to). Conversion has nothing to do with the kernel.

   Hugo.

> The conversion looked like it worked, I rebooted ok, and then it got
> corrupted quickly after I deleted the subvolumes that had the old ext4 data.
> I've since wiped that disk and done a fresh btrfs install on it, because I
> had to get some work done :)
> 
> Marc

-- 
Hugo Mills             | Essex: a branch of philothophy.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Marc MERLIN]

On Tue, May 23, 2017 at 03:51:43PM -0600, Chris Murphy wrote:
> On Tue, May 23, 2017 at 3:49 PM, Marc MERLIN <marc@merlins.org> wrote:
> > On Tue, May 23, 2017 at 03:38:01PM -0600, Chris Murphy wrote:
> >> > I've tried an ext4 to btrfs conversion 3 times in the last 3 years, it
> >> > never worked properly any of those times, sadly.
> >>
> >> Since the 4.6 total rewrite? There are also recent bug fixes related
> >> to convert in the changelog, it should be working now and if there are
> >> problems Qu is probably interested in getting them fixed.
> >
> > It was a 4.9 kernel from debian.
> 
> Convert is done by user space code, so btrfs-progs version is relevant.

Sigh, I think you found the issue. That system only had btrfs-tools 4.4 :(

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Marc MERLIN]

On Tue, May 23, 2017 at 02:53:21PM -0700, Marc MERLIN wrote:
> On Tue, May 23, 2017 at 03:51:43PM -0600, Chris Murphy wrote:
> > On Tue, May 23, 2017 at 3:49 PM, Marc MERLIN <marc@merlins.org> wrote:
> > > On Tue, May 23, 2017 at 03:38:01PM -0600, Chris Murphy wrote:
> > >> > I've tried an ext4 to btrfs conversion 3 times in the last 3 years, it
> > >> > never worked properly any of those times, sadly.
> > >>
> > >> Since the 4.6 total rewrite? There are also recent bug fixes related
> > >> to convert in the changelog, it should be working now and if there are
> > >> problems Qu is probably interested in getting them fixed.
> > >
> > > It was a 4.9 kernel from debian.
> > 
> > Convert is done by user space code, so btrfs-progs version is relevant.
> 
> Sigh, I think you found the issue. That system only had btrfs-tools 4.4 :(

On the plus side, it's good news for btrfs, that means the next time I may
need/try this again, it should actually work :)
(so thanks for fixing it)

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: BTRFS converted from EXT4 becomes read-only after reboot

[Andrei Borzenkov]

24.05.2017 00:49, Marc MERLIN пишет:
> On Tue, May 23, 2017 at 03:38:01PM -0600, Chris Murphy wrote:
>>> I've tried an ext4 to btrfs conversion 3 times in the last 3 years, it
>>> never worked properly any of those times, sadly.
>>
>> Since the 4.6 total rewrite? There are also recent bug fixes related
>> to convert in the changelog, it should be working now and if there are
>> problems Qu is probably interested in getting them fixed.
> 
> It was a 4.9 kernel from debian.

btrfs-convert is actually user space, so the question is what btrfsprogs
version you used.