Re: Trying to write to read-only block-device from XFS

Re: Trying to write to read-only block-device from XFS

[Andi Kleen]

Adding linux-fsdevel. Keeping full quote for context.

On Mon, Aug 27, 2018 at 10:12:54AM +0200, Carlos Maiolino wrote:
> Hi.
> 
> On Sat, Aug 25, 2018 at 09:25:33PM -0500, Eric Sandeen wrote:
> > 
> > 
> > On 8/25/18 9:09 AM, Andi Kleen wrote:
> > > On Fri, Aug 24, 2018 at 07:58:05PM -0500, Eric Sandeen wrote:
> > >> On 8/24/18 6:37 PM, Andi Kleen wrote:
> > >>>
> > >>> Seen this a few times now with recent kernels:
> > >>
> > >> Hi Andi -
> > >>
> > >> Dumb question maybe, but just to be clear -
> > >> Was this an RO mount on an RO device, or something else?
> > >> What type of mount was this?
> > > 
> > > Was just a normal RW partition on a normal SSD. I didn't do anything
> > > special to make it RO.
> > 
> > I don't know why it shows up as RO at this point, but maybe this fixes it:
> > 
> > commit b089cfd95d32638335c551651a8e00fd2c4edb0b
> > Author: Jens Axboe <axboe@kernel.dk>
> > Date:   Tue Aug 14 10:52:40 2018 -0600
> > 
> >     block: don't warn for flush on read-only device
> >     
> >     Don't warn for a flush issued to a read-only device. It's not strictly
> >     a writable command, as it doesn't change any on-media data by itself.
> >     
> >     Reported-by: Stefan Agner <stefan@agner.ch>
> >     Fixes: 721c7fc701c7 ("block: fail op_is_write() requests to read-only partitions")
> >     Signed-off-by: Jens Axboe <axboe@kernel.dk>
> 
> This kind of 'problem' is being reported quite often now. It all started with
> commit:
> 
> commit 721c7fc701c71f693307d274d2b346a1ecd4a534
> Author: Ilya Dryomov <idryomov@gmail.com>
> Date:   Thu Jan 11 14:09:11 2018 +0100
> 
>     block: fail op_is_write() requests to read-only partitions
> 
> Which basically added a generic check for RO devices into generic_make_request()
> code path, as a way to enforce ioctl(BLKROSET), so far, write requests were
> never checked.
> 
> 
> Then, Linus applied the following commit:
> 
> commit a32e236eb93e62a0f692e79b7c3c9636689559b9
> Author: Linus Torvalds <torvalds@linux-foundation.org>
> Date:   Fri Aug 3 12:22:09 2018 -0700
> 
>     Partially revert "block: fail op_is_write() requests to read-only partitions"
> 
> Once a few layers like LVM, didn't expect the device to enforce RO, which kept
> writing to the device while snapshotting, even if the device has been marked as
> RO.
> 
> And then came the commit mentioned by Eric:
> 
> commit b089cfd95d32638335c551651a8e00fd2c4edb0b
> Author: Jens Axboe <axboe@kernel.dk>
> Date:   Tue Aug 14 10:52:40 2018 -0600
> 
>     block: don't warn for flush on read-only device
> 
> 
> Which relaxed the rules a bit and basically let flushes to proceed without warnings.
> 
> Essentially, this isn't a XFS problem, and most likely you are using XFS on top
> of some layer like LVM, and/or using snapshots, and your underlying volume went
> read-only behind XFS for some reason.

Ok, so the blame is assigned, but the question is still how to avoid the
warning:

(1) either XFS needs to check for read only underlying more often
or
(2) the warning needs to be removed.

I assume (2) is far easier. Is (1) even possible without races? 
What is preferred? 


-Andi

Re: Trying to write to read-only block-device from XFS

[Eric Sandeen]

On 8/27/18 11:43 AM, Andi Kleen wrote:
> 
> Adding linux-fsdevel. Keeping full quote for context.
> 
> On Mon, Aug 27, 2018 at 10:12:54AM +0200, Carlos Maiolino wrote:
>> Hi.
>>
>> On Sat, Aug 25, 2018 at 09:25:33PM -0500, Eric Sandeen wrote:
>>>
>>>
>>> On 8/25/18 9:09 AM, Andi Kleen wrote:
>>>> On Fri, Aug 24, 2018 at 07:58:05PM -0500, Eric Sandeen wrote:
>>>>> On 8/24/18 6:37 PM, Andi Kleen wrote:
>>>>>>
>>>>>> Seen this a few times now with recent kernels:
>>>>>
>>>>> Hi Andi -
>>>>>
>>>>> Dumb question maybe, but just to be clear -
>>>>> Was this an RO mount on an RO device, or something else?
>>>>> What type of mount was this?
>>>>
>>>> Was just a normal RW partition on a normal SSD. I didn't do anything
>>>> special to make it RO.
>>>
>>> I don't know why it shows up as RO at this point, but maybe this fixes it:
>>>
>>> commit b089cfd95d32638335c551651a8e00fd2c4edb0b
>>> Author: Jens Axboe <axboe@kernel.dk>
>>> Date:   Tue Aug 14 10:52:40 2018 -0600
>>>
>>>     block: don't warn for flush on read-only device
>>>     
>>>     Don't warn for a flush issued to a read-only device. It's not strictly
>>>     a writable command, as it doesn't change any on-media data by itself.
>>>     
>>>     Reported-by: Stefan Agner <stefan@agner.ch>
>>>     Fixes: 721c7fc701c7 ("block: fail op_is_write() requests to read-only partitions")
>>>     Signed-off-by: Jens Axboe <axboe@kernel.dk>
>>
>> This kind of 'problem' is being reported quite often now. It all started with
>> commit:
>>
>> commit 721c7fc701c71f693307d274d2b346a1ecd4a534
>> Author: Ilya Dryomov <idryomov@gmail.com>
>> Date:   Thu Jan 11 14:09:11 2018 +0100
>>
>>     block: fail op_is_write() requests to read-only partitions
>>
>> Which basically added a generic check for RO devices into generic_make_request()
>> code path, as a way to enforce ioctl(BLKROSET), so far, write requests were
>> never checked.
>>
>>
>> Then, Linus applied the following commit:
>>
>> commit a32e236eb93e62a0f692e79b7c3c9636689559b9
>> Author: Linus Torvalds <torvalds@linux-foundation.org>
>> Date:   Fri Aug 3 12:22:09 2018 -0700
>>
>>     Partially revert "block: fail op_is_write() requests to read-only partitions"
>>
>> Once a few layers like LVM, didn't expect the device to enforce RO, which kept
>> writing to the device while snapshotting, even if the device has been marked as
>> RO.
>>
>> And then came the commit mentioned by Eric:
>>
>> commit b089cfd95d32638335c551651a8e00fd2c4edb0b
>> Author: Jens Axboe <axboe@kernel.dk>
>> Date:   Tue Aug 14 10:52:40 2018 -0600
>>
>>     block: don't warn for flush on read-only device
>>
>>
>> Which relaxed the rules a bit and basically let flushes to proceed without warnings.
>>
>> Essentially, this isn't a XFS problem, and most likely you are using XFS on top
>> of some layer like LVM, and/or using snapshots, and your underlying volume went
>> read-only behind XFS for some reason.
> 
> Ok, so the blame is assigned, but the question is still how to avoid the
> warning:
> 
> (1) either XFS needs to check for read only underlying more often
> or
> (2) the warning needs to be removed.
> 
> I assume (2) is far easier. Is (1) even possible without races? 
> What is preferred? 

Seems like the first question to answer is: why is your "plain RW partition
on an SSD" suddenly reporting itself as RO during umount?

That said, I don't see a reason for xfs to continue to check for an RO partition
during operation, if for some unexpected reason a block device starts returning
EIO due to a device going readonly, normal error handling should kick in.

-Eric

Re: Trying to write to read-only block-device from XFS

[Dave Chinner]

On Mon, Aug 27, 2018 at 09:43:43AM -0700, Andi Kleen wrote:
> 
> Adding linux-fsdevel. Keeping full quote for context.
> 
> On Mon, Aug 27, 2018 at 10:12:54AM +0200, Carlos Maiolino wrote:
> > Essentially, this isn't a XFS problem, and most likely you are using XFS on top
> > of some layer like LVM, and/or using snapshots, and your underlying volume went
> > read-only behind XFS for some reason.
> 
> Ok, so the blame is assigned, but the question is still how to avoid the
> warning:
> 
> (1) either XFS needs to check for read only underlying more often
> or
> (2) the warning needs to be removed.
> 
> I assume (2) is far easier. Is (1) even possible without races? 

I doubt (1) is possible to handle at all.

e.g. What's the filesystem supposed to do when the device goes RO
under it without warning? We've now got a heap of dirty state in
memory that we can't write, so in reality if this ever happens our
only option is to shutdown the filesystem, toss everything away and
unmount it.

And because the device is now read only, we can't get the filesystem
back into a consistent state by mounting it again because it can't
run log recovery as that requires writing to the device.

IOWs, having devices turn read only underneath active read/write
filesystems without warning is an unrecoverable failure from the
filesystem perspective.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Trying to write to read-only block-device from XFS

[Carlos Maiolino]

> 
> Ok, so the blame is assigned, but the question is still how to avoid the
> warning:

I'd say, unless you are using a volume manager that is doing something weird,
which essentially LVM did by keep writing to RO volumes for snapshots, you
should avoid the warning by checking why your device went into RO mode and
fix it.

Block devices shouldn't suddenly enter in RO mode without facing an error
condition or somebody forcing it into RO mode.

> 
> (1) either XFS needs to check for read only underlying more often
> or
> (2) the warning needs to be removed.
> 
> I assume (2) is far easier. Is (1) even possible without races? 
> What is preferred? 
> 

> 
> -Andi

-- 
Carlos

Re: Trying to write to read-only block-device from XFS

[Andi Kleen]

On Tue, Aug 28, 2018 at 09:54:20AM +0200, Carlos Maiolino wrote:
> > 
> > Ok, so the blame is assigned, but the question is still how to avoid the
> > warning:
> 
> I'd say, unless you are using a volume manager that is doing something weird,
> which essentially LVM did by keep writing to RO volumes for snapshots, you

I don't use any snapshots.

> should avoid the warning by checking why your device went into RO mode and
> fix it.

You seem to be very confused about the purpose of kernel warnings.

Users should never be able to cause warnings by any action. 
It's always a software bug of some sort.

-Andi

Re: Trying to write to read-only block-device from XFS

[Dave Chinner]

On Tue, Aug 28, 2018 at 07:40:25PM -0700, Andi Kleen wrote:
> On Tue, Aug 28, 2018 at 09:54:20AM +0200, Carlos Maiolino wrote:
> > > 
> > > Ok, so the blame is assigned, but the question is still how to avoid the
> > > warning:
> > 
> > I'd say, unless you are using a volume manager that is doing something weird,
> > which essentially LVM did by keep writing to RO volumes for snapshots, you
> 
> I don't use any snapshots.
> 
> > should avoid the warning by checking why your device went into RO mode and
> > fix it.
> 
> You seem to be very confused about the purpose of kernel warnings.
> 
> Users should never be able to cause warnings by any action. 
> It's always a software bug of some sort.

I don't know why you're trying to pick a fight with Carlos, Andi.

Neither the warning nor whatever is triggering the device to go RO
has anything to do with XFS, so taking potshots the messenger and
those trying to help you is out of line. We're just trying to get
enough information to be able to point you at the right people to
get your issue fixed.

To do that, there's /one/ question we need answered. A question both
Eric and Carlos have already asked you: how is the block device
being changed to RO while a RW filesystem is mounted on it?

FYI, the warning is /trivial/ to provoke manually by turning a block
device read only under an active filesystem with the blockdev
command:

root@test4~# blockdev --setro /dev/vdc
[515320.769159] ------------[ cut here ]------------
[515320.770078] generic_make_request: Trying to write to read-only block-device vdc (partno 0)
[515320.771611] WARNING: CPU: 11 PID: 8173 at block/blk-core.c:2171 generic_make_request_checks+0x308/0x4a0
[515320.773349] CPU: 11 PID: 8173 Comm: xfs_io Not tainted 4.18.0-dgc+ #648
[515320.774577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.1-1 04/01/2014
[515320.776122] RIP: 0010:generic_make_request_checks+0x308/0x4a0
[515320.777187] Code: 6c 03 00 00 48 8d 74 24 08 48 89 df c6 05 6c de fa 00 01 e8 ea 91 01 00 48 c7 c7 10 90 30 82 48 89 c6 44 89 ea e8 28 3d 9c ff <0f> 0b 48 8b 43 08 e9 b5 fd ff ff 48 8b 53 28 49 03 3
[515320.780642] RSP: 0018:ffffc90008e0faf8 EFLAGS: 00010282
[515320.781617] RAX: 0000000000000000 RBX: ffff880829db8180 RCX: 0000000000000006
[515320.782934] RDX: 0000000000000007 RSI: 0000000000000082 RDI: ffff88083fd15550
[515320.784272] RBP: ffff88023fab2000 R08: ffffffff8183f550 R09: 000000000000004e
[515320.785595] R10: ffffc90008e0fbc8 R11: ffffffff82e5c9ee R12: 0000000000001000
[515320.786910] R13: 0000000000000000 R14: ffffc90008e0fc90 R15: ffff880828cba180
[515320.788241] FS:  00007f6beac60840(0000) GS:ffff88083fd00000(0000) knlGS:0000000000000000
[515320.789738] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[515320.790802] CR2: 0000559f72e98000 CR3: 0000000829f2f005 CR4: 0000000000060ee0
[515320.792124] Call Trace:
[515320.792649]  ? blk_queue_enter+0x212/0x240
[515320.793423]  generic_make_request+0x78/0x450
[515320.794231]  ? iov_iter_get_pages+0xbe/0x2a0
[515320.795035]  ? submit_bio+0x6d/0x120
[515320.795714]  submit_bio+0x6d/0x120
[515320.796403]  iomap_dio_bio_actor+0x1b5/0x3a0
[515320.797218]  ? iomap_page_release.part.29+0x40/0x40
[515320.798136]  iomap_apply+0xb0/0x130
[515320.798803]  iomap_dio_rw+0x2a6/0x3c0
[515320.799504]  ? iomap_page_release.part.29+0x40/0x40
[515320.800458]  ? xfs_file_dio_aio_write+0x117/0x2e0
[515320.801345]  xfs_file_dio_aio_write+0x117/0x2e0
[515320.802205]  xfs_file_write_iter+0x83/0xb0
[515320.802988]  __vfs_write+0x109/0x190
[515320.803695]  vfs_write+0xb6/0x180
[515320.804535]  ksys_pwrite64+0x71/0x90
[515320.805212]  do_syscall_64+0x5a/0x180
[515320.805910]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
[515320.806852] RIP: 0033:0x7f6bebd5e024

That doesn't happen on a normal system under normal operation,
though, so we really need to know where it is coming from.  It's
likely something in userspace on your system is doing it but until
we know the source it is not clear where the incorrect behaviour
lies and what may need fixing.

So, can you please find what is changing the block device mode to RO
for us?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Trying to write to read-only block-device from XFS

[Carlos Maiolino]

On Wed, Aug 29, 2018 at 02:19:57PM +1000, Dave Chinner wrote:
> On Tue, Aug 28, 2018 at 07:40:25PM -0700, Andi Kleen wrote:
> > On Tue, Aug 28, 2018 at 09:54:20AM +0200, Carlos Maiolino wrote:
> > > > 
> > > > Ok, so the blame is assigned, but the question is still how to avoid the
> > > > warning:
> > > 
> > > I'd say, unless you are using a volume manager that is doing something weird,
> > > which essentially LVM did by keep writing to RO volumes for snapshots, you
> > 
> > I don't use any snapshots.
> > 
> > > should avoid the warning by checking why your device went into RO mode and
> > > fix it.
> > 
> > You seem to be very confused about the purpose of kernel warnings.
> > 
> > Users should never be able to cause warnings by any action. 
> > It's always a software bug of some sort.

The reason I asked you about snapshots is because the first problems reported
after the patches I mentioned previously were from LVM users, due the fact LVM use
(or used to) write to snapshot volumes even when they were set to RO and the
generic_make_request() wasn't honoring the RO status.

If you believe kernel warnings can't be triggered by users, I do apologize, but
I'm not the one confused here. You can assume warnings shouldn't be triggered by
users, but the reality is very different from it, and there are many ways users
can make kernel print out warnings.

As Dave mentioned, it's pretty straight forward to trigger such warning, setting
the block device to RO while the filesystem is still RW.

Have you ever tried to write random data directly to the block device while the
filesystem was still mounted? Try it, wait a few seconds and see the kernel
printing out warnings.

So, yes, there are several ways users can trigger warnings, as long as the user
has enough permissions to change some configuration which will lead to weird
behavior.
KERN_WARNING flags, in my humble view, means: "Something unexpected happened, but
not critical enough to shut down your system", which IMHO, doesn't necessarily
means users can't trigger it. And of course I can be wrong too.

But anyway, I kindly ask you to keep childish behavior off of the list, because
keep shooting people for no reason specially when they are trying to help isn't
neither professional nor is going to help you get help faster on next issues you
may have.

Cheers


> 
> I don't know why you're trying to pick a fight with Carlos, Andi.
> 
> Neither the warning nor whatever is triggering the device to go RO
> has anything to do with XFS, so taking potshots the messenger and
> those trying to help you is out of line. We're just trying to get
> enough information to be able to point you at the right people to
> get your issue fixed.
> 
> To do that, there's /one/ question we need answered. A question both
> Eric and Carlos have already asked you: how is the block device
> being changed to RO while a RW filesystem is mounted on it?
> 
> FYI, the warning is /trivial/ to provoke manually by turning a block
> device read only under an active filesystem with the blockdev
> command:
> 
> root@test4~# blockdev --setro /dev/vdc
> [515320.769159] ------------[ cut here ]------------
> [515320.770078] generic_make_request: Trying to write to read-only block-device vdc (partno 0)
> [515320.771611] WARNING: CPU: 11 PID: 8173 at block/blk-core.c:2171 generic_make_request_checks+0x308/0x4a0
> [515320.773349] CPU: 11 PID: 8173 Comm: xfs_io Not tainted 4.18.0-dgc+ #648
> [515320.774577] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.1-1 04/01/2014
> [515320.776122] RIP: 0010:generic_make_request_checks+0x308/0x4a0
> [515320.777187] Code: 6c 03 00 00 48 8d 74 24 08 48 89 df c6 05 6c de fa 00 01 e8 ea 91 01 00 48 c7 c7 10 90 30 82 48 89 c6 44 89 ea e8 28 3d 9c ff <0f> 0b 48 8b 43 08 e9 b5 fd ff ff 48 8b 53 28 49 03 3
> [515320.780642] RSP: 0018:ffffc90008e0faf8 EFLAGS: 00010282
> [515320.781617] RAX: 0000000000000000 RBX: ffff880829db8180 RCX: 0000000000000006
> [515320.782934] RDX: 0000000000000007 RSI: 0000000000000082 RDI: ffff88083fd15550
> [515320.784272] RBP: ffff88023fab2000 R08: ffffffff8183f550 R09: 000000000000004e
> [515320.785595] R10: ffffc90008e0fbc8 R11: ffffffff82e5c9ee R12: 0000000000001000
> [515320.786910] R13: 0000000000000000 R14: ffffc90008e0fc90 R15: ffff880828cba180
> [515320.788241] FS:  00007f6beac60840(0000) GS:ffff88083fd00000(0000) knlGS:0000000000000000
> [515320.789738] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [515320.790802] CR2: 0000559f72e98000 CR3: 0000000829f2f005 CR4: 0000000000060ee0
> [515320.792124] Call Trace:
> [515320.792649]  ? blk_queue_enter+0x212/0x240
> [515320.793423]  generic_make_request+0x78/0x450
> [515320.794231]  ? iov_iter_get_pages+0xbe/0x2a0
> [515320.795035]  ? submit_bio+0x6d/0x120
> [515320.795714]  submit_bio+0x6d/0x120
> [515320.796403]  iomap_dio_bio_actor+0x1b5/0x3a0
> [515320.797218]  ? iomap_page_release.part.29+0x40/0x40
> [515320.798136]  iomap_apply+0xb0/0x130
> [515320.798803]  iomap_dio_rw+0x2a6/0x3c0
> [515320.799504]  ? iomap_page_release.part.29+0x40/0x40
> [515320.800458]  ? xfs_file_dio_aio_write+0x117/0x2e0
> [515320.801345]  xfs_file_dio_aio_write+0x117/0x2e0
> [515320.802205]  xfs_file_write_iter+0x83/0xb0
> [515320.802988]  __vfs_write+0x109/0x190
> [515320.803695]  vfs_write+0xb6/0x180
> [515320.804535]  ksys_pwrite64+0x71/0x90
> [515320.805212]  do_syscall_64+0x5a/0x180
> [515320.805910]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
> [515320.806852] RIP: 0033:0x7f6bebd5e024
> 
> That doesn't happen on a normal system under normal operation,
> though, so we really need to know where it is coming from.  It's
> likely something in userspace on your system is doing it but until
> we know the source it is not clear where the incorrect behaviour
> lies and what may need fixing.
> 
> So, can you please find what is changing the block device mode to RO
> for us?
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com

-- 
Carlos