Re: Possible kernel fs block code regression in 6.2.3 umounting usb drives

Re: Possible kernel fs block code regression in 6.2.3 umounting usb drives

[Mike Cloaked]

With kerne. 6.2.3 if I simply plug in a usb external drive, mount it
and umount it, then the journal has a kernel Oops and I have submitted
a bug report, that includes the journal output, at
https://bugzilla.kernel.org/show_bug.cgi?id=217174

As soon as the usb drive is unmounted, the kernel Oops occurs, and the
machine hangs on shutdown and needs a hard reboot.

I have reproduced the same issue on three different machines, and in
each case downgrading back to kernel 6.2.2 resolves the issue and it
no longer occurs.

This would seem to be a regression in kernel 6.2.3

Mike C

Re: Possible kernel fs block code regression in 6.2.3 umounting usb drives

[Eric Biggers]

On Fri, Mar 10, 2023 at 07:33:37PM +0000, Mike Cloaked wrote:
> With kerne. 6.2.3 if I simply plug in a usb external drive, mount it
> and umount it, then the journal has a kernel Oops and I have submitted
> a bug report, that includes the journal output, at
> https://bugzilla.kernel.org/show_bug.cgi?id=217174
> 
> As soon as the usb drive is unmounted, the kernel Oops occurs, and the
> machine hangs on shutdown and needs a hard reboot.
> 
> I have reproduced the same issue on three different machines, and in
> each case downgrading back to kernel 6.2.2 resolves the issue and it
> no longer occurs.
> 
> This would seem to be a regression in kernel 6.2.3
> 
> Mike C

Thanks for reporting this!  If this is reliably reproducible and is known to be
a regression between v6.2.2 and v6.2.3, any chance you could bisect it to find
out the exact commit that introduced the bug?

For reference I'm also copying the stack trace from bugzilla below:

BUG: kernel NULL pointer dereference, address: 0000000000000028
 #PF: supervisor read access in kernel mode
 #PF: error_code(0x0000) - not-present page
 PGD 0 P4D 0
 Oops: 0000 [#1] PREEMPT SMP PTI
 CPU: 9 PID: 1118 Comm: lvcreate Tainted: G                T  6.2.3-1>
 Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z370 Ex>
 RIP: 0010:blk_throtl_update_limit_valid+0x1f/0x110
 Code: 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 41 54 49 89 fc>
 RSP: 0018:ffffb5fd01b47bb8 EFLAGS: 00010046
 RAX: 0000000000000000 RBX: ffff9d09040d8000 RCX: 0000000000000000
 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
 RBP: ffffffff97b2f648 R08: 0000000000000000 R09: 0000000000000000
 R10: 0000000000000000 R11: 0000000000000000 R12: ffff9d090fce2c00
 R13: ffff9d090aedf060 R14: ffff9d090aedf1c8 R15: ffff9d090aedf0d8
 FS:  00007f3896fc7240(0000) GS:ffff9d109f040000(0000) knlGS:00000000>
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 0000000000000028 CR3: 0000000111ce4003 CR4: 00000000003706e0
 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
 Call Trace:
  <TASK>
  throtl_pd_offline+0x40/0x70
  blkcg_deactivate_policy+0xab/0x140
  ? __pfx_dev_remove+0x10/0x10 [dm_mod]
  blk_throtl_exit+0x45/0x80
  disk_release+0x4a/0xf0
  device_release+0x34/0x90
  kobject_put+0x97/0x1d0
  cleanup_mapped_device+0xe0/0x170 [dm_mod]
  __dm_destroy+0x120/0x1e0 [dm_mod]
  dev_remove+0x11b/0x190 [dm_mod]
  ctl_ioctl+0x302/0x5b0 [dm_mod]
  dm_ctl_ioctl+0xe/0x20 [dm_mod]
  __x64_sys_ioctl+0x9c/0xe0
  do_syscall_64+0x5c/0x90
  entry_SYSCALL_64_after_hwframe+0x72/0xdc
 RIP: 0033:0x7f389745653f
 Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48>
 RSP: 002b:00007ffe5499e4f0 EFLAGS: 00000246 ORIG_RAX: 00000000000000>
 RAX: ffffffffffffffda RBX: 000055d198c3bec0 RCX: 00007f389745653f
 RDX: 000055d1994501b0 RSI: 00000000c138fd04 RDI: 0000000000000003
 RBP: 0000000000000006 R08: 000055d197547088 R09: 00007ffe5499e3a0
 R10: 0000000000000000 R11: 0000000000000246 R12: 000055d1974d10d6
 R13: 000055d199450260 R14: 000055d1974d10c7 R15: 000055d197545bbb
  </TASK>
 Modules linked in: dm_cache_smq dm_cache dm_persistent_data dm_bio_p>
  soundcore pcspkr intel_wmi_thunderbolt i2c_smbus mei sysimgblt inpu>
 CR2: 0000000000000028
 ---[ end trace 0000000000000000 ]---
 RIP: 0010:blk_throtl_update_limit_valid+0x1f/0x110
 Code: 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 41 54 49 89 fc>

Re: Possible kernel fs block code regression in 6.2.3 umounting usb drives

[Eric Biggers]

On Fri, Mar 10, 2023 at 12:14:10PM -0800, Eric Biggers wrote:
> On Fri, Mar 10, 2023 at 07:33:37PM +0000, Mike Cloaked wrote:
> > With kerne. 6.2.3 if I simply plug in a usb external drive, mount it
> > and umount it, then the journal has a kernel Oops and I have submitted
> > a bug report, that includes the journal output, at
> > https://bugzilla.kernel.org/show_bug.cgi?id=217174
> > 
> > As soon as the usb drive is unmounted, the kernel Oops occurs, and the
> > machine hangs on shutdown and needs a hard reboot.
> > 
> > I have reproduced the same issue on three different machines, and in
> > each case downgrading back to kernel 6.2.2 resolves the issue and it
> > no longer occurs.
> > 
> > This would seem to be a regression in kernel 6.2.3
> > 
> > Mike C
> 
> Thanks for reporting this!  If this is reliably reproducible and is known to be
> a regression between v6.2.2 and v6.2.3, any chance you could bisect it to find
> out the exact commit that introduced the bug?
> 
> For reference I'm also copying the stack trace from bugzilla below:
> 
> BUG: kernel NULL pointer dereference, address: 0000000000000028
>  #PF: supervisor read access in kernel mode
>  #PF: error_code(0x0000) - not-present page
>  PGD 0 P4D 0
>  Oops: 0000 [#1] PREEMPT SMP PTI
>  CPU: 9 PID: 1118 Comm: lvcreate Tainted: G                T  6.2.3-1>
>  Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z370 Ex>
>  RIP: 0010:blk_throtl_update_limit_valid+0x1f/0x110

BTW, the block/ commits between v6.2.2 and v6.2.3 were:

	blk-mq: avoid sleep in blk_mq_alloc_request_hctx
	blk-mq: remove stale comment for blk_mq_sched_mark_restart_hctx
	blk-mq: wait on correct sbitmap_queue in blk_mq_mark_tag_wait
	blk-mq: Fix potential io hung for shared sbitmap per tagset
	blk-mq: correct stale comment of .get_budget
	block: sync mixed merged request's failfast with 1st bio's
	block: Fix io statistics for cgroup in throttle path
	block: bio-integrity: Copy flags when bio_integrity_payload is cloned
	block: use proper return value from bio_failfast()
	blk-iocost: fix divide by 0 error in calc_lcoefs()
	blk-cgroup: dropping parent refcount after pd_free_fn() is done
	blk-cgroup: synchronize pd_free_fn() from blkg_free_workfn() and blkcg_deactivate_policy()
	block: don't allow multiple bios for IOCB_NOWAIT issue
	block: clear bio->bi_bdev when putting a bio back in the cache
	block: be a bit more careful in checking for NULL bdev while polling

Without having any in-depth knowledge here, I think "blk-cgroup: synchronize
pd_free_fn() from blkg_free_workfn() and blkcg_deactivate_policy()" looks the
most suspicious here...  I see that AUTOSEL selected it from a 3-patch series
without backporting patch 2, maybe that could be it?  Anyway, just a hunch.

- Eric

Re: Possible kernel fs block code regression in 6.2.3 umounting usb drives

[Jens Axboe]

On 3/10/23 1:16 PM, Eric Biggers wrote:
> On Fri, Mar 10, 2023 at 12:14:10PM -0800, Eric Biggers wrote:
>> On Fri, Mar 10, 2023 at 07:33:37PM +0000, Mike Cloaked wrote:
>>> With kerne. 6.2.3 if I simply plug in a usb external drive, mount it
>>> and umount it, then the journal has a kernel Oops and I have submitted
>>> a bug report, that includes the journal output, at
>>> https://bugzilla.kernel.org/show_bug.cgi?id=217174
>>>
>>> As soon as the usb drive is unmounted, the kernel Oops occurs, and the
>>> machine hangs on shutdown and needs a hard reboot.
>>>
>>> I have reproduced the same issue on three different machines, and in
>>> each case downgrading back to kernel 6.2.2 resolves the issue and it
>>> no longer occurs.
>>>
>>> This would seem to be a regression in kernel 6.2.3
>>>
>>> Mike C
>>
>> Thanks for reporting this!  If this is reliably reproducible and is known to be
>> a regression between v6.2.2 and v6.2.3, any chance you could bisect it to find
>> out the exact commit that introduced the bug?
>>
>> For reference I'm also copying the stack trace from bugzilla below:
>>
>> BUG: kernel NULL pointer dereference, address: 0000000000000028
>>  #PF: supervisor read access in kernel mode
>>  #PF: error_code(0x0000) - not-present page
>>  PGD 0 P4D 0
>>  Oops: 0000 [#1] PREEMPT SMP PTI
>>  CPU: 9 PID: 1118 Comm: lvcreate Tainted: G                T  6.2.3-1>
>>  Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z370 Ex>
>>  RIP: 0010:blk_throtl_update_limit_valid+0x1f/0x110
> 
> BTW, the block/ commits between v6.2.2 and v6.2.3 were:
> 
> 	blk-mq: avoid sleep in blk_mq_alloc_request_hctx
> 	blk-mq: remove stale comment for blk_mq_sched_mark_restart_hctx
> 	blk-mq: wait on correct sbitmap_queue in blk_mq_mark_tag_wait
> 	blk-mq: Fix potential io hung for shared sbitmap per tagset
> 	blk-mq: correct stale comment of .get_budget
> 	block: sync mixed merged request's failfast with 1st bio's
> 	block: Fix io statistics for cgroup in throttle path
> 	block: bio-integrity: Copy flags when bio_integrity_payload is cloned
> 	block: use proper return value from bio_failfast()
> 	blk-iocost: fix divide by 0 error in calc_lcoefs()
> 	blk-cgroup: dropping parent refcount after pd_free_fn() is done
> 	blk-cgroup: synchronize pd_free_fn() from blkg_free_workfn() and blkcg_deactivate_policy()
> 	block: don't allow multiple bios for IOCB_NOWAIT issue
> 	block: clear bio->bi_bdev when putting a bio back in the cache
> 	block: be a bit more careful in checking for NULL bdev while polling
> 
> Without having any in-depth knowledge here, I think "blk-cgroup: synchronize
> pd_free_fn() from blkg_free_workfn() and blkcg_deactivate_policy()" looks the
> most suspicious here...  I see that AUTOSEL selected it from a 3-patch series
> without backporting patch 2, maybe that could be it?  Anyway, just a hunch.

Was just looking at this too, the primary suspects would indeed be those
two blk-cgroup changes. And yes, they ended up in stable due to auto
selection, and very odd how it picked 2 and not the 3rd?!

But I would revert:

bfe46d2efe46c5c952f982e2ca94fe2ec5e58e2a
57a425badc05c2e87e9f25713e5c3c0298e4202c

in that order from 6.2.3 and see if that helps. Adding Yu.

-- 
Jens Axboe

Re: Possible kernel fs block code regression in 6.2.3 umounting usb drives

[Genes Lists]

On 3/10/23 15:23, Jens Axboe wrote:
> On 3/10/23 1:16 PM, Eric Biggers wrote:
...
> But I would revert:
> 
> bfe46d2efe46c5c952f982e2ca94fe2ec5e58e2a
> 57a425badc05c2e87e9f25713e5c3c0298e4202c
> 
> in that order from 6.2.3 and see if that helps. Adding Yu.
> 
Confirm the 2 Reverts fixed in my tests as well (nvme + sata drives).
Nasty crash - some needed to be power cycled as they hung on shutdown.

Thank you!

gene

Re: Possible kernel fs block code regression in 6.2.3 umounting usb drives

[Eric Biggers]

On Fri, Mar 10, 2023 at 04:08:21PM -0500, Genes Lists wrote:
> On 3/10/23 15:23, Jens Axboe wrote:
> > On 3/10/23 1:16 PM, Eric Biggers wrote:
> ...
> > But I would revert:
> > 
> > bfe46d2efe46c5c952f982e2ca94fe2ec5e58e2a
> > 57a425badc05c2e87e9f25713e5c3c0298e4202c
> > 
> > in that order from 6.2.3 and see if that helps. Adding Yu.
> > 
> Confirm the 2 Reverts fixed in my tests as well (nvme + sata drives).
> Nasty crash - some needed to be power cycled as they hung on shutdown.
> 
> Thank you!
> 
> gene
> 
> 

Great, thanks.  BTW, 6.1 is also affected.  A simple reproducer is to run:

	dmsetup create dev --table "0 128 zero"
	dmsetup remove dev

The following kconfigs are needed for the bug to be hit:

	CONFIG_BLK_CGROUP=y
	CONFIG_BLK_DEV_THROTTLING=y
	CONFIG_BLK_DEV_THROTTLING_LOW=y

Sasha or Greg, can you please revert the indicated commits from 6.1 and 6.2?

- Eric

Re: Possible kernel fs block code regression in 6.2.3 umounting usb drives

[Yu Kuai]

Hi,

在 2023/03/11 5:08, Genes Lists 写道:
> On 3/10/23 15:23, Jens Axboe wrote:
>> On 3/10/23 1:16 PM, Eric Biggers wrote:
> ...
>> But I would revert:
>>
>> bfe46d2efe46c5c952f982e2ca94fe2ec5e58e2a
>> 57a425badc05c2e87e9f25713e5c3c0298e4202c
>>
>> in that order from 6.2.3 and see if that helps. Adding Yu.

So, The reason is that patch dfd6200a0954 ("blk-cgroup: support to track
if policy is online") is missed, this will absolutely cause some
problems.

Thanks,
Kuai
>>
> Confirm the 2 Reverts fixed in my tests as well (nvme + sata drives).
> Nasty crash - some needed to be power cycled as they hung on shutdown.
> 
> Thank you!
> 
> gene
> 
> 
> .
> 

Re: Possible kernel fs block code regression in 6.2.3 umounting usb drives

[Greg Kroah-Hartman]

On Fri, Mar 10, 2023 at 02:53:19PM -0800, Eric Biggers wrote:
> On Fri, Mar 10, 2023 at 04:08:21PM -0500, Genes Lists wrote:
> > On 3/10/23 15:23, Jens Axboe wrote:
> > > On 3/10/23 1:16 PM, Eric Biggers wrote:
> > ...
> > > But I would revert:
> > > 
> > > bfe46d2efe46c5c952f982e2ca94fe2ec5e58e2a
> > > 57a425badc05c2e87e9f25713e5c3c0298e4202c
> > > 
> > > in that order from 6.2.3 and see if that helps. Adding Yu.
> > > 
> > Confirm the 2 Reverts fixed in my tests as well (nvme + sata drives).
> > Nasty crash - some needed to be power cycled as they hung on shutdown.
> > 
> > Thank you!
> > 
> > gene
> > 
> > 
> 
> Great, thanks.  BTW, 6.1 is also affected.  A simple reproducer is to run:
> 
> 	dmsetup create dev --table "0 128 zero"
> 	dmsetup remove dev
> 
> The following kconfigs are needed for the bug to be hit:
> 
> 	CONFIG_BLK_CGROUP=y
> 	CONFIG_BLK_DEV_THROTTLING=y
> 	CONFIG_BLK_DEV_THROTTLING_LOW=y
> 
> Sasha or Greg, can you please revert the indicated commits from 6.1 and 6.2?

Yes, will go do that right now, thanks for debugging this so quickly!

greg k-h

Re: Possible kernel fs block code regression in 6.2.3 umounting usb drives

[Jens Axboe]

n 3/11/23 2:32?AM, Greg Kroah-Hartman wrote:
> On Fri, Mar 10, 2023 at 02:53:19PM -0800, Eric Biggers wrote:
>> On Fri, Mar 10, 2023 at 04:08:21PM -0500, Genes Lists wrote:
>>> On 3/10/23 15:23, Jens Axboe wrote:
>>>> On 3/10/23 1:16?PM, Eric Biggers wrote:
>>> ...
>>>> But I would revert:
>>>>
>>>> bfe46d2efe46c5c952f982e2ca94fe2ec5e58e2a
>>>> 57a425badc05c2e87e9f25713e5c3c0298e4202c
>>>>
>>>> in that order from 6.2.3 and see if that helps. Adding Yu.
>>>>
>>> Confirm the 2 Reverts fixed in my tests as well (nvme + sata drives).
>>> Nasty crash - some needed to be power cycled as they hung on shutdown.
>>>
>>> Thank you!
>>>
>>> gene
>>>
>>>
>>
>> Great, thanks.  BTW, 6.1 is also affected.  A simple reproducer is to run:
>>
>> 	dmsetup create dev --table "0 128 zero"
>> 	dmsetup remove dev
>>
>> The following kconfigs are needed for the bug to be hit:
>>
>> 	CONFIG_BLK_CGROUP=y
>> 	CONFIG_BLK_DEV_THROTTLING=y
>> 	CONFIG_BLK_DEV_THROTTLING_LOW=y
>>
>> Sasha or Greg, can you please revert the indicated commits from 6.1 and 6.2?
> 
> Yes, will go do that right now, thanks for debugging this so quickly!

The issue here is that parts of a series was auto-selected. That seems
like a bad idea to do for stable. Just because something applies without
other parts of the series doesn't mean it's sane to backport by itself.

How do we prevent that from happening? Maybe we just need to default
to "if whole series doesn't pick cleanly, don't grab any parts of it
in auto-selection"? Exception being if it's explicitly marked for stable,
not uncommon to have a series that starts with a fix or two which should
go to stable, then the feature bits.

-- 
Jens Axboe