MQ-BFQ crashing on battery mode

MQ-BFQ crashing on battery mode

[Massimo Burcheri]

[-- Attachment #1: Type: text/plain, Size: 2140 bytes --]

Hello,


I got a kernel trace when unplugging the power supply, switching to battery
mode. I get the same kernel trace when booting on battery.
Both making the system unusable or breaking the boot.

The kernel call trace with symbols:

? blk_mq_requeue_request+0x...
? __scsi_queue_insert+0x...
? ata_scsi_var_len_cdb_xlat+0x
? __blk_mq_complete_request+0x...
? ata_scsi_translate+0x...
? ata_scsi_queuecmd+0x...
? scsi_dispatch_cmd+0x...
? scsi_queue_rq+0x...
? blk_mq_dispatch_rq_list+0x...
? kyber_dispatch_cur_domain+0x...
? kyber_completed_request+0x...
? blk_mq_sched_dispatch_requests+0x...
? __ blk_mq_run_hw_queue+0x...
? __blk_mq_delay_run_hw_queue+0x...
? blk_mq_run_hw_queue+0x...
? blk_mq_run_hw_queues+0x...
? blk_mq_requeue_work+0x...
? process_one_work+0x...
? worker_thread+0x...
? process_one_work+0x...
? kthread+0x...
? kthread_flush_work_fn+0x...
? ret_from_fork+0x...
Code: ...
RIP: sbitmap_queue_clear+0x...

Screenshot: 	https://ibin.co/4D34Ej3DWsqI.jpg
Kernel config: 	https://bpaste.net/show/870004e55123

Kernel: 4.17.11-ck

Setup:

btrfs-on-bcache-on-luks
btrfs options (rw,noatime,nodiratime,compress-
force=lzo,nossd,noacl,space_cache,autodefrag)

Using mq bfq scheduler for the hdd backing and kyber for the ssd caching device


Failed tests:
Tested many kernel down to 4.13.2 with Gentoo or Ck patchset. Sorry for not
including the vanilla sources in the test, I can provide if required.
Skipping services in the boot process didn't help, any next service leads to the
same trace.
Switching off the laptop-mode-tools daemon didn't help.
Switching all devices to "none" scheduler did not help.


Workaround:
After some tests and due to the *mq* call stack I was able to workaround by
disabling CONFIG_SCSI_MQ_DEFAULT and CONFIG_DM_MQ_DEFAULT and switching all
devices to cfq scheduler.
However with the MQ enabled kernel, only bfq, kyber and none are possible, while
the non-mq kernel can only set cfq. I guess this is intentional as the current
bfq implementation is a MQ only version and CFQ is a non-mq only version?

Best regards,
Massimo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: MQ-BFQ crashing on battery mode

[Paolo Valente]

> Il giorno 22 ago 2018, alle ore 10:20, Massimo Burcheri <massimo@burcheri.de> ha scritto:
> 
> Hello,
> 
> 
> I got a kernel trace when unplugging the power supply, switching to battery
> mode. I get the same kernel trace when booting on battery.
> Both making the system unusable or breaking the boot.
> 
> The kernel call trace with symbols:
> 
> ? blk_mq_requeue_request+0x...
> ? __scsi_queue_insert+0x...
> ? ata_scsi_var_len_cdb_xlat+0x
> ? __blk_mq_complete_request+0x...
> ? ata_scsi_translate+0x...
> ? ata_scsi_queuecmd+0x...
> ? scsi_dispatch_cmd+0x...
> ? scsi_queue_rq+0x...
> ? blk_mq_dispatch_rq_list+0x...
> ? kyber_dispatch_cur_domain+0x...
> ? kyber_completed_request+0x...
> ? blk_mq_sched_dispatch_requests+0x...
> ? __ blk_mq_run_hw_queue+0x...
> ? __blk_mq_delay_run_hw_queue+0x...
> ? blk_mq_run_hw_queue+0x...
> ? blk_mq_run_hw_queues+0x...
> ? blk_mq_requeue_work+0x...
> ? process_one_work+0x...
> ? worker_thread+0x...
> ? process_one_work+0x...
> ? kthread+0x...
> ? kthread_flush_work_fn+0x...
> ? ret_from_fork+0x...
> Code: ...
> RIP: sbitmap_queue_clear+0x...
> 
> Screenshot: 	https://ibin.co/4D34Ej3DWsqI.jpg
> Kernel config: 	https://bpaste.net/show/870004e55123
> 
> Kernel: 4.17.11-ck
> 
> Setup:
> 
> btrfs-on-bcache-on-luks
> btrfs options (rw,noatime,nodiratime,compress-
> force=lzo,nossd,noacl,space_cache,autodefrag)
> 
> Using mq bfq scheduler for the hdd backing and kyber for the ssd caching device
> 
> 

Hi,
I'm missing why you mention bfq in the subject, as, according to the
trace, the failure has not to do with bfq.  IIRC this failure, or a
very similar one, has been reported recently (and maybe fixed too).

Thanks,
Paolo

> Failed tests:
> Tested many kernel down to 4.13.2 with Gentoo or Ck patchset. Sorry for not
> including the vanilla sources in the test, I can provide if required.
> Skipping services in the boot process didn't help, any next service leads to the
> same trace.
> Switching off the laptop-mode-tools daemon didn't help.
> Switching all devices to "none" scheduler did not help.
> 
> 
> Workaround:
> After some tests and due to the *mq* call stack I was able to workaround by
> disabling CONFIG_SCSI_MQ_DEFAULT and CONFIG_DM_MQ_DEFAULT and switching all
> devices to cfq scheduler.
> However with the MQ enabled kernel, only bfq, kyber and none are possible, while
> the non-mq kernel can only set cfq. I guess this is intentional as the current
> bfq implementation is a MQ only version and CFQ is a non-mq only version?
> 
> Best regards,
> Massimo

Re: MQ-BFQ crashing on battery mode

[Jens Axboe]

On 8/22/18 2:20 AM, Massimo Burcheri wrote:
> Hello,
> 
> 
> I got a kernel trace when unplugging the power supply, switching to battery
> mode. I get the same kernel trace when booting on battery.
> Both making the system unusable or breaking the boot.
> 
> The kernel call trace with symbols:
> 
> ? blk_mq_requeue_request+0x...
> ? __scsi_queue_insert+0x...
> ? ata_scsi_var_len_cdb_xlat+0x
> ? __blk_mq_complete_request+0x...
> ? ata_scsi_translate+0x...
> ? ata_scsi_queuecmd+0x...
> ? scsi_dispatch_cmd+0x...
> ? scsi_queue_rq+0x...
> ? blk_mq_dispatch_rq_list+0x...
> ? kyber_dispatch_cur_domain+0x...
> ? kyber_completed_request+0x...
> ? blk_mq_sched_dispatch_requests+0x...
> ? __ blk_mq_run_hw_queue+0x...
> ? __blk_mq_delay_run_hw_queue+0x...
> ? blk_mq_run_hw_queue+0x...
> ? blk_mq_run_hw_queues+0x...
> ? blk_mq_requeue_work+0x...
> ? process_one_work+0x...
> ? worker_thread+0x...
> ? process_one_work+0x...
> ? kthread+0x...
> ? kthread_flush_work_fn+0x...
> ? ret_from_fork+0x...
> Code: ...
> RIP: sbitmap_queue_clear+0x...
> 
> Screenshot: 	https://ibin.co/4D34Ej3DWsqI.jpg
> Kernel config: 	https://bpaste.net/show/870004e55123
> 
> Kernel: 4.17.11-ck
> 
> Setup:
> 
> btrfs-on-bcache-on-luks
> btrfs options (rw,noatime,nodiratime,compress-
> force=lzo,nossd,noacl,space_cache,autodefrag)
> 
> Using mq bfq scheduler for the hdd backing and kyber for the ssd caching device
> 
> 
> Failed tests:
> Tested many kernel down to 4.13.2 with Gentoo or Ck patchset. Sorry for not
> including the vanilla sources in the test, I can provide if required.
> Skipping services in the boot process didn't help, any next service leads to the
> same trace.
> Switching off the laptop-mode-tools daemon didn't help.
> Switching all devices to "none" scheduler did not help.

Any chance you can try 4.18 at least?

> Workaround:
> After some tests and due to the *mq* call stack I was able to workaround by
> disabling CONFIG_SCSI_MQ_DEFAULT and CONFIG_DM_MQ_DEFAULT and switching all
> devices to cfq scheduler.
> However with the MQ enabled kernel, only bfq, kyber and none are possible, while
> the non-mq kernel can only set cfq. I guess this is intentional as the current
> bfq implementation is a MQ only version and CFQ is a non-mq only version?

Schedulers are specific to the queuing infrastructure, so certain types
will only be available with MQ and certain types will only be available
on !MQ.

-- 
Jens Axboe

Re: MQ-BFQ crashing on battery mode

[Massimo Burcheri]

[-- Attachment #1: Type: text/plain, Size: 1005 bytes --]

On Wed, 2018-08-22 at 11:42 -0600, Jens Axboe wrote:
> Any chance you can try 4.18 at least?

Of course, I tried the kernel.org sources now, sys-kernel/vanilla-sources-4.18.3 
on Gentoo. The issue is the same.

I wonder that nobody has noticed that issue that I can reproduce on many kernels
since 4.13 or even older. Is it a rare combination with my bcache, luks or
btrfs? Or hardware specific? Or scheduler specific, as even BFQ is not that well
tested on btrfs like the default one.

> Schedulers are specific to the queuing infrastructure, so certain types
> will only be available with MQ and certain types will only be available
> on !MQ.

I noticed that. It's just a bit intransparent from the menuconfig, options don't
mention if a scheduler is working on MQ or not, like CONFIG_IOSCHED_BFQ.
There it's written in the description, while CONFIG_MQ_IOSCHED_KYBER has the
note in the name.
Actually MQ schedulers could depend on BLK_MQ to make it obvious.

Best regards,
Massimo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: MQ-BFQ crashing on battery mode

[jianchao.wang]

Hi Massimo

On 08/24/2018 02:29 PM, Massimo Burcheri wrote:
> On Wed, 2018-08-22 at 11:42 -0600, Jens Axboe wrote:
>> Any chance you can try 4.18 at least?
> Of course, I tried the kernel.org sources now, sys-kernel/vanilla-sources-4.18.3 
> on Gentoo. The issue is the same.

In the first mail of this thread, you said:
"
Switching all devices to "none" scheduler did not help
"

So this issue should not related with io scheduler.

Would you please share the entire dmesg log or backtrace you got on 4.18.3 ?

Thanks
Jianchao

Re: blk-mq crashing on battery mode

[Massimo Burcheri]

[-- Attachment #1: Type: text/plain, Size: 425 bytes --]

On Fri, 2018-08-24 at 15:12 +0800, jianchao.wang wrote:
> So this issue should not related with io scheduler.

Hi, I can't localize the specific issue and have changed the subject to blk-mq.

> Would you please share the entire dmesg log or backtrace you got on 4.18.3 ?

Have you been able to reproduce this? I reproduced the backtrace on a vanilla
4.18.3:
https://ibin.co/4E3q5dETqA5T.jpg

Best regards,
Massimo

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]