uml segfault during I/O

uml segfault during I/O

[Ritesh Raj Sarraf]

[-- Attachment #1.1: Type: text/plain, Size: 2580 bytes --]

This happens on the 5.2 Linux kernel with Debian patches on top.
The configs details are:

Linux Host: 5.2 (Debian)
Linux UML: 5.2 (Debian)

This tends to happen very easily when doing good amount of I/O, in this
case, doing an `apt upgrade`


[  OK  ] Started Update UTMP about System Runlevel Changes.
NET: Registered protocol family 10
Segment Routing with IPv6
random: crng init done

Modules linked in: ipv6 crc_ccitt
Pid: 870, comm: kworker/0:1H Not tainted 5.2.17
RIP: 0033:[<00000000607e9c03>]
RSP: 000000009d40fde0  EFLAGS: 00010297
RAX: 0000000000000000 RBX: 0000000000004801 RCX: 000000009d40fe30
RDX: 0000000000000001 RSI: 000000009dbc35c0 RDI: 0000000000000000
RBP: 000000009d40fe30 R08: 0000000c00000204 R09: 8080808080808080
R10: fefefefefefefeff R11: 0000000000000246 R12: 0000000000000000
R13: 000000009dbc35c0 R14: 000000009dbc3608 R15: 000000009dbb9800
Kernel panic - not syncing: Segfault with no mm
CPU: 0 PID: 870 Comm: kworker/0:1H Not tainted 5.2.17 #2
Workqueue: kblockd blk_mq_requeue_work
Stack:
 00000000 100000000000001 607d4425 00000001
 9dbc35c0 00000000 00000000 00000000
 9dbb9800 607d44e9 9d40fe30 9d40fe30
Call Trace:
 [<607d4425>] ? blk_mq_sched_insert_request+0x0/0xff
 [<607d44e9>] ? blk_mq_sched_insert_request+0xc4/0xff
 [<607d4425>] ? blk_mq_sched_insert_request+0x0/0xff
 [<607d07c6>] ? blk_mq_requeue_work+0xd0/0x133
 [<60434285>] ? process_one_work+0x1e4/0x34c
 [<60431fd0>] ? move_linked_works+0x0/0x4f
 [<609791f0>] ? __schedule+0x0/0x41d
 [<6043569f>] ? wq_worker_running+0xd/0x2f
 [<60431fd0>] ? move_linked_works+0x0/0x4f
 [<60435362>] ? worker_thread+0x324/0x45e
 [<6043215f>] ? set_pf_worker+0x0/0x5e
 [<604167c7>] ? get_signals+0x0/0xa
 [<60439485>] ? __kthread_parkme+0x4a/0x94
 [<60421a53>] ? do_exit+0x0/0x948
 [<6043503e>] ? worker_thread+0x0/0x45e
 [<604399aa>] ? kthread+0x198/0x1a0
 [<603fea08>] ? new_thread_handler+0x82/0xb8

/home/rrs/bin/uml-debian: line 8: 16743 Aborted                 (core dumped) linux ubd0=~/rrs-home/Libvirt-Images/uml.img eth0=daemon mem=1024M rw
 16:58 ♒ ॐ   ☹ 😟=> 134  


And the linux process remains stray on the host machine.

rrs      18159  0.0  0.0 1057692 11496 ?       Ss   16:57   0:00 linux ubd0=/home/rrs/rrs-home/Libvirt-Images/uml.img eth0=daemon mem=1024M rw
rrs      18187  0.0  0.0 1057704 11496 ?       Ss   16:57   0:00 linux ubd0=/home/rrs/rrs-home/Libvirt-Images/uml.img eth0=daemon mem=1024M rw

-- 
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 152 bytes --]

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: uml segfault during I/O

[Ritesh Raj Sarraf]

[-- Attachment #1.1: Type: text/plain, Size: 1145 bytes --]

On Thu, 2019-10-17 at 14:02 +0100, Anton Ivanov wrote:
> Looking into that. I have not run into anything like that, but I have
> not used any of the legacy networking for 5 odd years now.
> 

Do you think this is related to networking ? I ask because there was no
network activity going on.

apt is just an example. The packages were all already downloaded and
there was no network activity. Rather, there was block I/O.

One thing I noticed, which may or may not be useful to this report. I
was booting the uml guest and immediately logging into it and running
the block I/O. And the segfault would occur.

If I, instead, booted the uml vm and let it remain idle for a minute or
so and then did the I/O, it worked fine. So I am not sure if there is
any lazy initialization happening in the background which gets
corrupted during immediate hot boot I/O.

On the other hand, if you think there can be any number of commands to
run locally that could generate more information, please assist me so.

Thanks,
Ritesh

-- 
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 152 bytes --]

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: uml segfault during I/O

[Anton Ivanov]

On 17/10/2019 14:30, Ritesh Raj Sarraf wrote:
> On Thu, 2019-10-17 at 14:02 +0100, Anton Ivanov wrote:
>> Looking into that. I have not run into anything like that, but I have
>> not used any of the legacy networking for 5 odd years now.
>>
> 
> Do you think this is related to networking ? I ask because there was no
> network activity going on.

no, it's disk somewhere. I managed to reproduce it with 5.2 stock on 
Debian 5.2 host.

> 
> apt is just an example. The packages were all already downloaded and
> there was no network activity. Rather, there was block I/O.

I am looking into that.

> 
> One thing I noticed, which may or may not be useful to this report. I
> was booting the uml guest and immediately logging into it and running
> the block I/O. And the segfault would occur.
> 
> If I, instead, booted the uml vm and let it remain idle for a minute or
> so and then did the I/O, it worked fine. So I am not sure if there is
> any lazy initialization happening in the background which gets
> corrupted during immediate hot boot I/O.

Interesting...

> 
> On the other hand, if you think there can be any number of commands to
> run locally that could generate more information, please assist me so.

As I said - I managed to reproduce it, I am looking at it. In first 
instance I am trying with a couple of version up/down to see if this is 
5.2 specific.

> 
> Thanks,
> Ritesh
> 
> 
> _______________________________________________
> linux-um mailing list
> linux-um@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-um
> 

-- 
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: uml segfault during I/O

[Anton Ivanov]

On 17/10/2019 14:33, Anton Ivanov wrote:
> 
> 
> On 17/10/2019 14:30, Ritesh Raj Sarraf wrote:
>> On Thu, 2019-10-17 at 14:02 +0100, Anton Ivanov wrote:
>>> Looking into that. I have not run into anything like that, but I have
>>> not used any of the legacy networking for 5 odd years now.
>>>
>>
>> Do you think this is related to networking ? I ask because there was no
>> network activity going on.
> 
> no, it's disk somewhere. I managed to reproduce it with 5.2 stock on 
> Debian 5.2 host.
> 
>>
>> apt is just an example. The packages were all already downloaded and
>> there was no network activity. Rather, there was block I/O.
> 
> I am looking into that.
> 
>>
>> One thing I noticed, which may or may not be useful to this report. I
>> was booting the uml guest and immediately logging into it and running
>> the block I/O. And the segfault would occur.
>>
>> If I, instead, booted the uml vm and let it remain idle for a minute or
>> so and then did the I/O, it worked fine. So I am not sure if there is
>> any lazy initialization happening in the background which gets
>> corrupted during immediate hot boot I/O.
> 
> Interesting...
> 
>>
>> On the other hand, if you think there can be any number of commands to
>> run locally that could generate more information, please assist me so.
> 
> As I said - I managed to reproduce it, I am looking at it. In first 
> instance I am trying with a couple of version up/down to see if this is 
> 5.2 specific.

I cannot even get it to start on 5.4-rc1, 5.3 shows the same symptoms.

> 
>>
>> Thanks,
>> Ritesh
>>
>>
>> _______________________________________________
>> linux-um mailing list
>> linux-um@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-um
>>
> 

-- 
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: uml segfault during I/O

[Anton Ivanov]

On 17/10/2019 16:03, Anton Ivanov wrote:
> 
> 
> On 17/10/2019 14:33, Anton Ivanov wrote:
>>
>>
>> On 17/10/2019 14:30, Ritesh Raj Sarraf wrote:
>>> On Thu, 2019-10-17 at 14:02 +0100, Anton Ivanov wrote:
>>>> Looking into that. I have not run into anything like that, but I have
>>>> not used any of the legacy networking for 5 odd years now.
>>>>
>>>
>>> Do you think this is related to networking ? I ask because there was no
>>> network activity going on.
>>
>> no, it's disk somewhere. I managed to reproduce it with 5.2 stock on 
>> Debian 5.2 host.
>>
>>>
>>> apt is just an example. The packages were all already downloaded and
>>> there was no network activity. Rather, there was block I/O.
>>
>> I am looking into that.
>>
>>>
>>> One thing I noticed, which may or may not be useful to this report. I
>>> was booting the uml guest and immediately logging into it and running
>>> the block I/O. And the segfault would occur.
>>>
>>> If I, instead, booted the uml vm and let it remain idle for a minute or
>>> so and then did the I/O, it worked fine. So I am not sure if there is
>>> any lazy initialization happening in the background which gets
>>> corrupted during immediate hot boot I/O.
>>
>> Interesting...
>>
>>>
>>> On the other hand, if you think there can be any number of commands to
>>> run locally that could generate more information, please assist me so.
>>
>> As I said - I managed to reproduce it, I am looking at it. In first 
>> instance I am trying with a couple of version up/down to see if this 
>> is 5.2 specific.
> 
> I cannot even get it to start on 5.4-rc1, 5.3 shows the same symptoms.

This is something outside the UBD driver as such. There were no notable 
changes to it since we ported it to block-mq and added DISCARD which was 
quite a while back.

I am going to check the other usual suspects such as IRQs, but that is 
something I test quite extensively when working on the networking side.

So I suspect that this is something outside UML which is showing only in 
UML for some reason.

A.

> 
>>
>>>
>>> Thanks,
>>> Ritesh
>>>
>>>
>>> _______________________________________________
>>> linux-um mailing list
>>> linux-um@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-um
>>>
>>
> 

-- 
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: uml segfault during I/O

[Anton Ivanov]

On 17/10/2019 17:29, Anton Ivanov wrote:
> 
> 
> On 17/10/2019 16:03, Anton Ivanov wrote:
>>
>>
>> On 17/10/2019 14:33, Anton Ivanov wrote:
>>>
>>>
>>> On 17/10/2019 14:30, Ritesh Raj Sarraf wrote:
>>>> On Thu, 2019-10-17 at 14:02 +0100, Anton Ivanov wrote:
>>>>> Looking into that. I have not run into anything like that, but I have
>>>>> not used any of the legacy networking for 5 odd years now.
>>>>>
>>>>
>>>> Do you think this is related to networking ? I ask because there was no
>>>> network activity going on.
>>>
>>> no, it's disk somewhere. I managed to reproduce it with 5.2 stock on 
>>> Debian 5.2 host.
>>>
>>>>
>>>> apt is just an example. The packages were all already downloaded and
>>>> there was no network activity. Rather, there was block I/O.
>>>
>>> I am looking into that.
>>>
>>>>
>>>> One thing I noticed, which may or may not be useful to this report. I
>>>> was booting the uml guest and immediately logging into it and running
>>>> the block I/O. And the segfault would occur.
>>>>
>>>> If I, instead, booted the uml vm and let it remain idle for a minute or
>>>> so and then did the I/O, it worked fine. So I am not sure if there is
>>>> any lazy initialization happening in the background which gets
>>>> corrupted during immediate hot boot I/O.
>>>
>>> Interesting...
>>>
>>>>
>>>> On the other hand, if you think there can be any number of commands to
>>>> run locally that could generate more information, please assist me so.
>>>
>>> As I said - I managed to reproduce it, I am looking at it. In first 
>>> instance I am trying with a couple of version up/down to see if this 
>>> is 5.2 specific.
>>
>> I cannot even get it to start on 5.4-rc1, 5.3 shows the same symptoms.
> 
> This is something outside the UBD driver as such. There were no notable 
> changes to it since we ported it to block-mq and added DISCARD which was 
> quite a while back.
> 
> I am going to check the other usual suspects such as IRQs, but that is 
> something I test quite extensively when working on the networking side.
> 
> So I suspect that this is something outside UML which is showing only in 
> UML for some reason.

Still happening with 5.2.21, albeit a bit more difficult to reproduce.

Looking at the backtraces it is ALWAYS a result of a re-queue.

[  263.990000]  [<60440633>] blk_mq_dispatch_rq_list+0xf3/0x5f0
[  263.990000]  [<60440501>] ? blk_mq_get_driver_tag+0xc1/0x100
[  263.990000]  [<60440540>] ? blk_mq_dispatch_rq_list+0x0/0x5f0
[  263.990000]  [<60445966>] blk_mq_do_dispatch_sched+0x66/0xe0
[  263.990000]  [<60446107>] blk_mq_sched_dispatch_requests+0x107/0x190
[  263.990000]  [<6043f130>] ? blk_mq_run_hw_queue+0x0/0x120
[  263.990000]  [<6043efb4>] __blk_mq_run_hw_queue+0x74/0xd0
[  263.990000]  [<6043f0d4>] __blk_mq_delay_run_hw_queue+0xc4/0xd0
[  263.990000]  [<6043f1d9>] blk_mq_run_hw_queue+0xa9/0x120
[  263.990000]  [<6043f292>] blk_mq_run_hw_queues+0x42/0x60
[  263.990000]  [<60440bc0>] ? blk_mq_request_bypass_insert+0x0/0x90
[  263.990000]  [<604462a0>] ? blk_mq_sched_insert_request+0x0/0x1c0
ALWAYS >>[  263.990000]  [<604414b0>] blk_mq_requeue_work+0x160/0x170
[  263.990000]  [<6008c91b>] process_one_work+0x1eb/0x490
[  263.990000]  [<607f5aa0>] ? __schedule+0x0/0x620
[  263.990000]  [<6008e000>] ? wq_worker_running+0x10/0x40
[  263.990000]  [<6008c730>] ? process_one_work+0x0/0x490
[  263.990000]  [<6008cc06>] worker_thread+0x46/0x670
[  263.990000]  [<600931c1>] ? __kthread_parkme+0xa1/0xd0
[  263.990000]  [<6008cbc0>] ? worker_thread+0x0/0x670
[  263.990000]  [<6008cbc0>] ? worker_thread+0x0/0x670
[  263.990000]  [<60093bc4>] kthread+0x194/0x1c0
[  263.990000]  [<6002a091>] new_thread_handler+0x81/0xc0


A.

> 
> A.
> 
>>
>>>
>>>>
>>>> Thanks,
>>>> Ritesh
>>>>
>>>>
>>>> _______________________________________________
>>>> linux-um mailing list
>>>> linux-um@lists.infradead.org
>>>> http://lists.infradead.org/mailman/listinfo/linux-um
>>>>
>>>
>>
> 

-- 
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: uml segfault during I/O

[Johannes Berg]

On Fri, 2019-10-18 at 08:35 +0100, Anton Ivanov wrote:
> 
> Still happening with 5.2.21, albeit a bit more difficult to reproduce.

Just randomly reviewing the code, isn't there a bug in io_thread()?

--- a/arch/um/drivers/ubd_kern.c
+++ b/arch/um/drivers/ubd_kern.c
@@ -1602,7 +1602,8 @@ int io_thread(void *arg)
                written = 0;
 
                do {
-                       res = os_write_file(kernel_fd, ((char *) io_req_buffer) + written, n);
+                       res = os_write_file(kernel_fd, ((char *) io_req_buffer) + written,
+                                           n - written);
                        if (res >= 0) {
                                written += res;
                        }

johannes


_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: uml segfault during I/O

[Anton Ivanov]

On 18/10/2019 09:57, Johannes Berg wrote:
> On Fri, 2019-10-18 at 08:35 +0100, Anton Ivanov wrote:
>>
>> Still happening with 5.2.21, albeit a bit more difficult to reproduce.
> 
> Just randomly reviewing the code, isn't there a bug in io_thread()?

Indeed. Well spotted. It is good that a short write to a block device is 
so rare :)

> 
> --- a/arch/um/drivers/ubd_kern.c
> +++ b/arch/um/drivers/ubd_kern.c
> @@ -1602,7 +1602,8 @@ int io_thread(void *arg)
>                  written = 0;
>   
>                  do {
> -                       res = os_write_file(kernel_fd, ((char *) io_req_buffer) + written, n);
> +                       res = os_write_file(kernel_fd, ((char *) io_req_buffer) + written,
> +                                           n - written);
>                          if (res >= 0) {
>                                  written += res;
>                          }
> 
> johannes
> 
> 
> _______________________________________________
> linux-um mailing list
> linux-um@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-um
> 

I have verified this.

It is always in a requeue and always after the io thread has returned 
EAGAIN on the attempt to submit a request.

That part of the code has been tested very heavily before and it works 
fine in 4.19

-- 
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: uml segfault during I/O

[Anton Ivanov]

On 18/10/2019 10:13, Anton Ivanov wrote:
> 
> 
> On 18/10/2019 09:57, Johannes Berg wrote:
>> On Fri, 2019-10-18 at 08:35 +0100, Anton Ivanov wrote:
>>>
>>> Still happening with 5.2.21, albeit a bit more difficult to reproduce.
>>
>> Just randomly reviewing the code, isn't there a bug in io_thread()?
> 
> Indeed. Well spotted. It is good that a short write to a block device is 
> so rare :)
> 
>>
>> --- a/arch/um/drivers/ubd_kern.c
>> +++ b/arch/um/drivers/ubd_kern.c
>> @@ -1602,7 +1602,8 @@ int io_thread(void *arg)
>>                  written = 0;
>>                  do {
>> -                       res = os_write_file(kernel_fd, ((char *) 
>> io_req_buffer) + written, n);
>> +                       res = os_write_file(kernel_fd, ((char *) 
>> io_req_buffer) + written,
>> +                                           n - written);
>>                          if (res >= 0) {
>>                                  written += res;
>>                          }
>>
>> johannes
>>
>>
>> _______________________________________________
>> linux-um mailing list
>> linux-um@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-um
>>
> 
> I have verified this.
> 
> It is always in a requeue and always after the io thread has returned 
> EAGAIN on the attempt to submit a request.
> 
> That part of the code has been tested very heavily before and it works 
> fine in 4.19
> 

Applied and it is not it.

This is something in the blk-mq code and we should forward the whole 
thread to the blk_mq maintainers.

A




-- 
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: uml segfault during I/O

[Anton Ivanov]

Adding Jens and Christoph

On 18/10/2019 08:35, Anton Ivanov wrote:
> 
> 
> On 17/10/2019 17:29, Anton Ivanov wrote:
>>
>>
>> On 17/10/2019 16:03, Anton Ivanov wrote:
>>>
>>>
>>> On 17/10/2019 14:33, Anton Ivanov wrote:
>>>>
>>>>
>>>> On 17/10/2019 14:30, Ritesh Raj Sarraf wrote:
>>>>> On Thu, 2019-10-17 at 14:02 +0100, Anton Ivanov wrote:
>>>>>> Looking into that. I have not run into anything like that, but I have
>>>>>> not used any of the legacy networking for 5 odd years now.
>>>>>>
>>>>>
>>>>> Do you think this is related to networking ? I ask because there 
>>>>> was no
>>>>> network activity going on.
>>>>
>>>> no, it's disk somewhere. I managed to reproduce it with 5.2 stock on 
>>>> Debian 5.2 host.
>>>>
>>>>>
>>>>> apt is just an example. The packages were all already downloaded and
>>>>> there was no network activity. Rather, there was block I/O.
>>>>
>>>> I am looking into that.
>>>>
>>>>>
>>>>> One thing I noticed, which may or may not be useful to this report. I
>>>>> was booting the uml guest and immediately logging into it and running
>>>>> the block I/O. And the segfault would occur.
>>>>>
>>>>> If I, instead, booted the uml vm and let it remain idle for a 
>>>>> minute or
>>>>> so and then did the I/O, it worked fine. So I am not sure if there is
>>>>> any lazy initialization happening in the background which gets
>>>>> corrupted during immediate hot boot I/O.
>>>>
>>>> Interesting...
>>>>
>>>>>
>>>>> On the other hand, if you think there can be any number of commands to
>>>>> run locally that could generate more information, please assist me so.
>>>>
>>>> As I said - I managed to reproduce it, I am looking at it. In first 
>>>> instance I am trying with a couple of version up/down to see if this 
>>>> is 5.2 specific.
>>>
>>> I cannot even get it to start on 5.4-rc1, 5.3 shows the same symptoms.
>>
>> This is something outside the UBD driver as such. There were no 
>> notable changes to it since we ported it to block-mq and added DISCARD 
>> which was quite a while back.
>>
>> I am going to check the other usual suspects such as IRQs, but that is 
>> something I test quite extensively when working on the networking side.
>>
>> So I suspect that this is something outside UML which is showing only 
>> in UML for some reason.
> 
> Still happening with 5.2.21, albeit a bit more difficult to reproduce.
> 
> Looking at the backtraces it is ALWAYS a result of a re-queue.
> 
> [  263.990000]  [<60440633>] blk_mq_dispatch_rq_list+0xf3/0x5f0
> [  263.990000]  [<60440501>] ? blk_mq_get_driver_tag+0xc1/0x100
> [  263.990000]  [<60440540>] ? blk_mq_dispatch_rq_list+0x0/0x5f0
> [  263.990000]  [<60445966>] blk_mq_do_dispatch_sched+0x66/0xe0
> [  263.990000]  [<60446107>] blk_mq_sched_dispatch_requests+0x107/0x190
> [  263.990000]  [<6043f130>] ? blk_mq_run_hw_queue+0x0/0x120
> [  263.990000]  [<6043efb4>] __blk_mq_run_hw_queue+0x74/0xd0
> [  263.990000]  [<6043f0d4>] __blk_mq_delay_run_hw_queue+0xc4/0xd0
> [  263.990000]  [<6043f1d9>] blk_mq_run_hw_queue+0xa9/0x120
> [  263.990000]  [<6043f292>] blk_mq_run_hw_queues+0x42/0x60
> [  263.990000]  [<60440bc0>] ? blk_mq_request_bypass_insert+0x0/0x90
> [  263.990000]  [<604462a0>] ? blk_mq_sched_insert_request+0x0/0x1c0
> ALWAYS >>[  263.990000]  [<604414b0>] blk_mq_requeue_work+0x160/0x170
> [  263.990000]  [<6008c91b>] process_one_work+0x1eb/0x490
> [  263.990000]  [<607f5aa0>] ? __schedule+0x0/0x620
> [  263.990000]  [<6008e000>] ? wq_worker_running+0x10/0x40
> [  263.990000]  [<6008c730>] ? process_one_work+0x0/0x490
> [  263.990000]  [<6008cc06>] worker_thread+0x46/0x670
> [  263.990000]  [<600931c1>] ? __kthread_parkme+0xa1/0xd0
> [  263.990000]  [<6008cbc0>] ? worker_thread+0x0/0x670
> [  263.990000]  [<6008cbc0>] ? worker_thread+0x0/0x670
> [  263.990000]  [<60093bc4>] kthread+0x194/0x1c0
> [  263.990000]  [<6002a091>] new_thread_handler+0x81/0xc0
> 
> 
> A.
> 
>>
>> A.
>>
>>>
>>>>
>>>>>
>>>>> Thanks,
>>>>> Ritesh
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> linux-um mailing list
>>>>> linux-um@lists.infradead.org
>>>>> http://lists.infradead.org/mailman/listinfo/linux-um
>>>>>
>>>>
>>>
>>
> 

To put a long story short we have a reproducible segfault when UML 
re-queues a block request in 5.x

This used to work in 4.x

We found a couple of minor things in UML when looking at it which we 
will fix shortly, but the root problem seems to be either in block_mq or 
in the way we are using it to re-queue requests.

-- 
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: uml segfault during I/O

[Anton Ivanov]

On 18/10/2019 10:55, Anton Ivanov wrote:
> Adding Jens and Christoph
> 
> On 18/10/2019 08:35, Anton Ivanov wrote:
>>
>>
>> On 17/10/2019 17:29, Anton Ivanov wrote:
>>>
>>>
>>> On 17/10/2019 16:03, Anton Ivanov wrote:
>>>>
>>>>
>>>> On 17/10/2019 14:33, Anton Ivanov wrote:
>>>>>
>>>>>
>>>>> On 17/10/2019 14:30, Ritesh Raj Sarraf wrote:
>>>>>> On Thu, 2019-10-17 at 14:02 +0100, Anton Ivanov wrote:
>>>>>>> Looking into that. I have not run into anything like that, but I 
>>>>>>> have
>>>>>>> not used any of the legacy networking for 5 odd years now.
>>>>>>>
>>>>>>
>>>>>> Do you think this is related to networking ? I ask because there 
>>>>>> was no
>>>>>> network activity going on.
>>>>>
>>>>> no, it's disk somewhere. I managed to reproduce it with 5.2 stock 
>>>>> on Debian 5.2 host.
>>>>>
>>>>>>
>>>>>> apt is just an example. The packages were all already downloaded and
>>>>>> there was no network activity. Rather, there was block I/O.
>>>>>
>>>>> I am looking into that.
>>>>>
>>>>>>
>>>>>> One thing I noticed, which may or may not be useful to this report. I
>>>>>> was booting the uml guest and immediately logging into it and running
>>>>>> the block I/O. And the segfault would occur.
>>>>>>
>>>>>> If I, instead, booted the uml vm and let it remain idle for a 
>>>>>> minute or
>>>>>> so and then did the I/O, it worked fine. So I am not sure if there is
>>>>>> any lazy initialization happening in the background which gets
>>>>>> corrupted during immediate hot boot I/O.
>>>>>
>>>>> Interesting...
>>>>>
>>>>>>
>>>>>> On the other hand, if you think there can be any number of 
>>>>>> commands to
>>>>>> run locally that could generate more information, please assist me 
>>>>>> so.
>>>>>
>>>>> As I said - I managed to reproduce it, I am looking at it. In first 
>>>>> instance I am trying with a couple of version up/down to see if 
>>>>> this is 5.2 specific.
>>>>
>>>> I cannot even get it to start on 5.4-rc1, 5.3 shows the same symptoms.
>>>
>>> This is something outside the UBD driver as such. There were no 
>>> notable changes to it since we ported it to block-mq and added 
>>> DISCARD which was quite a while back.
>>>
>>> I am going to check the other usual suspects such as IRQs, but that 
>>> is something I test quite extensively when working on the networking 
>>> side.
>>>
>>> So I suspect that this is something outside UML which is showing only 
>>> in UML for some reason.
>>
>> Still happening with 5.2.21, albeit a bit more difficult to reproduce.
>>
>> Looking at the backtraces it is ALWAYS a result of a re-queue.
>>
>> [  263.990000]  [<60440633>] blk_mq_dispatch_rq_list+0xf3/0x5f0
>> [  263.990000]  [<60440501>] ? blk_mq_get_driver_tag+0xc1/0x100
>> [  263.990000]  [<60440540>] ? blk_mq_dispatch_rq_list+0x0/0x5f0
>> [  263.990000]  [<60445966>] blk_mq_do_dispatch_sched+0x66/0xe0
>> [  263.990000]  [<60446107>] blk_mq_sched_dispatch_requests+0x107/0x190
>> [  263.990000]  [<6043f130>] ? blk_mq_run_hw_queue+0x0/0x120
>> [  263.990000]  [<6043efb4>] __blk_mq_run_hw_queue+0x74/0xd0
>> [  263.990000]  [<6043f0d4>] __blk_mq_delay_run_hw_queue+0xc4/0xd0
>> [  263.990000]  [<6043f1d9>] blk_mq_run_hw_queue+0xa9/0x120
>> [  263.990000]  [<6043f292>] blk_mq_run_hw_queues+0x42/0x60
>> [  263.990000]  [<60440bc0>] ? blk_mq_request_bypass_insert+0x0/0x90
>> [  263.990000]  [<604462a0>] ? blk_mq_sched_insert_request+0x0/0x1c0
>> ALWAYS >>[  263.990000]  [<604414b0>] blk_mq_requeue_work+0x160/0x170
>> [  263.990000]  [<6008c91b>] process_one_work+0x1eb/0x490
>> [  263.990000]  [<607f5aa0>] ? __schedule+0x0/0x620
>> [  263.990000]  [<6008e000>] ? wq_worker_running+0x10/0x40
>> [  263.990000]  [<6008c730>] ? process_one_work+0x0/0x490
>> [  263.990000]  [<6008cc06>] worker_thread+0x46/0x670
>> [  263.990000]  [<600931c1>] ? __kthread_parkme+0xa1/0xd0
>> [  263.990000]  [<6008cbc0>] ? worker_thread+0x0/0x670
>> [  263.990000]  [<6008cbc0>] ? worker_thread+0x0/0x670
>> [  263.990000]  [<60093bc4>] kthread+0x194/0x1c0
>> [  263.990000]  [<6002a091>] new_thread_handler+0x81/0xc0
>>
>>
>> A.
>>
>>>
>>> A.
>>>
>>>>
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Ritesh
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> linux-um mailing list
>>>>>> linux-um@lists.infradead.org
>>>>>> http://lists.infradead.org/mailman/listinfo/linux-um
>>>>>>
>>>>>
>>>>
>>>
>>
> 
> To put a long story short we have a reproducible segfault when UML 
> re-queues a block request in 5.x
> 
> This used to work in 4.x
> 
> We found a couple of minor things in UML when looking at it which we 
> will fix shortly, but the root problem seems to be either in block_mq or 
> in the way we are using it to re-queue requests.
> 

My (probably wrong) guess is that it is something related to the changes 
from 4.x to 5.x where blk-mq is no longer getting the hw context out of 
the cpu map.

For some reason it gets a null deref at some point.

I may be wrong of course.

I will not be able to pick this up again before Tuesday so if someone 
can have a go before then it will be greatly appreciated.

-- 
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: uml segfault during I/O

[Anton Ivanov]

On 18/10/2019 11:51, Anton Ivanov wrote:
>
>
> On 18/10/2019 10:55, Anton Ivanov wrote:
>> Adding Jens and Christoph
>>
>> On 18/10/2019 08:35, Anton Ivanov wrote:
>>>
>>>
>>> On 17/10/2019 17:29, Anton Ivanov wrote:
>>>>
>>>>
>>>> On 17/10/2019 16:03, Anton Ivanov wrote:
>>>>>
>>>>>
>>>>> On 17/10/2019 14:33, Anton Ivanov wrote:
>>>>>>
>>>>>>
>>>>>> On 17/10/2019 14:30, Ritesh Raj Sarraf wrote:
>>>>>>> On Thu, 2019-10-17 at 14:02 +0100, Anton Ivanov wrote:
>>>>>>>> Looking into that. I have not run into anything like that, but 
>>>>>>>> I have
>>>>>>>> not used any of the legacy networking for 5 odd years now.
>>>>>>>>
>>>>>>>
>>>>>>> Do you think this is related to networking ? I ask because there 
>>>>>>> was no
>>>>>>> network activity going on.
>>>>>>
>>>>>> no, it's disk somewhere. I managed to reproduce it with 5.2 stock 
>>>>>> on Debian 5.2 host.
>>>>>>
>>>>>>>
>>>>>>> apt is just an example. The packages were all already downloaded 
>>>>>>> and
>>>>>>> there was no network activity. Rather, there was block I/O.
>>>>>>
>>>>>> I am looking into that.
>>>>>>
>>>>>>>
>>>>>>> One thing I noticed, which may or may not be useful to this 
>>>>>>> report. I
>>>>>>> was booting the uml guest and immediately logging into it and 
>>>>>>> running
>>>>>>> the block I/O. And the segfault would occur.
>>>>>>>
>>>>>>> If I, instead, booted the uml vm and let it remain idle for a 
>>>>>>> minute or
>>>>>>> so and then did the I/O, it worked fine. So I am not sure if 
>>>>>>> there is
>>>>>>> any lazy initialization happening in the background which gets
>>>>>>> corrupted during immediate hot boot I/O.
>>>>>>
>>>>>> Interesting...
>>>>>>
>>>>>>>
>>>>>>> On the other hand, if you think there can be any number of 
>>>>>>> commands to
>>>>>>> run locally that could generate more information, please assist 
>>>>>>> me so.
>>>>>>
>>>>>> As I said - I managed to reproduce it, I am looking at it. In 
>>>>>> first instance I am trying with a couple of version up/down to 
>>>>>> see if this is 5.2 specific.
>>>>>
>>>>> I cannot even get it to start on 5.4-rc1, 5.3 shows the same 
>>>>> symptoms.
>>>>
>>>> This is something outside the UBD driver as such. There were no 
>>>> notable changes to it since we ported it to block-mq and added 
>>>> DISCARD which was quite a while back.
>>>>
>>>> I am going to check the other usual suspects such as IRQs, but that 
>>>> is something I test quite extensively when working on the 
>>>> networking side.
>>>>
>>>> So I suspect that this is something outside UML which is showing 
>>>> only in UML for some reason.
>>>
>>> Still happening with 5.2.21, albeit a bit more difficult to reproduce.
>>>
>>> Looking at the backtraces it is ALWAYS a result of a re-queue.
>>>
>>> [  263.990000]  [<60440633>] blk_mq_dispatch_rq_list+0xf3/0x5f0
>>> [  263.990000]  [<60440501>] ? blk_mq_get_driver_tag+0xc1/0x100
>>> [  263.990000]  [<60440540>] ? blk_mq_dispatch_rq_list+0x0/0x5f0
>>> [  263.990000]  [<60445966>] blk_mq_do_dispatch_sched+0x66/0xe0
>>> [  263.990000]  [<60446107>] blk_mq_sched_dispatch_requests+0x107/0x190
>>> [  263.990000]  [<6043f130>] ? blk_mq_run_hw_queue+0x0/0x120
>>> [  263.990000]  [<6043efb4>] __blk_mq_run_hw_queue+0x74/0xd0
>>> [  263.990000]  [<6043f0d4>] __blk_mq_delay_run_hw_queue+0xc4/0xd0
>>> [  263.990000]  [<6043f1d9>] blk_mq_run_hw_queue+0xa9/0x120
>>> [  263.990000]  [<6043f292>] blk_mq_run_hw_queues+0x42/0x60
>>> [  263.990000]  [<60440bc0>] ? blk_mq_request_bypass_insert+0x0/0x90
>>> [  263.990000]  [<604462a0>] ? blk_mq_sched_insert_request+0x0/0x1c0
>>> ALWAYS >>[  263.990000]  [<604414b0>] blk_mq_requeue_work+0x160/0x170
>>> [  263.990000]  [<6008c91b>] process_one_work+0x1eb/0x490
>>> [  263.990000]  [<607f5aa0>] ? __schedule+0x0/0x620
>>> [  263.990000]  [<6008e000>] ? wq_worker_running+0x10/0x40
>>> [  263.990000]  [<6008c730>] ? process_one_work+0x0/0x490
>>> [  263.990000]  [<6008cc06>] worker_thread+0x46/0x670
>>> [  263.990000]  [<600931c1>] ? __kthread_parkme+0xa1/0xd0
>>> [  263.990000]  [<6008cbc0>] ? worker_thread+0x0/0x670
>>> [  263.990000]  [<6008cbc0>] ? worker_thread+0x0/0x670
>>> [  263.990000]  [<60093bc4>] kthread+0x194/0x1c0
>>> [  263.990000]  [<6002a091>] new_thread_handler+0x81/0xc0
>>>
>>>
>>> A.
>>>
>>>>
>>>> A.
>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Ritesh
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> linux-um mailing list
>>>>>>> linux-um@lists.infradead.org
>>>>>>> http://lists.infradead.org/mailman/listinfo/linux-um
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> To put a long story short we have a reproducible segfault when UML 
>> re-queues a block request in 5.x
>>
>> This used to work in 4.x
>>
>> We found a couple of minor things in UML when looking at it which we 
>> will fix shortly, but the root problem seems to be either in block_mq 
>> or in the way we are using it to re-queue requests.
>>
>
> My (probably wrong) guess is that it is something related to the 
> changes from 4.x to 5.x where blk-mq is no longer getting the hw 
> context out of the cpu map.
>
> For some reason it gets a null deref at some point.
>
> I may be wrong of course.
>
> I will not be able to pick this up again before Tuesday so if someone 
> can have a go before then it will be greatly appreciated.
>
Unless I am mistaken It should not re-queue.

Xen does not: 
https://elixir.bootlin.com/linux/latest/source/drivers/block/xen-blkfront.c#L910

Unless I am mistaken, if you return a form of BUSY the the upper layers 
will re-queue for you.

That explains the crash as we have the same bio enqueued multiple times 
when the device stalls.

I am testing a patch and if it all tests out I will submit it shortly.

-- 

Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/


_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um