cfq-iosched.c: panic in cfq_dispatch_requests

cfq-iosched.c: panic in cfq_dispatch_requests

[Lina Lu]

Hi Vivek,

I did some test with blkio weight policy these days. The kernel panic 
once in a while. 
	
>From the calltrace, It 's in func cfq_dispatch_requests(+0x058D).
And I caught a divide by zero error with system monitor tools.
	
The panic happens when I first set 100 weight to two IO pids, then 
change the weight to 200, and begin to perform IO. The kernel panic
accurately when the IO begin.

I try to find the divide 0 code in cfq_dispatch_requests. This function
call the cfq_select_queue to select a queue, and cfq_select_queue
call cfq_choose_cfqg, then choose_service_tree(). There is a expression
of division in choose_service_tree.
	
        slice = group_slice * count /
                max_t(unsigned, cfqg->busy_queues_avg[cfqd->serving_prio],
                      cfq_group_busy_queues_wl(cfqd->serving_prio, cfqd, cfqg));

I don't know whether this code divide 0. Can you take a look of this issue?

Thanks
Lina 

Re: cfq-iosched.c: panic in cfq_dispatch_requests

[Vivek Goyal]

On Sun, Mar 20, 2011 at 03:23:23PM +0800, Lina Lu wrote:
> Hi Vivek,
> 
> I did some test with blkio weight policy these days. The kernel panic 
> once in a while. 
> 	
> >From the calltrace, It 's in func cfq_dispatch_requests(+0x058D).
> And I caught a divide by zero error with system monitor tools.

Hi Lina,

Can you please paste the full backtrace here.

> 	
> The panic happens when I first set 100 weight to two IO pids, then 
> change the weight to 200, and begin to perform IO. The kernel panic
> accurately when the IO begin.

So you notice this only if weights are changed otherwise not?

Recently Justin fixed a bug in CFQ where crash can happen when weights
are changed. This fix in Jens's block tree. Can you please apply this
patch and see if it resolves your issue too.

commit 8184f93eced1e304721c2a55c00d87d5a14f8907
Author: Justin TerAvest <teravest@google.com>
Date:   Thu Mar 17 16:12:36 2011 +0100

    cfq-iosched: Don't update group weights when on service tree

Thanks
Vivek

Re: Re: cfq-iosched.c: panic in cfq_dispatch_requests

[Lina Lu]

On 2011-03-21 22:08:19, Vivek Goyal wrote:
>On Sun, Mar 20, 2011 at 03:23:23PM +0800, Lina Lu wrote:
>> Hi Vivek,
>> 
>> I did some test with blkio weight policy these days. The kernel panic 
>> once in a while. 
>> 	
>> From the calltrace, It 's in func cfq_dispatch_requests(+0x058D).
>> And I caught a divide by zero error with system monitor tools.
>
>Hi Lina,
>
>Can you please paste the full backtrace here.
>
>> 	
>> The panic happens when I first set 100 weight to two IO pids, then 
>> change the weight to 200, and begin to perform IO. The kernel panic
>> accurately when the IO begin.
>
>So you notice this only if weights are changed otherwise not?

Hi Vivek

Thanks for this patch! I did objdump and find place that divide by zero.
It's in cfq_group_slice function, the st->total_weight is zero in my issue.
As described in the patch, this is because I change the weight when the
group on service tree. 

Thanks
Lina

>
>Recently Justin fixed a bug in CFQ where crash can happen when weights
>are changed. This fix in Jens's block tree. Can you please apply this
>patch and see if it resolves your issue too.
>
>commit 8184f93eced1e304721c2a55c00d87d5a14f8907
>Author: Justin TerAvest <teravest@google.com>
>Date:   Thu Mar 17 16:12:36 2011 +0100
>
>    cfq-iosched: Don't update group weights when on service tree
>
>Thanks
>Vivek
@

Re: Re: cfq-iosched.c: panic in cfq_dispatch_requests

[Vivek Goyal]

On Tue, Mar 22, 2011 at 11:32:47PM +0800, Lina Lu wrote:
> On 2011-03-21 22:08:19, Vivek Goyal wrote:
> >On Sun, Mar 20, 2011 at 03:23:23PM +0800, Lina Lu wrote:
> >> Hi Vivek,
> >> 
> >> I did some test with blkio weight policy these days. The kernel panic 
> >> once in a while. 
> >> 	
> >> From the calltrace, It 's in func cfq_dispatch_requests(+0x058D).
> >> And I caught a divide by zero error with system monitor tools.
> >
> >Hi Lina,
> >
> >Can you please paste the full backtrace here.
> >
> >> 	
> >> The panic happens when I first set 100 weight to two IO pids, then 
> >> change the weight to 200, and begin to perform IO. The kernel panic
> >> accurately when the IO begin.
> >
> >So you notice this only if weights are changed otherwise not?
> 
> Hi Vivek
> 
> Thanks for this patch! I did objdump and find place that divide by zero.
> It's in cfq_group_slice function, the st->total_weight is zero in my issue.
> As described in the patch, this is because I change the weight when the
> group on service tree. 

Ok, so this patch did solve the issue you are facing? 

Thanks
Vivek

Re: Re: Re: cfq-iosched.c: panic in cfq_dispatch_requests

[Lina Lu]

On 2011-03-22 23:39:48, Vivek Goyal wrote:
>On Tue, Mar 22, 2011 at 11:32:47PM +0800, Lina Lu wrote:
>> On 2011-03-21 22:08:19, Vivek Goyal wrote:
>> >On Sun, Mar 20, 2011 at 03:23:23PM +0800, Lina Lu wrote:
>> >> Hi Vivek,
>> >> 
>> >> I did some test with blkio weight policy these days. The kernel panic 
>> >> once in a while. 
>> >> 	
>> >> From the calltrace, It 's in func cfq_dispatch_requests(+0x058D).
>> >> And I caught a divide by zero error with system monitor tools.
>> >
>> >Hi Lina,
>> >
>> >Can you please paste the full backtrace here.
>> >
>> >> 	
>> >> The panic happens when I first set 100 weight to two IO pids, then 
>> >> change the weight to 200, and begin to perform IO. The kernel panic
>> >> accurately when the IO begin.
>> >
>> >So you notice this only if weights are changed otherwise not?
>> 
>> Hi Vivek
>> 
>> Thanks for this patch! I did objdump and find place that divide by zero.
>> It's in cfq_group_slice function, the st->total_weight is zero in my issue.
>> As described in the patch, this is because I change the weight when the
>> group on service tree. 
>
>Ok, so this patch did solve the issue you are facing? 
>

Yes, this patch did solve the issue. 

And I will do more test these days, if there is any other problem, I will keep
you know.

Thanks
Lina