as_arq scheduler alloc with 2.6.0-test8-mm1

as_arq scheduler alloc with 2.6.0-test8-mm1

[Mr Amit Patel]

Hi,
 
I am using 2.6.0-test8-mm1 kernel. I am using qlogic
driver patch with 
this kernel. Every time I insert qlogic driver my
as_arq pool size 
increases by 2000 objects of each size 76 bytes. I
have kgdb setup and I 
tried to put break point in various function in
as_iosched.c to see if I 
can see where it is getting allocated, but It never
hit the breakpoint. 
Can anyone tell me whether its as-i/o scheduler
problem or something in 
qlogic driver? How do I debug is problem?
 
After doing just insmod/rmmod for 20/30 times my
machine runs out of 
free pages. If I dont try to insert qlogic driver my
system runs fine for 
days without any problem. Following is the output of
/proc/slabinfo for 
as_arq after doing insmod/rmmod.

Any help as to how to debug this problem is
appreciated. 
 
Thanks
Amit
 
[root@Host200-w2k root]# cat /proc/slabinfo |grep
as_arq
as_arq              6251   6300     76   50    1 :
tunables   32   16    
0 : slabdata    126    126      0 : globalstat  165249
  6300  1708 
1538    0    1   82 : cpustat 245396  11366 247012  
3502
[root@Host200-w2k root]# cat /proc/slabinfo |grep
as_arq
as_arq              8291   8300     76   50    1 :
tunables   32   16    
0 : slabdata    166    166      0 : globalstat  167299
  8298  1749 
1539    0    1   82 : cpustat 247381  11530 247121  
3502
[root@Host200-w2k root]# 

__________________________________
Do you Yahoo!?
Exclusive Video Premiere - Britney Spears
http://launch.yahoo.com/promos/britneyspears/

Re: as_arq scheduler alloc with 2.6.0-test8-mm1

[Andrew Morton]

Mr Amit Patel <patelamitv@yahoo.com> wrote:
>
> I am using 2.6.0-test8-mm1 kernel. I am using qlogic
>  driver patch with 
>  this kernel

What qlogic driver patch is this?

Re: as_arq scheduler alloc with 2.6.0-test8-mm1

[Mr Amit Patel]

Hi Andrew,

Qlogic driver I am using is from Micheal Bligh mjb1
patch. I have not applied the full patch just the
qlogic driver.

Thanks
Amit
--- Andrew Morton <akpm@osdl.org> wrote:
> Mr Amit Patel <patelamitv@yahoo.com> wrote:
> >
> > I am using 2.6.0-test8-mm1 kernel. I am using
> qlogic
> >  driver patch with 
> >  this kernel
> 
> What qlogic driver patch is this?
> 


__________________________________
Do you Yahoo!?
Exclusive Video Premiere - Britney Spears
http://launch.yahoo.com/promos/britneyspears/

Re: as_arq scheduler alloc with 2.6.0-test8-mm1

[Mr Amit Patel]

Hi Andrew,

The qlogic driver is for Fibre Channel HBA QLA2342.
This is a beta driver which is part of the mjb1 patch
against 2.6.0-test8. As a part of driver insmod,
driver tries to find fiber channel device and maps it
to scsi block device. Actually I don't have any fibre
channel target attached, so driver does not find any
scsi devices and discovery finishes without adding any
block device. 

I am trying to go through driver scsi_scan process and
see when does actual allocation from as_arq happens.
But for some reason after going to kgdb I get SIGEMT
and I cannot debug further. What is causing SIGEMT
cause after doing some search looks like its actually
SIGUSR but linux treats it as SIGEMT. Is there any way
to prevent SIGEMT when I want to use kgdb ? 

Thanks for your help,

Amit
--- Andrew Morton <akpm@osdl.org> wrote:
> Mr Amit Patel <patelamitv@yahoo.com> wrote:
> >
> > I am using 2.6.0-test8-mm1 kernel. I am using
> qlogic
> >  driver patch with 
> >  this kernel
> 
> What qlogic driver patch is this?
> 


__________________________________
Do you Yahoo!?
Exclusive Video Premiere - Britney Spears
http://launch.yahoo.com/promos/britneyspears/

Re: as_arq scheduler alloc with 2.6.0-test8-mm1

[Andrew Morton]

Mr Amit Patel <patelamitv@yahoo.com> wrote:
>
> Hi Andrew,
> 
> The qlogic driver is for Fibre Channel HBA QLA2342.
> This is a beta driver which is part of the mjb1 patch
> against 2.6.0-test8. As a part of driver insmod,
> driver tries to find fiber channel device and maps it
> to scsi block device. Actually I don't have any fibre
> channel target attached, so driver does not find any
> scsi devices and discovery finishes without adding any
> block device. 
> 
> I am trying to go through driver scsi_scan process and
> see when does actual allocation from as_arq happens.
> But for some reason after going to kgdb I get SIGEMT
> and I cannot debug further. What is causing SIGEMT
> cause after doing some search looks like its actually
> SIGUSR but linux treats it as SIGEMT. Is there any way
> to prevent SIGEMT when I want to use kgdb ? 
> 

SIGEMT is kgdb's way of telling you the kernel oopsed.  The different
signal types used to convey useful information (a different type for NMI
watchdog expiry, for example).  But George then randomised them and I've
never worked out the new scheme...

kgdb is difficult to use with kernel modules.  There's probably a way of
doing it in 2.6 but I've never sat down and worked it out.  But as you
appear to be working on the SCSI core that's not a problem.

Re: as_arq scheduler alloc with 2.6.0-test8-mm1

[Nick Piggin]

Mr Amit Patel wrote:

>Hi Andrew,
>
>The qlogic driver is for Fibre Channel HBA QLA2342.
>This is a beta driver which is part of the mjb1 patch
>against 2.6.0-test8. As a part of driver insmod,
>driver tries to find fiber channel device and maps it
>to scsi block device. Actually I don't have any fibre
>channel target attached, so driver does not find any
>scsi devices and discovery finishes without adding any
>block device. 
>
>I am trying to go through driver scsi_scan process and
>see when does actual allocation from as_arq happens.
>But for some reason after going to kgdb I get SIGEMT
>and I cannot debug further. What is causing SIGEMT
>cause after doing some search looks like its actually
>SIGUSR but linux treats it as SIGEMT. Is there any way
>to prevent SIGEMT when I want to use kgdb ? 
>
>Thanks for your help,
>

Hi Amit,

I'm a little bit busy to look at this now, however someone
is looking into all these refcounting problems.

If you would like to narrow it down a bit, check that the
request queues that are allocated are all released when the
driver is unloaded (drivers/block/ll_rw_blk.c blk_alloc_queue
and blk_cleanup_queue). Just stick a couple of printks there
if your debugger isn't working.

Re: as_arq scheduler alloc with 2.6.0-test8-mm1

[Amit Patel]

Hi 

After doing some printk debugging looks like for each
block device we allocate elevator structure. But
during cleanup the elevator->elevator_data is never
freed. In printk I put printk in as-iosched.c:as_alloc
where I see elevator_data is allocated from as_arq
pool and it should have been freed as a part of block
device cleanup through elevator_release function. I
never see it coming to elevatore_release function. 

I will try to walk through code some more and see if I
can figure out who was supposed to call
elevator_release as part of cleanup. Let me know if I
am going on wrong track here.

Thanks
Amit
--- Nick Piggin <piggin@cyberone.com.au> wrote:
> 
> 
> Mr Amit Patel wrote:
> 
> >Hi Andrew,
> >
> >The qlogic driver is for Fibre Channel HBA QLA2342.
> >This is a beta driver which is part of the mjb1
> patch
> >against 2.6.0-test8. As a part of driver insmod,
> >driver tries to find fiber channel device and maps
> it
> >to scsi block device. Actually I don't have any
> fibre
> >channel target attached, so driver does not find
> any
> >scsi devices and discovery finishes without adding
> any
> >block device. 
> >
> >I am trying to go through driver scsi_scan process
> and
> >see when does actual allocation from as_arq
> happens.
> >But for some reason after going to kgdb I get
> SIGEMT
> >and I cannot debug further. What is causing SIGEMT
> >cause after doing some search looks like its
> actually
> >SIGUSR but linux treats it as SIGEMT. Is there any
> way
> >to prevent SIGEMT when I want to use kgdb ? 
> >
> >Thanks for your help,
> >
> 
> Hi Amit,
> 
> I'm a little bit busy to look at this now, however
> someone
> is looking into all these refcounting problems.
> 
> If you would like to narrow it down a bit, check
> that the
> request queues that are allocated are all released
> when the
> driver is unloaded (drivers/block/ll_rw_blk.c
> blk_alloc_queue
> and blk_cleanup_queue). Just stick a couple of
> printks there
> if your debugger isn't working.
> 
> 


__________________________________
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
http://shopping.yahoo.com

Re: as_arq scheduler alloc with 2.6.0-test8-mm1

[Amit Patel]

--- Nick Piggin <piggin@cyberone.com.au> wrote:
> 
> No, you're on the right track. Actually, in the
> block device
> you will allocate the queue, and the queue will then
> allocate
> the elevator. First see if the queue is being
> properly allocated
> and freed. If it is not, then the problem is in the
> SCSI layer.
> 
> 

Hi Nick,

During scsi_scan.c:scsi_probe_and_add_lun it allocates
scsi device and request_queue to send the
scsi_request. If during this scan it finds no device
attached to target it will deallocate request__queue.
But since there was no disk found it never called
add_disk and so it never go through blk_register_queue
function to register elevator queue. So during clean
up it just calles blk_cleanup_queue which does _not_
clean up elevator_data. 

This is just what I think is happening by looking at
the code. I might be missing something. But in this
case either blk_cleanup_queue should clean elevator
also as part of cleanup instead of it getting cleaned
up through blk_unregister_queue or scsi  layer needs
to make some changes during scan. I do not know what
implication it might have if cleanup is done as part
of blk_cleanup_queue on other kind of block devices. I
will try to put this cleanup as part of
blk_cleanup_queue to see if it works. 

Thanks
Amit
> 
> Amit Patel wrote:
> 
> >Hi 
> >
> >After doing some printk debugging looks like for
> each
> >block device we allocate elevator structure. But
> >during cleanup the elevator->elevator_data is never
> >freed. In printk I put printk in
> as-iosched.c:as_alloc
> >where I see elevator_data is allocated from as_arq
> >pool and it should have been freed as a part of
> block
> >device cleanup through elevator_release function. I
> >never see it coming to elevatore_release function. 
> >
> >I will try to walk through code some more and see
> if I
> >can figure out who was supposed to call
> >elevator_release as part of cleanup. Let me know if
> I
> >am going on wrong track here.
> >
> >Thanks
> >Amit
> >
> 


__________________________________
Do you Yahoo!?
Exclusive Video Premiere - Britney Spears
http://launch.yahoo.com/promos/britneyspears/

Re: as_arq scheduler alloc with 2.6.0-test8-mm1

[Nick Piggin]

Amit Patel wrote:

>--- Nick Piggin <piggin@cyberone.com.au> wrote:
>
>>No, you're on the right track. Actually, in the
>>block device
>>you will allocate the queue, and the queue will then
>>allocate
>>the elevator. First see if the queue is being
>>properly allocated
>>and freed. If it is not, then the problem is in the
>>SCSI layer.
>>
>>
>>
>
>Hi Nick,
>
>During scsi_scan.c:scsi_probe_and_add_lun it allocates
>scsi device and request_queue to send the
>scsi_request. If during this scan it finds no device
>attached to target it will deallocate request__queue.
>But since there was no disk found it never called
>add_disk and so it never go through blk_register_queue
>function to register elevator queue. So during clean
>up it just calles blk_cleanup_queue which does _not_
>clean up elevator_data. 
>
>This is just what I think is happening by looking at
>the code. I might be missing something. But in this
>case either blk_cleanup_queue should clean elevator
>also as part of cleanup instead of it getting cleaned
>up through blk_unregister_queue or scsi  layer needs
>to make some changes during scan. I do not know what
>implication it might have if cleanup is done as part
>of blk_cleanup_queue on other kind of block devices. I
>will try to put this cleanup as part of
>blk_cleanup_queue to see if it works. 
>

Yeah that sounds right. This is due to elv-select.patch. Its
quite a rats nest in there though :(

What should be happening is blk_alloc_queue should register
the queue kobject, and blk_init_queue should allocate the
the elevator kobject (which should be registered in the process).

add_disk should just be taking a reference on the queue, and
add a symlink in sysfs to the real queue (I think). I have a
patch to do this, but it is a bit ugly because I don't know
where in sysfs to put the queue object.

Anyway, if this isn't going to be done before 2.6.0, I guess
Andrew you should drop by elv-select patches for now. That
doesn't fix anything, but it hides some of the problems.