[3.1-rc6] kmalloc(64) leak from IDE

[3.1-rc6] kmalloc(64) leak from IDE

[Simon Kirby]

All sorts of fun with 3.1-rc!

On an older x86 box still using the IDE code, I'm seeing a kmalloc(64)
leak (according to slabtop) that basically OOM'd the box in a few days
(640 MB of RAM). This has popped up since 2.6.36, which ran for a long
time on this box with no problems. Issues seen on -rc5, so I rebuilt with
CONFIG_DEBUG_KMEMLEAK on 9d037a777695993ec7437e5f451647dea7919d4c and
/sys/kernel/debug/kmemleak filled up with size 64 traces involving IDE
requests. Every trace seems to contain idedisk_prep_fn():

unreferenced object 0xe30c00c0 (size 64):
  comm "md6_raid1", pid 255, jiffies 4294903935 (age 23889.704s)
  hex dump (first 32 bytes):
    00 00 00 00 00 00 00 ea 00 00 00 00 00 00 00 00  ................
    7e 00 00 00 20 00 00 00 01 00 00 00 00 00 00 00  ~... ...........
  backtrace:
    [<c1495c37>] kmemleak_alloc+0x27/0x50
    [<c10b2f4a>] kmem_cache_alloc_trace+0x8a/0x120
    [<c131fe47>] idedisk_prep_fn+0x37/0xf0
    [<c12758b3>] blk_peek_request+0xa3/0x1e0
    [<c1311f15>] __ide_requeue_and_plug+0x25/0x30
    [<c131257d>] do_ide_request+0x3d/0x4e0
    [<c1270ff4>] __blk_run_queue+0x14/0x20
    [<c127699c>] __make_request+0x21c/0x290
    [<c1274f26>] generic_make_request+0x1a6/0x490
    [<c127526c>] submit_bio+0x5c/0xd0
    [<c13a651b>] md_super_write+0x6b/0x80
    [<c13a67ec>] md_update_sb+0x2bc/0x540
    [<c13a7d51>] md_check_recovery+0x2c1/0x5f0
    [<c138abce>] raid1d+0x2e/0xd90
    [<c13a58d5>] md_thread+0xe5/0x110
    [<c1047df4>] kthread+0x74/0x80

unreferenced object 0xc1c3d900 (size 64):
  comm "hardirq", pid 0, jiffies 5819438 (age 829.636s)
  hex dump (first 32 bytes):
    00 00 00 00 00 00 00 ea 00 00 00 00 00 00 00 00  ................
    7e 00 00 00 20 00 00 00 01 00 00 00 00 00 00 00  ~... ...........
  backtrace:
    [<c1495c37>] kmemleak_alloc+0x27/0x50
    [<c10b2f4a>] kmem_cache_alloc_trace+0x8a/0x120
    [<c131fe47>] idedisk_prep_fn+0x37/0xf0
    [<c12758b3>] blk_peek_request+0xa3/0x1e0
    [<c1311f15>] __ide_requeue_and_plug+0x25/0x30
    [<c1311f2f>] ide_requeue_and_plug+0xf/0x20
    [<c1311fb8>] ide_intr+0x78/0x1e0
    [<c1065c14>] handle_irq_event_percpu+0x54/0x1d0
    [<c1065dac>] handle_irq_event+0x1c/0x30
    [<c1067b8c>] handle_level_irq+0x4c/0xa0
    [<ffffffff>] 0xffffffff

idedisk_prep_fn() seems to allocate a command and return it as
rq->special, but I'm not following what happens after that.

Simon-

Re: [3.1-rc6] kmalloc(64) leak from IDE

[Borislav Petkov]

On Thu, Sep 22, 2011 at 12:26:44AM -0700, Simon Kirby wrote:
> All sorts of fun with 3.1-rc!
> 
> On an older x86 box still using the IDE code, I'm seeing a kmalloc(64)
> leak (according to slabtop) that basically OOM'd the box in a few days
> (640 MB of RAM). This has popped up since 2.6.36, which ran for a long
> time on this box with no problems. Issues seen on -rc5, so I rebuilt with
> CONFIG_DEBUG_KMEMLEAK on 9d037a777695993ec7437e5f451647dea7919d4c and
> /sys/kernel/debug/kmemleak filled up with size 64 traces involving IDE
> requests. Every trace seems to contain idedisk_prep_fn():
> 
> unreferenced object 0xe30c00c0 (size 64):
>   comm "md6_raid1", pid 255, jiffies 4294903935 (age 23889.704s)
>   hex dump (first 32 bytes):
>     00 00 00 00 00 00 00 ea 00 00 00 00 00 00 00 00  ................
>     7e 00 00 00 20 00 00 00 01 00 00 00 00 00 00 00  ~... ...........
>   backtrace:
>     [<c1495c37>] kmemleak_alloc+0x27/0x50
>     [<c10b2f4a>] kmem_cache_alloc_trace+0x8a/0x120
>     [<c131fe47>] idedisk_prep_fn+0x37/0xf0
>     [<c12758b3>] blk_peek_request+0xa3/0x1e0
>     [<c1311f15>] __ide_requeue_and_plug+0x25/0x30
>     [<c131257d>] do_ide_request+0x3d/0x4e0
>     [<c1270ff4>] __blk_run_queue+0x14/0x20
>     [<c127699c>] __make_request+0x21c/0x290
>     [<c1274f26>] generic_make_request+0x1a6/0x490
>     [<c127526c>] submit_bio+0x5c/0xd0
>     [<c13a651b>] md_super_write+0x6b/0x80
>     [<c13a67ec>] md_update_sb+0x2bc/0x540
>     [<c13a7d51>] md_check_recovery+0x2c1/0x5f0
>     [<c138abce>] raid1d+0x2e/0xd90
>     [<c13a58d5>] md_thread+0xe5/0x110
>     [<c1047df4>] kthread+0x74/0x80
> 
> unreferenced object 0xc1c3d900 (size 64):
>   comm "hardirq", pid 0, jiffies 5819438 (age 829.636s)
>   hex dump (first 32 bytes):
>     00 00 00 00 00 00 00 ea 00 00 00 00 00 00 00 00  ................
>     7e 00 00 00 20 00 00 00 01 00 00 00 00 00 00 00  ~... ...........
>   backtrace:
>     [<c1495c37>] kmemleak_alloc+0x27/0x50
>     [<c10b2f4a>] kmem_cache_alloc_trace+0x8a/0x120
>     [<c131fe47>] idedisk_prep_fn+0x37/0xf0
>     [<c12758b3>] blk_peek_request+0xa3/0x1e0
>     [<c1311f15>] __ide_requeue_and_plug+0x25/0x30
>     [<c1311f2f>] ide_requeue_and_plug+0xf/0x20
>     [<c1311fb8>] ide_intr+0x78/0x1e0
>     [<c1065c14>] handle_irq_event_percpu+0x54/0x1d0
>     [<c1065dac>] handle_irq_event+0x1c/0x30
>     [<c1067b8c>] handle_level_irq+0x4c/0xa0
>     [<ffffffff>] 0xffffffff
> 
> idedisk_prep_fn() seems to allocate a command and return it as
> rq->special, but I'm not following what happens after that.

AFAIR, that's blk_peek_request - it calls q->prep_rq_fn which is
idedisk_prep_fn() and it unconditionally allocates those ide_cmd's
without freeing them and I can imagine the upper layer requeue one
request a couple of times back-to-back, leading to the leaks.

So maybe the following could work, it is a stab in the dark for all I
know:

--
diff --git a/drivers/ide/ide-disk.c b/drivers/ide/ide-disk.c
index 274798068a54..16f69be820c7 100644
--- a/drivers/ide/ide-disk.c
+++ b/drivers/ide/ide-disk.c
@@ -435,7 +435,12 @@ static int idedisk_prep_fn(struct request_queue *q, struct request *rq)
 	if (!(rq->cmd_flags & REQ_FLUSH))
 		return BLKPREP_OK;
 
-	cmd = kzalloc(sizeof(*cmd), GFP_ATOMIC);
+	if (rq->special) {
+		cmd = rq->special;
+		memset(cmd, 0, sizeof(*cmd));
+	} else {
+		cmd = kzalloc(sizeof(*cmd), GFP_ATOMIC);
+	}
 
 	/* FIXME: map struct ide_taskfile on rq->cmd[] */
 	BUG_ON(cmd == NULL);
--

Can you rerun it with kmemleak enabled and check whether it still
triggers?

Also, I'm sure you know IDE is deprecated, so what are the chances of
moving this box to libata? Also, can you send me your .config pls?

Thanks.

-- 
Regards/Gruss,
    Boris.

Re: [3.1-rc6] kmalloc(64) leak from IDE

[Simon Kirby]

On Thu, Sep 22, 2011 at 10:48:11AM +0200, Borislav Petkov wrote:

> On Thu, Sep 22, 2011 at 12:26:44AM -0700, Simon Kirby wrote:
> > All sorts of fun with 3.1-rc!
> > 
> > On an older x86 box still using the IDE code, I'm seeing a kmalloc(64)
> > leak (according to slabtop) that basically OOM'd the box in a few days
> > (640 MB of RAM). This has popped up since 2.6.36, which ran for a long
> > time on this box with no problems. Issues seen on -rc5, so I rebuilt with
> > CONFIG_DEBUG_KMEMLEAK on 9d037a777695993ec7437e5f451647dea7919d4c and
> > /sys/kernel/debug/kmemleak filled up with size 64 traces involving IDE
> > requests. Every trace seems to contain idedisk_prep_fn():
> > 
> > unreferenced object 0xe30c00c0 (size 64):
> >   comm "md6_raid1", pid 255, jiffies 4294903935 (age 23889.704s)
> >   hex dump (first 32 bytes):
> >     00 00 00 00 00 00 00 ea 00 00 00 00 00 00 00 00  ................
> >     7e 00 00 00 20 00 00 00 01 00 00 00 00 00 00 00  ~... ...........
> >   backtrace:
> >     [<c1495c37>] kmemleak_alloc+0x27/0x50
> >     [<c10b2f4a>] kmem_cache_alloc_trace+0x8a/0x120
> >     [<c131fe47>] idedisk_prep_fn+0x37/0xf0
> >     [<c12758b3>] blk_peek_request+0xa3/0x1e0
> >     [<c1311f15>] __ide_requeue_and_plug+0x25/0x30
> >     [<c131257d>] do_ide_request+0x3d/0x4e0
> >     [<c1270ff4>] __blk_run_queue+0x14/0x20
> >     [<c127699c>] __make_request+0x21c/0x290
> >     [<c1274f26>] generic_make_request+0x1a6/0x490
> >     [<c127526c>] submit_bio+0x5c/0xd0
> >     [<c13a651b>] md_super_write+0x6b/0x80
> >     [<c13a67ec>] md_update_sb+0x2bc/0x540
> >     [<c13a7d51>] md_check_recovery+0x2c1/0x5f0
> >     [<c138abce>] raid1d+0x2e/0xd90
> >     [<c13a58d5>] md_thread+0xe5/0x110
> >     [<c1047df4>] kthread+0x74/0x80
> > 
> > unreferenced object 0xc1c3d900 (size 64):
> >   comm "hardirq", pid 0, jiffies 5819438 (age 829.636s)
> >   hex dump (first 32 bytes):
> >     00 00 00 00 00 00 00 ea 00 00 00 00 00 00 00 00  ................
> >     7e 00 00 00 20 00 00 00 01 00 00 00 00 00 00 00  ~... ...........
> >   backtrace:
> >     [<c1495c37>] kmemleak_alloc+0x27/0x50
> >     [<c10b2f4a>] kmem_cache_alloc_trace+0x8a/0x120
> >     [<c131fe47>] idedisk_prep_fn+0x37/0xf0
> >     [<c12758b3>] blk_peek_request+0xa3/0x1e0
> >     [<c1311f15>] __ide_requeue_and_plug+0x25/0x30
> >     [<c1311f2f>] ide_requeue_and_plug+0xf/0x20
> >     [<c1311fb8>] ide_intr+0x78/0x1e0
> >     [<c1065c14>] handle_irq_event_percpu+0x54/0x1d0
> >     [<c1065dac>] handle_irq_event+0x1c/0x30
> >     [<c1067b8c>] handle_level_irq+0x4c/0xa0
> >     [<ffffffff>] 0xffffffff
> > 
> > idedisk_prep_fn() seems to allocate a command and return it as
> > rq->special, but I'm not following what happens after that.
> 
> AFAIR, that's blk_peek_request - it calls q->prep_rq_fn which is
> idedisk_prep_fn() and it unconditionally allocates those ide_cmd's
> without freeing them and I can imagine the upper layer requeue one
> request a couple of times back-to-back, leading to the leaks.
> 
> So maybe the following could work, it is a stab in the dark for all I
> know:
> 
> --
> diff --git a/drivers/ide/ide-disk.c b/drivers/ide/ide-disk.c
> index 274798068a54..16f69be820c7 100644
> --- a/drivers/ide/ide-disk.c
> +++ b/drivers/ide/ide-disk.c
> @@ -435,7 +435,12 @@ static int idedisk_prep_fn(struct request_queue *q, struct request *rq)
>  	if (!(rq->cmd_flags & REQ_FLUSH))
>  		return BLKPREP_OK;
>  
> -	cmd = kzalloc(sizeof(*cmd), GFP_ATOMIC);
> +	if (rq->special) {
> +		cmd = rq->special;
> +		memset(cmd, 0, sizeof(*cmd));
> +	} else {
> +		cmd = kzalloc(sizeof(*cmd), GFP_ATOMIC);
> +	}
>  
>  	/* FIXME: map struct ide_taskfile on rq->cmd[] */
>  	BUG_ON(cmd == NULL);
> --
> 
> Can you rerun it with kmemleak enabled and check whether it still
> triggers?

Yes, that seems to have made it stop complaining about the IDE path.
All I see from kmemleak now is:

unreferenced object 0xe7481a00 (size 256):
  comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
  hex dump (first 32 bytes):
    00 00 00 28 ff ff ef ff 60 78 4e e7 00 02 00 00  ...(....`xN.....
    47 01 f8 0c f8 0c 01 08 00 00 00 00 0c 03 00 00  G...............
  backtrace:
    [<c1495c47>] kmemleak_alloc+0x27/0x50
    [<c10b3563>] __kmalloc+0xf3/0x1c0
    [<c149e0b0>] pci_acpi_scan_root+0x11e/0x272
    [<c149916b>] acpi_pci_root_add+0x163/0x256
    [<c12adddc>] acpi_device_probe+0x3a/0xf4
    [<c1302e38>] driver_probe_device+0x68/0x160
    [<c1302fb9>] __driver_attach+0x89/0x90
    [<c1302718>] bus_for_each_dev+0x48/0x70
    [<c1302cc9>] driver_attach+0x19/0x20
    [<c130213f>] bus_add_driver+0x17f/0x240
    [<c1303345>] driver_register+0x65/0x120
    [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
    [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
    [<c1001030>] do_one_initcall+0x30/0x160
    [<c16b920b>] kernel_init+0x78/0x10c
    [<c14a1b76>] kernel_thread_helper+0x6/0xd
unreferenced object 0xe74e7860 (size 16):
  comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
  hex dump (first 16 bytes):
    50 43 49 20 42 75 73 20 30 30 30 30 3a 30 30 00  PCI Bus 0000:00.
  backtrace:
    [<c1495c47>] kmemleak_alloc+0x27/0x50
    [<c10b3563>] __kmalloc+0xf3/0x1c0
    [<c1290cce>] kvasprintf+0x2e/0x50
    [<c1290d01>] kasprintf+0x11/0x20
    [<c149e0da>] pci_acpi_scan_root+0x148/0x272
    [<c149916b>] acpi_pci_root_add+0x163/0x256
    [<c12adddc>] acpi_device_probe+0x3a/0xf4
    [<c1302e38>] driver_probe_device+0x68/0x160
    [<c1302fb9>] __driver_attach+0x89/0x90
    [<c1302718>] bus_for_each_dev+0x48/0x70
    [<c1302cc9>] driver_attach+0x19/0x20
    [<c130213f>] bus_add_driver+0x17f/0x240
    [<c1303345>] driver_register+0x65/0x120
    [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
    [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
    [<c1001030>] do_one_initcall+0x30/0x160

...which is probably a separate, non-recurring leak.

> Also, I'm sure you know IDE is deprecated, so what are the chances of
> moving this box to libata? Also, can you send me your .config pls?

Yeah, I was going to get around to that eventually. :) Config (and
earlier kmemleak output) here: http://0x.ca/sim/ref/3.1-rc6-blue/

Simon-

Re: [3.1-rc6] kmalloc(64) leak from IDE

[Borislav Petkov]

On Thu, Sep 22, 2011 at 01:23:37PM -0700, Simon Kirby wrote:
> Yes, that seems to have made it stop complaining about the IDE path.

Good, thanks for testing. It would be great if you left it running for
a couple of days like this to see whether there aren't any other issues
with the patch. I'll send it with a proper description to Dave soonish
since this is a real bug.

> All I see from kmemleak now is:

Yep, not IDE-related. Adding linux-acpi.

> unreferenced object 0xe7481a00 (size 256):
>   comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
>   hex dump (first 32 bytes):
>     00 00 00 28 ff ff ef ff 60 78 4e e7 00 02 00 00  ...(....`xN.....
>     47 01 f8 0c f8 0c 01 08 00 00 00 00 0c 03 00 00  G...............
>   backtrace:
>     [<c1495c47>] kmemleak_alloc+0x27/0x50
>     [<c10b3563>] __kmalloc+0xf3/0x1c0
>     [<c149e0b0>] pci_acpi_scan_root+0x11e/0x272
>     [<c149916b>] acpi_pci_root_add+0x163/0x256
>     [<c12adddc>] acpi_device_probe+0x3a/0xf4
>     [<c1302e38>] driver_probe_device+0x68/0x160
>     [<c1302fb9>] __driver_attach+0x89/0x90
>     [<c1302718>] bus_for_each_dev+0x48/0x70
>     [<c1302cc9>] driver_attach+0x19/0x20
>     [<c130213f>] bus_add_driver+0x17f/0x240
>     [<c1303345>] driver_register+0x65/0x120
>     [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
>     [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
>     [<c1001030>] do_one_initcall+0x30/0x160
>     [<c16b920b>] kernel_init+0x78/0x10c
>     [<c14a1b76>] kernel_thread_helper+0x6/0xd
> unreferenced object 0xe74e7860 (size 16):
>   comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
>   hex dump (first 16 bytes):
>     50 43 49 20 42 75 73 20 30 30 30 30 3a 30 30 00  PCI Bus 0000:00.
>   backtrace:
>     [<c1495c47>] kmemleak_alloc+0x27/0x50
>     [<c10b3563>] __kmalloc+0xf3/0x1c0
>     [<c1290cce>] kvasprintf+0x2e/0x50
>     [<c1290d01>] kasprintf+0x11/0x20
>     [<c149e0da>] pci_acpi_scan_root+0x148/0x272
>     [<c149916b>] acpi_pci_root_add+0x163/0x256
>     [<c12adddc>] acpi_device_probe+0x3a/0xf4
>     [<c1302e38>] driver_probe_device+0x68/0x160
>     [<c1302fb9>] __driver_attach+0x89/0x90
>     [<c1302718>] bus_for_each_dev+0x48/0x70
>     [<c1302cc9>] driver_attach+0x19/0x20
>     [<c130213f>] bus_add_driver+0x17f/0x240
>     [<c1303345>] driver_register+0x65/0x120
>     [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
>     [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
>     [<c1001030>] do_one_initcall+0x30/0x160
> 
> ...which is probably a separate, non-recurring leak.
> 
> > Also, I'm sure you know IDE is deprecated, so what are the chances of
> > moving this box to libata? Also, can you send me your .config pls?
> 
> Yeah, I was going to get around to that eventually. :) Config (and
> earlier kmemleak output) here: http://0x.ca/sim/ref/3.1-rc6-blue/

Ok, thanks.

-- 
Regards/Gruss,
    Boris.

Re: [3.1-rc6] kmalloc(64) leak from IDE

[Bernd Schubert]

On 09/23/2011 09:21 AM, Borislav Petkov wrote:
> On Thu, Sep 22, 2011 at 01:23:37PM -0700, Simon Kirby wrote:
>> Yes, that seems to have made it stop complaining about the IDE path.
>
> Good, thanks for testing. It would be great if you left it running for
> a couple of days like this to see whether there aren't any other issues
> with the patch. I'll send it with a proper description to Dave soonish
> since this is a real bug.
>
>> All I see from kmemleak now is:
>
> Yep, not IDE-related. Adding linux-acpi.


>
>> unreferenced object 0xe7481a00 (size 256):
>>    comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
>>    hex dump (first 32 bytes):
>>      00 00 00 28 ff ff ef ff 60 78 4e e7 00 02 00 00  ...(....`xN.....
>>      47 01 f8 0c f8 0c 01 08 00 00 00 00 0c 03 00 00  G...............
>>    backtrace:
>>      [<c1495c47>] kmemleak_alloc+0x27/0x50
>>      [<c10b3563>] __kmalloc+0xf3/0x1c0
>>      [<c149e0b0>] pci_acpi_scan_root+0x11e/0x272
>>      [<c149916b>] acpi_pci_root_add+0x163/0x256
>>      [<c12adddc>] acpi_device_probe+0x3a/0xf4
>>      [<c1302e38>] driver_probe_device+0x68/0x160
>>      [<c1302fb9>] __driver_attach+0x89/0x90
>>      [<c1302718>] bus_for_each_dev+0x48/0x70
>>      [<c1302cc9>] driver_attach+0x19/0x20
>>      [<c130213f>] bus_add_driver+0x17f/0x240
>>      [<c1303345>] driver_register+0x65/0x120
>>      [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
>>      [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
>>      [<c1001030>] do_one_initcall+0x30/0x160
>>      [<c16b920b>] kernel_init+0x78/0x10c
>>      [<c14a1b76>] kernel_thread_helper+0x6/0xd
>> unreferenced object 0xe74e7860 (size 16):
>>    comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
>>    hex dump (first 16 bytes):
>>      50 43 49 20 42 75 73 20 30 30 30 30 3a 30 30 00  PCI Bus 0000:00.
>>    backtrace:
>>      [<c1495c47>] kmemleak_alloc+0x27/0x50
>>      [<c10b3563>] __kmalloc+0xf3/0x1c0
>>      [<c1290cce>] kvasprintf+0x2e/0x50
>>      [<c1290d01>] kasprintf+0x11/0x20
>>      [<c149e0da>] pci_acpi_scan_root+0x148/0x272
>>      [<c149916b>] acpi_pci_root_add+0x163/0x256
>>      [<c12adddc>] acpi_device_probe+0x3a/0xf4
>>      [<c1302e38>] driver_probe_device+0x68/0x160
>>      [<c1302fb9>] __driver_attach+0x89/0x90
>>      [<c1302718>] bus_for_each_dev+0x48/0x70
>>      [<c1302cc9>] driver_attach+0x19/0x20
>>      [<c130213f>] bus_add_driver+0x17f/0x240
>>      [<c1303345>] driver_register+0x65/0x120
>>      [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
>>      [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
>>      [<c1001030>] do_one_initcall+0x30/0x160
>>
>> ...which is probably a separate, non-recurring leak.
>>
>>> Also, I'm sure you know IDE is deprecated, so what are the chances of
>>> moving this box to libata? Also, can you send me your .config pls?
>>
>> Yeah, I was going to get around to that eventually. :) Config (and
>> earlier kmemleak output) here: http://0x.ca/sim/ref/3.1-rc6-blue/
>
> Ok, thanks.
>

I think I reported those already some time ago:

https://lkml.org/lkml/2011/6/21/95


Cheers,
Bernd

Re: [3.1-rc6] kmalloc(64) leak from IDE

[Bjorn Helgaas]

On Fri, Sep 23, 2011 at 9:58 AM, Bernd Schubert
<bernd.schubert@itwm.fraunhofer.de> wrote:
> On 09/23/2011 09:21 AM, Borislav Petkov wrote:
>>
>> On Thu, Sep 22, 2011 at 01:23:37PM -0700, Simon Kirby wrote:
>>>
>>> Yes, that seems to have made it stop complaining about the IDE path.
>>
>> Good, thanks for testing. It would be great if you left it running for
>> a couple of days like this to see whether there aren't any other issues
>> with the patch. I'll send it with a proper description to Dave soonish
>> since this is a real bug.
>>
>>> All I see from kmemleak now is:
>>
>> Yep, not IDE-related. Adding linux-acpi.
>
>
>>
>>> unreferenced object 0xe7481a00 (size 256):
>>>   comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
>>>   hex dump (first 32 bytes):
>>>     00 00 00 28 ff ff ef ff 60 78 4e e7 00 02 00 00  ...(....`xN.....
>>>     47 01 f8 0c f8 0c 01 08 00 00 00 00 0c 03 00 00  G...............
>>>   backtrace:
>>>     [<c1495c47>] kmemleak_alloc+0x27/0x50
>>>     [<c10b3563>] __kmalloc+0xf3/0x1c0
>>>     [<c149e0b0>] pci_acpi_scan_root+0x11e/0x272
>>>     [<c149916b>] acpi_pci_root_add+0x163/0x256
>>>     [<c12adddc>] acpi_device_probe+0x3a/0xf4
>>>     [<c1302e38>] driver_probe_device+0x68/0x160
>>>     [<c1302fb9>] __driver_attach+0x89/0x90
>>>     [<c1302718>] bus_for_each_dev+0x48/0x70
>>>     [<c1302cc9>] driver_attach+0x19/0x20
>>>     [<c130213f>] bus_add_driver+0x17f/0x240
>>>     [<c1303345>] driver_register+0x65/0x120
>>>     [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
>>>     [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
>>>     [<c1001030>] do_one_initcall+0x30/0x160
>>>     [<c16b920b>] kernel_init+0x78/0x10c
>>>     [<c14a1b76>] kernel_thread_helper+0x6/0xd
>>> unreferenced object 0xe74e7860 (size 16):
>>>   comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
>>>   hex dump (first 16 bytes):
>>>     50 43 49 20 42 75 73 20 30 30 30 30 3a 30 30 00  PCI Bus 0000:00.
>>>   backtrace:
>>>     [<c1495c47>] kmemleak_alloc+0x27/0x50
>>>     [<c10b3563>] __kmalloc+0xf3/0x1c0
>>>     [<c1290cce>] kvasprintf+0x2e/0x50
>>>     [<c1290d01>] kasprintf+0x11/0x20
>>>     [<c149e0da>] pci_acpi_scan_root+0x148/0x272
>>>     [<c149916b>] acpi_pci_root_add+0x163/0x256
>>>     [<c12adddc>] acpi_device_probe+0x3a/0xf4
>>>     [<c1302e38>] driver_probe_device+0x68/0x160
>>>     [<c1302fb9>] __driver_attach+0x89/0x90
>>>     [<c1302718>] bus_for_each_dev+0x48/0x70
>>>     [<c1302cc9>] driver_attach+0x19/0x20
>>>     [<c130213f>] bus_add_driver+0x17f/0x240
>>>     [<c1303345>] driver_register+0x65/0x120
>>>     [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
>>>     [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
>>>     [<c1001030>] do_one_initcall+0x30/0x160
>>>
>>> ...which is probably a separate, non-recurring leak.
>>>
>>>> Also, I'm sure you know IDE is deprecated, so what are the chances of
>>>> moving this box to libata? Also, can you send me your .config pls?
>>>
>>> Yeah, I was going to get around to that eventually. :) Config (and
>>> earlier kmemleak output) here: http://0x.ca/sim/ref/3.1-rc6-blue/
>>
>> Ok, thanks.
>>
>
> I think I reported those already some time ago:
>
> https://lkml.org/lkml/2011/6/21/95

Rats.  And you even posted a patch
(https://lkml.org/lkml/2011/6/21/132).  I was cc'd, but unfortunately
to an old email address that no longer works.  I'll follow up on it
and make sure it's fixed (probably in 3.2 since it's minor and we're
so late in 3.1).

Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [3.1-rc6] kmalloc(64) leak from IDE

[Bjorn Helgaas]

On Fri, Sep 23, 2011 at 9:58 AM, Bernd Schubert
<bernd.schubert@itwm.fraunhofer.de> wrote:
> On 09/23/2011 09:21 AM, Borislav Petkov wrote:
>>
>> On Thu, Sep 22, 2011 at 01:23:37PM -0700, Simon Kirby wrote:
>>>
>>> Yes, that seems to have made it stop complaining about the IDE path.
>>
>> Good, thanks for testing. It would be great if you left it running for
>> a couple of days like this to see whether there aren't any other issues
>> with the patch. I'll send it with a proper description to Dave soonish
>> since this is a real bug.
>>
>>> All I see from kmemleak now is:
>>
>> Yep, not IDE-related. Adding linux-acpi.
>
>
>>
>>> unreferenced object 0xe7481a00 (size 256):
>>>   comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
>>>   hex dump (first 32 bytes):
>>>     00 00 00 28 ff ff ef ff 60 78 4e e7 00 02 00 00  ...(....`xN.....
>>>     47 01 f8 0c f8 0c 01 08 00 00 00 00 0c 03 00 00  G...............
>>>   backtrace:
>>>     [<c1495c47>] kmemleak_alloc+0x27/0x50
>>>     [<c10b3563>] __kmalloc+0xf3/0x1c0
>>>     [<c149e0b0>] pci_acpi_scan_root+0x11e/0x272
>>>     [<c149916b>] acpi_pci_root_add+0x163/0x256
>>>     [<c12adddc>] acpi_device_probe+0x3a/0xf4
>>>     [<c1302e38>] driver_probe_device+0x68/0x160
>>>     [<c1302fb9>] __driver_attach+0x89/0x90
>>>     [<c1302718>] bus_for_each_dev+0x48/0x70
>>>     [<c1302cc9>] driver_attach+0x19/0x20
>>>     [<c130213f>] bus_add_driver+0x17f/0x240
>>>     [<c1303345>] driver_register+0x65/0x120
>>>     [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
>>>     [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
>>>     [<c1001030>] do_one_initcall+0x30/0x160
>>>     [<c16b920b>] kernel_init+0x78/0x10c
>>>     [<c14a1b76>] kernel_thread_helper+0x6/0xd
>>> unreferenced object 0xe74e7860 (size 16):
>>>   comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
>>>   hex dump (first 16 bytes):
>>>     50 43 49 20 42 75 73 20 30 30 30 30 3a 30 30 00  PCI Bus 0000:00.
>>>   backtrace:
>>>     [<c1495c47>] kmemleak_alloc+0x27/0x50
>>>     [<c10b3563>] __kmalloc+0xf3/0x1c0
>>>     [<c1290cce>] kvasprintf+0x2e/0x50
>>>     [<c1290d01>] kasprintf+0x11/0x20
>>>     [<c149e0da>] pci_acpi_scan_root+0x148/0x272
>>>     [<c149916b>] acpi_pci_root_add+0x163/0x256
>>>     [<c12adddc>] acpi_device_probe+0x3a/0xf4
>>>     [<c1302e38>] driver_probe_device+0x68/0x160
>>>     [<c1302fb9>] __driver_attach+0x89/0x90
>>>     [<c1302718>] bus_for_each_dev+0x48/0x70
>>>     [<c1302cc9>] driver_attach+0x19/0x20
>>>     [<c130213f>] bus_add_driver+0x17f/0x240
>>>     [<c1303345>] driver_register+0x65/0x120
>>>     [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
>>>     [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
>>>     [<c1001030>] do_one_initcall+0x30/0x160
>>>
>>> ...which is probably a separate, non-recurring leak.
>>>
>>>> Also, I'm sure you know IDE is deprecated, so what are the chances of
>>>> moving this box to libata? Also, can you send me your .config pls?
>>>
>>> Yeah, I was going to get around to that eventually. :) Config (and
>>> earlier kmemleak output) here: http://0x.ca/sim/ref/3.1-rc6-blue/
>>
>> Ok, thanks.
>>
>
> I think I reported those already some time ago:
>
> https://lkml.org/lkml/2011/6/21/95

Rats.  And you even posted a patch
(https://lkml.org/lkml/2011/6/21/132).  I was cc'd, but unfortunately
to an old email address that no longer works.  I'll follow up on it
and make sure it's fixed (probably in 3.2 since it's minor and we're
so late in 3.1).

Bjorn

Re: [3.1-rc6] kmalloc(64) leak from IDE

[Bernd Schubert]

On 09/23/2011 06:08 PM, Bjorn Helgaas wrote:
> On Fri, Sep 23, 2011 at 9:58 AM, Bernd Schubert
> <bernd.schubert@itwm.fraunhofer.de>  wrote:
>> On 09/23/2011 09:21 AM, Borislav Petkov wrote:
>>>
>>> On Thu, Sep 22, 2011 at 01:23:37PM -0700, Simon Kirby wrote:
>>>>
>>>> Yes, that seems to have made it stop complaining about the IDE path.
>>>
>>> Good, thanks for testing. It would be great if you left it running for
>>> a couple of days like this to see whether there aren't any other issues
>>> with the patch. I'll send it with a proper description to Dave soonish
>>> since this is a real bug.
>>>
>>>> All I see from kmemleak now is:
>>>
>>> Yep, not IDE-related. Adding linux-acpi.
>>
>>
>>>
>>>> unreferenced object 0xe7481a00 (size 256):
>>>>    comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
>>>>    hex dump (first 32 bytes):
>>>>      00 00 00 28 ff ff ef ff 60 78 4e e7 00 02 00 00  ...(....`xN.....
>>>>      47 01 f8 0c f8 0c 01 08 00 00 00 00 0c 03 00 00  G...............
>>>>    backtrace:
>>>>      [<c1495c47>] kmemleak_alloc+0x27/0x50
>>>>      [<c10b3563>] __kmalloc+0xf3/0x1c0
>>>>      [<c149e0b0>] pci_acpi_scan_root+0x11e/0x272
>>>>      [<c149916b>] acpi_pci_root_add+0x163/0x256
>>>>      [<c12adddc>] acpi_device_probe+0x3a/0xf4
>>>>      [<c1302e38>] driver_probe_device+0x68/0x160
>>>>      [<c1302fb9>] __driver_attach+0x89/0x90
>>>>      [<c1302718>] bus_for_each_dev+0x48/0x70
>>>>      [<c1302cc9>] driver_attach+0x19/0x20
>>>>      [<c130213f>] bus_add_driver+0x17f/0x240
>>>>      [<c1303345>] driver_register+0x65/0x120
>>>>      [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
>>>>      [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
>>>>      [<c1001030>] do_one_initcall+0x30/0x160
>>>>      [<c16b920b>] kernel_init+0x78/0x10c
>>>>      [<c14a1b76>] kernel_thread_helper+0x6/0xd
>>>> unreferenced object 0xe74e7860 (size 16):
>>>>    comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
>>>>    hex dump (first 16 bytes):
>>>>      50 43 49 20 42 75 73 20 30 30 30 30 3a 30 30 00  PCI Bus 0000:00.
>>>>    backtrace:
>>>>      [<c1495c47>] kmemleak_alloc+0x27/0x50
>>>>      [<c10b3563>] __kmalloc+0xf3/0x1c0
>>>>      [<c1290cce>] kvasprintf+0x2e/0x50
>>>>      [<c1290d01>] kasprintf+0x11/0x20
>>>>      [<c149e0da>] pci_acpi_scan_root+0x148/0x272
>>>>      [<c149916b>] acpi_pci_root_add+0x163/0x256
>>>>      [<c12adddc>] acpi_device_probe+0x3a/0xf4
>>>>      [<c1302e38>] driver_probe_device+0x68/0x160
>>>>      [<c1302fb9>] __driver_attach+0x89/0x90
>>>>      [<c1302718>] bus_for_each_dev+0x48/0x70
>>>>      [<c1302cc9>] driver_attach+0x19/0x20
>>>>      [<c130213f>] bus_add_driver+0x17f/0x240
>>>>      [<c1303345>] driver_register+0x65/0x120
>>>>      [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
>>>>      [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
>>>>      [<c1001030>] do_one_initcall+0x30/0x160
>>>>
>>>> ...which is probably a separate, non-recurring leak.
>>>>
>>>>> Also, I'm sure you know IDE is deprecated, so what are the chances of
>>>>> moving this box to libata? Also, can you send me your .config pls?
>>>>
>>>> Yeah, I was going to get around to that eventually. :) Config (and
>>>> earlier kmemleak output) here: http://0x.ca/sim/ref/3.1-rc6-blue/
>>>
>>> Ok, thanks.
>>>
>>
>> I think I reported those already some time ago:
>>
>> https://lkml.org/lkml/2011/6/21/95
>
> Rats.  And you even posted a patch
> (https://lkml.org/lkml/2011/6/21/132).  I was cc'd, but unfortunately
> to an old email address that no longer works.  I'll follow up on it
> and make sure it's fixed (probably in 3.2 since it's minor and we're
> so late in 3.1).

Ah great, I thought you are simply too busy to look into it. We are all 
just changing our mail addresses too often :)

Do you mind if I ping you directly once kernel.org is up again? We have 
a few test systems here, that fail to boot with recent kernels (just a 
system hard reset). The issue was introduced in between 2.6.27 and 
2.6.32 and it is pci mmconfig related, probably due to a troublesome 
acpi bios. I filled a bugzilla some time ago and never got a responds on 
it. But I also need to update it with my latest finding...

Thanks,
Bernd

Re: [3.1-rc6] kmalloc(64) leak from IDE

[Bjorn Helgaas]

On Fri, Sep 23, 2011 at 10:34 AM, Bernd Schubert
<bernd.schubert@itwm.fraunhofer.de> wrote:
> On 09/23/2011 06:08 PM, Bjorn Helgaas wrote:
>>
>> On Fri, Sep 23, 2011 at 9:58 AM, Bernd Schubert
>> <bernd.schubert@itwm.fraunhofer.de>  wrote:
>>>
>>> On 09/23/2011 09:21 AM, Borislav Petkov wrote:
>>>>
>>>> On Thu, Sep 22, 2011 at 01:23:37PM -0700, Simon Kirby wrote:
>>>>>
>>>>> Yes, that seems to have made it stop complaining about the IDE path.
>>>>
>>>> Good, thanks for testing. It would be great if you left it running for
>>>> a couple of days like this to see whether there aren't any other issues
>>>> with the patch. I'll send it with a proper description to Dave soonish
>>>> since this is a real bug.
>>>>
>>>>> All I see from kmemleak now is:
>>>>
>>>> Yep, not IDE-related. Adding linux-acpi.
>>>
>>>
>>>>
>>>>> unreferenced object 0xe7481a00 (size 256):
>>>>>   comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
>>>>>   hex dump (first 32 bytes):
>>>>>     00 00 00 28 ff ff ef ff 60 78 4e e7 00 02 00 00  ...(....`xN.....
>>>>>     47 01 f8 0c f8 0c 01 08 00 00 00 00 0c 03 00 00  G...............
>>>>>   backtrace:
>>>>>     [<c1495c47>] kmemleak_alloc+0x27/0x50
>>>>>     [<c10b3563>] __kmalloc+0xf3/0x1c0
>>>>>     [<c149e0b0>] pci_acpi_scan_root+0x11e/0x272
>>>>>     [<c149916b>] acpi_pci_root_add+0x163/0x256
>>>>>     [<c12adddc>] acpi_device_probe+0x3a/0xf4
>>>>>     [<c1302e38>] driver_probe_device+0x68/0x160
>>>>>     [<c1302fb9>] __driver_attach+0x89/0x90
>>>>>     [<c1302718>] bus_for_each_dev+0x48/0x70
>>>>>     [<c1302cc9>] driver_attach+0x19/0x20
>>>>>     [<c130213f>] bus_add_driver+0x17f/0x240
>>>>>     [<c1303345>] driver_register+0x65/0x120
>>>>>     [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
>>>>>     [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
>>>>>     [<c1001030>] do_one_initcall+0x30/0x160
>>>>>     [<c16b920b>] kernel_init+0x78/0x10c
>>>>>     [<c14a1b76>] kernel_thread_helper+0x6/0xd
>>>>> unreferenced object 0xe74e7860 (size 16):
>>>>>   comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
>>>>>   hex dump (first 16 bytes):
>>>>>     50 43 49 20 42 75 73 20 30 30 30 30 3a 30 30 00  PCI Bus 0000:00.
>>>>>   backtrace:
>>>>>     [<c1495c47>] kmemleak_alloc+0x27/0x50
>>>>>     [<c10b3563>] __kmalloc+0xf3/0x1c0
>>>>>     [<c1290cce>] kvasprintf+0x2e/0x50
>>>>>     [<c1290d01>] kasprintf+0x11/0x20
>>>>>     [<c149e0da>] pci_acpi_scan_root+0x148/0x272
>>>>>     [<c149916b>] acpi_pci_root_add+0x163/0x256
>>>>>     [<c12adddc>] acpi_device_probe+0x3a/0xf4
>>>>>     [<c1302e38>] driver_probe_device+0x68/0x160
>>>>>     [<c1302fb9>] __driver_attach+0x89/0x90
>>>>>     [<c1302718>] bus_for_each_dev+0x48/0x70
>>>>>     [<c1302cc9>] driver_attach+0x19/0x20
>>>>>     [<c130213f>] bus_add_driver+0x17f/0x240
>>>>>     [<c1303345>] driver_register+0x65/0x120
>>>>>     [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
>>>>>     [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
>>>>>     [<c1001030>] do_one_initcall+0x30/0x160
>>>>>
>>>>> ...which is probably a separate, non-recurring leak.
>>>>>
>>>>>> Also, I'm sure you know IDE is deprecated, so what are the chances of
>>>>>> moving this box to libata? Also, can you send me your .config pls?
>>>>>
>>>>> Yeah, I was going to get around to that eventually. :) Config (and
>>>>> earlier kmemleak output) here: http://0x.ca/sim/ref/3.1-rc6-blue/
>>>>
>>>> Ok, thanks.
>>>>
>>>
>>> I think I reported those already some time ago:
>>>
>>> https://lkml.org/lkml/2011/6/21/95
>>
>> Rats.  And you even posted a patch
>> (https://lkml.org/lkml/2011/6/21/132).  I was cc'd, but unfortunately
>> to an old email address that no longer works.  I'll follow up on it
>> and make sure it's fixed (probably in 3.2 since it's minor and we're
>> so late in 3.1).
>
> Ah great, I thought you are simply too busy to look into it. We are all just
> changing our mail addresses too often :)
>
> Do you mind if I ping you directly once kernel.org is up again? We have a
> few test systems here, that fail to boot with recent kernels (just a system
> hard reset). The issue was introduced in between 2.6.27 and 2.6.32 and it is
> pci mmconfig related, probably due to a troublesome acpi bios. I filled a
> bugzilla some time ago and never got a responds on it. But I also need to
> update it with my latest finding...

Please do, and you don't need to wait for kernel.org to come back
(though it'd be good to include any bugzilla URLs if you know them).
Just email me and copy linux-pci (since it seems PCI
MMCONFIG-related), and we can try to make some progress.  It's very
important to fix regressions as quickly as possible.

Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [3.1-rc6] kmalloc(64) leak from IDE

[Bjorn Helgaas]

On Fri, Sep 23, 2011 at 10:34 AM, Bernd Schubert
<bernd.schubert@itwm.fraunhofer.de> wrote:
> On 09/23/2011 06:08 PM, Bjorn Helgaas wrote:
>>
>> On Fri, Sep 23, 2011 at 9:58 AM, Bernd Schubert
>> <bernd.schubert@itwm.fraunhofer.de>  wrote:
>>>
>>> On 09/23/2011 09:21 AM, Borislav Petkov wrote:
>>>>
>>>> On Thu, Sep 22, 2011 at 01:23:37PM -0700, Simon Kirby wrote:
>>>>>
>>>>> Yes, that seems to have made it stop complaining about the IDE path.
>>>>
>>>> Good, thanks for testing. It would be great if you left it running for
>>>> a couple of days like this to see whether there aren't any other issues
>>>> with the patch. I'll send it with a proper description to Dave soonish
>>>> since this is a real bug.
>>>>
>>>>> All I see from kmemleak now is:
>>>>
>>>> Yep, not IDE-related. Adding linux-acpi.
>>>
>>>
>>>>
>>>>> unreferenced object 0xe7481a00 (size 256):
>>>>>   comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
>>>>>   hex dump (first 32 bytes):
>>>>>     00 00 00 28 ff ff ef ff 60 78 4e e7 00 02 00 00  ...(....`xN.....
>>>>>     47 01 f8 0c f8 0c 01 08 00 00 00 00 0c 03 00 00  G...............
>>>>>   backtrace:
>>>>>     [<c1495c47>] kmemleak_alloc+0x27/0x50
>>>>>     [<c10b3563>] __kmalloc+0xf3/0x1c0
>>>>>     [<c149e0b0>] pci_acpi_scan_root+0x11e/0x272
>>>>>     [<c149916b>] acpi_pci_root_add+0x163/0x256
>>>>>     [<c12adddc>] acpi_device_probe+0x3a/0xf4
>>>>>     [<c1302e38>] driver_probe_device+0x68/0x160
>>>>>     [<c1302fb9>] __driver_attach+0x89/0x90
>>>>>     [<c1302718>] bus_for_each_dev+0x48/0x70
>>>>>     [<c1302cc9>] driver_attach+0x19/0x20
>>>>>     [<c130213f>] bus_add_driver+0x17f/0x240
>>>>>     [<c1303345>] driver_register+0x65/0x120
>>>>>     [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
>>>>>     [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
>>>>>     [<c1001030>] do_one_initcall+0x30/0x160
>>>>>     [<c16b920b>] kernel_init+0x78/0x10c
>>>>>     [<c14a1b76>] kernel_thread_helper+0x6/0xd
>>>>> unreferenced object 0xe74e7860 (size 16):
>>>>>   comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
>>>>>   hex dump (first 16 bytes):
>>>>>     50 43 49 20 42 75 73 20 30 30 30 30 3a 30 30 00  PCI Bus 0000:00.
>>>>>   backtrace:
>>>>>     [<c1495c47>] kmemleak_alloc+0x27/0x50
>>>>>     [<c10b3563>] __kmalloc+0xf3/0x1c0
>>>>>     [<c1290cce>] kvasprintf+0x2e/0x50
>>>>>     [<c1290d01>] kasprintf+0x11/0x20
>>>>>     [<c149e0da>] pci_acpi_scan_root+0x148/0x272
>>>>>     [<c149916b>] acpi_pci_root_add+0x163/0x256
>>>>>     [<c12adddc>] acpi_device_probe+0x3a/0xf4
>>>>>     [<c1302e38>] driver_probe_device+0x68/0x160
>>>>>     [<c1302fb9>] __driver_attach+0x89/0x90
>>>>>     [<c1302718>] bus_for_each_dev+0x48/0x70
>>>>>     [<c1302cc9>] driver_attach+0x19/0x20
>>>>>     [<c130213f>] bus_add_driver+0x17f/0x240
>>>>>     [<c1303345>] driver_register+0x65/0x120
>>>>>     [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
>>>>>     [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
>>>>>     [<c1001030>] do_one_initcall+0x30/0x160
>>>>>
>>>>> ...which is probably a separate, non-recurring leak.
>>>>>
>>>>>> Also, I'm sure you know IDE is deprecated, so what are the chances of
>>>>>> moving this box to libata? Also, can you send me your .config pls?
>>>>>
>>>>> Yeah, I was going to get around to that eventually. :) Config (and
>>>>> earlier kmemleak output) here: http://0x.ca/sim/ref/3.1-rc6-blue/
>>>>
>>>> Ok, thanks.
>>>>
>>>
>>> I think I reported those already some time ago:
>>>
>>> https://lkml.org/lkml/2011/6/21/95
>>
>> Rats.  And you even posted a patch
>> (https://lkml.org/lkml/2011/6/21/132).  I was cc'd, but unfortunately
>> to an old email address that no longer works.  I'll follow up on it
>> and make sure it's fixed (probably in 3.2 since it's minor and we're
>> so late in 3.1).
>
> Ah great, I thought you are simply too busy to look into it. We are all just
> changing our mail addresses too often :)
>
> Do you mind if I ping you directly once kernel.org is up again? We have a
> few test systems here, that fail to boot with recent kernels (just a system
> hard reset). The issue was introduced in between 2.6.27 and 2.6.32 and it is
> pci mmconfig related, probably due to a troublesome acpi bios. I filled a
> bugzilla some time ago and never got a responds on it. But I also need to
> update it with my latest finding...

Please do, and you don't need to wait for kernel.org to come back
(though it'd be good to include any bugzilla URLs if you know them).
Just email me and copy linux-pci (since it seems PCI
MMCONFIG-related), and we can try to make some progress.  It's very
important to fix regressions as quickly as possible.

Bjorn

Re: [3.1-rc6] kmalloc(64) leak from IDE

[Simon Kirby]

On Fri, Sep 23, 2011 at 09:21:18AM +0200, Borislav Petkov wrote:

> On Thu, Sep 22, 2011 at 01:23:37PM -0700, Simon Kirby wrote:
> > Yes, that seems to have made it stop complaining about the IDE path.
> 
> Good, thanks for testing. It would be great if you left it running for
> a couple of days like this to see whether there aren't any other issues
> with the patch. I'll send it with a proper description to Dave soonish
> since this is a real bug.

It has been up 22 hours now without breaking. I suppose you're worried
about the old command being used after it gets reused. Would it help to
bisect the regression to see how it used to work? That would let us know
how many stable kernels to patch, too.

Simon-

Re: [3.1-rc6] kmalloc(64) leak from IDE

[Bernd Schubert]

On 09/23/2011 06:40 PM, Bjorn Helgaas wrote:
> On Fri, Sep 23, 2011 at 10:34 AM, Bernd Schubert
> <bernd.schubert@itwm.fraunhofer.de>  wrote:
>> On 09/23/2011 06:08 PM, Bjorn Helgaas wrote:
>>>
>>> On Fri, Sep 23, 2011 at 9:58 AM, Bernd Schubert
>>> <bernd.schubert@itwm.fraunhofer.de>    wrote:
>>>>
>>>> On 09/23/2011 09:21 AM, Borislav Petkov wrote:
>>>>>
>>>>> On Thu, Sep 22, 2011 at 01:23:37PM -0700, Simon Kirby wrote:
>>>>>>
>>>>>> Yes, that seems to have made it stop complaining about the IDE path.
>>>>>
>>>>> Good, thanks for testing. It would be great if you left it running for
>>>>> a couple of days like this to see whether there aren't any other issues
>>>>> with the patch. I'll send it with a proper description to Dave soonish
>>>>> since this is a real bug.
>>>>>
>>>>>> All I see from kmemleak now is:
>>>>>
>>>>> Yep, not IDE-related. Adding linux-acpi.
>>>>
>>>>
>>>>>
>>>>>> unreferenced object 0xe7481a00 (size 256):
>>>>>>    comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
>>>>>>    hex dump (first 32 bytes):
>>>>>>      00 00 00 28 ff ff ef ff 60 78 4e e7 00 02 00 00  ...(....`xN.....
>>>>>>      47 01 f8 0c f8 0c 01 08 00 00 00 00 0c 03 00 00  G...............
>>>>>>    backtrace:
>>>>>>      [<c1495c47>] kmemleak_alloc+0x27/0x50
>>>>>>      [<c10b3563>] __kmalloc+0xf3/0x1c0
>>>>>>      [<c149e0b0>] pci_acpi_scan_root+0x11e/0x272
>>>>>>      [<c149916b>] acpi_pci_root_add+0x163/0x256
>>>>>>      [<c12adddc>] acpi_device_probe+0x3a/0xf4
>>>>>>      [<c1302e38>] driver_probe_device+0x68/0x160
>>>>>>      [<c1302fb9>] __driver_attach+0x89/0x90
>>>>>>      [<c1302718>] bus_for_each_dev+0x48/0x70
>>>>>>      [<c1302cc9>] driver_attach+0x19/0x20
>>>>>>      [<c130213f>] bus_add_driver+0x17f/0x240
>>>>>>      [<c1303345>] driver_register+0x65/0x120
>>>>>>      [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
>>>>>>      [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
>>>>>>      [<c1001030>] do_one_initcall+0x30/0x160
>>>>>>      [<c16b920b>] kernel_init+0x78/0x10c
>>>>>>      [<c14a1b76>] kernel_thread_helper+0x6/0xd
>>>>>> unreferenced object 0xe74e7860 (size 16):
>>>>>>    comm "swapper", pid 1, jiffies 4294892509 (age 515.560s)
>>>>>>    hex dump (first 16 bytes):
>>>>>>      50 43 49 20 42 75 73 20 30 30 30 30 3a 30 30 00  PCI Bus 0000:00.
>>>>>>    backtrace:
>>>>>>      [<c1495c47>] kmemleak_alloc+0x27/0x50
>>>>>>      [<c10b3563>] __kmalloc+0xf3/0x1c0
>>>>>>      [<c1290cce>] kvasprintf+0x2e/0x50
>>>>>>      [<c1290d01>] kasprintf+0x11/0x20
>>>>>>      [<c149e0da>] pci_acpi_scan_root+0x148/0x272
>>>>>>      [<c149916b>] acpi_pci_root_add+0x163/0x256
>>>>>>      [<c12adddc>] acpi_device_probe+0x3a/0xf4
>>>>>>      [<c1302e38>] driver_probe_device+0x68/0x160
>>>>>>      [<c1302fb9>] __driver_attach+0x89/0x90
>>>>>>      [<c1302718>] bus_for_each_dev+0x48/0x70
>>>>>>      [<c1302cc9>] driver_attach+0x19/0x20
>>>>>>      [<c130213f>] bus_add_driver+0x17f/0x240
>>>>>>      [<c1303345>] driver_register+0x65/0x120
>>>>>>      [<c12af273>] acpi_bus_register_driver+0x3a/0x3f
>>>>>>      [<c16d421f>] acpi_pci_root_init+0x1b/0x2a
>>>>>>      [<c1001030>] do_one_initcall+0x30/0x160
>>>>>>
>>>>>> ...which is probably a separate, non-recurring leak.
>>>>>>
>>>>>>> Also, I'm sure you know IDE is deprecated, so what are the chances of
>>>>>>> moving this box to libata? Also, can you send me your .config pls?
>>>>>>
>>>>>> Yeah, I was going to get around to that eventually. :) Config (and
>>>>>> earlier kmemleak output) here: http://0x.ca/sim/ref/3.1-rc6-blue/
>>>>>
>>>>> Ok, thanks.
>>>>>
>>>>
>>>> I think I reported those already some time ago:
>>>>
>>>> https://lkml.org/lkml/2011/6/21/95
>>>
>>> Rats.  And you even posted a patch
>>> (https://lkml.org/lkml/2011/6/21/132).  I was cc'd, but unfortunately
>>> to an old email address that no longer works.  I'll follow up on it
>>> and make sure it's fixed (probably in 3.2 since it's minor and we're
>>> so late in 3.1).
>>
>> Ah great, I thought you are simply too busy to look into it. We are all just
>> changing our mail addresses too often :)
>>
>> Do you mind if I ping you directly once kernel.org is up again? We have a
>> few test systems here, that fail to boot with recent kernels (just a system
>> hard reset). The issue was introduced in between 2.6.27 and 2.6.32 and it is
>> pci mmconfig related, probably due to a troublesome acpi bios. I filled a
>> bugzilla some time ago and never got a responds on it. But I also need to
>> update it with my latest finding...
>
> Please do, and you don't need to wait for kernel.org to come back
> (though it'd be good to include any bugzilla URLs if you know them).
> Just email me and copy linux-pci (since it seems PCI
> MMCONFIG-related), and we can try to make some progress.  It's very
> important to fix regressions as quickly as possible.

Hmm, I tried to analyze it already and I don't have the text and 
attachments I sent to the bugzilla ready on my disk to send it again. So 
it would be really good if we could wait just a little bit longer (I 
posted it several months ago already). I also found some kind of 
workaround for us and need the systems right now for our own development...


Thanks,
Bernd

Re: [3.1-rc6] kmalloc(64) leak from IDE

[Borislav Petkov]

On Fri, Sep 23, 2011 at 10:38:08AM -0700, Simon Kirby wrote:
> On Fri, Sep 23, 2011 at 09:21:18AM +0200, Borislav Petkov wrote:
> 
> > On Thu, Sep 22, 2011 at 01:23:37PM -0700, Simon Kirby wrote:
> > > Yes, that seems to have made it stop complaining about the IDE path.
> > 
> > Good, thanks for testing. It would be great if you left it running for
> > a couple of days like this to see whether there aren't any other issues
> > with the patch. I'll send it with a proper description to Dave soonish
> > since this is a real bug.
> 
> It has been up 22 hours now without breaking.

Good.

> I suppose you're worried about the old command being used after it
> gets reused.

Nah, we clear the old command before reusing it and when the request is
done, it gets freed, AFAICT, but...

> Would it help to bisect the regression to see how it used to work?
> That would let us know how many stable kernels to patch, too.

backporting it to -stable is a good point. I'll add the proper tagging
to the patch.

Thanks.

-- 
Regards/Gruss,
    Boris.

Re: [3.1-rc6] kmalloc(64) leak from IDE

[Simon Kirby]

On Sun, Sep 25, 2011 at 10:58:18AM +0200, Borislav Petkov wrote:

> On Fri, Sep 23, 2011 at 10:38:08AM -0700, Simon Kirby wrote:
> > On Fri, Sep 23, 2011 at 09:21:18AM +0200, Borislav Petkov wrote:
> > 
> > > On Thu, Sep 22, 2011 at 01:23:37PM -0700, Simon Kirby wrote:
> > > > Yes, that seems to have made it stop complaining about the IDE path.
> > > 
> > > Good, thanks for testing. It would be great if you left it running for
> > > a couple of days like this to see whether there aren't any other issues
> > > with the patch. I'll send it with a proper description to Dave soonish
> > > since this is a real bug.
> > 
> > It has been up 22 hours now without breaking.
> 
> Good.
> 
> > I suppose you're worried about the old command being used after it
> > gets reused.
> 
> Nah, we clear the old command before reusing it and when the request is
> done, it gets freed, AFAICT, but...

Ok, good. It's still running without any problem, and no new leaks
reported.

> > Would it help to bisect the regression to see how it used to work?
> > That would let us know how many stable kernels to patch, too.
> 
> backporting it to -stable is a good point. I'll add the proper tagging
> to the patch.

Do you know in which version the issue started, then?

If not, all I have to start with is that it was fine on 2.6.36, and I can
bisect it, if that would help.

Cheers,

Simon-

Re: [3.1-rc6] kmalloc(64) leak from IDE

[Borislav Petkov]

(forgot to Cc linux-ide earlier, sorry)

On Mon, Sep 26, 2011 at 01:05:50AM -0700, Simon Kirby wrote:
> Ok, good. It's still running without any problem, and no new leaks
> reported.

Ok.

[..]

> > backporting it to -stable is a good point. I'll add the proper tagging
> > to the patch.
> 
> Do you know in which version the issue started, then?
> 
> If not, all I have to start with is that it was fine on 2.6.36, and I can
> bisect it, if that would help.

This is exactly the question: AFAICT, it could be that changes in the
block layer at some point have caused the ide bust and since almost no
one tests ide...

The patch adding the dynamic ide_cmd struct allocation is
395d8ef5bebe547a80737692f9789d2e36da16f2 from 2008 and I don't think it
caused the issue then but I could also be remembering it wrong.

So I wouldn't bisect it but test stable trees after 2.6.36 to see
whether they have the issue, and if so, only then the patch should be
backported.

And this is not that easy now with k.org down but looking at

http://web.archive.org/web/20110725015737/http://kernel.org/

the only stable trees which need to be tested are 2.6.39 and 3.0.

How does that sound?

-- 
Regards/Gruss,
Boris.

Re: [3.1-rc6] kmalloc(64) leak from IDE

[Borislav Petkov]

On Tue, Sep 27, 2011 at 07:07:55PM +0200, Borislav Petkov wrote:
> (forgot to Cc linux-ide earlier, sorry)
> 
> On Mon, Sep 26, 2011 at 01:05:50AM -0700, Simon Kirby wrote:
> > Ok, good. It's still running without any problem, and no new leaks
> > reported.
> 
> Ok.
> 
> [..]
> 
> > > backporting it to -stable is a good point. I'll add the proper tagging
> > > to the patch.
> > 
> > Do you know in which version the issue started, then?
> > 
> > If not, all I have to start with is that it was fine on 2.6.36, and I can
> > bisect it, if that would help.
> 
> This is exactly the question: AFAICT, it could be that changes in the
> block layer at some point have caused the ide bust and since almost no
> one tests ide...
> 
> The patch adding the dynamic ide_cmd struct allocation is
> 395d8ef5bebe547a80737692f9789d2e36da16f2 from 2008 and I don't think it
> caused the issue then but I could also be remembering it wrong.
> 
> So I wouldn't bisect it but test stable trees after 2.6.36 to see
> whether they have the issue, and if so, only then the patch should be
> backported.
> 
> And this is not that easy now with k.org down but looking at
> 
> http://web.archive.org/web/20110725015737/http://kernel.org/
> 
> the only stable trees which need to be tested are 2.6.39 and 3.0.
> 
> How does that sound?

Btw, here's the patch, if you would like to test 2.6.39 once without it
to see whether kmemleak screams and once with it, I'll add the stable
tagging.

Thanks.

--
>From 96414ddbfecaaa3d265794c0792d816cf3c1e33d Mon Sep 17 00:00:00 2001
From: Borislav Petkov <bp@alien8.de>
Date: Sun, 25 Sep 2011 13:38:04 +0200
Subject: [PATCH] ide-disk: Fix request requeuing

Simon Kirby reported that on his RAID setup with idedisk underneath
the box OOMs after a couple of days of runtime. Running with
CONFIG_DEBUG_KMEMLEAK pointed to idedisk_prep_fn() which unconditionally
allocates an ide_cmd struct. However, ide_requeue_and_plug() can be
called more than once per request, either from the request issue or the
IRQ handler path and do blk_peek_request() ends up in idedisk_prep_fn()
repeatedly, allocating a struct ide_cmd everytime and "forgetting" the
previous pointer.

Make sure the code reuses the old allocated chunk.

Reported-and-tested-by: Simon Kirby <sim@hostway.ca>
Link: http://marc.info/?l=linux-kernel&m=131667641517919
Link: http://lkml.kernel.org/r/20110922072643.GA27232@hostway.ca
Signed-off-by: Borislav Petkov <bp@alien8.de>
---
 drivers/ide/ide-disk.c |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/drivers/ide/ide-disk.c b/drivers/ide/ide-disk.c
index 274798068a54..16f69be820c7 100644
--- a/drivers/ide/ide-disk.c
+++ b/drivers/ide/ide-disk.c
@@ -435,7 +435,12 @@ static int idedisk_prep_fn(struct request_queue *q, struct request *rq)
 	if (!(rq->cmd_flags & REQ_FLUSH))
 		return BLKPREP_OK;
 
-	cmd = kzalloc(sizeof(*cmd), GFP_ATOMIC);
+	if (rq->special) {
+		cmd = rq->special;
+		memset(cmd, 0, sizeof(*cmd));
+	} else {
+		cmd = kzalloc(sizeof(*cmd), GFP_ATOMIC);
+	}
 
 	/* FIXME: map struct ide_taskfile on rq->cmd[] */
 	BUG_ON(cmd == NULL);
-- 
1.7.5.3.401.gfb674


-- 
Regards/Gruss,
    Boris.

Re: [3.1-rc6] kmalloc(64) leak from IDE

[Simon Kirby]

On Thu, Sep 29, 2011 at 11:27:05AM +0200, Borislav Petkov wrote:

> On Tue, Sep 27, 2011 at 07:07:55PM +0200, Borislav Petkov wrote:
> > (forgot to Cc linux-ide earlier, sorry)
> > 
> > On Mon, Sep 26, 2011 at 01:05:50AM -0700, Simon Kirby wrote:
> > > Ok, good. It's still running without any problem, and no new leaks
> > > reported.
> > 
> > Ok.
> > 
> > [..]
> > 
> > > > backporting it to -stable is a good point. I'll add the proper tagging
> > > > to the patch.
> > > 
> > > Do you know in which version the issue started, then?
> > > 
> > > If not, all I have to start with is that it was fine on 2.6.36, and I can
> > > bisect it, if that would help.
> > 
> > This is exactly the question: AFAICT, it could be that changes in the
> > block layer at some point have caused the ide bust and since almost no
> > one tests ide...
> > 
> > The patch adding the dynamic ide_cmd struct allocation is
> > 395d8ef5bebe547a80737692f9789d2e36da16f2 from 2008 and I don't think it
> > caused the issue then but I could also be remembering it wrong.
> > 
> > So I wouldn't bisect it but test stable trees after 2.6.36 to see
> > whether they have the issue, and if so, only then the patch should be
> > backported.
> > 
> > And this is not that easy now with k.org down but looking at
> > 
> > http://web.archive.org/web/20110725015737/http://kernel.org/
> > 
> > the only stable trees which need to be tested are 2.6.39 and 3.0.
> > 
> > How does that sound?
> 
> Btw, here's the patch, if you would like to test 2.6.39 once without it
> to see whether kmemleak screams and once with it, I'll add the stable
> tagging.
> 
> Thanks.
> 
> --
> >From 96414ddbfecaaa3d265794c0792d816cf3c1e33d Mon Sep 17 00:00:00 2001
> From: Borislav Petkov <bp@alien8.de>
> Date: Sun, 25 Sep 2011 13:38:04 +0200
> Subject: [PATCH] ide-disk: Fix request requeuing
> 
> Simon Kirby reported that on his RAID setup with idedisk underneath
> the box OOMs after a couple of days of runtime. Running with
> CONFIG_DEBUG_KMEMLEAK pointed to idedisk_prep_fn() which unconditionally
> allocates an ide_cmd struct. However, ide_requeue_and_plug() can be
> called more than once per request, either from the request issue or the
> IRQ handler path and do blk_peek_request() ends up in idedisk_prep_fn()
> repeatedly, allocating a struct ide_cmd everytime and "forgetting" the
> previous pointer.
> 
> Make sure the code reuses the old allocated chunk.
> 
> Reported-and-tested-by: Simon Kirby <sim@hostway.ca>
> Link: http://marc.info/?l=linux-kernel&m=131667641517919
> Link: http://lkml.kernel.org/r/20110922072643.GA27232@hostway.ca
> Signed-off-by: Borislav Petkov <bp@alien8.de>
> ---
>  drivers/ide/ide-disk.c |    7 ++++++-
>  1 files changed, 6 insertions(+), 1 deletions(-)
> 
> diff --git a/drivers/ide/ide-disk.c b/drivers/ide/ide-disk.c
> index 274798068a54..16f69be820c7 100644
> --- a/drivers/ide/ide-disk.c
> +++ b/drivers/ide/ide-disk.c
> @@ -435,7 +435,12 @@ static int idedisk_prep_fn(struct request_queue *q, struct request *rq)
>  	if (!(rq->cmd_flags & REQ_FLUSH))
>  		return BLKPREP_OK;
>  
> -	cmd = kzalloc(sizeof(*cmd), GFP_ATOMIC);
> +	if (rq->special) {
> +		cmd = rq->special;
> +		memset(cmd, 0, sizeof(*cmd));
> +	} else {
> +		cmd = kzalloc(sizeof(*cmd), GFP_ATOMIC);
> +	}
>  
>  	/* FIXME: map struct ide_taskfile on rq->cmd[] */
>  	BUG_ON(cmd == NULL);
> -- 
> 1.7.5.3.401.gfb674

Tested against on 2.6.39 with and without this patch, and it definitely
leaks without it and does not leak with it. 3.0 and 3.1-rc8 also seems
good with the patch.

I tested on another IDE box (with an old Quantum Fireball 6.4GB!) and
even with software RAID, I could not see the leak, so I had to use the
original box, of course. The only difference I could see is the test
box has piix and the production box has via82cxxx, but anyway...

Thanks!

Simon-

Re: [3.1-rc6] kmalloc(64) leak from IDE

[Borislav Petkov]

On Thu, Sep 29, 2011 at 03:45:05PM -0700, Simon Kirby wrote:
> Tested against on 2.6.39 with and without this patch, and it
> definitely leaks without it and does not leak with it. 3.0 and 3.1-rc8
> also seems good with the patch.

Good job, thanks!

> I tested on another IDE box (with an old Quantum Fireball 6.4GB!) and

:-)

> even with software RAID, I could not see the leak, so I had to use the
> original box, of course. The only difference I could see is the test
> box has piix and the production box has via82cxxx, but anyway...

It might be because the production box drive is reporting of being
capable of doing write cache flushes, i.e. it should say something like

ideXX: cache flushes supported

in dmesg. idedisk_prep_fn() is contingent on that and is used only then
AFAICT.

Thanks again for testing.

-- 
Regards/Gruss,
    Boris.