linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [3.15-rc1 slab] Oops when reading /proc/slab_allocators
@ 2014-04-09 11:36 Tetsuo Handa
  2014-04-10  7:39 ` Joonsoo Kim
  0 siblings, 1 reply; 6+ messages in thread
From: Tetsuo Handa @ 2014-04-09 11:36 UTC (permalink / raw)
  To: iamjoonsoo.kim, ak, penberg; +Cc: linux-kernel

Hello.

I found that

  $ cat /proc/slab_allocators

causes an oops.

---------- dmesg start ----------
[   22.719620] BUG: unable to handle kernel paging request at ffff8800389b7ff8
[   22.719742] IP: [<ffffffff811b751d>] handle_slab+0x8d/0x1a0
[   22.719742] PGD 2d64067 PUD 2d65067 PMD 3fc20067 PTE 80000000389b7060
[   22.719742] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
[   22.719742] Modules linked in:
[   22.719742] CPU: 0 PID: 1 Comm: init Not tainted 3.14.0-12041-g75ff24f #297
[   22.719742] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[   22.719742] task: ffff88003decc010 ti: ffff88003ded0000 task.ti: ffff88003ded0000
[   22.719742] RIP: 0010:[<ffffffff811b751d>]  [<ffffffff811b751d>] handle_slab+0x8d/0x1a0
[   22.719742] RSP: 0018:ffff88003ded1d88  EFLAGS: 00000002
[   22.719742] RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff88003f625c80
[   22.719742] RDX: ffffea0000e26dc0 RSI: 0000000000000000 RDI: ffff88001763f000
[   22.719742] RBP: ffff88003ded1dd8 R08: ffff88001763f000 R09: ffff88001763f008
[   22.719742] R10: ffff88001763f010 R11: 0000000000001000 R12: ffff880017616f00
[   22.719742] R13: 0000000000000000 R14: ffff8800389b7000 R15: ffffea0000e26dc0
[   22.719742] FS:  00000000015b7860(0063) GS:ffff88003fa00000(0000) knlGS:0000000000000000
[   22.719742] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   22.719742] CR2: ffff8800389b7ff8 CR3: 0000000017619000 CR4: 00000000000006f0
[   22.719742] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   22.719742] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
[   22.719742] Stack:
[   22.719742]  ffff88001763f010 ffff88003f625c80 ffff88001763f000 ffff88001763f008
[   22.719742]  ffff88003ded1dd8 ffff88001763f000 ffff880017616f00 ffff88003f625c80
[   22.719742]  0000000000000400 ffffea0000e26dc0 ffff88003ded1e48 ffffffff811bcbd6
[   22.719742] Call Trace:
[   22.719742]  [<ffffffff811bcbd6>] leaks_show+0xf6/0x270
[   22.719742]  [<ffffffff811f529f>] seq_read+0x2af/0x440
[   22.719742]  [<ffffffff8123e213>] proc_reg_read+0x43/0x70
[   22.719742]  [<ffffffff811ccdcd>] vfs_read+0xad/0x190
[   22.719742]  [<ffffffff811ccf99>] SyS_read+0x59/0xd0
[   22.719742]  [<ffffffff81658412>] system_call_fastpath+0x16/0x1b
[   22.719742] Code: 75 18 e9 df 00 00 00 0f 1f 80 00 00 00 00 48 63 d0 3b 34 97 0f 84 cc 00 00 00 83 c0 01 39 c3 77 ed f6 41 1a 01 0f 84 02 01 00 00 <4f> 8b 64 1e f8 4d 85 e4 0f 84 ad 00 00 00 49 8b 39 4c 89 d3 89
[   22.719742] RIP  [<ffffffff811b751d>] handle_slab+0x8d/0x1a0
[   22.719742]  RSP <ffff88003ded1d88>
[   22.719742] CR2: ffff8800389b7ff8
[   22.719742] ---[ end trace b3602f55c39df20e ]---
---------- dmesg end ----------

Kernel config is at http://I-love.SAKURA.ne.jp/tmp/config-pre-3.15-rc1 .

The location is

  if (!add_caller(n, (unsigned long)*dbg_userword(c, p)))

in handle_slab() in mm/slab.c , and dbg_userword(c, p) is returning
an undereferencable value (0xffff8800389b7ff8) when
n=ffff880017645000, c=ffff88003f625c80, p=ffff8800389b7000,
c->flags=0x80010c00 and c->size=4096.

  static void **dbg_userword(struct kmem_cache *cachep, void *objp)
  {
          BUG_ON(!(cachep->flags & SLAB_STORE_USER));
          return (void **)(objp + cachep->size - BYTES_PER_WORD);
  }

Bisection reached commit b1cb0982 "slab: change the management method
of free objects of the slab".

  bad commit b1cb0982bdd6f57fed690f796659733350bb2cae
  good commit a57a49887eb3398d31bfaa8009531f7121d6537d
  untestable commit 56f295ef0dfa7e1d0be18deebe0c15fb6b2d9d5b
  untestable commit 0c3aa83e00a9cd93f08e7aa42fba01924aa5f2fc
  good commit 73293c2f900d0adbb6a415b312cd57976d5ae242

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: [3.15-rc1 slab] Oops when reading /proc/slab_allocators
  2014-04-09 11:36 [3.15-rc1 slab] Oops when reading /proc/slab_allocators Tetsuo Handa
@ 2014-04-10  7:39 ` Joonsoo Kim
  2014-04-10 11:54   ` Tetsuo Handa
  0 siblings, 1 reply; 6+ messages in thread
From: Joonsoo Kim @ 2014-04-10  7:39 UTC (permalink / raw)
  To: Tetsuo Handa; +Cc: ak, penberg, linux-kernel

On Wed, Apr 09, 2014 at 08:36:10PM +0900, Tetsuo Handa wrote:
> Hello.
> 
> I found that
> 
>   $ cat /proc/slab_allocators
> 
> causes an oops.
> 
> ---------- dmesg start ----------
> [   22.719620] BUG: unable to handle kernel paging request at ffff8800389b7ff8
> [   22.719742] IP: [<ffffffff811b751d>] handle_slab+0x8d/0x1a0
> [   22.719742] PGD 2d64067 PUD 2d65067 PMD 3fc20067 PTE 80000000389b7060
> [   22.719742] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
> [   22.719742] Modules linked in:
> [   22.719742] CPU: 0 PID: 1 Comm: init Not tainted 3.14.0-12041-g75ff24f #297
> [   22.719742] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
> [   22.719742] task: ffff88003decc010 ti: ffff88003ded0000 task.ti: ffff88003ded0000
> [   22.719742] RIP: 0010:[<ffffffff811b751d>]  [<ffffffff811b751d>] handle_slab+0x8d/0x1a0
> [   22.719742] RSP: 0018:ffff88003ded1d88  EFLAGS: 00000002
> [   22.719742] RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff88003f625c80
> [   22.719742] RDX: ffffea0000e26dc0 RSI: 0000000000000000 RDI: ffff88001763f000
> [   22.719742] RBP: ffff88003ded1dd8 R08: ffff88001763f000 R09: ffff88001763f008
> [   22.719742] R10: ffff88001763f010 R11: 0000000000001000 R12: ffff880017616f00
> [   22.719742] R13: 0000000000000000 R14: ffff8800389b7000 R15: ffffea0000e26dc0
> [   22.719742] FS:  00000000015b7860(0063) GS:ffff88003fa00000(0000) knlGS:0000000000000000
> [   22.719742] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   22.719742] CR2: ffff8800389b7ff8 CR3: 0000000017619000 CR4: 00000000000006f0
> [   22.719742] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [   22.719742] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
> [   22.719742] Stack:
> [   22.719742]  ffff88001763f010 ffff88003f625c80 ffff88001763f000 ffff88001763f008
> [   22.719742]  ffff88003ded1dd8 ffff88001763f000 ffff880017616f00 ffff88003f625c80
> [   22.719742]  0000000000000400 ffffea0000e26dc0 ffff88003ded1e48 ffffffff811bcbd6
> [   22.719742] Call Trace:
> [   22.719742]  [<ffffffff811bcbd6>] leaks_show+0xf6/0x270
> [   22.719742]  [<ffffffff811f529f>] seq_read+0x2af/0x440
> [   22.719742]  [<ffffffff8123e213>] proc_reg_read+0x43/0x70
> [   22.719742]  [<ffffffff811ccdcd>] vfs_read+0xad/0x190
> [   22.719742]  [<ffffffff811ccf99>] SyS_read+0x59/0xd0
> [   22.719742]  [<ffffffff81658412>] system_call_fastpath+0x16/0x1b
> [   22.719742] Code: 75 18 e9 df 00 00 00 0f 1f 80 00 00 00 00 48 63 d0 3b 34 97 0f 84 cc 00 00 00 83 c0 01 39 c3 77 ed f6 41 1a 01 0f 84 02 01 00 00 <4f> 8b 64 1e f8 4d 85 e4 0f 84 ad 00 00 00 49 8b 39 4c 89 d3 89
> [   22.719742] RIP  [<ffffffff811b751d>] handle_slab+0x8d/0x1a0
> [   22.719742]  RSP <ffff88003ded1d88>
> [   22.719742] CR2: ffff8800389b7ff8
> [   22.719742] ---[ end trace b3602f55c39df20e ]---
> ---------- dmesg end ----------
> 
> Kernel config is at http://I-love.SAKURA.ne.jp/tmp/config-pre-3.15-rc1 .
> 
> The location is
> 
>   if (!add_caller(n, (unsigned long)*dbg_userword(c, p)))
> 
> in handle_slab() in mm/slab.c , and dbg_userword(c, p) is returning
> an undereferencable value (0xffff8800389b7ff8) when
> n=ffff880017645000, c=ffff88003f625c80, p=ffff8800389b7000,
> c->flags=0x80010c00 and c->size=4096.
> 
>   static void **dbg_userword(struct kmem_cache *cachep, void *objp)
>   {
>           BUG_ON(!(cachep->flags & SLAB_STORE_USER));
>           return (void **)(objp + cachep->size - BYTES_PER_WORD);
>   }
> 
> Bisection reached commit b1cb0982 "slab: change the management method
> of free objects of the slab".
> 
>   bad commit b1cb0982bdd6f57fed690f796659733350bb2cae
>   good commit a57a49887eb3398d31bfaa8009531f7121d6537d
>   untestable commit 56f295ef0dfa7e1d0be18deebe0c15fb6b2d9d5b
>   untestable commit 0c3aa83e00a9cd93f08e7aa42fba01924aa5f2fc
>   good commit 73293c2f900d0adbb6a415b312cd57976d5ae242

Hello,

Thanks for reporting.

There was another report about this problem and I have already fixed
it, although it wasn't reviewed and merged. See following link.

https://lkml.org/lkml/2014/3/11/119

In that time, I thought that this problem wasn't related to my patches,
but your bisect report say that it is from my patches. I will dig into
the problem more deeply as soon as possible and make proper fix.

Thanks.



^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: [3.15-rc1 slab] Oops when reading /proc/slab_allocators
  2014-04-10  7:39 ` Joonsoo Kim
@ 2014-04-10 11:54   ` Tetsuo Handa
  2014-04-16  0:10     ` Joonsoo Kim
  0 siblings, 1 reply; 6+ messages in thread
From: Tetsuo Handa @ 2014-04-10 11:54 UTC (permalink / raw)
  To: iamjoonsoo.kim; +Cc: ak, penberg, linux-kernel

Joonsoo Kim wrote:
> There was another report about this problem and I have already fixed
> it, although it wasn't reviewed and merged. See following link.
> 
> https://lkml.org/lkml/2014/3/11/119

OK. That patch fixes this problem.

Commit b1cb0982 changed the way of walking objects, didn't it? Then,
it could happen that slab leak detector code walking all objects
triggers an oops.

> In that time, I thought that this problem wasn't related to my patches,
> but your bisect report say that it is from my patches. I will dig into
> the problem more deeply as soon as possible and make proper fix.

I see. Here is a minimal kernel config and test script for qemu on CentOS 6.5.
http://I-love.SAKURA.ne.jp/tmp/config-pre-3.15-rc1-min

---------- test.sh start ----------
#!/bin/sh
mkdir -p tmp/
cc -Wall -O3 --static -o tmp/init -x c - << "EOF"
#include <stdio.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/mount.h>
#include <sys/reboot.h> 
#include <linux/reboot.h> 

int main(int argc, char **argv)
{
	FILE *fp;
	mkdir("/proc", 0755);
	mount(NULL, "/proc", "proc", 0, NULL);
	fp = fopen("/proc/slab_allocators", "r");
	if (fp) {
		char buffer[4096];
		printf("Reading /proc/slab_allocators\n");
		while (fgets(buffer, sizeof(buffer), fp));
		fclose(fp);
	}
	printf("Shutting down.\n");
	reboot(LINUX_REBOOT_CMD_POWER_OFF);
	return 0;
}
EOF
(cd tmp; echo init | cpio -o -H newc | gzip -9) > initrd
exec /usr/libexec/qemu-kvm --kernel arch/x86/boot/bzImage --initrd initrd --nographic --append "console=ttyS0,115200n8"
---------- test.sh end ----------

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: [3.15-rc1 slab] Oops when reading /proc/slab_allocators
  2014-04-10 11:54   ` Tetsuo Handa
@ 2014-04-16  0:10     ` Joonsoo Kim
  2014-06-23 10:59       ` Tetsuo Handa
  0 siblings, 1 reply; 6+ messages in thread
From: Joonsoo Kim @ 2014-04-16  0:10 UTC (permalink / raw)
  To: Tetsuo Handa; +Cc: ak, penberg, linux-kernel

On Thu, Apr 10, 2014 at 08:54:37PM +0900, Tetsuo Handa wrote:
> Joonsoo Kim wrote:
> > There was another report about this problem and I have already fixed
> > it, although it wasn't reviewed and merged. See following link.
> > 
> > https://lkml.org/lkml/2014/3/11/119
> 
> OK. That patch fixes this problem.
> 
> Commit b1cb0982 changed the way of walking objects, didn't it? Then,
> it could happen that slab leak detector code walking all objects
> triggers an oops.

Yeah, although it doesn't walks all objects, it walks non-active
objects and it is problematic. This behaviour comes from my patches :)

> 
> > In that time, I thought that this problem wasn't related to my patches,
> > but your bisect report say that it is from my patches. I will dig into
> > the problem more deeply as soon as possible and make proper fix.
> 
> I see. Here is a minimal kernel config and test script for qemu on CentOS 6.5.
> http://I-love.SAKURA.ne.jp/tmp/config-pre-3.15-rc1-min

Thanks.
I already sent another patch to fix this bug and cc'ed you.

Thanks.

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: [3.15-rc1 slab] Oops when reading /proc/slab_allocators
  2014-04-16  0:10     ` Joonsoo Kim
@ 2014-06-23 10:59       ` Tetsuo Handa
  2014-06-25  1:26         ` Joonsoo Kim
  0 siblings, 1 reply; 6+ messages in thread
From: Tetsuo Handa @ 2014-06-23 10:59 UTC (permalink / raw)
  To: iamjoonsoo.kim; +Cc: ak, penberg, linux-kernel

Joonsoo Kim wrote:
> On Thu, Apr 10, 2014 at 08:54:37PM +0900, Tetsuo Handa wrote:
> > Joonsoo Kim wrote:
> > > There was another report about this problem and I have already fixed
> > > it, although it wasn't reviewed and merged. See following link.
> > > 
> > > https://lkml.org/lkml/2014/3/11/119
> > 
> > OK. That patch fixes this problem.
> > 
> > Commit b1cb0982 changed the way of walking objects, didn't it? Then,
> > it could happen that slab leak detector code walking all objects
> > triggers an oops.
> 
> Yeah, although it doesn't walks all objects, it walks non-active
> objects and it is problematic. This behaviour comes from my patches :)
> 
> > 
> > > In that time, I thought that this problem wasn't related to my patches,
> > > but your bisect report say that it is from my patches. I will dig into
> > > the problem more deeply as soon as possible and make proper fix.
> > 
> > I see. Here is a minimal kernel config and test script for qemu on CentOS 6.5.
> > http://I-love.SAKURA.ne.jp/tmp/config-pre-3.15-rc1-min
> 
> Thanks.
> I already sent another patch to fix this bug and cc'ed you.
> 
> Thanks.
> 

Hello.

I noticed that 3.16-rc2 shows the same bug.
Did you merge your patch for this bug into 3.15?

[   43.515887] BUG: unable to handle kernel paging request at ffff88003708aff8
[   43.519486] IP: [<ffffffff811c34ff>] handle_slab+0x8f/0x190
[   43.522267] PGD 31a5067 PUD 31a6067 PMD 7fc2b067 PTE 800000003708a060
[   43.525687] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
[   43.528033] Modules linked in: fuse ipv6 vhost_net macvtap macvlan vhost tun ppdev snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq dm_mod snd_seq_device snd_pcm snd_timer snd soundcore sg i2c_piix4 i2c_core parport_pc parport shpchp ext4(E) jbd2(E) mbcache(E) crc16(E) sd_mod(E) crc_t10dif(E) sr_mod(E) cdrom(E) vmxnet3(E) mptspi(E) mptscsih(E) mptbase(E) scsi_transport_spi(E) pata_acpi(E) ata_generic(E) ata_piix(E)
[   43.548360] CPU: 2 PID: 2088 Comm: cat Tainted: G        W   E 3.16.0-rc2 #401
[   43.551602] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
[   43.555332] task: ffff880034a59250 ti: ffff880035de0000 task.ti: ffff880035de0000
[   43.557070] RIP: 0010:[<ffffffff811c34ff>]  [<ffffffff811c34ff>] handle_slab+0x8f/0x190
[   43.559095] RSP: 0018:ffff880035de3d88  EFLAGS: 00010002
[   43.560368] RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffff8800476e75c0
[   43.562051] RDX: ffffea0000dc2280 RSI: 0000000000000001 RDI: ffff88003c19a000
[   43.563736] RBP: ffff880035de3dd8 R08: ffff88003c19a000 R09: ffff88003c19a008
[   43.565454] R10: ffff88003c19a010 R11: 0000000000001000 R12: ffff880048c3ff00
[   43.567137] R13: ffff8800476e75c0 R14: ffff88003708a000 R15: ffffea0000dc2280
[   43.568898] FS:  00007ffdc37d1700(0000) GS:ffff880079600000(0000) knlGS:0000000000000000
[   43.570904] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   43.572299] CR2: ffff88003708aff8 CR3: 00000000348ca000 CR4: 00000000000407e0
[   43.574082] Stack:
[   43.574598]  ffff8800476f1400 ffff8800476f1418 ffff8800476e75c0 0000000000000400
[   43.576500]  ffff880035de3dd8 ffff88003c19a000 ffff880048c3ff00 ffff8800476e75c0
[   43.576502]  0000000000000400 ffffea0000dc2280 ffff880035de3e48 ffffffff811c8bc6
[   43.576504] Call Trace:
[   43.576506]  [<ffffffff811c8bc6>] leaks_show+0xf6/0x270
[   43.576511]  [<ffffffff81200e8f>] seq_read+0x2af/0x440
[   43.576514]  [<ffffffff81245cb3>] proc_reg_read+0x43/0x70
[   43.576517]  [<ffffffff811d7adb>] vfs_read+0xab/0x120
[   43.576519]  [<ffffffff811fa173>] ? __fdget+0x13/0x20
[   43.576521]  [<ffffffff811d80f9>] SyS_read+0x59/0xd0
[   43.576526]  [<ffffffff8164ca12>] system_call_fastpath+0x16/0x1b
[   43.576551] Code: 75 18 e9 e5 00 00 00 0f 1f 44 00 00 89 c2 0f b6 14 17 39 da 0f 84 d2 00 00 00 83 c0 01 39 c6 77 eb f6 41 1a 01 0f 84 f8 00 00 00 <4f> 8b 6c 1e f8 4d 85 ed 0f 84 b3 00 00 00 49 8b 39 4d 89 d4 89 
[   43.576553] RIP  [<ffffffff811c34ff>] handle_slab+0x8f/0x190
[   43.576553]  RSP <ffff880035de3d88>
[   43.576554] CR2: ffff88003708aff8
[   43.576556] ---[ end trace 512dafa068f4bf70 ]---

^ permalink raw reply	[flat|nested] 6+ messages in thread
* Re: [3.15-rc1 slab] Oops when reading /proc/slab_allocators
  2014-06-23 10:59       ` Tetsuo Handa
@ 2014-06-25  1:26         ` Joonsoo Kim
  0 siblings, 0 replies; 6+ messages in thread
From: Joonsoo Kim @ 2014-06-25  1:26 UTC (permalink / raw)
  To: Tetsuo Handa; +Cc: ak, penberg, linux-kernel

On Mon, Jun 23, 2014 at 07:59:17PM +0900, Tetsuo Handa wrote:
> Joonsoo Kim wrote:
> > On Thu, Apr 10, 2014 at 08:54:37PM +0900, Tetsuo Handa wrote:
> > > Joonsoo Kim wrote:
> > > > There was another report about this problem and I have already fixed
> > > > it, although it wasn't reviewed and merged. See following link.
> > > > 
> > > > https://lkml.org/lkml/2014/3/11/119
> > > 
> > > OK. That patch fixes this problem.
> > > 
> > > Commit b1cb0982 changed the way of walking objects, didn't it? Then,
> > > it could happen that slab leak detector code walking all objects
> > > triggers an oops.
> > 
> > Yeah, although it doesn't walks all objects, it walks non-active
> > objects and it is problematic. This behaviour comes from my patches :)
> > 
> > > 
> > > > In that time, I thought that this problem wasn't related to my patches,
> > > > but your bisect report say that it is from my patches. I will dig into
> > > > the problem more deeply as soon as possible and make proper fix.
> > > 
> > > I see. Here is a minimal kernel config and test script for qemu on CentOS 6.5.
> > > http://I-love.SAKURA.ne.jp/tmp/config-pre-3.15-rc1-min
> > 
> > Thanks.
> > I already sent another patch to fix this bug and cc'ed you.
> > 
> > Thanks.
> > 
> 
> Hello.
> 
> I noticed that 3.16-rc2 shows the same bug.
> Did you merge your patch for this bug into 3.15?

Hello,

I'm afriad of not.
It is merged yesterday, so it will appear on 3.16-rc3.
Sorry for late.

Thanks.


^ permalink raw reply	[flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-06-25  1:21 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-04-09 11:36 [3.15-rc1 slab] Oops when reading /proc/slab_allocators Tetsuo Handa
2014-04-10  7:39 ` Joonsoo Kim
2014-04-10 11:54   ` Tetsuo Handa
2014-04-16  0:10     ` Joonsoo Kim
2014-06-23 10:59       ` Tetsuo Handa
2014-06-25  1:26         ` Joonsoo Kim

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).