* 2.6.31-rc1 crashes randomly on my Machine.
@ 2009-06-26 6:56 Zeno Davatz
2009-06-26 7:15 ` Al Viro
0 siblings, 1 reply; 6+ messages in thread
From: Zeno Davatz @ 2009-06-26 6:56 UTC (permalink / raw)
To: linux-kernel
Hi
Yesterday I done a fresh pull after Linus released 2.6.31-rc1. After
doing a clean boot, my machine started crashing randomly. I rebooted
several times. Also my X-server and Window-Manager (ion3) did not
start properly and was not really working. The machine suddenly hung
out of nowhere. What do you need? I attached my /var/log/messages
I got stuff like this on my screen:
Jun 25 21:19:12 zenogentoo Code: 00 00 00 c7 47 20 00 00 00 00 c7 47
24 00 00 00 00 c7 47 10 00 00 00 00 c7 47 14 00 00 00 00 c7 47 0c 00
00 00 00 e9 27 ff ff ff <ff> 89 e5 57 56 53 83 ec 34 89 45 d0 89 55 cc
89 4d c8 8b 70 6c
Jun 25 21:19:12 zenogentoo EIP: [<c10d1d35>] seq_read+0x0/0x3a5 SS:ESP
0068:f4b01f44
Jun 25 21:19:12 zenogentoo CR2: 0000000053565be5
Jun 25 21:19:12 zenogentoo ---[ end trace 6254fef9dc80950b ]---
Jun 25 21:19:12 zenogentoo BUG: unable to handle kernel paging request
at 53565be5
Jun 25 21:19:12 zenogentoo IP: [<c10d1d35>] seq_read+0x0/0x3a5
Jun 25 21:19:12 zenogentoo *pde = 00000000
Jun 25 21:19:12 zenogentoo Oops: 0002 [#27] PREEMPT SMP
Jun 25 21:19:12 zenogentoo last sysfs file: /sys/class/dmi/id/chassis_asset_tag
Jun 25 21:19:12 zenogentoo Modules linked in:
Jun 25 21:19:12 zenogentoo
Jun 25 21:19:12 zenogentoo Pid: 2961, comm: sh Tainted: G D
(2.6.31-rc1 #50) MS-7137
Jun 25 21:19:12 zenogentoo EIP: 0060:[<c10d1d35>] EFLAGS: 00010286 CPU: 0
Jun 25 21:19:12 zenogentoo EIP is at seq_read+0x0/0x3a5
Jun 25 21:19:12 zenogentoo EAX: f6b70400 EBX: f7279d00 ECX: 00000400
EDX: b7fb9000
Jun 25 21:19:12 zenogentoo ESI: fffffffb EDI: c10d1d35 EBP: f635df64
ESP: f635df44
Jun 25 21:19:12 zenogentoo DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jun 25 21:19:12 zenogentoo Process sh (pid: 2961, ti=f635c000
task=f64edb60 task.ti=f635c000)
Jun 25 21:19:12 zenogentoo Stack:
Jun 25 21:19:12 zenogentoo c10f29e2 f635df98 00000400 b7fb9000
f6b70400 f6b70400 b7fb9000 00000400
Jun 25 21:19:12 zenogentoo <0> f635df8c c10bbb37 f635df98 00000000
f64cd500 00000073 c10f298b f6b70400
Jun 25 21:19:12 zenogentoo <0> fffffff7 00000000 f635dfac c10bbc96
f635df98 00000000 00000000 00000000
Jun 25 21:19:12 zenogentoo Call Trace:
Jun 25 21:19:12 zenogentoo [<c10f29e2>] ? proc_reg_read+0x57/0x78
Jun 25 21:19:12 zenogentoo [<c10bbb37>] ? vfs_read+0x8b/0x141
Jun 25 21:19:12 zenogentoo [<c10f298b>] ? proc_reg_read+0x0/0x78
Jun 25 21:19:12 zenogentoo [<c10bbc96>] ? sys_read+0x3d/0x6b
Jun 25 21:19:12 zenogentoo [<c1021290>] ? sysenter_do_call+0x12/0x2c
Jun 25 21:19:12 zenogentoo Code: 00 00 00 c7 47 20 00 00 00 00 c7 47
24 00 00 00 00 c7 47 10 00 00 00 00 c7 47 14 00 00 00 00 c7 47 0c 00
00 00 00 e9 27 ff ff ff <ff> 89 e5 57 56 53 83 ec 34 89 45 d0 89 55 cc
89 4d c8 8b 70 6c
Jun 25 21:19:12 zenogentoo EIP: [<c10d1d35>] seq_read+0x0/0x3a5 SS:ESP
0068:f635df44
Jun 25 21:19:12 zenogentoo CR2: 0000000053565be5
Jun 25 21:19:12 zenogentoo ---[ end trace 6254fef9dc80950c ]---
See attachment.
lscpi:
00:00.0 Host bridge: Intel Corporation 82915G/P/GV/GL/PL/910GL Memory
Controller Hub (rev 0e)
00:02.0 VGA compatible controller: Intel Corporation 82915G/GV/910GL
Integrated Graphics Controller (rev 0e)
00:1b.0 Audio device: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6
Family) High Definition Audio Controller (rev 05)
00:1d.0 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6
Family) USB UHCI #1 (rev 05)
00:1d.1 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6
Family) USB UHCI #2 (rev 05)
00:1d.2 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6
Family) USB UHCI #3 (rev 05)
00:1d.3 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6
Family) USB UHCI #4 (rev 05)
00:1d.7 USB Controller: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6
Family) USB2 EHCI Controller (rev 05)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d5)
00:1f.0 ISA bridge: Intel Corporation 82801FB/FR (ICH6/ICH6R) LPC
Interface Bridge (rev 05)
00:1f.1 IDE interface: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6
Family) IDE Controller (rev 05)
00:1f.2 IDE interface: Intel Corporation 82801FB/FW (ICH6/ICH6W) SATA
Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family)
SMBus Controller (rev 05)
02:03.0 FireWire (IEEE 1394): VIA Technologies, Inc. IEEE 1394 Host
Controller (rev 80)
02:06.0 Ethernet controller: Intel Corporation 82541GI Gigabit
Ethernet Controller (rev 05)
Thank you for your Feedback.
Best
Zeno
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.6.31-rc1 crashes randomly on my Machine.
2009-06-26 6:56 2.6.31-rc1 crashes randomly on my Machine Zeno Davatz
@ 2009-06-26 7:15 ` Al Viro
2009-06-26 7:39 ` Al Viro
0 siblings, 1 reply; 6+ messages in thread
From: Al Viro @ 2009-06-26 7:15 UTC (permalink / raw)
To: Zeno Davatz; +Cc: linux-kernel
On Fri, Jun 26, 2009 at 08:56:52AM +0200, Zeno Davatz wrote:
> Jun 25 21:19:12 zenogentoo Code: 00 00 00 c7 47 20 00 00 00 00 c7 47
> 24 00 00 00 00 c7 47 10 00 00 00 00 c7 47 14 00 00 00 00 c7 47 0c 00
> 00 00 00 e9 27 ff ff ff <ff> 89 e5 57 56 53 83 ec 34 89 45 d0 89 55 cc
> 89 4d c8 8b 70 6c
> Jun 25 21:19:12 zenogentoo EIP: [<c10d1d35>] seq_read+0x0/0x3a5 SS:ESP
> 0068:f4b01f44
> Jun 25 21:19:12 zenogentoo CR2: 0000000053565be5
> Jun 25 21:19:12 zenogentoo ---[ end trace 6254fef9dc80950b ]---
> Jun 25 21:19:12 zenogentoo BUG: unable to handle kernel paging request
> at 53565be5
Real cute... Disassembly of that sucker:
decl 0x535657e5(%ecx)
which matches nicely the address in page fault. However, that doesn't
look even remotely plausible for a beginning of function. OTOH,
disassembly at one byte offset from that gives
mov %esp,%ebp
push %edi
push %esi
push %ebx
which is exactly what you'd expect to see in such place. IOW, you've
got an off-by-one - it had jumped at one byte before the actual entry
point of seq_read().
The interesting question is whether that's a memory corruption of some
kind or a linker fuckup. Check what does System.map have for
seq_read; if it's that address (c10d1d35), the odds are that you've
got something fishy going on with linking. Do objdump of vmlinux and
check the functions nearby; ditto for fs/seq_file.o.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.6.31-rc1 crashes randomly on my Machine.
2009-06-26 7:15 ` Al Viro
@ 2009-06-26 7:39 ` Al Viro
2009-06-27 11:10 ` Zeno Davatz
0 siblings, 1 reply; 6+ messages in thread
From: Al Viro @ 2009-06-26 7:39 UTC (permalink / raw)
To: Zeno Davatz; +Cc: linux-kernel
On Fri, Jun 26, 2009 at 08:15:21AM +0100, Al Viro wrote:
> On Fri, Jun 26, 2009 at 08:56:52AM +0200, Zeno Davatz wrote:
>
> > Jun 25 21:19:12 zenogentoo Code: 00 00 00 c7 47 20 00 00 00 00 c7 47
> > 24 00 00 00 00 c7 47 10 00 00 00 00 c7 47 14 00 00 00 00 c7 47 0c 00
> > 00 00 00 e9 27 ff ff ff <ff> 89 e5 57 56 53 83 ec 34 89 45 d0 89 55 cc
> > 89 4d c8 8b 70 6c
> > Jun 25 21:19:12 zenogentoo EIP: [<c10d1d35>] seq_read+0x0/0x3a5 SS:ESP
> > 0068:f4b01f44
> > Jun 25 21:19:12 zenogentoo CR2: 0000000053565be5
> > Jun 25 21:19:12 zenogentoo ---[ end trace 6254fef9dc80950b ]---
> > Jun 25 21:19:12 zenogentoo BUG: unable to handle kernel paging request
> > at 53565be5
>
> Real cute... Disassembly of that sucker:
> decl 0x535657e5(%ecx)
> which matches nicely the address in page fault. However, that doesn't
> look even remotely plausible for a beginning of function. OTOH,
> disassembly at one byte offset from that gives
> mov %esp,%ebp
> push %edi
> push %esi
> push %ebx
> which is exactly what you'd expect to see in such place.
Actually, it's not *quite* what you'd expect to see. What's missing is
push %ebp
as the first instruction, preceding that stuff. And it would take one
byte, so...
> IOW, you've
> got an off-by-one - it had jumped at one byte before the actual entry
> point of seq_read().
... this is not an off-by-one at all. The first byte of function code
got overwritten with 0xff. Code before that doesn't seem to be mangled -
it's
movl $0x0,0x20(%edi)
movl $0x0,0x24(%edi)
movl $0x0,0x10(%edi)
movl $0x0,0x14(%edi)
movl $0x0,0xc(%edi)
jmp <a bit back>
which is at least not entirely implausible. So it seems to be a memory
corruption in .text, which might or might not affect the directly
preceding bytes (0xe9 <signed 32bit int> is a relative jump, so there's
no way to tell whether this 0xff had been the only byte affected - it
would be preceded by 3 0xff coming from small negative integer anyway).
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.6.31-rc1 crashes randomly on my Machine.
2009-06-26 7:39 ` Al Viro
@ 2009-06-27 11:10 ` Zeno Davatz
2009-06-27 16:42 ` Al Viro
0 siblings, 1 reply; 6+ messages in thread
From: Zeno Davatz @ 2009-06-27 11:10 UTC (permalink / raw)
To: Al Viro; +Cc: linux-kernel
On Fri, Jun 26, 2009 at 9:39 AM, Al Viro<viro@zeniv.linux.org.uk> wrote:
> On Fri, Jun 26, 2009 at 08:15:21AM +0100, Al Viro wrote:
>> On Fri, Jun 26, 2009 at 08:56:52AM +0200, Zeno Davatz wrote:
>>
>> > Jun 25 21:19:12 zenogentoo Code: 00 00 00 c7 47 20 00 00 00 00 c7 47
>> > 24 00 00 00 00 c7 47 10 00 00 00 00 c7 47 14 00 00 00 00 c7 47 0c 00
>> > 00 00 00 e9 27 ff ff ff <ff> 89 e5 57 56 53 83 ec 34 89 45 d0 89 55 cc
>> > 89 4d c8 8b 70 6c
>> > Jun 25 21:19:12 zenogentoo EIP: [<c10d1d35>] seq_read+0x0/0x3a5 SS:ESP
>> > 0068:f4b01f44
>> > Jun 25 21:19:12 zenogentoo CR2: 0000000053565be5
>> > Jun 25 21:19:12 zenogentoo ---[ end trace 6254fef9dc80950b ]---
>> > Jun 25 21:19:12 zenogentoo BUG: unable to handle kernel paging request
>> > at 53565be5
>>
>> Real cute... Disassembly of that sucker:
>> decl 0x535657e5(%ecx)
>> which matches nicely the address in page fault. However, that doesn't
>> look even remotely plausible for a beginning of function. OTOH,
>> disassembly at one byte offset from that gives
>> mov %esp,%ebp
>> push %edi
>> push %esi
>> push %ebx
>> which is exactly what you'd expect to see in such place.
>
> Actually, it's not *quite* what you'd expect to see. What's missing is
> push %ebp
> as the first instruction, preceding that stuff. And it would take one
> byte, so...
>
>> IOW, you've
>> got an off-by-one - it had jumped at one byte before the actual entry
>> point of seq_read().
>
> ... this is not an off-by-one at all. The first byte of function code
> got overwritten with 0xff. Code before that doesn't seem to be mangled -
> it's
> movl $0x0,0x20(%edi)
> movl $0x0,0x24(%edi)
> movl $0x0,0x10(%edi)
> movl $0x0,0x14(%edi)
> movl $0x0,0xc(%edi)
> jmp <a bit back>
> which is at least not entirely implausible. So it seems to be a memory
> corruption in .text, which might or might not affect the directly
> preceding bytes (0xe9 <signed 32bit int> is a relative jump, so there's
> no way to tell whether this 0xff had been the only byte affected - it
> would be preceded by 3 0xff coming from small negative integer anyway).
I just done another pull from the Git repository of Linus and booted
from the latest 2.6.31-rc1 and my Machine still hangs after boot up,
with the following message at the end in /var/log/messages
Jun 27 03:01:52 zenogentoo Stack:
Jun 27 03:01:52 zenogentoo c10d14f2 f2eb9f5c c10ab407 00000400
b8033000 f6b43d80 f33cbe28 00000000
Jun 27 03:01:52 zenogentoo <0> 00000000 f65c9000 00001000 00000000
00000000 00000000 f721a100 fffffffb
Jun 27 03:01:52 zenogentoo <0> c10d13e5 f2eb9f64 c10f1522 f2eb9f98
00000400 b8033000 f6b43d80 f6b43d80
Jun 27 03:01:52 zenogentoo Call Trace:
Jun 27 03:01:52 zenogentoo [<c10d14f2>] ? seq_read+0x10d/0x3a5
Jun 27 03:01:52 zenogentoo [<c10ab407>] ? mmap_region+0x1bf/0x41a
Jun 27 03:01:52 zenogentoo [<c10d13e5>] ? seq_read+0x0/0x3a5
Jun 27 03:01:52 zenogentoo [<c10f1522>] ? proc_reg_read+0x57/0x78
Jun 27 03:01:52 zenogentoo [<c10bb257>] ? vfs_read+0x8b/0x141
Jun 27 03:01:52 zenogentoo [<c10f14cb>] ? proc_reg_read+0x0/0x78
Jun 27 03:01:52 zenogentoo [<c10bb3b6>] ? sys_read+0x3d/0x6b
Jun 27 03:01:52 zenogentoo [<c1021290>] ? sysenter_do_call+0x12/0x2c
Jun 27 03:01:52 zenogentoo Code: 0b fc f6 50 0b fc f6 01 00 00 00 00
00 00 00 60 0b fc f6 60 0b fc f6 00 00 00 00 00 00 00 00 00 00 00 00
08 00 00 00 00 00 00 00 <ff> ff ff ff ff ff ff ff 00 00 00 00 00 00 00
00 00 00 00 00 00
Jun 27 03:01:52 zenogentoo EIP: [<f6fc0b7c>] 0xf6fc0b7c SS:ESP 0068:f2eb9efc
Jun 27 03:01:52 zenogentoo ---[ end trace 1b3422263ead727b ]---
Jun 27 13:02:32 zenogentoo Stack:
Jun 27 13:02:32 zenogentoo 00000002 f7000a00 f7002564 f7002550
f7002540 f7000a00 c2686660 f707bf70
Jun 27 13:02:32 zenogentoo <0> c10b509e 00000000 00000000 f707bf70
c2689600 c2686660 f7059130 f707bfb8
Jun 27 13:02:32 zenogentoo <0> c105f8a7 c2688f80 00000001 fffedb31
c2684e00 c268960c c2689604 c2689600
Jun 27 13:02:32 zenogentoo Call Trace:
Jun 27 13:02:32 zenogentoo [<c10b509e>] ? cache_reap+0xbf/0xe9
Jun 27 13:02:32 zenogentoo [<c105f8a7>] ? worker_thread+0x158/0x23b
Jun 27 13:02:32 zenogentoo [<c10b4fdf>] ? cache_reap+0x0/0xe9
Jun 27 13:02:32 zenogentoo [<c1063a8f>] ? autoremove_wake_function+0x0/0x3a
Jun 27 13:02:32 zenogentoo [<c105f74f>] ? worker_thread+0x0/0x23b
Jun 27 13:02:32 zenogentoo [<c106376c>] ? kthread+0x6f/0x75
Jun 27 13:02:32 zenogentoo [<c10636fd>] ? kthread+0x0/0x75
Jun 27 13:02:32 zenogentoo [<c1021da7>] ? kernel_thread_helper+0x7/0x10
Jun 27 13:02:32 zenogentoo Code: 56 53 83 ec 10 89 45 e8 89 d6 89 4d
e4 85 c9 7e 79 8d 42 10 89 45 f0 3b 42 10 74 6e 8d 52 24 89 55 ec 31
ff eb 42 8b 13 8b 43 04 <89> 42 04 89 10 c7 03 00 01 10 00 c7 43 04 00
02 20 00 8b 55 e8
Jun 27 13:02:32 zenogentoo EIP: [<c10b4f7c>] drain_freelist+0x2f/0x92
SS:ESP 0068:f707bf34
Jun 27 13:02:32 zenogentoo CR2: 0000000000000104
Jun 27 13:02:32 zenogentoo ---[ end trace 1b3422263ead727d ]---
Also the date does not seem to be set correctly from the system (ion3
shows me some ??? where I normally get the time and date).
Best
Zeno
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.6.31-rc1 crashes randomly on my Machine.
2009-06-27 11:10 ` Zeno Davatz
@ 2009-06-27 16:42 ` Al Viro
2009-06-29 6:35 ` Zeno Davatz
0 siblings, 1 reply; 6+ messages in thread
From: Al Viro @ 2009-06-27 16:42 UTC (permalink / raw)
To: Zeno Davatz; +Cc: linux-kernel
On Sat, Jun 27, 2009 at 01:10:46PM +0200, Zeno Davatz wrote:
> > which is at least not entirely implausible. ?So it seems to be a memory
> > corruption in .text, which might or might not affect the directly
> > preceding bytes (0xe9 <signed 32bit int> is a relative jump, so there's
> > no way to tell whether this 0xff had been the only byte affected - it
> > would be preceded by 3 0xff coming from small negative integer anyway).
>
> I just done another pull from the Git repository of Linus and booted
> from the latest 2.6.31-rc1 and my Machine still hangs after boot up,
> with the following message at the end in /var/log/messages
>
> Jun 27 03:01:52 zenogentoo Stack:
> Jun 27 03:01:52 zenogentoo c10d14f2 f2eb9f5c c10ab407 00000400
> b8033000 f6b43d80 f33cbe28 00000000
> Jun 27 03:01:52 zenogentoo <0> 00000000 f65c9000 00001000 00000000
> 00000000 00000000 f721a100 fffffffb
> Jun 27 03:01:52 zenogentoo <0> c10d13e5 f2eb9f64 c10f1522 f2eb9f98
> 00000400 b8033000 f6b43d80 f6b43d80
> Jun 27 03:01:52 zenogentoo Call Trace:
> Jun 27 03:01:52 zenogentoo [<c10d14f2>] ? seq_read+0x10d/0x3a5
> Jun 27 03:01:52 zenogentoo [<c10ab407>] ? mmap_region+0x1bf/0x41a
> Jun 27 03:01:52 zenogentoo [<c10d13e5>] ? seq_read+0x0/0x3a5
> Jun 27 03:01:52 zenogentoo [<c10f1522>] ? proc_reg_read+0x57/0x78
> Jun 27 03:01:52 zenogentoo [<c10bb257>] ? vfs_read+0x8b/0x141
> Jun 27 03:01:52 zenogentoo [<c10f14cb>] ? proc_reg_read+0x0/0x78
> Jun 27 03:01:52 zenogentoo [<c10bb3b6>] ? sys_read+0x3d/0x6b
> Jun 27 03:01:52 zenogentoo [<c1021290>] ? sysenter_do_call+0x12/0x2c
> Jun 27 03:01:52 zenogentoo Code: 0b fc f6 50 0b fc f6 01 00 00 00 00
> 00 00 00 60 0b fc f6 60 0b fc f6 00 00 00 00 00 00 00 00 00 00 00 00
> 08 00 00 00 00 00 00 00 <ff> ff ff ff ff ff ff ff 00 00 00 00 00 00 00
> 00 00 00 00 00 00
> Jun 27 03:01:52 zenogentoo EIP: [<f6fc0b7c>] 0xf6fc0b7c SS:ESP 0068:f2eb9efc
> Jun 27 03:01:52 zenogentoo ---[ end trace 1b3422263ead727b ]---
Jumped to nowhere. For one thing, 0xf6fc0b7c is nowhere near the addresses
where the kernel code would live. For another, the contents of memory at
that address doesn't look code (a lot of 0, a lot of 0xff *and* several
32bit values that look like addresses nearby (0xf6fc0b50, 0xf6fc0b60).
IOW, some data structures; hell knows what it might have been, but we
have seq_read() seeing m->op->start that points somewhere strange.
Again, memory corruption of some kind. We have file->private_data that
might have been screwed up, or it might have been right pointer, but
the struct seq_file it points had been overwritten with some crap, or
that might have happened to the methods table ->op of that seq_file points
to...
Having looked at what seq_read() has compiled to in your kernel... what's
the value of ECX in that oops? That's where m->op ends up and a look at
that sucker might at least narrow it down.
Said that, at this point I'd
* run memtest, just to exclude the hardware crapping itself; nastier
coincidences happened
* try bisecting, if oopsen are easy to trigger.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 2.6.31-rc1 crashes randomly on my Machine.
2009-06-27 16:42 ` Al Viro
@ 2009-06-29 6:35 ` Zeno Davatz
0 siblings, 0 replies; 6+ messages in thread
From: Zeno Davatz @ 2009-06-29 6:35 UTC (permalink / raw)
To: Al Viro, linux-kernel
Dear Al
On Sat, Jun 27, 2009 at 6:42 PM, Al Viro<viro@zeniv.linux.org.uk> wrote:
> On Sat, Jun 27, 2009 at 01:10:46PM +0200, Zeno Davatz wrote:
>> > which is at least not entirely implausible. ?So it seems to be a memory
>> > corruption in .text, which might or might not affect the directly
>> > preceding bytes (0xe9 <signed 32bit int> is a relative jump, so there's
>> > no way to tell whether this 0xff had been the only byte affected - it
>> > would be preceded by 3 0xff coming from small negative integer anyway).
>>
>> I just done another pull from the Git repository of Linus and booted
>> from the latest 2.6.31-rc1 and my Machine still hangs after boot up,
>> with the following message at the end in /var/log/messages
>>
>> Jun 27 03:01:52 zenogentoo Stack:
>> Jun 27 03:01:52 zenogentoo c10d14f2 f2eb9f5c c10ab407 00000400
>> b8033000 f6b43d80 f33cbe28 00000000
>> Jun 27 03:01:52 zenogentoo <0> 00000000 f65c9000 00001000 00000000
>> 00000000 00000000 f721a100 fffffffb
>> Jun 27 03:01:52 zenogentoo <0> c10d13e5 f2eb9f64 c10f1522 f2eb9f98
>> 00000400 b8033000 f6b43d80 f6b43d80
>> Jun 27 03:01:52 zenogentoo Call Trace:
>> Jun 27 03:01:52 zenogentoo [<c10d14f2>] ? seq_read+0x10d/0x3a5
>> Jun 27 03:01:52 zenogentoo [<c10ab407>] ? mmap_region+0x1bf/0x41a
>> Jun 27 03:01:52 zenogentoo [<c10d13e5>] ? seq_read+0x0/0x3a5
>> Jun 27 03:01:52 zenogentoo [<c10f1522>] ? proc_reg_read+0x57/0x78
>> Jun 27 03:01:52 zenogentoo [<c10bb257>] ? vfs_read+0x8b/0x141
>> Jun 27 03:01:52 zenogentoo [<c10f14cb>] ? proc_reg_read+0x0/0x78
>> Jun 27 03:01:52 zenogentoo [<c10bb3b6>] ? sys_read+0x3d/0x6b
>> Jun 27 03:01:52 zenogentoo [<c1021290>] ? sysenter_do_call+0x12/0x2c
>> Jun 27 03:01:52 zenogentoo Code: 0b fc f6 50 0b fc f6 01 00 00 00 00
>> 00 00 00 60 0b fc f6 60 0b fc f6 00 00 00 00 00 00 00 00 00 00 00 00
>> 08 00 00 00 00 00 00 00 <ff> ff ff ff ff ff ff ff 00 00 00 00 00 00 00
>> 00 00 00 00 00 00
>> Jun 27 03:01:52 zenogentoo EIP: [<f6fc0b7c>] 0xf6fc0b7c SS:ESP 0068:f2eb9efc
>> Jun 27 03:01:52 zenogentoo ---[ end trace 1b3422263ead727b ]---
>
> Jumped to nowhere. For one thing, 0xf6fc0b7c is nowhere near the addresses
> where the kernel code would live. For another, the contents of memory at
> that address doesn't look code (a lot of 0, a lot of 0xff *and* several
> 32bit values that look like addresses nearby (0xf6fc0b50, 0xf6fc0b60).
> IOW, some data structures; hell knows what it might have been, but we
> have seq_read() seeing m->op->start that points somewhere strange.
>
> Again, memory corruption of some kind. We have file->private_data that
> might have been screwed up, or it might have been right pointer, but
> the struct seq_file it points had been overwritten with some crap, or
> that might have happened to the methods table ->op of that seq_file points
> to...
>
> Having looked at what seq_read() has compiled to in your kernel... what's
> the value of ECX in that oops? That's where m->op ends up and a look at
> that sucker might at least narrow it down.
My ECX Values are:
Jun 25 21:19:12 zenogentoo EAX: f6b70400 EBX: f7279d00 ECX: 00000400
EDX: b7fb9000
Jun 25 21:19:12 zenogentoo ESI: fffffffb EDI: c10d1d35 EBP: f635df64
ESP: f635df44
Jun 25 21:19:12 zenogentoo DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jun 25 21:19:12 zenogentoo Process sh (pid: 2961, ti=f635c000
task=f64edb60 task.ti=f635c000)
Jun 25 21:19:12 zenogentoo Stack:
Jun 25 21:19:12 zenogentoo c10f29e2 f635df98 00000400 b7fb9000
f6b70400 f6b70400 b7fb9000 00000400
Jun 25 21:19:12 zenogentoo <0> f635df8c c10bbb37 f635df98 00000000
f64cd500 00000073 c10f298b f6b70400
Jun 25 21:19:12 zenogentoo <0> fffffff7 00000000 f635dfac c10bbc96
f635df98 00000000 00000000 00000000
Jun 25 21:19:12 zenogentoo Call Trace:
Jun 25 21:19:12 zenogentoo [<c10f29e2>] ? proc_reg_read+0x57/0x78
Jun 25 21:19:12 zenogentoo [<c10bbb37>] ? vfs_read+0x8b/0x141
Jun 25 21:19:12 zenogentoo [<c10f298b>] ? proc_reg_read+0x0/0x78
Jun 25 21:19:12 zenogentoo [<c10bbc96>] ? sys_read+0x3d/0x6b
Jun 25 21:19:12 zenogentoo [<c1021290>] ? sysenter_do_call+0x12/0x2c
Jun 25 21:19:12 zenogentoo Code: 00 00 00 c7 47 20 00 00 00 00 c7 47
24 00 00 00 00 c7 47 10 00 00 00 00 c7 47 14 00 00 00 00 c7 47 0c 00
00 00 00 e9 27 ff ff ff <ff> 89 e5 57 56 53 83 ec 34 89 45 d0 89 55 cc
89 4d c8 8b 70 6c
Jun 25 21:19:12 zenogentoo EIP: [<c10d1d35>] seq_read+0x0/0x3a5 SS:ESP
0068:f635df44
Jun 25 21:19:12 zenogentoo CR2: 0000000053565be5
Jun 25 21:19:12 zenogentoo ---[ end trace 6254fef9dc80950c ]---
Jun 27 03:01:52 zenogentoo EAX: f33cbe00 EBX: fffffff4 ECX: f6077f20
EDX: f2eb9f2c
Jun 27 03:01:52 zenogentoo ESI: f33cbe00 EDI: c10d13e5 EBP: f2eb9f40
ESP: f2eb9efc
Jun 27 03:01:52 zenogentoo DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jun 27 03:01:52 zenogentoo Process sh (pid: 2889, ti=f2eb8000
task=f691ebe0 task.ti=f2eb8000)
Jun 27 03:01:52 zenogentoo Stack:
Jun 27 03:01:52 zenogentoo c10d14f2 f2eb9f5c c10ab407 00000400
b8033000 f6b43d80 f33cbe28 00000000
Jun 27 03:01:52 zenogentoo <0> 00000000 f65c9000 00001000 00000000
00000000 00000000 f721a100 fffffffb
Jun 27 03:01:52 zenogentoo <0> c10d13e5 f2eb9f64 c10f1522 f2eb9f98
00000400 b8033000 f6b43d80 f6b43d80
Jun 27 03:01:52 zenogentoo Call Trace:
Jun 27 03:01:52 zenogentoo [<c10d14f2>] ? seq_read+0x10d/0x3a5
Jun 27 03:01:52 zenogentoo [<c10ab407>] ? mmap_region+0x1bf/0x41a
Jun 27 03:01:52 zenogentoo [<c10d13e5>] ? seq_read+0x0/0x3a5
Jun 27 03:01:52 zenogentoo [<c10f1522>] ? proc_reg_read+0x57/0x78
Jun 27 03:01:52 zenogentoo [<c10bb257>] ? vfs_read+0x8b/0x141
Jun 27 03:01:52 zenogentoo [<c10f14cb>] ? proc_reg_read+0x0/0x78
Jun 27 03:01:52 zenogentoo [<c10bb3b6>] ? sys_read+0x3d/0x6b
Jun 27 03:01:52 zenogentoo [<c1021290>] ? sysenter_do_call+0x12/0x2c
Jun 27 03:01:52 zenogentoo Code: 0b fc f6 50 0b fc f6 01 00 00 00 00
00 00 00 60 0b fc f6 60 0b fc f6 00 00 00 00 00 00 00 00 00 00 00 00
08 00 00 00 00 00 00 00 <ff> ff ff ff ff ff ff ff 00 00 00 00 00 00 00
00 00 00 00 00 00
Jun 27 03:01:52 zenogentoo EIP: [<f6fc0b7c>] 0xf6fc0b7c SS:ESP 0068:f2eb9efc
Jun 27 03:01:52 zenogentoo ---[ end trace 1b3422263ead727b ]---
Jun 27 03:01:52 zenogentoo EAX: f33cbe00 EBX: fffffff4 ECX: f6077f20
EDX: f2ebdf2c
Jun 27 03:01:52 zenogentoo ESI: f33cbe00 EDI: c10d13e5 EBP: f2ebdf40
ESP: f2ebdefc
Jun 27 03:01:52 zenogentoo DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Jun 27 03:01:52 zenogentoo Process xmessage (pid: 2891, ti=f2ebc000
task=f71bb440 task.ti=f2ebc000)
Jun 27 03:01:52 zenogentoo Stack:
Jun 27 03:01:52 zenogentoo c10d14f2 f2ebdf5c c10ab407 00000400
b801f000 f6b20b80 f33cbe28 00000000
Jun 27 03:01:52 zenogentoo <0> 00000000 f65c9000 00001000 00000000
00000000 00000000 f721a100 fffffffb
Jun 27 03:01:52 zenogentoo <0> c10d13e5 f2ebdf64 c10f1522 f2ebdf98
00000400 b801f000 f6b20b80 f6b20b80
Jun 27 03:01:52 zenogentoo Call Trace:
Jun 27 03:01:52 zenogentoo [<c10d14f2>] ? seq_read+0x10d/0x3a5
Jun 27 03:01:52 zenogentoo [<c10ab407>] ? mmap_region+0x1bf/0x41a
Jun 27 03:01:52 zenogentoo [<c10d13e5>] ? seq_read+0x0/0x3a5
Jun 27 03:01:52 zenogentoo [<c10f1522>] ? proc_reg_read+0x57/0x78
Jun 27 03:01:52 zenogentoo [<c10bb257>] ? vfs_read+0x8b/0x141
Jun 27 03:01:52 zenogentoo [<c10f14cb>] ? proc_reg_read+0x0/0x78
Jun 27 03:01:52 zenogentoo [<c10bb3b6>] ? sys_read+0x3d/0x6b
Jun 27 03:01:52 zenogentoo [<c1021290>] ? sysenter_do_call+0x12/0x2c
Jun 27 03:01:52 zenogentoo Code: 0b fc f6 50 0b fc f6 01 00 00 00 00
00 00 00 60 0b fc f6 60 0b fc f6 00 00 00 00 00 00 00 00 00 00 00 00
08 00 00 00 00 00 00 00 <ff> ff ff ff ff ff ff ff 00 00 00 00 00 00 00
00 00 00 00 00 00
Jun 27 03:01:52 zenogentoo EIP: [<f6fc0b7c>] 0xf6fc0b7c SS:ESP 0068:f2ebdefc
Jun 27 03:01:52 zenogentoo ---[ end trace 1b3422263ead727c ]---
Jun 27 13:02:32 zenogentoo EAX: f695cc00 EBX: f6077f20 ECX: 00000002
EDX: 00000100
Jun 27 13:02:32 zenogentoo ESI: f7002540 EDI: 00000000 EBP: f707bf50
ESP: f707bf34
Jun 27 13:02:32 zenogentoo DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Jun 27 13:02:32 zenogentoo Process events/1 (pid: 11, ti=f707a000
task=f7059130 task.ti=f707a000)
Jun 27 13:02:32 zenogentoo Stack:
Jun 27 13:02:32 zenogentoo 00000002 f7000a00 f7002564 f7002550
f7002540 f7000a00 c2686660 f707bf70
Jun 27 13:02:32 zenogentoo <0> c10b509e 00000000 00000000 f707bf70
c2689600 c2686660 f7059130 f707bfb8
Jun 27 13:02:32 zenogentoo <0> c105f8a7 c2688f80 00000001 fffedb31
c2684e00 c268960c c2689604 c2689600
Jun 27 13:02:32 zenogentoo Call Trace:
Jun 27 13:02:32 zenogentoo [<c10b509e>] ? cache_reap+0xbf/0xe9
Jun 27 13:02:32 zenogentoo [<c105f8a7>] ? worker_thread+0x158/0x23b
Jun 27 13:02:32 zenogentoo [<c10b4fdf>] ? cache_reap+0x0/0xe9
Jun 27 13:02:32 zenogentoo [<c1063a8f>] ? autoremove_wake_function+0x0/0x3a
Jun 27 13:02:32 zenogentoo [<c105f74f>] ? worker_thread+0x0/0x23b
Jun 27 13:02:32 zenogentoo [<c106376c>] ? kthread+0x6f/0x75
Jun 27 13:02:32 zenogentoo [<c10636fd>] ? kthread+0x0/0x75
Jun 27 13:02:32 zenogentoo [<c1021da7>] ? kernel_thread_helper+0x7/0x10
Jun 27 13:02:32 zenogentoo Code: 56 53 83 ec 10 89 45 e8 89 d6 89 4d
e4 85 c9 7e 79 8d 42 10 89 45 f0 3b 42 10 74 6e 8d 52 24 89 55 ec 31
ff eb 42 8b 13 8b 43 04 <89> 42 04 89 10 c7 03 00 01 10 00 c7 43 04 00
02 20 00 8b 55 e8
Jun 27 13:02:32 zenogentoo EIP: [<c10b4f7c>] drain_freelist+0x2f/0x92
SS:ESP 0068:f707bf34
Jun 27 13:02:32 zenogentoo CR2: 0000000000000104
Jun 27 13:02:32 zenogentoo ---[ end trace 1b3422263ead727d ]---
I got a lot of the
ECX: 00000400
values around the hanging.
Best
Zeno
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2009-06-29 6:35 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-06-26 6:56 2.6.31-rc1 crashes randomly on my Machine Zeno Davatz
2009-06-26 7:15 ` Al Viro
2009-06-26 7:39 ` Al Viro
2009-06-27 11:10 ` Zeno Davatz
2009-06-27 16:42 ` Al Viro
2009-06-29 6:35 ` Zeno Davatz
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.