From: Ryan Richter <ryan@tau.solarneutrino.net>
To: Kai Makisara <Kai.Makisara@kolumbus.fi>
Cc: James Bottomley <James.Bottomley@SteelEye.com>,
Linus Torvalds <torvalds@osdl.org>,
Hugh Dickins <hugh@veritas.com>, Andrew Morton <akpm@osdl.org>,
linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org,
ryan@tau.solarneutrino.net
Subject: Re: Fw: crash on x86_64 - mm related?
Date: Wed, 4 Jan 2006 12:27:27 -0500 [thread overview]
Message-ID: <20060104172727.GA320@tau.solarneutrino.net> (raw)
In-Reply-To: <Pine.LNX.4.63.0512271807130.4955@kai.makisara.local>
On Tue, Dec 27, 2005 at 06:21:39PM +0200, Kai Makisara wrote:
> On Mon, 26 Dec 2005, Ryan Richter wrote:
>
> > On Thu, Dec 15, 2005 at 08:01:43PM -0800, James Bottomley wrote:
> > > On Thu, 2005-12-15 at 14:09 -0500, Ryan Richter wrote:
> > > > On Mon, Dec 12, 2005 at 12:24:42PM -0600, James Bottomley wrote:
> > > > > I'll find a fix for the real problem, but this patch isn't the cause.
> > > >
> > > > Is the patch set you posted yesterday supposed to fix this? If so, is
> > > > it available in patch form anywhere?
> > >
> > > No, I've been too busin integrating other people's patches to work on
> > > ones of my own. Try this.
> >
> > It was looking good, but...
> >
> >
> > Bad page state at free_hot_cold_page (in process 'taper', page ffff8100040e22e8)
> > flags:0x010000000000000c mapping:ffff810102bba238 mapcount:2 count:0
> > Backtrace:
> >
> Looks familiar ;-(
>
> I don't have any new ideas but I will tell you what I have tried. In order
> to get more information about what is happening, I inserted the patch at
> the end of this message to my kernel. The purpose of the patch was to
> print something about the page mappings (prt_pages) and to catch possible
> double freeing from st earlier (clear page pointers).
>
> Running dump gave me the following output:
Here's what I got:
st: page attributes before page_release 8
0: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1392956
1: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1397945
2: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1537473
3: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1398161
4: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1398261
5: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1392778
6: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1402858
7: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1396799
Bad page state at free_hot_cold_page (in process 'taper', page ffff81000427ff60)
flags:0x010000000000000c mapping:ffff810102bbaaf0 mapcount:2 count:0
Backtrace:
Call Trace:<ffffffff80150234>{bad_page+116} <ffffffff80150c3f>{free_hot_cold_page+105}
<ffffffff8028c534>{sgl_unmap_user_pages+124} <ffffffff8028826d>{release_buffering+27}
<ffffffff802888f9>{st_write+1670} <ffffffff8016d9bc>{vfs_write+173}
<ffffffff8016dac8>{sys_write+69} <ffffffff8010d762>{system_call+126}
Trying to fix it up, but a reboot is needed
Bad page state at free_hot_cold_page (in process 'taper', page ffff810003ed9290)
flags:0x010000000000000c mapping:ffff810102bbaaf0 mapcount:2 count:0
Backtrace:
Call Trace:<ffffffff80150234>{bad_page+116} <ffffffff80150c3f>{free_hot_cold_page+105}
<ffffffff8028c534>{sgl_unmap_user_pages+124} <ffffffff8028826d>{release_buffering+27}
<ffffffff802888f9>{st_write+1670} <ffffffff8016d9bc>{vfs_write+173}
<ffffffff8016dac8>{sys_write+69} <ffffffff8010d762>{system_call+126}
Trying to fix it up, but a reboot is needed
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at mm/swap.c:49
invalid operand: 0000 [1] SMP
CPU 1
Modules linked in: bonding
Pid: 2892, comm: taper Tainted: G B 2.6.15 #1
RIP: 0010:[<ffffffff8015751c>] <ffffffff8015751c>{put_page+96}
RSP: 0018:ffff81017a247e18 EFLAGS: 00010256
RAX: 0000000000000000 RBX: 00000000000000c0 RCX: ffff81000427ff60
RDX: ffff81000427ff60 RSI: 0000000000000001 RDI: ffff81000427ff60
RBP: 0000000000000006 R08: ffff81017a246000 R09: 0000000000000001
R10: ffff8100f6f31aa0 R11: 0000000000000046 R12: 0000000000000008
R13: ffff8100f6f9e068 R14: 0000000000000000 R15: 0000000000008000
FS: 00002aaaab53d880(0000) GS:ffffffff804a9880(0000) knlGS:00000000556b6920
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002aaaaab5ffff CR3: 00000001722be000 CR4: 00000000000006e0
Process taper (pid: 2892, threadinfo ffff81017a246000, task ffff81017e63a140)
Stack: 010000000000000c ffffffff8028c534 0000000000008000 0000000000008000
ffff8100f6f9e000 ffff810004840200 0000000000008000 0000000000000040
0000000000008000 ffffffff8028826d
Call Trace:<ffffffff8028c534>{sgl_unmap_user_pages+124} <ffffffff8028826d>{release_buffering+27}
<ffffffff802888f9>{st_write+1670} <ffffffff8016d9bc>{vfs_write+173}
<ffffffff8016dac8>{sys_write+69} <ffffffff8010d762>{system_call+126}
Code: 0f 0b 68 ae b1 36 80 c2 31 00 f0 83 42 08 ff 0f 98 c0 84 c0
RIP <ffffffff8015751c>{put_page+96} RSP <ffff81017a247e18>
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at mm/rmap.c:486
invalid operand: 0000 [2] SMP
CPU 1
Modules linked in: bonding
Pid: 2892, comm: taper Tainted: G B 2.6.15 #1
RIP: 0010:[<ffffffff80163736>] <ffffffff80163736>{page_remove_rmap+19}
RSP: 0018:ffff81017a247aa0 EFLAGS: 00010286
RAX: 00000000ffffffff RBX: ffff81000427ff60 RCX: 0000000000000020
RDX: 80000000e6db4067 RSI: 80000000e6db4067 RDI: ffff81000427ff60
RBP: 80000000e6db4067 R08: 0000000000000020 R09: 00002aaaaaafe000
R10: 00000000000e6db4 R11: 0000000000000000 R12: ffff810101c25480
R13: ffff8101722a07f0 R14: 00002aaaaaafe000 R15: 0000000000000000
FS: 00002aaaab53d880(0000) GS:ffffffff804a9880(0000) knlGS:00000000556b6920
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002aaaaab5ffff CR3: 00000001722be000 CR4: 00000000000006e0
Process taper (pid: 2892, threadinfo ffff81017a246000, task ffff81017e63a140)
Stack: ffffffff8015be23 ffff81017a247ae8 ffff81010190d310 ffffffc100000000
ffff81017f078180 ffff81017a247bb8 00002aaaaab62000 ffff81017d039368
00002aaaaab62000 ffff810172296aa8
Call Trace:<ffffffff8015be23>{zap_pte_range+464} <ffffffff8015c0ff>{unmap_page_range+476}
<ffffffff8015c24e>{unmap_vmas+238} <ffffffff8016167b>{exit_mmap+114}
<ffffffff8012d638>{mmput+34} <ffffffff8013214a>{do_exit+489}
<ffffffff8010f203>{die_nmi+0} <ffffffff8010f587>{do_invalid_op+145}
<ffffffff8015751c>{put_page+96} <ffffffff803448db>{thread_return+0}
<ffffffff8010e49d>{error_exit+0} <ffffffff8015751c>{put_page+96}
<ffffffff8028c534>{sgl_unmap_user_pages+124} <ffffffff8028826d>{release_buffering+27}
<ffffffff802888f9>{st_write+1670} <ffffffff8016d9bc>{vfs_write+173}
<ffffffff8016dac8>{sys_write+69} <ffffffff8010d762>{system_call+126}
Code: 0f 0b 68 b0 b2 36 80 c2 e6 01 48 83 ce ff bf 20 00 00 00 e9
RIP <ffffffff80163736>{page_remove_rmap+19} RSP <ffff81017a247aa0>
<1>Fixing recursive fault but reboot is needed!
Unable to handle kernel paging request at 0000000000100108 RIP:
<ffffffff80150d53>{buffered_rmqueue+120}
PGD 170d24067 PUD 1551d3067 PMD 0
Oops: 0002 [3] SMP
CPU 1
Modules linked in: bonding
Pid: 2898, comm: dumper Tainted: G B 2.6.15 #1
RIP: 0010:[<ffffffff80150d53>] <ffffffff80150d53>{buffered_rmqueue+120}
RSP: 0018:ffff8100bcc17a98 EFLAGS: 00010002
RAX: ffff810003ed92b8 RBX: ffff81010287a340 RCX: 0000000000200200
RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffff81000000f300
RBP: ffff81000000f300 R08: 0000000000000000 R09: 000000000000095b
R10: 0000000000000000 R11: 0000000000000002 R12: 0000000000000000
R13: 0000000000000000 R14: ffff810003ed9290 R15: 00000000000200d2
FS: 00002aaaab53d8e0(0000) GS:ffffffff804a9880(0000) knlGS:00000000556b6920
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 000000017d27f000 CR4: 00000000000006e0
Process dumper (pid: 2898, threadinfo ffff8100bcc16000, task ffff8100bd0d4140)
Stack: ffff81011ae74810 000000001ae74810 0000000000000282 ffff810100001c18
0000000000000044 0000000000000000 0000000000000000 ffff810100001c10
0000000000000002 ffffffff80150fe3
Call Trace:<ffffffff80150fe3>{get_page_from_freelist+130} <ffffffff80151067>{__alloc_pages+86}
<ffffffff8014e7f9>{generic_file_buffered_write+402}
<ffffffff80134113>{current_fs_time+97} <ffffffff8014ef95>{__generic_file_aio_write_nolock+891}
<ffffffff802ea7fb>{sock_common_recvmsg+45} <ffffffff802e7724>{sock_aio_read+252}
<ffffffff8014f1ff>{generic_file_aio_write+105} <ffffffff801a2979>{ext3_file_write+22}
<ffffffff8016d8d5>{do_sync_write+202} <ffffffff8017e9e8>{__pollwait+0}
<ffffffff80142cca>{autoremove_wake_function+0} <ffffffff8017f22a>{sys_select+952}
<ffffffff8016d9bc>{vfs_write+173} <ffffffff8016dac8>{sys_write+69}
<ffffffff8010d762>{system_call+126}
Code: 48 89 4a 08 48 89 11 48 c7 40 08 00 02 20 00 48 c7 00 00 01
RIP <ffffffff80150d53>{buffered_rmqueue+120} RSP <ffff8100bcc17a98>
CR2: 0000000000100108
<1>Unable to handle kernel paging request at 0000000000100108 RIP:
<ffffffff80150d53>{buffered_rmqueue+120}
PGD 170d24067 PUD 1551d3067 PMD 0
Oops: 0002 [4] SMP
CPU 1
Modules linked in: bonding
Pid: 2898, comm: dumper Tainted: G B 2.6.15 #1
RIP: 0010:[<ffffffff80150d53>] <ffffffff80150d53>{buffered_rmqueue+120}
RSP: 0018:ffff81010289bca8 EFLAGS: 00010002
RAX: ffff810003ed92b8 RBX: ffff81010287a340 RCX: 0000000000200200
RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffff81000000f300
RBP: ffff81000000f300 R08: 0000000000000000 R09: 000000000000095b
R10: 0000000000000000 R11: 0000000000000002 R12: 0000000000000000
R13: 0000000000000000 R14: ffff810003ed9290 R15: 0000000000020020
FS: 00002aaaab53d8e0(0000) GS:ffffffff804a9880(0000) knlGS:00000000556b6920
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 000000017d27f000 CR4: 00000000000006e0
Process dumper (pid: 2898, threadinfo ffff8100bcc16000, task ffff8100bd0d4140)
Stack: ffff8100e78d55f8 0000000000000000 0000000000000082 ffff810100000c08
0000000000000044 0000000000000000 0000000000000000 ffff810100000c00
0000000000000002 ffffffff80150fe3
Call Trace: <IRQ> <ffffffff80150fe3>{get_page_from_freelist+130}
<ffffffff80151067>{__alloc_pages+86} <ffffffff801545b1>{kmem_getpages+88}
<ffffffff80155789>{cache_grow+195} <ffffffff801559d0>{cache_alloc_refill+408}
<ffffffff80155feb>{__kmalloc+100} <ffffffff802eb11b>{__alloc_skb+83}
<ffffffff802663a6>{tg3_alloc_rx_skb+186} <ffffffff80266630>{tg3_rx+338}
<ffffffff80266998>{tg3_poll+135} <ffffffff802f0a85>{net_rx_action+129}
<ffffffff801342b8>{__do_softirq+80} <ffffffff8010eae3>{call_softirq+31}
<ffffffff80110384>{do_softirq+47} <ffffffff8010e34a>{apic_timer_interrupt+98}
<EOI> <ffffffff8034566b>{__down_read+147} <ffffffff803455ea>{__down_read+18}
<ffffffff8012d6f3>{mm_release+30} <ffffffff801316a0>{exit_mm+43}
<ffffffff8013214a>{do_exit+489} <ffffffff8011d3c6>{do_page_fault+1215}
<ffffffff801affef>{do_get_write_access+1277} <ffffffff8010e49d>{error_exit+0}
<ffffffff80150d53>{buffered_rmqueue+120} <ffffffff80150fe3>{get_page_from_freelist+130}
<ffffffff80151067>{__alloc_pages+86} <ffffffff8014e7f9>{generic_file_buffered_write+402}
<ffffffff80134113>{current_fs_time+97} <ffffffff8014ef95>{__generic_file_aio_write_nolock+891}
<ffffffff802ea7fb>{sock_common_recvmsg+45} <ffffffff802e7724>{sock_aio_read+252}
<ffffffff8014f1ff>{generic_file_aio_write+105} <ffffffff801a2979>{ext3_file_write+22}
<ffffffff8016d8d5>{do_sync_write+202} <ffffffff8017e9e8>{__pollwait+0}
<ffffffff80142cca>{autoremove_wake_function+0} <ffffffff8017f22a>{sys_select+952}
<ffffffff8016d9bc>{vfs_write+173} <ffffffff8016dac8>{sys_write+69}
<ffffffff8010d762>{system_call+126}
Code: 48 89 4a 08 48 89 11 48 c7 40 08 00 02 20 00 48 c7 00 00 01
RIP <ffffffff80150d53>{buffered_rmqueue+120} RSP <ffff81010289bca8>
CR2: 0000000000100108
<0>Kernel panic - not syncing: Aiee, killing interrupt handler!
-ryan
next prev parent reply other threads:[~2006-01-04 17:27 UTC|newest]
Thread overview: 99+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20051129092432.0f5742f0.akpm@osdl.org>
2005-11-29 18:34 ` Fw: crash on x86_64 - mm related? Ryan Richter
[not found] ` <Pine.LNX.4.63.0511292147120.5739@kai.makisara.local>
2005-11-29 20:31 ` Ryan Richter
2005-11-29 20:48 ` Kai Makisara
2005-11-29 20:58 ` Ryan Richter
2005-11-29 21:36 ` Kai Makisara
2005-11-30 5:12 ` Kai Makisara
2005-12-01 19:18 ` Kai Makisara
2005-12-01 19:38 ` Linus Torvalds
2005-12-01 19:56 ` Ryan Richter
2005-12-01 20:21 ` Hugh Dickins
2005-12-01 21:44 ` Kai Makisara
2005-12-02 18:03 ` Ryan Richter
2005-12-02 18:43 ` Jesper Juhl
2005-12-02 19:12 ` Hugh Dickins
2005-12-02 19:44 ` Ryan Richter
2005-12-02 20:40 ` Hugh Dickins
2005-12-03 17:29 ` Ryan Richter
2005-12-06 16:08 ` Ryan Richter
2005-12-06 20:31 ` Hugh Dickins
2005-12-06 20:43 ` Ryan Richter
2005-12-07 18:37 ` Hugh Dickins
2005-12-08 2:26 ` Ryan Richter
2005-12-12 16:54 ` Ryan Richter
2005-12-12 17:40 ` Linus Torvalds
2005-12-12 17:45 ` James Bottomley
2005-12-12 18:04 ` Ryan Richter
2005-12-12 18:09 ` Linus Torvalds
2005-12-12 18:24 ` James Bottomley
2005-12-15 19:09 ` Ryan Richter
2005-12-16 4:01 ` James Bottomley
2005-12-17 3:31 ` Ryan Richter
2005-12-26 23:42 ` Ryan Richter
2005-12-27 16:21 ` Kai Makisara
2006-01-03 19:03 ` Ryan Richter
2006-01-04 17:27 ` Ryan Richter [this message]
2006-01-04 21:48 ` Kai Makisara
2006-01-05 5:40 ` Ryan Richter
2006-01-05 20:12 ` Ryan Richter
2006-01-05 21:18 ` Linus Torvalds
2006-01-08 22:36 ` Ryan Richter
2006-01-09 3:31 ` Ryan Richter
2006-01-09 4:07 ` Linus Torvalds
2006-01-09 5:13 ` Andrew Morton
2006-01-09 5:45 ` Ryan Richter
2006-01-09 5:57 ` Andrew Morton
2006-01-09 9:44 ` Hugh Dickins
2006-01-09 18:53 ` Ryan Richter
2006-01-09 19:31 ` Hugh Dickins
2006-01-09 20:05 ` Ryan Richter
2006-01-18 0:12 ` Ryan Richter
2006-01-18 16:00 ` Hugh Dickins
2006-02-03 19:46 ` Hugh Dickins
2006-02-03 19:51 ` [PATCH] ib: don't doublefree pages from scatterlist Hugh Dickins
2006-02-03 23:13 ` Roland Dreier
2006-02-03 19:53 ` [PATCH] st: " Hugh Dickins
2006-02-03 20:38 ` Mike Christie
2006-02-03 21:16 ` Hugh Dickins
2006-02-04 12:10 ` Kai Makisara
2006-02-04 15:01 ` Hugh Dickins
2006-02-03 19:55 ` [PATCH] ipr: " Hugh Dickins
2006-02-03 22:06 ` Brian King
2006-02-04 0:26 ` Hugh Dickins
2006-02-05 21:35 ` Brian King
2006-02-06 9:32 ` Hugh Dickins
2006-02-06 9:46 ` David S. Miller
2006-02-06 14:46 ` Brian King
2006-02-06 16:45 ` Hugh Dickins
2006-02-06 17:38 ` James Bottomley
2006-02-06 19:15 ` Brian King
2006-02-06 21:11 ` Andi Kleen
2006-02-06 21:49 ` David S. Miller
2006-02-06 22:11 ` Hugh Dickins
2006-02-06 22:13 ` Andi Kleen
2006-02-07 3:09 ` Ryan Richter
2006-02-11 22:38 ` Ryan Richter
2006-02-12 18:57 ` Hugh Dickins
2006-02-12 21:29 ` Andi Kleen
2006-02-13 17:21 ` Hugh Dickins
2006-02-06 15:02 ` James Bottomley
2006-02-06 17:01 ` Hugh Dickins
2006-02-03 19:56 ` [PATCH] osst: " Hugh Dickins
2006-02-03 21:10 ` Fw: crash on x86_64 - mm related? Ryan Richter
2006-02-04 11:58 ` Kai Makisara
2006-02-04 14:46 ` Hugh Dickins
2006-01-05 22:09 ` Kai Makisara
2006-01-04 18:26 ` Ryan Richter
2005-12-07 18:30 ` Ryan Richter
2005-12-07 18:56 ` Hugh Dickins
2005-12-07 19:06 ` Ryan Richter
2005-12-06 17:57 ` Ryan Richter
2005-12-01 20:28 ` James Bottomley
2005-12-01 21:17 ` Kai Makisara
2005-12-02 13:45 ` Hugh Dickins
2005-12-02 17:59 ` Kai Makisara
2005-12-02 18:55 ` Hugh Dickins
2005-12-02 19:46 ` Kai Makisara
2005-12-02 20:47 ` Hugh Dickins
2005-12-04 9:29 ` Kai Makisara
2005-12-01 19:53 ` Ryan Richter
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20060104172727.GA320@tau.solarneutrino.net \
--to=ryan@tau.solarneutrino.net \
--cc=James.Bottomley@SteelEye.com \
--cc=Kai.Makisara@kolumbus.fi \
--cc=akpm@osdl.org \
--cc=hugh@veritas.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=torvalds@osdl.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).