Re: Fw: crash on x86_64 - mm related?

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Tue, Nov 29, 2005 at 09:24:32AM -0800, Ryan Richter wrote:

Not sure if this matters, but this apparently happened in two stages.
This first part happened during the backups, as I said earlier:

>  Bad page state at free_hot_cold_page (in process 'taper', page ffff81000260b6f8)
> flags:0x010000000000000c mapping:ffff8100355f1dd8 mapcount:2 count:0
> Backtrace:
> 
> Call Trace:<ffffffff80159f93>{bad_page+99} <ffffffff8015a965>{free_hot_cold_page+101}
>        <ffffffff80162007>{__page_cache_release+151} <ffffffff802b8fe8>{sgl_unmap_user_pages+120}
>        <ffffffff802b48fb>{release_buffering+27} <ffffffff802b4fb1>{st_write+1697}
>        <ffffffff8017af46>{vfs_write+198} <ffffffff8017b0a3>{sys_write+83}
>        <ffffffff8010db7a>{system_call+126} 
> Trying to fix it up, but a reboot is needed
> Bad page state at free_hot_cold_page (in process 'taper', page ffff81000260b6f8)
> flags:0x010000000000081c mapping:ffff81005c0fc310 mapcount:0 count:0
> Backtrace:
> 
> Call Trace:<ffffffff80159f93>{bad_page+99} <ffffffff8015a965>{free_hot_cold_page+101}
>        <ffffffff80162007>{__page_cache_release+151} <ffffffff802b8fe8>{sgl_unmap
> _user_pages+120}
>        <ffffffff802b48fb>{release_buffering+27} <ffffffff802b4fb1>{st_write+1697}
>        <ffffffff8017af46>{vfs_write+198} <ffffffff8017b0a3>{sys_write+83}
>        <ffffffff8010db7a>{system_call+126} 
> Trying to fix it up, but a reboot is needed
> ----------- [cut here ] --------- [please bite here ] ---------
> Kernel BUG at include/linux/mm.h:341
> invalid operand: 0000 [1] SMP 
> CPU 1 
> Modules linked in: bonding
> Pid: 2418, comm: taper Tainted: G    B 2.6.14.2 #1
> RIP: 0010:[<ffffffff802b8fcd>] <ffffffff802b8fcd>{sgl_unmap_user_pages+93}
> RSP: 0018:ffff810035725e18  EFLAGS: 00010256
> RAX: 0000000000000000 RBX: 0000000000000007 RCX: 000000000000000f
> RDX: 00000000000000e0 RSI: 0000000000000001 RDI: ffff81000260b6f8
> RBP: ffff810004852068 R08: 00000000ffffffff R09: 0000000000000000
> R10: 0000000000008000 R11: 0000000000000200 R12: 0000000000000008
> R13: 0000000000000000 R14: 0000000000008000 R15: ffff810004949d10
> FS:  00002aaaab53d880(0000) GS:ffffffff804db880(0000) knlGS:00000000556b6920
> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 00002aaaaaac0000 CR3: 0000000035691000 CR4: 00000000000006e0
> Process taper (pid: 2418, threadinfo ffff810035724000, task ffff81017d680300)
> Stack: ffff8101423f3600 ffff810004852000 0000000000000040 0000000000008000 
>        ffff810004949c00 ffffffff802b48fb ffff810004852000 ffffffff802b4fb1 
>        ffff810000000000 ffffffff00000001 
> Call Trace:<ffffffff802b48fb>{release_buffering+27} <ffffffff802b4fb1>{st_write+1697}
>        <ffffffff8017af46>{vfs_write+198} <ffffffff8017b0a3>{sys_write+83}
>        <ffffffff8010db7a>{system_call+126} 
> 
> Code: 0f 0b 68 ba 12 3a 80 c2 55 01 f0 83 47 08 ff 0f 98 c0 84 c0 
> RIP <ffffffff802b8fcd>{sgl_unmap_user_pages+93} RSP <ffff810035725e18>
>  ----------- [cut here ] --------- [please bite here ] ---------
> Kernel BUG at mm/rmap.c:487
> invalid operand: 0000 [2] SMP 
> CPU 1 
> Modules linked in: bonding
> Pid: 2418, comm: taper Tainted: G    B 2.6.14.2 #1
> RIP: 0010:[<ffffffff8016f3f7>] <ffffffff8016f3f7>{page_remove_rmap+39}
> RSP: 0018:ffff810035725ab0  EFLAGS: 00010286
> RAX: 00000000ffffffff RBX: ffff8100356976f8 RCX: ffff81000000f000
> RDX: 0000000000000000 RSI: 8000000064c69067 RDI: ffff81000260b6f8
> RBP: 00002aaaaaadf000 R08: 0000000000000000 R09: ffff81000260b688
> R10: 00000000fffffffa R11: 0000000000000000 R12: ffff810101c22380
> R13: 8000000064c69067 R14: ffff81000260b6f8 R15: 0000000000000000
> FS:  00002aaaab53d880(0000) GS:ffffffff804db880(0000) knlGS:00000000556b6920
> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 00002aaaaaac0000 CR3: 0000000035691000 CR4: 00000000000006e0
> Process taper (pid: 2418, threadinfo ffff810035724000, task ffff81017d680300)
> Stack: ffffffff80166ecd 00002aaaaab62000 ffff810035696aa8 00002aaaaab62000 
>        00002aaaaab62000 00002aaaaab61fff ffff810035695550 00002aaaaab62000 
>        ffffffff80167180 ffff810035725d68 
> Call Trace:<ffffffff80166ecd>{zap_pte_range+477} <ffffffff80167180>{unmap_page_range+496}
>        <ffffffff801672e5>{unmap_vmas+293} <ffffffff8016cfa2>{exit_mmap+162}
>        <ffffffff80131ce1>{mmput+49} <ffffffff801371c6>{do_exit+438}
>        <ffffffff8010f6f1>{die+81} <ffffffff8010f9df>{do_invalid_op+159}
>        <ffffffff802b8fcd>{sgl_unmap_user_pages+93} <ffffffff80381f76>{thread_return+86}
>        <ffffffff802a8662>{sym_setup_data_and_start+402} <ffffffff8010e84d>{error_exit+0}
>        <ffffffff802b8fcd>{sgl_unmap_user_pages+93} <ffffffff802b8fe8>{sgl_unmap_user_pages+120}
>        <ffffffff802b48fb>{release_buffering+27} <ffffffff802b4fb1>{st_write+1697}
>        <ffffffff8017af46>{vfs_write+198} <ffffffff8017b0a3>{sys_write+83}
>        <ffffffff8010db7a>{system_call+126} 
> 
> Code: 0f 0b 68 9b 35 3a 80 c2 e7 01 48 c7 c6 ff ff ff ff bf 20 00 
> RIP <ffffffff8016f3f7>{page_remove_rmap+39} RSP <ffff810035725ab0>
>  <1>Fixing recursive fault but reboot is needed!
> Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP: 
> <ffffffff801b9c7b>{ext3_prepare_write+27}
> PGD 355bc067 PUD 355c9067 PMD 0 
> Oops: 0000 [3] SMP 
> CPU 0 
> Modules linked in: bonding
> Pid: 2416, comm: driver Tainted: G    B 2.6.14.2 #1
> RIP: 0010:[<ffffffff801b9c7b>] <ffffffff801b9c7b>{ext3_prepare_write+27}
> RSP: 0018:ffff8100355e7b48  EFLAGS: 00010296
> RAX: 0000000000000000 RBX: ffffffff8040f660 RCX: 000000000000017d
> RDX: 0000000000000094 RSI: ffff81000260b6f8 RDI: ffff810035b09cc0
> RBP: 000000000000000e R08: 00000000fffffffa R09: 00000000000000e9
> R10: ffff81001190c818 R11: 0000000000000000 R12: ffff81000260b6f8
> R13: ffff81000260b6f8 R14: 000000000000017d R15: 0000000000000094
> FS:  00002aaaab53d8e0(0000) GS:ffffffff804db800(0000) knlGS:00000000555bc920
> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000000000000000 CR3: 0000000035555000 CR4: 00000000000006e0
> Process driver (pid: 2416, threadinfo ffff8100355e6000, task ffff8100f43e8a80)
> Stack: ffff81014e643310 ffffffff8040f660 000000000000000e ffff81000260b6f8 
>        ffff81005c0fc310 0000000000000094 00000000000000e9 ffffffff80158247 
>        0000000000000292 00002aaaaaac0000 
> Call Trace:<ffffffff80158247>{generic_file_buffered_write+551}
>        <ffffffff801c06bd>{__ext3_journal_stop+45} <ffffffff8019ec74>{__mark_inode_dirty+52}
>        <ffffffff801963ec>{inode_update_time+188} <ffffffff80158a08>{__generic_file_aio_write_nolock+936}
>        <ffffffff80381f76>{thread_return+86} <ffffffff8013d349>{lock_timer_base+41}
>        <ffffffff80158cfe>{generic_file_aio_write+110} <ffffffff801b7783>{ext3_file_write+35}
>        <ffffffff8017ae43>{do_sync_write+211} <ffffffff8018ecc0>{__pollwait+0}
>        <ffffffff8014a2b0>{autoremove_wake_function+0} <ffffffff8018f681>{sys_select+1153}
>        <ffffffff8017af46>{vfs_write+198} <ffffffff8017b0a3>{sys_write+83}
>        <ffffffff8010db7a>{system_call+126} 
> 
> Code: 48 8b 28 48 89 ef e8 aa 26 00 00 c7 44 24 04 00 00 00 00 89 
> RIP <ffffffff801b9c7b>{ext3_prepare_write+27} RSP <ffff8100355e7b48>
> CR2: 0000000000000000
>  <0>Bad page state at prep_new_page (in process 'dumper', page ffff81000260b6f8)
> flags:0x010000000000001d mapping:0000000000000000 mapcount:-1 count:1
> Backtrace:
> 
> Call Trace:<ffffffff80159f93>{bad_page+99} <ffffffff8015a371>{prep_new_page+65}
>        <ffffffff8015ab2e>{buffered_rmqueue+302} <ffffffff8015ad85>{__alloc_pages+261}
>        <ffffffff801581bd>{generic_file_buffered_write+413}
>        <ffffffff80139509>{current_fs_time+105} <ffffffff8019636e>{inode_update_time+62}
>        <ffffffff80158a08>{__generic_file_aio_write_nolock+936}
>        <ffffffff8031f4a4>{sock_common_recvmsg+52} <ffffffff8031bb30>{sock_aio_read+272}
>        <ffffffff80158cfe>{generic_file_aio_write+110} <ffffffff801b7783>{ext3_file_write+35}
>        <ffffffff8017ae43>{do_sync_write+211} <ffffffff8018ecc0>{__pollwait+0}
>        <ffffffff8014a2b0>{autoremove_wake_function+0} <ffffffff8018f681>{sys_select+1153}
>        <ffffffff8017af46>{vfs_write+198} <ffffffff8017b0a3>{sys_write+83}
>        <ffffffff8010db7a>{system_call+126} 
> Trying to fix it up, but a reboot is needed

Everything from here on happened several hours later while updatedb was
running.

> Bad page state at prep_new_page (in process 'find', page ffff81000260b6f8)
> flags:0x0100000000000064 mapping:ffff8100f3be9be9 mapcount:1 count:1
> Backtrace:
> 
> Call Trace:<ffffffff80159f93>{bad_page+99} <ffffffff8015a371>{prep_new_page+65}
>        <ffffffff8015ab2e>{buffered_rmqueue+302} <ffffffff8015ad85>{__alloc_pages+261}
>        <ffffffff8015e7a3>{kmem_getpages+99} <ffffffff8015fbb0>{cache_grow+192}
>        <ffffffff8015fe3b>{cache_alloc_refill+459} <ffffffff80160226>{kmem_cache_alloc+54}
>        <ffffffff80193831>{d_alloc+33} <ffffffff80188fe9>{real_lookup+105}
>        <ffffffff801893c0>{do_lookup+112} <ffffffff80189e07>{__link_path_walk+2551}
>        <ffffffff8018a382>{link_path_walk+178} <ffffffff8018a8ce>{path_lookup+446}
>        <ffffffff8018aa9e>{__user_walk+62} <ffffffff801849b6>{vfs_lstat+38}
>        <ffffffff80184dff>{sys_newlstat+31} <ffffffff8010db7a>{system_call+126}
>        
> Trying to fix it up, but a reboot is needed
> Unable to handle kernel paging request at 00002aaaab9c5b61 RIP: 
> <ffffffff8015fdba>{cache_alloc_refill+330}
> PGD c2512067 PUD c2513067 PMD 0 
> Oops: 0002 [4] SMP 
> CPU 0 
> Modules linked in: bonding
> Pid: 3011, comm: find Tainted: G    B 2.6.14.2 #1
> RIP: 0010:[<ffffffff8015fdba>] <ffffffff8015fdba>{cache_alloc_refill+330}
> RSP: 0018:ffff810112f05c28  EFLAGS: 00010082
> RAX: 00002aaaab9c5b59 RBX: 0000000000000010 RCX: 0000000000029ba6
> RDX: 00002aaaab9c5bb3 RSI: ffff810064c69040 RDI: ffff81000c01a288
> RBP: ffff8100f6fc4800 R08: ffff81000c01a250 R09: ffff81000c01a260
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff81000c01a240
> R13: ffff8100f6fc3640 R14: ffff81000c01a288 R15: 00000000000000d0
> FS:  00002aaaaae00640(0000) GS:ffffffff804db800(0000) knlGS:00000000555bc920
> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 00002aaaab9c5b61 CR3: 00000000c2bb3000 CR4: 00000000000006e0
> Process find (pid: 3011, threadinfo ffff810112f04000, task ffff810102b46040)
> Stack: ffff810112f05e68 ffff810179923cb8 fffffffffffffff4 ffff810112f05d28 
>        ffff810179923cb8 ffff810112f05d28 ffff810112f05e68 ffffffff80160226 
>        0000000000000292 ffffffff80193831 
> Call Trace:<ffffffff80160226>{kmem_cache_alloc+54} <ffffffff80193831>{d_alloc+33}
>        <ffffffff80188fe9>{real_lookup+105} <ffffffff801893c0>{do_lookup+112}
>        <ffffffff80189e07>{__link_path_walk+2551} <ffffffff8018a382>{link_path_walk+178}
>        <ffffffff8018a8ce>{path_lookup+446} <ffffffff8018aa9e>{__user_walk+62}
>        <ffffffff801849b6>{vfs_lstat+38} <ffffffff80184dff>{sys_newlstat+31}
>        <ffffffff8010db7a>{system_call+126} 
> 
> Code: 48 89 50 08 48 89 02 48 c7 46 08 00 02 20 00 83 7e 24 ff 48 
> RIP <ffffffff8015fdba>{cache_alloc_refill+330} RSP <ffff810112f05c28>
> CR2: 00002aaaab9c5b61
>  NMI Watchdog detected LOCKUP on CPU 1
> CPU 1 
> Modules linked in: bonding
> Pid: 7, comm: events/1 Tainted: G    B 2.6.14.2 #1
> RIP: 0010:[<ffffffff803837dd>] <ffffffff803837dd>{.text.lock.spinlock+118}
> RSP: 0018:ffff810004869dd0  EFLAGS: 00000086
> RAX: ffff81000c01a240 RBX: ffff81000c01a288 RCX: ffff8100f6fc3640
> RDX: 0000000000000003 RSI: 0000000000000003 RDI: ffff81000c01a288
> RBP: ffff810100009dc0 R08: 0000000000000000 R09: 0000000000000000
> R10: 00000000ffffffff R11: 0000000000000066 R12: 0000000000000000
> R13: ffff810100009dd0 R14: 0000000000000292 R15: ffff810100009e40
> FS:  00002aaaaae00640(0000) GS:ffffffff804db880(0000) knlGS:00000000556b6920
> CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> CR2: 00002aaaaaf1df40 CR3: 000000017f448000 CR4: 00000000000006e0
> Process events/1 (pid: 7, threadinfo ffff810004868000, task ffff8100f6fb6080)
> Stack: ffffffff8015e35b ffff8100f6fc3640 ffff810100009f60 0000000000000001 
>        ffff810100009e40 ffff8100f6fc3640 ffff8100f6fc38e0 ffff810100009f88 
>        ffffffff80161414 ffff810004869e58 
> Call Trace:<ffffffff8015e35b>{drain_alien_cache+123} <ffffffff80161414>{cache_reap+164}
>        <ffffffff80161370>{cache_reap+0} <ffffffff8014553c>{worker_thread+476}
>        <ffffffff8012ed70>{default_wake_function+0} <ffffffff8012ed70>{default_wake_function+0}
>        <ffffffff80145360>{worker_thread+0} <ffffffff80149c82>{kthread+146}
>        <ffffffff8010ea02>{child_rip+8} <ffffffff80145360>{worker_thread+0}
>        <ffffffff80149bf0>{kthread+0} <ffffffff8010e9fa>{child_rip+0}
>        
> 
> Code: 80 3f 00 7e f9 e9 59 fe ff ff e8 58 41 e9 ff e9 6f fe ff ff 
> console shuts up ...
>  <0>Kernel panic - not syncing: Aiee, killing interrupt handler!

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Tue, Nov 29, 2005 at 10:04:39PM +0200, Kai Makisara wrote:
> I looked at the driver and it seems that there is a bug: st_write calls 
> release_buffering at the end even when it has started an asynchronous 
> write. This means that it releases the mapping while it is being used!
> (I wonder why this has not been noticed earlier.)
> 
> The patch below (against 2.6.15-rc2) should fix this bug and some others 
> related to buffering. It is based on the patch "[PATCH] SCSI tape direct 
> i/o fixes" I sent to linux-scsi on Nov 21. The patch restores setting 
> pages dirty after reading and clears number of s/g segments when the 
> pointers are not valid any more.
> 
> The patch has been lightly tested with AMD64.

This applies cleanly to 2.6.14.2, do you forsee any problems using it
with that kernel?  I'd like to not change too many things at once.

If it should be OK, I'll boot this tonight or tomorrow - the backups run
every other night, so it won't get any testing until tomorrow night.

Thanks a lot,
-ryan

Re: Fw: crash on x86_64 - mm related?

[Kai Makisara]

On Tue, 29 Nov 2005, Ryan Richter wrote:

> On Tue, Nov 29, 2005 at 10:04:39PM +0200, Kai Makisara wrote:
> > I looked at the driver and it seems that there is a bug: st_write calls 
> > release_buffering at the end even when it has started an asynchronous 
> > write. This means that it releases the mapping while it is being used!
> > (I wonder why this has not been noticed earlier.)
> > 
> > The patch below (against 2.6.15-rc2) should fix this bug and some others 
> > related to buffering. It is based on the patch "[PATCH] SCSI tape direct 
> > i/o fixes" I sent to linux-scsi on Nov 21. The patch restores setting 
> > pages dirty after reading and clears number of s/g segments when the 
> > pointers are not valid any more.
> > 
> > The patch has been lightly tested with AMD64.
> 
> This applies cleanly to 2.6.14.2, do you forsee any problems using it
> with that kernel?  I'd like to not change too many things at once.
> 
No, I don't see any potential problems applying this patch to 2.6.14.2. 
There is nothing specific to 2.6.15-rc2.

If someone sees that there is something wrong, please yell. The 
main purpose of the patch is not to call release_buffering() at the end of 
st_write() when starting asynchronous write and call it in 
write_behind_check() instead.

> If it should be OK, I'll boot this tonight or tomorrow - the backups run
> every other night, so it won't get any testing until tomorrow night.
> 
> Thanks a lot,
> -ryan
> 
Thanks for reporting the problem and thanks in advance for testing.

-- 
Kai

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Tue, Nov 29, 2005 at 10:48:22PM +0200, Kai Makisara wrote:
> On Tue, 29 Nov 2005, Ryan Richter wrote:
> > This applies cleanly to 2.6.14.2, do you forsee any problems using it
> > with that kernel?  I'd like to not change too many things at once.
> > 
> No, I don't see any potential problems applying this patch to 2.6.14.2. 
> There is nothing specific to 2.6.15-rc2.
> 
> If someone sees that there is something wrong, please yell. The 
> main purpose of the patch is not to call release_buffering() at the end of 
> st_write() when starting asynchronous write and call it in 
> write_behind_check() instead.

OK, thanks.  I think I'll go ahead and advance to 2.6.14.3 since that
should theoretically not cause any problems.

One question: do you think the oopses that happened later that actually
crashed the box were from damage caused by this bug or is that a
different problem?

> > If it should be OK, I'll boot this tonight or tomorrow - the backups run
> > every other night, so it won't get any testing until tomorrow night.
> > 
> > Thanks a lot,
> > -ryan
> > 
> Thanks for reporting the problem and thanks in advance for testing.

Sure thing,
-ryan

Re: Fw: crash on x86_64 - mm related?

[Kai Makisara]

On Tue, 29 Nov 2005, Ryan Richter wrote:

> On Tue, Nov 29, 2005 at 10:48:22PM +0200, Kai Makisara wrote:
> > On Tue, 29 Nov 2005, Ryan Richter wrote:
> > > This applies cleanly to 2.6.14.2, do you forsee any problems using it
> > > with that kernel?  I'd like to not change too many things at once.
> > > 
> > No, I don't see any potential problems applying this patch to 2.6.14.2. 
> > There is nothing specific to 2.6.15-rc2.
> > 
> > If someone sees that there is something wrong, please yell. The 
> > main purpose of the patch is not to call release_buffering() at the end of 
> > st_write() when starting asynchronous write and call it in 
> > write_behind_check() instead.
> 
> OK, thanks.  I think I'll go ahead and advance to 2.6.14.3 since that
> should theoretically not cause any problems.
> 
> One question: do you think the oopses that happened later that actually
> crashed the box were from damage caused by this bug or is that a
> different problem?
> 
I looked at the oopses but, not knowing enough about what is happening 
inside the kernel, I can only hope that they are caused by the st bug(s). 
We will see after testing with the patch.

-- 
Kai

Re: Fw: crash on x86_64 - mm related?

[Kai Makisara]

On Tue, 29 Nov 2005, Kai Makisara wrote:

> On Tue, 29 Nov 2005, Ryan Richter wrote:
> 
> > On Tue, Nov 29, 2005 at 10:04:39PM +0200, Kai Makisara wrote:
> > > I looked at the driver and it seems that there is a bug: st_write calls 
> > > release_buffering at the end even when it has started an asynchronous 
> > > write. This means that it releases the mapping while it is being used!
> > > (I wonder why this has not been noticed earlier.)
> > > 
> > > The patch below (against 2.6.15-rc2) should fix this bug and some others 
> > > related to buffering. It is based on the patch "[PATCH] SCSI tape direct 
> > > i/o fixes" I sent to linux-scsi on Nov 21. The patch restores setting 
> > > pages dirty after reading and clears number of s/g segments when the 
> > > pointers are not valid any more.
> > > 
> > > The patch has been lightly tested with AMD64.
> > 
> > This applies cleanly to 2.6.14.2, do you forsee any problems using it
> > with that kernel?  I'd like to not change too many things at once.
> > 
> No, I don't see any potential problems applying this patch to 2.6.14.2. 
> There is nothing specific to 2.6.15-rc2.
> 
> If someone sees that there is something wrong, please yell. The 
> main purpose of the patch is not to call release_buffering() at the end of 
> st_write() when starting asynchronous write and call it in 
> write_behind_check() instead.
> 
Yelling!

The patch does not cause any problems but my theory about calling 
release_buffering() too early is wrong: asynchronous writes are not done 
with direct i/o. (The reason for this choice is that the API allows the 
user to do anything with the buffer between writes and so the SCSI write 
has to be finished before returning from write().)

The other fixes in the patch are valid but I don't see how they should fix 
this problem.

-- 
Kai

Re: Fw: crash on x86_64 - mm related?

[Kai Makisara]

On Tue, 29 Nov 2005, Andrew Morton wrote:

> 
> 
> Begin forwarded message:
> 
> Date: Tue, 29 Nov 2005 10:44:09 -0500
> From: Ryan Richter <ryan@tau.solarneutrino.net>
> To: linux-kernel@vger.kernel.org
> Cc: ryan@tau.solarneutrino.net
> Subject: crash on x86_64 - mm related?
> 
> 
> Hi, I booted 2.6.14.2 with the MPT fusion performance fix patch about a
> week ago on my file server.  The machine crashed lat night while it was
> doing backups.  You can see the voluminous kernel output below.
> 
> Someone else recently had seemingly the same thing happen, but didn't
> think it was a kernel problem.  You can read about it here:
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=338335
> 
> I will reply later today with the kernel .config, right now I have to
> wait for someone to reboot the machine first.
> 
> Any help would be appreciated,
> -ryan
> 
>  Bad page state at free_hot_cold_page (in process 'taper', page ffff81000260b6f8)
> flags:0x010000000000000c mapping:ffff8100355f1dd8 mapcount:2 count:0
> Backtrace:
> 
> Call Trace:<ffffffff80159f93>{bad_page+99} <ffffffff8015a965>{free_hot_cold_page+101}
>        <ffffffff80162007>{__page_cache_release+151} <ffffffff802b8fe8>{sgl_unmap_user_pages+120}
>        <ffffffff802b48fb>{release_buffering+27} <ffffffff802b4fb1>{st_write+1697}
>        <ffffffff8017af46>{vfs_write+198} <ffffffff8017b0a3>{sys_write+83}
>        <ffffffff8010db7a>{system_call+126} 
> Trying to fix it up, but a reboot is needed
> Bad page state at free_hot_cold_page (in process 'taper', page ffff81000260b6f8)
> flags:0x010000000000081c mapping:ffff81005c0fc310 mapcount:0 count:0
> Backtrace:
> 
> Call Trace:<ffffffff80159f93>{bad_page+99} <ffffffff8015a965>{free_hot_cold_page+101}
>        <ffffffff80162007>{__page_cache_release+151} <ffffffff802b8fe8>{sgl_unmap
> _user_pages+120}
>        <ffffffff802b48fb>{release_buffering+27} <ffffffff802b4fb1>{st_write+1697}
>        <ffffffff8017af46>{vfs_write+198} <ffffffff8017b0a3>{sys_write+83}
>        <ffffffff8010db7a>{system_call+126} 
> Trying to fix it up, but a reboot is needed
> ----------- [cut here ] --------- [please bite here ] ---------
> Kernel BUG at include/linux/mm.h:341
> invalid operand: 0000 [1] SMP 
> CPU 1 
> Modules linked in: bonding
> Pid: 2418, comm: taper Tainted: G    B 2.6.14.2 #1
> RIP: 0010:[<ffffffff802b8fcd>] <ffffffff802b8fcd>{sgl_unmap_user_pages+93}
> RSP: 0018:ffff810035725e18  EFLAGS: 00010256
> RAX: 0000000000000000 RBX: 0000000000000007 RCX: 000000000000000f
> RDX: 00000000000000e0 RSI: 0000000000000001 RDI: ffff81000260b6f8
> RBP: ffff810004852068 R08: 00000000ffffffff R09: 0000000000000000
> R10: 0000000000008000 R11: 0000000000000200 R12: 0000000000000008
> R13: 0000000000000000 R14: 0000000000008000 R15: ffff810004949d10
> FS:  00002aaaab53d880(0000) GS:ffffffff804db880(0000) knlGS:00000000556b6920
> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 00002aaaaaac0000 CR3: 0000000035691000 CR4: 00000000000006e0
> Process taper (pid: 2418, threadinfo ffff810035724000, task ffff81017d680300)
> Stack: ffff8101423f3600 ffff810004852000 0000000000000040 0000000000008000 
>        ffff810004949c00 ffffffff802b48fb ffff810004852000 ffffffff802b4fb1 
>        ffff810000000000 ffffffff00000001 
> Call Trace:<ffffffff802b48fb>{release_buffering+27} <ffffffff802b4fb1>{st_write+1697}
>        <ffffffff8017af46>{vfs_write+198} <ffffffff8017b0a3>{sys_write+83}
>        <ffffffff8010db7a>{system_call+126} 
> 
> Code: 0f 0b 68 ba 12 3a 80 c2 55 01 f0 83 47 08 ff 0f 98 c0 84 c0 
> RIP <ffffffff802b8fcd>{sgl_unmap_user_pages+93} RSP <ffff810035725e18>
>  ----------- [cut here ] --------- [please bite here ] ---------
> Kernel BUG at mm/rmap.c:487
> invalid operand: 0000 [2] SMP 
> CPU 1 
> Modules linked in: bonding
> Pid: 2418, comm: taper Tainted: G    B 2.6.14.2 #1
> RIP: 0010:[<ffffffff8016f3f7>] <ffffffff8016f3f7>{page_remove_rmap+39}
> RSP: 0018:ffff810035725ab0  EFLAGS: 00010286
> RAX: 00000000ffffffff RBX: ffff8100356976f8 RCX: ffff81000000f000
> RDX: 0000000000000000 RSI: 8000000064c69067 RDI: ffff81000260b6f8
> RBP: 00002aaaaaadf000 R08: 0000000000000000 R09: ffff81000260b688
> R10: 00000000fffffffa R11: 0000000000000000 R12: ffff810101c22380
> R13: 8000000064c69067 R14: ffff81000260b6f8 R15: 0000000000000000
> FS:  00002aaaab53d880(0000) GS:ffffffff804db880(0000) knlGS:00000000556b6920
> CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 00002aaaaaac0000 CR3: 0000000035691000 CR4: 00000000000006e0
> Process taper (pid: 2418, threadinfo ffff810035724000, task ffff81017d680300)
> Stack: ffffffff80166ecd 00002aaaaab62000 ffff810035696aa8 00002aaaaab62000 
>        00002aaaaab62000 00002aaaaab61fff ffff810035695550 00002aaaaab62000 
>        ffffffff80167180 ffff810035725d68 
> Call Trace:<ffffffff80166ecd>{zap_pte_range+477} <ffffffff80167180>{unmap_page_range+496}
>        <ffffffff801672e5>{unmap_vmas+293} <ffffffff8016cfa2>{exit_mmap+162}
>        <ffffffff80131ce1>{mmput+49} <ffffffff801371c6>{do_exit+438}
>        <ffffffff8010f6f1>{die+81} <ffffffff8010f9df>{do_invalid_op+159}
>        <ffffffff802b8fcd>{sgl_unmap_user_pages+93} <ffffffff80381f76>{thread_return+86}
>        <ffffffff802a8662>{sym_setup_data_and_start+402} <ffffffff8010e84d>{error_exit+0}
>        <ffffffff802b8fcd>{sgl_unmap_user_pages+93} <ffffffff802b8fe8>{sgl_unmap_user_pages+120}
>        <ffffffff802b48fb>{release_buffering+27} <ffffffff802b4fb1>{st_write+1697}
>        <ffffffff8017af46>{vfs_write+198} <ffffffff8017b0a3>{sys_write+83}
>        <ffffffff8010db7a>{system_call+126} 
> 
[ Rest of the oopses cut ]

I have installed amanda and learned to use it enough to do experiments 
with my main system. Unfortunately I have not been able to see any oopses.

My system is somewhat similar to yours but not completely. I have a single 
processor system with 1 GB memory whereas your system is a dual processor 
system with 5 GB memory. We both use the sym53c8xx driver to control the 
tape drive.

I have tried 2.6.14.2 and 2.6.15-rc3 kernels with and without the patch I 
sent earlier to the list. The first kernels did not have preemption and 
NUMA support enabled but later I configured the 2.6.14.2 kernel with both 
enabled. This is the nearest thing to your NUMA dual processor system but 
it does not seem to be near enough.

Since I can't reproduce the problem, I have to look at the oopses more 
carefully. Both yout oopses and those from 
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=338335 are quite similar 
at the beginning. First come one or more reports about "Bad page state at 
free_hot_cold_page". The mapping_count is always two and count is zero. 
This condition triggers the message.

The next thing is "Kernel BUG at include/linux/mm.h:341". This is in 
put_page(struct page *page) and points to page pointer being NULL.

The third event is "Kernel BUG at mm/rmap.c:487" which results from 
"BUG_ON(page_mapcount(page) < 0)". The page pointer has been used used 
earlier in page_remove_rmap().

I am not an mm expert and have no idea what could cause this sequence of 
events. Any ideas?

If someone has any ideas for my debugging, they are welcome. I will 
continue thinking about this but now I am out of useful ideas.

-- 
Kai

Re: Fw: crash on x86_64 - mm related?

[Linus Torvalds]

On Thu, 1 Dec 2005, Kai Makisara wrote:

> On Tue, 29 Nov 2005, Andrew Morton wrote:
> >
> >  Bad page state at free_hot_cold_page (in process 'taper', page ffff81000260b6f8)
> > flags:0x010000000000000c mapping:ffff8100355f1dd8 mapcount:2 count:0
> > Backtrace:

Ryan, can you test 2.6.15-rc4 and report what it does?

The "Bad page state" messages may (should) remain, but the crashes should 
be gone and the machine should hopefully continue functioning fine. And, 
perhaps more importantly, you should hopefully have a _new_ message about 
incomplete pfn mappings that should help pinpoint which driver causes 
this..

		Linus

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Thu, Dec 01, 2005 at 09:18:33PM +0200, Kai Makisara wrote:
> I have installed amanda and learned to use it enough to do experiments 
> with my main system. Unfortunately I have not been able to see any oopses.

Thanks a lot for doing this!  Our backups run every other night and
usually write 15-20GB to tape, and it was 2 weeks before we hit this.
So it's not very easy to reproduce.  They ran again last night (with the
patch) without incident.

There's one more thing that might be interesting: the morning before
this crash happened, this machine mysteriously rebooted for no apparent
reason.  There was nothing in the logs.  Unfortunately, the BIOS screen
wipes out the last ~24 lines of serial console output on reboot, so all
I can say is that if anything was printed it was much less than 24
lines.  There are other machines on the same UPS so it wasn't a power
glitch.  And it had just had an uptime of 170-some days with 2.6.11.3
(which had some annoying bugs but was stable), so it's not exactly prone
to random reboots...

I didn't think it was worth mentioning since I knew nothing about what
happened, but I noticed that it happened exactly one hour after the
6:37am cron job (updatedb etc.) - the same one that caused it to crash
the next day after the oopses during the backups.  Might it be related?

Let me know if there's any more info I can provide.

-ryan

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Thu, Dec 01, 2005 at 11:38:20AM -0800, Linus Torvalds wrote:
> 
> 
> On Thu, 1 Dec 2005, Kai Makisara wrote:
> 
> > On Tue, 29 Nov 2005, Andrew Morton wrote:
> > >
> > >  Bad page state at free_hot_cold_page (in process 'taper', page ffff81000260b6f8)
> > > flags:0x010000000000000c mapping:ffff8100355f1dd8 mapcount:2 count:0
> > > Backtrace:
> 
> Ryan, can you test 2.6.15-rc4 and report what it does?
> 
> The "Bad page state" messages may (should) remain, but the crashes should 
> be gone and the machine should hopefully continue functioning fine. And, 
> perhaps more importantly, you should hopefully have a _new_ message about 
> incomplete pfn mappings that should help pinpoint which driver causes 
> this..

Will do, I plan to take this machine down Saturday to run memtest86 for
a while (just to be sure - 2/3 of the RAM is new, but I should be seeing
machine checks if that were the problem, no?) so I'll boot this after that.

Thanks,
-ryan

Re: Fw: crash on x86_64 - mm related?

[Hugh Dickins]

On Thu, 1 Dec 2005, Ryan Richter wrote:
> On Thu, Dec 01, 2005 at 11:38:20AM -0800, Linus Torvalds wrote:
> > > >
> > > >  Bad page state at free_hot_cold_page (in process 'taper', page ffff81000260b6f8)
> > > > flags:0x010000000000000c mapping:ffff8100355f1dd8 mapcount:2 count:0
> > > > Backtrace:
> > 
> > Ryan, can you test 2.6.15-rc4 and report what it does?
> > 
> > The "Bad page state" messages may (should) remain, but the crashes should 
> > be gone and the machine should hopefully continue functioning fine. And, 
> > perhaps more importantly, you should hopefully have a _new_ message about 
> > incomplete pfn mappings that should help pinpoint which driver causes 
> > this..

But we already know which driver does this: drivers/scsi/st.c.

> Will do, I plan to take this machine down Saturday to run memtest86 for
> a while (just to be sure - 2/3 of the RAM is new, but I should be seeing
> machine checks if that were the problem, no?) so I'll boot this after that.

Nick and I had already been looking at drivers/scsi/{sg.c,st.c},
brought there by __put_page in sg.c's peculiar sg_rb_correct4mmap,
which we'd like to remove.  But that's irrelevant to your pain, except...

One extract from the patches I'd like to send Doug and Kai for 2.6.15
or 2.6.16 is this below: since the incomplete get_user_pages path omits
to reset res, but has already released all the pages, it will result in
premature freeing of user pages, and behaviour just like you've seen.

Though I'd have thought incomplete get_user_pages was an exceptional
case, and a bit surprised you'd encounter it.  Perhaps there's some
other premature freeing in the driver, and this instance has nothing
whatever to do with it.

If the problem were easily reproducible, it'd be great if you could
try this patch; but I think you've said it's not :-(

Hugh

--- 2.6.14/drivers/scsi/st.c	2005-10-28 01:02:08.000000000 +0100
+++ linux/drivers/scsi/st.c	2005-12-01 20:06:02.000000000 +0000
@@ -4511,6 +4511,7 @@ static int sgl_map_user_pages(struct sca
 	if (res > 0) {
 		for (j=0; j < res; j++)
 			page_cache_release(pages[j]);
+		res = 0;
 	}
 	kfree(pages);
 	return res;

Re: Fw: crash on x86_64 - mm related?

[James Bottomley]

On Thu, 2005-12-01 at 11:38 -0800, Linus Torvalds wrote:
> Ryan, can you test 2.6.15-rc4 and report what it does?
> 
> The "Bad page state" messages may (should) remain, but the crashes should 
> be gone and the machine should hopefully continue functioning fine. And, 
> perhaps more importantly, you should hopefully have a _new_ message about 
> incomplete pfn mappings that should help pinpoint which driver causes 
> this..

On a side note, I have Kai's patch in the scsi-rc-fixes tree which I'm
getting ready to push.  Can we get a consensus on whether it should be
removed before I merge upwards?

Thanks,

James

Re: Fw: crash on x86_64 - mm related?

[Kai Makisara]

On Thu, 1 Dec 2005, James Bottomley wrote:

> On Thu, 2005-12-01 at 11:38 -0800, Linus Torvalds wrote:
> > Ryan, can you test 2.6.15-rc4 and report what it does?
> > 
> > The "Bad page state" messages may (should) remain, but the crashes should 
> > be gone and the machine should hopefully continue functioning fine. And, 
> > perhaps more importantly, you should hopefully have a _new_ message about 
> > incomplete pfn mappings that should help pinpoint which driver causes 
> > this..
> 
> On a side note, I have Kai's patch in the scsi-rc-fixes tree which I'm
> getting ready to push.  Can we get a consensus on whether it should be
> removed before I merge upwards?
> 
I think it should be removed because it is based partly on a wrong 
assumption: asynchronous writes are _not_ done together with direct i/o. 
(I have also experimentally verified that this does not happen.)

The patch includes the patch I sent sent to linux-scsi on Nov 21. Nobody 
has commented it and I don't know if the user pages have to be explicitly 
marked dirty after the HBA has read data there. If they have to, then this 
earlier patch is valid. If not, I will send a patch for 2.6.16 to remove 
the latent code.

-- 
Kai

Re: Fw: crash on x86_64 - mm related?

[Kai Makisara]

On Thu, 1 Dec 2005, Hugh Dickins wrote:

...
> Nick and I had already been looking at drivers/scsi/{sg.c,st.c},
> brought there by __put_page in sg.c's peculiar sg_rb_correct4mmap,
> which we'd like to remove.  But that's irrelevant to your pain, except...
> 
> One extract from the patches I'd like to send Doug and Kai for 2.6.15
> or 2.6.16 is this below: since the incomplete get_user_pages path omits
> to reset res, but has already released all the pages, it will result in
> premature freeing of user pages, and behaviour just like you've seen.
> 
> Though I'd have thought incomplete get_user_pages was an exceptional
> case, and a bit surprised you'd encounter it.  Perhaps there's some
> other premature freeing in the driver, and this instance has nothing
> whatever to do with it.
> 
> If the problem were easily reproducible, it'd be great if you could
> try this patch; but I think you've said it's not :-(
> 
> Hugh
> 
> --- 2.6.14/drivers/scsi/st.c	2005-10-28 01:02:08.000000000 +0100
> +++ linux/drivers/scsi/st.c	2005-12-01 20:06:02.000000000 +0000
> @@ -4511,6 +4511,7 @@ static int sgl_map_user_pages(struct sca
>  	if (res > 0) {
>  		for (j=0; j < res; j++)
>  			page_cache_release(pages[j]);
> +		res = 0;
>  	}
>  	kfree(pages);
>  	return res;
> 
Whether this fix solves the current problem or not, it clearly fixes a 
bug. If someone wants to include this into a patch series, you can add
Signed-off-by: Kai Makisara <kai.makisara@kolumbus.fi>
If not, I will include this when I send patches next time.

Thanks for noticing this problem.

Kai

Re: Fw: crash on x86_64 - mm related?

[Hugh Dickins]

On Thu, 1 Dec 2005, Kai Makisara wrote:
> On Thu, 1 Dec 2005, James Bottomley wrote:
> > 
> > On a side note, I have Kai's patch in the scsi-rc-fixes tree which I'm
> > getting ready to push.  Can we get a consensus on whether it should be
> > removed before I merge upwards?

I too need to decide whether to base my little sg.c,st.c patchset
on top of Kai's (as I see in 2.6.15-rc3-mm1, I presume) or on top
of 2.6.15-rc4.

> I think it should be removed because it is based partly on a wrong 
> assumption: asynchronous writes are _not_ done together with direct i/o. 
> (I have also experimentally verified that this does not happen.)

I'm assuming from this that I'd best base on 2.6.15-rc4;
but by all means overrule me if you've changed your mind.

> The patch includes the patch I sent sent to linux-scsi on Nov 21. Nobody 
> has commented it and I don't know if the user pages have to be explicitly 
> marked dirty after the HBA has read data there. If they have to, then this 
> earlier patch is valid.

What I see in 2.6.15-rc3-mm1 looks like three patches.

One to do with resetting sg_segs to 0 at various points:
I've no appreciation of that patch at all.  If it would help you for
me to add that into my little set, please send me a comment for it.

One to add an "is_read" argument to release_buffering.  Yes, that's
a part of my set too, though in my case called "dirtied" (and I believe
that the call at the end of st_read can say 0, because that's just for
an error path: when it's really dirtied user memory, it'll be read_tape
that does the release_buffering).  sg.c was always saying dirtied, even
when writing from memory; st.c was always saying not dirtied, even when
reading into memory.  Usually the latter is okay, get_user_pages has
said dirty in advance; but under pressure there's a window whereby it's
not good enough.  And SetPageDirty can be counter-productive these days,
so your patch is incomplete in that regard: I'll explain more in mine.

One to move around where release_buffering is called from:
that's the part you've decided was wrong, or at least unnecessary.

> If not, I will send a patch for 2.6.16 to remove the latent code.

I didn't understand that bit, but I probably don't need to.

Hugh

Re: Fw: crash on x86_64 - mm related?

[Kai Makisara]

On Fri, 2 Dec 2005, Hugh Dickins wrote:

> On Thu, 1 Dec 2005, Kai Makisara wrote:
> > On Thu, 1 Dec 2005, James Bottomley wrote:
> > > 
> > > On a side note, I have Kai's patch in the scsi-rc-fixes tree which I'm
> > > getting ready to push.  Can we get a consensus on whether it should be
> > > removed before I merge upwards?
> 
> I too need to decide whether to base my little sg.c,st.c patchset
> on top of Kai's (as I see in 2.6.15-rc3-mm1, I presume) or on top
> of 2.6.15-rc4.
> 
> > I think it should be removed because it is based partly on a wrong 
> > assumption: asynchronous writes are _not_ done together with direct i/o. 
> > (I have also experimentally verified that this does not happen.)
> 
> I'm assuming from this that I'd best base on 2.6.15-rc4;
> but by all means overrule me if you've changed your mind.
> 
> > The patch includes the patch I sent sent to linux-scsi on Nov 21. Nobody 
> > has commented it and I don't know if the user pages have to be explicitly 
> > marked dirty after the HBA has read data there. If they have to, then this 
> > earlier patch is valid.
> 
> What I see in 2.6.15-rc3-mm1 looks like three patches.
> 
> One to do with resetting sg_segs to 0 at various points:
> I've no appreciation of that patch at all.  If it would help you for
> me to add that into my little set, please send me a comment for it.
> 
I include at the end of this message the patch I sent to linux-scsi 
earlier. It should clarify what are the useful parts of the later patch.

I think the release_buffering() call at the end of st_read must say 1. All 
returns use the same path (except the one returning -ERESTARTSYS).

This is the comment related to this part:
- the number of s/g segments has not always been zeroed when the page
  pointers become invalid

The changes setting sg_segs to 0 don't fix any known bugs. They will be 
necessary if/when Mike Christie's patches will be merged later. You can 
omit these things now if you want.

> One to add an "is_read" argument to release_buffering.  Yes, that's
> a part of my set too, though in my case called "dirtied" (and I believe
> that the call at the end of st_read can say 0, because that's just for
> an error path: when it's really dirtied user memory, it'll be read_tape
> that does the release_buffering).  sg.c was always saying dirtied, even
> when writing from memory; st.c was always saying not dirtied, even when
> reading into memory.  Usually the latter is okay, get_user_pages has
> said dirty in advance; but under pressure there's a window whereby it's
> not good enough.  And SetPageDirty can be counter-productive these days,
> so your patch is incomplete in that regard: I'll explain more in mine.
> 
Good, I will leave sorting out these things to to you. Any naming is OK 
with me.

I think the release_buffering() call at the end of st_read must say 1. All 
returns use the same path (except the one returning -ERESTARTSYS).

st.c did set pages dirty after reading before 2.6.0-test4. It disappeared 
when code was rearranged and I don't have any notes about why.

> One to move around where release_buffering is called from:
> that's the part you've decided was wrong, or at least unnecessary.
> 
It is unnecessary but does not cause any problems. It is wrong because it 
hints that asynchronous writes could be done with direct i/o.

> > If not, I will send a patch for 2.6.16 to remove the latent code.
> 
> I didn't understand that bit, but I probably don't need to.
> 
After the previous comments, you don't need to :-) The meaning was that I 
would remove the extra parameter and code setting pages dirty from 
sgl_unmap_pages() if that code would never be used.

> Hugh
> 
Thanks for looking at the code.

Kai

----------
This patch is against 2.6.15-rc2 and fixes the following two bugs in the SCSI
tape driver:
- the pages dirtied by reading data to user space have not been marked dirty
- the number of s/g segments has not always been zeroed when the page
  pointers become invalid

Signed-off-by: Kai Makisara <kai.makisara@kolumbus.fi>
---
--- linux-2.6.15-rc2/drivers/scsi/st.c	2005-11-20 22:10:00.000000000 +0200
+++ linux-2.6.15-rc2-k1/drivers/scsi/st.c	2005-11-20 22:33:25.000000000 +0200
@@ -17,7 +17,7 @@
    Last modified: 18-JAN-1998 Richard Gooch <rgooch@atnf.csiro.au> Devfs support
  */
 
-static char *verstr = "20050830";
+static char *verstr = "20051120";
 
 #include <linux/module.h>
 
@@ -1449,14 +1449,15 @@ static int setup_buffering(struct scsi_t
 
 
 /* Can be called more than once after each setup_buffer() */
-static void release_buffering(struct scsi_tape *STp)
+static void release_buffering(struct scsi_tape *STp, int is_read)
 {
 	struct st_buffer *STbp;
 
 	STbp = STp->buffer;
 	if (STbp->do_dio) {
-		sgl_unmap_user_pages(&(STbp->sg[0]), STbp->do_dio, 0);
+		sgl_unmap_user_pages(&(STbp->sg[0]), STbp->do_dio, is_read);
 		STbp->do_dio = 0;
+		STbp->sg_segs = 0;
 	}
 }
 
@@ -1729,7 +1730,7 @@ st_write(struct file *filp, const char _
  out:
 	if (SRpnt != NULL)
 		scsi_release_request(SRpnt);
-	release_buffering(STp);
+	release_buffering(STp, 0);
 	up(&STp->lock);
 
 	return retval;
@@ -1787,7 +1788,7 @@ static long read_tape(struct scsi_tape *
 	SRpnt = *aSRpnt;
 	SRpnt = st_do_scsi(SRpnt, STp, cmd, bytes, DMA_FROM_DEVICE,
 			   STp->device->timeout, MAX_RETRIES, 1);
-	release_buffering(STp);
+	release_buffering(STp, 1);
 	*aSRpnt = SRpnt;
 	if (!SRpnt)
 		return STbp->syscall_result;
@@ -2058,7 +2059,7 @@ st_read(struct file *filp, char __user *
 		SRpnt = NULL;
 	}
 	if (do_dio) {
-		release_buffering(STp);
+		release_buffering(STp, 1);
 		STbp->buffer_bytes = 0;
 	}
 	up(&STp->lock);
@@ -3670,6 +3671,7 @@ static void normalize_buffer(struct st_b
 	}
 	STbuffer->frp_segs = STbuffer->orig_frp_segs;
 	STbuffer->frp_sg_current = 0;
+	STbuffer->sg_segs = 0;
 }
 
 

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Thu, Dec 01, 2005 at 08:21:57PM +0000, Hugh Dickins wrote:
> If the problem were easily reproducible, it'd be great if you could
> try this patch; but I think you've said it's not :-(
> 
> Hugh
> 
> --- 2.6.14/drivers/scsi/st.c	2005-10-28 01:02:08.000000000 +0100
> +++ linux/drivers/scsi/st.c	2005-12-01 20:06:02.000000000 +0000
> @@ -4511,6 +4511,7 @@ static int sgl_map_user_pages(struct sca
>  	if (res > 0) {
>  		for (j=0; j < res; j++)
>  			page_cache_release(pages[j]);
> +		res = 0;
>  	}
>  	kfree(pages);
>  	return res;

I can throw this in and test it for sure.  I'll run the backups every
day for a while to speed up the testing also.

Could someone please tell me exactly which patches I should include in
the kernel I will boot tomorrow?  I haven't played with -rc for ages, so
I'm no longer sure which kernel I should start with (2.6.14 or
2.6.14.3?).  Are the MPT-fusion performance fix patches in -rc4, or if
not will they still apply?

Thanks,
-ryan

Re: Fw: crash on x86_64 - mm related?

[Jesper Juhl]

On 12/2/05, Ryan Richter <ryan@tau.solarneutrino.net> wrote:
[snip]
>
> Could someone please tell me exactly which patches I should include in
> the kernel I will boot tomorrow?  I haven't played with -rc for ages, so
> I'm no longer sure which kernel I should start with (2.6.14 or
> 2.6.14.3?).  Are the MPT-fusion performance fix patches in -rc4, or if
> not will they still apply?
>
The -rc patches apply to the base x.y.z kernel - in this case 2.6.14
For more info, read Documentation/applying-patches.txt in a recent
kernel source (or use this link:
http://sosdg.org/~coywolf/lxr/source/Documentation/applying-patches.txt
)


--
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: Fw: crash on x86_64 - mm related?

[Hugh Dickins]

On Fri, 2 Dec 2005, Kai Makisara wrote:
> On Fri, 2 Dec 2005, Hugh Dickins wrote:
> I include at the end of this message the patch I sent to linux-scsi 
> earlier. It should clarify what are the useful parts of the later patch.

Thanks, yes.  I'll leave out updating the verstr[],
I think that should be sent by you alone.

> I think the release_buffering() call at the end of st_read must say 1. All 
> returns use the same path (except the one returning -ERESTARTSYS).

Okay, if you insist.  Yes, all those returns pass that way, but if it
actually did some reading into the memory, it called read_tape, which
did the effective release_buffering immediately after st_do_scsi.

But perhaps I'm misreading it, and even if not, someone will come
along and "correct" it later, or change things around and make my
not-dirty assumption wrong.

It's just that after seeing how sg.c is claiming to dirty even readonly
memory, I'm excessively averse to saying we've dirtied memory we haven't.
My hangup, I'll get over it!

> st.c did set pages dirty after reading before 2.6.0-test4. It disappeared 
> when code was rearranged and I don't have any notes about why.

Possibly because of issues with hugetlb compound pages: David Gibson
raised that issue recently with respect to access_process_vm
(page[1].mapping is reused and crashes set_page_dirty), I'm thinking
we don't want to add PageCompound tests all over, and have mailed
Andrew separately for guidance on that.

Hugh

Re: Fw: crash on x86_64 - mm related?

[Hugh Dickins]

On Fri, 2 Dec 2005, Ryan Richter wrote:
> On Thu, Dec 01, 2005 at 08:21:57PM +0000, Hugh Dickins wrote:
> > If the problem were easily reproducible, it'd be great if you could
> > try this patch; but I think you've said it's not :-(
> 
> I can throw this in and test it for sure.  I'll run the backups every
> day for a while to speed up the testing also.

Thanks - though everyone seems to have agreed that the patch is needed
whatever: so although it would be interesting to know if it has fixed
your problem, we're not waiting on you to go forward with it (James
already invited Linus to pull).

> Could someone please tell me exactly which patches I should include in
> the kernel I will boot tomorrow?

You hit the problem with 2.6.14.2.  My own opinion would be to apply
just that patch to that release, or to 2.6.14.3 if you prefer.

Linus was suggesting 2.6.15-rc4 because there's some debug there which
might have helped identify the offending driver: but your backtrace
had already showed us the offending driver.  Well, not proven: if a
page is doubly freed, the one that suffers is not necessarily the one
that's guilty; but it's a reasonable expectation.  I don't think his
debug would tell us anything more.

> I haven't played with -rc for ages, so
> I'm no longer sure which kernel I should start with (2.6.14 or
> 2.6.14.3?).

If you do want 2.6.15-rc4, then it's 2.6.14 + patch-2.6.15-rc4.

> Are the MPT-fusion performance fix patches in -rc4, or if
> not will they still apply?

I don't know it myself.  Is that the one about the module performing
much better than the builtin?  I'd expect that to be in.  If you have
it, suggest you look at your 2.6.15-rc4 source to see if it's there.

Hugh

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Fri, Dec 02, 2005 at 07:12:19PM +0000, Hugh Dickins wrote:
> On Fri, 2 Dec 2005, Ryan Richter wrote:
> > On Thu, Dec 01, 2005 at 08:21:57PM +0000, Hugh Dickins wrote:
> > > If the problem were easily reproducible, it'd be great if you could
> > > try this patch; but I think you've said it's not :-(
> > 
> > I can throw this in and test it for sure.  I'll run the backups every
> > day for a while to speed up the testing also.
> 
> Thanks - though everyone seems to have agreed that the patch is needed
> whatever: so although it would be interesting to know if it has fixed
> your problem, we're not waiting on you to go forward with it (James
> already invited Linus to pull).
> 
> > Could someone please tell me exactly which patches I should include in
> > the kernel I will boot tomorrow?
> 
> You hit the problem with 2.6.14.2.  My own opinion would be to apply
> just that patch to that release, or to 2.6.14.3 if you prefer.
> 
> Linus was suggesting 2.6.15-rc4 because there's some debug there which
> might have helped identify the offending driver: but your backtrace
> had already showed us the offending driver.  Well, not proven: if a
> page is doubly freed, the one that suffers is not necessarily the one
> that's guilty; but it's a reasonable expectation.  I don't think his
> debug would tell us anything more.

OK, I guess I'll stick with 2.6.14.3 for now, plus your patch.  Should I
keep Kai's st.c patch?  There was some mention of other patches, are
those relevant?  Most of that discussion went over my head...

Thanks,
-ryan

Re: Fw: crash on x86_64 - mm related?

[Kai Makisara]

On Fri, 2 Dec 2005, Hugh Dickins wrote:

> On Fri, 2 Dec 2005, Kai Makisara wrote:
> > On Fri, 2 Dec 2005, Hugh Dickins wrote:
> > I include at the end of this message the patch I sent to linux-scsi 
> > earlier. It should clarify what are the useful parts of the later patch.
> 
> Thanks, yes.  I'll leave out updating the verstr[],
> I think that should be sent by you alone.
> 
> > I think the release_buffering() call at the end of st_read must say 1. All 
> > returns use the same path (except the one returning -ERESTARTSYS).
> 
> Okay, if you insist.  Yes, all those returns pass that way, but if it
> actually did some reading into the memory, it called read_tape, which
> did the effective release_buffering immediately after st_do_scsi.
> 
> But perhaps I'm misreading it, and even if not, someone will come
> along and "correct" it later, or change things around and make my
> not-dirty assumption wrong.
> 
Your analysis is correct. It is not necessary or useful to use dirtied = 1 
at the end of st_read(). It has been a long time since I introduced 
release_buffering() into st.c and I did not read all of the code now.

> It's just that after seeing how sg.c is claiming to dirty even readonly
> memory, I'm excessively averse to saying we've dirtied memory we haven't.
> My hangup, I'll get over it!
> 
Please don't. I have a very conservative attitude to these things: my 
priority is to make sure that the data is correct even if it is not the 
fastest code. I will happily let others point out when I am too 
conservative.

-- 
Kai

Re: Fw: crash on x86_64 - mm related?

[Hugh Dickins]

On Fri, 2 Dec 2005, Ryan Richter wrote:
> 
> OK, I guess I'll stick with 2.6.14.3 for now, plus your patch.  Should I
> keep Kai's st.c patch?  There was some mention of other patches, are
> those relevant?  Most of that discussion went over my head...

For the "Bad page state" premature freeing you were seeing, only my
patch should be relevant.  There are other patches in the works, yes,
and we have good reasons for them; but don't worry about them for this.

Hugh

Re: Fw: crash on x86_64 - mm related?

[Hugh Dickins]

On Fri, 2 Dec 2005, Kai Makisara wrote:
> On Fri, 2 Dec 2005, Hugh Dickins wrote:
> 
> > It's just that after seeing how sg.c is claiming to dirty even readonly
> > memory, I'm excessively averse to saying we've dirtied memory we haven't.
> > My hangup, I'll get over it!
> > 
> Please don't. I have a very conservative attitude to these things: my 
> priority is to make sure that the data is correct even if it is not the 
> fastest code. I will happily let others point out when I am too 
> conservative.

I'm not certain which way you're directing me now: a conservative
attitude suggests we play safe at the end of st_read, by saying we might
somehow have dirtied memory there, perhaps if someone changes sequence.

As I presently, incompletely have it, to maintain more similarity with
sg.c, I've actually moved away from "dirtied" to "rw" READ or WRITE,
and it will look odd to put WRITE at the end of st_read.

I'm giving up for the evening, and probably won't have a chance to do
more until Sunday - the PageCompound issue still under discussion too.

Hugh

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Fri, Dec 02, 2005 at 08:40:56PM +0000, Hugh Dickins wrote:
> On Fri, 2 Dec 2005, Ryan Richter wrote:
> > 
> > OK, I guess I'll stick with 2.6.14.3 for now, plus your patch.  Should I
> > keep Kai's st.c patch?  There was some mention of other patches, are
> > those relevant?  Most of that discussion went over my head...
> 
> For the "Bad page state" premature freeing you were seeing, only my
> patch should be relevant.  There are other patches in the works, yes,
> and we have good reasons for them; but don't worry about them for this.

OK, I've booted 2.6.14.3+mpt fusion patch+your patch.  Memtest86 found
nothing overnight, so it's not that easy.  Also the backups will run
daily for at least the next week or two, so I'll pipe up if anything
happens.

Thanks,
-ryan

Re: Fw: crash on x86_64 - mm related?

[Kai Makisara]

On Fri, 2 Dec 2005, Hugh Dickins wrote:

> On Fri, 2 Dec 2005, Kai Makisara wrote:
> > On Fri, 2 Dec 2005, Hugh Dickins wrote:
> > 
> > > It's just that after seeing how sg.c is claiming to dirty even readonly
> > > memory, I'm excessively averse to saying we've dirtied memory we haven't.
> > > My hangup, I'll get over it!
> > > 
> > Please don't. I have a very conservative attitude to these things: my 
> > priority is to make sure that the data is correct even if it is not the 
> > fastest code. I will happily let others point out when I am too 
> > conservative.
> 
> I'm not certain which way you're directing me now: a conservative
> attitude suggests we play safe at the end of st_read, by saying we might
> somehow have dirtied memory there, perhaps if someone changes sequence.
> 
You should do what you think is best. One of the driver maintainers tasks 
is to look at the changes and ask questions if there is something 
suspicious. I asked and you pointed out that this time my concern was not 
valid.

During this discussion you have mentioned good arguments favouring both 
choices. Peformancewise we are talking about a rather theoretical issue: 
in a production server this issue might be relevant a few times a year.

My current suggestion is to use the value 1 (to minimise the possibility 
of bugs due to future changes). If you want, you can add a comment saying 
that using 1 is not strictly necessary with the current design.

> As I presently, incompletely have it, to maintain more similarity with
> sg.c, I've actually moved away from "dirtied" to "rw" READ or WRITE,
> and it will look odd to put WRITE at the end of st_read.
> 
It might be best to use naming that does not bind the purpose of the 
parameter to any "abstract" (i.e., read or write) activity when the 
purpose is to tell the function that the buffer contents may have been 
altered.

> I'm giving up for the evening, and probably won't have a chance to do
> more until Sunday - the PageCompound issue still under discussion too.
> 
> Hugh
> 

-- 
Kai

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

Another crash during last night's backup:

Bad page state at free_hot_cold_page (in process 'taper', page ffff810002410cc0)
flags:0x010000000000000c mapping:ffff81017d5beb70 mapcount:2 count:0
Backtrace:

Call Trace:<ffffffff80159fd3>{bad_page+99} <ffffffff8015a9a5>{free_hot_cold_page+101}
       <ffffffff80162047>{__page_cache_release+151} <ffffffff802b9068>{sgl_unmap_user_pages+120}
       <ffffffff802b497b>{release_buffering+27} <ffffffff802b5031>{st_write+1697}
       <ffffffff8017af86>{vfs_write+198} <ffffffff8017b0e3>{sys_write+83}
       <ffffffff8010db7a>{system_call+126} 
Trying to fix it up, but a reboot is needed
Bad page state at free_hot_cold_page (in process 'taper', page ffff810002410cc0)
flags:0x010000000000081c mapping:ffff810064cd6910 mapcount:0 count:0
Backtrace:

Call Trace:<ffffffff80159fd3>{bad_page+99} <ffffffff8015a9a5>{free_hot_cold_page+101}
       <ffffffff80162047>{__page_cache_release+151} <ffffffff802b9068>{sgl_unmap_user_pages+120}
       <ffffffff802b497b>{release_buffering+27} <ffffffff802b5031>{st_write+1697}
       <ffffffff8017af86>{vfs_write+198} <ffffffff8017b0e3>{sys_write+83}
       <ffffffff8010db7a>{system_call+126} 
Trying to fix it up, but a reboot is needed
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at include/linux/mm.h:341
invalid operand: 0000 [1] SMP 
CPU 1 
Modules linked in: bonding
Pid: 11402, comm: taper Tainted: G    B 2.6.14.3 #1
RIP: 0010:[<ffffffff802b904d>] <ffffffff802b904d>{sgl_unmap_user_pages+93}
RSP: 0018:ffff8100ca6d9e18  EFLAGS: 00010256
RAX: 0000000000000000 RBX: 0000000000000007 RCX: 0000000000000005
RDX: 00000000000000e0 RSI: 0000000000000001 RDI: ffff810002410cc0
RBP: ffff810004990068 R08: 00000000ffffffff R09: 0000000000000000
R10: 0000000000008000 R11: 0000000000000200 R12: 0000000000000008
R13: 0000000000000000 R14: 0000000000008000 R15: ffff810004953d10
FS:  00002aaaab53d880(0000) GS:ffffffff804db880(0000) knlGS:00000000555bc920
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002aaaaab5ffff CR3: 000000008c49e000 CR4: 00000000000006e0
Process taper (pid: 11402, threadinfo ffff8100ca6d8000, task ffff81017dabc200)
Stack: ffff81017ff1e600 ffff810004990000 0000000000000040 0000000000008000 
       ffff810004953c00 ffffffff802b497b ffff810004990000 ffffffff802b5031 
       ffff810000000000 ffffffff00000001 
Call Trace:<ffffffff802b497b>{release_buffering+27} <ffffffff802b5031>{st_write+1697}
       <ffffffff8017af86>{vfs_write+198} <ffffffff8017b0e3>{sys_write+83}
       <ffffffff8010db7a>{system_call+126} 

Code: 0f 0b 68 3a 13 3a 80 c2 55 01 f0 83 47 08 ff 0f 98 c0 84 c0 
RIP <ffffffff802b904d>{sgl_unmap_user_pages+93} RSP <ffff8100ca6d9e18>
 ----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at mm/rmap.c:487
invalid operand: 0000 [2] SMP 
CPU 1 
Modules linked in: bonding
Pid: 11402, comm: taper Tainted: G    B 2.6.14.3 #1
RIP: 0010:[<ffffffff8016f437>] <ffffffff8016f437>{page_remove_rmap+39}
RSP: 0018:ffff8100ca6d9ab0  EFLAGS: 00010286
RAX: 00000000ffffffff RBX: ffff8100cc4e96f8 RCX: ffff81000000f000
RDX: 0000000000000000 RSI: 800000005bba8067 RDI: ffff810002410cc0
RBP: 00002aaaaaadf000 R08: 0000000000000000 R09: ffff810002802738
R10: 00000000fffffffa R11: 0000000000000000 R12: ffff810101c22380
R13: 800000005bba8067 R14: ffff810002410cc0 R15: 0000000000000000
FS:  00002aaaab53d880(0000) GS:ffffffff804db880(0000) knlGS:00000000555bc920
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002aaaaab5ffff CR3: 000000008c49e000 CR4: 00000000000006e0
Process taper (pid: 11402, threadinfo ffff8100ca6d8000, task ffff81017dabc200)
Stack: ffffffff80166f0d 00002aaaaab62000 ffff81006d16faa8 00002aaaaab62000 
       00002aaaaab62000 00002aaaaab61fff ffff8100b424c550 00002aaaaab62000 
       ffffffff801671c0 ffff8100ca6d9d68 
Call Trace:<ffffffff80166f0d>{zap_pte_range+477} <ffffffff801671c0>{unmap_page_range+496}
       <ffffffff80167325>{unmap_vmas+293} <ffffffff8016cfe2>{exit_mmap+162}
       <ffffffff80131ce1>{mmput+49} <ffffffff801371c6>{do_exit+438}
       <ffffffff8010f6f1>{die+81} <ffffffff8010f9df>{do_invalid_op+159}
       <ffffffff802b904d>{sgl_unmap_user_pages+93} <ffffffff80381ff6>{thread_return+86}
       <ffffffff802a86e2>{sym_setup_data_and_start+402} <ffffffff8010e84d>{error_exit+0}
       <ffffffff802b904d>{sgl_unmap_user_pages+93} <ffffffff802b9068>{sgl_unmap_user_pages+120}
       <ffffffff802b497b>{release_buffering+27} <ffffffff802b5031>{st_write+1697}
       <ffffffff8017af86>{vfs_write+198} <ffffffff8017b0e3>{sys_write+83}
       <ffffffff8010db7a>{system_call+126} 

Code: 0f 0b 68 1b 36 3a 80 c2 e7 01 48 c7 c6 ff ff ff ff bf 20 00 
RIP <ffffffff8016f437>{page_remove_rmap+39} RSP <ffff8100ca6d9ab0>
 <1>Fixing recursive fault but reboot is needed!
Bad page state at prep_new_page (in process 'dumper', page ffff810002410cc0)
flags:0x010000000000001c mapping:0000000000000000 mapcount:-1 count:0
Backtrace:

Call Trace:<ffffffff80159fd3>{bad_page+99} <ffffffff8015a3b1>{prep_new_page+65}
       <ffffffff8015ab6e>{buffered_rmqueue+302} <ffffffff8015adc5>{__alloc_pages+261}
       <ffffffff801581fd>{generic_file_buffered_write+413}
       <ffffffff801963ae>{inode_update_time+62} <ffffffff80158a48>{__generic_file_aio_write_nolock+936}
       <ffffffff8031f524>{sock_common_recvmsg+52}<0>Bad page state at free_hot_cold_page (in process 'dump', page ffff810002410cc0)
flags:0x010000000000001c mapping:0000000000000000 mapcount:-1 count:0
Backtrace:

Call Trace:<ffffffff80159fd3>{bad_page+99} <ffffffff8015a9a5>{free_hot_cold_page+101}
       <ffffffff8015b1d0>{__pagevec_free+32} <ffffffff801621dd>{release_pages+349}
       <ffffffff801c6ee0>{journal_stop+512} <ffffffff801c06fd>{__ext3_journal_stop+45}
       <ffffffff80162217>{__pagevec_release+23} <ffffffff801a0a02>{mpage_writepages+690}
       <ffffffff801ba1f0>{ext3_ordered_writepage+0} <ffffffff8019ef12>{__sync_single_inode+114}
       <ffffffff8019f223>{__writeback_single_inode+355} <ffffffff801567ff>{find_get_pages_tag+127}
       <ffffffff8016252a>{pagevec_lookup_tag+26} <ffffffff80155f0c>{wait_on_page_writeback_range+220}
       <ffffffff8019f421>{sync_sb_inodes+481} <ffffffff8019f664>{sync_inodes_sb+132}
       <ffffffff8019f734>{__sync_inodes+116} <ffffffff8019f7a1>{sync_inodes+17}
       <ffffffff8017c542>{do_sync+18} <ffffffff8017c59e>{sys_sync+14}
       <ffffffff8010db7a>{system_call+126} 
Trying to fix it up, but a reboot is needed
 <ffffffff8031bbb0>{sock_aio_read+272}
       <ffffffff80158d3e>{generic_file_aio_write+110} <ffffffff801b77c3>{ext3_file_write+35}
       <ffffffff8017ae83>{do_sync_write+211} <ffffffff8018ed00>{__pollwait+0}
       <ffffffff8014a2f0>{autoremove_wake_function+0} <ffffffff8018f6c1>{sys_select+1153}
       <ffffffff8017af86>{vfs_write+198} <ffffffff8017b0e3>{sys_write+83}
       <ffffffff8010db7a>{system_call+126} 
Trying to fix it up, but a reboot is needed
general protection fault: 0000 [3] SMP 
CPU 0 
Modules linked in: bonding
Pid: 1303, comm: kjournald Tainted: G    B 2.6.14.3 #1
RIP: 0010:[<ffffffff8015fdfa>] <ffffffff8015fdfa>{cache_alloc_refill+330}
RSP: 0018:ffff81017ed4dc38  EFLAGS: 00010012
RAX: cb0f489d027409f5 RBX: 000000000000002c RCX: 000000000000000f
RDX: ffff810004820760 RSI: ffff81005bba8000 RDI: ffff81001a184030
RBP: ffff81000483a400 R08: ffff810004820750 R09: ffff810004820760
R10: 0000000000000180 R11: 0000000000000000 R12: ffff810004820740
R13: ffff810004834440 R14: ffff810004820788 R15: 0000000000011200
FS:  00002aaaab00c4a0(0000) GS:ffffffff804db800(0000) knlGS:00000000555bc920
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002aaaaaac0000 CR3: 00000000a21ae000 CR4: 00000000000006e0
Process kjournald (pid: 1303, threadinfo ffff81017ed4c000, task ffff8101028bcf40)
Stack: ffffffff8014a2f0 0000000000011200 0000000000011210 ffff810004820ec0 
       0000000000000010 ffff81017eda3000 0000000000000200 ffffffff80160266 
       0000000000000296 ffffffff80159669 
Call Trace:<ffffffff8014a2f0>{autoremove_wake_function+0} <ffffffff80160266>{kmem_cache_alloc+54}
       <ffffffff80159669>{mempool_alloc+57} <ffffffff8014a320>{wake_bit_function+0}
       <ffffffff8017fdcc>{bio_alloc_bioset+28} <ffffffff8017ff50>{bio_alloc+16}
       <ffffffff8017f6c6>{submit_bh+150} <ffffffff8017f7ef>{ll_rw_block+143}
       <ffffffff801c82fa>{journal_commit_transaction+1114}
       <ffffffff80381fa0>{thread_return+0} <ffffffff80381ff6>{thread_return+86}
       <ffffffff8013d349>{lock_timer_base+41} <ffffffff801cb03e>{kjournald+238}
       <ffffffff8014a2f0>{autoremove_wake_function+0} <ffffffff8014a2f0>{autoremove_wake_function+0}
       <ffffffff801caf40>{commit_timeout+0} <ffffffff8010ea02>{child_rip+8}
       <ffffffff801caf50>{kjournald+0} <ffffffff8010e9fa>{child_rip+0}
       

Code: 48 89 50 08 48 89 02 48 c7 46 08 00 02 20 00 83 7e 24 ff 48 
RIP <ffffffff8015fdfa>{cache_alloc_refill+330} RSP <ffff81017ed4dc38>
 NMI Watchdog detected LOCKUP on CPU 1
CPU 1 
Modules linked in: bonding
Pid: 918, comm: md0_raid5 Tainted: G    B 2.6.14.3 #1
RIP: 0010:[<ffffffff8038385b>] <ffffffff8038385b>{.text.lock.spinlock+116}
RSP: 0018:ffff8101028a1c90  EFLAGS: 00000086
RAX: ffff810004820740 RBX: ffff810004820788 RCX: 000000000000000c
RDX: 0000000000000001 RSI: ffff81000000f000 RDI: ffff810004820788
RBP: ffff810004834440 R08: ffff8100048208c0 R09: 0000000000000000
R10: 00000000ffffffff R11: 0000000000000001 R12: 0000000000000000
R13: ffff8101028547c0 R14: ffff8101028547d0 R15: 0000000000000001
FS:  00002aaaab53d8e0(0000) GS:ffffffff804db880(0000) knlGS:00000000555bc920
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00002aaaaaac0000 CR3: 00000000a21af000 CR4: 00000000000006e0
Process md0_raid5 (pid: 918, threadinfo ffff8101028a0000, task ffff8101028bc1c0)

(serial console output cut off here)

-ryan

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

A little more of the last oops made it out to the syslog server:

Dec  6 00:35:50 xarello kernel:  NMI Watchdog detected LOCKUP on CPU 1
Dec  6 00:35:50 xarello kernel: Modules linked in: bonding
Dec  6 00:35:50 xarello kernel: RIP: 0010:[<ffffffff8038385b>] <ffffffff8038385b>{.text.lock.spinlock+116}
Dec  6 00:35:50 xarello kernel: RAX: ffff810004820740 RBX: ffff810004820788 RCX: 000000000000000c
Dec  6 00:35:50 xarello kernel: RBP: ffff810004834440 R08: ffff8100048208c0 R09: 0000000000000000
Dec  6 00:35:50 xarello kernel: FS:  00002aaaab53d8e0(0000) GS:ffffffff804db880(0000) knlGS:00000000555bc920
Dec  6 00:35:50 xarello kernel: CR2: 00002aaaaaac0000 CR3: 00000000a21af000 CR4: 00000000000006e0
Dec  6 00:35:50 xarello kernel: Stack: ffffffff801607d9 ffff810073bafac0 0000000000000282 ffff810004820ec0
Dec  6 00:35:50 xarello kernel:        ffffffff801597ca ffff810073bafac0
Dec  6 00:35:50 xarello kernel:        <ffffffff80180031>{bio_put+49} <ffffffff8017f62a>{end_bio_bh_io_sync+58}
Dec  6 00:35:50 xarello kernel:        <ffffffff80306d08>{handle_stripe+3752} <ffffffff80307885>{raid5d+741}
Dec  6 00:35:50 xarello kernel:        <ffffffff8030ddb9>{md_thread+281} <ffffffff8014a2f0>{autoremove_wake_function+0}
Dec  6 00:35:50 xarello kernel:        <ffffffff8030dca0>{md_thread+0} <ffffffff80149cc2>{kthread+146}
Dec  6 00:35:50 xarello kernel:        <ffffffff80149c30>{kthread+0} <ffffffff8010e9fa>{child_rip+0}
Dec  6 00:35:50 xarello kernel:
Dec  6 00:35:50 xarello kernel: console shuts up ...

-ryan

Re: Fw: crash on x86_64 - mm related?

[Hugh Dickins]

On Tue, 6 Dec 2005, Ryan Richter wrote:

> Another crash during last night's backup:
> 
> Bad page state at free_hot_cold_page (in process 'taper', page ffff810002410cc0)
> flags:0x010000000000000c mapping:ffff81017d5beb70 mapcount:2 count:0
> Bad page state at free_hot_cold_page (in process 'taper', page ffff810002410cc0)
> flags:0x010000000000081c mapping:ffff810064cd6910 mapcount:0 count:0
> Kernel BUG at include/linux/mm.h:341
> Pid: 11402, comm: taper Tainted: G    B 2.6.14.3 #1
> RIP: 0010:[<ffffffff802b904d>] <ffffffff802b904d>{sgl_unmap_user_pages+93}
> Kernel BUG at mm/rmap.c:487
> Pid: 11402, comm: taper Tainted: G    B 2.6.14.3 #1
> RIP: 0010:[<ffffffff8016f437>] <ffffffff8016f437>{page_remove_rmap+39}
> Bad page state at prep_new_page (in process 'dumper', page ffff810002410cc0)
> flags:0x010000000000001c mapping:0000000000000000 mapcount:-1 count:0
> general protection fault: 0000 [3] SMP 
> Pid: 1303, comm: kjournald Tainted: G    B 2.6.14.3 #1
> RIP: 0010:[<ffffffff8015fdfa>] <ffffffff8015fdfa>{cache_alloc_refill+330}
>  NMI Watchdog detected LOCKUP on CPU 1
> Pid: 918, comm: md0_raid5 Tainted: G    B 2.6.14.3 #1
> RIP: 0010:[<ffffffff8038385b>] <ffffffff8038385b>{.text.lock.spinlock+116}

Thanks for the further report.  And you had my st.c patch in along
with 2.6.14.3, but it still happened, very much like before (except the
latter errors, general protection fault onwards - but once we get into
using one page for two uses at the same time, anything can go wrong).

I've been staring and thinking, but no inspiration yet.

Hugh

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Tue, Dec 06, 2005 at 08:31:43PM +0000, Hugh Dickins wrote:
> Thanks for the further report.  And you had my st.c patch in along
> with 2.6.14.3, but it still happened, very much like before (except the
> latter errors, general protection fault onwards - but once we get into
> using one page for two uses at the same time, anything can go wrong).
> 
> I've been staring and thinking, but no inspiration yet.

That's correct, thanks for looking.  Let me know if there's anything I
could do get more information.

-ryan

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

I don't know if this is related, but in the last couple days I've seen
hundreds of these messages from this machine (and I haven't seen it
before):

Hangcheck: hangcheck value past margin!

-ryan

Re: Fw: crash on x86_64 - mm related?

[Hugh Dickins]

On Tue, 6 Dec 2005, Ryan Richter wrote:
> On Tue, Dec 06, 2005 at 08:31:43PM +0000, Hugh Dickins wrote:
> > Thanks for the further report.  And you had my st.c patch in along
> > with 2.6.14.3, but it still happened, very much like before (except the
> > latter errors, general protection fault onwards - but once we get into
> > using one page for two uses at the same time, anything can go wrong).
> > 
> > I've been staring and thinking, but no inspiration yet.
> 
> That's correct, thanks for looking.  Let me know if there's anything I
> could do get more information.

Still no luck with it, I'm afraid: I've found no error in st.c,
beyond what the patch already fixed, to account for what you see.

I have noticed that struct st_buffer's unsigned char do_dio ought to be
unsigned short, in order to accommodate its maximum value of 256; but
that would not account for your symptoms, nor does taper approach that
maximum.

The symptoms are consistent with that do_dio field (near the start of
struct st_buffer) being corrupted, infrequently but repeatedly - the
same page has been mis-released three times before your first error
message.  But I see nothing wrong with st.c's handling of do_dio,
nor with get_user_pages' returned count.

So I think the best I can suggest is that you rebuild your kernel with
CONFIG_DEBUG_SLAB=y, and see if that sheds any light.  And if you don't
mind doing so, advancing to 2.6.15-rc5 (which includes my patch) with
CONFIG_DEBUG_SLAB=y would be even better: there's a chance it's a bug
that's already been fixed.  Though that won't really be proved if you
don't hit the problem, since the new kernel will be sufficiently
different that it might just have shifted the bug aside.

To be honest, the symptoms seem a little too regular to blame someone
overrunning their slab; but I've searched in vain and must move on.

Hugh

Re: Fw: crash on x86_64 - mm related?

[Hugh Dickins]

On Wed, 7 Dec 2005, Ryan Richter wrote:

> I don't know if this is related, but in the last couple days I've seen
> hundreds of these messages from this machine (and I haven't seen it
> before):
> 
> Hangcheck: hangcheck value past margin!

I'm unclear whether you rebooted after the "general protection fault"
and "NMI Watchdog" messages (if you had to, I'm sure you did, but I
don't know whether the machine appeared to work on after those).

If you didn't reboot, then discount these "Hangcheck" messages as a
consequence of the earlier errors, and reboot as soon as convenient.
It's interesting for me to see the Bad page state, Bad page state,
BUG in mm.h, BUG in rmap.c; but after those you ought to reboot.

If you did reboot, well, maybe related, but it doesn't tell us much.

Hugh

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Wed, Dec 07, 2005 at 06:56:04PM +0000, Hugh Dickins wrote:
> On Wed, 7 Dec 2005, Ryan Richter wrote:
> 
> > I don't know if this is related, but in the last couple days I've seen
> > hundreds of these messages from this machine (and I haven't seen it
> > before):
> > 
> > Hangcheck: hangcheck value past margin!
> 
> I'm unclear whether you rebooted after the "general protection fault"
> and "NMI Watchdog" messages (if you had to, I'm sure you did, but I
> don't know whether the machine appeared to work on after those).
> 
> If you didn't reboot, then discount these "Hangcheck" messages as a
> consequence of the earlier errors, and reboot as soon as convenient.
> It's interesting for me to see the Bad page state, Bad page state,
> BUG in mm.h, BUG in rmap.c; but after those you ought to reboot.
> 
> If you did reboot, well, maybe related, but it doesn't tell us much.

Yep, the machine pretty much hung after that and was reset the next
morning.

-ryan

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Wed, Dec 07, 2005 at 06:37:22PM +0000, Hugh Dickins wrote:
> So I think the best I can suggest is that you rebuild your kernel with
> CONFIG_DEBUG_SLAB=y, and see if that sheds any light.  And if you don't
> mind doing so, advancing to 2.6.15-rc5 (which includes my patch) with
> CONFIG_DEBUG_SLAB=y would be even better: there's a chance it's a bug
> that's already been fixed.  Though that won't really be proved if you
> don't hit the problem, since the new kernel will be sufficiently
> different that it might just have shifted the bug aside.

OK, it's running 2.6.15-rc5 now, and no other patches.

Here's the new .config and dmesg (not much new):


#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.15-rc5
# Wed Dec  7 15:29:23 2005
#
CONFIG_X86_64=y
CONFIG_64BIT=y
CONFIG_X86=y
CONFIG_SEMAPHORE_SLEEPERS=y
CONFIG_MMU=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_CMPXCHG=y
CONFIG_EARLY_PRINTK=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32

#
# General setup
#
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
# CONFIG_POSIX_MQUEUE is not set
# CONFIG_BSD_PROCESS_ACCT is not set
CONFIG_SYSCTL=y
# CONFIG_AUDIT is not set
CONFIG_HOTPLUG=y
CONFIG_KOBJECT_UEVENT=y
# CONFIG_IKCONFIG is not set
# CONFIG_CPUSETS is not set
CONFIG_INITRAMFS_SOURCE=""
# CONFIG_EMBEDDED is not set
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SHMEM=y
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
CONFIG_OBSOLETE_MODPARM=y
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
# CONFIG_KMOD is not set
CONFIG_STOP_MACHINE=y

#
# Block layer
#
CONFIG_LBD=y

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_AS=y
# CONFIG_DEFAULT_DEADLINE is not set
# CONFIG_DEFAULT_CFQ is not set
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="anticipatory"

#
# Processor type and features
#
CONFIG_MK8=y
# CONFIG_MPSC is not set
# CONFIG_GENERIC_CPU is not set
CONFIG_X86_L1_CACHE_BYTES=64
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_TSC=y
CONFIG_X86_GOOD_APIC=y
CONFIG_MICROCODE=y
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set
CONFIG_X86_IO_APIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_MTRR=y
CONFIG_SMP=y
# CONFIG_SCHED_SMT is not set
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
# CONFIG_PREEMPT_BKL is not set
CONFIG_NUMA=y
CONFIG_K8_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
# CONFIG_NUMA_EMU is not set
CONFIG_ARCH_DISCONTIGMEM_ENABLE=y
CONFIG_ARCH_DISCONTIGMEM_DEFAULT=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_SELECT_MEMORY_MODEL=y
# CONFIG_FLATMEM_MANUAL is not set
CONFIG_DISCONTIGMEM_MANUAL=y
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_DISCONTIGMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_NEED_MULTIPLE_NODES=y
# CONFIG_SPARSEMEM_STATIC is not set
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_HAVE_ARCH_EARLY_PFN_TO_NID=y
CONFIG_NR_CPUS=8
# CONFIG_HOTPLUG_CPU is not set
CONFIG_HPET_TIMER=y
# CONFIG_X86_PM_TIMER is not set
CONFIG_HPET_EMULATE_RTC=y
CONFIG_GART_IOMMU=y
CONFIG_SWIOTLB=y
CONFIG_X86_MCE=y
# CONFIG_X86_MCE_INTEL is not set
CONFIG_X86_MCE_AMD=y
CONFIG_PHYSICAL_START=0x100000
# CONFIG_KEXEC is not set
# CONFIG_SECCOMP is not set
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_ISA_DMA_API=y
CONFIG_GENERIC_PENDING_IRQ=y

#
# Power management options
#
CONFIG_PM=y
CONFIG_PM_LEGACY=y
# CONFIG_PM_DEBUG is not set

#
# ACPI (Advanced Configuration and Power Interface) Support
#
CONFIG_ACPI=y
# CONFIG_ACPI_AC is not set
# CONFIG_ACPI_BATTERY is not set
CONFIG_ACPI_BUTTON=y
# CONFIG_ACPI_VIDEO is not set
# CONFIG_ACPI_HOTKEY is not set
CONFIG_ACPI_FAN=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_NUMA=y
# CONFIG_ACPI_ASUS is not set
# CONFIG_ACPI_IBM is not set
# CONFIG_ACPI_TOSHIBA is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_SYSTEM=y
# CONFIG_ACPI_CONTAINER is not set

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
# CONFIG_UNORDERED_IO is not set
# CONFIG_PCIEPORTBUS is not set
# CONFIG_PCI_MSI is not set
CONFIG_PCI_LEGACY_PROC=y
# CONFIG_PCI_DEBUG is not set

#
# PCCARD (PCMCIA/CardBus) support
#
# CONFIG_PCCARD is not set

#
# PCI Hotplug Support
#
# CONFIG_HOTPLUG_PCI is not set

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
# CONFIG_BINFMT_MISC is not set
CONFIG_IA32_EMULATION=y
# CONFIG_IA32_AOUT is not set
CONFIG_COMPAT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_UID16=y

#
# Networking
#
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_FIB_HASH=y
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_TUNNEL is not set
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_BIC=y
# CONFIG_IPV6 is not set
# CONFIG_NETFILTER is not set

#
# DCCP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_DCCP is not set

#
# SCTP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_SCTP is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set

#
# QoS and/or fair queueing
#
# CONFIG_NET_SCHED is not set

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_IEEE80211 is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
# CONFIG_FW_LOADER is not set
# CONFIG_DEBUG_DRIVER is not set

#
# Connector - unified userspace <-> kernelspace linker
#
# CONFIG_CONNECTOR is not set

#
# Memory Technology Devices (MTD)
#
# CONFIG_MTD is not set

#
# Parallel port support
#
CONFIG_PARPORT=y
CONFIG_PARPORT_PC=y
# CONFIG_PARPORT_SERIAL is not set
# CONFIG_PARPORT_PC_FIFO is not set
# CONFIG_PARPORT_PC_SUPERIO is not set
# CONFIG_PARPORT_GSC is not set
# CONFIG_PARPORT_1284 is not set

#
# Plug and Play support
#
# CONFIG_PNP is not set

#
# Block devices
#
CONFIG_BLK_DEV_FD=y
# CONFIG_PARIDE is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=y
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=4096
CONFIG_BLK_DEV_INITRD=y
# CONFIG_CDROM_PKTCDVD is not set
# CONFIG_ATA_OVER_ETH is not set

#
# ATA/ATAPI/MFM/RLL support
#
# CONFIG_IDE is not set

#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_ST=y
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=y
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=y
CONFIG_CHR_DEV_SCH=y

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
# CONFIG_SCSI_MULTI_LUN is not set
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set

#
# SCSI Transport Attributes
#
CONFIG_SCSI_SPI_ATTRS=y
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_ATTRS is not set

#
# SCSI low-level drivers
#
# CONFIG_ISCSI_TCP is not set
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
# CONFIG_SCSI_SATA is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_PPA is not set
# CONFIG_SCSI_IMM is not set
CONFIG_SCSI_SYM53C8XX_2=y
CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1
CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
# CONFIG_SCSI_SYM53C8XX_IOMAPPED is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_QLOGIC_FC is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
CONFIG_SCSI_QLA2XXX=y
# CONFIG_SCSI_QLA21XX is not set
# CONFIG_SCSI_QLA22XX is not set
# CONFIG_SCSI_QLA2300 is not set
# CONFIG_SCSI_QLA2322 is not set
# CONFIG_SCSI_QLA6312 is not set
# CONFIG_SCSI_QLA24XX is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_DEBUG is not set

#
# Multi-device support (RAID and LVM)
#
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
# CONFIG_MD_LINEAR is not set
# CONFIG_MD_RAID0 is not set
CONFIG_MD_RAID1=y
# CONFIG_MD_RAID10 is not set
CONFIG_MD_RAID5=y
# CONFIG_MD_RAID6 is not set
# CONFIG_MD_MULTIPATH is not set
# CONFIG_MD_FAULTY is not set
CONFIG_BLK_DEV_DM=y
# CONFIG_DM_CRYPT is not set
# CONFIG_DM_SNAPSHOT is not set
# CONFIG_DM_MIRROR is not set
# CONFIG_DM_ZERO is not set
# CONFIG_DM_MULTIPATH is not set

#
# Fusion MPT device support
#
CONFIG_FUSION=y
CONFIG_FUSION_SPI=y
# CONFIG_FUSION_FC is not set
# CONFIG_FUSION_SAS is not set
CONFIG_FUSION_MAX_SGE=128
CONFIG_FUSION_CTL=y

#
# IEEE 1394 (FireWire) support
#
# CONFIG_IEEE1394 is not set

#
# I2O device support
#
# CONFIG_I2O is not set

#
# Network device support
#
CONFIG_NETDEVICES=y
# CONFIG_DUMMY is not set
CONFIG_BONDING=m
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set

#
# ARCnet devices
#
# CONFIG_ARCNET is not set

#
# PHY device support
#

#
# Ethernet (10 or 100Mbit)
#
# CONFIG_NET_ETHERNET is not set

#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SIS190 is not set
# CONFIG_SKGE is not set
# CONFIG_SK98LIN is not set
CONFIG_TIGON3=y
# CONFIG_BNX2 is not set

#
# Ethernet (10000 Mbit)
#
# CONFIG_CHELSIO_T1 is not set
# CONFIG_IXGB is not set
# CONFIG_S2IO is not set

#
# Token Ring devices
#
# CONFIG_TR is not set

#
# Wireless LAN (non-hamradio)
#
# CONFIG_NET_RADIO is not set

#
# Wan interfaces
#
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
# CONFIG_SHAPER is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1600
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=1200
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_TSDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_INPUT_MOUSE is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_SERIAL_NONSTANDARD is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_ACPI=y
CONFIG_SERIAL_8250_NR_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
CONFIG_PRINTER=y
# CONFIG_LP_CONSOLE is not set
# CONFIG_PPDEV is not set
# CONFIG_TIPAR is not set

#
# IPMI
#
# CONFIG_IPMI_HANDLER is not set

#
# Watchdog Cards
#
CONFIG_WATCHDOG=y
# CONFIG_WATCHDOG_NOWAYOUT is not set

#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=y
# CONFIG_ACQUIRE_WDT is not set
# CONFIG_ADVANTECH_WDT is not set
# CONFIG_ALIM1535_WDT is not set
# CONFIG_ALIM7101_WDT is not set
# CONFIG_SC520_WDT is not set
# CONFIG_EUROTECH_WDT is not set
# CONFIG_IB700_WDT is not set
# CONFIG_IBMASR is not set
# CONFIG_WAFER_WDT is not set
# CONFIG_I6300ESB_WDT is not set
# CONFIG_I8XX_TCO is not set
# CONFIG_SC1200_WDT is not set
# CONFIG_60XX_WDT is not set
# CONFIG_SBC8360_WDT is not set
# CONFIG_CPU5_WDT is not set
# CONFIG_W83627HF_WDT is not set
# CONFIG_W83877F_WDT is not set
# CONFIG_W83977F_WDT is not set
# CONFIG_MACHZ_WDT is not set

#
# PCI-based Watchdog Cards
#
# CONFIG_PCIPCWATCHDOG is not set
# CONFIG_WDTPCI is not set

#
# USB-based Watchdog Cards
#
# CONFIG_USBPCWATCHDOG is not set
CONFIG_HW_RANDOM=y
# CONFIG_NVRAM is not set
CONFIG_RTC=y
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set

#
# Ftape, the floppy tape device driver
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
# CONFIG_AGP_INTEL is not set
# CONFIG_DRM is not set
# CONFIG_MWAVE is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_HPET is not set
CONFIG_HANGCHECK_TIMER=y

#
# TPM devices
#
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set

#
# I2C support
#
CONFIG_I2C=y
CONFIG_I2C_CHARDEV=y

#
# I2C Algorithms
#
# CONFIG_I2C_ALGOBIT is not set
# CONFIG_I2C_ALGOPCF is not set
# CONFIG_I2C_ALGOPCA is not set

#
# I2C Hardware Bus support
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
# CONFIG_I2C_AMD756 is not set
CONFIG_I2C_AMD8111=y
# CONFIG_I2C_I801 is not set
# CONFIG_I2C_I810 is not set
# CONFIG_I2C_PIIX4 is not set
CONFIG_I2C_ISA=y
# CONFIG_I2C_NFORCE2 is not set
# CONFIG_I2C_PARPORT is not set
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_PROSAVAGE is not set
# CONFIG_I2C_SAVAGE4 is not set
# CONFIG_SCx200_ACB is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_STUB is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set
# CONFIG_I2C_VOODOO3 is not set
# CONFIG_I2C_PCA_ISA is not set

#
# Miscellaneous I2C Chip support
#
# CONFIG_SENSORS_DS1337 is not set
# CONFIG_SENSORS_DS1374 is not set
# CONFIG_SENSORS_EEPROM is not set
# CONFIG_SENSORS_PCF8574 is not set
# CONFIG_SENSORS_PCA9539 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_SENSORS_RTC8564 is not set
# CONFIG_SENSORS_MAX6875 is not set
# CONFIG_RTC_X1205_I2C is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set

#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set

#
# Hardware Monitoring support
#
CONFIG_HWMON=y
CONFIG_HWMON_VID=y
# CONFIG_SENSORS_ADM1021 is not set
# CONFIG_SENSORS_ADM1025 is not set
# CONFIG_SENSORS_ADM1026 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ADM9240 is not set
# CONFIG_SENSORS_ASB100 is not set
# CONFIG_SENSORS_ATXP1 is not set
# CONFIG_SENSORS_DS1621 is not set
# CONFIG_SENSORS_FSCHER is not set
# CONFIG_SENSORS_FSCPOS is not set
# CONFIG_SENSORS_GL518SM is not set
# CONFIG_SENSORS_GL520SM is not set
# CONFIG_SENSORS_IT87 is not set
# CONFIG_SENSORS_LM63 is not set
# CONFIG_SENSORS_LM75 is not set
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
# CONFIG_SENSORS_LM80 is not set
# CONFIG_SENSORS_LM83 is not set
CONFIG_SENSORS_LM85=y
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_LM92 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_SIS5595 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
# CONFIG_SENSORS_SMSC47B397 is not set
# CONFIG_SENSORS_VIA686A is not set
CONFIG_SENSORS_W83781D=y
# CONFIG_SENSORS_W83792D is not set
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83627HF is not set
# CONFIG_SENSORS_W83627EHF is not set
# CONFIG_SENSORS_HDAPS is not set
# CONFIG_HWMON_DEBUG_CHIP is not set

#
# Misc devices
#
# CONFIG_IBM_ASM is not set

#
# Multimedia Capabilities Port drivers
#

#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set

#
# Digital Video Broadcasting Devices
#
# CONFIG_DVB is not set

#
# Graphics support
#
# CONFIG_FB is not set
# CONFIG_VIDEO_SELECT is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y

#
# Sound
#
# CONFIG_SOUND is not set

#
# USB support
#
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_BANDWIDTH is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set

#
# USB Host Controller Drivers
#
CONFIG_USB_EHCI_HCD=y
# CONFIG_USB_EHCI_SPLIT_ISO is not set
# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_OHCI_HCD is not set
CONFIG_USB_UHCI_HCD=y
# CONFIG_USB_SL811_HCD is not set

#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
# CONFIG_USB_PRINTER is not set

#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support'
#

#
# may also be needed; see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=y
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_DPCM is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set

#
# USB Input Devices
#
# CONFIG_USB_HID is not set

#
# USB HID Boot Protocol drivers
#
# CONFIG_USB_KBD is not set
# CONFIG_USB_MOUSE is not set
# CONFIG_USB_AIPTEK is not set
# CONFIG_USB_WACOM is not set
# CONFIG_USB_ACECAD is not set
# CONFIG_USB_KBTAB is not set
# CONFIG_USB_POWERMATE is not set
# CONFIG_USB_MTOUCH is not set
# CONFIG_USB_ITMTOUCH is not set
# CONFIG_USB_EGALAX is not set
# CONFIG_USB_YEALINK is not set
# CONFIG_USB_XPAD is not set
# CONFIG_USB_ATI_REMOTE is not set
# CONFIG_USB_KEYSPAN_REMOTE is not set
# CONFIG_USB_APPLETOUCH is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set

#
# USB Multimedia devices
#
# CONFIG_USB_DABUSB is not set

#
# Video4Linux support is needed for USB Multimedia device support
#

#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set
# CONFIG_USB_MON is not set

#
# USB port drivers
#
# CONFIG_USB_USS720 is not set

#
# USB Serial Converter support
#
# CONFIG_USB_SERIAL is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGETKIT is not set
# CONFIG_USB_PHIDGETSERVO is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TEST is not set

#
# USB DSL modem support
#

#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set

#
# MMC/SD Card support
#
# CONFIG_MMC is not set

#
# InfiniBand support
#
# CONFIG_INFINIBAND is not set

#
# SN Devices
#

#
# Firmware Drivers
#
# CONFIG_EDD is not set
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set

#
# File systems
#
CONFIG_EXT2_FS=y
# CONFIG_EXT2_FS_XATTR is not set
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=y
# CONFIG_EXT3_FS_XATTR is not set
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_FS_POSIX_ACL is not set
# CONFIG_XFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_INOTIFY is not set
# CONFIG_QUOTA is not set
CONFIG_DNOTIFY=y
# CONFIG_AUTOFS_FS is not set
# CONFIG_AUTOFS4_FS is not set
# CONFIG_FUSE_FS is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
# CONFIG_ZISOFS is not set
# CONFIG_UDF_FS is not set

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_RAMFS=y
# CONFIG_RELAYFS_FS is not set

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set

#
# Network File Systems
#
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
# CONFIG_NFS_V3_ACL is not set
# CONFIG_NFS_V4 is not set
# CONFIG_NFS_DIRECTIO is not set
CONFIG_NFSD=y
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V3_ACL is not set
# CONFIG_NFSD_V4 is not set
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
# CONFIG_RPCSEC_GSS_KRB5 is not set
# CONFIG_RPCSEC_GSS_SPKM3 is not set
# CONFIG_SMB_FS is not set
# CONFIG_CIFS is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
# CONFIG_9P_FS is not set

#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y

#
# Native Language Support
#
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
# CONFIG_NLS_ASCII is not set
CONFIG_NLS_ISO8859_1=y
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=y

#
# Instrumentation Support
#
# CONFIG_PROFILING is not set
# CONFIG_KPROBES is not set

#
# Kernel hacking
#
# CONFIG_PRINTK_TIME is not set
CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_LOG_BUF_SHIFT=17
# CONFIG_DETECT_SOFTLOCKUP is not set
# CONFIG_SCHEDSTATS is not set
CONFIG_DEBUG_SLAB=y
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_FS is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_FRAME_POINTER is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_INIT_DEBUG is not set
# CONFIG_IOMMU_DEBUG is not set

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set

#
# Cryptographic options
#
# CONFIG_CRYPTO is not set

#
# Hardware crypto devices
#

#
# Library routines
#
# CONFIG_CRC_CCITT is not set
# CONFIG_CRC16 is not set
# CONFIG_CRC32 is not set
# CONFIG_LIBCRC32C is not set


Bootdata ok (command line is BOOT_IMAGE=Linux ro root=854 console=ttyS0,19200n8)
Linux version 2.6.15-rc5 (root@xarello) (gcc version 3.3.5 (Debian 1:3.3.5-13)) #1 SMP Wed Dec 7 15:35:34 EST 2005
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 00000000f6ff0000 (usable)
 BIOS-e820: 00000000f6ff0000 - 00000000f6fff000 (ACPI data)
 BIOS-e820: 00000000f6fff000 - 00000000f7000000 (ACPI NVS)
 BIOS-e820: 00000000ff7c0000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 0000000180000000 (usable)
ACPI: RSDP (v002 ACPIAM                                ) @ 0x00000000000f4680
ACPI: XSDT (v001 A M I  OEMXSDT  0x06000318 MSFT 0x00000097) @ 0x00000000f6ff0100
ACPI: FADT (v001 A M I  OEMFACP  0x06000318 MSFT 0x00000097) @ 0x00000000f6ff0281
ACPI: MADT (v001 A M I  OEMAPIC  0x06000318 MSFT 0x00000097) @ 0x00000000f6ff0380
ACPI: OEMB (v001 A M I  OEMBIOS  0x06000318 MSFT 0x00000097) @ 0x00000000f6fff040
ACPI: ASF! (v001 AMIASF AMDSTRET 0x00000001 INTL 0x02002026) @ 0x00000000f6ff3530
ACPI: DSDT (v001  0ABCF 0ABCF007 0x00000007 INTL 0x02002026) @ 0x0000000000000000
Scanning NUMA topology in Northbridge 24
Number of nodes 2
Node 0 MemBase 0000000000000000 Limit 0000000100000000
Node 1 MemBase 0000000100000000 Limit 0000000180000000
Using 30 for the hash shift.
Using node hash shift of 30
Bootmem setup node 0 0000000000000000-0000000100000000
Bootmem setup node 1 0000000100000000-0000000180000000
On node 0 totalpages: 996171
  DMA zone: 2851 pages, LIFO batch:0
  DMA32 zone: 993320 pages, LIFO batch:31
  Normal zone: 0 pages, LIFO batch:0
  HighMem zone: 0 pages, LIFO batch:0
On node 1 totalpages: 517023
  DMA zone: 0 pages, LIFO batch:0
  DMA32 zone: 0 pages, LIFO batch:0
  Normal zone: 517023 pages, LIFO batch:31
  HighMem zone: 0 pages, LIFO batch:0
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 15:5 APIC version 16
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
Processor #1 15:5 APIC version 16
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, version 17, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x03] address[0xfebfe000] gsi_base[24])
IOAPIC[1]: apic_id 3, version 17, address 0xfebfe000, GSI 24-27
ACPI: IOAPIC (id[0x04] address[0xfebff000] gsi_base[28])
IOAPIC[2]: apic_id 4, version 17, address 0xfebff000, GSI 28-31
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Setting APIC routing to flat
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at f8000000 (gap: f7000000:87c0000)
Checking aperture...
CPU 0: aperture @ 1ee0000000 size 64 MB
Aperture from northbridge cpu 0 beyond 4GB. Ignoring.
No AGP bridge found
Your BIOS doesn't leave a aperture memory hole
Please enable the IOMMU option in the BIOS setup
This costs you 64 MB of RAM
Mapping aperture over 65536 KB of RAM @ 8000000
Built 2 zonelists
Kernel command line: BOOT_IMAGE=Linux ro root=854 console=ttyS0,19200n8
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 131072 bytes)
time.c: Using 1.193182 MHz PIT timer.
time.c: Detected 1394.571 MHz processor.
Console: colour VGA+ 80x25
Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
Memory: 5974956k/6291456k available (2612k kernel code, 168592k reserved, 1064k data, 244k init)
Calibrating delay using timer specific routine.. 2795.00 BogoMIPS (lpj=5590014)
Mount-cache hash table entries: 256
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 0(1) -> Node 0 -> Core 0
mtrr: v2.0 (20020519)
Using local APIC timer interrupts.
Detected 12.451 MHz APIC timer.
Booting processor 1/2 APIC 0x1
Initializing CPU#1
Calibrating delay using timer specific routine.. 2789.44 BogoMIPS (lpj=5578889)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 1(1) -> Node 1 -> Core 0
AMD Opteron(tm) Processor 240 stepping 01
CPU 1: Syncing TSC to CPU 0.
CPU 1: synchronized TSC with CPU 0 (last diff 7 cycles, maxerr 913 cycles)
Brought up 2 CPUs
time.c: Using PIT/TSC based timekeeping.
testing NMI watchdog ... OK.
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Subsystem revision 20050902
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI: Probing PCI hardware (bus 00)
Boot video device is 0000:03:06.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.GOLA._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.GOLB._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 *5 6 7 9 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 *10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 *9 10 11 12 14 15)
SCSI subsystem initialized
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
PCI-DMA: Disabling AGP.
PCI-DMA: aperture base @ 8000000 size 65536 KB
PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
PCI: Bridge: 0000:00:06.0
  IO window: b000-bfff
  MEM window: fca00000-feafffff
  PREFETCH window: ff500000-ff5fffff
PCI: Bridge: 0000:00:0a.0
  IO window: a000-afff
  MEM window: fc400000-fc9fffff
  PREFETCH window: ff400000-ff4fffff
PCI: Bridge: 0000:00:0b.0
  IO window: disabled.
  MEM window: fc300000-fc3fffff
  PREFETCH window: ff300000-ff3fffff
IA-32 Microcode Update Driver: v1.14 <tigran@veritas.com>
IA32 emulation $Id: sys_ia32.c,v 1.32 2002/03/24 13:02:28 ak Exp $
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
PCI: MSI quirk detected. pci_msi_quirk set.
PCI: MSI quirk detected. pci_msi_quirk set.
ACPI: Power Button (FF) [PWRF]
ACPI: Power Button (CM) [PWRB]
ACPI: Processor [CPU1] (supports 8 throttling states)
lp: driver loaded but no devices found
Real Time Clock Driver v1.12
hw_random: AMD768 system management I/O registers at 0x5000.
hw_random hardware driver 1.0.0 loaded
Software Watchdog Timer: 0.07 initialized. soft_noboot=0 soft_margin=60 sec (nowayout= 0)
Linux agpgart interface v0.101 (c) Dave Jones
Hangcheck: starting hangcheck timer 0.9.0 (tick is 180 seconds, margin is 60 seconds).
Hangcheck: Using monotonic_clock().
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
parport0: PC-style at 0x378 [PCSPP(,...)]
lp0: using parport0 (polling).
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
loop: loaded (max 8 devices)
tg3.c:v3.43 (Oct 24, 2005)
GSI 16 sharing vector 0xA9 and IRQ 16
ACPI: PCI Interrupt 0000:02:09.0[A] -> GSI 24 (level, low) -> IRQ 16
eth0: Tigon3 [partno(BCM95704A7) rev 2003 PHY(5704)] (PCIX:100MHz:64-bit) 10/100/1000BaseT Ethernet 00:e0:81:51:d6:c9
eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[1] 
eth0: dma_rwctrl[769f4000]
GSI 17 sharing vector 0xB1 and IRQ 17
ACPI: PCI Interrupt 0000:02:09.1[B] -> GSI 25 (level, low) -> IRQ 17
eth1: Tigon3 [partno(BCM95704A7) rev 2003 PHY(5704)] (PCIX:100MHz:64-bit) 10/100/1000BaseT Ethernet 00:e0:81:51:d6:ca
eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[1] 
eth1: dma_rwctrl[769f4000]
GSI 18 sharing vector 0xB9 and IRQ 18
ACPI: PCI Interrupt 0000:03:04.0[A] -> GSI 16 (level, low) -> IRQ 18
sym0: <875> rev 0x26 at pci 0000:03:04.0 irq 18
sym0: Symbios NVRAM, ID 7, Fast-20, SE, parity checking
sym0: open drain IRQ line driver, using on-chip SRAM
sym0: using LOAD/STORE-based firmware.
sym0: SCSI BUS has been reset.
scsi0 : sym-2.2.1
  Vendor: SONY      Model: SDX-400C          Rev: 0702
  Type:   Sequential-Access                  ANSI SCSI revision: 02
 target0:0:4: Beginning Domain Validation
 target0:0:4: asynchronous.
 target0:0:4: wide asynchronous.
 target0:0:4: FAST-20 WIDE SCSI 40.0 MB/s ST (50 ns, offset 15)
 target0:0:4: Domain Validation skipping write tests
 target0:0:4: Ending Domain Validation
  Vendor: OVERLAND  Model: LIBRARYPRO        Rev: 0417
  Type:   Medium Changer                     ANSI SCSI revision: 02
 target0:0:6: Beginning Domain Validation
 target0:0:6: asynchronous.
 target0:0:6: wide asynchronous.
 target0:0:6: FAST-10 WIDE SCSI 20.0 MB/s ST (100 ns, offset 15)
 target0:0:6: Domain Validation skipping write tests
 target0:0:6: Ending Domain Validation
st: Version 20050830, fixed bufsize 32768, s/g segs 256
st 0:0:4:0: Attached scsi tape st0<4>st0: try direct i/o: yes (alignment 512 B), max page reachable by HBA 1572864
st 0:0:4:0: Attached scsi generic sg0 type 1
 0:0:6:0: Attached scsi generic sg1 type 8
SCSI Media Changer driver v0.25 
ch0: type #1 (mt): 0x0+1 [medium transport]
ch0: type #2 (st): 0x1+18 [storage]
ch0: type #3 (ie): 0xd0+1 [import/export]
ch0: type #4 (dt): 0xe0+1 [data transfer]
ch0: dt 0xe0: ID 4, LUN 0, name: SONY     SDX-400C         0702
ch0: INITIALIZE ELEMENT STATUS, may take some time ...
ch0: ... finished
ch 0:0:6:0: Attached scsi changer ch0
Fusion MPT base driver 3.03.04
Copyright (c) 1999-2005 LSI Logic Corporation
Fusion MPT SPI Host driver 3.03.04
ACPI: PCI Interrupt 0000:02:0a.0[A] -> GSI 24 (level, low) -> IRQ 16
mptbase: Initiating ioc0 bringup
ioc0: 53C1030: Capabilities={Initiator}
scsi1 : ioc0: LSI53C1030, FwRev=01030600h, Ports=1, MaxQ=255, IRQ=16
  Vendor: SUPER     Model: GEM318            Rev: 0   
  Type:   Processor                          ANSI SCSI revision: 02
 1:0:8:0: Attached scsi generic sg2 type 3
  Vendor: SEAGATE   Model: ST3146807LC       Rev: 0006
  Type:   Direct-Access                      ANSI SCSI revision: 03
SCSI device sda: 286749488 512-byte hdwr sectors (146816 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 286749488 512-byte hdwr sectors (146816 MB)
SCSI device sda: drive cache: write back
 sda: sda1
sd 1:0:9:0: Attached scsi disk sda
sd 1:0:9:0: Attached scsi generic sg3 type 0
  Vendor: SEAGATE   Model: ST3146807LC       Rev: 0006
  Type:   Direct-Access                      ANSI SCSI revision: 03
SCSI device sdb: 286749488 512-byte hdwr sectors (146816 MB)
SCSI device sdb: drive cache: write back
SCSI device sdb: 286749488 512-byte hdwr sectors (146816 MB)
SCSI device sdb: drive cache: write back
 sdb: sdb1
sd 1:0:10:0: Attached scsi disk sdb
sd 1:0:10:0: Attached scsi generic sg4 type 0
  Vendor: SEAGATE   Model: ST3146807LC       Rev: 0006
  Type:   Direct-Access                      ANSI SCSI revision: 03
SCSI device sdc: 286749488 512-byte hdwr sectors (146816 MB)
SCSI device sdc: drive cache: write back
SCSI device sdc: 286749488 512-byte hdwr sectors (146816 MB)
SCSI device sdc: drive cache: write back
 sdc: sdc1
sd 1:0:11:0: Attached scsi disk sdc
sd 1:0:11:0: Attached scsi generic sg5 type 0
  Vendor: SEAGATE   Model: ST3146807LC       Rev: 0007
  Type:   Direct-Access                      ANSI SCSI revision: 03
SCSI device sdd: 286749488 512-byte hdwr sectors (146816 MB)
SCSI device sdd: drive cache: write back
SCSI device sdd: 286749488 512-byte hdwr sectors (146816 MB)
SCSI device sdd: drive cache: write back
 sdd: sdd1
sd 1:0:12:0: Attached scsi disk sdd
sd 1:0:12:0: Attached scsi generic sg6 type 0
  Vendor: SEAGATE   Model: ST3146807LC       Rev: 0007
  Type:   Direct-Access                      ANSI SCSI revision: 03
SCSI device sde: 286749488 512-byte hdwr sectors (146816 MB)
SCSI device sde: drive cache: write back
SCSI device sde: 286749488 512-byte hdwr sectors (146816 MB)
SCSI device sde: drive cache: write back
 sde: sde1
sd 1:0:13:0: Attached scsi disk sde
sd 1:0:13:0: Attached scsi generic sg7 type 0
ACPI: PCI Interrupt 0000:02:0a.1[B] -> GSI 25 (level, low) -> IRQ 17
mptbase: Initiating ioc1 bringup
ioc1: 53C1030: Capabilities={Initiator}
scsi2 : ioc1: LSI53C1030, FwRev=01030600h, Ports=1, MaxQ=255, IRQ=17
  Vendor: SEAGATE   Model: ST336607LW        Rev: 0007
  Type:   Direct-Access                      ANSI SCSI revision: 03
SCSI device sdf: 71687372 512-byte hdwr sectors (36704 MB)
SCSI device sdf: drive cache: write back
SCSI device sdf: 71687372 512-byte hdwr sectors (36704 MB)
SCSI device sdf: drive cache: write back
 sdf: sdf1 sdf2 sdf3 sdf4
sd 2:0:0:0: Attached scsi disk sdf
sd 2:0:0:0: Attached scsi generic sg8 type 0
Fusion MPT misc device (ioctl) driver 3.03.04
mptctl: Registered with Fusion MPT base driver
mptctl: /dev/mptctl @ (major,minor=10,220)
USB Universal Host Controller Interface driver v2.3
Initializing USB Mass Storage driver...
usbcore: registered new driver usb-storage
USB Mass Storage support registered.
mice: PS/2 mouse device common for all mice
i2c /dev entries driver
md: raid1 personality registered as nr 3
md: raid5 personality registered as nr 4
raid5: automatically using best checksumming function: generic_sse
   generic_sse:  4297.000 MB/sec
raid5: using function: generic_sse (4297.000 MB/sec)
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
device-mapper: 4.4.0-ioctl (2005-01-12) initialised: dm-devel@redhat.com
NET: Registered protocol family 2
IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
TCP bic registered
NET: Registered protocol family 1
NET: Registered protocol family 17
md: Autodetecting RAID arrays.
md: autorun ...
md: considering sde1 ...
md:  adding sde1 ...
md:  adding sdd1 ...
md:  adding sdc1 ...
md:  adding sdb1 ...
md:  adding sda1 ...
md: created md0
md: bind<sda1>
md: bind<sdb1>
md: bind<sdc1>
md: bind<sdd1>
md: bind<sde1>
md: running: <sde1><sdd1><sdc1><sdb1><sda1>
raid5: device sde1 operational as raid disk 4
raid5: device sdd1 operational as raid disk 3
raid5: device sdc1 operational as raid disk 2
raid5: device sdb1 operational as raid disk 1
raid5: device sda1 operational as raid disk 0
raid5: allocated 5312kB for md0
raid5: raid level 5 set md0 active with 5 out of 5 devices, algorithm 2
RAID5 conf printout:
 --- rd:5 wd:5 fd:0
 disk 0, o:1, dev:sda1
 disk 1, o:1, dev:sdb1
 disk 2, o:1, dev:sdc1
 disk 3, o:1, dev:sdd1
 disk 4, o:1, dev:sde1
md: ... autorun DONE.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 244k freed
Adding 2097136k swap on /dev/sdf2.  Priority:-1 extents:1 across:2097136k
EXT3 FS on sdf4, internal journal
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sdf3, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-32, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-34, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-33, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-35, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-36, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-38, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-39, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-40, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-41, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-2, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-3, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-4, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-5, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-6, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-7, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-8, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-9, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-10, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-11, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-12, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-13, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-14, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-15, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-16, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-17, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-18, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-19, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-20, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-21, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-22, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-23, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-24, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-25, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-26, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-27, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-28, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-29, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-30, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-31, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Ethernet Channel Bonding Driver: v2.6.5 (November 4, 2005)
bonding: MII link monitoring set to 100 ms
bonding: bond0: enslaving eth0 as an active interface with a down link.
bonding: bond0: enslaving eth1 as an active interface with a down link.
tg3: eth0: Link is up at 1000 Mbps, full duplex.
bonding: bond0: link status definitely up for interface eth0.
tg3: eth0: Flow control is off for TX and off for RX.
tg3: eth1: Link is up at 1000 Mbps, full duplex.
tg3: eth1: Flow control is off for TX and off for RX.
bonding: bond0: link status definitely up for interface eth1.
process `syslogd' is using obsolete setsockopt SO_BSDCOMPAT
st0: Block limits 2 - 16777215 bytes.
program mt is using a deprecated SCSI ioctl, please convert it to SG_IO
program mt is using a deprecated SCSI ioctl, please convert it to SG_IO
program mt is using a deprecated SCSI ioctl, please convert it to SG_IO
program mt is using a deprecated SCSI ioctl, please convert it to SG_IO

-ryan

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

And yet another crash, this time during boot:


mptbase: Initiating ioc1 bringup
ioc1: 53C1030: Capabilities={Initiator}
scsi2 : ioc1: LSI53C1030, FwRev=01030600h, Ports=1, MaxQ=255, IRQ=17
  Vendor: SEAGATE   Model: ST336607LW        Rev: 0007
  Type:   Direct-Access                      ANSI SCSI revision: 03
SCSI device sdf: 71687372 512-byte hdwr sectors (36704 MB)
SCSI device sdf: drive cache: write back
SCSI device sdf: 71687372 512-byte hdwr sectors (36704 MB)
SCSI device sdf: drive cache: write back
 sdf: sdf1 sdf2 sdf3 sdf4
sd 2:0:0:0: Attached scsi disk sdf
sd 2:0:0:0: Attached scsi generic sg8 type 0
general protection fault: 0000 [1] SMP
CPU 1
Modules linked in:
Pid: 0, comm: swapper Not tainted 2.6.15-rc5 #1
RIP: 0010:[<ffffffff802a0a04>] <ffffffff802a0a04>{scsi_run_queue+20}
RSP: 0000:ffff81010288be18  EFLAGS: 00010296
RAX: 0000000000000001 RBX: ffff81000c0ad770 RCX: 0000000000000001
RDX: 0000000000000001 RSI: ffffffff80222090 RDI: 6b6b6b6b6b6b6b6b
RBP: ffff8100f6fba230 R08: 0000000000000000 R09: ffff81000c0aa198
R10: ffff8100f6f88578 R11: 0000000000000001 R12: 0000000000000292
R13: ffff81000486e3e8 R14: ffff81000486e3e8 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffffffff804ee880(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000000101000 CR4: 00000000000006e0
Process swapper (pid: 0, threadinfo ffff81000482e000, task ffff81000482ae00)
Stack: 0000000000000001 ffff81000c0ad770 ffff8100f6fba230 0000000000000292
       0000000000000001 ffffffff802a0c4f ffff81000c0d93b8 ffff8100f6fba230
       0000000000000024 0000000000010000
Call Trace: <IRQ> <ffffffff802a0c4f>{scsi_end_request+207} <ffffffff802a11af>{scsi_io_completion+1039}
       <ffffffff8029c3c6>{scsi_finish_command+150} <ffffffff8029c2dd>{scsi_softirq+333}
       <ffffffff801386eb>{__do_softirq+107} <ffffffff8010eeeb>{call_softirq+31}
       <ffffffff80110781>{do_softirq+49} <ffffffff80110744>{do_IRQ+52}
       <ffffffff8010e10c>{ret_from_intr+0}  <EOI> <ffffffff8038866c>{thread_return+140}
       <ffffffff8010bb1a>{default_idle+58} <ffffffff8010bd61>{cpu_idle+97}


Code: f6 87 cd 01 00 00 80 48 8b 2f 74 05 e8 fb fe ff ff 48 8b 7d
RIP <ffffffff802a0a04>{scsi_run_queue+20} RSP <ffff81010288be18>
 <0>Kernel panic - not syncing: Aiee, killing interrupt handler!

-ryan

Re: Fw: crash on x86_64 - mm related?

[Linus Torvalds]

On Mon, 12 Dec 2005, Ryan Richter wrote:
>
> And yet another crash, this time during boot:

The instruction that crashes is

	testb  $0x80,0x1cd(%rdi)

with %rdi being 6b6b6b6b6b6b6b6b, which is the pattern that slab poisoning 
uses for free areas. 

I think it's the "sdev->single_lun" test at the very top of the function, 
where "sdev" was initialized with "q->queuedata". So it looks like 
somebody free'd the request_queue structure before the IO completed.

Definitely sounds like something screwy in SCSI.. I don't think this is VM 
related.

		Linus

Re: Fw: crash on x86_64 - mm related?

[James Bottomley]

On Mon, 2005-12-12 at 09:40 -0800, Linus Torvalds wrote:
> I think it's the "sdev->single_lun" test at the very top of the function, 
> where "sdev" was initialized with "q->queuedata". So it looks like 
> somebody free'd the request_queue structure before the IO completed.
> 
> Definitely sounds like something screwy in SCSI.. I don't think this is VM 
> related.

Welcome back home.

This is that SCSI patch you reversed just before you went away.  If you
put it back again, the problem will go away ...

James

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Mon, Dec 12, 2005 at 11:45:30AM -0600, James Bottomley wrote:
> On Mon, 2005-12-12 at 09:40 -0800, Linus Torvalds wrote:
> > I think it's the "sdev->single_lun" test at the very top of the
> > function, where "sdev" was initialized with "q->queuedata". So it
> > looks like somebody free'd the request_queue structure before the IO
> > completed.
> > 
> > Definitely sounds like something screwy in SCSI.. I don't think this
> > is VM related.

That was just a stupid guess to get someone's attention :)

> Welcome back home.
> 
> This is that SCSI patch you reversed just before you went away.  If
> you put it back again, the problem will go away ...
> 

Should I try this patch out?  If so, where can I find it?

Thanks,
-ryan

Re: Fw: crash on x86_64 - mm related?

[Linus Torvalds]

On Mon, 12 Dec 2005, James Bottomley wrote:
> 
> This is that SCSI patch you reversed just before you went away.  If you
> put it back again, the problem will go away ...

Well, that patch is definitely broken.

You say that it just causes a warning about sleeping in interrupt context, 
while I say that the warning is a serious error. If that semaphore _ever_ 
is write-locked, the whole machine will crash from trying to sleep when it 
cannot sleep.

So I can certainly undo the undo, but the fact is, the code is CRAP. I'd 
much rather get a real fix instead of having to select between two known 
bugs.

Please?

		Linus

Re: Fw: crash on x86_64 - mm related?

[James Bottomley]

On Mon, 2005-12-12 at 10:09 -0800, Linus Torvalds wrote:
> Well, that patch is definitely broken.

No, it's not; all it's doing is deferring the device_put() from the
scsi_put_command() to after the scsi_run_queue(), which doesn't fix the
sleep while atomic problem of the device release method.  In both cases
we still get the semaphore in atomic context problem which is caused by
scsi_reap_target() doing a device_del(), which I assumed (wrongly) was
valid from atomic context.

I'll fix the scsi_reap_target(), but it's nothing to do with the patch
you reversed.

> You say that it just causes a warning about sleeping in interrupt context, 
> while I say that the warning is a serious error. If that semaphore _ever_ 
> is write-locked, the whole machine will crash from trying to sleep when it 
> cannot sleep.
> 
> So I can certainly undo the undo, but the fact is, the code is CRAP. I'd 
> much rather get a real fix instead of having to select between two known 
> bugs.

I'll find a fix for the real problem, but this patch isn't the cause.

James

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Mon, Dec 12, 2005 at 12:24:42PM -0600, James Bottomley wrote:
> I'll find a fix for the real problem, but this patch isn't the cause.

Is the patch set you posted yesterday supposed to fix this?  If so, is
it available in patch form anywhere?

Thanks,
-ryan

Re: Fw: crash on x86_64 - mm related?

[James Bottomley]

On Thu, 2005-12-15 at 14:09 -0500, Ryan Richter wrote:
> On Mon, Dec 12, 2005 at 12:24:42PM -0600, James Bottomley wrote:
> > I'll find a fix for the real problem, but this patch isn't the cause.
> 
> Is the patch set you posted yesterday supposed to fix this?  If so, is
> it available in patch form anywhere?

No, I've been too busin integrating other people's patches to work on
ones of my own.  Try this.

James

diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c
--- a/drivers/scsi/scsi_scan.c
+++ b/drivers/scsi/scsi_scan.c
@@ -400,6 +400,35 @@ static struct scsi_target *scsi_alloc_ta
 	return found_target;
 }
 
+struct work_queue_wrapper {
+	struct work_struct	work;
+	struct scsi_target	*starget;
+};
+
+static void scsi_target_reap_work(void *data) {
+	struct work_queue_wrapper *wqw = (struct work_queue_wrapper *)data;
+	struct scsi_target *starget = wqw->starget;
+	struct Scsi_Host *shost = dev_to_shost(starget->dev.parent);
+	unsigned long flags;
+
+	kfree(wqw);
+
+	spin_lock_irqsave(shost->host_lock, flags);
+
+	if (--starget->reap_ref == 0 && list_empty(&starget->devices)) {
+		list_del_init(&starget->siblings);
+		spin_unlock_irqrestore(shost->host_lock, flags);
+		device_del(&starget->dev);
+		transport_unregister_device(&starget->dev);
+		put_device(&starget->dev);
+		return;
+
+	}
+	spin_unlock_irqrestore(shost->host_lock, flags);
+
+	return;
+}
+
 /**
  * scsi_target_reap - check to see if target is in use and destroy if not
  *
@@ -411,19 +440,18 @@ static struct scsi_target *scsi_alloc_ta
  */
 void scsi_target_reap(struct scsi_target *starget)
 {
-	struct Scsi_Host *shost = dev_to_shost(starget->dev.parent);
-	unsigned long flags;
-	spin_lock_irqsave(shost->host_lock, flags);
+	struct work_queue_wrapper *wqw = 
+		kzalloc(sizeof(struct work_queue_wrapper), GFP_ATOMIC);
 
-	if (--starget->reap_ref == 0 && list_empty(&starget->devices)) {
-		list_del_init(&starget->siblings);
-		spin_unlock_irqrestore(shost->host_lock, flags);
-		device_del(&starget->dev);
-		transport_unregister_device(&starget->dev);
-		put_device(&starget->dev);
+	if (!wqw) {
+		starget_printk(KERN_ERR, starget,
+			       "Failed to allocate memory in scsi_reap_target()\n");
 		return;
 	}
-	spin_unlock_irqrestore(shost->host_lock, flags);
+
+	INIT_WORK(&wqw->work, scsi_target_reap_work, wqw);
+	wqw->starget = starget;
+	schedule_work(&wqw->work);
 }
 
 /**

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Thu, Dec 15, 2005 at 08:01:43PM -0800, James Bottomley wrote:
> On Thu, 2005-12-15 at 14:09 -0500, Ryan Richter wrote:
> > On Mon, Dec 12, 2005 at 12:24:42PM -0600, James Bottomley wrote:
> > > I'll find a fix for the real problem, but this patch isn't the cause.
> > 
> > Is the patch set you posted yesterday supposed to fix this?  If so, is
> > it available in patch form anywhere?
> 
> No, I've been too busin integrating other people's patches to work on
> ones of my own.  Try this.

Thanks for the patch, I'm testing it now.  If this survives the next few
days I'll leave it running while I'm away, otherwise testing will have
to wait until the new year.

Cheers,
-ryan

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Thu, Dec 15, 2005 at 08:01:43PM -0800, James Bottomley wrote:
> On Thu, 2005-12-15 at 14:09 -0500, Ryan Richter wrote:
> > On Mon, Dec 12, 2005 at 12:24:42PM -0600, James Bottomley wrote:
> > > I'll find a fix for the real problem, but this patch isn't the cause.
> > 
> > Is the patch set you posted yesterday supposed to fix this?  If so, is
> > it available in patch form anywhere?
> 
> No, I've been too busin integrating other people's patches to work on
> ones of my own.  Try this.

It was looking good, but...


                   Bad page state at free_hot_cold_page (in process 'taper', page ffff8100040e22e8)
flags:0x010000000000000c mapping:ffff810102bba238 mapcount:2 count:0
Backtrace:

Call Trace:<ffffffff80158a94>{bad_page+116} <ffffffff801595df>{free_hot_cold_page+111}
       <ffffffff80160f2e>{__page_cache_release+158} <ffffffff802b8e05>{sgl_unmap_user_pages+53}
       <ffffffff802b477b>{release_buffering+27} <ffffffff802b4e31>{st_write+1697}
       <ffffffff80179e06>{vfs_write+198} <ffffffff80179f63>{sys_write+83}
       <ffffffff8010db6a>{system_call+126}
Trying to fix it up, but a reboot is needed
Bad page state at free_hot_cold_page (in process 'taper', page ffff8100040e2320)
flags:0x010000000000000c mapping:ffff810102bba238 mapcount:2 count:0
Backtrace:

Call Trace:<ffffffff80158a94>{bad_page+116} <ffffffff801595df>{free_hot_cold_page+111}
       <ffffffff80160f2e>{__page_cache_release+158} <ffffffff802b8e05>{sgl_unmap_user_pages+53}
       <ffffffff802b477b>{release_buffering+27} <ffffffff802b4e31>{st_write+1697}
       <ffffffff80179e06>{vfs_write+198} <ffffffff80179f63>{sys_write+83}
       <ffffffff8010db6a>{system_call+126}
Trying to fix it up, but a reboot is needed
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at mm/swap.c:49
invalid operand: 0000 [1] SMP
CPU 1
Modules linked in: bonding
Pid: 2202, comm: taper Tainted: G    B 2.6.15-rc5 #2
RIP: 0010:[<ffffffff80160b63>] <ffffffff80160b63>{put_page+99}
RSP: 0018:ffff810105be9e08  EFLAGS: 00010256
RAX: 0000000000000000 RBX: 0000000000000007 RCX: ffff8100040e22e8
RDX: ffff8100040e22e8 RSI: 0000000000000001 RDI: ffff8100040e22e8
RBP: 0000000000000008 R08: ffff810105be8000 R09: 0000000000000000
R10: 0000000000008000 R11: 0000000000000200 R12: 0000000000000000
R13: ffff8100f6f9e068 R14: 0000000000008000 R15: ffff810004857510
FS:  00002aaaab53d880(0000) GS:ffffffff804ec880(0000) knlGS:00000000555bc920
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002aaaaab5ffff CR3: 0000000105bdd000 CR4: 00000000000006e0
Process taper (pid: 2202, threadinfo ffff810105be8000, task ffff810169339800)
Stack: 0000000000000000 ffffffff802b8e05 ffff8100f6f1fd48 ffff8100f6f9e000
       0000000000000040 0000000000008000 ffff810004857400 ffffffff802b477b
       ffff8100f6f9e000 ffffffff802b4e31
Call Trace:<ffffffff802b8e05>{sgl_unmap_user_pages+53} <ffffffff802b477b>{release_buffering+27}
       <ffffffff802b4e31>{st_write+1697} <ffffffff80179e06>{vfs_write+198}
       <ffffffff80179f63>{sys_write+83} <ffffffff8010db6a>{system_call+126}


Code: 0f 0b 68 ce e4 3a 80 c2 31 00 66 66 90 f0 83 42 08 ff 0f 98
RIP <ffffffff80160b63>{put_page+99} RSP <ffff810105be9e08>
 ----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at mm/rmap.c:486
invalid operand: 0000 [2] SMP
CPU 1
Modules linked in: bonding
Pid: 2202, comm: taper Tainted: G    B 2.6.15-rc5 #2
RIP: 0010:[<ffffffff8016e363>] <ffffffff8016e363>{page_remove_rmap+19}
RSP: 0018:ffff810105be9a80  EFLAGS: 00010286
RAX: 00000000ffffffff RBX: ffff810105be37f0 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 00002aaaaaafe000 RDI: ffff8100040e22e8
RBP: 00002aaaaaafe000 R08: 80000000df77b067 R09: 000000000000001e
R10: 00000000fffffffa R11: 0000000000000001 R12: ffff810105be9ba8
R13: ffff8100040e22e8 R14: ffff810101c25480 R15: 80000000df77b067
FS:  00002aaaab53d880(0000) GS:ffffffff804ec880(0000) knlGS:00000000555bc920
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002aaaaab5ffff CR3: 0000000105bdd000 CR4: 00000000000006e0
Process taper (pid: 2202, threadinfo ffff810105be8000, task ffff810169339800)
Stack: ffffffff80165d0d ffff8101001559b8 ffffffc100000000 ffff81017f0d8e00
       00002aaaaab62000 ffff810105b9d368 00002aaaaab62000 ffff810105be2aa8
       ffff810105be9ba8 00002aaaaab62000
Call Trace:<ffffffff80165d0d>{zap_pte_range+477} <ffffffff80166027>{unmap_page_range+535}
       <ffffffff80166199>{unmap_vmas+265} <ffffffff8016bee7>{exit_mmap+119}
       <ffffffff80130e31>{mmput+49} <ffffffff8013619a>{do_exit+506}
       <ffffffff8010f731>{die+81} <ffffffff8010fa1f>{do_invalid_op+159}
       <ffffffff80160b63>{put_page+99} <ffffffff803870a0>{thread_return+0}
       <ffffffff802a85e2>{sym_setup_data_and_start+402} <ffffffff802a843b>{sym_queue_command+155}
       <ffffffff8010e8a5>{error_exit+0} <ffffffff80160b63>{put_page+99}
       <ffffffff802b8e05>{sgl_unmap_user_pages+53} <ffffffff802b477b>{release_buffering+27}
       <ffffffff802b4e31>{st_write+1697} <ffffffff80179e06>{vfs_write+198}
       <ffffffff80179f63>{sys_write+83} <ffffffff8010db6a>{system_call+126}


Code: 0f 0b 68 d0 e5 3a 80 c2 e6 01 66 66 90 48 c7 c6 ff ff ff ff
RIP <ffffffff8016e363>{page_remove_rmap+19} RSP <ffff810105be9a80>
 <1>Fixing recursive fault but reboot is needed!
Unable to handle kernel paging request at 0000000000100108 RIP:
<ffffffff80159705>{buffered_rmqueue+117}
PGD a7daa067 PUD ca442067 PMD 0
Oops: 0002 [3] SMP
CPU 1
Modules linked in: bonding
Pid: 2775, comm: dump Tainted: G    B 2.6.15-rc5 #2
RIP: 0010:[<ffffffff80159705>] <ffffffff80159705>{buffered_rmqueue+117}
RSP: 0018:ffff8100f4dedd88  EFLAGS: 00010006
RAX: ffff8100040e2348 RBX: ffff81010287a340 RCX: 0000000000200200
RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffff81000000f300
RBP: ffff81000000f300 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000002 R11: 0000000000000000 R12: ffff8100040e2320
R13: 0000000000000000 R14: 0000000000000000 R15: 00000000000200d2
FS:  00002aaaab22b090(0000) GS:ffffffff804ec880(0000) knlGS:00000000555bc920
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 00000000a7da9000 CR4: 00000000000006e0
Process dump (pid: 2775, threadinfo ffff8100f4dec000, task ffff810090a9b880)
Stack: 0000000000000282 ffff810100001c18 0000000000000044 0000000000000000
       0000000000000000 0000000000000002 00000000000200d2 ffffffff801599b9
       ffff810100001c10 ffff810100001c10
Call Trace:<ffffffff801599b9>{get_page_from_freelist+137} <ffffffff80159a40>{__alloc_pages+80}
       <ffffffff801868a2>{pipe_writev+626} <ffffffff80186b7a>{pipe_write+26}
       <ffffffff80179e06>{vfs_write+198} <ffffffff80179f63>{sys_write+83}
       <ffffffff8010db6a>{system_call+126}

Code: 48 89 4a 08 48 89 11 48 c7 40 08 00 02 20 00 48 c7 00 00 01
RIP <ffffffff80159705>{buffered_rmqueue+117} RSP <ffff8100f4dedd88>
CR2: 0000000000100108
 <0>Bad page state at prep_new_page (in process 'dumper', page ffff8100040e22e8)
flags:0x010000000000001c mapping:0000000000000000 mapcount:-1 count:0

-ryan

Re: Fw: crash on x86_64 - mm related?

[Kai Makisara]

On Mon, 26 Dec 2005, Ryan Richter wrote:

> On Thu, Dec 15, 2005 at 08:01:43PM -0800, James Bottomley wrote:
> > On Thu, 2005-12-15 at 14:09 -0500, Ryan Richter wrote:
> > > On Mon, Dec 12, 2005 at 12:24:42PM -0600, James Bottomley wrote:
> > > > I'll find a fix for the real problem, but this patch isn't the cause.
> > > 
> > > Is the patch set you posted yesterday supposed to fix this?  If so, is
> > > it available in patch form anywhere?
> > 
> > No, I've been too busin integrating other people's patches to work on
> > ones of my own.  Try this.
> 
> It was looking good, but...
> 
> 
>                    Bad page state at free_hot_cold_page (in process 'taper', page ffff8100040e22e8)
> flags:0x010000000000000c mapping:ffff810102bba238 mapcount:2 count:0
> Backtrace:
> 
Looks familiar ;-(

I don't have any new ideas but I will tell you what I have tried. In order 
to get more information about what is happening, I inserted the patch at 
the end of this message to my kernel. The purpose of the patch was to 
print something about the page mappings (prt_pages) and to catch possible 
double freeing from st earlier (clear page pointers).

Running dump gave me the following output:

st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81001e0e7698 mapcount:2 count:4 pfn:173706
 1: flags:0x010000000000006c mapping:ffff81001e0e7698 mapcount:2 count:4 pfn:67369
 2: flags:0x010000000000006c mapping:ffff81001e0e7698 mapcount:2 count:4 pfn:208590
 3: flags:0x010000000000006c mapping:ffff81001e0e7698 mapcount:2 count:4 pfn:201889
 4: flags:0x010000000000006c mapping:ffff81001e0e7698 mapcount:2 count:4 pfn:172689
 5: flags:0x010000000000006c mapping:ffff81001e0e7698 mapcount:2 count:4 pfn:172132
 6: flags:0x010000000000006c mapping:ffff81001e0e7698 mapcount:2 count:4 pfn:189032
 7: flags:0x010000000000006c mapping:ffff81001e0e7698 mapcount:2 count:4 pfn:188229

This was repeated 37 times during the 11.5 GB dump. This shows that the 
taper process is using the same buffer the whole time to write the 32 kB 
blocks to tape. mapcount is 2 as in your error condition but the use count 
is 4.

If nobody has better ideas, you could maybe try this.

Kai

--- linux-2.6.15-rc7/drivers/scsi/st.c	2005-12-25 10:39:09.000000000 +0200
+++ linux-2.6.15-rc7-k0/drivers/scsi/st.c	2005-12-27 18:05:31.000000000 +0200
@@ -4498,22 +4498,44 @@ static int sgl_map_user_pages(struct sca
 	return res;
 }
 
+static void prt_pages(struct scatterlist *sgl, const unsigned int nr_pages, char *s)
+{
+	int i;
+
+	printk("st: page attributes %s page_release %d\n", s, nr_pages);
+	for (i=0; i < nr_pages; i++) {
+		struct page *page = sgl[i].page;
+		printk("%2d: flags:0x%0*lx mapping:%p mapcount:%d count:%d pfn:%ld\n",
+		       i, (int)(2*sizeof(unsigned long)), (unsigned long)page->flags,
+		       page->mapping, page_mapcount(page), page_count(page), page_to_pfn(sgl[i].page));
+	}
+}
+
 
 /* And unmap them... */
 static int sgl_unmap_user_pages(struct scatterlist *sgl, const unsigned int nr_pages,
 				int dirtied)
 {
 	int i;
+	static int ctr;
 
+
+	if ((ctr++ % 10000) == 0)
+		prt_pages(sgl, nr_pages, "before");
 	for (i=0; i < nr_pages; i++) {
 		struct page *page = sgl[i].page;
 
+		if (!page) {
+			printk("st: trying double free: %d %d\n", i, nr_pages);
+			continue;
+		}
 		if (dirtied)
 			SetPageDirty(page);
 		/* FIXME: cache flush missing for rw==READ
 		 * FIXME: call the correct reference counting function
 		 */
 		page_cache_release(page);
+		sgl[i].page = NULL;
 	}
 
 	return 0;

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Tue, Dec 27, 2005 at 06:21:39PM +0200, Kai Makisara wrote:
> If nobody has better ideas, you could maybe try this.

Thanks, I just got back and I will boot a kernel with this patch tonight
before the backup run.

-ryan

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Tue, Dec 27, 2005 at 06:21:39PM +0200, Kai Makisara wrote:
> On Mon, 26 Dec 2005, Ryan Richter wrote:
> 
> > On Thu, Dec 15, 2005 at 08:01:43PM -0800, James Bottomley wrote:
> > > On Thu, 2005-12-15 at 14:09 -0500, Ryan Richter wrote:
> > > > On Mon, Dec 12, 2005 at 12:24:42PM -0600, James Bottomley wrote:
> > > > > I'll find a fix for the real problem, but this patch isn't the cause.
> > > > 
> > > > Is the patch set you posted yesterday supposed to fix this?  If so, is
> > > > it available in patch form anywhere?
> > > 
> > > No, I've been too busin integrating other people's patches to work on
> > > ones of my own.  Try this.
> > 
> > It was looking good, but...
> > 
> > 
> >                    Bad page state at free_hot_cold_page (in process 'taper', page ffff8100040e22e8)
> > flags:0x010000000000000c mapping:ffff810102bba238 mapcount:2 count:0
> > Backtrace:
> > 
> Looks familiar ;-(
> 
> I don't have any new ideas but I will tell you what I have tried. In order 
> to get more information about what is happening, I inserted the patch at 
> the end of this message to my kernel. The purpose of the patch was to 
> print something about the page mappings (prt_pages) and to catch possible 
> double freeing from st earlier (clear page pointers).
> 
> Running dump gave me the following output:

Here's what I got:


 st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1392956
 1: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1397945
 2: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1537473
 3: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1398161
 4: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1398261
 5: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1392778
 6: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1402858
 7: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1396799
Bad page state at free_hot_cold_page (in process 'taper', page ffff81000427ff60)
flags:0x010000000000000c mapping:ffff810102bbaaf0 mapcount:2 count:0
Backtrace:

Call Trace:<ffffffff80150234>{bad_page+116} <ffffffff80150c3f>{free_hot_cold_page+105}
       <ffffffff8028c534>{sgl_unmap_user_pages+124} <ffffffff8028826d>{release_buffering+27}
       <ffffffff802888f9>{st_write+1670} <ffffffff8016d9bc>{vfs_write+173}
       <ffffffff8016dac8>{sys_write+69} <ffffffff8010d762>{system_call+126}

Trying to fix it up, but a reboot is needed
Bad page state at free_hot_cold_page (in process 'taper', page ffff810003ed9290)
flags:0x010000000000000c mapping:ffff810102bbaaf0 mapcount:2 count:0
Backtrace:

Call Trace:<ffffffff80150234>{bad_page+116} <ffffffff80150c3f>{free_hot_cold_page+105}
       <ffffffff8028c534>{sgl_unmap_user_pages+124} <ffffffff8028826d>{release_buffering+27}
       <ffffffff802888f9>{st_write+1670} <ffffffff8016d9bc>{vfs_write+173}
       <ffffffff8016dac8>{sys_write+69} <ffffffff8010d762>{system_call+126}

Trying to fix it up, but a reboot is needed
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at mm/swap.c:49
invalid operand: 0000 [1] SMP
CPU 1
Modules linked in: bonding
Pid: 2892, comm: taper Tainted: G    B 2.6.15 #1
RIP: 0010:[<ffffffff8015751c>] <ffffffff8015751c>{put_page+96}
RSP: 0018:ffff81017a247e18  EFLAGS: 00010256
RAX: 0000000000000000 RBX: 00000000000000c0 RCX: ffff81000427ff60
RDX: ffff81000427ff60 RSI: 0000000000000001 RDI: ffff81000427ff60
RBP: 0000000000000006 R08: ffff81017a246000 R09: 0000000000000001
R10: ffff8100f6f31aa0 R11: 0000000000000046 R12: 0000000000000008
R13: ffff8100f6f9e068 R14: 0000000000000000 R15: 0000000000008000
FS:  00002aaaab53d880(0000) GS:ffffffff804a9880(0000) knlGS:00000000556b6920
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002aaaaab5ffff CR3: 00000001722be000 CR4: 00000000000006e0
Process taper (pid: 2892, threadinfo ffff81017a246000, task ffff81017e63a140)
Stack: 010000000000000c ffffffff8028c534 0000000000008000 0000000000008000
       ffff8100f6f9e000 ffff810004840200 0000000000008000 0000000000000040
       0000000000008000 ffffffff8028826d
Call Trace:<ffffffff8028c534>{sgl_unmap_user_pages+124} <ffffffff8028826d>{release_buffering+27}
       <ffffffff802888f9>{st_write+1670} <ffffffff8016d9bc>{vfs_write+173}
       <ffffffff8016dac8>{sys_write+69} <ffffffff8010d762>{system_call+126}


Code: 0f 0b 68 ae b1 36 80 c2 31 00 f0 83 42 08 ff 0f 98 c0 84 c0
RIP <ffffffff8015751c>{put_page+96} RSP <ffff81017a247e18>
 ----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at mm/rmap.c:486
invalid operand: 0000 [2] SMP
CPU 1
Modules linked in: bonding
Pid: 2892, comm: taper Tainted: G    B 2.6.15 #1
RIP: 0010:[<ffffffff80163736>] <ffffffff80163736>{page_remove_rmap+19}
RSP: 0018:ffff81017a247aa0  EFLAGS: 00010286
RAX: 00000000ffffffff RBX: ffff81000427ff60 RCX: 0000000000000020
RDX: 80000000e6db4067 RSI: 80000000e6db4067 RDI: ffff81000427ff60
RBP: 80000000e6db4067 R08: 0000000000000020 R09: 00002aaaaaafe000
R10: 00000000000e6db4 R11: 0000000000000000 R12: ffff810101c25480
R13: ffff8101722a07f0 R14: 00002aaaaaafe000 R15: 0000000000000000
FS:  00002aaaab53d880(0000) GS:ffffffff804a9880(0000) knlGS:00000000556b6920
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002aaaaab5ffff CR3: 00000001722be000 CR4: 00000000000006e0
Process taper (pid: 2892, threadinfo ffff81017a246000, task ffff81017e63a140)
Stack: ffffffff8015be23 ffff81017a247ae8 ffff81010190d310 ffffffc100000000
       ffff81017f078180 ffff81017a247bb8 00002aaaaab62000 ffff81017d039368
       00002aaaaab62000 ffff810172296aa8
Call Trace:<ffffffff8015be23>{zap_pte_range+464} <ffffffff8015c0ff>{unmap_page_range+476}
       <ffffffff8015c24e>{unmap_vmas+238} <ffffffff8016167b>{exit_mmap+114}
       <ffffffff8012d638>{mmput+34} <ffffffff8013214a>{do_exit+489}
       <ffffffff8010f203>{die_nmi+0} <ffffffff8010f587>{do_invalid_op+145}
       <ffffffff8015751c>{put_page+96} <ffffffff803448db>{thread_return+0}
       <ffffffff8010e49d>{error_exit+0} <ffffffff8015751c>{put_page+96}
       <ffffffff8028c534>{sgl_unmap_user_pages+124} <ffffffff8028826d>{release_buffering+27}
       <ffffffff802888f9>{st_write+1670} <ffffffff8016d9bc>{vfs_write+173}
       <ffffffff8016dac8>{sys_write+69} <ffffffff8010d762>{system_call+126}


Code: 0f 0b 68 b0 b2 36 80 c2 e6 01 48 83 ce ff bf 20 00 00 00 e9
RIP <ffffffff80163736>{page_remove_rmap+19} RSP <ffff81017a247aa0>
 <1>Fixing recursive fault but reboot is needed!
Unable to handle kernel paging request at 0000000000100108 RIP:
<ffffffff80150d53>{buffered_rmqueue+120}
PGD 170d24067 PUD 1551d3067 PMD 0
Oops: 0002 [3] SMP
CPU 1
Modules linked in: bonding
Pid: 2898, comm: dumper Tainted: G    B 2.6.15 #1
RIP: 0010:[<ffffffff80150d53>] <ffffffff80150d53>{buffered_rmqueue+120}
RSP: 0018:ffff8100bcc17a98  EFLAGS: 00010002
RAX: ffff810003ed92b8 RBX: ffff81010287a340 RCX: 0000000000200200
RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffff81000000f300
RBP: ffff81000000f300 R08: 0000000000000000 R09: 000000000000095b
R10: 0000000000000000 R11: 0000000000000002 R12: 0000000000000000
R13: 0000000000000000 R14: ffff810003ed9290 R15: 00000000000200d2
FS:  00002aaaab53d8e0(0000) GS:ffffffff804a9880(0000) knlGS:00000000556b6920
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 000000017d27f000 CR4: 00000000000006e0
Process dumper (pid: 2898, threadinfo ffff8100bcc16000, task ffff8100bd0d4140)
Stack: ffff81011ae74810 000000001ae74810 0000000000000282 ffff810100001c18
       0000000000000044 0000000000000000 0000000000000000 ffff810100001c10
       0000000000000002 ffffffff80150fe3
Call Trace:<ffffffff80150fe3>{get_page_from_freelist+130} <ffffffff80151067>{__alloc_pages+86}
       <ffffffff8014e7f9>{generic_file_buffered_write+402}
       <ffffffff80134113>{current_fs_time+97} <ffffffff8014ef95>{__generic_file_aio_write_nolock+891}
       <ffffffff802ea7fb>{sock_common_recvmsg+45} <ffffffff802e7724>{sock_aio_read+252}
       <ffffffff8014f1ff>{generic_file_aio_write+105} <ffffffff801a2979>{ext3_file_write+22}
       <ffffffff8016d8d5>{do_sync_write+202} <ffffffff8017e9e8>{__pollwait+0}
       <ffffffff80142cca>{autoremove_wake_function+0} <ffffffff8017f22a>{sys_select+952}
       <ffffffff8016d9bc>{vfs_write+173} <ffffffff8016dac8>{sys_write+69}
       <ffffffff8010d762>{system_call+126}

Code: 48 89 4a 08 48 89 11 48 c7 40 08 00 02 20 00 48 c7 00 00 01
RIP <ffffffff80150d53>{buffered_rmqueue+120} RSP <ffff8100bcc17a98>
CR2: 0000000000100108
 <1>Unable to handle kernel paging request at 0000000000100108 RIP:
<ffffffff80150d53>{buffered_rmqueue+120}
PGD 170d24067 PUD 1551d3067 PMD 0
Oops: 0002 [4] SMP
CPU 1
Modules linked in: bonding
Pid: 2898, comm: dumper Tainted: G    B 2.6.15 #1
RIP: 0010:[<ffffffff80150d53>] <ffffffff80150d53>{buffered_rmqueue+120}
RSP: 0018:ffff81010289bca8  EFLAGS: 00010002
RAX: ffff810003ed92b8 RBX: ffff81010287a340 RCX: 0000000000200200
RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffff81000000f300
RBP: ffff81000000f300 R08: 0000000000000000 R09: 000000000000095b
R10: 0000000000000000 R11: 0000000000000002 R12: 0000000000000000
R13: 0000000000000000 R14: ffff810003ed9290 R15: 0000000000020020
FS:  00002aaaab53d8e0(0000) GS:ffffffff804a9880(0000) knlGS:00000000556b6920
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 000000017d27f000 CR4: 00000000000006e0
Process dumper (pid: 2898, threadinfo ffff8100bcc16000, task ffff8100bd0d4140)
Stack: ffff8100e78d55f8 0000000000000000 0000000000000082 ffff810100000c08
       0000000000000044 0000000000000000 0000000000000000 ffff810100000c00
       0000000000000002 ffffffff80150fe3
Call Trace: <IRQ> <ffffffff80150fe3>{get_page_from_freelist+130}
       <ffffffff80151067>{__alloc_pages+86} <ffffffff801545b1>{kmem_getpages+88}
       <ffffffff80155789>{cache_grow+195} <ffffffff801559d0>{cache_alloc_refill+408}
       <ffffffff80155feb>{__kmalloc+100} <ffffffff802eb11b>{__alloc_skb+83}
       <ffffffff802663a6>{tg3_alloc_rx_skb+186} <ffffffff80266630>{tg3_rx+338}
       <ffffffff80266998>{tg3_poll+135} <ffffffff802f0a85>{net_rx_action+129}
       <ffffffff801342b8>{__do_softirq+80} <ffffffff8010eae3>{call_softirq+31}
       <ffffffff80110384>{do_softirq+47} <ffffffff8010e34a>{apic_timer_interrupt+98}
        <EOI> <ffffffff8034566b>{__down_read+147} <ffffffff803455ea>{__down_read+18}
       <ffffffff8012d6f3>{mm_release+30} <ffffffff801316a0>{exit_mm+43}
       <ffffffff8013214a>{do_exit+489} <ffffffff8011d3c6>{do_page_fault+1215}
       <ffffffff801affef>{do_get_write_access+1277} <ffffffff8010e49d>{error_exit+0}
       <ffffffff80150d53>{buffered_rmqueue+120} <ffffffff80150fe3>{get_page_from_freelist+130}
       <ffffffff80151067>{__alloc_pages+86} <ffffffff8014e7f9>{generic_file_buffered_write+402}
       <ffffffff80134113>{current_fs_time+97} <ffffffff8014ef95>{__generic_file_aio_write_nolock+891}
       <ffffffff802ea7fb>{sock_common_recvmsg+45} <ffffffff802e7724>{sock_aio_read+252}
       <ffffffff8014f1ff>{generic_file_aio_write+105} <ffffffff801a2979>{ext3_file_write+22}
       <ffffffff8016d8d5>{do_sync_write+202} <ffffffff8017e9e8>{__pollwait+0}
       <ffffffff80142cca>{autoremove_wake_function+0} <ffffffff8017f22a>{sys_select+952}
       <ffffffff8016d9bc>{vfs_write+173} <ffffffff8016dac8>{sys_write+69}
       <ffffffff8010d762>{system_call+126}

Code: 48 89 4a 08 48 89 11 48 c7 40 08 00 02 20 00 48 c7 00 00 01
RIP <ffffffff80150d53>{buffered_rmqueue+120} RSP <ffff81010289bca8>
CR2: 0000000000100108
 <0>Kernel panic - not syncing: Aiee, killing interrupt handler!


-ryan

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Tue, Dec 27, 2005 at 06:21:39PM +0200, Kai Makisara wrote:
> Running dump gave me the following output:

In case it's interesting, here's what I got from flushing the unfinished
backup to tape:

st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff8100f6eb4520 mapcount:2 count:4 pfn:981656
 1: flags:0x010000000000006c mapping:ffff8100f6eb4520 mapcount:2 count:4 pfn:981657
 2: flags:0x010000000000006c mapping:ffff8100f6eb4520 mapcount:2 count:4 pfn:981658
 3: flags:0x010000000000006c mapping:ffff8100f6eb4520 mapcount:2 count:4 pfn:981659
 4: flags:0x010000000000006c mapping:ffff8100f6eb4520 mapcount:2 count:4 pfn:981660
 5: flags:0x010000000000006c mapping:ffff8100f6eb4520 mapcount:2 count:4 pfn:981661
 6: flags:0x010000000000006c mapping:ffff8100f6eb4520 mapcount:2 count:4 pfn:981662
 7: flags:0x010000000000006c mapping:ffff8100f6eb4520 mapcount:2 count:4 pfn:981663

No crash this time.

-ryan

Re: Fw: crash on x86_64 - mm related?

[Kai Makisara]

On Wed, 4 Jan 2006, Ryan Richter wrote:

> On Tue, Dec 27, 2005 at 06:21:39PM +0200, Kai Makisara wrote:
...
> > I don't have any new ideas but I will tell you what I have tried. In order 
> > to get more information about what is happening, I inserted the patch at 
> > the end of this message to my kernel. The purpose of the patch was to 
> > print something about the page mappings (prt_pages) and to catch possible 
> > double freeing from st earlier (clear page pointers).
> > 
> > Running dump gave me the following output:
> 
> Here's what I got:
> 
> 
>  st: page attributes before page_release 8
>  0: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1392956
>  1: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1397945
>  2: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1537473
>  3: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1398161
>  4: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1398261
>  5: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1392778
>  6: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1402858
>  7: flags:0x060000000000006c mapping:ffff810102bba238 mapcount:2 count:4 pfn:1396799

This looks similar to my output except that the page flags are 
0x060000000000006c in your case whereas I have 0x010000000000006c.

I have run amanda with a patch modified so that it prints results for 32 
consecutive writes each 10000 writes (if ((ctr++ % 10000) < 32)). This 
showed that taper (at least in my case) is actually using a buffer of 20 x 
32 kB = 640 kB. In all writes the mapping, counts, and flags were the 
same. Instead of remapping a single 32 kB buffer, st is cyclically mapping 
20 32 kB buffers.

> Bad page state at free_hot_cold_page (in process 'taper', page ffff81000427ff60)
> flags:0x010000000000000c mapping:ffff810102bbaaf0 mapcount:2 count:0

The mapping and flags have changed. The flags have lost one bit at the 
high end and the PG_lru and PG_active flags.

The patch sets the page pointer to NULL after page_cache_release() is 
called in st. This means that st is not calling page_cache_release() more 
than once for each page returned by get_user_pages(). The page state 
seems to get corrupted somewhere.

I have to continue thinking.

-- 
Kai

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Wed, Jan 04, 2006 at 11:48:52PM +0200, Kai Makisara wrote:
> This looks similar to my output except that the page flags are 
> 0x060000000000006c in your case whereas I have 0x010000000000006c.

Well, here's another.  It seems to happen much more often with 2.6.15.


 st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff8100f6eb4520 mapcount:2 count:4 pfn:981656
 1: flags:0x010000000000006c mapping:ffff8100f6eb4520 mapcount:2 count:4 pfn:981657
 2: flags:0x010000000000006c mapping:ffff8100f6eb4520 mapcount:2 count:4 pfn:981658
 3: flags:0x010000000000006c mapping:ffff8100f6eb4520 mapcount:2 count:4 pfn:981659
 4: flags:0x010000000000006c mapping:ffff8100f6eb4520 mapcount:2 count:4 pfn:981660
 5: flags:0x010000000000006c mapping:ffff8100f6eb4520 mapcount:2 count:4 pfn:981661
 6: flags:0x010000000000006c mapping:ffff8100f6eb4520 mapcount:2 count:4 pfn:981662
 7: flags:0x010000000000006c mapping:ffff8100f6eb4520 mapcount:2 count:4 pfn:981663
Bad page state at free_hot_cold_page (in process 'taper', page ffff810004422e88)
flags:0x010000000000000c mapping:ffff81017513fe58 mapcount:2 count:0
Backtrace:

Call Trace:<ffffffff80150234>{bad_page+116} <ffffffff80150c3f>{free_hot_cold_page+105}
       <ffffffff8028c534>{sgl_unmap_user_pages+124} <ffffffff8028826d>{release_buffering+27}
       <ffffffff802888f9>{st_write+1670} <ffffffff8016d9bc>{vfs_write+173}
       <ffffffff8016dac8>{sys_write+69} <ffffffff8010d762>{system_call+126}

Trying to fix it up, but a reboot is needed
Bad page state at free_hot_cold_page (in process 'taper', page ffff8100019b4b30)
flags:0x010000000000000c mapping:ffff81017513fe58 mapcount:2 count:0
Backtrace:

Call Trace:<ffffffff80150234>{bad_page+116} <ffffffff80150c3f>{free_hot_cold_page+105}
       <ffffffff8028c534>{sgl_unmap_user_pages+124} <ffffffff8028826d>{release_buffering+27}
       <ffffffff802888f9>{st_write+1670} <ffffffff8016d9bc>{vfs_write+173}
       <ffffffff8016dac8>{sys_write+69} <ffffffff8010d762>{system_call+126}

Trying to fix it up, but a reboot is needed
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at mm/swap.c:49
invalid operand: 0000 [1] SMP
CPU 0
Modules linked in: bonding
Pid: 6392, comm: taper Tainted: G    B 2.6.15 #1
RIP: 0010:[<ffffffff8015751c>] <ffffffff8015751c>{put_page+96}
RSP: 0018:ffff81017514be18  EFLAGS: 00010256
RAX: 0000000000000000 RBX: 00000000000000c0 RCX: ffff810004422e88
RDX: ffff810004422e88 RSI: 0000000000000001 RDI: ffff810004422e88
RBP: 0000000000000006 R08: ffff81017514a000 R09: 0000000000000001
R10: ffff8100049237f8 R11: 0000000000000046 R12: 0000000000000008
R13: ffff8100f6f84068 R14: 0000000000000000 R15: 0000000000008000
FS:  00002aaaab53d880(0000) GS:ffffffff804a9800(0000) knlGS:00000000555bc920
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002aaaaab5ffff CR3: 0000000175146000 CR4: 00000000000006e0
Process taper (pid: 6392, threadinfo ffff81017514a000, task ffff81017da40300)
Stack: 010000000000000c ffffffff8028c534 0000000000008000 0000000000008000
       ffff8100f6f84000 ffff81000493e400 0000000000008000 0000000000000040
       0000000000008000 ffffffff8028826d
Call Trace:<ffffffff8028c534>{sgl_unmap_user_pages+124} <ffffffff8028826d>{release_buffering+27}
       <ffffffff802888f9>{st_write+1670} <ffffffff8016d9bc>{vfs_write+173}
       <ffffffff8016dac8>{sys_write+69} <ffffffff8010d762>{system_call+126}


Code: 0f 0b 68 ae b1 36 80 c2 31 00 f0 83 42 08 ff 0f 98 c0 84 c0
RIP <ffffffff8015751c>{put_page+96} RSP <ffff81017514be18>
 ----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at mm/rmap.c:486
invalid operand: 0000 [2] SMP
CPU 0
Modules linked in: bonding
Pid: 6392, comm: taper Tainted: G    B 2.6.15 #1
RIP: 0010:[<ffffffff80163736>] <ffffffff80163736>{page_remove_rmap+19}
RSP: 0018:ffff81017514baa0  EFLAGS: 00010286
RAX: 00000000ffffffff RBX: ffff810004422e88 RCX: 0000000000000020
RDX: 80000000ee567067 RSI: 80000000ee567067 RDI: ffff810004422e88
RBP: 80000000ee567067 R08: 0000000000000020 R09: 00002aaaaaafe000
R10: 00000000000ee567 R11: 0000000000000000 R12: ffff81000c002280
R13: ffff8100edd037f0 R14: 00002aaaaaafe000 R15: 0000000000000000
FS:  00002aaaab53d880(0000) GS:ffffffff804a9800(0000) knlGS:00000000555bc920
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002aaaaab5ffff CR3: 0000000175146000 CR4: 00000000000006e0
Process taper (pid: 6392, threadinfo ffff81017514a000, task ffff81017da40300)
Stack: ffffffff8015be23 ffff81017514bae8 ffff8100044058b8 ffffffc100000000
       ffff81017fb0cc40 ffff81017514bbb8 00002aaaaab62000 ffff81017da75700
       00002aaaaab62000 ffff8100edd02aa8
Call Trace:<ffffffff8015be23>{zap_pte_range+464} <ffffffff8015c0ff>{unmap_page_range+476}
       <ffffffff8015c24e>{unmap_vmas+238} <ffffffff8016167b>{exit_mmap+114}
       <ffffffff8012d638>{mmput+34} <ffffffff8013214a>{do_exit+489}
       <ffffffff8010f203>{die_nmi+0} <ffffffff8010f587>{do_invalid_op+145}
       <ffffffff8015751c>{put_page+96} <ffffffff803448db>{thread_return+0}
       <ffffffff8010e49d>{error_exit+0} <ffffffff8015751c>{put_page+96}
       <ffffffff8028c534>{sgl_unmap_user_pages+124} <ffffffff8028826d>{release_buffering+27}
       <ffffffff802888f9>{st_write+1670} <ffffffff8016d9bc>{vfs_write+173}
       <ffffffff8016dac8>{sys_write+69} <ffffffff8010d762>{system_call+126}


Code: 0f 0b 68 b0 b2 36 80 c2 e6 01 48 83 ce ff bf 20 00 00 00 e9
RIP <ffffffff80163736>{page_remove_rmap+19} RSP <ffff81017514baa0>
 <1>Fixing recursive fault but reboot is needed!
Bad page state at prep_new_page (in process 'gzip', page ffff810004422e88)
flags:0x010000000000001c mapping:0000000000000000 mapcount:-1 count:0
Backtrace:

Call Trace:<ffffffff80150234>{bad_page+116} <ffffffff801506eb>{prep_new_page+70}
       <ffffffff80150e12>{buffered_rmqueue+311} <ffffffff80150fe3>{get_page_from_freelist+130}
       <ffffffff80151067>{__alloc_pages+86} <ffffffff8015df88>{do_anonymous_page+69}
       <ffffffff8015e6f9>{__handle_mm_fault+396} <ffffffff8011d10f>{do_page_fault+520}
       <ffffffff80142cca>{autoremove_wake_function+0} <ffffffff8019722f>{dnotify_parent+28}
       <ffffffff8016d7dc>{vfs_read+212} <ffffffff8010e49d>{error_exit+0}

Trying to fix it up, but a reboot is needed
Bad page state at prep_new_page (in process 'dumper', page ffff8100019b4b30)
flags:0x0100000000000064 mapping:ffff81017de5eaf9 mapcount:1 count:1
Backtrace:

Call Trace:<ffffffff80150234>{bad_page+116} <ffffffff801506eb>{prep_new_page+70}
       <ffffffff80150e12>{buffered_rmqueue+311} <ffffffff80150fe3>{get_page_from_freelist+130}
       <ffffffff80151067>{__alloc_pages+86} <ffffffff8014e7f9>{generic_file_buffered_write+402}
       <ffffffff80134113>{current_fs_time+97} <ffffffff8014ef95>{__generic_file_aio_write_nolock+891}
       <ffffffff802ea7fb>{sock_common_recvmsg+45} <ffffffff802e7724>{sock_aio_read+252}
       <ffffffff8014f1ff>{generic_file_aio_write+105} <ffffffff801a2979>{ext3_file_write+22}
       <ffffffff8016d8d5>{do_sync_write+202} <ffffffff8017e9e8>{__pollwait+0}
       <ffffffff80142cca>{autoremove_wake_function+0} <ffffffff8017f22a>{sys_select+952}
       <ffffffff8016d9bc>{vfs_write+173} <0>Bad page state at prep_new_page (in process 'dumper', page ffff810001a24af8)
flags:0x0100000000000064 mapping:ffff81017de5eaf9 mapcount:1 count:1
Backtrace:

Call Trace: <IRQ> <ffffffff80150234>{bad_page+116} <ffffffff801506eb>{prep_new_page+70}
       <ffffffff80150e12>{buffered_rmqueue+311} <ffffffff80150fe3>{get_page_from_freelist+130}
       <ffffffff80151067>{__alloc_pages+86} <ffffffff801545b1>{kmem_getpages+88}
       <ffffffff80155789>{cache_grow+195} <ffffffff801559d0>{cache_alloc_refill+408}
       <ffffffff80155feb>{__kmalloc+100} <ffffffff802eb11b>{__alloc_skb+83}
       <ffffffff802663a6>{tg3_alloc_rx_skb+186} <ffffffff80266630>{tg3_rx+338}
       <ffffffff80266998>{tg3_poll+135} <ffffffff802f0a85>{net_rx_action+129}
       <ffffffff801342b8>{__do_softirq+80} <ffffffff8010eae3>{call_softirq+31}
       <ffffffff80110384>{do_softirq+47} <ffffffff8010e34a>{apic_timer_interrupt+98}
        <EOI> <ffffffff8013008d>{release_console_sem+76} <ffffffff8012ff8d>{vprintk+543}
       <ffffffff8012ae27>{__wake_up+54} <ffffffff8016d9bc>{vfs_write+173}
       <ffffffff8012fd66>{printk+141} <ffffffff80149172>{module_text_address+48}
       <ffffffff8010edd8>{show_trace+430} <ffffffff8010eecf>{dump_stack+9}
       <ffffffff80150234>{bad_page+116} <ffffffff801506eb>{prep_new_page+70}
       <ffffffff80150e12>{buffered_rmqueue+311} <ffffffff80150fe3>{get_page_from_freelist+130}
       <ffffffff80151067>{__alloc_pages+86} <ffffffff8014e7f9>{generic_file_buffered_write+402}
       <ffffffff80134113>{current_fs_time+97} <ffffffff8014ef95>{__generic_file_aio_write_nolock+891}
       <ffffffff802ea7fb>{sock_common_recvmsg+45} <ffffffff802e7724>{sock_aio_read+252}
       <ffffffff8014f1ff>{generic_file_aio_write+105} <ffffffff801a2979>{ext3_file_write+22}
       <ffffffff8016d8d5>{do_sync_write+202} <ffffffff8017e9e8>{__pollwait+0}
       <ffffffff80142cca>{autoremove_wake_function+0} <ffffffff8017f22a>{sys_select+952}
       <ffffffff8016d9bc>{vfs_write+173} <ffffffff8016dac8>{sys_write+69}
       <ffffffff8010d762>{system_call+126}
Trying to fix it up, but a reboot is needed
<ffffffff8016dac8>{sys_write+69}
       <ffffffff8010d762>{system_call+126}
Trying to fix it up, but a reboot is needed
Bad page state at prep_new_page (in process 'dumper', page ffff81000449ab00)
flags:0x0100000000000064 mapping:ffff81017de5eaf9 mapcount:1 count:1
Backtrace:

Call Trace:<ffffffff80150234>{bad_page+116} <ffffffff801506eb>{prep_new_page+70}

-ryan

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Wed, Jan 04, 2006 at 11:48:52PM +0200, Kai Makisara wrote:
> > Here's what I got:

Another one.  I can't keep running this kernel - nearly all of our
backup tapes are erased now.  If a drive were to fail today, we would
lose hundreds of GB of irreplacible data.  I'm going back to 2.6.11.3
until we have a full set of backups again.


 st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff810102bbaaf0 mapcount:2 count:4 pfn:1553908
 1: flags:0x060000000000006c mapping:ffff810102bbaaf0 mapcount:2 count:4 pfn:1553846
 2: flags:0x060000000000006c mapping:ffff810102bbaaf0 mapcount:2 count:4 pfn:1553907
 3: flags:0x060000000000006c mapping:ffff810102bbaaf0 mapcount:2 count:4 pfn:1554431
 4: flags:0x060000000000006c mapping:ffff810102bbaaf0 mapcount:2 count:4 pfn:1553947
 5: flags:0x060000000000006c mapping:ffff810102bbaaf0 mapcount:2 count:4 pfn:1553919
 6: flags:0x060000000000006c mapping:ffff810102bbaaf0 mapcount:2 count:4 pfn:1553940
 7: flags:0x060000000000006c mapping:ffff810102bbaaf0 mapcount:2 count:4 pfn:1553918
Bad page state at free_hot_cold_page (in process 'taper', page ffff81000455f7b0)
flags:0x010000000000000c mapping:ffff810102bbaaf0 mapcount:2 count:0
Backtrace:

Call Trace:<ffffffff80150234>{bad_page+116} <ffffffff80150c3f>{free_hot_cold_page+105}
       <ffffffff8028c534>{sgl_unmap_user_pages+124} <ffffffff8028826d>{release_buffering+27}
       <ffffffff802888f9>{st_write+1670} <ffffffff8016d9bc>{vfs_write+173}
       <ffffffff8016dac8>{sys_write+69} <ffffffff8010d762>{system_call+126}

Trying to fix it up, but a reboot is needed
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at mm/swap.c:49
invalid operand: 0000 [1] SMP
CPU 0
Modules linked in: bonding
Pid: 2166, comm: taper Tainted: G    B 2.6.15 #1
RIP: 0010:[<ffffffff8015751c>] <ffffffff8015751c>{put_page+96}
RSP: 0018:ffff81017b6bfe18  EFLAGS: 00010256
RAX: 0000000000000000 RBX: 00000000000000e0 RCX: ffff81000455f7b0
RDX: ffff81000455f7b0 RSI: 0000000000000001 RDI: ffff81000455f7b0
RBP: 0000000000000007 R08: ffff81017b6be000 R09: 0000000000000001
R10: ffff810004878aa0 R11: 0000000000000046 R12: 0000000000000008
R13: ffff8100f6f9e068 R14: 0000000000000000 R15: 0000000000008000
FS:  00002aaaab53d880(0000) GS:ffffffff804a9800(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002aaaaab5ffff CR3: 000000017b06e000 CR4: 00000000000006e0
Process taper (pid: 2166, threadinfo ffff81017b6be000, task ffff81017b6b4700)
Stack: 010000000000000c ffffffff8028c534 0000000000008000 0000000000008000
       ffff8100f6f9e000 ffff81000c06e200 0000000000008000 0000000000000040
       0000000000008000 ffffffff8028826d
Call Trace:<ffffffff8028c534>{sgl_unmap_user_pages+124} <ffffffff8028826d>{release_buffering+27}
       <ffffffff802888f9>{st_write+1670} <ffffffff8016d9bc>{vfs_write+173}
       <ffffffff8016dac8>{sys_write+69} <ffffffff8010d762>{system_call+126}


Code: 0f 0b 68 ae b1 36 80 c2 31 00 f0 83 42 08 ff 0f 98 c0 84 c0
RIP <ffffffff8015751c>{put_page+96} RSP <ffff81017b6bfe18>
 ----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at mm/swap.c:215
invalid operand: 0000 [2] SMP
CPU 0
Modules linked in: bonding
Pid: 2166, comm: taper Tainted: G    B 2.6.15 #1
RIP: 0010:[<ffffffff801578d9>] <ffffffff801578d9>{release_pages+79}
RSP: 0018:ffff81017b6bfb08  EFLAGS: 00010256
RAX: 0000000000000000 RBX: ffff81000455f7b0 RCX: ffff81000000f518
RDX: ffff81000455f7b0 RSI: 0000000000000006 RDI: ffff81000c006dd0
RBP: 0000000000000000 R08: ffffffff803cfa68 R09: 0000000000000286
R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000005
R13: 0000000000000006 R14: ffff81000c006dd0 R15: 0000000000008000
FS:  00002aaaab53d880(0000) GS:ffffffff804a9800(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002aaaaab5ffff CR3: 000000017b06e000 CR4: 00000000000006e0
Process taper (pid: 2166, threadinfo ffff81017b6be000, task ffff81017b6b4700)
Stack: 0000000000000000 0000000000000000 0000000000008000 ffff81017b6bfd68
       0000000000000000 0000000000000000 ffffffff80362e14 000000000000000f
       0000000000000000 0000000000000000
Call Trace:<ffffffff8012fd66>{printk+141} <ffffffff80157b74>{__pagevec_lru_add+195}
       <ffffffff801577b9>{lru_add_drain+32} <ffffffff8016162a>{exit_mmap+33}
       <ffffffff8012d638>{mmput+34} <ffffffff8013214a>{do_exit+489}
       <ffffffff8010f203>{die_nmi+0} <ffffffff8010f587>{do_invalid_op+145}
       <ffffffff8015751c>{put_page+96} <ffffffff803448db>{thread_return+0}
       <ffffffff8010e49d>{error_exit+0} <ffffffff8015751c>{put_page+96}
       <ffffffff8028c534>{sgl_unmap_user_pages+124} <ffffffff8028826d>{release_buffering+27}
       <ffffffff802888f9>{st_write+1670} <ffffffff8016d9bc>{vfs_write+173}
       <ffffffff8016dac8>{sys_write+69} <ffffffff8010d762>{system_call+126}


Code: 0f 0b 68 ae b1 36 80 c2 d7 00 f0 83 43 08 ff 0f 98 c0 84 c0
RIP <ffffffff801578d9>{release_pages+79} RSP <ffff81017b6bfb08>
 <1>Fixing recursive fault but reboot is needed!
Bad page state at prep_new_page (in process 'nfsd', page ffff81000455f7b0)
flags:0x010000000000002c mapping:0000000000000000 mapcount:0 count:0
Backtrace:

Call Trace:<ffffffff80150234>{bad_page+116} <ffffffff801506eb>{prep_new_page+70}
       <ffffffff80150e12>{buffered_rmqueue+311} <ffffffff80150fe3>{get_page_from_freelist+130}
       <ffffffff80151067>{__alloc_pages+86} <ffffffff801545b1>{kmem_getpages+88}
       <ffffffff80155789>{cache_grow+195} <ffffffff801559d0>{cache_alloc_refill+408}
       <ffffffff80155d56>{kmem_cache_alloc+51} <ffffffff80341e2a>{cache_make_upcall+85}
       <ffffffff80340c99>{cache_check+183} <ffffffff801defc6>{exp_find_key+118}
       <ffffffff8012ae27>{__wake_up+54} <ffffffff80341f00>{cache_make_upcall+299}
       <ffffffff801d9d9f>{fh_verify+401} <ffffffff801e224c>{nfsd3_proc_getattr+129}
       <ffffffff801d844b>{nfsd_dispatch+226} <ffffffff8033be8c>{svc_process+1003}
       <ffffffff8012ad76>{default_wake_function+0} <ffffffff801d8018>{nfsd+0}
       <ffffffff801d81e1>{nfsd+457} <ffffffff8010e652>{child_rip+8}
       <ffffffff801d8018>{nfsd+0} <ffffffff801d8018>{nfsd+0}
       <ffffffff8010e64a>{child_rip+0}
Trying to fix it up, but a reboot is needed
Bad page state at prep_new_page (in process 'nfsd', page ffff81000455b840)
flags:0x010000000000002c mapping:ffff810102bbaaf0 mapcount:2 count:3
Backtrace:

Call Trace:<ffffffff80150234>{bad_page+116} <ffffffff801506eb>{prep_new_page+70}
       <ffffffff80150e12>{buffered_rmqueue+311} <ffffffff80150fe3>{get_page_from_freelist+130}
       <ffffffff801d8018>{nfsd+0} <ffffffff80151067>{__alloc_pages+86}
       <ffffffff801d8018>{nfsd+0} <ffffffff8033d9d8>{svc_recv+291}
       <ffffffff8012ad76>{default_wake_function+0} <ffffffff8012ad76>{default_wake_function+0}
       <ffffffff801d8018>{nfsd+0} <ffffffff801d8125>{nfsd+269}
       <ffffffff8010e652>{child_rip+8} <ffffffff801d8018>{nfsd+0}
       <ffffffff801d8018>{nfsd+0} <ffffffff8010e64a>{child_rip+0}

Trying to fix it up, but a reboot is needed
Bad page state at prep_new_page (in process 'zsh', page ffff81000455f3c0)
flags:0x010000000000002c mapping:ffff810102bbaaf0 mapcount:2 count:3
Backtrace:

Call Trace:<ffffffff80150234>{bad_page+116} <ffffffff801506eb>{prep_new_page+70}
       <ffffffff80150e12>{buffered_rmqueue+311} <ffffffff80150fe3>{get_page_from_freelist+130}
       <ffffffff80151067>{__alloc_pages+86} <ffffffff8015d50c>{do_wp_page+400}
       <ffffffff8015e7df>{__handle_mm_fault+626} <ffffffff8011d10f>{do_page_fault+520}
       <ffffffff80205410>{__put_user_4+32} <ffffffff8010e49d>{error_exit+0}

Trying to fix it up, but a reboot is needed
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at mm/swap.c:303
invalid operand: 0000 [3] SMP
CPU 0
Modules linked in: bonding
Pid: 171, comm: pdflush Tainted: G    B 2.6.15 #1
RIP: 0010:[<ffffffff80157b10>] <ffffffff80157b10>{__pagevec_lru_add+95}
RSP: 0018:ffff8100f6fbdba8  EFLAGS: 00010086
RAX: 00000000ffffffff RBX: ffff81000000f300 RCX: 000000000000000f
RDX: ffffffff804ebe60 RSI: ffff81000000f000 RDI: ffff81000000f500
RBP: ffff81000c006dc0 R08: 0000000000000000 R09: ffffffff801a4f68
R10: 0000000000000001 R11: ffff81017fc63c00 R12: ffff810004474798
R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000001
FS:  00002aaaaae00640(0000) GS:ffffffff804a9800(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007fffffd68f60 CR3: 000000017b223000 CR4: 00000000000006e0
Process pdflush (pid: 171, threadinfo ffff8100f6fbc000, task ffff8100f6fbaf80)
Stack: ffff81008bb1af60 ffff8100f6fbdc48 ffff8100f6fbde78 ffff81017d204350
       ffff8100f6f21978 ffffffff801577b9 ffff81017fc63c00 ffffffff80157a00
       ffff810101b3b930 ffffffff8018e5a1
Call Trace:<ffffffff801577b9>{lru_add_drain+32} <ffffffff80157a00>{__pagevec_release+9}
       <ffffffff8018e5a1>{mpage_writepages+663} <ffffffff801a4f81>{ext3_ordered_writepage+0}
       <ffffffff801534f1>{mapping_tagged+58} <ffffffff8018cd2e>{__sync_single_inode+108}
       <ffffffff8018d027>{__writeback_single_inode+353} <ffffffff8013817f>{process_timeout+0}
       <ffffffff802e1a2d>{dm_table_any_congested+18} <ffffffff802e1a2d>{dm_table_any_congested+18}
       <ffffffff8018d204>{sync_sb_inodes+468} <ffffffff8014280f>{keventd_create_kthread+0}
       <ffffffff8018d36e>{writeback_inodes+133} <ffffffff801536c8>{pdflush+0}
       <ffffffff80152ce5>{background_writeout+102} <ffffffff80153621>{__pdflush+289}
       <ffffffff80153702>{pdflush+58} <ffffffff80152c7f>{background_writeout+0}
       <ffffffff801427e6>{kthread+129} <ffffffff8010e652>{child_rip+8}
       <ffffffff8014280f>{keventd_create_kthread+0} <ffffffff80142765>{kthread+0}
       <ffffffff8010e64a>{child_rip+0}

Code: 0f 0b 68 ae b1 36 80 c2 2f 01 48 8b 93 18 02 00 00 49 8d 44
RIP <ffffffff80157b10>{__pagevec_lru_add+95} RSP <ffff8100f6fbdba8>
 NMI Watchdog detected LOCKUP on CPU 0
CPU 0
Modules linked in: bonding
Pid: 1619, comm: irqbalance Tainted: G    B 2.6.15 #1
RIP: 0010:[<ffffffff80345d2f>] <ffffffff80345d2f>{.text.lock.spinlock+32}
RSP: 0018:ffff8100f648de90  EFLAGS: 00000086
RAX: ffff81000000f300 RBX: ffff81000000f300 RCX: 00002aaaaaac0000
RDX: ffffffff804ebe60 RSI: ffff8100f09339e0 RDI: ffff81000000f500
RBP: ffff81000c006dc0 R08: 00002aaaaaac1000 R09: 00000000ffffffff
R10: ffff81000c399da8 R11: ffff81000c399088 R12: ffff810004474798
R13: 0000000000000000 R14: 00002aaaaaac1000 R15: 00002aaaaaac0000
FS:  00002aaaaae00640(0000) GS:ffffffff804a9800(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002aaaaaac0000 CR3: 00000000f6486000 CR4: 00000000000006e0
Process irqbalance (pid: 1619, threadinfo ffff8100f648c000, task ffff810004aac340)
Stack: ffffffff80157b03 ffff81008bb1af60 ffffffff804e7320 ffff8100f65c6580
       ffff81000c399d78 ffff81000c399088 ffffffff801577b9 ffff81000c399088
       ffffffff80160eb9 ffff8100f09339e0
Call Trace:<ffffffff80157b03>{__pagevec_lru_add+82} <ffffffff801577b9>{lru_add_drain+32}
       <ffffffff80160eb9>{unmap_region+65} <ffffffff801612d9>{do_munmap+387}
       <ffffffff80161329>{sys_munmap+57} <ffffffff8010d762>{system_call+126}


Code: f3 90 83 3f 00 7e f9 e9 66 fe ff ff f3 90 83 3f 00 7e f9 e9
console shuts up ...
 <0>Kernel panic - not syncing: Aiee, killing interrupt handler!

-ryan

Re: Fw: crash on x86_64 - mm related?

[Linus Torvalds]

On Thu, 5 Jan 2006, Ryan Richter wrote:
> 
> Another one.  I can't keep running this kernel - nearly all of our
> backup tapes are erased now.  If a drive were to fail today, we would
> lose hundreds of GB of irreplacible data.  I'm going back to 2.6.11.3
> until we have a full set of backups again.

Yeah, don't trash your backups.

If/when you try something again, how about these two trivial one-liners?

I'm not 100% sure the mapcount sanity check is strictly speaking right (no 
locking between mapcount/pagecount comparison), but the page count really 
should never fall below the mapcount, so aside from races that I don't 
think can be triggered in practice, this might be very useful to find 
where those pesky page counts suddenly disappear..

Right now we get the oops "too late" - something has decremented the page
count way too far, but we don't know what it was, and the actual function 
that triggers it _seems_ to be harmless.

The other part of the patch is to clear "page" when it's being free'd, in 
case somebody tries to free the same thing twice. I don't see how that 
could happen either, but...

The patch is untested in every way. No guarantees.

		Linus

---
diff --git a/drivers/scsi/st.c b/drivers/scsi/st.c
index c4aade8..18e60e1 100644
--- a/drivers/scsi/st.c
+++ b/drivers/scsi/st.c
@@ -4481,6 +4481,7 @@ static int sgl_unmap_user_pages(struct s
 	for (i=0; i < nr_pages; i++) {
 		struct page *page = sgl[i].page;
 
+		sgl[i].page = NULL;
 		if (dirtied)
 			SetPageDirty(page);
 		/* FIXME: cache flush missing for rw==READ
diff --git a/include/linux/mm.h b/include/linux/mm.h
index a06a84d..daf504d 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -299,6 +299,7 @@ struct page {
 #define put_page_testzero(p)				\
 	({						\
 		BUG_ON(page_count(p) == 0);		\
+		BUG_ON(page_count(p) <= page_mapcount(p));	\
 		atomic_add_negative(-1, &(p)->_count);	\
 	})
 

Re: Fw: crash on x86_64 - mm related?

[Kai Makisara]

On Thu, 5 Jan 2006, Ryan Richter wrote:

> On Wed, Jan 04, 2006 at 11:48:52PM +0200, Kai Makisara wrote:
> > > Here's what I got:
> 
> Another one.  I can't keep running this kernel - nearly all of our
> backup tapes are erased now.  If a drive were to fail today, we would
> lose hundreds of GB of irreplacible data.  I'm going back to 2.6.11.3
> until we have a full set of backups again.
> 
Yes, don't risk the backups. If you don't want to change the kernel, you 
can load the st module with the option try_direct_io=0. If you do this, st 
will use the driver's buffer for reading and writing and you don't see 
this problem.

If you later have an opportunity to try the patch Linus suggested, it 
might provide interesting information. However, if all goes well, the st 
driver will not be using these mapping functions in 2.6.16 any more (see 
Mike Christie's changes in "[GIT PATCH] SCSI update for 2.6.15" sent to 
linux-scsi by James Bottomley yesterday). I am not sure this fixes your 
problem but it certainly changes the code that is being debugged.

-- 
Kai

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Thu, Jan 05, 2006 at 01:18:57PM -0800, Linus Torvalds wrote:

OK, I can trash a few tapes again now.

> ---
> diff --git a/drivers/scsi/st.c b/drivers/scsi/st.c
> index c4aade8..18e60e1 100644
> --- a/drivers/scsi/st.c
> +++ b/drivers/scsi/st.c
> @@ -4481,6 +4481,7 @@ static int sgl_unmap_user_pages(struct s
>  	for (i=0; i < nr_pages; i++) {
>  		struct page *page = sgl[i].page;
>  
> +		sgl[i].page = NULL;
>  		if (dirtied)
>  			SetPageDirty(page);
>  		/* FIXME: cache flush missing for rw==READ

This hunk fails on 2.6.15+Kai's patch.  There's already a
sgl[i].page = NULL;
in there - here's what my copy of the function looks like:

/* And unmap them... */
static int sgl_unmap_user_pages(struct scatterlist *sgl, const unsigned int nr_pages,
                                int dirtied)
{
        int i;
        static int ctr;


        if ((ctr++ % 10000) == 0)
                prt_pages(sgl, nr_pages, "before");
        for (i=0; i < nr_pages; i++) {
                struct page *page = sgl[i].page;

                if (!page) {
                        printk("st: trying double free: %d %d\n", i, nr_pages);
                        continue;
                }
                if (dirtied)
                        SetPageDirty(page);
                /* FIXME: cache flush missing for rw==READ
                 * FIXME: call the correct reference counting function
                 */
                page_cache_release(page);
                sgl[i].page = NULL;
        }

        return 0;
}

I left it like that.

> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index a06a84d..daf504d 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -299,6 +299,7 @@ struct page {
>  #define put_page_testzero(p)				\
>  	({						\
>  		BUG_ON(page_count(p) == 0);		\
> +		BUG_ON(page_count(p) <= page_mapcount(p));	\
>  		atomic_add_negative(-1, &(p)->_count);	\
>  	})
>

The first backup run with this patch didn't crash - that's the first
time with 2.6.15.  I wonder if it's a coincidence.  Nothing else has
changed, AFAICS, but the kernel output is different (or there's more of
it):

st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933526
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933496
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933498
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933499
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933500
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933501
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933525
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933523
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007909
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963403
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:992653
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963620
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:985187
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:966121
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1007992
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933594
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933522
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963590
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:931906
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933763
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963777
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933718
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963619
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:207313
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933522
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963590
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:931906
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933763
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963777
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933718
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963619
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:207313
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933606
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933323
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963420
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:959854
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962968
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933602
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933600
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933339
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933606
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933323
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963420
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:959854
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962968
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933602
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933600
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933339
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933606
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933323
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963420
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:959854
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962968
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933602
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933600
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933339
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933606
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933323
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963420
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:959854
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962968
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933602
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933600
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933339
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933606
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933323
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963420
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:959854
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962968
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933602
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933600
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933339
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933606
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933323
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963420
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:959854
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962968
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933602
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933600
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933339
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933606
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933323
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963420
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:959854
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962968
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933602
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933600
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933339
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933606
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933323
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963420
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:959854
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962968
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933602
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933600
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933339
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933606
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933323
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963420
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:959854
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962968
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933602
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933600
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933339
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933606
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933323
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963420
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:959854
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962968
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933602
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933600
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933339
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933606
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933323
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963420
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:959854
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962968
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933602
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933600
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933339
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933606
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933323
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:963420
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:959854
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962968
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933602
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933600
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:933339
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1009784
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:186937
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1008366
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:968112
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962986
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932668
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932637
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962435
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1009784
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:186937
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1008366
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:968112
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962986
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932668
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932637
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962435
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1009784
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:186937
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1008366
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:968112
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962986
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932668
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932637
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962435
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1009784
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:186937
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1008366
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:968112
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962986
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932668
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932637
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962435
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1009784
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:186937
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1008366
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:968112
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962986
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932668
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932637
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962435
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1009784
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:186937
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1008366
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:968112
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962986
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932668
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932637
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962435
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1009784
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:186937
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1008366
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:968112
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962986
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932668
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932637
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962435
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1009784
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:186937
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1008366
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:968112
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962986
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932668
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932637
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962435
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1009784
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:186937
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1008366
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:968112
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962986
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932668
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932637
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962435
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1009784
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:186937
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1008366
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:968112
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962986
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932668
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932637
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962435
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1009784
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:186937
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1008366
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:968112
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962986
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932668
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932637
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962435
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1009784
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:186937
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1008366
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:968112
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962986
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932668
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932637
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962435
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1009784
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:186937
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1008366
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:968112
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962986
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932668
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932637
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962435
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1009784
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:186937
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1008366
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:968112
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962986
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932668
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932637
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962435
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1009784
 1: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:186937
 2: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:1008366
 3: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:968112
 4: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962986
 5: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932668
 6: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:932637
 7: flags:0x010000000000006c mapping:ffff81000c354278 mapcount:2 count:4 pfn:962435

There are 48 of those, not quite the same as the number of filesystems
written to tape (58).

I'll run a few more tonight.

-ryan  

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Thu, Jan 05, 2006 at 01:18:57PM -0800, Linus Torvalds wrote:
> 
> 
> On Thu, 5 Jan 2006, Ryan Richter wrote:
> > 
> > Another one.  I can't keep running this kernel - nearly all of our
> > backup tapes are erased now.  If a drive were to fail today, we would
> > lose hundreds of GB of irreplacible data.  I'm going back to 2.6.11.3
> > until we have a full set of backups again.
> 
> Yeah, don't trash your backups.
> 
> If/when you try something again, how about these two trivial one-liners?

Here's what I get:


 st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:59059
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931269
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931299
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931297
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931270
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931271
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931293
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931292
st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1363318
 1: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1316938
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1094504
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1206029
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1567257
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1161075
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1097099
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1532602
st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1363318
 1: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1316938
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1094504
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1206029
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1567257
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1161075
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1097099
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1532602
st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1363318
 1: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1316938
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1094504
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1206029
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1567257
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1161075
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1097099
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1532602
st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1363318
 1: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1316938
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1094504
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1206029
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1567257
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1161075
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1097099
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1532602
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930553
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930554
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930555
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930556
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930557
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930558
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930559
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930368
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1004900
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:929996
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1487794
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1482747
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1490011
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559513
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559672
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1531152
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1004900
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:929996
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1487794
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1482747
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1490011
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559513
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559672
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1531152
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1004900
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:929996
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1487794
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1482747
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1490011
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559513
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559672
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1531152
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1004900
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:929996
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1487794
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1482747
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1490011
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559513
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559672
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1531152
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:990157
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931663
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931296
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931434
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931288
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931289
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930934
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930935
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:990157
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931663
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931296
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931434
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931288
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931289
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930934
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930935
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:990157
 1: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931663
 2: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931296
 3: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931434
 4: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931288
 5: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:931289
 6: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930934
 7: flags:0x010000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:930935
st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559143
 1: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1531129
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1554810
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1151109
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1102551
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1096291
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1364840
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1271161
st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559143
 1: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1531129
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1554810
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1151109
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1102551
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1096291
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1364840
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1271161
st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559143
 1: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1531129
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1554810
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1151109
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1102551
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1096291
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1364840
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1271161
st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559143
 1: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1531129
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1554810
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1151109
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1102551
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1096291
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1364840
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1271161
st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559143
 1: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1531129
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1554810
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1151109
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1102551
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1096291
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1364840
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1271161
st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559143
 1: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1531129
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1554810
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1151109
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1102551
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1096291
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1364840
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1271161
st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559143
 1: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1531129
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1554810
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1151109
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1102551
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1096291
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1364840
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1271161
st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559143
 1: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1531129
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1554810
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1151109
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1102551
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1096291
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1364840
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1271161
st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559143
 1: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1531129
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1554810
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1151109
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1102551
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1096291
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1364840
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1271161
st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559143
 1: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1531129
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1554810
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1151109
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1102551
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1096291
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1364840
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1271161
st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559143
 1: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1531129
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1554810
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1151109
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1102551
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1096291
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1364840
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1271161
st: page attributes before page_release 8
 0: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1559143
 1: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1531129
 2: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1554810
 3: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1151109
 4: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1102551
 5: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1096291
 6: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1364840
 7: flags:0x060000000000006c mapping:ffff81000c34db30 mapcount:2 count:4 pfn:1271161
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:578294
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91168
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:93080
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:738043
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91194
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:622028
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:145120
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219869
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:578294
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91168
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:93080
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:738043
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91194
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:622028
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:145120
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219869
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:578294
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91168
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:93080
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:738043
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91194
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:622028
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:145120
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219869
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:578294
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91168
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:93080
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:738043
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91194
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:622028
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:145120
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219869
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:578294
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91168
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:93080
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:738043
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91194
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:622028
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:145120
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219869
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:578294
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91168
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:93080
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:738043
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91194
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:622028
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:145120
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219869
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:578294
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91168
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:93080
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:738043
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91194
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:622028
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:145120
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219869
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:578294
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91168
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:93080
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:738043
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91194
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:622028
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:145120
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219869
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:578294
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91168
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:93080
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:738043
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91194
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:622028
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:145120
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219869
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:578294
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91168
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:93080
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:738043
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91194
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:622028
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:145120
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219869
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:578294
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91168
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:93080
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:738043
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91194
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:622028
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:145120
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219869
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:578294
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91168
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:93080
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:738043
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91194
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:622028
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:145120
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219869
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:578294
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91168
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:93080
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:738043
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91194
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:622028
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:145120
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219869
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:578294
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91168
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:93080
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:738043
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91194
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:622028
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:145120
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219869
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:578294
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91168
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:93080
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:738043
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:91194
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:622028
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:145120
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219869
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:585737
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:748879
 2: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1104048
 3: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1370594
 4: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1065893
 5: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1425661
 6: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1525898
 7: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1295844
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:585737
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:748879
 2: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1104048
 3: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1370594
 4: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1065893
 5: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1425661
 6: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1525898
 7: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1295844
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:585737
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:748879
 2: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1104048
 3: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1370594
 4: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1065893
 5: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1425661
 6: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1525898
 7: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1295844
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:585737
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:748879
 2: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1104048
 3: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1370594
 4: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1065893
 5: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1425661
 6: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1525898
 7: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1295844
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:585737
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:748879
 2: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1104048
 3: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1370594
 4: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1065893
 5: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1425661
 6: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1525898
 7: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1295844
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:585737
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:748879
 2: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1104048
 3: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1370594
 4: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1065893
 5: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1425661
 6: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1525898
 7: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1295844
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:585737
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:748879
 2: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1104048
 3: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1370594
 4: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1065893
 5: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1425661
 6: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1525898
 7: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1295844
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:585737
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:748879
 2: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1104048
 3: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1370594
 4: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1065893
 5: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1425661
 6: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1525898
 7: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1295844
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:585737
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:748879
 2: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1104048
 3: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1370594
 4: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1065893
 5: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1425661
 6: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1525898
 7: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1295844
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:585737
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:748879
 2: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1104048
 3: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1370594
 4: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1065893
 5: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1425661
 6: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1525898
 7: flags:0x060000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:1295844
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:677238
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:628761
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:427387
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:564253
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:378426
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:268240
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:29652
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:875401
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:677238
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:628761
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:427387
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:564253
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:378426
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:268240
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:29652
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:875401
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:677238
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:628761
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:427387
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:564253
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:378426
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:268240
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:29652
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:875401
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:677238
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:628761
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:427387
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:564253
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:378426
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:268240
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:29652
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:875401
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:677238
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:628761
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:427387
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:564253
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:378426
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:268240
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:29652
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:875401
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:677238
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:628761
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:427387
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:564253
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:378426
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:268240
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:29652
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:875401
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:677238
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:628761
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:427387
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:564253
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:378426
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:268240
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:29652
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:875401
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:677238
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:628761
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:427387
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:564253
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:378426
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:268240
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:29652
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:875401
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:677238
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:628761
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:427387
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:564253
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:378426
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:268240
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:29652
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:875401
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:677238
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:628761
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:427387
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:564253
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:378426
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:268240
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:29652
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:875401
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:227805
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219103
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:126968
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:601873
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:393638
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:96658
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:930644
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:694582
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:227805
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219103
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:126968
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:601873
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:393638
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:96658
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:930644
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:694582
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:227805
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219103
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:126968
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:601873
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:393638
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:96658
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:930644
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:694582
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:227805
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219103
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:126968
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:601873
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:393638
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:96658
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:930644
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:694582
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:227805
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219103
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:126968
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:601873
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:393638
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:96658
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:930644
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:694582
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:227805
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219103
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:126968
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:601873
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:393638
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:96658
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:930644
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:694582
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:227805
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219103
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:126968
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:601873
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:393638
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:96658
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:930644
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:694582
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:227805
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219103
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:126968
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:601873
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:393638
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:96658
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:930644
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:694582
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:227805
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219103
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:126968
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:601873
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:393638
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:96658
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:930644
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:694582
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:227805
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219103
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:126968
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:601873
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:393638
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:96658
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:930644
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:694582
st: page attributes before page_release 8
 0: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:227805
 1: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:219103
 2: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:126968
 3: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:601873
 4: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:393638
 5: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:96658
 6: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:930644
 7: flags:0x010000000000006c mapping:ffff81000c34d278 mapcount:2 count:4 pfn:694582
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at mm/swap.c:49
invalid operand: 0000 [1] SMP
CPU 0
Modules linked in: bonding
Pid: 4501, comm: taper Not tainted 2.6.15 #2
RIP: 0010:[<ffffffff80157596>] <ffffffff80157596>{put_page+178}
RSP: 0018:ffff8101453d9e18  EFLAGS: 00010246
RAX: 0000000000000002 RBX: 00000000000000e0 RCX: ffff810002f1ea10
RDX: 0000000000000002 RSI: 0000000000000008 RDI: ffff810002f1ea10
RBP: 0000000000000007 R08: ffff8101453d8000 R09: 0000000000000001
R10: ffff8100f6f3daa0 R11: 0000000000000046 R12: 0000000000000008
R13: ffff8100f6f9e068 R14: 0000000000000000 R15: 0000000000008000
FS:  00002aaaab53d880(0000) GS:ffffffff804a9800(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002aaaaaac2000 CR3: 000000014d29c000 CR4: 00000000000006e0
Process taper (pid: 4501, threadinfo ffff8101453d8000, task ffff81017d0143c0)
Stack: ffff810002f1ea10 ffffffff8028c614 0000000000008000 0000000000008000
       ffff8100f6f9e000 ffff8100f6f2a200 0000000000008000 0000000000000040
       0000000000008000 ffffffff8028834d
Call Trace:<ffffffff8028c614>{sgl_unmap_user_pages+124} <ffffffff8028834d>{release_buffering+27}
       <ffffffff802889d9>{st_write+1670} <ffffffff8016da90>{vfs_write+173}
       <ffffffff8016db9c>{sys_write+69} <ffffffff8010d762>{system_call+126}


Code: 0f 0b 68 8e b2 36 80 c2 31 00 f0 83 41 08 ff 0f 98 c0 84 c0
RIP <ffffffff80157596>{put_page+178} RSP <ffff8101453d9e18>
 <0>Bad page state at free_hot_cold_page (in process 'taper', page ffff810002f1ea10)
flags:0x010000000000001c mapping:ffff810102bba238 mapcount:0 count:0
Backtrace:

Call Trace:<ffffffff80150234>{bad_page+116} <ffffffff80150c3f>{free_hot_cold_page+105}
       <ffffffff80151393>{__pagevec_free+28} <ffffffff80157a8d>{release_pages+393}
       <ffffffff80165239>{free_pages_and_swap_cache+108} <ffffffff80161795>{exit_mmap+184}
       <ffffffff8012d638>{mmput+34} <ffffffff8013214a>{do_exit+489}
       <ffffffff80132472>{sys_exit_group+0} <ffffffff8010d762>{system_call+126}

Trying to fix it up, but a reboot is needed

-ryan

Re: Fw: crash on x86_64 - mm related?

[Linus Torvalds]

On Sun, 8 Jan 2006, Ryan Richter wrote:
>
> Kernel BUG at mm/swap.c:49

Well, it sure triggered.

> Process taper (pid: 4501, threadinfo ffff8101453d8000, task ffff81017d0143c0)
> Call Trace:<ffffffff8028c614>{sgl_unmap_user_pages+124}
>		 <ffffffff8028834d>{release_buffering+27}

and it's that same sgl_unmap_user_pages() that keeps on triggering it.

Which was not what I was hoping for. I was hoping we'd see somebody _else_ 
decrementing the page count below the map count, and get a new clue.

However, the page flags you show later on (0x1c) ended up making me take 
notice of something. That's "dirty", and maybe it's from

	if (dirtied)
		SetPageDirty(page);

in that same sgl_unmap_user_pages() routine.. And it strikes me that that 
is bogus.

Code like that should use "set_page_dirty()", which does the appropriate 
callbacks to the filesystem for that page. I wonder if the bug is simply 
because the ST code just sets the dirty bit without telling anybody else 
about it...

Gaah. Hugh, Nick?

		Linus

Re: Fw: crash on x86_64 - mm related?

[Andrew Morton]

Linus Torvalds <torvalds@osdl.org> wrote:
>
> 
> 
> On Sun, 8 Jan 2006, Ryan Richter wrote:
> >
> > Kernel BUG at mm/swap.c:49
> 
> Well, it sure triggered.
> 
> > Process taper (pid: 4501, threadinfo ffff8101453d8000, task ffff81017d0143c0)
> > Call Trace:<ffffffff8028c614>{sgl_unmap_user_pages+124}
> >		 <ffffffff8028834d>{release_buffering+27}
> 
> and it's that same sgl_unmap_user_pages() that keeps on triggering it.
> 
> Which was not what I was hoping for. I was hoping we'd see somebody _else_ 
> decrementing the page count below the map count, and get a new clue.
> 
> However, the page flags you show later on (0x1c) ended up making me take 
> notice of something. That's "dirty", and maybe it's from
> 
> 	if (dirtied)
> 		SetPageDirty(page);
> 
> in that same sgl_unmap_user_pages() routine.. And it strikes me that that 
> is bogus.
> 
> Code like that should use "set_page_dirty()", which does the appropriate 
> callbacks to the filesystem for that page. I wonder if the bug is simply 
> because the ST code just sets the dirty bit without telling anybody else 
> about it...
> 

It should be using set_page_dirty_lock().  As should st_unmap_user_pages().
 I doubt if this would explain a refcounting problem though.

Ryan, It might be worth poisoning the thing, see if the completion is being
called twice:


diff -puN drivers/scsi/st.c~a drivers/scsi/st.c
--- devel/drivers/scsi/st.c~a	2006-01-08 21:11:47.000000000 -0800
+++ devel-akpm/drivers/scsi/st.c	2006-01-08 21:12:13.000000000 -0800
@@ -4482,11 +4482,12 @@ static int sgl_unmap_user_pages(struct s
 		struct page *page = sgl[i].page;
 
 		if (dirtied)
-			SetPageDirty(page);
+			set_page_dirty_lock(page);
 		/* FIXME: cache flush missing for rw==READ
 		 * FIXME: call the correct reference counting function
 		 */
 		page_cache_release(page);
+		sgl[i].page = NULL;
 	}
 
 	return 0;
_

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Sun, Jan 08, 2006 at 09:13:21PM -0800, Andrew Morton wrote:
> It should be using set_page_dirty_lock().  As should st_unmap_user_pages().
>  I doubt if this would explain a refcounting problem though.
> 
> Ryan, It might be worth poisoning the thing, see if the completion is being
> called twice:
> 
> 
> diff -puN drivers/scsi/st.c~a drivers/scsi/st.c
> --- devel/drivers/scsi/st.c~a	2006-01-08 21:11:47.000000000 -0800
> +++ devel-akpm/drivers/scsi/st.c	2006-01-08 21:12:13.000000000 -0800
> @@ -4482,11 +4482,12 @@ static int sgl_unmap_user_pages(struct s
>  		struct page *page = sgl[i].page;
>  
>  		if (dirtied)
> -			SetPageDirty(page);
> +			set_page_dirty_lock(page);
>  		/* FIXME: cache flush missing for rw==READ
>  		 * FIXME: call the correct reference counting function
>  		 */
>  		page_cache_release(page);
> +		sgl[i].page = NULL;
>  	}
>  
>  	return 0;
> _
> 

Which version does this patch apply to?

Thanks,
-ryan

Re: Fw: crash on x86_64 - mm related?

[Andrew Morton]

Ryan Richter <ryan@tau.solarneutrino.net> wrote:
>
> On Sun, Jan 08, 2006 at 09:13:21PM -0800, Andrew Morton wrote:
> > It should be using set_page_dirty_lock().  As should st_unmap_user_pages().
> >  I doubt if this would explain a refcounting problem though.
> > 
> > Ryan, It might be worth poisoning the thing, see if the completion is being
> > called twice:
> > 
> > 
> > diff -puN drivers/scsi/st.c~a drivers/scsi/st.c
> > --- devel/drivers/scsi/st.c~a	2006-01-08 21:11:47.000000000 -0800
> > +++ devel-akpm/drivers/scsi/st.c	2006-01-08 21:12:13.000000000 -0800
> > @@ -4482,11 +4482,12 @@ static int sgl_unmap_user_pages(struct s
> >  		struct page *page = sgl[i].page;
> >  
> >  		if (dirtied)
> > -			SetPageDirty(page);
> > +			set_page_dirty_lock(page);
> >  		/* FIXME: cache flush missing for rw==READ
> >  		 * FIXME: call the correct reference counting function
> >  		 */
> >  		page_cache_release(page);
> > +		sgl[i].page = NULL;
> >  	}
> >  
> >  	return 0;
> > _
> > 
> 
> Which version does this patch apply to?
> 

Oh, I see we've already tried the poisoning thing.

So you just want to change the one line:

--- devel/drivers/scsi/st.c~a	2006-01-08 21:57:25.000000000 -0800
+++ devel-akpm/drivers/scsi/st.c	2006-01-08 21:57:38.000000000 -0800
@@ -4509,7 +4509,7 @@ static int sgl_unmap_user_pages(struct s
 		struct page *page = sgl[i].page;
 
 		if (dirtied)
-			SetPageDirty(page);
+			set_page_dirty_lock(page);
 		/* FIXME: cache flush missing for rw==READ
 		 * FIXME: call the correct reference counting function
 		 */
_

Re: Fw: crash on x86_64 - mm related?

[Hugh Dickins]

On Sun, 8 Jan 2006, Linus Torvalds wrote:
> 
> Code like that should use "set_page_dirty()", which does the appropriate 
> callbacks to the filesystem for that page. I wonder if the bug is simply 
> because the ST code just sets the dirty bit without telling anybody else 
> about it...

Yes, it should be using set_page_dirty_lock(), and that is already known
about (I have patches for this and similar sg.c, but the sg.c case is
tougher and not yet finished); but entirely irrelevant to Ryan's case.

Quite apart from the fact that he's doing backups to tape (not dirtying
the memory from this driver), you'll find that it even passes dirty 0
when reading into the memory (another bug; whereas sg.c conversely says
it's always dirtying when it isn't).  So there's no point in Ryan
fiddling with the SetPageDirty.

It's an intriguing problem because it's signature is so regular,
and I've spent many hours trying to work out how it might come about,
but unsuccessfully so far.  Your adjustment to the put_page_testzero
BUG_ON was a good idea, but it still hasn't shone a light.  I'm
keeping quiet until I find something useful to add.

Perhaps we just need a few more people to add sgl[i].page = NULL ;)

Hugh

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Mon, Jan 09, 2006 at 09:44:26AM +0000, Hugh Dickins wrote:
> On Sun, 8 Jan 2006, Linus Torvalds wrote:
> > 
> > Code like that should use "set_page_dirty()", which does the appropriate 
> > callbacks to the filesystem for that page. I wonder if the bug is simply 
> > because the ST code just sets the dirty bit without telling anybody else 
> > about it...
> 
> Yes, it should be using set_page_dirty_lock(), and that is already known
> about (I have patches for this and similar sg.c, but the sg.c case is
> tougher and not yet finished); but entirely irrelevant to Ryan's case.
> 
> Quite apart from the fact that he's doing backups to tape (not dirtying
> the memory from this driver), you'll find that it even passes dirty 0
> when reading into the memory (another bug; whereas sg.c conversely says
> it's always dirtying when it isn't).  So there's no point in Ryan
> fiddling with the SetPageDirty.

One thing I forgot to mention was that 2.6.11.3 had the problem too when
I reverted to it.  I remember now that the person who made the debian
bug report for this said it only happened with a 64-bit userspace - and
I switched from a 32- to 64-bit userspace when I did 2.6.11 -> 2.6.14
(and I'm too lazy to switch back).

To get the backups back, I just ran a recent kernel with
try_direct_io=0.  If there's nothing further for me to test at this
time, I guess I'll go back to doing that until there's something to try.
Is that OK?

-ryan

Re: Fw: crash on x86_64 - mm related?

[Hugh Dickins]

On Mon, 9 Jan 2006, Ryan Richter wrote:
> 
> One thing I forgot to mention was that 2.6.11.3 had the problem too when
> I reverted to it.  I remember now that the person who made the debian
> bug report for this said it only happened with a 64-bit userspace - and
> I switched from a 32- to 64-bit userspace when I did 2.6.11 -> 2.6.14
> (and I'm too lazy to switch back).

I remembered you reported it originally on 2.6.14.N, so I wasn't
searching amongst the 2.6.15 changes at all.  Thanks for the info
that it's at least as old as 2.6.11.3.

> To get the backups back, I just ran a recent kernel with
> try_direct_io=0.  If there's nothing further for me to test at this
> time, I guess I'll go back to doing that until there's something to try.
> Is that OK?

I think we'll allow you the luxury of making successful backups for now ;)

Thanks for all your work on this, I'm sure it's irritating to you that
we haven't found the answer yet.  I'm still clueless about it (despite
the excellent clues you've provided).  And personally I don't like
asking someone "try this, try that" until I've a pretty good hypothesis
to devise a patch to test out.  Still thinking it over.  Someone else
may have a better idea of what to try next.

Hugh

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Mon, Jan 09, 2006 at 07:31:11PM +0000, Hugh Dickins wrote:
> On Mon, 9 Jan 2006, Ryan Richter wrote:
> > To get the backups back, I just ran a recent kernel with
> > try_direct_io=0.  If there's nothing further for me to test at this
> > time, I guess I'll go back to doing that until there's something to try.
> > Is that OK?
> 
> I think we'll allow you the luxury of making successful backups for now ;)
> 
> Thanks for all your work on this, I'm sure it's irritating to you that
> we haven't found the answer yet.  I'm still clueless about it (despite
> the excellent clues you've provided).  And personally I don't like
> asking someone "try this, try that" until I've a pretty good hypothesis
> to devise a patch to test out.  Still thinking it over.  Someone else
> may have a better idea of what to try next.

The episode where I blew through half the tapes was mostly my fault.  I
can avoid that in the future while still doing destructive testing, so
you don't have to be too reluctant to give me things to test.  I just
wanted to make sure there was no further utility in getting oopses from
Linus's patch.

And try_direct_io=0 works just fine, and doesn't seem to have any
performance impact or anything, so at this point I'm not losing any
sleep.

Thanks,
-ryan

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

This machine experienced another random reboot today, nothing in the
logs or on the console etc.  This is the 3rd time now since I upgraded
from 2.6.11.3.  Is there any way to debug something like this?  I'm
fairly certain it's not hardware-related.  Might this have something to
do with the st problem?

Thanks,
-ryan

Re: Fw: crash on x86_64 - mm related?

[Hugh Dickins]

(A useless answer from me, sorry: can anyone else help?)

On Tue, 17 Jan 2006, Ryan Richter wrote:

> This machine experienced another random reboot today, nothing in the
> logs or on the console etc.  This is the 3rd time now since I upgraded
> from 2.6.11.3.  Is there any way to debug something like this?

Nasty.  I hope someone else can suggest something constructive.

> I'm fairly certain it's not hardware-related.
> Might this have something to do with the st problem?

Well, it might: I've not been able to explain that at all, so cannot
rule out a relation to your reboots.  The st problem (Bad page state,
mapcount 2 while count 0, from sgl_unmap_user_pages) ought to be a lot
easier to debug than a random reboot: but I've still no suggestion.

Hugh

Re: Fw: crash on x86_64 - mm related?

[Hugh Dickins]

On Wed, 18 Jan 2006, Hugh Dickins wrote:
> On Tue, 17 Jan 2006, Ryan Richter wrote:
> 
> > This machine experienced another random reboot today, nothing in the
> > logs or on the console etc.  This is the 3rd time now since I upgraded
> > from 2.6.11.3.  Is there any way to debug something like this?
> 
> Nasty.  I hope someone else can suggest something constructive.

Are they still happening?

> > I'm fairly certain it's not hardware-related.
> > Might this have something to do with the st problem?
> 
> Well, it might: I've not been able to explain that at all, so cannot
> rule out a relation to your reboots.

If no "Bad page state" has occurred earlier, we can now be sure that
these reboots are (unfortunately) unrelated to the st problem.

> The st problem (Bad page state,
> mapcount 2 while count 0, from sgl_unmap_user_pages) ought to be a lot
> easier to debug than a random reboot: but I've still no suggestion.

I guessed it last week, Ryan verified that booting with "iommu=nomerge"
worked around it, and then that the 2.6.15 patch below fixes it.

On some architectures (alpha, ia64, parisc, powerpc, sparc64, x86_64,
more?) in some configurations, dma_map_sg or pci_map_sg may coalesce
scatterlist entries, copying later entries down: bad news if we then
go on to use that coalesced scatterlist to free the pages afterwards -
we're quite liable to free the same page twice.

James' DMA-API.txt is perfectly clear that "The mapping process is
allowed to destroy information in the sg".  I wonder that this hasn't
been noticed before.  Perhaps we should have a debug option to destroy
everything in the scatterlist when unmapping.

The st code has moved on a little since 2.6.15, I'll send the 2.6.16-rc2
patch to Kai and lists in reply to this mail.  I've looked around for
similar culprits - fewer than I was expecting, but I'm not sure if
I've found them all:

drivers/infiniband/core/uverbs_mem.c
drivers/scsi/ipr.c
drivers/scsi/osst.c
drivers/scsi/sg.c

I'll reply with _untested_ patches to the maintainers and list for
those too.  Except for sg.c, which I'll send privately to Doug -
I've wasted a lot of time not quite understanding it, and don't
want to expose my ignorance in public.

Hugh

--- 2.6.15/drivers/scsi/st.c	2006-01-03 03:21:10.000000000 +0000
+++ linux/drivers/scsi/st.c	2006-01-29 17:21:47.000000000 +0000
@@ -188,11 +188,11 @@ static int from_buffer(struct st_buffer 
 static void move_buffer_data(struct st_buffer *, int);
 static void buf_to_sg(struct st_buffer *, unsigned int);
 
-static int st_map_user_pages(struct scatterlist *, const unsigned int, 
+static int st_map_user_pages(struct st_buffer *, const unsigned int, 
 			     unsigned long, size_t, int, unsigned long);
-static int sgl_map_user_pages(struct scatterlist *, const unsigned int, 
+static int sgl_map_user_pages(struct st_buffer *, const unsigned int, 
 			      unsigned long, size_t, int);
-static int sgl_unmap_user_pages(struct scatterlist *, const unsigned int, int);
+static int sgl_unmap_user_pages(struct st_buffer *, const unsigned int, int);
 
 static int st_probe(struct device *);
 static int st_remove(struct device *);
@@ -1402,7 +1402,7 @@ static int setup_buffering(struct scsi_t
 		i = STp->try_dio && try_wdio;
 	if (i && ((unsigned long)buf & queue_dma_alignment(
 					STp->device->request_queue)) == 0) {
-		i = st_map_user_pages(&(STbp->sg[0]), STbp->use_sg,
+		i = st_map_user_pages(STbp, STbp->use_sg,
 				      (unsigned long)buf, count, (is_read ? READ : WRITE),
 				      STp->max_pfn);
 		if (i > 0) {
@@ -1455,7 +1455,7 @@ static void release_buffering(struct scs
 
 	STbp = STp->buffer;
 	if (STbp->do_dio) {
-		sgl_unmap_user_pages(&(STbp->sg[0]), STbp->do_dio, 0);
+		sgl_unmap_user_pages(STbp, STbp->do_dio, 0);
 		STbp->do_dio = 0;
 	}
 }
@@ -4396,32 +4396,33 @@ static void do_create_class_files(struct
    - any page is above max_pfn
    (i.e., either completely successful or fails)
 */
-static int st_map_user_pages(struct scatterlist *sgl, const unsigned int max_pages, 
+static int st_map_user_pages(struct st_buffer *STbp, const unsigned int max_pages, 
 			     unsigned long uaddr, size_t count, int rw,
 			     unsigned long max_pfn)
 {
 	int i, nr_pages;
 
-	nr_pages = sgl_map_user_pages(sgl, max_pages, uaddr, count, rw);
+	nr_pages = sgl_map_user_pages(STbp, max_pages, uaddr, count, rw);
 	if (nr_pages <= 0)
 		return nr_pages;
 
 	for (i=0; i < nr_pages; i++) {
-		if (page_to_pfn(sgl[i].page) > max_pfn)
+		if (page_to_pfn(STbp->sg[i].page) > max_pfn)
 			goto out_unmap;
 	}
 	return nr_pages;
 
  out_unmap:
-	sgl_unmap_user_pages(sgl, nr_pages, 0);
+	sgl_unmap_user_pages(STbp, nr_pages, 0);
 	return 0;
 }
 
 
 /* The following functions may be useful for a larger audience. */
-static int sgl_map_user_pages(struct scatterlist *sgl, const unsigned int max_pages, 
+static int sgl_map_user_pages(struct st_buffer *STbp, const unsigned int max_pages, 
 			      unsigned long uaddr, size_t count, int rw)
 {
+	struct scatterlist *sgl = STbp->sg;
 	unsigned long end = (uaddr + count + PAGE_SIZE - 1) >> PAGE_SHIFT;
 	unsigned long start = uaddr >> PAGE_SHIFT;
 	const int nr_pages = end - start;
@@ -4485,7 +4486,7 @@ static int sgl_map_user_pages(struct sca
 		sgl[0].length = count;
 	}
 
-	kfree(pages);
+	STbp->sg_pages = pages;
 	return nr_pages;
 
  out_unmap:
@@ -4500,13 +4501,13 @@ static int sgl_map_user_pages(struct sca
 
 
 /* And unmap them... */
-static int sgl_unmap_user_pages(struct scatterlist *sgl, const unsigned int nr_pages,
+static int sgl_unmap_user_pages(struct st_buffer *STbp, const unsigned int nr_pages,
 				int dirtied)
 {
 	int i;
 
 	for (i=0; i < nr_pages; i++) {
-		struct page *page = sgl[i].page;
+		struct page *page = STbp->sg_pages[i];
 
 		if (dirtied)
 			SetPageDirty(page);
@@ -4516,5 +4517,7 @@ static int sgl_unmap_user_pages(struct s
 		page_cache_release(page);
 	}
 
+	kfree(STbp->sg_pages);
+	STbp->sg_pages = NULL;
 	return 0;
 }
--- 2.6.15/drivers/scsi/st.h	2005-10-28 01:02:08.000000000 +0100
+++ linux/drivers/scsi/st.h	2006-01-29 16:52:02.000000000 +0000
@@ -37,6 +37,7 @@ struct st_buffer {
 	unsigned short frp_segs;	/* number of buffer segments */
 	unsigned int frp_sg_current;	/* driver buffer length currently in s/g list */
 	struct st_buf_fragment *frp;	/* the allocated buffer fragment list */
+	struct page **sg_pages;		/* pages to be freed from s/g list */
 	struct scatterlist sg[1];	/* MUST BE last item */
 };
 

[PATCH] ib: don't doublefree pages from scatterlist

[Hugh Dickins]

On some architectures, mapping the scatterlist may coalesce entries:
if that coalesced list is then used for freeing the pages afterwards,
there's a danger that pages may be doubly freed (and others leaked).

Fix Infiniband's __ib_umem_release by freeing from a separate array
beyond the scatterlist: IB_UMEM_MAX_PAGE_CHUNK lowered to fit one page.

Signed-off-by: Hugh Dickins <hugh@veritas.com>
---
Warning: untested!  And please double-check the adjusted definition of
IB_UMEM_MAX_PAGE_CHUNK - the old definition was avoiding "sizeof"s, but
I don't understand why.

 drivers/infiniband/core/uverbs_mem.c |   22 ++++++++++++++++------
 include/rdma/ib_verbs.h              |    3 +--
 2 files changed, 17 insertions(+), 8 deletions(-)

--- 2.6.16-rc2/drivers/infiniband/core/uverbs_mem.c	2005-10-28 01:02:08.000000000 +0100
+++ linux/drivers/infiniband/core/uverbs_mem.c	2006-02-03 09:59:37.000000000 +0000
@@ -49,15 +49,18 @@ struct ib_umem_account_work {
 static void __ib_umem_release(struct ib_device *dev, struct ib_umem *umem, int dirty)
 {
 	struct ib_umem_chunk *chunk, *tmp;
+	struct page **sg_pages;
 	int i;
 
 	list_for_each_entry_safe(chunk, tmp, &umem->chunk_list, list) {
 		dma_unmap_sg(dev->dma_device, chunk->page_list,
 			     chunk->nents, DMA_BIDIRECTIONAL);
+		/* Scatterlist may have been coalesced: free saved pagelist */
+		sg_pages = (struct page **) (chunk->page_list + chunk->nents);
 		for (i = 0; i < chunk->nents; ++i) {
 			if (umem->writable && dirty)
-				set_page_dirty_lock(chunk->page_list[i].page);
-			put_page(chunk->page_list[i].page);
+				set_page_dirty_lock(sg_pages[i]);
+			put_page(sg_pages[i]);
 		}
 
 		kfree(chunk);
@@ -69,11 +72,13 @@ int ib_umem_get(struct ib_device *dev, s
 {
 	struct page **page_list;
 	struct ib_umem_chunk *chunk;
+	struct page **sg_pages;
 	unsigned long locked;
 	unsigned long lock_limit;
 	unsigned long cur_base;
 	unsigned long npages;
 	int ret = 0;
+	int nents;
 	int off;
 	int i;
 
@@ -121,16 +126,21 @@ int ib_umem_get(struct ib_device *dev, s
 		off = 0;
 
 		while (ret) {
-			chunk = kmalloc(sizeof *chunk + sizeof (struct scatterlist) *
-					min_t(int, ret, IB_UMEM_MAX_PAGE_CHUNK),
+			nents = min_t(int, ret, IB_UMEM_MAX_PAGE_CHUNK);
+			chunk = kmalloc(sizeof *chunk +
+					sizeof (struct scatterlist) * nents +
+					sizeof (struct page *) * nents,
 					GFP_KERNEL);
 			if (!chunk) {
 				ret = -ENOMEM;
 				goto out;
 			}
+			/* Save pages to be freed in array beyond scatterlist */
+			sg_pages = (struct page **) (chunk->page_list + nents);
 
-			chunk->nents = min_t(int, ret, IB_UMEM_MAX_PAGE_CHUNK);
+			chunk->nents = nents;
 			for (i = 0; i < chunk->nents; ++i) {
+				sg_pages[i] =
 				chunk->page_list[i].page   = page_list[i + off];
 				chunk->page_list[i].offset = 0;
 				chunk->page_list[i].length = PAGE_SIZE;
@@ -142,7 +152,7 @@ int ib_umem_get(struct ib_device *dev, s
 						 DMA_BIDIRECTIONAL);
 			if (chunk->nmap <= 0) {
 				for (i = 0; i < chunk->nents; ++i)
-					put_page(chunk->page_list[i].page);
+					put_page(sg_pages[i]);
 				kfree(chunk);
 
 				ret = -ENOMEM;
--- 2.6.16-rc2/include/rdma/ib_verbs.h	2006-02-03 09:32:50.000000000 +0000
+++ linux/include/rdma/ib_verbs.h	2006-02-03 09:59:37.000000000 +0000
@@ -696,8 +696,7 @@ struct ib_udata {
 
 #define IB_UMEM_MAX_PAGE_CHUNK						\
 	((PAGE_SIZE - offsetof(struct ib_umem_chunk, page_list)) /	\
-	 ((void *) &((struct ib_umem_chunk *) 0)->page_list[1] -	\
-	  (void *) &((struct ib_umem_chunk *) 0)->page_list[0]))
+	 (sizeof(struct scatterlist) + sizeof(struct page *)))
 
 struct ib_umem_object {
 	struct ib_uobject	uobject;

[PATCH] st: don't doublefree pages from scatterlist

[Hugh Dickins]

On some architectures, mapping the scatterlist may coalesce entries:
if that coalesced list is then used for freeing the pages afterwards,
there's a danger that pages may be doubly freed (and others leaked).

Fix SCSI Tape's sgl_unmap_user_pages by freeing from the pagelist used
in sgl_map_user_pages.  Fixes Ryan Richter's crash on x86_64, with Bad
page state mapcount 2 from sgl_unmap_user_pages, and consequent mayhem.

But it's quite confusing for st's (un)map_user_pages to be named sgl_,
with sg's named st_: while we're changing its arg, rename st's to
st_map_user_pages and st_unmap_user_pages (and do sg's separately).

Signed-off-by: Hugh Dickins <hugh@veritas.com>
---

 drivers/scsi/st.c |   20 ++++++++++++--------
 drivers/scsi/st.h |    1 +
 2 files changed, 13 insertions(+), 8 deletions(-)

--- 2.6.16-rc2/drivers/scsi/st.c	2006-02-03 09:32:25.000000000 +0000
+++ linux/drivers/scsi/st.c	2006-02-03 09:59:37.000000000 +0000
@@ -188,9 +188,9 @@ static int from_buffer(struct st_buffer 
 static void move_buffer_data(struct st_buffer *, int);
 static void buf_to_sg(struct st_buffer *, unsigned int);
 
-static int sgl_map_user_pages(struct scatterlist *, const unsigned int, 
+static int st_map_user_pages(struct st_buffer *, const unsigned int, 
 			      unsigned long, size_t, int);
-static int sgl_unmap_user_pages(struct scatterlist *, const unsigned int, int);
+static int st_unmap_user_pages(struct st_buffer *, const unsigned int, int);
 
 static int st_probe(struct device *);
 static int st_remove(struct device *);
@@ -1404,7 +1404,7 @@ static int setup_buffering(struct scsi_t
 
 	if (i && ((unsigned long)buf & queue_dma_alignment(
 					STp->device->request_queue)) == 0) {
-		i = sgl_map_user_pages(&(STbp->sg[0]), STbp->use_sg,
+		i = st_map_user_pages(STbp, STbp->use_sg,
 				      (unsigned long)buf, count, (is_read ? READ : WRITE));
 		if (i > 0) {
 			STbp->do_dio = i;
@@ -1456,7 +1456,7 @@ static void release_buffering(struct scs
 
 	STbp = STp->buffer;
 	if (STbp->do_dio) {
-		sgl_unmap_user_pages(&(STbp->sg[0]), STbp->do_dio, is_read);
+		st_unmap_user_pages(STbp, STbp->do_dio, is_read);
 		STbp->do_dio = 0;
 		STbp->sg_segs = 0;
 	}
@@ -4368,13 +4368,14 @@ static void do_create_class_files(struct
 }
 
 /* The following functions may be useful for a larger audience. */
-static int sgl_map_user_pages(struct scatterlist *sgl, const unsigned int max_pages, 
+static int st_map_user_pages(struct st_buffer *STbp, const unsigned int max_pages, 
 			      unsigned long uaddr, size_t count, int rw)
 {
 	unsigned long end = (uaddr + count + PAGE_SIZE - 1) >> PAGE_SHIFT;
 	unsigned long start = uaddr >> PAGE_SHIFT;
 	const int nr_pages = end - start;
 	int res, i, j;
+	struct scatterlist *sgl = STbp->sg;
 	struct page **pages;
 
 	/* User attempted Overflow! */
@@ -4434,7 +4435,7 @@ static int sgl_map_user_pages(struct sca
 		sgl[0].length = count;
 	}
 
-	kfree(pages);
+	STbp->sg_pages = pages;
 	return nr_pages;
 
  out_unmap:
@@ -4449,13 +4450,14 @@ static int sgl_map_user_pages(struct sca
 
 
 /* And unmap them... */
-static int sgl_unmap_user_pages(struct scatterlist *sgl, const unsigned int nr_pages,
+static int st_unmap_user_pages(struct st_buffer *STbp, const unsigned int nr_pages,
 				int dirtied)
 {
 	int i;
 
+	/* Scatterlist entries may have been coalesced: free saved pagelist */
 	for (i=0; i < nr_pages; i++) {
-		struct page *page = sgl[i].page;
+		struct page *page = STbp->sg_pages[i];
 
 		if (dirtied)
 			SetPageDirty(page);
@@ -4465,5 +4467,7 @@ static int sgl_unmap_user_pages(struct s
 		page_cache_release(page);
 	}
 
+	kfree(STbp->sg_pages);
+	STbp->sg_pages = NULL;
 	return 0;
 }
--- 2.6.16-rc2/drivers/scsi/st.h	2006-02-03 09:32:25.000000000 +0000
+++ linux/drivers/scsi/st.h	2006-02-03 09:59:37.000000000 +0000
@@ -49,6 +49,7 @@ struct st_buffer {
 	unsigned short frp_segs;	/* number of buffer segments */
 	unsigned int frp_sg_current;	/* driver buffer length currently in s/g list */
 	struct st_buf_fragment *frp;	/* the allocated buffer fragment list */
+	struct page **sg_pages;		/* pages to be freed from s/g list */
 	struct scatterlist sg[1];	/* MUST BE last item */
 };
 

[PATCH] ipr: don't doublefree pages from scatterlist

[Hugh Dickins]

On some architectures, mapping the scatterlist may coalesce entries:
if that coalesced list is then used for freeing the pages afterwards,
there's a danger that pages may be doubly freed (and others leaked).

Fix Power RAID's ipr_free_ucode_buffer by freeing from a separate array
beyond the scatterlist.

Signed-off-by: Hugh Dickins <hugh@veritas.com>
---
Warning: untested!

 drivers/scsi/ipr.c |   12 ++++++++++--
 1 files changed, 10 insertions(+), 2 deletions(-)

--- 2.6.16-rc2/drivers/scsi/ipr.c	2006-02-03 09:32:24.000000000 +0000
+++ linux/drivers/scsi/ipr.c	2006-02-03 09:59:37.000000000 +0000
@@ -2538,6 +2538,7 @@ static struct ipr_sglist *ipr_alloc_ucod
 	int sg_size, order, bsize_elem, num_elem, i, j;
 	struct ipr_sglist *sglist;
 	struct scatterlist *scatterlist;
+	struct page **sg_pages;
 	struct page *page;
 
 	/* Get the minimum size per scatter/gather element */
@@ -2557,7 +2558,8 @@ static struct ipr_sglist *ipr_alloc_ucod
 
 	/* Allocate a scatter/gather list for the DMA */
 	sglist = kzalloc(sizeof(struct ipr_sglist) +
-			 (sizeof(struct scatterlist) * (num_elem - 1)),
+			 (sizeof(struct scatterlist) * (num_elem - 1)) +
+			 (sizeof(struct page *) * num_elem),
 			 GFP_KERNEL);
 
 	if (sglist == NULL) {
@@ -2566,6 +2568,8 @@ static struct ipr_sglist *ipr_alloc_ucod
 	}
 
 	scatterlist = sglist->scatterlist;
+	/* Save pages to be freed in array beyond scatterlist */
+	sg_pages = (struct page **) (scatterlist + num_elem);
 
 	sglist->order = order;
 	sglist->num_sg = num_elem;
@@ -2584,6 +2588,7 @@ static struct ipr_sglist *ipr_alloc_ucod
 		}
 
 		scatterlist[i].page = page;
+		sg_pages[i] = page;
 	}
 
 	return sglist;
@@ -2601,10 +2606,13 @@ static struct ipr_sglist *ipr_alloc_ucod
  **/
 static void ipr_free_ucode_buffer(struct ipr_sglist *sglist)
 {
+	struct page **sg_pages;
 	int i;
 
+	/* Scatterlist entries may have been coalesced: free saved pagelist */
+	sg_pages = (struct page **) (sglist->scatterlist + sglist->num_sg);
 	for (i = 0; i < sglist->num_sg; i++)
-		__free_pages(sglist->scatterlist[i].page, sglist->order);
+		__free_pages(sg_pages[i], sglist->order);
 
 	kfree(sglist);
 }

[PATCH] osst: don't doublefree pages from scatterlist

[Hugh Dickins]

On some architectures, mapping the scatterlist may coalesce entries:
if that coalesced list is then used for freeing the pages afterwards,
there's a danger that pages may be doubly freed (and others leaked).

Fix OnStream SCSI Tape's normalize_buffer by freeing from a second list
beyond the I/O scatterlist, saving page pointers with their lengths.

Signed-off-by: Hugh Dickins <hugh@veritas.com>
---
Warning: untested!

 drivers/scsi/osst.c |   13 ++++++++++---
 1 files changed, 10 insertions(+), 3 deletions(-)

--- 2.6.16-rc2/drivers/scsi/osst.c	2006-01-03 03:21:10.000000000 +0000
+++ linux/drivers/scsi/osst.c	2006-02-03 09:59:37.000000000 +0000
@@ -5152,7 +5152,8 @@ static struct osst_buffer * new_tape_buf
 	else
 		priority = GFP_KERNEL;
 
-	i = sizeof(struct osst_buffer) + (osst_max_sg_segs - 1) * sizeof(struct scatterlist);
+	i = sizeof(struct osst_buffer) +
+		(2 * osst_max_sg_segs - 1) * sizeof(struct scatterlist);
 	tb = (struct osst_buffer *)kmalloc(i, priority);
 	if (!tb) {
 		printk(KERN_NOTICE "osst :I: Can't allocate new tape buffer.\n");
@@ -5227,6 +5228,8 @@ static int enlarge_buffer(struct osst_bu
 #if DEBUG
 			STbuffer->buffer_size = got;
 #endif
+			memcpy(STbuffer->sg + STbuffer->sg_segs, STbuffer->sg,
+				STbuffer->sg_segs * sizeof(STbuffer->sg[0]));
 			normalize_buffer(STbuffer);
 			return 0;
 		}
@@ -5235,6 +5238,9 @@ static int enlarge_buffer(struct osst_bu
 		STbuffer->buffer_size = got;
 		STbuffer->sg_segs = ++segs;
 	}
+	/* Save uncoalesced scatterlist of pages to be freed */
+	memcpy(STbuffer->sg + STbuffer->sg_segs, STbuffer->sg,
+		STbuffer->sg_segs * sizeof(STbuffer->sg[0]));
 #if DEBUG
 	if (debugging) {
 		printk(OSST_DEB_MSG
@@ -5254,9 +5260,10 @@ static int enlarge_buffer(struct osst_bu
 /* Release the segments */
 static void normalize_buffer(struct osst_buffer *STbuffer)
 {
-  int i, order, b_size;
+	int i, order, b_size;
 
-	for (i=0; i < STbuffer->sg_segs; i++) {
+	/* Scatterlist entries may have been coalesced: free saved pagelist */
+	for (i = STbuffer->sg_segs; i < 2*STbuffer->sg_segs; i++) {
 
 		for (b_size = PAGE_SIZE, order = 0;
 		     b_size < STbuffer->sg[i].length;

Re: [PATCH] st: don't doublefree pages from scatterlist

[Mike Christie]

Hugh Dickins wrote:
> On some architectures, mapping the scatterlist may coalesce entries:
> if that coalesced list is then used for freeing the pages afterwards,
> there's a danger that pages may be doubly freed (and others leaked).
> 
> Fix SCSI Tape's sgl_unmap_user_pages by freeing from the pagelist used
> in sgl_map_user_pages.  Fixes Ryan Richter's crash on x86_64, with Bad
> page state mapcount 2 from sgl_unmap_user_pages, and consequent mayhem.
> 

Is this crash occuring with 2.6.16-rc1? I ask becuase in that kernel the 
scatterlist passed into scsi_execute_async

if (scsi_execute_async(STp->device, cmd, direction,
			&((STp->buffer)->sg[0]), bytes,

is not the same one that gets send down to the device/HBA.

scsi_execute_async takes the scatterlist passed to it from st or sg, 
uses it as a hint to build a request + bios, then later when the request 
is sent to the device a new scatterlist is sent to the device and the 
device does the pci/dma operation on that scatterlist from the 
block/scsi layer.

Re: Fw: crash on x86_64 - mm related?

[Ryan Richter]

On Fri, Feb 03, 2006 at 07:46:24PM +0000, Hugh Dickins wrote:
> On Wed, 18 Jan 2006, Hugh Dickins wrote:
> > On Tue, 17 Jan 2006, Ryan Richter wrote:
> > 
> > > This machine experienced another random reboot today, nothing in the
> > > logs or on the console etc.  This is the 3rd time now since I upgraded
> > > from 2.6.11.3.  Is there any way to debug something like this?
> > 
> > Nasty.  I hope someone else can suggest something constructive.
> 
> Are they still happening?
> 
> > > I'm fairly certain it's not hardware-related.
> > > Might this have something to do with the st problem?
> > 
> > Well, it might: I've not been able to explain that at all, so cannot
> > rule out a relation to your reboots.
> 
> If no "Bad page state" has occurred earlier, we can now be sure that
> these reboots are (unfortunately) unrelated to the st problem.

It must not be related then, since it never stayed up more than a few
seconds after that happened.

I've tried disabling the hangcheck timer and the software watchdog as a
shot in the dark, but it's only been a week since then and these seem to
happen about every 3 weeks or so.  Quite a pain to debug.

-ryan

Re: [PATCH] st: don't doublefree pages from scatterlist

[Hugh Dickins]

On Fri, 3 Feb 2006, Mike Christie wrote:
> Hugh Dickins wrote:
> > On some architectures, mapping the scatterlist may coalesce entries:
> > if that coalesced list is then used for freeing the pages afterwards,
> > there's a danger that pages may be doubly freed (and others leaked).
> > 
> > Fix SCSI Tape's sgl_unmap_user_pages by freeing from the pagelist used
> > in sgl_map_user_pages.  Fixes Ryan Richter's crash on x86_64, with Bad
> > page state mapcount 2 from sgl_unmap_user_pages, and consequent mayhem.
> 
> Is this crash occuring with 2.6.16-rc1?

I do believe 2.6.15 is the latest release that Ryan has tried (and,
strangely, nobody else has ever reported it - identifiably, anyway).
Interested in trying unpatched 2.6.16-rc2, Ryan?

> I ask becuase in that kernel the scatterlist passed into scsi_execute_async
> 
> if (scsi_execute_async(STp->device, cmd, direction,
> &((STp->buffer)->sg[0]), bytes,
> 
> is not the same one that gets send down to the device/HBA.

Wow, great info, thanks.

> scsi_execute_async takes the scatterlist passed to it from st or sg, uses it
> as a hint to build a request + bios, then later when the request is sent to
> the device a new scatterlist is sent to the device and the device does the
> pci/dma operation on that scatterlist from the block/scsi layer.

Very interesting.  James can confirm, but I think that means everybody
can ignore my drivers/scsi/st.c patch for 2.6.16-rc, and the unposted
sg.c patches for the same issue that I was going to send Doug.

But the Infiniband, ipr and osst patches still look to be necessary -
unless maintainers can point to a similar intervening layer.

Hugh

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[Brian King]

Hugh Dickins wrote:
> On some architectures, mapping the scatterlist may coalesce entries:
> if that coalesced list is then used for freeing the pages afterwards,
> there's a danger that pages may be doubly freed (and others leaked).

I don't understand why this is necessary... Comparing the bug you fixed
in st with ipr's usage of scatterlists, there is a bit of a difference.
st is dealing with user pages and calling page_cache_release to release
the pages, and I can begin to understand why there might be a problem
in that code. page_cache_release looks at the pages themselves to figure
out how many compound pages there are. This could certainly result in
double free's occurring.

Looking at ipr, however, it is simply doing alloc_pages
and __free_pages. __free_pages passes in the page allocation order, so
I don't think I would have the double free problem.

As I understand it, pci_map_sg only modifies the dma_address and dma_length
fields when things get coalesced. If it were to coalese pages by
turning them into compound pages then I would agree that ipr might have
a problem, but I don't think this to be the case...


Brian

> 
> Fix Power RAID's ipr_free_ucode_buffer by freeing from a separate array
> beyond the scatterlist.
> 
> Signed-off-by: Hugh Dickins <hugh@veritas.com>
> ---
> Warning: untested!
> 
>  drivers/scsi/ipr.c |   12 ++++++++++--
>  1 files changed, 10 insertions(+), 2 deletions(-)
> 
> --- 2.6.16-rc2/drivers/scsi/ipr.c	2006-02-03 09:32:24.000000000 +0000
> +++ linux/drivers/scsi/ipr.c	2006-02-03 09:59:37.000000000 +0000
> @@ -2538,6 +2538,7 @@ static struct ipr_sglist *ipr_alloc_ucod
>  	int sg_size, order, bsize_elem, num_elem, i, j;
>  	struct ipr_sglist *sglist;
>  	struct scatterlist *scatterlist;
> +	struct page **sg_pages;
>  	struct page *page;
>  
>  	/* Get the minimum size per scatter/gather element */
> @@ -2557,7 +2558,8 @@ static struct ipr_sglist *ipr_alloc_ucod
>  
>  	/* Allocate a scatter/gather list for the DMA */
>  	sglist = kzalloc(sizeof(struct ipr_sglist) +
> -			 (sizeof(struct scatterlist) * (num_elem - 1)),
> +			 (sizeof(struct scatterlist) * (num_elem - 1)) +
> +			 (sizeof(struct page *) * num_elem),
>  			 GFP_KERNEL);
>  
>  	if (sglist == NULL) {
> @@ -2566,6 +2568,8 @@ static struct ipr_sglist *ipr_alloc_ucod
>  	}
>  
>  	scatterlist = sglist->scatterlist;
> +	/* Save pages to be freed in array beyond scatterlist */
> +	sg_pages = (struct page **) (scatterlist + num_elem);
>  
>  	sglist->order = order;
>  	sglist->num_sg = num_elem;
> @@ -2584,6 +2588,7 @@ static struct ipr_sglist *ipr_alloc_ucod
>  		}
>  
>  		scatterlist[i].page = page;
> +		sg_pages[i] = page;
>  	}
>  
>  	return sglist;
> @@ -2601,10 +2606,13 @@ static struct ipr_sglist *ipr_alloc_ucod
>   **/
>  static void ipr_free_ucode_buffer(struct ipr_sglist *sglist)
>  {
> +	struct page **sg_pages;
>  	int i;
>  
> +	/* Scatterlist entries may have been coalesced: free saved pagelist */
> +	sg_pages = (struct page **) (sglist->scatterlist + sglist->num_sg);
>  	for (i = 0; i < sglist->num_sg; i++)
> -		__free_pages(sglist->scatterlist[i].page, sglist->order);
> +		__free_pages(sg_pages[i], sglist->order);
>  
>  	kfree(sglist);
>  }
> 


-- 
Brian King
eServer Storage I/O
IBM Linux Technology Center

Re: [PATCH] ib: don't doublefree pages from scatterlist

[Roland Dreier]

Thanks, Hugh.  This is definitely a real bug caused by an embarassing
oversight on my part.  I will test and apply to my trees.

 > Warning: untested!  And please double-check the adjusted definition of
 > IB_UMEM_MAX_PAGE_CHUNK - the old definition was avoiding "sizeof"s, but
 > I don't understand why.

The old definition of IB_UMEM_MAX_PAGE_CHUNK came from my paranoia:

 >  #define IB_UMEM_MAX_PAGE_CHUNK						\
 >  	((PAGE_SIZE - offsetof(struct ib_umem_chunk, page_list)) /	\
 > -	 ((void *) &((struct ib_umem_chunk *) 0)->page_list[1] -	\
 > -	  (void *) &((struct ib_umem_chunk *) 0)->page_list[0]))
 > +	 (sizeof(struct scatterlist) + sizeof(struct page *)))

I was afraid that some compiler somewhere might add in some padding
that would cause sizeof (struct scatterlist) to be smaller than the
entries in the array end up being, but now I've convinced myself that
this can't happen -- if it could then things like ARRAY_SIZE() would
be stuffed as well.

So I think your version is correct and clearer.

Thanks,
  Roland

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[Hugh Dickins]

On Fri, 3 Feb 2006, Brian King wrote:
> Hugh Dickins wrote:
> > On some architectures, mapping the scatterlist may coalesce entries:
> > if that coalesced list is then used for freeing the pages afterwards,
> > there's a danger that pages may be doubly freed (and others leaked).
> 
> I don't understand why this is necessary... Comparing the bug you fixed
> in st with ipr's usage of scatterlists, there is a bit of a difference.
> st is dealing with user pages and calling page_cache_release to release
> the pages, and I can begin to understand why there might be a problem
> in that code.

Yes, certainly the st and ipr cases seem different, and originally
I was thinking it was only an issue in connection with get_user_pages.

> page_cache_release looks at the pages themselves to figure
> out how many compound pages there are. This could certainly result in
> double free's occurring.

I don't follow you there, but better if I try to explain how I see it.

> Looking at ipr, however, it is simply doing alloc_pages
> and __free_pages. __free_pages passes in the page allocation order, so
> I don't think I would have the double free problem.
> 
> As I understand it, pci_map_sg only modifies the dma_address and dma_length
> fields when things get coalesced. If it were to coalese pages by
> turning them into compound pages then I would agree that ipr might have
> a problem, but I don't think this to be the case...

It's not fully defined what it does (intentionally: internal detail).
But as I understand it, what it's likely to do is coalesce entries as
far as it can (causes no problem in itself) then shift down and attempt
to coalesce the entries above.  It's the shifting down that gives the
problem.

Imagine we start with sglist

struct page *a, length A
struct page *b, length B
struct page *c, length C

and pci_map_sg finds a+A brings it to b (I'm writing loosely, mixing
the page pointers and their corresponding virtual addresses), but b+B
does not match c.  Then it'll coalesce the first two entries, and
shift down the third to second place, turning sglist into

struct page *a, length A+B
struct page *c, length C
struct page *c, length C

So (again writing loosely, mixing up lengths and orders) on return the
caller will __free_pages(a, A+B) (itself probably wrong since a and b
were likely not buddies to start out with), __free_pages(c, C),
__free_pages(c, C) - doubly freeing c, ensuing mayhem.

Maybe it doesn't change length A to A+B, just doing that in dma_length;
but caller is still going to free c twice.

Agree?

Hugh

Re: Fw: crash on x86_64 - mm related?

[Kai Makisara]

On Fri, 3 Feb 2006, Hugh Dickins wrote:

> On Wed, 18 Jan 2006, Hugh Dickins wrote:
> > On Tue, 17 Jan 2006, Ryan Richter wrote:
> > 
...
> > The st problem (Bad page state,
> > mapcount 2 while count 0, from sgl_unmap_user_pages) ought to be a lot
> > easier to debug than a random reboot: but I've still no suggestion.
> 
> I guessed it last week, Ryan verified that booting with "iommu=nomerge"
> worked around it, and then that the 2.6.15 patch below fixes it.
> 

I have tested the patch and did not find any problems. I suppose you will 
send this to the stable team for inclusion into 2.6.15.x? You can add

Signed-off-by: Kai Makisara <Kai.Makisara@kolumbus.fi>

Thanks for fixing this problem.

-- 
Kai

Re: [PATCH] st: don't doublefree pages from scatterlist

[Kai Makisara]

On Fri, 3 Feb 2006, Hugh Dickins wrote:

> On Fri, 3 Feb 2006, Mike Christie wrote:
...
> > I ask becuase in that kernel the scatterlist passed into scsi_execute_async
> > 
> > if (scsi_execute_async(STp->device, cmd, direction,
> > &((STp->buffer)->sg[0]), bytes,
> > 
> > is not the same one that gets send down to the device/HBA.
> 
> Wow, great info, thanks.
> 
> > scsi_execute_async takes the scatterlist passed to it from st or sg, uses it
> > as a hint to build a request + bios, then later when the request is sent to
> > the device a new scatterlist is sent to the device and the device does the
> > pci/dma operation on that scatterlist from the block/scsi layer.
> 
> Very interesting.  James can confirm, but I think that means everybody
> can ignore my drivers/scsi/st.c patch for 2.6.16-rc, and the unposted
> sg.c patches for the same issue that I was going to send Doug.
> 
The patch st is not necessary now but I don't think it would be a bad idea 
to include it anyway. My reasoning is based on that the patch is very 
inexpensive, it basically moves freeing of an array to another place. 
The reasons for inclusion are:
- someone reviewing the code may wonder why the change to 2.6.15.x is
  not in 2.6.x >= 16; 2.6.16 would need at least a comment about this
- it does decouple st from any dependencies about what happens to
  the s/g array at the lower levels
- if the s/g array will at some future time be again passed directly to 
  dma mapping, we would not face the problem again

I don't have any firm opinion either way.

-- 
Kai

Re: Fw: crash on x86_64 - mm related?

[Hugh Dickins]

On Sat, 4 Feb 2006, Kai Makisara wrote:
> On Fri, 3 Feb 2006, Hugh Dickins wrote:
> > > The st problem (Bad page state,
> > > mapcount 2 while count 0, from sgl_unmap_user_pages) ought to be a lot
> > > easier to debug than a random reboot: but I've still no suggestion.
> > 
> > I guessed it last week, Ryan verified that booting with "iommu=nomerge"
> > worked around it, and then that the 2.6.15 patch below fixes it.
> 
> I have tested the patch and did not find any problems. I suppose you will 

Thanks.

> send this to the stable team for inclusion into 2.6.15.x? You can add

Yes, this one does fit the criteria for stable (even though the patch
fortuitously turns out not to be necessary in 2.6.16): it fixes a serious
observed user problem.  Not sure what to do with the associated patches
to other drivers though, since those problems have not been knowingly
observed: probably best leave them to maintainer's discretion.

> Signed-off-by: Kai Makisara <Kai.Makisara@kolumbus.fi>

Thanks,
Hugh

Re: [PATCH] st: don't doublefree pages from scatterlist

[Hugh Dickins]

On Sat, 4 Feb 2006, Kai Makisara wrote:
> On Fri, 3 Feb 2006, Hugh Dickins wrote:
> > 
> > Very interesting.  James can confirm, but I think that means everybody
> > can ignore my drivers/scsi/st.c patch for 2.6.16-rc, and the unposted
> > sg.c patches for the same issue that I was going to send Doug.
> > 
> The patch st is not necessary now but I don't think it would be a bad idea 
> to include it anyway. My reasoning is based on that the patch is very 
> inexpensive, it basically moves freeing of an array to another place. 
> The reasons for inclusion are:
> - someone reviewing the code may wonder why the change to 2.6.15.x is
>   not in 2.6.x >= 16; 2.6.16 would need at least a comment about this
> - it does decouple st from any dependencies about what happens to
>   the s/g array at the lower levels
> - if the s/g array will at some future time be again passed directly to 
>   dma mapping, we would not face the problem again
> 
> I don't have any firm opinion either way.

Fair points, I don't have a firm opinion either, it's entirely up to you
as maintainer.

But I'd be less keen to force the equivalent changes into 2.6.16's sg.c.

Saving sg_pages from get_user_pages fits in quite unobtrusively in st.c,
and its enlarge_buffer/normalize_buffer pages are already safely isolated
from the scatterlist by the separate frp_segs array.  But sg.c presently
uses the scatterlist for the latter case too, so would need more change:
except that, thanks to Mike, we find it doesn't need changing now.
Your suggestion of a comment would be most appropriate there.

(I've not forgotten our outstanding SetPageDirty to set_page_dirty_lock
issue, but still striving to avoid too much nuisance in sg.c's case.)

Hugh

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[Brian King]

Hugh Dickins wrote:
> On Fri, 3 Feb 2006, Brian King wrote:
>>Hugh Dickins wrote:
>>>On some architectures, mapping the scatterlist may coalesce entries:
>>>if that coalesced list is then used for freeing the pages afterwards,
>>>there's a danger that pages may be doubly freed (and others leaked).
>>I don't understand why this is necessary... Comparing the bug you fixed
>>in st with ipr's usage of scatterlists, there is a bit of a difference.
>>st is dealing with user pages and calling page_cache_release to release
>>the pages, and I can begin to understand why there might be a problem
>>in that code.
> 
> Yes, certainly the st and ipr cases seem different, and originally
> I was thinking it was only an issue in connection with get_user_pages.
> 
>>page_cache_release looks at the pages themselves to figure
>>out how many compound pages there are. This could certainly result in
>>double free's occurring.
> 
> I don't follow you there, but better if I try to explain how I see it.
> 
>>Looking at ipr, however, it is simply doing alloc_pages
>>and __free_pages. __free_pages passes in the page allocation order, so
>>I don't think I would have the double free problem.
>>
>>As I understand it, pci_map_sg only modifies the dma_address and dma_length
>>fields when things get coalesced. If it were to coalese pages by
>>turning them into compound pages then I would agree that ipr might have
>>a problem, but I don't think this to be the case...
> 
> It's not fully defined what it does (intentionally: internal detail).

I did a quick check of all the different architectures and was unable
to find any architecture that does what you describe below. All coalescing
appears to be done by only modifying the dma_length and dma_address fields in
the scatterlist.

> But as I understand it, what it's likely to do is coalesce entries as
> far as it can (causes no problem in itself) then shift down and attempt
> to coalesce the entries above.  It's the shifting down that gives the
> problem.
> 
> Imagine we start with sglist
> 
> struct page *a, length A
> struct page *b, length B
> struct page *c, length C
> 
> and pci_map_sg finds a+A brings it to b (I'm writing loosely, mixing
> the page pointers and their corresponding virtual addresses), but b+B
> does not match c.  Then it'll coalesce the first two entries, and
> shift down the third to second place, turning sglist into
> 
> struct page *a, length A+B
> struct page *c, length C
> struct page *c, length C

Disagree. This is how I would expect the sglist to look after pci_map_sg:

struct page *a, length A, dma_addr a, dma_length A+B
struct page *b, length B, dma_addr c, dma_length C
struct page *c, length C, dma_addr n/a, dma_length n/a

> 
> So (again writing loosely, mixing up lengths and orders) on return the
> caller will __free_pages(a, A+B) (itself probably wrong since a and b
> were likely not buddies to start out with), __free_pages(c, C),
> __free_pages(c, C) - doubly freeing c, ensuing mayhem.
> 
> Maybe it doesn't change length A to A+B, just doing that in dma_length;
> but caller is still going to free c twice.
> 
> Agree?

No. I can't find any code in any architecture that modifies either the length,
page, or offset fields in the *_map_sg routines.

I'm still failing to see an actual bug in the ipr driver. Unless there is a
good reason for pushing additional complexity on every caller of pci_map_sg
to do additional bookkeeping, such as an architecture that is actually modifying
fields in the scatterlist other than the dma_* fields, then I think it makes
more sense to clarify the API to say that pci_map_sg will not modify these fields.

Brian

-- 
Brian King
eServer Storage I/O
IBM Linux Technology Center

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[Hugh Dickins]

On Sun, 5 Feb 2006, Brian King wrote:
> 
> No. I can't find any code in any architecture that modifies either the length,
> page, or offset fields in the *_map_sg routines.

>From looking at the source, the architectures I found to be doing
scatterlist coalescing in some cases were alpha, ia64, parisc (some
code under drivers), powerpc, sparc64 and x86_64.

I agree with you that it would be possible for them to do the coalescing
by just adjusting dma_address and dma_length (though it's architecture-
dependent whether there are such fields at all), not interfering with
the input page and length; and maybe some of them do proceed that way.
I didn't find the coalescing code in any of them very easy to follow.

So please examine arch/x86_64/kernel/pci_gart.c gart_map_sg (and
dma_map_cont which it calls): x86_64 was the architecture on which
the problem was really found with drivers/scsi/st.c, and avoided
by that boot option iommu=nomerge.

Lines like "*sout = *s;" and "*sout = sg[start];" are structure-
copying whole scallerlist entries from one position in the list
to another, without explicit mention of the page and length fields.

> I'm still failing to see an actual bug in the ipr driver. Unless there is a
> good reason for pushing additional complexity on every caller of pci_map_sg
> to do additional bookkeeping, such as an architecture that is actually
> modifying
> fields in the scatterlist other than the dma_* fields, then I think it makes
> more sense to clarify the API to say that pci_map_sg will not modify these
> fields.

The API is commendably clear that they may be modified.  Like you I'd
prefer it to be otherwise, and instead guarantee that they won't be:
that would prevent this kind of surprise.  But it's not how it is;
and looking at the various pieces of coalescing code, I quickly
abandoned my first inclination, to adjust them to make it so.

Hugh

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[David S. Miller]

From: Hugh Dickins <hugh@veritas.com>
Date: Mon, 6 Feb 2006 09:32:54 +0000 (GMT)

> From looking at the source, the architectures I found to be doing
> scatterlist coalescing in some cases were alpha, ia64, parisc (some
> code under drivers), powerpc, sparc64 and x86_64.
> 
> I agree with you that it would be possible for them to do the coalescing
> by just adjusting dma_address and dma_length (though it's architecture-
> dependent whether there are such fields at all), not interfering with
> the input page and length; and maybe some of them do proceed that way.
> I didn't find the coalescing code in any of them very easy to follow.
> 
> So please examine arch/x86_64/kernel/pci_gart.c gart_map_sg (and
> dma_map_cont which it calls): x86_64 was the architecture on which
> the problem was really found with drivers/scsi/st.c, and avoided
> by that boot option iommu=nomerge.
> 
> Lines like "*sout = *s;" and "*sout = sg[start];" are structure-
> copying whole scallerlist entries from one position in the list
> to another, without explicit mention of the page and length fields.

That's a bug, frankly.  Sparc64 doesn't need to do anything like
that.  Spamming the page pointers is really really bogus and I'm
surprised this doesn't make more stuff explode.

It was never the intention to allow the DMA mapping support code
to modify the page, offset, and length members of the scatterlist.
Only the DMA components.

I'd really prefer that those assignments get fixed and an explicit
note added to Documentation/DMA-mapping.txt about this.

It's rediculious that these generic subsystem drivers need to
know about this. :)

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[Brian King]

[-- Attachment #1: Type: text/plain, Size: 1730 bytes --]

David S. Miller wrote:
> From: Hugh Dickins <hugh@veritas.com>
> Date: Mon, 6 Feb 2006 09:32:54 +0000 (GMT)
> 
>> From looking at the source, the architectures I found to be doing
>> scatterlist coalescing in some cases were alpha, ia64, parisc (some
>> code under drivers), powerpc, sparc64 and x86_64.
>>
>> I agree with you that it would be possible for them to do the coalescing
>> by just adjusting dma_address and dma_length (though it's architecture-
>> dependent whether there are such fields at all), not interfering with
>> the input page and length; and maybe some of them do proceed that way.
>> I didn't find the coalescing code in any of them very easy to follow.
>>
>> So please examine arch/x86_64/kernel/pci_gart.c gart_map_sg (and
>> dma_map_cont which it calls): x86_64 was the architecture on which
>> the problem was really found with drivers/scsi/st.c, and avoided
>> by that boot option iommu=nomerge.
>>
>> Lines like "*sout = *s;" and "*sout = sg[start];" are structure-
>> copying whole scallerlist entries from one position in the list
>> to another, without explicit mention of the page and length fields.
> 
> That's a bug, frankly.  Sparc64 doesn't need to do anything like
> that.  Spamming the page pointers is really really bogus and I'm
> surprised this doesn't make more stuff explode.
> 
> It was never the intention to allow the DMA mapping support code
> to modify the page, offset, and length members of the scatterlist.
> Only the DMA components.
> 
> I'd really prefer that those assignments get fixed and an explicit
> note added to Documentation/DMA-mapping.txt about this.

How about this for a documentation fix?

Brian


-- 
Brian King
eServer Storage I/O
IBM Linux Technology Center

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: dma_mapping_clarification.patch --]
[-- Type: text/x-patch; name="dma_mapping_clarification.patch", Size: 1452 bytes --]


The current pci_map_sg API is a bit unclear what it is allowed
to modify in the passed scatterlist when coalescing entries.
Clarify the pci_map_sg API to prevent it from modifying
the page, offset, and length fields in the scatterlist.

Signed-off-by: Brian King <brking@us.ibm.com>
---

 linux-2.6-bjking1/Documentation/DMA-mapping.txt |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletion(-)

diff -puN Documentation/DMA-mapping.txt~dma_mapping_clarification Documentation/DMA-mapping.txt
--- linux-2.6/Documentation/DMA-mapping.txt~dma_mapping_clarification	2006-02-06 08:23:10.000000000 -0600
+++ linux-2.6-bjking1/Documentation/DMA-mapping.txt	2006-02-06 08:38:43.000000000 -0600
@@ -513,7 +513,10 @@ consecutive sglist entries can be merged
 ends and the second one starts on a page boundary - in fact this is a huge
 advantage for cards which either cannot do scatter-gather or have very
 limited number of scatter-gather entries) and returns the actual number
-of sg entries it mapped them to. On failure 0 is returned.
+of sg entries it mapped them to. The implementation is free to do this
+by modifying the scatterlist fields specified for DMA. The scatterlist
+fields used as an input to this function (i.e. page, offset, and length)
+will NOT be modified. On failure 0 is returned.
 
 Then you should loop count times (note: this can be less than nents times)
 and use sg_dma_address() and sg_dma_len() macros where you previously
_

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[James Bottomley]

On Mon, 2006-02-06 at 01:46 -0800, David S. Miller wrote:
> That's a bug, frankly.  Sparc64 doesn't need to do anything like
> that.  Spamming the page pointers is really really bogus and I'm
> surprised this doesn't make more stuff explode.
> 
> It was never the intention to allow the DMA mapping support code
> to modify the page, offset, and length members of the scatterlist.
> Only the DMA components.
> 
> I'd really prefer that those assignments get fixed and an explicit
> note added to Documentation/DMA-mapping.txt about this.
> 
> It's rediculious that these generic subsystem drivers need to
> know about this. :)

I complained about this x86_64 behaviour ages ago.  Andi claimed it was
the only way they could get there merging algorithm to work.  It
actually triggered a bug in SCSI because in-flight I/O that was rejected
gets unmapped and then remapped (which was, originally, not working).
They finally fixed it by making the unmap put back the original
scatterlist.  Perhaps this should go to linux-arch just in case anyone
else copied x86_64?

James

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[Hugh Dickins]

On Mon, 6 Feb 2006, Brian King wrote:
> David S. Miller wrote:
> > From: Hugh Dickins <hugh@veritas.com>
> > Date: Mon, 6 Feb 2006 09:32:54 +0000 (GMT)
> > 
> >> From looking at the source, the architectures I found to be doing
> >> scatterlist coalescing in some cases were alpha, ia64, parisc (some
> >> code under drivers), powerpc, sparc64 and x86_64.
> >>
> >> I agree with you that it would be possible for them to do the coalescing
> >> by just adjusting dma_address and dma_length (though it's architecture-
> >> dependent whether there are such fields at all), not interfering with
> >> the input page and length; and maybe some of them do proceed that way.
> >> I didn't find the coalescing code in any of them very easy to follow.
> >>
> >> So please examine arch/x86_64/kernel/pci_gart.c gart_map_sg (and
> >> dma_map_cont which it calls): x86_64 was the architecture on which
> >> the problem was really found with drivers/scsi/st.c, and avoided
> >> by that boot option iommu=nomerge.
> >>
> >> Lines like "*sout = *s;" and "*sout = sg[start];" are structure-
> >> copying whole scallerlist entries from one position in the list
> >> to another, without explicit mention of the page and length fields.
> > 
> > That's a bug, frankly.  Sparc64 doesn't need to do anything like
> > that.  Spamming the page pointers is really really bogus and I'm
> > surprised this doesn't make more stuff explode.
> > 
> > It was never the intention to allow the DMA mapping support code
> > to modify the page, offset, and length members of the scatterlist.
> > Only the DMA components.
> > 
> > I'd really prefer that those assignments get fixed and an explicit
> > note added to Documentation/DMA-mapping.txt about this.
> 
> How about this for a documentation fix?

Now that David and (to my surprise) James have come out clearly in
favour of your interpretation, yes, this would certainly be an
improvement.

But I'd also want James or someone to clarify the paragraph
"Please note that the sg cannot be mapped again if it has been mapped once.
 The mapping process is allowed to destroy information in the sg."
which I took as explicitly allowing what x86_64 does in gart_map_sg.
I thought James had a scenario in mind which demands this wholesale
destruction, but it seems not; and I now read that first sentence as
saying the sg must be unmapped before it can be mapped a second time,
not that it can only be mapped once.

And add a paragraph explaining that really the one array of scatterlist
entries should be regarded as two arrays of possibly different lengths,
the page,offset,length array and the dma_address,dma_length array:
because entries of the latter may be coalesced, so that in the end
the dma_address in a scatterlength entry may bear no relation to the
page pointer in that same entry, but to the page pointer in a later entry.
Though it gets hard to explain given that not all architectures coalesce,
so may not even have a separate dma_length field; or use different naming.
If you can express this better than I, please do!

But all this presupposes that someone is suddenly going to change the
x86_64 gart_map_sg (and subfunctions), or else force its iommu=nomerge:
that won't be me.

Hugh

> The current pci_map_sg API is a bit unclear what it is allowed
> to modify in the passed scatterlist when coalescing entries.
> Clarify the pci_map_sg API to prevent it from modifying
> the page, offset, and length fields in the scatterlist.
> 
> Signed-off-by: Brian King <brking@us.ibm.com>
> ---
> 
>  linux-2.6-bjking1/Documentation/DMA-mapping.txt |    5 ++++-
>  1 files changed, 4 insertions(+), 1 deletion(-)
> 
> diff -puN Documentation/DMA-mapping.txt~dma_mapping_clarification Documentation/DMA-mapping.txt
> --- linux-2.6/Documentation/DMA-mapping.txt~dma_mapping_clarification	2006-02-06 08:23:10.000000000 -0600
> +++ linux-2.6-bjking1/Documentation/DMA-mapping.txt	2006-02-06 08:38:43.000000000 -0600
> @@ -513,7 +513,10 @@ consecutive sglist entries can be merged
>  ends and the second one starts on a page boundary - in fact this is a huge
>  advantage for cards which either cannot do scatter-gather or have very
>  limited number of scatter-gather entries) and returns the actual number
> -of sg entries it mapped them to. On failure 0 is returned.
> +of sg entries it mapped them to. The implementation is free to do this
> +by modifying the scatterlist fields specified for DMA. The scatterlist
> +fields used as an input to this function (i.e. page, offset, and length)
> +will NOT be modified. On failure 0 is returned.
>  
>  Then you should loop count times (note: this can be less than nents times)
>  and use sg_dma_address() and sg_dma_len() macros where you previously

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[Hugh Dickins]

On Mon, 6 Feb 2006, James Bottomley wrote:
> On Mon, 2006-02-06 at 01:46 -0800, David S. Miller wrote:
> > That's a bug, frankly.  Sparc64 doesn't need to do anything like
> > that.  Spamming the page pointers is really really bogus and I'm
> > surprised this doesn't make more stuff explode.
> > 
> > It was never the intention to allow the DMA mapping support code
> > to modify the page, offset, and length members of the scatterlist.
> > Only the DMA components.
> > 
> > I'd really prefer that those assignments get fixed and an explicit
> > note added to Documentation/DMA-mapping.txt about this.
> > 
> > It's rediculious that these generic subsystem drivers need to
> > know about this. :)
> 
> I complained about this x86_64 behaviour ages ago.

While I agree with you, David and Brian that this would be much nicer
for driver writers to know that the page,offset,length in the sg list
would not be touched by map_sg, I am surprised that you didn't say so
in DMA-API.txt, and said only that map_sg would destroy information
there.  x86_64 seems to conform to that destruction!

> Andi claimed it was
> the only way they could get there merging algorithm to work.  It
> actually triggered a bug in SCSI because in-flight I/O that was rejected
> gets unmapped and then remapped (which was, originally, not working).
> They finally fixed it by making the unmap put back the original
> scatterlist.

I don't quite get that, but whatever, it would be a good idea to cc Andi.

> Perhaps this should go to linux-arch just in case anyone
> else copied x86_64?

Okay, cc'd to linux-arch also.  The one thing I have checked is that the
coalescing architectures (if I actually got them right) do all declare
a dma_length (perhaps differently named) distinct from input length,
so wouldn't need that added.  Although I can see that x86_64 does
modify the page,offset,length fields we'd prefer it not to, I didn't
find any of the architecture's coalescing routines easy to follow,
and would hesitate to declare any of them safe in this regard.

I'd better also forward my original mail on the fix to Ryan's bug
to Andi and linux-arch also.

Hugh

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[James Bottomley]

On Mon, 2006-02-06 at 16:45 +0000, Hugh Dickins wrote:
> But I'd also want James or someone to clarify the paragraph
> "Please note that the sg cannot be mapped again if it has been mapped once.
>  The mapping process is allowed to destroy information in the sg."
> which I took as explicitly allowing what x86_64 does in gart_map_sg.
> I thought James had a scenario in mind which demands this wholesale
> destruction, but it seems not; and I now read that first sentence as
> saying the sg must be unmapped before it can be mapped a second time,
> not that it can only be mapped once.
> 
> And add a paragraph explaining that really the one array of scatterlist
> entries should be regarded as two arrays of possibly different lengths,
> the page,offset,length array and the dma_address,dma_length array:
> because entries of the latter may be coalesced, so that in the end
> the dma_address in a scatterlength entry may bear no relation to the
> page pointer in that same entry, but to the page pointer in a later entry.
> Though it gets hard to explain given that not all architectures coalesce,
> so may not even have a separate dma_length field; or use different naming.
> If you can express this better than I, please do!

Yes, I added that piece after the x86_64 problem.  The original x86_64
bug was that you couldn't do dma_map_sg() then dma_unmap_sg() then
dma_map_sg() on the same scatterlist (the map was destroying information
which wasn't restored on the unmap, so the second map produced an
incorrect scatterlist), which was causing a bug in all SCSI drivers
(because that's the way SCSI requeueing works).

James

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[Brian King]

[-- Attachment #1: Type: text/plain, Size: 1827 bytes --]

James Bottomley wrote:
> On Mon, 2006-02-06 at 16:45 +0000, Hugh Dickins wrote:
>> But I'd also want James or someone to clarify the paragraph
>> "Please note that the sg cannot be mapped again if it has been mapped once.
>>  The mapping process is allowed to destroy information in the sg."
>> which I took as explicitly allowing what x86_64 does in gart_map_sg.
>> I thought James had a scenario in mind which demands this wholesale
>> destruction, but it seems not; and I now read that first sentence as
>> saying the sg must be unmapped before it can be mapped a second time,
>> not that it can only be mapped once.
>>
>> And add a paragraph explaining that really the one array of scatterlist
>> entries should be regarded as two arrays of possibly different lengths,
>> the page,offset,length array and the dma_address,dma_length array:
>> because entries of the latter may be coalesced, so that in the end
>> the dma_address in a scatterlength entry may bear no relation to the
>> page pointer in that same entry, but to the page pointer in a later entry.
>> Though it gets hard to explain given that not all architectures coalesce,
>> so may not even have a separate dma_length field; or use different naming.
>> If you can express this better than I, please do!
> 
> Yes, I added that piece after the x86_64 problem.  The original x86_64
> bug was that you couldn't do dma_map_sg() then dma_unmap_sg() then
> dma_map_sg() on the same scatterlist (the map was destroying information
> which wasn't restored on the unmap, so the second map produced an
> incorrect scatterlist), which was causing a bug in all SCSI drivers
> (because that's the way SCSI requeueing works).

Here is a second try at a documentation update. Any takers to fixup x86_64?

Brian

-- 
Brian King
eServer Storage I/O
IBM Linux Technology Center

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: dma_mapping_clarification.patch --]
[-- Type: text/x-patch; name="dma_mapping_clarification.patch", Size: 2407 bytes --]


The current pci_map_sg API is a bit unclear what it is allowed
to modify in the passed scatterlist when coalescing entries.
Clarify the pci_map_sg API to prevent it from modifying
the page, offset, and length fields in the scatterlist.

Signed-off-by: Brian King <brking@us.ibm.com>
---

 linux-2.6-bjking1/Documentation/DMA-API.txt     |    5 +++--
 linux-2.6-bjking1/Documentation/DMA-mapping.txt |    5 ++++-
 2 files changed, 7 insertions(+), 3 deletions(-)

diff -puN Documentation/DMA-mapping.txt~dma_mapping_clarification Documentation/DMA-mapping.txt
--- linux-2.6/Documentation/DMA-mapping.txt~dma_mapping_clarification	2006-02-06 13:05:52.000000000 -0600
+++ linux-2.6-bjking1/Documentation/DMA-mapping.txt	2006-02-06 13:05:52.000000000 -0600
@@ -513,7 +513,10 @@ consecutive sglist entries can be merged
 ends and the second one starts on a page boundary - in fact this is a huge
 advantage for cards which either cannot do scatter-gather or have very
 limited number of scatter-gather entries) and returns the actual number
-of sg entries it mapped them to. On failure 0 is returned.
+of sg entries it mapped them to. The implementation is free to do this
+by modifying the scatterlist fields specified for DMA. The scatterlist
+fields used as an input to this function (i.e. page, offset, and length)
+will NOT be modified. On failure 0 is returned.
 
 Then you should loop count times (note: this can be less than nents times)
 and use sg_dma_address() and sg_dma_len() macros where you previously
diff -puN Documentation/DMA-API.txt~dma_mapping_clarification Documentation/DMA-API.txt
--- linux-2.6/Documentation/DMA-API.txt~dma_mapping_clarification	2006-02-06 13:06:17.000000000 -0600
+++ linux-2.6-bjking1/Documentation/DMA-API.txt	2006-02-06 13:07:40.000000000 -0600
@@ -318,8 +318,9 @@ than <nents> passed in if the block laye
 elements of the scatter/gather list are physically adjacent and thus
 may be mapped with a single entry).
 
-Please note that the sg cannot be mapped again if it has been mapped once.
-The mapping process is allowed to destroy information in the sg.
+Please note that the sg can be mapped again, as long as it is unmapped
+first. The mapping process is only allowed to modify the scatterlist
+fields related to DMA.
 
 As with the other mapping interfaces, dma_map_sg can fail. When it
 does, 0 is returned and a driver must take appropriate action. It is
_

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[Andi Kleen]

On Monday 06 February 2006 17:45, Hugh Dickins wrote:

> 
> But all this presupposes that someone is suddenly going to change the
> x86_64 gart_map_sg (and subfunctions), or else force its iommu=nomerge:
> that won't be me.

Ok i changed it to conform to the gospel. I gave it some basic pounding LTP/dd IO with 
and without IOMMU force, but it's not that well tested. More testing welcome.

-Andi

Don't touch the non DMA members in the sg list in dma_map_sg in the IOMMU

Some drivers (in particular ipr) ran into problems because they
reused the sg lists after passing them to pci_map_sg(). The merging
procedure in the K8 GART IOMMU corrupted the state. This patch
changes it to only touch the dma* entries during merging,
but not the other fields.  Approach suggested by Dave Miller.

I did some basic tests with CONFIG_IOMMU_DEBUG (LTP, large dd) 
and without and it hold up, but needs more testing.

Signed-off-by: Andi Kleen <ak@suse.de>

Index: linux/arch/x86_64/kernel/pci-gart.c
===================================================================
--- linux.orig/arch/x86_64/kernel/pci-gart.c
+++ linux/arch/x86_64/kernel/pci-gart.c
@@ -300,7 +300,7 @@ void gart_unmap_sg(struct device *dev, s
 
 	for (i = 0; i < nents; i++) {
 		struct scatterlist *s = &sg[i];
-		if (!s->dma_length || !s->length)
+		if (!s->dma_length)
 			break;
 		dma_unmap_single(dev, s->dma_address, s->dma_length, dir);
 	}
@@ -354,7 +354,6 @@ static int __dma_map_cont(struct scatter
 		
 		BUG_ON(i > start && s->offset);
 		if (i == start) {
-			*sout = *s; 
 			sout->dma_address = iommu_bus_base;
 			sout->dma_address += iommu_page*PAGE_SIZE + s->offset;
 			sout->dma_length = s->length;
@@ -369,7 +368,7 @@ static int __dma_map_cont(struct scatter
 			SET_LEAK(iommu_page);
 			addr += PAGE_SIZE;
 			iommu_page++;
-	} 
+		} 
 	} 
 	BUG_ON(iommu_page - iommu_start != pages);	
 	return 0;
@@ -381,7 +380,6 @@ static inline int dma_map_cont(struct sc
 {
 	if (!need) { 
 		BUG_ON(stopat - start != 1);
-		*sout = sg[start]; 
 		sout->dma_length = sg[start].length; 
 		return 0;
 	} 

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[David S. Miller]

From: Andi Kleen <ak@suse.de>
Date: Mon, 6 Feb 2006 22:11:29 +0100

> Don't touch the non DMA members in the sg list in dma_map_sg in the IOMMU
> 
> Some drivers (in particular ipr) ran into problems because they
> reused the sg lists after passing them to pci_map_sg(). The merging
> procedure in the K8 GART IOMMU corrupted the state. This patch
> changes it to only touch the dma* entries during merging,
> but not the other fields.  Approach suggested by Dave Miller.
> 
> I did some basic tests with CONFIG_IOMMU_DEBUG (LTP, large dd) 
> and without and it hold up, but needs more testing.
> 
> Signed-off-by: Andi Kleen <ak@suse.de>

Thanks for doing this work Andi.

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[Hugh Dickins]

On Mon, 6 Feb 2006, Andi Kleen wrote:
> On Monday 06 February 2006 17:45, Hugh Dickins wrote:
> > 
> > But all this presupposes that someone is suddenly going to change the
> > x86_64 gart_map_sg (and subfunctions), or else force its iommu=nomerge:
> > that won't be me.
> 
> Ok i changed it to conform to the gospel. I gave it some basic pounding LTP/dd IO with 
> and without IOMMU force, but it's not that well tested. More testing welcome.

Great, thanks Andi.  One small correction to the comment...

> Don't touch the non DMA members in the sg list in dma_map_sg in the IOMMU
> 
> Some drivers (in particular ipr) ran into problems because they

No, the problem hasn't knowingly been sighted on ipr, it was the
st driver that Ryan's been seeing it with - ipr just came from
my looking around for like instances.

> reused the sg lists after passing them to pci_map_sg(). The merging
> procedure in the K8 GART IOMMU corrupted the state. This patch
> changes it to only touch the dma* entries during merging,
> but not the other fields.  Approach suggested by Dave Miller.
> 
> I did some basic tests with CONFIG_IOMMU_DEBUG (LTP, large dd) 
> and without and it hold up, but needs more testing.

Below is, I think, the 2.6.15 equivalent of the patch Andi posted.
Ryan cannot effectively test Andi's patch on 2.6.16-rc because Mike
Christie's scsi_execute_async changes have serendipitously fixed
the st instance.  Ryan, would you be able to test the patch below
on 2.6.15 without my st.c,st.h patch?

But I fear Ryan may be growing a little tired of this surfeit
of fixes: he's waited months for one, now he's been given one
workaround and three fixes.

Thanks,
Hugh

--- 2.6.15/arch/x86_64/kernel/pci-gart.c	2006-01-03 03:21:10.000000000 +0000
+++ linux/arch/x86_64/kernel/pci-gart.c	2006-02-06 21:33:17.000000000 +0000
@@ -477,7 +477,6 @@ static int __dma_map_cont(struct scatter
 		
 		BUG_ON(i > start && s->offset);
 		if (i == start) {
-			*sout = *s; 
 			sout->dma_address = iommu_bus_base;
 			sout->dma_address += iommu_page*PAGE_SIZE + s->offset;
 			sout->dma_length = s->length;
@@ -492,7 +491,7 @@ static int __dma_map_cont(struct scatter
 			SET_LEAK(iommu_page);
 			addr += PAGE_SIZE;
 			iommu_page++;
-	} 
+		} 
 	} 
 	BUG_ON(iommu_page - iommu_start != pages);	
 	return 0;
@@ -504,7 +503,6 @@ static inline int dma_map_cont(struct sc
 {
 	if (!need) { 
 		BUG_ON(stopat - start != 1);
-		*sout = sg[start]; 
 		sout->dma_length = sg[start].length; 
 		return 0;
 	} 
@@ -622,7 +620,7 @@ void dma_unmap_sg(struct device *dev, st
 	}
 	for (i = 0; i < nents; i++) { 
 		struct scatterlist *s = &sg[i];
-		if (!s->dma_length || !s->length) 
+		if (!s->dma_length) 
 			break;
 		dma_unmap_single(dev, s->dma_address, s->dma_length, dir);
 	}

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[Andi Kleen]

On Monday 06 February 2006 23:11, Hugh Dickins wrote:
> On Mon, 6 Feb 2006, Andi Kleen wrote:
> > On Monday 06 February 2006 17:45, Hugh Dickins wrote:
> > > 
> > > But all this presupposes that someone is suddenly going to change the
> > > x86_64 gart_map_sg (and subfunctions), or else force its iommu=nomerge:
> > > that won't be me.
> > 
> > Ok i changed it to conform to the gospel. I gave it some basic pounding LTP/dd IO with 
> > and without IOMMU force, but it's not that well tested. More testing welcome.
> 
> Great, thanks Andi.  One small correction to the comment...
> 
> > Don't touch the non DMA members in the sg list in dma_map_sg in the IOMMU
> > 
> > Some drivers (in particular ipr) ran into problems because they
> 
> No, the problem hasn't knowingly been sighted on ipr,

Ok. I was wondering anyways why people should use ipr on x86-64.

> it was the 
> st driver that Ryan's been seeing it with - ipr just came from
> my looking around for like instances.

I will fix the comment. Thanks.

-Andi

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[Ryan Richter]

On Mon, Feb 06, 2006 at 10:11:09PM +0000, Hugh Dickins wrote:
> Below is, I think, the 2.6.15 equivalent of the patch Andi posted.
> Ryan cannot effectively test Andi's patch on 2.6.16-rc because Mike
> Christie's scsi_execute_async changes have serendipitously fixed
> the st instance.  Ryan, would you be able to test the patch below
> on 2.6.15 without my st.c,st.h patch?

Sure, I'll try this out.  It'll probably have to wait for the weekend
again.

-ryan

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[Ryan Richter]

On Mon, Feb 06, 2006 at 10:11:09PM +0000, Hugh Dickins wrote:
> Below is, I think, the 2.6.15 equivalent of the patch Andi posted.
> Ryan cannot effectively test Andi's patch on 2.6.16-rc because Mike
> Christie's scsi_execute_async changes have serendipitously fixed
> the st instance.  Ryan, would you be able to test the patch below
> on 2.6.15 without my st.c,st.h patch?

This patch survived 6 runs, and I'll keep running it.

-ryan

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[Hugh Dickins]

On Sat, 11 Feb 2006, Ryan Richter wrote:
> On Mon, Feb 06, 2006 at 10:11:09PM +0000, Hugh Dickins wrote:
> > Below is, I think, the 2.6.15 equivalent of the patch Andi posted.
> > Ryan cannot effectively test Andi's patch on 2.6.16-rc because Mike
> > Christie's scsi_execute_async changes have serendipitously fixed
> > the st instance.  Ryan, would you be able to test the patch below
> > on 2.6.15 without my st.c,st.h patch?
> 
> This patch survived 6 runs, and I'll keep running it.

Many thanks for all your efforts on this, Ryan: for reporting the bug,
for your patience in waiting for a diagnosis and a fix, and for
uncomplainingly testing so many workarounds and fixes.

I'll say a mean thing, but please take it as the compliment it's
intended to be: I hope you find lots more bugs, because you're the
rare someone we can depend on to see them through to the final fix!
Nah, you've had your share, you deserve a break.

Andi, do you feel this x86_64-gart-dma-merge.patch has now had enough
testing?  I've read through your gart_map_sg() more carefully now, and
the patch looks right to me (it seems to be a misunderstanding that the
lines you're now deleting were ever in).  I've also read through, less
carefully, the sg coalescing routines in other architectures, and found
no such offending code in them.  Brian did the same earlier and found
nothing.

So I'd like to believe that your x86_64-gart-dma-merge.patch is the
final answer to this issue, and see it go forward into 2.6.16-rc -
if you feel it's ready now.  Then we can just throw away those driver
patches I posted a week ago (including the "ipr" one of this thread).

Thanks,
Hugh

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[Andi Kleen]

On Sunday 12 February 2006 19:57, Hugh Dickins wrote:

> So I'd like to believe that your x86_64-gart-dma-merge.patch is the
> final answer to this issue, and see it go forward into 2.6.16-rc -
> if you feel it's ready now.  Then we can just throw away those driver
> patches I posted a week ago (including the "ipr" one of this thread).

Yes, it's probably ready to go. I will submit it later if Andrew doesn't
beat me.

-Andi

Re: [PATCH] ipr: don't doublefree pages from scatterlist

[Hugh Dickins]

On Sun, 12 Feb 2006, Andi Kleen wrote:
> On Sunday 12 February 2006 19:57, Hugh Dickins wrote:
> 
> > So I'd like to believe that your x86_64-gart-dma-merge.patch is the
> > final answer to this issue, and see it go forward into 2.6.16-rc -
> > if you feel it's ready now.  Then we can just throw away those driver
> > patches I posted a week ago (including the "ipr" one of this thread).
> 
> Yes, it's probably ready to go. I will submit it later if Andrew doesn't
> beat me.

And you got the fix into 2.6.16-rc3, last on the train: many thanks.
Andrew, please now drop from -mm my unnecessary, driver-complicating

ib-dont-doublefree-pages-from-scatterlist.patch
ipr-dont-doublefree-pages-from-scatterlist.patch
osst-dont-doublefree-pages-from-scatterlist.patch

Thanks,
Hugh