From: Eran Liberty <liberty@extricom.com>
To: Jon Smirl <jonsmirl@gmail.com>
Cc: Steven Rostedt <rostedt@goodmis.org>,
Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>,
linux-kernel@vger.kernel.org, linuxppc-dev@ozlabs.org,
Steven Rostedt <srostedt@redhat.com>,
Alan Modra <amodra@au1.ibm.com>,
Scott Wood <scottwood@freescale.com>,
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Subject: Re: ftrace introduces instability into kernel 2.6.27(-rc2,-rc3)
Date: Wed, 20 Aug 2008 21:23:28 +0300 [thread overview]
Message-ID: <48AC6120.1080400@extricom.com> (raw)
In-Reply-To: <9e4733910808200755y1128ae56p6a1235684bfbb3ec@mail.gmail.com>
Jon Smirl wrote:
> Manually edit the broken binary to change the order of the restore and
> see if the problem disappears. That will keep everything else
> constant.
>
>
checked with objdump where __d_lookup()
> powerpc-linux-gnu-objdump -dr --start-address=0xc0065790 vmlinux |
grep "<__d_lookup>:"
c00b91d8 <__d_lookup>:
> liberty@liberty:~/svn/exsw1600-2.6.27-rc2$ powerpc-linux-gnu-objdump
-dr --start-address=0xc00b91d8 vmlinux| head -n 100
vmlinux: file format elf32-powerpc
Disassembly of section .text:
c00b91d8 <__d_lookup>:
c00b91d8: 7c 08 02 a6 mflr r0
c00b91dc: 90 01 00 04 stw r0,4(r1)
....
c00b92bc: 4e 80 04 21 bctrl
c00b92c0: 2f 83 00 00 cmpwi cr7,r3,0
c00b92c4: 41 9e 00 50 beq- cr7,c00b9314 <__d_lookup+0x13c>
c00b92c8: 83 de 00 00 lwz r30,0(r30)
c00b92cc: 2f 9e 00 00 cmpwi cr7,r30,0
c00b92d0: 40 9e ff 98 bne+ cr7,c00b9268 <__d_lookup+0x90>
c00b92d4: 38 60 00 00 li r3,0
c00b92d8: 81 61 00 00 lwz r11,0(r1)
c00b92dc: 80 0b 00 04 lwz r0,4(r11)
c00b92e0: 7d 61 5b 78 mr r1,r11
<=== As explained by Steve, these two should be replaced ===>
c00b92e4: bb 0b ff e0 lmw r24,-32(r11)
c00b92e8: 7c 08 03 a6 mtlr r0
c00b92ec: 4e 80 00 20 blr
c00b92f0: 80 04 00 04 lwz r0,4(r4)
....
c00b9330: 7f a3 eb 78 mr r3,r29
c00b9334: 4b ff ff a4 b c00b92d8 <__d_lookup+0x100>
c00b9338 <d_lookup>:
c00b9338: 7c 08 02 a6 mflr r0
on the target I fired up xmon and replaced them.
~ # echo x > /proc/sysrq-trigger
SysRq : Entering xmon
Vector: 0 at [df51fdb8]
pc: c0025960: sysrq_handle_xmon+0x60/0x64
lr: c0025960: sysrq_handle_xmon+0x60/0x64
sp: df51fe80
msr: 21000
current = 0xdc22a9a0
pid = 1698, comm = echo
WARNING: exception is not recoverable, can't continue
enter ? for help
[df51fe90] c0193c38 __handle_sysrq+0xa8/0x178
[df51fec0] c00ee818 write_sysrq_trigger+0x78/0x7c
[df51fed0] c00e65e4 proc_reg_write+0x5c/0x84
[df51fef0] c00a299c vfs_write+0xc8/0x180
[df51ff10] c00a2f40 sys_write+0x5c/0xa4
[df51ff40] c0010554 ret_from_syscall+0x0/0x3c
SP (bffe87e0) is in userspace
mon> di c00b92d0
c00b92d0 409eff98 bne cr7,c00b9268 # __d_lookup+0x90/0x160
c00b92d4 38600000 li r3,0
c00b92d8 81610000 lwz r11,0(r1)
c00b92dc 800b0004 lwz r0,4(r11)
c00b92e0 7d615b78 mr r1,r11
<=== wrong order ===>
c00b92e4 bb0bffe0 lmw r24,-32(r11)
c00b92e8 7c0803a6 mtlr r0
c00b92ec 4e800020 blr
c00b92f0 80040004 lwz r0,4(r4)
c00b92f4 7f80c800 cmpw cr7,r0,r25
c00b92f8 409eff64 bne cr7,c00b925c # __d_lookup+0x84/0x160
c00b92fc 80640008 lwz r3,8(r4)
c00b9300 7f25cb78 mr r5,r25
c00b9304 7f04c378 mr r4,r24
c00b9308 4bf5ccf1 bl c0015ff8 # memcmp+0x0/0x30
c00b930c 2f830000 cmpwi cr7,r3,0
mon> m c00b92e0
c00b92e0 7d l
c00b92e0 7d615b78 bb0bffe0
c00b92e4 bb0bffe0 7d615b78
c00b92e8 7c0803a6 x
mon> di c00b92d0
c00b92d0 409eff98 bne cr7,c00b9268 # __d_lookup+0x90/0x160
c00b92d4 38600000 li r3,0
c00b92d8 81610000 lwz r11,0(r1)
c00b92dc 800b0004 lwz r0,4(r11)
c00b92e0 bb0bffe0 lmw r24,-32(r11)
<=== right order ===>
c00b92e4 7d615b78 mr r1,r11
c00b92e8 7c0803a6 mtlr r0
c00b92ec 4e800020 blr
c00b92f0 80040004 lwz r0,4(r4)
c00b92f4 7f80c800 cmpw cr7,r0,r25
c00b92f8 409eff64 bne cr7,c00b925c # __d_lookup+0x84/0x160
c00b92fc 80640008 lwz r3,8(r4)
c00b9300 7f25cb78 mr r5,r25
c00b9304 7f04c378 mr r4,r24
c00b9308 4bf5ccf1 bl c0015ff8 # memcmp+0x0/0x30
c00b930c 2f830000 cmpwi cr7,r3,0
mon> X
~ #
And took it for a test drive
~# while [ 1 ] ; do find / > /dev/null ; echo . ; done
.
.
.
[ 10 min later ]
.
.
.
Stable! Yeepee.
So how do we get this to be right in the first place and everywhere?
-- Liberty
next prev parent reply other threads:[~2008-08-20 18:25 UTC|newest]
Thread overview: 81+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-06-18 14:18 2.6.24-rc4: pci_remove_bus_device() => pci_scan_child_bus() => pci_bus_add_devices bug? Eran Liberty
2008-07-20 10:31 ` [PATCH 2.6.24-rc4] PCI: refuse to re-add a device to a bus upon pci_scan_child_bus() Eran Liberty
2008-07-20 16:48 ` Eran Liberty
2008-07-21 19:18 ` [PATCH 2.6.26] " Eran Liberty
2008-07-21 19:49 ` Matthew Wilcox
2008-07-22 8:21 ` eran liberty
2008-07-22 11:49 ` Matthew Wilcox
2008-07-22 13:08 ` Eran Liberty
2008-07-22 13:14 ` Eran Liberty
2008-07-22 14:13 ` Matthew Wilcox
2008-07-22 15:25 ` Eran Liberty
2008-07-22 16:52 ` Matthew Wilcox
2008-07-22 17:41 ` Eran Liberty
2008-07-22 18:11 ` Matthew Wilcox
2008-07-23 18:31 ` Eran Liberty
2008-07-27 11:01 ` Eran Liberty
2008-07-27 15:08 ` Matthew Wilcox
2008-08-18 8:08 ` ftrace introduces instability into kernel 2.6.27(-rc2,-rc3) Eran Liberty
2008-08-18 15:07 ` Steven Rostedt
2008-08-18 15:47 ` Mathieu Desnoyers
2008-08-18 16:12 ` Steven Rostedt
2008-08-18 17:04 ` Mathieu Desnoyers
2008-08-18 17:21 ` Scott Wood
2008-08-18 18:27 ` Steven Rostedt
2008-08-18 18:29 ` Scott Wood
2008-08-19 1:53 ` Benjamin Herrenschmidt
2008-08-19 2:28 ` Steven Rostedt
2008-08-19 2:39 ` Benjamin Herrenschmidt
2008-08-19 2:41 ` Steven Rostedt
2008-08-19 2:47 ` Mathieu Desnoyers
2008-08-19 3:32 ` Steven Rostedt
2008-08-19 3:36 ` Mathieu Desnoyers
2008-08-19 4:00 ` Steven Rostedt
2008-08-19 16:47 ` Steven Rostedt
2008-08-19 17:34 ` Mathieu Desnoyers
2008-08-19 21:08 ` Steven Rostedt
2008-08-20 9:40 ` Nick Piggin
2008-08-19 21:47 ` Benjamin Herrenschmidt
2008-08-19 23:58 ` Jeremy Fitzhardinge
2008-08-20 1:17 ` Benjamin Herrenschmidt
2008-08-19 2:56 ` Benjamin Herrenschmidt
2008-08-19 3:12 ` Steven Rostedt
2008-08-19 4:17 ` Benjamin Herrenschmidt
2008-08-20 7:18 ` Benjamin Herrenschmidt
2008-08-20 13:14 ` Steven Rostedt
2008-08-20 13:19 ` Steven Rostedt
2008-08-20 13:36 ` Eran Liberty
2008-08-20 13:43 ` Steven Rostedt
2008-08-20 14:02 ` Eran Liberty
2008-08-20 14:55 ` Jon Smirl
2008-08-20 15:23 ` Steven Rostedt
2008-08-20 18:23 ` Eran Liberty [this message]
2008-08-20 18:33 ` Steven Rostedt
2008-08-20 15:27 ` Steven Rostedt
2008-08-20 21:37 ` Benjamin Herrenschmidt
2008-08-20 14:16 ` Josh Boyer
2008-08-20 14:22 ` Steven Rostedt
2008-08-20 14:50 ` Josh Boyer
2008-09-15 16:30 ` [PATCH 2.6.26] SERIAL DRIVER: Handle Multiple consecutive sysrq from the serial Eran Liberty
2008-09-17 23:46 ` Andrew Morton
2008-09-18 6:58 ` Eran Liberty
2008-08-20 21:36 ` ftrace introduces instability into kernel 2.6.27(-rc2,-rc3) Benjamin Herrenschmidt
2008-08-20 21:44 ` Steven Rostedt
2008-08-18 18:47 ` Steven Rostedt
2008-08-18 18:56 ` Scott Wood
2008-08-18 19:28 ` Steven Rostedt
2008-08-18 18:25 ` Eran Liberty
2008-08-18 18:41 ` Mathieu Desnoyers
2008-08-19 1:54 ` Benjamin Herrenschmidt
2008-08-19 9:56 ` Eran Liberty
2008-08-19 13:02 ` Mathieu Desnoyers
2008-08-19 21:46 ` Benjamin Herrenschmidt
2008-08-18 18:50 ` Steven Rostedt
2008-08-19 12:09 ` Eran Liberty
2008-08-19 13:05 ` Mathieu Desnoyers
2008-08-19 14:21 ` Eran Liberty
2008-08-19 14:42 ` Mathieu Desnoyers
2008-08-19 20:15 ` Steven Rostedt
2008-08-20 11:18 ` Eran Liberty
2008-08-20 13:12 ` Steven Rostedt
2008-08-19 1:51 ` Benjamin Herrenschmidt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=48AC6120.1080400@extricom.com \
--to=liberty@extricom.com \
--cc=amodra@au1.ibm.com \
--cc=jonsmirl@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linuxppc-dev@ozlabs.org \
--cc=mathieu.desnoyers@polymtl.ca \
--cc=paulmck@linux.vnet.ibm.com \
--cc=rostedt@goodmis.org \
--cc=scottwood@freescale.com \
--cc=srostedt@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).