bad pmd

bad pmd

[Aric D. Blumer]

Hi.  I'm using the long-term stable kernel 2.6.32 on a PXA320 platform,
and I'm seeing errors like the following:

/home/aric/sdg/git/linux/mm/memory.c:144: bad pmd 8040542e.

I have seen these messages on both the 2.6.32.15 and 2.6.32.24 kernels
(haven't tried others).  Can someone tell me what the message means?  I
suspect memory is being clobbered.  One interesting thing is that
whenever that message is printed, the 8040542e is always the same.  I
have not been able to establish any correlation yet with what causes it.

Thanks for any insight you can give.

Aric.

bad pmd

[Russell King - ARM Linux]

On Wed, Dec 01, 2010 at 02:54:26PM -0500, Aric D. Blumer wrote:
> Hi.  I'm using the long-term stable kernel 2.6.32 on a PXA320 platform,
> and I'm seeing errors like the following:
> 
> /home/aric/sdg/git/linux/mm/memory.c:144: bad pmd 8040542e.
> 
> I have seen these messages on both the 2.6.32.15 and 2.6.32.24 kernels
> (haven't tried others).  Can someone tell me what the message means?  I
> suspect memory is being clobbered.  One interesting thing is that
> whenever that message is printed, the 8040542e is always the same.  I
> have not been able to establish any correlation yet with what causes it.

A pmd value of 0x8040542e is a section mapping, which the generic MM
code will not understand.

It is for address 0x80400000, is read/writable from SVC mode, inaccessible
from user mode, domain 1 (which is normally for 'user' memory), and has
a memory type of TEXCB=10111.

As standard mainline doesn't create mappings with TEX=101, and we don't
create mappings with the 'user' domain using sections, the question this
immediately raises is: have you modified this kernel?

bad pmd

[Aric D. Blumer]

On 12/01/2010 03:14 PM, Russell King - ARM Linux wrote:
> On Wed, Dec 01, 2010 at 02:54:26PM -0500, Aric D. Blumer wrote:
>> Hi.  I'm using the long-term stable kernel 2.6.32 on a PXA320 platform,
>> and I'm seeing errors like the following:
>>
>> /home/aric/sdg/git/linux/mm/memory.c:144: bad pmd 8040542e.
>>
>> I have seen these messages on both the 2.6.32.15 and 2.6.32.24 kernels
>> (haven't tried others).  Can someone tell me what the message means?  I
>> suspect memory is being clobbered.  One interesting thing is that
>> whenever that message is printed, the 8040542e is always the same.  I
>> have not been able to establish any correlation yet with what causes it.
> A pmd value of 0x8040542e is a section mapping, which the generic MM
> code will not understand.
>
> It is for address 0x80400000, is read/writable from SVC mode, inaccessible
> from user mode, domain 1 (which is normally for 'user' memory), and has
> a memory type of TEXCB=10111.
>
> As standard mainline doesn't create mappings with TEX=101, and we don't
> create mappings with the 'user' domain using sections, the question this
> immediately raises is: have you modified this kernel?

Thanks for the info, Russell.  We have modified this kernel in two
ways:  1) We have added code to support the platform (GPIOs,
touchscreen, bluetooth UART, etc.).   2) It has the patches for Android
merged in.

It doesn't look like the Android patches do any mappings different from
mainline, but the bad entry looks very much like a real page table
entry.  But, supposing that memory is being trampled, can any driver
mess up the page tables, or is a special processor mode required?  Could
a rogue DMA trample page table memory?  Can you suggest how to determine
what the address of the bad page table entry is?

I'll start removing non-critical drivers to see if I can isolate the
cause, but I have a bit more information in the meantime:  I put
__backtrace() into pmd_clear_bad(), and I always see a read() system
call sequence like this when the error occurs:

[    8.894213] /home/aric/sdg/git/linux/mm/memory.c:144: bad pmd 8040542e.
[    8.901133] [<c0099d50>] (pmd_clear_bad+0x0/0x40) from [<c00a4b18>]
(walk_page_range+0x22c/0x230)
[    8.910128]  r4:80600000
[    8.912839] [<c00a48ec>] (walk_page_range+0x0/0x230) from
[<c00eecf8>] (show_smap+0x84/0x17c)
[    8.921589] [<c00eec74>] (show_smap+0x0/0x17c) from [<c00ca60c>]
(seq_read+0x314/0x48c)
[    8.929810] [<c00ca2f8>] (seq_read+0x0/0x48c) from [<c00b1348>]
(vfs_read+0xb8/0x16c)
[    8.937761] [<c00b1290>] (vfs_read+0x0/0x16c) from [<c00b16c8>]
(sys_read+0x44/0x74)
[    8.945603]  r8:c002e108 r7:00000000 r6:00012000 r5:fffffff7 r4:cf32bf80
[    8.952644] [<c00b1684>] (sys_read+0x0/0x74) from [<c002df60>]
(ret_fast_syscall+0x0/0x28)
[    8.961019]  r7:00000003 r6:5a5cbc68 r5:afe14cfd r4:afe3bdfc
[    8.968572] /home/aric/sdg/git/linux/mm/memory.c:144: bad pmd 8040542e.
[    8.975402] [<c0099d50>] (pmd_clear_bad+0x0/0x40) from [<c00a4b18>]
(walk_page_range+0x22c/0x230)
[    8.984386]  r4:80c00000
[    8.987062] [<c00a48ec>] (walk_page_range+0x0/0x230) from
[<c00eecf8>] (show_smap+0x84/0x17c)
[    8.995789] [<c00eec74>] (show_smap+0x0/0x17c) from [<c00ca60c>]
(seq_read+0x314/0x48c)
[    9.003994] [<c00ca2f8>] (seq_read+0x0/0x48c) from [<c00b1348>]
(vfs_read+0xb8/0x16c)
[    9.012013] [<c00b1290>] (vfs_read+0x0/0x16c) from [<c00b16c8>]
(sys_read+0x44/0x74)
[    9.019908]  r8:c002e108 r7:00000000 r6:00012800 r5:fffffff7 r4:cf32bf80
[    9.026846] [<c00b1684>] (sys_read+0x0/0x74) from [<c002df60>]
(ret_fast_syscall+0x0/0x28)
[    9.035224]  r7:00000003 r6:5a5cbc40 r5:afe14cfd r4:afe3bdfc

bad pmd

[Aric D. Blumer]

On 12/01/2010 09:35 PM, Aric D. Blumer wrote:
> On 12/01/2010 03:14 PM, Russell King - ARM Linux wrote:
>> On Wed, Dec 01, 2010 at 02:54:26PM -0500, Aric D. Blumer wrote:
>>> Hi.  I'm using the long-term stable kernel 2.6.32 on a PXA320 platform,
>>> and I'm seeing errors like the following:
>>>
>>> /home/aric/sdg/git/linux/mm/memory.c:144: bad pmd 8040542e.
>>>
>>> I have seen these messages on both the 2.6.32.15 and 2.6.32.24 kernels
>>> (haven't tried others).  Can someone tell me what the message means?  I
>>> suspect memory is being clobbered.  One interesting thing is that
>>> whenever that message is printed, the 8040542e is always the same.  I
>>> have not been able to establish any correlation yet with what causes it.
>> A pmd value of 0x8040542e is a section mapping, which the generic MM
>> code will not understand.
>>
>> It is for address 0x80400000, is read/writable from SVC mode, inaccessible
>> from user mode, domain 1 (which is normally for 'user' memory), and has
>> a memory type of TEXCB=10111.
>>
>> As standard mainline doesn't create mappings with TEX=101, and we don't
>> create mappings with the 'user' domain using sections, the question this
>> immediately raises is: have you modified this kernel?
> Thanks for the info, Russell.  We have modified this kernel in two
> ways:  1) We have added code to support the platform (GPIOs,
> touchscreen, bluetooth UART, etc.).   2) It has the patches for Android
> merged in.
>
> It doesn't look like the Android patches do any mappings different from
> mainline, but the bad entry looks very much like a real page table
> entry.  But, supposing that memory is being trampled, can any driver
> mess up the page tables, or is a special processor mode required?  Could
> a rogue DMA trample page table memory?  Can you suggest how to determine
> what the address of the bad page table entry is?
>
> I'll start removing non-critical drivers to see if I can isolate the
> cause. . . .

Matt Reimer and I believe we have found what is going on here.  I've put
in a fix (no failures yet), but I wanted to bounce it off anyone interested.

The PXA platform does not use the "bad pmd" mapping that Russell
describes above under normal circumstances, but the PXA resume code
(arch/arm/mach-pxa/sleep.S) does on resume:

        @ temporarily map resume_turn_on_mmu into the page table,
        @ otherwise prefetch abort occurs after MMU is turned on
        mov     r1, r7
        bic     r1, r1, #0x00ff
        bic     r1, r1, #0x3f00
        ldr     r2, =0x542e

        adr     r3, resume_turn_on_mmu
        mov     r3, r3, lsr #20
        orr     r4, r2, r3, lsl #20
        ldr     r5, [r1, r3, lsl #2]
        str     r4, [r1, r3, lsl #2]

        @ Mapping page table address in the page table
        mov     r6, r1, lsr #20
        orr     r7, r2, r6, lsl #20
        ldr     r8, [r1, r6, lsl #2]
        str     r7, [r1, r6, lsl #2]

The first four lines of code is where the 0x8040542e page table entry
comes from.  The two 'bic' instructions ensure that r7 contains only the
page table address and not any of the status bits.  This code is fine. 
The problem lies with pxa3xx_resume_after_mmu assuming that r1 is
unmodified, but resume_turn_on_mmu clobbers r1.  It clobbers it with a
read of the page table address, but it does not mask the lower bits of
that register:

bad pmd

[Russell King - ARM Linux]

On Tue, Dec 07, 2010 at 12:26:45PM -0500, Aric D. Blumer wrote:
> Matt Reimer and I believe we have found what is going on here.  I've put
> in a fix (no failures yet), but I wanted to bounce it off anyone interested.
> 
> The PXA platform does not use the "bad pmd" mapping that Russell
> describes above under normal circumstances, but the PXA resume code
> (arch/arm/mach-pxa/sleep.S) does on resume:

Good find.

> I'm using the patch below as a fix for now, but it's hard to know what
> registers are available.  Might be better to just mask it off the lower
> bits in r1 again.
> 
>         @ Let us ensure we jump to resume_after_mmu only when the mcr above
>         @ actually took effect.  They call it the "cpwait" operation.
> -       mrc     p15, 0, r1, c2, c0, 0           @ queue a dependency on CP15
> -       sub     pc, r2, r1, lsr #32             @ jump to virtual addr
> +       mrc     p15, 0, r0, c2, c0, 0           @ queue a dependency on CP15
> +       sub     pc, r2, r0, lsr #32             @ jump to virtual addr

I don't see anything wrong with this.  Any PXA people want to ack this?

bad pmd

[Matt Reimer]

On Wed, Dec 8, 2010 at 9:02 AM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Tue, Dec 07, 2010 at 12:26:45PM -0500, Aric D. Blumer wrote:
>> Matt Reimer and I believe we have found what is going on here. ?I've put
>> in a fix (no failures yet), but I wanted to bounce it off anyone interested.
>>
>> The PXA platform does not use the "bad pmd" mapping that Russell
>> describes above under normal circumstances, but the PXA resume code
>> (arch/arm/mach-pxa/sleep.S) does on resume:
>
> Good find.
>
>> I'm using the patch below as a fix for now, but it's hard to know what
>> registers are available. ?Might be better to just mask it off the lower
>> bits in r1 again.
>>
>> ? ? ? ? @ Let us ensure we jump to resume_after_mmu only when the mcr above
>> ? ? ? ? @ actually took effect. ?They call it the "cpwait" operation.
>> - ? ? ? mrc ? ? p15, 0, r1, c2, c0, 0 ? ? ? ? ? @ queue a dependency on CP15
>> - ? ? ? sub ? ? pc, r2, r1, lsr #32 ? ? ? ? ? ? @ jump to virtual addr
>> + ? ? ? mrc ? ? p15, 0, r0, c2, c0, 0 ? ? ? ? ? @ queue a dependency on CP15
>> + ? ? ? sub ? ? pc, r2, r0, lsr #32 ? ? ? ? ? ? @ jump to virtual addr
>
> I don't see anything wrong with this. ?Any PXA people want to ack this?

This patch fixes the problem for us. We have yet to see a failure with
this patch applied.

Eric, do you concur that this is a proper fix?

Matt

bad pmd

[Eric Miao]

On Tue, Dec 14, 2010 at 11:08 PM, Matt Reimer <mreimer@sdgsystems.com> wrote:
> On Wed, Dec 8, 2010 at 9:02 AM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
>> On Tue, Dec 07, 2010 at 12:26:45PM -0500, Aric D. Blumer wrote:
>>> Matt Reimer and I believe we have found what is going on here. ?I've put
>>> in a fix (no failures yet), but I wanted to bounce it off anyone interested.
>>>
>>> The PXA platform does not use the "bad pmd" mapping that Russell
>>> describes above under normal circumstances, but the PXA resume code
>>> (arch/arm/mach-pxa/sleep.S) does on resume:
>>
>> Good find.
>>
>>> I'm using the patch below as a fix for now, but it's hard to know what
>>> registers are available. ?Might be better to just mask it off the lower
>>> bits in r1 again.
>>>
>>> ? ? ? ? @ Let us ensure we jump to resume_after_mmu only when the mcr above
>>> ? ? ? ? @ actually took effect. ?They call it the "cpwait" operation.
>>> - ? ? ? mrc ? ? p15, 0, r1, c2, c0, 0 ? ? ? ? ? @ queue a dependency on CP15
>>> - ? ? ? sub ? ? pc, r2, r1, lsr #32 ? ? ? ? ? ? @ jump to virtual addr
>>> + ? ? ? mrc ? ? p15, 0, r0, c2, c0, 0 ? ? ? ? ? @ queue a dependency on CP15
>>> + ? ? ? sub ? ? pc, r2, r0, lsr #32 ? ? ? ? ? ? @ jump to virtual addr
>>
>> I don't see anything wrong with this. ?Any PXA people want to ack this?

Me neither.

Acked-by: Eric Miao <eric.y.miao@gmail.com>

>
> This patch fixes the problem for us. We have yet to see a failure with
> this patch applied.
>
> Eric, do you concur that this is a proper fix?
>

This patch is good. Thanks for figuring this out. My bad replying so
late.

Russell,

Still time for this fix to get into .37?

> Matt
>

[PATCH] [ARM] pxa: fix page table corruption on resume

[Matt Reimer]

From: Aric D. Blumer <aric@sdgsystems.com>

Before this patch, the following error would sometimes occur after a
resume on pxa3xx:

    /path/to/mm/memory.c:144: bad pmd 8040542e.

The problem was that a temporary page table mapping was being improperly
restored.

The PXA3xx resume code creates a temporary mapping of resume_turn_on_mmu
to avoid a prefetch abort.  The pxa3xx_resume_after_mmu code requires
that the r1 register holding the address of this mapping not be
modified, however, resume_turn_on_mmu does modify it. It is mostly
correct in that r1 receives the base table address, but it may also
get other bits in 13:0.  This results in pxa3xx_resume_after_mmu
restoring the original mapping to the wrong place, corrupting memory
and leaving the temporary mapping in place.

Signed-off-by: Matt Reimer <mreimer@sdgsystems.com>
---
 arch/arm/mach-pxa/sleep.S |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/arm/mach-pxa/sleep.S b/arch/arm/mach-pxa/sleep.S
index 52c30b0..ae00811 100644
--- a/arch/arm/mach-pxa/sleep.S
+++ b/arch/arm/mach-pxa/sleep.S
@@ -353,8 +353,8 @@ resume_turn_on_mmu:
 
 	@ Let us ensure we jump to resume_after_mmu only when the mcr above
 	@ actually took effect.  They call it the "cpwait" operation.
-	mrc	p15, 0, r1, c2, c0, 0		@ queue a dependency on CP15
-	sub	pc, r2, r1, lsr #32		@ jump to virtual addr
+	mrc	p15, 0, r0, c2, c0, 0		@ queue a dependency on CP15
+	sub	pc, r2, r0, lsr #32		@ jump to virtual addr
 	nop
 	nop
 	nop
-- 
1.7.0.4

bad pmd

[Matt Reimer]

On Tue, Dec 28, 2010 at 11:51 PM, Eric Miao <eric.y.miao@gmail.com> wrote:
> On Tue, Dec 14, 2010 at 11:08 PM, Matt Reimer <mreimer@sdgsystems.com> wrote:
>> On Wed, Dec 8, 2010 at 9:02 AM, Russell King - ARM Linux
>> <linux@arm.linux.org.uk> wrote:
>>> On Tue, Dec 07, 2010 at 12:26:45PM -0500, Aric D. Blumer wrote:
>>>> Matt Reimer and I believe we have found what is going on here. ?I've put
>>>> in a fix (no failures yet), but I wanted to bounce it off anyone interested.
>>>>
>>>> The PXA platform does not use the "bad pmd" mapping that Russell
>>>> describes above under normal circumstances, but the PXA resume code
>>>> (arch/arm/mach-pxa/sleep.S) does on resume:
>>>
>>> Good find.
>>>
>>>> I'm using the patch below as a fix for now, but it's hard to know what
>>>> registers are available. ?Might be better to just mask it off the lower
>>>> bits in r1 again.
>>>>
>>>> ? ? ? ? @ Let us ensure we jump to resume_after_mmu only when the mcr above
>>>> ? ? ? ? @ actually took effect. ?They call it the "cpwait" operation.
>>>> - ? ? ? mrc ? ? p15, 0, r1, c2, c0, 0 ? ? ? ? ? @ queue a dependency on CP15
>>>> - ? ? ? sub ? ? pc, r2, r1, lsr #32 ? ? ? ? ? ? @ jump to virtual addr
>>>> + ? ? ? mrc ? ? p15, 0, r0, c2, c0, 0 ? ? ? ? ? @ queue a dependency on CP15
>>>> + ? ? ? sub ? ? pc, r2, r0, lsr #32 ? ? ? ? ? ? @ jump to virtual addr
>>>
>>> I don't see anything wrong with this. ?Any PXA people want to ack this?
>
> Me neither.
>
> Acked-by: Eric Miao <eric.y.miao@gmail.com>
>
>>
>> This patch fixes the problem for us. We have yet to see a failure with
>> this patch applied.
>>
>> Eric, do you concur that this is a proper fix?
>>
>
> This patch is good. Thanks for figuring this out. My bad replying so
> late.
>
> Russell,
>
> Still time for this fix to get into .37?

I've just resent the patch with a proper commit message.

Matt

bad pmd

[Russell King - ARM Linux]

On Wed, Dec 29, 2010 at 11:23:18AM -0500, Matt Reimer wrote:
> On Tue, Dec 28, 2010 at 11:51 PM, Eric Miao <eric.y.miao@gmail.com> wrote:
> > Russell,
> >
> > Still time for this fix to get into .37?

Yes, if someone collects up the acks and queues it in the patch system.

> I've just resent the patch with a proper commit message.
> 
> Matt

[PATCH] [ARM] pxa: fix page table corruption on resume

[Eric Miao]

On Thu, Dec 30, 2010 at 12:18 AM, Matt Reimer <mreimer@sdgsystems.com> wrote:
> From: Aric D. Blumer <aric@sdgsystems.com>
>
> Before this patch, the following error would sometimes occur after a
> resume on pxa3xx:
>
> ? ?/path/to/mm/memory.c:144: bad pmd 8040542e.
>
> The problem was that a temporary page table mapping was being improperly
> restored.
>
> The PXA3xx resume code creates a temporary mapping of resume_turn_on_mmu
> to avoid a prefetch abort. ?The pxa3xx_resume_after_mmu code requires
> that the r1 register holding the address of this mapping not be
> modified, however, resume_turn_on_mmu does modify it. It is mostly
> correct in that r1 receives the base table address, but it may also
> get other bits in 13:0. ?This results in pxa3xx_resume_after_mmu
> restoring the original mapping to the wrong place, corrupting memory
> and leaving the temporary mapping in place.
>
> Signed-off-by: Matt Reimer <mreimer@sdgsystems.com>

Applied.

> ---
> ?arch/arm/mach-pxa/sleep.S | ? ?4 ++--
> ?1 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm/mach-pxa/sleep.S b/arch/arm/mach-pxa/sleep.S
> index 52c30b0..ae00811 100644
> --- a/arch/arm/mach-pxa/sleep.S
> +++ b/arch/arm/mach-pxa/sleep.S
> @@ -353,8 +353,8 @@ resume_turn_on_mmu:
>
> ? ? ? ?@ Let us ensure we jump to resume_after_mmu only when the mcr above
> ? ? ? ?@ actually took effect. ?They call it the "cpwait" operation.
> - ? ? ? mrc ? ? p15, 0, r1, c2, c0, 0 ? ? ? ? ? @ queue a dependency on CP15
> - ? ? ? sub ? ? pc, r2, r1, lsr #32 ? ? ? ? ? ? @ jump to virtual addr
> + ? ? ? mrc ? ? p15, 0, r0, c2, c0, 0 ? ? ? ? ? @ queue a dependency on CP15
> + ? ? ? sub ? ? pc, r2, r0, lsr #32 ? ? ? ? ? ? @ jump to virtual addr
> ? ? ? ?nop
> ? ? ? ?nop
> ? ? ? ?nop
> --
> 1.7.0.4
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>