Mapping an executable page

Mapping an executable page

[Thomas De Schampheleire]

Hi,

To cover a specific reset scenario, I need to jump back to the reset
vector of a powerpc processor (e500mc core). In order to be able to
jump there directly, the code where I jump to should have a TLB
mapping associated with it.

I tried achieving this as follows:

                typedef void (*funcptr)(void);

                void __iomem *vaddr = __ioremap(0xfffff000, 0x1000,
(_PAGE_BASE | _PAGE_KERNEL_RWX));
                printk(KERN_ERR "reboot_helper: 0xfffff000 mapped to
%p\n", vaddr);

                /* Disable interrupts to avoid the boot code to be
interrupted */
                local_irq_disable();

                funcptr resetvector = (funcptr)(vaddr + 0xfec);
                resetvector();

Unfortunately, I'm experiencing problems with this approach. I get :

[   23.384639] reboot_helper: event: val=1
[   23.384699] reboot_helper: 0xfffff000 mapped to f127e000
[   23.384781] reboot_helper: 0xfffe1000 mapped to f1420000
[   23.384856] Unable to handle kernel paging request for instruction fetch
[   23.384949] Faulting instruction address: 0xf126b8d0
[   23.385021] Oops: Kernel access of bad area, sig: 11 [#1]
[   23.385096] P4080 DS
[   23.385129] last sysfs file: /sys/class/uio/uio0/name
[   23.385200] Modules linked in: reboot_helper
[   23.385310] NIP: f126b8d0 LR: f127a190 CTR: f127efec
[   23.385382] REGS: ec459cf0 TRAP: 0400   Not tainted  (2.6.34.6-hg378747c1a102
-dirty)
[   23.385489] MSR: 00029002 <EE,ME,CE>  CR: 22002082  XER: 20000000
[   23.385591] TASK = ec08a590[1094] 'init' THREAD: ec458000
[   23.385664] GPR00: 00000001 ec459da0 ec08a590 00000042 0000388b ffffffff c01e
4388 00000000
[   23.385800] GPR08: 00000001 c0490000 00000001 c04972d8 0fffffff 100bea58 0000
0000 00000201
[   23.385936] GPR16: ff800002 ff80003f 00000000 00000000 00000001 effff000 c04b
8000 00000000
[   23.386072] GPR24: bfdb9be8 100891a4 bfdb9d7c 00000000 00000000 fffffffe 0000
0001 f127efec
[   23.386216] NIP [f126b8d0] 0xf126b8d0
[   23.386275] LR [f127a190] isam_reboot_handler+0xa0/0xc4 [reboot_helper]
[   23.386366] Call Trace:
[   23.386410] [ec459da0] [f127a168] isam_reboot_handler+0x78/0xc4 [reboot_helpe
r] (unreliable)
[   23.386534] [ec459db0] [c00422a0] notifier_call_chain+0x5c/0xc8
[   23.386624] [ec459dd0] [c00426d4] __blocking_notifier_call_chain+0x5c/0x88
[   23.386725] [ec459e00] [c0036850] kernel_restart_prepare+0x20/0x44
[   23.386816] [ec459e10] [c00368c4] kernel_restart+0x18/0x5c
[   23.386899] [ec459e20] [c0036a94] sys_reboot+0x184/0x1cc
[   23.386980] [ec459f40] [c000fbe0] ret_from_syscall+0x0/0x3c
[   23.387059] Instruction dump:
[   23.387104] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XX
XXXXXX
[   23.387228] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XX
XXXXXX
[   23.387355] ---[ end trace 48808de79275a83d ]---


Although I realize that what I need to achieve is unconventional, what
is the correct way of mapping a certain address range into memory, and
be able to execute from it?

Thanks,
Thomas

Re: Mapping an executable page

[Tabi Timur-B04825]

On Fri, May 27, 2011 at 8:25 AM, Thomas De Schampheleire
<patrickdepinguin+linuxppc@gmail.com> wrote:

> Although I realize that what I need to achieve is unconventional, what
> is the correct way of mapping a certain address range into memory, and
> be able to execute from it?

Have you tried looking at the actual TLB entry for this page to see if
it's correct?  Also, you might need to do some kind of instruction
cache flushing before you jump to that page.

--=20
Timur Tabi
Linux kernel developer at Freescale=

Re: Mapping an executable page

[McClintock Matthew-B29882]

On Fri, May 27, 2011 at 8:25 AM, Thomas De Schampheleire
<patrickdepinguin+linuxppc@gmail.com> wrote:
> Although I realize that what I need to achieve is unconventional, what
> is the correct way of mapping a certain address range into memory, and
> be able to execute from it?

Can you look at using mpic_reset_core in arch/powerpc/sysdev/mpic.c?
kexec on 85xx uses this to reset other cores the the hold off spin
loop.

-M=

Re: Mapping an executable page

[Thomas De Schampheleire]

Hi,

On Sun, May 29, 2011 at 4:53 PM, Tabi Timur-B04825 <B04825@freescale.com> w=
rote:
> On Fri, May 27, 2011 at 8:25 AM, Thomas De Schampheleire
> <patrickdepinguin+linuxppc@gmail.com> wrote:
>
>> Although I realize that what I need to achieve is unconventional, what
>> is the correct way of mapping a certain address range into memory, and
>> be able to execute from it?
>
> Have you tried looking at the actual TLB entry for this page to see if
> it's correct? =A0Also, you might need to do some kind of instruction
> cache flushing before you jump to that page.

Sorry to have delayed this so long.
I had to do quite some experimentation to get it finally working. Here
are my findings:
* to map a page as executable, the following does indeed work:
void __iomem *vaddr =3D __ioremap(map_start, map_size, (_PAGE_BASE |
_PAGE_KERNEL_RWX));

* However, if you jump to an address in that page, you'll have to make
sure that the entire code that executes is mapped (make map_size large
enough).

* When that range spanned multiple pages, I faced the issue of only
one page being actually mapped in the TLBs. My assumption is that the
call to __ioremap not necessarily updates the TLBs, but mainly some
kernel-internal tables. The actual TLB mapping presumably happens when
a data exception occurs.
Unfortunately, since I left the Linux kernel and jumped to other
(boot) code that reassigns the exception vectors, the kernel-internal
tables are not used anymore, and the exception handler cannot update
the TLBs correctly.

* Therefore, to make sure that the mapping I intended with __ioremap()
is actually reflected in the TLB tables, I added dummy reads of each
page in the TLB, prior to jumping to the boot code, as follows:
                /* make sure memory is read, once every 4Kbyte is enough */
                for (p =3D vaddr; p < vaddr + map_size; p +=3D 0x1000) {
                        unsigned long dummy =3D *(volatile unsigned long *)=
p;
                        (void)dummy;
                }

* After these changes (make sure all code is mapped + make sure to
read all pages so that the TLBs are updated), my scenario works fine.

Best regards,
Thomas

Re: Mapping an executable page

[Timur Tabi]

Thomas De Schampheleire wrote:

> * However, if you jump to an address in that page, you'll have to make
> sure that the entire code that executes is mapped (make map_size large
> enough).

Well, that seems obvious.

> * When that range spanned multiple pages, I faced the issue of only
> one page being actually mapped in the TLBs. My assumption is that the
> call to __ioremap not necessarily updates the TLBs, but mainly some
> kernel-internal tables. The actual TLB mapping presumably happens when
> a data exception occurs.

Hmmm.... I find that surprising.  Memory allocated via ioremap() is supposed to
be available in interrupt handlers, where TLB mappings can't be created
on-the-fly.  I'm not sure that your observation is correct.

> * Therefore, to make sure that the mapping I intended with __ioremap()
> is actually reflected in the TLB tables, I added dummy reads of each
> page in the TLB, prior to jumping to the boot code, as follows:
>                 /* make sure memory is read, once every 4Kbyte is enough */
>                 for (p = vaddr; p < vaddr + map_size; p += 0x1000) {

You should at least use PAGE_SIZE instead of 0x1000.

>                         unsigned long dummy = *(volatile unsigned long *)p;
>                         (void)dummy;
>                 }
> 
> * After these changes (make sure all code is mapped + make sure to
> read all pages so that the TLBs are updated), my scenario works fine.

I still find it hard to believe that this is necessary.

-- 
Timur Tabi
Linux kernel developer at Freescale

Re: Mapping an executable page

[Timur Tabi]

Timur Tabi wrote:
> Hmmm.... I find that surprising.  Memory allocated via ioremap() is supposed to
> be available in interrupt handlers, where TLB mappings can't be created
> on-the-fly.  I'm not sure that your observation is correct.

Ok, it turns out I'm wrong.  As long as the page is in the page tables (i.e.
physically present in RAM), you can take a TLB miss in an interrupt handler, and
the TLB miss handler will create a TLB for you.

This means that ...

>                         unsigned long dummy = *(volatile unsigned long *)p;
>                         (void)dummy;
>                 }
> 
> * After these changes (make sure all code is mapped + make sure to
> read all pages so that the TLBs are updated), my scenario works fine.

is not going to work reliably, because it assumes that the TLBs created by your
multiple ioremap() calls will still be there when your code is called.

If you use just a single ioremap() call, but still touch every page, that should
work for you just as well.

-- 
Timur Tabi
Linux kernel developer at Freescale

Re: Mapping an executable page

[Scott Wood]

On Tue, 14 Jun 2011 10:56:31 +0200
Thomas De Schampheleire <patrickdepinguin+linuxppc@gmail.com> wrote:

> * Therefore, to make sure that the mapping I intended with __ioremap()
> is actually reflected in the TLB tables, I added dummy reads of each
> page in the TLB, prior to jumping to the boot code, as follows:
>                 /* make sure memory is read, once every 4Kbyte is enough */
>                 for (p = vaddr; p < vaddr + map_size; p += 0x1000) {
>                         unsigned long dummy = *(volatile unsigned long *)p;
>                         (void)dummy;
>                 }
> 
> * After these changes (make sure all code is mapped + make sure to
> read all pages so that the TLBs are updated), my scenario works fine.

This is fragile -- you are assuming that it's possible to fit this
set of pages in TLB0 all at once, and that none of them will be
evicted/invalidated by the time you're done.

If you really need to do this, I sugest using settlbcam() from
arch/powerpc/mm/fsl_booke_mmu.c to create TLB1 entries with IPROT set.

Better still if you could live with whatever memory the kernel has already
pinned.

-Scott

Re: Mapping an executable page

[Thomas De Schampheleire]

On Tue, Jun 14, 2011 at 10:02 PM, Timur Tabi <timur@freescale.com> wrote:
> Thomas De Schampheleire wrote:
>
>> * However, if you jump to an address in that page, you'll have to make
>> sure that the entire code that executes is mapped (make map_size large
>> enough).
>
> Well, that seems obvious.

Agreed.

>
>> * When that range spanned multiple pages, I faced the issue of only
>> one page being actually mapped in the TLBs. My assumption is that the
>> call to __ioremap not necessarily updates the TLBs, but mainly some
>> kernel-internal tables. The actual TLB mapping presumably happens when
>> a data exception occurs.
>
> Hmmm.... I find that surprising. =A0Memory allocated via ioremap() is sup=
posed to
> be available in interrupt handlers, where TLB mappings can't be created
> on-the-fly. =A0I'm not sure that your observation is correct.
>
>> * Therefore, to make sure that the mapping I intended with __ioremap()
>> is actually reflected in the TLB tables, I added dummy reads of each
>> page in the TLB, prior to jumping to the boot code, as follows:
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* make sure memory is read, once every =
4Kbyte is enough */
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 for (p =3D vaddr; p < vaddr + map_size; =
p +=3D 0x1000) {
>
> You should at least use PAGE_SIZE instead of 0x1000.

Thanks, I fixed this.

Thomas

Re: Mapping an executable page

[Thomas De Schampheleire]

On Tue, Jun 14, 2011 at 10:07 PM, Timur Tabi <timur@freescale.com> wrote:
> Timur Tabi wrote:
>> Hmmm.... I find that surprising. =A0Memory allocated via ioremap() is su=
pposed to
>> be available in interrupt handlers, where TLB mappings can't be created
>> on-the-fly. =A0I'm not sure that your observation is correct.
>
> Ok, it turns out I'm wrong. =A0As long as the page is in the page tables =
(i.e.
> physically present in RAM), you can take a TLB miss in an interrupt handl=
er, and
> the TLB miss handler will create a TLB for you.
>
> This means that ...
>
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 unsigned long dummy =3D =
*(volatile unsigned long *)p;
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 (void)dummy;
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 }
>>
>> * After these changes (make sure all code is mapped + make sure to
>> read all pages so that the TLBs are updated), my scenario works fine.
>
> is not going to work reliably, because it assumes that the TLBs created b=
y your
> multiple ioremap() calls will still be there when your code is called.
>
> If you use just a single ioremap() call, but still touch every page, that=
 should
> work for you just as well.

I am using a single __ioremap call.
You have a point about the reliability of this: if an interrupt occurs
between the mapping or dummy reading, and the point where the actual
code is executing, some TLB entries may have been replaced, right?
I think I can make it more reliable by dummy reading the pages *after*
I disabled interrupts on that processor, immediately before jumping to
the boot code. Is that correct?
(note that I have to disable interrupts anyhow for the boot code to
work properly without interruptions to 'linux land'.

Thanks for your input,
Thomas

Re: Mapping an executable page

[Thomas De Schampheleire]

On Tue, Jun 14, 2011 at 10:26 PM, Scott Wood <scottwood@freescale.com> wrot=
e:
> On Tue, 14 Jun 2011 10:56:31 +0200
> Thomas De Schampheleire <patrickdepinguin+linuxppc@gmail.com> wrote:
>
>> * Therefore, to make sure that the mapping I intended with __ioremap()
>> is actually reflected in the TLB tables, I added dummy reads of each
>> page in the TLB, prior to jumping to the boot code, as follows:
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* make sure memory is read, once every =
4Kbyte is enough */
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 for (p =3D vaddr; p < vaddr + map_size; =
p +=3D 0x1000) {
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 unsigned long dummy =3D =
*(volatile unsigned long *)p;
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 (void)dummy;
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 }
>>
>> * After these changes (make sure all code is mapped + make sure to
>> read all pages so that the TLBs are updated), my scenario works fine.
>
> This is fragile -- you are assuming that it's possible to fit this
> set of pages in TLB0 all at once, and that none of them will be
> evicted/invalidated by the time you're done.

You're right. I think that disabling interrupts (which I can do
because I'm in a reset scenario) should fix this right? See also my
reply to Timur Tabi's post.

>
> If you really need to do this, I sugest using settlbcam() from
> arch/powerpc/mm/fsl_booke_mmu.c to create TLB1 entries with IPROT set.

Unfortunately, settlbcam is not exported to modules. Since I prefer to
be able to do all this from a kernel module, I cannot use that
function. Thanks for the suggestion though.

>
> Better still if you could live with whatever memory the kernel has alread=
y
> pinned.

In this case it is not possible. I need to jump to boot code which is
residing somewhere in physical RAM, outside the kernel memory ranges.

Best regards,
Thomas

Re: Mapping an executable page

[Tabi Timur-B04825]

Thomas De Schampheleire wrote:
> I think I can make it more reliable by dummy reading the pages*after*
> I disabled interrupts on that processor, immediately before jumping to
> the boot code. Is that correct?

That sounds logical to me.

BTW, since you're already doing something non-standard with your module, wh=
y=20
don't you just make settlbcam exported?

--=20
Timur Tabi
Linux kernel developer at Freescale=

Re: Mapping an executable page

[Thomas De Schampheleire]

On Wed, Jun 22, 2011 at 1:40 PM, Tabi Timur-B04825 <B04825@freescale.com> wrote:
> Thomas De Schampheleire wrote:
>> I think I can make it more reliable by dummy reading the pages*after*
>> I disabled interrupts on that processor, immediately before jumping to
>> the boot code. Is that correct?
>
> That sounds logical to me.
>
> BTW, since you're already doing something non-standard with your module, why
> don't you just make settlbcam exported?

It's not because I need to do something non-standard that I like to
disregard any rule, convention, or good practice :-)
I prefer to follow the kernel 'rules' and practices as closely as
possible, primarily by not messing with the kernel at all.
Unfortunately, in this case, there does not seem to be another way.

Exporting settlbcam causes me to make kernel changes (not module
changes). Then, if I want to update to a newer kernel version, I have
to re-apply the change.
By using a kernel module in a 'standard' way, I limit the number of
upgrade issues.

Best regards,
Thomas