use of BAT before taking over the MMU

use of BAT before taking over the MMU

[Albert Cahalan]

On the prom boot path, with the firmware supposed to
be managing the MMU, there is a case where:

1. Linux changes some BAT registers.
2. Bits 0x00000070 are/become set in the MSR.
3. Linux takes an MMU fault.
4. The firmware handles it.

AFAIK, you can't expect the firmware to leave the BAT alone.
If the firmware provides mapping services by using the BAT
as a software-filled TLB, Linux's BAT changes may be lost.

You also can't expect that your BAT changes will not conflict
with mappings that the firmware uses for itself. The firmware
might write to your new BAT mapping, relying on those virtual
addresses to be something else entirely.

Re: use of BAT before taking over the MMU

[Segher Boessenkool]

> On the prom boot path, with the firmware supposed to
> be managing the MMU, there is a case where:
>
> 1. Linux changes some BAT registers.
> 2. Bits 0x00000070 are/become set in the MSR.
> 3. Linux takes an MMU fault.
> 4. The firmware handles it.
>
> AFAIK, you can't expect the firmware to leave the BAT alone.
> If the firmware provides mapping services by using the BAT
> as a software-filled TLB, Linux's BAT changes may be lost.
>
> You also can't expect that your BAT changes will not conflict
> with mappings that the firmware uses for itself. The firmware
> might write to your new BAT mapping, relying on those virtual
> addresses to be something else entirely.

The PowerPC OF binding requires the firmware to save and restore
the BATs on entry to / exit from the firmware.


Segher

Re: use of BAT before taking over the MMU

[Albert Cahalan]

On Mon, Oct 4, 2010 at 12:25 AM, Segher Boessenkool
<segher@kernel.crashing.org> wrote:

> The PowerPC OF binding requires the firmware to save and restore
> the BATs on entry to / exit from the firmware.

This sucks, because using the BAT is **much** easier for
the firmware. In my case, it also means I don't need to worry
about Linux stomping on anything -- I have nothing in RAM.

(this is an emulator, but I don't want to cheat too much
because adding special cases affects performance)

Page tables would need to go in RAM. If Linux wants to
use that memory...? It seems that Linux does tend to ask;
will it panic if I refuse? Are there addresses Linux won't
ever ask for?

It also looks like I could just start up Linux w/o the MMU on.
I'm just making 1:1 mappings anyway.

Re: use of BAT before taking over the MMU

[Segher Boessenkool]

>> The PowerPC OF binding requires the firmware to save and restore
>> the BATs on entry to / exit from the firmware.
>
> This sucks, because using the BAT is **much** easier for
> the firmware. In my case, it also means I don't need to worry
> about Linux stomping on anything -- I have nothing in RAM.

It's easy to save and restore the BATs on firmware entry/exit,
where do you see a problem?

> Page tables would need to go in RAM. If Linux wants to
> use that memory...?

Same way as when the client uses *any* memory OF has claimed:
it has to stop calling OF from then on.

> It seems that Linux does tend to ask;
> will it panic if I refuse? Are there addresses Linux won't
> ever ask for?

Linux always "kills" OF pretty early on.

> It also looks like I could just start up Linux w/o the MMU on.

Yes, that's common and works.  You will still need BATs (or
something else) to do I/O from the firmware though.


Segher

Re: use of BAT before taking over the MMU

[Benjamin Herrenschmidt]

On Sat, 2010-10-02 at 14:32 -0400, Albert Cahalan wrote:
> On the prom boot path, with the firmware supposed to
> be managing the MMU, there is a case where:
> 
> 1. Linux changes some BAT registers.
> 2. Bits 0x00000070 are/become set in the MSR.
> 3. Linux takes an MMU fault.

Meeep ! Linux should never take an MMU fault at that point :-) If it
does, then there's a bug somewhere that needs squashing.

> 4. The firmware handles it.
> 
> AFAIK, you can't expect the firmware to leave the BAT alone.
> If the firmware provides mapping services by using the BAT
> as a software-filled TLB, Linux's BAT changes may be lost.
> 
> You also can't expect that your BAT changes will not conflict
> with mappings that the firmware uses for itself. The firmware
> might write to your new BAT mapping, relying on those virtual
> addresses to be something else entirely.

Right, which is why the moment Linux takes over the BATs, it shouldn't
call into FW anymore nor take faults.

Cheers,
Ben.

Re: use of BAT before taking over the MMU

[Benjamin Herrenschmidt]

On Mon, 2010-10-04 at 06:25 +0200, Segher Boessenkool wrote:
> > On the prom boot path, with the firmware supposed to
> > be managing the MMU, there is a case where:
> >
> > 1. Linux changes some BAT registers.
> > 2. Bits 0x00000070 are/become set in the MSR.
> > 3. Linux takes an MMU fault.
> > 4. The firmware handles it.
> >
> > AFAIK, you can't expect the firmware to leave the BAT alone.
> > If the firmware provides mapping services by using the BAT
> > as a software-filled TLB, Linux's BAT changes may be lost.
> >
> > You also can't expect that your BAT changes will not conflict
> > with mappings that the firmware uses for itself. The firmware
> > might write to your new BAT mapping, relying on those virtual
> > addresses to be something else entirely.
> 
> The PowerPC OF binding requires the firmware to save and restore
> the BATs on entry to / exit from the firmware.

I'm not sure he was talking about OF here... In any case, we don't muck
around with BATs until after we're done with OF anyways.

Cheers,
Ben.

Re: use of BAT before taking over the MMU

[Benjamin Herrenschmidt]

On Mon, 2010-10-04 at 12:06 -0400, Albert Cahalan wrote:
> On Mon, Oct 4, 2010 at 12:25 AM, Segher Boessenkool
> <segher@kernel.crashing.org> wrote:
> 
> > The PowerPC OF binding requires the firmware to save and restore
> > the BATs on entry to / exit from the firmware.
> 
> This sucks, because using the BAT is **much** easier for
> the firmware. In my case, it also means I don't need to worry
> about Linux stomping on anything -- I have nothing in RAM.

What FW are we talking about here anyways ? As I said, Linux isn't
supposed to take any fault from the moment it starts setting up BATs to
the moment it takes over exceptions and can handle faults all by itself.

> (this is an emulator, but I don't want to cheat too much
> because adding special cases affects performance)
> 
> Page tables would need to go in RAM. If Linux wants to
> use that memory...? It seems that Linux does tend to ask;
> will it panic if I refuse? Are there addresses Linux won't
> ever ask for?
> 
> It also looks like I could just start up Linux w/o the MMU on.
> I'm just making 1:1 mappings anyway.

Cheers,
Ben.

Re: use of BAT before taking over the MMU

[Albert Cahalan]

On Tue, Oct 5, 2010 at 4:10 AM, Benjamin Herrenschmidt
<benh@kernel.crashing.org> wrote:
> On Mon, 2010-10-04 at 06:25 +0200, Segher Boessenkool wrote:
>> > On the prom boot path, with the firmware supposed to
>> > be managing the MMU, there is a case where:
>> >
>> > 1. Linux changes some BAT registers.
>> > 2. Bits 0x00000070 are/become set in the MSR.
>> > 3. Linux takes an MMU fault.
>> > 4. The firmware handles it.
>> >
>> > AFAIK, you can't expect the firmware to leave the BAT alone.
>> > If the firmware provides mapping services by using the BAT
>> > as a software-filled TLB, Linux's BAT changes may be lost.
>> >
>> > You also can't expect that your BAT changes will not conflict
>> > with mappings that the firmware uses for itself. The firmware
>> > might write to your new BAT mapping, relying on those virtual
>> > addresses to be something else entirely.
>>
>> The PowerPC OF binding requires the firmware to save and restore
>> the BATs on entry to / exit from the firmware.

That would defeat the purpose of setting them.
They are used to provide Linux with mappings.

> I'm not sure he was talking about OF here... In any case, we don't muck
> around with BATs until after we're done with OF anyways.

It's the Open Firmware client interface, not Firmworks code.
I wrote it. It just does hypervisor-like calls to an emulator which
does the real work. Essentially I have a prom_call opcode
and a handle_exception opcode. I claim to be an MPC7400.

The kernel is sitting at address 0, MSR becomes 0x70 when
an rfi goes to address 0. This will be head_32.S with r5 set,
so Linux will call prom_init to fetch the device tree.

Of course that faults immediately, so I have a handler that
loads IBAT0 with a 128 KiB mapping. I treat the BAT like a
direct-mapped software-loaded TLB. (like MIPS arch MMU)

I threw in a hack to skip to the next BAT when the chosen
one is not a 128 KiB 1:1 mapping. That seems to work, but
sure doesn't feel right.

Note that Linux can fail even with a firmware that doesn't touch
the BAT registers. The MMU is on, and 0xc0000000 may be
where the firmware expects to have... MMIO for the console,
the client interface entry point, a forth stack, whatever.
The BAT takes priority, and thus the firmware splatters stuff
right onto the kernel or dies trying to read something it left there.

Re: use of BAT before taking over the MMU

[Benjamin Herrenschmidt]

> >> The PowerPC OF binding requires the firmware to save and restore
> >> the BATs on entry to / exit from the firmware.
> 
> That would defeat the purpose of setting them.
> They are used to provide Linux with mappings.

How so ? As long as they are present when executing Linux code,
I don't see the problem if they contain different mappings while
inside FW...

> > I'm not sure he was talking about OF here... In any case, we don't muck
> > around with BATs until after we're done with OF anyways.
> 
> It's the Open Firmware client interface, not Firmworks code.
> I wrote it. It just does hypervisor-like calls to an emulator which
> does the real work. Essentially I have a prom_call opcode
> and a handle_exception opcode. I claim to be an MPC7400.
> 
> The kernel is sitting at address 0, MSR becomes 0x70 when
> an rfi goes to address 0. This will be head_32.S with r5 set,
> so Linux will call prom_init to fetch the device tree.

Right. Within prom_init we don't touch the BATs, we are still under FW
control at this stage.

> Of course that faults immediately,

It shouldn't but it can. When you enter Linux with an OF "client
interface" entry, you are responsible for having reasonable mappings for
executing the client code. You are allowed to fault these if you want
to, and do whatever you want, as long as the prom_init code can execute.
IE. Linux hasn't set any mappings nor BATs at this stage, it's operating
as a client program of the FW and entirely relies on the FW to provide
the necessary MMU setup until the end of prom_init.

>  so I have a handler that
> loads IBAT0 with a 128 KiB mapping. I treat the BAT like a
> direct-mapped software-loaded TLB. (like MIPS arch MMU)
> 
> I threw in a hack to skip to the next BAT when the chosen
> one is not a 128 KiB 1:1 mapping. That seems to work, but
> sure doesn't feel right.

Why would you need to do that ? Linux hasn't done anything to mappings
yet and only cares about what you have setup...

Linux will throw away your BATs etc... only after it returns from
prom_init at which point it won't call your FW anymore anyways.

> Note that Linux can fail even with a firmware that doesn't touch
> the BAT registers. The MMU is on, and 0xc0000000 may be
> where the firmware expects to have... MMIO for the console,
> the client interface entry point, a forth stack, whatever.

Right, but Linux doesn't establish nor relies on mappings at c0000000 at
this point. prom_init is relocatable code that can execute from any
address.

> The BAT takes priority, and thus the firmware splatters stuff
> right onto the kernel or dies trying to read something it left there.

I'm not entirely sure what you are doing but it certainly sounds
wrong :-)

As I said, at this stage, Linux is just a "normal" client interface
program. It's happy to run from any address you put it with any mapping
you provide until it returns from prom_init at which point it takes over
the MMU.

Ben.

Re: use of BAT before taking over the MMU

[Segher Boessenkool]

>> The PowerPC OF binding requires the firmware to save and restore
>> the BATs on entry to / exit from the firmware.
>
> I'm not sure he was talking about OF here...

Yeah, I thought I was on a different mailing list.  It's still
sort-of relevant though.

> In any case, we don't muck
> around with BATs until after we're done with OF anyways.

"Lower than the lowest common divider", yeah ;-)


Segher

Re: use of BAT before taking over the MMU

[Segher Boessenkool]

>>> The PowerPC OF binding requires the firmware to save and restore
>>> the BATs on entry to / exit from the firmware.
>
> That would defeat the purpose of setting them.
> They are used to provide Linux with mappings.

The initial state the OS has for the BATs is what the firmware
provides it with, sure.

An OS shouldn't expect to have more than its own program image
RAM mapped, in general.

> Of course that faults immediately, so I have a handler that
> loads IBAT0 with a 128 KiB mapping. I treat the BAT like a
> direct-mapped software-loaded TLB. (like MIPS arch MMU)

Just map the first 256MB and don't worry about anything else?
Seems a lot simpler to me ;-)

> Note that Linux can fail even with a firmware that doesn't touch
> the BAT registers. The MMU is on,

You can boot Linux with the MMU off as well.

> and 0xc0000000 may be
> where the firmware expects to have... MMIO for the console,
> the client interface entry point, a forth stack, whatever.
> The BAT takes priority, and thus the firmware splatters stuff
> right onto the kernel or dies trying to read something it left there.

Like I said, you're supposed to swap OS BATs with firmware BATs
in your client interface entry and exit.  You have to switch
a lot of other registers there as well already, so that's no
big deal.


Segher

Re: use of BAT before taking over the MMU

[Albert Cahalan]

On Tue, Oct 5, 2010 at 11:31 AM, Segher Boessenkool
<segher@kernel.crashing.org> wrote:

> An OS shouldn't expect to have more than its own program image
> RAM mapped, in general.

Linux actually makes calls to allocate more. I'm thankful
that Linux always specifies an address, so I was able to
get away with simply returning success. I wonder how this
works for a firmware implementation that resides in RAM,
using the memory that Linux demands. Must the firmware
move itself out of the way?

>> Of course that faults immediately, so I have a handler that
>> loads IBAT0 with a 128 KiB mapping. I treat the BAT like a
>> direct-mapped software-loaded TLB. (like MIPS arch MMU)
>
> Just map the first 256MB and don't worry about anything else?
> Seems a lot simpler to me ;-)

I was expecting that Linux would demand plenty of mappings,
including small ones and ones for IO. I was preparing myself
for that.

>> Note that Linux can fail even with a firmware that doesn't touch
>> the BAT registers. The MMU is on,
>
> You can boot Linux with the MMU off as well.

That wasn't obvious for the prom_init path. IEEE docs seemed
to suggest that the firmware must provide MMU handling.

>> and 0xc0000000 may be
>> where the firmware expects to have... MMIO for the console,
>> the client interface entry point, a forth stack, whatever.
>> The BAT takes priority, and thus the firmware splatters stuff
>> right onto the kernel or dies trying to read something it left there.
>
> Like I said, you're supposed to swap OS BATs with firmware BATs
> in your client interface entry and exit. =A0You have to switch
> a lot of other registers there as well already, so that's no
> big deal.

Well no. This isn't real hardware. My prom entry point looks
something like this:

eciwx r0,r0,r0
blr

My ISI and DSI handlers look something like this:

ecowx r0,r0,r0
rfi

The firmware doesn't need **any** registers. It's magic. I was
just using the BAT registers to map what Linux wanted mapped.

Anyway, I'm no longer able to reproduce the problem. I think
something unrelated was causing strange behavior. This is a
bit of a surprise since I would've expected a crash. Oh well.

Re: use of BAT before taking over the MMU

[Benjamin Herrenschmidt]

On Wed, 2010-10-06 at 22:05 -0400, Albert Cahalan wrote:
> On Tue, Oct 5, 2010 at 11:31 AM, Segher Boessenkool
> <segher@kernel.crashing.org> wrote:
> 
> > An OS shouldn't expect to have more than its own program image
> > RAM mapped, in general.
> 
> Linux actually makes calls to allocate more. I'm thankful
> that Linux always specifies an address, so I was able to
> get away with simply returning success. I wonder how this
> works for a firmware implementation that resides in RAM,
> using the memory that Linux demands. Must the firmware
> move itself out of the way?

No, Linux will retry somewhere else :-)

> >> Of course that faults immediately, so I have a handler that
> >> loads IBAT0 with a 128 KiB mapping. I treat the BAT like a
> >> direct-mapped software-loaded TLB. (like MIPS arch MMU)
> >
> > Just map the first 256MB and don't worry about anything else?
> > Seems a lot simpler to me ;-)
> 
> I was expecting that Linux would demand plenty of mappings,
> including small ones and ones for IO. I was preparing myself
> for that.

No, not during prom_init. It's really just a trampoline code that sucks
out the device-tree and does a few other things. Once that's complete,
Linux takes over the MMU and from that point on, your FW is dead.

> >> Note that Linux can fail even with a firmware that doesn't touch
> >> the BAT registers. The MMU is on,
> >
> > You can boot Linux with the MMU off as well.
> 
> That wasn't obvious for the prom_init path. IEEE docs seemed
> to suggest that the firmware must provide MMU handling.

1275 powerpc binding specifies both mode of operations. Linux doesn't
care which one is active

> >> and 0xc0000000 may be
> >> where the firmware expects to have... MMIO for the console,
> >> the client interface entry point, a forth stack, whatever.
> >> The BAT takes priority, and thus the firmware splatters stuff
> >> right onto the kernel or dies trying to read something it left there.
> >
> > Like I said, you're supposed to swap OS BATs with firmware BATs
> > in your client interface entry and exit.  You have to switch
> > a lot of other registers there as well already, so that's no
> > big deal.
> 
> Well no. This isn't real hardware. My prom entry point looks
> something like this:
> 
> eciwx r0,r0,r0
> blr
> 
> My ISI and DSI handlers look something like this:
> 
> ecowx r0,r0,r0
> rfi
> 
> The firmware doesn't need **any** registers. It's magic. I was
> just using the BAT registers to map what Linux wanted mapped.

Which is really just what any client program wants: You start with the
program ELF sections, and whatever it allocates with claim() calls.

> Anyway, I'm no longer able to reproduce the problem. I think
> something unrelated was causing strange behavior. This is a
> bit of a surprise since I would've expected a crash. Oh well.

Hard to tell from my side :-) But as I said, Linux doesn't rely on any
mapping at c0000000 or any BAT at this point. By the time it sets up
BATs the firmware is long gone and Linux is fully in control of the MMU.

Cheers,
Ben.