64 bit operations w/32 bit kernel

64 bit operations w/32 bit kernel

[Finney, Steve]

What would be the downside to enabling 64 bit operations in user space on a 32 bit kernel (setting the PX bit in the status register?). The particular issue is that I want to access 64 bit-memory mapped registers, and I really need to do it as an atomic operation. I tried borrowing sibyte/64bit.h from the kernel, but I get an illegal instruction on the double ops.

Also, assuming this isn't a horrible idea, is there any obvious single place where "default" values in the CP0 status register get set?

Thanks,
sf

64 bit operations w/32 bit kernel

[Finney, Steve]

What would be the downside to enabling 64 bit operations in user space on a 32 bit kernel (setting the PX bit in the status register?). The particular issue is that I want to access 64 bit-memory mapped registers, and I really need to do it as an atomic operation. I tried borrowing sibyte/64bit.h from the kernel, but I get an illegal instruction on the double ops.

Also, assuming this isn't a horrible idea, is there any obvious single place where "default" values in the CP0 status register get set?

Thanks,
sf

Re: 64 bit operations w/32 bit kernel

[Michael Uhler]

On Mon, 2003-09-29 at 10:31, Finney, Steve wrote:
> What would be the downside to enabling 64 bit operations in user space on a 32 bit kernel (setting the PX bit in the status register?). The particular issue is that I want to access 64 bit-memory mapped registers, and I really need to do it as an atomic operation. I tried borrowing sibyte/64bit.h from the kernel, but I get an illegal instruction on the double ops.
> 
The most glaring problem is you violate the rule that the 64-bit GPRs
are sign-extended when running a 32-bit binary.  There are all kinds
of assumptions in the hardware and software that depend on the
GPRs being sign-extended, and to violate this will risk some
serious instability of the software.

> Also, assuming this isn't a horrible idea, is there any obvious single place where "default" values in the CP0 status register get set?
> 
> Thanks,
> sf
-- 

Michael Uhler, Chief Technology Officer
MIPS Technologies, Inc.  Email: uhler@mips.com  Pager:uhler_p@mips.com
1225 Charleston Road     Voice:  (650)567-5025  FAX:   (650)567-5225
Mountain View, CA 94043  Mobile: (650)868-6870  Admin: (650)567-5085

Re: 64 bit operations w/32 bit kernel

[Kip Walker]

Michael Uhler wrote:
> 
> On Mon, 2003-09-29 at 10:31, Finney, Steve wrote:
> > What would be the downside to enabling 64 bit operations in user space on a 32 bit kernel (setting the PX bit in the status register?). The particular issue is that I want to access 64 bit-memory mapped registers, and I really need to do it as an atomic operation. I tried borrowing sibyte/64bit.h from the kernel, but I get an illegal instruction on the double ops.
> >
> The most glaring problem is you violate the rule that the 64-bit GPRs
> are sign-extended when running a 32-bit binary.  There are all kinds
> of assumptions in the hardware and software that depend on the
> GPRs being sign-extended, and to violate this will risk some
> serious instability of the software.

Not to mention that the kernel won't preserve the upper 32 bits across
interrupts and system calls, if you even manage to get 64-bit values in
the registers in the first place.

Kip

Re: 64 bit operations w/32 bit kernel

[Ralf Baechle]

On Mon, Sep 29, 2003 at 07:31:57AM -1000, Finney, Steve wrote:

> What would be the downside to enabling 64 bit operations in user space
> on a 32 bit kernel (setting the PX bit in the status register?). The
> particular issue is that I want to access 64 bit-memory mapped registers,
> and I really need to do it as an atomic operation. I tried borrowing
> sibyte/64bit.h from the kernel, but I get an illegal instruction on the
> double ops.

Common design bug in hardware, imho ...

> Also, assuming this isn't a horrible idea, is there any obvious single
> place where "default" values in the CP0 status register get set?

There isn't.

What you want really is a 64-bit kernel.  On a 64-bit kernel even for
processes running in 32-bit address spaces (o32, N32) the processor
will run with the UX bit enabled.  o32 userspace still lives in the
assumption that registers are 32-bit so only those bits will be restored
in function calls etc.  N32 (where userspace isn't ready for prime time
yet) does guarantee that.  And N64 (userspace similarly not ready for
prime time) obviously is fully 64-bit everything.

  Ralf

Re: 64 bit operations w/32 bit kernel

[Maciej W. Rozycki]

On Tue, 30 Sep 2003, Ralf Baechle wrote:

> > What would be the downside to enabling 64 bit operations in user space
> > on a 32 bit kernel (setting the PX bit in the status register?). The
> > particular issue is that I want to access 64 bit-memory mapped registers,
> > and I really need to do it as an atomic operation. I tried borrowing
> > sibyte/64bit.h from the kernel, but I get an illegal instruction on the
> > double ops.
> 
> Common design bug in hardware, imho ...

 Well, ioremap() can be used to get at these areas (as well as any
others), whether we call it a bug or not.

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: 64 bit operations w/32 bit kernel

[Michael Uhler]

On Tue, 2003-09-30 at 09:00, Ralf Baechle wrote:
> On Mon, Sep 29, 2003 at 07:31:57AM -1000, Finney, Steve wrote:
> 
> > What would be the downside to enabling 64 bit operations in user space
> > on a 32 bit kernel (setting the PX bit in the status register?). The
> > particular issue is that I want to access 64 bit-memory mapped registers,
> > and I really need to do it as an atomic operation. I tried borrowing
> > sibyte/64bit.h from the kernel, but I get an illegal instruction on the
> > double ops.
> 
> Common design bug in hardware, imho ...
> 
> > Also, assuming this isn't a horrible idea, is there any obvious single
> > place where "default" values in the CP0 status register get set?
> 
> There isn't.
> 
> What you want really is a 64-bit kernel.  On a 64-bit kernel even for
> processes running in 32-bit address spaces (o32, N32) the processor
> will run with the UX bit enabled.  o32 userspace still lives in the
> assumption that registers are 32-bit so only those bits will be restored
> in function calls etc.  N32 (where userspace isn't ready for prime time
> yet) does guarantee that.  And N64 (userspace similarly not ready for
> prime time) obviously is fully 64-bit everything.

I don't think you want to run o32 processes with the UX bit set.  UX not
only enables 64-bit addressing (which you can, in software, make look
like 32-bit addressing), it also enables access to the 64-bit opcodes.
This means that you are going to get unexpected and potentially
unreproducible results.

N32 is a 64-bit data model with 32-bit addresses, so you're OK there.

/gmu

> 
>   Ralf
-- 

Michael Uhler, Chief Technology Officer
MIPS Technologies, Inc.  Email: uhler@mips.com  Pager:uhler_p@mips.com
1225 Charleston Road     Voice:  (650)567-5025  FAX:   (650)567-5225
Mountain View, CA 94043  Mobile: (650)868-6870  Admin: (650)567-5085

Re: 64 bit operations w/32 bit kernel

[Ralf Baechle]

On Tue, Sep 30, 2003 at 08:04:42PM +0200, Maciej W. Rozycki wrote:

> > > What would be the downside to enabling 64 bit operations in user space
> > > on a 32 bit kernel (setting the PX bit in the status register?). The
> > > particular issue is that I want to access 64 bit-memory mapped registers,
> > > and I really need to do it as an atomic operation. I tried borrowing
> > > sibyte/64bit.h from the kernel, but I get an illegal instruction on the
> > > double ops.
> > 
> > Common design bug in hardware, imho ...
> 
>  Well, ioremap() can be used to get at these areas (as well as any
> others), whether we call it a bug or not.

What I called a bug is the necessity to access hardware registers with
64-bit loads and stores in some systems as opposed to of 32-bit
instructions - that simply doesn't work from 32-bit universes.

To clarify, it was my understanding of Steve's problem he needs 64-bit
loads and stores, not something in the 64-bit physical address space.
The later problem obviously would get a different answer.

  Ralf

Re: 64 bit operations w/32 bit kernel

[Maciej W. Rozycki]

On 30 Sep 2003, Michael Uhler wrote:

> > What you want really is a 64-bit kernel.  On a 64-bit kernel even for
> > processes running in 32-bit address spaces (o32, N32) the processor
> > will run with the UX bit enabled.  o32 userspace still lives in the
> > assumption that registers are 32-bit so only those bits will be restored
> > in function calls etc.  N32 (where userspace isn't ready for prime time
> > yet) does guarantee that.  And N64 (userspace similarly not ready for
> > prime time) obviously is fully 64-bit everything.
> 
> I don't think you want to run o32 processes with the UX bit set.  UX not
> only enables 64-bit addressing (which you can, in software, make look
> like 32-bit addressing), it also enables access to the 64-bit opcodes.
> This means that you are going to get unexpected and potentially
> unreproducible results.

 Well, I think this is OK -- 64-bit opcodes are generally useless for
software built for the o32 ABI, so they should not normally happen in
regular code.  Perhaps some fancy hand-coded assembly might try to use
them to get unusual results, including an invalid opcode trap handler for
the processors that do not support them at all.  But I don't think we
should try to work hard to prevent broken software from shooting into its
foot.

 And the advantage is we have a single TLB refill handler.

  Maciej

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: 64 bit operations w/32 bit kernel

[Michael Uhler]

On Tue, 2003-09-30 at 12:10, Maciej W. Rozycki wrote:
> On 30 Sep 2003, Michael Uhler wrote:
> 
> > > What you want really is a 64-bit kernel.  On a 64-bit kernel even for
> > > processes running in 32-bit address spaces (o32, N32) the processor
> > > will run with the UX bit enabled.  o32 userspace still lives in the
> > > assumption that registers are 32-bit so only those bits will be restored
> > > in function calls etc.  N32 (where userspace isn't ready for prime time
> > > yet) does guarantee that.  And N64 (userspace similarly not ready for
> > > prime time) obviously is fully 64-bit everything.
> > 
> > I don't think you want to run o32 processes with the UX bit set.  UX not
> > only enables 64-bit addressing (which you can, in software, make look
> > like 32-bit addressing), it also enables access to the 64-bit opcodes.
> > This means that you are going to get unexpected and potentially
> > unreproducible results.
> 
>  Well, I think this is OK -- 64-bit opcodes are generally useless for
> software built for the o32 ABI, so they should not normally happen in
> regular code.  Perhaps some fancy hand-coded assembly might try to use
> them to get unusual results, including an invalid opcode trap handler for
> the processors that do not support them at all.  But I don't think we
> should try to work hard to prevent broken software from shooting into its
> foot.

I'm not a real fan of architecting software to dismiss broken (or rogue)
programs since it tends to open up a whole lot of holes that cause
O/S crashes.  For instance, an o32 program should never be able to pass
a non-sign-extended value thru a GPR to the O/S in a system call.  How
many places in the O/S implicitly assume this is true?  The architecture
was never intended to run real 32-bit programs with 64-bit ops enabled,
and I would strongly urge you not to do this now.

> 
>  And the advantage is we have a single TLB refill handler.

This hardly seems compelling given how short the handlers are.  Further,
since N64 needs the 64-bit address support anyway, setting UX and PX
correctly will allow you to run the N32 code with the TLB handler
instead of the XTLB handler.  That way the XTLB handler is only
invoked when a real extended address reference happens.  That seems
cleaner to me, but I admit I'm not the one that has to write the
code.

/gmu

> 
>   Maciej
-- 

Michael Uhler, Chief Technology Officer
MIPS Technologies, Inc.  Email: uhler@mips.com  Pager:uhler_p@mips.com
1225 Charleston Road     Voice:  (650)567-5025  FAX:   (650)567-5225
Mountain View, CA 94043  Mobile: (650)868-6870  Admin: (650)567-5085

Re: 64 bit operations w/32 bit kernel

[Ralf Baechle]

On Tue, Sep 30, 2003 at 11:29:28AM -0700, Michael Uhler wrote:

> > There isn't.
> > 
> > What you want really is a 64-bit kernel.  On a 64-bit kernel even for
> > processes running in 32-bit address spaces (o32, N32) the processor
> > will run with the UX bit enabled.  o32 userspace still lives in the
> > assumption that registers are 32-bit so only those bits will be restored
> > in function calls etc.  N32 (where userspace isn't ready for prime time
> > yet) does guarantee that.  And N64 (userspace similarly not ready for
> > prime time) obviously is fully 64-bit everything.
> 
> I don't think you want to run o32 processes with the UX bit set.  UX not
> only enables 64-bit addressing (which you can, in software, make look
> like 32-bit addressing), it also enables access to the 64-bit opcodes.
> This means that you are going to get unexpected and potentially
> unreproducible results.

Currently the 64-bit kernel is running everything in userspace with UX
enabled, in part that's lazyness, in part it was also made in the hope
optimized libraries for o32 would take advantage of 64-bit computing.

Linux btw. doesn't use the MIPS64 PX flag.  It's only supported on MIPS64
and it we simply didn't consider it's use much of a benefit so far -
something that probably is arguable ...

  Ralf

Re: 64 bit operations w/32 bit kernel

[Maciej W. Rozycki]

On Tue, 30 Sep 2003, Ralf Baechle wrote:

> What I called a bug is the necessity to access hardware registers with
> 64-bit loads and stores in some systems as opposed to of 32-bit
> instructions - that simply doesn't work from 32-bit universes.
> 
> To clarify, it was my understanding of Steve's problem he needs 64-bit
> loads and stores, not something in the 64-bit physical address space.
> The later problem obviously would get a different answer.

 I must have missed the detail.  Well, if 64-bit transfers are needed,
then going for the 64-bit kernel is about the only way.  Or, as a wild
hack, perhaps "ldc1" and "sdc1" can be used, if it's known the FP is
present.

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: 64 bit operations w/32 bit kernel

[Maciej W. Rozycki]

On 30 Sep 2003, Michael Uhler wrote:

> >  Well, I think this is OK -- 64-bit opcodes are generally useless for
> > software built for the o32 ABI, so they should not normally happen in
> > regular code.  Perhaps some fancy hand-coded assembly might try to use
> > them to get unusual results, including an invalid opcode trap handler for
> > the processors that do not support them at all.  But I don't think we
> > should try to work hard to prevent broken software from shooting into its
> > foot.
> 
> I'm not a real fan of architecting software to dismiss broken (or rogue)
> programs since it tends to open up a whole lot of holes that cause
> O/S crashes.  For instance, an o32 program should never be able to pass
> a non-sign-extended value thru a GPR to the O/S in a system call.  How
> many places in the O/S implicitly assume this is true?  The architecture

 That actually we get right -- the arguments at the stack are obviously
32-bit and the ones in argument registers are explicitly sign-extended
from their 32 LSBs in our syscall prologue for o32 syscalls.

> was never intended to run real 32-bit programs with 64-bit ops enabled,
> and I would strongly urge you not to do this now.

 After a bit of thinking, I consider this not to be a real problem.  Apart
from the kernel interface, which sanitizes values passed, the rest is pure
userland, where allowing undefined operation with 64-bit opcodes cannot
really hurt.  Of course running a buggy or malicious program may lead to
bad results or loss of data, but it'll be limited to the user responsible
for running such software and the root user by definition has to know what
he is doing and specifically he is responsible for not running untrusted
software on critical systems.

 That said, I don't really have a strong preference either way -- it just
doesn't seem to be worth the hassle for me to explicitly defend against
such a marginal case.  Although it may be good to try validating this
assumption with `crashme'. 

  Maciej

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: 64 bit operations w/32 bit kernel

[Ralf Baechle]

On Wed, Oct 01, 2003 at 06:26:02AM +0200, Maciej W. Rozycki wrote:

> > was never intended to run real 32-bit programs with 64-bit ops enabled,
> > and I would strongly urge you not to do this now.
> 
>  After a bit of thinking, I consider this not to be a real problem.  Apart
> from the kernel interface, which sanitizes values passed, the rest is pure
> userland, where allowing undefined operation with 64-bit opcodes cannot
> really hurt.  Of course running a buggy or malicious program may lead to
> bad results or loss of data, but it'll be limited to the user responsible
> for running such software and the root user by definition has to know what
> he is doing and specifically he is responsible for not running untrusted
> software on critical systems.
> 
>  That said, I don't really have a strong preference either way -- it just
> doesn't seem to be worth the hassle for me to explicitly defend against
> such a marginal case.  Although it may be good to try validating this
> assumption with `crashme'. 

It's a while since this last has been done and all bugs showing up were
fixed ...

  Ralf

RE: 64 bit operations w/32 bit kernel

[Finney, Steve]

> 
> What you want really is a 64-bit kernel.  On a 64-bit kernel even for
> processes running in 32-bit address spaces (o32, N32) the processor
> will run with the UX bit enabled.  o32 userspace still lives in the
> assumption that registers are 32-bit so only those bits will 
> be restored
> in function calls etc.  N32 (where userspace isn't ready for 
> prime time
> yet) does guarantee that.  And N64 (userspace similarly not ready for
> prime time) obviously is fully 64-bit everything.

What is the page table space impact of a 64 bit kernel on an architecture like the Sibyte with discontiguous physical address spaces?  Do you still waste 36 MB of page table space on the "hole", or does it even double because the page table entries are bigger for the wider addresses? Or do the issues become irrelevant with the larger address space?

Thanks,
sf

RE: 64 bit operations w/32 bit kernel

[Finney, Steve]

> 
> What you want really is a 64-bit kernel.  On a 64-bit kernel even for
> processes running in 32-bit address spaces (o32, N32) the processor
> will run with the UX bit enabled.  o32 userspace still lives in the
> assumption that registers are 32-bit so only those bits will 
> be restored
> in function calls etc.  N32 (where userspace isn't ready for 
> prime time
> yet) does guarantee that.  And N64 (userspace similarly not ready for
> prime time) obviously is fully 64-bit everything.

What is the page table space impact of a 64 bit kernel on an architecture like the Sibyte with discontiguous physical address spaces?  Do you still waste 36 MB of page table space on the "hole", or does it even double because the page table entries are bigger for the wider addresses? Or do the issues become irrelevant with the larger address space?

Thanks,
sf