Does the R10K family support the "wait" instruction?

Does the R10K family support the "wait" instruction?

[Joshua Kinard]

Does anyone know if the R1x000 family of CPUs support the "wait" instruction?
The 'check_wait' function in arch/mips/kernel/idle.c doesn't have a case for
any of the R10K CPUs, and I can't find any specific guidance in the final R10K
manual produced by Renesas, nor in the MIPS-IV instruction set.  It appears
this was added in MIPS-II, and the R4K CPUs use it, with one version for when
interrupts are enabled, and one where they're disabled.  Since a lot of CPUs
tend to reuse R4K-compatible code, I wasn't sure.

Kinda-assuming it doesn't, since the R10K lacks any notion of reduced power
operation.

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
6144R/F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Re: Does the R10K family support the "wait" instruction?

[Ralf Baechle]

On Sun, Mar 26, 2017 at 09:50:08PM -0400, Joshua Kinard wrote:

> Does anyone know if the R1x000 family of CPUs support the "wait" instruction?
> The 'check_wait' function in arch/mips/kernel/idle.c doesn't have a case for
> any of the R10K CPUs, and I can't find any specific guidance in the final R10K
> manual produced by Renesas, nor in the MIPS-IV instruction set.  It appears
> this was added in MIPS-II, and the R4K CPUs use it, with one version for when
> interrupts are enabled, and one where they're disabled.  Since a lot of CPUs
> tend to reuse R4K-compatible code, I wasn't sure.

Interesting, didn't know Renesas did another R10000 manual.  Presumably
they only rebranded NEC's manual?

If you have any documentation to indicate a MIPS II CPU to support WAIT,
I'm interested.  From all that I know the feature was introduced by the
R4600.

> Kinda-assuming it doesn't, since the R10K lacks any notion of reduced power
> operation.

The R10000 like many of the older MIPS CPU took a hardware-only approach
to low-power operation, that is there are no knobs, no instructions for
software to optimize the power consumption.  That also means, no WAIT
instruction.  Other bits of low-power support in R4x00-processors are
fake also, for example the c0_status.rp "reduced power" bit is documented
but fairly hidden erratas say it's not implemneted.  So just throw in another
plutonium slab and all will be good ;-)

But back to the WAIT instruction.  WAIT uses major opcode COP0 which has
bits 31..27 = 010000.  Traditionally MIPS doesn't fully decode the minor
opcodes, so an unsupported WAIT instruction will not cause an Reserved
Instruction exception anyway because the COP0 opcode does exist.

So one could basically pretend WAIT did exist on all MIPS CPUs, even
R2000 even though it doesn't - but check_wait is paranoid and really
avoids WAIT unless it's officially supported and useful.

Another complexity is that WAIT instruction was defined for many years
to leave it to an implementation if the pipeline would ever restart
after a WAIT instruction was executed with interrupts disabled.  This
is fairly braindead because in a typical idle loop of an OS such as Linux

        for (;;) {
                if (wait_available && !need_resched())
                        asm("wait");
                schedule();
        }

there is the race condition between checking need_resched() and
actually executing WAIT that need_resched() might change, so WAIT-mode
might be entered falsely causing a scheduling delay of up to one tick.
The solution is to disable interrupts like:

        for (;;) {
		disable_irq();
                if (wait_available && !need_resched())
                        asm("wait");
		enable_irq();
                schedule();
        }

But some MIPS CPUs might lockup for good if this is attempted ...

This turned out to be a longer writeup than I meant to because I
recognized much of this is not documented anywhere so I did elaborate
a bit and add it as a new wiki page:

https://www.linux-mips.org/wiki/WAIT_instruction

  Ralf

Re: Does the R10K family support the "wait" instruction?

[Joshua Kinard]

[-- Attachment #1: Type: text/plain, Size: 6036 bytes --]

On 03/27/2017 09:05, Ralf Baechle wrote:
> On Sun, Mar 26, 2017 at 09:50:08PM -0400, Joshua Kinard wrote:
> 
>> Does anyone know if the R1x000 family of CPUs support the "wait" instruction?
>> The 'check_wait' function in arch/mips/kernel/idle.c doesn't have a case for
>> any of the R10K CPUs, and I can't find any specific guidance in the final R10K
>> manual produced by Renesas, nor in the MIPS-IV instruction set.  It appears
>> this was added in MIPS-II, and the R4K CPUs use it, with one version for when
>> interrupts are enabled, and one where they're disabled.  Since a lot of CPUs
>> tend to reuse R4K-compatible code, I wasn't sure.
> 
> Interesting, didn't know Renesas did another R10000 manual.  Presumably
> they only rebranded NEC's manual?

Yup, mostly.  It's document U10278EJ4V0UM00, 4th edition, though I can't get
the search engine on their site to cough a link up, so I've attached the copy I
found.  The manual looks like it was dated from March 2001, which covers at
least up to R12000, but has a "phased-out/discontinued" stamp and a notice to
customers dated April 2010.

I still, to this date, have never come across any documentation on the R14000
or R16000.  That must be internal to SGI and HP (used in their NonStop servers)
and never saw the light of day.  Technically HP now, since they recently
scooped up what was left of SGI.


> If you have any documentation to indicate a MIPS II CPU to support WAIT,
> I'm interested.  From all that I know the feature was introduced by the
> R4600.

One of my Google searches, using the keywords "mips wait instruction", has been
returning search results from a Harvard computer sciences course detailing the
"instructional operating system OS/161", which uses an instructional CPU that
borrows heavily from the R3000 (which they dub MIPS-161):

http://os161.eecs.harvard.edu/documentation/sys161-1.99.07/mips.html

In there, they state:
"The WAIT instruction has been borrowed from MIPS-II. This operation puts the
processor into a low-power state and suspends execution until some external
event occurs, such as an interrupt. Since the exact behavior of WAIT is not
clearly specified anywhere I could find, the MIPS-161 behavior is as follows"

So it could be that the instructor for that course simply got some wrong
information, but good luck teaching Google that.  I figure once its spider
crawls this e-mail in the archives, it'll further strengthen hits like the above.


>> Kinda-assuming it doesn't, since the R10K lacks any notion of reduced power
>> operation.
> 
> The R10000 like many of the older MIPS CPU took a hardware-only approach
> to low-power operation, that is there are no knobs, no instructions for
> software to optimize the power consumption.  That also means, no WAIT
> instruction.  Other bits of low-power support in R4x00-processors are
> fake also, for example the c0_status.rp "reduced power" bit is documented
> but fairly hidden erratas say it's not implemneted.  So just throw in another
> plutonium slab and all will be good ;-)

It might make sense to add the R10K cases to the switch in cpu_wait, right
above the "default" block, with a comment stating that R10K doesn't support
wait.  That way it's documented in the right spot in case this question ever
comes up in the future.  I can send a patch for this later on if interested.


> But back to the WAIT instruction.  WAIT uses major opcode COP0 which has
> bits 31..27 = 010000.  Traditionally MIPS doesn't fully decode the minor
> opcodes, so an unsupported WAIT instruction will not cause an Reserved
> Instruction exception anyway because the COP0 opcode does exist.

That would explain why when I tried it briefly on an Octane kernel, nothing
happened.  So I wasn't sure if it was supported or not.


> So one could basically pretend WAIT did exist on all MIPS CPUs, even
> R2000 even though it doesn't - but check_wait is paranoid and really
> avoids WAIT unless it's officially supported and useful.
> 
> Another complexity is that WAIT instruction was defined for many years
> to leave it to an implementation if the pipeline would ever restart
> after a WAIT instruction was executed with interrupts disabled.  This
> is fairly braindead because in a typical idle loop of an OS such as Linux
> 
>         for (;;) {
>                 if (wait_available && !need_resched())
>                         asm("wait");
>                 schedule();
>         }
> 
> there is the race condition between checking need_resched() and
> actually executing WAIT that need_resched() might change, so WAIT-mode
> might be entered falsely causing a scheduling delay of up to one tick.
> The solution is to disable interrupts like:
> 
>         for (;;) {
> 		disable_irq();
>                 if (wait_available && !need_resched())
>                         asm("wait");
> 		enable_irq();
>                 schedule();
>         }
> 
> But some MIPS CPUs might lockup for good if this is attempted ...
> 
> This turned out to be a longer writeup than I meant to because I
> recognized much of this is not documented anywhere so I did elaborate
> a bit and add it as a new wiki page:
> 
> https://www.linux-mips.org/wiki/WAIT_instruction

And this is where I think I am seeing the IP27's main issue, a lock up having
to do with a race condition in enabling/disabling IRQs, and possibly the
registers getting clobbered in the process, leading to the resumed process
getting garbage data.

Several of my google searches on a race in enabling/disabling IRQs kept coming
back with hits to the wait instruction, and since I couldn't find any
information on if the R10K supported it, decided to write my e-mail.  So I can
scratch that angle off of my list.

Thanks!

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
6144R/F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

[-- Attachment #2: r10000-um-200103-4th-renesas.pdf --]
[-- Type: application/pdf, Size: 1571138 bytes --]

Re: Does the R10K family support the "wait" instruction?

[Maciej W. Rozycki]

On Mon, 27 Mar 2017, Joshua Kinard wrote:

> > If you have any documentation to indicate a MIPS II CPU to support WAIT,
> > I'm interested.  From all that I know the feature was introduced by the
> > R4600.
> 
> One of my Google searches, using the keywords "mips wait instruction", has been
> returning search results from a Harvard computer sciences course detailing the
> "instructional operating system OS/161", which uses an instructional CPU that
> borrows heavily from the R3000 (which they dub MIPS-161):
> 
> http://os161.eecs.harvard.edu/documentation/sys161-1.99.07/mips.html
> 
> In there, they state:
> "The WAIT instruction has been borrowed from MIPS-II. This operation puts the
> processor into a low-power state and suspends execution until some external
> event occurs, such as an interrupt. Since the exact behavior of WAIT is not
> clearly specified anywhere I could find, the MIPS-161 behavior is as follows"
> 
> So it could be that the instructor for that course simply got some wrong
> information, but good luck teaching Google that.  I figure once its spider
> crawls this e-mail in the archives, it'll further strengthen hits like the above.

 FWIW according to the MIPS IV ISA manual[1], which seems to be the most 
comprehensive legacy MIPS ISA instruction reference, COP0 encodings, and 
hence the WAIT instruction, have never been a part of any of the legacy 
MIPS ISAs, they have always been implementation-specific.

  Maciej

References:

[1] "MIPS IV Instruction Set", MIPS Technologies, Inc., Revision 3.2, By
    Charles Price, September, 1995, Section A 8.3.1 "Coprocessor 0 - 
    COP0", p. A-176
    <http://techpubs.sgi.com/library/manuals/2000/007-2597-001/pdf/007-2597-001.pdf>

Re: Does the R10K family support the "wait" instruction?

[Ralf Baechle]

On Mon, Mar 27, 2017 at 05:04:27PM -0400, Joshua Kinard wrote:

> On 03/27/2017 09:05, Ralf Baechle wrote:
> > On Sun, Mar 26, 2017 at 09:50:08PM -0400, Joshua Kinard wrote:
> > 
> >> Does anyone know if the R1x000 family of CPUs support the "wait" instruction?
> >> The 'check_wait' function in arch/mips/kernel/idle.c doesn't have a case for
> >> any of the R10K CPUs, and I can't find any specific guidance in the final R10K
> >> manual produced by Renesas, nor in the MIPS-IV instruction set.  It appears
> >> this was added in MIPS-II, and the R4K CPUs use it, with one version for when
> >> interrupts are enabled, and one where they're disabled.  Since a lot of CPUs
> >> tend to reuse R4K-compatible code, I wasn't sure.
> > 
> > Interesting, didn't know Renesas did another R10000 manual.  Presumably
> > they only rebranded NEC's manual?
> 
> Yup, mostly.  It's document U10278EJ4V0UM00, 4th edition, though I can't get
> the search engine on their site to cough a link up, so I've attached the copy I
> found.  The manual looks like it was dated from March 2001, which covers at
> least up to R12000, but has a "phased-out/discontinued" stamp and a notice to
> customers dated April 2010.
> 
> I still, to this date, have never come across any documentation on the R14000
> or R16000.  That must be internal to SGI and HP (used in their NonStop servers)
> and never saw the light of day.  Technically HP now, since they recently
> scooped up what was left of SGI.
> 
> 
> > If you have any documentation to indicate a MIPS II CPU to support WAIT,
> > I'm interested.  From all that I know the feature was introduced by the
> > R4600.
> 
> One of my Google searches, using the keywords "mips wait instruction", has been
> returning search results from a Harvard computer sciences course detailing the
> "instructional operating system OS/161", which uses an instructional CPU that
> borrows heavily from the R3000 (which they dub MIPS-161):
> 
> http://os161.eecs.harvard.edu/documentation/sys161-1.99.07/mips.html
> 
> In there, they state:
> "The WAIT instruction has been borrowed from MIPS-II. This operation puts the
> processor into a low-power state and suspends execution until some external
> event occurs, such as an interrupt. Since the exact behavior of WAIT is not
> clearly specified anywhere I could find, the MIPS-161 behavior is as follows"
> 
> So it could be that the instructor for that course simply got some wrong
> information, but good luck teaching Google that.  I figure once its spider
> crawls this e-mail in the archives, it'll further strengthen hits like the above.

Hint, MIPS II was defined by the '89 R6000, a CPU implemented in ECL.  That's
a technology that sucks the wiring from the wall (3 CPU 90MHz R6000 = 6kW!).

Best I can say the R4600 was the first CPU to support the WAIT instruction.

There were a number of not quite proper MIPS II CPUs which appeared to be
like MIPS I pimped with a bit of MIPS III.  IDT called theirs "enhanced
MIPS II".

> It might make sense to add the R10K cases to the switch in cpu_wait, right
> above the "default" block, with a comment stating that R10K doesn't support
> wait.  That way it's documented in the right spot in case this question ever
> comes up in the future.  I can send a patch for this later on if interested.

Yeah, one might do that for documentation purposes.  It won't result in
kernel bloat after all.

  Ralf