Time to remove platforms/cell?

Time to remove platforms/cell?

[Michael Ellerman]

Discuss ...

cheers

Re: Time to remove platforms/cell?

[Marc Dietrich]

[-- Attachment #1: Type: text/plain, Size: 144 bytes --]

Am Donnerstag, 17. September 2015, 15:28:47 schrieb Michael Ellerman:
> Discuss ...

as long as Geoff still maintains the ps3 port - why?

Marc

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: Time to remove platforms/cell?

[Arnd Bergmann]

On Thursday 17 September 2015 10:43:39 Marc Dietrich wrote:
> Am Donnerstag, 17. September 2015, 15:28:47 schrieb Michael Ellerman:
> > Discuss ...
> 
> as long as Geoff still maintains the ps3 port - why?
> 
We already removed celleb a while ago, which was arguably the least
commonly used one.

Within platforms/cell, we have three separate portions that we need
to look at:

a) Common ps3/ibm parts: spufs, oprofile
   We should not remove them before we plan to also remove platforms/ps3
   support, which doesn't seem likely in the near term. We can move them
   to platforms/ps3 if we decide to remove the rest.

b) Support for IBM blades:
   It is unlikely that there are QS20 blades still around and being used
   at all, but there is very little code specific to them.
   For QS21/QS22, there are probably still a few in existence, but
   I have no idea whether anybody would consider running a 4.x kernel
   on them. There are also some customer-specific Cell machines that
   are vaguely related to QS22 and that are in a similar state.
   I don't mind removing the code, but if anybody is still using it
   on new kernels, we should be prepared to put it back.

c) QPACE. We know who the three users were, and they have upgraded to
   QPACE2 (based on Intel Xeon Phi) this year. Removing this would be
   appreciated as it lets us clean up one of the ugly corners of the
   8250 uart driver.

	Arnd

Re: Time to remove platforms/cell?

[Michael Ellerman]

On Thu, 2015-09-17 at 10:43 +0200, Marc Dietrich wrote:
> Am Donnerstag, 17. September 2015, 15:28:47 schrieb Michael Ellerman:
> > Discuss ...
> 
> as long as Geoff still maintains the ps3 port - why?

Because even if Geoff maintains the ps3 port, there's still a non-zero cost to
us for carrying the Cell code.

To be clear there's more to Cell than just ps3. The platforms/cell code is
mainly about supporting the IBM Cell Blades, as well as spufs, which is also
used on ps3.

See for example commit 74b5037baa20 ("powerpc/mm: Fix pte_pagesize_index()
crash on 4K w/64K hash"):

  https://git.kernel.org/cgit/linux/kernel/git/powerpc/linux.git/commit/?id=74b5037baa2011a2799e2c43adde7d171b072f9e

Which is a fix for crashes we were seeing on non-cell machines, caused by some
obscure code that was added for Cell. It took several of us a few days to track
that one down.

But the main motivation would just be to drop code that no one's using. From
December 2013 until April 2015 the Cell machines (not ps3) were broken in
mainline and no one noticed. I now have automated boot tests to prevent that
happening again, but it makes me think no one is using those machines much
anymore.


You also raise a good point, which is that ps3 is a separate platform, so we
could actually keep that but get rid of platforms/cell. If we did that we'd
need to move spufs out of platforms/cell though.

Looking at the code size:

	platforms/ps3	 	 6 KSLOC
	platforms/cell		11 KSLOC
	platforms/cell/spufs	 7 KSLOC
	other cell code		 2 KSLOC

So dropping platforms/cell but not ps3 would save us ~6 KSLOC.


Having said all that, this email was mainly a fishing expedition to see if
anyone still cares about Cell support at all. It sounds like you at least are
running mainline kernels on PS3?

cheers

Re: Time to remove platforms/cell?

[Michael Ellerman]

On Thu, 2015-09-17 at 11:31 +0200, Arnd Bergmann wrote:
> On Thursday 17 September 2015 10:43:39 Marc Dietrich wrote:
> > Am Donnerstag, 17. September 2015, 15:28:47 schrieb Michael Ellerman:
> > > Discuss ...
> > 
> > as long as Geoff still maintains the ps3 port - why?
> > 
> We already removed celleb a while ago, which was arguably the least
> commonly used one.

Hi Arnd,

Thanks for the input, I probably should have CC'ed you, sorry!

> Within platforms/cell, we have three separate portions that we need
> to look at:
> 
> a) Common ps3/ibm parts: spufs, oprofile
>    We should not remove them before we plan to also remove platforms/ps3
>    support, which doesn't seem likely in the near term. We can move them
>    to platforms/ps3 if we decide to remove the rest.

Right. I wouldn't bet money that the oprofile code still works, but spufs
definitely would have to stay.

> b) Support for IBM blades:
>    It is unlikely that there are QS20 blades still around and being used
>    at all, but there is very little code specific to them.

Wasn't spider QS20 only? So that would be a bit of code that could go,
including (some of) the io-workarounds code (I think).

>    For QS21/QS22, there are probably still a few in existence, but
>    I have no idea whether anybody would consider running a 4.x kernel
>    on them. There are also some customer-specific Cell machines that
>    are vaguely related to QS22 and that are in a similar state.
>    I don't mind removing the code, but if anybody is still using it
>    on new kernels, we should be prepared to put it back.

Yeah, that's my impression. We obviously still have a QS22, but really we're
only keeping it alive for testing purposes.

Hopefully this email will bring some more folks out of the woodwork.

> c) QPACE. We know who the three users were, and they have upgraded to
>    QPACE2 (based on Intel Xeon Phi) this year. Removing this would be
>    appreciated as it lets us clean up one of the ugly corners of the
>    8250 uart driver.

Ah excellent info.

I had done some googling on QPACE and from the web site there is no suggestion
that it's been shutdown, but I guess that's the nature of web pages, they tend
not to get updated:

  http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/QPACE/QPACE_node.html


So we can probably remove the QPACE code sooner rather than later, unless
someone yells.

cheers

Re: Time to remove platforms/cell?

[Marc Dietrich]

[-- Attachment #1: Type: text/plain, Size: 635 bytes --]

Am Donnerstag, 17. September 2015, 19:45:38 schrieb Michael Ellerman:
> On Thu, 2015-09-17 at 10:43 +0200, Marc Dietrich wrote:
> > Am Donnerstag, 17. September 2015, 15:28:47 schrieb Michael Ellerman:
>
> [...]
>
> Having said all that, this email was mainly a fishing expedition to see if
> anyone still cares about Cell support at all. It sounds like you at least
> are running mainline kernels on PS3?

well, I like to but I couldn't get the kernel booting for some reason. Still 
trying. Beside this, I have nothing against dropping the other cell systems, 
but I'm only a "fun user", so my vote probably isn't counting :-)

Marc

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: Time to remove platforms/cell?

[Arnd Bergmann]

On Thursday 17 September 2015 20:05:25 Michael Ellerman wrote:
> On Thu, 2015-09-17 at 11:31 +0200, Arnd Bergmann wrote:
> > b) Support for IBM blades:
> >    It is unlikely that there are QS20 blades still around and being used
> >    at all, but there is very little code specific to them.
> 
> Wasn't spider QS20 only? So that would be a bit of code that could go,
> including (some of) the io-workarounds code (I think).

Spider is also used on PS3, so if anyone is still using that as bare-metal
(hacked) rather than OtherOS mode, it would be needed. Don't know if we
ever supported that upstream though.

> >    For QS21/QS22, there are probably still a few in existence, but
> >    I have no idea whether anybody would consider running a 4.x kernel
> >    on them. There are also some customer-specific Cell machines that
> >    are vaguely related to QS22 and that are in a similar state.
> >    I don't mind removing the code, but if anybody is still using it
> >    on new kernels, we should be prepared to put it back.
> 
> Yeah, that's my impression. We obviously still have a QS22, but really we're
> only keeping it alive for testing purposes.
> 
> Hopefully this email will bring some more folks out of the woodwork.

FWIW, the last data points from Mercury and Fixstars (who were both offering
OEM products based on QS2x) were from 2009. IBM officially stopped selling the
QS22 in January 2012, which is much later than what I remembered as the day
they stopped offering support.

	Arnd

Re: Time to remove platforms/cell?

[Arnd Bergmann]

On Thursday 17 September 2015 12:28:13 Marc Dietrich wrote:
> Am Donnerstag, 17. September 2015, 19:45:38 schrieb Michael Ellerman:
> > On Thu, 2015-09-17 at 10:43 +0200, Marc Dietrich wrote:
> > > Am Donnerstag, 17. September 2015, 15:28:47 schrieb Michael Ellerman:
> >
> > [...]
> >
> > Having said all that, this email was mainly a fishing expedition to see if
> > anyone still cares about Cell support at all. It sounds like you at least
> > are running mainline kernels on PS3?
> 
> well, I like to but I couldn't get the kernel booting for some reason. Still 
> trying. Beside this, I have nothing against dropping the other cell systems, 
> but I'm only a "fun user", so my vote probably isn't counting 

No, your vote counts at least as much as a commercial user, because
"fun users" are more likely to contribute back patches when something
breaks.

	Arnd

Re: Time to remove platforms/cell?

[Geoff Levand]

On Thu, 2015-09-17 at 20:05 +1000, Michael Ellerman wrote:

> Right. I wouldn't bet money that the oprofile code still works, but spufs
> definitely would have to stay.

oprofile on PS3 needed additional patches [1] that I never
merged upstream, and no longer plan to, so oprofile support
should be removed.

If someone in the future needs oprofile it wouldn't be too
hard to get what is there now working again. 

[1] http://git.kernel.org/cgit/linux/kernel/git/geoff/ps3-linux.git/log/?h=ps3-queue-v3.7

-Geoff

Re: Time to remove platforms/cell?

[Michael Ellerman]

On Thu, 2015-09-17 at 13:15 +0200, Arnd Bergmann wrote:
> On Thursday 17 September 2015 12:28:13 Marc Dietrich wrote:
> > Am Donnerstag, 17. September 2015, 19:45:38 schrieb Michael Ellerman:
> > > On Thu, 2015-09-17 at 10:43 +0200, Marc Dietrich wrote:
> > > > Am Donnerstag, 17. September 2015, 15:28:47 schrieb Michael Ellerman:
> > >
> > > [...]
> > >
> > > Having said all that, this email was mainly a fishing expedition to see if
> > > anyone still cares about Cell support at all. It sounds like you at least
> > > are running mainline kernels on PS3?
> > 
> > well, I like to but I couldn't get the kernel booting for some reason. Still 
> > trying. Beside this, I have nothing against dropping the other cell systems, 
> > but I'm only a "fun user", so my vote probably isn't counting 
> 
> No, your vote counts at least as much as a commercial user, because
> "fun users" are more likely to contribute back patches when something
> breaks.

Yep. I'm quite happy for us to try and maintain support indefinitely if folks
are still using the systems, and especially if they're able to report the odd
breakage that happens.

cheers

Re: Time to remove platforms/cell?

[Michael Ellerman]

On Thu, 2015-09-17 at 12:28 +0200, Marc Dietrich wrote:
> Am Donnerstag, 17. September 2015, 19:45:38 schrieb Michael Ellerman:
> > On Thu, 2015-09-17 at 10:43 +0200, Marc Dietrich wrote:
> > > Am Donnerstag, 17. September 2015, 15:28:47 schrieb Michael Ellerman:
> >
> > [...]
> >
> > Having said all that, this email was mainly a fishing expedition to see if
> > anyone still cares about Cell support at all. It sounds like you at least
> > are running mainline kernels on PS3?
> 
> well, I like to but I couldn't get the kernel booting for some reason. Still 
> trying.

OK. If you need help feel free to post on here. There's no guarantee anyone
will have free time to help, but you can always ask :)

cheers

Re: Time to remove platforms/cell?

[Geoff Levand]

Hi,

On Thu, 2015-09-17 at 12:28 +0200, Marc Dietrich wrote:
> well, I like to but I couldn't get the kernel booting for some reason. Still 
> trying. 

I just tested the ps3-queue branch (v4.3-rc4 based) of my ps3-linux git
repo and it works OK.

Do you see any output on the display, maybe some penguins at least?

-Geoff

Re: Time to remove platforms/cell?

[Marc Dietrich]

[-- Attachment #1: Type: text/plain, Size: 789 bytes --]

Hi Geoff,

Am Sonntag, 4. Oktober 2015, 12:27:19 schrieb Geoff Levand:
> Hi,
> 
> On Thu, 2015-09-17 at 12:28 +0200, Marc Dietrich wrote:
> > well, I like to but I couldn't get the kernel booting for some reason.
> > Still trying.
> 
> I just tested the ps3-queue branch (v4.3-rc4 based) of my ps3-linux git
> repo and it works OK.
> 
> Do you see any output on the display, maybe some penguins at least


I tried with ps3-queue and still no luck. Petitboot just says "Booting kernel 
..." thats all - no output.

FW is 3.15 of course
Petitboot says (white-09.09.01-15.56)
compiled kernel with gcc 4.4.2 (fedora 13)
git clean -fdx; make ps3_defconfig; make vmlinux modules; make modules_install
strip --strip-unneeded vmlinux
boot this kernel

Maybe still something missing?

Thanks!

Marc

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: Time to remove platforms/cell?

[Geoff Levand]

On Mon, 2015-10-05 at 12:27 +0200, Marc Dietrich wrote:
> Am Sonntag, 4. Oktober 2015, 12:27:19 schrieb Geoff Levand:
> > I just tested the ps3-queue branch (v4.3-rc4 based) of my ps3-linux
> > git
> > repo and it works OK.
> > 
> > Do you see any output on the display, maybe some penguins at least
> 
> 
> I tried with ps3-queue and still no luck. Petitboot just says
> "Booting kernel 
> ..." thats all - no output.
> 
> FW is 3.15 of course
> Petitboot says (white-09.09.01-15.56)
> compiled kernel with gcc 4.4.2 (fedora 13)
> git clean -fdx; make ps3_defconfig; make vmlinux modules; make
> modules_install
> strip --strip-unneeded vmlinux

I think I can reproduce your problem, but didn't have time to look into
it.  I'll try to find time over the weekend.

If you want to try some things yourself, one is to exit to the
petitboot shell and try a manual kexec from there to see if there are
any messages that might explain what's going wrong.  Another is to use
git bisect to try to locate the kernel commit that introduced the
problem.  Let me know anything you find if you try.

-Geoff

Re: Time to remove platforms/cell?

[Geoff Levand]

> Hi,
> 
> > On Mon, 2015-10-05 at 12:27 +0200, Marc Dietrich wrote:
> > > I tried with ps3-queue and still no luck. Petitboot just says
> > > "Booting kernel
> > > ..." thats all - no output.
> > > 
> > > FW is 3.15 of course
> > > Petitboot says (white-09.09.01-15.56)
> 
> I spent a few minutes on this, and it seems the 2nd stage kernel
> panics when
> setting up the SPUs.  You could try with CONFIG_SPU_FS=n.
> 
> I'll look into it more as I have time.
> 
> -Geoff

Re: Time to remove platforms/cell?

[Geoff Levand]

Hi Marc,

On Thu, 2015-10-08 at 11:10 -0700, Geoff Levand wrote:
> > On Mon, 2015-10-05 at 12:27 +0200, Marc Dietrich wrote:
> > > > I tried with ps3-queue and still no luck. Petitboot just says
> > > > "Booting kernel
> > > > ..." thats all - no output.
> > > > 
> > > > FW is 3.15 of course
> > > > Petitboot says (white-09.09.01-15.56)
> > 
> > I spent a few minutes on this, and it seems the 2nd stage kernel
> > panics when
> > setting up the SPUs.  You could try with CONFIG_SPU_FS=n.

With the 4.2-rc4 kernel, kexec seems to work when CONFIG_SPU_FS=n.  I
have not tried with petitboot release white-09.09.01-15.56 yet though.

-Geoff

Re: Time to remove platforms/cell?

[Marc Dietrich]

[-- Attachment #1: Type: text/plain, Size: 920 bytes --]

Hi Geoff,

Am Freitag, 9. Oktober 2015, 10:45:42 schrieb Geoff Levand:
> Hi Marc,
> 
> On Thu, 2015-10-08 at 11:10 -0700, Geoff Levand wrote:
> > > On Mon, 2015-10-05 at 12:27 +0200, Marc Dietrich wrote:
> > > > > I tried with ps3-queue and still no luck. Petitboot just says
> > > > > "Booting kernel
> > > > > ..." thats all - no output.
> > > > > 
> > > > > FW is 3.15 of course
> > > > > Petitboot says (white-09.09.01-15.56)
> > > 
> > > I spent a few minutes on this, and it seems the 2nd stage kernel
> > > panics when
> > > setting up the SPUs.  You could try with CONFIG_SPU_FS=n.
> 
> With the 4.2-rc4 kernel, kexec seems to work when CONFIG_SPU_FS=n.  I
> have not tried with petitboot release white-09.09.01-15.56 yet though.

I guess you mean 4.3-rc4. Still no luck with CONFIG_SPU_FS=n. No output even 
with direct calling kexec. Can you put a working kernel/initrd to some place 
so I can try this?

Marc

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: Time to remove platforms/cell?

[Geoff Levand]

Hi,

On Fri, 2015-10-09 at 23:13 +0200, Marc Dietrich wrote:
> Am Freitag, 9. Oktober 2015, 10:45:42 schrieb Geoff Levand:
> > With the 4.2-rc4 kernel, kexec seems to work when CONFIG_SPU_FS=n. 
> >  I
> > have not tried with petitboot release white-09.09.01-15.56 yet though.
> 
> I guess you mean 4.3-rc4. Still no luck with CONFIG_SPU_FS=n. No output even 
> with direct calling kexec. Can you put a working kernel/initrd to some place 
> so I can try this?

I did some more work on this, but still did not find out what the problem
is.  Kexec from 4.3 -> 4.3 works, but with the white-09.09.01-15.56
petitboot (2.6.30.9) -> 4.3 gets a kernel panic:

 -> early_setup(), dt_ptr: 0x7fff000
Kernel panic - not syncing: BUG: Failed verifying flat device tree, bad version?

I'll look at it some more as I find time.

-Geoff

Re: Time to remove platforms/cell?

[Michael Ellerman]

On Thu, 2015-10-22 at 12:15 -0700, Geoff Levand wrote:

> Hi,
> 
> On Fri, 2015-10-09 at 23:13 +0200, Marc Dietrich wrote:

> > Am Freitag, 9. Oktober 2015, 10:45:42 schrieb Geoff Levand:

> > > With the 4.2-rc4 kernel, kexec seems to work when CONFIG_SPU_FS=n. 
> > >  I
> > > have not tried with petitboot release white-09.09.01-15.56 yet though.
> > 
> > I guess you mean 4.3-rc4. Still no luck with CONFIG_SPU_FS=n. No output even 
> > with direct calling kexec. Can you put a working kernel/initrd to some place 
> > so I can try this?
> 
> I did some more work on this, but still did not find out what the problem
> is.  Kexec from 4.3 -> 4.3 works, but with the white-09.09.01-15.56
> petitboot (2.6.30.9) -> 4.3 gets a kernel panic:
> 
>  -> early_setup(), dt_ptr: 0x7fff000
> Kernel panic - not syncing: BUG: Failed verifying flat device tree, bad version?
> 
> I'll look at it some more as I find time.

That says that the device tree firmware gave you (ie. from kexec), is using an
old version of the device tree format.

I can't remember off the top of my head which version you need, but basically
newer kernels require a newer device tree format. So your kexec might be too
old?

cheers

Re: Time to remove platforms/cell?

[Marc Dietrich]

[-- Attachment #1: Type: text/plain, Size: 1705 bytes --]

Am Freitag, 23. Oktober 2015, 12:54:56 schrieb Michael Ellerman:
> On Thu, 2015-10-22 at 12:15 -0700, Geoff Levand wrote:
> > Hi,
> > 
> > On Fri, 2015-10-09 at 23:13 +0200, Marc Dietrich wrote:
> > > Am Freitag, 9. Oktober 2015, 10:45:42 schrieb Geoff Levand:
> > > > With the 4.2-rc4 kernel, kexec seems to work when CONFIG_SPU_FS=n.
> > > > 
> > > >  I
> > > > 
> > > > have not tried with petitboot release white-09.09.01-15.56 yet though.
> > > 
> > > I guess you mean 4.3-rc4. Still no luck with CONFIG_SPU_FS=n. No output
> > > even with direct calling kexec. Can you put a working kernel/initrd to
> > > some place so I can try this?
> > 
> > I did some more work on this, but still did not find out what the problem
> > is.  Kexec from 4.3 -> 4.3 works, but with the white-09.09.01-15.56
> > 
> > petitboot (2.6.30.9) -> 4.3 gets a kernel panic:
> >  -> early_setup(), dt_ptr: 0x7fff000
> > 
> > Kernel panic - not syncing: BUG: Failed verifying flat device tree, bad
> > version?
> > 
> > I'll look at it some more as I find time.
> 
> That says that the device tree firmware gave you (ie. from kexec), is using
> an old version of the device tree format.
> 
> I can't remember off the top of my head which version you need, but
> basically newer kernels require a newer device tree format. So your kexec
> might be too old?

that reminds me about a different question. I said before that I use plain 
vmlinux for the kernel image. This works at least with 2.6.35 (the newest 
kernel I got booting so far). But I also saw that a device tree is being 
compiled and linked with the kernel to produce a dt image in 
arch/powerpc/boot. Is it possible that newer kernels require a device tree?

Marc

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]