Can we drop upstream Linux x32 support?

Can we drop upstream Linux x32 support?

[Andy Lutomirski]

Hi all-

I'm seriously considering sending a patch to remove x32 support from
upstream Linux.  Here are some problems with it:

1. It's not entirely clear that it has users.  As far as I know, it's
supported on Gentoo and Debian, and the Debian popcon graph for x32
has been falling off dramatically.  I don't think that any enterprise
distro has ever supported x32.

2. The way that system calls work is very strange.  Most syscalls on
x32 enter through their *native* (i.e. not COMPAT_SYSCALL_DEFINE)
entry point, and this is intentional.  For example, adjtimex() uses
the native entry, not the compat entry, because x32's struct timex
matches the x86_64 layout.  But a handful of syscalls have separate
entry points -- these are the syscalls starting at 512.  These enter
through the COMPAT_SYSCALL_DEFINE entry points.

The x32 syscalls that are *not* in the 512 range violate all semblance
of kernel syscall convention.  In the syscall handlers,
in_compat_syscall() returns true, but the COMPAT_SYSCALL_DEFINE entry
is not invoked.   This is nutty and risks breaking things when people
refactor their syscall implementations.  And no one tests these
things.  Similarly, if someone calls any of the syscalls below 512 but
sets bit 31 in RAX, then the native entry will be called with
in_compat_set().

Conversely, if you call a syscall in the 512 range with bit 31
*clear*, then the compat entry is set with in_compat_syscall()
*clear*.  This is also nutty.

Finally, the kernel has a weird distinction between CONFIG_X86_X32_ABI
and and CONFIG_X86_X32, which I suspect results in incorrect builds if
the host doesn't have an x32 toolchain installed.

I propose that we make CONFIG_X86_X32 depend on BROKEN for a release
or two and then remove all the code if no one complains.  If anyone
wants to re-add it, IMO they're welcome to do so, but they need to do
it in a way that is maintainable.

--Andy

Re: Can we drop upstream Linux x32 support?

[Linus Torvalds]

On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski <luto@kernel.org> wrote:
>
> I'm seriously considering sending a patch to remove x32 support from
> upstream Linux.  Here are some problems with it:

I talked to Arnd (I think - we were talking about all the crazy ABI's,
but maybe it was with somebody else) about exactly this in Edinburgh.

Apparently the main real use case is for extreme benchmarking. It's
the only use-case where the complexity of maintaining a whole
development environment and distro is worth it, it seems. Apparently a
number of Spec submissions have been done with the x32 model.

I'm not opposed to trying to sunset the support, but let's see who complains..

              Linus

Re: Can we drop upstream Linux x32 support?

[hpa]

On December 10, 2018 5:40:33 PM PST, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski <luto@kernel.org>
>wrote:
>>
>> I'm seriously considering sending a patch to remove x32 support from
>> upstream Linux.  Here are some problems with it:
>
>I talked to Arnd (I think - we were talking about all the crazy ABI's,
>but maybe it was with somebody else) about exactly this in Edinburgh.
>
>Apparently the main real use case is for extreme benchmarking. It's
>the only use-case where the complexity of maintaining a whole
>development environment and distro is worth it, it seems. Apparently a
>number of Spec submissions have been done with the x32 model.
>
>I'm not opposed to trying to sunset the support, but let's see who
>complains..
>
>              Linus

The use case aside, I need to address the technical issues in this post; some of the behaviors that Andy is pointing out area quite intentional, even if they are perhaps somewhat confusing at first glance. That being said, some were due to tradeoffs that might have been wrong.
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Can we drop upstream Linux x32 support?

[Florian Weimer]

* Linus Torvalds:

> Apparently the main real use case is for extreme benchmarking. It's
> the only use-case where the complexity of maintaining a whole
> development environment and distro is worth it, it seems. Apparently a
> number of Spec submissions have been done with the x32 model.

Are you sure GCC was used for that?

ICC has a 32-bit pointer mode which does not change the (64-bit) kernel
interface.

Thanks,
Florian

Re: Can we drop upstream Linux x32 support?

[Thorsten Glaser]

Note: please keep me in Cc because I am not subscribed.

Linus Torvalds dixit:

>I'm not opposed to trying to sunset the support, but let's see who complains..

I will hereby complain.

I’m using Debian/x32 on my main desktop at work, and do
occasionally help out with porting issues. It’s a good
way to make more out of 64-bit machines without going
all 64 bit; it’s also helped me find bugs in software.
It’s a nice architectural idea, and a way forward for
things that are constricted to 32 bits while opening
up stuff like 64-bit time_t without taking up half the
available CPU registers (while more than doubling the
number of the available CPU registers, too).

I was also considering investing a nontrivial amount of
work into porting klibc to x32, since hpa does not wish
to do it himself. Thankfully I have only done a bit yet.

Furthermore, x32 was the first of the many *64ilp32
architectures; I know I’ve seen amd64ilp32 and at least
one other I don’t recall. It will have prototyped many
of the problems users of these will run in, and I’d prefer
to keep it (completely selfish because I don’t wish to
have to crossgrade a whole system yet again).

Thanks for your consideration,
//mirabilos
--  
When he found out that the m68k port was in a pretty bad shape, he did
not, like many before him, shrug and move on; instead, he took it upon
himself to start compiling things, just so he could compile his shell.
How's that for dedication. -- Wouter, about my Debian/m68k revival

Re: Can we drop upstream Linux x32 support?

[Andy Lutomirski]

On Tue, Dec 11, 2018 at 2:14 PM Thorsten Glaser <tg@mirbsd.de> wrote:
>
> Note: please keep me in Cc because I am not subscribed.
>
> Linus Torvalds dixit:
>
> >I'm not opposed to trying to sunset the support, but let's see who complains..
>
> I will hereby complain.
>
> I’m using Debian/x32 on my main desktop at work, and do
> occasionally help out with porting issues. It’s a good
> way to make more out of 64-bit machines without going
> all 64 bit; it’s also helped me find bugs in software.
> It’s a nice architectural idea, and a way forward for
> things that are constricted to 32 bits while opening
> up stuff like 64-bit time_t without taking up half the
> available CPU registers (while more than doubling the
> number of the available CPU registers, too).

Thanks for responding!

I suppose the question is: are you enough of a user to justify the
continued maintenance effort.

>
> I was also considering investing a nontrivial amount of
> work into porting klibc to x32, since hpa does not wish
> to do it himself. Thankfully I have only done a bit yet.
>
> Furthermore, x32 was the first of the many *64ilp32
> architectures; I know I’ve seen amd64ilp32 and at least
> one other I don’t recall. It will have prototyped many
> of the problems users of these will run in, and I’d prefer
> to keep it (completely selfish because I don’t wish to
> have to crossgrade a whole system yet again).

it kind of seems like arm64's lesson is "don't do it like x32".

There's some effort going on right now to make it possible to add
syscalls without having to muck with every single architecture.  I
don't really want x32 to derail that effort.  I suppose we could say
that x32 stays but that it simply gets no new syscalls, but that seems
a bit lame.  Unfortunately, on x86, x32 really is a third ABI that is
not compatible in a structure-memory-layout sense with the other two.
What happens if someone adds a struct like:

struct nasty_on_x32 {
  __kernel_long_t a;
  void * __user b;
};

On x86_64, that's two 8-byte fields.  On x86_32, it's two four-byte
fields.  On x32, it's an 8-byte field and a 4-byte field.  Now what?

I'm sure we could have some magic gcc plugin or other nifty tool that gives us:

copy_from_user(struct struct_name, kernel_ptr, user_ptr);

where it automatically generates code for all possible ABIs to copy
over the struct and dispatches dynamically based on the current
syscall ABI, but I have trouble imagining anyone volunteering to
actually do this work.  Instead we get ad hoc fixes for each syscall,
along the lines of preadv64v2(), which get done when somebody notices
a problem.

Linus, any advice here?

--Andy

Re: Can we drop upstream Linux x32 support?

[Thorsten Glaser]

Andy Lutomirski dixit:

>What happens if someone adds a struct like:
>
>struct nasty_on_x32 {
>  __kernel_long_t a;
>  void * __user b;
>};
>
>On x86_64, that's two 8-byte fields.  On x86_32, it's two four-byte
>fields.  On x32, it's an 8-byte field and a 4-byte field.  Now what?

Yes, that’s indeed ugly. I understand. But don’t we already have
this problem with architectures which support multiple ABIs at the
same time? An amd64 kernel with i386 userspace comes to mind, or
the multiple MIPS ABIs.

>I'm sure we could have some magic gcc plugin or other nifty tool that
>gives us:
>
>copy_from_user(struct struct_name, kernel_ptr, user_ptr);

Something like that might be useful. Generate call stubs, which
then call the syscall implementation with the actual user-space
struct contents as arguments. Hm, that might be too generic to
be useful. Generate macros that can read from or write specific
structures to userspace?

I think something like this could solve other more general problems
as well, so it might be “nice to have anyway”. Of course it’s work,
and I’m not involved enough in Linux kernel programming to be able
to usefully help with it (doing too much elsewhere already).

>actually do this work.  Instead we get ad hoc fixes for each syscall,
>along the lines of preadv64v2(), which get done when somebody notices

Yes, that’s absolutely ugly and ridiculous and all kinds of bad.

On the other hand, from my current experience, someone (Arnd?) noticed
all the currently existing baddies for x32 already and fixed them.

New syscalls are indeed an issue, but perhaps something generating
copyinout stubs could help. This might allow other architectures
that could do with a new ABI but have until now feared the overhead
as well. (IIRC, m68k could do with a new ABI that reserves a register
for TLS, but Geert would know. At the same time, time_t and off_t could
be bumped to 64 bit. Something like that. If changing sizes of types
shared between kernel and user spaces is not something feared…)

Thanks for considering,
//mirabilos
-- 
„Cool, /usr/share/doc/mksh/examples/uhr.gz ist ja ein Grund,
mksh auf jedem System zu installieren.“
	-- XTaran auf der OpenRheinRuhr, ganz begeistert
(EN: “[…]uhr.gz is a reason to install mksh on every system.”)

Re: Can we drop upstream Linux x32 support?

[Arnd Bergmann]

On Wed, Dec 12, 2018 at 12:38 AM Thorsten Glaser <tg@mirbsd.de> wrote:
>
> Andy Lutomirski dixit:
>
> >What happens if someone adds a struct like:
> >
> >struct nasty_on_x32 {
> >  __kernel_long_t a;
> >  void * __user b;
> >};
> >
> >On x86_64, that's two 8-byte fields.  On x86_32, it's two four-byte
> >fields.  On x32, it's an 8-byte field and a 4-byte field.  Now what?
>
> Yes, that’s indeed ugly. I understand. But don’t we already have
> this problem with architectures which support multiple ABIs at the
> same time? An amd64 kernel with i386 userspace comes to mind, or
> the multiple MIPS ABIs.

The main trouble that I see with x32 is in how it's different
from all the other compat architectures. We have a number
of drivers that support not only two ABIs (native and compat)
but also a third one specifically for x32, and more drivers
that fail on x32 because they never implemented the third
case.

Note that this is not a problem for MIPS n32 and aarch64ilp32,
which generally stick to the regular compat ABI (each compat
architecture has some minor quirks somewhere, much less so
than x32 does).

> >actually do this work.  Instead we get ad hoc fixes for each syscall,
> >along the lines of preadv64v2(), which get done when somebody notices
>
> Yes, that’s absolutely ugly and ridiculous and all kinds of bad.
>
> On the other hand, from my current experience, someone (Arnd?) noticed
> all the currently existing baddies for x32 already and fixed them.

I did some, and others did more, but it's hard to ever do a complete
job, and even harder to prevent new bugs from creeping in.

> New syscalls are indeed an issue, but perhaps something generating
> copyinout stubs could help. This might allow other architectures
> that could do with a new ABI but have until now feared the overhead
> as well. (IIRC, m68k could do with a new ABI that reserves a register
> for TLS, but Geert would know. At the same time, time_t and off_t could
> be bumped to 64 bit. Something like that. If changing sizes of types
> shared between kernel and user spaces is not something feared…)

Creating a completely new ABI for an existing architecutre is generally
something you want to avoid, as this means all existing user space
has to be recompiled.

All architectures support 64-bit off_t already, and soon will support
64-bit time_t as well, at this point (as of this week's linux-next)
we mostly just need to assign a couple of syscall numbers for each
32-bit architectures.

        Arnd

Re: Can we drop upstream Linux x32 support?

[Andy Lutomirski]

> On Dec 11, 2018, at 3:35 PM, Thorsten Glaser <tg@mirbsd.de> wrote:
>
> Andy Lutomirski dixit:
>
>> What happens if someone adds a struct like:
>>
>> struct nasty_on_x32 {
>> __kernel_long_t a;
>> void * __user b;
>> };
>>
>> On x86_64, that's two 8-byte fields.  On x86_32, it's two four-byte
>> fields.  On x32, it's an 8-byte field and a 4-byte field.  Now what?
>
> Yes, that’s indeed ugly. I understand. But don’t we already have
> this problem with architectures which support multiple ABIs at the
> same time? An amd64 kernel with i386 userspace comes to mind, or
> the multiple MIPS ABIs.

That’s the thing, though: the whole generic kernel compat
infrastructure assumes there are at most two ABIs: native and, if
enabled and relevant, compat. x32 breaks this entirely.

>
>> I'm sure we could have some magic gcc plugin or other nifty tool that
>> gives us:
>>
>> copy_from_user(struct struct_name, kernel_ptr, user_ptr);
>
> Something like that might be useful. Generate call stubs, which
> then call the syscall implementation with the actual user-space
> struct contents as arguments. Hm, that might be too generic to
> be useful. Generate macros that can read from or write specific
> structures to userspace?
>
> I think something like this could solve other more general problems
> as well, so it might be “nice to have anyway”. Of course it’s work,
> and I’m not involved enough in Linux kernel programming to be able
> to usefully help with it (doing too much elsewhere already).
>
>> actually do this work.  Instead we get ad hoc fixes for each syscall,
>> along the lines of preadv64v2(), which get done when somebody notices
>
> Yes, that’s absolutely ugly and ridiculous and all kinds of bad.
>
> On the other hand, from my current experience, someone (Arnd?) noticed
> all the currently existing baddies for x32 already and fixed them.
>
> New syscalls are indeed an issue, but perhaps something generating
> copyinout stubs could help. This might allow other architectures
> that could do with a new ABI but have until now feared the overhead
> as well. (IIRC, m68k could do with a new ABI that reserves a register
> for TLS, but Geert would know. At the same time, time_t and off_t could
> be bumped to 64 bit. Something like that. If changing sizes of types
> shared between kernel and user spaces is not something feared…)

Magic autogenerated stubs would be great.  Difficult, too, given
unions, multiplexers, cmsg, etc.

I suppose I will see how bad it would be to split out the x32 syscall
table and at least isolate the mess to some extent.

IMO the real right solution would be to push the whole problem to
userspace: get an ILP32 system working with almost or entirely LP64
syscalls.  POSIX support might have to be a bit flexible, but still.
How hard would it be to have __attribute__((ilp64)), with an optional
warning if any embedded structs are not ilp64?  This plus a wrapper to
make sure that mmap puts everything below 4GB ought to do the trick.
Or something like what arm64 is proposing where the kernel ABI has
32-bit long doesn’t seem too horrible.

Re: Can we drop upstream Linux x32 support?

[Thorsten Glaser]

Andy Lutomirski dixit:

>That’s the thing, though: the whole generic kernel compat
>infrastructure assumes there are at most two ABIs: native and, if
>enabled and relevant, compat. x32 breaks this entirely.

MIPS had o32, n32, n64 since like forever.

ARM has old ABI, EABI and now 64-bit.

Other architectures are yet to come.

>IMO the real right solution would be to push the whole problem to
>userspace: get an ILP32 system working with almost or entirely LP64

Is this a reflex of Linux kernel developers? ;-)

I doubt that userspace is the right place for this, remember
the recent glibc vs. syscalls debate. It would also need to
multiply across various libcs.

>How hard would it be to have __attribute__((ilp64)), with an optional
>warning if any embedded structs are not ilp64?  This plus a wrapper to

You mean LP64. Impossible, because LP64 vs. ILP32 is not the only
difference between amd64 and x32.

bye,
//mirabilos
-- 
I believe no one can invent an algorithm. One just happens to hit upon it
when God enlightens him. Or only God invents algorithms, we merely copy them.
If you don't believe in God, just consider God as Nature if you won't deny
existence.		-- Coywolf Qi Hunt

Re: Can we drop upstream Linux x32 support?

[Arnd Bergmann]

On Wed, Dec 12, 2018 at 3:38 AM Thorsten Glaser <tg@mirbsd.de> wrote:
>
> Andy Lutomirski dixit:
>
> >That’s the thing, though: the whole generic kernel compat
> >infrastructure assumes there are at most two ABIs: native and, if
> >enabled and relevant, compat. x32 breaks this entirely.
>
> MIPS had o32, n32, n64 since like forever.

o32 and n32 are practically the same, the only difference on the
syscall ABI that I can see are the actual syscall numbers, and
the 'struct sigcontext' definition.

> ARM has old ABI, EABI and now 64-bit.

arm64 intentionally did not attempt to support OABI user space
because of this, and as I said the ilp32 ABI follows what MIPS
n32 does using the same data structures as aarch32 (corresponding
to mips o32).

> >How hard would it be to have __attribute__((ilp64)), with an optional
> >warning if any embedded structs are not ilp64?  This plus a wrapper to
>
> You mean LP64. Impossible, because LP64 vs. ILP32 is not the only
> difference between amd64 and x32.

I think the above is what Intel's compiler does, and similar to what they
do for mixing big-endian and little-endian code (!). Generally possible yes,
but a lot of work, as well as error-prone and not particular appealing for
the GNU toolchain IMHO.

        Arnd

Re: Can we drop upstream Linux x32 support?

[Joseph Myers]

On Wed, 12 Dec 2018, Arnd Bergmann wrote:

> > MIPS had o32, n32, n64 since like forever.
> 
> o32 and n32 are practically the same, the only difference on the
> syscall ABI that I can see are the actual syscall numbers, and
> the 'struct sigcontext' definition.

And for syscalls that have 64-bit arguments, n32 generally passes those in 
a single register (like n64), not pairs of registers (like o32).  But, 
yes, userspace structure layout for n32 is generally much closer to o32 
than to n64.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Can we drop upstream Linux x32 support?

[Ivan Ivanov]

Please don't drop x86 support from the Linux kernel ! You may not
realize it, but there are plenty of people who care about x86 systems.
I have a good old Pentium 4 based computer which is 32-bit, using it
for some weird Linux experiments + retro gaming, and if I can't run
new Linux kernels there - it would be SAD
ср, 12 дек. 2018 г. в 21:15, Joseph Myers <joseph@codesourcery.com>:
>
> On Wed, 12 Dec 2018, Arnd Bergmann wrote:
>
> > > MIPS had o32, n32, n64 since like forever.
> >
> > o32 and n32 are practically the same, the only difference on the
> > syscall ABI that I can see are the actual syscall numbers, and
> > the 'struct sigcontext' definition.
>
> And for syscalls that have 64-bit arguments, n32 generally passes those in
> a single register (like n64), not pairs of registers (like o32).  But,
> yes, userspace structure layout for n32 is generally much closer to o32
> than to n64.
>
> --
> Joseph S. Myers
> joseph@codesourcery.com

Re: Can we drop upstream Linux x32 support?

[Andy Lutomirski]

On Wed, Dec 12, 2018 at 10:50 AM Ivan Ivanov <qmastery16@gmail.com> wrote:
>
> Please don't drop x86 support from the Linux kernel ! You may not
> realize it, but there are plenty of people who care about x86 systems.
> I have a good old Pentium 4 based computer which is 32-bit, using it
> for some weird Linux experiments + retro gaming, and if I can't run
> new Linux kernels there - it would be SAD

Just to clarify: no one is proposing to drop 32-bit hardware support
or normal x86 compatibility support from new kernels.

That being said, you should seriously consider replacing that P4.
Unless you live somewhere with *very* cheap power or you barely use
it, replacing it will pay for itself quite quickly.

Re: Can we drop upstream Linux x32 support?

[Ivan Ivanov]

My main reason on staying at Pentium 4 is that it does not contain
Intel ME backdoor (it came starting with Core 2 Duo)
ср, 12 дек. 2018 г. в 22:12, Andy Lutomirski <luto@kernel.org>:
>
> On Wed, Dec 12, 2018 at 10:50 AM Ivan Ivanov <qmastery16@gmail.com> wrote:
> >
> > Please don't drop x86 support from the Linux kernel ! You may not
> > realize it, but there are plenty of people who care about x86 systems.
> > I have a good old Pentium 4 based computer which is 32-bit, using it
> > for some weird Linux experiments + retro gaming, and if I can't run
> > new Linux kernels there - it would be SAD
>
> Just to clarify: no one is proposing to drop 32-bit hardware support
> or normal x86 compatibility support from new kernels.
>
> That being said, you should seriously consider replacing that P4.
> Unless you live somewhere with *very* cheap power or you barely use
> it, replacing it will pay for itself quite quickly.

Re: Can we drop upstream Linux x32 support?

[Andy Lutomirski]

> On Dec 11, 2018, at 6:33 PM, Thorsten Glaser <tg@mirbsd.de> wrote:
>
> Andy Lutomirski dixit:
>

>
>> IMO the real right solution would be to push the whole problem to
>> userspace: get an ILP32 system working with almost or entirely LP64
>
> Is this a reflex of Linux kernel developers? ;-)
>
> I doubt that userspace is the right place for this, remember
> the recent glibc vs. syscalls debate. It would also need to
> multiply across various libcs.
>
>> How hard would it be to have __attribute__((ilp64)), with an optional
>> warning if any embedded structs are not ilp64?  This plus a wrapper to
>
> You mean LP64. Impossible, because LP64 vs. ILP32 is not the only
> difference between amd64 and x32.

I mean LP64.  And I'm not suggesting that ILP32 is the only difference
between x32 and x86_64, nor am I suggesting that a technique like this
would implement x32 -- I'm suggesting it would implement something
better than x32.

The kernel, as a practical matter, supports two ABIs on 64-bit builds:
LP64 and ILP32.  ILP32 is what the kernel calls "compat".  ("compat"
comes with other baggage -- it generally has a 32-bit signal frame,
syscall arguments are mostly limited to 32 bits, etc.)  Allowing a
user program that runs in 64-bit mode to issue compat syscalls is not
really a big deal.  x86_64 has allowed this forever using int $0x80 --
it's just slow.  Adding a faster mechanism would be straightforward.
As I understand it, the arm64 ilp32 proposal involves using a genuine
ILP32 model for user code, so the syscalls will all (except for signal
handling) go through the compat path.

x32 is not this at all.  The kernel ABI part of x32 isn't ILP32.  It's
IP32, 32-bit size_t, and *64-bit* long.  The core kernel doesn't
really support this.  The only good things I can think of about it are
that (a) it proves that somewhat odd ABIs are possible, at least in
principle, and (b) three users have come out of the woodwork to say
that they use it.

I'm proposing another alternative.  Given that x32 already proves that
the user bitness model doesn't have to match the kernel model (in x32,
user "long" is 32-bit but the kernel ABI "long" is 64-bit), I'm
proposing extending this to just make the kernel ABI be LP64.  So
__kernel_size_t would be 64-bit and pointers in kernel data structures
would be 64-bit.  In other words, most or all of the kernel ABI would
just match x86_64.

As far as I can tell, the only thing that really needs unusual
toolchain features here is that C doesn't have an extra-wide pointer
type.  The kernel headers would need a way to say "this pointer is
still logically a pointer, and user code may assume that it's 32 bits,
but it has 8-byte alignment."

So I guess I'm proposing that people who really like x32 and are
willing to invest effort it in instead invest some effort into getting
something similar to work using the normal x86_64 syscalls.  And I'm
hinting to the ARM folks on the thread that they consider this
approach for arm64.

There's an added benefit to my proposal over x32.  With my proposal,
an ILP32 program could plausibly call into a true 64-bit C library
using a straightforward FFI translation.
>
> bye,
> //mirabilos
> --
> I believe no one can invent an algorithm. One just happens to hit upon it
> when God enlightens him. Or only God invents algorithms, we merely copy them.
> If you don't believe in God, just consider God as Nature if you won't deny
> existence.        -- Coywolf Qi Hunt

Re: Can we drop upstream Linux x32 support?

[Rich Felker]

On Wed, Dec 12, 2018 at 08:39:53AM -0800, Andy Lutomirski wrote:
> > On Dec 11, 2018, at 6:33 PM, Thorsten Glaser <tg@mirbsd.de> wrote:
> >
> > Andy Lutomirski dixit:
> >
> 
> >
> >> IMO the real right solution would be to push the whole problem to
> >> userspace: get an ILP32 system working with almost or entirely LP64
> >
> > Is this a reflex of Linux kernel developers? ;-)
> >
> > I doubt that userspace is the right place for this, remember
> > the recent glibc vs. syscalls debate. It would also need to
> > multiply across various libcs.
> >
> >> How hard would it be to have __attribute__((ilp64)), with an optional
> >> warning if any embedded structs are not ilp64?  This plus a wrapper to
> >
> > You mean LP64. Impossible, because LP64 vs. ILP32 is not the only
> > difference between amd64 and x32.
> 
> I mean LP64.  And I'm not suggesting that ILP32 is the only difference
> between x32 and x86_64, nor am I suggesting that a technique like this
> would implement x32 -- I'm suggesting it would implement something
> better than x32.
> 
> The kernel, as a practical matter, supports two ABIs on 64-bit builds:
> LP64 and ILP32.  ILP32 is what the kernel calls "compat".  ("compat"
> comes with other baggage -- it generally has a 32-bit signal frame,
> syscall arguments are mostly limited to 32 bits, etc.)  Allowing a
> user program that runs in 64-bit mode to issue compat syscalls is not
> really a big deal.  x86_64 has allowed this forever using int $0x80 --
> it's just slow.  Adding a faster mechanism would be straightforward.
> As I understand it, the arm64 ilp32 proposal involves using a genuine
> ILP32 model for user code, so the syscalls will all (except for signal
> handling) go through the compat path.
> 
> x32 is not this at all.  The kernel ABI part of x32 isn't ILP32.  It's
> IP32, 32-bit size_t, and *64-bit* long.  The core kernel doesn't
> really support this.  The only good things I can think of about it are
> that (a) it proves that somewhat odd ABIs are possible, at least in
> principle, and (b) three users have come out of the woodwork to say
> that they use it.
> 
> I'm proposing another alternative.  Given that x32 already proves that
> the user bitness model doesn't have to match the kernel model (in x32,
> user "long" is 32-bit but the kernel ABI "long" is 64-bit), I'm
> proposing extending this to just make the kernel ABI be LP64.  So
> __kernel_size_t would be 64-bit and pointers in kernel data structures
> would be 64-bit.  In other words, most or all of the kernel ABI would
> just match x86_64.
> 
> As far as I can tell, the only thing that really needs unusual
> toolchain features here is that C doesn't have an extra-wide pointer
> type.  The kernel headers would need a way to say "this pointer is
> still logically a pointer, and user code may assume that it's 32 bits,
> but it has 8-byte alignment."

None of this works on the userspace/C side, nor should any attempt be
made to make it work. Types fundamentally cannot have alignments
larger than their size. If you want to make the alignment of some
pointers 8, you have to make their size 8, and then you just have LP64
again if you did it for all pointers.

If on the other hand you tried to make just some pointers "wide
pointers", you'd also be completely breaking the specified API
contracts of standard interfaces. For example in struct iovec's
iov_base, &foo->iov_base is no longer a valid pointer to an object of
type void* that you can pass to interfaces expecting void**. Sloppy
misunderstandings like what you're making now are exactly why x32 is
already broken and buggy (&foo->tv_nsec already has wrong type for
struct timespec foo).

If you wanted to do this and have it work, libc would have to
completely wrap and convert all the types from userspace structures
with the right types matching the specified API contracts to
kernel-interface structures that can be passed to the LP64 syscalls.
This could be done, but it's a big task, and I don't think anyone
wants to do it, especially if it would be yet another new thing that
doesn't exactly match the already-existing x32. The cost-benefit ratio
is just too high.

> There's an added benefit to my proposal over x32.  With my proposal,
> an ILP32 program could plausibly call into a true 64-bit C library
> using a straightforward FFI translation.

Unless it's a thin, "pure" library that doesn't need anything from
libc, or needs sufficiently little that it could be satisfied by some
shims, this would necessarily require having two libcs in the same
process, which is not going to work.

Rich

Re: Can we drop upstream Linux x32 support?

[Andy Lutomirski]

On Wed, Dec 12, 2018 at 8:52 AM Rich Felker <dalias@libc.org> wrote:
>
> On Wed, Dec 12, 2018 at 08:39:53AM -0800, Andy Lutomirski wrote:
> > > On Dec 11, 2018, at 6:33 PM, Thorsten Glaser <tg@mirbsd.de> wrote:
> > >
> > > Andy Lutomirski dixit:
> > >
> >
> > >
> > >> IMO the real right solution would be to push the whole problem to
> > >> userspace: get an ILP32 system working with almost or entirely LP64
> > >
> > > Is this a reflex of Linux kernel developers? ;-)
> > >
> > > I doubt that userspace is the right place for this, remember
> > > the recent glibc vs. syscalls debate. It would also need to
> > > multiply across various libcs.
> > >
> > >> How hard would it be to have __attribute__((ilp64)), with an optional
> > >> warning if any embedded structs are not ilp64?  This plus a wrapper to
> > >
> > > You mean LP64. Impossible, because LP64 vs. ILP32 is not the only
> > > difference between amd64 and x32.
> >
> > I mean LP64.  And I'm not suggesting that ILP32 is the only difference
> > between x32 and x86_64, nor am I suggesting that a technique like this
> > would implement x32 -- I'm suggesting it would implement something
> > better than x32.
> >
> > The kernel, as a practical matter, supports two ABIs on 64-bit builds:
> > LP64 and ILP32.  ILP32 is what the kernel calls "compat".  ("compat"
> > comes with other baggage -- it generally has a 32-bit signal frame,
> > syscall arguments are mostly limited to 32 bits, etc.)  Allowing a
> > user program that runs in 64-bit mode to issue compat syscalls is not
> > really a big deal.  x86_64 has allowed this forever using int $0x80 --
> > it's just slow.  Adding a faster mechanism would be straightforward.
> > As I understand it, the arm64 ilp32 proposal involves using a genuine
> > ILP32 model for user code, so the syscalls will all (except for signal
> > handling) go through the compat path.
> >
> > x32 is not this at all.  The kernel ABI part of x32 isn't ILP32.  It's
> > IP32, 32-bit size_t, and *64-bit* long.  The core kernel doesn't
> > really support this.  The only good things I can think of about it are
> > that (a) it proves that somewhat odd ABIs are possible, at least in
> > principle, and (b) three users have come out of the woodwork to say
> > that they use it.
> >
> > I'm proposing another alternative.  Given that x32 already proves that
> > the user bitness model doesn't have to match the kernel model (in x32,
> > user "long" is 32-bit but the kernel ABI "long" is 64-bit), I'm
> > proposing extending this to just make the kernel ABI be LP64.  So
> > __kernel_size_t would be 64-bit and pointers in kernel data structures
> > would be 64-bit.  In other words, most or all of the kernel ABI would
> > just match x86_64.
> >
> > As far as I can tell, the only thing that really needs unusual
> > toolchain features here is that C doesn't have an extra-wide pointer
> > type.  The kernel headers would need a way to say "this pointer is
> > still logically a pointer, and user code may assume that it's 32 bits,
> > but it has 8-byte alignment."
>
> None of this works on the userspace/C side, nor should any attempt be
> made to make it work. Types fundamentally cannot have alignments
> larger than their size. If you want to make the alignment of some
> pointers 8, you have to make their size 8, and then you just have LP64
> again if you did it for all pointers.
>
> If on the other hand you tried to make just some pointers "wide
> pointers", you'd also be completely breaking the specified API
> contracts of standard interfaces. For example in struct iovec's
> iov_base, &foo->iov_base is no longer a valid pointer to an object of
> type void* that you can pass to interfaces expecting void**. Sloppy
> misunderstandings like what you're making now are exactly why x32 is
> already broken and buggy (&foo->tv_nsec already has wrong type for
> struct timespec foo).

I don't think it's quite that broken.  For the struct iovec example,
we currently have:

           struct iovec {
               void  *iov_base;    /* Starting address */
               size_t iov_len;     /* Number of bytes to transfer */
           };

we could have, instead: (pardon any whitespace damage)

           struct iovec {
               void  *iov_base;    /* Starting address */
               uint32_t __pad0;
               size_t iov_len;     /* Number of bytes to transfer */
               uint32_t __pad1;
           } __attribute__((aligned(8));

or the same thing but where iov_len is uint64_t.  A pointer to
iov_base still works exactly as expected.  Something would need to be
done to ensure that the padding is all zeroed, which might be a real
problem.

No one wants to actually type all the macro gunk into the headers to
make this work, but this type of transformation is what I have in mind
when the compiler is asked to handle the headers.  Or there could
potentially be a tool that automatically consumes the uapi headers and
spits out modified headers like this.

Realistically, I think a much better model would be to use true ILP32
code, where all the memory layouts in the uapi match i386.

> Unless it's a thin, "pure" library that doesn't need anything from
> libc, or needs sufficiently little that it could be satisfied by some
> shims, this would necessarily require having two libcs in the same
> process, which is not going to work.
>
>

That's a good point.

Re: Can we drop upstream Linux x32 support?

[Catalin Marinas]

On Wed, Dec 12, 2018 at 10:03:30AM -0800, Andy Lutomirski wrote:
> On Wed, Dec 12, 2018 at 8:52 AM Rich Felker <dalias@libc.org> wrote:
> > On Wed, Dec 12, 2018 at 08:39:53AM -0800, Andy Lutomirski wrote:
> > > I'm proposing another alternative.  Given that x32 already proves that
> > > the user bitness model doesn't have to match the kernel model (in x32,
> > > user "long" is 32-bit but the kernel ABI "long" is 64-bit), I'm
> > > proposing extending this to just make the kernel ABI be LP64.  So
> > > __kernel_size_t would be 64-bit and pointers in kernel data structures
> > > would be 64-bit.  In other words, most or all of the kernel ABI would
> > > just match x86_64.
> > >
> > > As far as I can tell, the only thing that really needs unusual
> > > toolchain features here is that C doesn't have an extra-wide pointer
> > > type.  The kernel headers would need a way to say "this pointer is
> > > still logically a pointer, and user code may assume that it's 32 bits,
> > > but it has 8-byte alignment."
> >
> > None of this works on the userspace/C side, nor should any attempt be
> > made to make it work. Types fundamentally cannot have alignments
> > larger than their size. If you want to make the alignment of some
> > pointers 8, you have to make their size 8, and then you just have LP64
> > again if you did it for all pointers.
> >
> > If on the other hand you tried to make just some pointers "wide
> > pointers", you'd also be completely breaking the specified API
> > contracts of standard interfaces. For example in struct iovec's
> > iov_base, &foo->iov_base is no longer a valid pointer to an object of
> > type void* that you can pass to interfaces expecting void**. Sloppy
> > misunderstandings like what you're making now are exactly why x32 is
> > already broken and buggy (&foo->tv_nsec already has wrong type for
> > struct timespec foo).
> 
> I don't think it's quite that broken.  For the struct iovec example,
> we currently have:
> 
>            struct iovec {
>                void  *iov_base;    /* Starting address */
>                size_t iov_len;     /* Number of bytes to transfer */
>            };
> 
> we could have, instead: (pardon any whitespace damage)
> 
>            struct iovec {
>                void  *iov_base;    /* Starting address */
>                uint32_t __pad0;
>                size_t iov_len;     /* Number of bytes to transfer */
>                uint32_t __pad1;
>            } __attribute__((aligned(8));
> 
> or the same thing but where iov_len is uint64_t.  A pointer to
> iov_base still works exactly as expected.  Something would need to be
> done to ensure that the padding is all zeroed, which might be a real
> problem.

We looked at this approach briefly for arm64/ILP32 and zeroing the pads
was the biggest problem. User programs would not explicitly zero the pad
and I'm not sure the compiler would be any smarter. This means it's the
kernel's responsibility to zero the pad (around get_user,
copy_from_user), so it doesn't actually simplify the kernel side of the
syscall interface.

If the data flow goes the other way (kernel to user), this approach
works fine.

> No one wants to actually type all the macro gunk into the headers to
> make this work, but this type of transformation is what I have in mind
> when the compiler is asked to handle the headers.  Or there could
> potentially be a tool that automatically consumes the uapi headers and
> spits out modified headers like this.

If the compiler can handle the zeroing, that would be great, though not
sure how (some __attribute__((zero)) which generates a type constructor
for such structure; it kind of departs from what the C language offers).

> Realistically, I think a much better model would be to use true ILP32
> code, where all the memory layouts in the uapi match i386.

The conclusion we came to on arm64 was that an ILP32 ABI should not
really be any different from a _new_ 32-bit architecture ABI. It differs
from arm32 a bit (different syscall numbers, off_t is 64-bit,
sigcontext) but not significantly as it is still able to use the
majority of the compat_sys_* wrappers.

-- 
Catalin

Re: Can we drop upstream Linux x32 support?

[Rich Felker]

On Thu, Dec 13, 2018 at 12:40:25PM +0000, Catalin Marinas wrote:
> On Wed, Dec 12, 2018 at 10:03:30AM -0800, Andy Lutomirski wrote:
> > On Wed, Dec 12, 2018 at 8:52 AM Rich Felker <dalias@libc.org> wrote:
> > > On Wed, Dec 12, 2018 at 08:39:53AM -0800, Andy Lutomirski wrote:
> > > > I'm proposing another alternative.  Given that x32 already proves that
> > > > the user bitness model doesn't have to match the kernel model (in x32,
> > > > user "long" is 32-bit but the kernel ABI "long" is 64-bit), I'm
> > > > proposing extending this to just make the kernel ABI be LP64.  So
> > > > __kernel_size_t would be 64-bit and pointers in kernel data structures
> > > > would be 64-bit.  In other words, most or all of the kernel ABI would
> > > > just match x86_64.
> > > >
> > > > As far as I can tell, the only thing that really needs unusual
> > > > toolchain features here is that C doesn't have an extra-wide pointer
> > > > type.  The kernel headers would need a way to say "this pointer is
> > > > still logically a pointer, and user code may assume that it's 32 bits,
> > > > but it has 8-byte alignment."
> > >
> > > None of this works on the userspace/C side, nor should any attempt be
> > > made to make it work. Types fundamentally cannot have alignments
> > > larger than their size. If you want to make the alignment of some
> > > pointers 8, you have to make their size 8, and then you just have LP64
> > > again if you did it for all pointers.
> > >
> > > If on the other hand you tried to make just some pointers "wide
> > > pointers", you'd also be completely breaking the specified API
> > > contracts of standard interfaces. For example in struct iovec's
> > > iov_base, &foo->iov_base is no longer a valid pointer to an object of
> > > type void* that you can pass to interfaces expecting void**. Sloppy
> > > misunderstandings like what you're making now are exactly why x32 is
> > > already broken and buggy (&foo->tv_nsec already has wrong type for
> > > struct timespec foo).
> > 
> > I don't think it's quite that broken.  For the struct iovec example,
> > we currently have:
> > 
> >            struct iovec {
> >                void  *iov_base;    /* Starting address */
> >                size_t iov_len;     /* Number of bytes to transfer */
> >            };
> > 
> > we could have, instead: (pardon any whitespace damage)
> > 
> >            struct iovec {
> >                void  *iov_base;    /* Starting address */
> >                uint32_t __pad0;
> >                size_t iov_len;     /* Number of bytes to transfer */
> >                uint32_t __pad1;
> >            } __attribute__((aligned(8));
> > 
> > or the same thing but where iov_len is uint64_t.  A pointer to
> > iov_base still works exactly as expected.  Something would need to be
> > done to ensure that the padding is all zeroed, which might be a real
> > problem.
> 
> We looked at this approach briefly for arm64/ILP32 and zeroing the pads
> was the biggest problem. User programs would not explicitly zero the pad
> and I'm not sure the compiler would be any smarter. This means it's the
> kernel's responsibility to zero the pad (around get_user,
> copy_from_user), so it doesn't actually simplify the kernel side of the
> syscall interface.
> 
> If the data flow goes the other way (kernel to user), this approach
> works fine.
> 
> > No one wants to actually type all the macro gunk into the headers to
> > make this work, but this type of transformation is what I have in mind
> > when the compiler is asked to handle the headers.  Or there could
> > potentially be a tool that automatically consumes the uapi headers and
> > spits out modified headers like this.
> 
> If the compiler can handle the zeroing, that would be great, though not
> sure how (some __attribute__((zero)) which generates a type constructor
> for such structure; it kind of departs from what the C language offers).

The compiler fundamentally can't. At the very least it would require
effective type tracking, which requires shadow memory and is even more
controversial than -fstrict-aliasing (because in a sense it's a
stronger version thereof). But even effective type tracking would not
help, since you can have things like:

	struct iovec *iov = malloc(sizeof *iov);
	scanf("%p %zu", &iov->iov_base, &iov->iov_len);

where no store to the object via the struct type ever happens and the
only stores that do happen are invisible across translation unit
boundaries. (Ignore that scanf here is awful; it's just a canonical
example of a function that would store the members via pointers to
them.)

The kernel-side approach could work if the kernel had some markup for
fields that need to be zero- or sign-extended when copied from user in
a 32-bit process and applied them at copy time. That could also fix
the existing tv_nsec issue. I'm not sure how difficult/costly it would
be though.

Rich

Re: Can we drop upstream Linux x32 support?

[Florian Weimer]

* Rich Felker:

>> If the compiler can handle the zeroing, that would be great, though not
>> sure how (some __attribute__((zero)) which generates a type constructor
>> for such structure; it kind of departs from what the C language offers).
>
> The compiler fundamentally can't. At the very least it would require
> effective type tracking, which requires shadow memory and is even more
> controversial than -fstrict-aliasing (because in a sense it's a
> stronger version thereof).

It's possible to do it with the right types.  See _Bool on 32-bit Darwin
PowerPC for an example, which is four bytes instead of the usual one.

Similarly, we could have integer types with trap representations.
Whether it is a good idea is a different matter, but the amount of
compiler magic required is actually limited.

Thanks,
Florian

Re: Can we drop upstream Linux x32 support?

[Rich Felker]

On Thu, Dec 13, 2018 at 05:04:59PM +0100, Florian Weimer wrote:
> * Rich Felker:
> 
> >> If the compiler can handle the zeroing, that would be great, though not
> >> sure how (some __attribute__((zero)) which generates a type constructor
> >> for such structure; it kind of departs from what the C language offers).
> >
> > The compiler fundamentally can't. At the very least it would require
> > effective type tracking, which requires shadow memory and is even more
> > controversial than -fstrict-aliasing (because in a sense it's a
> > stronger version thereof).
> 
> It's possible to do it with the right types.  See _Bool on 32-bit Darwin
> PowerPC for an example, which is four bytes instead of the usual one.
> 
> Similarly, we could have integer types with trap representations.
> Whether it is a good idea is a different matter, but the amount of
> compiler magic required is actually limited.

If you do this you just have LP64 with value range restricted to
32-bit. This does not accomplish anything useful since it uses just as
much memory as normal LP64 but you don't get anything beneficial out
of it.

Rich

Re: Can we drop upstream Linux x32 support?

[Florian Weimer]

* Rich Felker:

> On Thu, Dec 13, 2018 at 05:04:59PM +0100, Florian Weimer wrote:
>> * Rich Felker:
>> 
>> >> If the compiler can handle the zeroing, that would be great, though not
>> >> sure how (some __attribute__((zero)) which generates a type constructor
>> >> for such structure; it kind of departs from what the C language offers).
>> >
>> > The compiler fundamentally can't. At the very least it would require
>> > effective type tracking, which requires shadow memory and is even more
>> > controversial than -fstrict-aliasing (because in a sense it's a
>> > stronger version thereof).
>> 
>> It's possible to do it with the right types.  See _Bool on 32-bit Darwin
>> PowerPC for an example, which is four bytes instead of the usual one.
>> 
>> Similarly, we could have integer types with trap representations.
>> Whether it is a good idea is a different matter, but the amount of
>> compiler magic required is actually limited.
>
> If you do this you just have LP64 with value range restricted to
> 32-bit.

You have to a type different from long int for the relevant struct
fields.  This type would have zero padding.

Florian

Re: Can we drop upstream Linux x32 support?

[Rich Felker]

On Fri, Dec 14, 2018 at 12:42:34PM +0100, Florian Weimer wrote:
> * Rich Felker:
> 
> > On Thu, Dec 13, 2018 at 05:04:59PM +0100, Florian Weimer wrote:
> >> * Rich Felker:
> >> 
> >> >> If the compiler can handle the zeroing, that would be great, though not
> >> >> sure how (some __attribute__((zero)) which generates a type constructor
> >> >> for such structure; it kind of departs from what the C language offers).
> >> >
> >> > The compiler fundamentally can't. At the very least it would require
> >> > effective type tracking, which requires shadow memory and is even more
> >> > controversial than -fstrict-aliasing (because in a sense it's a
> >> > stronger version thereof).
> >> 
> >> It's possible to do it with the right types.  See _Bool on 32-bit Darwin
> >> PowerPC for an example, which is four bytes instead of the usual one.
> >> 
> >> Similarly, we could have integer types with trap representations.
> >> Whether it is a good idea is a different matter, but the amount of
> >> compiler magic required is actually limited.
> >
> > If you do this you just have LP64 with value range restricted to
> > 32-bit.
> 
> You have to a type different from long int for the relevant struct
> fields.  This type would have zero padding.

Just upthread (Message-ID: <20181212165237.GT23599@brightrain.aerifal.cx>)
I explained why this does not work:

>>> If on the other hand you tried to make just some pointers "wide
>>> pointers", you'd also be completely breaking the specified API
>>> contracts of standard interfaces. For example in struct iovec's
>>> iov_base, &foo->iov_base is no longer a valid pointer to an object of
>>> type void* that you can pass to interfaces expecting void**. Sloppy
>>> misunderstandings like what you're making now are exactly why x32 is
>>> already broken and buggy (&foo->tv_nsec already has wrong type for
>>> struct timespec foo).

Rich

Re: Can we drop upstream Linux x32 support?

[Joseph Myers]

On Thu, 13 Dec 2018, Florian Weimer wrote:

> Similarly, we could have integer types with trap representations.

C++2a will require two's complement representation for integer types, with 
no trap representations (other than for bool, where only 0 and 1 are valid 
representations).  It seems very plausible that C will follow.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Can we drop upstream Linux x32 support?

[Thorsten Glaser]

Andy Lutomirski dixit:

>x32 is not this at all.  The kernel ABI part of x32 isn't ILP32.  It's
>IP32, 32-bit size_t, and *64-bit* long.  The core kernel doesn't

Yeah, I was looking at this from userspace PoV, as I said I’m not
a Linux kernel programmer.

In BSD we have register_t which is probably the equivalent to your
__kernel_long_t? Maybe removing the “long” from the name helps.

But yes, x32 is just a (second to i386) ILP32 userspace API in an
amd64 kernel. This does imply mapping on the userspace (x32) to
kernel (amd64) boundary and back. I would have thought full struct
member mapping, as dalias described, to be the most robust.

>something similar to work using the normal x86_64 syscalls.  And I'm

But those would require the longer structs etc. and therefore
lose all the benefits of x32…

bye,
//mirabilos
-- 
„Cool, /usr/share/doc/mksh/examples/uhr.gz ist ja ein Grund,
mksh auf jedem System zu installieren.“
	-- XTaran auf der OpenRheinRuhr, ganz begeistert
(EN: “[…]uhr.gz is a reason to install mksh on every system.”)

Re: Can we drop upstream Linux x32 support?

[Rich Felker]

On Tue, Dec 11, 2018 at 03:22:43PM -0800, Andy Lutomirski wrote:
> On Tue, Dec 11, 2018 at 2:14 PM Thorsten Glaser <tg@mirbsd.de> wrote:
> > I was also considering investing a nontrivial amount of
> > work into porting klibc to x32, since hpa does not wish
> > to do it himself. Thankfully I have only done a bit yet.
> >
> > Furthermore, x32 was the first of the many *64ilp32
> > architectures; I know I’ve seen amd64ilp32 and at least
> > one other I don’t recall. It will have prototyped many
> > of the problems users of these will run in, and I’d prefer
> > to keep it (completely selfish because I don’t wish to
> > have to crossgrade a whole system yet again).
> 
> it kind of seems like arm64's lesson is "don't do it like x32".
> 
> There's some effort going on right now to make it possible to add
> syscalls without having to muck with every single architecture.  I
> don't really want x32 to derail that effort.  I suppose we could say
> that x32 stays but that it simply gets no new syscalls, but that seems
> a bit lame.  Unfortunately, on x86, x32 really is a third ABI that is
> not compatible in a structure-memory-layout sense with the other two.
> What happens if someone adds a struct like:
> 
> struct nasty_on_x32 {
>   __kernel_long_t a;
>   void * __user b;
> };
> 
> On x86_64, that's two 8-byte fields.  On x86_32, it's two four-byte
> fields.  On x32, it's an 8-byte field and a 4-byte field.  Now what?
> 
> I'm sure we could have some magic gcc plugin or other nifty tool that gives us:
> 
> copy_from_user(struct struct_name, kernel_ptr, user_ptr);
> 
> where it automatically generates code for all possible ABIs to copy
> over the struct and dispatches dynamically based on the current
> syscall ABI, but I have trouble imagining anyone volunteering to
> actually do this work.  Instead we get ad hoc fixes for each syscall,
> along the lines of preadv64v2(), which get done when somebody notices
> a problem.

For what it's worth, this is exactly what I plan to do if/when I ever
get to doing my kernel with multiple/universal userspace ABI support:
pure-data markup for the data structures (and syscall argument
arrangements) userspace is speaking, with generic marshalling code.
But I don't think it's easy to retrofit to Linux, and probably
wouldn't be welcome there.

Rich

Re: Can we drop upstream Linux x32 support?

[Maciej W. Rozycki]

On Tue, 11 Dec 2018, Thorsten Glaser wrote:

> Furthermore, x32 was the first of the many *64ilp32
> architectures; I know I’ve seen amd64ilp32 and at least
> one other I don’t recall.

 Maybe you mean n32 MIPS, complete kernel+userland support for which has 
been available in Linux from 2001?

  Maciej

Re: Can we drop upstream Linux x32 support?

[Olof Johansson]

On Tue, Dec 11, 2018 at 9:40 AM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski <luto@kernel.org> wrote:
> >
> > I'm seriously considering sending a patch to remove x32 support from
> > upstream Linux.  Here are some problems with it:
>
> I talked to Arnd (I think - we were talking about all the crazy ABI's,
> but maybe it was with somebody else) about exactly this in Edinburgh.
>
> Apparently the main real use case is for extreme benchmarking. It's
> the only use-case where the complexity of maintaining a whole
> development environment and distro is worth it, it seems. Apparently a
> number of Spec submissions have been done with the x32 model.
>
> I'm not opposed to trying to sunset the support, but let's see who complains..

I'm just a single user. I do rely on it though, FWIW.

I hadn't finished my benchmarking in Edinburgh, but for my new machine
that does kernel builds 24/7, I ended up going with x32 userspace (in
a container).

Main reason is that it's a free ~10% improvement in runtime over
64-bit. I.e. GCC-as-a-workload is quite a bit faster as x32,
supposedly mostly due to smaller D cache footprints (I ran out of
cycles to tinker with back and forth perf data collection and settled
down on just running it).

Running classic 32-bit (i386? i686? whatever it's called) is about
half as good. I.e. even then I get a ~5% performance win. Less than
x32, but still better than 64-bit userspace.


-Olof

Re: Can we drop upstream Linux x32 support?

[Lance Richardson]

On Thu, Dec 13, 2018 at 9:39 AM Olof Johansson <olof@lixom.net> wrote:
>
> On Tue, Dec 11, 2018 at 9:40 AM Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >
> > On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski <luto@kernel.org> wrote:
> > >
> > > I'm seriously considering sending a patch to remove x32 support from
> > > upstream Linux.  Here are some problems with it:
> >
> > I talked to Arnd (I think - we were talking about all the crazy ABI's,
> > but maybe it was with somebody else) about exactly this in Edinburgh.
> >
> > Apparently the main real use case is for extreme benchmarking. It's
> > the only use-case where the complexity of maintaining a whole
> > development environment and distro is worth it, it seems. Apparently a
> > number of Spec submissions have been done with the x32 model.
> >
> > I'm not opposed to trying to sunset the support, but let's see who complains..
>
> I'm just a single user. I do rely on it though, FWIW.
>
> I hadn't finished my benchmarking in Edinburgh, but for my new machine
> that does kernel builds 24/7, I ended up going with x32 userspace (in
> a container).
>
> Main reason is that it's a free ~10% improvement in runtime over
> 64-bit. I.e. GCC-as-a-workload is quite a bit faster as x32,
> supposedly mostly due to smaller D cache footprints (I ran out of
> cycles to tinker with back and forth perf data collection and settled
> down on just running it).
>
> Running classic 32-bit (i386? i686? whatever it's called) is about
> half as good. I.e. even then I get a ~5% performance win. Less than
> x32, but still better than 64-bit userspace.
>
>
> -Olof

I'm familiar with two embedded Linux systems using x32 ABI for the
following reasons:
   - Significant performance benefits over i386 ABI.
   - Smaller memory footprint than x86_64 (pointer-heavy data structures)
   - Large legacy software base with many ILP32 assumptions, much
     smaller effort to move to x32 than x86_64.

Some examples of (relatively minor) problems encountered with x32:
   - Time-related data type mismatch in socket options (fixed)
     https://patchwork.ozlabs.org/patch/904254/

   - Userspace overrun with select() (patch proposed)
     https://patchwork.kernel.org/patch/10245677/

   - glibc get_phys_pages() doesn't work correctly with x32
     (assumes that struct sysinfo fields are not wider than "long").

So, one small vote for keeping x32 with the hope that support for it
can continue to be improved...

   - Lance

Re: Can we drop upstream Linux x32 support?

[Richard Purdie]

On Mon, 2018-12-10 at 17:40 -0800, Linus Torvalds wrote:
> On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski <luto@kernel.org>
> wrote:
> > I'm seriously considering sending a patch to remove x32 support
> > from
> > upstream Linux.  Here are some problems with it:
> 
> I talked to Arnd (I think - we were talking about all the crazy
> ABI's,
> but maybe it was with somebody else) about exactly this in Edinburgh.
> 
> Apparently the main real use case is for extreme benchmarking. It's
> the only use-case where the complexity of maintaining a whole
> development environment and distro is worth it, it seems. Apparently
> a
> number of Spec submissions have been done with the x32 model.
> 
> I'm not opposed to trying to sunset the support, but let's see who
> complains..

FWIW, the Yocto Project has working x32 support which it regularly
builds, tests and has as a supported configuration. We did help
identify and fix many of the places userspace didn't support it.

As with open source in general, its hard to know how many people are
actually using it. I don't think there are huge numbers but we do
occasionally get bug reports.

Cheers,

Richard

Re: Can we drop upstream Linux x32 support?

[H.J. Lu]

On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski <luto@kernel.org> wrote:
>
> Hi all-
>
> I'm seriously considering sending a patch to remove x32 support from
> upstream Linux.  Here are some problems with it:
>
> 1. It's not entirely clear that it has users.  As far as I know, it's
> supported on Gentoo and Debian, and the Debian popcon graph for x32
> has been falling off dramatically.  I don't think that any enterprise
> distro has ever supported x32.

I have been posting x32 GCC results for years:

https://gcc.gnu.org/ml/gcc-testresults/2018-12/msg01358.html

> 2. The way that system calls work is very strange.  Most syscalls on
> x32 enter through their *native* (i.e. not COMPAT_SYSCALL_DEFINE)
> entry point, and this is intentional.  For example, adjtimex() uses
> the native entry, not the compat entry, because x32's struct timex
> matches the x86_64 layout.  But a handful of syscalls have separate

This becomes less an issue with 64-bit time_t.

> entry points -- these are the syscalls starting at 512.  These enter
> throuh the COMPAT_SYSCALL_DEFINE entry points.
>
> The x32 syscalls that are *not* in the 512 range violate all semblance
> of kernel syscall convention.  In the syscall handlers,
> in_compat_syscall() returns true, but the COMPAT_SYSCALL_DEFINE entry
> is not invoked.   This is nutty and risks breaking things when people
> refactor their syscall implementations.  And no one tests these
> things.  Similarly, if someone calls any of the syscalls below 512 but
> sets bit 31 in RAX, then the native entry will be called with
> in_compat_set().
>
> Conversely, if you call a syscall in the 512 range with bit 31
> *clear*, then the compat entry is set with in_compat_syscall()
> *clear*.  This is also nutty.

This is to share syscalls between LP64 and ILP32 (x32) in x86-64 kernel.

> Finally, the kernel has a weird distinction between CONFIG_X86_X32_ABI
> and and CONFIG_X86_X32, which I suspect results in incorrect builds if
> the host doesn't have an x32 toolchain installed.

X86-64 binutils and GCC can be used to build x86-64 kernel with x32 support.

> I propose that we make CONFIG_X86_X32 depend on BROKEN for a release
> or two and then remove all the code if no one complains.  If anyone
> wants to re-add it, IMO they're welcome to do so, but they need to do
> it in a way that is maintainable.


-- 
H.J.

Re: Can we drop upstream Linux x32 support?

[Andy Lutomirski]

On Mon, Dec 10, 2018 at 7:15 PM H.J. Lu <hjl.tools@gmail.com> wrote:
>
> On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski <luto@kernel.org> wrote:
> >
> > Hi all-
> >
> > I'm seriously considering sending a patch to remove x32 support from
> > upstream Linux.  Here are some problems with it:
> >
> > 1. It's not entirely clear that it has users.  As far as I know, it's
> > supported on Gentoo and Debian, and the Debian popcon graph for x32
> > has been falling off dramatically.  I don't think that any enterprise
> > distro has ever supported x32.
>
> I have been posting x32 GCC results for years:
>
> https://gcc.gnu.org/ml/gcc-testresults/2018-12/msg01358.html

Right.  My question wasn't whether x32 had developers -- it was
whether it had users.  If the only users are a small handful of people
who keep the toolchain and working and some people who benchmark it,
then I think the case for keeping it in upstream Linux is a bit weak.

>
> > 2. The way that system calls work is very strange.  Most syscalls on
> > x32 enter through their *native* (i.e. not COMPAT_SYSCALL_DEFINE)
> > entry point, and this is intentional.  For example, adjtimex() uses
> > the native entry, not the compat entry, because x32's struct timex
> > matches the x86_64 layout.  But a handful of syscalls have separate
>
> This becomes less an issue with 64-bit time_t.
>
> > entry points -- these are the syscalls starting at 512.  These enter
> > throuh the COMPAT_SYSCALL_DEFINE entry points.
> >
> > The x32 syscalls that are *not* in the 512 range violate all semblance
> > of kernel syscall convention.  In the syscall handlers,
> > in_compat_syscall() returns true, but the COMPAT_SYSCALL_DEFINE entry
> > is not invoked.   This is nutty and risks breaking things when people
> > refactor their syscall implementations.  And no one tests these
> > things.  Similarly, if someone calls any of the syscalls below 512 but
> > sets bit 31 in RAX, then the native entry will be called with
> > in_compat_set().
> >
> > Conversely, if you call a syscall in the 512 range with bit 31
> > *clear*, then the compat entry is set with in_compat_syscall()
> > *clear*.  This is also nutty.
>
> This is to share syscalls between LP64 and ILP32 (x32) in x86-64 kernel.
>

I tried to understand what's going on.  As far as I can tell, most of
the magic is the fact that __kernel_long_t and __kernel_ulong_t are
64-bit as seen by x32 user code.  This means that a decent number of
uapi structures are the same on x32 and x86_64.  Syscalls that only
use structures like this should route to the x86_64 entry points.  But
the implementation is still highly dubious -- in_compat_syscall() will
be *true* in such system calls, which means that, if someone changes:

SYSCALL_DEFINE1(some_func, struct some_struct __user *, ptr)
{
  /* x32 goes here, but it's entirely non-obvious unless you read the
x86 syscall table */
  native impl;
}

COMPAT_SYSCALL_DEFINE1(some_func, struct compat_some_struct __user *, ptr)
{
  compat impl;
}

to the Obviously Equivalent (tm):

SYSCALL_DEFINE1(some_func, struct some_struct __user *, ptr)
{
  struct some_struct kernel_val;
  if (in_compat_syscall()) {
    get_compat_some_struct(&kernel_val, ptr);
  } else {
    copy_from_user(&kernel_val, ptr, sizeof(struct some_struct));
  }
  do the work;
}

then x32 breaks.

And I don't even know how x32 is supposed to support some hypothetical
syscall like this:

long sys_nasty(struct adjtimex *a, struct iovec *b);

where one argument has x32 and x86_64 matching but the other has x32
and x86_32 matching.

This whole thing seems extremely fragile.

Re: Can we drop upstream Linux x32 support?

[Arnd Bergmann]

On Tue, Dec 11, 2018 at 6:35 AM Andy Lutomirski <luto@kernel.org> wrote:
> On Mon, Dec 10, 2018 at 7:15 PM H.J. Lu <hjl.tools@gmail.com> wrote:
> > On Mon, Dec 10, 2018 at 5:23 PM Andy Lutomirski <luto@kernel.org> wrote:
> Right.  My question wasn't whether x32 had developers -- it was
> whether it had users.  If the only users are a small handful of people
> who keep the toolchain and working and some people who benchmark it,
> then I think the case for keeping it in upstream Linux is a bit weak.

+1

> > > Conversely, if you call a syscall in the 512 range with bit 31
> > > *clear*, then the compat entry is set with in_compat_syscall()
> > > *clear*.  This is also nutty.
> >
> > This is to share syscalls between LP64 and ILP32 (x32) in x86-64 kernel.
> >
>
> I tried to understand what's going on.  As far as I can tell, most of
> the magic is the fact that __kernel_long_t and __kernel_ulong_t are
> 64-bit as seen by x32 user code.  This means that a decent number of
> uapi structures are the same on x32 and x86_64.  Syscalls that only
> use structures like this should route to the x86_64 entry points.  But
> the implementation is still highly dubious -- in_compat_syscall() will
> be *true* in such system calls,

I think the fundamental issue was that the intention had always been
to use only the 64-bit entry points for system calls, but the most
complex one we have -- ioctl() -- has to use the compat entry point
because device drivers define their own data structures using 'long'
and pointer members and they need translation, as well as
matching in_compat_syscall() checks. This in turn breaks down
again whenever a driver defines an ioctl command that takes
a __kernel_long_t or a derived type like timespec as its argument.

> which means that, if someone changes:
>
...
> where one argument has x32 and x86_64 matching but the other has x32
> and x86_32 matching.
>
> This whole thing seems extremely fragile.

It definitely is. We have lots of workarounds specifically for
x32 in device drivers, but in the time_t conversion for y2038
I still found ones that had not been caught earlier, and for each
y2038 conversion that someone did to a driver or syscall, we have
to make sure that it doesn't break x32 in the process.

         Arnd

Re: Can we drop upstream Linux x32 support?

[Catalin Marinas]

On Tue, Dec 11, 2018 at 10:02:45AM +0100, Arnd Bergmann wrote:
> On Tue, Dec 11, 2018 at 6:35 AM Andy Lutomirski <luto@kernel.org> wrote:
> > I tried to understand what's going on.  As far as I can tell, most of
> > the magic is the fact that __kernel_long_t and __kernel_ulong_t are
> > 64-bit as seen by x32 user code.  This means that a decent number of
> > uapi structures are the same on x32 and x86_64.  Syscalls that only
> > use structures like this should route to the x86_64 entry points.  But
> > the implementation is still highly dubious -- in_compat_syscall() will
> > be *true* in such system calls,
> 
> I think the fundamental issue was that the intention had always been
> to use only the 64-bit entry points for system calls, but the most
> complex one we have -- ioctl() -- has to use the compat entry point
> because device drivers define their own data structures using 'long'
> and pointer members and they need translation, as well as
> matching in_compat_syscall() checks. This in turn breaks down
> again whenever a driver defines an ioctl command that takes
> a __kernel_long_t or a derived type like timespec as its argument.

With arm64 ILP32 we tried to avoid the ioctl() problem by having
__kernel_long_t 32-bit, IOW mimicking the arm32 ABI (compat). The
biggest pain point is signals where the state is completely different
from arm32 (more, wider registers) and can't be dealt with by the compat
layer.

Fortunately, we haven't merge it yet as we have the same dilemma about
real users and who's going to regularly test the ABI in the long run. In
the meantime, watching this thread with interest ;).

-- 
Catalin

Re: Can we drop upstream Linux x32 support?

[Florian Weimer]

* Catalin Marinas:

> On Tue, Dec 11, 2018 at 10:02:45AM +0100, Arnd Bergmann wrote:
>> On Tue, Dec 11, 2018 at 6:35 AM Andy Lutomirski <luto@kernel.org> wrote:
>> > I tried to understand what's going on.  As far as I can tell, most of
>> > the magic is the fact that __kernel_long_t and __kernel_ulong_t are
>> > 64-bit as seen by x32 user code.  This means that a decent number of
>> > uapi structures are the same on x32 and x86_64.  Syscalls that only
>> > use structures like this should route to the x86_64 entry points.  But
>> > the implementation is still highly dubious -- in_compat_syscall() will
>> > be *true* in such system calls,
>> 
>> I think the fundamental issue was that the intention had always been
>> to use only the 64-bit entry points for system calls, but the most
>> complex one we have -- ioctl() -- has to use the compat entry point
>> because device drivers define their own data structures using 'long'
>> and pointer members and they need translation, as well as
>> matching in_compat_syscall() checks. This in turn breaks down
>> again whenever a driver defines an ioctl command that takes
>> a __kernel_long_t or a derived type like timespec as its argument.
>
> With arm64 ILP32 we tried to avoid the ioctl() problem by having
> __kernel_long_t 32-bit, IOW mimicking the arm32 ABI (compat). The
> biggest pain point is signals where the state is completely different
> from arm32 (more, wider registers) and can't be dealt with by the compat
> layer.

I would expect to approach this from the opposite direction: use 64-bit
types in places where the 64-bit kernel interface uses 64-bit types.
After all, not everyone who is interested in ILP32 has a companion
32-bit architecture which could serve as a model for the application
ABI.

(If there are conflicts with POSIX, then POSIX needs to be fixed to
support this.)

Thanks,
Florian

Re: Can we drop upstream Linux x32 support?

[Catalin Marinas]

On Tue, Dec 11, 2018 at 12:37:42PM +0100, Florian Weimer wrote:
> * Catalin Marinas:
> > On Tue, Dec 11, 2018 at 10:02:45AM +0100, Arnd Bergmann wrote:
> >> On Tue, Dec 11, 2018 at 6:35 AM Andy Lutomirski <luto@kernel.org> wrote:
> >> > I tried to understand what's going on.  As far as I can tell, most of
> >> > the magic is the fact that __kernel_long_t and __kernel_ulong_t are
> >> > 64-bit as seen by x32 user code.  This means that a decent number of
> >> > uapi structures are the same on x32 and x86_64.  Syscalls that only
> >> > use structures like this should route to the x86_64 entry points.  But
> >> > the implementation is still highly dubious -- in_compat_syscall() will
> >> > be *true* in such system calls,
> >> 
> >> I think the fundamental issue was that the intention had always been
> >> to use only the 64-bit entry points for system calls, but the most
> >> complex one we have -- ioctl() -- has to use the compat entry point
> >> because device drivers define their own data structures using 'long'
> >> and pointer members and they need translation, as well as
> >> matching in_compat_syscall() checks. This in turn breaks down
> >> again whenever a driver defines an ioctl command that takes
> >> a __kernel_long_t or a derived type like timespec as its argument.
> >
> > With arm64 ILP32 we tried to avoid the ioctl() problem by having
> > __kernel_long_t 32-bit, IOW mimicking the arm32 ABI (compat). The
> > biggest pain point is signals where the state is completely different
> > from arm32 (more, wider registers) and can't be dealt with by the compat
> > layer.
> 
> I would expect to approach this from the opposite direction: use 64-bit
> types in places where the 64-bit kernel interface uses 64-bit types.
> After all, not everyone who is interested in ILP32 has a companion
> 32-bit architecture which could serve as a model for the application
> ABI.

I fully agree with you that if someone wants ILP32 for a 64-bit only
architecture, they should use the 64-bit kernel interface and ensure
POSIX is adjusted.

In the arm64 context, both options were discussed with the libc
community complaining that a partial 64-bit syscall ABI breaks POSIX
while the potential users were just asking for a 32-bit ABI to run their
existing software stack on ARMv8 machines without native 32-bit support
(until they complete the migration to 64-bit).

> (If there are conflicts with POSIX, then POSIX needs to be fixed to
> support this.)

This would have been nice but no-one volunteered and, more importantly,
there was no conclusive argument that ARM ILP32 is better than LP64
(well, apart from a minority of benchmarks) and something that people
would want to migrate to. Given that the only credible case made was
about legacy code, we decided to go ahead with a (mostly) compat 32-bit
ABI.

-- 
Catalin

Re: Can we drop upstream Linux x32 support?

[Christian Brauner]

On Mon, Dec 10, 2018 at 05:23:39PM -0800, Andy Lutomirski wrote:
> Hi all-
> 
> I'm seriously considering sending a patch to remove x32 support from
> upstream Linux.  Here are some problems with it:
> 
> 1. It's not entirely clear that it has users.  As far as I know, it's
> supported on Gentoo and Debian, and the Debian popcon graph for x32
> has been falling off dramatically.  I don't think that any enterprise
> distro has ever supported x32.
> 
> 2. The way that system calls work is very strange.  Most syscalls on
> x32 enter through their *native* (i.e. not COMPAT_SYSCALL_DEFINE)
> entry point, and this is intentional.  For example, adjtimex() uses
> the native entry, not the compat entry, because x32's struct timex
> matches the x86_64 layout.  But a handful of syscalls have separate
> entry points -- these are the syscalls starting at 512.  These enter
> through the COMPAT_SYSCALL_DEFINE entry points.
> 
> The x32 syscalls that are *not* in the 512 range violate all semblance
> of kernel syscall convention.  In the syscall handlers,
> in_compat_syscall() returns true, but the COMPAT_SYSCALL_DEFINE entry
> is not invoked.   This is nutty and risks breaking things when people
> refactor their syscall implementations.  And no one tests these
> things.  Similarly, if someone calls any of the syscalls below 512 but
> sets bit 31 in RAX, then the native entry will be called with
> in_compat_set().
> 
> Conversely, if you call a syscall in the 512 range with bit 31
> *clear*, then the compat entry is set with in_compat_syscall()
> *clear*.  This is also nutty.
> 
> Finally, the kernel has a weird distinction between CONFIG_X86_X32_ABI
> and and CONFIG_X86_X32, which I suspect results in incorrect builds if
> the host doesn't have an x32 toolchain installed.
> 
> I propose that we make CONFIG_X86_X32 depend on BROKEN for a release
> or two and then remove all the code if no one complains.  If anyone

Based on the discussion we had at the beginning of the pidfd_send_signal
syscall patchset I think this is a good idea. For once, the complex
compat handling can make adding new syscalls that need to rely on compat
types because of precedent established by older syscalls icky.

> wants to re-add it, IMO they're welcome to do so, but they need to do
> it in a way that is maintainable.
> 
> --Andy

Re: Can we drop upstream Linux x32 support?

[John Paul Adrian Glaubitz]

Hello!

I'm Debian's principal maintainer of the x32 port.

On 12/11/18 2:23 AM, Andy Lutomirski wrote:
> 1. It's not entirely clear that it has users.  As far as I know, it's
> supported on Gentoo and Debian, and the Debian popcon graph for x32
> has been falling off dramatically.  I don't think that any enterprise
> distro has ever supported x32.

There are definitely some users of this port. I don't know the actual
number, but I hear from users from time to time. As for the popcon
curve, I wouldn't say it has dropped dramatically as it was never
high in the first place.

> https://popcon.debian.org/stat/sub-x32.png

It seems that the highest number of recorded users was 18 and it's
now down to 7. Keep in mind though that popcon participation is
opt-in, so the actual number of users should be higher. According
to popcon, there are also only 172331 Debian installations on x86_64:

> https://popcon.debian.org/

As for the enterprise support, this seems to be correct. I don't know
of any enterprise distribution with x32 support either.

> 2. The way that system calls work is very strange.  Most syscalls on
> x32 enter through their *native* (i.e. not COMPAT_SYSCALL_DEFINE)
> entry point, and this is intentional.  For example, adjtimex() uses
> the native entry, not the compat entry, because x32's struct timex
> matches the x86_64 layout.  But a handful of syscalls have separate
> entry points -- these are the syscalls starting at 512.  These enter
> through the COMPAT_SYSCALL_DEFINE entry points.
> 
> The x32 syscalls that are *not* in the 512 range violate all semblance
> of kernel syscall convention.  In the syscall handlers,
> in_compat_syscall() returns true, but the COMPAT_SYSCALL_DEFINE entry
> is not invoked.   This is nutty and risks breaking things when people
> refactor their syscall implementations.  And no one tests these
> things.  Similarly, if someone calls any of the syscalls below 512 but
> sets bit 31 in RAX, then the native entry will be called with
> in_compat_set().
> 
> Conversely, if you call a syscall in the 512 range with bit 31
> *clear*, then the compat entry is set with in_compat_syscall()
> *clear*.  This is also nutty.

I can't say anything about the syscall interface. However, what I do know
is that the weird combination of a 32-bit userland with a 64-bit kernel
interface is sometimes causing issues. For example, application code usually
expects things like time_t to be 32-bit on a 32-bit system. However, this
isn't the case for x32 which is why code fails to build.

Additionally, x32 support in many applications is either rudimentary
or broken. For example, while LLVM has support for x32, the backend
isn't really stable on this target meaning that compilers like clang
or Rust are partially broken or crash. I'm not sure whether anyone
is interested in fixing this.

It's also that the performance benefits of x32 are often eaten up by
the fact that none of the scripted languages that I know of provide
a JIT that supports x32. Thus, things like Javascript are either
unsupported or slow on x32.

> I propose that we make CONFIG_X86_X32 depend on BROKEN for a release
> or two and then remove all the code if no one complains.  If anyone
> wants to re-add it, IMO they're welcome to do so, but they need to do
> it in a way that is maintainable.

I'm not terribly opposed to this change. I'm usually for keeping support
for things that people are using, but the maintenance is a huge burden
to upstream projects, I'm fine with letting it go.

There are other architectures in the kernel like Alpha, HPPA, M68K, PowerPC,
SH and SPARC that I care much more about than x32.

If x32 is eventually to be removed, we should also take care of removing
x32 support from userland code. From the top of my head, this would at least
concern:

 * OpenJDK
 * LLVM
 * gcc
 * glibc
 * Rust
 * binutils

I can take care of these once I know about the decision regarding the kernel.
Usually, it's a matter of grepping the commit history for "x32" and revert
the corresponding commits.

Thanks,
Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Can we drop upstream Linux x32 support?

[Florian Weimer]

* John Paul Adrian Glaubitz:

> As for the enterprise support, this seems to be correct. I don't know
> of any enterprise distribution with x32 support either.

Me neither.  I would expect a pure userspace port, with limitations in
what ioctls you can use, and perhaps support from GCC to share more APIs
with the kernel.

(I do not know of any plans to work on this.)

> I'm not terribly opposed to this change. I'm usually for keeping support
> for things that people are using, but the maintenance is a huge burden
> to upstream projects, I'm fine with letting it go.

Thank you for sharing your perspective.

> If x32 is eventually to be removed, we should also take care of removing
> x32 support from userland code. From the top of my head, this would at least
> concern:
>
>  * OpenJDK

Note that OpenJDK (well, Hotspot) has its own 32-bit pointer support for
the Java heap (compressed oops), so only the native code parts (and JNI)
benefit from x32 anyway.

Thanks,
Florian

Re: Can we drop upstream Linux x32 support?

[John Paul Adrian Glaubitz]

On 12/11/18 11:37 AM, Florian Weimer wrote:
> Note that OpenJDK (well, Hotspot) has its own 32-bit pointer support for
> the Java heap (compressed oops), so only the native code parts (and JNI)
> benefit from x32 anyway.

Yeah, I was actually only talking about this single commit by me to add
Zero build support for x86_64-linux-gnux32:

> http://hg.openjdk.java.net/jdk/jdk/rev/9ce4a0d718c7

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Can we drop upstream Linux x32 support?

[Thorsten Glaser]

John Paul Adrian Glaubitz dixit:

>I can't say anything about the syscall interface. However, what I do know
>is that the weird combination of a 32-bit userland with a 64-bit kernel
>interface is sometimes causing issues. For example, application code usually

Yes, but more and more ${foo}64ilp32 architectures are popping up.

>Additionally, x32 support in many applications is either rudimentary

If a signal is sent that this kind of architectures will stay, some
people might be convinced to fix that.

>It's also that the performance benefits of x32 are often eaten up by
>the fact that none of the scripted languages that I know of provide

Non-JITted languages like yours truly’s shell do benefit from it,
though. (mksh works just fine on LP64 but its internal structures
pack massively better on ILP32, for example.)

>If x32 is eventually to be removed, we should also take care of removing
>x32 support from userland code. From the top of my head, this would at least

I don’t think so. The patches also contain
– stuff to support 64-bit time_t on 32-bit architectures, e.g:
- bugfixes like printf("%lld", (long long)timet_value) instead
  of assuming time_t fits into a long (also important for other
  operating systems…)
- generally switching from generic types like long to specific
  types like size_t, ptrdiff_t, etc.
- there was one more but after having written two eMails I forgot it
- oh and, of course, they lay the base for e.g. amd64ilp32 support

bye,
//mirabilos
-- 
FWIW, I'm quite impressed with mksh interactively. I thought it was much
*much* more bare bones. But it turns out it beats the living hell out of
ksh93 in that respect. I'd even consider it for my daily use if I hadn't
wasted half my life on my zsh setup. :-) -- Frank Terbeck in #!/bin/mksh

Re: Can we drop upstream Linux x32 support?

[Rich Felker]

On Tue, Dec 11, 2018 at 09:59:48PM +0000, Thorsten Glaser wrote:
> John Paul Adrian Glaubitz dixit:
> 
> >I can't say anything about the syscall interface. However, what I do know
> >is that the weird combination of a 32-bit userland with a 64-bit kernel
> >interface is sometimes causing issues. For example, application code usually
> 
> Yes, but more and more ${foo}64ilp32 architectures are popping up.
> 
> >Additionally, x32 support in many applications is either rudimentary
> 
> If a signal is sent that this kind of architectures will stay, some
> people might be convinced to fix that.
> 
> >It's also that the performance benefits of x32 are often eaten up by
> >the fact that none of the scripted languages that I know of provide
> 
> Non-JITted languages like yours truly’s shell do benefit from it,
> though. (mksh works just fine on LP64 but its internal structures
> pack massively better on ILP32, for example.)

gcc also benefits massively, up to 2x or more performance improvement
from ILP32, just because you can double your -jN to make. The "or
more" comes from cache utilization also improving. Actually I often
use i486 gcc even on 64-bit systems for this reason, and have observed
this order of improvement despite the lack of registers and worse call
ABI. Presumably x32 would be better, but I've avoided it because of
how buggy it's been in the past.

> >If x32 is eventually to be removed, we should also take care of removing
> >x32 support from userland code. From the top of my head, this would at least
> 
> I don’t think so. The patches also contain
> – stuff to support 64-bit time_t on 32-bit architectures, e.g:
> - bugfixes like printf("%lld", (long long)timet_value) instead
>   of assuming time_t fits into a long (also important for other
>   operating systems…)
> - generally switching from generic types like long to specific
>   types like size_t, ptrdiff_t, etc.
> - there was one more but after having written two eMails I forgot it
> - oh and, of course, they lay the base for e.g. amd64ilp32 support

I would not classify any of that as part of "removing x32 support",
but rather as "reintroducing broken assumptions". I think "removing
x32 support" was intended to mean places where there's
explicitly-x32-specific code or tooling, like asm functions/fragments
in libc, x32 versions of the dispatch asm in mesa, x32 code
generation, etc. Obviously (at least I hope it's obvious) nonportable
assumptions should not be reintroduced regardless of whether x32 is
dropped.

FWIW x32 is nice as a test case for catching such nonportable
assumptions and getting them fixed.

Rich

Re: Can we drop upstream Linux x32 support?

[Kevin Easton]

On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote:
...
> I can't say anything about the syscall interface. However, what I do know
> is that the weird combination of a 32-bit userland with a 64-bit kernel
> interface is sometimes causing issues. For example, application code usually
> expects things like time_t to be 32-bit on a 32-bit system. However, this
> isn't the case for x32 which is why code fails to build.

OpenBSD and NetBSD both have 64-bit time_t on 32-bit systems and have
had for four or five years at this point.

    - Kevin

Re: Can we drop upstream Linux x32 support?

[Richard Weinberger]

On Thu, Dec 13, 2018 at 6:03 AM Kevin Easton <kevin@guarana.org> wrote:
>
> On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote:
> ...
> > I can't say anything about the syscall interface. However, what I do know
> > is that the weird combination of a 32-bit userland with a 64-bit kernel
> > interface is sometimes causing issues. For example, application code usually
> > expects things like time_t to be 32-bit on a 32-bit system. However, this
> > isn't the case for x32 which is why code fails to build.
>
> OpenBSD and NetBSD both have 64-bit time_t on 32-bit systems and have
> had for four or five years at this point.

They can also do flag-day changes and break existing applications, Linux not.

-- 
Thanks,
//richard

Re: Can we drop upstream Linux x32 support?

[Kevin Easton]

On Thu, Dec 13, 2018 at 10:05:14AM +0100, Richard Weinberger wrote:
> On Thu, Dec 13, 2018 at 6:03 AM Kevin Easton <kevin@guarana.org> wrote:
> >
> > On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote:
> > ...
> > > I can't say anything about the syscall interface. However, what I do know
> > > is that the weird combination of a 32-bit userland with a 64-bit kernel
> > > interface is sometimes causing issues. For example, application code usually
> > > expects things like time_t to be 32-bit on a 32-bit system. However, this
> > > isn't the case for x32 which is why code fails to build.
> >
> > OpenBSD and NetBSD both have 64-bit time_t on 32-bit systems and have
> > had for four or five years at this point.
> 
> They can also do flag-day changes and break existing applications, Linux not.

Sure, but the point is that most widely-used software has probably by
now come in to contact with systems where time_t is bigger than long.

    - Kevin

RE: Can we drop upstream Linux x32 support?

[David Laight]

From: Richard Weinberger
> Sent: 13 December 2018 09:05
> On Thu, Dec 13, 2018 at 6:03 AM Kevin Easton <kevin@guarana.org> wrote:
> >
> > On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote:
> > ...
> > > I can't say anything about the syscall interface. However, what I do know
> > > is that the weird combination of a 32-bit userland with a 64-bit kernel
> > > interface is sometimes causing issues. For example, application code usually
> > > expects things like time_t to be 32-bit on a 32-bit system. However, this
> > > isn't the case for x32 which is why code fails to build.
> >
> > OpenBSD and NetBSD both have 64-bit time_t on 32-bit systems and have
> > had for four or five years at this point.
> 
> They can also do flag-day changes and break existing applications, Linux not.

Not true at all.
The binary compatibility in NetBSD is probably better than that in Linux
and goes back a long way.

For the time_t changes new system calls numbers were assigned where needed.
The system headers and libc were updated so that recompiled code would
use the new system calls.

The only real advantage that NetBSD has is that its libc (and standard
utilities) are released with the kernel making it much easier to get
applications to use the new features.

This was also done a very long time ago when file offsets were extended
to 64 bits.

Some of the system calls have quite a few 'compatibility' versions.
As well as the ones for emulations of other operating systems.
It has been possible to run copies of firefox compiled for Linux
under NetBSD.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: Can we drop upstream Linux x32 support?

[Richard Weinberger]

Am Freitag, 14. Dezember 2018, 15:38:53 CET schrieb David Laight:
> From: Richard Weinberger
> > Sent: 13 December 2018 09:05
> > On Thu, Dec 13, 2018 at 6:03 AM Kevin Easton <kevin@guarana.org> wrote:
> > >
> > > On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote:
> > > ...
> > > > I can't say anything about the syscall interface. However, what I do know
> > > > is that the weird combination of a 32-bit userland with a 64-bit kernel
> > > > interface is sometimes causing issues. For example, application code usually
> > > > expects things like time_t to be 32-bit on a 32-bit system. However, this
> > > > isn't the case for x32 which is why code fails to build.
> > >
> > > OpenBSD and NetBSD both have 64-bit time_t on 32-bit systems and have
> > > had for four or five years at this point.
> > 
> > They can also do flag-day changes and break existing applications, Linux not.
> 
> Not true at all.
> The binary compatibility in NetBSD is probably better than that in Linux
> and goes back a long way.
> 
> For the time_t changes new system calls numbers were assigned where needed.
> The system headers and libc were updated so that recompiled code would
> use the new system calls.
> 
> The only real advantage that NetBSD has is that its libc (and standard
> utilities) are released with the kernel making it much easier to get
> applications to use the new features.
> 
> This was also done a very long time ago when file offsets were extended
> to 64 bits.
> 
> Some of the system calls have quite a few 'compatibility' versions.
> As well as the ones for emulations of other operating systems.
> It has been possible to run copies of firefox compiled for Linux
> under NetBSD.

I stand corrected, I was under the impression that NetBSD went the same
path as OpenBSD did. Thanks for pointing this out.

Thanks,
//richard

Re: Can we drop upstream Linux x32 support?

[Rich Felker]

On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote:
> I can't say anything about the syscall interface. However, what I do know
> is that the weird combination of a 32-bit userland with a 64-bit kernel
> interface is sometimes causing issues. For example, application code usually
> expects things like time_t to be 32-bit on a 32-bit system. However, this
> isn't the case for x32 which is why code fails to build.

I don't see any basis for this claim about expecting time_t to be
32-bit. I've encountered some programs that "implicitly assume" this
by virtue of assuming they can cast time_t to long to print it, or
similar. IIRC this was an issue in busybox at one point; I'm not sure
if it's been fixed. But any software that runs on non-Linux unices has
long been corrected. If not, 2038 is sufficiently close that catching
and correcting any such remaining bugs is more useful than covering
them up and making the broken code work as expected.

Rich

Re: Can we drop upstream Linux x32 support?

[Bernd Petrovitsch]

[-- Attachment #1: Type: text/plain, Size: 2564 bytes --]

On 13/12/2018 17:02, Rich Felker wrote:
> On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote:
>> I can't say anything about the syscall interface. However, what I do know
>> is that the weird combination of a 32-bit userland with a 64-bit kernel
>> interface is sometimes causing issues. For example, application code usually
>> expects things like time_t to be 32-bit on a 32-bit system. However, this

IMHO this just historically grown (as in "it has been forever that way"
- it sounds way better in Viennese dialect though;-).

>> isn't the case for x32 which is why code fails to build.
> 
> I don't see any basis for this claim about expecting time_t to be
> 32-bit. I've encountered some programs that "implicitly assume" this
> by virtue of assuming they can cast time_t to long to print it, or
> similar. IIRC this was an issue in busybox at one point; I'm not sure
> if it's been fixed. But any software that runs on non-Linux unices has
> long been corrected. If not, 2038 is sufficiently close that catching
> and correcting any such remaining bugs is more useful than covering
> them up and making the broken code work as expected.

Yup, unconditionally providing 64bit
time_t/timespec/timeval/...-equivalents with libc and syscall support
also for 32bit architectures (and deprecating all 32bit versions) should
be the way to go.

FWIW I have
----  snip  ----
#if defined __x86_64__
# if defined __ILP32__		// x32
#  define PRI_time_t	"lld"	// for time_t
#  define PRI_nsec_t	"lld"	// for tv_nsec in struct timespec
# else				// x86_64
#  define PRI_time_t	"ld"	// for time_t
#  define PRI_nsec_t	"ld"	// for tv_nsec in struct timespec
# endif
#else				// i[3-6]68
# define PRI_time_t	"ld"	// for time_t
# define PRI_nsec_t	"ld"	// for tv_nsec in struct timespec
#endif
----  snip  ----
in my userspace code for printf() and friends - I don't know how libc's
react to such a patch (and I don't care for the name of the macros as
long it's obviously clear for which type they are).
I assume/fear we won't get additional modifiers into the relevant
standards for libc types (as they are far more like pid_t, uid_t etc.).
And casting to u/intmaxptr_t to get a defined printf()-modifier doesn't
look appealing to me to "solve" such issues.

MfG,
	Bernd
-- 
"I dislike type abstraction if it has no real reason. And saving
on typing is not a good reason - if your typing speed is the main
issue when you're coding, you're doing something seriously wrong."
    - Linus Torvalds

[-- Attachment #2: pEpkey.asc --]
[-- Type: application/pgp-keys, Size: 2513 bytes --]

Re: Can we drop upstream Linux x32 support?

[Rich Felker]

On Fri, Dec 14, 2018 at 03:13:10PM +0100, Bernd Petrovitsch wrote:
> On 13/12/2018 17:02, Rich Felker wrote:
> > On Tue, Dec 11, 2018 at 11:29:14AM +0100, John Paul Adrian Glaubitz wrote:
> >> I can't say anything about the syscall interface. However, what I do know
> >> is that the weird combination of a 32-bit userland with a 64-bit kernel
> >> interface is sometimes causing issues. For example, application code usually
> >> expects things like time_t to be 32-bit on a 32-bit system. However, this
> 
> IMHO this just historically grown (as in "it has been forever that way"
> - it sounds way better in Viennese dialect though;-).
> 
> >> isn't the case for x32 which is why code fails to build.
> > 
> > I don't see any basis for this claim about expecting time_t to be
> > 32-bit. I've encountered some programs that "implicitly assume" this
> > by virtue of assuming they can cast time_t to long to print it, or
> > similar. IIRC this was an issue in busybox at one point; I'm not sure
> > if it's been fixed. But any software that runs on non-Linux unices has
> > long been corrected. If not, 2038 is sufficiently close that catching
> > and correcting any such remaining bugs is more useful than covering
> > them up and making the broken code work as expected.
> 
> Yup, unconditionally providing 64bit
> time_t/timespec/timeval/...-equivalents with libc and syscall support
> also for 32bit architectures (and deprecating all 32bit versions) should
> be the way to go.
> 
> FWIW I have
> ----  snip  ----
> #if defined __x86_64__
> # if defined __ILP32__		// x32
> #  define PRI_time_t	"lld"	// for time_t
> #  define PRI_nsec_t	"lld"	// for tv_nsec in struct timespec
> # else				// x86_64
> #  define PRI_time_t	"ld"	// for time_t
> #  define PRI_nsec_t	"ld"	// for tv_nsec in struct timespec
> # endif
> #else				// i[3-6]68
> # define PRI_time_t	"ld"	// for time_t
> # define PRI_nsec_t	"ld"	// for tv_nsec in struct timespec
> #endif
> ----  snip  ----
> in my userspace code for printf() and friends - I don't know how libc's
> react to such a patch (and I don't care for the name of the macros as
> long it's obviously clear for which type they are).
> I assume/fear we won't get additional modifiers into the relevant
> standards for libc types (as they are far more like pid_t, uid_t etc.).
> And casting to u/intmaxptr_t to get a defined printf()-modifier doesn't
> look appealing to me to "solve" such issues.

This is all useless (and wrong since tv_nsec is required to have type
long as part of C and POSIX, regardless of ILP32-vs-LP64; that's a bug
in glibc's x32). Just do:

	printf("%jd", (intmax_t)t);

Saving 2 or 3 insns (for sign or zero extension) around a call to
printf is not going to make any measurable difference to performance
or any significant difference to size, and it's immeasurably more
readable than the awful PRI* macros and the
adjacent-string-concatenation they rely on.

Rich

Re: Can we drop upstream Linux x32 support?

[Bernd Petrovitsch]

[-- Attachment #1: Type: text/plain, Size: 2391 bytes --]

On 14/12/2018 17:17, Rich Felker wrote:
> On Fri, Dec 14, 2018 at 03:13:10PM +0100, Bernd Petrovitsch wrote:
[..]
>> FWIW I have
>> ----  snip  ----
>> #if defined __x86_64__
>> # if defined __ILP32__		// x32
>> #  define PRI_time_t	"lld"	// for time_t
>> #  define PRI_nsec_t	"lld"	// for tv_nsec in struct timespec
>> # else				// x86_64
>> #  define PRI_time_t	"ld"	// for time_t
>> #  define PRI_nsec_t	"ld"	// for tv_nsec in struct timespec
>> # endif
>> #else				// i[3-6]68
>> # define PRI_time_t	"ld"	// for time_t
>> # define PRI_nsec_t	"ld"	// for tv_nsec in struct timespec
>> #endif
>> ----  snip  ----
>> in my userspace code for printf() and friends - I don't know how libc's
>> react to such a patch (and I don't care for the name of the macros as
>> long it's obviously clear for which type they are).
>> I assume/fear we won't get additional modifiers into the relevant
>> standards for libc types (as they are far more like pid_t, uid_t etc.).
>> And casting to u/intmaxptr_t to get a defined printf()-modifier doesn't
>> look appealing to me to "solve" such issues.
> 
> This is all useless (and wrong since tv_nsec is required to have type
> long as part of C and POSIX, regardless of ILP32-vs-LP64; that's a bug

Thanks. OK, I didn't know that - and 32bit is enough to represent 1E9
(as a second won't have more nanosecs).
Hmm, can we fix that in the x32 world?
Sry, I'm not the expert on glibc vs ABI va syscall interface vs breakage
there though.

> in glibc's x32). Just do:
> 
> 	printf("%jd", (intmax_t)t);
> 
> Saving 2 or 3 insns (for sign or zero extension) around a call to
> printf is not going to make any measurable difference to performance

Until someone comes up with hardware with ASIC support for 1k bit int's
and (ab)uses intmax_t for that. SCNR ....

> or any significant difference to size, and it's immeasurably more
> readable than the awful PRI* macros and the
> adjacent-string-concatenation they rely on.

One gets used to the PRI_* macros over time (and there no calculated
format strings in my world) - and type casts are not better in my eyes ...

MfG,
	Bernd
-- 
"I dislike type abstraction if it has no real reason. And saving
on typing is not a good reason - if your typing speed is the main
issue when you're coding, you're doing something seriously wrong."
    - Linus Torvalds

[-- Attachment #2: pEpkey.asc --]
[-- Type: application/pgp-keys, Size: 2513 bytes --]

Re: Can we drop upstream Linux x32 support?

[Florian Weimer]

* Rich Felker:

> This is all useless (and wrong since tv_nsec is required to have type
> long as part of C and POSIX, regardless of ILP32-vs-LP64; that's a bug
> in glibc's x32).

We should be able to fix standards if they prove unworkable in practice.
In my opinion, if standards require complex solutions where an obvious
and simple solution exists, then standards are wrong.

Thanks,
Florian

Re: Can we drop upstream Linux x32 support?

[Rich Felker]

On Fri, Dec 14, 2018 at 05:38:33PM +0100, Florian Weimer wrote:
> * Rich Felker:
> 
> > This is all useless (and wrong since tv_nsec is required to have type
> > long as part of C and POSIX, regardless of ILP32-vs-LP64; that's a bug
> > in glibc's x32).
> 
> We should be able to fix standards if they prove unworkable in practice.
> In my opinion, if standards require complex solutions where an obvious
> and simple solution exists, then standards are wrong.

The requirement doesn't mandate complex solutions. There's nothing
complex about tv_nsec being long. long is the smallest type that C
guarantees to be large enough to store the range of values, which is
forever fixed and can't grow (because the definition of "nano" prefix
is fixed :). The type has been long ever since the structure was
introduced, and its being long means that there's lots of (correct!)
code using %ld (e.g. ".%.9ld" to format results as a decimal without
using floating point approximations) to print it. There might also be
code taking pointers to it to pass to functions, etc.

The only reason a "complex" need arises is that Linux did something
horribly wrong here, ignoring the specified type, when introducing an
obscure subarch that almost nobody uses. This kind of mistake is
becoming a theme in Linux (see also: msghdr). Application authors
should not have to pay the price for fixing this by retrofitting yet
another silly type like "snseconds_t" or something into programs to
accommodate the mistakes of x32.

Rich

Re: Can we drop upstream Linux x32 support?

[Andy Lutomirski]

> On Dec 14, 2018, at 8:55 AM, Rich Felker <dalias@libc.org> wrote:
> 
>> On Fri, Dec 14, 2018 at 05:38:33PM +0100, Florian Weimer wrote:
>> * Rich Felker:
>> 
>>> This is all useless (and wrong since tv_nsec is required to have type
>>> long as part of C and POSIX, regardless of ILP32-vs-LP64; that's a bug
>>> in glibc's x32).
>> 
>> We should be able to fix standards if they prove unworkable in practice.
>> In my opinion, if standards require complex solutions where an obvious
>> and simple solution exists, then standards are wrong.
> 
> The requirement doesn't mandate complex solutions. There's nothing
> complex about tv_nsec being long. long is the smallest type that C
> guarantees to be large enough to store the range of values, which is
> forever fixed and can't grow (because the definition of "nano" prefix
> is fixed :). The type has been long ever since the structure was
> introduced, and its being long means that there's lots of (correct!)
> code using %ld (e.g. ".%.9ld" to format results as a decimal without
> using floating point approximations) to print it. There might also be
> code taking pointers to it to pass to functions, etc.
> 
> The only reason a "complex" need arises is that Linux did something
> horribly wrong here, ignoring the specified type, when introducing an
> obscure subarch that almost nobody uses. This kind of mistake is
> becoming a theme in Linux (see also: msghdr). Application authors
> should not have to pay the price for fixing this by retrofitting yet
> another silly type like "snseconds_t" or something into programs to
> accommodate the mistakes of x32.
> 
> 

Does anyone know *why* Linux’s x32 has __kernel_long_t defined as long long?  I assume that this is where this bug, and most of the other bugs, came from.

This may be silly, but the kernel could plausibly add a x32v2 where long is genuinely 32-bit, and then maybe we could drop the old x32 at some point.  From all the discussion so far, it seems like there really is some demand for ILP32, but it’s still not really clear that preserving compatibility with existing x32 binaries in the long run is critical.

Re: Can we drop upstream Linux x32 support?

[Lance Richardson]

My impression is it was mostly a desire to reuse existing x86_64 system calls
as much as possible without modification or additional compat layer work.

The 64-bit time requirement seems to have come from an lkml discussion, which
has quite a bit of interesting background about x32:
    https://lkml.org/lkml/2011/8/26/415
    https://lkml.org/lkml/2011/8/26/453


On Fri, Dec 14, 2018 at 2:05 PM Andy Lutomirski <luto@amacapital.net> wrote:
>
>
>
> > On Dec 14, 2018, at 8:55 AM, Rich Felker <dalias@libc.org> wrote:
> >
> >> On Fri, Dec 14, 2018 at 05:38:33PM +0100, Florian Weimer wrote:
> >> * Rich Felker:
> >>
> >>> This is all useless (and wrong since tv_nsec is required to have type
> >>> long as part of C and POSIX, regardless of ILP32-vs-LP64; that's a bug
> >>> in glibc's x32).
> >>
> >> We should be able to fix standards if they prove unworkable in practice.
> >> In my opinion, if standards require complex solutions where an obvious
> >> and simple solution exists, then standards are wrong.
> >
> > The requirement doesn't mandate complex solutions. There's nothing
> > complex about tv_nsec being long. long is the smallest type that C
> > guarantees to be large enough to store the range of values, which is
> > forever fixed and can't grow (because the definition of "nano" prefix
> > is fixed :). The type has been long ever since the structure was
> > introduced, and its being long means that there's lots of (correct!)
> > code using %ld (e.g. ".%.9ld" to format results as a decimal without
> > using floating point approximations) to print it. There might also be
> > code taking pointers to it to pass to functions, etc.
> >
> > The only reason a "complex" need arises is that Linux did something
> > horribly wrong here, ignoring the specified type, when introducing an
> > obscure subarch that almost nobody uses. This kind of mistake is
> > becoming a theme in Linux (see also: msghdr). Application authors
> > should not have to pay the price for fixing this by retrofitting yet
> > another silly type like "snseconds_t" or something into programs to
> > accommodate the mistakes of x32.
> >
> >
>
> Does anyone know *why* Linux’s x32 has __kernel_long_t defined as long long?  I assume that this is where this bug, and most of the other bugs, came from.
>
> This may be silly, but the kernel could plausibly add a x32v2 where long is genuinely 32-bit, and then maybe we could drop the old x32 at some point.  From all the discussion so far, it seems like there really is some demand for ILP32, but it’s still not really clear that preserving compatibility with existing x32 binaries in the long run is critical.

Re: Can we drop upstream Linux x32 support?

[Linus Torvalds]

On Fri, Dec 14, 2018 at 10:58 AM Andy Lutomirski <luto@amacapital.net> wrote:
>
> Does anyone know *why* Linux’s x32 has __kernel_long_t defined as long long?

It *needs* to be long long, since the headers are used for builds in
user mode using ILP32.

Since __kernel_long_t is a 64-bit (the _kernel_ is not ILP32), you
need to use "long long" when building in ILP32.

Obviously, it could be something like

   #ifdef __KERNEL__
    typedef long __kernel_long_t;
   #else
    typedef long long __kernel_long_t;
   #endif

or similar to make it more obvious what's going on.

Or we could encourage all the uapi header files to always just use
explicit sizing like __u64, but some of the structures really end up
being "kernel long size" for sad historical reasons. Not lovely, but
there we are..

              Linus

Re: Can we drop upstream Linux x32 support?

[Andy Lutomirski]

On Fri, Dec 14, 2018 at 12:13 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Fri, Dec 14, 2018 at 10:58 AM Andy Lutomirski <luto@amacapital.net> wrote:
> >
> > Does anyone know *why* Linux’s x32 has __kernel_long_t defined as long long?
>
> It *needs* to be long long, since the headers are used for builds in
> user mode using ILP32.
>
> Since __kernel_long_t is a 64-bit (the _kernel_ is not ILP32), you
> need to use "long long" when building in ILP32.
>
> Obviously, it could be something like
>
>    #ifdef __KERNEL__
>     typedef long __kernel_long_t;
>    #else
>     typedef long long __kernel_long_t;
>    #endif
>
> or similar to make it more obvious what's going on.
>
> Or we could encourage all the uapi header files to always just use
> explicit sizing like __u64, but some of the structures really end up
> being "kernel long size" for sad historical reasons. Not lovely, but
> there we are..
>

This is probably water under the bridge, but I disagree with you here.
Or rather, I agree with you in principle but I really don't like the
way it turned out.

For legacy uapi structs (and probably some new ones too, sigh), as a
practical matter, user code is going to shove them at the C compiler,
and the C compiler is going to interpret them in the usual way, and
either we need a usermode translation layer or the kernel needs to
deal with the result.

It's a nice thought that, by convincing an x32 compiler that
__kernel_long_t is 64 bits, we end up with the x32 struct being
compatible with the native Linux struct, but it only works for structs
where the only ABI-dependent type is long.  But the real structs in
uapi aren't all like this.  We have struct iovec:

struct iovec
{
        void __user *iov_base;  /* BSD uses caddr_t (1003.1g requires void *) */
        __kernel_size_t iov_len; /* Must be size_t (1003.1g) */
};

Whoops, this one looks the same on x32 and i386, but neither one of
them match x86_64, and, just randomly grepping around a bit, I see:

struct snd_hwdep_dsp_image {
        unsigned int index;             /* W: DSP index */
        unsigned char name[64];         /* W: ID (e.g. file name) */
        unsigned char __user *image;    /* W: binary image */
        size_t length;                  /* W: size of image in bytes */
        unsigned long driver_data;      /* W: driver-specific data */
};

struct __sysctl_args {
        int __user *name;
        int nlen;
        void __user *oldval;
        size_t __user *oldlenp;
        void __user *newval;
        size_t newlen;
        unsigned long __unused[4];
};

If these had been switched from "unsigned long" to __kernel_ulong_t,
they would have had three different layouts on i386, x32, and x86_64.

So now we have a situation where, if we were to make x32 work 100%,
the whole kernel would need to recognize that there are three possible
ABIs, not two.  And this sucks.

So I think it would have been a better choice to let long be 32-bit on
x32 and to therefore make x32 match the x86_32 "compat" layout as much
as possible.  Sure, this would make x32 be more of a second-class
citizen, but I think it would have worked better, had fewer bugs, and
been more maintainable.

--Andy

Re: Can we drop upstream Linux x32 support?

[Thomas Schöbel-Theuer]

On 12/11/18 02:23, Andy Lutomirski wrote:
> I'm seriously considering sending a patch to remove x32 support from
> upstream Linux.

I am downstream maintainer of several self-patched kernels at 1&1 Ionos. 
The kernels are rolled out to several tenthousands of production servers 
running in several datacenters and in multiple continents.

Currently, we have a few thousands of servers relying on 32bit ABIs in 
some thousands of VMs and/or containers of various types (LXC, OpenVZ, etc).

Here is my private opinion, not speaking for 1&1: at some point the 
future, 32bit userspace support needs to be dropped anyway, somewhen in 
future. This is inevitable in the very long term.

Thus the discussion should be about _timing_ / _roadmaps_, but not about 
the fact as such.

My suggestion:

1) please release / declare a new LTS kernel, with upstream support for 
at least 5 years (as usual). Currently, only 4.4 and 4.9 are marked as 
LTS. Either mark another existing stable kernel as LTS, or a future one.

2) please _announce_ _now_ that after the _next_ LTS kernel (whichever 
you want to declare as such), you will _afterwards_ drop the legacy 
32bit support for 64 kernels (I am deliberately using "management speak" 
here).

=> result: the industry should have to fair chance to deal with such a 
roadmap. Yes, it will hurt some people, but they will have enough time 
for their migration projects.

Example: I know that out of several millions of customers of webhosting, 
a very low number of them have some very old legacy 32bit software 
installed in their webspace. This cannot be supported forever. But the 
number of such cases is very small, and there just needs to be enough 
time for finding a solution for those few customers.

3) the next development kernel _after_ that LTS release can then 
immediately drop the 32bit support. Enterprise users should have enough 
time for planning, and for lots of internal projects modernizing their 
infrastructure. Usually, they will need to do this anyway in the long term.

A roadmap should be _reliable_ for planning, and there should be no 
"unexpected surprises". That's the most important requirements.

Notice: 5 years is what I know will very likely work for 1&1 Ionos. I 
cannot speak for other large-scale enterprise users, but I think most of 
them should be able to deal with suchalike intervals in a clear roadmap.


Addendum / side note: I know of cases where _critical_ / sometimes even 
proprietary 32bit enterprise software needs to run for several 
_decades_. Please don't drop KVM/qemu support for 32bit guests using 
_old_ (unchanged) 32bit kernels, which are just happen to run on 64bit 
hypervisors. But AFAICS that was not the intention of this initiative. 
Just keep the latter in mind as another (independent) requirement.


Cheers,

Thomas

Re: Can we drop upstream Linux x32 support?

[Andy Lutomirski]

On Fri, Dec 14, 2018 at 1:22 PM Thomas Schöbel-Theuer
<thomas@schoebel-theuer.de> wrote:
>
> On 12/11/18 02:23, Andy Lutomirski wrote:
> > I'm seriously considering sending a patch to remove x32 support from
> > upstream Linux.
>
> I am downstream maintainer of several self-patched kernels at 1&1 Ionos.
> The kernels are rolled out to several tenthousands of production servers
> running in several datacenters and in multiple continents.
>
> Currently, we have a few thousands of servers relying on 32bit ABIs in
> some thousands of VMs and/or containers of various types (LXC, OpenVZ, etc).

I should clarify again: I am not suggesting that we drop 32-bit
support in the forseeable future.  (Well, I might eventually suggest
that we drop support for 32-bit *hardware* at some point, but not for
32-bit compat software.)  Linux's compat code is quite robust and is
even fairly maintainable.

I'm talking about x32, which is a different beast.

Re: Can we drop upstream Linux x32 support?

[Thomas Schöbel-Theuer]

On 12/14/18 22:24, Andy Lutomirski wrote:
> I should clarify again: I am not suggesting that we drop 32-bit
> support in the forseeable future.  (Well, I might eventually suggest
> that we drop support for 32-bit *hardware* at some point, but not for
> 32-bit compat software.)  Linux's compat code is quite robust and is
> even fairly maintainable.
>
> I'm talking about x32, which is a different beast.
>
OK, this does not really make a big difference for large-scale 
enterprise users. Some day in the very distant future any legacy system 
will disappear, eventually.

So from my viewpoint the mentioned roadmap / timing requirements will 
remain the same, whatever you are dropping.

Enterprise-critical use cases will probably need to be migrated to 
KVM/qemu together with their old kernel versions, anyway (because the 
original hardware will be no longer available in a few decades).

Re: Can we drop upstream Linux x32 support?

[Thomas Schoebel-Theuer]

On 12/14/18 22:41, Thomas Schöbel-Theuer wrote:
> On 12/14/18 22:24, Andy Lutomirski wrote:
>>
>> I'm talking about x32, which is a different beast.
>>
>
> So from my viewpoint the mentioned roadmap / timing requirements will 
> remain the same, whatever you are dropping.
>
> Enterprise-critical use cases will probably need to be migrated to 
> KVM/qemu together with their old kernel versions, anyway (because the 
> original hardware will be no longer available in a few decades).
>

Here is a systematic approach to the problem.


AFAICS legacy 32bit userspace code (which exists in some notable masses) 
can be executed at least in the following ways:


1) natively on 32bit-capable hardware, under 32bit kernels. Besides 
legacy hardware, this also encompasses most current Intel / AMD 64bit 
hardware in 32bit compatibility mode.

2) under 64bit kernels, using the 32bit compat layer from practically 
any kernel version.

3) under KVM/qemu.


When you just drop 1), users have a fair chance by migrating to any of 
the other two possibilities.

As explained, a time frame of ~5 years should work for the vast majority.

If you clearly explain the migration paths to your users (and to the 
press), I think it will be acceptable.


[side note: I know of a single legacy instance which is now ~20 years 
old, but makes a revenue of several millions per month. These guys have 
large quantities of legacy hardware in stock. And they have enough money 
to hire a downstream maintainer in case of emergency.]


Fatal problems would only arise if you would drop all three 
possibilities in the very long term.


In ~100 years, possibility 3) should be sufficient for handling use 
cases like preservation of historic documents. The latter is roughly 
equivalent to running binary-only MSDOS, Windows NT, and similar, even 
in 100 years, and even non-natively under future hardware architectures.

Re: Can we drop upstream Linux x32 support?

[Andy Lutomirski]

> On Dec 14, 2018, at 11:41 PM, Thomas Schoebel-Theuer <tst@schoebel-theuer.de> wrote:
> 
>> On 12/14/18 22:41, Thomas Schöbel-Theuer wrote:
>>> On 12/14/18 22:24, Andy Lutomirski wrote:
>>> 
>>> I'm talking about x32, which is a different beast.
>>> 
>> 
>> So from my viewpoint the mentioned roadmap / timing requirements will remain the same, whatever you are dropping.
>> 
>> Enterprise-critical use cases will probably need to be migrated to KVM/qemu together with their old kernel versions, anyway (because the original hardware will be no longer available in a few decades).
>> 
> 
> Here is a systematic approach to the problem.
> 

This is way off topic. Let’s not discuss it on this thread, please.

Re: Can we drop upstream Linux x32 support?

[Florian Weimer]

* Thomas Schöbel-Theuer:

> 2) please _announce_ _now_ that after the _next_ LTS kernel (whichever
> you want to declare as such), you will _afterwards_ drop the legacy
> 32bit support for 64 kernels (I am deliberately using "management
> speak" here).
>
> => result: the industry should have to fair chance to deal with such a
> roadmap. Yes, it will hurt some people, but they will have enough time
> for their migration projects.
>
> Example: I know that out of several millions of customers of
> webhosting, a very low number of them have some very old legacy 32bit
> software installed in their webspace. This cannot be supported
> forever. But the number of such cases is very small, and there just
> needs to be enough time for finding a solution for those few
> customers.
>
> 3) the next development kernel _after_ that LTS release can then
> immediately drop the 32bit support. Enterprise users should have
> enough time for planning, and for lots of internal projects
> modernizing their infrastructure. Usually, they will need to do this
> anyway in the long term.

We've already phased out support for all 32-bit architectures except
i386 in our products, and i386 is obviously next.  (We never supported
x32 in the first place.)

It becomes increasingly difficult to build a 32-bit userspace that meets
backwards-compatibility needs.  We want to use SSE2 (to avoid excess
precision for double) and avoid relying on stack realignment (for
compatibility with applications that use the old i386 ABI which did not
require stack realignment).  We also have to build the distribution with
a full complement of hardening flags.  This results in a combination of
flags that is poorly tested in upstream GCC.  The i386 register file
isn't large enough to support all these features at the same time and
combine them with function arguments passed in registers (which some
programs enable manually via function attributes).

So even if we keep the kernel interface, in the forseeable future, I
expect that it will be difficult to build a full, contemporary 32-bit
userspace on i386.

Thanks,
Florian

Re: Can we drop upstream Linux x32 support?

[Rich Felker]

On Wed, Jan 09, 2019 at 01:41:14PM +0100, Florian Weimer wrote:
> * Thomas Schöbel-Theuer:
> 
> > 2) please _announce_ _now_ that after the _next_ LTS kernel (whichever
> > you want to declare as such), you will _afterwards_ drop the legacy
> > 32bit support for 64 kernels (I am deliberately using "management
> > speak" here).
> >
> > => result: the industry should have to fair chance to deal with such a
> > roadmap. Yes, it will hurt some people, but they will have enough time
> > for their migration projects.
> >
> > Example: I know that out of several millions of customers of
> > webhosting, a very low number of them have some very old legacy 32bit
> > software installed in their webspace. This cannot be supported
> > forever. But the number of such cases is very small, and there just
> > needs to be enough time for finding a solution for those few
> > customers.
> >
> > 3) the next development kernel _after_ that LTS release can then
> > immediately drop the 32bit support. Enterprise users should have
> > enough time for planning, and for lots of internal projects
> > modernizing their infrastructure. Usually, they will need to do this
> > anyway in the long term.
> 
> We've already phased out support for all 32-bit architectures except
> i386 in our products, and i386 is obviously next.  (We never supported
> x32 in the first place.)
> 
> It becomes increasingly difficult to build a 32-bit userspace that meets
> backwards-compatibility needs.  We want to use SSE2 (to avoid excess
> precision for double) and avoid relying on stack realignment (for
> compatibility with applications that use the old i386 ABI which did not
> require stack realignment).  We also have to build the distribution with
> a full complement of hardening flags.  This results in a combination of
> flags that is poorly tested in upstream GCC.  The i386 register file
> isn't large enough to support all these features at the same time and
> combine them with function arguments passed in registers (which some
> programs enable manually via function attributes).
> 
> So even if we keep the kernel interface, in the forseeable future, I
> expect that it will be difficult to build a full, contemporary 32-bit
> userspace on i386.

I guess that's informative of how one company's distro process works,
but it's not representative. Your customers are enterprise and
big-server (probably mostly the former) oriented which is exactly the
domain where 32-bit is of course irrelevant except for running legacy
applications. Where it matters are embedded and other systems striving
for resource efficiency.

For what it's worth, 32-bit archs including i386 and many others are
well-supported in Debian with no forseeable EOL I'm aware of, and most
if not all of the musl-libc-based distros I'm familiar with support
32-bit archs including i386.

I don't think waning relevance of 32-bit is a reasonable argument
against x32 since x32's relevance is in exactly the places where
32-bit is already relevant and preferred for important reasons.

Rich

Re: Can we drop upstream Linux x32 support?

[Enrico Weigelt, metux IT consult]

On 14.12.18 22:16, Thomas Schöbel-Theuer wrote:

Hi,

> Currently, we have a few thousands of servers relying on 32bit ABIs in> some thousands of VMs and/or containers of various types (LXC,
OpenVZ,> etc).
Similar w/ my clients, but in Industrie/Embedded/i4.0 world. We have
thousands of devices in the field, that need support for at least a
decade.

Right now, I'm working on a new device, that - even having a 64bit cpu -
will still run 32bit userland for various reasons (memory footprint
is one of them).

Even on the data center side, where we have *a lot* of containers
(eg. container-based microservices on minimal distros like alpine
or ptxdist - as well as many different, often highly specialized,
legacy applications), 32bit userland indeed is a major factor.

Dropping it would have huge economic consequences:

* massive costs for re-deployment / migration of thousands of different
  applications
* field roll of thousands of different embedded / industrial devices,
  as a major cpu arch
* huge increase in memory consumption and io load in container-
  based microservice environments
* loosing x86 for a huge portion of the embedded / small devices world

I don't even dare to estimate the economic damage. (not even speaking
about reputation loss of the Linux community)

> Here is my private opinion, not speaking for 1&1: at some point the> future, 32bit userspace support needs to be dropped anyway, somewhen
in> future. This is inevitable in the very long term.

Very, very, long term. Just the devices I'm coping w/ will have
remaining lifetime of at least 10..15 years. That's a really long time
in IT world. Nobody knows whether x86 at all will play a big role then.

> 1) please release / declare a new LTS kernel, with upstream support for
> at least 5 years (as usual). Currently, only 4.4 and 4.9 are marked as
> LTS. Either mark another existing stable kernel as LTS, or a future one.

That might be okay in enterprise world (even though, here I'm regularily
stumbling across much older code in production), but in industrial
world, the product lifetimes are much longer - 20+yrs years
usualstandard.are pretty common.

--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info@metux.net -- +49-151-27565287