Re: Continuing the UAPI split

Re: Continuing the UAPI split

[Christian Brauner]

[+Cc Florian, glibc]

On November 7, 2019 12:17:50 PM GMT+01:00, Elichai Turkel <elichai.turkel@gmail.com> wrote:
>Hi,
>I'm working on a library that calls syscalls directly to the kernel.
>`make hedears_install` is a great command to auto generate the UAPI
>headers that are needed to call the kernel.
>
>But the headers are still missing a bunch of defines for flags and
>structs.
>
>I wanted to know if patches to move more things from `./include/linux`
>to `./include/uapi/linux` are welcome (obviously only
>typedefs/defines/structs that are required for the syscalls)
>
>I think the UAPI is really close to getting a complete set of things
>needed to communicate with the syscalls, but still not quite there. I
>would like to push patches whenever I see missing things that my
>library needs (that way it will be incrementally and by usage only).
>
>Would love to get feedback.
>Thanks,
>Elichai.

Re: Continuing the UAPI split

[Florian Weimer]

* Christian Brauner:

> [+Cc Florian, glibc]
>
> On November 7, 2019 12:17:50 PM GMT+01:00, Elichai Turkel <elichai.turkel@gmail.com> wrote:
>>Hi,
>>I'm working on a library that calls syscalls directly to the kernel.
>>`make hedears_install` is a great command to auto generate the UAPI
>>headers that are needed to call the kernel.
>>
>>But the headers are still missing a bunch of defines for flags and
>>structs.
>>
>>I wanted to know if patches to move more things from `./include/linux`
>>to `./include/uapi/linux` are welcome (obviously only
>>typedefs/defines/structs that are required for the syscalls)
>>
>>I think the UAPI is really close to getting a complete set of things
>>needed to communicate with the syscalls, but still not quite there. I
>>would like to push patches whenever I see missing things that my
>>library needs (that way it will be incrementally and by usage only).

The kernel uses some POSIX names with POSIX-incompatible definitions,
e.g. struct msghdr.  Some libcs prioritize POSIX conformance over kernel
conformance and implement userspace translation for mismatch types.
When building against such libcs, it becomes difficult to use UAPI and
libc headers in a single translation unit.  (It is already difficult
today in some cases.)

I don't know a good solution here.  Not using POSIX names in UAPI
headers is one option.  Previously, we have tried to use preprocessor
macros to coordinate definitions, but did not work well in practice
(only few conflicts were ever resolved).  It does not help at all when
the definitions are incompatible.

Thanks,
Florian

Re: Continuing the UAPI split

[Elichai Turkel]

On Thu, Nov 7, 2019 at 2:10 PM Florian Weimer <fweimer@redhat.com> wrote:
>
> The kernel uses some POSIX names with POSIX-incompatible definitions,
> e.g. struct msghdr.  Some libcs prioritize POSIX conformance over kernel
> conformance and implement userspace translation for mismatch types.
> When building against such libcs, it becomes difficult to use UAPI and
> libc headers in a single translation unit.  (It is already difficult
> today in some cases.)
>
> I don't know a good solution here.  Not using POSIX names in UAPI
> headers is one option.  Previously, we have tried to use preprocessor
> macros to coordinate definitions, but did not work well in practice
> (only few conflicts were ever resolved).  It does not help at all when
> the definitions are incompatible.
>
> Thanks,
> Florian
>

Thanks for the response,
I'm not sure what are you suggesting exactly.
A rename to the structs/types so they won't collide with libc?
Prioritizing POSIX conformance in the kernel(I think that ship has long sailed)?
Or just giving up and telling users they can't just directly include
both libc headers and kernel headers?
(I'm actually not sure how it works now. because there are definitely
collision and if we are using ifdefs and undefs black magic you still
end up with a single declaration in the end that might not be
compatible with both libc and kernel)

I'd personally really like to see a better separation between (g)libc
and the kernel. I

Thanks,
Elichai.

-- 
PGP: 5607C93B5F86650C

Re: Continuing the UAPI split

[Florian Weimer]

* Elichai Turkel:

> Thanks for the response,
> I'm not sure what are you suggesting exactly.
> A rename to the structs/types so they won't collide with libc?

Or use namespaces.  I mean, we have had proper technical solutions for
this issue for more than 20 years now.  I know that this isn't going to
happen, though.

> Prioritizing POSIX conformance in the kernel(I think that ship has long sailed)?

That doesn't really work anyway if userspace has different versions of
the type depending on feature macros.  Examples are struct stat or soon
struct timespec on 32-bit architectures.

> Or just giving up and telling users they can't just directly include
> both libc headers and kernel headers?

That's what I've been telling users if they encounter fringe cases I
can't fix on the glibc side.

It might also help if top-level types for end user use would be in
separate headers.  This is what we started doing internally in glibc, to
disentangle our own headers and eliminate cyclic dependencies.

> (I'm actually not sure how it works now. because there are definitely
> collision and if we are using ifdefs and undefs black magic you still
> end up with a single declaration in the end that might not be
> compatible with both libc and kernel)

Officially, it's supposed to work today.  We have one glibc developer
who says that it's easy to solve, but I disagree strongly.  The fact
that the problems are not fixed promptly suggests that it's anything but
simple.

What I've been doing instead is to include UAPI headers directly from
glibc headers if it's technically feasible.  (It's not possible in POSIX
mode, where we have to manage the set of exported symbols closely, or
generally with old compilers which do not have __has_include.)  See
<bits/statx.h>, <bits/statx-generic.h> etc. in current glibc for an
example.

If you add more type definitions to kernel headers, this might interfere
with such usage of UAPI headers because today, we need to provide our
own definitions for many things not in UAPI headers, and the currently
deployed glibc headers are simply not compatible with such future
changes.

Thanks,
Florian

Re: Continuing the UAPI split

[Christian Brauner]

On Thu, Nov 07, 2019 at 02:23:26PM +0100, Florian Weimer wrote:
> * Elichai Turkel:
> 
> > Thanks for the response,
> > I'm not sure what are you suggesting exactly.
> > A rename to the structs/types so they won't collide with libc?
> 
> Or use namespaces.  I mean, we have had proper technical solutions for
> this issue for more than 20 years now.  I know that this isn't going to
> happen, though.
> 
> > Prioritizing POSIX conformance in the kernel(I think that ship has long sailed)?
> 
> That doesn't really work anyway if userspace has different versions of
> the type depending on feature macros.  Examples are struct stat or soon
> struct timespec on 32-bit architectures.
> 
> > Or just giving up and telling users they can't just directly include
> > both libc headers and kernel headers?
> 
> That's what I've been telling users if they encounter fringe cases I
> can't fix on the glibc side.
> 
> It might also help if top-level types for end user use would be in
> separate headers.  This is what we started doing internally in glibc, to
> disentangle our own headers and eliminate cyclic dependencies.
> 
> > (I'm actually not sure how it works now. because there are definitely
> > collision and if we are using ifdefs and undefs black magic you still
> > end up with a single declaration in the end that might not be
> > compatible with both libc and kernel)
> 
> Officially, it's supposed to work today.  We have one glibc developer
> who says that it's easy to solve, but I disagree strongly.  The fact
> that the problems are not fixed promptly suggests that it's anything but
> simple.
> 
> What I've been doing instead is to include UAPI headers directly from
> glibc headers if it's technically feasible.  (It's not possible in POSIX

A problem I recently ran into that is related are problems with
sys/wait.h and linux/wait.h.
How P_{PID,PGID,PIDFD} used by the waitid() syscall are defined is
different for the two headers.
linux/wait.h uses #define for P_{PID,PGID,PIDFD} whereas sys/wait.h
uses an enum.
The problem is that if I simply don't rely on sys/wait.h and just do
#ifndef P_PID
#define P_PID <value>
#endif
where value is what the syscall expects then technically I need to call
the waitid() syscall directly since it's not at all guaranteed - afaict
- that the P_PID enum == P_PID define that glibc uses for its waitid()
syscall wrapper.
So I'm now in a scenario where I can't call the glibc wrapper for
waitid() with the linux/wait.h defines and I can't call the syscall
directly (e.g. because I want to make use of the syscall's rusage
argument that glibc doesn't expose) with sys/wait.h's P_PID enum.
I'm not sure what the right solution is here...

Christian

Re: Continuing the UAPI split

[Florian Weimer]

* Christian Brauner:

> A problem I recently ran into that is related are problems with
> sys/wait.h and linux/wait.h.
> How P_{PID,PGID,PIDFD} used by the waitid() syscall are defined is
> different for the two headers.
> linux/wait.h uses #define for P_{PID,PGID,PIDFD} whereas sys/wait.h
> uses an enum.
> The problem is that if I simply don't rely on sys/wait.h and just do
> #ifndef P_PID
> #define P_PID <value>
> #endif
> where value is what the syscall expects then technically I need to call
> the waitid() syscall directly since it's not at all guaranteed - afaict
> - that the P_PID enum == P_PID define that glibc uses for its waitid()
> syscall wrapper.

Right, and this is where POSIX mandates that there is a type idtype_t
which is an enum, an that it has members P_PID etc.

What we could do is:

typedef enum
{
  P_ALL,		/* Wait for any child.  */
#define P_ALL P_ALL
  P_PID,		/* Wait for specified process.  */
#define P_PID P_PID
  P_PGID		/* Wait for members of process group.  */
#define P_PGID P_PGID
} idtype_t;

The other header can then use #ifdef.  You'll see that pattern in some
cases already.

But that will only work if you include glibc headers first.  The generic
approach uses some shared macro for the coordination so that things work
both ways.

The other issue here is that it gets rather iffy from a language point
of view if the kernel wants to add additional constants and glibc has
still the old idtype_t definition.

> So I'm now in a scenario where I can't call the glibc wrapper for
> waitid() with the linux/wait.h defines and I can't call the syscall
> directly (e.g. because I want to make use of the syscall's rusage
> argument that glibc doesn't expose) with sys/wait.h's P_PID enum.
> I'm not sure what the right solution is here...

Yes, it's a hard problem.  waitid is particularly annoying because POSIX
and the kernel have such different function prototypes.  We could
perhaps expose the actual waitid system call under a different name, and
use int for the ID type parameter.  But that needs someone to write a
patch.  (My efforts to add syscall wrappers have stalled unfortunately.)

Thanks,
Florian

Re: Continuing the UAPI split

[Christian Brauner]

On Thu, Nov 07, 2019 at 02:47:54PM +0100, Florian Weimer wrote:
> * Christian Brauner:
> 
> > A problem I recently ran into that is related are problems with
> > sys/wait.h and linux/wait.h.
> > How P_{PID,PGID,PIDFD} used by the waitid() syscall are defined is
> > different for the two headers.
> > linux/wait.h uses #define for P_{PID,PGID,PIDFD} whereas sys/wait.h
> > uses an enum.
> > The problem is that if I simply don't rely on sys/wait.h and just do
> > #ifndef P_PID
> > #define P_PID <value>
> > #endif
> > where value is what the syscall expects then technically I need to call
> > the waitid() syscall directly since it's not at all guaranteed - afaict
> > - that the P_PID enum == P_PID define that glibc uses for its waitid()
> > syscall wrapper.
> 
> Right, and this is where POSIX mandates that there is a type idtype_t
> which is an enum, an that it has members P_PID etc.
> 
> What we could do is:
> 
> typedef enum
> {
>   P_ALL,		/* Wait for any child.  */
> #define P_ALL P_ALL
>   P_PID,		/* Wait for specified process.  */
> #define P_PID P_PID
>   P_PGID		/* Wait for members of process group.  */
> #define P_PGID P_PGID
> } idtype_t;

Right, that sounds feasible.
(Nit/slightly off-topic: David (forgot to Cc him...) once reminded me
 that we do prefer to explicitly set the enum value so that there are no
 accidental changes, e.g.:

typedef enum
{
  P_ALL = 0,		/* Wait for any child.  */
#define P_ALL P_ALL
  P_PID = 1,		/* Wait for specified process.  */
#define P_PID P_PID
  P_PGID = 2,		/* Wait for members of process group.  */
#define P_PGID P_PGID
  P_PIDFD = 3,		/* Wait via pidfds.  */
#define P_PIDFD P_PIDFD
} idtype_t;

)

> 
> The other header can then use #ifdef.  You'll see that pattern in some
> cases already.
> 
> But that will only work if you include glibc headers first.  The generic
> approach uses some shared macro for the coordination so that things work
> both ways.

We saw a conflict at least on Fedora for the pidfd-tests with the new
P_PIDFD type defined in linux/wait.h but not yet in sys/wait.h and it
was exactly caused by wrong inclusion order. :)

> 
> The other issue here is that it gets rather iffy from a language point
> of view if the kernel wants to add additional constants and glibc has
> still the old idtype_t definition.
> 
> > So I'm now in a scenario where I can't call the glibc wrapper for
> > waitid() with the linux/wait.h defines and I can't call the syscall
> > directly (e.g. because I want to make use of the syscall's rusage
> > argument that glibc doesn't expose) with sys/wait.h's P_PID enum.
> > I'm not sure what the right solution is here...
> 
> Yes, it's a hard problem.  waitid is particularly annoying because POSIX
> and the kernel have such different function prototypes.  We could
> perhaps expose the actual waitid system call under a different name, and

Wouldn't be the worst idea.

> use int for the ID type parameter.  But that needs someone to write a
> patch.  (My efforts to add syscall wrappers have stalled unfortunately.)

This could be a good Google Summer of Code or Outreachy project or
something tbh. In general, some more noise around this would probably
help. Especially with the ability to use Gerrit to submit patches that
might make it easier for people to contribute...
Do we officially document somewhere how to add glibc syscall wrappers?

Christian

Re: Continuing the UAPI split

[Carlos O'Donell]

On 11/7/19 8:23 AM, Florian Weimer wrote:
> Officially, it's supposed to work today.  We have one glibc developer
> who says that it's easy to solve, but I disagree strongly.  The fact
> that the problems are not fixed promptly suggests that it's anything but
> simple.

Is that one glibc developer me? :-)

They aren't easy to solve, but there is a magic process in place.

Getting the definitions to line up is part of the work involved.

Sometimes they may not line up, in that case it doesn't work.

> What I've been doing instead is to include UAPI headers directly from
> glibc headers if it's technically feasible.  (It's not possible in POSIX
> mode, where we have to manage the set of exported symbols closely, or
> generally with old compilers which do not have __has_include.)  See
> <bits/statx.h>, <bits/statx-generic.h> etc. in current glibc for an
> example.

That's really the best way to solve it if you can.

> If you add more type definitions to kernel headers, this might interfere
> with such usage of UAPI headers because today, we need to provide our
> own definitions for many things not in UAPI headers, and the currently
> deployed glibc headers are simply not compatible with such future
> changes.

Right.

-- 
Cheers,
Carlos.

Re: Continuing the UAPI split

[Szabolcs Nagy]

On 07/11/2019 13:03, Elichai Turkel wrote:
> A rename to the structs/types so they won't collide with libc?
> Prioritizing POSIX conformance in the kernel(I think that ship has long sailed)?

hosted c code can only use linux api headers reliably
if those respect the iso c and posix rules.

linux should maintain a clean set of c headers (or
interface descriptions from which such headers can be
generated).

providing clean headers that work in both hosted and
freestanding environments is not trivial: e.g. the
former requires type definitions to use libc types,
the latter requires type definitions that don't depend
on the libc.

e.g. a possible approach is to use iso c implementation
reserved namespace for all linux api identifiers, so
it's entirely independent of libc and non-conflicting,
then libc headers can replicate declarations that it
wants to expose (with different names and libc types).
(there are other ways, but the current headers are hard
to use which affects a lot of users)

> Or just giving up and telling users they can't just directly include
> both libc headers and kernel headers?

including both libc and linux headers is fragile and
will break differently across the different linux
libc implementations.

Re: Continuing the UAPI split

[Carlos O'Donell]

On 11/7/19 11:21 AM, Szabolcs Nagy wrote:
>> Or just giving up and telling users they can't just directly include
>> both libc headers and kernel headers?
> 
> including both libc and linux headers is fragile and
> will break differently across the different linux
> libc implementations.

We saw this all the time working in embedded.

There ends up being a rather tight version coupling that
happens, and that's unfortunate, but a function of cost.

Fixing this so it works in the various cases is costly.

-- 
Cheers,
Carlos.

Re: Continuing the UAPI split

[Florian Weimer]

* Carlos O'Donell:

> On 11/7/19 11:21 AM, Szabolcs Nagy wrote:
>>> Or just giving up and telling users they can't just directly include
>>> both libc headers and kernel headers?
>> 
>> including both libc and linux headers is fragile and
>> will break differently across the different linux
>> libc implementations.
>
> We saw this all the time working in embedded.

Do you mean you saw problems while you working in the embedded space?

Re: Continuing the UAPI split

[Carlos O'Donell]

On 11/7/19 3:32 PM, Florian Weimer wrote:
> * Carlos O'Donell:
> 
>> On 11/7/19 11:21 AM, Szabolcs Nagy wrote:
>>>> Or just giving up and telling users they can't just directly include
>>>> both libc headers and kernel headers?
>>>
>>> including both libc and linux headers is fragile and
>>> will break differently across the different linux
>>> libc implementations.
>>
>> We saw this all the time working in embedded.
> 
> Do you mean you saw problems while you working in the embedded space?
 
Yes, embedded Linux to be specific.

There is a strong coupling between the kernel version and the toolchain
version, specifically because the strategies we take to solve these
problems end up being brittle in this regard.

Too new a kernel and you get new header problems not solved by your
old libc. Too new a libc and the kernel doesn't have the header
coordination fixes required for the newer software that needed the
newer libc.

Does that clarify my point?

-- 
Cheers,
Carlos.

Re: Continuing the UAPI split

[Florian Weimer]

* Carlos O'Donell:

> On 11/7/19 3:32 PM, Florian Weimer wrote:
>> * Carlos O'Donell:
>> 
>>> On 11/7/19 11:21 AM, Szabolcs Nagy wrote:
>>>>> Or just giving up and telling users they can't just directly include
>>>>> both libc headers and kernel headers?
>>>>
>>>> including both libc and linux headers is fragile and
>>>> will break differently across the different linux
>>>> libc implementations.
>>>
>>> We saw this all the time working in embedded.
>> 
>> Do you mean you saw problems while you working in the embedded space?
>  
> Yes, embedded Linux to be specific.
>
> There is a strong coupling between the kernel version and the toolchain
> version, specifically because the strategies we take to solve these
> problems end up being brittle in this regard.
>
> Too new a kernel and you get new header problems not solved by your
> old libc. Too new a libc and the kernel doesn't have the header
> coordination fixes required for the newer software that needed the
> newer libc.
>
> Does that clarify my point?

Yes, it does.  It wasn't clear to me if you wanted to say that this
was actually working for the embedded case.