* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 [not found] ` <CAHk-=wgoX0pVqNMMOcrhq=nuOfoZB_3qihyHB3y1S8qo=MDs6w@mail.gmail.com> @ 2021-09-03 23:04 ` Nathan Chancellor 2021-09-04 8:01 ` Florian Weimer 0 siblings, 1 reply; 27+ messages in thread From: Nathan Chancellor @ 2021-09-03 23:04 UTC (permalink / raw) To: Linus Torvalds, Masahiro Yamada Cc: Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On 9/3/2021 3:53 PM, Linus Torvalds wrote:> On Thu, Sep 2, 2021 at 4:31 PM Masahiro Yamada <masahiroy@kernel.org> wrote: >> >> I fixed the warnings observed in the previous PR. > > Ok, let's try it again. > >> - Add <linux/stdarg.h> to the kernel source instead of borrowing >> <stdarg.h> from the compiler. > > So I certainly agree with the reasoning, but this worries me a bit. > > stdarg is truly intimately an internal compiler file, in ways that > stddef (to pick another example) isn't. > > Yeah, yeah, offsetof() is "kind of compiler internal", and we end up > using __compiler_offsetof(), but in the absence of that we *can* just > do it by hand. So offsetof() really is one of those things where we > can just do our own version if some compiler is being difficult. > > But va_start and friends absolutely *must* match the exact compiler version. > > It does look like both gcc and clang have just standardized on using > __builtin_xyz for all the different stdarg things, and so I approve of > what that <linux/stdarg.h> ended up looking like. > > But at the same time, it does make me go "ok, this is a big new > assumption that we've consciously avoided for a long time". > > Nick is already on the cc here for other reasons, but let's add the > clang-built list and Nathan explicitly. Because this basically > codifies that > > typedef __builtin_va_list va_list; > #define va_start(v, l) __builtin_va_start(v, l) > #define va_end(v) __builtin_va_end(v) > #define va_arg(v, T) __builtin_va_arg(v, T) > #define va_copy(d, s) __builtin_va_copy(d, s) > > being the way all the supported compilers work. > > Did people talk to any gcc maintainers too? We don't have the same > kind of "gcc kernel people" list or contacts. The above builtins have > been the case for a long long time for gcc, so I don't think it's > wrong or likely to change, but I think it would be a good thing to > just make compiler people aware of how we're now relying on that > explicitly. We set up the linux-toolchains mailing list after Plumbers 2020 to have a common place that kernel developers can bring issues and discussion to both clang and GCC folks. I am not sure who exactly from the GCC world is subscribed but I have added it now to see. > (Side note: Linux using the compiler <stdarg.h> goes so far back that > it very much predates all those nice builtins. I still have memories > of <stdarg.h> being a collection of nasty per-architecture messes back > in the bad old days. So I'm actually happy we can do this now, but > there most definitely was a time when we really really had to use the > compiler-provided stdarg.h). Cheers, Nathan ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-03 23:04 ` [GIT PULL v2] Kbuild updates for v5.15-rc1 Nathan Chancellor @ 2021-09-04 8:01 ` Florian Weimer 2021-09-04 13:19 ` Segher Boessenkool 0 siblings, 1 reply; 27+ messages in thread From: Florian Weimer @ 2021-09-04 8:01 UTC (permalink / raw) To: Nathan Chancellor Cc: Linus Torvalds, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains * Nathan Chancellor: > We set up the linux-toolchains mailing list after Plumbers 2020 to > have a common place that kernel developers can bring issues and > discussion to both clang and GCC folks. I am not sure who exactly from > the GCC world is subscribed but I have added it now to see. Someone said that they “agree with the reasoning”, but the original patch does not provide one. It looks like it's about preventing the use of compiler-supplied header files, but even that doesn't really answer the question: why? Especially since some parts of the kernel actually need some of those header files. Thanks, Florian ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-04 8:01 ` Florian Weimer @ 2021-09-04 13:19 ` Segher Boessenkool 2021-09-04 15:19 ` Florian Weimer 0 siblings, 1 reply; 27+ messages in thread From: Segher Boessenkool @ 2021-09-04 13:19 UTC (permalink / raw) To: Florian Weimer Cc: Nathan Chancellor, Linus Torvalds, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Sat, Sep 04, 2021 at 10:01:07AM +0200, Florian Weimer wrote: > * Nathan Chancellor: > > We set up the linux-toolchains mailing list after Plumbers 2020 to > > have a common place that kernel developers can bring issues and > > discussion to both clang and GCC folks. I am not sure who exactly from > > the GCC world is subscribed but I have added it now to see. > > Someone said that they “agree with the reasoning”, but the original > patch does not provide one. It looks like it's about preventing the use > of compiler-supplied header files, but even that doesn't really answer > the question: why? > > Especially since some parts of the kernel actually need some of those > header files. Let me quote the original mail (I had to dig it out of the archives as well, no nice threading, too lazy, sorry): > On Thu, Sep 2, 2021 at 4:31 PM Masahiro Yamada <masahiroy@kernel.org> wrote: > > > > I fixed the warnings observed in the previous PR. > > Ok, let's try it again. > > > - Add <linux/stdarg.h> to the kernel source instead of borrowing > > <stdarg.h> from the compiler. > > So I certainly agree with the reasoning, but this worries me a bit. > > stdarg is truly intimately an internal compiler file, in ways that > stddef (to pick another example) isn't. > > Yeah, yeah, offsetof() is "kind of compiler internal", and we end up > using __compiler_offsetof(), but in the absence of that we *can* just > do it by hand. So offsetof() really is one of those things where we > can just do our own version if some compiler is being difficult. > > But va_start and friends absolutely *must* match the exact compiler version. > > It does look like both gcc and clang have just standardized on using > __builtin_xyz for all the different stdarg things, and so I approve of > what that <linux/stdarg.h> ended up looking like. > > But at the same time, it does make me go "ok, this is a big new > assumption that we've consciously avoided for a long time". > > Nick is already on the cc here for other reasons, but let's add the > clang-built list and Nathan explicitly. Because this basically > codifies that > > typedef __builtin_va_list va_list; > #define va_start(v, l) __builtin_va_start(v, l) > #define va_end(v) __builtin_va_end(v) > #define va_arg(v, T) __builtin_va_arg(v, T) > #define va_copy(d, s) __builtin_va_copy(d, s) > > being the way all the supported compilers work. > > Did people talk to any gcc maintainers too? We don't have the same > kind of "gcc kernel people" list or contacts. The above builtins have > been the case for a long long time for gcc, so I don't think it's > wrong or likely to change, but I think it would be a good thing to > just make compiler people aware of how we're now relying on that > explicitly. > > (Side note: Linux using the compiler <stdarg.h> goes so far back that > it very much predates all those nice builtins. I still have memories > of <stdarg.h> being a collection of nasty per-architecture messes back > in the bad old days. So I'm actually happy we can do this now, but > there most definitely was a time when we really really had to use the > compiler-provided stdarg.h). > > Linus <stdarg.h> is a header that any C implementation is required to provide to give some certain functionality (one type and four macros, in this case, mentioned above). No implementation is allowed to put anything in those headers that can conflict with anything in user code or in some implementation's internals, and I haven't heard of any implementation breaking in that way for decades, there is absolutely no reason not to use <stdarg.h>. It is one of the few headers required from freestanding implementations even (and <stddef.h> is another for that matter: the full list is <float.h>, <iso646.h>, <limits.h>, <stdalign.h>, <stdarg.h>, <stdbool.h>, <stddef.h>, <stdint.h>, and <stdnoreturn.h>). I recommend using this. It is what it is for. It works in all compilers. Not using it is not writing in C. Segher ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-04 13:19 ` Segher Boessenkool @ 2021-09-04 15:19 ` Florian Weimer 2021-09-04 16:19 ` Segher Boessenkool 2021-09-04 17:22 ` Linus Torvalds 0 siblings, 2 replies; 27+ messages in thread From: Florian Weimer @ 2021-09-04 15:19 UTC (permalink / raw) To: Segher Boessenkool Cc: Nathan Chancellor, Linus Torvalds, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains * Segher Boessenkool: > Let me quote the original mail (I had to dig it out of the archives as > well, no nice threading, too lazy, sorry): It still doesn't say why. I did see a reference to fleeting reference to <stdatomic.h> and <float.h>. My conjecture is that the real reason is avoid atomic emulation (softatomic?) and softfloat code. It's not related to <stdarg.h> at all: this header is replaced so that GCC's include subdirectory can be dropped from the include search path. What I don't know if this is to avoid obscure linker failures related to libatomic/softfloat (obviously not great) or run-time failures (worse). In any case, it would be nice to know what the real motivation is. After all, <stdatomic.h> is exactly like <stdarg.h> in that it's possible to use its functionality even without the header file. The __atomic builtins are even documented in the GCC manual (unlike <stdatomic.h>), which is why some programmers prefer them over the standard interface. And then there's the _Atomic keyword itself, whose use can easily result in calls to libatomic functions, too. So blocking <stdatomic.h> makes little sense to me. I don't know enough about softfloat if blocking the inclusion of <float.h> is worth it. Thanks, Florian ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-04 15:19 ` Florian Weimer @ 2021-09-04 16:19 ` Segher Boessenkool 2021-09-04 17:22 ` Linus Torvalds 1 sibling, 0 replies; 27+ messages in thread From: Segher Boessenkool @ 2021-09-04 16:19 UTC (permalink / raw) To: Florian Weimer Cc: Nathan Chancellor, Linus Torvalds, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Sat, Sep 04, 2021 at 05:19:21PM +0200, Florian Weimer wrote: > * Segher Boessenkool: > > > Let me quote the original mail (I had to dig it out of the archives as > > well, no nice threading, too lazy, sorry): > > It still doesn't say why. I did see a reference to fleeting reference > to <stdatomic.h> and <float.h>. Yeah... I dug out the actual patch from linux-kbuild: https://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git/commit/?h=kbuild-v5.15&id=c0891ac15f0428ffa81b2e818d416bdf3cb74ab6 The reasoning in there is completely wrong. <stdarg.h> is not a "userspace header". Instead, it is a standard header, required for some functionality in C. It also says "GPL 2 version of <stdarg.h> can be extracted from http://archive.debian.org/debian/pool/main/g/gcc-4.2/gcc-4.2_4.2.4.orig.tar.gz" which seems to suggest you cannot use stuff from GPLv3-licensed GCC. This is just wrong. The header in question says """ Under Section 7 of GPL version 3, you are granted additional permissions described in the GCC Runtime Library Exception, version 3.1, as published by the Free Software Foundation. """ And <https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=COPYING.RUNTIME> reads in part """ 1. Grant of Additional Permission. You have permission to propagate a work of Target Code formed by combining the Runtime Library with Independent Modules, even if such propagation would otherwise violate the terms of GPLv3, provided that all Target Code was generated by Eligible Compilation Processes. You may then convey such a combination under terms of your choice, consistent with the licensing of the Independent Modules. """ which says that if you compile with GCC, then it is perfectly fine if it uses the standard C headers, it does not make your work GPL-licenced. > After all, <stdatomic.h> is exactly like <stdarg.h> in that it's > possible to use its functionality even without the header file. The > __atomic builtins are even documented in the GCC manual (unlike > <stdatomic.h>), which is why some programmers prefer them over the > standard interface. And then there's the _Atomic keyword itself, whose > use can easily result in calls to libatomic functions, too. So blocking > <stdatomic.h> makes little sense to me. > > I don't know enough about softfloat if blocking the inclusion of > <float.h> is worth it. Blocking the use of <float.h> is pretty useless: it is possible to do millions of things in the kernel source that are frowned upon, or actively bad, or anything in between or more extreme. That is what code review is for. If it would be a common mistake (it is not afaik) you can warn for it from checkpatch.pl or something. The patch is just re-implementing part of the standard GCC <stdarg.h>, so that it will only work with recent GCC (and maybe clang as well if it implements the GCC internal interfaces correctly (i.e. compatibly) here, and the same for other compilers). Almost all of the <stdarg.h> GCC itself uses is the same, but it also is compatible to the various C standards if this header is included indirectly. That is all just some ifdeffery anyway, so doesn't influence compilation times noticeably, and all that. - * - So as far as I can see the motivation behind the patch is a) a misunderstanding of what standard C headers are, are for, etc.; and b) a misunderstanding of the GPL and the GCC runtime exception to it. The patch makes things worse than they were. If on the contrary Linux would use *more* standard compiler headers, say <stddef.h>, then insidious bugs like that fixed by c46bbf5d2def would be prevented. Segher ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-04 15:19 ` Florian Weimer 2021-09-04 16:19 ` Segher Boessenkool @ 2021-09-04 17:22 ` Linus Torvalds 2021-09-04 19:15 ` Segher Boessenkool 2021-09-06 6:54 ` Florian Weimer 1 sibling, 2 replies; 27+ messages in thread From: Linus Torvalds @ 2021-09-04 17:22 UTC (permalink / raw) To: Florian Weimer Cc: Segher Boessenkool, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Sat, Sep 4, 2021 at 8:19 AM Florian Weimer <fweimer@redhat.com> wrote: > > In any case, it would be nice to know what the real motivation is. I don't know about the original motivation, but the reason I like that patch after-the-fact is that I've actually been in situations where I test out self-built compilers without installing them. Then it's convenient to have a completely standalone kernel tree. Nobody cares about things like <stdatomic.h> They are completely irrelevant for the kernel, exactly because we've always just done our own, or used __builtin_xyz() for things. <stdarg.h> is the only actual core header file that we have always taken from the installed system headers - because historically it really was required. Everything else we can just do ourselves. Linus ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-04 17:22 ` Linus Torvalds @ 2021-09-04 19:15 ` Segher Boessenkool 2021-09-04 19:58 ` Linus Torvalds 2021-09-06 6:54 ` Florian Weimer 1 sibling, 1 reply; 27+ messages in thread From: Segher Boessenkool @ 2021-09-04 19:15 UTC (permalink / raw) To: Linus Torvalds Cc: Florian Weimer, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Sat, Sep 04, 2021 at 10:22:25AM -0700, Linus Torvalds wrote: > On Sat, Sep 4, 2021 at 8:19 AM Florian Weimer <fweimer@redhat.com> wrote: > > In any case, it would be nice to know what the real motivation is. > > I don't know about the original motivation, but the reason I like that > patch after-the-fact is that I've actually been in situations where I > test out self-built compilers without installing them. Then you probably know that that is quite hard to do correctly. And installing the compiler (into a temporary dir that you can just blow away after you are done with it) is trivial (see for example <https://gcc.gnu.org/install/finalinstall.html> line 11 or so). Also, the standard headers are *part of* the compiler. Including those (like <stdarg.h> here) that are required for freestanding implementations. > Then it's convenient to have a completely standalone kernel tree. Yes, and you certainly do not want to mix in anything userspace, accidentally or not. But these "freestanding" headers already have to make sure they do not do that, and they are used by many OSes and embedded-style things, so you can have some confidence that this actually works. And the C standard requires you to use these headers to use some features of the C language (variable arguments in this specific case). You can of course mimic whatever some implementation does and hope that will work on some other implementations, and on future versions of that first implementation as well. But that is imo not a good idea, just using the standard headers (which are required, and are there on all implementations I have heard of, without bugs) is a much simpler idea, much more future-proof, much less maintenance. > Nobody cares about things like <stdatomic.h> They are completely > irrelevant for the kernel, exactly because we've always just done our > own, or used __builtin_xyz() for things. The Linux kernel needs (or wants) stronger primitives than provided there, yeah, so you could not even use it as the underlying implementation for the semantics the kernel wants (which it could do with <stdint.h> to implement u64 etc.) > <stdarg.h> is the only actual core header file that we have always > taken from the installed system headers - because historically it > really was required. Everything else we can just do ourselves. You also need <stdint.h> and <stddef.h>. These are much simpler to implement of course, but see for example the c46bbf5d2def commit I mentioned before. Some of these compiler headers might have been buggy (or non-existent) historically, but that is very long ago, you really can expect at least C99 to be there (and work correctly) nowadays :-) Segher ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-04 19:15 ` Segher Boessenkool @ 2021-09-04 19:58 ` Linus Torvalds 2021-09-06 15:46 ` Segher Boessenkool 0 siblings, 1 reply; 27+ messages in thread From: Linus Torvalds @ 2021-09-04 19:58 UTC (permalink / raw) To: Segher Boessenkool Cc: Florian Weimer, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Sat, Sep 4, 2021 at 12:18 PM Segher Boessenkool <segher@kernel.crashing.org> wrote: > > And the C standard requires you to use these headers to use some > features of the C language That's really the point here: we don't care AT ALL. The C standard headers are simply not relevant for the kernel. Never have been, never will be. We care about the low-level compiler infrastructure, not the standard C headers. Those standards are designed for - and try to cater to - a completely different audience. We do atomics, and we do not care at all about the mis-designed C++ standard atomics. We do our own type system, and again, we don't care at all about the C "official" type system and odd name rules that change from one version to the other. That has always been the case. We generally *cannot* use the system header files, because they bring in things that the kernel simply cannot handle. That's entirely obvious for things like <stdio.h>, but it's actually true even for other things. > You also need <stdint.h> and <stddef.h>. No, you really don't. We avoid those intentionally, and always have. Because the system header files have never been a good match. Now, <stdarg.h> is actually special for the kernel, exactly because unlike other header files, that one really ended up being something that historically wasn't exposed in any other way and wasn't doable sanely inside the kernel. But it does look like gcc and clang have both ended up exposing the interfaces the same way, using the __buildin_xyz model to just wrap the standard names in a namespace-clean way. That really wasn't historically true. Back in the bad old days of varargs etc, you literally had "va_list" be a "char *" and the accessor functions were magic macros that updated things by hand by the size of the arguments etc... So <stdarg.h> is historically very very special, and tied to compiler implementation details. In ways that a lot of other "standard C header files" very much are *not* - many of those are about the types that the system provides. And in fact often the kernel is the *source* and definition of parts of them. Linus ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-04 19:58 ` Linus Torvalds @ 2021-09-06 15:46 ` Segher Boessenkool 2021-09-06 16:30 ` Linus Torvalds 0 siblings, 1 reply; 27+ messages in thread From: Segher Boessenkool @ 2021-09-06 15:46 UTC (permalink / raw) To: Linus Torvalds Cc: Florian Weimer, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Sat, Sep 04, 2021 at 12:58:50PM -0700, Linus Torvalds wrote: > On Sat, Sep 4, 2021 at 12:18 PM Segher Boessenkool > <segher@kernel.crashing.org> wrote: > > And the C standard requires you to use these headers to use some > > features of the C language > > That's really the point here: we don't care AT ALL. > > The C standard headers are simply not relevant for the kernel. Never > have been, never will be. Yes they are. It is the single standard way to get the functionality you want. These are not library headers. These headers are provided by the compiler itself. (The one exception is limits.h, but you have no use for that in the kernel anyway). > We care about the low-level compiler infrastructure, not the standard C headers. I don't know why you think you can separate that. Take <stdarg.h> -- there is no other (portable, standard) way to implement receiving variadic arguments. Yes you can mimic what GCC currently does in its <stdarg.h>, and hope that keeps working on later versions of GCC, and on all older compiler versions you care about, and that it works on other compilers as well. > Those standards are designed for - and try to cater to - a completely > different audience. """ This International Standard specifies the form and establishes the interpretation of programs expressed in the programming language C. Its purpose is to promote portability, reliability, maintainability, and efficient execution of C language programs on a variety of computing systems. """ Is the kernel not written in C? The C standard is not the POSIX standard, or anything like it. > We do our own type system, and again, we don't care at all about the C > "official" type system and odd name rules that change from one version > to the other. <stdint.h> has existed since C99, and has not relevantly changed since. And you *do* care about this feature, deeply. That is why the kernel has u8, for the exact same reason standard C has uint8_t: to get exact width integer types. The kernel could just include the standard header and then typedef uint8_t u8; instead of the dance it currently goes through. > We generally *cannot* use the system header files, because they bring > in things that the kernel simply cannot handle. Like? These header files are part of the compiler. The compiler is required to provide these to be a C compiler. Without using those header files there is no portable way to use some important features of C. These are not library headers. These are not headers provided by some third party. These headers are an intricate part of the compiler itself. > > You also need <stdint.h> and <stddef.h>. > > No, you really don't. > > We avoid those intentionally, and always have. Because the system > header files have never been a good match. Linux re-implements all of it. It *does* need it, it just has its own implementation if it. It needs the features it provides. > But it does look like gcc and clang have both ended up exposing the > interfaces the same way, using the __buildin_xyz model to just wrap > the standard names in a namespace-clean way. > > That really wasn't historically true. Back in the bad old days of > varargs etc, you literally had "va_list" be a "char *" and the > accessor functions were magic macros that updated things by hand by > the size of the arguments etc... Before C90 there was <varargs.h> instead, and that was like that indeed. The standard C <stdarg.h> header *cannot* be implemented in terms of other building blocks in C. This is the underlying theme of all standard C headers. > In ways that a lot of other "standard C header files" very much are > *not* - many of those are about the types that the system provides. > > And in fact often the kernel is the *source* and definition of parts of them. That is not true at all. These headers in no way depend on Linux or any other OS. Some of it is mandated by the C standard, some of it is mandated by the low-level ABI used (the size of an int for example, that kind of thing). But none of it is from any kernel. I get that long ago in the dark ages Linux had to implement many things from scratch. And that implementing things from scratch is more fun anyway. Also, everything seems to work now, tinkering with it is not necessarily the best idea. But that does not mean that if you had to start over you should not use the standard C headers, or that it would be a bad idea to use them more often now. Segher ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-06 15:46 ` Segher Boessenkool @ 2021-09-06 16:30 ` Linus Torvalds 2021-09-06 17:27 ` Segher Boessenkool 0 siblings, 1 reply; 27+ messages in thread From: Linus Torvalds @ 2021-09-06 16:30 UTC (permalink / raw) To: Segher Boessenkool Cc: Florian Weimer, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Mon, Sep 6, 2021 at 8:50 AM Segher Boessenkool <segher@kernel.crashing.org> wrote: > > > Yes they are. It is the single standard way to get the functionality > you want. Sorry, but 'standard' is the part that the kernel simply doesn't do. Just face it. The kernel isn't written with all those standards that are designed for entirely different things in mind. > I don't know why you think you can separate that. Take <stdarg.h> -- > there is no other (portable, standard) way to implement receiving > variadic arguments. You clearly haven't actually read the thread. That was my whole argument. For the _historical_ situation. Linus ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-06 16:30 ` Linus Torvalds @ 2021-09-06 17:27 ` Segher Boessenkool 2021-09-06 18:11 ` Linus Torvalds 0 siblings, 1 reply; 27+ messages in thread From: Segher Boessenkool @ 2021-09-06 17:27 UTC (permalink / raw) To: Linus Torvalds Cc: Florian Weimer, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Mon, Sep 06, 2021 at 09:30:01AM -0700, Linus Torvalds wrote: > On Mon, Sep 6, 2021 at 8:50 AM Segher Boessenkool > <segher@kernel.crashing.org> wrote: > > Yes they are. It is the single standard way to get the functionality > > you want. > > Sorry, but 'standard' is the part that the kernel simply doesn't do. > > Just face it. The kernel isn't written with all those standards that > are designed for entirely different things in mind. The kernel is written in C, and C is meant in part for implementing things like kernels. When I say "standard C" I mean "without the GNU C extensions". All C compilers implement standard C. Only GCC implements all the GNU C extensions, and implements most of them correctly. > > I don't know why you think you can separate that. Take <stdarg.h> -- > > there is no other (portable, standard) way to implement receiving > > variadic arguments. > > You clearly haven't actually read the thread. ? Are you refering to something more than what is at <https://lore.kernel.org/lkml/CAK7LNAQ0Q6CdXaD-dVGj_e3O3JYs_crpejWKpXHYQJYxyk-1VQ@mail.gmail.com/> ? I did read that. If I draw other conclusions, well, that happens. > That was my whole argument. For the _historical_ situation. But this whole thread is about removing uses of <stdarg.h>, and eventually removing the -nostdinc -isystem $(shell $(CC) -print-file-name=include) thing (except in some places where it is next to impossible to reimplement the compiler's implementation (instead of just very hard and useless work, making all this more fragile in the process)). That is not the past, that is the future. A bleak future, if technical decisions are based on FUD like "compiler headers are dangerous". Segher ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-06 17:27 ` Segher Boessenkool @ 2021-09-06 18:11 ` Linus Torvalds 2021-09-06 18:27 ` Florian Weimer 0 siblings, 1 reply; 27+ messages in thread From: Linus Torvalds @ 2021-09-06 18:11 UTC (permalink / raw) To: Segher Boessenkool Cc: Florian Weimer, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Mon, Sep 6, 2021 at 10:30 AM Segher Boessenkool <segher@kernel.crashing.org> wrote: > > But this whole thread is about removing uses of <stdarg.h>, and > eventually removing the > -nostdinc -isystem $(shell $(CC) -print-file-name=include) Yes. But your argument against it was based on that past argument of it being compiler-specific, in ways that are not true any more. Happily, it clearly is the case that compilers have standardized their internal implementation of this in ways that means that we actually _could_ do this for <stdarg.h>. Exactly the way we do it for pretty much every other header file. IOW, <stdarg.h> is no longer the special thing it used to be. We use the compiler intrinsics without the C library header files for everything else, so doing so for <stdarg.h> seems to actually be a clarification and improvement. Linus ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-06 18:11 ` Linus Torvalds @ 2021-09-06 18:27 ` Florian Weimer 2021-09-06 19:48 ` Segher Boessenkool 0 siblings, 1 reply; 27+ messages in thread From: Florian Weimer @ 2021-09-06 18:27 UTC (permalink / raw) To: Linus Torvalds Cc: Segher Boessenkool, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains * Linus Torvalds: > We use the compiler intrinsics without the C library header files for > everything else, so doing so for <stdarg.h> seems to actually be a > clarification and improvement. This is an exaggeration. On several architectures, the kernel cannot use the vector built-ins directly. Some of the implementing headers are very special and intertwined with the compiler. <stdarg.h> is currently not such a case, but it's just not technically not feasible to avoid dependencies on all compiler headers. I think this considerably weakens the case against <stdarg.h> because the compiler version is so obviously harmless. What the kernel is doing here is imposing an unnecesary constraint on compiler development. Basically, you are telling compiler writers that implementing features with the help of header files is a bad idea because it makes it more difficult to use them from the kernel. (See the proposed exceptions for vector code.) Thanks, Florian ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-06 18:27 ` Florian Weimer @ 2021-09-06 19:48 ` Segher Boessenkool 2021-09-06 20:14 ` Jakub Jelinek 0 siblings, 1 reply; 27+ messages in thread From: Segher Boessenkool @ 2021-09-06 19:48 UTC (permalink / raw) To: Florian Weimer Cc: Linus Torvalds, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Mon, Sep 06, 2021 at 08:27:25PM +0200, Florian Weimer wrote: > * Linus Torvalds: > > > We use the compiler intrinsics without the C library header files for > > everything else, so doing so for <stdarg.h> seems to actually be a > > clarification and improvement. > > This is an exaggeration. On several architectures, the kernel cannot > use the vector built-ins directly. Some of the implementing headers are > very special and intertwined with the compiler. <stdarg.h> is currently > not such a case, but it's just not technically not feasible to avoid > dependencies on all compiler headers. I think this considerably weakens > the case against <stdarg.h> because the compiler version is so obviously > harmless. Exactly Florian. Thank you for so clearly making the point. > What the kernel is doing here is imposing an unnecesary constraint on > compiler development. Basically, you are telling compiler writers that > implementing features with the help of header files is a bad idea > because it makes it more difficult to use them from the kernel. (See > the proposed exceptions for vector code.) Either it will constrain the compiler development, or perhaps more likely, building the kernel will break in ways that the kernel people will blame the compiler developers for. The compiler headers (standard or arch-specific, same reason here) are there because it decouples the user (that doesn't mean "userland", it means the kernel here) from the builtins. Decoupling has many advantages. The most obvious in general is you can use nicer names in a header file, names that can step on the user's toes (like "bool" vs. "_Bool", which is essentially all that <stdbool.h> does). But another huge advantage of decoupling is it allows the compiler more freedom in bugfixing (or any other maintenance / new development). It is low probability that there are bugs in the compiler's standard headers, and it's not likely the kernel's ad-hoc imitation of it has bugs, this is all so small after all (but have I mentioned the c46bbf5d2def commit?) So there is no big pressure for changing anything here. But OTOH it clearly is not a good idea to remove the existing uses of standard headers. No upsides, various downsides, and some of those can be very costly. Segher ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-06 19:48 ` Segher Boessenkool @ 2021-09-06 20:14 ` Jakub Jelinek 2021-09-06 21:08 ` Linus Torvalds 0 siblings, 1 reply; 27+ messages in thread From: Jakub Jelinek @ 2021-09-06 20:14 UTC (permalink / raw) To: Segher Boessenkool Cc: Florian Weimer, Linus Torvalds, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Mon, Sep 06, 2021 at 02:48:08PM -0500, Segher Boessenkool wrote: > > This is an exaggeration. On several architectures, the kernel cannot > > use the vector built-ins directly. Some of the implementing headers are > > very special and intertwined with the compiler. <stdarg.h> is currently > > not such a case, but it's just not technically not feasible to avoid > > dependencies on all compiler headers. I think this considerably weakens > > the case against <stdarg.h> because the compiler version is so obviously > > harmless. > > Exactly Florian. Thank you for so clearly making the point. Yes. While stdarg.h builtins haven't changed for a while (we had some since ~ 1999, changed them incompatibly in 2000 and again in 2002 and not since then and I'm not aware of plans to change them in the near future), e.g. for the backend intrinsics, we change some every year or two, the only guaranteed APIs are those provided by the headers (x86intrin.h/*mmintrin.h etc. on x86, arm_{neon,sve}.h etc. on arm*, ...) and the underlying builtins are sometimes removed, further ones added, etc. stdarg.h is also such a header, although admittedly much simpler and less likely to change. I must say I don't understand the argument about uninstalled compilers, uninstalled compilers work just fine if one supplies the right include path for them, for C it is trivial and apparently the kernel has been already doing that through -nostdinc -isystem .../include, for C++ a little bit harder but we have a script for that. Jakub ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-06 20:14 ` Jakub Jelinek @ 2021-09-06 21:08 ` Linus Torvalds 2021-09-06 21:24 ` Linus Torvalds ` (2 more replies) 0 siblings, 3 replies; 27+ messages in thread From: Linus Torvalds @ 2021-09-06 21:08 UTC (permalink / raw) To: Jakub Jelinek Cc: Segher Boessenkool, Florian Weimer, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Mon, Sep 6, 2021 at 1:14 PM Jakub Jelinek <jakub@redhat.com> wrote: > > the only guaranteed APIs are > those provided by the headers (x86intrin.h/*mmintrin.h etc. on x86, > arm_{neon,sve}.h etc. on arm*, ...) You guys realize we don't use those, do you? And you don't seem to realize that you are actively arguing *AGAINST* what you think you argue for. That "immintrin.h" file, for example, is simply not usable for the kernel. I just checked. Why? Because it ends up doing exactly all those things that MUST NOT be done for the kernel. In file included from /usr/lib/gcc/x86_64-redhat-linux/11/include/xmmintrin.h:34, from /usr/lib/gcc/x86_64-redhat-linux/11/include/immintrin.h:31, from t.c:1: /usr/lib/gcc/x86_64-redhat-linux/11/include/mm_malloc.h:27:10: fatal error: stdlib.h: No such file or directory 27 | #include <stdlib.h> | ^~~~~~~~~~ Oops. And no, it doesn't work trying to include some specific avx2intrin.h file either: /usr/lib/gcc/x86_64-redhat-linux/11/include/avx2intrin.h:25:3: error: #error "Never use <avx2intrin.h> directly; include <immintrin.h> instead." 25 | # error "Never use <avx2intrin.h> directly; include <immintrin.h> instead." | ^~~~~ Very similar things happens if you try to use that <stdint.h> file that somebody mentioned earlier. Guys, you don't understand how limited the kernel header files are - on purpose. You also don't seem to realize how hard it is to separate out the user-land crap that we really cannot use, and must not use. And you think that we're making it harder for compiler people, but that's not at all the case. You really don't want to deal with us saying "you can't do that" when you do something that is Yes, <stdarg.h> has historically worked for us, and it's pretty much the only one. All your arguments about how people need to use the standard headers are basically worthless, because you have never actually tried to use them in a standalone project, have you? So just face it - stdarg.h is special. And it's not clear that there's any reason why the kernel should include the one that comes with the compiler, when the kernel cannot use any of the other header files that come with the compiler anyway. And ALL of your arguments about how we must use compiler header files are COMPLETE GARBAGE, because you didn't even look at them, did you? See? So stop making arguments out of ignorance. Because that's literally what you are doing. You've never tried to make those header files standalone, and you don't have any idea of how nasty it would be if you were forced to. Linus ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-06 21:08 ` Linus Torvalds @ 2021-09-06 21:24 ` Linus Torvalds 2021-09-07 14:41 ` Segher Boessenkool 2021-09-06 21:52 ` Jakub Jelinek 2021-09-07 14:29 ` Segher Boessenkool 2 siblings, 1 reply; 27+ messages in thread From: Linus Torvalds @ 2021-09-06 21:24 UTC (permalink / raw) To: Jakub Jelinek Cc: Segher Boessenkool, Florian Weimer, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Mon, Sep 6, 2021 at 2:08 PM Linus Torvalds <torvalds@linuxfoundation.org> wrote: > > And you think that we're making it harder for compiler people, but > that's not at all the case. > > You really don't want to deal with us saying "you can't do that" when > you do something that is That got cut short when I went off to adding the examples of errors that happen for those intrinsics headers. But it was supposed to be "when you do something that is not valid in the kernel". There are some *very* core header files that the kernel cannot include from outside. That "stdlib.h" thing already came up in the errors I quoted. But I think you'll find that you guys want to include things like <errno.h> too, and you'll probably add others (<types.h>? things like that) simply because they always work fine in user space, and you'd not even notice. Header file include chains get messy very quickly, and very easily. I'm pretty sure you guys don't really want to deal with the pain that is crazy kernel people that have their very bare environment. So you may *think* you want the kernel to use your header files "because compiler portability". Instead, you should be very thankful that we don't, and that you don't have to deal with our mess any more than you already do. Linus ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-06 21:24 ` Linus Torvalds @ 2021-09-07 14:41 ` Segher Boessenkool 0 siblings, 0 replies; 27+ messages in thread From: Segher Boessenkool @ 2021-09-07 14:41 UTC (permalink / raw) To: Linus Torvalds Cc: Jakub Jelinek, Florian Weimer, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Mon, Sep 06, 2021 at 02:24:39PM -0700, Linus Torvalds wrote: > There are some *very* core header files that the kernel cannot include > from outside. That "stdlib.h" thing already came up in the errors I > quoted. > > But I think you'll find that you guys want to include things like > <errno.h> too, and you'll probably add others (<types.h>? things like > that) simply because they always work fine in user space, and you'd > not even notice. Guess what. We actually test this. We do notice. Except we don't, because all those problems do not actually exist. Long ago there were issues. We do not live long ago now. > I'm pretty sure you guys don't really want to deal with the pain that > is crazy kernel people that have their very bare environment. There are many other users that use freestanding environments. Most of them do use the standard headers. > So you may *think* you want the kernel to use your header files > "because compiler portability". Instead, you should be very thankful > that we don't, and that you don't have to deal with our mess any more > than you already do. We would like it to be *less* pain, *less* unnecessary work, that is why we would like the kernel to use the compiler headers. Instead of what the current patches do: getting rid of more of them, which will end up as more work for everyone. Segher ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-06 21:08 ` Linus Torvalds 2021-09-06 21:24 ` Linus Torvalds @ 2021-09-06 21:52 ` Jakub Jelinek 2021-09-06 22:24 ` Linus Torvalds 2021-09-07 14:52 ` Segher Boessenkool 2021-09-07 14:29 ` Segher Boessenkool 2 siblings, 2 replies; 27+ messages in thread From: Jakub Jelinek @ 2021-09-06 21:52 UTC (permalink / raw) To: Linus Torvalds Cc: Segher Boessenkool, Florian Weimer, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Mon, Sep 06, 2021 at 02:08:58PM -0700, Linus Torvalds wrote: > > the only guaranteed APIs are > > those provided by the headers (x86intrin.h/*mmintrin.h etc. on x86, > > arm_{neon,sve}.h etc. on arm*, ...) > > You guys realize we don't use those, do you? The x86intrin.h/*mmintrin.h on x86 indeed not, arm_neon.h is used in the kernel: arch/arm64/include/asm/neon-intrinsics.h:#include <arm_neon.h> arch/arm64/lib/xor-neon.c:#include <asm/neon-intrinsics.h> The arm intrinsics headers are similar to the x86 ones in that the only supported APIs are the ones provided by the headers, the underlying builtins can change. > That "immintrin.h" file, for example, is simply not usable for the > kernel. I just checked. > > Why? Because it ends up doing exactly all those things that MUST NOT > be done for the kernel. > > In file included from > /usr/lib/gcc/x86_64-redhat-linux/11/include/xmmintrin.h:34, > from > /usr/lib/gcc/x86_64-redhat-linux/11/include/immintrin.h:31, > from t.c:1: > /usr/lib/gcc/x86_64-redhat-linux/11/include/mm_malloc.h:27:10: > fatal error: stdlib.h: No such file or directory > 27 | #include <stdlib.h> > | ^~~~~~~~~~ > > Oops. It is actually not that bad, stdlib.h is indeed included there because of 2 intrinsics out of more than 5000 and when one doesn't need those, just #define _MM_MALLOC_H_INCLUDED #include <x86intrin.h> will get rid of the stdlib.h include and those 2 APIs that wouldn't be usable in the kernel anyway. There is a stddef.h include too and that's it (I must say I don't see the reason for that include though). Other compiler provided headers (not talking about C++ now) also have no or very limited includes, including stddef.h, stdarg.h, stdatomic.h, etc. The only exceptions are tgmath.h which isn't usable without libc math.h/complex.h, in some cases stdint.h and limits.h which are in some configurations provided both by the C library and the compiler and include each other in that case (but e.g. stdint.h has an alternate version that only uses compiler provided builtin macros) and openacc.h. Sure, the glibc headers are a different thing. Jakub ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-06 21:52 ` Jakub Jelinek @ 2021-09-06 22:24 ` Linus Torvalds 2021-09-07 15:26 ` Segher Boessenkool 2021-09-07 14:52 ` Segher Boessenkool 1 sibling, 1 reply; 27+ messages in thread From: Linus Torvalds @ 2021-09-06 22:24 UTC (permalink / raw) To: Jakub Jelinek Cc: Segher Boessenkool, Florian Weimer, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Mon, Sep 6, 2021 at 2:52 PM Jakub Jelinek <jakub@redhat.com> wrote: > > It is actually not that bad, stdlib.h is indeed included there because of 2 > intrinsics out of more than 5000 and when one doesn't need those, just > #define _MM_MALLOC_H_INCLUDED .. and on clang? In other words, your suggestion is unworkable, and actively works against the whole point of "use standard headers". It's a joke. And not in a good way. There, I think you have to undefine __STDC_HOSTED__. Maybe by using -ffreestanding? Except if you use -ffreestanding, you lose some very bvasic functionality (*), so we don't do that except for very special code (our 16-bit realmode code does it, for example, and some very low-level library implementations). Just face the facts. Those header files weren't designed for the kernel, and we've done all the special x86 FPU stuff using inline asms. Which is fine. But it all just proves your arguments about "you guys have to use our header files" to be just a pipe dream, and not true. The compiler header files may in some very limited situations be usable. But in the general case? No. And they should very much *not*at*all* be considered some kind of sacred "this is the only way to do things". Because that is clearly not true, and has *never* been true. The usable header files are the odd special case, not the general case. Really. Is it really so hard to just admit that the kernel shouldn't use those headers? When we have 30 years of experience in doing exactly that? Linus (*) iirc, with -ffreestanding gcc doesn't do all the basic memcpy() optimizations. But I forget the exact details. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-06 22:24 ` Linus Torvalds @ 2021-09-07 15:26 ` Segher Boessenkool 0 siblings, 0 replies; 27+ messages in thread From: Segher Boessenkool @ 2021-09-07 15:26 UTC (permalink / raw) To: Linus Torvalds Cc: Jakub Jelinek, Florian Weimer, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Mon, Sep 06, 2021 at 03:24:41PM -0700, Linus Torvalds wrote: > On Mon, Sep 6, 2021 at 2:52 PM Jakub Jelinek <jakub@redhat.com> wrote: > > > > It is actually not that bad, stdlib.h is indeed included there because of 2 > > intrinsics out of more than 5000 and when one doesn't need those, just > > #define _MM_MALLOC_H_INCLUDED > > .. and on clang? Clang apparently has __MM_MALLOC_H as header guard here. But Clang does say #if __STDC_HOSTED__ #include <mm_malloc.h> #endif so they do not have this bug in the first place. GCC should fix this as well. I filed <https://gcc.gnu.org/PR102231>. Thanks for bringing thisd to our attention! > There, I think you have to undefine __STDC_HOSTED__. Maybe by using > -ffreestanding? That defines it to 0, instead, as required by the C standard: $ :|gcc -E -dM -|grep HOSTED #define __STDC_HOSTED__ 1 $ :|gcc -E -dM - -ffreestanding|grep HOSTED #define __STDC_HOSTED__ 0 Yes, that is how this works: the command line flag says how the macro should be defined by the compiler (and changing it in your code is UB btw). > Except if you use -ffreestanding, you lose some very bvasic > functionality (*), (see below) > And they should very much *not*at*all* be considered some kind of > sacred "this is the only way to do things". Because that is clearly > not true, and has *never* been true. Take for example <stdint.h>. There is no other way to find out what types to use for exact-width integers. Unless you really want to do typedef unsigned int __attribute__((mode (DI))) u64; but that is rather unportable to other compilers but GCC. <stdint.h> *is* the only portable way for getting exact-width integers (and some other things). This is not something we decide, it is not something you can decide, it follows directly from the C standard. Using it is easy. Not using it leads to the forest of include files and preprocessor conditionals the kernel currently uses. > The usable header files are the odd special case, not the general case. > > Really. > > Is it really so hard to just admit that the kernel shouldn't use those > headers? When we have 30 years of experience in doing exactly that? That isn't the core issue at hand. Yes, Linux has implemented all of it manually historically, and there even were good reasons for some of that. That does not mean that it would be a good idea to throw out the few standard C headers it does use, and implement those with compiler internal interfaces instead, which we advice you against. > (*) iirc, with -ffreestanding gcc doesn't do all the basic memcpy() > optimizations. But I forget the exact details. I would love to hear any details about that. I do know the Glibc headers do (or did?) some premature optimisation wrt. some mem* and str*, hurting performance on newer compilers. But you should not be using the Glibc headers for compiling the kernel at all anyway, so maybe there was some bug thereabouts? Segher ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-06 21:52 ` Jakub Jelinek 2021-09-06 22:24 ` Linus Torvalds @ 2021-09-07 14:52 ` Segher Boessenkool 2021-09-09 5:14 ` Masahiro Yamada 1 sibling, 1 reply; 27+ messages in thread From: Segher Boessenkool @ 2021-09-07 14:52 UTC (permalink / raw) To: Jakub Jelinek Cc: Linus Torvalds, Florian Weimer, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Mon, Sep 06, 2021 at 11:52:18PM +0200, Jakub Jelinek wrote: > On Mon, Sep 06, 2021 at 02:08:58PM -0700, Linus Torvalds wrote: > There is a stddef.h include too and that's it > (I must say I don't see the reason for that include though). Yeah me neither. Maybe the header used NULL before? > Other compiler provided headers (not talking about C++ now) also have no > or very limited includes, including stddef.h, stdarg.h, stdatomic.h, etc. > The only exceptions are tgmath.h which isn't usable without libc > math.h/complex.h, <tgmath.h> is only for hosted environments. That requires a C library for GCC (we do not implement this stuff ourselves). The compiler and the C library have to work together to get this done, and the relation between GCC and Glibc has been a bit too tight for this, it is true. But a kernel build is not in a hosted environment. > in some cases stdint.h and limits.h which are in some > configurations provided both by the C library and the compiler and include > each other in that case (but e.g. stdint.h has an alternate version that > only uses compiler provided builtin macros) and openacc.h. On what targets is <stdint.h> still problematic? And <limits.h>? Segher ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-07 14:52 ` Segher Boessenkool @ 2021-09-09 5:14 ` Masahiro Yamada 0 siblings, 0 replies; 27+ messages in thread From: Masahiro Yamada @ 2021-09-09 5:14 UTC (permalink / raw) To: Segher Boessenkool Cc: Jakub Jelinek, Linus Torvalds, Florian Weimer, Nathan Chancellor, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Tue, Sep 7, 2021 at 11:54 PM Segher Boessenkool <segher@kernel.crashing.org> wrote: > > On Mon, Sep 06, 2021 at 11:52:18PM +0200, Jakub Jelinek wrote: > > On Mon, Sep 06, 2021 at 02:08:58PM -0700, Linus Torvalds wrote: > > There is a stddef.h include too and that's it > > (I must say I don't see the reason for that include though). > > Yeah me neither. Maybe the header used NULL before? > > > Other compiler provided headers (not talking about C++ now) also have no > > or very limited includes, including stddef.h, stdarg.h, stdatomic.h, etc. > > The only exceptions are tgmath.h which isn't usable without libc > > math.h/complex.h, > > <tgmath.h> is only for hosted environments. That requires a C library > for GCC (we do not implement this stuff ourselves). The compiler and > the C library have to work together to get this done, and the relation > between GCC and Glibc has been a bit too tight for this, it is true. > > But a kernel build is not in a hosted environment. > > > in some cases stdint.h and limits.h which are in some > > configurations provided both by the C library and the compiler and include > > each other in that case (but e.g. stdint.h has an alternate version that > > only uses compiler provided builtin macros) and openacc.h. > > On what targets is <stdint.h> still problematic? And <limits.h>? Since commit 0c79a8e29b5fcbcbfd611daf9d500cfad8370fcf all architectures in the kernel space use the same fixed-width types, which are defined by include/uapi/asm-generic/int-ll64.h So, u32 is always 'unsigned int', and u64 is always 'unsigned long long'. It is convenient for printk() in common code because we can always use the 'll' prefix for u64. u32 foo = 1; u64 bar = 1; printk("foo = %u\n", foo); printk("bar = %llu\n, bar); If we use compiler-provided <stdint.h>, it is not convenient for printk() because uint64_t is 'unsigned long' on some compilers and 'unsigned long long' on others. <intypes.h> provides macros such as PRIx64 uint32_t foo = 1; uint64_t bar = 1; printk("foo = %" PRIu32 "\n", foo); printk("bar = %" PRIu64 "\n", bar); This works, but ends up with ugly code, which I want to avoid. -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-06 21:08 ` Linus Torvalds 2021-09-06 21:24 ` Linus Torvalds 2021-09-06 21:52 ` Jakub Jelinek @ 2021-09-07 14:29 ` Segher Boessenkool 2 siblings, 0 replies; 27+ messages in thread From: Segher Boessenkool @ 2021-09-07 14:29 UTC (permalink / raw) To: Linus Torvalds Cc: Jakub Jelinek, Florian Weimer, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Mon, Sep 06, 2021 at 02:08:58PM -0700, Linus Torvalds wrote: > On Mon, Sep 6, 2021 at 1:14 PM Jakub Jelinek <jakub@redhat.com> wrote: > > > > the only guaranteed APIs are > > those provided by the headers (x86intrin.h/*mmintrin.h etc. on x86, > > arm_{neon,sve}.h etc. on arm*, ...) > > You guys realize we don't use those, do you? Linux does use the Arm and Power vector intrinsics (arm_neon.h and altivec.h). > And you don't seem to realize that you are actively arguing *AGAINST* > what you think you argue for. I have no idea what you think we are arguing for, only what we are. > That "immintrin.h" file, for example, is simply not usable for the > kernel. I just checked. Yes. It cannot be used in freestanding environments; it includes a header file we don't ship (<mm_malloc.h>, via <xmmintrin.h>). Or perhaps we do ship it, but only on native systems? Same issue, anyway. The top comment in the latter says /* Implemented from the specification included in the Intel C++ Compiler User Guide and Reference, version 9.0. */ so the shortcoming exists there already probably? > 27 | #include <stdlib.h> > | ^~~~~~~~~~ > > Oops. Yup, that is not a (freestanding) standard C header. > Very similar things happens if you try to use that <stdint.h> file > that somebody mentioned earlier. Not at all. <stdint.h> *is* a standard C header, and we ship it, it always works. If you find problems with it, please report them! > You also don't seem to realize how hard it is to separate out the > user-land crap that we really cannot use, and must not use. We have worked very hard for many years to make this better. For the freestanding headers only <limits.h> is still problematic, and you do not need that one anyway, it is legacy more than anything else. Segher ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-04 17:22 ` Linus Torvalds 2021-09-04 19:15 ` Segher Boessenkool @ 2021-09-06 6:54 ` Florian Weimer 2021-09-06 16:02 ` Alexey Dobriyan 1 sibling, 1 reply; 27+ messages in thread From: Florian Weimer @ 2021-09-06 6:54 UTC (permalink / raw) To: Alexey Dobriyan Cc: Linus Torvalds, Segher Boessenkool, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains * Linus Torvalds: > On Sat, Sep 4, 2021 at 8:19 AM Florian Weimer <fweimer@redhat.com> wrote: >> >> In any case, it would be nice to know what the real motivation is. > > I don't know about the original motivation, but the reason I like that > patch after-the-fact is that I've actually been in situations where I > test out self-built compilers without installing them. Does this really simplify matters? Why wouldn't the gcc compiler driver find cc1, but not be able to pass the right path options, so that the include/ subdirectory can be located as well? > Then it's convenient to have a completely standalone kernel tree. The final patch in the series is here: isystem: delete global -isystem compile option <https://lore.kernel.org/linux-kernel/YQhY40teUJcTc5H4@localhost.localdomain/> It's still not self-contained. And it seems that there has been quite a bit of fallout from the removal of <stddef.h>. > Nobody cares about things like <stdatomic.h> They are completely > irrelevant for the kernel, exactly because we've always just done our > own, or used __builtin_xyz() for things. Apparently, some people care enough about <stdatomic.h> to prevent its use. I still have not seen an explanation. Maybe it's because we haven't Cc:ed the patch author so far (oops). Alexey, why are <stdatomic.h> and <float.h> so special that you called them out in your patch? If it's about unintended use of libatomic, then maybe we should work on a proper compiler option that also works for __atomic builtins and the _Atomic keyword. Thanks, Florian ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-06 6:54 ` Florian Weimer @ 2021-09-06 16:02 ` Alexey Dobriyan 2021-09-08 3:09 ` Masahiro Yamada 0 siblings, 1 reply; 27+ messages in thread From: Alexey Dobriyan @ 2021-09-06 16:02 UTC (permalink / raw) To: Florian Weimer Cc: Linus Torvalds, Segher Boessenkool, Nathan Chancellor, Masahiro Yamada, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Mon, Sep 06, 2021 at 08:54:13AM +0200, Florian Weimer wrote: > * Linus Torvalds: > > > On Sat, Sep 4, 2021 at 8:19 AM Florian Weimer <fweimer@redhat.com> wrote: > >> > >> In any case, it would be nice to know what the real motivation is. > > > > I don't know about the original motivation, but the reason I like that > > patch after-the-fact is that I've actually been in situations where I > > test out self-built compilers without installing them. > > Does this really simplify matters? Why wouldn't the gcc compiler driver > find cc1, but not be able to pass the right path options, so that the > include/ subdirectory can be located as well? > > > Then it's convenient to have a completely standalone kernel tree. > > The final patch in the series is here: > > isystem: delete global -isystem compile option > <https://lore.kernel.org/linux-kernel/YQhY40teUJcTc5H4@localhost.localdomain/> > > It's still not self-contained. What do you mean? Mainline has 1/3 and 2/3 now: c0891ac15f0428ffa81b2e818d416bdf3cb74ab6 isystem: ship and use stdarg.h 39f75da7bcc829ddc4d40bb60d0e95520de7898b isystem: trim/fixup stdarg.h and other headers 3/3 is stuck in -next: https://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git/log/?h=for-next I'm not sure why. If the patch is bad it should be dropped from -next as well. If it is good, it should be in mainline, otherwise more compile time failures will happen. > And it seems that there has been quite a > bit of fallout from the removal of <stddef.h>. > > > Nobody cares about things like <stdatomic.h> They are completely > > irrelevant for the kernel, exactly because we've always just done our > > own, or used __builtin_xyz() for things. > > Apparently, some people care enough about <stdatomic.h> to prevent its > use. I still have not seen an explanation. Maybe it's because we > haven't Cc:ed the patch author so far (oops). > > Alexey, why are <stdatomic.h> and <float.h> so special that you called > them out in your patch? > > If it's about unintended use of libatomic, then maybe we should work on > a proper compiler option that also works for __atomic builtins and the > _Atomic keyword. stdatomic.h isn't magic really. I looked at what gcc here ships and found these headers. Clearly kernel doesn't want alien stdatomic.h injections because kernel has their own atomic model. Kernel doesn't want any floating point shenanigans either. I think I saw 1 instance of "float" usage but it was harmless (some macro which is converted to an integer at compile time) Kernel doesn't want any future stuff either unless vetted. I can only repeat what I wrote when sending previous versions: kernel clearly isolates itself from userspace, -isystem merely step in the same direction. Other direction (kernel uses what standard says should be available) is fine in principle but it is not my decision to make. And it is more painful, just try to s/u8/uint8_t/g and see what happens. Or, worse, #define and && #define or || Just try it. I also want to note that kernel version are slightly incompatible, but better! * bool should be a macro (module_param(bool) breaks) but it better for everyone if it is a typedef, * true and false should be macros, but they look better in preprocessor output if they are enum. * SHRT_MAX is of type "int", which is silly because typeof(short) != typeof(SHRT_MAX) Practice of many trivial headers is in general worse for compile times, because open/read/parse/close can't be faster than global -Dnoreturn=_Noreturn ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [GIT PULL v2] Kbuild updates for v5.15-rc1 2021-09-06 16:02 ` Alexey Dobriyan @ 2021-09-08 3:09 ` Masahiro Yamada 0 siblings, 0 replies; 27+ messages in thread From: Masahiro Yamada @ 2021-09-08 3:09 UTC (permalink / raw) To: Alexey Dobriyan Cc: Florian Weimer, Linus Torvalds, Segher Boessenkool, Nathan Chancellor, Nick Desaulniers, Linux Kbuild mailing list, Linux Kernel Mailing List, clang-built-linux, llvm, linux-toolchains On Tue, Sep 7, 2021 at 1:02 AM Alexey Dobriyan <adobriyan@gmail.com> wrote: > > On Mon, Sep 06, 2021 at 08:54:13AM +0200, Florian Weimer wrote: > > * Linus Torvalds: > > > > > On Sat, Sep 4, 2021 at 8:19 AM Florian Weimer <fweimer@redhat.com> wrote: > > >> > > >> In any case, it would be nice to know what the real motivation is. > > > > > > I don't know about the original motivation, but the reason I like that > > > patch after-the-fact is that I've actually been in situations where I > > > test out self-built compilers without installing them. > > > > Does this really simplify matters? Why wouldn't the gcc compiler driver > > find cc1, but not be able to pass the right path options, so that the > > include/ subdirectory can be located as well? > > > > > Then it's convenient to have a completely standalone kernel tree. > > > > The final patch in the series is here: > > > > isystem: delete global -isystem compile option > > <https://lore.kernel.org/linux-kernel/YQhY40teUJcTc5H4@localhost.localdomain/> > > > > It's still not self-contained. > > What do you mean? > > Mainline has 1/3 and 2/3 now: > > c0891ac15f0428ffa81b2e818d416bdf3cb74ab6 isystem: ship and use stdarg.h > 39f75da7bcc829ddc4d40bb60d0e95520de7898b isystem: trim/fixup stdarg.h and other headers > > 3/3 is stuck in -next: > > https://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git/log/?h=for-next > > I'm not sure why. If the patch is bad it should be dropped from -next > as well. If it is good, it should be in mainline, otherwise more > compile time failures will happen. See https://lore.kernel.org/all/20210906084947.4f65761d@canb.auug.org.au/ Your 3/3 correctly detected a new use of <stddef.h> in the drm tree. Stephen Rothwell pointed it out a long time ago, and fixed it in linux-next. But, the drm maintainers did not fix it in time. I could not fix it either since the bad commit, b97060a99b01b4, was not in my tree. Now it is mainlined, so my plan is to do s/<stddef.h>/<linux/stddef.h>/ in my tree, then include your 3/3 in my second pull request in this MW. -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 27+ messages in thread
end of thread, other threads:[~2021-09-09 5:15 UTC | newest] Thread overview: 27+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <CAK7LNAQ0Q6CdXaD-dVGj_e3O3JYs_crpejWKpXHYQJYxyk-1VQ@mail.gmail.com> [not found] ` <CAHk-=wgoX0pVqNMMOcrhq=nuOfoZB_3qihyHB3y1S8qo=MDs6w@mail.gmail.com> 2021-09-03 23:04 ` [GIT PULL v2] Kbuild updates for v5.15-rc1 Nathan Chancellor 2021-09-04 8:01 ` Florian Weimer 2021-09-04 13:19 ` Segher Boessenkool 2021-09-04 15:19 ` Florian Weimer 2021-09-04 16:19 ` Segher Boessenkool 2021-09-04 17:22 ` Linus Torvalds 2021-09-04 19:15 ` Segher Boessenkool 2021-09-04 19:58 ` Linus Torvalds 2021-09-06 15:46 ` Segher Boessenkool 2021-09-06 16:30 ` Linus Torvalds 2021-09-06 17:27 ` Segher Boessenkool 2021-09-06 18:11 ` Linus Torvalds 2021-09-06 18:27 ` Florian Weimer 2021-09-06 19:48 ` Segher Boessenkool 2021-09-06 20:14 ` Jakub Jelinek 2021-09-06 21:08 ` Linus Torvalds 2021-09-06 21:24 ` Linus Torvalds 2021-09-07 14:41 ` Segher Boessenkool 2021-09-06 21:52 ` Jakub Jelinek 2021-09-06 22:24 ` Linus Torvalds 2021-09-07 15:26 ` Segher Boessenkool 2021-09-07 14:52 ` Segher Boessenkool 2021-09-09 5:14 ` Masahiro Yamada 2021-09-07 14:29 ` Segher Boessenkool 2021-09-06 6:54 ` Florian Weimer 2021-09-06 16:02 ` Alexey Dobriyan 2021-09-08 3:09 ` Masahiro Yamada
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).