* s390 depending on cc-options makes it difficult to configure @ 2019-12-09 16:41 Jeremy Cline 2019-12-10 9:01 ` Michal Kubecek 0 siblings, 1 reply; 9+ messages in thread From: Jeremy Cline @ 2019-12-09 16:41 UTC (permalink / raw) To: Heiko Carstens, Vasily Gorbik; +Cc: linux-s390, linux-kernel Hi folks, Commit 5474080a3a0a ("s390/Kconfig: make use of 'depends on cc-option'") makes it difficult to produce an s390 configuration for Fedora and Red Hat kernels. The issue is I have the following configurations: CONFIG_MARCH_Z13=y CONFIG_TUNE_Z14=y # CONFIG_TUNE_DEFAULT is not set When the configuration is prepared on a non-s390x host without a compiler with -march=z* it changes CONFIG_TUNE_DEFAULT to y which, as far as I can tell, leads to a kernel tuned for z13 instead of z14. Fedora and Red Hat build processes produce complete configurations from snippets on any available host in the build infrastructure which very frequently is *not* s390. I did a quick search and couldn't find any other examples of Kconfigs depending on march or mtune compiler flags and it seems like it'd generally problematic for people preparing configurations. Regards, Jeremy ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: s390 depending on cc-options makes it difficult to configure 2019-12-09 16:41 s390 depending on cc-options makes it difficult to configure Jeremy Cline @ 2019-12-10 9:01 ` Michal Kubecek 2019-12-11 17:18 ` Jeremy Cline 0 siblings, 1 reply; 9+ messages in thread From: Michal Kubecek @ 2019-12-10 9:01 UTC (permalink / raw) To: linux-kernel; +Cc: Jeremy Cline, Heiko Carstens, Vasily Gorbik, linux-s390 On Mon, Dec 09, 2019 at 11:41:55AM -0500, Jeremy Cline wrote: > Hi folks, > > Commit 5474080a3a0a ("s390/Kconfig: make use of 'depends on cc-option'") > makes it difficult to produce an s390 configuration for Fedora and Red > Hat kernels. > > The issue is I have the following configurations: > > CONFIG_MARCH_Z13=y > CONFIG_TUNE_Z14=y > # CONFIG_TUNE_DEFAULT is not set > > When the configuration is prepared on a non-s390x host without a > compiler with -march=z* it changes CONFIG_TUNE_DEFAULT to y which, as > far as I can tell, leads to a kernel tuned for z13 instead of z14. > Fedora and Red Hat build processes produce complete configurations from > snippets on any available host in the build infrastructure which very > frequently is *not* s390. We have exactly the same problem. Our developers need to update config files for different architectures and different kernel versions on their machines which are usually x86_64 but that often produces different configs than the real build environment. This is not an issue for upstream development as one usually updates configs on the same system where the build takes place but it's a big problem for distribution maintainers. > I did a quick search and couldn't find any other examples of Kconfigs > depending on march or mtune compiler flags and it seems like it'd > generally problematic for people preparing configurations. There are more issues like this. In general, since 4.17 or 4.18, the resulting config depends on both architecture and compiler version. Earlier, you could simply run "ARCH=... make oldconfig" (or menuconfig) to update configs for all architectures and distribution versions. Today, you need to use the right compiler version (results with e.g. 4.8, 7.4 and 9.2 differ) and architecture. At the moment, I'm working around the issue by using chroot environments with target distributions (e.g. openSUSE Tumbleweed) and set of cross compilers for supported architectures but it's far from perfect and even this way, there are problemantic points, e.g. BPFILTER_UMH which depends on gcc being able to not only compile but also link. IMHO the key problem is that .config mixes configuration with description of build environment. I have an idea of a solution which would consist of - an option to extract "config" options which describe build environment (i.e. their values are determined by running some command, rather than reading from a file or asking user) into a cache file - an option telling "make *config" to use such cache file for these environment "config" options instead of running the test scripts (and probably issue an error if an environment option is missing) This would be perhaps even easier if we also separated the environment descripton into a different file so that we would only need an option telling scripts/kconfig/conf not to update the environment description. I'm not sure if that would be acceptable for upstream, though. Unfortunately I didn't find time to implement this yet. Michal Kubecek ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: s390 depending on cc-options makes it difficult to configure 2019-12-10 9:01 ` Michal Kubecek @ 2019-12-11 17:18 ` Jeremy Cline 2020-02-10 18:49 ` Philipp Rudo 0 siblings, 1 reply; 9+ messages in thread From: Jeremy Cline @ 2019-12-11 17:18 UTC (permalink / raw) To: Michal Kubecek; +Cc: linux-kernel, Heiko Carstens, Vasily Gorbik, linux-s390 On Tue, Dec 10, 2019 at 10:01:08AM +0100, Michal Kubecek wrote: > On Mon, Dec 09, 2019 at 11:41:55AM -0500, Jeremy Cline wrote: > > Hi folks, > > > > Commit 5474080a3a0a ("s390/Kconfig: make use of 'depends on cc-option'") > > makes it difficult to produce an s390 configuration for Fedora and Red > > Hat kernels. > > > > The issue is I have the following configurations: > > > > CONFIG_MARCH_Z13=y > > CONFIG_TUNE_Z14=y > > # CONFIG_TUNE_DEFAULT is not set > > > > When the configuration is prepared on a non-s390x host without a > > compiler with -march=z* it changes CONFIG_TUNE_DEFAULT to y which, as > > far as I can tell, leads to a kernel tuned for z13 instead of z14. > > Fedora and Red Hat build processes produce complete configurations from > > snippets on any available host in the build infrastructure which very > > frequently is *not* s390. > > We have exactly the same problem. Our developers need to update config > files for different architectures and different kernel versions on their > machines which are usually x86_64 but that often produces different > configs than the real build environment. > Glad (or sad?) to hear we're not the only ones hitting this. > This is not an issue for upstream development as one usually updates > configs on the same system where the build takes place but it's a big > problem for distribution maintainers. > > > I did a quick search and couldn't find any other examples of Kconfigs > > depending on march or mtune compiler flags and it seems like it'd > > generally problematic for people preparing configurations. > > There are more issues like this. In general, since 4.17 or 4.18, the > resulting config depends on both architecture and compiler version. > Earlier, you could simply run "ARCH=... make oldconfig" (or menuconfig) > to update configs for all architectures and distribution versions. > Today, you need to use the right compiler version (results with e.g. > 4.8, 7.4 and 9.2 differ) and architecture. > Yeah, that's also troublesome. This is by no means the first problem related to the environment at configuration time, but it the most bothersome to work around (at least for Fedora kernel configuration). > At the moment, I'm working around the issue by using chroot environments > with target distributions (e.g. openSUSE Tumbleweed) and set of cross > compilers for supported architectures but it's far from perfect and even > this way, there are problemantic points, e.g. BPFILTER_UMH which depends > on gcc being able to not only compile but also link. > > IMHO the key problem is that .config mixes configuration with > description of build environment. I have an idea of a solution which > would consist of > > - an option to extract "config" options which describe build > environment (i.e. their values are determined by running some > command, rather than reading from a file or asking user) into > a cache file > - an option telling "make *config" to use such cache file for these > environment "config" options instead of running the test scripts > (and probably issue an error if an environment option is missing) > I agree that the issue is mixing kernel configuration with build environment. I suppose a cache file would work, but it still sounds like a difficult process that is working around that fact that folks are coupling the configuration step with the build step. I would advocate that this patch be reverted and an effort made to not mix build environment checks into the configuration. I'm much happier for the build to fail because the configuration can't be satisfied by the environment than I am for the configuration to quietly change or for the tools to not allow me to make the configuration in the first place. Ideally the tools would warn the user if their environment won't build the configuration, but that's a nice-to-have. - Jeremy ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: s390 depending on cc-options makes it difficult to configure 2019-12-11 17:18 ` Jeremy Cline @ 2020-02-10 18:49 ` Philipp Rudo 2020-02-14 3:31 ` Masahiro Yamada 0 siblings, 1 reply; 9+ messages in thread From: Philipp Rudo @ 2020-02-10 18:49 UTC (permalink / raw) To: Jeremy Cline Cc: Michal Kubecek, linux-kernel, Heiko Carstens, Vasily Gorbik, linux-s390, Masahiro Yamada Hey Jeremy, Hey Michal, sorry for the late response. The mail got lost in the pre-xmas rush... In my opinion the problem goes beyond s390 and the commit you mentioned. So I'm also adding Masahiro as Kconfig maintainer and author of cc-option. On Wed, 11 Dec 2019 12:18:22 -0500 Jeremy Cline <jcline@redhat.com> wrote: > On Tue, Dec 10, 2019 at 10:01:08AM +0100, Michal Kubecek wrote: > > On Mon, Dec 09, 2019 at 11:41:55AM -0500, Jeremy Cline wrote: > > > Hi folks, > > > > > > Commit 5474080a3a0a ("s390/Kconfig: make use of 'depends on cc-option'") > > > makes it difficult to produce an s390 configuration for Fedora and Red > > > Hat kernels. > > > > > > The issue is I have the following configurations: > > > > > > CONFIG_MARCH_Z13=y > > > CONFIG_TUNE_Z14=y > > > # CONFIG_TUNE_DEFAULT is not set > > > > > > When the configuration is prepared on a non-s390x host without a > > > compiler with -march=z* it changes CONFIG_TUNE_DEFAULT to y which, as > > > far as I can tell, leads to a kernel tuned for z13 instead of z14. > > > Fedora and Red Hat build processes produce complete configurations from > > > snippets on any available host in the build infrastructure which very > > > frequently is *not* s390. > > > > We have exactly the same problem. Our developers need to update config > > files for different architectures and different kernel versions on their > > machines which are usually x86_64 but that often produces different > > configs than the real build environment. > > > > This is not an issue for upstream development as one usually updates > > configs on the same system where the build takes place but it's a big > > problem for distribution maintainers. If I recall correct the goal was to avoid trouble with clang, as it does not support all processor types with -march. But yeah, in the original consideration we only thought about upstream development and forgot the distros. > > > I did a quick search and couldn't find any other examples of Kconfigs > > > depending on march or mtune compiler flags and it seems like it'd > > > generally problematic for people preparing configurations. True, but not the whole story. Power and Arm64 use cc-option to check for -mstack-protector*, which do not exist on s390. So you have the same problem when you prepare a config for any of them on s390. Thus simply reverting the commit you mentioned above does not solve the problem but merely hides one symptom. Which also means that the original problem will return over and over again in the future. An other reason why I don't think it makes sens to revert the commit is that it would make cc-option as a whole useless. What's the benefit in having cc-option when you are not allowed to use it? Or less provocative, in which use cases is allowed to use cc-option? > > There are more issues like this. In general, since 4.17 or 4.18, the > > resulting config depends on both architecture and compiler version. > > Earlier, you could simply run "ARCH=... make oldconfig" (or menuconfig) > > to update configs for all architectures and distribution versions. > > Today, you need to use the right compiler version (results with e.g. > > 4.8, 7.4 and 9.2 differ) and architecture. > > > > Yeah, that's also troublesome. This is by no means the first problem > related to the environment at configuration time, but it the most > bothersome to work around (at least for Fedora kernel configuration). > > > At the moment, I'm working around the issue by using chroot environments > > with target distributions (e.g. openSUSE Tumbleweed) and set of cross > > compilers for supported architectures but it's far from perfect and even > > this way, there are problemantic points, e.g. BPFILTER_UMH which depends > > on gcc being able to not only compile but also link. > > > > IMHO the key problem is that .config mixes configuration with > > description of build environment. I have an idea of a solution which > > would consist of > > > > - an option to extract "config" options which describe build > > environment (i.e. their values are determined by running some > > command, rather than reading from a file or asking user) into > > a cache file > > - an option telling "make *config" to use such cache file for these > > environment "config" options instead of running the test scripts > > (and probably issue an error if an environment option is missing) > > > > I agree that the issue is mixing kernel configuration with build > environment. I suppose a cache file would work, but it still sounds like > a difficult process that is working around that fact that folks are > coupling the configuration step with the build step. An other solution would be a "I know better" switch which simply disables cc-option for that run. That would allow the use of cc-option for upstream development and provide a simple way for distros to turn it off. > I would advocate that this patch be reverted and an effort made to not > mix build environment checks into the configuration. I'm much happier > for the build to fail because the configuration can't be satisfied by > the environment than I am for the configuration to quietly change or for > the tools to not allow me to make the configuration in the first place. > Ideally the tools would warn the user if their environment won't build > the configuration, but that's a nice-to-have. I too would prefer to have a warning instead of the config being silently changed. But again, the problem goes beyond what was reported. @Masahiro: What do you think about it? Thanks Philipp ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: s390 depending on cc-options makes it difficult to configure 2020-02-10 18:49 ` Philipp Rudo @ 2020-02-14 3:31 ` Masahiro Yamada 2020-02-14 20:35 ` Jeremy Cline 0 siblings, 1 reply; 9+ messages in thread From: Masahiro Yamada @ 2020-02-14 3:31 UTC (permalink / raw) To: Philipp Rudo Cc: Jeremy Cline, Michal Kubecek, Linux Kernel Mailing List, Heiko Carstens, Vasily Gorbik, linux-s390 [-- Attachment #1: Type: text/plain, Size: 7206 bytes --] Hi. On Tue, Feb 11, 2020 at 3:49 AM Philipp Rudo <prudo@linux.ibm.com> wrote: > > Hey Jeremy, > Hey Michal, > > sorry for the late response. The mail got lost in the pre-xmas rush... > > In my opinion the problem goes beyond s390 and the commit you mentioned. So I'm > also adding Masahiro as Kconfig maintainer and author of cc-option. I did not notice the former discussion. Thanks for CC'ing me. > On Wed, 11 Dec 2019 12:18:22 -0500 > Jeremy Cline <jcline@redhat.com> wrote: > > > On Tue, Dec 10, 2019 at 10:01:08AM +0100, Michal Kubecek wrote: > > > On Mon, Dec 09, 2019 at 11:41:55AM -0500, Jeremy Cline wrote: > > > > Hi folks, > > > > > > > > Commit 5474080a3a0a ("s390/Kconfig: make use of 'depends on cc-option'") > > > > makes it difficult to produce an s390 configuration for Fedora and Red > > > > Hat kernels. > > > > > > > > The issue is I have the following configurations: > > > > > > > > CONFIG_MARCH_Z13=y > > > > CONFIG_TUNE_Z14=y > > > > # CONFIG_TUNE_DEFAULT is not set > > > > > > > > When the configuration is prepared on a non-s390x host without a > > > > compiler with -march=z* it changes CONFIG_TUNE_DEFAULT to y which, as > > > > far as I can tell, leads to a kernel tuned for z13 instead of z14. > > > > Fedora and Red Hat build processes produce complete configurations from > > > > snippets on any available host in the build infrastructure which very > > > > frequently is *not* s390. > > > > > > We have exactly the same problem. Our developers need to update config > > > files for different architectures and different kernel versions on their > > > machines which are usually x86_64 but that often produces different > > > configs than the real build environment. > > > > > > This is not an issue for upstream development as one usually updates > > > configs on the same system where the build takes place but it's a big > > > problem for distribution maintainers. > > If I recall correct the goal was to avoid trouble with clang, as it does not > support all processor types with -march. But yeah, in the original > consideration we only thought about upstream development and forgot the > distros. > > > > I did a quick search and couldn't find any other examples of Kconfigs > > > > depending on march or mtune compiler flags and it seems like it'd > > > > generally problematic for people preparing configurations. > > True, but not the whole story. Power and Arm64 use cc-option to check for > -mstack-protector*, which do not exist on s390. So you have the same problem > when you prepare a config for any of them on s390. Thus simply reverting the > commit you mentioned above does not solve the problem but merely hides one > symptom. Which also means that the original problem will return over and over > again in the future. > > An other reason why I don't think it makes sens to revert the commit is that it > would make cc-option as a whole useless. What's the benefit in having cc-option > when you are not allowed to use it? Or less provocative, in which use cases is > allowed to use cc-option? You are right. Reverting the particular s390 commit is not the solution. > > > There are more issues like this. In general, since 4.17 or 4.18, the > > > resulting config depends on both architecture and compiler version. > > > Earlier, you could simply run "ARCH=... make oldconfig" (or menuconfig) > > > to update configs for all architectures and distribution versions. > > > Today, you need to use the right compiler version (results with e.g. > > > 4.8, 7.4 and 9.2 differ) and architecture. > > > > > > > Yeah, that's also troublesome. This is by no means the first problem > > related to the environment at configuration time, but it the most > > bothersome to work around (at least for Fedora kernel configuration). > > > > > At the moment, I'm working around the issue by using chroot environments > > > with target distributions (e.g. openSUSE Tumbleweed) and set of cross > > > compilers for supported architectures but it's far from perfect and even > > > this way, there are problemantic points, e.g. BPFILTER_UMH which depends > > > on gcc being able to not only compile but also link. > > > > > > IMHO the key problem is that .config mixes configuration with > > > description of build environment. I have an idea of a solution which > > > would consist of > > > > > > - an option to extract "config" options which describe build > > > environment (i.e. their values are determined by running some > > > command, rather than reading from a file or asking user) into > > > a cache file > > > - an option telling "make *config" to use such cache file for these > > > environment "config" options instead of running the test scripts > > > (and probably issue an error if an environment option is missing) > > > > > > > I agree that the issue is mixing kernel configuration with build > > environment. I suppose a cache file would work, but it still sounds like > > a difficult process that is working around that fact that folks are > > coupling the configuration step with the build step. > > An other solution would be a "I know better" switch which simply disables > cc-option for that run. That would allow the use of cc-option for upstream > development and provide a simple way for distros to turn it off. > > > I would advocate that this patch be reverted and an effort made to not > > mix build environment checks into the configuration. I'm much happier > > for the build to fail because the configuration can't be satisfied by > > the environment than I am for the configuration to quietly change or for > > the tools to not allow me to make the configuration in the first place. > > Ideally the tools would warn the user if their environment won't build > > the configuration, but that's a nice-to-have. > > I too would prefer to have a warning instead of the config being silently > changed. But again, the problem goes beyond what was reported. > > @Masahiro: What do you think about it? > > Thanks > Philipp > The problem for Jeremy and Michal is, it is difficult to get a full-feature cross-compiler for every arch. One idea to workaround this is to use a fake script that accepts any flag, and use it as $(CC) in Kconfig. RFC patch is attached. This is not a perfect solution, of course. Evaluating the compiler in the Kconfig stage conceptually has a conflict with the workflow of distro maintainers. I think the only way to solve it completely is, ultimately, go back to pre 4.18 situation. But, I am not sure if upstream people want to do it. At least, Linus was happy to do compiler-tests in Kconfig. I already got several criticism about the new feature in Kconfig because it broke the workflow of distro maintainers. Sorry about that. The idea from Michal, separation of the build environment description, would work too. IIRC, the crosstool-ng project generates some Kconfig files based on the environment. In hindsight, Kconfig did not need to have cc-option but it was how I implemented. I just thought it would be cleaner to put cc-option and the CONFIG option depending on it very close. Anyway, comments to the attachment are appreciated. -- Best Regards Masahiro Yamada [-- Attachment #2: 0001-kbuild-add-dummy-toolchains-to-enable-all-cc-option-.patch --] [-- Type: application/x-patch, Size: 4680 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: s390 depending on cc-options makes it difficult to configure 2020-02-14 3:31 ` Masahiro Yamada @ 2020-02-14 20:35 ` Jeremy Cline 2020-02-16 4:08 ` Masahiro Yamada 0 siblings, 1 reply; 9+ messages in thread From: Jeremy Cline @ 2020-02-14 20:35 UTC (permalink / raw) To: Masahiro Yamada Cc: Philipp Rudo, Michal Kubecek, Linux Kernel Mailing List, Heiko Carstens, Vasily Gorbik, linux-s390 On Fri, Feb 14, 2020 at 12:31:05PM +0900, Masahiro Yamada wrote: > Hi. > > On Tue, Feb 11, 2020 at 3:49 AM Philipp Rudo <prudo@linux.ibm.com> wrote: > > > > Hey Jeremy, > > Hey Michal, > > > > sorry for the late response. The mail got lost in the pre-xmas rush... > > > > In my opinion the problem goes beyond s390 and the commit you mentioned. So I'm > > also adding Masahiro as Kconfig maintainer and author of cc-option. > > > I did not notice the former discussion. > Thanks for CC'ing me. > > > > > > On Wed, 11 Dec 2019 12:18:22 -0500 > > Jeremy Cline <jcline@redhat.com> wrote: > > > > > On Tue, Dec 10, 2019 at 10:01:08AM +0100, Michal Kubecek wrote: > > > > On Mon, Dec 09, 2019 at 11:41:55AM -0500, Jeremy Cline wrote: > > > > > Hi folks, > > > > > > > > > > Commit 5474080a3a0a ("s390/Kconfig: make use of 'depends on cc-option'") > > > > > makes it difficult to produce an s390 configuration for Fedora and Red > > > > > Hat kernels. > > > > > > > > > > The issue is I have the following configurations: > > > > > > > > > > CONFIG_MARCH_Z13=y > > > > > CONFIG_TUNE_Z14=y > > > > > # CONFIG_TUNE_DEFAULT is not set > > > > > > > > > > When the configuration is prepared on a non-s390x host without a > > > > > compiler with -march=z* it changes CONFIG_TUNE_DEFAULT to y which, as > > > > > far as I can tell, leads to a kernel tuned for z13 instead of z14. > > > > > Fedora and Red Hat build processes produce complete configurations from > > > > > snippets on any available host in the build infrastructure which very > > > > > frequently is *not* s390. > > > > > > > > We have exactly the same problem. Our developers need to update config > > > > files for different architectures and different kernel versions on their > > > > machines which are usually x86_64 but that often produces different > > > > configs than the real build environment. > > > > > > > > This is not an issue for upstream development as one usually updates > > > > configs on the same system where the build takes place but it's a big > > > > problem for distribution maintainers. > > > > If I recall correct the goal was to avoid trouble with clang, as it does not > > support all processor types with -march. But yeah, in the original > > consideration we only thought about upstream development and forgot the > > distros. > > > > > I did a quick search and couldn't find any other examples of Kconfigs > > > > > depending on march or mtune compiler flags and it seems like it'd > > > > > generally problematic for people preparing configurations. > > > > True, but not the whole story. Power and Arm64 use cc-option to check for > > -mstack-protector*, which do not exist on s390. So you have the same problem > > when you prepare a config for any of them on s390. Thus simply reverting the > > commit you mentioned above does not solve the problem but merely hides one > > symptom. Which also means that the original problem will return over and over > > again in the future. > > > > An other reason why I don't think it makes sens to revert the commit is that it > > would make cc-option as a whole useless. What's the benefit in having cc-option > > when you are not allowed to use it? Or less provocative, in which use cases is > > allowed to use cc-option? > > > You are right. > Reverting the particular s390 commit is not the solution. > > > > > > There are more issues like this. In general, since 4.17 or 4.18, the > > > > resulting config depends on both architecture and compiler version. > > > > Earlier, you could simply run "ARCH=... make oldconfig" (or menuconfig) > > > > to update configs for all architectures and distribution versions. > > > > Today, you need to use the right compiler version (results with e.g. > > > > 4.8, 7.4 and 9.2 differ) and architecture. > > > > > > > > > > Yeah, that's also troublesome. This is by no means the first problem > > > related to the environment at configuration time, but it the most > > > bothersome to work around (at least for Fedora kernel configuration). > > > > > > > At the moment, I'm working around the issue by using chroot environments > > > > with target distributions (e.g. openSUSE Tumbleweed) and set of cross > > > > compilers for supported architectures but it's far from perfect and even > > > > this way, there are problemantic points, e.g. BPFILTER_UMH which depends > > > > on gcc being able to not only compile but also link. > > > > > > > > IMHO the key problem is that .config mixes configuration with > > > > description of build environment. I have an idea of a solution which > > > > would consist of > > > > > > > > - an option to extract "config" options which describe build > > > > environment (i.e. their values are determined by running some > > > > command, rather than reading from a file or asking user) into > > > > a cache file > > > > - an option telling "make *config" to use such cache file for these > > > > environment "config" options instead of running the test scripts > > > > (and probably issue an error if an environment option is missing) > > > > > > > > > > I agree that the issue is mixing kernel configuration with build > > > environment. I suppose a cache file would work, but it still sounds like > > > a difficult process that is working around that fact that folks are > > > coupling the configuration step with the build step. > > > > An other solution would be a "I know better" switch which simply disables > > cc-option for that run. That would allow the use of cc-option for upstream > > development and provide a simple way for distros to turn it off. > > > > > I would advocate that this patch be reverted and an effort made to not > > > mix build environment checks into the configuration. I'm much happier > > > for the build to fail because the configuration can't be satisfied by > > > the environment than I am for the configuration to quietly change or for > > > the tools to not allow me to make the configuration in the first place. > > > Ideally the tools would warn the user if their environment won't build > > > the configuration, but that's a nice-to-have. > > > > I too would prefer to have a warning instead of the config being silently > > changed. But again, the problem goes beyond what was reported. > > > > @Masahiro: What do you think about it? > > > > Thanks > > Philipp > > > > > The problem for Jeremy and Michal is, > it is difficult to get a full-feature cross-compiler > for every arch. > Indeed. > One idea to workaround this is > to use a fake script that accepts any flag, > and use it as $(CC) in Kconfig. > > RFC patch is attached. > > This is not a perfect solution, of course. > The attached patch doesn't looks like it'd work for what we need, although I wonder if it's easier to just check when cc-options is defined for an environment variable or something and always return y instead of calling out to $(CC) at all. Comes to the same thing, I suppose. > > Evaluating the compiler in the Kconfig stage > conceptually has a conflict with the workflow > of distro maintainers. > > I think the only way to solve it completely is, > ultimately, go back to pre 4.18 situation. > But, I am not sure if upstream people want to do it. > At least, Linus was happy to do compiler-tests > in Kconfig. > > I already got several criticism about the > new feature in Kconfig because it broke the > workflow of distro maintainers. Sorry about that. > No worries, it's a tough balancing act between upstream users and distros. It's not caused me *that* much bother. > > The idea from Michal, separation of the build environment > description, would work too. > IIRC, the crosstool-ng project generates some > Kconfig files based on the environment. > In hindsight, Kconfig did not need to have cc-option > but it was how I implemented. I just thought it would be cleaner to > put cc-option and the CONFIG option depending on it very close. > > Anyway, comments to the attachment are appreciated. > I believe it would solve our problem so from that perspective, it looks good to me. Thanks, Jeremy ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: s390 depending on cc-options makes it difficult to configure 2020-02-14 20:35 ` Jeremy Cline @ 2020-02-16 4:08 ` Masahiro Yamada 2020-02-16 18:57 ` Jeremy Cline 0 siblings, 1 reply; 9+ messages in thread From: Masahiro Yamada @ 2020-02-16 4:08 UTC (permalink / raw) To: Jeremy Cline Cc: Philipp Rudo, Michal Kubecek, Linux Kernel Mailing List, Heiko Carstens, Vasily Gorbik, linux-s390 On Sat, Feb 15, 2020 at 5:35 AM Jeremy Cline <jcline@redhat.com> wrote: > > On Fri, Feb 14, 2020 at 12:31:05PM +0900, Masahiro Yamada wrote: > > Hi. > > > > On Tue, Feb 11, 2020 at 3:49 AM Philipp Rudo <prudo@linux.ibm.com> wrote: > > > > > > Hey Jeremy, > > > Hey Michal, > > > > > > sorry for the late response. The mail got lost in the pre-xmas rush... > > > > > > In my opinion the problem goes beyond s390 and the commit you mentioned. So I'm > > > also adding Masahiro as Kconfig maintainer and author of cc-option. > > > > > > I did not notice the former discussion. > > Thanks for CC'ing me. > > > > > > > > > > > On Wed, 11 Dec 2019 12:18:22 -0500 > > > Jeremy Cline <jcline@redhat.com> wrote: > > > > > > > On Tue, Dec 10, 2019 at 10:01:08AM +0100, Michal Kubecek wrote: > > > > > On Mon, Dec 09, 2019 at 11:41:55AM -0500, Jeremy Cline wrote: > > > > > > Hi folks, > > > > > > > > > > > > Commit 5474080a3a0a ("s390/Kconfig: make use of 'depends on cc-option'") > > > > > > makes it difficult to produce an s390 configuration for Fedora and Red > > > > > > Hat kernels. > > > > > > > > > > > > The issue is I have the following configurations: > > > > > > > > > > > > CONFIG_MARCH_Z13=y > > > > > > CONFIG_TUNE_Z14=y > > > > > > # CONFIG_TUNE_DEFAULT is not set > > > > > > > > > > > > When the configuration is prepared on a non-s390x host without a > > > > > > compiler with -march=z* it changes CONFIG_TUNE_DEFAULT to y which, as > > > > > > far as I can tell, leads to a kernel tuned for z13 instead of z14. > > > > > > Fedora and Red Hat build processes produce complete configurations from > > > > > > snippets on any available host in the build infrastructure which very > > > > > > frequently is *not* s390. > > > > > > > > > > We have exactly the same problem. Our developers need to update config > > > > > files for different architectures and different kernel versions on their > > > > > machines which are usually x86_64 but that often produces different > > > > > configs than the real build environment. > > > > > > > > > > This is not an issue for upstream development as one usually updates > > > > > configs on the same system where the build takes place but it's a big > > > > > problem for distribution maintainers. > > > > > > If I recall correct the goal was to avoid trouble with clang, as it does not > > > support all processor types with -march. But yeah, in the original > > > consideration we only thought about upstream development and forgot the > > > distros. > > > > > > I did a quick search and couldn't find any other examples of Kconfigs > > > > > > depending on march or mtune compiler flags and it seems like it'd > > > > > > generally problematic for people preparing configurations. > > > > > > True, but not the whole story. Power and Arm64 use cc-option to check for > > > -mstack-protector*, which do not exist on s390. So you have the same problem > > > when you prepare a config for any of them on s390. Thus simply reverting the > > > commit you mentioned above does not solve the problem but merely hides one > > > symptom. Which also means that the original problem will return over and over > > > again in the future. > > > > > > An other reason why I don't think it makes sens to revert the commit is that it > > > would make cc-option as a whole useless. What's the benefit in having cc-option > > > when you are not allowed to use it? Or less provocative, in which use cases is > > > allowed to use cc-option? > > > > > > You are right. > > Reverting the particular s390 commit is not the solution. > > > > > > > > > There are more issues like this. In general, since 4.17 or 4.18, the > > > > > resulting config depends on both architecture and compiler version. > > > > > Earlier, you could simply run "ARCH=... make oldconfig" (or menuconfig) > > > > > to update configs for all architectures and distribution versions. > > > > > Today, you need to use the right compiler version (results with e.g. > > > > > 4.8, 7.4 and 9.2 differ) and architecture. > > > > > > > > > > > > > Yeah, that's also troublesome. This is by no means the first problem > > > > related to the environment at configuration time, but it the most > > > > bothersome to work around (at least for Fedora kernel configuration). > > > > > > > > > At the moment, I'm working around the issue by using chroot environments > > > > > with target distributions (e.g. openSUSE Tumbleweed) and set of cross > > > > > compilers for supported architectures but it's far from perfect and even > > > > > this way, there are problemantic points, e.g. BPFILTER_UMH which depends > > > > > on gcc being able to not only compile but also link. > > > > > > > > > > IMHO the key problem is that .config mixes configuration with > > > > > description of build environment. I have an idea of a solution which > > > > > would consist of > > > > > > > > > > - an option to extract "config" options which describe build > > > > > environment (i.e. their values are determined by running some > > > > > command, rather than reading from a file or asking user) into > > > > > a cache file > > > > > - an option telling "make *config" to use such cache file for these > > > > > environment "config" options instead of running the test scripts > > > > > (and probably issue an error if an environment option is missing) > > > > > > > > > > > > > I agree that the issue is mixing kernel configuration with build > > > > environment. I suppose a cache file would work, but it still sounds like > > > > a difficult process that is working around that fact that folks are > > > > coupling the configuration step with the build step. > > > > > > An other solution would be a "I know better" switch which simply disables > > > cc-option for that run. That would allow the use of cc-option for upstream > > > development and provide a simple way for distros to turn it off. > > > > > > > I would advocate that this patch be reverted and an effort made to not > > > > mix build environment checks into the configuration. I'm much happier > > > > for the build to fail because the configuration can't be satisfied by > > > > the environment than I am for the configuration to quietly change or for > > > > the tools to not allow me to make the configuration in the first place. > > > > Ideally the tools would warn the user if their environment won't build > > > > the configuration, but that's a nice-to-have. > > > > > > I too would prefer to have a warning instead of the config being silently > > > changed. But again, the problem goes beyond what was reported. > > > > > > @Masahiro: What do you think about it? > > > > > > Thanks > > > Philipp > > > > > > > > > The problem for Jeremy and Michal is, > > it is difficult to get a full-feature cross-compiler > > for every arch. > > > > Indeed. > > > One idea to workaround this is > > to use a fake script that accepts any flag, > > and use it as $(CC) in Kconfig. > > > > RFC patch is attached. > > > > This is not a perfect solution, of course. > > > > The attached patch doesn't looks like it'd work for what we need, I thought turning all cc-options to y would work for what you need. With this, you can enable CONFIG_MARCH_Z13=y and CONFIG_TUNE_Z14=y instead of CONFIG_TUNE_DEFAULT. If this approach does not work for you, what is your requirement? > although I wonder if it's easier to just check when cc-options is > defined for an environment variable or something and always return y > instead of calling out to $(CC) at all. Comes to the same thing, I > suppose. The macro definition in scripts/Kconfig.include takes precedence over any environment variable. So, if you want to hack it from the environment, you need to change the code somehow. The scripts/dummy-tools/ approach does not change anything for the use-case in upstream. The result is the same, of course. Masahiro Yamada > > > > Evaluating the compiler in the Kconfig stage > > conceptually has a conflict with the workflow > > of distro maintainers. > > > > I think the only way to solve it completely is, > > ultimately, go back to pre 4.18 situation. > > But, I am not sure if upstream people want to do it. > > At least, Linus was happy to do compiler-tests > > in Kconfig. > > > > I already got several criticism about the > > new feature in Kconfig because it broke the > > workflow of distro maintainers. Sorry about that. > > > > No worries, it's a tough balancing act between upstream users and > distros. It's not caused me *that* much bother. > > > > > The idea from Michal, separation of the build environment > > description, would work too. > > IIRC, the crosstool-ng project generates some > > Kconfig files based on the environment. > > In hindsight, Kconfig did not need to have cc-option > > but it was how I implemented. I just thought it would be cleaner to > > put cc-option and the CONFIG option depending on it very close. > > > > Anyway, comments to the attachment are appreciated. > > > > I believe it would solve our problem so from that perspective, it looks > good to me. > > Thanks, > Jeremy > -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: s390 depending on cc-options makes it difficult to configure 2020-02-16 4:08 ` Masahiro Yamada @ 2020-02-16 18:57 ` Jeremy Cline 2020-02-18 9:37 ` Philipp Rudo 0 siblings, 1 reply; 9+ messages in thread From: Jeremy Cline @ 2020-02-16 18:57 UTC (permalink / raw) To: Masahiro Yamada Cc: Philipp Rudo, Michal Kubecek, Linux Kernel Mailing List, Heiko Carstens, Vasily Gorbik, linux-s390 On Sun, Feb 16, 2020 at 01:08:49PM +0900, Masahiro Yamada wrote: > On Sat, Feb 15, 2020 at 5:35 AM Jeremy Cline <jcline@redhat.com> wrote: > > > > On Fri, Feb 14, 2020 at 12:31:05PM +0900, Masahiro Yamada wrote: > > > Hi. > > > > > > On Tue, Feb 11, 2020 at 3:49 AM Philipp Rudo <prudo@linux.ibm.com> wrote: > > > > > > > > Hey Jeremy, > > > > Hey Michal, > > > > > > > > sorry for the late response. The mail got lost in the pre-xmas rush... > > > > > > > > In my opinion the problem goes beyond s390 and the commit you mentioned. So I'm > > > > also adding Masahiro as Kconfig maintainer and author of cc-option. > > > > > > > > > I did not notice the former discussion. > > > Thanks for CC'ing me. > > > > > > > > > > > > > > > > On Wed, 11 Dec 2019 12:18:22 -0500 > > > > Jeremy Cline <jcline@redhat.com> wrote: > > > > > > > > > On Tue, Dec 10, 2019 at 10:01:08AM +0100, Michal Kubecek wrote: > > > > > > On Mon, Dec 09, 2019 at 11:41:55AM -0500, Jeremy Cline wrote: > > > > > > > Hi folks, > > > > > > > > > > > > > > Commit 5474080a3a0a ("s390/Kconfig: make use of 'depends on cc-option'") > > > > > > > makes it difficult to produce an s390 configuration for Fedora and Red > > > > > > > Hat kernels. > > > > > > > > > > > > > > The issue is I have the following configurations: > > > > > > > > > > > > > > CONFIG_MARCH_Z13=y > > > > > > > CONFIG_TUNE_Z14=y > > > > > > > # CONFIG_TUNE_DEFAULT is not set > > > > > > > > > > > > > > When the configuration is prepared on a non-s390x host without a > > > > > > > compiler with -march=z* it changes CONFIG_TUNE_DEFAULT to y which, as > > > > > > > far as I can tell, leads to a kernel tuned for z13 instead of z14. > > > > > > > Fedora and Red Hat build processes produce complete configurations from > > > > > > > snippets on any available host in the build infrastructure which very > > > > > > > frequently is *not* s390. > > > > > > > > > > > > We have exactly the same problem. Our developers need to update config > > > > > > files for different architectures and different kernel versions on their > > > > > > machines which are usually x86_64 but that often produces different > > > > > > configs than the real build environment. > > > > > > > > > > > > This is not an issue for upstream development as one usually updates > > > > > > configs on the same system where the build takes place but it's a big > > > > > > problem for distribution maintainers. > > > > > > > > If I recall correct the goal was to avoid trouble with clang, as it does not > > > > support all processor types with -march. But yeah, in the original > > > > consideration we only thought about upstream development and forgot the > > > > distros. > > > > > > > I did a quick search and couldn't find any other examples of Kconfigs > > > > > > > depending on march or mtune compiler flags and it seems like it'd > > > > > > > generally problematic for people preparing configurations. > > > > > > > > True, but not the whole story. Power and Arm64 use cc-option to check for > > > > -mstack-protector*, which do not exist on s390. So you have the same problem > > > > when you prepare a config for any of them on s390. Thus simply reverting the > > > > commit you mentioned above does not solve the problem but merely hides one > > > > symptom. Which also means that the original problem will return over and over > > > > again in the future. > > > > > > > > An other reason why I don't think it makes sens to revert the commit is that it > > > > would make cc-option as a whole useless. What's the benefit in having cc-option > > > > when you are not allowed to use it? Or less provocative, in which use cases is > > > > allowed to use cc-option? > > > > > > > > > You are right. > > > Reverting the particular s390 commit is not the solution. > > > > > > > > > > > > There are more issues like this. In general, since 4.17 or 4.18, the > > > > > > resulting config depends on both architecture and compiler version. > > > > > > Earlier, you could simply run "ARCH=... make oldconfig" (or menuconfig) > > > > > > to update configs for all architectures and distribution versions. > > > > > > Today, you need to use the right compiler version (results with e.g. > > > > > > 4.8, 7.4 and 9.2 differ) and architecture. > > > > > > > > > > > > > > > > Yeah, that's also troublesome. This is by no means the first problem > > > > > related to the environment at configuration time, but it the most > > > > > bothersome to work around (at least for Fedora kernel configuration). > > > > > > > > > > > At the moment, I'm working around the issue by using chroot environments > > > > > > with target distributions (e.g. openSUSE Tumbleweed) and set of cross > > > > > > compilers for supported architectures but it's far from perfect and even > > > > > > this way, there are problemantic points, e.g. BPFILTER_UMH which depends > > > > > > on gcc being able to not only compile but also link. > > > > > > > > > > > > IMHO the key problem is that .config mixes configuration with > > > > > > description of build environment. I have an idea of a solution which > > > > > > would consist of > > > > > > > > > > > > - an option to extract "config" options which describe build > > > > > > environment (i.e. their values are determined by running some > > > > > > command, rather than reading from a file or asking user) into > > > > > > a cache file > > > > > > - an option telling "make *config" to use such cache file for these > > > > > > environment "config" options instead of running the test scripts > > > > > > (and probably issue an error if an environment option is missing) > > > > > > > > > > > > > > > > I agree that the issue is mixing kernel configuration with build > > > > > environment. I suppose a cache file would work, but it still sounds like > > > > > a difficult process that is working around that fact that folks are > > > > > coupling the configuration step with the build step. > > > > > > > > An other solution would be a "I know better" switch which simply disables > > > > cc-option for that run. That would allow the use of cc-option for upstream > > > > development and provide a simple way for distros to turn it off. > > > > > > > > > I would advocate that this patch be reverted and an effort made to not > > > > > mix build environment checks into the configuration. I'm much happier > > > > > for the build to fail because the configuration can't be satisfied by > > > > > the environment than I am for the configuration to quietly change or for > > > > > the tools to not allow me to make the configuration in the first place. > > > > > Ideally the tools would warn the user if their environment won't build > > > > > the configuration, but that's a nice-to-have. > > > > > > > > I too would prefer to have a warning instead of the config being silently > > > > changed. But again, the problem goes beyond what was reported. > > > > > > > > @Masahiro: What do you think about it? > > > > > > > > Thanks > > > > Philipp > > > > > > > > > > > > > The problem for Jeremy and Michal is, > > > it is difficult to get a full-feature cross-compiler > > > for every arch. > > > > > > > Indeed. > > > > > One idea to workaround this is > > > to use a fake script that accepts any flag, > > > and use it as $(CC) in Kconfig. > > > > > > RFC patch is attached. > > > > > > This is not a perfect solution, of course. > > > > > > > The attached patch doesn't looks like it'd work for what we need, > > I thought turning all cc-options to y would work > for what you need. > > With this, you can enable > CONFIG_MARCH_Z13=y and CONFIG_TUNE_Z14=y > instead of CONFIG_TUNE_DEFAULT. > > If this approach does not work for you, > what is your requirement? > Oof, this was an awful typo. It *would* work for what we need. Sorry for the confusion :(. > > > > although I wonder if it's easier to just check when cc-options is > > defined for an environment variable or something and always return y > > instead of calling out to $(CC) at all. Comes to the same thing, I > > suppose. > > > The macro definition in scripts/Kconfig.include > takes precedence over any environment variable. > > So, if you want to hack it from the environment, > you need to change the code somehow. > > The scripts/dummy-tools/ approach does not change > anything for the use-case in upstream. > > The result is the same, of course. > Indeed. Since I'm not maintaining it I don't have a particularly strong opinion about the approach. Whatever you like most works for me. - Jeremy ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: s390 depending on cc-options makes it difficult to configure 2020-02-16 18:57 ` Jeremy Cline @ 2020-02-18 9:37 ` Philipp Rudo 0 siblings, 0 replies; 9+ messages in thread From: Philipp Rudo @ 2020-02-18 9:37 UTC (permalink / raw) To: Jeremy Cline Cc: Masahiro Yamada, Michal Kubecek, Linux Kernel Mailing List, Heiko Carstens, Vasily Gorbik, linux-s390 Hi Masahiro, On Sun, 16 Feb 2020 13:57:59 -0500 Jeremy Cline <jcline@redhat.com> wrote: > On Sun, Feb 16, 2020 at 01:08:49PM +0900, Masahiro Yamada wrote: > > On Sat, Feb 15, 2020 at 5:35 AM Jeremy Cline <jcline@redhat.com> wrote: > > > > > > On Fri, Feb 14, 2020 at 12:31:05PM +0900, Masahiro Yamada wrote: > > > > Hi. > > > > > > > > On Tue, Feb 11, 2020 at 3:49 AM Philipp Rudo <prudo@linux.ibm.com> wrote: [...] > > > > One idea to workaround this is > > > > to use a fake script that accepts any flag, > > > > and use it as $(CC) in Kconfig. > > > > > > > > RFC patch is attached. > > > > > > > > This is not a perfect solution, of course. I like the approach as it gives both sides what they need without blowing up Kconfig. > > > The attached patch doesn't looks like it'd work for what we need, > > > > I thought turning all cc-options to y would work > > for what you need. > > > > With this, you can enable > > CONFIG_MARCH_Z13=y and CONFIG_TUNE_Z14=y > > instead of CONFIG_TUNE_DEFAULT. > > > > If this approach does not work for you, > > what is your requirement? > > > > Oof, this was an awful typo. It *would* work for what we need. Sorry for > the confusion :(. > > > > > > > > although I wonder if it's easier to just check when cc-options is > > > defined for an environment variable or something and always return y > > > instead of calling out to $(CC) at all. Comes to the same thing, I > > > suppose. > > > > > > The macro definition in scripts/Kconfig.include > > takes precedence over any environment variable. > > > > So, if you want to hack it from the environment, > > you need to change the code somehow. > > > > The scripts/dummy-tools/ approach does not change > > anything for the use-case in upstream. > > > > The result is the same, of course. > > > > Indeed. Since I'm not maintaining it I don't have a particularly strong > opinion about the approach. Whatever you like most works for me. I agree with Jeremy. Just pick what you think is best. Thanks a lot Philipp ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2020-02-18 9:38 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-12-09 16:41 s390 depending on cc-options makes it difficult to configure Jeremy Cline 2019-12-10 9:01 ` Michal Kubecek 2019-12-11 17:18 ` Jeremy Cline 2020-02-10 18:49 ` Philipp Rudo 2020-02-14 3:31 ` Masahiro Yamada 2020-02-14 20:35 ` Jeremy Cline 2020-02-16 4:08 ` Masahiro Yamada 2020-02-16 18:57 ` Jeremy Cline 2020-02-18 9:37 ` Philipp Rudo
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).