[Ksummit-discuss] [TECH TOPIC] tools/Makefile: Fix Many Many problems and inconsistencies

[Ksummit-discuss] [TECH TOPIC] tools/Makefile: Fix Many Many problems and inconsistencies

[Randy Dunlap]

and subdir Makefiles.

Examples:

Use/honor O=outputdir consistently instead of building in <kerneltree>/tools.
(check/compare kernel commit bf35182ffcd00d8b36d56210ffdac110e5624d7d)

Honor MAKEFLAGS (well, they aren't even passed to tools/Makefile AFAICT.
from an execution log:
make LDFLAGS= MAKEFLAGS="" O=/local/lnx/kernel/lnx-47/TOOLS subdir=tools -C ../tools/ all

Use make's "findstring" correctly (see patch below)

There are lots of other problems unless I have just had too much too drink tonight,
so here's the TECH TOPIC:

In a 1.5 hour code crunch session, get a bunch of interested people together to fix
a lot of problems quickly.  Then I will be a guinea pig tester.  :)


Self-nomination: /me.
Others interested:
Arnaldo (acme); Michal Marek.


Thanks.
---

From: Randy Dunlap <rdunlap@infradead.org>

Use GCC make's "findstring findthis,instring" correctly;
i.e., reverse its arguments in tools/scripts/Makefiles.include.

Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
---
 tools/scripts/Makefile.include |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

This never worked, right?  because of the argument ordering
and because MAKEFLAGS=""!

--- lnx-47.orig/tools/scripts/Makefile.include
+++ lnx-47/tools/scripts/Makefile.include
@@ -40,7 +40,7 @@ EXTRA_WARNINGS += -Wundef
 EXTRA_WARNINGS += -Wwrite-strings
 EXTRA_WARNINGS += -Wformat
 
-ifneq ($(findstring $(MAKEFLAGS), w),w)
+ifneq ($(findstring w,$(MAKEFLAGS)),w)
 PRINT_DIR = --no-print-directory
 else
 NO_SUBDIR = :
@@ -58,7 +58,7 @@ descend = \
 QUIET_SUBDIR0  = +$(MAKE) $(COMMAND_O) -C # space to separate -C and subdir
 QUIET_SUBDIR1  =
 
-ifneq ($(findstring $(MAKEFLAGS),s),s)
+ifneq ($(findstring s,$(MAKEFLAGS)),s)
   ifneq ($(V),1)
 	QUIET_CC       = @echo '  CC       '$@;
 	QUIET_CC_FPIC  = @echo '  CC FPIC  '$@;

Re: [Ksummit-discuss] [TECH TOPIC] tools/Makefile: Fix Many Many problems and inconsistencies

[Wangnan (F)]

On 2016/8/2 11:46, Randy Dunlap wrote:
> and subdir Makefiles.
>
> Examples:
>
> Use/honor O=outputdir consistently instead of building in <kerneltree>/tools.
> (check/compare kernel commit bf35182ffcd00d8b36d56210ffdac110e5624d7d)
>
> Honor MAKEFLAGS (well, they aren't even passed to tools/Makefile AFAICT.
> from an execution log:
> make LDFLAGS= MAKEFLAGS="" O=/local/lnx/kernel/lnx-47/TOOLS subdir=tools -C ../tools/ all
>
> Use make's "findstring" correctly (see patch below)
>
> There are lots of other problems unless I have just had too much too drink tonight,
> so here's the TECH TOPIC:
>
> In a 1.5 hour code crunch session, get a bunch of interested people together to fix
> a lot of problems quickly.  Then I will be a guinea pig tester.  :)
>
>
> Self-nomination: /me.
> Others interested:
> Arnaldo (acme); Michal Marek.

Please count me in. I worked in tools/Makefile and tools/perf/Makefile, 
solved
some problems to make it work for my cross-compiling environment. 'O=' 
is important
to me.

Thank you.

Re: [Ksummit-discuss] [TECH TOPIC] tools/Makefile: Fix Many Many problems and inconsistencies

[Josh Triplett]

On Mon, Aug 01, 2016 at 08:46:05PM -0700, Randy Dunlap wrote:
> and subdir Makefiles.
> 
> Examples:
> 
> Use/honor O=outputdir consistently instead of building in <kerneltree>/tools.
> (check/compare kernel commit bf35182ffcd00d8b36d56210ffdac110e5624d7d)
> 
> Honor MAKEFLAGS (well, they aren't even passed to tools/Makefile AFAICT.
> from an execution log:
> make LDFLAGS= MAKEFLAGS="" O=/local/lnx/kernel/lnx-47/TOOLS subdir=tools -C ../tools/ all
> 
> Use make's "findstring" correctly (see patch below)
> 
> There are lots of other problems unless I have just had too much too drink tonight,
> so here's the TECH TOPIC:
> 
> In a 1.5 hour code crunch session, get a bunch of interested people together to fix
> a lot of problems quickly.  Then I will be a guinea pig tester.  :)

Another pattern to look for: any instance of
ifeq ($(CONFIG_FOO),y)
where CONFIG_FOO is a tristate symbol should generally become
ifneq ($(CONFIG_FOO),n)

A higher-level item that I think would make a good topic: extracting
Kconfig into a library usable outside the kernel.  Several other
projects use a copied and hacked-up version of Kconfig.  I'd love to see
it more widely used as a build-time configuration system for more
projects, but not by forking it out of the kernel each time.  I've heard
kbuild/kconfig maintainers mention that they're interested in seeing
this done, but don't have the bandwidth to do it.

The thing that makes this a topic, rather than just a thing someone
needs to just do and submit a patch for, is determining the necessary
requirements from both the kernel and other projects, so that Kconfig
doesn't depend on the kernel but that the kernel build system doesn't
get any more complex to support that.

Re: [Ksummit-discuss] [TECH TOPIC] tools/Makefile: Fix Many Many problems and inconsistencies

[Arnd Bergmann]

On Monday, August 1, 2016 10:02:33 PM CEST Josh Triplett wrote:
> On Mon, Aug 01, 2016 at 08:46:05PM -0700, Randy Dunlap wrote:
> > and subdir Makefiles.
> > 
> > Examples:
> > 
> > Use/honor O=outputdir consistently instead of building in <kerneltree>/tools.
> > (check/compare kernel commit bf35182ffcd00d8b36d56210ffdac110e5624d7d)
> > 
> > Honor MAKEFLAGS (well, they aren't even passed to tools/Makefile AFAICT.
> > from an execution log:
> > make LDFLAGS= MAKEFLAGS="" O=/local/lnx/kernel/lnx-47/TOOLS subdir=tools -C ../tools/ all
> > 
> > Use make's "findstring" correctly (see patch below)
> > 
> > There are lots of other problems unless I have just had too much too drink tonight,
> > so here's the TECH TOPIC:
> > 
> > In a 1.5 hour code crunch session, get a bunch of interested people together to fix
> > a lot of problems quickly.  Then I will be a guinea pig tester.  :)

I'd be interested in this too.

> Another pattern to look for: any instance of
> ifeq ($(CONFIG_FOO),y)
> where CONFIG_FOO is a tristate symbol should generally become
> ifneq ($(CONFIG_FOO),n)

I tend to use

ifdef CONFIG_FOO

which seems more straightforward (fewer parentheses). Anything wrong
with that?

	Arnd

Re: [Ksummit-discuss] [TECH TOPIC] tools/Makefile: Fix Many Many problems and inconsistencies

[Josh Triplett]

On Tue, Aug 02, 2016 at 09:31:05AM +0200, Arnd Bergmann wrote:
> On Monday, August 1, 2016 10:02:33 PM CEST Josh Triplett wrote:
> > On Mon, Aug 01, 2016 at 08:46:05PM -0700, Randy Dunlap wrote:
> > > and subdir Makefiles.
> > > 
> > > Examples:
> > > 
> > > Use/honor O=outputdir consistently instead of building in <kerneltree>/tools.
> > > (check/compare kernel commit bf35182ffcd00d8b36d56210ffdac110e5624d7d)
> > > 
> > > Honor MAKEFLAGS (well, they aren't even passed to tools/Makefile AFAICT.
> > > from an execution log:
> > > make LDFLAGS= MAKEFLAGS="" O=/local/lnx/kernel/lnx-47/TOOLS subdir=tools -C ../tools/ all
> > > 
> > > Use make's "findstring" correctly (see patch below)
> > > 
> > > There are lots of other problems unless I have just had too much too drink tonight,
> > > so here's the TECH TOPIC:
> > > 
> > > In a 1.5 hour code crunch session, get a bunch of interested people together to fix
> > > a lot of problems quickly.  Then I will be a guinea pig tester.  :)
> 
> I'd be interested in this too.
> 
> > Another pattern to look for: any instance of
> > ifeq ($(CONFIG_FOO),y)
> > where CONFIG_FOO is a tristate symbol should generally become
> > ifneq ($(CONFIG_FOO),n)
> 
> I tend to use
> 
> ifdef CONFIG_FOO
> 
> which seems more straightforward (fewer parentheses). Anything wrong
> with that?

No, that's fine.  And I had a typo above, as well; I meant to write:
ifneq ($(CONFIG_FOO),)

Re: [Ksummit-discuss] [TECH TOPIC] tools/Makefile: Fix Many Many problems and inconsistencies

[Alexandre Belloni]

On 01/08/2016 at 22:02:33 -0700, Josh Triplett wrote :
> A higher-level item that I think would make a good topic: extracting
> Kconfig into a library usable outside the kernel.  Several other
> projects use a copied and hacked-up version of Kconfig.  I'd love to see
> it more widely used as a build-time configuration system for more
> projects, but not by forking it out of the kernel each time.  I've heard
> kbuild/kconfig maintainers mention that they're interested in seeing
> this done, but don't have the bandwidth to do it.
> 
> The thing that makes this a topic, rather than just a thing someone
> needs to just do and submit a patch for, is determining the necessary
> requirements from both the kernel and other projects, so that Kconfig
> doesn't depend on the kernel but that the kernel build system doesn't
> get any more complex to support that.

Well, what about http://ymorin.is-a-geek.org/projects/kconfig-frontends
by the Kconfig maintainer?


-- 
Alexandre Belloni, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

Re: [Ksummit-discuss] [TECH TOPIC] tools/Makefile: Fix Many Many problems and inconsistencies

[Josh Triplett]

On Tue, Aug 02, 2016 at 10:41:09AM +0200, Alexandre Belloni wrote:
> On 01/08/2016 at 22:02:33 -0700, Josh Triplett wrote :
> > A higher-level item that I think would make a good topic: extracting
> > Kconfig into a library usable outside the kernel.  Several other
> > projects use a copied and hacked-up version of Kconfig.  I'd love to see
> > it more widely used as a build-time configuration system for more
> > projects, but not by forking it out of the kernel each time.  I've heard
> > kbuild/kconfig maintainers mention that they're interested in seeing
> > this done, but don't have the bandwidth to do it.
> > 
> > The thing that makes this a topic, rather than just a thing someone
> > needs to just do and submit a patch for, is determining the necessary
> > requirements from both the kernel and other projects, so that Kconfig
> > doesn't depend on the kernel but that the kernel build system doesn't
> > get any more complex to support that.
> 
> Well, what about http://ymorin.is-a-geek.org/projects/kconfig-frontends
> by the Kconfig maintainer?

Last updated in 2013, and not directly used by Linux, so it's still
effectively a fork, just an attempt to maintain *one* fork instead of
several.

Re: [Ksummit-discuss] [TECH TOPIC] tools/Makefile: Fix Many Many problems and inconsistencies

[Arnaldo Carvalho de Melo]

Em Tue, Aug 02, 2016 at 12:53:23PM +0800, Wangnan (F) escreveu:
> On 2016/8/2 11:46, Randy Dunlap wrote:
> > and subdir Makefiles.

> > Examples:
> > 
> > Use/honor O=outputdir consistently instead of building in <kerneltree>/tools.
> > (check/compare kernel commit bf35182ffcd00d8b36d56210ffdac110e5624d7d)
> > 
> > Honor MAKEFLAGS (well, they aren't even passed to tools/Makefile AFAICT.
> > from an execution log:
> > make LDFLAGS= MAKEFLAGS="" O=/local/lnx/kernel/lnx-47/TOOLS subdir=tools -C ../tools/ all
> > 
> > Use make's "findstring" correctly (see patch below)
> > 
> > There are lots of other problems unless I have just had too much too drink tonight,
> > so here's the TECH TOPIC:
> > 
> > In a 1.5 hour code crunch session, get a bunch of interested people together to fix
> > a lot of problems quickly.  Then I will be a guinea pig tester.  :)
> > 
> > Self-nomination: /me.
> > Others interested:
> > Arnaldo (acme); Michal Marek.
 
> Please count me in. I worked in tools/Makefile and tools/perf/Makefile,
> solved
> some problems to make it work for my cross-compiling environment. 'O=' is
> important
> to me.

I'll be in town the whole week, to attend plumbers, will be happy to
talk about tools/ related issues.

- Arnaldo

Re: [Ksummit-discuss] [TECH TOPIC] tools/Makefile: Fix Many Many problems and inconsistencies

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 1907 bytes --]

[Slowly catching up on ksummit-discuss]

On Mon, 2016-08-01 at 20:46 -0700, Randy Dunlap wrote:
> and subdir Makefiles.
> 
> Examples:
> 
> Use/honor O=outputdir consistently instead of building in <kerneltree>/tools.
> (check/compare kernel commit bf35182ffcd00d8b36d56210ffdac110e5624d7d)
> 
> Honor MAKEFLAGS (well, they aren't even passed to tools/Makefile AFAICT.
> from an execution log:
> make LDFLAGS= MAKEFLAGS="" O=/local/lnx/kernel/lnx-47/TOOLS subdir=tools -C ../tools/ all
> 
> Use make's "findstring" correctly (see patch below)
> 
> There are lots of other problems unless I have just had too much too drink tonight,
> so here's the TECH TOPIC:
> 
> In a 1.5 hour code crunch session, get a bunch of interested people together to fix
> a lot of problems quickly.  Then I will be a guinea pig tester.  :)
[...]

I don't know how much can be done in that time.  I've had some
recurring pains in packaging tools/:

1. Many different build systems
   - Inconsistent support for configuration variables (not just 'O')
   - usbip isn't included in a recursive build, presumably because
     it uses autotools

2. Tools include UAPI headers in one of two ways, neither of which is
   reliable:
   - Assume the current headers are on the system include path
   - Include unprocessed UAPI headers through a relative path

   The right thing to do is to run 'make headers_install' and add
   usr/ to the front of the system include path.  But we'd want a
   way to avoid re-doing that when the UAPI headers haven't changed.

3. Tools frequently fail to build in stable releases (sometimes on
   specific architectures) - seems like tools/ is not covered by CI
   or it's ignored

This last point is more of a core topic though.

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: [Ksummit-discuss] [TECH TOPIC] tools/Makefile: Fix Many Many problems and inconsistencies

[Arnaldo Carvalho de Melo]

Em Sun, Sep 04, 2016 at 10:15:02PM +0100, Ben Hutchings escreveu:
> [Slowly catching up on ksummit-discuss]
> 
> On Mon, 2016-08-01 at 20:46 -0700, Randy Dunlap wrote:
> > and subdir Makefiles.
> > 
> > Examples:
> > 
> > Use/honor O=outputdir consistently instead of building in <kerneltree>/tools.
> > (check/compare kernel commit bf35182ffcd00d8b36d56210ffdac110e5624d7d)
> > 
> > Honor MAKEFLAGS (well, they aren't even passed to tools/Makefile AFAICT.
> > from an execution log:
> > make LDFLAGS= MAKEFLAGS="" O=/local/lnx/kernel/lnx-47/TOOLS subdir=tools -C ../tools/ all
> > 
> > Use make's "findstring" correctly (see patch below)
> > 
> > There are lots of other problems unless I have just had too much too drink tonight,
> > so here's the TECH TOPIC:
> > 
> > In a 1.5 hour code crunch session, get a bunch of interested people together to fix
> > a lot of problems quickly.  Then I will be a guinea pig tester.  :)
> [...]
> 
> I don't know how much can be done in that time.  I've had some
> recurring pains in packaging tools/:
> 
> 1. Many different build systems
>    - Inconsistent support for configuration variables (not just 'O')
>    - usbip isn't included in a recursive build, presumably because
>      it uses autotools

Right, that needs improving, I haven't looked at anything outside
tools/{arch,build,lib,include,objtool,perf}
 
> 2. Tools include UAPI headers in one of two ways, neither of which is
>    reliable:
>    - Assume the current headers are on the system include path
>    - Include unprocessed UAPI headers through a relative path
> 
>    The right thing to do is to run 'make headers_install' and add
>    usr/ to the front of the system include path.  But we'd want a
>    way to avoid re-doing that when the UAPI headers haven't changed.

Again, haven't checked outside the above list of directories, by now,
tools/perf/ doesn't use anything outside tools/, are you talking about
other tools that touch kernel source files outside tools/?
 
> 3. Tools frequently fail to build in stable releases (sometimes on
>    specific architectures) - seems like tools/ is not covered by CI
>    or it's ignored

The list above I've been running over a set of docker image
repositories, now publicly available, each with a set of tags for distro
releases and cross build environments.

https://hub.docker.com/r/acmel/linux-perf-tools-build-alpine     
https://hub.docker.com/r/acmel/linux-perf-tools-build-android-ndk
https://hub.docker.com/r/acmel/linux-perf-tools-build-archlinux  
https://hub.docker.com/r/acmel/linux-perf-tools-build-centos     
https://hub.docker.com/r/acmel/linux-perf-tools-build-debian     
https://hub.docker.com/r/acmel/linux-perf-tools-build-fedora     
https://hub.docker.com/r/acmel/linux-perf-tools-build-mageia     
https://hub.docker.com/r/acmel/linux-perf-tools-build-opensuse   
https://hub.docker.com/r/acmel/linux-perf-tools-build-ubuntu

> This last point is more of a core topic though.
> 
> Ben.
> 
> -- 
> Ben Hutchings
> Once a job is fouled up, anything done to improve it makes it worse.



> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [TECH TOPIC] tools/Makefile: Fix Many Many problems and inconsistencies

[Bamvor Zhang Jian]

On 6 September 2016 at 22:57, Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> Em Sun, Sep 04, 2016 at 10:15:02PM +0100, Ben Hutchings escreveu:
>> [Slowly catching up on ksummit-discuss]
>>
>> On Mon, 2016-08-01 at 20:46 -0700, Randy Dunlap wrote:
>> > and subdir Makefiles.
>> >
>> > Examples:
>> >
>> > Use/honor O=outputdir consistently instead of building in <kerneltree>/tools.
>> > (check/compare kernel commit bf35182ffcd00d8b36d56210ffdac110e5624d7d)
>> >
>> > Honor MAKEFLAGS (well, they aren't even passed to tools/Makefile AFAICT.
>> > from an execution log:
>> > make LDFLAGS= MAKEFLAGS="" O=/local/lnx/kernel/lnx-47/TOOLS subdir=tools -C ../tools/ all
>> >
>> > Use make's "findstring" correctly (see patch below)
>> >
>> > There are lots of other problems unless I have just had too much too drink tonight,
>> > so here's the TECH TOPIC:
>> >
>> > In a 1.5 hour code crunch session, get a bunch of interested people together to fix
>> > a lot of problems quickly.  Then I will be a guinea pig tester.  :)
>> [...]
>>
>> I don't know how much can be done in that time.  I've had some
>> recurring pains in packaging tools/:
>>
>> 1. Many different build systems
>>    - Inconsistent support for configuration variables (not just 'O')
>>    - usbip isn't included in a recursive build, presumably because
>>      it uses autotools
>
>Right, that needs improving, I haven't looked at anything outside
>tools/{arch,build,lib,include,objtool,perf}
I am working on enable O= support for kselftest(tools/testing/selftest).
Kbuild or tools/build/Makefile.include seems too heavy for kselftest.
Those Makefile could easily link lots of objects to single binary,
while in kselftest, almost all the single binary is compiled from one
single source respectively, I could redirect the output directory
through the explicit rules.
If there is a plan to build a generic build system(at least for tools),
I hope kselftest could be counted in. Then I could suspend my work and
waiting for this solutions.
>
>> 2. Tools include UAPI headers in one of two ways, neither of which is
>>    reliable:
>>    - Assume the current headers are on the system include path
>>    - Include unprocessed UAPI headers through a relative path
>>
>>    The right thing to do is to run 'make headers_install' and add
>>    usr/ to the front of the system include path.  But we'd want a
>>    way to avoid re-doing that when the UAPI headers haven't changed.
>
>Again, haven't checked outside the above list of directories, by now,
>tools/perf/ doesn't use anything outside tools/, are you talking about
>other tools that touch kernel source files outside tools/?
E.g. tools/gpio and some testcases in tools/testing/selftest


Regards

Bamvor

>
>> 3. Tools frequently fail to build in stable releases (sometimes on
>>    specific architectures) - seems like tools/ is not covered by CI
>>    or it's ignored
>
> The list above I've been running over a set of docker image
> repositories, now publicly available, each with a set of tags for distro
> releases and cross build environments.
>
> https://hub.docker.com/r/acmel/linux-perf-tools-build-alpine
> https://hub.docker.com/r/acmel/linux-perf-tools-build-android-ndk
> https://hub.docker.com/r/acmel/linux-perf-tools-build-archlinux
> https://hub.docker.com/r/acmel/linux-perf-tools-build-centos
> https://hub.docker.com/r/acmel/linux-perf-tools-build-debian
> https://hub.docker.com/r/acmel/linux-perf-tools-build-fedora
> https://hub.docker.com/r/acmel/linux-perf-tools-build-mageia
> https://hub.docker.com/r/acmel/linux-perf-tools-build-opensuse
> https://hub.docker.com/r/acmel/linux-perf-tools-build-ubuntu
>
>> This last point is more of a core topic though.
>>
>> Ben.
>>
>> --
>> Ben Hutchings
>> Once a job is fouled up, anything done to improve it makes it worse.
>
>
>
>> _______________________________________________
>> Ksummit-discuss mailing list
>> Ksummit-discuss@lists.linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
>
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
>