strange dependancy generation bug?

strange dependancy generation bug?

[Dave Jones]

Recently (about the same time that the V=0 stuff was changed over
in kbuild), I noticed that the dependancy generation stuff seems to be
executed more often.

An example:

(davej@halogen:linux-2.5)$ make arch/i386/kernel/cpu/cpufreq/powernow-k7.o V=1
make -f scripts/Makefile.build obj=scripts
make -f scripts/Makefile.build obj=scripts/genksyms
make -f scripts/Makefile.build obj=arch/i386/kernel/cpu/cpufreq arch/i386/kernel/cpu/cpufreq/powernow-k7.o
  gcc -Wp,-MD,arch/i386/kernel/cpu/cpufreq/.powernow-k7.o.d -D__KERNEL__ -Iinclude -Wall -Wstrict-prototypes -Wno-trigraphs -O2 -fno-strict-aliasing -fno-common -pipe -mpreferred-stack-boundary=2 -march=i686 -Iinclude/asm-i386/mach-default -nostdinc -iwithprefix include    -DKBUILD_BASENAME=powernow_k7 -DKBUILD_MODNAME=powernow_k7 -c -o arch/i386/kernel/cpu/cpufreq/.tmp_powernow-k7.o arch/i386/kernel/cpu/cpufreq/powernow-k7.c
scripts/fixdep arch/i386/kernel/cpu/cpufreq/.powernow-k7.o.d arch/i386/kernel/cpu/cpufreq/powernow-k7.o 'gcc -Wp,-MD,arch/i386/kernel/cpu/cpufreq/.powernow-k7.o.d -D__KERNEL__ -Iinclude -Wall -Wstrict-prototypes -Wno-trigraphs -O2 -fno-strict-aliasing -fno-common -pipe -mpreferred-stack-boundary=2 -march=i686 -Iinclude/asm-i386/mach-default -nostdinc -iwithprefix include    -DKBUILD_BASENAME=powernow_k7 -DKBUILD_MODNAME=powernow_k7 -c -o arch/i386/kernel/cpu/cpufreq/.tmp_powernow-k7.o arch/i386/kernel/cpu/cpufreq/powernow-k7.c' > arch/i386/kernel/cpu/cpufreq/.powernow-k7.o.tmp; rm -f arch/i386/kernel/cpu/cpufreq/.powernow-k7.o.d; mv -f arch/i386/kernel/cpu/cpufreq/.powernow-k7.o.tmp arch/i386/kernel/cpu/cpufreq/.powernow-k7.o.cmd


First question that springs to mind, is why does the fixdep stuff get run _after_ doing the compile ?
Second, is why does this always happen? Even on subsequent builds.

		Dave

Re: strange dependancy generation bug?

[Sam Ravnborg]

On Tue, Jun 03, 2003 at 01:07:42PM +0100, Dave Jones wrote:
> Recently (about the same time that the V=0 stuff was changed over
> in kbuild), I noticed that the dependancy generation stuff seems to be
> executed more often.

> First question that springs to mind, is why does the fixdep stuff
> get run _after_ doing the compile ?

Dependency information is always generated when a file is compiled.
The build system has no knowledge about the actual changes and there can
be changes in dependencies. The dependency generation is done by the
option:
-Wp,-MD,arch/i386/kernel/cpu/cpufreq/.powernow-k7.o.d

This tell the preprocessor to generate a makefile fragment
that includes all dependencies.
But this is not enough - a file must be recompiled if one of the CONFIG_
options used in that particualr file - or any of the dependencies - has
changed.
Therefore the second step is to run the fixdep command.
Fixdep parses all included files, and when it see a CONFIG_* symbol
it generate a dependency on the corresponding file located in
include/config/*

split-config has already updated a hirachy of files from the
active .config. When .config is changed relevant files under
include/config/* are updated. Relevant files represent only changed
config options.

fixdep is called with the commandline as argument. this is used by kbuild
to detect changes in command line parameters. If any flags are changed
this will force a recompile as well.

> Second, is why does this always happen? Even on subsequent builds.
As explained above fixdep needs to be run each time a file is
compiled.

But the real question is why you start seeing the invocation
of fixdep. I do not see it on my machine.
I asssume that you see fixdep invocation even with "make V=0".
This is a bug!

Counting this one I have now three independent reports where
kbuild displayed the invocation of fixdep.

I have tried to narrow down the root cause.
Both users were running debian unstable.
Different shells.
GNU make 3.80

I tried to install GNU make 3.80 - but I still do not see the problem.
What happens is that within Makefile.build there is used multi line
definition, where each new-line causes make to launch a new sub-shell.
The command for the second sub-shell is echoed, even though make is told
not to do so. 

I have a patch to fix this that I will send to Linus tonight.

	Sam

Re: strange dependancy generation bug?

[Ricky Beam]

On Tue, 3 Jun 2003, Sam Ravnborg wrote:
>In make a so-called "canned command sequence" is generated.
>Quote from 'info make':
>
>	On the other hand, prefix characters on the command line that refers
>	to a canned sequence apply to every line in the sequence.  So the rule:
>
>	     frob.out: frob.in
>        	     @$(frobnicate)

(I seriously cannot believe there needs to be a protracted debate about
 what's broken and who's at fault -- make or Makefile.  What we see is
 what we get and it's a 5 second fix.  Beyond that, it's a problem for
 the GNU Make maintainers, not lkml.)

Weither make is using 1 shell or 10 doesn't matter.  Make is the thing
emitting the command to the console as well as feeding the shell.  It's
a matter of one command or more than one command.

The Kbuild system is rather complex.  If you look deep enough, you'll
see where the '@' is being applied.  What is actually happening looks
like this:
	define rule_for_act_of_frobnication
		stuff_that_should_be_one_line
				\
		but_isn't
	endef

	frobnicate = @set -e; \
			$(rule_for_act_of_frobnication);

	frob.out: frob.in
		$(call frobnicate)

There's a big difference in @$(call frobnicate) and $(call frobnicate).
Both ultimately resovle to two line of text.  In the first case, all of
it hidden.  In the second, only the lines explicitly hidden will be hidden.

Make isn't broken.  Makefile.build is.  To me, this was immediately obvious
and confirmed after a few seconds searching for the "@". (It looks like
someone hit [enter] where they shouldn't have.)

--Ricky

Re: strange dependancy generation bug?

[Stig Brautaset]

On Jun 03 2003, Sam wrote:
> On Tue, Jun 03, 2003 at 08:11:54PM +0100, Stig Brautaset wrote:
> > > What happens is that within Makefile.build there is used multi line
> > > definition, where each new-line causes make to launch a new sub-shell.
> > > The command for the second sub-shell is echoed, even though make is told
> > > not to do so. 
> > 
> > I beg to differ. Since make launches a new subshell, the commands in the
> > second subshell is _not_ told to shut up, and thus is echoed. No?
> 
> In make a so-called "canned command sequence" is generated.
> Quote from 'info make':
> 
> 	On the other hand, prefix characters on the command line that refers
> 	to a canned sequence apply to every line in the sequence.  So the rule:
> 
> 	     frob.out: frob.in
>         	     @$(frobnicate)
> 
> 	does not echo _any_ commands.
> 
> 
> In kbuild this is exactly what happens.
> So according to make info the command for the second sub-shell should not
> be echoed.

Indeed. It seem I haven't read the docs carefully enough. I was just
looking at cause and effect -- not how it's _supposed_ to work. ;) 

Here, with two subshells, the latter part is echoed, with one it is not. 
However, in the light of this (to me) new information I'm at a loss as
to what causes this to happen.

Stig
-- 
brautaset.org

Re: strange dependancy generation bug?

[Sam Ravnborg]

On Tue, Jun 03, 2003 at 08:11:54PM +0100, Stig Brautaset wrote:
> > What happens is that within Makefile.build there is used multi line
> > definition, where each new-line causes make to launch a new sub-shell.
> > The command for the second sub-shell is echoed, even though make is told
> > not to do so. 
> 
> I beg to differ. Since make launches a new subshell, the commands in the
> second subshell is _not_ told to shut up, and thus is echoed. No?

In make a so-called "canned command sequence" is generated.
Quote from 'info make':

	On the other hand, prefix characters on the command line that refers
	to a canned sequence apply to every line in the sequence.  So the rule:

	     frob.out: frob.in
        	     @$(frobnicate)

	does not echo _any_ commands.


In kbuild this is exactly what happens.
So according to make info the command for the second sub-shell should not
be echoed.

	Sam

Re: strange dependancy generation bug?

[Stig Brautaset]

On Jun 03 2003, Sam wrote:
... 
> But the real question is why you start seeing the invocation
> of fixdep. I do not see it on my machine.
> I asssume that you see fixdep invocation even with "make V=0".
> This is a bug!
> 
> Counting this one I have now three independent reports where
> kbuild displayed the invocation of fixdep.
> 
> I have tried to narrow down the root cause.
> Both users were running debian unstable.
> Different shells.
> GNU make 3.80
> 
> I tried to install GNU make 3.80 - but I still do not see the problem.
> What happens is that within Makefile.build there is used multi line
> definition, where each new-line causes make to launch a new sub-shell.
> The command for the second sub-shell is echoed, even though make is told
> not to do so. 

I beg to differ. Since make launches a new subshell, the commands in the
second subshell is _not_ told to shut up, and thus is echoed. No?


Stig
-- 
brautaset.org