Automatic dependencies are out of sync

Automatic dependencies are out of sync

[Juergen Gross]

I think we have a major problem in our build system regarding
automatic dependencies.

Starting with a new tree (after git clone or make clean) I have
no dependency files (*.d2) anywhere:

$ make clean
$ find . -name '*.d2' | wc -l
0

Doing "make" will produce only some of them in a very limited number of
directories:

$ make
$ find . -name '*.d2' | wc -l
279
$ find . -name '*.d2' | xargs -n 1 dirname | sort -u
./tools/firmware/xen-dir/xen-root/xen/arch/x86
./tools/firmware/xen-dir/xen-root/xen/arch/x86/x86_64
./tools/firmware/xen-dir/xen-root/xen/common
./tools/firmware/xen-dir/xen-root/xen/common/compat
./xen/arch/x86
./xen/arch/x86/x86_64
./xen/common
./xen/common/compat

And only after the next "make" we have all of the *.d2 files available:

$ make
$ find . -name '*.d2' | wc -l
969

I guess the reason for that is that the *.d2 files only depend on the
*.d files which are built together with the *.o files. They are not
needed for building the product files, so they seem to be always one
make step behind.

Now comes the weird part (that's why I started to look into this):
When I now do:

$ touch tools/libxc/include/xenctrl.h
$ make

I get build failures in tools/tests/depriv (lots of unknown types like
uint64_t). The .d file in the depriv directory suddenly contains only
7 instead of 44 entries, while the .d2 file is still okay (it is one
make "older", do you remember?):

$ wc .*.d*
   7   14  546 .depriv-fd-checker.d
  44   88 1923 .depriv-fd-checker.d2

And doing another make will succeed again, but now both dependency files
are cut down to 7 entries (sure they are: .d2 is rebuilt from .d, while
.d would only be rebuilt if .o is being built, which isn't happening as
no file it depends on has changed):

$ make
$ wc .*.d*
  7  14 546 .depriv-fd-checker.d
  7  14 336 .depriv-fd-checker.d2

I have no immediate idea how to solve that.


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Automatic dependencies are out of sync

[Juergen Gross]

On 05/09/18 16:05, Juergen Gross wrote:
> I think we have a major problem in our build system regarding
> automatic dependencies.
> 
> Starting with a new tree (after git clone or make clean) I have
> no dependency files (*.d2) anywhere:
> 
> $ make clean
> $ find . -name '*.d2' | wc -l
> 0
> 
> Doing "make" will produce only some of them in a very limited number of
> directories:
> 
> $ make
> $ find . -name '*.d2' | wc -l
> 279
> $ find . -name '*.d2' | xargs -n 1 dirname | sort -u
> ./tools/firmware/xen-dir/xen-root/xen/arch/x86
> ./tools/firmware/xen-dir/xen-root/xen/arch/x86/x86_64
> ./tools/firmware/xen-dir/xen-root/xen/common
> ./tools/firmware/xen-dir/xen-root/xen/common/compat
> ./xen/arch/x86
> ./xen/arch/x86/x86_64
> ./xen/common
> ./xen/common/compat
> 
> And only after the next "make" we have all of the *.d2 files available:
> 
> $ make
> $ find . -name '*.d2' | wc -l
> 969
> 
> I guess the reason for that is that the *.d2 files only depend on the
> *.d files which are built together with the *.o files. They are not
> needed for building the product files, so they seem to be always one
> make step behind.
> 
> Now comes the weird part (that's why I started to look into this):
> When I now do:
> 
> $ touch tools/libxc/include/xenctrl.h
> $ make
> 
> I get build failures in tools/tests/depriv (lots of unknown types like
> uint64_t). The .d file in the depriv directory suddenly contains only
> 7 instead of 44 entries, while the .d2 file is still okay (it is one
> make "older", do you remember?):
> 
> $ wc .*.d*
>    7   14  546 .depriv-fd-checker.d
>   44   88 1923 .depriv-fd-checker.d2
> 
> And doing another make will succeed again, but now both dependency files
> are cut down to 7 entries (sure they are: .d2 is rebuilt from .d, while
> .d would only be rebuilt if .o is being built, which isn't happening as
> no file it depends on has changed):
> 
> $ make
> $ wc .*.d*
>   7  14 546 .depriv-fd-checker.d
>   7  14 336 .depriv-fd-checker.d2
> 
> I have no immediate idea how to solve that.

I've setup a little example Makefile solving the problem (just to show
the correct dependencies, needs to be adapted for naming the .d and .d2
files and how to build the .d2):

-->8 snip here 8<--

DEPS := tst.d2

all: tst $(DEPS)

%.d2: %.d
        cp $< $@

%.o %.d: %.c
        gcc -MMD -o $(patsubst %.c,%.o,$<) -c $<

%: %.o
        gcc $< -o $@

-include $(DEPS)

-->8 snip here 8<--

So the basic ideas are:

- add a rule for constructing the .d files
- let the build depend on the .d2 files

I hope I didn't miss anything.

For cases like xen/arch/x86/mm/guest_walk_*.o obviously some special
handling is needed, but this should be rather straightforward.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Automatic dependencies are out of sync

[Jan Beulich]

>>> On 06.09.18 at 09:34, <jgross@suse.com> wrote:
> On 05/09/18 16:05, Juergen Gross wrote:
>> I think we have a major problem in our build system regarding
>> automatic dependencies.
>> 
>> Starting with a new tree (after git clone or make clean) I have
>> no dependency files (*.d2) anywhere:
>> 
>> $ make clean
>> $ find . -name '*.d2' | wc -l
>> 0
>> 
>> Doing "make" will produce only some of them in a very limited number of
>> directories:
>> 
>> $ make
>> $ find . -name '*.d2' | wc -l
>> 279
>> $ find . -name '*.d2' | xargs -n 1 dirname | sort -u
>> ./tools/firmware/xen-dir/xen-root/xen/arch/x86
>> ./tools/firmware/xen-dir/xen-root/xen/arch/x86/x86_64
>> ./tools/firmware/xen-dir/xen-root/xen/common
>> ./tools/firmware/xen-dir/xen-root/xen/common/compat
>> ./xen/arch/x86
>> ./xen/arch/x86/x86_64
>> ./xen/common
>> ./xen/common/compat
>> 
>> And only after the next "make" we have all of the *.d2 files available:
>> 
>> $ make
>> $ find . -name '*.d2' | wc -l
>> 969
>> 
>> I guess the reason for that is that the *.d2 files only depend on the
>> *.d files which are built together with the *.o files. They are not
>> needed for building the product files, so they seem to be always one
>> make step behind.
>> 
>> Now comes the weird part (that's why I started to look into this):
>> When I now do:
>> 
>> $ touch tools/libxc/include/xenctrl.h
>> $ make
>> 
>> I get build failures in tools/tests/depriv (lots of unknown types like
>> uint64_t). The .d file in the depriv directory suddenly contains only
>> 7 instead of 44 entries, while the .d2 file is still okay (it is one
>> make "older", do you remember?):
>> 
>> $ wc .*.d*
>>    7   14  546 .depriv-fd-checker.d
>>   44   88 1923 .depriv-fd-checker.d2
>> 
>> And doing another make will succeed again, but now both dependency files
>> are cut down to 7 entries (sure they are: .d2 is rebuilt from .d, while
>> .d would only be rebuilt if .o is being built, which isn't happening as
>> no file it depends on has changed):
>> 
>> $ make
>> $ wc .*.d*
>>   7  14 546 .depriv-fd-checker.d
>>   7  14 336 .depriv-fd-checker.d2
>> 
>> I have no immediate idea how to solve that.
> 
> I've setup a little example Makefile solving the problem (just to show
> the correct dependencies, needs to be adapted for naming the .d and .d2
> files and how to build the .d2):
> 
> -->8 snip here 8<--
> 
> DEPS := tst.d2
> 
> all: tst $(DEPS)

-include $(DEPS) already ought to have the effect of such a dependency,
since all makefiles are checked for rules of how to re-make them.

> %.d2: %.d
>         cp $< $@

Such a rule already exists in ./Config.mk.

> %.o %.d: %.c
>         gcc -MMD -o $(patsubst %.c,%.o,$<) -c $<

Doesn't this result in gcc to be invoked twice, perhaps resulting in
corrupt .o and/or .d? I think %.d wants to depend on %.o, without
a command.

> %: %.o
>         gcc $< -o $@
> 
> -include $(DEPS)
> 
> -->8 snip here 8<--
> 
> So the basic ideas are:
> 
> - add a rule for constructing the .d files
> - let the build depend on the .d2 files

IOW I wonder whether this really is any different from what we
do now (minus bugs/quirks in make itself, of course). And from this
as well as your original mail I still don't understand what's actually
broken with the current approach.

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Automatic dependencies are out of sync

[Juergen Gross]

On 06/09/18 10:10, Jan Beulich wrote:
>>>> On 06.09.18 at 09:34, <jgross@suse.com> wrote:
>> On 05/09/18 16:05, Juergen Gross wrote:
>>> I think we have a major problem in our build system regarding
>>> automatic dependencies.
>>>
>>> Starting with a new tree (after git clone or make clean) I have
>>> no dependency files (*.d2) anywhere:
>>>
>>> $ make clean
>>> $ find . -name '*.d2' | wc -l
>>> 0
>>>
>>> Doing "make" will produce only some of them in a very limited number of
>>> directories:
>>>
>>> $ make
>>> $ find . -name '*.d2' | wc -l
>>> 279
>>> $ find . -name '*.d2' | xargs -n 1 dirname | sort -u
>>> ./tools/firmware/xen-dir/xen-root/xen/arch/x86
>>> ./tools/firmware/xen-dir/xen-root/xen/arch/x86/x86_64
>>> ./tools/firmware/xen-dir/xen-root/xen/common
>>> ./tools/firmware/xen-dir/xen-root/xen/common/compat
>>> ./xen/arch/x86
>>> ./xen/arch/x86/x86_64
>>> ./xen/common
>>> ./xen/common/compat
>>>
>>> And only after the next "make" we have all of the *.d2 files available:
>>>
>>> $ make
>>> $ find . -name '*.d2' | wc -l
>>> 969
>>>
>>> I guess the reason for that is that the *.d2 files only depend on the
>>> *.d files which are built together with the *.o files. They are not
>>> needed for building the product files, so they seem to be always one
>>> make step behind.
>>>
>>> Now comes the weird part (that's why I started to look into this):
>>> When I now do:
>>>
>>> $ touch tools/libxc/include/xenctrl.h
>>> $ make
>>>
>>> I get build failures in tools/tests/depriv (lots of unknown types like
>>> uint64_t). The .d file in the depriv directory suddenly contains only
>>> 7 instead of 44 entries, while the .d2 file is still okay (it is one
>>> make "older", do you remember?):
>>>
>>> $ wc .*.d*
>>>    7   14  546 .depriv-fd-checker.d
>>>   44   88 1923 .depriv-fd-checker.d2
>>>
>>> And doing another make will succeed again, but now both dependency files
>>> are cut down to 7 entries (sure they are: .d2 is rebuilt from .d, while
>>> .d would only be rebuilt if .o is being built, which isn't happening as
>>> no file it depends on has changed):
>>>
>>> $ make
>>> $ wc .*.d*
>>>   7  14 546 .depriv-fd-checker.d
>>>   7  14 336 .depriv-fd-checker.d2
>>>
>>> I have no immediate idea how to solve that.
>>
>> I've setup a little example Makefile solving the problem (just to show
>> the correct dependencies, needs to be adapted for naming the .d and .d2
>> files and how to build the .d2):
>>
>> -->8 snip here 8<--
>>
>> DEPS := tst.d2
>>
>> all: tst $(DEPS)
> 
> -include $(DEPS) already ought to have the effect of such a dependency,
> since all makefiles are checked for rules of how to re-make them.

Obviously this isn't the case. Otherwise there would be .d2 files more
common after doing a make.

> 
>> %.d2: %.d
>>         cp $< $@
> 
> Such a rule already exists in ./Config.mk.

Right, that was just needed in my little test Makefile.

> 
>> %.o %.d: %.c
>>         gcc -MMD -o $(patsubst %.c,%.o,$<) -c $<
> 
> Doesn't this result in gcc to be invoked twice, perhaps resulting in
> corrupt .o and/or .d? I think %.d wants to depend on %.o, without
> a command.

No, that's perfectly fine. make will invoke the command only once, its
just not clear for which target (that's the reason I need to use the
$(patsubst %.c,%.o,$<) instead of $@, which might be the .o _or_ the .d
file).

From the make docs:

  Pattern rules may have more than one target. Unlike normal rules, this
  does not act as many different rules with the same prerequisites and
  recipe. If a pattern rule has multiple targets, make knows that the
  rule’s recipe is responsible for making all of the targets. The recipe
  is executed only once to make all the targets.

> 
>> %: %.o
>>         gcc $< -o $@
>>
>> -include $(DEPS)
>>
>> -->8 snip here 8<--
>>
>> So the basic ideas are:
>>
>> - add a rule for constructing the .d files
>> - let the build depend on the .d2 files
> 
> IOW I wonder whether this really is any different from what we
> do now (minus bugs/quirks in make itself, of course). And from this
> as well as your original mail I still don't understand what's actually
> broken with the current approach.

The main problem is that the .d2 files used for determining which object
files need to be (re-)built are based on the build before the last one.
I'm not sure this is always the case, but at least when starting with a
clean tree I need two invocations of "make" to get all .d2 files built.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Automatic dependencies are out of sync

[Jan Beulich]

>>> On 06.09.18 at 10:27, <jgross@suse.com> wrote:
> On 06/09/18 10:10, Jan Beulich wrote:
>>>>> On 06.09.18 at 09:34, <jgross@suse.com> wrote:
>>> I've setup a little example Makefile solving the problem (just to show
>>> the correct dependencies, needs to be adapted for naming the .d and .d2
>>> files and how to build the .d2):
>>>
>>> -->8 snip here 8<--
>>>
>>> DEPS := tst.d2
>>>
>>> all: tst $(DEPS)
>> 
>> -include $(DEPS) already ought to have the effect of such a dependency,
>> since all makefiles are checked for rules of how to re-make them.
> 
> Obviously this isn't the case. Otherwise there would be .d2 files more
> common after doing a make.

Well, be this (mis?)behavior is what needs explaining first.

>>> %.o %.d: %.c
>>>         gcc -MMD -o $(patsubst %.c,%.o,$<) -c $<
>> 
>> Doesn't this result in gcc to be invoked twice, perhaps resulting in
>> corrupt .o and/or .d? I think %.d wants to depend on %.o, without
>> a command.
> 
> No, that's perfectly fine. make will invoke the command only once, its
> just not clear for which target (that's the reason I need to use the
> $(patsubst %.c,%.o,$<) instead of $@, which might be the .o _or_ the .d
> file).
> 
> From the make docs:
> 
>   Pattern rules may have more than one target. Unlike normal rules, this
>   does not act as many different rules with the same prerequisites and
>   recipe. If a pattern rule has multiple targets, make knows that the
>   rule’s recipe is responsible for making all of the targets. The recipe
>   is executed only once to make all the targets.

Oh, right. But then there's no need to play games - just use $*.

>>> %: %.o
>>>         gcc $< -o $@
>>>
>>> -include $(DEPS)
>>>
>>> -->8 snip here 8<--
>>>
>>> So the basic ideas are:
>>>
>>> - add a rule for constructing the .d files
>>> - let the build depend on the .d2 files
>> 
>> IOW I wonder whether this really is any different from what we
>> do now (minus bugs/quirks in make itself, of course). And from this
>> as well as your original mail I still don't understand what's actually
>> broken with the current approach.
> 
> The main problem is that the .d2 files used for determining which object
> files need to be (re-)built are based on the build before the last one.
> I'm not sure this is always the case, but at least when starting with a
> clean tree I need two invocations of "make" to get all .d2 files built.

But that's correct: They're not needed _until_ a rebuild happens.
And by way of make's rebuilding of makefiles (if there are suitable
rules) they should appear _before_ any .o gets rebuilt, and even
before make evaluates which ones need rebuilding.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Automatic dependencies are out of sync

[Andrew Cooper]

On 06/09/18 09:27, Juergen Gross wrote:
> On 06/09/18 10:10, Jan Beulich wrote:
>
>>> %: %.o
>>>         gcc $< -o $@
>>>
>>> -include $(DEPS)
>>>
>>> -->8 snip here 8<--
>>>
>>> So the basic ideas are:
>>>
>>> - add a rule for constructing the .d files
>>> - let the build depend on the .d2 files
>> IOW I wonder whether this really is any different from what we
>> do now (minus bugs/quirks in make itself, of course). And from this
>> as well as your original mail I still don't understand what's actually
>> broken with the current approach.
> The main problem is that the .d2 files used for determining which object
> files need to be (re-)built are based on the build before the last one.
> I'm not sure this is always the case, but at least when starting with a
> clean tree I need two invocations of "make" to get all .d2 files built.

As an alternative, can we see about fixing the real bug and getting rid
of .d2 all together?

I still revert that change for some of my builds, because it completely
breaks the incremental build (by causing most things to be rebuilt).

Whatever the underlying issue is, fixing it by sed'ing the dependency
files is gross hack, rather than a fix.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Automatic dependencies are out of sync

[Juergen Gross]

On 06/09/18 12:26, Jan Beulich wrote:
>>>> On 06.09.18 at 10:27, <jgross@suse.com> wrote:
>> On 06/09/18 10:10, Jan Beulich wrote:
>>>>>> On 06.09.18 at 09:34, <jgross@suse.com> wrote:
>>>> I've setup a little example Makefile solving the problem (just to show
>>>> the correct dependencies, needs to be adapted for naming the .d and .d2
>>>> files and how to build the .d2):
>>>>
>>>> -->8 snip here 8<--
>>>>
>>>> DEPS := tst.d2
>>>>
>>>> all: tst $(DEPS)
>>>
>>> -include $(DEPS) already ought to have the effect of such a dependency,
>>> since all makefiles are checked for rules of how to re-make them.
>>
>> Obviously this isn't the case. Otherwise there would be .d2 files more
>> common after doing a make.
> 
> Well, be this (mis?)behavior is what needs explaining first.

In my example Makefile this is working as you are expecting it: even
with removing the dependency of "all" on $(DEPS) everything is fine.

I suspect there are no .d2 files after the first make due to:

DEPS = .*.d
DEPS_INCLUDE = $(addsuffix .d2, $(basename $(wildcard $(DEPS))))

In the beginning there is no .d file, so DEPS_INCLUDE will be empty.

> 
>>>> %.o %.d: %.c
>>>>         gcc -MMD -o $(patsubst %.c,%.o,$<) -c $<
>>>
>>> Doesn't this result in gcc to be invoked twice, perhaps resulting in
>>> corrupt .o and/or .d? I think %.d wants to depend on %.o, without
>>> a command.
>>
>> No, that's perfectly fine. make will invoke the command only once, its
>> just not clear for which target (that's the reason I need to use the
>> $(patsubst %.c,%.o,$<) instead of $@, which might be the .o _or_ the .d
>> file).
>>
>> From the make docs:
>>
>>   Pattern rules may have more than one target. Unlike normal rules, this
>>   does not act as many different rules with the same prerequisites and
>>   recipe. If a pattern rule has multiple targets, make knows that the
>>   rule’s recipe is responsible for making all of the targets. The recipe
>>   is executed only once to make all the targets.
> 
> Oh, right. But then there's no need to play games - just use $*.

Aah, right. Good idea.

> 
>>>> %: %.o
>>>>         gcc $< -o $@
>>>>
>>>> -include $(DEPS)
>>>>
>>>> -->8 snip here 8<--
>>>>
>>>> So the basic ideas are:
>>>>
>>>> - add a rule for constructing the .d files
>>>> - let the build depend on the .d2 files
>>>
>>> IOW I wonder whether this really is any different from what we
>>> do now (minus bugs/quirks in make itself, of course). And from this
>>> as well as your original mail I still don't understand what's actually
>>> broken with the current approach.
>>
>> The main problem is that the .d2 files used for determining which object
>> files need to be (re-)built are based on the build before the last one.
>> I'm not sure this is always the case, but at least when starting with a
>> clean tree I need two invocations of "make" to get all .d2 files built.
> 
> But that's correct: They're not needed _until_ a rebuild happens.
> And by way of make's rebuilding of makefiles (if there are suitable
> rules) they should appear _before_ any .o gets rebuilt, and even
> before make evaluates which ones need rebuilding.

Okay. Then the issue I'm after seems to be a different one.

And now I know what is wrong with the build in tools/tests/depriv:

There are no .o files specified for depriv-fd-checker, so it is built
via:

gcc $(CFLAGS) depriv-fd-checker.c $(LDFLAGS) -o depriv-fd-checker

I'm suspecting the source file is specified via $<, as the resulting
command line after touching one of the headers is:

gcc $(CFLAGS) depriv-fd-checker.c <all-headers-from-d2-file> $(LDFLAGS)
-o depriv-fd-checker

So all files the program is depending on are added into the parameters
of gcc. This is the reason for the build error I'm seeing.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel