[PATCH RFC] build: detect outdated configure outputs

[PATCH RFC] build: detect outdated configure outputs

[Roger Pau Monne]

The Xen build system relies on configure to parse some .in files in
order to do substitutions based on the data gathered from configure.

The main issue with those substitutions done at the configure level is
that make is not able to detect when they go out of date because the
.in file has been modified, and hence it's possible to end up in a
situation where .in files have been modified but the build is using
outdated ones. This is made even worse because the 'clean' targets
don't remove the output of the .in parsing, so doing a typical `make
clean && make` will still use the old files without complaining.
Note that 'clean' not removing the output of the .in transformations
is the right behavior, otherwise Xen would require re-executing the
configure script after each clean.

Attempt to improve the situation by adding a global rule that spot the
outdated files as long as they are properly listed as makefile target
prerequisites.

Ultimately those substitutions should be part of the build phase, not
the configure one.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
RFC because I'm not sure if there's some better way to handle this.
Also I think we would want to make sure all the .in outputs are
properly listed as target prerequisites, or else this won't work.

Also not sure whether this will break some other usage of .in files
I'm not aware.
---
 Config.mk | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/Config.mk b/Config.mk
index a56a971028..a81d384899 100644
--- a/Config.mk
+++ b/Config.mk
@@ -65,6 +65,10 @@ DEPS_RM = $(DEPS) $(DEPS_INCLUDE)
 %.d2: %.d
 	sed "s!\(^\| \)$$PWD/! !" $^ >$@.tmp && mv -f $@.tmp $@
 
+# Make sure the configure generated files are up to date
+%: %.in
+	$(error $@ is outdated, please re-run configure)
+
 include $(XEN_ROOT)/config/$(XEN_OS).mk
 include $(XEN_ROOT)/config/$(XEN_TARGET_ARCH).mk
 
-- 
2.30.1

Re: [PATCH RFC] build: detect outdated configure outputs

[Jan Beulich]

On 11.03.2021 12:46, Roger Pau Monne wrote:
> The Xen build system relies on configure to parse some .in files in
> order to do substitutions based on the data gathered from configure.
> 
> The main issue with those substitutions done at the configure level is
> that make is not able to detect when they go out of date because the
> .in file has been modified, and hence it's possible to end up in a
> situation where .in files have been modified but the build is using
> outdated ones. This is made even worse because the 'clean' targets
> don't remove the output of the .in parsing, so doing a typical `make
> clean && make` will still use the old files without complaining.
> Note that 'clean' not removing the output of the .in transformations
> is the right behavior, otherwise Xen would require re-executing the
> configure script after each clean.
> 
> Attempt to improve the situation by adding a global rule that spot the
> outdated files as long as they are properly listed as makefile target
> prerequisites.
> 
> Ultimately those substitutions should be part of the build phase, not
> the configure one.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> ---
> RFC because I'm not sure if there's some better way to handle this.
> Also I think we would want to make sure all the .in outputs are
> properly listed as target prerequisites, or else this won't work.
> 
> Also not sure whether this will break some other usage of .in files
> I'm not aware.

There are a number of such files in the tree which aren't used to
record configure results. Whether their existence could actually
case a problem with this approach I can't tell. Would it be
possible to ...

> --- a/Config.mk
> +++ b/Config.mk
> @@ -65,6 +65,10 @@ DEPS_RM = $(DEPS) $(DEPS_INCLUDE)
>  %.d2: %.d
>  	sed "s!\(^\| \)$$PWD/! !" $^ >$@.tmp && mv -f $@.tmp $@
>  
> +# Make sure the configure generated files are up to date
> +%: %.in
> +	$(error $@ is outdated, please re-run configure)

... make this a static pattern rule for just the file names that
are actually processed / produced by configure? Of course it
wouldn't be very nice to have to keep in sync that list and what
the various configure.ac scripts list in AC_CONFIG_FILES() et al.
But not listing the targets explicitly would always risk the rule
to kick in for a file where it's not supposed to apply.

Jan

Re: [PATCH RFC] build: detect outdated configure outputs

[Roger Pau Monné]

On Thu, Mar 11, 2021 at 02:01:53PM +0100, Jan Beulich wrote:
> On 11.03.2021 12:46, Roger Pau Monne wrote:
> > The Xen build system relies on configure to parse some .in files in
> > order to do substitutions based on the data gathered from configure.
> > 
> > The main issue with those substitutions done at the configure level is
> > that make is not able to detect when they go out of date because the
> > .in file has been modified, and hence it's possible to end up in a
> > situation where .in files have been modified but the build is using
> > outdated ones. This is made even worse because the 'clean' targets
> > don't remove the output of the .in parsing, so doing a typical `make
> > clean && make` will still use the old files without complaining.
> > Note that 'clean' not removing the output of the .in transformations
> > is the right behavior, otherwise Xen would require re-executing the
> > configure script after each clean.
> > 
> > Attempt to improve the situation by adding a global rule that spot the
> > outdated files as long as they are properly listed as makefile target
> > prerequisites.
> > 
> > Ultimately those substitutions should be part of the build phase, not
> > the configure one.
> > 
> > Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> > ---
> > RFC because I'm not sure if there's some better way to handle this.
> > Also I think we would want to make sure all the .in outputs are
> > properly listed as target prerequisites, or else this won't work.
> > 
> > Also not sure whether this will break some other usage of .in files
> > I'm not aware.
> 
> There are a number of such files in the tree which aren't used to
> record configure results. Whether their existence could actually
> case a problem with this approach I can't tell. Would it be
> possible to ...

I think having other .in files in the tree is not a problem with the
target I've added, as it would only apply to files that have an .in
pair and are used as prerequisites.

> > --- a/Config.mk
> > +++ b/Config.mk
> > @@ -65,6 +65,10 @@ DEPS_RM = $(DEPS) $(DEPS_INCLUDE)
> >  %.d2: %.d
> >  	sed "s!\(^\| \)$$PWD/! !" $^ >$@.tmp && mv -f $@.tmp $@
> >  
> > +# Make sure the configure generated files are up to date
> > +%: %.in
> > +	$(error $@ is outdated, please re-run configure)
> 
> ... make this a static pattern rule for just the file names that
> are actually processed / produced by configure? Of course it
> wouldn't be very nice to have to keep in sync that list and what
> the various configure.ac scripts list in AC_CONFIG_FILES() et al.
> But not listing the targets explicitly would always risk the rule
> to kick in for a file where it's not supposed to apply.

Yes, I think it's going to be a pain to keep the list updated, as it's
part of two different configure.ac files.

From a quick look there are other .in files used in a similar way by
ocaml, so those targets will override the pattern rule (%: %.in) in
some places, for example in tools/ocaml/libs/xl/Makefile. I think this
should be fine.

Thanks, Roger.