[Xen-devel] [PATCH 0/2] tools/shim: Bodge things harder

[Xen-devel] [PATCH 0/2] tools/shim: Bodge things harder

[Andrew Cooper]

This logic is all terrible.  This series should resolve the reported build
failure, but definitely isn't a "proper" fix.

Andrew Cooper (2):
  tools/shim: Fix race condition creating linkfarm.stamp
  tools/shim: Apply more duct tape to the linkfarm logic

 tools/firmware/xen-dir/Makefile | 27 +++++++++++++++++++++++----
 1 file changed, 23 insertions(+), 4 deletions(-)

-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 1/2] tools/shim: Fix race condition creating linkfarm.stamp

[Andrew Cooper]

In the case the while loop gets interrupted, the target musn't appear as
up-to-date.  The mov $X.tmp $X must be the last action of the rule.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Ian Jackson <Ian.Jackson@citrix.com>
CC: Wei Liu <wl@xen.org>
CC: Jan Beulich <JBeulich@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: George Dunlap <george.dunlap@eu.citrix.com>
CC: Sander Eikelenboom <linux@eikelenboom.it>
---
 tools/firmware/xen-dir/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/firmware/xen-dir/Makefile b/tools/firmware/xen-dir/Makefile
index 697bbbd57b..df3f5a7006 100644
--- a/tools/firmware/xen-dir/Makefile
+++ b/tools/firmware/xen-dir/Makefile
@@ -32,9 +32,9 @@ linkfarm.stamp: $(DEP_DIRS) $(DEP_FILES) FORCE
 		echo $(f) >> linkfarm.stamp.tmp ;)
 	cmp -s linkfarm.stamp.tmp linkfarm.stamp && \
 		rm linkfarm.stamp.tmp || { \
+		cat linkfarm.stamp.tmp | while read f; \
+		  do rm -f "$(D)/$$f"; ln -s "$(XEN_ROOT)/$$f" "$(D)/$$f"; done; \
 		mv linkfarm.stamp.tmp linkfarm.stamp; \
-		cat linkfarm.stamp | while read f; \
-		  do rm -f "$(D)/$$f"; ln -s "$(XEN_ROOT)/$$f" "$(D)/$$f"; done \
 		}
 
 # Copy enough of the tree to build the shim hypervisor
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH 2/2] tools/shim: Apply more duct tape to the linkfarm logic

[Andrew Cooper]

Sander reported a build failure which manifests as `make; make install`
failing with:

  <snip>/cross-install -m0644 -p xen-dir/xen-shim //usr/local/lib/xen/boot/xen-shim
  install: cannot stat 'xen-dir/xen-shim': No such file or directory
  make[4]: *** [Makefile:52: install] Error 1
  make[4]: Leaving directory '/usr/src/new/xen-unstable/tools/firmware'

It has subsequently been seen intermittently by OSSTest.  This was caused by
c/s 32b1d628 triggering a preexisting linkfarm bug for partial rebuilds.

Between the first `make` and the subsequent `make install`, the linkfarm logic
observes new final build products and regenerates the linkfarm.  This includes
a distclean, which throws away everything from the first `make`.

As the xen-shim rule use a symlink, the link itself remains still up-to-date
but is broken due to the distclean, which causes install to fail.

Update the linkfarm logic to not regenerate itself when build artefacts
appear.  This isn't a comprehensive fix but is the best which can be done
easily.  Any further effort would be better spent making out-of-tree builds
work for Xen.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Ian Jackson <Ian.Jackson@citrix.com>
CC: Wei Liu <wl@xen.org>
CC: Jan Beulich <JBeulich@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: George Dunlap <george.dunlap@eu.citrix.com>
CC: Sander Eikelenboom <linux@eikelenboom.it>
---
 tools/firmware/xen-dir/Makefile | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/tools/firmware/xen-dir/Makefile b/tools/firmware/xen-dir/Makefile
index df3f5a7006..538931e9b4 100644
--- a/tools/firmware/xen-dir/Makefile
+++ b/tools/firmware/xen-dir/Makefile
@@ -14,6 +14,26 @@ LINK_FILES=Config.mk
 DEP_DIRS=$(foreach i, $(LINK_DIRS), $(XEN_ROOT)/$(i))
 DEP_FILES=$(foreach i, $(LINK_FILES), $(XEN_ROOT)/$(i))
 
+# Exclude some intermediate files and final build products
+LINK_EXCLUDES := '*.[isoa]' '.*.d' '.*.d2' '.config'
+LINK_EXCLUDES += '*.map' 'xen' 'xen.gz' 'xen.efi' 'xen-syms'
+
+# This is all a giant mess and doesn't really work.
+#
+# The correct solution is to fix Xen to be able to do out-of-tree builds.
+#
+# Until that happens, we set up a linkfarm by iterating over the xen/ tree,
+# linking source files.  This is repeated each time we enter this directory,
+# which poses a problem for a two-step "make; make install" build process.
+#
+# Any time the list of files to link changes, we relink all files, then
+# distclean to take out not-easy-to-classify intermediate files.  This is to
+# support easy development of the shim, but has a side effect of clobbering
+# the already-built shim.
+#
+# $(LINK_EXCLUDES) should be set such that a parallel build of shim and xen/
+# doesn't cause a subsequent `make install` to decide to regenerate the
+# linkfarm.  This means that all final build artefacts must be excluded.
 linkfarm.stamp: $(DEP_DIRS) $(DEP_FILES) FORCE
 	mkdir -p $(D)
 	rm -f linkfarm.stamp.tmp
@@ -25,8 +45,7 @@ linkfarm.stamp: $(DEP_DIRS) $(DEP_FILES) FORCE
 			sed 's,^$(XEN_ROOT)/$(d)/,,g' | xargs mkdir -p .);) \
 	$(foreach d, $(LINK_DIRS), \
 		(cd $(XEN_ROOT); \
-		 find $(d) ! -type l -type f \
-		 $(addprefix ! -name , '*.[isoa]' '.*.d' '.*.d2')) \
+		 find $(d) ! -type l -type f $(addprefix ! -name ,$(LINK_EXCLUDES))) \
 		 >> linkfarm.stamp.tmp ; ) \
 	$(foreach f, $(LINK_FILES), \
 		echo $(f) >> linkfarm.stamp.tmp ;)
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 1/2] tools/shim: Fix race condition creating linkfarm.stamp

[Ian Jackson]

Andrew Cooper writes ("[PATCH 1/2] tools/shim: Fix race condition creating linkfarm.stamp"):
> In the case the while loop gets interrupted, the target musn't appear as
> up-to-date.  The mov $X.tmp $X must be the last action of the rule.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 2/2] tools/shim: Apply more duct tape to the linkfarm logic

[Ian Jackson]

Andrew Cooper writes ("[PATCH 2/2] tools/shim: Apply more duct tape to the linkfarm logic"):
> Sander reported a build failure which manifests as `make; make install`
> failing with:

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

> Update the linkfarm logic to not regenerate itself when build artefacts
> appear.  This isn't a comprehensive fix but is the best which can be done
> easily.  Any further effort would be better spent making out-of-tree builds
> work for Xen.

Fair enough.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 0/2] tools/shim: Bodge things harder

[Sander Eikelenboom]

On 02/09/2019 18:41, Andrew Cooper wrote:
> This logic is all terrible.  This series should resolve the reported build
> failure, but definitely isn't a "proper" fix.
> 
> Andrew Cooper (2):
>   tools/shim: Fix race condition creating linkfarm.stamp
>   tools/shim: Apply more duct tape to the linkfarm logic
> 
>  tools/firmware/xen-dir/Makefile | 27 +++++++++++++++++++++++----
>  1 file changed, 23 insertions(+), 4 deletions(-)
> 

Thanks Andrew, Just tested and it works for me!

--
Sander

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel