[PATCH v3] firmware/shim : filter output files during Xen tree setup

[PATCH v3] firmware/shim : filter output files during Xen tree setup

[Christopher Clark]

Exclude named output files from the Xen tree setup.

The linkfarm.stamp content will differ between top level "make"
and "make install" invocations, due to the introduction of these
output files that are produced during the "make" build.

Filter these out to prevent an unnecessary rebuild of the shim
during "make install", after "make" within a fresh source tree.

Excluded from consideration with this change: differences in stamp
content when performing incremental builds in an existing tree.

Signed-off-by: Christopher Clark <christopher.clark6@baesystems.com>
---
Changes in v3: added '.xen.efi.*' '.xen-syms.*' to the exclude list.

Tested with: Xen 4.10.1, 4.11.0 and staging,
Yocto poky, OpenEmbedded meta-openembedded, meta-virtualization
with binutils 2.3.0 with x86_64-pep target enabled.

 tools/firmware/xen-dir/Makefile | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/tools/firmware/xen-dir/Makefile b/tools/firmware/xen-dir/Makefile
index 84648c3..af789d0 100644
--- a/tools/firmware/xen-dir/Makefile
+++ b/tools/firmware/xen-dir/Makefile
@@ -11,6 +11,11 @@ D=xen-root
 LINK_DIRS=config xen
 LINK_FILES=Config.mk
 
+# Files to exclude from the link farm
+EXCLUDE_FILES=xen xen.gz xen-syms xen-syms.map xen.efi xen.efi.map \
+              efi.lds xen.lds mkelf32 mkreloc \
+              '.xen.efi.*' '.xen-syms.*'
+
 DEP_DIRS=$(foreach i, $(LINK_DIRS), $(XEN_ROOT)/$(i))
 DEP_FILES=$(foreach i, $(LINK_FILES), $(XEN_ROOT)/$(i))
 
@@ -26,7 +31,8 @@ linkfarm.stamp: $(DEP_DIRS) $(DEP_FILES) FORCE
 	$(foreach d, $(LINK_DIRS), \
 		(cd $(XEN_ROOT); \
 		 find $(d) ! -type l -type f \
-		 $(addprefix ! -name , '*.[isoa]' '.*.d' '.*.d2')) \
+		 $(addprefix ! -name , '*.[isoa]' '.*.d' '.*.d2' \
+		                       $(EXCLUDE_FILES) )) \
 		 >> linkfarm.stamp.tmp ; ) \
 	$(foreach f, $(LINK_FILES), \
 		echo $(f) >> linkfarm.stamp.tmp ;)
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH v3] firmware/shim : filter output files during Xen tree setup

[Jan Beulich]

>>> On 26.07.18 at 23:16, <christopher.w.clark@gmail.com> wrote:
> Exclude named output files from the Xen tree setup.
> 
> The linkfarm.stamp content will differ between top level "make"
> and "make install" invocations, due to the introduction of these
> output files that are produced during the "make" build.
> 
> Filter these out to prevent an unnecessary rebuild of the shim
> during "make install", after "make" within a fresh source tree.
> 
> Excluded from consideration with this change: differences in stamp
> content when performing incremental builds in an existing tree.

I don't understand this (as well as you most recent remark on the
v2 thread): The "make install" invocation _is_ an incremental
rebuild. Hence I don't understand how excluding some but not all
generated files helps. But I'm not going to exclude that this is
simply because I don't understand well enough the logic in
xen-dir/Makefile when to trigger a rebuild.

> Signed-off-by: Christopher Clark <christopher.clark6@baesystems.com>
> ---
> Changes in v3: added '.xen.efi.*' '.xen-syms.*' to the exclude list.
> 
> Tested with: Xen 4.10.1, 4.11.0 and staging,
> Yocto poky, OpenEmbedded meta-openembedded, meta-virtualization
> with binutils 2.3.0 with x86_64-pep target enabled.

I sincerely hope you mean 2.30 or some such (halfway recent) here.

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH v3] firmware/shim : filter output files during Xen tree setup

[Christopher Clark]

[-- Attachment #1.1: Type: text/plain, Size: 4322 bytes --]

On Fri, Jul 27, 2018 at 1:34 AM, Jan Beulich <JBeulich@suse.com> wrote:

> >>> On 26.07.18 at 23:16, <christopher.w.clark@gmail.com> wrote:
> > Exclude named output files from the Xen tree setup.
> >
> > The linkfarm.stamp content will differ between top level "make"
> > and "make install" invocations, due to the introduction of these
> > output files that are produced during the "make" build.
> >
> > Filter these out to prevent an unnecessary rebuild of the shim
> > during "make install", after "make" within a fresh source tree.
> >
> > Excluded from consideration with this change: differences in stamp
> > content when performing incremental builds in an existing tree.
>
> I don't understand this (as well as you most recent remark on the
> v2 thread): The "make install" invocation _is_ an incremental
> rebuild. Hence I don't understand how excluding some but not all
> generated files helps. But I'm not going to exclude that this is
> simply because I don't understand well enough the logic in
> xen-dir/Makefile when to trigger a rebuild.
>
>
>
OK, so "make install" is considered to be an incremental rebuild here -
which makes sense - because in the implementation, it causes almost the
same actions, within the same tree, as the preceding "make":

"make" has the default target of all, which only depends on dist, dist
depends on the dist-<COMPONENT>s, and those each depend on
install-<COMPONENT>s. "make install" depends on install-<COMPONENT>s. A
main difference with "make install" is the destination directories that are
populated with the output.

Here's what's actually going on: it's parallel builds that are not working
correctly ("make -j <N>", where N>1). When multiple threads are available
to build with, the tools build starts before the xen subsystem build
finishes, so the first contents of linkfarm.stamp is a snapshot of a moving
tree, and it ends up missing the final xen subsystem build products.

The reason the v3 patch (mostly) works is that it lists the later build
products of the xen subsystem. The earlier ones, such as the headers, are
added to the xen tree long enough before the tools build gets started, so
they are witnessed and captured in the linkfarm.stamp file. I say the v3
patch mostly works, because I've found that its short exclusion list is
missing "xen.gz.new", which is briefly present as a temporary file and can
end up caught in the linkfarm.stamp file. It would need to be added to the
exclusion list if continuing with that filtering approach.

An alternative approach is to serialize the xen and tools subsystem builds
in the top level Makefile, not allowing the tools build to proceed until
the xen build is complete. I don't currently have a patch to propose to do
this. In the absence of that in place, the workaround is simple: build just
the xen subsystem alone first.

To demonstrate the behaviour described above:
===
With a single-threaded build, which will behave correctly:

make
cp ./tools/firmware/xen-dir/linkfarm.stamp ../linkfarm.stamp.compile
make install
diff -u ../linkfarm.stamp.compile ./tools/firmware/xen-dir/linkfarm.stamp

linkfarm.stamp.compile contents will be populated after the xen subsystem
build is complete, so contents of linkfarm.stamp will not change when
invoking "make install" and the diff will be empty.

===
With a multi-threaded build on an SMP machine (eg. an 8-way build host),
which may show the defect:

JOBS="12"
make -j $JOBS
cp ./tools/firmware/xen-dir/linkfarm.stamp ../linkfarm.stamp.compile
make -j $JOBS install
diff -u ../linkfarm.stamp.compile ./tools/firmware/xen-dir/linkfarm.stamp

linkfarm.stamp.compile contents will likely be populated _before_ the xen
subsystem build is complete, and if so the diff will _not_ be empty. The
shim will then rebuild during the "install" build.

Since this is racey, different results are visible with different numbers
of threads and different load on the machine. In testing, in one test 40
threads actually built just as well as the single threaded case, but each
of 2, 4, 8, 12, 16 threads showed different contents of the incomplete
linkfarm.stamp files after the first compile.

===
Workaround to build with multiple threads and no patch applied: build the
xen subsystem alone first:

make -j $JOBS xen
make -j $JOBS
make -j $JOBS install

===

Christopher

[-- Attachment #1.2: Type: text/html, Size: 5678 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH v3] firmware/shim : filter output files during Xen tree setup

[Jan Beulich]

>>> On 30.07.18 at 03:22, <christopher.w.clark@gmail.com> wrote:
> On Fri, Jul 27, 2018 at 1:34 AM, Jan Beulich <JBeulich@suse.com> wrote:
> 
>> >>> On 26.07.18 at 23:16, <christopher.w.clark@gmail.com> wrote:
>> > Exclude named output files from the Xen tree setup.
>> >
>> > The linkfarm.stamp content will differ between top level "make"
>> > and "make install" invocations, due to the introduction of these
>> > output files that are produced during the "make" build.
>> >
>> > Filter these out to prevent an unnecessary rebuild of the shim
>> > during "make install", after "make" within a fresh source tree.
>> >
>> > Excluded from consideration with this change: differences in stamp
>> > content when performing incremental builds in an existing tree.
>>
>> I don't understand this (as well as you most recent remark on the
>> v2 thread): The "make install" invocation _is_ an incremental
>> rebuild. Hence I don't understand how excluding some but not all
>> generated files helps. But I'm not going to exclude that this is
>> simply because I don't understand well enough the logic in
>> xen-dir/Makefile when to trigger a rebuild.
>>
>>
>>
> OK, so "make install" is considered to be an incremental rebuild here -
> which makes sense - because in the implementation, it causes almost the
> same actions, within the same tree, as the preceding "make":
> 
> "make" has the default target of all, which only depends on dist, dist
> depends on the dist-<COMPONENT>s, and those each depend on
> install-<COMPONENT>s. "make install" depends on install-<COMPONENT>s. A
> main difference with "make install" is the destination directories that are
> populated with the output.
> 
> Here's what's actually going on: it's parallel builds that are not working
> correctly ("make -j <N>", where N>1). When multiple threads are available
> to build with, the tools build starts before the xen subsystem build
> finishes, so the first contents of linkfarm.stamp is a snapshot of a moving
> tree, and it ends up missing the final xen subsystem build products.
> 
> The reason the v3 patch (mostly) works is that it lists the later build
> products of the xen subsystem. The earlier ones, such as the headers, are
> added to the xen tree long enough before the tools build gets started, so
> they are witnessed and captured in the linkfarm.stamp file. I say the v3
> patch mostly works, because I've found that its short exclusion list is
> missing "xen.gz.new", which is briefly present as a temporary file and can
> end up caught in the linkfarm.stamp file. It would need to be added to the
> exclusion list if continuing with that filtering approach.
> 
> An alternative approach is to serialize the xen and tools subsystem builds
> in the top level Makefile, not allowing the tools build to proceed until
> the xen build is complete. I don't currently have a patch to propose to do
> this. In the absence of that in place, the workaround is simple: build just
> the xen subsystem alone first.

Thanks for the thorough analysis. I don't think the goal should be to
restrict the two builds to happen one after the other. Instead, as said
before, the goal ought to be to disregard _all_ generated files when
producing the stamp file.

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel