[PATCH] build: centralize / unify asm-offsets generation

[PATCH] build: centralize / unify asm-offsets generation

[Jan Beulich]

Except for an additional prereq Arm and x86 have the same needs here,
and Arm can also benefit from the recent x86 side improvement. Recurse
into arch/*/ only for a phony include target (doing nothing on Arm),
and handle asm-offsets itself entirely locally to xen/Makefile.

Signed-off-by: Jan Beulich <jbeulich@suse.com>

--- a/.gitignore
+++ b/.gitignore
@@ -318,7 +318,6 @@
 xen/arch/x86/efi/check.efi
 xen/arch/x86/efi/mkreloc
 xen/arch/*/xen.lds
-xen/arch/*/asm-offsets.s
 xen/arch/*/efi/boot.c
 xen/arch/*/efi/compat.c
 xen/arch/*/efi/ebmalloc.c
@@ -325,6 +324,7 @@
 xen/arch/*/efi/efi.h
 xen/arch/*/efi/pe.c
 xen/arch/*/efi/runtime.c
+xen/asm-offsets.s
 xen/common/config_data.S
 xen/common/config.gz
 xen/include/headers*.chk
--- a/xen/Makefile
+++ b/xen/Makefile
@@ -341,7 +341,7 @@ _clean: delete-unfresh-files
 	find . \( -name "*.o" -o -name ".*.d" -o -name ".*.d2" \
 		-o -name "*.gcno" -o -name ".*.cmd" \) -exec rm -f {} \;
 	rm -f include/asm $(TARGET) $(TARGET).gz $(TARGET).efi $(TARGET).efi.map $(TARGET)-syms $(TARGET)-syms.map *~ core
-	rm -f include/asm-*/asm-offsets.h
+	rm -f asm-offsets.s include/asm-*/asm-offsets.h
 	rm -f .banner
 
 .PHONY: _distclean
@@ -362,7 +362,7 @@ $(TARGET): delete-unfresh-files
 		done; \
 		true
 	$(MAKE) -f $(BASEDIR)/Rules.mk -C include
-	$(MAKE) -f $(BASEDIR)/Rules.mk -C arch/$(TARGET_ARCH) asm-offsets.s
+	$(MAKE) -f $(BASEDIR)/Rules.mk -C arch/$(TARGET_ARCH) include
 	$(MAKE) -f $(BASEDIR)/Rules.mk include/asm-$(TARGET_ARCH)/asm-offsets.h
 	$(MAKE) -f $(BASEDIR)/Rules.mk -C arch/$(TARGET_ARCH) $@
 
@@ -399,7 +399,11 @@ include/xen/compile.h: include/xen/compi
 	@sed -rf tools/process-banner.sed < .banner >> $@.new
 	@mv -f $@.new $@
 
-include/asm-$(TARGET_ARCH)/asm-offsets.h: arch/$(TARGET_ARCH)/asm-offsets.s
+asm-offsets.s: arch/$(TARGET_ARCH)/$(TARGET_SUBARCH)/asm-offsets.c
+	$(CC) $(filter-out -Wa$(comma)% -flto,$(c_flags)) -S -g0 -o $@.new -MQ $@ $<
+	$(call move-if-changed,$@.new,$@)
+
+include/asm-$(TARGET_ARCH)/asm-offsets.h: asm-offsets.s
 	@(set -e; \
 	  echo "/*"; \
 	  echo " * DO NOT MODIFY."; \
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -131,8 +131,8 @@ $(TARGET)-syms: prelink.o xen.lds
 		>$(@D)/$(@F).map
 	rm -f $(@D)/.$(@F).[0-9]*
 
-asm-offsets.s: $(TARGET_SUBARCH)/asm-offsets.c
-	$(CC) $(filter-out -flto,$(c_flags)) -S -o $@ $<
+.PHONY: include
+include:
 
 xen.lds: xen.lds.S
 	$(CPP) -P $(a_flags) -MQ $@ -o $@ $<
@@ -141,6 +141,6 @@ dtb.o: $(CONFIG_DTB_FILE)
 
 .PHONY: clean
 clean::
-	rm -f asm-offsets.s xen.lds
+	rm -f xen.lds
 	rm -f $(BASEDIR)/.xen-syms.[0-9]*
 	rm -f $(TARGET).efi
--- a/xen/arch/x86/Makefile
+++ b/xen/arch/x86/Makefile
@@ -244,9 +244,8 @@ endif
 efi/buildid.o efi/relocs-dummy.o: $(BASEDIR)/arch/x86/efi/built_in.o
 efi/buildid.o efi/relocs-dummy.o: ;
 
-asm-offsets.s: $(TARGET_SUBARCH)/asm-offsets.c $(BASEDIR)/include/asm-x86/asm-macros.h
-	$(CC) $(filter-out -Wa$(comma)% -flto,$(c_flags)) -S -g0 -o $@.new -MQ $@ $<
-	$(call move-if-changed,$@.new,$@)
+.PHONY: include
+include: $(BASEDIR)/include/asm-x86/asm-macros.h
 
 asm-macros.i: CFLAGS-y += -D__ASSEMBLY__ -P
 
@@ -276,7 +275,7 @@ hweight.o: CFLAGS-y += $(foreach reg,cx
 
 .PHONY: clean
 clean::
-	rm -f asm-offsets.s *.lds *.new boot/*.o boot/*~ boot/core boot/mkelf32
+	rm -f *.lds *.new boot/*.o boot/*~ boot/core boot/mkelf32
 	rm -f asm-macros.i $(BASEDIR)/include/asm-x86/asm-macros.*
 	rm -f $(BASEDIR)/.xen-syms.[0-9]* boot/.*.d $(BASEDIR)/.xen.elf32
 	rm -f $(BASEDIR)/.xen.efi.[0-9]* efi/*.efi efi/mkreloc

Re: [PATCH] build: centralize / unify asm-offsets generation

[Julien Grall]

Hi Jan,

On 01/04/2021 09:33, Jan Beulich wrote:
> --- a/xen/Makefile
> +++ b/xen/Makefile
> @@ -341,7 +341,7 @@ _clean: delete-unfresh-files
>   	find . \( -name "*.o" -o -name ".*.d" -o -name ".*.d2" \
>   		-o -name "*.gcno" -o -name ".*.cmd" \) -exec rm -f {} \;
>   	rm -f include/asm $(TARGET) $(TARGET).gz $(TARGET).efi $(TARGET).efi.map $(TARGET)-syms $(TARGET)-syms.map *~ core
> -	rm -f include/asm-*/asm-offsets.h
> +	rm -f asm-offsets.s include/asm-*/asm-offsets.h
>   	rm -f .banner
>   
>   .PHONY: _distclean
> @@ -362,7 +362,7 @@ $(TARGET): delete-unfresh-files
>   		done; \
>   		true
>   	$(MAKE) -f $(BASEDIR)/Rules.mk -C include
> -	$(MAKE) -f $(BASEDIR)/Rules.mk -C arch/$(TARGET_ARCH) asm-offsets.s
> +	$(MAKE) -f $(BASEDIR)/Rules.mk -C arch/$(TARGET_ARCH) include
>   	$(MAKE) -f $(BASEDIR)/Rules.mk include/asm-$(TARGET_ARCH)/asm-offsets.h
>   	$(MAKE) -f $(BASEDIR)/Rules.mk -C arch/$(TARGET_ARCH) $@
>   
> @@ -399,7 +399,11 @@ include/xen/compile.h: include/xen/compi
>   	@sed -rf tools/process-banner.sed < .banner >> $@.new
>   	@mv -f $@.new $@
>   
> -include/asm-$(TARGET_ARCH)/asm-offsets.h: arch/$(TARGET_ARCH)/asm-offsets.s
> +asm-offsets.s: arch/$(TARGET_ARCH)/$(TARGET_SUBARCH)/asm-offsets.c
> +	$(CC) $(filter-out -Wa$(comma)% -flto,$(c_flags)) -S -g0 -o $@.new -MQ $@ $<
> +	$(call move-if-changed,$@.new,$@)
> +
> +include/asm-$(TARGET_ARCH)/asm-offsets.h: asm-offsets.s
>   	@(set -e; \
>   	  echo "/*"; \
>   	  echo " * DO NOT MODIFY."; \
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -131,8 +131,8 @@ $(TARGET)-syms: prelink.o xen.lds
>   		>$(@D)/$(@F).map
>   	rm -f $(@D)/.$(@F).[0-9]*
>   
> -asm-offsets.s: $(TARGET_SUBARCH)/asm-offsets.c
> -	$(CC) $(filter-out -flto,$(c_flags)) -S -o $@ $<
> +.PHONY: include
> +include:

 From a generic PoV, this target is meant to generate arch-specific 
include. Is that correct?

Cheers,

-- 
Julien Grall

Re: [PATCH] build: centralize / unify asm-offsets generation

[Jan Beulich]

On 01.04.2021 17:05, Julien Grall wrote:
> On 01/04/2021 09:33, Jan Beulich wrote:
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -131,8 +131,8 @@ $(TARGET)-syms: prelink.o xen.lds
>>   		>$(@D)/$(@F).map
>>   	rm -f $(@D)/.$(@F).[0-9]*
>>   
>> -asm-offsets.s: $(TARGET_SUBARCH)/asm-offsets.c
>> -	$(CC) $(filter-out -flto,$(c_flags)) -S -o $@ $<
>> +.PHONY: include
>> +include:
> 
>  From a generic PoV, this target is meant to generate arch-specific 
> include. Is that correct?

Yes, like is happening on x86 already.

Jan

Ping: [PATCH] build: centralize / unify asm-offsets generation

[Jan Beulich]

On 01.04.2021 10:33, Jan Beulich wrote:
> Except for an additional prereq Arm and x86 have the same needs here,
> and Arm can also benefit from the recent x86 side improvement. Recurse
> into arch/*/ only for a phony include target (doing nothing on Arm),
> and handle asm-offsets itself entirely locally to xen/Makefile.
> 
> Signed-off-by: Jan Beulich <jbeulich@suse.com>

Comments / acks, anyone?

Thanks, Jan

> --- a/.gitignore
> +++ b/.gitignore
> @@ -318,7 +318,6 @@
>  xen/arch/x86/efi/check.efi
>  xen/arch/x86/efi/mkreloc
>  xen/arch/*/xen.lds
> -xen/arch/*/asm-offsets.s
>  xen/arch/*/efi/boot.c
>  xen/arch/*/efi/compat.c
>  xen/arch/*/efi/ebmalloc.c
> @@ -325,6 +324,7 @@
>  xen/arch/*/efi/efi.h
>  xen/arch/*/efi/pe.c
>  xen/arch/*/efi/runtime.c
> +xen/asm-offsets.s
>  xen/common/config_data.S
>  xen/common/config.gz
>  xen/include/headers*.chk
> --- a/xen/Makefile
> +++ b/xen/Makefile
> @@ -341,7 +341,7 @@ _clean: delete-unfresh-files
>  	find . \( -name "*.o" -o -name ".*.d" -o -name ".*.d2" \
>  		-o -name "*.gcno" -o -name ".*.cmd" \) -exec rm -f {} \;
>  	rm -f include/asm $(TARGET) $(TARGET).gz $(TARGET).efi $(TARGET).efi.map $(TARGET)-syms $(TARGET)-syms.map *~ core
> -	rm -f include/asm-*/asm-offsets.h
> +	rm -f asm-offsets.s include/asm-*/asm-offsets.h
>  	rm -f .banner
>  
>  .PHONY: _distclean
> @@ -362,7 +362,7 @@ $(TARGET): delete-unfresh-files
>  		done; \
>  		true
>  	$(MAKE) -f $(BASEDIR)/Rules.mk -C include
> -	$(MAKE) -f $(BASEDIR)/Rules.mk -C arch/$(TARGET_ARCH) asm-offsets.s
> +	$(MAKE) -f $(BASEDIR)/Rules.mk -C arch/$(TARGET_ARCH) include
>  	$(MAKE) -f $(BASEDIR)/Rules.mk include/asm-$(TARGET_ARCH)/asm-offsets.h
>  	$(MAKE) -f $(BASEDIR)/Rules.mk -C arch/$(TARGET_ARCH) $@
>  
> @@ -399,7 +399,11 @@ include/xen/compile.h: include/xen/compi
>  	@sed -rf tools/process-banner.sed < .banner >> $@.new
>  	@mv -f $@.new $@
>  
> -include/asm-$(TARGET_ARCH)/asm-offsets.h: arch/$(TARGET_ARCH)/asm-offsets.s
> +asm-offsets.s: arch/$(TARGET_ARCH)/$(TARGET_SUBARCH)/asm-offsets.c
> +	$(CC) $(filter-out -Wa$(comma)% -flto,$(c_flags)) -S -g0 -o $@.new -MQ $@ $<
> +	$(call move-if-changed,$@.new,$@)
> +
> +include/asm-$(TARGET_ARCH)/asm-offsets.h: asm-offsets.s
>  	@(set -e; \
>  	  echo "/*"; \
>  	  echo " * DO NOT MODIFY."; \
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -131,8 +131,8 @@ $(TARGET)-syms: prelink.o xen.lds
>  		>$(@D)/$(@F).map
>  	rm -f $(@D)/.$(@F).[0-9]*
>  
> -asm-offsets.s: $(TARGET_SUBARCH)/asm-offsets.c
> -	$(CC) $(filter-out -flto,$(c_flags)) -S -o $@ $<
> +.PHONY: include
> +include:
>  
>  xen.lds: xen.lds.S
>  	$(CPP) -P $(a_flags) -MQ $@ -o $@ $<
> @@ -141,6 +141,6 @@ dtb.o: $(CONFIG_DTB_FILE)
>  
>  .PHONY: clean
>  clean::
> -	rm -f asm-offsets.s xen.lds
> +	rm -f xen.lds
>  	rm -f $(BASEDIR)/.xen-syms.[0-9]*
>  	rm -f $(TARGET).efi
> --- a/xen/arch/x86/Makefile
> +++ b/xen/arch/x86/Makefile
> @@ -244,9 +244,8 @@ endif
>  efi/buildid.o efi/relocs-dummy.o: $(BASEDIR)/arch/x86/efi/built_in.o
>  efi/buildid.o efi/relocs-dummy.o: ;
>  
> -asm-offsets.s: $(TARGET_SUBARCH)/asm-offsets.c $(BASEDIR)/include/asm-x86/asm-macros.h
> -	$(CC) $(filter-out -Wa$(comma)% -flto,$(c_flags)) -S -g0 -o $@.new -MQ $@ $<
> -	$(call move-if-changed,$@.new,$@)
> +.PHONY: include
> +include: $(BASEDIR)/include/asm-x86/asm-macros.h
>  
>  asm-macros.i: CFLAGS-y += -D__ASSEMBLY__ -P
>  
> @@ -276,7 +275,7 @@ hweight.o: CFLAGS-y += $(foreach reg,cx
>  
>  .PHONY: clean
>  clean::
> -	rm -f asm-offsets.s *.lds *.new boot/*.o boot/*~ boot/core boot/mkelf32
> +	rm -f *.lds *.new boot/*.o boot/*~ boot/core boot/mkelf32
>  	rm -f asm-macros.i $(BASEDIR)/include/asm-x86/asm-macros.*
>  	rm -f $(BASEDIR)/.xen-syms.[0-9]* boot/.*.d $(BASEDIR)/.xen.elf32
>  	rm -f $(BASEDIR)/.xen.efi.[0-9]* efi/*.efi efi/mkreloc
> 

Re: [PATCH] build: centralize / unify asm-offsets generation

[Roger Pau Monné]

On Thu, Apr 01, 2021 at 10:33:47AM +0200, Jan Beulich wrote:
> Except for an additional prereq Arm and x86 have the same needs here,
> and Arm can also benefit from the recent x86 side improvement. Recurse
> into arch/*/ only for a phony include target (doing nothing on Arm),
> and handle asm-offsets itself entirely locally to xen/Makefile.
> 
> Signed-off-by: Jan Beulich <jbeulich@suse.com>
> 
> --- a/.gitignore
> +++ b/.gitignore
> @@ -318,7 +318,6 @@
>  xen/arch/x86/efi/check.efi
>  xen/arch/x86/efi/mkreloc
>  xen/arch/*/xen.lds
> -xen/arch/*/asm-offsets.s
>  xen/arch/*/efi/boot.c
>  xen/arch/*/efi/compat.c
>  xen/arch/*/efi/ebmalloc.c
> @@ -325,6 +324,7 @@
>  xen/arch/*/efi/efi.h
>  xen/arch/*/efi/pe.c
>  xen/arch/*/efi/runtime.c
> +xen/asm-offsets.s
>  xen/common/config_data.S
>  xen/common/config.gz
>  xen/include/headers*.chk
> --- a/xen/Makefile
> +++ b/xen/Makefile
> @@ -341,7 +341,7 @@ _clean: delete-unfresh-files
>  	find . \( -name "*.o" -o -name ".*.d" -o -name ".*.d2" \
>  		-o -name "*.gcno" -o -name ".*.cmd" \) -exec rm -f {} \;
>  	rm -f include/asm $(TARGET) $(TARGET).gz $(TARGET).efi $(TARGET).efi.map $(TARGET)-syms $(TARGET)-syms.map *~ core
> -	rm -f include/asm-*/asm-offsets.h
> +	rm -f asm-offsets.s include/asm-*/asm-offsets.h
>  	rm -f .banner
>  
>  .PHONY: _distclean
> @@ -362,7 +362,7 @@ $(TARGET): delete-unfresh-files
>  		done; \
>  		true
>  	$(MAKE) -f $(BASEDIR)/Rules.mk -C include
> -	$(MAKE) -f $(BASEDIR)/Rules.mk -C arch/$(TARGET_ARCH) asm-offsets.s
> +	$(MAKE) -f $(BASEDIR)/Rules.mk -C arch/$(TARGET_ARCH) include
>  	$(MAKE) -f $(BASEDIR)/Rules.mk include/asm-$(TARGET_ARCH)/asm-offsets.h
>  	$(MAKE) -f $(BASEDIR)/Rules.mk -C arch/$(TARGET_ARCH) $@
>  
> @@ -399,7 +399,11 @@ include/xen/compile.h: include/xen/compi
>  	@sed -rf tools/process-banner.sed < .banner >> $@.new
>  	@mv -f $@.new $@
>  
> -include/asm-$(TARGET_ARCH)/asm-offsets.h: arch/$(TARGET_ARCH)/asm-offsets.s
> +asm-offsets.s: arch/$(TARGET_ARCH)/$(TARGET_SUBARCH)/asm-offsets.c
> +	$(CC) $(filter-out -Wa$(comma)% -flto,$(c_flags)) -S -g0 -o $@.new -MQ $@ $<
> +	$(call move-if-changed,$@.new,$@)

Won't it be more natural to keep the .s file in arch/$(TARGET_ARCH)?

Thanks, Roger.

Re: [PATCH] build: centralize / unify asm-offsets generation

[Jan Beulich]

On 20.04.2021 17:29, Roger Pau Monné wrote:
> On Thu, Apr 01, 2021 at 10:33:47AM +0200, Jan Beulich wrote:
>> @@ -399,7 +399,11 @@ include/xen/compile.h: include/xen/compi
>>  	@sed -rf tools/process-banner.sed < .banner >> $@.new
>>  	@mv -f $@.new $@
>>  
>> -include/asm-$(TARGET_ARCH)/asm-offsets.h: arch/$(TARGET_ARCH)/asm-offsets.s
>> +asm-offsets.s: arch/$(TARGET_ARCH)/$(TARGET_SUBARCH)/asm-offsets.c
>> +	$(CC) $(filter-out -Wa$(comma)% -flto,$(c_flags)) -S -g0 -o $@.new -MQ $@ $<
>> +	$(call move-if-changed,$@.new,$@)
> 
> Won't it be more natural to keep the .s file in arch/$(TARGET_ARCH)?

Yes and no: Yes as far as the actual file location is concerned.
No when considering where it gets generated: I generally consider
it risky to generate files outside of the directory where make
currently runs. There may be good reasons for certain exceptions,
but personally I don't see this case as good enough a reason.

Somewhat related - if doing as you suggest, which Makefile's
clean: rule should clean up that file in your opinion?

Nevertheless, if there's general agreement that keeping the file
there is better, I'd make the change and simply ignore my unhappy
feelings about it.

Jan

Re: [PATCH] build: centralize / unify asm-offsets generation

[Roger Pau Monné]

On Tue, Apr 20, 2021 at 05:47:49PM +0200, Jan Beulich wrote:
> On 20.04.2021 17:29, Roger Pau Monné wrote:
> > On Thu, Apr 01, 2021 at 10:33:47AM +0200, Jan Beulich wrote:
> >> @@ -399,7 +399,11 @@ include/xen/compile.h: include/xen/compi
> >>  	@sed -rf tools/process-banner.sed < .banner >> $@.new
> >>  	@mv -f $@.new $@
> >>  
> >> -include/asm-$(TARGET_ARCH)/asm-offsets.h: arch/$(TARGET_ARCH)/asm-offsets.s
> >> +asm-offsets.s: arch/$(TARGET_ARCH)/$(TARGET_SUBARCH)/asm-offsets.c
> >> +	$(CC) $(filter-out -Wa$(comma)% -flto,$(c_flags)) -S -g0 -o $@.new -MQ $@ $<
> >> +	$(call move-if-changed,$@.new,$@)
> > 
> > Won't it be more natural to keep the .s file in arch/$(TARGET_ARCH)?
> 
> Yes and no: Yes as far as the actual file location is concerned.
> No when considering where it gets generated: I generally consider
> it risky to generate files outside of the directory where make
> currently runs. There may be good reasons for certain exceptions,
> but personally I don't see this case as good enough a reason.
> 
> Somewhat related - if doing as you suggest, which Makefile's
> clean: rule should clean up that file in your opinion?

The clean rule should be in the makefile where it's generated IMO,
same as asm-offsets.h clean rule currently in xen/Makefile.

> Nevertheless, if there's general agreement that keeping the file
> there is better, I'd make the change and simply ignore my unhappy
> feelings about it.

I don't have a strong opinion, it just feels weird to have this IMO
stray asm-offsets.s outside of it's arch directory, taking into
account that we have asm-offsets.h generated from xen/Makefile into an
arch specific directory already as a precedent in that makefile.

Thanks, Roger.

Re: [PATCH] build: centralize / unify asm-offsets generation

[Jan Beulich]

On 20.04.2021 18:20, Roger Pau Monné wrote:
> On Tue, Apr 20, 2021 at 05:47:49PM +0200, Jan Beulich wrote:
>> On 20.04.2021 17:29, Roger Pau Monné wrote:
>>> On Thu, Apr 01, 2021 at 10:33:47AM +0200, Jan Beulich wrote:
>>>> @@ -399,7 +399,11 @@ include/xen/compile.h: include/xen/compi
>>>>  	@sed -rf tools/process-banner.sed < .banner >> $@.new
>>>>  	@mv -f $@.new $@
>>>>  
>>>> -include/asm-$(TARGET_ARCH)/asm-offsets.h: arch/$(TARGET_ARCH)/asm-offsets.s
>>>> +asm-offsets.s: arch/$(TARGET_ARCH)/$(TARGET_SUBARCH)/asm-offsets.c
>>>> +	$(CC) $(filter-out -Wa$(comma)% -flto,$(c_flags)) -S -g0 -o $@.new -MQ $@ $<
>>>> +	$(call move-if-changed,$@.new,$@)
>>>
>>> Won't it be more natural to keep the .s file in arch/$(TARGET_ARCH)?
>>
>> Yes and no: Yes as far as the actual file location is concerned.
>> No when considering where it gets generated: I generally consider
>> it risky to generate files outside of the directory where make
>> currently runs. There may be good reasons for certain exceptions,
>> but personally I don't see this case as good enough a reason.
>>
>> Somewhat related - if doing as you suggest, which Makefile's
>> clean: rule should clean up that file in your opinion?
> 
> The clean rule should be in the makefile where it's generated IMO,
> same as asm-offsets.h clean rule currently in xen/Makefile.
> 
>> Nevertheless, if there's general agreement that keeping the file
>> there is better, I'd make the change and simply ignore my unhappy
>> feelings about it.
> 
> I don't have a strong opinion, it just feels weird to have this IMO
> stray asm-offsets.s outside of it's arch directory, taking into
> account that we have asm-offsets.h generated from xen/Makefile into an
> arch specific directory already as a precedent in that makefile.

Well, asm-offsets.h generation doesn't involve the compiler, hence
no .*.d files get generated and want including later. For
asm-offsets.s to have dependencies properly honored, if we
generated it in xen/arch/<arch>, .asm-offsets.d would also end up
there, and hence including of it would need separately taking care
of.

Jan

Re: [PATCH] build: centralize / unify asm-offsets generation

[Roger Pau Monné]

On Wed, Apr 21, 2021 at 04:09:03PM +0200, Jan Beulich wrote:
> On 20.04.2021 18:20, Roger Pau Monné wrote:
> > On Tue, Apr 20, 2021 at 05:47:49PM +0200, Jan Beulich wrote:
> >> On 20.04.2021 17:29, Roger Pau Monné wrote:
> >>> On Thu, Apr 01, 2021 at 10:33:47AM +0200, Jan Beulich wrote:
> >>>> @@ -399,7 +399,11 @@ include/xen/compile.h: include/xen/compi
> >>>>  	@sed -rf tools/process-banner.sed < .banner >> $@.new
> >>>>  	@mv -f $@.new $@
> >>>>  
> >>>> -include/asm-$(TARGET_ARCH)/asm-offsets.h: arch/$(TARGET_ARCH)/asm-offsets.s
> >>>> +asm-offsets.s: arch/$(TARGET_ARCH)/$(TARGET_SUBARCH)/asm-offsets.c
> >>>> +	$(CC) $(filter-out -Wa$(comma)% -flto,$(c_flags)) -S -g0 -o $@.new -MQ $@ $<
> >>>> +	$(call move-if-changed,$@.new,$@)
> >>>
> >>> Won't it be more natural to keep the .s file in arch/$(TARGET_ARCH)?
> >>
> >> Yes and no: Yes as far as the actual file location is concerned.
> >> No when considering where it gets generated: I generally consider
> >> it risky to generate files outside of the directory where make
> >> currently runs. There may be good reasons for certain exceptions,
> >> but personally I don't see this case as good enough a reason.
> >>
> >> Somewhat related - if doing as you suggest, which Makefile's
> >> clean: rule should clean up that file in your opinion?
> > 
> > The clean rule should be in the makefile where it's generated IMO,
> > same as asm-offsets.h clean rule currently in xen/Makefile.
> > 
> >> Nevertheless, if there's general agreement that keeping the file
> >> there is better, I'd make the change and simply ignore my unhappy
> >> feelings about it.
> > 
> > I don't have a strong opinion, it just feels weird to have this IMO
> > stray asm-offsets.s outside of it's arch directory, taking into
> > account that we have asm-offsets.h generated from xen/Makefile into an
> > arch specific directory already as a precedent in that makefile.
> 
> Well, asm-offsets.h generation doesn't involve the compiler, hence
> no .*.d files get generated and want including later. For
> asm-offsets.s to have dependencies properly honored, if we
> generated it in xen/arch/<arch>, .asm-offsets.d would also end up
> there, and hence including of it would need separately taking care
> of.

I'm not sure I understand what do you mean with inclusion need taking
care of separately. Isn't it expected for .d file to reside in the
same directory as the output, and hence picked up automatically?

Thanks, Roger.

Re: [PATCH] build: centralize / unify asm-offsets generation

[Jan Beulich]

On 27.04.2021 11:05, Roger Pau Monné wrote:
> On Wed, Apr 21, 2021 at 04:09:03PM +0200, Jan Beulich wrote:
>> On 20.04.2021 18:20, Roger Pau Monné wrote:
>>> On Tue, Apr 20, 2021 at 05:47:49PM +0200, Jan Beulich wrote:
>>>> On 20.04.2021 17:29, Roger Pau Monné wrote:
>>>>> On Thu, Apr 01, 2021 at 10:33:47AM +0200, Jan Beulich wrote:
>>>>>> @@ -399,7 +399,11 @@ include/xen/compile.h: include/xen/compi
>>>>>>  	@sed -rf tools/process-banner.sed < .banner >> $@.new
>>>>>>  	@mv -f $@.new $@
>>>>>>  
>>>>>> -include/asm-$(TARGET_ARCH)/asm-offsets.h: arch/$(TARGET_ARCH)/asm-offsets.s
>>>>>> +asm-offsets.s: arch/$(TARGET_ARCH)/$(TARGET_SUBARCH)/asm-offsets.c
>>>>>> +	$(CC) $(filter-out -Wa$(comma)% -flto,$(c_flags)) -S -g0 -o $@.new -MQ $@ $<
>>>>>> +	$(call move-if-changed,$@.new,$@)
>>>>>
>>>>> Won't it be more natural to keep the .s file in arch/$(TARGET_ARCH)?
>>>>
>>>> Yes and no: Yes as far as the actual file location is concerned.
>>>> No when considering where it gets generated: I generally consider
>>>> it risky to generate files outside of the directory where make
>>>> currently runs. There may be good reasons for certain exceptions,
>>>> but personally I don't see this case as good enough a reason.
>>>>
>>>> Somewhat related - if doing as you suggest, which Makefile's
>>>> clean: rule should clean up that file in your opinion?
>>>
>>> The clean rule should be in the makefile where it's generated IMO,
>>> same as asm-offsets.h clean rule currently in xen/Makefile.
>>>
>>>> Nevertheless, if there's general agreement that keeping the file
>>>> there is better, I'd make the change and simply ignore my unhappy
>>>> feelings about it.
>>>
>>> I don't have a strong opinion, it just feels weird to have this IMO
>>> stray asm-offsets.s outside of it's arch directory, taking into
>>> account that we have asm-offsets.h generated from xen/Makefile into an
>>> arch specific directory already as a precedent in that makefile.
>>
>> Well, asm-offsets.h generation doesn't involve the compiler, hence
>> no .*.d files get generated and want including later. For
>> asm-offsets.s to have dependencies properly honored, if we
>> generated it in xen/arch/<arch>, .asm-offsets.d would also end up
>> there, and hence including of it would need separately taking care
>> of.
> 
> I'm not sure I understand what do you mean with inclusion need taking
> care of separately. Isn't it expected for .d file to reside in the
> same directory as the output,

Yes, ...

> and hence picked up automatically?

... and hence they _wouldn't_ be picked up automatically while
building in xen/ if the .s file got created in xen/arch/<arch>/.

Jan

Re: [PATCH] build: centralize / unify asm-offsets generation

[Roger Pau Monné]

On Tue, Apr 27, 2021 at 02:30:35PM +0200, Jan Beulich wrote:
> On 27.04.2021 11:05, Roger Pau Monné wrote:
> > On Wed, Apr 21, 2021 at 04:09:03PM +0200, Jan Beulich wrote:
> >> On 20.04.2021 18:20, Roger Pau Monné wrote:
> >>> On Tue, Apr 20, 2021 at 05:47:49PM +0200, Jan Beulich wrote:
> >>>> On 20.04.2021 17:29, Roger Pau Monné wrote:
> >>>>> On Thu, Apr 01, 2021 at 10:33:47AM +0200, Jan Beulich wrote:
> >>>>>> @@ -399,7 +399,11 @@ include/xen/compile.h: include/xen/compi
> >>>>>>  	@sed -rf tools/process-banner.sed < .banner >> $@.new
> >>>>>>  	@mv -f $@.new $@
> >>>>>>  
> >>>>>> -include/asm-$(TARGET_ARCH)/asm-offsets.h: arch/$(TARGET_ARCH)/asm-offsets.s
> >>>>>> +asm-offsets.s: arch/$(TARGET_ARCH)/$(TARGET_SUBARCH)/asm-offsets.c
> >>>>>> +	$(CC) $(filter-out -Wa$(comma)% -flto,$(c_flags)) -S -g0 -o $@.new -MQ $@ $<
> >>>>>> +	$(call move-if-changed,$@.new,$@)
> >>>>>
> >>>>> Won't it be more natural to keep the .s file in arch/$(TARGET_ARCH)?
> >>>>
> >>>> Yes and no: Yes as far as the actual file location is concerned.
> >>>> No when considering where it gets generated: I generally consider
> >>>> it risky to generate files outside of the directory where make
> >>>> currently runs. There may be good reasons for certain exceptions,
> >>>> but personally I don't see this case as good enough a reason.
> >>>>
> >>>> Somewhat related - if doing as you suggest, which Makefile's
> >>>> clean: rule should clean up that file in your opinion?
> >>>
> >>> The clean rule should be in the makefile where it's generated IMO,
> >>> same as asm-offsets.h clean rule currently in xen/Makefile.
> >>>
> >>>> Nevertheless, if there's general agreement that keeping the file
> >>>> there is better, I'd make the change and simply ignore my unhappy
> >>>> feelings about it.
> >>>
> >>> I don't have a strong opinion, it just feels weird to have this IMO
> >>> stray asm-offsets.s outside of it's arch directory, taking into
> >>> account that we have asm-offsets.h generated from xen/Makefile into an
> >>> arch specific directory already as a precedent in that makefile.
> >>
> >> Well, asm-offsets.h generation doesn't involve the compiler, hence
> >> no .*.d files get generated and want including later. For
> >> asm-offsets.s to have dependencies properly honored, if we
> >> generated it in xen/arch/<arch>, .asm-offsets.d would also end up
> >> there, and hence including of it would need separately taking care
> >> of.
> > 
> > I'm not sure I understand what do you mean with inclusion need taking
> > care of separately. Isn't it expected for .d file to reside in the
> > same directory as the output,
> 
> Yes, ...
> 
> > and hence picked up automatically?
> 
> ... and hence they _wouldn't_ be picked up automatically while
> building in xen/ if the .s file got created in xen/arch/<arch>/.

Hm, so that would prevent re-building the target when the dependencies
changed as the .d file being in the arch directory would attempt the
rebuild from there instead of the top level xen/?

I guess the alternative would be to force a rebuild of the target
every time, in order to not depend on the .d logic?

Thanks, Roger.

Re: [PATCH] build: centralize / unify asm-offsets generation

[Jan Beulich]

On 27.04.2021 15:31, Roger Pau Monné wrote:
> On Tue, Apr 27, 2021 at 02:30:35PM +0200, Jan Beulich wrote:
>> On 27.04.2021 11:05, Roger Pau Monné wrote:
>>> On Wed, Apr 21, 2021 at 04:09:03PM +0200, Jan Beulich wrote:
>>>> On 20.04.2021 18:20, Roger Pau Monné wrote:
>>>>> On Tue, Apr 20, 2021 at 05:47:49PM +0200, Jan Beulich wrote:
>>>>>> On 20.04.2021 17:29, Roger Pau Monné wrote:
>>>>>>> On Thu, Apr 01, 2021 at 10:33:47AM +0200, Jan Beulich wrote:
>>>>>>>> @@ -399,7 +399,11 @@ include/xen/compile.h: include/xen/compi
>>>>>>>>  	@sed -rf tools/process-banner.sed < .banner >> $@.new
>>>>>>>>  	@mv -f $@.new $@
>>>>>>>>  
>>>>>>>> -include/asm-$(TARGET_ARCH)/asm-offsets.h: arch/$(TARGET_ARCH)/asm-offsets.s
>>>>>>>> +asm-offsets.s: arch/$(TARGET_ARCH)/$(TARGET_SUBARCH)/asm-offsets.c
>>>>>>>> +	$(CC) $(filter-out -Wa$(comma)% -flto,$(c_flags)) -S -g0 -o $@.new -MQ $@ $<
>>>>>>>> +	$(call move-if-changed,$@.new,$@)
>>>>>>>
>>>>>>> Won't it be more natural to keep the .s file in arch/$(TARGET_ARCH)?
>>>>>>
>>>>>> Yes and no: Yes as far as the actual file location is concerned.
>>>>>> No when considering where it gets generated: I generally consider
>>>>>> it risky to generate files outside of the directory where make
>>>>>> currently runs. There may be good reasons for certain exceptions,
>>>>>> but personally I don't see this case as good enough a reason.
>>>>>>
>>>>>> Somewhat related - if doing as you suggest, which Makefile's
>>>>>> clean: rule should clean up that file in your opinion?
>>>>>
>>>>> The clean rule should be in the makefile where it's generated IMO,
>>>>> same as asm-offsets.h clean rule currently in xen/Makefile.
>>>>>
>>>>>> Nevertheless, if there's general agreement that keeping the file
>>>>>> there is better, I'd make the change and simply ignore my unhappy
>>>>>> feelings about it.
>>>>>
>>>>> I don't have a strong opinion, it just feels weird to have this IMO
>>>>> stray asm-offsets.s outside of it's arch directory, taking into
>>>>> account that we have asm-offsets.h generated from xen/Makefile into an
>>>>> arch specific directory already as a precedent in that makefile.
>>>>
>>>> Well, asm-offsets.h generation doesn't involve the compiler, hence
>>>> no .*.d files get generated and want including later. For
>>>> asm-offsets.s to have dependencies properly honored, if we
>>>> generated it in xen/arch/<arch>, .asm-offsets.d would also end up
>>>> there, and hence including of it would need separately taking care
>>>> of.
>>>
>>> I'm not sure I understand what do you mean with inclusion need taking
>>> care of separately. Isn't it expected for .d file to reside in the
>>> same directory as the output,
>>
>> Yes, ...
>>
>>> and hence picked up automatically?
>>
>> ... and hence they _wouldn't_ be picked up automatically while
>> building in xen/ if the .s file got created in xen/arch/<arch>/.
> 
> Hm, so that would prevent re-building the target when the dependencies
> changed as the .d file being in the arch directory would attempt the
> rebuild from there instead of the top level xen/?

No, in the arch directory nothing should happen at all, as there's
no rule to rebuild asm-offsets.s. And if we built it in the subarch
directory (where asm-offsets.c lives), the wrong rule would kick in
(the general one compiling C to assembly).

> I guess the alternative would be to force a rebuild of the target
> every time, in order to not depend on the .d logic?

Simply rebuilding always is not going to be good: There should be
no re-building at all when actually just installing Xen. Hence
while move-if-changed would be able to suppress the bulk of the
fallout, this would still be too much rebuilding for my taste in
that specific case.

The option I've been hinting at was to explicitly include the .d
files from the arch dir. But I don't really like this either ...

Jan

Re: [PATCH] build: centralize / unify asm-offsets generation

[Roger Pau Monné]

On Tue, Apr 27, 2021 at 03:40:24PM +0200, Jan Beulich wrote:
> On 27.04.2021 15:31, Roger Pau Monné wrote:
> > On Tue, Apr 27, 2021 at 02:30:35PM +0200, Jan Beulich wrote:
> >> On 27.04.2021 11:05, Roger Pau Monné wrote:
> >>> On Wed, Apr 21, 2021 at 04:09:03PM +0200, Jan Beulich wrote:
> >>>> On 20.04.2021 18:20, Roger Pau Monné wrote:
> >>>>> On Tue, Apr 20, 2021 at 05:47:49PM +0200, Jan Beulich wrote:
> >>>>>> On 20.04.2021 17:29, Roger Pau Monné wrote:
> >>>>>>> On Thu, Apr 01, 2021 at 10:33:47AM +0200, Jan Beulich wrote:
> >>>>>>>> @@ -399,7 +399,11 @@ include/xen/compile.h: include/xen/compi
> >>>>>>>>  	@sed -rf tools/process-banner.sed < .banner >> $@.new
> >>>>>>>>  	@mv -f $@.new $@
> >>>>>>>>  
> >>>>>>>> -include/asm-$(TARGET_ARCH)/asm-offsets.h: arch/$(TARGET_ARCH)/asm-offsets.s
> >>>>>>>> +asm-offsets.s: arch/$(TARGET_ARCH)/$(TARGET_SUBARCH)/asm-offsets.c
> >>>>>>>> +	$(CC) $(filter-out -Wa$(comma)% -flto,$(c_flags)) -S -g0 -o $@.new -MQ $@ $<
> >>>>>>>> +	$(call move-if-changed,$@.new,$@)
> >>>>>>>
> >>>>>>> Won't it be more natural to keep the .s file in arch/$(TARGET_ARCH)?
> >>>>>>
> >>>>>> Yes and no: Yes as far as the actual file location is concerned.
> >>>>>> No when considering where it gets generated: I generally consider
> >>>>>> it risky to generate files outside of the directory where make
> >>>>>> currently runs. There may be good reasons for certain exceptions,
> >>>>>> but personally I don't see this case as good enough a reason.
> >>>>>>
> >>>>>> Somewhat related - if doing as you suggest, which Makefile's
> >>>>>> clean: rule should clean up that file in your opinion?
> >>>>>
> >>>>> The clean rule should be in the makefile where it's generated IMO,
> >>>>> same as asm-offsets.h clean rule currently in xen/Makefile.
> >>>>>
> >>>>>> Nevertheless, if there's general agreement that keeping the file
> >>>>>> there is better, I'd make the change and simply ignore my unhappy
> >>>>>> feelings about it.
> >>>>>
> >>>>> I don't have a strong opinion, it just feels weird to have this IMO
> >>>>> stray asm-offsets.s outside of it's arch directory, taking into
> >>>>> account that we have asm-offsets.h generated from xen/Makefile into an
> >>>>> arch specific directory already as a precedent in that makefile.
> >>>>
> >>>> Well, asm-offsets.h generation doesn't involve the compiler, hence
> >>>> no .*.d files get generated and want including later. For
> >>>> asm-offsets.s to have dependencies properly honored, if we
> >>>> generated it in xen/arch/<arch>, .asm-offsets.d would also end up
> >>>> there, and hence including of it would need separately taking care
> >>>> of.
> >>>
> >>> I'm not sure I understand what do you mean with inclusion need taking
> >>> care of separately. Isn't it expected for .d file to reside in the
> >>> same directory as the output,
> >>
> >> Yes, ...
> >>
> >>> and hence picked up automatically?
> >>
> >> ... and hence they _wouldn't_ be picked up automatically while
> >> building in xen/ if the .s file got created in xen/arch/<arch>/.
> > 
> > Hm, so that would prevent re-building the target when the dependencies
> > changed as the .d file being in the arch directory would attempt the
> > rebuild from there instead of the top level xen/?
> 
> No, in the arch directory nothing should happen at all, as there's
> no rule to rebuild asm-offsets.s. And if we built it in the subarch
> directory (where asm-offsets.c lives), the wrong rule would kick in
> (the general one compiling C to assembly).
> 
> > I guess the alternative would be to force a rebuild of the target
> > every time, in order to not depend on the .d logic?
> 
> Simply rebuilding always is not going to be good: There should be
> no re-building at all when actually just installing Xen. Hence
> while move-if-changed would be able to suppress the bulk of the
> fallout, this would still be too much rebuilding for my taste in
> that specific case.
> 
> The option I've been hinting at was to explicitly include the .d
> files from the arch dir. But I don't really like this either ...

It's hard to tell whether I would prefer that option without seeing
it. In any case, the change is an improvement overall as the logic
gets shared between architectures, so I don't plan to hold it just
because of the placement nit:

Acked-by: Roger Pau Monné <roger.pau@citrix.com>

Thanks, Roger.

Ping: [PATCH] build: centralize / unify asm-offsets generation

[Jan Beulich]

On 27.04.2021 16:13, Roger Pau Monné wrote:
> Acked-by: Roger Pau Monné <roger.pau@citrix.com>

Thanks Roger.

Julien, Stefano, may I ask for an Arm side ack (or otherwise) here
as well?

Thanks, Jan

Re: Ping: [PATCH] build: centralize / unify asm-offsets generation

[Julien Grall]

Hi Jan,

Sorry for the late answer.

On 29/04/2021 10:18, Jan Beulich wrote:
> On 27.04.2021 16:13, Roger Pau Monné wrote:
>> Acked-by: Roger Pau Monné <roger.pau@citrix.com>
> 
> Thanks Roger.
> 
> Julien, Stefano, may I ask for an Arm side ack (or otherwise) here
> as well?

Acked-by: Julien Grall <jgrall@amazon.com>

I will let you commit the patch because I wasn't sure if there was some 
tweaks required based on the conversation between Roger and you.

Cheers,

-- 
Julien Grall