linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH] scripts: Fix incremental build header re-generation
@ 2021-03-01 14:22 Jeevan Shriram
  2021-03-02  3:36 ` Masahiro Yamada
  0 siblings, 1 reply; 4+ messages in thread
From: Jeevan Shriram @ 2021-03-01 14:22 UTC (permalink / raw)
  To: linux-kbuild; +Cc: Jeevan Shriram, linux-kernel, tsoni

compile.h and autoconf.h are ignored when checking headers sha as they
are always re-generated for every kernel compilation. However,
these two headers are packaged into kheaders tar. During incremental
compilation of kernel, kheaders tar file is always generated and re-packaged
irrespective of the changes in headers.

Change-Id: I7a64faebb81df44c32230b0fea1d6df09d7ce66f
Signed-off-by: Jeevan Shriram <jshriram@codeaurora.org>
---
 kernel/gen_kheaders.sh | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
index c1510f0..5499f72 100755
--- a/kernel/gen_kheaders.sh
+++ b/kernel/gen_kheaders.sh
@@ -51,8 +51,7 @@ this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
 if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
 if [ -f kernel/kheaders.md5 ] &&
 	[ "$(head -n 1 kernel/kheaders.md5)" = "$headers_md5" ] &&
-	[ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ] &&
-	[ "$(tail -n 1 kernel/kheaders.md5)" = "$tarfile_md5" ]; then
+	[ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ]; then
 		exit
 fi
 
-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project


^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: [PATCH] scripts: Fix incremental build header re-generation
  2021-03-01 14:22 [PATCH] scripts: Fix incremental build header re-generation Jeevan Shriram
@ 2021-03-02  3:36 ` Masahiro Yamada
  2021-03-18  2:54   ` Jeevan Shriram
  0 siblings, 1 reply; 4+ messages in thread
From: Masahiro Yamada @ 2021-03-02  3:36 UTC (permalink / raw)
  To: Jeevan Shriram
  Cc: Linux Kbuild mailing list, Linux Kernel Mailing List, Trilok Soni

On Mon, Mar 1, 2021 at 11:23 PM Jeevan Shriram <jshriram@codeaurora.org> wrote:
>
> compile.h and autoconf.h are ignored when checking headers sha as they
> are always re-generated for every kernel compilation. However,
> these two headers are packaged into kheaders tar. During incremental
> compilation of kernel, kheaders tar file is always generated and re-packaged
> irrespective of the changes in headers.


I do not see this problem.
Could you describe the steps to reproduce it, please?







> Change-Id: I7a64faebb81df44c32230b0fea1d6df09d7ce66f
> Signed-off-by: Jeevan Shriram <jshriram@codeaurora.org>
> ---
>  kernel/gen_kheaders.sh | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
> index c1510f0..5499f72 100755
> --- a/kernel/gen_kheaders.sh
> +++ b/kernel/gen_kheaders.sh
> @@ -51,8 +51,7 @@ this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
>  if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
>  if [ -f kernel/kheaders.md5 ] &&
>         [ "$(head -n 1 kernel/kheaders.md5)" = "$headers_md5" ] &&
> -       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ] &&
> -       [ "$(tail -n 1 kernel/kheaders.md5)" = "$tarfile_md5" ]; then
> +       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ]; then
>                 exit
>  fi
>
> --
> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
> a Linux Foundation Collaborative Project
>


-- 
Best Regards
Masahiro Yamada

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH] scripts: Fix incremental build header re-generation
  2021-03-02  3:36 ` Masahiro Yamada
@ 2021-03-18  2:54   ` Jeevan Shriram
  2021-03-24  6:27     ` Masahiro Yamada
  0 siblings, 1 reply; 4+ messages in thread
From: Jeevan Shriram @ 2021-03-18  2:54 UTC (permalink / raw)
  To: Masahiro Yamada
  Cc: Linux Kbuild mailing list, Linux Kernel Mailing List, Trilok Soni


On 3/1/2021 7:36 PM, Masahiro Yamada wrote:
> On Mon, Mar 1, 2021 at 11:23 PM Jeevan Shriram <jshriram@codeaurora.org> wrote:
>> compile.h and autoconf.h are ignored when checking headers sha as they
>> are always re-generated for every kernel compilation. However,
>> these two headers are packaged into kheaders tar. During incremental
>> compilation of kernel, kheaders tar file is always generated and re-packaged
>> irrespective of the changes in headers.
>
> I do not see this problem.
> Could you describe the steps to reproduce it, please?
>
Without making any changes in the kernel or it's headers, re-compile the kernel. i.e.,incremental kernel build without any changes.
I have added following log in gen_kheaders.sh script for confirming the hash differences.

diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
index b7425a0..ee542a0 100755
--- a/kernel/gen_kheaders.sh
+++ b/kernel/gen_kheaders.sh
@@ -40,6 +40,10 @@ obj_files_md5="$(find $dir_list -name "*.h"                     |
  # Any changes to this script will also cause a rebuild of the archive.
  this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
  if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
+
+echo "Old Tar file $tarfile_md5"
+echo "New Tar file hash $this_file_md5"
+
  if [ -f kernel/kheaders.md5 ] &&
         [ "$(cat kernel/kheaders.md5|head -1)" == "$src_files_md5" ] &&
         [ "$(cat kernel/kheaders.md5|head -2|tail -1)" == "$obj_files_md5" ] &&

log output :
89306 19:29:02.109961   CHK     kernel/kheaders_data.tar.xz
89307 19:29:02.109971 Old Tar file 2aa6990e4183c31a862951f4bcac037e
89308 19:29:02.109982 New Tar file hash ecf84e700c7cacfe8b35a0905859582d

>> Change-Id: I7a64faebb81df44c32230b0fea1d6df09d7ce66f
>> Signed-off-by: Jeevan Shriram <jshriram@codeaurora.org>
>> ---
>>   kernel/gen_kheaders.sh | 3 +--
>>   1 file changed, 1 insertion(+), 2 deletions(-)
>>
>> diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
>> index c1510f0..5499f72 100755
>> --- a/kernel/gen_kheaders.sh
>> +++ b/kernel/gen_kheaders.sh
>> @@ -51,8 +51,7 @@ this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
>>   if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
>>   if [ -f kernel/kheaders.md5 ] &&
>>          [ "$(head -n 1 kernel/kheaders.md5)" = "$headers_md5" ] &&
>> -       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ] &&
>> -       [ "$(tail -n 1 kernel/kheaders.md5)" = "$tarfile_md5" ]; then
>> +       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ]; then
>>                  exit
>>   fi
>>
>> --
>> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
>> a Linux Foundation Collaborative Project
>>
>

^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: [PATCH] scripts: Fix incremental build header re-generation
  2021-03-18  2:54   ` Jeevan Shriram
@ 2021-03-24  6:27     ` Masahiro Yamada
  0 siblings, 0 replies; 4+ messages in thread
From: Masahiro Yamada @ 2021-03-24  6:27 UTC (permalink / raw)
  To: Jeevan Shriram
  Cc: Linux Kbuild mailing list, Linux Kernel Mailing List, Trilok Soni

On Thu, Mar 18, 2021 at 11:55 AM Jeevan Shriram <jshriram@codeaurora.org> wrote:
>
>
> On 3/1/2021 7:36 PM, Masahiro Yamada wrote:
> > On Mon, Mar 1, 2021 at 11:23 PM Jeevan Shriram <jshriram@codeaurora.org> wrote:
> >> compile.h and autoconf.h are ignored when checking headers sha as they
> >> are always re-generated for every kernel compilation. However,
> >> these two headers are packaged into kheaders tar. During incremental
> >> compilation of kernel, kheaders tar file is always generated and re-packaged
> >> irrespective of the changes in headers.
> >
> > I do not see this problem.
> > Could you describe the steps to reproduce it, please?
> >
> Without making any changes in the kernel or it's headers, re-compile the kernel. i.e.,incremental kernel build without any changes.
> I have added following log in gen_kheaders.sh script for confirming the hash differences.


Of course they are different because you are comparing
the hashes of two different files.



>
> diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
> index b7425a0..ee542a0 100755
> --- a/kernel/gen_kheaders.sh
> +++ b/kernel/gen_kheaders.sh
> @@ -40,6 +40,10 @@ obj_files_md5="$(find $dir_list -name "*.h"                     |
>   # Any changes to this script will also cause a rebuild of the archive.
>   this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
>   if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
> +
> +echo "Old Tar file $tarfile_md5"


This is the hash of kernel/kheaders_data.tar.xz


> +echo "New Tar file hash $this_file_md5"


This is the hash of kernel/gen_kheaders.sh




> +
>   if [ -f kernel/kheaders.md5 ] &&
>          [ "$(cat kernel/kheaders.md5|head -1)" == "$src_files_md5" ] &&
>          [ "$(cat kernel/kheaders.md5|head -2|tail -1)" == "$obj_files_md5" ] &&
>
> log output :
> 89306 19:29:02.109961   CHK     kernel/kheaders_data.tar.xz
> 89307 19:29:02.109971 Old Tar file 2aa6990e4183c31a862951f4bcac037e
> 89308 19:29:02.109982 New Tar file hash ecf84e700c7cacfe8b35a0905859582d


I do not understand what your claim is.



CHK     kernel/kheaders_data.tar.xz

is displayed when it is checking the hash in order to
determine if the tarball should be updated.


GEN     kernel/kheaders_data.tar.xz

is displayed when the tarball is really updated.





[Incremental build with no change]


masahiro@oscar:~/ref/linux$ make -j24
  DESCEND  objtool
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  CHK     kernel/kheaders_data.tar.xz
Kernel: arch/x86/boot/bzImage is ready  (#2)


[Touch one header and rebuild]

masahiro@oscar:~/ref/linux$ touch include/uapi/drm/exynos_drm.h
masahiro@oscar:~/ref/linux$ make -j24
  DESCEND  objtool
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  CHK     kernel/kheaders_data.tar.xz
  GEN     kernel/kheaders_data.tar.xz
  CC      kernel/kheaders.o
  AR      kernel/built-in.a
  GEN     .version
  CHK     include/generated/compile.h
  UPD     include/generated/compile.h
  CC      init/version.o
  AR      init/built-in.a
  LD      vmlinux.o
  MODPOST vmlinux.symvers
  MODINFO modules.builtin.modinfo
  GEN     modules.builtin
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  SORTTAB vmlinux
  SYSMAP  System.map
  MODPOST Module.symvers
  CC      arch/x86/boot/version.o
  VOFFSET arch/x86/boot/compressed/../voffset.h
  OBJCOPY arch/x86/boot/compressed/vmlinux.bin
  RELOCS  arch/x86/boot/compressed/vmlinux.relocs
  CC      arch/x86/boot/compressed/kaslr.o
  CC      arch/x86/boot/compressed/misc.o
  GZIP    arch/x86/boot/compressed/vmlinux.bin.gz
  MKPIGGY arch/x86/boot/compressed/piggy.S
  AS      arch/x86/boot/compressed/piggy.o
  LD      arch/x86/boot/compressed/vmlinux
  ZOFFSET arch/x86/boot/zoffset.h
  OBJCOPY arch/x86/boot/vmlinux.bin
  AS      arch/x86/boot/header.o
  LD      arch/x86/boot/setup.elf
  OBJCOPY arch/x86/boot/setup.bin
  BUILD   arch/x86/boot/bzImage
Kernel: arch/x86/boot/bzImage is ready  (#3)







> >> Change-Id: I7a64faebb81df44c32230b0fea1d6df09d7ce66f
> >> Signed-off-by: Jeevan Shriram <jshriram@codeaurora.org>
> >> ---
> >>   kernel/gen_kheaders.sh | 3 +--
> >>   1 file changed, 1 insertion(+), 2 deletions(-)
> >>
> >> diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
> >> index c1510f0..5499f72 100755
> >> --- a/kernel/gen_kheaders.sh
> >> +++ b/kernel/gen_kheaders.sh
> >> @@ -51,8 +51,7 @@ this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
> >>   if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
> >>   if [ -f kernel/kheaders.md5 ] &&
> >>          [ "$(head -n 1 kernel/kheaders.md5)" = "$headers_md5" ] &&
> >> -       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ] &&
> >> -       [ "$(tail -n 1 kernel/kheaders.md5)" = "$tarfile_md5" ]; then
> >> +       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ]; then
> >>                  exit
> >>   fi
> >>
> >> --
> >> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
> >> a Linux Foundation Collaborative Project
> >>
> >
--
Best Regards
Masahiro Yamada

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2021-03-24  6:29 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-01 14:22 [PATCH] scripts: Fix incremental build header re-generation Jeevan Shriram
2021-03-02  3:36 ` Masahiro Yamada
2021-03-18  2:54   ` Jeevan Shriram
2021-03-24  6:27     ` Masahiro Yamada

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).