[PATCH] kbuild: add documentation of KBUILD_BUILD_VERSION

[PATCH] kbuild: add documentation of KBUILD_BUILD_VERSION

[Nicholas Mc Guire]

KBUILD_BUILD_VERSION is currently not documented but it is
needed when rebuilding a kernel that should result in the identical
binary. This is a brief documentation of KBUILD_BUILD_VERSION.

Signed-off-by: Nicholas Mc Guire <hofrat@osadl.org>
---

Patch is against 4.0-rc6 (localversion-next is -next-20150402)

 Documentation/kbuild/kbuild.txt |    8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/Documentation/kbuild/kbuild.txt b/Documentation/kbuild/kbuild.txt
index 6466704..7ad01d9 100644
--- a/Documentation/kbuild/kbuild.txt
+++ b/Documentation/kbuild/kbuild.txt
@@ -213,6 +213,14 @@ UTS_VERSION definition (uname -v in the running kernel). The value has to
 be a string that can be passed to date -d. The default value
 is the output of the date command at one point during build.
 
+KBUILD_BUILD_VERSION
+--------------------------------------------------
+Setting this to an integer overrides the build version number used in the
+UTS_VERSION definition (uname -v in the running kernel shown as e.g. #1).
+The default value is set to 0 if .version does not yet exist and is
+generated by incrementing the current value in .version by one if it
+already exists.
+
 KBUILD_BUILD_USER, KBUILD_BUILD_HOST
 --------------------------------------------------
 These two variables allow to override the user@host string displayed during
-- 
1.7.10.4

Re: [PATCH] kbuild: add documentation of KBUILD_BUILD_VERSION

[Jonathan Corbet]

On Sun,  5 Apr 2015 08:44:28 +0200
Nicholas Mc Guire <hofrat@osadl.org> wrote:

> KBUILD_BUILD_VERSION is currently not documented but it is
> needed when rebuilding a kernel that should result in the identical
> binary. This is a brief documentation of KBUILD_BUILD_VERSION.

Can we add something like the above to the document itself so that
readers have an idea of why they might want to tweak this?

Either way, I can take it in the docs tree if that's best..Michal?

Thanks,

jon

Re: [PATCH] kbuild: add documentation of KBUILD_BUILD_VERSION

[Nicholas Mc Guire]

On Sat, 11 Apr 2015, Jonathan Corbet wrote:

> On Sun,  5 Apr 2015 08:44:28 +0200
> Nicholas Mc Guire <hofrat@osadl.org> wrote:
> 
> > KBUILD_BUILD_VERSION is currently not documented but it is
> > needed when rebuilding a kernel that should result in the identical
> > binary. This is a brief documentation of KBUILD_BUILD_VERSION.
> 
> Can we add something like the above to the document itself so that
> readers have an idea of why they might want to tweak this?
> 
> Either way, I can take it in the docs tree if that's best..Michal?
>
I thought of that but it would be inconsistent as all other descriptions
here are only the function not the use. I did not find a file where 
the problem of identical rebuild would really fit.

thx!
hofrat 

Re: [PATCH] kbuild: add documentation of KBUILD_BUILD_VERSION

[Michal Marek]

Dne 11.4.2015 v 15:20 Nicholas Mc Guire napsal(a):
> On Sat, 11 Apr 2015, Jonathan Corbet wrote:
> 
>> On Sun,  5 Apr 2015 08:44:28 +0200
>> Nicholas Mc Guire <hofrat@osadl.org> wrote:
>>
>>> KBUILD_BUILD_VERSION is currently not documented but it is
>>> needed when rebuilding a kernel that should result in the identical
>>> binary. This is a brief documentation of KBUILD_BUILD_VERSION.
>>
>> Can we add something like the above to the document itself so that
>> readers have an idea of why they might want to tweak this?
>>
>> Either way, I can take it in the docs tree if that's best..Michal?
>>
> I thought of that but it would be inconsistent as all other descriptions
> here are only the function not the use.

Most of the entries in this file predate efforts at deterministic
builds, so I'd prefer usefulness over consistency here :-).


> I did not find a file where 
> the problem of identical rebuild would really fit.

... unless, of course, you want to start a new file covering this topic.
Because it's not just the few override variables, but also some options
have to be turned off (I remember CONFIG_GCOV_KERNEL) and the paths must
be the same, or relative paths must be used.

Michal

Re: [PATCH] kbuild: add documentation of KBUILD_BUILD_VERSION

[Nicholas Mc Guire]

On Sat, 11 Apr 2015, Michal Marek wrote:

> Dne 11.4.2015 v 15:20 Nicholas Mc Guire napsal(a):
> > On Sat, 11 Apr 2015, Jonathan Corbet wrote:
> > 
> >> On Sun,  5 Apr 2015 08:44:28 +0200
> >> Nicholas Mc Guire <hofrat@osadl.org> wrote:
> >>
> >>> KBUILD_BUILD_VERSION is currently not documented but it is
> >>> needed when rebuilding a kernel that should result in the identical
> >>> binary. This is a brief documentation of KBUILD_BUILD_VERSION.
> >>
> >> Can we add something like the above to the document itself so that
> >> readers have an idea of why they might want to tweak this?
> >>
> >> Either way, I can take it in the docs tree if that's best..Michal?
> >>
> > I thought of that but it would be inconsistent as all other descriptions
> > here are only the function not the use.
> 
> Most of the entries in this file predate efforts at deterministic
> builds, so I'd prefer usefulness over consistency here :-).
> 
> 
> > I did not find a file where 
> > the problem of identical rebuild would really fit.
> 
> ... unless, of course, you want to start a new file covering this topic.
> Because it's not just the few override variables, but also some options
> have to be turned off (I remember CONFIG_GCOV_KERNEL) and the paths must
> be the same, or relative paths must be used.
>
The identical path really only should be needed for GCOV as it compiles
path info into the binary - are you aware of any other cnfig option that
would need the same path ?

It could be in a short document but it could also just be placed right
under the descriptions of the relevant variables - something like:


Use of KBUILD_BUILD_{TIMESTAMP,VERSION,USER,HOST}
--------------------------------------------------

When the kernel is built, Kbuild stores the build timestamp, version, user
and hostname in include/generated/compile.h. By default this information
is extracted from the build-environment on every build. To override this
behavior and rebuild an identical binary kernel, given a running kernel,
the original compile time information needs to be passed to Kbuild.

Prerequisites:
 * Original kernel config available (e.g extract it from /proc/config.gz)
 * Identical source tree
 * Identical toolchain
 * Same source path if building with CONFIG_GCOV_KERNEL
 * Module signature (CONFIG_MODULE_SIG) must be disabled

The build process for rebuilding the identical binary kernel can extract the
remaining necessary information from dmesg and set the Kbuild and make env:

from the boot log or dmesg in the running system:
[    0.000000] Linux version 4.0.0-rc6-sil2 (hofrat@debian) (gcc version 4.7.2
   (Debian 4.7.2-5) ) #0 SMP Mon Apr 6 01:49:56 EDT 2015
                       ^     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
KBUILD_BUILD_VERSION---'                  |
     KBUILD_BUILD_TIMESTAMP---------------'

$ export KBUILD_BUILD_VERSION=0
$ export KBUILD_BUILD_TIMESTAMP="Mon Apr 6 01:49:56 EDT 2015"
$ export KBUILD_BUILD_USER=hofrat
$ export KBUILD_BUILD_HOST=debian
$ git reset --hard v4.0-rc6
$ zcat /proc/config.gz > .config
$ make oldconfig
$ make -j 8 EXTRAVERSION=-rc6-sil2



probably not quite sufficient yet - but atleast for 
defconfigs and a few other trials it worked for me.

If that makes sense to cleanup I'll try to figure out what
other config options could cause problems and submit it
as a patch to kbuild.txt

thx!
hofrat

Re: [PATCH] kbuild: add documentation of KBUILD_BUILD_VERSION

[Michal Marek]

On 2015-04-12 06:08, Nicholas Mc Guire wrote:
> On Sat, 11 Apr 2015, Michal Marek wrote:
> 
>> Dne 11.4.2015 v 15:20 Nicholas Mc Guire napsal(a):
>>> On Sat, 11 Apr 2015, Jonathan Corbet wrote:
>>>
>>>> On Sun,  5 Apr 2015 08:44:28 +0200
>>>> Nicholas Mc Guire <hofrat@osadl.org> wrote:
>>>>
>>>>> KBUILD_BUILD_VERSION is currently not documented but it is
>>>>> needed when rebuilding a kernel that should result in the identical
>>>>> binary. This is a brief documentation of KBUILD_BUILD_VERSION.
>>>>
>>>> Can we add something like the above to the document itself so that
>>>> readers have an idea of why they might want to tweak this?
>>>>
>>>> Either way, I can take it in the docs tree if that's best..Michal?
>>>>
>>> I thought of that but it would be inconsistent as all other descriptions
>>> here are only the function not the use.
>>
>> Most of the entries in this file predate efforts at deterministic
>> builds, so I'd prefer usefulness over consistency here :-).
>>
>>
>>> I did not find a file where 
>>> the problem of identical rebuild would really fit.
>>
>> ... unless, of course, you want to start a new file covering this topic.
>> Because it's not just the few override variables, but also some options
>> have to be turned off (I remember CONFIG_GCOV_KERNEL) and the paths must
>> be the same, or relative paths must be used.
>>
> The identical path really only should be needed for GCOV as it compiles
> path info into the binary - are you aware of any other cnfig option that
> would need the same path ?

The paths appear in BUG() messages if you are using O=<path>. This can
be solver by either not using O= or using O=<subdir>, in which case
kbuild uses '..' as path to the srctree.

IIRC the problem with GCOV was that gcc was adding a randomly named
symbol to each object file.


> It could be in a short document but it could also just be placed right
> under the descriptions of the relevant variables - something like:
> 
> 
> Use of KBUILD_BUILD_{TIMESTAMP,VERSION,USER,HOST}
> --------------------------------------------------
> 
> When the kernel is built, Kbuild stores the build timestamp, version, user
> and hostname in include/generated/compile.h. By default this information
> is extracted from the build-environment on every build. To override this
> behavior and rebuild an identical binary kernel, given a running kernel,
> the original compile time information needs to be passed to Kbuild.
> 
> Prerequisites:
>  * Original kernel config available (e.g extract it from /proc/config.gz)
>  * Identical source tree
>  * Identical toolchain
>  * Same source path if building with CONFIG_GCOV_KERNEL
>  * Module signature (CONFIG_MODULE_SIG) must be disabled

I think that only MODULE_SIG_ALL is problematic, the signature checking
code itself is OK.


> If that makes sense to cleanup I'll try to figure out what
> other config options could cause problems and submit it
> as a patch to kbuild.txt

Michal

Re: [PATCH] kbuild: add documentation of KBUILD_BUILD_VERSION

[Nicholas Mc Guire]

On Mon, 13 Apr 2015, Michal Marek wrote:

> On 2015-04-12 06:08, Nicholas Mc Guire wrote:
> > On Sat, 11 Apr 2015, Michal Marek wrote:
> > 
> >> Dne 11.4.2015 v 15:20 Nicholas Mc Guire napsal(a):
> >>> On Sat, 11 Apr 2015, Jonathan Corbet wrote:
> >>>
> >>>> On Sun,  5 Apr 2015 08:44:28 +0200
> >>>> Nicholas Mc Guire <hofrat@osadl.org> wrote:
> >>>>
> >>>>> KBUILD_BUILD_VERSION is currently not documented but it is
> >>>>> needed when rebuilding a kernel that should result in the identical
> >>>>> binary. This is a brief documentation of KBUILD_BUILD_VERSION.
> >>>>
> >>>> Can we add something like the above to the document itself so that
> >>>> readers have an idea of why they might want to tweak this?
> >>>>
> >>>> Either way, I can take it in the docs tree if that's best..Michal?
> >>>>
> >>> I thought of that but it would be inconsistent as all other descriptions
> >>> here are only the function not the use.
> >>
> >> Most of the entries in this file predate efforts at deterministic
> >> builds, so I'd prefer usefulness over consistency here :-).
> >>
> >>
> >>> I did not find a file where 
> >>> the problem of identical rebuild would really fit.
> >>
> >> ... unless, of course, you want to start a new file covering this topic.
> >> Because it's not just the few override variables, but also some options
> >> have to be turned off (I remember CONFIG_GCOV_KERNEL) and the paths must
> >> be the same, or relative paths must be used.
> >>
> > The identical path really only should be needed for GCOV as it compiles
> > path info into the binary - are you aware of any other cnfig option that
> > would need the same path ?
> 
> The paths appear in BUG() messages if you are using O=<path>. This can
> be solver by either not using O= or using O=<subdir>, in which case
> kbuild uses '..' as path to the srctree.
> 
> IIRC the problem with GCOV was that gcc was adding a randomly named
> symbol to each object file.
>

well it also simply compiles the absolute path into the binary
so that is in the checksum.
 
> 
> > It could be in a short document but it could also just be placed right
> > under the descriptions of the relevant variables - something like:
> > 
> > 
> > Use of KBUILD_BUILD_{TIMESTAMP,VERSION,USER,HOST}
> > --------------------------------------------------
> > 
> > When the kernel is built, Kbuild stores the build timestamp, version, user
> > and hostname in include/generated/compile.h. By default this information
> > is extracted from the build-environment on every build. To override this
> > behavior and rebuild an identical binary kernel, given a running kernel,
> > the original compile time information needs to be passed to Kbuild.
> > 
> > Prerequisites:
> >  * Original kernel config available (e.g extract it from /proc/config.gz)
> >  * Identical source tree
> >  * Identical toolchain
> >  * Same source path if building with CONFIG_GCOV_KERNEL
> >  * Module signature (CONFIG_MODULE_SIG) must be disabled
> 
> I think that only MODULE_SIG_ALL is problematic, the signature checking
> code itself is OK.
>

thanks - will check that - not sure if I had _SIG AND _ALL on while testing

thanks for the notes - running a randbuild script to see if this holds or not
should give atleast some level of confidence that the
exeptions are complete.

thx!
hofrat