From mboxrd@z Thu Jan 1 00:00:00 1970 From: Arnout Vandecappelle Date: Thu, 12 May 2016 22:05:36 +0200 Subject: [Buildroot] [PATCH 09/34] reproducibility/libglib2: allow removing codegen In-Reply-To: <20160510194257.GE13285@hermes.click-hack.org> References: <20160430074358.GE1781@hermes.click-hack.org> <1462002570-14706-1-git-send-email-gilles.chanteperdrix@xenomai.org> <1462002570-14706-9-git-send-email-gilles.chanteperdrix@xenomai.org> <88864677-f612-bd6b-3158-5c2980bb9825@mind.be> <20160508202511.GT13285@hermes.click-hack.org> <00b9819d-dbe3-52ce-75f7-6d7a8217f47a@mind.be> <20160510194257.GE13285@hermes.click-hack.org> Message-ID: List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: buildroot@busybox.net On 05/10/16 21:42, Gilles Chanteperdrix wrote: > On Tue, May 10, 2016 at 01:40:30AM +0200, Arnout Vandecappelle wrote: >> On 05/08/16 22:25, Gilles Chanteperdrix wrote: >>> On Sat, May 07, 2016 at 11:04:25PM +0200, Arnout Vandecappelle wrote: >>>> On 04/30/16 09:49, Gilles Chanteperdrix wrote: >>>>> But this is not sufficient, compiling the python bytecode with the >>>>> same interpreter in different environments yields different binaries, >>>> >>>> Er, this is worrisome... You are saying that we don't have a chance in hell of >>>> generating reproducible python bytecode? >>> >>> I am not a python specialist, but it seems to me four bytes in the >>> python generated bytecode are the build timestamp, so unless there >>> is a way to override it with SOURCE_DATE_EPOCH, I do not see that >>> possible. >> >> I've checked the docs. What is saved is the timestamp of the .py file, not the >> build time. > > Mmmm I don't remember. I would run a compilation manually, twice at > a different time, to make sure that the problem is only the file > date. I did that - admittedly with just a few seconds difference. Both in python2.7.11 and python3.5, they were identical when compiling a second time, and different after touch'ing. > >> >> I think it would make sense to run 'touch -d @$(SOURCE_DATE_EPOCH)' on all >> files after patching to capture this aspect. This was also proposed for >> Fedora[1] (though there it was only for the .py files). Not sure what happened >> with that proposal in the end. > > I think I tried running "touch" before compiling, unfortunately > playing with file dates before running "make" is a bit like playing > with fire. For instance with autotools based projects for which > autoreconf is not run, the project may use versions of the autotools > not installed on the user machine, and because of file dates may > want to rerun autoconf or autmake and whine because the right > versions are not available. Doing this only for .py files is much > more reasonable. I tested this as well, with make-3.81 and make-4.04. Both of them do _not_ rebuild if the timestamps are identical. And since the idea is to use touch -d @$(SOURCE_DATE_EPOCH), all timestamps will be identical. But it does indeed mean that if a package has a generated file with an earlier date than the source files, it will now suddenly no longer be rebuilt. I don't know how the various other build systems (waf, scons) will behave. There is indeed a risk, but I'd say we deal with that when it happens. Actually I think we would prefer to detect such issues. Regards, Arnout -- Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF