[PATCH 0/3] Improving build reproducibility

[PATCH 0/3] Improving build reproducibility

[Jonathan Nieder]

Hi,

The reproducible builds <https://reproducible-builds.org/> project has
been working on making it possible to verify that binary packages of
open source projects were built from the source they were claimed to
have been built from.

To that end, Debian has been carrying patches 1-2 for a while.  Patch
3 is a Google-internal patch with a related but distinct goal of
making builds go faster.

I think these should be ready to apply.  Thoughts of all kinds welcome.

Sincerely,
Anders Kaseorg (2):
  Documentation: allow overriding timestamps of generated asciidoc
  git-gui: sort entries in optimized tclIndex

Jonathan Nieder (1):
  generate-cmdlist: avoid non-deterministic output

 Documentation/Makefile               | 7 +++++--
 Documentation/technical/api-index.sh | 5 +++++
 generate-cmdlist.sh                  | 2 +-
 git-gui/Makefile                     | 2 +-
 4 files changed, 12 insertions(+), 4 deletions(-)

[PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc

[Jonathan Nieder]

From: Anders Kaseorg <andersk@mit.edu>
Date: Wed, 30 Nov 2016 22:21:15 -0500

Allow overriding the timestamp in generated documentation by setting
SOURCE_DATE_EPOCH to the number of seconds since 1970-01-01 00:00:00
UTC to use.

This makes the generated documentation reproducible from the source
code as long as that variable is set, without losing the last-modified
dates in the default build.

With this change, the package passes Debian's build reproducibility
test (https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal).

The goal is to make it easier to verify that binary packages of open
source projects were built from the source they were claimed to have
been built from.  https://reproducible-builds.org/ has more details.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
Perhaps this should e.g. be taking the latest timestamp of all its
inputs.  That would be straightforward to do, but what's here is what
we've been running with for the past year, so I'd rather stick to it,
at least as a starting point.

Another tweak I'd be interested in is allowing asciidoc to take the
timestamp as a parameter instead of inferring it from mtimes.
Asciidoc accepts an "--attribute footer-style=none" parameter, but I'm
not aware of an "--attribute footer-date=<foo>" parameter to keep the
footer but override its date.

 Documentation/Makefile               | 7 +++++--
 Documentation/technical/api-index.sh | 5 +++++
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/Documentation/Makefile b/Documentation/Makefile
index 2ab65561af..dfec29f36f 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -410,6 +410,7 @@ $(patsubst %.txt,%.texi,$(MAN_TXT)): %.texi : %.xml
 howto-index.txt: howto-index.sh $(wildcard howto/*.txt)
 	$(QUIET_GEN)$(RM) $@+ $@ && \
 	'$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) >$@+ && \
+	$(if $(SOURCE_DATE_EPOCH),touch -d '@$(SOURCE_DATE_EPOCH)' $@+ &&) \
 	mv $@+ $@
 
 $(patsubst %,%.html,$(ARTICLES)) : %.html : %.txt
@@ -420,8 +421,10 @@ WEBDOC_DEST = /pub/software/scm/git/docs
 howto/%.html: ASCIIDOC_EXTRA += -a git-relative-html-prefix=../
 $(patsubst %.txt,%.html,$(wildcard howto/*.txt)): %.html : %.txt
 	$(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
-	sed -e '1,/^$$/d' $< | \
-	$(TXT_TO_HTML) - >$@+ && \
+	sed -e '1,/^$$/d' $< > $<+ && \
+	$(if $(SOURCE_DATE_EPOCH),touch -d '@$(SOURCE_DATE_EPOCH)' $<+ &&) \
+	$(TXT_TO_HTML) -o $@+ $<+ && \
+	rm $<+ && \
 	mv $@+ $@
 
 install-webdoc : html
diff --git a/Documentation/technical/api-index.sh b/Documentation/technical/api-index.sh
index 9c3f4131b8..07b3909627 100755
--- a/Documentation/technical/api-index.sh
+++ b/Documentation/technical/api-index.sh
@@ -20,6 +20,11 @@
 	sed -n -e '/^\/\/ table of contents end/,$p' "$skel"
 ) >api-index.txt+
 
+if test "${SOURCE_DATE_EPOCH:+set}"
+then
+	touch -d "@$SOURCE_DATE_EPOCH" api-index.txt+
+fi
+
 if test -f api-index.txt && cmp api-index.txt api-index.txt+ >/dev/null
 then
 	rm -f api-index.txt+
-- 
2.15.0.448.gf294e3d99a

[PATCH 2/3] git-gui: sort entries in optimized tclIndex

[Jonathan Nieder]

From: Anders Kaseorg <andersk@mit.edu>
Date: Wed, 16 Nov 2016 16:37:17 -0500

auto_mkindex expands wildcards in directory order, which depends on
the underlying filesystem.  To improve build reproducibility, sort the
list of *.tcl files in the Makefile.

The unoptimized loading case was previously fixed in gitgui-0.21.0~14
(git-gui: sort entries in tclIndex, 2015-01-26).

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 918a8de369..f10caedaa7 100644
--- a/Makefile
+++ b/Makefile
@@ -254,7 +254,7 @@ $(ALL_MSGFILES): %.msg : %.po
 lib/tclIndex: $(ALL_LIBFILES) GIT-GUI-VARS
 	$(QUIET_INDEX)if echo \
 	  $(foreach p,$(PRELOAD_FILES),source $p\;) \
-	  auto_mkindex lib '*.tcl' \
+	  auto_mkindex lib $(patsubst lib/%,%,$(sort $(ALL_LIBFILES))) \
 	| $(TCL_PATH) $(QUIET_2DEVNULL); then : ok; \
 	else \
 	 echo >&2 "    * $(TCL_PATH) failed; using unoptimized loading"; \
-- 
2.15.0.448.gf294e3d99a

[PATCH 3/3] generate-cmdlist: avoid non-deterministic output

[Jonathan Nieder]

Date: Fri, 1 Jul 2016 17:32:00 -0700

Non-determinism makes it harder for build tools to discover when a
target needs to be rebuilt.

generate-cmdlist.sh stores the full path in a comment:

 /* Automatically generated by /build/git-agojiD/git-2.15.0/generate-cmdlist.sh */

Use the file name alone instead.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
That's the end of the series.  Thanks for reading.

 generate-cmdlist.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/generate-cmdlist.sh b/generate-cmdlist.sh
index ab0d1b0c06..eeea4b67ea 100755
--- a/generate-cmdlist.sh
+++ b/generate-cmdlist.sh
@@ -1,6 +1,6 @@
 #!/bin/sh
 
-echo "/* Automatically generated by $0 */
+echo "/* Automatically generated by generate-cmdlist.sh */
 struct cmdname_help {
 	char name[16];
 	char help[80];
-- 
2.15.0.448.gf294e3d99a

Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2474 bytes --]

On Tue, Nov 21, 2017 at 03:34:32PM -0800, Jonathan Nieder wrote:
> From: Anders Kaseorg <andersk@mit.edu>
> Date: Wed, 30 Nov 2016 22:21:15 -0500
> 
> Allow overriding the timestamp in generated documentation by setting
> SOURCE_DATE_EPOCH to the number of seconds since 1970-01-01 00:00:00
> UTC to use.
> 
> This makes the generated documentation reproducible from the source
> code as long as that variable is set, without losing the last-modified
> dates in the default build.

Thanks for this.  I had planned on either submitting this patch myself,
or working on a similar one, but I ran into the issue I'll mention
below and hadn't finished looking at it.

My research on this determined that Asciidoctor 1.5.5 and newer handle
this properly, because they honor SOURCE_BUILD_EPOCH.  It's only with
AsciiDoc that this is an issue.

> diff --git a/Documentation/Makefile b/Documentation/Makefile
> index 2ab65561af..dfec29f36f 100644
> --- a/Documentation/Makefile
> +++ b/Documentation/Makefile
> @@ -410,6 +410,7 @@ $(patsubst %.txt,%.texi,$(MAN_TXT)): %.texi : %.xml
>  howto-index.txt: howto-index.sh $(wildcard howto/*.txt)
>  	$(QUIET_GEN)$(RM) $@+ $@ && \
>  	'$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) >$@+ && \
> +	$(if $(SOURCE_DATE_EPOCH),touch -d '@$(SOURCE_DATE_EPOCH)' $@+ &&) \

touch -d @SECONDS isn't POSIX compliant, and non-Linux systems don't
provide it.  POSIX only allows certain fixed format, and I assume that
non-Linux parties (maybe OpenBSD) will want to have reproducible builds
as well.

It's unfortunate for shell users that this variable is in seconds from
the epoch, since there's no portable way to format such a time in shell.
(POSIX doesn't allow date(1) to format anything but the current time.)

My proposed solution was to use Perl to do so, and simply require that
if you wanted a reproducible build, then you had to have Perl.  That
would, of course, require a separate variable in the Makefile holding
the formatted date.

Maybe something like the following in the Makefile:

ifndef NO_PERL
SOURCE_DATE_TIMESTAMP=$(shell perl -MPOSIX -e 'print strftime("%FT%TZ", gmtime($ENV{SOURCE_DATE_EPOCH}));')
endif

and then:

+	$(if $(SOURCE_DATE_TIMESTAMP),touch -d '$(SOURCE_DATE_TIMESTAMP)' $@+ &&) \
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc

[Jonathan Nieder]

Hi,

brian m. carlson wrote:
> On Tue, Nov 21, 2017 at 03:34:32PM -0800, Jonathan Nieder wrote:

>> --- a/Documentation/Makefile
>> +++ b/Documentation/Makefile
>> @@ -410,6 +410,7 @@ $(patsubst %.txt,%.texi,$(MAN_TXT)): %.texi : %.xml
>>  howto-index.txt: howto-index.sh $(wildcard howto/*.txt)
>>  	$(QUIET_GEN)$(RM) $@+ $@ && \
>>  	'$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) >$@+ && \
>> +	$(if $(SOURCE_DATE_EPOCH),touch -d '@$(SOURCE_DATE_EPOCH)' $@+ &&) \
>
> touch -d @SECONDS isn't POSIX compliant, and non-Linux systems don't
> provide it.  POSIX only allows certain fixed format, and I assume that
> non-Linux parties (maybe OpenBSD) will want to have reproducible builds
> as well.

Interesting.  My knee-jerk preference is still to go with this patch
as-is for now, since the non-portability only triggers when
SOURCE_DATE_EPOCH is set.

> It's unfortunate for shell users that this variable is in seconds from
> the epoch, since there's no portable way to format such a time in shell.
> (POSIX doesn't allow date(1) to format anything but the current time.)
>
> My proposed solution was to use Perl to do so, and simply require that
> if you wanted a reproducible build, then you had to have Perl.  That
> would, of course, require a separate variable in the Makefile holding
> the formatted date.
>
> Maybe something like the following in the Makefile:
>
> ifndef NO_PERL
> SOURCE_DATE_TIMESTAMP=$(shell perl -MPOSIX -e 'print strftime("%FT%TZ", gmtime($ENV{SOURCE_DATE_EPOCH}));')
> endif
>
> and then:
>
> +	$(if $(SOURCE_DATE_TIMESTAMP),touch -d '$(SOURCE_DATE_TIMESTAMP)' $@+ &&) \

Neat.  I can play with this a little.

http://asciidoc.org/CHANGELOG.html is stale but asciidoc still seems
to be getting changes at https://github.com/asciidoc/asciidoc.  I
wonder how difficult it would be to add any required SOURCE_DATE_EPOCH
support there.

Longer term, I wonder what it would take to move to a markup language
that is more widely known, like commonmark.

Thanks,
Jonathan

Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

>> touch -d @SECONDS isn't POSIX compliant, and non-Linux systems don't
>> provide it.  POSIX only allows certain fixed format, and I assume that
>> non-Linux parties (maybe OpenBSD) will want to have reproducible builds
>> as well.
>
> Interesting.  My knee-jerk preference is still to go with this patch
> as-is for now, since the non-portability only triggers when
> SOURCE_DATE_EPOCH is set.

As long as this patch is kept Debian-only, that is a sensible stance
to take.  I am not sure (note: this is different from "I do not think")
if it is also OK for the wider public, though.

I wondered if this affects the dirtyness of the build, regardless of
how file timestamps are mucked with.  It turns out that we do not
use "describe --dirty" in the GIT-VERSION-GEN script, so perhaps
it would be OK.

Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc

[Anders Kaseorg]

On Tue, 21 Nov 2017, Jonathan Nieder wrote:
> http://asciidoc.org/CHANGELOG.html is stale but asciidoc still seems
> to be getting changes at https://github.com/asciidoc/asciidoc.  I
> wonder how difficult it would be to add any required SOURCE_DATE_EPOCH
> support there.

In fact I already did (https://github.com/asciidoc/asciidoc/pull/106), 
which is why I’d been holding off on trying to upstream this Git patch.  
The trouble was, the AsciiDoc developers had not been cutting new releases 
“because nobody knows how” 
(https://github.com/asciidoc/asciidoc/issues/103#issuecomment-322077321). 
However, it looks like AsciiDoc 8.6.10 was recently tagged and Debian got 
a 8.6.10-1 package yesterday, so I guess that trouble has been quietly 
resolved.

That should make this Git patch unnecessary.  (You’re of course still 
welcome to take it if you think build reproducibility with old AsciiDoc 
versions is worthwhile.)

Anders

Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc

[Junio C Hamano]

Anders Kaseorg <andersk@mit.edu> writes:

> That should make this Git patch unnecessary.  (You’re of course still 
> welcome to take it if you think build reproducibility with old AsciiDoc 
> versions is worthwhile.)

Thanks.  

I've queued these three only so that I won't lose track, but will
not hastily merge them down (yet) until I hear from people.

Re: [PATCH 1/3] Documentation: allow overriding timestamps of generated asciidoc

[Jonathan Nieder]

Anders Kaseorg wrote:
> On Tue, 21 Nov 2017, Jonathan Nieder wrote:

>> http://asciidoc.org/CHANGELOG.html is stale but asciidoc still seems
>> to be getting changes at https://github.com/asciidoc/asciidoc.  I
>> wonder how difficult it would be to add any required SOURCE_DATE_EPOCH
>> support there.
>
> In fact I already did (https://github.com/asciidoc/asciidoc/pull/106),
> which is why I’d been holding off on trying to upstream this Git patch.
> The trouble was, the AsciiDoc developers had not been cutting new releases
> “because nobody knows how”
> (https://github.com/asciidoc/asciidoc/issues/103#issuecomment-322077321).
> However, it looks like AsciiDoc 8.6.10 was recently tagged and Debian got
> a 8.6.10-1 package yesterday, so I guess that trouble has been quietly
> resolved.

Ah, lovely.  I'll add a build-time dependency on that version to the
Debian package.

Junio, please disregard this patch (patch 1/3).

Thanks,
Jonathan