implement a stable 'Last updated' in Documentation

implement a stable 'Last updated' in Documentation

[Olaf Hering]

Several files in Documentation have an unstable 'Last updated' timestamp. The
reason is that their mtime changes every time, which prevents reproducible
builds.

341 technical/api-index.txt: technical/api-index-skel.txt \
342         technical/api-index.sh $(patsubst %,%.txt,$(API_DOCS))
343         $(QUIET_GEN)cd technical && '$(SHELL_PATH_SQ)' ./api-index.sh

388 howto-index.txt: howto-index.sh $(wildcard howto/*.txt)
389         $(QUIET_GEN)$(RM) $@+ $@ && \
390         '$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) >$@+ && \
391         mv $@+ $@

399 $(patsubst %.txt,%.html,$(wildcard howto/*.txt)): %.html : %.txt
400         $(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
401         sed -e '1,/^$$/d' $< | \
402         $(TXT_TO_HTML) - >$@+ && \
403         mv $@+ $@

What file timestamp should be used for them? Likely "../version"?
The final file, before passing it to asciidoc, should get a fixed timestamp
with 'touch -r $reference_file $file'.

Olaf

Re: implement a stable 'Last updated' in Documentation

[Olaf Hering]

On Mon, Jan 26, Olaf Hering wrote:

> Several files in Documentation have an unstable 'Last updated' timestamp. The
> reason is that their mtime changes every time, which prevents reproducible
> builds.
> 
> 341 technical/api-index.txt: technical/api-index-skel.txt \
> 342         technical/api-index.sh $(patsubst %,%.txt,$(API_DOCS))
> 343         $(QUIET_GEN)cd technical && '$(SHELL_PATH_SQ)' ./api-index.sh
> 
> 388 howto-index.txt: howto-index.sh $(wildcard howto/*.txt)
> 389         $(QUIET_GEN)$(RM) $@+ $@ && \
> 390         '$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) >$@+ && \
> 391         mv $@+ $@
> 
> 399 $(patsubst %.txt,%.html,$(wildcard howto/*.txt)): %.html : %.txt
> 400         $(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
> 401         sed -e '1,/^$$/d' $< | \
> 402         $(TXT_TO_HTML) - >$@+ && \
> 403         mv $@+ $@
> 
> What file timestamp should be used for them? Likely "../version"?
> The final file, before passing it to asciidoc, should get a fixed timestamp
> with 'touch -r $reference_file $file'.

I came up with this change. If anything happens to depend on
howto-index.txt and technical/api-index.txt then this part needs rework.

diff --git a/Documentation/Makefile b/Documentation/Makefile
index 3e39e28..25ad0bd 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -6,6 +6,7 @@ TECH_DOCS =
 ARTICLES =
 SP_ARTICLES =
 OBSOLETE_HTML =
+TIMESTAMP_FILE = ../version
 
 MAN1_TXT += $(filter-out \
 		$(addsuffix .txt, $(ARTICLES) $(SP_ARTICLES)), \
@@ -341,6 +342,7 @@ user-manual.xml: user-manual.txt user-manual.conf
 technical/api-index.txt: technical/api-index-skel.txt \
 	technical/api-index.sh $(patsubst %,%.txt,$(API_DOCS))
 	$(QUIET_GEN)cd technical && '$(SHELL_PATH_SQ)' ./api-index.sh
+	touch -r '$(TIMESTAMP_FILE)' $@
 
 technical/%.html: ASCIIDOC_EXTRA += -a git-relative-html-prefix=../
 $(patsubst %,%.html,$(API_DOCS) technical/api-index $(TECH_DOCS)): %.html : %.txt asciidoc.conf
@@ -388,6 +390,7 @@ $(patsubst %.txt,%.texi,$(MAN_TXT)): %.texi : %.xml
 howto-index.txt: howto-index.sh $(wildcard howto/*.txt)
 	$(QUIET_GEN)$(RM) $@+ $@ && \
 	'$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) >$@+ && \
+	touch -r '$(TIMESTAMP_FILE)' $@+ && \
 	mv $@+ $@
 
 $(patsubst %,%.html,$(ARTICLES)) : %.html : %.txt
@@ -397,9 +400,12 @@ WEBDOC_DEST = /pub/software/scm/git/docs
 
 howto/%.html: ASCIIDOC_EXTRA += -a git-relative-html-prefix=../
 $(patsubst %.txt,%.html,$(wildcard howto/*.txt)): %.html : %.txt
-	$(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
-	sed -e '1,/^$$/d' $< | \
-	$(TXT_TO_HTML) - >$@+ && \
+	$(QUIET_ASCIIDOC)$(RM) $@++ $@+ $@ && \
+	sed -e '1,/^$$/d' $< >$@++ && \
+	touch -r '$(TIMESTAMP_FILE)' $@++ && \
+	$(TXT_TO_HTML) -o $@+ $@++ && \
+	$(RM) $@++ && \
+	touch $@+ && \
 	mv $@+ $@
 
 install-webdoc : html

Olaf

Re: implement a stable 'Last updated' in Documentation

[Junio C Hamano]

Olaf Hering <olaf@aepfle.de> writes:

> Several files in Documentation have an unstable 'Last updated' timestamp. The
> reason is that their mtime changes every time, which prevents reproducible
> builds.
>
> 341 technical/api-index.txt: technical/api-index-skel.txt \
> 342         technical/api-index.sh $(patsubst %,%.txt,$(API_DOCS))
> 343         $(QUIET_GEN)cd technical && '$(SHELL_PATH_SQ)' ./api-index.sh
>
> 388 howto-index.txt: howto-index.sh $(wildcard howto/*.txt)
> 389         $(QUIET_GEN)$(RM) $@+ $@ && \
> 390         '$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) >$@+ && \
> 391         mv $@+ $@
>
> 399 $(patsubst %.txt,%.html,$(wildcard howto/*.txt)): %.html : %.txt
> 400         $(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
> 401         sed -e '1,/^$$/d' $< | \
> 402         $(TXT_TO_HTML) - >$@+ && \
> 403         mv $@+ $@

All of them seem to have dependencies so it seems to me that two
builds back to back without actually changing their input would not
re-build anything.  What am I missing???

> What file timestamp should be used for them? Likely "../version"?

I tend to think the "Last updated" timestamp taken from the
filesystem timestamp is a bad practice inherited by these tools from
the days back when nobody used any revision control systems.

If I check out v1.8.5 and build documentation now, it does not
matter if the generated documentation had the timestamp of the
checkout of individual files or that of a single file generated
during the build process, i.e. ../version.  Neither has much
relevance to the time the actual contents of the documentation was
prepared or what vintage of the software the documentation is for.

I am fine with branding generated documentation with the version
number (i.e. "You are reading the documentation for Git version
2.2.0").  Replacing that statement with "You are reading the
documentation for a version of Git that was committed on such and
such time" is also fine.  But using file timestamp would not help
either.

And that practice of using file timestamp is doubly bad by leading
misguided people to want to set timestamps of all the files by
copying commit timestamp.  That is backwards, I would think.

Re: implement a stable 'Last updated' in Documentation

[Olaf Hering]

On Tue, Jan 27, Junio C Hamano wrote:

> Olaf Hering <olaf@aepfle.de> writes:
> 
> > Several files in Documentation have an unstable 'Last updated' timestamp. The
> > reason is that their mtime changes every time, which prevents reproducible
> > builds.
> >
> > 341 technical/api-index.txt: technical/api-index-skel.txt \
> > 342         technical/api-index.sh $(patsubst %,%.txt,$(API_DOCS))
> > 343         $(QUIET_GEN)cd technical && '$(SHELL_PATH_SQ)' ./api-index.sh
> >
> > 388 howto-index.txt: howto-index.sh $(wildcard howto/*.txt)
> > 389         $(QUIET_GEN)$(RM) $@+ $@ && \
> > 390         '$(SHELL_PATH_SQ)' ./howto-index.sh $(sort $(wildcard howto/*.txt)) >$@+ && \
> > 391         mv $@+ $@
> >
> > 399 $(patsubst %.txt,%.html,$(wildcard howto/*.txt)): %.html : %.txt
> > 400         $(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
> > 401         sed -e '1,/^$$/d' $< | \
> > 402         $(TXT_TO_HTML) - >$@+ && \
> > 403         mv $@+ $@
> 
> All of them seem to have dependencies so it seems to me that two
> builds back to back without actually changing their input would not
> re-build anything.  What am I missing???

Per default the files referenced do not exist, so a build from releases
will generate the files. But I agree, perhaps there should be some
refactoring to handle dependencies properly.
In any case, if the 'Last updated' is unavoidable some reference is
required, see my patch in the other mail.

> > What file timestamp should be used for them? Likely "../version"?
> 
> I tend to think the "Last updated" timestamp taken from the
> filesystem timestamp is a bad practice inherited by these tools from
> the days back when nobody used any revision control systems.

I'm not sure. The bug is that such 'Last updated' line exists at all in
the default output. Noone asked for it, noone really needs it. And it
makes it impossible to get reproducible builds.
But so far I found no way to avoid the 'Last updated' output. The docs
and google suggest something like :last-updated-time!:, which did not
work for me.

Olaf

Re: implement a stable 'Last updated' in Documentation

[Junio C Hamano]

Olaf Hering <olaf@aepfle.de> writes:

> On Tue, Jan 27, Junio C Hamano wrote:
>
>> > What file timestamp should be used for them? Likely "../version"?
>> 
>> I tend to think the "Last updated" timestamp taken from the
>> filesystem timestamp is a bad practice inherited by these tools from
>> the days back when nobody used any revision control systems.
>
> I'm not sure. The bug is that such 'Last updated' line exists at all in
> the default output. Noone asked for it, noone really needs it. And it
> makes it impossible to get reproducible builds.

Amen to that ;-)

Re: implement a stable 'Last updated' in Documentation

[Michael J Gruber]

Junio C Hamano schrieb am 29.01.2015 um 07:18:
> Olaf Hering <olaf@aepfle.de> writes:
> 
>> On Tue, Jan 27, Junio C Hamano wrote:
>>
>>>> What file timestamp should be used for them? Likely "../version"?
>>>
>>> I tend to think the "Last updated" timestamp taken from the
>>> filesystem timestamp is a bad practice inherited by these tools from
>>> the days back when nobody used any revision control systems.
>>
>> I'm not sure. The bug is that such 'Last updated' line exists at all in
>> the default output. Noone asked for it, noone really needs it. And it
>> makes it impossible to get reproducible builds.
> 
> Amen to that ;-)
> 

It's a shame one can't simply replace the [footer-text] template which
asciidoc insists on.

It turns out asciidoc 8.6.9-3 and later will habe a knob to turn:

https://github.com/asciidoc/asciidoc/pull/9

I'll try and get my hands on it to see whether we can simply use that.
I'm wondering though which is more useful - the version of the tree the
doc is processed from, or the version of the last commit changing the
corresponding doc source file. The first one changes even when the doc
source is unchanged (but is stable between reruns, of course).

Michael

Re: implement a stable 'Last updated' in Documentation

[Jeff King]

On Fri, Jan 30, 2015 at 11:05:36AM +0100, Michael J Gruber wrote:

> It's a shame one can't simply replace the [footer-text] template which
> asciidoc insists on.
> 
> It turns out asciidoc 8.6.9-3 and later will habe a knob to turn:
> 
> https://github.com/asciidoc/asciidoc/pull/9
> 
> I'll try and get my hands on it to see whether we can simply use that.
> I'm wondering though which is more useful - the version of the tree the
> doc is processed from, or the version of the last commit changing the
> corresponding doc source file. The first one changes even when the doc
> source is unchanged (but is stable between reruns, of course).

I have 8.6.9-3 installed (it is part of Debian testing/unstable now),
and confirmed that:

diff --git a/Documentation/asciidoc.conf b/Documentation/asciidoc.conf
index 2c16c53..10c777e 100644
--- a/Documentation/asciidoc.conf
+++ b/Documentation/asciidoc.conf
@@ -21,6 +21,7 @@ tilde=~
 apostrophe='
 backtick=`
 litdd=--
+footer-style=none
 
 ifdef::backend-docbook[]
 [linkgit-inlinemacro]

drops the "last-updated" footer.

But note that this only affects the generated HTML. The manpages still
get the date in their footer. But this isn't an asciidoc-ism at all;
it's added by docbook when converting the xml to roff. I'm sure there is
a way to tweak that, too, but looking at docbook gives me nightmares.

-Peff

Re: implement a stable 'Last updated' in Documentation

[Olaf Hering]

On Fri, Jan 30, Jeff King wrote:

> I have 8.6.9-3 installed (it is part of Debian testing/unstable now),
> and confirmed that:
> 
> diff --git a/Documentation/asciidoc.conf b/Documentation/asciidoc.conf
> index 2c16c53..10c777e 100644
> --- a/Documentation/asciidoc.conf
> +++ b/Documentation/asciidoc.conf
> @@ -21,6 +21,7 @@ tilde=~
>  apostrophe='
>  backtick=`
>  litdd=--
> +footer-style=none
>  
>  ifdef::backend-docbook[]
>  [linkgit-inlinemacro]
> 
> drops the "last-updated" footer.

This does not remove "Last updated" from files like
using-merge-subtree.html for me, using asciidoc-8.6.8.

Olaf

Re: implement a stable 'Last updated' in Documentation

[Jeff King]

On Tue, Feb 10, 2015 at 04:17:47PM +0100, Olaf Hering wrote:

> On Fri, Jan 30, Jeff King wrote:
> 
> > I have 8.6.9-3 installed (it is part of Debian testing/unstable now),
> > and confirmed that:
> > 
> > diff --git a/Documentation/asciidoc.conf b/Documentation/asciidoc.conf
> > index 2c16c53..10c777e 100644
> > --- a/Documentation/asciidoc.conf
> > +++ b/Documentation/asciidoc.conf
> > @@ -21,6 +21,7 @@ tilde=~
> >  apostrophe='
> >  backtick=`
> >  litdd=--
> > +footer-style=none
> >  
> >  ifdef::backend-docbook[]
> >  [linkgit-inlinemacro]
> > 
> > drops the "last-updated" footer.
> 
> This does not remove "Last updated" from files like
> using-merge-subtree.html for me, using asciidoc-8.6.8.

Yes, the feature is part of 8.6.9-3.

-Peff

Re: implement a stable 'Last updated' in Documentation

[Olaf Hering]

Hey, asciidoc made a move, so this patch is good to go:
https://github.com/asciidoc/asciidoc/pull/9

Thanks.

Olaf

On Tue, Feb 10, Jeff King wrote:

> On Tue, Feb 10, 2015 at 04:17:47PM +0100, Olaf Hering wrote:
> 
> > On Fri, Jan 30, Jeff King wrote:
> > 
> > > I have 8.6.9-3 installed (it is part of Debian testing/unstable now),
> > > and confirmed that:
> > > 
> > > diff --git a/Documentation/asciidoc.conf b/Documentation/asciidoc.conf
> > > index 2c16c53..10c777e 100644
> > > --- a/Documentation/asciidoc.conf
> > > +++ b/Documentation/asciidoc.conf
> > > @@ -21,6 +21,7 @@ tilde=~
> > >  apostrophe='
> > >  backtick=`
> > >  litdd=--
> > > +footer-style=none
> > >  
> > >  ifdef::backend-docbook[]
> > >  [linkgit-inlinemacro]
> > > 
> > > drops the "last-updated" footer.
> > 
> > This does not remove "Last updated" from files like
> > using-merge-subtree.html for me, using asciidoc-8.6.8.
> 
> Yes, the feature is part of 8.6.9-3.
> 
> -Peff

Re: implement a stable 'Last updated' in Documentation

[Jeff King]

On Thu, Sep 01, 2016 at 04:37:36PM +0200, Olaf Hering wrote:

> Hey, asciidoc made a move, so this patch is good to go:
> https://github.com/asciidoc/asciidoc/pull/9

Sine this thread is 18 months old, I needed some recap to remember what
we were talking about. :)

It's here:

  http://public-inbox.org/git/20150126172409.GA15204@aepfle.de/T/#u

and the gist of it is that we'd like to drop the "Last updated" footer
from the HTML version of the manpages, but older versions of asciidoc
did not provide a mechanism.

The patch you quoted adds "footer-style=none", which would do the trick.
But I have two open questions:

  1. What does this do on older versions of asciidoc? Is it silently
     ignored (ok), or does it generate an error (bad)?

  2. This covers the HTML versions, but not the roff manpages (which
     are generated by docbook). Do we have a way to tweak the date in
     the latter?

     I don't think that's necessarily a requirement for this patch, but
     it is worth thinking about at the same time.

Assuming the answer to (1) is "ok" and (2) is "no, but it's hard because
docbook is scary, so let's punt", then somebody needs to write up the
commit message and send the actual patch to the list. Would you like to
try that?

-Peff