[PATCH v2] travis-ci: build documentation

[PATCH v2] travis-ci: build documentation

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

Run "make doc" as separate Travis CI build job to check if all
documentation can be built without errors.

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---

diff to v1:
* add quick sanity check to documentation results (thanks Matthieu)
* fix typo in commits message (thanks Stefan)
* move 'make doc' into seperate Travis CI build job (thanks Peff)
* started to move CI helper scripts to /ci (thanks Matthieu & Junio)

I really like the idea of the "/ci" directory. Over time I plan to
migrate all non Travis-CI related code from .travis.yml to this directory.
This would ease the transition to or parallel execution with another CI
system (e.g. GitLab CI to test Git on Windows).

Thanks,
Lars


 .travis.yml              | 15 +++++++++++++++
 ci/test-documentation.sh | 23 +++++++++++++++++++++++
 2 files changed, 38 insertions(+)
 create mode 100755 ci/test-documentation.sh

diff --git a/.travis.yml b/.travis.yml
index 78e433b..9f71d23 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -32,6 +32,21 @@ env:
     # t9816 occasionally fails with "TAP out of sequence errors" on Travis CI OS X
     - GIT_SKIP_TESTS="t9810 t9816"

+matrix:
+  include:
+    - env: Documentation
+      os: linux
+      compiler: clang
+      addons:
+        apt:
+          packages:
+          - asciidoc
+          - xmlto
+      before_install:
+      before_script: make doc
+      script: ci/test-documentation.sh
+      after_failure:
+
 before_install:
   - >
     case "${TRAVIS_OS_NAME:-linux}" in
diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh
new file mode 100755
index 0000000..329ff4b
--- /dev/null
+++ b/ci/test-documentation.sh
@@ -0,0 +1,23 @@
+#!/bin/sh
+#
+# Perform a quick sanity check on documentation generated with 'make doc'.
+#
+
+set -e
+
+test_file_count () {
+    SUFFIX=$1
+    EXPECTED_COUNT=$2
+    ACTUAL_COUNT=$(find Documentation -type f -name "*.$SUFFIX" | wc -l)
+    echo "$ACTUAL_COUNT *.$SUFFIX files found. $EXPECTED_COUNT expected."
+    test $ACTUAL_COUNT -eq $EXPECTED_COUNT
+}
+
+test -s Documentation/git.html
+test -s Documentation/git.xml
+test -s Documentation/git.1
+
+# The follow numbers need to be adjusted when new documentation is added.
+test_file_count html 233
+test_file_count xml 171
+test_file_count 1 152
--
2.5.1

Re: [PATCH v2] travis-ci: build documentation

[Jeff King]

On Fri, Apr 29, 2016 at 11:35:34AM +0200, larsxschneider@gmail.com wrote:

> +# The follow numbers need to be adjusted when new documentation is added.
> +test_file_count html 233
> +test_file_count xml 171
> +test_file_count 1 152

This seems like it will be really flaky and a pain in the future. I'm
not really sure what it's accomplishing, either. The earlier steps would
complain if something failed to render, wouldn't they? At some point we
have to have some faith in "make doc".

-Peff

Re: [PATCH v2] travis-ci: build documentation

[Matthieu Moy]

Jeff King <peff@peff.net> writes:

> On Fri, Apr 29, 2016 at 11:35:34AM +0200, larsxschneider@gmail.com wrote:
>
>> +# The follow numbers need to be adjusted when new documentation is added.
>> +test_file_count html 233
>> +test_file_count xml 171
>> +test_file_count 1 152
>
> This seems like it will be really flaky and a pain in the future. I'm
> not really sure what it's accomplishing, either. The earlier steps would
> complain if something failed to render, wouldn't they? At some point we
> have to have some faith in "make doc".

I agree. My proposal to check for a handful of generated files was just
because this extra paranoia was almost free (just 3 lines of code that
won't need particular maintenance).

In this case, I'm afraid the maintenance cost is much bigger than the
expected benefits.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: [PATCH v2] travis-ci: build documentation

[Lars Schneider]

> On 29 Apr 2016, at 14:21, Matthieu Moy <Matthieu.Moy@grenoble-inp.fr> wrote:
> 
> Jeff King <peff@peff.net> writes:
> 
>> On Fri, Apr 29, 2016 at 11:35:34AM +0200, larsxschneider@gmail.com wrote:
>> 
>>> +# The follow numbers need to be adjusted when new documentation is added.
>>> +test_file_count html 233
>>> +test_file_count xml 171
>>> +test_file_count 1 152
>> 
>> This seems like it will be really flaky and a pain in the future. I'm
>> not really sure what it's accomplishing, either. The earlier steps would
>> complain if something failed to render, wouldn't they? At some point we
>> have to have some faith in "make doc".
> 
> I agree. My proposal to check for a handful of generated files was just
> because this extra paranoia was almost free (just 3 lines of code that
> won't need particular maintenance).
> 
> In this case, I'm afraid the maintenance cost is much bigger than the
> expected benefits.

I agree, too. I wasn't sure about this check. That's why I added
the little comment above to point out the problem.

Should I reroll?

Thanks,
Lars

Re: [PATCH v2] travis-ci: build documentation

[Jeff King]

On Fri, Apr 29, 2016 at 04:22:05PM +0200, Lars Schneider wrote:

> >>> +# The follow numbers need to be adjusted when new documentation is added.
> >>> +test_file_count html 233
> >>> +test_file_count xml 171
> >>> +test_file_count 1 152
> [...]
> I agree, too. I wasn't sure about this check. That's why I added
> the little comment above to point out the problem.
> 
> Should I reroll?

IMHO, yes.

-Peff

Re: [PATCH v2] travis-ci: build documentation

[Matthieu Moy]

larsxschneider@gmail.com writes:

> +      before_install:
> +      before_script: make doc
> +      script: ci/test-documentation.sh

If you are to re-roll, I think before_script and script should be
merged, i.e. just write

script: ci/test-documentation.sh

and have ci/test-documentation.sh be:

#!/bin/sh

set -e

make doc
test -s Documentation/git.html
test -s Documentation/git.xml
test -s Documentation/git.1

In the perspective of using another CI tool, everything within ci/ is
potentially shared between systems so I'd tend to minimize the content
of .travis.yml in favor of ci/*

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: [PATCH v2] travis-ci: build documentation

[Stefan Beller]

On Fri, Apr 29, 2016 at 5:21 AM, Matthieu Moy
<Matthieu.Moy@grenoble-inp.fr> wrote:
> Jeff King <peff@peff.net> writes:
>
>> On Fri, Apr 29, 2016 at 11:35:34AM +0200, larsxschneider@gmail.com wrote:
>>
>>> +# The follow numbers need to be adjusted when new documentation is added.
>>> +test_file_count html 233
>>> +test_file_count xml 171
>>> +test_file_count 1 152
>>
>> This seems like it will be really flaky and a pain in the future. I'm
>> not really sure what it's accomplishing, either. The earlier steps would
>> complain if something failed to render, wouldn't they? At some point we
>> have to have some faith in "make doc".
>
> I agree. My proposal to check for a handful of generated files was just
> because this extra paranoia was almost free (just 3 lines of code that
> won't need particular maintenance).
>
> In this case, I'm afraid the maintenance cost is much bigger than the
> expected benefits.

So you proposed to check a handful files for its exact content?

This could be less of maintenance if we'd check with a "larger as" operator
such as

    test_file_count_more_than html 200

using an arbitrary slightly smaller number.


>
> --
> Matthieu Moy
> http://www-verimag.imag.fr/~moy/
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] travis-ci: build documentation

[Matthieu Moy]

Stefan Beller <sbeller@google.com> writes:

> On Fri, Apr 29, 2016 at 5:21 AM, Matthieu Moy
> <Matthieu.Moy@grenoble-inp.fr> wrote:
>> Jeff King <peff@peff.net> writes:
>>
>>> On Fri, Apr 29, 2016 at 11:35:34AM +0200, larsxschneider@gmail.com wrote:
>>>
>>>> +# The follow numbers need to be adjusted when new documentation is added.
>>>> +test_file_count html 233
>>>> +test_file_count xml 171
>>>> +test_file_count 1 152
>>>
>>> This seems like it will be really flaky and a pain in the future. I'm
>>> not really sure what it's accomplishing, either. The earlier steps would
>>> complain if something failed to render, wouldn't they? At some point we
>>> have to have some faith in "make doc".
>>
>> I agree. My proposal to check for a handful of generated files was just
>> because this extra paranoia was almost free (just 3 lines of code that
>> won't need particular maintenance).
>>
>> In this case, I'm afraid the maintenance cost is much bigger than the
>> expected benefits.
>
> So you proposed to check a handful files for its exact content?

No, just to check that the files exist and are non-empty, i.e. the "test
-s" part of Lars' patch.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: [PATCH v2] travis-ci: build documentation

[Lars Schneider]

On 29 Apr 2016, at 19:27, Stefan Beller <sbeller@google.com> wrote:

> On Fri, Apr 29, 2016 at 5:21 AM, Matthieu Moy
> <Matthieu.Moy@grenoble-inp.fr> wrote:
>> Jeff King <peff@peff.net> writes:
>> 
>>> On Fri, Apr 29, 2016 at 11:35:34AM +0200, larsxschneider@gmail.com wrote:
>>> 
>>>> +# The follow numbers need to be adjusted when new documentation is added.
>>>> +test_file_count html 233
>>>> +test_file_count xml 171
>>>> +test_file_count 1 152
>>> 
>>> This seems like it will be really flaky and a pain in the future. I'm
>>> not really sure what it's accomplishing, either. The earlier steps would
>>> complain if something failed to render, wouldn't they? At some point we
>>> have to have some faith in "make doc".
>> 
>> I agree. My proposal to check for a handful of generated files was just
>> because this extra paranoia was almost free (just 3 lines of code that
>> won't need particular maintenance).
>> 
>> In this case, I'm afraid the maintenance cost is much bigger than the
>> expected benefits.
> 
> So you proposed to check a handful files for its exact content?
> 
> This could be less of maintenance if we'd check with a "larger as" operator
> such as
> 
>    test_file_count_more_than html 200
> 
> using an arbitrary slightly smaller number.

Well, I was thinking about testing against something like 
$(find . -type f -name "git*.txt" | wc -l) but it the end
all of this is not really meaningful I think...

- Lars


> 
> 
>> 
>> --
>> Matthieu Moy
>> http://www-verimag.imag.fr/~moy/
>> --
>> To unsubscribe from this list: send the line "unsubscribe git" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] travis-ci: build documentation

[Junio C Hamano]

Lars Schneider <larsxschneider@gmail.com> writes:

>> This could be less of maintenance if we'd check with a "larger as" operator
>> such as
>> 
>>    test_file_count_more_than html 200
>> 
>> using an arbitrary slightly smaller number.
>
> Well, I was thinking about testing against something like 
> $(find . -type f -name "git*.txt" | wc -l) but it the end
> all of this is not really meaningful I think...

Either is too much, I would say--I have too much faith in the exit
status from "make doc", I guess.

What would _REALLY_ be nice is a check that lets us catch an error
like this deliberate breakage:

    diff --git a/Documentation/diff-options.txt b/Documentation/diff-options.txt
    index 4b0318e..a684f2d 100644
    --- a/Documentation/diff-options.txt
    +++ b/Documentation/diff-options.txt
    @@ -560,4 +560,4 @@ endif::git-format-patch[]
            Do not show any source or destination prefix.

     For more detailed explanation on these common options, see also
    -linkgit:gitdiffcore[7].
    +linkgit:gitdiffnore[7].

We just get a dangling link in the result without an error exit from
"make doc"; neither "test -s" nor "size is about what we expect"
would catch such a breakage, though.

Other things that might be of interest are

    make check-builtins
    make check-docs

but I am not sure if the latter built target is up to date (it has a
whiltelist that needs to stay current).  We rarely add new commands
these days, so it is easy to forget what these build targets try to
check, which makes them good candidates to be thrown into the set of
automated tests.

[PATCH v3 0/2] travis-ci: build documentation

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

diff to v2:
* remove file count check for generated documentation as it is too flaky
* run `make doc` in `test-documentation.sh` to reduce the code in
  .travis.yml and ease the execution in other CI systems
* add linkgit check
* fix existing linkgit errors
* add `make check-builtins` and `make check-docs`

Thanks to Jeff, Matthieu, Stefan, and Junio for the review!

Cheers,
Lars

Lars Schneider (2):
  Documentation: fix linkgit references
  travis-ci: build documentation

 .travis.yml                         | 15 +++++++++++++++
 Documentation/config.txt            |  6 +++---
 Documentation/git-check-ignore.txt  |  2 +-
 Documentation/git-filter-branch.txt |  4 ++--
 Documentation/git-for-each-ref.txt  |  2 +-
 Documentation/git-help.txt          |  4 ++--
 Documentation/git-instaweb.txt      |  4 ++--
 Documentation/git-sh-i18n.txt       |  2 +-
 ci/test-documentation.sh            | 24 ++++++++++++++++++++++++
 9 files changed, 51 insertions(+), 12 deletions(-)
 create mode 100755 ci/test-documentation.sh

--
2.5.1

[PATCH v3 1/2] Documentation: fix linkgit references

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 Documentation/config.txt            | 6 +++---
 Documentation/git-check-ignore.txt  | 2 +-
 Documentation/git-filter-branch.txt | 4 ++--
 Documentation/git-for-each-ref.txt  | 2 +-
 Documentation/git-help.txt          | 4 ++--
 Documentation/git-instaweb.txt      | 4 ++--
 Documentation/git-sh-i18n.txt       | 2 +-
 7 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index c7bbe98..c5f1d6b 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -894,7 +894,7 @@ branch.<name>.description::
 browser.<tool>.cmd::
 	Specify the command to invoke the specified browser. The
 	specified command is evaluated in shell with the URLs passed
-	as arguments. (See linkgit:git-web{litdd}browse[1].)
+	as arguments. (See linkgit:git-web--browse[1].)
 
 browser.<tool>.path::
 	Override the path for the given tool that may be used to
@@ -1494,7 +1494,7 @@ gui.diffContext::
 	made by the linkgit:git-gui[1]. The default is "5".
 
 gui.displayUntracked::
-	Determines if linkgit::git-gui[1] shows untracked files
+	Determines if linkgit:git-gui[1] shows untracked files
 	in the file list. The default is "true".
 
 gui.encoding::
@@ -1665,7 +1665,7 @@ http.cookieFile::
 	File containing previously stored cookie lines which should be used
 	in the Git http session, if they match the server. The file format
 	of the file to read cookies from should be plain HTTP headers or
-	the Netscape/Mozilla cookie file format (see linkgit:curl[1]).
+	the Netscape/Mozilla cookie file format (see `curl(1)`).
 	NOTE that the file specified with http.cookieFile is only used as
 	input unless http.saveCookies is set.
 
diff --git a/Documentation/git-check-ignore.txt b/Documentation/git-check-ignore.txt
index e94367a..9a85998 100644
--- a/Documentation/git-check-ignore.txt
+++ b/Documentation/git-check-ignore.txt
@@ -112,7 +112,7 @@ EXIT STATUS
 SEE ALSO
 --------
 linkgit:gitignore[5]
-linkgit:gitconfig[5]
+linkgit:git-config[5]
 linkgit:git-ls-files[1]
 
 GIT
diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 73fd9e8..6538cb1 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -205,8 +205,8 @@ to other tags will be rewritten to point to the underlying commit.
 Remap to ancestor
 ~~~~~~~~~~~~~~~~~
 
-By using linkgit:rev-list[1] arguments, e.g., path limiters, you can limit the
-set of revisions which get rewritten. However, positive refs on the command
+By using linkgit:git-rev-list[1] arguments, e.g., path limiters, you can limit
+the set of revisions which get rewritten. However, positive refs on the command
 line are distinguished: we don't let them be excluded by such limiters. For
 this purpose, they are instead rewritten to point at the nearest ancestor that
 was not excluded.
diff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt
index c52578b..d9d406d 100644
--- a/Documentation/git-for-each-ref.txt
+++ b/Documentation/git-for-each-ref.txt
@@ -179,7 +179,7 @@ returns an empty string instead.
 
 As a special case for the date-type fields, you may specify a format for
 the date by adding `:` followed by date format name (see the
-values the `--date` option to linkgit::git-rev-list[1] takes).
+values the `--date` option to linkgit:git-rev-list[1] takes).
 
 
 EXAMPLES
diff --git a/Documentation/git-help.txt b/Documentation/git-help.txt
index 3956525..7af18f9 100644
--- a/Documentation/git-help.txt
+++ b/Documentation/git-help.txt
@@ -72,7 +72,7 @@ The web browser can be specified using the configuration variable
 'help.browser', or 'web.browser' if the former is not set. If none of
 these config variables is set, the 'git web{litdd}browse' helper script
 (called by 'git help') will pick a suitable default. See
-linkgit:git-web{litdd}browse[1] for more information about this.
+linkgit:git-web--browse[1] for more information about this.
 
 CONFIGURATION VARIABLES
 -----------------------
@@ -95,7 +95,7 @@ help.browser, web.browser and browser.<tool>.path
 The 'help.browser', 'web.browser' and 'browser.<tool>.path' will also
 be checked if the 'web' format is chosen (either by command-line
 option or configuration variable). See '-w|--web' in the OPTIONS
-section above and linkgit:git-web{litdd}browse[1].
+section above and linkgit:git-web--browse[1].
 
 man.viewer
 ~~~~~~~~~~
diff --git a/Documentation/git-instaweb.txt b/Documentation/git-instaweb.txt
index cc75b25..b95cc15 100644
--- a/Documentation/git-instaweb.txt
+++ b/Documentation/git-instaweb.txt
@@ -46,7 +46,7 @@ OPTIONS
 	The web browser that should be used to view the gitweb
 	page. This will be passed to the 'git web{litdd}browse' helper
 	script along with the URL of the gitweb instance. See
-	linkgit:git-web{litdd}browse[1] for more information about this. If
+	linkgit:git-web--browse[1] for more information about this. If
 	the script fails, the URL will be printed to stdout.
 
 start::
@@ -82,7 +82,7 @@ You may specify configuration in your .git/config
 
 If the configuration variable 'instaweb.browser' is not set,
 'web.browser' will be used instead if it is defined. See
-linkgit:git-web{litdd}browse[1] for more information about this.
+linkgit:git-web--browse[1] for more information about this.
 
 SEE ALSO
 --------
diff --git a/Documentation/git-sh-i18n.txt b/Documentation/git-sh-i18n.txt
index 60cf49c..eafa55a 100644
--- a/Documentation/git-sh-i18n.txt
+++ b/Documentation/git-sh-i18n.txt
@@ -35,7 +35,7 @@ gettext::
 eval_gettext::
 	Currently a dummy fall-through function implemented as a wrapper
 	around `printf(1)` with variables expanded by the
-	linkgit:git-sh-i18n{litdd}envsubst[1] helper. Will be replaced by a
+	linkgit:git-sh-i18n--envsubst[1] helper. Will be replaced by a
 	real gettext implementation in a later version.
 
 GIT
-- 
2.5.1

[PATCH v3 2/2] travis-ci: build documentation

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

Build documentation as separate Travis CI job to check for
documentation errors.

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 .travis.yml              | 15 +++++++++++++++
 ci/test-documentation.sh | 24 ++++++++++++++++++++++++
 2 files changed, 39 insertions(+)
 create mode 100755 ci/test-documentation.sh

diff --git a/.travis.yml b/.travis.yml
index 78e433b..55299bd 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -32,6 +32,21 @@ env:
     # t9816 occasionally fails with "TAP out of sequence errors" on Travis CI OS X
     - GIT_SKIP_TESTS="t9810 t9816"
 
+matrix:
+  include:
+    - env: Documentation
+      os: linux
+      compiler: clang
+      addons:
+        apt:
+          packages:
+          - asciidoc
+          - xmlto
+      before_install:
+      before_script:
+      script: ci/test-documentation.sh
+      after_failure:
+
 before_install:
   - >
     case "${TRAVIS_OS_NAME:-linux}" in
diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh
new file mode 100755
index 0000000..889e6fd
--- /dev/null
+++ b/ci/test-documentation.sh
@@ -0,0 +1,24 @@
+#!/bin/sh
+#
+# Perform sanity checks on documentation and build it.
+#
+
+set -e
+
+LINKS=$(grep --recursive --only-matching --no-filename --perl-regexp \
+    '(?<=linkgit:).*?(?=\[\d+\])' Documentation/* \
+    | sort -u \
+)
+
+for LINK in $LINKS; do
+    echo "Checking linkgit:$LINK..."
+    test -s Documentation/$LINK.txt
+done
+
+make check-builtins
+make check-docs
+make doc
+
+test -s Documentation/git.html
+test -s Documentation/git.xml
+test -s Documentation/git.1
-- 
2.5.1

Re: [PATCH v3 1/2] Documentation: fix linkgit references

[Jeff King]

On Mon, May 02, 2016 at 10:20:04PM +0200, larsxschneider@gmail.com wrote:

> From: Lars Schneider <larsxschneider@gmail.com>
> 
> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
> ---

Fix how? Your commit message doesn't say why this is a good idea. Since
this is v3, I'm guessing that reasoning is on the list, but it needs to
be summarized here in the commit message.

> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index c7bbe98..c5f1d6b 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -894,7 +894,7 @@ branch.<name>.description::
>  browser.<tool>.cmd::
>  	Specify the command to invoke the specified browser. The
>  	specified command is evaluated in shell with the URLs passed
> -	as arguments. (See linkgit:git-web{litdd}browse[1].)
> +	as arguments. (See linkgit:git-web--browse[1].)

The existing code renders fine for me with "make git-config.1". But with
your patch, I get a unicode emdash, which is wrong:

--- old	2016-05-02 16:27:53.242050262 -0400
+++ new	2016-05-02 16:27:57.742050360 -0400
@@ -978,7 +978,7 @@
 
        browser.<tool>.cmd
            Specify the command to invoke the specified browser. The specified command is evaluated in shell with the
-           URLs passed as arguments. (See git-web--browse(1).)
+           URLs passed as arguments. (See git-web—browse(1).)
 
        browser.<tool>.path
            Override the path for the given tool that may be used to browse HTML help (see -w option in git-help(1))

In case it's hard to see with your font, the generated roff looks like
this:

-\fBgit-web--browse\fR(1)\&.)
+\fBgit-web\(embrowse\fR(1)\&.)

So I think that's a step backwards. I did check the asciidoctor
rendering on git-scm.com, though, and it gets the {litdd} case wrong. So
I think it does need fixing, but we need a solution that looks correct
in both cases. Maybe linkgit:`git-web--browse`[1] would work; it seems
OK with my version of asciidoc, but I have a feeling it will run into
the same problem with asciidoctor (if it's not respecting {litdd} in
that context, it's probably also not respecting backticks).

-Peff

Re: [PATCH v3 2/2] travis-ci: build documentation

[Junio C Hamano]

larsxschneider@gmail.com writes:

> +set -e
> +
> +LINKS=$(grep --recursive --only-matching --no-filename --perl-regexp \
> +    '(?<=linkgit:).*?(?=\[\d+\])' Documentation/* \
> +    | sort -u \
> +)
> +
> +for LINK in $LINKS; do
> +    echo "Checking linkgit:$LINK..."
> +    test -s Documentation/$LINK.txt
> +done

Please separate the above link check out of this step and do so
separately after the move of test body to a separate script
settles.

When you reintroduce the tests, please make sure the shell script
follow the coding style of other scripts.  E.g. I do not think the
last one in the $(...) needs a backslash continuation at all.  I am
assuming that you are doing this only on Linux, in which case use of
GNUism with grep may be fine.

> +make check-builtins
> +make check-docs
> +make doc
> +
> +test -s Documentation/git.html
> +test -s Documentation/git.xml
> +test -s Documentation/git.1

Re: [PATCH v3 2/2] travis-ci: build documentation

[Lars Schneider]

On 02 May 2016, at 22:45, Junio C Hamano <gitster@pobox.com> wrote:

> larsxschneider@gmail.com writes:
> 
>> +set -e
>> +
>> +LINKS=$(grep --recursive --only-matching --no-filename --perl-regexp \
>> +    '(?<=linkgit:).*?(?=\[\d+\])' Documentation/* \
>> +    | sort -u \
>> +)
>> +
>> +for LINK in $LINKS; do
>> +    echo "Checking linkgit:$LINK..."
>> +    test -s Documentation/$LINK.txt
>> +done
> 
> Please separate the above link check out of this step and do so
> separately after the move of test body to a separate script
> settles.
OK. I also wonder if the link check should rather go to the 
"check-docs" Makefile target?


> When you reintroduce the tests, please make sure the shell script
> follow the coding style of other scripts.  E.g. I do not think the
> last one in the $(...) needs a backslash continuation at all.  I am
> assuming that you are doing this only on Linux, in which case use of
> GNUism with grep may be fine.
OK.

Thanks for the review,
Lars


> 
>> +make check-builtins
>> +make check-docs
>> +make doc
>> +
>> +test -s Documentation/git.html
>> +test -s Documentation/git.xml
>> +test -s Documentation/git.1

Re: [PATCH v3 1/2] Documentation: fix linkgit references

[Lars Schneider]

On 02 May 2016, at 22:34, Jeff King <peff@peff.net> wrote:

> On Mon, May 02, 2016 at 10:20:04PM +0200, larsxschneider@gmail.com wrote:
> 
>> From: Lars Schneider <larsxschneider@gmail.com>
>> 
>> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
>> ---
> 
> Fix how? Your commit message doesn't say why this is a good idea. Since
> this is v3, I'm guessing that reasoning is on the list, but it needs to
> be summarized here in the commit message.
You are right, I should have explained my thinking a bit more detailed. 
A few of the fixed linkgit references are just typos, e.g.:

-linkgit:gitconfig[5]
+linkgit:git-config[5]

-values the `--date` option to linkgit::git-rev-list[1] takes).
+values the `--date` option to linkgit:git-rev-list[1] takes).

-	the Netscape/Mozilla cookie file format (see linkgit:curl[1]).
+	the Netscape/Mozilla cookie file format (see `curl(1)`).

I mistakenly assumed the "{litdd}" was a typo/bad search replace, too.
I checked this website and thought my change would fix it, too:

https://git-scm.com/docs/git-config

There it is rendered as "(See git-web{litdd}browse[1].)" and the link
is broken.


>> diff --git a/Documentation/config.txt b/Documentation/config.txt
>> index c7bbe98..c5f1d6b 100644
>> --- a/Documentation/config.txt
>> +++ b/Documentation/config.txt
>> @@ -894,7 +894,7 @@ branch.<name>.description::
>> browser.<tool>.cmd::
>> 	Specify the command to invoke the specified browser. The
>> 	specified command is evaluated in shell with the URLs passed
>> -	as arguments. (See linkgit:git-web{litdd}browse[1].)
>> +	as arguments. (See linkgit:git-web--browse[1].)
> 
> The existing code renders fine for me with "make git-config.1". But with
> your patch, I get a unicode emdash, which is wrong:
> 
> --- old	2016-05-02 16:27:53.242050262 -0400
> +++ new	2016-05-02 16:27:57.742050360 -0400
> @@ -978,7 +978,7 @@
> 
>        browser.<tool>.cmd
>            Specify the command to invoke the specified browser. The specified command is evaluated in shell with the
> -           URLs passed as arguments. (See git-web--browse(1).)
> +           URLs passed as arguments. (See git-web—browse(1).)
> 
>        browser.<tool>.path
>            Override the path for the given tool that may be used to browse HTML help (see -w option in git-help(1))
> 
> In case it's hard to see with your font, the generated roff looks like
> this:
> 
> -\fBgit-web--browse\fR(1)\&.)
> +\fBgit-web\(embrowse\fR(1)\&.)

I can confirm. Sorry, I indeed missed that.


> So I think that's a step backwards. I did check the asciidoctor
> rendering on git-scm.com, though, and it gets the {litdd} case wrong. So
> I think it does need fixing, but we need a solution that looks correct
> in both cases. Maybe linkgit:`git-web--browse`[1] would work; it seems
> OK with my version of asciidoc, but I have a feeling it will run into
> the same problem with asciidoctor (if it's not respecting {litdd} in
> that context, it's probably also not respecting backticks).
I will play with this to find a solution. Would it be an option to
replace "--" with "-"? Why do we need two dashes if they cause trouble?

Thanks for the review,
Lars

Re: [PATCH v3 1/2] Documentation: fix linkgit references

[Junio C Hamano]

Lars Schneider <larsxschneider@gmail.com> writes:

> On 02 May 2016, at 22:34, Jeff King <peff@peff.net> wrote:
>
>> On Mon, May 02, 2016 at 10:20:04PM +0200, larsxschneider@gmail.com wrote:
>> 
>>> From: Lars Schneider <larsxschneider@gmail.com>
>>> 
>>> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
>>> ---
>> 
>> Fix how? Your commit message doesn't say why this is a good idea. Since
>> this is v3, I'm guessing that reasoning is on the list, but it needs to
>> be summarized here in the commit message.
> You are right, I should have explained my thinking a bit more detailed. 
> A few of the fixed linkgit references are just typos, e.g.:
>
> -linkgit:gitconfig[5]
> +linkgit:git-config[5]

I didn't run "git blame" yet, so it might be that we used to have it
in section 5, but "git config --help" puts the latter in section 1.

Even though you obviously were fooled by AsciiDoctor regarding
literal double dash {litdd}, other parts of the changes did look
good typofixes.  Thanks for starting this.

Re: [PATCH v3 2/2] travis-ci: build documentation

[Junio C Hamano]

Lars Schneider <larsxschneider@gmail.com> writes:

> On 02 May 2016, at 22:45, Junio C Hamano <gitster@pobox.com> wrote:
>
>> larsxschneider@gmail.com writes:
>> 
>>> +set -e
>>> +
>>> +LINKS=$(grep --recursive --only-matching --no-filename --perl-regexp \
>>> +    '(?<=linkgit:).*?(?=\[\d+\])' Documentation/* \
>>> +    | sort -u \
>>> +)
>>> +
>>> +for LINK in $LINKS; do
>>> +    echo "Checking linkgit:$LINK..."
>>> +    test -s Documentation/$LINK.txt
>>> +done
>> 
>> Please separate the above link check out of this step and do so
>> separately after the move of test body to a separate script
>> settles.
>
> OK. I also wonder if the link check should rather go to the 
> "check-docs" Makefile target?

That sounds like a good direction.

Which in turn means that people on all platforms are welcome to run
it, which in turn means that the script must be even more portable,
with avoiding GNUism and bash-isms etc.

Re: [PATCH v3 1/2] Documentation: fix linkgit references

[Jeff King]

On Tue, May 03, 2016 at 10:30:09AM +0200, Lars Schneider wrote:

> > So I think that's a step backwards. I did check the asciidoctor
> > rendering on git-scm.com, though, and it gets the {litdd} case wrong. So
> > I think it does need fixing, but we need a solution that looks correct
> > in both cases. Maybe linkgit:`git-web--browse`[1] would work; it seems
> > OK with my version of asciidoc, but I have a feeling it will run into
> > the same problem with asciidoctor (if it's not respecting {litdd} in
> > that context, it's probably also not respecting backticks).
> I will play with this to find a solution. Would it be an option to
> replace "--" with "-"? Why do we need two dashes if they cause trouble?

We use two dashes to signify "internal" programs that users should not
rely on. So "git-web-browse" would be something we'd expect to support
forever, but "git-web--browse" is an implementation detail of one of our
scripts (that just happens to require an extra program).

So I don't think we want to switch away from that convention just to
make the documentation work.

AFAICT, the {litdd} is working fine with asciidoc; it's only asciidoctor
that is the problem. So the first step may be talking with asciidoctor
folks to see if it's a bug, or if they have a recommended workaround.

-Peff

Re: [PATCH v3 1/2] Documentation: fix linkgit references

[Jeff King]

On Tue, May 03, 2016 at 08:41:37AM -0700, Junio C Hamano wrote:

> Even though you obviously were fooled by AsciiDoctor regarding
> literal double dash {litdd}, other parts of the changes did look
> good typofixes.  Thanks for starting this.

Yeah, I stopped reading after seeing the first hunk, but I agree the
rest look trivial.

-Peff

Re: [PATCH v3 2/2] travis-ci: build documentation

[Lars Schneider]

On 03 May 2016, at 17:43, Junio C Hamano <gitster@pobox.com> wrote:

> Lars Schneider <larsxschneider@gmail.com> writes:
> 
>> On 02 May 2016, at 22:45, Junio C Hamano <gitster@pobox.com> wrote:
>> 
>>> larsxschneider@gmail.com writes:
>>> 
>>>> +set -e
>>>> +
>>>> +LINKS=$(grep --recursive --only-matching --no-filename --perl-regexp \
>>>> +    '(?<=linkgit:).*?(?=\[\d+\])' Documentation/* \
>>>> +    | sort -u \
>>>> +)
>>>> +
>>>> +for LINK in $LINKS; do
>>>> +    echo "Checking linkgit:$LINK..."
>>>> +    test -s Documentation/$LINK.txt
>>>> +done
>>> 
>>> Please separate the above link check out of this step and do so
>>> separately after the move of test body to a separate script
>>> settles.
>> 
>> OK. I also wonder if the link check should rather go to the 
>> "check-docs" Makefile target?
> 
> That sounds like a good direction.
> 
> Which in turn means that people on all platforms are welcome to run
> it, which in turn means that the script must be even more portable,
> with avoiding GNUism and bash-isms etc.

OK. I am not that experienced with shell scripting and therefore it
is hard for me to distinguish between the different shell features.
Do you know/can you recommend the most basic shell to test/work 
with? A quick Google search told me that "dash" from Ubuntu seems
to be a good baseline as it aims to support pretty much only POSIX [1].

Thanks,
Lars

[1] http://www.cyberciti.biz/faq/debian-ubuntu-linux-binbash-vs-bindash-vs-binshshell/

Re: [PATCH v3 2/2] travis-ci: build documentation

[Eric Wong]

Lars Schneider <larsxschneider@gmail.com> wrote:
> OK. I am not that experienced with shell scripting and therefore it
> is hard for me to distinguish between the different shell features.
> Do you know/can you recommend the most basic shell to test/work 
> with? A quick Google search told me that "dash" from Ubuntu seems
> to be a good baseline as it aims to support pretty much only POSIX [1].

Yes, I recommend dash as your /bin/sh if you're using a
Debian-based system ("Debian Almquist shell") such as Ubuntu.

Also, the Debian "devscripts" package has a tool called
"checkbashisms" which I have also found very useful.

Maybe ksh(93) or posh could be good, too, but it's been a
while...

I also rely on checking the manpages-posix and manpages-posix-dev
in the "non-free" section of Debian ("multiverse" in Ubuntu?)

Re: [PATCH v3 2/2] travis-ci: build documentation

[Junio C Hamano]

Lars Schneider <larsxschneider@gmail.com> writes:

> A quick Google search told me that "dash" from Ubuntu seems
> to be a good baseline as it aims to support pretty much only POSIX [1].

Yeah that is a good and safe starting point.

[PATCH v4 0/2] travis-ci: build documentation

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

diff to v3:
* Revert the change from "{litdd}" to "--" in the documentation.
  "{litdd}" is rendered wrong in some HTML output [1], but "--"
  breaks the roff output ... I will investigate this and try to fix
  it in a future patch.
* I removed the doc link checker for now. I will try to reintroduce
  an improved version of the link checker as part of `make check-docs`
  in a future patch.

Thanks Peff and Junio for the review,
Lars

[1] https://git-scm.com/docs/git-config

Lars Schneider (2):
  Documentation: fix linkgit references
  travis-ci: build documentation

 .travis.yml                         | 15 +++++++++++++++
 Documentation/config.txt            |  4 ++--
 Documentation/git-check-ignore.txt  |  2 +-
 Documentation/git-filter-branch.txt |  4 ++--
 Documentation/git-for-each-ref.txt  |  2 +-
 ci/test-documentation.sh            | 14 ++++++++++++++
 6 files changed, 35 insertions(+), 6 deletions(-)
 create mode 100755 ci/test-documentation.sh

--
2.5.1

[PATCH v4 1/2] Documentation: fix linkgit references

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 Documentation/config.txt            | 4 ++--
 Documentation/git-check-ignore.txt  | 2 +-
 Documentation/git-filter-branch.txt | 4 ++--
 Documentation/git-for-each-ref.txt  | 2 +-
 4 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index c7bbe98..5683400 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -1494,7 +1494,7 @@ gui.diffContext::
 	made by the linkgit:git-gui[1]. The default is "5".
 
 gui.displayUntracked::
-	Determines if linkgit::git-gui[1] shows untracked files
+	Determines if linkgit:git-gui[1] shows untracked files
 	in the file list. The default is "true".
 
 gui.encoding::
@@ -1665,7 +1665,7 @@ http.cookieFile::
 	File containing previously stored cookie lines which should be used
 	in the Git http session, if they match the server. The file format
 	of the file to read cookies from should be plain HTTP headers or
-	the Netscape/Mozilla cookie file format (see linkgit:curl[1]).
+	the Netscape/Mozilla cookie file format (see `curl(1)`).
 	NOTE that the file specified with http.cookieFile is only used as
 	input unless http.saveCookies is set.
 
diff --git a/Documentation/git-check-ignore.txt b/Documentation/git-check-ignore.txt
index e94367a..9a85998 100644
--- a/Documentation/git-check-ignore.txt
+++ b/Documentation/git-check-ignore.txt
@@ -112,7 +112,7 @@ EXIT STATUS
 SEE ALSO
 --------
 linkgit:gitignore[5]
-linkgit:gitconfig[5]
+linkgit:git-config[5]
 linkgit:git-ls-files[1]
 
 GIT
diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 73fd9e8..6538cb1 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -205,8 +205,8 @@ to other tags will be rewritten to point to the underlying commit.
 Remap to ancestor
 ~~~~~~~~~~~~~~~~~
 
-By using linkgit:rev-list[1] arguments, e.g., path limiters, you can limit the
-set of revisions which get rewritten. However, positive refs on the command
+By using linkgit:git-rev-list[1] arguments, e.g., path limiters, you can limit
+the set of revisions which get rewritten. However, positive refs on the command
 line are distinguished: we don't let them be excluded by such limiters. For
 this purpose, they are instead rewritten to point at the nearest ancestor that
 was not excluded.
diff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt
index c52578b..d9d406d 100644
--- a/Documentation/git-for-each-ref.txt
+++ b/Documentation/git-for-each-ref.txt
@@ -179,7 +179,7 @@ returns an empty string instead.
 
 As a special case for the date-type fields, you may specify a format for
 the date by adding `:` followed by date format name (see the
-values the `--date` option to linkgit::git-rev-list[1] takes).
+values the `--date` option to linkgit:git-rev-list[1] takes).
 
 
 EXAMPLES
-- 
2.5.1

[PATCH v4 2/2] travis-ci: build documentation

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

Build documentation as separate Travis CI job to check for
documentation errors.

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 .travis.yml              | 15 +++++++++++++++
 ci/test-documentation.sh | 14 ++++++++++++++
 2 files changed, 29 insertions(+)
 create mode 100755 ci/test-documentation.sh

diff --git a/.travis.yml b/.travis.yml
index 78e433b..55299bd 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -32,6 +32,21 @@ env:
     # t9816 occasionally fails with "TAP out of sequence errors" on Travis CI OS X
     - GIT_SKIP_TESTS="t9810 t9816"
 
+matrix:
+  include:
+    - env: Documentation
+      os: linux
+      compiler: clang
+      addons:
+        apt:
+          packages:
+          - asciidoc
+          - xmlto
+      before_install:
+      before_script:
+      script: ci/test-documentation.sh
+      after_failure:
+
 before_install:
   - >
     case "${TRAVIS_OS_NAME:-linux}" in
diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh
new file mode 100755
index 0000000..579d540
--- /dev/null
+++ b/ci/test-documentation.sh
@@ -0,0 +1,14 @@
+#!/bin/sh
+#
+# Perform sanity checks on documentation and build it.
+#
+
+set -e
+
+make check-builtins
+make check-docs
+make doc
+
+test -s Documentation/git.html
+test -s Documentation/git.xml
+test -s Documentation/git.1
-- 
2.5.1

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Lars Schneider]

On 04 May 2016, at 10:38, larsxschneider@gmail.com wrote:

> From: Lars Schneider <larsxschneider@gmail.com>
> 
> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
> ---
> Documentation/config.txt            | 4 ++--
> Documentation/git-check-ignore.txt  | 2 +-
> Documentation/git-filter-branch.txt | 4 ++--
> Documentation/git-for-each-ref.txt  | 2 +-
> 4 files changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index c7bbe98..5683400 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -1494,7 +1494,7 @@ gui.diffContext::
> 	made by the linkgit:git-gui[1]. The default is "5".
> 
> gui.displayUntracked::
> -	Determines if linkgit::git-gui[1] shows untracked files
> +	Determines if linkgit:git-gui[1] shows untracked files
> 	in the file list. The default is "true".
> 
> gui.encoding::
> @@ -1665,7 +1665,7 @@ http.cookieFile::
> 	File containing previously stored cookie lines which should be used
> 	in the Git http session, if they match the server. The file format
> 	of the file to read cookies from should be plain HTTP headers or
> -	the Netscape/Mozilla cookie file format (see linkgit:curl[1]).
> +	the Netscape/Mozilla cookie file format (see `curl(1)`).
> 	NOTE that the file specified with http.cookieFile is only used as
> 	input unless http.saveCookies is set.
> 
> diff --git a/Documentation/git-check-ignore.txt b/Documentation/git-check-ignore.txt
> index e94367a..9a85998 100644
> --- a/Documentation/git-check-ignore.txt
> +++ b/Documentation/git-check-ignore.txt
> @@ -112,7 +112,7 @@ EXIT STATUS
> SEE ALSO
> --------
> linkgit:gitignore[5]
> -linkgit:gitconfig[5]
> +linkgit:git-config[5]
> linkgit:git-ls-files[1]
> 
> GIT
> diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
> index 73fd9e8..6538cb1 100644
> --- a/Documentation/git-filter-branch.txt
> +++ b/Documentation/git-filter-branch.txt
> @@ -205,8 +205,8 @@ to other tags will be rewritten to point to the underlying commit.
> Remap to ancestor
> ~~~~~~~~~~~~~~~~~
> 
> -By using linkgit:rev-list[1] arguments, e.g., path limiters, you can limit the
> -set of revisions which get rewritten. However, positive refs on the command
> +By using linkgit:git-rev-list[1] arguments, e.g., path limiters, you can limit
> +the set of revisions which get rewritten. However, positive refs on the command

All other linkgit fixes seem legimiate to me although I am not sure of this case

-linkgit:rev-list[1] 
+linkgit:git-rev-list[1]
  
"rev-list" works but I think "git-rev-list" would be the canonical form?
See: https://git-scm.com/docs/git-filter-branch


> line are distinguished: we don't let them be excluded by such limiters. For
> this purpose, they are instead rewritten to point at the nearest ancestor that
> was not excluded.
> diff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt
> index c52578b..d9d406d 100644
> --- a/Documentation/git-for-each-ref.txt
> +++ b/Documentation/git-for-each-ref.txt
> @@ -179,7 +179,7 @@ returns an empty string instead.
> 
> As a special case for the date-type fields, you may specify a format for
> the date by adding `:` followed by date format name (see the
> -values the `--date` option to linkgit::git-rev-list[1] takes).
> +values the `--date` option to linkgit:git-rev-list[1] takes).
> 
> 
> EXAMPLES
> -- 
> 2.5.1
> 

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Jeff King]

On Wed, May 04, 2016 at 10:43:04AM +0200, Lars Schneider wrote:

> > diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
> > index 73fd9e8..6538cb1 100644
> > --- a/Documentation/git-filter-branch.txt
> > +++ b/Documentation/git-filter-branch.txt
> > @@ -205,8 +205,8 @@ to other tags will be rewritten to point to the underlying commit.
> > Remap to ancestor
> > ~~~~~~~~~~~~~~~~~
> > 
> > -By using linkgit:rev-list[1] arguments, e.g., path limiters, you can limit the
> > -set of revisions which get rewritten. However, positive refs on the command
> > +By using linkgit:git-rev-list[1] arguments, e.g., path limiters, you can limit
> > +the set of revisions which get rewritten. However, positive refs on the command
> 
> All other linkgit fixes seem legimiate to me although I am not sure of this case
> 
> -linkgit:rev-list[1] 
> +linkgit:git-rev-list[1]
>   
> "rev-list" works but I think "git-rev-list" would be the canonical form?
> See: https://git-scm.com/docs/git-filter-branch

It should definitely be "git-rev-list". The "linkgit" macro will format
whatever text you feed it. For the manpages, that doesn't matter,
because they don't actually hyperlink. But for other formats (like
HTML), using just "rev-list" will generate a broken link.

-Peff

Re: [PATCH v3 2/2] travis-ci: build documentation

[Christian Couder]

On Wed, May 4, 2016 at 10:04 AM, Lars Schneider
<larsxschneider@gmail.com> wrote:
>
> On 03 May 2016, at 17:43, Junio C Hamano <gitster@pobox.com> wrote:
>
>> Lars Schneider <larsxschneider@gmail.com> writes:
>>
>>> On 02 May 2016, at 22:45, Junio C Hamano <gitster@pobox.com> wrote:
>>>
>>>> larsxschneider@gmail.com writes:
>>>>
>>>>> +set -e
>>>>> +
>>>>> +LINKS=$(grep --recursive --only-matching --no-filename --perl-regexp \
>>>>> +    '(?<=linkgit:).*?(?=\[\d+\])' Documentation/* \
>>>>> +    | sort -u \
>>>>> +)
>>>>> +
>>>>> +for LINK in $LINKS; do
>>>>> +    echo "Checking linkgit:$LINK..."
>>>>> +    test -s Documentation/$LINK.txt
>>>>> +done
>>>>
>>>> Please separate the above link check out of this step and do so
>>>> separately after the move of test body to a separate script
>>>> settles.
>>>
>>> OK. I also wonder if the link check should rather go to the
>>> "check-docs" Makefile target?
>>
>> That sounds like a good direction.
>>
>> Which in turn means that people on all platforms are welcome to run
>> it, which in turn means that the script must be even more portable,
>> with avoiding GNUism and bash-isms etc.
>
> OK. I am not that experienced with shell scripting and therefore it
> is hard for me to distinguish between the different shell features.
> Do you know/can you recommend the most basic shell to test/work
> with? A quick Google search told me that "dash" from Ubuntu seems
> to be a good baseline as it aims to support pretty much only POSIX [1].

You might also want to check:

http://www.shellcheck.net/

Re: [PATCH v3 2/2] travis-ci: build documentation

[Matthieu Moy]

Christian Couder <christian.couder@gmail.com> writes:

> You might also want to check:
>
> http://www.shellcheck.net/

Indeed, it's also an excellent tool to check for common mistakes in
shell scripts (there are many, and shellcheck is good at pointing them
out and explaining them).

http://www.shellcheck.net/

(Available online and as a command-line tool)

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Ramsay Jones]

On 04/05/16 09:43, Lars Schneider wrote:
> 
> On 04 May 2016, at 10:38, larsxschneider@gmail.com wrote:
> 
>> From: Lars Schneider <larsxschneider@gmail.com>
>>
>> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
>> ---
>> Documentation/config.txt            | 4 ++--
>> Documentation/git-check-ignore.txt  | 2 +-
>> Documentation/git-filter-branch.txt | 4 ++--
>> Documentation/git-for-each-ref.txt  | 2 +-
>> 4 files changed, 6 insertions(+), 6 deletions(-)
>>
>> diff --git a/Documentation/config.txt b/Documentation/config.txt
>> index c7bbe98..5683400 100644
>> --- a/Documentation/config.txt
>> +++ b/Documentation/config.txt
>> @@ -1494,7 +1494,7 @@ gui.diffContext::
>> 	made by the linkgit:git-gui[1]. The default is "5".
>>
>> gui.displayUntracked::
>> -	Determines if linkgit::git-gui[1] shows untracked files
>> +	Determines if linkgit:git-gui[1] shows untracked files
>> 	in the file list. The default is "true".
>>
>> gui.encoding::
>> @@ -1665,7 +1665,7 @@ http.cookieFile::
>> 	File containing previously stored cookie lines which should be used
>> 	in the Git http session, if they match the server. The file format
>> 	of the file to read cookies from should be plain HTTP headers or
>> -	the Netscape/Mozilla cookie file format (see linkgit:curl[1]).
>> +	the Netscape/Mozilla cookie file format (see `curl(1)`).
>> 	NOTE that the file specified with http.cookieFile is only used as
>> 	input unless http.saveCookies is set.
>>
>> diff --git a/Documentation/git-check-ignore.txt b/Documentation/git-check-ignore.txt
>> index e94367a..9a85998 100644
>> --- a/Documentation/git-check-ignore.txt
>> +++ b/Documentation/git-check-ignore.txt
>> @@ -112,7 +112,7 @@ EXIT STATUS
>> SEE ALSO
>> --------
>> linkgit:gitignore[5]
>> -linkgit:gitconfig[5]
>> +linkgit:git-config[5]

I think Junio already noted, git-config is in section 1 not 5.

ATB,
Ramsay Jones

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Junio C Hamano]

Ramsay Jones <ramsay@ramsayjones.plus.com> writes:

>>> diff --git a/Documentation/git-check-ignore.txt b/Documentation/git-check-ignore.txt
>>> index e94367a..9a85998 100644
>>> --- a/Documentation/git-check-ignore.txt
>>> +++ b/Documentation/git-check-ignore.txt
>>> @@ -112,7 +112,7 @@ EXIT STATUS
>>> SEE ALSO
>>> --------
>>> linkgit:gitignore[5]
>>> -linkgit:gitconfig[5]
>>> +linkgit:git-config[5]
>
> I think Junio already noted, git-config is in section 1 not 5.

Not just that, I am afraid.  This came from 368aa529 (add
git-check-ignore sub-command, 2013-01-06) and it added these:

+linkgit:gitignore[5]
+linkgit:gitconfig[5]
+linkgit:git-ls-files[5]

The last one was later corrected, but who knows what other mistakes
there are?

So I used the script attached at the bottom to audit the whole
thing, and the result is here.

-- >8 --
Documentation/config.txt:1497: nongit link: :git-gui[1]
Documentation/config.txt:1662: nongit link: curl[1]
Documentation/diff-options.txt:274: wrong section (should be 5): gitattributes[1]
Documentation/git-check-ignore.txt:115: no such source: gitconfig[5]
Documentation/git-filter-branch.txt:208: nongit link: rev-list[1]
Documentation/git-for-each-ref.txt:182: nongit link: :git-rev-list[1]
Documentation/git-notes.txt:405: wrong section (should be 1): git[7]
Documentation/technical/api-credentials.txt:246: wrong section (should be 1): git-credential[7]
Documentation/technical/api-credentials.txt:271: wrong section (should be 1): git-config[5]
-- 8< --

I do not think there is any false positive above, so perhaps the
checker script below can be used as the link checker we discussed?

-- >8 --
#!/bin/sh

git grep -l linkgit: Documentation/ |
while read path
do
	perl -e '
	sub report {
		my ($where, $what, $error) = @_;
		print "$where: $error: $what\n";
	}

	sub grab_section {
		my ($page) = @_;
		open my $fh, "<", "Documentation/$page.txt";
		my $firstline = <$fh>;
		chomp $firstline;
		close $fh;
		my ($section) = ($firstline =~ /.*\((\d)\)$/);
		return $section;
	}

	while (<>) {
		my $where = "$ARGV:$.";
		while (s/linkgit:((.*?)\[(\d)\])//) {
			my ($target, $page, $section) = ($1, $2, $3);

			# De-AsciiDoc
			$page =~ s/{litdd}/--/g;

			if ($page !~ /^git/) {
				report($where, $target, "nongit link");
				next;
			}
			if (! -f "Documentation/$page.txt") {
				report($where, $target, "no such source");
				next;
			}
			$real_section = grab_section($page); 
			if ($real_section != $section) {
				report($where, $target,
					"wrong section (should be $real_section)");
				next;
			}
		}
	}
        ' "$path"
done

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> So I used the script attached at the bottom to audit the whole
> thing, and the result is here.
> ...

While I was at it, I just did this to make the documentation set
lint clean.

-- >8 --

There are a handful of incorrect "linkgit:<page>[<section>]"
instances in our documentation set.

 * Some have an extra colon after "linkgit:"; fix them by removing
   the extra colon;

 * Some refer to a page outside the Git suite, namely curl(1); fix
   them by using the `curl(1)` that already appears on the same page
   for the same purpose of referring the readers to its manual page.

 * Some spell the name of the page incorrectly, e.g. "rev-list" when
   they mean "git-rev-list"; fix them.

 * Some list the manual section incorrectly; fix them to make sure
   they match what is at the top of the target of the link.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/config.txt                    | 4 ++--
 Documentation/diff-options.txt              | 2 +-
 Documentation/everyday.txto                 | 2 +-
 Documentation/git-check-ignore.txt          | 2 +-
 Documentation/git-filter-branch.txt         | 2 +-
 Documentation/git-for-each-ref.txt          | 2 +-
 Documentation/git-notes.txt                 | 2 +-
 Documentation/technical/api-credentials.txt | 4 ++--
 8 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 42d2b50..f37e16b 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -1494,7 +1494,7 @@ gui.diffContext::
 	made by the linkgit:git-gui[1]. The default is "5".
 
 gui.displayUntracked::
-	Determines if linkgit::git-gui[1] shows untracked files
+	Determines if linkgit:git-gui[1] shows untracked files
 	in the file list. The default is "true".
 
 gui.encoding::
@@ -1659,7 +1659,7 @@ http.cookieFile::
 	File containing previously stored cookie lines which should be used
 	in the Git http session, if they match the server. The file format
 	of the file to read cookies from should be plain HTTP headers or
-	the Netscape/Mozilla cookie file format (see linkgit:curl[1]).
+	the Netscape/Mozilla cookie file format (see `curl(1)`).
 	NOTE that the file specified with http.cookieFile is only used as
 	input unless http.saveCookies is set.
 
diff --git a/Documentation/diff-options.txt b/Documentation/diff-options.txt
index 4b0318e..3cb3015 100644
--- a/Documentation/diff-options.txt
+++ b/Documentation/diff-options.txt
@@ -271,7 +271,7 @@ For example, `--word-diff-regex=.` will treat each character as a word
 and, correspondingly, show differences character by character.
 +
 The regex can also be set via a diff driver or configuration option, see
-linkgit:gitattributes[1] or linkgit:git-config[1].  Giving it explicitly
+linkgit:gitattributes[5] or linkgit:git-config[1].  Giving it explicitly
 overrides any diff driver or configuration setting.  Diff drivers
 override configuration settings.
 
diff --git a/Documentation/everyday.txto b/Documentation/everyday.txto
index c5047d8..ae555bd 100644
--- a/Documentation/everyday.txto
+++ b/Documentation/everyday.txto
@@ -1,7 +1,7 @@
 Everyday Git With 20 Commands Or So
 ===================================
 
-This document has been moved to linkgit:giteveryday[1].
+This document has been moved to linkgit:giteveryday[7].
 
 Please let the owners of the referring site know so that they can update the
 link you clicked to get here.
diff --git a/Documentation/git-check-ignore.txt b/Documentation/git-check-ignore.txt
index e94367a..611754f 100644
--- a/Documentation/git-check-ignore.txt
+++ b/Documentation/git-check-ignore.txt
@@ -112,7 +112,7 @@ EXIT STATUS
 SEE ALSO
 --------
 linkgit:gitignore[5]
-linkgit:gitconfig[5]
+linkgit:git-config[1]
 linkgit:git-ls-files[1]
 
 GIT
diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 73fd9e8..003731f 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -205,7 +205,7 @@ to other tags will be rewritten to point to the underlying commit.
 Remap to ancestor
 ~~~~~~~~~~~~~~~~~
 
-By using linkgit:rev-list[1] arguments, e.g., path limiters, you can limit the
+By using linkgit:git-rev-list[1] arguments, e.g., path limiters, you can limit the
 set of revisions which get rewritten. However, positive refs on the command
 line are distinguished: we don't let them be excluded by such limiters. For
 this purpose, they are instead rewritten to point at the nearest ancestor that
diff --git a/Documentation/git-for-each-ref.txt b/Documentation/git-for-each-ref.txt
index c52578b..d9d406d 100644
--- a/Documentation/git-for-each-ref.txt
+++ b/Documentation/git-for-each-ref.txt
@@ -179,7 +179,7 @@ returns an empty string instead.
 
 As a special case for the date-type fields, you may specify a format for
 the date by adding `:` followed by date format name (see the
-values the `--date` option to linkgit::git-rev-list[1] takes).
+values the `--date` option to linkgit:git-rev-list[1] takes).
 
 
 EXAMPLES
diff --git a/Documentation/git-notes.txt b/Documentation/git-notes.txt
index 8de3499..9c4fd68 100644
--- a/Documentation/git-notes.txt
+++ b/Documentation/git-notes.txt
@@ -402,4 +402,4 @@ on the `notes.rewrite.<command>` and `notes.rewriteRef` settings.
 
 GIT
 ---
-Part of the linkgit:git[7] suite
+Part of the linkgit:git[1] suite
diff --git a/Documentation/technical/api-credentials.txt b/Documentation/technical/api-credentials.txt
index e44426d..75368f2 100644
--- a/Documentation/technical/api-credentials.txt
+++ b/Documentation/technical/api-credentials.txt
@@ -243,7 +243,7 @@ appended to its command line, which is one of:
 The details of the credential will be provided on the helper's stdin
 stream. The exact format is the same as the input/output format of the
 `git credential` plumbing command (see the section `INPUT/OUTPUT
-FORMAT` in linkgit:git-credential[7] for a detailed specification).
+FORMAT` in linkgit:git-credential[1] for a detailed specification).
 
 For a `get` operation, the helper should produce a list of attributes
 on stdout in the same format. A helper is free to produce a subset, or
@@ -268,4 +268,4 @@ See also
 
 linkgit:gitcredentials[7]
 
-linkgit:git-config[5] (See configuration variables `credential.*`)
+linkgit:git-config[1] (See configuration variables `credential.*`)

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> I do not think there is any false positive above, so perhaps the
> checker script below can be used as the link checker we discussed?

-- >8 --
Subject: [PATCH] ci: validate "gitlink:" in documentation

It is easy to add incorrect "linkgit:<page>[<section>]" references
to our documentation suite.  Catch these common classes of errors:

 * Referring to Documentation/<page>.txt that does not exist.

 * Referring to a <page> outside the Git suite.  In general, <page>
   must begin with "git".

 * Listing the manual <section> incorrectly.  The first line of the
   Documentation/<page>.txt must end with "(<section>)".

with a new script ci/lint-gitlink.sh.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 ci/lint-gitlink.sh | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
 create mode 100755 ci/lint-gitlink.sh

diff --git a/ci/lint-gitlink.sh b/ci/lint-gitlink.sh
new file mode 100755
index 0000000..2379626
--- /dev/null
+++ b/ci/lint-gitlink.sh
@@ -0,0 +1,47 @@
+#!/bin/sh
+
+git grep -l linkgit: Documentation/ |
+while read path
+do
+	perl -e '
+	sub report {
+		my ($where, $what, $error) = @_;
+		print "$where: $error: $what\n";
+	}
+
+	sub grab_section {
+		my ($page) = @_;
+		open my $fh, "<", "Documentation/$page.txt";
+		my $firstline = <$fh>;
+		chomp $firstline;
+		close $fh;
+		my ($section) = ($firstline =~ /.*\((\d)\)$/);
+		return $section;
+	}
+
+	while (<>) {
+		my $where = "$ARGV:$.";
+		while (s/linkgit:((.*?)\[(\d)\])//) {
+			my ($target, $page, $section) = ($1, $2, $3);
+
+			# De-AsciiDoc
+			$page =~ s/{litdd}/--/g;
+
+			if ($page !~ /^git/) {
+				report($where, $target, "nongit link");
+				next;
+			}
+			if (! -f "Documentation/$page.txt") {
+				report($where, $target, "no such source");
+				next;
+			}
+			$real_section = grab_section($page);
+			if ($real_section != $section) {
+				report($where, $target,
+					"wrong section (should be $real_section)");
+				next;
+			}
+		}
+	}
+	' "$path"
+done
-- 
2.8.2-498-g6350fe8

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Jeff King]

On Wed, May 04, 2016 at 11:52:52AM -0700, Junio C Hamano wrote:

> > I do not think there is any false positive above, so perhaps the
> > checker script below can be used as the link checker we discussed?
> 
> -- >8 --
> Subject: [PATCH] ci: validate "gitlink:" in documentation
> 
> It is easy to add incorrect "linkgit:<page>[<section>]" references
> to our documentation suite.  Catch these common classes of errors:
> 
>  * Referring to Documentation/<page>.txt that does not exist.
> 
>  * Referring to a <page> outside the Git suite.  In general, <page>
>    must begin with "git".
> 
>  * Listing the manual <section> incorrectly.  The first line of the
>    Documentation/<page>.txt must end with "(<section>)".
> 
> with a new script ci/lint-gitlink.sh.

It seems like this is something we _should_ be able to do while asciidoc
is actually expanding the macro, rather than as a separate step. But:

  1. I don't think stock asciidoc has this much flexibility in its
     macros.

  2. There are some ordering questions (e.g., while building "foo.1",
     "bar.1" may not be built yet, so we still have to massage requests
     for "bar.1" into "bar.txt".

Likewise, I think we could build the whole HTML source and then actually
just look for broken links in it. But that script would probably end up
looking similar to this one, with s/linkgit/href/. But it does more
directly measure what we want, which is that the rendered doc is usable;
it would catch something like using "--" instead of "{litdd}".

> +#!/bin/sh
> +
> +git grep -l linkgit: Documentation/ |
> +while read path
> +do
> +	perl -e '

Is it worth just making this a perl script, rather than a shell script
with a giant inline perl script? Perl is actually really good at doing
that "grep" as it reads the file. :)

Besides being slightly more efficient (reading the files only once
rather than filtering once via grep and then re-reading via perl). But
more importantly, it avoids a layer of quoting, which makes it less
likely to make a mistake by using single-quote in the perl script (I do
not see any errors in what you wrote, though).

-Peff

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Likewise, I think we could build the whole HTML source and then actually
> just look for broken links in it. But that script would probably end up
> looking similar to this one, with s/linkgit/href/. But it does more
> directly measure what we want, which is that the rendered doc is usable;

I debated about this myself, but chose to inspect the source
material, as that approach is easier to give actionable lint output
to the user that points out the file:lineno to be corrected.

> it would catch something like using "--" instead of "{litdd}".

That is true indeed.  With the "source" approach, that would indeed
be harder.

>> +#!/bin/sh
>> +
>> +git grep -l linkgit: Documentation/ |
>> +while read path
>> +do
>> +	perl -e '
>
> Is it worth just making this a perl script, rather than a shell script
> with a giant inline perl script? Perl is actually really good at doing
> that "grep" as it reads the file. :)

OK.

-- >8 --
From: Junio C Hamano <gitster@pobox.com>
Date: Wed, 4 May 2016 11:48:06 -0700
Subject: [PATCH v2] ci: validate "gitlink:" in documentation

It is easy to add incorrect "linkgit:<page>[<section>]" references
to our documentation suite.  Catch these common classes of errors:

 * Referring to Documentation/<page>.txt that does not exist.

 * Referring to a <page> outside the Git suite.  In general, <page>
   must begin with "git".

 * Listing the manual <section> incorrectly.  The first line of the
   Documentation/<page>.txt must end with "(<section>)".

with a new script "ci/lint-gitlink".

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 ci/lint-gitlink | 60 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 60 insertions(+)
 create mode 100755 ci/lint-gitlink

diff --git a/ci/lint-gitlink b/ci/lint-gitlink
new file mode 100755
index 0000000..bb73e89
--- /dev/null
+++ b/ci/lint-gitlink
@@ -0,0 +1,60 @@
+#!/usr/bin/perl
+
+use File::Find;
+
+my $found_errors = 0;
+
+sub report {
+	my ($where, $what, $error) = @_;
+	print "$where: $error: $what\n";
+	$found_errors = 1;
+}
+
+sub grab_section {
+	my ($page) = @_;
+	open my $fh, "<", "Documentation/$page.txt";
+	my $firstline = <$fh>;
+	chomp $firstline;
+	close $fh;
+	my ($section) = ($firstline =~ /.*\((\d)\)$/);
+	return $section;
+}
+
+sub lint {
+	my ($file) = @_;
+	open my $fh, "<", $file
+		or return;
+	while (<$fh>) {
+		my $where = "$file:$.";
+		while (s/linkgit:((.*?)\[(\d)\])//) {
+			my ($target, $page, $section) = ($1, $2, $3);
+
+			# De-AsciiDoc
+			$page =~ s/{litdd}/--/g;
+
+			if ($page !~ /^git/) {
+				report($where, $target, "nongit link");
+				next;
+			}
+			if (! -f "Documentation/$page.txt") {
+				report($where, $target, "no such source");
+				next;
+			}
+			$real_section = grab_section($page);
+			if ($real_section != $section) {
+				report($where, $target,
+					"wrong section (should be $real_section)");
+				next;
+			}
+		}
+	}
+	close $fh;
+}
+
+sub lint_it {
+	lint($File::Find::name) if -f;
+}
+
+find({ wanted => \&lint_it, no_chdir => 1 }, "Documentation");
+
+exit $found_errors;
-- 
2.8.2-498-g6350fe8

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Jeff King]

On Wed, May 04, 2016 at 12:57:31PM -0700, Junio C Hamano wrote:

> > Is it worth just making this a perl script, rather than a shell script
> > with a giant inline perl script? Perl is actually really good at doing
> > that "grep" as it reads the file. :)
> 
> OK.

Hmm. This new version uses File::Find:

> +sub lint_it {
> +	lint($File::Find::name) if -f;
> +}
> +
> +find({ wanted => \&lint_it, no_chdir => 1 }, "Documentation");

That will inspect non-source files, too.

Would:

  open(my $files, '-|', qw(git ls-files));
  while (<$files>) {
    chomp;
    ...
  }

make sense? Or a simpler but non-streaming spelling:

  my @files = map { chomp; $_ } `git ls-files`;

Or just taking the list of files on the command line as your original
did, and feeding `ls-files` from the caller. That also lets you do
"link-gitlink git-foo.txt", etc.

-Peff

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Wed, May 04, 2016 at 12:57:31PM -0700, Junio C Hamano wrote:
>
>> > Is it worth just making this a perl script, rather than a shell script
>> > with a giant inline perl script? Perl is actually really good at doing
>> > that "grep" as it reads the file. :)
>> 
>> OK.
>
> Hmm. This new version uses File::Find:
>
>> +sub lint_it {
>> +	lint($File::Find::name) if -f;
>> +}
>> +
>> +find({ wanted => \&lint_it, no_chdir => 1 }, "Documentation");
>
> That will inspect non-source files, too.
>
> Would:
>
>   open(my $files, '-|', qw(git ls-files));
>   while (<$files>) {
>     chomp;
>     ...
>   }
>
> make sense? Or a simpler but non-streaming spelling:
>
>   my @files = map { chomp; $_ } `git ls-files`;

I forgot to say that I wanted not to rely on "git" (i.e. OK to use
this on tarball extract).

> Or just taking the list of files on the command line as your original
> did, and feeding `ls-files` from the caller. That also lets you do
> "link-gitlink git-foo.txt", etc.

Yes, I think that is the most sensible.

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Jeff King]

On Wed, May 04, 2016 at 02:15:39PM -0700, Junio C Hamano wrote:

> > make sense? Or a simpler but non-streaming spelling:
> >
> >   my @files = map { chomp; $_ } `git ls-files`;
> 
> I forgot to say that I wanted not to rely on "git" (i.e. OK to use
> this on tarball extract).

Oh, that's a good idea.

> > Or just taking the list of files on the command line as your original
> > did, and feeding `ls-files` from the caller. That also lets you do
> > "link-gitlink git-foo.txt", etc.
> 
> Yes, I think that is the most sensible.

Yeah, and then the Makefile can drive it from $(MAN_TXT), etc, without
requiring git (which I think is what you were getting at, but just
spelling it out for myself and the list).

-Peff

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Junio C Hamano]

Third time's a charm, perhaps?
 
-- >8 --
Subject: [PATCH] ci: validate "gitlink:" in documentation

It is easy to add incorrect "linkgit:<page>[<section>]" references
to our documentation suite.  Catch these common classes of errors:

 * Referring to Documentation/<page>.txt that does not exist.

 * Referring to a <page> outside the Git suite.  In general, <page>
   must begin with "git".

 * Listing the manual <section> incorrectly.  The first line of the
   Documentation/<page>.txt must end with "(<section>)".

with a new script "ci/lint-gitlink", and drive it from "make check-docs".

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/Makefile |  5 +++++
 Makefile               |  1 +
 ci/lint-gitlink        | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 62 insertions(+)
 create mode 100755 ci/lint-gitlink

diff --git a/Documentation/Makefile b/Documentation/Makefile
index 3e39e28..e9cd43d 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -204,6 +204,7 @@ ifndef V
 	QUIET_DBLATEX	= @echo '   ' DBLATEX $@;
 	QUIET_XSLTPROC	= @echo '   ' XSLTPROC $@;
 	QUIET_GEN	= @echo '   ' GEN $@;
+	QUIET_LINT	= @echo '   ' LINT $@;
 	QUIET_STDERR	= 2> /dev/null
 	QUIET_SUBDIR0	= +@subdir=
 	QUIET_SUBDIR1	= ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
@@ -427,4 +428,8 @@ quick-install-html: require-htmlrepo
 print-man1:
 	@for i in $(MAN1_TXT); do echo $$i; done
 
+lint-docs::
+	$(QUIET_LINT)$(foreach txt,$(patsubst %.html,%.txt,$(DOC_HTML)), \
+	../ci/lint-gitlink $(txt))
+
 .PHONY: FORCE
diff --git a/Makefile b/Makefile
index 2742a69..61bd0ab 100644
--- a/Makefile
+++ b/Makefile
@@ -2496,6 +2496,7 @@ ALL_COMMANDS += git-gui git-citool
 
 .PHONY: check-docs
 check-docs::
+	$(MAKE) -C Documentation lint-docs
 	@(for v in $(ALL_COMMANDS); \
 	do \
 		case "$$v" in \
diff --git a/ci/lint-gitlink b/ci/lint-gitlink
new file mode 100755
index 0000000..6b6bf91
--- /dev/null
+++ b/ci/lint-gitlink
@@ -0,0 +1,56 @@
+#!/usr/bin/perl
+
+my $found_errors = 0;
+
+sub report {
+	my ($where, $what, $error) = @_;
+	print "$where: $error: $what\n";
+	$found_errors = 1;
+}
+
+sub grab_section {
+	my ($page) = @_;
+	open my $fh, "<", "$page.txt";
+	my $firstline = <$fh>;
+	chomp $firstline;
+	close $fh;
+	my ($section) = ($firstline =~ /.*\((\d)\)$/);
+	return $section;
+}
+
+sub lint {
+	my ($file) = @_;
+	open my $fh, "<", $file
+		or return;
+	while (<$fh>) {
+		my $where = "$file:$.";
+		while (s/linkgit:((.*?)\[(\d)\])//) {
+			my ($target, $page, $section) = ($1, $2, $3);
+
+			# De-AsciiDoc
+			$page =~ s/{litdd}/--/g;
+
+			if ($page !~ /^git/) {
+				report($where, $target, "nongit link");
+				next;
+			}
+			if (! -f "$page.txt") {
+				report($where, $target, "no such source");
+				next;
+			}
+			$real_section = grab_section($page);
+			if ($real_section != $section) {
+				report($where, $target,
+					"wrong section (should be $real_section)");
+				next;
+			}
+		}
+	}
+	close $fh;
+}
+
+for (@ARGV) {
+	lint($_);
+}
+
+exit $found_errors;
-- 
2.8.2-498-g6350fe8

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Jeff King]

On Wed, May 04, 2016 at 02:34:23PM -0700, Junio C Hamano wrote:

> Third time's a charm, perhaps?
>  
> -- >8 --
> Subject: [PATCH] ci: validate "gitlink:" in documentation
> 
> It is easy to add incorrect "linkgit:<page>[<section>]" references
> to our documentation suite.  Catch these common classes of errors:
> 
>  * Referring to Documentation/<page>.txt that does not exist.
> 
>  * Referring to a <page> outside the Git suite.  In general, <page>
>    must begin with "git".
> 
>  * Listing the manual <section> incorrectly.  The first line of the
>    Documentation/<page>.txt must end with "(<section>)".
> 
> with a new script "ci/lint-gitlink", and drive it from "make check-docs".
> 
> Signed-off-by: Junio C Hamano <gitster@pobox.com>
> ---

This looks good to me. Two minor nits that may not be worth addressing:

> +lint-docs::
> +	$(QUIET_LINT)$(foreach txt,$(patsubst %.html,%.txt,$(DOC_HTML)), \
> +	../ci/lint-gitlink $(txt))
> +

This dependency feels funny. Wouldn't CI want to invoke this as:

  make -C Documentation lint-docs

IOW, Documentation owns the script (just like t/ owns its own lint
scripts like check-non-portable-shell.pl), and CI always just triggers
the make-driven checks, just as a normal developer would?

> +sub lint {
> +	my ($file) = @_;
> +	open my $fh, "<", $file
> +		or return;
> +	while (<$fh>) {
> +		my $where = "$file:$.";
> [... actual linting of line ...]
> +}
> +
> +for (@ARGV) {
> +	lint($_);
> +}

The usual perl way here would be:

  while(<>) {
	my $where = "$ARGV:$.";
	... actual linting of line ...
  }

where "<>" automagically reads from files on the command-line or stdin
if none were specified (and sets $ARGV as appropriate). But maybe you
prefer not to handle the stdin case (it is a benefit sometimes, but an
annoyance if you accidentally end up with an empty file list).

-Peff

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> This dependency feels funny. Wouldn't CI want to invoke this as:
>
>   make -C Documentation lint-docs

I expected CI to do this instead

	make check-docs

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Ævar Arnfjörð Bjarmason]

On Wed, May 4, 2016 at 10:06 PM, Jeff King <peff@peff.net> wrote:
> Would:
>
>   open(my $files, '-|', qw(git ls-files));
>   while (<$files>) {
>     chomp;
>     ...
>   }
>
> make sense? Or a simpler but non-streaming spelling:
>
>   my @files = map { chomp; $_ } `git ls-files`;

As a minor sidenote you can equivalently write that as:

    chomp(my @files = `git ls-files`);

I.e. chomp itself when given a list will chomp each item of the list.
So no need for a map.

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Jeff King]

On Wed, May 04, 2016 at 02:52:27PM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > This dependency feels funny. Wouldn't CI want to invoke this as:
> >
> >   make -C Documentation lint-docs
> 
> I expected CI to do this instead
> 
> 	make check-docs

Ah, sure, that makes even more sense. But I think the point remains,
which is that your perl script is an implementation detail of the
Makefile process. I thought the "ci" directory was supposed to be for
ci-specific scripts that would be driven directly by Travis, etc.

-Peff

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Jeff King]

On Wed, May 04, 2016 at 11:54:52PM +0200, Ævar Arnfjörð Bjarmason wrote:

> >   my @files = map { chomp; $_ } `git ls-files`;
> 
> As a minor sidenote you can equivalently write that as:
> 
>     chomp(my @files = `git ls-files`);
> 
> I.e. chomp itself when given a list will chomp each item of the list.
> So no need for a map.

Right, thanks, that is more readable. I use the map form reflexively
because I am often doing stuff like:

  my @list = do {
    open(my $fh, '<', $fn);
    map { chomp; $_ }
  };

(or replacing "do" with an actual function which is returning a
grotesque pipeline of grep/map, etc).

-Peff

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> But I think the point remains,
> which is that your perl script is an implementation detail of the
> Makefile process. I thought the "ci" directory was supposed to be for
> ci-specific scripts that would be driven directly by Travis, etc.

True.  Documentation/ has tools like built-docdep.perl,
howto-index.sh, etc., so it can live next to it.

Thanks.

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Jeff King <peff@peff.net> writes:
>
>> But I think the point remains,
>> which is that your perl script is an implementation detail of the
>> Makefile process. I thought the "ci" directory was supposed to be for
>> ci-specific scripts that would be driven directly by Travis, etc.
>
> True.  Documentation/ has tools like built-docdep.perl,
> howto-index.sh, etc., so it can live next to it.
>
> Thanks.

Gaah, the Makefile part of the final patch is wrong; we do not check
the included sources at all if we only passed the top-level targets'
sources.

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Gaah, the Makefile part of the final patch is wrong; we do not check
> the included sources at all if we only passed the top-level targets'
> sources.

I ended up queuing an enhanced version of File::Find based one on
'pu', but I won't be posting it here today.

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Jeff King]

On Wed, May 04, 2016 at 04:28:31PM -0700, Junio C Hamano wrote:

> Junio C Hamano <gitster@pobox.com> writes:
> 
> > Gaah, the Makefile part of the final patch is wrong; we do not check
> > the included sources at all if we only passed the top-level targets'
> > sources.
> 
> I ended up queuing an enhanced version of File::Find based one on
> 'pu', but I won't be posting it here today.

Hmm. It seems like we should still be able to drive it from the Makefile
using the dependencies generated by build-docdep.

-Peff

Re: [PATCH v4 1/2] Documentation: fix linkgit references

[Jeff King]

On Wed, May 04, 2016 at 07:41:53PM -0400, Jeff King wrote:

> On Wed, May 04, 2016 at 04:28:31PM -0700, Junio C Hamano wrote:
> 
> > Junio C Hamano <gitster@pobox.com> writes:
> > 
> > > Gaah, the Makefile part of the final patch is wrong; we do not check
> > > the included sources at all if we only passed the top-level targets'
> > > sources.
> > 
> > I ended up queuing an enhanced version of File::Find based one on
> > 'pu', but I won't be posting it here today.
> 
> Hmm. It seems like we should still be able to drive it from the Makefile
> using the dependencies generated by build-docdep.

Having just read that script, it is blindly looking at "*.txt", so...as
long as that is what your script is doing (and it looks like the version
in pu is), I don't think there's any reason to get more complicated. The
only difference is that build-docdep does not even bother looking in
subdirectories (maybe it should, since we now build stuff in technical,
I think).

-Peff

Re: [PATCH v4 2/2] travis-ci: build documentation

[Junio C Hamano]

larsxschneider@gmail.com writes:

> From: Lars Schneider <larsxschneider@gmail.com>
>
> Build documentation as separate Travis CI job to check for
> documentation errors.
>
> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
> ---
>  .travis.yml              | 15 +++++++++++++++
>  ci/test-documentation.sh | 14 ++++++++++++++
>  2 files changed, 29 insertions(+)
>  create mode 100755 ci/test-documentation.sh
>
> diff --git a/.travis.yml b/.travis.yml
> index 78e433b..55299bd 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -32,6 +32,21 @@ env:
>      # t9816 occasionally fails with "TAP out of sequence errors" on Travis CI OS X
>      - GIT_SKIP_TESTS="t9810 t9816"

Completely offtopic, but this looks like this is made to apply to
all archs, not limited to OSX?  It of course would be ideal to see
why they fail only on OSX and fix them, but shouldn't the blacklist
at least limited to the platform with the problem?

Re: [PATCH v4 2/2] travis-ci: build documentation

[Lars Schneider]

> On 10 May 2016, at 19:12, Junio C Hamano <gitster@pobox.com> wrote:
> 
> larsxschneider@gmail.com writes:
> 
>> From: Lars Schneider <larsxschneider@gmail.com>
>> 
>> Build documentation as separate Travis CI job to check for
>> documentation errors.
>> 
>> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
>> ---
>> .travis.yml              | 15 +++++++++++++++
>> ci/test-documentation.sh | 14 ++++++++++++++
>> 2 files changed, 29 insertions(+)
>> create mode 100755 ci/test-documentation.sh
>> 
>> diff --git a/.travis.yml b/.travis.yml
>> index 78e433b..55299bd 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -32,6 +32,21 @@ env:
>>     # t9816 occasionally fails with "TAP out of sequence errors" on Travis CI OS X
>>     - GIT_SKIP_TESTS="t9810 t9816"
> 
> Completely offtopic, but this looks like this is made to apply to
> all archs, not limited to OSX?  It of course would be ideal to see
> why they fail only on OSX and fix them, but shouldn't the blacklist
> at least limited to the platform with the problem?

As far as I remember the test was flaky on Linux and OSX. The problem
was not Git. The problem was the Perforce Server "p4d" p4 daemon which
would refuse to die sometimes.

I run a bigger number of Travis tests and it looks like the problem is
gone. Out of 60 runs only just one failed and t9810 / t9816 passed:
https://travis-ci.org/larsxschneider/git/jobs/129383851
(BTW: does anyone know how I can make prove show me the running tests?
I would like to know which test died there..)

It looks like as Perforce solved the t9810/t9816 problem with the update 
from 15.2 to 16.1 that we made in 31f3c86 "travis-ci: update Git-LFS 
and P4 to the latest version". I will post a patch to enable these tests,
again.

Thanks,
Lars