[PATCH 0 of 2 v2] improve checking for documentation tools and formatting

[PATCH 0 of 2 v2] improve checking for documentation tools and formatting

[Matt Wilson]

This version makes running ./configure a prerequisite for building
documentation. If a tool is not detected by ./configure and is not
specified by the user, the make variable will be empty. So I've
switched to using make conditionals instead of which(1).

Matt

[PATCH 1 of 2 v2] tools: check for documentation generation tools at configure time

[Matt Wilson]

It is sometimes hard to discover all the optional tools that should be
on a system to build all available Xen documentation. By checking for
documentation generation tools at ./configure time and displaying a
warning, Xen packagers will more easily learn about new optional build
dependencies, like markdown, when they are introduced.

Changes since v1:
 * require that ./configure be run before building docs
 * remove Docs.mk and make Tools.mk the canonical locaiton where
   docs tools are defined (via ./configure)
 * fold in checking for markdown_py

Signed-off-by: Matt Wilson <msw@amazon.com>

diff -r d7e4efa17fb0 -r 512b4e0c49f3 README
--- a/README	Tue Aug 28 15:35:08 2012 -0700
+++ b/README	Thu Aug 30 10:51:00 2012 -0700
@@ -28,8 +28,9 @@
 your system. For full documentation, see the Xen User Manual. If this
 is a pre-built release then you can find the manual at:
  dist/install/usr/share/doc/xen/pdf/user.pdf
-If you have a source release, then 'make -C docs' will build the
-manual at docs/pdf/user.pdf.
+If you have a source release and the required documentation generation
+tools, then './configure; make -C docs' will build the manual at
+docs/pdf/user.pdf.
 
 Quick-Start Guide
 =================
@@ -59,7 +60,6 @@
     * GNU gettext
     * 16-bit x86 assembler, loader and compiler (dev86 rpm or bin86 & bcc debs)
     * ACPI ASL compiler (iasl)
-    * markdown
 
 In addition to the above there are a number of optional build
 prerequisites. Omitting these will cause the related features to be
@@ -67,6 +67,7 @@
     * Development install of Ocaml (e.g. ocaml-nox and
       ocaml-findlib). Required to build ocaml components which
       includes the alternative ocaml xenstored.
+    * markdown
 
 Second, you need to acquire a suitable kernel for use in domain 0. If
 possible you should use a kernel provided by your OS distributor. If
diff -r d7e4efa17fb0 -r 512b4e0c49f3 config/Tools.mk.in
--- a/config/Tools.mk.in	Tue Aug 28 15:35:08 2012 -0700
+++ b/config/Tools.mk.in	Thu Aug 30 10:51:00 2012 -0700
@@ -22,6 +22,18 @@
 LD86                := @LD86@
 BCC                 := @BCC@
 IASL                := @IASL@
+PS2PDF              := @PS2PDF@
+DVIPS               := @DVIPS@
+LATEX               := @LATEX@
+FIG2DEV             := @FIG2DEV@
+LATEX2HTML          := @LATEX2HTML@
+DOXYGEN             := @DOXYGEN@
+POD2MAN             := @POD2MAN@
+POD2HTML            := @POD2HTML@
+POD2TEXT            := @POD2TEXT@
+DOT                 := @DOT@
+NEATO               := @NEATO@
+MARKDOWN            := @MARKDOWN@
 
 # Extra folder for libs/includes
 PREPEND_INCLUDES    := @PREPEND_INCLUDES@
diff -r d7e4efa17fb0 -r 512b4e0c49f3 docs/Docs.mk
--- a/docs/Docs.mk	Tue Aug 28 15:35:08 2012 -0700
+++ /dev/null	Thu Jan 01 00:00:00 1970 +0000
@@ -1,12 +0,0 @@
-PS2PDF		:= ps2pdf
-DVIPS		:= dvips
-LATEX		:= latex
-FIG2DEV		:= fig2dev
-LATEX2HTML	:= latex2html
-DOXYGEN		:= doxygen
-POD2MAN		:= pod2man
-POD2HTML	:= pod2html
-POD2TEXT	:= pod2text
-DOT		:= dot
-NEATO		:= neato
-MARKDOWN	:= markdown
diff -r d7e4efa17fb0 -r 512b4e0c49f3 docs/Makefile
--- a/docs/Makefile	Tue Aug 28 15:35:08 2012 -0700
+++ b/docs/Makefile	Thu Aug 30 10:51:00 2012 -0700
@@ -2,7 +2,7 @@
 
 XEN_ROOT=$(CURDIR)/..
 include $(XEN_ROOT)/Config.mk
-include $(XEN_ROOT)/docs/Docs.mk
+-include $(XEN_ROOT)/config/Tools.mk
 
 VERSION		= xen-unstable
 
@@ -26,10 +26,12 @@
 
 .PHONY: build
 build: html txt man-pages
-	@if which $(DOT) 1>/dev/null 2>/dev/null ; then              \
-	$(MAKE) -C xen-api build ; else                              \
-        echo "Graphviz (dot) not installed; skipping xen-api." ; fi
+ifdef DOT
+	$(MAKE) -C xen-api build
 	rm -f *.aux *.dvi *.bbl *.blg *.glo *.idx *.ilg *.log *.ind *.toc
+else
+	@echo "Graphviz (dot) not installed; skipping xen-api."
+endif
 
 .PHONY: dev-docs
 dev-docs: python-dev-docs
@@ -42,18 +44,21 @@
 
 .PHONY: python-dev-docs
 python-dev-docs:
-	@mkdir -v -p api/tools/python
-	@set -e ; if which $(DOXYGEN) 1>/dev/null 2>/dev/null; then \
-        echo "Running doxygen to generate Python tools APIs ... "; \
-	$(DOXYGEN) Doxyfile;                                       \
-	$(MAKE) -C api/tools/python/latex ; else                   \
-        echo "Doxygen not installed; skipping python-dev-docs."; fi
+ifdef DOXYGEN
+	@echo "Running doxygen to generate Python tools APIs ... "
+	mkdir -v -p api/tools/python
+	$(DOXYGEN) Doxyfile && $(MAKE) -C api/tools/python/latex
+else
+	@echo "Doxygen not installed; skipping python-dev-docs."
+endif
 
 .PHONY: man-pages
 man-pages:
-	@if which $(POD2MAN) 1>/dev/null 2>/dev/null; then \
-	$(MAKE) $(DOC_MAN1) $(DOC_MAN5); else              \
-	echo "pod2man not installed; skipping man-pages."; fi
+ifdef POD2MAN
+	$(MAKE) $(DOC_MAN1) $(DOC_MAN5)
+else
+	@echo "pod2man not installed; skipping man-pages."
+endif
 
 man1/%.1: man/%.pod.1 Makefile
 	$(INSTALL_DIR) $(@D)
@@ -94,30 +99,40 @@
 	perl -w -- ./gen-html-index -i INDEX html $(DOC_HTML)
 
 html/%.html: %.markdown
-	@$(INSTALL_DIR) $(@D)
-	@set -e ; if which $(MARKDOWN) 1>/dev/null 2>/dev/null; then \
-	echo "Running markdown to generate $*.html ... "; \
+	$(INSTALL_DIR) $(@D)
+ifdef MARKDOWN
+	@echo "Running markdown to generate $*.html ... "
 	$(MARKDOWN) $< > $@.tmp ; \
-	$(call move-if-changed,$@.tmp,$@) ; else \
-	echo "markdown not installed; skipping $*.html."; fi
+	$(call move-if-changed,$@.tmp,$@)
+else
+	@echo "markdown not installed; skipping $*.html."
+endif
 
 html/%.txt: %.txt
-	@$(INSTALL_DIR) $(@D)
+	$(INSTALL_DIR) $(@D)
 	cp $< $@
 
 html/man/%.1.html: man/%.pod.1 Makefile
 	$(INSTALL_DIR) $(@D)
+ifdef POD2HTML
 	$(POD2HTML) --infile=$< --outfile=$@.tmp
 	$(call move-if-changed,$@.tmp,$@)
+else
+	@echo "pod2html not installed; skipping $<."
+endif
 
 html/man/%.5.html: man/%.pod.5 Makefile
 	$(INSTALL_DIR) $(@D)
+ifdef POD2HTML
 	$(POD2HTML) --infile=$< --outfile=$@.tmp
 	$(call move-if-changed,$@.tmp,$@)
+else
+	@echo "pod2html not installed; skipping $<."
+endif
 
 html/hypercall/index.html: ./xen-headers
 	rm -rf $(@D)
-	@$(INSTALL_DIR) $(@D)
+	$(INSTALL_DIR) $(@D)
 	./xen-headers -O $(@D) \
 		-T 'arch-x86_64 - Xen public headers' \
 		-X arch-ia64 -X arch-x86_32 -X xen-x86_32 -X arch-arm \
@@ -137,11 +152,24 @@
 
 txt/man/%.1.txt: man/%.pod.1 Makefile
 	$(INSTALL_DIR) $(@D)
+ifdef POD2TEXT
 	$(POD2TEXT) $< $@.tmp
 	$(call move-if-changed,$@.tmp,$@)
+else
+	@echo "pod2text not installed; skipping $<."
+endif
 
 txt/man/%.5.txt: man/%.pod.5 Makefile
 	$(INSTALL_DIR) $(@D)
+ifdef POD2TEXT
 	$(POD2TEXT) $< $@.tmp
 	$(call move-if-changed,$@.tmp,$@)
+else
+	@echo "pod2text not installed; skipping $<."
+endif
 
+ifeq (,$(findstring clean,$(MAKECMDGOALS)))
+$(XEN_ROOT)/config/Tools.mk:
+	$(error You have to run ./configure before building docs)
+endif
+
diff -r d7e4efa17fb0 -r 512b4e0c49f3 docs/xen-api/Makefile
--- a/docs/xen-api/Makefile	Tue Aug 28 15:35:08 2012 -0700
+++ b/docs/xen-api/Makefile	Thu Aug 30 10:51:00 2012 -0700
@@ -2,7 +2,7 @@
 
 XEN_ROOT=$(CURDIR)/../..
 include $(XEN_ROOT)/Config.mk
-include $(XEN_ROOT)/docs/Docs.mk
+-include $(XEN_ROOT)/config/Tools.mk
 
 
 TEX := $(wildcard *.tex)
@@ -42,3 +42,8 @@
 .PHONY: clean
 clean:
 	rm -f *.pdf *.ps *.dvi *.aux *.log *.out $(EPSDOT)
+
+ifeq (,$(findstring clean,$(MAKECMDGOALS)))
+$(XEN_ROOT)/config/Tools.mk:
+	$(error You have to run ./configure before building docs)
+endif
diff -r d7e4efa17fb0 -r 512b4e0c49f3 tools/configure.ac
--- a/tools/configure.ac	Tue Aug 28 15:35:08 2012 -0700
+++ b/tools/configure.ac	Thu Aug 30 10:51:00 2012 -0700
@@ -34,6 +34,7 @@
 m4_include([m4/curses.m4])
 m4_include([m4/pthread.m4])
 m4_include([m4/ptyfuncs.m4])
+m4_include([m4/docs_tool.m4])
 
 # Enable/disable options
 AX_ARG_DEFAULT_DISABLE([githttp], [Download GIT repositories via HTTP])
@@ -80,6 +81,18 @@
 AC_PATH_PROG([BISON], [bison])
 AC_PATH_PROG([FLEX], [flex])
 AX_PATH_PROG_OR_FAIL([PERL], [perl])
+AX_DOCS_TOOL_PROG([PS2PDF], [ps2pdf])
+AX_DOCS_TOOL_PROG([DVIPS], [dvips])
+AX_DOCS_TOOL_PROG([LATEX], [latex])
+AX_DOCS_TOOL_PROG([FIG2DEV], [fig2dev])
+AX_DOCS_TOOL_PROG([LATEX2HTML], [latex2html])
+AX_DOCS_TOOL_PROG([DOXYGEN], [doxygen])
+AX_DOCS_TOOL_PROG([POD2MAN], [pod2man])
+AX_DOCS_TOOL_PROG([POD2HTML], [pod2html])
+AX_DOCS_TOOL_PROG([POD2TEXT], [pod2text])
+AX_DOCS_TOOL_PROG([DOT], [dot])
+AX_DOCS_TOOL_PROG([NEATO], [neato])
+AX_DOCS_TOOL_PROGS([MARKDOWN], [markdown markdown_py])
 AS_IF([test "x$xapi" = "xy"], [
     AX_PATH_PROG_OR_FAIL([CURL], [curl-config])
     AX_PATH_PROG_OR_FAIL([XML], [xml2-config])
diff -r d7e4efa17fb0 -r 512b4e0c49f3 tools/m4/docs_tool.m4
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/tools/m4/docs_tool.m4	Thu Aug 30 10:51:00 2012 -0700
@@ -0,0 +1,17 @@
+AC_DEFUN([AX_DOCS_TOOL_PROG], [
+dnl
+    AC_ARG_VAR([$1], [Path to $2 tool])
+    AC_PATH_PROG([$1], [$2])
+    AS_IF([! test -x "$ac_cv_path_$1"], [
+        AC_MSG_WARN([$2 is not available so some documentation won't be built])
+    ])
+])
+
+AC_DEFUN([AX_DOCS_TOOL_PROGS], [
+dnl
+    AC_ARG_VAR([$1], [Path to $2 tool])
+    AC_PATH_PROGS([$1], [$2])
+    AS_IF([! test -x "$ac_cv_path_$1"], [
+        AC_MSG_WARN([$2 is not available so some documentation won't be built])
+    ])
+])

[PATCH 2 of 2 v2] docs: use elinks to format markdown-generated html to text

[Matt Wilson]

Markdown, while easy to read and write, isn't the most consumable
format for users reading documentation on a terminal. This patch uses
lynx to format markdown produced HTML into text files.

Changes since v3:
 * check for html to text dump tool in ./configure
 * switch to using elinks
 * allow command line flags to dump tool to be specified

Signed-off-by: Matt Wilson <msw@amazon.com>

diff -r 512b4e0c49f3 -r 651347cccff7 config/Tools.mk.in
--- a/config/Tools.mk.in	Thu Aug 30 10:51:00 2012 -0700
+++ b/config/Tools.mk.in	Thu Aug 30 11:56:01 2012 -0700
@@ -34,6 +34,8 @@
 DOT                 := @DOT@
 NEATO               := @NEATO@
 MARKDOWN            := @MARKDOWN@
+HTMLDUMP            := @HTMLDUMP@
+HTMLDUMPFLAGS       := @HTMLDUMPFLAGS@
 
 # Extra folder for libs/includes
 PREPEND_INCLUDES    := @PREPEND_INCLUDES@
diff -r 512b4e0c49f3 -r 651347cccff7 docs/Makefile
--- a/docs/Makefile	Thu Aug 30 10:51:00 2012 -0700
+++ b/docs/Makefile	Thu Aug 30 11:56:01 2012 -0700
@@ -146,9 +146,20 @@
 	$(call move-if-changed,$@.tmp,$@)
 
 txt/%.txt: %.markdown
-	$(INSTALL_DIR) $(@D)
-	cp $< $@.tmp
+	@$(INSTALL_DIR) $(@D)
+ifdef MARKDOWN
+ifdef HTMLDUMP
+	@echo "Running markdown to generate $*.txt ... "; \
+	$(MARKDOWN) $< | $(HTMLDUMP) $(HTMLDUMPFLAGS) > $@.tmp
 	$(call move-if-changed,$@.tmp,$@)
+else
+	@echo "html dump tool (like elinks) not installed; just copying $<." \;
+	cp $< $@;
+endif
+else
+	@echo "markdown not installed; just copying $<." \;
+	cp $< $@;
+endif
 
 txt/man/%.1.txt: man/%.pod.1 Makefile
 	$(INSTALL_DIR) $(@D)
diff -r 512b4e0c49f3 -r 651347cccff7 tools/configure.ac
--- a/tools/configure.ac	Thu Aug 30 10:51:00 2012 -0700
+++ b/tools/configure.ac	Thu Aug 30 11:56:01 2012 -0700
@@ -93,6 +93,16 @@
 AX_DOCS_TOOL_PROG([DOT], [dot])
 AX_DOCS_TOOL_PROG([NEATO], [neato])
 AX_DOCS_TOOL_PROGS([MARKDOWN], [markdown markdown_py])
+AC_ARG_VAR([HTMLDUMP],
+           [Path to html-to-text generation tool (default: elinks)])
+AC_PATH_PROG([HTMLDUMP], [elinks])
+AS_IF([! test -x "$ac_cv_path_HTMLDUMP"], [
+    AC_MSG_WARN([$ac_cv_path_HTMLDUMP is not available so text documentation will be unformatted markdown])
+])
+AC_SUBST([HTMLDUMPFLAGS], ["-dump"])
+AC_ARG_VAR([HTMLDUMPFLAGS], [Flags passed to html to text translation tool])
+
+
 AS_IF([test "x$xapi" = "xy"], [
     AX_PATH_PROG_OR_FAIL([CURL], [curl-config])
     AX_PATH_PROG_OR_FAIL([XML], [xml2-config])

Re: [PATCH 1 of 2 v2] tools: check for documentation generation tools at configure time

[Ian Campbell]

On Thu, 2012-08-30 at 19:59 +0100, Matt Wilson wrote:
> It is sometimes hard to discover all the optional tools that should be
> on a system to build all available Xen documentation. By checking for
> documentation generation tools at ./configure time and displaying a
> warning, Xen packagers will more easily learn about new optional build
> dependencies, like markdown, when they are introduced.
> 
> Changes since v1:
>  * require that ./configure be run before building docs
>  * remove Docs.mk and make Tools.mk the canonical locaiton where
>    docs tools are defined (via ./configure)
>  * fold in checking for markdown_py
> 
> Signed-off-by: Matt Wilson <msw@amazon.com>
> 
> diff -r d7e4efa17fb0 -r 512b4e0c49f3 README
> --- a/README	Tue Aug 28 15:35:08 2012 -0700
> +++ b/README	Thu Aug 30 10:51:00 2012 -0700
> @@ -28,8 +28,9 @@
>  your system. For full documentation, see the Xen User Manual. If this
>  is a pre-built release then you can find the manual at:
>   dist/install/usr/share/doc/xen/pdf/user.pdf
> -If you have a source release, then 'make -C docs' will build the
> -manual at docs/pdf/user.pdf.
> +If you have a source release and the required documentation generation
> +tools, then './configure; make -C docs' will build the manual at
> +docs/pdf/user.pdf.

This document was removed in 24563:4271634e4c86, looks like we missed
this reference. Could you nuke it as you go please?

> diff -r d7e4efa17fb0 -r 512b4e0c49f3 tools/m4/docs_tool.m4
> --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
> +++ b/tools/m4/docs_tool.m4	Thu Aug 30 10:51:00 2012 -0700
> @@ -0,0 +1,17 @@
> +AC_DEFUN([AX_DOCS_TOOL_PROG], [
> +dnl
> +    AC_ARG_VAR([$1], [Path to $2 tool])
> +    AC_PATH_PROG([$1], [$2])
> +    AS_IF([! test -x "$ac_cv_path_$1"], [
> +        AC_MSG_WARN([$2 is not available so some documentation won't be built])
> +    ])
> +])
> +
> +AC_DEFUN([AX_DOCS_TOOL_PROGS], [
> +dnl
> +    AC_ARG_VAR([$1], [Path to $2 tool])

Does this do something sensible when $2 is a space separated list?

Do we need both PROG and PROGS variants? PROG is just a special case of
PROGS, isn't it? Although seeing that AC apparently has both variants I
guess there's a reason for that.

> +    AC_PATH_PROGS([$1], [$2])
> +    AS_IF([! test -x "$ac_cv_path_$1"], [
> +        AC_MSG_WARN([$2 is not available so some documentation won't be built])
> +    ])
> +])

Re: [PATCH 2 of 2 v2] docs: use elinks to format markdown-generated html to text

[Ian Campbell]

On Thu, 2012-08-30 at 20:00 +0100, Matt Wilson wrote:
> Markdown, while easy to read and write, isn't the most consumable
> format for users reading documentation on a terminal. This patch uses
> lynx to format markdown produced HTML into text files.

s/lynx/elinks/ in the comment.

> diff -r 512b4e0c49f3 -r 651347cccff7 config/Tools.mk.in
> --- a/config/Tools.mk.in	Thu Aug 30 10:51:00 2012 -0700
> +++ b/config/Tools.mk.in	Thu Aug 30 11:56:01 2012 -0700
> @@ -34,6 +34,8 @@
>  DOT                 := @DOT@
>  NEATO               := @NEATO@
>  MARKDOWN            := @MARKDOWN@
> +HTMLDUMP            := @HTMLDUMP@
> +HTMLDUMPFLAGS       := @HTMLDUMPFLAGS@

Does HTMLDUMP="elinks -dump" not work?

Ian.

Re: [PATCH 1 of 2 v2] tools: check for documentation generation tools at configure time

[Ian Campbell]

On Fri, 2012-08-31 at 10:25 +0100, Ian Campbell wrote:
> > +AC_DEFUN([AX_DOCS_TOOL_PROGS], [
> > +dnl
> > +    AC_ARG_VAR([$1], [Path to $2 tool])
> 
> Does this do something sensible when $2 is a space separated list?

Not especially:
$ ./configure --help| grep mark
   MARKDOWN    Path to markdown markdown_py tool

Ian.

Re: [PATCH 1 of 2 v2] tools: check for documentation generation tools at configure time

[Ian Campbell]

On Thu, 2012-08-30 at 19:59 +0100, Matt Wilson wrote:
> It is sometimes hard to discover all the optional tools that should be
> on a system to build all available Xen documentation. By checking for
> documentation generation tools at ./configure time and displaying a
> warning, Xen packagers will more easily learn about new optional build
> dependencies, like markdown, when they are introduced.

Having remembered to rerun autogen.sh I can confirm that I get the same
set of docs as I did before on my regular build machine. I get no
warnings about missing tools so I suppose I'm building all the docs.

(note to self, when this patch goes in check that the cronjob running
http://xenbits.xen.org/docs/unstable/ still works. reminders
appreciated...)

Ian.

Re: [PATCH 2 of 2 v2] docs: use elinks to format markdown-generated html to text

[Matt Wilson]

On Fri, Aug 31, 2012 at 10:26:51AM +0100, Ian Campbell wrote:
> On Thu, 2012-08-30 at 20:00 +0100, Matt Wilson wrote:
> > Markdown, while easy to read and write, isn't the most consumable
> > format for users reading documentation on a terminal. This patch uses
> > lynx to format markdown produced HTML into text files.
> 
> s/lynx/elinks/ in the comment.
> 
> > diff -r 512b4e0c49f3 -r 651347cccff7 config/Tools.mk.in
> > --- a/config/Tools.mk.in	Thu Aug 30 10:51:00 2012 -0700
> > +++ b/config/Tools.mk.in	Thu Aug 30 11:56:01 2012 -0700
> > @@ -34,6 +34,8 @@
> >  DOT                 := @DOT@
> >  NEATO               := @NEATO@
> >  MARKDOWN            := @MARKDOWN@
> > +HTMLDUMP            := @HTMLDUMP@
> > +HTMLDUMPFLAGS       := @HTMLDUMPFLAGS@
> 
> Does HTMLDUMP="elinks -dump" not work?

I don't think so. AC_PATH_PROG() will just want the program name. This
follows the CC / CFLAGS model.

Matt

Re: [PATCH 1 of 2 v2] tools: check for documentation generation tools at configure time

[Matt Wilson]

On Fri, Aug 31, 2012 at 01:30:00PM +0100, Ian Campbell wrote:
> On Fri, 2012-08-31 at 10:25 +0100, Ian Campbell wrote:
> > > +AC_DEFUN([AX_DOCS_TOOL_PROGS], [
> > > +dnl
> > > +    AC_ARG_VAR([$1], [Path to $2 tool])
> > 
> > Does this do something sensible when $2 is a space separated list?
> 
> Not especially:
> $ ./configure --help| grep mark
>    MARKDOWN    Path to markdown markdown_py tool

I'll fix that up.

Matt