[RFC] A first shot at asciidoc-based formatted docs

[RFC] A first shot at asciidoc-based formatted docs

[Jonathan Corbet]

So here is a proof-of-concept series showing how a fully asciidoc-based
toolchain might work.  Lots of hackery here, this isn't meant to be applied
to anything at this point, but it's a good start.  What this series has is:

 - Jani Nikula's patch adding asciidoc output to kernel-doc.  Thanks for
   doing this!  It was the kickstart that was needed to get this process
   going.

 - Tweak docproc to handle asciidoc template files.  If a template ends in
   ".adt", it's an asciidoc template; it's processed pretty much the same
   way, except that kernel-doc gets the -asciidoc argument.

 - Bash on the Makefile to get it to process asciidoc templates into HTML.
   Naturally this was where most of the time got spent.  *Only* HTML output
   works at the moment.

 - Convert tracepoints.html to tpoint.adt as a proof of concept.  It works,
   and the output is much pleasing, IMO.

I'm sure there's a thousand details to deal with, and there is the issue of
the other output formats.  asciidoctor claims to be able to create man
pages, but I've not tried that yet; neither tool will do PDF.  Maybe we
could rely on pandoc to do that.  Otherwise, getting to asciidoc to XML is
straightforward, so it should be possible to use xmlto as is done now.

It's all in the doc/asciidoc branch of git://git.lwn.net/linux.git if
anybody wants to mess with it.

Comments?

jon

[PATCH 1/4] kernel-doc: add support for asciidoc output

[Jonathan Corbet]

From: Jani Nikula <jani.nikula@intel.com>

Add new -asciidoc option to produce asciidoc output from kernel-doc. The
output is formatted internally, with no dependencies on external
tools. Any asciidoc formatting present in kernel-doc will naturally be
present in the resulting asciidoc as well.

Highlighting of functions(), &struct structures, &enum enumerations,
@parameters, etc. will be done by means of asciidoc. Anchors and
cross-references are added as well, providing hyperlinking support in
the result processed by asciidoc(1).

This support is non-conflicting and orthogonal to the patches adding
asciidoc support to the kernel-doc DocBook output [1]. Those patches
bolt on to the current document generation pipeline; there is currently
none for native asciidoc in the kernel (though ideas have been discussed
[2]). At this time, this patch should be considered a worthwhile
standalone improvement to kernel-doc, independent of the rest.

[1] http://mid.gmane.org/1448471279-19748-1-git-send-email-daniel.vetter@ffwll.ch
[2] http://mid.gmane.org/20160114131823.2ff43a0c@lwn.net

Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Daniel Vetter <daniel@ffwll.ch>
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/kernel-doc | 237 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 237 insertions(+)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index c37255b..06f1649 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -201,6 +201,8 @@ my $type_param = '\@(\w+)';
 my $type_struct = '\&((struct\s*)*[_\w]+)';
 my $type_struct_xml = '\\&amp;((struct\s*)*[_\w]+)';
 my $type_env = '(\$\w+)';
+my $type_enum_full = '\&(enum)\s*([_\w]+)';
+my $type_struct_full = '\&(struct)\s*([_\w]+)';
 
 # Output conversion substitutions.
 #  One for each output format
@@ -266,6 +268,17 @@ my @highlights_text = (
 		      );
 my $blankline_text = "";
 
+# asciidoc-mode
+my @highlights_asciidoc = (
+                       [$type_constant, "`\$1`"],
+                       [$type_func, "<<func_\$1,`\$1()`>>"],
+                       [$type_struct_full, "<<\$1_\$2,`\$1 \$2`>>"],
+                       [$type_enum_full, "<<\$1_\$2,`\$1 \$2`>>"],
+                       [$type_struct, "<<struct_\$1,`\$1`>>"],
+                       [$type_param, "*\$1*"]
+		      );
+my $blankline_asciidoc = "\n";
+
 # list mode
 my @highlights_list = (
                        [$type_constant, "\$1"],
@@ -402,6 +415,10 @@ while ($ARGV[0] =~ m/^-(.*)/) {
 	$output_mode = "text";
 	@highlights = @highlights_text;
 	$blankline = $blankline_text;
+    } elsif ($cmd eq "-asciidoc") {
+	$output_mode = "asciidoc";
+	@highlights = @highlights_asciidoc;
+	$blankline = $blankline_asciidoc;
     } elsif ($cmd eq "-docbook") {
 	$output_mode = "xml";
 	@highlights = @highlights_xml;
@@ -1713,6 +1730,214 @@ sub output_blockhead_text(%) {
     }
 }
 
+##
+# output in asciidoc
+sub output_highlight_asciidoc {
+    my $contents = join "\n",@_;
+    my $line;
+
+    # undo the evil effects of xml_escape() earlier
+    $contents = xml_unescape($contents);
+
+    eval $dohighlight;
+    die $@ if $@;
+
+    foreach $line (split "\n", $contents) {
+	if ($line eq "") {
+	    print $lineprefix, $blankline;
+	} else {
+	    $line =~ s/\\\\\\/\&/g;
+	    print $lineprefix, $line;
+	}
+	print "\n";
+    }
+}
+
+sub output_function_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter, $section);
+    my $start;
+
+    print "[[func_$args{'function'}]]\n";
+    print "=== " . $args{'function'} . " ===\n";
+    print $args{'purpose'} . "\n\n";
+
+    print "----------\n";
+    if ($args{'functiontype'} ne "") {
+	$start = $args{'functiontype'} . " " . $args{'function'} . " (";
+    } else {
+	$start = $args{'function'} . " (";
+    }
+    print $start;
+
+    my $count = 0;
+    foreach my $parameter (@{$args{'parameterlist'}}) {
+	$type = $args{'parametertypes'}{$parameter};
+	if ($type =~ m/([^\(]*\(\*)\s*\)\s*\(([^\)]*)\)/) {
+	    # pointer-to-function
+	    print $1 . $parameter . ") (" . $2;
+	} else {
+	    print $type . " " . $parameter;
+	}
+	if ($count != $#{$args{'parameterlist'}}) {
+	    $count++;
+	    print ",\n";
+	    print " " x length($start);
+	} else {
+	    print ");\n\n";
+	}
+    }
+    print "----------\n";
+
+    print ".Parameters\n";
+    print "[horizontal]\n";
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	my $parameter_name = $parameter;
+	$parameter_name =~ s/\[.*//;
+	$type = $args{'parametertypes'}{$parameter};
+
+	print "`$type $parameter`::\n";
+	if ($args{'parameterdescs'}{$parameter_name} ne $undescribed) {
+	    $blankline = "+";
+	    output_highlight_asciidoc($args{'parameterdescs'}{$parameter_name});
+	    $blankline = "\n";
+	} else {
+	    print "_undescribed_\n";
+	}
+	print "\n";
+    }
+    output_section_asciidoc(@_);
+}
+
+sub output_section_asciidoc(%) {
+    my %args = %{$_[0]};
+    my $section;
+
+    foreach $section (@{$args{'sectionlist'}}) {
+	print ".$section\n\n";
+	output_highlight_asciidoc($args{'sections'}{$section});
+	print "\n";
+    }
+    print "\n";
+}
+
+sub output_enum_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter);
+    my $count;
+
+    print "[[enum_$args{'enum'}]]\n";
+    print "=== enum " . $args{'enum'} . " ===\n";
+    print $args{'purpose'} . "\n\n";
+
+    print "----------\n";
+    print "enum " . $args{'enum'} . " {\n";
+    $count = 0;
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	print "\t$parameter";
+	if ($count != $#{$args{'parameterlist'}}) {
+	    $count++;
+	    print ",";
+	}
+	print "\n";
+    }
+    print "};\n";
+    print "----------\n";
+
+    print ".Constants\n";
+    print "[horizontal]\n";
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	print "`$parameter`::\n";
+	if ($args{'parameterdescs'}{$parameter} ne $undescribed) {
+	    $blankline = "+";
+	    output_highlight_asciidoc($args{'parameterdescs'}{$parameter});
+	    $blankline = "\n";
+	} else {
+	    print "_undescribed_\n";
+	}
+	print "\n";
+    }
+
+    output_section_asciidoc(@_);
+}
+
+sub output_typedef_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter);
+    my $count;
+
+    print "[[$args{'typedef'}]]\n";
+    print "=== typedef " . $args{'typedef'} . " ===\n";
+    print $args{'purpose'} . "\n\n";
+
+    output_section_asciidoc(@_);
+}
+
+sub output_struct_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter);
+
+    print "[[$args{'type'}_$args{'struct'}]]\n";
+    print "=== " . $args{'type'} . " " . $args{'struct'} . " ===\n";
+    print $args{'purpose'} . "\n\n";
+
+    print "----------\n";
+    print $args{'type'} . " " . $args{'struct'} . " {\n";
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	if ($parameter =~ /^#/) {
+	    print "$parameter\n";
+	    next;
+	}
+
+	my $parameter_name = $parameter;
+	$parameter_name =~ s/\[.*//;
+
+	($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next;
+	$type = $args{'parametertypes'}{$parameter};
+	if ($type =~ m/([^\(]*\(\*)\s*\)\s*\(([^\)]*)\)/) {
+	    # pointer-to-function
+	    print "\t$1 $parameter) ($2);\n";
+	} elsif ($type =~ m/^(.*?)\s*(:.*)/) {
+	    # bitfield
+	    print "\t$1 $parameter$2;\n";
+	} else {
+	    print "\t" . $type . " " . $parameter . ";\n";
+	}
+    }
+    print "};\n";
+    print "----------\n";
+
+    print ".Members\n";
+    print "[horizontal]\n";
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	($parameter =~ /^#/) && next;
+
+	my $parameter_name = $parameter;
+	$parameter_name =~ s/\[.*//;
+
+	($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next;
+	$type = $args{'parametertypes'}{$parameter};
+	print "`$type $parameter`" . "::\n";
+	$blankline = "+";
+	output_highlight_asciidoc($args{'parameterdescs'}{$parameter_name});
+	$blankline = "\n";
+	print "\n";
+    }
+    print "\n";
+    output_section_asciidoc(@_);
+}
+
+sub output_blockhead_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter, $section);
+
+    foreach $section (@{$args{'sectionlist'}}) {
+	print "=== $section ===\n";
+	output_highlight_asciidoc($args{'sections'}{$section});
+	print "\n";
+    }
+}
+
 ## list mode output functions
 
 sub output_function_list(%) {
@@ -2414,6 +2639,18 @@ sub xml_escape($) {
 	return $text;
 }
 
+# xml_unescape: reverse the effects of xml_escape
+sub xml_unescape($) {
+	my $text = shift;
+	if (($output_mode eq "text") || ($output_mode eq "man")) {
+		return $text;
+	}
+	$text =~ s/\\\\\\amp;/\&/g;
+	$text =~ s/\\\\\\lt;/</g;
+	$text =~ s/\\\\\\gt;/>/g;
+	return $text;
+}
+
 # convert local escape strings to html
 # local escape strings look like:  '\\\\menmonic:' (that's 4 backslashes)
 sub local_unescape($) {
-- 
2.7.0

[PATCH 2/4] docproc: handle asciidoc templates

[Jonathan Corbet]

There's really nothing different that needs to be done except for invoking
kernel-doc with the -asciidoc argument.  Look at the input file name to
recognize asciidoc templates, so no special command-line flags are needed.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/docproc.c | 25 ++++++++++++++++++++-----
 1 file changed, 20 insertions(+), 5 deletions(-)

diff --git a/scripts/docproc.c b/scripts/docproc.c
index e267e621..89284b5 100644
--- a/scripts/docproc.c
+++ b/scripts/docproc.c
@@ -68,12 +68,15 @@ FILELINE * docsection;
 #define KERNELDOCPATH "scripts/"
 #define KERNELDOC     "kernel-doc"
 #define DOCBOOK       "-docbook"
+#define ASCIIDOC      "-asciidoc"
 #define LIST          "-list"
 #define FUNCTION      "-function"
 #define NOFUNCTION    "-nofunction"
 #define NODOCSECTIONS "-no-doc-sections"
 #define SHOWNOTFOUND  "-show-not-found"
 
+static char *doc_format = DOCBOOK;
+
 static char *srctree, *kernsrctree;
 
 static char **all_list = NULL;
@@ -242,7 +245,7 @@ static void find_export_symbols(char * filename)
 /*
  * Document all external or internal functions in a file.
  * Call kernel-doc with following parameters:
- * kernel-doc -docbook -nofunction function_name1 filename
+ * kernel-doc [-docbook|-asciidoc] -nofunction function_name1 filename
  * Function names are obtained from all the src files
  * by find_export_symbols.
  * intfunc uses -nofunction
@@ -263,7 +266,7 @@ static void docfunctions(char * filename, char * type)
 		exit(1);
 	}
 	vec[idx++] = KERNELDOC;
-	vec[idx++] = DOCBOOK;
+	vec[idx++] = doc_format;
 	vec[idx++] = NODOCSECTIONS;
 	for (i=0; i < symfilecnt; i++) {
 		struct symfile * sym = &symfilelist[i];
@@ -275,7 +278,7 @@ static void docfunctions(char * filename, char * type)
 	}
 	vec[idx++]     = filename;
 	vec[idx] = NULL;
-	printf("<!-- %s -->\n", filename);
+	/* printf("<!-- %s -->\n", filename); */
 	exec_kernel_doc(vec);
 	fflush(stdout);
 	free(vec);
@@ -294,7 +297,7 @@ static void singfunc(char * filename, char * line)
 	int i, idx = 0;
 	int startofsym = 1;
 	vec[idx++] = KERNELDOC;
-	vec[idx++] = DOCBOOK;
+	vec[idx++] = doc_format;
 	vec[idx++] = SHOWNOTFOUND;
 
 	/* Split line up in individual parameters preceded by FUNCTION */
@@ -343,7 +346,7 @@ static void docsect(char *filename, char *line)
 	free(s);
 
 	vec[0] = KERNELDOC;
-	vec[1] = DOCBOOK;
+	vec[1] = doc_format;
 	vec[2] = SHOWNOTFOUND;
 	vec[3] = FUNCTION;
 	vec[4] = line;
@@ -497,6 +500,16 @@ static void parse_file(FILE *infile)
 	fflush(stdout);
 }
 
+/*
+ * Is this an asciidoc template?  Answer the question by seeing if its
+ * name ends in ".adt".
+ */
+static int is_asciidoc(const char *file)
+{
+	int len = strlen(file);
+
+	return len > 4 && ! strcmp(file + len - 4, ".adt");
+}
 
 int main(int argc, char *argv[])
 {
@@ -520,6 +533,8 @@ int main(int argc, char *argv[])
 		perror(argv[2]);
 		exit(2);
 	}
+	if (is_asciidoc(argv[2]))
+		doc_format = ASCIIDOC;
 
 	if (strcmp("doc", argv[1]) == 0) {
 		/* Need to do this in two passes.
-- 
2.7.0

[PATCH 3/4] Docs: Makefile tweaks for asciidoc templates

[Jonathan Corbet]

This is a hatchet job, but it's something to start with.  Generalize some
of the string manipulation to not assume that templates have a ".tmpl"
suffix, and add rules to translate asciidoc templates to HTML.  Nothing for
any other output formats at this point.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 Documentation/DocBook/Makefile | 72 +++++++++++++++++++++++++-----------------
 1 file changed, 43 insertions(+), 29 deletions(-)

diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile
index d70f9b6..f04e8c8 100644
--- a/Documentation/DocBook/Makefile
+++ b/Documentation/DocBook/Makefile
@@ -6,16 +6,16 @@
 # To add a new book the only step required is to add the book to the
 # list of DOCBOOKS.
 
-DOCBOOKS := z8530book.xml device-drivers.xml \
-	    kernel-hacking.xml kernel-locking.xml deviceiobook.xml \
-	    writing_usb_driver.xml networking.xml \
-	    kernel-api.xml filesystems.xml lsm.xml usb.xml kgdb.xml \
-	    gadget.xml libata.xml mtdnand.xml librs.xml rapidio.xml \
-	    genericirq.xml s390-drivers.xml uio-howto.xml scsi.xml \
-	    80211.xml debugobjects.xml sh.xml regulator.xml \
-	    alsa-driver-api.xml writing-an-alsa-driver.xml \
-	    tracepoint.xml gpu.xml media_api.xml w1.xml \
-	    writing_musb_glue_layer.xml crypto-API.xml iio.xml
+DOCBOOKS := z8530book device-drivers \
+	    kernel-hacking kernel-locking deviceiobook \
+	    writing_usb_driver networking \
+	    kernel-api filesystems lsm usb kgdb \
+	    gadget libata mtdnand librs rapidio \
+	    genericirq s390-drivers uio-howto scsi \
+	    80211 debugobjects sh regulator \
+	    alsa-driver-api writing-an-alsa-driver \
+	    tracepoint gpu media_api w1 \
+	    writing_musb_glue_layer crypto-API iio
 
 include Documentation/DocBook/media/Makefile
 
@@ -39,21 +39,21 @@ PHONY += xmldocs sgmldocs psdocs pdfdocs htmldocs mandocs installmandocs cleando
 
 targets += $(DOCBOOKS)
 BOOKS := $(addprefix $(obj)/,$(DOCBOOKS))
-xmldocs: $(BOOKS)
+xmldocs: $(addsuffix .xml, $(BOOKS))
 sgmldocs: xmldocs
 
-PS := $(patsubst %.xml, %.ps, $(BOOKS))
+PS := $(addsuffix .ps, $(BOOKS))
 psdocs: $(PS)
 
-PDF := $(patsubst %.xml, %.pdf, $(BOOKS))
+PDF := $(addsuffix .pdf, $(BOOKS))
 pdfdocs: $(PDF)
 
-HTML := $(sort $(patsubst %.xml, %.html, $(BOOKS)))
+HTML := $(sort $(addsuffix .html, $(BOOKS)))
 htmldocs: $(HTML)
 	$(call cmd,build_main_index)
 	$(call install_media_images)
 
-MAN := $(patsubst %.xml, %.9, $(BOOKS))
+MAN := $(addsuffix .9, $(BOOKS))
 mandocs: $(MAN)
 	find $(obj)/man -name '*.9' | xargs gzip -nf
 
@@ -103,6 +103,19 @@ endef
 # Tell kbuild to always build the programs
 always := $(hostprogs-y)
 
+#
+# asciidoc stuff.
+#
+quiet_cmd_ad2html = ASCIIDOC	$@
+      cmd_ad2html = asciidoc -b html $< > $@
+
+%.ad: %.adt $(KERNELDOC) $(DOCPROC) FORCE
+	$(call cmd,docproc)
+
+%.html: %.ad
+	$(call cmd,ad2html)
+
+
 notfoundtemplate = echo "*** You have to install docbook-utils or xmlto ***"; \
 		   exit 1
 db2xtemplate = db2TYPE -o $(dir $@) $<
@@ -234,22 +247,23 @@ dochelp:
 
 ###
 # Temporary files left by various tools
-clean-files := $(DOCBOOKS) \
-	$(patsubst %.xml, %.dvi,     $(DOCBOOKS)) \
-	$(patsubst %.xml, %.aux,     $(DOCBOOKS)) \
-	$(patsubst %.xml, %.tex,     $(DOCBOOKS)) \
-	$(patsubst %.xml, %.log,     $(DOCBOOKS)) \
-	$(patsubst %.xml, %.out,     $(DOCBOOKS)) \
-	$(patsubst %.xml, %.ps,      $(DOCBOOKS)) \
-	$(patsubst %.xml, %.pdf,     $(DOCBOOKS)) \
-	$(patsubst %.xml, %.html,    $(DOCBOOKS)) \
-	$(patsubst %.xml, %.9,       $(DOCBOOKS)) \
-	$(patsubst %.xml, %.aux.xml, $(DOCBOOKS)) \
-	$(patsubst %.xml, %.xml.db,  $(DOCBOOKS)) \
-	$(patsubst %.xml, %.xml,     $(DOCBOOKS)) \
+clean-files := \
+	$(addsuffix .dvi,     $(DOCBOOKS)) \
+	$(addsuffix .aux,     $(DOCBOOKS)) \
+	$(addsuffix .tex,     $(DOCBOOKS)) \
+	$(addsuffix .log,     $(DOCBOOKS)) \
+	$(addsuffix .out,     $(DOCBOOKS)) \
+	$(addsuffix .ps,      $(DOCBOOKS)) \
+	$(addsuffix .pdf,     $(DOCBOOKS)) \
+	$(addsuffix .html,    $(DOCBOOKS)) \
+	$(addsuffix .9,       $(DOCBOOKS)) \
+	$(addsuffix .aux.xml, $(DOCBOOKS)) \
+	$(addsuffix .xml.db,  $(DOCBOOKS)) \
+	$(addsuffix .xml,     $(DOCBOOKS)) \
+	$(addsuffix .ad,      $(DOCBOOKS)) \
 	$(index)
 
-clean-dirs := $(patsubst %.xml,%,$(DOCBOOKS)) man
+clean-dirs := $(DOCBOOKS) man
 
 cleandocs: cleanmediadocs
 	$(Q)rm -f $(call objectify, $(clean-files))
-- 
2.7.0

[PATCH 4/4] Docs: add a sample asciidoc template

[Jonathan Corbet]

This is just a copy of tracepoints.tmpl (because it was short!) converted
into asciidoc.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 Documentation/DocBook/Makefile   |  2 +-
 Documentation/DocBook/tpoint.adt | 64 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 65 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/DocBook/tpoint.adt

diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile
index f04e8c8..2e5195e 100644
--- a/Documentation/DocBook/Makefile
+++ b/Documentation/DocBook/Makefile
@@ -15,7 +15,7 @@ DOCBOOKS := z8530book device-drivers \
 	    80211 debugobjects sh regulator \
 	    alsa-driver-api writing-an-alsa-driver \
 	    tracepoint gpu media_api w1 \
-	    writing_musb_glue_layer crypto-API iio
+	    writing_musb_glue_layer crypto-API iio tpoint
 
 include Documentation/DocBook/media/Makefile
 
diff --git a/Documentation/DocBook/tpoint.adt b/Documentation/DocBook/tpoint.adt
new file mode 100644
index 0000000..f93863a
--- /dev/null
+++ b/Documentation/DocBook/tpoint.adt
@@ -0,0 +1,64 @@
+The Linux Kernel Tracepoint API
+===============================
+Jason Baron, William Cohen
+
+.This document
+****************************************************************
+This documentation is free software; you can redistribute
+it and/or modify it under the terms of the GNU General Public
+License as published by the Free Software Foundation; either
+version 2 of the License, or (at your option) any later
+version.
+
+This program is distributed in the hope that it will be
+useful, but WITHOUT ANY WARRANTY; without even the implied
+warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+See the GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public
+License along with this program; if not, write to the Free
+Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+MA 02111-1307 USA
+
+For more details see the file COPYING in the source
+distribution of Linux.
+****************************************************************
+
+Introduction
+------------
+
+Tracepoints are static probe points that are located in strategic points
+throughout the kernel. 'Probes' register/unregister with tracepoints
+via a callback mechanism. The 'probes' are strictly typed functions that
+are passed a unique set of parameters defined by each tracepoint.
+
+From this simple callback mechanism, 'probes' can be used to profile, debug,
+and understand kernel behavior. There are a number of tools that provide a
+framework for using 'probes'. These tools include Systemtap, ftrace, and
+LTTng.
+
+Tracepoints are defined in a number of header files via various macros. Thus,
+the purpose of this document is to provide a clear accounting of the available
+tracepoints. The intention is to understand not only what tracepoints are
+available but also to understand where future tracepoints might be added.
+
+The API presented has functions of the form:
++trace_tracepointname(function parameters)+. These are the
+tracepoints callbacks that are found throughout the code. Registering and
+unregistering probes with these callback sites is covered in the
+'Documentation/trace/*' directory.
+
+IRQ
+---
+!Iinclude/trace/events/irq.h
+
+SIGNAL
+------
+!Iinclude/trace/events/signal.h
+
+Block IO
+!Iinclude/trace/events/block.h
+
+Workqueue
+---------
+!Iinclude/trace/events/workqueue.h
-- 
2.7.0

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jani Nikula]

On Tue, 26 Jan 2016, Jonathan Corbet <corbet@lwn.net> wrote:
> So here is a proof-of-concept series showing how a fully asciidoc-based
> toolchain might work.  Lots of hackery here, this isn't meant to be applied
> to anything at this point, but it's a good start.  What this series has is:
>
>  - Jani Nikula's patch adding asciidoc output to kernel-doc.  Thanks for
>    doing this!  It was the kickstart that was needed to get this process
>    going.

Hooray! :)

>  - Tweak docproc to handle asciidoc template files.  If a template ends in
>    ".adt", it's an asciidoc template; it's processed pretty much the same
>    way, except that kernel-doc gets the -asciidoc argument.
>
>  - Bash on the Makefile to get it to process asciidoc templates into HTML.
>    Naturally this was where most of the time got spent.  *Only* HTML output
>    works at the moment.
>
>  - Convert tracepoints.html to tpoint.adt as a proof of concept.  It works,
>    and the output is much pleasing, IMO.
>
> I'm sure there's a thousand details to deal with, and there is the issue of
> the other output formats.  asciidoctor claims to be able to create man
> pages, but I've not tried that yet; neither tool will do PDF.  Maybe we
> could rely on pandoc to do that.  Otherwise, getting to asciidoc to XML is
> straightforward, so it should be possible to use xmlto as is done now.
>
> It's all in the doc/asciidoc branch of git://git.lwn.net/linux.git if
> anybody wants to mess with it.
>
> Comments?

I'm afraid we've done some overlapping work in the mean time, but I'm
happy we've both looked at the tool chain, and can have a more
meaningful conversation now.

I first took roughly the same approach as you did. I was really
impressed with the speed and the beauty of the produced HTML. The
trouble is, neither asciidoc nor asciidoctor can produce chunked (split
to several pages) HTML directly. This is a showstopper for the gpu
document which turns into 1.3 MB of HTML, which looks pretty but is a
paint to navigate. To do chunked output, you have to output DocBook and
handle that like we do now. So while I would like to have asciidoc
generate HTML directly for speed and beauty, I ended up going the
asciidoc to DocBook path. The upside is all the output formats are
supported.

(We could of course split the source documents, but then I believe we'd
have lots of trouble cross-referencing between the documents. I could be
proven wrong. I'd *like* to be proven wrong.)

One significant difference between our approaches is that I ditched
docproc out of the equation. Instead of having the docproc ! directives
in the asciidoc, I opted for using asciidoc's native include macro, with
specially crafted filename suffixes to specify what parts of the source
to include. Those files are then generated as intermediate asciidoc
files using kernel-doc, with dependencies set right. There's a bunch of
scripting involved, but it's pretty straight forward. 

So you'd have, for example,

include::drivers/gpu/drm/drm_ioctl.c,export[]

to include all the exported functions, or

include::drivers/gpu/drm/i915/i915_irq.c,doc,interrupt_handling[]

to include the DOC: section. I think I'd like some version of this,
regardless of whether asciidoc generates HTML or DocBook. It lets make
parallelize all of kernel-doc processing for free, which is a big win.

Patches follow, also available in kernel-asciidoc branch of
git://people.freedesktop.org/~jani/drm (web interface
http://cgit.freedesktop.org/~jani/drm/log/?h=kernel-asciidoc)

BR,
Jani.


Jani Nikula (10):
  kernel-doc: rewrite usage description, remove duplicated comments
  kernel-doc: add support for asciidoc output
  kernel-doc: support printing exported and non-exported symbols
  kernel-doc: add support for printing DOC: comments with escaped names
  scripts: add asciidoc-includes to extract includes from asciidoc
  scripts: add a kernel-doc helper for special invocation
  scripts: add tool for generating asciidoc dependencies and rules
  scripts: add a crude converter from DocBook tmpl to asciidoc
  Documentation: convert gpu.tmpl to gpu.txt
  Documentation: build asciidoc documentation

 Documentation/DocBook/Makefile |   37 +-
 Documentation/DocBook/gpu.tmpl | 3499 ----------------------------------------
 Documentation/DocBook/gpu.txt  | 1183 ++++++++++++++
 scripts/asciidoc-includes      |    6 +
 scripts/kernel-doc             |  365 ++++-
 scripts/kernel-doc-deps        |   66 +
 scripts/kernel-doc-helper      |   70 +
 scripts/tmpl2asciidoc          |   39 +
 8 files changed, 1709 insertions(+), 3556 deletions(-)
 delete mode 100644 Documentation/DocBook/gpu.tmpl
 create mode 100644 Documentation/DocBook/gpu.txt
 create mode 100755 scripts/asciidoc-includes
 create mode 100755 scripts/kernel-doc-deps
 create mode 100755 scripts/kernel-doc-helper
 create mode 100755 scripts/tmpl2asciidoc

-- 
2.1.4

[RFC 01/10] kernel-doc: rewrite usage description, remove duplicated comments

[Jani Nikula]

Instead of having the kernel-doc usage in both comments and in output to
the user, merge them all to one here document. While at it, imrove the
text and make it pretty. Give shoemaker's children some shoes.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 83 ++++++++++++++++++++++++------------------------------
 1 file changed, 37 insertions(+), 46 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index c37255bb620d..29fd5cabb657 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -39,41 +39,43 @@ use strict;
 # 25/07/2012 - Added support for HTML5
 # -- Dan Luedtke <mail@danrl.de>
 
-#
-# This will read a 'c' file and scan for embedded comments in the
-# style of gnome comments (+minor extensions - see below).
-#
-
-# Note: This only supports 'c'.
-
-# usage:
-# kernel-doc [ -docbook | -html | -html5 | -text | -man | -list ]
-#            [ -no-doc-sections ]
-#            [ -function funcname [ -function funcname ...] ]
-#            c file(s)s > outputfile
-# or
-#            [ -nofunction funcname [ -function funcname ...] ]
-#            c file(s)s > outputfile
-#
-#  Set output format using one of -docbook -html -html5 -text or -man.
-#  Default is man.
-#  The -list format is for internal use by docproc.
-#
-#  -no-doc-sections
-#	Do not output DOC: sections
-#
-#  -function funcname
-#	If set, then only generate documentation for the given function(s) or
-#	DOC: section titles.  All other functions and DOC: sections are ignored.
-#
-#  -nofunction funcname
-#	If set, then only generate documentation for the other function(s)/DOC:
-#	sections. Cannot be used together with -function (yes, that's a bug --
-#	perl hackers can fix it 8))
-#
-#  c files - list of 'c' files to process
-#
-#  All output goes to stdout, with errors to stderr.
+sub usage {
+    my $message = <<"EOF";
+Usage: $0 [OPTION ...] FILE ...
+
+Read C language source or header FILEs, extract embedded documentation comments,
+and print formatted documentation to standard output.
+
+The documentation comments are identified by "/**" opening comment mark. See
+Documentation/kernel-doc-nano-HOWTO.txt for the documentation comment syntax.
+
+Output format selection (mutually exclusive):
+  -docbook		Output DocBook format.
+  -html			Output HTML format.
+  -html5		Output HTML5 format.
+  -list			Output symbol list format. This is for use by docproc.
+  -man			Output troff manual page format. This is the default.
+  -text			Output plain text format.
+
+Output selection (mutually exclusive):
+  -function NAME	Only output documentation for the given function(s)
+			or DOC: section title(s). All other functions and DOC:
+			sections are ignored. May be specified multiple times.
+  -nofunction NAME	Do NOT output documentation for the given function(s);
+			only output documentation for the other functions and
+			DOC: sections. May be specified multiple times.
+
+Output selection modifiers:
+  -no-doc-sections	Do not output DOC: sections.
+
+Other parameters:
+  -v			Verbose output, more warnings and other information.
+  -h			Print this help.
+
+EOF
+    print $message;
+    exit 1;
+}
 
 #
 # format of comments.
@@ -437,17 +439,6 @@ while ($ARGV[0] =~ m/^-(.*)/) {
 
 # continue execution near EOF;
 
-sub usage {
-    print "Usage: $0 [ -docbook | -html | -html5 | -text | -man | -list ]\n";
-    print "         [ -no-doc-sections ]\n";
-    print "         [ -function funcname [ -function funcname ...] ]\n";
-    print "         [ -nofunction funcname [ -nofunction funcname ...] ]\n";
-    print "         [ -v ]\n";
-    print "         c source file(s) > outputfile\n";
-    print "         -v : verbose output, more warnings & other info listed\n";
-    exit 1;
-}
-
 # get kernel version from env
 sub get_kernel_version() {
     my $version = 'unknown kernel version';
-- 
2.1.4

[RFC 02/10] kernel-doc: add support for asciidoc output

[Jani Nikula]

Add new -asciidoc option to produce asciidoc output from kernel-doc. The
output is formatted internally, with no dependencies on external
tools. Any asciidoc formatting present in kernel-doc will naturally be
present in the resulting asciidoc as well.

Highlighting of functions(), &struct structures, &enum enumerations,
@parameters, etc. will be done by means of asciidoc. Anchors and
cross-references are added as well, providing hyperlinking support in
the result processed by asciidoc(1).

This support is non-conflicting and orthogonal to the patches adding
asciidoc support to the kernel-doc DocBook output [1]. Those patches
bolt on to the current document generation pipeline; there is currently
none for native asciidoc in the kernel (though ideas have been discussed
[2]). At this time, this patch should be considered a worthwhile
standalone improvement to kernel-doc, independent of the rest.

[1] http://mid.gmane.org/1448471279-19748-1-git-send-email-daniel.vetter@ffwll.ch
[2] http://mid.gmane.org/20160114131823.2ff43a0c@lwn.net

Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Daniel Vetter <daniel@ffwll.ch>
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 238 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 238 insertions(+)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 29fd5cabb657..15077f910e81 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -50,6 +50,7 @@ The documentation comments are identified by "/**" opening comment mark. See
 Documentation/kernel-doc-nano-HOWTO.txt for the documentation comment syntax.
 
 Output format selection (mutually exclusive):
+  -asciidoc		Output AsciiDoc format.
   -docbook		Output DocBook format.
   -html			Output HTML format.
   -html5		Output HTML5 format.
@@ -203,6 +204,8 @@ my $type_param = '\@(\w+)';
 my $type_struct = '\&((struct\s*)*[_\w]+)';
 my $type_struct_xml = '\\&amp;((struct\s*)*[_\w]+)';
 my $type_env = '(\$\w+)';
+my $type_enum_full = '\&(enum)\s*([_\w]+)';
+my $type_struct_full = '\&(struct)\s*([_\w]+)';
 
 # Output conversion substitutions.
 #  One for each output format
@@ -268,6 +271,17 @@ my @highlights_text = (
 		      );
 my $blankline_text = "";
 
+# asciidoc-mode
+my @highlights_asciidoc = (
+                       [$type_constant, "`\$1`"],
+                       [$type_func, "<<func_\$1,`\$1()`>>"],
+                       [$type_struct_full, "<<\$1_\$2,`\$1 \$2`>>"],
+                       [$type_enum_full, "<<\$1_\$2,`\$1 \$2`>>"],
+                       [$type_struct, "<<struct_\$1,`\$1`>>"],
+                       [$type_param, "*\$1*"]
+		      );
+my $blankline_asciidoc = "\n";
+
 # list mode
 my @highlights_list = (
                        [$type_constant, "\$1"],
@@ -404,6 +418,10 @@ while ($ARGV[0] =~ m/^-(.*)/) {
 	$output_mode = "text";
 	@highlights = @highlights_text;
 	$blankline = $blankline_text;
+    } elsif ($cmd eq "-asciidoc") {
+	$output_mode = "asciidoc";
+	@highlights = @highlights_asciidoc;
+	$blankline = $blankline_asciidoc;
     } elsif ($cmd eq "-docbook") {
 	$output_mode = "xml";
 	@highlights = @highlights_xml;
@@ -1704,6 +1722,214 @@ sub output_blockhead_text(%) {
     }
 }
 
+##
+# output in asciidoc
+sub output_highlight_asciidoc {
+    my $contents = join "\n",@_;
+    my $line;
+
+    # undo the evil effects of xml_escape() earlier
+    $contents = xml_unescape($contents);
+
+    eval $dohighlight;
+    die $@ if $@;
+
+    foreach $line (split "\n", $contents) {
+	if ($line eq "") {
+	    print $lineprefix, $blankline;
+	} else {
+	    $line =~ s/\\\\\\/\&/g;
+	    print $lineprefix, $line;
+	}
+	print "\n";
+    }
+}
+
+sub output_function_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter, $section);
+    my $start;
+
+    print "[[func_$args{'function'}]]\n";
+    print "=== " . $args{'function'} . " ===\n";
+    print $args{'purpose'} . "\n\n";
+
+    print "----------\n";
+    if ($args{'functiontype'} ne "") {
+	$start = $args{'functiontype'} . " " . $args{'function'} . " (";
+    } else {
+	$start = $args{'function'} . " (";
+    }
+    print $start;
+
+    my $count = 0;
+    foreach my $parameter (@{$args{'parameterlist'}}) {
+	$type = $args{'parametertypes'}{$parameter};
+	if ($type =~ m/([^\(]*\(\*)\s*\)\s*\(([^\)]*)\)/) {
+	    # pointer-to-function
+	    print $1 . $parameter . ") (" . $2;
+	} else {
+	    print $type . " " . $parameter;
+	}
+	if ($count != $#{$args{'parameterlist'}}) {
+	    $count++;
+	    print ",\n";
+	    print " " x length($start);
+	} else {
+	    print ");\n\n";
+	}
+    }
+    print "----------\n";
+
+    print ".Parameters\n";
+    print "[horizontal]\n";
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	my $parameter_name = $parameter;
+	$parameter_name =~ s/\[.*//;
+	$type = $args{'parametertypes'}{$parameter};
+
+	print "`$type $parameter`::\n";
+	if ($args{'parameterdescs'}{$parameter_name} ne $undescribed) {
+	    $blankline = "+";
+	    output_highlight_asciidoc($args{'parameterdescs'}{$parameter_name});
+	    $blankline = "\n";
+	} else {
+	    print "_undescribed_\n";
+	}
+	print "\n";
+    }
+    output_section_asciidoc(@_);
+}
+
+sub output_section_asciidoc(%) {
+    my %args = %{$_[0]};
+    my $section;
+
+    foreach $section (@{$args{'sectionlist'}}) {
+	print ".$section\n\n";
+	output_highlight_asciidoc($args{'sections'}{$section});
+	print "\n";
+    }
+    print "\n";
+}
+
+sub output_enum_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter);
+    my $count;
+
+    print "[[enum_$args{'enum'}]]\n";
+    print "=== enum " . $args{'enum'} . " ===\n";
+    print $args{'purpose'} . "\n\n";
+
+    print "----------\n";
+    print "enum " . $args{'enum'} . " {\n";
+    $count = 0;
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	print "\t$parameter";
+	if ($count != $#{$args{'parameterlist'}}) {
+	    $count++;
+	    print ",";
+	}
+	print "\n";
+    }
+    print "};\n";
+    print "----------\n";
+
+    print ".Constants\n";
+    print "[horizontal]\n";
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	print "`$parameter`::\n";
+	if ($args{'parameterdescs'}{$parameter} ne $undescribed) {
+	    $blankline = "+";
+	    output_highlight_asciidoc($args{'parameterdescs'}{$parameter});
+	    $blankline = "\n";
+	} else {
+	    print "_undescribed_\n";
+	}
+	print "\n";
+    }
+
+    output_section_asciidoc(@_);
+}
+
+sub output_typedef_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter);
+    my $count;
+
+    print "[[$args{'typedef'}]]\n";
+    print "=== typedef " . $args{'typedef'} . " ===\n";
+    print $args{'purpose'} . "\n\n";
+
+    output_section_asciidoc(@_);
+}
+
+sub output_struct_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter);
+
+    print "[[$args{'type'}_$args{'struct'}]]\n";
+    print "=== " . $args{'type'} . " " . $args{'struct'} . " ===\n";
+    print $args{'purpose'} . "\n\n";
+
+    print "----------\n";
+    print $args{'type'} . " " . $args{'struct'} . " {\n";
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	if ($parameter =~ /^#/) {
+	    print "$parameter\n";
+	    next;
+	}
+
+	my $parameter_name = $parameter;
+	$parameter_name =~ s/\[.*//;
+
+	($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next;
+	$type = $args{'parametertypes'}{$parameter};
+	if ($type =~ m/([^\(]*\(\*)\s*\)\s*\(([^\)]*)\)/) {
+	    # pointer-to-function
+	    print "\t$1 $parameter) ($2);\n";
+	} elsif ($type =~ m/^(.*?)\s*(:.*)/) {
+	    # bitfield
+	    print "\t$1 $parameter$2;\n";
+	} else {
+	    print "\t" . $type . " " . $parameter . ";\n";
+	}
+    }
+    print "};\n";
+    print "----------\n";
+
+    print ".Members\n";
+    print "[horizontal]\n";
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	($parameter =~ /^#/) && next;
+
+	my $parameter_name = $parameter;
+	$parameter_name =~ s/\[.*//;
+
+	($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next;
+	$type = $args{'parametertypes'}{$parameter};
+	print "`$type $parameter`" . "::\n";
+	$blankline = "+";
+	output_highlight_asciidoc($args{'parameterdescs'}{$parameter_name});
+	$blankline = "\n";
+	print "\n";
+    }
+    print "\n";
+    output_section_asciidoc(@_);
+}
+
+sub output_blockhead_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter, $section);
+
+    foreach $section (@{$args{'sectionlist'}}) {
+	print "=== $section ===\n";
+	output_highlight_asciidoc($args{'sections'}{$section});
+	print "\n";
+    }
+}
+
 ## list mode output functions
 
 sub output_function_list(%) {
@@ -2405,6 +2631,18 @@ sub xml_escape($) {
 	return $text;
 }
 
+# xml_unescape: reverse the effects of xml_escape
+sub xml_unescape($) {
+	my $text = shift;
+	if (($output_mode eq "text") || ($output_mode eq "man")) {
+		return $text;
+	}
+	$text =~ s/\\\\\\amp;/\&/g;
+	$text =~ s/\\\\\\lt;/</g;
+	$text =~ s/\\\\\\gt;/>/g;
+	return $text;
+}
+
 # convert local escape strings to html
 # local escape strings look like:  '\\\\menmonic:' (that's 4 backslashes)
 sub local_unescape($) {
-- 
2.1.4

[RFC 03/10] kernel-doc: support printing exported and non-exported symbols

[Jani Nikula]

Currently we use docproc to figure out which symbols are exported, and
then docproc calls kernel-doc on specific functions, to get
documentation on exported functions. According to git blame and docproc
comments, this is due to historical reasons, as functions and their
corresponding EXPORT_SYMBOL* may have been in different files. However
for more than ten years the recommendation in CodingStyle has been to
place the EXPORT_SYMBOL* immediately after the closing function brace
line.

Additionally, the kernel-doc comments for functions are generally placed
above the function definition in the .c files (i.e. where the
EXPORT_SYMBOL* is) rather than above the declaration in the .h
files. There are some exceptions to this, but AFAICT none of these are
included in DocBook documentation using the "!E" docproc directive.

Therefore, assuming the EXPORT_SYMBOL* and kernel-doc are with the
function definition, kernel-doc can extract the exported vs. not
information by making two passes on the input file. Add support for that
via the new -export and -internal parameters.

[This should cover most of the cases. If it's not enough, it's trivial
to allow specifying additional files to look for EXPORT_SYMBOL* in.]

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 29 +++++++++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 15077f910e81..ee2ac9137a43 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -59,6 +59,12 @@ Output format selection (mutually exclusive):
   -text			Output plain text format.
 
 Output selection (mutually exclusive):
+  -export		Only output documentation for symbols that have been
+			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
+			in the same FILE.
+  -internal		Only output documentation for symbols that have NOT been
+			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
+			in the same FILE.
   -function NAME	Only output documentation for the given function(s)
 			or DOC: section title(s). All other functions and DOC:
 			sections are ignored. May be specified multiple times.
@@ -380,6 +386,7 @@ my $doc_block = $doc_com . 'DOC:\s*(.*)?';
 my $doc_split_start = '^\s*/\*\*\s*$';
 my $doc_split_sect = '\s*\*\s*(@[\w\s]+):(.*)';
 my $doc_split_end = '^\s*\*/\s*$';
+my $export_symbol = '^\s*EXPORT_SYMBOL(_GPL)?\s*\(\s*(\w+)\s*\)\s*;';
 
 my %constants;
 my %parameterdescs;
@@ -444,6 +451,12 @@ while ($ARGV[0] =~ m/^-(.*)/) {
 	$function_only = 2;
 	$function = shift @ARGV;
 	$function_table{$function} = 1;
+    } elsif ($cmd eq "-export") { # only exported symbols
+	$function_only = 3;
+	%function_table = ()
+    } elsif ($cmd eq "-internal") { # only non-exported symbols
+	$function_only = 4;
+	%function_table = ()
     } elsif ($cmd eq "-v") {
 	$verbose = 1;
     } elsif (($cmd eq "-h") || ($cmd eq "--help")) {
@@ -1976,8 +1989,10 @@ sub output_declaration {
     my $functype = shift;
     my $func = "output_${functype}_$output_mode";
     if (($function_only==0) ||
-	( $function_only == 1 && defined($function_table{$name})) ||
-	( $function_only == 2 && !($functype eq "function" && defined($function_table{$name}))))
+	( ($function_only == 1 || $function_only == 3) &&
+	  defined($function_table{$name})) ||
+	( ($function_only == 2 || $function_only == 4) &&
+	  !($functype eq "function" && defined($function_table{$name}))))
     {
 	&$func(@_);
 	$section_counter++;
@@ -2682,6 +2697,16 @@ sub process_file($) {
 
     $. = 1;
 
+    # two passes for -export and -internal
+    if ($function_only == 3 || $function_only == 4) {
+	while (<IN>) {
+	    if (/$export_symbol/o) {
+		$function_table{$2} = 1;
+	    }
+	}
+	seek(IN, 0, 0);
+    }
+
     $section_counter = 0;
     while (<IN>) {
 	while (s/\\\s*$//) {
-- 
2.1.4

[RFC 04/10] kernel-doc: add support for printing DOC: comments with escaped names

[Jani Nikula]

-function supports printing named DOC: sections, but spaces and braces
and quotes etc. are allowed in section titles. This is tricky to handle
in scripts, let alone Makefiles.

Add a new -doc parameter for dumping doc sections (to not convolute
-function more than it already is), with support for "escaped" names
with everything non-alphanumeric repaced with underscores, in addition
to verbatim names.

For example, all these three now do the same thing:

$ scripts/kernel-doc -function "Panel Self Refresh (PSR/SRD)" drivers/gpu/drm/i915/intel_psr.c
$ scripts/kernel-doc -doc "Panel Self Refresh (PSR/SRD)" drivers/gpu/drm/i915/intel_psr.c
$ scripts/kernel-doc -doc "Panel_Self_Refresh__PSR_SRD_" drivers/gpu/drm/i915/intel_psr.c

Use of -function for extracting DOC: sections should probably be
deprecated, but keep it around for backward compatibility.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index ee2ac9137a43..0e410daa92a9 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -71,6 +71,9 @@ Output selection (mutually exclusive):
   -nofunction NAME	Do NOT output documentation for the given function(s);
 			only output documentation for the other functions and
 			DOC: sections. May be specified multiple times.
+  -doc NAME		Only output documentation for the given DOC: section
+			titles. NAME is matched both as-is and with all
+			non-alphanumeric characters replaced with underscores.
 
 Output selection modifiers:
   -no-doc-sections	Do not output DOC: sections.
@@ -457,6 +460,10 @@ while ($ARGV[0] =~ m/^-(.*)/) {
     } elsif ($cmd eq "-internal") { # only non-exported symbols
 	$function_only = 4;
 	%function_table = ()
+    } elsif ($cmd eq "-doc") { # to only output specific doc sections
+	$function_only = 5;
+	$function = shift @ARGV;
+	$function_table{$function} = 1;
     } elsif ($cmd eq "-v") {
 	$verbose = 1;
     } elsif (($cmd eq "-h") || ($cmd eq "--help")) {
@@ -520,14 +527,20 @@ sub dump_doc_section {
     my $file = shift;
     my $name = shift;
     my $contents = join "\n", @_;
+    my $escaped_name = $name;
 
     if ($no_doc_sections) {
         return;
     }
 
+    $escaped_name =~ s/[^a-zA-Z0-9]/_/g;
+
     if (($function_only == 0) ||
 	( $function_only == 1 && defined($function_table{$name})) ||
-	( $function_only == 2 && !defined($function_table{$name})))
+	( $function_only == 2 && !defined($function_table{$name})) ||
+	( $function_only == 5 &&
+	  (defined($function_table{$name}) ||
+	   defined($function_table{$escaped_name}))))
     {
 	dump_section($file, $name, $contents);
 	output_blockhead({'sectionlist' => \@sectionlist,
-- 
2.1.4

[RFC 05/10] scripts: add asciidoc-includes to extract includes from asciidoc

[Jani Nikula]

Given an asciidoc input file in standard input, print the files included
using asciidoc include macros to standard output, one per line. This
will be helpful in asciidoc dependency generation.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/asciidoc-includes | 6 ++++++
 1 file changed, 6 insertions(+)
 create mode 100755 scripts/asciidoc-includes

diff --git a/scripts/asciidoc-includes b/scripts/asciidoc-includes
new file mode 100755
index 000000000000..d832035275eb
--- /dev/null
+++ b/scripts/asciidoc-includes
@@ -0,0 +1,6 @@
+#!/bin/sh
+# Given an asciidoc input file in standard input, print the files included using
+# asciidoc include macros to standard output, one per line.
+
+INCLUDE="^include::\([^[]*\)\[.*\]"
+exec sed "/${INCLUDE}/!d; s/${INCLUDE}/\1/"
-- 
2.1.4

[RFC 06/10] scripts: add a kernel-doc helper for special invocation

[Jani Nikula]

Add a helper for calling kernel-doc, with the -w parameter describing
the parameters to pass on to kernel-doc, in a special format.

[This is easier to explain and will make more sense in the context of
the following patches. The -w parameter is expected to be a filename
with the kernel-doc parameters encoded at the end of the filename. This
could be included in kernel-doc itself, but it feels like polluting the
tool. So here's a helper, at least for now.]

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc-helper | 70 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)
 create mode 100755 scripts/kernel-doc-helper

diff --git a/scripts/kernel-doc-helper b/scripts/kernel-doc-helper
new file mode 100755
index 000000000000..1e26266f9feb
--- /dev/null
+++ b/scripts/kernel-doc-helper
@@ -0,0 +1,70 @@
+#!/bin/bash
+
+KERNELDOC=kernel-doc
+OUTPUT_FORMAT=-asciidoc
+
+WHAT=
+
+usage()
+{
+	cat <<EOF
+$ kernel-doc-helper [-k KERNELDOC] -w WHAT SOURCE
+
+Run KERNELDOC (kernel-doc by default) on SOURCE to extract information specified
+by WHAT.
+
+WHAT is an arbitrary string ending in special suffixes that are converted to
+kernel-doc parameters.
+EOF
+	exit 0
+}
+
+while getopts "hw:k:" opt; do
+	case "$opt" in
+		h)
+			usage
+			;;
+		w)
+			WHAT="$OPTARG"
+			;;
+		k)
+			KERNELDOC="$OPTARG"
+			;;
+		*)
+			exit 1
+	esac
+done
+shift `expr $OPTIND - 1`
+
+if [ "$#" = "0" ]; then
+	echo "$0: source file missing" >&2
+	exit 1
+fi
+
+if [ -z "$WHAT" ]; then
+	echo "$0: output selection (-w) missing" >&2
+	exit 1
+fi
+
+SOURCE=$1
+
+case $WHAT in
+	*,export)
+		exec $KERNELDOC $OUTPUT_FORMAT -export $SOURCE
+		;;
+	*,internal)
+		exec $KERNELDOC $OUTPUT_FORMAT -internal $SOURCE
+		;;
+	*,function,*)
+		name=${WHAT##*,function,}
+		exec $KERNELDOC $OUTPUT_FORMAT -function $name $SOURCE
+		;;
+	*,doc,*)
+		name=${WHAT##*,doc,}
+		exec $KERNELDOC $OUTPUT_FORMAT -doc $name $SOURCE
+		;;
+	*)
+		echo "$0: unknown output selection '$WHAT'" >&2
+		exit 1
+		;;
+esac
-- 
2.1.4

[RFC 07/10] scripts: add tool for generating asciidoc dependencies and rules

[Jani Nikula]

Given a list of special format filenames, generate dependencies and
rules for kernel-doc invocation on those files, to be included in to
Makefiles.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc-deps | 66 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)
 create mode 100755 scripts/kernel-doc-deps

diff --git a/scripts/kernel-doc-deps b/scripts/kernel-doc-deps
new file mode 100755
index 000000000000..c348863b9354
--- /dev/null
+++ b/scripts/kernel-doc-deps
@@ -0,0 +1,66 @@
+#!/bin/bash
+
+SRCTREE=
+DSTTREE=
+DEP=
+KERNELDOC=
+TOOL=kernel-doc-helper
+
+while getopts "k:t:s:d:m:" opt; do
+	case "$opt" in
+		k)
+			KERNELDOC="$OPTARG"
+			;;
+		t)
+			TOOL="$OPTARG"
+			;;
+		s)
+			SRCTREE="$OPTARG/"
+			;;
+		d)
+			DSTTREE="$OPTARG/"
+			;;
+		m)
+			DEP="$OPTARG"
+			;;
+		*)
+			echo "fail"
+			exit 1
+	esac
+done
+shift `expr $OPTIND - 1`
+
+ACTION="\tmkdir \$()\n\t${TOOL} -w \$@ \$^ > \$@"
+
+# $1 = filename
+# $2 = for what type
+print_rule()
+{
+	echo "$1: $2"
+	echo -e "\tmkdir -p \$(dir \$@)"
+	echo -e "\t${TOOL} -k ${KERNELDOC} -w \$@ \$^ > \$@"
+	echo "$DEP: $1"
+}
+
+# given a list of filenames with special notation, generate deps
+
+while read file; do
+	case $file in
+		# all of these are C/H source files
+		*,export)
+			print_rule "${DSTTREE}$file" "${SRCTREE}${file%,export}"
+			;;
+		*,internal)
+			print_rule "${DSTTREE}$file" "${SRCTREE}${file%,internal}"
+			;;
+		*,function,*)
+			print_rule "${DSTTREE}$file" "${SRCTREE}${file%,function,*}"
+			;;
+		*,doc,*)
+			print_rule "${DSTTREE}$file" "${SRCTREE}${file%,doc,*}"
+			;;
+		*)
+			print_rule "${DSTTREE}/${file%,txt,} " "${SRCTREE}${file%,F,*}"
+			;;
+	esac
+done
-- 
2.1.4

[RFC 08/10] scripts: add a crude converter from DocBook tmpl to asciidoc

[Jani Nikula]

Use some pre- and post-processing to handle the "![EIFPCD]" docproc
directives in the DocBook, and let pandoc do the rest. Manual editing
will be required, but this will give a big jump start.

The asciidoc result would be nicer without the pandoc --no-wrap option,
but unfortunately pandoc also wraps the docproc directives within
<pre></pre> tags, breaking their post-processing. There's probably a way
around this, but I couldn't be bothered for this proof of concept.

The post-processing converts the directives to the new format of having
comma separated filename suffixes describe the content to be included.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/tmpl2asciidoc | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
 create mode 100755 scripts/tmpl2asciidoc

diff --git a/scripts/tmpl2asciidoc b/scripts/tmpl2asciidoc
new file mode 100755
index 000000000000..092400cc702c
--- /dev/null
+++ b/scripts/tmpl2asciidoc
@@ -0,0 +1,39 @@
+#!/bin/bash
+# a crude converter from DocBook tmpl in STDIN to AsciiDoc in STDOUT
+
+sed 's/^\(!.*\)/<pre>\1<\/pre>/' |\
+	pandoc --atx-headers --no-wrap -f docbook -t asciidoc |\
+	while read line; do
+		case "$line" in
+			!E*)
+				file=${line#!?}
+				echo "include::$file,export[]"
+				;;
+			!I*)
+				file=${line#!?}
+				echo "include::$file,internal[]"
+				;;
+			!F*)
+				file=${line#!?}
+				file=${file%% *}
+				functions=${line#* }
+				for f in $functions; do
+					echo "include::$file,function,$f[]"
+				done
+				;;
+			!P*)
+				file=${line#!?}
+				file=${file%% *}
+				doc=${line#* }
+				doc=${doc//[^A-Za-z0-9]/_}
+				echo "include::$file,doc,$doc[]"
+				;;
+			!C*|!D*)
+				echo "$0: WARNING: unsupported: $line" >&2
+				echo "// $line"
+				;;
+			*)
+				echo -E "$line"
+				;;
+		esac
+	done
-- 
2.1.4

[RFC 09/10] Documentation: convert gpu.tmpl to gpu.txt

[Jani Nikula]

This is the verbatim result of

$ scripts/tmpl2asciidoc < Documentation/DocBook/gpu.tmpl > Documentation/DocBook/gpu.txt
$ rm Documentation/DocBook/gpu.tmpl

It should probably be split up to separate documents for GPU drivers
etc. using include macros within asciidoc, but this is for demonstration
purposes.

Also, the pandoc --no-wrap is ugly, as explained in the commit message
for the conversion tool.

The output is worth looking at for the new style includes for kernel-doc
comments, for example:

include::drivers/gpu/drm/drm_irq.c,export[]
include::include/drm/drmP.h,function,drm_crtc_vblank_waitqueue[]

Signed-off-by: Jani Nikula <jani.nikula@intel.com>


*ATTENTION*: The patch was big and contained long lines which git send-email
refused to mail (damn --no-wrap again). I just dropped it out and left this as
placeholder. The commit is at:

http://cgit.freedesktop.org/~jani/drm/commit/?h=kernel-asciidoc&id=aa4b1d229401e83c98a38c2b2fe1d9f3012bb5b5

---
 Documentation/DocBook/gpu.tmpl | 3499 ----------------------------------------
 Documentation/DocBook/gpu.txt  | 1183 ++++++++++++++
 2 files changed, 1183 insertions(+), 3499 deletions(-)
 delete mode 100644 Documentation/DocBook/gpu.tmpl
 create mode 100644 Documentation/DocBook/gpu.txt

[RFC 10/10] Documentation: build asciidoc documentation

[Jani Nikula]

Finally wrap up all the work in the preceding patches, and generate a
pipeline from asciidoc documentation as part of the DocBook
documentation.

The biggest difference in building DocBook xml from asciidoc compared to
the docproc pipeline is the use of intermediate asciidoc snippets for
the kernel-doc generated documentation, their inclusion into the source
asciidoc using the markup's own directives, with the dependencies set
right.

This is a big improvement in the build process:

First, kernel-doc is only called on the source files that have actually
changed. (Also, asciidoc is only called on the high level documents
where the included documentation has actually changed.)

Second, all of the kernel-doc processing parallelizes.

The xmlto proessing of the result is still slow, but the build time
improvement for the kernel-doc/asciidoc part is huge.

It's not yet all rosy though:

- The intermediate files should be put in a separate subdirectory
  instead of DocBook where it pollutes everything. Sorry, this is rather
  ugly now.

- The clean targets don't work properly.

- Some of the choices made in kernel-doc asciidoc support looked
  beautiful with the default asciidoc html output; however the DocBook
  pipeline is a different beast with different templates, and it's not
  so pretty anymore. (Particularly the numbered "tables" for parameters
  are ugly.)

- The cross-referencing probably needs more looking into. I think I've
  seen some weird stuff there. Also, since the kernel-doc asciidoc turns
  everything into xrefs, the later cross-referencing stage complains
  about dead links.

NOTE: When switching between this and master, be sure to clean the
Documentation/DocBook/*.cmd files. They don't seem to depend right, and
wasted me a few hours debugging my makefiles.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/DocBook/Makefile | 37 +++++++++++++++++++++++++++++--------
 Documentation/DocBook/gpu.txt  |  2 +-
 2 files changed, 30 insertions(+), 9 deletions(-)

diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile
index d70f9b68174e..aed990080aba 100644
--- a/Documentation/DocBook/Makefile
+++ b/Documentation/DocBook/Makefile
@@ -4,7 +4,9 @@
 # See Documentation/kernel-doc-nano-HOWTO.txt for instruction in how
 # to document the SRC - and how to read it.
 # To add a new book the only step required is to add the book to the
-# list of DOCBOOKS.
+# list of DOCBOOKS or ASCIIDOC_BOOKS.
+
+ASCIIDOC_BOOKS := gpu.txt
 
 DOCBOOKS := z8530book.xml device-drivers.xml \
 	    kernel-hacking.xml kernel-locking.xml deviceiobook.xml \
@@ -14,18 +16,19 @@ DOCBOOKS := z8530book.xml device-drivers.xml \
 	    genericirq.xml s390-drivers.xml uio-howto.xml scsi.xml \
 	    80211.xml debugobjects.xml sh.xml regulator.xml \
 	    alsa-driver-api.xml writing-an-alsa-driver.xml \
-	    tracepoint.xml gpu.xml media_api.xml w1.xml \
-	    writing_musb_glue_layer.xml crypto-API.xml iio.xml
+	    tracepoint.xml media_api.xml w1.xml \
+	    writing_musb_glue_layer.xml crypto-API.xml iio.xml \
+	    $(ASCIIDOC_BOOKS:%.txt=%.xml)
 
 include Documentation/DocBook/media/Makefile
 
 ###
 # The build process is as follows (targets):
-#              (xmldocs) [by docproc]
-# file.tmpl --> file.xml +--> file.ps   (psdocs)   [by db2ps or xmlto]
-#                        +--> file.pdf  (pdfdocs)  [by db2pdf or xmlto]
-#                        +--> DIR=file  (htmldocs) [by xmlto]
-#                        +--> man/      (mandocs)  [by xmlto]
+#                (xmldocs) [by docproc or asciidoc]
+# file.tmpl -+               +--> file.ps   (psdocs)   [by db2ps or xmlto]
+#            +--> file.xml --+--> file.pdf  (pdfdocs)  [by db2pdf or xmlto]
+# file.txt  -+               +--> DIR=file  (htmldocs) [by xmlto]
+#                            +--> man/      (mandocs)  [by xmlto]
 
 
 # for PDF and PS output you can choose between xmlto and docbook-utils tools
@@ -69,6 +72,9 @@ KERNELDOCXMLREF = $(srctree)/scripts/kernel-doc-xml-ref
 KERNELDOC       = $(srctree)/scripts/kernel-doc
 DOCPROC         = $(objtree)/scripts/docproc
 CHECK_LC_CTYPE = $(objtree)/scripts/check-lc_ctype
+ASCIIDOC_INCLUDES	= $(srctree)/scripts/asciidoc-includes
+KERNELDOC_DEPS		= $(srctree)/scripts/kernel-doc-deps
+KERNELDOC_HELPER	= $(srctree)/scripts/kernel-doc-helper
 
 # Use a fixed encoding - UTF-8 if the C library has support built-in
 # or ASCII if not
@@ -79,6 +85,21 @@ XMLTOFLAGS = -m $(srctree)/$(src)/stylesheet.xsl
 XMLTOFLAGS += --skip-validation
 
 ###
+# Docbook pipeline for asciidoc and asciidoctor (both should work).
+%.xml: %.txt
+	asciidoctor -b docbook -o $@ $<
+
+# Generate dependencies and rules for building intermediate asciidoc files
+# included from the asciidoc high level documents. The intermediate files are
+# generated from kernel sources using kernel-doc, according to the file names
+# and suffixes in the asciidoc high level documents.
+%.deps: %.txt
+	$(ASCIIDOC_INCLUDES) < $< | $(KERNELDOC_DEPS) -k $(KERNELDOC) -t $(KERNELDOC_HELPER) -s $(srctree) -d $(obj) -m $(@:%.deps=%.xml) > $@
+
+# Include (and generate as necessary) the dependencies.
+-include $(addprefix $(obj)/,$(ASCIIDOC_BOOKS:%.txt=%.deps))
+
+###
 # DOCPROC is used for two purposes:
 # 1) To generate a dependency list for a .tmpl file
 # 2) To preprocess a .tmpl file and call kernel-doc with
diff --git a/Documentation/DocBook/gpu.txt b/Documentation/DocBook/gpu.txt
index 71aa1473461e..9ba9247d1c43 100644
--- a/Documentation/DocBook/gpu.txt
+++ b/Documentation/DocBook/gpu.txt
@@ -6,7 +6,7 @@ The Linux DRM layer contains code intended to support the needs of complex graph
 
 A note on versions: this guide covers features found in the DRM tree, including the TTM memory manager, output configuration and mode setting, and the new vblank internals, in addition to all the regular features found in current kernels.
 
-[Insert diagram of typical DRM stack here]
+// Insert diagram of typical DRM stack here
 
 == Style Guidelines
 
-- 
2.1.4

Re: [RFC] A first shot at asciidoc-based formatted docs

[Daniel Vetter]

On Tue, Jan 26, 2016 at 1:08 PM, Jani Nikula <jani.nikula@intel.com> wrote:
> On Tue, 26 Jan 2016, Jonathan Corbet <corbet@lwn.net> wrote:
>> So here is a proof-of-concept series showing how a fully asciidoc-based
>> toolchain might work.  Lots of hackery here, this isn't meant to be applied
>> to anything at this point, but it's a good start.  What this series has is:
>>
>>  - Jani Nikula's patch adding asciidoc output to kernel-doc.  Thanks for
>>    doing this!  It was the kickstart that was needed to get this process
>>    going.
>
> Hooray! :)
>
>>  - Tweak docproc to handle asciidoc template files.  If a template ends in
>>    ".adt", it's an asciidoc template; it's processed pretty much the same
>>    way, except that kernel-doc gets the -asciidoc argument.
>>
>>  - Bash on the Makefile to get it to process asciidoc templates into HTML.
>>    Naturally this was where most of the time got spent.  *Only* HTML output
>>    works at the moment.
>>
>>  - Convert tracepoints.html to tpoint.adt as a proof of concept.  It works,
>>    and the output is much pleasing, IMO.
>>
>> I'm sure there's a thousand details to deal with, and there is the issue of
>> the other output formats.  asciidoctor claims to be able to create man
>> pages, but I've not tried that yet; neither tool will do PDF.  Maybe we
>> could rely on pandoc to do that.  Otherwise, getting to asciidoc to XML is
>> straightforward, so it should be possible to use xmlto as is done now.
>>
>> It's all in the doc/asciidoc branch of git://git.lwn.net/linux.git if
>> anybody wants to mess with it.
>>
>> Comments?
>
> I'm afraid we've done some overlapping work in the mean time, but I'm
> happy we've both looked at the tool chain, and can have a more
> meaningful conversation now.
>
> I first took roughly the same approach as you did. I was really
> impressed with the speed and the beauty of the produced HTML. The
> trouble is, neither asciidoc nor asciidoctor can produce chunked (split
> to several pages) HTML directly. This is a showstopper for the gpu
> document which turns into 1.3 MB of HTML, which looks pretty but is a
> paint to navigate. To do chunked output, you have to output DocBook and
> handle that like we do now. So while I would like to have asciidoc
> generate HTML directly for speed and beauty, I ended up going the
> asciidoc to DocBook path. The upside is all the output formats are
> supported.

This is a big bummer since with the parralized kernel-doc processing
using Makefiles and using asciidoctor even building something big like
the gpu docs is down to 2-3 seconds. From a clean tree, so not even
counting incremental speed-ups. Unfortunately asciidoc doesn't have an
built-in tooling (there's some experimental extensions) to split
things up.
-Daniel

> (We could of course split the source documents, but then I believe we'd
> have lots of trouble cross-referencing between the documents. I could be
> proven wrong. I'd *like* to be proven wrong.)
>
> One significant difference between our approaches is that I ditched
> docproc out of the equation. Instead of having the docproc ! directives
> in the asciidoc, I opted for using asciidoc's native include macro, with
> specially crafted filename suffixes to specify what parts of the source
> to include. Those files are then generated as intermediate asciidoc
> files using kernel-doc, with dependencies set right. There's a bunch of
> scripting involved, but it's pretty straight forward.
>
> So you'd have, for example,
>
> include::drivers/gpu/drm/drm_ioctl.c,export[]
>
> to include all the exported functions, or
>
> include::drivers/gpu/drm/i915/i915_irq.c,doc,interrupt_handling[]
>
> to include the DOC: section. I think I'd like some version of this,
> regardless of whether asciidoc generates HTML or DocBook. It lets make
> parallelize all of kernel-doc processing for free, which is a big win.
>
> Patches follow, also available in kernel-asciidoc branch of
> git://people.freedesktop.org/~jani/drm (web interface
> http://cgit.freedesktop.org/~jani/drm/log/?h=kernel-asciidoc)
>
> BR,
> Jani.
>
>
> Jani Nikula (10):
>   kernel-doc: rewrite usage description, remove duplicated comments
>   kernel-doc: add support for asciidoc output
>   kernel-doc: support printing exported and non-exported symbols
>   kernel-doc: add support for printing DOC: comments with escaped names
>   scripts: add asciidoc-includes to extract includes from asciidoc
>   scripts: add a kernel-doc helper for special invocation
>   scripts: add tool for generating asciidoc dependencies and rules
>   scripts: add a crude converter from DocBook tmpl to asciidoc
>   Documentation: convert gpu.tmpl to gpu.txt
>   Documentation: build asciidoc documentation
>
>  Documentation/DocBook/Makefile |   37 +-
>  Documentation/DocBook/gpu.tmpl | 3499 ----------------------------------------
>  Documentation/DocBook/gpu.txt  | 1183 ++++++++++++++
>  scripts/asciidoc-includes      |    6 +
>  scripts/kernel-doc             |  365 ++++-
>  scripts/kernel-doc-deps        |   66 +
>  scripts/kernel-doc-helper      |   70 +
>  scripts/tmpl2asciidoc          |   39 +
>  8 files changed, 1709 insertions(+), 3556 deletions(-)
>  delete mode 100644 Documentation/DocBook/gpu.tmpl
>  create mode 100644 Documentation/DocBook/gpu.txt
>  create mode 100755 scripts/asciidoc-includes
>  create mode 100755 scripts/kernel-doc-deps
>  create mode 100755 scripts/kernel-doc-helper
>  create mode 100755 scripts/tmpl2asciidoc
>
> --
> 2.1.4
>



-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jani Nikula]

On Tue, 26 Jan 2016, Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
> On Tue, Jan 26, 2016 at 1:08 PM, Jani Nikula <jani.nikula@intel.com> wrote:
>> I first took roughly the same approach as you did. I was really
>> impressed with the speed and the beauty of the produced HTML. The
>> trouble is, neither asciidoc nor asciidoctor can produce chunked (split
>> to several pages) HTML directly. This is a showstopper for the gpu
>> document which turns into 1.3 MB of HTML, which looks pretty but is a
>> paint to navigate. To do chunked output, you have to output DocBook and
>> handle that like we do now. So while I would like to have asciidoc
>> generate HTML directly for speed and beauty, I ended up going the
>> asciidoc to DocBook path. The upside is all the output formats are
>> supported.
>
> This is a big bummer since with the parralized kernel-doc processing
> using Makefiles and using asciidoctor even building something big like
> the gpu docs is down to 2-3 seconds. From a clean tree, so not even
> counting incremental speed-ups. Unfortunately asciidoc doesn't have an
> built-in tooling (there's some experimental extensions) to split
> things up.

Basically asciidoc -> HTML is about as fast as asciidoc -> XML, and with
parallel kernel-doc it really is fast. Sadly the XML -> HTML part still
takes forever.

I just want to emphasize that we can get parallel kernel-doc with either
pipeline. It is also possible to enable both pipelines, i.e. have a fast
path HTML generation with few external dependencies and the Swiss army
knife slow path with XML.

I should also remind us that the original goal was to enable lightweight
markup for documentation. This seems very much achievable now. We don't
have to solve all the existing problems with the XML pipeline right
now. And asciidoc suits this well, as it can also feed to the existing
pipeline.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jonathan Corbet]

On Tue, 26 Jan 2016 14:08:45 +0200
Jani Nikula <jani.nikula@intel.com> wrote:

> I'm afraid we've done some overlapping work in the mean time, but I'm
> happy we've both looked at the tool chain, and can have a more
> meaningful conversation now.

Overlapping work is just how this kernel thing works :)

> I first took roughly the same approach as you did. I was really
> impressed with the speed and the beauty of the produced HTML. The
> trouble is, neither asciidoc nor asciidoctor can produce chunked (split
> to several pages) HTML directly. This is a showstopper for the gpu
> document which turns into 1.3 MB of HTML, which looks pretty but is a
> paint to navigate. To do chunked output, you have to output DocBook and
> handle that like we do now. So while I would like to have asciidoc
> generate HTML directly for speed and beauty, I ended up going the
> asciidoc to DocBook path. The upside is all the output formats are
> supported.

So I'm not really going to have time to dig too much more into this until
after LCA, so these are really quick impressions for now.

I would *really* like to get the XML step out of the processing path if
possible.  It adds complexity, makes it harder for others to build the
docs, makes things more fragile, and slows it all down.  It seems to me
that it should be possible to do that.

The issues, it seems, are splitting the output files and format support.
The latter isn't really an issue, I don't think; there are tools to do all
kinds of format conversions.  The only one that's even slightly weird is
man, and kernel-doc already has some (unused, I think) provisions for
doing that.  We could generate man pages directly without much pain.

For HTML-page splitting, we can see if the tools can help us, consider
splitting the template files, or do the splitting in a postprocessing
step.  Docproc (or whatever replaces it) could also maybe do that work.
It doesn't seem to me something that should force the inclusion of an
entire XML-based processing step.  



But then, I'm here spouting ideas without any proof to back them up again,
so who knows...:)

Thanks for doing this work.  As you said, it has been demonstrated 

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jonathan Corbet]

On Tue, 26 Jan 2016 07:48:21 -0700
Jonathan Corbet <corbet@lwn.net> wrote:

> For HTML-page splitting, we can see if the tools can help us, consider
> splitting the template files, or do the splitting in a postprocessing
> step.  Docproc (or whatever replaces it) could also maybe do that work.
> It doesn't seem to me something that should force the inclusion of an
> entire XML-based processing step.  
> 
> 
> 
> But then, I'm here spouting ideas without any proof to back them up again,
> so who knows...:)
> 
> Thanks for doing this work.  As you said, it has been demonstrated 

So let's try this again without hitting that "send" button prematurely,
sigh.  Claws is annoying sometimes.

You'd said:

> (We could of course split the source documents, but then I believe we'd
> have lots of trouble cross-referencing between the documents. I could be
> proven wrong. I'd *like* to be proven wrong.)

I'd like to solve the cross-reference problem anyway; the documents
shouldn't be silos unto themselves.  Sphinx and ReST can do that; I *bet*
the other formats can too.  Maybe I can dig into some of that in my
upcoming airplane time (LAX->MEL, 15:45, sigh).

> I should also remind us that the original goal was to enable lightweight
> markup for documentation. This seems very much achievable now.

Agreed, nice work.

jon

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jonathan Corbet]

On Tue, 26 Jan 2016 14:08:45 +0200
Jani Nikula <jani.nikula@intel.com> wrote:

> I'm afraid we've done some overlapping work in the mean time, but I'm
> happy we've both looked at the tool chain, and can have a more
> meaningful conversation now.

[Adding Keith since you said you wanted to be a part of this - let us know
when you've had enough!]

So I've spent a bit of time looking at this, and quite a bit more time
talking with various folks at LCA.  There is pretty much universal
agreement that this is interesting work and the direction we'd like to
go.  My current hope is that we can merge some version of it for 4.6 and
see where it goes from there.

So naturally I have some thoughts on the whole thing...

 - I would like to format directly to HTML if at all possible.  It seems
   it should be possible to get a table of contents into the files, and
   the feedback I got was that a TOC would be enough for navigation - it
   would not be necessary to split the files at that point.  We might
   still want to try to figure that out too, though.  In any case, this
   isn't a show stopper, in that we can change it anytime if a better way
   shows up.  But I'd like to have it in mind.

 - Asciidoc templates and processing should happen in a new directory
   (perhaps imaginatively called "asciidoc"); having them in a directory
   called "DocBook" seems a little weird.  More importantly, though, I'd
   like to separate them out as a fresh start, and not mess with the
   existing DocBook templates until we decide we don't need them anymore.
   If we could end up with a cleaner, simpler makefile in the process,
   that would be a bonus.

 - I'm not sold on the new inclusion mechanism.  Creating thousands of
   little files and tracking them for dependencies and such doesn't seem
   like a simplification or a path toward better performance.  I would
   like to at least consider keeping the direct-from-source inclusion.  

 - Insisting on EXPORT_SYMBOL being in the same file doesn't seem like
   it's going to work for now; that could maybe change after Al's work
   goes in, which could be fairly soon.

Please let me know your thoughts on the above.  Do you think you can find
some time over the next month for this?  I'll try to shake loose some time
too, but, well, $EXCUSES...

Many thanks for doing this work!

jon

Re: [RFC] A first shot at asciidoc-based formatted docs

[Daniel Vetter]

On Wed, Feb 10, 2016 at 1:09 AM, Jonathan Corbet <corbet@lwn.net> wrote:
> On Tue, 26 Jan 2016 14:08:45 +0200
> Jani Nikula <jani.nikula@intel.com> wrote:
>
>> I'm afraid we've done some overlapping work in the mean time, but I'm
>> happy we've both looked at the tool chain, and can have a more
>> meaningful conversation now.
>
> [Adding Keith since you said you wanted to be a part of this - let us know
> when you've had enough!]
>
> So I've spent a bit of time looking at this, and quite a bit more time
> talking with various folks at LCA.  There is pretty much universal
> agreement that this is interesting work and the direction we'd like to
> go.  My current hope is that we can merge some version of it for 4.6 and
> see where it goes from there.
>
> So naturally I have some thoughts on the whole thing...
>
>  - I would like to format directly to HTML if at all possible.  It seems
>    it should be possible to get a table of contents into the files, and
>    the feedback I got was that a TOC would be enough for navigation - it
>    would not be necessary to split the files at that point.  We might
>    still want to try to figure that out too, though.  In any case, this
>    isn't a show stopper, in that we can change it anytime if a better way
>    shows up.  But I'd like to have it in mind.

I think for 4.6 it'd be best to go with the hybrid asciidoc->docbook
toolchain, since that's less disruptive. And with that we can also
fully concentrating on the frontend, and how it'll look and behave.

Once that's solid we can look into the icing on the cake for later
kernels I think.

>  - Asciidoc templates and processing should happen in a new directory
>    (perhaps imaginatively called "asciidoc"); having them in a directory
>    called "DocBook" seems a little weird.  More importantly, though, I'd
>    like to separate them out as a fresh start, and not mess with the
>    existing DocBook templates until we decide we don't need them anymore.
>    If we could end up with a cleaner, simpler makefile in the process,
>    that would be a bonus.

For the long term dream plan of including other .txt files from the
existing pile of unstructured docs, do we really want a separate
asciidoc directory? Or just .asciidoc as a special extension?

>  - I'm not sold on the new inclusion mechanism.  Creating thousands of
>    little files and tracking them for dependencies and such doesn't seem
>    like a simplification or a path toward better performance.  I would
>    like to at least consider keeping the direct-from-source inclusion.

The motivation behind the new inclusion mechanism isn't the speed-up
due to parallelization, but being able to use native asciidoc
includes. With those you can pass options to e.g. shift the hierarchy.
With that you can do subheadings in DOC: sections and then seamlessly
include them. Or similar stuff.

The speed-up due to parallelization is just a small bonus.

Also generating thousands of files is totally not unheard of in the kernel:

$ find include/config | wc -l
2623

None of those are in git.

>  - Insisting on EXPORT_SYMBOL being in the same file doesn't seem like
>    it's going to work for now; that could maybe change after Al's work
>    goes in, which could be fairly soon.

Hm, assuming Al gets his stuff into 4.6 could we just assume this? It
holds true for gpu docs already I think, and most other subsystems.
The trouble iirc is all around asm and similar stuff, and we can't
kerneldoc asm afaik.

> Please let me know your thoughts on the above.  Do you think you can find
> some time over the next month for this?  I'll try to shake loose some time
> too, but, well, $EXCUSES...

One more thing we discussed: Did you ping kbuild folks already? Or
want to get some agreement on the overall build process first?

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jani Nikula]

[Sorry this turned out a long email, I didn't have the time to write a
short one.]

On Wed, 10 Feb 2016, Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
> On Wed, Feb 10, 2016 at 1:09 AM, Jonathan Corbet <corbet@lwn.net> wrote:
>> On Tue, 26 Jan 2016 14:08:45 +0200
>> Jani Nikula <jani.nikula@intel.com> wrote:
>>
>>> I'm afraid we've done some overlapping work in the mean time, but I'm
>>> happy we've both looked at the tool chain, and can have a more
>>> meaningful conversation now.
>>
>> [Adding Keith since you said you wanted to be a part of this - let us know
>> when you've had enough!]
>>
>> So I've spent a bit of time looking at this, and quite a bit more time
>> talking with various folks at LCA.  There is pretty much universal
>> agreement that this is interesting work and the direction we'd like to
>> go.  My current hope is that we can merge some version of it for 4.6 and
>> see where it goes from there.
>>
>> So naturally I have some thoughts on the whole thing...
>>
>>  - I would like to format directly to HTML if at all possible.  It seems
>>    it should be possible to get a table of contents into the files, and
>>    the feedback I got was that a TOC would be enough for navigation - it
>>    would not be necessary to split the files at that point.  We might
>>    still want to try to figure that out too, though.  In any case, this
>>    isn't a show stopper, in that we can change it anytime if a better way
>>    shows up.  But I'd like to have it in mind.
>
> I think for 4.6 it'd be best to go with the hybrid asciidoc->docbook
> toolchain, since that's less disruptive. And with that we can also
> fully concentrating on the frontend, and how it'll look and behave.

I'd like to clarify the end goal a bit more before deciding what to do
next. In particular, is the aim to have asciidoc->HTML only or dual
asciidoc->HTML and asciidoc->XML->whatever? Or independent
asciidoc->HTML first, with the existing DocBook on the side until
everything's converted? Something else?

Direct asciidoc->HTML has the problem I mentioned that there is no
chunked output. If the source is big (as-is or via asciidoc includes)
the output is big. The current gpu.tmpl turned way too big. We could
alleviate that by splitting the source documents into smaller pieces (in
gpu.tmpl case it's desirable no matter what), and tying them together
via cross-references and TOC rather than asciidoc includes.

The problem with this, in turn, is that I don't really know how
automatic cross-referencing between kernel-doc comments would turn out
then (e.g. i915 kernel-doc references a symbol in drm core kernel-doc
after gpu.tmpl split) as asciidoc would process the files
independently. A kernel-doc comment writer shouldn't have to know which
document the referenced symbol is in... We could do post-processing I
guess, but I'd really like to get rid of the homebrew aspects here.

Is it acceptable to have dead links when referencing symbols outside of
the document in question, for the time being, until someone figures out
a nice way to do this?

> Once that's solid we can look into the icing on the cake for later
> kernels I think.
>
>>  - Asciidoc templates and processing should happen in a new directory
>>    (perhaps imaginatively called "asciidoc"); having them in a directory
>>    called "DocBook" seems a little weird.  More importantly, though, I'd
>>    like to separate them out as a fresh start, and not mess with the
>>    existing DocBook templates until we decide we don't need them anymore.
>>    If we could end up with a cleaner, simpler makefile in the process,
>>    that would be a bonus.
>
> For the long term dream plan of including other .txt files from the
> existing pile of unstructured docs, do we really want a separate
> asciidoc directory? Or just .asciidoc as a special extension?

Also in my dream world you could have asciidoc files anywhere in the
Documentation tree, with a Makefile per directory identifying which ones
should be processed as asciidoc. I might even name them all .txt, and
you wouldn't have to rename existing "almost markup" plain text files to
have them processed, just fix the markup and update the Makefile. (FWIW
asciidoc suggests .txt extension, though asciidoctor suggests .adoc or
.asciidoc.) I think this would better promote a gradual transition to
lightweight markup, with easier to review patches. Also you mentioned
there's no structure under Documentation. Allowing asciidoc files
anywhere would, I think, help gradual restructuring.

The output could be a subdirectory (one per output format?) under
Documentation.

>>  - I'm not sold on the new inclusion mechanism.  Creating thousands of
>>    little files and tracking them for dependencies and such doesn't seem
>>    like a simplification or a path toward better performance.  I would
>>    like to at least consider keeping the direct-from-source inclusion.
>
> The motivation behind the new inclusion mechanism isn't the speed-up
> due to parallelization, but being able to use native asciidoc
> includes. With those you can pass options to e.g. shift the hierarchy.
> With that you can do subheadings in DOC: sections and then seamlessly
> include them. Or similar stuff.
>
> The speed-up due to parallelization is just a small bonus.
>
> Also generating thousands of files is totally not unheard of in the kernel:
>
> $ find include/config | wc -l
> 2623
>
> None of those are in git.

Yes, my main motivation here was to get rid of the preprocessing step
(currently tmpl->xml). I wanted to have the source documents in pure
markup which could be directly processed by asciidoc. I wanted to have
the editor markup helpers and syntax highlighting just work, with no
extra non-markup cruft to confuse it. (For example, emacs tells me the
current tmpl files are invalid XML because of the docproc directives.)
This ties back to the dream above; just have .txt files with no
preprocessing step, IMO it's less confusing for actually writing the
docs.

I didn't think there'd be anything weird about having thousands of
intermediate files generated from source files, with dependencies set
and working, just like we have .o files.

Sure, the mechanism is a proof-of-concept, rough around the edges, and
needs to stow away the intermediate files better, but I still think it's
a conceptually better approach than adding a layer of homebrew when we
have a chance to break away from that. And there's the bonus of getting
parallelization, which I think just backs the concept.

I did try to make asciidoc filters and plugins work for including
kernel-doc, which might have been a better match to what Jon wants, but
without docproc in between. I didn't quite manage to make that work, and
there's the problem they're both incompatible with asciidoctor.

>>  - Insisting on EXPORT_SYMBOL being in the same file doesn't seem like
>>    it's going to work for now; that could maybe change after Al's work
>>    goes in, which could be fairly soon.
>
> Hm, assuming Al gets his stuff into 4.6 could we just assume this? It
> holds true for gpu docs already I think, and most other subsystems.
> The trouble iirc is all around asm and similar stuff, and we can't
> kerneldoc asm afaik.

I'd turn this around. IMO the problem isn't insisting EXPORT_SYMBOL is
in the same file as the definition of the symbol. The problem is
insisting that the kernel-doc comment is in the same file as the
EXPORT_SYMBOL and the definition. Particularly include/media has plenty
of kernel-doc in headers with the declarations.

If we can't insist on that, we could teach kernel-doc to scan a list of
other files for the EXPORT_SYMBOLs, instead of having that logic
externally in docproc. This should be trivial, especially if you know
perl. (Unfortunately this might get a little tricky with the include
syntax.)

This was mostly driven by the desire to get rid of the docproc
preprocessing step.

>> Please let me know your thoughts on the above.  Do you think you can find
>> some time over the next month for this?  I'll try to shake loose some time
>> too, but, well, $EXCUSES...

If we can come up with a plan where I can be reasonably sure the
polished effort isn't going down the drain... ;)

> One more thing we discussed: Did you ping kbuild folks already? Or
> want to get some agreement on the overall build process first?

I think CONFIG_BUILD_DOCSRC vs. having documentation targets directly in
Documentation/Makefile (instead of top level make issuing recursive make
in Documentation/DocBook/Makefile) should be reconciliated
somehow. Frankly, I find it odd that the hostprog targets under
Documentation seem to be better class citizens than documentation
targets. Not saying they can't both be there, but they should coexist.

BR,
Jani.



-- 
Jani Nikula, Intel Open Source Technology Center

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jani Nikula]

On Wed, 10 Feb 2016, Jani Nikula <jani.nikula@intel.com> wrote:
>> On Wed, Feb 10, 2016 at 1:09 AM, Jonathan Corbet <corbet@lwn.net> wrote:
>>>  - I'm not sold on the new inclusion mechanism.  Creating thousands of
>>>    little files and tracking them for dependencies and such doesn't seem
>>>    like a simplification or a path toward better performance.  I would
>>>    like to at least consider keeping the direct-from-source inclusion.

...

> Yes, my main motivation here was to get rid of the preprocessing step
> (currently tmpl->xml). I wanted to have the source documents in pure
> markup which could be directly processed by asciidoc. I wanted to have
> the editor markup helpers and syntax highlighting just work, with no
> extra non-markup cruft to confuse it. (For example, emacs tells me the
> current tmpl files are invalid XML because of the docproc directives.)
> This ties back to the dream above; just have .txt files with no
> preprocessing step, IMO it's less confusing for actually writing the
> docs.

I suppose a compromise could be to put the docproc directives in
asciidoc comments to keep the files pure asciidoc and to hide the
preprocessing step from the document writers, i.e. call them asciidoc
and name them .txt instead of .tmpl or something. While I'm not thrilled
about the idea of keeping docproc around, this would be progress, would
avoid the EXPORT_SYMBOL problem for now, and, most importantly, wouldn't
block us from doing what I suggested as a future iteration.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jonathan Corbet]

On Wed, 10 Feb 2016 09:07:22 +0100
Daniel Vetter <daniel.vetter@ffwll.ch> wrote:

> I think for 4.6 it'd be best to go with the hybrid asciidoc->docbook
> toolchain, since that's less disruptive. And with that we can also
> fully concentrating on the frontend, and how it'll look and behave.

That can be fine, I'd just like to have the end goal in mind.  For the near
future we should go with what actually works, definitely.

> >  - Asciidoc templates and processing should happen in a new directory
> >    (perhaps imaginatively called "asciidoc"); having them in a directory
> >    called "DocBook" seems a little weird.  More importantly, though, I'd
> >    like to separate them out as a fresh start, and not mess with the
> >    existing DocBook templates until we decide we don't need them anymore.
> >    If we could end up with a cleaner, simpler makefile in the process,
> >    that would be a bonus.  
> 
> For the long term dream plan of including other .txt files from the
> existing pile of unstructured docs, do we really want a separate
> asciidoc directory? Or just .asciidoc as a special extension?

That's a good question.  I can certainly see the value of mixing the
templates with the rest, but that's a longer-term thing.  I'd sure like to
clean up the main Documentation/ directory before we start scattering
asciidoc stuff around there.  For the moment, I think my preference is
still to focus this work in one place where it's easily found and played
with.  Straightening out this directory is going to involve a fair amount
of moving stuff around, adding a few template files to that debt isn't
going to change the situation much.

> >  - I'm not sold on the new inclusion mechanism.  Creating thousands of
> >    little files and tracking them for dependencies and such doesn't seem
> >    like a simplification or a path toward better performance.  I would
> >    like to at least consider keeping the direct-from-source inclusion.  
> 
> The motivation behind the new inclusion mechanism isn't the speed-up
> due to parallelization, but being able to use native asciidoc
> includes. With those you can pass options to e.g. shift the hierarchy.
> With that you can do subheadings in DOC: sections and then seamlessly
> include them. Or similar stuff.
> 
> The speed-up due to parallelization is just a small bonus.
> 
> Also generating thousands of files is totally not unheard of in the kernel:
> 
> $ find include/config | wc -l
> 2623
> 
> None of those are in git.

Well, we are talking about an order of magnitude more files...  Still, I
said "not sold on" rather than "violently opposed to".  It does seem that
there are some good reasons for doing things this way, including, as Jani
said, getting rid of docproc and not mixing in a weird alien include
syntax.  I'm still not 100% sold, but I'll not hold up a working patch on
this point.

> >  - Insisting on EXPORT_SYMBOL being in the same file doesn't seem like
> >    it's going to work for now; that could maybe change after Al's work
> >    goes in, which could be fairly soon.  
> 
> Hm, assuming Al gets his stuff into 4.6 could we just assume this? It
> holds true for gpu docs already I think, and most other subsystems.
> The trouble iirc is all around asm and similar stuff, and we can't
> kerneldoc asm afaik.

asm stuff and things built into libraries.  But it does seem that this is
well on the way toward being fixed.

> One more thing we discussed: Did you ping kbuild folks already? Or
> want to get some agreement on the overall build process first?

Not yet.  It's worth doing...it would be nice, someday, if docs makefiles
could just have lines like:

      adoc-y += drm.txt ...

but I think we should work out how the pieces fit together before we get
too worried about the details of the build system.  What we have now isn't
particularly well integrated, we're not likely to make it all that much
worse...

Thanks,

jon

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jonathan Corbet]

On Wed, 10 Feb 2016 16:03:38 +0200
Jani Nikula <jani.nikula@intel.com> wrote:

> I'd like to clarify the end goal a bit more before deciding what to do
> next. In particular, is the aim to have asciidoc->HTML only or dual
> asciidoc->HTML and asciidoc->XML->whatever? Or independent
> asciidoc->HTML first, with the existing DocBook on the side until
> everything's converted? Something else?

asciidoc->HTML on its own isn't viable, I think; we do have people wanting
other formats.  Though one might well ask when somebody last successfully
generated PDF...maybe it's not worth the trouble.  I would like epub
someday...

There's also people who actually use the man-page output.  I don't think
that should require the xml step; getting rid of that might make it
possible to do "make mandocs" and have it finish before the next merge
window opens...

> Direct asciidoc->HTML has the problem I mentioned that there is no
> chunked output. If the source is big (as-is or via asciidoc includes)
> the output is big. The current gpu.tmpl turned way too big. We could
> alleviate that by splitting the source documents into smaller pieces (in
> gpu.tmpl case it's desirable no matter what), and tying them together
> via cross-references and TOC rather than asciidoc includes.

We talked about that a bit in Geelong; the short-term idea was to generate
a TOC and use CSS to place it correctly.  Daniel, if I heard you correctly,
you thought that would be a fine solution that would remove the need for
chunked output.  Keith seemed interested in looking into this too.

I would still like to look into splitting up the output.  One would *think*
we ought to be able to do that without the whole docbook infrastructure,
but, then, I'm known to be a naive optimist...

> The problem with this, in turn, is that I don't really know how
> automatic cross-referencing between kernel-doc comments would turn out
> then (e.g. i915 kernel-doc references a symbol in drm core kernel-doc
> after gpu.tmpl split) as asciidoc would process the files
> independently. A kernel-doc comment writer shouldn't have to know which
> document the referenced symbol is in... We could do post-processing I
> guess, but I'd really like to get rid of the homebrew aspects here.
> 
> Is it acceptable to have dead links when referencing symbols outside of
> the document in question, for the time being, until someone figures out
> a nice way to do this?

Short-term *maybe*, but I think we'd want to figure that one out quickly. 

> Also in my dream world you could have asciidoc files anywhere in the
> Documentation tree, with a Makefile per directory identifying which ones
> should be processed as asciidoc. I might even name them all .txt, and
> you wouldn't have to rename existing "almost markup" plain text files to
> have them processed, just fix the markup and update the Makefile. (FWIW
> asciidoc suggests .txt extension, though asciidoctor suggests .adoc or
> .asciidoc.) I think this would better promote a gradual transition to
> lightweight markup, with easier to review patches. Also you mentioned
> there's no structure under Documentation. Allowing asciidoc files
> anywhere would, I think, help gradual restructuring.

I agree with all of this, but I still think that, for the short term while
we're figuring out how all this works, it's better to concentrate it in one
place where people can actually find it...

> I'd turn this around. IMO the problem isn't insisting EXPORT_SYMBOL is
> in the same file as the definition of the symbol. The problem is
> insisting that the kernel-doc comment is in the same file as the
> EXPORT_SYMBOL and the definition. Particularly include/media has plenty
> of kernel-doc in headers with the declarations.
> 
> If we can't insist on that, we could teach kernel-doc to scan a list of
> other files for the EXPORT_SYMBOLs, instead of having that logic
> externally in docproc. This should be trivial, especially if you know
> perl. (Unfortunately this might get a little tricky with the include
> syntax.)
> 
> This was mostly driven by the desire to get rid of the docproc
> preprocessing step.

...and that's a worthy goal.  In an ideal world, it's all found together.
I think we should probably proceed with the idea that the EXPORT_SYMBOL
issue can be dealt with.

> >> Please let me know your thoughts on the above.  Do you think you can find
> >> some time over the next month for this?  I'll try to shake loose some time
> >> too, but, well, $EXCUSES...  
> 
> If we can come up with a plan where I can be reasonably sure the
> polished effort isn't going down the drain... ;)

Seems we should be able to do that.  We want this stuff, even I'm not so
dumb as to send it down the drain when things are so close...:)

jon

Re: [RFC] A first shot at asciidoc-based formatted docs

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 2589 bytes --]

Jonathan Corbet <corbet@lwn.net> writes:

> [Adding Keith since you said you wanted to be a part of this - let us know
> when you've had enough!]

Thanks.

>  - I would like to format directly to HTML if at all possible.

Agreed. asciidoc's docbook path seems to only increase the amount of
software involved.

>    It seems it should be possible to get a table of contents into the
>    files, and the feedback I got was that a TOC would be enough for
>    navigation

I spent a few hours on the flight home reading asciidoc source code, and
it is a stream processor with a stack. The output is generated with some
simple templates, one for each backend. Here's the xhtml1.1 template for
section level sections: (sect1 in the .conf file):

        [sect1]
        <div class="sect1{style? {style}}{role? {role}}">
        <h2{id? id="{id}"}>{numbered?{sectnum} }{title}</h2>
        <div class="sectionbody">
        |
        </div>
        </div>

The contents of the section get inserted at the |; it's nesting, so
[sect2] bits would get expanded while being processed.

Each asciidoc backend has dramatically different functionality. It's
pretty clear to me that the 'docbook' backend has the best support for
larger documents as that provides 'book-scale' processing bits. I've
recently written a book in asciidoc using the docbook backend, and the
html and pdf results are quite comparable. Using the html backend from
asciidoc yields a significantly different result.

I think it should be pretty easy to hack asciidoc to add diversions to
hold TOC contents while generating the rest of the doc and then replay
the diversion into the final document. Something like:

        [sect1]
        <div class="sect1{style? {style}}{role? {role}}">
        <h2{id? id="{id}"}>{numbered?{sectnum} }{title}</h2>
        |>"table-of-contents"<dl class="toc">
                <dt>
                        <span class="section">
                                <a href="{id}">{numbered?{sectnum} {title}</a>
                        </span>
                </dt>
        </dl>
        <div class="sectionbody">
        |
        </div>
        </div>

At the end of the document, we'd have some way of wrapping the diversion
in suitable additional bits to  complete the TOC, which would then be
formatted by CSS.

This same technique could be used to create lists of figures and tables.

The goal would be to create an html document which could be used without
javascript, and that would work without css as well.

-- 
-keith

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 810 bytes --]

Re: [RFC] A first shot at asciidoc-based formatted docs

[Jani Nikula]

On Thu, 11 Feb 2016, Keith Packard <keithp@keithp.com> wrote:
> I think it should be pretty easy to hack asciidoc to add diversions to
> hold TOC contents while generating the rest of the doc and then replay
> the diversion into the final document.

One of the chief complaints with the current pipeline (and some of the
proposals) has been the need to install lots of tools with lots of
dependencies. I would like to avoid the need to install bleeding edge
tools and stick to what's already widely available in distros. Thus I
would like to avoid hacking asciidoc for our needs.

Also, I'd really like to not have to decide between asciidoc and
asciidoctor, and only use features supported by both. Let the users pick
which one suits them better.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: [RFC] A first shot at asciidoc-based formatted docs

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 1482 bytes --]

Jonathan Corbet <corbet@lwn.net> writes:

> asciidoc->HTML on its own isn't viable, I think; we do have people wanting
> other formats.  Though one might well ask when somebody last successfully
> generated PDF...maybe it's not worth the trouble.  I would like epub
> someday...

I'm hopeful that I can hack up asciidoc to generate usable HTML
directly. Once we've got HTML, we've got epub.

If we don't want to use docbook for pdf, asciidoc has a native latex
backend. That's in about the same shape as the html backend. Would that
be better than docbook?

> There's also people who actually use the man-page output.  I don't think
> that should require the xml step; getting rid of that might make it
> possible to do "make mandocs" and have it finish before the next merge
> window opens...

Adding a troff backend to asciidoc would be simple enough; I'm not sure
what other method you'd suggest here.

> We talked about that a bit in Geelong; the short-term idea was to generate
> a TOC and use CSS to place it correctly.  Daniel, if I heard you correctly,
> you thought that would be a fine solution that would remove the need for
> chunked output.  Keith seemed interested in looking into this too.

Here's an example that takes the docbook output with some simple CSS
hacks to place the TOC alongside the document in a separate scrolling
list. With a small bit of javascript, I'm pretty sure that could have
collapsible entries.

-- 
-keith

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 810 bytes --]

Re: [RFC] A first shot at asciidoc-based formatted docs

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 1160 bytes --]

Jani Nikula <jani.nikula@intel.com> writes:

> One of the chief complaints with the current pipeline (and some of the
> proposals) has been the need to install lots of tools with lots of
> dependencies. I would like to avoid the need to install bleeding edge
> tools and stick to what's already widely available in distros. Thus I
> would like to avoid hacking asciidoc for our needs.

Agreed. That means using docbook for now; the native html output from
asciidoc is simply not usable for anything more complicated than a short
web page. However, getting ready to collapse the pipeline by eliminating
docbook seems like a good medium-term goal.

> Also, I'd really like to not have to decide between asciidoc and
> asciidoctor, and only use features supported by both. Let the users pick
> which one suits them better.

That's harder; you'll have much different output from the two
processors. I'd encourage the selection of one of these two tools
instead of trying to support both. I've settled on using only asciidoc
for my other projects because it doesn't require the installation of a
whole new language environment.

-- 
-keith

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 810 bytes --]

Re: [RFC] A first shot at asciidoc-based formatted docs

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 912 bytes --]

Keith Packard <keithp@keithp.com> writes:

> The goal would be to create an html document which could be used without
> javascript, and that would work without css as well.

I've managed to hack up asciidoc to generate the TOC within the
document, rather than requiring javascript. The changes are fairly
minor, and seem to add a nice generalization to the asciidoc environment
which should be useful in other contexts.

The changes consist of two bits -- the first is to allow the diversion
of some text from .conf file sections, the second is to postpone some
attribute processing to a second pass over the document so that the TOC
can be inserted in the desired location, instead of requiring that it be
placed at the bottom.

I've sent these changes upstream, and also pushed them to a personal
asciidoc git repository at :

        git clone git://keithp.com/git/asciidoc

-- 
-keith

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 810 bytes --]