[PATCH for-4.11 v2 0/9] Provide support matrix generator

[PATCH for-4.11 v2 0/9] Provide support matrix generator

[Ian Jackson]

This series provides code to generate a feature support matrix, to
replace the one on the wiki.  You can see an example of the output
here:
  http://xenbits.xen.org/people/iwj/2018/support-matrix-example-v2/t.html
There is also an accompanying SUPPORT.html to make the links for 4.11
work.

Patches 1-3 fix syntax trouble in SUPPORT.md (and commit us to using
pandoc format for it).  Patches 4-7 arrange to generate a SUPPORT.html
at the toplevel of the generated docs.  These should all be backported
to 4.10.

Patches 8 and 9 are the support matrix generator.  They are not hooked
into any of the xen.git makefiles but it seemed best to have them here
in xen.git/docs rather than floating about in some ad hoc crontab
repo.

  ar 1/9 SUPPORT.md: Syntax: Fix some bullet lists
  ar 2/9 SUPPORT.md: Syntax: Fix a typo "States"
 m r 3/9 SUPPORT.md: Syntax: Provide a title rather than a
   r 4/9 docs/gen-html-index: Extract titles from HTML documents
   r 5/9 docs/gen-html-index: Support documents at the toplevel
   r 6/9 docs/Makefile: Introduce GENERATE_PANDOC_RULE_RAW
   r 7/9 docs/Makefile: Format SUPPORT.md into the toplevel
 +   8/9 docs: Provide parse-support-md
 +   9/9 docs: Provide support-matrix-generate, to generate ...

 a = acked (by a tools or REST maintainer)
 r = release-acked for 4.11
 + = new patch
 m = modified commit message

I think if the output is tolerably accurate, it would be valuable to
get this set up as a cron job ASAP.  Stylistic and formatting
improvements etc. can be done on top.

My plan is that the cron job will use the matrix generator from
xen.git#master.

For my reference, this was generated with
   docs/support-matrix-generate HEAD SUPPORT.html refs/heads/wip.support-stmt-NN https://xenbits.xen.org/docs/NN-testing/SUPPORT.html >t.html
  rsync -vP t.html SUPPORT.html xenbits:public_html/2018/support-matrix-example-v1/

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 1/9] SUPPORT.md: Syntax: Fix some bullet lists

[Ian Jackson]

Continuations of bullet list items must be indented by exactly 4
spaces (according to pandoc_markdown(5) on Debian jessie).

This is most easily achieved by making the bullet list items have two
spaces before the `*'.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Release-acked-by: Juergen Gross <jgross@suse.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
---
 SUPPORT.md | 36 ++++++++++++++++++------------------
 1 file changed, 18 insertions(+), 18 deletions(-)

diff --git a/SUPPORT.md b/SUPPORT.md
index c72a25b..1c5220b 100644
--- a/SUPPORT.md
+++ b/SUPPORT.md
@@ -783,40 +783,40 @@ What is the risk of it exhibiting bugs?
 
 General answers to the above:
 
- * **Here be dragons**
+  * **Here be dragons**
 
-   Pretty likely to still crash / fail to work.
-   Not recommended unless you like life on the bleeding edge.
+    Pretty likely to still crash / fail to work.
+    Not recommended unless you like life on the bleeding edge.
 
- * **Quirky**
+  * **Quirky**
 
-   Mostly works but may have odd behavior here and there.
-   Recommended for playing around or for non-production use cases.
+    Mostly works but may have odd behavior here and there.
+    Recommended for playing around or for non-production use cases.
 
- * **Normal**
+  * **Normal**
 
-   Ready for production use
+    Ready for production use
 
 ### Interface stability
 
 If I build a system based on the current interfaces,
 will they still work when I upgrade to the next version?
 
- * **Not stable**
+  * **Not stable**
 
-   Interface is still in the early stages and
-   still fairly likely to be broken in future updates.
+    Interface is still in the early stages and
+    still fairly likely to be broken in future updates.
 
- * **Provisionally stable**
+  * **Provisionally stable**
 
-   We're not yet promising backwards compatibility,
-   but we think this is probably the final form of the interface.
-   It may still require some tweaks.
+    We're not yet promising backwards compatibility,
+    but we think this is probably the final form of the interface.
+    It may still require some tweaks.
 
- * **Stable**
+  * **Stable**
 
-   We will try very hard to avoid breaking backwards  compatibility,
-   and to fix any regressions that are reported.
+    We will try very hard to avoid breaking backwards  compatibility,
+    and to fix any regressions that are reported.
 
 ### Security supported
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 2/9] SUPPORT.md: Syntax: Fix a typo "States"

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Release-acked-by: Juergen Gross <jgross@suse.com>
Acked-by: George Dunlap <george.dunlap@citrix.com>
---
 SUPPORT.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/SUPPORT.md b/SUPPORT.md
index 1c5220b..e447069 100644
--- a/SUPPORT.md
+++ b/SUPPORT.md
@@ -360,7 +360,7 @@ Guest-side driver capable of speaking the Xen PV block protocol
     Status, FreeBSD: Supported, Security support external
     Status, NetBSD: Supported, Security support external
     Status, OpenBSD: Supported, Security support external
-    States, Windows: Supported
+    Status, Windows: Supported
 
 Guest-side driver capable of speaking the Xen PV networking protocol
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 3/9] SUPPORT.md: Syntax: Provide a title rather than a spurious empty section

[Ian Jackson]

This commits (more or less) this file to be processed with pandoc,
rather than other markdown processors.  There is, unfortunately, no
widely-accepted way to declare a title for the document.

I tested feeding the document to markdown(1) on Debian jessie and it
reproduced the % line as if it were simple text.  I guess many other
markdown processors will do something similarly tolerable.  My
internet searches did not discover a markdown processor that used
lines starting with & for something else.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Release-acked-by: Juergen Gross <jgross@suse.com>
---
v2: Discuss the compatibility problems with & in the commit message.
---
 SUPPORT.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/SUPPORT.md b/SUPPORT.md
index e447069..264b23f 100644
--- a/SUPPORT.md
+++ b/SUPPORT.md
@@ -1,4 +1,4 @@
-# Support statement for this release
+% Support statement for this release
 
 This document describes the support status
 and in particular the security support status of the Xen branch
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 3/9] SUPPORT.md: Syntax: Provide a title rather than a spurious empty section

[George Dunlap]

On 04/11/2018 04:35 PM, Ian Jackson wrote:
> This commits (more or less) this file to be processed with pandoc,
> rather than other markdown processors.  There is, unfortunately, no
> widely-accepted way to declare a title for the document.
> 
> I tested feeding the document to markdown(1) on Debian jessie and it
> reproduced the % line as if it were simple text.  I guess many other
> markdown processors will do something similarly tolerable.  My
> internet searches did not discover a markdown processor that used
> lines starting with & for something else.

s/&/%/?

In any case:

Acked-by: George Dunlap <george.dunlap@citrix.com>

> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> Release-acked-by: Juergen Gross <jgross@suse.com>
> ---
> v2: Discuss the compatibility problems with & in the commit message.
> ---
>  SUPPORT.md | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/SUPPORT.md b/SUPPORT.md
> index e447069..264b23f 100644
> --- a/SUPPORT.md
> +++ b/SUPPORT.md
> @@ -1,4 +1,4 @@
> -# Support statement for this release
> +% Support statement for this release
>  
>  This document describes the support status
>  and in particular the security support status of the Xen branch
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 3/9] SUPPORT.md: Syntax: Provide a title rather than a spurious empty section

[Ian Jackson]

George Dunlap writes ("Re: [PATCH 3/9] SUPPORT.md: Syntax: Provide a title rather than a spurious empty section"):
> On 04/11/2018 04:35 PM, Ian Jackson wrote:
> > I tested feeding the document to markdown(1) on Debian jessie and it
> > reproduced the % line as if it were simple text.  I guess many other
> > markdown processors will do something similarly tolerable.  My
> > internet searches did not discover a markdown processor that used
> > lines starting with & for something else.
> 
> s/&/%/?

Oops, yes.

> In any case:
> 
> Acked-by: George Dunlap <george.dunlap@citrix.com>

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 4/9] docs/gen-html-index: Extract titles from HTML documents

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Release-acked-by: Juergen Gross <jgross@suse.com>
---
 docs/gen-html-index | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/docs/gen-html-index b/docs/gen-html-index
index e9792bf..5b43b42 100644
--- a/docs/gen-html-index
+++ b/docs/gen-html-index
@@ -10,6 +10,7 @@ use warnings;
 use Getopt::Long;
 use IO::File;
 use File::Basename;
+use HTML::TreeBuilder::XPath;
 
 Getopt::Long::Configure('bundling');
 
@@ -64,6 +65,18 @@ sub make_linktext ($) {
     return "$1($2)" if $l =~ m,^man/(.*)\.([0-9].*)\.html,;
     $l =~ s/.(?:html|txt)$//g;
     return $index{$l} if exists $index{$l};
+
+    my $from_html;
+    eval {
+        my $tree = new HTML::TreeBuilder::XPath;
+        my $f = "$outdir/$l.html";
+        open F, '<', $f or die "$l $f $!";
+        $tree->parse_file(\*F) or die;
+        close F;
+        $from_html = $tree->findvalue("/html/head/title");
+    };
+    return $from_html if $from_html;
+
     return basename($l);
 }
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 5/9] docs/gen-html-index: Support documents at the toplevel

[Ian Jackson]

There are none yet.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Release-acked-by: Juergen Gross <jgross@suse.com>
---
 docs/gen-html-index | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/docs/gen-html-index b/docs/gen-html-index
index 5b43b42..8258e2b 100644
--- a/docs/gen-html-index
+++ b/docs/gen-html-index
@@ -137,6 +137,10 @@ sub dirs($)
     return @dirs;
 }
 
+foreach my $of (grep { !m{/} } @docs) {
+    $top .= make_link($of,'');
+}
+
 foreach my $od (sort { $a cmp $b } uniq map { dirs($_) } @docs) {
     my @d = (grep /^\Q$od\E/, @docs);
     if ( @d == 1 and $d[0] eq "$od/index.html" )
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 6/9] docs/Makefile: Introduce GENERATE_PANDOC_RULE_RAW

[Ian Jackson]

We are going to want to format SUPPORT.md which does not match the
filename patterns in docs/.  So provide a way to make an ad-hoc rule
using pandoc with the standard options.

No functional change in this patch.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Release-acked-by: Juergen Gross <jgross@suse.com>
---
 docs/Makefile | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/docs/Makefile b/docs/Makefile
index 6743fa3..d82463f 100644
--- a/docs/Makefile
+++ b/docs/Makefile
@@ -237,17 +237,18 @@ txt/%.txt: %.markdown
 	$(INSTALL_DATA) $< $@
 
 # Metarule for generating pandoc rules.
-define GENERATE_PANDOC_RULE
-# $(1) is the target documentation format. $(2) is the source format.
-
-$(1)/%.$(1): %.$(2)
+define GENERATE_PANDOC_RULE_RAW
+$(1): $(2)
 ifneq ($(PANDOC),)
 	@$(INSTALL_DIR) $$(@D)
 	$(PANDOC) --number-sections --toc --standalone $$< --output $$@
 else
 	@echo "pandoc not installed; skipping $$@"
 endif
-
+endef
+define GENERATE_PANDOC_RULE
+# $(1) is the target documentation format. $(2) is the source format.
+$(call GENERATE_PANDOC_RULE_RAW,$(1)/%.$(1),%.$(2))
 endef
 $(eval $(call GENERATE_PANDOC_RULE,pdf,pandoc))   # pdf/%.pdf: %.pandoc
 $(eval $(call GENERATE_PANDOC_RULE,txt,pandoc))   # txt/%.txt: %.pandoc
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 7/9] docs/Makefile: Format SUPPORT.md into the toplevel

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Release-acked-by: Juergen Gross <jgross@suse.com>
---
 docs/Makefile | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/docs/Makefile b/docs/Makefile
index d82463f..b300bb6 100644
--- a/docs/Makefile
+++ b/docs/Makefile
@@ -28,7 +28,8 @@ DOC_MAN7 := $(patsubst man/%.pod.7,man7/%.7,$(MAN7SRC-y)) \
 	        $(patsubst man/%.markdown.7,man7/%.7,$(MAN7SRC-y))
 DOC_MAN8 := $(patsubst man/%.pod.8,man8/%.8,$(MAN8SRC-y)) \
 	        $(patsubst man/%.markdown.8,man8/%.8,$(MAN8SRC-y))
-DOC_HTML := $(patsubst %.markdown,html/%.html,$(MARKDOWNSRC-y)) \
+DOC_HTML := html/SUPPORT.html \
+            $(patsubst %.markdown,html/%.html,$(MARKDOWNSRC-y)) \
             $(patsubst %.pandoc,html/%.html,$(PANDOCSRC-y)) \
             $(patsubst man/%.markdown.1,html/man/%.1.html,$(MAN1SRC-y)) \
             $(patsubst man/%.markdown.5,html/man/%.5.html,$(MAN5SRC-y)) \
@@ -255,6 +256,8 @@ $(eval $(call GENERATE_PANDOC_RULE,txt,pandoc))   # txt/%.txt: %.pandoc
 $(eval $(call GENERATE_PANDOC_RULE,html,pandoc))  # html/%.html: %.pandoc
 $(eval $(call GENERATE_PANDOC_RULE,pdf,markdown)) # pdf/%.pdf: %.markdown
 
+$(eval $(call GENERATE_PANDOC_RULE_RAW,html/SUPPORT.html,$(XEN_ROOT)/SUPPORT.md)) # pdf/%.pdf: %.markdown
+
 ifeq (,$(findstring clean,$(MAKECMDGOALS)))
 $(XEN_ROOT)/config/Docs.mk:
 	$(error You have to run ./configure before building docs)
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 8/9] docs: Provide parse-support-md

[Ian Jackson]

This utility reads json format pandoc output, from parsing one or more
SUPPORT.md files, and generates an HTML table element containing the
principal version and feature information.

This is rather hairier than I anticipated when I started out; hence
the 400-odd-line Perl script.

Machinery to assemble the appropriate inputs for parse-support-md
will be in the next commit.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: New in this version of the series.
---
 docs/parse-support-md | 410 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 410 insertions(+)
 create mode 100755 docs/parse-support-md

diff --git a/docs/parse-support-md b/docs/parse-support-md
new file mode 100755
index 0000000..b882705
--- /dev/null
+++ b/docs/parse-support-md
@@ -0,0 +1,410 @@
+#!/usr/bin/perl -w
+#
+# Written with reference to pandoc_markdown from Debian jessie
+# We require atx-style headers
+#
+# usage:
+#   pandoc -t json SUPPORT.md >j-unstable
+#   git-cat-file ... | pandoc -t json >j-4.10
+#   docs/parse-support-md \
+#            j-unstable https://xenbits/unstable/SUPPORT.html
+#            j-4.10 https://xenbits/4.10/SUPPORT.html
+# or equivalent
+
+use strict;
+use JSON;
+use Tie::IxHash;
+use IO::File;
+use CGI qw(escapeHTML);
+use Data::Dumper;
+
+#---------- accumulating input/output ----------
+
+# This combines information from all of the input files.
+
+sub new_sectlist () { { } };
+our $toplevel_sectlist = new_sectlist();
+# an $sectlist is
+#   { }                 nothing seen yet
+#   a tied hashref      something seen
+# (tied $sectlist)    is an object of type Tie::IxHash
+# $sectlist->{KEY} a $sectnode:
+# $sectlist->{KEY}{Status}[VI] = absent or markdown content
+# $sectlist->{KEY}{HasText}[VI] = trueish iff there was a Para
+# $sectlist->{KEY}{Children} = a further $sectlist
+# $sectlist->{KEY}{Key} = KEY
+# $sectlist->{KEY}{Anchor} = value for < id="" > in the pandoc html
+#
+# A $sectnode represents a single section from the original markdown
+# document.  Its subsections are in Children.
+#
+# Also, the input syntax:
+#    Status, something or other: Supported
+# is treated as a $sectnode, is as if it were a subsection -
+# one called `something or other'.
+#
+# KEY is the Anchor, or derived from the `something or other'.
+# It is used to match up identical features in different versions.
+
+#---------- state for this input file ----------
+
+our $version_index;
+our @version_urls;
+
+our @insections;
+# $insections[]{Key} = string
+# $insections[]{Anchor} = string or undef
+# $insections[]{Headline} = markdown content
+
+our $had_unknown;
+our $current_sectnode;
+# adding new variable ?  it must be reset in r_toplevel
+
+#---------- parsing ----------
+
+sub ri_Header {
+    my ($c) = @_;
+    my ($level, $infos, $hl) = @$c;
+#print STDERR 'RI_HEADER ', Dumper($c, \@c);
+    my ($id) = @$infos;
+    die unless $level >= 1;
+    die unless $level-2 <= $#insections;
+    $#insections = $level-2;
+    push @insections,
+        {
+         Key => $id,
+         Anchor => $id,
+         Headline => $hl,
+        };
+#print STDERR Dumper(\@insections);
+    $current_sectnode = undef;
+}
+
+sub ri_Para {
+    if ($current_sectnode) {
+        $current_sectnode->{HasText}[$version_index] = 1;
+    }
+};
+
+sub parse_feature_entry ($) {
+    my ($value) = @_;
+    die unless @insections;
+
+    my $sectnode;
+    my $anchor = '';
+    foreach my $s (@insections) {
+        my $sectlist = $sectnode
+            ? $sectnode->{Children} : $toplevel_sectlist;
+        my $key = $s->{Key};
+        $anchor = $s->{Anchor} if $s->{Anchor};
+        tie %$sectlist, 'Tie::IxHash' unless tied %$sectlist;
+#print STDERR "PARSE_FEATURE_ENTRY ", Dumper($s);
+        $sectlist->{$key} //=
+            {
+             Children => new_sectlist(),
+             Headline => $s->{Headline},
+             Key => $key,
+             Anchor => $anchor,
+            };
+        $sectnode = $sectlist->{$key};
+    }
+    die unless $sectnode;
+    $sectnode->{Status}[$version_index] = $value;
+    $current_sectnode = $sectnode;
+}
+
+sub ri_CodeBlock {
+    my ($c) = @_;
+    my ($infos, $text) = @$c;
+
+    if ($text =~ m{^(?: Functional\ completeness 
+                   | Functional\ stability
+                   | Interface\ stability
+                   | Security\ supported ) \:}x) {
+        # ignore this
+        return;
+    }
+    die "$had_unknown / $text ?" if $had_unknown;
+
+    my $toplevel = $text =~ m{^Xen-Version:};
+
+    foreach my $l (split /\n/, $text) {
+        $l =~ s/\s*$//;
+        next unless $l =~ m/\S/;
+
+        my ($descr, $value) =
+            $toplevel
+            ? $l =~ m{^([A-Z][-A-Z0-9a-z]+)\:\s+(\S.*)$}
+            : $l =~ m{^(?:Status|Supported)(?:\,\s*([^:]+))?\:\s+(\S.*)$}
+            or die ("$text\n^ cannot parse status codeblock line:".
+                    ($toplevel and 'top').
+                    "\n$l\n ?");
+        if (length $descr) {
+            die unless @insections;
+            my $key = lc $descr;
+            $key =~ y/ /-/;
+            $key =~ y/-0-9A-Za-z//cd;
+            $key = $insections[$#insections]{Anchor}.'--'.$key;
+            push @insections,
+                {
+                 Key => $key,
+                 Headline => [{ t => 'Str', c => $descr }],
+                };
+        }
+        parse_feature_entry $value;
+        if (length $descr) {
+            pop @insections;
+        }
+    }
+}
+
+sub process_unknown {
+    my ($c, $e) = @_;
+    $had_unknown = Dumper($e);
+}
+
+sub r_content ($) {
+    my ($i) = @_;
+    foreach my $e (@$i) {
+        my $f = ${*::}{"ri_$e->{t}"};
+        $f //= \&process_unknown;
+        $f->($e->{c}, $e);
+    }
+}
+
+sub r_toplevel ($) {
+    my ($i) = @_;
+
+    die unless defined $version_index;
+
+    @insections = ();
+    $had_unknown = undef;
+    $current_sectnode = undef;
+
+    foreach my $e (@$i) {
+        next unless ref $e eq 'ARRAY';
+        r_content $e;
+    }
+}
+
+sub read_inputs () {
+    $version_index = 0;
+
+    local $/;
+    undef $/;
+
+    while (my $f = shift @ARGV) {
+        push @version_urls, shift @ARGV;
+        eval {
+            open F, '<', $f or die $!;
+            my $input_toplevel = decode_json <F>;
+            r_toplevel $input_toplevel;
+        };
+        die "$@\nwhile processing input file $f\n" if $@;
+        $version_index++;
+    }
+}
+
+#---------- reprocessing ----------
+
+# variables generated by analyse_reprocess:
+our $maxdepth;
+
+sub pandoc2html_inline ($) {
+    my ($content) = @_;
+
+    my $json_fh = IO::File::new_tmpfile or die $!;
+    print $json_fh to_json([
+                            { unMeta => { } },
+                            [{ t => 'Para', c => $content }],
+                           ]) or die $!;
+    flush $json_fh or die $!;
+    seek $json_fh,0,0 or die $!;
+
+    my $c = open PD, "-|" // die $!;
+    if (!$c) {
+        open STDIN, "<&", $json_fh;
+        exec qw(pandoc -f json) or die $!;
+    }
+
+    local $/;
+    undef $/;
+    my $html = <PD>;
+    $?=$!=0;
+    if (!close PD) {
+        eval {
+            seek $json_fh,0,0 or die $!;
+            open STDIN, '<&', $json_fh or die $!;
+            system 'json_pp';
+        };
+        die "\n $? $!";
+    }
+
+    $html =~ s{^\<p\>}{} or die "$html ?";
+    $html =~ s{\</p\>$}{} or die "$html ?";
+    $html =~ s{\n$}{};
+    return $html;
+}
+
+sub reprocess_sectlist ($$);
+
+sub reprocess_sectnode ($$) {
+    my ($sectnode, $d) = @_;
+
+    $sectnode->{Depth} = $d;
+
+    if ($sectnode->{Status}) {
+        $maxdepth = $d if $d > $maxdepth;
+    }
+
+    if ($sectnode->{Headline}) {
+#            print STDERR Dumper($sectnode);
+        $sectnode->{Headline} =
+            pandoc2html_inline $sectnode->{Headline};
+    }
+
+    reprocess_sectlist $sectnode->{Children}, $d;
+}
+
+sub reprocess_sectlist ($$) {
+    my ($sectlist, $d) = @_;
+    $d++;
+
+    foreach my $sectnode (values %$sectlist) {
+        reprocess_sectnode $sectnode, $d;
+    }
+}
+
+sub count_rows_sectlist ($);
+
+sub count_rows_sectnode ($) {
+    my ($sectnode) = @_;
+    my $rows = 0;
+    $rows++ if $sectnode->{Status};
+    $rows += count_rows_sectlist $sectnode->{Children};
+    $sectnode->{Rows} = $rows;
+    return $rows;
+}
+
+sub count_rows_sectlist ($) {
+    my ($sectlist) = @_;
+    my $rows = 0;
+    foreach my $sectnode (values %$sectlist) {
+        $rows += count_rows_sectnode $sectnode;
+    }
+    return $rows;
+}
+
+# After reprocess_sectlist,
+#    ->{Headline}   is in html
+#    ->{Status}     is in plain text
+
+sub analyse_reprocess () {
+    $maxdepth = 0;
+    reprocess_sectlist $toplevel_sectlist, 0;
+}
+
+#---------- output ----------
+
+sub o { print @_ or die $!; }
+
+our @pending_headings;
+
+sub write_output_row ($) {
+    my ($sectnode) = @_;
+#    print STDERR 'WOR ', Dumper($d, $sectnode);
+    o('<tr>');
+    my $span = sub {
+        my ($rowcol, $n) = @_;
+        o(sprintf ' %sspan="%d"', $rowcol, $n) if $n != 1;
+    };
+    # This is all a bit tricky because (i) the input is hierarchical
+    # with variable depth, whereas the output has to have a fixed
+    # number of heading columns on the LHS; (ii) the HTML
+    # colspan/rowspan system means that when we are writing out, we
+    # have to not write table elements for table entries which have
+    # already been written with a span instruction that covers what we
+    # would write now.
+    while (my $heading = shift @pending_headings) {
+        o('<th valign="top"');
+        o(sprintf ' id="%s"', $heading->{Key});
+        $span->('row', $heading->{Rows});
+        $span->('col', $maxdepth - $heading->{Depth} + 1)
+            if !%{ $heading->{Children} };
+        o(' align="left">');
+        o($heading->{Headline});
+        o('</th>');
+    }
+    if (%{ $sectnode->{Children} }) {
+        # we suppressed the colspan above, but we do need to make the gap
+        my $n = $maxdepth - $sectnode->{Depth};
+        die 'XX '. Dumper($n, $sectnode) if $n<0;
+        if ($n) {
+            o('<td');
+            $span->('col', $n);
+            o('></td>');
+        }
+    }
+    for (my $i=0; $i<@version_urls; $i++) {
+        my $st = $sectnode->{Status}[$i];
+        $st //= '-';
+        o('<td>');
+        my $end_a = '';
+        if ($sectnode->{Key} eq 'release-support--xen-version') {
+            o(sprintf '<a href="%s">', $version_urls[$i]);
+            $end_a = '</a>';
+        }
+        o(escapeHTML($st));
+        if ($sectnode->{HasText}[$i] && $sectnode->{Anchor}) {
+            o(sprintf '<a href="%s#%s">[*]</a>',
+              $version_urls[$i], $sectnode->{Anchor});
+        }
+        o($end_a);
+        o('</td>');
+    }
+    o("</tr>\n");
+}      
+
+sub write_output_sectlist ($);
+sub write_output_sectlist ($) {
+    my ($sectlist) = @_;
+    foreach my $key (keys %$sectlist) {
+        my $sectnode = $sectlist->{$key};
+        push @pending_headings, $sectnode;
+        write_output_row $sectnode if $sectnode->{Status};
+        write_output_sectlist $sectnode->{Children};
+    }
+}
+
+sub write_output () {
+    o('<table rules="all">');
+    write_output_sectlist $toplevel_sectlist;
+    o('</table>');
+}
+
+#---------- main program ----------
+
+open DEBUG, '>', '/dev/null' or die $!;
+if (@ARGV && $ARGV[0] eq '-D') {
+    shift @ARGV;
+    open DEBUG, '>&2' or die $!;
+}
+
+die unless @ARGV;
+die if $ARGV[0] =~ m/^-/;
+die if @ARGV % 2;
+
+read_inputs();
+
+#use Data::Dumper;
+#print DEBUG Dumper($toplevel_sectlist);
+
+analyse_reprocess();
+# Now Headline is in HTML
+
+count_rows_sectlist($toplevel_sectlist);
+
+#use Data::Dumper;
+print DEBUG Dumper($toplevel_sectlist);
+
+write_output();
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 8/9] docs: Provide parse-support-md

[Juergen Gross]

On 11/04/18 17:35, Ian Jackson wrote:
> This utility reads json format pandoc output, from parsing one or more
> SUPPORT.md files, and generates an HTML table element containing the
> principal version and feature information.
> 
> This is rather hairier than I anticipated when I started out; hence
> the 400-odd-line Perl script.
> 
> Machinery to assemble the appropriate inputs for parse-support-md
> will be in the next commit.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Release-acked-by: Juergen Gross <jgross@suse.com>


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 9/9] docs: Provide support-matrix-generate, to generate a support matrix in HTML

[Ian Jackson]

This archaeology script:
 - figures out what the current and previous Xen versions were
 - looks for appropriate git branches for them
 - finds SUPPORT.md for each one
 - feeds its findings to parse-support-md

We do not intend to integrate this into docs/Makefile, because it
relies on the git history.  Instead, we will take the rune provided in
the head comment and paste a variant of it into an appropriate cronjob
on xenbits.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 .gitignore                         |   1 +
 docs/misc/support-matrix-head.html |  41 +++++++++
 docs/support-matrix-generate       | 180 +++++++++++++++++++++++++++++++++++++
 3 files changed, 222 insertions(+)
 create mode 100644 docs/misc/support-matrix-head.html
 create mode 100755 docs/support-matrix-generate

diff --git a/.gitignore b/.gitignore
index cd57530..7004349 100644
--- a/.gitignore
+++ b/.gitignore
@@ -43,6 +43,7 @@ config/Paths.mk
 
 build-*
 dist/*
+docs/tmp.*
 docs/html/
 docs/man/xl.cfg.pod.5
 docs/man/xl.pod.1
diff --git a/docs/misc/support-matrix-head.html b/docs/misc/support-matrix-head.html
new file mode 100644
index 0000000..7cc2776
--- /dev/null
+++ b/docs/misc/support-matrix-head.html
@@ -0,0 +1,41 @@
+<html>
+  <head>
+    <title>Xen versions and feature support matrix</title>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+  </head>
+  <body>
+    <h1>Xen versions and features support matrix</h1>
+
+    This table summarises the support status of Xen releases,
+    and of individual features within each release.
+
+    <h2>Important notes</h2>
+
+    The matrix is extracted automatically
+    from the formal support status documents
+    in each Xen release.
+    The full formal support status document
+    is linked to from the column heading for each version.
+
+    <p>
+    The individual entries are summaries;
+    where a specific entry has more information in the full
+    document a link, denoted [*], is provided.
+    The statuses Supported, Experimental, and so on,
+    are likewise defined in the full document.
+
+    <p>
+    Sometimes the same feature, or a similar feature,
+    is named differently
+    in the documentation for different releases.
+    In such cases the table will show it as
+    two separate features,
+    with a discontinuity in support,
+    even though support may have been continuous.
+
+    <p>
+    The support status of versions earlier than listed here
+    is documented
+    <a href="https://wiki.xenproject.org/wiki/Xen_Project_Release_Features">on the wiki</a>.
+
+    <h2>Support Matrix</h2>
diff --git a/docs/support-matrix-generate b/docs/support-matrix-generate
new file mode 100755
index 0000000..17630c6
--- /dev/null
+++ b/docs/support-matrix-generate
@@ -0,0 +1,180 @@
+#!/bin/bash
+
+# usage:
+#   cd xen.git
+#   docs/support-matrix-generate                                    \
+#       refs/remotes/origin/master                                  \
+#           https://xenbits.xen.org/docs/unstable/SUPPORT.html      \
+#       refs/remotes/origin/stable-NN                               \
+#           https://xenbits.xen.org/docs/NN-testing/SUPPORT.html    \
+#
+# NN is a *literal* in the above rune!  It will be substituted with
+# the appropriate version number.
+#
+# The idea is that we use staging's version of this script, and it
+# looks into the git history and various git remote tracking refs to
+# find the various versions of SUPPORT.md.
+#
+# The arguments specify the git refs to look in, and also the URLs for
+# the SUPPORT.html (which are needed so that we can make
+# cross-reference links).  We provide the ref and url (i) for unstable
+# (ii) in template form for all previous versions.
+
+# Algorithm:
+#
+# We start with `refs/remotes/origin/master' and process its
+# SUPPORT.md into json.
+#
+# Then we try to find the next previous revision.  This is done by
+# extracting the current version number from xen/Makefile.  (We make
+# some slight assumption about how xen/Makefile's xenversion target
+# works, because we want to be able to do this without checking out
+# the whole tree for the version in question.)  Then we use git log on
+# xen/Makefile to try to find a commit where the version changed.
+# This gives us the previous version number, NN.
+#
+# That is substituted into the `refs/remotes/origin/stable-NN'
+# argument to get the tip of the relevant branch.  That in turns
+# contains another SUPPORT.md.  We keep going until either the ref
+# itself is missing, or we get to a ref with no SUPPORT.md.
+
+set -e
+set -o posix
+set -o pipefail
+
+fail () { echo >&2 "$0: $1"; exit 12; }
+
+case "$#.$1" in
+    4.[^-]*) ;;
+    *) fail 'bad usage' ;;
+esac
+
+current_ref=$1
+current_url=$2
+pattern_ref=$3
+pattern_url=$4
+
+tmp_prefix="docs/tmp.support-matrix"
+tmp_mdfile="$tmp_prefix.md"
+tmp_revisions="$tmp_prefix.revisions.html"
+
+versionfile=xen/Makefile
+tmp_versionfile="$tmp_prefix.xen.make"
+
+args=()
+
+cat docs/misc/support-matrix-head.html
+
+debug () {
+    echo "<!-- $* -->"
+}
+
+select_commitish () {
+    commitish=$1
+    debug "select_commitish $commitish"
+    obj="$(printf "%s:SUPPORT.md" "$commitish")"
+    exists="$(printf "%s" "$obj" | git cat-file --batch-check)"
+    >"$tmp_mdfile"
+    case "$exists" in
+        *' missing')
+            rm "$tmp_mdfile"
+            ;;
+        *' blob '*)
+            git cat-file blob "$obj" >"$tmp_mdfile"
+            ;;
+        *) fail "?? $current_url $exists ?";;
+    esac
+}
+
+commitish_version () {
+    case "$commitish" in
+        refs/*)
+            # this is how to find out if a ref exists
+            local gfer=$(git for-each-ref "[r]${commitish#r}")
+            if [ "x$gfer" = x ]; then return; fi
+            ;;
+    esac
+
+    git cat-file blob "$commitish:$versionfile" >"$tmp_versionfile"
+    version=$(make --no-print-directory -C docs \
+                   -f "${tmp_versionfile#docs/}" xenversion)
+    case "$version" in
+        *.*.*) version="${version%.*}" ;;
+    esac
+    printf "%s\n" "${version%%-*}"
+}
+
+exec 4>"$tmp_revisions"
+
+while true; do
+    select_commitish "$current_ref"
+    current_version=$(commitish_version)
+    debug "current_version=$current_version"
+
+    if ! [ -e "$tmp_mdfile" ]; then break; fi
+
+    cat >&4 <<END
+<tr>
+<td align="center">$current_version</td>
+END
+    git >&4 log -n1 --pretty=tformat:'
+<td><code>%ci</code></td>
+<td><code>%H</code></td>
+' "$current_ref"
+    cat >&4 <<END
+</tr>
+END
+
+    current_jsonfile="$tmp_prefix.$current_version.json"
+    pandoc -t json <"$tmp_mdfile" >"$current_jsonfile"
+
+    args+=("$current_jsonfile" "$current_url")
+
+    # find previous version
+    search_commit="$current_ref"
+    while true; do
+        search_commit=$(git-log --pretty=format:%H -n1 \
+                        -G 'XEN.*VERSION' $search_commit -- $versionfile)
+        if ! [ "$search_commit" ]; then search_version=''; break; fi
+
+        search_commit="$search_commit~"
+        select_commitish "$search_commit"
+        search_version=$(commitish_version)
+        debug "search_version=$search_version"
+        if [ "x$search_version" != "x$current_version" ]; then break; fi
+    done
+
+    if [ "x$search_version" = x ]; then break; fi
+
+    # have found the previous version
+    current_ref=${pattern_ref/NN/$search_version}
+    current_url=${pattern_url/NN/$search_version}
+done
+
+debug "${args[*]}"
+docs/parse-support-md "${args[@]}"
+
+cat <<END
+    <h2>Source materials</h2>
+      Generated from xen.git
+      by docs/support-matrix-generate and docs/parse-support-md.
+      <p>
+      Input revisions of SUPPORT.md used:
+    <table>
+    <tr>
+      <th>Version</th>
+      <th align="left">Commit date</th>
+      <th align="left">Git commit</th>
+END
+
+cat "$tmp_revisions"
+
+generated=$(TZ=UTC date --iso-8601=minutes)
+
+cat <<END
+    </table>
+    <p>
+    Last checked/updated/regenerated: ${generated/T/ }
+  </body>
+</html>
+END
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 9/9] docs: Provide support-matrix-generate, to generate a support matrix in HTML

[Juergen Gross]

On 11/04/18 17:35, Ian Jackson wrote:
> This archaeology script:
>  - figures out what the current and previous Xen versions were
>  - looks for appropriate git branches for them
>  - finds SUPPORT.md for each one
>  - feeds its findings to parse-support-md
> 
> We do not intend to integrate this into docs/Makefile, because it
> relies on the git history.  Instead, we will take the rune provided in
> the head comment and paste a variant of it into an appropriate cronjob
> on xenbits.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Release-acked-by: Juergen Gross <jgross@suse.com>


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-4.11 v2 0/9] Provide support matrix generator

[Lars Kurth]

On 11/04/2018, 16:36, "Ian Jackson" <ian.jackson@eu.citrix.com> wrote:

    This series provides code to generate a feature support matrix, to
    replace the one on the wiki.  You can see an example of the output
    here:
      http://xenbits.xen.org/people/iwj/2018/support-matrix-example-v2/t.html
    There is also an accompanying SUPPORT.html to make the links for 4.11
    work.
    
    Patches 1-3 fix syntax trouble in SUPPORT.md (and commit us to using
    pandoc format for it).  Patches 4-7 arrange to generate a SUPPORT.html
    at the toplevel of the generated docs.  These should all be backported
    to 4.10.
    
    Patches 8 and 9 are the support matrix generator.  They are not hooked
    into any of the xen.git makefiles but it seemed best to have them here
    in xen.git/docs rather than floating about in some ad hoc crontab
    repo.
    
      ar 1/9 SUPPORT.md: Syntax: Fix some bullet lists
      ar 2/9 SUPPORT.md: Syntax: Fix a typo "States"
     m r 3/9 SUPPORT.md: Syntax: Provide a title rather than a
       r 4/9 docs/gen-html-index: Extract titles from HTML documents
       r 5/9 docs/gen-html-index: Support documents at the toplevel
       r 6/9 docs/Makefile: Introduce GENERATE_PANDOC_RULE_RAW
       r 7/9 docs/Makefile: Format SUPPORT.md into the toplevel
     +   8/9 docs: Provide parse-support-md
     +   9/9 docs: Provide support-matrix-generate, to generate ...
    
     a = acked (by a tools or REST maintainer)
     r = release-acked for 4.11
     + = new patch
     m = modified commit message
    
    I think if the output is tolerably accurate, it would be valuable to
    get this set up as a cron job ASAP.  Stylistic and formatting
    improvements etc. can be done on top.
    
    My plan is that the cron job will use the matrix generator from
    xen.git#master.
    
    For my reference, this was generated with
       docs/support-matrix-generate HEAD SUPPORT.html refs/heads/wip.support-stmt-NN https://xenbits.xen.org/docs/NN-testing/SUPPORT.html >t.html
      rsync -vP t.html SUPPORT.html xenbits:public_html/2018/support-matrix-example-v1/

Acked-by: Lars Kurth <lars.kurth@citrix.com>

Ian, this looks good. 

Only a small comment: for table entries such as
x86/PCI Device Passthrough	x86 PV		Supported, with caveats	Supported, with caveats
x86 HVM					Supported, with caveats[*]	Supported, with caveats[*]

The footnote [*] should probably apply to all items in the tree, not just the last one. Also, the 4.10 link didn't work (probably because the SUPPORT.md format changes need backporting)

Regards
Lars

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH for-4.10 0/9] SUPPORT.md backports to support matrix generation

[Ian Jackson]

This is a backport of my two recent series to fix bugs in SUPPORT.md,
format it as part of the published docs, and use relative position of
the `Status' stanza compared to descriptive text to indicate whether
the text is a caveat that deserves a footnote.

Most of this is uncontroversial and I have run the build and checked
that the resulting index.html is good.

It would be good if someone could review particularly the way I
resolved the conflicts in the big patch to SUPPORT.md, ie in
  [PATCH 8/9] SUPPORT.md: Move descriptions up before Status info

The result for 4.10's
SUPPORT.md is here:
  https://xenbits.xen.org/people/iwj/2018/support-matrix-example-C/SUPPORT.html

I have also checked that (with the patch I am about to send for
staging's parse-support-md) that this generates a sensible-looking
support matrix.  The resulting support matrix can be found here:
  https://xenbits.xen.org/people/iwj/2018/support-matrix-example-C/t.html
(The hyperlink references for staging are to the live xenbits version;
the references for 4.10 are to the example output file mentioned above.)

I have squashed the fix to not depend on HTML::TreeBuilder::XPath into
the patch that introduces the dependency.

For my reference, this was made as follows:
  docs/support-matrix-generate -D HEAD https://xenbits.xen.org/docs/unstable-staging/SUPPORT.html refs/heads/wip.support-stmt-NN-2 SUPPORT.html 2>&1 >docs/html/t.html |less
  rsync -LvP docs/html/{t,SUPPORT}.html xenbits:public_html/2018/support-matrix-example-C

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 5/9] docs/gen-html-index: Support documents at the toplevel

[Ian Jackson]

There are none yet.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Release-acked-by: Juergen Gross <jgross@suse.com>
Acked-by: Lars Kurth <lars.kurth@citrix.com>
(cherry picked from commit 1e4a834a8f5d970e68cff6d9c16710194bc46537)
---
 docs/gen-html-index | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/docs/gen-html-index b/docs/gen-html-index
index 410674e..4fad6db 100644
--- a/docs/gen-html-index
+++ b/docs/gen-html-index
@@ -140,6 +140,10 @@ sub dirs($)
     return @dirs;
 }
 
+foreach my $of (grep { !m{/} } @docs) {
+    $top .= make_link($of,'');
+}
+
 foreach my $od (sort { $a cmp $b } uniq map { dirs($_) } @docs) {
     my @d = (grep /^\Q$od\E/, @docs);
     if ( @d == 1 and $d[0] eq "$od/index.html" )
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel