[PATCH 0/3] docs: User oriented documentation

[PATCH 0/3] docs: User oriented documentation

[Andrew Cooper]

This is a project I've been musing over for a long time now, to try and
address Xen's almost complete absense of documentation.

This series, plus some other in-progress conversion of the command line doc,
is available to view at:

  https://andrewcoop-xen.readthedocs.io/en/latest/

This is read-the-docs's automatic CI build of documentation from a branch on
gitlab.  Observe that the docs don't look like they are out of the 90's, and
are automatically translated into PDF and ePUB format as well.

In due course I'll see about updating xenbits.xen.org/docs to render this as
well, but I don't have sufficient tuits at the moment.

Andrew Cooper (3):
  docs/sphinx: Skeleton setup
  docs/rst: Use pandoc to render ReStructuredText
  docs/admin-guide: Boot time microcode loading

 docs/Makefile                          |  24 +++-
 docs/admin-guide/index.rst             |   5 +
 docs/admin-guide/microcode-loading.rst | 103 ++++++++++++++++++
 docs/conf.py                           | 193 +++++++++++++++++++++++++++++++++
 docs/index.rst                         |  10 ++
 docs/sphinx/requirements.txt           |   3 +
 6 files changed, 336 insertions(+), 2 deletions(-)
 create mode 100644 docs/admin-guide/index.rst
 create mode 100644 docs/admin-guide/microcode-loading.rst
 create mode 100644 docs/conf.py
 create mode 100644 docs/index.rst
 create mode 100644 docs/sphinx/requirements.txt

-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 1/3] docs/sphinx: Skeleton setup

[Andrew Cooper]

Sphinx is a documentation system, which is popular for technical writing.  It
uses ReStructuredText as its markup syntax, and is designed for whole-project
documentation, rather than the misc assortment of individual files that we
currently have.

This is a skeleton setup which just enough infrastructure to render an empty
set of pages.  It will become better integrated into Xen's docs system when it
becomes less WIP.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: George Dunlap <George.Dunlap@eu.citrix.com>
CC: Ian Jackson <ian.jackson@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Wei Liu <wei.liu2@citrix.com>
CC: Julien Grall <julien.grall@arm.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Lars Kurth <lars.kurth@citrix.com>
---
 docs/Makefile                |  12 +++
 docs/conf.py                 | 193 +++++++++++++++++++++++++++++++++++++++++++
 docs/index.rst               |   2 +
 docs/sphinx/requirements.txt |   3 +
 4 files changed, 210 insertions(+)
 create mode 100644 docs/conf.py
 create mode 100644 docs/index.rst
 create mode 100644 docs/sphinx/requirements.txt

diff --git a/docs/Makefile b/docs/Makefile
index 44aebf0..eda4bd8 100644
--- a/docs/Makefile
+++ b/docs/Makefile
@@ -37,6 +37,18 @@ all: build
 .PHONY: build
 build: html txt pdf man-pages figs
 
+# WIP Sphinx/RST documentation.  Use virtualenv to get a suitable environment
+#
+# xen.git$ virtualenv venv-docs
+# xen.git$ . venv-docs/bin/activate
+# (venv-docs)xen.git$ pip install -r docs/sphinx/requirements.txt
+# (venv-docs)xen.git$ make -C docs/ sphinx-html
+# (venv-docs)xen.git$ deactivate
+# xen.git$
+.PHONY: sphinx-html
+sphinx-html:
+	sphinx-build -b html . sphinx/html
+
 .PHONY: html
 html: $(DOC_HTML) html/index.html
 
diff --git a/docs/conf.py b/docs/conf.py
new file mode 100644
index 0000000..73b7b9b
--- /dev/null
+++ b/docs/conf.py
@@ -0,0 +1,193 @@
+# -*- coding: utf-8 -*-
+#
+# Configuration file for the Sphinx documentation builder.
+#
+# This file does only contain a selection of the most common options. For a
+# full list see the documentation:
+# http://www.sphinx-doc.org/en/master/config
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+# import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
+
+
+# -- Project information -----------------------------------------------------
+
+project = u'Xen'
+copyright = u'2019, The Xen development community'
+author = u'The Xen development community'
+
+# Pull the Xen version straight out of the Makefile
+try:
+    xen_ver = xen_subver = xen_extra = None
+
+    for line in open(u"../xen/Makefile"):
+        if line.startswith(u"export XEN_VERSION"):
+            xen_ver = line.split(u"=")[1].strip()
+        elif line.startswith(u"export XEN_SUBVERSION"):
+            xen_subver = line.split(u"=")[1].strip()
+        elif line.startswith(u"export XEN_EXTRAVERSION"):
+            xen_extra = line.split(u"=")[1].split(u"$", 1)[0].strip()
+except:
+    pass
+finally:
+    if xen_ver and xen_subver and xen_extra:
+        version = xen_ver + u"." + xen_subver
+        release = version + xen_extra
+    else:
+        version = release = u"unknown version"
+
+# -- General configuration ---------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+#
+needs_sphinx = '1.4'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = []
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+#
+# source_suffix = ['.rst', '.md']
+source_suffix = '.rst'
+
+# The master toctree document.
+master_doc = 'index'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = [u'sphinx/output', 'Thumbs.db', '.DS_Store']
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = None
+
+primary_domain = 'c'
+highlight_language = 'none'
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+try:
+    import sphinx_rtd_theme
+    html_theme = 'sphinx_rtd_theme'
+    html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
+except ImportError:
+    sys.stderr.write('Warning: The Sphinx \'sphinx_rtd_theme\' HTML theme was not found. Make sure you have the theme installed to produce pretty HTML output. Falling back to the default theme.\n')
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#
+# html_theme_options = {}
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = []
+
+# Custom sidebar templates, must be a dictionary that maps document names
+# to template names.
+#
+# The default sidebars (for documents that don't match any pattern) are
+# defined by theme itself.  Builtin themes are using these templates by
+# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
+# 'searchbox.html']``.
+#
+# html_sidebars = {}
+
+
+# -- Options for HTMLHelp output ---------------------------------------------
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'Xendoc'
+
+
+# -- Options for LaTeX output ------------------------------------------------
+
+latex_elements = {
+    # The paper size ('letterpaper' or 'a4paper').
+    #
+    # 'papersize': 'letterpaper',
+
+    # The font size ('10pt', '11pt' or '12pt').
+    #
+    # 'pointsize': '10pt',
+
+    # Additional stuff for the LaTeX preamble.
+    #
+    # 'preamble': '',
+
+    # Latex figure (float) alignment
+    #
+    # 'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+#  author, documentclass [howto, manual, or own class]).
+latex_documents = [
+    (master_doc, 'Xen.tex', u'Xen Documentation',
+     u'The Xen development community', 'manual'),
+]
+
+
+# -- Options for manual page output ------------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    (master_doc, 'xen', u'Xen Documentation',
+     [author], 1)
+]
+
+
+# -- Options for Texinfo output ----------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+    (master_doc, 'Xen', u'Xen Documentation',
+     author, 'Xen', 'One line description of project.',
+     'Miscellaneous'),
+]
+
+
+# -- Options for Epub output -------------------------------------------------
+
+# Bibliographic Dublin Core info.
+epub_title = project
+
+# The unique identifier of the text. This can be a ISBN number
+# or the project homepage.
+#
+# epub_identifier = ''
+
+# A unique identification for the text.
+#
+# epub_uid = ''
+
+# A list of files that should not be packed into the epub file.
+epub_exclude_files = ['search.html']
diff --git a/docs/index.rst b/docs/index.rst
new file mode 100644
index 0000000..732ebe0
--- /dev/null
+++ b/docs/index.rst
@@ -0,0 +1,2 @@
+The Xen Hypervisor documentation
+================================
diff --git a/docs/sphinx/requirements.txt b/docs/sphinx/requirements.txt
new file mode 100644
index 0000000..fa1393a
--- /dev/null
+++ b/docs/sphinx/requirements.txt
@@ -0,0 +1,3 @@
+docutils == 0.12
+Sphinx == 1.4.9
+sphinx_rtd_theme
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 2/3] docs/rst: Use pandoc to render ReStructuredText

[Andrew Cooper]

Sphinx uses ReStructuredText as its markup format.  Although missing the
project wide integration, individual *.rst files can be rendered by pandoc to
suppliement our existing ad-hoc documentation.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: George Dunlap <George.Dunlap@eu.citrix.com>
CC: Ian Jackson <ian.jackson@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Wei Liu <wei.liu2@citrix.com>
CC: Julien Grall <julien.grall@arm.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Lars Kurth <lars.kurth@citrix.com>
---
 docs/Makefile | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/docs/Makefile b/docs/Makefile
index eda4bd8..0c1228c 100644
--- a/docs/Makefile
+++ b/docs/Makefile
@@ -11,6 +11,8 @@ MAN_SECTIONS    := 1 5 7 8
 # Documentation sources to build
 MAN-SRC-y := $(sort $(basename $(wildcard man/*.pod man/*.pandoc)))
 
+RST-SRC-y := $(sort $(filter-out %index.rst,$(shell find * -type f -name '*.rst' -print)))
+
 TXTSRC-y := $(sort $(shell find misc -name '*.txt' -print))
 
 PANDOCSRC-y := $(sort $(shell find designs/ features/ misc/ process/ specs/ -name '*.pandoc' -print))
@@ -22,13 +24,16 @@ $(foreach i,$(MAN_SECTIONS), \
 
 DOC_HTML := html/SUPPORT.html \
             $(patsubst %.pandoc,html/%.html,$(PANDOCSRC-y)) \
+            $(patsubst %.rst,html/%.html,$(RST-SRC-y)) \
             $(patsubst %,html/%.html,$(MAN-SRC-y)) \
             $(patsubst %.txt,html/%.txt,$(TXTSRC-y)) \
             $(patsubst %,html/hypercall/%/index.html,$(DOC_ARCHES))
 DOC_TXT  := $(patsubst %.txt,txt/%.txt,$(TXTSRC-y)) \
             $(patsubst %.pandoc,txt/%.txt,$(PANDOCSRC-y)) \
+            $(patsubst %.rst,txt/%.txt,$(RST-SRC-y)) \
             $(patsubst %,txt/%.txt,$(MAN-SRC-y))
-DOC_PDF  := $(patsubst %.pandoc,pdf/%.pdf,$(PANDOCSRC-y))
+DOC_PDF  := $(patsubst %.pandoc,pdf/%.pdf,$(PANDOCSRC-y)) \
+            $(patsubst %.rst,pdf/%.pdf,$(RST-SRC-y))
 
 # Top level build targets
 .PHONY: all
@@ -71,7 +76,7 @@ clean: clean-man-pages
 	$(MAKE) -C figs clean
 	rm -rf .word_count *.aux *.dvi *.bbl *.blg *.glo *.idx *~
 	rm -rf *.ilg *.log *.ind *.toc *.bak *.tmp core
-	rm -rf html txt pdf
+	rm -rf html txt pdf sphinx/html
 
 .PHONY: distclean
 distclean: clean
@@ -232,8 +237,11 @@ define GENERATE_PANDOC_RULE
 $(call GENERATE_PANDOC_RULE_RAW,$(1)/%.$(1),%.$(2))
 endef
 $(eval $(call GENERATE_PANDOC_RULE,pdf,pandoc))   # pdf/%.pdf: %.pandoc
+$(eval $(call GENERATE_PANDOC_RULE,pdf,rst))      # pdf/%.pdf: %.rst
 $(eval $(call GENERATE_PANDOC_RULE,txt,pandoc))   # txt/%.txt: %.pandoc
+$(eval $(call GENERATE_PANDOC_RULE,txt,rst))      # txt/%.txt: %.rst
 $(eval $(call GENERATE_PANDOC_RULE,html,pandoc))  # html/%.html: %.pandoc
+$(eval $(call GENERATE_PANDOC_RULE,html,rst))     # html/%.html: %.rst
 
 $(eval $(call GENERATE_PANDOC_RULE_RAW,html/SUPPORT.html,$(XEN_ROOT)/SUPPORT.md))
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 3/3] docs/admin-guide: Boot time microcode loading

[Andrew Cooper]

Recent discussion on xen-devel has demonstrated that Xen existing microcode
loading support isn't adequately documented.  Take the opportunity to address
this, and start some end-user focused documentation.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: George Dunlap <George.Dunlap@eu.citrix.com>
CC: Ian Jackson <ian.jackson@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Tim Deegan <tim@xen.org>
CC: Wei Liu <wei.liu2@citrix.com>
CC: Julien Grall <julien.grall@arm.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Lars Kurth <lars.kurth@citrix.com>

I've deliberately omitted runtime microcode loading at this point, because it
is currently rather broken and its implementation is in flux.  This document
can be extended in due course.
---
 docs/admin-guide/index.rst             |   5 ++
 docs/admin-guide/microcode-loading.rst | 103 +++++++++++++++++++++++++++++++++
 docs/index.rst                         |   8 +++
 3 files changed, 116 insertions(+)
 create mode 100644 docs/admin-guide/index.rst
 create mode 100644 docs/admin-guide/microcode-loading.rst

diff --git a/docs/admin-guide/index.rst b/docs/admin-guide/index.rst
new file mode 100644
index 0000000..4d9bcb4
--- /dev/null
+++ b/docs/admin-guide/index.rst
@@ -0,0 +1,5 @@
+User documentation
+==================
+
+.. toctree::
+   microcode-loading
diff --git a/docs/admin-guide/microcode-loading.rst b/docs/admin-guide/microcode-loading.rst
new file mode 100644
index 0000000..58393b8
--- /dev/null
+++ b/docs/admin-guide/microcode-loading.rst
@@ -0,0 +1,103 @@
+Microcode Loading
+=================
+
+Like many other pieces of hardware, CPUs themselves have errata which are
+discovered after shipping, and need to be addressed in the field.  Microcode
+can be considered as firmware for the processor, and new microcode is
+published as needed by the processor vendors.
+
+Microcode is included as part of the system firmware by an OEM, and a system
+firmware update is the preferred way of obtaining updated microcode.  However,
+this is often not the most expedient way to get updated microcode, so Xen
+supports loading microcode itself.
+
+Distros typically package microcode updates for users, and may provide hooks
+to cause microcode to be automatically loaded at boot time.  Consult your dom0
+distro guidance for microcode loading.
+
+Microcode can make almost arbitrary changes to the processor, including to
+software visible features.  This includes removing features (e.g. the Haswell
+TSX errata which necessitated disabling the feature entirely), or the addition
+of brand new features (e.g. the Spectre v2 controls to work around speculative
+execution vulnerabilities).
+
+
+Boot time microcode loading
+---------------------------
+
+Where possible, microcode should be loaded at boot time.  This allows the CPU
+to be updated to its eventual configuration before Xen starts making setup
+decisions based on the visible features.
+
+Xen will report during boot if it performed a microcode update.  e.g.::
+
+  [root@host ~]# xl dmesg | grep microcode
+  (XEN) microcode: CPU0 updated from revision 0x1a to 0x25, date = 2018-04-02
+  (XEN) microcode: CPU2 updated from revision 0x1a to 0x25, date = 2018-04-02
+  (XEN) microcode: CPU4 updated from revision 0x1a to 0x25, date = 2018-04-02
+  (XEN) microcode: CPU6 updated from revision 0x1a to 0x25, date = 2018-04-02
+
+The exact details printed are system and microcode specific.  After boot, the
+current microcode version can obtained from with dom0.  e.g.::
+
+  [root@host ~]# head /proc/cpuinfo
+  processor    : 0
+  vendor_id    : GenuineIntel
+  cpu family   : 6
+  model        : 60
+  model name   : Intel(R) Xeon(R) CPU E3-1240 v3 @ 3.40GHz
+  stepping     : 3
+  microcode    : 0x25
+  cpu MHz      : 3392.148
+  cache size   : 8192 KB
+  physical id  : 0
+
+
+Loading microcode from a single file
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Xen handles microcode blobs in the binary form shipped by vendors, which is
+also the form the processor uses.  This binary form contains header
+information which Xen and various userspace tools can use to identify the
+correct blob for a specific CPU.
+
+Tools such as dracut will identify the correct blob for the current CPU, which
+will be a few kilobytes, for minimal overhead during boot.
+
+Additionally, Xen is capable of handling a number of blobs concatenated
+together, and will locate the appropriate blob based on the header
+information.
+
+This option is less efficient during boot, but may be preferred in situations
+where the exact CPU details aren't known ahead of booting (e.g. install
+media).
+
+The file containing the blob(s) needs to be accessible to Xen as early as
+possible.
+
+* For multiboot/multiboot2 boots, this is achieved by loading the blob as a
+  multiboot module.  The ``ucode=$num`` command line option can be used to
+  identify which multiboot module contains the microcode, including negative
+  indexing to count from the end.
+
+* For EFI boots, there isn't really a concept of modules.  A microcode file
+  can be specified in the EFI configuration file with ``ucode=$file``.  Use of
+  this mechanism will override any ``ucode=`` settings on the command line.
+
+
+Loading microcode from a Linux initrd
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For systems using a Linux based dom0, it usually suffices to install the
+appropriate distro package, and add ``ucode=scan`` to Xen's command line.
+
+Xen is compatible with the Linux initrd microcode protocol.  The initrd is
+expected to be generated with an uncompressed CPIO fragment at the beginning
+which contains contains one of these two files::
+
+  kernel/x86/microcode/GenuineIntel.bin
+  kernel/x86/microcode/AuthenticAMD.bin
+
+The ``ucode=scan`` command line option will cause Xen to search through all
+modules to find any CPIO archives, and look up the applicable file in the CPIO
+archive.
diff --git a/docs/index.rst b/docs/index.rst
index 732ebe0..ab3d0e0 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -1,2 +1,10 @@
 The Xen Hypervisor documentation
 ================================
+
+User documentation
+------------------
+
+.. toctree::
+   :maxdepth: 2
+
+   admin-guide/index
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 3/3] docs/admin-guide: Boot time microcode loading

[Jan Beulich]

>>> On 19.03.19 at 17:20, <andrew.cooper3@citrix.com> wrote:
> Recent discussion on xen-devel has demonstrated that Xen existing microcode
> loading support isn't adequately documented.  Take the opportunity to address
> this, and start some end-user focused documentation.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Acked-by: Jan Beulich <jbeulich@suse.com>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 0/3] docs: User oriented documentation

[Wei Liu]

On Tue, Mar 19, 2019 at 04:20:04PM +0000, Andrew Cooper wrote:
> This is a project I've been musing over for a long time now, to try and
> address Xen's almost complete absense of documentation.
> 
> This series, plus some other in-progress conversion of the command line doc,
> is available to view at:
> 
>   https://andrewcoop-xen.readthedocs.io/en/latest/
> 
> This is read-the-docs's automatic CI build of documentation from a branch on
> gitlab.  Observe that the docs don't look like they are out of the 90's, and
> are automatically translated into PDF and ePUB format as well.
> 
> In due course I'll see about updating xenbits.xen.org/docs to render this as
> well, but I don't have sufficient tuits at the moment.
> 
> Andrew Cooper (3):
>   docs/sphinx: Skeleton setup
>   docs/rst: Use pandoc to render ReStructuredText
>   docs/admin-guide: Boot time microcode loading

I don't think these changes introduce new dependencies in the build.
Assuming my observation is correct:

Acked-by: Wei Liu <wei.liu2@citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 0/3] docs: User oriented documentation

[Andrew Cooper]

On 20/03/2019 11:20, Wei Liu wrote:
> On Tue, Mar 19, 2019 at 04:20:04PM +0000, Andrew Cooper wrote:
>> This is a project I've been musing over for a long time now, to try and
>> address Xen's almost complete absense of documentation.
>>
>> This series, plus some other in-progress conversion of the command line doc,
>> is available to view at:
>>
>>   https://andrewcoop-xen.readthedocs.io/en/latest/
>>
>> This is read-the-docs's automatic CI build of documentation from a branch on
>> gitlab.  Observe that the docs don't look like they are out of the 90's, and
>> are automatically translated into PDF and ePUB format as well.
>>
>> In due course I'll see about updating xenbits.xen.org/docs to render this as
>> well, but I don't have sufficient tuits at the moment.
>>
>> Andrew Cooper (3):
>>   docs/sphinx: Skeleton setup
>>   docs/rst: Use pandoc to render ReStructuredText
>>   docs/admin-guide: Boot time microcode loading
> I don't think these changes introduce new dependencies in the build.
> Assuming my observation is correct:
>
> Acked-by: Wei Liu <wei.liu2@citrix.com>

In the short term, no.  In due course, we'll need virtualenv and a bit
more integration for anyone wanting to build the sphinx tree locally,
but I'm fairly sure we get virtualenv automatically by already having
python as a build dependency.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 0/3] docs: User oriented documentation

[Wei Liu]

On Wed, Mar 20, 2019 at 11:23:57AM +0000, Andrew Cooper wrote:
> On 20/03/2019 11:20, Wei Liu wrote:
> > On Tue, Mar 19, 2019 at 04:20:04PM +0000, Andrew Cooper wrote:
> >> This is a project I've been musing over for a long time now, to try and
> >> address Xen's almost complete absense of documentation.
> >>
> >> This series, plus some other in-progress conversion of the command line doc,
> >> is available to view at:
> >>
> >>   https://andrewcoop-xen.readthedocs.io/en/latest/
> >>
> >> This is read-the-docs's automatic CI build of documentation from a branch on
> >> gitlab.  Observe that the docs don't look like they are out of the 90's, and
> >> are automatically translated into PDF and ePUB format as well.
> >>
> >> In due course I'll see about updating xenbits.xen.org/docs to render this as
> >> well, but I don't have sufficient tuits at the moment.
> >>
> >> Andrew Cooper (3):
> >>   docs/sphinx: Skeleton setup
> >>   docs/rst: Use pandoc to render ReStructuredText
> >>   docs/admin-guide: Boot time microcode loading
> > I don't think these changes introduce new dependencies in the build.
> > Assuming my observation is correct:
> >
> > Acked-by: Wei Liu <wei.liu2@citrix.com>
> 
> In the short term, no.  In due course, we'll need virtualenv and a bit
> more integration for anyone wanting to build the sphinx tree locally,
> but I'm fairly sure we get virtualenv automatically by already having
> python as a build dependency.

I don't think python depends on virtualenv -- it's the other way around
on Debian.

Using virtualenv can isolate build from host python, but that's it.  I
don't think virtualenv is a hard dependency . Distros already package
sphinx. 

Wei.

> 
> ~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 0/3] docs: User oriented documentation

[Andrew Cooper]

On 20/03/2019 11:28, Wei Liu wrote:
> On Wed, Mar 20, 2019 at 11:23:57AM +0000, Andrew Cooper wrote:
>> On 20/03/2019 11:20, Wei Liu wrote:
>>> On Tue, Mar 19, 2019 at 04:20:04PM +0000, Andrew Cooper wrote:
>>>> This is a project I've been musing over for a long time now, to try and
>>>> address Xen's almost complete absense of documentation.
>>>>
>>>> This series, plus some other in-progress conversion of the command line doc,
>>>> is available to view at:
>>>>
>>>>   https://andrewcoop-xen.readthedocs.io/en/latest/
>>>>
>>>> This is read-the-docs's automatic CI build of documentation from a branch on
>>>> gitlab.  Observe that the docs don't look like they are out of the 90's, and
>>>> are automatically translated into PDF and ePUB format as well.
>>>>
>>>> In due course I'll see about updating xenbits.xen.org/docs to render this as
>>>> well, but I don't have sufficient tuits at the moment.
>>>>
>>>> Andrew Cooper (3):
>>>>   docs/sphinx: Skeleton setup
>>>>   docs/rst: Use pandoc to render ReStructuredText
>>>>   docs/admin-guide: Boot time microcode loading
>>> I don't think these changes introduce new dependencies in the build.
>>> Assuming my observation is correct:
>>>
>>> Acked-by: Wei Liu <wei.liu2@citrix.com>
>> In the short term, no.  In due course, we'll need virtualenv and a bit
>> more integration for anyone wanting to build the sphinx tree locally,
>> but I'm fairly sure we get virtualenv automatically by already having
>> python as a build dependency.
> I don't think python depends on virtualenv -- it's the other way around
> on Debian.
>
> Using virtualenv can isolate build from host python, but that's it.  I
> don't think virtualenv is a hard dependency . Distros already package
> sphinx.

The point of using virtualenv is to get a known-compatible set of
dependencies.  See docs/sphinx/requirements.txt in patch 1.

Use of the distro-packaged versions of sphinx/rtd-theme/docutils may
work, but can be very hit-and-miss.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 0/3] docs: User oriented documentation

[Ian Jackson]

Andrew Cooper writes ("Re: [PATCH 0/3] docs: User oriented documentation"):
> The point of using virtualenv is to get a known-compatible set of
> dependencies. See docs/sphinx/requirements.txt in patch 1.

I don't think we need to have more curlbashware in the Xen build tree.
Sphinx 1.4.9 is what is needed according to the metadata in your patch
1.  It is in Debian stable.  Likewise your metadata says docutils 0.12
is needed and Debian stable has 0.13.1.

> Use of the distro-packaged versions of sphinx/rtd-theme/docutils may
> work, but can be very hit-and-miss.

On irc you wrote:

 11:39 <Diziet> Why on earth do we need [virtualenv] ?
 11:44 <andyhhp> in this case, because stretch has a known-buggy
                 combination of docutils and sphinx

That does not seem to be documented in your
docs/sphinx/requirements.txt.

Is the fix available in stretch-backports ?

I really don't want to install virtualenv on xenbits.  Nor do I want
anyone doing so ad-hoc in their filespace.  This is not an appropriate
approach to software distribution and management.  And yes, I know
that I am rejecting the underlying design principle of things like
virtualenv, NPM, etc.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 0/3] docs: User oriented documentation

[Andrew Cooper]

On 20/03/2019 11:49, Ian Jackson wrote:
> Andrew Cooper writes ("Re: [PATCH 0/3] docs: User oriented documentation"):
>> The point of using virtualenv is to get a known-compatible set of
>> dependencies. See docs/sphinx/requirements.txt in patch 1.
> I don't think we need to have more curlbashware in the Xen build tree.
> Sphinx 1.4.9 is what is needed according to the metadata in your patch
> 1.  It is in Debian stable.  Likewise your metadata says docutils 0.12
> is needed and Debian stable has 0.13.1.

That's perhaps all well and fine.  Debian Jessie doesn't have a
combination compatible with Xen's needs.

I haven't checked other distros.

>
>> Use of the distro-packaged versions of sphinx/rtd-theme/docutils may
>> work, but can be very hit-and-miss.
> On irc you wrote:
>
>  11:39 <Diziet> Why on earth do we need [virtualenv] ?
>  11:44 <andyhhp> in this case, because stretch has a known-buggy
>                  combination of docutils and sphinx
>
> That does not seem to be documented in your
> docs/sphinx/requirements.txt.
>
> Is the fix available in stretch-backports ?

It turns out that I mis-interpreted the compatibility note in
https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html# and
stretch is probably fine.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 0/3] docs: User oriented documentation

[Ian Jackson]

Andrew Cooper writes ("Re: [PATCH 0/3] docs: User oriented documentation"):
> It turns out that I mis-interpreted the compatibility note in
> https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html# and
> stretch is probably fine.

OK, good.

Well, I think we could say that to build this documentation you have
to have the relevant tools.  sphinx is certainly well-regarded and
this is probably a worthwhile upgrade.

What do the earlier tools do ?  Should we print some warning and
disable the build, or will they generate slightly-mangled output, or
what ?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 0/3] docs: User oriented documentation

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 1134 bytes --]

On 20/03/2019 16:43, Ian Jackson wrote:
> Andrew Cooper writes ("Re: [PATCH 0/3] docs: User oriented documentation"):
>> It turns out that I mis-interpreted the compatibility note in
>> https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html# and
>> stretch is probably fine.
> OK, good.
>
> Well, I think we could say that to build this documentation you have
> to have the relevant tools.  sphinx is certainly well-regarded and
> this is probably a worthwhile upgrade.
>
> What do the earlier tools do ?  Should we print some warning and
> disable the build, or will they generate slightly-mangled output, or
> what ?

conf.py has version check in it.

On Jessie with native packages, the result is:

andrewcoop@andrewcoop:/local/xen.git/xen$ make -C ../docs/ sphinx-html
make: Entering directory '/local/xen.git/docs'
sphinx-build -b html . sphinx/html
Running Sphinx v1.2.3

Sphinx version error:
This project needs at least Sphinx v1.4 and therefore cannot be built with this version.
Makefile:55: recipe for target 'sphinx-html' failed
make: *** [sphinx-html] Error 1
make: Leaving directory '/local/xen.git/docs'

~Andrew

[-- Attachment #1.2: Type: text/html, Size: 1891 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 0/3] docs: User oriented documentation

[Wei Liu]

On Wed, Mar 20, 2019 at 11:35:10AM +0000, Andrew Cooper wrote:
> On 20/03/2019 11:28, Wei Liu wrote:
> > On Wed, Mar 20, 2019 at 11:23:57AM +0000, Andrew Cooper wrote:
> >> On 20/03/2019 11:20, Wei Liu wrote:
> >>> On Tue, Mar 19, 2019 at 04:20:04PM +0000, Andrew Cooper wrote:
> >>>> This is a project I've been musing over for a long time now, to try and
> >>>> address Xen's almost complete absense of documentation.
> >>>>
> >>>> This series, plus some other in-progress conversion of the command line doc,
> >>>> is available to view at:
> >>>>
> >>>>   https://andrewcoop-xen.readthedocs.io/en/latest/
> >>>>
> >>>> This is read-the-docs's automatic CI build of documentation from a branch on
> >>>> gitlab.  Observe that the docs don't look like they are out of the 90's, and
> >>>> are automatically translated into PDF and ePUB format as well.
> >>>>
> >>>> In due course I'll see about updating xenbits.xen.org/docs to render this as
> >>>> well, but I don't have sufficient tuits at the moment.
> >>>>
> >>>> Andrew Cooper (3):
> >>>>   docs/sphinx: Skeleton setup
> >>>>   docs/rst: Use pandoc to render ReStructuredText
> >>>>   docs/admin-guide: Boot time microcode loading
> >>> I don't think these changes introduce new dependencies in the build.
> >>> Assuming my observation is correct:
> >>>
> >>> Acked-by: Wei Liu <wei.liu2@citrix.com>
> >> In the short term, no.  In due course, we'll need virtualenv and a bit
> >> more integration for anyone wanting to build the sphinx tree locally,
> >> but I'm fairly sure we get virtualenv automatically by already having
> >> python as a build dependency.
> > I don't think python depends on virtualenv -- it's the other way around
> > on Debian.
> >
> > Using virtualenv can isolate build from host python, but that's it.  I
> > don't think virtualenv is a hard dependency . Distros already package
> > sphinx.
> 
> The point of using virtualenv is to get a known-compatible set of
> dependencies.  See docs/sphinx/requirements.txt in patch 1.
> 
> Use of the distro-packaged versions of sphinx/rtd-theme/docutils may
> work, but can be very hit-and-miss.

They can be fixed in due course.

In any case, I don't think virtualenv is a hard requirement -- I was
thinking more along the line of developer / packager workflow. I think
it would be beneficial to have this series in tree, so people can
join the effort.

What to do on xenbits is another matter.

Wei.

> 
> ~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 0/3] docs: User oriented documentation

[George Dunlap]

On 3/20/19 12:02 PM, Wei Liu wrote:
> On Wed, Mar 20, 2019 at 11:35:10AM +0000, Andrew Cooper wrote:
>> On 20/03/2019 11:28, Wei Liu wrote:
>>> On Wed, Mar 20, 2019 at 11:23:57AM +0000, Andrew Cooper wrote:
>>>> On 20/03/2019 11:20, Wei Liu wrote:
>>>>> On Tue, Mar 19, 2019 at 04:20:04PM +0000, Andrew Cooper wrote:
>>>>>> This is a project I've been musing over for a long time now, to try and
>>>>>> address Xen's almost complete absense of documentation.
>>>>>>
>>>>>> This series, plus some other in-progress conversion of the command line doc,
>>>>>> is available to view at:
>>>>>>
>>>>>>   https://andrewcoop-xen.readthedocs.io/en/latest/
>>>>>>
>>>>>> This is read-the-docs's automatic CI build of documentation from a branch on
>>>>>> gitlab.  Observe that the docs don't look like they are out of the 90's, and
>>>>>> are automatically translated into PDF and ePUB format as well.
>>>>>>
>>>>>> In due course I'll see about updating xenbits.xen.org/docs to render this as
>>>>>> well, but I don't have sufficient tuits at the moment.
>>>>>>
>>>>>> Andrew Cooper (3):
>>>>>>   docs/sphinx: Skeleton setup
>>>>>>   docs/rst: Use pandoc to render ReStructuredText
>>>>>>   docs/admin-guide: Boot time microcode loading
>>>>> I don't think these changes introduce new dependencies in the build.
>>>>> Assuming my observation is correct:
>>>>>
>>>>> Acked-by: Wei Liu <wei.liu2@citrix.com>
>>>> In the short term, no.  In due course, we'll need virtualenv and a bit
>>>> more integration for anyone wanting to build the sphinx tree locally,
>>>> but I'm fairly sure we get virtualenv automatically by already having
>>>> python as a build dependency.
>>> I don't think python depends on virtualenv -- it's the other way around
>>> on Debian.
>>>
>>> Using virtualenv can isolate build from host python, but that's it.  I
>>> don't think virtualenv is a hard dependency . Distros already package
>>> sphinx.
>>
>> The point of using virtualenv is to get a known-compatible set of
>> dependencies.  See docs/sphinx/requirements.txt in patch 1.
>>
>> Use of the distro-packaged versions of sphinx/rtd-theme/docutils may
>> work, but can be very hit-and-miss.
> 
> They can be fixed in due course.
> 
> In any case, I don't think virtualenv is a hard requirement -- I was
> thinking more along the line of developer / packager workflow. I think
> it would be beneficial to have this series in tree, so people can
> join the effort.
> 
> What to do on xenbits is another matter.

It seems like setting up a suitable sphinx environment should be done by
the user (perhaps with some pointers).  Our make system shouldn't be
creating its own virtual environment.  Having some scripts in
automation/ would be useful, but it should be separate from the main doc
build.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 0/3] docs: User oriented documentation

[Andrew Cooper]

On 20/03/2019 16:49, George Dunlap wrote:
> On 3/20/19 12:02 PM, Wei Liu wrote:
>> On Wed, Mar 20, 2019 at 11:35:10AM +0000, Andrew Cooper wrote:
>>> On 20/03/2019 11:28, Wei Liu wrote:
>>>> On Wed, Mar 20, 2019 at 11:23:57AM +0000, Andrew Cooper wrote:
>>>>> On 20/03/2019 11:20, Wei Liu wrote:
>>>>>> On Tue, Mar 19, 2019 at 04:20:04PM +0000, Andrew Cooper wrote:
>>>>>>> This is a project I've been musing over for a long time now, to try and
>>>>>>> address Xen's almost complete absense of documentation.
>>>>>>>
>>>>>>> This series, plus some other in-progress conversion of the command line doc,
>>>>>>> is available to view at:
>>>>>>>
>>>>>>>   https://andrewcoop-xen.readthedocs.io/en/latest/
>>>>>>>
>>>>>>> This is read-the-docs's automatic CI build of documentation from a branch on
>>>>>>> gitlab.  Observe that the docs don't look like they are out of the 90's, and
>>>>>>> are automatically translated into PDF and ePUB format as well.
>>>>>>>
>>>>>>> In due course I'll see about updating xenbits.xen.org/docs to render this as
>>>>>>> well, but I don't have sufficient tuits at the moment.
>>>>>>>
>>>>>>> Andrew Cooper (3):
>>>>>>>   docs/sphinx: Skeleton setup
>>>>>>>   docs/rst: Use pandoc to render ReStructuredText
>>>>>>>   docs/admin-guide: Boot time microcode loading
>>>>>> I don't think these changes introduce new dependencies in the build.
>>>>>> Assuming my observation is correct:
>>>>>>
>>>>>> Acked-by: Wei Liu <wei.liu2@citrix.com>
>>>>> In the short term, no.  In due course, we'll need virtualenv and a bit
>>>>> more integration for anyone wanting to build the sphinx tree locally,
>>>>> but I'm fairly sure we get virtualenv automatically by already having
>>>>> python as a build dependency.
>>>> I don't think python depends on virtualenv -- it's the other way around
>>>> on Debian.
>>>>
>>>> Using virtualenv can isolate build from host python, but that's it.  I
>>>> don't think virtualenv is a hard dependency . Distros already package
>>>> sphinx.
>>> The point of using virtualenv is to get a known-compatible set of
>>> dependencies.  See docs/sphinx/requirements.txt in patch 1.
>>>
>>> Use of the distro-packaged versions of sphinx/rtd-theme/docutils may
>>> work, but can be very hit-and-miss.
>> They can be fixed in due course.
>>
>> In any case, I don't think virtualenv is a hard requirement -- I was
>> thinking more along the line of developer / packager workflow. I think
>> it would be beneficial to have this series in tree, so people can
>> join the effort.
>>
>> What to do on xenbits is another matter.
> It seems like setting up a suitable sphinx environment should be done by
> the user (perhaps with some pointers).  Our make system shouldn't be
> creating its own virtual environment.  Having some scripts in
> automation/ would be useful, but it should be separate from the main doc
> build.

This patch series very deliberately doesn't set up a virtualenv,
(because doing that automatically is not something I'm happy with Xen's
makefiles doing) but does have minimal instructions beside the build
target and one file committed into the tree to allow users to do it
easily.  See patch 1.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel