Possible changes to the project docs

Possible changes to the project docs

[Richard Purdie]

Since Robert was talking about future docs wide changes, I wanted to
put an idea out there. Its something a few of us have talked a little
about. At this point it is an idea, not a decision and something we're
thinking we need to explore and evaluate.

Basically, the thought is whether to move from docbook to sphinx.

Other projects like the kernel have switched and it could assist some
of our website/publishing challenges. It might also be easier for
people to contribute to and there are wider and more modern tools
supporting that format.

It'd be a big change, particularly if we consider we may need to
convert previous releases of the docs as well as the current version.
That said, if we were going to do this, the time could be now...

Cheers,

Richard

Re: [docs] Possible changes to the project docs

[Trevor Woerner]

On Wed 2020-04-15 @ 11:19:25 AM, Richard Purdie wrote:
> Basically, the thought is whether to move from docbook to sphinx.

This sounds like a good idea to me.

Prior to yesterday I had never looked into sphinx before. Skimming some
information about sphinx, it looks like it's flexible with regards to the
exact formatting you use for, for example, section headings. If that's true, I
believe we'll need a "coding standard" in place before proceeding. We wouldn't
want the BSP Manual to use:

	section 1
	^^^^^^^^^
	section 1.1
	===========
	section 1.1.1
	-------------

and the Reference Manual to use:

	section 1
	=========
	section 1.1
	***********
	section 1.1.1
	^^^^^^^^^^^^^

Best regards,
	Trevor

Re: [docs] Possible changes to the project docs

[Robert P. J. Day]

On Wed, 15 Apr 2020, Trevor Woerner wrote:

> On Wed 2020-04-15 @ 11:19:25 AM, Richard Purdie wrote:
> > Basically, the thought is whether to move from docbook to sphinx.
>
> This sounds like a good idea to me.
>
> Prior to yesterday I had never looked into sphinx before. Skimming some
> information about sphinx, it looks like it's flexible with regards to the
> exact formatting you use for, for example, section headings. If that's true, I
> believe we'll need a "coding standard" in place before proceeding. We wouldn't
> want the BSP Manual to use:
>
> 	section 1
> 	^^^^^^^^^
> 	section 1.1
> 	===========
> 	section 1.1.1
> 	-------------
>
> and the Reference Manual to use:
>
> 	section 1
> 	=========
> 	section 1.1
> 	***********
> 	section 1.1.1
> 	^^^^^^^^^^^^^

  absolutely, and i'll throw out one more "coding standard" that i
initially loathed, but came to appreciate -- keep sentences on the
same physical line.

  i first ran into that standard when contributing to the "pro git"
book, and i hated, hated, hated it as it naturally made long sentences
go on forever off the right side of the terminal window.

  however, i grew to like it because it makes side-by-side comparison
trivial and if you ended up with a sentence like that, it made you
reconsider whether you were doing this correctly.

  anyway, yes to the above.

rday

p.s. i've played briefly with sphinx on my fedora 31/32 system -- has
anyone set up a full processing toolchain that can just be purloined
to get started? even though it looks pretty simple.

Re: [docs] Possible changes to the project docs

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 3061 bytes --]

On Thu, Apr 16, 2020 at 6:51 PM Robert P. J. Day <rpjday@crashcourse.ca>
wrote:

> On Wed, 15 Apr 2020, Trevor Woerner wrote:
>
> > On Wed 2020-04-15 @ 11:19:25 AM, Richard Purdie wrote:
> > > Basically, the thought is whether to move from docbook to sphinx.
> >
> > This sounds like a good idea to me.
> >
> > Prior to yesterday I had never looked into sphinx before. Skimming some
> > information about sphinx, it looks like it's flexible with regards to the
> > exact formatting you use for, for example, section headings. If that's
> true, I
> > believe we'll need a "coding standard" in place before proceeding. We
> wouldn't
> > want the BSP Manual to use:
> >
> >       section 1
> >       ^^^^^^^^^
> >       section 1.1
> >       ===========
> >       section 1.1.1
> >       -------------
> >
> > and the Reference Manual to use:
> >
> >       section 1
> >       =========
> >       section 1.1
> >       ***********
> >       section 1.1.1
> >       ^^^^^^^^^^^^^
>
>   absolutely, and i'll throw out one more "coding standard" that i
> initially loathed, but came to appreciate -- keep sentences on the
> same physical line.
>
>   i first ran into that standard when contributing to the "pro git"
> book, and i hated, hated, hated it as it naturally made long sentences
> go on forever off the right side of the terminal window.
>
>   however, i grew to like it because it makes side-by-side comparison
> trivial and if you ended up with a sentence like that, it made you
> reconsider whether you were doing this correctly.
>
>   anyway, yes to the above.
>

I added this in one of my commits messages, to start with:

==
    sphinx: implement headings

    The YP manuals layout is organized as:

    Book
      Chapter
        Section
          Section
            Section

    During the conversion with pandoc, some of the headings are not
    converted properly. Let's decide for now to use the following headings
    styles:

    Book              => overline ===
      Chapter         => overline ***
        Section       => ====
          Section     => ----
            Section   => ^^^^
              Section => """"

    At least with this change, we now have the same TOC with Sphinx and
    DocBook.
==


> rday
>
> p.s. i've played briefly with sphinx on my fedora 31/32 system -- has
> anyone set up a full processing toolchain that can just be purloined
> to get started? even though it looks pretty simple.
>

As I put in one of my emails before you can use my dev branch:

https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx

then it's just
pip3 install sphinx
# to use the non default theme
pip3 install sphinx_rtd_theme

cd documentation
make html

and it should generate exactly that on your local folder
https://people.linaro.org/~nicolas.dechesne/yp-docs/

I am happy to take patches in this branch (or create/maintain a contrib
branch if more folks want to play/experiment and send patches).




>

[-- Attachment #2: Type: text/html, Size: 4669 bytes --]

Re: [docs] Possible changes to the project docs

[Robert P. J. Day]

On Thu, 16 Apr 2020, Nicolas Dechesne wrote:

> As I put in one of my emails before you can use my dev branch:
>
> https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx
>
> then it's just
> pip3 install sphinx
> # to use the non default theme
> pip3 install sphinx_rtd_theme
>
> cd documentation make html
>
> and it should generate exactly that on your local folder
> https://people.linaro.org/~nicolas.dechesne/yp-docs/
>
> I am happy to take patches in this branch (or create/maintain a
> contrib branch if more folks want to play/experiment and send
> patches).

  definitely need some mechanism to handle docbook entities and basic
tags like <filename> ... i'm still poring over pandoc, can it be
configured to deal with that?

  will go read more ...

rday

Re: [docs] Possible changes to the project docs

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 1711 bytes --]

On Thu, Apr 16, 2020 at 7:46 PM Robert P. J. Day <rpjday@crashcourse.ca>
wrote:

> On Thu, 16 Apr 2020, Nicolas Dechesne wrote:
>
> > As I put in one of my emails before you can use my dev branch:
> >
> > https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx
> >
> > then it's just
> > pip3 install sphinx
> > # to use the non default theme
> > pip3 install sphinx_rtd_theme
> >
> > cd documentation make html
> >
> > and it should generate exactly that on your local folder
> > https://people.linaro.org/~nicolas.dechesne/yp-docs/
> >
> > I am happy to take patches in this branch (or create/maintain a
> > contrib branch if more folks want to play/experiment and send
> > patches).
>
>   definitely need some mechanism to handle docbook entities and basic
> tags like <filename> ... i'm still poring over pandoc, can it be
> configured to deal with that?
>
>   will go read more ...
>

It does 'most of the time'. See:
https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp.xml.html#bsp-layers

where the docbook source is:
       Each repository is a BSP layer supported by the Yocto Project



        (e.g. <filename>meta-raspberrypi</filename> and



        <filename>meta-intel</filename>).




I am not sure if it always does the *right* thing though.. and I have seen
occurrences where it failed. I am wondering if we should 'fix' pandoc , or
just spend some time fixing the output of pandoc. I don't think we will
avoid reading/checking each manual, anyways.. I was thinking we could use
pandoc for 80% of the work, some regex for 15% and then manually finish the
work.. it might be a bit optimistic ;)


rday
> 
>

[-- Attachment #2: Type: text/html, Size: 3511 bytes --]

Re: [docs] Possible changes to the project docs

[Robert P. J. Day]

[-- Attachment #1: Type: text/plain, Size: 3069 bytes --]

On Thu, 16 Apr 2020, Nicolas Dechesne wrote:

>
>
> On Thu, Apr 16, 2020 at 7:46 PM Robert P. J. Day <rpjday@crashcourse.ca> wrote:
>       On Thu, 16 Apr 2020, Nicolas Dechesne wrote:
>
>       > As I put in one of my emails before you can use my dev branch:
>       >
>       > https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx
>       >
>       > then it's just
>       > pip3 install sphinx
>       > # to use the non default theme
>       > pip3 install sphinx_rtd_theme
>       >
>       > cd documentation make html
>       >
>       > and it should generate exactly that on your local folder
>       > https://people.linaro.org/~nicolas.dechesne/yp-docs/
>       >
>       > I am happy to take patches in this branch (or create/maintain a
>       > contrib branch if more folks want to play/experiment and send
>       > patches).
>
>         definitely need some mechanism to handle docbook entities and basic
>       tags like <filename> ... i'm still poring over pandoc, can it be
>       configured to deal with that?
>
>         will go read more ...
>
>
> It does 'most of the time'. See:
> https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp.xml.html#bsp-layers
>
> where the docbook source is:
>        Each repository is a BSP layer supported by the Yocto Project                                  
>                                                                                                      
>                          
>         (e.g. <filename>meta-raspberrypi</filename> and                                              
>                                                                                                      
>                            
>         <filename>meta-intel</filename>).                                                            
>                                                                                                      
>                            
>  I am not sure if it always does the *right* thing though.. and I
> have seen occurrences where it failed. I am wondering if we should
> 'fix' pandoc , or just spend some time fixing the output of pandoc.
> I don't think we will avoid reading/checking each manual, anyways..
> I was thinking we could use pandoc for 80% of the work, some regex
> for 15% and then manually finish the work.. it might be a bit
> optimistic ;)

  interesting ... if you go to the top-level page:

https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp-guide.xml.html

the <filename> tag in the third note is *not* migrated over, you can
still see the tag. i suspect there will be plenty of tweaking, but i'm
still totally on-board with this, and am willing to contribute.

rday

Re: [docs] Possible changes to the project docs

[Mark Morton]

Hi:

I'd like to actually start planning this conversion to Sphinx. So far myself, Rob D., Nicolas, and Tim O. are all interested in helping out.

I'd like to have a Zoom meeting or something similar to get started on where to go from Nico's proof of concept. Please let me know if you want to attend - I'm thinking sometime early next week.

Thanks,
-Mark M. 

> -----Original Message-----
> From: docs@lists.yoctoproject.org <docs@lists.yoctoproject.org> On Behalf
> Of Robert P. J. Day
> Sent: Thursday, April 16, 2020 11:06 AM
> To: docs@lists.yoctoproject.org
> Subject: Re: [docs] Possible changes to the project docs
> 
> On Thu, 16 Apr 2020, Nicolas Dechesne wrote:
> 
> >
> >
> > On Thu, Apr 16, 2020 at 7:46 PM Robert P. J. Day <rpjday@crashcourse.ca>
> wrote:
> >       On Thu, 16 Apr 2020, Nicolas Dechesne wrote:
> >
> >       > As I put in one of my emails before you can use my dev branch:
> >       >
> >       > https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx
> >       >
> >       > then it's just
> >       > pip3 install sphinx
> >       > # to use the non default theme
> >       > pip3 install sphinx_rtd_theme
> >       >
> >       > cd documentation make html
> >       >
> >       > and it should generate exactly that on your local folder
> >       > https://people.linaro.org/~nicolas.dechesne/yp-docs/
> >       >
> >       > I am happy to take patches in this branch (or create/maintain a
> >       > contrib branch if more folks want to play/experiment and send
> >       > patches).
> >
> >         definitely need some mechanism to handle docbook entities and basic
> >       tags like <filename> ... i'm still poring over pandoc, can it be
> >       configured to deal with that?
> >
> >         will go read more ...
> >
> >
> > It does 'most of the time'. See:
> > https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp.xml.
> > html#bsp-layers
> >
> > where the docbook source is:
> >        Each repository is a BSP layer supported by the Yocto Project
> >
> >
> >         (e.g. <filename>meta-raspberrypi</filename> and
> >
> >
> >         <filename>meta-intel</filename>).
> >
> >
> >  I am not sure if it always does the *right* thing though.. and I have
> > seen occurrences where it failed. I am wondering if we should 'fix'
> > pandoc , or just spend some time fixing the output of pandoc.
> > I don't think we will avoid reading/checking each manual, anyways..
> > I was thinking we could use pandoc for 80% of the work, some regex for
> > 15% and then manually finish the work.. it might be a bit optimistic
> > ;)
> 
>   interesting ... if you go to the top-level page:
> 
> https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp-
> guide.xml.html
> 
> the <filename> tag in the third note is *not* migrated over, you can still see
> the tag. i suspect there will be plenty of tweaking, but i'm still totally on-board
> with this, and am willing to contribute.
> 
> rday

Re: [docs] Possible changes to the project docs

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 3439 bytes --]

hi Mark,


On Thu, May 21, 2020 at 10:42 PM Mark Morton <Mark.Morton@windriver.com>
wrote:

> Hi:
>
> I'd like to actually start planning this conversion to Sphinx. So far
> myself, Rob D., Nicolas, and Tim O. are all interested in helping out.
>
> I'd like to have a Zoom meeting or something similar to get started on
> where to go from Nico's proof of concept. Please let me know if you want to
> attend - I'm thinking sometime early next week.
>

Thanks for starting this. I will be happy to help. I can prepare an
'initial' plan to get us started.


>
> Thanks,
> -Mark M.
>
> > -----Original Message-----
> > From: docs@lists.yoctoproject.org <docs@lists.yoctoproject.org> On
> Behalf
> > Of Robert P. J. Day
> > Sent: Thursday, April 16, 2020 11:06 AM
> > To: docs@lists.yoctoproject.org
> > Subject: Re: [docs] Possible changes to the project docs
> >
> > On Thu, 16 Apr 2020, Nicolas Dechesne wrote:
> >
> > >
> > >
> > > On Thu, Apr 16, 2020 at 7:46 PM Robert P. J. Day <
> rpjday@crashcourse.ca>
> > wrote:
> > >       On Thu, 16 Apr 2020, Nicolas Dechesne wrote:
> > >
> > >       > As I put in one of my emails before you can use my dev branch:
> > >       >
> > >       >
> https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx
> > >       >
> > >       > then it's just
> > >       > pip3 install sphinx
> > >       > # to use the non default theme
> > >       > pip3 install sphinx_rtd_theme
> > >       >
> > >       > cd documentation make html
> > >       >
> > >       > and it should generate exactly that on your local folder
> > >       > https://people.linaro.org/~nicolas.dechesne/yp-docs/
> > >       >
> > >       > I am happy to take patches in this branch (or create/maintain a
> > >       > contrib branch if more folks want to play/experiment and send
> > >       > patches).
> > >
> > >         definitely need some mechanism to handle docbook entities and
> basic
> > >       tags like <filename> ... i'm still poring over pandoc, can it be
> > >       configured to deal with that?
> > >
> > >         will go read more ...
> > >
> > >
> > > It does 'most of the time'. See:
> > > https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp.xml.
> > > html#bsp-layers
> > >
> > > where the docbook source is:
> > >        Each repository is a BSP layer supported by the Yocto Project
> > >
> > >
> > >         (e.g. <filename>meta-raspberrypi</filename> and
> > >
> > >
> > >         <filename>meta-intel</filename>).
> > >
> > >
> > >  I am not sure if it always does the *right* thing though.. and I have
> > > seen occurrences where it failed. I am wondering if we should 'fix'
> > > pandoc , or just spend some time fixing the output of pandoc.
> > > I don't think we will avoid reading/checking each manual, anyways..
> > > I was thinking we could use pandoc for 80% of the work, some regex for
> > > 15% and then manually finish the work.. it might be a bit optimistic
> > > ;)
> >
> >   interesting ... if you go to the top-level page:
> >
> > https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp-
> > guide.xml.html
> >
> > the <filename> tag in the third note is *not* migrated over, you can
> still see
> > the tag. i suspect there will be plenty of tweaking, but i'm still
> totally on-board
> > with this, and am willing to contribute.
> >
> > rday
> 
>

[-- Attachment #2: Type: text/html, Size: 5402 bytes --]

Re: [docs] Possible changes to the project docs

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 3749 bytes --]

On Mon, May 25, 2020 at 9:33 AM Nicolas Dechesne <
nicolas.dechesne@linaro.org> wrote:

> hi Mark,
>
>
> On Thu, May 21, 2020 at 10:42 PM Mark Morton <Mark.Morton@windriver.com>
> wrote:
>
>> Hi:
>>
>> I'd like to actually start planning this conversion to Sphinx. So far
>> myself, Rob D., Nicolas, and Tim O. are all interested in helping out.
>>
>> I'd like to have a Zoom meeting or something similar to get started on
>> where to go from Nico's proof of concept. Please let me know if you want to
>> attend - I'm thinking sometime early next week.
>>
>
> Thanks for starting this. I will be happy to help. I can prepare an
> 'initial' plan to get us started.
>

Well.. I've started this wiki:
https://wiki.yoctoproject.org/wiki/Documentation_Sphinx


>
>
>>
>> Thanks,
>> -Mark M.
>>
>> > -----Original Message-----
>> > From: docs@lists.yoctoproject.org <docs@lists.yoctoproject.org> On
>> Behalf
>> > Of Robert P. J. Day
>> > Sent: Thursday, April 16, 2020 11:06 AM
>> > To: docs@lists.yoctoproject.org
>> > Subject: Re: [docs] Possible changes to the project docs
>> >
>> > On Thu, 16 Apr 2020, Nicolas Dechesne wrote:
>> >
>> > >
>> > >
>> > > On Thu, Apr 16, 2020 at 7:46 PM Robert P. J. Day <
>> rpjday@crashcourse.ca>
>> > wrote:
>> > >       On Thu, 16 Apr 2020, Nicolas Dechesne wrote:
>> > >
>> > >       > As I put in one of my emails before you can use my dev branch:
>> > >       >
>> > >       >
>> https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx
>> > >       >
>> > >       > then it's just
>> > >       > pip3 install sphinx
>> > >       > # to use the non default theme
>> > >       > pip3 install sphinx_rtd_theme
>> > >       >
>> > >       > cd documentation make html
>> > >       >
>> > >       > and it should generate exactly that on your local folder
>> > >       > https://people.linaro.org/~nicolas.dechesne/yp-docs/
>> > >       >
>> > >       > I am happy to take patches in this branch (or create/maintain
>> a
>> > >       > contrib branch if more folks want to play/experiment and send
>> > >       > patches).
>> > >
>> > >         definitely need some mechanism to handle docbook entities and
>> basic
>> > >       tags like <filename> ... i'm still poring over pandoc, can it be
>> > >       configured to deal with that?
>> > >
>> > >         will go read more ...
>> > >
>> > >
>> > > It does 'most of the time'. See:
>> > > https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp.xml
>> .
>> > > html#bsp-layers
>> > >
>> > > where the docbook source is:
>> > >        Each repository is a BSP layer supported by the Yocto Project
>> > >
>> > >
>> > >         (e.g. <filename>meta-raspberrypi</filename> and
>> > >
>> > >
>> > >         <filename>meta-intel</filename>).
>> > >
>> > >
>> > >  I am not sure if it always does the *right* thing though.. and I have
>> > > seen occurrences where it failed. I am wondering if we should 'fix'
>> > > pandoc , or just spend some time fixing the output of pandoc.
>> > > I don't think we will avoid reading/checking each manual, anyways..
>> > > I was thinking we could use pandoc for 80% of the work, some regex for
>> > > 15% and then manually finish the work.. it might be a bit optimistic
>> > > ;)
>> >
>> >   interesting ... if you go to the top-level page:
>> >
>> > https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp-
>> > guide.xml.html
>> >
>> > the <filename> tag in the third note is *not* migrated over, you can
>> still see
>> > the tag. i suspect there will be plenty of tweaking, but i'm still
>> totally on-board
>> > with this, and am willing to contribute.
>> >
>> > rday
>> 
>>
>

[-- Attachment #2: Type: text/html, Size: 6205 bytes --]

Re: Possible changes to the project docs

[Mark Van De Vyver]

[-- Attachment #1: Type: text/plain, Size: 2169 bytes --]

Hmm, it seems the interwebs ate my prior message.   TLDR;

Considering AsciiDoc, MkDoc-Materialize, Docusarus, Sphinx and Antora

* A switch from DocBook is a net benefit regardless of what you switch too.
* Making docs more accessible to dev and user contributors is the key.
* I regard ascidoc/md/rst as near enough that they don't swing the decision - in terms of being approachable to devs and users.
* A potential discriminator is l10n/i18n - on the face of it they all seem to be imperfect (wip) and adequate.
* Is there any other strategic benefit to be obtained?

I think so.  Looking at https://antora.org/ this stands out:
*The ability to retrieve and aggregate all the content from different git repositories.*

This provides the following headroom:
* Move documentation into the repo that holds the related code.  Bringing documentation closer to devs who are more likely to write some preliminary documentation and correct documentation errors/ambiguities.
* Allow individual repo's/tools to evolve at their own pace - a release that repo/tool X would like to make 'yesterday' can be made and everyone can be confident the docs are still consistent (assumes that tool/repo-Y does not make any statements about tool/repo-X in the Y-docs)
* Allow the project to treat a repo as a 1st class citizen even if it is not under the openembedded.org or yoctoproject.org
* Corollary: Allow any repo outside of openembedded.org or yoctoproject.org to incorporate whichever of the tools/repos they need (git cloning & forking model allows/encourages that) and present documentation that is integrated and targeted (Antora could - eventually - allow/encourage that).
* Give the Yocto-project flexibility in how to organize itself going forward - documentation is not an impediment.

Of course there is a learning curve and the docs need to be refactored back to their home tool/repo.
To my mind there is a categorical distinction that can be made between the DocBook alternatives.

The questions is whether any of the above conjectures are of value or attractive?

HTH?
--
Kind regards
Mark

Mark Van de Vyver, B.Bus(Hons), PhD(Dist)

[-- Attachment #2: Type: text/html, Size: 2441 bytes --]

Re: [docs] Possible changes to the project docs

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 5328 bytes --]

hi Mark,

thanks a lot for joining the discussion! This is very helpful and much
appreciated.

On Thu, May 28, 2020 at 7:49 AM Mark Van De Vyver <mark@taqtiqa.com> wrote:

> Hmm, it seems the interwebs ate my prior message.   TLDR;
>

I believe you replied privately to me, your message is not lost and is in
good hands , well mine ;)


>
> Considering AsciiDoc, MkDoc-Materialize, Docusarus, Sphinx and Antora
>
> * A switch from DocBook is a net benefit regardless of what you switch too.
> * Making docs more accessible to dev and user contributors is the key.
> * I regard ascidoc/md/rst as near enough that they don't swing the
> decision - in terms of being approachable to devs and users.

* A potential discriminator is l10n/i18n - on the face of it they all seem
> to be imperfect (wip) and adequate.
>

We don't have plans for that now, even if it might be a good thing for the
project, I think we struggle enough with one language today.


> * Is there any other strategic benefit to be obtained?
>
The main goal is to ensure documentation is 'easier' to read and write,
encouraging more contributors, while keeping the reputation of the quality
of the project's documentation.


>
> I think so.  Looking at https://antora.org/ this stands out:
> *The ability to retrieve and aggregate all the content from different git
> repositories.*


I think there are 2 sides to the problem:
1. the format of the documentation: DocBook (XML) vs reST vs Asciidoc. I
think it's well accepted that Markdown is 'not enough' for large tech
documentation/book
2. the tools available to process the documentation (sphinx, asciidoctor,
antora.. ). Antora is a toolset on top of Asciidoc.

The main reasons why we've started to explore/experiment with reST/Sphinx
(as opposed to Asciidoc) are:
1. this is what Python uses, and our project is intimately linked to
Python. Sphinx can be extended with plugins in Python, and we have very
capable Python developers around here.
2. we've closely studied what happened in the Linux kernel community, and
their thought process when they moved away from Docbook a couple of years
ago, which is nicely summarized in the following articles:
https://lwn.net/Articles/671496/
https://lwn.net/Articles/692704/
https://lwn.net/Articles/692705/
Even though the initial plan was to move from Docbook to Asciidoc, they
ended up using reST/Sphinx in the end. Given the proximity of the YP and
kernel communities, we should pay close attention to what they've done.

That said, no decision has been made, Sphinx is just the first proof of
concept which have been made. So now is definitely the right time to
discuss all options!


>
> This provides the following headroom:
> * Move documentation into the repo that holds the related code.  Bringing
> documentation closer to devs who are more likely to write some preliminary
> documentation and correct documentation errors/ambiguities.
>
I agree that the project documentation would benefit from some inline
documentation. Not for everything, but maybe for classes, variables,
functions, it would be easier to maintain if the documentation was 'closer'
to the doc. The kernel documentation uses a mix of inline doc vs separate
doc already (with sphinx).


> * Allow individual repo's/tools to evolve at their own pace - a release
> that repo/tool X would like to make 'yesterday' can be made and everyone
> can be confident the docs are still consistent (assumes that tool/repo-Y
> does not make any statements about tool/repo-X in the Y-docs)
> * Allow the project to treat a repo as a 1st class citizen even if it is
> not under the openembedded.org or yoctoproject.org

I don't think there is such a concept of a 1st class citizen. The
Openembedded ecosystem is made of a very large amount of layers, and only a
minority of layers are hosted on oe.org or yp.org. I believe that all
layers are treated equally, as soon as they are listed/added to the OE
layer index maybe.


> * Corollary: Allow any repo outside of openembedded.org or
> yoctoproject.org to incorporate whichever of the tools/repos they need
> (git cloning & forking model allows/encourages that) and present
> documentation that is integrated and targeted (Antora could - eventually -
> allow/encourage that).
>

The YP documentation 'project' is about publishing documentation for what
makes up the Yocto Project (OE-core, bitbake, ..). The current Docbook
structure is probably too complex (learning curve) to be adopted in other
layers. So anything we could improve to make it simpler for other layers to
benefit from doc infrastructure would be a nice bonus.


> * Give the Yocto-project flexibility in how to organize itself going
> forward - documentation is not an impediment.
>
> Of course there is a learning curve and the docs need to be refactored
> back to their home tool/repo.
> To my mind there is a categorical distinction that can be made between the
> DocBook alternatives.
>

Can you provide more information about what Antora vs Asciidoc/asciidoctor?


>
> The questions is whether any of the above conjectures are of value or
> attractive?
>
> HTH?
> --
> Kind regards
> Mark
>
> Mark Van de Vyver, B.Bus(Hons), PhD(Dist)
> 
>

[-- Attachment #2: Type: text/html, Size: 7746 bytes --]

Re: [docs] Possible changes to the project docs

[Robert P. J. Day]

[-- Attachment #1: Type: text/plain, Size: 1030 bytes --]

On Thu, 28 May 2020, Nicolas Dechesne wrote:

... snip ...

> The main reasons why we've started to explore/experiment with reST/Sphinx (as opposed to
> Asciidoc) are:
> 1. this is what Python uses, and our project is intimately linked to Python. Sphinx can
> be extended with plugins in Python, and we have very capable Python developers around
> here.
> 2. we've closely studied what happened in the Linux kernel community, and their thought
> process when they moved away from Docbook a couple of years ago, which is nicely
> summarized in the following articles:
> https://lwn.net/Articles/671496/ 
> https://lwn.net/Articles/692704/
> https://lwn.net/Articles/692705/
> Even though the initial plan was to move from Docbook to Asciidoc, they ended up using
> reST/Sphinx in the end. Given the proximity of the YP and kernel communities, we should
> pay close attention to what they've done.

... snip ...

  FWIW, the red hat folks working on podman/libpod docs are moving to
sphinx as well.

rday

Re: [docs] Possible changes to the project docs

[Robert P. J. Day]

On Thu, 28 May 2020, Nicolas Dechesne wrote:

> hi Mark,
>
> thanks a lot for joining the discussion! This is very helpful and much appreciated.

  ... snip ...

>       Considering AsciiDoc, MkDoc-Materialize, Docusarus, Sphinx and Antora
>
>       * A switch from DocBook is a net benefit regardless of what you switch too.

  the one aspect of docbook i will miss is actual semantic markup
which can then be customized with stylesheets, but that benefit is
outweighed by all the downsides of the complexity. and given that i'm
poring over the sphinx docs right now and playing with it, yeah,
sphinx is my choice. :-)

rday

Re: [docs] Possible changes to the project docs

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 1099 bytes --]

On Thu, May 28, 2020 at 10:29 AM Robert P. J. Day <rpjday@crashcourse.ca>
wrote:

> On Thu, 28 May 2020, Nicolas Dechesne wrote:
>
> > hi Mark,
> >
> > thanks a lot for joining the discussion! This is very helpful and much
> appreciated.
>
>   ... snip ...
>
> >       Considering AsciiDoc, MkDoc-Materialize, Docusarus, Sphinx and
> Antora
> >
> >       * A switch from DocBook is a net benefit regardless of what you
> switch too.
>
>   the one aspect of docbook i will miss is actual semantic markup
> which can then be customized with stylesheets, but that benefit is
> outweighed by all the downsides of the complexity. and given that i'm
> poring over the sphinx docs right now and playing with it, yeah,
> sphinx is my choice. :-)
>

Ah, nice. If you have any patches that are relevant for the sphinx proof of
concept, please share them. I started a sphinx2 branch here:
https://git.linaro.org/people/nicolas.dechesne/yocto-docs.git/log/?h=sphinx2

with a 'better' commit log structure. I have also done the pandoc
conversion for all manuals.


>
> rday
>

[-- Attachment #2: Type: text/html, Size: 1761 bytes --]

Re: Possible changes to the project docs

[Mark Van De Vyver]

[-- Attachment #1: Type: text/plain, Size: 11158 bytes --]

On Thu, May 28, 2020 at 12:54 AM, Nicolas Dechesne wrote:

> 
> hi Mark,
> 
> thanks a lot for joining the discussion! This is very helpful and much
> appreciated.
> 
> On Thu, May 28, 2020 at 7:49 AM Mark Van De Vyver < mark@taqtiqa.com >
> wrote:
> 
>> Hmm, it seems the interwebs ate my prior message.   TLDR;
> 
> 
> I believe you replied privately to me, your message is not lost and is in
> good hands , well mine ;)
> 
> 
>> Considering AsciiDoc, MkDoc-Materialize, Docusarus, Sphinx and Antora
>> 
>> * A switch from DocBook is a net benefit regardless of what you switch
>> too.
>> * Making docs more accessible to dev and user contributors is the key.
>> * I regard ascidoc/md/rst as near enough that they don't swing the
>> decision - in terms of being approachable to devs and users.
> 
> 
>> * A potential discriminator is l10n/i18n - on the face of it they all seem
>> to be imperfect (wip) and adequate.
> 
> 
> We don't have plans for that now, even if it might be a good thing for the
> project, I think we struggle enough with one language today.
> 
> 

Agreed.
Unless there is a group dedicated to a translation, it would only make sense to restrict the translation to some automated service - google translate at worse or some thing like Transifex. Transifex offer a freelicense to "Transifex for free to Open Source projects ( http://docs.transifex.com/faq/#what-is-considered-an-open-source-project ) that have no funding, revenue, and/or commercialization model".  Not clear if we qualify for that.
The question is whether the cost of a 'rough' translation is worth the benefit of new users who are technically profficient but whose English is not at a level to grok the docs fully.
Thinking here of the proverbial 14yo who has just got board/device xyz and stumbles upon yoctoproject.org.

> 
> 
> 
>> * Is there any other strategic benefit to be obtained?
> 
> The main goal is to ensure documentation is 'easier' to read and write,
> encouraging more contributors, while keeping the reputation of the quality
> of the project's documentation.
> 
> 
>> I think so.  Looking at https://antora.org/ this stands out:
>> *The ability to retrieve and aggregate all the content from different git
>> repositories.*
> 
> 
> I think there are 2 sides to the problem:
> 1. the format of the documentation: DocBook (XML) vs reST vs Asciidoc. I
> think it's well accepted that Markdown is 'not enough' for large tech
> documentation/book
> 
> 

MD is ruled out, to my mind, by the lack of multi-repo support;
However it is worth noting the scope of the syntax/functionality should be sufficient - is see here ( https://cloudflow.io/docs/current ) for a larger project and here ( https://jlenv.github.io/jlenv ) for my Julia port of rbenv (devs here might be familiar with the Python port of rbenv).
Hence, I would argue that, which ever of asciidoc or rst is used, we encourage users employ to a subset of the syntax to keep the docs simple and maintainable.
AFAICT we're not dealing Julia or Linux graphics code levels of complexity.  Happy to be corrected.

> 
> 2. the tools available to process the documentation (sphinx, asciidoctor,
> antora.. ). Antora is a toolset on top of Asciidoc.
> 
> The main reasons why we've started to explore/experiment with reST/Sphinx
> (as opposed to Asciidoc) are:
> 1. this is what Python uses, and our project is intimately linked to
> Python. Sphinx can be extended with plugins in Python, and we have very
> capable Python developers around here.
> 
> 

Developer productivity is key.
I'd argue constraining to tool X's functionality.
Again, I'm happy to be wrong, yet AFAICT there shouldn't be a level of complexity in this projects documentation requirements that exceeds these tools.

> 
> 2. we've closely studied what happened in the Linux kernel community, and
> their thought process when they moved away from Docbook a couple of years
> ago, which is nicely summarized in the following articles:
> https://lwn.net/Articles/671496/
> https://lwn.net/Articles/692704/
> https://lwn.net/Articles/692705/
> Even though the initial plan was to move from Docbook to Asciidoc, they
> ended up using reST/Sphinx in the end. Given the proximity of the YP and
> kernel communities, we should pay close attention to what they've done.
> 
> 

Agreed - I read that transition slightly differently: The kernel doc tooling was imperfect but generally tenable.  The Linux graphics code base complexity made the docs untenable, and forced an alternative to be sought. Once the graphics code base needs were satisfied it obviously made sense to standardize.  Just as Julia has use cases of that compexity.
Are we really dealing with that level of difficulty (as disctinct from very elborate/long chains of scripts/routines - each conceptually relatively straightword in isloation).

> 
> 
> That said, no decision has been made, Sphinx is just the first proof of
> concept which have been made. So now is definitely the right time to
> discuss all options!
> 
> 

Great. So I spent some time digging further (See Transifex note above)...

> 
> 
>> This provides the following headroom:
>> * Move documentation into the repo that holds the related code.  Bringing
>> documentation closer to devs who are more likely to write some preliminary
>> documentation and correct documentation errors/ambiguities.
> 
> I agree that the project documentation would benefit from some inline
> documentation. Not for everything, but maybe for classes, variables,
> functions, it would be easier to maintain if the documentation was
> 'closer' to the doc. The kernel documentation uses a mix of inline doc vs
> separate doc already (with sphinx).
> 

Agreed. Pretty much every tool can handle some level of rendering inline docs that should be adequate.
What Antora allows is doc sharing between repos, and AFAICT, Sphinx doesn't provide that headroom.

> 
> 
> 
> 
>> * Allow individual repo's/tools to evolve at their own pace - a release
>> that repo/tool X would like to make 'yesterday' can be made and everyone
>> can be confident the docs are still consistent (assumes that tool/repo-Y
>> does not make any statements about tool/repo-X in the Y-docs)
>> * Allow the project to treat a repo as a 1st class citizen even if it is
>> not under the openembedded.org ( http://openembedded.org ) or yoctoproject.org
>> ( http://yoctoproject.org )
> 
> I don't think there is such a concept of a 1st class citizen. The
> Openembedded ecosystem is made of a very large amount of layers, and only
> a minority of layers are hosted on oe.org ( http://oe.org ) or yp.org (
> http://yp.org ). I believe that all layers are treated equally, as soon as
> they are listed/added to the OE layer index maybe.
> 
> 

Yet such layers cannot present integrated documentation.
Lets take a concrete existing example:
http://git.yoctoproject.org/cgit/cgit.cgi/meta-amd/tree/?h=dunfell

Ideally there would be a way for that project to present their (currently) secondary documentation (assume the eight (8) md files are ported to which ever syntax or rst or asciidoc) integrated with the yocto/oe parts of their documentation.  Even if this is only for some group internal document portal or readthedocs or some such portal.
In fact on the dunfell branch the current DOCUMENTATION.md suggests a Antora " playbook ( https://docs.antora.org/antora/2.2/playbook/ ) " (untested):

site:
 title: Layer XYZ Documentation

content:
 sources:
 - url: .
   branches: dunfell
   start_path: documentation/introduction - url: git://git.yoctoproject.org/poky.git branches: - dunfell start_path: documentation/overview-manual - url: git://git.yoctoproject.org/poky.git branches: - dunfell start_path: documentation/ brief-yoctoprojectqs ( http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/documentation/brief-yoctoprojectqs?h=dunfell )
 - url: git://git.yoctoproject.org/poky.git branches: - dunfell start_path: documentation/ toaster-manual ( http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/documentation/brief-yoctoprojectqs?h=dunfell )
 - url: git://git.yoctoproject.org/poky.git branches: - dunfell start_path: documentation/ sdk-manual ( http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/documentation/brief-yoctoprojectqs?h=dunfell )
output: clean: true dir: public 
Take a second concrete example:
http://git.yoctoproject.org/cgit/cgit.cgi/meta-raspberrypi

The proverbial 12-14yo with a board now has three sets of documentation to contend with (following the README.md):
1. https://meta-raspberrypi.readthedocs.io
2. Yocto/OE
3. raspberrypi.org

> 
> 
> 
>> * Corollary: Allow any repo outside of openembedded.org (
>> http://openembedded.org ) or yoctoproject.org ( http://yoctoproject.org ) to
>> incorporate whichever of the tools/repos they need (git cloning & forking
>> model allows/encourages that) and present documentation that is integrated
>> and targeted (Antora could - eventually - allow/encourage that).
> 
> 
> The YP documentation 'project' is about publishing documentation for what
> makes up the Yocto Project (OE-core, bitbake, ..).
> 

Agreed. I suppose I'm arguing the project specific elements of the documentation (i.e not reusable by/of interest to a repo/layer) could (ideally) be limited to page header, footer and brand image detail - everything else could be reusable by layers as they think best fits their userbase/audience.

> 
> The current Docbook structure is probably too complex (learning curve) to
> be adopted in other layers. So anything we could improve to make it
> simpler for other layers to benefit from doc infrastructure would be a
> nice bonus.
> 
> 
>> * Give the Yocto-project flexibility in how to organize itself going
>> forward - documentation is not an impediment.
>> 
>> Of course there is a learning curve and the docs need to be refactored
>> back to their home tool/repo.
>> To my mind there is a categorical distinction that can be made between the
>> DocBook alternatives.
> 
> 
> Can you provide more information about what Antora vs
> Asciidoc/asciidoctor?
> 
> 

Personally I've never encountered the need use asciidoc. Until now - I'll likely refactor the github.com/jlenv org documents so that each repo's documentation is pulled into a whole under jlenv.github.io/.
My understanding Antora builds on asciidoctor.js.
I'd argue for whichever tool is used: Project and layer docs should limit themselves to that tools built-in functionality - patching a doc tool detracts dev cycles.
I'd argue to limit documentation to what is effectively the markdown level of functionality - but layer/repo maintainers are best placed to make this call.

Kind regards
Mark

> 
> 
> 
>> The questions is whether any of the above conjectures are of value or
>> attractive?
>> 
>> HTH?
>> --
>> Kind regards
>> Mark
>> 
>> Mark Van de Vyver, B.Bus(Hons), PhD(Dist)
>> 
> 
> 

--
Kind regards
Mark

Mark Van de Vyver, B.Bus(Hons), PhD(Dist)

[-- Attachment #2: Type: text/html, Size: 17733 bytes --]

Re: [docs] Possible changes to the project docs

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 4248 bytes --]

hi Mark (Morton)

On Tue, May 26, 2020 at 12:43 PM Nicolas Dechesne <
nicolas.dechesne@linaro.org> wrote:

>
>
> On Mon, May 25, 2020 at 9:33 AM Nicolas Dechesne <
> nicolas.dechesne@linaro.org> wrote:
>
>> hi Mark,
>>
>>
>> On Thu, May 21, 2020 at 10:42 PM Mark Morton <Mark.Morton@windriver.com>
>> wrote:
>>
>>> Hi:
>>>
>>> I'd like to actually start planning this conversion to Sphinx. So far
>>> myself, Rob D., Nicolas, and Tim O. are all interested in helping out.
>>>
>>> I'd like to have a Zoom meeting or something similar to get started on
>>> where to go from Nico's proof of concept. Please let me know if you want to
>>> attend - I'm thinking sometime early next week.
>>>
>>
>> Thanks for starting this. I will be happy to help. I can prepare an
>> 'initial' plan to get us started.
>>
>
> Well.. I've started this wiki:
> https://wiki.yoctoproject.org/wiki/Documentation_Sphinx
>

Can we find a time next week to get things going? I am in CET time zone,
fwiw. How about any of the following time (I don't know where you are so
using PST timezone)

* Monday  9AM PST
* Tuesday 10AM PST
* Wed 9AM or 10AM PST


>
>
>>
>>
>>>
>>> Thanks,
>>> -Mark M.
>>>
>>> > -----Original Message-----
>>> > From: docs@lists.yoctoproject.org <docs@lists.yoctoproject.org> On
>>> Behalf
>>> > Of Robert P. J. Day
>>> > Sent: Thursday, April 16, 2020 11:06 AM
>>> > To: docs@lists.yoctoproject.org
>>> > Subject: Re: [docs] Possible changes to the project docs
>>> >
>>> > On Thu, 16 Apr 2020, Nicolas Dechesne wrote:
>>> >
>>> > >
>>> > >
>>> > > On Thu, Apr 16, 2020 at 7:46 PM Robert P. J. Day <
>>> rpjday@crashcourse.ca>
>>> > wrote:
>>> > >       On Thu, 16 Apr 2020, Nicolas Dechesne wrote:
>>> > >
>>> > >       > As I put in one of my emails before you can use my dev
>>> branch:
>>> > >       >
>>> > >       >
>>> https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx
>>> > >       >
>>> > >       > then it's just
>>> > >       > pip3 install sphinx
>>> > >       > # to use the non default theme
>>> > >       > pip3 install sphinx_rtd_theme
>>> > >       >
>>> > >       > cd documentation make html
>>> > >       >
>>> > >       > and it should generate exactly that on your local folder
>>> > >       > https://people.linaro.org/~nicolas.dechesne/yp-docs/
>>> > >       >
>>> > >       > I am happy to take patches in this branch (or
>>> create/maintain a
>>> > >       > contrib branch if more folks want to play/experiment and send
>>> > >       > patches).
>>> > >
>>> > >         definitely need some mechanism to handle docbook entities
>>> and basic
>>> > >       tags like <filename> ... i'm still poring over pandoc, can it
>>> be
>>> > >       configured to deal with that?
>>> > >
>>> > >         will go read more ...
>>> > >
>>> > >
>>> > > It does 'most of the time'. See:
>>> > >
>>> https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp.xml.
>>> > > html#bsp-layers
>>> > >
>>> > > where the docbook source is:
>>> > >        Each repository is a BSP layer supported by the Yocto Project
>>> > >
>>> > >
>>> > >         (e.g. <filename>meta-raspberrypi</filename> and
>>> > >
>>> > >
>>> > >         <filename>meta-intel</filename>).
>>> > >
>>> > >
>>> > >  I am not sure if it always does the *right* thing though.. and I
>>> have
>>> > > seen occurrences where it failed. I am wondering if we should 'fix'
>>> > > pandoc , or just spend some time fixing the output of pandoc.
>>> > > I don't think we will avoid reading/checking each manual, anyways..
>>> > > I was thinking we could use pandoc for 80% of the work, some regex
>>> for
>>> > > 15% and then manually finish the work.. it might be a bit optimistic
>>> > > ;)
>>> >
>>> >   interesting ... if you go to the top-level page:
>>> >
>>> > https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp-
>>> > guide.xml.html
>>> >
>>> > the <filename> tag in the third note is *not* migrated over, you can
>>> still see
>>> > the tag. i suspect there will be plenty of tweaking, but i'm still
>>> totally on-board
>>> > with this, and am willing to contribute.
>>> >
>>> > rday
>>> 
>>>
>>

[-- Attachment #2: Type: text/html, Size: 7150 bytes --]

Re: [docs] Possible changes to the project docs

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 6249 bytes --]

On Fri, Jun 12, 2020 at 1:52 PM Nicolas Dechesne <
nicolas.dechesne@linaro.org> wrote:

> hi Mark (Morton)
>
> On Tue, May 26, 2020 at 12:43 PM Nicolas Dechesne <
> nicolas.dechesne@linaro.org> wrote:
>
>>
>>
>> On Mon, May 25, 2020 at 9:33 AM Nicolas Dechesne <
>> nicolas.dechesne@linaro.org> wrote:
>>
>>> hi Mark,
>>>
>>>
>>> On Thu, May 21, 2020 at 10:42 PM Mark Morton <Mark.Morton@windriver.com>
>>> wrote:
>>>
>>>> Hi:
>>>>
>>>> I'd like to actually start planning this conversion to Sphinx. So far
>>>> myself, Rob D., Nicolas, and Tim O. are all interested in helping out.
>>>>
>>>> I'd like to have a Zoom meeting or something similar to get started on
>>>> where to go from Nico's proof of concept. Please let me know if you want to
>>>> attend - I'm thinking sometime early next week.
>>>>
>>>
>>> Thanks for starting this. I will be happy to help. I can prepare an
>>> 'initial' plan to get us started.
>>>
>>
>> Well.. I've started this wiki:
>> https://wiki.yoctoproject.org/wiki/Documentation_Sphinx
>>
>
> Can we find a time next week to get things going? I am in CET time zone,
> fwiw. How about any of the following time (I don't know where you are so
> using PST timezone)
>
> * Monday  9AM PST
> * Tuesday 10AM PST
> * Wed 9AM or 10AM PST
>

Just to update everyone, we have agreed to have a first meeting this week,
on Wed at 9AM PST time. The meeting has been added in the YP Google
Calendar as well (refer to
https://www.yoctoproject.org/public-virtual-meetings/)

And I am sharing the meeting details here as well

https://zoom.us/j/97588912425
<https://www.google.com/url?q=https://zoom.us/j/97588912425&sa=D&ust=1592682564242000&usg=AOvVaw261x8lemP-eSIOdWUOiTEc>
Meeting ID: 975 8891 2425 One tap mobile +12532158782,,97588912425# US
(Tacoma) +13462487799,,97588912425# US (Houston) Dial by your location +1
253 215 8782 US (Tacoma) +1 346 248 7799 US (Houston) +1 669 900 6833 US
(San Jose) +1 301 715 8592 US (Germantown) +1 312 626 6799 US (Chicago) +1
646 558 8656 US (New York) 877 369 0926 US Toll-free 855 880 1246 US
Toll-free +44 208 080 6591 United Kingdom +44 208 080 6592 United Kingdom
+44 330 088 5830 United Kingdom +44 131 460 1196 United Kingdom +44 203 481
5237 United Kingdom +44 203 481 5240 United Kingdom 0 800 260 5801 United
Kingdom Toll-free 0 800 031 5717 United Kingdom Toll-free +60 3 9212 1727
Malaysia +60 3 3099 2229 Malaysia 1 800 818 425 Malaysia Toll-free +55 21
3958 7888 Brazil +55 11 4680 6788 Brazil 0 800 878 3108 Brazil Toll-free 0
800 282 5751 Brazil Toll-free +52 229 910 0061 Mexico +52 554 161 4288
Mexico +52 556 826 9800 Mexico 800 733 4118 Mexico Toll-free 800 062 7287
Mexico Toll-free +64 9 884 6780 New Zealand +64 4 886 0026 New Zealand 0
800 527 544 New Zealand Toll-free 0 800 002 260 New Zealand Toll-free
Meeting ID: 975 8891 2425
Find your local number: https://zoom.us/u/adB5zrMbQ8
<https://www.google.com/url?q=https://zoom.us/u/adB5zrMbQ8&sa=D&ust=1592682564242000&usg=AOvVaw0DXi0h4wUmkZ03JQTC7xqf>



>
>>
>>
>>>
>>>
>>>>
>>>> Thanks,
>>>> -Mark M.
>>>>
>>>> > -----Original Message-----
>>>> > From: docs@lists.yoctoproject.org <docs@lists.yoctoproject.org> On
>>>> Behalf
>>>> > Of Robert P. J. Day
>>>> > Sent: Thursday, April 16, 2020 11:06 AM
>>>> > To: docs@lists.yoctoproject.org
>>>> > Subject: Re: [docs] Possible changes to the project docs
>>>> >
>>>> > On Thu, 16 Apr 2020, Nicolas Dechesne wrote:
>>>> >
>>>> > >
>>>> > >
>>>> > > On Thu, Apr 16, 2020 at 7:46 PM Robert P. J. Day <
>>>> rpjday@crashcourse.ca>
>>>> > wrote:
>>>> > >       On Thu, 16 Apr 2020, Nicolas Dechesne wrote:
>>>> > >
>>>> > >       > As I put in one of my emails before you can use my dev
>>>> branch:
>>>> > >       >
>>>> > >       >
>>>> https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx
>>>> > >       >
>>>> > >       > then it's just
>>>> > >       > pip3 install sphinx
>>>> > >       > # to use the non default theme
>>>> > >       > pip3 install sphinx_rtd_theme
>>>> > >       >
>>>> > >       > cd documentation make html
>>>> > >       >
>>>> > >       > and it should generate exactly that on your local folder
>>>> > >       > https://people.linaro.org/~nicolas.dechesne/yp-docs/
>>>> > >       >
>>>> > >       > I am happy to take patches in this branch (or
>>>> create/maintain a
>>>> > >       > contrib branch if more folks want to play/experiment and
>>>> send
>>>> > >       > patches).
>>>> > >
>>>> > >         definitely need some mechanism to handle docbook entities
>>>> and basic
>>>> > >       tags like <filename> ... i'm still poring over pandoc, can it
>>>> be
>>>> > >       configured to deal with that?
>>>> > >
>>>> > >         will go read more ...
>>>> > >
>>>> > >
>>>> > > It does 'most of the time'. See:
>>>> > >
>>>> https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp.xml.
>>>> > > html#bsp-layers
>>>> > >
>>>> > > where the docbook source is:
>>>> > >        Each repository is a BSP layer supported by the Yocto Project
>>>> > >
>>>> > >
>>>> > >         (e.g. <filename>meta-raspberrypi</filename> and
>>>> > >
>>>> > >
>>>> > >         <filename>meta-intel</filename>).
>>>> > >
>>>> > >
>>>> > >  I am not sure if it always does the *right* thing though.. and I
>>>> have
>>>> > > seen occurrences where it failed. I am wondering if we should 'fix'
>>>> > > pandoc , or just spend some time fixing the output of pandoc.
>>>> > > I don't think we will avoid reading/checking each manual, anyways..
>>>> > > I was thinking we could use pandoc for 80% of the work, some regex
>>>> for
>>>> > > 15% and then manually finish the work.. it might be a bit optimistic
>>>> > > ;)
>>>> >
>>>> >   interesting ... if you go to the top-level page:
>>>> >
>>>> > https://people.linaro.org/~nicolas.dechesne/yp-docs/bsp-guide/bsp-
>>>> > guide.xml.html
>>>> >
>>>> > the <filename> tag in the third note is *not* migrated over, you can
>>>> still see
>>>> > the tag. i suspect there will be plenty of tweaking, but i'm still
>>>> totally on-board
>>>> > with this, and am willing to contribute.
>>>> >
>>>> > rday
>>>> 
>>>>
>>>

[-- Attachment #2: Type: text/html, Size: 10527 bytes --]

Re: Possible changes to the project docs

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 2958 bytes --]

hi,

On Wed, Apr 15, 2020 at 12:19 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> Since Robert was talking about future docs wide changes, I wanted to
> put an idea out there. Its something a few of us have talked a little
> about. At this point it is an idea, not a decision and something we're
> thinking we need to explore and evaluate.
>
> Basically, the thought is whether to move from docbook to sphinx.
>

Good timing. I have been tossing around this idea... As much as I love
using and reading the YP docs/manual, I know that maintaining and updating
the documentation 'sources' is rather difficult.


> Other projects like the kernel have switched and it could assist some
> of our website/publishing challenges. It might also be easier for
> people to contribute to and there are wider and more modern tools
> supporting that format.
>

Right, and they claimed in https://lwn.net/Articles/692704/:
"if the documentation can be written in plain text rather than DocBook XML,
it's more likely to be written in the first place."

I have spent no more than a couple of hours on it, trying to understand why
the kernel moved and how Sphinx works. The outcome is really
interesting...

I use a conversion tool for DocBook .xml to Sphinx/ReST file format
(pandoc). Of course there are conversion gothas.. but the documentation
source code becomes much easier.

See:
https://people.linaro.org/~nicolas.dechesne/yp-docs/_sources/ref-manual/ref-release-process.xml.rst.txt
vs
http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/documentation/ref-manual/ref-release-process.xml

Or even for the variable glossary (sphinx has builtin support for that):
https://people.linaro.org/~nicolas.dechesne/yp-docs/_sources/ref-manual/ref-variables.xml.rst

vs
http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/documentation/ref-manual/ref-variables.xml

I have picked the same theme as the Linux kernel documentation, but of
course, the theme is just HTML and CSS, so it's entirely customizable.

For now, only 3 manuals were converted, the html output looks like that:
https://people.linaro.org/~nicolas.dechesne/yp-docs/

I have made a couple of patches to show how to get there:
https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx

It was straightforward to install anything i needed:
sudo apt install pandoc
pip3 install sphinx
# to use the non default theme
pip3 install sphinx_rtd_theme

then it's just:
make html


> It'd be a big change, particularly if we consider we may need to
> convert previous releases of the docs as well as the current version.
> That said, if we were going to do this, the time could be now...
>

The intent is just to show where this could lead us. We need to ensure that
it will not affect the quality of our documentation, which is obviously one
of the project's gems!


>
> Cheers,
>
> Richard
>
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 4985 bytes --]

Re: [docs] Possible changes to the project docs

[akuster]

[-- Attachment #1: Type: text/plain, Size: 3768 bytes --]



On 4/15/20 8:37 AM, Nicolas Dechesne wrote:
> hi,
>
> On Wed, Apr 15, 2020 at 12:19 PM Richard Purdie
> <richard.purdie@linuxfoundation.org
> <mailto:richard.purdie@linuxfoundation.org>> wrote:
>
>     Since Robert was talking about future docs wide changes, I wanted to
>     put an idea out there. Its something a few of us have talked a little
>     about. At this point it is an idea, not a decision and something we're
>     thinking we need to explore and evaluate.
>
>     Basically, the thought is whether to move from docbook to sphinx.
>
>
> Good timing. I have been tossing around this idea... As much as I love
> using and reading the YP docs/manual, I know that maintaining and
> updating the documentation 'sources' is rather difficult.

its not that bad IMHO : )

>
>
>     Other projects like the kernel have switched and it could assist some
>     of our website/publishing challenges. It might also be easier for
>     people to contribute to and there are wider and more modern tools
>     supporting that format.
>
>
> Right, and they claimed in https://lwn.net/Articles/692704/:
> "if the documentation can be written in plain text rather than DocBook
> XML, it's more likely to be written in the first place."
>
> I have spent no more than a couple of hours on it, trying to
> understand why the kernel moved and how Sphinx works. The outcome is
> really interesting...   
>
> I use a conversion tool for DocBook .xml to Sphinx/ReST file format
> (pandoc). Of course there are conversion gothas.. but the
> documentation source code becomes much easier.

Cool, a conversion tool. will have to look into it.
>
> See:
> https://people.linaro.org/~nicolas.dechesne/yp-docs/_sources/ref-manual/ref-release-process.xml.rst.txt
> vs
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/documentation/ref-manual/ref-release-process.xml
>
> Or even for the variable glossary (sphinx has builtin support for that):
> https://people.linaro.org/~nicolas.dechesne/yp-docs/_sources/ref-manual/ref-variables.xml.rst 
> vs
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/documentation/ref-manual/ref-variables.xml
>
> I have picked the same theme as the Linux kernel documentation, but of
> course, the theme is just HTML and CSS, so it's entirely customizable. 
>
> For now, only 3 manuals were converted, the html output looks like that:
> https://people.linaro.org/~nicolas.dechesne/yp-docs/
>
> I have made a couple of patches to show how to get there:
> https://git.linaro.org/people/nicolas.dechesne/poky.git/log/?h=sphinx 
>
> It was straightforward to install anything i needed:
> sudo apt install pandoc
> pip3 install sphinx
> # to use the non default theme
> pip3 install sphinx_rtd_theme
>
> then it's just:
> make html
>
>
>     It'd be a big change, particularly if we consider we may need to
>     convert previous releases of the docs as well as the current version.
>     That said, if we were going to do this, the time could be now...
>
>
> The intent is just to show where this could lead us. We need to ensure
> that it will not affect the quality of our documentation, which is
> obviously one of the project's gems!

I am in the middle of doing clean up work to remove "Common" code into
their own files. This allow us to make a fix in one place instead of
updating all the manuals.

This is what happens for every new dot release.  I did that for Zeus and
Warrior and is silly to have to do and is what prompted to clean this
up.  I hope sphinx will allow a similar structure as updating 11 files
with the same information is not a good use of time.

- armin
>  
>
>
>     Cheers,
>
>     Richard
>
>
>
>
>
>
> 


[-- Attachment #2: Type: text/html, Size: 9024 bytes --]

Re: [docs] Possible changes to the project docs

[Robert P. J. Day]

On Wed, 15 Apr 2020, Richard Purdie wrote:

> Since Robert was talking about future docs wide changes, I wanted to
> put an idea out there. Its something a few of us have talked a
> little about. At this point it is an idea, not a decision and
> something we're thinking we need to explore and evaluate.
>
> Basically, the thought is whether to move from docbook to sphinx.
>
> Other projects like the kernel have switched and it could assist
> some of our website/publishing challenges. It might also be easier
> for people to contribute to and there are wider and more modern
> tools supporting that format.
>
> It'd be a big change, particularly if we consider we may need to
> convert previous releases of the docs as well as the current
> version. That said, if we were going to do this, the time could be
> now...

  ok, time to weigh in now that i have a minute ... i am totally on
board with a rewriting in sphinx, as that would not only make it
easier for others to contribute, but it's something that, given some
minimal direction, could be handed off to an intern of some kind or --
as i see already mentioned in subsequent replies -- fed through an
automatic translator (eg., pandoc).

  it would also be the right time to restructure some of the docs
(migration manual all on its own, etc, etc.).

  the only downside to the switch is that, as arcane as docbook and
xsl stylesheets can be at times, they do support pretty much infinite
and incredibly refined customization. however, i don't think the YP
docs really scream for that level of customization.

  i'm for it.

rday