[sphinx] updates as of Aug 5th.

[sphinx] updates as of Aug 5th.

[Nicolas Dechesne]

hi there,

I will be away from the Interweb the next two weeks. So I wanted to
share some updates about how the YP docs migration to Sphinx is doing.
[it is a fairly long email].

To be honest, I wouldn't mind a little bit of help, and I will give
some hints about how you can help!

We are still working on publishing the current Sphinx docs on
yoctoproject.org, in the meantime, it's here for YP and Bitbake
documentation.

https://people.linaro.org/~nicolas.dechesne/yp-docs/html/index.html
https://people.linaro.org/~nicolas.dechesne/bitbake-docs/html/

All changes are pushed to the 'sphinx' branch in yocto-docs and in
bitbake trees.

Summary of the recent changes:

* I imported some CSS config from the current YP documentation, as a
result the look and feel is much closer to what we currently have.
There are still things to improve, but it's better on my eyes at
least.
* I added all images/figures, since they were missed by the automatic
conversion.
* I added a general index (it's a builtin Sphinx feature), which uses
all terms from the glossary. It is a nice feature, which looks
promising (it's a general index across all documents)
* I fixed a large amount to cross references (~2200 instances) to
terms and sections using a series of Python regexp and also another
400 instances were fixed manually.. when my regexp failed.
* I added extlinks Sphinx support to refer to external links/website,
and fixed them up in the source tree.
* I added intersphinx support which allows a Sphinx documentation
project to refer to section, terms, references from another Sphinx
documentation project. This is really good, since we can cross
reference bitbake terms directly from the YP documentation, eg..
:term:`bitbake:BB_FOO` will 'just work'.

The new documentation source code (.rst files) are really good to work
with, and it's much better to work with ReST files than XML, without
any surprise. The conversion is just a lot of work, and I already
spent some significant amount of work on it.. I hope you will find it
useful and I really hope it will achieve its goal to make it more
attractive to contribute to Yocto Project documentation project.

The way Sphinx works is that we have a single documentation project
that includes all the YP 'books'. It is an important change, we no
longer have X independent books, but have one project with all of
them. We need to work on how we want to publish the documentation
after the Sphinx migration.

Richard is trying to add support to easily switch between YP version
within the doc (e.g. like the Python doc), and we will host the 'old'
documentation as well. e.g. when choosing the 'thud' version, we want
to show a static page (not a Sphinx doc) that points to the thud html
doc.

And now here is the section you are all expecting the most! How can you help?

1. first you could learn about Sphinx and get familiar with the new
syntax. You should review all the patches one by one, to learn from
what I did! The sphinx documentation is good too. But looking at a
real world example is much easier.

2. there are still plenty of broken links.. it's mostly cross
references to section from one doc to another. DocBook used to have
'variables' (set in poky.ent) for each document (http link) and had
some macro to append an anchor to create a link. This whole mechanism
is not working well with Sphinx, and we should use Sphinx 'references'
(e.g. :ref: syntax) instead. Maybe some of that can be automated with
regexp, but I am not sure (very often it's multi line text and the
replacement pattern is not always obvious).. Just to give an idea
about how much work there is:
 $ git grep YOCTO_DOCS_.*_URL */*.rst  | wc -l
475

3. During the conversion from DocBook the 'code' or 'shell' sections
have been messed up, and is lost... it's quite unfortunate as this
will require some good amount of work to convert.. In Sphinx, we would
do this:
.. code-block:: shell
   $ cd $HOME/poky
   $ source oe-init-build-env
And it would render as we expect it to be.
This task is typically something that would distribute well among many
developers..

4. There are probably some other conversion hiccups at random places
(color, font type, text, ...), so I think we will have to do a careful
review of the HTML to find such issues and fix them. To be honest, I
think it's a rather good thing to do anyways

5. Theme or CSS improvements. We are using the standard 'read the doc'
theme. I would like to get feedback about the visual quality of the
resulting HTML project. If anyone has idea to improve it, that would
be much appreciated.

thanks!
nico

Re: [docs] [sphinx] updates as of Aug 5th.

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 5232 bytes --]

Nico,

Thank you SO much for all this work. And especially thank you for taking
the time to write this very explicit email.

I plan to work on this come hell or high water during the time you will be
away.

On Wed, Aug 5, 2020 at 2:54 PM Nicolas Dechesne <nicolas.dechesne@linaro.org>
wrote:

> hi there,
>
> I will be away from the Interweb the next two weeks. So I wanted to
> share some updates about how the YP docs migration to Sphinx is doing.
> [it is a fairly long email].
>
> To be honest, I wouldn't mind a little bit of help, and I will give
> some hints about how you can help!
>
> We are still working on publishing the current Sphinx docs on
> yoctoproject.org, in the meantime, it's here for YP and Bitbake
> documentation.
>
> https://people.linaro.org/~nicolas.dechesne/yp-docs/html/index.html
> https://people.linaro.org/~nicolas.dechesne/bitbake-docs/html/
>
> All changes are pushed to the 'sphinx' branch in yocto-docs and in
> bitbake trees.
>
> Summary of the recent changes:
>
> * I imported some CSS config from the current YP documentation, as a
> result the look and feel is much closer to what we currently have.
> There are still things to improve, but it's better on my eyes at
> least.
> * I added all images/figures, since they were missed by the automatic
> conversion.
> * I added a general index (it's a builtin Sphinx feature), which uses
> all terms from the glossary. It is a nice feature, which looks
> promising (it's a general index across all documents)
> * I fixed a large amount to cross references (~2200 instances) to


2200!!!


> terms and sections using a series of Python regexp and also another
> 400 instances were fixed manually.. when my regexp failed.

400!!!


> * I added extlinks Sphinx support to refer to external links/website,
> and fixed them up in the source tree.
> * I added intersphinx support which allows a Sphinx documentation
> project to refer to section, terms, references from another Sphinx
> documentation project. This is really good, since we can cross
> reference bitbake terms directly from the YP documentation, eg..
> :term:`bitbake:BB_FOO` will 'just work'.
>
> The new documentation source code (.rst files) are really good to work
> with, and it's much better to work with ReST files than XML, without
> any surprise. The conversion is just a lot of work, and I already
> spent some significant amount of work on it.. I hope you will find it
> useful and I really hope it will achieve its goal to make it more
> attractive to contribute to Yocto Project documentation project.
>
> The way Sphinx works is that we have a single documentation project
> that includes all the YP 'books'. It is an important change, we no
> longer have X independent books, but have one project with all of
> them. We need to work on how we want to publish the documentation
> after the Sphinx migration.
>
> Richard is trying to add support to easily switch between YP version
> within the doc (e.g. like the Python doc), and we will host the 'old'
> documentation as well. e.g. when choosing the 'thud' version, we want
> to show a static page (not a Sphinx doc) that points to the thud html
> doc.
>
> And now here is the section you are all expecting the most! How can you
> help?
>
> 1. first you could learn about Sphinx and get familiar with the new
> syntax. You should review all the patches one by one, to learn from
> what I did! The sphinx documentation is good too. But looking at a
> real world example is much easier.
>
> 2. there are still plenty of broken links.. it's mostly cross
> references to section from one doc to another. DocBook used to have
> 'variables' (set in poky.ent) for each document (http link) and had
> some macro to append an anchor to create a link. This whole mechanism
> is not working well with Sphinx, and we should use Sphinx 'references'
> (e.g. :ref: syntax) instead. Maybe some of that can be automated with
> regexp, but I am not sure (very often it's multi line text and the
> replacement pattern is not always obvious).. Just to give an idea
> about how much work there is:
>  $ git grep YOCTO_DOCS_.*_URL */*.rst  | wc -l
> 475
>
> 3. During the conversion from DocBook the 'code' or 'shell' sections
> have been messed up, and is lost... it's quite unfortunate as this
> will require some good amount of work to convert.. In Sphinx, we would
> do this:
> .. code-block:: shell
>    $ cd $HOME/poky
>    $ source oe-init-build-env
> And it would render as we expect it to be.
> This task is typically something that would distribute well among many
> developers..
>
> 4. There are probably some other conversion hiccups at random places
> (color, font type, text, ...), so I think we will have to do a careful
> review of the HTML to find such issues and fix them. To be honest, I
> think it's a rather good thing to do anyways
>
> 5. Theme or CSS improvements. We are using the standard 'read the doc'
> theme. I would like to get feedback about the visual quality of the
> resulting HTML project. If anyone has idea to improve it, that would
> be much appreciated.
>
> thanks!
> nico
> 
>

[-- Attachment #2: Type: text/html, Size: 6577 bytes --]

Re: [docs] [sphinx] updates as of Aug 5th.

[Mark Morton]

[-- Attachment #1: Type: text/plain, Size: 5517 bytes --]

Hi Tim and Nico:

I have some clock cycles myself – please coordinate with me so we don’t end up duplicating each other’s work.

Thanks,
-Mark M.

From: docs@lists.yoctoproject.org <docs@lists.yoctoproject.org> On Behalf Of Tim Orling
Sent: Wednesday, August 5, 2020 7:35 PM
To: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Cc: docs@lists.yoctoproject.org
Subject: Re: [docs] [sphinx] updates as of Aug 5th.

Nico,

Thank you SO much for all this work. And especially thank you for taking the time to write this very explicit email.

I plan to work on this come hell or high water during the time you will be away.

On Wed, Aug 5, 2020 at 2:54 PM Nicolas Dechesne <nicolas.dechesne@linaro.org<mailto:nicolas.dechesne@linaro.org>> wrote:
hi there,

I will be away from the Interweb the next two weeks. So I wanted to
share some updates about how the YP docs migration to Sphinx is doing.
[it is a fairly long email].

To be honest, I wouldn't mind a little bit of help, and I will give
some hints about how you can help!

We are still working on publishing the current Sphinx docs on
yoctoproject.org<http://yoctoproject.org>, in the meantime, it's here for YP and Bitbake
documentation.

https://people.linaro.org/~nicolas.dechesne/yp-docs/html/index.html
https://people.linaro.org/~nicolas.dechesne/bitbake-docs/html/

All changes are pushed to the 'sphinx' branch in yocto-docs and in
bitbake trees.

Summary of the recent changes:

* I imported some CSS config from the current YP documentation, as a
result the look and feel is much closer to what we currently have.
There are still things to improve, but it's better on my eyes at
least.
* I added all images/figures, since they were missed by the automatic
conversion.
* I added a general index (it's a builtin Sphinx feature), which uses
all terms from the glossary. It is a nice feature, which looks
promising (it's a general index across all documents)
* I fixed a large amount to cross references (~2200 instances) to

2200!!!


terms and sections using a series of Python regexp and also another
400 instances were fixed manually.. when my regexp failed.
400!!!


* I added extlinks Sphinx support to refer to external links/website,
and fixed them up in the source tree.
* I added intersphinx support which allows a Sphinx documentation
project to refer to section, terms, references from another Sphinx
documentation project. This is really good, since we can cross
reference bitbake terms directly from the YP documentation, eg..
:term:`bitbake:BB_FOO` will 'just work'.

The new documentation source code (.rst files) are really good to work
with, and it's much better to work with ReST files than XML, without
any surprise. The conversion is just a lot of work, and I already
spent some significant amount of work on it.. I hope you will find it
useful and I really hope it will achieve its goal to make it more
attractive to contribute to Yocto Project documentation project.

The way Sphinx works is that we have a single documentation project
that includes all the YP 'books'. It is an important change, we no
longer have X independent books, but have one project with all of
them. We need to work on how we want to publish the documentation
after the Sphinx migration.

Richard is trying to add support to easily switch between YP version
within the doc (e.g. like the Python doc), and we will host the 'old'
documentation as well. e.g. when choosing the 'thud' version, we want
to show a static page (not a Sphinx doc) that points to the thud html
doc.

And now here is the section you are all expecting the most! How can you help?

1. first you could learn about Sphinx and get familiar with the new
syntax. You should review all the patches one by one, to learn from
what I did! The sphinx documentation is good too. But looking at a
real world example is much easier.

2. there are still plenty of broken links.. it's mostly cross
references to section from one doc to another. DocBook used to have
'variables' (set in poky.ent) for each document (http link) and had
some macro to append an anchor to create a link. This whole mechanism
is not working well with Sphinx, and we should use Sphinx 'references'
(e.g. :ref: syntax) instead. Maybe some of that can be automated with
regexp, but I am not sure (very often it's multi line text and the
replacement pattern is not always obvious).. Just to give an idea
about how much work there is:
 $ git grep YOCTO_DOCS_.*_URL */*.rst  | wc -l
475

3. During the conversion from DocBook the 'code' or 'shell' sections
have been messed up, and is lost... it's quite unfortunate as this
will require some good amount of work to convert.. In Sphinx, we would
do this:
.. code-block:: shell
   $ cd $HOME/poky
   $ source oe-init-build-env
And it would render as we expect it to be.
This task is typically something that would distribute well among many
developers..

4. There are probably some other conversion hiccups at random places
(color, font type, text, ...), so I think we will have to do a careful
review of the HTML to find such issues and fix them. To be honest, I
think it's a rather good thing to do anyways

5. Theme or CSS improvements. We are using the standard 'read the doc'
theme. I would like to get feedback about the visual quality of the
resulting HTML project. If anyone has idea to improve it, that would
be much appreciated.

thanks!
nico

[-- Attachment #2: Type: text/html, Size: 10707 bytes --]

Re: [docs] [sphinx] updates as of Aug 5th.

[Robert P. J. Day]

Quoting Mark Morton <Mark.Morton@windriver.com>:

> Hi Tim and Nico:
>
> I have some clock cycles myself – please coordinate with me so we  
> don’t end up duplicating each other’s work.

... snip ...

I feel guilty that I've been AWOL buried in a current (Yocto Project)
contract, so I'll spend this weekend catching up and promise I'll find
some time to help out.

rday

Re: [docs] [sphinx] updates as of Aug 5th.

[Robert Berger]

Hi,

My comments are in-line

On 06/08/2020 00:54, Nicolas Dechesne wrote:
> hi there,
> 
> I will be away from the Interweb the next two weeks. So I wanted to
> share some updates about how the YP docs migration to Sphinx is doing.
> [it is a fairly long email].
> 
> To be honest, I wouldn't mind a little bit of help, and I will give
> some hints about how you can help!
> 
> We are still working on publishing the current Sphinx docs on
> yoctoproject.org, in the meantime, it's here for YP and Bitbake
> documentation.
> 
> https://people.linaro.org/~nicolas.dechesne/yp-docs/html/index.html
> https://people.linaro.org/~nicolas.dechesne/bitbake-docs/html/
> 
> All changes are pushed to the 'sphinx' branch in yocto-docs and in
> bitbake trees.

As long as chapters/sub chapters and variables are have some kind of 
stable links this should work for me.

To give you some context:

I my training I have many links from my slides to the good old doc.
e.g.

in my latex magic I have something like:
{\tt \href{\MegaManualVarSearch LAYERSERIES\_COMPAT}{LAYERSERIES\_COMPAT 
(MM)}}

which ends up into a link pointing to a downloaded version of the good 
old Mega manual:

file:///xxx/mega-manual/mega-manual.html#var-LAYERSERIES_COMPAT

It looks like I could do the same with your stuff:

https://people.linaro.org/~nicolas.dechesne/yp-docs/html/ref-manual/ref-variables.html#term-LAYERSERIES_COMPAT

and I hope there will be a Mega Manual which contains them all ;)

Regards,

Robert

> 
> 
> 

Re: [docs] [sphinx] updates as of Aug 5th.

[Richard Purdie]

Hi Nico/All,

Thanks for putting that list together, its really useful.

On Wed, 2020-08-05 at 23:54 +0200, Nicolas Dechesne wrote:
> I will be away from the Interweb the next two weeks. So I wanted to
> share some updates about how the YP docs migration to Sphinx is
> doing.
>
> [...]
> 
> Richard is trying to add support to easily switch between YP version
> within the doc (e.g. like the Python doc), and we will host the 'old'
> documentation as well. e.g. when choosing the 'thud' version, we want
> to show a static page (not a Sphinx doc) that points to the thud html
> doc.

I hope you're having some good away time! I'm sending this while I have
the context in my head ready for when you get back. Its taken me a lot
more work than I thought but I think I have something ready to share in
this area now. I've published some sample output at:

https://www.rpsys.net/yocto-docs/

Basically this combines the existing docbooks docs into a structure
with the new sphinx generated manual such that you can select and
nagivate around and see the different versions/options.

I'd ultimately like to replace what we have at 
https://www.yoctoproject.org/docs/ with this. I appreciate there are
four links there we still need to add into the new docs page:

https://www.yoctoproject.org/docs/what-i-wish-id-known/ 
https://www.yoctoproject.org/docs/transitioning-to-a-custom-environment 
https://www.yoctoproject.org/software-overview/ 
https://wiki.yoctoproject.org/wiki/TipsAndTricks

but other than that I think its much more navigable and usable for
users. Features/changes include:

* Clearly showing which version you've viewing (yes, css needs work)
* Drop down selection of recent versions
* Drop down selection of the format you want 
  (individual web pages or all in one mega manual)
* Older documents with newer point releases show a warning in red at 
  the top
* Obsolete releases show a warning in red at the top
* All the releases are listed on the main page for search engine 
  visibility which should improve their indexing and avoid some of the 
  version issues we've had (I've gone back to sumo at present) and 
  make it clear to users how 'behind' a given version is

I know the sphinx transition isn't complete so the "dev" manual version
has all the issues still needing to be resolved with the transition.
The CSS also needs a little more work to help with the colours/layout.

I'm interested to hear what people think of this...

Code is in the yocto-docs repo on the sphinx-rp branch. "make -f
Makefile.sphinx" will generate the docs I've shared and the generated
output in _build/final is designed to overlay the existing docbook docs
and all fit together correctly.

Cheers,

Richard

Re: [docs] [sphinx] updates as of Aug 5th.

[Nicolas Dechesne]

hi Robert,

thanks for your time!

On Tue, Aug 11, 2020 at 6:07 PM Robert Berger@yocto.user
<robert.berger.yocto.user@gmail.com> wrote:
>
> Hi,
>
> My comments are in-line
>
> On 06/08/2020 00:54, Nicolas Dechesne wrote:
> > hi there,
> >
> > I will be away from the Interweb the next two weeks. So I wanted to
> > share some updates about how the YP docs migration to Sphinx is doing.
> > [it is a fairly long email].
> >
> > To be honest, I wouldn't mind a little bit of help, and I will give
> > some hints about how you can help!
> >
> > We are still working on publishing the current Sphinx docs on
> > yoctoproject.org, in the meantime, it's here for YP and Bitbake
> > documentation.
> >
> > https://people.linaro.org/~nicolas.dechesne/yp-docs/html/index.html
> > https://people.linaro.org/~nicolas.dechesne/bitbake-docs/html/
> >
> > All changes are pushed to the 'sphinx' branch in yocto-docs and in
> > bitbake trees.
>
> As long as chapters/sub chapters and variables are have some kind of
> stable links this should work for me.

Chapters/sections links will be stable. There is one difference
between how it's done with docbook files since the links were "hand
crafted", with Sphinx, they are generated from the 'section' name, but
they won't change unless we change the section name. e.g.:

https://people.linaro.org/~nicolas.dechesne/yp-docs/html/overview-manual/overview-manual-concepts.html#shared-state

I am planning to rename the individual files to avoid the duplication
in the url (e.g. overview-manual/overview-manual-concepts.rst should
be overview-manual/concepts.rst).

>
> To give you some context:
>
> I my training I have many links from my slides to the good old doc.
> e.g.
>
> in my latex magic I have something like:
> {\tt \href{\MegaManualVarSearch LAYERSERIES\_COMPAT}{LAYERSERIES\_COMPAT
> (MM)}}
>
> which ends up into a link pointing to a downloaded version of the good
> old Mega manual:
>
> file:///xxx/mega-manual/mega-manual.html#var-LAYERSERIES_COMPAT
>
> It looks like I could do the same with your stuff:
>
> https://people.linaro.org/~nicolas.dechesne/yp-docs/html/ref-manual/ref-variables.html#term-LAYERSERIES_COMPAT

yes, that's correct.

>
> and I hope there will be a Mega Manual which contains them all ;)

Here again there is a nice feature with Sphinx. We can generate a
global 'index' table that is automatically generated with all 'terms'
from all the glossaries, see:
https://people.linaro.org/~nicolas.dechesne/yp-docs/html/genindex.html

e.g. there can be more than 1 glossary. Currently we have 2:
* the variable glossary from the reference manual:
https://people.linaro.org/~nicolas.dechesne/yp-docs/html/_sources/ref-manual/ref-variables.rst.txt
* the terms glossary:
https://people.linaro.org/~nicolas.dechesne/yp-docs/html/_sources/ref-manual/ref-terms.rst.txt

We can also add any term from any place within the whole documentation
set.  I like that a lot.

>
> Regards,
>
> Robert
>
> >
> > 
> >
>

Re: [docs] [sphinx] updates as of Aug 5th.

[Nicolas Dechesne]

hi Richard,

On Thu, Aug 20, 2020 at 12:59 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> Hi Nico/All,
>
> Thanks for putting that list together, its really useful.
>
> On Wed, 2020-08-05 at 23:54 +0200, Nicolas Dechesne wrote:
> > I will be away from the Interweb the next two weeks. So I wanted to
> > share some updates about how the YP docs migration to Sphinx is
> > doing.
> >
> > [...]
> >
> > Richard is trying to add support to easily switch between YP version
> > within the doc (e.g. like the Python doc), and we will host the 'old'
> > documentation as well. e.g. when choosing the 'thud' version, we want
> > to show a static page (not a Sphinx doc) that points to the thud html
> > doc.
>
> I hope you're having some good away time! I'm sending this while I have
> the context in my head ready for when you get back. Its taken me a lot
> more work than I thought but I think I have something ready to share in
> this area now. I've published some sample output at:
>
> https://www.rpsys.net/yocto-docs/

I really like the output, and the added features. I don't like as much
how it's done though (the Makefile changes and the multi-pass sphinx
runs..) but that's an implementation detail ;-) But with the current
output we can create a standalone YP documentation 'web site' that
contains all our old docs (for all releases) independently of DocBook
vs Sphinx!

>
> Basically this combines the existing docbooks docs into a structure
> with the new sphinx generated manual such that you can select and
> nagivate around and see the different versions/options.
>
> I'd ultimately like to replace what we have at
> https://www.yoctoproject.org/docs/ with this. I appreciate there are
> four links there we still need to add into the new docs page:
>
> https://www.yoctoproject.org/docs/what-i-wish-id-known/
> https://www.yoctoproject.org/docs/transitioning-to-a-custom-environment

These 2 could potentially be converted into Sphinx RST files, assuming
we don't need them on the YP 'website'.

> https://www.yoctoproject.org/software-overview/
> https://wiki.yoctoproject.org/wiki/TipsAndTricks
>
> but other than that I think its much more navigable and usable for
> users. Features/changes include:
>
> * Clearly showing which version you've viewing (yes, css needs work)
> * Drop down selection of recent versions
> * Drop down selection of the format you want
>   (individual web pages or all in one mega manual)
> * Older documents with newer point releases show a warning in red at
>   the top
> * Obsolete releases show a warning in red at the top
> * All the releases are listed on the main page for search engine
>   visibility which should improve their indexing and avoid some of the
>   version issues we've had (I've gone back to sumo at present) and
>   make it clear to users how 'behind' a given version is
>
> I know the sphinx transition isn't complete so the "dev" manual version
> has all the issues still needing to be resolved with the transition.
> The CSS also needs a little more work to help with the colours/layout.
>
> I'm interested to hear what people think of this...
>
> Code is in the yocto-docs repo on the sphinx-rp branch. "make -f
> Makefile.sphinx" will generate the docs I've shared and the generated
> output in _build/final is designed to overlay the existing docbook docs
> and all fit together correctly.

Sphinx has a fairly advanced extension mechanism support. My gut
feeling tells me that a lot of what you did can be achieved with a
nice extension.. I will need to think about that.
See https://www.sphinx-doc.org/en/master/extdev/index.html#dev-extensions

For example, here is a simple extension used in the kernel doc project:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/sphinx/maintainers_include.py

But we can interact with the Sphinx builder at any stage of the
process. For example the whole generation of the 'old' index files
could be done this way. I would like to figure out how to produce what
you have in a single Sphinx run... But for now let's focus on the
actual output you have and get some feedback from the users as much as
possible, whether this is useful.

>
> Cheers,
>
> Richard
>
>
>

Re: [docs] [sphinx] updates as of Aug 5th.

[Richard Purdie]

Hi Nico,

On Mon, 2020-08-24 at 17:48 +0200, Nicolas Dechesne wrote:
> On Thu, Aug 20, 2020 at 12:59 AM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > I hope you're having some good away time! I'm sending this while I
> > have
> > the context in my head ready for when you get back. Its taken me a
> > lot
> > more work than I thought but I think I have something ready to
> > share in
> > this area now. I've published some sample output at:
> > 
> > https://www.rpsys.net/yocto-docs/
> 
> I really like the output, and the added features. I don't like as
> much how it's done though (the Makefile changes and the multi-pass
> sphinx runs..) but that's an implementation detail ;-) But with the
> current output we can create a standalone YP documentation 'web site'
> that contains all our old docs (for all releases) independently of
> DocBook vs Sphinx!

I don't particularly like what I did to make that happen either. It
felt like I just hacked sphinx to make it do what we wanted. They are
separate "doc" entities in their own right with no connection with each
other apart from the links so that seemed to require separate sphinx
build runs.

Obviously we can separate out the make target, probably use more
makefile magic to generate the files rather than my horrific for loop
and so on so even if we're stuck with this, it can be neater. I was
focusing on making it generate something which looked like what I think
we need.

> > Basically this combines the existing docbooks docs into a structure
> > with the new sphinx generated manual such that you can select and
> > nagivate around and see the different versions/options.
> > 
> > I'd ultimately like to replace what we have at
> > https://www.yoctoproject.org/docs/ with this. I appreciate there
> > are
> > four links there we still need to add into the new docs page:
> > 
> > https://www.yoctoproject.org/docs/what-i-wish-id-known/
> > https://www.yoctoproject.org/docs/transitioning-to-a-custom-environment
> 
> These 2 could potentially be converted into Sphinx RST files,
> assuming we don't need them on the YP 'website'.

I ended up adding links to these on the top level index page so I
effectively handled that. They're docs Tracey wrote and I think wants
as part of the website but we can discuss that in due course if needed.

> > https://www.yoctoproject.org/software-overview/
> > https://wiki.yoctoproject.org/wiki/TipsAndTricks

I also worked links to those into the index page since I sent this
email.

> > but other than that I think its much more navigable and usable for
> > users. Features/changes include:
> > 
> > * Clearly showing which version you've viewing (yes, css needs
> > work)
> > * Drop down selection of recent versions
> > * Drop down selection of the format you want
> >   (individual web pages or all in one mega manual)
> > * Older documents with newer point releases show a warning in red
> > at
> >   the top
> > * Obsolete releases show a warning in red at the top
> > * All the releases are listed on the main page for search engine
> >   visibility which should improve their indexing and avoid some of
> > the
> >   version issues we've had (I've gone back to sumo at present) and
> >   make it clear to users how 'behind' a given version is
> > 
> > I know the sphinx transition isn't complete so the "dev" manual
> > version
> > has all the issues still needing to be resolved with the
> > transition.
> > The CSS also needs a little more work to help with the
> > colours/layout.
> > 
> > I'm interested to hear what people think of this...
> > 
> > Code is in the yocto-docs repo on the sphinx-rp branch. "make -f
> > Makefile.sphinx" will generate the docs I've shared and the
> > generated
> > output in _build/final is designed to overlay the existing docbook
> > docs
> > and all fit together correctly.
> 
> Sphinx has a fairly advanced extension mechanism support. My gut
> feeling tells me that a lot of what you did can be achieved with a
> nice extension.. I will need to think about that.
> See 
> https://www.sphinx-doc.org/en/master/extdev/index.html#dev-extensions

I will take a look, the challenge is that the way I've done it, they're
each their own document and I think they perhaps need to be to keep the
navigation working the way we need it.

> For example, here is a simple extension used in the kernel doc
> project:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/sphinx/maintainers_include.py
> 
> But we can interact with the Sphinx builder at any stage of the
> process. For example the whole generation of the 'old' index files
> could be done this way. I would like to figure out how to produce
> what you have in a single Sphinx run... But for now let's focus on
> the actual output you have and get some feedback from the users as
> much as possible, whether this is useful.

I did mention it in a few of the meetings last week and so far its had
positive reviews and is something people would seem to find useful.

Cheers,

Richard

Re: [docs] [sphinx] updates as of Aug 5th.

[Robert Berger]

Hi Nic,

As far as I can see this looks good.
As soon as the official mega manual and bitbake manual are replaced by 
Sphinx I'll give it a spin.

Regards,

Robert

On 24/08/2020 18:36, Nicolas Dechesne wrote:
> hi Robert,
> 
> thanks for your time!
> 
> On Tue, Aug 11, 2020 at 6:07 PM Robert Berger@yocto.user
> <robert.berger.yocto.user@gmail.com> wrote:
>>
>> Hi,
>>
>> My comments are in-line
>>
>> On 06/08/2020 00:54, Nicolas Dechesne wrote:
>>> hi there,
>>>
>>> I will be away from the Interweb the next two weeks. So I wanted to
>>> share some updates about how the YP docs migration to Sphinx is doing.
>>> [it is a fairly long email].
>>>
>>> To be honest, I wouldn't mind a little bit of help, and I will give
>>> some hints about how you can help!
>>>
>>> We are still working on publishing the current Sphinx docs on
>>> yoctoproject.org, in the meantime, it's here for YP and Bitbake
>>> documentation.
>>>
>>> https://people.linaro.org/~nicolas.dechesne/yp-docs/html/index.html
>>> https://people.linaro.org/~nicolas.dechesne/bitbake-docs/html/
>>>
>>> All changes are pushed to the 'sphinx' branch in yocto-docs and in
>>> bitbake trees.
>>
>> As long as chapters/sub chapters and variables are have some kind of
>> stable links this should work for me.
> 
> Chapters/sections links will be stable. There is one difference
> between how it's done with docbook files since the links were "hand
> crafted", with Sphinx, they are generated from the 'section' name, but
> they won't change unless we change the section name. e.g.:
> 
> https://people.linaro.org/~nicolas.dechesne/yp-docs/html/overview-manual/overview-manual-concepts.html#shared-state
> 
> I am planning to rename the individual files to avoid the duplication
> in the url (e.g. overview-manual/overview-manual-concepts.rst should
> be overview-manual/concepts.rst).
> 
>>
>> To give you some context:
>>
>> I my training I have many links from my slides to the good old doc.
>> e.g.
>>
>> in my latex magic I have something like:
>> {\tt \href{\MegaManualVarSearch LAYERSERIES\_COMPAT}{LAYERSERIES\_COMPAT
>> (MM)}}
>>
>> which ends up into a link pointing to a downloaded version of the good
>> old Mega manual:
>>
>> file:///xxx/mega-manual/mega-manual.html#var-LAYERSERIES_COMPAT
>>
>> It looks like I could do the same with your stuff:
>>
>> https://people.linaro.org/~nicolas.dechesne/yp-docs/html/ref-manual/ref-variables.html#term-LAYERSERIES_COMPAT
> 
> yes, that's correct.
> 
>>
>> and I hope there will be a Mega Manual which contains them all ;)
> 
> Here again there is a nice feature with Sphinx. We can generate a
> global 'index' table that is automatically generated with all 'terms'
> from all the glossaries, see:
> https://people.linaro.org/~nicolas.dechesne/yp-docs/html/genindex.html
> 
> e.g. there can be more than 1 glossary. Currently we have 2:
> * the variable glossary from the reference manual:
> https://people.linaro.org/~nicolas.dechesne/yp-docs/html/_sources/ref-manual/ref-variables.rst.txt
> * the terms glossary:
> https://people.linaro.org/~nicolas.dechesne/yp-docs/html/_sources/ref-manual/ref-terms.rst.txt
> 
> We can also add any term from any place within the whole documentation
> set.  I like that a lot.
> 
>>
>> Regards,
>>
>> Robert
>>
>>>
>>> 
>>>
>>