[sphinx] updates as of Sep 10th.

[sphinx] updates as of Sep 10th.

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 3102 bytes --]

hi there,

some more updates on our Sphinx migration task. It turned out to be a
rather huge task.. "Never ask engineers to make estimates" applies quite
well here ;)

I think we are getting close to where we wanted to be. As you should know
by now, we are planning to switch for 3.2, and the sphinx migration is kind
of blocking M3 right now. Richard and I have spent quite a bit of time on
the docs lately to make the final push.

Any help in this last mile would be much appreciated.. The 'last mile' is
to review each doc and finalize what's left which was not converted
properly by pandoc, nor by any of the regexp find/replace we've done..
Unfortunately this last step is manual and time consuming.

We are planning to merge the sphinx branch in yocto-docs and bitbake repo
any time now. We haven't removed the docbook files yet, we will do the
removal once we claim to be done.

In terms of publishing the docs, our plan is to have a brand new YP docs
'website'. On that website, we will publish docs for each release/branch,
and it will be easy to navigate between versions.

We have also created 'transition' pages in Sphinx for docs from all YP
releases from 0.9 to 3.1.x. For each release, we have a 'static' page with
links to the DocBook HTML manuals. So that the YP docs website will be the
*only* place to look for our docs moving forward.

In each of the 2 sphinx branches, we've done the following:
1. add makefile / sphinx boilerplate
2. used pandoc to convert all .xml into .rst. it's a gross conversion.. far
from perfect...
3. added a bunch of commits to fix up what's left from the conversion
(images, links, ..)
4. Finally we are doing a manual review of each file and fixing the last
things. At this point, what's the most painful is that the 'code-block'
sections are not converted automatically.

If anybody has any cycle now, we need to finish #4. The current status is
that:
* brief-yoctoprojectqs/* is done.
* bitbake user manual is half done, and I am doing it
* bsp-guide/*: i started it, and will finish
* ref-manual/*: RP is on it.
* there are plenty of warnings when we build, i would like to get rid of
them all!

Anything else is work that remains to be done. Before starting anything,
please send a quick email here to let everyone know, to avoid duplication.
If you have any improvement, please share your patches, I will take them
into the sphinx/master branch. And I will continue to work on the remaining
files that no one would pick.

And since a picture is worth a thousand words...

The new doc website is here:
https://docs.yoctoproject.org/

If you want to help, please review the "WIP" patches on
https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx
And pick one of the manuals which are left! It is not straightforward to
get started, I have to warn you.. don't hesitate to ask me questions by
email or on irc (ndec) if you need any help.

And as a reminder, instructions to build can be found here:
https://wiki.yoctoproject.org/wiki/Documentation_Sphinx

cheers
nico

[-- Attachment #2: Type: text/html, Size: 3904 bytes --]

Re: [docs] [sphinx] updates as of Sep 10th.

[akuster]

On 9/10/20 3:03 PM, Nicolas Dechesne wrote:
> hi there,
>
> some more updates on our Sphinx migration task. It turned out to be a
> rather huge task.. "Never ask engineers to make estimates" applies
> quite well here ;)
>
> I think we are getting close to where we wanted to be. As you should
> know by now, we are planning to switch for 3.2, and the sphinx
> migration is kind of blocking M3 right now. Richard and I have spent
> quite a bit of time on the docs lately to make the final push.
>
> Any help in this last mile would be much appreciated.. The 'last mile'
> is to review each doc and finalize what's left which was not converted
> properly by pandoc, nor by any of the regexp find/replace we've done..
> Unfortunately this last step is manual and time consuming.
>
> We are planning to merge the sphinx branch in yocto-docs and bitbake
> repo any time now. We haven't removed the docbook files yet, we will
> do the removal once we claim to be done.
>
> In terms of publishing the docs, our plan is to have a brand new YP
> docs 'website'. On that website, we will publish docs for each
> release/branch, and it will be easy to navigate between versions. 
>
> We have also created 'transition' pages in Sphinx for docs from all YP
> releases from 0.9 to 3.1.x. For each release, we have a 'static' page
> with links to the DocBook HTML manuals. So that the YP docs website
> will be the *only* place to look for our docs moving forward.
>
> In each of the 2 sphinx branches, we've done the following:
> 1. add makefile / sphinx boilerplate
> 2. used pandoc to convert all .xml into .rst. it's a gross
> conversion.. far from perfect...
> 3. added a bunch of commits to fix up what's left from the conversion
> (images, links, ..)
> 4. Finally we are doing a manual review of each file and fixing the
> last things. At this point, what's the most painful is that the
> 'code-block' sections are not converted automatically.
>
> If anybody has any cycle now, we need to finish #4. The current status
> is that:
> * brief-yoctoprojectqs/* is done.
> * bitbake user manual is half done, and I am doing it
> * bsp-guide/*: i started it, and will finish
> * ref-manual/*: RP is on it.
> * there are plenty of warnings when we build, i would like to get rid
> of them all!


>
> Anything else is work that remains to be done. Before starting
> anything, please send a quick email here to let everyone know, to
> avoid duplication. If you have any improvement, please share your
> patches, I will take them into the sphinx/master branch. And I will
> continue to work on the remaining files that no one would pick.
>
> And since a picture is worth a thousand words...
>
> The new doc website is here:
> https://docs.yoctoproject.org/
>
> If you want to help, please review the "WIP" patches on 
> https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx
> And pick one of the manuals which are left! It is not straightforward
> to get started, I have to warn you.. don't hesitate to ask me
> questions by email or on irc (ndec) if you need any help.
Are we leaving the Mega manual for last or are we dropping it?

I will take the "overview-manual" if its not taken.

- armin

>
> And as a reminder, instructions to build can be found here:
> https://wiki.yoctoproject.org/wiki/Documentation_Sphinx
>
> cheers
> nico
>
>  
>
> 

Re: [docs] [sphinx] updates as of Sep 10th.

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 4898 bytes --]

On Fri, Sep 11, 2020 at 1:39 AM akuster808 <akuster808@gmail.com> wrote:

>
>
> On 9/10/20 3:03 PM, Nicolas Dechesne wrote:
> > hi there,
> >
> > some more updates on our Sphinx migration task. It turned out to be a
> > rather huge task.. "Never ask engineers to make estimates" applies
> > quite well here ;)
> >
> > I think we are getting close to where we wanted to be. As you should
> > know by now, we are planning to switch for 3.2, and the sphinx
> > migration is kind of blocking M3 right now. Richard and I have spent
> > quite a bit of time on the docs lately to make the final push.
> >
> > Any help in this last mile would be much appreciated.. The 'last mile'
> > is to review each doc and finalize what's left which was not converted
> > properly by pandoc, nor by any of the regexp find/replace we've done..
> > Unfortunately this last step is manual and time consuming.
> >
> > We are planning to merge the sphinx branch in yocto-docs and bitbake
> > repo any time now. We haven't removed the docbook files yet, we will
> > do the removal once we claim to be done.
> >
> > In terms of publishing the docs, our plan is to have a brand new YP
> > docs 'website'. On that website, we will publish docs for each
> > release/branch, and it will be easy to navigate between versions.
> >
> > We have also created 'transition' pages in Sphinx for docs from all YP
> > releases from 0.9 to 3.1.x. For each release, we have a 'static' page
> > with links to the DocBook HTML manuals. So that the YP docs website
> > will be the *only* place to look for our docs moving forward.
> >
> > In each of the 2 sphinx branches, we've done the following:
> > 1. add makefile / sphinx boilerplate
> > 2. used pandoc to convert all .xml into .rst. it's a gross
> > conversion.. far from perfect...
> > 3. added a bunch of commits to fix up what's left from the conversion
> > (images, links, ..)
> > 4. Finally we are doing a manual review of each file and fixing the
> > last things. At this point, what's the most painful is that the
> > 'code-block' sections are not converted automatically.
> >
> > If anybody has any cycle now, we need to finish #4. The current status
> > is that:
> > * brief-yoctoprojectqs/* is done.
> > * bitbake user manual is half done, and I am doing it
> > * bsp-guide/*: i started it, and will finish
> > * ref-manual/*: RP is on it.
> > * there are plenty of warnings when we build, i would like to get rid
> > of them all!
>
>
> >
> > Anything else is work that remains to be done. Before starting
> > anything, please send a quick email here to let everyone know, to
> > avoid duplication. If you have any improvement, please share your
> > patches, I will take them into the sphinx/master branch. And I will
> > continue to work on the remaining files that no one would pick.
> >
> > And since a picture is worth a thousand words...
> >
> > The new doc website is here:
> > https://docs.yoctoproject.org/
> >
> > If you want to help, please review the "WIP" patches on
> > https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/log/?h=sphinx
> > And pick one of the manuals which are left! It is not straightforward
> > to get started, I have to warn you.. don't hesitate to ask me
> > questions by email or on irc (ndec) if you need any help.
> Are we leaving the Mega manual for last or are we dropping it?
>

No, it's there. Sphinx can produce both "html" files, e.g. a 'website' with
all pages and the navigation bar, but it also has a target called
'singlehtml' which produces a single html page with all the doc. we are
publishing it as well (see the mega manual link in the left bar, or the
drop down selector on the top  you can choose between the regular site or
the mega manual). The mega manuel is:
https://docs.yoctoproject.org/singleindex.html

I am thinking about perhaps renaming it..

There are a few short coming with the Sphinx singlehtml target, the main
one being that Sphinx removes table of content in the singlehtml output.
You can move from manual to manual using the left navigation bar, but there
is no TOC in the HTML.. It's a known issue.. i am hoping we can fix that
later, but I think it's out of scope for 3.2.. At least the use case of
"grepping" through the mega manual is still supported.

I believe the new docs website will bring new ways of using the doc as
well. For example a general index is automatically generated/managed with
glossary terms, and over time we can include more terms in the index:
https://docs.yoctoproject.org/genindex.html



>
> I will take the "overview-manual" if its not taken.
>

thank you!


>
> - armin
>
> >
> > And as a reminder, instructions to build can be found here:
> > https://wiki.yoctoproject.org/wiki/Documentation_Sphinx
> >
> > cheers
> > nico
> >
> >
> >
> > 
>
>
>

[-- Attachment #2: Type: text/html, Size: 6603 bytes --]

Re: [docs] [sphinx] updates as of Sep 10th.

[Richard Purdie]

On Fri, 2020-09-11 at 00:03 +0200, Nicolas Dechesne wrote:
> If anybody has any cycle now, we need to finish #4. The current
> status is that:
> * brief-yoctoprojectqs/* is done.
> * bitbake user manual is half done, and I am doing it
> * bsp-guide/*: i started it, and will finish
> * ref-manual/*: RP is on it.

This is done, at least as a first pass with hopefully the bulk of
issues fixed.

> * there are plenty of warnings when we build, i would like to get rid
> of them all!
> 
> Anything else is work that remains to be done. Before starting
> anything, please send a quick email here to let everyone know, to
> avoid duplication. If you have any improvement, please share your
> patches, I will take them into the sphinx/master branch. And I will
> continue to work on the remaining files that no one would pick.

I'll take dev-manual/* next.

Cheers,

Richard

Re: [docs] [sphinx] updates as of Sep 10th.

[Richard Purdie]

On Sun, 2020-09-13 at 16:42 +0100, Richard Purdie via
lists.yoctoproject.org wrote:
> On Fri, 2020-09-11 at 00:03 +0200, Nicolas Dechesne wrote:
> > If anybody has any cycle now, we need to finish #4. The current
> > status is that:
> > * brief-yoctoprojectqs/* is done.
> > * bitbake user manual is half done, and I am doing it
> > * bsp-guide/*: i started it, and will finish
> > * ref-manual/*: RP is on it.
> 
> This is done, at least as a first pass with hopefully the bulk of
> issues fixed.
> 
> > * there are plenty of warnings when we build, i would like to get
> > rid
> > of them all!
> > 
> > Anything else is work that remains to be done. Before starting
> > anything, please send a quick email here to let everyone know, to
> > avoid duplication. If you have any improvement, please share your
> > patches, I will take them into the sphinx/master branch. And I will
> > continue to work on the remaining files that no one would pick.
> 
> I'll take dev-manual/* next.

dev-manual is on the branch, I'll look at sdk-manual next.

Cheers,

Richard

Re: [docs] [sphinx] updates as of Sep 10th.

[Richard Purdie]

On Mon, 2020-09-14 at 09:50 +0100, Richard Purdie via
lists.yoctoproject.org wrote:
> On Sun, 2020-09-13 at 16:42 +0100, Richard Purdie via
> lists.yoctoproject.org wrote:
> > On Fri, 2020-09-11 at 00:03 +0200, Nicolas Dechesne wrote:
> > > If anybody has any cycle now, we need to finish #4. The current
> > > status is that:
> > > * brief-yoctoprojectqs/* is done.
> > > * bitbake user manual is half done, and I am doing it
> > > * bsp-guide/*: i started it, and will finish
> > > * ref-manual/*: RP is on it.
> > 
> > This is done, at least as a first pass with hopefully the bulk of
> > issues fixed.
> > 
> > > * there are plenty of warnings when we build, i would like to get
> > > rid
> > > of them all!
> > > 
> > > Anything else is work that remains to be done. Before starting
> > > anything, please send a quick email here to let everyone know, to
> > > avoid duplication. If you have any improvement, please share your
> > > patches, I will take them into the sphinx/master branch. And I
> > > will
> > > continue to work on the remaining files that no one would pick.
> > 
> > I'll take dev-manual/* next.
> 
> dev-manual is on the branch, I'll look at sdk-manual next.

sdk-manual is on the branch, kernel-dev is my next target.

Cheers,

Richard

Re: [docs] [sphinx] updates as of Sep 10th.

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 1947 bytes --]

On Mon, Sep 14, 2020 at 5:26 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Mon, 2020-09-14 at 09:50 +0100, Richard Purdie via
> lists.yoctoproject.org wrote:
> > On Sun, 2020-09-13 at 16:42 +0100, Richard Purdie via
> > lists.yoctoproject.org wrote:
> > > On Fri, 2020-09-11 at 00:03 +0200, Nicolas Dechesne wrote:
> > > > If anybody has any cycle now, we need to finish #4. The current
> > > > status is that:
> > > > * brief-yoctoprojectqs/* is done.
> > > > * bitbake user manual is half done, and I am doing it
> > > > * bsp-guide/*: i started it, and will finish
> > > > * ref-manual/*: RP is on it.
> > >
> > > This is done, at least as a first pass with hopefully the bulk of
> > > issues fixed.
> > >
> > > > * there are plenty of warnings when we build, i would like to get
> > > > rid
> > > > of them all!
> > > >
> > > > Anything else is work that remains to be done. Before starting
> > > > anything, please send a quick email here to let everyone know, to
> > > > avoid duplication. If you have any improvement, please share your
> > > > patches, I will take them into the sphinx/master branch. And I
> > > > will
> > > > continue to work on the remaining files that no one would pick.
> > >
> > > I'll take dev-manual/* next.
> >
> > dev-manual is on the branch, I'll look at sdk-manual next.
>
> sdk-manual is on the branch, kernel-dev is my next target.
>

I am now done with bsp-guide and bitbake.  Based on Richard's last comment,
the remaining docs to process are:
overview-manual
profile-manual
test-manual
toaster-manual

They are rather 'small' books. I believe Armin is looking at
overview-manual and Mark at test-manual. I will take profile-manual next.

I've also updated https://docs.yoctoproject.org/ with all the latest
changes combined. If you can use the new website and report any issues,
please let us know.


> Cheers,
>
> Richard
>
>

[-- Attachment #2: Type: text/html, Size: 3010 bytes --]

Re: [docs] [sphinx] updates as of Sep 10th.

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 2175 bytes --]

On Mon, Sep 14, 2020 at 3:05 PM Nicolas Dechesne <
nicolas.dechesne@linaro.org> wrote:

>
>
> On Mon, Sep 14, 2020 at 5:26 PM Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
>
>> On Mon, 2020-09-14 at 09:50 +0100, Richard Purdie via
>> lists.yoctoproject.org wrote:
>> > On Sun, 2020-09-13 at 16:42 +0100, Richard Purdie via
>> > lists.yoctoproject.org wrote:
>> > > On Fri, 2020-09-11 at 00:03 +0200, Nicolas Dechesne wrote:
>> > > > If anybody has any cycle now, we need to finish #4. The current
>> > > > status is that:
>> > > > * brief-yoctoprojectqs/* is done.
>> > > > * bitbake user manual is half done, and I am doing it
>> > > > * bsp-guide/*: i started it, and will finish
>> > > > * ref-manual/*: RP is on it.
>> > >
>> > > This is done, at least as a first pass with hopefully the bulk of
>> > > issues fixed.
>> > >
>> > > > * there are plenty of warnings when we build, i would like to get
>> > > > rid
>> > > > of them all!
>> > > >
>> > > > Anything else is work that remains to be done. Before starting
>> > > > anything, please send a quick email here to let everyone know, to
>> > > > avoid duplication. If you have any improvement, please share your
>> > > > patches, I will take them into the sphinx/master branch. And I
>> > > > will
>> > > > continue to work on the remaining files that no one would pick.
>> > >
>> > > I'll take dev-manual/* next.
>> >
>> > dev-manual is on the branch, I'll look at sdk-manual next.
>>
>> sdk-manual is on the branch, kernel-dev is my next target.
>>
>
> I am now done with bsp-guide and bitbake.  Based on Richard's last
> comment, the remaining docs to process are:
> overview-manual
> profile-manual
> test-manual
> toaster-manual
>
> They are rather 'small' books. I believe Armin is looking at
> overview-manual and Mark at test-manual. I will take profile-manual next.
>

I'll take a crack at toaster-manual today.
--Tim

>
> I've also updated https://docs.yoctoproject.org/ with all the latest
> changes combined. If you can use the new website and report any issues,
> please let us know.
>
>
>> Cheers,
>>
>> Richard
>>
>> 
>

[-- Attachment #2: Type: text/html, Size: 3723 bytes --]

Re: [docs] [sphinx] updates as of Sep 10th.

[Nicolas Dechesne]

[-- Attachment #1: Type: text/plain, Size: 2702 bytes --]

On Tue, Sep 15, 2020 at 5:44 PM Tim Orling <ticotimo@gmail.com> wrote:

>
>
> On Mon, Sep 14, 2020 at 3:05 PM Nicolas Dechesne <
> nicolas.dechesne@linaro.org> wrote:
>
>>
>>
>> On Mon, Sep 14, 2020 at 5:26 PM Richard Purdie <
>> richard.purdie@linuxfoundation.org> wrote:
>>
>>> On Mon, 2020-09-14 at 09:50 +0100, Richard Purdie via
>>> lists.yoctoproject.org wrote:
>>> > On Sun, 2020-09-13 at 16:42 +0100, Richard Purdie via
>>> > lists.yoctoproject.org wrote:
>>> > > On Fri, 2020-09-11 at 00:03 +0200, Nicolas Dechesne wrote:
>>> > > > If anybody has any cycle now, we need to finish #4. The current
>>> > > > status is that:
>>> > > > * brief-yoctoprojectqs/* is done.
>>> > > > * bitbake user manual is half done, and I am doing it
>>> > > > * bsp-guide/*: i started it, and will finish
>>> > > > * ref-manual/*: RP is on it.
>>> > >
>>> > > This is done, at least as a first pass with hopefully the bulk of
>>> > > issues fixed.
>>> > >
>>> > > > * there are plenty of warnings when we build, i would like to get
>>> > > > rid
>>> > > > of them all!
>>> > > >
>>> > > > Anything else is work that remains to be done. Before starting
>>> > > > anything, please send a quick email here to let everyone know, to
>>> > > > avoid duplication. If you have any improvement, please share your
>>> > > > patches, I will take them into the sphinx/master branch. And I
>>> > > > will
>>> > > > continue to work on the remaining files that no one would pick.
>>> > >
>>> > > I'll take dev-manual/* next.
>>> >
>>> > dev-manual is on the branch, I'll look at sdk-manual next.
>>>
>>> sdk-manual is on the branch, kernel-dev is my next target.
>>>
>>
>> I am now done with bsp-guide and bitbake.  Based on Richard's last
>> comment, the remaining docs to process are:
>> overview-manual
>> profile-manual
>> test-manual
>> toaster-manual
>>
>> They are rather 'small' books. I believe Armin is looking at
>> overview-manual and Mark at test-manual. I will take profile-manual next.
>>
>
> I'll take a crack at toaster-manual today.
>

thanks!

I've finished profile-manual today. And excluded adt-manual from Sphinx
(deprecated doc). At this point, we have:
Richard on kernel-dev
Mark on test-manual
Armin on overview-manual
Tim on toaster-manual

Which means everything is now covered! Please send patches on the list for
review, and if you have any trouble with Sphinx , let me know!



> --Tim
>
>>
>> I've also updated https://docs.yoctoproject.org/ with all the latest
>> changes combined. If you can use the new website and report any issues,
>> please let us know.
>>
>>
>>> Cheers,
>>>
>>> Richard
>>>
>>> 
>>
>

[-- Attachment #2: Type: text/html, Size: 4798 bytes --]

Re: [docs] [sphinx] updates as of Sep 10th.

[Richard Purdie]

On Tue, 2020-09-15 at 18:48 +0200, Nicolas Dechesne wrote:
> I've finished profile-manual today. And excluded adt-manual from
> Sphinx (deprecated doc). At this point, we have:
> Richard on kernel-dev

This is now done.

I've also gone back over the reference manual and fixed a ton of issues
I missed the first time as I had a better process for it.

> Mark on test-manual
> Armin on overview-manual
> Tim on toaster-manual
> 
> Which means everything is now covered! Please send patches on the
> list for review, and if you have any trouble with Sphinx , let me
> know!

I've split the migration guide into separate files which should avoid
warnings. I've also tweaked the css handling of "em" handling thanks to
tips from Nico, to better match the existing style.

One problem I've noticed is that example blocks don't need "\" escaping
and so we have a ton of "\\" loose in them we now need to fix.

Its looking ever better though, quite excited to see it coming
together.

Cheers,

Richard

Re: [docs] [sphinx] updates as of Sep 10th.

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 1331 bytes --]

On Tue, Sep 15, 2020 at 3:17 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Tue, 2020-09-15 at 18:48 +0200, Nicolas Dechesne wrote:
>
> > I've finished profile-manual today. And excluded adt-manual from
>
> > Sphinx (deprecated doc). At this point, we have:
>
> > Richard on kernel-dev
>
>
>
> This is now done.
>
>
>
> I've also gone back over the reference manual and fixed a ton of issues
>
> I missed the first time as I had a better process for it.
>
>
>
> > Mark on test-manual
>
> > Armin on overview-manual
>
> > Tim on toaster-manual
>
> >
>
> > Which means everything is now covered! Please send patches on the
>
> > list for review, and if you have any trouble with Sphinx , let me
>
> > know!
>
>
>
> I've split the migration guide into separate files which should avoid
>
> warnings. I've also tweaked the css handling of "em" handling thanks to
>
> tips from Nico, to better match the existing style.
>
>
>
> One problem I've noticed is that example blocks don't need "\" escaping
>
> and so we have a ton of "\\" loose in them we now need to fix.
>

I noticed this during my toaster-manual edits.


>
>
> Its looking ever better though, quite excited to see it coming
>
> together.
>
>
>
> Cheers,
>
>
>
> Richard
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 1913 bytes --]