ksummit.lists.linux.dev archive mirror
 help / color / mirror / Atom feed
* [TECH TOPIC] What kernel documentation could be
@ 2022-06-17 20:57 Jonathan Corbet
  2022-06-17 20:57 ` [Ksummit-discuss] " Jonathan Corbet
                   ` (3 more replies)
  0 siblings, 4 replies; 40+ messages in thread
From: Jonathan Corbet @ 2022-06-17 20:57 UTC (permalink / raw)
  To: ksummit-discuss, ksummit

The development community has put a lot of work into the kernel's
documentation directory in recent years, with visible results. But the
kernel's documentation still falls far short of the standard set by many
other projects, and there is a great deal of "tribal knowledge" in our
community that is not set down. I'd like to look at the successes and
failures of the work so far, but intent to focus on a discussion of what
our documentation should be and what we can do to get it there.

Questions to discuss include:

 - Bringing an coherent overall structure to Documentation/

 - Making it easier for developers to improve the documentation.

 - What we would like from Sphinx and what we can do to make it happen

 - Making the docs build system less ugly and more maintainable

 - Integrating rustdoc documentation

jon

^ permalink raw reply	[flat|nested] 40+ messages in thread

* [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-17 20:57 [TECH TOPIC] What kernel documentation could be Jonathan Corbet
@ 2022-06-17 20:57 ` Jonathan Corbet
  2022-06-17 21:48 ` Laurent Pinchart
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 40+ messages in thread
From: Jonathan Corbet @ 2022-06-17 20:57 UTC (permalink / raw)


The development community has put a lot of work into the kernel's
documentation directory in recent years, with visible results. But the
kernel's documentation still falls far short of the standard set by many
other projects, and there is a great deal of "tribal knowledge" in our
community that is not set down. I'd like to look at the successes and
failures of the work so far, but intent to focus on a discussion of what
our documentation should be and what we can do to get it there.

Questions to discuss include:

 - Bringing an coherent overall structure to Documentation/

 - Making it easier for developers to improve the documentation.

 - What we would like from Sphinx and what we can do to make it happen

 - Making the docs build system less ugly and more maintainable

 - Integrating rustdoc documentation

jon

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-17 20:57 [TECH TOPIC] What kernel documentation could be Jonathan Corbet
  2022-06-17 20:57 ` [Ksummit-discuss] " Jonathan Corbet
@ 2022-06-17 21:48 ` Laurent Pinchart
  2022-06-17 21:48   ` Laurent Pinchart
  2022-06-27 15:18   ` Jonathan Corbet
  2022-06-18  8:24 ` Mauro Carvalho Chehab
  2022-06-25 17:43 ` Steven Rostedt
  3 siblings, 2 replies; 40+ messages in thread
From: Laurent Pinchart @ 2022-06-17 21:48 UTC (permalink / raw)
  To: Jonathan Corbet; +Cc: ksummit-discuss, ksummit

Hi Jon,

On Fri, Jun 17, 2022 at 02:57:10PM -0600, Jonathan Corbet wrote:
> The development community has put a lot of work into the kernel's
> documentation directory in recent years, with visible results. But the
> kernel's documentation still falls far short of the standard set by many
> other projects, and there is a great deal of "tribal knowledge" in our
> community that is not set down. I'd like to look at the successes and
> failures of the work so far, but intent to focus on a discussion of what
> our documentation should be and what we can do to get it there.
> 
> Questions to discuss include:
> 
>  - Bringing an coherent overall structure to Documentation/
> 
>  - Making it easier for developers to improve the documentation.

I've been wondering recently if it would help to have a place where we
can record documentation wishes.

It has happened quite a few times that my searches for particular pieces
of documentation in the kernel were unfruitful, or that the related
documentation left me none the wiser (one example, which is just an
example and not finger-pointing, is the runtime PM documentation in
Documentation/power/runtime_pm.rst that feels like reading a
calculus-related PHD thesis when all you want is to understand how to
add two numbers). I rarely had time to write documentation patches as a
result, and my pains were thus left unnoticed by maintainers and
developers, who may be willing to improve the documentation if they knew
that users were struggling.

>  - What we would like from Sphinx and what we can do to make it happen
> 
>  - Making the docs build system less ugly and more maintainable
> 
>  - Integrating rustdoc documentation

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 40+ messages in thread

* [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-17 21:48 ` Laurent Pinchart
@ 2022-06-17 21:48   ` Laurent Pinchart
  2022-06-27 15:18   ` Jonathan Corbet
  1 sibling, 0 replies; 40+ messages in thread
From: Laurent Pinchart @ 2022-06-17 21:48 UTC (permalink / raw)


Hi Jon,

On Fri, Jun 17, 2022 at 02:57:10PM -0600, Jonathan Corbet wrote:
> The development community has put a lot of work into the kernel's
> documentation directory in recent years, with visible results. But the
> kernel's documentation still falls far short of the standard set by many
> other projects, and there is a great deal of "tribal knowledge" in our
> community that is not set down. I'd like to look at the successes and
> failures of the work so far, but intent to focus on a discussion of what
> our documentation should be and what we can do to get it there.
> 
> Questions to discuss include:
> 
>  - Bringing an coherent overall structure to Documentation/
> 
>  - Making it easier for developers to improve the documentation.

I've been wondering recently if it would help to have a place where we
can record documentation wishes.

It has happened quite a few times that my searches for particular pieces
of documentation in the kernel were unfruitful, or that the related
documentation left me none the wiser (one example, which is just an
example and not finger-pointing, is the runtime PM documentation in
Documentation/power/runtime_pm.rst that feels like reading a
calculus-related PHD thesis when all you want is to understand how to
add two numbers). I rarely had time to write documentation patches as a
result, and my pains were thus left unnoticed by maintainers and
developers, who may be willing to improve the documentation if they knew
that users were struggling.

>  - What we would like from Sphinx and what we can do to make it happen
> 
>  - Making the docs build system less ugly and more maintainable
> 
>  - Integrating rustdoc documentation

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-17 20:57 [TECH TOPIC] What kernel documentation could be Jonathan Corbet
  2022-06-17 20:57 ` [Ksummit-discuss] " Jonathan Corbet
  2022-06-17 21:48 ` Laurent Pinchart
@ 2022-06-18  8:24 ` Mauro Carvalho Chehab
  2022-06-18  8:24   ` Mauro Carvalho Chehab
                     ` (2 more replies)
  2022-06-25 17:43 ` Steven Rostedt
  3 siblings, 3 replies; 40+ messages in thread
From: Mauro Carvalho Chehab @ 2022-06-18  8:24 UTC (permalink / raw)
  To: Jonathan Corbet; +Cc: ksummit-discuss, ksummit

Hi John,

Em Fri, 17 Jun 2022 14:57:10 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> The development community has put a lot of work into the kernel's
> documentation directory in recent years, with visible results. But the
> kernel's documentation still falls far short of the standard set by many
> other projects, and there is a great deal of "tribal knowledge" in our
> community that is not set down. I'd like to look at the successes and
> failures of the work so far, but intent to focus on a discussion of what
> our documentation should be and what we can do to get it there.
> 
> Questions to discuss include:
> 
>  - Bringing an coherent overall structure to Documentation/

Fully agreed. We dedicated a lot of resources on the past years to
convert documentation to ReST and on kernel-doc, but not so much
on placing the topics at the right order.

There are a huge amount of such documentation that describe border cases,
or have just kernel-doc tags without any (or very small) descriptions.
Kernel-doc markups are important, as they help to keep the documentation
updated, but explaining subsystem's principles is equally important, as
it can save a lot of time from maintainers if the contributors are aware
on how and why the subsystem's kAPIs were organized the way it is.

We should imagine documentation as if this is a series of books describing
every aspect of the kernel. So, I would expect it to be organized with
some structure that would place the topics on some order that would make
easier for people to read, being either new contributors or experienced
developers that need to touch other parts of the Kernel that are out of
his past experiences.

For instance to the kAPI documentation, I would expect something like:

	- How to contribute / Submission process;
	- An introductory chapter that would describe the most used
	  components that all developers need to know, like:

		- how to write a driver;
		- fs principles;
		- scheduler principles;
		- mm most used functions (kmalloc & friends);
		- wait queues;
		- kernel threads;
		- softirqs;
		- arch-dependent items to consider (like bit order), weak
		  memory model, etc;
		- ...

	  IMO, we could take a look at the index of some already-existing 
	  books like LDD 3, Linux Kernel Development and others in order to
	  get a rough idea about the items to be covered there.
	- Bus-related subsystems (USB, PCI, I2C, ...);
	- Then, place per-subsystem's documentation, all having their
	  texts explaining basic principles.

IMO, we should write an index file and a couple of new ReST files with an
skeleton for the above, and then ask people to help filling the blanks.

>  - Making it easier for developers to improve the documentation.

A well-prepared skeleton would make life easier. From time to time, we
have discussions and patches shifting documentation between different
directories.

>  - What we would like from Sphinx and what we can do to make it happen

The big missing feature on Sphinx itself is with regards to per-C-type
domain. So, if we have one struct and one function both called "foo",
the cross-references will be broken. This issue is known since Sphinx
3.1, and there are already patches fixing it since then (I remember
testing them) but, up to today, the Sphinx upstream patches meant
to fix it weren't applied yet (as far as I can tell).

Another problem is with regards to the documentation's build time.

One feature we're not using yet is intersphinx. Right now, we can
build parts of the documentation with:

	make SPHINXDIRS="foo" htmldocs

But, if "foo" have cross-references to "bar", the build will produce
warnings and the documentation's cross-references to "bar" won't work.

Properly setting interphinx would allow this to work, as such references
would point, instead, to some remote place (like kernel.org).

This could be used to help improving the documentation's build time 
during doc development.

Also, I would love to have a good way to just produce html (and pdf) from
the documents I'm actually working with. The way I do it right now is
that I create a "Documentation/foo" directory, adding there just the docs
I'm currently working with, placing them on a fake index.rst file.

This way, I can build them really fast, without needing to rebuild
everything at the same book. Perhaps something like that could be 
automated - or Sphinx itself could support something like:

	make htmldocs SPHINXFILES="foo.rst bar.rst"

>  - Making the docs build system less ugly and more maintainable

I guess shifting the minimal Sphinx version would help to remove some
bad things there.

One thing that probably be solved on a different way is to have
a better solution for things like:

	Functions for feature 1
	=======================

	.. kernel-doc:: include/some_header.h
	   :doc: Feature 1

	.. kernel-doc:: include/some_header.h
	   :functions:
		func1
		func2

	Functions for feature 2
	=======================

	.. kernel-doc:: include/some_header.h
	   :doc: Feature 2

	.. kernel-doc:: include/some_header.h
	   :functions:
		func3
		func4

Perhaps we could change Kernel-doc in a way that doing just:

	.. kernel-doc:: include/some_header.h

would be enough.

> - Integrating rustdoc documentation

Won't comment much about that, as never touched any of those.

Life would be a lot easier on this side if rustdoc could
support ReST.

Regards,
Mauro

^ permalink raw reply	[flat|nested] 40+ messages in thread

* [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-18  8:24 ` Mauro Carvalho Chehab
@ 2022-06-18  8:24   ` Mauro Carvalho Chehab
  2022-06-18 11:03   ` Miguel Ojeda
  2022-06-23  9:18   ` Jani Nikula
  2 siblings, 0 replies; 40+ messages in thread
From: Mauro Carvalho Chehab @ 2022-06-18  8:24 UTC (permalink / raw)


Hi John,

Em Fri, 17 Jun 2022 14:57:10 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> The development community has put a lot of work into the kernel's
> documentation directory in recent years, with visible results. But the
> kernel's documentation still falls far short of the standard set by many
> other projects, and there is a great deal of "tribal knowledge" in our
> community that is not set down. I'd like to look at the successes and
> failures of the work so far, but intent to focus on a discussion of what
> our documentation should be and what we can do to get it there.
> 
> Questions to discuss include:
> 
>  - Bringing an coherent overall structure to Documentation/

Fully agreed. We dedicated a lot of resources on the past years to
convert documentation to ReST and on kernel-doc, but not so much
on placing the topics at the right order.

There are a huge amount of such documentation that describe border cases,
or have just kernel-doc tags without any (or very small) descriptions.
Kernel-doc markups are important, as they help to keep the documentation
updated, but explaining subsystem's principles is equally important, as
it can save a lot of time from maintainers if the contributors are aware
on how and why the subsystem's kAPIs were organized the way it is.

We should imagine documentation as if this is a series of books describing
every aspect of the kernel. So, I would expect it to be organized with
some structure that would place the topics on some order that would make
easier for people to read, being either new contributors or experienced
developers that need to touch other parts of the Kernel that are out of
his past experiences.

For instance to the kAPI documentation, I would expect something like:

	- How to contribute / Submission process;
	- An introductory chapter that would describe the most used
	  components that all developers need to know, like:

		- how to write a driver;
		- fs principles;
		- scheduler principles;
		- mm most used functions (kmalloc & friends);
		- wait queues;
		- kernel threads;
		- softirqs;
		- arch-dependent items to consider (like bit order), weak
		  memory model, etc;
		- ...

	  IMO, we could take a look at the index of some already-existing 
	  books like LDD 3, Linux Kernel Development and others in order to
	  get a rough idea about the items to be covered there.
	- Bus-related subsystems (USB, PCI, I2C, ...);
	- Then, place per-subsystem's documentation, all having their
	  texts explaining basic principles.

IMO, we should write an index file and a couple of new ReST files with an
skeleton for the above, and then ask people to help filling the blanks.

>  - Making it easier for developers to improve the documentation.

A well-prepared skeleton would make life easier. From time to time, we
have discussions and patches shifting documentation between different
directories.

>  - What we would like from Sphinx and what we can do to make it happen

The big missing feature on Sphinx itself is with regards to per-C-type
domain. So, if we have one struct and one function both called "foo",
the cross-references will be broken. This issue is known since Sphinx
3.1, and there are already patches fixing it since then (I remember
testing them) but, up to today, the Sphinx upstream patches meant
to fix it weren't applied yet (as far as I can tell).

Another problem is with regards to the documentation's build time.

One feature we're not using yet is intersphinx. Right now, we can
build parts of the documentation with:

	make SPHINXDIRS="foo" htmldocs

But, if "foo" have cross-references to "bar", the build will produce
warnings and the documentation's cross-references to "bar" won't work.

Properly setting interphinx would allow this to work, as such references
would point, instead, to some remote place (like kernel.org).

This could be used to help improving the documentation's build time 
during doc development.

Also, I would love to have a good way to just produce html (and pdf) from
the documents I'm actually working with. The way I do it right now is
that I create a "Documentation/foo" directory, adding there just the docs
I'm currently working with, placing them on a fake index.rst file.

This way, I can build them really fast, without needing to rebuild
everything at the same book. Perhaps something like that could be 
automated - or Sphinx itself could support something like:

	make htmldocs SPHINXFILES="foo.rst bar.rst"

>  - Making the docs build system less ugly and more maintainable

I guess shifting the minimal Sphinx version would help to remove some
bad things there.

One thing that probably be solved on a different way is to have
a better solution for things like:

	Functions for feature 1
	=======================

	.. kernel-doc:: include/some_header.h
	   :doc: Feature 1

	.. kernel-doc:: include/some_header.h
	   :functions:
		func1
		func2

	Functions for feature 2
	=======================

	.. kernel-doc:: include/some_header.h
	   :doc: Feature 2

	.. kernel-doc:: include/some_header.h
	   :functions:
		func3
		func4

Perhaps we could change Kernel-doc in a way that doing just:

	.. kernel-doc:: include/some_header.h

would be enough.

> - Integrating rustdoc documentation

Won't comment much about that, as never touched any of those.

Life would be a lot easier on this side if rustdoc could
support ReST.

Regards,
Mauro

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-18  8:24 ` Mauro Carvalho Chehab
  2022-06-18  8:24   ` Mauro Carvalho Chehab
@ 2022-06-18 11:03   ` Miguel Ojeda
  2022-06-18 11:16     ` Mauro Carvalho Chehab
  2022-06-23  9:18   ` Jani Nikula
  2 siblings, 1 reply; 40+ messages in thread
From: Miguel Ojeda @ 2022-06-18 11:03 UTC (permalink / raw)
  To: Mauro Carvalho Chehab; +Cc: Jonathan Corbet, ksummit, ksummit

On Sat, Jun 18, 2022 at 10:24 AM Mauro Carvalho Chehab
<mchehab@kernel.org> wrote:
>
> Life would be a lot easier on this side if rustdoc could
> support ReST.

Not sure what you mean by this. Do you mean writing ReST for Rust
documentation? For the files inside `Documentation/rust/`, we already
use ReST [*] like the rest. For the source code docs (the ones handled
by rustdoc), it outputs HTML. Or do you mean using rustdoc to generate
`Doc/`s or similar?

[*] Though it would be nice to be able to use Markdown (but that is a
different topic).

Cheers,
Miguel

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-18 11:03   ` Miguel Ojeda
@ 2022-06-18 11:16     ` Mauro Carvalho Chehab
  2022-06-18 11:16       ` Mauro Carvalho Chehab
  2022-06-18 14:37       ` Miguel Ojeda
  0 siblings, 2 replies; 40+ messages in thread
From: Mauro Carvalho Chehab @ 2022-06-18 11:16 UTC (permalink / raw)
  To: Miguel Ojeda; +Cc: Jonathan Corbet, ksummit, ksummit

Em Sat, 18 Jun 2022 13:03:38 +0200
Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> escreveu:

> On Sat, Jun 18, 2022 at 10:24 AM Mauro Carvalho Chehab
> <mchehab@kernel.org> wrote:
> >
> > Life would be a lot easier on this side if rustdoc could
> > support ReST.  
> 
> Not sure what you mean by this. Do you mean writing ReST for Rust
> documentation? For the files inside `Documentation/rust/`, we already
> use ReST [*] like the rest. For the source code docs (the ones handled
> by rustdoc), it outputs HTML. Or do you mean using rustdoc to generate
> `Doc/`s or similar?

I meant that rustdoc could generate ReST output, being included inside
the Kernel documentation on a way similar to kerneldoc. This would
integrate better with our building system, which can produce output docs
on 3 different formats (html, epub and pdf).

Regards,
Mauro


> 
> [*] Though it would be nice to be able to use Markdown (but that is a
> different topic).
> 
> Cheers,
> Miguel

^ permalink raw reply	[flat|nested] 40+ messages in thread

* [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-18 11:16     ` Mauro Carvalho Chehab
@ 2022-06-18 11:16       ` Mauro Carvalho Chehab
  2022-06-18 14:37       ` Miguel Ojeda
  1 sibling, 0 replies; 40+ messages in thread
From: Mauro Carvalho Chehab @ 2022-06-18 11:16 UTC (permalink / raw)


Em Sat, 18 Jun 2022 13:03:38 +0200
Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> escreveu:

> On Sat, Jun 18, 2022 at 10:24 AM Mauro Carvalho Chehab
> <mchehab@kernel.org> wrote:
> >
> > Life would be a lot easier on this side if rustdoc could
> > support ReST.  
> 
> Not sure what you mean by this. Do you mean writing ReST for Rust
> documentation? For the files inside `Documentation/rust/`, we already
> use ReST [*] like the rest. For the source code docs (the ones handled
> by rustdoc), it outputs HTML. Or do you mean using rustdoc to generate
> `Doc/`s or similar?

I meant that rustdoc could generate ReST output, being included inside
the Kernel documentation on a way similar to kerneldoc. This would
integrate better with our building system, which can produce output docs
on 3 different formats (html, epub and pdf).

Regards,
Mauro


> 
> [*] Though it would be nice to be able to use Markdown (but that is a
> different topic).
> 
> Cheers,
> Miguel

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-18 11:16     ` Mauro Carvalho Chehab
  2022-06-18 11:16       ` Mauro Carvalho Chehab
@ 2022-06-18 14:37       ` Miguel Ojeda
  1 sibling, 0 replies; 40+ messages in thread
From: Miguel Ojeda @ 2022-06-18 14:37 UTC (permalink / raw)
  To: Mauro Carvalho Chehab; +Cc: Jonathan Corbet, ksummit, ksummit

On Sat, Jun 18, 2022 at 1:16 PM Mauro Carvalho Chehab
<mchehab@kernel.org> wrote:
>
> I meant that rustdoc could generate ReST output, being included inside
> the Kernel documentation on a way similar to kerneldoc. This would
> integrate better with our building system, which can produce output docs
> on 3 different formats (html, epub and pdf).

That may lose some of the advantages of the rustdoc output (tailored
to Rust code, navigable source code view, etc.).

It may be possible to encode a lot of the information in ReST and then
replicate functionality with Sphinx/plugins, possibly having a Rust
domain etc., but I am not sure it would be worth it (for the HTML
output at least).

Cheers,
Miguel

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-18  8:24 ` Mauro Carvalho Chehab
  2022-06-18  8:24   ` Mauro Carvalho Chehab
  2022-06-18 11:03   ` Miguel Ojeda
@ 2022-06-23  9:18   ` Jani Nikula
  2022-06-23  9:57     ` Mauro Carvalho Chehab
  2 siblings, 1 reply; 40+ messages in thread
From: Jani Nikula @ 2022-06-23  9:18 UTC (permalink / raw)
  To: Mauro Carvalho Chehab, Jonathan Corbet; +Cc: ksummit-discuss, ksummit

On Sat, 18 Jun 2022, Mauro Carvalho Chehab <mchehab@kernel.org> wrote:
> The big missing feature on Sphinx itself is with regards to per-C-type
> domain. So, if we have one struct and one function both called "foo",
> the cross-references will be broken. This issue is known since Sphinx
> 3.1, and there are already patches fixing it since then (I remember
> testing them) but, up to today, the Sphinx upstream patches meant
> to fix it weren't applied yet (as far as I can tell).

https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#namespacing

Integrating that needs to be done carefully, though, to not make a mess
of it.

> One thing that probably be solved on a different way is to have
> a better solution for things like:
>
> 	Functions for feature 1
> 	=======================
>
> 	.. kernel-doc:: include/some_header.h
> 	   :doc: Feature 1
>
> 	.. kernel-doc:: include/some_header.h
> 	   :functions:
> 		func1
> 		func2
>
> 	Functions for feature 2
> 	=======================
>
> 	.. kernel-doc:: include/some_header.h
> 	   :doc: Feature 2
>
> 	.. kernel-doc:: include/some_header.h
> 	   :functions:
> 		func3
> 		func4

Yeah, currently that leads to parsing the header four times by
kernel-doc the script. The solution would be to finally convert the
script to a proper python Sphinx extension that can do caching. (This is
how it works in Hawkmoth, FWIW.)

> Perhaps we could change Kernel-doc in a way that doing just:
>
> 	.. kernel-doc:: include/some_header.h
>
> would be enough.

The order in nicely flowing documentation is not necessarily the same as
the order in nicely flowing source code. I expect it to be much more
acceptable to tweak the rst to achieve this than to do source code
rearrangement to generate nice documentation.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Graphics Center

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-23  9:18   ` Jani Nikula
@ 2022-06-23  9:57     ` Mauro Carvalho Chehab
  2022-06-23 10:30       ` Jani Nikula
  2022-06-23 13:40       ` Jonathan Corbet
  0 siblings, 2 replies; 40+ messages in thread
From: Mauro Carvalho Chehab @ 2022-06-23  9:57 UTC (permalink / raw)
  To: Jani Nikula; +Cc: Jonathan Corbet, ksummit-discuss, ksummit

Em Thu, 23 Jun 2022 12:18:50 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Sat, 18 Jun 2022, Mauro Carvalho Chehab <mchehab@kernel.org> wrote:
> > The big missing feature on Sphinx itself is with regards to per-C-type
> > domain. So, if we have one struct and one function both called "foo",
> > the cross-references will be broken. This issue is known since Sphinx
> > 3.1, and there are already patches fixing it since then (I remember
> > testing them) but, up to today, the Sphinx upstream patches meant
> > to fix it weren't applied yet (as far as I can tell).  
> 
> https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#namespacing
> 
> Integrating that needs to be done carefully, though, to not make a mess
> of it.
> 
> > One thing that probably be solved on a different way is to have
> > a better solution for things like:
> >
> > 	Functions for feature 1
> > 	=======================
> >
> > 	.. kernel-doc:: include/some_header.h
> > 	   :doc: Feature 1
> >
> > 	.. kernel-doc:: include/some_header.h
> > 	   :functions:
> > 		func1
> > 		func2
> >
> > 	Functions for feature 2
> > 	=======================
> >
> > 	.. kernel-doc:: include/some_header.h
> > 	   :doc: Feature 2
> >
> > 	.. kernel-doc:: include/some_header.h
> > 	   :functions:
> > 		func3
> > 		func4  
> 
> Yeah, currently that leads to parsing the header four times by
> kernel-doc the script.

Yes.

> The solution would be to finally convert the
> script to a proper python Sphinx extension that can do caching. (This is
> how it works in Hawkmoth, FWIW.)

That's one solution, but see: there is already a python extension
that currently calls kernel-doc everytime. It could, instead,
cache the rst returned by its first run (or a parsed version of it)
and use the cached results the other 3 times.

Porting kernel-doc to python could be doable, but not trivial, due to several
reasons:

- it should keep running standalone, as otherwise debugging parsing issues
  on kernel-doc would be a lot harder. In particular, kernel-doc --none is
  really helpful to report kernel-doc tag errors;
- regressions will likely be introduced on a change like that;
- regular contributors to kernel-doc will need to ramp up with the newer
  version;
- a port like that could increase the script run time, as the
  optimizations and regular expressions there could behave different on
  python.

> > Perhaps we could change Kernel-doc in a way that doing just:
> >
> > 	.. kernel-doc:: include/some_header.h
> >
> > would be enough.  
> 
> The order in nicely flowing documentation is not necessarily the same as
> the order in nicely flowing source code. I expect it to be much more
> acceptable to tweak the rst to achieve this than to do source code
> rearrangement to generate nice documentation.

True, but independently if the script would be rewritten in python or not,
one way would be to enrich the 'DOCS:' kernel-doc tag in order to mention 
there the symbols that belong to each part of the document, e. g. something
like:

	/**
	 * DOC: foo
	 *
	 * Some comments...
	 *
	 * symbols:
	 *     foo
	 *     bar
	 */

One advantage is that all documentation will be on a single place,
so hopefully it would be easier to maintain.

Also, on documents using `DOC:` with such new `symbols` tag, kernel-doc 
could validate if all documented symbols are singly included at all `DOC:`
sessions and if any symbols are missed/renamed/removed.

Regards,
Mauro

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-23  9:57     ` Mauro Carvalho Chehab
@ 2022-06-23 10:30       ` Jani Nikula
  2022-06-23 13:40       ` Jonathan Corbet
  1 sibling, 0 replies; 40+ messages in thread
From: Jani Nikula @ 2022-06-23 10:30 UTC (permalink / raw)
  To: Mauro Carvalho Chehab; +Cc: Jonathan Corbet, ksummit-discuss, ksummit

On Thu, 23 Jun 2022, Mauro Carvalho Chehab <mchehab@kernel.org> wrote:
> Em Thu, 23 Jun 2022 12:18:50 +0300
> Jani Nikula <jani.nikula@intel.com> escreveu:
>
>> On Sat, 18 Jun 2022, Mauro Carvalho Chehab <mchehab@kernel.org> wrote:
>> > The big missing feature on Sphinx itself is with regards to per-C-type
>> > domain. So, if we have one struct and one function both called "foo",
>> > the cross-references will be broken. This issue is known since Sphinx
>> > 3.1, and there are already patches fixing it since then (I remember
>> > testing them) but, up to today, the Sphinx upstream patches meant
>> > to fix it weren't applied yet (as far as I can tell).  
>> 
>> https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#namespacing
>> 
>> Integrating that needs to be done carefully, though, to not make a mess
>> of it.
>> 
>> > One thing that probably be solved on a different way is to have
>> > a better solution for things like:
>> >
>> > 	Functions for feature 1
>> > 	=======================
>> >
>> > 	.. kernel-doc:: include/some_header.h
>> > 	   :doc: Feature 1
>> >
>> > 	.. kernel-doc:: include/some_header.h
>> > 	   :functions:
>> > 		func1
>> > 		func2
>> >
>> > 	Functions for feature 2
>> > 	=======================
>> >
>> > 	.. kernel-doc:: include/some_header.h
>> > 	   :doc: Feature 2
>> >
>> > 	.. kernel-doc:: include/some_header.h
>> > 	   :functions:
>> > 		func3
>> > 		func4  
>> 
>> Yeah, currently that leads to parsing the header four times by
>> kernel-doc the script.
>
> Yes.
>
>> The solution would be to finally convert the
>> script to a proper python Sphinx extension that can do caching. (This is
>> how it works in Hawkmoth, FWIW.)
>
> That's one solution, but see: there is already a python extension
> that currently calls kernel-doc everytime. It could, instead,
> cache the rst returned by its first run (or a parsed version of it)
> and use the cached results the other 3 times.

This would mean having kernel-doc the perl script output meta
information about the constructs, complicating it even more.

> Porting kernel-doc to python could be doable, but not trivial, due to several
> reasons:
>
> - it should keep running standalone, as otherwise debugging parsing issues
>   on kernel-doc would be a lot harder. In particular, kernel-doc --none is
>   really helpful to report kernel-doc tag errors;

This is trivial and sensible to do in the python extension too.

> - regressions will likely be introduced on a change like that;

Any change can cause regressions... but previously a diff on the html
outputs generated by both versions has been an effective way of
verifying the changes.

> - regular contributors to kernel-doc will need to ramp up with the newer
>   version;

You say it like it's a bad thing. :p

Sometimes I regret not throwing kernel-doc the script in the curb while
doing the Sphinx conversion.

> - a port like that could increase the script run time, as the
>   optimizations and regular expressions there could behave different on
>   python.

This is certainly possible. It's somewhat countered by the python
extension being able to cache stuff.

>
>> > Perhaps we could change Kernel-doc in a way that doing just:
>> >
>> > 	.. kernel-doc:: include/some_header.h
>> >
>> > would be enough.  
>> 
>> The order in nicely flowing documentation is not necessarily the same as
>> the order in nicely flowing source code. I expect it to be much more
>> acceptable to tweak the rst to achieve this than to do source code
>> rearrangement to generate nice documentation.
>
> True, but independently if the script would be rewritten in python or not,
> one way would be to enrich the 'DOCS:' kernel-doc tag in order to mention 
> there the symbols that belong to each part of the document, e. g. something
> like:
>
> 	/**
> 	 * DOC: foo
> 	 *
> 	 * Some comments...
> 	 *
> 	 * symbols:
> 	 *     foo
> 	 *     bar
> 	 */
>

I believe preprocessing of what's supposed to be rst leads to
problems. It's difficult to define proper escaping to ensure stuff gets
parsed at the correct stage in the correct way. It should just be rst.

Part of the reason the old docbook thing was such a big mess were the
"impedance mismatches" at each separate stage of processing.

In general, stick to the mechanisms provided by rst and Sphinx. Extend
on them using Sphinx extensions, not by manually parsing rst.

BR,
Jani.

> One advantage is that all documentation will be on a single place,
> so hopefully it would be easier to maintain.
>
> Also, on documents using `DOC:` with such new `symbols` tag, kernel-doc 
> could validate if all documented symbols are singly included at all `DOC:`
> sessions and if any symbols are missed/renamed/removed.
>
> Regards,
> Mauro

-- 
Jani Nikula, Intel Open Source Graphics Center

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-23  9:57     ` Mauro Carvalho Chehab
  2022-06-23 10:30       ` Jani Nikula
@ 2022-06-23 13:40       ` Jonathan Corbet
  2022-06-24  7:33         ` Mauro Carvalho Chehab
  1 sibling, 1 reply; 40+ messages in thread
From: Jonathan Corbet @ 2022-06-23 13:40 UTC (permalink / raw)
  To: Mauro Carvalho Chehab, Jani Nikula; +Cc: ksummit-discuss, ksummit

Mauro Carvalho Chehab <mchehab@kernel.org> writes:

>> The solution would be to finally convert the
>> script to a proper python Sphinx extension that can do caching. (This is
>> how it works in Hawkmoth, FWIW.)

I've been pondering on this for a bit, and would like to do it, but I
don't know when I might find the time for it.

> That's one solution, but see: there is already a python extension
> that currently calls kernel-doc everytime. It could, instead,
> cache the rst returned by its first run (or a parsed version of it)
> and use the cached results the other 3 times.
>
> Porting kernel-doc to python could be doable, but not trivial, due to several
> reasons:
>
> - it should keep running standalone, as otherwise debugging parsing issues
>   on kernel-doc would be a lot harder. In particular, kernel-doc --none is
>   really helpful to report kernel-doc tag errors;

Yes, of course.  As Jani noted, that's just how you would do it, not a
problem. 

> - regressions will likely be introduced on a change like that;

The nice thing is ... we already have a really nice regression test in
the form of the current docs build and diff.

> - regular contributors to kernel-doc will need to ramp up with the newer
>   version;

We have those?  That script is a nightmare and nobody goes near it if
they can possibly avoid it.  I would expect to have more contributors
with a decent Python version that doesn't include 25 years of regex
accretion.

> - a port like that could increase the script run time, as the
>   optimizations and regular expressions there could behave different on
>   python.

It could also decrease it by improving caching opportunities, getting
rid of a lot of fork()/exec() pairs and Perl interpreter startups, etc.

I've actually, in a spare moment or two, been doing some profiling of
the kernel docs build and trying to track down the sources of the
slowness.  I am thinking that nearly 700 *million* calls to the iterator
for the C-domain symbol list might have something to do with it...

> True, but independently if the script would be rewritten in python or not,
> one way would be to enrich the 'DOCS:' kernel-doc tag in order to mention 
> there the symbols that belong to each part of the document, e. g. something
> like:
>
> 	/**
> 	 * DOC: foo
> 	 *
> 	 * Some comments...
> 	 *
> 	 * symbols:
> 	 *     foo
> 	 *     bar
> 	 */
>
> One advantage is that all documentation will be on a single place,
> so hopefully it would be easier to maintain.

I'm not quite sure I get this...you want to put the TOC tree in the
source comments?  This looks like the kind of thing that nobody ever
remembers to update, but maybe I'm missing something.

Thanks,

jon


^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-23 13:40       ` Jonathan Corbet
@ 2022-06-24  7:33         ` Mauro Carvalho Chehab
  2022-06-24 16:37           ` Markus Heiser
  2022-06-24 22:57           ` Jonathan Corbet
  0 siblings, 2 replies; 40+ messages in thread
From: Mauro Carvalho Chehab @ 2022-06-24  7:33 UTC (permalink / raw)
  To: Jonathan Corbet; +Cc: Jani Nikula, ksummit-discuss, ksummit, Markus Heiser

Em Thu, 23 Jun 2022 07:40:45 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab@kernel.org> writes:
> 
> >> The solution would be to finally convert the
> >> script to a proper python Sphinx extension that can do caching. (This is
> >> how it works in Hawkmoth, FWIW.)  
> 
> I've been pondering on this for a bit, and would like to do it, but I
> don't know when I might find the time for it.

There is already a version of kernel-doc written in Python, made by
Markus Heiser:

	https://github.com/return42/linuxdoc/blob/master/linuxdoc/kernel_doc.py

It could be a starting point.

> > That's one solution, but see: there is already a python extension
> > that currently calls kernel-doc everytime. It could, instead,
> > cache the rst returned by its first run (or a parsed version of it)
> > and use the cached results the other 3 times.
> >
> > Porting kernel-doc to python could be doable, but not trivial, due to several
> > reasons:
> >
> > - it should keep running standalone, as otherwise debugging parsing issues
> >   on kernel-doc would be a lot harder. In particular, kernel-doc --none is
> >   really helpful to report kernel-doc tag errors;  
> 
> Yes, of course.  As Jani noted, that's just how you would do it, not a
> problem. 
> 
> > - regressions will likely be introduced on a change like that;  
> 
> The nice thing is ... we already have a really nice regression test in
> the form of the current docs build and diff.

True. We can test the results with both versions and even check how
performance is affected.

> > - regular contributors to kernel-doc will need to ramp up with the newer
> >   version;  
> 
> We have those?  That script is a nightmare and nobody goes near it if
> they can possibly avoid it.  I would expect to have more contributors
> with a decent Python version that doesn't include 25 years of regex
> accretion.

Well, the kernel-doc version from Markus in python is ~23% bigger
than our current kernel-doc. Granted, it could be due to comments and
blank lines, but basically, the same regexes that are in perl would
also be need to be replicated in python, as parsing a C code with
regexes is not a trivial task.

So, in terms of complexity, I doubt much would change by porting it
to python.

Now, one of the things that the kernel-doc does is that it has the
parse code and the output logic, which actually has an an abstraction
to let it to produce results on different formats (currently, none, man
and rst - we dropped some other formats from it). This abstraction
increases its complexity. This is something that doesn't need to be
replicated on any ports. 

> > - a port like that could increase the script run time, as the
> >   optimizations and regular expressions there could behave different on
> >   python.  
> 
> It could also decrease it by improving caching opportunities, getting
> rid of a lot of fork()/exec() pairs and Perl interpreter startups, etc.
> 
> I've actually, in a spare moment or two, been doing some profiling of
> the kernel docs build and trying to track down the sources of the
> slowness.  I am thinking that nearly 700 *million* calls to the iterator
> for the C-domain symbol list might have something to do with it...

Wow, that's a lot!

> > True, but independently if the script would be rewritten in python or not,
> > one way would be to enrich the 'DOCS:' kernel-doc tag in order to mention 
> > there the symbols that belong to each part of the document, e. g. something
> > like:
> >
> > 	/**
> > 	 * DOC: foo
> > 	 *
> > 	 * Some comments...
> > 	 *
> > 	 * symbols:
> > 	 *     foo
> > 	 *     bar
> > 	 */
> >
> > One advantage is that all documentation will be on a single place,
> > so hopefully it would be easier to maintain.  
> 
> I'm not quite sure I get this...you want to put the TOC tree in the
> source comments?  This looks like the kind of thing that nobody ever
> remembers to update, but maybe I'm missing something.

No, it won't generate a TOC tree. It would instead reorder how
kernel-doc would output the symbols.

That's no different than what we have already at the .kernel-doc
directives, e. g. it would be a replacement for:


	.. kernel-doc:: include/some_header.h
	   :doc: foo

	.. kernel-doc:: include/some_header.h
	   :functions:
		foo
		bar

The problem we currently have is that the above pattern means that
one or more .rst files would contain a list of symbols that are
actually documented at some_header.h. People updating such file
will very likely forget to update .rst files, leading to missing
documentation. Also, the same symbol could be included on different
.rst files.

I remember I had to fix myself duplicated symbols inclusion while
trying to reduce the docs build time on a few *.h files that
were included hundreds of times. I bet if someone checks again,
duplicated inclusions of the same symbol and missed symbols that
are documented on their sources, but aren't included at any .rst file.

Currently, detecting it is very hard, because the symbol lists
are on different files and the same header are sometimes included
on different rst files. By placing the symbol list inside the source
file, it makes very simple for kernel-doc to check if all documented 
symbols are inside the "DOC:" markups, producing errors otherwise.

Regards,
Mauro

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-24  7:33         ` Mauro Carvalho Chehab
@ 2022-06-24 16:37           ` Markus Heiser
  2022-06-27 15:27             ` Jonathan Corbet
  2022-06-24 22:57           ` Jonathan Corbet
  1 sibling, 1 reply; 40+ messages in thread
From: Markus Heiser @ 2022-06-24 16:37 UTC (permalink / raw)
  To: Mauro Carvalho Chehab, Jonathan Corbet
  Cc: Jani Nikula, ksummit-discuss, ksummit

Hi Jon, Mauro,

   with great interest I have read this thread !

If I can help, I'm happy to participate again.  TBH: The kernel-doc
implementation from mine could only be a piece of the puzzle to
improve the process (not without raising other questions).

But similar to Jani, I see the priority problems on a different level
level.

We have too many demands that reach into the smallest niches.  I
believe that some parts must be generalized and some request must
be abandoned.

In my opinion, many problems are homemade, to take just one example:

IMO It is unnecessary that the build-chain must run on all
platforms and with all distributions.

Who observes the Sphinx-doc & docutils development since (>15)
years is aware that with various (old) Sphinx-doc & docutils
versions no stable results can be produced, not without
complicating the build-chain.  And this is exactly the situation
we are facing today.

The build chain of documentation has nothing to do with kernel
development (at least in my opinion) and should be decoupled from it:
maintaining one defined build environment is enough work ... this
becomes especially clear if you (as Jani recommends) rely more on
sphinx-modules and widely used tools.

Another point on which I now have a clear view are the regular
expressions: no one likes them (me too), but I can't think of a
general solution (parser) given the number of requirements for
parsing source code we have.

I would like to support the kernel community again, but at the
moment I have difficulties to follow the many exceptions. I would
be happy if you keep me in CC .. may be I find my place again
back in the kernel-doc development :-)

Thanks and with best regards

   -- Markus --


Am 24.06.22 um 09:33 schrieb Mauro Carvalho Chehab:
> Em Thu, 23 Jun 2022 07:40:45 -0600
> Jonathan Corbet <corbet@lwn.net> escreveu:
> 
>> Mauro Carvalho Chehab <mchehab@kernel.org> writes:
>>
>>>> The solution would be to finally convert the
>>>> script to a proper python Sphinx extension that can do caching. (This is
>>>> how it works in Hawkmoth, FWIW.)
>>
>> I've been pondering on this for a bit, and would like to do it, but I
>> don't know when I might find the time for it.
> 
> There is already a version of kernel-doc written in Python, made by
> Markus Heiser:
> 
> 	https://github.com/return42/linuxdoc/blob/master/linuxdoc/kernel_doc.py
> 
> It could be a starting point.
> 
>>> That's one solution, but see: there is already a python extension
>>> that currently calls kernel-doc everytime. It could, instead,
>>> cache the rst returned by its first run (or a parsed version of it)
>>> and use the cached results the other 3 times.
>>>
>>> Porting kernel-doc to python could be doable, but not trivial, due to several
>>> reasons:
>>>
>>> - it should keep running standalone, as otherwise debugging parsing issues
>>>    on kernel-doc would be a lot harder. In particular, kernel-doc --none is
>>>    really helpful to report kernel-doc tag errors;
>>
>> Yes, of course.  As Jani noted, that's just how you would do it, not a
>> problem.
>>
>>> - regressions will likely be introduced on a change like that;
>>
>> The nice thing is ... we already have a really nice regression test in
>> the form of the current docs build and diff.
> 
> True. We can test the results with both versions and even check how
> performance is affected.
> 
>>> - regular contributors to kernel-doc will need to ramp up with the newer
>>>    version;
>>
>> We have those?  That script is a nightmare and nobody goes near it if
>> they can possibly avoid it.  I would expect to have more contributors
>> with a decent Python version that doesn't include 25 years of regex
>> accretion.
> 
> Well, the kernel-doc version from Markus in python is ~23% bigger
> than our current kernel-doc. Granted, it could be due to comments and
> blank lines, but basically, the same regexes that are in perl would
> also be need to be replicated in python, as parsing a C code with
> regexes is not a trivial task.
> 
> So, in terms of complexity, I doubt much would change by porting it
> to python.
> 
> Now, one of the things that the kernel-doc does is that it has the
> parse code and the output logic, which actually has an an abstraction
> to let it to produce results on different formats (currently, none, man
> and rst - we dropped some other formats from it). This abstraction
> increases its complexity. This is something that doesn't need to be
> replicated on any ports.
> 
>>> - a port like that could increase the script run time, as the
>>>    optimizations and regular expressions there could behave different on
>>>    python.
>>
>> It could also decrease it by improving caching opportunities, getting
>> rid of a lot of fork()/exec() pairs and Perl interpreter startups, etc.
>>
>> I've actually, in a spare moment or two, been doing some profiling of
>> the kernel docs build and trying to track down the sources of the
>> slowness.  I am thinking that nearly 700 *million* calls to the iterator
>> for the C-domain symbol list might have something to do with it...
> 
> Wow, that's a lot!
> 
>>> True, but independently if the script would be rewritten in python or not,
>>> one way would be to enrich the 'DOCS:' kernel-doc tag in order to mention
>>> there the symbols that belong to each part of the document, e. g. something
>>> like:
>>>
>>> 	/**
>>> 	 * DOC: foo
>>> 	 *
>>> 	 * Some comments...
>>> 	 *
>>> 	 * symbols:
>>> 	 *     foo
>>> 	 *     bar
>>> 	 */
>>>
>>> One advantage is that all documentation will be on a single place,
>>> so hopefully it would be easier to maintain.
>>
>> I'm not quite sure I get this...you want to put the TOC tree in the
>> source comments?  This looks like the kind of thing that nobody ever
>> remembers to update, but maybe I'm missing something.
> 
> No, it won't generate a TOC tree. It would instead reorder how
> kernel-doc would output the symbols.
> 
> That's no different than what we have already at the .kernel-doc
> directives, e. g. it would be a replacement for:
> 
> 
> 	.. kernel-doc:: include/some_header.h
> 	   :doc: foo
> 
> 	.. kernel-doc:: include/some_header.h
> 	   :functions:
> 		foo
> 		bar
> 
> The problem we currently have is that the above pattern means that
> one or more .rst files would contain a list of symbols that are
> actually documented at some_header.h. People updating such file
> will very likely forget to update .rst files, leading to missing
> documentation. Also, the same symbol could be included on different
> .rst files.
> 
> I remember I had to fix myself duplicated symbols inclusion while
> trying to reduce the docs build time on a few *.h files that
> were included hundreds of times. I bet if someone checks again,
> duplicated inclusions of the same symbol and missed symbols that
> are documented on their sources, but aren't included at any .rst file.
> 
> Currently, detecting it is very hard, because the symbol lists
> are on different files and the same header are sometimes included
> on different rst files. By placing the symbol list inside the source
> file, it makes very simple for kernel-doc to check if all documented
> symbols are inside the "DOC:" markups, producing errors otherwise.
> 
> Regards,
> Mauro

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-24  7:33         ` Mauro Carvalho Chehab
  2022-06-24 16:37           ` Markus Heiser
@ 2022-06-24 22:57           ` Jonathan Corbet
  2022-06-25  9:10             ` Mauro Carvalho Chehab
  1 sibling, 1 reply; 40+ messages in thread
From: Jonathan Corbet @ 2022-06-24 22:57 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Jani Nikula, ksummit-discuss, ksummit, Markus Heiser

Mauro Carvalho Chehab <mchehab@kernel.org> writes:

> Em Thu, 23 Jun 2022 07:40:45 -0600
> Jonathan Corbet <corbet@lwn.net> escreveu:
>> I've actually, in a spare moment or two, been doing some profiling of
>> the kernel docs build and trying to track down the sources of the
>> slowness.  I am thinking that nearly 700 *million* calls to the iterator
>> for the C-domain symbol list might have something to do with it...
>
> Wow, that's a lot!

Just for the curious ...

The Sphinx C domain code makes this elaborate tree data structure out of
all the identifier names it has picked up - including the names of
function parameters when they are provided in prototypes for some
reason.  This is the structure that is consulted whenever we want to
resolve a cross-reference, which is fairly often with automarkup
enabled.

How does it work?  The algorithm is, as far as I can tell:

 - Serialize the whole tree in a sort of breadth-first traversal

 - Make a linear pass through the *entire* list, comparing the
   identifier name with each entry and accumulating a list of all the
   matches found.

 - Return the first match and throw away the rest.

The result is O(n^2) behavior and, in the kernel docs build, n gets to
be fairly large.

I went in with a crowbar and sledgehammer and replaced some of the list
searches with a dict lookup, resulting in about a 20% speedup in the
full htmldocs build with Sphinx 5.0.2.  A couple of automarkup
optimizations result in about a 27% speedup overall.  And I didn't break
any cross-references.

I think it's possible to do better, but this is a start.  I'll post the
automarkup changes as soon as I can, but I need to verify them across
the whole range of Sphinx versions.  For the Sphinx stuff, I'll need to
turn my hatchetwork into something presentable and figure out how to
contribute it upstream, but it seems worth the effort.

Thanks,

jon

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-24 22:57           ` Jonathan Corbet
@ 2022-06-25  9:10             ` Mauro Carvalho Chehab
  2022-06-25 14:00               ` Jonathan Corbet
  0 siblings, 1 reply; 40+ messages in thread
From: Mauro Carvalho Chehab @ 2022-06-25  9:10 UTC (permalink / raw)
  To: Jonathan Corbet; +Cc: Jani Nikula, ksummit-discuss, ksummit, Markus Heiser

Em Fri, 24 Jun 2022 16:57:56 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab@kernel.org> writes:
> 
> > Em Thu, 23 Jun 2022 07:40:45 -0600
> > Jonathan Corbet <corbet@lwn.net> escreveu:  
> >> I've actually, in a spare moment or two, been doing some profiling of
> >> the kernel docs build and trying to track down the sources of the
> >> slowness.  I am thinking that nearly 700 *million* calls to the iterator
> >> for the C-domain symbol list might have something to do with it...  
> >
> > Wow, that's a lot!  
> 
> Just for the curious ...
> 
> The Sphinx C domain code makes this elaborate tree data structure out of
> all the identifier names it has picked up - including the names of
> function parameters when they are provided in prototypes for some
> reason.  This is the structure that is consulted whenever we want to
> resolve a cross-reference, which is fairly often with automarkup
> enabled.
> 
> How does it work?  The algorithm is, as far as I can tell:
> 
>  - Serialize the whole tree in a sort of breadth-first traversal
> 
>  - Make a linear pass through the *entire* list, comparing the
>    identifier name with each entry and accumulating a list of all the
>    matches found.
> 
>  - Return the first match and throw away the rest.
> 
> The result is O(n^2) behavior and, in the kernel docs build, n gets to
> be fairly large.
> 
> I went in with a crowbar and sledgehammer and replaced some of the list
> searches with a dict lookup, resulting in about a 20% speedup in the
> full htmldocs build with Sphinx 5.0.2. 

Great to have a 20% speedup here! Yet, I would expect an even better
performance improvement by replacing 700 million calls from linear search
to dict, as it would change from O(n^2) to O(1) at the average case [1],
but the speedup gain actually depends on the actual number of symbols we
have defined.

[1] According with:
	https://wiki.python.org/moin/TimeComplexity#dict
    Worse case is O(n) - when collisions are common which is unlikely.

> A couple of automarkup
> optimizations result in about a 27% speedup overall.  And I didn't break
> any cross-references.

Great!

> I think it's possible to do better, but this is a start.  I'll post the
> automarkup changes as soon as I can, but I need to verify them across
> the whole range of Sphinx versions. 

> For the Sphinx stuff, I'll need to
> turn my hatchetwork into something presentable and figure out how to
> contribute it upstream, but it seems worth the effort.

Indeed.

Regards,
Mauro

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-25  9:10             ` Mauro Carvalho Chehab
@ 2022-06-25 14:00               ` Jonathan Corbet
  2022-06-25 18:11                 ` Linus Torvalds
  0 siblings, 1 reply; 40+ messages in thread
From: Jonathan Corbet @ 2022-06-25 14:00 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Jani Nikula, ksummit-discuss, ksummit, Markus Heiser

Mauro Carvalho Chehab <mchehab@kernel.org> writes:

> Em Fri, 24 Jun 2022 16:57:56 -0600
> Jonathan Corbet <corbet@lwn.net> escreveu:

>> I went in with a crowbar and sledgehammer and replaced some of the list
>> searches with a dict lookup, resulting in about a 20% speedup in the
>> full htmldocs build with Sphinx 5.0.2. 
>
> Great to have a 20% speedup here! Yet, I would expect an even better
> performance improvement by replacing 700 million calls from linear search
> to dict, as it would change from O(n^2) to O(1) at the average case [1],
> but the speedup gain actually depends on the actual number of symbols we
> have defined.

I said "some" - this was a proof-of-concept hack.  The complexity of
their symbol lookup code is ... daunting ... and not something that gets
fixed in one place.  More research is required...

jon

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [TECH TOPIC] What kernel documentation could be
  2022-06-17 20:57 [TECH TOPIC] What kernel documentation could be Jonathan Corbet
                   ` (2 preceding siblings ...)
  2022-06-18  8:24 ` Mauro Carvalho Chehab
@ 2022-06-25 17:43 ` Steven Rostedt
  2022-06-25 17:48   ` Laurent Pinchart
  3 siblings, 1 reply; 40+ messages in thread
From: Steven Rostedt @ 2022-06-25 17:43 UTC (permalink / raw)
  To: Jonathan Corbet; +Cc: ksummit-discuss, ksummit

On Fri, 17 Jun 2022 14:57:10 -0600
Jonathan Corbet <corbet@lwn.net> wrote:

> The development community has put a lot of work into the kernel's
> documentation directory in recent years, with visible results. But the
> kernel's documentation still falls far short of the standard set by many
> other projects, and there is a great deal of "tribal knowledge" in our
> community that is not set down. I'd like to look at the successes and
> failures of the work so far, but intent to focus on a discussion of what
> our documentation should be and what we can do to get it there.
> 
> Questions to discuss include:
> 
>  - Bringing an coherent overall structure to Documentation/
> 
>  - Making it easier for developers to improve the documentation.


Is there a way to mark comments in the code as "design docs". And I
don't mean the kernel-doc that is already above functions.

I have lots of comments about the design of complex code just above the
code it is describing. Which is much more likely to get updated when
the design changes or gets new features.

I found my design documents I have under Documentation to become
grossly obsolete over time, where as the comments above the code I can
keep up with.

Is there already some kind of mark up that we can add to the comments to state:

 ** .design.trace/ftrace.rst .ringbuffer. **

or something that can be extracted into another document? With the
above example, I could have in Documentatino/trace/ftrace.rst a

 .design .ringbuffer.

And the comment from the code would be extracted into the html
documents that are created?

Or maybe it would be better to reference the code from the documentation?

 .include kerne/trace/ring_buffer.c .ringbuffer.

That will look for a tag in kernel/trace/ring_buffer.c called
".ringbuffer" and pull in the comment into the document under Documentation.

Thoughts?

-- Steve


> 
>  - What we would like from Sphinx and what we can do to make it happen
> 
>  - Making the docs build system less ugly and more maintainable
> 
>  - Integrating rustdoc documentation
> 
> jon


^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [TECH TOPIC] What kernel documentation could be
  2022-06-25 17:43 ` Steven Rostedt
@ 2022-06-25 17:48   ` Laurent Pinchart
  0 siblings, 0 replies; 40+ messages in thread
From: Laurent Pinchart @ 2022-06-25 17:48 UTC (permalink / raw)
  To: Steven Rostedt; +Cc: Jonathan Corbet, ksummit-discuss, ksummit

Hi Steven,

On Sat, Jun 25, 2022 at 01:43:28PM -0400, Steven Rostedt wrote:
> On Fri, 17 Jun 2022 14:57:10 -0600 Jonathan Corbet wrote:
> 
> > The development community has put a lot of work into the kernel's
> > documentation directory in recent years, with visible results. But the
> > kernel's documentation still falls far short of the standard set by many
> > other projects, and there is a great deal of "tribal knowledge" in our
> > community that is not set down. I'd like to look at the successes and
> > failures of the work so far, but intent to focus on a discussion of what
> > our documentation should be and what we can do to get it there.
> > 
> > Questions to discuss include:
> > 
> >  - Bringing an coherent overall structure to Documentation/
> > 
> >  - Making it easier for developers to improve the documentation.
> 
> Is there a way to mark comments in the code as "design docs". And I
> don't mean the kernel-doc that is already above functions.
> 
> I have lots of comments about the design of complex code just above the
> code it is describing. Which is much more likely to get updated when
> the design changes or gets new features.
> 
> I found my design documents I have under Documentation to become
> grossly obsolete over time, where as the comments above the code I can
> keep up with.
> 
> Is there already some kind of mark up that we can add to the comments to state:
> 
>  ** .design.trace/ftrace.rst .ringbuffer. **
> 
> or something that can be extracted into another document? With the
> above example, I could have in Documentatino/trace/ftrace.rst a
> 
>  .design .ringbuffer.
> 
> And the comment from the code would be extracted into the html
> documents that are created?
> 
> Or maybe it would be better to reference the code from the documentation?
> 
>  .include kerne/trace/ring_buffer.c .ringbuffer.
> 
> That will look for a tag in kernel/trace/ring_buffer.c called
> ".ringbuffer" and pull in the comment into the document under Documentation.
> 
> Thoughts?

See the "DOC: overview" block in drivers/gpu/drm/drm_atomic_helper.c,
and how it is integrated in Documentation/gpu/drm-kms-helpers.rst with

.. kernel-doc:: drivers/gpu/drm/drm_atomic_helper.c
   :doc: overview

Is that what you're looking for ?

> >  - What we would like from Sphinx and what we can do to make it happen
> > 
> >  - Making the docs build system less ugly and more maintainable
> > 
> >  - Integrating rustdoc documentation

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-25 14:00               ` Jonathan Corbet
@ 2022-06-25 18:11                 ` Linus Torvalds
  2022-06-26  7:55                   ` Mauro Carvalho Chehab
                                     ` (2 more replies)
  0 siblings, 3 replies; 40+ messages in thread
From: Linus Torvalds @ 2022-06-25 18:11 UTC (permalink / raw)
  To: Jonathan Corbet
  Cc: Mauro Carvalho Chehab, Jani Nikula, ksummit, ksummit, Markus Heiser

On Sat, Jun 25, 2022 at 7:00 AM Jonathan Corbet <corbet@lwn.net> wrote:
>
> I said "some" - this was a proof-of-concept hack.  The complexity of
> their symbol lookup code is ... daunting ... and not something that gets
> fixed in one place.  More research is required...

Note that at least for me, the issue with building docs isn't hugely
performance related.

Yes, yes, it would be good if it was faster. But..

The primary problems I see with building the docs (and thus checking
them, the same way I do an allmodconfig build test) is

 (a) it's insanely noisy, which makes and "check that it's ok" ENTIRELY USELESS.

Seriously. There's a very solid reason why I require the standard
kernel build to be entirely warning-free. Warnings are *BAD*. Even a
single false-positive warning invalidates all other warnings.

And the doc build isn't about some "single warning" thing.

 (b) it requires some unusual build tools

Now, (b) may some historical oddity, but at least if you have Fedora
installed and it still has sphinx version 2.2.11 or something like
that.

And when you try to build docs, the makefile gives you some random pip
install thing that is entirely bogus.

The proper fix is to actually remove that old sphinx environment
entirely, and install "python3-sphinx" instead, but that's not at all
what the "to upgrade Sphinx" docs in the kernel say when you try to
make the docs.

Anyway, (b) is one of those "the docs about the docs are wrong" things
and ironic - but easily fixable if you know about it.

But (a) then makes it all entirely pointless as far as I'm concerned.
The doc build could take five seconds, and I *still* wouldn't bother,
because building docs doesn't actually do anything useful for me. It's
just noise.

                  Linus

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-25 18:11                 ` Linus Torvalds
@ 2022-06-26  7:55                   ` Mauro Carvalho Chehab
  2022-06-26  9:26                     ` Mauro Carvalho Chehab
  2022-06-26  9:53                     ` Mauro Carvalho Chehab
  2022-06-27 15:34                   ` Jonathan Corbet
  2022-07-02 10:52                   ` Mauro Carvalho Chehab
  2 siblings, 2 replies; 40+ messages in thread
From: Mauro Carvalho Chehab @ 2022-06-26  7:55 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: Jonathan Corbet, Jani Nikula, ksummit, ksummit, Markus Heiser

Em Sat, 25 Jun 2022 11:11:40 -0700
Linus Torvalds <torvalds@linux-foundation.org> escreveu:

> On Sat, Jun 25, 2022 at 7:00 AM Jonathan Corbet <corbet@lwn.net> wrote:
> >
> > I said "some" - this was a proof-of-concept hack.  The complexity of
> > their symbol lookup code is ... daunting ... and not something that gets
> > fixed in one place.  More research is required...  
> 
> Note that at least for me, the issue with building docs isn't hugely
> performance related.
> 
> Yes, yes, it would be good if it was faster. But..
> 
> The primary problems I see with building the docs (and thus checking
> them, the same way I do an allmodconfig build test) is
> 
>  (a) it's insanely noisy, which makes and "check that it's ok" ENTIRELY USELESS.

When the environment is set for building docs, almost all output there
are actually due to build errors/warnings that got introduced by patches
touching documentation that were never build-tested.

There are 4 components that report errors during "make htmldocs". 

1. Documentation cross-reference check. 

	$ scripts/documentation-file-ref-check --warn
	Warning: Documentation/admin-guide/kernel-parameters.txt references a file that doesn't exist: Documentation/virt/kvm/amd-memory-encryption.rst
	Warning: Documentation/dev-tools/kunit/run_wrapper.rst references a file that doesn't exist: Documentation/dev-tools/kunit/non_uml.rst
	...

Those warnings are result of build-untested patches moving/renaming/removing 
documentation without updating all documentation.

On almost all Kernel versions, I usually send a couple of patch series
addressing them, but incomplete patches that forget about updating references
keep being merged.

Without htmldocs build tests by patch authors, this is an endless job.

2. ABI documentation check:

	$ ./scripts/get_abi.pl validate
	Warning: /sys/bus/iio/devices/iio:deviceX/fault_ovuv is defined 2 times:  Documentation/ABI/testing/sysfs-bus-iio-temperature-max31865:0  Documentation/ABI/testing/sysfs-bus-iio-temperature-max31856:14
	Warning: /sys/bus/iio/devices/iio:deviceX/in_filter_notch_center_frequency is defined 2 times:  Documentation/ABI/testing/sysfs-bus-iio:1948  Documentation/ABI/testing/sysfs-bus-iio-temperature-max31865:12
	Warning: /sys/bus/iio/devices/triggerX/sampling_frequency is defined 2 times:  Documentation/ABI/testing/sysfs-bus-iio:96  Documentation/ABI/testing/sysfs-bus-iio-timer-stm32:92
	Warning: /sys/devices/system/cpu/cpuX/topology/core_id is defined 2 times:  Documentation/ABI/stable/sysfs-devices-system-cpu:38  Documentation/ABI/testing/sysfs-devices-system-cpu:69

Those warnings indicate problems with userspace API definitions, as the 
same symbol has different meanings.

Talking about ABI, the output is even worse if the script is used to check
if the ABIs used on a running system are all documented or not. On my
laptop (Kernel 5.18.5-200.fc36.x86_64), I'm getting, for instance, 2815
undocumented or badly-specified sysfs ABI symbols: 

	$ sudo ./scripts/get_abi.pl undefined >errors
	Reading /sys directory contents...done.
	Converting ABI What fields into regexes...done.
	4:59: processing sysfs files... done
	$ wc -l errors
	2815

3. kernel-doc warnings.

Those indicate kernel-doc markups with troubles. The output is the same
as running:

	$ ./scripts/kernel-doc --none ./include/linux/fscache.h
	./include/linux/fscache.h:269: warning: Function parameter or member 'cookie' not described in 'fscache_use_cookie'
	./include/linux/fscache.h:269: warning: Excess function parameter 'object' description in 'fscache_use_cookie'
	./include/linux/fscache.h:286: warning: Function parameter or member 'cookie' not described in 'fscache_unuse_cookie'
	./include/linux/fscache.h:286: warning: Excess function parameter 'object' description in 'fscache_unuse_cookie'

In this particular case, kernel-doc markup describes a non-existent "object"
symbol, while the function prototype has "cookie". Probably result of some
rename at the function prototype that was not validated via html build.

4. Sphinx warnings.

Most of them are one line warnings:

	Documentation/arm/samsung-s3c24xx/cpufreq.rst:2: WARNING: Explicit markup ends without a blank line; unexpected unindent.
	Documentation/admin-guide/device-mapper/writecache.rst:23: WARNING: Unexpected indentation.
	Documentation/vm/highmem:166: ./include/linux/highmem.h:154: WARNING: Inline emphasis start-string without end-string.
	Documentation/vm/highmem:166: ./include/linux/highmem.h:157: WARNING: Inline emphasis start-string without end-string.

But on errors, Sphinx is very noisy, as it outputs the offending
lines or tables. 

For instance, on next-20220620, some patch broke a table.
The error there is very noisy, as it outputs the entire broken table.

> Seriously. There's a very solid reason why I require the standard
> kernel build to be entirely warning-free. Warnings are *BAD*. Even a
> single false-positive warning invalidates all other warnings.

All the Sphinx warnings could be avoided if people start testing
documentation changes. A policy of making kernel htmldocs warning-free
would have solved all the above issues.

I can submit some patches addressing issues along this week against
linux-next (or against your tree?), but without developers trying to
build documentation, we'll keep having issues.

> And the doc build isn't about some "single warning" thing.
> 
>  (b) it requires some unusual build tools
> 
> Now, (b) may some historical oddity, but at least if you have Fedora
> installed and it still has sphinx version 2.2.11 or something like
> that.
>
> And when you try to build docs, the makefile gives you some random pip
> install thing that is entirely bogus.

That's likely the case. You probably installed Sphinx via pip
on, let's say, Fedora 35. After upgrading to Fedora 36, python version
got incremented, and the past install won't work anymore, because it
would be trying to use patch libraries from a different directory. 

Since this changeset:

	a8b380c379ef ("scripts: sphinx-pre-install: only ask to activate valid venvs")

the documentation build checks if this command works:

	sphinx-build --version

If it doesn't, it will give instructions about how to install Sphinx.

> The proper fix is to actually remove that old sphinx environment
> entirely, and install "python3-sphinx" instead, but that's not at all
> what the "to upgrade Sphinx" docs in the kernel say when you try to
> make the docs.

The script which checks for Sphinx availability is capable of either
recommend distro-specific Sphinx package or the usage of python venv.

As Sphinx release cycle is fast, it was opted to use venv by default,
so, it currently recommends the usage of pip. 

The same script called at htmldocs build time can be used to recommend
the usage of the distro-provided package. Running it in Fedora, it would
give:

	./scripts/sphinx-pre-install --no-virtualenv
	Detected OS: Fedora release 36 (Thirty Six).
	ERROR: please install "python-sphinx", otherwise, build won't work.
	Warning: better to also install "sphinx_rtd_theme".
	Warning: better to also install "texlive-ctex".
	You should run:

		sudo dnf install -y python3-sphinx python3-sphinx_rtd_theme texlive-ctex
	note: If you want pdf, you need at least Sphinx 2.4.4.

	Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 955.

It would be trivial to change the default to recommend the usage of the
distro package. Yet:

1. Newer versions of Sphinx are two times slower than 2.4.x;

2. Recommending the same version everywhere ensures that the
   documentation will be compatible with Sphinx >= 2.4.

3. Older LTS distros may not have a Sphinx version compatible with the
   Kernel.

> Anyway, (b) is one of those "the docs about the docs are wrong" things
> and ironic - but easily fixable if you know about it.
> 
> But (a) then makes it all entirely pointless as far as I'm concerned.
> The doc build could take five seconds, and I *still* wouldn't bother,
> because building docs doesn't actually do anything useful for me. It's
> just noise.


The hope is that, if building it would be fast enough, developers and
maintainers would actually check if the build succeeds without warnings,
as ultimately, the big noise is due to patches merged without being
even build-tested.

Regards,
Mauro


^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-26  7:55                   ` Mauro Carvalho Chehab
@ 2022-06-26  9:26                     ` Mauro Carvalho Chehab
  2022-06-26  9:53                     ` Mauro Carvalho Chehab
  1 sibling, 0 replies; 40+ messages in thread
From: Mauro Carvalho Chehab @ 2022-06-26  9:26 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: Jonathan Corbet, Jani Nikula, ksummit, ksummit, Markus Heiser

Em Sun, 26 Jun 2022 08:55:24 +0100
Mauro Carvalho Chehab <mchehab@kernel.org> escreveu:

> Em Sat, 25 Jun 2022 11:11:40 -0700
> Linus Torvalds <torvalds@linux-foundation.org> escreveu:
> 
> > On Sat, Jun 25, 2022 at 7:00 AM Jonathan Corbet <corbet@lwn.net> wrote:
> > >
> > > I said "some" - this was a proof-of-concept hack.  The complexity of
> > > their symbol lookup code is ... daunting ... and not something that gets
> > > fixed in one place.  More research is required...  
> > 
> > Note that at least for me, the issue with building docs isn't hugely
> > performance related.
> > 
> > Yes, yes, it would be good if it was faster. But..
> > 
> > The primary problems I see with building the docs (and thus checking
> > them, the same way I do an allmodconfig build test) is
> > 
> >  (a) it's insanely noisy, which makes and "check that it's ok" ENTIRELY USELESS.
> 
> When the environment is set for building docs, almost all output there
> are actually due to build errors/warnings that got introduced by patches
> touching documentation that were never build-tested.
> 
> There are 4 components that report errors during "make htmldocs". 
> 
> 1. Documentation cross-reference check. 
> 
> 	$ scripts/documentation-file-ref-check --warn
> 	Warning: Documentation/admin-guide/kernel-parameters.txt references a file that doesn't exist: Documentation/virt/kvm/amd-memory-encryption.rst
> 	Warning: Documentation/dev-tools/kunit/run_wrapper.rst references a file that doesn't exist: Documentation/dev-tools/kunit/non_uml.rst
> 	...
> 
> Those warnings are result of build-untested patches moving/renaming/removing 
> documentation without updating all documentation.
> 
> On almost all Kernel versions, I usually send a couple of patch series
> addressing them, but incomplete patches that forget about updating references
> keep being merged.
> 
> Without htmldocs build tests by patch authors, this is an endless job.

Just sent a patch series addressing the existing issues against
next-20220624:

	https://lore.kernel.org/linux-doc/cover.1656234456.git.mchehab@kernel.org/T/#t

With that, the number of warnings will be reduced to only 3:

	$ ./scripts/documentation-file-ref-check 
	Documentation/dev-tools/kunit/run_wrapper.rst: Documentation/dev-tools/kunit/non_uml.rst
	Documentation/devicetree/bindings/regulator/siliconmitus,sm5703-regulator.yaml: Documentation/devicetree/bindings/mfd/siliconmitus,sm5703.yaml
	drivers/acpi/device_pm.c: Documentation/firmware-guide/acpi/low-power-probe.rst

Those are actually useful for doc discussions, as they show one of
the issues we're currently having.

Sometimes, support for some functionality crosses multiple subsystems.
On such cases, part of the support may end being merged on a future
linux-next version - or even on a future Kernel version.

That seems to be the case of sm5703 support. The regulator bindings got
already merged, but the mfd ones weren't.

So, there's not much that could be done to fix
siliconmitus,sm5703-regulator.yaml broken reference to
mfd/siliconmitus,sm5703.yaml.

The same seems to apply to low-power-probe.rst: the patch adding
it was not merged yet at linux-next:

	https://lore.kernel.org/all/20210128232729.16064-3-sakari.ailus@linux.intel.com/

but the patch using it at drivers/acpi/device_pm.c was merged.

Regards,
Mauro

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-26  7:55                   ` Mauro Carvalho Chehab
  2022-06-26  9:26                     ` Mauro Carvalho Chehab
@ 2022-06-26  9:53                     ` Mauro Carvalho Chehab
  2022-06-27 15:28                       ` Liam Howlett
  1 sibling, 1 reply; 40+ messages in thread
From: Mauro Carvalho Chehab @ 2022-06-26  9:53 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: Jonathan Corbet, Jani Nikula, ksummit, ksummit, Markus Heiser

Em Sun, 26 Jun 2022 08:55:24 +0100
Mauro Carvalho Chehab <mchehab@kernel.org> escreveu:

> Em Sat, 25 Jun 2022 11:11:40 -0700
> Linus Torvalds <torvalds@linux-foundation.org> escreveu:
> 
> > On Sat, Jun 25, 2022 at 7:00 AM Jonathan Corbet <corbet@lwn.net> wrote:  
> > >
> > > I said "some" - this was a proof-of-concept hack.  The complexity of
> > > their symbol lookup code is ... daunting ... and not something that gets
> > > fixed in one place.  More research is required...    
> > 
> > Note that at least for me, the issue with building docs isn't hugely
> > performance related.
> > 
> > Yes, yes, it would be good if it was faster. But..
> > 
> > The primary problems I see with building the docs (and thus checking
> > them, the same way I do an allmodconfig build test) is
> > 
> >  (a) it's insanely noisy, which makes and "check that it's ok" ENTIRELY USELESS.  
> 
> When the environment is set for building docs, almost all output there
> are actually due to build errors/warnings that got introduced by patches
> touching documentation that were never build-tested.
> 
> There are 4 components that report errors during "make htmldocs". 
> 
> 1. Documentation cross-reference check. 
...
> 2. ABI documentation check:
...
> 3. kernel-doc warnings.
...

Btw, once we fix the errors from the above checks, one of the things that 
could be done in order to avoid noisy doc builds would be to run this 
during normal Kernel build, if CONFIG_WERROR is set (and if .git is present
at the source build dir):

	./scripts/documentation-file-ref-check --warn
	./scripts/get_abi.pl validate
	./scripts/kernel-doc --none $(git grep kernel-doc $(git ls-files Documentation/|grep -v kernel-doc.rst)|perl -ne 'print "$1\n" if (m/kernel-doc::\s*(\S+)/);'|sort|uniq) 

aborting the build on such warnings.

On my notebook (i5-10210U), the above takes ~8 seconds to run. So, it 
won't make much difference at the build time, and doing that would have
avoided ~100 warnings during htmldocs build against current linux-next.

Regards,
Mauro

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-17 21:48 ` Laurent Pinchart
  2022-06-17 21:48   ` Laurent Pinchart
@ 2022-06-27 15:18   ` Jonathan Corbet
  1 sibling, 0 replies; 40+ messages in thread
From: Jonathan Corbet @ 2022-06-27 15:18 UTC (permalink / raw)
  To: Laurent Pinchart; +Cc: ksummit-discuss, ksummit

[yes, I'm running even slower than usual...]

Laurent Pinchart <laurent.pinchart@ideasonboard.com> writes:

> I've been wondering recently if it would help to have a place where we
> can record documentation wishes.

One of the conclusions that came out of the LSFMM session on MM
documentation was that having an overall structure in place makes it
easier for people to fill in pieces.  If we could impose more of this
kind of structure on our docs as a whole, one obvious way to express
wishes of this sort would be to add an empty section where the need
exists and hope that it inspires somebody to fill it in.

Thanks,

jon

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-24 16:37           ` Markus Heiser
@ 2022-06-27 15:27             ` Jonathan Corbet
  2022-06-27 15:31               ` Christoph Hellwig
  2022-06-28  7:43               ` Mauro Carvalho Chehab
  0 siblings, 2 replies; 40+ messages in thread
From: Jonathan Corbet @ 2022-06-27 15:27 UTC (permalink / raw)
  To: Markus Heiser, Mauro Carvalho Chehab
  Cc: Jani Nikula, ksummit-discuss, ksummit

Markus Heiser <markus.heiser@darmarit.de> writes:

> IMO It is unnecessary that the build-chain must run on all
> platforms and with all distributions.
>
> Who observes the Sphinx-doc & docutils development since (>15)
> years is aware that with various (old) Sphinx-doc & docutils
> versions no stable results can be produced, not without
> complicating the build-chain.  And this is exactly the situation
> we are facing today.
>
> The build chain of documentation has nothing to do with kernel
> development (at least in my opinion) and should be decoupled from it:
> maintaining one defined build environment is enough work ... this
> becomes especially clear if you (as Jani recommends) rely more on
> sphinx-modules and widely used tools.

The counterargument to this is that we want as many developers as
possible to be able to build the docs and contribute to them.  We can't
complain that developers have broken the docs build if we don't do what
we can to help them do the build themselves.

One of our longstanding contributors is on Sphinx 1.8.5:

  https://lwn.net/ml/linux-doc/4c403239-3c71-4ab9-2168-f7e9d77008b2%40infradead.org/

I would like to narrow our range of supported versions, but I really
don't want to cut people out.

The real question, perhaps, is whether requiring people to set up a
virtualenv to run a supported version would be too much.  It's something
I worry about.

> Another point on which I now have a clear view are the regular
> expressions: no one likes them (me too), but I can't think of a
> general solution (parser) given the number of requirements for
> parsing source code we have.

I fear you're right.  There are nice C parsers for Python, but they
don't work at the level we need; we need the comments, unexpanded
macros, etc.

That said, the regex expression in kernel-doc is out of hand, with no
documentation, lots of duplication, etc.

> I would like to support the kernel community again, but at the
> moment I have difficulties to follow the many exceptions. I would
> be happy if you keep me in CC .. may be I find my place again
> back in the kernel-doc development :-)

We would be glad to have you back.

Remember that the reason I was hesitant to take your kernel-doc rewrite
at the time was that we were already thrashing too many things and I
wanted to hold at least one piece stable.  Hopefully we're past that
phase now...:)

Thanks,

jon

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-26  9:53                     ` Mauro Carvalho Chehab
@ 2022-06-27 15:28                       ` Liam Howlett
  2022-06-27 15:54                         ` Christian Brauner
                                           ` (2 more replies)
  0 siblings, 3 replies; 40+ messages in thread
From: Liam Howlett @ 2022-06-27 15:28 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Linus Torvalds, Jonathan Corbet, Jani Nikula, ksummit, ksummit,
	Markus Heiser

* Mauro Carvalho Chehab <mchehab@kernel.org> [220626 05:53]:
> Em Sun, 26 Jun 2022 08:55:24 +0100
> Mauro Carvalho Chehab <mchehab@kernel.org> escreveu:
> 
> > Em Sat, 25 Jun 2022 11:11:40 -0700
> > Linus Torvalds <torvalds@linux-foundation.org> escreveu:
> > 
> > > On Sat, Jun 25, 2022 at 7:00 AM Jonathan Corbet <corbet@lwn.net> wrote:  
> > > >
> > > > I said "some" - this was a proof-of-concept hack.  The complexity of
> > > > their symbol lookup code is ... daunting ... and not something that gets
> > > > fixed in one place.  More research is required...    
> > > 
> > > Note that at least for me, the issue with building docs isn't hugely
> > > performance related.
> > > 
> > > Yes, yes, it would be good if it was faster. But..
> > > 
> > > The primary problems I see with building the docs (and thus checking
> > > them, the same way I do an allmodconfig build test) is
> > > 
> > >  (a) it's insanely noisy, which makes and "check that it's ok" ENTIRELY USELESS.  
> > 
> > When the environment is set for building docs, almost all output there
> > are actually due to build errors/warnings that got introduced by patches
> > touching documentation that were never build-tested.
> > 
> > There are 4 components that report errors during "make htmldocs". 
> > 
> > 1. Documentation cross-reference check. 
> ...
> > 2. ABI documentation check:
> ...
> > 3. kernel-doc warnings.
> ...
> 
> Btw, once we fix the errors from the above checks, one of the things that 
> could be done in order to avoid noisy doc builds would be to run this 
> during normal Kernel build, if CONFIG_WERROR is set (and if .git is present
> at the source build dir):
> 
> 	./scripts/documentation-file-ref-check --warn
> 	./scripts/get_abi.pl validate
> 	./scripts/kernel-doc --none $(git grep kernel-doc $(git ls-files Documentation/|grep -v kernel-doc.rst)|perl -ne 'print "$1\n" if (m/kernel-doc::\s*(\S+)/);'|sort|uniq) 
> 
> aborting the build on such warnings.
> 
> On my notebook (i5-10210U), the above takes ~8 seconds to run. So, it 
> won't make much difference at the build time, and doing that would have
> avoided ~100 warnings during htmldocs build against current linux-next.
> 

Couldn't we add this to the build bots and ask the authors to fix the
commits?


Thanks,
Liam

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-27 15:27             ` Jonathan Corbet
@ 2022-06-27 15:31               ` Christoph Hellwig
  2022-06-28  7:43               ` Mauro Carvalho Chehab
  1 sibling, 0 replies; 40+ messages in thread
From: Christoph Hellwig @ 2022-06-27 15:31 UTC (permalink / raw)
  To: Jonathan Corbet
  Cc: Markus Heiser, Mauro Carvalho Chehab, Jani Nikula,
	ksummit-discuss, ksummit

On Mon, Jun 27, 2022 at 09:27:54AM -0600, Jonathan Corbet wrote:
> The real question, perhaps, is whether requiring people to set up a
> virtualenv to run a supported version would be too much.  It's something
> I worry about.

I know for on that I will not setup pything venvs as they are massive
pain.  Anyting that isn't available in a somewhat recent distro using
the package manage is a complete no-go.  And software that changes so
frequently that this question even arises seem rather suspect to me
in general for something like documentation that should have a wide
range of contributors.

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-25 18:11                 ` Linus Torvalds
  2022-06-26  7:55                   ` Mauro Carvalho Chehab
@ 2022-06-27 15:34                   ` Jonathan Corbet
  2022-06-27 17:07                     ` Linus Torvalds
  2022-07-02 10:52                   ` Mauro Carvalho Chehab
  2 siblings, 1 reply; 40+ messages in thread
From: Jonathan Corbet @ 2022-06-27 15:34 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: Mauro Carvalho Chehab, Jani Nikula, ksummit, ksummit, Markus Heiser

Linus Torvalds <torvalds@linux-foundation.org> writes:

> On Sat, Jun 25, 2022 at 7:00 AM Jonathan Corbet <corbet@lwn.net> wrote:
>>
>> I said "some" - this was a proof-of-concept hack.  The complexity of
>> their symbol lookup code is ... daunting ... and not something that gets
>> fixed in one place.  More research is required...
>
> Note that at least for me, the issue with building docs isn't hugely
> performance related.

Those of us who do lots of docs builds see it a bit differently :)

> Yes, yes, it would be good if it was faster. But..
>
> The primary problems I see with building the docs (and thus checking
> them, the same way I do an allmodconfig build test) is
>
>  (a) it's insanely noisy, which makes and "check that it's ok" ENTIRELY USELESS.
>
> Seriously. There's a very solid reason why I require the standard
> kernel build to be entirely warning-free. Warnings are *BAD*. Even a
> single false-positive warning invalidates all other warnings.

I totally agree, which is why fixing warnings has been at the top of the
priority list for a long time.  That has mostly been done, with Mauro
having taken on the bulk of that work.

It is hard, though, to maintain the warning-free condition.  Warnings
are easy to add and most developers don't check.  Most documentation
changes don't go through the docs tree, so I can't enforce that kind of
check (and trying to force docs changes through my tree would be a cure
far worse than the disease).

Making the docs build faster might make it easier to insist that more
checking is done.

> And the doc build isn't about some "single warning" thing.
>
>  (b) it requires some unusual build tools
>
> Now, (b) may some historical oddity, but at least if you have Fedora
> installed and it still has sphinx version 2.2.11 or something like
> that.
>
> And when you try to build docs, the makefile gives you some random pip
> install thing that is entirely bogus.

I'd be curious to know about how you got into that situation.  Fedora
handles this stuff pretty readily.  I'll take another look at what we
have there.

jon

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-27 15:28                       ` Liam Howlett
@ 2022-06-27 15:54                         ` Christian Brauner
  2022-06-27 16:27                         ` Mark Brown
  2022-06-28 16:13                         ` Luck, Tony
  2 siblings, 0 replies; 40+ messages in thread
From: Christian Brauner @ 2022-06-27 15:54 UTC (permalink / raw)
  To: Liam Howlett
  Cc: Mauro Carvalho Chehab, Linus Torvalds, Jonathan Corbet,
	Jani Nikula, ksummit, ksummit, Markus Heiser

On Mon, Jun 27, 2022 at 03:28:40PM +0000, Liam Howlett wrote:
> * Mauro Carvalho Chehab <mchehab@kernel.org> [220626 05:53]:
> > Em Sun, 26 Jun 2022 08:55:24 +0100
> > Mauro Carvalho Chehab <mchehab@kernel.org> escreveu:
> > 
> > > Em Sat, 25 Jun 2022 11:11:40 -0700
> > > Linus Torvalds <torvalds@linux-foundation.org> escreveu:
> > > 
> > > > On Sat, Jun 25, 2022 at 7:00 AM Jonathan Corbet <corbet@lwn.net> wrote:  
> > > > >
> > > > > I said "some" - this was a proof-of-concept hack.  The complexity of
> > > > > their symbol lookup code is ... daunting ... and not something that gets
> > > > > fixed in one place.  More research is required...    
> > > > 
> > > > Note that at least for me, the issue with building docs isn't hugely
> > > > performance related.
> > > > 
> > > > Yes, yes, it would be good if it was faster. But..
> > > > 
> > > > The primary problems I see with building the docs (and thus checking
> > > > them, the same way I do an allmodconfig build test) is
> > > > 
> > > >  (a) it's insanely noisy, which makes and "check that it's ok" ENTIRELY USELESS.  
> > > 
> > > When the environment is set for building docs, almost all output there
> > > are actually due to build errors/warnings that got introduced by patches
> > > touching documentation that were never build-tested.
> > > 
> > > There are 4 components that report errors during "make htmldocs". 
> > > 
> > > 1. Documentation cross-reference check. 
> > ...
> > > 2. ABI documentation check:
> > ...
> > > 3. kernel-doc warnings.
> > ...
> > 
> > Btw, once we fix the errors from the above checks, one of the things that 
> > could be done in order to avoid noisy doc builds would be to run this 
> > during normal Kernel build, if CONFIG_WERROR is set (and if .git is present
> > at the source build dir):
> > 
> > 	./scripts/documentation-file-ref-check --warn
> > 	./scripts/get_abi.pl validate
> > 	./scripts/kernel-doc --none $(git grep kernel-doc $(git ls-files Documentation/|grep -v kernel-doc.rst)|perl -ne 'print "$1\n" if (m/kernel-doc::\s*(\S+)/);'|sort|uniq) 
> > 
> > aborting the build on such warnings.
> > 
> > On my notebook (i5-10210U), the above takes ~8 seconds to run. So, it 
> > won't make much difference at the build time, and doing that would have
> > avoided ~100 warnings during htmldocs build against current linux-next.
> > 
> 
> Couldn't we add this to the build bots and ask the authors to fix the
> commits?

Isn't for-next already integrated? Maybe not. I got a mail from Andrew
telling me I had a kernel-doc copy-paste error.

Integration with build-bots would be great. I generally aim to add
kernel-doc for all non-trivial code that I add. But during different
patchset versions copy-paste errors do sometimes happen (e.g., for git
grep/sed calls).

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-27 15:28                       ` Liam Howlett
  2022-06-27 15:54                         ` Christian Brauner
@ 2022-06-27 16:27                         ` Mark Brown
  2022-06-28 10:53                           ` Mauro Carvalho Chehab
  2022-06-28 16:13                         ` Luck, Tony
  2 siblings, 1 reply; 40+ messages in thread
From: Mark Brown @ 2022-06-27 16:27 UTC (permalink / raw)
  To: Liam Howlett
  Cc: Mauro Carvalho Chehab, Linus Torvalds, Jonathan Corbet,
	Jani Nikula, ksummit, ksummit, Markus Heiser

[-- Attachment #1: Type: text/plain, Size: 1374 bytes --]

On Mon, Jun 27, 2022 at 03:28:40PM +0000, Liam Howlett wrote:
> * Mauro Carvalho Chehab <mchehab@kernel.org> [220626 05:53]:

> > Btw, once we fix the errors from the above checks, one of the things that 
> > could be done in order to avoid noisy doc builds would be to run this 
> > during normal Kernel build, if CONFIG_WERROR is set (and if .git is present
> > at the source build dir):

> > 	./scripts/documentation-file-ref-check --warn
> > 	./scripts/get_abi.pl validate
> > 	./scripts/kernel-doc --none $(git grep kernel-doc $(git ls-files Documentation/|grep -v kernel-doc.rst)|perl -ne 'print "$1\n" if (m/kernel-doc::\s*(\S+)/);'|sort|uniq) 

That last one is quite the command line...

> > aborting the build on such warnings.

> > On my notebook (i5-10210U), the above takes ~8 seconds to run. So, it 
> > won't make much difference at the build time, and doing that would have
> > avoided ~100 warnings during htmldocs build against current linux-next.

> Couldn't we add this to the build bots and ask the authors to fix the
> commits?

There's reports for at least some of this for -next already, at least
for newly introduced warnings.  If we could get 0day or something else
that picks stuff off the lists that would help a lot I think, as would
getting it into the bots that people who like to fix up warnings tend to
be using.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-27 15:34                   ` Jonathan Corbet
@ 2022-06-27 17:07                     ` Linus Torvalds
  0 siblings, 0 replies; 40+ messages in thread
From: Linus Torvalds @ 2022-06-27 17:07 UTC (permalink / raw)
  To: Jonathan Corbet
  Cc: Mauro Carvalho Chehab, Jani Nikula, ksummit, ksummit, Markus Heiser

On Mon, Jun 27, 2022 at 8:34 AM Jonathan Corbet <corbet@lwn.net> wrote:
>
> >  (b) it requires some unusual build tools
> >
> > Now, (b) may some historical oddity, but at least if you have Fedora
> > installed and it still has sphinx version 2.2.11 or something like
> > that.
> >
> > And when you try to build docs, the makefile gives you some random pip
> > install thing that is entirely bogus.
>
> I'd be curious to know about how you got into that situation.  Fedora
> handles this stuff pretty readily.  I'll take another look at what we
> have there.

At some point I just did "dnf install sphinx", and it got me a sphinx install.

So then the kernel doc build system sees "oh, you need a newer version
of sphinx" or whatever, and suggests you do that horrible virtual
python sandbox thing and do a random pip install in that, which I
really don't want to do.

So I ignore it entirely, obviously.

End result: it took me a *loong* time to go "I wonder why modern
Fedora ships with an ancient version of sphinx, since the modern
version is apparently 4.x something".

And only at THAT point do you realize that there are three completely
different projects called "sphinx".

There's the docbuild one that comes from python, which is the sphinx I want.

There's the search engine 'sphinx', which is what is packaged as
"sphinx" in Fedora, and which the documentation code obviously refuses
to use.

There's also CMU sphinx, which is a voice recognition system.

End result: building the docs can be really confusing if you have the
wrong sphinx installed, and the "help message" is actually misleading
and very unhelpful.

              Linus

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-27 15:27             ` Jonathan Corbet
  2022-06-27 15:31               ` Christoph Hellwig
@ 2022-06-28  7:43               ` Mauro Carvalho Chehab
  2022-06-28  7:57                 ` Geert Uytterhoeven
  1 sibling, 1 reply; 40+ messages in thread
From: Mauro Carvalho Chehab @ 2022-06-28  7:43 UTC (permalink / raw)
  To: Jonathan Corbet; +Cc: Markus Heiser, Jani Nikula, ksummit-discuss, ksummit

Em Mon, 27 Jun 2022 09:27:54 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Markus Heiser <markus.heiser@darmarit.de> writes:
> 
> > IMO It is unnecessary that the build-chain must run on all
> > platforms and with all distributions.
> >
> > Who observes the Sphinx-doc & docutils development since (>15)
> > years is aware that with various (old) Sphinx-doc & docutils
> > versions no stable results can be produced, not without
> > complicating the build-chain.  And this is exactly the situation
> > we are facing today.
> >
> > The build chain of documentation has nothing to do with kernel
> > development (at least in my opinion) and should be decoupled from it:
> > maintaining one defined build environment is enough work ... this
> > becomes especially clear if you (as Jani recommends) rely more on
> > sphinx-modules and widely used tools.  
> 
> The counterargument to this is that we want as many developers as
> possible to be able to build the docs and contribute to them.  We can't
> complain that developers have broken the docs build if we don't do what
> we can to help them do the build themselves.

Agreed.

> One of our longstanding contributors is on Sphinx 1.8.5:
> 
>   https://lwn.net/ml/linux-doc/4c403239-3c71-4ab9-2168-f7e9d77008b2%40infradead.org/

If he migrates to Leap 15.4, he'll either get 2.3.1 (default package)
or 4.2.0 (alternative package):

	https://download.opensuse.org/distribution/leap/15.4/repo/oss/noarch/python3-Sphinx-2.3.1-150400.1.7.noarch.rpm
	https://download.opensuse.org/distribution/leap/15.4/repo/oss/noarch/python3-Sphinx_4_2_0-4.2.0-150400.11.6.noarch.rpm

> I would like to narrow our range of supported versions, but I really
> don't want to cut people out.
> 
> The real question, perhaps, is whether requiring people to set up a
> virtualenv to run a supported version would be too much.  It's something
> I worry about.

Installing distro-specific packages likely covers a broader group of
developers than requiring virtualenv/venv.

However, if we decide to move on, removing the version-dependent
hacks from the build system, it would mean that the minimal version
would be 3.1.0. On such case, there won't be Sphinx packages on some
LTS distros. Developers using such distros will be forced to use 
virtualenv/venv.

In any case, the script which checks if sphinx-build runs can recommend
either way. Yet, as the current default is venv, tests are needed to check
if it recommending python3-sphinx packages still work on all supported
distros.

Such script could also be changed to detect if the distro has a
native package >= 3.1.0, recommending it by default, falling back to
virtualenv/venv otherwise.

Regards,
Mauro


^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-28  7:43               ` Mauro Carvalho Chehab
@ 2022-06-28  7:57                 ` Geert Uytterhoeven
  2022-06-28 11:01                   ` Mauro Carvalho Chehab
  0 siblings, 1 reply; 40+ messages in thread
From: Geert Uytterhoeven @ 2022-06-28  7:57 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Jonathan Corbet, Markus Heiser, Jani Nikula, ksummit, ksummit

Hi Mauro,

On Tue, Jun 28, 2022 at 9:43 AM Mauro Carvalho Chehab
<mchehab@kernel.org> wrote:
> Em Mon, 27 Jun 2022 09:27:54 -0600
> Jonathan Corbet <corbet@lwn.net> escreveu:
> > Markus Heiser <markus.heiser@darmarit.de> writes:
> > > IMO It is unnecessary that the build-chain must run on all
> > > platforms and with all distributions.
> > >
> > > Who observes the Sphinx-doc & docutils development since (>15)
> > > years is aware that with various (old) Sphinx-doc & docutils
> > > versions no stable results can be produced, not without
> > > complicating the build-chain.  And this is exactly the situation
> > > we are facing today.
> > >
> > > The build chain of documentation has nothing to do with kernel
> > > development (at least in my opinion) and should be decoupled from it:
> > > maintaining one defined build environment is enough work ... this
> > > becomes especially clear if you (as Jani recommends) rely more on
> > > sphinx-modules and widely used tools.
> >
> > The counterargument to this is that we want as many developers as
> > possible to be able to build the docs and contribute to them.  We can't
> > complain that developers have broken the docs build if we don't do what
> > we can to help them do the build themselves.
>
> Agreed.
>
> > One of our longstanding contributors is on Sphinx 1.8.5:
> >
> >   https://lwn.net/ml/linux-doc/4c403239-3c71-4ab9-2168-f7e9d77008b2%40infradead.org/

Ubuntu 20.04 LTS also has 1.8.5.
Looks like I'll get 4.3.2 after upgrading to 22.04 LTS (which is only
about two months old).

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-27 16:27                         ` Mark Brown
@ 2022-06-28 10:53                           ` Mauro Carvalho Chehab
  0 siblings, 0 replies; 40+ messages in thread
From: Mauro Carvalho Chehab @ 2022-06-28 10:53 UTC (permalink / raw)
  To: Mark Brown
  Cc: Liam Howlett, Linus Torvalds, Jonathan Corbet, Jani Nikula,
	ksummit, ksummit, Markus Heiser

Em Mon, 27 Jun 2022 17:27:27 +0100
Mark Brown <broonie@kernel.org> escreveu:

> On Mon, Jun 27, 2022 at 03:28:40PM +0000, Liam Howlett wrote:
> > * Mauro Carvalho Chehab <mchehab@kernel.org> [220626 05:53]:  
> 
> > > Btw, once we fix the errors from the above checks, one of the things that 
> > > could be done in order to avoid noisy doc builds would be to run this 
> > > during normal Kernel build, if CONFIG_WERROR is set (and if .git is present
> > > at the source build dir):  
> 
> > > 	./scripts/documentation-file-ref-check --warn

Sent a patch series addressing those at linux-next:
	https://lore.kernel.org/linux-doc/cover.1656234456.git.mchehab@kernel.org/T/#t

> > > 	./scripts/get_abi.pl validate

I submited a patch fixing one such issue:
	https://lore.kernel.org/all/1e92337c1ef74f5eb9e1c1871e20b858b490d269.1656235926.git.mchehab@kernel.org/ 

and Jonathan Cameron submitted a series addressing those:

	https://lore.kernel.org/linux-iio/20220626165511.602202-1-jic23@kernel.org/T/#t 

> > > 	./scripts/kernel-doc --none $(git grep kernel-doc $(git ls-files Documentation/|grep -v kernel-doc.rst)|perl -ne 'print "$1\n" if (m/kernel-doc::\s*(\S+)/);'|sort|uniq)   

Sent a patch series addressing all of those, also against linux-next:

	https://lore.kernel.org/linux-doc/cover.1656409369.git.mchehab@kernel.org/T/#t

next-20220627 has currently:
	32 cross broken docs cross reference warnings;
	 4 duplicated ABI symbols;
	60 kernel-doc warnings.

After having those three series applied, the 3 above commands would
produce just 3 warnings:

	Warning: Documentation/dev-tools/kunit/run_wrapper.rst references a file that doesn't exist: Documentation/dev-tools/kunit/non_uml.rst
	Warning: Documentation/devicetree/bindings/regulator/siliconmitus,sm5703-regulator.yaml references a file that doesn't exist: Documentation/devicetree/bindings/mfd/siliconmitus,sm5703.yaml
	Warning: drivers/acpi/device_pm.c references a file that doesn't exist: Documentation/firmware-guide/acpi/low-power-probe.rst

Those 3 warnings depend on non-merged documents.

After those ~100 warnings that can be quickly checked with the 3 above
command lines in a couple of seconds, there are the Sphinx errors.
Reducing the build time (currently at 10 mins order) to something more
palatable could help having more people checking them before submitting
patches and waiting for 0day results.

> That last one is quite the command line...

True. Yet, placing it at docs/Makefile for the "make all" target should be 
trivial and will hopefully make more Kernel developers to check for some of 
the documentation issues. 

The other two are already called, depending on some config vars:

	# Check for broken documentation file references
	ifeq ($(CONFIG_WARN_MISSING_DOCUMENTS),y)
	$(shell $(srctree)/scripts/documentation-file-ref-check --warn)
	endif

	# Check for broken ABI files
	ifeq ($(CONFIG_WARN_ABI_ERRORS),y)
	$(shell $(srctree)/scripts/get_abi.pl validate --dir $(srctree)/Documentation/ABI)
	endif

But they don't currently stop the build if CONFIG_WERROR is enabled.


> 
> > > aborting the build on such warnings.  
> 
> > > On my notebook (i5-10210U), the above takes ~8 seconds to run. So, it 
> > > won't make much difference at the build time, and doing that would have
> > > avoided ~100 warnings during htmldocs build against current linux-next.  
> 
> > Couldn't we add this to the build bots and ask the authors to fix the
> > commits?  
> 
> There's reports for at least some of this for -next already, at least
> for newly introduced warnings.  If we could get 0day or something else
> that picks stuff off the lists that would help a lot I think, as would
> getting it into the bots that people who like to fix up warnings tend to
> be using.

Yes, newly-introduced warnings are monitored there, but still patches
adding new warnings keep being merged, requiring extra periodic janitorial
work in order to reduce the build noise that would otherwise increase
with time.

Regards,
Mauro

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-28  7:57                 ` Geert Uytterhoeven
@ 2022-06-28 11:01                   ` Mauro Carvalho Chehab
  2022-07-02 12:43                     ` Stephen Rothwell
  0 siblings, 1 reply; 40+ messages in thread
From: Mauro Carvalho Chehab @ 2022-06-28 11:01 UTC (permalink / raw)
  To: Geert Uytterhoeven
  Cc: Jonathan Corbet, Markus Heiser, Jani Nikula, ksummit, ksummit

Em Tue, 28 Jun 2022 09:57:29 +0200
Geert Uytterhoeven <geert@linux-m68k.org> escreveu:

> Hi Mauro,
> 
> On Tue, Jun 28, 2022 at 9:43 AM Mauro Carvalho Chehab
> <mchehab@kernel.org> wrote:
> > Em Mon, 27 Jun 2022 09:27:54 -0600
> > Jonathan Corbet <corbet@lwn.net> escreveu:  
> > > Markus Heiser <markus.heiser@darmarit.de> writes:  
> > > > IMO It is unnecessary that the build-chain must run on all
> > > > platforms and with all distributions.
> > > >
> > > > Who observes the Sphinx-doc & docutils development since (>15)
> > > > years is aware that with various (old) Sphinx-doc & docutils
> > > > versions no stable results can be produced, not without
> > > > complicating the build-chain.  And this is exactly the situation
> > > > we are facing today.
> > > >
> > > > The build chain of documentation has nothing to do with kernel
> > > > development (at least in my opinion) and should be decoupled from it:
> > > > maintaining one defined build environment is enough work ... this
> > > > becomes especially clear if you (as Jani recommends) rely more on
> > > > sphinx-modules and widely used tools.  
> > >
> > > The counterargument to this is that we want as many developers as
> > > possible to be able to build the docs and contribute to them.  We can't
> > > complain that developers have broken the docs build if we don't do what
> > > we can to help them do the build themselves.  
> >
> > Agreed.
> >  
> > > One of our longstanding contributors is on Sphinx 1.8.5:
> > >
> > >   https://lwn.net/ml/linux-doc/4c403239-3c71-4ab9-2168-f7e9d77008b2%40infradead.org/  
> 
> Ubuntu 20.04 LTS also has 1.8.5.
> Looks like I'll get 4.3.2 after upgrading to 22.04 LTS (which is only
> about two months old).

Yeah, LTS distros are the main reason why the default recommendation is to
install Sphinx via virtualenv/venv.

With RHEL9, SUSE 15 SP4 (plus CenOS and openSUSE Leap versions of them),
plus Ubuntu 22.04, maybe we can change the default to recommend installing
it via distro-provided packages. Yet, Debian bulseye has 2.5.0. Only
bookworm, scheduled for mid 2023, will come with Sphinx > 2.

Regards,
Mauro




^ permalink raw reply	[flat|nested] 40+ messages in thread

* RE: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-27 15:28                       ` Liam Howlett
  2022-06-27 15:54                         ` Christian Brauner
  2022-06-27 16:27                         ` Mark Brown
@ 2022-06-28 16:13                         ` Luck, Tony
  2 siblings, 0 replies; 40+ messages in thread
From: Luck, Tony @ 2022-06-28 16:13 UTC (permalink / raw)
  To: Liam Howlett, Mauro Carvalho Chehab
  Cc: Linus Torvalds, Jonathan Corbet, Nikula, Jani, ksummit, ksummit,
	Markus Heiser

>> On my notebook (i5-10210U), the above takes ~8 seconds to run. So, it 
>> won't make much difference at the build time, and doing that would have
>> avoided ~100 warnings during htmldocs build against current linux-next.
>> 
>
> Couldn't we add this to the build bots and ask the authors to fix the
> commits?

I poked the zero-day bot owners. They are looking into it.

Reminder that the zero-day folks are open to suggestions on how to
improve coverage.  Just ping me or Dave Hansen and we can pass
on requests.

-Tony

^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-25 18:11                 ` Linus Torvalds
  2022-06-26  7:55                   ` Mauro Carvalho Chehab
  2022-06-27 15:34                   ` Jonathan Corbet
@ 2022-07-02 10:52                   ` Mauro Carvalho Chehab
  2 siblings, 0 replies; 40+ messages in thread
From: Mauro Carvalho Chehab @ 2022-07-02 10:52 UTC (permalink / raw)
  To: Linus Torvalds
  Cc: Jonathan Corbet, Jani Nikula, ksummit, ksummit, Markus Heiser

Em Sat, 25 Jun 2022 11:11:40 -0700
Linus Torvalds <torvalds@linux-foundation.org> escreveu:

> On Sat, Jun 25, 2022 at 7:00 AM Jonathan Corbet <corbet@lwn.net> wrote:
> >
> > I said "some" - this was a proof-of-concept hack.  The complexity of
> > their symbol lookup code is ... daunting ... and not something that gets
> > fixed in one place.  More research is required...  
> 
> Note that at least for me, the issue with building docs isn't hugely
> performance related.
> 
> Yes, yes, it would be good if it was faster. But..
> 
> The primary problems I see with building the docs (and thus checking
> them, the same way I do an allmodconfig build test) is
> 
>  (a) it's insanely noisy, which makes and "check that it's ok" ENTIRELY USELESS.

I solved almost all warnings that can be fixed against -next.
Sending patches today.

However, there is an special case where warnings will be hit for perfectly
fine code, when docs are built with Sphinx 3.0 and upper:

	Documentation/driver-api/usb/usb:164: ./drivers/usb/core/message.c:967: WARNING: Duplicate C declaration, also defined at driver-api/usb/gadget:783.
	Declaration is '.. c:function:: int usb_string (struct usb_device *dev, int index, char *buf, size_t size)'.
	Documentation/driver-api/usb/usb.rst:967: WARNING: Duplicate C declaration, also defined at driver-api/usb/gadget:783.
	Declaration is '.. c:struct:: usb_string'.

In this specific case, there are both a struct and a function named 
"usb_string". So, the above is not an issue. While this is not fixed,
there will be a 22 lines noise (11 two-line warnings) after my fixes[1].

The root cause is that Sphinx has a single namespace for symbols, no matter
if the symbol is a function, a struct, an enum, etc. So, when the same
name is declared on multiple places, like in the case of "usb_string",
the cross-references will either point to the function or to the struct.
While this is annoying, it is usually not a big deal, as they are
typically placed at the same rst file.

While this problem affects *all* Sphinx versions so far, Sphinx 2.x.x 
doesn't warn when this happen, but newer versions complain about that. 
This is a well known issue and there are patches fixing it since 
Sphinx 3.1:

	https://github.com/sphinx-doc/sphinx/pull/8313

Those patches were expected to be merged on Sphinx 3.3, but ended
not being merged so far, as applying them would break support for
a Sphinx extension that it is used on several places.

---

[1] I got one case where the warning was actually reporting an issue:

	Documentation/driver-api/hte/tegra194-hte:28: ./drivers/gpio/gpiolib.c:2464: WARNING: Duplicate C declaration, also defined at driver-api>
	Declaration is '.. c:function:: int gpiod_enable_hw_timestamp_ns (struct gpio_desc *desc, unsigned long flags)'.
	Documentation/driver-api/hte/tegra194-hte.rst:2464: WARNING: Duplicate C declaration, also defined at driver-api/gpio/index:2464.
	Declaration is '.. c:function:: int gpiod_enable_hw_timestamp_ns(struct gpio_desc *desc, unsigned long flags)'.

Basically drivers/gpio/gpiolib.c is included twice for the same
symbols, one at Documentation/driver-api/gpio/index.rst, and then 
Documentation/driver-api/hte/tegra194-hte.rst includes it again for 
gpiod_enable_hw_timestamp_ns() and gpiod_disable_hw_timestamp_ns().

Regards,
Mauro





^ permalink raw reply	[flat|nested] 40+ messages in thread

* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be
  2022-06-28 11:01                   ` Mauro Carvalho Chehab
@ 2022-07-02 12:43                     ` Stephen Rothwell
  0 siblings, 0 replies; 40+ messages in thread
From: Stephen Rothwell @ 2022-07-02 12:43 UTC (permalink / raw)
  To: Mauro Carvalho Chehab
  Cc: Geert Uytterhoeven, Jonathan Corbet, Markus Heiser, Jani Nikula,
	ksummit, ksummit

[-- Attachment #1: Type: text/plain, Size: 530 bytes --]

Hi Mauro,

On Tue, 28 Jun 2022 12:01:42 +0100 Mauro Carvalho Chehab <mchehab@kernel.org> wrote:
>
> With RHEL9, SUSE 15 SP4 (plus CenOS and openSUSE Leap versions of them),
> plus Ubuntu 22.04, maybe we can change the default to recommend installing
> it via distro-provided packages. Yet, Debian bulseye has 2.5.0. Only
> bookworm, scheduled for mid 2023, will come with Sphinx > 2.

Debian current Stable (Bullseye) has python3-sphinx version 3.4.3
(python-sphinx is version 1.8.4).

-- 
Cheers,
Stephen Rothwell

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 40+ messages in thread

end of thread, other threads:[~2022-07-02 12:43 UTC | newest]

Thread overview: 40+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-17 20:57 [TECH TOPIC] What kernel documentation could be Jonathan Corbet
2022-06-17 20:57 ` [Ksummit-discuss] " Jonathan Corbet
2022-06-17 21:48 ` Laurent Pinchart
2022-06-17 21:48   ` Laurent Pinchart
2022-06-27 15:18   ` Jonathan Corbet
2022-06-18  8:24 ` Mauro Carvalho Chehab
2022-06-18  8:24   ` Mauro Carvalho Chehab
2022-06-18 11:03   ` Miguel Ojeda
2022-06-18 11:16     ` Mauro Carvalho Chehab
2022-06-18 11:16       ` Mauro Carvalho Chehab
2022-06-18 14:37       ` Miguel Ojeda
2022-06-23  9:18   ` Jani Nikula
2022-06-23  9:57     ` Mauro Carvalho Chehab
2022-06-23 10:30       ` Jani Nikula
2022-06-23 13:40       ` Jonathan Corbet
2022-06-24  7:33         ` Mauro Carvalho Chehab
2022-06-24 16:37           ` Markus Heiser
2022-06-27 15:27             ` Jonathan Corbet
2022-06-27 15:31               ` Christoph Hellwig
2022-06-28  7:43               ` Mauro Carvalho Chehab
2022-06-28  7:57                 ` Geert Uytterhoeven
2022-06-28 11:01                   ` Mauro Carvalho Chehab
2022-07-02 12:43                     ` Stephen Rothwell
2022-06-24 22:57           ` Jonathan Corbet
2022-06-25  9:10             ` Mauro Carvalho Chehab
2022-06-25 14:00               ` Jonathan Corbet
2022-06-25 18:11                 ` Linus Torvalds
2022-06-26  7:55                   ` Mauro Carvalho Chehab
2022-06-26  9:26                     ` Mauro Carvalho Chehab
2022-06-26  9:53                     ` Mauro Carvalho Chehab
2022-06-27 15:28                       ` Liam Howlett
2022-06-27 15:54                         ` Christian Brauner
2022-06-27 16:27                         ` Mark Brown
2022-06-28 10:53                           ` Mauro Carvalho Chehab
2022-06-28 16:13                         ` Luck, Tony
2022-06-27 15:34                   ` Jonathan Corbet
2022-06-27 17:07                     ` Linus Torvalds
2022-07-02 10:52                   ` Mauro Carvalho Chehab
2022-06-25 17:43 ` Steven Rostedt
2022-06-25 17:48   ` Laurent Pinchart

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).