* [TECH TOPIC] What kernel documentation could be @ 2022-06-17 20:57 Jonathan Corbet 2022-06-17 20:57 ` [Ksummit-discuss] " Jonathan Corbet ` (3 more replies) 0 siblings, 4 replies; 40+ messages in thread From: Jonathan Corbet @ 2022-06-17 20:57 UTC (permalink / raw) To: ksummit-discuss, ksummit The development community has put a lot of work into the kernel's documentation directory in recent years, with visible results. But the kernel's documentation still falls far short of the standard set by many other projects, and there is a great deal of "tribal knowledge" in our community that is not set down. I'd like to look at the successes and failures of the work so far, but intent to focus on a discussion of what our documentation should be and what we can do to get it there. Questions to discuss include: - Bringing an coherent overall structure to Documentation/ - Making it easier for developers to improve the documentation. - What we would like from Sphinx and what we can do to make it happen - Making the docs build system less ugly and more maintainable - Integrating rustdoc documentation jon ^ permalink raw reply [flat|nested] 40+ messages in thread
* [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-17 20:57 [TECH TOPIC] What kernel documentation could be Jonathan Corbet @ 2022-06-17 20:57 ` Jonathan Corbet 2022-06-17 21:48 ` Laurent Pinchart ` (2 subsequent siblings) 3 siblings, 0 replies; 40+ messages in thread From: Jonathan Corbet @ 2022-06-17 20:57 UTC (permalink / raw) The development community has put a lot of work into the kernel's documentation directory in recent years, with visible results. But the kernel's documentation still falls far short of the standard set by many other projects, and there is a great deal of "tribal knowledge" in our community that is not set down. I'd like to look at the successes and failures of the work so far, but intent to focus on a discussion of what our documentation should be and what we can do to get it there. Questions to discuss include: - Bringing an coherent overall structure to Documentation/ - Making it easier for developers to improve the documentation. - What we would like from Sphinx and what we can do to make it happen - Making the docs build system less ugly and more maintainable - Integrating rustdoc documentation jon ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-17 20:57 [TECH TOPIC] What kernel documentation could be Jonathan Corbet 2022-06-17 20:57 ` [Ksummit-discuss] " Jonathan Corbet @ 2022-06-17 21:48 ` Laurent Pinchart 2022-06-17 21:48 ` Laurent Pinchart 2022-06-27 15:18 ` Jonathan Corbet 2022-06-18 8:24 ` Mauro Carvalho Chehab 2022-06-25 17:43 ` Steven Rostedt 3 siblings, 2 replies; 40+ messages in thread From: Laurent Pinchart @ 2022-06-17 21:48 UTC (permalink / raw) To: Jonathan Corbet; +Cc: ksummit-discuss, ksummit Hi Jon, On Fri, Jun 17, 2022 at 02:57:10PM -0600, Jonathan Corbet wrote: > The development community has put a lot of work into the kernel's > documentation directory in recent years, with visible results. But the > kernel's documentation still falls far short of the standard set by many > other projects, and there is a great deal of "tribal knowledge" in our > community that is not set down. I'd like to look at the successes and > failures of the work so far, but intent to focus on a discussion of what > our documentation should be and what we can do to get it there. > > Questions to discuss include: > > - Bringing an coherent overall structure to Documentation/ > > - Making it easier for developers to improve the documentation. I've been wondering recently if it would help to have a place where we can record documentation wishes. It has happened quite a few times that my searches for particular pieces of documentation in the kernel were unfruitful, or that the related documentation left me none the wiser (one example, which is just an example and not finger-pointing, is the runtime PM documentation in Documentation/power/runtime_pm.rst that feels like reading a calculus-related PHD thesis when all you want is to understand how to add two numbers). I rarely had time to write documentation patches as a result, and my pains were thus left unnoticed by maintainers and developers, who may be willing to improve the documentation if they knew that users were struggling. > - What we would like from Sphinx and what we can do to make it happen > > - Making the docs build system less ugly and more maintainable > > - Integrating rustdoc documentation -- Regards, Laurent Pinchart ^ permalink raw reply [flat|nested] 40+ messages in thread
* [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-17 21:48 ` Laurent Pinchart @ 2022-06-17 21:48 ` Laurent Pinchart 2022-06-27 15:18 ` Jonathan Corbet 1 sibling, 0 replies; 40+ messages in thread From: Laurent Pinchart @ 2022-06-17 21:48 UTC (permalink / raw) Hi Jon, On Fri, Jun 17, 2022 at 02:57:10PM -0600, Jonathan Corbet wrote: > The development community has put a lot of work into the kernel's > documentation directory in recent years, with visible results. But the > kernel's documentation still falls far short of the standard set by many > other projects, and there is a great deal of "tribal knowledge" in our > community that is not set down. I'd like to look at the successes and > failures of the work so far, but intent to focus on a discussion of what > our documentation should be and what we can do to get it there. > > Questions to discuss include: > > - Bringing an coherent overall structure to Documentation/ > > - Making it easier for developers to improve the documentation. I've been wondering recently if it would help to have a place where we can record documentation wishes. It has happened quite a few times that my searches for particular pieces of documentation in the kernel were unfruitful, or that the related documentation left me none the wiser (one example, which is just an example and not finger-pointing, is the runtime PM documentation in Documentation/power/runtime_pm.rst that feels like reading a calculus-related PHD thesis when all you want is to understand how to add two numbers). I rarely had time to write documentation patches as a result, and my pains were thus left unnoticed by maintainers and developers, who may be willing to improve the documentation if they knew that users were struggling. > - What we would like from Sphinx and what we can do to make it happen > > - Making the docs build system less ugly and more maintainable > > - Integrating rustdoc documentation -- Regards, Laurent Pinchart ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-17 21:48 ` Laurent Pinchart 2022-06-17 21:48 ` Laurent Pinchart @ 2022-06-27 15:18 ` Jonathan Corbet 1 sibling, 0 replies; 40+ messages in thread From: Jonathan Corbet @ 2022-06-27 15:18 UTC (permalink / raw) To: Laurent Pinchart; +Cc: ksummit-discuss, ksummit [yes, I'm running even slower than usual...] Laurent Pinchart <laurent.pinchart@ideasonboard.com> writes: > I've been wondering recently if it would help to have a place where we > can record documentation wishes. One of the conclusions that came out of the LSFMM session on MM documentation was that having an overall structure in place makes it easier for people to fill in pieces. If we could impose more of this kind of structure on our docs as a whole, one obvious way to express wishes of this sort would be to add an empty section where the need exists and hope that it inspires somebody to fill it in. Thanks, jon ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-17 20:57 [TECH TOPIC] What kernel documentation could be Jonathan Corbet 2022-06-17 20:57 ` [Ksummit-discuss] " Jonathan Corbet 2022-06-17 21:48 ` Laurent Pinchart @ 2022-06-18 8:24 ` Mauro Carvalho Chehab 2022-06-18 8:24 ` Mauro Carvalho Chehab ` (2 more replies) 2022-06-25 17:43 ` Steven Rostedt 3 siblings, 3 replies; 40+ messages in thread From: Mauro Carvalho Chehab @ 2022-06-18 8:24 UTC (permalink / raw) To: Jonathan Corbet; +Cc: ksummit-discuss, ksummit Hi John, Em Fri, 17 Jun 2022 14:57:10 -0600 Jonathan Corbet <corbet@lwn.net> escreveu: > The development community has put a lot of work into the kernel's > documentation directory in recent years, with visible results. But the > kernel's documentation still falls far short of the standard set by many > other projects, and there is a great deal of "tribal knowledge" in our > community that is not set down. I'd like to look at the successes and > failures of the work so far, but intent to focus on a discussion of what > our documentation should be and what we can do to get it there. > > Questions to discuss include: > > - Bringing an coherent overall structure to Documentation/ Fully agreed. We dedicated a lot of resources on the past years to convert documentation to ReST and on kernel-doc, but not so much on placing the topics at the right order. There are a huge amount of such documentation that describe border cases, or have just kernel-doc tags without any (or very small) descriptions. Kernel-doc markups are important, as they help to keep the documentation updated, but explaining subsystem's principles is equally important, as it can save a lot of time from maintainers if the contributors are aware on how and why the subsystem's kAPIs were organized the way it is. We should imagine documentation as if this is a series of books describing every aspect of the kernel. So, I would expect it to be organized with some structure that would place the topics on some order that would make easier for people to read, being either new contributors or experienced developers that need to touch other parts of the Kernel that are out of his past experiences. For instance to the kAPI documentation, I would expect something like: - How to contribute / Submission process; - An introductory chapter that would describe the most used components that all developers need to know, like: - how to write a driver; - fs principles; - scheduler principles; - mm most used functions (kmalloc & friends); - wait queues; - kernel threads; - softirqs; - arch-dependent items to consider (like bit order), weak memory model, etc; - ... IMO, we could take a look at the index of some already-existing books like LDD 3, Linux Kernel Development and others in order to get a rough idea about the items to be covered there. - Bus-related subsystems (USB, PCI, I2C, ...); - Then, place per-subsystem's documentation, all having their texts explaining basic principles. IMO, we should write an index file and a couple of new ReST files with an skeleton for the above, and then ask people to help filling the blanks. > - Making it easier for developers to improve the documentation. A well-prepared skeleton would make life easier. From time to time, we have discussions and patches shifting documentation between different directories. > - What we would like from Sphinx and what we can do to make it happen The big missing feature on Sphinx itself is with regards to per-C-type domain. So, if we have one struct and one function both called "foo", the cross-references will be broken. This issue is known since Sphinx 3.1, and there are already patches fixing it since then (I remember testing them) but, up to today, the Sphinx upstream patches meant to fix it weren't applied yet (as far as I can tell). Another problem is with regards to the documentation's build time. One feature we're not using yet is intersphinx. Right now, we can build parts of the documentation with: make SPHINXDIRS="foo" htmldocs But, if "foo" have cross-references to "bar", the build will produce warnings and the documentation's cross-references to "bar" won't work. Properly setting interphinx would allow this to work, as such references would point, instead, to some remote place (like kernel.org). This could be used to help improving the documentation's build time during doc development. Also, I would love to have a good way to just produce html (and pdf) from the documents I'm actually working with. The way I do it right now is that I create a "Documentation/foo" directory, adding there just the docs I'm currently working with, placing them on a fake index.rst file. This way, I can build them really fast, without needing to rebuild everything at the same book. Perhaps something like that could be automated - or Sphinx itself could support something like: make htmldocs SPHINXFILES="foo.rst bar.rst" > - Making the docs build system less ugly and more maintainable I guess shifting the minimal Sphinx version would help to remove some bad things there. One thing that probably be solved on a different way is to have a better solution for things like: Functions for feature 1 ======================= .. kernel-doc:: include/some_header.h :doc: Feature 1 .. kernel-doc:: include/some_header.h :functions: func1 func2 Functions for feature 2 ======================= .. kernel-doc:: include/some_header.h :doc: Feature 2 .. kernel-doc:: include/some_header.h :functions: func3 func4 Perhaps we could change Kernel-doc in a way that doing just: .. kernel-doc:: include/some_header.h would be enough. > - Integrating rustdoc documentation Won't comment much about that, as never touched any of those. Life would be a lot easier on this side if rustdoc could support ReST. Regards, Mauro ^ permalink raw reply [flat|nested] 40+ messages in thread
* [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-18 8:24 ` Mauro Carvalho Chehab @ 2022-06-18 8:24 ` Mauro Carvalho Chehab 2022-06-18 11:03 ` Miguel Ojeda 2022-06-23 9:18 ` Jani Nikula 2 siblings, 0 replies; 40+ messages in thread From: Mauro Carvalho Chehab @ 2022-06-18 8:24 UTC (permalink / raw) Hi John, Em Fri, 17 Jun 2022 14:57:10 -0600 Jonathan Corbet <corbet@lwn.net> escreveu: > The development community has put a lot of work into the kernel's > documentation directory in recent years, with visible results. But the > kernel's documentation still falls far short of the standard set by many > other projects, and there is a great deal of "tribal knowledge" in our > community that is not set down. I'd like to look at the successes and > failures of the work so far, but intent to focus on a discussion of what > our documentation should be and what we can do to get it there. > > Questions to discuss include: > > - Bringing an coherent overall structure to Documentation/ Fully agreed. We dedicated a lot of resources on the past years to convert documentation to ReST and on kernel-doc, but not so much on placing the topics at the right order. There are a huge amount of such documentation that describe border cases, or have just kernel-doc tags without any (or very small) descriptions. Kernel-doc markups are important, as they help to keep the documentation updated, but explaining subsystem's principles is equally important, as it can save a lot of time from maintainers if the contributors are aware on how and why the subsystem's kAPIs were organized the way it is. We should imagine documentation as if this is a series of books describing every aspect of the kernel. So, I would expect it to be organized with some structure that would place the topics on some order that would make easier for people to read, being either new contributors or experienced developers that need to touch other parts of the Kernel that are out of his past experiences. For instance to the kAPI documentation, I would expect something like: - How to contribute / Submission process; - An introductory chapter that would describe the most used components that all developers need to know, like: - how to write a driver; - fs principles; - scheduler principles; - mm most used functions (kmalloc & friends); - wait queues; - kernel threads; - softirqs; - arch-dependent items to consider (like bit order), weak memory model, etc; - ... IMO, we could take a look at the index of some already-existing books like LDD 3, Linux Kernel Development and others in order to get a rough idea about the items to be covered there. - Bus-related subsystems (USB, PCI, I2C, ...); - Then, place per-subsystem's documentation, all having their texts explaining basic principles. IMO, we should write an index file and a couple of new ReST files with an skeleton for the above, and then ask people to help filling the blanks. > - Making it easier for developers to improve the documentation. A well-prepared skeleton would make life easier. From time to time, we have discussions and patches shifting documentation between different directories. > - What we would like from Sphinx and what we can do to make it happen The big missing feature on Sphinx itself is with regards to per-C-type domain. So, if we have one struct and one function both called "foo", the cross-references will be broken. This issue is known since Sphinx 3.1, and there are already patches fixing it since then (I remember testing them) but, up to today, the Sphinx upstream patches meant to fix it weren't applied yet (as far as I can tell). Another problem is with regards to the documentation's build time. One feature we're not using yet is intersphinx. Right now, we can build parts of the documentation with: make SPHINXDIRS="foo" htmldocs But, if "foo" have cross-references to "bar", the build will produce warnings and the documentation's cross-references to "bar" won't work. Properly setting interphinx would allow this to work, as such references would point, instead, to some remote place (like kernel.org). This could be used to help improving the documentation's build time during doc development. Also, I would love to have a good way to just produce html (and pdf) from the documents I'm actually working with. The way I do it right now is that I create a "Documentation/foo" directory, adding there just the docs I'm currently working with, placing them on a fake index.rst file. This way, I can build them really fast, without needing to rebuild everything at the same book. Perhaps something like that could be automated - or Sphinx itself could support something like: make htmldocs SPHINXFILES="foo.rst bar.rst" > - Making the docs build system less ugly and more maintainable I guess shifting the minimal Sphinx version would help to remove some bad things there. One thing that probably be solved on a different way is to have a better solution for things like: Functions for feature 1 ======================= .. kernel-doc:: include/some_header.h :doc: Feature 1 .. kernel-doc:: include/some_header.h :functions: func1 func2 Functions for feature 2 ======================= .. kernel-doc:: include/some_header.h :doc: Feature 2 .. kernel-doc:: include/some_header.h :functions: func3 func4 Perhaps we could change Kernel-doc in a way that doing just: .. kernel-doc:: include/some_header.h would be enough. > - Integrating rustdoc documentation Won't comment much about that, as never touched any of those. Life would be a lot easier on this side if rustdoc could support ReST. Regards, Mauro ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-18 8:24 ` Mauro Carvalho Chehab 2022-06-18 8:24 ` Mauro Carvalho Chehab @ 2022-06-18 11:03 ` Miguel Ojeda 2022-06-18 11:16 ` Mauro Carvalho Chehab 2022-06-23 9:18 ` Jani Nikula 2 siblings, 1 reply; 40+ messages in thread From: Miguel Ojeda @ 2022-06-18 11:03 UTC (permalink / raw) To: Mauro Carvalho Chehab; +Cc: Jonathan Corbet, ksummit, ksummit On Sat, Jun 18, 2022 at 10:24 AM Mauro Carvalho Chehab <mchehab@kernel.org> wrote: > > Life would be a lot easier on this side if rustdoc could > support ReST. Not sure what you mean by this. Do you mean writing ReST for Rust documentation? For the files inside `Documentation/rust/`, we already use ReST [*] like the rest. For the source code docs (the ones handled by rustdoc), it outputs HTML. Or do you mean using rustdoc to generate `Doc/`s or similar? [*] Though it would be nice to be able to use Markdown (but that is a different topic). Cheers, Miguel ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-18 11:03 ` Miguel Ojeda @ 2022-06-18 11:16 ` Mauro Carvalho Chehab 2022-06-18 11:16 ` Mauro Carvalho Chehab 2022-06-18 14:37 ` Miguel Ojeda 0 siblings, 2 replies; 40+ messages in thread From: Mauro Carvalho Chehab @ 2022-06-18 11:16 UTC (permalink / raw) To: Miguel Ojeda; +Cc: Jonathan Corbet, ksummit, ksummit Em Sat, 18 Jun 2022 13:03:38 +0200 Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> escreveu: > On Sat, Jun 18, 2022 at 10:24 AM Mauro Carvalho Chehab > <mchehab@kernel.org> wrote: > > > > Life would be a lot easier on this side if rustdoc could > > support ReST. > > Not sure what you mean by this. Do you mean writing ReST for Rust > documentation? For the files inside `Documentation/rust/`, we already > use ReST [*] like the rest. For the source code docs (the ones handled > by rustdoc), it outputs HTML. Or do you mean using rustdoc to generate > `Doc/`s or similar? I meant that rustdoc could generate ReST output, being included inside the Kernel documentation on a way similar to kerneldoc. This would integrate better with our building system, which can produce output docs on 3 different formats (html, epub and pdf). Regards, Mauro > > [*] Though it would be nice to be able to use Markdown (but that is a > different topic). > > Cheers, > Miguel ^ permalink raw reply [flat|nested] 40+ messages in thread
* [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-18 11:16 ` Mauro Carvalho Chehab @ 2022-06-18 11:16 ` Mauro Carvalho Chehab 2022-06-18 14:37 ` Miguel Ojeda 1 sibling, 0 replies; 40+ messages in thread From: Mauro Carvalho Chehab @ 2022-06-18 11:16 UTC (permalink / raw) Em Sat, 18 Jun 2022 13:03:38 +0200 Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> escreveu: > On Sat, Jun 18, 2022 at 10:24 AM Mauro Carvalho Chehab > <mchehab@kernel.org> wrote: > > > > Life would be a lot easier on this side if rustdoc could > > support ReST. > > Not sure what you mean by this. Do you mean writing ReST for Rust > documentation? For the files inside `Documentation/rust/`, we already > use ReST [*] like the rest. For the source code docs (the ones handled > by rustdoc), it outputs HTML. Or do you mean using rustdoc to generate > `Doc/`s or similar? I meant that rustdoc could generate ReST output, being included inside the Kernel documentation on a way similar to kerneldoc. This would integrate better with our building system, which can produce output docs on 3 different formats (html, epub and pdf). Regards, Mauro > > [*] Though it would be nice to be able to use Markdown (but that is a > different topic). > > Cheers, > Miguel ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-18 11:16 ` Mauro Carvalho Chehab 2022-06-18 11:16 ` Mauro Carvalho Chehab @ 2022-06-18 14:37 ` Miguel Ojeda 1 sibling, 0 replies; 40+ messages in thread From: Miguel Ojeda @ 2022-06-18 14:37 UTC (permalink / raw) To: Mauro Carvalho Chehab; +Cc: Jonathan Corbet, ksummit, ksummit On Sat, Jun 18, 2022 at 1:16 PM Mauro Carvalho Chehab <mchehab@kernel.org> wrote: > > I meant that rustdoc could generate ReST output, being included inside > the Kernel documentation on a way similar to kerneldoc. This would > integrate better with our building system, which can produce output docs > on 3 different formats (html, epub and pdf). That may lose some of the advantages of the rustdoc output (tailored to Rust code, navigable source code view, etc.). It may be possible to encode a lot of the information in ReST and then replicate functionality with Sphinx/plugins, possibly having a Rust domain etc., but I am not sure it would be worth it (for the HTML output at least). Cheers, Miguel ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-18 8:24 ` Mauro Carvalho Chehab 2022-06-18 8:24 ` Mauro Carvalho Chehab 2022-06-18 11:03 ` Miguel Ojeda @ 2022-06-23 9:18 ` Jani Nikula 2022-06-23 9:57 ` Mauro Carvalho Chehab 2 siblings, 1 reply; 40+ messages in thread From: Jani Nikula @ 2022-06-23 9:18 UTC (permalink / raw) To: Mauro Carvalho Chehab, Jonathan Corbet; +Cc: ksummit-discuss, ksummit On Sat, 18 Jun 2022, Mauro Carvalho Chehab <mchehab@kernel.org> wrote: > The big missing feature on Sphinx itself is with regards to per-C-type > domain. So, if we have one struct and one function both called "foo", > the cross-references will be broken. This issue is known since Sphinx > 3.1, and there are already patches fixing it since then (I remember > testing them) but, up to today, the Sphinx upstream patches meant > to fix it weren't applied yet (as far as I can tell). https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#namespacing Integrating that needs to be done carefully, though, to not make a mess of it. > One thing that probably be solved on a different way is to have > a better solution for things like: > > Functions for feature 1 > ======================= > > .. kernel-doc:: include/some_header.h > :doc: Feature 1 > > .. kernel-doc:: include/some_header.h > :functions: > func1 > func2 > > Functions for feature 2 > ======================= > > .. kernel-doc:: include/some_header.h > :doc: Feature 2 > > .. kernel-doc:: include/some_header.h > :functions: > func3 > func4 Yeah, currently that leads to parsing the header four times by kernel-doc the script. The solution would be to finally convert the script to a proper python Sphinx extension that can do caching. (This is how it works in Hawkmoth, FWIW.) > Perhaps we could change Kernel-doc in a way that doing just: > > .. kernel-doc:: include/some_header.h > > would be enough. The order in nicely flowing documentation is not necessarily the same as the order in nicely flowing source code. I expect it to be much more acceptable to tweak the rst to achieve this than to do source code rearrangement to generate nice documentation. BR, Jani. -- Jani Nikula, Intel Open Source Graphics Center ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-23 9:18 ` Jani Nikula @ 2022-06-23 9:57 ` Mauro Carvalho Chehab 2022-06-23 10:30 ` Jani Nikula 2022-06-23 13:40 ` Jonathan Corbet 0 siblings, 2 replies; 40+ messages in thread From: Mauro Carvalho Chehab @ 2022-06-23 9:57 UTC (permalink / raw) To: Jani Nikula; +Cc: Jonathan Corbet, ksummit-discuss, ksummit Em Thu, 23 Jun 2022 12:18:50 +0300 Jani Nikula <jani.nikula@intel.com> escreveu: > On Sat, 18 Jun 2022, Mauro Carvalho Chehab <mchehab@kernel.org> wrote: > > The big missing feature on Sphinx itself is with regards to per-C-type > > domain. So, if we have one struct and one function both called "foo", > > the cross-references will be broken. This issue is known since Sphinx > > 3.1, and there are already patches fixing it since then (I remember > > testing them) but, up to today, the Sphinx upstream patches meant > > to fix it weren't applied yet (as far as I can tell). > > https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#namespacing > > Integrating that needs to be done carefully, though, to not make a mess > of it. > > > One thing that probably be solved on a different way is to have > > a better solution for things like: > > > > Functions for feature 1 > > ======================= > > > > .. kernel-doc:: include/some_header.h > > :doc: Feature 1 > > > > .. kernel-doc:: include/some_header.h > > :functions: > > func1 > > func2 > > > > Functions for feature 2 > > ======================= > > > > .. kernel-doc:: include/some_header.h > > :doc: Feature 2 > > > > .. kernel-doc:: include/some_header.h > > :functions: > > func3 > > func4 > > Yeah, currently that leads to parsing the header four times by > kernel-doc the script. Yes. > The solution would be to finally convert the > script to a proper python Sphinx extension that can do caching. (This is > how it works in Hawkmoth, FWIW.) That's one solution, but see: there is already a python extension that currently calls kernel-doc everytime. It could, instead, cache the rst returned by its first run (or a parsed version of it) and use the cached results the other 3 times. Porting kernel-doc to python could be doable, but not trivial, due to several reasons: - it should keep running standalone, as otherwise debugging parsing issues on kernel-doc would be a lot harder. In particular, kernel-doc --none is really helpful to report kernel-doc tag errors; - regressions will likely be introduced on a change like that; - regular contributors to kernel-doc will need to ramp up with the newer version; - a port like that could increase the script run time, as the optimizations and regular expressions there could behave different on python. > > Perhaps we could change Kernel-doc in a way that doing just: > > > > .. kernel-doc:: include/some_header.h > > > > would be enough. > > The order in nicely flowing documentation is not necessarily the same as > the order in nicely flowing source code. I expect it to be much more > acceptable to tweak the rst to achieve this than to do source code > rearrangement to generate nice documentation. True, but independently if the script would be rewritten in python or not, one way would be to enrich the 'DOCS:' kernel-doc tag in order to mention there the symbols that belong to each part of the document, e. g. something like: /** * DOC: foo * * Some comments... * * symbols: * foo * bar */ One advantage is that all documentation will be on a single place, so hopefully it would be easier to maintain. Also, on documents using `DOC:` with such new `symbols` tag, kernel-doc could validate if all documented symbols are singly included at all `DOC:` sessions and if any symbols are missed/renamed/removed. Regards, Mauro ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-23 9:57 ` Mauro Carvalho Chehab @ 2022-06-23 10:30 ` Jani Nikula 2022-06-23 13:40 ` Jonathan Corbet 1 sibling, 0 replies; 40+ messages in thread From: Jani Nikula @ 2022-06-23 10:30 UTC (permalink / raw) To: Mauro Carvalho Chehab; +Cc: Jonathan Corbet, ksummit-discuss, ksummit On Thu, 23 Jun 2022, Mauro Carvalho Chehab <mchehab@kernel.org> wrote: > Em Thu, 23 Jun 2022 12:18:50 +0300 > Jani Nikula <jani.nikula@intel.com> escreveu: > >> On Sat, 18 Jun 2022, Mauro Carvalho Chehab <mchehab@kernel.org> wrote: >> > The big missing feature on Sphinx itself is with regards to per-C-type >> > domain. So, if we have one struct and one function both called "foo", >> > the cross-references will be broken. This issue is known since Sphinx >> > 3.1, and there are already patches fixing it since then (I remember >> > testing them) but, up to today, the Sphinx upstream patches meant >> > to fix it weren't applied yet (as far as I can tell). >> >> https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#namespacing >> >> Integrating that needs to be done carefully, though, to not make a mess >> of it. >> >> > One thing that probably be solved on a different way is to have >> > a better solution for things like: >> > >> > Functions for feature 1 >> > ======================= >> > >> > .. kernel-doc:: include/some_header.h >> > :doc: Feature 1 >> > >> > .. kernel-doc:: include/some_header.h >> > :functions: >> > func1 >> > func2 >> > >> > Functions for feature 2 >> > ======================= >> > >> > .. kernel-doc:: include/some_header.h >> > :doc: Feature 2 >> > >> > .. kernel-doc:: include/some_header.h >> > :functions: >> > func3 >> > func4 >> >> Yeah, currently that leads to parsing the header four times by >> kernel-doc the script. > > Yes. > >> The solution would be to finally convert the >> script to a proper python Sphinx extension that can do caching. (This is >> how it works in Hawkmoth, FWIW.) > > That's one solution, but see: there is already a python extension > that currently calls kernel-doc everytime. It could, instead, > cache the rst returned by its first run (or a parsed version of it) > and use the cached results the other 3 times. This would mean having kernel-doc the perl script output meta information about the constructs, complicating it even more. > Porting kernel-doc to python could be doable, but not trivial, due to several > reasons: > > - it should keep running standalone, as otherwise debugging parsing issues > on kernel-doc would be a lot harder. In particular, kernel-doc --none is > really helpful to report kernel-doc tag errors; This is trivial and sensible to do in the python extension too. > - regressions will likely be introduced on a change like that; Any change can cause regressions... but previously a diff on the html outputs generated by both versions has been an effective way of verifying the changes. > - regular contributors to kernel-doc will need to ramp up with the newer > version; You say it like it's a bad thing. :p Sometimes I regret not throwing kernel-doc the script in the curb while doing the Sphinx conversion. > - a port like that could increase the script run time, as the > optimizations and regular expressions there could behave different on > python. This is certainly possible. It's somewhat countered by the python extension being able to cache stuff. > >> > Perhaps we could change Kernel-doc in a way that doing just: >> > >> > .. kernel-doc:: include/some_header.h >> > >> > would be enough. >> >> The order in nicely flowing documentation is not necessarily the same as >> the order in nicely flowing source code. I expect it to be much more >> acceptable to tweak the rst to achieve this than to do source code >> rearrangement to generate nice documentation. > > True, but independently if the script would be rewritten in python or not, > one way would be to enrich the 'DOCS:' kernel-doc tag in order to mention > there the symbols that belong to each part of the document, e. g. something > like: > > /** > * DOC: foo > * > * Some comments... > * > * symbols: > * foo > * bar > */ > I believe preprocessing of what's supposed to be rst leads to problems. It's difficult to define proper escaping to ensure stuff gets parsed at the correct stage in the correct way. It should just be rst. Part of the reason the old docbook thing was such a big mess were the "impedance mismatches" at each separate stage of processing. In general, stick to the mechanisms provided by rst and Sphinx. Extend on them using Sphinx extensions, not by manually parsing rst. BR, Jani. > One advantage is that all documentation will be on a single place, > so hopefully it would be easier to maintain. > > Also, on documents using `DOC:` with such new `symbols` tag, kernel-doc > could validate if all documented symbols are singly included at all `DOC:` > sessions and if any symbols are missed/renamed/removed. > > Regards, > Mauro -- Jani Nikula, Intel Open Source Graphics Center ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-23 9:57 ` Mauro Carvalho Chehab 2022-06-23 10:30 ` Jani Nikula @ 2022-06-23 13:40 ` Jonathan Corbet 2022-06-24 7:33 ` Mauro Carvalho Chehab 1 sibling, 1 reply; 40+ messages in thread From: Jonathan Corbet @ 2022-06-23 13:40 UTC (permalink / raw) To: Mauro Carvalho Chehab, Jani Nikula; +Cc: ksummit-discuss, ksummit Mauro Carvalho Chehab <mchehab@kernel.org> writes: >> The solution would be to finally convert the >> script to a proper python Sphinx extension that can do caching. (This is >> how it works in Hawkmoth, FWIW.) I've been pondering on this for a bit, and would like to do it, but I don't know when I might find the time for it. > That's one solution, but see: there is already a python extension > that currently calls kernel-doc everytime. It could, instead, > cache the rst returned by its first run (or a parsed version of it) > and use the cached results the other 3 times. > > Porting kernel-doc to python could be doable, but not trivial, due to several > reasons: > > - it should keep running standalone, as otherwise debugging parsing issues > on kernel-doc would be a lot harder. In particular, kernel-doc --none is > really helpful to report kernel-doc tag errors; Yes, of course. As Jani noted, that's just how you would do it, not a problem. > - regressions will likely be introduced on a change like that; The nice thing is ... we already have a really nice regression test in the form of the current docs build and diff. > - regular contributors to kernel-doc will need to ramp up with the newer > version; We have those? That script is a nightmare and nobody goes near it if they can possibly avoid it. I would expect to have more contributors with a decent Python version that doesn't include 25 years of regex accretion. > - a port like that could increase the script run time, as the > optimizations and regular expressions there could behave different on > python. It could also decrease it by improving caching opportunities, getting rid of a lot of fork()/exec() pairs and Perl interpreter startups, etc. I've actually, in a spare moment or two, been doing some profiling of the kernel docs build and trying to track down the sources of the slowness. I am thinking that nearly 700 *million* calls to the iterator for the C-domain symbol list might have something to do with it... > True, but independently if the script would be rewritten in python or not, > one way would be to enrich the 'DOCS:' kernel-doc tag in order to mention > there the symbols that belong to each part of the document, e. g. something > like: > > /** > * DOC: foo > * > * Some comments... > * > * symbols: > * foo > * bar > */ > > One advantage is that all documentation will be on a single place, > so hopefully it would be easier to maintain. I'm not quite sure I get this...you want to put the TOC tree in the source comments? This looks like the kind of thing that nobody ever remembers to update, but maybe I'm missing something. Thanks, jon ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-23 13:40 ` Jonathan Corbet @ 2022-06-24 7:33 ` Mauro Carvalho Chehab 2022-06-24 16:37 ` Markus Heiser 2022-06-24 22:57 ` Jonathan Corbet 0 siblings, 2 replies; 40+ messages in thread From: Mauro Carvalho Chehab @ 2022-06-24 7:33 UTC (permalink / raw) To: Jonathan Corbet; +Cc: Jani Nikula, ksummit-discuss, ksummit, Markus Heiser Em Thu, 23 Jun 2022 07:40:45 -0600 Jonathan Corbet <corbet@lwn.net> escreveu: > Mauro Carvalho Chehab <mchehab@kernel.org> writes: > > >> The solution would be to finally convert the > >> script to a proper python Sphinx extension that can do caching. (This is > >> how it works in Hawkmoth, FWIW.) > > I've been pondering on this for a bit, and would like to do it, but I > don't know when I might find the time for it. There is already a version of kernel-doc written in Python, made by Markus Heiser: https://github.com/return42/linuxdoc/blob/master/linuxdoc/kernel_doc.py It could be a starting point. > > That's one solution, but see: there is already a python extension > > that currently calls kernel-doc everytime. It could, instead, > > cache the rst returned by its first run (or a parsed version of it) > > and use the cached results the other 3 times. > > > > Porting kernel-doc to python could be doable, but not trivial, due to several > > reasons: > > > > - it should keep running standalone, as otherwise debugging parsing issues > > on kernel-doc would be a lot harder. In particular, kernel-doc --none is > > really helpful to report kernel-doc tag errors; > > Yes, of course. As Jani noted, that's just how you would do it, not a > problem. > > > - regressions will likely be introduced on a change like that; > > The nice thing is ... we already have a really nice regression test in > the form of the current docs build and diff. True. We can test the results with both versions and even check how performance is affected. > > - regular contributors to kernel-doc will need to ramp up with the newer > > version; > > We have those? That script is a nightmare and nobody goes near it if > they can possibly avoid it. I would expect to have more contributors > with a decent Python version that doesn't include 25 years of regex > accretion. Well, the kernel-doc version from Markus in python is ~23% bigger than our current kernel-doc. Granted, it could be due to comments and blank lines, but basically, the same regexes that are in perl would also be need to be replicated in python, as parsing a C code with regexes is not a trivial task. So, in terms of complexity, I doubt much would change by porting it to python. Now, one of the things that the kernel-doc does is that it has the parse code and the output logic, which actually has an an abstraction to let it to produce results on different formats (currently, none, man and rst - we dropped some other formats from it). This abstraction increases its complexity. This is something that doesn't need to be replicated on any ports. > > - a port like that could increase the script run time, as the > > optimizations and regular expressions there could behave different on > > python. > > It could also decrease it by improving caching opportunities, getting > rid of a lot of fork()/exec() pairs and Perl interpreter startups, etc. > > I've actually, in a spare moment or two, been doing some profiling of > the kernel docs build and trying to track down the sources of the > slowness. I am thinking that nearly 700 *million* calls to the iterator > for the C-domain symbol list might have something to do with it... Wow, that's a lot! > > True, but independently if the script would be rewritten in python or not, > > one way would be to enrich the 'DOCS:' kernel-doc tag in order to mention > > there the symbols that belong to each part of the document, e. g. something > > like: > > > > /** > > * DOC: foo > > * > > * Some comments... > > * > > * symbols: > > * foo > > * bar > > */ > > > > One advantage is that all documentation will be on a single place, > > so hopefully it would be easier to maintain. > > I'm not quite sure I get this...you want to put the TOC tree in the > source comments? This looks like the kind of thing that nobody ever > remembers to update, but maybe I'm missing something. No, it won't generate a TOC tree. It would instead reorder how kernel-doc would output the symbols. That's no different than what we have already at the .kernel-doc directives, e. g. it would be a replacement for: .. kernel-doc:: include/some_header.h :doc: foo .. kernel-doc:: include/some_header.h :functions: foo bar The problem we currently have is that the above pattern means that one or more .rst files would contain a list of symbols that are actually documented at some_header.h. People updating such file will very likely forget to update .rst files, leading to missing documentation. Also, the same symbol could be included on different .rst files. I remember I had to fix myself duplicated symbols inclusion while trying to reduce the docs build time on a few *.h files that were included hundreds of times. I bet if someone checks again, duplicated inclusions of the same symbol and missed symbols that are documented on their sources, but aren't included at any .rst file. Currently, detecting it is very hard, because the symbol lists are on different files and the same header are sometimes included on different rst files. By placing the symbol list inside the source file, it makes very simple for kernel-doc to check if all documented symbols are inside the "DOC:" markups, producing errors otherwise. Regards, Mauro ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-24 7:33 ` Mauro Carvalho Chehab @ 2022-06-24 16:37 ` Markus Heiser 2022-06-27 15:27 ` Jonathan Corbet 2022-06-24 22:57 ` Jonathan Corbet 1 sibling, 1 reply; 40+ messages in thread From: Markus Heiser @ 2022-06-24 16:37 UTC (permalink / raw) To: Mauro Carvalho Chehab, Jonathan Corbet Cc: Jani Nikula, ksummit-discuss, ksummit Hi Jon, Mauro, with great interest I have read this thread ! If I can help, I'm happy to participate again. TBH: The kernel-doc implementation from mine could only be a piece of the puzzle to improve the process (not without raising other questions). But similar to Jani, I see the priority problems on a different level level. We have too many demands that reach into the smallest niches. I believe that some parts must be generalized and some request must be abandoned. In my opinion, many problems are homemade, to take just one example: IMO It is unnecessary that the build-chain must run on all platforms and with all distributions. Who observes the Sphinx-doc & docutils development since (>15) years is aware that with various (old) Sphinx-doc & docutils versions no stable results can be produced, not without complicating the build-chain. And this is exactly the situation we are facing today. The build chain of documentation has nothing to do with kernel development (at least in my opinion) and should be decoupled from it: maintaining one defined build environment is enough work ... this becomes especially clear if you (as Jani recommends) rely more on sphinx-modules and widely used tools. Another point on which I now have a clear view are the regular expressions: no one likes them (me too), but I can't think of a general solution (parser) given the number of requirements for parsing source code we have. I would like to support the kernel community again, but at the moment I have difficulties to follow the many exceptions. I would be happy if you keep me in CC .. may be I find my place again back in the kernel-doc development :-) Thanks and with best regards -- Markus -- Am 24.06.22 um 09:33 schrieb Mauro Carvalho Chehab: > Em Thu, 23 Jun 2022 07:40:45 -0600 > Jonathan Corbet <corbet@lwn.net> escreveu: > >> Mauro Carvalho Chehab <mchehab@kernel.org> writes: >> >>>> The solution would be to finally convert the >>>> script to a proper python Sphinx extension that can do caching. (This is >>>> how it works in Hawkmoth, FWIW.) >> >> I've been pondering on this for a bit, and would like to do it, but I >> don't know when I might find the time for it. > > There is already a version of kernel-doc written in Python, made by > Markus Heiser: > > https://github.com/return42/linuxdoc/blob/master/linuxdoc/kernel_doc.py > > It could be a starting point. > >>> That's one solution, but see: there is already a python extension >>> that currently calls kernel-doc everytime. It could, instead, >>> cache the rst returned by its first run (or a parsed version of it) >>> and use the cached results the other 3 times. >>> >>> Porting kernel-doc to python could be doable, but not trivial, due to several >>> reasons: >>> >>> - it should keep running standalone, as otherwise debugging parsing issues >>> on kernel-doc would be a lot harder. In particular, kernel-doc --none is >>> really helpful to report kernel-doc tag errors; >> >> Yes, of course. As Jani noted, that's just how you would do it, not a >> problem. >> >>> - regressions will likely be introduced on a change like that; >> >> The nice thing is ... we already have a really nice regression test in >> the form of the current docs build and diff. > > True. We can test the results with both versions and even check how > performance is affected. > >>> - regular contributors to kernel-doc will need to ramp up with the newer >>> version; >> >> We have those? That script is a nightmare and nobody goes near it if >> they can possibly avoid it. I would expect to have more contributors >> with a decent Python version that doesn't include 25 years of regex >> accretion. > > Well, the kernel-doc version from Markus in python is ~23% bigger > than our current kernel-doc. Granted, it could be due to comments and > blank lines, but basically, the same regexes that are in perl would > also be need to be replicated in python, as parsing a C code with > regexes is not a trivial task. > > So, in terms of complexity, I doubt much would change by porting it > to python. > > Now, one of the things that the kernel-doc does is that it has the > parse code and the output logic, which actually has an an abstraction > to let it to produce results on different formats (currently, none, man > and rst - we dropped some other formats from it). This abstraction > increases its complexity. This is something that doesn't need to be > replicated on any ports. > >>> - a port like that could increase the script run time, as the >>> optimizations and regular expressions there could behave different on >>> python. >> >> It could also decrease it by improving caching opportunities, getting >> rid of a lot of fork()/exec() pairs and Perl interpreter startups, etc. >> >> I've actually, in a spare moment or two, been doing some profiling of >> the kernel docs build and trying to track down the sources of the >> slowness. I am thinking that nearly 700 *million* calls to the iterator >> for the C-domain symbol list might have something to do with it... > > Wow, that's a lot! > >>> True, but independently if the script would be rewritten in python or not, >>> one way would be to enrich the 'DOCS:' kernel-doc tag in order to mention >>> there the symbols that belong to each part of the document, e. g. something >>> like: >>> >>> /** >>> * DOC: foo >>> * >>> * Some comments... >>> * >>> * symbols: >>> * foo >>> * bar >>> */ >>> >>> One advantage is that all documentation will be on a single place, >>> so hopefully it would be easier to maintain. >> >> I'm not quite sure I get this...you want to put the TOC tree in the >> source comments? This looks like the kind of thing that nobody ever >> remembers to update, but maybe I'm missing something. > > No, it won't generate a TOC tree. It would instead reorder how > kernel-doc would output the symbols. > > That's no different than what we have already at the .kernel-doc > directives, e. g. it would be a replacement for: > > > .. kernel-doc:: include/some_header.h > :doc: foo > > .. kernel-doc:: include/some_header.h > :functions: > foo > bar > > The problem we currently have is that the above pattern means that > one or more .rst files would contain a list of symbols that are > actually documented at some_header.h. People updating such file > will very likely forget to update .rst files, leading to missing > documentation. Also, the same symbol could be included on different > .rst files. > > I remember I had to fix myself duplicated symbols inclusion while > trying to reduce the docs build time on a few *.h files that > were included hundreds of times. I bet if someone checks again, > duplicated inclusions of the same symbol and missed symbols that > are documented on their sources, but aren't included at any .rst file. > > Currently, detecting it is very hard, because the symbol lists > are on different files and the same header are sometimes included > on different rst files. By placing the symbol list inside the source > file, it makes very simple for kernel-doc to check if all documented > symbols are inside the "DOC:" markups, producing errors otherwise. > > Regards, > Mauro ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-24 16:37 ` Markus Heiser @ 2022-06-27 15:27 ` Jonathan Corbet 2022-06-27 15:31 ` Christoph Hellwig 2022-06-28 7:43 ` Mauro Carvalho Chehab 0 siblings, 2 replies; 40+ messages in thread From: Jonathan Corbet @ 2022-06-27 15:27 UTC (permalink / raw) To: Markus Heiser, Mauro Carvalho Chehab Cc: Jani Nikula, ksummit-discuss, ksummit Markus Heiser <markus.heiser@darmarit.de> writes: > IMO It is unnecessary that the build-chain must run on all > platforms and with all distributions. > > Who observes the Sphinx-doc & docutils development since (>15) > years is aware that with various (old) Sphinx-doc & docutils > versions no stable results can be produced, not without > complicating the build-chain. And this is exactly the situation > we are facing today. > > The build chain of documentation has nothing to do with kernel > development (at least in my opinion) and should be decoupled from it: > maintaining one defined build environment is enough work ... this > becomes especially clear if you (as Jani recommends) rely more on > sphinx-modules and widely used tools. The counterargument to this is that we want as many developers as possible to be able to build the docs and contribute to them. We can't complain that developers have broken the docs build if we don't do what we can to help them do the build themselves. One of our longstanding contributors is on Sphinx 1.8.5: https://lwn.net/ml/linux-doc/4c403239-3c71-4ab9-2168-f7e9d77008b2%40infradead.org/ I would like to narrow our range of supported versions, but I really don't want to cut people out. The real question, perhaps, is whether requiring people to set up a virtualenv to run a supported version would be too much. It's something I worry about. > Another point on which I now have a clear view are the regular > expressions: no one likes them (me too), but I can't think of a > general solution (parser) given the number of requirements for > parsing source code we have. I fear you're right. There are nice C parsers for Python, but they don't work at the level we need; we need the comments, unexpanded macros, etc. That said, the regex expression in kernel-doc is out of hand, with no documentation, lots of duplication, etc. > I would like to support the kernel community again, but at the > moment I have difficulties to follow the many exceptions. I would > be happy if you keep me in CC .. may be I find my place again > back in the kernel-doc development :-) We would be glad to have you back. Remember that the reason I was hesitant to take your kernel-doc rewrite at the time was that we were already thrashing too many things and I wanted to hold at least one piece stable. Hopefully we're past that phase now...:) Thanks, jon ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-27 15:27 ` Jonathan Corbet @ 2022-06-27 15:31 ` Christoph Hellwig 2022-06-28 7:43 ` Mauro Carvalho Chehab 1 sibling, 0 replies; 40+ messages in thread From: Christoph Hellwig @ 2022-06-27 15:31 UTC (permalink / raw) To: Jonathan Corbet Cc: Markus Heiser, Mauro Carvalho Chehab, Jani Nikula, ksummit-discuss, ksummit On Mon, Jun 27, 2022 at 09:27:54AM -0600, Jonathan Corbet wrote: > The real question, perhaps, is whether requiring people to set up a > virtualenv to run a supported version would be too much. It's something > I worry about. I know for on that I will not setup pything venvs as they are massive pain. Anyting that isn't available in a somewhat recent distro using the package manage is a complete no-go. And software that changes so frequently that this question even arises seem rather suspect to me in general for something like documentation that should have a wide range of contributors. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-27 15:27 ` Jonathan Corbet 2022-06-27 15:31 ` Christoph Hellwig @ 2022-06-28 7:43 ` Mauro Carvalho Chehab 2022-06-28 7:57 ` Geert Uytterhoeven 1 sibling, 1 reply; 40+ messages in thread From: Mauro Carvalho Chehab @ 2022-06-28 7:43 UTC (permalink / raw) To: Jonathan Corbet; +Cc: Markus Heiser, Jani Nikula, ksummit-discuss, ksummit Em Mon, 27 Jun 2022 09:27:54 -0600 Jonathan Corbet <corbet@lwn.net> escreveu: > Markus Heiser <markus.heiser@darmarit.de> writes: > > > IMO It is unnecessary that the build-chain must run on all > > platforms and with all distributions. > > > > Who observes the Sphinx-doc & docutils development since (>15) > > years is aware that with various (old) Sphinx-doc & docutils > > versions no stable results can be produced, not without > > complicating the build-chain. And this is exactly the situation > > we are facing today. > > > > The build chain of documentation has nothing to do with kernel > > development (at least in my opinion) and should be decoupled from it: > > maintaining one defined build environment is enough work ... this > > becomes especially clear if you (as Jani recommends) rely more on > > sphinx-modules and widely used tools. > > The counterargument to this is that we want as many developers as > possible to be able to build the docs and contribute to them. We can't > complain that developers have broken the docs build if we don't do what > we can to help them do the build themselves. Agreed. > One of our longstanding contributors is on Sphinx 1.8.5: > > https://lwn.net/ml/linux-doc/4c403239-3c71-4ab9-2168-f7e9d77008b2%40infradead.org/ If he migrates to Leap 15.4, he'll either get 2.3.1 (default package) or 4.2.0 (alternative package): https://download.opensuse.org/distribution/leap/15.4/repo/oss/noarch/python3-Sphinx-2.3.1-150400.1.7.noarch.rpm https://download.opensuse.org/distribution/leap/15.4/repo/oss/noarch/python3-Sphinx_4_2_0-4.2.0-150400.11.6.noarch.rpm > I would like to narrow our range of supported versions, but I really > don't want to cut people out. > > The real question, perhaps, is whether requiring people to set up a > virtualenv to run a supported version would be too much. It's something > I worry about. Installing distro-specific packages likely covers a broader group of developers than requiring virtualenv/venv. However, if we decide to move on, removing the version-dependent hacks from the build system, it would mean that the minimal version would be 3.1.0. On such case, there won't be Sphinx packages on some LTS distros. Developers using such distros will be forced to use virtualenv/venv. In any case, the script which checks if sphinx-build runs can recommend either way. Yet, as the current default is venv, tests are needed to check if it recommending python3-sphinx packages still work on all supported distros. Such script could also be changed to detect if the distro has a native package >= 3.1.0, recommending it by default, falling back to virtualenv/venv otherwise. Regards, Mauro ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-28 7:43 ` Mauro Carvalho Chehab @ 2022-06-28 7:57 ` Geert Uytterhoeven 2022-06-28 11:01 ` Mauro Carvalho Chehab 0 siblings, 1 reply; 40+ messages in thread From: Geert Uytterhoeven @ 2022-06-28 7:57 UTC (permalink / raw) To: Mauro Carvalho Chehab Cc: Jonathan Corbet, Markus Heiser, Jani Nikula, ksummit, ksummit Hi Mauro, On Tue, Jun 28, 2022 at 9:43 AM Mauro Carvalho Chehab <mchehab@kernel.org> wrote: > Em Mon, 27 Jun 2022 09:27:54 -0600 > Jonathan Corbet <corbet@lwn.net> escreveu: > > Markus Heiser <markus.heiser@darmarit.de> writes: > > > IMO It is unnecessary that the build-chain must run on all > > > platforms and with all distributions. > > > > > > Who observes the Sphinx-doc & docutils development since (>15) > > > years is aware that with various (old) Sphinx-doc & docutils > > > versions no stable results can be produced, not without > > > complicating the build-chain. And this is exactly the situation > > > we are facing today. > > > > > > The build chain of documentation has nothing to do with kernel > > > development (at least in my opinion) and should be decoupled from it: > > > maintaining one defined build environment is enough work ... this > > > becomes especially clear if you (as Jani recommends) rely more on > > > sphinx-modules and widely used tools. > > > > The counterargument to this is that we want as many developers as > > possible to be able to build the docs and contribute to them. We can't > > complain that developers have broken the docs build if we don't do what > > we can to help them do the build themselves. > > Agreed. > > > One of our longstanding contributors is on Sphinx 1.8.5: > > > > https://lwn.net/ml/linux-doc/4c403239-3c71-4ab9-2168-f7e9d77008b2%40infradead.org/ Ubuntu 20.04 LTS also has 1.8.5. Looks like I'll get 4.3.2 after upgrading to 22.04 LTS (which is only about two months old). Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-28 7:57 ` Geert Uytterhoeven @ 2022-06-28 11:01 ` Mauro Carvalho Chehab 2022-07-02 12:43 ` Stephen Rothwell 0 siblings, 1 reply; 40+ messages in thread From: Mauro Carvalho Chehab @ 2022-06-28 11:01 UTC (permalink / raw) To: Geert Uytterhoeven Cc: Jonathan Corbet, Markus Heiser, Jani Nikula, ksummit, ksummit Em Tue, 28 Jun 2022 09:57:29 +0200 Geert Uytterhoeven <geert@linux-m68k.org> escreveu: > Hi Mauro, > > On Tue, Jun 28, 2022 at 9:43 AM Mauro Carvalho Chehab > <mchehab@kernel.org> wrote: > > Em Mon, 27 Jun 2022 09:27:54 -0600 > > Jonathan Corbet <corbet@lwn.net> escreveu: > > > Markus Heiser <markus.heiser@darmarit.de> writes: > > > > IMO It is unnecessary that the build-chain must run on all > > > > platforms and with all distributions. > > > > > > > > Who observes the Sphinx-doc & docutils development since (>15) > > > > years is aware that with various (old) Sphinx-doc & docutils > > > > versions no stable results can be produced, not without > > > > complicating the build-chain. And this is exactly the situation > > > > we are facing today. > > > > > > > > The build chain of documentation has nothing to do with kernel > > > > development (at least in my opinion) and should be decoupled from it: > > > > maintaining one defined build environment is enough work ... this > > > > becomes especially clear if you (as Jani recommends) rely more on > > > > sphinx-modules and widely used tools. > > > > > > The counterargument to this is that we want as many developers as > > > possible to be able to build the docs and contribute to them. We can't > > > complain that developers have broken the docs build if we don't do what > > > we can to help them do the build themselves. > > > > Agreed. > > > > > One of our longstanding contributors is on Sphinx 1.8.5: > > > > > > https://lwn.net/ml/linux-doc/4c403239-3c71-4ab9-2168-f7e9d77008b2%40infradead.org/ > > Ubuntu 20.04 LTS also has 1.8.5. > Looks like I'll get 4.3.2 after upgrading to 22.04 LTS (which is only > about two months old). Yeah, LTS distros are the main reason why the default recommendation is to install Sphinx via virtualenv/venv. With RHEL9, SUSE 15 SP4 (plus CenOS and openSUSE Leap versions of them), plus Ubuntu 22.04, maybe we can change the default to recommend installing it via distro-provided packages. Yet, Debian bulseye has 2.5.0. Only bookworm, scheduled for mid 2023, will come with Sphinx > 2. Regards, Mauro ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-28 11:01 ` Mauro Carvalho Chehab @ 2022-07-02 12:43 ` Stephen Rothwell 0 siblings, 0 replies; 40+ messages in thread From: Stephen Rothwell @ 2022-07-02 12:43 UTC (permalink / raw) To: Mauro Carvalho Chehab Cc: Geert Uytterhoeven, Jonathan Corbet, Markus Heiser, Jani Nikula, ksummit, ksummit [-- Attachment #1: Type: text/plain, Size: 530 bytes --] Hi Mauro, On Tue, 28 Jun 2022 12:01:42 +0100 Mauro Carvalho Chehab <mchehab@kernel.org> wrote: > > With RHEL9, SUSE 15 SP4 (plus CenOS and openSUSE Leap versions of them), > plus Ubuntu 22.04, maybe we can change the default to recommend installing > it via distro-provided packages. Yet, Debian bulseye has 2.5.0. Only > bookworm, scheduled for mid 2023, will come with Sphinx > 2. Debian current Stable (Bullseye) has python3-sphinx version 3.4.3 (python-sphinx is version 1.8.4). -- Cheers, Stephen Rothwell [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-24 7:33 ` Mauro Carvalho Chehab 2022-06-24 16:37 ` Markus Heiser @ 2022-06-24 22:57 ` Jonathan Corbet 2022-06-25 9:10 ` Mauro Carvalho Chehab 1 sibling, 1 reply; 40+ messages in thread From: Jonathan Corbet @ 2022-06-24 22:57 UTC (permalink / raw) To: Mauro Carvalho Chehab Cc: Jani Nikula, ksummit-discuss, ksummit, Markus Heiser Mauro Carvalho Chehab <mchehab@kernel.org> writes: > Em Thu, 23 Jun 2022 07:40:45 -0600 > Jonathan Corbet <corbet@lwn.net> escreveu: >> I've actually, in a spare moment or two, been doing some profiling of >> the kernel docs build and trying to track down the sources of the >> slowness. I am thinking that nearly 700 *million* calls to the iterator >> for the C-domain symbol list might have something to do with it... > > Wow, that's a lot! Just for the curious ... The Sphinx C domain code makes this elaborate tree data structure out of all the identifier names it has picked up - including the names of function parameters when they are provided in prototypes for some reason. This is the structure that is consulted whenever we want to resolve a cross-reference, which is fairly often with automarkup enabled. How does it work? The algorithm is, as far as I can tell: - Serialize the whole tree in a sort of breadth-first traversal - Make a linear pass through the *entire* list, comparing the identifier name with each entry and accumulating a list of all the matches found. - Return the first match and throw away the rest. The result is O(n^2) behavior and, in the kernel docs build, n gets to be fairly large. I went in with a crowbar and sledgehammer and replaced some of the list searches with a dict lookup, resulting in about a 20% speedup in the full htmldocs build with Sphinx 5.0.2. A couple of automarkup optimizations result in about a 27% speedup overall. And I didn't break any cross-references. I think it's possible to do better, but this is a start. I'll post the automarkup changes as soon as I can, but I need to verify them across the whole range of Sphinx versions. For the Sphinx stuff, I'll need to turn my hatchetwork into something presentable and figure out how to contribute it upstream, but it seems worth the effort. Thanks, jon ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-24 22:57 ` Jonathan Corbet @ 2022-06-25 9:10 ` Mauro Carvalho Chehab 2022-06-25 14:00 ` Jonathan Corbet 0 siblings, 1 reply; 40+ messages in thread From: Mauro Carvalho Chehab @ 2022-06-25 9:10 UTC (permalink / raw) To: Jonathan Corbet; +Cc: Jani Nikula, ksummit-discuss, ksummit, Markus Heiser Em Fri, 24 Jun 2022 16:57:56 -0600 Jonathan Corbet <corbet@lwn.net> escreveu: > Mauro Carvalho Chehab <mchehab@kernel.org> writes: > > > Em Thu, 23 Jun 2022 07:40:45 -0600 > > Jonathan Corbet <corbet@lwn.net> escreveu: > >> I've actually, in a spare moment or two, been doing some profiling of > >> the kernel docs build and trying to track down the sources of the > >> slowness. I am thinking that nearly 700 *million* calls to the iterator > >> for the C-domain symbol list might have something to do with it... > > > > Wow, that's a lot! > > Just for the curious ... > > The Sphinx C domain code makes this elaborate tree data structure out of > all the identifier names it has picked up - including the names of > function parameters when they are provided in prototypes for some > reason. This is the structure that is consulted whenever we want to > resolve a cross-reference, which is fairly often with automarkup > enabled. > > How does it work? The algorithm is, as far as I can tell: > > - Serialize the whole tree in a sort of breadth-first traversal > > - Make a linear pass through the *entire* list, comparing the > identifier name with each entry and accumulating a list of all the > matches found. > > - Return the first match and throw away the rest. > > The result is O(n^2) behavior and, in the kernel docs build, n gets to > be fairly large. > > I went in with a crowbar and sledgehammer and replaced some of the list > searches with a dict lookup, resulting in about a 20% speedup in the > full htmldocs build with Sphinx 5.0.2. Great to have a 20% speedup here! Yet, I would expect an even better performance improvement by replacing 700 million calls from linear search to dict, as it would change from O(n^2) to O(1) at the average case [1], but the speedup gain actually depends on the actual number of symbols we have defined. [1] According with: https://wiki.python.org/moin/TimeComplexity#dict Worse case is O(n) - when collisions are common which is unlikely. > A couple of automarkup > optimizations result in about a 27% speedup overall. And I didn't break > any cross-references. Great! > I think it's possible to do better, but this is a start. I'll post the > automarkup changes as soon as I can, but I need to verify them across > the whole range of Sphinx versions. > For the Sphinx stuff, I'll need to > turn my hatchetwork into something presentable and figure out how to > contribute it upstream, but it seems worth the effort. Indeed. Regards, Mauro ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-25 9:10 ` Mauro Carvalho Chehab @ 2022-06-25 14:00 ` Jonathan Corbet 2022-06-25 18:11 ` Linus Torvalds 0 siblings, 1 reply; 40+ messages in thread From: Jonathan Corbet @ 2022-06-25 14:00 UTC (permalink / raw) To: Mauro Carvalho Chehab Cc: Jani Nikula, ksummit-discuss, ksummit, Markus Heiser Mauro Carvalho Chehab <mchehab@kernel.org> writes: > Em Fri, 24 Jun 2022 16:57:56 -0600 > Jonathan Corbet <corbet@lwn.net> escreveu: >> I went in with a crowbar and sledgehammer and replaced some of the list >> searches with a dict lookup, resulting in about a 20% speedup in the >> full htmldocs build with Sphinx 5.0.2. > > Great to have a 20% speedup here! Yet, I would expect an even better > performance improvement by replacing 700 million calls from linear search > to dict, as it would change from O(n^2) to O(1) at the average case [1], > but the speedup gain actually depends on the actual number of symbols we > have defined. I said "some" - this was a proof-of-concept hack. The complexity of their symbol lookup code is ... daunting ... and not something that gets fixed in one place. More research is required... jon ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-25 14:00 ` Jonathan Corbet @ 2022-06-25 18:11 ` Linus Torvalds 2022-06-26 7:55 ` Mauro Carvalho Chehab ` (2 more replies) 0 siblings, 3 replies; 40+ messages in thread From: Linus Torvalds @ 2022-06-25 18:11 UTC (permalink / raw) To: Jonathan Corbet Cc: Mauro Carvalho Chehab, Jani Nikula, ksummit, ksummit, Markus Heiser On Sat, Jun 25, 2022 at 7:00 AM Jonathan Corbet <corbet@lwn.net> wrote: > > I said "some" - this was a proof-of-concept hack. The complexity of > their symbol lookup code is ... daunting ... and not something that gets > fixed in one place. More research is required... Note that at least for me, the issue with building docs isn't hugely performance related. Yes, yes, it would be good if it was faster. But.. The primary problems I see with building the docs (and thus checking them, the same way I do an allmodconfig build test) is (a) it's insanely noisy, which makes and "check that it's ok" ENTIRELY USELESS. Seriously. There's a very solid reason why I require the standard kernel build to be entirely warning-free. Warnings are *BAD*. Even a single false-positive warning invalidates all other warnings. And the doc build isn't about some "single warning" thing. (b) it requires some unusual build tools Now, (b) may some historical oddity, but at least if you have Fedora installed and it still has sphinx version 2.2.11 or something like that. And when you try to build docs, the makefile gives you some random pip install thing that is entirely bogus. The proper fix is to actually remove that old sphinx environment entirely, and install "python3-sphinx" instead, but that's not at all what the "to upgrade Sphinx" docs in the kernel say when you try to make the docs. Anyway, (b) is one of those "the docs about the docs are wrong" things and ironic - but easily fixable if you know about it. But (a) then makes it all entirely pointless as far as I'm concerned. The doc build could take five seconds, and I *still* wouldn't bother, because building docs doesn't actually do anything useful for me. It's just noise. Linus ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-25 18:11 ` Linus Torvalds @ 2022-06-26 7:55 ` Mauro Carvalho Chehab 2022-06-26 9:26 ` Mauro Carvalho Chehab 2022-06-26 9:53 ` Mauro Carvalho Chehab 2022-06-27 15:34 ` Jonathan Corbet 2022-07-02 10:52 ` Mauro Carvalho Chehab 2 siblings, 2 replies; 40+ messages in thread From: Mauro Carvalho Chehab @ 2022-06-26 7:55 UTC (permalink / raw) To: Linus Torvalds Cc: Jonathan Corbet, Jani Nikula, ksummit, ksummit, Markus Heiser Em Sat, 25 Jun 2022 11:11:40 -0700 Linus Torvalds <torvalds@linux-foundation.org> escreveu: > On Sat, Jun 25, 2022 at 7:00 AM Jonathan Corbet <corbet@lwn.net> wrote: > > > > I said "some" - this was a proof-of-concept hack. The complexity of > > their symbol lookup code is ... daunting ... and not something that gets > > fixed in one place. More research is required... > > Note that at least for me, the issue with building docs isn't hugely > performance related. > > Yes, yes, it would be good if it was faster. But.. > > The primary problems I see with building the docs (and thus checking > them, the same way I do an allmodconfig build test) is > > (a) it's insanely noisy, which makes and "check that it's ok" ENTIRELY USELESS. When the environment is set for building docs, almost all output there are actually due to build errors/warnings that got introduced by patches touching documentation that were never build-tested. There are 4 components that report errors during "make htmldocs". 1. Documentation cross-reference check. $ scripts/documentation-file-ref-check --warn Warning: Documentation/admin-guide/kernel-parameters.txt references a file that doesn't exist: Documentation/virt/kvm/amd-memory-encryption.rst Warning: Documentation/dev-tools/kunit/run_wrapper.rst references a file that doesn't exist: Documentation/dev-tools/kunit/non_uml.rst ... Those warnings are result of build-untested patches moving/renaming/removing documentation without updating all documentation. On almost all Kernel versions, I usually send a couple of patch series addressing them, but incomplete patches that forget about updating references keep being merged. Without htmldocs build tests by patch authors, this is an endless job. 2. ABI documentation check: $ ./scripts/get_abi.pl validate Warning: /sys/bus/iio/devices/iio:deviceX/fault_ovuv is defined 2 times: Documentation/ABI/testing/sysfs-bus-iio-temperature-max31865:0 Documentation/ABI/testing/sysfs-bus-iio-temperature-max31856:14 Warning: /sys/bus/iio/devices/iio:deviceX/in_filter_notch_center_frequency is defined 2 times: Documentation/ABI/testing/sysfs-bus-iio:1948 Documentation/ABI/testing/sysfs-bus-iio-temperature-max31865:12 Warning: /sys/bus/iio/devices/triggerX/sampling_frequency is defined 2 times: Documentation/ABI/testing/sysfs-bus-iio:96 Documentation/ABI/testing/sysfs-bus-iio-timer-stm32:92 Warning: /sys/devices/system/cpu/cpuX/topology/core_id is defined 2 times: Documentation/ABI/stable/sysfs-devices-system-cpu:38 Documentation/ABI/testing/sysfs-devices-system-cpu:69 Those warnings indicate problems with userspace API definitions, as the same symbol has different meanings. Talking about ABI, the output is even worse if the script is used to check if the ABIs used on a running system are all documented or not. On my laptop (Kernel 5.18.5-200.fc36.x86_64), I'm getting, for instance, 2815 undocumented or badly-specified sysfs ABI symbols: $ sudo ./scripts/get_abi.pl undefined >errors Reading /sys directory contents...done. Converting ABI What fields into regexes...done. 4:59: processing sysfs files... done $ wc -l errors 2815 3. kernel-doc warnings. Those indicate kernel-doc markups with troubles. The output is the same as running: $ ./scripts/kernel-doc --none ./include/linux/fscache.h ./include/linux/fscache.h:269: warning: Function parameter or member 'cookie' not described in 'fscache_use_cookie' ./include/linux/fscache.h:269: warning: Excess function parameter 'object' description in 'fscache_use_cookie' ./include/linux/fscache.h:286: warning: Function parameter or member 'cookie' not described in 'fscache_unuse_cookie' ./include/linux/fscache.h:286: warning: Excess function parameter 'object' description in 'fscache_unuse_cookie' In this particular case, kernel-doc markup describes a non-existent "object" symbol, while the function prototype has "cookie". Probably result of some rename at the function prototype that was not validated via html build. 4. Sphinx warnings. Most of them are one line warnings: Documentation/arm/samsung-s3c24xx/cpufreq.rst:2: WARNING: Explicit markup ends without a blank line; unexpected unindent. Documentation/admin-guide/device-mapper/writecache.rst:23: WARNING: Unexpected indentation. Documentation/vm/highmem:166: ./include/linux/highmem.h:154: WARNING: Inline emphasis start-string without end-string. Documentation/vm/highmem:166: ./include/linux/highmem.h:157: WARNING: Inline emphasis start-string without end-string. But on errors, Sphinx is very noisy, as it outputs the offending lines or tables. For instance, on next-20220620, some patch broke a table. The error there is very noisy, as it outputs the entire broken table. > Seriously. There's a very solid reason why I require the standard > kernel build to be entirely warning-free. Warnings are *BAD*. Even a > single false-positive warning invalidates all other warnings. All the Sphinx warnings could be avoided if people start testing documentation changes. A policy of making kernel htmldocs warning-free would have solved all the above issues. I can submit some patches addressing issues along this week against linux-next (or against your tree?), but without developers trying to build documentation, we'll keep having issues. > And the doc build isn't about some "single warning" thing. > > (b) it requires some unusual build tools > > Now, (b) may some historical oddity, but at least if you have Fedora > installed and it still has sphinx version 2.2.11 or something like > that. > > And when you try to build docs, the makefile gives you some random pip > install thing that is entirely bogus. That's likely the case. You probably installed Sphinx via pip on, let's say, Fedora 35. After upgrading to Fedora 36, python version got incremented, and the past install won't work anymore, because it would be trying to use patch libraries from a different directory. Since this changeset: a8b380c379ef ("scripts: sphinx-pre-install: only ask to activate valid venvs") the documentation build checks if this command works: sphinx-build --version If it doesn't, it will give instructions about how to install Sphinx. > The proper fix is to actually remove that old sphinx environment > entirely, and install "python3-sphinx" instead, but that's not at all > what the "to upgrade Sphinx" docs in the kernel say when you try to > make the docs. The script which checks for Sphinx availability is capable of either recommend distro-specific Sphinx package or the usage of python venv. As Sphinx release cycle is fast, it was opted to use venv by default, so, it currently recommends the usage of pip. The same script called at htmldocs build time can be used to recommend the usage of the distro-provided package. Running it in Fedora, it would give: ./scripts/sphinx-pre-install --no-virtualenv Detected OS: Fedora release 36 (Thirty Six). ERROR: please install "python-sphinx", otherwise, build won't work. Warning: better to also install "sphinx_rtd_theme". Warning: better to also install "texlive-ctex". You should run: sudo dnf install -y python3-sphinx python3-sphinx_rtd_theme texlive-ctex note: If you want pdf, you need at least Sphinx 2.4.4. Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 955. It would be trivial to change the default to recommend the usage of the distro package. Yet: 1. Newer versions of Sphinx are two times slower than 2.4.x; 2. Recommending the same version everywhere ensures that the documentation will be compatible with Sphinx >= 2.4. 3. Older LTS distros may not have a Sphinx version compatible with the Kernel. > Anyway, (b) is one of those "the docs about the docs are wrong" things > and ironic - but easily fixable if you know about it. > > But (a) then makes it all entirely pointless as far as I'm concerned. > The doc build could take five seconds, and I *still* wouldn't bother, > because building docs doesn't actually do anything useful for me. It's > just noise. The hope is that, if building it would be fast enough, developers and maintainers would actually check if the build succeeds without warnings, as ultimately, the big noise is due to patches merged without being even build-tested. Regards, Mauro ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-26 7:55 ` Mauro Carvalho Chehab @ 2022-06-26 9:26 ` Mauro Carvalho Chehab 2022-06-26 9:53 ` Mauro Carvalho Chehab 1 sibling, 0 replies; 40+ messages in thread From: Mauro Carvalho Chehab @ 2022-06-26 9:26 UTC (permalink / raw) To: Linus Torvalds Cc: Jonathan Corbet, Jani Nikula, ksummit, ksummit, Markus Heiser Em Sun, 26 Jun 2022 08:55:24 +0100 Mauro Carvalho Chehab <mchehab@kernel.org> escreveu: > Em Sat, 25 Jun 2022 11:11:40 -0700 > Linus Torvalds <torvalds@linux-foundation.org> escreveu: > > > On Sat, Jun 25, 2022 at 7:00 AM Jonathan Corbet <corbet@lwn.net> wrote: > > > > > > I said "some" - this was a proof-of-concept hack. The complexity of > > > their symbol lookup code is ... daunting ... and not something that gets > > > fixed in one place. More research is required... > > > > Note that at least for me, the issue with building docs isn't hugely > > performance related. > > > > Yes, yes, it would be good if it was faster. But.. > > > > The primary problems I see with building the docs (and thus checking > > them, the same way I do an allmodconfig build test) is > > > > (a) it's insanely noisy, which makes and "check that it's ok" ENTIRELY USELESS. > > When the environment is set for building docs, almost all output there > are actually due to build errors/warnings that got introduced by patches > touching documentation that were never build-tested. > > There are 4 components that report errors during "make htmldocs". > > 1. Documentation cross-reference check. > > $ scripts/documentation-file-ref-check --warn > Warning: Documentation/admin-guide/kernel-parameters.txt references a file that doesn't exist: Documentation/virt/kvm/amd-memory-encryption.rst > Warning: Documentation/dev-tools/kunit/run_wrapper.rst references a file that doesn't exist: Documentation/dev-tools/kunit/non_uml.rst > ... > > Those warnings are result of build-untested patches moving/renaming/removing > documentation without updating all documentation. > > On almost all Kernel versions, I usually send a couple of patch series > addressing them, but incomplete patches that forget about updating references > keep being merged. > > Without htmldocs build tests by patch authors, this is an endless job. Just sent a patch series addressing the existing issues against next-20220624: https://lore.kernel.org/linux-doc/cover.1656234456.git.mchehab@kernel.org/T/#t With that, the number of warnings will be reduced to only 3: $ ./scripts/documentation-file-ref-check Documentation/dev-tools/kunit/run_wrapper.rst: Documentation/dev-tools/kunit/non_uml.rst Documentation/devicetree/bindings/regulator/siliconmitus,sm5703-regulator.yaml: Documentation/devicetree/bindings/mfd/siliconmitus,sm5703.yaml drivers/acpi/device_pm.c: Documentation/firmware-guide/acpi/low-power-probe.rst Those are actually useful for doc discussions, as they show one of the issues we're currently having. Sometimes, support for some functionality crosses multiple subsystems. On such cases, part of the support may end being merged on a future linux-next version - or even on a future Kernel version. That seems to be the case of sm5703 support. The regulator bindings got already merged, but the mfd ones weren't. So, there's not much that could be done to fix siliconmitus,sm5703-regulator.yaml broken reference to mfd/siliconmitus,sm5703.yaml. The same seems to apply to low-power-probe.rst: the patch adding it was not merged yet at linux-next: https://lore.kernel.org/all/20210128232729.16064-3-sakari.ailus@linux.intel.com/ but the patch using it at drivers/acpi/device_pm.c was merged. Regards, Mauro ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-26 7:55 ` Mauro Carvalho Chehab 2022-06-26 9:26 ` Mauro Carvalho Chehab @ 2022-06-26 9:53 ` Mauro Carvalho Chehab 2022-06-27 15:28 ` Liam Howlett 1 sibling, 1 reply; 40+ messages in thread From: Mauro Carvalho Chehab @ 2022-06-26 9:53 UTC (permalink / raw) To: Linus Torvalds Cc: Jonathan Corbet, Jani Nikula, ksummit, ksummit, Markus Heiser Em Sun, 26 Jun 2022 08:55:24 +0100 Mauro Carvalho Chehab <mchehab@kernel.org> escreveu: > Em Sat, 25 Jun 2022 11:11:40 -0700 > Linus Torvalds <torvalds@linux-foundation.org> escreveu: > > > On Sat, Jun 25, 2022 at 7:00 AM Jonathan Corbet <corbet@lwn.net> wrote: > > > > > > I said "some" - this was a proof-of-concept hack. The complexity of > > > their symbol lookup code is ... daunting ... and not something that gets > > > fixed in one place. More research is required... > > > > Note that at least for me, the issue with building docs isn't hugely > > performance related. > > > > Yes, yes, it would be good if it was faster. But.. > > > > The primary problems I see with building the docs (and thus checking > > them, the same way I do an allmodconfig build test) is > > > > (a) it's insanely noisy, which makes and "check that it's ok" ENTIRELY USELESS. > > When the environment is set for building docs, almost all output there > are actually due to build errors/warnings that got introduced by patches > touching documentation that were never build-tested. > > There are 4 components that report errors during "make htmldocs". > > 1. Documentation cross-reference check. ... > 2. ABI documentation check: ... > 3. kernel-doc warnings. ... Btw, once we fix the errors from the above checks, one of the things that could be done in order to avoid noisy doc builds would be to run this during normal Kernel build, if CONFIG_WERROR is set (and if .git is present at the source build dir): ./scripts/documentation-file-ref-check --warn ./scripts/get_abi.pl validate ./scripts/kernel-doc --none $(git grep kernel-doc $(git ls-files Documentation/|grep -v kernel-doc.rst)|perl -ne 'print "$1\n" if (m/kernel-doc::\s*(\S+)/);'|sort|uniq) aborting the build on such warnings. On my notebook (i5-10210U), the above takes ~8 seconds to run. So, it won't make much difference at the build time, and doing that would have avoided ~100 warnings during htmldocs build against current linux-next. Regards, Mauro ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-26 9:53 ` Mauro Carvalho Chehab @ 2022-06-27 15:28 ` Liam Howlett 2022-06-27 15:54 ` Christian Brauner ` (2 more replies) 0 siblings, 3 replies; 40+ messages in thread From: Liam Howlett @ 2022-06-27 15:28 UTC (permalink / raw) To: Mauro Carvalho Chehab Cc: Linus Torvalds, Jonathan Corbet, Jani Nikula, ksummit, ksummit, Markus Heiser * Mauro Carvalho Chehab <mchehab@kernel.org> [220626 05:53]: > Em Sun, 26 Jun 2022 08:55:24 +0100 > Mauro Carvalho Chehab <mchehab@kernel.org> escreveu: > > > Em Sat, 25 Jun 2022 11:11:40 -0700 > > Linus Torvalds <torvalds@linux-foundation.org> escreveu: > > > > > On Sat, Jun 25, 2022 at 7:00 AM Jonathan Corbet <corbet@lwn.net> wrote: > > > > > > > > I said "some" - this was a proof-of-concept hack. The complexity of > > > > their symbol lookup code is ... daunting ... and not something that gets > > > > fixed in one place. More research is required... > > > > > > Note that at least for me, the issue with building docs isn't hugely > > > performance related. > > > > > > Yes, yes, it would be good if it was faster. But.. > > > > > > The primary problems I see with building the docs (and thus checking > > > them, the same way I do an allmodconfig build test) is > > > > > > (a) it's insanely noisy, which makes and "check that it's ok" ENTIRELY USELESS. > > > > When the environment is set for building docs, almost all output there > > are actually due to build errors/warnings that got introduced by patches > > touching documentation that were never build-tested. > > > > There are 4 components that report errors during "make htmldocs". > > > > 1. Documentation cross-reference check. > ... > > 2. ABI documentation check: > ... > > 3. kernel-doc warnings. > ... > > Btw, once we fix the errors from the above checks, one of the things that > could be done in order to avoid noisy doc builds would be to run this > during normal Kernel build, if CONFIG_WERROR is set (and if .git is present > at the source build dir): > > ./scripts/documentation-file-ref-check --warn > ./scripts/get_abi.pl validate > ./scripts/kernel-doc --none $(git grep kernel-doc $(git ls-files Documentation/|grep -v kernel-doc.rst)|perl -ne 'print "$1\n" if (m/kernel-doc::\s*(\S+)/);'|sort|uniq) > > aborting the build on such warnings. > > On my notebook (i5-10210U), the above takes ~8 seconds to run. So, it > won't make much difference at the build time, and doing that would have > avoided ~100 warnings during htmldocs build against current linux-next. > Couldn't we add this to the build bots and ask the authors to fix the commits? Thanks, Liam ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-27 15:28 ` Liam Howlett @ 2022-06-27 15:54 ` Christian Brauner 2022-06-27 16:27 ` Mark Brown 2022-06-28 16:13 ` Luck, Tony 2 siblings, 0 replies; 40+ messages in thread From: Christian Brauner @ 2022-06-27 15:54 UTC (permalink / raw) To: Liam Howlett Cc: Mauro Carvalho Chehab, Linus Torvalds, Jonathan Corbet, Jani Nikula, ksummit, ksummit, Markus Heiser On Mon, Jun 27, 2022 at 03:28:40PM +0000, Liam Howlett wrote: > * Mauro Carvalho Chehab <mchehab@kernel.org> [220626 05:53]: > > Em Sun, 26 Jun 2022 08:55:24 +0100 > > Mauro Carvalho Chehab <mchehab@kernel.org> escreveu: > > > > > Em Sat, 25 Jun 2022 11:11:40 -0700 > > > Linus Torvalds <torvalds@linux-foundation.org> escreveu: > > > > > > > On Sat, Jun 25, 2022 at 7:00 AM Jonathan Corbet <corbet@lwn.net> wrote: > > > > > > > > > > I said "some" - this was a proof-of-concept hack. The complexity of > > > > > their symbol lookup code is ... daunting ... and not something that gets > > > > > fixed in one place. More research is required... > > > > > > > > Note that at least for me, the issue with building docs isn't hugely > > > > performance related. > > > > > > > > Yes, yes, it would be good if it was faster. But.. > > > > > > > > The primary problems I see with building the docs (and thus checking > > > > them, the same way I do an allmodconfig build test) is > > > > > > > > (a) it's insanely noisy, which makes and "check that it's ok" ENTIRELY USELESS. > > > > > > When the environment is set for building docs, almost all output there > > > are actually due to build errors/warnings that got introduced by patches > > > touching documentation that were never build-tested. > > > > > > There are 4 components that report errors during "make htmldocs". > > > > > > 1. Documentation cross-reference check. > > ... > > > 2. ABI documentation check: > > ... > > > 3. kernel-doc warnings. > > ... > > > > Btw, once we fix the errors from the above checks, one of the things that > > could be done in order to avoid noisy doc builds would be to run this > > during normal Kernel build, if CONFIG_WERROR is set (and if .git is present > > at the source build dir): > > > > ./scripts/documentation-file-ref-check --warn > > ./scripts/get_abi.pl validate > > ./scripts/kernel-doc --none $(git grep kernel-doc $(git ls-files Documentation/|grep -v kernel-doc.rst)|perl -ne 'print "$1\n" if (m/kernel-doc::\s*(\S+)/);'|sort|uniq) > > > > aborting the build on such warnings. > > > > On my notebook (i5-10210U), the above takes ~8 seconds to run. So, it > > won't make much difference at the build time, and doing that would have > > avoided ~100 warnings during htmldocs build against current linux-next. > > > > Couldn't we add this to the build bots and ask the authors to fix the > commits? Isn't for-next already integrated? Maybe not. I got a mail from Andrew telling me I had a kernel-doc copy-paste error. Integration with build-bots would be great. I generally aim to add kernel-doc for all non-trivial code that I add. But during different patchset versions copy-paste errors do sometimes happen (e.g., for git grep/sed calls). ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-27 15:28 ` Liam Howlett 2022-06-27 15:54 ` Christian Brauner @ 2022-06-27 16:27 ` Mark Brown 2022-06-28 10:53 ` Mauro Carvalho Chehab 2022-06-28 16:13 ` Luck, Tony 2 siblings, 1 reply; 40+ messages in thread From: Mark Brown @ 2022-06-27 16:27 UTC (permalink / raw) To: Liam Howlett Cc: Mauro Carvalho Chehab, Linus Torvalds, Jonathan Corbet, Jani Nikula, ksummit, ksummit, Markus Heiser [-- Attachment #1: Type: text/plain, Size: 1374 bytes --] On Mon, Jun 27, 2022 at 03:28:40PM +0000, Liam Howlett wrote: > * Mauro Carvalho Chehab <mchehab@kernel.org> [220626 05:53]: > > Btw, once we fix the errors from the above checks, one of the things that > > could be done in order to avoid noisy doc builds would be to run this > > during normal Kernel build, if CONFIG_WERROR is set (and if .git is present > > at the source build dir): > > ./scripts/documentation-file-ref-check --warn > > ./scripts/get_abi.pl validate > > ./scripts/kernel-doc --none $(git grep kernel-doc $(git ls-files Documentation/|grep -v kernel-doc.rst)|perl -ne 'print "$1\n" if (m/kernel-doc::\s*(\S+)/);'|sort|uniq) That last one is quite the command line... > > aborting the build on such warnings. > > On my notebook (i5-10210U), the above takes ~8 seconds to run. So, it > > won't make much difference at the build time, and doing that would have > > avoided ~100 warnings during htmldocs build against current linux-next. > Couldn't we add this to the build bots and ask the authors to fix the > commits? There's reports for at least some of this for -next already, at least for newly introduced warnings. If we could get 0day or something else that picks stuff off the lists that would help a lot I think, as would getting it into the bots that people who like to fix up warnings tend to be using. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-27 16:27 ` Mark Brown @ 2022-06-28 10:53 ` Mauro Carvalho Chehab 0 siblings, 0 replies; 40+ messages in thread From: Mauro Carvalho Chehab @ 2022-06-28 10:53 UTC (permalink / raw) To: Mark Brown Cc: Liam Howlett, Linus Torvalds, Jonathan Corbet, Jani Nikula, ksummit, ksummit, Markus Heiser Em Mon, 27 Jun 2022 17:27:27 +0100 Mark Brown <broonie@kernel.org> escreveu: > On Mon, Jun 27, 2022 at 03:28:40PM +0000, Liam Howlett wrote: > > * Mauro Carvalho Chehab <mchehab@kernel.org> [220626 05:53]: > > > > Btw, once we fix the errors from the above checks, one of the things that > > > could be done in order to avoid noisy doc builds would be to run this > > > during normal Kernel build, if CONFIG_WERROR is set (and if .git is present > > > at the source build dir): > > > > ./scripts/documentation-file-ref-check --warn Sent a patch series addressing those at linux-next: https://lore.kernel.org/linux-doc/cover.1656234456.git.mchehab@kernel.org/T/#t > > > ./scripts/get_abi.pl validate I submited a patch fixing one such issue: https://lore.kernel.org/all/1e92337c1ef74f5eb9e1c1871e20b858b490d269.1656235926.git.mchehab@kernel.org/ and Jonathan Cameron submitted a series addressing those: https://lore.kernel.org/linux-iio/20220626165511.602202-1-jic23@kernel.org/T/#t > > > ./scripts/kernel-doc --none $(git grep kernel-doc $(git ls-files Documentation/|grep -v kernel-doc.rst)|perl -ne 'print "$1\n" if (m/kernel-doc::\s*(\S+)/);'|sort|uniq) Sent a patch series addressing all of those, also against linux-next: https://lore.kernel.org/linux-doc/cover.1656409369.git.mchehab@kernel.org/T/#t next-20220627 has currently: 32 cross broken docs cross reference warnings; 4 duplicated ABI symbols; 60 kernel-doc warnings. After having those three series applied, the 3 above commands would produce just 3 warnings: Warning: Documentation/dev-tools/kunit/run_wrapper.rst references a file that doesn't exist: Documentation/dev-tools/kunit/non_uml.rst Warning: Documentation/devicetree/bindings/regulator/siliconmitus,sm5703-regulator.yaml references a file that doesn't exist: Documentation/devicetree/bindings/mfd/siliconmitus,sm5703.yaml Warning: drivers/acpi/device_pm.c references a file that doesn't exist: Documentation/firmware-guide/acpi/low-power-probe.rst Those 3 warnings depend on non-merged documents. After those ~100 warnings that can be quickly checked with the 3 above command lines in a couple of seconds, there are the Sphinx errors. Reducing the build time (currently at 10 mins order) to something more palatable could help having more people checking them before submitting patches and waiting for 0day results. > That last one is quite the command line... True. Yet, placing it at docs/Makefile for the "make all" target should be trivial and will hopefully make more Kernel developers to check for some of the documentation issues. The other two are already called, depending on some config vars: # Check for broken documentation file references ifeq ($(CONFIG_WARN_MISSING_DOCUMENTS),y) $(shell $(srctree)/scripts/documentation-file-ref-check --warn) endif # Check for broken ABI files ifeq ($(CONFIG_WARN_ABI_ERRORS),y) $(shell $(srctree)/scripts/get_abi.pl validate --dir $(srctree)/Documentation/ABI) endif But they don't currently stop the build if CONFIG_WERROR is enabled. > > > > aborting the build on such warnings. > > > > On my notebook (i5-10210U), the above takes ~8 seconds to run. So, it > > > won't make much difference at the build time, and doing that would have > > > avoided ~100 warnings during htmldocs build against current linux-next. > > > Couldn't we add this to the build bots and ask the authors to fix the > > commits? > > There's reports for at least some of this for -next already, at least > for newly introduced warnings. If we could get 0day or something else > that picks stuff off the lists that would help a lot I think, as would > getting it into the bots that people who like to fix up warnings tend to > be using. Yes, newly-introduced warnings are monitored there, but still patches adding new warnings keep being merged, requiring extra periodic janitorial work in order to reduce the build noise that would otherwise increase with time. Regards, Mauro ^ permalink raw reply [flat|nested] 40+ messages in thread
* RE: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-27 15:28 ` Liam Howlett 2022-06-27 15:54 ` Christian Brauner 2022-06-27 16:27 ` Mark Brown @ 2022-06-28 16:13 ` Luck, Tony 2 siblings, 0 replies; 40+ messages in thread From: Luck, Tony @ 2022-06-28 16:13 UTC (permalink / raw) To: Liam Howlett, Mauro Carvalho Chehab Cc: Linus Torvalds, Jonathan Corbet, Nikula, Jani, ksummit, ksummit, Markus Heiser >> On my notebook (i5-10210U), the above takes ~8 seconds to run. So, it >> won't make much difference at the build time, and doing that would have >> avoided ~100 warnings during htmldocs build against current linux-next. >> > > Couldn't we add this to the build bots and ask the authors to fix the > commits? I poked the zero-day bot owners. They are looking into it. Reminder that the zero-day folks are open to suggestions on how to improve coverage. Just ping me or Dave Hansen and we can pass on requests. -Tony ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-25 18:11 ` Linus Torvalds 2022-06-26 7:55 ` Mauro Carvalho Chehab @ 2022-06-27 15:34 ` Jonathan Corbet 2022-06-27 17:07 ` Linus Torvalds 2022-07-02 10:52 ` Mauro Carvalho Chehab 2 siblings, 1 reply; 40+ messages in thread From: Jonathan Corbet @ 2022-06-27 15:34 UTC (permalink / raw) To: Linus Torvalds Cc: Mauro Carvalho Chehab, Jani Nikula, ksummit, ksummit, Markus Heiser Linus Torvalds <torvalds@linux-foundation.org> writes: > On Sat, Jun 25, 2022 at 7:00 AM Jonathan Corbet <corbet@lwn.net> wrote: >> >> I said "some" - this was a proof-of-concept hack. The complexity of >> their symbol lookup code is ... daunting ... and not something that gets >> fixed in one place. More research is required... > > Note that at least for me, the issue with building docs isn't hugely > performance related. Those of us who do lots of docs builds see it a bit differently :) > Yes, yes, it would be good if it was faster. But.. > > The primary problems I see with building the docs (and thus checking > them, the same way I do an allmodconfig build test) is > > (a) it's insanely noisy, which makes and "check that it's ok" ENTIRELY USELESS. > > Seriously. There's a very solid reason why I require the standard > kernel build to be entirely warning-free. Warnings are *BAD*. Even a > single false-positive warning invalidates all other warnings. I totally agree, which is why fixing warnings has been at the top of the priority list for a long time. That has mostly been done, with Mauro having taken on the bulk of that work. It is hard, though, to maintain the warning-free condition. Warnings are easy to add and most developers don't check. Most documentation changes don't go through the docs tree, so I can't enforce that kind of check (and trying to force docs changes through my tree would be a cure far worse than the disease). Making the docs build faster might make it easier to insist that more checking is done. > And the doc build isn't about some "single warning" thing. > > (b) it requires some unusual build tools > > Now, (b) may some historical oddity, but at least if you have Fedora > installed and it still has sphinx version 2.2.11 or something like > that. > > And when you try to build docs, the makefile gives you some random pip > install thing that is entirely bogus. I'd be curious to know about how you got into that situation. Fedora handles this stuff pretty readily. I'll take another look at what we have there. jon ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-27 15:34 ` Jonathan Corbet @ 2022-06-27 17:07 ` Linus Torvalds 0 siblings, 0 replies; 40+ messages in thread From: Linus Torvalds @ 2022-06-27 17:07 UTC (permalink / raw) To: Jonathan Corbet Cc: Mauro Carvalho Chehab, Jani Nikula, ksummit, ksummit, Markus Heiser On Mon, Jun 27, 2022 at 8:34 AM Jonathan Corbet <corbet@lwn.net> wrote: > > > (b) it requires some unusual build tools > > > > Now, (b) may some historical oddity, but at least if you have Fedora > > installed and it still has sphinx version 2.2.11 or something like > > that. > > > > And when you try to build docs, the makefile gives you some random pip > > install thing that is entirely bogus. > > I'd be curious to know about how you got into that situation. Fedora > handles this stuff pretty readily. I'll take another look at what we > have there. At some point I just did "dnf install sphinx", and it got me a sphinx install. So then the kernel doc build system sees "oh, you need a newer version of sphinx" or whatever, and suggests you do that horrible virtual python sandbox thing and do a random pip install in that, which I really don't want to do. So I ignore it entirely, obviously. End result: it took me a *loong* time to go "I wonder why modern Fedora ships with an ancient version of sphinx, since the modern version is apparently 4.x something". And only at THAT point do you realize that there are three completely different projects called "sphinx". There's the docbuild one that comes from python, which is the sphinx I want. There's the search engine 'sphinx', which is what is packaged as "sphinx" in Fedora, and which the documentation code obviously refuses to use. There's also CMU sphinx, which is a voice recognition system. End result: building the docs can be really confusing if you have the wrong sphinx installed, and the "help message" is actually misleading and very unhelpful. Linus ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] What kernel documentation could be 2022-06-25 18:11 ` Linus Torvalds 2022-06-26 7:55 ` Mauro Carvalho Chehab 2022-06-27 15:34 ` Jonathan Corbet @ 2022-07-02 10:52 ` Mauro Carvalho Chehab 2 siblings, 0 replies; 40+ messages in thread From: Mauro Carvalho Chehab @ 2022-07-02 10:52 UTC (permalink / raw) To: Linus Torvalds Cc: Jonathan Corbet, Jani Nikula, ksummit, ksummit, Markus Heiser Em Sat, 25 Jun 2022 11:11:40 -0700 Linus Torvalds <torvalds@linux-foundation.org> escreveu: > On Sat, Jun 25, 2022 at 7:00 AM Jonathan Corbet <corbet@lwn.net> wrote: > > > > I said "some" - this was a proof-of-concept hack. The complexity of > > their symbol lookup code is ... daunting ... and not something that gets > > fixed in one place. More research is required... > > Note that at least for me, the issue with building docs isn't hugely > performance related. > > Yes, yes, it would be good if it was faster. But.. > > The primary problems I see with building the docs (and thus checking > them, the same way I do an allmodconfig build test) is > > (a) it's insanely noisy, which makes and "check that it's ok" ENTIRELY USELESS. I solved almost all warnings that can be fixed against -next. Sending patches today. However, there is an special case where warnings will be hit for perfectly fine code, when docs are built with Sphinx 3.0 and upper: Documentation/driver-api/usb/usb:164: ./drivers/usb/core/message.c:967: WARNING: Duplicate C declaration, also defined at driver-api/usb/gadget:783. Declaration is '.. c:function:: int usb_string (struct usb_device *dev, int index, char *buf, size_t size)'. Documentation/driver-api/usb/usb.rst:967: WARNING: Duplicate C declaration, also defined at driver-api/usb/gadget:783. Declaration is '.. c:struct:: usb_string'. In this specific case, there are both a struct and a function named "usb_string". So, the above is not an issue. While this is not fixed, there will be a 22 lines noise (11 two-line warnings) after my fixes[1]. The root cause is that Sphinx has a single namespace for symbols, no matter if the symbol is a function, a struct, an enum, etc. So, when the same name is declared on multiple places, like in the case of "usb_string", the cross-references will either point to the function or to the struct. While this is annoying, it is usually not a big deal, as they are typically placed at the same rst file. While this problem affects *all* Sphinx versions so far, Sphinx 2.x.x doesn't warn when this happen, but newer versions complain about that. This is a well known issue and there are patches fixing it since Sphinx 3.1: https://github.com/sphinx-doc/sphinx/pull/8313 Those patches were expected to be merged on Sphinx 3.3, but ended not being merged so far, as applying them would break support for a Sphinx extension that it is used on several places. --- [1] I got one case where the warning was actually reporting an issue: Documentation/driver-api/hte/tegra194-hte:28: ./drivers/gpio/gpiolib.c:2464: WARNING: Duplicate C declaration, also defined at driver-api> Declaration is '.. c:function:: int gpiod_enable_hw_timestamp_ns (struct gpio_desc *desc, unsigned long flags)'. Documentation/driver-api/hte/tegra194-hte.rst:2464: WARNING: Duplicate C declaration, also defined at driver-api/gpio/index:2464. Declaration is '.. c:function:: int gpiod_enable_hw_timestamp_ns(struct gpio_desc *desc, unsigned long flags)'. Basically drivers/gpio/gpiolib.c is included twice for the same symbols, one at Documentation/driver-api/gpio/index.rst, and then Documentation/driver-api/hte/tegra194-hte.rst includes it again for gpiod_enable_hw_timestamp_ns() and gpiod_disable_hw_timestamp_ns(). Regards, Mauro ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [TECH TOPIC] What kernel documentation could be 2022-06-17 20:57 [TECH TOPIC] What kernel documentation could be Jonathan Corbet ` (2 preceding siblings ...) 2022-06-18 8:24 ` Mauro Carvalho Chehab @ 2022-06-25 17:43 ` Steven Rostedt 2022-06-25 17:48 ` Laurent Pinchart 3 siblings, 1 reply; 40+ messages in thread From: Steven Rostedt @ 2022-06-25 17:43 UTC (permalink / raw) To: Jonathan Corbet; +Cc: ksummit-discuss, ksummit On Fri, 17 Jun 2022 14:57:10 -0600 Jonathan Corbet <corbet@lwn.net> wrote: > The development community has put a lot of work into the kernel's > documentation directory in recent years, with visible results. But the > kernel's documentation still falls far short of the standard set by many > other projects, and there is a great deal of "tribal knowledge" in our > community that is not set down. I'd like to look at the successes and > failures of the work so far, but intent to focus on a discussion of what > our documentation should be and what we can do to get it there. > > Questions to discuss include: > > - Bringing an coherent overall structure to Documentation/ > > - Making it easier for developers to improve the documentation. Is there a way to mark comments in the code as "design docs". And I don't mean the kernel-doc that is already above functions. I have lots of comments about the design of complex code just above the code it is describing. Which is much more likely to get updated when the design changes or gets new features. I found my design documents I have under Documentation to become grossly obsolete over time, where as the comments above the code I can keep up with. Is there already some kind of mark up that we can add to the comments to state: ** .design.trace/ftrace.rst .ringbuffer. ** or something that can be extracted into another document? With the above example, I could have in Documentatino/trace/ftrace.rst a .design .ringbuffer. And the comment from the code would be extracted into the html documents that are created? Or maybe it would be better to reference the code from the documentation? .include kerne/trace/ring_buffer.c .ringbuffer. That will look for a tag in kernel/trace/ring_buffer.c called ".ringbuffer" and pull in the comment into the document under Documentation. Thoughts? -- Steve > > - What we would like from Sphinx and what we can do to make it happen > > - Making the docs build system less ugly and more maintainable > > - Integrating rustdoc documentation > > jon ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [TECH TOPIC] What kernel documentation could be 2022-06-25 17:43 ` Steven Rostedt @ 2022-06-25 17:48 ` Laurent Pinchart 0 siblings, 0 replies; 40+ messages in thread From: Laurent Pinchart @ 2022-06-25 17:48 UTC (permalink / raw) To: Steven Rostedt; +Cc: Jonathan Corbet, ksummit-discuss, ksummit Hi Steven, On Sat, Jun 25, 2022 at 01:43:28PM -0400, Steven Rostedt wrote: > On Fri, 17 Jun 2022 14:57:10 -0600 Jonathan Corbet wrote: > > > The development community has put a lot of work into the kernel's > > documentation directory in recent years, with visible results. But the > > kernel's documentation still falls far short of the standard set by many > > other projects, and there is a great deal of "tribal knowledge" in our > > community that is not set down. I'd like to look at the successes and > > failures of the work so far, but intent to focus on a discussion of what > > our documentation should be and what we can do to get it there. > > > > Questions to discuss include: > > > > - Bringing an coherent overall structure to Documentation/ > > > > - Making it easier for developers to improve the documentation. > > Is there a way to mark comments in the code as "design docs". And I > don't mean the kernel-doc that is already above functions. > > I have lots of comments about the design of complex code just above the > code it is describing. Which is much more likely to get updated when > the design changes or gets new features. > > I found my design documents I have under Documentation to become > grossly obsolete over time, where as the comments above the code I can > keep up with. > > Is there already some kind of mark up that we can add to the comments to state: > > ** .design.trace/ftrace.rst .ringbuffer. ** > > or something that can be extracted into another document? With the > above example, I could have in Documentatino/trace/ftrace.rst a > > .design .ringbuffer. > > And the comment from the code would be extracted into the html > documents that are created? > > Or maybe it would be better to reference the code from the documentation? > > .include kerne/trace/ring_buffer.c .ringbuffer. > > That will look for a tag in kernel/trace/ring_buffer.c called > ".ringbuffer" and pull in the comment into the document under Documentation. > > Thoughts? See the "DOC: overview" block in drivers/gpu/drm/drm_atomic_helper.c, and how it is integrated in Documentation/gpu/drm-kms-helpers.rst with .. kernel-doc:: drivers/gpu/drm/drm_atomic_helper.c :doc: overview Is that what you're looking for ? > > - What we would like from Sphinx and what we can do to make it happen > > > > - Making the docs build system less ugly and more maintainable > > > > - Integrating rustdoc documentation -- Regards, Laurent Pinchart ^ permalink raw reply [flat|nested] 40+ messages in thread
end of thread, other threads:[~2022-07-02 12:43 UTC | newest] Thread overview: 40+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-06-17 20:57 [TECH TOPIC] What kernel documentation could be Jonathan Corbet 2022-06-17 20:57 ` [Ksummit-discuss] " Jonathan Corbet 2022-06-17 21:48 ` Laurent Pinchart 2022-06-17 21:48 ` Laurent Pinchart 2022-06-27 15:18 ` Jonathan Corbet 2022-06-18 8:24 ` Mauro Carvalho Chehab 2022-06-18 8:24 ` Mauro Carvalho Chehab 2022-06-18 11:03 ` Miguel Ojeda 2022-06-18 11:16 ` Mauro Carvalho Chehab 2022-06-18 11:16 ` Mauro Carvalho Chehab 2022-06-18 14:37 ` Miguel Ojeda 2022-06-23 9:18 ` Jani Nikula 2022-06-23 9:57 ` Mauro Carvalho Chehab 2022-06-23 10:30 ` Jani Nikula 2022-06-23 13:40 ` Jonathan Corbet 2022-06-24 7:33 ` Mauro Carvalho Chehab 2022-06-24 16:37 ` Markus Heiser 2022-06-27 15:27 ` Jonathan Corbet 2022-06-27 15:31 ` Christoph Hellwig 2022-06-28 7:43 ` Mauro Carvalho Chehab 2022-06-28 7:57 ` Geert Uytterhoeven 2022-06-28 11:01 ` Mauro Carvalho Chehab 2022-07-02 12:43 ` Stephen Rothwell 2022-06-24 22:57 ` Jonathan Corbet 2022-06-25 9:10 ` Mauro Carvalho Chehab 2022-06-25 14:00 ` Jonathan Corbet 2022-06-25 18:11 ` Linus Torvalds 2022-06-26 7:55 ` Mauro Carvalho Chehab 2022-06-26 9:26 ` Mauro Carvalho Chehab 2022-06-26 9:53 ` Mauro Carvalho Chehab 2022-06-27 15:28 ` Liam Howlett 2022-06-27 15:54 ` Christian Brauner 2022-06-27 16:27 ` Mark Brown 2022-06-28 10:53 ` Mauro Carvalho Chehab 2022-06-28 16:13 ` Luck, Tony 2022-06-27 15:34 ` Jonathan Corbet 2022-06-27 17:07 ` Linus Torvalds 2022-07-02 10:52 ` Mauro Carvalho Chehab 2022-06-25 17:43 ` Steven Rostedt 2022-06-25 17:48 ` Laurent Pinchart
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).