[RFC] scripts: kernel-doc: Major kernel-doc rework

[RFC] scripts: kernel-doc: Major kernel-doc rework

[Tomasz Warniełło]

This is in fact a release notification of a major kernel-doc script
refurbishment I have done. My work has reached a stage, which can be
considered a world sync point and here we are.

I'm not sending all the patches to the Linux mailing list, as I prefer
to check what you think before I emit about 500 emails.

I've parked the lot for you to inspect here:
https://salsa.debian.org/guest/kernel-doc

This also helps me report the bug fixes. See the issue tracker:
https://salsa.debian.org/guest/kernel-doc/-/issues

A FEW PRACTICAL WORDS

Before I move on, I'd like to ask you for support. I can't find a job -
neither as a junior, nor even part time. So if you find my effort
valuable and are able to, send me some Dollars, Euro or other Yen or
Ruble to my PayPal account linked with:

tomasz.warniello@gmail.com

If you have a job offer for me, the address stays the same. I consider
myself a general Linux assistant, in case you wonder.

Thank you.

BASIC INFO

The main area of my rework's focus is modularisation. The leading theme
is getting rid of the globals. In the meantime, smaller chunks of code
get assigned to packages, instance interfaces are built and wrap instance
variables, while helper functions land in their container functions,
various variables get lexicalised in their scopes, etc.

What is also done are countless simplifications, reductions, merges,
unifications, but also splits and duplications where logic and
architecture suggests so or requires.

The starting point for this rework is the official Linux tree commit
2a987e65025e2b79c6d453b78cb5985ac6e5eb26. It is also the base for all
tests performed throughout the process. (Except it changes slightly
with bug fixes.)

The initial fifteen patches are the POD series I published on the Kernel
mailing list. It was already version 3, which somehow got forgotten or
whatever happened to it. You can find it here:
https://lore.kernel.org/linux-doc/20220104015946.529524-1-tomasz.warniello@gmail.com/

One exception here is the copyright patch, which now is version 4.

The final, release version offers:
- ca. 20 bug fixes - big and small
- a few extra features
- no wild globals
- execution time not much worse than the original, despite the internal
  bureaucracy - in fact much better, though maybe not with single files

PATCH ERRATA

For the first ca. 200 patches, when I did indentation and spacing
corrections, I also simplified parentheses. Then I thought I'd better
cover this activity in patch descriptions. Correcting all the early ones
feels far beyond my powers.

PATCH MARKS

1. Enhancements
Patches marked with * at the end of the title contain changes that go
slightly against the master. I'm sometimes unsure if they're enhancements
or bug fixes.

2. Uncertainty
Patches marked with (?) at the of the title are those, which I consider
difficult to check. They may be ok, but should be examined with special
care.

SERIES GUIDE

The first patch in the repository contains the base, ie. the master
version of the script. Please let me know if I should have included
something essential like a license, etc. I've done my research and
came to the conclusion the script can move on its own. But...

The last patch contains this note and a few test scripts I used.

All the patches between them contain only changes to kernel-doc itself.
They are all signed-off.

TESTING

Initially it was chaotic with rst conclusions based on HTML builds and
with scant notes only in every some patch. But it soon got better and
more intelligent. I stated to inspect pure rst and included a note in
every patch unless no test was actually performed. It stayed so ever
after with light tests based on a 200 file sample and heavier tests
covering all available files.

The test reference is the master Linux commit, changing with bugfixes.

REUSE AND PORTING

The series is ready to be pulled by the Linux upstream at will. Of course
I can send it to the Linux mailing list - just let me know. The patches
are signed-off, patch-checked, and ready to go, the best of my knowledge.

That is - except for things like indentation, which is not worse than in
the master if you ask me, but it definitely is far from any claim of
correctness. Not in the conventional sense.

A few patches may undoubtedly be considered egoistic, like the Gedit
syntax highlight helper, or many of the indentation corrections. What can
I say? I needed them. Without them I would be fighting with an uncessant
swarm of tiny difficulties, and would quite likely never get to the end.

What else then? You might be interested in porting bug fixes, which may be
or may not be easy, depending on the case. You might also want to use my
version as an output reference. It's there for you as a whole and as parts.
It's up to you, so help yourselves as you please.

FINAL THOUGHTS

I could have used the POD space prepared for documentation and written
at least general summaries for all modules. But that's another adventure.
I need a break now.

There is still a lot to do around this script. parse_body is a noble warm up
candidate.

Re: [RFC] scripts: kernel-doc: Major kernel-doc rework

[Jonathan Corbet]

Tomasz Warniełło <tomasz.warniello@gmail.com> writes:

> This is in fact a release notification of a major kernel-doc script
> refurbishment I have done. My work has reached a stage, which can be
> considered a world sync point and here we are.
>
> I'm not sending all the patches to the Linux mailing list, as I prefer
> to check what you think before I emit about 500 emails.
>
> I've parked the lot for you to inspect here:
> https://salsa.debian.org/guest/kernel-doc
>
> This also helps me report the bug fixes. See the issue tracker:
> https://salsa.debian.org/guest/kernel-doc/-/issues

So I have done you a disservice with this work; I've been distracted and
it has languished.  I apologize.  It's at the top of the list and will
be the next thing I put some serious attention to.

Thanks,

jon

Re: [RFC] scripts: kernel-doc: Major kernel-doc rework

[Jonathan Corbet]

Tomasz Warniełło <tomasz.warniello@gmail.com> writes:

> This is in fact a release notification of a major kernel-doc script
> refurbishment I have done. My work has reached a stage, which can be
> considered a world sync point and here we are.
>
> I'm not sending all the patches to the Linux mailing list, as I prefer
> to check what you think before I emit about 500 emails.
>
> I've parked the lot for you to inspect here:
> https://salsa.debian.org/guest/kernel-doc
>
> This also helps me report the bug fixes. See the issue tracker:
> https://salsa.debian.org/guest/kernel-doc/-/issues

I've taken a quick look - that's a lot of stuff.  Thanks for not sending
it all; you would have gotten some unhappy answers.

Some overall thoughts:

 - Work like this needs to be broken up into digestible batches.  Let's
   start with the POD stuff that I've (finally) commented on; other
   stuff can come later.

 - The coding style in the new work is very unkernellike; that will make
   it harder to get this work merged.

 - But let's take a step back and ask: why are we doing all of this work
   in the first place?  What is the benefit to the kernel community from
   all this churn, and a growth of the kernel-doc script by over 2,000
   lines (even if an awful lot of them are blank)?

I'm serious about that last question; do we really want to invest that
kind of effort into this nasty old script?  Or, if we're going to do
such a thing, should we maybe start with Markus's rewrite into Python
instead?  If we're going to thrash the code and make it unrecognizable,
perhaps we should move to a language that is consistent with the rest of
the docs build system and which, I believe, is easier for more kernel
developers to deal with?

I am *not* saying that this work cannot be accepted, and I certainly do
not want to alienate somebody who is actually able to look at kernel-doc
and not have their eyes bleed out.  But I am saying that, before
launching into a hundreds-of-patches journey, we should know where we're
going and why we are doing it.

See what I'm getting at?

Thanks,

jon

Re: [RFC] scripts: kernel-doc: Major kernel-doc rework

[Tomasz Warniełło]

On Wed, 16 Feb 2022 16:45:25 -0700
Jonathan Corbet <corbet@lwn.net> wrote:

> Some overall thoughts:
> 
>  - Work like this needs to be broken up into digestible batches.  Let's
>    start with the POD stuff that I've (finally) commented on; other
>    stuff can come later.

I will prepare the POD part. As for the rest, you will need a porter if you
decide this is the way to go. I could help with explanations if something is
unclear and as long as the questions come soon. The winter is almost over and
I need to switch to other activities like earning money.

>  - The coding style in the new work is very unkernellike; that will make
>    it harder to get this work merged.

Yes. I can only add a thought: it seems unnatural to me to have the same style
for various languages.

>  - But let's take a step back and ask: why are we doing all of this work
>    in the first place?  What is the benefit to the kernel community from
>    all this churn, and a growth of the kernel-doc script by over 2,000
>    lines (even if an awful lot of them are blank)?

My motivation was just climbing this mountain, nothing else. I don't even
know, how important this script is to whom. I can only guess. Neither am I
a C programmer, nor want to become one. So this question is to another part
of the community.

I could though think of the benefits to the maintainers. In this respect I've
moved things forward quite a bit, I gather.

As for the blank lines, they are just one character each and trivial to get
rid of.

> I'm serious about that last question; do we really want to invest that
> kind of effort into this nasty old script?  Or, if we're going to do
> such a thing, should we maybe start with Markus's rewrite into Python
> instead?  If we're going to thrash the code and make it unrecognizable,
> perhaps we should move to a language that is consistent with the rest of
> the docs build system and which, I believe, is easier for more kernel
> developers to deal with?

I understand. I didn't know there was a rewrite. I'm not into Python, but
if you could post a link, I'd take a look out of curiosity.

If the community prefers Python, what can I say about this? Their choice.

Personally, I'd rather play with translating this script to Raku (aka Perl 6).
I also wonder, how Perl 5 will transition to Perl 7. A question to Perl
experts.

> I am *not* saying that this work cannot be accepted, and I certainly do
> not want to alienate somebody who is actually able to look at kernel-doc
> and not have their eyes bleed out.  But I am saying that, before
> launching into a hundreds-of-patches journey, we should know where we're
> going and why we are doing it.

I agree.

> See what I'm getting at?

Yes. Thanks.

Tomasz

Re: [RFC] scripts: kernel-doc: Major kernel-doc rework

[Jonathan Corbet]

Tomasz Warniełło <tomasz.warniello@gmail.com> writes:

> I will prepare the POD part. As for the rest, you will need a porter if you
> decide this is the way to go. I could help with explanations if something is
> unclear and as long as the questions come soon. The winter is almost over and
> I need to switch to other activities like earning money.

Understood.

> I understand. I didn't know there was a rewrite. I'm not into Python, but
> if you could post a link, I'd take a look out of curiosity.
>
> If the community prefers Python, what can I say about this? Their choice.

This is the kernel community you're talking about - saying that it
prefers any language (except possibly C) is asking for big-time trouble
:)

*I* prefer Python, and the Sphinx side of things is necessarily in
Python, so I'd be happy to see kernel-doc move over.  That said, others
certainly disagree.

Markus's work was here:

  https://lore.kernel.org/lkml/1485287564-24205-1-git-send-email-markus.heiser@darmarit.de/

At the time, we were just trying to get the RST transition done, and
swapping out the kernel-doc script seemed like a major distraction that
we didn't need, so this never got looked at as seriously as I would have
liked.

> Personally, I'd rather play with translating this script to Raku (aka
> Perl 6).

Trying to add yet another language dependency is another path to
unhappiness, we're not going to go there.

Thanks,

jon

Re: [RFC] scripts: kernel-doc: Major kernel-doc rework

[Randy Dunlap]

On 2/17/22 09:04, Jonathan Corbet wrote:
> Tomasz Warniełło <tomasz.warniello@gmail.com> writes:
> 
>> I will prepare the POD part. As for the rest, you will need a porter if you
>> decide this is the way to go. I could help with explanations if something is
>> unclear and as long as the questions come soon. The winter is almost over and
>> I need to switch to other activities like earning money.
> 
> Understood.
> 
>> I understand. I didn't know there was a rewrite. I'm not into Python, but
>> if you could post a link, I'd take a look out of curiosity.
>>
>> If the community prefers Python, what can I say about this? Their choice.
> 
> This is the kernel community you're talking about - saying that it
> prefers any language (except possibly C) is asking for big-time trouble
> :)
> 
> *I* prefer Python, and the Sphinx side of things is necessarily in
> Python, so I'd be happy to see kernel-doc move over.  That said, others
> certainly disagree.
> 
> Markus's work was here:
> 
>   https://lore.kernel.org/lkml/1485287564-24205-1-git-send-email-markus.heiser@darmarit.de/

I can barely modify Python script, but I think that this would be a good move overall.
Then I could learn more Python.

> At the time, we were just trying to get the RST transition done, and
> swapping out the kernel-doc script seemed like a major distraction that
> we didn't need, so this never got looked at as seriously as I would have
> liked.
> 
>> Personally, I'd rather play with translating this script to Raku (aka
>> Perl 6).
> 
> Trying to add yet another language dependency is another path to
> unhappiness, we're not going to go there.


-- 
~Randy

Re: [RFC] scripts: kernel-doc: Major kernel-doc rework

[Tomasz Warniełło]

On Thu, 17 Feb 2022 10:04:23 -0700
Jonathan Corbet <corbet@lwn.net> wrote:

> This is the kernel community you're talking about - saying that it
> prefers any language (except possibly C) is asking for big-time trouble
> :)

It should be written in C then.

> Markus's work was here:
> 
>   https://lore.kernel.org/lkml/1485287564-24205-1-git-send-email-markus.heiser@darmarit.de/
> 
> At the time, we were just trying to get the RST transition done, and
> swapping out the kernel-doc script seemed like a major distraction that
> we didn't need, so this never got looked at as seriously as I would have
> liked.

Thanks. Interesting.

I should have mentioned getting rid of the parser states in my release
announcement.

Re: [RFC] scripts: kernel-doc: Major kernel-doc rework

[Matthew Wilcox]

On Thu, Feb 17, 2022 at 10:04:23AM -0700, Jonathan Corbet wrote:
> *I* prefer Python, and the Sphinx side of things is necessarily in
> Python, so I'd be happy to see kernel-doc move over.  That said, others
> certainly disagree.
> 
> Markus's work was here:
> 
>   https://lore.kernel.org/lkml/1485287564-24205-1-git-send-email-markus.heiser@darmarit.de/
> 
> At the time, we were just trying to get the RST transition done, and
> swapping out the kernel-doc script seemed like a major distraction that
> we didn't need, so this never got looked at as seriously as I would have
> liked.

Personally, I'd like to see us switch over to
https://github.com/jnikula/hawkmoth

but I don't have time to work on the rough edges.  I really hate the
kernel-doc style; I think it makes us write very stilted documentation,
full of parameter descriptions like:

 function() - Do the thing to a page.
 @page: The page.

which really serves nobody.  Being able to write:

 function() - Do the thing to @page

is easier to both write and read.

Re: [RFC] scripts: kernel-doc: Major kernel-doc rework

[Jani Nikula]

On Fri, 18 Feb 2022, Matthew Wilcox <willy@infradead.org> wrote:
> On Thu, Feb 17, 2022 at 10:04:23AM -0700, Jonathan Corbet wrote:
>> *I* prefer Python, and the Sphinx side of things is necessarily in
>> Python, so I'd be happy to see kernel-doc move over.  That said, others
>> certainly disagree.
>> 
>> Markus's work was here:
>> 
>>   https://lore.kernel.org/lkml/1485287564-24205-1-git-send-email-markus.heiser@darmarit.de/
>> 
>> At the time, we were just trying to get the RST transition done, and
>> swapping out the kernel-doc script seemed like a major distraction that
>> we didn't need, so this never got looked at as seriously as I would have
>> liked.
>
> Personally, I'd like to see us switch over to
> https://github.com/jnikula/hawkmoth
>
> but I don't have time to work on the rough edges.

Basically Hawkmoth is my idea how C documentation comments should be
incorporated to Sphinx *if* there is no legacy to care about. In that
sense, I never wrote it with the kernel documentation in mind, on the
contrary it's a hobby project where I can just not think about the
kernel at all. ;)

With that in mind, I'd really like to hear what the rough edges are that
you see. Or are they kernel specific? Preferrably as issues on the
project page if you don't mind.

> I really hate the kernel-doc style; I think it makes us write very
> stilted documentation, full of parameter descriptions like:
>
>  function() - Do the thing to a page.
>  @page: The page.
>
> which really serves nobody.  Being able to write:
>
>  function() - Do the thing to @page
>
> is easier to both write and read.

I tend to agree. Though Hawkmoth does not take a stand here, basically
the assumption is that the documentation comments are pure rst. And that
was one of the goals, no parsing of the comments beyond removing the C
comment markers. It's up to the users or projects to dictate what the
comments should look like. There's some plugin support for users to add
their own filters, e.g. there could be kernel-doc format support via
that.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Graphics Center