docs: automarkup.py

docs: automarkup.py

[Randy Dunlap]

Hi,

Can anyone explain this?  maybe even suggest a fix for it?

This has been around for a few weeks AFAIK. I haven't see a patch for it,
but I could have missed it.

(since 17e02586ed185 in August/2023; oh, that just fixed the move
of files to the Documentation/arch/ subdir, so maybe even longer)


In file Documentation/ABI/testing/sysfs-bus-papr-pmem:

		response to H_SCM_HEALTH hcall. The details of the bit
		flags returned in response to this hcall is available
		at 'Documentation/arch/powerpc/papr_hcalls.rst'. Below are
		the flags reported in this sysfs file:

kernel-doc reports:

linux-next-20231222/Documentation/ABI/testing/sysfs-bus-papr-pmem:2: WARNING: unknown document: '/powerpc/papr_hcalls'

and the output file Documentation/output/admin-guide/abi-testing.html says:

response to H_SCM_HEALTH hcall. The details of the bit
flags returned in response to this hcall is available
at '<span class="xref std std-doc">/powerpc/papr_hcalls</span>' . Below are
the flags reported in this sysfs file:</p>


so the leading "Documentation/arch" is being removed from the filename
AFAICT.

I tried changing the quoted filename from single quotes to double back quotes
`` and I tried it without any quotes, all with the same results.

-- 
#Randy

Re: docs: automarkup.py

[Vegard Nossum]

On 27/12/2023 08:55, Randy Dunlap wrote:
> Can anyone explain this?  maybe even suggest a fix for it?
> 
> This has been around for a few weeks AFAIK. I haven't see a patch for it,
> but I could have missed it.
> 
> (since 17e02586ed185 in August/2023; oh, that just fixed the move
> of files to the Documentation/arch/ subdir, so maybe even longer)
> 
> 
> In file Documentation/ABI/testing/sysfs-bus-papr-pmem:
> 
> 		response to H_SCM_HEALTH hcall. The details of the bit
> 		flags returned in response to this hcall is available
> 		at 'Documentation/arch/powerpc/papr_hcalls.rst'. Below are
> 		the flags reported in this sysfs file:
> 
> kernel-doc reports:
> 
> linux-next-20231222/Documentation/ABI/testing/sysfs-bus-papr-pmem:2: WARNING: unknown document: '/powerpc/papr_hcalls'
> 
> and the output file Documentation/output/admin-guide/abi-testing.html says:
> 
> response to H_SCM_HEALTH hcall. The details of the bit
> flags returned in response to this hcall is available
> at '<span class="xref std std-doc">/powerpc/papr_hcalls</span>' . Below are
> the flags reported in this sysfs file:</p>
> 
> 
> so the leading "Documentation/arch" is being removed from the filename
> AFAICT.
> 
> I tried changing the quoted filename from single quotes to double back quotes
> `` and I tried it without any quotes, all with the same results.
> 

I don't see that here, there is no warning and it renders properly.

If you go on https://docs.kernel.org/admin-guide/abi-testing.html then
it says 6.7.0-rc7 and (AFAICT) it also links/renders properly.

Maybe try building in a fresh clone/worktree just to verify there isn't
some old file somewhere that didn't get cleaned out/updated?


Vegard

Re: docs: automarkup.py

[Mauro Carvalho Chehab]

Em Tue, 26 Dec 2023 23:55:26 -0800
Randy Dunlap <rdunlap@infradead.org> escreveu:

> Hi,
> 
> Can anyone explain this?  maybe even suggest a fix for it?
> 
> This has been around for a few weeks AFAIK. I haven't see a patch for it,
> but I could have missed it.
> 
> (since 17e02586ed185 in August/2023; oh, that just fixed the move
> of files to the Documentation/arch/ subdir, so maybe even longer)
> 
> 
> In file Documentation/ABI/testing/sysfs-bus-papr-pmem:
> 
> 		response to H_SCM_HEALTH hcall. The details of the bit
> 		flags returned in response to this hcall is available
> 		at 'Documentation/arch/powerpc/papr_hcalls.rst'. Below are
> 		the flags reported in this sysfs file:
> 
> kernel-doc reports:
> 
> linux-next-20231222/Documentation/ABI/testing/sysfs-bus-papr-pmem:2: WARNING: unknown document: '/powerpc/papr_hcalls'
> 
> and the output file Documentation/output/admin-guide/abi-testing.html says:
> 
> response to H_SCM_HEALTH hcall. The details of the bit
> flags returned in response to this hcall is available
> at '<span class="xref std std-doc">/powerpc/papr_hcalls</span>' . Below are
> the flags reported in this sysfs file:</p>

With sphinx-build version 6.2.1 and latest linux-next, I'm getting:

<snip>
<p>Defined on file <a class="reference internal" href="#abi-file-testing-sysfs-bus-papr-pmem"><span class="std std-ref">sysfs-bus-papr-pmem</span></a></p>
<p>(RO) Report flags indicating various states of a
papr-pmem NVDIMM device. Each flag maps to a one or
more bits set in the dimm-health-bitmap retrieved in
response to H_SCM_HEALTH hcall. The details of the bit
flags returned in response to this hcall is available
at '<a class="reference internal" href="../arch/powerpc/papr_hcalls.html"><span class="doc">Hypercall Op-codes (hcalls)</span></a>' .
</snip>

It seems that references are properly evaluated there.

> so the leading "Documentation/arch" is being removed from the filename
> AFAICT.

Actually, this is not related to automarkup.py, but, instead to
get_abi automation. You can see how this is processed by running
get_abi.pl script, e. g.:

<snip>
$ ./scripts/get_abi.pl search nmemX/papr/flags

/sys/bus/nd/devices/nmemX/papr/flags
------------------------------------

Kernel version:		v5.8
Date:			Apr, 2020
Contact:		linuxppc-dev <linuxppc-dev@lists.ozlabs.org>, nvdimm@lists.linux.dev,
Defined on file(s):	Documentation/ABI/testing/sysfs-bus-papr-pmem

Description:

(RO) Report flags indicating various states of a
papr-pmem NVDIMM device. Each flag maps to a one or
more bits set in the dimm-health-bitmap retrieved in
response to H_SCM_HEALTH hcall. The details of the bit
flags returned in response to this hcall is available
at 'Documentation/arch/powerpc/papr_hcalls.rst' . Below are
the flags reported in this sysfs file:

* "not_armed"
                  Indicates that NVDIMM contents will not
                  survive a power cycle.
* "flush_fail"
                  Indicates that NVDIMM contents
                  couldn't be flushed during last
                  shut-down event.
* "restore_fail"
                  Indicates that NVDIMM contents
                  couldn't be restored during NVDIMM
                  initialization.
* "encrypted"
                  NVDIMM contents are encrypted.
* "smart_notify"
                  There is health event for the NVDIMM.
* "scrubbed"
                  Indicating that contents of the
                  NVDIMM have been scrubbed.
* "locked"
                  Indicating that NVDIMM contents can't
                  be modified until next power cycle.
</snip>

The output there is not in ReST format. There is a "rest" parameter
to generate the version actually used by Sphinx.

When "./scripts/get_abi.pl rest" is used, it converts any reference
of Documentation/xxx.rst into[1]:

	:doc:`/xxx`

Which is actually a relative link, pointing to the file at:

	<doc_source_dir>/xxx.rst

The references there are relative to the doc root directory passed
to sphinx-build (Documentation/). So, the above shall create a
cross-reference to Documentation/xxx.rst using the document title as the
name of the reference, so this will become:

	<a class="reference internal" href="../arch/powerpc/papr_hcalls.html"><span class="doc">Hypercall Op-codes (hcalls)</span></a>

[1] see https://docs.readthedocs.io/en/stable/guides/cross-referencing-with-sphinx.html#the-doc-role

See how a similar link is converted to ReST format:

<snip>
$ ./scripts/get_abi.pl rest |grep -A20 /mte_tcf_preferred
Warning: file Documentation/ABI/testing/sysfs-platform-silicom#20:
	What '/sys/devices/platform/silicom-platform/power_cycle' doesn't have a description
| **\/sys\/devices\/system\/cpu\/cpuX\/mte_tcf_preferred** |
+----------------------------------------------------------+

Defined on file :ref:`sysfs-devices-system-cpu <abi_file_testing_sysfs_devices_system_cpu>`

Preferred MTE tag checking mode

When a user program specifies more than one MTE tag checking
mode, this sysfs node is used to specify which mode should
be preferred when scheduling a task on that CPU. Possible
values:

================  ==============================================
"sync"            Prefer synchronous mode
"asymm"           Prefer asymmetric mode
"async"           Prefer asynchronous mode
================  ==============================================

See also: :doc:`/arch/arm64/memory-tagging-extension`
<snip>

> I tried changing the quoted filename from single quotes to double back quotes
> `` and I tried it without any quotes, all with the same results.

With a quote like "`", the above would be evaluated to:
	
See also: `:doc:`/arch/arm64/memory-tagging-extension``

which will probably cause problems when sphinx parses it.

That's said, I remember that some old Sphinx versions had issues 
with the doc: tag. On some legacy versions, the <doc_source_dir>
is set in a wrong way, ignoring the path used at the sphinx-build
directory parameter.

Thanks,
Mauro

Re: docs: automarkup.py

[Randy Dunlap]

Hi Vegard and Mauro,

Thanks for your assistance.
(more below)


On 12/27/23 01:08, Vegard Nossum wrote:
> 
> On 27/12/2023 08:55, Randy Dunlap wrote:
>> Can anyone explain this?  maybe even suggest a fix for it?
>>
>> This has been around for a few weeks AFAIK. I haven't see a patch for it,
>> but I could have missed it.
>>
>> (since 17e02586ed185 in August/2023; oh, that just fixed the move
>> of files to the Documentation/arch/ subdir, so maybe even longer)
>>
>>
>> In file Documentation/ABI/testing/sysfs-bus-papr-pmem:
>>
>>         response to H_SCM_HEALTH hcall. The details of the bit
>>         flags returned in response to this hcall is available
>>         at 'Documentation/arch/powerpc/papr_hcalls.rst'. Below are
>>         the flags reported in this sysfs file:
>>
>> kernel-doc reports:
>>
>> linux-next-20231222/Documentation/ABI/testing/sysfs-bus-papr-pmem:2: WARNING: unknown document: '/powerpc/papr_hcalls'
>>
>> and the output file Documentation/output/admin-guide/abi-testing.html says:
>>
>> response to H_SCM_HEALTH hcall. The details of the bit
>> flags returned in response to this hcall is available
>> at '<span class="xref std std-doc">/powerpc/papr_hcalls</span>' . Below are
>> the flags reported in this sysfs file:</p>
>>
>>
>> so the leading "Documentation/arch" is being removed from the filename
>> AFAICT.
>>
>> I tried changing the quoted filename from single quotes to double back quotes
>> `` and I tried it without any quotes, all with the same results.
>>
> 
> I don't see that here, there is no warning and it renders properly.
> 
> If you go on https://docs.kernel.org/admin-guide/abi-testing.html then
> it says 6.7.0-rc7 and (AFAICT) it also links/renders properly.

Yes, I probably should have checked there earlier. :)

> Maybe try building in a fresh clone/worktree just to verify there isn't
> some old file somewhere that didn't get cleaned out/updated?

That does work but it made me suspicious.

I was building in a kernel tree that was built from a tarball
and linux-next daily patches.  That leaves behind some *.orig files
(since I use 'patch -b' for "backups").

If I remove the Documentation/ABI/*.orig files, there is no issue
like I had erroneously reported here.
Maybe get_abi.pl is (also) processing the *.orig files and that
somehow causes it to produce the confusing output.

Anyway, nothing to see here. Move along. :)

Thanks again.

-- 
#Randy

Re: docs: automarkup.py

[Mauro Carvalho Chehab]

Em Wed, 27 Dec 2023 14:20:05 -0800
Randy Dunlap <rdunlap@infradead.org> escreveu:

> Hi Vegard and Mauro,
> 
> Thanks for your assistance.
> (more below)
> 
> 
> On 12/27/23 01:08, Vegard Nossum wrote:
> > 
> > On 27/12/2023 08:55, Randy Dunlap wrote:  
> >> Can anyone explain this?  maybe even suggest a fix for it?
> >>
> >> This has been around for a few weeks AFAIK. I haven't see a patch for it,
> >> but I could have missed it.
> >>
> >> (since 17e02586ed185 in August/2023; oh, that just fixed the move
> >> of files to the Documentation/arch/ subdir, so maybe even longer)
> >>
> >>
> >> In file Documentation/ABI/testing/sysfs-bus-papr-pmem:
> >>
> >>         response to H_SCM_HEALTH hcall. The details of the bit
> >>         flags returned in response to this hcall is available
> >>         at 'Documentation/arch/powerpc/papr_hcalls.rst'. Below are
> >>         the flags reported in this sysfs file:
> >>
> >> kernel-doc reports:
> >>
> >> linux-next-20231222/Documentation/ABI/testing/sysfs-bus-papr-pmem:2: WARNING: unknown document: '/powerpc/papr_hcalls'
> >>
> >> and the output file Documentation/output/admin-guide/abi-testing.html says:
> >>
> >> response to H_SCM_HEALTH hcall. The details of the bit
> >> flags returned in response to this hcall is available
> >> at '<span class="xref std std-doc">/powerpc/papr_hcalls</span>' . Below are
> >> the flags reported in this sysfs file:</p>
> >>
> >>
> >> so the leading "Documentation/arch" is being removed from the filename
> >> AFAICT.
> >>
> >> I tried changing the quoted filename from single quotes to double back quotes
> >> `` and I tried it without any quotes, all with the same results.
> >>  
> > 
> > I don't see that here, there is no warning and it renders properly.
> > 
> > If you go on https://docs.kernel.org/admin-guide/abi-testing.html then
> > it says 6.7.0-rc7 and (AFAICT) it also links/renders properly.  
> 
> Yes, I probably should have checked there earlier. :)
> 
> > Maybe try building in a fresh clone/worktree just to verify there isn't
> > some old file somewhere that didn't get cleaned out/updated?  
> 
> That does work but it made me suspicious.
> 
> I was building in a kernel tree that was built from a tarball
> and linux-next daily patches.  That leaves behind some *.orig files
> (since I use 'patch -b' for "backups").
> 
> If I remove the Documentation/ABI/*.orig files, there is no issue
> like I had erroneously reported here.
> Maybe get_abi.pl is (also) processing the *.orig files and that
> somehow causes it to produce the confusing output.

The logic which parse files at get_abi.pl is this one:

    sub parse_abi {
        my $file = $File::Find::name;

        my $mode = (stat($file))[2];
        return if ($mode & S_IFDIR);
        return if ($file =~ m,/README,);

Currently, it ignores just README and directories.

It wouldn't be hard to add something like:

	return if ($file =~ m,/\.(rej|org|orig|bak)$,);

to ignore certain extensions. On such case, we would need to explicitly
add the extensions to be Ignored.

We could, instead, do a broader regex:

	return if ($file =~ m,/\.,);

to ignore everything that would have a dot at the file name, but that
could be problematic if we ever end adding files with extensions
there.

Regards,
Mauro

Re: docs: automarkup.py

[Randy Dunlap]

On 12/28/23 08:22, Mauro Carvalho Chehab wrote:
> Em Wed, 27 Dec 2023 14:20:05 -0800
> Randy Dunlap <rdunlap@infradead.org> escreveu:
> 
>> Hi Vegard and Mauro,
>>
>> Thanks for your assistance.
>> (more below)
>>
>>
>> On 12/27/23 01:08, Vegard Nossum wrote:
>>>
>>> On 27/12/2023 08:55, Randy Dunlap wrote:  
>>>> Can anyone explain this?  maybe even suggest a fix for it?
>>>>
>>>> This has been around for a few weeks AFAIK. I haven't see a patch for it,
>>>> but I could have missed it.
>>>>
>>>> (since 17e02586ed185 in August/2023; oh, that just fixed the move
>>>> of files to the Documentation/arch/ subdir, so maybe even longer)
>>>>
>>>>
>>>> In file Documentation/ABI/testing/sysfs-bus-papr-pmem:
>>>>
>>>>         response to H_SCM_HEALTH hcall. The details of the bit
>>>>         flags returned in response to this hcall is available
>>>>         at 'Documentation/arch/powerpc/papr_hcalls.rst'. Below are
>>>>         the flags reported in this sysfs file:
>>>>
>>>> kernel-doc reports:
>>>>
>>>> linux-next-20231222/Documentation/ABI/testing/sysfs-bus-papr-pmem:2: WARNING: unknown document: '/powerpc/papr_hcalls'
>>>>
>>>> and the output file Documentation/output/admin-guide/abi-testing.html says:
>>>>
>>>> response to H_SCM_HEALTH hcall. The details of the bit
>>>> flags returned in response to this hcall is available
>>>> at '<span class="xref std std-doc">/powerpc/papr_hcalls</span>' . Below are
>>>> the flags reported in this sysfs file:</p>
>>>>
>>>>
>>>> so the leading "Documentation/arch" is being removed from the filename
>>>> AFAICT.
>>>>
>>>> I tried changing the quoted filename from single quotes to double back quotes
>>>> `` and I tried it without any quotes, all with the same results.
>>>>  
>>>
>>> I don't see that here, there is no warning and it renders properly.
>>>
>>> If you go on https://docs.kernel.org/admin-guide/abi-testing.html then
>>> it says 6.7.0-rc7 and (AFAICT) it also links/renders properly.  
>>
>> Yes, I probably should have checked there earlier. :)
>>
>>> Maybe try building in a fresh clone/worktree just to verify there isn't
>>> some old file somewhere that didn't get cleaned out/updated?  
>>
>> That does work but it made me suspicious.
>>
>> I was building in a kernel tree that was built from a tarball
>> and linux-next daily patches.  That leaves behind some *.orig files
>> (since I use 'patch -b' for "backups").
>>
>> If I remove the Documentation/ABI/*.orig files, there is no issue
>> like I had erroneously reported here.
>> Maybe get_abi.pl is (also) processing the *.orig files and that
>> somehow causes it to produce the confusing output.
> 
> The logic which parse files at get_abi.pl is this one:
> 
>     sub parse_abi {
>         my $file = $File::Find::name;
> 
>         my $mode = (stat($file))[2];
>         return if ($mode & S_IFDIR);
>         return if ($file =~ m,/README,);
> 
> Currently, it ignores just README and directories.
> 
> It wouldn't be hard to add something like:
> 
> 	return if ($file =~ m,/\.(rej|org|orig|bak)$,);

That didn't quite work as is. I changed it to this:

+       return if ($file =~ m,\.(rej|org|orig|bak)$,);

which works (dropping the leading "/").

> to ignore certain extensions. On such case, we would need to explicitly
> add the extensions to be Ignored.
> 
> We could, instead, do a broader regex:
> 
> 	return if ($file =~ m,/\.,);

That line is already present in the script, for skipping
hidden (dot) files.

It does not skip filenames like "papr_hcalls.orig" -- at least I
still see this problem when that line is present.

I wouldn't suggest skipping any file that has a '.' in its name.

> 
> to ignore everything that would have a dot at the file name, but that
> could be problematic if we ever end adding files with extensions
> there.
> 
> Regards,
> Mauro
> 

Thanks.
-- 
#Randy

Re: docs: automarkup.py

[Mauro Carvalho Chehab]

Em Thu, 28 Dec 2023 12:01:00 -0800
Randy Dunlap <rdunlap@infradead.org> escreveu:

> On 12/28/23 08:22, Mauro Carvalho Chehab wrote:
> > Em Wed, 27 Dec 2023 14:20:05 -0800
> > Randy Dunlap <rdunlap@infradead.org> escreveu:
> >   
> >> Hi Vegard and Mauro,
> >>
> >> Thanks for your assistance.
> >> (more below)
> >>
> >>
> >> On 12/27/23 01:08, Vegard Nossum wrote:  
> >>>
> >>> On 27/12/2023 08:55, Randy Dunlap wrote:    
> >>>> Can anyone explain this?  maybe even suggest a fix for it?
> >>>>
> >>>> This has been around for a few weeks AFAIK. I haven't see a patch for it,
> >>>> but I could have missed it.
> >>>>
> >>>> (since 17e02586ed185 in August/2023; oh, that just fixed the move
> >>>> of files to the Documentation/arch/ subdir, so maybe even longer)
> >>>>
> >>>>
> >>>> In file Documentation/ABI/testing/sysfs-bus-papr-pmem:
> >>>>
> >>>>         response to H_SCM_HEALTH hcall. The details of the bit
> >>>>         flags returned in response to this hcall is available
> >>>>         at 'Documentation/arch/powerpc/papr_hcalls.rst'. Below are
> >>>>         the flags reported in this sysfs file:
> >>>>
> >>>> kernel-doc reports:
> >>>>
> >>>> linux-next-20231222/Documentation/ABI/testing/sysfs-bus-papr-pmem:2: WARNING: unknown document: '/powerpc/papr_hcalls'
> >>>>
> >>>> and the output file Documentation/output/admin-guide/abi-testing.html says:
> >>>>
> >>>> response to H_SCM_HEALTH hcall. The details of the bit
> >>>> flags returned in response to this hcall is available
> >>>> at '<span class="xref std std-doc">/powerpc/papr_hcalls</span>' . Below are
> >>>> the flags reported in this sysfs file:</p>
> >>>>
> >>>>
> >>>> so the leading "Documentation/arch" is being removed from the filename
> >>>> AFAICT.
> >>>>
> >>>> I tried changing the quoted filename from single quotes to double back quotes
> >>>> `` and I tried it without any quotes, all with the same results.
> >>>>    
> >>>
> >>> I don't see that here, there is no warning and it renders properly.
> >>>
> >>> If you go on https://docs.kernel.org/admin-guide/abi-testing.html then
> >>> it says 6.7.0-rc7 and (AFAICT) it also links/renders properly.    
> >>
> >> Yes, I probably should have checked there earlier. :)
> >>  
> >>> Maybe try building in a fresh clone/worktree just to verify there isn't
> >>> some old file somewhere that didn't get cleaned out/updated?    
> >>
> >> That does work but it made me suspicious.
> >>
> >> I was building in a kernel tree that was built from a tarball
> >> and linux-next daily patches.  That leaves behind some *.orig files
> >> (since I use 'patch -b' for "backups").
> >>
> >> If I remove the Documentation/ABI/*.orig files, there is no issue
> >> like I had erroneously reported here.
> >> Maybe get_abi.pl is (also) processing the *.orig files and that
> >> somehow causes it to produce the confusing output.  
> > 
> > The logic which parse files at get_abi.pl is this one:
> > 
> >     sub parse_abi {
> >         my $file = $File::Find::name;
> > 
> >         my $mode = (stat($file))[2];
> >         return if ($mode & S_IFDIR);
> >         return if ($file =~ m,/README,);
> > 
> > Currently, it ignores just README and directories.
> > 
> > It wouldn't be hard to add something like:
> > 
> > 	return if ($file =~ m,/\.(rej|org|orig|bak)$,);  
> 
> That didn't quite work as is. I changed it to this:
> 
> +       return if ($file =~ m,\.(rej|org|orig|bak)$,);
> 
> which works (dropping the leading "/").

Great! If you want, feel free to submit a patch with that with
my Acked-by.

> 
> > to ignore certain extensions. On such case, we would need to explicitly
> > add the extensions to be Ignored.
> > 
> > We could, instead, do a broader regex:
> > 
> > 	return if ($file =~ m,/\.,);  
> 
> That line is already present in the script, for skipping
> hidden (dot) files.
> 
> It does not skip filenames like "papr_hcalls.orig" -- at least I
> still see this problem when that line is present.
> 
> I wouldn't suggest skipping any file that has a '.' in its name.

Yeah, agreed.

Regards,
Mauro