[RFC] doc: fix code snippet build warnings

[RFC] doc: fix code snippet build warnings

[Tobin C. Harding]

Posting as RFC in the hope that someone knows how to massage sphinx
correctly to fix this patch.

Currently function kernel-doc contains a multi-line code snippet. This
is causing sphinx to emit 5 build warnings

	WARNING: Unexpected indentation.
	WARNING: Unexpected indentation.
	WARNING: Block quote ends without a blank line; unexpected unindent.
	WARNING: Block quote ends without a blank line; unexpected unindent.
	WARNING: Inline literal start-string without end-string.

And the snippet is not rendering correctly in HTML.

We can stop shpinx complaining by using '::' instead of the currently
used '``' however this still does not render correctly in HTML. The
rendering is [arguably] better but still incorrect. Sphinx renders two
function calls thus:

	:c:func:`rcu_read_lock()`;

The rest of the snippet does however have correct spacing.

Use '::' to pre-fix code snippet. Clears build warnings but does not
render correctly.

Signed-off-by: Tobin C. Harding <me@tobin.cc>
---

To view current broken rendering see

https://www.kernel.org/doc/html/latest/core-api/kernel-api.html?highlight=rcu_pointer_handoff#c.rcu_pointer_handoff

 include/linux/rcupdate.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index a6ddc42f87a5..cc10e772e3e9 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -568,7 +568,8 @@ static inline void rcu_preempt_sleep_check(void) { }
  * is handed off from RCU to some other synchronization mechanism, for
  * example, reference counting or locking.  In C11, it would map to
  * kill_dependency().  It could be used as follows:
- * ``
+ * ::
+ *
  *	rcu_read_lock();
  *	p = rcu_dereference(gp);
  *	long_lived = is_long_lived(p);
@@ -579,7 +580,6 @@ static inline void rcu_preempt_sleep_check(void) { }
  *			p = rcu_pointer_handoff(p);
  *	}
  *	rcu_read_unlock();
- *``
  */
 #define rcu_pointer_handoff(p) (p)
 
-- 
2.7.4

Re: [RFC] doc: fix code snippet build warnings

[Paul E. McKenney]

On Wed, Jan 10, 2018 at 03:04:53PM +1100, Tobin C. Harding wrote:
> Posting as RFC in the hope that someone knows how to massage sphinx
> correctly to fix this patch.

I would welcome that.  ;-)

> Currently function kernel-doc contains a multi-line code snippet. This
> is causing sphinx to emit 5 build warnings
> 
> 	WARNING: Unexpected indentation.
> 	WARNING: Unexpected indentation.
> 	WARNING: Block quote ends without a blank line; unexpected unindent.
> 	WARNING: Block quote ends without a blank line; unexpected unindent.
> 	WARNING: Inline literal start-string without end-string.
> 
> And the snippet is not rendering correctly in HTML.
> 
> We can stop shpinx complaining by using '::' instead of the currently
> used '``' however this still does not render correctly in HTML. The
> rendering is [arguably] better but still incorrect. Sphinx renders two
> function calls thus:
> 
> 	:c:func:`rcu_read_lock()`;
> 
> The rest of the snippet does however have correct spacing.
> 
> Use '::' to pre-fix code snippet. Clears build warnings but does not
> render correctly.

If the usual docbook suspects ack this, I would be happy to carry it.

Cue debate over silent vs. noisy errors.  ;-)

							Thanx, Paul

> Signed-off-by: Tobin C. Harding <me@tobin.cc>
> ---
> 
> To view current broken rendering see
> 
> https://www.kernel.org/doc/html/latest/core-api/kernel-api.html?highlight=rcu_pointer_handoff#c.rcu_pointer_handoff
> 
>  include/linux/rcupdate.h | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> index a6ddc42f87a5..cc10e772e3e9 100644
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -568,7 +568,8 @@ static inline void rcu_preempt_sleep_check(void) { }
>   * is handed off from RCU to some other synchronization mechanism, for
>   * example, reference counting or locking.  In C11, it would map to
>   * kill_dependency().  It could be used as follows:
> - * ``
> + * ::
> + *
>   *	rcu_read_lock();
>   *	p = rcu_dereference(gp);
>   *	long_lived = is_long_lived(p);
> @@ -579,7 +580,6 @@ static inline void rcu_preempt_sleep_check(void) { }
>   *			p = rcu_pointer_handoff(p);
>   *	}
>   *	rcu_read_unlock();
> - *``
>   */
>  #define rcu_pointer_handoff(p) (p)
> 
> -- 
> 2.7.4
> 

Re: [RFC] doc: fix code snippet build warnings

[Tobin C. Harding]

On Wed, Jan 10, 2018 at 08:37:02AM -0800, Paul E. McKenney wrote:
> On Wed, Jan 10, 2018 at 03:04:53PM +1100, Tobin C. Harding wrote:
> > Posting as RFC in the hope that someone knows how to massage sphinx
> > correctly to fix this patch.
> 
> I would welcome that.  ;-)
> 
> > Currently function kernel-doc contains a multi-line code snippet. This
> > is causing sphinx to emit 5 build warnings
> > 
> > 	WARNING: Unexpected indentation.
> > 	WARNING: Unexpected indentation.
> > 	WARNING: Block quote ends without a blank line; unexpected unindent.
> > 	WARNING: Block quote ends without a blank line; unexpected unindent.
> > 	WARNING: Inline literal start-string without end-string.
> > 
> > And the snippet is not rendering correctly in HTML.
> > 
> > We can stop shpinx complaining by using '::' instead of the currently
> > used '``' however this still does not render correctly in HTML. The
> > rendering is [arguably] better but still incorrect. Sphinx renders two
> > function calls thus:
> > 
> > 	:c:func:`rcu_read_lock()`;
> > 
> > The rest of the snippet does however have correct spacing.
> > 
> > Use '::' to pre-fix code snippet. Clears build warnings but does not
> > render correctly.
> 
> If the usual docbook suspects ack this, I would be happy to carry it.
> 
> Cue debate over silent vs. noisy errors.  ;-)

Besides making me laugh out loud I did not think of this issue while
patching. FWIW, now you have mentioned it, I favour noisy errors :)

thanks,
Tobin.

Re: [RFC] doc: fix code snippet build warnings

[Paul E. McKenney]

On Thu, Jan 11, 2018 at 07:30:07AM +1100, Tobin C. Harding wrote:
> On Wed, Jan 10, 2018 at 08:37:02AM -0800, Paul E. McKenney wrote:
> > On Wed, Jan 10, 2018 at 03:04:53PM +1100, Tobin C. Harding wrote:
> > > Posting as RFC in the hope that someone knows how to massage sphinx
> > > correctly to fix this patch.
> > 
> > I would welcome that.  ;-)
> > 
> > > Currently function kernel-doc contains a multi-line code snippet. This
> > > is causing sphinx to emit 5 build warnings
> > > 
> > > 	WARNING: Unexpected indentation.
> > > 	WARNING: Unexpected indentation.
> > > 	WARNING: Block quote ends without a blank line; unexpected unindent.
> > > 	WARNING: Block quote ends without a blank line; unexpected unindent.
> > > 	WARNING: Inline literal start-string without end-string.
> > > 
> > > And the snippet is not rendering correctly in HTML.
> > > 
> > > We can stop shpinx complaining by using '::' instead of the currently
> > > used '``' however this still does not render correctly in HTML. The
> > > rendering is [arguably] better but still incorrect. Sphinx renders two
> > > function calls thus:
> > > 
> > > 	:c:func:`rcu_read_lock()`;
> > > 
> > > The rest of the snippet does however have correct spacing.
> > > 
> > > Use '::' to pre-fix code snippet. Clears build warnings but does not
> > > render correctly.
> > 
> > If the usual docbook suspects ack this, I would be happy to carry it.
> > 
> > Cue debate over silent vs. noisy errors.  ;-)
> 
> Besides making me laugh out loud I did not think of this issue while
> patching. FWIW, now you have mentioned it, I favour noisy errors :)

Very good, will leave this one alone, then.  ;-)

							Thanx, Paul

Re: [RFC] doc: fix code snippet build warnings

[Jonathan Corbet]

On Wed, 10 Jan 2018 15:04:53 +1100
"Tobin C. Harding" <me@tobin.cc> wrote:

> Posting as RFC in the hope that someone knows how to massage sphinx
> correctly to fix this patch.
> 
> Currently function kernel-doc contains a multi-line code snippet. This
> is causing sphinx to emit 5 build warnings
> 
> 	WARNING: Unexpected indentation.
> 	WARNING: Unexpected indentation.
> 	WARNING: Block quote ends without a blank line; unexpected unindent.
> 	WARNING: Block quote ends without a blank line; unexpected unindent.
> 	WARNING: Inline literal start-string without end-string.
> 
> And the snippet is not rendering correctly in HTML.
> 
> We can stop shpinx complaining by using '::' instead of the currently
> used '``' however this still does not render correctly in HTML. The
> rendering is [arguably] better but still incorrect. Sphinx renders two
> function calls thus:
> 
> 	:c:func:`rcu_read_lock()`;
> 
> The rest of the snippet does however have correct spacing.

The behavior when `` was used is not surprising, that really just does a
font change.  Once you went with a literal block (with "::") though, the
situation changes a bit.  That really should work.

I looked a bit.  This isn't a sphinx (or "shpinx" :) problem, the bug is
in kernel-doc.  Once we go into the literal mode, it shouldn't be
screwing around with the text anymore.  Of course, kernel-doc doesn't
understand enough RST to know that.  I'm a little nervous about trying to
teach it more, but maybe we have to do that; we should certainly be able
to put code snippets into the docs and have them come through unmolested.

I'll try to look more closely at that shortly.  Meanwhile, this patch
makes things better than the were before.  That said...

> Use '::' to pre-fix code snippet. Clears build warnings but does not
> render correctly.
> 
> Signed-off-by: Tobin C. Harding <me@tobin.cc>
> ---
> 
> To view current broken rendering see
> 
> https://www.kernel.org/doc/html/latest/core-api/kernel-api.html?highlight=rcu_pointer_handoff#c.rcu_pointer_handoff
> 
>  include/linux/rcupdate.h | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> index a6ddc42f87a5..cc10e772e3e9 100644
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -568,7 +568,8 @@ static inline void rcu_preempt_sleep_check(void) { }
>   * is handed off from RCU to some other synchronization mechanism, for
>   * example, reference counting or locking.  In C11, it would map to
>   * kill_dependency().  It could be used as follows:
> - * ``
> + * ::

...rather than adding a separate "::" line, you can just
s/follows:/follows::/ and the Right Thing will happen (to the same extent
that it does now, anyway.

Thanks,

jon

Re: [RFC] doc: fix code snippet build warnings

[Tobin C. Harding]

On Wed, Jan 10, 2018 at 02:59:58PM -0700, Jonathan Corbet wrote:
> On Wed, 10 Jan 2018 15:04:53 +1100
> "Tobin C. Harding" <me@tobin.cc> wrote:
> 
> > Posting as RFC in the hope that someone knows how to massage sphinx
> > correctly to fix this patch.
> > 
> > Currently function kernel-doc contains a multi-line code snippet. This
> > is causing sphinx to emit 5 build warnings
> > 
> > 	WARNING: Unexpected indentation.
> > 	WARNING: Unexpected indentation.
> > 	WARNING: Block quote ends without a blank line; unexpected unindent.
> > 	WARNING: Block quote ends without a blank line; unexpected unindent.
> > 	WARNING: Inline literal start-string without end-string.
> > 
> > And the snippet is not rendering correctly in HTML.
> > 
> > We can stop shpinx complaining by using '::' instead of the currently
> > used '``' however this still does not render correctly in HTML. The
> > rendering is [arguably] better but still incorrect. Sphinx renders two
> > function calls thus:
> > 
> > 	:c:func:`rcu_read_lock()`;
> > 
> > The rest of the snippet does however have correct spacing.
> 
> The behavior when `` was used is not surprising, that really just does a
> font change.  Once you went with a literal block (with "::") though, the
> situation changes a bit.  That really should work.
> 
> I looked a bit.  This isn't a sphinx (or "shpinx" :) problem, the bug is
> in kernel-doc.  Once we go into the literal mode, it shouldn't be
> screwing around with the text anymore.  Of course, kernel-doc doesn't
> understand enough RST to know that.  I'm a little nervous about trying to
> teach it more, but maybe we have to do that; we should certainly be able
> to put code snippets into the docs and have them come through unmolested.
> 
> I'll try to look more closely at that shortly.  Meanwhile, this patch
> makes things better than the were before.  That said...
> 
> > Use '::' to pre-fix code snippet. Clears build warnings but does not
> > render correctly.
> > 
> > Signed-off-by: Tobin C. Harding <me@tobin.cc>
> > ---
> > 
> > To view current broken rendering see
> > 
> > https://www.kernel.org/doc/html/latest/core-api/kernel-api.html?highlight=rcu_pointer_handoff#c.rcu_pointer_handoff
> > 
> >  include/linux/rcupdate.h | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> > index a6ddc42f87a5..cc10e772e3e9 100644
> > --- a/include/linux/rcupdate.h
> > +++ b/include/linux/rcupdate.h
> > @@ -568,7 +568,8 @@ static inline void rcu_preempt_sleep_check(void) { }
> >   * is handed off from RCU to some other synchronization mechanism, for
> >   * example, reference counting or locking.  In C11, it would map to
> >   * kill_dependency().  It could be used as follows:
> > - * ``
> > + * ::
> 
> ...rather than adding a separate "::" line, you can just
> s/follows:/follows::/ and the Right Thing will happen (to the same extent
> that it does now, anyway.

Except that it will render as

	kill_dependency().  It could be used as follows

instead of
	kill_dependency().  It could be used as follows:

(note: final colon)

Also the diff will be bigger.  These two reasons led me to the patch as
it is.  I'm happy to re-spin with your suggested change though.

thanks,
Tobin.

Re: [RFC] doc: fix code snippet build warnings

[Tobin C. Harding]

On Thu, Jan 11, 2018 at 09:25:31AM +1100, Tobin C. Harding wrote:
> On Wed, Jan 10, 2018 at 02:59:58PM -0700, Jonathan Corbet wrote:
> > On Wed, 10 Jan 2018 15:04:53 +1100
> > "Tobin C. Harding" <me@tobin.cc> wrote:
> > 
> > > Posting as RFC in the hope that someone knows how to massage sphinx
> > > correctly to fix this patch.
> > > 
> > > Currently function kernel-doc contains a multi-line code snippet. This
> > > is causing sphinx to emit 5 build warnings
> > > 
> > > 	WARNING: Unexpected indentation.
> > > 	WARNING: Unexpected indentation.
> > > 	WARNING: Block quote ends without a blank line; unexpected unindent.
> > > 	WARNING: Block quote ends without a blank line; unexpected unindent.
> > > 	WARNING: Inline literal start-string without end-string.
> > > 
> > > And the snippet is not rendering correctly in HTML.
> > > 
> > > We can stop shpinx complaining by using '::' instead of the currently
> > > used '``' however this still does not render correctly in HTML. The
> > > rendering is [arguably] better but still incorrect. Sphinx renders two
> > > function calls thus:
> > > 
> > > 	:c:func:`rcu_read_lock()`;
> > > 
> > > The rest of the snippet does however have correct spacing.
> > 
> > The behavior when `` was used is not surprising, that really just does a
> > font change.  Once you went with a literal block (with "::") though, the
> > situation changes a bit.  That really should work.
> > 
> > I looked a bit.  This isn't a sphinx (or "shpinx" :) problem, the bug is
> > in kernel-doc.  Once we go into the literal mode, it shouldn't be
> > screwing around with the text anymore.  Of course, kernel-doc doesn't
> > understand enough RST to know that.  I'm a little nervous about trying to
> > teach it more, but maybe we have to do that; we should certainly be able
> > to put code snippets into the docs and have them come through unmolested.
> > 
> > I'll try to look more closely at that shortly.  Meanwhile, this patch
> > makes things better than the were before.  That said...
> > 
> > > Use '::' to pre-fix code snippet. Clears build warnings but does not
> > > render correctly.
> > > 
> > > Signed-off-by: Tobin C. Harding <me@tobin.cc>
> > > ---
> > > 
> > > To view current broken rendering see
> > > 
> > > https://www.kernel.org/doc/html/latest/core-api/kernel-api.html?highlight=rcu_pointer_handoff#c.rcu_pointer_handoff
> > > 
> > >  include/linux/rcupdate.h | 4 ++--
> > >  1 file changed, 2 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> > > index a6ddc42f87a5..cc10e772e3e9 100644
> > > --- a/include/linux/rcupdate.h
> > > +++ b/include/linux/rcupdate.h
> > > @@ -568,7 +568,8 @@ static inline void rcu_preempt_sleep_check(void) { }
> > >   * is handed off from RCU to some other synchronization mechanism, for
> > >   * example, reference counting or locking.  In C11, it would map to
> > >   * kill_dependency().  It could be used as follows:
> > > - * ``
> > > + * ::
> > 
> > ...rather than adding a separate "::" line, you can just
> > s/follows:/follows::/ and the Right Thing will happen (to the same extent
> > that it does now, anyway.
> 
> Except that it will render as
> 
> 	kill_dependency().  It could be used as follows
> 
> instead of
> 	kill_dependency().  It could be used as follows:
> 
> (note: final colon)
> 
> Also the diff will be bigger.  These two reasons led me to the patch as
> it is.  I'm happy to re-spin with your suggested change though.

Woops, not 're-spin' - we are on an RFC.  I'll wait for your response
then submit a PATCH

thanks,
Tobin.

Re: [RFC] doc: fix code snippet build warnings

[Jani Nikula]

On Thu, 11 Jan 2018, "Tobin C. Harding" <me@tobin.cc> wrote:
> On Wed, Jan 10, 2018 at 02:59:58PM -0700, Jonathan Corbet wrote:
>> ...rather than adding a separate "::" line, you can just
>> s/follows:/follows::/ and the Right Thing will happen (to the same extent
>> that it does now, anyway.
>
> Except that it will render as
>
> 	kill_dependency().  It could be used as follows
>
> instead of
> 	kill_dependency().  It could be used as follows:
>
> (note: final colon)

Quoting [1], "The :: may be tacked onto the very end of any
paragraph. The :: will be omitted if it is preceded by whitespace. The
:: will be converted to a single colon if preceded by text"

BR,
Jani.


[1] http://docutils.sourceforge.net/docs/user/rst/quickref.html#literal-blocks



-- 
Jani Nikula, Intel Open Source Technology Center

Re: [RFC] doc: fix code snippet build warnings

[Jani Nikula]

On Wed, 10 Jan 2018, Jonathan Corbet <corbet@lwn.net> wrote:
> On Wed, 10 Jan 2018 15:04:53 +1100
> "Tobin C. Harding" <me@tobin.cc> wrote:
>
>> Posting as RFC in the hope that someone knows how to massage sphinx
>> correctly to fix this patch.
>> 
>> Currently function kernel-doc contains a multi-line code snippet. This
>> is causing sphinx to emit 5 build warnings
>> 
>> 	WARNING: Unexpected indentation.
>> 	WARNING: Unexpected indentation.
>> 	WARNING: Block quote ends without a blank line; unexpected unindent.
>> 	WARNING: Block quote ends without a blank line; unexpected unindent.
>> 	WARNING: Inline literal start-string without end-string.
>> 
>> And the snippet is not rendering correctly in HTML.
>> 
>> We can stop shpinx complaining by using '::' instead of the currently
>> used '``' however this still does not render correctly in HTML. The
>> rendering is [arguably] better but still incorrect. Sphinx renders two
>> function calls thus:
>> 
>> 	:c:func:`rcu_read_lock()`;
>> 
>> The rest of the snippet does however have correct spacing.
>
> The behavior when `` was used is not surprising, that really just does a
> font change.  Once you went with a literal block (with "::") though, the
> situation changes a bit.  That really should work.
>
> I looked a bit.  This isn't a sphinx (or "shpinx" :) problem, the bug is
> in kernel-doc.  Once we go into the literal mode, it shouldn't be
> screwing around with the text anymore.  Of course, kernel-doc doesn't
> understand enough RST to know that.  I'm a little nervous about trying to
> teach it more, but maybe we have to do that; we should certainly be able
> to put code snippets into the docs and have them come through unmolested.

I think eventually the Right Thing would be to move most of kernel-doc's
mucking of the comments (i.e. highlights) into kernel-doc the Sphinx
extension. I've toyed with the idea, but it's not as trivial as I would
like it to be.

Basically it involves flagging all the kernel-doc nodes during the read
phase, and registering handlers to post process the doctrees. At that
stage, it's no longer simple regexp replaces, it's replacing doctree
nodes with ones that have reference nodes within them. It's more
complicated, but at that stage we can ignore stuff that should stay
verbatim. I think it's also possible to check that the reference targets
actually exist. In short, at that phase we have all the knowledge about
the rst structure, parsed by Sphinx, and we don't have to duplicate that
knowledge in kernel-doc the perl script.

Note that this has nothing to do with swithing to Python based
kernel-doc suggested by Markus previously.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: [RFC] doc: fix code snippet build warnings

[Markus Heiser]

> Am 16.01.2018 um 11:22 schrieb Jani Nikula <jani.nikula@linux.intel.com>:
> 
> On Wed, 10 Jan 2018, Jonathan Corbet <corbet@lwn.net> wrote:
>> On Wed, 10 Jan 2018 15:04:53 +1100
>> "Tobin C. Harding" <me@tobin.cc> wrote:
>> 
>>> Posting as RFC in the hope that someone knows how to massage sphinx
>>> correctly to fix this patch.
>>> 
>>> Currently function kernel-doc contains a multi-line code snippet. This
>>> is causing sphinx to emit 5 build warnings
>>> 
>>> 	WARNING: Unexpected indentation.
>>> 	WARNING: Unexpected indentation.
>>> 	WARNING: Block quote ends without a blank line; unexpected unindent.
>>> 	WARNING: Block quote ends without a blank line; unexpected unindent.
>>> 	WARNING: Inline literal start-string without end-string.
>>> 
>>> And the snippet is not rendering correctly in HTML.
>>> 
>>> We can stop shpinx complaining by using '::' instead of the currently
>>> used '``' however this still does not render correctly in HTML. The
>>> rendering is [arguably] better but still incorrect. Sphinx renders two
>>> function calls thus:
>>> 
>>> 	:c:func:`rcu_read_lock()`;
>>> 
>>> The rest of the snippet does however have correct spacing.
>> 
>> The behavior when `` was used is not surprising, that really just does a
>> font change.  Once you went with a literal block (with "::") though, the
>> situation changes a bit.  That really should work.
>> 
>> I looked a bit.  This isn't a sphinx (or "shpinx" :) problem, the bug is
>> in kernel-doc.  Once we go into the literal mode, it shouldn't be
>> screwing around with the text anymore.  Of course, kernel-doc doesn't
>> understand enough RST to know that.  I'm a little nervous about trying to
>> teach it more, but maybe we have to do that; we should certainly be able
>> to put code snippets into the docs and have them come through unmolested.
> 
> I think eventually the Right Thing would be to move most of kernel-doc's
> mucking of the comments (i.e. highlights) into kernel-doc the Sphinx
> extension. I've toyed with the idea, but it's not as trivial as I would
> like it to be.
> 
> Basically it involves flagging all the kernel-doc nodes during the read
> phase, and registering handlers to post process the doctrees. At that
> stage, it's no longer simple regexp replaces, it's replacing doctree
> nodes with ones that have reference nodes within them. It's more
> complicated,

Hi Jani, my thoughts about:  

1.) The reST syntax [1] (the parser part) is not *extendable*, we
can only add new roles [2] or directives.

2.) the kernel-doc syntax is weak and ambiguous. This
remains mainly in tagging only with a start-tag or only with a end-tag 
e.g:

* sectioning:   "Return:" --> end-tag just ":"
* fields:       "@arg1:"  --> better
* highlight/ref: start tag [@%$&] / no end tag

even if I don't know how (1.), if the highlight-syntax becomes a part
of the reST syntax we have to quote [@%$&] elsewhere. This will break
with valid reST documents which is unacceptable.

Thats why I think we need to pre-parse highlighting in the kernel-doc
Perl script, even if it is a bit hackish (how should it be otherwise,
if the syntax is already hackish). I haven't had the time to implement
such a highlight parser right now, but this would be the first shot of
mine. 

An alternative might be to use roles [2] but this means we have to
break with the the kernel-doc (highlight) syntax which is IMO also
unacceptable ... tricky situation :o

[1] http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html
[2] http://docutils.sourceforge.net/docs/howto/rst-roles.html


> but at that stage we can ignore stuff that should stay
> verbatim. I think it's also possible to check that the reference targets
> actually exist. In short, at that phase we have all the knowledge about
> the rst structure, parsed by Sphinx, and we don't have to duplicate that
> knowledge in kernel-doc the perl script.
> 
> Note that this has nothing to do with swithing to Python based
> kernel-doc suggested by Markus previously.

;)

-- Markus --

Re: [RFC] doc: fix code snippet build warnings

[Jani Nikula]

On Tue, 16 Jan 2018, Markus Heiser <markus.heiser@darmarit.de> wrote:
>> Am 16.01.2018 um 11:22 schrieb Jani Nikula <jani.nikula@linux.intel.com>:
>> 
>> On Wed, 10 Jan 2018, Jonathan Corbet <corbet@lwn.net> wrote:
>>> On Wed, 10 Jan 2018 15:04:53 +1100
>>> "Tobin C. Harding" <me@tobin.cc> wrote:
>>> 
>>>> Posting as RFC in the hope that someone knows how to massage sphinx
>>>> correctly to fix this patch.
>>>> 
>>>> Currently function kernel-doc contains a multi-line code snippet. This
>>>> is causing sphinx to emit 5 build warnings
>>>> 
>>>> 	WARNING: Unexpected indentation.
>>>> 	WARNING: Unexpected indentation.
>>>> 	WARNING: Block quote ends without a blank line; unexpected unindent.
>>>> 	WARNING: Block quote ends without a blank line; unexpected unindent.
>>>> 	WARNING: Inline literal start-string without end-string.
>>>> 
>>>> And the snippet is not rendering correctly in HTML.
>>>> 
>>>> We can stop shpinx complaining by using '::' instead of the currently
>>>> used '``' however this still does not render correctly in HTML. The
>>>> rendering is [arguably] better but still incorrect. Sphinx renders two
>>>> function calls thus:
>>>> 
>>>> 	:c:func:`rcu_read_lock()`;
>>>> 
>>>> The rest of the snippet does however have correct spacing.
>>> 
>>> The behavior when `` was used is not surprising, that really just does a
>>> font change.  Once you went with a literal block (with "::") though, the
>>> situation changes a bit.  That really should work.
>>> 
>>> I looked a bit.  This isn't a sphinx (or "shpinx" :) problem, the bug is
>>> in kernel-doc.  Once we go into the literal mode, it shouldn't be
>>> screwing around with the text anymore.  Of course, kernel-doc doesn't
>>> understand enough RST to know that.  I'm a little nervous about trying to
>>> teach it more, but maybe we have to do that; we should certainly be able
>>> to put code snippets into the docs and have them come through unmolested.
>> 
>> I think eventually the Right Thing would be to move most of kernel-doc's
>> mucking of the comments (i.e. highlights) into kernel-doc the Sphinx
>> extension. I've toyed with the idea, but it's not as trivial as I would
>> like it to be.
>> 
>> Basically it involves flagging all the kernel-doc nodes during the read
>> phase, and registering handlers to post process the doctrees. At that
>> stage, it's no longer simple regexp replaces, it's replacing doctree
>> nodes with ones that have reference nodes within them. It's more
>> complicated,
>
> Hi Jani, my thoughts about:  
>
> 1.) The reST syntax [1] (the parser part) is not *extendable*, we
> can only add new roles [2] or directives.

Right, but we've *already* extended it using kernel-doc anyway. I'm just
suggesting a way to do that *after* all other rst parsing has been
done. It's much easier to not confuse the rst parser this way. And we
can trivially skip reference nodes or bold text etc. already created by
Sphinx. And we can limit this to nodes generated from kernel-doc
comments if we want.

So for example, we can check if a text node contains, say, "struct foo",
check if struct foo documentation exists, and turn that into a reference
automatically. In most cases that's the right thing to do. Same with
function() stuff. And even when it's actually wrong, it doesn't screw up
the rendering like in the example in this thread, but rather just
generates a bogus link.

BR,
Jani.

>
> 2.) the kernel-doc syntax is weak and ambiguous. This
> remains mainly in tagging only with a start-tag or only with a end-tag 
> e.g:
>
> * sectioning:   "Return:" --> end-tag just ":"
> * fields:       "@arg1:"  --> better
> * highlight/ref: start tag [@%$&] / no end tag
>
> even if I don't know how (1.), if the highlight-syntax becomes a part
> of the reST syntax we have to quote [@%$&] elsewhere. This will break
> with valid reST documents which is unacceptable.
>
> Thats why I think we need to pre-parse highlighting in the kernel-doc
> Perl script, even if it is a bit hackish (how should it be otherwise,
> if the syntax is already hackish). I haven't had the time to implement
> such a highlight parser right now, but this would be the first shot of
> mine. 
>
> An alternative might be to use roles [2] but this means we have to
> break with the the kernel-doc (highlight) syntax which is IMO also
> unacceptable ... tricky situation :o
>
> [1] http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html
> [2] http://docutils.sourceforge.net/docs/howto/rst-roles.html
>
>
>> but at that stage we can ignore stuff that should stay
>> verbatim. I think it's also possible to check that the reference targets
>> actually exist. In short, at that phase we have all the knowledge about
>> the rst structure, parsed by Sphinx, and we don't have to duplicate that
>> knowledge in kernel-doc the perl script.
>> 
>> Note that this has nothing to do with swithing to Python based
>> kernel-doc suggested by Markus previously.
>
> ;)
>
> -- Markus --

-- 
Jani Nikula, Intel Open Source Technology Center