[PATCH 0/8] Clean up kernel-doc and fix literal-block handling

[PATCH 0/8] Clean up kernel-doc and fix literal-block handling

[Jonathan Corbet]

So once upon a time I set out to fix the problem reported by Tobin wherein
a literal block within a kerneldoc comment would be corrupted in
processing.  On the way, though, I got annoyed at the way I have to learn
how kernel-doc works from the beginning every time I tear into it.

As a result, seven of the following eight patches just get rid of some dead
code and reorganize the rest - mostly turning the 500-line process_file()
function into something a bit more rational.  Sphinx output is unchanged
after these are applied.  Then, at the end, there's a tweak to stop messing
with literal blocks.

If anybody was unaware that I've not done any serious Perl since the
1990's, they will certainly understand that fact now.

Jonathan Corbet (8):
  docs: kernel-doc: Get rid of xml_escape() and friends
  docs: kernel-doc: Rename and split STATE_FIELD
  docs: kernel-doc: Move STATE_NORMAL processing into its own function
  docs: kernel-doc: Move STATE_NAME processing into its own function
  docs: kernel-doc: Move STATE_BODY processing to a separate function
  docs: kernel-doc: Move STATE_PROTO processing into its own function
  docs: kernel-doc: Finish moving STATE_* code out of process_file()
  docs: kernel-doc: Don't mangle literal code blocks in comments

 scripts/kernel-doc | 658 +++++++++++++++++++++++++++++------------------------
 1 file changed, 357 insertions(+), 301 deletions(-)

-- 
2.14.3

[PATCH 1/8] docs: kernel-doc: Get rid of xml_escape() and friends

[Jonathan Corbet]

XML escaping is a worry that came with DocBook, which we no longer have any
dealings with.  So get rid of the useless xml_escape()/xml_unescape()
functions.  No change to the generated output.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/kernel-doc | 65 ++++++++----------------------------------------------
 1 file changed, 9 insertions(+), 56 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index fee8952037b1..5aa4ce211fc6 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -553,10 +553,9 @@ sub output_highlight {
 	}
 	if ($line eq ""){
 	    if (! $output_preformatted) {
-		print $lineprefix, local_unescape($blankline);
+		print $lineprefix, $blankline;
 	    }
 	} else {
-	    $line =~ s/\\\\\\/\&/g;
 	    if ($output_mode eq "man" && substr($line, 0, 1) eq ".") {
 		print "\\&$line";
 	    } else {
@@ -751,9 +750,6 @@ sub output_highlight_rst {
     my $contents = join "\n",@_;
     my $line;
 
-    # undo the evil effects of xml_escape() earlier
-    $contents = xml_unescape($contents);
-
     eval $dohighlight;
     die $@ if $@;
 
@@ -1422,8 +1418,6 @@ sub push_parameter($$$$) {
 		}
 	}
 
-	$param = xml_escape($param);
-
 	# strip spaces from $param so that it is one continuous string
 	# on @parameterlist;
 	# this fixes a problem where check_sections() cannot find
@@ -1748,47 +1742,6 @@ sub process_proto_type($$) {
     }
 }
 
-# xml_escape: replace <, >, and & in the text stream;
-#
-# however, formatting controls that are generated internally/locally in the
-# kernel-doc script are not escaped here; instead, they begin life like
-# $blankline_html (4 of '\' followed by a mnemonic + ':'), then these strings
-# are converted to their mnemonic-expected output, without the 4 * '\' & ':',
-# just before actual output; (this is done by local_unescape())
-sub xml_escape($) {
-	my $text = shift;
-	if ($output_mode eq "man") {
-		return $text;
-	}
-	$text =~ s/\&/\\\\\\amp;/g;
-	$text =~ s/\</\\\\\\lt;/g;
-	$text =~ s/\>/\\\\\\gt;/g;
-	return $text;
-}
-
-# xml_unescape: reverse the effects of xml_escape
-sub xml_unescape($) {
-	my $text = shift;
-	if ($output_mode eq "man") {
-		return $text;
-	}
-	$text =~ s/\\\\\\amp;/\&/g;
-	$text =~ s/\\\\\\lt;/</g;
-	$text =~ s/\\\\\\gt;/>/g;
-	return $text;
-}
-
-# convert local escape strings to html
-# local escape strings look like:  '\\\\menmonic:' (that's 4 backslashes)
-sub local_unescape($) {
-	my $text = shift;
-	if ($output_mode eq "man") {
-		return $text;
-	}
-	$text =~ s/\\\\\\\\lt:/</g;
-	$text =~ s/\\\\\\\\gt:/>/g;
-	return $text;
-}
 
 sub map_filename($) {
     my $file;
@@ -1889,7 +1842,7 @@ sub process_file($) {
 		    $descr =~ s/^\s*//;
 		    $descr =~ s/\s*$//;
 		    $descr =~ s/\s+/ /g;
-		    $declaration_purpose = xml_escape($descr);
+		    $declaration_purpose = $descr;
 		    $in_purpose = 1;
 		} else {
 		    $declaration_purpose = "";
@@ -1944,7 +1897,7 @@ sub process_file($) {
 			print STDERR "${file}:$.: warning: contents before sections\n";
 			++$warnings;
 		    }
-		    dump_section($file, $section, xml_escape($contents));
+		    dump_section($file, $section, $contents);
 		    $section = $section_default;
 		}
 
@@ -1962,7 +1915,7 @@ sub process_file($) {
 		$leading_space = undef;
 	    } elsif (/$doc_end/) {
 		if (($contents ne "") && ($contents ne "\n")) {
-		    dump_section($file, $section, xml_escape($contents));
+		    dump_section($file, $section, $contents);
 		    $section = $section_default;
 		    $contents = "";
 		}
@@ -1981,7 +1934,7 @@ sub process_file($) {
 		# @parameter line to signify start of description
 		if ($1 eq "") {
 		    if ($section =~ m/^@/ || $section eq $section_context) {
-			dump_section($file, $section, xml_escape($contents));
+			dump_section($file, $section, $contents);
 			$section = $section_default;
 			$contents = "";
                         $new_start_line = $.;
@@ -1992,7 +1945,7 @@ sub process_file($) {
 		} elsif ($in_purpose == 1) {
 		    # Continued declaration purpose
 		    chomp($declaration_purpose);
-		    $declaration_purpose .= " " . xml_escape($1);
+		    $declaration_purpose .= " " . $1;
 		    $declaration_purpose =~ s/\s+/ /g;
 		} else {
 		    my $cont = $1;
@@ -2030,7 +1983,7 @@ sub process_file($) {
 	    # Documentation block end */
 	    } elsif (/$doc_inline_end/) {
 		if (($contents ne "") && ($contents ne "\n")) {
-		    dump_section($file, $section, xml_escape($contents));
+		    dump_section($file, $section, $contents);
 		    $section = $section_default;
 		    $contents = "";
 		}
@@ -2057,7 +2010,7 @@ sub process_file($) {
 		$contents = $2;
 		if ($contents ne "") {
 		    $contents .= "\n";
-		    dump_section($file, $section, xml_escape($contents));
+		    dump_section($file, $section, $contents);
 		    $section = $section_default;
 		    $contents = "";
 		}
@@ -2072,7 +2025,7 @@ sub process_file($) {
 	} elsif ($state == STATE_DOCBLOCK) {
 		if (/$doc_end/)
 		{
-			dump_doc_section($file, $section, xml_escape($contents));
+			dump_doc_section($file, $section, $contents);
 			$section = $section_default;
 			$contents = "";
 			$function = "";
-- 
2.14.3

[PATCH 2/8] docs: kernel-doc: Rename and split STATE_FIELD

[Jonathan Corbet]

STATE_FIELD describes a parser state that can handle any part of a
kerneldoc comment body; rename it to STATE_BODY to reflect that.

The $in_purpose variable was a hidden substate of STATE_FIELD; get rid of
it and make a proper state (STATE_BODY_MAYBE) instead.  This will make the
subsequent process_file() splitup easier.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/kernel-doc | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 5aa4ce211fc6..ad30c52f91ef 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -328,10 +328,11 @@ my $lineprefix="";
 use constant {
     STATE_NORMAL        => 0, # normal code
     STATE_NAME          => 1, # looking for function name
-    STATE_FIELD         => 2, # scanning field start
-    STATE_PROTO         => 3, # scanning prototype
-    STATE_DOCBLOCK      => 4, # documentation block
-    STATE_INLINE        => 5, # gathering documentation outside main block
+    STATE_BODY_MAYBE    => 2, # body - or maybe more description
+    STATE_BODY          => 3, # the body of the comment
+    STATE_PROTO         => 4, # scanning prototype
+    STATE_DOCBLOCK      => 5, # documentation block
+    STATE_INLINE        => 6, # gathering documentation outside main block
 };
 my $state;
 my $in_doc_sect;
@@ -1784,7 +1785,6 @@ sub process_file($) {
     my $identifier;
     my $func;
     my $descr;
-    my $in_purpose = 0;
     my $initial_section_counter = $section_counter;
     my ($orig_file) = @_;
     my $leading_space;
@@ -1830,7 +1830,7 @@ sub process_file($) {
 		    $identifier = $1;
 		}
 
-		$state = STATE_FIELD;
+		$state = STATE_BODY;
 		# if there's no @param blocks need to set up default section
 		# here
 		$contents = "";
@@ -1843,7 +1843,7 @@ sub process_file($) {
 		    $descr =~ s/\s*$//;
 		    $descr =~ s/\s+/ /g;
 		    $declaration_purpose = $descr;
-		    $in_purpose = 1;
+		    $state = STATE_BODY_MAYBE;
 		} else {
 		    $declaration_purpose = "";
 		}
@@ -1875,7 +1875,7 @@ sub process_file($) {
 		++$warnings;
 		$state = STATE_NORMAL;
 	    }
-	} elsif ($state == STATE_FIELD) {	# look for head: lines, and include content
+	} elsif ($state == STATE_BODY || $state == STATE_BODY_MAYBE) {
 	    if (/$doc_sect/i) { # case insensitive for supported section names
 		$newsection = $1;
 		$newcontents = $2;
@@ -1902,7 +1902,7 @@ sub process_file($) {
 		}
 
 		$in_doc_sect = 1;
-		$in_purpose = 0;
+		$state = STATE_BODY;
 		$contents = $newcontents;
                 $new_start_line = $.;
 		while (substr($contents, 0, 1) eq " ") {
@@ -1941,8 +1941,8 @@ sub process_file($) {
 		    } else {
 			$contents .= "\n";
 		    }
-		    $in_purpose = 0;
-		} elsif ($in_purpose == 1) {
+		    $state = STATE_BODY;
+		} elsif ($state == STATE_BODY_MAYBE) {
 		    # Continued declaration purpose
 		    chomp($declaration_purpose);
 		    $declaration_purpose .= " " . $1;
-- 
2.14.3

[PATCH 3/8] docs: kernel-doc: Move STATE_NORMAL processing into its own function

[Jonathan Corbet]

Begin the process of splitting up the nearly 500-line process_file()
function by moving STATE_NORMAL processing to a separate function.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/kernel-doc | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index ad30c52f91ef..65150b7c8472 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -1780,6 +1780,21 @@ sub process_export_file($) {
     close(IN);
 }
 
+#
+# Parsers for the various processing states.
+#
+# STATE_NORMAL: looking for the /** to begin everything.
+#
+sub process_normal() {
+    if (/$doc_start/o) {
+	$state = STATE_NAME;	# next line is always the function name
+	$in_doc_sect = 0;
+	$declaration_start_line = $. + 1;
+    }
+}
+
+
+
 sub process_file($) {
     my $file;
     my $identifier;
@@ -1807,11 +1822,7 @@ sub process_file($) {
 	# Replace tabs by spaces
         while ($_ =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {};
 	if ($state == STATE_NORMAL) {
-	    if (/$doc_start/o) {
-		$state = STATE_NAME;	# next line is always the function name
-		$in_doc_sect = 0;
-		$declaration_start_line = $. + 1;
-	    }
+	    process_normal();
 	} elsif ($state == STATE_NAME) {# this line is the function name (always)
 	    if (/$doc_block/o) {
 		$state = STATE_DOCBLOCK;
-- 
2.14.3

[PATCH 4/8] docs: kernel-doc: Move STATE_NAME processing into its own function

[Jonathan Corbet]

Move this code out of process_file() in the name of readability and
maintainability.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/kernel-doc | 137 ++++++++++++++++++++++++++++-------------------------
 1 file changed, 72 insertions(+), 65 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 65150b7c8472..a27c7016f72d 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -1793,13 +1793,81 @@ sub process_normal() {
     }
 }
 
+#
+# STATE_NAME: Looking for the "name - description" line
+#
+sub process_name($$) {
+    my $file = shift;
+    my $identifier;
+    my $descr;
+
+    if (/$doc_block/o) {
+	$state = STATE_DOCBLOCK;
+	$contents = "";
+	$new_start_line = $. + 1;
+
+	if ( $1 eq "" ) {
+	    $section = $section_intro;
+	} else {
+	    $section = $1;
+	}
+    }
+    elsif (/$doc_decl/o) {
+	$identifier = $1;
+	if (/\s*([\w\s]+?)\s*-/) {
+	    $identifier = $1;
+	}
 
+	$state = STATE_BODY;
+	# if there's no @param blocks need to set up default section
+	# here
+	$contents = "";
+	$section = $section_default;
+	$new_start_line = $. + 1;
+	if (/-(.*)/) {
+	    # strip leading/trailing/multiple spaces
+	    $descr= $1;
+	    $descr =~ s/^\s*//;
+	    $descr =~ s/\s*$//;
+	    $descr =~ s/\s+/ /g;
+	    $declaration_purpose = $descr;
+	    $state = STATE_BODY_MAYBE;
+	} else {
+	    $declaration_purpose = "";
+	}
+
+	if (($declaration_purpose eq "") && $verbose) {
+	    print STDERR "${file}:$.: warning: missing initial short description on line:\n";
+	    print STDERR $_;
+	    ++$warnings;
+	}
+
+	if ($identifier =~ m/^struct/) {
+	    $decl_type = 'struct';
+	} elsif ($identifier =~ m/^union/) {
+	    $decl_type = 'union';
+	} elsif ($identifier =~ m/^enum/) {
+	    $decl_type = 'enum';
+	} elsif ($identifier =~ m/^typedef/) {
+	    $decl_type = 'typedef';
+	} else {
+	    $decl_type = 'function';
+	}
+
+	if ($verbose) {
+	    print STDERR "${file}:$.: info: Scanning doc for $identifier\n";
+	}
+    } else {
+	print STDERR "${file}:$.: warning: Cannot understand $_ on line $.",
+	    " - I thought it was a doc line\n";
+	++$warnings;
+	$state = STATE_NORMAL;
+    }
+}
 
 sub process_file($) {
     my $file;
-    my $identifier;
     my $func;
-    my $descr;
     my $initial_section_counter = $section_counter;
     my ($orig_file) = @_;
     my $leading_space;
@@ -1823,69 +1891,8 @@ sub process_file($) {
         while ($_ =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {};
 	if ($state == STATE_NORMAL) {
 	    process_normal();
-	} elsif ($state == STATE_NAME) {# this line is the function name (always)
-	    if (/$doc_block/o) {
-		$state = STATE_DOCBLOCK;
-		$contents = "";
-                $new_start_line = $. + 1;
-
-		if ( $1 eq "" ) {
-			$section = $section_intro;
-		} else {
-			$section = $1;
-		}
-	    }
-	    elsif (/$doc_decl/o) {
-		$identifier = $1;
-		if (/\s*([\w\s]+?)\s*-/) {
-		    $identifier = $1;
-		}
-
-		$state = STATE_BODY;
-		# if there's no @param blocks need to set up default section
-		# here
-		$contents = "";
-		$section = $section_default;
-		$new_start_line = $. + 1;
-		if (/-(.*)/) {
-		    # strip leading/trailing/multiple spaces
-		    $descr= $1;
-		    $descr =~ s/^\s*//;
-		    $descr =~ s/\s*$//;
-		    $descr =~ s/\s+/ /g;
-		    $declaration_purpose = $descr;
-		    $state = STATE_BODY_MAYBE;
-		} else {
-		    $declaration_purpose = "";
-		}
-
-		if (($declaration_purpose eq "") && $verbose) {
-			print STDERR "${file}:$.: warning: missing initial short description on line:\n";
-			print STDERR $_;
-			++$warnings;
-		}
-
-		if ($identifier =~ m/^struct/) {
-		    $decl_type = 'struct';
-		} elsif ($identifier =~ m/^union/) {
-		    $decl_type = 'union';
-		} elsif ($identifier =~ m/^enum/) {
-		    $decl_type = 'enum';
-		} elsif ($identifier =~ m/^typedef/) {
-		    $decl_type = 'typedef';
-		} else {
-		    $decl_type = 'function';
-		}
-
-		if ($verbose) {
-		    print STDERR "${file}:$.: info: Scanning doc for $identifier\n";
-		}
-	    } else {
-		print STDERR "${file}:$.: warning: Cannot understand $_ on line $.",
-		" - I thought it was a doc line\n";
-		++$warnings;
-		$state = STATE_NORMAL;
-	    }
+	} elsif ($state == STATE_NAME) {
+	    process_name($file, $_);
 	} elsif ($state == STATE_BODY || $state == STATE_BODY_MAYBE) {
 	    if (/$doc_sect/i) { # case insensitive for supported section names
 		$newsection = $1;
-- 
2.14.3

[PATCH 5/8] docs: kernel-doc: Move STATE_BODY processing to a separate function

[Jonathan Corbet]

Also group the pseudo-global $leading_space variable with its peers.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/kernel-doc | 193 ++++++++++++++++++++++++++++-------------------------
 1 file changed, 101 insertions(+), 92 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index a27c7016f72d..a6a7bb46ea29 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -336,6 +336,7 @@ use constant {
 };
 my $state;
 my $in_doc_sect;
+my $leading_space;
 
 # Inline documentation state
 use constant {
@@ -1865,12 +1866,110 @@ sub process_name($$) {
     }
 }
 
+
+#
+# STATE_BODY and STATE_BODY_MAYBE: the bulk of a kerneldoc comment.
+#
+sub process_body($$) {
+    my $file = shift;
+
+    if (/$doc_sect/i) { # case insensitive for supported section names
+	$newsection = $1;
+	$newcontents = $2;
+
+	# map the supported section names to the canonical names
+	if ($newsection =~ m/^description$/i) {
+	    $newsection = $section_default;
+	} elsif ($newsection =~ m/^context$/i) {
+	    $newsection = $section_context;
+	} elsif ($newsection =~ m/^returns?$/i) {
+	    $newsection = $section_return;
+	} elsif ($newsection =~ m/^\@return$/) {
+	    # special: @return is a section, not a param description
+	    $newsection = $section_return;
+	}
+
+	if (($contents ne "") && ($contents ne "\n")) {
+	    if (!$in_doc_sect && $verbose) {
+		print STDERR "${file}:$.: warning: contents before sections\n";
+		++$warnings;
+	    }
+	    dump_section($file, $section, $contents);
+	    $section = $section_default;
+	}
+
+	$in_doc_sect = 1;
+	$state = STATE_BODY;
+	$contents = $newcontents;
+	$new_start_line = $.;
+	while (substr($contents, 0, 1) eq " ") {
+	    $contents = substr($contents, 1);
+	}
+	if ($contents ne "") {
+	    $contents .= "\n";
+	}
+	$section = $newsection;
+	$leading_space = undef;
+    } elsif (/$doc_end/) {
+	if (($contents ne "") && ($contents ne "\n")) {
+	    dump_section($file, $section, $contents);
+	    $section = $section_default;
+	    $contents = "";
+	}
+	# look for doc_com + <text> + doc_end:
+	if ($_ =~ m'\s*\*\s*[a-zA-Z_0-9:\.]+\*/') {
+	    print STDERR "${file}:$.: warning: suspicious ending line: $_";
+	    ++$warnings;
+	}
+
+	$prototype = "";
+	$state = STATE_PROTO;
+	$brcount = 0;
+    } elsif (/$doc_content/) {
+	# miguel-style comment kludge, look for blank lines after
+	# @parameter line to signify start of description
+	if ($1 eq "") {
+	    if ($section =~ m/^@/ || $section eq $section_context) {
+		dump_section($file, $section, $contents);
+		$section = $section_default;
+		$contents = "";
+		$new_start_line = $.;
+	    } else {
+		$contents .= "\n";
+	    }
+	    $state = STATE_BODY;
+	} elsif ($state == STATE_BODY_MAYBE) {
+	    # Continued declaration purpose
+	    chomp($declaration_purpose);
+	    $declaration_purpose .= " " . $1;
+	    $declaration_purpose =~ s/\s+/ /g;
+	} else {
+	    my $cont = $1;
+	    if ($section =~ m/^@/ || $section eq $section_context) {
+		if (!defined $leading_space) {
+		    if ($cont =~ m/^(\s+)/) {
+			$leading_space = $1;
+		    } else {
+			$leading_space = "";
+		    }
+		}
+		$cont =~ s/^$leading_space//;
+	    }
+	    $contents .= $cont . "\n";
+	}
+    } else {
+	# i dont know - bad line?  ignore.
+	print STDERR "${file}:$.: warning: bad line: $_";
+	++$warnings;
+    }
+}
+
+
 sub process_file($) {
     my $file;
     my $func;
     my $initial_section_counter = $section_counter;
     my ($orig_file) = @_;
-    my $leading_space;
 
     $file = map_filename($orig_file);
 
@@ -1894,97 +1993,7 @@ sub process_file($) {
 	} elsif ($state == STATE_NAME) {
 	    process_name($file, $_);
 	} elsif ($state == STATE_BODY || $state == STATE_BODY_MAYBE) {
-	    if (/$doc_sect/i) { # case insensitive for supported section names
-		$newsection = $1;
-		$newcontents = $2;
-
-		# map the supported section names to the canonical names
-		if ($newsection =~ m/^description$/i) {
-		    $newsection = $section_default;
-		} elsif ($newsection =~ m/^context$/i) {
-		    $newsection = $section_context;
-		} elsif ($newsection =~ m/^returns?$/i) {
-		    $newsection = $section_return;
-		} elsif ($newsection =~ m/^\@return$/) {
-		    # special: @return is a section, not a param description
-		    $newsection = $section_return;
-		}
-
-		if (($contents ne "") && ($contents ne "\n")) {
-		    if (!$in_doc_sect && $verbose) {
-			print STDERR "${file}:$.: warning: contents before sections\n";
-			++$warnings;
-		    }
-		    dump_section($file, $section, $contents);
-		    $section = $section_default;
-		}
-
-		$in_doc_sect = 1;
-		$state = STATE_BODY;
-		$contents = $newcontents;
-                $new_start_line = $.;
-		while (substr($contents, 0, 1) eq " ") {
-		    $contents = substr($contents, 1);
-		}
-		if ($contents ne "") {
-		    $contents .= "\n";
-		}
-		$section = $newsection;
-		$leading_space = undef;
-	    } elsif (/$doc_end/) {
-		if (($contents ne "") && ($contents ne "\n")) {
-		    dump_section($file, $section, $contents);
-		    $section = $section_default;
-		    $contents = "";
-		}
-		# look for doc_com + <text> + doc_end:
-		if ($_ =~ m'\s*\*\s*[a-zA-Z_0-9:\.]+\*/') {
-		    print STDERR "${file}:$.: warning: suspicious ending line: $_";
-		    ++$warnings;
-		}
-
-		$prototype = "";
-		$state = STATE_PROTO;
-		$brcount = 0;
-#		print STDERR "end of doc comment, looking for prototype\n";
-	    } elsif (/$doc_content/) {
-		# miguel-style comment kludge, look for blank lines after
-		# @parameter line to signify start of description
-		if ($1 eq "") {
-		    if ($section =~ m/^@/ || $section eq $section_context) {
-			dump_section($file, $section, $contents);
-			$section = $section_default;
-			$contents = "";
-                        $new_start_line = $.;
-		    } else {
-			$contents .= "\n";
-		    }
-		    $state = STATE_BODY;
-		} elsif ($state == STATE_BODY_MAYBE) {
-		    # Continued declaration purpose
-		    chomp($declaration_purpose);
-		    $declaration_purpose .= " " . $1;
-		    $declaration_purpose =~ s/\s+/ /g;
-		} else {
-		    my $cont = $1;
-		    if ($section =~ m/^@/ || $section eq $section_context) {
-			if (!defined $leading_space) {
-			    if ($cont =~ m/^(\s+)/) {
-				$leading_space = $1;
-			    } else {
-				$leading_space = "";
-			    }
-			}
-
-			$cont =~ s/^$leading_space//;
-		    }
-		    $contents .= $cont . "\n";
-		}
-	    } else {
-		# i dont know - bad line?  ignore.
-		print STDERR "${file}:$.: warning: bad line: $_";
-		++$warnings;
-	    }
+	    process_body($file, $_);
 	} elsif ($state == STATE_INLINE) { # scanning for inline parameters
 	    # First line (state 1) needs to be a @parameter
 	    if ($inline_doc_state == STATE_INLINE_NAME && /$doc_inline_sect/o) {
-- 
2.14.3

[PATCH 6/8] docs: kernel-doc: Move STATE_PROTO processing into its own function

[Jonathan Corbet]

Move the top-level prototype-processing code out of process_file().

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/kernel-doc | 46 ++++++++++++++++++++++++++++------------------
 1 file changed, 28 insertions(+), 18 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index a6a7bb46ea29..2deddb876156 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -1965,6 +1965,32 @@ sub process_body($$) {
 }
 
 
+#
+# STATE_PROTO: reading a function/whatever prototype.
+#
+sub process_proto($$) {
+    my $file = shift;
+
+    if (/$doc_inline_oneline/) {
+	$section = $1;
+	$contents = $2;
+	if ($contents ne "") {
+	    $contents .= "\n";
+	    dump_section($file, $section, $contents);
+	    $section = $section_default;
+	    $contents = "";
+	}
+    } elsif (/$doc_inline_start/) {
+	$state = STATE_INLINE;
+	$inline_doc_state = STATE_INLINE_NAME;
+    } elsif ($decl_type eq 'function') {
+	process_proto_function($_, $file);
+    } else {
+	process_proto_type($_, $file);
+    }
+}
+
+
 sub process_file($) {
     my $file;
     my $func;
@@ -2031,24 +2057,8 @@ sub process_file($) {
 		    ++$warnings;
 		}
 	    }
-	} elsif ($state == STATE_PROTO) {	# scanning for function '{' (end of prototype)
-	    if (/$doc_inline_oneline/) {
-		$section = $1;
-		$contents = $2;
-		if ($contents ne "") {
-		    $contents .= "\n";
-		    dump_section($file, $section, $contents);
-		    $section = $section_default;
-		    $contents = "";
-		}
-	    } elsif (/$doc_inline_start/) {
-		$state = STATE_INLINE;
-		$inline_doc_state = STATE_INLINE_NAME;
-	    } elsif ($decl_type eq 'function') {
-		process_proto_function($_, $file);
-	    } else {
-		process_proto_type($_, $file);
-	    }
+	} elsif ($state == STATE_PROTO) {
+	    process_proto($file, $_);
 	} elsif ($state == STATE_DOCBLOCK) {
 		if (/$doc_end/)
 		{
-- 
2.14.3

[PATCH 7/8] docs: kernel-doc: Finish moving STATE_* code out of process_file()

[Jonathan Corbet]

Move STATE_INLINE and STATE_DOCBLOCK code out of process_file(), which now
actually fits on a single screen.  Delete an unused variable and add a
couple of comments while I'm at it.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/kernel-doc | 145 ++++++++++++++++++++++++++++++-----------------------
 1 file changed, 83 insertions(+), 62 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 2deddb876156..c6c9370a1e49 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -1990,10 +1990,86 @@ sub process_proto($$) {
     }
 }
 
+#
+# STATE_DOCBLOCK: within a DOC: block.
+#
+sub process_docblock($$) {
+    my $file = shift;
+
+    if (/$doc_end/)
+    {
+	dump_doc_section($file, $section, $contents);
+	$section = $section_default;
+	$contents = "";
+	$function = "";
+	%parameterdescs = ();
+	%parametertypes = ();
+	@parameterlist = ();
+	%sections = ();
+	@sectionlist = ();
+	$prototype = "";
+	$state = STATE_NORMAL;
+    }
+    elsif (/$doc_content/)
+    {
+	if ( $1 eq "" )
+	{
+	    $contents .= $blankline;
+	}
+	else
+	{
+	    $contents .= $1 . "\n";
+	}
+    }
+}
+
+#
+# STATE_INLINE: docbook comments within a prototype.
+#
+sub process_inline($$) {
+    my $file = shift;
+
+    # First line (state 1) needs to be a @parameter
+    if ($inline_doc_state == STATE_INLINE_NAME && /$doc_inline_sect/o) {
+	$section = $1;
+	$contents = $2;
+	$new_start_line = $.;
+	if ($contents ne "") {
+	    while (substr($contents, 0, 1) eq " ") {
+		$contents = substr($contents, 1);
+	    }
+	    $contents .= "\n";
+	}
+	$inline_doc_state = STATE_INLINE_TEXT;
+	# Documentation block end */
+    } elsif (/$doc_inline_end/) {
+	if (($contents ne "") && ($contents ne "\n")) {
+	    dump_section($file, $section, $contents);
+	    $section = $section_default;
+	    $contents = "";
+	}
+	$state = STATE_PROTO;
+	$inline_doc_state = STATE_INLINE_NA;
+	# Regular text
+    } elsif (/$doc_content/) {
+	if ($inline_doc_state == STATE_INLINE_TEXT) {
+	    $contents .= $1 . "\n";
+	    # nuke leading blank lines
+	    if ($contents =~ /^\s*$/) {
+		$contents = "";
+	    }
+	} elsif ($inline_doc_state == STATE_INLINE_NAME) {
+	    $inline_doc_state = STATE_INLINE_ERROR;
+	    print STDERR "${file}:$.: warning: ";
+	    print STDERR "Incorrect use of kernel-doc format: $_";
+	    ++$warnings;
+	}
+    }
+}
+
 
 sub process_file($) {
     my $file;
-    my $func;
     my $initial_section_counter = $section_counter;
     my ($orig_file) = @_;
 
@@ -2014,6 +2090,8 @@ sub process_file($) {
 	}
 	# Replace tabs by spaces
         while ($_ =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {};
+
+	# Hand this line to the appropriate state handler
 	if ($state == STATE_NORMAL) {
 	    process_normal();
 	} elsif ($state == STATE_NAME) {
@@ -2021,72 +2099,15 @@ sub process_file($) {
 	} elsif ($state == STATE_BODY || $state == STATE_BODY_MAYBE) {
 	    process_body($file, $_);
 	} elsif ($state == STATE_INLINE) { # scanning for inline parameters
-	    # First line (state 1) needs to be a @parameter
-	    if ($inline_doc_state == STATE_INLINE_NAME && /$doc_inline_sect/o) {
-		$section = $1;
-		$contents = $2;
-                $new_start_line = $.;
-		if ($contents ne "") {
-		    while (substr($contents, 0, 1) eq " ") {
-			$contents = substr($contents, 1);
-		    }
-		    $contents .= "\n";
-		}
-		$inline_doc_state = STATE_INLINE_TEXT;
-	    # Documentation block end */
-	    } elsif (/$doc_inline_end/) {
-		if (($contents ne "") && ($contents ne "\n")) {
-		    dump_section($file, $section, $contents);
-		    $section = $section_default;
-		    $contents = "";
-		}
-		$state = STATE_PROTO;
-		$inline_doc_state = STATE_INLINE_NA;
-	    # Regular text
-	    } elsif (/$doc_content/) {
-		if ($inline_doc_state == STATE_INLINE_TEXT) {
-		    $contents .= $1 . "\n";
-		    # nuke leading blank lines
-		    if ($contents =~ /^\s*$/) {
-			$contents = "";
-		    }
-		} elsif ($inline_doc_state == STATE_INLINE_NAME) {
-		    $inline_doc_state = STATE_INLINE_ERROR;
-		    print STDERR "${file}:$.: warning: ";
-		    print STDERR "Incorrect use of kernel-doc format: $_";
-		    ++$warnings;
-		}
-	    }
+	    process_inline($file, $_);
 	} elsif ($state == STATE_PROTO) {
 	    process_proto($file, $_);
 	} elsif ($state == STATE_DOCBLOCK) {
-		if (/$doc_end/)
-		{
-			dump_doc_section($file, $section, $contents);
-			$section = $section_default;
-			$contents = "";
-			$function = "";
-			%parameterdescs = ();
-			%parametertypes = ();
-			@parameterlist = ();
-			%sections = ();
-			@sectionlist = ();
-			$prototype = "";
-			$state = STATE_NORMAL;
-		}
-		elsif (/$doc_content/)
-		{
-			if ( $1 eq "" )
-			{
-				$contents .= $blankline;
-			}
-			else
-			{
-				$contents .= $1 . "\n";
-			}
-		}
+	    process_docblock($file, $_);
 	}
     }
+
+    # Make sure we got something interesting.
     if ($initial_section_counter == $section_counter) {
 	if ($output_mode ne "none") {
 	    print STDERR "${file}:1: warning: no structured comments found\n";
-- 
2.14.3

[PATCH 8/8] docs: kernel-doc: Don't mangle literal code blocks in comments

[Jonathan Corbet]

It can be useful to put code snippets into kerneldoc comments; that can be
done with the "::" operator at the end of a line like this::

   if (desperate)
       run_in_circles();

kernel-doc currently fails to understand these literal blocks and applies
its normal markup to them, which is then treated as literal by sphinx.  The
result is unsightly markup instead of a useful code snippet.

Apply a hack to the output code to recognize literal blocks and avoid
performing any special markup on them.  It's ugly, but that means it fits
in well with the rest of the script.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/kernel-doc | 55 +++++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 50 insertions(+), 5 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index c6c9370a1e49..c984f82cb897 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -748,14 +748,59 @@ sub output_blockhead_rst(%) {
     }
 }
 
-sub output_highlight_rst {
-    my $contents = join "\n",@_;
-    my $line;
-
+#
+# Apply the RST highlights to a sub-block of text.
+#   
+sub highlight_block($) {
+    # The dohighlight kludge requires the text be called $contents
+    my $contents = shift;
     eval $dohighlight;
     die $@ if $@;
+    return $contents;
+}
 
-    foreach $line (split "\n", $contents) {
+sub output_highlight_rst {
+    my $input = join "\n",@_;
+    my $output = "";
+    my $line;
+    my $in_literal = 0;
+    my $litprefix;
+    my $block = "";
+
+    # The "dohighlight" hack requires that the data be called "$contents"
+    foreach $line (split "\n",$input) {
+	#
+	# If we're in a literal block, see if we should drop out
+	# of it.  Otherwise pass the line straight through unmunged.
+	#
+	if ($in_literal) {
+	    if (! ($line =~ /$litprefix/ || $line =~ /^\s*$/)) {
+		$in_literal = 0;
+	    }
+	    else {
+		$output .= $line . "\n";
+	    }
+	}
+	#
+	# Not in a literal block (or just dropped out)
+	#
+	if (! $in_literal) {
+	    $block .= $line . "\n";
+	    if ($line =~ /^[^.].*::$/) {
+		$in_literal = 1;
+		# Note current indentation - we'll go as long as it's deeper.
+		$line =~ /^(\s*)/;
+		$litprefix = '^' . $1 . ' ';
+		$output .= highlight_block($block);
+		$block = ""
+	    }
+	}
+    }
+
+    if ($block) {
+	$output .= highlight_block($block);
+    }
+    foreach $line (split "\n", $output) {
 	print $lineprefix . $line . "\n";
     }
 }
-- 
2.14.3

Re: [PATCH 0/8] Clean up kernel-doc and fix literal-block handling

[Tobin C. Harding]

On Wed, Feb 07, 2018 at 10:26:16AM -0700, Jonathan Corbet wrote:
> So once upon a time I set out to fix the problem reported by Tobin wherein
> a literal block within a kerneldoc comment would be corrupted in
> processing.  On the way, though, I got annoyed at the way I have to learn
> how kernel-doc works from the beginning every time I tear into it.

Awesome. The patch that exposed that problem hasn't landed yet but I
tested this series and can confirm that it fixes it.

FTR to test I did:

- update function docs for include/linux/rcupdate.h:rcu_pointer_hadoff()

  	removed enclosing '``' and added leading '::'

- apply this series
- rebuild docs
- profit

thanks,
Tobin.

Re: [PATCH 7/8] docs: kernel-doc: Finish moving STATE_* code out of process_file()

[Tobin C. Harding]

On Wed, Feb 07, 2018 at 10:26:23AM -0700, Jonathan Corbet wrote:
> Move STATE_INLINE and STATE_DOCBLOCK code out of process_file(), which now
> actually fits on a single screen.  Delete an unused variable and add a
> couple of comments while I'm at it.
> 
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
> ---
>  scripts/kernel-doc | 145 ++++++++++++++++++++++++++++++-----------------------
>  1 file changed, 83 insertions(+), 62 deletions(-)
> 
> diff --git a/scripts/kernel-doc b/scripts/kernel-doc
> index 2deddb876156..c6c9370a1e49 100755
> --- a/scripts/kernel-doc
> +++ b/scripts/kernel-doc
> @@ -1990,10 +1990,86 @@ sub process_proto($$) {
>      }
>  }
>  
> +#
> +# STATE_DOCBLOCK: within a DOC: block.
> +#
> +sub process_docblock($$) {
> +    my $file = shift;
> +
> +    if (/$doc_end/)
> +    {
> +	dump_doc_section($file, $section, $contents);
> +	$section = $section_default;
> +	$contents = "";
> +	$function = "";
> +	%parameterdescs = ();
> +	%parametertypes = ();
> +	@parameterlist = ();
> +	%sections = ();
> +	@sectionlist = ();
> +	$prototype = "";
> +	$state = STATE_NORMAL;
> +    }
> +    elsif (/$doc_content/)
> +    {
> +	if ( $1 eq "" )
> +	{
> +	    $contents .= $blankline;
> +	}
> +	else
> +	{
> +	    $contents .= $1 . "\n";
> +	}
> +    }
> +}

It doesn't appear to be introduced by you but the brace positions are
non-uniform in this patch.

if
{
  ...
}
else
{
  ...
}

instead of

if {
  ...
} else {
  eee
}

Hope this helps,
Tobin.


(rest of patch left intentionally for reference)

> +
> +#
> +# STATE_INLINE: docbook comments within a prototype.
> +#
> +sub process_inline($$) {
> +    my $file = shift;
> +
> +    # First line (state 1) needs to be a @parameter
> +    if ($inline_doc_state == STATE_INLINE_NAME && /$doc_inline_sect/o) {
> +	$section = $1;
> +	$contents = $2;
> +	$new_start_line = $.;
> +	if ($contents ne "") {
> +	    while (substr($contents, 0, 1) eq " ") {
> +		$contents = substr($contents, 1);
> +	    }
> +	    $contents .= "\n";
> +	}
> +	$inline_doc_state = STATE_INLINE_TEXT;
> +	# Documentation block end */
> +    } elsif (/$doc_inline_end/) {
> +	if (($contents ne "") && ($contents ne "\n")) {
> +	    dump_section($file, $section, $contents);
> +	    $section = $section_default;
> +	    $contents = "";
> +	}
> +	$state = STATE_PROTO;
> +	$inline_doc_state = STATE_INLINE_NA;
> +	# Regular text
> +    } elsif (/$doc_content/) {
> +	if ($inline_doc_state == STATE_INLINE_TEXT) {
> +	    $contents .= $1 . "\n";
> +	    # nuke leading blank lines
> +	    if ($contents =~ /^\s*$/) {
> +		$contents = "";
> +	    }
> +	} elsif ($inline_doc_state == STATE_INLINE_NAME) {
> +	    $inline_doc_state = STATE_INLINE_ERROR;
> +	    print STDERR "${file}:$.: warning: ";
> +	    print STDERR "Incorrect use of kernel-doc format: $_";
> +	    ++$warnings;
> +	}
> +    }
> +}
> +
>  
>  sub process_file($) {
>      my $file;
> -    my $func;
>      my $initial_section_counter = $section_counter;
>      my ($orig_file) = @_;
>  
> @@ -2014,6 +2090,8 @@ sub process_file($) {
>  	}
>  	# Replace tabs by spaces
>          while ($_ =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {};
> +
> +	# Hand this line to the appropriate state handler
>  	if ($state == STATE_NORMAL) {
>  	    process_normal();
>  	} elsif ($state == STATE_NAME) {
> @@ -2021,72 +2099,15 @@ sub process_file($) {
>  	} elsif ($state == STATE_BODY || $state == STATE_BODY_MAYBE) {
>  	    process_body($file, $_);
>  	} elsif ($state == STATE_INLINE) { # scanning for inline parameters
> -	    # First line (state 1) needs to be a @parameter
> -	    if ($inline_doc_state == STATE_INLINE_NAME && /$doc_inline_sect/o) {
> -		$section = $1;
> -		$contents = $2;
> -                $new_start_line = $.;
> -		if ($contents ne "") {
> -		    while (substr($contents, 0, 1) eq " ") {
> -			$contents = substr($contents, 1);
> -		    }
> -		    $contents .= "\n";
> -		}
> -		$inline_doc_state = STATE_INLINE_TEXT;
> -	    # Documentation block end */
> -	    } elsif (/$doc_inline_end/) {
> -		if (($contents ne "") && ($contents ne "\n")) {
> -		    dump_section($file, $section, $contents);
> -		    $section = $section_default;
> -		    $contents = "";
> -		}
> -		$state = STATE_PROTO;
> -		$inline_doc_state = STATE_INLINE_NA;
> -	    # Regular text
> -	    } elsif (/$doc_content/) {
> -		if ($inline_doc_state == STATE_INLINE_TEXT) {
> -		    $contents .= $1 . "\n";
> -		    # nuke leading blank lines
> -		    if ($contents =~ /^\s*$/) {
> -			$contents = "";
> -		    }
> -		} elsif ($inline_doc_state == STATE_INLINE_NAME) {
> -		    $inline_doc_state = STATE_INLINE_ERROR;
> -		    print STDERR "${file}:$.: warning: ";
> -		    print STDERR "Incorrect use of kernel-doc format: $_";
> -		    ++$warnings;
> -		}
> -	    }
> +	    process_inline($file, $_);
>  	} elsif ($state == STATE_PROTO) {
>  	    process_proto($file, $_);
>  	} elsif ($state == STATE_DOCBLOCK) {
> -		if (/$doc_end/)
> -		{
> -			dump_doc_section($file, $section, $contents);
> -			$section = $section_default;
> -			$contents = "";
> -			$function = "";
> -			%parameterdescs = ();
> -			%parametertypes = ();
> -			@parameterlist = ();
> -			%sections = ();
> -			@sectionlist = ();
> -			$prototype = "";
> -			$state = STATE_NORMAL;
> -		}
> -		elsif (/$doc_content/)
> -		{
> -			if ( $1 eq "" )
> -			{
> -				$contents .= $blankline;
> -			}
> -			else
> -			{
> -				$contents .= $1 . "\n";
> -			}
> -		}
> +	    process_docblock($file, $_);
>  	}
>      }
> +
> +    # Make sure we got something interesting.
>      if ($initial_section_counter == $section_counter) {
>  	if ($output_mode ne "none") {
>  	    print STDERR "${file}:1: warning: no structured comments found\n";
> -- 
> 2.14.3
> 

Re: [PATCH 8/8] docs: kernel-doc: Don't mangle literal code blocks in comments

[Tobin C. Harding]

On Wed, Feb 07, 2018 at 10:26:24AM -0700, Jonathan Corbet wrote:

[snip]

>    if (desperate)
>        run_in_circles();

this is gold :)

Re: [PATCH 7/8] docs: kernel-doc: Finish moving STATE_* code out of process_file()

[Jonathan Corbet]

On Thu, 8 Feb 2018 13:29:53 +1100
"Tobin C. Harding" <me@tobin.cc> wrote:

> It doesn't appear to be introduced by you but the brace positions are
> non-uniform in this patch.
> 
> if
> {
>   ...
> }
> else
> {
>   ...
> }
> 
> instead of
> 
> if {
>   ...
> } else {
>   eee
> }

True, that's worth fixing up while I'm in the neighborhood.  I'll do that
before I commit the set.

Thanks,

jon

Re: [PATCH 1/8] docs: kernel-doc: Get rid of xml_escape() and friends

[Jani Nikula]

On Wed, 07 Feb 2018, Jonathan Corbet <corbet@lwn.net> wrote:
> XML escaping is a worry that came with DocBook, which we no longer have any
> dealings with.  So get rid of the useless xml_escape()/xml_unescape()
> functions.  No change to the generated output.

I think this will break at least the -docbook output option, perhaps
also -html and -html5 options. If you think it's okay to break them,
would it not be better to just axe those off first?

I guess this boils down to, is kernel-doc the script a general purpose
tool, or just a very specific part of the kernel documentation build
process?

FWIW I think the latter, and why don't you throw docbook/html support
out already!


BR,
Jani.


>
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
> ---
>  scripts/kernel-doc | 65 ++++++++----------------------------------------------
>  1 file changed, 9 insertions(+), 56 deletions(-)
>
> diff --git a/scripts/kernel-doc b/scripts/kernel-doc
> index fee8952037b1..5aa4ce211fc6 100755
> --- a/scripts/kernel-doc
> +++ b/scripts/kernel-doc
> @@ -553,10 +553,9 @@ sub output_highlight {
>  	}
>  	if ($line eq ""){
>  	    if (! $output_preformatted) {
> -		print $lineprefix, local_unescape($blankline);
> +		print $lineprefix, $blankline;
>  	    }
>  	} else {
> -	    $line =~ s/\\\\\\/\&/g;
>  	    if ($output_mode eq "man" && substr($line, 0, 1) eq ".") {
>  		print "\\&$line";
>  	    } else {
> @@ -751,9 +750,6 @@ sub output_highlight_rst {
>      my $contents = join "\n",@_;
>      my $line;
>  
> -    # undo the evil effects of xml_escape() earlier
> -    $contents = xml_unescape($contents);
> -
>      eval $dohighlight;
>      die $@ if $@;
>  
> @@ -1422,8 +1418,6 @@ sub push_parameter($$$$) {
>  		}
>  	}
>  
> -	$param = xml_escape($param);
> -
>  	# strip spaces from $param so that it is one continuous string
>  	# on @parameterlist;
>  	# this fixes a problem where check_sections() cannot find
> @@ -1748,47 +1742,6 @@ sub process_proto_type($$) {
>      }
>  }
>  
> -# xml_escape: replace <, >, and & in the text stream;
> -#
> -# however, formatting controls that are generated internally/locally in the
> -# kernel-doc script are not escaped here; instead, they begin life like
> -# $blankline_html (4 of '\' followed by a mnemonic + ':'), then these strings
> -# are converted to their mnemonic-expected output, without the 4 * '\' & ':',
> -# just before actual output; (this is done by local_unescape())
> -sub xml_escape($) {
> -	my $text = shift;
> -	if ($output_mode eq "man") {
> -		return $text;
> -	}
> -	$text =~ s/\&/\\\\\\amp;/g;
> -	$text =~ s/\</\\\\\\lt;/g;
> -	$text =~ s/\>/\\\\\\gt;/g;
> -	return $text;
> -}
> -
> -# xml_unescape: reverse the effects of xml_escape
> -sub xml_unescape($) {
> -	my $text = shift;
> -	if ($output_mode eq "man") {
> -		return $text;
> -	}
> -	$text =~ s/\\\\\\amp;/\&/g;
> -	$text =~ s/\\\\\\lt;/</g;
> -	$text =~ s/\\\\\\gt;/>/g;
> -	return $text;
> -}
> -
> -# convert local escape strings to html
> -# local escape strings look like:  '\\\\menmonic:' (that's 4 backslashes)
> -sub local_unescape($) {
> -	my $text = shift;
> -	if ($output_mode eq "man") {
> -		return $text;
> -	}
> -	$text =~ s/\\\\\\\\lt:/</g;
> -	$text =~ s/\\\\\\\\gt:/>/g;
> -	return $text;
> -}
>  
>  sub map_filename($) {
>      my $file;
> @@ -1889,7 +1842,7 @@ sub process_file($) {
>  		    $descr =~ s/^\s*//;
>  		    $descr =~ s/\s*$//;
>  		    $descr =~ s/\s+/ /g;
> -		    $declaration_purpose = xml_escape($descr);
> +		    $declaration_purpose = $descr;
>  		    $in_purpose = 1;
>  		} else {
>  		    $declaration_purpose = "";
> @@ -1944,7 +1897,7 @@ sub process_file($) {
>  			print STDERR "${file}:$.: warning: contents before sections\n";
>  			++$warnings;
>  		    }
> -		    dump_section($file, $section, xml_escape($contents));
> +		    dump_section($file, $section, $contents);
>  		    $section = $section_default;
>  		}
>  
> @@ -1962,7 +1915,7 @@ sub process_file($) {
>  		$leading_space = undef;
>  	    } elsif (/$doc_end/) {
>  		if (($contents ne "") && ($contents ne "\n")) {
> -		    dump_section($file, $section, xml_escape($contents));
> +		    dump_section($file, $section, $contents);
>  		    $section = $section_default;
>  		    $contents = "";
>  		}
> @@ -1981,7 +1934,7 @@ sub process_file($) {
>  		# @parameter line to signify start of description
>  		if ($1 eq "") {
>  		    if ($section =~ m/^@/ || $section eq $section_context) {
> -			dump_section($file, $section, xml_escape($contents));
> +			dump_section($file, $section, $contents);
>  			$section = $section_default;
>  			$contents = "";
>                          $new_start_line = $.;
> @@ -1992,7 +1945,7 @@ sub process_file($) {
>  		} elsif ($in_purpose == 1) {
>  		    # Continued declaration purpose
>  		    chomp($declaration_purpose);
> -		    $declaration_purpose .= " " . xml_escape($1);
> +		    $declaration_purpose .= " " . $1;
>  		    $declaration_purpose =~ s/\s+/ /g;
>  		} else {
>  		    my $cont = $1;
> @@ -2030,7 +1983,7 @@ sub process_file($) {
>  	    # Documentation block end */
>  	    } elsif (/$doc_inline_end/) {
>  		if (($contents ne "") && ($contents ne "\n")) {
> -		    dump_section($file, $section, xml_escape($contents));
> +		    dump_section($file, $section, $contents);
>  		    $section = $section_default;
>  		    $contents = "";
>  		}
> @@ -2057,7 +2010,7 @@ sub process_file($) {
>  		$contents = $2;
>  		if ($contents ne "") {
>  		    $contents .= "\n";
> -		    dump_section($file, $section, xml_escape($contents));
> +		    dump_section($file, $section, $contents);
>  		    $section = $section_default;
>  		    $contents = "";
>  		}
> @@ -2072,7 +2025,7 @@ sub process_file($) {
>  	} elsif ($state == STATE_DOCBLOCK) {
>  		if (/$doc_end/)
>  		{
> -			dump_doc_section($file, $section, xml_escape($contents));
> +			dump_doc_section($file, $section, $contents);
>  			$section = $section_default;
>  			$contents = "";
>  			$function = "";

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH 2/8] docs: kernel-doc: Rename and split STATE_FIELD

[Jani Nikula]

On Wed, 07 Feb 2018, Jonathan Corbet <corbet@lwn.net> wrote:
> STATE_FIELD describes a parser state that can handle any part of a
> kerneldoc comment body; rename it to STATE_BODY to reflect that.
>
> The $in_purpose variable was a hidden substate of STATE_FIELD; get rid of
> it and make a proper state (STATE_BODY_MAYBE) instead.  This will make the
> subsequent process_file() splitup easier.
>
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>

I really wanted to avoid leaving behind any evidence that I've ever
reviewed perl, but my sympathy for you updating the script won. But just
barely.

Reviewed-by: Jani Nikula <jani.nikula@intel.com>

> ---
>  scripts/kernel-doc | 22 +++++++++++-----------
>  1 file changed, 11 insertions(+), 11 deletions(-)
>
> diff --git a/scripts/kernel-doc b/scripts/kernel-doc
> index 5aa4ce211fc6..ad30c52f91ef 100755
> --- a/scripts/kernel-doc
> +++ b/scripts/kernel-doc
> @@ -328,10 +328,11 @@ my $lineprefix="";
>  use constant {
>      STATE_NORMAL        => 0, # normal code
>      STATE_NAME          => 1, # looking for function name
> -    STATE_FIELD         => 2, # scanning field start
> -    STATE_PROTO         => 3, # scanning prototype
> -    STATE_DOCBLOCK      => 4, # documentation block
> -    STATE_INLINE        => 5, # gathering documentation outside main block
> +    STATE_BODY_MAYBE    => 2, # body - or maybe more description
> +    STATE_BODY          => 3, # the body of the comment
> +    STATE_PROTO         => 4, # scanning prototype
> +    STATE_DOCBLOCK      => 5, # documentation block
> +    STATE_INLINE        => 6, # gathering documentation outside main block
>  };
>  my $state;
>  my $in_doc_sect;
> @@ -1784,7 +1785,6 @@ sub process_file($) {
>      my $identifier;
>      my $func;
>      my $descr;
> -    my $in_purpose = 0;
>      my $initial_section_counter = $section_counter;
>      my ($orig_file) = @_;
>      my $leading_space;
> @@ -1830,7 +1830,7 @@ sub process_file($) {
>  		    $identifier = $1;
>  		}
>  
> -		$state = STATE_FIELD;
> +		$state = STATE_BODY;
>  		# if there's no @param blocks need to set up default section
>  		# here
>  		$contents = "";
> @@ -1843,7 +1843,7 @@ sub process_file($) {
>  		    $descr =~ s/\s*$//;
>  		    $descr =~ s/\s+/ /g;
>  		    $declaration_purpose = $descr;
> -		    $in_purpose = 1;
> +		    $state = STATE_BODY_MAYBE;
>  		} else {
>  		    $declaration_purpose = "";
>  		}
> @@ -1875,7 +1875,7 @@ sub process_file($) {
>  		++$warnings;
>  		$state = STATE_NORMAL;
>  	    }
> -	} elsif ($state == STATE_FIELD) {	# look for head: lines, and include content
> +	} elsif ($state == STATE_BODY || $state == STATE_BODY_MAYBE) {
>  	    if (/$doc_sect/i) { # case insensitive for supported section names
>  		$newsection = $1;
>  		$newcontents = $2;
> @@ -1902,7 +1902,7 @@ sub process_file($) {
>  		}
>  
>  		$in_doc_sect = 1;
> -		$in_purpose = 0;
> +		$state = STATE_BODY;
>  		$contents = $newcontents;
>                  $new_start_line = $.;
>  		while (substr($contents, 0, 1) eq " ") {
> @@ -1941,8 +1941,8 @@ sub process_file($) {
>  		    } else {
>  			$contents .= "\n";
>  		    }
> -		    $in_purpose = 0;
> -		} elsif ($in_purpose == 1) {
> +		    $state = STATE_BODY;
> +		} elsif ($state == STATE_BODY_MAYBE) {
>  		    # Continued declaration purpose
>  		    chomp($declaration_purpose);
>  		    $declaration_purpose .= " " . $1;

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH 3/8] docs: kernel-doc: Move STATE_NORMAL processing into its own function

[Jani Nikula]

On Wed, 07 Feb 2018, Jonathan Corbet <corbet@lwn.net> wrote:
> Begin the process of splitting up the nearly 500-line process_file()
> function by moving STATE_NORMAL processing to a separate function.
>
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
> ---
>  scripts/kernel-doc | 21 ++++++++++++++++-----
>  1 file changed, 16 insertions(+), 5 deletions(-)
>
> diff --git a/scripts/kernel-doc b/scripts/kernel-doc
> index ad30c52f91ef..65150b7c8472 100755
> --- a/scripts/kernel-doc
> +++ b/scripts/kernel-doc
> @@ -1780,6 +1780,21 @@ sub process_export_file($) {
>      close(IN);
>  }
>  
> +#
> +# Parsers for the various processing states.
> +#
> +# STATE_NORMAL: looking for the /** to begin everything.
> +#
> +sub process_normal() {
> +    if (/$doc_start/o) {
> +	$state = STATE_NAME;	# next line is always the function name
> +	$in_doc_sect = 0;
> +	$declaration_start_line = $. + 1;
> +    }
> +}
> +
> +
> +

Superfluous whitespace, but fixed in the next patch so I'll look the
other way.

Reviewed-by: Jani Nikula <jani.nikula@intel.com>

>  sub process_file($) {
>      my $file;
>      my $identifier;
> @@ -1807,11 +1822,7 @@ sub process_file($) {
>  	# Replace tabs by spaces
>          while ($_ =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {};
>  	if ($state == STATE_NORMAL) {
> -	    if (/$doc_start/o) {
> -		$state = STATE_NAME;	# next line is always the function name
> -		$in_doc_sect = 0;
> -		$declaration_start_line = $. + 1;
> -	    }
> +	    process_normal();
>  	} elsif ($state == STATE_NAME) {# this line is the function name (always)
>  	    if (/$doc_block/o) {
>  		$state = STATE_DOCBLOCK;

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH 4/8] docs: kernel-doc: Move STATE_NAME processing into its own function

[Jani Nikula]

On Wed, 07 Feb 2018, Jonathan Corbet <corbet@lwn.net> wrote:
> Move this code out of process_file() in the name of readability and
> maintainability.
>
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>

I admit I don't fully understand the semantics of perl parameter
passing, but looks okay.

Reviewed-by: Jani Nikula <jani.nikula@intel.com>

> ---
>  scripts/kernel-doc | 137 ++++++++++++++++++++++++++++-------------------------
>  1 file changed, 72 insertions(+), 65 deletions(-)
>
> diff --git a/scripts/kernel-doc b/scripts/kernel-doc
> index 65150b7c8472..a27c7016f72d 100755
> --- a/scripts/kernel-doc
> +++ b/scripts/kernel-doc
> @@ -1793,13 +1793,81 @@ sub process_normal() {
>      }
>  }
>  
> +#
> +# STATE_NAME: Looking for the "name - description" line
> +#
> +sub process_name($$) {
> +    my $file = shift;
> +    my $identifier;
> +    my $descr;
> +
> +    if (/$doc_block/o) {
> +	$state = STATE_DOCBLOCK;
> +	$contents = "";
> +	$new_start_line = $. + 1;
> +
> +	if ( $1 eq "" ) {
> +	    $section = $section_intro;
> +	} else {
> +	    $section = $1;
> +	}
> +    }
> +    elsif (/$doc_decl/o) {
> +	$identifier = $1;
> +	if (/\s*([\w\s]+?)\s*-/) {
> +	    $identifier = $1;
> +	}
>  
> +	$state = STATE_BODY;
> +	# if there's no @param blocks need to set up default section
> +	# here
> +	$contents = "";
> +	$section = $section_default;
> +	$new_start_line = $. + 1;
> +	if (/-(.*)/) {
> +	    # strip leading/trailing/multiple spaces
> +	    $descr= $1;
> +	    $descr =~ s/^\s*//;
> +	    $descr =~ s/\s*$//;
> +	    $descr =~ s/\s+/ /g;
> +	    $declaration_purpose = $descr;
> +	    $state = STATE_BODY_MAYBE;
> +	} else {
> +	    $declaration_purpose = "";
> +	}
> +
> +	if (($declaration_purpose eq "") && $verbose) {
> +	    print STDERR "${file}:$.: warning: missing initial short description on line:\n";
> +	    print STDERR $_;
> +	    ++$warnings;
> +	}
> +
> +	if ($identifier =~ m/^struct/) {
> +	    $decl_type = 'struct';
> +	} elsif ($identifier =~ m/^union/) {
> +	    $decl_type = 'union';
> +	} elsif ($identifier =~ m/^enum/) {
> +	    $decl_type = 'enum';
> +	} elsif ($identifier =~ m/^typedef/) {
> +	    $decl_type = 'typedef';
> +	} else {
> +	    $decl_type = 'function';
> +	}
> +
> +	if ($verbose) {
> +	    print STDERR "${file}:$.: info: Scanning doc for $identifier\n";
> +	}
> +    } else {
> +	print STDERR "${file}:$.: warning: Cannot understand $_ on line $.",
> +	    " - I thought it was a doc line\n";
> +	++$warnings;
> +	$state = STATE_NORMAL;
> +    }
> +}
>  
>  sub process_file($) {
>      my $file;
> -    my $identifier;
>      my $func;
> -    my $descr;
>      my $initial_section_counter = $section_counter;
>      my ($orig_file) = @_;
>      my $leading_space;
> @@ -1823,69 +1891,8 @@ sub process_file($) {
>          while ($_ =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {};
>  	if ($state == STATE_NORMAL) {
>  	    process_normal();
> -	} elsif ($state == STATE_NAME) {# this line is the function name (always)
> -	    if (/$doc_block/o) {
> -		$state = STATE_DOCBLOCK;
> -		$contents = "";
> -                $new_start_line = $. + 1;
> -
> -		if ( $1 eq "" ) {
> -			$section = $section_intro;
> -		} else {
> -			$section = $1;
> -		}
> -	    }
> -	    elsif (/$doc_decl/o) {
> -		$identifier = $1;
> -		if (/\s*([\w\s]+?)\s*-/) {
> -		    $identifier = $1;
> -		}
> -
> -		$state = STATE_BODY;
> -		# if there's no @param blocks need to set up default section
> -		# here
> -		$contents = "";
> -		$section = $section_default;
> -		$new_start_line = $. + 1;
> -		if (/-(.*)/) {
> -		    # strip leading/trailing/multiple spaces
> -		    $descr= $1;
> -		    $descr =~ s/^\s*//;
> -		    $descr =~ s/\s*$//;
> -		    $descr =~ s/\s+/ /g;
> -		    $declaration_purpose = $descr;
> -		    $state = STATE_BODY_MAYBE;
> -		} else {
> -		    $declaration_purpose = "";
> -		}
> -
> -		if (($declaration_purpose eq "") && $verbose) {
> -			print STDERR "${file}:$.: warning: missing initial short description on line:\n";
> -			print STDERR $_;
> -			++$warnings;
> -		}
> -
> -		if ($identifier =~ m/^struct/) {
> -		    $decl_type = 'struct';
> -		} elsif ($identifier =~ m/^union/) {
> -		    $decl_type = 'union';
> -		} elsif ($identifier =~ m/^enum/) {
> -		    $decl_type = 'enum';
> -		} elsif ($identifier =~ m/^typedef/) {
> -		    $decl_type = 'typedef';
> -		} else {
> -		    $decl_type = 'function';
> -		}
> -
> -		if ($verbose) {
> -		    print STDERR "${file}:$.: info: Scanning doc for $identifier\n";
> -		}
> -	    } else {
> -		print STDERR "${file}:$.: warning: Cannot understand $_ on line $.",
> -		" - I thought it was a doc line\n";
> -		++$warnings;
> -		$state = STATE_NORMAL;
> -	    }
> +	} elsif ($state == STATE_NAME) {
> +	    process_name($file, $_);
>  	} elsif ($state == STATE_BODY || $state == STATE_BODY_MAYBE) {
>  	    if (/$doc_sect/i) { # case insensitive for supported section names
>  		$newsection = $1;

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH 5/8] docs: kernel-doc: Move STATE_BODY processing to a separate function

[Jani Nikula]

On Wed, 07 Feb 2018, Jonathan Corbet <corbet@lwn.net> wrote:
> Also group the pseudo-global $leading_space variable with its peers.
>
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
> ---
>  scripts/kernel-doc | 193 ++++++++++++++++++++++++++++-------------------------
>  1 file changed, 101 insertions(+), 92 deletions(-)
>
> diff --git a/scripts/kernel-doc b/scripts/kernel-doc
> index a27c7016f72d..a6a7bb46ea29 100755
> --- a/scripts/kernel-doc
> +++ b/scripts/kernel-doc
> @@ -336,6 +336,7 @@ use constant {
>  };
>  my $state;
>  my $in_doc_sect;
> +my $leading_space;
>  
>  # Inline documentation state
>  use constant {
> @@ -1865,12 +1866,110 @@ sub process_name($$) {
>      }
>  }
>  
> +
> +#
> +# STATE_BODY and STATE_BODY_MAYBE: the bulk of a kerneldoc comment.
> +#
> +sub process_body($$) {
> +    my $file = shift;
> +
> +    if (/$doc_sect/i) { # case insensitive for supported section names
> +	$newsection = $1;
> +	$newcontents = $2;
> +
> +	# map the supported section names to the canonical names
> +	if ($newsection =~ m/^description$/i) {
> +	    $newsection = $section_default;
> +	} elsif ($newsection =~ m/^context$/i) {
> +	    $newsection = $section_context;
> +	} elsif ($newsection =~ m/^returns?$/i) {
> +	    $newsection = $section_return;
> +	} elsif ($newsection =~ m/^\@return$/) {
> +	    # special: @return is a section, not a param description
> +	    $newsection = $section_return;
> +	}
> +
> +	if (($contents ne "") && ($contents ne "\n")) {
> +	    if (!$in_doc_sect && $verbose) {
> +		print STDERR "${file}:$.: warning: contents before sections\n";
> +		++$warnings;
> +	    }
> +	    dump_section($file, $section, $contents);
> +	    $section = $section_default;
> +	}
> +
> +	$in_doc_sect = 1;
> +	$state = STATE_BODY;
> +	$contents = $newcontents;
> +	$new_start_line = $.;
> +	while (substr($contents, 0, 1) eq " ") {
> +	    $contents = substr($contents, 1);
> +	}
> +	if ($contents ne "") {
> +	    $contents .= "\n";
> +	}
> +	$section = $newsection;
> +	$leading_space = undef;
> +    } elsif (/$doc_end/) {
> +	if (($contents ne "") && ($contents ne "\n")) {
> +	    dump_section($file, $section, $contents);
> +	    $section = $section_default;
> +	    $contents = "";
> +	}
> +	# look for doc_com + <text> + doc_end:
> +	if ($_ =~ m'\s*\*\s*[a-zA-Z_0-9:\.]+\*/') {
> +	    print STDERR "${file}:$.: warning: suspicious ending line: $_";
> +	    ++$warnings;
> +	}
> +
> +	$prototype = "";
> +	$state = STATE_PROTO;
> +	$brcount = 0;
> +    } elsif (/$doc_content/) {
> +	# miguel-style comment kludge, look for blank lines after
> +	# @parameter line to signify start of description

The "miguel-style" always intrigued me, but its origin predates git
history. Does anyone know?

Reviewed-by: Jani Nikula <jani.nikula@intel.com>

> +	if ($1 eq "") {
> +	    if ($section =~ m/^@/ || $section eq $section_context) {
> +		dump_section($file, $section, $contents);
> +		$section = $section_default;
> +		$contents = "";
> +		$new_start_line = $.;
> +	    } else {
> +		$contents .= "\n";
> +	    }
> +	    $state = STATE_BODY;
> +	} elsif ($state == STATE_BODY_MAYBE) {
> +	    # Continued declaration purpose
> +	    chomp($declaration_purpose);
> +	    $declaration_purpose .= " " . $1;
> +	    $declaration_purpose =~ s/\s+/ /g;
> +	} else {
> +	    my $cont = $1;
> +	    if ($section =~ m/^@/ || $section eq $section_context) {
> +		if (!defined $leading_space) {
> +		    if ($cont =~ m/^(\s+)/) {
> +			$leading_space = $1;
> +		    } else {
> +			$leading_space = "";
> +		    }
> +		}
> +		$cont =~ s/^$leading_space//;
> +	    }
> +	    $contents .= $cont . "\n";
> +	}
> +    } else {
> +	# i dont know - bad line?  ignore.
> +	print STDERR "${file}:$.: warning: bad line: $_";
> +	++$warnings;
> +    }
> +}
> +
> +
>  sub process_file($) {
>      my $file;
>      my $func;
>      my $initial_section_counter = $section_counter;
>      my ($orig_file) = @_;
> -    my $leading_space;
>  
>      $file = map_filename($orig_file);
>  
> @@ -1894,97 +1993,7 @@ sub process_file($) {
>  	} elsif ($state == STATE_NAME) {
>  	    process_name($file, $_);
>  	} elsif ($state == STATE_BODY || $state == STATE_BODY_MAYBE) {
> -	    if (/$doc_sect/i) { # case insensitive for supported section names
> -		$newsection = $1;
> -		$newcontents = $2;
> -
> -		# map the supported section names to the canonical names
> -		if ($newsection =~ m/^description$/i) {
> -		    $newsection = $section_default;
> -		} elsif ($newsection =~ m/^context$/i) {
> -		    $newsection = $section_context;
> -		} elsif ($newsection =~ m/^returns?$/i) {
> -		    $newsection = $section_return;
> -		} elsif ($newsection =~ m/^\@return$/) {
> -		    # special: @return is a section, not a param description
> -		    $newsection = $section_return;
> -		}
> -
> -		if (($contents ne "") && ($contents ne "\n")) {
> -		    if (!$in_doc_sect && $verbose) {
> -			print STDERR "${file}:$.: warning: contents before sections\n";
> -			++$warnings;
> -		    }
> -		    dump_section($file, $section, $contents);
> -		    $section = $section_default;
> -		}
> -
> -		$in_doc_sect = 1;
> -		$state = STATE_BODY;
> -		$contents = $newcontents;
> -                $new_start_line = $.;
> -		while (substr($contents, 0, 1) eq " ") {
> -		    $contents = substr($contents, 1);
> -		}
> -		if ($contents ne "") {
> -		    $contents .= "\n";
> -		}
> -		$section = $newsection;
> -		$leading_space = undef;
> -	    } elsif (/$doc_end/) {
> -		if (($contents ne "") && ($contents ne "\n")) {
> -		    dump_section($file, $section, $contents);
> -		    $section = $section_default;
> -		    $contents = "";
> -		}
> -		# look for doc_com + <text> + doc_end:
> -		if ($_ =~ m'\s*\*\s*[a-zA-Z_0-9:\.]+\*/') {
> -		    print STDERR "${file}:$.: warning: suspicious ending line: $_";
> -		    ++$warnings;
> -		}
> -
> -		$prototype = "";
> -		$state = STATE_PROTO;
> -		$brcount = 0;
> -#		print STDERR "end of doc comment, looking for prototype\n";
> -	    } elsif (/$doc_content/) {
> -		# miguel-style comment kludge, look for blank lines after
> -		# @parameter line to signify start of description
> -		if ($1 eq "") {
> -		    if ($section =~ m/^@/ || $section eq $section_context) {
> -			dump_section($file, $section, $contents);
> -			$section = $section_default;
> -			$contents = "";
> -                        $new_start_line = $.;
> -		    } else {
> -			$contents .= "\n";
> -		    }
> -		    $state = STATE_BODY;
> -		} elsif ($state == STATE_BODY_MAYBE) {
> -		    # Continued declaration purpose
> -		    chomp($declaration_purpose);
> -		    $declaration_purpose .= " " . $1;
> -		    $declaration_purpose =~ s/\s+/ /g;
> -		} else {
> -		    my $cont = $1;
> -		    if ($section =~ m/^@/ || $section eq $section_context) {
> -			if (!defined $leading_space) {
> -			    if ($cont =~ m/^(\s+)/) {
> -				$leading_space = $1;
> -			    } else {
> -				$leading_space = "";
> -			    }
> -			}
> -
> -			$cont =~ s/^$leading_space//;
> -		    }
> -		    $contents .= $cont . "\n";
> -		}
> -	    } else {
> -		# i dont know - bad line?  ignore.
> -		print STDERR "${file}:$.: warning: bad line: $_";
> -		++$warnings;
> -	    }
> +	    process_body($file, $_);
>  	} elsif ($state == STATE_INLINE) { # scanning for inline parameters
>  	    # First line (state 1) needs to be a @parameter
>  	    if ($inline_doc_state == STATE_INLINE_NAME && /$doc_inline_sect/o) {

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH 6/8] docs: kernel-doc: Move STATE_PROTO processing into its own function

[Jani Nikula]

On Wed, 07 Feb 2018, Jonathan Corbet <corbet@lwn.net> wrote:
> Move the top-level prototype-processing code out of process_file().
>
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>

Reviewed-by: Jani Nikula <jani.nikula@intel.com>

> ---
>  scripts/kernel-doc | 46 ++++++++++++++++++++++++++++------------------
>  1 file changed, 28 insertions(+), 18 deletions(-)
>
> diff --git a/scripts/kernel-doc b/scripts/kernel-doc
> index a6a7bb46ea29..2deddb876156 100755
> --- a/scripts/kernel-doc
> +++ b/scripts/kernel-doc
> @@ -1965,6 +1965,32 @@ sub process_body($$) {
>  }
>  
>  
> +#
> +# STATE_PROTO: reading a function/whatever prototype.
> +#
> +sub process_proto($$) {
> +    my $file = shift;
> +
> +    if (/$doc_inline_oneline/) {
> +	$section = $1;
> +	$contents = $2;
> +	if ($contents ne "") {
> +	    $contents .= "\n";
> +	    dump_section($file, $section, $contents);
> +	    $section = $section_default;
> +	    $contents = "";
> +	}
> +    } elsif (/$doc_inline_start/) {
> +	$state = STATE_INLINE;
> +	$inline_doc_state = STATE_INLINE_NAME;
> +    } elsif ($decl_type eq 'function') {
> +	process_proto_function($_, $file);
> +    } else {
> +	process_proto_type($_, $file);
> +    }
> +}
> +
> +
>  sub process_file($) {
>      my $file;
>      my $func;
> @@ -2031,24 +2057,8 @@ sub process_file($) {
>  		    ++$warnings;
>  		}
>  	    }
> -	} elsif ($state == STATE_PROTO) {	# scanning for function '{' (end of prototype)
> -	    if (/$doc_inline_oneline/) {
> -		$section = $1;
> -		$contents = $2;
> -		if ($contents ne "") {
> -		    $contents .= "\n";
> -		    dump_section($file, $section, $contents);
> -		    $section = $section_default;
> -		    $contents = "";
> -		}
> -	    } elsif (/$doc_inline_start/) {
> -		$state = STATE_INLINE;
> -		$inline_doc_state = STATE_INLINE_NAME;
> -	    } elsif ($decl_type eq 'function') {
> -		process_proto_function($_, $file);
> -	    } else {
> -		process_proto_type($_, $file);
> -	    }
> +	} elsif ($state == STATE_PROTO) {
> +	    process_proto($file, $_);
>  	} elsif ($state == STATE_DOCBLOCK) {
>  		if (/$doc_end/)
>  		{

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH 7/8] docs: kernel-doc: Finish moving STATE_* code out of process_file()

[Jani Nikula]

On Thu, 08 Feb 2018, Jonathan Corbet <corbet@lwn.net> wrote:
> On Thu, 8 Feb 2018 13:29:53 +1100
> "Tobin C. Harding" <me@tobin.cc> wrote:
>
>> It doesn't appear to be introduced by you but the brace positions are
>> non-uniform in this patch.
>> 
>> if
>> {
>>   ...
>> }
>> else
>> {
>>   ...
>> }
>> 
>> instead of
>> 
>> if {
>>   ...
>> } else {
>>   eee
>> }
>
> True, that's worth fixing up while I'm in the neighborhood.  I'll do that
> before I commit the set.

With that,

Reviewed-by: Jani Nikula <jani.nikula@intel.com>


>
> Thanks,
>
> jon
> --
> To unsubscribe from this list: send the line "unsubscribe linux-doc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH 8/8] docs: kernel-doc: Don't mangle literal code blocks in comments

[Jani Nikula]

On Wed, 07 Feb 2018, Jonathan Corbet <corbet@lwn.net> wrote:
> It can be useful to put code snippets into kerneldoc comments; that can be
> done with the "::" operator at the end of a line like this::
>
>    if (desperate)
>        run_in_circles();
>
> kernel-doc currently fails to understand these literal blocks and applies
> its normal markup to them, which is then treated as literal by sphinx.  The
> result is unsightly markup instead of a useful code snippet.
>
> Apply a hack to the output code to recognize literal blocks and avoid
> performing any special markup on them.  It's ugly, but that means it fits
> in well with the rest of the script.
>
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
> ---
>  scripts/kernel-doc | 55 +++++++++++++++++++++++++++++++++++++++++++++++++-----
>  1 file changed, 50 insertions(+), 5 deletions(-)
>
> diff --git a/scripts/kernel-doc b/scripts/kernel-doc
> index c6c9370a1e49..c984f82cb897 100755
> --- a/scripts/kernel-doc
> +++ b/scripts/kernel-doc
> @@ -748,14 +748,59 @@ sub output_blockhead_rst(%) {
>      }
>  }
>  
> -sub output_highlight_rst {
> -    my $contents = join "\n",@_;
> -    my $line;
> -
> +#
> +# Apply the RST highlights to a sub-block of text.
> +#   
> +sub highlight_block($) {
> +    # The dohighlight kludge requires the text be called $contents
> +    my $contents = shift;
>      eval $dohighlight;
>      die $@ if $@;
> +    return $contents;
> +}
>  
> -    foreach $line (split "\n", $contents) {
> +sub output_highlight_rst {
> +    my $input = join "\n",@_;
> +    my $output = "";
> +    my $line;
> +    my $in_literal = 0;
> +    my $litprefix;
> +    my $block = "";
> +
> +    # The "dohighlight" hack requires that the data be called "$contents"
> +    foreach $line (split "\n",$input) {
> +	#
> +	# If we're in a literal block, see if we should drop out
> +	# of it.  Otherwise pass the line straight through unmunged.
> +	#
> +	if ($in_literal) {
> +	    if (! ($line =~ /$litprefix/ || $line =~ /^\s*$/)) {
> +		$in_literal = 0;
> +	    }
> +	    else {
> +		$output .= $line . "\n";
> +	    }
> +	}
> +	#
> +	# Not in a literal block (or just dropped out)
> +	#
> +	if (! $in_literal) {
> +	    $block .= $line . "\n";
> +	    if ($line =~ /^[^.].*::$/) {

I think you should also add "code-block:: <language>" to the
regexp. There are only a few uses now, but I think someone's bound to
hit the same problem with those.

Perhaps also extract the regexp to a variable with a self-documenting
name.

Thanks for doing this. Not that I like it, but as you say, it fits right
in the script. I have some ideas on how to do all of the highlights
nicely as post-processing in the Sphinx extension, but we need this now
and not somewhere in the distant future.

BR,
Jani.

> +		$in_literal = 1;
> +		# Note current indentation - we'll go as long as it's deeper.
> +		$line =~ /^(\s*)/;
> +		$litprefix = '^' . $1 . ' ';
> +		$output .= highlight_block($block);
> +		$block = ""
> +	    }
> +	}
> +    }
> +
> +    if ($block) {
> +	$output .= highlight_block($block);
> +    }
> +    foreach $line (split "\n", $output) {
>  	print $lineprefix . $line . "\n";
>      }
>  }

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH 1/8] docs: kernel-doc: Get rid of xml_escape() and friends

[Jonathan Corbet]

On Fri, 09 Feb 2018 11:09:27 +0200
Jani Nikula <jani.nikula@linux.intel.com> wrote:

> On Wed, 07 Feb 2018, Jonathan Corbet <corbet@lwn.net> wrote:
> > XML escaping is a worry that came with DocBook, which we no longer have any
> > dealings with.  So get rid of the useless xml_escape()/xml_unescape()
> > functions.  No change to the generated output.  
> 
> I think this will break at least the -docbook output option, perhaps
> also -html and -html5 options. If you think it's okay to break them,
> would it not be better to just axe those off first?
> 
> I guess this boils down to, is kernel-doc the script a general purpose
> tool, or just a very specific part of the kernel documentation build
> process?
> 
> FWIW I think the latter, and why don't you throw docbook/html support
> out already!

See commit b05142675310 :)

Thanks,

jon

Re: [PATCH 5/8] docs: kernel-doc: Move STATE_BODY processing to a separate function

[Linus Torvalds]

On Fri, Feb 9, 2018 at 1:32 AM, Jani Nikula <jani.nikula@linux.intel.com> wrote:
>> +     # miguel-style comment kludge, look for blank lines after
>> +     # @parameter line to signify start of description
>
> The "miguel-style" always intrigued me, but its origin predates git
> history. Does anyone know?

It came with the original script in 2.3.52-pre1, somewhere around March 2000.

(Historical footnote: there was no actual 2.3.52 ever released - it
was supposed turn into 2.4, but there was a long line of 2.3.99-pre
kernels before that then finally happened early 2001).

But no, back then we didn't track things well enough to have an actual
reason. And even with git, we probably wouldn't have had a reason
since it came in the initial patch.

You'd have to go search emails if you really care. That "around March
2000" would at least give you a starting point for searches.

               Linus

Re: [PATCH 5/8] docs: kernel-doc: Move STATE_BODY processing to a separate function

[Jani Nikula]

On Fri, 09 Feb 2018, Linus Torvalds <torvalds@linux-foundation.org> wrote:
> On Fri, Feb 9, 2018 at 1:32 AM, Jani Nikula <jani.nikula@linux.intel.com> wrote:
>>> +     # miguel-style comment kludge, look for blank lines after
>>> +     # @parameter line to signify start of description
>>
>> The "miguel-style" always intrigued me, but its origin predates git
>> history. Does anyone know?
>
> It came with the original script in 2.3.52-pre1, somewhere around March 2000.
>
> (Historical footnote: there was no actual 2.3.52 ever released - it
> was supposed turn into 2.4, but there was a long line of 2.3.99-pre
> kernels before that then finally happened early 2001).
>
> But no, back then we didn't track things well enough to have an actual
> reason. And even with git, we probably wouldn't have had a reason
> since it came in the initial patch.
>
> You'd have to go search emails if you really care. That "around March
> 2000" would at least give you a starting point for searches.

Thanks for the background digging! I had a cursory look at the list
archives, nothing much jumped at me but I found that the script was
originally called gdoc. A little googling revealed that other forks of
the script predating inclusion to the kernel also included the comment.

I'll bet whoever added that didn't think someone would wonder about it
two decades later.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [PATCH 8/8] docs: kernel-doc: Don't mangle literal code blocks in comments

[Markus Heiser]

> Am 07.02.2018 um 18:26 schrieb Jonathan Corbet <corbet@lwn.net>:
> 
> It can be useful to put code snippets into kerneldoc comments; that can be
> done with the "::" operator at the end of a line like this::
> 
>  if (desperate)
>      run_in_circles();
> 
> kernel-doc currently fails to understand these literal blocks and applies
> its normal markup to them, which is then treated as literal by sphinx.  The
> result is unsightly markup instead of a useful code snippet.
> 
> Apply a hack to the output code to recognize literal blocks and avoid
> performing any special markup on them.  It's ugly, but that means it fits
> in well with the rest of the script.
> 
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>


[...]

FYI; added similar patch to python version of kernel-doc:

 https://github.com/return42/linuxdoc/commit/3205b8a68

may you like to add regexpr for code-block directive to your patch (jani mentioned already).

-- Markus --