[Ksummit-discuss] [TECH TOPIC] Documentation

[Ksummit-discuss] [TECH TOPIC] Documentation

[Jonathan Corbet]

What could be more fun than talking about kernel documentation?  Things we
could get into:

 - The state of the RST transition, what remains to be done, whether it's
   all just useless churn that makes the documentation worse, etc.

 - Things we'd like to improve in the documentation toolchain.

 - Overall organization of Documentation/ and moving docs when the need
   arises.  It seems I end up fighting about this more than just about
   anything else, but I think it's important to organize our docs for the
   convenience of the people using them.

 - The ultimate vision for kernel docs (for now).  RST conversion and
   imposing some organization are important, but they will not,
   themselves, give us a coherent set of documentation.  What can we do to
   have documentation that is useful, current, and maintainable, rather
   than the dusty attic we have now?

jon

Re: [Ksummit-discuss] [TECH TOPIC] Documentation

[Shuah Khan]

On 6/12/19 8:54 AM, Jonathan Corbet wrote:
> What could be more fun than talking about kernel documentation?  Things we
> could get into:
> 
>   - The state of the RST transition, what remains to be done, whether it's
>     all just useless churn that makes the documentation worse, etc.
> 
>   - Things we'd like to improve in the documentation toolchain.
> 
>   - Overall organization of Documentation/ and moving docs when the need
>     arises.  It seems I end up fighting about this more than just about
>     anything else, but I think it's important to organize our docs for the
>     convenience of the people using them.
> 
>   - The ultimate vision for kernel docs (for now).  RST conversion and
>     imposing some organization are important, but they will not,
>     themselves, give us a coherent set of documentation.  What can we do to
>     have documentation that is useful, current, and maintainable, rather
>     than the dusty attic we have now?
> 

The first step is identifying the ones that are out of date and/or
incorrect. I think this is probably the most time consuming part
and the second is start updating.

I am currently putting together a task list for mentorship program
with a goal to create tasks that would more meaningful and helpful
instead of whitespace patches.

I will gladly add documentation tasks to the list to help improve
the documentation.

thanks,
-- Shuah

Re: [Ksummit-discuss] [TECH TOPIC] Documentation

[Martin K. Petersen]

Shuah,

> I am currently putting together a task list for mentorship program
> with a goal to create tasks that would more meaningful and helpful
> instead of whitespace patches.

If somebody is looking for meaningful documentation work, I have a
substantial project in need of volunteers: The target stack wiki is
stale, unmaintained, and often inaccessible. I would love for somebody
to convert all that information to RST so we can stick it in the kernel
and keep it in sync with the code changes.

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [Ksummit-discuss] [TECH TOPIC] Documentation

[Shuah Khan]

On 6/12/19 1:12 PM, Martin K. Petersen wrote:
> 
> Shuah,
> 
>> I am currently putting together a task list for mentorship program
>> with a goal to create tasks that would more meaningful and helpful
>> instead of whitespace patches.
> 
> If somebody is looking for meaningful documentation work, I have a
> substantial project in need of volunteers: The target stack wiki is
> stale, unmaintained, and often inaccessible. I would love for somebody
> to convert all that information to RST so we can stick it in the kernel
> and keep it in sync with the code changes.
> 

Awesome. Please send the project description to me. Feel free to break
it up as needed. We can chat offline about it.

thanks,
-- Shuah

Re: [Ksummit-discuss] [TECH TOPIC] Documentation

[Mauro Carvalho Chehab]

Em Wed, 12 Jun 2019 12:22:55 -0600
Shuah Khan <skhan@linuxfoundation.org> escreveu:

> On 6/12/19 8:54 AM, Jonathan Corbet wrote:
> > What could be more fun than talking about kernel documentation?  Things we
> > could get into:
> > 
> >   - The state of the RST transition, what remains to be done, whether it's
> >     all just useless churn that makes the documentation worse, etc.
> > 
> >   - Things we'd like to improve in the documentation toolchain.
> > 
> >   - Overall organization of Documentation/ and moving docs when the need
> >     arises.  It seems I end up fighting about this more than just about
> >     anything else, but I think it's important to organize our docs for the
> >     convenience of the people using them.
> > 
> >   - The ultimate vision for kernel docs (for now).  RST conversion and
> >     imposing some organization are important, but they will not,
> >     themselves, give us a coherent set of documentation.  What can we do to
> >     have documentation that is useful, current, and maintainable, rather
> >     than the dusty attic we have now?
> >   
> 
> The first step is identifying the ones that are out of date and/or
> incorrect. I think this is probably the most time consuming part
> and the second is start updating.

I would do just the opposite: convert the documentation, then
update it.

The reason is that, when someone sends a patch to the documentation,
people tend to look more closely on it, and we start receiving
updates for it.

Ok, this doesn't happen every time, but it is still worth.

There's another thing to consider here: there are lots of docs
written for "stable" subsystems, in the sense that the documented 
code doesn't change for a lot of time (except for trivial patches
not directly related to it.

While doing large chunks of code conversion, I noticed that there
are lots of still valid documents that fit on this.

There's a big issue with those "stable" subsystems: they tend
to not have any active maintainer anymore. So, there are not
much people that are willing to actively review patches to it.

So, my advice here is to really invert things:

- do the conversion;
- find a good place to put the converted the book;
- add kernel-doc markups to be sure that the symbols will always
  match upstream;
- review the remaining content of the docs.

> I am currently putting together a task list for mentorship program
> with a goal to create tasks that would more meaningful and helpful
> instead of whitespace patches.
> 
> I will gladly add documentation tasks to the list to help improve
> the documentation.

I can surely mentor people interested on doing such tasks.

> 
> thanks,
> -- Shuah
> 
> 
> 
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss



Thanks,
Mauro

Re: [Ksummit-discuss] [TECH TOPIC] Documentation

[Shuah Khan]

On 6/13/19 8:25 AM, Mauro Carvalho Chehab wrote:
> Em Wed, 12 Jun 2019 12:22:55 -0600
> Shuah Khan <skhan@linuxfoundation.org> escreveu:
> 
>> On 6/12/19 8:54 AM, Jonathan Corbet wrote:
>>> What could be more fun than talking about kernel documentation?  Things we
>>> could get into:
>>>
>>>    - The state of the RST transition, what remains to be done, whether it's
>>>      all just useless churn that makes the documentation worse, etc.
>>>
>>>    - Things we'd like to improve in the documentation toolchain.
>>>
>>>    - Overall organization of Documentation/ and moving docs when the need
>>>      arises.  It seems I end up fighting about this more than just about
>>>      anything else, but I think it's important to organize our docs for the
>>>      convenience of the people using them.
>>>
>>>    - The ultimate vision for kernel docs (for now).  RST conversion and
>>>      imposing some organization are important, but they will not,
>>>      themselves, give us a coherent set of documentation.  What can we do to
>>>      have documentation that is useful, current, and maintainable, rather
>>>      than the dusty attic we have now?
>>>    
>>
>> The first step is identifying the ones that are out of date and/or
>> incorrect. I think this is probably the most time consuming part
>> and the second is start updating.
> 
> I would do just the opposite: convert the documentation, then
> update it.
> 

Yes. That is fine. I am referring to the outdated part. Converting
and updating them would work just fine.

> The reason is that, when someone sends a patch to the documentation,
> people tend to look more closely on it, and we start receiving
> updates for it.
> 


> Ok, this doesn't happen every time, but it is still worth.
> 
> There's another thing to consider here: there are lots of docs
> written for "stable" subsystems, in the sense that the documented
> code doesn't change for a lot of time (except for trivial patches
> not directly related to it.
> 
> While doing large chunks of code conversion, I noticed that there
> are lots of still valid documents that fit on this.
> 
> There's a big issue with those "stable" subsystems: they tend
> to not have any active maintainer anymore. So, there are not
> much people that are willing to actively review patches to it.
> 
> So, my advice here is to really invert things:
> 
> - do the conversion;

Even this can be made into tasks. If you would like to experiment
with and see how it works, send me a list of documents that you
would like to see converted first.

> - find a good place to put the converted the book;
> - add kernel-doc markups to be sure that the symbols will always
>    match upstream;
> - review the remaining content of the docs.
> 
>> I am currently putting together a task list for mentorship program
>> with a goal to create tasks that would more meaningful and helpful
>> instead of whitespace patches.
>>
>> I will gladly add documentation tasks to the list to help improve
>> the documentation.
> 
> I can surely mentor people interested on doing such tasks.
> 

That will be great.

thanks,
-- Shuah

Re: [Ksummit-discuss] [TECH TOPIC] Documentation

[Mauro Carvalho Chehab]

Em Fri, 14 Jun 2019 15:40:26 -0600
Shuah Khan <skhan@linuxfoundation.org> escreveu:

> On 6/13/19 8:25 AM, Mauro Carvalho Chehab wrote:
> > Em Wed, 12 Jun 2019 12:22:55 -0600
> > Shuah Khan <skhan@linuxfoundation.org> escreveu:
> >   

> > So, my advice here is to really invert things:
> > 
> > - do the conversion;  
> 
> Even this can be made into tasks. If you would like to experiment
> with and see how it works, send me a list of documents that you
> would like to see converted first.

After the patchsets I'm working it:

	https://git.linuxtv.org/mchehab/experimental.git/log/?h=convert_rst_renames_v5

There will be very few directories that doesn't any any .rst file:

$ for i in $(find Documentation/ -name '*.txt' | sed -E "s,(Documentation/[^\/+]+).*,\1,"|uniq|grep -v output |grep -v binding|grep -v devicetree); do if [ "$(find $i -name '*.rst')" == "" ]; then if [ -d $i ]; then echo $i; fi; fi; done

Documentation/platform
Documentation/scsi
Documentation/RCU
Documentation/cpu-freq
Documentation/ABI
Documentation/sphinx
Documentation/features
Documentation/acpi

Among those:

- There's nothing to be done at the sphinx directory.
- The cpu-freq maintainer won't want changes there, as the stuff inside
  that specific dir is obsolete.
- If I'm not mistaken, there's already a pending patchset for the
  acpi directory too.
- Platform has just a single file, easily convertible to ReST.
  I suspect that it should be moved to the laptops dir too.
  I guess I wrote a patch for it, but it seems it got lost on
  some rebase. Anyway, I can take care of this one.	
- We are handling the ABI directory on a different way. 

So, I guess that what it was left behind is:

	Documentation/scsi
	Documentation/RCU
	Documentation/features

You should notice, however, that there are some other directories
that have a mix of rst and txt files, with also have some files 
that could be needing conversion:

$ for i in $(find Documentation/ -name '*.txt' | sed -E "s,(Documentation/[^\/+]+).*,\1,"|uniq|grep -v output |grep -v binding|grep -v devicetree); do if [ -d $i ]; then echo $i; fi; done
Documentation/device-mapper
Documentation/sh
Documentation/trace
Documentation/misc-devices
Documentation/arm64
Documentation/platform
Documentation/scsi
Documentation/RCU
Documentation/virtual
Documentation/cpu-freq
Documentation/admin-guide
Documentation/block
Documentation/ABI
Documentation/translations
Documentation/PCI
Documentation/filesystems
Documentation/networking
Documentation/sphinx
Documentation/features
Documentation/netlabel
Documentation/acpi
Documentation/crypto
Documentation/sparc

I suspect that those dirs with contain botn .rst and .txt files are
in the process of being updated and converted by the subsystem
maintainers, but double check is needed.

Plus other places where there are files whose filename doesn't have any
extension at all and could be a text file to be converted:

$ for i in $(find Documentation/  -type f ! -name "*.*" | sed -E "s,(Documentation/[^\/+]+).*,\1,"|uniq|grep -v output |grep -v binding|grep -v devicetree); do if [ -d $i ]; then echo $i; fi; done
Documentation/misc-devices
Documentation/target
Documentation/firmware_class
Documentation/isdn
Documentation/media
Documentation/parisc
Documentation/spi
Documentation/scsi
Documentation/hwmon
Documentation/EDID
Documentation/nios2
Documentation/virtual
Documentation/i2c
Documentation/usb
Documentation/ABI
Documentation/translations
Documentation/openrisc
Documentation/w1
Documentation/filesystems
Documentation/networking
Documentation/auxdisplay

There are a few exceptions that should be kept as plain text
files. I remember of two such cases:

- a configuration file whose extension is .txt;
- a dump of chapter extracted from an old arch-specific manual
  with seems to be provided by its manufacturer.

While the last one could eventually be converted, it sounded
too much work for too little gain - and a format change may
eventually require some negotiation with the copyright holder.

Thanks,
Mauro

Re: [Ksummit-discuss] [TECH TOPIC] Documentation

[Mauro Carvalho Chehab]

Em Fri, 14 Jun 2019 21:05:36 -0300
Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:

> Em Fri, 14 Jun 2019 15:40:26 -0600
> Shuah Khan <skhan@linuxfoundation.org> escreveu:
> 
> > On 6/13/19 8:25 AM, Mauro Carvalho Chehab wrote:  
> > > Em Wed, 12 Jun 2019 12:22:55 -0600
> > > Shuah Khan <skhan@linuxfoundation.org> escreveu:
> > >     
> 
> > > So, my advice here is to really invert things:
> > > 
> > > - do the conversion;    
> > 
> > Even this can be made into tasks. If you would like to experiment
> > with and see how it works, send me a list of documents that you
> > would like to see converted first.  
> 
> After the patchsets I'm working it:
> 
> 	https://git.linuxtv.org/mchehab/experimental.git/log/?h=convert_rst_renames_v5
> 
> There will be very few directories that doesn't any any .rst file:
> 
> $ for i in $(find Documentation/ -name '*.txt' | sed -E "s,(Documentation/[^\/+]+).*,\1,"|uniq|grep -v output |grep -v binding|grep -v devicetree); do if [ "$(find $i -name '*.rst')" == "" ]; then if [ -d $i ]; then echo $i; fi; fi; done
> 
> Documentation/platform
> Documentation/scsi
> Documentation/RCU
> Documentation/cpu-freq
> Documentation/ABI
> Documentation/sphinx
> Documentation/features
> Documentation/acpi
> 
> Among those:
> 
> - There's nothing to be done at the sphinx directory.
> - The cpu-freq maintainer won't want changes there, as the stuff inside
>   that specific dir is obsolete.
> - If I'm not mistaken, there's already a pending patchset for the
>   acpi directory too.
> - Platform has just a single file, easily convertible to ReST.
>   I suspect that it should be moved to the laptops dir too.
>   I guess I wrote a patch for it, but it seems it got lost on
>   some rebase. Anyway, I can take care of this one.	
> - We are handling the ABI directory on a different way. 
> 
> So, I guess that what it was left behind is:
> 
> 	Documentation/scsi
> 	Documentation/RCU


> 	Documentation/features

This is actually a special case. I don't think that a plain txt->rst
conversion is the right thing to be done here.

IMO, it requires some further discussions.

The files there have a standard format, with is (poorly) documented
at Documentation/features/arch-support.txt.

Getting a random file there, the format has a description part and a
table (Documentation/features/debug/kprobes/arch-support.txt):

	# Feature name:          kprobes
	#         Kconfig:       HAVE_KPROBES
	#         description:   arch supports live patched kernel probe
	#
	    -----------------------
	    |         arch |status|
	    -----------------------
	    |       alpha: | TODO |
	    |         arc: |  ok  |
	    |         arm: |  ok  |
	...
	    |      xtensa: | TODO |
	    -----------------------

E. g. each file contains the name of a feature, its Kconfig
symbol, a description and a table with shows what architectures
support and what architectures don't.

I actually started converting it, but the results don't look nice.

The thing is: it is a lot easier for a "feature" maintainer to have
a single file with a per-architecture status, as he's free to update
the file without needing to be concerned about merge conflicts.

So, the current way makes a perfect sense from a developer's PoV.

-

However, for a Kernel user's perspective, he may be interested on
looking at this as a completely different way.

1) he may want to buy a hardware whose support a certain feature
   subset;

2) He has already some hardware, any may want to identify if a
   certain feature is there.


For (1), the best would be a table like:

=======  =====  ===== =====   =====       ======
Feature  alpha  arc   arm     arm64 ....  xtensa
=======  =====  ===== =====   =====       ======
KASAN    TODO   TODO  TODO    ok          ok
kdb      TODO   ok    ok      ok          TODO
...
=======  =====  ===== =====   =====       ======

(eventually with Kconfig and description, but we need to double check
if it won't bee to big at the pdf output).


And for (2), one file per architecture, with something like:


Feature status at x86 Architecture
==================================

=======   =================  ======  ==============================================
Feature   Kconfig            Status  Description
=======   =================  ======  ==============================================
KASAN	  HAVE_ARCH_KASAN    ok      arch supports the KASAN runtime memory checker
kgdb      HAVE_ARCH_KGDB     ok      arch supports the kGDB kernel debugger
...
=======   =================  ======  ==============================================


So, IMHO, the best thing to do with the feature files is an approach similar
to the one we're doing with ABI, e. g.:

1) improve the contents of "arch-support.txt" file for it to precisely
describe the format of the file, including the header. While not required,
eventually we could change the headers to something less prune to human
errors, e. g. to something like:

	name:	       kprobes
	Kconfig:       HAVE_KPROBES
	description:   arch supports live patched kernel probe

2) have a script similar to get_abi.pl that would parse, validate
and produce a per-architecture output file;

3) add a per-arch features file that will be automatically
generated by the script.

The script should be able to produce valid ReST output on all the
above formats, e. g.:

1) output per feature;

2) output per architecture;

3) output as a matrix feature x architecture.

This way, we could add the feature lists on multiple books:

	- admin-doc: feature x architecture view
	- arch-specific view: feature status for such specific arch
	- kAPI book: per feature output

Comments?

Thanks,
Mauro

Re: [Ksummit-discuss] [TECH TOPIC] Documentation

[Mauro Carvalho Chehab]

Em Mon, 17 Jun 2019 07:12:06 -0300
Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:

Answering myself here...

> > 	Documentation/features  
> 
> This is actually a special case. I don't think that a plain txt->rst
> conversion is the right thing to be done here.
> 
> IMO, it requires some further discussions.

> The script should be able to produce valid ReST output on all the
> above formats, e. g.:
> 
> 1) output per feature;
> 2) output per architecture;
> 3) output as a matrix feature x architecture.

I'm actually thinking on something like:

1) A page that could be part of a feature documentation
   (On this example, it documents scheduler ARCH_HAS_MEMBARRIER_SYNC_CORE)

	https://www.infradead.org/~mchehab/rst_features/feature_membarrier-sync-core.html

2) A page that could be part of the per-archtecture books
   (here, arm64 one):

	https://www.infradead.org/~mchehab/rst_features/feature_arm64.html

3) A page with a matrix with all architectures and all features:

	https://www.infradead.org/~mchehab/rst_features/feature_matrix.html

I just wrote a patch with such script.

I should be submitting it soon to the ML for review, after some final
cleanups.

Thanks,
Mauro

[Ksummit-discuss] [PATCH RFC] scripts: add a script to handle Documentation/features

[Mauro Carvalho Chehab]

The Documentation/features contains a set of parseable files.
It is not worth converting them to ReST format, as they're
useful the way it is. It is, however, interesting to parse
them and produce output on different formats:

1) Output the contents of a feature in ReST format;

2) Output what features a given architecture supports;

3) Output a matrix with features x architectures.

Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---

As commented at KS mailing list, converting the Documentation/features
file to ReST may not be the best way to handle it. 

This script allows validating the features files and to  generate ReST files 
on three different formats.

The goal is to support it via a sphinx extension, in order to be able to add
the features inside the Kernel documentation.

 scripts/get_feat.pl | 470 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 470 insertions(+)
 create mode 100755 scripts/get_feat.pl

diff --git a/scripts/get_feat.pl b/scripts/get_feat.pl
new file mode 100755
index 000000000000..c5a267b12f49
--- /dev/null
+++ b/scripts/get_feat.pl
@@ -0,0 +1,470 @@
+#!/usr/bin/perl
+
+use strict;
+use Pod::Usage;
+use Getopt::Long;
+use File::Find;
+use Fcntl ':mode';
+
+my $help;
+my $man;
+my $debug;
+my $arch;
+my $feat;
+my $prefix="Documentation/features";
+
+GetOptions(
+	"debug|d+" => \$debug,
+	'help|?' => \$help,
+	'arch=s' => \$arch,
+	'feat=s' => \$feat,
+	man => \$man
+) or pod2usage(2);
+
+pod2usage(1) if $help;
+pod2usage(-exitstatus => 0, -verbose => 2) if $man;
+
+pod2usage(2) if (scalar @ARGV < 1 || @ARGV > 2);
+
+my ($cmd, $arg) = @ARGV;
+
+pod2usage(2) if ($cmd ne "current" && $cmd ne "rest" && $cmd ne "validate");
+
+require Data::Dumper if ($debug);
+
+my %data;
+my %archs;
+
+#
+# Displays an error message, printing file name and line
+#
+sub parse_error($$$$) {
+	my ($file, $ln, $msg, $data) = @_;
+
+	$data =~ s/\s+$/\n/;
+
+	print STDERR "Warning: file $file#$ln:\n\t$msg";
+
+	if ($data ne "") {
+		print STDERR ". Line\n\t\t$data";
+	} else {
+	    print STDERR "\n";
+	}
+}
+
+#
+# Parse a features file, storing its contents at %data
+#
+
+my $h_name = "Feature";
+my $h_kconfig = "Kconfig";
+my $h_description = "Description";
+my $h_subsys = "Subsystem";
+my $h_status = "Status";
+my $h_arch = "Architecture";
+
+my $max_size_name = length($h_name);
+my $max_size_kconfig = length($h_kconfig);
+my $max_size_description = length($h_description);
+my $max_size_subsys = length($h_subsys);
+my $max_size_status = length($h_status);
+my $max_size_arch = length($h_arch);
+
+sub parse_feat {
+	my $file = $File::Find::name;
+
+	my $mode = (stat($file))[2];
+	return if ($mode & S_IFDIR);
+	return if ($file =~ m,($prefix)/arch-support.txt,);
+	return if (!($file =~ m,arch-support.txt$,));
+
+	my $subsys = "";
+	$subsys = $2 if ( m,.*($prefix)/([^/]+).*,);
+
+	if (length($subsys) > $max_size_subsys) {
+		$max_size_subsys = length($subsys);
+	}
+
+	my $name;
+	my $kconfig;
+	my $description;
+	my $comments = "";
+	my $last_status;
+	my $ln;
+	my %arch_table;
+
+	print STDERR "Opening $file\n" if ($debug > 1);
+	open IN, $file;
+
+	while(<IN>) {
+		$ln++;
+
+		if (m/^\#\s+Feature\s+name:\s*(.*\S)/) {
+			$name = $1;
+			if (length($name) > $max_size_name) {
+				$max_size_name = length($name);
+			}
+			next;
+		}
+		if (m/^\#\s+Kconfig:\s*(.*\S)/) {
+			$kconfig = $1;
+			if (length($kconfig) > $max_size_kconfig) {
+				$max_size_kconfig = length($kconfig);
+			}
+			next;
+		}
+		if (m/^\#\s+description:\s*(.*\S)/) {
+			$description = $1;
+			if (length($description) > $max_size_description) {
+				$max_size_description = length($description);
+			}
+			next;
+		}
+		next if (m/^\\s*$/);
+		next if (m/^\s*\-+\s*$/);
+		next if (m/^\s*\|\s*arch\s*\|\s*status\s*\|\s*$/);
+
+		if (m/^\#\s*(.*)/) {
+			$comments .= "$1\n";
+			next;
+		}
+		if (m/^\s*\|\s*(\S+):\s*\|\s*(\S+)\s*\|\s*$/) {
+			my $a = $1;
+			my $status = $2;
+
+			if (length($status) > $max_size_status) {
+				$max_size_status = length($status);
+			}
+			if (length($a) > $max_size_arch) {
+				$max_size_arch = length($a);
+			}
+
+			$archs{$a} = 1;
+			$arch_table{$a} = $status;
+			next;
+		}
+
+		#Everything else is an error
+		parse_error($file, $ln, "line is invalid", $_);
+	}
+	close IN;
+
+	if (!$name) {
+		parse_error($file, $ln, "Feature name not found", "");
+		return;
+	}
+
+	parse_error($file, $ln, "Subsystem not found", "") if (!$subsys);
+	parse_error($file, $ln, "Kconfig not found", "") if (!$kconfig);
+	parse_error($file, $ln, "Description not found", "") if (!$description);
+
+	if (!%arch_table) {
+		parse_error($file, $ln, "Architecture table not found", "");
+		return;
+	}
+
+	$data{$name}->{where} = $file;
+	$data{$name}->{subsys} = $subsys;
+	$data{$name}->{kconfig} = $kconfig;
+	$data{$name}->{description} = $description;
+	$data{$name}->{comments} = $comments;
+	$data{$name}->{table} = \%arch_table;
+}
+
+#
+# Output feature(s) for a given architecture
+#
+sub output_arch_table {
+	my $title = "Feature status on $arch architecture";
+
+	print "=" x length($title) . "\n";
+	print "$title\n";
+	print "=" x length($title) . "\n\n";
+
+	print "=" x $max_size_subsys;
+	print "  ";
+	print "=" x $max_size_name;
+	print "  ";
+	print "=" x $max_size_kconfig;
+	print "  ";
+	print "=" x $max_size_status;
+	print "  ";
+	print "=" x $max_size_description;
+	print "\n";
+	printf "%-${max_size_subsys}s  ", $h_subsys;
+	printf "%-${max_size_name}s  ", $h_name;
+	printf "%-${max_size_kconfig}s  ", $h_kconfig;
+	printf "%-${max_size_status}s  ", $h_status;
+	printf "%-${max_size_description}s\n", $h_description;
+	print "=" x $max_size_subsys;
+	print "  ";
+	print "=" x $max_size_name;
+	print "  ";
+	print "=" x $max_size_kconfig;
+	print "  ";
+	print "=" x $max_size_status;
+	print "  ";
+	print "=" x $max_size_description;
+	print "\n";
+
+	foreach my $name (sort {
+				($data{$a}->{subsys} cmp $data{$b}->{subsys}) ||
+				($data{$a}->{name} cmp $data{$b}->{name})
+			       } keys %data) {
+		next if ($feat && $name ne $feat);
+
+		my %arch_table = %{$data{$name}->{table}};
+		printf "%-${max_size_subsys}s  ", $data{$name}->{subsys};
+		printf "%-${max_size_name}s  ", $name;
+		printf "%-${max_size_kconfig}s  ", $data{$name}->{kconfig};
+		printf "%-${max_size_status}s  ", $arch_table{$arch};
+		printf "%-${max_size_description}s\n", $data{$name}->{description};
+	}
+
+	print "=" x $max_size_subsys;
+	print "  ";
+	print "=" x $max_size_name;
+	print "  ";
+	print "=" x $max_size_kconfig;
+	print "  ";
+	print "=" x $max_size_status;
+	print "  ";
+	print "=" x $max_size_description;
+	print "\n";
+}
+
+#
+# Output a feature on all architectures
+#
+sub output_feature {
+	my $title = "Feature $feat";
+
+	print "=" x length($title) . "\n";
+	print "$title\n";
+	print "=" x length($title) . "\n\n";
+
+	print ":Subsystem: $data{$feat}->{subsys} \n" if ($data{$feat}->{subsys});
+	print ":Kconfig: $data{$feat}->{kconfig} \n" if ($data{$feat}->{kconfig});
+
+	my $desc = $data{$feat}->{description};
+	$desc =~ s/^([a-z])/\U$1/;
+	$desc =~ s/\.?\s*//;
+	print "\n$desc.\n\n";
+
+	my $com = $data{$feat}->{comments};
+	$com =~ s/^\s+//;
+	$com =~ s/\s+$//;
+	if ($com) {
+		print "Comments\n";
+		print "--------\n\n";
+		print "$com\n\n";
+	}
+
+	print "=" x $max_size_arch;
+	print "  ";
+	print "=" x $max_size_status;
+	print "\n";
+
+	printf "%-${max_size_arch}s  ", $h_arch;
+	printf "%-${max_size_status}s", $h_status . "\n";
+
+	print "=" x $max_size_arch;
+	print "  ";
+	print "=" x $max_size_status;
+	print "\n";
+
+	my %arch_table = %{$data{$feat}->{table}};
+	foreach my $arch (sort keys %arch_table) {
+		printf "%-${max_size_arch}s  ", $arch;
+		printf "%-${max_size_status}s\n", $arch_table{$arch};
+	}
+
+	print "=" x $max_size_arch;
+	print "  ";
+	print "=" x $max_size_status;
+	print "\n";
+}
+
+#
+# Output all features for all architectures
+#
+
+sub matrix_lines {
+	print "=" x $max_size_subsys;
+	print "  ";
+	print "=" x $max_size_name;
+	print "  ";
+
+	foreach my $arch (sort keys %archs) {
+		my $len = $max_size_status;
+
+		$len = length($arch) if ($len < length($arch));
+
+		print "=" x $len;
+		print "  ";
+	}
+	print "=" x $max_size_kconfig;
+	print "  ";
+	print "=" x $max_size_description;
+	print "\n";
+}
+
+sub output_matrix {
+
+	my $title = "Feature List (feature x architecture)";
+
+	print "=" x length($title) . "\n";
+	print "$title\n";
+	print "=" x length($title) . "\n\n";
+
+	matrix_lines;
+
+	printf "%-${max_size_subsys}s  ", $h_subsys;
+	printf "%-${max_size_name}s  ", $h_name;
+
+	foreach my $arch (sort keys %archs) {
+		printf "%-${max_size_status}s  ", $arch;
+	}
+	printf "%-${max_size_kconfig}s  ", $h_kconfig;
+	printf "%-${max_size_description}s\n", $h_description;
+
+	matrix_lines;
+
+	foreach my $name (sort {
+				($data{$a}->{subsys} cmp $data{$b}->{subsys}) ||
+				($data{$a}->{name} cmp $data{$b}->{name})
+			       } keys %data) {
+		printf "%-${max_size_subsys}s  ", $data{$name}->{subsys};
+		printf "%-${max_size_name}s  ", $name;
+
+		my %arch_table = %{$data{$name}->{table}};
+
+		foreach my $arch (sort keys %arch_table) {
+			my $len = $max_size_status;
+
+			$len = length($arch) if ($len < length($arch));
+
+			printf "%-${len}s  ", $arch_table{$arch};
+		}
+		printf "%-${max_size_kconfig}s  ", $data{$name}->{kconfig};
+		printf "%-${max_size_description}s\n", $data{$name}->{description};
+	}
+
+	matrix_lines;
+}
+
+
+#
+# Parses all feature files located at $prefix dir
+#
+find({wanted =>\&parse_feat, no_chdir => 1}, $prefix);
+
+print STDERR Data::Dumper->Dump([\%data], [qw(*data)]) if ($debug);
+
+#
+# Handles the command
+#
+if ($cmd eq "current") {
+	$arch = qx(uname -m | sed 's/x86_64/x86/' | sed 's/i386/x86/');
+	$arch =~s/\s+$//;
+}
+
+if ($cmd ne "validate") {
+	if ($arch) {
+		output_arch_table;
+	} elsif ($feat) {
+		output_feature;
+	} else {
+		output_matrix;
+	}
+}
+
+__END__
+
+=head1 NAME
+
+get_feat.pl - parse the Linux Feature files and produce a ReST book.
+
+=head1 SYNOPSIS
+
+B<get_feat.pl> [--debug] [--man] [--help] [--dir=<dir>]
+	       [--arch=<arch>] [--feat=<feature>] <COMAND> [<ARGUMENT>]
+
+Where <COMMAND> can be:
+
+=over 8
+
+B<current>               - output features for this machine's architecture
+
+B<rest>                  - output features in ReST markup language
+
+B<validate>              - validate the feature contents
+
+=back
+
+=head1 OPTIONS
+
+=over 8
+
+=item B<--arch>
+
+Output features for an specific architecture, optionally filtering for
+a single specific feature.
+
+=item B<--feat>
+
+Output features for a single specific architecture.
+
+=item B<--dir>
+
+Changes the location of the Feature files. By default, it uses
+the Documentation/features directory.
+
+=item B<--debug>
+
+Put the script in verbose mode, useful for debugging. Can be called multiple
+times, to increase verbosity.
+
+=item B<--help>
+
+Prints a brief help message and exits.
+
+=item B<--man>
+
+Prints the manual page and exits.
+
+=back
+
+=head1 DESCRIPTION
+
+Parse the Linux feature files from Documentation/features (by default),
+optionally producing results at ReST format.
+
+It supports output data per architecture, per feature or a
+feature x arch matrix.
+
+When used with B<rest> command, it will use either one of the tree formats:
+
+If neither B<--arch> or B<--feature> arguments are used, it will output a
+matrix with features per architecture.
+
+If B<--arch> argument is used, it will output the features availability for
+a given architecture.
+
+If B<--feat> argument is used, it will output the content of the feature
+file using ReStructured Text markup.
+
+=head1 BUGS
+
+Report bugs to Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
+
+=head1 COPYRIGHT
+
+Copyright (c) 2019 by Mauro Carvalho Chehab <mchehab+samsung@kernel.org>.
+
+License GPLv2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>.
+
+This is free software: you are free to change and redistribute it.
+There is NO WARRANTY, to the extent permitted by law.
+
+=cut
-- 
2.21.0

[PATCH RFC] scripts: add a script to handle Documentation/features

[Mauro Carvalho Chehab]

The Documentation/features contains a set of parseable files.
It is not worth converting them to ReST format, as they're
useful the way it is. It is, however, interesting to parse
them and produce output on different formats:

1) Output the contents of a feature in ReST format;

2) Output what features a given architecture supports;

3) Output a matrix with features x architectures.

Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
---

As commented at KS mailing list, converting the Documentation/features
file to ReST may not be the best way to handle it. 

This script allows validating the features files and to  generate ReST files 
on three different formats.

The goal is to support it via a sphinx extension, in order to be able to add
the features inside the Kernel documentation.

 scripts/get_feat.pl | 470 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 470 insertions(+)
 create mode 100755 scripts/get_feat.pl

diff --git a/scripts/get_feat.pl b/scripts/get_feat.pl
new file mode 100755
index 000000000000..c5a267b12f49
--- /dev/null
+++ b/scripts/get_feat.pl
@@ -0,0 +1,470 @@
+#!/usr/bin/perl
+
+use strict;
+use Pod::Usage;
+use Getopt::Long;
+use File::Find;
+use Fcntl ':mode';
+
+my $help;
+my $man;
+my $debug;
+my $arch;
+my $feat;
+my $prefix="Documentation/features";
+
+GetOptions(
+	"debug|d+" => \$debug,
+	'help|?' => \$help,
+	'arch=s' => \$arch,
+	'feat=s' => \$feat,
+	man => \$man
+) or pod2usage(2);
+
+pod2usage(1) if $help;
+pod2usage(-exitstatus => 0, -verbose => 2) if $man;
+
+pod2usage(2) if (scalar @ARGV < 1 || @ARGV > 2);
+
+my ($cmd, $arg) = @ARGV;
+
+pod2usage(2) if ($cmd ne "current" && $cmd ne "rest" && $cmd ne "validate");
+
+require Data::Dumper if ($debug);
+
+my %data;
+my %archs;
+
+#
+# Displays an error message, printing file name and line
+#
+sub parse_error($$$$) {
+	my ($file, $ln, $msg, $data) = @_;
+
+	$data =~ s/\s+$/\n/;
+
+	print STDERR "Warning: file $file#$ln:\n\t$msg";
+
+	if ($data ne "") {
+		print STDERR ". Line\n\t\t$data";
+	} else {
+	    print STDERR "\n";
+	}
+}
+
+#
+# Parse a features file, storing its contents at %data
+#
+
+my $h_name = "Feature";
+my $h_kconfig = "Kconfig";
+my $h_description = "Description";
+my $h_subsys = "Subsystem";
+my $h_status = "Status";
+my $h_arch = "Architecture";
+
+my $max_size_name = length($h_name);
+my $max_size_kconfig = length($h_kconfig);
+my $max_size_description = length($h_description);
+my $max_size_subsys = length($h_subsys);
+my $max_size_status = length($h_status);
+my $max_size_arch = length($h_arch);
+
+sub parse_feat {
+	my $file = $File::Find::name;
+
+	my $mode = (stat($file))[2];
+	return if ($mode & S_IFDIR);
+	return if ($file =~ m,($prefix)/arch-support.txt,);
+	return if (!($file =~ m,arch-support.txt$,));
+
+	my $subsys = "";
+	$subsys = $2 if ( m,.*($prefix)/([^/]+).*,);
+
+	if (length($subsys) > $max_size_subsys) {
+		$max_size_subsys = length($subsys);
+	}
+
+	my $name;
+	my $kconfig;
+	my $description;
+	my $comments = "";
+	my $last_status;
+	my $ln;
+	my %arch_table;
+
+	print STDERR "Opening $file\n" if ($debug > 1);
+	open IN, $file;
+
+	while(<IN>) {
+		$ln++;
+
+		if (m/^\#\s+Feature\s+name:\s*(.*\S)/) {
+			$name = $1;
+			if (length($name) > $max_size_name) {
+				$max_size_name = length($name);
+			}
+			next;
+		}
+		if (m/^\#\s+Kconfig:\s*(.*\S)/) {
+			$kconfig = $1;
+			if (length($kconfig) > $max_size_kconfig) {
+				$max_size_kconfig = length($kconfig);
+			}
+			next;
+		}
+		if (m/^\#\s+description:\s*(.*\S)/) {
+			$description = $1;
+			if (length($description) > $max_size_description) {
+				$max_size_description = length($description);
+			}
+			next;
+		}
+		next if (m/^\\s*$/);
+		next if (m/^\s*\-+\s*$/);
+		next if (m/^\s*\|\s*arch\s*\|\s*status\s*\|\s*$/);
+
+		if (m/^\#\s*(.*)/) {
+			$comments .= "$1\n";
+			next;
+		}
+		if (m/^\s*\|\s*(\S+):\s*\|\s*(\S+)\s*\|\s*$/) {
+			my $a = $1;
+			my $status = $2;
+
+			if (length($status) > $max_size_status) {
+				$max_size_status = length($status);
+			}
+			if (length($a) > $max_size_arch) {
+				$max_size_arch = length($a);
+			}
+
+			$archs{$a} = 1;
+			$arch_table{$a} = $status;
+			next;
+		}
+
+		#Everything else is an error
+		parse_error($file, $ln, "line is invalid", $_);
+	}
+	close IN;
+
+	if (!$name) {
+		parse_error($file, $ln, "Feature name not found", "");
+		return;
+	}
+
+	parse_error($file, $ln, "Subsystem not found", "") if (!$subsys);
+	parse_error($file, $ln, "Kconfig not found", "") if (!$kconfig);
+	parse_error($file, $ln, "Description not found", "") if (!$description);
+
+	if (!%arch_table) {
+		parse_error($file, $ln, "Architecture table not found", "");
+		return;
+	}
+
+	$data{$name}->{where} = $file;
+	$data{$name}->{subsys} = $subsys;
+	$data{$name}->{kconfig} = $kconfig;
+	$data{$name}->{description} = $description;
+	$data{$name}->{comments} = $comments;
+	$data{$name}->{table} = \%arch_table;
+}
+
+#
+# Output feature(s) for a given architecture
+#
+sub output_arch_table {
+	my $title = "Feature status on $arch architecture";
+
+	print "=" x length($title) . "\n";
+	print "$title\n";
+	print "=" x length($title) . "\n\n";
+
+	print "=" x $max_size_subsys;
+	print "  ";
+	print "=" x $max_size_name;
+	print "  ";
+	print "=" x $max_size_kconfig;
+	print "  ";
+	print "=" x $max_size_status;
+	print "  ";
+	print "=" x $max_size_description;
+	print "\n";
+	printf "%-${max_size_subsys}s  ", $h_subsys;
+	printf "%-${max_size_name}s  ", $h_name;
+	printf "%-${max_size_kconfig}s  ", $h_kconfig;
+	printf "%-${max_size_status}s  ", $h_status;
+	printf "%-${max_size_description}s\n", $h_description;
+	print "=" x $max_size_subsys;
+	print "  ";
+	print "=" x $max_size_name;
+	print "  ";
+	print "=" x $max_size_kconfig;
+	print "  ";
+	print "=" x $max_size_status;
+	print "  ";
+	print "=" x $max_size_description;
+	print "\n";
+
+	foreach my $name (sort {
+				($data{$a}->{subsys} cmp $data{$b}->{subsys}) ||
+				($data{$a}->{name} cmp $data{$b}->{name})
+			       } keys %data) {
+		next if ($feat && $name ne $feat);
+
+		my %arch_table = %{$data{$name}->{table}};
+		printf "%-${max_size_subsys}s  ", $data{$name}->{subsys};
+		printf "%-${max_size_name}s  ", $name;
+		printf "%-${max_size_kconfig}s  ", $data{$name}->{kconfig};
+		printf "%-${max_size_status}s  ", $arch_table{$arch};
+		printf "%-${max_size_description}s\n", $data{$name}->{description};
+	}
+
+	print "=" x $max_size_subsys;
+	print "  ";
+	print "=" x $max_size_name;
+	print "  ";
+	print "=" x $max_size_kconfig;
+	print "  ";
+	print "=" x $max_size_status;
+	print "  ";
+	print "=" x $max_size_description;
+	print "\n";
+}
+
+#
+# Output a feature on all architectures
+#
+sub output_feature {
+	my $title = "Feature $feat";
+
+	print "=" x length($title) . "\n";
+	print "$title\n";
+	print "=" x length($title) . "\n\n";
+
+	print ":Subsystem: $data{$feat}->{subsys} \n" if ($data{$feat}->{subsys});
+	print ":Kconfig: $data{$feat}->{kconfig} \n" if ($data{$feat}->{kconfig});
+
+	my $desc = $data{$feat}->{description};
+	$desc =~ s/^([a-z])/\U$1/;
+	$desc =~ s/\.?\s*//;
+	print "\n$desc.\n\n";
+
+	my $com = $data{$feat}->{comments};
+	$com =~ s/^\s+//;
+	$com =~ s/\s+$//;
+	if ($com) {
+		print "Comments\n";
+		print "--------\n\n";
+		print "$com\n\n";
+	}
+
+	print "=" x $max_size_arch;
+	print "  ";
+	print "=" x $max_size_status;
+	print "\n";
+
+	printf "%-${max_size_arch}s  ", $h_arch;
+	printf "%-${max_size_status}s", $h_status . "\n";
+
+	print "=" x $max_size_arch;
+	print "  ";
+	print "=" x $max_size_status;
+	print "\n";
+
+	my %arch_table = %{$data{$feat}->{table}};
+	foreach my $arch (sort keys %arch_table) {
+		printf "%-${max_size_arch}s  ", $arch;
+		printf "%-${max_size_status}s\n", $arch_table{$arch};
+	}
+
+	print "=" x $max_size_arch;
+	print "  ";
+	print "=" x $max_size_status;
+	print "\n";
+}
+
+#
+# Output all features for all architectures
+#
+
+sub matrix_lines {
+	print "=" x $max_size_subsys;
+	print "  ";
+	print "=" x $max_size_name;
+	print "  ";
+
+	foreach my $arch (sort keys %archs) {
+		my $len = $max_size_status;
+
+		$len = length($arch) if ($len < length($arch));
+
+		print "=" x $len;
+		print "  ";
+	}
+	print "=" x $max_size_kconfig;
+	print "  ";
+	print "=" x $max_size_description;
+	print "\n";
+}
+
+sub output_matrix {
+
+	my $title = "Feature List (feature x architecture)";
+
+	print "=" x length($title) . "\n";
+	print "$title\n";
+	print "=" x length($title) . "\n\n";
+
+	matrix_lines;
+
+	printf "%-${max_size_subsys}s  ", $h_subsys;
+	printf "%-${max_size_name}s  ", $h_name;
+
+	foreach my $arch (sort keys %archs) {
+		printf "%-${max_size_status}s  ", $arch;
+	}
+	printf "%-${max_size_kconfig}s  ", $h_kconfig;
+	printf "%-${max_size_description}s\n", $h_description;
+
+	matrix_lines;
+
+	foreach my $name (sort {
+				($data{$a}->{subsys} cmp $data{$b}->{subsys}) ||
+				($data{$a}->{name} cmp $data{$b}->{name})
+			       } keys %data) {
+		printf "%-${max_size_subsys}s  ", $data{$name}->{subsys};
+		printf "%-${max_size_name}s  ", $name;
+
+		my %arch_table = %{$data{$name}->{table}};
+
+		foreach my $arch (sort keys %arch_table) {
+			my $len = $max_size_status;
+
+			$len = length($arch) if ($len < length($arch));
+
+			printf "%-${len}s  ", $arch_table{$arch};
+		}
+		printf "%-${max_size_kconfig}s  ", $data{$name}->{kconfig};
+		printf "%-${max_size_description}s\n", $data{$name}->{description};
+	}
+
+	matrix_lines;
+}
+
+
+#
+# Parses all feature files located at $prefix dir
+#
+find({wanted =>\&parse_feat, no_chdir => 1}, $prefix);
+
+print STDERR Data::Dumper->Dump([\%data], [qw(*data)]) if ($debug);
+
+#
+# Handles the command
+#
+if ($cmd eq "current") {
+	$arch = qx(uname -m | sed 's/x86_64/x86/' | sed 's/i386/x86/');
+	$arch =~s/\s+$//;
+}
+
+if ($cmd ne "validate") {
+	if ($arch) {
+		output_arch_table;
+	} elsif ($feat) {
+		output_feature;
+	} else {
+		output_matrix;
+	}
+}
+
+__END__
+
+=head1 NAME
+
+get_feat.pl - parse the Linux Feature files and produce a ReST book.
+
+=head1 SYNOPSIS
+
+B<get_feat.pl> [--debug] [--man] [--help] [--dir=<dir>]
+	       [--arch=<arch>] [--feat=<feature>] <COMAND> [<ARGUMENT>]
+
+Where <COMMAND> can be:
+
+=over 8
+
+B<current>               - output features for this machine's architecture
+
+B<rest>                  - output features in ReST markup language
+
+B<validate>              - validate the feature contents
+
+=back
+
+=head1 OPTIONS
+
+=over 8
+
+=item B<--arch>
+
+Output features for an specific architecture, optionally filtering for
+a single specific feature.
+
+=item B<--feat>
+
+Output features for a single specific architecture.
+
+=item B<--dir>
+
+Changes the location of the Feature files. By default, it uses
+the Documentation/features directory.
+
+=item B<--debug>
+
+Put the script in verbose mode, useful for debugging. Can be called multiple
+times, to increase verbosity.
+
+=item B<--help>
+
+Prints a brief help message and exits.
+
+=item B<--man>
+
+Prints the manual page and exits.
+
+=back
+
+=head1 DESCRIPTION
+
+Parse the Linux feature files from Documentation/features (by default),
+optionally producing results at ReST format.
+
+It supports output data per architecture, per feature or a
+feature x arch matrix.
+
+When used with B<rest> command, it will use either one of the tree formats:
+
+If neither B<--arch> or B<--feature> arguments are used, it will output a
+matrix with features per architecture.
+
+If B<--arch> argument is used, it will output the features availability for
+a given architecture.
+
+If B<--feat> argument is used, it will output the content of the feature
+file using ReStructured Text markup.
+
+=head1 BUGS
+
+Report bugs to Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
+
+=head1 COPYRIGHT
+
+Copyright (c) 2019 by Mauro Carvalho Chehab <mchehab+samsung@kernel.org>.
+
+License GPLv2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>.
+
+This is free software: you are free to change and redistribute it.
+There is NO WARRANTY, to the extent permitted by law.
+
+=cut
-- 
2.21.0

Re: [Ksummit-discuss] [PATCH RFC] scripts: add a script to handle Documentation/features

[Greg Kroah-Hartman]

On Mon, Jun 17, 2019 at 03:05:07PM -0300, Mauro Carvalho Chehab wrote:
> The Documentation/features contains a set of parseable files.
> It is not worth converting them to ReST format, as they're
> useful the way it is. It is, however, interesting to parse
> them and produce output on different formats:
> 
> 1) Output the contents of a feature in ReST format;
> 
> 2) Output what features a given architecture supports;
> 
> 3) Output a matrix with features x architectures.
> 
> Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> ---
> 
> As commented at KS mailing list, converting the Documentation/features
> file to ReST may not be the best way to handle it. 
> 
> This script allows validating the features files and to  generate ReST files 
> on three different formats.
> 
> The goal is to support it via a sphinx extension, in order to be able to add
> the features inside the Kernel documentation.
> 
>  scripts/get_feat.pl | 470 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 470 insertions(+)
>  create mode 100755 scripts/get_feat.pl
> 
> diff --git a/scripts/get_feat.pl b/scripts/get_feat.pl
> new file mode 100755
> index 000000000000..c5a267b12f49
> --- /dev/null
> +++ b/scripts/get_feat.pl
> @@ -0,0 +1,470 @@
> +#!/usr/bin/perl
> +

No SPDX line :(

Re: [PATCH RFC] scripts: add a script to handle Documentation/features

[Greg Kroah-Hartman]

On Mon, Jun 17, 2019 at 03:05:07PM -0300, Mauro Carvalho Chehab wrote:
> The Documentation/features contains a set of parseable files.
> It is not worth converting them to ReST format, as they're
> useful the way it is. It is, however, interesting to parse
> them and produce output on different formats:
> 
> 1) Output the contents of a feature in ReST format;
> 
> 2) Output what features a given architecture supports;
> 
> 3) Output a matrix with features x architectures.
> 
> Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> ---
> 
> As commented at KS mailing list, converting the Documentation/features
> file to ReST may not be the best way to handle it. 
> 
> This script allows validating the features files and to  generate ReST files 
> on three different formats.
> 
> The goal is to support it via a sphinx extension, in order to be able to add
> the features inside the Kernel documentation.
> 
>  scripts/get_feat.pl | 470 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 470 insertions(+)
>  create mode 100755 scripts/get_feat.pl
> 
> diff --git a/scripts/get_feat.pl b/scripts/get_feat.pl
> new file mode 100755
> index 000000000000..c5a267b12f49
> --- /dev/null
> +++ b/scripts/get_feat.pl
> @@ -0,0 +1,470 @@
> +#!/usr/bin/perl
> +

No SPDX line :(

Re: [Ksummit-discuss] [PATCH RFC] scripts: add a script to handle Documentation/features

[Mauro Carvalho Chehab]

Em Mon, 17 Jun 2019 20:11:16 +0200
Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:

> On Mon, Jun 17, 2019 at 03:05:07PM -0300, Mauro Carvalho Chehab wrote:
> > The Documentation/features contains a set of parseable files.
> > It is not worth converting them to ReST format, as they're
> > useful the way it is. It is, however, interesting to parse
> > them and produce output on different formats:
> > 
> > 1) Output the contents of a feature in ReST format;
> > 
> > 2) Output what features a given architecture supports;
> > 
> > 3) Output a matrix with features x architectures.
> > 
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> > ---
> > 
> > As commented at KS mailing list, converting the Documentation/features
> > file to ReST may not be the best way to handle it. 
> > 
> > This script allows validating the features files and to  generate ReST files 
> > on three different formats.
> > 
> > The goal is to support it via a sphinx extension, in order to be able to add
> > the features inside the Kernel documentation.
> > 
> >  scripts/get_feat.pl | 470 ++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 470 insertions(+)
> >  create mode 100755 scripts/get_feat.pl
> > 
> > diff --git a/scripts/get_feat.pl b/scripts/get_feat.pl
> > new file mode 100755
> > index 000000000000..c5a267b12f49
> > --- /dev/null
> > +++ b/scripts/get_feat.pl
> > @@ -0,0 +1,470 @@
> > +#!/usr/bin/perl
> > +  
> 
> No SPDX line :(

Added.

I also added a Sphinx extension to handle it as well. You'll notice that it
is almost a copy of kernel_abi.py. 

With regards to patch 2/2, it will generate both a feature x arch matrix 
at the admin-guide and a x86-specific features list.

IMO, it makes sense to have a per-arch feature file just like the one
I added to x86. As the patches converting documentation for other archs
are still being merged via docs tree, before adding the features list to
the other arch documents, it seems better to wait to do it after the
next merge window.

Those patches are applied after the ABI patches on this dir:

	https://git.linuxtv.org/mchehab/experimental.git/log/?h=abi_patches_v4.1

The output with both scripts are at:

	https://www.infradead.org/~mchehab/rst_features/index.html

The relevant parts are: ABI:

	https://www.infradead.org/~mchehab/rst_features/admin-guide/abi.html

Feature list x architecture (at admin-guide:

	https://www.infradead.org/~mchehab/rst_features/admin-guide/features.html

X86 features:

	https://www.infradead.org/~mchehab/rst_features/x86/features.html

While I didn't write a patch, with the new get_feat.pl script, we can probably
get rid of the previous shell script at:

	Documentation/features/list-arch.sh

As calling:

	./scripts/get_feat.pl current

Will output the same content (with a different format, though).

Thanks,
Mauro

Re: [PATCH RFC] scripts: add a script to handle Documentation/features

[Mauro Carvalho Chehab]

Em Mon, 17 Jun 2019 20:11:16 +0200
Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu:

> On Mon, Jun 17, 2019 at 03:05:07PM -0300, Mauro Carvalho Chehab wrote:
> > The Documentation/features contains a set of parseable files.
> > It is not worth converting them to ReST format, as they're
> > useful the way it is. It is, however, interesting to parse
> > them and produce output on different formats:
> > 
> > 1) Output the contents of a feature in ReST format;
> > 
> > 2) Output what features a given architecture supports;
> > 
> > 3) Output a matrix with features x architectures.
> > 
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
> > ---
> > 
> > As commented at KS mailing list, converting the Documentation/features
> > file to ReST may not be the best way to handle it. 
> > 
> > This script allows validating the features files and to  generate ReST files 
> > on three different formats.
> > 
> > The goal is to support it via a sphinx extension, in order to be able to add
> > the features inside the Kernel documentation.
> > 
> >  scripts/get_feat.pl | 470 ++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 470 insertions(+)
> >  create mode 100755 scripts/get_feat.pl
> > 
> > diff --git a/scripts/get_feat.pl b/scripts/get_feat.pl
> > new file mode 100755
> > index 000000000000..c5a267b12f49
> > --- /dev/null
> > +++ b/scripts/get_feat.pl
> > @@ -0,0 +1,470 @@
> > +#!/usr/bin/perl
> > +  
> 
> No SPDX line :(

Added.

I also added a Sphinx extension to handle it as well. You'll notice that it
is almost a copy of kernel_abi.py. 

With regards to patch 2/2, it will generate both a feature x arch matrix 
at the admin-guide and a x86-specific features list.

IMO, it makes sense to have a per-arch feature file just like the one
I added to x86. As the patches converting documentation for other archs
are still being merged via docs tree, before adding the features list to
the other arch documents, it seems better to wait to do it after the
next merge window.

Those patches are applied after the ABI patches on this dir:

	https://git.linuxtv.org/mchehab/experimental.git/log/?h=abi_patches_v4.1

The output with both scripts are at:

	https://www.infradead.org/~mchehab/rst_features/index.html

The relevant parts are: ABI:

	https://www.infradead.org/~mchehab/rst_features/admin-guide/abi.html

Feature list x architecture (at admin-guide:

	https://www.infradead.org/~mchehab/rst_features/admin-guide/features.html

X86 features:

	https://www.infradead.org/~mchehab/rst_features/x86/features.html

While I didn't write a patch, with the new get_feat.pl script, we can probably
get rid of the previous shell script at:

	Documentation/features/list-arch.sh

As calling:

	./scripts/get_feat.pl current

Will output the same content (with a different format, though).

Thanks,
Mauro

Re: [Ksummit-discuss] [TECH TOPIC] Documentation

[Kate Stewart]

On Wed, Jun 12, 2019 at 9:56 AM Jonathan Corbet <corbet@lwn.net> wrote:
>
> What could be more fun than talking about kernel documentation?  Things we
> could get into:
>
>  - The state of the RST transition, what remains to be done, whether it's
>    all just useless churn that makes the documentation worse, etc.
>
>  - Things we'd like to improve in the documentation toolchain.
>
>  - Overall organization of Documentation/ and moving docs when the need
>    arises.  It seems I end up fighting about this more than just about
>    anything else, but I think it's important to organize our docs for the
>    convenience of the people using them.
>
>  - The ultimate vision for kernel docs (for now).  RST conversion and
>    imposing some organization are important, but they will not,
>    themselves, give us a coherent set of documentation.  What can we do to
>    have documentation that is useful, current, and maintainable, rather
>    than the dusty attic we have now?

Also,  it would be great if we could talk about cleaning up the
documentation licensing, so it too can have its licenses be
automatically detected.   :-)

Kate

Re: [Ksummit-discuss] [TECH TOPIC] Documentation

[Mauro Carvalho Chehab]

Em Wed, 12 Jun 2019 15:33:36 -0500
Kate Stewart <kstewart@linuxfoundation.org> escreveu:

> On Wed, Jun 12, 2019 at 9:56 AM Jonathan Corbet <corbet@lwn.net> wrote:
> >
> > What could be more fun than talking about kernel documentation?  Things we
> > could get into:
> >
> >  - The state of the RST transition, what remains to be done, whether it's
> >    all just useless churn that makes the documentation worse, etc.
> >
> >  - Things we'd like to improve in the documentation toolchain.
> >
> >  - Overall organization of Documentation/ and moving docs when the need
> >    arises.  It seems I end up fighting about this more than just about
> >    anything else, but I think it's important to organize our docs for the
> >    convenience of the people using them.
> >
> >  - The ultimate vision for kernel docs (for now).  RST conversion and
> >    imposing some organization are important, but they will not,
> >    themselves, give us a coherent set of documentation.  What can we do to
> >    have documentation that is useful, current, and maintainable, rather
> >    than the dusty attic we have now?  
> 
> Also,  it would be great if we could talk about cleaning up the
> documentation licensing, so it too can have its licenses be
> automatically detected.   :-)

With that regards, I'm still waiting for a solution with for the
GFDL license:

	https://github.com/spdx/license-list-XML/issues/686

All documents under Documentation/media now have a SPDX header,
except for the ones that were originally licensed under
the free version of GNU Free Document License (e. g. with
no invariant sections, no Front-Cover texts and no Back-Cover
texts) or that are dual-licensed GFDL/GPL.

While we don't have a SPDX tag for those, we can't finish adding
SPDX headers there though :-(


Thanks,
Mauro

Re: [Ksummit-discuss] [TECH TOPIC] Documentation

[Mauro Carvalho Chehab]

Em Wed, 12 Jun 2019 08:54:05 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> What could be more fun than talking about kernel documentation?

That sounds a topic that we should have some discussion. IMO, the best
would be to put people on some round tables and exchange some ideas
about how we can improve the process. I would try to do it as a
half-day topic, if the agenda would allow, as a standard 50' min
slot doesn't seem to be enough to address the needed things there.

> Things we
> could get into:
> 
>  - The state of the RST transition, what remains to be done, whether it's
>    all just useless churn that makes the documentation worse, etc.

I have a large changeset (with about one patch per Documentation/
dir) with manually written patches with addresses a large amount of
the remaining stuff. I'm hoping to have most of it applied before
KS :-)

There will still be 4 or 5 directories with lots of documentation
files on it, plus ABI and DT. 

>  - Things we'd like to improve in the documentation toolchain.

I remember once I submitted a patch with a script capable of
parsing the files at ABI and produce a parsed content, with the
goal of validating the ABI files against its syntax and to
generate an ABI book.

IMO, it makes sense to have one book containing the Linux ABI, but, 
as the patch didn't have much attention, and I got sidetracked with
something else, I ended giving up on trying to push it. 

Perhaps by KS it would be time to rescue it from some old git tree
and see if it would be worth the time to work on it.

> 
>  - Overall organization of Documentation/ and moving docs when the need
>    arises.  It seems I end up fighting about this more than just about
>    anything else, but I think it's important to organize our docs for the
>    convenience of the people using them.

Fully agreed. It is not always clear where to place some converted
docs.

Also, in the case of driver-specific information, it is not unusual to
have both kernel and user's information at the same file. Ideally, the
best would be to split, but I'm not sure if it would worth the time
for doing such changes.

>  - The ultimate vision for kernel docs (for now).  RST conversion and
>    imposing some organization are important, but they will not,
>    themselves, give us a coherent set of documentation.  What can we do to
>    have documentation that is useful, current, and maintainable, rather
>    than the dusty attic we have now?

Yeah, RST conversion and enforcing an organization of them is important,
as it will enforce a single "coding" style for the documents, but this
is only a small part of the issue.

IMO, we should work to attract people focused on maintaining docs.

I've been doing some userspace maintainership on a few media-related
projects. Among them, I'm maintaining a KDE media player (Kaffeine).

The KDE project has a group of people that it is focused only on
writing documentation, and one group of people per translation language.

IMO, if we want to have a good set of documents, we should have a
group of people working mainly with a "documentation hat", reviewing,
improving and rewriting large chunks of documents.

If we ever have such kind of people working around the Kernel,
I guess the first task would be to imagine what kind of things a new 
Kernel developer needs to know and try to structure a coherent
documentation from that. Such group could use the existing Kernel
printed books as a starting point (perhaps using existing books
documentation that their authors could release them under GPLv2),
and modifying the texts to use what we have nowadays inside the
Kernel's tree.

I guess the main problem is that, for this to work, someone
(possibly a non-profit org) would likely need to sponsor such work,
as this would probably require more time than we could do on our
spare time.

Thanks,
Mauro

Re: [Ksummit-discuss] [TECH TOPIC] Documentation

[Greg KH]

On Thu, Jun 13, 2019 at 11:57:12AM -0300, Mauro Carvalho Chehab wrote:
> >  - Things we'd like to improve in the documentation toolchain.
> 
> I remember once I submitted a patch with a script capable of
> parsing the files at ABI and produce a parsed content, with the
> goal of validating the ABI files against its syntax and to
> generate an ABI book.
> 
> IMO, it makes sense to have one book containing the Linux ABI, but, 
> as the patch didn't have much attention, and I got sidetracked with
> something else, I ended giving up on trying to push it. 
> 
> Perhaps by KS it would be time to rescue it from some old git tree
> and see if it would be worth the time to work on it.

I liked that patcheset.  If you can dig it up, I can work on reviving it
if you don't want to.

thanks,

greg k-h

Re: [Ksummit-discuss] [TECH TOPIC] Documentation

[Mauro Carvalho Chehab]

Em Thu, 13 Jun 2019 20:48:08 +0200
Greg KH <greg@kroah.com> escreveu:

> On Thu, Jun 13, 2019 at 11:57:12AM -0300, Mauro Carvalho Chehab wrote:
> > >  - Things we'd like to improve in the documentation toolchain.  
> > 
> > I remember once I submitted a patch with a script capable of
> > parsing the files at ABI and produce a parsed content, with the
> > goal of validating the ABI files against its syntax and to
> > generate an ABI book.
> > 
> > IMO, it makes sense to have one book containing the Linux ABI, but, 
> > as the patch didn't have much attention, and I got sidetracked with
> > something else, I ended giving up on trying to push it. 
> > 
> > Perhaps by KS it would be time to rescue it from some old git tree
> > and see if it would be worth the time to work on it.  
> 
> I liked that patcheset.  If you can dig it up, I can work on reviving it
> if you don't want to.

The patches are here:

	https://git.linuxtv.org/mchehab/experimental-old.git/commit/?h=abi_docs_v3&id=da4b94205aced79df7389ca6f2ec4328fb8604a7

I'll rebase them on the top of linux-next and re-submit.

Thanks,
Mauro

Re: [Ksummit-discuss] [TECH TOPIC] Documentation

[Kees Cook]

On Wed, Jun 12, 2019 at 08:54:05AM -0600, Jonathan Corbet wrote:
>  - Things we'd like to improve in the documentation toolchain.

Just to put it into this thread... I've seen a recurring request for
markup improvement to have Sphinx be smart enough to linkify:

	function()

instead of needing to do:

	:c:func:`iounmap()`

And similarly for: struct namegoeshere

-- 
Kees Cook

Re: [Ksummit-discuss] [TECH TOPIC] Documentation

[Jonathan Corbet]

On Thu, 20 Jun 2019 11:36:38 -0700
Kees Cook <keescook@chromium.org> wrote:

> Just to put it into this thread... I've seen a recurring request for
> markup improvement to have Sphinx be smart enough to linkify:
> 
> 	function()
> 
> instead of needing to do:
> 
> 	:c:func:`iounmap()`

I already have a patch to do that:

	https://lwn.net/ml/linux-doc/20190425200125.12302-1-corbet@lwn.net/

It needs another bit of attention before it's ready, and I've not yet
managed to do that.  Soon, I promise :)

Thanks,

jon

Re: [Ksummit-discuss] [TECH TOPIC] Documentation

[Mauro Carvalho Chehab]

Em Wed, 12 Jun 2019 08:54:05 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> What could be more fun than talking about kernel documentation?  Things we
> could get into:
> 
>  - The state of the RST transition

Assuming that all patches on my development branch gets merged, we should
have:

	- about 300 files missing conversion;
	- about 40 files under Documentation/*.txt that needs to be
	  moved to a subdir and be renamed to *.rst.

>  - Overall organization of Documentation/ and moving docs when the need
>    arises.

Looking at Documentation/index.rst, I guess we're aiming to have something 
like:

	Documentation/
	├── ABI
	├── admin-guide
	├── kbuild
	├── arch
	│   ├── arm
	│   ├── arm64
	│   ...
	│   ├── x86
	│   │   ├── i386
	│   │   └── x86_64
	│   └── xtensa
	├── core-api
	├── devicetree
	├── dev-tools
	├── doc-guide
	├── driver-api
	├── features
	├── fault-injection
	├── filesystems
	├── firmware-guide
	├── kernel-hacking
	├── livepatch
	├── maintainer
	├── process
	├── trace
	└── translations

Btw, right now, ext4 fs docs is on two separate parts of 
Documentation/index.rst:

   filesystems/index
   filesystems/ext4/index

We should probably get rid of filesystems/ext4/index entry (and the 
corresponding PDF entry at conf.py).

-

IMO, the main work to be done in order to achieve that is related to
Driver's subsystem documentation.

What I've been doing so far - at least for most (if not all) driver docs 
that carry more than one documentation type at the same subdir (kABI, 
uABI and/or admin-guide) is to keep the directory as-is, adding them under
this section at Documentation/index.rst:

	Kernel API documentation
	------------------------

There are a couple of reasons why I opted for this strategy when
I did such conversions:

1) There are *lots* of docs that contain all 3 types of information
   on it on a single file.

2) On media, we use SPHINXDIRS to produce the media book from our
   devel tree:

	https://linuxtv.org/downloads/v4l-dvb-apis-new/index.html

   When documents are built with either PDF or SPHINXDIRS, each subdir
   will be on a different book and all inter-book cross-references
   will break.
   
   For this to be fixed, intersphinx extension would be required,
   but this would probably require a per-subsystem mapping 
   (for example, saying that the site used to resolve media
   broken cross references is linuxtv.org). 

   Maintaining it can be painful, as we would have a big table
   at conf.py with subsystem-specific stuff.

3) So far, I was unable to split even the media docs. Shame on
   me! The reason is that this is not an easy task.

One interesting example is the open() documentation at the
media media uAPI book:

	Documentation/media/uapi/v4l/open.rst

This file actually contains a lot of sysadmin relevant data (so, it
is a good candidate for the admin-guide).

Yet, it was written focused on what a media uAPI developer needs
to know. So, it also mentions Kernel userspace API syscalls:
open(), read(), close() - with has cross-references to other parts
of the uAPI book.

Splitting this file on two separate books won't be that easy.

Ideally, we should split what's there at media/uapi into admin-guide
and userspace-api, but this would mean *a lot* of effort. Not sure
if it is worth the time.

Also, while a sysadmin might want to know what a /dev/video0 device
means, the intended audience is really uAPI developers, as an user
will just click on its GUI to call a media application.

>  - The ultimate vision for kernel docs (for now).  RST conversion and
>    imposing some organization are important, but they will not,
>    themselves, give us a coherent set of documentation.  What can we do to
>    have documentation that is useful, current, and maintainable, rather
>    than the dusty attic we have now?

The more I think the more I'm convinced that the best way to proceed would
be to use some Kernel books as an example and organize the main index.rst
files on a way that it will cover what a newbie Kernel developer would need
to know.

For example, looking at LDD3 organization (https://lwn.net/Kernel/LDD3/),
we could use its index as an starting point for what a driver-api book
should contain:

	Chapter 1: An Introduction to Device Drivers
	Chapter 3: Char Drivers
	Chapter 6: Advanced Char Driver Operations
	Chapter 9: Communicating with Hardware
	Chapter 12: PCI Drivers
	Chapter 13: USB Drivers
	Chapter 14: The Linux Device Model
	Chapter 16: Block Drivers
	Chapter 17: Network Drivers
	Chapter 18: TTY Drivers

Just looking on these, our driver-api book seem to be missing the
texts that would glue its contents, e. g. an introduction to device
drivers, to char/block devices and about how to communicate with the
hardware.

It can also help to identify the contents that a driver developer
would need from a core-api and a Kernel development bookset:

	Chapter 2: Building and Running Modules
	Chapter 4: Debugging Techniques
	Chapter 5: Concurrency and Race Conditions
	Chapter 7: Time, Delays, and Deferred Work
	Chapter 8: Allocating Memory
	Chapter 10: Interrupt Handling
	Chapter 11: Data Types in the Kernel

I would do the same with other Linux and Linux Kernel related books.

-

Btw, if the authors of some existing old books release their stuff under
GPLv2 and allow us to import their contents, we could try to import some
parts of it that aren't too obsolete. The new automarkup extension
can help a lot to identify outdated documents, as it won't be able
to solve the func() calls.

Thanks,
Mauro