* [Ksummit-discuss] [TECH TOPIC] Documentation @ 2019-06-12 14:54 Jonathan Corbet 2019-06-12 18:22 ` Shuah Khan ` (4 more replies) 0 siblings, 5 replies; 20+ messages in thread From: Jonathan Corbet @ 2019-06-12 14:54 UTC (permalink / raw) To: ksummit-discuss What could be more fun than talking about kernel documentation? Things we could get into: - The state of the RST transition, what remains to be done, whether it's all just useless churn that makes the documentation worse, etc. - Things we'd like to improve in the documentation toolchain. - Overall organization of Documentation/ and moving docs when the need arises. It seems I end up fighting about this more than just about anything else, but I think it's important to organize our docs for the convenience of the people using them. - The ultimate vision for kernel docs (for now). RST conversion and imposing some organization are important, but they will not, themselves, give us a coherent set of documentation. What can we do to have documentation that is useful, current, and maintainable, rather than the dusty attic we have now? jon ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] Documentation 2019-06-12 14:54 [Ksummit-discuss] [TECH TOPIC] Documentation Jonathan Corbet @ 2019-06-12 18:22 ` Shuah Khan 2019-06-12 19:12 ` Martin K. Petersen 2019-06-13 14:25 ` Mauro Carvalho Chehab 2019-06-12 20:33 ` [Ksummit-discuss] [TECH TOPIC] Documentation Kate Stewart ` (3 subsequent siblings) 4 siblings, 2 replies; 20+ messages in thread From: Shuah Khan @ 2019-06-12 18:22 UTC (permalink / raw) To: ksummit-discuss On 6/12/19 8:54 AM, Jonathan Corbet wrote: > What could be more fun than talking about kernel documentation? Things we > could get into: > > - The state of the RST transition, what remains to be done, whether it's > all just useless churn that makes the documentation worse, etc. > > - Things we'd like to improve in the documentation toolchain. > > - Overall organization of Documentation/ and moving docs when the need > arises. It seems I end up fighting about this more than just about > anything else, but I think it's important to organize our docs for the > convenience of the people using them. > > - The ultimate vision for kernel docs (for now). RST conversion and > imposing some organization are important, but they will not, > themselves, give us a coherent set of documentation. What can we do to > have documentation that is useful, current, and maintainable, rather > than the dusty attic we have now? > The first step is identifying the ones that are out of date and/or incorrect. I think this is probably the most time consuming part and the second is start updating. I am currently putting together a task list for mentorship program with a goal to create tasks that would more meaningful and helpful instead of whitespace patches. I will gladly add documentation tasks to the list to help improve the documentation. thanks, -- Shuah ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] Documentation 2019-06-12 18:22 ` Shuah Khan @ 2019-06-12 19:12 ` Martin K. Petersen 2019-06-12 19:43 ` Shuah Khan 2019-06-13 14:25 ` Mauro Carvalho Chehab 1 sibling, 1 reply; 20+ messages in thread From: Martin K. Petersen @ 2019-06-12 19:12 UTC (permalink / raw) To: Shuah Khan; +Cc: ksummit-discuss Shuah, > I am currently putting together a task list for mentorship program > with a goal to create tasks that would more meaningful and helpful > instead of whitespace patches. If somebody is looking for meaningful documentation work, I have a substantial project in need of volunteers: The target stack wiki is stale, unmaintained, and often inaccessible. I would love for somebody to convert all that information to RST so we can stick it in the kernel and keep it in sync with the code changes. -- Martin K. Petersen Oracle Linux Engineering ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] Documentation 2019-06-12 19:12 ` Martin K. Petersen @ 2019-06-12 19:43 ` Shuah Khan 0 siblings, 0 replies; 20+ messages in thread From: Shuah Khan @ 2019-06-12 19:43 UTC (permalink / raw) To: Martin K. Petersen; +Cc: ksummit-discuss On 6/12/19 1:12 PM, Martin K. Petersen wrote: > > Shuah, > >> I am currently putting together a task list for mentorship program >> with a goal to create tasks that would more meaningful and helpful >> instead of whitespace patches. > > If somebody is looking for meaningful documentation work, I have a > substantial project in need of volunteers: The target stack wiki is > stale, unmaintained, and often inaccessible. I would love for somebody > to convert all that information to RST so we can stick it in the kernel > and keep it in sync with the code changes. > Awesome. Please send the project description to me. Feel free to break it up as needed. We can chat offline about it. thanks, -- Shuah ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] Documentation 2019-06-12 18:22 ` Shuah Khan 2019-06-12 19:12 ` Martin K. Petersen @ 2019-06-13 14:25 ` Mauro Carvalho Chehab 2019-06-14 21:40 ` Shuah Khan 1 sibling, 1 reply; 20+ messages in thread From: Mauro Carvalho Chehab @ 2019-06-13 14:25 UTC (permalink / raw) To: Shuah Khan; +Cc: ksummit-discuss Em Wed, 12 Jun 2019 12:22:55 -0600 Shuah Khan <skhan@linuxfoundation.org> escreveu: > On 6/12/19 8:54 AM, Jonathan Corbet wrote: > > What could be more fun than talking about kernel documentation? Things we > > could get into: > > > > - The state of the RST transition, what remains to be done, whether it's > > all just useless churn that makes the documentation worse, etc. > > > > - Things we'd like to improve in the documentation toolchain. > > > > - Overall organization of Documentation/ and moving docs when the need > > arises. It seems I end up fighting about this more than just about > > anything else, but I think it's important to organize our docs for the > > convenience of the people using them. > > > > - The ultimate vision for kernel docs (for now). RST conversion and > > imposing some organization are important, but they will not, > > themselves, give us a coherent set of documentation. What can we do to > > have documentation that is useful, current, and maintainable, rather > > than the dusty attic we have now? > > > > The first step is identifying the ones that are out of date and/or > incorrect. I think this is probably the most time consuming part > and the second is start updating. I would do just the opposite: convert the documentation, then update it. The reason is that, when someone sends a patch to the documentation, people tend to look more closely on it, and we start receiving updates for it. Ok, this doesn't happen every time, but it is still worth. There's another thing to consider here: there are lots of docs written for "stable" subsystems, in the sense that the documented code doesn't change for a lot of time (except for trivial patches not directly related to it. While doing large chunks of code conversion, I noticed that there are lots of still valid documents that fit on this. There's a big issue with those "stable" subsystems: they tend to not have any active maintainer anymore. So, there are not much people that are willing to actively review patches to it. So, my advice here is to really invert things: - do the conversion; - find a good place to put the converted the book; - add kernel-doc markups to be sure that the symbols will always match upstream; - review the remaining content of the docs. > I am currently putting together a task list for mentorship program > with a goal to create tasks that would more meaningful and helpful > instead of whitespace patches. > > I will gladly add documentation tasks to the list to help improve > the documentation. I can surely mentor people interested on doing such tasks. > > thanks, > -- Shuah > > > > _______________________________________________ > Ksummit-discuss mailing list > Ksummit-discuss@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss Thanks, Mauro ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] Documentation 2019-06-13 14:25 ` Mauro Carvalho Chehab @ 2019-06-14 21:40 ` Shuah Khan 2019-06-15 0:05 ` Mauro Carvalho Chehab 0 siblings, 1 reply; 20+ messages in thread From: Shuah Khan @ 2019-06-14 21:40 UTC (permalink / raw) To: Mauro Carvalho Chehab; +Cc: ksummit-discuss On 6/13/19 8:25 AM, Mauro Carvalho Chehab wrote: > Em Wed, 12 Jun 2019 12:22:55 -0600 > Shuah Khan <skhan@linuxfoundation.org> escreveu: > >> On 6/12/19 8:54 AM, Jonathan Corbet wrote: >>> What could be more fun than talking about kernel documentation? Things we >>> could get into: >>> >>> - The state of the RST transition, what remains to be done, whether it's >>> all just useless churn that makes the documentation worse, etc. >>> >>> - Things we'd like to improve in the documentation toolchain. >>> >>> - Overall organization of Documentation/ and moving docs when the need >>> arises. It seems I end up fighting about this more than just about >>> anything else, but I think it's important to organize our docs for the >>> convenience of the people using them. >>> >>> - The ultimate vision for kernel docs (for now). RST conversion and >>> imposing some organization are important, but they will not, >>> themselves, give us a coherent set of documentation. What can we do to >>> have documentation that is useful, current, and maintainable, rather >>> than the dusty attic we have now? >>> >> >> The first step is identifying the ones that are out of date and/or >> incorrect. I think this is probably the most time consuming part >> and the second is start updating. > > I would do just the opposite: convert the documentation, then > update it. > Yes. That is fine. I am referring to the outdated part. Converting and updating them would work just fine. > The reason is that, when someone sends a patch to the documentation, > people tend to look more closely on it, and we start receiving > updates for it. > > Ok, this doesn't happen every time, but it is still worth. > > There's another thing to consider here: there are lots of docs > written for "stable" subsystems, in the sense that the documented > code doesn't change for a lot of time (except for trivial patches > not directly related to it. > > While doing large chunks of code conversion, I noticed that there > are lots of still valid documents that fit on this. > > There's a big issue with those "stable" subsystems: they tend > to not have any active maintainer anymore. So, there are not > much people that are willing to actively review patches to it. > > So, my advice here is to really invert things: > > - do the conversion; Even this can be made into tasks. If you would like to experiment with and see how it works, send me a list of documents that you would like to see converted first. > - find a good place to put the converted the book; > - add kernel-doc markups to be sure that the symbols will always > match upstream; > - review the remaining content of the docs. > >> I am currently putting together a task list for mentorship program >> with a goal to create tasks that would more meaningful and helpful >> instead of whitespace patches. >> >> I will gladly add documentation tasks to the list to help improve >> the documentation. > > I can surely mentor people interested on doing such tasks. > That will be great. thanks, -- Shuah ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] Documentation 2019-06-14 21:40 ` Shuah Khan @ 2019-06-15 0:05 ` Mauro Carvalho Chehab 2019-06-17 10:12 ` Mauro Carvalho Chehab 0 siblings, 1 reply; 20+ messages in thread From: Mauro Carvalho Chehab @ 2019-06-15 0:05 UTC (permalink / raw) To: Shuah Khan; +Cc: ksummit-discuss Em Fri, 14 Jun 2019 15:40:26 -0600 Shuah Khan <skhan@linuxfoundation.org> escreveu: > On 6/13/19 8:25 AM, Mauro Carvalho Chehab wrote: > > Em Wed, 12 Jun 2019 12:22:55 -0600 > > Shuah Khan <skhan@linuxfoundation.org> escreveu: > > > > So, my advice here is to really invert things: > > > > - do the conversion; > > Even this can be made into tasks. If you would like to experiment > with and see how it works, send me a list of documents that you > would like to see converted first. After the patchsets I'm working it: https://git.linuxtv.org/mchehab/experimental.git/log/?h=convert_rst_renames_v5 There will be very few directories that doesn't any any .rst file: $ for i in $(find Documentation/ -name '*.txt' | sed -E "s,(Documentation/[^\/+]+).*,\1,"|uniq|grep -v output |grep -v binding|grep -v devicetree); do if [ "$(find $i -name '*.rst')" == "" ]; then if [ -d $i ]; then echo $i; fi; fi; done Documentation/platform Documentation/scsi Documentation/RCU Documentation/cpu-freq Documentation/ABI Documentation/sphinx Documentation/features Documentation/acpi Among those: - There's nothing to be done at the sphinx directory. - The cpu-freq maintainer won't want changes there, as the stuff inside that specific dir is obsolete. - If I'm not mistaken, there's already a pending patchset for the acpi directory too. - Platform has just a single file, easily convertible to ReST. I suspect that it should be moved to the laptops dir too. I guess I wrote a patch for it, but it seems it got lost on some rebase. Anyway, I can take care of this one. - We are handling the ABI directory on a different way. So, I guess that what it was left behind is: Documentation/scsi Documentation/RCU Documentation/features You should notice, however, that there are some other directories that have a mix of rst and txt files, with also have some files that could be needing conversion: $ for i in $(find Documentation/ -name '*.txt' | sed -E "s,(Documentation/[^\/+]+).*,\1,"|uniq|grep -v output |grep -v binding|grep -v devicetree); do if [ -d $i ]; then echo $i; fi; done Documentation/device-mapper Documentation/sh Documentation/trace Documentation/misc-devices Documentation/arm64 Documentation/platform Documentation/scsi Documentation/RCU Documentation/virtual Documentation/cpu-freq Documentation/admin-guide Documentation/block Documentation/ABI Documentation/translations Documentation/PCI Documentation/filesystems Documentation/networking Documentation/sphinx Documentation/features Documentation/netlabel Documentation/acpi Documentation/crypto Documentation/sparc I suspect that those dirs with contain botn .rst and .txt files are in the process of being updated and converted by the subsystem maintainers, but double check is needed. Plus other places where there are files whose filename doesn't have any extension at all and could be a text file to be converted: $ for i in $(find Documentation/ -type f ! -name "*.*" | sed -E "s,(Documentation/[^\/+]+).*,\1,"|uniq|grep -v output |grep -v binding|grep -v devicetree); do if [ -d $i ]; then echo $i; fi; done Documentation/misc-devices Documentation/target Documentation/firmware_class Documentation/isdn Documentation/media Documentation/parisc Documentation/spi Documentation/scsi Documentation/hwmon Documentation/EDID Documentation/nios2 Documentation/virtual Documentation/i2c Documentation/usb Documentation/ABI Documentation/translations Documentation/openrisc Documentation/w1 Documentation/filesystems Documentation/networking Documentation/auxdisplay There are a few exceptions that should be kept as plain text files. I remember of two such cases: - a configuration file whose extension is .txt; - a dump of chapter extracted from an old arch-specific manual with seems to be provided by its manufacturer. While the last one could eventually be converted, it sounded too much work for too little gain - and a format change may eventually require some negotiation with the copyright holder. Thanks, Mauro ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] Documentation 2019-06-15 0:05 ` Mauro Carvalho Chehab @ 2019-06-17 10:12 ` Mauro Carvalho Chehab 2019-06-17 17:21 ` Mauro Carvalho Chehab 0 siblings, 1 reply; 20+ messages in thread From: Mauro Carvalho Chehab @ 2019-06-17 10:12 UTC (permalink / raw) To: Shuah Khan, Jonathan Corbet; +Cc: ksummit-discuss Em Fri, 14 Jun 2019 21:05:36 -0300 Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu: > Em Fri, 14 Jun 2019 15:40:26 -0600 > Shuah Khan <skhan@linuxfoundation.org> escreveu: > > > On 6/13/19 8:25 AM, Mauro Carvalho Chehab wrote: > > > Em Wed, 12 Jun 2019 12:22:55 -0600 > > > Shuah Khan <skhan@linuxfoundation.org> escreveu: > > > > > > > So, my advice here is to really invert things: > > > > > > - do the conversion; > > > > Even this can be made into tasks. If you would like to experiment > > with and see how it works, send me a list of documents that you > > would like to see converted first. > > After the patchsets I'm working it: > > https://git.linuxtv.org/mchehab/experimental.git/log/?h=convert_rst_renames_v5 > > There will be very few directories that doesn't any any .rst file: > > $ for i in $(find Documentation/ -name '*.txt' | sed -E "s,(Documentation/[^\/+]+).*,\1,"|uniq|grep -v output |grep -v binding|grep -v devicetree); do if [ "$(find $i -name '*.rst')" == "" ]; then if [ -d $i ]; then echo $i; fi; fi; done > > Documentation/platform > Documentation/scsi > Documentation/RCU > Documentation/cpu-freq > Documentation/ABI > Documentation/sphinx > Documentation/features > Documentation/acpi > > Among those: > > - There's nothing to be done at the sphinx directory. > - The cpu-freq maintainer won't want changes there, as the stuff inside > that specific dir is obsolete. > - If I'm not mistaken, there's already a pending patchset for the > acpi directory too. > - Platform has just a single file, easily convertible to ReST. > I suspect that it should be moved to the laptops dir too. > I guess I wrote a patch for it, but it seems it got lost on > some rebase. Anyway, I can take care of this one. > - We are handling the ABI directory on a different way. > > So, I guess that what it was left behind is: > > Documentation/scsi > Documentation/RCU > Documentation/features This is actually a special case. I don't think that a plain txt->rst conversion is the right thing to be done here. IMO, it requires some further discussions. The files there have a standard format, with is (poorly) documented at Documentation/features/arch-support.txt. Getting a random file there, the format has a description part and a table (Documentation/features/debug/kprobes/arch-support.txt): # Feature name: kprobes # Kconfig: HAVE_KPROBES # description: arch supports live patched kernel probe # ----------------------- | arch |status| ----------------------- | alpha: | TODO | | arc: | ok | | arm: | ok | ... | xtensa: | TODO | ----------------------- E. g. each file contains the name of a feature, its Kconfig symbol, a description and a table with shows what architectures support and what architectures don't. I actually started converting it, but the results don't look nice. The thing is: it is a lot easier for a "feature" maintainer to have a single file with a per-architecture status, as he's free to update the file without needing to be concerned about merge conflicts. So, the current way makes a perfect sense from a developer's PoV. - However, for a Kernel user's perspective, he may be interested on looking at this as a completely different way. 1) he may want to buy a hardware whose support a certain feature subset; 2) He has already some hardware, any may want to identify if a certain feature is there. For (1), the best would be a table like: ======= ===== ===== ===== ===== ====== Feature alpha arc arm arm64 .... xtensa ======= ===== ===== ===== ===== ====== KASAN TODO TODO TODO ok ok kdb TODO ok ok ok TODO ... ======= ===== ===== ===== ===== ====== (eventually with Kconfig and description, but we need to double check if it won't bee to big at the pdf output). And for (2), one file per architecture, with something like: Feature status at x86 Architecture ================================== ======= ================= ====== ============================================== Feature Kconfig Status Description ======= ================= ====== ============================================== KASAN HAVE_ARCH_KASAN ok arch supports the KASAN runtime memory checker kgdb HAVE_ARCH_KGDB ok arch supports the kGDB kernel debugger ... ======= ================= ====== ============================================== So, IMHO, the best thing to do with the feature files is an approach similar to the one we're doing with ABI, e. g.: 1) improve the contents of "arch-support.txt" file for it to precisely describe the format of the file, including the header. While not required, eventually we could change the headers to something less prune to human errors, e. g. to something like: name: kprobes Kconfig: HAVE_KPROBES description: arch supports live patched kernel probe 2) have a script similar to get_abi.pl that would parse, validate and produce a per-architecture output file; 3) add a per-arch features file that will be automatically generated by the script. The script should be able to produce valid ReST output on all the above formats, e. g.: 1) output per feature; 2) output per architecture; 3) output as a matrix feature x architecture. This way, we could add the feature lists on multiple books: - admin-doc: feature x architecture view - arch-specific view: feature status for such specific arch - kAPI book: per feature output Comments? Thanks, Mauro ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] Documentation 2019-06-17 10:12 ` Mauro Carvalho Chehab @ 2019-06-17 17:21 ` Mauro Carvalho Chehab 2019-06-17 18:05 ` [Ksummit-discuss] [PATCH RFC] scripts: add a script to handle Documentation/features Mauro Carvalho Chehab 0 siblings, 1 reply; 20+ messages in thread From: Mauro Carvalho Chehab @ 2019-06-17 17:21 UTC (permalink / raw) To: Shuah Khan, Jonathan Corbet; +Cc: ksummit-discuss Em Mon, 17 Jun 2019 07:12:06 -0300 Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu: Answering myself here... > > Documentation/features > > This is actually a special case. I don't think that a plain txt->rst > conversion is the right thing to be done here. > > IMO, it requires some further discussions. > The script should be able to produce valid ReST output on all the > above formats, e. g.: > > 1) output per feature; > 2) output per architecture; > 3) output as a matrix feature x architecture. I'm actually thinking on something like: 1) A page that could be part of a feature documentation (On this example, it documents scheduler ARCH_HAS_MEMBARRIER_SYNC_CORE) https://www.infradead.org/~mchehab/rst_features/feature_membarrier-sync-core.html 2) A page that could be part of the per-archtecture books (here, arm64 one): https://www.infradead.org/~mchehab/rst_features/feature_arm64.html 3) A page with a matrix with all architectures and all features: https://www.infradead.org/~mchehab/rst_features/feature_matrix.html I just wrote a patch with such script. I should be submitting it soon to the ML for review, after some final cleanups. Thanks, Mauro ^ permalink raw reply [flat|nested] 20+ messages in thread
* [Ksummit-discuss] [PATCH RFC] scripts: add a script to handle Documentation/features 2019-06-17 17:21 ` Mauro Carvalho Chehab @ 2019-06-17 18:05 ` Mauro Carvalho Chehab 2019-06-17 18:11 ` Greg Kroah-Hartman 0 siblings, 1 reply; 20+ messages in thread From: Mauro Carvalho Chehab @ 2019-06-17 18:05 UTC (permalink / raw) To: Linux Doc Mailing List, Jonathan Corbet, Greg Kroah-Hartman Cc: Mauro Carvalho Chehab, linux-kernel, ksummit-discuss The Documentation/features contains a set of parseable files. It is not worth converting them to ReST format, as they're useful the way it is. It is, however, interesting to parse them and produce output on different formats: 1) Output the contents of a feature in ReST format; 2) Output what features a given architecture supports; 3) Output a matrix with features x architectures. Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org> --- As commented at KS mailing list, converting the Documentation/features file to ReST may not be the best way to handle it. This script allows validating the features files and to generate ReST files on three different formats. The goal is to support it via a sphinx extension, in order to be able to add the features inside the Kernel documentation. scripts/get_feat.pl | 470 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 470 insertions(+) create mode 100755 scripts/get_feat.pl diff --git a/scripts/get_feat.pl b/scripts/get_feat.pl new file mode 100755 index 000000000000..c5a267b12f49 --- /dev/null +++ b/scripts/get_feat.pl @@ -0,0 +1,470 @@ +#!/usr/bin/perl + +use strict; +use Pod::Usage; +use Getopt::Long; +use File::Find; +use Fcntl ':mode'; + +my $help; +my $man; +my $debug; +my $arch; +my $feat; +my $prefix="Documentation/features"; + +GetOptions( + "debug|d+" => \$debug, + 'help|?' => \$help, + 'arch=s' => \$arch, + 'feat=s' => \$feat, + man => \$man +) or pod2usage(2); + +pod2usage(1) if $help; +pod2usage(-exitstatus => 0, -verbose => 2) if $man; + +pod2usage(2) if (scalar @ARGV < 1 || @ARGV > 2); + +my ($cmd, $arg) = @ARGV; + +pod2usage(2) if ($cmd ne "current" && $cmd ne "rest" && $cmd ne "validate"); + +require Data::Dumper if ($debug); + +my %data; +my %archs; + +# +# Displays an error message, printing file name and line +# +sub parse_error($$$$) { + my ($file, $ln, $msg, $data) = @_; + + $data =~ s/\s+$/\n/; + + print STDERR "Warning: file $file#$ln:\n\t$msg"; + + if ($data ne "") { + print STDERR ". Line\n\t\t$data"; + } else { + print STDERR "\n"; + } +} + +# +# Parse a features file, storing its contents at %data +# + +my $h_name = "Feature"; +my $h_kconfig = "Kconfig"; +my $h_description = "Description"; +my $h_subsys = "Subsystem"; +my $h_status = "Status"; +my $h_arch = "Architecture"; + +my $max_size_name = length($h_name); +my $max_size_kconfig = length($h_kconfig); +my $max_size_description = length($h_description); +my $max_size_subsys = length($h_subsys); +my $max_size_status = length($h_status); +my $max_size_arch = length($h_arch); + +sub parse_feat { + my $file = $File::Find::name; + + my $mode = (stat($file))[2]; + return if ($mode & S_IFDIR); + return if ($file =~ m,($prefix)/arch-support.txt,); + return if (!($file =~ m,arch-support.txt$,)); + + my $subsys = ""; + $subsys = $2 if ( m,.*($prefix)/([^/]+).*,); + + if (length($subsys) > $max_size_subsys) { + $max_size_subsys = length($subsys); + } + + my $name; + my $kconfig; + my $description; + my $comments = ""; + my $last_status; + my $ln; + my %arch_table; + + print STDERR "Opening $file\n" if ($debug > 1); + open IN, $file; + + while(<IN>) { + $ln++; + + if (m/^\#\s+Feature\s+name:\s*(.*\S)/) { + $name = $1; + if (length($name) > $max_size_name) { + $max_size_name = length($name); + } + next; + } + if (m/^\#\s+Kconfig:\s*(.*\S)/) { + $kconfig = $1; + if (length($kconfig) > $max_size_kconfig) { + $max_size_kconfig = length($kconfig); + } + next; + } + if (m/^\#\s+description:\s*(.*\S)/) { + $description = $1; + if (length($description) > $max_size_description) { + $max_size_description = length($description); + } + next; + } + next if (m/^\\s*$/); + next if (m/^\s*\-+\s*$/); + next if (m/^\s*\|\s*arch\s*\|\s*status\s*\|\s*$/); + + if (m/^\#\s*(.*)/) { + $comments .= "$1\n"; + next; + } + if (m/^\s*\|\s*(\S+):\s*\|\s*(\S+)\s*\|\s*$/) { + my $a = $1; + my $status = $2; + + if (length($status) > $max_size_status) { + $max_size_status = length($status); + } + if (length($a) > $max_size_arch) { + $max_size_arch = length($a); + } + + $archs{$a} = 1; + $arch_table{$a} = $status; + next; + } + + #Everything else is an error + parse_error($file, $ln, "line is invalid", $_); + } + close IN; + + if (!$name) { + parse_error($file, $ln, "Feature name not found", ""); + return; + } + + parse_error($file, $ln, "Subsystem not found", "") if (!$subsys); + parse_error($file, $ln, "Kconfig not found", "") if (!$kconfig); + parse_error($file, $ln, "Description not found", "") if (!$description); + + if (!%arch_table) { + parse_error($file, $ln, "Architecture table not found", ""); + return; + } + + $data{$name}->{where} = $file; + $data{$name}->{subsys} = $subsys; + $data{$name}->{kconfig} = $kconfig; + $data{$name}->{description} = $description; + $data{$name}->{comments} = $comments; + $data{$name}->{table} = \%arch_table; +} + +# +# Output feature(s) for a given architecture +# +sub output_arch_table { + my $title = "Feature status on $arch architecture"; + + print "=" x length($title) . "\n"; + print "$title\n"; + print "=" x length($title) . "\n\n"; + + print "=" x $max_size_subsys; + print " "; + print "=" x $max_size_name; + print " "; + print "=" x $max_size_kconfig; + print " "; + print "=" x $max_size_status; + print " "; + print "=" x $max_size_description; + print "\n"; + printf "%-${max_size_subsys}s ", $h_subsys; + printf "%-${max_size_name}s ", $h_name; + printf "%-${max_size_kconfig}s ", $h_kconfig; + printf "%-${max_size_status}s ", $h_status; + printf "%-${max_size_description}s\n", $h_description; + print "=" x $max_size_subsys; + print " "; + print "=" x $max_size_name; + print " "; + print "=" x $max_size_kconfig; + print " "; + print "=" x $max_size_status; + print " "; + print "=" x $max_size_description; + print "\n"; + + foreach my $name (sort { + ($data{$a}->{subsys} cmp $data{$b}->{subsys}) || + ($data{$a}->{name} cmp $data{$b}->{name}) + } keys %data) { + next if ($feat && $name ne $feat); + + my %arch_table = %{$data{$name}->{table}}; + printf "%-${max_size_subsys}s ", $data{$name}->{subsys}; + printf "%-${max_size_name}s ", $name; + printf "%-${max_size_kconfig}s ", $data{$name}->{kconfig}; + printf "%-${max_size_status}s ", $arch_table{$arch}; + printf "%-${max_size_description}s\n", $data{$name}->{description}; + } + + print "=" x $max_size_subsys; + print " "; + print "=" x $max_size_name; + print " "; + print "=" x $max_size_kconfig; + print " "; + print "=" x $max_size_status; + print " "; + print "=" x $max_size_description; + print "\n"; +} + +# +# Output a feature on all architectures +# +sub output_feature { + my $title = "Feature $feat"; + + print "=" x length($title) . "\n"; + print "$title\n"; + print "=" x length($title) . "\n\n"; + + print ":Subsystem: $data{$feat}->{subsys} \n" if ($data{$feat}->{subsys}); + print ":Kconfig: $data{$feat}->{kconfig} \n" if ($data{$feat}->{kconfig}); + + my $desc = $data{$feat}->{description}; + $desc =~ s/^([a-z])/\U$1/; + $desc =~ s/\.?\s*//; + print "\n$desc.\n\n"; + + my $com = $data{$feat}->{comments}; + $com =~ s/^\s+//; + $com =~ s/\s+$//; + if ($com) { + print "Comments\n"; + print "--------\n\n"; + print "$com\n\n"; + } + + print "=" x $max_size_arch; + print " "; + print "=" x $max_size_status; + print "\n"; + + printf "%-${max_size_arch}s ", $h_arch; + printf "%-${max_size_status}s", $h_status . "\n"; + + print "=" x $max_size_arch; + print " "; + print "=" x $max_size_status; + print "\n"; + + my %arch_table = %{$data{$feat}->{table}}; + foreach my $arch (sort keys %arch_table) { + printf "%-${max_size_arch}s ", $arch; + printf "%-${max_size_status}s\n", $arch_table{$arch}; + } + + print "=" x $max_size_arch; + print " "; + print "=" x $max_size_status; + print "\n"; +} + +# +# Output all features for all architectures +# + +sub matrix_lines { + print "=" x $max_size_subsys; + print " "; + print "=" x $max_size_name; + print " "; + + foreach my $arch (sort keys %archs) { + my $len = $max_size_status; + + $len = length($arch) if ($len < length($arch)); + + print "=" x $len; + print " "; + } + print "=" x $max_size_kconfig; + print " "; + print "=" x $max_size_description; + print "\n"; +} + +sub output_matrix { + + my $title = "Feature List (feature x architecture)"; + + print "=" x length($title) . "\n"; + print "$title\n"; + print "=" x length($title) . "\n\n"; + + matrix_lines; + + printf "%-${max_size_subsys}s ", $h_subsys; + printf "%-${max_size_name}s ", $h_name; + + foreach my $arch (sort keys %archs) { + printf "%-${max_size_status}s ", $arch; + } + printf "%-${max_size_kconfig}s ", $h_kconfig; + printf "%-${max_size_description}s\n", $h_description; + + matrix_lines; + + foreach my $name (sort { + ($data{$a}->{subsys} cmp $data{$b}->{subsys}) || + ($data{$a}->{name} cmp $data{$b}->{name}) + } keys %data) { + printf "%-${max_size_subsys}s ", $data{$name}->{subsys}; + printf "%-${max_size_name}s ", $name; + + my %arch_table = %{$data{$name}->{table}}; + + foreach my $arch (sort keys %arch_table) { + my $len = $max_size_status; + + $len = length($arch) if ($len < length($arch)); + + printf "%-${len}s ", $arch_table{$arch}; + } + printf "%-${max_size_kconfig}s ", $data{$name}->{kconfig}; + printf "%-${max_size_description}s\n", $data{$name}->{description}; + } + + matrix_lines; +} + + +# +# Parses all feature files located at $prefix dir +# +find({wanted =>\&parse_feat, no_chdir => 1}, $prefix); + +print STDERR Data::Dumper->Dump([\%data], [qw(*data)]) if ($debug); + +# +# Handles the command +# +if ($cmd eq "current") { + $arch = qx(uname -m | sed 's/x86_64/x86/' | sed 's/i386/x86/'); + $arch =~s/\s+$//; +} + +if ($cmd ne "validate") { + if ($arch) { + output_arch_table; + } elsif ($feat) { + output_feature; + } else { + output_matrix; + } +} + +__END__ + +=head1 NAME + +get_feat.pl - parse the Linux Feature files and produce a ReST book. + +=head1 SYNOPSIS + +B<get_feat.pl> [--debug] [--man] [--help] [--dir=<dir>] + [--arch=<arch>] [--feat=<feature>] <COMAND> [<ARGUMENT>] + +Where <COMMAND> can be: + +=over 8 + +B<current> - output features for this machine's architecture + +B<rest> - output features in ReST markup language + +B<validate> - validate the feature contents + +=back + +=head1 OPTIONS + +=over 8 + +=item B<--arch> + +Output features for an specific architecture, optionally filtering for +a single specific feature. + +=item B<--feat> + +Output features for a single specific architecture. + +=item B<--dir> + +Changes the location of the Feature files. By default, it uses +the Documentation/features directory. + +=item B<--debug> + +Put the script in verbose mode, useful for debugging. Can be called multiple +times, to increase verbosity. + +=item B<--help> + +Prints a brief help message and exits. + +=item B<--man> + +Prints the manual page and exits. + +=back + +=head1 DESCRIPTION + +Parse the Linux feature files from Documentation/features (by default), +optionally producing results at ReST format. + +It supports output data per architecture, per feature or a +feature x arch matrix. + +When used with B<rest> command, it will use either one of the tree formats: + +If neither B<--arch> or B<--feature> arguments are used, it will output a +matrix with features per architecture. + +If B<--arch> argument is used, it will output the features availability for +a given architecture. + +If B<--feat> argument is used, it will output the content of the feature +file using ReStructured Text markup. + +=head1 BUGS + +Report bugs to Mauro Carvalho Chehab <mchehab+samsung@kernel.org> + +=head1 COPYRIGHT + +Copyright (c) 2019 by Mauro Carvalho Chehab <mchehab+samsung@kernel.org>. + +License GPLv2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>. + +This is free software: you are free to change and redistribute it. +There is NO WARRANTY, to the extent permitted by law. + +=cut -- 2.21.0 ^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [PATCH RFC] scripts: add a script to handle Documentation/features 2019-06-17 18:05 ` [Ksummit-discuss] [PATCH RFC] scripts: add a script to handle Documentation/features Mauro Carvalho Chehab @ 2019-06-17 18:11 ` Greg Kroah-Hartman 2019-06-17 19:45 ` Mauro Carvalho Chehab 0 siblings, 1 reply; 20+ messages in thread From: Greg Kroah-Hartman @ 2019-06-17 18:11 UTC (permalink / raw) To: Mauro Carvalho Chehab Cc: ksummit-discuss, linux-kernel, Linux Doc Mailing List On Mon, Jun 17, 2019 at 03:05:07PM -0300, Mauro Carvalho Chehab wrote: > The Documentation/features contains a set of parseable files. > It is not worth converting them to ReST format, as they're > useful the way it is. It is, however, interesting to parse > them and produce output on different formats: > > 1) Output the contents of a feature in ReST format; > > 2) Output what features a given architecture supports; > > 3) Output a matrix with features x architectures. > > Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org> > --- > > As commented at KS mailing list, converting the Documentation/features > file to ReST may not be the best way to handle it. > > This script allows validating the features files and to generate ReST files > on three different formats. > > The goal is to support it via a sphinx extension, in order to be able to add > the features inside the Kernel documentation. > > scripts/get_feat.pl | 470 ++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 470 insertions(+) > create mode 100755 scripts/get_feat.pl > > diff --git a/scripts/get_feat.pl b/scripts/get_feat.pl > new file mode 100755 > index 000000000000..c5a267b12f49 > --- /dev/null > +++ b/scripts/get_feat.pl > @@ -0,0 +1,470 @@ > +#!/usr/bin/perl > + No SPDX line :( ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [PATCH RFC] scripts: add a script to handle Documentation/features 2019-06-17 18:11 ` Greg Kroah-Hartman @ 2019-06-17 19:45 ` Mauro Carvalho Chehab 0 siblings, 0 replies; 20+ messages in thread From: Mauro Carvalho Chehab @ 2019-06-17 19:45 UTC (permalink / raw) To: Greg Kroah-Hartman; +Cc: ksummit-discuss, linux-kernel, Linux Doc Mailing List Em Mon, 17 Jun 2019 20:11:16 +0200 Greg Kroah-Hartman <gregkh@linuxfoundation.org> escreveu: > On Mon, Jun 17, 2019 at 03:05:07PM -0300, Mauro Carvalho Chehab wrote: > > The Documentation/features contains a set of parseable files. > > It is not worth converting them to ReST format, as they're > > useful the way it is. It is, however, interesting to parse > > them and produce output on different formats: > > > > 1) Output the contents of a feature in ReST format; > > > > 2) Output what features a given architecture supports; > > > > 3) Output a matrix with features x architectures. > > > > Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org> > > --- > > > > As commented at KS mailing list, converting the Documentation/features > > file to ReST may not be the best way to handle it. > > > > This script allows validating the features files and to generate ReST files > > on three different formats. > > > > The goal is to support it via a sphinx extension, in order to be able to add > > the features inside the Kernel documentation. > > > > scripts/get_feat.pl | 470 ++++++++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 470 insertions(+) > > create mode 100755 scripts/get_feat.pl > > > > diff --git a/scripts/get_feat.pl b/scripts/get_feat.pl > > new file mode 100755 > > index 000000000000..c5a267b12f49 > > --- /dev/null > > +++ b/scripts/get_feat.pl > > @@ -0,0 +1,470 @@ > > +#!/usr/bin/perl > > + > > No SPDX line :( Added. I also added a Sphinx extension to handle it as well. You'll notice that it is almost a copy of kernel_abi.py. With regards to patch 2/2, it will generate both a feature x arch matrix at the admin-guide and a x86-specific features list. IMO, it makes sense to have a per-arch feature file just like the one I added to x86. As the patches converting documentation for other archs are still being merged via docs tree, before adding the features list to the other arch documents, it seems better to wait to do it after the next merge window. Those patches are applied after the ABI patches on this dir: https://git.linuxtv.org/mchehab/experimental.git/log/?h=abi_patches_v4.1 The output with both scripts are at: https://www.infradead.org/~mchehab/rst_features/index.html The relevant parts are: ABI: https://www.infradead.org/~mchehab/rst_features/admin-guide/abi.html Feature list x architecture (at admin-guide: https://www.infradead.org/~mchehab/rst_features/admin-guide/features.html X86 features: https://www.infradead.org/~mchehab/rst_features/x86/features.html While I didn't write a patch, with the new get_feat.pl script, we can probably get rid of the previous shell script at: Documentation/features/list-arch.sh As calling: ./scripts/get_feat.pl current Will output the same content (with a different format, though). Thanks, Mauro ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] Documentation 2019-06-12 14:54 [Ksummit-discuss] [TECH TOPIC] Documentation Jonathan Corbet 2019-06-12 18:22 ` Shuah Khan @ 2019-06-12 20:33 ` Kate Stewart 2019-06-13 14:17 ` Mauro Carvalho Chehab 2019-06-13 14:57 ` Mauro Carvalho Chehab ` (2 subsequent siblings) 4 siblings, 1 reply; 20+ messages in thread From: Kate Stewart @ 2019-06-12 20:33 UTC (permalink / raw) To: Jonathan Corbet; +Cc: ksummit-discuss On Wed, Jun 12, 2019 at 9:56 AM Jonathan Corbet <corbet@lwn.net> wrote: > > What could be more fun than talking about kernel documentation? Things we > could get into: > > - The state of the RST transition, what remains to be done, whether it's > all just useless churn that makes the documentation worse, etc. > > - Things we'd like to improve in the documentation toolchain. > > - Overall organization of Documentation/ and moving docs when the need > arises. It seems I end up fighting about this more than just about > anything else, but I think it's important to organize our docs for the > convenience of the people using them. > > - The ultimate vision for kernel docs (for now). RST conversion and > imposing some organization are important, but they will not, > themselves, give us a coherent set of documentation. What can we do to > have documentation that is useful, current, and maintainable, rather > than the dusty attic we have now? Also, it would be great if we could talk about cleaning up the documentation licensing, so it too can have its licenses be automatically detected. :-) Kate ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] Documentation 2019-06-12 20:33 ` [Ksummit-discuss] [TECH TOPIC] Documentation Kate Stewart @ 2019-06-13 14:17 ` Mauro Carvalho Chehab 0 siblings, 0 replies; 20+ messages in thread From: Mauro Carvalho Chehab @ 2019-06-13 14:17 UTC (permalink / raw) To: Kate Stewart; +Cc: ksummit-discuss Em Wed, 12 Jun 2019 15:33:36 -0500 Kate Stewart <kstewart@linuxfoundation.org> escreveu: > On Wed, Jun 12, 2019 at 9:56 AM Jonathan Corbet <corbet@lwn.net> wrote: > > > > What could be more fun than talking about kernel documentation? Things we > > could get into: > > > > - The state of the RST transition, what remains to be done, whether it's > > all just useless churn that makes the documentation worse, etc. > > > > - Things we'd like to improve in the documentation toolchain. > > > > - Overall organization of Documentation/ and moving docs when the need > > arises. It seems I end up fighting about this more than just about > > anything else, but I think it's important to organize our docs for the > > convenience of the people using them. > > > > - The ultimate vision for kernel docs (for now). RST conversion and > > imposing some organization are important, but they will not, > > themselves, give us a coherent set of documentation. What can we do to > > have documentation that is useful, current, and maintainable, rather > > than the dusty attic we have now? > > Also, it would be great if we could talk about cleaning up the > documentation licensing, so it too can have its licenses be > automatically detected. :-) With that regards, I'm still waiting for a solution with for the GFDL license: https://github.com/spdx/license-list-XML/issues/686 All documents under Documentation/media now have a SPDX header, except for the ones that were originally licensed under the free version of GNU Free Document License (e. g. with no invariant sections, no Front-Cover texts and no Back-Cover texts) or that are dual-licensed GFDL/GPL. While we don't have a SPDX tag for those, we can't finish adding SPDX headers there though :-( Thanks, Mauro ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] Documentation 2019-06-12 14:54 [Ksummit-discuss] [TECH TOPIC] Documentation Jonathan Corbet 2019-06-12 18:22 ` Shuah Khan 2019-06-12 20:33 ` [Ksummit-discuss] [TECH TOPIC] Documentation Kate Stewart @ 2019-06-13 14:57 ` Mauro Carvalho Chehab 2019-06-13 18:48 ` Greg KH 2019-06-20 18:36 ` Kees Cook 2019-07-22 14:52 ` Mauro Carvalho Chehab 4 siblings, 1 reply; 20+ messages in thread From: Mauro Carvalho Chehab @ 2019-06-13 14:57 UTC (permalink / raw) To: Jonathan Corbet; +Cc: ksummit-discuss Em Wed, 12 Jun 2019 08:54:05 -0600 Jonathan Corbet <corbet@lwn.net> escreveu: > What could be more fun than talking about kernel documentation? That sounds a topic that we should have some discussion. IMO, the best would be to put people on some round tables and exchange some ideas about how we can improve the process. I would try to do it as a half-day topic, if the agenda would allow, as a standard 50' min slot doesn't seem to be enough to address the needed things there. > Things we > could get into: > > - The state of the RST transition, what remains to be done, whether it's > all just useless churn that makes the documentation worse, etc. I have a large changeset (with about one patch per Documentation/ dir) with manually written patches with addresses a large amount of the remaining stuff. I'm hoping to have most of it applied before KS :-) There will still be 4 or 5 directories with lots of documentation files on it, plus ABI and DT. > - Things we'd like to improve in the documentation toolchain. I remember once I submitted a patch with a script capable of parsing the files at ABI and produce a parsed content, with the goal of validating the ABI files against its syntax and to generate an ABI book. IMO, it makes sense to have one book containing the Linux ABI, but, as the patch didn't have much attention, and I got sidetracked with something else, I ended giving up on trying to push it. Perhaps by KS it would be time to rescue it from some old git tree and see if it would be worth the time to work on it. > > - Overall organization of Documentation/ and moving docs when the need > arises. It seems I end up fighting about this more than just about > anything else, but I think it's important to organize our docs for the > convenience of the people using them. Fully agreed. It is not always clear where to place some converted docs. Also, in the case of driver-specific information, it is not unusual to have both kernel and user's information at the same file. Ideally, the best would be to split, but I'm not sure if it would worth the time for doing such changes. > - The ultimate vision for kernel docs (for now). RST conversion and > imposing some organization are important, but they will not, > themselves, give us a coherent set of documentation. What can we do to > have documentation that is useful, current, and maintainable, rather > than the dusty attic we have now? Yeah, RST conversion and enforcing an organization of them is important, as it will enforce a single "coding" style for the documents, but this is only a small part of the issue. IMO, we should work to attract people focused on maintaining docs. I've been doing some userspace maintainership on a few media-related projects. Among them, I'm maintaining a KDE media player (Kaffeine). The KDE project has a group of people that it is focused only on writing documentation, and one group of people per translation language. IMO, if we want to have a good set of documents, we should have a group of people working mainly with a "documentation hat", reviewing, improving and rewriting large chunks of documents. If we ever have such kind of people working around the Kernel, I guess the first task would be to imagine what kind of things a new Kernel developer needs to know and try to structure a coherent documentation from that. Such group could use the existing Kernel printed books as a starting point (perhaps using existing books documentation that their authors could release them under GPLv2), and modifying the texts to use what we have nowadays inside the Kernel's tree. I guess the main problem is that, for this to work, someone (possibly a non-profit org) would likely need to sponsor such work, as this would probably require more time than we could do on our spare time. Thanks, Mauro ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] Documentation 2019-06-13 14:57 ` Mauro Carvalho Chehab @ 2019-06-13 18:48 ` Greg KH 2019-06-13 19:01 ` Mauro Carvalho Chehab 0 siblings, 1 reply; 20+ messages in thread From: Greg KH @ 2019-06-13 18:48 UTC (permalink / raw) To: Mauro Carvalho Chehab; +Cc: ksummit-discuss On Thu, Jun 13, 2019 at 11:57:12AM -0300, Mauro Carvalho Chehab wrote: > > - Things we'd like to improve in the documentation toolchain. > > I remember once I submitted a patch with a script capable of > parsing the files at ABI and produce a parsed content, with the > goal of validating the ABI files against its syntax and to > generate an ABI book. > > IMO, it makes sense to have one book containing the Linux ABI, but, > as the patch didn't have much attention, and I got sidetracked with > something else, I ended giving up on trying to push it. > > Perhaps by KS it would be time to rescue it from some old git tree > and see if it would be worth the time to work on it. I liked that patcheset. If you can dig it up, I can work on reviving it if you don't want to. thanks, greg k-h ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] Documentation 2019-06-13 18:48 ` Greg KH @ 2019-06-13 19:01 ` Mauro Carvalho Chehab 0 siblings, 0 replies; 20+ messages in thread From: Mauro Carvalho Chehab @ 2019-06-13 19:01 UTC (permalink / raw) To: Greg KH; +Cc: ksummit-discuss Em Thu, 13 Jun 2019 20:48:08 +0200 Greg KH <greg@kroah.com> escreveu: > On Thu, Jun 13, 2019 at 11:57:12AM -0300, Mauro Carvalho Chehab wrote: > > > - Things we'd like to improve in the documentation toolchain. > > > > I remember once I submitted a patch with a script capable of > > parsing the files at ABI and produce a parsed content, with the > > goal of validating the ABI files against its syntax and to > > generate an ABI book. > > > > IMO, it makes sense to have one book containing the Linux ABI, but, > > as the patch didn't have much attention, and I got sidetracked with > > something else, I ended giving up on trying to push it. > > > > Perhaps by KS it would be time to rescue it from some old git tree > > and see if it would be worth the time to work on it. > > I liked that patcheset. If you can dig it up, I can work on reviving it > if you don't want to. The patches are here: https://git.linuxtv.org/mchehab/experimental-old.git/commit/?h=abi_docs_v3&id=da4b94205aced79df7389ca6f2ec4328fb8604a7 I'll rebase them on the top of linux-next and re-submit. Thanks, Mauro ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] Documentation 2019-06-12 14:54 [Ksummit-discuss] [TECH TOPIC] Documentation Jonathan Corbet ` (2 preceding siblings ...) 2019-06-13 14:57 ` Mauro Carvalho Chehab @ 2019-06-20 18:36 ` Kees Cook 2019-06-20 19:28 ` Jonathan Corbet 2019-07-22 14:52 ` Mauro Carvalho Chehab 4 siblings, 1 reply; 20+ messages in thread From: Kees Cook @ 2019-06-20 18:36 UTC (permalink / raw) To: Jonathan Corbet; +Cc: ksummit-discuss On Wed, Jun 12, 2019 at 08:54:05AM -0600, Jonathan Corbet wrote: > - Things we'd like to improve in the documentation toolchain. Just to put it into this thread... I've seen a recurring request for markup improvement to have Sphinx be smart enough to linkify: function() instead of needing to do: :c:func:`iounmap()` And similarly for: struct namegoeshere -- Kees Cook ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] Documentation 2019-06-20 18:36 ` Kees Cook @ 2019-06-20 19:28 ` Jonathan Corbet 0 siblings, 0 replies; 20+ messages in thread From: Jonathan Corbet @ 2019-06-20 19:28 UTC (permalink / raw) To: Kees Cook; +Cc: ksummit-discuss On Thu, 20 Jun 2019 11:36:38 -0700 Kees Cook <keescook@chromium.org> wrote: > Just to put it into this thread... I've seen a recurring request for > markup improvement to have Sphinx be smart enough to linkify: > > function() > > instead of needing to do: > > :c:func:`iounmap()` I already have a patch to do that: https://lwn.net/ml/linux-doc/20190425200125.12302-1-corbet@lwn.net/ It needs another bit of attention before it's ready, and I've not yet managed to do that. Soon, I promise :) Thanks, jon ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Ksummit-discuss] [TECH TOPIC] Documentation 2019-06-12 14:54 [Ksummit-discuss] [TECH TOPIC] Documentation Jonathan Corbet ` (3 preceding siblings ...) 2019-06-20 18:36 ` Kees Cook @ 2019-07-22 14:52 ` Mauro Carvalho Chehab 4 siblings, 0 replies; 20+ messages in thread From: Mauro Carvalho Chehab @ 2019-07-22 14:52 UTC (permalink / raw) To: Jonathan Corbet; +Cc: ksummit-discuss Em Wed, 12 Jun 2019 08:54:05 -0600 Jonathan Corbet <corbet@lwn.net> escreveu: > What could be more fun than talking about kernel documentation? Things we > could get into: > > - The state of the RST transition Assuming that all patches on my development branch gets merged, we should have: - about 300 files missing conversion; - about 40 files under Documentation/*.txt that needs to be moved to a subdir and be renamed to *.rst. > - Overall organization of Documentation/ and moving docs when the need > arises. Looking at Documentation/index.rst, I guess we're aiming to have something like: Documentation/ ├── ABI ├── admin-guide ├── kbuild ├── arch │ ├── arm │ ├── arm64 │ ... │ ├── x86 │ │ ├── i386 │ │ └── x86_64 │ └── xtensa ├── core-api ├── devicetree ├── dev-tools ├── doc-guide ├── driver-api ├── features ├── fault-injection ├── filesystems ├── firmware-guide ├── kernel-hacking ├── livepatch ├── maintainer ├── process ├── trace └── translations Btw, right now, ext4 fs docs is on two separate parts of Documentation/index.rst: filesystems/index filesystems/ext4/index We should probably get rid of filesystems/ext4/index entry (and the corresponding PDF entry at conf.py). - IMO, the main work to be done in order to achieve that is related to Driver's subsystem documentation. What I've been doing so far - at least for most (if not all) driver docs that carry more than one documentation type at the same subdir (kABI, uABI and/or admin-guide) is to keep the directory as-is, adding them under this section at Documentation/index.rst: Kernel API documentation ------------------------ There are a couple of reasons why I opted for this strategy when I did such conversions: 1) There are *lots* of docs that contain all 3 types of information on it on a single file. 2) On media, we use SPHINXDIRS to produce the media book from our devel tree: https://linuxtv.org/downloads/v4l-dvb-apis-new/index.html When documents are built with either PDF or SPHINXDIRS, each subdir will be on a different book and all inter-book cross-references will break. For this to be fixed, intersphinx extension would be required, but this would probably require a per-subsystem mapping (for example, saying that the site used to resolve media broken cross references is linuxtv.org). Maintaining it can be painful, as we would have a big table at conf.py with subsystem-specific stuff. 3) So far, I was unable to split even the media docs. Shame on me! The reason is that this is not an easy task. One interesting example is the open() documentation at the media media uAPI book: Documentation/media/uapi/v4l/open.rst This file actually contains a lot of sysadmin relevant data (so, it is a good candidate for the admin-guide). Yet, it was written focused on what a media uAPI developer needs to know. So, it also mentions Kernel userspace API syscalls: open(), read(), close() - with has cross-references to other parts of the uAPI book. Splitting this file on two separate books won't be that easy. Ideally, we should split what's there at media/uapi into admin-guide and userspace-api, but this would mean *a lot* of effort. Not sure if it is worth the time. Also, while a sysadmin might want to know what a /dev/video0 device means, the intended audience is really uAPI developers, as an user will just click on its GUI to call a media application. > - The ultimate vision for kernel docs (for now). RST conversion and > imposing some organization are important, but they will not, > themselves, give us a coherent set of documentation. What can we do to > have documentation that is useful, current, and maintainable, rather > than the dusty attic we have now? The more I think the more I'm convinced that the best way to proceed would be to use some Kernel books as an example and organize the main index.rst files on a way that it will cover what a newbie Kernel developer would need to know. For example, looking at LDD3 organization (https://lwn.net/Kernel/LDD3/), we could use its index as an starting point for what a driver-api book should contain: Chapter 1: An Introduction to Device Drivers Chapter 3: Char Drivers Chapter 6: Advanced Char Driver Operations Chapter 9: Communicating with Hardware Chapter 12: PCI Drivers Chapter 13: USB Drivers Chapter 14: The Linux Device Model Chapter 16: Block Drivers Chapter 17: Network Drivers Chapter 18: TTY Drivers Just looking on these, our driver-api book seem to be missing the texts that would glue its contents, e. g. an introduction to device drivers, to char/block devices and about how to communicate with the hardware. It can also help to identify the contents that a driver developer would need from a core-api and a Kernel development bookset: Chapter 2: Building and Running Modules Chapter 4: Debugging Techniques Chapter 5: Concurrency and Race Conditions Chapter 7: Time, Delays, and Deferred Work Chapter 8: Allocating Memory Chapter 10: Interrupt Handling Chapter 11: Data Types in the Kernel I would do the same with other Linux and Linux Kernel related books. - Btw, if the authors of some existing old books release their stuff under GPLv2 and allow us to import their contents, we could try to import some parts of it that aren't too obsolete. The new automarkup extension can help a lot to identify outdated documents, as it won't be able to solve the func() calls. Thanks, Mauro ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2019-07-22 14:53 UTC | newest] Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-06-12 14:54 [Ksummit-discuss] [TECH TOPIC] Documentation Jonathan Corbet 2019-06-12 18:22 ` Shuah Khan 2019-06-12 19:12 ` Martin K. Petersen 2019-06-12 19:43 ` Shuah Khan 2019-06-13 14:25 ` Mauro Carvalho Chehab 2019-06-14 21:40 ` Shuah Khan 2019-06-15 0:05 ` Mauro Carvalho Chehab 2019-06-17 10:12 ` Mauro Carvalho Chehab 2019-06-17 17:21 ` Mauro Carvalho Chehab 2019-06-17 18:05 ` [Ksummit-discuss] [PATCH RFC] scripts: add a script to handle Documentation/features Mauro Carvalho Chehab 2019-06-17 18:11 ` Greg Kroah-Hartman 2019-06-17 19:45 ` Mauro Carvalho Chehab 2019-06-12 20:33 ` [Ksummit-discuss] [TECH TOPIC] Documentation Kate Stewart 2019-06-13 14:17 ` Mauro Carvalho Chehab 2019-06-13 14:57 ` Mauro Carvalho Chehab 2019-06-13 18:48 ` Greg KH 2019-06-13 19:01 ` Mauro Carvalho Chehab 2019-06-20 18:36 ` Kees Cook 2019-06-20 19:28 ` Jonathan Corbet 2019-07-22 14:52 ` Mauro Carvalho Chehab
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).