[PATCH V2] get_maintainer: Prepare for separate MAINTAINERS files

[PATCH V2] get_maintainer: Prepare for separate MAINTAINERS files

[Joe Perches]

Allow for MAINTAINERS to become a directory and if it is,
read all the files in the directory for maintained sections.

Also look for all files named MAINTAINERS in all subdirectories
excluding the .git directory.

This adds ~.3 seconds of CPU on an Intel i5-6200 with an SSD.

Miscellanea:

o Create a read_maintainer_file subroutine from the existing code
o Test only the existence of MAINTAINERS, not whether it's a file

Signed-off-by: Joe Perches <joe@perches.com>
---

v2: Search for MAINTAINERS in subdirectories, ignoring .git

 scripts/get_maintainer.pl | 84 +++++++++++++++++++++++++++++++++--------------
 1 file changed, 59 insertions(+), 25 deletions(-)

diff --git a/scripts/get_maintainer.pl b/scripts/get_maintainer.pl
index 3bd5f4f30235..56d5d3ef4c81 100755
--- a/scripts/get_maintainer.pl
+++ b/scripts/get_maintainer.pl
@@ -18,6 +18,7 @@ my $V = '0.26';
 
 use Getopt::Long qw(:config no_auto_abbrev);
 use Cwd;
+use File::Find;
 
 my $cur_path = fastgetcwd() . '/';
 my $lk_path = "./";
@@ -307,36 +308,69 @@ if (!top_of_kernel_tree($lk_path)) {
 
 my @typevalue = ();
 my %keyword_hash;
+my @mfiles = ();
 
-open (my $maint, '<', "${lk_path}MAINTAINERS")
-    or die "$P: Can't open MAINTAINERS: $!\n";
-while (<$maint>) {
-    my $line = $_;
-
-    if ($line =~ m/^([A-Z]):\s*(.*)/) {
-	my $type = $1;
-	my $value = $2;
-
-	##Filename pattern matching
-	if ($type eq "F" || $type eq "X") {
-	    $value =~ s@\.@\\\.@g;       ##Convert . to \.
-	    $value =~ s/\*/\.\*/g;       ##Convert * to .*
-	    $value =~ s/\?/\./g;         ##Convert ? to .
-	    ##if pattern is a directory and it lacks a trailing slash, add one
-	    if ((-d $value)) {
-		$value =~ s@([^/])$@$1/@;
+sub read_maintainer_file {
+    my ($file) = @_;
+
+    open (my $maint, '<', "$file")
+	or die "$P: Can't open MAINTAINERS file '$file': $!\n";
+    while (<$maint>) {
+	my $line = $_;
+
+	if ($line =~ m/^([A-Z]):\s*(.*)/) {
+	    my $type = $1;
+	    my $value = $2;
+
+	    ##Filename pattern matching
+	    if ($type eq "F" || $type eq "X") {
+		$value =~ s@\.@\\\.@g;       ##Convert . to \.
+		$value =~ s/\*/\.\*/g;       ##Convert * to .*
+		$value =~ s/\?/\./g;         ##Convert ? to .
+		##if pattern is a directory and it lacks a trailing slash, add one
+		if ((-d $value)) {
+		    $value =~ s@([^/])$@$1/@;
+		}
+	    } elsif ($type eq "K") {
+		$keyword_hash{@typevalue} = $value;
 	    }
-	} elsif ($type eq "K") {
-	    $keyword_hash{@typevalue} = $value;
+	    push(@typevalue, "$type:$value");
+	} elsif (!(/^\s*$/ || /^\s*\#/)) {
+	    $line =~ s/\n$//g;
+	    push(@typevalue, $line);
 	}
-	push(@typevalue, "$type:$value");
-    } elsif (!/^(\s)*$/) {
-	$line =~ s/\n$//g;
-	push(@typevalue, $line);
+    }
+    close($maint);
+}
+
+sub find_is_maintainer_file {
+    my $file = $File::Find::name;
+    if (-f $file && $file =~ m@/MAINTAINERS$@) {
+	push(@mfiles, $file);
+    }
+}
+
+sub find_ignore_git {
+    return grep { $_ !~ /^\.git$/; } @_;
+}
+
+if (-d "${lk_path}MAINTAINERS") {
+    opendir(DIR, "${lk_path}MAINTAINERS") or die $!;
+    my @files = readdir(DIR);
+    closedir(DIR);
+    foreach my $file (@files) {
+	push(@mfiles, "${lk_path}MAINTAINERS/$file") if ($file !~ /^\./);
     }
 }
-close($maint);
 
+find( { wanted => \&find_is_maintainer_file,
+	preprocess => \&find_ignore_git,
+	no_chdir => 1,
+      }, "${lk_path}");
+
+foreach my $file (@mfiles) {
+    read_maintainer_file("$file");
+}
 
 #
 # Read mail address map
@@ -873,7 +907,7 @@ sub top_of_kernel_tree {
     if (   (-f "${lk_path}COPYING")
 	&& (-f "${lk_path}CREDITS")
 	&& (-f "${lk_path}Kbuild")
-	&& (-f "${lk_path}MAINTAINERS")
+	&& (-e "${lk_path}MAINTAINERS")
 	&& (-f "${lk_path}Makefile")
 	&& (-f "${lk_path}README")
 	&& (-d "${lk_path}Documentation")
-- 
2.10.0.rc2.1.g053435c

Re: [PATCH V2] get_maintainer: Prepare for separate MAINTAINERS files

[Joe Perches]

On Sun, 2017-07-23 at 13:32 -0700, Joe Perches wrote:
> Allow for MAINTAINERS to become a directory and if it is,
> read all the files in the directory for maintained sections.

ping?

> Also look for all files named MAINTAINERS in all subdirectories
> excluding the .git directory.
> 
> This adds ~.3 seconds of CPU on an Intel i5-6200 with an SSD.
> 
> Miscellanea:
> 
> o Create a read_maintainer_file subroutine from the existing code
> o Test only the existence of MAINTAINERS, not whether it's a file
> 
> Signed-off-by: Joe Perches <joe@perches.com>
> ---
> 
> v2: Search for MAINTAINERS in subdirectories, ignoring .git
> 
>  scripts/get_maintainer.pl | 84 +++++++++++++++++++++++++++++++++--------------
>  1 file changed, 59 insertions(+), 25 deletions(-)
> 
> diff --git a/scripts/get_maintainer.pl b/scripts/get_maintainer.pl
> index 3bd5f4f30235..56d5d3ef4c81 100755
> --- a/scripts/get_maintainer.pl
> +++ b/scripts/get_maintainer.pl
> @@ -18,6 +18,7 @@ my $V = '0.26';
>  
>  use Getopt::Long qw(:config no_auto_abbrev);
>  use Cwd;
> +use File::Find;
>  
>  my $cur_path = fastgetcwd() . '/';
>  my $lk_path = "./";
> @@ -307,36 +308,69 @@ if (!top_of_kernel_tree($lk_path)) {
>  
>  my @typevalue = ();
>  my %keyword_hash;
> +my @mfiles = ();
>  
> -open (my $maint, '<', "${lk_path}MAINTAINERS")
> -    or die "$P: Can't open MAINTAINERS: $!\n";
> -while (<$maint>) {
> -    my $line = $_;
> -
> -    if ($line =~ m/^([A-Z]):\s*(.*)/) {
> -	my $type = $1;
> -	my $value = $2;
> -
> -	##Filename pattern matching
> -	if ($type eq "F" || $type eq "X") {
> -	    $value =~ s@\.@\\\.@g;       ##Convert . to \.
> -	    $value =~ s/\*/\.\*/g;       ##Convert * to .*
> -	    $value =~ s/\?/\./g;         ##Convert ? to .
> -	    ##if pattern is a directory and it lacks a trailing slash, add one
> -	    if ((-d $value)) {
> -		$value =~ s@([^/])$@$1/@;
> +sub read_maintainer_file {
> +    my ($file) = @_;
> +
> +    open (my $maint, '<', "$file")
> +	or die "$P: Can't open MAINTAINERS file '$file': $!\n";
> +    while (<$maint>) {
> +	my $line = $_;
> +
> +	if ($line =~ m/^([A-Z]):\s*(.*)/) {
> +	    my $type = $1;
> +	    my $value = $2;
> +
> +	    ##Filename pattern matching
> +	    if ($type eq "F" || $type eq "X") {
> +		$value =~ s@\.@\\\.@g;       ##Convert . to \.
> +		$value =~ s/\*/\.\*/g;       ##Convert * to .*
> +		$value =~ s/\?/\./g;         ##Convert ? to .
> +		##if pattern is a directory and it lacks a trailing slash, add one
> +		if ((-d $value)) {
> +		    $value =~ s@([^/])$@$1/@;
> +		}
> +	    } elsif ($type eq "K") {
> +		$keyword_hash{@typevalue} = $value;
>  	    }
> -	} elsif ($type eq "K") {
> -	    $keyword_hash{@typevalue} = $value;
> +	    push(@typevalue, "$type:$value");
> +	} elsif (!(/^\s*$/ || /^\s*\#/)) {
> +	    $line =~ s/\n$//g;
> +	    push(@typevalue, $line);
>  	}
> -	push(@typevalue, "$type:$value");
> -    } elsif (!/^(\s)*$/) {
> -	$line =~ s/\n$//g;
> -	push(@typevalue, $line);
> +    }
> +    close($maint);
> +}
> +
> +sub find_is_maintainer_file {
> +    my $file = $File::Find::name;
> +    if (-f $file && $file =~ m@/MAINTAINERS$@) {
> +	push(@mfiles, $file);
> +    }
> +}
> +
> +sub find_ignore_git {
> +    return grep { $_ !~ /^\.git$/; } @_;
> +}
> +
> +if (-d "${lk_path}MAINTAINERS") {
> +    opendir(DIR, "${lk_path}MAINTAINERS") or die $!;
> +    my @files = readdir(DIR);
> +    closedir(DIR);
> +    foreach my $file (@files) {
> +	push(@mfiles, "${lk_path}MAINTAINERS/$file") if ($file !~ /^\./);
>      }
>  }
> -close($maint);
>  
> +find( { wanted => \&find_is_maintainer_file,
> +	preprocess => \&find_ignore_git,
> +	no_chdir => 1,
> +      }, "${lk_path}");
> +
> +foreach my $file (@mfiles) {
> +    read_maintainer_file("$file");
> +}
>  
>  #
>  # Read mail address map
> @@ -873,7 +907,7 @@ sub top_of_kernel_tree {
>      if (   (-f "${lk_path}COPYING")
>  	&& (-f "${lk_path}CREDITS")
>  	&& (-f "${lk_path}Kbuild")
> -	&& (-f "${lk_path}MAINTAINERS")
> +	&& (-e "${lk_path}MAINTAINERS")
>  	&& (-f "${lk_path}Makefile")
>  	&& (-f "${lk_path}README")
>  	&& (-d "${lk_path}Documentation")

Re: [PATCH V2] get_maintainer: Prepare for separate MAINTAINERS files

[Randy Dunlap]

On 08/02/2017 01:04 AM, Joe Perches wrote:
> On Sun, 2017-07-23 at 13:32 -0700, Joe Perches wrote:
>> Allow for MAINTAINERS to become a directory and if it is,
>> read all the files in the directory for maintained sections.
> 
> ping?

I often use $EDITOR to read or search the MAINTAINERS file.

IMO, the parse-maintainters.pl (sorting) script makes the need for separate
MAINTAINERS files much less important since the file can be "fixed" easily
at any time.


>> Also look for all files named MAINTAINERS in all subdirectories
>> excluding the .git directory.
>>
>> This adds ~.3 seconds of CPU on an Intel i5-6200 with an SSD.
>>
>> Miscellanea:
>>
>> o Create a read_maintainer_file subroutine from the existing code
>> o Test only the existence of MAINTAINERS, not whether it's a file
>>
>> Signed-off-by: Joe Perches <joe@perches.com>
>> ---
>>
>> v2: Search for MAINTAINERS in subdirectories, ignoring .git
>>
>>  scripts/get_maintainer.pl | 84 +++++++++++++++++++++++++++++++++--------------
>>  1 file changed, 59 insertions(+), 25 deletions(-)


-- 
~Randy

Re: [PATCH V2] get_maintainer: Prepare for separate MAINTAINERS files

[Linus Torvalds]

On Wed, Aug 2, 2017 at 11:06 AM, Randy Dunlap <rdunlap@infradead.org> wrote:
>
> IMO, the parse-maintainters.pl (sorting) script makes the need for separate
> MAINTAINERS files much less important since the file can be "fixed" easily
> at any time.

For me it's not the "fixing". It's the inevitable merge mess, and the
two hundred commits that I have to go through.

That said, the extra time just to look for MAINTAINERS files makes me
unhappy. It may be just .3s on Joe's machine, but it's presumably much
more when things aren't in the filesystem caches. I (like apparently
Joe) have an SSD so it's not a big deal for me, but..

Just having a single MAINTAINERS directory would alleviate that concern.

             Linus

Re: [PATCH V2] get_maintainer: Prepare for separate MAINTAINERS files

[Joe Perches]

On Wed, 2017-08-02 at 11:15 -0700, Linus Torvalds wrote:
> On Wed, Aug 2, 2017 at 11:06 AM, Randy Dunlap <rdunlap@infradead.org> wrote:
> > 
> > IMO, the parse-maintainters.pl (sorting) script makes the need for separate
> > MAINTAINERS files much less important since the file can be "fixed" easily
> > at any time.
> 
> For me it's not the "fixing". It's the inevitable merge mess, and the
> two hundred commits that I have to go through.
> 
> That said, the extra time just to look for MAINTAINERS files makes me
> unhappy. It may be just .3s on Joe's machine, but it's presumably much
> more when things aren't in the filesystem caches. I (like apparently
> Joe) have an SSD so it's not a big deal for me, but..
> 
> Just having a single MAINTAINERS directory would alleviate that concern.

Yeah.  Quite a bit.  With an i5-6200 and an SSD:

Looking for all MAINTAINERS files:

Cold cache find:

$ time find . -not \( -path ./.git -prune \) -name MAINTAINERS
./drivers/staging/unisys/MAINTAINERS
./MAINTAINERS

real	0m3.129s
user	0m0.368s
sys	0m0.828s

Warm cache find:

$ time find . -not \( -path ./.git -prune \) -name MAINTAINERS
./drivers/staging/unisys/MAINTAINERS
./MAINTAINERS

real	0m0.151s
user	0m0.084s
sys	0m0.064s

vs looking for all files in MAINTAINERS directory

Cold cache:

$ time find MAINTAINERS.tmp/ | wc -l
1736

real	0m0.037s
user	0m0.016s
sys	0m0.012s

Warm cache:

$ time find MAINTAINERS.tmp/ | wc -l
1736

real	0m0.017s
user	0m0.012s
sys	0m0.008s

Re: [PATCH V2] get_maintainer: Prepare for separate MAINTAINERS files

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1589 bytes --]

Hi!

On Wed 2017-08-02 11:15:09, Linus Torvalds wrote:
> On Wed, Aug 2, 2017 at 11:06 AM, Randy Dunlap <rdunlap@infradead.org> wrote:
> >
> > IMO, the parse-maintainters.pl (sorting) script makes the need for separate
> > MAINTAINERS files much less important since the file can be "fixed" easily
> > at any time.
> 
> For me it's not the "fixing". It's the inevitable merge mess, and the
> two hundred commits that I have to go through.
> 
> That said, the extra time just to look for MAINTAINERS files makes me
> unhappy. It may be just .3s on Joe's machine, but it's presumably much
> more when things aren't in the filesystem caches. I (like apparently
> Joe) have an SSD so it's not a big deal for me, but..
> 
> Just having a single MAINTAINERS directory would alleviate that
>concern.

Well, I am one of those slow-spinning-rust users. (I do have SSD here,
but bcache is not exactly easy to configure with already-existing
setup).

Using git is already pretty painful... but I believe having
net/MAINTAINERS file which clearly tells you who maintains this
directory would save time even for me. Grepping MAINTAINERS is not
currently very easy ("is it NET subsystem or NETWORK subsystem?", is
it listed as "ALSA" or "ADVANCED LINUX SOUND..."?) and splitting it to
directories would help a lot.

Having single directory with all the MAINTAINERS files would be even
worse than current situation..
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [PATCH V2] get_maintainer: Prepare for separate MAINTAINERS files

[Joe Perches]

On Tue, 2017-08-15 at 00:09 +0200, Pavel Machek wrote:
> Hi!
> 
> On Wed 2017-08-02 11:15:09, Linus Torvalds wrote:
> > On Wed, Aug 2, 2017 at 11:06 AM, Randy Dunlap <rdunlap@infradead.org> wrote:
> > > 
> > > IMO, the parse-maintainters.pl (sorting) script makes the need for separate
> > > MAINTAINERS files much less important since the file can be "fixed" easily
> > > at any time.
> > 
> > For me it's not the "fixing". It's the inevitable merge mess, and the
> > two hundred commits that I have to go through.
> > 
> > That said, the extra time just to look for MAINTAINERS files makes me
> > unhappy. It may be just .3s on Joe's machine, but it's presumably much
> > more when things aren't in the filesystem caches. I (like apparently
> > Joe) have an SSD so it's not a big deal for me, but..
> > 
> > Just having a single MAINTAINERS directory would alleviate that
> > concern.
> 
> Well, I am one of those slow-spinning-rust users. (I do have SSD here,
> but bcache is not exactly easy to configure with already-existing
> setup).
> 
> Using git is already pretty painful... but I believe having
> net/MAINTAINERS file which clearly tells you who maintains this
> directory would save time even for me. Grepping MAINTAINERS is not
> currently very easy ("is it NET subsystem or NETWORK subsystem?", is
> it listed as "ALSA" or "ADVANCED LINUX SOUND..."?) and splitting it to
> directories would help a lot.
> 
> Having single directory with all the MAINTAINERS files would be even
> worse than current situation..

What does not work well about scripts/get_maintainer.pl ?