[PATCH v5] checkpatch: add new exception to repeated word check

[PATCH v5] checkpatch: add new exception to repeated word check

[Dwaipayan Ray]

Recently, commit 4f6ad8aa1eac ("checkpatch: move repeated word test")
moved the repeated word test to check for more file types. But after
this, if checkpatch.pl is run on MAINTAINERS, it generates several
new warnings of the type:

WARNING: Possible repeated word: 'git'

For example:
WARNING: Possible repeated word: 'git'
+T:	git git://git.kernel.org/pub/scm/linux/kernel/git/rw/uml.git

So, the pattern "git git://..." is a false positive in this case.

There are several other combinations which may produce a wrong
warning message, such as "@size size", ":Begin begin", etc.

Extend repeated word check to compare the characters before and
after the word matches.

If there is a non whitespace character before the first word or a
non whitespace character excluding punctuation characters after
the second word, then the check is skipped and the warning is avoided.

Also add case insensitive word matching to the repeated word check.

Link: https://lore.kernel.org/linux-kernel-mentees/81b6a0bb2c7b9256361573f7a13201ebcd4876f1.camel@perches.com/
Suggested-by: Joe Perches <joe@perches.com>
Suggested-by: Lukas Bulwahn <lukas.bulwahn@gmail.com>
Signed-off-by: Dwaipayan Ray <dwaipayanray1@gmail.com>
---
 scripts/checkpatch.pl | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl
index f1a4e61917eb..ec380626bebc 100755
--- a/scripts/checkpatch.pl
+++ b/scripts/checkpatch.pl
@@ -3052,19 +3052,30 @@ sub process {
 
 # check for repeated words separated by a single space
 		if ($rawline =~ /^\+/ || $in_commit_log) {
+			pos($rawline) = 1 if (!$in_commit_log);
 			while ($rawline =~ /\b($word_pattern) (?=($word_pattern))/g) {
 
 				my $first = $1;
 				my $second = $2;
-
+				my $start_pos = $-[1];
+				my $end_pos = $+[2];
 				if ($first =~ /(?:struct|union|enum)/) {
 					pos($rawline) += length($first) + length($second) + 1;
 					next;
 				}
 
-				next if ($first ne $second);
+				next if (lc($first) ne lc($second));
 				next if ($first eq 'long');
 
+				# check for character before and after the word matches
+				my $start_char = '';
+				my $end_char = '';
+				$start_char = substr($rawline, $start_pos - 1, 1) if ($start_pos > ($in_commit_log ? 0 : 1));
+				$end_char = substr($rawline, $end_pos, 1) if ($end_pos < length($rawline));
+
+				next if ($start_char =~ /^\S$/);
+				next if (index(" \t.,;?!", $end_char) == -1);
+
 				if (WARN("REPEATED_WORD",
 					 "Possible repeated word: '$first'\n" . $herecurr) &&
 				    $fix) {
-- 
2.27.0

Re: [PATCH v5] checkpatch: add new exception to repeated word check

[Joe Perches]

On Sat, 2020-10-17 at 21:57 +0530, Dwaipayan Ray wrote:
> Recently, commit 4f6ad8aa1eac ("checkpatch: move repeated word test")
> moved the repeated word test to check for more file types. But after
> this, if checkpatch.pl is run on MAINTAINERS, it generates several
> new warnings of the type:

NAK.

Slow down and test before you send more patch versions.

> +				next if (index(" \t.,;?!", $end_char) == -1);

what does this do?

Re: [PATCH v5] checkpatch: add new exception to repeated word check

[Dwaipayan Ray]

On Sat, Oct 17, 2020 at 10:03 PM Joe Perches <joe@perches.com> wrote:
>
> On Sat, 2020-10-17 at 21:57 +0530, Dwaipayan Ray wrote:
> > Recently, commit 4f6ad8aa1eac ("checkpatch: move repeated word test")
> > moved the repeated word test to check for more file types. But after
> > this, if checkpatch.pl is run on MAINTAINERS, it generates several
> > new warnings of the type:
>
> NAK.
>
> Slow down and test before you send more patch versions.
>
> > +                             next if (index(" \t.,;?!", $end_char) == -1);
>
> what does this do?

Um, it checks if end_char is not present in " \t.,;?!".
If end_char doesn't belong to this list, then the check shall
skip. That is the test will skip for "word word:", but will produce
a warning for "word word." or "word word?", etc.

Shouldn't this itself be the case or am I perhaps going wrong
somewhere?

Thanks,
Dwaipayan.

Re: [PATCH v5] checkpatch: add new exception to repeated word check

[Joe Perches]

On Sat, 2020-10-17 at 22:19 +0530, Dwaipayan Ray wrote:
> On Sat, Oct 17, 2020 at 10:03 PM Joe Perches <joe@perches.com> wrote:
> > On Sat, 2020-10-17 at 21:57 +0530, Dwaipayan Ray wrote:
> > > Recently, commit 4f6ad8aa1eac ("checkpatch: move repeated word test")
> > > moved the repeated word test to check for more file types. But after
> > > this, if checkpatch.pl is run on MAINTAINERS, it generates several
> > > new warnings of the type:
> > 
> > NAK.
> > 
> > Slow down and test before you send more patch versions.
> > 
> > > +                             next if (index(" \t.,;?!", $end_char) == -1);
> > 
> > what does this do?
> 
> Um, it checks if end_char is not present in " \t.,;?!".
> If end_char doesn't belong to this list, then the check shall
> skip. That is the test will skip for "word word:", but will produce
> a warning for "word word." or "word word?", etc.
> 
> Shouldn't this itself be the case or am I perhaps going wrong
> somewhere?

No, you were right, I was being a numbskull.

btw: I think this should set
	my $end_char = ' ';
not '' 

so that if the last word on a line is a
repeat the test still works.

Re: [PATCH v5] checkpatch: add new exception to repeated word check

[Dwaipayan Ray]

On Sun, Oct 18, 2020 at 11:50 AM Joe Perches <joe@perches.com> wrote:
>
> On Sat, 2020-10-17 at 22:19 +0530, Dwaipayan Ray wrote:
> > On Sat, Oct 17, 2020 at 10:03 PM Joe Perches <joe@perches.com> wrote:
> > > On Sat, 2020-10-17 at 21:57 +0530, Dwaipayan Ray wrote:
> > > > Recently, commit 4f6ad8aa1eac ("checkpatch: move repeated word test")
> > > > moved the repeated word test to check for more file types. But after
> > > > this, if checkpatch.pl is run on MAINTAINERS, it generates several
> > > > new warnings of the type:
> > >
> > > NAK.
> > >
> > > Slow down and test before you send more patch versions.
> > >
> > > > +                             next if (index(" \t.,;?!", $end_char) == -1);
> > >
> > > what does this do?
> >
> > Um, it checks if end_char is not present in " \t.,;?!".
> > If end_char doesn't belong to this list, then the check shall
> > skip. That is the test will skip for "word word:", but will produce
> > a warning for "word word." or "word word?", etc.
> >
> > Shouldn't this itself be the case or am I perhaps going wrong
> > somewhere?
>
> No, you were right, I was being a numbskull.
>
> btw: I think this should set
>         my $end_char = ' ';
> not ''
>
> so that if the last word on a line is a
> repeat the test still works.
>
Hi,
Umm, index() function seems to return 0 when an empty string is
passed. I tried this:

print index(" \t.,;?!", '');

It output 0 in my case. So last words on a line seems to work.
I don't know if this changes with the perl version though.

So given this, will it be necessary to change end_char to ' ' ?
or perhaps change both start_char and end_char to a ' ' to maintain
uniformity?

Thanks,
Dwaipayan.

Re: [PATCH v5] checkpatch: add new exception to repeated word check

[Joe Perches]

On Sun, 2020-10-18 at 12:10 +0530, Dwaipayan Ray wrote:
> print index(" \t.,;?!", '');
> 
> It output 0 in my case. So last words on a line seems to work.
> I don't know if this changes with the perl version though.
> 
> So given this, will it be necessary to change end_char to ' ' ?
> or perhaps change both start_char and end_char to a ' ' to maintain
> uniformity?

It seems it's fine even back to perl 5.6, so it's fine as is.

cheers, Joe

Re: [PATCH v5] checkpatch: add new exception to repeated word check

[Dwaipayan Ray]

On Sun, Oct 18, 2020 at 12:41 PM Joe Perches <joe@perches.com> wrote:
>
> On Sun, 2020-10-18 at 12:10 +0530, Dwaipayan Ray wrote:
> > print index(" \t.,;?!", '');
> >
> > It output 0 in my case. So last words on a line seems to work.
> > I don't know if this changes with the perl version though.
> >
> > So given this, will it be necessary to change end_char to ' ' ?
> > or perhaps change both start_char and end_char to a ' ' to maintain
> > uniformity?
>
> It seems it's fine even back to perl 5.6, so it's fine as is.
>
> cheers, Joe
>
Okay thanks!
Please let me know if any other change is needed
or if it's good to go.

Regards,
Dwaipayan.

Re: [PATCH v5] checkpatch: add new exception to repeated word check

[Joe Perches]

On Sat, 2020-10-17 at 21:57 +0530, Dwaipayan Ray wrote:
> Recently, commit 4f6ad8aa1eac ("checkpatch: move repeated word test")
> moved the repeated word test to check for more file types. But after
> this, if checkpatch.pl is run on MAINTAINERS, it generates several
> new warnings of the type:

Andrew, can you pick this up please?

Acked-by: Joe Perches <joe@perches.com>

> WARNING: Possible repeated word: 'git'
> 
> For example:
> WARNING: Possible repeated word: 'git'
> +T:	git git://git.kernel.org/pub/scm/linux/kernel/git/rw/uml.git
> 
> So, the pattern "git git://..." is a false positive in this case.
> 
> There are several other combinations which may produce a wrong
> warning message, such as "@size size", ":Begin begin", etc.
> 
> Extend repeated word check to compare the characters before and
> after the word matches.
> 
> If there is a non whitespace character before the first word or a
> non whitespace character excluding punctuation characters after
> the second word, then the check is skipped and the warning is avoided.
> 
> Also add case insensitive word matching to the repeated word check.
> 
> Link: https://lore.kernel.org/linux-kernel-mentees/81b6a0bb2c7b9256361573f7a13201ebcd4876f1.camel@perches.com/
> Suggested-by: Joe Perches <joe@perches.com>
> Suggested-by: Lukas Bulwahn <lukas.bulwahn@gmail.com>
> Signed-off-by: Dwaipayan Ray <dwaipayanray1@gmail.com>
> ---
>  scripts/checkpatch.pl | 15 +++++++++++++--
>  1 file changed, 13 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl
> index f1a4e61917eb..ec380626bebc 100755
> --- a/scripts/checkpatch.pl
> +++ b/scripts/checkpatch.pl
> @@ -3052,19 +3052,30 @@ sub process {
>  
>  # check for repeated words separated by a single space
>  		if ($rawline =~ /^\+/ || $in_commit_log) {
> +			pos($rawline) = 1 if (!$in_commit_log);
>  			while ($rawline =~ /\b($word_pattern) (?=($word_pattern))/g) {
>  
>  				my $first = $1;
>  				my $second = $2;
> -
> +				my $start_pos = $-[1];
> +				my $end_pos = $+[2];
>  				if ($first =~ /(?:struct|union|enum)/) {
>  					pos($rawline) += length($first) + length($second) + 1;
>  					next;
>  				}
>  
> -				next if ($first ne $second);
> +				next if (lc($first) ne lc($second));
>  				next if ($first eq 'long');
>  
> +				# check for character before and after the word matches
> +				my $start_char = '';
> +				my $end_char = '';
> +				$start_char = substr($rawline, $start_pos - 1, 1) if ($start_pos > ($in_commit_log ? 0 : 1));
> +				$end_char = substr($rawline, $end_pos, 1) if ($end_pos < length($rawline));
> +
> +				next if ($start_char =~ /^\S$/);
> +				next if (index(" \t.,;?!", $end_char) == -1);
> +
>  				if (WARN("REPEATED_WORD",
>  					 "Possible repeated word: '$first'\n" . $herecurr) &&
>  				    $fix) {