[PATCH] checkpatch: warn when patch exceeds a maximum message size

[PATCH] checkpatch: warn when patch exceeds a maximum message size

[Bruce Allan]

Patch submissions which exceed a maximum message size imposed by email
list servers can be silently blocked.  Warn when a patch message exceeds
the maximum message size (100,000 characters) allowed by Majordomo at
vger.kernel.org since that is where most Linux email lists are served.
Also provide a command-line option to specify a different maximum size.

Signed-off-by: Bruce Allan <bruce.w.allan@intel.com>
---

 scripts/checkpatch.pl |   15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl
index 34eb216..79bd7fd6 100755
--- a/scripts/checkpatch.pl
+++ b/scripts/checkpatch.pl
@@ -40,6 +40,7 @@ my @ignore = ();
 my $help = 0;
 my $configuration_file = ".checkpatch.conf";
 my $max_line_length = 80;
+my $max_msg_size = 100000;	# Max allowed by Majordomo at vger.kernel.org
 my $ignore_perl_version = 0;
 my $minimum_perl_version = 5.10.0;
 
@@ -62,6 +63,7 @@ Options:
   --types TYPE(,TYPE2...)    show only these comma separated message types
   --ignore TYPE(,TYPE2...)   ignore various comma separated message types
   --max-line-length=n        set the maximum line length, if exceeded, warn
+  --max-msg-size=n           set the maximum message size, if exceeded, warn
   --show-types               show the message "types" in the output
   --root=PATH                PATH to the kernel tree root
   --no-summary               suppress the per-file summary
@@ -130,6 +132,7 @@ GetOptions(
 	'types=s'	=> \@use,
 	'show-types!'	=> \$show_types,
 	'max-line-length=i' => \$max_line_length,
+	'max-msg-size=i' => \$max_msg_size,
 	'root=s'	=> \$root,
 	'summary!'	=> \$summary,
 	'mailback!'	=> \$mailback,
@@ -4600,6 +4603,18 @@ sub process {
 		exit(0);
 	}
 
+	# Warn if patch exceeds the maximum message size
+	if (!$file) {
+		my $cnt_chars = 0;
+		foreach my $rawline (@rawlines) {
+			$cnt_chars += length($rawline) + 1;
+		}
+		if ($cnt_chars > $max_msg_size) {
+			WARN("PATCH_TOO_LARGE",
+			     "patch exceeds maximum message size ($max_msg_size) and might be blocked by email list server(s).\n");
+		}
+	}
+
 	if (!$is_patch) {
 		ERROR("NOT_UNIFIED_DIFF",
 		      "Does not appear to be a unified-diff format patch\n");

Re: [PATCH] checkpatch: warn when patch exceeds a maximum message size

[Joe Perches]

On Thu, 2014-06-05 at 17:00 -0700, Bruce Allan wrote:
> the maximum message size (100,000 characters) allowed by Majordomo at
> vger.kernel.org since that is where most Linux email lists are served.

I believe this is incorrect and the patch is unnecessary.

It'd maybe more interesting if the email header sizes
were calculated and if > 1024, then some bleating message
were emitted.

> Also provide a command-line option to specify a different maximum size.
> 
> Signed-off-by: Bruce Allan <bruce.w.allan@intel.com>
> ---
> 
>  scripts/checkpatch.pl |   15 +++++++++++++++
>  1 file changed, 15 insertions(+)
> 
> diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl
> index 34eb216..79bd7fd6 100755
> --- a/scripts/checkpatch.pl
> +++ b/scripts/checkpatch.pl
> @@ -40,6 +40,7 @@ my @ignore = ();
>  my $help = 0;
>  my $configuration_file = ".checkpatch.conf";
>  my $max_line_length = 80;
> +my $max_msg_size = 100000;	# Max allowed by Majordomo at vger.kernel.org
>  my $ignore_perl_version = 0;
>  my $minimum_perl_version = 5.10.0;
>  
> @@ -62,6 +63,7 @@ Options:
>    --types TYPE(,TYPE2...)    show only these comma separated message types
>    --ignore TYPE(,TYPE2...)   ignore various comma separated message types
>    --max-line-length=n        set the maximum line length, if exceeded, warn
> +  --max-msg-size=n           set the maximum message size, if exceeded, warn
>    --show-types               show the message "types" in the output
>    --root=PATH                PATH to the kernel tree root
>    --no-summary               suppress the per-file summary
> @@ -130,6 +132,7 @@ GetOptions(
>  	'types=s'	=> \@use,
>  	'show-types!'	=> \$show_types,
>  	'max-line-length=i' => \$max_line_length,
> +	'max-msg-size=i' => \$max_msg_size,
>  	'root=s'	=> \$root,
>  	'summary!'	=> \$summary,
>  	'mailback!'	=> \$mailback,
> @@ -4600,6 +4603,18 @@ sub process {
>  		exit(0);
>  	}
>  
> +	# Warn if patch exceeds the maximum message size
> +	if (!$file) {
> +		my $cnt_chars = 0;
> +		foreach my $rawline (@rawlines) {
> +			$cnt_chars += length($rawline) + 1;
> +		}
> +		if ($cnt_chars > $max_msg_size) {
> +			WARN("PATCH_TOO_LARGE",
> +			     "patch exceeds maximum message size ($max_msg_size) and might be blocked by email list server(s).\n");
> +		}
> +	}
> +
>  	if (!$is_patch) {
>  		ERROR("NOT_UNIFIED_DIFF",
>  		      "Does not appear to be a unified-diff format patch\n");
> 

RE: [PATCH] checkpatch: warn when patch exceeds a maximum message size

[Allan, Bruce W]

> -----Original Message-----
> From: Joe Perches [mailto:joe@perches.com]
> Sent: Thursday, June 05, 2014 6:36 PM
> To: Allan, Bruce W
> Cc: apw@canonical.com; linux-kernel@vger.kernel.org; postmaster
> Subject: Re: [PATCH] checkpatch: warn when patch exceeds a maximum
> message size
> 
> On Thu, 2014-06-05 at 17:00 -0700, Bruce Allan wrote:
> > the maximum message size (100,000 characters) allowed by Majordomo at
> > vger.kernel.org since that is where most Linux email lists are served.
> 
> I believe this is incorrect and the patch is unnecessary.

Actually, it is correct (it even says so at http://vger.kernel.org/majordomo-info.html).
We just experienced 2 out of 4 patches in the initial RFC submittal of the Intel(R)
QuickAssist Technology (QAT) driver being blocked by Majordomo from getting out
on the linux-crypto mailing list.  It took a while to figure out the problem and had to
break up those two blocked patches into smaller ones which went through fine.  I
submitted this checkpatch patch in hopes to prevent others from having to figure out
the same.

> 
> It'd maybe more interesting if the email header sizes
> were calculated and if > 1024, then some bleating message
> were emitted.
> 
> > Also provide a command-line option to specify a different maximum size.
> >
> > Signed-off-by: Bruce Allan <bruce.w.allan@intel.com>
> > ---
> >
> >  scripts/checkpatch.pl |   15 +++++++++++++++
> >  1 file changed, 15 insertions(+)
> >
> > diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl
> > index 34eb216..79bd7fd6 100755
> > --- a/scripts/checkpatch.pl
> > +++ b/scripts/checkpatch.pl
> > @@ -40,6 +40,7 @@ my @ignore = ();
> >  my $help = 0;
> >  my $configuration_file = ".checkpatch.conf";
> >  my $max_line_length = 80;
> > +my $max_msg_size = 100000;	# Max allowed by Majordomo at
> vger.kernel.org
> >  my $ignore_perl_version = 0;
> >  my $minimum_perl_version = 5.10.0;
> >
> > @@ -62,6 +63,7 @@ Options:
> >    --types TYPE(,TYPE2...)    show only these comma separated message
> types
> >    --ignore TYPE(,TYPE2...)   ignore various comma separated message
> types
> >    --max-line-length=n        set the maximum line length, if exceeded, warn
> > +  --max-msg-size=n           set the maximum message size, if exceeded,
> warn
> >    --show-types               show the message "types" in the output
> >    --root=PATH                PATH to the kernel tree root
> >    --no-summary               suppress the per-file summary
> > @@ -130,6 +132,7 @@ GetOptions(
> >  	'types=s'	=> \@use,
> >  	'show-types!'	=> \$show_types,
> >  	'max-line-length=i' => \$max_line_length,
> > +	'max-msg-size=i' => \$max_msg_size,
> >  	'root=s'	=> \$root,
> >  	'summary!'	=> \$summary,
> >  	'mailback!'	=> \$mailback,
> > @@ -4600,6 +4603,18 @@ sub process {
> >  		exit(0);
> >  	}
> >
> > +	# Warn if patch exceeds the maximum message size
> > +	if (!$file) {
> > +		my $cnt_chars = 0;
> > +		foreach my $rawline (@rawlines) {
> > +			$cnt_chars += length($rawline) + 1;
> > +		}
> > +		if ($cnt_chars > $max_msg_size) {
> > +			WARN("PATCH_TOO_LARGE",
> > +			     "patch exceeds maximum message size
> ($max_msg_size) and might be blocked by email list server(s).\n");
> > +		}
> > +	}
> > +
> >  	if (!$is_patch) {
> >  		ERROR("NOT_UNIFIED_DIFF",
> >  		      "Does not appear to be a unified-diff format patch\n");
> >
> 
> 

Re: [PATCH] checkpatch: warn when patch exceeds a maximum message size

[Joe Perches]

On Fri, 2014-06-06 at 15:08 +0000, Allan, Bruce W wrote:
> > -----Original Message-----
> > From: Joe Perches [mailto:joe@perches.com]
> > Sent: Thursday, June 05, 2014 6:36 PM
> > To: Allan, Bruce W
> > Cc: apw@canonical.com; linux-kernel@vger.kernel.org; postmaster
> > Subject: Re: [PATCH] checkpatch: warn when patch exceeds a maximum
> > message size
> > 
> > On Thu, 2014-06-05 at 17:00 -0700, Bruce Allan wrote:
> > > the maximum message size (100,000 characters) allowed by Majordomo at
> > > vger.kernel.org since that is where most Linux email lists are served.
> > 
> > I believe this is incorrect and the patch is unnecessary.
> 
> Actually, it is correct (it even says so at http://vger.kernel.org/majordomo-info.html).

where?

All it says there that's relevant is:

------------------------------------------
All email sent to there must be TEXT/PLAIN, there can be no multipart
messages, no VCARDs, nothing ``fancy''. In presence of such things,
Majordomo will very likely do the wrong thing. 

When you send there email, do make sure that all of the email headers,
both visible and transport level, have same addresses in them. People
experience problems when for example ``From:'', ``Sender:'' and possible
``Reply-To:'' headers present different addresses. The most common
manifestation is complete silence from VGER! 
------------------------------------------

> We just experienced 2 out of 4 patches in the initial RFC submittal of the Intel(R)
> QuickAssist Technology (QAT) driver being blocked by Majordomo from getting out
> on the linux-crypto mailing list.

Possible, but the largest patch I've seen recently
on linux-kernel is about .5 Meg, 5 times your limit.

>   It took a while to figure out the problem and had to
> break up those two blocked patches into smaller ones which went through fine.  I
> submitted this checkpatch patch in hopes to prevent others from having to figure out
> the same.

Maybe specific to crypto?

> > It'd maybe more interesting if the email header sizes
> > were calculated and if > 1024, then some bleating message
> > were emitted.

This specific bit, email headers > 1K in total size,
is an actual failure mode btw.

Re: [PATCH] checkpatch: warn when patch exceeds a maximum message size

[Joe Perches]

On Fri, 2014-06-06 at 08:26 -0700, Joe Perches wrote:
> On Fri, 2014-06-06 at 15:08 +0000, Allan, Bruce W wrote:
> > From: Joe Perches [mailto:joe@perches.com]
> > Sent: Thursday, June 05, 2014 6:36 PM
> > > On Thu, 2014-06-05 at 17:00 -0700, Bruce Allan wrote:
> > > > the maximum message size (100,000 characters) allowed by Majordomo at
> > > > vger.kernel.org since that is where most Linux email lists are served.
> > > I believe this is incorrect and the patch is unnecessary.
> > Actually, it is correct (it even says so at http://vger.kernel.org/majordomo-info.html).
> where?

Duh, near the very end it does have:

	Message size exceeding 100 000 characters causes blocking. 

But I believe that info is out of date.

RE: [PATCH] checkpatch: warn when patch exceeds a maximum message size

[Allan, Bruce W]

> -----Original Message-----
> From: Joe Perches [mailto:joe@perches.com]
> Sent: Friday, June 06, 2014 8:27 AM
> To: Allan, Bruce W
> Cc: apw@canonical.com; linux-kernel@vger.kernel.org; postmaster
> Subject: Re: [PATCH] checkpatch: warn when patch exceeds a maximum
> message size
> 
> On Fri, 2014-06-06 at 15:08 +0000, Allan, Bruce W wrote:
> > > -----Original Message-----
> > > From: Joe Perches [mailto:joe@perches.com]
> > > Sent: Thursday, June 05, 2014 6:36 PM
> > > To: Allan, Bruce W
> > > Cc: apw@canonical.com; linux-kernel@vger.kernel.org; postmaster
> > > Subject: Re: [PATCH] checkpatch: warn when patch exceeds a maximum
> > > message size
> > >
> > > On Thu, 2014-06-05 at 17:00 -0700, Bruce Allan wrote:
> > > > the maximum message size (100,000 characters) allowed by
> Majordomo at
> > > > vger.kernel.org since that is where most Linux email lists are served.
> > >
> > > I believe this is incorrect and the patch is unnecessary.
> >
> > Actually, it is correct (it even says so at
> http://vger.kernel.org/majordomo-info.html).
> 
> where?
> 
> All it says there that's relevant is:
> 
> ------------------------------------------
> All email sent to there must be TEXT/PLAIN, there can be no multipart
> messages, no VCARDs, nothing ``fancy''. In presence of such things,
> Majordomo will very likely do the wrong thing.
> 
> When you send there email, do make sure that all of the email headers,
> both visible and transport level, have same addresses in them. People
> experience problems when for example ``From:'', ``Sender:'' and possible
> ``Reply-To:'' headers present different addresses. The most common
> manifestation is complete silence from VGER!
> ------------------------------------------
> 

http://vger.kernel.org/majordomo-info.html#taboo says:
The Majordomo is configured with a set of filter rules which when triggered will send
the email to "/dev/null". 
(List owner actually, but they are overworked elsewere, and use "d" button usually...)
<snip>
* Message size exceeding 100 000 characters causes blocking.


> > We just experienced 2 out of 4 patches in the initial RFC submittal of the
> Intel(R)
> > QuickAssist Technology (QAT) driver being blocked by Majordomo from
> getting out
> > on the linux-crypto mailing list.
> 
> Possible, but the largest patch I've seen recently
> on linux-kernel is about .5 Meg, 5 times your limit.
> 
> >   It took a while to figure out the problem and had to
> > break up those two blocked patches into smaller ones which went
> through fine.  I
> > submitted this checkpatch patch in hopes to prevent others from having
> to figure out
> > the same.
> 
> Maybe specific to crypto?
> 
> > > It'd maybe more interesting if the email header sizes
> > > were calculated and if > 1024, then some bleating message
> > > were emitted.
> 
> This specific bit, email headers > 1K in total size,
> is an actual failure mode btw.

RE: [PATCH] checkpatch: warn when patch exceeds a maximum message size

[Allan, Bruce W]

> -----Original Message-----
> From: Joe Perches [mailto:joe@perches.com]
> Sent: Friday, June 06, 2014 8:34 AM
> To: Allan, Bruce W
> Cc: apw@canonical.com; linux-kernel@vger.kernel.org; postmaster
> Subject: Re: [PATCH] checkpatch: warn when patch exceeds a maximum
> message size
> 
> On Fri, 2014-06-06 at 08:26 -0700, Joe Perches wrote:
> > On Fri, 2014-06-06 at 15:08 +0000, Allan, Bruce W wrote:
> > > From: Joe Perches [mailto:joe@perches.com]
> > > Sent: Thursday, June 05, 2014 6:36 PM
> > > > On Thu, 2014-06-05 at 17:00 -0700, Bruce Allan wrote:
> > > > > the maximum message size (100,000 characters) allowed by
> Majordomo at
> > > > > vger.kernel.org since that is where most Linux email lists are served.
> > > > I believe this is incorrect and the patch is unnecessary.
> > > Actually, it is correct (it even says so at
> http://vger.kernel.org/majordomo-info.html).
> > where?
> 
> Duh, near the very end it does have:
> 
> 	Message size exceeding 100 000 characters causes blocking.
> 
> But I believe that info is out of date.
> 

If it is out of date, can you explain how/why we got stung by it?

Re: [PATCH] checkpatch: warn when patch exceeds a maximum message size

[Joe Perches]

On Fri, 2014-06-06 at 15:35 +0000, Allan, Bruce W wrote:
> > 	Message size exceeding 100 000 characters causes blocking.
> > 
> > But I believe that info is out of date.
> > 
> If it is out of date, can you explain how/why we got stung by it?

Dunno.

Too many recipients maybe?

cc'ing more than 20 or so people can run up against
that 1024 total byte header limit.

Maybe send me a copy of the original rejected posting?

RE: [PATCH] checkpatch: warn when patch exceeds a maximum message size

[Allan, Bruce W]

> -----Original Message-----
> From: Joe Perches [mailto:joe@perches.com]
> Sent: Friday, June 06, 2014 8:45 AM
> To: Allan, Bruce W
> Cc: apw@canonical.com; linux-kernel@vger.kernel.org; postmaster
> Subject: Re: [PATCH] checkpatch: warn when patch exceeds a maximum
> message size
> 
> On Fri, 2014-06-06 at 15:35 +0000, Allan, Bruce W wrote:
> > > 	Message size exceeding 100 000 characters causes blocking.
> > >
> > > But I believe that info is out of date.
> > >
> > If it is out of date, can you explain how/why we got stung by it?
> 
> Dunno.
> 
> Too many recipients maybe?
> 
> cc'ing more than 20 or so people can run up against
> that 1024 total byte header limit.
> 
> Maybe send me a copy of the original rejected posting?

Will-do, off-list.