RFC: git send-email and error handling

RFC: git send-email and error handling

[Paul Gortmaker]

I came across a situation today where the behaviour of git send-email
kind of surprised me -- in that it seemed definitely less than ideal
for the use case I had.

For stable linux kernel releases, it is very common to send
hundreds of patches at once, to people you've never contacted before,
simply because the are called out in CC: or SOB: lines of a patch
that has been cherry picked.  So it is highly likely that you may
hit full inboxes, expired accounts and so on.

The command line (git 1.7.4.4) is typically something like:

git send-email --to stable@kernel.org --to linux-kernel@vger.kernel.org \
   --cc stable-review@kernel.org   some_patch_dir

So, let me get to what happened today:  After sending 113 out of 209
patches, it came to the 114th patch, and gave me this:

(mbox) Adding cc: Dmitry Torokhov <dmitry.torokhov@gmail.com> from line 'From: Dmitry Torokhov <dmitry.torokhov@gmail.com>'
(body) Adding cc: Dmitry Torokhov <dtor@mail.ru> from line 'Signed-off-by: Dmitry Torokhov <dtor@mail.ru>'
(body) Adding cc: Paul Gortmaker <paul.gortmaker@windriver.com> from line 'Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>'
5.2.1 <dtor@mail.ru>... Mailbox disabled for this recipient

Then, taking that as a hard error, it simply exited,
leaving me scrambling to figure out how to quickly fix the
offending patch and continue with the unsent queue.

 From my point of view, the right thing to do here would have
been to ignore the error on the harvested mail address, and continue
on through the rest of the queue.  Or even interactively ask me what
to do when it saw the 5.2.1 failure.  But maybe that wouldn't be
right for everyone.  I didn't see anything in the GSE man page
that would let me configure this behaviour either.

Anyway, I thought I'd mention it and see where the discussion went.

Thanks,
Paul.

Re: RFC: git send-email and error handling

[Jeff King]

On Thu, Apr 14, 2011 at 04:10:12PM -0400, Paul Gortmaker wrote:

> The command line (git 1.7.4.4) is typically something like:
> 
> git send-email --to stable@kernel.org --to linux-kernel@vger.kernel.org \
>    --cc stable-review@kernel.org   some_patch_dir
> 
> So, let me get to what happened today:  After sending 113 out of 209
> patches, it came to the 114th patch, and gave me this:
> 
> (mbox) Adding cc: Dmitry Torokhov <dmitry.torokhov@gmail.com> from line 'From: Dmitry Torokhov <dmitry.torokhov@gmail.com>'
> (body) Adding cc: Dmitry Torokhov <dtor@mail.ru> from line 'Signed-off-by: Dmitry Torokhov <dtor@mail.ru>'
> (body) Adding cc: Paul Gortmaker <paul.gortmaker@windriver.com> from line 'Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>'
> 5.2.1 <dtor@mail.ru>... Mailbox disabled for this recipient
> 
> Then, taking that as a hard error, it simply exited,
> leaving me scrambling to figure out how to quickly fix the
> offending patch and continue with the unsent queue.

I suspect part of the issue is that your mail setup is unlike that of
most people. Usually, we would deliver to some local MTA (either by SMTP
directly, by sendmail that speaks SMTP to a smarthost, or by sendmail
that queues directly); that MTA would queue the message in a spool for
you, and attempt delivery asynchronously.  So the errors generally come
all or nothing. You can queue mail, or you can't, and trying again and
again after each error is just wasteful and annoying. Eventual errors
are reported back to you as bounces.

Your setup seems different; it looks like your sendmail (or the SMTP
server you connect to) actually routes the mail without queueing at all,
and you synchronously get the final word on whether it can be delivered.
Or maybe it just connects to the recipient site and checks that "RCPT
TO" works before actually queueing. It's hard to say from the snippet
above. What's your MTA?

I can see in your case how it would be preferable to keep going through
the list and then assemble a set of errors at the end. But that should
be configurable, because most setups won't want that behavior.

>  From my point of view, the right thing to do here would have
> been to ignore the error on the harvested mail address, and continue
> on through the rest of the queue.

It's a little tricky, because send-email may not know the details of
what happened, especially if this behavior comes from a sendmail
wrapper rather than SMTP. We dump the message and a list of recipients
to an external program, and then get back a "yes it was sent" or "no it
was not" code. So we can't do anything clever, like say "Well, it was
sent, but not to one particular address, but that's OK because that
address was auto-harvested from a signed-off-by line".

Whether we can do better depends on your MTA. _If_ you are sending via
SMTP, and _if_ it will reject particular recipients at the time of "rcpt
to", then we could do something that clever. Given that the 5.2.1
message appeared on your terminal, and that it should not have been
generated by git-send-email, that implies to me you are using the local
sendmail binary.

> Or even interactively ask me what to do when it saw the 5.2.1 failure.
> But maybe that wouldn't be right for everyone.  I didn't see anything
> in the GSE man page that would let me configure this behaviour either.

The problem there is that the message probably was not actually sent (or
at least, sendmail presumably returned an error code to git, which is
why git stopped). And you, as a human, saw that the error was something
survivable. But you can't just tell git "it's OK to continue". You need
to actually fix the problem and re-send, which means telling git that
the one particular address is not interesting. And that is a lot more
interface than just yes/no.

I would think what you really want is a system that tries to send
everything, keeps track of which recipients are to receive which
message, and then marks success or failure for each. At the end, you
would find that dtor@mail.ru didn't receive anything, and realize you
don't care. And you don't have to worry about restarting a failed
send-email and sending duplicates. You know who got what.

If that sounds good, then you should consider switching MTAs, because
that is exactly what the job of an MTA is. :)

-Peff

Re: RFC: git send-email and error handling

[Paul Gortmaker]

On 11-04-14 05:09 PM, Jeff King wrote:
> On Thu, Apr 14, 2011 at 04:10:12PM -0400, Paul Gortmaker wrote:
> 
>> The command line (git 1.7.4.4) is typically something like:
>>
>> git send-email --to stable@kernel.org --to linux-kernel@vger.kernel.org \
>>    --cc stable-review@kernel.org   some_patch_dir
>>
>> So, let me get to what happened today:  After sending 113 out of 209
>> patches, it came to the 114th patch, and gave me this:
>>
>> (mbox) Adding cc: Dmitry Torokhov <dmitry.torokhov@gmail.com> from line 'From: Dmitry Torokhov <dmitry.torokhov@gmail.com>'
>> (body) Adding cc: Dmitry Torokhov <dtor@mail.ru> from line 'Signed-off-by: Dmitry Torokhov <dtor@mail.ru>'
>> (body) Adding cc: Paul Gortmaker <paul.gortmaker@windriver.com> from line 'Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>'
>> 5.2.1 <dtor@mail.ru>... Mailbox disabled for this recipient
>>
>> Then, taking that as a hard error, it simply exited,
>> leaving me scrambling to figure out how to quickly fix the
>> offending patch and continue with the unsent queue.
> 
> I suspect part of the issue is that your mail setup is unlike that of
> most people. Usually, we would deliver to some local MTA (either by SMTP
> directly, by sendmail that speaks SMTP to a smarthost, or by sendmail
> that queues directly); that MTA would queue the message in a spool for
> you, and attempt delivery asynchronously.  So the errors generally come
> all or nothing. You can queue mail, or you can't, and trying again and
> again after each error is just wasteful and annoying. Eventual errors
> are reported back to you as bounces.
> 
> Your setup seems different; it looks like your sendmail (or the SMTP
> server you connect to) actually routes the mail without queueing at all,
> and you synchronously get the final word on whether it can be delivered.
> Or maybe it just connects to the recipient site and checks that "RCPT
> TO" works before actually queueing. It's hard to say from the snippet
> above. What's your MTA?

Yes, the above is true -- I'm not queuing anything locally or involving
a local MTA.  I've set sendemail.smtpserver in my ~/.gitconfig to the
hostname of an infrastructural server running sendmail (telnet 25 doesnt
show me what version, but I'm told it is sendmail).  This configuration
gives me the most portability to run on any random machine within our
org without having to wonder if it has a locally installed and correctly
configured MTA -- it works so well, I've even abused git send-email to
dispatch random (non-patch) mails from ad-hoc scripts on occasions, simply
because I know everyone has git installed somewhere in $PATH.

> 
> I can see in your case how it would be preferable to keep going through
> the list and then assemble a set of errors at the end. But that should
> be configurable, because most setups won't want that behavior.
> 
>>  From my point of view, the right thing to do here would have
>> been to ignore the error on the harvested mail address, and continue
>> on through the rest of the queue.
> 
> It's a little tricky, because send-email may not know the details of
> what happened, especially if this behavior comes from a sendmail
> wrapper rather than SMTP. We dump the message and a list of recipients
> to an external program, and then get back a "yes it was sent" or "no it
> was not" code. So we can't do anything clever, like say "Well, it was
> sent, but not to one particular address, but that's OK because that
> address was auto-harvested from a signed-off-by line".

True.  I wonder if there is some flexibility in what we do, depending
on whether the setting is a local binary like /usr/bin/sendmail, vs.
a hostname of a server, like it was in my case...

> 
> Whether we can do better depends on your MTA. _If_ you are sending via
> SMTP, and _if_ it will reject particular recipients at the time of "rcpt
> to", then we could do something that clever. Given that the 5.2.1
> message appeared on your terminal, and that it should not have been
> generated by git-send-email, that implies to me you are using the local
> sendmail binary.
> 
>> Or even interactively ask me what to do when it saw the 5.2.1 failure.
>> But maybe that wouldn't be right for everyone.  I didn't see anything
>> in the GSE man page that would let me configure this behaviour either.
> 
> The problem there is that the message probably was not actually sent (or
> at least, sendmail presumably returned an error code to git, which is
> why git stopped). And you, as a human, saw that the error was something
> survivable. But you can't just tell git "it's OK to continue". You need
> to actually fix the problem and re-send, which means telling git that
> the one particular address is not interesting. And that is a lot more
> interface than just yes/no.
> 
> I would think what you really want is a system that tries to send
> everything, keeps track of which recipients are to receive which
> message, and then marks success or failure for each. At the end, you
> would find that dtor@mail.ru didn't receive anything, and realize you
> don't care. And you don't have to worry about restarting a failed
> send-email and sending duplicates. You know who got what.
> 
> If that sounds good, then you should consider switching MTAs, because
> that is exactly what the job of an MTA is. :)

I could set up a local queue, and for the big batches of patches I
may do that, but the portability of having sendemail.smtpserver pointing
at a host instead of a program is too nice for me to be willing to
give that up globally.

P.

> 
> -Peff

Re: RFC: git send-email and error handling

[Jeff King]

On Thu, Apr 14, 2011 at 08:30:26PM -0400, Paul Gortmaker wrote:

> > Your setup seems different; it looks like your sendmail (or the SMTP
> > server you connect to) actually routes the mail without queueing at all,
> > and you synchronously get the final word on whether it can be delivered.
> > Or maybe it just connects to the recipient site and checks that "RCPT
> > TO" works before actually queueing. It's hard to say from the snippet
> > above. What's your MTA?
> 
> Yes, the above is true -- I'm not queuing anything locally or involving
> a local MTA.  I've set sendemail.smtpserver in my ~/.gitconfig to the
> hostname of an infrastructural server running sendmail (telnet 25 doesnt
> show me what version, but I'm told it is sendmail).

Ah. I'm not up on my sendmail config, but googling around, there are
apparently milters that will do this kind of "call-ahead" rcpt checking.

> This configuration gives me the most portability to run on any random
> machine within our org without having to wonder if it has a locally
> installed and correctly configured MTA -- it works so well, I've even
> abused git send-email to dispatch random (non-patch) mails from ad-hoc
> scripts on occasions, simply because I know everyone has git installed
> somewhere in $PATH.

Yeah, I think submitting to a central server is a very sane config if
you don't have reliably local delivery.

> > It's a little tricky, because send-email may not know the details of
> > what happened, especially if this behavior comes from a sendmail
> > wrapper rather than SMTP. We dump the message and a list of recipients
> > to an external program, and then get back a "yes it was sent" or "no it
> > was not" code. So we can't do anything clever, like say "Well, it was
> > sent, but not to one particular address, but that's OK because that
> > address was auto-harvested from a signed-off-by line".
> 
> True.  I wonder if there is some flexibility in what we do, depending
> on whether the setting is a local binary like /usr/bin/sendmail, vs.
> a hostname of a server, like it was in my case...

Sure. Since you are actually doing SMTP, you have much more flexibility
in knowing what errors happen. Look in git-send-email.perl's
send_message, around line 1118. We use the Mail::SMTP module, but we
just feed it the whole recipient list and barf if any of them is
rejected. You could probably remember which recipients are "important"
(i.e., given on the command line) and which were pulled automatically
from the commit information, and then feed each recipient individually.
If important ones fail, abort the message. If an unimportant one fails,
send the message anyway, but remember the bad address and report the
error at the end.

That wouldn't help people using a sendmail binary, but there's nothing
we can do. That transport simply doesn't supply as much information, so
it can't take advantage of the new feature. But it will be no worse off
for you adding the feature for SMTP users.

-Peff

[RFC/PATCH] git-send-email: Remember sources of Cc addresses

[Jakub Narebski]

Instead of using @initial_cc to remember --cc=<address> command line
options, and @cc to remember Cc addresses derived from message body
and --cc-cmd=<command> and ultimately gather all Cc addresses,
use %cc hash to remember from where Cc addresses came from:
 * command line --cc=<address> ('initial'),
 * "From:" email header in patch ('from'),
 * "Cc:"   email header in patch ('cc'),
 * signoff lines and Cc lines in message body ('body'),
 * result of running <command> from --cc-cmd=<command> ('cc-cmd').

This is pure refactoring: we don't use this information, but gather
together all of those in @cc variable local to send_message()
subroutine.  No changes in behavior.

While at it make assignment to $to variable in send_message()
up, to make it more clear that @recipients is just @to then.

Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
On Fri, 15 Apr 2011, Jeff King wrote:
> On Thu, Apr 14, 2011 at 08:30:26PM -0400, Paul Gortmaker wrote:

>> True.  I wonder if there is some flexibility in what we do, depending
>> on whether the setting is a local binary like /usr/bin/sendmail, vs.
>> a hostname of a server, like it was in my case...
> 
> Sure. Since you are actually doing SMTP, you have much more flexibility
> in knowing what errors happen. Look in git-send-email.perl's
> send_message, around line 1118. We use the Mail::SMTP module, but we
> just feed it the whole recipient list and barf if any of them is
> rejected. You could probably remember which recipients are "important"
> (i.e., given on the command line) and which were pulled automatically
> from the commit information, and then feed each recipient individually.
> If important ones fail, abort the message. If an unimportant one fails,
> send the message anyway, but remember the bad address and report the
> error at the end.
> 
> That wouldn't help people using a sendmail binary, but there's nothing
> we can do. That transport simply doesn't supply as much information, so
> it can't take advantage of the new feature. But it will be no worse off
> for you adding the feature for SMTP users.

This is an RFC patch preparing the way, so to speak, by remembering
where each Cc address came from.  We could in the future treat
$cc{'body'} / all_cc('body') differently from the rest of all_cc().

Is the approach taken here sane?

 git-send-email.perl |   52 ++++++++++++++++++++++++++++++--------------------
 1 files changed, 31 insertions(+), 21 deletions(-)

diff --git a/git-send-email.perl b/git-send-email.perl
index 1c6b1a8..7d75a1e 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -140,10 +140,18 @@ my $smtp;
 my $auth;
 
 # Variables we fill in automatically, or via prompting:
-my (@to,$no_to,@initial_to,@cc,$no_cc,@initial_cc,@bcclist,$no_bcc,@xh,
+my (@to,$no_to,@initial_to,%cc,$no_cc,@bcclist,$no_bcc,@xh,
 	$initial_reply_to,$initial_subject,@files,
 	$author,$sender,$smtp_authpass,$annotate,$compose,$time);
 
+sub all_cc {
+	my @keys = @_;
+	@keys = qw(initial from cc body cc-cmd) unless @keys;
+	return map { ref($_) ? @$_ : () } @cc{@keys};
+
+	#return map { ref($_) ? @$_ : () } values %cc;
+}
+
 my $envelope_sender;
 
 # Example reply to:
@@ -221,7 +229,7 @@ my %config_settings = (
     "smtpdomain" => \$smtp_domain,
     "to" => \@initial_to,
     "tocmd" => \$to_cmd,
-    "cc" => \@initial_cc,
+    "cc" => \@{$cc{'initial'}},
     "cccmd" => \$cc_cmd,
     "aliasfiletype" => \$aliasfiletype,
     "bcc" => \@bcclist,
@@ -281,7 +289,7 @@ my $rc = GetOptions("sender|from=s" => \$sender,
 		    "to=s" => \@initial_to,
 		    "to-cmd=s" => \$to_cmd,
 		    "no-to" => \$no_to,
-		    "cc=s" => \@initial_cc,
+		    "cc=s" => \@{$cc{'initial'}},
 		    "no-cc" => \$no_cc,
 		    "bcc=s" => \@bcclist,
 		    "no-bcc" => \$no_bcc,
@@ -423,7 +431,7 @@ foreach my $entry (@initial_to) {
 	die "Comma in --to entry: $entry'\n" unless $entry !~ m/,/;
 }
 
-foreach my $entry (@initial_cc) {
+foreach my $entry (@{$cc{'initial'}}) {
 	die "Comma in --cc entry: $entry'\n" unless $entry !~ m/,/;
 }
 
@@ -753,7 +761,7 @@ sub expand_one_alias {
 
 @initial_to = expand_aliases(@initial_to);
 @initial_to = (map { sanitize_address($_) } @initial_to);
-@initial_cc = expand_aliases(@initial_cc);
+@{$cc{'initial'}} = expand_aliases(@{$cc{'initial'}});
 @bcclist = expand_aliases(@bcclist);
 
 if ($thread && !defined $initial_reply_to && $prompting) {
@@ -959,12 +967,14 @@ sub maildomain {
 
 sub send_message {
 	my @recipients = unique_email_list(@to);
-	@cc = (grep { my $cc = extract_valid_address($_);
-		      not grep { $cc eq $_ || $_ =~ /<\Q${cc}\E>$/ } @recipients
-		    }
-	       map { sanitize_address($_) }
-	       @cc);
-	my $to = join (",\n\t", @recipients);
+	my $to = join(",\n\t", @recipients);
+	my @cc =
+		grep {
+			my $cc = extract_valid_address($_);
+			not grep { $cc eq $_ || $_ =~ /<\Q${cc}\E>$/ } @recipients
+		}
+		map { sanitize_address($_) }
+		all_cc();
 	@recipients = unique_email_list(@recipients,@cc,@bcclist);
 	@recipients = (map { extract_valid_address($_) } @recipients);
 	my $date = format_2822_time($time++);
@@ -1159,7 +1169,7 @@ foreach my $t (@files) {
 	my $has_content_type;
 	my $body_encoding;
 	@to = ();
-	@cc = ();
+	$cc{$_} = [] foreach (qw(from cc body));
 	@xh = ();
 	my $input_format = undef;
 	my @header = ();
@@ -1197,7 +1207,7 @@ foreach my $t (@files) {
 				next if $suppress_cc{'self'} and $author eq $sender;
 				printf("(mbox) Adding cc: %s from line '%s'\n",
 					$1, $_) unless $quiet;
-				push @cc, $1;
+				push @{$cc{'from'}}, $1;
 			}
 			elsif (/^To:\s+(.*)$/) {
 				foreach my $addr (parse_address_line($1)) {
@@ -1215,7 +1225,7 @@ foreach my $t (@files) {
 					}
 					printf("(mbox) Adding cc: %s from line '%s'\n",
 						$addr, $_) unless $quiet;
-					push @cc, $addr;
+					push @{$cc{'cc'}}, $addr;
 				}
 			}
 			elsif (/^Content-type:/i) {
@@ -1239,10 +1249,10 @@ foreach my $t (@files) {
 			# line 2 = subject
 			# So let's support that, too.
 			$input_format = 'lots';
-			if (@cc == 0 && !$suppress_cc{'cc'}) {
+			if (all_cc() == 0 && !$suppress_cc{'cc'}) {
 				printf("(non-mbox) Adding cc: %s from line '%s'\n",
 					$_, $_) unless $quiet;
-				push @cc, $_;
+				push @{$cc{'cc'}}, $_;
 			} elsif (!defined $subject) {
 				$subject = $_;
 			}
@@ -1261,7 +1271,7 @@ foreach my $t (@files) {
 				next if $suppress_cc{'sob'} and $what =~ /Signed-off-by/i;
 				next if $suppress_cc{'bodycc'} and $what =~ /Cc/i;
 			}
-			push @cc, $c;
+			push @{$cc{'body'}}, $c;
 			printf("(body) Adding cc: %s from line '%s'\n",
 				$c, $_) unless $quiet;
 		}
@@ -1270,7 +1280,7 @@ foreach my $t (@files) {
 
 	push @to, recipients_cmd("to-cmd", "to", $to_cmd, $t)
 		if defined $to_cmd;
-	push @cc, recipients_cmd("cc-cmd", "cc", $cc_cmd, $t)
+	push @{$cc{'cc-cmd'}}, recipients_cmd("cc-cmd", "cc", $cc_cmd, $t)
 		if defined $cc_cmd && !$suppress_cc{'cccmd'};
 
 	if ($broken_encoding{$t} && !$has_content_type) {
@@ -1308,14 +1318,14 @@ foreach my $t (@files) {
 		}
 	}
 
+	my $has_cc = all_cc();
 	$needs_confirm = (
 		$confirm eq "always" or
-		($confirm =~ /^(?:auto|cc)$/ && @cc) or
+		($confirm =~ /^(?:auto|cc)$/ && $has_cc) or
 		($confirm =~ /^(?:auto|compose)$/ && $compose && $message_num == 1));
-	$needs_confirm = "inform" if ($needs_confirm && $confirm_unconfigured && @cc);
+	$needs_confirm = "inform" if ($needs_confirm && $confirm_unconfigured && $has_cc);
 
 	@to = (@initial_to, @to);
-	@cc = (@initial_cc, @cc);
 
 	my $message_was_sent = send_message();
 
-- 
1.7.5

Re: [RFC/PATCH] git-send-email: Remember sources of Cc addresses

[Jeff King]

On Wed, May 04, 2011 at 06:12:08PM +0200, Jakub Narebski wrote:

> > Sure. Since you are actually doing SMTP, you have much more flexibility
> > in knowing what errors happen. Look in git-send-email.perl's
> > send_message, around line 1118. We use the Mail::SMTP module, but we
> > just feed it the whole recipient list and barf if any of them is
> > rejected. You could probably remember which recipients are "important"
> > (i.e., given on the command line) and which were pulled automatically
> > from the commit information, and then feed each recipient individually.
> > If important ones fail, abort the message. If an unimportant one fails,
> > send the message anyway, but remember the bad address and report the
> > error at the end.
> [...]
> This is an RFC patch preparing the way, so to speak, by remembering
> where each Cc address came from.  We could in the future treat
> $cc{'body'} / all_cc('body') differently from the rest of all_cc().
> 
> Is the approach taken here sane?

Yeah, from my cursory read, it looks like a good step forward, and I
didn't see any obvious bugs.

You'll need still more refactoring in send_message to treat them
differently at the SMTP level. We collapse all of the addresses down to
a single list via unique_email_list (and we obviously want to keep this
unique-ifying step), but that final list will have to remember where
each address came from.

> +sub all_cc {
> +	my @keys = @_;
> +	@keys = qw(initial from cc body cc-cmd) unless @keys;
> +	return map { ref($_) ? @$_ : () } @cc{@keys};
> +
> +	#return map { ref($_) ? @$_ : () } values %cc;
> +}

Nit: debugging cruft. :)

-Peff

[RFC/PATCH 2/2] git-send-email: Do not require that addresses added from body be valid

[Jakub Narebski]

On Wed, 4 May 2011, Jeff King wrote:
> On Wed, May 04, 2011 at 06:12:08PM +0200, Jakub Narebski wrote:
> > On Fri, 15 Apr 2011, Jeff King wrote:
> > >
> > > Sure. Since you are actually doing SMTP, you have much more flexibility
> > > in knowing what errors happen. Look in git-send-email.perl's
> > > send_message, around line 1118. We use the Mail::SMTP module, but we
> > > just feed it the whole recipient list and barf if any of them is
> > > rejected. You could probably remember which recipients are "important"
> > > (i.e., given on the command line) and which were pulled automatically
> > > from the commit information, and then feed each recipient individually.
> > > If important ones fail, abort the message. If an unimportant one fails,
> > > send the message anyway, but remember the bad address and report the
> > > error at the end.
> > [...]
> > This is an RFC patch preparing the way, so to speak, by remembering
> > where each Cc address came from.  We could in the future treat
> > $cc{'body'} / all_cc('body') differently from the rest of all_cc().
> > 
> > Is the approach taken here sane?
> 
> Yeah, from my cursory read, it looks like a good step forward, and I
> didn't see any obvious bugs.
> 
> You'll need still more refactoring in send_message to treat them
> differently at the SMTP level. We collapse all of the addresses down to
> a single list via unique_email_list (and we obviously want to keep this
> unique-ifying step), but that final list will have to remember where
> each address came from.

Below there is RFC patch that implements it by separating @cc and @cc_extra
and later @recipients and @recipients_extra.

How about it?

It does not warn about bad addresses from body, and there are no tests yet!

> > +sub all_cc {
> > +	my @keys = @_;
> > +	@keys = qw(initial from cc body cc-cmd) unless @keys;
> > +	return map { ref($_) ? @$_ : () } @cc{@keys};
> > +
> > +	#return map { ref($_) ? @$_ : () } values %cc;
> > +}
> 
> Nit: debugging cruft. :)

Right, I'll fix it in final version.

-- >8 ---------- >8 --
Subject: [PATCH] git-send-email: Do not require that addresses added from
 body be valid

Do not barf if any of recipients that was pulled automatically from
commit information is rejected.  This covers [currently] addresses
from 'Cc:' and 'Signed-off-by:' lines in message body.

This is possible only if we are using Net::SMTP or Net::SMTP::SSL
(it means that --smtp-server / sendemail.smtpserver is set to
internet address of outgoing SMTP server to use), because only then we
have control over which addresses are to be checked.

Currently no warnings that "unimportant" addresses were rejected are
shown; we should report errors at the end.

No test for this new behavior: perhaps we could adapt some test from
Net::SMTP module distribution...

Cc: Jeff King <peff@peff.net>
Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
 git-send-email.perl |   40 +++++++++++++++++++++++++---------------
 1 files changed, 25 insertions(+), 15 deletions(-)

diff --git a/git-send-email.perl b/git-send-email.perl
index 7d75a1e..e758fd9 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -968,22 +968,32 @@ sub maildomain {
 sub send_message {
 	my @recipients = unique_email_list(@to);
 	my $to = join(",\n\t", @recipients);
-	my @cc =
-		grep {
-			my $cc = extract_valid_address($_);
-			not grep { $cc eq $_ || $_ =~ /<\Q${cc}\E>$/ } @recipients
-		}
-		map { sanitize_address($_) }
-		all_cc();
-	@recipients = unique_email_list(@recipients,@cc,@bcclist);
+	my $sanitize_cc = sub {
+		return
+			grep {
+				my $cc = extract_valid_address($_);
+				not grep { $cc eq $_ || $_ =~ /<\Q${cc}\E>$/ } @recipients
+			}
+			map { sanitize_address($_) }
+			all_cc(@_);
+	};
+	my @cc = $sanitize_cc->(qw(initial from cc cc-cmd));
+	my @cc_extra = $sanitize_cc->(qw(body));
+
+	my (%seen, @recipients_extra);
+	@recipients = unique_email_list(\%seen,@recipients,@cc,@bcclist);
 	@recipients = (map { extract_valid_address($_) } @recipients);
+	@recipients_extra =
+		map { extract_valid_address($_) }
+		unique_email_list(\%seen,@cc_extra);
+
 	my $date = format_2822_time($time++);
 	my $gitversion = '@@GIT_VERSION@@';
 	if ($gitversion =~ m/..GIT_VERSION../) {
 	    $gitversion = Git::version();
 	}
 
-	my $cc = join(",\n\t", unique_email_list(@cc));
+	my $cc = join(",\n\t", unique_email_list(@cc,@cc_extra));
 	my $ccline = "";
 	if ($cc ne '') {
 		$ccline = "\nCc: $cc";
@@ -1007,7 +1017,7 @@ X-Mailer: git-send-email $gitversion
 		$header .= join("\n", @xh) . "\n";
 	}
 
-	my @sendmail_parameters = ('-i', @recipients);
+	my @sendmail_parameters = ('-i', @recipients,@recipients_extra);
 	my $raw_from = $sanitized_sender;
 	if (defined $envelope_sender && $envelope_sender ne "auto") {
 		$raw_from = $envelope_sender;
@@ -1126,6 +1136,7 @@ X-Mailer: git-send-email $gitversion
 
 		$smtp->mail( $raw_from ) or die $smtp->message;
 		$smtp->to( @recipients ) or die $smtp->message;
+		$smtp->to( @recipients_extra, { Notify => ['NEVER'], SkipBad => 1 });
 		$smtp->data or die $smtp->message;
 		$smtp->datasend("$header\n$message") or die $smtp->message;
 		$smtp->dataend() or die $smtp->message;
@@ -1138,8 +1149,8 @@ X-Mailer: git-send-email $gitversion
 		if ($smtp_server !~ m#^/#) {
 			print "Server: $smtp_server\n";
 			print "MAIL FROM:<$raw_from>\n";
-			foreach my $entry (@recipients) {
-			    print "RCPT TO:<$entry>\n";
+			foreach my $entry (@recipients,@recipients_extra) {
+				print "RCPT TO:<$entry>\n";
 			}
 		} else {
 			print "Sendmail: $smtp_server ".join(' ',@sendmail_parameters)."\n";
@@ -1375,13 +1386,12 @@ sub cleanup_compose_files {
 $smtp->quit if $smtp;
 
 sub unique_email_list {
-	my %seen;
+	my $seen = ref $_[0] eq 'HASH' ? shift : {};
 	my @emails;
 
 	foreach my $entry (@_) {
 		if (my $clean = extract_valid_address($entry)) {
-			$seen{$clean} ||= 0;
-			next if $seen{$clean}++;
+			next if $seen->{$clean}++;
 			push @emails, $entry;
 		} else {
 			print STDERR "W: unable to extract a valid address",
-- 
1.7.5

[RFC/PATCH 3/2 (squash!)] git-send-email: Warn about rejected automatically added recipients

[Jakub Narebski]

Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
On Thu, 5 May 2011, Jakub Narebski wrote:
> On Wed, 4 May 2011, Jeff King wrote:
>> On Wed, May 04, 2011 at 06:12:08PM +0200, Jakub Narebski wrote:
>>> On Fri, 15 Apr 2011, Jeff King wrote:

>>>> [...] You could probably remember which recipients are "important"
>>>> (i.e., given on the command line) and which were pulled automatically
>>>> from the commit information, and then feed each recipient individually.
>>>> If important ones fail, abort the message. If an unimportant one fails,
>>>> send the message anyway, but remember the bad address and report the
>>>> error at the end.
> 
> It does not warn about bad addresses from body, and there are no tests yet!

Now it does warn, though I don't know if we should warn after each message,
or all together at the end, and if we should warn only about _new_ addresses.

Still no tests, and no idea how to write one...

Also if it is to be standalone commit, it needs better commit message.
But if it is to squashed with previous, it doesn't ;-)

 git-send-email.perl |   11 ++++++++++-
 1 files changed, 10 insertions(+), 1 deletions(-)

diff --git a/git-send-email.perl b/git-send-email.perl
index e758fd9..b8d4fb9 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -1136,11 +1136,20 @@ X-Mailer: git-send-email $gitversion
 
 		$smtp->mail( $raw_from ) or die $smtp->message;
 		$smtp->to( @recipients ) or die $smtp->message;
-		$smtp->to( @recipients_extra, { Notify => ['NEVER'], SkipBad => 1 });
+		my @good_recips =
+			$smtp->to( @recipients_extra, { Notify => ['NEVER'], SkipBad => 1 });
 		$smtp->data or die $smtp->message;
 		$smtp->datasend("$header\n$message") or die $smtp->message;
 		$smtp->dataend() or die $smtp->message;
 		$smtp->code =~ /250|200/ or die "Failed to send $subject\n".$smtp->message;
+
+		%seen = ();
+		unique_email_list(\%seen, @good_recips);
+		# bad recipients are those not seen on list of good recipients
+		my @bad_recips = unique_email_list(\%seen, @recipients_extra);
+		@bad_recips and
+			warn "W: The following addresses added from body were rejected:\n\t".
+				join("\n\t", @bad_recips)."\n";
 	}
 	if ($quiet) {
 		printf (($dry_run ? "Dry-" : "")."Sent %s\n", $subject);
-- 
1.7.5

Re: [RFC/PATCH 2/2] git-send-email: Do not require that addresses added from body be valid

[Jakub Narebski]

On Thu, 5 May 2011, Jakub Narebski wrote:

>                 $smtp->to( @recipients ) or die $smtp->message;
> +               $smtp->to( @recipients_extra, { Notify => ['NEVER'], SkipBad => 1 });
                                                  ^^^^^^^^^^^^^^^^^^^^

Note: contrary to what I thought this doesn't mean to not send any
notification request, but to send DSN (Delivery Status Notification)
request of 'NEVER'.  For example when using smtp.gmail.com gives:

  Net::SMTP::recipient: DSN option not supported by host at ./git-send-email line 1140

So the underlined part has to be removed.

-- 
Jakub Narebski
Poland