[PATCH] git-send-email: parse all messages in mbox

[PATCH] git-send-email: parse all messages in mbox

[Vitaly Mayatskikh]

Currently git-send-email sends all mbox in one email. This seems to be wrong,
because mbox can contain several messages. For example,
`git format-patch --stdout' forms mbox file with all patches in it.

This patch allows git send-email to send several messages from one mbox
separately.

Signed-off-by: Vitaly Mayatskikh <v.mayatskih@gmail.com>
---
 git-send-email.perl |  278 ++++++++++++++++++++++++++-------------------------
 1 files changed, 141 insertions(+), 137 deletions(-)

diff --git a/git-send-email.perl b/git-send-email.perl
index cccbf45..614fcbd 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -984,168 +984,172 @@ $message_num = 0;
 foreach my $t (@files) {
 	open(F,"<",$t) or die "can't open file $t";
 
-	my $author = undef;
-	my $author_encoding;
-	my $has_content_type;
-	my $body_encoding;
-	@cc = ();
-	@xh = ();
-	my $input_format = undef;
-	my @header = ();
-	$message = "";
-	$message_num++;
-	# First unfold multiline header fields
-	while(<F>) {
-		last if /^\s*$/;
-		if (/^\s+\S/ and @header) {
-			chomp($header[$#header]);
-			s/^\s+/ /;
-			$header[$#header] .= $_;
-	    } else {
-			push(@header, $_);
-		}
-	}
-	# Now parse the header
-	foreach(@header) {
-		if (/^From /) {
-			$input_format = 'mbox';
-			next;
-		}
-		chomp;
-		if (!defined $input_format && /^[-A-Za-z]+:\s/) {
-			$input_format = 'mbox';
+	while (1) {
+		my $author = undef;
+		my $author_encoding;
+		my $has_content_type;
+		my $body_encoding;
+		@cc = ();
+		@xh = ();
+		my $input_format = undef;
+		my @header = ();
+		$message = "";
+		$message_num++;
+		# First unfold multiline header fields
+		while(<F>) {
+			last if /^\s*$/;
+			if (/^\s+\S/ and @header) {
+				chomp($header[$#header]);
+				s/^\s+/ /;
+				$header[$#header] .= $_;
+			} else {
+				push(@header, $_);
+			}
 		}
-
-		if (defined $input_format && $input_format eq 'mbox') {
-			if (/^Subject:\s+(.*)$/) {
-				$subject = $1;
+		# Now parse the header
+		foreach(@header) {
+			if (/^From /) {
+				$input_format = 'mbox';
+				next;
 			}
-			elsif (/^From:\s+(.*)$/) {
-				($author, $author_encoding) = unquote_rfc2047($1);
-				next if $suppress_cc{'author'};
-				next if $suppress_cc{'self'} and $author eq $sender;
-				printf("(mbox) Adding cc: %s from line '%s'\n",
-					$1, $_) unless $quiet;
-				push @cc, $1;
+			chomp;
+			if (!defined $input_format && /^[-A-Za-z]+:\s/) {
+				$input_format = 'mbox';
 			}
-			elsif (/^Cc:\s+(.*)$/) {
-				foreach my $addr (parse_address_line($1)) {
-					if (unquote_rfc2047($addr) eq $sender) {
-						next if ($suppress_cc{'self'});
-					} else {
-						next if ($suppress_cc{'cc'});
-					}
+
+			if (defined $input_format && $input_format eq 'mbox') {
+				if (/^Subject:\s+(.*)$/) {
+					$subject = $1;
+				}
+				elsif (/^From:\s+(.*)$/) {
+					($author, $author_encoding) = unquote_rfc2047($1);
+					next if $suppress_cc{'author'};
+					next if $suppress_cc{'self'} and $author eq $sender;
 					printf("(mbox) Adding cc: %s from line '%s'\n",
-						$addr, $_) unless $quiet;
-					push @cc, $addr;
+					       $1, $_) unless $quiet;
+					push @cc, $1;
 				}
-			}
-			elsif (/^Content-type:/i) {
-				$has_content_type = 1;
-				if (/charset="?([^ "]+)/) {
-					$body_encoding = $1;
+				elsif (/^Cc:\s+(.*)$/) {
+					foreach my $addr (parse_address_line($1)) {
+						if (unquote_rfc2047($addr) eq $sender) {
+							next if ($suppress_cc{'self'});
+						} else {
+							next if ($suppress_cc{'cc'});
+						}
+						printf("(mbox) Adding cc: %s from line '%s'\n",
+						       $addr, $_) unless $quiet;
+						push @cc, $addr;
+					}
+				}
+				elsif (/^Content-type:/i) {
+					$has_content_type = 1;
+					if (/charset="?([^ "]+)/) {
+						$body_encoding = $1;
+					}
+					push @xh, $_;
+				}
+				elsif (/^Message-Id: (.*)/i) {
+					$message_id = $1;
+				}
+				elsif (!/^Date:\s/ && /^[-A-Za-z]+:\s+\S/) {
+					push @xh, $_;
 				}
-				push @xh, $_;
-			}
-			elsif (/^Message-Id: (.*)/i) {
-				$message_id = $1;
-			}
-			elsif (!/^Date:\s/ && /^[-A-Za-z]+:\s+\S/) {
-				push @xh, $_;
-			}
 
-		} else {
-			# In the traditional
-			# "send lots of email" format,
-			# line 1 = cc
-			# line 2 = subject
-			# So let's support that, too.
-			$input_format = 'lots';
-			if (@cc == 0 && !$suppress_cc{'cc'}) {
-				printf("(non-mbox) Adding cc: %s from line '%s'\n",
-					$_, $_) unless $quiet;
-				push @cc, $_;
-			} elsif (!defined $subject) {
-				$subject = $_;
+			} else {
+				# In the traditional
+				# "send lots of email" format,
+				# line 1 = cc
+				# line 2 = subject
+				# So let's support that, too.
+				$input_format = 'lots';
+				if (@cc == 0 && !$suppress_cc{'cc'}) {
+					printf("(non-mbox) Adding cc: %s from line '%s'\n",
+					       $_, $_) unless $quiet;
+					push @cc, $_;
+				} elsif (!defined $subject) {
+					$subject = $_;
+				}
 			}
 		}
-	}
-	# Now parse the message body
-	while(<F>) {
-		$message .=  $_;
-		if (/^(Signed-off-by|Cc): (.*)$/i) {
-			chomp;
-			my ($what, $c) = ($1, $2);
-			chomp $c;
-			if ($c eq $sender) {
-				next if ($suppress_cc{'self'});
-			} else {
-				next if $suppress_cc{'sob'} and $what =~ /Signed-off-by/i;
-				next if $suppress_cc{'bodycc'} and $what =~ /Cc/i;
+		# Now parse the message body
+		while(<F>) {
+			last if /^From /; # Next message in file
+			$message .=  $_;
+			if (/^(Signed-off-by|Cc): (.*)$/i) {
+				chomp;
+				my ($what, $c) = ($1, $2);
+				chomp $c;
+				if ($c eq $sender) {
+					next if ($suppress_cc{'self'});
+				} else {
+					next if $suppress_cc{'sob'} and $what =~ /Signed-off-by/i;
+					next if $suppress_cc{'bodycc'} and $what =~ /Cc/i;
+				}
+				push @cc, $c;
+				printf("(body) Adding cc: %s from line '%s'\n",
+				       $c, $_) unless $quiet;
 			}
-			push @cc, $c;
-			printf("(body) Adding cc: %s from line '%s'\n",
-				$c, $_) unless $quiet;
 		}
-	}
-	close F;
 
-	if (defined $cc_cmd && !$suppress_cc{'cccmd'}) {
-		open(F, "$cc_cmd $t |")
-			or die "(cc-cmd) Could not execute '$cc_cmd'";
-		while(<F>) {
-			my $c = $_;
-			$c =~ s/^\s*//g;
-			$c =~ s/\n$//g;
-			next if ($c eq $sender and $suppress_from);
-			push @cc, $c;
-			printf("(cc-cmd) Adding cc: %s from: '%s'\n",
-				$c, $cc_cmd) unless $quiet;
+		if (defined $cc_cmd && !$suppress_cc{'cccmd'}) {
+			open(CC_CMD, "$cc_cmd $t |")
+			    or die "(cc-cmd) Could not execute '$cc_cmd'";
+			while(<CC_CMD>) {
+				my $c = $_;
+				$c =~ s/^\s*//g;
+				$c =~ s/\n$//g;
+				next if ($c eq $sender and $suppress_from);
+				push @cc, $c;
+				printf("(cc-cmd) Adding cc: %s from: '%s'\n",
+				       $c, $cc_cmd) unless $quiet;
+			}
+			close CC_CMD
+			    or die "(cc-cmd) failed to close pipe to '$cc_cmd'";
 		}
-		close F
-			or die "(cc-cmd) failed to close pipe to '$cc_cmd'";
-	}
 
-	if (defined $author and $author ne $sender) {
-		$message = "From: $author\n\n$message";
-		if (defined $author_encoding) {
-			if ($has_content_type) {
-				if ($body_encoding eq $author_encoding) {
-					# ok, we already have the right encoding
+		if (defined $author and $author ne $sender) {
+			$message = "From: $author\n\n$message";
+			if (defined $author_encoding) {
+				if ($has_content_type) {
+					if ($body_encoding eq $author_encoding) {
+						# ok, we already have the right encoding
+					}
+					else {
+						# uh oh, we should re-encode
+					}
 				}
 				else {
-					# uh oh, we should re-encode
+					push @xh,
+					'MIME-Version: 1.0',
+					"Content-Type: text/plain; charset=$author_encoding",
+					'Content-Transfer-Encoding: 8bit';
 				}
 			}
-			else {
-				push @xh,
-				  'MIME-Version: 1.0',
-				  "Content-Type: text/plain; charset=$author_encoding",
-				  'Content-Transfer-Encoding: 8bit';
-			}
 		}
-	}
 
-	$needs_confirm = (
-		$confirm eq "always" or
-		($confirm =~ /^(?:auto|cc)$/ && @cc) or
-		($confirm =~ /^(?:auto|compose)$/ && $compose && $message_num == 1));
-	$needs_confirm = "inform" if ($needs_confirm && $confirm_unconfigured && @cc);
+		$needs_confirm = (
+			$confirm eq "always" or
+			($confirm =~ /^(?:auto|cc)$/ && @cc) or
+			($confirm =~ /^(?:auto|compose)$/ && $compose && $message_num == 1));
+		$needs_confirm = "inform" if ($needs_confirm && $confirm_unconfigured && @cc);
 
-	@cc = (@initial_cc, @cc);
+		@cc = (@initial_cc, @cc);
 
-	send_message();
+		send_message();
 
-	# set up for the next message
-	if ($chain_reply_to || !defined $reply_to || length($reply_to) == 0) {
-		$reply_to = $message_id;
-		if (length $references > 0) {
-			$references .= "\n $message_id";
-		} else {
-			$references = "$message_id";
+		# set up for the next message
+		if ($chain_reply_to || !defined $reply_to || length($reply_to) == 0) {
+			$reply_to = $message_id;
+			if (length $references > 0) {
+				$references .= "\n $message_id";
+			} else {
+				$references = "$message_id";
+			}
 		}
+		$message_id = undef;
+		last unless (<F>);
 	}
-	$message_id = undef;
+	close F;
 }
 
 cleanup_compose_files();

-- 
wbr, Vitaly

Re: [PATCH] git-send-email: parse all messages in mbox

[Junio C Hamano]

Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:

> Currently git-send-email sends all mbox in one email. This seems to be wrong,
> because mbox can contain several messages. For example,
> `git format-patch --stdout' forms mbox file with all patches in it.
>
> This patch allows git send-email to send several messages from one mbox
> separately.

I suspect nobody would comment on it because with reindentation this was
extremely painful to review.

I _think_ the gist of your change is that (1) here you stop slurping the
body when you see a line that begins with "From "...

> +		# Now parse the message body
> +		while(<F>) {
> +			last if /^From /; # Next message in file

... and (2) the "last unless (<F>)" at the end of the loop where you
attempt to see if there is any more lines to be read from the stream (and
if so go back and continue from the header parsing).

> -	# set up for the next message
> -	if ($chain_reply_to || !defined $reply_to || length($reply_to) == 0) {
> -		$reply_to = $message_id;
> -		if (length $references > 0) {
> -			$references .= "\n $message_id";
> -		} else {
> -			$references = "$message_id";
> +		# set up for the next message
> +		if ($chain_reply_to || !defined $reply_to || length($reply_to) == 0) {
> +			$reply_to = $message_id;
> +			if (length $references > 0) {
> +				$references .= "\n $message_id";
> +			} else {
> +				$references = "$message_id";
> +			}
>  		}
> +		$message_id = undef;
> +		last unless (<F>);
>  	}
> -	$message_id = undef;
> +	close F;

But I think the code structure is wrong.  When you said "last if /^From /"
earlier, you have already read that line (and you do not unread it), and
here with "unless (<F>)" you are reading yet another line (one line after
the UNIX-From line) and discarding it.  The next round loses the real
RFC2822 header line you discarded with this unless (<F>), and begins from
the next line, and would break "First unfold multiline header fields"
logic among other things.

But this is only from reviewing a patch with reindentation noise so I
might have missed some subtle issues.

Can you make this into two patches for easier review?  One to split out
the existing loop for a single input stream into a helper function without
changing any behaviour (i.e. the loop reads everything to the end), and
then as a follow-up patch introduce "when we see a UNIX-From line we are
at the beginning of the next message so return early" logic to the helper?

IOW, after the two-patch series, the current main-loop may look
something like:

	my $unread_line = undef;
	while (1) {
		$unread_line = handle_one_stream(\*F, $unread_line);
                last if (!defined $unread_line);
	}
        close(\*F);

and your new handle_one_stream() sub will look something like:

	sub handle_one_stream {
		my ($fh, $last_line) = @_;
		local ($_);

		my $author = undef;
                my $author_encoding;
                my $has_content_type;
                my $body_encoding;
                @cc = ();
                @xh = ();
                my $input_format = undef;
                my @header = ();
                $message = "";
                $message_num++;

                # First unfold multiline header fields
                while (1) {
			if (defined $last_line) {
				$_ = $last_line;
                                $last_line = undef;
			} else {
                        	$_ = <F>;
			}
                        ...
		}
                # Now parse the header
                ...
                # Now parse the message body
                $last_line = undef;
                while (1) {
                	$_ = <F>;
			if (/^From /) {
				# This is the beginning of the
                                # next message; unread it.
				$last_line = $_;
                                last;
                        }
                        $message .= $_;
			...
		}
		return $last_line;
	}

Re: [PATCH] git-send-email: parse all messages in mbox

[Vitaly Mayatskikh]

At Tue, 12 May 2009 16:27:21 -0700, Junio C Hamano wrote:

Thanks for your review Junio! Once I'll found a free minute, I'll
refresh my almost forgotten Perl knowledge and try to deuglify the
patch.

-- 
wbr, Vitaly

Re: [PATCH] git-send-email: parse all messages in mbox

[Jay Soffian]

On Mon, May 11, 2009 at 9:46 AM, Vitaly Mayatskikh
<v.mayatskih@gmail.com> wrote:
> Currently git-send-email sends all mbox in one email. This seems to be wrong,
> because mbox can contain several messages. For example,
> `git format-patch --stdout' forms mbox file with all patches in it.

Just out of curiosity, what is the motivation for this patch?

By default, format-patch generates a file per commit, which is what
send-email currently expects. You can also specify one or more commits
to send-email directly, in which case it runs format-patch on your
behalf. And, send-email cannot handle messages on stdin, so "git
format-patch | git send-email" is not a valid workflow, even after
this patch.

So the only way I can see this being useful is if you're doing something like:

$ git format-patch --stdout > mbox
$ git send-email mbox

Or you can combine those in bash with:

$ git send-email <(git format-patch --stdout)

But, why? The only way this patch would make any sense to me is if it
_also_ extended send-email to read its messages from stdin.

j.

Re: [PATCH] git-send-email: parse all messages in mbox

[Vitaly Mayatskikh]

At Thu, 14 May 2009 10:48:33 -0400, Jay Soffian wrote:

> On Mon, May 11, 2009 at 9:46 AM, Vitaly Mayatskikh
> <v.mayatskih@gmail.com> wrote:
> > Currently git-send-email sends all mbox in one email. This seems to be wrong,
> > because mbox can contain several messages. For example,
> > `git format-patch --stdout' forms mbox file with all patches in it.
> 
> Just out of curiosity, what is the motivation for this patch?
> 
> By default, format-patch generates a file per commit, which is what
> send-email currently expects. You can also specify one or more commits
> to send-email directly, in which case it runs format-patch on your
> behalf. And, send-email cannot handle messages on stdin, so "git
> format-patch | git send-email" is not a valid workflow, even after
> this patch.
> 
> So the only way I can see this being useful is if you're doing something like:
> 
> $ git format-patch --stdout > mbox
> $ git send-email mbox
> 
> Or you can combine those in bash with:
> 
> $ git send-email <(git format-patch --stdout)
> 
> But, why? The only way this patch would make any sense to me is if it
> _also_ extended send-email to read its messages from stdin.

Because I want to edit cover letter, make --dry-run, etc. And it's
more handy to me to have one entity (file) then a dozen of separate
patches.

-- 
wbr, Vitaly