[PATCH] unblock and unignore SIGPIPE

[PATCH] unblock and unignore SIGPIPE

[Patrick Reynolds]

Blocked and ignored signals -- but not caught signals -- are inherited
across exec.  Some callers with sloppy signal-handling behavior can call
git with SIGPIPE blocked or ignored, even non-deterministically.  When
SIGPIPE is blocked or ignored, several git commands can run indefinitely,
ignoring EPIPE returns from write() calls, even when the process that
called them has gone away.  Our specific case involved a pipe of git
diff-tree output to a script that reads a limited amount of diff data.

In an ideal world, git would never be called with SIGPIPE blocked or
ignored.  But in the real world, several real potential callers, including
Perl, Apache, and Unicorn, sometimes spawn subprocesses with SIGPIPE
ignored.  It is easier and more productive to harden git against this
mistake than to clean it up in every potential parent process.

Signed-off-by: Patrick Reynolds <patrick.reynolds@github.com>
---
 cache.h            |  1 +
 git.c              |  5 +++++
 setup.c            | 11 +++++++++++
 t/t0012-sigpipe.sh | 27 +++++++++++++++++++++++++++
 4 files changed, 44 insertions(+)
 create mode 100755 t/t0012-sigpipe.sh

diff --git a/cache.h b/cache.h
index fcb511d..0a89fc1 100644
--- a/cache.h
+++ b/cache.h
@@ -463,6 +463,7 @@ extern int set_git_dir_init(const char *git_dir, const char *real_git_dir, int);
 extern int init_db(const char *template_dir, unsigned int flags);
 
 extern void sanitize_stdfds(void);
+extern void sanitize_signals(void);
 extern int daemonize(void);
 
 #define alloc_nr(x) (((x)+16)*3/2)
diff --git a/git.c b/git.c
index 9c49519..d6b221b 100644
--- a/git.c
+++ b/git.c
@@ -611,6 +611,11 @@ int main(int argc, char **av)
 	 */
 	sanitize_stdfds();
 
+	/*
+	 * Make sure we aren't ignoring or blocking SIGPIPE.
+	 */
+	sanitize_signals();
+
 	git_setup_gettext();
 
 	trace_command_performance(argv);
diff --git a/setup.c b/setup.c
index 0a22f8b..7aa4b01 100644
--- a/setup.c
+++ b/setup.c
@@ -865,3 +865,14 @@ int daemonize(void)
 	return 0;
 #endif
 }
+
+/* un-ignore and un-block SIGPIPE */
+void sanitize_signals(void)
+{
+	sigset_t unblock;
+
+	sigemptyset(&unblock);
+	sigaddset(&unblock, SIGPIPE);
+	sigprocmask(SIG_UNBLOCK, &unblock, NULL);
+	signal(SIGPIPE, SIG_DFL);
+}
diff --git a/t/t0012-sigpipe.sh b/t/t0012-sigpipe.sh
new file mode 100755
index 0000000..213cde3
--- /dev/null
+++ b/t/t0012-sigpipe.sh
@@ -0,0 +1,27 @@
+#!/bin/sh
+
+test_description='check handling of SIGPIPE'
+. ./test-lib.sh
+
+test_expect_success 'create blob' '
+	test-genrandom foo 16384 >file &&
+	git add file
+'
+
+large_git () {
+	for i in $(test_seq 1 100); do
+		git diff --staged --binary || return $?
+	done
+}
+
+test_expect_success 'git dies with SIGPIPE' '
+	OUT=$( ((large_git; echo $? 1>&3) | true) 3>&1 )
+	test "$OUT" -eq 141
+'
+
+test_expect_success 'git dies with SIGPIPE even if parent ignores it' '
+	OUT=$( ((trap "" PIPE; large_git; echo $? 1>&3) | true) 3>&1 )
+	test "$OUT" -eq 141
+'
+
+test_done
-- 
2.0.1

Re: [PATCH] unblock and unignore SIGPIPE

[Eric Wong]

Patrick Reynolds <patrick.reynolds@github.com> wrote:
> But in the real world, several real potential callers, including
> Perl, Apache, and Unicorn, sometimes spawn subprocesses with SIGPIPE
> ignored.

s/Unicorn/Ruby/

But unicorn would ignore SIGPIPE it if Ruby did not; relying on SIGPIPE
while doing any multiplexed I/O doesn't work well.

Re: [PATCH] unblock and unignore SIGPIPE

[Patrick Reynolds]

On Sun, Aug 17, 2014 at 8:14 PM, Eric Wong <normalperson@yhbt.net> wrote:
> But unicorn would ignore SIGPIPE it if Ruby did not; relying on SIGPIPE
> while doing any multiplexed I/O doesn't work well.

Exactly.  Callers block SIGPIPE for their own legitimate reasons, but they
don't consistently unblock it before spawning a git subprocess that needs
the default SIGPIPE behavior.  Easier to fix it in git than in every potential
caller.

--Patrick

Re: [PATCH] unblock and unignore SIGPIPE

[Junio C Hamano]

Patrick Reynolds <patrick.reynolds@github.com> writes:

> Blocked and ignored signals -- but not caught signals -- are inherited
> across exec.  Some callers with sloppy signal-handling behavior can call
> git with SIGPIPE blocked or ignored, even non-deterministically.  When
> SIGPIPE is blocked or ignored, several git commands can run indefinitely,
> ignoring EPIPE returns from write() calls, even when the process that
> called them has gone away.  Our specific case involved a pipe of git
> diff-tree output to a script that reads a limited amount of diff data.
>
> In an ideal world, git would never be called with SIGPIPE blocked or
> ignored.  But in the real world, several real potential callers, including
> Perl, Apache, and Unicorn, sometimes spawn subprocesses with SIGPIPE
> ignored.  It is easier and more productive to harden git against this
> mistake than to clean it up in every potential parent process.
>
> Signed-off-by: Patrick Reynolds <patrick.reynolds@github.com>

I missed this one when it was posted.

I have a suspicion that $gmane/256544 (relevant parties Cc'ed) is
cured by this (even though the other topic came much later than this
change)?

> diff --git a/git.c b/git.c
> index 9c49519..d6b221b 100644
> --- a/git.c
> +++ b/git.c
> @@ -611,6 +611,11 @@ int main(int argc, char **av)
>  	 */
>  	sanitize_stdfds();
>  
> +	/*
> +	 * Make sure we aren't ignoring or blocking SIGPIPE.
> +	 */
> +	sanitize_signals();
> +
>  	git_setup_gettext();
>  
>  	trace_command_performance(argv);
> diff --git a/setup.c b/setup.c
> index 0a22f8b..7aa4b01 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -865,3 +865,14 @@ int daemonize(void)
>  	return 0;
>  #endif
>  }
> +
> +/* un-ignore and un-block SIGPIPE */
> +void sanitize_signals(void)
> +{
> +	sigset_t unblock;
> +
> +	sigemptyset(&unblock);
> +	sigaddset(&unblock, SIGPIPE);
> +	sigprocmask(SIG_UNBLOCK, &unblock, NULL);
> +	signal(SIGPIPE, SIG_DFL);

With the only caller in git.c, there is not a good reason that we
would want to have this as a global in a different file (I think the
patch merely follows the pattern of sanitize-fds, but that one has
to be called from many places).

After making the function static to git.c, it would also be a good
idea to rename it to match the comment at the sole callsite,
e.g. "restore_sigpipe_to_default()" or something.  Then the comment
at the callsite can be removed.

Later somebody else may want to add other signals handled similarly
for whatever reason (I do not think of any signal or any good reason
at this moment).  But they will have to come up with much better
justification than "the function name says 'sanitize'" if we named
it to something specific to SIGPIPE.  And that good justification
will guide us what kind of signals need to be "sanitized" and find a
better verb than "sanitize", if it ever happens.

> diff --git a/t/t0012-sigpipe.sh b/t/t0012-sigpipe.sh
> new file mode 100755
> index 0000000..213cde3
> --- /dev/null
> +++ b/t/t0012-sigpipe.sh
> @@ -0,0 +1,27 @@
> +#!/bin/sh

Hmmm, do we really need to allocate a new test number just for these
two tests, instead of folding it into an existing one?

> +test_expect_success 'create blob' '
> +	test-genrandom foo 16384 >file &&
> +	git add file
> +'
> +
> +large_git () {
> +	for i in $(test_seq 1 100); do
> +		git diff --staged --binary || return $?

Write it more like this:

	for i in $(...)
        do
		git diff --cached --binary || return
	done

to (1) avoid unnecessary semicolon before "do", (2) prefer the true
option name over a synonym, and (3) omit unnecessary $? that is
given to "return".

Summing them up, something like this squashed in would give us a
good end result, perhaps?

---
 cache.h            |  1 -
 git.c              | 25 +++++++++++++++++++++----
 setup.c            | 11 -----------
 t/t0000-basic.sh   | 17 +++++++++++++++++
 t/t0012-sigpipe.sh | 27 ---------------------------
 5 files changed, 38 insertions(+), 43 deletions(-)
 delete mode 100755 t/t0012-sigpipe.sh

diff --git a/cache.h b/cache.h
index 0a89fc1..fcb511d 100644
--- a/cache.h
+++ b/cache.h
@@ -463,7 +463,6 @@ extern int set_git_dir_init(const char *git_dir, const char *real_git_dir, int);
 extern int init_db(const char *template_dir, unsigned int flags);
 
 extern void sanitize_stdfds(void);
-extern void sanitize_signals(void);
 extern int daemonize(void);
 
 #define alloc_nr(x) (((x)+16)*3/2)
diff --git a/git.c b/git.c
index d6b221b..c87bacd 100644
--- a/git.c
+++ b/git.c
@@ -592,6 +592,26 @@ static int run_argv(int *argcp, const char ***argv)
 	return done_alias;
 }
 
+/*
+ * Many parts of Git have subprograms communicate via pipe, expect the
+ * upstream of the pipe to die with SIGPIPE and the downstream process
+ * even knows to check and handle EPIPE correctly.  Some third-party
+ * programs that ignore or block SIGPIPE for their own reason forget
+ * to restore SIGPIPE handling to the default before spawning Git and
+ * break this carefully orchestrated machinery.
+ *
+ * Restore the way SIGPIPE is handled to default, which is what we
+ * expect.
+ */
+static void restore_sigpipe_to_default(void)
+{
+	sigset_t unblock;
+
+	sigemptyset(&unblock);
+	sigaddset(&unblock, SIGPIPE);
+	sigprocmask(SIG_UNBLOCK, &unblock, NULL);
+	signal(SIGPIPE, SIG_DFL);
+}
 
 int main(int argc, char **av)
 {
@@ -611,10 +631,7 @@ int main(int argc, char **av)
 	 */
 	sanitize_stdfds();
 
-	/*
-	 * Make sure we aren't ignoring or blocking SIGPIPE.
-	 */
-	sanitize_signals();
+	restore_sigpipe_to_default();
 
 	git_setup_gettext();
 
diff --git a/setup.c b/setup.c
index 7aa4b01..0a22f8b 100644
--- a/setup.c
+++ b/setup.c
@@ -865,14 +865,3 @@ int daemonize(void)
 	return 0;
 #endif
 }
-
-/* un-ignore and un-block SIGPIPE */
-void sanitize_signals(void)
-{
-	sigset_t unblock;
-
-	sigemptyset(&unblock);
-	sigaddset(&unblock, SIGPIPE);
-	sigprocmask(SIG_UNBLOCK, &unblock, NULL);
-	signal(SIGPIPE, SIG_DFL);
-}
diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh
index f10ba4a..21a0b19 100755
--- a/t/t0000-basic.sh
+++ b/t/t0000-basic.sh
@@ -1063,4 +1063,21 @@ test_expect_success 'very long name in the index handled sanely' '
 	test $len = 4098
 '
 
+large_git () {
+	for i in $(test_seq 1 100)
+	do
+		git ls-files -s || return
+	done
+}
+
+test_expect_success 'a constipated git dies with SIGPIPE' '
+	OUT=$( ((large_git; echo $? 1>&3) | :) 3>&1 )
+	test "$OUT" -eq 141
+'
+
+test_expect_success 'a constipated git dies with SIGPIPE even if parent ignores it' '
+	OUT=$( ((trap "" PIPE; large_git; echo $? 1>&3) | :) 3>&1 )
+	test "$OUT" -eq 141
+'
+
 test_done
diff --git a/t/t0012-sigpipe.sh b/t/t0012-sigpipe.sh
deleted file mode 100755
index 213cde3..0000000
--- a/t/t0012-sigpipe.sh
+++ /dev/null
@@ -1,27 +0,0 @@
-#!/bin/sh
-
-test_description='check handling of SIGPIPE'
-. ./test-lib.sh
-
-test_expect_success 'create blob' '
-	test-genrandom foo 16384 >file &&
-	git add file
-'
-
-large_git () {
-	for i in $(test_seq 1 100); do
-		git diff --staged --binary || return $?
-	done
-}
-
-test_expect_success 'git dies with SIGPIPE' '
-	OUT=$( ((large_git; echo $? 1>&3) | true) 3>&1 )
-	test "$OUT" -eq 141
-'
-
-test_expect_success 'git dies with SIGPIPE even if parent ignores it' '
-	OUT=$( ((trap "" PIPE; large_git; echo $? 1>&3) | true) 3>&1 )
-	test "$OUT" -eq 141
-'
-
-test_done
-- 
2.1.0-403-g099cf47

Re: [PATCH] unblock and unignore SIGPIPE

[Jeff King]

On Tue, Sep 16, 2014 at 02:43:43PM -0700, Junio C Hamano wrote:

> > +/* un-ignore and un-block SIGPIPE */
> > +void sanitize_signals(void)
> > +{
> > +	sigset_t unblock;
> > +
> > +	sigemptyset(&unblock);
> > +	sigaddset(&unblock, SIGPIPE);
> > +	sigprocmask(SIG_UNBLOCK, &unblock, NULL);
> > +	signal(SIGPIPE, SIG_DFL);
> 
> With the only caller in git.c, there is not a good reason that we
> would want to have this as a global in a different file (I think the
> patch merely follows the pattern of sanitize-fds, but that one has
> to be called from many places).

Would we want to call it from external C commands, too? For the most
part, git.c is the entry point for running git commands, and any
sanitizing it does will be inherited by sub-commands. But it _is_ still
legal to call dashed commands individually, and even required in some
cases (e.g., git-upload-pack for ssh clients).

> > diff --git a/t/t0012-sigpipe.sh b/t/t0012-sigpipe.sh
> > new file mode 100755
> > index 0000000..213cde3
> > --- /dev/null
> > +++ b/t/t0012-sigpipe.sh
> > @@ -0,0 +1,27 @@
> > +#!/bin/sh
> 
> Hmmm, do we really need to allocate a new test number just for these
> two tests, instead of folding it into an existing one?

I see in your proposed patch below you put them into t0000. I wonder if
t0005 would be a more obvious place.

-Peff

Re: [PATCH] unblock and unignore SIGPIPE

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> I see in your proposed patch below you put them into t0000. I wonder if
> t0005 would be a more obvious place.

Yup.  That is a good idea.

Re: [PATCH] unblock and unignore SIGPIPE

[Patrick Reynolds]

On Wed, Sep 17, 2014 at 3:11 AM, Jeff King <peff@peff.net> wrote:
> Would we want to call it from external C commands, too? For the most
> part, git.c is the entry point for running git commands, and any
> sanitizing it does will be inherited by sub-commands. But it _is_ still
> legal to call dashed commands individually, and even required in some
> cases (e.g., git-upload-pack for ssh clients).

git-upload-pack is protected pretty well from SIGPIPE shenanigans,
because its stdout all goes through write_or_die, as of cdf4fb8.  We
did, long ago, have some EPIPE problems with upload-pack and SSH
clients, but it all predates cdf4fb8.

So I think it's redundant to unblock SIGPIPE in git-upload-pack.

I'll tidy up as Junio recommended, recheck the tests, and submit
an updated patch shortly.

--Patrick

Re: [PATCH] unblock and unignore SIGPIPE

[Junio C Hamano]

Thanks; just to save time, you may want to look at what has already
been queued on 'pu'.


On Thu, Sep 18, 2014 at 7:35 AM, Patrick Reynolds <piki@github.com> wrote:
> On Wed, Sep 17, 2014 at 3:11 AM, Jeff King <peff@peff.net> wrote:
>> Would we want to call it from external C commands, too? For the most
>> part, git.c is the entry point for running git commands, and any
>> sanitizing it does will be inherited by sub-commands. But it _is_ still
>> legal to call dashed commands individually, and even required in some
>> cases (e.g., git-upload-pack for ssh clients).
>
> git-upload-pack is protected pretty well from SIGPIPE shenanigans,
> because its stdout all goes through write_or_die, as of cdf4fb8.  We
> did, long ago, have some EPIPE problems with upload-pack and SSH
> clients, but it all predates cdf4fb8.
>
> So I think it's redundant to unblock SIGPIPE in git-upload-pack.
>
> I'll tidy up as Junio recommended, recheck the tests, and submit
> an updated patch shortly.
>
> --Patrick