[PATCH RESEND] exec: don't force_sigsegv processes with a pending fatal signal

[PATCH RESEND] exec: don't force_sigsegv processes with a pending fatal signal

[Ivan Delalande]

We were seeing unexplained segfaults in coreutils processes and other
basic utilities that we tracked down to binfmt_elf failing to load
segments for ld.so. Digging further, the actual problem seems to occur
when a process gets sigkilled while it is still being loaded by the
kernel. In our case when _do_page_fault goes for a retry it will return
early as it first checks for fatal_signal_pending(), so load_elf_interp
also returns with error and as a result search_binary_handler will
force_sigsegv() which is pretty confusing as nothing actually failed
here.

Fixes: 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()")
Reference: https://lkml.org/lkml/2013/2/14/5
Signed-off-by: Ivan Delalande <colona@arista.com>
---
 fs/exec.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/fs/exec.c b/fs/exec.c
index bdd0eacefdf5..6e8007edbb2d 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm)
 		if (retval < 0 && !bprm->mm) {
 			/* we got to flush_old_exec() and failed after it */
 			read_unlock(&binfmt_lock);
-			force_sigsegv(SIGSEGV, current);
+			if (!fatal_signal_pending(current))
+				force_sigsegv(SIGSEGV, current);
 			return retval;
 		}
 		if (retval != -ENOEXEC || !bprm->file) {
-- 
2.18.0

Re: [PATCH RESEND] exec: don't force_sigsegv processes with a pending fatal signal

[Dmitry Safonov]

Hi Ivan,

2018-07-31 1:56 GMT+01:00 Ivan Delalande <colona@arista.com>:
> We were seeing unexplained segfaults in coreutils processes and other
> basic utilities that we tracked down to binfmt_elf failing to load
> segments for ld.so. Digging further, the actual problem seems to occur
> when a process gets sigkilled while it is still being loaded by the
> kernel. In our case when _do_page_fault goes for a retry it will return
> early as it first checks for fatal_signal_pending(), so load_elf_interp
> also returns with error and as a result search_binary_handler will
> force_sigsegv() which is pretty confusing as nothing actually failed
> here.
>
> Fixes: 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()")
> Reference: https://lkml.org/lkml/2013/2/14/5
> Signed-off-by: Ivan Delalande <colona@arista.com>

+Cc: Oleg Nesterov <oleg@redhat.com>
+Cc: Andy Lutomirski <luto@kernel.org>

> ---
>  fs/exec.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/fs/exec.c b/fs/exec.c
> index bdd0eacefdf5..6e8007edbb2d 100644
> --- a/fs/exec.c
> +++ b/fs/exec.c
> @@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm)
>                 if (retval < 0 && !bprm->mm) {
>                         /* we got to flush_old_exec() and failed after it */
>                         read_unlock(&binfmt_lock);
> -                       force_sigsegv(SIGSEGV, current);
> +                       if (!fatal_signal_pending(current))
> +                               force_sigsegv(SIGSEGV, current);

I would suggest to add something like:
: if (print_fatal_signals)
:         pr_info("load_binary() failed: %d\n", retval);

It was interesting to catch that it actually segfaults during loading,
probably will save someone a couple of minutes too ;-)

>                         return retval;
>                 }
>                 if (retval != -ENOEXEC || !bprm->file) {

Thanks,
             Dmitry

Re: [PATCH RESEND] exec: don't force_sigsegv processes with a pending fatal signal

[Dmitry Safonov]

2018-08-02 20:53 GMT+01:00 Dmitry Safonov <0x7f454c46@gmail.com>:
> Hi Ivan,
>
> 2018-07-31 1:56 GMT+01:00 Ivan Delalande <colona@arista.com>:
>> We were seeing unexplained segfaults in coreutils processes and other
>> basic utilities that we tracked down to binfmt_elf failing to load
>> segments for ld.so. Digging further, the actual problem seems to occur
>> when a process gets sigkilled while it is still being loaded by the
>> kernel. In our case when _do_page_fault goes for a retry it will return
>> early as it first checks for fatal_signal_pending(), so load_elf_interp
>> also returns with error and as a result search_binary_handler will
>> force_sigsegv() which is pretty confusing as nothing actually failed
>> here.
>>
>> Fixes: 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()")
>> Reference: https://lkml.org/lkml/2013/2/14/5
>> Signed-off-by: Ivan Delalande <colona@arista.com>
>
> +Cc: Oleg Nesterov <oleg@redhat.com>
> +Cc: Andy Lutomirski <luto@kernel.org>

Also worth to add to commit message an example of previously
user-visible message in dmesg:

[ 1545.889604] potentially unexpected fatal signal 11.
[ 1545.889614] CPU: 2 PID: 7462 Comm: grep Tainted: P           O   3.18.28 #1
[ 1545.889617] Hardware name: Celestica D4040/D4040, BIOS 5.6.5 08/18/2016
[ 1545.889621] task: ffff880011282280 ti: ffff880100938000 task.ti:
ffff880100938000
[ 1545.889624] RIP: 0023:[<00000000f760eb70>]  [<00000000f760eb70>] 0xf760eb70
[ 1545.889641] RSP: 002b:00000000fffa3454  EFLAGS: 00000296
[ 1545.889644] RAX: fffffffffffffff2 RBX: 00000000f7a5c3e8 RCX: 00000000f7a5c718
[ 1545.889647] RDX: 00000000f7a5b230 RSI: 00000000f7a5c718 RDI: 00000000f757c000
[ 1545.889650] RBP: 00000000f7a5c3e8 R08: 0000000000000000 R09: 0000000000000000
[ 1545.889653] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 1545.889656] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 1545.889659] FS:  0000000000000000(0000) GS:ffff88017fb00000(0000)
knlGS:0000000000000000
[ 1545.889662] CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 1545.889665] CR2: 00000000f77d7838 CR3: 000000010bb90000 CR4: 00000000001007e0

(which now will be suppressed if there was a fatal signal)

>
>> ---
>>  fs/exec.c | 3 ++-
>>  1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/fs/exec.c b/fs/exec.c
>> index bdd0eacefdf5..6e8007edbb2d 100644
>> --- a/fs/exec.c
>> +++ b/fs/exec.c
>> @@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm)
>>                 if (retval < 0 && !bprm->mm) {
>>                         /* we got to flush_old_exec() and failed after it */
>>                         read_unlock(&binfmt_lock);
>> -                       force_sigsegv(SIGSEGV, current);
>> +                       if (!fatal_signal_pending(current))
>> +                               force_sigsegv(SIGSEGV, current);
>
> I would suggest to add something like:
> : if (print_fatal_signals)
> :         pr_info("load_binary() failed: %d\n", retval);
>
> It was interesting to catch that it actually segfaults during loading,
> probably will save someone a couple of minutes too ;-)

Not sure if it's easy to trigger, but it might require a ratelimit too..

>
>>                         return retval;
>>                 }
>>                 if (retval != -ENOEXEC || !bprm->file) {

-- 
             Dmitry

Re: [PATCH RESEND] exec: don't force_sigsegv processes with a pending fatal signal

[Oleg Nesterov]

On 08/02, Dmitry Safonov wrote:
>
> Hi Ivan,
>
> 2018-07-31 1:56 GMT+01:00 Ivan Delalande <colona@arista.com>:
> > We were seeing unexplained segfaults in coreutils processes and other
> > basic utilities that we tracked down to binfmt_elf failing to load
> > segments for ld.so. Digging further, the actual problem seems to occur
> > when a process gets sigkilled while it is still being loaded by the
> > kernel. In our case when _do_page_fault goes for a retry it will return
> > early as it first checks for fatal_signal_pending(), so load_elf_interp
> > also returns with error and as a result search_binary_handler will
> > force_sigsegv() which is pretty confusing as nothing actually failed
> > here.
> >
> > Fixes: 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()")
> > Reference: https://lkml.org/lkml/2013/2/14/5
> > Signed-off-by: Ivan Delalande <colona@arista.com>
>
> +Cc: Oleg Nesterov <oleg@redhat.com>
> +Cc: Andy Lutomirski <luto@kernel.org>

Thanks...

and sorry, I fail to understand the problem and what/how this patch tries to fix.

Hmm. After I read the next email from Dmitry it seems to me that the whole purpose
of this patch is to avoid print_fatal_signal()? If yes, the changelog should clearly
explain this.

> > --- a/fs/exec.c
> > +++ b/fs/exec.c
> > @@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm)
> >                 if (retval < 0 && !bprm->mm) {
> >                         /* we got to flush_old_exec() and failed after it */
> >                         read_unlock(&binfmt_lock);
> > -                       force_sigsegv(SIGSEGV, current);
> > +                       if (!fatal_signal_pending(current))
> > +                               force_sigsegv(SIGSEGV, current);

I won't argue, but may be force_sigsegv() should check fatal_signal_pending()
itself. setup_rt_frame() can too fail if fatal_signal_pending() by the same
reason.

Oleg.

Re: [PATCH RESEND] exec: don't force_sigsegv processes with a pending fatal signal

[Ivan Delalande]

Hi,

On Fri, Aug 03, 2018 at 03:39:24PM +0200, Oleg Nesterov wrote:
> On 08/02, Dmitry Safonov wrote:
> > 2018-07-31 1:56 GMT+01:00 Ivan Delalande <colona@arista.com>:
> > > We were seeing unexplained segfaults in coreutils processes and other
> > > basic utilities that we tracked down to binfmt_elf failing to load
> > > segments for ld.so. Digging further, the actual problem seems to occur
> > > when a process gets sigkilled while it is still being loaded by the
> > > kernel. In our case when _do_page_fault goes for a retry it will return
> > > early as it first checks for fatal_signal_pending(), so load_elf_interp
> > > also returns with error and as a result search_binary_handler will
> > > force_sigsegv() which is pretty confusing as nothing actually failed
> > > here.
> > >
> > > Fixes: 19d860a140be ("handle suicide on late failure exits in execve() in search_binary_handler()")
> > > Reference: https://lkml.org/lkml/2013/2/14/5
> > > Signed-off-by: Ivan Delalande <colona@arista.com>
> >
> > +Cc: Oleg Nesterov <oleg@redhat.com>
> > +Cc: Andy Lutomirski <luto@kernel.org>
> 
> Thanks...
> 
> and sorry, I fail to understand the problem and what/how this patch tries to fix.
> 
> Hmm. After I read the next email from Dmitry it seems to me that the whole purpose
> of this patch is to avoid print_fatal_signal()? If yes, the changelog should clearly
> explain this.

Sorry about that, yes this is purely to avoid printing the segfault
messages for these processes when they were in fact killed.
I'll definitely send a v2 to clarify that, and probably add the helpful
message Dimitry suggested as well.

> > > --- a/fs/exec.c
> > > +++ b/fs/exec.c
> > > @@ -1656,7 +1656,8 @@ int search_binary_handler(struct linux_binprm *bprm)
> > >                 if (retval < 0 && !bprm->mm) {
> > >                         /* we got to flush_old_exec() and failed after it */
> > >                         read_unlock(&binfmt_lock);
> > > -                       force_sigsegv(SIGSEGV, current);
> > > +                       if (!fatal_signal_pending(current))
> > > +                               force_sigsegv(SIGSEGV, current);
> 
> I won't argue, but may be force_sigsegv() should check fatal_signal_pending()
> itself. setup_rt_frame() can too fail if fatal_signal_pending() by the same
> reason.

I'm not sure, I think it would feel out of place in force_sigsegv() as
other callers might not expect this check in different contexts. I could
add a similar call to fatal_signal_pending() in signal_setup_done()
though, if you think we can hit the same problem from setup_rt_frame().

Thanks,
-- 
Ivan Delalande
Arista Networks