[PATCH] kernel: release ptraced tasks before zap_pid_ns_processes

[PATCH] kernel: release ptraced tasks before zap_pid_ns_processes

[Andrei Vagin]

Currently, exit_ptrace() adds all ptraced tasks in a dead list, than
zap_pid_ns_processes() waits all tasks in a current pid ns, and only
then tasks from the dead list are released.

zap_pid_ns_processes() can stuck on waiting tasks from the dead list. In
this case, we will have one unkillable process with one or more dead
children.

Cc: Oleg Nesterov <oleg@redhat.com>
Cc: "Eric W. Biederman" <ebiederm@xmission.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Andrei Vagin <avagin@gmail.com>
---
 kernel/exit.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/kernel/exit.c b/kernel/exit.c
index 0e21e6d21f35..ccaa6f6549ba 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -664,9 +664,6 @@ static void forget_original_parent(struct task_struct *father,
 {
 	struct task_struct *p, *t, *reaper;
 
-	if (unlikely(!list_empty(&father->ptraced)))
-		exit_ptrace(father, dead);
-
 	/* Can drop and reacquire tasklist_lock */
 	reaper = find_child_reaper(father);
 	if (list_empty(&father->children))
@@ -705,8 +702,18 @@ static void exit_notify(struct task_struct *tsk, int group_dead)
 	LIST_HEAD(dead);
 
 	write_lock_irq(&tasklist_lock);
-	forget_original_parent(tsk, &dead);
+	if (unlikely(!list_empty(&tsk->ptraced)))
+		exit_ptrace(tsk, &dead);
+	write_unlock_irq(&tasklist_lock);
+
+	/* Ptraced tasks have to be released before zap_pid_ns_processes(). */
+	list_for_each_entry_safe(p, n, &dead, ptrace_entry) {
+		list_del_init(&p->ptrace_entry);
+		release_task(p);
+	}
 
+	write_lock_irq(&tasklist_lock);
+	forget_original_parent(tsk, &dead);
 	if (group_dead)
 		kill_orphaned_pgrp(tsk->group_leader, NULL);
 
-- 
2.17.2

Re: [PATCH] kernel: release ptraced tasks before zap_pid_ns_processes

[Andrei Vagin]

[-- Attachment #1: Type: text/plain, Size: 2980 bytes --]

Hello,

The reproducer for this problem is attached to this message.

Below you can find its effect without this fix:

We have one alive process which stucks in
zap_pid_ns_processes:

$ ps axf 
... 
11831 pts/0    S      0:00 [ptrace_pidns] 
11833 pts/0    Zl     0:00  \_ [ptrace_pidns] <defunct> 
 
$ cat /proc/11831/stack  
[<0>] do_wait+0x1fa/0x2c0 
[<0>] kernel_wait4+0x9e/0x150 
[<0>] zap_pid_ns_processes+0x17d/0x270 
[<0>] do_exit+0xa15/0xbd0 
[<0>] do_group_exit+0x47/0xc0 
[<0>] get_signal+0x28c/0x850 
[<0>] do_signal+0x36/0x630 
[<0>] exit_to_usermode_loop+0x62/0xc0 
[<0>] prepare_exit_to_usermode+0xb4/0xe0 
[<0>] retint_user+0x8/0x18 
[<0>] 0xffffffffffffffff 


The child process has two threads which were ptraced by parent:
$ ls  /proc/11833/task/ 
11833  11834 

The parent can't wait the child, becase a thread group isn't empty, but
a thread is in the dead list:

$ cat /proc/1183{1,3,4}/status | grep '\(NSpid\|TracerPid\|State\)' 
State:  S (sleeping) 
TracerPid:      0 
NSpid:  11831   1 
State:  Z (zombie) 
TracerPid:      0 
NSpid:  11833   2 
State:  X (dead) 
TracerPid:      0 
NSpid:  11834   3

On Wed, Jan 02, 2019 at 12:59:39PM -0800, Andrei Vagin wrote:
> Currently, exit_ptrace() adds all ptraced tasks in a dead list, than
> zap_pid_ns_processes() waits all tasks in a current pid ns, and only
> then tasks from the dead list are released.
> 
> zap_pid_ns_processes() can stuck on waiting tasks from the dead list. In
> this case, we will have one unkillable process with one or more dead
> children.
> 
> Cc: Oleg Nesterov <oleg@redhat.com>
> Cc: "Eric W. Biederman" <ebiederm@xmission.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Signed-off-by: Andrei Vagin <avagin@gmail.com>
> ---
>  kernel/exit.c | 15 +++++++++++----
>  1 file changed, 11 insertions(+), 4 deletions(-)
> 
> diff --git a/kernel/exit.c b/kernel/exit.c
> index 0e21e6d21f35..ccaa6f6549ba 100644
> --- a/kernel/exit.c
> +++ b/kernel/exit.c
> @@ -664,9 +664,6 @@ static void forget_original_parent(struct task_struct *father,
>  {
>  	struct task_struct *p, *t, *reaper;
>  
> -	if (unlikely(!list_empty(&father->ptraced)))
> -		exit_ptrace(father, dead);
> -
>  	/* Can drop and reacquire tasklist_lock */
>  	reaper = find_child_reaper(father);
>  	if (list_empty(&father->children))
> @@ -705,8 +702,18 @@ static void exit_notify(struct task_struct *tsk, int group_dead)
>  	LIST_HEAD(dead);
>  
>  	write_lock_irq(&tasklist_lock);
> -	forget_original_parent(tsk, &dead);
> +	if (unlikely(!list_empty(&tsk->ptraced)))
> +		exit_ptrace(tsk, &dead);
> +	write_unlock_irq(&tasklist_lock);
> +
> +	/* Ptraced tasks have to be released before zap_pid_ns_processes(). */
> +	list_for_each_entry_safe(p, n, &dead, ptrace_entry) {
> +		list_del_init(&p->ptrace_entry);
> +		release_task(p);
> +	}
>  
> +	write_lock_irq(&tasklist_lock);
> +	forget_original_parent(tsk, &dead);
>  	if (group_dead)
>  		kill_orphaned_pgrp(tsk->group_leader, NULL);
>  
> -- 
> 2.17.2
> 

[-- Attachment #2: ptrace_pidns.c --]
[-- Type: text/plain, Size: 1096 bytes --]

#define _GNU_SOURCE         /* See feature_test_macros(7) */
#include <unistd.h>
#include <sys/ptrace.h>
#include <sys/types.h>
#include <sys/time.h>
#include <sys/resource.h>
#include <sys/wait.h>
#include <sys/syscall.h>   /* For SYS_xxx definitions */
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>


int pfd[2];
void *thread_fn()
{
	pid_t tid = syscall(SYS_gettid);
	write(pfd[1], &tid, sizeof(tid));
	sleep(1000);
	return NULL;
}

int main()
{
	pid_t pid, tid, ppid = getpid();
	pthread_t t;

	if (pipe(pfd))
		return 1;

	pid = fork();
	if (pid < 0)
		return 1;
	if (pid == 0) {
		pthread_create(&t, NULL, thread_fn, (void *)(unsigned long)ppid);
		sleep(1000);
		return 0;
	}
	printf("fork: %d\n", pid);

	if (read(pfd[0], &tid, sizeof(tid)) != sizeof(tid))
		return 1;
	printf("thread: %d\n", tid);
	if (ptrace(PTRACE_ATTACH, tid, 0, 0))
		return 1;
	if (wait4(tid, NULL, __WALL, NULL) != tid)
		return 1;
	if (ptrace(PTRACE_ATTACH, pid, 0, 0))
		return 1;
	if (wait4(pid, NULL, __WALL, NULL) != pid)
		return 1;

	kill(pid, SIGKILL);

	*((int *)(0)) = 0xdead;

	return 0;
}

Re: [PATCH] kernel: release ptraced tasks before zap_pid_ns_processes

[Andrei Vagin]

Oops, Al is here by mistake. Sorry about that.

On Wed, Jan 02, 2019 at 12:59:39PM -0800, Andrei Vagin wrote:
> Currently, exit_ptrace() adds all ptraced tasks in a dead list, than
> zap_pid_ns_processes() waits all tasks in a current pid ns, and only
> then tasks from the dead list are released.
> 
> zap_pid_ns_processes() can stuck on waiting tasks from the dead list. In
> this case, we will have one unkillable process with one or more dead
> children.
> 
> Cc: Oleg Nesterov <oleg@redhat.com>
> Cc: "Eric W. Biederman" <ebiederm@xmission.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Signed-off-by: Andrei Vagin <avagin@gmail.com>
> ---
>  kernel/exit.c | 15 +++++++++++----
>  1 file changed, 11 insertions(+), 4 deletions(-)
> 
> diff --git a/kernel/exit.c b/kernel/exit.c
> index 0e21e6d21f35..ccaa6f6549ba 100644
> --- a/kernel/exit.c
> +++ b/kernel/exit.c
> @@ -664,9 +664,6 @@ static void forget_original_parent(struct task_struct *father,
>  {
>  	struct task_struct *p, *t, *reaper;
>  
> -	if (unlikely(!list_empty(&father->ptraced)))
> -		exit_ptrace(father, dead);
> -
>  	/* Can drop and reacquire tasklist_lock */
>  	reaper = find_child_reaper(father);
>  	if (list_empty(&father->children))
> @@ -705,8 +702,18 @@ static void exit_notify(struct task_struct *tsk, int group_dead)
>  	LIST_HEAD(dead);
>  
>  	write_lock_irq(&tasklist_lock);
> -	forget_original_parent(tsk, &dead);
> +	if (unlikely(!list_empty(&tsk->ptraced)))
> +		exit_ptrace(tsk, &dead);
> +	write_unlock_irq(&tasklist_lock);
> +
> +	/* Ptraced tasks have to be released before zap_pid_ns_processes(). */
> +	list_for_each_entry_safe(p, n, &dead, ptrace_entry) {
> +		list_del_init(&p->ptrace_entry);
> +		release_task(p);
> +	}
>  
> +	write_lock_irq(&tasklist_lock);
> +	forget_original_parent(tsk, &dead);
>  	if (group_dead)
>  		kill_orphaned_pgrp(tsk->group_leader, NULL);
>  
> -- 
> 2.17.2
> 

Re: [PATCH] kernel: release ptraced tasks before zap_pid_ns_processes

[Oleg Nesterov]

Sorry for delay, vacation,

On 01/02, Andrei Vagin wrote:
>
> zap_pid_ns_processes() can stuck on waiting tasks from the dead list. In
> this case, we will have one unkillable process with one or more dead
> children.

Thanks!

> --- a/kernel/exit.c
> +++ b/kernel/exit.c
> @@ -664,9 +664,6 @@ static void forget_original_parent(struct task_struct *father,
>  {
>  	struct task_struct *p, *t, *reaper;
>  
> -	if (unlikely(!list_empty(&father->ptraced)))
> -		exit_ptrace(father, dead);
> -
>  	/* Can drop and reacquire tasklist_lock */
>  	reaper = find_child_reaper(father);
>  	if (list_empty(&father->children))
> @@ -705,8 +702,18 @@ static void exit_notify(struct task_struct *tsk, int group_dead)
>  	LIST_HEAD(dead);
>  
>  	write_lock_irq(&tasklist_lock);
> -	forget_original_parent(tsk, &dead);
> +	if (unlikely(!list_empty(&tsk->ptraced)))
> +		exit_ptrace(tsk, &dead);
> +	write_unlock_irq(&tasklist_lock);
> +
> +	/* Ptraced tasks have to be released before zap_pid_ns_processes(). */
> +	list_for_each_entry_safe(p, n, &dead, ptrace_entry) {
> +		list_del_init(&p->ptrace_entry);
> +		release_task(p);
> +	}
>  
> +	write_lock_irq(&tasklist_lock);
> +	forget_original_parent(tsk, &dead);
>  	if (group_dead)
>  		kill_orphaned_pgrp(tsk->group_leader, NULL);

How about a different fix below? It avoids additional write_lock/unlock(tasklist),
and another list_for_each_entry_safe(dead) loop is called only if it is actually
needed.

Or I missed something?

Oleg.


--- x/kernel/exit.c
+++ x/kernel/exit.c
@@ -558,12 +558,14 @@ static struct task_struct *find_alive_th
 	return NULL;
 }
 
-static struct task_struct *find_child_reaper(struct task_struct *father)
+static struct task_struct *find_child_reaper(struct task_struct *father,
+						struct list_head *dead)
 	__releases(&tasklist_lock)
 	__acquires(&tasklist_lock)
 {
 	struct pid_namespace *pid_ns = task_active_pid_ns(father);
 	struct task_struct *reaper = pid_ns->child_reaper;
+	struct task_struct *p, *n;
 
 	if (likely(reaper != father))
 		return reaper;
@@ -579,6 +581,11 @@ static struct task_struct *find_child_re
 		panic("Attempted to kill init! exitcode=0x%08x\n",
 			father->signal->group_exit_code ?: father->exit_code);
 	}
+
+	list_for_each_entry_safe(p, n, &dead, ptrace_entry) {
+		list_del_init(&p->ptrace_entry);
+		release_task(p);
+	}
 	zap_pid_ns_processes(pid_ns);
 	write_lock_irq(&tasklist_lock);
 
@@ -668,7 +675,7 @@ static void forget_original_parent(struc
 		exit_ptrace(father, dead);
 
 	/* Can drop and reacquire tasklist_lock */
-	reaper = find_child_reaper(father);
+	reaper = find_child_reaper(father, dead);
 	if (list_empty(&father->children))
 		return;
 

Re: [PATCH] kernel: release ptraced tasks before zap_pid_ns_processes

[Andrei Vagin]

On Tue, Jan 08, 2019 at 05:50:43PM +0100, Oleg Nesterov wrote:
> Sorry for delay, vacation,
> 
> On 01/02, Andrei Vagin wrote:
> >
> > zap_pid_ns_processes() can stuck on waiting tasks from the dead list. In
> > this case, we will have one unkillable process with one or more dead
> > children.
> 
> Thanks!
> 
> > --- a/kernel/exit.c
> > +++ b/kernel/exit.c
> > @@ -664,9 +664,6 @@ static void forget_original_parent(struct task_struct *father,
> >  {
> >  	struct task_struct *p, *t, *reaper;
> >  
> > -	if (unlikely(!list_empty(&father->ptraced)))
> > -		exit_ptrace(father, dead);
> > -
> >  	/* Can drop and reacquire tasklist_lock */
> >  	reaper = find_child_reaper(father);
> >  	if (list_empty(&father->children))
> > @@ -705,8 +702,18 @@ static void exit_notify(struct task_struct *tsk, int group_dead)
> >  	LIST_HEAD(dead);
> >  
> >  	write_lock_irq(&tasklist_lock);
> > -	forget_original_parent(tsk, &dead);
> > +	if (unlikely(!list_empty(&tsk->ptraced)))
> > +		exit_ptrace(tsk, &dead);
> > +	write_unlock_irq(&tasklist_lock);
> > +
> > +	/* Ptraced tasks have to be released before zap_pid_ns_processes(). */
> > +	list_for_each_entry_safe(p, n, &dead, ptrace_entry) {
> > +		list_del_init(&p->ptrace_entry);
> > +		release_task(p);
> > +	}
> >  
> > +	write_lock_irq(&tasklist_lock);
> > +	forget_original_parent(tsk, &dead);
> >  	if (group_dead)
> >  		kill_orphaned_pgrp(tsk->group_leader, NULL);
> 
> How about a different fix below? It avoids additional write_lock/unlock(tasklist),
> and another list_for_each_entry_safe(dead) loop is called only if it is actually
> needed.
> 
> Or I missed something?

No, you don't. The patch looks really nice. Thanks!

BTW: We probably need to add the "Fixes:" tag, but I am not sure to which
commit, it looks like the issue is here for years.

Acked-by: Andrei Vagin <avagin@gmail.com>

> 
> Oleg.
> 
> 
> --- x/kernel/exit.c
> +++ x/kernel/exit.c
> @@ -558,12 +558,14 @@ static struct task_struct *find_alive_th
>  	return NULL;
>  }
>  
> -static struct task_struct *find_child_reaper(struct task_struct *father)
> +static struct task_struct *find_child_reaper(struct task_struct *father,
> +						struct list_head *dead)
>  	__releases(&tasklist_lock)
>  	__acquires(&tasklist_lock)
>  {
>  	struct pid_namespace *pid_ns = task_active_pid_ns(father);
>  	struct task_struct *reaper = pid_ns->child_reaper;
> +	struct task_struct *p, *n;
>  
>  	if (likely(reaper != father))
>  		return reaper;
> @@ -579,6 +581,11 @@ static struct task_struct *find_child_re
>  		panic("Attempted to kill init! exitcode=0x%08x\n",
>  			father->signal->group_exit_code ?: father->exit_code);
>  	}
> +
> +	list_for_each_entry_safe(p, n, &dead, ptrace_entry) {
> +		list_del_init(&p->ptrace_entry);
> +		release_task(p);
> +	}
>  	zap_pid_ns_processes(pid_ns);
>  	write_lock_irq(&tasklist_lock);
>  
> @@ -668,7 +675,7 @@ static void forget_original_parent(struc
>  		exit_ptrace(father, dead);
>  
>  	/* Can drop and reacquire tasklist_lock */
> -	reaper = find_child_reaper(father);
> +	reaper = find_child_reaper(father, dead);
>  	if (list_empty(&father->children))
>  		return;
>  
> 

Re: [PATCH] kernel: release ptraced tasks before zap_pid_ns_processes

[Oleg Nesterov]

On 01/08, Andrei Vagin wrote:
>
> No, you don't. The patch looks really nice. Thanks!
>
> BTW: We probably need to add the "Fixes:" tag, but I am not sure to which
> commit, it looks like the issue is here for years.

I hate you ;) Ok, OK, this was broken by me in 7c8bd2322c7fd973d089b27de55e29c92c667a06
"exit: ptrace: shift "reap dead" code from exit_ptrace() to forget_original_parent()"

> Acked-by: Andrei Vagin <avagin@gmail.com>

Thanks, but if you agree with this patch may I ask you to send v2?

You found the real problem, my version is only slightly better than yours,
I don't want to steal the "From" tag. Feel free to add mine signed-of-by.

Oleg.

Re: [PATCH] kernel: release ptraced tasks before zap_pid_ns_processes

[Andrei Vagin]

On Wed, Jan 09, 2019 at 06:08:30PM +0100, Oleg Nesterov wrote:
> On 01/08, Andrei Vagin wrote:
> >
> > No, you don't. The patch looks really nice. Thanks!
> >
> > BTW: We probably need to add the "Fixes:" tag, but I am not sure to which
> > commit, it looks like the issue is here for years.
> 
> I hate you ;)

I know :)

> Ok, OK, this was broken by me in 7c8bd2322c7fd973d089b27de55e29c92c667a06
> "exit: ptrace: shift "reap dead" code from exit_ptrace() to forget_original_parent()"
> 
> > Acked-by: Andrei Vagin <avagin@gmail.com>
> 
> Thanks, but if you agree with this patch may I ask you to send v2?
> 
> You found the real problem, my version is only slightly better than yours,
> I don't want to steal the "From" tag. Feel free to add mine signed-of-by.

Thanks Oleg! I will send a second version of this patch.

> 
> Oleg.
>