[PATCH v1 1/2] Add polling support to pidfd

[PATCH v1 1/2] Add polling support to pidfd

[joel]

pidfd are file descriptors referring to a process created with the
CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs pidfd
polling support to replace code that currently checks for existence of
/proc/pid for knowing that a process that is signalled to be killed has
died, which is both racy and slow. The pidfd poll approach is race-free,
and also allows the LMK to do other things (such as by polling on other
fds) while awaiting the process being killed to die.

It prevents a situation where a PID is reused between when LMK sends a
kill signal and checks for existence of the PID, since the wrong PID is
now possibly checked for existence.

In this patch, we follow the same existing mechanism in the kernel used
when the parent of the task group is to be notified (do_notify_parent).
This is when the tasks waiting on a poll of pidfd are also awakened.

We have decided to include the waitqueue in struct pid for the following
reasons:
1. The wait queue has to survive for the lifetime of the poll. Including
it in task_struct would not be option in this case because the task can
be reaped and destroyed before the poll returns.

2. By including the struct pid for the waitqueue means that during
de_thread(), the new thread group leader automatically gets the new
waitqueue/pid even though its task_struct is different.

Appropriate test cases are added in the second patch to provide coverage
of all the cases the patch is handling.

Andy had a similar patch [1] in the past which was a good reference
however this patch tries to handle different situations properly related
to thread group existence, and how/where it notifies. And also solves
other bugs (waitqueue lifetime).  Daniel had a similar patch [2]
recently which this patch supercedes.

[1] https://lore.kernel.org/patchwork/patch/345098/
[2] https://lore.kernel.org/lkml/20181029175322.189042-1-dancol at google.com/

Cc: luto at amacapital.net
Cc: rostedt at goodmis.org
Cc: dancol at google.com
Cc: sspatil at google.com
Cc: christian at brauner.io
Cc: jannh at google.com
Cc: surenb at google.com
Cc: timmurray at google.com
Cc: Jonathan Kowalski <bl0pbl33p at gmail.com>
Cc: torvalds at linux-foundation.org
Cc: kernel-team at android.com
Co-developed-by: Daniel Colascione <dancol at google.com>
Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>

---

RFC -> v1:
* Based on CLONE_PIDFD patches: https://lwn.net/Articles/786244/
* Updated selftests.
* Renamed poll wake function to do_notify_pidfd.
* Removed depending on EXIT flags
* Removed POLLERR flag since semantics are controversial and
  we don't have usecases for it right now (later we can add if there's
  a need for it).

 include/linux/pid.h |  3 +++
 kernel/fork.c       | 33 +++++++++++++++++++++++++++++++++
 kernel/pid.c        |  2 ++
 kernel/signal.c     | 14 ++++++++++++++
 4 files changed, 52 insertions(+)

diff --git a/include/linux/pid.h b/include/linux/pid.h
index 3c8ef5a199ca..1484db6ca8d1 100644
--- a/include/linux/pid.h
+++ b/include/linux/pid.h
@@ -3,6 +3,7 @@
 #define _LINUX_PID_H
 
 #include <linux/rculist.h>
+#include <linux/wait.h>
 
 enum pid_type
 {
@@ -60,6 +61,8 @@ struct pid
 	unsigned int level;
 	/* lists of tasks that use this pid */
 	struct hlist_head tasks[PIDTYPE_MAX];
+	/* wait queue for pidfd notifications */
+	wait_queue_head_t wait_pidfd;
 	struct rcu_head rcu;
 	struct upid numbers[1];
 };
diff --git a/kernel/fork.c b/kernel/fork.c
index 5525837ed80e..fb3b614f6456 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -1685,8 +1685,41 @@ static void pidfd_show_fdinfo(struct seq_file *m, struct file *f)
 }
 #endif
 
+static unsigned int pidfd_poll(struct file *file, struct poll_table_struct *pts)
+{
+	struct task_struct *task;
+	struct pid *pid;
+	int poll_flags = 0;
+
+	/*
+	 * tasklist_lock must be held because to avoid racing with
+	 * changes in exit_state and wake up. Basically to avoid:
+	 *
+	 * P0: read exit_state = 0
+	 * P1: write exit_state = EXIT_DEAD
+	 * P1: Do a wake up - wq is empty, so do nothing
+	 * P0: Queue for polling - wait forever.
+	 */
+	read_lock(&tasklist_lock);
+	pid = file->private_data;
+	task = pid_task(pid, PIDTYPE_PID);
+	WARN_ON_ONCE(task && !thread_group_leader(task));
+
+	if (!task || (task->exit_state && thread_group_empty(task)))
+		poll_flags = POLLIN | POLLRDNORM;
+
+	if (!poll_flags)
+		poll_wait(file, &pid->wait_pidfd, pts);
+
+	read_unlock(&tasklist_lock);
+
+	return poll_flags;
+}
+
+
 const struct file_operations pidfd_fops = {
 	.release = pidfd_release,
+	.poll = pidfd_poll,
 #ifdef CONFIG_PROC_FS
 	.show_fdinfo = pidfd_show_fdinfo,
 #endif
diff --git a/kernel/pid.c b/kernel/pid.c
index 20881598bdfa..5c90c239242f 100644
--- a/kernel/pid.c
+++ b/kernel/pid.c
@@ -214,6 +214,8 @@ struct pid *alloc_pid(struct pid_namespace *ns)
 	for (type = 0; type < PIDTYPE_MAX; ++type)
 		INIT_HLIST_HEAD(&pid->tasks[type]);
 
+	init_waitqueue_head(&pid->wait_pidfd);
+
 	upid = pid->numbers + ns->level;
 	spin_lock_irq(&pidmap_lock);
 	if (!(ns->pid_allocated & PIDNS_ADDING))
diff --git a/kernel/signal.c b/kernel/signal.c
index 1581140f2d99..16e7718316e5 100644
--- a/kernel/signal.c
+++ b/kernel/signal.c
@@ -1800,6 +1800,17 @@ int send_sigqueue(struct sigqueue *q, struct pid *pid, enum pid_type type)
 	return ret;
 }
 
+static void do_notify_pidfd(struct task_struct *task)
+{
+	struct pid *pid;
+
+	lockdep_assert_held(&tasklist_lock);
+
+	pid = get_task_pid(task, PIDTYPE_PID);
+	wake_up_all(&pid->wait_pidfd);
+	put_pid(pid);
+}
+
 /*
  * Let a parent know about the death of a child.
  * For a stopped/continued status change, use do_notify_parent_cldstop instead.
@@ -1823,6 +1834,9 @@ bool do_notify_parent(struct task_struct *tsk, int sig)
 	BUG_ON(!tsk->ptrace &&
 	       (tsk->group_leader != tsk || !thread_group_empty(tsk)));
 
+	/* Wake up all pidfd waiters */
+	do_notify_pidfd(tsk);
+
 	if (sig != SIGCHLD) {
 		/*
 		 * This is only possible if parent == real_parent.
-- 
2.21.0.593.g511ec345e18-goog

[PATCH v1 1/2] Add polling support to pidfd

[Joel Fernandes (Google)]

pidfd are file descriptors referring to a process created with the
CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs pidfd
polling support to replace code that currently checks for existence of
/proc/pid for knowing that a process that is signalled to be killed has
died, which is both racy and slow. The pidfd poll approach is race-free,
and also allows the LMK to do other things (such as by polling on other
fds) while awaiting the process being killed to die.

It prevents a situation where a PID is reused between when LMK sends a
kill signal and checks for existence of the PID, since the wrong PID is
now possibly checked for existence.

In this patch, we follow the same existing mechanism in the kernel used
when the parent of the task group is to be notified (do_notify_parent).
This is when the tasks waiting on a poll of pidfd are also awakened.

We have decided to include the waitqueue in struct pid for the following
reasons:
1. The wait queue has to survive for the lifetime of the poll. Including
it in task_struct would not be option in this case because the task can
be reaped and destroyed before the poll returns.

2. By including the struct pid for the waitqueue means that during
de_thread(), the new thread group leader automatically gets the new
waitqueue/pid even though its task_struct is different.

Appropriate test cases are added in the second patch to provide coverage
of all the cases the patch is handling.

Andy had a similar patch [1] in the past which was a good reference
however this patch tries to handle different situations properly related
to thread group existence, and how/where it notifies. And also solves
other bugs (waitqueue lifetime).  Daniel had a similar patch [2]
recently which this patch supercedes.

[1] https://lore.kernel.org/patchwork/patch/345098/
[2] https://lore.kernel.org/lkml/20181029175322.189042-1-dancol at google.com/

Cc: luto at amacapital.net
Cc: rostedt at goodmis.org
Cc: dancol at google.com
Cc: sspatil at google.com
Cc: christian at brauner.io
Cc: jannh at google.com
Cc: surenb at google.com
Cc: timmurray at google.com
Cc: Jonathan Kowalski <bl0pbl33p at gmail.com>
Cc: torvalds at linux-foundation.org
Cc: kernel-team at android.com
Co-developed-by: Daniel Colascione <dancol at google.com>
Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>

---

RFC -> v1:
* Based on CLONE_PIDFD patches: https://lwn.net/Articles/786244/
* Updated selftests.
* Renamed poll wake function to do_notify_pidfd.
* Removed depending on EXIT flags
* Removed POLLERR flag since semantics are controversial and
  we don't have usecases for it right now (later we can add if there's
  a need for it).

 include/linux/pid.h |  3 +++
 kernel/fork.c       | 33 +++++++++++++++++++++++++++++++++
 kernel/pid.c        |  2 ++
 kernel/signal.c     | 14 ++++++++++++++
 4 files changed, 52 insertions(+)

diff --git a/include/linux/pid.h b/include/linux/pid.h
index 3c8ef5a199ca..1484db6ca8d1 100644
--- a/include/linux/pid.h
+++ b/include/linux/pid.h
@@ -3,6 +3,7 @@
 #define _LINUX_PID_H
 
 #include <linux/rculist.h>
+#include <linux/wait.h>
 
 enum pid_type
 {
@@ -60,6 +61,8 @@ struct pid
 	unsigned int level;
 	/* lists of tasks that use this pid */
 	struct hlist_head tasks[PIDTYPE_MAX];
+	/* wait queue for pidfd notifications */
+	wait_queue_head_t wait_pidfd;
 	struct rcu_head rcu;
 	struct upid numbers[1];
 };
diff --git a/kernel/fork.c b/kernel/fork.c
index 5525837ed80e..fb3b614f6456 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -1685,8 +1685,41 @@ static void pidfd_show_fdinfo(struct seq_file *m, struct file *f)
 }
 #endif
 
+static unsigned int pidfd_poll(struct file *file, struct poll_table_struct *pts)
+{
+	struct task_struct *task;
+	struct pid *pid;
+	int poll_flags = 0;
+
+	/*
+	 * tasklist_lock must be held because to avoid racing with
+	 * changes in exit_state and wake up. Basically to avoid:
+	 *
+	 * P0: read exit_state = 0
+	 * P1: write exit_state = EXIT_DEAD
+	 * P1: Do a wake up - wq is empty, so do nothing
+	 * P0: Queue for polling - wait forever.
+	 */
+	read_lock(&tasklist_lock);
+	pid = file->private_data;
+	task = pid_task(pid, PIDTYPE_PID);
+	WARN_ON_ONCE(task && !thread_group_leader(task));
+
+	if (!task || (task->exit_state && thread_group_empty(task)))
+		poll_flags = POLLIN | POLLRDNORM;
+
+	if (!poll_flags)
+		poll_wait(file, &pid->wait_pidfd, pts);
+
+	read_unlock(&tasklist_lock);
+
+	return poll_flags;
+}
+
+
 const struct file_operations pidfd_fops = {
 	.release = pidfd_release,
+	.poll = pidfd_poll,
 #ifdef CONFIG_PROC_FS
 	.show_fdinfo = pidfd_show_fdinfo,
 #endif
diff --git a/kernel/pid.c b/kernel/pid.c
index 20881598bdfa..5c90c239242f 100644
--- a/kernel/pid.c
+++ b/kernel/pid.c
@@ -214,6 +214,8 @@ struct pid *alloc_pid(struct pid_namespace *ns)
 	for (type = 0; type < PIDTYPE_MAX; ++type)
 		INIT_HLIST_HEAD(&pid->tasks[type]);
 
+	init_waitqueue_head(&pid->wait_pidfd);
+
 	upid = pid->numbers + ns->level;
 	spin_lock_irq(&pidmap_lock);
 	if (!(ns->pid_allocated & PIDNS_ADDING))
diff --git a/kernel/signal.c b/kernel/signal.c
index 1581140f2d99..16e7718316e5 100644
--- a/kernel/signal.c
+++ b/kernel/signal.c
@@ -1800,6 +1800,17 @@ int send_sigqueue(struct sigqueue *q, struct pid *pid, enum pid_type type)
 	return ret;
 }
 
+static void do_notify_pidfd(struct task_struct *task)
+{
+	struct pid *pid;
+
+	lockdep_assert_held(&tasklist_lock);
+
+	pid = get_task_pid(task, PIDTYPE_PID);
+	wake_up_all(&pid->wait_pidfd);
+	put_pid(pid);
+}
+
 /*
  * Let a parent know about the death of a child.
  * For a stopped/continued status change, use do_notify_parent_cldstop instead.
@@ -1823,6 +1834,9 @@ bool do_notify_parent(struct task_struct *tsk, int sig)
 	BUG_ON(!tsk->ptrace &&
 	       (tsk->group_leader != tsk || !thread_group_empty(tsk)));
 
+	/* Wake up all pidfd waiters */
+	do_notify_pidfd(tsk);
+
 	if (sig != SIGCHLD) {
 		/*
 		 * This is only possible if parent == real_parent.
-- 
2.21.0.593.g511ec345e18-goog

[PATCH v1 2/2] Add selftests for pidfd polling

[joel]

Other than verifying pidfd based polling, the tests make sure that
wait semantics are preserved with the pidfd poll. Notably the 2 cases:
1. If a thread group leader exits while threads still there, then no
   pidfd poll notifcation should happen.
2. If a non-thread group leader does an execve, then the thread group
   leader is signaled to exit and is replaced with the execing thread
   as the new leader, however the parent is not notified in this case.

Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>
---
 tools/testing/selftests/pidfd/Makefile     |   2 +-
 tools/testing/selftests/pidfd/pidfd_test.c | 198 +++++++++++++++++++++
 2 files changed, 199 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/pidfd/Makefile b/tools/testing/selftests/pidfd/Makefile
index deaf8073bc06..4b31c14f273c 100644
--- a/tools/testing/selftests/pidfd/Makefile
+++ b/tools/testing/selftests/pidfd/Makefile
@@ -1,4 +1,4 @@
-CFLAGS += -g -I../../../../usr/include/
+CFLAGS += -g -I../../../../usr/include/ -lpthread
 
 TEST_GEN_PROGS := pidfd_test
 
diff --git a/tools/testing/selftests/pidfd/pidfd_test.c b/tools/testing/selftests/pidfd/pidfd_test.c
index d59378a93782..e887f807645e 100644
--- a/tools/testing/selftests/pidfd/pidfd_test.c
+++ b/tools/testing/selftests/pidfd/pidfd_test.c
@@ -4,18 +4,42 @@
 #include <errno.h>
 #include <fcntl.h>
 #include <linux/types.h>
+#include <pthread.h>
 #include <sched.h>
 #include <signal.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 #include <syscall.h>
+#include <sys/epoll.h>
+#include <sys/mman.h>
 #include <sys/mount.h>
 #include <sys/wait.h>
+#include <time.h>
 #include <unistd.h>
 
 #include "../kselftest.h"
 
+#define CHILD_THREAD_MIN_WAIT 3 /* seconds */
+#define MAX_EVENTS 5
+#define __NR_pidfd_send_signal 424
+
+#ifndef CLONE_PIDFD
+#define CLONE_PIDFD 0x00001000
+#endif
+
+static pid_t pidfd_clone(int flags, int *pidfd, int (*fn)(void *))
+{
+	size_t stack_size = 1024;
+	char *stack[1024] = { 0 };
+
+#ifdef __ia64__
+	return __clone2(fn, stack, stack_size, flags | SIGCHLD, NULL, pidfd);
+#else
+	return clone(fn, stack + stack_size, flags | SIGCHLD, NULL, pidfd);
+#endif
+}
+
 static inline int sys_pidfd_send_signal(int pidfd, int sig, siginfo_t *info,
 					unsigned int flags)
 {
@@ -368,10 +392,184 @@ static int test_pidfd_send_signal_syscall_support(void)
 	return 0;
 }
 
+void *test_pidfd_poll_exec_thread(void *priv)
+{
+	char waittime[256];
+
+	ksft_print_msg("Child Thread: starting. pid %d tid %d ; and sleeping\n",
+			getpid(), syscall(SYS_gettid));
+	ksft_print_msg("Child Thread: doing exec of sleep\n");
+
+	sprintf(waittime, "%d", CHILD_THREAD_MIN_WAIT);
+	execl("/bin/sleep", "sleep", waittime, (char *)NULL);
+
+	ksft_print_msg("Child Thread: DONE. pid %d tid %d\n",
+			getpid(), syscall(SYS_gettid));
+	return NULL;
+}
+
+static int poll_pidfd(const char *test_name, int pidfd)
+{
+	int c;
+	int epoll_fd = epoll_create1(0);
+	struct epoll_event event, events[MAX_EVENTS];
+
+	if (epoll_fd == -1)
+		ksft_exit_fail_msg("%s test: Failed to create epoll file descriptor\n",
+				   test_name);
+
+	event.events = EPOLLIN;
+	event.data.fd = pidfd;
+
+	if (epoll_ctl(epoll_fd, EPOLL_CTL_ADD, pidfd, &event)) {
+		ksft_print_msg("%s test: Failed to add epoll file descriptor: Skipping\n",
+			       test_name);
+		_exit(PIDFD_SKIP);
+	}
+
+	c = epoll_wait(epoll_fd, events, MAX_EVENTS, 5000);
+	if (c != 1 || !(events[0].events & EPOLLIN))
+		ksft_exit_fail_msg("%s test: Unexpected epoll_wait result (c=%d, events=%x)\n",
+				   test_name, c, events[0].events);
+
+	close(epoll_fd);
+	return events[0].events;
+
+}
+
+static int child_poll_exec_test(void *args)
+{
+	pthread_t t1;
+
+	ksft_print_msg("Child (pidfd): starting. pid %d tid %d\n", getpid(),
+			syscall(SYS_gettid));
+	pthread_create(&t1, NULL, test_pidfd_poll_exec_thread, NULL);
+	/*
+	 * Exec in the non-leader thread will destroy the leader immediately.
+	 * If the wait in the parent returns too soon, the test fails.
+	 */
+	while (1)
+		;
+}
+
+int test_pidfd_poll_exec(int use_waitpid)
+{
+	int pid, pidfd = 0;
+	int status, ret;
+	pthread_t t1;
+	time_t prog_start = time(NULL);
+	const char *test_name = "pidfd_poll check for premature notification on child thread exec";
+
+	ksft_print_msg("Parent: pid: %d\n", getpid());
+	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_exec_test);
+
+	ksft_print_msg("Parent: Waiting for Child (%d) to complete.\n", pid);
+
+	if (use_waitpid) {
+		ret = waitpid(pid, &status, 0);
+		if (ret == -1)
+			ksft_print_msg("Parent: error\n");
+
+		if (ret == pid)
+			ksft_print_msg("Parent: Child process waited for.\n");
+	} else {
+		poll_pidfd(test_name, pidfd);
+	}
+
+	time_t prog_time = time(NULL) - prog_start;
+
+	ksft_print_msg("Time waited for child: %lu\n", prog_time);
+
+	close(pidfd);
+
+	if (prog_time < CHILD_THREAD_MIN_WAIT || prog_time > CHILD_THREAD_MIN_WAIT + 2)
+		ksft_exit_fail_msg("%s test: Failed\n", test_name);
+	else
+		ksft_test_result_pass("%s test: Passed\n", test_name);
+}
+
+void *test_pidfd_poll_leader_exit_thread(void *priv)
+{
+	char waittime[256];
+
+	ksft_print_msg("Child Thread: starting. pid %d tid %d ; and sleeping\n",
+			getpid(), syscall(SYS_gettid));
+	sleep(CHILD_THREAD_MIN_WAIT);
+	ksft_print_msg("Child Thread: DONE. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
+	return NULL;
+}
+
+static time_t *child_exit_secs;
+static int child_poll_leader_exit_test(void *args)
+{
+	pthread_t t1, t2;
+
+	ksft_print_msg("Child: starting. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
+	pthread_create(&t1, NULL, test_pidfd_poll_leader_exit_thread, NULL);
+	pthread_create(&t2, NULL, test_pidfd_poll_leader_exit_thread, NULL);
+
+	/*
+	 * glibc exit calls exit_group syscall, so explicity call exit only
+	 * so that only the group leader exits, leaving the threads alone.
+	 */
+	*child_exit_secs = time(NULL);
+	syscall(SYS_exit, 0);
+}
+
+int test_pidfd_poll_leader_exit(int use_waitpid)
+{
+	int pid, pidfd = 0;
+	int status, ret;
+	time_t prog_start = time(NULL);
+	const char *test_name = "pidfd_poll check for premature notification on non-empty"
+				"group leader exit";
+
+	child_exit_secs = mmap(NULL, sizeof *child_exit_secs, PROT_READ | PROT_WRITE,
+			MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+
+	ksft_print_msg("Parent: pid: %d\n", getpid());
+	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_leader_exit_test);
+
+	ksft_print_msg("Parent: Waiting for Child (%d) to complete.\n", pid);
+
+	if (use_waitpid) {
+		ret = waitpid(pid, &status, 0);
+		if (ret == -1)
+			ksft_print_msg("Parent: error\n");
+	} else {
+		/*
+		 * This sleep tests for the case where if the child exits, and is in
+		 * EXIT_ZOMBIE, but the thread group leader is non-empty, then the poll
+		 * doesn't prematurely return even though there are active threads
+		 */
+		sleep(1);
+		poll_pidfd(test_name, pidfd);
+	}
+
+	if (ret == pid)
+		ksft_print_msg("Parent: Child process waited for.\n");
+
+	time_t since_child_exit = time(NULL) - *child_exit_secs;
+
+	ksft_print_msg("Time since child exit: %lu\n", since_child_exit);
+
+	close(pidfd);
+
+	if (since_child_exit < CHILD_THREAD_MIN_WAIT ||
+			since_child_exit > CHILD_THREAD_MIN_WAIT + 2)
+		ksft_exit_fail_msg("%s test: Failed\n", test_name);
+	else
+		ksft_test_result_pass("%s test: Passed\n", test_name);
+}
+
 int main(int argc, char **argv)
 {
 	ksft_print_header();
 
+	test_pidfd_poll_exec(0);
+	test_pidfd_poll_exec(1);
+	test_pidfd_poll_leader_exit(0);
+	test_pidfd_poll_leader_exit(1);
 	test_pidfd_send_signal_syscall_support();
 	test_pidfd_send_signal_simple_success();
 	test_pidfd_send_signal_exited_fail();
-- 
2.21.0.593.g511ec345e18-goog

[PATCH v1 2/2] Add selftests for pidfd polling

[Joel Fernandes (Google)]

Other than verifying pidfd based polling, the tests make sure that
wait semantics are preserved with the pidfd poll. Notably the 2 cases:
1. If a thread group leader exits while threads still there, then no
   pidfd poll notifcation should happen.
2. If a non-thread group leader does an execve, then the thread group
   leader is signaled to exit and is replaced with the execing thread
   as the new leader, however the parent is not notified in this case.

Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>
---
 tools/testing/selftests/pidfd/Makefile     |   2 +-
 tools/testing/selftests/pidfd/pidfd_test.c | 198 +++++++++++++++++++++
 2 files changed, 199 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/pidfd/Makefile b/tools/testing/selftests/pidfd/Makefile
index deaf8073bc06..4b31c14f273c 100644
--- a/tools/testing/selftests/pidfd/Makefile
+++ b/tools/testing/selftests/pidfd/Makefile
@@ -1,4 +1,4 @@
-CFLAGS += -g -I../../../../usr/include/
+CFLAGS += -g -I../../../../usr/include/ -lpthread
 
 TEST_GEN_PROGS := pidfd_test
 
diff --git a/tools/testing/selftests/pidfd/pidfd_test.c b/tools/testing/selftests/pidfd/pidfd_test.c
index d59378a93782..e887f807645e 100644
--- a/tools/testing/selftests/pidfd/pidfd_test.c
+++ b/tools/testing/selftests/pidfd/pidfd_test.c
@@ -4,18 +4,42 @@
 #include <errno.h>
 #include <fcntl.h>
 #include <linux/types.h>
+#include <pthread.h>
 #include <sched.h>
 #include <signal.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 #include <syscall.h>
+#include <sys/epoll.h>
+#include <sys/mman.h>
 #include <sys/mount.h>
 #include <sys/wait.h>
+#include <time.h>
 #include <unistd.h>
 
 #include "../kselftest.h"
 
+#define CHILD_THREAD_MIN_WAIT 3 /* seconds */
+#define MAX_EVENTS 5
+#define __NR_pidfd_send_signal 424
+
+#ifndef CLONE_PIDFD
+#define CLONE_PIDFD 0x00001000
+#endif
+
+static pid_t pidfd_clone(int flags, int *pidfd, int (*fn)(void *))
+{
+	size_t stack_size = 1024;
+	char *stack[1024] = { 0 };
+
+#ifdef __ia64__
+	return __clone2(fn, stack, stack_size, flags | SIGCHLD, NULL, pidfd);
+#else
+	return clone(fn, stack + stack_size, flags | SIGCHLD, NULL, pidfd);
+#endif
+}
+
 static inline int sys_pidfd_send_signal(int pidfd, int sig, siginfo_t *info,
 					unsigned int flags)
 {
@@ -368,10 +392,184 @@ static int test_pidfd_send_signal_syscall_support(void)
 	return 0;
 }
 
+void *test_pidfd_poll_exec_thread(void *priv)
+{
+	char waittime[256];
+
+	ksft_print_msg("Child Thread: starting. pid %d tid %d ; and sleeping\n",
+			getpid(), syscall(SYS_gettid));
+	ksft_print_msg("Child Thread: doing exec of sleep\n");
+
+	sprintf(waittime, "%d", CHILD_THREAD_MIN_WAIT);
+	execl("/bin/sleep", "sleep", waittime, (char *)NULL);
+
+	ksft_print_msg("Child Thread: DONE. pid %d tid %d\n",
+			getpid(), syscall(SYS_gettid));
+	return NULL;
+}
+
+static int poll_pidfd(const char *test_name, int pidfd)
+{
+	int c;
+	int epoll_fd = epoll_create1(0);
+	struct epoll_event event, events[MAX_EVENTS];
+
+	if (epoll_fd == -1)
+		ksft_exit_fail_msg("%s test: Failed to create epoll file descriptor\n",
+				   test_name);
+
+	event.events = EPOLLIN;
+	event.data.fd = pidfd;
+
+	if (epoll_ctl(epoll_fd, EPOLL_CTL_ADD, pidfd, &event)) {
+		ksft_print_msg("%s test: Failed to add epoll file descriptor: Skipping\n",
+			       test_name);
+		_exit(PIDFD_SKIP);
+	}
+
+	c = epoll_wait(epoll_fd, events, MAX_EVENTS, 5000);
+	if (c != 1 || !(events[0].events & EPOLLIN))
+		ksft_exit_fail_msg("%s test: Unexpected epoll_wait result (c=%d, events=%x)\n",
+				   test_name, c, events[0].events);
+
+	close(epoll_fd);
+	return events[0].events;
+
+}
+
+static int child_poll_exec_test(void *args)
+{
+	pthread_t t1;
+
+	ksft_print_msg("Child (pidfd): starting. pid %d tid %d\n", getpid(),
+			syscall(SYS_gettid));
+	pthread_create(&t1, NULL, test_pidfd_poll_exec_thread, NULL);
+	/*
+	 * Exec in the non-leader thread will destroy the leader immediately.
+	 * If the wait in the parent returns too soon, the test fails.
+	 */
+	while (1)
+		;
+}
+
+int test_pidfd_poll_exec(int use_waitpid)
+{
+	int pid, pidfd = 0;
+	int status, ret;
+	pthread_t t1;
+	time_t prog_start = time(NULL);
+	const char *test_name = "pidfd_poll check for premature notification on child thread exec";
+
+	ksft_print_msg("Parent: pid: %d\n", getpid());
+	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_exec_test);
+
+	ksft_print_msg("Parent: Waiting for Child (%d) to complete.\n", pid);
+
+	if (use_waitpid) {
+		ret = waitpid(pid, &status, 0);
+		if (ret == -1)
+			ksft_print_msg("Parent: error\n");
+
+		if (ret == pid)
+			ksft_print_msg("Parent: Child process waited for.\n");
+	} else {
+		poll_pidfd(test_name, pidfd);
+	}
+
+	time_t prog_time = time(NULL) - prog_start;
+
+	ksft_print_msg("Time waited for child: %lu\n", prog_time);
+
+	close(pidfd);
+
+	if (prog_time < CHILD_THREAD_MIN_WAIT || prog_time > CHILD_THREAD_MIN_WAIT + 2)
+		ksft_exit_fail_msg("%s test: Failed\n", test_name);
+	else
+		ksft_test_result_pass("%s test: Passed\n", test_name);
+}
+
+void *test_pidfd_poll_leader_exit_thread(void *priv)
+{
+	char waittime[256];
+
+	ksft_print_msg("Child Thread: starting. pid %d tid %d ; and sleeping\n",
+			getpid(), syscall(SYS_gettid));
+	sleep(CHILD_THREAD_MIN_WAIT);
+	ksft_print_msg("Child Thread: DONE. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
+	return NULL;
+}
+
+static time_t *child_exit_secs;
+static int child_poll_leader_exit_test(void *args)
+{
+	pthread_t t1, t2;
+
+	ksft_print_msg("Child: starting. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
+	pthread_create(&t1, NULL, test_pidfd_poll_leader_exit_thread, NULL);
+	pthread_create(&t2, NULL, test_pidfd_poll_leader_exit_thread, NULL);
+
+	/*
+	 * glibc exit calls exit_group syscall, so explicity call exit only
+	 * so that only the group leader exits, leaving the threads alone.
+	 */
+	*child_exit_secs = time(NULL);
+	syscall(SYS_exit, 0);
+}
+
+int test_pidfd_poll_leader_exit(int use_waitpid)
+{
+	int pid, pidfd = 0;
+	int status, ret;
+	time_t prog_start = time(NULL);
+	const char *test_name = "pidfd_poll check for premature notification on non-empty"
+				"group leader exit";
+
+	child_exit_secs = mmap(NULL, sizeof *child_exit_secs, PROT_READ | PROT_WRITE,
+			MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+
+	ksft_print_msg("Parent: pid: %d\n", getpid());
+	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_leader_exit_test);
+
+	ksft_print_msg("Parent: Waiting for Child (%d) to complete.\n", pid);
+
+	if (use_waitpid) {
+		ret = waitpid(pid, &status, 0);
+		if (ret == -1)
+			ksft_print_msg("Parent: error\n");
+	} else {
+		/*
+		 * This sleep tests for the case where if the child exits, and is in
+		 * EXIT_ZOMBIE, but the thread group leader is non-empty, then the poll
+		 * doesn't prematurely return even though there are active threads
+		 */
+		sleep(1);
+		poll_pidfd(test_name, pidfd);
+	}
+
+	if (ret == pid)
+		ksft_print_msg("Parent: Child process waited for.\n");
+
+	time_t since_child_exit = time(NULL) - *child_exit_secs;
+
+	ksft_print_msg("Time since child exit: %lu\n", since_child_exit);
+
+	close(pidfd);
+
+	if (since_child_exit < CHILD_THREAD_MIN_WAIT ||
+			since_child_exit > CHILD_THREAD_MIN_WAIT + 2)
+		ksft_exit_fail_msg("%s test: Failed\n", test_name);
+	else
+		ksft_test_result_pass("%s test: Passed\n", test_name);
+}
+
 int main(int argc, char **argv)
 {
 	ksft_print_header();
 
+	test_pidfd_poll_exec(0);
+	test_pidfd_poll_exec(1);
+	test_pidfd_poll_leader_exit(0);
+	test_pidfd_poll_leader_exit(1);
 	test_pidfd_send_signal_syscall_support();
 	test_pidfd_send_signal_simple_success();
 	test_pidfd_send_signal_exited_fail();
-- 
2.21.0.593.g511ec345e18-goog

[PATCH v1 2/2] Add selftests for pidfd polling

[tycho]

On Thu, Apr 25, 2019 at 03:00:10PM -0400, Joel Fernandes (Google) wrote:
>
> +void *test_pidfd_poll_exec_thread(void *priv)

I think everything in this file can be static, there's this one and
3-4 below.

> +int test_pidfd_poll_exec(int use_waitpid)
> +{
> +	int pid, pidfd = 0;
> +	int status, ret;
> +	pthread_t t1;
> +	time_t prog_start = time(NULL);
> +	const char *test_name = "pidfd_poll check for premature notification on child thread exec";
> +
> +	ksft_print_msg("Parent: pid: %d\n", getpid());
> +	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_exec_test);

If pidfd_clone() fails here, I think things will go haywire below.

> +	ksft_print_msg("Parent: Waiting for Child (%d) to complete.\n", pid);
> +
> +	if (use_waitpid) {
> +		ret = waitpid(pid, &status, 0);
> +		if (ret == -1)
> +			ksft_print_msg("Parent: error\n");
> +
> +		if (ret == pid)
> +			ksft_print_msg("Parent: Child process waited for.\n");
> +	} else {
> +		poll_pidfd(test_name, pidfd);
> +	}
> +
> +	time_t prog_time = time(NULL) - prog_start;
> +
> +	ksft_print_msg("Time waited for child: %lu\n", prog_time);
> +
> +	close(pidfd);
> +
> +	if (prog_time < CHILD_THREAD_MIN_WAIT || prog_time > CHILD_THREAD_MIN_WAIT + 2)
> +		ksft_exit_fail_msg("%s test: Failed\n", test_name);
> +	else
> +		ksft_test_result_pass("%s test: Passed\n", test_name);
> +}
> +
> +void *test_pidfd_poll_leader_exit_thread(void *priv)
> +{
> +	char waittime[256];
> +
> +	ksft_print_msg("Child Thread: starting. pid %d tid %d ; and sleeping\n",
> +			getpid(), syscall(SYS_gettid));
> +	sleep(CHILD_THREAD_MIN_WAIT);
> +	ksft_print_msg("Child Thread: DONE. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
> +	return NULL;
> +}
> +
> +static time_t *child_exit_secs;
> +static int child_poll_leader_exit_test(void *args)
> +{
> +	pthread_t t1, t2;
> +
> +	ksft_print_msg("Child: starting. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
> +	pthread_create(&t1, NULL, test_pidfd_poll_leader_exit_thread, NULL);
> +	pthread_create(&t2, NULL, test_pidfd_poll_leader_exit_thread, NULL);
> +
> +	/*
> +	 * glibc exit calls exit_group syscall, so explicity call exit only
> +	 * so that only the group leader exits, leaving the threads alone.
> +	 */
> +	*child_exit_secs = time(NULL);
> +	syscall(SYS_exit, 0);
> +}
> +
> +int test_pidfd_poll_leader_exit(int use_waitpid)
> +{
> +	int pid, pidfd = 0;
> +	int status, ret;
> +	time_t prog_start = time(NULL);
> +	const char *test_name = "pidfd_poll check for premature notification on non-empty"
> +				"group leader exit";
> +
> +	child_exit_secs = mmap(NULL, sizeof *child_exit_secs, PROT_READ | PROT_WRITE,
> +			MAP_SHARED | MAP_ANONYMOUS, -1, 0);
> +
> +	ksft_print_msg("Parent: pid: %d\n", getpid());
> +	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_leader_exit_test);

Same problem here, I think.

Tycho

[PATCH v1 2/2] Add selftests for pidfd polling

[Tycho Andersen]

On Thu, Apr 25, 2019@03:00:10PM -0400, Joel Fernandes (Google) wrote:
>
> +void *test_pidfd_poll_exec_thread(void *priv)

I think everything in this file can be static, there's this one and
3-4 below.

> +int test_pidfd_poll_exec(int use_waitpid)
> +{
> +	int pid, pidfd = 0;
> +	int status, ret;
> +	pthread_t t1;
> +	time_t prog_start = time(NULL);
> +	const char *test_name = "pidfd_poll check for premature notification on child thread exec";
> +
> +	ksft_print_msg("Parent: pid: %d\n", getpid());
> +	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_exec_test);

If pidfd_clone() fails here, I think things will go haywire below.

> +	ksft_print_msg("Parent: Waiting for Child (%d) to complete.\n", pid);
> +
> +	if (use_waitpid) {
> +		ret = waitpid(pid, &status, 0);
> +		if (ret == -1)
> +			ksft_print_msg("Parent: error\n");
> +
> +		if (ret == pid)
> +			ksft_print_msg("Parent: Child process waited for.\n");
> +	} else {
> +		poll_pidfd(test_name, pidfd);
> +	}
> +
> +	time_t prog_time = time(NULL) - prog_start;
> +
> +	ksft_print_msg("Time waited for child: %lu\n", prog_time);
> +
> +	close(pidfd);
> +
> +	if (prog_time < CHILD_THREAD_MIN_WAIT || prog_time > CHILD_THREAD_MIN_WAIT + 2)
> +		ksft_exit_fail_msg("%s test: Failed\n", test_name);
> +	else
> +		ksft_test_result_pass("%s test: Passed\n", test_name);
> +}
> +
> +void *test_pidfd_poll_leader_exit_thread(void *priv)
> +{
> +	char waittime[256];
> +
> +	ksft_print_msg("Child Thread: starting. pid %d tid %d ; and sleeping\n",
> +			getpid(), syscall(SYS_gettid));
> +	sleep(CHILD_THREAD_MIN_WAIT);
> +	ksft_print_msg("Child Thread: DONE. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
> +	return NULL;
> +}
> +
> +static time_t *child_exit_secs;
> +static int child_poll_leader_exit_test(void *args)
> +{
> +	pthread_t t1, t2;
> +
> +	ksft_print_msg("Child: starting. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
> +	pthread_create(&t1, NULL, test_pidfd_poll_leader_exit_thread, NULL);
> +	pthread_create(&t2, NULL, test_pidfd_poll_leader_exit_thread, NULL);
> +
> +	/*
> +	 * glibc exit calls exit_group syscall, so explicity call exit only
> +	 * so that only the group leader exits, leaving the threads alone.
> +	 */
> +	*child_exit_secs = time(NULL);
> +	syscall(SYS_exit, 0);
> +}
> +
> +int test_pidfd_poll_leader_exit(int use_waitpid)
> +{
> +	int pid, pidfd = 0;
> +	int status, ret;
> +	time_t prog_start = time(NULL);
> +	const char *test_name = "pidfd_poll check for premature notification on non-empty"
> +				"group leader exit";
> +
> +	child_exit_secs = mmap(NULL, sizeof *child_exit_secs, PROT_READ | PROT_WRITE,
> +			MAP_SHARED | MAP_ANONYMOUS, -1, 0);
> +
> +	ksft_print_msg("Parent: pid: %d\n", getpid());
> +	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_leader_exit_test);

Same problem here, I think.

Tycho

[PATCH v1 2/2] Add selftests for pidfd polling

[christian]

On Thu, Apr 25, 2019 at 03:00:10PM -0400, Joel Fernandes (Google) wrote:
> Other than verifying pidfd based polling, the tests make sure that
> wait semantics are preserved with the pidfd poll. Notably the 2 cases:
> 1. If a thread group leader exits while threads still there, then no
>    pidfd poll notifcation should happen.
> 2. If a non-thread group leader does an execve, then the thread group
>    leader is signaled to exit and is replaced with the execing thread
>    as the new leader, however the parent is not notified in this case.
> 
> Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>
> ---
>  tools/testing/selftests/pidfd/Makefile     |   2 +-
>  tools/testing/selftests/pidfd/pidfd_test.c | 198 +++++++++++++++++++++
>  2 files changed, 199 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/testing/selftests/pidfd/Makefile b/tools/testing/selftests/pidfd/Makefile
> index deaf8073bc06..4b31c14f273c 100644
> --- a/tools/testing/selftests/pidfd/Makefile
> +++ b/tools/testing/selftests/pidfd/Makefile
> @@ -1,4 +1,4 @@
> -CFLAGS += -g -I../../../../usr/include/
> +CFLAGS += -g -I../../../../usr/include/ -lpthread
>  
>  TEST_GEN_PROGS := pidfd_test
>  
> diff --git a/tools/testing/selftests/pidfd/pidfd_test.c b/tools/testing/selftests/pidfd/pidfd_test.c
> index d59378a93782..e887f807645e 100644
> --- a/tools/testing/selftests/pidfd/pidfd_test.c
> +++ b/tools/testing/selftests/pidfd/pidfd_test.c
> @@ -4,18 +4,42 @@
>  #include <errno.h>
>  #include <fcntl.h>
>  #include <linux/types.h>
> +#include <pthread.h>
>  #include <sched.h>
>  #include <signal.h>
>  #include <stdio.h>
>  #include <stdlib.h>
>  #include <string.h>
>  #include <syscall.h>
> +#include <sys/epoll.h>
> +#include <sys/mman.h>
>  #include <sys/mount.h>
>  #include <sys/wait.h>
> +#include <time.h>
>  #include <unistd.h>
>  
>  #include "../kselftest.h"
>  
> +#define CHILD_THREAD_MIN_WAIT 3 /* seconds */
> +#define MAX_EVENTS 5
> +#define __NR_pidfd_send_signal 424

Should probably be ifndefed as well.

> +
> +#ifndef CLONE_PIDFD
> +#define CLONE_PIDFD 0x00001000
> +#endif
> +
> +static pid_t pidfd_clone(int flags, int *pidfd, int (*fn)(void *))
> +{
> +	size_t stack_size = 1024;
> +	char *stack[1024] = { 0 };
> +
> +#ifdef __ia64__
> +	return __clone2(fn, stack, stack_size, flags | SIGCHLD, NULL, pidfd);
> +#else
> +	return clone(fn, stack + stack_size, flags | SIGCHLD, NULL, pidfd);
> +#endif
> +}
> +
>  static inline int sys_pidfd_send_signal(int pidfd, int sig, siginfo_t *info,
>  					unsigned int flags)
>  {
> @@ -368,10 +392,184 @@ static int test_pidfd_send_signal_syscall_support(void)
>  	return 0;
>  }
>  
> +void *test_pidfd_poll_exec_thread(void *priv)
> +{
> +	char waittime[256];
> +
> +	ksft_print_msg("Child Thread: starting. pid %d tid %d ; and sleeping\n",
> +			getpid(), syscall(SYS_gettid));
> +	ksft_print_msg("Child Thread: doing exec of sleep\n");
> +
> +	sprintf(waittime, "%d", CHILD_THREAD_MIN_WAIT);

> +#define CHILD_THREAD_MIN_SLEEP "3" /* seconds */

Could also be

#define str(s) _str(s)
#define _str(s) #s
#define CHILD_THREAD_MIN_SLEEP 3

execl("/bin/sleep", "sleep", str(CHILD_THREAD_MIN_SLEEP), (char *)NULL);

getting rid of waittime, and snprintf().

> +	execl("/bin/sleep", "sleep", waittime, (char *)NULL);
> +
> +	ksft_print_msg("Child Thread: DONE. pid %d tid %d\n",
> +			getpid(), syscall(SYS_gettid));
> +	return NULL;
> +}
> +
> +static int poll_pidfd(const char *test_name, int pidfd)
> +{
> +	int c;
> +	int epoll_fd = epoll_create1(0);

You probably don't need the epoll_fd after an exec, so:
int epoll_fd = epoll_create1(EPOLL_CLOEXEC);

> +	struct epoll_event event, events[MAX_EVENTS];
> +
> +	if (epoll_fd == -1)
> +		ksft_exit_fail_msg("%s test: Failed to create epoll file descriptor\n",
> +				   test_name);

I think logging the errno is helpful here. 

> +
> +	event.events = EPOLLIN;
> +	event.data.fd = pidfd;
> +
> +	if (epoll_ctl(epoll_fd, EPOLL_CTL_ADD, pidfd, &event)) {
> +		ksft_print_msg("%s test: Failed to add epoll file descriptor: Skipping\n",
> +			       test_name);

I think logging the errno is helpful here. 

> +		_exit(PIDFD_SKIP);

Why do you skip when you can't add the pidfd to the epoll loop? Why
shouldn't this be a test failure?

> +	}
> +
> +	c = epoll_wait(epoll_fd, events, MAX_EVENTS, 5000);

Uhm 5000 timeout? Either do a -1 or something that is noticeably
shorter, please. :)

> +	if (c != 1 || !(events[0].events & EPOLLIN))
> +		ksft_exit_fail_msg("%s test: Unexpected epoll_wait result (c=%d, events=%x)\n",
> +				   test_name, c, events[0].events);

I think logging the errno is helpful here. 

> +
> +	close(epoll_fd);
> +	return events[0].events;
> +
> +}
> +
> +static int child_poll_exec_test(void *args)
> +{
> +	pthread_t t1;
> +
> +	ksft_print_msg("Child (pidfd): starting. pid %d tid %d\n", getpid(),
> +			syscall(SYS_gettid));
> +	pthread_create(&t1, NULL, test_pidfd_poll_exec_thread, NULL);
> +	/*
> +	 * Exec in the non-leader thread will destroy the leader immediately.
> +	 * If the wait in the parent returns too soon, the test fails.
> +	 */
> +	while (1)
> +		;

Wouldn't sleep(<some-value>) be better here or at least a:

while (true)
        sleep(<some-sensible-value);

instead of a busy loop?

> +}
> +
> +int test_pidfd_poll_exec(int use_waitpid)
> +{
> +	int pid, pidfd = 0;
> +	int status, ret;
> +	pthread_t t1;
> +	time_t prog_start = time(NULL);
> +	const char *test_name = "pidfd_poll check for premature notification on child thread exec";
> +
> +	ksft_print_msg("Parent: pid: %d\n", getpid());
> +	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_exec_test);

That needs to check for error aka
if (pid < 0)
I think Tycho mentioned this already.

> +
> +	ksft_print_msg("Parent: Waiting for Child (%d) to complete.\n", pid);
> +
> +	if (use_waitpid) {
> +		ret = waitpid(pid, &status, 0);
> +		if (ret == -1)
> +			ksft_print_msg("Parent: error\n");
> +
> +		if (ret == pid)
> +			ksft_print_msg("Parent: Child process waited for.\n");
> +	} else {
> +		poll_pidfd(test_name, pidfd);

Either make poll_pidfd() void or check the error value. One of the two.

> +	}
> +
> +	time_t prog_time = time(NULL) - prog_start;
> +
> +	ksft_print_msg("Time waited for child: %lu\n", prog_time);
> +
> +	close(pidfd);
> +
> +	if (prog_time < CHILD_THREAD_MIN_WAIT || prog_time > CHILD_THREAD_MIN_WAIT + 2)

This timing-based testing seems kinda odd to be honest. Can't we do
something better than this?

> +		ksft_exit_fail_msg("%s test: Failed\n", test_name);
> +	else
> +		ksft_test_result_pass("%s test: Passed\n", test_name);
> +}
> +
> +void *test_pidfd_poll_leader_exit_thread(void *priv)
> +{
> +	char waittime[256];

Unused variable

> +
> +	ksft_print_msg("Child Thread: starting. pid %d tid %d ; and sleeping\n",
> +			getpid(), syscall(SYS_gettid));
> +	sleep(CHILD_THREAD_MIN_WAIT);
> +	ksft_print_msg("Child Thread: DONE. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
> +	return NULL;
> +}
> +
> +static time_t *child_exit_secs;
> +static int child_poll_leader_exit_test(void *args)
> +{
> +	pthread_t t1, t2;
> +
> +	ksft_print_msg("Child: starting. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
> +	pthread_create(&t1, NULL, test_pidfd_poll_leader_exit_thread, NULL);
> +	pthread_create(&t2, NULL, test_pidfd_poll_leader_exit_thread, NULL);
> +
> +	/*
> +	 * glibc exit calls exit_group syscall, so explicity call exit only
> +	 * so that only the group leader exits, leaving the threads alone.
> +	 */
> +	*child_exit_secs = time(NULL);
> +	syscall(SYS_exit, 0);
> +}
> +
> +int test_pidfd_poll_leader_exit(int use_waitpid)

static

> +{
> +	int pid, pidfd = 0;
> +	int status, ret;
> +	time_t prog_start = time(NULL);
> +	const char *test_name = "pidfd_poll check for premature notification on non-empty"
> +				"group leader exit";
> +
> +	child_exit_secs = mmap(NULL, sizeof *child_exit_secs, PROT_READ | PROT_WRITE,
> +			MAP_SHARED | MAP_ANONYMOUS, -1, 0);

Error checking, please:

if (child_exit_secs == MAP_FAILED)

> +
> +	ksft_print_msg("Parent: pid: %d\n", getpid());
> +	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_leader_exit_test);

Error checking, please:

if (pid < 0)

> +
> +	ksft_print_msg("Parent: Waiting for Child (%d) to complete.\n", pid);
> +
> +	if (use_waitpid) {
> +		ret = waitpid(pid, &status, 0);
> +		if (ret == -1)
> +			ksft_print_msg("Parent: error\n");
> +	} else {
> +		/*
> +		 * This sleep tests for the case where if the child exits, and is in
> +		 * EXIT_ZOMBIE, but the thread group leader is non-empty, then the poll
> +		 * doesn't prematurely return even though there are active threads
> +		 */
> +		sleep(1);
> +		poll_pidfd(test_name, pidfd);

Make poll_pidfd() void or check error, please.

> +	}
> +
> +	if (ret == pid)
> +		ksft_print_msg("Parent: Child process waited for.\n");
> +
> +	time_t since_child_exit = time(NULL) - *child_exit_secs;
> +
> +	ksft_print_msg("Time since child exit: %lu\n", since_child_exit);
> +
> +	close(pidfd);
> +
> +	if (since_child_exit < CHILD_THREAD_MIN_WAIT ||
> +			since_child_exit > CHILD_THREAD_MIN_WAIT + 2)

This looks very magical. Especially without a comment. Now you add
random +2. Please comment it or better, come up with a non-timing
based test.

> +		ksft_exit_fail_msg("%s test: Failed\n", test_name);
> +	else
> +		ksft_test_result_pass("%s test: Passed\n", test_name);
> +}
> +
>  int main(int argc, char **argv)
>  {
>  	ksft_print_header();
>  
> +	test_pidfd_poll_exec(0);
> +	test_pidfd_poll_exec(1);
> +	test_pidfd_poll_leader_exit(0);
> +	test_pidfd_poll_leader_exit(1);
>  	test_pidfd_send_signal_syscall_support();
>  	test_pidfd_send_signal_simple_success();
>  	test_pidfd_send_signal_exited_fail();
> -- 
> 2.21.0.593.g511ec345e18-goog
> 

[PATCH v1 2/2] Add selftests for pidfd polling

[Christian Brauner]

On Thu, Apr 25, 2019@03:00:10PM -0400, Joel Fernandes (Google) wrote:
> Other than verifying pidfd based polling, the tests make sure that
> wait semantics are preserved with the pidfd poll. Notably the 2 cases:
> 1. If a thread group leader exits while threads still there, then no
>    pidfd poll notifcation should happen.
> 2. If a non-thread group leader does an execve, then the thread group
>    leader is signaled to exit and is replaced with the execing thread
>    as the new leader, however the parent is not notified in this case.
> 
> Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>
> ---
>  tools/testing/selftests/pidfd/Makefile     |   2 +-
>  tools/testing/selftests/pidfd/pidfd_test.c | 198 +++++++++++++++++++++
>  2 files changed, 199 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/testing/selftests/pidfd/Makefile b/tools/testing/selftests/pidfd/Makefile
> index deaf8073bc06..4b31c14f273c 100644
> --- a/tools/testing/selftests/pidfd/Makefile
> +++ b/tools/testing/selftests/pidfd/Makefile
> @@ -1,4 +1,4 @@
> -CFLAGS += -g -I../../../../usr/include/
> +CFLAGS += -g -I../../../../usr/include/ -lpthread
>  
>  TEST_GEN_PROGS := pidfd_test
>  
> diff --git a/tools/testing/selftests/pidfd/pidfd_test.c b/tools/testing/selftests/pidfd/pidfd_test.c
> index d59378a93782..e887f807645e 100644
> --- a/tools/testing/selftests/pidfd/pidfd_test.c
> +++ b/tools/testing/selftests/pidfd/pidfd_test.c
> @@ -4,18 +4,42 @@
>  #include <errno.h>
>  #include <fcntl.h>
>  #include <linux/types.h>
> +#include <pthread.h>
>  #include <sched.h>
>  #include <signal.h>
>  #include <stdio.h>
>  #include <stdlib.h>
>  #include <string.h>
>  #include <syscall.h>
> +#include <sys/epoll.h>
> +#include <sys/mman.h>
>  #include <sys/mount.h>
>  #include <sys/wait.h>
> +#include <time.h>
>  #include <unistd.h>
>  
>  #include "../kselftest.h"
>  
> +#define CHILD_THREAD_MIN_WAIT 3 /* seconds */
> +#define MAX_EVENTS 5
> +#define __NR_pidfd_send_signal 424

Should probably be ifndefed as well.

> +
> +#ifndef CLONE_PIDFD
> +#define CLONE_PIDFD 0x00001000
> +#endif
> +
> +static pid_t pidfd_clone(int flags, int *pidfd, int (*fn)(void *))
> +{
> +	size_t stack_size = 1024;
> +	char *stack[1024] = { 0 };
> +
> +#ifdef __ia64__
> +	return __clone2(fn, stack, stack_size, flags | SIGCHLD, NULL, pidfd);
> +#else
> +	return clone(fn, stack + stack_size, flags | SIGCHLD, NULL, pidfd);
> +#endif
> +}
> +
>  static inline int sys_pidfd_send_signal(int pidfd, int sig, siginfo_t *info,
>  					unsigned int flags)
>  {
> @@ -368,10 +392,184 @@ static int test_pidfd_send_signal_syscall_support(void)
>  	return 0;
>  }
>  
> +void *test_pidfd_poll_exec_thread(void *priv)
> +{
> +	char waittime[256];
> +
> +	ksft_print_msg("Child Thread: starting. pid %d tid %d ; and sleeping\n",
> +			getpid(), syscall(SYS_gettid));
> +	ksft_print_msg("Child Thread: doing exec of sleep\n");
> +
> +	sprintf(waittime, "%d", CHILD_THREAD_MIN_WAIT);

> +#define CHILD_THREAD_MIN_SLEEP "3" /* seconds */

Could also be

#define str(s) _str(s)
#define _str(s) #s
#define CHILD_THREAD_MIN_SLEEP 3

execl("/bin/sleep", "sleep", str(CHILD_THREAD_MIN_SLEEP), (char *)NULL);

getting rid of waittime, and snprintf().

> +	execl("/bin/sleep", "sleep", waittime, (char *)NULL);
> +
> +	ksft_print_msg("Child Thread: DONE. pid %d tid %d\n",
> +			getpid(), syscall(SYS_gettid));
> +	return NULL;
> +}
> +
> +static int poll_pidfd(const char *test_name, int pidfd)
> +{
> +	int c;
> +	int epoll_fd = epoll_create1(0);

You probably don't need the epoll_fd after an exec, so:
int epoll_fd = epoll_create1(EPOLL_CLOEXEC);

> +	struct epoll_event event, events[MAX_EVENTS];
> +
> +	if (epoll_fd == -1)
> +		ksft_exit_fail_msg("%s test: Failed to create epoll file descriptor\n",
> +				   test_name);

I think logging the errno is helpful here. 

> +
> +	event.events = EPOLLIN;
> +	event.data.fd = pidfd;
> +
> +	if (epoll_ctl(epoll_fd, EPOLL_CTL_ADD, pidfd, &event)) {
> +		ksft_print_msg("%s test: Failed to add epoll file descriptor: Skipping\n",
> +			       test_name);

I think logging the errno is helpful here. 

> +		_exit(PIDFD_SKIP);

Why do you skip when you can't add the pidfd to the epoll loop? Why
shouldn't this be a test failure?

> +	}
> +
> +	c = epoll_wait(epoll_fd, events, MAX_EVENTS, 5000);

Uhm 5000 timeout? Either do a -1 or something that is noticeably
shorter, please. :)

> +	if (c != 1 || !(events[0].events & EPOLLIN))
> +		ksft_exit_fail_msg("%s test: Unexpected epoll_wait result (c=%d, events=%x)\n",
> +				   test_name, c, events[0].events);

I think logging the errno is helpful here. 

> +
> +	close(epoll_fd);
> +	return events[0].events;
> +
> +}
> +
> +static int child_poll_exec_test(void *args)
> +{
> +	pthread_t t1;
> +
> +	ksft_print_msg("Child (pidfd): starting. pid %d tid %d\n", getpid(),
> +			syscall(SYS_gettid));
> +	pthread_create(&t1, NULL, test_pidfd_poll_exec_thread, NULL);
> +	/*
> +	 * Exec in the non-leader thread will destroy the leader immediately.
> +	 * If the wait in the parent returns too soon, the test fails.
> +	 */
> +	while (1)
> +		;

Wouldn't sleep(<some-value>) be better here or at least a:

while (true)
        sleep(<some-sensible-value);

instead of a busy loop?

> +}
> +
> +int test_pidfd_poll_exec(int use_waitpid)
> +{
> +	int pid, pidfd = 0;
> +	int status, ret;
> +	pthread_t t1;
> +	time_t prog_start = time(NULL);
> +	const char *test_name = "pidfd_poll check for premature notification on child thread exec";
> +
> +	ksft_print_msg("Parent: pid: %d\n", getpid());
> +	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_exec_test);

That needs to check for error aka
if (pid < 0)
I think Tycho mentioned this already.

> +
> +	ksft_print_msg("Parent: Waiting for Child (%d) to complete.\n", pid);
> +
> +	if (use_waitpid) {
> +		ret = waitpid(pid, &status, 0);
> +		if (ret == -1)
> +			ksft_print_msg("Parent: error\n");
> +
> +		if (ret == pid)
> +			ksft_print_msg("Parent: Child process waited for.\n");
> +	} else {
> +		poll_pidfd(test_name, pidfd);

Either make poll_pidfd() void or check the error value. One of the two.

> +	}
> +
> +	time_t prog_time = time(NULL) - prog_start;
> +
> +	ksft_print_msg("Time waited for child: %lu\n", prog_time);
> +
> +	close(pidfd);
> +
> +	if (prog_time < CHILD_THREAD_MIN_WAIT || prog_time > CHILD_THREAD_MIN_WAIT + 2)

This timing-based testing seems kinda odd to be honest. Can't we do
something better than this?

> +		ksft_exit_fail_msg("%s test: Failed\n", test_name);
> +	else
> +		ksft_test_result_pass("%s test: Passed\n", test_name);
> +}
> +
> +void *test_pidfd_poll_leader_exit_thread(void *priv)
> +{
> +	char waittime[256];

Unused variable

> +
> +	ksft_print_msg("Child Thread: starting. pid %d tid %d ; and sleeping\n",
> +			getpid(), syscall(SYS_gettid));
> +	sleep(CHILD_THREAD_MIN_WAIT);
> +	ksft_print_msg("Child Thread: DONE. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
> +	return NULL;
> +}
> +
> +static time_t *child_exit_secs;
> +static int child_poll_leader_exit_test(void *args)
> +{
> +	pthread_t t1, t2;
> +
> +	ksft_print_msg("Child: starting. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
> +	pthread_create(&t1, NULL, test_pidfd_poll_leader_exit_thread, NULL);
> +	pthread_create(&t2, NULL, test_pidfd_poll_leader_exit_thread, NULL);
> +
> +	/*
> +	 * glibc exit calls exit_group syscall, so explicity call exit only
> +	 * so that only the group leader exits, leaving the threads alone.
> +	 */
> +	*child_exit_secs = time(NULL);
> +	syscall(SYS_exit, 0);
> +}
> +
> +int test_pidfd_poll_leader_exit(int use_waitpid)

static

> +{
> +	int pid, pidfd = 0;
> +	int status, ret;
> +	time_t prog_start = time(NULL);
> +	const char *test_name = "pidfd_poll check for premature notification on non-empty"
> +				"group leader exit";
> +
> +	child_exit_secs = mmap(NULL, sizeof *child_exit_secs, PROT_READ | PROT_WRITE,
> +			MAP_SHARED | MAP_ANONYMOUS, -1, 0);

Error checking, please:

if (child_exit_secs == MAP_FAILED)

> +
> +	ksft_print_msg("Parent: pid: %d\n", getpid());
> +	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_leader_exit_test);

Error checking, please:

if (pid < 0)

> +
> +	ksft_print_msg("Parent: Waiting for Child (%d) to complete.\n", pid);
> +
> +	if (use_waitpid) {
> +		ret = waitpid(pid, &status, 0);
> +		if (ret == -1)
> +			ksft_print_msg("Parent: error\n");
> +	} else {
> +		/*
> +		 * This sleep tests for the case where if the child exits, and is in
> +		 * EXIT_ZOMBIE, but the thread group leader is non-empty, then the poll
> +		 * doesn't prematurely return even though there are active threads
> +		 */
> +		sleep(1);
> +		poll_pidfd(test_name, pidfd);

Make poll_pidfd() void or check error, please.

> +	}
> +
> +	if (ret == pid)
> +		ksft_print_msg("Parent: Child process waited for.\n");
> +
> +	time_t since_child_exit = time(NULL) - *child_exit_secs;
> +
> +	ksft_print_msg("Time since child exit: %lu\n", since_child_exit);
> +
> +	close(pidfd);
> +
> +	if (since_child_exit < CHILD_THREAD_MIN_WAIT ||
> +			since_child_exit > CHILD_THREAD_MIN_WAIT + 2)

This looks very magical. Especially without a comment. Now you add
random +2. Please comment it or better, come up with a non-timing
based test.

> +		ksft_exit_fail_msg("%s test: Failed\n", test_name);
> +	else
> +		ksft_test_result_pass("%s test: Passed\n", test_name);
> +}
> +
>  int main(int argc, char **argv)
>  {
>  	ksft_print_header();
>  
> +	test_pidfd_poll_exec(0);
> +	test_pidfd_poll_exec(1);
> +	test_pidfd_poll_leader_exit(0);
> +	test_pidfd_poll_leader_exit(1);
>  	test_pidfd_send_signal_syscall_support();
>  	test_pidfd_send_signal_simple_success();
>  	test_pidfd_send_signal_exited_fail();
> -- 
> 2.21.0.593.g511ec345e18-goog
> 

[PATCH v1 2/2] Add selftests for pidfd polling

[dancol]

On Thu, Apr 25, 2019 at 2:29 PM Christian Brauner <christian at brauner.io> wrote:
> This timing-based testing seems kinda odd to be honest. Can't we do
> something better than this?

Agreed. Timing-based tests have a substantial risk of becoming flaky.
We ought to be able to make these tests fully deterministic and not
subject to breakage from odd scheduling outcomes. We don't have
sleepable events for everything, granted, but sleep-waiting on a
condition with exponential backoff is fine in test code. In general,
if you start with a robust test, you can insert a sleep(100) anywhere
and not break the logic. Violating this rule always causes pain sooner
or later.

Other thoughts: IMHO, using poll(2) instead of epoll would simplify
the test code, and I think we can get away with calling
pthread_exit(3) instead of SYS_exit.

[PATCH v1 2/2] Add selftests for pidfd polling

[Daniel Colascione]

On Thu, Apr 25, 2019@2:29 PM Christian Brauner <christian@brauner.io> wrote:
> This timing-based testing seems kinda odd to be honest. Can't we do
> something better than this?

Agreed. Timing-based tests have a substantial risk of becoming flaky.
We ought to be able to make these tests fully deterministic and not
subject to breakage from odd scheduling outcomes. We don't have
sleepable events for everything, granted, but sleep-waiting on a
condition with exponential backoff is fine in test code. In general,
if you start with a robust test, you can insert a sleep(100) anywhere
and not break the logic. Violating this rule always causes pain sooner
or later.

Other thoughts: IMHO, using poll(2) instead of epoll would simplify
the test code, and I think we can get away with calling
pthread_exit(3) instead of SYS_exit.

[PATCH v1 1/2] Add polling support to pidfd

[christian]

On Thu, Apr 25, 2019 at 03:00:09PM -0400, Joel Fernandes (Google) wrote:
> pidfd are file descriptors referring to a process created with the
> CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs pidfd
> polling support to replace code that currently checks for existence of
> /proc/pid for knowing that a process that is signalled to be killed has
> died, which is both racy and slow. The pidfd poll approach is race-free,
> and also allows the LMK to do other things (such as by polling on other
> fds) while awaiting the process being killed to die.

Thanks for the patch!

Ok, let me be a little bit anal.
Please start the commit message with what this patch does and then add
the justification why. You just say the "pidfd-poll" approach. You can
probably assume that CLONE_PIDFD is available for this patch. So
something like:

"This patch makes pidfds pollable. Specifically, it allows listeners to
be informed when the process the pidfd referes to exits. This patch only
introduces the ability to poll thread-group leaders since pidfds
currently can only reference those..."

Then justify the use-case and then go into implementation details.
That's usually how I would think about this:
- Change the codebase to do X
- Why do we need X
- Are there any technical details worth mentioning in the commit message
(- Are there any controversial points that people stumbled upon but that
  have been settled sufficiently.)

> pidfd are file descriptors referring to a process created with the
> CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs pidfd
> polling support to replace code that currently checks for existence of
> /proc/pid for knowing that a process that is signalled to be killed has
> died, which is both racy and slow. The pidfd poll approach is race-free,
> and also allows the LMK to do other things (such as by polling on other
> fds) while awaiting the process being killed to die.

> 
> It prevents a situation where a PID is reused between when LMK sends a
> kill signal and checks for existence of the PID, since the wrong PID is
> now possibly checked for existence.
> 
> In this patch, we follow the same existing mechanism in the kernel used
> when the parent of the task group is to be notified (do_notify_parent).
> This is when the tasks waiting on a poll of pidfd are also awakened.
> 
> We have decided to include the waitqueue in struct pid for the following
> reasons:
> 1. The wait queue has to survive for the lifetime of the poll. Including
> it in task_struct would not be option in this case because the task can
> be reaped and destroyed before the poll returns.
> 
> 2. By including the struct pid for the waitqueue means that during
> de_thread(), the new thread group leader automatically gets the new
> waitqueue/pid even though its task_struct is different.
> 
> Appropriate test cases are added in the second patch to provide coverage
> of all the cases the patch is handling.
> 
> Andy had a similar patch [1] in the past which was a good reference
> however this patch tries to handle different situations properly related
> to thread group existence, and how/where it notifies. And also solves
> other bugs (waitqueue lifetime).  Daniel had a similar patch [2]
> recently which this patch supercedes.
> 
> [1] https://lore.kernel.org/patchwork/patch/345098/
> [2] https://lore.kernel.org/lkml/20181029175322.189042-1-dancol at google.com/
> 
> Cc: luto at amacapital.net
> Cc: rostedt at goodmis.org
> Cc: dancol at google.com
> Cc: sspatil at google.com
> Cc: christian at brauner.io
> Cc: jannh at google.com
> Cc: surenb at google.com
> Cc: timmurray at google.com
> Cc: Jonathan Kowalski <bl0pbl33p at gmail.com>
> Cc: torvalds at linux-foundation.org
> Cc: kernel-team at android.com

These should all be in the form:

Cc: Firstname Lastname <email at address.com>

There are people missing from the Cc that really should be there...

Even though he usually doesn't respond that often, please Cc Al on this.
If he responds it's usually rather important.

Oleg has reviewed your RFC patch quite substantially and given valuable
feedback and has an opinion on this thing and is best acquainted with
the exit code. So please add him to the Cc of the commit message in the
appropriate form and also add him to the Cc of the thread.

Probably also want linux-api for good measure since a lot of people are
subscribed that would care about pollable pidfds. I'd also add Kees
since he had some interest in this work and David (Howells).

> Co-developed-by: Daniel Colascione <dancol at google.com>

Every CDB needs to give a SOB as well.

> Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>
> 
> ---
> 
> RFC -> v1:
> * Based on CLONE_PIDFD patches: https://lwn.net/Articles/786244/
> * Updated selftests.
> * Renamed poll wake function to do_notify_pidfd.
> * Removed depending on EXIT flags
> * Removed POLLERR flag since semantics are controversial and
>   we don't have usecases for it right now (later we can add if there's
>   a need for it).
> 
>  include/linux/pid.h |  3 +++
>  kernel/fork.c       | 33 +++++++++++++++++++++++++++++++++
>  kernel/pid.c        |  2 ++
>  kernel/signal.c     | 14 ++++++++++++++
>  4 files changed, 52 insertions(+)
> 
> diff --git a/include/linux/pid.h b/include/linux/pid.h
> index 3c8ef5a199ca..1484db6ca8d1 100644
> --- a/include/linux/pid.h
> +++ b/include/linux/pid.h
> @@ -3,6 +3,7 @@
>  #define _LINUX_PID_H
>  
>  #include <linux/rculist.h>
> +#include <linux/wait.h>
>  
>  enum pid_type
>  {
> @@ -60,6 +61,8 @@ struct pid
>  	unsigned int level;
>  	/* lists of tasks that use this pid */
>  	struct hlist_head tasks[PIDTYPE_MAX];
> +	/* wait queue for pidfd notifications */
> +	wait_queue_head_t wait_pidfd;
>  	struct rcu_head rcu;
>  	struct upid numbers[1];
>  };
> diff --git a/kernel/fork.c b/kernel/fork.c
> index 5525837ed80e..fb3b614f6456 100644
> --- a/kernel/fork.c
> +++ b/kernel/fork.c
> @@ -1685,8 +1685,41 @@ static void pidfd_show_fdinfo(struct seq_file *m, struct file *f)
>  }
>  #endif
>  
> +static unsigned int pidfd_poll(struct file *file, struct poll_table_struct *pts)
> +{
> +	struct task_struct *task;
> +	struct pid *pid;
> +	int poll_flags = 0;
> +
> +	/*
> +	 * tasklist_lock must be held because to avoid racing with
> +	 * changes in exit_state and wake up. Basically to avoid:
> +	 *
> +	 * P0: read exit_state = 0
> +	 * P1: write exit_state = EXIT_DEAD
> +	 * P1: Do a wake up - wq is empty, so do nothing
> +	 * P0: Queue for polling - wait forever.
> +	 */
> +	read_lock(&tasklist_lock);
> +	pid = file->private_data;
> +	task = pid_task(pid, PIDTYPE_PID);
> +	WARN_ON_ONCE(task && !thread_group_leader(task));
> +
> +	if (!task || (task->exit_state && thread_group_empty(task)))
> +		poll_flags = POLLIN | POLLRDNORM;

So we block until the thread-group is empty? Hm, the thread-group leader
remains in zombie state until all threads are gone. Should probably just
be a short comment somewhere that callers are only informed about a
whole thread-group exit and not about when the thread-group leader has
actually exited.

I would like the ability to extend this interface in the future to allow
for actually reading data from the pidfd on EPOLLIN.
POSIX specifies that POLLIN and POLLRDNORM are set even if the
message to be read is zero. So one cheap way of doing this would
probably be to do a 0 read/ioctl. That wouldn't hurt your very limited
usecase and people could test whether the read returned non-0 data and
if so they know this interface got extended. If we never extend it here
it won't matter.

> +
> +	if (!poll_flags)
> +		poll_wait(file, &pid->wait_pidfd, pts);
> +
> +	read_unlock(&tasklist_lock);
> +
> +	return poll_flags;
> +}


> +
> +
>  const struct file_operations pidfd_fops = {
>  	.release = pidfd_release,
> +	.poll = pidfd_poll,
>  #ifdef CONFIG_PROC_FS
>  	.show_fdinfo = pidfd_show_fdinfo,
>  #endif
> diff --git a/kernel/pid.c b/kernel/pid.c
> index 20881598bdfa..5c90c239242f 100644
> --- a/kernel/pid.c
> +++ b/kernel/pid.c
> @@ -214,6 +214,8 @@ struct pid *alloc_pid(struct pid_namespace *ns)
>  	for (type = 0; type < PIDTYPE_MAX; ++type)
>  		INIT_HLIST_HEAD(&pid->tasks[type]);
>  
> +	init_waitqueue_head(&pid->wait_pidfd);
> +
>  	upid = pid->numbers + ns->level;
>  	spin_lock_irq(&pidmap_lock);
>  	if (!(ns->pid_allocated & PIDNS_ADDING))
> diff --git a/kernel/signal.c b/kernel/signal.c
> index 1581140f2d99..16e7718316e5 100644
> --- a/kernel/signal.c
> +++ b/kernel/signal.c
> @@ -1800,6 +1800,17 @@ int send_sigqueue(struct sigqueue *q, struct pid *pid, enum pid_type type)
>  	return ret;
>  }
>  
> +static void do_notify_pidfd(struct task_struct *task)

Maybe a short command that this helper can only be called when we know
that task is a thread-group leader wouldn't hurt so there's no confusion
later.

> +{
> +	struct pid *pid;
> +
> +	lockdep_assert_held(&tasklist_lock);
> +
> +	pid = get_task_pid(task, PIDTYPE_PID);
> +	wake_up_all(&pid->wait_pidfd);
> +	put_pid(pid);
> +}
> +
>  /*
>   * Let a parent know about the death of a child.
>   * For a stopped/continued status change, use do_notify_parent_cldstop instead.
> @@ -1823,6 +1834,9 @@ bool do_notify_parent(struct task_struct *tsk, int sig)
>  	BUG_ON(!tsk->ptrace &&
>  	       (tsk->group_leader != tsk || !thread_group_empty(tsk)));
>  
> +	/* Wake up all pidfd waiters */
> +	do_notify_pidfd(tsk);
> +
>  	if (sig != SIGCHLD) {
>  		/*
>  		 * This is only possible if parent == real_parent.
> -- 
> 2.21.0.593.g511ec345e18-goog

[PATCH v1 1/2] Add polling support to pidfd

[Christian Brauner]

On Thu, Apr 25, 2019@03:00:09PM -0400, Joel Fernandes (Google) wrote:
> pidfd are file descriptors referring to a process created with the
> CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs pidfd
> polling support to replace code that currently checks for existence of
> /proc/pid for knowing that a process that is signalled to be killed has
> died, which is both racy and slow. The pidfd poll approach is race-free,
> and also allows the LMK to do other things (such as by polling on other
> fds) while awaiting the process being killed to die.

Thanks for the patch!

Ok, let me be a little bit anal.
Please start the commit message with what this patch does and then add
the justification why. You just say the "pidfd-poll" approach. You can
probably assume that CLONE_PIDFD is available for this patch. So
something like:

"This patch makes pidfds pollable. Specifically, it allows listeners to
be informed when the process the pidfd referes to exits. This patch only
introduces the ability to poll thread-group leaders since pidfds
currently can only reference those..."

Then justify the use-case and then go into implementation details.
That's usually how I would think about this:
- Change the codebase to do X
- Why do we need X
- Are there any technical details worth mentioning in the commit message
(- Are there any controversial points that people stumbled upon but that
  have been settled sufficiently.)

> pidfd are file descriptors referring to a process created with the
> CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs pidfd
> polling support to replace code that currently checks for existence of
> /proc/pid for knowing that a process that is signalled to be killed has
> died, which is both racy and slow. The pidfd poll approach is race-free,
> and also allows the LMK to do other things (such as by polling on other
> fds) while awaiting the process being killed to die.

> 
> It prevents a situation where a PID is reused between when LMK sends a
> kill signal and checks for existence of the PID, since the wrong PID is
> now possibly checked for existence.
> 
> In this patch, we follow the same existing mechanism in the kernel used
> when the parent of the task group is to be notified (do_notify_parent).
> This is when the tasks waiting on a poll of pidfd are also awakened.
> 
> We have decided to include the waitqueue in struct pid for the following
> reasons:
> 1. The wait queue has to survive for the lifetime of the poll. Including
> it in task_struct would not be option in this case because the task can
> be reaped and destroyed before the poll returns.
> 
> 2. By including the struct pid for the waitqueue means that during
> de_thread(), the new thread group leader automatically gets the new
> waitqueue/pid even though its task_struct is different.
> 
> Appropriate test cases are added in the second patch to provide coverage
> of all the cases the patch is handling.
> 
> Andy had a similar patch [1] in the past which was a good reference
> however this patch tries to handle different situations properly related
> to thread group existence, and how/where it notifies. And also solves
> other bugs (waitqueue lifetime).  Daniel had a similar patch [2]
> recently which this patch supercedes.
> 
> [1] https://lore.kernel.org/patchwork/patch/345098/
> [2] https://lore.kernel.org/lkml/20181029175322.189042-1-dancol at google.com/
> 
> Cc: luto at amacapital.net
> Cc: rostedt at goodmis.org
> Cc: dancol at google.com
> Cc: sspatil at google.com
> Cc: christian at brauner.io
> Cc: jannh at google.com
> Cc: surenb at google.com
> Cc: timmurray at google.com
> Cc: Jonathan Kowalski <bl0pbl33p at gmail.com>
> Cc: torvalds at linux-foundation.org
> Cc: kernel-team at android.com

These should all be in the form:

Cc: Firstname Lastname <email at address.com>

There are people missing from the Cc that really should be there...

Even though he usually doesn't respond that often, please Cc Al on this.
If he responds it's usually rather important.

Oleg has reviewed your RFC patch quite substantially and given valuable
feedback and has an opinion on this thing and is best acquainted with
the exit code. So please add him to the Cc of the commit message in the
appropriate form and also add him to the Cc of the thread.

Probably also want linux-api for good measure since a lot of people are
subscribed that would care about pollable pidfds. I'd also add Kees
since he had some interest in this work and David (Howells).

> Co-developed-by: Daniel Colascione <dancol at google.com>

Every CDB needs to give a SOB as well.

> Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>
> 
> ---
> 
> RFC -> v1:
> * Based on CLONE_PIDFD patches: https://lwn.net/Articles/786244/
> * Updated selftests.
> * Renamed poll wake function to do_notify_pidfd.
> * Removed depending on EXIT flags
> * Removed POLLERR flag since semantics are controversial and
>   we don't have usecases for it right now (later we can add if there's
>   a need for it).
> 
>  include/linux/pid.h |  3 +++
>  kernel/fork.c       | 33 +++++++++++++++++++++++++++++++++
>  kernel/pid.c        |  2 ++
>  kernel/signal.c     | 14 ++++++++++++++
>  4 files changed, 52 insertions(+)
> 
> diff --git a/include/linux/pid.h b/include/linux/pid.h
> index 3c8ef5a199ca..1484db6ca8d1 100644
> --- a/include/linux/pid.h
> +++ b/include/linux/pid.h
> @@ -3,6 +3,7 @@
>  #define _LINUX_PID_H
>  
>  #include <linux/rculist.h>
> +#include <linux/wait.h>
>  
>  enum pid_type
>  {
> @@ -60,6 +61,8 @@ struct pid
>  	unsigned int level;
>  	/* lists of tasks that use this pid */
>  	struct hlist_head tasks[PIDTYPE_MAX];
> +	/* wait queue for pidfd notifications */
> +	wait_queue_head_t wait_pidfd;
>  	struct rcu_head rcu;
>  	struct upid numbers[1];
>  };
> diff --git a/kernel/fork.c b/kernel/fork.c
> index 5525837ed80e..fb3b614f6456 100644
> --- a/kernel/fork.c
> +++ b/kernel/fork.c
> @@ -1685,8 +1685,41 @@ static void pidfd_show_fdinfo(struct seq_file *m, struct file *f)
>  }
>  #endif
>  
> +static unsigned int pidfd_poll(struct file *file, struct poll_table_struct *pts)
> +{
> +	struct task_struct *task;
> +	struct pid *pid;
> +	int poll_flags = 0;
> +
> +	/*
> +	 * tasklist_lock must be held because to avoid racing with
> +	 * changes in exit_state and wake up. Basically to avoid:
> +	 *
> +	 * P0: read exit_state = 0
> +	 * P1: write exit_state = EXIT_DEAD
> +	 * P1: Do a wake up - wq is empty, so do nothing
> +	 * P0: Queue for polling - wait forever.
> +	 */
> +	read_lock(&tasklist_lock);
> +	pid = file->private_data;
> +	task = pid_task(pid, PIDTYPE_PID);
> +	WARN_ON_ONCE(task && !thread_group_leader(task));
> +
> +	if (!task || (task->exit_state && thread_group_empty(task)))
> +		poll_flags = POLLIN | POLLRDNORM;

So we block until the thread-group is empty? Hm, the thread-group leader
remains in zombie state until all threads are gone. Should probably just
be a short comment somewhere that callers are only informed about a
whole thread-group exit and not about when the thread-group leader has
actually exited.

I would like the ability to extend this interface in the future to allow
for actually reading data from the pidfd on EPOLLIN.
POSIX specifies that POLLIN and POLLRDNORM are set even if the
message to be read is zero. So one cheap way of doing this would
probably be to do a 0 read/ioctl. That wouldn't hurt your very limited
usecase and people could test whether the read returned non-0 data and
if so they know this interface got extended. If we never extend it here
it won't matter.

> +
> +	if (!poll_flags)
> +		poll_wait(file, &pid->wait_pidfd, pts);
> +
> +	read_unlock(&tasklist_lock);
> +
> +	return poll_flags;
> +}


> +
> +
>  const struct file_operations pidfd_fops = {
>  	.release = pidfd_release,
> +	.poll = pidfd_poll,
>  #ifdef CONFIG_PROC_FS
>  	.show_fdinfo = pidfd_show_fdinfo,
>  #endif
> diff --git a/kernel/pid.c b/kernel/pid.c
> index 20881598bdfa..5c90c239242f 100644
> --- a/kernel/pid.c
> +++ b/kernel/pid.c
> @@ -214,6 +214,8 @@ struct pid *alloc_pid(struct pid_namespace *ns)
>  	for (type = 0; type < PIDTYPE_MAX; ++type)
>  		INIT_HLIST_HEAD(&pid->tasks[type]);
>  
> +	init_waitqueue_head(&pid->wait_pidfd);
> +
>  	upid = pid->numbers + ns->level;
>  	spin_lock_irq(&pidmap_lock);
>  	if (!(ns->pid_allocated & PIDNS_ADDING))
> diff --git a/kernel/signal.c b/kernel/signal.c
> index 1581140f2d99..16e7718316e5 100644
> --- a/kernel/signal.c
> +++ b/kernel/signal.c
> @@ -1800,6 +1800,17 @@ int send_sigqueue(struct sigqueue *q, struct pid *pid, enum pid_type type)
>  	return ret;
>  }
>  
> +static void do_notify_pidfd(struct task_struct *task)

Maybe a short command that this helper can only be called when we know
that task is a thread-group leader wouldn't hurt so there's no confusion
later.

> +{
> +	struct pid *pid;
> +
> +	lockdep_assert_held(&tasklist_lock);
> +
> +	pid = get_task_pid(task, PIDTYPE_PID);
> +	wake_up_all(&pid->wait_pidfd);
> +	put_pid(pid);
> +}
> +
>  /*
>   * Let a parent know about the death of a child.
>   * For a stopped/continued status change, use do_notify_parent_cldstop instead.
> @@ -1823,6 +1834,9 @@ bool do_notify_parent(struct task_struct *tsk, int sig)
>  	BUG_ON(!tsk->ptrace &&
>  	       (tsk->group_leader != tsk || !thread_group_empty(tsk)));
>  
> +	/* Wake up all pidfd waiters */
> +	do_notify_pidfd(tsk);
> +
>  	if (sig != SIGCHLD) {
>  		/*
>  		 * This is only possible if parent == real_parent.
> -- 
> 2.21.0.593.g511ec345e18-goog

[PATCH v1 2/2] Add selftests for pidfd polling

[joel]

On Thu, Apr 25, 2019 at 11:29:18PM +0200, Christian Brauner wrote:
> On Thu, Apr 25, 2019 at 03:00:10PM -0400, Joel Fernandes (Google) wrote:
> > Other than verifying pidfd based polling, the tests make sure that
> > wait semantics are preserved with the pidfd poll. Notably the 2 cases:
> > 1. If a thread group leader exits while threads still there, then no
> >    pidfd poll notifcation should happen.
> > 2. If a non-thread group leader does an execve, then the thread group
> >    leader is signaled to exit and is replaced with the execing thread
> >    as the new leader, however the parent is not notified in this case.
> > 
> > Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>
> > ---
> >  tools/testing/selftests/pidfd/Makefile     |   2 +-
> >  tools/testing/selftests/pidfd/pidfd_test.c | 198 +++++++++++++++++++++
> >  2 files changed, 199 insertions(+), 1 deletion(-)
> > 
> > diff --git a/tools/testing/selftests/pidfd/Makefile b/tools/testing/selftests/pidfd/Makefile
> > index deaf8073bc06..4b31c14f273c 100644
> > --- a/tools/testing/selftests/pidfd/Makefile
> > +++ b/tools/testing/selftests/pidfd/Makefile
> > @@ -1,4 +1,4 @@
> > -CFLAGS += -g -I../../../../usr/include/
> > +CFLAGS += -g -I../../../../usr/include/ -lpthread
> >  
> >  TEST_GEN_PROGS := pidfd_test
> >  
> > diff --git a/tools/testing/selftests/pidfd/pidfd_test.c b/tools/testing/selftests/pidfd/pidfd_test.c
> > index d59378a93782..e887f807645e 100644
> > --- a/tools/testing/selftests/pidfd/pidfd_test.c
> > +++ b/tools/testing/selftests/pidfd/pidfd_test.c
> > @@ -4,18 +4,42 @@
> >  #include <errno.h>
> >  #include <fcntl.h>
> >  #include <linux/types.h>
> > +#include <pthread.h>
> >  #include <sched.h>
> >  #include <signal.h>
> >  #include <stdio.h>
> >  #include <stdlib.h>
> >  #include <string.h>
> >  #include <syscall.h>
> > +#include <sys/epoll.h>
> > +#include <sys/mman.h>
> >  #include <sys/mount.h>
> >  #include <sys/wait.h>
> > +#include <time.h>
> >  #include <unistd.h>
> >  
> >  #include "../kselftest.h"
> >  
> > +#define CHILD_THREAD_MIN_WAIT 3 /* seconds */
> > +#define MAX_EVENTS 5
> > +#define __NR_pidfd_send_signal 424
> 
> Should probably be ifndefed as well.

done

> > +#ifndef CLONE_PIDFD
> > +#define CLONE_PIDFD 0x00001000
> > +#endif
> > +
> > +static pid_t pidfd_clone(int flags, int *pidfd, int (*fn)(void *))
> > +{
> > +	size_t stack_size = 1024;
> > +	char *stack[1024] = { 0 };
> > +
> > +#ifdef __ia64__
> > +	return __clone2(fn, stack, stack_size, flags | SIGCHLD, NULL, pidfd);
> > +#else
> > +	return clone(fn, stack + stack_size, flags | SIGCHLD, NULL, pidfd);
> > +#endif
> > +}
> > +
> >  static inline int sys_pidfd_send_signal(int pidfd, int sig, siginfo_t *info,
> >  					unsigned int flags)
> >  {
> > @@ -368,10 +392,184 @@ static int test_pidfd_send_signal_syscall_support(void)
> >  	return 0;
> >  }
> >  
> > +void *test_pidfd_poll_exec_thread(void *priv)
> > +{
> > +	char waittime[256];
> > +
> > +	ksft_print_msg("Child Thread: starting. pid %d tid %d ; and sleeping\n",
> > +			getpid(), syscall(SYS_gettid));
> > +	ksft_print_msg("Child Thread: doing exec of sleep\n");
> > +
> > +	sprintf(waittime, "%d", CHILD_THREAD_MIN_WAIT);
> 
> > +#define CHILD_THREAD_MIN_SLEEP "3" /* seconds */
> 
> Could also be
> 
> #define str(s) _str(s)
> #define _str(s) #s
> #define CHILD_THREAD_MIN_SLEEP 3
> 
> execl("/bin/sleep", "sleep", str(CHILD_THREAD_MIN_SLEEP), (char *)NULL);
> 
> getting rid of waittime, and snprintf().

yep, much better, thanks.

> > +	execl("/bin/sleep", "sleep", waittime, (char *)NULL);
> > +
> > +	ksft_print_msg("Child Thread: DONE. pid %d tid %d\n",
> > +			getpid(), syscall(SYS_gettid));
> > +	return NULL;
> > +}
> > +
> > +static int poll_pidfd(const char *test_name, int pidfd)
> > +{
> > +	int c;
> > +	int epoll_fd = epoll_create1(0);
> 
> You probably don't need the epoll_fd after an exec, so:
> int epoll_fd = epoll_create1(EPOLL_CLOEXEC);

done

> > +	struct epoll_event event, events[MAX_EVENTS];
> > +
> > +	if (epoll_fd == -1)
> > +		ksft_exit_fail_msg("%s test: Failed to create epoll file descriptor\n",
> > +				   test_name);
> 
> I think logging the errno is helpful here. 
> 
> > +
> > +	event.events = EPOLLIN;
> > +	event.data.fd = pidfd;
> > +
> > +	if (epoll_ctl(epoll_fd, EPOLL_CTL_ADD, pidfd, &event)) {
> > +		ksft_print_msg("%s test: Failed to add epoll file descriptor: Skipping\n",
> > +			       test_name);
> 
> I think logging the errno is helpful here. 

no where else in other tests are we logging this. I don't have a preference.
Should ksft_exit_fail_msg() do this automatically? Although it could be
logging a stale errno if it did.  Anyway I added logging of errno here, as
you suggest.

> > +		_exit(PIDFD_SKIP);
> 
> Why do you skip when you can't add the pidfd to the epoll loop? Why
> shouldn't this be a test failure?

The original approach was to do this for proc pidfd, which means older
kernels could get a pidfd but couldn't do poll, in this case I wanted the
test to be skipped. Since we are now basing this on CLONE_PIDFD, there is
less of a reason for that. So I will just do ksft_exit_fail_msg() here.

> > +	}
> > +
> > +	c = epoll_wait(epoll_fd, events, MAX_EVENTS, 5000);
> 
> Uhm 5000 timeout? Either do a -1 or something that is noticeably
> shorter, please. :)

I want a timeout for the case where epoll_wait blocks indefinitely, in which
case it should be a test failure.

> > +	if (c != 1 || !(events[0].events & EPOLLIN))
> > +		ksft_exit_fail_msg("%s test: Unexpected epoll_wait result (c=%d, events=%x)\n",
> > +				   test_name, c, events[0].events);
> 
> I think logging the errno is helpful here. 

Ok, done.

> > +
> > +	close(epoll_fd);
> > +	return events[0].events;
> > +
> > +}
> > +
> > +static int child_poll_exec_test(void *args)
> > +{
> > +	pthread_t t1;
> > +
> > +	ksft_print_msg("Child (pidfd): starting. pid %d tid %d\n", getpid(),
> > +			syscall(SYS_gettid));
> > +	pthread_create(&t1, NULL, test_pidfd_poll_exec_thread, NULL);
> > +	/*
> > +	 * Exec in the non-leader thread will destroy the leader immediately.
> > +	 * If the wait in the parent returns too soon, the test fails.
> > +	 */
> > +	while (1)
> > +		;
> 
> Wouldn't sleep(<some-value>) be better here or at least a:
> 
> while (true)
>         sleep(<some-sensible-value);
> 
> instead of a busy loop?

Good catch, I will do sleep(1);

> > +}
> > +
> > +int test_pidfd_poll_exec(int use_waitpid)
> > +{
> > +	int pid, pidfd = 0;
> > +	int status, ret;
> > +	pthread_t t1;
> > +	time_t prog_start = time(NULL);
> > +	const char *test_name = "pidfd_poll check for premature notification on child thread exec";
> > +
> > +	ksft_print_msg("Parent: pid: %d\n", getpid());
> > +	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_exec_test);
> 
> That needs to check for error aka
> if (pid < 0)
> I think Tycho mentioned this already.

fixed, thanks to Tycho as well!

> > +
> > +	ksft_print_msg("Parent: Waiting for Child (%d) to complete.\n", pid);
> > +
> > +	if (use_waitpid) {
> > +		ret = waitpid(pid, &status, 0);
> > +		if (ret == -1)
> > +			ksft_print_msg("Parent: error\n");
> > +
> > +		if (ret == pid)
> > +			ksft_print_msg("Parent: Child process waited for.\n");
> > +	} else {
> > +		poll_pidfd(test_name, pidfd);
> 
> Either make poll_pidfd() void or check the error value. One of the two.

done

> > +	}
> > +
> > +	time_t prog_time = time(NULL) - prog_start;
> > +
> > +	ksft_print_msg("Time waited for child: %lu\n", prog_time);
> > +
> > +	close(pidfd);
> > +
> > +	if (prog_time < CHILD_THREAD_MIN_WAIT || prog_time > CHILD_THREAD_MIN_WAIT + 2)
> 
> This timing-based testing seems kinda odd to be honest. Can't we do
> something better than this?

will try..

> > +		ksft_exit_fail_msg("%s test: Failed\n", test_name);
> > +	else
> > +		ksft_test_result_pass("%s test: Passed\n", test_name);
> > +}
> > +
> > +void *test_pidfd_poll_leader_exit_thread(void *priv)
> > +{
> > +	char waittime[256];
> 
> Unused variable

ouch, fixed

> > +
> > +	ksft_print_msg("Child Thread: starting. pid %d tid %d ; and sleeping\n",
> > +			getpid(), syscall(SYS_gettid));
> > +	sleep(CHILD_THREAD_MIN_WAIT);
> > +	ksft_print_msg("Child Thread: DONE. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
> > +	return NULL;
> > +}
> > +
> > +static time_t *child_exit_secs;
> > +static int child_poll_leader_exit_test(void *args)
> > +{
> > +	pthread_t t1, t2;
> > +
> > +	ksft_print_msg("Child: starting. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
> > +	pthread_create(&t1, NULL, test_pidfd_poll_leader_exit_thread, NULL);
> > +	pthread_create(&t2, NULL, test_pidfd_poll_leader_exit_thread, NULL);
> > +
> > +	/*
> > +	 * glibc exit calls exit_group syscall, so explicity call exit only
> > +	 * so that only the group leader exits, leaving the threads alone.
> > +	 */
> > +	*child_exit_secs = time(NULL);
> > +	syscall(SYS_exit, 0);
> > +}
> > +
> > +int test_pidfd_poll_leader_exit(int use_waitpid)
> 
> static

fixed

> > +{
> > +	int pid, pidfd = 0;
> > +	int status, ret;
> > +	time_t prog_start = time(NULL);
> > +	const char *test_name = "pidfd_poll check for premature notification on non-empty"
> > +				"group leader exit";
> > +
> > +	child_exit_secs = mmap(NULL, sizeof *child_exit_secs, PROT_READ | PROT_WRITE,
> > +			MAP_SHARED | MAP_ANONYMOUS, -1, 0);
> 
> Error checking, please:
> 
> if (child_exit_secs == MAP_FAILED)

done

> > +
> > +	ksft_print_msg("Parent: pid: %d\n", getpid());
> > +	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_leader_exit_test);
> 
> Error checking, please:
> 
> if (pid < 0)

done

> > +
> > +	ksft_print_msg("Parent: Waiting for Child (%d) to complete.\n", pid);
> > +
> > +	if (use_waitpid) {
> > +		ret = waitpid(pid, &status, 0);
> > +		if (ret == -1)
> > +			ksft_print_msg("Parent: error\n");
> > +	} else {
> > +		/*
> > +		 * This sleep tests for the case where if the child exits, and is in
> > +		 * EXIT_ZOMBIE, but the thread group leader is non-empty, then the poll
> > +		 * doesn't prematurely return even though there are active threads
> > +		 */
> > +		sleep(1);
> > +		poll_pidfd(test_name, pidfd);
> 
> Make poll_pidfd() void or check error, please.

done, made void

> > +	}
> > +
> > +	if (ret == pid)
> > +		ksft_print_msg("Parent: Child process waited for.\n");
> > +
> > +	time_t since_child_exit = time(NULL) - *child_exit_secs;
> > +
> > +	ksft_print_msg("Time since child exit: %lu\n", since_child_exit);
> > +
> > +	close(pidfd);
> > +
> > +	if (since_child_exit < CHILD_THREAD_MIN_WAIT ||
> > +			since_child_exit > CHILD_THREAD_MIN_WAIT + 2)
> 
> This looks very magical. Especially without a comment. Now you add
> random +2. Please comment it or better, come up with a non-timing
> based test.

Will try a non-timing test, need to plan it out. Other comments are addressed
and will post again soon, thanks!

 - Joel

[PATCH v1 2/2] Add selftests for pidfd polling

[Joel Fernandes]

On Thu, Apr 25, 2019@11:29:18PM +0200, Christian Brauner wrote:
> On Thu, Apr 25, 2019@03:00:10PM -0400, Joel Fernandes (Google) wrote:
> > Other than verifying pidfd based polling, the tests make sure that
> > wait semantics are preserved with the pidfd poll. Notably the 2 cases:
> > 1. If a thread group leader exits while threads still there, then no
> >    pidfd poll notifcation should happen.
> > 2. If a non-thread group leader does an execve, then the thread group
> >    leader is signaled to exit and is replaced with the execing thread
> >    as the new leader, however the parent is not notified in this case.
> > 
> > Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>
> > ---
> >  tools/testing/selftests/pidfd/Makefile     |   2 +-
> >  tools/testing/selftests/pidfd/pidfd_test.c | 198 +++++++++++++++++++++
> >  2 files changed, 199 insertions(+), 1 deletion(-)
> > 
> > diff --git a/tools/testing/selftests/pidfd/Makefile b/tools/testing/selftests/pidfd/Makefile
> > index deaf8073bc06..4b31c14f273c 100644
> > --- a/tools/testing/selftests/pidfd/Makefile
> > +++ b/tools/testing/selftests/pidfd/Makefile
> > @@ -1,4 +1,4 @@
> > -CFLAGS += -g -I../../../../usr/include/
> > +CFLAGS += -g -I../../../../usr/include/ -lpthread
> >  
> >  TEST_GEN_PROGS := pidfd_test
> >  
> > diff --git a/tools/testing/selftests/pidfd/pidfd_test.c b/tools/testing/selftests/pidfd/pidfd_test.c
> > index d59378a93782..e887f807645e 100644
> > --- a/tools/testing/selftests/pidfd/pidfd_test.c
> > +++ b/tools/testing/selftests/pidfd/pidfd_test.c
> > @@ -4,18 +4,42 @@
> >  #include <errno.h>
> >  #include <fcntl.h>
> >  #include <linux/types.h>
> > +#include <pthread.h>
> >  #include <sched.h>
> >  #include <signal.h>
> >  #include <stdio.h>
> >  #include <stdlib.h>
> >  #include <string.h>
> >  #include <syscall.h>
> > +#include <sys/epoll.h>
> > +#include <sys/mman.h>
> >  #include <sys/mount.h>
> >  #include <sys/wait.h>
> > +#include <time.h>
> >  #include <unistd.h>
> >  
> >  #include "../kselftest.h"
> >  
> > +#define CHILD_THREAD_MIN_WAIT 3 /* seconds */
> > +#define MAX_EVENTS 5
> > +#define __NR_pidfd_send_signal 424
> 
> Should probably be ifndefed as well.

done

> > +#ifndef CLONE_PIDFD
> > +#define CLONE_PIDFD 0x00001000
> > +#endif
> > +
> > +static pid_t pidfd_clone(int flags, int *pidfd, int (*fn)(void *))
> > +{
> > +	size_t stack_size = 1024;
> > +	char *stack[1024] = { 0 };
> > +
> > +#ifdef __ia64__
> > +	return __clone2(fn, stack, stack_size, flags | SIGCHLD, NULL, pidfd);
> > +#else
> > +	return clone(fn, stack + stack_size, flags | SIGCHLD, NULL, pidfd);
> > +#endif
> > +}
> > +
> >  static inline int sys_pidfd_send_signal(int pidfd, int sig, siginfo_t *info,
> >  					unsigned int flags)
> >  {
> > @@ -368,10 +392,184 @@ static int test_pidfd_send_signal_syscall_support(void)
> >  	return 0;
> >  }
> >  
> > +void *test_pidfd_poll_exec_thread(void *priv)
> > +{
> > +	char waittime[256];
> > +
> > +	ksft_print_msg("Child Thread: starting. pid %d tid %d ; and sleeping\n",
> > +			getpid(), syscall(SYS_gettid));
> > +	ksft_print_msg("Child Thread: doing exec of sleep\n");
> > +
> > +	sprintf(waittime, "%d", CHILD_THREAD_MIN_WAIT);
> 
> > +#define CHILD_THREAD_MIN_SLEEP "3" /* seconds */
> 
> Could also be
> 
> #define str(s) _str(s)
> #define _str(s) #s
> #define CHILD_THREAD_MIN_SLEEP 3
> 
> execl("/bin/sleep", "sleep", str(CHILD_THREAD_MIN_SLEEP), (char *)NULL);
> 
> getting rid of waittime, and snprintf().

yep, much better, thanks.

> > +	execl("/bin/sleep", "sleep", waittime, (char *)NULL);
> > +
> > +	ksft_print_msg("Child Thread: DONE. pid %d tid %d\n",
> > +			getpid(), syscall(SYS_gettid));
> > +	return NULL;
> > +}
> > +
> > +static int poll_pidfd(const char *test_name, int pidfd)
> > +{
> > +	int c;
> > +	int epoll_fd = epoll_create1(0);
> 
> You probably don't need the epoll_fd after an exec, so:
> int epoll_fd = epoll_create1(EPOLL_CLOEXEC);

done

> > +	struct epoll_event event, events[MAX_EVENTS];
> > +
> > +	if (epoll_fd == -1)
> > +		ksft_exit_fail_msg("%s test: Failed to create epoll file descriptor\n",
> > +				   test_name);
> 
> I think logging the errno is helpful here. 
> 
> > +
> > +	event.events = EPOLLIN;
> > +	event.data.fd = pidfd;
> > +
> > +	if (epoll_ctl(epoll_fd, EPOLL_CTL_ADD, pidfd, &event)) {
> > +		ksft_print_msg("%s test: Failed to add epoll file descriptor: Skipping\n",
> > +			       test_name);
> 
> I think logging the errno is helpful here. 

no where else in other tests are we logging this. I don't have a preference.
Should ksft_exit_fail_msg() do this automatically? Although it could be
logging a stale errno if it did.  Anyway I added logging of errno here, as
you suggest.

> > +		_exit(PIDFD_SKIP);
> 
> Why do you skip when you can't add the pidfd to the epoll loop? Why
> shouldn't this be a test failure?

The original approach was to do this for proc pidfd, which means older
kernels could get a pidfd but couldn't do poll, in this case I wanted the
test to be skipped. Since we are now basing this on CLONE_PIDFD, there is
less of a reason for that. So I will just do ksft_exit_fail_msg() here.

> > +	}
> > +
> > +	c = epoll_wait(epoll_fd, events, MAX_EVENTS, 5000);
> 
> Uhm 5000 timeout? Either do a -1 or something that is noticeably
> shorter, please. :)

I want a timeout for the case where epoll_wait blocks indefinitely, in which
case it should be a test failure.

> > +	if (c != 1 || !(events[0].events & EPOLLIN))
> > +		ksft_exit_fail_msg("%s test: Unexpected epoll_wait result (c=%d, events=%x)\n",
> > +				   test_name, c, events[0].events);
> 
> I think logging the errno is helpful here. 

Ok, done.

> > +
> > +	close(epoll_fd);
> > +	return events[0].events;
> > +
> > +}
> > +
> > +static int child_poll_exec_test(void *args)
> > +{
> > +	pthread_t t1;
> > +
> > +	ksft_print_msg("Child (pidfd): starting. pid %d tid %d\n", getpid(),
> > +			syscall(SYS_gettid));
> > +	pthread_create(&t1, NULL, test_pidfd_poll_exec_thread, NULL);
> > +	/*
> > +	 * Exec in the non-leader thread will destroy the leader immediately.
> > +	 * If the wait in the parent returns too soon, the test fails.
> > +	 */
> > +	while (1)
> > +		;
> 
> Wouldn't sleep(<some-value>) be better here or at least a:
> 
> while (true)
>         sleep(<some-sensible-value);
> 
> instead of a busy loop?

Good catch, I will do sleep(1);

> > +}
> > +
> > +int test_pidfd_poll_exec(int use_waitpid)
> > +{
> > +	int pid, pidfd = 0;
> > +	int status, ret;
> > +	pthread_t t1;
> > +	time_t prog_start = time(NULL);
> > +	const char *test_name = "pidfd_poll check for premature notification on child thread exec";
> > +
> > +	ksft_print_msg("Parent: pid: %d\n", getpid());
> > +	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_exec_test);
> 
> That needs to check for error aka
> if (pid < 0)
> I think Tycho mentioned this already.

fixed, thanks to Tycho as well!

> > +
> > +	ksft_print_msg("Parent: Waiting for Child (%d) to complete.\n", pid);
> > +
> > +	if (use_waitpid) {
> > +		ret = waitpid(pid, &status, 0);
> > +		if (ret == -1)
> > +			ksft_print_msg("Parent: error\n");
> > +
> > +		if (ret == pid)
> > +			ksft_print_msg("Parent: Child process waited for.\n");
> > +	} else {
> > +		poll_pidfd(test_name, pidfd);
> 
> Either make poll_pidfd() void or check the error value. One of the two.

done

> > +	}
> > +
> > +	time_t prog_time = time(NULL) - prog_start;
> > +
> > +	ksft_print_msg("Time waited for child: %lu\n", prog_time);
> > +
> > +	close(pidfd);
> > +
> > +	if (prog_time < CHILD_THREAD_MIN_WAIT || prog_time > CHILD_THREAD_MIN_WAIT + 2)
> 
> This timing-based testing seems kinda odd to be honest. Can't we do
> something better than this?

will try..

> > +		ksft_exit_fail_msg("%s test: Failed\n", test_name);
> > +	else
> > +		ksft_test_result_pass("%s test: Passed\n", test_name);
> > +}
> > +
> > +void *test_pidfd_poll_leader_exit_thread(void *priv)
> > +{
> > +	char waittime[256];
> 
> Unused variable

ouch, fixed

> > +
> > +	ksft_print_msg("Child Thread: starting. pid %d tid %d ; and sleeping\n",
> > +			getpid(), syscall(SYS_gettid));
> > +	sleep(CHILD_THREAD_MIN_WAIT);
> > +	ksft_print_msg("Child Thread: DONE. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
> > +	return NULL;
> > +}
> > +
> > +static time_t *child_exit_secs;
> > +static int child_poll_leader_exit_test(void *args)
> > +{
> > +	pthread_t t1, t2;
> > +
> > +	ksft_print_msg("Child: starting. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
> > +	pthread_create(&t1, NULL, test_pidfd_poll_leader_exit_thread, NULL);
> > +	pthread_create(&t2, NULL, test_pidfd_poll_leader_exit_thread, NULL);
> > +
> > +	/*
> > +	 * glibc exit calls exit_group syscall, so explicity call exit only
> > +	 * so that only the group leader exits, leaving the threads alone.
> > +	 */
> > +	*child_exit_secs = time(NULL);
> > +	syscall(SYS_exit, 0);
> > +}
> > +
> > +int test_pidfd_poll_leader_exit(int use_waitpid)
> 
> static

fixed

> > +{
> > +	int pid, pidfd = 0;
> > +	int status, ret;
> > +	time_t prog_start = time(NULL);
> > +	const char *test_name = "pidfd_poll check for premature notification on non-empty"
> > +				"group leader exit";
> > +
> > +	child_exit_secs = mmap(NULL, sizeof *child_exit_secs, PROT_READ | PROT_WRITE,
> > +			MAP_SHARED | MAP_ANONYMOUS, -1, 0);
> 
> Error checking, please:
> 
> if (child_exit_secs == MAP_FAILED)

done

> > +
> > +	ksft_print_msg("Parent: pid: %d\n", getpid());
> > +	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_leader_exit_test);
> 
> Error checking, please:
> 
> if (pid < 0)

done

> > +
> > +	ksft_print_msg("Parent: Waiting for Child (%d) to complete.\n", pid);
> > +
> > +	if (use_waitpid) {
> > +		ret = waitpid(pid, &status, 0);
> > +		if (ret == -1)
> > +			ksft_print_msg("Parent: error\n");
> > +	} else {
> > +		/*
> > +		 * This sleep tests for the case where if the child exits, and is in
> > +		 * EXIT_ZOMBIE, but the thread group leader is non-empty, then the poll
> > +		 * doesn't prematurely return even though there are active threads
> > +		 */
> > +		sleep(1);
> > +		poll_pidfd(test_name, pidfd);
> 
> Make poll_pidfd() void or check error, please.

done, made void

> > +	}
> > +
> > +	if (ret == pid)
> > +		ksft_print_msg("Parent: Child process waited for.\n");
> > +
> > +	time_t since_child_exit = time(NULL) - *child_exit_secs;
> > +
> > +	ksft_print_msg("Time since child exit: %lu\n", since_child_exit);
> > +
> > +	close(pidfd);
> > +
> > +	if (since_child_exit < CHILD_THREAD_MIN_WAIT ||
> > +			since_child_exit > CHILD_THREAD_MIN_WAIT + 2)
> 
> This looks very magical. Especially without a comment. Now you add
> random +2. Please comment it or better, come up with a non-timing
> based test.

Will try a non-timing test, need to plan it out. Other comments are addressed
and will post again soon, thanks!

 - Joel

[PATCH v1 2/2] Add selftests for pidfd polling

[joel]

On Thu, Apr 25, 2019 at 02:00:35PM -0600, Tycho Andersen wrote:
> On Thu, Apr 25, 2019 at 03:00:10PM -0400, Joel Fernandes (Google) wrote:
> >
> > +void *test_pidfd_poll_exec_thread(void *priv)
> 
> I think everything in this file can be static, there's this one and
> 3-4 below.
> 
> > +int test_pidfd_poll_exec(int use_waitpid)
> > +{
> > +	int pid, pidfd = 0;
> > +	int status, ret;
> > +	pthread_t t1;
> > +	time_t prog_start = time(NULL);
> > +	const char *test_name = "pidfd_poll check for premature notification on child thread exec";
> > +
> > +	ksft_print_msg("Parent: pid: %d\n", getpid());
> > +	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_exec_test);
> 
> If pidfd_clone() fails here, I think things will go haywire below.
> 
> > +	ksft_print_msg("Parent: Waiting for Child (%d) to complete.\n", pid);
> > +
> > +	if (use_waitpid) {
> > +		ret = waitpid(pid, &status, 0);
> > +		if (ret == -1)
> > +			ksft_print_msg("Parent: error\n");
> > +
> > +		if (ret == pid)
> > +			ksft_print_msg("Parent: Child process waited for.\n");
> > +	} else {
> > +		poll_pidfd(test_name, pidfd);
> > +	}
> > +
> > +	time_t prog_time = time(NULL) - prog_start;
> > +
> > +	ksft_print_msg("Time waited for child: %lu\n", prog_time);
> > +
> > +	close(pidfd);
> > +
> > +	if (prog_time < CHILD_THREAD_MIN_WAIT || prog_time > CHILD_THREAD_MIN_WAIT + 2)
> > +		ksft_exit_fail_msg("%s test: Failed\n", test_name);
> > +	else
> > +		ksft_test_result_pass("%s test: Passed\n", test_name);
> > +}
> > +
> > +void *test_pidfd_poll_leader_exit_thread(void *priv)
> > +{
> > +	char waittime[256];
> > +
> > +	ksft_print_msg("Child Thread: starting. pid %d tid %d ; and sleeping\n",
> > +			getpid(), syscall(SYS_gettid));
> > +	sleep(CHILD_THREAD_MIN_WAIT);
> > +	ksft_print_msg("Child Thread: DONE. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
> > +	return NULL;
> > +}
> > +
> > +static time_t *child_exit_secs;
> > +static int child_poll_leader_exit_test(void *args)
> > +{
> > +	pthread_t t1, t2;
> > +
> > +	ksft_print_msg("Child: starting. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
> > +	pthread_create(&t1, NULL, test_pidfd_poll_leader_exit_thread, NULL);
> > +	pthread_create(&t2, NULL, test_pidfd_poll_leader_exit_thread, NULL);
> > +
> > +	/*
> > +	 * glibc exit calls exit_group syscall, so explicity call exit only
> > +	 * so that only the group leader exits, leaving the threads alone.
> > +	 */
> > +	*child_exit_secs = time(NULL);
> > +	syscall(SYS_exit, 0);
> > +}
> > +
> > +int test_pidfd_poll_leader_exit(int use_waitpid)
> > +{
> > +	int pid, pidfd = 0;
> > +	int status, ret;
> > +	time_t prog_start = time(NULL);
> > +	const char *test_name = "pidfd_poll check for premature notification on non-empty"
> > +				"group leader exit";
> > +
> > +	child_exit_secs = mmap(NULL, sizeof *child_exit_secs, PROT_READ | PROT_WRITE,
> > +			MAP_SHARED | MAP_ANONYMOUS, -1, 0);
> > +
> > +	ksft_print_msg("Parent: pid: %d\n", getpid());
> > +	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_leader_exit_test);
> 
> Same problem here, I think.


All comments address and fixed in the next revision, thanks!

 - Joel

[PATCH v1 2/2] Add selftests for pidfd polling

[Joel Fernandes]

On Thu, Apr 25, 2019@02:00:35PM -0600, Tycho Andersen wrote:
> On Thu, Apr 25, 2019@03:00:10PM -0400, Joel Fernandes (Google) wrote:
> >
> > +void *test_pidfd_poll_exec_thread(void *priv)
> 
> I think everything in this file can be static, there's this one and
> 3-4 below.
> 
> > +int test_pidfd_poll_exec(int use_waitpid)
> > +{
> > +	int pid, pidfd = 0;
> > +	int status, ret;
> > +	pthread_t t1;
> > +	time_t prog_start = time(NULL);
> > +	const char *test_name = "pidfd_poll check for premature notification on child thread exec";
> > +
> > +	ksft_print_msg("Parent: pid: %d\n", getpid());
> > +	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_exec_test);
> 
> If pidfd_clone() fails here, I think things will go haywire below.
> 
> > +	ksft_print_msg("Parent: Waiting for Child (%d) to complete.\n", pid);
> > +
> > +	if (use_waitpid) {
> > +		ret = waitpid(pid, &status, 0);
> > +		if (ret == -1)
> > +			ksft_print_msg("Parent: error\n");
> > +
> > +		if (ret == pid)
> > +			ksft_print_msg("Parent: Child process waited for.\n");
> > +	} else {
> > +		poll_pidfd(test_name, pidfd);
> > +	}
> > +
> > +	time_t prog_time = time(NULL) - prog_start;
> > +
> > +	ksft_print_msg("Time waited for child: %lu\n", prog_time);
> > +
> > +	close(pidfd);
> > +
> > +	if (prog_time < CHILD_THREAD_MIN_WAIT || prog_time > CHILD_THREAD_MIN_WAIT + 2)
> > +		ksft_exit_fail_msg("%s test: Failed\n", test_name);
> > +	else
> > +		ksft_test_result_pass("%s test: Passed\n", test_name);
> > +}
> > +
> > +void *test_pidfd_poll_leader_exit_thread(void *priv)
> > +{
> > +	char waittime[256];
> > +
> > +	ksft_print_msg("Child Thread: starting. pid %d tid %d ; and sleeping\n",
> > +			getpid(), syscall(SYS_gettid));
> > +	sleep(CHILD_THREAD_MIN_WAIT);
> > +	ksft_print_msg("Child Thread: DONE. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
> > +	return NULL;
> > +}
> > +
> > +static time_t *child_exit_secs;
> > +static int child_poll_leader_exit_test(void *args)
> > +{
> > +	pthread_t t1, t2;
> > +
> > +	ksft_print_msg("Child: starting. pid %d tid %d\n", getpid(), syscall(SYS_gettid));
> > +	pthread_create(&t1, NULL, test_pidfd_poll_leader_exit_thread, NULL);
> > +	pthread_create(&t2, NULL, test_pidfd_poll_leader_exit_thread, NULL);
> > +
> > +	/*
> > +	 * glibc exit calls exit_group syscall, so explicity call exit only
> > +	 * so that only the group leader exits, leaving the threads alone.
> > +	 */
> > +	*child_exit_secs = time(NULL);
> > +	syscall(SYS_exit, 0);
> > +}
> > +
> > +int test_pidfd_poll_leader_exit(int use_waitpid)
> > +{
> > +	int pid, pidfd = 0;
> > +	int status, ret;
> > +	time_t prog_start = time(NULL);
> > +	const char *test_name = "pidfd_poll check for premature notification on non-empty"
> > +				"group leader exit";
> > +
> > +	child_exit_secs = mmap(NULL, sizeof *child_exit_secs, PROT_READ | PROT_WRITE,
> > +			MAP_SHARED | MAP_ANONYMOUS, -1, 0);
> > +
> > +	ksft_print_msg("Parent: pid: %d\n", getpid());
> > +	pid = pidfd_clone(CLONE_PIDFD, &pidfd, child_poll_leader_exit_test);
> 
> Same problem here, I think.


All comments address and fixed in the next revision, thanks!

 - Joel

[PATCH v1 1/2] Add polling support to pidfd

[joel]

On Fri, Apr 26, 2019 at 12:24:04AM +0200, Christian Brauner wrote:
> On Thu, Apr 25, 2019 at 03:00:09PM -0400, Joel Fernandes (Google) wrote:
> > pidfd are file descriptors referring to a process created with the
> > CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs pidfd
> > polling support to replace code that currently checks for existence of
> > /proc/pid for knowing that a process that is signalled to be killed has
> > died, which is both racy and slow. The pidfd poll approach is race-free,
> > and also allows the LMK to do other things (such as by polling on other
> > fds) while awaiting the process being killed to die.
> 
> Thanks for the patch!
> 
> Ok, let me be a little bit anal.
> Please start the commit message with what this patch does and then add

The subject title is "Add polling support to pidfd", but ok I should write a
better commit message.

> the justification why. You just say the "pidfd-poll" approach. You can
> probably assume that CLONE_PIDFD is available for this patch. So
> something like:
> 
> "This patch makes pidfds pollable. Specifically, it allows listeners to
> be informed when the process the pidfd referes to exits. This patch only
> introduces the ability to poll thread-group leaders since pidfds
> currently can only reference those..."
> 
> Then justify the use-case and then go into implementation details.
> That's usually how I would think about this:
> - Change the codebase to do X
> - Why do we need X
> - Are there any technical details worth mentioning in the commit message
> (- Are there any controversial points that people stumbled upon but that
>   have been settled sufficiently.)

Generally the "how" in the patch should be in the code, but ok.

I changed the first 3 paragraphs of the changelog to the following, is that
better? :

Android low memory killer (LMK) needs to know when a process dies once
it is sent the kill signal. It does so by checking for the existence of
/proc/pid which is both racy and slow. For example, if a PID is reused
between when LMK sends a kill signal and checks for existence of the
PID, since the wrong PID is now possibly checked for existence.

This patch adds polling support to pidfd. Using the polling support, LMK
will be able to get notified when a process exists in race-free and fast
way, and allows the LMK to do other things (such as by polling on other
fds) while awaiting the process being killed to die.

For notification to polling processes, we follow the same existing
mechanism in the kernel used when the parent of the task group is to be
notified of a child's death (do_notify_parent).  This is precisely when
the tasks waiting on a poll of pidfd are also awakened in this patch.

> > pidfd are file descriptors referring to a process created with the
> > CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs pidfd
> > polling support to replace code that currently checks for existence of
> > /proc/pid for knowing that a process that is signalled to be killed has
> > died, which is both racy and slow. The pidfd poll approach is race-free,
> > and also allows the LMK to do other things (such as by polling on other
> > fds) while awaiting the process being killed to die.
> 
> > 
> > It prevents a situation where a PID is reused between when LMK sends a
> > kill signal and checks for existence of the PID, since the wrong PID is
> > now possibly checked for existence.
> > 
> > In this patch, we follow the same existing mechanism in the kernel used
> > when the parent of the task group is to be notified (do_notify_parent).
> > This is when the tasks waiting on a poll of pidfd are also awakened.
> > 
> > We have decided to include the waitqueue in struct pid for the following
> > reasons:
> > 1. The wait queue has to survive for the lifetime of the poll. Including
> > it in task_struct would not be option in this case because the task can
> > be reaped and destroyed before the poll returns.
> > 
> > 2. By including the struct pid for the waitqueue means that during
> > de_thread(), the new thread group leader automatically gets the new
> > waitqueue/pid even though its task_struct is different.
> > 
> > Appropriate test cases are added in the second patch to provide coverage
> > of all the cases the patch is handling.
> > 
> > Andy had a similar patch [1] in the past which was a good reference
> > however this patch tries to handle different situations properly related
> > to thread group existence, and how/where it notifies. And also solves
> > other bugs (waitqueue lifetime).  Daniel had a similar patch [2]
> > recently which this patch supercedes.
> > 
> > [1] https://lore.kernel.org/patchwork/patch/345098/
> > [2] https://lore.kernel.org/lkml/20181029175322.189042-1-dancol at google.com/
> > 
> > Cc: luto at amacapital.net
> > Cc: rostedt at goodmis.org
> > Cc: dancol at google.com
> > Cc: sspatil at google.com
> > Cc: christian at brauner.io
> > Cc: jannh at google.com
> > Cc: surenb at google.com
> > Cc: timmurray at google.com
> > Cc: Jonathan Kowalski <bl0pbl33p at gmail.com>
> > Cc: torvalds at linux-foundation.org
> > Cc: kernel-team at android.com
> 
> These should all be in the form:
> 
> Cc: Firstname Lastname <email at address.com>

If this bothers you too much, I can also just remove the CC list from the
changelog here, and include it in my invocation of git-send-email instead..
but I have seen commits in the tree that don't follow this rule.

> 
> There are people missing from the Cc that really should be there...

If you look at the CC list of the email, people in the get_maintainer.pl
script were also added. I did run get_maintainer.pl and checkpatch. But ok, I
will add the folks you are suggesting as well. Thanks.

> Even though he usually doesn't respond that often, please Cc Al on this.
> If he responds it's usually rather important.

No issues on that, but I am wondering if he should also be in MAINTAINERS
file somewhere such that get_maintainer.pl does pick him up. I added him.

> Oleg has reviewed your RFC patch quite substantially and given valuable
> feedback and has an opinion on this thing and is best acquainted with
> the exit code. So please add him to the Cc of the commit message in the
> appropriate form and also add him to the Cc of the thread.

Done.

> Probably also want linux-api for good measure since a lot of people are
> subscribed that would care about pollable pidfds. I'd also add Kees
> since he had some interest in this work and David (Howells).

Done, I added all of them and CC will go out to them next time. Thanks.

> 
> > Co-developed-by: Daniel Colascione <dancol at google.com>
> 
> Every CDB needs to give a SOB as well.

Ok, done. thanks.

> 
> > Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>
> > 
> > ---
> > 
> > RFC -> v1:
> > * Based on CLONE_PIDFD patches: https://lwn.net/Articles/786244/
> > * Updated selftests.
> > * Renamed poll wake function to do_notify_pidfd.
> > * Removed depending on EXIT flags
> > * Removed POLLERR flag since semantics are controversial and
> >   we don't have usecases for it right now (later we can add if there's
> >   a need for it).
> > 
> >  include/linux/pid.h |  3 +++
> >  kernel/fork.c       | 33 +++++++++++++++++++++++++++++++++
> >  kernel/pid.c        |  2 ++
> >  kernel/signal.c     | 14 ++++++++++++++
> >  4 files changed, 52 insertions(+)
> > 
> > diff --git a/include/linux/pid.h b/include/linux/pid.h
> > index 3c8ef5a199ca..1484db6ca8d1 100644
> > --- a/include/linux/pid.h
> > +++ b/include/linux/pid.h
> > @@ -3,6 +3,7 @@
> >  #define _LINUX_PID_H
> >  
> >  #include <linux/rculist.h>
> > +#include <linux/wait.h>
> >  
> >  enum pid_type
> >  {
> > @@ -60,6 +61,8 @@ struct pid
> >  	unsigned int level;
> >  	/* lists of tasks that use this pid */
> >  	struct hlist_head tasks[PIDTYPE_MAX];
> > +	/* wait queue for pidfd notifications */
> > +	wait_queue_head_t wait_pidfd;
> >  	struct rcu_head rcu;
> >  	struct upid numbers[1];
> >  };
> > diff --git a/kernel/fork.c b/kernel/fork.c
> > index 5525837ed80e..fb3b614f6456 100644
> > --- a/kernel/fork.c
> > +++ b/kernel/fork.c
> > @@ -1685,8 +1685,41 @@ static void pidfd_show_fdinfo(struct seq_file *m, struct file *f)
> >  }
> >  #endif
> >  
> > +static unsigned int pidfd_poll(struct file *file, struct poll_table_struct *pts)
> > +{
> > +	struct task_struct *task;
> > +	struct pid *pid;
> > +	int poll_flags = 0;
> > +
> > +	/*
> > +	 * tasklist_lock must be held because to avoid racing with
> > +	 * changes in exit_state and wake up. Basically to avoid:
> > +	 *
> > +	 * P0: read exit_state = 0
> > +	 * P1: write exit_state = EXIT_DEAD
> > +	 * P1: Do a wake up - wq is empty, so do nothing
> > +	 * P0: Queue for polling - wait forever.
> > +	 */
> > +	read_lock(&tasklist_lock);
> > +	pid = file->private_data;
> > +	task = pid_task(pid, PIDTYPE_PID);
> > +	WARN_ON_ONCE(task && !thread_group_leader(task));
> > +
> > +	if (!task || (task->exit_state && thread_group_empty(task)))
> > +		poll_flags = POLLIN | POLLRDNORM;
> 
> So we block until the thread-group is empty? Hm, the thread-group leader
> remains in zombie state until all threads are gone. Should probably just
> be a short comment somewhere that callers are only informed about a
> whole thread-group exit and not about when the thread-group leader has
> actually exited.

Ok, I'll add a comment.

> I would like the ability to extend this interface in the future to allow
> for actually reading data from the pidfd on EPOLLIN.
> POSIX specifies that POLLIN and POLLRDNORM are set even if the
> message to be read is zero. So one cheap way of doing this would
> probably be to do a 0 read/ioctl. That wouldn't hurt your very limited
> usecase and people could test whether the read returned non-0 data and
> if so they know this interface got extended. If we never extend it here
> it won't matter.

I am a bit confused. What specific changes to this patch are you proposing?
This patch makes poll block until the process exits. In the future, we can
make it unblock for a other reasons as well, that's fine with me. I don't see
how this patch prevents such extensions.

> > +	if (!poll_flags)
> > +		poll_wait(file, &pid->wait_pidfd, pts);
> > +
> > +	read_unlock(&tasklist_lock);
> > +
> > +	return poll_flags;
> > +}
> 
> 
> > +
> > +
> >  const struct file_operations pidfd_fops = {
> >  	.release = pidfd_release,
> > +	.poll = pidfd_poll,
> >  #ifdef CONFIG_PROC_FS
> >  	.show_fdinfo = pidfd_show_fdinfo,
> >  #endif
> > diff --git a/kernel/pid.c b/kernel/pid.c
> > index 20881598bdfa..5c90c239242f 100644
> > --- a/kernel/pid.c
> > +++ b/kernel/pid.c
> > @@ -214,6 +214,8 @@ struct pid *alloc_pid(struct pid_namespace *ns)
> >  	for (type = 0; type < PIDTYPE_MAX; ++type)
> >  		INIT_HLIST_HEAD(&pid->tasks[type]);
> >  
> > +	init_waitqueue_head(&pid->wait_pidfd);
> > +
> >  	upid = pid->numbers + ns->level;
> >  	spin_lock_irq(&pidmap_lock);
> >  	if (!(ns->pid_allocated & PIDNS_ADDING))
> > diff --git a/kernel/signal.c b/kernel/signal.c
> > index 1581140f2d99..16e7718316e5 100644
> > --- a/kernel/signal.c
> > +++ b/kernel/signal.c
> > @@ -1800,6 +1800,17 @@ int send_sigqueue(struct sigqueue *q, struct pid *pid, enum pid_type type)
> >  	return ret;
> >  }
> >  
> > +static void do_notify_pidfd(struct task_struct *task)
> 
> Maybe a short command that this helper can only be called when we know
> that task is a thread-group leader wouldn't hurt so there's no confusion
> later.

Ok, will do.

thanks,

 - Joel

[PATCH v1 1/2] Add polling support to pidfd

[Joel Fernandes]

On Fri, Apr 26, 2019@12:24:04AM +0200, Christian Brauner wrote:
> On Thu, Apr 25, 2019@03:00:09PM -0400, Joel Fernandes (Google) wrote:
> > pidfd are file descriptors referring to a process created with the
> > CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs pidfd
> > polling support to replace code that currently checks for existence of
> > /proc/pid for knowing that a process that is signalled to be killed has
> > died, which is both racy and slow. The pidfd poll approach is race-free,
> > and also allows the LMK to do other things (such as by polling on other
> > fds) while awaiting the process being killed to die.
> 
> Thanks for the patch!
> 
> Ok, let me be a little bit anal.
> Please start the commit message with what this patch does and then add

The subject title is "Add polling support to pidfd", but ok I should write a
better commit message.

> the justification why. You just say the "pidfd-poll" approach. You can
> probably assume that CLONE_PIDFD is available for this patch. So
> something like:
> 
> "This patch makes pidfds pollable. Specifically, it allows listeners to
> be informed when the process the pidfd referes to exits. This patch only
> introduces the ability to poll thread-group leaders since pidfds
> currently can only reference those..."
> 
> Then justify the use-case and then go into implementation details.
> That's usually how I would think about this:
> - Change the codebase to do X
> - Why do we need X
> - Are there any technical details worth mentioning in the commit message
> (- Are there any controversial points that people stumbled upon but that
>   have been settled sufficiently.)

Generally the "how" in the patch should be in the code, but ok.

I changed the first 3 paragraphs of the changelog to the following, is that
better? :

Android low memory killer (LMK) needs to know when a process dies once
it is sent the kill signal. It does so by checking for the existence of
/proc/pid which is both racy and slow. For example, if a PID is reused
between when LMK sends a kill signal and checks for existence of the
PID, since the wrong PID is now possibly checked for existence.

This patch adds polling support to pidfd. Using the polling support, LMK
will be able to get notified when a process exists in race-free and fast
way, and allows the LMK to do other things (such as by polling on other
fds) while awaiting the process being killed to die.

For notification to polling processes, we follow the same existing
mechanism in the kernel used when the parent of the task group is to be
notified of a child's death (do_notify_parent).  This is precisely when
the tasks waiting on a poll of pidfd are also awakened in this patch.

> > pidfd are file descriptors referring to a process created with the
> > CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs pidfd
> > polling support to replace code that currently checks for existence of
> > /proc/pid for knowing that a process that is signalled to be killed has
> > died, which is both racy and slow. The pidfd poll approach is race-free,
> > and also allows the LMK to do other things (such as by polling on other
> > fds) while awaiting the process being killed to die.
> 
> > 
> > It prevents a situation where a PID is reused between when LMK sends a
> > kill signal and checks for existence of the PID, since the wrong PID is
> > now possibly checked for existence.
> > 
> > In this patch, we follow the same existing mechanism in the kernel used
> > when the parent of the task group is to be notified (do_notify_parent).
> > This is when the tasks waiting on a poll of pidfd are also awakened.
> > 
> > We have decided to include the waitqueue in struct pid for the following
> > reasons:
> > 1. The wait queue has to survive for the lifetime of the poll. Including
> > it in task_struct would not be option in this case because the task can
> > be reaped and destroyed before the poll returns.
> > 
> > 2. By including the struct pid for the waitqueue means that during
> > de_thread(), the new thread group leader automatically gets the new
> > waitqueue/pid even though its task_struct is different.
> > 
> > Appropriate test cases are added in the second patch to provide coverage
> > of all the cases the patch is handling.
> > 
> > Andy had a similar patch [1] in the past which was a good reference
> > however this patch tries to handle different situations properly related
> > to thread group existence, and how/where it notifies. And also solves
> > other bugs (waitqueue lifetime).  Daniel had a similar patch [2]
> > recently which this patch supercedes.
> > 
> > [1] https://lore.kernel.org/patchwork/patch/345098/
> > [2] https://lore.kernel.org/lkml/20181029175322.189042-1-dancol at google.com/
> > 
> > Cc: luto at amacapital.net
> > Cc: rostedt at goodmis.org
> > Cc: dancol at google.com
> > Cc: sspatil at google.com
> > Cc: christian at brauner.io
> > Cc: jannh at google.com
> > Cc: surenb at google.com
> > Cc: timmurray at google.com
> > Cc: Jonathan Kowalski <bl0pbl33p at gmail.com>
> > Cc: torvalds at linux-foundation.org
> > Cc: kernel-team at android.com
> 
> These should all be in the form:
> 
> Cc: Firstname Lastname <email at address.com>

If this bothers you too much, I can also just remove the CC list from the
changelog here, and include it in my invocation of git-send-email instead..
but I have seen commits in the tree that don't follow this rule.

> 
> There are people missing from the Cc that really should be there...

If you look at the CC list of the email, people in the get_maintainer.pl
script were also added. I did run get_maintainer.pl and checkpatch. But ok, I
will add the folks you are suggesting as well. Thanks.

> Even though he usually doesn't respond that often, please Cc Al on this.
> If he responds it's usually rather important.

No issues on that, but I am wondering if he should also be in MAINTAINERS
file somewhere such that get_maintainer.pl does pick him up. I added him.

> Oleg has reviewed your RFC patch quite substantially and given valuable
> feedback and has an opinion on this thing and is best acquainted with
> the exit code. So please add him to the Cc of the commit message in the
> appropriate form and also add him to the Cc of the thread.

Done.

> Probably also want linux-api for good measure since a lot of people are
> subscribed that would care about pollable pidfds. I'd also add Kees
> since he had some interest in this work and David (Howells).

Done, I added all of them and CC will go out to them next time. Thanks.

> 
> > Co-developed-by: Daniel Colascione <dancol at google.com>
> 
> Every CDB needs to give a SOB as well.

Ok, done. thanks.

> 
> > Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>
> > 
> > ---
> > 
> > RFC -> v1:
> > * Based on CLONE_PIDFD patches: https://lwn.net/Articles/786244/
> > * Updated selftests.
> > * Renamed poll wake function to do_notify_pidfd.
> > * Removed depending on EXIT flags
> > * Removed POLLERR flag since semantics are controversial and
> >   we don't have usecases for it right now (later we can add if there's
> >   a need for it).
> > 
> >  include/linux/pid.h |  3 +++
> >  kernel/fork.c       | 33 +++++++++++++++++++++++++++++++++
> >  kernel/pid.c        |  2 ++
> >  kernel/signal.c     | 14 ++++++++++++++
> >  4 files changed, 52 insertions(+)
> > 
> > diff --git a/include/linux/pid.h b/include/linux/pid.h
> > index 3c8ef5a199ca..1484db6ca8d1 100644
> > --- a/include/linux/pid.h
> > +++ b/include/linux/pid.h
> > @@ -3,6 +3,7 @@
> >  #define _LINUX_PID_H
> >  
> >  #include <linux/rculist.h>
> > +#include <linux/wait.h>
> >  
> >  enum pid_type
> >  {
> > @@ -60,6 +61,8 @@ struct pid
> >  	unsigned int level;
> >  	/* lists of tasks that use this pid */
> >  	struct hlist_head tasks[PIDTYPE_MAX];
> > +	/* wait queue for pidfd notifications */
> > +	wait_queue_head_t wait_pidfd;
> >  	struct rcu_head rcu;
> >  	struct upid numbers[1];
> >  };
> > diff --git a/kernel/fork.c b/kernel/fork.c
> > index 5525837ed80e..fb3b614f6456 100644
> > --- a/kernel/fork.c
> > +++ b/kernel/fork.c
> > @@ -1685,8 +1685,41 @@ static void pidfd_show_fdinfo(struct seq_file *m, struct file *f)
> >  }
> >  #endif
> >  
> > +static unsigned int pidfd_poll(struct file *file, struct poll_table_struct *pts)
> > +{
> > +	struct task_struct *task;
> > +	struct pid *pid;
> > +	int poll_flags = 0;
> > +
> > +	/*
> > +	 * tasklist_lock must be held because to avoid racing with
> > +	 * changes in exit_state and wake up. Basically to avoid:
> > +	 *
> > +	 * P0: read exit_state = 0
> > +	 * P1: write exit_state = EXIT_DEAD
> > +	 * P1: Do a wake up - wq is empty, so do nothing
> > +	 * P0: Queue for polling - wait forever.
> > +	 */
> > +	read_lock(&tasklist_lock);
> > +	pid = file->private_data;
> > +	task = pid_task(pid, PIDTYPE_PID);
> > +	WARN_ON_ONCE(task && !thread_group_leader(task));
> > +
> > +	if (!task || (task->exit_state && thread_group_empty(task)))
> > +		poll_flags = POLLIN | POLLRDNORM;
> 
> So we block until the thread-group is empty? Hm, the thread-group leader
> remains in zombie state until all threads are gone. Should probably just
> be a short comment somewhere that callers are only informed about a
> whole thread-group exit and not about when the thread-group leader has
> actually exited.

Ok, I'll add a comment.

> I would like the ability to extend this interface in the future to allow
> for actually reading data from the pidfd on EPOLLIN.
> POSIX specifies that POLLIN and POLLRDNORM are set even if the
> message to be read is zero. So one cheap way of doing this would
> probably be to do a 0 read/ioctl. That wouldn't hurt your very limited
> usecase and people could test whether the read returned non-0 data and
> if so they know this interface got extended. If we never extend it here
> it won't matter.

I am a bit confused. What specific changes to this patch are you proposing?
This patch makes poll block until the process exits. In the future, we can
make it unblock for a other reasons as well, that's fine with me. I don't see
how this patch prevents such extensions.

> > +	if (!poll_flags)
> > +		poll_wait(file, &pid->wait_pidfd, pts);
> > +
> > +	read_unlock(&tasklist_lock);
> > +
> > +	return poll_flags;
> > +}
> 
> 
> > +
> > +
> >  const struct file_operations pidfd_fops = {
> >  	.release = pidfd_release,
> > +	.poll = pidfd_poll,
> >  #ifdef CONFIG_PROC_FS
> >  	.show_fdinfo = pidfd_show_fdinfo,
> >  #endif
> > diff --git a/kernel/pid.c b/kernel/pid.c
> > index 20881598bdfa..5c90c239242f 100644
> > --- a/kernel/pid.c
> > +++ b/kernel/pid.c
> > @@ -214,6 +214,8 @@ struct pid *alloc_pid(struct pid_namespace *ns)
> >  	for (type = 0; type < PIDTYPE_MAX; ++type)
> >  		INIT_HLIST_HEAD(&pid->tasks[type]);
> >  
> > +	init_waitqueue_head(&pid->wait_pidfd);
> > +
> >  	upid = pid->numbers + ns->level;
> >  	spin_lock_irq(&pidmap_lock);
> >  	if (!(ns->pid_allocated & PIDNS_ADDING))
> > diff --git a/kernel/signal.c b/kernel/signal.c
> > index 1581140f2d99..16e7718316e5 100644
> > --- a/kernel/signal.c
> > +++ b/kernel/signal.c
> > @@ -1800,6 +1800,17 @@ int send_sigqueue(struct sigqueue *q, struct pid *pid, enum pid_type type)
> >  	return ret;
> >  }
> >  
> > +static void do_notify_pidfd(struct task_struct *task)
> 
> Maybe a short command that this helper can only be called when we know
> that task is a thread-group leader wouldn't hurt so there's no confusion
> later.

Ok, will do.

thanks,

 - Joel

[PATCH v1 1/2] Add polling support to pidfd

[christian]

On Thu, Apr 25, 2019 at 03:00:09PM -0400, Joel Fernandes (Google) wrote:
> pidfd are file descriptors referring to a process created with the
> CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs pidfd
> polling support to replace code that currently checks for existence of
> /proc/pid for knowing that a process that is signalled to be killed has
> died, which is both racy and slow. The pidfd poll approach is race-free,
> and also allows the LMK to do other things (such as by polling on other
> fds) while awaiting the process being killed to die.
> 
> It prevents a situation where a PID is reused between when LMK sends a
> kill signal and checks for existence of the PID, since the wrong PID is
> now possibly checked for existence.
> 
> In this patch, we follow the same existing mechanism in the kernel used
> when the parent of the task group is to be notified (do_notify_parent).
> This is when the tasks waiting on a poll of pidfd are also awakened.
> 
> We have decided to include the waitqueue in struct pid for the following
> reasons:
> 1. The wait queue has to survive for the lifetime of the poll. Including
> it in task_struct would not be option in this case because the task can
> be reaped and destroyed before the poll returns.
> 
> 2. By including the struct pid for the waitqueue means that during
> de_thread(), the new thread group leader automatically gets the new
> waitqueue/pid even though its task_struct is different.
> 
> Appropriate test cases are added in the second patch to provide coverage
> of all the cases the patch is handling.
> 
> Andy had a similar patch [1] in the past which was a good reference
> however this patch tries to handle different situations properly related
> to thread group existence, and how/where it notifies. And also solves
> other bugs (waitqueue lifetime).  Daniel had a similar patch [2]
> recently which this patch supercedes.
> 
> [1] https://lore.kernel.org/patchwork/patch/345098/
> [2] https://lore.kernel.org/lkml/20181029175322.189042-1-dancol at google.com/
> 
> Cc: luto at amacapital.net
> Cc: rostedt at goodmis.org
> Cc: dancol at google.com
> Cc: sspatil at google.com
> Cc: christian at brauner.io
> Cc: jannh at google.com
> Cc: surenb at google.com
> Cc: timmurray at google.com
> Cc: Jonathan Kowalski <bl0pbl33p at gmail.com>
> Cc: torvalds at linux-foundation.org
> Cc: kernel-team at android.com

That should be of the form:

Cc: First Name <email at address.com>


> Co-developed-by: Daniel Colascione <dancol at google.com>

Every CDB needs to come with a SOB.

> Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>
> 
> ---
> 
> RFC -> v1:
> * Based on CLONE_PIDFD patches: https://lwn.net/Articles/786244/
> * Updated selftests.
> * Renamed poll wake function to do_notify_pidfd.
> * Removed depending on EXIT flags
> * Removed POLLERR flag since semantics are controversial and
>   we don't have usecases for it right now (later we can add if there's
>   a need for it).
> 
>  include/linux/pid.h |  3 +++
>  kernel/fork.c       | 33 +++++++++++++++++++++++++++++++++
>  kernel/pid.c        |  2 ++
>  kernel/signal.c     | 14 ++++++++++++++
>  4 files changed, 52 insertions(+)
> 
> diff --git a/include/linux/pid.h b/include/linux/pid.h
> index 3c8ef5a199ca..1484db6ca8d1 100644
> --- a/include/linux/pid.h
> +++ b/include/linux/pid.h
> @@ -3,6 +3,7 @@
>  #define _LINUX_PID_H
>  
>  #include <linux/rculist.h>
> +#include <linux/wait.h>
>  
>  enum pid_type
>  {
> @@ -60,6 +61,8 @@ struct pid
>  	unsigned int level;
>  	/* lists of tasks that use this pid */
>  	struct hlist_head tasks[PIDTYPE_MAX];
> +	/* wait queue for pidfd notifications */
> +	wait_queue_head_t wait_pidfd;
>  	struct rcu_head rcu;
>  	struct upid numbers[1];
>  };
> diff --git a/kernel/fork.c b/kernel/fork.c
> index 5525837ed80e..fb3b614f6456 100644
> --- a/kernel/fork.c
> +++ b/kernel/fork.c
> @@ -1685,8 +1685,41 @@ static void pidfd_show_fdinfo(struct seq_file *m, struct file *f)
>  }
>  #endif
>  
> +static unsigned int pidfd_poll(struct file *file, struct poll_table_struct *pts)
> +{
> +	struct task_struct *task;
> +	struct pid *pid;
> +	int poll_flags = 0;
> +
> +	/*
> +	 * tasklist_lock must be held because to avoid racing with
> +	 * changes in exit_state and wake up. Basically to avoid:
> +	 *
> +	 * P0: read exit_state = 0
> +	 * P1: write exit_state = EXIT_DEAD
> +	 * P1: Do a wake up - wq is empty, so do nothing
> +	 * P0: Queue for polling - wait forever.
> +	 */
> +	read_lock(&tasklist_lock);
> +	pid = file->private_data;
> +	task = pid_task(pid, PIDTYPE_PID);
> +	WARN_ON_ONCE(task && !thread_group_leader(task));
> +
> +	if (!task || (task->exit_state && thread_group_empty(task)))
> +		poll_flags = POLLIN | POLLRDNORM;
> +
> +	if (!poll_flags)
> +		poll_wait(file, &pid->wait_pidfd, pts);
> +
> +	read_unlock(&tasklist_lock);
> +
> +	return poll_flags;
> +}
> +
> +
>  const struct file_operations pidfd_fops = {
>  	.release = pidfd_release,
> +	.poll = pidfd_poll,
>  #ifdef CONFIG_PROC_FS
>  	.show_fdinfo = pidfd_show_fdinfo,
>  #endif
> diff --git a/kernel/pid.c b/kernel/pid.c
> index 20881598bdfa..5c90c239242f 100644
> --- a/kernel/pid.c
> +++ b/kernel/pid.c
> @@ -214,6 +214,8 @@ struct pid *alloc_pid(struct pid_namespace *ns)
>  	for (type = 0; type < PIDTYPE_MAX; ++type)
>  		INIT_HLIST_HEAD(&pid->tasks[type]);
>  
> +	init_waitqueue_head(&pid->wait_pidfd);
> +
>  	upid = pid->numbers + ns->level;
>  	spin_lock_irq(&pidmap_lock);
>  	if (!(ns->pid_allocated & PIDNS_ADDING))
> diff --git a/kernel/signal.c b/kernel/signal.c
> index 1581140f2d99..16e7718316e5 100644
> --- a/kernel/signal.c
> +++ b/kernel/signal.c
> @@ -1800,6 +1800,17 @@ int send_sigqueue(struct sigqueue *q, struct pid *pid, enum pid_type type)
>  	return ret;
>  }
>  
> +static void do_notify_pidfd(struct task_struct *task)
> +{
> +	struct pid *pid;
> +
> +	lockdep_assert_held(&tasklist_lock);
> +
> +	pid = get_task_pid(task, PIDTYPE_PID);
> +	wake_up_all(&pid->wait_pidfd);
> +	put_pid(pid);
> +}
> +
>  /*
>   * Let a parent know about the death of a child.
>   * For a stopped/continued status change, use do_notify_parent_cldstop instead.
> @@ -1823,6 +1834,9 @@ bool do_notify_parent(struct task_struct *tsk, int sig)
>  	BUG_ON(!tsk->ptrace &&
>  	       (tsk->group_leader != tsk || !thread_group_empty(tsk)));
>  
> +	/* Wake up all pidfd waiters */
> +	do_notify_pidfd(tsk);
> +
>  	if (sig != SIGCHLD) {
>  		/*
>  		 * This is only possible if parent == real_parent.
> -- 
> 2.21.0.593.g511ec345e18-goog

[PATCH v1 1/2] Add polling support to pidfd

[Christian Brauner]

On Thu, Apr 25, 2019@03:00:09PM -0400, Joel Fernandes (Google) wrote:
> pidfd are file descriptors referring to a process created with the
> CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs pidfd
> polling support to replace code that currently checks for existence of
> /proc/pid for knowing that a process that is signalled to be killed has
> died, which is both racy and slow. The pidfd poll approach is race-free,
> and also allows the LMK to do other things (such as by polling on other
> fds) while awaiting the process being killed to die.
> 
> It prevents a situation where a PID is reused between when LMK sends a
> kill signal and checks for existence of the PID, since the wrong PID is
> now possibly checked for existence.
> 
> In this patch, we follow the same existing mechanism in the kernel used
> when the parent of the task group is to be notified (do_notify_parent).
> This is when the tasks waiting on a poll of pidfd are also awakened.
> 
> We have decided to include the waitqueue in struct pid for the following
> reasons:
> 1. The wait queue has to survive for the lifetime of the poll. Including
> it in task_struct would not be option in this case because the task can
> be reaped and destroyed before the poll returns.
> 
> 2. By including the struct pid for the waitqueue means that during
> de_thread(), the new thread group leader automatically gets the new
> waitqueue/pid even though its task_struct is different.
> 
> Appropriate test cases are added in the second patch to provide coverage
> of all the cases the patch is handling.
> 
> Andy had a similar patch [1] in the past which was a good reference
> however this patch tries to handle different situations properly related
> to thread group existence, and how/where it notifies. And also solves
> other bugs (waitqueue lifetime).  Daniel had a similar patch [2]
> recently which this patch supercedes.
> 
> [1] https://lore.kernel.org/patchwork/patch/345098/
> [2] https://lore.kernel.org/lkml/20181029175322.189042-1-dancol at google.com/
> 
> Cc: luto at amacapital.net
> Cc: rostedt at goodmis.org
> Cc: dancol at google.com
> Cc: sspatil at google.com
> Cc: christian at brauner.io
> Cc: jannh at google.com
> Cc: surenb at google.com
> Cc: timmurray at google.com
> Cc: Jonathan Kowalski <bl0pbl33p at gmail.com>
> Cc: torvalds at linux-foundation.org
> Cc: kernel-team at android.com

That should be of the form:

Cc: First Name <email at address.com>


> Co-developed-by: Daniel Colascione <dancol at google.com>

Every CDB needs to come with a SOB.

> Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>
> 
> ---
> 
> RFC -> v1:
> * Based on CLONE_PIDFD patches: https://lwn.net/Articles/786244/
> * Updated selftests.
> * Renamed poll wake function to do_notify_pidfd.
> * Removed depending on EXIT flags
> * Removed POLLERR flag since semantics are controversial and
>   we don't have usecases for it right now (later we can add if there's
>   a need for it).
> 
>  include/linux/pid.h |  3 +++
>  kernel/fork.c       | 33 +++++++++++++++++++++++++++++++++
>  kernel/pid.c        |  2 ++
>  kernel/signal.c     | 14 ++++++++++++++
>  4 files changed, 52 insertions(+)
> 
> diff --git a/include/linux/pid.h b/include/linux/pid.h
> index 3c8ef5a199ca..1484db6ca8d1 100644
> --- a/include/linux/pid.h
> +++ b/include/linux/pid.h
> @@ -3,6 +3,7 @@
>  #define _LINUX_PID_H
>  
>  #include <linux/rculist.h>
> +#include <linux/wait.h>
>  
>  enum pid_type
>  {
> @@ -60,6 +61,8 @@ struct pid
>  	unsigned int level;
>  	/* lists of tasks that use this pid */
>  	struct hlist_head tasks[PIDTYPE_MAX];
> +	/* wait queue for pidfd notifications */
> +	wait_queue_head_t wait_pidfd;
>  	struct rcu_head rcu;
>  	struct upid numbers[1];
>  };
> diff --git a/kernel/fork.c b/kernel/fork.c
> index 5525837ed80e..fb3b614f6456 100644
> --- a/kernel/fork.c
> +++ b/kernel/fork.c
> @@ -1685,8 +1685,41 @@ static void pidfd_show_fdinfo(struct seq_file *m, struct file *f)
>  }
>  #endif
>  
> +static unsigned int pidfd_poll(struct file *file, struct poll_table_struct *pts)
> +{
> +	struct task_struct *task;
> +	struct pid *pid;
> +	int poll_flags = 0;
> +
> +	/*
> +	 * tasklist_lock must be held because to avoid racing with
> +	 * changes in exit_state and wake up. Basically to avoid:
> +	 *
> +	 * P0: read exit_state = 0
> +	 * P1: write exit_state = EXIT_DEAD
> +	 * P1: Do a wake up - wq is empty, so do nothing
> +	 * P0: Queue for polling - wait forever.
> +	 */
> +	read_lock(&tasklist_lock);
> +	pid = file->private_data;
> +	task = pid_task(pid, PIDTYPE_PID);
> +	WARN_ON_ONCE(task && !thread_group_leader(task));
> +
> +	if (!task || (task->exit_state && thread_group_empty(task)))
> +		poll_flags = POLLIN | POLLRDNORM;
> +
> +	if (!poll_flags)
> +		poll_wait(file, &pid->wait_pidfd, pts);
> +
> +	read_unlock(&tasklist_lock);
> +
> +	return poll_flags;
> +}
> +
> +
>  const struct file_operations pidfd_fops = {
>  	.release = pidfd_release,
> +	.poll = pidfd_poll,
>  #ifdef CONFIG_PROC_FS
>  	.show_fdinfo = pidfd_show_fdinfo,
>  #endif
> diff --git a/kernel/pid.c b/kernel/pid.c
> index 20881598bdfa..5c90c239242f 100644
> --- a/kernel/pid.c
> +++ b/kernel/pid.c
> @@ -214,6 +214,8 @@ struct pid *alloc_pid(struct pid_namespace *ns)
>  	for (type = 0; type < PIDTYPE_MAX; ++type)
>  		INIT_HLIST_HEAD(&pid->tasks[type]);
>  
> +	init_waitqueue_head(&pid->wait_pidfd);
> +
>  	upid = pid->numbers + ns->level;
>  	spin_lock_irq(&pidmap_lock);
>  	if (!(ns->pid_allocated & PIDNS_ADDING))
> diff --git a/kernel/signal.c b/kernel/signal.c
> index 1581140f2d99..16e7718316e5 100644
> --- a/kernel/signal.c
> +++ b/kernel/signal.c
> @@ -1800,6 +1800,17 @@ int send_sigqueue(struct sigqueue *q, struct pid *pid, enum pid_type type)
>  	return ret;
>  }
>  
> +static void do_notify_pidfd(struct task_struct *task)
> +{
> +	struct pid *pid;
> +
> +	lockdep_assert_held(&tasklist_lock);
> +
> +	pid = get_task_pid(task, PIDTYPE_PID);
> +	wake_up_all(&pid->wait_pidfd);
> +	put_pid(pid);
> +}
> +
>  /*
>   * Let a parent know about the death of a child.
>   * For a stopped/continued status change, use do_notify_parent_cldstop instead.
> @@ -1823,6 +1834,9 @@ bool do_notify_parent(struct task_struct *tsk, int sig)
>  	BUG_ON(!tsk->ptrace &&
>  	       (tsk->group_leader != tsk || !thread_group_empty(tsk)));
>  
> +	/* Wake up all pidfd waiters */
> +	do_notify_pidfd(tsk);
> +
>  	if (sig != SIGCHLD) {
>  		/*
>  		 * This is only possible if parent == real_parent.
> -- 
> 2.21.0.593.g511ec345e18-goog

[PATCH v1 1/2] Add polling support to pidfd

[christian]

On Fri, Apr 26, 2019 at 10:23:37AM -0400, Joel Fernandes wrote:
> On Fri, Apr 26, 2019 at 12:24:04AM +0200, Christian Brauner wrote:
> > On Thu, Apr 25, 2019 at 03:00:09PM -0400, Joel Fernandes (Google) wrote:
> > > pidfd are file descriptors referring to a process created with the
> > > CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs pidfd
> > > polling support to replace code that currently checks for existence of
> > > /proc/pid for knowing that a process that is signalled to be killed has
> > > died, which is both racy and slow. The pidfd poll approach is race-free,
> > > and also allows the LMK to do other things (such as by polling on other
> > > fds) while awaiting the process being killed to die.
> > 
> > Thanks for the patch!
> > 
> > Ok, let me be a little bit anal.
> > Please start the commit message with what this patch does and then add
> 
> The subject title is "Add polling support to pidfd", but ok I should write a
> better commit message.

Yeah, it's really just that we should really just have a simple
paragraph that expresses this makes the codebase do X.

> 
> > the justification why. You just say the "pidfd-poll" approach. You can
> > probably assume that CLONE_PIDFD is available for this patch. So
> > something like:
> > 
> > "This patch makes pidfds pollable. Specifically, it allows listeners to
> > be informed when the process the pidfd referes to exits. This patch only
> > introduces the ability to poll thread-group leaders since pidfds
> > currently can only reference those..."
> > 
> > Then justify the use-case and then go into implementation details.
> > That's usually how I would think about this:
> > - Change the codebase to do X
> > - Why do we need X
> > - Are there any technical details worth mentioning in the commit message
> > (- Are there any controversial points that people stumbled upon but that
> >   have been settled sufficiently.)
> 
> Generally the "how" in the patch should be in the code, but ok.

That's why I said: technical details that are worth mentioning.
Sometimes you have controversial bits that are obviously to be
understood in the code but it still might be worth explaining why one
had to do it this way. Like say what we did for the pidfd_send_signal()
thing where we explained why O_PATH is disallowed.

> 
> I changed the first 3 paragraphs of the changelog to the following, is that
> better? :
> 
> Android low memory killer (LMK) needs to know when a process dies once
> it is sent the kill signal. It does so by checking for the existence of
> /proc/pid which is both racy and slow. For example, if a PID is reused
> between when LMK sends a kill signal and checks for existence of the
> PID, since the wrong PID is now possibly checked for existence.
> 
> This patch adds polling support to pidfd. Using the polling support, LMK
> will be able to get notified when a process exists in race-free and fast
> way, and allows the LMK to do other things (such as by polling on other
> fds) while awaiting the process being killed to die.
> 
> For notification to polling processes, we follow the same existing
> mechanism in the kernel used when the parent of the task group is to be
> notified of a child's death (do_notify_parent).  This is precisely when
> the tasks waiting on a poll of pidfd are also awakened in this patch.
> 
> > > pidfd are file descriptors referring to a process created with the
> > > CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs pidfd
> > > polling support to replace code that currently checks for existence of
> > > /proc/pid for knowing that a process that is signalled to be killed has
> > > died, which is both racy and slow. The pidfd poll approach is race-free,
> > > and also allows the LMK to do other things (such as by polling on other
> > > fds) while awaiting the process being killed to die.
> > 
> > > 
> > > It prevents a situation where a PID is reused between when LMK sends a
> > > kill signal and checks for existence of the PID, since the wrong PID is
> > > now possibly checked for existence.
> > > 
> > > In this patch, we follow the same existing mechanism in the kernel used
> > > when the parent of the task group is to be notified (do_notify_parent).
> > > This is when the tasks waiting on a poll of pidfd are also awakened.
> > > 
> > > We have decided to include the waitqueue in struct pid for the following
> > > reasons:
> > > 1. The wait queue has to survive for the lifetime of the poll. Including
> > > it in task_struct would not be option in this case because the task can
> > > be reaped and destroyed before the poll returns.
> > > 
> > > 2. By including the struct pid for the waitqueue means that during
> > > de_thread(), the new thread group leader automatically gets the new
> > > waitqueue/pid even though its task_struct is different.
> > > 
> > > Appropriate test cases are added in the second patch to provide coverage
> > > of all the cases the patch is handling.
> > > 
> > > Andy had a similar patch [1] in the past which was a good reference
> > > however this patch tries to handle different situations properly related
> > > to thread group existence, and how/where it notifies. And also solves
> > > other bugs (waitqueue lifetime).  Daniel had a similar patch [2]
> > > recently which this patch supercedes.
> > > 
> > > [1] https://lore.kernel.org/patchwork/patch/345098/
> > > [2] https://lore.kernel.org/lkml/20181029175322.189042-1-dancol at google.com/
> > > 
> > > Cc: luto at amacapital.net
> > > Cc: rostedt at goodmis.org
> > > Cc: dancol at google.com
> > > Cc: sspatil at google.com
> > > Cc: christian at brauner.io
> > > Cc: jannh at google.com
> > > Cc: surenb at google.com
> > > Cc: timmurray at google.com
> > > Cc: Jonathan Kowalski <bl0pbl33p at gmail.com>
> > > Cc: torvalds at linux-foundation.org
> > > Cc: kernel-team at android.com
> > 
> > These should all be in the form:
> > 
> > Cc: Firstname Lastname <email at address.com>
> 
> If this bothers you too much, I can also just remove the CC list from the
> changelog here, and include it in my invocation of git-send-email instead..
> but I have seen commits in the tree that don't follow this rule.

Yeah, but they should. There are people with multiple emails over the
years and they might not necessarily contain their first and last
name. And I don't want to have to mailmap them or sm. Having their names
in there just makes it easier. Also, every single other DCO-*related*
line follows:

Random J Developer <random at developer.example.org>

This should too. If others are sloppy and allow this, fine. No reason we
should.

> 
> > 
> > There are people missing from the Cc that really should be there...
> 
> If you look at the CC list of the email, people in the get_maintainer.pl
> script were also added. I did run get_maintainer.pl and checkpatch. But ok, I
> will add the folks you are suggesting as well. Thanks.

get_maintainer.pl is not the last word. 

> 
> > Even though he usually doesn't respond that often, please Cc Al on this.
> > If he responds it's usually rather important.
> 
> No issues on that, but I am wondering if he should also be in MAINTAINERS
> file somewhere such that get_maintainer.pl does pick him up. I added him.

It's often not about someone being a maintainer but whether or not they
have valuable input.

"[...] This tag documents that potentially interested parties have been
included in the discussion."

> 
> > Oleg has reviewed your RFC patch quite substantially and given valuable
> > feedback and has an opinion on this thing and is best acquainted with
> > the exit code. So please add him to the Cc of the commit message in the
> > appropriate form and also add him to the Cc of the thread.
> 
> Done.

Thanks!

> 
> > Probably also want linux-api for good measure since a lot of people are
> > subscribed that would care about pollable pidfds. I'd also add Kees
> > since he had some interest in this work and David (Howells).
> 
> Done, I added all of them and CC will go out to them next time. Thanks.

Cool. That really wasn't a "you've done this wrong". It's rather really
just to make sure that everyone who might catch a big f*ck up on our
part has had a chance to tell us so. :)

> 
> > 
> > > Co-developed-by: Daniel Colascione <dancol at google.com>
> > 
> > Every CDB needs to give a SOB as well.
> 
> Ok, done. thanks.

Fwiw, I only learned this recently too.

> 
> > 
> > > Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>
> > > 
> > > ---
> > > 
> > > RFC -> v1:
> > > * Based on CLONE_PIDFD patches: https://lwn.net/Articles/786244/
> > > * Updated selftests.
> > > * Renamed poll wake function to do_notify_pidfd.
> > > * Removed depending on EXIT flags
> > > * Removed POLLERR flag since semantics are controversial and
> > >   we don't have usecases for it right now (later we can add if there's
> > >   a need for it).
> > > 
> > >  include/linux/pid.h |  3 +++
> > >  kernel/fork.c       | 33 +++++++++++++++++++++++++++++++++
> > >  kernel/pid.c        |  2 ++
> > >  kernel/signal.c     | 14 ++++++++++++++
> > >  4 files changed, 52 insertions(+)
> > > 
> > > diff --git a/include/linux/pid.h b/include/linux/pid.h
> > > index 3c8ef5a199ca..1484db6ca8d1 100644
> > > --- a/include/linux/pid.h
> > > +++ b/include/linux/pid.h
> > > @@ -3,6 +3,7 @@
> > >  #define _LINUX_PID_H
> > >  
> > >  #include <linux/rculist.h>
> > > +#include <linux/wait.h>
> > >  
> > >  enum pid_type
> > >  {
> > > @@ -60,6 +61,8 @@ struct pid
> > >  	unsigned int level;
> > >  	/* lists of tasks that use this pid */
> > >  	struct hlist_head tasks[PIDTYPE_MAX];
> > > +	/* wait queue for pidfd notifications */
> > > +	wait_queue_head_t wait_pidfd;
> > >  	struct rcu_head rcu;
> > >  	struct upid numbers[1];
> > >  };
> > > diff --git a/kernel/fork.c b/kernel/fork.c
> > > index 5525837ed80e..fb3b614f6456 100644
> > > --- a/kernel/fork.c
> > > +++ b/kernel/fork.c
> > > @@ -1685,8 +1685,41 @@ static void pidfd_show_fdinfo(struct seq_file *m, struct file *f)
> > >  }
> > >  #endif
> > >  
> > > +static unsigned int pidfd_poll(struct file *file, struct poll_table_struct *pts)
> > > +{
> > > +	struct task_struct *task;
> > > +	struct pid *pid;
> > > +	int poll_flags = 0;
> > > +
> > > +	/*
> > > +	 * tasklist_lock must be held because to avoid racing with
> > > +	 * changes in exit_state and wake up. Basically to avoid:
> > > +	 *
> > > +	 * P0: read exit_state = 0
> > > +	 * P1: write exit_state = EXIT_DEAD
> > > +	 * P1: Do a wake up - wq is empty, so do nothing
> > > +	 * P0: Queue for polling - wait forever.
> > > +	 */
> > > +	read_lock(&tasklist_lock);
> > > +	pid = file->private_data;
> > > +	task = pid_task(pid, PIDTYPE_PID);
> > > +	WARN_ON_ONCE(task && !thread_group_leader(task));
> > > +
> > > +	if (!task || (task->exit_state && thread_group_empty(task)))
> > > +		poll_flags = POLLIN | POLLRDNORM;
> > 
> > So we block until the thread-group is empty? Hm, the thread-group leader
> > remains in zombie state until all threads are gone. Should probably just
> > be a short comment somewhere that callers are only informed about a
> > whole thread-group exit and not about when the thread-group leader has
> > actually exited.
> 
> Ok, I'll add a comment.
> 
> > I would like the ability to extend this interface in the future to allow
> > for actually reading data from the pidfd on EPOLLIN.
> > POSIX specifies that POLLIN and POLLRDNORM are set even if the
> > message to be read is zero. So one cheap way of doing this would
> > probably be to do a 0 read/ioctl. That wouldn't hurt your very limited
> > usecase and people could test whether the read returned non-0 data and
> > if so they know this interface got extended. If we never extend it here
> > it won't matter.
> 
> I am a bit confused. What specific changes to this patch are you proposing?
> This patch makes poll block until the process exits. In the future, we can
> make it unblock for a other reasons as well, that's fine with me. I don't see
> how this patch prevents such extensions.

I guess I should've asked the following:
What happens right now, when you get EPOLLIN on the pidfd and you and
out of ignorance you do:

read(pidfd, ...)

> 
> > > +	if (!poll_flags)
> > > +		poll_wait(file, &pid->wait_pidfd, pts);
> > > +
> > > +	read_unlock(&tasklist_lock);
> > > +
> > > +	return poll_flags;
> > > +}
> > 
> > 
> > > +
> > > +
> > >  const struct file_operations pidfd_fops = {
> > >  	.release = pidfd_release,
> > > +	.poll = pidfd_poll,
> > >  #ifdef CONFIG_PROC_FS
> > >  	.show_fdinfo = pidfd_show_fdinfo,
> > >  #endif
> > > diff --git a/kernel/pid.c b/kernel/pid.c
> > > index 20881598bdfa..5c90c239242f 100644
> > > --- a/kernel/pid.c
> > > +++ b/kernel/pid.c
> > > @@ -214,6 +214,8 @@ struct pid *alloc_pid(struct pid_namespace *ns)
> > >  	for (type = 0; type < PIDTYPE_MAX; ++type)
> > >  		INIT_HLIST_HEAD(&pid->tasks[type]);
> > >  
> > > +	init_waitqueue_head(&pid->wait_pidfd);
> > > +
> > >  	upid = pid->numbers + ns->level;
> > >  	spin_lock_irq(&pidmap_lock);
> > >  	if (!(ns->pid_allocated & PIDNS_ADDING))
> > > diff --git a/kernel/signal.c b/kernel/signal.c
> > > index 1581140f2d99..16e7718316e5 100644
> > > --- a/kernel/signal.c
> > > +++ b/kernel/signal.c
> > > @@ -1800,6 +1800,17 @@ int send_sigqueue(struct sigqueue *q, struct pid *pid, enum pid_type type)
> > >  	return ret;
> > >  }
> > >  
> > > +static void do_notify_pidfd(struct task_struct *task)
> > 
> > Maybe a short command that this helper can only be called when we know
> > that task is a thread-group leader wouldn't hurt so there's no confusion
> > later.
> 
> Ok, will do.
> 
> thanks,
> 
>  - Joel
> 

[PATCH v1 1/2] Add polling support to pidfd

[Christian Brauner]

On Fri, Apr 26, 2019@10:23:37AM -0400, Joel Fernandes wrote:
> On Fri, Apr 26, 2019@12:24:04AM +0200, Christian Brauner wrote:
> > On Thu, Apr 25, 2019@03:00:09PM -0400, Joel Fernandes (Google) wrote:
> > > pidfd are file descriptors referring to a process created with the
> > > CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs pidfd
> > > polling support to replace code that currently checks for existence of
> > > /proc/pid for knowing that a process that is signalled to be killed has
> > > died, which is both racy and slow. The pidfd poll approach is race-free,
> > > and also allows the LMK to do other things (such as by polling on other
> > > fds) while awaiting the process being killed to die.
> > 
> > Thanks for the patch!
> > 
> > Ok, let me be a little bit anal.
> > Please start the commit message with what this patch does and then add
> 
> The subject title is "Add polling support to pidfd", but ok I should write a
> better commit message.

Yeah, it's really just that we should really just have a simple
paragraph that expresses this makes the codebase do X.

> 
> > the justification why. You just say the "pidfd-poll" approach. You can
> > probably assume that CLONE_PIDFD is available for this patch. So
> > something like:
> > 
> > "This patch makes pidfds pollable. Specifically, it allows listeners to
> > be informed when the process the pidfd referes to exits. This patch only
> > introduces the ability to poll thread-group leaders since pidfds
> > currently can only reference those..."
> > 
> > Then justify the use-case and then go into implementation details.
> > That's usually how I would think about this:
> > - Change the codebase to do X
> > - Why do we need X
> > - Are there any technical details worth mentioning in the commit message
> > (- Are there any controversial points that people stumbled upon but that
> >   have been settled sufficiently.)
> 
> Generally the "how" in the patch should be in the code, but ok.

That's why I said: technical details that are worth mentioning.
Sometimes you have controversial bits that are obviously to be
understood in the code but it still might be worth explaining why one
had to do it this way. Like say what we did for the pidfd_send_signal()
thing where we explained why O_PATH is disallowed.

> 
> I changed the first 3 paragraphs of the changelog to the following, is that
> better? :
> 
> Android low memory killer (LMK) needs to know when a process dies once
> it is sent the kill signal. It does so by checking for the existence of
> /proc/pid which is both racy and slow. For example, if a PID is reused
> between when LMK sends a kill signal and checks for existence of the
> PID, since the wrong PID is now possibly checked for existence.
> 
> This patch adds polling support to pidfd. Using the polling support, LMK
> will be able to get notified when a process exists in race-free and fast
> way, and allows the LMK to do other things (such as by polling on other
> fds) while awaiting the process being killed to die.
> 
> For notification to polling processes, we follow the same existing
> mechanism in the kernel used when the parent of the task group is to be
> notified of a child's death (do_notify_parent).  This is precisely when
> the tasks waiting on a poll of pidfd are also awakened in this patch.
> 
> > > pidfd are file descriptors referring to a process created with the
> > > CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs pidfd
> > > polling support to replace code that currently checks for existence of
> > > /proc/pid for knowing that a process that is signalled to be killed has
> > > died, which is both racy and slow. The pidfd poll approach is race-free,
> > > and also allows the LMK to do other things (such as by polling on other
> > > fds) while awaiting the process being killed to die.
> > 
> > > 
> > > It prevents a situation where a PID is reused between when LMK sends a
> > > kill signal and checks for existence of the PID, since the wrong PID is
> > > now possibly checked for existence.
> > > 
> > > In this patch, we follow the same existing mechanism in the kernel used
> > > when the parent of the task group is to be notified (do_notify_parent).
> > > This is when the tasks waiting on a poll of pidfd are also awakened.
> > > 
> > > We have decided to include the waitqueue in struct pid for the following
> > > reasons:
> > > 1. The wait queue has to survive for the lifetime of the poll. Including
> > > it in task_struct would not be option in this case because the task can
> > > be reaped and destroyed before the poll returns.
> > > 
> > > 2. By including the struct pid for the waitqueue means that during
> > > de_thread(), the new thread group leader automatically gets the new
> > > waitqueue/pid even though its task_struct is different.
> > > 
> > > Appropriate test cases are added in the second patch to provide coverage
> > > of all the cases the patch is handling.
> > > 
> > > Andy had a similar patch [1] in the past which was a good reference
> > > however this patch tries to handle different situations properly related
> > > to thread group existence, and how/where it notifies. And also solves
> > > other bugs (waitqueue lifetime).  Daniel had a similar patch [2]
> > > recently which this patch supercedes.
> > > 
> > > [1] https://lore.kernel.org/patchwork/patch/345098/
> > > [2] https://lore.kernel.org/lkml/20181029175322.189042-1-dancol at google.com/
> > > 
> > > Cc: luto at amacapital.net
> > > Cc: rostedt at goodmis.org
> > > Cc: dancol at google.com
> > > Cc: sspatil at google.com
> > > Cc: christian at brauner.io
> > > Cc: jannh at google.com
> > > Cc: surenb at google.com
> > > Cc: timmurray at google.com
> > > Cc: Jonathan Kowalski <bl0pbl33p at gmail.com>
> > > Cc: torvalds at linux-foundation.org
> > > Cc: kernel-team at android.com
> > 
> > These should all be in the form:
> > 
> > Cc: Firstname Lastname <email at address.com>
> 
> If this bothers you too much, I can also just remove the CC list from the
> changelog here, and include it in my invocation of git-send-email instead..
> but I have seen commits in the tree that don't follow this rule.

Yeah, but they should. There are people with multiple emails over the
years and they might not necessarily contain their first and last
name. And I don't want to have to mailmap them or sm. Having their names
in there just makes it easier. Also, every single other DCO-*related*
line follows:

Random J Developer <random at developer.example.org>

This should too. If others are sloppy and allow this, fine. No reason we
should.

> 
> > 
> > There are people missing from the Cc that really should be there...
> 
> If you look at the CC list of the email, people in the get_maintainer.pl
> script were also added. I did run get_maintainer.pl and checkpatch. But ok, I
> will add the folks you are suggesting as well. Thanks.

get_maintainer.pl is not the last word. 

> 
> > Even though he usually doesn't respond that often, please Cc Al on this.
> > If he responds it's usually rather important.
> 
> No issues on that, but I am wondering if he should also be in MAINTAINERS
> file somewhere such that get_maintainer.pl does pick him up. I added him.

It's often not about someone being a maintainer but whether or not they
have valuable input.

"[...] This tag documents that potentially interested parties have been
included in the discussion."

> 
> > Oleg has reviewed your RFC patch quite substantially and given valuable
> > feedback and has an opinion on this thing and is best acquainted with
> > the exit code. So please add him to the Cc of the commit message in the
> > appropriate form and also add him to the Cc of the thread.
> 
> Done.

Thanks!

> 
> > Probably also want linux-api for good measure since a lot of people are
> > subscribed that would care about pollable pidfds. I'd also add Kees
> > since he had some interest in this work and David (Howells).
> 
> Done, I added all of them and CC will go out to them next time. Thanks.

Cool. That really wasn't a "you've done this wrong". It's rather really
just to make sure that everyone who might catch a big f*ck up on our
part has had a chance to tell us so. :)

> 
> > 
> > > Co-developed-by: Daniel Colascione <dancol at google.com>
> > 
> > Every CDB needs to give a SOB as well.
> 
> Ok, done. thanks.

Fwiw, I only learned this recently too.

> 
> > 
> > > Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>
> > > 
> > > ---
> > > 
> > > RFC -> v1:
> > > * Based on CLONE_PIDFD patches: https://lwn.net/Articles/786244/
> > > * Updated selftests.
> > > * Renamed poll wake function to do_notify_pidfd.
> > > * Removed depending on EXIT flags
> > > * Removed POLLERR flag since semantics are controversial and
> > >   we don't have usecases for it right now (later we can add if there's
> > >   a need for it).
> > > 
> > >  include/linux/pid.h |  3 +++
> > >  kernel/fork.c       | 33 +++++++++++++++++++++++++++++++++
> > >  kernel/pid.c        |  2 ++
> > >  kernel/signal.c     | 14 ++++++++++++++
> > >  4 files changed, 52 insertions(+)
> > > 
> > > diff --git a/include/linux/pid.h b/include/linux/pid.h
> > > index 3c8ef5a199ca..1484db6ca8d1 100644
> > > --- a/include/linux/pid.h
> > > +++ b/include/linux/pid.h
> > > @@ -3,6 +3,7 @@
> > >  #define _LINUX_PID_H
> > >  
> > >  #include <linux/rculist.h>
> > > +#include <linux/wait.h>
> > >  
> > >  enum pid_type
> > >  {
> > > @@ -60,6 +61,8 @@ struct pid
> > >  	unsigned int level;
> > >  	/* lists of tasks that use this pid */
> > >  	struct hlist_head tasks[PIDTYPE_MAX];
> > > +	/* wait queue for pidfd notifications */
> > > +	wait_queue_head_t wait_pidfd;
> > >  	struct rcu_head rcu;
> > >  	struct upid numbers[1];
> > >  };
> > > diff --git a/kernel/fork.c b/kernel/fork.c
> > > index 5525837ed80e..fb3b614f6456 100644
> > > --- a/kernel/fork.c
> > > +++ b/kernel/fork.c
> > > @@ -1685,8 +1685,41 @@ static void pidfd_show_fdinfo(struct seq_file *m, struct file *f)
> > >  }
> > >  #endif
> > >  
> > > +static unsigned int pidfd_poll(struct file *file, struct poll_table_struct *pts)
> > > +{
> > > +	struct task_struct *task;
> > > +	struct pid *pid;
> > > +	int poll_flags = 0;
> > > +
> > > +	/*
> > > +	 * tasklist_lock must be held because to avoid racing with
> > > +	 * changes in exit_state and wake up. Basically to avoid:
> > > +	 *
> > > +	 * P0: read exit_state = 0
> > > +	 * P1: write exit_state = EXIT_DEAD
> > > +	 * P1: Do a wake up - wq is empty, so do nothing
> > > +	 * P0: Queue for polling - wait forever.
> > > +	 */
> > > +	read_lock(&tasklist_lock);
> > > +	pid = file->private_data;
> > > +	task = pid_task(pid, PIDTYPE_PID);
> > > +	WARN_ON_ONCE(task && !thread_group_leader(task));
> > > +
> > > +	if (!task || (task->exit_state && thread_group_empty(task)))
> > > +		poll_flags = POLLIN | POLLRDNORM;
> > 
> > So we block until the thread-group is empty? Hm, the thread-group leader
> > remains in zombie state until all threads are gone. Should probably just
> > be a short comment somewhere that callers are only informed about a
> > whole thread-group exit and not about when the thread-group leader has
> > actually exited.
> 
> Ok, I'll add a comment.
> 
> > I would like the ability to extend this interface in the future to allow
> > for actually reading data from the pidfd on EPOLLIN.
> > POSIX specifies that POLLIN and POLLRDNORM are set even if the
> > message to be read is zero. So one cheap way of doing this would
> > probably be to do a 0 read/ioctl. That wouldn't hurt your very limited
> > usecase and people could test whether the read returned non-0 data and
> > if so they know this interface got extended. If we never extend it here
> > it won't matter.
> 
> I am a bit confused. What specific changes to this patch are you proposing?
> This patch makes poll block until the process exits. In the future, we can
> make it unblock for a other reasons as well, that's fine with me. I don't see
> how this patch prevents such extensions.

I guess I should've asked the following:
What happens right now, when you get EPOLLIN on the pidfd and you and
out of ignorance you do:

read(pidfd, ...)

> 
> > > +	if (!poll_flags)
> > > +		poll_wait(file, &pid->wait_pidfd, pts);
> > > +
> > > +	read_unlock(&tasklist_lock);
> > > +
> > > +	return poll_flags;
> > > +}
> > 
> > 
> > > +
> > > +
> > >  const struct file_operations pidfd_fops = {
> > >  	.release = pidfd_release,
> > > +	.poll = pidfd_poll,
> > >  #ifdef CONFIG_PROC_FS
> > >  	.show_fdinfo = pidfd_show_fdinfo,
> > >  #endif
> > > diff --git a/kernel/pid.c b/kernel/pid.c
> > > index 20881598bdfa..5c90c239242f 100644
> > > --- a/kernel/pid.c
> > > +++ b/kernel/pid.c
> > > @@ -214,6 +214,8 @@ struct pid *alloc_pid(struct pid_namespace *ns)
> > >  	for (type = 0; type < PIDTYPE_MAX; ++type)
> > >  		INIT_HLIST_HEAD(&pid->tasks[type]);
> > >  
> > > +	init_waitqueue_head(&pid->wait_pidfd);
> > > +
> > >  	upid = pid->numbers + ns->level;
> > >  	spin_lock_irq(&pidmap_lock);
> > >  	if (!(ns->pid_allocated & PIDNS_ADDING))
> > > diff --git a/kernel/signal.c b/kernel/signal.c
> > > index 1581140f2d99..16e7718316e5 100644
> > > --- a/kernel/signal.c
> > > +++ b/kernel/signal.c
> > > @@ -1800,6 +1800,17 @@ int send_sigqueue(struct sigqueue *q, struct pid *pid, enum pid_type type)
> > >  	return ret;
> > >  }
> > >  
> > > +static void do_notify_pidfd(struct task_struct *task)
> > 
> > Maybe a short command that this helper can only be called when we know
> > that task is a thread-group leader wouldn't hurt so there's no confusion
> > later.
> 
> Ok, will do.
> 
> thanks,
> 
>  - Joel
> 

[PATCH v1 1/2] Add polling support to pidfd

[christian]

On April 26, 2019 5:21:40 PM GMT+02:00, Christian Brauner <christian at brauner.io> wrote:
>On Fri, Apr 26, 2019 at 10:23:37AM -0400, Joel Fernandes wrote:
>> On Fri, Apr 26, 2019 at 12:24:04AM +0200, Christian Brauner wrote:
>> > On Thu, Apr 25, 2019 at 03:00:09PM -0400, Joel Fernandes (Google)
>wrote:
>> > > pidfd are file descriptors referring to a process created with
>the
>> > > CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs
>pidfd
>> > > polling support to replace code that currently checks for
>existence of
>> > > /proc/pid for knowing that a process that is signalled to be
>killed has
>> > > died, which is both racy and slow. The pidfd poll approach is
>race-free,
>> > > and also allows the LMK to do other things (such as by polling on
>other
>> > > fds) while awaiting the process being killed to die.
>> > 
>> > Thanks for the patch!
>> > 
>> > Ok, let me be a little bit anal.
>> > Please start the commit message with what this patch does and then
>add
>> 
>> The subject title is "Add polling support to pidfd", but ok I should
>write a
>> better commit message.
>
>Yeah, it's really just that we should really just have a simple
>paragraph that expresses this makes the codebase do X.
>
>> 
>> > the justification why. You just say the "pidfd-poll" approach. You
>can
>> > probably assume that CLONE_PIDFD is available for this patch. So
>> > something like:
>> > 
>> > "This patch makes pidfds pollable. Specifically, it allows
>listeners to
>> > be informed when the process the pidfd referes to exits. This patch
>only
>> > introduces the ability to poll thread-group leaders since pidfds
>> > currently can only reference those..."
>> > 
>> > Then justify the use-case and then go into implementation details.
>> > That's usually how I would think about this:
>> > - Change the codebase to do X
>> > - Why do we need X
>> > - Are there any technical details worth mentioning in the commit
>message
>> > (- Are there any controversial points that people stumbled upon but
>that
>> >   have been settled sufficiently.)
>> 
>> Generally the "how" in the patch should be in the code, but ok.
>
>That's why I said: technical details that are worth mentioning.
>Sometimes you have controversial bits that are obviously to be
>understood in the code but it still might be worth explaining why one
>had to do it this way. Like say what we did for the pidfd_send_signal()
>thing where we explained why O_PATH is disallowed.
>
>> 
>> I changed the first 3 paragraphs of the changelog to the following,
>is that
>> better? :
>> 
>> Android low memory killer (LMK) needs to know when a process dies
>once
>> it is sent the kill signal. It does so by checking for the existence
>of
>> /proc/pid which is both racy and slow. For example, if a PID is
>reused
>> between when LMK sends a kill signal and checks for existence of the
>> PID, since the wrong PID is now possibly checked for existence.
>> 
>> This patch adds polling support to pidfd. Using the polling support,
>LMK
>> will be able to get notified when a process exists in race-free and
>fast
>> way, and allows the LMK to do other things (such as by polling on
>other
>> fds) while awaiting the process being killed to die.
>> 
>> For notification to polling processes, we follow the same existing
>> mechanism in the kernel used when the parent of the task group is to
>be
>> notified of a child's death (do_notify_parent).  This is precisely
>when
>> the tasks waiting on a poll of pidfd are also awakened in this patch.
>> 
>> > > pidfd are file descriptors referring to a process created with
>the
>> > > CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs
>pidfd
>> > > polling support to replace code that currently checks for
>existence of
>> > > /proc/pid for knowing that a process that is signalled to be
>killed has
>> > > died, which is both racy and slow. The pidfd poll approach is
>race-free,
>> > > and also allows the LMK to do other things (such as by polling on
>other
>> > > fds) while awaiting the process being killed to die.
>> > 
>> > > 
>> > > It prevents a situation where a PID is reused between when LMK
>sends a
>> > > kill signal and checks for existence of the PID, since the wrong
>PID is
>> > > now possibly checked for existence.
>> > > 
>> > > In this patch, we follow the same existing mechanism in the
>kernel used
>> > > when the parent of the task group is to be notified
>(do_notify_parent).
>> > > This is when the tasks waiting on a poll of pidfd are also
>awakened.
>> > > 
>> > > We have decided to include the waitqueue in struct pid for the
>following
>> > > reasons:
>> > > 1. The wait queue has to survive for the lifetime of the poll.
>Including
>> > > it in task_struct would not be option in this case because the
>task can
>> > > be reaped and destroyed before the poll returns.
>> > > 
>> > > 2. By including the struct pid for the waitqueue means that
>during
>> > > de_thread(), the new thread group leader automatically gets the
>new
>> > > waitqueue/pid even though its task_struct is different.
>> > > 
>> > > Appropriate test cases are added in the second patch to provide
>coverage
>> > > of all the cases the patch is handling.
>> > > 
>> > > Andy had a similar patch [1] in the past which was a good
>reference
>> > > however this patch tries to handle different situations properly
>related
>> > > to thread group existence, and how/where it notifies. And also
>solves
>> > > other bugs (waitqueue lifetime).  Daniel had a similar patch [2]
>> > > recently which this patch supercedes.
>> > > 
>> > > [1] https://lore.kernel.org/patchwork/patch/345098/
>> > > [2]
>https://lore.kernel.org/lkml/20181029175322.189042-1-dancol at google.com/
>> > > 
>> > > Cc: luto at amacapital.net
>> > > Cc: rostedt at goodmis.org
>> > > Cc: dancol at google.com
>> > > Cc: sspatil at google.com
>> > > Cc: christian at brauner.io
>> > > Cc: jannh at google.com
>> > > Cc: surenb at google.com
>> > > Cc: timmurray at google.com
>> > > Cc: Jonathan Kowalski <bl0pbl33p at gmail.com>
>> > > Cc: torvalds at linux-foundation.org
>> > > Cc: kernel-team at android.com
>> > 
>> > These should all be in the form:
>> > 
>> > Cc: Firstname Lastname <email at address.com>
>> 
>> If this bothers you too much, I can also just remove the CC list from
>the
>> changelog here, and include it in my invocation of git-send-email
>instead..
>> but I have seen commits in the tree that don't follow this rule.
>
>Yeah, but they should. There are people with multiple emails over the
>years and they might not necessarily contain their first and last
>name. And I don't want to have to mailmap them or sm. Having their
>names
>in there just makes it easier. Also, every single other DCO-*related*
>line follows:
>
>Random J Developer <random at developer.example.org>
>
>This should too. If others are sloppy and allow this, fine. No reason
>we
>should.
>
>> 
>> > 
>> > There are people missing from the Cc that really should be there...
>> 
>> If you look at the CC list of the email, people in the
>get_maintainer.pl
>> script were also added. I did run get_maintainer.pl and checkpatch.
>But ok, I
>> will add the folks you are suggesting as well. Thanks.
>
>get_maintainer.pl is not the last word. 
>
>> 
>> > Even though he usually doesn't respond that often, please Cc Al on
>this.
>> > If he responds it's usually rather important.
>> 
>> No issues on that, but I am wondering if he should also be in
>MAINTAINERS
>> file somewhere such that get_maintainer.pl does pick him up. I added
>him.
>
>It's often not about someone being a maintainer but whether or not they
>have valuable input.
>
>"[...] This tag documents that potentially interested parties have been
>included in the discussion."
>
>> 
>> > Oleg has reviewed your RFC patch quite substantially and given
>valuable
>> > feedback and has an opinion on this thing and is best acquainted
>with
>> > the exit code. So please add him to the Cc of the commit message in
>the
>> > appropriate form and also add him to the Cc of the thread.
>> 
>> Done.
>
>Thanks!
>
>> 
>> > Probably also want linux-api for good measure since a lot of people
>are
>> > subscribed that would care about pollable pidfds. I'd also add Kees
>> > since he had some interest in this work and David (Howells).
>> 
>> Done, I added all of them and CC will go out to them next time.
>Thanks.
>
>Cool. That really wasn't a "you've done this wrong". It's rather really
>just to make sure that everyone who might catch a big f*ck up on our
>part has had a chance to tell us so. :)
>
>> 
>> > 
>> > > Co-developed-by: Daniel Colascione <dancol at google.com>
>> > 
>> > Every CDB needs to give a SOB as well.
>> 
>> Ok, done. thanks.
>
>Fwiw, I only learned this recently too.
>
>> 
>> > 
>> > > Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>
>> > > 
>> > > ---
>> > > 
>> > > RFC -> v1:
>> > > * Based on CLONE_PIDFD patches: https://lwn.net/Articles/786244/
>> > > * Updated selftests.
>> > > * Renamed poll wake function to do_notify_pidfd.
>> > > * Removed depending on EXIT flags
>> > > * Removed POLLERR flag since semantics are controversial and
>> > >   we don't have usecases for it right now (later we can add if
>there's
>> > >   a need for it).
>> > > 
>> > >  include/linux/pid.h |  3 +++
>> > >  kernel/fork.c       | 33 +++++++++++++++++++++++++++++++++
>> > >  kernel/pid.c        |  2 ++
>> > >  kernel/signal.c     | 14 ++++++++++++++
>> > >  4 files changed, 52 insertions(+)
>> > > 
>> > > diff --git a/include/linux/pid.h b/include/linux/pid.h
>> > > index 3c8ef5a199ca..1484db6ca8d1 100644
>> > > --- a/include/linux/pid.h
>> > > +++ b/include/linux/pid.h
>> > > @@ -3,6 +3,7 @@
>> > >  #define _LINUX_PID_H
>> > >  
>> > >  #include <linux/rculist.h>
>> > > +#include <linux/wait.h>
>> > >  
>> > >  enum pid_type
>> > >  {
>> > > @@ -60,6 +61,8 @@ struct pid
>> > >  	unsigned int level;
>> > >  	/* lists of tasks that use this pid */
>> > >  	struct hlist_head tasks[PIDTYPE_MAX];
>> > > +	/* wait queue for pidfd notifications */
>> > > +	wait_queue_head_t wait_pidfd;
>> > >  	struct rcu_head rcu;
>> > >  	struct upid numbers[1];
>> > >  };
>> > > diff --git a/kernel/fork.c b/kernel/fork.c
>> > > index 5525837ed80e..fb3b614f6456 100644
>> > > --- a/kernel/fork.c
>> > > +++ b/kernel/fork.c
>> > > @@ -1685,8 +1685,41 @@ static void pidfd_show_fdinfo(struct
>seq_file *m, struct file *f)
>> > >  }
>> > >  #endif
>> > >  
>> > > +static unsigned int pidfd_poll(struct file *file, struct
>poll_table_struct *pts)
>> > > +{
>> > > +	struct task_struct *task;
>> > > +	struct pid *pid;
>> > > +	int poll_flags = 0;
>> > > +
>> > > +	/*
>> > > +	 * tasklist_lock must be held because to avoid racing with
>> > > +	 * changes in exit_state and wake up. Basically to avoid:
>> > > +	 *
>> > > +	 * P0: read exit_state = 0
>> > > +	 * P1: write exit_state = EXIT_DEAD
>> > > +	 * P1: Do a wake up - wq is empty, so do nothing
>> > > +	 * P0: Queue for polling - wait forever.
>> > > +	 */
>> > > +	read_lock(&tasklist_lock);
>> > > +	pid = file->private_data;
>> > > +	task = pid_task(pid, PIDTYPE_PID);
>> > > +	WARN_ON_ONCE(task && !thread_group_leader(task));
>> > > +
>> > > +	if (!task || (task->exit_state && thread_group_empty(task)))
>> > > +		poll_flags = POLLIN | POLLRDNORM;
>> > 
>> > So we block until the thread-group is empty? Hm, the thread-group
>leader
>> > remains in zombie state until all threads are gone. Should probably
>just
>> > be a short comment somewhere that callers are only informed about a
>> > whole thread-group exit and not about when the thread-group leader
>has
>> > actually exited.
>> 
>> Ok, I'll add a comment.
>> 
>> > I would like the ability to extend this interface in the future to
>allow
>> > for actually reading data from the pidfd on EPOLLIN.
>> > POSIX specifies that POLLIN and POLLRDNORM are set even if the
>> > message to be read is zero. So one cheap way of doing this would
>> > probably be to do a 0 read/ioctl. That wouldn't hurt your very
>limited
>> > usecase and people could test whether the read returned non-0 data
>and
>> > if so they know this interface got extended. If we never extend it
>here
>> > it won't matter.
>> 
>> I am a bit confused. What specific changes to this patch are you
>proposing?
>> This patch makes poll block until the process exits. In the future,
>we can
>> make it unblock for a other reasons as well, that's fine with me. I
>don't see
>> how this patch prevents such extensions.
>
>I guess I should've asked the following:
>What happens right now, when you get EPOLLIN on the pidfd and you and
>out of ignorance you do:
>
>read(pidfd, ...)

I guess it returns EINVAL which is fine.
So you can ignore that comment.

>
>> 
>> > > +	if (!poll_flags)
>> > > +		poll_wait(file, &pid->wait_pidfd, pts);
>> > > +
>> > > +	read_unlock(&tasklist_lock);
>> > > +
>> > > +	return poll_flags;
>> > > +}
>> > 
>> > 
>> > > +
>> > > +
>> > >  const struct file_operations pidfd_fops = {
>> > >  	.release = pidfd_release,
>> > > +	.poll = pidfd_poll,
>> > >  #ifdef CONFIG_PROC_FS
>> > >  	.show_fdinfo = pidfd_show_fdinfo,
>> > >  #endif
>> > > diff --git a/kernel/pid.c b/kernel/pid.c
>> > > index 20881598bdfa..5c90c239242f 100644
>> > > --- a/kernel/pid.c
>> > > +++ b/kernel/pid.c
>> > > @@ -214,6 +214,8 @@ struct pid *alloc_pid(struct pid_namespace
>*ns)
>> > >  	for (type = 0; type < PIDTYPE_MAX; ++type)
>> > >  		INIT_HLIST_HEAD(&pid->tasks[type]);
>> > >  
>> > > +	init_waitqueue_head(&pid->wait_pidfd);
>> > > +
>> > >  	upid = pid->numbers + ns->level;
>> > >  	spin_lock_irq(&pidmap_lock);
>> > >  	if (!(ns->pid_allocated & PIDNS_ADDING))
>> > > diff --git a/kernel/signal.c b/kernel/signal.c
>> > > index 1581140f2d99..16e7718316e5 100644
>> > > --- a/kernel/signal.c
>> > > +++ b/kernel/signal.c
>> > > @@ -1800,6 +1800,17 @@ int send_sigqueue(struct sigqueue *q,
>struct pid *pid, enum pid_type type)
>> > >  	return ret;
>> > >  }
>> > >  
>> > > +static void do_notify_pidfd(struct task_struct *task)
>> > 
>> > Maybe a short command that this helper can only be called when we
>know
>> > that task is a thread-group leader wouldn't hurt so there's no
>confusion
>> > later.
>> 
>> Ok, will do.
>> 
>> thanks,
>> 
>>  - Joel
>> 

[PATCH v1 1/2] Add polling support to pidfd

[Christian Brauner]

On April 26, 2019 5:21:40 PM GMT+02:00, Christian Brauner <christian@brauner.io> wrote:
>On Fri, Apr 26, 2019@10:23:37AM -0400, Joel Fernandes wrote:
>> On Fri, Apr 26, 2019@12:24:04AM +0200, Christian Brauner wrote:
>> > On Thu, Apr 25, 2019 at 03:00:09PM -0400, Joel Fernandes (Google)
>wrote:
>> > > pidfd are file descriptors referring to a process created with
>the
>> > > CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs
>pidfd
>> > > polling support to replace code that currently checks for
>existence of
>> > > /proc/pid for knowing that a process that is signalled to be
>killed has
>> > > died, which is both racy and slow. The pidfd poll approach is
>race-free,
>> > > and also allows the LMK to do other things (such as by polling on
>other
>> > > fds) while awaiting the process being killed to die.
>> > 
>> > Thanks for the patch!
>> > 
>> > Ok, let me be a little bit anal.
>> > Please start the commit message with what this patch does and then
>add
>> 
>> The subject title is "Add polling support to pidfd", but ok I should
>write a
>> better commit message.
>
>Yeah, it's really just that we should really just have a simple
>paragraph that expresses this makes the codebase do X.
>
>> 
>> > the justification why. You just say the "pidfd-poll" approach. You
>can
>> > probably assume that CLONE_PIDFD is available for this patch. So
>> > something like:
>> > 
>> > "This patch makes pidfds pollable. Specifically, it allows
>listeners to
>> > be informed when the process the pidfd referes to exits. This patch
>only
>> > introduces the ability to poll thread-group leaders since pidfds
>> > currently can only reference those..."
>> > 
>> > Then justify the use-case and then go into implementation details.
>> > That's usually how I would think about this:
>> > - Change the codebase to do X
>> > - Why do we need X
>> > - Are there any technical details worth mentioning in the commit
>message
>> > (- Are there any controversial points that people stumbled upon but
>that
>> >   have been settled sufficiently.)
>> 
>> Generally the "how" in the patch should be in the code, but ok.
>
>That's why I said: technical details that are worth mentioning.
>Sometimes you have controversial bits that are obviously to be
>understood in the code but it still might be worth explaining why one
>had to do it this way. Like say what we did for the pidfd_send_signal()
>thing where we explained why O_PATH is disallowed.
>
>> 
>> I changed the first 3 paragraphs of the changelog to the following,
>is that
>> better? :
>> 
>> Android low memory killer (LMK) needs to know when a process dies
>once
>> it is sent the kill signal. It does so by checking for the existence
>of
>> /proc/pid which is both racy and slow. For example, if a PID is
>reused
>> between when LMK sends a kill signal and checks for existence of the
>> PID, since the wrong PID is now possibly checked for existence.
>> 
>> This patch adds polling support to pidfd. Using the polling support,
>LMK
>> will be able to get notified when a process exists in race-free and
>fast
>> way, and allows the LMK to do other things (such as by polling on
>other
>> fds) while awaiting the process being killed to die.
>> 
>> For notification to polling processes, we follow the same existing
>> mechanism in the kernel used when the parent of the task group is to
>be
>> notified of a child's death (do_notify_parent).  This is precisely
>when
>> the tasks waiting on a poll of pidfd are also awakened in this patch.
>> 
>> > > pidfd are file descriptors referring to a process created with
>the
>> > > CLONE_PIDFD clone(2) flag. Android low memory killer (LMK) needs
>pidfd
>> > > polling support to replace code that currently checks for
>existence of
>> > > /proc/pid for knowing that a process that is signalled to be
>killed has
>> > > died, which is both racy and slow. The pidfd poll approach is
>race-free,
>> > > and also allows the LMK to do other things (such as by polling on
>other
>> > > fds) while awaiting the process being killed to die.
>> > 
>> > > 
>> > > It prevents a situation where a PID is reused between when LMK
>sends a
>> > > kill signal and checks for existence of the PID, since the wrong
>PID is
>> > > now possibly checked for existence.
>> > > 
>> > > In this patch, we follow the same existing mechanism in the
>kernel used
>> > > when the parent of the task group is to be notified
>(do_notify_parent).
>> > > This is when the tasks waiting on a poll of pidfd are also
>awakened.
>> > > 
>> > > We have decided to include the waitqueue in struct pid for the
>following
>> > > reasons:
>> > > 1. The wait queue has to survive for the lifetime of the poll.
>Including
>> > > it in task_struct would not be option in this case because the
>task can
>> > > be reaped and destroyed before the poll returns.
>> > > 
>> > > 2. By including the struct pid for the waitqueue means that
>during
>> > > de_thread(), the new thread group leader automatically gets the
>new
>> > > waitqueue/pid even though its task_struct is different.
>> > > 
>> > > Appropriate test cases are added in the second patch to provide
>coverage
>> > > of all the cases the patch is handling.
>> > > 
>> > > Andy had a similar patch [1] in the past which was a good
>reference
>> > > however this patch tries to handle different situations properly
>related
>> > > to thread group existence, and how/where it notifies. And also
>solves
>> > > other bugs (waitqueue lifetime).  Daniel had a similar patch [2]
>> > > recently which this patch supercedes.
>> > > 
>> > > [1] https://lore.kernel.org/patchwork/patch/345098/
>> > > [2]
>https://lore.kernel.org/lkml/20181029175322.189042-1-dancol at google.com/
>> > > 
>> > > Cc: luto at amacapital.net
>> > > Cc: rostedt at goodmis.org
>> > > Cc: dancol at google.com
>> > > Cc: sspatil at google.com
>> > > Cc: christian at brauner.io
>> > > Cc: jannh at google.com
>> > > Cc: surenb at google.com
>> > > Cc: timmurray at google.com
>> > > Cc: Jonathan Kowalski <bl0pbl33p at gmail.com>
>> > > Cc: torvalds at linux-foundation.org
>> > > Cc: kernel-team at android.com
>> > 
>> > These should all be in the form:
>> > 
>> > Cc: Firstname Lastname <email at address.com>
>> 
>> If this bothers you too much, I can also just remove the CC list from
>the
>> changelog here, and include it in my invocation of git-send-email
>instead..
>> but I have seen commits in the tree that don't follow this rule.
>
>Yeah, but they should. There are people with multiple emails over the
>years and they might not necessarily contain their first and last
>name. And I don't want to have to mailmap them or sm. Having their
>names
>in there just makes it easier. Also, every single other DCO-*related*
>line follows:
>
>Random J Developer <random at developer.example.org>
>
>This should too. If others are sloppy and allow this, fine. No reason
>we
>should.
>
>> 
>> > 
>> > There are people missing from the Cc that really should be there...
>> 
>> If you look at the CC list of the email, people in the
>get_maintainer.pl
>> script were also added. I did run get_maintainer.pl and checkpatch.
>But ok, I
>> will add the folks you are suggesting as well. Thanks.
>
>get_maintainer.pl is not the last word. 
>
>> 
>> > Even though he usually doesn't respond that often, please Cc Al on
>this.
>> > If he responds it's usually rather important.
>> 
>> No issues on that, but I am wondering if he should also be in
>MAINTAINERS
>> file somewhere such that get_maintainer.pl does pick him up. I added
>him.
>
>It's often not about someone being a maintainer but whether or not they
>have valuable input.
>
>"[...] This tag documents that potentially interested parties have been
>included in the discussion."
>
>> 
>> > Oleg has reviewed your RFC patch quite substantially and given
>valuable
>> > feedback and has an opinion on this thing and is best acquainted
>with
>> > the exit code. So please add him to the Cc of the commit message in
>the
>> > appropriate form and also add him to the Cc of the thread.
>> 
>> Done.
>
>Thanks!
>
>> 
>> > Probably also want linux-api for good measure since a lot of people
>are
>> > subscribed that would care about pollable pidfds. I'd also add Kees
>> > since he had some interest in this work and David (Howells).
>> 
>> Done, I added all of them and CC will go out to them next time.
>Thanks.
>
>Cool. That really wasn't a "you've done this wrong". It's rather really
>just to make sure that everyone who might catch a big f*ck up on our
>part has had a chance to tell us so. :)
>
>> 
>> > 
>> > > Co-developed-by: Daniel Colascione <dancol at google.com>
>> > 
>> > Every CDB needs to give a SOB as well.
>> 
>> Ok, done. thanks.
>
>Fwiw, I only learned this recently too.
>
>> 
>> > 
>> > > Signed-off-by: Joel Fernandes (Google) <joel at joelfernandes.org>
>> > > 
>> > > ---
>> > > 
>> > > RFC -> v1:
>> > > * Based on CLONE_PIDFD patches: https://lwn.net/Articles/786244/
>> > > * Updated selftests.
>> > > * Renamed poll wake function to do_notify_pidfd.
>> > > * Removed depending on EXIT flags
>> > > * Removed POLLERR flag since semantics are controversial and
>> > >   we don't have usecases for it right now (later we can add if
>there's
>> > >   a need for it).
>> > > 
>> > >  include/linux/pid.h |  3 +++
>> > >  kernel/fork.c       | 33 +++++++++++++++++++++++++++++++++
>> > >  kernel/pid.c        |  2 ++
>> > >  kernel/signal.c     | 14 ++++++++++++++
>> > >  4 files changed, 52 insertions(+)
>> > > 
>> > > diff --git a/include/linux/pid.h b/include/linux/pid.h
>> > > index 3c8ef5a199ca..1484db6ca8d1 100644
>> > > --- a/include/linux/pid.h
>> > > +++ b/include/linux/pid.h
>> > > @@ -3,6 +3,7 @@
>> > >  #define _LINUX_PID_H
>> > >  
>> > >  #include <linux/rculist.h>
>> > > +#include <linux/wait.h>
>> > >  
>> > >  enum pid_type
>> > >  {
>> > > @@ -60,6 +61,8 @@ struct pid
>> > >  	unsigned int level;
>> > >  	/* lists of tasks that use this pid */
>> > >  	struct hlist_head tasks[PIDTYPE_MAX];
>> > > +	/* wait queue for pidfd notifications */
>> > > +	wait_queue_head_t wait_pidfd;
>> > >  	struct rcu_head rcu;
>> > >  	struct upid numbers[1];
>> > >  };
>> > > diff --git a/kernel/fork.c b/kernel/fork.c
>> > > index 5525837ed80e..fb3b614f6456 100644
>> > > --- a/kernel/fork.c
>> > > +++ b/kernel/fork.c
>> > > @@ -1685,8 +1685,41 @@ static void pidfd_show_fdinfo(struct
>seq_file *m, struct file *f)
>> > >  }
>> > >  #endif
>> > >  
>> > > +static unsigned int pidfd_poll(struct file *file, struct
>poll_table_struct *pts)
>> > > +{
>> > > +	struct task_struct *task;
>> > > +	struct pid *pid;
>> > > +	int poll_flags = 0;
>> > > +
>> > > +	/*
>> > > +	 * tasklist_lock must be held because to avoid racing with
>> > > +	 * changes in exit_state and wake up. Basically to avoid:
>> > > +	 *
>> > > +	 * P0: read exit_state = 0
>> > > +	 * P1: write exit_state = EXIT_DEAD
>> > > +	 * P1: Do a wake up - wq is empty, so do nothing
>> > > +	 * P0: Queue for polling - wait forever.
>> > > +	 */
>> > > +	read_lock(&tasklist_lock);
>> > > +	pid = file->private_data;
>> > > +	task = pid_task(pid, PIDTYPE_PID);
>> > > +	WARN_ON_ONCE(task && !thread_group_leader(task));
>> > > +
>> > > +	if (!task || (task->exit_state && thread_group_empty(task)))
>> > > +		poll_flags = POLLIN | POLLRDNORM;
>> > 
>> > So we block until the thread-group is empty? Hm, the thread-group
>leader
>> > remains in zombie state until all threads are gone. Should probably
>just
>> > be a short comment somewhere that callers are only informed about a
>> > whole thread-group exit and not about when the thread-group leader
>has
>> > actually exited.
>> 
>> Ok, I'll add a comment.
>> 
>> > I would like the ability to extend this interface in the future to
>allow
>> > for actually reading data from the pidfd on EPOLLIN.
>> > POSIX specifies that POLLIN and POLLRDNORM are set even if the
>> > message to be read is zero. So one cheap way of doing this would
>> > probably be to do a 0 read/ioctl. That wouldn't hurt your very
>limited
>> > usecase and people could test whether the read returned non-0 data
>and
>> > if so they know this interface got extended. If we never extend it
>here
>> > it won't matter.
>> 
>> I am a bit confused. What specific changes to this patch are you
>proposing?
>> This patch makes poll block until the process exits. In the future,
>we can
>> make it unblock for a other reasons as well, that's fine with me. I
>don't see
>> how this patch prevents such extensions.
>
>I guess I should've asked the following:
>What happens right now, when you get EPOLLIN on the pidfd and you and
>out of ignorance you do:
>
>read(pidfd, ...)

I guess it returns EINVAL which is fine.
So you can ignore that comment.

>
>> 
>> > > +	if (!poll_flags)
>> > > +		poll_wait(file, &pid->wait_pidfd, pts);
>> > > +
>> > > +	read_unlock(&tasklist_lock);
>> > > +
>> > > +	return poll_flags;
>> > > +}
>> > 
>> > 
>> > > +
>> > > +
>> > >  const struct file_operations pidfd_fops = {
>> > >  	.release = pidfd_release,
>> > > +	.poll = pidfd_poll,
>> > >  #ifdef CONFIG_PROC_FS
>> > >  	.show_fdinfo = pidfd_show_fdinfo,
>> > >  #endif
>> > > diff --git a/kernel/pid.c b/kernel/pid.c
>> > > index 20881598bdfa..5c90c239242f 100644
>> > > --- a/kernel/pid.c
>> > > +++ b/kernel/pid.c
>> > > @@ -214,6 +214,8 @@ struct pid *alloc_pid(struct pid_namespace
>*ns)
>> > >  	for (type = 0; type < PIDTYPE_MAX; ++type)
>> > >  		INIT_HLIST_HEAD(&pid->tasks[type]);
>> > >  
>> > > +	init_waitqueue_head(&pid->wait_pidfd);
>> > > +
>> > >  	upid = pid->numbers + ns->level;
>> > >  	spin_lock_irq(&pidmap_lock);
>> > >  	if (!(ns->pid_allocated & PIDNS_ADDING))
>> > > diff --git a/kernel/signal.c b/kernel/signal.c
>> > > index 1581140f2d99..16e7718316e5 100644
>> > > --- a/kernel/signal.c
>> > > +++ b/kernel/signal.c
>> > > @@ -1800,6 +1800,17 @@ int send_sigqueue(struct sigqueue *q,
>struct pid *pid, enum pid_type type)
>> > >  	return ret;
>> > >  }
>> > >  
>> > > +static void do_notify_pidfd(struct task_struct *task)
>> > 
>> > Maybe a short command that this helper can only be called when we
>know
>> > that task is a thread-group leader wouldn't hurt so there's no
>confusion
>> > later.
>> 
>> Ok, will do.
>> 
>> thanks,
>> 
>>  - Joel
>> 

[PATCH v1 2/2] Add selftests for pidfd polling

[joel]

On Thu, Apr 25, 2019 at 03:07:48PM -0700, Daniel Colascione wrote:
> On Thu, Apr 25, 2019 at 2:29 PM Christian Brauner <christian at brauner.io> wrote:
> > This timing-based testing seems kinda odd to be honest. Can't we do
> > something better than this?
> 
> Agreed. Timing-based tests have a substantial risk of becoming flaky.
> We ought to be able to make these tests fully deterministic and not
> subject to breakage from odd scheduling outcomes. We don't have
> sleepable events for everything, granted, but sleep-waiting on a
> condition with exponential backoff is fine in test code. In general,
> if you start with a robust test, you can insert a sleep(100) anywhere
> and not break the logic. Violating this rule always causes pain sooner
> or later.

I prefer if you can be more specific about how to redesign the test. Please
go through the code and make suggestions there. The tests have not been flaky
in my experience. Some tests do depend on timing like the preemptoff tests,
that can't be helped. Or a performance test that calculates framedrops.

In this case, we want to make sure that the poll unblocks at the right "time"
that is when the non-leader thread exits, and not when the leader thread
exits (test 1), or when the non-leader thread exits and not when the same
non-leader previous did an execve (test 2).

These are inherently timing related. Yes it is true that if this runs in a VM
and if the VM CPU is preempted for a couple seconds, then the test can fail
falsely. Still I would argue such a failure scenario of a multi-second CPU
lock-up can cause more serious issues like RCU stalls, and that's not a test
issue. We can increase the sleep intervals if you want, to reduce the risk of
such scenarios.

I would love to make the test not depend on timing, but I don't know how. And
the tests caught issues that I had in my development flow, so the tests
worked quite well in my experience.

thanks,

 - Joel

[PATCH v1 2/2] Add selftests for pidfd polling

[Joel Fernandes]

On Thu, Apr 25, 2019@03:07:48PM -0700, Daniel Colascione wrote:
> On Thu, Apr 25, 2019@2:29 PM Christian Brauner <christian@brauner.io> wrote:
> > This timing-based testing seems kinda odd to be honest. Can't we do
> > something better than this?
> 
> Agreed. Timing-based tests have a substantial risk of becoming flaky.
> We ought to be able to make these tests fully deterministic and not
> subject to breakage from odd scheduling outcomes. We don't have
> sleepable events for everything, granted, but sleep-waiting on a
> condition with exponential backoff is fine in test code. In general,
> if you start with a robust test, you can insert a sleep(100) anywhere
> and not break the logic. Violating this rule always causes pain sooner
> or later.

I prefer if you can be more specific about how to redesign the test. Please
go through the code and make suggestions there. The tests have not been flaky
in my experience. Some tests do depend on timing like the preemptoff tests,
that can't be helped. Or a performance test that calculates framedrops.

In this case, we want to make sure that the poll unblocks at the right "time"
that is when the non-leader thread exits, and not when the leader thread
exits (test 1), or when the non-leader thread exits and not when the same
non-leader previous did an execve (test 2).

These are inherently timing related. Yes it is true that if this runs in a VM
and if the VM CPU is preempted for a couple seconds, then the test can fail
falsely. Still I would argue such a failure scenario of a multi-second CPU
lock-up can cause more serious issues like RCU stalls, and that's not a test
issue. We can increase the sleep intervals if you want, to reduce the risk of
such scenarios.

I would love to make the test not depend on timing, but I don't know how. And
the tests caught issues that I had in my development flow, so the tests
worked quite well in my experience.

thanks,

 - Joel

[PATCH v1 2/2] Add selftests for pidfd polling

[dancol]

On Fri, Apr 26, 2019 at 10:26 AM Joel Fernandes <joel at joelfernandes.org> wrote:
> On Thu, Apr 25, 2019 at 03:07:48PM -0700, Daniel Colascione wrote:
> > On Thu, Apr 25, 2019 at 2:29 PM Christian Brauner <christian at brauner.io> wrote:
> > > This timing-based testing seems kinda odd to be honest. Can't we do
> > > something better than this?
> >
> > Agreed. Timing-based tests have a substantial risk of becoming flaky.
> > We ought to be able to make these tests fully deterministic and not
> > subject to breakage from odd scheduling outcomes. We don't have
> > sleepable events for everything, granted, but sleep-waiting on a
> > condition with exponential backoff is fine in test code. In general,
> > if you start with a robust test, you can insert a sleep(100) anywhere
> > and not break the logic. Violating this rule always causes pain sooner
> > or later.
>
> I prefer if you can be more specific about how to redesign the test. Please
> go through the code and make suggestions there. The tests have not been flaky
> in my experience.

You've been running them in an ideal environment.

Some tests do depend on timing like the preemptoff tests,
> that can't be helped. Or a performance test that calculates framedrops.

Performance tests are *about* timing. This is a functional test. Here,
we care about sequencing, not timing, and using a bare sleep instead
of sleeping with a condition check (see below) is always flaky.

> In this case, we want to make sure that the poll unblocks at the right "time"
> that is when the non-leader thread exits, and not when the leader thread
> exits (test 1), or when the non-leader thread exits and not when the same
> non-leader previous did an execve (test 2).

Instead of sleeping, you want to wait for some condition. Right now,
in a bunch of places, the test does something like this:

do_something()
sleep(SOME_TIMEOUT)
check(some_condition())

You can replace each of these clauses with something like this:

do_something()
start_time = now()
while(!some_condition() && now() - start_time < LONG_TIMEOUT)
  sleep(SHORT_DELAY)
check(some_condition())

This way, you're insensitive to timing, up to LONG_TIMEOUT (which can
be something like a minute). Yes, you can always write
sleep(LARGE_TIMEOUT) instead, but a good, robust value of LONG_TIMEOUT
(which should be tens of seconds) would make the test take far too
long to run in the happy case.

Note that this code is fine:

check(!some_condition())
sleep(SOME_REASONABLE_TIMEOUT)
check(!some_condition())

It's okay to sleep for a little while and check that something did
*not* happen, but it's not okay for the test to *fail* due to
scheduling delays. The difference is that
sleeping-and-checking-that-something-didn't-happen can only generate
false negatives when checking for failures, and it's much better from
a code health perspective for a test to sometimes fail to detect a bug
than for it to fire occasionally when there's no bug actually present.

> These are inherently timing related.

No they aren't. We don't care how long these operations take. We only
care that they happen in the right order.

(Well, we do care about performance, but not for the purposes of this
functional test.)

> Yes it is true that if this runs in a VM
> and if the VM CPU is preempted for a couple seconds, then the test can fail
> falsely. Still I would argue such a failure scenario of a multi-second CPU
> lock-up can cause more serious issues like RCU stalls, and that's not a test
> issue. We can increase the sleep intervals if you want, to reduce the risk of
> such scenarios.
>
> I would love to make the test not depend on timing, but I don't know how.

For threads, implement some_condition() above by opening a /proc
directory to the task you want. You can look by death by looking for
zombie status in stat or ESRCH.

If you want to test that poll() actually unblocks on exit (as opposed
to EPOLLIN-ing immediately when the waited process is already dead),
do something like this:

- [Main test thread] Start subprocess, getting a pidfd
- [Subprocess] Wait forever
- [Main test thread] Start a waiter thread
- [Waiter test thread] poll(2) (or epoll, if you insist) on process exit
- [Main test thread] sleep(FAIRLY_SHORT_TIMEOUT)
- [Main test thread] Check that the subprocess is alive
- [Main test thread] pthread_tryjoin_np (make sure the waiter thread
is still alive)
- [Main test thread] Kill the subprocess (or one of its threads, for
testing the leader-exit case)
- [Main test thread] pthread_timedjoin_np(LONG_TIMEOUT) the waiter thread
- [Waiter test thread] poll(2) returns and thread exits
- [Main test thread] pthread_join returns: test succeeds (or the
pthread_timedjoin_np fails with ETIMEOUT, it means poll(2) didn't
unblock, and the test should fail).

Tests that sleep for synchronization *do* end up being flaky. That the
flakiness doesn't show up in local iterative testing doesn't mean that
the test is adequately robust.

[PATCH v1 2/2] Add selftests for pidfd polling

[Daniel Colascione]

On Fri, Apr 26, 2019@10:26 AM Joel Fernandes <joel@joelfernandes.org> wrote:
> On Thu, Apr 25, 2019@03:07:48PM -0700, Daniel Colascione wrote:
> > On Thu, Apr 25, 2019@2:29 PM Christian Brauner <christian@brauner.io> wrote:
> > > This timing-based testing seems kinda odd to be honest. Can't we do
> > > something better than this?
> >
> > Agreed. Timing-based tests have a substantial risk of becoming flaky.
> > We ought to be able to make these tests fully deterministic and not
> > subject to breakage from odd scheduling outcomes. We don't have
> > sleepable events for everything, granted, but sleep-waiting on a
> > condition with exponential backoff is fine in test code. In general,
> > if you start with a robust test, you can insert a sleep(100) anywhere
> > and not break the logic. Violating this rule always causes pain sooner
> > or later.
>
> I prefer if you can be more specific about how to redesign the test. Please
> go through the code and make suggestions there. The tests have not been flaky
> in my experience.

You've been running them in an ideal environment.

Some tests do depend on timing like the preemptoff tests,
> that can't be helped. Or a performance test that calculates framedrops.

Performance tests are *about* timing. This is a functional test. Here,
we care about sequencing, not timing, and using a bare sleep instead
of sleeping with a condition check (see below) is always flaky.

> In this case, we want to make sure that the poll unblocks at the right "time"
> that is when the non-leader thread exits, and not when the leader thread
> exits (test 1), or when the non-leader thread exits and not when the same
> non-leader previous did an execve (test 2).

Instead of sleeping, you want to wait for some condition. Right now,
in a bunch of places, the test does something like this:

do_something()
sleep(SOME_TIMEOUT)
check(some_condition())

You can replace each of these clauses with something like this:

do_something()
start_time = now()
while(!some_condition() && now() - start_time < LONG_TIMEOUT)
  sleep(SHORT_DELAY)
check(some_condition())

This way, you're insensitive to timing, up to LONG_TIMEOUT (which can
be something like a minute). Yes, you can always write
sleep(LARGE_TIMEOUT) instead, but a good, robust value of LONG_TIMEOUT
(which should be tens of seconds) would make the test take far too
long to run in the happy case.

Note that this code is fine:

check(!some_condition())
sleep(SOME_REASONABLE_TIMEOUT)
check(!some_condition())

It's okay to sleep for a little while and check that something did
*not* happen, but it's not okay for the test to *fail* due to
scheduling delays. The difference is that
sleeping-and-checking-that-something-didn't-happen can only generate
false negatives when checking for failures, and it's much better from
a code health perspective for a test to sometimes fail to detect a bug
than for it to fire occasionally when there's no bug actually present.

> These are inherently timing related.

No they aren't. We don't care how long these operations take. We only
care that they happen in the right order.

(Well, we do care about performance, but not for the purposes of this
functional test.)

> Yes it is true that if this runs in a VM
> and if the VM CPU is preempted for a couple seconds, then the test can fail
> falsely. Still I would argue such a failure scenario of a multi-second CPU
> lock-up can cause more serious issues like RCU stalls, and that's not a test
> issue. We can increase the sleep intervals if you want, to reduce the risk of
> such scenarios.
>
> I would love to make the test not depend on timing, but I don't know how.

For threads, implement some_condition() above by opening a /proc
directory to the task you want. You can look by death by looking for
zombie status in stat or ESRCH.

If you want to test that poll() actually unblocks on exit (as opposed
to EPOLLIN-ing immediately when the waited process is already dead),
do something like this:

- [Main test thread] Start subprocess, getting a pidfd
- [Subprocess] Wait forever
- [Main test thread] Start a waiter thread
- [Waiter test thread] poll(2) (or epoll, if you insist) on process exit
- [Main test thread] sleep(FAIRLY_SHORT_TIMEOUT)
- [Main test thread] Check that the subprocess is alive
- [Main test thread] pthread_tryjoin_np (make sure the waiter thread
is still alive)
- [Main test thread] Kill the subprocess (or one of its threads, for
testing the leader-exit case)
- [Main test thread] pthread_timedjoin_np(LONG_TIMEOUT) the waiter thread
- [Waiter test thread] poll(2) returns and thread exits
- [Main test thread] pthread_join returns: test succeeds (or the
pthread_timedjoin_np fails with ETIMEOUT, it means poll(2) didn't
unblock, and the test should fail).

Tests that sleep for synchronization *do* end up being flaky. That the
flakiness doesn't show up in local iterative testing doesn't mean that
the test is adequately robust.

[PATCH v1 2/2] Add selftests for pidfd polling

[joel]

On Fri, Apr 26, 2019 at 12:35:40PM -0700, Daniel Colascione wrote:
> On Fri, Apr 26, 2019 at 10:26 AM Joel Fernandes <joel at joelfernandes.org> wrote:
> > On Thu, Apr 25, 2019 at 03:07:48PM -0700, Daniel Colascione wrote:
> > > On Thu, Apr 25, 2019 at 2:29 PM Christian Brauner <christian at brauner.io> wrote:
> > > > This timing-based testing seems kinda odd to be honest. Can't we do
> > > > something better than this?
> > >
> > > Agreed. Timing-based tests have a substantial risk of becoming flaky.
> > > We ought to be able to make these tests fully deterministic and not
> > > subject to breakage from odd scheduling outcomes. We don't have
> > > sleepable events for everything, granted, but sleep-waiting on a
> > > condition with exponential backoff is fine in test code. In general,
> > > if you start with a robust test, you can insert a sleep(100) anywhere
> > > and not break the logic. Violating this rule always causes pain sooner
> > > or later.
> >
> > I prefer if you can be more specific about how to redesign the test. Please
> > go through the code and make suggestions there. The tests have not been flaky
> > in my experience.
> 
> You've been running them in an ideal environment.

One would hope for a reliable test environment.

> > In this case, we want to make sure that the poll unblocks at the right "time"
> > that is when the non-leader thread exits, and not when the leader thread
> > exits (test 1), or when the non-leader thread exits and not when the same
> > non-leader previous did an execve (test 2).
> 
> Instead of sleeping, you want to wait for some condition. Right now,
> in a bunch of places, the test does something like this:
> 
> do_something()
> sleep(SOME_TIMEOUT)
> check(some_condition())

No. I don't have anything like "some_condition()". My some_condition() is
just the difference in time.

> 
> You can replace each of these clauses with something like this:
> 
> do_something()
> start_time = now()
> while(!some_condition() && now() - start_time < LONG_TIMEOUT)
>   sleep(SHORT_DELAY)
> check(some_condition())
> 
> This way, you're insensitive to timing, up to LONG_TIMEOUT (which can
> be something like a minute). Yes, you can always write
> sleep(LARGE_TIMEOUT) instead, but a good, robust value of LONG_TIMEOUT
> (which should be tens of seconds) would make the test take far too
> long to run in the happy case.

Yes, but try implementing some_condition()  :-). It is easy to talk in the
abstract, I think it would be more productive if you can come up with an
implementation/patchh of the test itself and send a patch for that. I know
you wrote some pseudocode below, but it is a complex reimplementation that I
don't think will make the test more robust. I mean reading /proc/pid stat?
yuck :) You are welcome to send a patch though if you have a better
implementation.

> Note that this code is fine:
> 
> check(!some_condition())
> sleep(SOME_REASONABLE_TIMEOUT)
> check(!some_condition())
> 
> It's okay to sleep for a little while and check that something did
> *not* happen, but it's not okay for the test to *fail* due to
> scheduling delays. The difference is that

As I said, multi-second scheduling delay are really unacceptable anyway. I
bet many kselftest may fail on a "bad" system like that way, that does not
mean the test is flaky. If there are any reports in the future that the test
fails or is flaky, I am happy to address them at that time. The tests work
and catch bugs reliably as I have seen. We could also make the test task as
RT if scheduling class is a concern.

I don't think its worth bikeshedding about hypothetical issues.

thanks,

 - Joel

[PATCH v1 2/2] Add selftests for pidfd polling

[Joel Fernandes]

On Fri, Apr 26, 2019@12:35:40PM -0700, Daniel Colascione wrote:
> On Fri, Apr 26, 2019@10:26 AM Joel Fernandes <joel@joelfernandes.org> wrote:
> > On Thu, Apr 25, 2019@03:07:48PM -0700, Daniel Colascione wrote:
> > > On Thu, Apr 25, 2019@2:29 PM Christian Brauner <christian@brauner.io> wrote:
> > > > This timing-based testing seems kinda odd to be honest. Can't we do
> > > > something better than this?
> > >
> > > Agreed. Timing-based tests have a substantial risk of becoming flaky.
> > > We ought to be able to make these tests fully deterministic and not
> > > subject to breakage from odd scheduling outcomes. We don't have
> > > sleepable events for everything, granted, but sleep-waiting on a
> > > condition with exponential backoff is fine in test code. In general,
> > > if you start with a robust test, you can insert a sleep(100) anywhere
> > > and not break the logic. Violating this rule always causes pain sooner
> > > or later.
> >
> > I prefer if you can be more specific about how to redesign the test. Please
> > go through the code and make suggestions there. The tests have not been flaky
> > in my experience.
> 
> You've been running them in an ideal environment.

One would hope for a reliable test environment.

> > In this case, we want to make sure that the poll unblocks at the right "time"
> > that is when the non-leader thread exits, and not when the leader thread
> > exits (test 1), or when the non-leader thread exits and not when the same
> > non-leader previous did an execve (test 2).
> 
> Instead of sleeping, you want to wait for some condition. Right now,
> in a bunch of places, the test does something like this:
> 
> do_something()
> sleep(SOME_TIMEOUT)
> check(some_condition())

No. I don't have anything like "some_condition()". My some_condition() is
just the difference in time.

> 
> You can replace each of these clauses with something like this:
> 
> do_something()
> start_time = now()
> while(!some_condition() && now() - start_time < LONG_TIMEOUT)
>   sleep(SHORT_DELAY)
> check(some_condition())
> 
> This way, you're insensitive to timing, up to LONG_TIMEOUT (which can
> be something like a minute). Yes, you can always write
> sleep(LARGE_TIMEOUT) instead, but a good, robust value of LONG_TIMEOUT
> (which should be tens of seconds) would make the test take far too
> long to run in the happy case.

Yes, but try implementing some_condition()  :-). It is easy to talk in the
abstract, I think it would be more productive if you can come up with an
implementation/patchh of the test itself and send a patch for that. I know
you wrote some pseudocode below, but it is a complex reimplementation that I
don't think will make the test more robust. I mean reading /proc/pid stat?
yuck :) You are welcome to send a patch though if you have a better
implementation.

> Note that this code is fine:
> 
> check(!some_condition())
> sleep(SOME_REASONABLE_TIMEOUT)
> check(!some_condition())
> 
> It's okay to sleep for a little while and check that something did
> *not* happen, but it's not okay for the test to *fail* due to
> scheduling delays. The difference is that

As I said, multi-second scheduling delay are really unacceptable anyway. I
bet many kselftest may fail on a "bad" system like that way, that does not
mean the test is flaky. If there are any reports in the future that the test
fails or is flaky, I am happy to address them at that time. The tests work
and catch bugs reliably as I have seen. We could also make the test task as
RT if scheduling class is a concern.

I don't think its worth bikeshedding about hypothetical issues.

thanks,

 - Joel

[PATCH v1 1/2] Add polling support to pidfd

[oleg]

Thanks for cc'ing me...

On 04/26, Christian Brauner wrote:
>
> On Thu, Apr 25, 2019 at 03:00:09PM -0400, Joel Fernandes (Google) wrote:
> > +static unsigned int pidfd_poll(struct file *file, struct poll_table_struct *pts)
> > +{
> > +	struct task_struct *task;
> > +	struct pid *pid;
> > +	int poll_flags = 0;
> > +
> > +	/*
> > +	 * tasklist_lock must be held because to avoid racing with
> > +	 * changes in exit_state and wake up. Basically to avoid:
> > +	 *
> > +	 * P0: read exit_state = 0
> > +	 * P1: write exit_state = EXIT_DEAD
> > +	 * P1: Do a wake up - wq is empty, so do nothing
> > +	 * P0: Queue for polling - wait forever.
> > +	 */
> > +	read_lock(&tasklist_lock);
> > +	pid = file->private_data;
> > +	task = pid_task(pid, PIDTYPE_PID);
> > +	WARN_ON_ONCE(task && !thread_group_leader(task));
> > +
> > +	if (!task || (task->exit_state && thread_group_empty(task)))
> > +		poll_flags = POLLIN | POLLRDNORM;

Joel, I still can't understand why do we need tasklist... and I don't really
understand the comment. The code looks as if you are trying to avoid poll_wait(),
but this would be strange.

OK, why can't pidfd_poll() do

	poll_wait(file, &pid->wait_pidfd, pts);

	rcu_read_lock();
	task = pid_task(pid, PIDTYPE_PID);
	if (!task || task->exit_state && thread_group_empty(task))
		poll_flags = POLLIN | ...;
	rcu_read_unlock();

	return poll_flags;

?

> > +static void do_notify_pidfd(struct task_struct *task)
>
> Maybe a short command that this helper can only be called when we know
> that task is a thread-group leader wouldn't hurt so there's no confusion
> later.

Not really. If the task is traced, do_notify_parent() (and thus do_notify_pidfd())
can be called to notify the debugger even if the task is not a leader and/or if
it is not the last thread. The latter means a spurious wakeup for pidfd_poll().

> > +{
> > +	struct pid *pid;
> > +
> > +	lockdep_assert_held(&tasklist_lock);
> > +
> > +	pid = get_task_pid(task, PIDTYPE_PID);
> > +	wake_up_all(&pid->wait_pidfd);
> > +	put_pid(pid);

Why get/put?

Oleg.

[PATCH v1 1/2] Add polling support to pidfd

[Oleg Nesterov]

Thanks for cc'ing me...

On 04/26, Christian Brauner wrote:
>
> On Thu, Apr 25, 2019@03:00:09PM -0400, Joel Fernandes (Google) wrote:
> > +static unsigned int pidfd_poll(struct file *file, struct poll_table_struct *pts)
> > +{
> > +	struct task_struct *task;
> > +	struct pid *pid;
> > +	int poll_flags = 0;
> > +
> > +	/*
> > +	 * tasklist_lock must be held because to avoid racing with
> > +	 * changes in exit_state and wake up. Basically to avoid:
> > +	 *
> > +	 * P0: read exit_state = 0
> > +	 * P1: write exit_state = EXIT_DEAD
> > +	 * P1: Do a wake up - wq is empty, so do nothing
> > +	 * P0: Queue for polling - wait forever.
> > +	 */
> > +	read_lock(&tasklist_lock);
> > +	pid = file->private_data;
> > +	task = pid_task(pid, PIDTYPE_PID);
> > +	WARN_ON_ONCE(task && !thread_group_leader(task));
> > +
> > +	if (!task || (task->exit_state && thread_group_empty(task)))
> > +		poll_flags = POLLIN | POLLRDNORM;

Joel, I still can't understand why do we need tasklist... and I don't really
understand the comment. The code looks as if you are trying to avoid poll_wait(),
but this would be strange.

OK, why can't pidfd_poll() do

	poll_wait(file, &pid->wait_pidfd, pts);

	rcu_read_lock();
	task = pid_task(pid, PIDTYPE_PID);
	if (!task || task->exit_state && thread_group_empty(task))
		poll_flags = POLLIN | ...;
	rcu_read_unlock();

	return poll_flags;

?

> > +static void do_notify_pidfd(struct task_struct *task)
>
> Maybe a short command that this helper can only be called when we know
> that task is a thread-group leader wouldn't hurt so there's no confusion
> later.

Not really. If the task is traced, do_notify_parent() (and thus do_notify_pidfd())
can be called to notify the debugger even if the task is not a leader and/or if
it is not the last thread. The latter means a spurious wakeup for pidfd_poll().

> > +{
> > +	struct pid *pid;
> > +
> > +	lockdep_assert_held(&tasklist_lock);
> > +
> > +	pid = get_task_pid(task, PIDTYPE_PID);
> > +	wake_up_all(&pid->wait_pidfd);
> > +	put_pid(pid);

Why get/put?

Oleg.

[PATCH v1 1/2] Add polling support to pidfd

[joel]

On Sun, Apr 28, 2019 at 06:24:06PM +0200, Oleg Nesterov wrote:
> Thanks for cc'ing me...
> 
> On 04/26, Christian Brauner wrote:
> >
> > On Thu, Apr 25, 2019 at 03:00:09PM -0400, Joel Fernandes (Google) wrote:
> > > +static unsigned int pidfd_poll(struct file *file, struct poll_table_struct *pts)
> > > +{
> > > +	struct task_struct *task;
> > > +	struct pid *pid;
> > > +	int poll_flags = 0;
> > > +
> > > +	/*
> > > +	 * tasklist_lock must be held because to avoid racing with
> > > +	 * changes in exit_state and wake up. Basically to avoid:
> > > +	 *
> > > +	 * P0: read exit_state = 0
> > > +	 * P1: write exit_state = EXIT_DEAD
> > > +	 * P1: Do a wake up - wq is empty, so do nothing
> > > +	 * P0: Queue for polling - wait forever.
> > > +	 */
> > > +	read_lock(&tasklist_lock);
> > > +	pid = file->private_data;
> > > +	task = pid_task(pid, PIDTYPE_PID);
> > > +	WARN_ON_ONCE(task && !thread_group_leader(task));
> > > +
> > > +	if (!task || (task->exit_state && thread_group_empty(task)))
> > > +		poll_flags = POLLIN | POLLRDNORM;
> 
> Joel, I still can't understand why do we need tasklist... and I don't really
> understand the comment. The code looks as if you are trying to avoid poll_wait(),
> but this would be strange.
> 
> OK, why can't pidfd_poll() do
> 
> 	poll_wait(file, &pid->wait_pidfd, pts);
> 
> 	rcu_read_lock();
> 	task = pid_task(pid, PIDTYPE_PID);
> 	if (!task || task->exit_state && thread_group_empty(task))
> 		poll_flags = POLLIN | ...;
> 	rcu_read_unlock();
> 
> 	return poll_flags;
> 
> ?

Oh that's much better Oleg, and would avoid the race I had in mind: Basically
I was acquiring the tasklist_lock to avoid a case where a polling task is not
woken up because it was added to the waitqueue too late. The reading of the
exit_state and the conditional adding to the wait queue, needed to be atomic.
Otherwise something like the following may be possible:

Task A (poller)		Task B (exiting task being polled)
------------            ----------------
poll() called
			exit_state is set to non-zero
read exit_state
			wake_up_all()

add_wait_queue()
----------------------------------------------

However, in your code above, it is avoided because we get:

Task A (poller)		Task B (exiting task being polled)
------------            ----------------
poll() called
add_wait_queue()
			exit_state is set to non-zero
read exit_state
remove_wait_queue()
			wake_up_all()

I don't see any other issues with your code above so I can try it out and
update the patches. Thanks.

> > > +static void do_notify_pidfd(struct task_struct *task)
> >
> > Maybe a short command that this helper can only be called when we know
> > that task is a thread-group leader wouldn't hurt so there's no confusion
> > later.
> 
> Not really. If the task is traced, do_notify_parent() (and thus do_notify_pidfd())
> can be called to notify the debugger even if the task is not a leader and/or if
> it is not the last thread. The latter means a spurious wakeup for pidfd_poll().

Seems like you are replying to Christian's point. I agree with you.

> > > +{
> > > +	struct pid *pid;
> > > +
> > > +	lockdep_assert_held(&tasklist_lock);
> > > +
> > > +	pid = get_task_pid(task, PIDTYPE_PID);
> > > +	wake_up_all(&pid->wait_pidfd);
> > > +	put_pid(pid);
> 
> Why get/put?

Yes, pid_task() should do it. Will update it. Thanks!

 - Joel

[PATCH v1 1/2] Add polling support to pidfd

[Joel Fernandes]

On Sun, Apr 28, 2019@06:24:06PM +0200, Oleg Nesterov wrote:
> Thanks for cc'ing me...
> 
> On 04/26, Christian Brauner wrote:
> >
> > On Thu, Apr 25, 2019@03:00:09PM -0400, Joel Fernandes (Google) wrote:
> > > +static unsigned int pidfd_poll(struct file *file, struct poll_table_struct *pts)
> > > +{
> > > +	struct task_struct *task;
> > > +	struct pid *pid;
> > > +	int poll_flags = 0;
> > > +
> > > +	/*
> > > +	 * tasklist_lock must be held because to avoid racing with
> > > +	 * changes in exit_state and wake up. Basically to avoid:
> > > +	 *
> > > +	 * P0: read exit_state = 0
> > > +	 * P1: write exit_state = EXIT_DEAD
> > > +	 * P1: Do a wake up - wq is empty, so do nothing
> > > +	 * P0: Queue for polling - wait forever.
> > > +	 */
> > > +	read_lock(&tasklist_lock);
> > > +	pid = file->private_data;
> > > +	task = pid_task(pid, PIDTYPE_PID);
> > > +	WARN_ON_ONCE(task && !thread_group_leader(task));
> > > +
> > > +	if (!task || (task->exit_state && thread_group_empty(task)))
> > > +		poll_flags = POLLIN | POLLRDNORM;
> 
> Joel, I still can't understand why do we need tasklist... and I don't really
> understand the comment. The code looks as if you are trying to avoid poll_wait(),
> but this would be strange.
> 
> OK, why can't pidfd_poll() do
> 
> 	poll_wait(file, &pid->wait_pidfd, pts);
> 
> 	rcu_read_lock();
> 	task = pid_task(pid, PIDTYPE_PID);
> 	if (!task || task->exit_state && thread_group_empty(task))
> 		poll_flags = POLLIN | ...;
> 	rcu_read_unlock();
> 
> 	return poll_flags;
> 
> ?

Oh that's much better Oleg, and would avoid the race I had in mind: Basically
I was acquiring the tasklist_lock to avoid a case where a polling task is not
woken up because it was added to the waitqueue too late. The reading of the
exit_state and the conditional adding to the wait queue, needed to be atomic.
Otherwise something like the following may be possible:

Task A (poller)		Task B (exiting task being polled)
------------            ----------------
poll() called
			exit_state is set to non-zero
read exit_state
			wake_up_all()

add_wait_queue()
----------------------------------------------

However, in your code above, it is avoided because we get:

Task A (poller)		Task B (exiting task being polled)
------------            ----------------
poll() called
add_wait_queue()
			exit_state is set to non-zero
read exit_state
remove_wait_queue()
			wake_up_all()

I don't see any other issues with your code above so I can try it out and
update the patches. Thanks.

> > > +static void do_notify_pidfd(struct task_struct *task)
> >
> > Maybe a short command that this helper can only be called when we know
> > that task is a thread-group leader wouldn't hurt so there's no confusion
> > later.
> 
> Not really. If the task is traced, do_notify_parent() (and thus do_notify_pidfd())
> can be called to notify the debugger even if the task is not a leader and/or if
> it is not the last thread. The latter means a spurious wakeup for pidfd_poll().

Seems like you are replying to Christian's point. I agree with you.

> > > +{
> > > +	struct pid *pid;
> > > +
> > > +	lockdep_assert_held(&tasklist_lock);
> > > +
> > > +	pid = get_task_pid(task, PIDTYPE_PID);
> > > +	wake_up_all(&pid->wait_pidfd);
> > > +	put_pid(pid);
> 
> Why get/put?

Yes, pid_task() should do it. Will update it. Thanks!

 - Joel

[PATCH v1 1/2] Add polling support to pidfd

[joel]

On Mon, Apr 29, 2019 at 10:02:45AM -0400, Joel Fernandes wrote:
> On Sun, Apr 28, 2019 at 06:24:06PM +0200, Oleg Nesterov wrote:
[snip]
> > > > +{
> > > > +	struct pid *pid;
> > > > +
> > > > +	lockdep_assert_held(&tasklist_lock);
> > > > +
> > > > +	pid = get_task_pid(task, PIDTYPE_PID);
> > > > +	wake_up_all(&pid->wait_pidfd);
> > > > +	put_pid(pid);
> > 
> > Why get/put?
> 
> Yes, pid_task() should do it. Will update it. Thanks!

I spoke too soon. We need the task's pid of type PIDTYPE_PID. How else can we
get it? This does an atomic_inc on the pid->count, so we need to put_pid()
after we are done with it. Did I miss something?

thanks,

 - Joel

 

[PATCH v1 1/2] Add polling support to pidfd

[Joel Fernandes]

On Mon, Apr 29, 2019@10:02:45AM -0400, Joel Fernandes wrote:
> On Sun, Apr 28, 2019@06:24:06PM +0200, Oleg Nesterov wrote:
[snip]
> > > > +{
> > > > +	struct pid *pid;
> > > > +
> > > > +	lockdep_assert_held(&tasklist_lock);
> > > > +
> > > > +	pid = get_task_pid(task, PIDTYPE_PID);
> > > > +	wake_up_all(&pid->wait_pidfd);
> > > > +	put_pid(pid);
> > 
> > Why get/put?
> 
> Yes, pid_task() should do it. Will update it. Thanks!

I spoke too soon. We need the task's pid of type PIDTYPE_PID. How else can we
get it? This does an atomic_inc on the pid->count, so we need to put_pid()
after we are done with it. Did I miss something?

thanks,

 - Joel

 

[PATCH v1 1/2] Add polling support to pidfd

[oleg]

On 04/29, Joel Fernandes wrote:
>
> However, in your code above, it is avoided because we get:
>
> Task A (poller)		Task B (exiting task being polled)
> ------------            ----------------
> poll() called
> add_wait_queue()
> 			exit_state is set to non-zero
> read exit_state
> remove_wait_queue()
> 			wake_up_all()

just to clarify... No, sys_poll() path doesn't do remove_wait_queue() until
it returns to user mode, and that is why we can't race with set-exit_code +
wake_up().

pidfd_poll() can race with the exiting task, miss exit_code != 0, and return
zero. However, do_poll() won't block after that and pidfd_poll() will be called
again.

Oleg.

[PATCH v1 1/2] Add polling support to pidfd

[Oleg Nesterov]

On 04/29, Joel Fernandes wrote:
>
> However, in your code above, it is avoided because we get:
>
> Task A (poller)		Task B (exiting task being polled)
> ------------            ----------------
> poll() called
> add_wait_queue()
> 			exit_state is set to non-zero
> read exit_state
> remove_wait_queue()
> 			wake_up_all()

just to clarify... No, sys_poll() path doesn't do remove_wait_queue() until
it returns to user mode, and that is why we can't race with set-exit_code +
wake_up().

pidfd_poll() can race with the exiting task, miss exit_code != 0, and return
zero. However, do_poll() won't block after that and pidfd_poll() will be called
again.

Oleg.

[PATCH v1 1/2] Add polling support to pidfd

[oleg]

On 04/29, Joel Fernandes wrote:
>
> On Mon, Apr 29, 2019 at 10:02:45AM -0400, Joel Fernandes wrote:
> > On Sun, Apr 28, 2019 at 06:24:06PM +0200, Oleg Nesterov wrote:
> [snip]
> > > > > +{
> > > > > +	struct pid *pid;
> > > > > +
> > > > > +	lockdep_assert_held(&tasklist_lock);
> > > > > +
> > > > > +	pid = get_task_pid(task, PIDTYPE_PID);
> > > > > +	wake_up_all(&pid->wait_pidfd);
> > > > > +	put_pid(pid);
> > >
> > > Why get/put?
> >
> > Yes, pid_task() should do it. Will update it. Thanks!
>
> I spoke too soon. We need the task's pid of type PIDTYPE_PID. How else can we
> get it? This does an atomic_inc on the pid->count, so we need to put_pid()
> after we are done with it. Did I miss something?

Just use task_pid(task);

Oleg.

[PATCH v1 1/2] Add polling support to pidfd

[Oleg Nesterov]

On 04/29, Joel Fernandes wrote:
>
> On Mon, Apr 29, 2019@10:02:45AM -0400, Joel Fernandes wrote:
> > On Sun, Apr 28, 2019@06:24:06PM +0200, Oleg Nesterov wrote:
> [snip]
> > > > > +{
> > > > > +	struct pid *pid;
> > > > > +
> > > > > +	lockdep_assert_held(&tasklist_lock);
> > > > > +
> > > > > +	pid = get_task_pid(task, PIDTYPE_PID);
> > > > > +	wake_up_all(&pid->wait_pidfd);
> > > > > +	put_pid(pid);
> > >
> > > Why get/put?
> >
> > Yes, pid_task() should do it. Will update it. Thanks!
>
> I spoke too soon. We need the task's pid of type PIDTYPE_PID. How else can we
> get it? This does an atomic_inc on the pid->count, so we need to put_pid()
> after we are done with it. Did I miss something?

Just use task_pid(task);

Oleg.

[PATCH v1 1/2] Add polling support to pidfd

[joel]

On Mon, Apr 29, 2019 at 04:20:30PM +0200, Oleg Nesterov wrote:
> On 04/29, Joel Fernandes wrote:
> >
> > However, in your code above, it is avoided because we get:
> >
> > Task A (poller)		Task B (exiting task being polled)
> > ------------            ----------------
> > poll() called
> > add_wait_queue()
> > 			exit_state is set to non-zero
> > read exit_state
> > remove_wait_queue()
> > 			wake_up_all()
> 
> just to clarify... No, sys_poll() path doesn't do remove_wait_queue() until
> it returns to user mode, and that is why we can't race with set-exit_code +
> wake_up().

I didn't follow what you mean, the removal from the waitqueue happens in
free_poll_entry() called from poll_freewait() which happens from
do_sys_poll() which is before the syscall returns to user mode. Could you
explain more?

> pidfd_poll() can race with the exiting task, miss exit_code != 0, and return
> zero. However, do_poll() won't block after that and pidfd_poll() will be called
> again.

Here also I didn't follow what you mean. If exit_code is read as 0 in
pidfd_poll(), then in do_poll() the count will be 0 and it will block in
poll_schedule_timeout(). Right? But above you're saying it wont block.
Also if you could show a timing diagram of this different race you're talking
about, that will make things clear. It is a bit hard for me to picture
otherwise.

Also, I will use task_pid() for getting the pid from the task, as you suggest
in the other thread.

thanks,

- Joel

[PATCH v1 1/2] Add polling support to pidfd

[Joel Fernandes]

On Mon, Apr 29, 2019@04:20:30PM +0200, Oleg Nesterov wrote:
> On 04/29, Joel Fernandes wrote:
> >
> > However, in your code above, it is avoided because we get:
> >
> > Task A (poller)		Task B (exiting task being polled)
> > ------------            ----------------
> > poll() called
> > add_wait_queue()
> > 			exit_state is set to non-zero
> > read exit_state
> > remove_wait_queue()
> > 			wake_up_all()
> 
> just to clarify... No, sys_poll() path doesn't do remove_wait_queue() until
> it returns to user mode, and that is why we can't race with set-exit_code +
> wake_up().

I didn't follow what you mean, the removal from the waitqueue happens in
free_poll_entry() called from poll_freewait() which happens from
do_sys_poll() which is before the syscall returns to user mode. Could you
explain more?

> pidfd_poll() can race with the exiting task, miss exit_code != 0, and return
> zero. However, do_poll() won't block after that and pidfd_poll() will be called
> again.

Here also I didn't follow what you mean. If exit_code is read as 0 in
pidfd_poll(), then in do_poll() the count will be 0 and it will block in
poll_schedule_timeout(). Right? But above you're saying it wont block.
Also if you could show a timing diagram of this different race you're talking
about, that will make things clear. It is a bit hard for me to picture
otherwise.

Also, I will use task_pid() for getting the pid from the task, as you suggest
in the other thread.

thanks,

- Joel

[PATCH v1 1/2] Add polling support to pidfd

[oleg]

On 04/29, Joel Fernandes wrote:
>
> On Mon, Apr 29, 2019 at 04:20:30PM +0200, Oleg Nesterov wrote:
> > On 04/29, Joel Fernandes wrote:
> > >
> > > However, in your code above, it is avoided because we get:
> > >
> > > Task A (poller)		Task B (exiting task being polled)
> > > ------------            ----------------
> > > poll() called
> > > add_wait_queue()
> > > 			exit_state is set to non-zero
> > > read exit_state
> > > remove_wait_queue()
> > > 			wake_up_all()
> >
> > just to clarify... No, sys_poll() path doesn't do remove_wait_queue() until
> > it returns to user mode, and that is why we can't race with set-exit_code +
> > wake_up().
>
> I didn't follow what you mean, the removal from the waitqueue happens in
> free_poll_entry() called from poll_freewait() which happens from
> do_sys_poll() which is before the syscall returns to user mode. Could you
> explain more?

Hmm. I do not really understand the question... Sure, do_sys_poll() does
poll_freewait() before sysret or even before return from syscall, but why
does this matter? This is the exit path, it frees the memory, does fput(),
etc, f_op->poll() won't be call after that.

> > pidfd_poll() can race with the exiting task, miss exit_code != 0, and return
> > zero. However, do_poll() won't block after that and pidfd_poll() will be called
> > again.
>
> Here also I didn't follow what you mean. If exit_code is read as 0 in
> pidfd_poll(), then in do_poll() the count will be 0 and it will block in
> poll_schedule_timeout(). Right?

No. Please note the pwq->triggered check and please read __pollwake().

But if you want to understand this you can forget about poll/select. It is
a bit complicated, in particular because it has to do set_current_state()
right  before schedule() and thus it plays games with pwq->triggered. But in
essence this doesn't differ too much from the plain wait_event-like code
(although you can also look at wait_woken/woken_wake_function).

If remove_wait_queue() could happem before wake_up_all() (like in your pseudo-
code above), then pidfd_poll() or any other ->poll() method could miss _both_
the condition and wakeup. But sys_poll() doesn't do this, so it is fine to miss
the condition and rely on wake_up_all() which ensures we won't block and the
next iteration must see condition == T.

Oleg.

[PATCH v1 1/2] Add polling support to pidfd

[Oleg Nesterov]

On 04/29, Joel Fernandes wrote:
>
> On Mon, Apr 29, 2019@04:20:30PM +0200, Oleg Nesterov wrote:
> > On 04/29, Joel Fernandes wrote:
> > >
> > > However, in your code above, it is avoided because we get:
> > >
> > > Task A (poller)		Task B (exiting task being polled)
> > > ------------            ----------------
> > > poll() called
> > > add_wait_queue()
> > > 			exit_state is set to non-zero
> > > read exit_state
> > > remove_wait_queue()
> > > 			wake_up_all()
> >
> > just to clarify... No, sys_poll() path doesn't do remove_wait_queue() until
> > it returns to user mode, and that is why we can't race with set-exit_code +
> > wake_up().
>
> I didn't follow what you mean, the removal from the waitqueue happens in
> free_poll_entry() called from poll_freewait() which happens from
> do_sys_poll() which is before the syscall returns to user mode. Could you
> explain more?

Hmm. I do not really understand the question... Sure, do_sys_poll() does
poll_freewait() before sysret or even before return from syscall, but why
does this matter? This is the exit path, it frees the memory, does fput(),
etc, f_op->poll() won't be call after that.

> > pidfd_poll() can race with the exiting task, miss exit_code != 0, and return
> > zero. However, do_poll() won't block after that and pidfd_poll() will be called
> > again.
>
> Here also I didn't follow what you mean. If exit_code is read as 0 in
> pidfd_poll(), then in do_poll() the count will be 0 and it will block in
> poll_schedule_timeout(). Right?

No. Please note the pwq->triggered check and please read __pollwake().

But if you want to understand this you can forget about poll/select. It is
a bit complicated, in particular because it has to do set_current_state()
right  before schedule() and thus it plays games with pwq->triggered. But in
essence this doesn't differ too much from the plain wait_event-like code
(although you can also look at wait_woken/woken_wake_function).

If remove_wait_queue() could happem before wake_up_all() (like in your pseudo-
code above), then pidfd_poll() or any other ->poll() method could miss _both_
the condition and wakeup. But sys_poll() doesn't do this, so it is fine to miss
the condition and rely on wake_up_all() which ensures we won't block and the
next iteration must see condition == T.

Oleg.

[PATCH v1 1/2] Add polling support to pidfd

[oleg]

On 04/30, Oleg Nesterov wrote:
>
> > > pidfd_poll() can race with the exiting task, miss exit_code != 0, and return
> > > zero. However, do_poll() won't block after that and pidfd_poll() will be called
> > > again.
> >
> > Here also I didn't follow what you mean. If exit_code is read as 0 in
> > pidfd_poll(), then in do_poll() the count will be 0 and it will block in
> > poll_schedule_timeout(). Right?
>
> No. Please note the pwq->triggered check and please read __pollwake().
>
> But if you want to understand this you can forget about poll/select. It is
> a bit complicated, in particular because it has to do set_current_state()
> right  before schedule() and thus it plays games with pwq->triggered. But in
> essence this doesn't differ too much from the plain wait_event-like code
> (although you can also look at wait_woken/woken_wake_function).
>
> If remove_wait_queue() could happem before wake_up_all() (like in your pseudo-
> code above), then pidfd_poll() or any other ->poll() method could miss _both_
> the condition and wakeup. But sys_poll() doesn't do this, so it is fine to miss
> the condition and rely on wake_up_all() which ensures we won't block and the
> next iteration must see condition == T.

Oh, just in case... If it is not clear, of course I am talking about the case
when wake_up_call() was already called when we check the condition. Otherwise
everything is simple.

Oleg.

[PATCH v1 1/2] Add polling support to pidfd

[Oleg Nesterov]

On 04/30, Oleg Nesterov wrote:
>
> > > pidfd_poll() can race with the exiting task, miss exit_code != 0, and return
> > > zero. However, do_poll() won't block after that and pidfd_poll() will be called
> > > again.
> >
> > Here also I didn't follow what you mean. If exit_code is read as 0 in
> > pidfd_poll(), then in do_poll() the count will be 0 and it will block in
> > poll_schedule_timeout(). Right?
>
> No. Please note the pwq->triggered check and please read __pollwake().
>
> But if you want to understand this you can forget about poll/select. It is
> a bit complicated, in particular because it has to do set_current_state()
> right  before schedule() and thus it plays games with pwq->triggered. But in
> essence this doesn't differ too much from the plain wait_event-like code
> (although you can also look at wait_woken/woken_wake_function).
>
> If remove_wait_queue() could happem before wake_up_all() (like in your pseudo-
> code above), then pidfd_poll() or any other ->poll() method could miss _both_
> the condition and wakeup. But sys_poll() doesn't do this, so it is fine to miss
> the condition and rely on wake_up_all() which ensures we won't block and the
> next iteration must see condition == T.

Oh, just in case... If it is not clear, of course I am talking about the case
when wake_up_call() was already called when we check the condition. Otherwise
everything is simple.

Oleg.

[PATCH v1 1/2] Add polling support to pidfd

[joel]

On Tue, Apr 30, 2019 at 01:53:33PM +0200, Oleg Nesterov wrote:
> On 04/29, Joel Fernandes wrote:
> >
> > On Mon, Apr 29, 2019 at 04:20:30PM +0200, Oleg Nesterov wrote:
> > > On 04/29, Joel Fernandes wrote:
> > > >
> > > > However, in your code above, it is avoided because we get:
> > > >
> > > > Task A (poller)		Task B (exiting task being polled)
> > > > ------------            ----------------
> > > > poll() called
> > > > add_wait_queue()
> > > > 			exit_state is set to non-zero
> > > > read exit_state
> > > > remove_wait_queue()
> > > > 			wake_up_all()
> > >
> > > just to clarify... No, sys_poll() path doesn't do remove_wait_queue() until
> > > it returns to user mode, and that is why we can't race with set-exit_code +
> > > wake_up().
> >
> > I didn't follow what you mean, the removal from the waitqueue happens in
> > free_poll_entry() called from poll_freewait() which happens from
> > do_sys_poll() which is before the syscall returns to user mode. Could you
> > explain more?
> 
> Hmm. I do not really understand the question... Sure, do_sys_poll() does
> poll_freewait() before sysret or even before return from syscall, but why
> does this matter? This is the exit path, it frees the memory, does fput(),
> etc, f_op->poll() won't be call after that.

Ok, we are on the same page on this.

> > > pidfd_poll() can race with the exiting task, miss exit_code != 0, and return
> > > zero. However, do_poll() won't block after that and pidfd_poll() will be called
> > > again.
> >
> > Here also I didn't follow what you mean. If exit_code is read as 0 in
> > pidfd_poll(), then in do_poll() the count will be 0 and it will block in
> > poll_schedule_timeout(). Right?
> 
> No. Please note the pwq->triggered check and please read __pollwake().
> 
> But if you want to understand this you can forget about poll/select. It is
> a bit complicated, in particular because it has to do set_current_state()
> right  before schedule() and thus it plays games with pwq->triggered. But in
> essence this doesn't differ too much from the plain wait_event-like code
> (although you can also look at wait_woken/woken_wake_function).
> 
> If remove_wait_queue() could happem before wake_up_all() (like in your pseudo-
> code above), then pidfd_poll() or any other ->poll() method could miss _both_
> the condition and wakeup. But sys_poll() doesn't do this, so it is fine to miss
> the condition and rely on wake_up_all() which ensures we won't block and the
> next iteration must see condition == T.

Agreed. In my pseudo-code above, I meant removal from waitqueue only once we
are not going to be blocking in poll and returning to userspace. I may have
messed the sequence of events, but my point was to show the race I had in
mind (missing a wake up due to adding to the waitqueue too late inside
pidfd_poll()).  Anyway, I will repost with your suggested change and send it
soon. Thanks for the discussions.

thanks,

 - Joel

[PATCH v1 1/2] Add polling support to pidfd

[Joel Fernandes]

On Tue, Apr 30, 2019@01:53:33PM +0200, Oleg Nesterov wrote:
> On 04/29, Joel Fernandes wrote:
> >
> > On Mon, Apr 29, 2019@04:20:30PM +0200, Oleg Nesterov wrote:
> > > On 04/29, Joel Fernandes wrote:
> > > >
> > > > However, in your code above, it is avoided because we get:
> > > >
> > > > Task A (poller)		Task B (exiting task being polled)
> > > > ------------            ----------------
> > > > poll() called
> > > > add_wait_queue()
> > > > 			exit_state is set to non-zero
> > > > read exit_state
> > > > remove_wait_queue()
> > > > 			wake_up_all()
> > >
> > > just to clarify... No, sys_poll() path doesn't do remove_wait_queue() until
> > > it returns to user mode, and that is why we can't race with set-exit_code +
> > > wake_up().
> >
> > I didn't follow what you mean, the removal from the waitqueue happens in
> > free_poll_entry() called from poll_freewait() which happens from
> > do_sys_poll() which is before the syscall returns to user mode. Could you
> > explain more?
> 
> Hmm. I do not really understand the question... Sure, do_sys_poll() does
> poll_freewait() before sysret or even before return from syscall, but why
> does this matter? This is the exit path, it frees the memory, does fput(),
> etc, f_op->poll() won't be call after that.

Ok, we are on the same page on this.

> > > pidfd_poll() can race with the exiting task, miss exit_code != 0, and return
> > > zero. However, do_poll() won't block after that and pidfd_poll() will be called
> > > again.
> >
> > Here also I didn't follow what you mean. If exit_code is read as 0 in
> > pidfd_poll(), then in do_poll() the count will be 0 and it will block in
> > poll_schedule_timeout(). Right?
> 
> No. Please note the pwq->triggered check and please read __pollwake().
> 
> But if you want to understand this you can forget about poll/select. It is
> a bit complicated, in particular because it has to do set_current_state()
> right  before schedule() and thus it plays games with pwq->triggered. But in
> essence this doesn't differ too much from the plain wait_event-like code
> (although you can also look at wait_woken/woken_wake_function).
> 
> If remove_wait_queue() could happem before wake_up_all() (like in your pseudo-
> code above), then pidfd_poll() or any other ->poll() method could miss _both_
> the condition and wakeup. But sys_poll() doesn't do this, so it is fine to miss
> the condition and rely on wake_up_all() which ensures we won't block and the
> next iteration must see condition == T.

Agreed. In my pseudo-code above, I meant removal from waitqueue only once we
are not going to be blocking in poll and returning to userspace. I may have
messed the sequence of events, but my point was to show the race I had in
mind (missing a wake up due to adding to the waitqueue too late inside
pidfd_poll()).  Anyway, I will repost with your suggested change and send it
soon. Thanks for the discussions.

thanks,

 - Joel