From: Alex Belits <abelits@marvell.com> To: "frederic@kernel.org" <frederic@kernel.org>, "rostedt@goodmis.org" <rostedt@goodmis.org> Cc: Prasun Kapoor <pkapoor@marvell.com>, "mingo@kernel.org" <mingo@kernel.org>, "davem@davemloft.net" <davem@davemloft.net>, "linux-api@vger.kernel.org" <linux-api@vger.kernel.org>, "peterz@infradead.org" <peterz@infradead.org>, "linux-arch@vger.kernel.org" <linux-arch@vger.kernel.org>, "catalin.marinas@arm.com" <catalin.marinas@arm.com>, "tglx@linutronix.de" <tglx@linutronix.de>, "will@kernel.org" <will@kernel.org>, "linux-arm-kernel@lists.infradead.org" <linux-arm-kernel@lists.infradead.org>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "netdev@vger.kernel.org" <netdev@vger.kernel.org> Subject: [PATCH v4 12/13] task_isolation: ringbuffer: don't interrupt CPUs running isolated tasks on buffer resize Date: Wed, 22 Jul 2020 14:59:13 +0000 [thread overview] Message-ID: <041a4c844b4a0f69aba6df4686e1952dd0f91931.camel@marvell.com> (raw) In-Reply-To: <04be044c1bcd76b7438b7563edc35383417f12c8.camel@marvell.com> From: Yuri Norov <ynorov@marvell.com> CPUs running isolated tasks are in userspace, so they don't have to perform ring buffer updates immediately. If ring_buffer_resize() schedules the update on those CPUs, isolation is broken. To prevent that, updates for CPUs running isolated tasks are performed locally, like for offline CPUs. A race condition between this update and isolation breaking is avoided at the cost of disabling per_cpu buffer writing for the time of update when it coincides with isolation breaking. Signed-off-by: Yuri Norov <ynorov@marvell.com> [abelits@marvell.com: updated to prevent race with isolation breaking] Signed-off-by: Alex Belits <abelits@marvell.com> --- kernel/trace/ring_buffer.c | 63 ++++++++++++++++++++++++++++++++++---- 1 file changed, 57 insertions(+), 6 deletions(-) diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c index 00867ff82412..22d4731f0def 100644 --- a/kernel/trace/ring_buffer.c +++ b/kernel/trace/ring_buffer.c @@ -21,6 +21,7 @@ #include <linux/delay.h> #include <linux/slab.h> #include <linux/init.h> +#include <linux/isolation.h> #include <linux/hash.h> #include <linux/list.h> #include <linux/cpu.h> @@ -1705,6 +1706,38 @@ static void update_pages_handler(struct work_struct *work) complete(&cpu_buffer->update_done); } +static bool update_if_isolated(struct ring_buffer_per_cpu *cpu_buffer, + int cpu) +{ + bool rv = false; + + smp_rmb(); + if (task_isolation_on_cpu(cpu)) { + /* + * CPU is running isolated task. Since it may lose + * isolation and re-enter kernel simultaneously with + * this update, disable recording until it's done. + */ + atomic_inc(&cpu_buffer->record_disabled); + /* Make sure, update is done, and isolation state is current */ + smp_mb(); + if (task_isolation_on_cpu(cpu)) { + /* + * If CPU is still running isolated task, we + * can be sure that breaking isolation will + * happen while recording is disabled, and CPU + * will not touch this buffer until the update + * is done. + */ + rb_update_pages(cpu_buffer); + cpu_buffer->nr_pages_to_update = 0; + rv = true; + } + atomic_dec(&cpu_buffer->record_disabled); + } + return rv; +} + /** * ring_buffer_resize - resize the ring buffer * @buffer: the buffer to resize. @@ -1794,13 +1827,22 @@ int ring_buffer_resize(struct trace_buffer *buffer, unsigned long size, if (!cpu_buffer->nr_pages_to_update) continue; - /* Can't run something on an offline CPU. */ + /* + * Can't run something on an offline CPU. + * + * CPUs running isolated tasks don't have to + * update ring buffers until they exit + * isolation because they are in + * userspace. Use the procedure that prevents + * race condition with isolation breaking. + */ if (!cpu_online(cpu)) { rb_update_pages(cpu_buffer); cpu_buffer->nr_pages_to_update = 0; } else { - schedule_work_on(cpu, - &cpu_buffer->update_pages_work); + if (!update_if_isolated(cpu_buffer, cpu)) + schedule_work_on(cpu, + &cpu_buffer->update_pages_work); } } @@ -1849,13 +1891,22 @@ int ring_buffer_resize(struct trace_buffer *buffer, unsigned long size, get_online_cpus(); - /* Can't run something on an offline CPU. */ + /* + * Can't run something on an offline CPU. + * + * CPUs running isolated tasks don't have to update + * ring buffers until they exit isolation because they + * are in userspace. Use the procedure that prevents + * race condition with isolation breaking. + */ if (!cpu_online(cpu_id)) rb_update_pages(cpu_buffer); else { - schedule_work_on(cpu_id, + if (!update_if_isolated(cpu_buffer, cpu_id)) + schedule_work_on(cpu_id, &cpu_buffer->update_pages_work); - wait_for_completion(&cpu_buffer->update_done); + wait_for_completion(&cpu_buffer->update_done); + } } cpu_buffer->nr_pages_to_update = 0; -- 2.26.2
WARNING: multiple messages have this Message-ID (diff)
From: Alex Belits <abelits@marvell.com> To: "frederic@kernel.org" <frederic@kernel.org>, "rostedt@goodmis.org" <rostedt@goodmis.org> Cc: "linux-arch@vger.kernel.org" <linux-arch@vger.kernel.org>, "catalin.marinas@arm.com" <catalin.marinas@arm.com>, "peterz@infradead.org" <peterz@infradead.org>, Prasun Kapoor <pkapoor@marvell.com>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "davem@davemloft.net" <davem@davemloft.net>, "netdev@vger.kernel.org" <netdev@vger.kernel.org>, "linux-api@vger.kernel.org" <linux-api@vger.kernel.org>, "tglx@linutronix.de" <tglx@linutronix.de>, "will@kernel.org" <will@kernel.org>, "mingo@kernel.org" <mingo@kernel.org>, "linux-arm-kernel@lists.infradead.org" <linux-arm-kernel@lists.infradead.org> Subject: [PATCH v4 12/13] task_isolation: ringbuffer: don't interrupt CPUs running isolated tasks on buffer resize Date: Wed, 22 Jul 2020 14:59:13 +0000 [thread overview] Message-ID: <041a4c844b4a0f69aba6df4686e1952dd0f91931.camel@marvell.com> (raw) In-Reply-To: <04be044c1bcd76b7438b7563edc35383417f12c8.camel@marvell.com> From: Yuri Norov <ynorov@marvell.com> CPUs running isolated tasks are in userspace, so they don't have to perform ring buffer updates immediately. If ring_buffer_resize() schedules the update on those CPUs, isolation is broken. To prevent that, updates for CPUs running isolated tasks are performed locally, like for offline CPUs. A race condition between this update and isolation breaking is avoided at the cost of disabling per_cpu buffer writing for the time of update when it coincides with isolation breaking. Signed-off-by: Yuri Norov <ynorov@marvell.com> [abelits@marvell.com: updated to prevent race with isolation breaking] Signed-off-by: Alex Belits <abelits@marvell.com> --- kernel/trace/ring_buffer.c | 63 ++++++++++++++++++++++++++++++++++---- 1 file changed, 57 insertions(+), 6 deletions(-) diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c index 00867ff82412..22d4731f0def 100644 --- a/kernel/trace/ring_buffer.c +++ b/kernel/trace/ring_buffer.c @@ -21,6 +21,7 @@ #include <linux/delay.h> #include <linux/slab.h> #include <linux/init.h> +#include <linux/isolation.h> #include <linux/hash.h> #include <linux/list.h> #include <linux/cpu.h> @@ -1705,6 +1706,38 @@ static void update_pages_handler(struct work_struct *work) complete(&cpu_buffer->update_done); } +static bool update_if_isolated(struct ring_buffer_per_cpu *cpu_buffer, + int cpu) +{ + bool rv = false; + + smp_rmb(); + if (task_isolation_on_cpu(cpu)) { + /* + * CPU is running isolated task. Since it may lose + * isolation and re-enter kernel simultaneously with + * this update, disable recording until it's done. + */ + atomic_inc(&cpu_buffer->record_disabled); + /* Make sure, update is done, and isolation state is current */ + smp_mb(); + if (task_isolation_on_cpu(cpu)) { + /* + * If CPU is still running isolated task, we + * can be sure that breaking isolation will + * happen while recording is disabled, and CPU + * will not touch this buffer until the update + * is done. + */ + rb_update_pages(cpu_buffer); + cpu_buffer->nr_pages_to_update = 0; + rv = true; + } + atomic_dec(&cpu_buffer->record_disabled); + } + return rv; +} + /** * ring_buffer_resize - resize the ring buffer * @buffer: the buffer to resize. @@ -1794,13 +1827,22 @@ int ring_buffer_resize(struct trace_buffer *buffer, unsigned long size, if (!cpu_buffer->nr_pages_to_update) continue; - /* Can't run something on an offline CPU. */ + /* + * Can't run something on an offline CPU. + * + * CPUs running isolated tasks don't have to + * update ring buffers until they exit + * isolation because they are in + * userspace. Use the procedure that prevents + * race condition with isolation breaking. + */ if (!cpu_online(cpu)) { rb_update_pages(cpu_buffer); cpu_buffer->nr_pages_to_update = 0; } else { - schedule_work_on(cpu, - &cpu_buffer->update_pages_work); + if (!update_if_isolated(cpu_buffer, cpu)) + schedule_work_on(cpu, + &cpu_buffer->update_pages_work); } } @@ -1849,13 +1891,22 @@ int ring_buffer_resize(struct trace_buffer *buffer, unsigned long size, get_online_cpus(); - /* Can't run something on an offline CPU. */ + /* + * Can't run something on an offline CPU. + * + * CPUs running isolated tasks don't have to update + * ring buffers until they exit isolation because they + * are in userspace. Use the procedure that prevents + * race condition with isolation breaking. + */ if (!cpu_online(cpu_id)) rb_update_pages(cpu_buffer); else { - schedule_work_on(cpu_id, + if (!update_if_isolated(cpu_buffer, cpu_id)) + schedule_work_on(cpu_id, &cpu_buffer->update_pages_work); - wait_for_completion(&cpu_buffer->update_done); + wait_for_completion(&cpu_buffer->update_done); + } } cpu_buffer->nr_pages_to_update = 0; -- 2.26.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2020-07-22 14:59 UTC|newest] Thread overview: 133+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-07-22 14:44 [PATCH v4 00/13] "Task_isolation" mode Alex Belits 2020-07-22 14:44 ` Alex Belits 2020-07-22 14:44 ` Alex Belits 2020-07-22 14:47 ` [PATCH v4 01/13] task_isolation: vmstat: add quiet_vmstat_sync function Alex Belits 2020-07-22 14:47 ` Alex Belits 2020-07-22 14:47 ` Alex Belits 2020-07-22 14:47 ` Alex Belits 2020-07-22 14:48 ` [PATCH v4 02/13] task_isolation: vmstat: add vmstat_idle function Alex Belits 2020-07-22 14:48 ` Alex Belits 2020-07-22 14:48 ` Alex Belits 2020-07-22 14:49 ` [PATCH v4 03/13] task_isolation: userspace hard isolation from kernel Alex Belits 2020-07-22 14:49 ` Alex Belits 2020-07-22 14:49 ` Alex Belits 2020-10-01 13:56 ` Frederic Weisbecker 2020-10-01 13:56 ` Frederic Weisbecker 2020-10-04 14:44 ` [EXT] " Alex Belits 2020-10-04 14:44 ` Alex Belits 2020-10-04 23:14 ` Frederic Weisbecker 2020-10-04 23:14 ` Frederic Weisbecker 2020-10-05 18:52 ` Nitesh Narayan Lal 2020-10-05 18:52 ` Nitesh Narayan Lal 2020-10-06 10:35 ` Frederic Weisbecker 2020-10-06 10:35 ` Frederic Weisbecker 2020-10-17 1:13 ` Alex Belits 2020-10-17 1:13 ` Alex Belits 2020-10-17 1:08 ` Alex Belits 2020-10-17 1:08 ` Alex Belits 2020-10-17 16:08 ` Thomas Gleixner 2020-10-17 16:08 ` Thomas Gleixner 2020-10-17 16:15 ` Alex Belits 2020-10-17 16:15 ` Alex Belits 2020-10-17 20:03 ` Thomas Gleixner 2020-10-17 20:03 ` Thomas Gleixner 2020-10-06 11:01 ` Alex Belits 2020-10-06 11:01 ` Alex Belits 2020-10-01 14:40 ` Frederic Weisbecker 2020-10-01 14:40 ` Frederic Weisbecker 2020-10-04 15:01 ` [EXT] " Alex Belits 2020-10-04 15:01 ` Alex Belits 2020-07-22 14:51 ` [PATCH v4 04/13] task_isolation: Add task isolation hooks to arch-independent code Alex Belits 2020-07-22 14:51 ` Alex Belits 2020-07-22 14:51 ` Alex Belits 2020-07-22 14:51 ` Alex Belits 2020-07-22 14:51 ` [PATCH v4 05/13] task_isolation: Add xen-specific hook Alex Belits 2020-07-22 14:51 ` Alex Belits 2020-07-22 14:51 ` Alex Belits 2020-07-22 14:53 ` [PATCH 06/13] task_isolation: Add driver-specific hooks Alex Belits 2020-07-22 14:53 ` Alex Belits 2020-07-22 14:53 ` Alex Belits 2020-07-22 14:54 ` [PATCH v4 07/13] task_isolation: arch/x86: enable task isolation functionality Alex Belits 2020-07-22 14:54 ` Alex Belits 2020-07-22 14:54 ` Alex Belits 2020-07-22 14:55 ` [PATCH 08/13] task_isolation: arch/arm64: " Alex Belits 2020-07-22 14:55 ` Alex Belits 2020-07-22 14:55 ` Alex Belits 2020-07-22 14:55 ` Alex Belits 2020-07-22 14:56 ` [PATCH v4 09/13] task_isolation: arch/arm: " Alex Belits 2020-07-22 14:56 ` Alex Belits 2020-07-22 14:56 ` Alex Belits 2020-07-22 14:56 ` Alex Belits 2020-07-22 14:57 ` [PATCH v4 10/13] task_isolation: don't interrupt CPUs with tick_nohz_full_kick_cpu() Alex Belits 2020-07-22 14:57 ` Alex Belits 2020-07-22 14:57 ` Alex Belits 2020-10-01 14:44 ` Frederic Weisbecker 2020-10-01 14:44 ` Frederic Weisbecker 2020-10-04 15:22 ` [EXT] " Alex Belits 2020-10-04 15:22 ` Alex Belits 2020-10-06 21:41 ` Frederic Weisbecker 2020-10-06 21:41 ` Frederic Weisbecker 2020-10-17 0:17 ` Alex Belits 2020-10-17 0:17 ` Alex Belits 2020-07-22 14:58 ` [PATCH v4 11/13] task_isolation: net: don't flush backlog on CPUs running isolated tasks Alex Belits 2020-07-22 14:58 ` Alex Belits 2020-07-22 14:58 ` Alex Belits 2020-07-22 14:58 ` Alex Belits 2020-10-01 14:47 ` Frederic Weisbecker 2020-10-01 14:47 ` Frederic Weisbecker 2020-10-04 17:12 ` [EXT] " Alex Belits 2020-10-04 17:12 ` Alex Belits 2021-01-22 14:13 ` Marcelo Tosatti 2021-01-22 14:13 ` Marcelo Tosatti 2021-01-22 16:13 ` Paolo Abeni 2021-01-22 16:13 ` Paolo Abeni 2020-07-22 14:59 ` Alex Belits [this message] 2020-07-22 14:59 ` [PATCH v4 12/13] task_isolation: ringbuffer: don't interrupt CPUs running isolated tasks on buffer resize Alex Belits 2020-07-22 14:59 ` Alex Belits 2020-07-22 14:59 ` [PATCH 13/13] task_isolation: kick_all_cpus_sync: don't kick isolated cpus Alex Belits 2020-07-22 14:59 ` Alex Belits 2020-07-22 14:59 ` Alex Belits 2020-07-22 14:59 ` Alex Belits 2020-07-23 13:17 ` [PATCH v4 00/13] "Task_isolation" mode Thomas Gleixner 2020-07-23 13:17 ` Thomas Gleixner 2020-07-23 13:17 ` Thomas Gleixner 2020-07-23 14:26 ` Peter Zijlstra 2020-07-23 14:26 ` Peter Zijlstra 2020-07-23 14:26 ` Peter Zijlstra 2020-07-23 14:53 ` Thomas Gleixner 2020-07-23 14:53 ` Thomas Gleixner 2020-07-23 14:53 ` Thomas Gleixner 2020-07-23 14:29 ` Peter Zijlstra 2020-07-23 14:29 ` Peter Zijlstra 2020-07-23 14:29 ` Peter Zijlstra 2020-07-23 15:41 ` [EXT] " Alex Belits 2020-07-23 15:41 ` Alex Belits 2020-07-23 15:41 ` Alex Belits 2020-07-23 15:48 ` Peter Zijlstra 2020-07-23 15:48 ` Peter Zijlstra 2020-07-23 15:48 ` Peter Zijlstra 2020-07-23 16:19 ` Alex Belits 2020-07-23 16:19 ` Alex Belits 2020-07-23 16:19 ` Alex Belits 2020-07-23 15:18 ` Alex Belits 2020-07-23 15:18 ` Alex Belits 2020-07-23 15:18 ` Alex Belits 2020-07-23 15:49 ` Peter Zijlstra 2020-07-23 15:49 ` Peter Zijlstra 2020-07-23 15:49 ` Peter Zijlstra 2020-07-23 15:49 ` Peter Zijlstra 2020-07-23 16:50 ` Alex Belits 2020-07-23 16:50 ` Alex Belits 2020-07-23 16:50 ` Alex Belits 2020-07-23 21:44 ` Thomas Gleixner 2020-07-23 21:44 ` Thomas Gleixner 2020-07-23 21:44 ` Thomas Gleixner 2020-07-24 3:00 ` [EXT] " Alex Belits 2020-07-24 3:00 ` Alex Belits 2020-07-24 3:00 ` Alex Belits 2020-07-24 16:08 ` Thomas Gleixner 2020-07-24 16:08 ` Thomas Gleixner 2020-07-24 16:08 ` Thomas Gleixner 2020-07-23 21:31 ` Thomas Gleixner 2020-07-23 21:31 ` Thomas Gleixner 2020-07-23 21:31 ` Thomas Gleixner
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=041a4c844b4a0f69aba6df4686e1952dd0f91931.camel@marvell.com \ --to=abelits@marvell.com \ --cc=catalin.marinas@arm.com \ --cc=davem@davemloft.net \ --cc=frederic@kernel.org \ --cc=linux-api@vger.kernel.org \ --cc=linux-arch@vger.kernel.org \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=mingo@kernel.org \ --cc=netdev@vger.kernel.org \ --cc=peterz@infradead.org \ --cc=pkapoor@marvell.com \ --cc=rostedt@goodmis.org \ --cc=tglx@linutronix.de \ --cc=will@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.