From: Manfred Spraul <manfred@colorfullife.com>
To: Jiebin Sun <jiebin.sun@intel.com>,
akpm@linux-foundation.org, vasily.averin@linux.dev,
shakeelb@google.com, dennis@kernel.org, tj@kernel.org,
cl@linux.com, ebiederm@xmission.com, legion@kernel.org,
alexander.mikhalitsyn@virtuozzo.com, linux-mm@kvack.org,
linux-kernel@vger.kernel.org
Cc: tim.c.chen@intel.com, feng.tang@intel.com, ying.huang@intel.com,
tianyou.li@intel.com, wangyang.guo@intel.com,
Tim Chen <tim.c.chen@linux.intel.com>,
kernel test robot <lkp@intel.com>,
1vier1@web.de
Subject: Re: [PATCH v6 1/2] percpu: Add percpu_counter_add_local and percpu_counter_sub_local
Date: Sun, 18 Sep 2022 13:08:16 +0200 [thread overview]
Message-ID: <b736022c-5028-a06e-5edb-f5cb526b0821@colorfullife.com> (raw)
In-Reply-To: <20220913192538.3023708-2-jiebin.sun@intel.com>
[-- Attachment #1: Type: text/plain, Size: 3332 bytes --]
Hi Jiebin,
On 9/13/22 21:25, Jiebin Sun wrote:
>
> +/*
> + * With percpu_counter_add_local() and percpu_counter_sub_local(), counts
> + * are accumulated in local per cpu counter and not in fbc->count until
> + * local count overflows PERCPU_COUNTER_LOCAL_BATCH. This makes counter
> + * write efficient.
> + * But percpu_counter_sum(), instead of percpu_counter_read(), needs to be
> + * used to add up the counts from each CPU to account for all the local
> + * counts. So percpu_counter_add_local() and percpu_counter_sub_local()
> + * should be used when a counter is updated frequently and read rarely.
> + */
> +static inline void
> +percpu_counter_add_local(struct percpu_counter *fbc, s64 amount)
> +{
> + percpu_counter_add_batch(fbc, amount, PERCPU_COUNTER_LOCAL_BATCH);
> +}
> +
Unrelated to your patch, and not relevant for ipc/msg as the functions
are not called from interrupts, but:
Aren't there races with interrupts?
> *
> * This function is both preempt and irq safe. The former is due to
> explicit
> * preemption disable. The latter is guaranteed by the fact that the
> slow path
> * is explicitly protected by an irq-safe spinlock whereas the fast
> patch uses
> * this_cpu_add which is irq-safe by definition. Hence there is no need
> muck
> * with irq state before calling this one
> */
> void percpu_counter_add_batch(struct percpu_counter *fbc, s64 amount,
> s32 batch)
> {
> s64 count;
>
> preempt_disable();
> count = __this_cpu_read(*fbc->counters) + amount;
> if (abs(count) >= batch) {
> unsigned long flags;
> raw_spin_lock_irqsave(&fbc->lock, flags);
> fbc->count += count;
> __this_cpu_sub(*fbc->counters, count - amount);
> raw_spin_unlock_irqrestore(&fbc->lock, flags);
> } else {
> this_cpu_add(*fbc->counters, amount);
> }
> preempt_enable();
> }
> EXPORT_SYMBOL(percpu_counter_add_batch);
>
>
Race 1:
start: __this_cpu_read(*fbc->counters) = INT_MAX-1.
Call: per_cpu_counter_add_batch(fbc, 1, INT_MAX);
Result:
count=INT_MAX;
if (abs(count) >= batch) { // branch taken
before the raw_spin_lock_irqsave():
Interrupt
Within interrupt:
per_cpu_counter_add_batch(fbc, -2*(INT_MAX-1), INT_MAX)
count=-(INT_MAX-1);
branch not taken
this_cpu_add() updates fbc->counters, new value is -(INT_MAX-1)
exit interrupt
raw_spin_lock_irqsave()
__this_cpu_sub(*fbc->counters, count - amount)
will substract INT_MAX-1 from *fbc->counters. But the value is already
-(INT_MAX-1) -> underflow.
Race 2: (much simpler)
start: __this_cpu_read(*fbc->counters) = 0.
Call: per_cpu_counter_add_batch(fbc, INT_MAX-1, INT_MAX);
amont=INT_MAX-1;
- branch not taken.
before this_cpu_add(): interrupt
within the interrupt: call per_cpu_counter_add_batch(fbc, INT_MAX-1,
INT_MAX)
new value of *fbc->counters: INT_MAX-1.
exit interrupt
outside interrupt:
this_cpu_add(*fbc->counters, amount);
<<< overflow.
Attached is an incomplete patch (untested).
If needed, I could check the whole file and add/move the required
local_irq_save() calls.
--
Manfred
[-- Attachment #2: 0001-lib-percpu_counter-RFC-potential-overflow-underflow.patch --]
[-- Type: text/x-patch, Size: 1893 bytes --]
From 6a1d2a4beb180241b63f9bf57454bbe031915dd1 Mon Sep 17 00:00:00 2001
From: Manfred Spraul <manfred@colorfullife.com>
Date: Sun, 18 Sep 2022 12:17:27 +0200
Subject: [PATCH] lib/percpu_counter: [RFC] potential overflow/underflow
If an interrupt happens between __this_cpu_read(*fbc->counters) and
this_cpu_add(*fbc->counters, amount), and that interrupt modifies
the per_cpu_counter, then the this_cpu_add() after the interrupt
returns may under/overflow.
Thus: Disable interrupts.
Note: The patch is incomplete, if the race is real, then
more functions than just percpu_counter_add_batch() needs to be
updated.
Especially, the !CONFIG_SMP code looks wrong to me as well:
> static inline void
> percpu_counter_add(struct percpu_counter *fbc, s64 amount)
> {
> preempt_disable();
> fbc->count += amount;
> preempt_enable();
> }
The update of fbc->count is not IRQ safe.
Signed-off-by: Manfred Spraul <manfred@colorfullife.com>
---
lib/percpu_counter.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/lib/percpu_counter.c b/lib/percpu_counter.c
index ed610b75dc32..39de94d59b4f 100644
--- a/lib/percpu_counter.c
+++ b/lib/percpu_counter.c
@@ -82,18 +82,20 @@ EXPORT_SYMBOL(percpu_counter_set);
void percpu_counter_add_batch(struct percpu_counter *fbc, s64 amount, s32 batch)
{
s64 count;
+ unsigned long flags;
preempt_disable();
+ local_irq_save(flags);
count = __this_cpu_read(*fbc->counters) + amount;
if (abs(count) >= batch) {
- unsigned long flags;
- raw_spin_lock_irqsave(&fbc->lock, flags);
+ raw_spin_lock(&fbc->lock);
fbc->count += count;
__this_cpu_sub(*fbc->counters, count - amount);
- raw_spin_unlock_irqrestore(&fbc->lock, flags);
+ raw_spin_unlock(&fbc->lock);
} else {
this_cpu_add(*fbc->counters, amount);
}
+ local_irq_restore(flags);
preempt_enable();
}
EXPORT_SYMBOL(percpu_counter_add_batch);
--
2.37.2
next prev parent reply other threads:[~2022-09-18 11:08 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-02 15:22 [PATCH] ipc/msg.c: mitigate the lock contention with percpu counter Jiebin Sun
2022-09-02 16:06 ` Andrew Morton
2022-09-05 11:54 ` Sun, Jiebin
2022-09-02 16:27 ` Shakeel Butt
2022-09-05 12:02 ` Sun, Jiebin
2022-09-06 18:44 ` Tim Chen
2022-09-07 9:39 ` Sun, Jiebin
2022-09-07 20:43 ` Andrew Morton
2022-09-07 17:25 ` [PATCH v4] ipc/msg: " Jiebin Sun
2022-09-07 16:01 ` Tim Chen
2022-09-07 21:34 ` Andrew Morton
2022-09-07 22:10 ` Tim Chen
2022-09-08 8:25 ` Sun, Jiebin
2022-09-08 15:38 ` Andrew Morton
2022-09-08 16:15 ` Dennis Zhou
2022-09-03 19:35 ` [PATCH] ipc/msg.c: " Manfred Spraul
2022-09-05 12:12 ` Sun, Jiebin
2022-09-05 19:35 ` [PATCH v2 0/2] ipc/msg: mitigate the lock contention in ipc/msg Jiebin Sun
2022-09-05 19:35 ` [PATCH v2 2/2] ipc/msg: mitigate the lock contention with percpu counter Jiebin Sun
2022-09-05 19:35 ` [PATCH v2 1/2] percpu: Add percpu_counter_add_local Jiebin Sun
2022-09-05 19:31 ` Shakeel Butt
2022-09-06 8:41 ` Sun, Jiebin
2022-09-06 16:54 ` [PATCH v3 0/2] ipc/msg: mitigate the lock contention in ipc/msg Jiebin Sun
2022-09-06 16:54 ` [PATCH v3 1/2] percpu: Add percpu_counter_add_local Jiebin Sun
2022-09-06 16:54 ` [PATCH v3 2/2] ipc/msg: mitigate the lock contention with percpu counter Jiebin Sun
2022-09-09 20:36 ` [PATCH v5 0/2] ipc/msg: mitigate the lock contention in ipc/msg Jiebin Sun
2022-09-09 20:36 ` [PATCH v5 1/2] percpu: Add percpu_counter_add_local and percpu_counter_sub_local Jiebin Sun
2022-09-09 16:37 ` Tim Chen
2022-09-10 1:37 ` kernel test robot
2022-09-10 8:15 ` kernel test robot
2022-09-10 8:26 ` kernel test robot
2022-09-09 20:36 ` [PATCH v5 2/2] ipc/msg: mitigate the lock contention with percpu counter Jiebin Sun
2022-09-09 16:11 ` Tim Chen
2022-09-13 19:25 ` [PATCH v6 0/2] ipc/msg: mitigate the lock contention in ipc/msg Jiebin Sun
2022-09-13 19:25 ` [PATCH v6 1/2] percpu: Add percpu_counter_add_local and percpu_counter_sub_local Jiebin Sun
2022-09-18 11:08 ` Manfred Spraul [this message]
2022-09-20 6:01 ` Sun, Jiebin
2022-09-13 19:25 ` [PATCH v6 2/2] ipc/msg: mitigate the lock contention with percpu counter Jiebin Sun
2022-09-18 12:53 ` Manfred Spraul
2022-09-20 2:36 ` Sun, Jiebin
2022-09-20 4:53 ` Manfred Spraul
2022-09-20 5:50 ` Sun, Jiebin
2022-09-20 15:08 ` [PATCH] ipc/msg: avoid negative value by overflow in msginfo Jiebin Sun
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b736022c-5028-a06e-5edb-f5cb526b0821@colorfullife.com \
--to=manfred@colorfullife.com \
--cc=1vier1@web.de \
--cc=akpm@linux-foundation.org \
--cc=alexander.mikhalitsyn@virtuozzo.com \
--cc=cl@linux.com \
--cc=dennis@kernel.org \
--cc=ebiederm@xmission.com \
--cc=feng.tang@intel.com \
--cc=jiebin.sun@intel.com \
--cc=legion@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=lkp@intel.com \
--cc=shakeelb@google.com \
--cc=tianyou.li@intel.com \
--cc=tim.c.chen@intel.com \
--cc=tim.c.chen@linux.intel.com \
--cc=tj@kernel.org \
--cc=vasily.averin@linux.dev \
--cc=wangyang.guo@intel.com \
--cc=ying.huang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).