From: Steffen Klassert <steffen.klassert@secunet.com>
To: Frederic Weisbecker <frederic@kernel.org>
Cc: Varad Gautam <varad.gautam@suse.com>,
<linux-kernel@vger.kernel.org>,
linux-rt-users <linux-rt-users@vger.kernel.org>,
<netdev@vger.kernel.org>,
Herbert Xu <herbert@gondor.apana.org.au>,
"David S. Miller" <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>, Florian Westphal <fw@strlen.de>,
"Ahmed S. Darwish" <a.darwish@linutronix.de>,
<stable@vger.kernel.org>
Subject: Re: [PATCH] xfrm: policy: Restructure RCU-read locking in xfrm_sk_policy_lookup
Date: Tue, 22 Jun 2021 14:33:43 +0200 [thread overview]
Message-ID: <20210622123343.GD40979@gauss3.secunet.de> (raw)
In-Reply-To: <20210622115124.GA109262@lothringen>
On Tue, Jun 22, 2021 at 01:51:24PM +0200, Frederic Weisbecker wrote:
> On Tue, Jun 22, 2021 at 01:21:59PM +0200, Steffen Klassert wrote:
> > On Mon, Jun 21, 2021 at 01:05:28PM +0200, Steffen Klassert wrote:
> > > On Mon, Jun 21, 2021 at 11:11:18AM +0200, Varad Gautam wrote:
> > > >
> > > > Right, I misread the call chain - security_xfrm_policy_lookup does not reach
> > > > xfrm_policy_lookup, making this patch unnecessary. The bug I have is:
> > > >
> > > > T1, holding hash_resize_mutex and sleeping inside synchronize_rcu:
> > > >
> > > > __schedule
> > > > schedule
> > > > schedule_timeout
> > > > wait_for_completion
> > > > __wait_rcu_gp
> > > > synchronize_rcu
> > > > xfrm_hash_resize
> > > >
> > > > And T2 producing RCU-stalls since it blocked on the mutex:
> > > >
> > > > __schedule
> > > > schedule
> > > > __rt_mutex_slowlock
> > > > rt_mutex_slowlock_locked
> > > > rt_mutex_slowlock
> > > > xfrm_policy_lookup_bytype.constprop.77
> > >
> > > Ugh, why does xfrm_policy_lookup_bytype use a mutex? This is called
> > > in the receive path inside a sofirq.
> > >
> > > The bug was introduced by:
> > >
> > > commit 77cc278f7b202e4f16f8596837219d02cb090b96
> > > Author: Ahmed S. Darwish <a.darwish@linutronix.de>
> > > Date: Mon Jul 20 17:55:22 2020 +0200
> > >
> > > xfrm: policy: Use sequence counters with associated lock
> > >
> > > A sequence counter write side critical section must be protected by some
> > > form of locking to serialize writers. If the serialization primitive is
> > > not disabling preemption implicitly, preemption has to be explicitly
> > > disabled before entering the sequence counter write side critical
> > > section.
> > >
> > > A plain seqcount_t does not contain the information of which lock must
> > > be held when entering a write side critical section.
> > >
> > > Use the new seqcount_spinlock_t and seqcount_mutex_t data types instead,
> > > which allow to associate a lock with the sequence counter. This enables
> > > lockdep to verify that the lock used for writer serialization is held
> > > when the write side critical section is entered.
> > >
> > > If lockdep is disabled this lock association is compiled out and has
> > > neither storage size nor runtime overhead.
> > >
> > > Signed-off-by: Ahmed S. Darwish <a.darwish@linutronix.de>
> > > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > > Link: https://lkml.kernel.org/r/20200720155530.1173732-17-a.darwish@linutronix.de
> > >
> > > This uses a seqcount_mutex_t for xfrm_policy_hash_generation, that's
> > > wrong.
> >
> > Varad, can you try to replace the seqcount_mutex_t for xfrm_policy_hash_generation
> > by a seqcount_spinlock_t? I'm not familiar with that seqcount changes,
> > but we should not end up with using a mutex in this codepath.
>
> Something like this? (beware, untested, also I don't know if the read side
> should then disable bh, doesn't look necessary for PREEMPT_RT, but I may be
> missing something...)
Looking a bit deeper into this it seems that the problem is that
xfrm_policy_hash_generation and hash_resize_mutex do not protect
the same thing.
hash_resize_mutex protects user configuration against a worker thread
that rebalances the hash buckets. xfrm_policy_hash_generation protects
user configuration against the data path that runs in softirq.
Finally the following line from xfrm_init() relates these two:
seqcount_mutex_init(&xfrm_policy_hash_generation, &hash_resize_mutex);
That looks a bit odd. This line was also introduced with the above
mentioned patch.
next prev parent reply other threads:[~2021-06-22 12:33 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-18 14:11 [PATCH] xfrm: policy: Restructure RCU-read locking in xfrm_sk_policy_lookup Varad Gautam
2021-06-21 8:29 ` Steffen Klassert
2021-06-21 9:11 ` Varad Gautam
2021-06-21 11:05 ` Steffen Klassert
2021-06-22 11:21 ` Steffen Klassert
2021-06-22 11:51 ` Frederic Weisbecker
2021-06-22 12:33 ` Steffen Klassert [this message]
2021-06-23 5:27 ` Steffen Klassert
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210622123343.GD40979@gauss3.secunet.de \
--to=steffen.klassert@secunet.com \
--cc=a.darwish@linutronix.de \
--cc=davem@davemloft.net \
--cc=frederic@kernel.org \
--cc=fw@strlen.de \
--cc=herbert@gondor.apana.org.au \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rt-users@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=stable@vger.kernel.org \
--cc=varad.gautam@suse.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).