From: Matthew Wilcox <willy@infradead.org>
To: jackm <jackm@dev.mellanox.co.il>
Cc: hans.westgaard.ry@oracle.com,
"Doug Ledford" <dledford@redhat.com>,
"Jason Gunthorpe" <jgg@mellanox.com>,
"Matthew Wilcox" <mawilcox@microsoft.com>,
linux-rdma@vger.kernel.org,
"Håkon Bugge" <haakon.bugge@oracle.com>,
"Parav Pandit" <parav@mellanox.com>,
"Pravin Shedge" <pravin.shedge4linux@gmail.com>,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH 2/2] IB/mad: Use IDR for agent IDs
Date: Tue, 12 Jun 2018 05:12:51 -0700 [thread overview]
Message-ID: <20180612121251.GC19433@bombadil.infradead.org> (raw)
In-Reply-To: <20180612115046.00007213@dev.mellanox.co.il>
On Tue, Jun 12, 2018 at 11:50:46AM +0300, jackm wrote:
> On Fri, 8 Jun 2018 10:42:18 -0700
> Matthew Wilcox <willy@infradead.org> wrote:
>
> > + rcu_read_lock();
> > + mad_agent = idr_find(&ib_mad_clients, hi_tid);
> > + if (mad_agent
> > && !atomic_inc_not_zero(&mad_agent->refcount))
> > + mad_agent = NULL;
> > + rcu_read_unlock();
>
> Hi Matthew,
>
> I don't see the flow which can explain using atomic_inc_not_zero() here.
>
> The refcount will go to zero only when unregister_mad_agent() is
> called (code below, see asterisks):
> idr_lock(&ib_mad_clients);
> *** idr_remove(&ib_mad_clients, mad_agent_priv->agent.hi_tid);
> idr_unlock(&ib_mad_clients);
>
> flush_workqueue(port_priv->wq);
> ib_cancel_rmpp_recvs(mad_agent_priv);
>
> *** deref_mad_agent(mad_agent_priv);
> [JPM] The call to idr_find in the interrupt context
> would need to occur here for the refcount to have a
> possibility of being zero.
> Shouldn't idr_find in the interrupt context fail, since
> idr_remove has already been invoked?
RCU is tricky. Here's the flow:
CPU 0 CPU 1
rcu_read_lock();
mad_agent = idr_find(&ib_mad_clients, hi_tid);
idr_lock(&ib_mad_clients);
idr_remove(&ib_mad_clients, mad_agent_priv->agent.hi_tid);
idr_unlock(&ib_mad_clients);
flush_workqueue(port_priv->wq);
ib_cancel_rmpp_recvs(mad_agent_priv);
deref_mad_agent(mad_agent_priv);
Now, you're going to argue that CPU 0 is running in interrupt context, but
with virtualisation, it can have the CPU taken away from it at any time.
This window which looks like a couple of instructions long can actually
be seconds long.
> wait_for_completion(&mad_agent_priv->comp);
>
> The refcount will be able to go to zero only after deref_mad_agent is
> called above. Before this, however, idr_remove() has been called --
> so, if my understanding is correct, the idr_find call in
> find_mad_agent() should not succeed since the refcount can get to zero
> only AFTER the idr_remove call.
>
> Could you please explain the flow which can result in idr_find
> succeeding (in the interrupt context) after idr_remove has been invoked
> (in the process context)? Will idr_find succeed even after
> idr_remove, and only fail after kfree_rcu is invoked as well? (or,
> maybe after some garbage-collection delay?)
Ordering is weird in SMP systems. You can appear to have causality
violations when you're operating locklessly (and rcu_read_lock()
is essentially lockless). So we can absolutely observe the store to
agent->refcount before we observe the store to idr->agent.
Documentation/memory-barriers.txt has a LOT more information on this.
next prev parent reply other threads:[~2018-06-12 12:12 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-06-08 17:42 [PATCH 0/2] Convert IB/mad to use an IDR for agent IDs Matthew Wilcox
2018-06-08 17:42 ` [PATCH 1/2] IB/mad: Agent registration is process context only Matthew Wilcox
2018-06-12 20:38 ` Jason Gunthorpe
2018-06-08 17:42 ` [PATCH 2/2] IB/mad: Use IDR for agent IDs Matthew Wilcox
2018-06-10 6:30 ` Leon Romanovsky
2018-06-10 10:43 ` Matthew Wilcox
2018-06-10 12:25 ` Leon Romanovsky
2018-06-10 20:30 ` Jason Gunthorpe
2018-06-11 4:34 ` Leon Romanovsky
2018-06-11 4:42 ` Jason Gunthorpe
2018-06-11 6:19 ` jackm
2018-06-11 16:19 ` Jason Gunthorpe
2018-06-12 4:59 ` jackm
2018-06-12 14:33 ` Jason Gunthorpe
2018-06-12 8:50 ` jackm
2018-06-12 12:12 ` Matthew Wilcox [this message]
2018-06-12 20:33 ` Jason Gunthorpe
2018-06-13 0:07 ` Matthew Wilcox
2018-06-13 7:36 ` Leon Romanovsky
2018-06-13 7:56 ` jackm
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180612121251.GC19433@bombadil.infradead.org \
--to=willy@infradead.org \
--cc=dledford@redhat.com \
--cc=haakon.bugge@oracle.com \
--cc=hans.westgaard.ry@oracle.com \
--cc=jackm@dev.mellanox.co.il \
--cc=jgg@mellanox.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=mawilcox@microsoft.com \
--cc=parav@mellanox.com \
--cc=pravin.shedge4linux@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).