From: Xin Long <lucien.xin@gmail.com>
To: Lee Jones <lee.jones@linaro.org>
Cc: Jakub Kicinski <kuba@kernel.org>,
LKML <linux-kernel@vger.kernel.org>,
Vlad Yasevich <vyasevich@gmail.com>,
Neil Horman <nhorman@tuxdriver.com>,
Marcelo Ricardo Leitner <marcelo.leitner@gmail.com>,
"David S. Miller" <davem@davemloft.net>,
lksctp developers <linux-sctp@vger.kernel.org>,
"H.P. Yarroll" <piggy@acm.org>,
Karl Knutson <karl@athena.chicago.il.us>,
Jon Grimm <jgrimm@us.ibm.com>,
Xingang Guo <xingang.guo@intel.com>,
Hui Huang <hui.huang@nokia.com>,
Sridhar Samudrala <sri@us.ibm.com>,
Daisy Chang <daisyc@us.ibm.com>, Ryan Layer <rmlayer@us.ibm.com>,
Kevin Gao <kevin.gao@intel.com>,
network dev <netdev@vger.kernel.org>
Subject: Re: [RESEND 2/2] sctp: hold cached endpoints to prevent possible UAF
Date: Thu, 16 Dec 2021 13:22:05 -0500 [thread overview]
Message-ID: <CADvbK_cxMbYwkuN_ZUvHY-7ahc9ff+jbuPkKn6CA=yqMk=SKVw@mail.gmail.com> (raw)
In-Reply-To: <CADvbK_cexKiVATn=dPrWqoS0qM-bM0UcSkx8Xqz5ibEKQizDVg@mail.gmail.com>
(
On Thu, Dec 16, 2021 at 1:12 PM Xin Long <lucien.xin@gmail.com> wrote:
>
> On Thu, Dec 16, 2021 at 12:14 PM Lee Jones <lee.jones@linaro.org> wrote:
> >
> > On Thu, 16 Dec 2021, Lee Jones wrote:
> >
> > > On Thu, 16 Dec 2021, Xin Long wrote:
> > >
> > > > On Thu, Dec 16, 2021 at 11:39 AM Lee Jones <lee.jones@linaro.org> wrote:
> > > > >
> > > > > On Thu, 16 Dec 2021, Xin Long wrote:
> > > > >
> > > > > > On Wed, Dec 15, 2021 at 8:48 PM Jakub Kicinski <kuba@kernel.org> wrote:
> > > > > > >
> > > > > > > On Tue, 14 Dec 2021 21:57:32 +0000 Lee Jones wrote:
> > > > > > > > The cause of the resultant dump_stack() reported below is a
> > > > > > > > dereference of a freed pointer to 'struct sctp_endpoint' in
> > > > > > > > sctp_sock_dump().
> > > > > > > >
> > > > > > > > This race condition occurs when a transport is cached into its
> > > > > > > > associated hash table followed by an endpoint/sock migration to a new
> > > > > > > > association in sctp_assoc_migrate() prior to their subsequent use in
> > > > > > > > sctp_diag_dump() which uses sctp_for_each_transport() to walk the hash
> > > > > > > > table calling into sctp_sock_dump() where the dereference occurs.
> > > > >
> > > > > > in sctp_sock_dump():
> > > > > > struct sock *sk = ep->base.sk;
> > > > > > ... <--[1]
> > > > > > lock_sock(sk);
> > > > > >
> > > > > > Do you mean in [1], the sk is peeled off and gets freed elsewhere?
> > > > >
> > > > > 'ep' and 'sk' are both switched out for new ones in sctp_sock_migrate().
> > > > >
> > > > > > if that's true, it's still late to do sock_hold(sk) in your this patch.
> > > > >
> > > > > No, that's not right.
> > > > >
> > > > > The schedule happens *inside* the lock_sock() call.
> > > > Sorry, I don't follow this.
> > > > We can't expect when the schedule happens, why do you think this
> > > > can never be scheduled before the lock_sock() call?
> > >
> > > True, but I've had this running for hours and it hasn't reproduced.
> I understand, but it's a crash, we shouldn't take any risk that it
> will never happen.
> you may try to add a usleep() before the lock_sock call to reproduce it.
>
> > >
> > > Without this patch, I can reproduce this in around 2 seconds.
> > >
> > > The C-repro for this is pretty intense!
> > >
> > > If you want to be *sure* that a schedule will never happen, we can
> > > take a reference directly with:
> > >
> > > ep = sctp_endpoint_hold(tsp->asoc->ep);
> > > sk = sock_hold(ep->base.sk);
> > >
> > > Which was my original plan before I soak tested this submitted patch
> > > for hours without any sign of reproducing the issue.
> we tried to not export sctp_obj_hold/put(), that's why we had
> sctp_for_each_transport().
>
> ep itself holds a reference of sk when it's alive, so it's weird to do
> these 2 together.
>
> > >
> > > > If the sock is peeled off or is being freed, we shouldn't dump this sock,
> > > > and it's better to skip it.
> > >
> > > I guess we can do that too.
> > >
> > > Are you suggesting sctp_sock_migrate() as the call site?
> diff --git a/net/sctp/socket.c b/net/sctp/socket.c
> index 85ac2e901ffc..56ea7a0e2add 100644
> --- a/net/sctp/socket.c
> +++ b/net/sctp/socket.c
> @@ -9868,6 +9868,7 @@ static int sctp_sock_migrate(struct sock *oldsk,
> struct sock *newsk,
> inet_sk_set_state(newsk, SCTP_SS_ESTABLISHED);
> }
>
> + sock_set_flag(oldsk, SOCK_RCU_FREE);
> release_sock(newsk);
>
> return 0;
>
> SOCK_RCU_FREE is set to the previous sk, so that this sk will not
> be freed between rcu_read_lock() and rcu_read_unlock().
>
> >
> > Also, when are you planning on testing the flag?
> SOCK_RCU_FREE flag is used when freeing sk in sk_destruct(),
> and if it's set, it will be freed in the next grace period of RCU.
>
> >
> > Won't that suffer with the same issue(s)?
> diff --git a/net/sctp/diag.c b/net/sctp/diag.c
> index 7970d786c4a2..b4c4acd9e67e 100644
> --- a/net/sctp/diag.c
> +++ b/net/sctp/diag.c
> @@ -309,16 +309,21 @@ static int sctp_tsp_dump_one(struct
> sctp_transport *tsp, void *p)
>
> static int sctp_sock_dump(struct sctp_transport *tsp, void *p)
> {
> - struct sctp_endpoint *ep = tsp->asoc->ep;
> struct sctp_comm_param *commp = p;
> - struct sock *sk = ep->base.sk;
> struct sk_buff *skb = commp->skb;
> struct netlink_callback *cb = commp->cb;
> const struct inet_diag_req_v2 *r = commp->r;
> struct sctp_association *assoc;
> + struct sctp_endpoint *ep;
> + struct sock *sk;
> int err = 0;
>
> + rcu_read_lock();
> + ep = tsp->asoc->ep;
> + sk = ep->base.sk;
> lock_sock(sk);
Unfortunately, this isn't going to work, as lock_sock() may sleep,
and is not allowed to be called understand rcu_read_lock() :(
> + if (tsp->asoc->ep != ep)
> + goto release;
> list_for_each_entry(assoc, &ep->asocs, asocs) {
> if (cb->args[4] < cb->args[1])
> goto next;
> @@ -358,6 +363,7 @@ static int sctp_sock_dump(struct sctp_transport
> *tsp, void *p)
> cb->args[4] = 0;
> release:
> release_sock(sk);
> + rcu_read_unlock();
> return err;
> }
>
> rcu_read_lock() will make sure sk from tsp->asoc->ep->base.sk will not
> be freed until rcu_read_unlock().
>
> That's all I have. Do you see any other way to fix this?
>
> Thanks.
>
> >
> > --
> > Lee Jones [李琼斯]
> > Senior Technical Lead - Developer Services
> > Linaro.org │ Open source software for Arm SoCs
> > Follow Linaro: Facebook | Twitter | Blog
next prev parent reply other threads:[~2021-12-16 18:22 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-12-14 21:57 [RESEND 1/2] sctp: export sctp_endpoint_{hold,put}() for use by seperate modules Lee Jones
2021-12-14 21:57 ` [RESEND 2/2] sctp: hold cached endpoints to prevent possible UAF Lee Jones
2021-12-16 1:48 ` Jakub Kicinski
2021-12-16 16:12 ` Xin Long
2021-12-16 16:39 ` Lee Jones
2021-12-16 16:52 ` Xin Long
2021-12-16 17:13 ` Lee Jones
2021-12-16 17:14 ` Lee Jones
2021-12-16 18:12 ` Xin Long
2021-12-16 18:22 ` Xin Long [this message]
2021-12-16 19:03 ` Lee Jones
2021-12-19 21:20 ` Xin Long
2021-12-20 8:56 ` Lee Jones
2021-12-21 19:52 ` Xin Long
2021-12-22 10:58 ` Lee Jones
2021-12-16 18:25 ` Lee Jones
2021-12-16 20:44 ` Jakub Kicinski
2021-12-17 11:21 ` Lee Jones
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CADvbK_cxMbYwkuN_ZUvHY-7ahc9ff+jbuPkKn6CA=yqMk=SKVw@mail.gmail.com' \
--to=lucien.xin@gmail.com \
--cc=daisyc@us.ibm.com \
--cc=davem@davemloft.net \
--cc=hui.huang@nokia.com \
--cc=jgrimm@us.ibm.com \
--cc=karl@athena.chicago.il.us \
--cc=kevin.gao@intel.com \
--cc=kuba@kernel.org \
--cc=lee.jones@linaro.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-sctp@vger.kernel.org \
--cc=marcelo.leitner@gmail.com \
--cc=netdev@vger.kernel.org \
--cc=nhorman@tuxdriver.com \
--cc=piggy@acm.org \
--cc=rmlayer@us.ibm.com \
--cc=sri@us.ibm.com \
--cc=vyasevich@gmail.com \
--cc=xingang.guo@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).