From: Leon Romanovsky <leon@kernel.org>
To: Divya Indi <divya.indi@oracle.com>
Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org,
"Jason Gunthorpe" <jgg@ziepe.ca>,
"Kaike Wan" <kaike.wan@intel.com>,
"Gerd Rausch" <gerd.rausch@oracle.com>,
"Håkon Bugge" <haakon.bugge@oracle.com>,
"Srinivas Eeda" <srinivas.eeda@oracle.com>,
"Rama Nichanamatlu" <rama.nichanamatlu@oracle.com>,
"Doug Ledford" <dledford@redhat.com>
Subject: Re: [PATCH v3] IB/sa: Resolving use-after-free in ib_nl_send_msg
Date: Tue, 9 Jun 2020 10:00:26 +0300 [thread overview]
Message-ID: <20200609070026.GJ164174@unreal> (raw)
In-Reply-To: <1591627576-920-2-git-send-email-divya.indi@oracle.com>
On Mon, Jun 08, 2020 at 07:46:16AM -0700, Divya Indi wrote:
> Commit 3ebd2fd0d011 ("IB/sa: Put netlink request into the request list before sending")'
> -
> 1. Adds the query to the request list before ib_nl_snd_msg.
> 2. Removes ib_nl_send_msg from within the spinlock which also makes it
> possible to allocate memory with GFP_KERNEL.
>
> However, if there is a delay in sending out the request (For
> eg: Delay due to low memory situation) the timer to handle request timeout
> might kick in before the request is sent out to ibacm via netlink.
> ib_nl_request_timeout may release the query causing a use after free situation
> while accessing the query in ib_nl_send_msg.
>
> Call Trace for the above race:
>
> [<ffffffffa02f43cb>] ? ib_pack+0x17b/0x240 [ib_core]
> [<ffffffffa032aef1>] ib_sa_path_rec_get+0x181/0x200 [ib_sa]
> [<ffffffffa0379db0>] rdma_resolve_route+0x3c0/0x8d0 [rdma_cm]
> [<ffffffffa0374450>] ? cma_bind_port+0xa0/0xa0 [rdma_cm]
> [<ffffffffa040f850>] ? rds_rdma_cm_event_handler_cmn+0x850/0x850
> [rds_rdma]
> [<ffffffffa040f22c>] rds_rdma_cm_event_handler_cmn+0x22c/0x850
> [rds_rdma]
> [<ffffffffa040f860>] rds_rdma_cm_event_handler+0x10/0x20 [rds_rdma]
> [<ffffffffa037778e>] addr_handler+0x9e/0x140 [rdma_cm]
> [<ffffffffa026cdb4>] process_req+0x134/0x190 [ib_addr]
> [<ffffffff810a02f9>] process_one_work+0x169/0x4a0
> [<ffffffff810a0b2b>] worker_thread+0x5b/0x560
> [<ffffffff810a0ad0>] ? flush_delayed_work+0x50/0x50
> [<ffffffff810a68fb>] kthread+0xcb/0xf0
> [<ffffffff816ec49a>] ? __schedule+0x24a/0x810
> [<ffffffff816ec49a>] ? __schedule+0x24a/0x810
> [<ffffffff810a6830>] ? kthread_create_on_node+0x180/0x180
> [<ffffffff816f25a7>] ret_from_fork+0x47/0x90
> [<ffffffff810a6830>] ? kthread_create_on_node+0x180/0x180
> ....
> RIP [<ffffffffa03296cd>] send_mad+0x33d/0x5d0 [ib_sa]
>
> To resolve the above issue -
> 1. Add the req to the request list only after the request has been sent out.
> 2. To handle the race where response comes in before adding request to
> the request list, send(rdma_nl_multicast) and add to list while holding the
> spinlock - request_lock.
> 3. Use GFP_NOWAIT for rdma_nl_multicast since it is called while holding
> a spinlock. In case of memory allocation failure, request will go out to SA.
>
> Signed-off-by: Divya Indi <divya.indi@oracle.com>
> Fixes: 3ebd2fd0d011 ("IB/sa: Put netlink request into the request list
> before sending")
Author SOB should be after "Fixes" line.
> ---
> drivers/infiniband/core/sa_query.c | 34 +++++++++++++++++-----------------
> 1 file changed, 17 insertions(+), 17 deletions(-)
>
> diff --git a/drivers/infiniband/core/sa_query.c b/drivers/infiniband/core/sa_query.c
> index 74e0058..042c99b 100644
> --- a/drivers/infiniband/core/sa_query.c
> +++ b/drivers/infiniband/core/sa_query.c
> @@ -836,6 +836,9 @@ static int ib_nl_send_msg(struct ib_sa_query *query, gfp_t gfp_mask)
> void *data;
> struct ib_sa_mad *mad;
> int len;
> + unsigned long flags;
> + unsigned long delay;
> + int ret;
>
> mad = query->mad_buf->mad;
> len = ib_nl_get_path_rec_attrs_len(mad->sa_hdr.comp_mask);
> @@ -860,35 +863,32 @@ static int ib_nl_send_msg(struct ib_sa_query *query, gfp_t gfp_mask)
> /* Repair the nlmsg header length */
> nlmsg_end(skb, nlh);
>
> - return rdma_nl_multicast(&init_net, skb, RDMA_NL_GROUP_LS, gfp_mask);
> + spin_lock_irqsave(&ib_nl_request_lock, flags);
> + ret = rdma_nl_multicast(&init_net, skb, RDMA_NL_GROUP_LS, GFP_NOWAIT);
It is hard to be convinced that this is correct solution. The mix of
gfp_flags and GFP_NOWAIT at the same time and usage of
ib_nl_request_lock to protect lists and suddenly rdma_nl_multicast() too
makes this code unreadable/non-maintainable.
> + if (!ret) {
Please use kernel coding style.
if (ret) {
spin_unlock_irqrestore(&ib_nl_request_lock, flags);
return ret;
}
....
> + /* Put the request on the list.*/
> + delay = msecs_to_jiffies(sa_local_svc_timeout_ms);
> + query->timeout = delay + jiffies;
> + list_add_tail(&query->list, &ib_nl_request_list);
> + /* Start the timeout if this is the only request */
> + if (ib_nl_request_list.next == &query->list)
> + queue_delayed_work(ib_nl_wq, &ib_nl_timed_work, delay);
> + }
> + spin_unlock_irqrestore(&ib_nl_request_lock, flags);
> +
> + return ret;
> }
>
> static int ib_nl_make_request(struct ib_sa_query *query, gfp_t gfp_mask)
> {
> - unsigned long flags;
> - unsigned long delay;
> int ret;
>
> INIT_LIST_HEAD(&query->list);
> query->seq = (u32)atomic_inc_return(&ib_nl_sa_request_seq);
>
> - /* Put the request on the list first.*/
> - spin_lock_irqsave(&ib_nl_request_lock, flags);
> - delay = msecs_to_jiffies(sa_local_svc_timeout_ms);
> - query->timeout = delay + jiffies;
> - list_add_tail(&query->list, &ib_nl_request_list);
> - /* Start the timeout if this is the only request */
> - if (ib_nl_request_list.next == &query->list)
> - queue_delayed_work(ib_nl_wq, &ib_nl_timed_work, delay);
> - spin_unlock_irqrestore(&ib_nl_request_lock, flags);
> -
> ret = ib_nl_send_msg(query, gfp_mask);
> if (ret) {
> ret = -EIO;
> - /* Remove the request */
> - spin_lock_irqsave(&ib_nl_request_lock, flags);
> - list_del(&query->list);
> - spin_unlock_irqrestore(&ib_nl_request_lock, flags);
> }
Brackets should be removed too.
>
> return ret;
> --
> 1.8.3.1
>
next prev parent reply other threads:[~2020-06-09 7:00 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-06-08 14:46 Review Request Divya Indi
2020-06-08 14:46 ` [PATCH v3] IB/sa: Resolving use-after-free in ib_nl_send_msg Divya Indi
2020-06-09 7:00 ` Leon Romanovsky [this message]
2020-06-09 14:45 ` Divya Indi
2020-06-14 6:41 ` Leon Romanovsky
2020-06-16 17:56 ` Divya Indi
2020-06-17 5:17 ` Leon Romanovsky
2020-06-17 18:23 ` Jason Gunthorpe
2020-06-21 7:12 ` Leon Romanovsky
2020-06-17 18:24 ` Jason Gunthorpe
2020-06-20 0:43 ` Divya Indi
2020-06-09 7:03 ` Review Request Leon Romanovsky
2020-06-09 15:44 ` Divya Indi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200609070026.GJ164174@unreal \
--to=leon@kernel.org \
--cc=divya.indi@oracle.com \
--cc=dledford@redhat.com \
--cc=gerd.rausch@oracle.com \
--cc=haakon.bugge@oracle.com \
--cc=jgg@ziepe.ca \
--cc=kaike.wan@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=rama.nichanamatlu@oracle.com \
--cc=srinivas.eeda@oracle.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).