From: Ming Lei <ming.lei@redhat.com>
To: "yukuai (C)" <yukuai3@huawei.com>
Cc: axboe@kernel.dk, josef@toxicpanda.com, hch@infradead.org,
linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
nbd@other.debian.org, yi.zhang@huawei.com
Subject: Re: [PATCH v5 5/6] nbd: convert to use blk_mq_find_and_get_req()
Date: Tue, 14 Sep 2021 22:33:32 +0800 [thread overview]
Message-ID: <YUCyvDDG0gOzaFfR@T590> (raw)
In-Reply-To: <39e628cc-496c-ba20-b53a-fbeecc1d7e4e@huawei.com>
On Tue, Sep 14, 2021 at 05:08:00PM +0800, yukuai (C) wrote:
> On 2021/09/14 15:46, Ming Lei wrote:
> > On Tue, Sep 14, 2021 at 03:13:38PM +0800, yukuai (C) wrote:
> > > On 2021/09/14 14:44, Ming Lei wrote:
> > > > On Tue, Sep 14, 2021 at 11:11:06AM +0800, yukuai (C) wrote:
> > > > > On 2021/09/14 9:11, Ming Lei wrote:
> > > > > > On Thu, Sep 09, 2021 at 10:12:55PM +0800, Yu Kuai wrote:
> > > > > > > blk_mq_tag_to_rq() can only ensure to return valid request in
> > > > > > > following situation:
> > > > > > >
> > > > > > > 1) client send request message to server first
> > > > > > > submit_bio
> > > > > > > ...
> > > > > > > blk_mq_get_tag
> > > > > > > ...
> > > > > > > blk_mq_get_driver_tag
> > > > > > > ...
> > > > > > > nbd_queue_rq
> > > > > > > nbd_handle_cmd
> > > > > > > nbd_send_cmd
> > > > > > >
> > > > > > > 2) client receive respond message from server
> > > > > > > recv_work
> > > > > > > nbd_read_stat
> > > > > > > blk_mq_tag_to_rq
> > > > > > >
> > > > > > > If step 1) is missing, blk_mq_tag_to_rq() will return a stale
> > > > > > > request, which might be freed. Thus convert to use
> > > > > > > blk_mq_find_and_get_req() to make sure the returned request is not
> > > > > > > freed.
> > > > > >
> > > > > > But NBD_CMD_INFLIGHT has been added for checking if the reply is
> > > > > > expected, do we still need blk_mq_find_and_get_req() for covering
> > > > > > this issue? BTW, request and its payload is pre-allocated, so there
> > > > > > isn't real use-after-free.
> > > > >
> > > > > Hi, Ming
> > > > >
> > > > > Checking NBD_CMD_INFLIGHT relied on the request founded by tag is valid,
> > > > > not the other way round.
> > > > >
> > > > > nbd_read_stat
> > > > > req = blk_mq_tag_to_rq()
> > > > > cmd = blk_mq_rq_to_pdu(req)
> > > > > mutex_lock(cmd->lock)
> > > > > checking NBD_CMD_INFLIGHT
> > > >
> > > > Request and its payload is pre-allocated, and either req->ref or cmd->lock can
> > > > serve the same purpose here. Once cmd->lock is held, you can check if the cmd is
> > > > inflight or not. If it isn't inflight, just return -ENOENT. Is there any
> > > > problem to handle in this way?
> > >
> > > Hi, Ming
> > >
> > > in nbd_read_stat:
> > >
> > > 1) get a request by tag first
> > > 2) get nbd_cmd by the request
> > > 3) hold cmd->lock and check if cmd is inflight
> > >
> > > If we want to check if the cmd is inflight in step 3), we have to do
> > > setp 1) and 2) first. As I explained in patch 0, blk_mq_tag_to_rq()
> > > can't make sure the returned request is not freed:
> > >
> > > nbd_read_stat
> > > blk_mq_sched_free_requests
> > > blk_mq_free_rqs
> > > blk_mq_tag_to_rq
> > > -> get rq before clear mapping
> > > blk_mq_clear_rq_mapping
> > > __free_pages -> rq is freed
> > > blk_mq_request_started -> UAF
> >
> > If the above can happen, blk_mq_find_and_get_req() may not fix it too, just
>
> Hi, Ming
>
> Why can't blk_mq_find_and_get_req() fix it? I can't think of any
> scenario that might have problem currently.
The principle behind blk_mq_find_and_get_req() is that if one request's
ref is grabbed, the queue's usage counter is guaranteed to be grabbed,
and this way isn't straight-forward.
Yeah, it can fix the issue, but I don't think it is good to call it in
fast path cause tags->lock is required.
>
> > wondering why not take the following simpler way for avoiding the UAF?
> >
> > diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
> > index 5170a630778d..dfa5cce71f66 100644
> > --- a/drivers/block/nbd.c
> > +++ b/drivers/block/nbd.c
> > @@ -795,9 +795,13 @@ static void recv_work(struct work_struct *work)
> > work);
> > struct nbd_device *nbd = args->nbd;
> > struct nbd_config *config = nbd->config;
> > + struct request_queue *q = nbd->disk->queue;
> > struct nbd_cmd *cmd;
> > struct request *rq;
> > + if (!percpu_ref_tryget(&q->q_usage_counter))
> > + return;
> > +
>
> We can't make sure freeze_queue is called before this, thus this approch
> can't fix the problem, right?
> nbd_read_stat
> blk_mq_tag_to_rq
> elevator_switch
> blk_mq_freeze_queue(q);
> elevator_switch_mq
> elevator_exit
> blk_mq_sched_free_requests
> blk_mq_request_started -> UAF
No, blk_mq_freeze_queue() waits until .q_usage_counter becomes zero, so
there won't be any concurrent nbd_read_stat() during switching elevator
if ->q_usage_counter is grabbed in recv_work().
Thanks,
Ming
next prev parent reply other threads:[~2021-09-14 14:33 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-09 14:12 [PATCH v5 0/6] handle unexpected message from server Yu Kuai
2021-09-09 14:12 ` [PATCH v5 1/6] nbd: don't handle response without a corresponding request message Yu Kuai
2021-09-14 0:54 ` Ming Lei
2021-09-09 14:12 ` [PATCH v5 2/6] nbd: make sure request completion won't concurrent Yu Kuai
2021-09-14 0:57 ` Ming Lei
2021-09-14 3:11 ` yukuai (C)
2021-09-09 14:12 ` [PATCH v5 3/6] nbd: check sock index in nbd_read_stat() Yu Kuai
2021-09-09 14:12 ` [PATCH v5 4/6] blk-mq: export two symbols to get request by tag Yu Kuai
2021-09-09 14:12 ` [PATCH v5 5/6] nbd: convert to use blk_mq_find_and_get_req() Yu Kuai
2021-09-14 1:11 ` Ming Lei
2021-09-14 3:11 ` yukuai (C)
2021-09-14 6:44 ` Ming Lei
2021-09-14 7:13 ` yukuai (C)
2021-09-14 7:46 ` Ming Lei
2021-09-14 9:08 ` yukuai (C)
2021-09-14 9:12 ` yukuai (C)
2021-09-14 14:33 ` Ming Lei [this message]
2021-09-14 9:19 ` yukuai (C)
2021-09-14 14:37 ` Ming Lei
2021-09-15 1:54 ` yukuai (C)
2021-09-15 3:16 ` Ming Lei
2021-09-15 3:36 ` yukuai (C)
2021-09-15 3:46 ` Ming Lei
2021-09-09 14:12 ` [PATCH v5 6/6] nbd: don't start request if nbd_queue_rq() failed Yu Kuai
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YUCyvDDG0gOzaFfR@T590 \
--to=ming.lei@redhat.com \
--cc=axboe@kernel.dk \
--cc=hch@infradead.org \
--cc=josef@toxicpanda.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=nbd@other.debian.org \
--cc=yi.zhang@huawei.com \
--cc=yukuai3@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).