From: Dexuan Cui <decui@microsoft.com>
To: kimbrownkd <kimbrownkd@gmail.com>
Cc: Michael Kelley <mikelley@microsoft.com>,
Long Li <longli@microsoft.com>,
Sasha Levin <Alexander.Levin@microsoft.com>,
Stephen Hemminger <sthemmin@microsoft.com>,
KY Srinivasan <kys@microsoft.com>,
Haiyang Zhang <haiyangz@microsoft.com>,
"devel@linuxdriverproject.org" <devel@linuxdriverproject.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH] Drivers: hv: vmbus: Add mutex lock to channel show functions
Date: Tue, 22 Jan 2019 18:40:30 +0000 [thread overview]
Message-ID: <PU1P153MB01696D28CDDC098307EBD5AFBF980@PU1P153MB0169.APCP153.PROD.OUTLOOK.COM> (raw)
In-Reply-To: <20190122064246.GA28613@ubu-Virtual-Machine>
> From: Kimberly Brown <kimbrownkd@gmail.com>
> Sent: Monday, January 21, 2019 10:43 PM
> > > @@ -1421,7 +1422,10 @@ static ssize_t vmbus_chan_attr_show(struct
> > > kobject *kobj,
> > > if (chan->state != CHANNEL_OPENED_STATE)
> > > return -EINVAL;
> > >
> > > - return attribute->show(chan, buf);
> > > + mutex_lock(&vmbus_connection.channel_mutex);
> > > + ret = attribute->show(chan, buf);
> > > + mutex_unlock(&vmbus_connection.channel_mutex);
> > > + return ret;
> > > }
> >
> > It looks this patch is already able to fix the original issue:
> > 6712cc9c2211 ("vmbus: don't return values for uninitalized channels"),
> > as chan->state can't be CHANNEL_OPENED_STATE when
> > channel->ringbuffer_page is not set up yet, or has been freed.
> > -- Dexuan
>
> I think that patch 6712cc9c2211 fixes the problem when the channel is
> not set up yet, but it doesn't fix the problem when the channel is being
> closed
Yeah, now I see your point.
> The channel could be freed after the check that "chan->state" is
> CHANNEL_OPENED_STATE, while the "attribute->show()" function is running.
IMO the channel struct itself can't be freed while the "attribute->show()"
function is running, because I suppose the sysfs interface should have an extra
reference to channel->kobj (see vmbus_add_channel_kobj()), so before the sysfs
files are removed, the channel struct itself can't disappear.
(I didn't check the related code very carefully, so I could be wrong. :-) )
But as you pointed, at least for sub-channels, channel->ringbuffer_page
can indeed disappear in vmbus_close() -> ... -> vmbus_free_ring(), and
the "attribute->show()" could crash when the race happens.
Adding channel_mutex here seems to be able to fix the race for
sub-channels, as the same channel_mutex is used in vmbus_disconnect_ring().
For a primary channel, vmbus_close() -> vmbus_free_ring() can still
free channel->ringbuffer_page without notifying the "attribute->show()".
We may also need to acquire/release the channel_mutex in vmbus_close()?
> Actually, there should be checks that "chan" is not null and that
> "chan->state" is CHANNEL_OPENED_STATE within the locked section. I'll
> need to fix that.
I suppose "chan" can not be NULL here (see the above).
Checking "chan->state" may not help to completely fix the race, because
there is no locking/synchronization code in
vmbus_close_internal() when we test and change "channel->state".
I guess we may need to check if channel->ringbuffer_page is NULL in
the "attribute->show()".
PS, to prove that a race condition does exist and can really cause a panic or
something, I usually add some msleep() delays in different paths so that I
can reproduce the crash every time I take a special action, e.g. when I read
the sysfs files of a NIC, I try to remove hv_netvsc driver. This way, I can prove
a patch can indeed help, at least it can fix the crash which would happen
without the patch. :-)
-- Dexuan
next prev parent reply other threads:[~2019-01-22 18:41 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-01-22 2:07 [PATCH] Drivers: hv: vmbus: Add mutex lock to channel show functions Kimberly Brown
2019-01-22 3:46 ` Dexuan Cui
2019-01-22 6:42 ` Kimberly Brown
2019-01-22 18:40 ` Dexuan Cui [this message]
2019-01-28 19:58 ` Kimberly Brown
2019-01-29 19:20 ` Dexuan Cui
2019-01-31 15:19 ` Sasha Levin
2019-01-31 16:45 ` Michael Kelley
2019-01-31 17:47 ` Kimberly Brown
2019-02-01 14:13 ` Sasha Levin
2019-02-01 18:24 ` Dexuan Cui
2019-02-02 20:07 ` Kimberly Brown
2019-02-15 1:54 ` Sasha Levin
2019-02-15 2:27 ` Kimberly Brown
2019-02-22 3:46 ` [PATCH v2 0/2] Fix a race condition vulnerability in "_show" functions Kimberly Brown
2019-02-22 3:47 ` [PATCH v2 1/2] Drivers: hv: vmbus: Refactor chan->state if statement Kimberly Brown
2019-02-24 16:54 ` Michael Kelley
2019-02-22 3:47 ` [PATCH v2 2/2] Drivers: hv: vmbus: Add a channel ring buffer mutex lock Kimberly Brown
2019-02-24 16:53 ` Michael Kelley
2019-02-26 6:24 ` Kimberly Brown
2019-03-14 20:04 ` [PATCH v3 0/3] Drivers: hv: vmbus: Fix a race condition in "_show" functions Kimberly Brown
2019-03-14 20:05 ` [PATCH v3 1/3] Drivers: hv: vmbus: Refactor chan->state if statement Kimberly Brown
2019-03-14 20:05 ` [PATCH v3 2/3] Drivers: hv: vmbus: Set ring_info field to 0 and remove memset Kimberly Brown
2019-03-29 16:01 ` Michael Kelley
2019-03-14 20:05 ` [PATCH v3 3/3] Drivers: hv: vmbus: Fix race condition with new ring_buffer_info mutex Kimberly Brown
2019-03-14 22:45 ` Stephen Hemminger
2019-03-17 1:49 ` Kimberly Brown
2019-03-20 20:06 ` Stephen Hemminger
2019-03-21 3:47 ` Kimberly Brown
2019-03-21 16:04 ` Michael Kelley
2019-03-28 4:30 ` Kimberly Brown
2019-03-28 18:42 ` Stephen Hemminger
2019-03-29 16:04 ` Michael Kelley
2019-04-10 22:59 ` [PATCH v3 0/3] Drivers: hv: vmbus: Fix a race condition in "_show" functions Sasha Levin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=PU1P153MB01696D28CDDC098307EBD5AFBF980@PU1P153MB0169.APCP153.PROD.OUTLOOK.COM \
--to=decui@microsoft.com \
--cc=Alexander.Levin@microsoft.com \
--cc=devel@linuxdriverproject.org \
--cc=haiyangz@microsoft.com \
--cc=kimbrownkd@gmail.com \
--cc=kys@microsoft.com \
--cc=linux-kernel@vger.kernel.org \
--cc=longli@microsoft.com \
--cc=mikelley@microsoft.com \
--cc=sthemmin@microsoft.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).