Linux-HyperV Archive on lore.kernel.org
 help / color / Atom feed
From: Michael Kelley <mikelley@microsoft.com>
To: Long Li <longli@microsoft.com>,
	"longli@linuxonhyperv.com" <longli@linuxonhyperv.com>,
	KY Srinivasan <kys@microsoft.com>,
	Haiyang Zhang <haiyangz@microsoft.com>,
	Stephen Hemminger <sthemmin@microsoft.com>,
	Wei Liu <wei.liu@kernel.org>,
	Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
	Rob Herring <robh@kernel.org>,
	Bjorn Helgaas <bhelgaas@google.com>,
	"linux-hyperv@vger.kernel.org" <linux-hyperv@vger.kernel.org>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	Dexuan Cui <decui@microsoft.com>
Subject: RE: [PATCH] PCI: hv: Fix a race condition when removing the device
Date: Wed, 21 Apr 2021 21:05:46 +0000
Message-ID: <MWHPR21MB15933F28861A2C448C3F233DD7479@MWHPR21MB1593.namprd21.prod.outlook.com> (raw)
In-Reply-To: <BYAPR21MB12711AF8B782FAA4B492D0CFCE479@BYAPR21MB1271.namprd21.prod.outlook.com>

From: Long Li <longli@microsoft.com> Sent: Wednesday, April 21, 2021 12:57 PM
> > From: longli@linuxonhyperv.com <longli@linuxonhyperv.com>  Sent:
> > Monday, April 19, 2021 12:21 PM
> > >
> > > On removing the device, any work item (hv_pci_devices_present() or
> > > hv_pci_eject_device()) scheduled on workqueue hbus->wq may still be
> > > running and race with hv_pci_remove().
> > >
> > > This can happen because the host may send PCI_EJECT or
> > > PCI_BUS_RELATIONS(2) and decide to rescind the channel immediately
> > after that.
> > >
> > > Fix this by flushing/stopping the workqueue of hbus before doing hbus
> > remove.
> >
> > I can see that this change follows the same pattern as in hv_pci_suspend().
> > The comments there give a full explanation of the issue and the solution.  But
> > interestingly, the current code also has a reference count mechanism on the
> > hbus.  And code near the end of hv_pci_remove() decrements the reference
> > count and then waits for all users to finish before destroying the workqueue.
> > With this change, is this reference counting mechanism still needed?   If the
> > workqueue has already been emptied, it seems like the
> > wait_for_completion() near the end of hv_pci_remove() would never be
> > waiting for anything.  It makes me wonder if moving the reference count
> > checking code from near the end of hv_pci_remove() up to near the beginning
> > would solve the problem as well (and maybe in hv_pci_suspend also?).
> 
> Yes I think put_hvpcibus() and get_hvpcibus() can be removed, as we have changed to use
> a dedicated workqueue for hbus since they were introduced.
> 
> But we still need to call tasklet_disable/enable() the same way hv_pci_suspend() does, the
> reason is that we need to protect hbus->state. This value needs to be consistent for the
> driver. For example, a CPU may decide to schedule a work on a work queue that we just
> flushed or destroyed, by reading the wrong hbus->state.
> 

Yes, I would agree the tasklet disable/enable are needed, especially since tasklet_disable()
is what ensures that the tasklet is not currently running.

If the hbus ref counting isn't needed any longer, I would strongly recommend adding
a patch to the series that removes it.  This synchronization stuff is hard enough to
understand and reason about; having a leftover mechanism that doesn't really do
anything useful makes it nearly impossible. :-)

Dexuan -- I'm hoping you can take a look as well and see if you agree.

Michael

> >
> > Michael
> >
> > >
> > > Signed-off-by: Long Li <longli@microsoft.com>
> > > ---
> > >  drivers/pci/controller/pci-hyperv.c | 11 +++++++++++
> > >  1 file changed, 11 insertions(+)
> > >
> > > diff --git a/drivers/pci/controller/pci-hyperv.c
> > > b/drivers/pci/controller/pci-hyperv.c
> > > index 27a17a1e4a7c..116815404313 100644
> > > --- a/drivers/pci/controller/pci-hyperv.c
> > > +++ b/drivers/pci/controller/pci-hyperv.c
> > > @@ -3305,6 +3305,17 @@ static int hv_pci_remove(struct hv_device
> > > *hdev)
> > >
> > >  	hbus = hv_get_drvdata(hdev);
> > >  	if (hbus->state == hv_pcibus_installed) {
> > > +		tasklet_disable(&hdev->channel->callback_event);
> > > +		hbus->state = hv_pcibus_removing;
> > > +		tasklet_enable(&hdev->channel->callback_event);
> > > +
> > > +		flush_workqueue(hbus->wq);
> > > +		/*
> > > +		 * At this point, no work is running or can be scheduled
> > > +		 * on hbus-wq. We can't race with hv_pci_devices_present()
> > > +		 * or hv_pci_eject_device(), it's safe to proceed.
> > > +		 */
> > > +
> > >  		/* Remove the bus from PCI's point of view. */
> > >  		pci_lock_rescan_remove();
> > >  		pci_stop_root_bus(hbus->pci_bus);
> > > --
> > > 2.27.0


  reply index

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-19 19:20 longli
2021-04-21 17:33 ` Michael Kelley
2021-04-21 19:56   ` Long Li
2021-04-21 21:05     ` Michael Kelley [this message]
2021-04-22  2:31       ` Dexuan Cui
2021-04-22  3:57         ` Long Li
2021-04-22  4:19           ` Long Li

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=MWHPR21MB15933F28861A2C448C3F233DD7479@MWHPR21MB1593.namprd21.prod.outlook.com \
    --to=mikelley@microsoft.com \
    --cc=bhelgaas@google.com \
    --cc=decui@microsoft.com \
    --cc=haiyangz@microsoft.com \
    --cc=kys@microsoft.com \
    --cc=linux-hyperv@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=longli@linuxonhyperv.com \
    --cc=longli@microsoft.com \
    --cc=lorenzo.pieralisi@arm.com \
    --cc=robh@kernel.org \
    --cc=sthemmin@microsoft.com \
    --cc=wei.liu@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-HyperV Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-hyperv/0 linux-hyperv/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-hyperv linux-hyperv/ https://lore.kernel.org/linux-hyperv \
		linux-hyperv@vger.kernel.org
	public-inbox-index linux-hyperv

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-hyperv


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git