[PATCH] nvme-pci: avoid dereference of symbol from unloaded module

[PATCH] nvme-pci: avoid dereference of symbol from unloaded module

[Ming Lei]

The 'remove_work' may be scheduled to run after nvme_remove()
returns since we can't simply cancel it in nvme_remove() for
avoiding deadlock. Once nvme_remove() returns, this module(nvme)
can be unloaded.

On the other hand, nvme_put_ctrl() calls ctr->ops->free_ctrl
which may point to nvme_pci_free_ctrl() in unloaded module.

This patch avoids this issue by holding the module refcount before
scheduling 'remove_work'.

Signed-off-by: Ming Lei <ming.lei at redhat.com>
---
 drivers/nvme/host/pci.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 3f5a04c586ce..3ed8fd7218d4 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2134,8 +2134,12 @@ static void nvme_remove_dead_ctrl(struct nvme_dev *dev, int status)
 
 	kref_get(&dev->ctrl.kref);
 	nvme_dev_disable(dev, false);
-	if (!schedule_work(&dev->remove_work))
+
+	__module_get(THIS_MODULE);
+	if (!schedule_work(&dev->remove_work)) {
 		nvme_put_ctrl(&dev->ctrl);
+		module_put(THIS_MODULE);
+	}
 }
 
 static void nvme_reset_work(struct work_struct *work)
@@ -2231,10 +2235,20 @@ static void nvme_remove_dead_ctrl_work(struct work_struct *work)
 	struct nvme_dev *dev = container_of(work, struct nvme_dev, remove_work);
 	struct pci_dev *pdev = to_pci_dev(dev->dev);
 
+	/*
+	 * Inside nvme_remove(), we can't simply cancel this 'remove_work'
+	 * for avoiding deadlock, so this work function may be run after
+	 * nvme_remove() returns, and this module may have been removed
+	 * at that time. We have to get the module refcount before scheduling
+	 * 'remove_work', otherwise nvme_put_ctrl() may deference symbols
+	 * of the unloaded module via ctrl->ops.
+	 */
+
 	nvme_kill_queues(&dev->ctrl);
 	if (pci_get_drvdata(pdev))
 		device_release_driver(&pdev->dev);
 	nvme_put_ctrl(&dev->ctrl);
+	module_put(THIS_MODULE);
 }
 
 static int nvme_pci_reg_read32(struct nvme_ctrl *ctrl, u32 off, u32 *val)
-- 
2.9.5

[PATCH] nvme-pci: avoid dereference of symbol from unloaded module

[Sagi Grimberg]

> The 'remove_work' may be scheduled to run after nvme_remove()
> returns since we can't simply cancel it in nvme_remove() for
> avoiding deadlock. Once nvme_remove() returns, this module(nvme)
> can be unloaded.
> 
> On the other hand, nvme_put_ctrl() calls ctr->ops->free_ctrl
> which may point to nvme_pci_free_ctrl() in unloaded module.
> 
> This patch avoids this issue by holding the module refcount before
> scheduling 'remove_work'.

Why not simply flushing all scheduled works in module exit?

[PATCH] nvme-pci: avoid dereference of symbol from unloaded module

[Ming Lei]

On Thu, Nov 02, 2017@02:10:16PM +0200, Sagi Grimberg wrote:
> 
> > The 'remove_work' may be scheduled to run after nvme_remove()
> > returns since we can't simply cancel it in nvme_remove() for
> > avoiding deadlock. Once nvme_remove() returns, this module(nvme)
> > can be unloaded.
> > 
> > On the other hand, nvme_put_ctrl() calls ctr->ops->free_ctrl
> > which may point to nvme_pci_free_ctrl() in unloaded module.
> > 
> > This patch avoids this issue by holding the module refcount before
> > scheduling 'remove_work'.
> 
> Why not simply flushing all scheduled works in module exit?

The nvme_wq is shared by all kinds of host(PCI, FC, ...), so
not good to flush all queued work in 'nvme_wq' just before
removing pci host module only.

Not mention 'remove_work' is actually scheduled by system wq
instead of nvme_wq.

-- 
Ming

[PATCH] nvme-pci: avoid dereference of symbol from unloaded module

[Christoph Hellwig]

On Thu, Nov 02, 2017@08:29:27PM +0800, Ming Lei wrote:
> On Thu, Nov 02, 2017@02:10:16PM +0200, Sagi Grimberg wrote:
> > 
> > > The 'remove_work' may be scheduled to run after nvme_remove()
> > > returns since we can't simply cancel it in nvme_remove() for
> > > avoiding deadlock. Once nvme_remove() returns, this module(nvme)
> > > can be unloaded.
> > > 
> > > On the other hand, nvme_put_ctrl() calls ctr->ops->free_ctrl
> > > which may point to nvme_pci_free_ctrl() in unloaded module.
> > > 
> > > This patch avoids this issue by holding the module refcount before
> > > scheduling 'remove_work'.
> > 
> > Why not simply flushing all scheduled works in module exit?
> 
> The nvme_wq is shared by all kinds of host(PCI, FC, ...), so
> not good to flush all queued work in 'nvme_wq' just before
> removing pci host module only.
> 
> Not mention 'remove_work' is actually scheduled by system wq
> instead of nvme_wq.

Still seems better than messing with module reference counts.

[PATCH] nvme-pci: avoid dereference of symbol from unloaded module

[Ming Lei]

On Wed, Nov 08, 2017@10:15:31AM -0800, Christoph Hellwig wrote:
> On Thu, Nov 02, 2017@08:29:27PM +0800, Ming Lei wrote:
> > On Thu, Nov 02, 2017@02:10:16PM +0200, Sagi Grimberg wrote:
> > > 
> > > > The 'remove_work' may be scheduled to run after nvme_remove()
> > > > returns since we can't simply cancel it in nvme_remove() for
> > > > avoiding deadlock. Once nvme_remove() returns, this module(nvme)
> > > > can be unloaded.
> > > > 
> > > > On the other hand, nvme_put_ctrl() calls ctr->ops->free_ctrl
> > > > which may point to nvme_pci_free_ctrl() in unloaded module.
> > > > 
> > > > This patch avoids this issue by holding the module refcount before
> > > > scheduling 'remove_work'.
> > > 
> > > Why not simply flushing all scheduled works in module exit?
> > 
> > The nvme_wq is shared by all kinds of host(PCI, FC, ...), so
> > not good to flush all queued work in 'nvme_wq' just before
> > removing pci host module only.
> > 
> > Not mention 'remove_work' is actually scheduled by system wq
> > instead of nvme_wq.
> 
> Still seems better than messing with module reference counts.

If 'remove_work' can be queued via 'nvme_wq', I think it is doable to
call flush_workqueue(nvme_wq) in nvme_exit(): drivers/nvme/host/pci.c.

Otherwise, it may not be a good idea to flush the global wq, please
see comment of 'flush_scheduled_work()'.

-- 
Ming

[PATCH] nvme-pci: avoid dereference of symbol from unloaded module

[Sagi Grimberg]

>> The nvme_wq is shared by all kinds of host(PCI, FC, ...), so
>> not good to flush all queued work in 'nvme_wq' just before
>> removing pci host module only.
>>
>> Not mention 'remove_work' is actually scheduled by system wq
>> instead of nvme_wq.
> 
> Still seems better than messing with module reference counts.

And also does not change the behavior (failing module unload).

[PATCH] nvme-pci: avoid dereference of symbol from unloaded module

[Sagi Grimberg]

> If 'remove_work' can be queued via 'nvme_wq', I think it is doable to
> call flush_workqueue(nvme_wq) in nvme_exit(): drivers/nvme/host/pci.c.

Lets do that.