Re: [PATCH v2] nvme: align io queue count with allocted nvme_queue in nvme_probe

[Weiping Zhang <zwp10758@gmail.com>]

On Thu, Apr 23, 2020 at 6:25 PM Max Gurtovoy <maxg@mellanox.com> wrote:
>
>
> On 4/23/2020 10:59 AM, Weiping Zhang wrote:
> > Since the commit 147b27e4bd0 "nvme-pci: allocate device queues storage space at probe"
> > nvme_alloc_queue will not alloc struct nvme_queue any more.
> > If user change write/poll_queues to larger than the number of
> > allocated queue in nvme_probe, nvme_alloc_queue will touch
> > the memory out of boundary.
> >
> > This patch add nr_allocated_queues for struct nvme_dev to record how
> > many queues alloctaed in nvme_probe, then nvme driver will not use
> > more queues than nr_allocated_queues when user update queue count
> > and do a controller reset.
> >
> > Since global module parameter can be changed at rumtime, so it's not
> > safe to use these two parameter directly in the following functions:
> > nvme_dbbuf_dma_alloc
> > nvme_dbbuf_dma_free
> > nvme_calc_irq_sets
> > nvme_setup_io_queues
> >
> > This patch also add nr_write_queues, nr_poll_queues for
> > struct nvme_dev and io_queues_reload for struct nvme_ctrl, that allow
> > per-controller reload module parmater when reset controller. The nvme
> > driver will not reload module parameter(write/poll_queues) by default
> > when reset controller. If user want to reload them, they should enable
> > it by echo 1 > /sys/block/<nvme_disk>/device/io_queues_reload.
> >
> > By now, nvme pci driver allow user change io queue count for each
> > type(write, read, poll) within nr_allocated_queue, that's to say, if
> > user want to change queue dynamically by reset controller, they should
> > setup io queues as many as possiable when laod nvme module, and then
> > tune io queue count for each type.
>
> typo: laod --> load
>
OK, fix it in V3.
>
> > Signed-off-by: Weiping Zhang <zhangweiping@didiglobal.com>
> > ---
> > Changes since V1:
> >   * don't use module parameter nvme_dbbuf_dma_free, nvme_dbbuf_dma_alloc
> >       and nvme_calc_irq_sets.
> >   * add per-controller sysfs file io_queues_reload to enable/disable
> >       reload global module parameter.
> >
> >   drivers/nvme/host/core.c | 29 +++++++++++++++++++++
> >   drivers/nvme/host/nvme.h |  1 +
> >   drivers/nvme/host/pci.c  | 55 +++++++++++++++++++++++-----------------
> >   3 files changed, 62 insertions(+), 23 deletions(-)
> >
> > diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> > index dfb064b4334f..80172192a9d8 100644
> > --- a/drivers/nvme/host/core.c
> > +++ b/drivers/nvme/host/core.c
> > @@ -3357,6 +3357,34 @@ static ssize_t nvme_sysfs_show_address(struct device *dev,
> >   }
> >   static DEVICE_ATTR(address, S_IRUGO, nvme_sysfs_show_address, NULL);
> >
> > +static ssize_t nvme_sysfs_io_queues_reload_show(struct device *dev,
> > +                                      struct device_attribute *attr,
> > +                                      char *buf)
> > +{
> > +     struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
> > +
> > +     return snprintf(buf, PAGE_SIZE, "%d\n",
> > +             ctrl->io_queues_reload ? 1 : 0);
> > +}
> > +
> > +static ssize_t nvme_sysfs_io_queues_reload_store(struct device *dev,
> > +                             struct device_attribute *attr, const char *buf,
> > +                             size_t count)
> > +{
> > +     struct nvme_ctrl *ctrl = dev_get_drvdata(dev);
> > +     bool val;
> > +
> > +     if (kstrtobool(buf, &val))
> > +             return -EINVAL;
> > +     ctrl->io_queues_reload = val;
> > +
> > +     return count;
> > +}
> > +
> > +static DEVICE_ATTR(io_queues_reload, S_IRUGO | S_IWUSR,
> > +             nvme_sysfs_io_queues_reload_show,
> > +             nvme_sysfs_io_queues_reload_store);
> > +
> >   static struct attribute *nvme_dev_attrs[] = {
> >       &dev_attr_reset_controller.attr,
> >       &dev_attr_rescan_controller.attr,
> > @@ -3374,6 +3402,7 @@ static struct attribute *nvme_dev_attrs[] = {
> >       &dev_attr_sqsize.attr,
> >       &dev_attr_hostnqn.attr,
> >       &dev_attr_hostid.attr,
> > +     &dev_attr_io_queues_reload.attr,
> >       NULL
> >   };
>
> Well for fabrics controllers it doesn't mean anything.
>
> maybe we can do it non-visible for fabrics ?
>

Make sense, fix in v3.

Thanks

> _______________________________________________
> linux-nvme mailing list
> linux-nvme@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme