From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_2 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB338C433E9 for ; Wed, 10 Feb 2021 18:30:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C43B964E7C for ; Wed, 10 Feb 2021 18:30:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233398AbhBJS3r (ORCPT ); Wed, 10 Feb 2021 13:29:47 -0500 Received: from frasgout.his.huawei.com ([185.176.79.56]:2539 "EHLO frasgout.his.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233065AbhBJSTO (ORCPT ); Wed, 10 Feb 2021 13:19:14 -0500 Received: from fraeml712-chm.china.huawei.com (unknown [172.18.147.207]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4DbSYk6bX5z67kqq; Thu, 11 Feb 2021 02:13:30 +0800 (CST) Received: from lhreml710-chm.china.huawei.com (10.201.108.61) by fraeml712-chm.china.huawei.com (10.206.15.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2106.2; Wed, 10 Feb 2021 19:18:26 +0100 Received: from localhost (10.47.67.2) by lhreml710-chm.china.huawei.com (10.201.108.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2106.2; Wed, 10 Feb 2021 18:18:25 +0000 Date: Wed, 10 Feb 2021 18:17:25 +0000 From: Jonathan Cameron To: Ben Widawsky CC: , , , , , Bjorn Helgaas , "Chris Browy" , Christoph Hellwig , "Dan Williams" , David Hildenbrand , David Rientjes , Ira Weiny , "Jon Masters" , Rafael Wysocki , Randy Dunlap , Vishal Verma , "John Groves (jgroves)" , "Kelley, Sean V" Subject: Re: [PATCH v2 3/8] cxl/mem: Register CXL memX devices Message-ID: <20210210181725.00007865@Huawei.com> In-Reply-To: <20210210000259.635748-4-ben.widawsky@intel.com> References: <20210210000259.635748-1-ben.widawsky@intel.com> <20210210000259.635748-4-ben.widawsky@intel.com> Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; i686-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.47.67.2] X-ClientProxiedBy: lhreml725-chm.china.huawei.com (10.201.108.76) To lhreml710-chm.china.huawei.com (10.201.108.61) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 9 Feb 2021 16:02:54 -0800 Ben Widawsky wrote: > From: Dan Williams > > Create the /sys/bus/cxl hierarchy to enumerate: > > * Memory Devices (per-endpoint control devices) > > * Memory Address Space Devices (platform address ranges with > interleaving, performance, and persistence attributes) > > * Memory Regions (active provisioned memory from an address space device > that is in use as System RAM or delegated to libnvdimm as Persistent > Memory regions). > > For now, only the per-endpoint control devices are registered on the > 'cxl' bus. However, going forward it will provide a mechanism to > coordinate cross-device interleave. > > Signed-off-by: Dan Williams > Signed-off-by: Ben Widawsky One stray header, and a request for a tiny bit of reordering to make it easier to chase through creation and destruction. Either way with the header move to earlier patch I'm fine with this one. Reviewed-by: Jonathan Cameron > --- > Documentation/ABI/testing/sysfs-bus-cxl | 26 ++ > .../driver-api/cxl/memory-devices.rst | 17 + > drivers/cxl/Makefile | 3 + > drivers/cxl/bus.c | 29 ++ > drivers/cxl/cxl.h | 4 + > drivers/cxl/mem.c | 301 +++++++++++++++++- > 6 files changed, 378 insertions(+), 2 deletions(-) > create mode 100644 Documentation/ABI/testing/sysfs-bus-cxl > create mode 100644 drivers/cxl/bus.c > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h > index 745f5e0bfce3..b3c56fa6e126 100644 > --- a/drivers/cxl/cxl.h > +++ b/drivers/cxl/cxl.h > @@ -3,6 +3,7 @@ > > #ifndef __CXL_H__ > #define __CXL_H__ > +#include Why is this coming in now? Feels like it should have been in earlier patch that started using struct range > > #include > #include > @@ -55,6 +56,7 @@ > (FIELD_GET(CXLMDEV_RESET_NEEDED_MASK, status) != \ > CXLMDEV_RESET_NEEDED_NOT) > > +struct cxl_memdev; > /** > * struct cxl_mem - A CXL memory device > * @pdev: The PCI device associated with this CXL device. > @@ -72,6 +74,7 @@ > struct cxl_mem { > struct pci_dev *pdev; > void __iomem *regs; > + struct cxl_memdev *cxlmd; > > void __iomem *status_regs; > void __iomem *mbox_regs; > @@ -90,4 +93,5 @@ struct cxl_mem { > } ram; > }; > > +extern struct bus_type cxl_bus_type; > #endif /* __CXL_H__ */ > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c > index 0a868a15badc..8bbd2495e237 100644 > --- a/drivers/cxl/mem.c > +++ b/drivers/cxl/mem.c > @@ -1,11 +1,36 @@ > > + > +static void cxl_memdev_release(struct device *dev) > +{ > + struct cxl_memdev *cxlmd = to_cxl_memdev(dev); > + > + percpu_ref_exit(&cxlmd->ops_active); > + ida_free(&cxl_memdev_ida, cxlmd->id); > + kfree(cxlmd); > +} > + ... > +static int cxl_mem_add_memdev(struct cxl_mem *cxlm) > +{ > + struct pci_dev *pdev = cxlm->pdev; > + struct cxl_memdev *cxlmd; > + struct device *dev; > + struct cdev *cdev; > + int rc; > + > + cxlmd = kzalloc(sizeof(*cxlmd), GFP_KERNEL); > + if (!cxlmd) > + return -ENOMEM; > + init_completion(&cxlmd->ops_dead); > + > + /* > + * @cxlm is deallocated when the driver unbinds so operations > + * that are using it need to hold a live reference. > + */ > + cxlmd->cxlm = cxlm; > + rc = percpu_ref_init(&cxlmd->ops_active, cxlmdev_ops_active_release, 0, > + GFP_KERNEL); > + if (rc) > + goto err_ref; > + > + rc = ida_alloc_range(&cxl_memdev_ida, 0, CXL_MEM_MAX_DEVS, GFP_KERNEL); > + if (rc < 0) > + goto err_id; > + cxlmd->id = rc; > + > + dev = &cxlmd->dev; > + device_initialize(dev); > + dev->parent = &pdev->dev; > + dev->bus = &cxl_bus_type; > + dev->devt = MKDEV(cxl_mem_major, cxlmd->id); > + dev->type = &cxl_memdev_type; > + dev_set_name(dev, "mem%d", cxlmd->id); > + > + cdev = &cxlmd->cdev; > + cdev_init(cdev, &cxl_memdev_fops); > + > + rc = cdev_device_add(cdev, dev); > + if (rc) > + goto err_add; > + > + return devm_add_action_or_reset(dev->parent, cxlmdev_unregister, cxlmd); This had me scratching my head. The cxlmdev_unregister() if called normally or in the _or_reset() results in percpu_ref_kill(&cxlmd->ops_active); cdev_device_del(&cxlmd->cdev, dev); wait_for_completion(&cxlmd->ops_dead); cxlmd->cxlm = NULL; put_device(dev); /* If last ref this will result in */ percpu_ref_exit(&cxlmd->ops_active); ida_free(&cxl_memdev_ida, cxlmd->id); kfree(cxlmd); So it's doing all the correct things but not necessarily in the obvious order. For simplicity of review perhaps it's worth reordering probe a bit to get the ida immediately after the cxlmd alloc and for the cxlmdev_unregister() perhaps reorder the cdev_device_del() before the percpu_ref_kill(). Trivial obvious as the ordering has no affect but makes it easy for reviewers to tick off setup vs tear down parts. > + > +err_add: > + ida_free(&cxl_memdev_ida, cxlmd->id); > +err_id: > + /* > + * Theoretically userspace could have already entered the fops, > + * so flush ops_active. > + */ > + percpu_ref_kill(&cxlmd->ops_active); > + wait_for_completion(&cxlmd->ops_dead); > + percpu_ref_exit(&cxlmd->ops_active); > +err_ref: > + kfree(cxlmd); > + > + return rc; > +} > +