From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1162311AbdAFA6W convert rfc822-to-8bit (ORCPT ); Thu, 5 Jan 2017 19:58:22 -0500 Received: from mga02.intel.com ([134.134.136.20]:64819 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753975AbdAFA6P (ORCPT ); Thu, 5 Jan 2017 19:58:15 -0500 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.33,323,1477983600"; d="scan'208";a="1079725010" From: "Tantilov, Emil S" To: Gavin Shan CC: "linux-pci@vger.kernel.org" , "intel-wired-lan@lists.osuosl.org" , "Duyck, Alexander H" , "netdev@vger.kernel.org" , "linux-kernel@vger.kernel.org" Subject: RE: [PATCH 2/2] PCI: lock each enable/disable num_vfs operation in sysfs Thread-Topic: [PATCH 2/2] PCI: lock each enable/disable num_vfs operation in sysfs Thread-Index: AQHSZjB5ZAYjBAVYMU+wnZEUoUh0JqEoeDSggAEBVYCAASGHwA== Date: Fri, 6 Jan 2017 00:55:08 +0000 Message-ID: <87618083B2453E4A8714035B62D679925074474B@FMSMSX105.amr.corp.intel.com> References: <20170104004826.17866.77074.stgit@localhost6.localdomain6> <20170104004831.17866.11537.stgit@localhost6.localdomain6> <20170104021531.GA567@gwshan> <87618083B2453E4A8714035B62D67992507430EB@FMSMSX105.amr.corp.intel.com> <20170104231139.GA3726@gwshan> In-Reply-To: <20170104231139.GA3726@gwshan> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiODBkNzAzZjYtYjFkYS00YzIxLWJiZjMtMmFmMTJkMGVkYTk1IiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX0lDIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE2LjIuMTEuMCIsIlRydXN0ZWRMYWJlbEhhc2giOiJcL3VOWFIxM3pqT2dRMmhUWG00N2Rkd1VcL0hNXC9YVFNGdk9xVXdtbVRtaTlFPSJ9 x-ctpclassification: CTP_IC x-originating-ip: [10.1.200.106] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8BIT MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org >-----Original Message----- >From: Gavin Shan [mailto:gwshan@linux.vnet.ibm.com] >Sent: Wednesday, January 04, 2017 3:12 PM >To: Tantilov, Emil S >Cc: Gavin Shan ; linux-pci@vger.kernel.org; >intel-wired-lan@lists.osuosl.org; Duyck, Alexander H >; netdev@vger.kernel.org; linux- >kernel@vger.kernel.org >Subject: Re: [PATCH 2/2] PCI: lock each enable/disable num_vfs operation in >sysfs > >On Wed, Jan 04, 2017 at 04:00:20PM +0000, Tantilov, Emil S wrote: >>>On Tue, Jan 03, 2017 at 04:48:31PM -0800, Emil Tantilov wrote: >>>>Enabling/disabling SRIOV via sysfs by echo-ing multiple values >>>>simultaneously: >>>> >>>>echo 63 > /sys/class/net/ethX/device/sriov_numvfs& >>>>echo 63 > /sys/class/net/ethX/device/sriov_numvfs >>>> >>>>sleep 5 >>>> >>>>echo 0 > /sys/class/net/ethX/device/sriov_numvfs& >>>>echo 0 > /sys/class/net/ethX/device/sriov_numvfs >>>> >>>>Results in the following bug: >>>> >>>>kernel BUG at drivers/pci/iov.c:495! >>>>invalid opcode: 0000 [#1] SMP >>>>CPU: 1 PID: 8050 Comm: bash Tainted: G W 4.9.0-rc7-net-next #2092 >>>>RIP: 0010:[] >>>> [] pci_iov_release+0x57/0x60 >>>> >>>>Call Trace: >>>> [] pci_release_dev+0x26/0x70 >>>> [] device_release+0x3e/0xb0 >>>> [] kobject_cleanup+0x67/0x180 >>>> [] kobject_put+0x2d/0x60 >>>> [] put_device+0x17/0x20 >>>> [] pci_dev_put+0x1a/0x20 >>>> [] pci_get_dev_by_id+0x5b/0x90 >>>> [] pci_get_subsys+0x35/0x40 >>>> [] pci_get_device+0x18/0x20 >>>> [] pci_get_domain_bus_and_slot+0x2b/0x60 >>>> [] pci_iov_remove_virtfn+0x57/0x180 >>>> [] pci_disable_sriov+0x65/0x140 >>>> [] ixgbe_disable_sriov+0xc7/0x1d0 [ixgbe] >>>> [] ixgbe_pci_sriov_configure+0x3d/0x170 [ixgbe] >>>> [] sriov_numvfs_store+0xdc/0x130 >>>>... >>>>RIP [] pci_iov_release+0x57/0x60 >>>> >>>>Use the existing mutex lock to protect each enable/disable operation. >>>> >>>>CC: Alexander Duyck >>>>Signed-off-by: Emil Tantilov >>> >>>Emil, It's going to change semantics of pci_enable_sriov() and >pci_disable_sriov(). >>>They can be invoked when writing to the sysfs entry, or loading PF's >>>driver. With the change applied, the lock (pf->sriov->lock) isn't acquired and released >>>in the PF's driver loading path. >> >>The enablement of SRIOV on driver load is done via deprecated module parameter. >>Perhaps we can just remove it, although there are probably still people that use it >>and may not be happy if we get rid of it. >> > >Yeah, some drivers are still using the interface. So we cannot affect it >until it can be droped. > >>>I think the reasonable way would be adding a flag in "struct sriov", to >>>indicate someone is accessing the IOV capability through sysfs file. With this, the >>>code returns with "-EBUSY" immediately for contenders. With it, nothing is going >>>to be changed in PF's driver loading path. >> >>Flag is what I initially had in mind, but did not want to add extra locking if we >>can make use of the existing. >> > >The problem is sriov->lock wasn't introduced to protect the whole IOV capability. >Instead, it protects the allocation of virtual bus (if needed). In your patch, >it will be used to protect the whole IOV capability, ensure accessing the >IOV capability exclusively. So the usage of this lock is changed. > > code extracted from pci.h: > > struct pci_sriov { > : > struct mutex lock; /* lock for VF bus */ > : > } > >The lock was introduced by commit d1b054da8 ("PCI: initialize and release >SR-IOV capability"). If I'm correct enough, I don't think this lock is needed when >pci_enable_sriov() or pci_disable_sriov() are called in driver because of >module >parameters. I don't see the usage case calling pci_disable_sriov() while >previous pci_enable_sriov() isn't finished yet. Also, it's not needed in EEH >subsystem. >So I think the lock can be dropped, then it can be used to protect sysfs path. That's pretty much what this patch does, except I kept the locking for EEH since it is the only driver that calls pci_iov_add/remove_virtfn() directly. I'll write it up and run some tests, although I have no way to test EEH. >>>Also, there are some minor comments as below and I guess most of them won't >>>be applied if you take my suggestion eventually. However, I'm trying to make >>>the comments complete. >> >>Thanks a lot for reviewing! >> >>> >>>>--- >>>> drivers/pci/pci-sysfs.c | 24 +++++++++++++++++------- >>>> 1 file changed, 17 insertions(+), 7 deletions(-) >>>> >>>>diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c >>>>index 0666287..5b54cf5 100644 >>>>--- a/drivers/pci/pci-sysfs.c >>>>+++ b/drivers/pci/pci-sysfs.c >>>>@@ -472,7 +472,9 @@ static ssize_t sriov_numvfs_store(struct device >*dev, >>>> const char *buf, size_t count) >>>> { >>>> struct pci_dev *pdev = to_pci_dev(dev); >>>>+ struct pci_sriov *iov = pdev->sriov; >>>> int ret; >>>>+ >>> >>>Unnecessary change. >>> >>>> u16 num_vfs; >>>> >>>> ret = kstrtou16(buf, 0, &num_vfs); >>>>@@ -482,38 +484,46 @@ static ssize_t sriov_numvfs_store(struct device >>>*dev, >>>> if (num_vfs > pci_sriov_get_totalvfs(pdev)) >>>> return -ERANGE; >>>> >>>>+ mutex_lock(&iov->dev->sriov->lock); >>>>+ >>>> if (num_vfs == pdev->sriov->num_VFs) >>>>- return count; /* no change */ >>>>+ goto exit; >>>> >>>> /* is PF driver loaded w/callback */ >>>> if (!pdev->driver || !pdev->driver->sriov_configure) { >>>> dev_info(&pdev->dev, "Driver doesn't support SRIOV >>>configuration via sysfs\n"); >>>>- return -ENOSYS; >>>>+ ret = -EINVAL; >>>>+ goto exit; >>> >>>Why we need change the error code here? >> >>checkpatch was complaining about the use of the ENOSYS error code being specific >>and even though it was not my patch introducing it I had to change it to shut it up. >> > >Right, it's reserved for attempt to call nonexisting syscall, but I think >ENOENT might be more indicative than EINVAL in this specific case? ENOENT is for a missing file, but if we got this far in the code then there must've been a sysfs file. This is pretty straightforward "not supported" error, which is why I picked EINVAL. Thanks, Emil