From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753539AbdC0PLy (ORCPT ); Mon, 27 Mar 2017 11:11:54 -0400 Received: from mail-db5eur01on0088.outbound.protection.outlook.com ([104.47.2.88]:54336 "EHLO EUR01-DB5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751124AbdC0PLu (ORCPT ); Mon, 27 Mar 2017 11:11:50 -0400 Authentication-Results: mellanox.com; dkim=none (message not signed) header.d=none;mellanox.com; dmarc=none action=none header.from=mellanox.com; Subject: Re: [v2] PCI: Add an option to control probing of VFs before enabling SR-IOV To: References: <1490198038-20465-1-git-send-email-bodong@mellanox.com> CC: , , , Eli Cohen From: Bodong Wang Message-ID: <272b9606-1b39-0b41-4c19-a35cfe5d789a@mellanox.com> Date: Mon, 27 Mar 2017 10:11:10 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <1490198038-20465-1-git-send-email-bodong@mellanox.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [172.87.190.14] X-ClientProxiedBy: CY4PR04CA0048.namprd04.prod.outlook.com (10.172.133.34) To AM5PR0502MB3060.eurprd05.prod.outlook.com (10.175.39.10) X-MS-Office365-Filtering-Correlation-Id: e483bfde-2927-4c3e-f8dc-08d47523963f X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(48565401081);SRVR:AM5PR0502MB3060; X-Microsoft-Exchange-Diagnostics: 1;AM5PR0502MB3060;3:v0icI+i/zHQTlCpSZkviyHW8kVoKpNOefzh+FjBCK4DguZKN7UyVmhcHpTN+6ciJZxQoYP4KdF+FATao14+daxmv0YlLP+iHNvPUgiyinQTX1RP343cUlyZT89v0SSMJrFHqfw1U1+YHZiSH0kQKJ4QpGaOhyiqDkvT50tIoMBsLDXeoTlxlEvGFWXlcj3b19AE5L3tdbNdbNFmEBmDFg/zPY1zYNi3fl8Aj0+tJYokHjPfGq278YJvb2wjf0RB5MUQWYrwY2pA+Dys7J8B1Xca4kPdFMCF1ED+z0lltNZg=;25:F9s+qrBV1v7+Xeo53KuZ1Oe+jgb7kGtY1yFrRBJW+728OSrOlEW8XKghRHkotv88foT+dHcbL43HS3yNxvrl5QFLyw7XjvtjR/iOB9cDqM1JPDrtdM/JuoMuu0r2RmUkwByubQFRDDmyAq9+vxTXKyMprYRXYmeUH7xMPhM4gBDenerPMHdnXSOJ5PrEhZV5tQOFVmEUBouYD/OdYxjD01DRFZtLZOKpfnML95w1neLRClmWZnzz0f24CEB+Ss3tMTRSTIxAmpc97jzK0xUh1EOgIGkEMTn0b3aSAv6puQjDn17wrllbKKK5UdZPVQ4fzWra0tpOuZb6AFZeoPLDYAVB8Le5uqNL5T1dDW12Vq0YbyNXmWdH04Cfv48Q8lRqjwySJO5vHgzTsrvf5AxbLuwvrmGfl7OJpry6xLszvsZ/N6mH8KsBJ8momwlm1RTdt7gzqf0s+dYp6n0nUV604w== X-Microsoft-Exchange-Diagnostics: 1;AM5PR0502MB3060;31:KsrEOqRM5xX+0IZHUpKmPg+RnxMJOLaRotojKyS0G6aXEHnLSpWUiQAa6MgSWuNX48uI9vuHKMKR0cuAklOOSMvI9bLlgRN0iTGzWEkG0B+vk1Vu48y0bybzgg8p1BQj/KiUVigKGv5gDEHHDcwKMHMjOPhhYQNedx4m3HE2jAiyr2OXjhtZgUaXvdbZfsvOwUEdKq6NhOCRJ2SQQoInYYu0RihWTk30c56c7fYzP6M=;20:heMOfP1q6U5geS9TFKiymR5mNGx8laNGUHgqg2rlPRzJYiR2+zmJ5+ta19KcPv3Wyuec5qIpgUc2/fn9fWAy0nO1c/XLzmMwYHLgjXCSrtMxdyWf4sdv7M0s/Rv0no+8Gi/RmTeplASr97JtJSerAeCxR98Qu48Ok7DwIAzEshmcwwRnqz/VX/Q/+IaNhTi7m6IUtdLXS7w7cCXT475cE42mnTzsEvo8GVgGmN8G5KX4S5eR9WbzpD4n/sxxdiyjx0DoRAEOHrGZQdERpLAsjEd+HaCYU0Em4c+peTpMlhIKy0unB+Xisbw0CMNSoR+GD27cleVHz7EKkveN16oG5q60jXoWSvgzVhj35st5u8IdrJypuwR0KhOnGwliiInibvsgLXIrx+T9aeS6V0omDi9pyJ5gwM0bS/FVAyxsgBW91e+7hNS2pTLFCA0d5esic3FoDxguPl9mvmCDWfgXiT4voBj5s7BPF3ila4eqFeJx86TFXSOpLHmGS38j2mdr X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(104084551191319); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040375)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046)(6055026)(6041248)(20161123555025)(20161123562025)(20161123564025)(20161123558025)(20161123560025)(6072148);SRVR:AM5PR0502MB3060;BCL:0;PCL:0;RULEID:;SRVR:AM5PR0502MB3060; X-Microsoft-Exchange-Diagnostics: 1;AM5PR0502MB3060;4:yku72wd0s/DN39xN/iP+tPx6XI/S1nwixiyRyP8Zq8AMQHmXDQMNOhGOBE4vUeWgka27gaRwpIN91rX8ihOitSUjVpIFUkLVwWpd+RFcIUyf+B1wfJoN0hvTKA9yT+4MBX8jKOKgrQY2x+vZjJ8QzBYH0JnmkrMYrKAvJKlVSkAz3i4iwJ0bcLczm6NJg1jPMAFpvdEicz42TAXEVdMQ92FP+iLBlMysKTUVkOQ294KZEy5VZ1+MTRa8YL8Me5z4o5Fe4QiZQlAW7ciHssbjPqDqflogyGedsmZ/hYt9xq8OLAF0MwQ6LHjEhnwu4YSlAh7sBt+QuVVd1Mczd7VdiMhECUtuWBhXdra/nI7Z4gQbuoQa07s4zLL7AUq10zot1fki0QUY0ZdaX3nfLlUL5MgTnssgOUiPOuX4e0mu0+lSKMWMzoBYeYlkj6QAsS78gCyNYfqGFjue4DQdPZtZEqFu6RocLUfdwzQ9NCJwn0bzQOikHSc5a8mWVFAi35YUD43H9V0McaE4ny3+t37wY8Uh4NnTmm5MJ3Ax399NfgOEXTroWE/a61jH62+/yj9/KSu6jMVQNn3tuZV7tox2P5jaLM809wXjJdzWCWVFkEW0DL/o0Ipexjj9M3pYsjHZ7k4UYfvPe1aHSWhSctebz66yapGvpQxyUkHHR22kI9A= X-Forefront-PRVS: 02596AB7DA X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(6049001)(39860400002)(39450400003)(39840400002)(39410400002)(39850400002)(24454002)(377454003)(50986999)(65826007)(189998001)(5660300001)(64126003)(2906002)(76176999)(54356999)(36756003)(86362001)(7736002)(305945005)(2351001)(42186005)(31696002)(31686004)(47776003)(66066001)(65806001)(65956001)(33646002)(230700001)(81166006)(8676002)(4326008)(6916009)(2950100002)(6666003)(23746002)(77096006)(229853002)(6486002)(6246003)(53936002)(25786009)(53546009)(110136004)(38730400002)(107886003)(54906002)(3846002)(6116002);DIR:OUT;SFP:1101;SCL:1;SRVR:AM5PR0502MB3060;H:[10.12.149.45];FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;AM5PR0502MB3060;23:sQM2HzCmf/zClCT51HQs+YpNlqTLF90JUEQ?= =?Windows-1252?Q?V1j0sRGeSmCU6oEn7QP1BqJFQg/kv2Qqmv9mGA/9PdGK1VZXtHbXrIZO?= =?Windows-1252?Q?xqsRaJ8ZCZHUY4mn9otoQL6m4DLUmFI76IQ3/NrNvzySe261S4C59dCi?= =?Windows-1252?Q?37OfVmR4hLthAFq/GpkZVG+P0I942Q2veuLBWsW2NssCGB3rzkzmuPE6?= =?Windows-1252?Q?YP85rmrdYYce1fL/Kkv8UidUUcT5wGK2G3NDX0iRlaQvYcjamcS48AJc?= =?Windows-1252?Q?sD/qg2R5CMSldzrY0Zv3N9Mr3NHg8Z59ldjwtZlNNkJHji1mVK+QUO56?= =?Windows-1252?Q?qMsAYpHEY2byiwdFOv1GchsENNDV2G1Djslxf0F2roN9rnRWw95SSaF9?= =?Windows-1252?Q?AJcs7KFmiobIP59ugaSjd6N2CWlmEbtCE6yAXH8b7DvOsYI3A9m30Gxe?= =?Windows-1252?Q?y0kmwhMfgNc8yuM1zlJVDYJdiEzP0ydfG7qoPViWJmFYRIlIWl1Cn62c?= =?Windows-1252?Q?+7n7RT9GviOHeJTq3euGanLHT7SQ7FC49EV03ywqLlPX/HNM2MwvrDRC?= =?Windows-1252?Q?SDfJkLeeOiBR+Z09sgjSF8zBqZDVVvhERUnYDaJBtdB3wHcbtU4In9ZX?= =?Windows-1252?Q?ER7xn90xX4gTHVwib0HE0q9SPcYP36VJrxmKeUp6tnjutqPP0k1xg3du?= =?Windows-1252?Q?HGHT2ihC+Nz6N8bCT3znUxz9YaVybDF3ELTHqisYldhTjQQYLjF9dFtQ?= =?Windows-1252?Q?sdtWNthUW6r+PZeeUPTXnW2xNHbQhvXFsmACf1dxgRNTEALxBvtZ+3NN?= =?Windows-1252?Q?8Bb3i6GB8ZlImbVulTF8aAb58nNWZlDCXI1Zm6/16We0VdMFrJabfFNK?= =?Windows-1252?Q?5nwKn94Td1r8bipICgXt5oIqez4Qcp6aMSD+u58enBrUqEB1ozdTm0eg?= =?Windows-1252?Q?kVkksNG3+AP8NTnA1gXry9UXzgqmWDuE6b/gyCi+mnydk6js0CEce4Wy?= =?Windows-1252?Q?Jn1vu0A7VJUK0R+3uAdJA60TJmjVnCZE1zzamRdUh3KgqKy0ZL/WaN5V?= =?Windows-1252?Q?Ahh4/3Qurn2u5ZnbpUSSQRh9uKafc7IIcBpUQ5+BlzN9uWx0T3Br0j9V?= =?Windows-1252?Q?y0yqi73F7snUlgUWWWgmrZMEKuBH/kcsQJ0AMHgOEv/xDsCf7x5hf/9r?= =?Windows-1252?Q?x6aesEC6/XVmm2oErY/SIYGHLQlJYKzQP6sH94Bqr2CdIt7reXvszV/c?= =?Windows-1252?Q?FEmzggNOWU/gZ91e+3IkxwjRGF4UnIIQJuWDHPz3aeh355EjvDiTQegC?= =?Windows-1252?Q?P738HouVxQczzY+UcYFU5MuvEHRKXJJ/kwn6FPlaQ/eKSfE2a6EhYRH/?= =?Windows-1252?Q?e8N0VjLah60BR?= X-Microsoft-Exchange-Diagnostics: 1;AM5PR0502MB3060;6:SmBnQ0ddtJjacEEZAOPIrL46G9IhSfsFd5UXSiE0kRoyp+zYL0FMhLau1Wno3Y7nPS5AcXUttJOQ9NFbAl36wWHJR3ExKGil/DpZsPmfSdAdjig2hqnHNdKdI4R0eglVEOGOM/GBmUz43r86JuwIGTum44/P1eDeGW6p0tj8v616mPY0dmYTZ7wZmTFTTveJkLra2HcPidMCsD2+ujHj00Bpf1a8dfHyUrxa/u1NXLHC/zPP6WToT8ZjgGCCTEH4Z3XfVInaQx5RjQHdo/nlxofY5NCiVoBy3+ahcW1vKinVpkQq1RulPLJzCzPLNbVHh0JUJZ6tHxBx1qghJdozdyithgaQWnwiunp6nuwyU0X5QtiwBZaV1mwNdO3i1BvLWq4Vyz0n1uQVG/rmZ6EMzoCRc5v8ARyfJq0pGbP+Zb4=;5:4kWHb9bF5c/uVKsAqVIM95xTu/H3B9iQ8LBzE8UYimjjZihB2I+NBHGkliHd3nqqv2BIkSb2IeL4p+0bgMCn9zKoFyiirI3AsVcwxShb0S1PhHSa8x5pC1W85v2JrvYsllwAvgI+a9+nz9f/7AbF+A==;24:GSTTDkemdogpGgUdrWZPpPjBmL4K1pmSrejscSFU955ZCWL3QlT0znUd8t/lDwIuECj0wdWJoyC/NyfI/lZ9zI9gCyyTKbBdZvcBEKZZCA8= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;AM5PR0502MB3060;7:Uvy1XgAZiO+HdKCnQlTXh1PRICcYx343eAlBdCih3D9Ab2DMYKxA5bs7yQG080p463S1ctGofd+RMbCcBw/soBoYDozFs9g+PHjnQziAPojmrKX8rRkm5EMnGcTnS3/bYGiKlUsXjjL+7JlC9Kw1lVW7mVuO15LAFZeRIdgBGez/u1X2f21H4XQkqsqChRJXFLt9iiTrumuNwPfH6YFdqbmGQ8RC+DRjcnNK4ziiavYYAzRJAD/c4XRg2WkgZNOaETxFgbrZCRjYMWuUmEO6GqaRGNL9Le2cDLKOId2eG7NG3MKgiexV3KICRMrGWJMN2qvcTfGNM95qGmO1PjDNWQ== X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2017 15:11:45.5159 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5PR0502MB3060 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 3/22/2017 10:53 AM, bodong@mellanox.com wrote: > From: Bodong Wang > > Sometimes it is not desirable to probe the virtual functions after > SRIOV is enabled. This can save host side resource usage by VF > instances which would be eventually probed to VMs. > > Add a new PCI sysfs interface "sriov_probe_vfs" to control that > from the PF, all current callers still retain the same functionality. > To modify it, echo 0/n/N (disable probe) or 1/y/Y (enable probe) to > > /sys/bus/pci/devices//sriov_probe_vfs > > Note that, the choice must be made before enabling VFs. The change > will not take effect if VFs are already enabled. Simply, one can set > sriov_numvfs to 0, choose whether to probe or not, and then resume > sriov_numvfs. > > Signed-off-by: Bodong Wang > Signed-off-by: Eli Cohen > Reviewed-by: Gavin Shan > --- > Documentation/PCI/pci-iov-howto.txt | 10 ++++++++++ > drivers/pci/iov.c | 1 + > drivers/pci/pci-driver.c | 22 ++++++++++++++++++---- > drivers/pci/pci-sysfs.c | 28 ++++++++++++++++++++++++++++ > drivers/pci/pci.h | 1 + > 5 files changed, 58 insertions(+), 4 deletions(-) > > diff --git a/Documentation/PCI/pci-iov-howto.txt b/Documentation/PCI/pci-iov-howto.txt > index 2d91ae2..902a528 100644 > --- a/Documentation/PCI/pci-iov-howto.txt > +++ b/Documentation/PCI/pci-iov-howto.txt > @@ -68,6 +68,16 @@ To disable SR-IOV capability: > echo 0 > \ > /sys/bus/pci/devices//sriov_numvfs > > +To enable probing VFs by a compatible driver on the host: > +Before enabling SR-IOV capabilities, do: > + echo 1 > \ > + /sys/bus/pci/devices//sriov_probe_vfs > + > +To disable probing VFs by a compatible driver on the host: > +Before enabling SR-IOV capabilities, do: > + echo 0 > \ > + /sys/bus/pci/devices//sriov_probe_vfs > + > 3.2 Usage example > > Following piece of code illustrates the usage of the SR-IOV API. > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c > index 2479ae8..70691de 100644 > --- a/drivers/pci/iov.c > +++ b/drivers/pci/iov.c > @@ -450,6 +450,7 @@ static int sriov_init(struct pci_dev *dev, int pos) > iov->total_VFs = total; > iov->pgsz = pgsz; > iov->self = dev; > + iov->probe_vfs = true; > pci_read_config_dword(dev, pos + PCI_SRIOV_CAP, &iov->cap); > pci_read_config_byte(dev, pos + PCI_SRIOV_FUNC_LINK, &iov->link); > if (pci_pcie_type(dev) == PCI_EXP_TYPE_RC_END) > diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c > index afa7271..2a1cf84 100644 > --- a/drivers/pci/pci-driver.c > +++ b/drivers/pci/pci-driver.c > @@ -394,6 +394,18 @@ void __weak pcibios_free_irq(struct pci_dev *dev) > { > } > > +#ifdef CONFIG_PCI_IOV > +static inline bool pci_device_can_probe(struct pci_dev *pdev) > +{ > + return (!pdev->is_virtfn || pdev->physfn->sriov->probe_vfs); > +} > +#else > +static inline bool pci_device_can_probe(struct pci_dev *pdev) > +{ > + return true; > +} > +#endif > + > static int pci_device_probe(struct device *dev) > { > int error; > @@ -405,10 +417,12 @@ static int pci_device_probe(struct device *dev) > return error; > > pci_dev_get(pci_dev); > - error = __pci_device_probe(drv, pci_dev); > - if (error) { > - pcibios_free_irq(pci_dev); > - pci_dev_put(pci_dev); > + if (pci_device_can_probe(pci_dev)) { > + error = __pci_device_probe(drv, pci_dev); > + if (error) { > + pcibios_free_irq(pci_dev); > + pci_dev_put(pci_dev); > + } > } > > return error; > diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c > index 25d010d..1d5b89d 100644 > --- a/drivers/pci/pci-sysfs.c > +++ b/drivers/pci/pci-sysfs.c > @@ -526,10 +526,37 @@ static ssize_t sriov_numvfs_store(struct device *dev, > return count; > } > > +static ssize_t sriov_probe_vfs_show(struct device *dev, > + struct device_attribute *attr, > + char *buf) > +{ > + struct pci_dev *pdev = to_pci_dev(dev); > + > + return sprintf(buf, "%u\n", pdev->sriov->probe_vfs); > +} > + > +static ssize_t sriov_probe_vfs_store(struct device *dev, > + struct device_attribute *attr, > + const char *buf, size_t count) > +{ > + struct pci_dev *pdev = to_pci_dev(dev); > + bool probe_vfs; > + > + if (kstrtobool(buf, &probe_vfs) < 0) > + return -EINVAL; > + > + pdev->sriov->probe_vfs = probe_vfs; > + > + return count; > +} > + > static struct device_attribute sriov_totalvfs_attr = __ATTR_RO(sriov_totalvfs); > static struct device_attribute sriov_numvfs_attr = > __ATTR(sriov_numvfs, (S_IRUGO|S_IWUSR|S_IWGRP), > sriov_numvfs_show, sriov_numvfs_store); > +static struct device_attribute sriov_probe_vfs_attr = > + __ATTR(sriov_probe_vfs, (S_IRUGO|S_IWUSR|S_IWGRP), > + sriov_probe_vfs_show, sriov_probe_vfs_store); > #endif /* CONFIG_PCI_IOV */ > > static ssize_t driver_override_store(struct device *dev, > @@ -1549,6 +1576,7 @@ static umode_t pci_dev_hp_attrs_are_visible(struct kobject *kobj, > static struct attribute *sriov_dev_attrs[] = { > &sriov_totalvfs_attr.attr, > &sriov_numvfs_attr.attr, > + &sriov_probe_vfs_attr.attr, > NULL, > }; > > diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h > index 8dd38e6..b7d8127d 100644 > --- a/drivers/pci/pci.h > +++ b/drivers/pci/pci.h > @@ -272,6 +272,7 @@ struct pci_sriov { > struct pci_dev *self; /* this PF */ > struct mutex lock; /* lock for setting sriov_numvfs in sysfs */ > resource_size_t barsz[PCI_SRIOV_NUM_BARS]; /* VF BAR size */ > + bool probe_vfs; /* control probing of VFs */ > }; > > #ifdef CONFIG_PCI_ATS Hi Bjorn, could you please comment? Thanks.