From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933380AbdCaPHr (ORCPT ); Fri, 31 Mar 2017 11:07:47 -0400 Received: from mail-he1eur01on0087.outbound.protection.outlook.com ([104.47.0.87]:64082 "EHLO EUR01-HE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932934AbdCaPHo (ORCPT ); Fri, 31 Mar 2017 11:07:44 -0400 Authentication-Results: mellanox.com; dkim=none (message not signed) header.d=none;mellanox.com; dmarc=none action=none header.from=mellanox.com; Subject: Re: [v2] PCI: Add an option to control probing of VFs before enabling SR-IOV To: References: <1490198038-20465-1-git-send-email-bodong@mellanox.com> <272b9606-1b39-0b41-4c19-a35cfe5d789a@mellanox.com> CC: , , , Eli Cohen From: Bodong Wang Message-ID: <9ccd6738-0db3-280e-5bed-d2498fa732d7@mellanox.com> Date: Fri, 31 Mar 2017 10:07:24 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <272b9606-1b39-0b41-4c19-a35cfe5d789a@mellanox.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [172.87.190.14] X-ClientProxiedBy: BN6PR20CA0023.namprd20.prod.outlook.com (10.173.158.161) To AM5PR0502MB3060.eurprd05.prod.outlook.com (10.175.39.10) X-MS-Office365-Filtering-Correlation-Id: 19c5e99f-832d-4959-286a-08d47847a97c X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(48565401081)(201703131423075)(201703031133081);SRVR:AM5PR0502MB3060; X-Microsoft-Exchange-Diagnostics: 1;AM5PR0502MB3060;3:2XBmkFtjKO9HILsS+Yh1jnBa65kyVK2RvDcpwSaAL190K1VkY1y/zLr6uJOHBih3zhs4uLl5ZofYkidVrHuFuJU3scduXeXzrXBvyfh0oF25S0dguhtzGBZyVKajeRRtb6GrLQIWTSDk3iLjTtPFEGjuo93PrxvuL7rDkhaorkaY3Dl8Gfr7aCYvHEOc4Ap99j7UDYlkdrKYKIOVNsGvtUw19R4pyAwAJ1aNRJpZeRAU76fJIMAj1GEqI4kmmOxprRCCV9BRpja5LEZhXAxjAageOP8LNwYzwc8x68piuibJPZ8b2+FmyXu4e5ShXb0WH9gSgLrKu+UQP/VIDCL5jJ0TJgEmcF3McEsRggf3VYE=;25:Hsr9+jf4cTuwRC1eQQx5kHmB/NTivYkyPaC3ha7gdyLMyE0InXEvvhbfXXMQdzplVOAe9UOxSwazirsa978ft+9NObImN1cab/3Js4fEXdvEDrftnE4i0ZFkJj3zQDv1B4KLJ65Ft3TCvag99t+l9XMMXuwXHSEVnGCf4u5Jio5sNUEF3ChViW+ny2u5DKqiOT1cX7Mir1IyOJAzdCmR9cUeTpHQSQeUFxL9l8pZu4HG9X8VrpLDj4JEy4T2e7VE8ALmBXTK+eC3YRJKllJZ67Hfyt9lpERVJK0L3HW6slqidIWCdmOCWau6ZNq6+mpGIYb9sjXkM2vc0NXxch+bZMnR96t4hVUZ3UAEZua+Px93K7DlwP7Bh/srVFFFZsGaJl7E5M3qgI42/EkO97MBYVk4XxcTQvMD37PdYpMp4rmZ50Bs+Vxqghotjc1nu4J022WVkFtFoCoRY0DW2/rBRA== X-Microsoft-Exchange-Diagnostics: 1;AM5PR0502MB3060;31:FyKyy/0DoYimbZhon07muG+0x7QNcMKg/gTKLSOT/wka4m8j/QaYYzVkk21C7V7nBEQSzauzP0myCCrXpq8abZ0/fAQWV6rOp30+1wlBOJ+DjaGwUJ3aU1ePB6CLivby5PwfHu9LQ/rFc9q5aGJuwsv+PD2myGTo1Vp8aWNADXiD4yBTJMRaBvAon/yWwZKUH4dG7s5c/fh+Ob0ZkogtgqSdfIWF87+frui+hNI1hJrf3/OEr3HQJ0QozInk/QAf;20:Pw0tWTTHjoNFDWXhUj6Des6rcHuCQd91dHxXAe62DHz77UTb0stAhgJBUVFUQCDx7sr3p6YGnd1KuPYx0f+45VI9J8jFE1atKE9Nb4z2taTlXyfEXxsisYZ1XZshq5lDMmMklxmwu+76wjuKYWk6OLJaKFU5frjZtdrev+mLylDNJBrV4jxQ6+TqSNApF/FcAcGm7cz3EQYPeFtXJArN+8xCSqNghVsPlNwQWea9FvY8o61jPqZNJa0epBoiv8Mn7gCzhx/co80nq/cENKobYxbhzHN2JJcbSvnNkGFuvBBlUM7588i+496uQLTW5/Clvow+CfOIpNG4LW2Pr4EGQNioNB0PErP5YNKUdcX8SUCzx9oXxZC/fDwbqm1BXQfin8Vy81rMAie+r059NTDZb+flkKEwfCBECbKlDbZsc9SKR648xM1zUY+jLv9h5recEoEylb4TeESaxsVTQYwbOG+SYcrJbYY8TK0QJ4ejbkT+N5IdTxrsrbJUDwOkbP0q X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(104084551191319); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040450)(601004)(2401047)(8121501046)(5005006)(3002001)(93006095)(93001095)(10201501046)(6055026)(6041248)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(20161123555025)(20161123560025)(20161123562025)(6072148);SRVR:AM5PR0502MB3060;BCL:0;PCL:0;RULEID:;SRVR:AM5PR0502MB3060; X-Microsoft-Exchange-Diagnostics: 1;AM5PR0502MB3060;4:VNYYeiwi2OYq5KZ9LfIYtIw7rpOeIjs0LGPiEqLTb7H10dUzqTey/aWufh3DvXvr/w8xyKj7LioxJ3wE95/iHdB0teXDX7JJCCLXGFMhoSA+nkYBIFpegrBABCMTOoxdhKwRjldYlmFzIKF/2vKD8vHppdteov5gLFK281VACPpdH/gfqb2S/OuiUK3kJxuvNRLKJLUgEC0xQDdkRRqm/6MSl2DT2LF29hJxtb5kF72PAupdjJYUJ4P89qWvg1BM5X/4isNNUw9boZBlQZo8zsCHP2gZbJus9IjwWhrfoc7LKIYd2cJU0xF5rL5fOwgm7MfdqtvY3Es2anMbYG2ZC8hxaSAXZsDr1sr2tmQIq8d6s8DRmDU4IRj2VHa7Y4fHjfy/VKASX+RjlktOV6L/xeHrqNEV+WgI/k3dF8o9bdRzRLdyqBhe9MICXKBs27fXGRdxEDlBFUCq2Y3U/8HrMUnATf6ZR67zP2yUr7mygMeVZ6H9DBqShQ5MuaB8tHhvVedN5q8BEjIeNw3OXLS+MPJAy/azUs2MGE7qupg0qjTOLq9J304j2KZYEFpoJ/na7j6wKX272dm5KxBBLT1equGraj090+xsEkSNG35yeNd7DUP1cX63MRVNm+2XzRyE349aJfwCSYXiWM31PuMUKmoieY6YMB/S3I0L9IHYRKZXd0HnvHsA9LnPh2CFree6AaOP5aI72kyY1f8t9YR81VZIwSB2uLOG6ek61zmlGBRCyz76WRCa3hJKD3+COWriqCfbmpZfIQ6mZnU1U+Vo4A== X-Forefront-PRVS: 02638D901B X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6009001)(6049001)(39450400003)(39850400002)(39840400002)(39860400002)(39400400002)(39410400002)(24454002)(377454003)(53936002)(305945005)(50986999)(8676002)(31696002)(81166006)(2906002)(54906002)(76176999)(36756003)(4001350100001)(86362001)(5660300001)(31686004)(33646002)(64126003)(65956001)(2351001)(189998001)(65806001)(6116002)(3846002)(54356999)(47776003)(66066001)(230700001)(65826007)(6486002)(50466002)(42186005)(107886003)(110136004)(4326008)(38730400002)(25786009)(23746002)(53546009)(6916009)(229853002)(6666003)(77096006)(6246003)(83506001)(2950100002);DIR:OUT;SFP:1101;SCL:1;SRVR:AM5PR0502MB3060;H:[10.12.146.139];FPR:;SPF:None;MLV:sfv;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;AM5PR0502MB3060;23:ykVFYBi1RjBZxGCLADpYrhjWsA+qnBC6a4+?= =?Windows-1252?Q?AZoPKRet5D6ukB/omLfoRT22nxB+1668TYNq3+JSfqFCUkqp4eFlS7YI?= =?Windows-1252?Q?Qt67h952dFM/wM3jAQgutjcS6T88k8sq9l1R0ISClNSMJRTHPriCijrR?= =?Windows-1252?Q?Foojpje9MVgg8Gvf9KbV/S0Uh6Jnh79fgrBMZx+U7WuJ8RqDgZsgrxcr?= =?Windows-1252?Q?k1TlnpDX85vpu30s55/IP2uMKNQnd7CeO5YY36kVfiGJ1ygAyDxiBAGw?= =?Windows-1252?Q?A+T25pQY7rpGgSRQQnFGnoD9H4tZRx91EepcGNc/dMJ86UfwpW28XeoV?= =?Windows-1252?Q?kEsW+8/+NipUNJCf3H1vrgVLVqA+SJMrqxlJpmeDGwnr4wQXaQZ7GxL7?= =?Windows-1252?Q?S5HPL15Ij+MuiZnfmwTZFzMMRbUVJxkUxnx42gSmKnaNrWZBU48INpwZ?= =?Windows-1252?Q?YS0aCRJA93OLoAzAg7O3VcVe0p1y3ko8QoslByt0W3d7qQLYQoEY7xrF?= =?Windows-1252?Q?93VSRuGUHBmUO+lRMQcFHJCI/Ojfqf74e5sH9i6jDFeUNLnEZHQKq6o5?= =?Windows-1252?Q?cEAGwawjjXeaCoY5VN2eCwy0z3njnpMdbtfCq9CveVyBW+Wqv864W/8t?= =?Windows-1252?Q?k1kUq8K8lIt+aMgkklUkW6WtslvJs3f54QbNAenq2HrXzGjf5dQarqJH?= =?Windows-1252?Q?tuzyd3TrAQTo9Maq4GdV1vFKPPQxNK4oY+bXmsbX6d2plCtIjLNZOptk?= =?Windows-1252?Q?q/cnB2NcMhEe/NgE1Jr+chEyX+09WXC3xaI86V3D8513OWOeMhbgUHdE?= =?Windows-1252?Q?j6B+sfnskrEU2WmM/zpNG7ilhDxrf6Dj9UE+ZUtXNVF+8n3boqT9/sW/?= =?Windows-1252?Q?Z0BjtaezQbG2umh+bUybOS2wuPhvPmYQuSoRwv0rFLg1ZMiNHTEIMYES?= =?Windows-1252?Q?se2P9eChbyvQWr3xHwbRtQrnYNj5DAkb8LhBSQTHrgY3jzgHQv9CpYeE?= =?Windows-1252?Q?sqqmdGpsf0wMg9xnbJFDHlsZKS10ykfZ9nZ5s431YFkn51ye/dPLBw0y?= =?Windows-1252?Q?binAlZQj6r/RbCaHGnEq8wvlHFThitO3M8CqUPdw7JUOt22mKOAXLmbb?= =?Windows-1252?Q?3aAss2vD2o27G54+LkF/jlESbkYGtFHCEesKmc8KIYUSYWEW18jQ0UgV?= =?Windows-1252?Q?UoxNfcfJbM7hkNTtWM3JI6UtA4NYsuBK5XjCOabkIXjwhHK9qAipvK14?= =?Windows-1252?Q?zg2kP3NRXR604gby0ad45cOINRgIGlZ1I2O5qnmdZO2KVQ3hz/Ic+68H?= =?Windows-1252?Q?VGCgvoY3CQhck+Pnq95Oif7b5LykmRfeg0DZZXoCszm5cA4AeKSpI3qV?= =?Windows-1252?Q?NTZAQWq2n7xbYtaSvNUNTZgN3QvvGvIUPBdonqli4HwfXnq/2IDjujSy?= =?Windows-1252?Q?BcnolhWaM4Z9X6/TjGTXGWU9rTwTLo8Xza6HhJQvCPw=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1;AM5PR0502MB3060;6:zyynmZDFFCX9yypiebr9XaW1G6E5LqupEQ6pdMkI9cXpokmqKWpivn5q9l+V+AssbnU8xXMVmJYdbHmPbfo69qlwmmgKVh94RfmAuf971P5wpzkbWMBHNte72XI9pdJXAmIjX1EdBOx37o5AT9YhcbHJ/iqY4xxK5sGjnYY2zV5meHAJMYBgd7WvuZRSiVqv/6mMci0a92sHAGPVxReFkqCEVSFRXULX/4YDweJnx2PfT1uEGAxgHuZiQ8VtJWxM8kJgPQGgDUy+vGq5NgmNq2SQxLcCNyyjdiLz/hcBemDqy0VddZdfYrabfNwv3Q94XZ5XGpSRcQR1lN+JJOw4xiqFUyrgAN4wDd1XNpc7fdXMGZcv+WQk/TODG+0fvZmUzfEo9qDEm8GutTx0r8rpGIHIHq91rU7lyasggzEPzx4=;5:/20UCo/QQvvboaDlTEaESRmW8xpu8T7TRRYF3PANk6nCI1mZQ8oYhpiXpWoyqqMJTeCjhIlDNafpIlo76QW3+uZDmsTz7QTvhfdgE0mCDW0U3vKz2MM46HAuD5tp0CfCD3F3+nAlyOMPno7AaZOauQ==;24:GHy4dUppNF36+O+razANVbftFsjjGtxAP6yw0FxoN+dQLWtaJWhGdvk012Lcm2TfKDkbvgR+7VDgRtNrQC2F76mlL10Zc7b8CkgVJipBW08= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;AM5PR0502MB3060;7:Hy8ehPbIoK1L2iO7WcjykS3I9iEXCNY3S5QLV48mixmrAUmmYymTZDx/XiAXQR+iulVuzLCU02kqbzCav9d1+WnKoOqfSuF/UfM5ljHP/D8dE1/nw07XCp7e7OF7FhmJW64E4pycziJH0sVNL9FpUR3ChYGXgfojGLPlFhtcEl4jHiBXH8/r3fXXvsyzIhdSbsFwFiOZltDXy5P8bjbUBGoZ/SztSYOg7PjOnfC8D0M8g0Ont0OMa1Y9thhmNmGquLVa11jbrhyldGZv0pFEL3VC/zVPzsccGnr9NOPN8tPUVU/D/NZs4X0b9QLa0XwtgxSuXO4FoMUGwQYp2bvHMA== X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Mar 2017 15:07:33.6751 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5PR0502MB3060 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 3/27/2017 10:11 AM, Bodong Wang wrote: > On 3/22/2017 10:53 AM, bodong@mellanox.com wrote: >> From: Bodong Wang >> >> Sometimes it is not desirable to probe the virtual functions after >> SRIOV is enabled. This can save host side resource usage by VF >> instances which would be eventually probed to VMs. >> >> Add a new PCI sysfs interface "sriov_probe_vfs" to control that >> from the PF, all current callers still retain the same functionality. >> To modify it, echo 0/n/N (disable probe) or 1/y/Y (enable probe) to >> >> /sys/bus/pci/devices//sriov_probe_vfs >> >> Note that, the choice must be made before enabling VFs. The change >> will not take effect if VFs are already enabled. Simply, one can set >> sriov_numvfs to 0, choose whether to probe or not, and then resume >> sriov_numvfs. >> >> Signed-off-by: Bodong Wang >> Signed-off-by: Eli Cohen >> Reviewed-by: Gavin Shan >> --- >> Documentation/PCI/pci-iov-howto.txt | 10 ++++++++++ >> drivers/pci/iov.c | 1 + >> drivers/pci/pci-driver.c | 22 ++++++++++++++++++---- >> drivers/pci/pci-sysfs.c | 28 ++++++++++++++++++++++++++++ >> drivers/pci/pci.h | 1 + >> 5 files changed, 58 insertions(+), 4 deletions(-) >> >> diff --git a/Documentation/PCI/pci-iov-howto.txt >> b/Documentation/PCI/pci-iov-howto.txt >> index 2d91ae2..902a528 100644 >> --- a/Documentation/PCI/pci-iov-howto.txt >> +++ b/Documentation/PCI/pci-iov-howto.txt >> @@ -68,6 +68,16 @@ To disable SR-IOV capability: >> echo 0 > \ >> /sys/bus/pci/devices//sriov_numvfs >> +To enable probing VFs by a compatible driver on the host: >> +Before enabling SR-IOV capabilities, do: >> + echo 1 > \ >> + /sys/bus/pci/devices//sriov_probe_vfs >> + >> +To disable probing VFs by a compatible driver on the host: >> +Before enabling SR-IOV capabilities, do: >> + echo 0 > \ >> + /sys/bus/pci/devices//sriov_probe_vfs >> + >> 3.2 Usage example >> Following piece of code illustrates the usage of the SR-IOV API. >> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c >> index 2479ae8..70691de 100644 >> --- a/drivers/pci/iov.c >> +++ b/drivers/pci/iov.c >> @@ -450,6 +450,7 @@ static int sriov_init(struct pci_dev *dev, int pos) >> iov->total_VFs = total; >> iov->pgsz = pgsz; >> iov->self = dev; >> + iov->probe_vfs = true; >> pci_read_config_dword(dev, pos + PCI_SRIOV_CAP, &iov->cap); >> pci_read_config_byte(dev, pos + PCI_SRIOV_FUNC_LINK, &iov->link); >> if (pci_pcie_type(dev) == PCI_EXP_TYPE_RC_END) >> diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c >> index afa7271..2a1cf84 100644 >> --- a/drivers/pci/pci-driver.c >> +++ b/drivers/pci/pci-driver.c >> @@ -394,6 +394,18 @@ void __weak pcibios_free_irq(struct pci_dev *dev) >> { >> } >> +#ifdef CONFIG_PCI_IOV >> +static inline bool pci_device_can_probe(struct pci_dev *pdev) >> +{ >> + return (!pdev->is_virtfn || pdev->physfn->sriov->probe_vfs); >> +} >> +#else >> +static inline bool pci_device_can_probe(struct pci_dev *pdev) >> +{ >> + return true; >> +} >> +#endif >> + >> static int pci_device_probe(struct device *dev) >> { >> int error; >> @@ -405,10 +417,12 @@ static int pci_device_probe(struct device *dev) >> return error; >> pci_dev_get(pci_dev); >> - error = __pci_device_probe(drv, pci_dev); >> - if (error) { >> - pcibios_free_irq(pci_dev); >> - pci_dev_put(pci_dev); >> + if (pci_device_can_probe(pci_dev)) { >> + error = __pci_device_probe(drv, pci_dev); >> + if (error) { >> + pcibios_free_irq(pci_dev); >> + pci_dev_put(pci_dev); >> + } >> } >> return error; >> diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c >> index 25d010d..1d5b89d 100644 >> --- a/drivers/pci/pci-sysfs.c >> +++ b/drivers/pci/pci-sysfs.c >> @@ -526,10 +526,37 @@ static ssize_t sriov_numvfs_store(struct device >> *dev, >> return count; >> } >> +static ssize_t sriov_probe_vfs_show(struct device *dev, >> + struct device_attribute *attr, >> + char *buf) >> +{ >> + struct pci_dev *pdev = to_pci_dev(dev); >> + >> + return sprintf(buf, "%u\n", pdev->sriov->probe_vfs); >> +} >> + >> +static ssize_t sriov_probe_vfs_store(struct device *dev, >> + struct device_attribute *attr, >> + const char *buf, size_t count) >> +{ >> + struct pci_dev *pdev = to_pci_dev(dev); >> + bool probe_vfs; >> + >> + if (kstrtobool(buf, &probe_vfs) < 0) >> + return -EINVAL; >> + >> + pdev->sriov->probe_vfs = probe_vfs; >> + >> + return count; >> +} >> + >> static struct device_attribute sriov_totalvfs_attr = >> __ATTR_RO(sriov_totalvfs); >> static struct device_attribute sriov_numvfs_attr = >> __ATTR(sriov_numvfs, (S_IRUGO|S_IWUSR|S_IWGRP), >> sriov_numvfs_show, sriov_numvfs_store); >> +static struct device_attribute sriov_probe_vfs_attr = >> + __ATTR(sriov_probe_vfs, (S_IRUGO|S_IWUSR|S_IWGRP), >> + sriov_probe_vfs_show, sriov_probe_vfs_store); >> #endif /* CONFIG_PCI_IOV */ >> static ssize_t driver_override_store(struct device *dev, >> @@ -1549,6 +1576,7 @@ static umode_t >> pci_dev_hp_attrs_are_visible(struct kobject *kobj, >> static struct attribute *sriov_dev_attrs[] = { >> &sriov_totalvfs_attr.attr, >> &sriov_numvfs_attr.attr, >> + &sriov_probe_vfs_attr.attr, >> NULL, >> }; >> diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h >> index 8dd38e6..b7d8127d 100644 >> --- a/drivers/pci/pci.h >> +++ b/drivers/pci/pci.h >> @@ -272,6 +272,7 @@ struct pci_sriov { >> struct pci_dev *self; /* this PF */ >> struct mutex lock; /* lock for setting sriov_numvfs in sysfs */ >> resource_size_t barsz[PCI_SRIOV_NUM_BARS]; /* VF BAR size */ >> + bool probe_vfs; /* control probing of VFs */ >> }; >> #ifdef CONFIG_PCI_ATS > > Hi Bjorn, could you please comment? > > Thanks. > Didn't see any comment on this patch. Is there any plan to pull this patch in? Thanks.