From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660094.outbound.protection.outlook.com [40.107.66.94]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id CF64B2034862E for ; Wed, 9 May 2018 05:38:54 -0700 (PDT) From: "Stephen Bates" Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Date: Wed, 9 May 2018 12:38:53 +0000 Message-ID: References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> <3584a6ac-95c7-5d23-1859-aee30605776e@deltatee.com> <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508144341.0441b676@w520.home> <20180508152631.50fd583c@w520.home> <354F7407-0DC7-470C-B9AA-74FDF9C46B08@raithlin.com> <20180508160336.0935ddde@w520.home> <20905682-9440-7d4b-0260-99d3dc794c3d@deltatee.com> <20180508163206.7d3bf383@w520.home> <2ce7908d-1422-55f8-aeba-94ccce2cb484@deltatee.com> In-Reply-To: <2ce7908d-1422-55f8-aeba-94ccce2cb484@deltatee.com> Content-Language: en-US Content-ID: MIME-Version: 1.0 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Logan Gunthorpe , Dan Williams , Alex Williamson Cc: Jens Axboe , Keith Busch , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-nvme@lists.infradead.org" , Christoph Hellwig , "linux-block@vger.kernel.org" , =?utf-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Jason Gunthorpe , Bjorn Helgaas , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= List-ID: Hi Logan > Yeah, I'm having a hard time coming up with an easy enough solution for > the user. I agree with Dan though, the bus renumbering risk would be > fairly low in the custom hardware seeing the switches are likely going > to be directly soldered to the same board with the CPU. I am afraid that soldered down assumption may not be valid. More and more PCIe cards with PCIe switches on them are becoming available and people are using these to connect servers to arrays of NVMe SSDs which may make the topology more dynamic. Stephen _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: "Stephen Bates" To: Logan Gunthorpe , Dan Williams , Alex Williamson CC: =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , "Bjorn Helgaas" , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "linux-block@vger.kernel.org" , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , =?utf-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Benjamin Herrenschmidt Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Date: Wed, 9 May 2018 12:38:53 +0000 Message-ID: References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> <3584a6ac-95c7-5d23-1859-aee30605776e@deltatee.com> <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508144341.0441b676@w520.home> <20180508152631.50fd583c@w520.home> <354F7407-0DC7-470C-B9AA-74FDF9C46B08@raithlin.com> <20180508160336.0935ddde@w520.home> <20905682-9440-7d4b-0260-99d3dc794c3d@deltatee.com> <20180508163206.7d3bf383@w520.home> <2ce7908d-1422-55f8-aeba-94ccce2cb484@deltatee.com> In-Reply-To: <2ce7908d-1422-55f8-aeba-94ccce2cb484@deltatee.com> Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 List-ID: SGkgTG9nYW4NCg0KPiAgICBZZWFoLCBJJ20gaGF2aW5nIGEgaGFyZCB0aW1lIGNvbWluZyB1cCB3 aXRoIGFuIGVhc3kgZW5vdWdoIHNvbHV0aW9uIGZvcg0KPiAgICB0aGUgdXNlci4gSSBhZ3JlZSB3 aXRoIERhbiB0aG91Z2gsIHRoZSBidXMgcmVudW1iZXJpbmcgcmlzayB3b3VsZCBiZQ0KPiAgICBm YWlybHkgbG93IGluIHRoZSBjdXN0b20gaGFyZHdhcmUgc2VlaW5nIHRoZSBzd2l0Y2hlcyBhcmUg bGlrZWx5IGdvaW5nDQo+ICAgIHRvIGJlIGRpcmVjdGx5IHNvbGRlcmVkIHRvIHRoZSBzYW1lIGJv YXJkIHdpdGggdGhlIENQVS4NCiAgICANCkkgYW0gYWZyYWlkIHRoYXQgc29sZGVyZWQgZG93biBh c3N1bXB0aW9uIG1heSBub3QgYmUgdmFsaWQuIE1vcmUgYW5kIG1vcmUgUENJZSBjYXJkcyB3aXRo IFBDSWUgc3dpdGNoZXMgb24gdGhlbSBhcmUgYmVjb21pbmcgYXZhaWxhYmxlIGFuZCBwZW9wbGUg YXJlIHVzaW5nIHRoZXNlIHRvIGNvbm5lY3Qgc2VydmVycyB0byBhcnJheXMgb2YgTlZNZSBTU0Rz IHdoaWNoIG1heSBtYWtlIHRoZSB0b3BvbG9neSBtb3JlIGR5bmFtaWMuDQoNClN0ZXBoZW4NCg0K From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Stephen Bates" Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Date: Wed, 9 May 2018 12:38:53 +0000 Message-ID: References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> <3584a6ac-95c7-5d23-1859-aee30605776e@deltatee.com> <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508144341.0441b676@w520.home> <20180508152631.50fd583c@w520.home> <354F7407-0DC7-470C-B9AA-74FDF9C46B08@raithlin.com> <20180508160336.0935ddde@w520.home> <20905682-9440-7d4b-0260-99d3dc794c3d@deltatee.com> <20180508163206.7d3bf383@w520.home> <2ce7908d-1422-55f8-aeba-94ccce2cb484@deltatee.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <2ce7908d-1422-55f8-aeba-94ccce2cb484-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org> Content-Language: en-US Content-ID: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Logan Gunthorpe , Dan Williams , Alex Williamson Cc: Jens Axboe , Keith Busch , "linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org" , "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org" , Christoph Hellwig , "linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , =?utf-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Jason Gunthorpe , Bjorn Helgaas , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= List-Id: linux-rdma@vger.kernel.org Hi Logan > Yeah, I'm having a hard time coming up with an easy enough solution for > the user. I agree with Dan though, the bus renumbering risk would be > fairly low in the custom hardware seeing the switches are likely going > to be directly soldered to the same board with the CPU. I am afraid that soldered down assumption may not be valid. More and more PCIe cards with PCIe switches on them are becoming available and people are using these to connect servers to arrays of NVMe SSDs which may make the topology more dynamic. Stephen From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934710AbeEIMi6 (ORCPT ); Wed, 9 May 2018 08:38:58 -0400 Received: from mail-eopbgr660129.outbound.protection.outlook.com ([40.107.66.129]:27669 "EHLO CAN01-QB1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932875AbeEIMiz (ORCPT ); Wed, 9 May 2018 08:38:55 -0400 From: "Stephen Bates" To: Logan Gunthorpe , Dan Williams , Alex Williamson CC: =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , "Bjorn Helgaas" , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "linux-block@vger.kernel.org" , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , =?utf-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Benjamin Herrenschmidt Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Thread-Topic: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Thread-Index: AQHT21soiBVnp6SJuEuLCiH6kBzTLKQk+zIAgACHcYCAAJmxAIAABkiAgAAoBgCAAAW2gIAAA0YAgAAHvYCAAAGOgIAABt+AgAABmICAAApggP//n96AgABqfgCAAAHhgIAABhYAgAAH84CAAAQigIAAe+sA Date: Wed, 9 May 2018 12:38:53 +0000 Message-ID: References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> <3584a6ac-95c7-5d23-1859-aee30605776e@deltatee.com> <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508144341.0441b676@w520.home> <20180508152631.50fd583c@w520.home> <354F7407-0DC7-470C-B9AA-74FDF9C46B08@raithlin.com> <20180508160336.0935ddde@w520.home> <20905682-9440-7d4b-0260-99d3dc794c3d@deltatee.com> <20180508163206.7d3bf383@w520.home> <2ce7908d-1422-55f8-aeba-94ccce2cb484@deltatee.com> In-Reply-To: <2ce7908d-1422-55f8-aeba-94ccce2cb484@deltatee.com> Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/10.d.0.180505 authentication-results: spf=none (sender IP is ) smtp.mailfrom=sbates@raithlin.com; x-originating-ip: [70.65.250.31] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;YTOPR0101MB2202;7:/YJ/AGrM/ao0SiG4+EJqadBdJvl1YTpWO2WN+yv3S6GM/nl7hthqZt04PxS09v9IoKXTfsFHaXaDnGO64citkmt88pF0RjxTrPV3pwSemwgvdzLG/qwyw4zoRIFI+lV7EG9ZHhElro35zMpVHltMf6fHUVxVqdaBWssNYfW1RdbTc0z11ViK1qtRaEts+3kzU/TUFmOG8HhM8hLCpLFSLMZzs2ONlYTLCqImoFEhZqmpxOLaXzVI1BudGHbeNnuJ x-ms-exchange-antispam-srfa-diagnostics: SOS; x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(7020095)(4652020)(7021125)(5600026)(4534165)(7022125)(4603075)(4627221)(201702281549075)(7048125)(7024125)(7027125)(7028125)(7023125)(2017052603328)(7153060)(7193020);SRVR:YTOPR0101MB2202; x-ms-traffictypediagnostic: YTOPR0101MB2202: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(21532816269658); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(6040522)(2401047)(8121501046)(5005006)(3002001)(10201501046)(3231254)(944501410)(52105095)(93006095)(93001095)(149027)(150027)(6041310)(20161123558120)(2016111802025)(20161123564045)(20161123562045)(20161123560045)(6072148)(6043046)(201708071742011);SRVR:YTOPR0101MB2202;BCL:0;PCL:0;RULEID:;SRVR:YTOPR0101MB2202; x-forefront-prvs: 0667289FF8 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(39380400002)(346002)(376002)(366004)(39830400003)(396003)(189003)(199004)(6512007)(97736004)(8666007)(8676002)(86362001)(33656002)(3660700001)(3280700002)(2906002)(81156014)(8936002)(14454004)(81166006)(58126008)(53936002)(106356001)(316002)(4326008)(105586002)(110136005)(305945005)(68736007)(7736002)(36756003)(93886005)(54906003)(26005)(446003)(186003)(76176011)(478600001)(102836004)(59450400001)(82746002)(486006)(2616005)(11346002)(6246003)(25786009)(7416002)(2900100001)(6506007)(5660300001)(476003)(6116002)(6486002)(83716003)(3846002)(66066001)(5250100002)(99286004)(229853002)(6436002);DIR:OUT;SFP:1102;SCL:1;SRVR:YTOPR0101MB2202;H:YTOPR0101MB2043.CANPRD01.PROD.OUTLOOK.COM;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; x-microsoft-antispam-message-info: 4Uqa89nHtTRYPj475iZ0wd+4PnGWWCdYHE4J+wlz4KwThMtzo1aTE0rDuCXN2BjcNsLuh0JHddl1/GKsFfFbMN2h48IPHubBQx0jD3vi8yVcqB+VFLb/k0PppI/yUtS2r8ILhULCVNL1cRL8mSN70qioymGYN+3iSB/07H73QF9Z6L2Qxdg9aK8QjpQSkWQt spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: a657985d-0624-4eb5-0714-08d5b5a9d2a6 X-OriginatorOrg: raithlin.com X-MS-Exchange-CrossTenant-Network-Message-Id: a657985d-0624-4eb5-0714-08d5b5a9d2a6 X-MS-Exchange-CrossTenant-originalarrivaltime: 09 May 2018 12:38:53.0364 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 18519031-7ff4-4cbb-bbcb-c3252d330f4b X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB2202 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id w49Cd3kt004356 Hi Logan > Yeah, I'm having a hard time coming up with an easy enough solution for > the user. I agree with Dan though, the bus renumbering risk would be > fairly low in the custom hardware seeing the switches are likely going > to be directly soldered to the same board with the CPU. I am afraid that soldered down assumption may not be valid. More and more PCIe cards with PCIe switches on them are becoming available and people are using these to connect servers to arrays of NVMe SSDs which may make the topology more dynamic. Stephen From mboxrd@z Thu Jan 1 00:00:00 1970 From: sbates@raithlin.com (Stephen Bates) Date: Wed, 9 May 2018 12:38:53 +0000 Subject: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches In-Reply-To: <2ce7908d-1422-55f8-aeba-94ccce2cb484@deltatee.com> References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> <3584a6ac-95c7-5d23-1859-aee30605776e@deltatee.com> <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508144341.0441b676@w520.home> <20180508152631.50fd583c@w520.home> <354F7407-0DC7-470C-B9AA-74FDF9C46B08@raithlin.com> <20180508160336.0935ddde@w520.home> <20905682-9440-7d4b-0260-99d3dc794c3d@deltatee.com> <20180508163206.7d3bf383@w520.home> <2ce7908d-1422-55f8-aeba-94ccce2cb484@deltatee.com> Message-ID: Hi Logan > Yeah, I'm having a hard time coming up with an easy enough solution for > the user. I agree with Dan though, the bus renumbering risk would be > fairly low in the custom hardware seeing the switches are likely going > to be directly soldered to the same board with the CPU. I am afraid that soldered down assumption may not be valid. More and more PCIe cards with PCIe switches on them are becoming available and people are using these to connect servers to arrays of NVMe SSDs which may make the topology more dynamic. Stephen