From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM02-BL2-obe.outbound.protection.outlook.com (mail-bl2nam02on0048.outbound.protection.outlook.com [104.47.38.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 1E347207E540E for ; Tue, 8 May 2018 09:50:41 -0700 (PDT) Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> From: =?UTF-8?Q?Christian_K=c3=b6nig?= Message-ID: Date: Tue, 8 May 2018 18:50:25 +0200 MIME-Version: 1.0 In-Reply-To: <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Logan Gunthorpe , Bjorn Helgaas , Alex Williamson Cc: Jens Axboe , Keith Busch , linux-nvdimm@lists.01.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Jason Gunthorpe , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-ID: QW0gMDguMDUuMjAxOCB1bSAxODoyNyBzY2hyaWViIExvZ2FuIEd1bnRob3JwZToKPgo+IE9uIDA4 LzA1LzE4IDAxOjE3IEFNLCBDaHJpc3RpYW4gS8O2bmlnIHdyb3RlOgo+PiBBTUQgQVBVcyBtYW5k YXRvcnkgbmVlZCB0aGUgQUNTIGZsYWcgc2V0IGZvciB0aGUgR1BVIGludGVncmF0ZWQgaW4gdGhl Cj4+IENQVSB3aGVuIElPTU1VIGlzIGVuYWJsZWQgb3Igb3RoZXJ3aXNlIHlvdSB3aWxsIGJyZWFr IFNWTS4KPiBXZWxsLCBnaXZlbiB0aGF0IHRoZSBjdXJyZW50IHNldCBvbmx5IGRpc2FibGVzIEFD UyBiaXRzIG9uIGJyaWRnZXMKPiAocHJldmlvdXMgdmVyc2lvbnMgd2VyZSBvbmx5IG9uIHN3aXRj aGVzKSB0aGlzIHNob3VsZG4ndCBiZSBhbiBpc3N1ZSBmb3IKPiBpbnRlZ3JhdGVkIGRldmljZXMu IFdlIGRvIG5vdCBkaXNhYmxlIEFDUyBmbGFncyBnbG9iYWxseS4KCk9rLCB0aGF0IGlzIGF0IGxl YXN0IGEgc3RlcCBpbiB0aGUgcmlnaHQgZGlyZWN0aW9uLiBCdXQgSSB0aGluayB3ZSAKc2VyaW91 c2x5IG5lZWQgdG8gdGVzdCB0aGF0IGZvciBzaWRlIGVmZmVjdHMuCgo+Cj4+IEFuZCB3aGF0IGV4 YWN0bHkgaXMgdGhlIHByb2JsZW0gaGVyZT8gSSdtIGN1cnJlbnRseSB0ZXN0aW5nIFAyUCB3aXRo Cj4+IEdQVXMgaW4gZGlmZmVyZW50IElPTU1VIGRvbWFpbnMgYW5kIGF0IGxlYXN0IHdpdGggQU1E IElPTU1VcyB0aGF0IHdvcmtzCj4+IHBlcmZlY3RseSBmaW5lLgo+IEluIGFkZGl0aW9uIHRvIFN0 ZXBoZW4ncyBjb21tZW50cywgc2VlaW5nIHdlJ3ZlIGVzdGFibGlzaGVkIGEgZ2VuZXJhbAo+IG5l ZWQgdG8gYXZvaWQgdGhlIHJvb3QgY29tcGxleCAodW50aWwgd2UgaGF2ZSBhIHdoaXRlbGlzdCBh dCBsZWFzdCkgd2UKPiBtdXN0IGhhdmUgQUNTIGRpc2FibGVkIGFsb25nIHRoZSBwYXRoIGJldHdl ZW4gdGhlIGRldmljZXMuIE90aGVyd2lzZSwKPiBhbGwgVExQcyB3aWxsIGdvIHRocm91Z2ggdGhl IHJvb3QgY29tcGxleCBhbmQgaWYgdGhlcmUgaXMgbm8gc3VwcG9ydCBpdAo+IHdpbGwgZmFpbC4K CldlbGwgSSdtIG5vdCBhbiBleHBlcnQgb24gdGhpcywgYnV0IGlmIEknbSBub3QgY29tcGxldGVs eSBtaXN0YWtlbiB0aGF0IAppcyBub3QgY29ycmVjdC4KCkUuZy4gdHJhbnNhY3Rpb25zIGFyZSBp bml0aWFsbHkgc2VuZCB0byB0aGUgcm9vdCBjb21wbGV4IGZvciAKdHJhbnNsYXRpb24sIHRoYXQn cyBmb3Igc3VyZS4gQnV0IGF0IGxlYXN0IGZvciBBTUQgR1BVcyB0aGUgcm9vdCBjb21wbGV4IAph bnN3ZXJzIHdpdGggdGhlIHRyYW5zbGF0ZWQgYWRkcmVzcyB3aGljaCBpcyB0aGVuIGNhY2hlZCBp biB0aGUgZGV2aWNlLgoKU28gZnVydGhlciB0cmFuc2FjdGlvbnMgZm9yIHRoZSBzYW1lIGFkZHJl c3MgcmFuZ2UgdGhlbiBnbyBkaXJlY3RseSB0byAKdGhlIGRlc3RpbmF0aW9uLgoKV2hhdCB5b3Ug ZG9uJ3Qgd2FudCBpcyBkZXZpY2UgaXNvbGF0aW9uLCBjYXVzZSBpbiB0aGlzIGNhc2UgdGhlIHJv b3QgCmNvbXBsZXggaGFuZGxlcyB0aGUgdHJhbnNhY3Rpb24gdGhlbXNlbHZlcy4gSUlSQyB0aGVy ZSB3aGVyZSBhbHNvIApzb21ldGhpbmcgbGlrZSAiZm9yY2VfaXNvbGF0aW9uIiBhbmQgIm5vYnlw YXNzIiBwYXJhbWV0ZXJzIGZvciB0aGUgSU9NTVUgCnRvIGNvbnRyb2wgdGhhdCBiZWhhdmlvci4K Ckl0J3MgYWxyZWFkeSBsYXRlIGhlcmUsIGJ1dCBnb2luZyB0byBkaWcgdXAgdGhlIGRvY3VtZW50 YXRpb24gZm9yIHRoYXQgCnRvbW9ycm93IGFuZC9vciBjb250YWN0IGEgaGFyZHdhcmUgZW5naW5l ZXIgaW52b2x2ZWQgaW4gdGhlIEFDUyBzcGVjLgoKUmVnYXJkcywKQ2hyaXN0aWFuLgoKPgo+IElm IHRoZSBjb25zZW5zdXMgaXMgd2Ugd2FudCBhIGNvbW1hbmQgbGluZSBvcHRpb24sIHRoZW4gc28g YmUgaXQuIEJ1dAo+IHdlJ2xsIGhhdmUgdG8gZGVueSBwcmV0dHkgbXVjaCBhbGwgUDJQIHRyYW5z YWN0aW9ucyB1bmxlc3MgdGhlIHVzZXIKPiBjb3JyZWN0bHkgZGlzYWJsZXMgQUNTIGFsb25nIHRo ZSBwYXRoIHVzaW5nIHRoZSBjb21tYW5kIGxpbmUgb3B0aW9uIGFuZAo+IHRoaXMgaXMgcmVhbGx5 IGFubm95aW5nIGZvciB1c2VycyBvZiB0aGlzIGZ1bmN0aW9uYWxpdHkgdG8gdW5kZXJzdGFuZAo+ IGhvdyB0byBkbyB0aGF0IGNvcnJlY3RseS4KPgo+IExvZ2FuCgpfX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fXwpMaW51eC1udmRpbW0gbWFpbGluZyBsaXN0Ckxp bnV4LW52ZGltbUBsaXN0cy4wMS5vcmcKaHR0cHM6Ly9saXN0cy4wMS5vcmcvbWFpbG1hbi9saXN0 aW5mby9saW51eC1udmRpbW0K From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches To: Logan Gunthorpe , Bjorn Helgaas , Alex Williamson Cc: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@lists.01.org, linux-block@vger.kernel.org, Stephen Bates , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Benjamin Herrenschmidt References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> From: =?UTF-8?Q?Christian_K=c3=b6nig?= Message-ID: Date: Tue, 8 May 2018 18:50:25 +0200 MIME-Version: 1.0 In-Reply-To: <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> Content-Type: text/plain; charset=utf-8; format=flowed Return-Path: christian.koenig@amd.com List-ID: Am 08.05.2018 um 18:27 schrieb Logan Gunthorpe: > > On 08/05/18 01:17 AM, Christian König wrote: >> AMD APUs mandatory need the ACS flag set for the GPU integrated in the >> CPU when IOMMU is enabled or otherwise you will break SVM. > Well, given that the current set only disables ACS bits on bridges > (previous versions were only on switches) this shouldn't be an issue for > integrated devices. We do not disable ACS flags globally. Ok, that is at least a step in the right direction. But I think we seriously need to test that for side effects. > >> And what exactly is the problem here? I'm currently testing P2P with >> GPUs in different IOMMU domains and at least with AMD IOMMUs that works >> perfectly fine. > In addition to Stephen's comments, seeing we've established a general > need to avoid the root complex (until we have a whitelist at least) we > must have ACS disabled along the path between the devices. Otherwise, > all TLPs will go through the root complex and if there is no support it > will fail. Well I'm not an expert on this, but if I'm not completely mistaken that is not correct. E.g. transactions are initially send to the root complex for translation, that's for sure. But at least for AMD GPUs the root complex answers with the translated address which is then cached in the device. So further transactions for the same address range then go directly to the destination. What you don't want is device isolation, cause in this case the root complex handles the transaction themselves. IIRC there where also something like "force_isolation" and "nobypass" parameters for the IOMMU to control that behavior. It's already late here, but going to dig up the documentation for that tomorrow and/or contact a hardware engineer involved in the ACS spec. Regards, Christian. > > If the consensus is we want a command line option, then so be it. But > we'll have to deny pretty much all P2P transactions unless the user > correctly disables ACS along the path using the command line option and > this is really annoying for users of this functionality to understand > how to do that correctly. > > Logan From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?UTF-8?Q?Christian_K=c3=b6nig?= Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Date: Tue, 8 May 2018 18:50:25 +0200 Message-ID: References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8"; Format="flowed" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <0b4183ef-e720-204b-9e85-b9eaf7a4136a-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org> Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Logan Gunthorpe , Bjorn Helgaas , Alex Williamson Cc: Jens Axboe , Keith Busch , linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Jason Gunthorpe , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-Id: linux-rdma@vger.kernel.org QW0gMDguMDUuMjAxOCB1bSAxODoyNyBzY2hyaWViIExvZ2FuIEd1bnRob3JwZToKPgo+IE9uIDA4 LzA1LzE4IDAxOjE3IEFNLCBDaHJpc3RpYW4gS8O2bmlnIHdyb3RlOgo+PiBBTUQgQVBVcyBtYW5k YXRvcnkgbmVlZCB0aGUgQUNTIGZsYWcgc2V0IGZvciB0aGUgR1BVIGludGVncmF0ZWQgaW4gdGhl Cj4+IENQVSB3aGVuIElPTU1VIGlzIGVuYWJsZWQgb3Igb3RoZXJ3aXNlIHlvdSB3aWxsIGJyZWFr IFNWTS4KPiBXZWxsLCBnaXZlbiB0aGF0IHRoZSBjdXJyZW50IHNldCBvbmx5IGRpc2FibGVzIEFD UyBiaXRzIG9uIGJyaWRnZXMKPiAocHJldmlvdXMgdmVyc2lvbnMgd2VyZSBvbmx5IG9uIHN3aXRj aGVzKSB0aGlzIHNob3VsZG4ndCBiZSBhbiBpc3N1ZSBmb3IKPiBpbnRlZ3JhdGVkIGRldmljZXMu IFdlIGRvIG5vdCBkaXNhYmxlIEFDUyBmbGFncyBnbG9iYWxseS4KCk9rLCB0aGF0IGlzIGF0IGxl YXN0IGEgc3RlcCBpbiB0aGUgcmlnaHQgZGlyZWN0aW9uLiBCdXQgSSB0aGluayB3ZSAKc2VyaW91 c2x5IG5lZWQgdG8gdGVzdCB0aGF0IGZvciBzaWRlIGVmZmVjdHMuCgo+Cj4+IEFuZCB3aGF0IGV4 YWN0bHkgaXMgdGhlIHByb2JsZW0gaGVyZT8gSSdtIGN1cnJlbnRseSB0ZXN0aW5nIFAyUCB3aXRo Cj4+IEdQVXMgaW4gZGlmZmVyZW50IElPTU1VIGRvbWFpbnMgYW5kIGF0IGxlYXN0IHdpdGggQU1E IElPTU1VcyB0aGF0IHdvcmtzCj4+IHBlcmZlY3RseSBmaW5lLgo+IEluIGFkZGl0aW9uIHRvIFN0 ZXBoZW4ncyBjb21tZW50cywgc2VlaW5nIHdlJ3ZlIGVzdGFibGlzaGVkIGEgZ2VuZXJhbAo+IG5l ZWQgdG8gYXZvaWQgdGhlIHJvb3QgY29tcGxleCAodW50aWwgd2UgaGF2ZSBhIHdoaXRlbGlzdCBh dCBsZWFzdCkgd2UKPiBtdXN0IGhhdmUgQUNTIGRpc2FibGVkIGFsb25nIHRoZSBwYXRoIGJldHdl ZW4gdGhlIGRldmljZXMuIE90aGVyd2lzZSwKPiBhbGwgVExQcyB3aWxsIGdvIHRocm91Z2ggdGhl IHJvb3QgY29tcGxleCBhbmQgaWYgdGhlcmUgaXMgbm8gc3VwcG9ydCBpdAo+IHdpbGwgZmFpbC4K CldlbGwgSSdtIG5vdCBhbiBleHBlcnQgb24gdGhpcywgYnV0IGlmIEknbSBub3QgY29tcGxldGVs eSBtaXN0YWtlbiB0aGF0IAppcyBub3QgY29ycmVjdC4KCkUuZy4gdHJhbnNhY3Rpb25zIGFyZSBp bml0aWFsbHkgc2VuZCB0byB0aGUgcm9vdCBjb21wbGV4IGZvciAKdHJhbnNsYXRpb24sIHRoYXQn cyBmb3Igc3VyZS4gQnV0IGF0IGxlYXN0IGZvciBBTUQgR1BVcyB0aGUgcm9vdCBjb21wbGV4IAph bnN3ZXJzIHdpdGggdGhlIHRyYW5zbGF0ZWQgYWRkcmVzcyB3aGljaCBpcyB0aGVuIGNhY2hlZCBp biB0aGUgZGV2aWNlLgoKU28gZnVydGhlciB0cmFuc2FjdGlvbnMgZm9yIHRoZSBzYW1lIGFkZHJl c3MgcmFuZ2UgdGhlbiBnbyBkaXJlY3RseSB0byAKdGhlIGRlc3RpbmF0aW9uLgoKV2hhdCB5b3Ug ZG9uJ3Qgd2FudCBpcyBkZXZpY2UgaXNvbGF0aW9uLCBjYXVzZSBpbiB0aGlzIGNhc2UgdGhlIHJv b3QgCmNvbXBsZXggaGFuZGxlcyB0aGUgdHJhbnNhY3Rpb24gdGhlbXNlbHZlcy4gSUlSQyB0aGVy ZSB3aGVyZSBhbHNvIApzb21ldGhpbmcgbGlrZSAiZm9yY2VfaXNvbGF0aW9uIiBhbmQgIm5vYnlw YXNzIiBwYXJhbWV0ZXJzIGZvciB0aGUgSU9NTVUgCnRvIGNvbnRyb2wgdGhhdCBiZWhhdmlvci4K Ckl0J3MgYWxyZWFkeSBsYXRlIGhlcmUsIGJ1dCBnb2luZyB0byBkaWcgdXAgdGhlIGRvY3VtZW50 YXRpb24gZm9yIHRoYXQgCnRvbW9ycm93IGFuZC9vciBjb250YWN0IGEgaGFyZHdhcmUgZW5naW5l ZXIgaW52b2x2ZWQgaW4gdGhlIEFDUyBzcGVjLgoKUmVnYXJkcywKQ2hyaXN0aWFuLgoKPgo+IElm IHRoZSBjb25zZW5zdXMgaXMgd2Ugd2FudCBhIGNvbW1hbmQgbGluZSBvcHRpb24sIHRoZW4gc28g YmUgaXQuIEJ1dAo+IHdlJ2xsIGhhdmUgdG8gZGVueSBwcmV0dHkgbXVjaCBhbGwgUDJQIHRyYW5z YWN0aW9ucyB1bmxlc3MgdGhlIHVzZXIKPiBjb3JyZWN0bHkgZGlzYWJsZXMgQUNTIGFsb25nIHRo ZSBwYXRoIHVzaW5nIHRoZSBjb21tYW5kIGxpbmUgb3B0aW9uIGFuZAo+IHRoaXMgaXMgcmVhbGx5 IGFubm95aW5nIGZvciB1c2VycyBvZiB0aGlzIGZ1bmN0aW9uYWxpdHkgdG8gdW5kZXJzdGFuZAo+ IGhvdyB0byBkbyB0aGF0IGNvcnJlY3RseS4KPgo+IExvZ2FuCgpfX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fXwpMaW51eC1udmRpbW0gbWFpbGluZyBsaXN0Ckxp bnV4LW52ZGltbUBsaXN0cy4wMS5vcmcKaHR0cHM6Ly9saXN0cy4wMS5vcmcvbWFpbG1hbi9saXN0 aW5mby9saW51eC1udmRpbW0K From mboxrd@z Thu Jan 1 00:00:00 1970 From: christian.koenig@amd.com (=?UTF-8?Q?Christian_K=c3=b6nig?=) Date: Tue, 8 May 2018 18:50:25 +0200 Subject: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches In-Reply-To: <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> Message-ID: Am 08.05.2018 um 18:27 schrieb Logan Gunthorpe: > > On 08/05/18 01:17 AM, Christian K?nig wrote: >> AMD APUs mandatory need the ACS flag set for the GPU integrated in the >> CPU when IOMMU is enabled or otherwise you will break SVM. > Well, given that the current set only disables ACS bits on bridges > (previous versions were only on switches) this shouldn't be an issue for > integrated devices. We do not disable ACS flags globally. Ok, that is at least a step in the right direction. But I think we seriously need to test that for side effects. > >> And what exactly is the problem here? I'm currently testing P2P with >> GPUs in different IOMMU domains and at least with AMD IOMMUs that works >> perfectly fine. > In addition to Stephen's comments, seeing we've established a general > need to avoid the root complex (until we have a whitelist at least) we > must have ACS disabled along the path between the devices. Otherwise, > all TLPs will go through the root complex and if there is no support it > will fail. Well I'm not an expert on this, but if I'm not completely mistaken that is not correct. E.g. transactions are initially send to the root complex for translation, that's for sure. But at least for AMD GPUs the root complex answers with the translated address which is then cached in the device. So further transactions for the same address range then go directly to the destination. What you don't want is device isolation, cause in this case the root complex handles the transaction themselves. IIRC there where also something like "force_isolation" and "nobypass" parameters for the IOMMU to control that behavior. It's already late here, but going to dig up the documentation for that tomorrow and/or contact a hardware engineer involved in the ACS spec. Regards, Christian. > > If the consensus is we want a command line option, then so be it. But > we'll have to deny pretty much all P2P transactions unless the user > correctly disables ACS along the path using the command line option and > this is really annoying for users of this functionality to understand > how to do that correctly. > > Logan