From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660137.outbound.protection.outlook.com [40.107.66.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id D1FCB202E5CCE for ; Tue, 8 May 2018 07:25:12 -0700 (PDT) From: "Stephen Bates" Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Date: Tue, 8 May 2018 14:25:10 +0000 Message-ID: <479F2F93-903B-4A8A-8989-A4BD0E5E1471@raithlin.com> References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> In-Reply-To: Content-Language: en-US Content-ID: <3CD50565E70F874B8651DD7C38C3ECA2@CANPRD01.PROD.OUTLOOK.COM> MIME-Version: 1.0 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , Bjorn Cc: Jens Axboe , Keith Busch , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-block@vger.kernel.org" , =?utf-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Jason Gunthorpe , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-ID: Hi Christian > AMD APUs mandatory need the ACS flag set for the GPU integrated in the > CPU when IOMMU is enabled or otherwise you will break SVM. OK but in this case aren't you losing (many of) the benefits of P2P since all DMAs will now get routed up to the IOMMU before being passed down to the destination PCIe EP? > Similar problems arise when you do this for dedicated GPU, but we > haven't upstreamed the support for this yet. Hmm, as above. With ACS enabled on all downstream ports any P2P enabled DMA will be routed to the IOMMU which removes a lot of the benefit. > So that is a clear NAK from my side for the approach. Do you have an alternative? This is the approach we arrived it after a reasonably lengthy discussion on the mailing lists. Alex, are you still comfortable with this approach? > And what exactly is the problem here? We had a pretty lengthy discussion on this topic on one of the previous revisions. The issue is that currently there is no mechanism in the IOMMU code to inform VMs if IOMMU groupings change. Since p2pdma can dynamically change its topology (due to PCI hotplug) we had to be cognizant of the fact that ACS settings could change. Since there is no way to currently handle changing ACS settings and hence IOMMU groupings the consensus was to simply disable ACS on all ports in a p2pdma domain. This effectively makes all the devices in the p2pdma domain part of the same IOMMU grouping. The plan will be to address this in time and add a mechanism for IOMMU grouping changes and notification to VMs but that's not part of this series. Note you are still allowed to have ACS functioning on other PCI domains so if you do not a plurality of IOMMU groupings you can still achieve it (but you can't do p2pdma across IOMMU groupings, which is safe). > I'm currently testing P2P with GPUs in different IOMMU domains and at least with AMD IOMMUs that works perfectly fine. Yup that should work though again I have to ask are you disabling ACS on the ports between the two peer devices to get the p2p benefit? If not you are not getting all the performance benefit (due to IOMMU routing), if you are then there are obviously security implications between those IOMMU domains if they are assigned to different VMs. And now the issue is if new devices are added and the p2p topology needed to change there would be no way to inform the VMs of any IOMMU group change. Cheers Stephen _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: "Stephen Bates" To: =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , "Bjorn Helgaas" , Logan Gunthorpe , "Alex Williamson" CC: "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "linux-block@vger.kernel.org" , "Christoph Hellwig" , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , =?utf-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Benjamin Herrenschmidt Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Date: Tue, 8 May 2018 14:25:10 +0000 Message-ID: <479F2F93-903B-4A8A-8989-A4BD0E5E1471@raithlin.com> References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> In-Reply-To: Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 List-ID: ICAgIA0KSGkgQ2hyaXN0aWFuDQoNCj4gQU1EIEFQVXMgbWFuZGF0b3J5IG5lZWQgdGhlIEFDUyBm bGFnIHNldCBmb3IgdGhlIEdQVSBpbnRlZ3JhdGVkIGluIHRoZSANCj4gQ1BVIHdoZW4gSU9NTVUg aXMgZW5hYmxlZCBvciBvdGhlcndpc2UgeW91IHdpbGwgYnJlYWsgU1ZNLg0KDQpPSyBidXQgaW4g dGhpcyBjYXNlIGFyZW4ndCB5b3UgbG9zaW5nIChtYW55IG9mKSB0aGUgYmVuZWZpdHMgb2YgUDJQ IHNpbmNlIGFsbCBETUFzIHdpbGwgbm93IGdldCByb3V0ZWQgdXAgdG8gdGhlIElPTU1VIGJlZm9y ZSBiZWluZyBwYXNzZWQgZG93biB0byB0aGUgZGVzdGluYXRpb24gUENJZSBFUD8NCg0KPiBTaW1p bGFyIHByb2JsZW1zIGFyaXNlIHdoZW4geW91IGRvIHRoaXMgZm9yIGRlZGljYXRlZCBHUFUsIGJ1 dCB3ZSANCj4gaGF2ZW4ndCB1cHN0cmVhbWVkIHRoZSBzdXBwb3J0IGZvciB0aGlzIHlldC4NCg0K SG1tLCBhcyBhYm92ZS4gV2l0aCBBQ1MgZW5hYmxlZCBvbiBhbGwgZG93bnN0cmVhbSBwb3J0cyBh bnkgUDJQIGVuYWJsZWQgRE1BIHdpbGwgYmUgcm91dGVkIHRvIHRoZSBJT01NVSB3aGljaCByZW1v dmVzIGEgbG90IG9mIHRoZSBiZW5lZml0LiANCiAgICANCj4gU28gdGhhdCBpcyBhIGNsZWFyIE5B SyBmcm9tIG15IHNpZGUgZm9yIHRoZSBhcHByb2FjaC4NCg0KRG8geW91IGhhdmUgYW4gYWx0ZXJu YXRpdmU/IFRoaXMgaXMgdGhlIGFwcHJvYWNoIHdlIGFycml2ZWQgaXQgYWZ0ZXIgYSByZWFzb25h Ymx5IGxlbmd0aHkgZGlzY3Vzc2lvbiBvbiB0aGUgbWFpbGluZyBsaXN0cy4gQWxleCwgYXJlIHlv dSBzdGlsbCBjb21mb3J0YWJsZSB3aXRoIHRoaXMgYXBwcm9hY2g/DQogICAgDQo+IEFuZCB3aGF0 IGV4YWN0bHkgaXMgdGhlIHByb2JsZW0gaGVyZT8NCiANCldlIGhhZCBhIHByZXR0eSBsZW5ndGh5 IGRpc2N1c3Npb24gb24gdGhpcyB0b3BpYyBvbiBvbmUgb2YgdGhlIHByZXZpb3VzIHJldmlzaW9u cy4gVGhlIGlzc3VlIGlzIHRoYXQgY3VycmVudGx5IHRoZXJlIGlzIG5vIG1lY2hhbmlzbSBpbiB0 aGUgSU9NTVUgY29kZSB0byBpbmZvcm0gVk1zIGlmIElPTU1VIGdyb3VwaW5ncyBjaGFuZ2UuIFNp bmNlIHAycGRtYSBjYW4gZHluYW1pY2FsbHkgY2hhbmdlIGl0cyB0b3BvbG9neSAoZHVlIHRvIFBD SSBob3RwbHVnKSB3ZSBoYWQgdG8gYmUgY29nbml6YW50IG9mIHRoZSBmYWN0IHRoYXQgQUNTIHNl dHRpbmdzIGNvdWxkIGNoYW5nZS4gU2luY2UgdGhlcmUgaXMgbm8gd2F5IHRvIGN1cnJlbnRseSBo YW5kbGUgY2hhbmdpbmcgQUNTIHNldHRpbmdzIGFuZCBoZW5jZSBJT01NVSBncm91cGluZ3MgdGhl IGNvbnNlbnN1cyB3YXMgdG8gc2ltcGx5IGRpc2FibGUgQUNTIG9uIGFsbCBwb3J0cyBpbiBhIHAy cGRtYSBkb21haW4uIFRoaXMgZWZmZWN0aXZlbHkgbWFrZXMgYWxsIHRoZSBkZXZpY2VzIGluIHRo ZSBwMnBkbWEgZG9tYWluIHBhcnQgb2YgdGhlIHNhbWUgSU9NTVUgZ3JvdXBpbmcuIFRoZSBwbGFu IHdpbGwgYmUgdG8gYWRkcmVzcyB0aGlzIGluIHRpbWUgYW5kIGFkZCBhIG1lY2hhbmlzbSBmb3Ig SU9NTVUgZ3JvdXBpbmcgY2hhbmdlcyBhbmQgbm90aWZpY2F0aW9uIHRvIFZNcyBidXQgdGhhdCdz IG5vdCBwYXJ0IG9mIHRoaXMgc2VyaWVzLiBOb3RlIHlvdSBhcmUgc3RpbGwgYWxsb3dlZCB0byBo YXZlIEFDUyBmdW5jdGlvbmluZyBvbiBvdGhlciBQQ0kgZG9tYWlucyBzbyBpZiB5b3UgZG8gbm90 IGEgcGx1cmFsaXR5IG9mIElPTU1VIGdyb3VwaW5ncyB5b3UgY2FuIHN0aWxsIGFjaGlldmUgaXQg KGJ1dCB5b3UgY2FuJ3QgZG8gcDJwZG1hIGFjcm9zcyBJT01NVSBncm91cGluZ3MsIHdoaWNoIGlz IHNhZmUpLg0KDQo+IEknbSBjdXJyZW50bHkgdGVzdGluZyBQMlAgd2l0aCAgR1BVcyBpbiBkaWZm ZXJlbnQgSU9NTVUgZG9tYWlucyBhbmQgYXQgbGVhc3Qgd2l0aCBBTUQgSU9NTVVzIHRoYXQgd29y a3MgcGVyZmVjdGx5IGZpbmUuDQoNCll1cCB0aGF0IHNob3VsZCB3b3JrIHRob3VnaCBhZ2FpbiBJ IGhhdmUgdG8gYXNrIGFyZSB5b3UgZGlzYWJsaW5nIEFDUyBvbiB0aGUgcG9ydHMgYmV0d2VlbiB0 aGUgdHdvIHBlZXIgZGV2aWNlcyB0byBnZXQgdGhlIHAycCBiZW5lZml0PyBJZiBub3QgeW91IGFy ZSBub3QgZ2V0dGluZyBhbGwgdGhlIHBlcmZvcm1hbmNlIGJlbmVmaXQgKGR1ZSB0byBJT01NVSBy b3V0aW5nKSwgaWYgeW91IGFyZSB0aGVuIHRoZXJlIGFyZSBvYnZpb3VzbHkgc2VjdXJpdHkgaW1w bGljYXRpb25zIGJldHdlZW4gdGhvc2UgSU9NTVUgZG9tYWlucyBpZiB0aGV5IGFyZSBhc3NpZ25l ZCB0byBkaWZmZXJlbnQgVk1zLiBBbmQgbm93IHRoZSBpc3N1ZSBpcyBpZiBuZXcgZGV2aWNlcyBh cmUgYWRkZWQgYW5kIHRoZSBwMnAgdG9wb2xvZ3kgbmVlZGVkIHRvIGNoYW5nZSB0aGVyZSB3b3Vs ZCBiZSBubyB3YXkgdG8gaW5mb3JtIHRoZSBWTXMgb2YgYW55IElPTU1VIGdyb3VwIGNoYW5nZS4g DQoNCkNoZWVycw0KDQpTdGVwaGVuDQogICAgDQoNCg== From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Stephen Bates" Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Date: Tue, 8 May 2018 14:25:10 +0000 Message-ID: <479F2F93-903B-4A8A-8989-A4BD0E5E1471@raithlin.com> References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: Content-Language: en-US Content-ID: <3CD50565E70F874B8651DD7C38C3ECA2-JkAt9bkEularoOM5E8FhRbjFIynDaujOfM0AETQt39g@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , Bjorn Helgaas , Logan Gunthorpe , Alex Williamson Cc: Jens Axboe , Keith Busch , "linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org" , "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org" , "linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , =?utf-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Jason Gunthorpe , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-Id: linux-rdma@vger.kernel.org Hi Christian > AMD APUs mandatory need the ACS flag set for the GPU integrated in the > CPU when IOMMU is enabled or otherwise you will break SVM. OK but in this case aren't you losing (many of) the benefits of P2P since all DMAs will now get routed up to the IOMMU before being passed down to the destination PCIe EP? > Similar problems arise when you do this for dedicated GPU, but we > haven't upstreamed the support for this yet. Hmm, as above. With ACS enabled on all downstream ports any P2P enabled DMA will be routed to the IOMMU which removes a lot of the benefit. > So that is a clear NAK from my side for the approach. Do you have an alternative? This is the approach we arrived it after a reasonably lengthy discussion on the mailing lists. Alex, are you still comfortable with this approach? > And what exactly is the problem here? We had a pretty lengthy discussion on this topic on one of the previous revisions. The issue is that currently there is no mechanism in the IOMMU code to inform VMs if IOMMU groupings change. Since p2pdma can dynamically change its topology (due to PCI hotplug) we had to be cognizant of the fact that ACS settings could change. Since there is no way to currently handle changing ACS settings and hence IOMMU groupings the consensus was to simply disable ACS on all ports in a p2pdma domain. This effectively makes all the devices in the p2pdma domain part of the same IOMMU grouping. The plan will be to address this in time and add a mechanism for IOMMU grouping changes and notification to VMs but that's not part of this series. Note you are still allowed to have ACS functioning on other PCI dom ains so if you do not a plurality of IOMMU groupings you can still achieve it (but you can't do p2pdma across IOMMU groupings, which is safe). > I'm currently testing P2P with GPUs in different IOMMU domains and at least with AMD IOMMUs that works perfectly fine. Yup that should work though again I have to ask are you disabling ACS on the ports between the two peer devices to get the p2p benefit? If not you are not getting all the performance benefit (due to IOMMU routing), if you are then there are obviously security implications between those IOMMU domains if they are assigned to different VMs. And now the issue is if new devices are added and the p2p topology needed to change there would be no way to inform the VMs of any IOMMU group change. Cheers Stephen From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932480AbeEHOZR (ORCPT ); Tue, 8 May 2018 10:25:17 -0400 Received: from mail-eopbgr660100.outbound.protection.outlook.com ([40.107.66.100]:44304 "EHLO CAN01-QB1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932193AbeEHOZN (ORCPT ); Tue, 8 May 2018 10:25:13 -0400 From: "Stephen Bates" To: =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , "Bjorn Helgaas" , Logan Gunthorpe , "Alex Williamson" CC: "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "linux-block@vger.kernel.org" , "Christoph Hellwig" , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , =?utf-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Benjamin Herrenschmidt Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Thread-Topic: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Thread-Index: AQHT21soiBVnp6SJuEuLCiH6kBzTLKQk+zIAgACHcYCAABLOAA== Date: Tue, 8 May 2018 14:25:10 +0000 Message-ID: <479F2F93-903B-4A8A-8989-A4BD0E5E1471@raithlin.com> References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> In-Reply-To: Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/10.c.0.180410 authentication-results: spf=none (sender IP is ) smtp.mailfrom=sbates@raithlin.com; x-originating-ip: [70.65.250.31] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;YTXPR0101MB0781;7:tWkmYY1HCzg3gno61gZCoKjBCUizK8krcouBy5gD+ZM5vgxxtC0f0yaSToYy65+JJHjcVv2HZmKIWIyMo997sCFnCLYoU52maJ46R9Iy7GHYPgYPnLbyvhJ/SFZjQ9YsFG/vpWKzPTyoXkaRaX9skUPhDe3JCdA2MGFt6YnI0E14UzIgNDTY2JLN7cei5pBa072MBlr1mR3x/3p3aTiYHbMWb7i2+l6wPzT5/45f0bPWexmODNA0NW2VNesWhM6q x-ms-exchange-antispam-srfa-diagnostics: SOS; x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(7020095)(4652020)(7021125)(5600026)(4534165)(7022125)(4603075)(4627221)(201702281549075)(7048125)(7024125)(7027125)(7028125)(7023125)(2017052603328)(7153060)(7193020);SRVR:YTXPR0101MB0781; x-ms-traffictypediagnostic: YTXPR0101MB0781: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(192374486261705)(100405760836317); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(6040522)(2401047)(8121501046)(5005006)(3231254)(944501410)(52105095)(3002001)(93006095)(93001095)(10201501046)(149027)(150027)(6041310)(20161123562045)(20161123558120)(2016111802025)(20161123564045)(20161123560045)(6043046)(6072148)(201708071742011);SRVR:YTXPR0101MB0781;BCL:0;PCL:0;RULEID:;SRVR:YTXPR0101MB0781; x-forefront-prvs: 0666E15D35 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(396003)(366004)(346002)(376002)(39830400003)(39380400002)(189003)(199004)(2616005)(82746002)(66066001)(5660300001)(7416002)(5250100002)(83716003)(446003)(58126008)(97736004)(86362001)(110136005)(316002)(68736007)(33656002)(54906003)(486006)(476003)(14454004)(11346002)(186003)(8666007)(53936002)(4326008)(36756003)(7736002)(6512007)(305945005)(81156014)(8936002)(26005)(2906002)(76176011)(6506007)(3660700001)(81166006)(106356001)(59450400001)(102836004)(105586002)(3280700002)(6116002)(6486002)(229853002)(478600001)(3846002)(2900100001)(99286004)(6246003)(25786009)(6436002)(8676002)(93886005);DIR:OUT;SFP:1102;SCL:1;SRVR:YTXPR0101MB0781;H:YTXPR0101MB2045.CANPRD01.PROD.OUTLOOK.COM;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; x-microsoft-antispam-message-info: DBwkeQLayZvkFFI7F3ED4jTW+AJrGm3Lvv/xfYlKWjGvuG6g+FyHIovOV8aTShYI/Rgwthb3PS4jq0KrMc++bOY/5TzRhG/+i1VCu4UOZOf8Bn7cAHZbAjPJcpYai20zyj1y2BE/EoJg2COJHBHicMR5gFrKjm9Nm83VGyXFA6x/XP6ps855zFIWo0ZZgfRB spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: <3CD50565E70F874B8651DD7C38C3ECA2@CANPRD01.PROD.OUTLOOK.COM> MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: f297e2fe-0333-4f8a-1060-08d5b4ef8182 X-OriginatorOrg: raithlin.com X-MS-Exchange-CrossTenant-Network-Message-Id: f297e2fe-0333-4f8a-1060-08d5b4ef8182 X-MS-Exchange-CrossTenant-originalarrivaltime: 08 May 2018 14:25:10.5136 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 18519031-7ff4-4cbb-bbcb-c3252d330f4b X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR0101MB0781 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id w48EQdaN019456 Hi Christian > AMD APUs mandatory need the ACS flag set for the GPU integrated in the > CPU when IOMMU is enabled or otherwise you will break SVM. OK but in this case aren't you losing (many of) the benefits of P2P since all DMAs will now get routed up to the IOMMU before being passed down to the destination PCIe EP? > Similar problems arise when you do this for dedicated GPU, but we > haven't upstreamed the support for this yet. Hmm, as above. With ACS enabled on all downstream ports any P2P enabled DMA will be routed to the IOMMU which removes a lot of the benefit. > So that is a clear NAK from my side for the approach. Do you have an alternative? This is the approach we arrived it after a reasonably lengthy discussion on the mailing lists. Alex, are you still comfortable with this approach? > And what exactly is the problem here? We had a pretty lengthy discussion on this topic on one of the previous revisions. The issue is that currently there is no mechanism in the IOMMU code to inform VMs if IOMMU groupings change. Since p2pdma can dynamically change its topology (due to PCI hotplug) we had to be cognizant of the fact that ACS settings could change. Since there is no way to currently handle changing ACS settings and hence IOMMU groupings the consensus was to simply disable ACS on all ports in a p2pdma domain. This effectively makes all the devices in the p2pdma domain part of the same IOMMU grouping. The plan will be to address this in time and add a mechanism for IOMMU grouping changes and notification to VMs but that's not part of this series. Note you are still allowed to have ACS functioning on other PCI domains so if you do not a plurality of IOMMU groupings you can still achieve it (but you can't do p2pdma across IOMMU groupings, which is safe). > I'm currently testing P2P with GPUs in different IOMMU domains and at least with AMD IOMMUs that works perfectly fine. Yup that should work though again I have to ask are you disabling ACS on the ports between the two peer devices to get the p2p benefit? If not you are not getting all the performance benefit (due to IOMMU routing), if you are then there are obviously security implications between those IOMMU domains if they are assigned to different VMs. And now the issue is if new devices are added and the p2p topology needed to change there would be no way to inform the VMs of any IOMMU group change. Cheers Stephen From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Return-Path: From: "Stephen Bates" To: =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , "Bjorn Helgaas" , Logan Gunthorpe , "Alex Williamson" Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Date: Tue, 8 May 2018 14:25:10 +0000 Message-ID: <479F2F93-903B-4A8A-8989-A4BD0E5E1471@raithlin.com> References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> In-Reply-To: MIME-Version: 1.0 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jens Axboe , Keith Busch , Sagi Grimberg , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-block@vger.kernel.org" , =?utf-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Jason Gunthorpe , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , Dan Williams , Christoph Hellwig Content-Type: text/plain; charset="us-ascii" Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+bjorn=helgaas.com@lists.infradead.org List-ID: Hi Christian > AMD APUs mandatory need the ACS flag set for the GPU integrated in the > CPU when IOMMU is enabled or otherwise you will break SVM. OK but in this case aren't you losing (many of) the benefits of P2P since all DMAs will now get routed up to the IOMMU before being passed down to the destination PCIe EP? > Similar problems arise when you do this for dedicated GPU, but we > haven't upstreamed the support for this yet. Hmm, as above. With ACS enabled on all downstream ports any P2P enabled DMA will be routed to the IOMMU which removes a lot of the benefit. > So that is a clear NAK from my side for the approach. Do you have an alternative? This is the approach we arrived it after a reasonably lengthy discussion on the mailing lists. Alex, are you still comfortable with this approach? > And what exactly is the problem here? We had a pretty lengthy discussion on this topic on one of the previous revisions. The issue is that currently there is no mechanism in the IOMMU code to inform VMs if IOMMU groupings change. Since p2pdma can dynamically change its topology (due to PCI hotplug) we had to be cognizant of the fact that ACS settings could change. Since there is no way to currently handle changing ACS settings and hence IOMMU groupings the consensus was to simply disable ACS on all ports in a p2pdma domain. This effectively makes all the devices in the p2pdma domain part of the same IOMMU grouping. The plan will be to address this in time and add a mechanism for IOMMU grouping changes and notification to VMs but that's not part of this series. Note you are still allowed to have ACS functioning on other PCI domains so if you do not a plurality of IOMMU groupings you can still achieve it (but you can't do p2pdma across IOMMU groupings, which is safe). > I'm currently testing P2P with GPUs in different IOMMU domains and at least with AMD IOMMUs that works perfectly fine. Yup that should work though again I have to ask are you disabling ACS on the ports between the two peer devices to get the p2p benefit? If not you are not getting all the performance benefit (due to IOMMU routing), if you are then there are obviously security implications between those IOMMU domains if they are assigned to different VMs. And now the issue is if new devices are added and the p2p topology needed to change there would be no way to inform the VMs of any IOMMU group change. Cheers Stephen _______________________________________________ Linux-nvme mailing list Linux-nvme@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-nvme From mboxrd@z Thu Jan 1 00:00:00 1970 From: sbates@raithlin.com (Stephen Bates) Date: Tue, 8 May 2018 14:25:10 +0000 Subject: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches In-Reply-To: References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> Message-ID: <479F2F93-903B-4A8A-8989-A4BD0E5E1471@raithlin.com> Hi Christian > AMD APUs mandatory need the ACS flag set for the GPU integrated in the > CPU when IOMMU is enabled or otherwise you will break SVM. OK but in this case aren't you losing (many of) the benefits of P2P since all DMAs will now get routed up to the IOMMU before being passed down to the destination PCIe EP? > Similar problems arise when you do this for dedicated GPU, but we > haven't upstreamed the support for this yet. Hmm, as above. With ACS enabled on all downstream ports any P2P enabled DMA will be routed to the IOMMU which removes a lot of the benefit. > So that is a clear NAK from my side for the approach. Do you have an alternative? This is the approach we arrived it after a reasonably lengthy discussion on the mailing lists. Alex, are you still comfortable with this approach? > And what exactly is the problem here? We had a pretty lengthy discussion on this topic on one of the previous revisions. The issue is that currently there is no mechanism in the IOMMU code to inform VMs if IOMMU groupings change. Since p2pdma can dynamically change its topology (due to PCI hotplug) we had to be cognizant of the fact that ACS settings could change. Since there is no way to currently handle changing ACS settings and hence IOMMU groupings the consensus was to simply disable ACS on all ports in a p2pdma domain. This effectively makes all the devices in the p2pdma domain part of the same IOMMU grouping. The plan will be to address this in time and add a mechanism for IOMMU grouping changes and notification to VMs but that's not part of this series. Note you are still allowed to have ACS functioning on other PCI domains so if you do not a plurality of IOMMU groupings you can still achieve it (but you can't do p2pdma across IOMMU groupings, which is safe). > I'm currently testing P2P with GPUs in different IOMMU domains and at least with AMD IOMMUs that works perfectly fine. Yup that should work though again I have to ask are you disabling ACS on the ports between the two peer devices to get the p2p benefit? If not you are not getting all the performance benefit (due to IOMMU routing), if you are then there are obviously security implications between those IOMMU domains if they are assigned to different VMs. And now the issue is if new devices are added and the p2p topology needed to change there would be no way to inform the VMs of any IOMMU group change. Cheers Stephen