From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670124.outbound.protection.outlook.com [40.107.67.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 709A920965DFD for ; Thu, 10 May 2018 07:20:02 -0700 (PDT) From: "Stephen Bates" Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Date: Thu, 10 May 2018 14:20:00 +0000 Message-ID: <405531AE-8315-4A4F-9B0C-8DBE49BFCAB4@raithlin.com> References: <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508205005.GC15608@redhat.com> <7FFB9603-DF9F-4441-82E9-46037CB6C0DE@raithlin.com> <1775CC56-4651-422F-953A-18E024D3717C@raithlin.com> <20180509160722.GB4140@redhat.com> <366A8132-B88A-40F7-BDE3-DA542E45FC0C@raithlin.com> <20180509174952.GC4140@redhat.com> In-Reply-To: <20180509174952.GC4140@redhat.com> Content-Language: en-US Content-ID: <88BFB02BD50775468182FF5860ACE23A@CANPRD01.PROD.OUTLOOK.COM> MIME-Version: 1.0 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Jerome Glisse Cc: Jens Axboe , Keith Busch , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-nvme@lists.infradead.org" , Christoph Hellwig , "linux-block@vger.kernel.org" , Alex Williamson , Jason Gunthorpe , Bjorn Helgaas , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= List-ID: Hi Jerome > As it is tie to PASID this is done using IOMMU so looks for caller > of amd_iommu_bind_pasid() or intel_svm_bind_mm() in GPU the existing > user is the AMD GPU driver see: Ah thanks. This cleared things up for me. A quick search shows there are still no users of intel_svm_bind_mm() but I see the AMD version used in that GPU driver. One thing I could not grok from the code how the GPU driver indicates which DMA events require ATS translations and which do not. I am assuming the driver implements someway of indicating that and its not just a global ON or OFF for all DMAs? The reason I ask is that I looking at if NVMe was to support ATS what would need to be added in the NVMe spec above and beyond what we have in PCI ATS to support efficient use of ATS (for example would we need a flag in the submission queue entries to indicate a particular IO's SGL/PRP should undergo ATS). Cheers Stephen _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: "Stephen Bates" To: Jerome Glisse CC: =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , "Logan Gunthorpe" , Alex Williamson , Bjorn Helgaas , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "linux-block@vger.kernel.org" , "Christoph Hellwig" , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , "Benjamin Herrenschmidt" Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Date: Thu, 10 May 2018 14:20:00 +0000 Message-ID: <405531AE-8315-4A4F-9B0C-8DBE49BFCAB4@raithlin.com> References: <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508205005.GC15608@redhat.com> <7FFB9603-DF9F-4441-82E9-46037CB6C0DE@raithlin.com> <1775CC56-4651-422F-953A-18E024D3717C@raithlin.com> <20180509160722.GB4140@redhat.com> <366A8132-B88A-40F7-BDE3-DA542E45FC0C@raithlin.com> <20180509174952.GC4140@redhat.com> In-Reply-To: <20180509174952.GC4140@redhat.com> Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 List-ID: SGkgSmVyb21lDQoNCj4gQXMgaXQgaXMgdGllIHRvIFBBU0lEIHRoaXMgaXMgZG9uZSB1c2luZyBJ T01NVSBzbyBsb29rcyBmb3IgY2FsbGVyDQo+IG9mIGFtZF9pb21tdV9iaW5kX3Bhc2lkKCkgb3Ig aW50ZWxfc3ZtX2JpbmRfbW0oKSBpbiBHUFUgdGhlIGV4aXN0aW5nDQo+ICB1c2VyIGlzIHRoZSBB TUQgR1BVIGRyaXZlciBzZWU6DQogICAgDQpBaCB0aGFua3MuIFRoaXMgY2xlYXJlZCB0aGluZ3Mg dXAgZm9yIG1lLiBBIHF1aWNrIHNlYXJjaCBzaG93cyB0aGVyZSBhcmUgc3RpbGwgbm8gdXNlcnMg b2YgaW50ZWxfc3ZtX2JpbmRfbW0oKSBidXQgSSBzZWUgdGhlIEFNRCB2ZXJzaW9uIHVzZWQgaW4g dGhhdCBHUFUgZHJpdmVyLg0KDQpPbmUgdGhpbmcgSSBjb3VsZCBub3QgZ3JvayBmcm9tIHRoZSBj b2RlIGhvdyB0aGUgR1BVIGRyaXZlciBpbmRpY2F0ZXMgd2hpY2ggRE1BIGV2ZW50cyByZXF1aXJl IEFUUyB0cmFuc2xhdGlvbnMgYW5kIHdoaWNoIGRvIG5vdC4gSSBhbSBhc3N1bWluZyB0aGUgZHJp dmVyIGltcGxlbWVudHMgc29tZXdheSBvZiBpbmRpY2F0aW5nIHRoYXQgYW5kIGl0cyBub3QganVz dCBhIGdsb2JhbCBPTiBvciBPRkYgZm9yIGFsbCBETUFzPyBUaGUgcmVhc29uIEkgYXNrIGlzIHRo YXQgSSBsb29raW5nIGF0IGlmIE5WTWUgd2FzIHRvIHN1cHBvcnQgQVRTIHdoYXQgd291bGQgbmVl ZCB0byBiZSBhZGRlZCBpbiB0aGUgTlZNZSBzcGVjIGFib3ZlIGFuZCBiZXlvbmQgd2hhdCB3ZSBo YXZlIGluIFBDSSBBVFMgdG8gc3VwcG9ydCBlZmZpY2llbnQgdXNlIG9mIEFUUyAoZm9yIGV4YW1w bGUgd291bGQgd2UgbmVlZCBhIGZsYWcgaW4gdGhlIHN1Ym1pc3Npb24gcXVldWUgZW50cmllcyB0 byBpbmRpY2F0ZSBhIHBhcnRpY3VsYXIgSU8ncyBTR0wvUFJQIHNob3VsZCB1bmRlcmdvIEFUUyku DQoNCkNoZWVycw0KDQpTdGVwaGVuICAgIA0KDQo= From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Stephen Bates" Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Date: Thu, 10 May 2018 14:20:00 +0000 Message-ID: <405531AE-8315-4A4F-9B0C-8DBE49BFCAB4@raithlin.com> References: <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508205005.GC15608@redhat.com> <7FFB9603-DF9F-4441-82E9-46037CB6C0DE@raithlin.com> <1775CC56-4651-422F-953A-18E024D3717C@raithlin.com> <20180509160722.GB4140@redhat.com> <366A8132-B88A-40F7-BDE3-DA542E45FC0C@raithlin.com> <20180509174952.GC4140@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20180509174952.GC4140-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> Content-Language: en-US Content-ID: <88BFB02BD50775468182FF5860ACE23A-JkAt9bkEularoOM5E8FhRbjFIynDaujOfM0AETQt39g@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Jerome Glisse Cc: Jens Axboe , Keith Busch , "linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org" , "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org" , Christoph Hellwig , "linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Alex Williamson , Jason Gunthorpe , Bjorn Helgaas , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= List-Id: linux-rdma@vger.kernel.org Hi Jerome > As it is tie to PASID this is done using IOMMU so looks for caller > of amd_iommu_bind_pasid() or intel_svm_bind_mm() in GPU the existing > user is the AMD GPU driver see: Ah thanks. This cleared things up for me. A quick search shows there are still no users of intel_svm_bind_mm() but I see the AMD version used in that GPU driver. One thing I could not grok from the code how the GPU driver indicates which DMA events require ATS translations and which do not. I am assuming the driver implements someway of indicating that and its not just a global ON or OFF for all DMAs? The reason I ask is that I looking at if NVMe was to support ATS what would need to be added in the NVMe spec above and beyond what we have in PCI ATS to support efficient use of ATS (for example would we need a flag in the submission queue entries to indicate a particular IO's SGL/PRP should undergo ATS). Cheers Stephen From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S965700AbeEJOUG (ORCPT ); Thu, 10 May 2018 10:20:06 -0400 Received: from mail-eopbgr670136.outbound.protection.outlook.com ([40.107.67.136]:10735 "EHLO CAN01-TO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S965516AbeEJOUC (ORCPT ); Thu, 10 May 2018 10:20:02 -0400 From: "Stephen Bates" To: Jerome Glisse CC: =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , "Logan Gunthorpe" , Alex Williamson , Bjorn Helgaas , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "linux-block@vger.kernel.org" , "Christoph Hellwig" , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , "Benjamin Herrenschmidt" Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Thread-Topic: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Thread-Index: AQHT21soiBVnp6SJuEuLCiH6kBzTLKQk+zIAgACHcYCAAJmxAIAABkiAgAAoBgCAAAW2gIAAA0YAgAAHvYCAAAGOgIAACKoAgACuAYCAAGxLgP//vUwAgABrwID//6HiAIAAesGAgADzGoA= Date: Thu, 10 May 2018 14:20:00 +0000 Message-ID: <405531AE-8315-4A4F-9B0C-8DBE49BFCAB4@raithlin.com> References: <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508205005.GC15608@redhat.com> <7FFB9603-DF9F-4441-82E9-46037CB6C0DE@raithlin.com> <1775CC56-4651-422F-953A-18E024D3717C@raithlin.com> <20180509160722.GB4140@redhat.com> <366A8132-B88A-40F7-BDE3-DA542E45FC0C@raithlin.com> <20180509174952.GC4140@redhat.com> In-Reply-To: <20180509174952.GC4140@redhat.com> Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/10.d.0.180505 authentication-results: spf=none (sender IP is ) smtp.mailfrom=sbates@raithlin.com; x-originating-ip: [70.65.250.31] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;YTXSPR01MB023;7:dG7wma3xmgTCgrr0zLLfj9MwOBtXCATGRkk7Z24S4b2ekurWHCGOTjpE4BHXAe5FJKaX50/dusGrdknPG5LyGvoxaauDuojV+Uw6bGBJYen9CAudT2Y+SJAcDiBdm07D4Qb+CPdylh+9/a4b9HPXj6zAnbcPz3iz9j8606MiyykCRRiBCN8nTUMeX3tW6hQMrCXxhmF/QGKGPRiQJOJa69Lt7e8oA6RBR/pOaiDHK5okZq59DAuDge9NEXqpnQuT x-ms-exchange-antispam-srfa-diagnostics: SOS; x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(7020095)(4652020)(7021125)(4534165)(7022125)(4603075)(4627221)(201702281549075)(7048125)(7024125)(7027125)(7028125)(7023125)(5600026)(2017052603328)(7153060)(7193020);SRVR:YTXSPR01MB023; x-ms-traffictypediagnostic: YTXSPR01MB023: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(6040522)(2401047)(5005006)(8121501046)(10201501046)(3231254)(944501410)(52105095)(93006095)(93001095)(3002001)(149027)(150027)(6041310)(20161123560045)(20161123562045)(20161123564045)(2016111802025)(20161123558120)(6043046)(6072148)(201708071742011);SRVR:YTXSPR01MB023;BCL:0;PCL:0;RULEID:;SRVR:YTXSPR01MB023; x-forefront-prvs: 066898046A x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(366004)(39830400003)(376002)(346002)(396003)(39380400002)(189003)(199004)(81166006)(81156014)(97736004)(11346002)(14454004)(106356001)(105586002)(3660700001)(8936002)(2906002)(6506007)(83716003)(6916009)(3846002)(68736007)(6116002)(2616005)(476003)(446003)(8676002)(7736002)(99286004)(305945005)(102836004)(186003)(5250100002)(86362001)(7416002)(5660300001)(26005)(25786009)(82746002)(478600001)(76176011)(3280700002)(33656002)(4326008)(6486002)(53936002)(6246003)(6436002)(8666007)(229853002)(2900100001)(93886005)(58126008)(316002)(66066001)(486006)(54906003)(36756003)(6512007);DIR:OUT;SFP:1102;SCL:1;SRVR:YTXSPR01MB023;H:YTXPR0101MB2045.CANPRD01.PROD.OUTLOOK.COM;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; x-microsoft-antispam-message-info: RxZtqftJEk00I2MNG0RUwAO5W52U/ssdvU64qwPm5APag+9hbsMk8FC1Et02LqDl8cNesnL5yib0lmg4YBnWKgfPv4t9tECpOdIYmAQEUlfiEc8SbvXukrD2Bx62bJWtGKMPfbMrsV2j7M0wKTIipx8wZqwlDpEwuI8KgSyww5rty3T/CdRY8eg9q0Xxj+3e spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: <88BFB02BD50775468182FF5860ACE23A@CANPRD01.PROD.OUTLOOK.COM> MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 0fed9e8a-756c-4d46-f565-08d5b6811d5a X-OriginatorOrg: raithlin.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0fed9e8a-756c-4d46-f565-08d5b6811d5a X-MS-Exchange-CrossTenant-originalarrivaltime: 10 May 2018 14:20:00.1236 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 18519031-7ff4-4cbb-bbcb-c3252d330f4b X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXSPR01MB023 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id w4AEKBnU003421 Hi Jerome > As it is tie to PASID this is done using IOMMU so looks for caller > of amd_iommu_bind_pasid() or intel_svm_bind_mm() in GPU the existing > user is the AMD GPU driver see: Ah thanks. This cleared things up for me. A quick search shows there are still no users of intel_svm_bind_mm() but I see the AMD version used in that GPU driver. One thing I could not grok from the code how the GPU driver indicates which DMA events require ATS translations and which do not. I am assuming the driver implements someway of indicating that and its not just a global ON or OFF for all DMAs? The reason I ask is that I looking at if NVMe was to support ATS what would need to be added in the NVMe spec above and beyond what we have in PCI ATS to support efficient use of ATS (for example would we need a flag in the submission queue entries to indicate a particular IO's SGL/PRP should undergo ATS). Cheers Stephen From mboxrd@z Thu Jan 1 00:00:00 1970 From: sbates@raithlin.com (Stephen Bates) Date: Thu, 10 May 2018 14:20:00 +0000 Subject: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches In-Reply-To: <20180509174952.GC4140@redhat.com> References: <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508205005.GC15608@redhat.com> <7FFB9603-DF9F-4441-82E9-46037CB6C0DE@raithlin.com> <1775CC56-4651-422F-953A-18E024D3717C@raithlin.com> <20180509160722.GB4140@redhat.com> <366A8132-B88A-40F7-BDE3-DA542E45FC0C@raithlin.com> <20180509174952.GC4140@redhat.com> Message-ID: <405531AE-8315-4A4F-9B0C-8DBE49BFCAB4@raithlin.com> Hi Jerome > As it is tie to PASID this is done using IOMMU so looks for caller > of amd_iommu_bind_pasid() or intel_svm_bind_mm() in GPU the existing > user is the AMD GPU driver see: Ah thanks. This cleared things up for me. A quick search shows there are still no users of intel_svm_bind_mm() but I see the AMD version used in that GPU driver. One thing I could not grok from the code how the GPU driver indicates which DMA events require ATS translations and which do not. I am assuming the driver implements someway of indicating that and its not just a global ON or OFF for all DMAs? The reason I ask is that I looking at if NVMe was to support ATS what would need to be added in the NVMe spec above and beyond what we have in PCI ATS to support efficient use of ATS (for example would we need a flag in the submission queue entries to indicate a particular IO's SGL/PRP should undergo ATS). Cheers Stephen