From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660090.outbound.protection.outlook.com [40.107.66.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id D3E9E2096AEC2 for ; Wed, 9 May 2018 05:44:23 -0700 (PDT) From: "Stephen Bates" Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Date: Wed, 9 May 2018 12:44:21 +0000 Message-ID: References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> <3584a6ac-95c7-5d23-1859-aee30605776e@deltatee.com> <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508144341.0441b676@w520.home> <20180508152631.50fd583c@w520.home> <354F7407-0DC7-470C-B9AA-74FDF9C46B08@raithlin.com> <20180508160336.0935ddde@w520.home> <11819753-af26-4632-8580-d4a47127a3b2@redhat.com> In-Reply-To: <11819753-af26-4632-8580-d4a47127a3b2@redhat.com> Content-Language: en-US Content-ID: <42C078A13685944FAC4A17915C5D134F@CANPRD01.PROD.OUTLOOK.COM> MIME-Version: 1.0 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Don Dutile , Alex Williamson Cc: Jens Axboe , Keith Busch , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-nvme@lists.infradead.org" , Christoph Hellwig , "linux-block@vger.kernel.org" , =?utf-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Jason Gunthorpe , Bjorn Helgaas , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= List-ID: Hi Don > RDMA VFs lend themselves to NVMEoF w/device-assignment.... need a way to > put NVME 'resources' into an assignable/manageable object for 'IOMMU-grouping', > which is really a 'DMA security domain' and less an 'IOMMU grouping domain'. Ha, I like your term "DMA Security Domain" which sounds about right for what we are discussing with p2pdma and ACS disablement ;-). The problem is that ACS is, in some ways, too big of hammer for what we want here in the sense that it is either on or off for the bridge or MF EP we enable/disable it for. ACS can't filter the TLPs by address or ID though PCI-SIG are having some discussions on extending ACS. That's a long term solution and won't be applicable to us for some time. NVMe SSDs that support SR-IOV are coming to market but we can't assume all NVMe SSDs with support SR-IOV. That will probably be a pretty high end-feature... Stephen _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: "Stephen Bates" To: Don Dutile , Alex Williamson CC: Logan Gunthorpe , =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , "Bjorn Helgaas" , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "linux-block@vger.kernel.org" , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , =?utf-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Benjamin Herrenschmidt Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Date: Wed, 9 May 2018 12:44:21 +0000 Message-ID: References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> <3584a6ac-95c7-5d23-1859-aee30605776e@deltatee.com> <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508144341.0441b676@w520.home> <20180508152631.50fd583c@w520.home> <354F7407-0DC7-470C-B9AA-74FDF9C46B08@raithlin.com> <20180508160336.0935ddde@w520.home> <11819753-af26-4632-8580-d4a47127a3b2@redhat.com> In-Reply-To: <11819753-af26-4632-8580-d4a47127a3b2@redhat.com> Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 List-ID: SGkgRG9uDQoNCj4gICAgUkRNQSBWRnMgbGVuZCB0aGVtc2VsdmVzIHRvIE5WTUVvRiB3L2Rldmlj ZS1hc3NpZ25tZW50Li4uLiBuZWVkIGEgd2F5IHRvDQo+ICAgIHB1dCBOVk1FICdyZXNvdXJjZXMn IGludG8gYW4gYXNzaWduYWJsZS9tYW5hZ2VhYmxlIG9iamVjdCBmb3IgJ0lPTU1VLWdyb3VwaW5n JywNCj4gICAgd2hpY2ggaXMgcmVhbGx5IGEgJ0RNQSBzZWN1cml0eSBkb21haW4nIGFuZCBsZXNz IGFuICdJT01NVSBncm91cGluZyBkb21haW4nLg0KICAgIA0KSGEsIEkgbGlrZSB5b3VyIHRlcm0g IkRNQSBTZWN1cml0eSBEb21haW4iIHdoaWNoIHNvdW5kcyBhYm91dCByaWdodCBmb3Igd2hhdCB3 ZSBhcmUgZGlzY3Vzc2luZyB3aXRoIHAycGRtYSBhbmQgQUNTIGRpc2FibGVtZW50IDstKS4gVGhl IHByb2JsZW0gaXMgdGhhdCBBQ1MgaXMsIGluIHNvbWUgd2F5cywgdG9vIGJpZyBvZiBoYW1tZXIg Zm9yIHdoYXQgd2Ugd2FudCBoZXJlIGluIHRoZSBzZW5zZSB0aGF0IGl0IGlzIGVpdGhlciBvbiBv ciBvZmYgZm9yIHRoZSBicmlkZ2Ugb3IgTUYgRVAgd2UgZW5hYmxlL2Rpc2FibGUgaXQgZm9yLiBB Q1MgY2FuJ3QgZmlsdGVyIHRoZSBUTFBzIGJ5IGFkZHJlc3Mgb3IgSUQgdGhvdWdoIFBDSS1TSUcg YXJlIGhhdmluZyBzb21lIGRpc2N1c3Npb25zIG9uIGV4dGVuZGluZyBBQ1MuIFRoYXQncyBhIGxv bmcgdGVybSBzb2x1dGlvbiBhbmQgd29uJ3QgYmUgYXBwbGljYWJsZSB0byB1cyBmb3Igc29tZSB0 aW1lLg0KDQpOVk1lIFNTRHMgdGhhdCBzdXBwb3J0IFNSLUlPViBhcmUgY29taW5nIHRvIG1hcmtl dCBidXQgd2UgY2FuJ3QgYXNzdW1lIGFsbCBOVk1lIFNTRHMgd2l0aCBzdXBwb3J0IFNSLUlPVi4g VGhhdCB3aWxsIHByb2JhYmx5IGJlIGEgcHJldHR5IGhpZ2ggZW5kLWZlYXR1cmUuLi4NCg0KU3Rl cGhlbg0KICAgIA0KICAgIA0KDQo= From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Stephen Bates" Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Date: Wed, 9 May 2018 12:44:21 +0000 Message-ID: References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> <3584a6ac-95c7-5d23-1859-aee30605776e@deltatee.com> <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508144341.0441b676@w520.home> <20180508152631.50fd583c@w520.home> <354F7407-0DC7-470C-B9AA-74FDF9C46B08@raithlin.com> <20180508160336.0935ddde@w520.home> <11819753-af26-4632-8580-d4a47127a3b2@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <11819753-af26-4632-8580-d4a47127a3b2-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> Content-Language: en-US Content-ID: <42C078A13685944FAC4A17915C5D134F-JkAt9bkEularoOM5E8FhRbjFIynDaujOfM0AETQt39g@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Don Dutile , Alex Williamson Cc: Jens Axboe , Keith Busch , "linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org" , "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org" , Christoph Hellwig , "linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , =?utf-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Jason Gunthorpe , Bjorn Helgaas , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= List-Id: linux-rdma@vger.kernel.org Hi Don > RDMA VFs lend themselves to NVMEoF w/device-assignment.... need a way to > put NVME 'resources' into an assignable/manageable object for 'IOMMU-grouping', > which is really a 'DMA security domain' and less an 'IOMMU grouping domain'. Ha, I like your term "DMA Security Domain" which sounds about right for what we are discussing with p2pdma and ACS disablement ;-). The problem is that ACS is, in some ways, too big of hammer for what we want here in the sense that it is either on or off for the bridge or MF EP we enable/disable it for. ACS can't filter the TLPs by address or ID though PCI-SIG are having some discussions on extending ACS. That's a long term solution and won't be applicable to us for some time. NVMe SSDs that support SR-IOV are coming to market but we can't assume all NVMe SSDs with support SR-IOV. That will probably be a pretty high end-feature... Stephen From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934728AbeEIMo0 (ORCPT ); Wed, 9 May 2018 08:44:26 -0400 Received: from mail-eopbgr670098.outbound.protection.outlook.com ([40.107.67.98]:38285 "EHLO CAN01-TO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S933953AbeEIMoX (ORCPT ); Wed, 9 May 2018 08:44:23 -0400 From: "Stephen Bates" To: Don Dutile , Alex Williamson CC: Logan Gunthorpe , =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= , "Bjorn Helgaas" , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "linux-block@vger.kernel.org" , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , =?utf-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Benjamin Herrenschmidt Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Thread-Topic: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Thread-Index: AQHT21soiBVnp6SJuEuLCiH6kBzTLKQk+zIAgACHcYCAAJmxAIAABkiAgAAoBgCAAAW2gIAAA0YAgAAHvYCAAAGOgIAABt+AgAABmICAAApggP//n96AgABqfgCAAATvgIAAjJCA Date: Wed, 9 May 2018 12:44:21 +0000 Message-ID: References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> <3584a6ac-95c7-5d23-1859-aee30605776e@deltatee.com> <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508144341.0441b676@w520.home> <20180508152631.50fd583c@w520.home> <354F7407-0DC7-470C-B9AA-74FDF9C46B08@raithlin.com> <20180508160336.0935ddde@w520.home> <11819753-af26-4632-8580-d4a47127a3b2@redhat.com> In-Reply-To: <11819753-af26-4632-8580-d4a47127a3b2@redhat.com> Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/10.d.0.180505 authentication-results: spf=none (sender IP is ) smtp.mailfrom=sbates@raithlin.com; x-originating-ip: [70.65.250.31] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;YTOPR0101MB1020;7:QJQURTDvQ3VgxwqhOYal/h9gw2PJJTZ0R82xXxc/ZA2EqFu6hsGhgoNXCDbpJctmax0Ob7ZuDEoS6j/oxkOGhkzV5hXYr+jhhgPV6YomsO2gLoChGhjwkOLZuq2KftZVU2qbXhVbKPX17FT/acN36to0dTUbMeROOdJ8U0entGjqe0Li0ThgLYNa8QqUMZWiCt2pwcDEhanBZEbWsPpD7GNfajP8hzJ6NVCBF8QHtVppXpZQ+fISiGb3hezysYni x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(7020095)(4652020)(7021125)(4534165)(7022125)(4603075)(4627221)(201702281549075)(7048125)(7024125)(7027125)(7028125)(7023125)(5600026)(2017052603328)(7153060)(7193020);SRVR:YTOPR0101MB1020; x-ms-traffictypediagnostic: YTOPR0101MB1020: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(192374486261705); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(6040522)(2401047)(8121501046)(5005006)(3002001)(10201501046)(93006095)(93001095)(3231254)(944501410)(52105095)(149027)(150027)(6041310)(2016111802025)(20161123558120)(20161123564045)(20161123562045)(20161123560045)(6072148)(6043046)(201708071742011);SRVR:YTOPR0101MB1020;BCL:0;PCL:0;RULEID:;SRVR:YTOPR0101MB1020; x-forefront-prvs: 0667289FF8 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(346002)(366004)(376002)(396003)(39830400003)(39380400002)(189003)(199004)(3280700002)(99286004)(3660700001)(6486002)(6506007)(229853002)(5250100002)(486006)(6246003)(86362001)(53936002)(26005)(33656002)(7736002)(14454004)(68736007)(2906002)(97736004)(81156014)(8936002)(6512007)(59450400001)(8676002)(8666007)(36756003)(106356001)(102836004)(105586002)(6436002)(76176011)(4326008)(66066001)(316002)(5660300001)(81166006)(478600001)(83716003)(93886005)(305945005)(58126008)(446003)(476003)(186003)(11346002)(7416002)(25786009)(110136005)(3846002)(54906003)(6116002)(82746002)(2900100001)(2616005);DIR:OUT;SFP:1102;SCL:1;SRVR:YTOPR0101MB1020;H:YTOPR0101MB2043.CANPRD01.PROD.OUTLOOK.COM;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; x-microsoft-antispam-message-info: DBeUl/QOO9aF6T3CKM6soglZkgls2Xuq3zFlfBfbmRBLd/uUGOMlMIhXn3fZqCc5QpWG5MrHlw9SQNlThLSXVP+nVm1gSfR+IFSODdT400g0bTgjl5P+cTvb9bd2eD0lntkOsX89DgQW3gHeYSbFBuijaMiwjUb7yptoFBc74NGw//fTwU5HwOCktCvXQqBQ spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: <42C078A13685944FAC4A17915C5D134F@CANPRD01.PROD.OUTLOOK.COM> MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 1e5844b6-b15b-4532-5fe7-08d5b5aa968b X-OriginatorOrg: raithlin.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1e5844b6-b15b-4532-5fe7-08d5b5aa968b X-MS-Exchange-CrossTenant-originalarrivaltime: 09 May 2018 12:44:21.6952 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 18519031-7ff4-4cbb-bbcb-c3252d330f4b X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB1020 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id w49CiWmS004380 Hi Don > RDMA VFs lend themselves to NVMEoF w/device-assignment.... need a way to > put NVME 'resources' into an assignable/manageable object for 'IOMMU-grouping', > which is really a 'DMA security domain' and less an 'IOMMU grouping domain'. Ha, I like your term "DMA Security Domain" which sounds about right for what we are discussing with p2pdma and ACS disablement ;-). The problem is that ACS is, in some ways, too big of hammer for what we want here in the sense that it is either on or off for the bridge or MF EP we enable/disable it for. ACS can't filter the TLPs by address or ID though PCI-SIG are having some discussions on extending ACS. That's a long term solution and won't be applicable to us for some time. NVMe SSDs that support SR-IOV are coming to market but we can't assume all NVMe SSDs with support SR-IOV. That will probably be a pretty high end-feature... Stephen From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Return-Path: From: "Stephen Bates" To: Don Dutile , Alex Williamson Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Date: Wed, 9 May 2018 12:44:21 +0000 Message-ID: References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> <3584a6ac-95c7-5d23-1859-aee30605776e@deltatee.com> <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508144341.0441b676@w520.home> <20180508152631.50fd583c@w520.home> <354F7407-0DC7-470C-B9AA-74FDF9C46B08@raithlin.com> <20180508160336.0935ddde@w520.home> <11819753-af26-4632-8580-d4a47127a3b2@redhat.com> In-Reply-To: <11819753-af26-4632-8580-d4a47127a3b2@redhat.com> MIME-Version: 1.0 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jens Axboe , Keith Busch , Sagi Grimberg , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-nvme@lists.infradead.org" , Christoph Hellwig , "linux-block@vger.kernel.org" , =?utf-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Jason Gunthorpe , Bjorn Helgaas , Benjamin Herrenschmidt , Bjorn Helgaas , Max Gurtovoy , Dan Williams , Logan Gunthorpe , =?utf-8?B?Q2hyaXN0aWFuIEvDtm5pZw==?= Content-Type: text/plain; charset="us-ascii" Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+bjorn=helgaas.com@lists.infradead.org List-ID: Hi Don > RDMA VFs lend themselves to NVMEoF w/device-assignment.... need a way to > put NVME 'resources' into an assignable/manageable object for 'IOMMU-grouping', > which is really a 'DMA security domain' and less an 'IOMMU grouping domain'. Ha, I like your term "DMA Security Domain" which sounds about right for what we are discussing with p2pdma and ACS disablement ;-). The problem is that ACS is, in some ways, too big of hammer for what we want here in the sense that it is either on or off for the bridge or MF EP we enable/disable it for. ACS can't filter the TLPs by address or ID though PCI-SIG are having some discussions on extending ACS. That's a long term solution and won't be applicable to us for some time. NVMe SSDs that support SR-IOV are coming to market but we can't assume all NVMe SSDs with support SR-IOV. That will probably be a pretty high end-feature... Stephen _______________________________________________ Linux-nvme mailing list Linux-nvme@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-nvme From mboxrd@z Thu Jan 1 00:00:00 1970 From: sbates@raithlin.com (Stephen Bates) Date: Wed, 9 May 2018 12:44:21 +0000 Subject: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches In-Reply-To: <11819753-af26-4632-8580-d4a47127a3b2@redhat.com> References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> <3584a6ac-95c7-5d23-1859-aee30605776e@deltatee.com> <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508144341.0441b676@w520.home> <20180508152631.50fd583c@w520.home> <354F7407-0DC7-470C-B9AA-74FDF9C46B08@raithlin.com> <20180508160336.0935ddde@w520.home> <11819753-af26-4632-8580-d4a47127a3b2@redhat.com> Message-ID: Hi Don > RDMA VFs lend themselves to NVMEoF w/device-assignment.... need a way to > put NVME 'resources' into an assignable/manageable object for 'IOMMU-grouping', > which is really a 'DMA security domain' and less an 'IOMMU grouping domain'. Ha, I like your term "DMA Security Domain" which sounds about right for what we are discussing with p2pdma and ACS disablement ;-). The problem is that ACS is, in some ways, too big of hammer for what we want here in the sense that it is either on or off for the bridge or MF EP we enable/disable it for. ACS can't filter the TLPs by address or ID though PCI-SIG are having some discussions on extending ACS. That's a long term solution and won't be applicable to us for some time. NVMe SSDs that support SR-IOV are coming to market but we can't assume all NVMe SSDs with support SR-IOV. That will probably be a pretty high end-feature... Stephen