From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Stephen Bates" Subject: Re: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory Date: Tue, 25 Apr 2017 21:23:30 +0000 Message-ID: References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1492034124.7236.77.camel@kernel.crashing.org> <81888a1e-eb0d-cbbc-dc66-0a09c32e4ea2@deltatee.com> <20170413232631.GB24910@bhelgaas-glaptop.roam.corp.google.com> <20170414041656.GA30694@obsidianresearch.com> <1492169849.25766.3.camel@kernel.crashing.org> <630c1c63-ff17-1116-e069-2b8f93e50fa2@deltatee.com> <20170414190452.GA15679@bhelgaas-glaptop.roam.corp.google.com> <1492207643.25766.18.camel@kernel.crashing.org> <1492311719.25766.37.camel@kernel.crashing.org> <5e43818e-8c6b-8be8-23ff-b798633d2a73@deltatee.com> <1492381907.25766.49.camel@kernel.crashing.org> <1493019397.3171.118.camel@oracle.com> <9b6c0830-a728-c7ca-e6c6-2135f3f760ed@deltatee.com> <1493101803.3171.246.camel@oracle.com> <0cc95df5-b9dd-6493-15fe-771d535c1020@deltatee.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <0cc95df5-b9dd-6493-15fe-771d535c1020-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org> Content-Language: en-US Content-ID: <2DE64262AC700A45B4EEDB3B78B07FB4-JkAt9bkEularoOM5E8FhRbjFIynDaujOfM0AETQt39g@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Logan Gunthorpe , Knut Omang , Benjamin Herrenschmidt , Dan Williams Cc: Jens Axboe , Keith Busch , "James E.J. Bottomley" , "Martin K. Petersen" , "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Steve Wise , "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org" , Jason Gunthorpe , Jerome Glisse , Bjorn Helgaas , linux-scsi , linux-nvdimm , Max Gurtovoy , Christoph Hellwig List-Id: linux-nvdimm@lists.01.org Pj4gWWVzLCB0aGF0J3Mgd2h5IEkgdXNlZCAnc2lnbmlmaWNhbnQnLiBPbmUgZ29vZCB0aGluZyBp cyB0aGF0IGdpdmVuIHJlc291cmNlcyANCj4+IGl0IGNhbiBlYXNpbHkgYmUgZG9uZSBpbiBwYXJh bGxlbCB3aXRoIG90aGVyIGRldmVsb3BtZW50LCBhbmQgd2lsbCBnaXZlIGFkZGl0aW9uYWwNCj4+ IGluc2lnaHQgb2Ygc29tZSBmb3JtLg0KPg0KPll1cCwgd2VsbCBpZiBzb21lb25lIHdhbnRzIHRv IHN0YXJ0IHdvcmtpbmcgb24gYW4gZW11bGF0ZWQgUkRNQSBkZXZpY2UNCj50aGF0IGFjdHVhbGx5 IHNpbXVsYXRlcyBwcm9wZXIgRE1BIHRyYW5zZmVycyB0aGF0IHdvdWxkIGJlIGdyZWF0IQ0KDQpH aXZlIHRoYXQgZWFjaCBSRE1BIHZlbmRvcuKAmXMgZGV2aWNlcyBleHBvc2UgYSBkaWZmZXJlbnQg TU1JTyBJIGRvbuKAmXQgZXhwZWN0IHRoaXMgdG8gaGFwcGVuIGFueXRpbWUgc29vbi4NCg0KPiBZ ZXMsIHRoZSBudm1lIGRldmljZSBpbiBxZW11IGhhcyBhIENNQiBidWZmZXIgd2hpY2ggaXMgYSBn b29kIGNob2ljZSB0bw0KPiB0ZXN0IHdpdGggYnV0IHdlIGRvbid0IGhhdmUgY29kZSB0byB1c2Ug aXQgZm9yIHAycCB0cmFuc2ZlcnMgaW4gdGhlDQo+a2VybmVsIHNvIGl0IGlzIGEgYml0IGF3a3dh cmQuDQoNCk5vdGUgdGhlIENNQiBjb2RlIGlzIG5vdCBpbiB1cHN0cmVhbSBRRU1VLCBpdOKAmXMg aW4gS2VpdGjigJlzIGZvcmsgWzFdLiBJIHdpbGwgc2VlIGlmIEkgY2FuIHB1c2ggdGhpcyB1cHN0 cmVhbS4NCg0KU3RlcGhlbg0KDQpbMV0gZ2l0Oi8vZ2l0LmluZnJhZGVhZC5vcmcvdXNlcnMva2J1 c2NoL3FlbXUtbnZtZS5naXQNCg0KDQpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fXwpMaW51eC1udmRpbW0gbWFpbGluZyBsaXN0CkxpbnV4LW52ZGltbUBsaXN0 cy4wMS5vcmcKaHR0cHM6Ly9saXN0cy4wMS5vcmcvbWFpbG1hbi9saXN0aW5mby9saW51eC1udmRp bW0K From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1954876AbdDYVXn (ORCPT ); Tue, 25 Apr 2017 17:23:43 -0400 Received: from mail-eopbgr670127.outbound.protection.outlook.com ([40.107.67.127]:49568 "EHLO CAN01-TO1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1954854AbdDYVXd (ORCPT ); Tue, 25 Apr 2017 17:23:33 -0400 From: "Stephen Bates" To: Logan Gunthorpe , Knut Omang , Benjamin Herrenschmidt , Dan Williams CC: Bjorn Helgaas , Jason Gunthorpe , Christoph Hellwig , "Sagi Grimberg" , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , "Max Gurtovoy" , Keith Busch , "linux-pci@vger.kernel.org" , linux-scsi , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , linux-nvdimm , "linux-kernel@vger.kernel.org" , Jerome Glisse Subject: Re: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory Thread-Topic: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory Thread-Index: AQHSqaLO8AzeU4AEL0W+Kr5FXh2edaHBRjQAgADFuoCAAE/FAIABiQcAgAAiw4CAAFElAIAAexaAgABijwCAABpxAIAAMv6AgAFIEwCAAEq/AIAAUdKAgADXooCAAAt/AIAAY7aAgAuYi4CAAJC4gIAA7wSAgACw+ID//9TqAA== Date: Tue, 25 Apr 2017 21:23:30 +0000 Message-ID: References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1492034124.7236.77.camel@kernel.crashing.org> <81888a1e-eb0d-cbbc-dc66-0a09c32e4ea2@deltatee.com> <20170413232631.GB24910@bhelgaas-glaptop.roam.corp.google.com> <20170414041656.GA30694@obsidianresearch.com> <1492169849.25766.3.camel@kernel.crashing.org> <630c1c63-ff17-1116-e069-2b8f93e50fa2@deltatee.com> <20170414190452.GA15679@bhelgaas-glaptop.roam.corp.google.com> <1492207643.25766.18.camel@kernel.crashing.org> <1492311719.25766.37.camel@kernel.crashing.org> <5e43818e-8c6b-8be8-23ff-b798633d2a73@deltatee.com> <1492381907.25766.49.camel@kernel.crashing.org> <1493019397.3171.118.camel@oracle.com> <9b6c0830-a728-c7ca-e6c6-2135f3f760ed@deltatee.com> <1493101803.3171.246.camel@oracle.com> <0cc95df5-b9dd-6493-15fe-771d535c1020@deltatee.com> In-Reply-To: <0cc95df5-b9dd-6493-15fe-771d535c1020@deltatee.com> Accept-Language: en-CA, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/f.20.0.170309 authentication-results: deltatee.com; dkim=none (message not signed) header.d=none;deltatee.com; dmarc=none action=none header.from=raithlin.com; x-originating-ip: [70.65.232.47] x-microsoft-exchange-diagnostics: 1;YTOPR01MB0619;7:zU6Q7KQZoZoPihA3cVOgfFBoG087TCj1WUB8OnVyA+m/qB9hLRseNAZLFqCxCCLyc2S1h7mZtSYR/EPKlF2zdPZWCJZHZKsCN6MsD97gQRBK39bkoCqwKSxNgdYXcXbVn1EVUJVzciJSNVcn6mBIrZEZEdKmkpnjiR25kaoCgmiTbX2zN9y4IlgSVG2kuHD3csLnL4OdB/OymcZiCW2vPxKMYZ6Xpeskfk9rkieBnYmsA8okWFBhxIrTxac0UyT7Vt9xs5mfBIiC5b/t31gY1NCX3N+bczr2POnh1HPlV84784wouztZ0UBP8WslYDhRtrBqPbCeXA/cbQlkyaMMUg== x-ms-office365-filtering-correlation-id: f19c376b-e951-4435-0a45-08d48c2151fd x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(2017030254075)(201703131423075)(201702281549075);SRVR:YTOPR01MB0619; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(244540007438412)(155532106045638); x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(6040450)(2401047)(8121501046)(5005006)(93006095)(93001095)(3002001)(10201501046)(6041248)(20161123560025)(2016111802025)(20161123555025)(20161123564025)(20161123562025)(201703131423075)(201702281528075)(201703061421075)(6043046)(6072148);SRVR:YTOPR01MB0619;BCL:0;PCL:0;RULEID:;SRVR:YTOPR01MB0619; x-forefront-prvs: 0288CD37D9 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(39400400002)(39410400002)(39830400002)(39450400003)(8676002)(36756003)(4326008)(7736002)(7416002)(83506001)(38730400002)(305945005)(54906002)(81166006)(8936002)(2906002)(189998001)(82746002)(4001350100001)(83716003)(66066001)(33656002)(86362001)(25786009)(3280700002)(54356999)(2900100001)(6246003)(122556002)(3660700001)(6512007)(6436002)(6506006)(230783001)(102836003)(6116002)(6486002)(5660300001)(53936002)(229853002)(2950100002)(50986999)(3846002)(76176999)(93886004)(77096006);DIR:OUT;SFP:1102;SCL:1;SRVR:YTOPR01MB0619;H:YTOPR01MB0619.CANPRD01.PROD.OUTLOOK.COM;FPR:;SPF:None;MLV:sfv;LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: <2DE64262AC700A45B4EEDB3B78B07FB4@CANPRD01.PROD.OUTLOOK.COM> MIME-Version: 1.0 X-OriginatorOrg: raithlin.com X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Apr 2017 21:23:30.2669 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 18519031-7ff4-4cbb-bbcb-c3252d330f4b X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR01MB0619 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id v3PLOYbo007718 >> Yes, that's why I used 'significant'. One good thing is that given resources >> it can easily be done in parallel with other development, and will give additional >> insight of some form. > >Yup, well if someone wants to start working on an emulated RDMA device >that actually simulates proper DMA transfers that would be great! Give that each RDMA vendor’s devices expose a different MMIO I don’t expect this to happen anytime soon. > Yes, the nvme device in qemu has a CMB buffer which is a good choice to > test with but we don't have code to use it for p2p transfers in the >kernel so it is a bit awkward. Note the CMB code is not in upstream QEMU, it’s in Keith’s fork [1]. I will see if I can push this upstream. Stephen [1] git://git.infradead.org/users/kbusch/qemu-nvme.git From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Return-Path: From: "Stephen Bates" To: Logan Gunthorpe , Knut Omang , Benjamin Herrenschmidt , Dan Williams CC: Bjorn Helgaas , Jason Gunthorpe , Christoph Hellwig , "Sagi Grimberg" , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , "Max Gurtovoy" , Keith Busch , "linux-pci@vger.kernel.org" , linux-scsi , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , linux-nvdimm , "linux-kernel@vger.kernel.org" , Jerome Glisse Subject: Re: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory Date: Tue, 25 Apr 2017 21:23:30 +0000 Message-ID: References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1492034124.7236.77.camel@kernel.crashing.org> <81888a1e-eb0d-cbbc-dc66-0a09c32e4ea2@deltatee.com> <20170413232631.GB24910@bhelgaas-glaptop.roam.corp.google.com> <20170414041656.GA30694@obsidianresearch.com> <1492169849.25766.3.camel@kernel.crashing.org> <630c1c63-ff17-1116-e069-2b8f93e50fa2@deltatee.com> <20170414190452.GA15679@bhelgaas-glaptop.roam.corp.google.com> <1492207643.25766.18.camel@kernel.crashing.org> <1492311719.25766.37.camel@kernel.crashing.org> <5e43818e-8c6b-8be8-23ff-b798633d2a73@deltatee.com> <1492381907.25766.49.camel@kernel.crashing.org> <1493019397.3171.118.camel@oracle.com> <9b6c0830-a728-c7ca-e6c6-2135f3f760ed@deltatee.com> <1493101803.3171.246.camel@oracle.com> <0cc95df5-b9dd-6493-15fe-771d535c1020@deltatee.com> In-Reply-To: <0cc95df5-b9dd-6493-15fe-771d535c1020@deltatee.com> Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 List-ID: Pj4gWWVzLCB0aGF0J3Mgd2h5IEkgdXNlZCAnc2lnbmlmaWNhbnQnLiBPbmUgZ29vZCB0aGluZyBp cyB0aGF0IGdpdmVuIHJlc291cmNlcyANCj4+IGl0IGNhbiBlYXNpbHkgYmUgZG9uZSBpbiBwYXJh bGxlbCB3aXRoIG90aGVyIGRldmVsb3BtZW50LCBhbmQgd2lsbCBnaXZlIGFkZGl0aW9uYWwNCj4+ IGluc2lnaHQgb2Ygc29tZSBmb3JtLg0KPg0KPll1cCwgd2VsbCBpZiBzb21lb25lIHdhbnRzIHRv IHN0YXJ0IHdvcmtpbmcgb24gYW4gZW11bGF0ZWQgUkRNQSBkZXZpY2UNCj50aGF0IGFjdHVhbGx5 IHNpbXVsYXRlcyBwcm9wZXIgRE1BIHRyYW5zZmVycyB0aGF0IHdvdWxkIGJlIGdyZWF0IQ0KDQpH aXZlIHRoYXQgZWFjaCBSRE1BIHZlbmRvcuKAmXMgZGV2aWNlcyBleHBvc2UgYSBkaWZmZXJlbnQg TU1JTyBJIGRvbuKAmXQgZXhwZWN0IHRoaXMgdG8gaGFwcGVuIGFueXRpbWUgc29vbi4NCg0KPiBZ ZXMsIHRoZSBudm1lIGRldmljZSBpbiBxZW11IGhhcyBhIENNQiBidWZmZXIgd2hpY2ggaXMgYSBn b29kIGNob2ljZSB0bw0KPiB0ZXN0IHdpdGggYnV0IHdlIGRvbid0IGhhdmUgY29kZSB0byB1c2Ug aXQgZm9yIHAycCB0cmFuc2ZlcnMgaW4gdGhlDQo+a2VybmVsIHNvIGl0IGlzIGEgYml0IGF3a3dh cmQuDQoNCk5vdGUgdGhlIENNQiBjb2RlIGlzIG5vdCBpbiB1cHN0cmVhbSBRRU1VLCBpdOKAmXMg aW4gS2VpdGjigJlzIGZvcmsgWzFdLiBJIHdpbGwgc2VlIGlmIEkgY2FuIHB1c2ggdGhpcyB1cHN0 cmVhbS4NCg0KU3RlcGhlbg0KDQpbMV0gZ2l0Oi8vZ2l0LmluZnJhZGVhZC5vcmcvdXNlcnMva2J1 c2NoL3FlbXUtbnZtZS5naXQNCg0KDQo= From mboxrd@z Thu Jan 1 00:00:00 1970 From: sbates@raithlin.com (Stephen Bates) Date: Tue, 25 Apr 2017 21:23:30 +0000 Subject: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory In-Reply-To: <0cc95df5-b9dd-6493-15fe-771d535c1020@deltatee.com> References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1492034124.7236.77.camel@kernel.crashing.org> <81888a1e-eb0d-cbbc-dc66-0a09c32e4ea2@deltatee.com> <20170413232631.GB24910@bhelgaas-glaptop.roam.corp.google.com> <20170414041656.GA30694@obsidianresearch.com> <1492169849.25766.3.camel@kernel.crashing.org> <630c1c63-ff17-1116-e069-2b8f93e50fa2@deltatee.com> <20170414190452.GA15679@bhelgaas-glaptop.roam.corp.google.com> <1492207643.25766.18.camel@kernel.crashing.org> <1492311719.25766.37.camel@kernel.crashing.org> <5e43818e-8c6b-8be8-23ff-b798633d2a73@deltatee.com> <1492381907.25766.49.camel@kernel.crashing.org> <1493019397.3171.118.camel@oracle.com> <9b6c0830-a728-c7ca-e6c6-2135f3f760ed@deltatee.com> <1493101803.3171.246.camel@oracle.com> <0cc95df5-b9dd-6493-15fe-771d535c1020@deltatee.com> Message-ID: >> Yes, that's why I used 'significant'. One good thing is that given resources >> it can easily be done in parallel with other development, and will give additional >> insight of some form. > >Yup, well if someone wants to start working on an emulated RDMA device >that actually simulates proper DMA transfers that would be great! Give that each RDMA vendor?s devices expose a different MMIO I don?t expect this to happen anytime soon. > Yes, the nvme device in qemu has a CMB buffer which is a good choice to > test with but we don't have code to use it for p2p transfers in the >kernel so it is a bit awkward. Note the CMB code is not in upstream QEMU, it?s in Keith?s fork [1]. I will see if I can push this upstream. Stephen [1] git://git.infradead.org/users/kbusch/qemu-nvme.git