From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ale.deltatee.com (ale.deltatee.com [207.54.116.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id A41DF224E6926 for ; Thu, 1 Mar 2018 09:10:39 -0800 (PST) References: <20180228234006.21093-1-logang@deltatee.com> <20180228234006.21093-7-logang@deltatee.com> <72e90cc2-c05a-3ed8-42c7-e4d7a0d535d8@grimberg.me> From: Logan Gunthorpe Message-ID: <3527faf1-3f3a-88d0-caa8-d94d75437d73@deltatee.com> Date: Thu, 1 Mar 2018 10:16:24 -0700 MIME-Version: 1.0 In-Reply-To: <72e90cc2-c05a-3ed8-42c7-e4d7a0d535d8@grimberg.me> Content-Language: en-US Subject: Re: [PATCH v2 06/10] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]() List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Sagi Grimberg , linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@lists.01.org, linux-block@vger.kernel.org Cc: Jens Axboe , Benjamin Herrenschmidt , Alex Williamson , Keith Busch , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Jason Gunthorpe , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-ID: SGV5IFNhZ2ksCgpUaGFua3MgZm9yIHRoZSByZXZpZXchCgoKT24gMDEvMDMvMTggMDM6MzIgQU0s IFNhZ2kgR3JpbWJlcmcgd3JvdGU6Cj4+IMKgIGludCByZG1hX3J3X2N0eF9pbml0KHN0cnVjdCBy ZG1hX3J3X2N0eCAqY3R4LCBzdHJ1Y3QgaWJfcXAgKnFwLCB1OCAKPj4gcG9ydF9udW0sCj4+IMKg wqDCoMKgwqDCoMKgwqDCoCBzdHJ1Y3Qgc2NhdHRlcmxpc3QgKnNnLCB1MzIgc2dfY250LCB1MzIg c2dfb2Zmc2V0LAo+PiAtwqDCoMKgwqDCoMKgwqAgdTY0IHJlbW90ZV9hZGRyLCB1MzIgcmtleSwg ZW51bSBkbWFfZGF0YV9kaXJlY3Rpb24gZGlyKQo+PiArwqDCoMKgwqDCoMKgwqAgdTY0IHJlbW90 ZV9hZGRyLCB1MzIgcmtleSwgZW51bSBkbWFfZGF0YV9kaXJlY3Rpb24gZGlyLAo+PiArwqDCoMKg wqDCoMKgwqAgdW5zaWduZWQgaW50IGZsYWdzKQo+PiDCoCB7Cj4+IMKgwqDCoMKgwqAgc3RydWN0 IGliX2RldmljZSAqZGV2ID0gcXAtPnBkLT5kZXZpY2U7Cj4+IMKgwqDCoMKgwqAgaW50IHJldDsK Pj4gLcKgwqDCoCByZXQgPSBpYl9kbWFfbWFwX3NnKGRldiwgc2csIHNnX2NudCwgZGlyKTsKPj4g K8KgwqDCoCBpZiAoZmxhZ3MgJiBSRE1BX1JXX0NUWF9GTEFHX1BDSV9QMlBETUEpCj4+ICvCoMKg wqDCoMKgwqDCoCByZXQgPSBwY2lfcDJwZG1hX21hcF9zZyhkZXYtPmRtYV9kZXZpY2UsIHNnLCBz Z19jbnQsIGRpcik7Cj4+ICvCoMKgwqAgZWxzZQo+PiArwqDCoMKgwqDCoMKgwqAgcmV0ID0gaWJf ZG1hX21hcF9zZyhkZXYsIHNnLCBzZ19jbnQsIGRpcik7Cj4+ICsKPiAKPiBXaHkgbm90IHVzZSBp c19wY2lfcDJwZG1hX3BhZ2Uoc2cpIGluc3RlYWQgb2YgYSBmbGFnPyBJdCB3b3VsZCBiZSBzbwo+ IG11Y2ggY2xlYW5lci4uLgoKWWVzLCB0aGF0IHNvdW5kcyBsaWtlIGEgZ29vZCBpZGVhLiBXZSBj YW4gbWFrZSB0aGF0IGNoYW5nZSBmb3IgdjMuCgpMb2dhbgpfX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fXwpMaW51eC1udmRpbW0gbWFpbGluZyBsaXN0CkxpbnV4 LW52ZGltbUBsaXN0cy4wMS5vcmcKaHR0cHM6Ly9saXN0cy4wMS5vcmcvbWFpbG1hbi9saXN0aW5m by9saW51eC1udmRpbW0K From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: To: Sagi Grimberg , linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@lists.01.org, linux-block@vger.kernel.org Cc: Stephen Bates , Christoph Hellwig , Jens Axboe , Keith Busch , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Benjamin Herrenschmidt , Alex Williamson References: <20180228234006.21093-1-logang@deltatee.com> <20180228234006.21093-7-logang@deltatee.com> <72e90cc2-c05a-3ed8-42c7-e4d7a0d535d8@grimberg.me> From: Logan Gunthorpe Message-ID: <3527faf1-3f3a-88d0-caa8-d94d75437d73@deltatee.com> Date: Thu, 1 Mar 2018 10:16:24 -0700 MIME-Version: 1.0 In-Reply-To: <72e90cc2-c05a-3ed8-42c7-e4d7a0d535d8@grimberg.me> Content-Type: text/plain; charset=utf-8; format=flowed Subject: Re: [PATCH v2 06/10] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]() List-ID: Hey Sagi, Thanks for the review! On 01/03/18 03:32 AM, Sagi Grimberg wrote: >>   int rdma_rw_ctx_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 >> port_num, >>           struct scatterlist *sg, u32 sg_cnt, u32 sg_offset, >> -        u64 remote_addr, u32 rkey, enum dma_data_direction dir) >> +        u64 remote_addr, u32 rkey, enum dma_data_direction dir, >> +        unsigned int flags) >>   { >>       struct ib_device *dev = qp->pd->device; >>       int ret; >> -    ret = ib_dma_map_sg(dev, sg, sg_cnt, dir); >> +    if (flags & RDMA_RW_CTX_FLAG_PCI_P2PDMA) >> +        ret = pci_p2pdma_map_sg(dev->dma_device, sg, sg_cnt, dir); >> +    else >> +        ret = ib_dma_map_sg(dev, sg, sg_cnt, dir); >> + > > Why not use is_pci_p2pdma_page(sg) instead of a flag? It would be so > much cleaner... Yes, that sounds like a good idea. We can make that change for v3. Logan From mboxrd@z Thu Jan 1 00:00:00 1970 From: Logan Gunthorpe Subject: Re: [PATCH v2 06/10] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]() Date: Thu, 1 Mar 2018 10:16:24 -0700 Message-ID: <3527faf1-3f3a-88d0-caa8-d94d75437d73@deltatee.com> References: <20180228234006.21093-1-logang@deltatee.com> <20180228234006.21093-7-logang@deltatee.com> <72e90cc2-c05a-3ed8-42c7-e4d7a0d535d8@grimberg.me> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8"; Format="flowed" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <72e90cc2-c05a-3ed8-42c7-e4d7a0d535d8-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Sagi Grimberg , linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org, linux-block-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Cc: Jens Axboe , Benjamin Herrenschmidt , Alex Williamson , Keith Busch , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Jason Gunthorpe , Bjorn Helgaas , Max Gurtovoy , Christoph Hellwig List-Id: linux-rdma@vger.kernel.org SGV5IFNhZ2ksCgpUaGFua3MgZm9yIHRoZSByZXZpZXchCgoKT24gMDEvMDMvMTggMDM6MzIgQU0s IFNhZ2kgR3JpbWJlcmcgd3JvdGU6Cj4+IMKgIGludCByZG1hX3J3X2N0eF9pbml0KHN0cnVjdCBy ZG1hX3J3X2N0eCAqY3R4LCBzdHJ1Y3QgaWJfcXAgKnFwLCB1OCAKPj4gcG9ydF9udW0sCj4+IMKg wqDCoMKgwqDCoMKgwqDCoCBzdHJ1Y3Qgc2NhdHRlcmxpc3QgKnNnLCB1MzIgc2dfY250LCB1MzIg c2dfb2Zmc2V0LAo+PiAtwqDCoMKgwqDCoMKgwqAgdTY0IHJlbW90ZV9hZGRyLCB1MzIgcmtleSwg ZW51bSBkbWFfZGF0YV9kaXJlY3Rpb24gZGlyKQo+PiArwqDCoMKgwqDCoMKgwqAgdTY0IHJlbW90 ZV9hZGRyLCB1MzIgcmtleSwgZW51bSBkbWFfZGF0YV9kaXJlY3Rpb24gZGlyLAo+PiArwqDCoMKg wqDCoMKgwqAgdW5zaWduZWQgaW50IGZsYWdzKQo+PiDCoCB7Cj4+IMKgwqDCoMKgwqAgc3RydWN0 IGliX2RldmljZSAqZGV2ID0gcXAtPnBkLT5kZXZpY2U7Cj4+IMKgwqDCoMKgwqAgaW50IHJldDsK Pj4gLcKgwqDCoCByZXQgPSBpYl9kbWFfbWFwX3NnKGRldiwgc2csIHNnX2NudCwgZGlyKTsKPj4g K8KgwqDCoCBpZiAoZmxhZ3MgJiBSRE1BX1JXX0NUWF9GTEFHX1BDSV9QMlBETUEpCj4+ICvCoMKg wqDCoMKgwqDCoCByZXQgPSBwY2lfcDJwZG1hX21hcF9zZyhkZXYtPmRtYV9kZXZpY2UsIHNnLCBz Z19jbnQsIGRpcik7Cj4+ICvCoMKgwqAgZWxzZQo+PiArwqDCoMKgwqDCoMKgwqAgcmV0ID0gaWJf ZG1hX21hcF9zZyhkZXYsIHNnLCBzZ19jbnQsIGRpcik7Cj4+ICsKPiAKPiBXaHkgbm90IHVzZSBp c19wY2lfcDJwZG1hX3BhZ2Uoc2cpIGluc3RlYWQgb2YgYSBmbGFnPyBJdCB3b3VsZCBiZSBzbwo+ IG11Y2ggY2xlYW5lci4uLgoKWWVzLCB0aGF0IHNvdW5kcyBsaWtlIGEgZ29vZCBpZGVhLiBXZSBj YW4gbWFrZSB0aGF0IGNoYW5nZSBmb3IgdjMuCgpMb2dhbgpfX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fXwpMaW51eC1udmRpbW0gbWFpbGluZyBsaXN0CkxpbnV4 LW52ZGltbUBsaXN0cy4wMS5vcmcKaHR0cHM6Ly9saXN0cy4wMS5vcmcvbWFpbG1hbi9saXN0aW5m by9saW51eC1udmRpbW0K From mboxrd@z Thu Jan 1 00:00:00 1970 From: logang@deltatee.com (Logan Gunthorpe) Date: Thu, 1 Mar 2018 10:16:24 -0700 Subject: [PATCH v2 06/10] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]() In-Reply-To: <72e90cc2-c05a-3ed8-42c7-e4d7a0d535d8@grimberg.me> References: <20180228234006.21093-1-logang@deltatee.com> <20180228234006.21093-7-logang@deltatee.com> <72e90cc2-c05a-3ed8-42c7-e4d7a0d535d8@grimberg.me> Message-ID: <3527faf1-3f3a-88d0-caa8-d94d75437d73@deltatee.com> Hey Sagi, Thanks for the review! On 01/03/18 03:32 AM, Sagi Grimberg wrote: >> ? int rdma_rw_ctx_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 >> port_num, >> ????????? struct scatterlist *sg, u32 sg_cnt, u32 sg_offset, >> -??????? u64 remote_addr, u32 rkey, enum dma_data_direction dir) >> +??????? u64 remote_addr, u32 rkey, enum dma_data_direction dir, >> +??????? unsigned int flags) >> ? { >> ????? struct ib_device *dev = qp->pd->device; >> ????? int ret; >> -??? ret = ib_dma_map_sg(dev, sg, sg_cnt, dir); >> +??? if (flags & RDMA_RW_CTX_FLAG_PCI_P2PDMA) >> +??????? ret = pci_p2pdma_map_sg(dev->dma_device, sg, sg_cnt, dir); >> +??? else >> +??????? ret = ib_dma_map_sg(dev, sg, sg_cnt, dir); >> + > > Why not use is_pci_p2pdma_page(sg) instead of a flag? It would be so > much cleaner... Yes, that sounds like a good idea. We can make that change for v3. Logan