From mboxrd@z Thu Jan 1 00:00:00 1970 From: Benjamin Herrenschmidt Subject: Re: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory Date: Fri, 14 Apr 2017 21:39:52 +1000 Message-ID: <1492169992.25766.5.camel@au1.ibm.com> References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1491974532.7236.43.camel@kernel.crashing.org> <5ac22496-56ec-025d-f153-140001d2a7f9@deltatee.com> <1492034124.7236.77.camel@kernel.crashing.org> <81888a1e-eb0d-cbbc-dc66-0a09c32e4ea2@deltatee.com> <20170413232631.GB24910@bhelgaas-glaptop.roam.corp.google.com> <20170414041656.GA30694@obsidianresearch.com> <08c32f0d-6c7c-b65f-6453-dde0d7c173d1@deltatee.com> <1492169879.25766.4.camel@kernel.crashing.org> Reply-To: benh-8fk3Idey6ehBDgjK7y7TUQ@public.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1492169879.25766.4.camel-XVmvHMARGAS8U2dJNN8I7kB+6BGkLq7r@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Logan Gunthorpe , Jason Gunthorpe , Bjorn Helgaas Cc: Jens Axboe , "James E.J. Bottomley" , "Martin K. Petersen" , linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-pci-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Steve Wise , linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, Keith Busch , Jerome Glisse , linux-scsi-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nvdimm-y27Ovi1pjclAfugRpC6u6w@public.gmane.org, Max Gurtovoy , Christoph Hellwig List-Id: linux-nvdimm@lists.01.org On Fri, 2017-04-14 at 21:37 +1000, Benjamin Herrenschmidt wrote: > On Thu, 2017-04-13 at 22:40 -0600, Logan Gunthorpe wrote: > > > > On 13/04/17 10:16 PM, Jason Gunthorpe wrote: > > > I'd suggest just detecting if there is any translation in bus > > > addresses anywhere and just hard disabling P2P on such systems. > > > > That's a fantastic suggestion. It simplifies things significantly. > > Unless there are any significant objections I think I will plan on > > doing > > that. > > I object. Note: It would also make your stuff fundamentally incompatible with KVM guest pass-through since KVM plays with remapping BARs all over the place. Ben. > > > On modern hardware with 64 bit BARs there is very little reason > > > to > > > have translation, so I think this is a legacy feature. > > > > Yes, p2pmem users are likely to be designing systems around it (ie > > JBOFs) and not trying to shoehorn it onto legacy architectures. > > > > At the very least, it makes sense to leave it out and if someone > > comes > > along who cares they can put in the effort to support the address > > translation. > > > > Thanks, > > > > Logan From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752799AbdDNLlG (ORCPT ); Fri, 14 Apr 2017 07:41:06 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:34016 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751587AbdDNLlC (ORCPT ); Fri, 14 Apr 2017 07:41:02 -0400 Subject: Re: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory From: Benjamin Herrenschmidt Reply-To: benh@au1.ibm.com To: Logan Gunthorpe , Jason Gunthorpe , Bjorn Helgaas Cc: Christoph Hellwig , Sagi Grimberg , "James E.J. Bottomley" , "Martin K. Petersen" , Jens Axboe , Steve Wise , Stephen Bates , Max Gurtovoy , Dan Williams , Keith Busch , linux-pci@vger.kernel.org, linux-scsi@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org, linux-nvdimm@ml01.01.org, linux-kernel@vger.kernel.org, Jerome Glisse Date: Fri, 14 Apr 2017 21:39:52 +1000 In-Reply-To: <1492169879.25766.4.camel@kernel.crashing.org> References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1491974532.7236.43.camel@kernel.crashing.org> <5ac22496-56ec-025d-f153-140001d2a7f9@deltatee.com> <1492034124.7236.77.camel@kernel.crashing.org> <81888a1e-eb0d-cbbc-dc66-0a09c32e4ea2@deltatee.com> <20170413232631.GB24910@bhelgaas-glaptop.roam.corp.google.com> <20170414041656.GA30694@obsidianresearch.com> <08c32f0d-6c7c-b65f-6453-dde0d7c173d1@deltatee.com> <1492169879.25766.4.camel@kernel.crashing.org> Organization: IBM Australia Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.22.6 (3.22.6-1.fc25) Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-TM-AS-MML: disable x-cbid: 17041411-0044-0000-0000-00000241FED7 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17041411-0045-0000-0000-000006C907EE Message-Id: <1492169992.25766.5.camel@au1.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-04-14_07:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1702020001 definitions=main-1704140102 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, 2017-04-14 at 21:37 +1000, Benjamin Herrenschmidt wrote: > On Thu, 2017-04-13 at 22:40 -0600, Logan Gunthorpe wrote: > > > > On 13/04/17 10:16 PM, Jason Gunthorpe wrote: > > > I'd suggest just detecting if there is any translation in bus > > > addresses anywhere and just hard disabling P2P on such systems. > > > > That's a fantastic suggestion. It simplifies things significantly. > > Unless there are any significant objections I think I will plan on > > doing > > that. > > I object. Note: It would also make your stuff fundamentally incompatible with KVM guest pass-through since KVM plays with remapping BARs all over the place. Ben. > > > On modern hardware with 64 bit BARs there is very little reason > > > to > > > have translation, so I think this is a legacy feature. > > > > Yes, p2pmem users are likely to be designing systems around it (ie > > JBOFs) and not trying to shoehorn it onto legacy architectures. > > > > At the very least, it makes sense to leave it out and if someone > > comes > > along who cares they can put in the effort to support the address > > translation. > > > > Thanks, > > > > Logan From mboxrd@z Thu Jan 1 00:00:00 1970 From: benh@au1.ibm.com (Benjamin Herrenschmidt) Date: Fri, 14 Apr 2017 21:39:52 +1000 Subject: [RFC 0/8] Copy Offload with Peer-to-Peer PCI Memory In-Reply-To: <1492169879.25766.4.camel@kernel.crashing.org> References: <1490911959-5146-1-git-send-email-logang@deltatee.com> <1491974532.7236.43.camel@kernel.crashing.org> <5ac22496-56ec-025d-f153-140001d2a7f9@deltatee.com> <1492034124.7236.77.camel@kernel.crashing.org> <81888a1e-eb0d-cbbc-dc66-0a09c32e4ea2@deltatee.com> <20170413232631.GB24910@bhelgaas-glaptop.roam.corp.google.com> <20170414041656.GA30694@obsidianresearch.com> <08c32f0d-6c7c-b65f-6453-dde0d7c173d1@deltatee.com> <1492169879.25766.4.camel@kernel.crashing.org> Message-ID: <1492169992.25766.5.camel@au1.ibm.com> On Fri, 2017-04-14@21:37 +1000, Benjamin Herrenschmidt wrote: > On Thu, 2017-04-13@22:40 -0600, Logan Gunthorpe wrote: > > > > On 13/04/17 10:16 PM, Jason Gunthorpe wrote: > > > I'd suggest just detecting if there is any translation in bus > > > addresses anywhere and just hard disabling P2P on such systems. > > > > That's a fantastic suggestion. It simplifies things significantly. > > Unless there are any significant objections I think I will plan on > > doing > > that. > > I object. Note: It would also make your stuff fundamentally incompatible with KVM guest pass-through since KVM plays with remapping BARs all over the place. Ben. > > > On modern hardware with 64 bit BARs there is very little reason > > > to > > > have translation, so I think this is a legacy feature. > > > > Yes, p2pmem users are likely to be designing systems around it (ie > > JBOFs) and not trying to shoehorn it onto legacy architectures. > > > > At the very least, it makes sense to leave it out and if someone > > comes > > along who cares they can put in the effort to support the address > > translation. > > > > Thanks, > > > > Logan