From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752602AbdFPSES (ORCPT ); Fri, 16 Jun 2017 14:04:18 -0400 Received: from mx1.redhat.com ([209.132.183.28]:58212 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752551AbdFPSEQ (ORCPT ); Fri, 16 Jun 2017 14:04:16 -0400 DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com D805B80C06 Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=jglisse@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com D805B80C06 Date: Fri, 16 Jun 2017 14:04:13 -0400 From: Jerome Glisse To: "Bridgman, John" Cc: "akpm@linux-foundation.org" , "linux-kernel@vger.kernel.org" , "linux-mm@kvack.org" , Dan Williams , "Kirill A . Shutemov" , John Hubbard , "Sander, Ben" , "Kuehling, Felix" Subject: Re: [HMM 00/15] HMM (Heterogeneous Memory Management) v23 Message-ID: <20170616180412.GC2420@redhat.com> References: <20170524172024.30810-1-jglisse@redhat.com> <20170616144737.GA2420@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.8.0 (2017-02-23) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Fri, 16 Jun 2017 18:04:16 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Jun 16, 2017 at 05:55:52PM +0000, Bridgman, John wrote: > >-----Original Message----- > >From: Jerome Glisse [mailto:jglisse@redhat.com] > >Sent: Friday, June 16, 2017 10:48 AM > >To: Bridgman, John > >Cc: akpm@linux-foundation.org; linux-kernel@vger.kernel.org; linux- > >mm@kvack.org; Dan Williams; Kirill A . Shutemov; John Hubbard; Sander, Ben; > >Kuehling, Felix > >Subject: Re: [HMM 00/15] HMM (Heterogeneous Memory Management) v23 > > > >On Fri, Jun 16, 2017 at 07:22:05AM +0000, Bridgman, John wrote: > >> Hi Jerome, > >> > >> I'm just getting back to this; sorry for the late responses. > >> > >> Your description of HMM talks about blocking CPU accesses when a page > >> has been migrated to device memory, and you treat that as a "given" in > >> the HMM design. Other than BAR limits, coherency between CPU and > >> device caches and performance on read-intensive CPU accesses to device > >> memory are there any other reasons for this ? > > > >Correct this is the list of reasons for it. Note that HMM is more of a toolboox > >that one monolithic thing. For instance you also have the HMM-CDM patchset > >that does allow to have GPU memory map to the CPU but this rely on CAPI or > >CCIX to keep same memory model garanty. > > > > > >> The reason I'm asking is that we make fairly heavy use of large BAR > >> support which allows the CPU to directly access all of the device > >> memory on each of the GPUs, albeit without cache coherency, and there > >> are some cases where it appears that allowing CPU access to the page > >> in device memory would be more efficient than constantly migrating > >> back and forth. > > > >The thing is we are designing toward random program and we can not make > >any assumption on what kind of instruction a program might run on such > >memory. So if program try to do atomic on it iirc it is un- define what is > >suppose to happen. > > Thanks... thought I was missing something from the list. Agree that we > need to provide consistent behaviour, and we definitely care about atomics. > If we could get consistent behaviour with the page still in device memory > are you aware of any other problems related to HMM itself ? Well only way to get consistent is with CCIX or CAPI bus, i would need to do an in depth reading of PCIE but from my memory this isn't doable with any of the existing PCIE standard. Note that i have HMM-CDM especially for the case you have cache coherent device memory that behave just like regular memory. When you use HMM-CDM and you migrate to GPU memory the page is still map into the CPU address space. HMM-CDM is a separate patchset that i posted couple days ago. So if you have cache coherent device memory that behave like regular memory what you want is HMM-CDM and when migrated thing are still map into the CPU page table. Jérôme