From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751116AbdCQP73 (ORCPT ); Fri, 17 Mar 2017 11:59:29 -0400 Received: from mx1.redhat.com ([209.132.183.28]:57220 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750999AbdCQP72 (ORCPT ); Fri, 17 Mar 2017 11:59:28 -0400 DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 6FE5781254 Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx01.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=jglisse@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 6FE5781254 Date: Fri, 17 Mar 2017 11:57:38 -0400 From: Jerome Glisse To: Bob Liu Cc: Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Hubbard , Naoya Horiguchi , David Nellans Subject: Re: [HMM 00/16] HMM (Heterogeneous Memory Management) v18 Message-ID: <20170317155737.GB7582@redhat.com> References: <1489680335-6594-1-git-send-email-jglisse@redhat.com> <20170316134321.c5cf727c21abf89b7e6708a2@linux-foundation.org> <20170316234950.GA5725@redhat.com> <3ff0fc0b-eb2a-a0d2-d8f6-82045a445324@huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <3ff0fc0b-eb2a-a0d2-d8f6-82045a445324@huawei.com> User-Agent: Mutt/1.7.1 (2016-10-04) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.25]); Fri, 17 Mar 2017 15:57:43 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Mar 17, 2017 at 04:29:10PM +0800, Bob Liu wrote: > On 2017/3/17 7:49, Jerome Glisse wrote: > > On Thu, Mar 16, 2017 at 01:43:21PM -0700, Andrew Morton wrote: > >> On Thu, 16 Mar 2017 12:05:19 -0400 J__r__me Glisse wrote: > >> > >>> Cliff note: > >> > >> "Cliff's notes" isn't appropriate for a large feature such as this. > >> Where's the long-form description? One which permits readers to fully > >> understand the requirements, design, alternative designs, the > >> implementation, the interface(s), etc? > >> > >> Have you ever spoken about HMM at a conference? If so, the supporting > >> presentation documents might help here. That's the level of detail > >> which should be presented here. > > > > Longer description of patchset rational, motivation and design choices > > were given in the first few posting of the patchset to which i included > > a link in my cover letter. Also given that i presented that for last 3 > > or 4 years to mm summit and kernel summit i thought that by now peoples > > were familiar about the topic and wanted to spare them the long version. > > My bad. > > > > I attach a patch that is a first stab at a Documentation/hmm.txt that > > explain the motivation and rational behind HMM. I can probably add a > > section about how to use HMM from device driver point of view. > > > > Please, that would be very helpful! > > > +3) Share address space and migration > > + > > +HMM intends to provide two main features. First one is to share the address > > +space by duplication the CPU page table into the device page table so same > > +address point to same memory and this for any valid main memory address in > > +the process address space. > > Is this an optional feature? > I mean the device don't have to duplicate the CPU page table. > But only make use of the second(migration) feature. Correct each feature can be use on its own without the other. > > +The second mechanism HMM provide is a new kind of ZONE_DEVICE memory that does > > +allow to allocate a struct page for each page of the device memory. Those page > > +are special because the CPU can not map them. They however allow to migrate > > +main memory to device memory using exhisting migration mechanism and everything > > +looks like if page was swap out to disk from CPU point of view. Using a struct > > +page gives the easiest and cleanest integration with existing mm mechanisms. > > +Again here HMM only provide helpers, first to hotplug new ZONE_DEVICE memory > > +for the device memory and second to perform migration. Policy decision of what > > +and when to migrate things is left to the device driver. > > + > > +Note that any CPU acess to a device page trigger a page fault which initiate a > > +migration back to system memory so that CPU can access it. > > A bit confused here, do you mean CPU access to a main memory page but that page has > been migrated to device memory? > Then a page fault will be triggered and initiate a migration back. If you migrate the page backing address A from a main memory page to a device page and then CPU try to access address A then you get a page fault because device memory is not accessible by CPU. The page fault is exactly as if the page was swap out to disk from kernel point of view. At any point in time there is only one and one page backing an address either a regular main memory page or device page. There is no change here to this fundamental fact in respect to mm. The only difference is that device page are not accessible by CPU. Cheers, Jérôme From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk0-f198.google.com (mail-qk0-f198.google.com [209.85.220.198]) by kanga.kvack.org (Postfix) with ESMTP id 601996B038C for ; Fri, 17 Mar 2017 11:57:44 -0400 (EDT) Received: by mail-qk0-f198.google.com with SMTP id j127so72110694qke.2 for ; Fri, 17 Mar 2017 08:57:44 -0700 (PDT) Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id h62si6744479qkc.61.2017.03.17.08.57.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 17 Mar 2017 08:57:43 -0700 (PDT) Date: Fri, 17 Mar 2017 11:57:38 -0400 From: Jerome Glisse Subject: Re: [HMM 00/16] HMM (Heterogeneous Memory Management) v18 Message-ID: <20170317155737.GB7582@redhat.com> References: <1489680335-6594-1-git-send-email-jglisse@redhat.com> <20170316134321.c5cf727c21abf89b7e6708a2@linux-foundation.org> <20170316234950.GA5725@redhat.com> <3ff0fc0b-eb2a-a0d2-d8f6-82045a445324@huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <3ff0fc0b-eb2a-a0d2-d8f6-82045a445324@huawei.com> Sender: owner-linux-mm@kvack.org List-ID: To: Bob Liu Cc: Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Hubbard , Naoya Horiguchi , David Nellans On Fri, Mar 17, 2017 at 04:29:10PM +0800, Bob Liu wrote: > On 2017/3/17 7:49, Jerome Glisse wrote: > > On Thu, Mar 16, 2017 at 01:43:21PM -0700, Andrew Morton wrote: > >> On Thu, 16 Mar 2017 12:05:19 -0400 J__r__me Glisse wrote: > >> > >>> Cliff note: > >> > >> "Cliff's notes" isn't appropriate for a large feature such as this. > >> Where's the long-form description? One which permits readers to fully > >> understand the requirements, design, alternative designs, the > >> implementation, the interface(s), etc? > >> > >> Have you ever spoken about HMM at a conference? If so, the supporting > >> presentation documents might help here. That's the level of detail > >> which should be presented here. > > > > Longer description of patchset rational, motivation and design choices > > were given in the first few posting of the patchset to which i included > > a link in my cover letter. Also given that i presented that for last 3 > > or 4 years to mm summit and kernel summit i thought that by now peoples > > were familiar about the topic and wanted to spare them the long version. > > My bad. > > > > I attach a patch that is a first stab at a Documentation/hmm.txt that > > explain the motivation and rational behind HMM. I can probably add a > > section about how to use HMM from device driver point of view. > > > > Please, that would be very helpful! > > > +3) Share address space and migration > > + > > +HMM intends to provide two main features. First one is to share the address > > +space by duplication the CPU page table into the device page table so same > > +address point to same memory and this for any valid main memory address in > > +the process address space. > > Is this an optional feature? > I mean the device don't have to duplicate the CPU page table. > But only make use of the second(migration) feature. Correct each feature can be use on its own without the other. > > +The second mechanism HMM provide is a new kind of ZONE_DEVICE memory that does > > +allow to allocate a struct page for each page of the device memory. Those page > > +are special because the CPU can not map them. They however allow to migrate > > +main memory to device memory using exhisting migration mechanism and everything > > +looks like if page was swap out to disk from CPU point of view. Using a struct > > +page gives the easiest and cleanest integration with existing mm mechanisms. > > +Again here HMM only provide helpers, first to hotplug new ZONE_DEVICE memory > > +for the device memory and second to perform migration. Policy decision of what > > +and when to migrate things is left to the device driver. > > + > > +Note that any CPU acess to a device page trigger a page fault which initiate a > > +migration back to system memory so that CPU can access it. > > A bit confused here, do you mean CPU access to a main memory page but that page has > been migrated to device memory? > Then a page fault will be triggered and initiate a migration back. If you migrate the page backing address A from a main memory page to a device page and then CPU try to access address A then you get a page fault because device memory is not accessible by CPU. The page fault is exactly as if the page was swap out to disk from kernel point of view. At any point in time there is only one and one page backing an address either a regular main memory page or device page. There is no change here to this fundamental fact in respect to mm. The only difference is that device page are not accessible by CPU. Cheers, Jerome -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org