From: Jerome Glisse <jglisse@redhat.com> To: Logan Gunthorpe <logang@deltatee.com> Cc: Dan Williams <dan.j.williams@intel.com>, Andi Kleen <ak@linux.intel.com>, Linux MM <linux-mm@kvack.org>, Andrew Morton <akpm@linux-foundation.org>, Linux Kernel Mailing List <linux-kernel@vger.kernel.org>, "Rafael J. Wysocki" <rafael@kernel.org>, Ross Zwisler <ross.zwisler@linux.intel.com>, Dave Hansen <dave.hansen@intel.com>, Haggai Eran <haggaie@mellanox.com>, balbirs@au1.ibm.com, "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>, Benjamin Herrenschmidt <benh@kernel.crashing.org>, "Kuehling, Felix" <felix.kuehling@amd.com>, Philip.Yang@amd.com, "Koenig, Christian" <christian.koenig@amd.com>, "Blinzer, Paul" <Paul.Blinzer@amd.com>, John Hubbard <jhubbard@nvidia.com>, rcampbell@nvidia.com Subject: Re: [RFC PATCH 02/14] mm/hms: heterogenenous memory system (HMS) documentation Date: Tue, 4 Dec 2018 14:22:21 -0500 [thread overview] Message-ID: <20181204192221.GG2937@redhat.com> (raw) In-Reply-To: <de7c1099-2717-6396-bf56-c4ab4085ee83@deltatee.com> On Tue, Dec 04, 2018 at 12:11:42PM -0700, Logan Gunthorpe wrote: > > > On 2018-12-04 11:57 a.m., Jerome Glisse wrote: > >> That sounds needlessly restrictive. Let the kernel arbitrate what > >> memory an application gets, don't design a system where applications > >> are hard coded to a memory type. Applications can hint, or optionally > >> specify an override and the kernel can react accordingly. > > > > You do not want to randomly use non cache coherent memory inside your > > application :) This is not gonna go well with C++ or atomic :) Yes they > > are legitimate use case where application can decide to give up cache > > coherency temporarily for a range of virtual address. But the application > > needs to understand what it is doing and opt in to do that knowing full > > well that. The version thing allows for scenario like. You do not have > > to define a new version with every new type of memory. If your new memory > > has all the properties of v1 than you expose it as v1 and old application > > on the new platform will use your new memory type being non the wiser. > > I agree with Dan and the general idea that this version thing is really > ugly. Define some standard attributes so the application can say "I want > cache-coherent, high bandwidth memory". If there's some future > new-memory attribute, then the application needs to know about it to > request it. So version is a bad prefix, what about type, prefixing target with a type id. So that application that are looking for a certain type of memory (which has a set of define properties) can select them. Having a type file inside the directory and hopping application will read that sysfs file is a recipies for failure from my point of view. While having it in the directory name is making sure that the application has some idea of what it is doing. > > Also, in the same vein, I think it's wrong to have the API enumerate all > the different memory available in the system. The API should simply > allow userspace to say it wants memory that can be accessed by a set of > initiators with a certain set of attributes and the bind call tries to > fulfill that or fallback on system memory/hmm migration/whatever. We have existing application that use topology today to partition their workload and do load balancing. Those application leverage the fact that they are only running on a small set of known platform with known topology here i want to provide a common API so that topology can be queried in a standard by application. Yes basic application will not leverage all this information and will be happy enough with give me memory that will be fast for initiator A and B. That can easily be implemented inside userspace library which dumbs down the topology on behalf of application. I believe that proposing a new infrastructure should allow for maximum expressiveness. The HMS API in this proposal allow to express any kind of directed graph hence i do not see any limitation going forward. At the same time userspace library can easily dumbs this down for average Joe/Jane application. Cheers, Jérôme
WARNING: multiple messages have this Message-ID (diff)
From: Jerome Glisse <jglisse@redhat.com> To: Logan Gunthorpe <logang@deltatee.com> Cc: Dan Williams <dan.j.williams@intel.com>, Andi Kleen <ak@linux.intel.com>, Linux MM <linux-mm@kvack.org>, Andrew Morton <akpm@linux-foundation.org>, Linux Kernel Mailing List <linux-kernel@vger.kernel.org>, "Rafael J. Wysocki" <rafael@kernel.org>, Ross Zwisler <ross.zwisler@linux.intel.com>, Dave Hansen <dave.hansen@intel.com>, Haggai Eran <haggaie@mellanox.com>, balbirs@au1.ibm.com, "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>, Benjamin Herrenschmidt <benh@kernel.crashing.org>, "Kuehling, Felix" <felix.kuehling@amd.com>, Philip.Yang@amd.com, "Koenig, Christian" <christian.koenig@amd.com>, "Blinzer, Paul" <Paul.Blinzer@amd.com>, John Hubbard <jhubbard@nvidia.com>, rcampbell@nvidia.com Subject: Re: [RFC PATCH 02/14] mm/hms: heterogenenous memory system (HMS) documentation Date: Tue, 4 Dec 2018 14:22:21 -0500 [thread overview] Message-ID: <20181204192221.GG2937@redhat.com> (raw) In-Reply-To: <de7c1099-2717-6396-bf56-c4ab4085ee83@deltatee.com> On Tue, Dec 04, 2018 at 12:11:42PM -0700, Logan Gunthorpe wrote: > > > On 2018-12-04 11:57 a.m., Jerome Glisse wrote: > >> That sounds needlessly restrictive. Let the kernel arbitrate what > >> memory an application gets, don't design a system where applications > >> are hard coded to a memory type. Applications can hint, or optionally > >> specify an override and the kernel can react accordingly. > > > > You do not want to randomly use non cache coherent memory inside your > > application :) This is not gonna go well with C++ or atomic :) Yes they > > are legitimate use case where application can decide to give up cache > > coherency temporarily for a range of virtual address. But the application > > needs to understand what it is doing and opt in to do that knowing full > > well that. The version thing allows for scenario like. You do not have > > to define a new version with every new type of memory. If your new memory > > has all the properties of v1 than you expose it as v1 and old application > > on the new platform will use your new memory type being non the wiser. > > I agree with Dan and the general idea that this version thing is really > ugly. Define some standard attributes so the application can say "I want > cache-coherent, high bandwidth memory". If there's some future > new-memory attribute, then the application needs to know about it to > request it. So version is a bad prefix, what about type, prefixing target with a type id. So that application that are looking for a certain type of memory (which has a set of define properties) can select them. Having a type file inside the directory and hopping application will read that sysfs file is a recipies for failure from my point of view. While having it in the directory name is making sure that the application has some idea of what it is doing. > > Also, in the same vein, I think it's wrong to have the API enumerate all > the different memory available in the system. The API should simply > allow userspace to say it wants memory that can be accessed by a set of > initiators with a certain set of attributes and the bind call tries to > fulfill that or fallback on system memory/hmm migration/whatever. We have existing application that use topology today to partition their workload and do load balancing. Those application leverage the fact that they are only running on a small set of known platform with known topology here i want to provide a common API so that topology can be queried in a standard by application. Yes basic application will not leverage all this information and will be happy enough with give me memory that will be fast for initiator A and B. That can easily be implemented inside userspace library which dumbs down the topology on behalf of application. I believe that proposing a new infrastructure should allow for maximum expressiveness. The HMS API in this proposal allow to express any kind of directed graph hence i do not see any limitation going forward. At the same time userspace library can easily dumbs this down for average Joe/Jane application. Cheers, J�r�me
next prev parent reply other threads:[~2018-12-04 19:22 UTC|newest] Thread overview: 171+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-12-03 23:34 [RFC PATCH 00/14] Heterogeneous Memory System (HMS) and hbind() jglisse 2018-12-03 23:34 ` jglisse 2018-12-03 23:34 ` [RFC PATCH 01/14] mm/hms: heterogeneous memory system (sysfs infrastructure) jglisse 2018-12-03 23:34 ` [RFC PATCH 02/14] mm/hms: heterogenenous memory system (HMS) documentation jglisse 2018-12-04 17:06 ` Andi Kleen 2018-12-04 17:06 ` Andi Kleen 2018-12-04 18:24 ` Jerome Glisse 2018-12-04 18:24 ` Jerome Glisse 2018-12-04 18:31 ` Dan Williams 2018-12-04 18:57 ` Jerome Glisse 2018-12-04 18:57 ` Jerome Glisse 2018-12-04 19:11 ` Logan Gunthorpe 2018-12-04 19:22 ` Jerome Glisse [this message] 2018-12-04 19:22 ` Jerome Glisse 2018-12-04 19:41 ` Logan Gunthorpe 2018-12-04 20:13 ` Jerome Glisse 2018-12-04 20:13 ` Jerome Glisse 2018-12-04 20:30 ` Logan Gunthorpe 2018-12-04 20:59 ` Jerome Glisse 2018-12-04 20:59 ` Jerome Glisse 2018-12-04 21:19 ` Logan Gunthorpe 2018-12-04 21:51 ` Jerome Glisse 2018-12-04 21:51 ` Jerome Glisse 2018-12-04 22:16 ` Logan Gunthorpe 2018-12-04 23:56 ` Jerome Glisse 2018-12-04 23:56 ` Jerome Glisse 2018-12-05 1:15 ` Logan Gunthorpe 2018-12-05 2:31 ` Jerome Glisse 2018-12-05 2:31 ` Jerome Glisse 2018-12-05 17:41 ` Logan Gunthorpe 2018-12-05 18:07 ` Jerome Glisse 2018-12-05 18:07 ` Jerome Glisse 2018-12-05 18:20 ` Logan Gunthorpe 2018-12-05 18:33 ` Jerome Glisse 2018-12-05 18:33 ` Jerome Glisse 2018-12-05 18:48 ` Logan Gunthorpe 2018-12-05 18:55 ` Jerome Glisse 2018-12-05 18:55 ` Jerome Glisse 2018-12-05 19:10 ` Logan Gunthorpe 2018-12-05 22:58 ` Jerome Glisse 2018-12-05 22:58 ` Jerome Glisse 2018-12-05 23:09 ` Logan Gunthorpe 2018-12-05 23:20 ` Jerome Glisse 2018-12-05 23:20 ` Jerome Glisse 2018-12-05 23:23 ` Logan Gunthorpe 2018-12-05 23:27 ` Jerome Glisse 2018-12-06 0:08 ` Dan Williams 2018-12-05 2:34 ` Dan Williams 2018-12-05 2:37 ` Jerome Glisse 2018-12-05 2:37 ` Jerome Glisse 2018-12-05 17:25 ` Logan Gunthorpe 2018-12-05 18:01 ` Jerome Glisse 2018-12-05 18:01 ` Jerome Glisse 2018-12-04 20:14 ` Andi Kleen 2018-12-04 20:47 ` Logan Gunthorpe 2018-12-04 21:15 ` Jerome Glisse 2018-12-04 21:15 ` Jerome Glisse 2018-12-05 0:54 ` Kuehling, Felix 2018-12-04 19:19 ` Dan Williams 2018-12-04 19:32 ` Jerome Glisse 2018-12-04 19:32 ` Jerome Glisse 2018-12-04 20:12 ` Andi Kleen 2018-12-04 20:41 ` Jerome Glisse 2018-12-04 20:41 ` Jerome Glisse 2018-12-05 4:36 ` Aneesh Kumar K.V 2018-12-05 4:41 ` Jerome Glisse 2018-12-05 4:41 ` Jerome Glisse 2018-12-05 10:52 ` Mike Rapoport 2018-12-05 10:52 ` Mike Rapoport 2018-12-03 23:34 ` [RFC PATCH 03/14] mm/hms: add target memory to heterogeneous memory system infrastructure jglisse 2018-12-03 23:34 ` [RFC PATCH 04/14] mm/hms: add initiator " jglisse 2018-12-03 23:35 ` [RFC PATCH 05/14] mm/hms: add link " jglisse 2018-12-03 23:35 ` [RFC PATCH 06/14] mm/hms: add bridge " jglisse 2018-12-03 23:35 ` [RFC PATCH 07/14] mm/hms: register main memory with heterogenenous memory system jglisse 2018-12-03 23:35 ` [RFC PATCH 08/14] mm/hms: register main CPUs " jglisse 2018-12-03 23:35 ` [RFC PATCH 09/14] mm/hms: hbind() for heterogeneous memory system (aka mbind() for HMS) jglisse 2018-12-03 23:35 ` [RFC PATCH 10/14] mm/hbind: add heterogeneous memory policy tracking infrastructure jglisse 2018-12-03 23:35 ` [RFC PATCH 11/14] mm/hbind: add bind command to heterogeneous memory policy jglisse 2018-12-03 23:35 ` [RFC PATCH 12/14] mm/hbind: add migrate command to hbind() ioctl jglisse 2018-12-03 23:35 ` [RFC PATCH 13/14] drm/nouveau: register GPU under heterogeneous memory system jglisse 2018-12-03 23:35 ` [RFC PATCH 14/14] test/hms: tests for " jglisse 2018-12-04 7:44 ` [RFC PATCH 00/14] Heterogeneous Memory System (HMS) and hbind() Aneesh Kumar K.V 2018-12-04 7:44 ` Aneesh Kumar K.V 2018-12-04 14:44 ` Jerome Glisse 2018-12-04 14:44 ` Jerome Glisse 2018-12-04 14:44 ` Jerome Glisse 2018-12-04 18:02 ` Dave Hansen 2018-12-04 18:02 ` Dave Hansen 2018-12-04 18:49 ` Jerome Glisse 2018-12-04 18:49 ` Jerome Glisse 2018-12-04 18:49 ` Jerome Glisse 2018-12-04 18:54 ` Dave Hansen 2018-12-04 18:54 ` Dave Hansen 2018-12-04 19:11 ` Jerome Glisse 2018-12-04 19:11 ` Jerome Glisse 2018-12-04 19:11 ` Jerome Glisse 2018-12-04 21:37 ` Dave Hansen 2018-12-04 21:37 ` Dave Hansen 2018-12-04 21:57 ` Jerome Glisse 2018-12-04 21:57 ` Jerome Glisse 2018-12-04 21:57 ` Jerome Glisse 2018-12-04 23:58 ` Dave Hansen 2018-12-04 23:58 ` Dave Hansen 2018-12-05 0:29 ` Jerome Glisse 2018-12-05 0:29 ` Jerome Glisse 2018-12-05 0:29 ` Jerome Glisse 2018-12-05 1:22 ` Kuehling, Felix 2018-12-05 1:22 ` Kuehling, Felix 2018-12-05 1:22 ` Kuehling, Felix 2018-12-05 11:27 ` Aneesh Kumar K.V 2018-12-05 11:27 ` Aneesh Kumar K.V 2018-12-05 16:09 ` Jerome Glisse 2018-12-05 16:09 ` Jerome Glisse 2018-12-05 16:09 ` Jerome Glisse 2018-12-04 23:54 ` Dave Hansen 2018-12-04 23:54 ` Dave Hansen 2018-12-05 0:15 ` Jerome Glisse 2018-12-05 0:15 ` Jerome Glisse 2018-12-05 0:15 ` Jerome Glisse 2018-12-05 1:06 ` Dave Hansen 2018-12-05 1:06 ` Dave Hansen 2018-12-05 2:13 ` Jerome Glisse 2018-12-05 2:13 ` Jerome Glisse 2018-12-05 2:13 ` Jerome Glisse 2018-12-05 17:27 ` Dave Hansen 2018-12-05 17:27 ` Dave Hansen 2018-12-05 17:53 ` Jerome Glisse 2018-12-05 17:53 ` Jerome Glisse 2018-12-05 17:53 ` Jerome Glisse 2018-12-06 18:25 ` Dave Hansen 2018-12-06 18:25 ` Dave Hansen 2018-12-06 19:20 ` Jerome Glisse 2018-12-06 19:20 ` Jerome Glisse 2018-12-06 19:20 ` Jerome Glisse 2018-12-06 19:31 ` Dave Hansen 2018-12-06 19:31 ` Dave Hansen 2018-12-06 20:11 ` Logan Gunthorpe 2018-12-06 20:11 ` Logan Gunthorpe 2018-12-06 22:04 ` Dave Hansen 2018-12-06 22:04 ` Dave Hansen 2018-12-06 22:39 ` Jerome Glisse 2018-12-06 22:39 ` Jerome Glisse 2018-12-06 22:39 ` Jerome Glisse 2018-12-06 23:09 ` Dave Hansen 2018-12-06 23:09 ` Dave Hansen 2018-12-06 23:28 ` Logan Gunthorpe 2018-12-06 23:28 ` Logan Gunthorpe 2018-12-06 23:34 ` Dave Hansen 2018-12-06 23:34 ` Dave Hansen 2018-12-06 23:38 ` Dave Hansen 2018-12-06 23:38 ` Dave Hansen 2018-12-06 23:48 ` Logan Gunthorpe 2018-12-06 23:48 ` Logan Gunthorpe 2018-12-07 0:20 ` Jerome Glisse 2018-12-07 0:20 ` Jerome Glisse 2018-12-07 0:20 ` Jerome Glisse 2018-12-07 15:06 ` Jonathan Cameron 2018-12-07 15:06 ` Jonathan Cameron 2018-12-07 15:06 ` Jonathan Cameron 2018-12-07 19:37 ` Jerome Glisse 2018-12-07 19:37 ` Jerome Glisse 2018-12-07 19:37 ` Jerome Glisse 2018-12-07 0:15 ` Jerome Glisse 2018-12-07 0:15 ` Jerome Glisse 2018-12-07 0:15 ` Jerome Glisse 2018-12-06 20:27 ` Jerome Glisse 2018-12-06 20:27 ` Jerome Glisse 2018-12-06 20:27 ` Jerome Glisse 2018-12-06 21:46 ` Jerome Glisse 2018-12-06 21:46 ` Jerome Glisse 2018-12-06 21:46 ` Jerome Glisse
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20181204192221.GG2937@redhat.com \ --to=jglisse@redhat.com \ --cc=Paul.Blinzer@amd.com \ --cc=Philip.Yang@amd.com \ --cc=ak@linux.intel.com \ --cc=akpm@linux-foundation.org \ --cc=aneesh.kumar@linux.ibm.com \ --cc=balbirs@au1.ibm.com \ --cc=benh@kernel.crashing.org \ --cc=christian.koenig@amd.com \ --cc=dan.j.williams@intel.com \ --cc=dave.hansen@intel.com \ --cc=felix.kuehling@amd.com \ --cc=haggaie@mellanox.com \ --cc=jhubbard@nvidia.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=logang@deltatee.com \ --cc=rafael@kernel.org \ --cc=rcampbell@nvidia.com \ --cc=ross.zwisler@linux.intel.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.