From: Jerome Glisse <jglisse@redhat.com> To: Logan Gunthorpe <logang@deltatee.com> Cc: Dan Williams <dan.j.williams@intel.com>, Andi Kleen <ak@linux.intel.com>, Linux MM <linux-mm@kvack.org>, Andrew Morton <akpm@linux-foundation.org>, Linux Kernel Mailing List <linux-kernel@vger.kernel.org>, "Rafael J. Wysocki" <rafael@kernel.org>, Dave Hansen <dave.hansen@intel.com>, Haggai Eran <haggaie@mellanox.com>, balbirs@au1.ibm.com, "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>, Benjamin Herrenschmidt <benh@kernel.crashing.org>, "Kuehling, Felix" <felix.kuehling@amd.com>, Philip.Yang@amd.com, "Koenig, Christian" <christian.koenig@amd.com>, "Blinzer, Paul" <Paul.Blinzer@amd.com>, John Hubbard <jhubbard@nvidia.com>, rcampbell@nvidia.com Subject: Re: [RFC PATCH 02/14] mm/hms: heterogenenous memory system (HMS) documentation Date: Wed, 5 Dec 2018 13:07:57 -0500 [thread overview] Message-ID: <20181205180756.GI3536@redhat.com> (raw) In-Reply-To: <a5ae63ff-a913-25af-4648-4ebf91775412@deltatee.com> On Wed, Dec 05, 2018 at 10:41:56AM -0700, Logan Gunthorpe wrote: > > > On 2018-12-04 7:31 p.m., Jerome Glisse wrote: > > How can i express multiple link, or memory that is only accessible > > by a subset of the devices/CPUs. In today model they are back in > > assumption like everyone can access all the node which do not hold > > in what i am trying to do. > > Well multiple links are easy when you have a 'link' bus. Just add > another link device under the bus. So you are telling do what i am doing in this patchset but not under HMS directory ? > > Technically, the accessibility issue is already encoded in sysfs. For > example, through the PCI tree you can determine which ACS bits are set > and determine which devices are behind the same root bridge the same way > we do in the kernel p2pdma subsystem. This is all bus specific which is > fine, but if we want to change that, we should have a common way for > existing buses to describe these attributes in the existing tree. The > new 'link' bus devices would have to have some way to describe cases if > memory isn't accessible in some way across it. What i am looking at is much more complex than just access bit. It is a whole set of properties attach to each path (can it be cache coherent ? can it do atomic ? what is the access granularity ? what is the bandwidth ? is it dedicated link ? ...) > > But really, I would say the kernel is responsible for telling you when > memory is accessible to a list of initiators, so it should be part of > the checks in a theoretical hbind api. This is already the approach > p2pdma takes in-kernel: we have functions that tell you if two PCI > devices can talk to each other and we have functions to give you memory > accessible by a set of devices. What we don't have is a special tree > that p2pdma users have to walk through to determine accessibility. You do not need it, but i do need it they are user out there that are already depending on the information by getting it through non standard way. I do want to provide a standard way for userspace to get this. They are real user out there and i believe their would be more user if we had a standard way to provide it. You do not believe in it fine. I will do more work in userspace and more example and i will come back with more hard evidence until i convince enough people. > > In my eye's, you are just conflating a bunch of different issues that > are better solved independently in the existing frameworks we have. And > if they were tackled individually, you'd have a much easier time getting > them merged one by one. I don't think i can convince you otherwise. They are user that use topology please looks at the links i provided, those folks have running program _today_ they rely on non standard API and would like to move toward standard API it would improve their life. On top of that i argue that more people would use that information if it were available to them. I agree that i have no hard evidence to back that up and that it is just a feeling but you can not disprove me either as this is a chicken and egg problem, you can not prove people will not use an API if the API is not there to be use. Cheers, Jérôme
WARNING: multiple messages have this Message-ID (diff)
From: Jerome Glisse <jglisse@redhat.com> To: Logan Gunthorpe <logang@deltatee.com> Cc: Dan Williams <dan.j.williams@intel.com>, Andi Kleen <ak@linux.intel.com>, Linux MM <linux-mm@kvack.org>, Andrew Morton <akpm@linux-foundation.org>, Linux Kernel Mailing List <linux-kernel@vger.kernel.org>, "Rafael J. Wysocki" <rafael@kernel.org>, Dave Hansen <dave.hansen@intel.com>, Haggai Eran <haggaie@mellanox.com>, balbirs@au1.ibm.com, "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>, Benjamin Herrenschmidt <benh@kernel.crashing.org>, "Kuehling, Felix" <felix.kuehling@amd.com>, Philip.Yang@amd.com, "Koenig, Christian" <christian.koenig@amd.com>, "Blinzer, Paul" <Paul.Blinzer@amd.com>, John Hubbard <jhubbard@nvidia.com>, rcampbell@nvidia.com Subject: Re: [RFC PATCH 02/14] mm/hms: heterogenenous memory system (HMS) documentation Date: Wed, 5 Dec 2018 13:07:57 -0500 [thread overview] Message-ID: <20181205180756.GI3536@redhat.com> (raw) In-Reply-To: <a5ae63ff-a913-25af-4648-4ebf91775412@deltatee.com> On Wed, Dec 05, 2018 at 10:41:56AM -0700, Logan Gunthorpe wrote: > > > On 2018-12-04 7:31 p.m., Jerome Glisse wrote: > > How can i express multiple link, or memory that is only accessible > > by a subset of the devices/CPUs. In today model they are back in > > assumption like everyone can access all the node which do not hold > > in what i am trying to do. > > Well multiple links are easy when you have a 'link' bus. Just add > another link device under the bus. So you are telling do what i am doing in this patchset but not under HMS directory ? > > Technically, the accessibility issue is already encoded in sysfs. For > example, through the PCI tree you can determine which ACS bits are set > and determine which devices are behind the same root bridge the same way > we do in the kernel p2pdma subsystem. This is all bus specific which is > fine, but if we want to change that, we should have a common way for > existing buses to describe these attributes in the existing tree. The > new 'link' bus devices would have to have some way to describe cases if > memory isn't accessible in some way across it. What i am looking at is much more complex than just access bit. It is a whole set of properties attach to each path (can it be cache coherent ? can it do atomic ? what is the access granularity ? what is the bandwidth ? is it dedicated link ? ...) > > But really, I would say the kernel is responsible for telling you when > memory is accessible to a list of initiators, so it should be part of > the checks in a theoretical hbind api. This is already the approach > p2pdma takes in-kernel: we have functions that tell you if two PCI > devices can talk to each other and we have functions to give you memory > accessible by a set of devices. What we don't have is a special tree > that p2pdma users have to walk through to determine accessibility. You do not need it, but i do need it they are user out there that are already depending on the information by getting it through non standard way. I do want to provide a standard way for userspace to get this. They are real user out there and i believe their would be more user if we had a standard way to provide it. You do not believe in it fine. I will do more work in userspace and more example and i will come back with more hard evidence until i convince enough people. > > In my eye's, you are just conflating a bunch of different issues that > are better solved independently in the existing frameworks we have. And > if they were tackled individually, you'd have a much easier time getting > them merged one by one. I don't think i can convince you otherwise. They are user that use topology please looks at the links i provided, those folks have running program _today_ they rely on non standard API and would like to move toward standard API it would improve their life. On top of that i argue that more people would use that information if it were available to them. I agree that i have no hard evidence to back that up and that it is just a feeling but you can not disprove me either as this is a chicken and egg problem, you can not prove people will not use an API if the API is not there to be use. Cheers, J�r�me
next prev parent reply other threads:[~2018-12-05 18:08 UTC|newest] Thread overview: 171+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-12-03 23:34 [RFC PATCH 00/14] Heterogeneous Memory System (HMS) and hbind() jglisse 2018-12-03 23:34 ` jglisse 2018-12-03 23:34 ` [RFC PATCH 01/14] mm/hms: heterogeneous memory system (sysfs infrastructure) jglisse 2018-12-03 23:34 ` [RFC PATCH 02/14] mm/hms: heterogenenous memory system (HMS) documentation jglisse 2018-12-04 17:06 ` Andi Kleen 2018-12-04 17:06 ` Andi Kleen 2018-12-04 18:24 ` Jerome Glisse 2018-12-04 18:24 ` Jerome Glisse 2018-12-04 18:31 ` Dan Williams 2018-12-04 18:57 ` Jerome Glisse 2018-12-04 18:57 ` Jerome Glisse 2018-12-04 19:11 ` Logan Gunthorpe 2018-12-04 19:22 ` Jerome Glisse 2018-12-04 19:22 ` Jerome Glisse 2018-12-04 19:41 ` Logan Gunthorpe 2018-12-04 20:13 ` Jerome Glisse 2018-12-04 20:13 ` Jerome Glisse 2018-12-04 20:30 ` Logan Gunthorpe 2018-12-04 20:59 ` Jerome Glisse 2018-12-04 20:59 ` Jerome Glisse 2018-12-04 21:19 ` Logan Gunthorpe 2018-12-04 21:51 ` Jerome Glisse 2018-12-04 21:51 ` Jerome Glisse 2018-12-04 22:16 ` Logan Gunthorpe 2018-12-04 23:56 ` Jerome Glisse 2018-12-04 23:56 ` Jerome Glisse 2018-12-05 1:15 ` Logan Gunthorpe 2018-12-05 2:31 ` Jerome Glisse 2018-12-05 2:31 ` Jerome Glisse 2018-12-05 17:41 ` Logan Gunthorpe 2018-12-05 18:07 ` Jerome Glisse [this message] 2018-12-05 18:07 ` Jerome Glisse 2018-12-05 18:20 ` Logan Gunthorpe 2018-12-05 18:33 ` Jerome Glisse 2018-12-05 18:33 ` Jerome Glisse 2018-12-05 18:48 ` Logan Gunthorpe 2018-12-05 18:55 ` Jerome Glisse 2018-12-05 18:55 ` Jerome Glisse 2018-12-05 19:10 ` Logan Gunthorpe 2018-12-05 22:58 ` Jerome Glisse 2018-12-05 22:58 ` Jerome Glisse 2018-12-05 23:09 ` Logan Gunthorpe 2018-12-05 23:20 ` Jerome Glisse 2018-12-05 23:20 ` Jerome Glisse 2018-12-05 23:23 ` Logan Gunthorpe 2018-12-05 23:27 ` Jerome Glisse 2018-12-06 0:08 ` Dan Williams 2018-12-05 2:34 ` Dan Williams 2018-12-05 2:37 ` Jerome Glisse 2018-12-05 2:37 ` Jerome Glisse 2018-12-05 17:25 ` Logan Gunthorpe 2018-12-05 18:01 ` Jerome Glisse 2018-12-05 18:01 ` Jerome Glisse 2018-12-04 20:14 ` Andi Kleen 2018-12-04 20:47 ` Logan Gunthorpe 2018-12-04 21:15 ` Jerome Glisse 2018-12-04 21:15 ` Jerome Glisse 2018-12-05 0:54 ` Kuehling, Felix 2018-12-04 19:19 ` Dan Williams 2018-12-04 19:32 ` Jerome Glisse 2018-12-04 19:32 ` Jerome Glisse 2018-12-04 20:12 ` Andi Kleen 2018-12-04 20:41 ` Jerome Glisse 2018-12-04 20:41 ` Jerome Glisse 2018-12-05 4:36 ` Aneesh Kumar K.V 2018-12-05 4:41 ` Jerome Glisse 2018-12-05 4:41 ` Jerome Glisse 2018-12-05 10:52 ` Mike Rapoport 2018-12-05 10:52 ` Mike Rapoport 2018-12-03 23:34 ` [RFC PATCH 03/14] mm/hms: add target memory to heterogeneous memory system infrastructure jglisse 2018-12-03 23:34 ` [RFC PATCH 04/14] mm/hms: add initiator " jglisse 2018-12-03 23:35 ` [RFC PATCH 05/14] mm/hms: add link " jglisse 2018-12-03 23:35 ` [RFC PATCH 06/14] mm/hms: add bridge " jglisse 2018-12-03 23:35 ` [RFC PATCH 07/14] mm/hms: register main memory with heterogenenous memory system jglisse 2018-12-03 23:35 ` [RFC PATCH 08/14] mm/hms: register main CPUs " jglisse 2018-12-03 23:35 ` [RFC PATCH 09/14] mm/hms: hbind() for heterogeneous memory system (aka mbind() for HMS) jglisse 2018-12-03 23:35 ` [RFC PATCH 10/14] mm/hbind: add heterogeneous memory policy tracking infrastructure jglisse 2018-12-03 23:35 ` [RFC PATCH 11/14] mm/hbind: add bind command to heterogeneous memory policy jglisse 2018-12-03 23:35 ` [RFC PATCH 12/14] mm/hbind: add migrate command to hbind() ioctl jglisse 2018-12-03 23:35 ` [RFC PATCH 13/14] drm/nouveau: register GPU under heterogeneous memory system jglisse 2018-12-03 23:35 ` [RFC PATCH 14/14] test/hms: tests for " jglisse 2018-12-04 7:44 ` [RFC PATCH 00/14] Heterogeneous Memory System (HMS) and hbind() Aneesh Kumar K.V 2018-12-04 7:44 ` Aneesh Kumar K.V 2018-12-04 14:44 ` Jerome Glisse 2018-12-04 14:44 ` Jerome Glisse 2018-12-04 14:44 ` Jerome Glisse 2018-12-04 18:02 ` Dave Hansen 2018-12-04 18:02 ` Dave Hansen 2018-12-04 18:49 ` Jerome Glisse 2018-12-04 18:49 ` Jerome Glisse 2018-12-04 18:49 ` Jerome Glisse 2018-12-04 18:54 ` Dave Hansen 2018-12-04 18:54 ` Dave Hansen 2018-12-04 19:11 ` Jerome Glisse 2018-12-04 19:11 ` Jerome Glisse 2018-12-04 19:11 ` Jerome Glisse 2018-12-04 21:37 ` Dave Hansen 2018-12-04 21:37 ` Dave Hansen 2018-12-04 21:57 ` Jerome Glisse 2018-12-04 21:57 ` Jerome Glisse 2018-12-04 21:57 ` Jerome Glisse 2018-12-04 23:58 ` Dave Hansen 2018-12-04 23:58 ` Dave Hansen 2018-12-05 0:29 ` Jerome Glisse 2018-12-05 0:29 ` Jerome Glisse 2018-12-05 0:29 ` Jerome Glisse 2018-12-05 1:22 ` Kuehling, Felix 2018-12-05 1:22 ` Kuehling, Felix 2018-12-05 1:22 ` Kuehling, Felix 2018-12-05 11:27 ` Aneesh Kumar K.V 2018-12-05 11:27 ` Aneesh Kumar K.V 2018-12-05 16:09 ` Jerome Glisse 2018-12-05 16:09 ` Jerome Glisse 2018-12-05 16:09 ` Jerome Glisse 2018-12-04 23:54 ` Dave Hansen 2018-12-04 23:54 ` Dave Hansen 2018-12-05 0:15 ` Jerome Glisse 2018-12-05 0:15 ` Jerome Glisse 2018-12-05 0:15 ` Jerome Glisse 2018-12-05 1:06 ` Dave Hansen 2018-12-05 1:06 ` Dave Hansen 2018-12-05 2:13 ` Jerome Glisse 2018-12-05 2:13 ` Jerome Glisse 2018-12-05 2:13 ` Jerome Glisse 2018-12-05 17:27 ` Dave Hansen 2018-12-05 17:27 ` Dave Hansen 2018-12-05 17:53 ` Jerome Glisse 2018-12-05 17:53 ` Jerome Glisse 2018-12-05 17:53 ` Jerome Glisse 2018-12-06 18:25 ` Dave Hansen 2018-12-06 18:25 ` Dave Hansen 2018-12-06 19:20 ` Jerome Glisse 2018-12-06 19:20 ` Jerome Glisse 2018-12-06 19:20 ` Jerome Glisse 2018-12-06 19:31 ` Dave Hansen 2018-12-06 19:31 ` Dave Hansen 2018-12-06 20:11 ` Logan Gunthorpe 2018-12-06 20:11 ` Logan Gunthorpe 2018-12-06 22:04 ` Dave Hansen 2018-12-06 22:04 ` Dave Hansen 2018-12-06 22:39 ` Jerome Glisse 2018-12-06 22:39 ` Jerome Glisse 2018-12-06 22:39 ` Jerome Glisse 2018-12-06 23:09 ` Dave Hansen 2018-12-06 23:09 ` Dave Hansen 2018-12-06 23:28 ` Logan Gunthorpe 2018-12-06 23:28 ` Logan Gunthorpe 2018-12-06 23:34 ` Dave Hansen 2018-12-06 23:34 ` Dave Hansen 2018-12-06 23:38 ` Dave Hansen 2018-12-06 23:38 ` Dave Hansen 2018-12-06 23:48 ` Logan Gunthorpe 2018-12-06 23:48 ` Logan Gunthorpe 2018-12-07 0:20 ` Jerome Glisse 2018-12-07 0:20 ` Jerome Glisse 2018-12-07 0:20 ` Jerome Glisse 2018-12-07 15:06 ` Jonathan Cameron 2018-12-07 15:06 ` Jonathan Cameron 2018-12-07 15:06 ` Jonathan Cameron 2018-12-07 19:37 ` Jerome Glisse 2018-12-07 19:37 ` Jerome Glisse 2018-12-07 19:37 ` Jerome Glisse 2018-12-07 0:15 ` Jerome Glisse 2018-12-07 0:15 ` Jerome Glisse 2018-12-07 0:15 ` Jerome Glisse 2018-12-06 20:27 ` Jerome Glisse 2018-12-06 20:27 ` Jerome Glisse 2018-12-06 20:27 ` Jerome Glisse 2018-12-06 21:46 ` Jerome Glisse 2018-12-06 21:46 ` Jerome Glisse 2018-12-06 21:46 ` Jerome Glisse
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20181205180756.GI3536@redhat.com \ --to=jglisse@redhat.com \ --cc=Paul.Blinzer@amd.com \ --cc=Philip.Yang@amd.com \ --cc=ak@linux.intel.com \ --cc=akpm@linux-foundation.org \ --cc=aneesh.kumar@linux.ibm.com \ --cc=balbirs@au1.ibm.com \ --cc=benh@kernel.crashing.org \ --cc=christian.koenig@amd.com \ --cc=dan.j.williams@intel.com \ --cc=dave.hansen@intel.com \ --cc=felix.kuehling@amd.com \ --cc=haggaie@mellanox.com \ --cc=jhubbard@nvidia.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=logang@deltatee.com \ --cc=rafael@kernel.org \ --cc=rcampbell@nvidia.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.