From: Jerome Glisse <jglisse@redhat.com>
To: Logan Gunthorpe <logang@deltatee.com>
Cc: Dan Williams <dan.j.williams@intel.com>,
Andi Kleen <ak@linux.intel.com>, Linux MM <linux-mm@kvack.org>,
Andrew Morton <akpm@linux-foundation.org>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
"Rafael J. Wysocki" <rafael@kernel.org>,
Dave Hansen <dave.hansen@intel.com>,
Haggai Eran <haggaie@mellanox.com>,
balbirs@au1.ibm.com,
"Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
Benjamin Herrenschmidt <benh@kernel.crashing.org>,
"Kuehling, Felix" <felix.kuehling@amd.com>,
Philip.Yang@amd.com, "Koenig,
Christian" <christian.koenig@amd.com>,
"Blinzer, Paul" <Paul.Blinzer@amd.com>,
John Hubbard <jhubbard@nvidia.com>,
rcampbell@nvidia.com
Subject: Re: [RFC PATCH 02/14] mm/hms: heterogenenous memory system (HMS) documentation
Date: Wed, 5 Dec 2018 13:01:28 -0500 [thread overview]
Message-ID: <20181205180127.GH3536@redhat.com> (raw)
In-Reply-To: <2f53e0c0-a8af-b003-5bd7-a341431908df@deltatee.com>
On Wed, Dec 05, 2018 at 10:25:31AM -0700, Logan Gunthorpe wrote:
>
>
> On 2018-12-04 7:37 p.m., Jerome Glisse wrote:
> >>
> >> This came up before for apis even better defined than HMS as well as
> >> more limited scope, i.e. experimental ABI availability only for -rc
> >> kernels. Linus said this:
> >>
> >> "There are no loopholes. No "but it's been only one release". No, no,
> >> no. The whole point is that users are supposed to be able to *trust*
> >> the kernel. If we do something, we keep on doing it.
> >>
> >> And if it makes it harder to add new user-visible interfaces, then
> >> that's a *good* thing." [1]
> >>
> >> The takeaway being don't land work-in-progress ABIs in the kernel.
> >> Once an application depends on it, there are no more incompatible
> >> changes possible regardless of the warnings, experimental notices, or
> >> "staging" designation. DAX is experimental because there are cases
> >> where it currently does not work with respect to another kernel
> >> feature like xfs-reflink, RDMA. The plan is to fix those, not continue
> >> to hide behind an experimental designation, and fix them in a way that
> >> preserves the user visible behavior that has already been exposed,
> >> i.e. no regressions.
> >>
> >> [1]: https://lists.linuxfoundation.org/pipermail/ksummit-discuss/2017-August/004742.html
> >
> > So i guess i am heading down the vXX road ... such is my life :)
>
> I recommend against it. I really haven't been convinced by any of your
> arguments for having a second topology tree. The existing topology tree
> in sysfs already better describes the links between hardware right now,
> except for the missing GPU links (and those should be addressable within
> the GPU community). Plus, maybe, some other enhancements to sockets/numa
> node descriptions if there's something missing there.
>
> Then, 'hbind' is another issue but I suspect it would be better
> implemented as an ioctl on existing GPU interfaces. I certainly can't
> see any benefit in using it myself.
>
> It's better to take an approach that would be less controversial with
> the community than to brow beat them with a patch set 20+ times until
> they take it.
So here is what i am gonna do because i need this code now. I am gonna
split the helper code that does policy and hbind out from its sysfs
peerage and i am gonna turn it into helpers that each device driver
can use. I will move the sysfs and syscall to be a patchset on its own
which use the exact same above infrastructure.
This means that i am loosing feature as it means that userspace can
not provide a list of multiple device memory to use (which is much more
common that you might think) but at least i can provide something for
the single device case through ioctl.
I am not giving up on sysfs or syscall as this is needed long term so
i am gonna improve it, port existing userspace (OpenCL, ROCm, ...) to
use it (in branch) and demonstrate how it get use by end application.
I will beat it again and again until either i convince people through
hard evidence or i get bored. I do not get bored easily :)
Cheers,
Jérôme
next prev parent reply other threads:[~2018-12-05 18:01 UTC|newest]
Thread overview: 94+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-12-03 23:34 [RFC PATCH 00/14] Heterogeneous Memory System (HMS) and hbind() jglisse
2018-12-03 23:34 ` [RFC PATCH 01/14] mm/hms: heterogeneous memory system (sysfs infrastructure) jglisse
2018-12-03 23:34 ` [RFC PATCH 02/14] mm/hms: heterogenenous memory system (HMS) documentation jglisse
2018-12-04 17:06 ` Andi Kleen
2018-12-04 18:24 ` Jerome Glisse
2018-12-04 18:31 ` Dan Williams
2018-12-04 18:57 ` Jerome Glisse
2018-12-04 19:11 ` Logan Gunthorpe
2018-12-04 19:22 ` Jerome Glisse
2018-12-04 19:41 ` Logan Gunthorpe
2018-12-04 20:13 ` Jerome Glisse
2018-12-04 20:30 ` Logan Gunthorpe
2018-12-04 20:59 ` Jerome Glisse
2018-12-04 21:19 ` Logan Gunthorpe
2018-12-04 21:51 ` Jerome Glisse
2018-12-04 22:16 ` Logan Gunthorpe
2018-12-04 23:56 ` Jerome Glisse
2018-12-05 1:15 ` Logan Gunthorpe
2018-12-05 2:31 ` Jerome Glisse
2018-12-05 17:41 ` Logan Gunthorpe
2018-12-05 18:07 ` Jerome Glisse
2018-12-05 18:20 ` Logan Gunthorpe
2018-12-05 18:33 ` Jerome Glisse
2018-12-05 18:48 ` Logan Gunthorpe
2018-12-05 18:55 ` Jerome Glisse
2018-12-05 19:10 ` Logan Gunthorpe
2018-12-05 22:58 ` Jerome Glisse
2018-12-05 23:09 ` Logan Gunthorpe
2018-12-05 23:20 ` Jerome Glisse
2018-12-05 23:23 ` Logan Gunthorpe
2018-12-05 23:27 ` Jerome Glisse
2018-12-06 0:08 ` Dan Williams
2018-12-05 2:34 ` Dan Williams
2018-12-05 2:37 ` Jerome Glisse
2018-12-05 17:25 ` Logan Gunthorpe
2018-12-05 18:01 ` Jerome Glisse [this message]
2018-12-04 20:14 ` Andi Kleen
2018-12-04 20:47 ` Logan Gunthorpe
2018-12-04 21:15 ` Jerome Glisse
2018-12-04 19:19 ` Dan Williams
2018-12-04 19:32 ` Jerome Glisse
2018-12-04 20:12 ` Andi Kleen
2018-12-04 20:41 ` Jerome Glisse
2018-12-05 4:36 ` Aneesh Kumar K.V
2018-12-05 4:41 ` Jerome Glisse
2018-12-05 10:52 ` Mike Rapoport
2018-12-03 23:34 ` [RFC PATCH 03/14] mm/hms: add target memory to heterogeneous memory system infrastructure jglisse
2018-12-03 23:34 ` [RFC PATCH 04/14] mm/hms: add initiator " jglisse
2018-12-03 23:35 ` [RFC PATCH 05/14] mm/hms: add link " jglisse
2018-12-03 23:35 ` [RFC PATCH 06/14] mm/hms: add bridge " jglisse
2018-12-03 23:35 ` [RFC PATCH 07/14] mm/hms: register main memory with heterogenenous memory system jglisse
2018-12-03 23:35 ` [RFC PATCH 08/14] mm/hms: register main CPUs " jglisse
2018-12-03 23:35 ` [RFC PATCH 09/14] mm/hms: hbind() for heterogeneous memory system (aka mbind() for HMS) jglisse
2018-12-03 23:35 ` [RFC PATCH 10/14] mm/hbind: add heterogeneous memory policy tracking infrastructure jglisse
2018-12-03 23:35 ` [RFC PATCH 11/14] mm/hbind: add bind command to heterogeneous memory policy jglisse
2018-12-03 23:35 ` [RFC PATCH 12/14] mm/hbind: add migrate command to hbind() ioctl jglisse
2018-12-03 23:35 ` [RFC PATCH 13/14] drm/nouveau: register GPU under heterogeneous memory system jglisse
2018-12-03 23:35 ` [RFC PATCH 14/14] test/hms: tests for " jglisse
2018-12-04 7:44 ` [RFC PATCH 00/14] Heterogeneous Memory System (HMS) and hbind() Aneesh Kumar K.V
2018-12-04 14:44 ` Jerome Glisse
2018-12-04 18:02 ` Dave Hansen
2018-12-04 18:49 ` Jerome Glisse
2018-12-04 18:54 ` Dave Hansen
2018-12-04 19:11 ` Jerome Glisse
2018-12-04 21:37 ` Dave Hansen
2018-12-04 21:57 ` Jerome Glisse
2018-12-04 23:58 ` Dave Hansen
2018-12-05 0:29 ` Jerome Glisse
2018-12-05 1:22 ` Kuehling, Felix
2018-12-05 11:27 ` Aneesh Kumar K.V
2018-12-05 16:09 ` Jerome Glisse
2018-12-04 23:54 ` Dave Hansen
2018-12-05 0:15 ` Jerome Glisse
2018-12-05 1:06 ` Dave Hansen
2018-12-05 2:13 ` Jerome Glisse
2018-12-05 17:27 ` Dave Hansen
2018-12-05 17:53 ` Jerome Glisse
2018-12-06 18:25 ` Dave Hansen
2018-12-06 19:20 ` Jerome Glisse
2018-12-06 19:31 ` Dave Hansen
2018-12-06 20:11 ` Logan Gunthorpe
2018-12-06 22:04 ` Dave Hansen
2018-12-06 22:39 ` Jerome Glisse
2018-12-06 23:09 ` Dave Hansen
2018-12-06 23:28 ` Logan Gunthorpe
2018-12-06 23:34 ` Dave Hansen
2018-12-06 23:38 ` Dave Hansen
2018-12-06 23:48 ` Logan Gunthorpe
2018-12-07 0:20 ` Jerome Glisse
2018-12-07 15:06 ` Jonathan Cameron
2018-12-07 19:37 ` Jerome Glisse
2018-12-07 0:15 ` Jerome Glisse
2018-12-06 20:27 ` Jerome Glisse
2018-12-06 21:46 ` Jerome Glisse
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20181205180127.GH3536@redhat.com \
--to=jglisse@redhat.com \
--cc=Paul.Blinzer@amd.com \
--cc=Philip.Yang@amd.com \
--cc=ak@linux.intel.com \
--cc=akpm@linux-foundation.org \
--cc=aneesh.kumar@linux.ibm.com \
--cc=balbirs@au1.ibm.com \
--cc=benh@kernel.crashing.org \
--cc=christian.koenig@amd.com \
--cc=dan.j.williams@intel.com \
--cc=dave.hansen@intel.com \
--cc=felix.kuehling@amd.com \
--cc=haggaie@mellanox.com \
--cc=jhubbard@nvidia.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=logang@deltatee.com \
--cc=rafael@kernel.org \
--cc=rcampbell@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).