From: Jason Gunthorpe <jgg@nvidia.com>
To: Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>
Cc: Leon Romanovsky <leon@kernel.org>,
"Marciniszyn, Mike" <mike.marciniszyn@cornelisnetworks.com>,
Doug Ledford <dledford@redhat.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>
Subject: Re: [PATCH rdma-next] RDMA/rdmavt: Decouple QP and SGE lists allocations
Date: Tue, 25 May 2021 10:13:58 -0300 [thread overview]
Message-ID: <20210525131358.GU1002214@nvidia.com> (raw)
In-Reply-To: <983802a6-0fa2-e181-832e-13a2d5f0fa82@cornelisnetworks.com>
On Thu, May 20, 2021 at 06:02:09PM -0400, Dennis Dalessandro wrote:
> > I don't want to encourage other drivers to do the same thing.
>
> I would imagine they would get the same push back we are getting here. I
> don't think this would encourage anyone honestly.
Then we are back to making infrastructure that is only useful for one,
arguably wrong, driver.
> > The correct thing to do today in 2021 is to use the standard NUMA
> > memory policy on already node-affine threads. The memory policy goes
> > into the kernel and normal non-_node allocations will obey it. When
> > combined with an appropriate node-affine HCA this will work as you are
> > expecting right now.
>
> So we shouldn't see any issue in the normal case is what you are
> saying. I'd like to believe that, proving it is not easy though.
Well, I said you have to setup the userspace properly, I'm not sure it
just works out of the box.
> > However you can't do anything like that while the kernel has the _node
> > annotations, that overrides the NUMA memory policy and breaks the
> > policy system!
>
> Does our driver doing this break the entire system? I'm not sure how that's
> possible.
It breaks your driver part of it, and if we lift it to the core code
then it breaks all drivers, so it is a hard no-go.
> Is there an effort to get rid of these per node allocations so
> ultimately we won't have a choice at some point?
Unlikely, subtle stuff like this will just be left broken in drivers
nobody cares about..
Jason
next prev parent reply other threads:[~2021-05-25 13:14 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-11 10:36 [PATCH rdma-next] RDMA/rdmavt: Decouple QP and SGE lists allocations Leon Romanovsky
2021-05-11 10:59 ` Haakon Bugge
2021-05-11 12:34 ` Leon Romanovsky
2021-05-11 19:15 ` Marciniszyn, Mike
2021-05-11 19:27 ` Leon Romanovsky
2021-05-11 19:39 ` Marciniszyn, Mike
2021-05-12 4:08 ` Dennis Dalessandro
2021-05-12 12:13 ` Leon Romanovsky
2021-05-12 12:45 ` Dennis Dalessandro
2021-05-11 12:26 ` Dennis Dalessandro
2021-05-11 12:34 ` Leon Romanovsky
2021-05-12 12:25 ` Marciniszyn, Mike
2021-05-12 12:50 ` Leon Romanovsky
2021-05-13 19:03 ` Dennis Dalessandro
2021-05-13 19:15 ` Jason Gunthorpe
2021-05-13 19:31 ` Dennis Dalessandro
2021-05-14 13:02 ` Jason Gunthorpe
2021-05-14 14:07 ` Dennis Dalessandro
2021-05-14 14:35 ` Jason Gunthorpe
2021-05-14 15:00 ` Marciniszyn, Mike
2021-05-14 15:02 ` Jason Gunthorpe
2021-05-19 7:50 ` Leon Romanovsky
2021-05-19 11:56 ` Dennis Dalessandro
2021-05-19 18:29 ` Jason Gunthorpe
2021-05-19 19:49 ` Dennis Dalessandro
2021-05-19 20:26 ` Jason Gunthorpe
2021-05-20 22:02 ` Dennis Dalessandro
2021-05-21 6:29 ` Leon Romanovsky
2021-05-25 13:13 ` Jason Gunthorpe [this message]
2021-05-25 14:10 ` Dennis Dalessandro
2021-05-25 14:20 ` Jason Gunthorpe
2021-05-25 14:29 ` Dennis Dalessandro
2021-06-28 21:59 ` Marciniszyn, Mike
2021-06-28 23:19 ` Jason Gunthorpe
2021-07-04 6:34 ` Leon Romanovsky
2021-06-02 4:33 ` Leon Romanovsky
2021-05-16 10:56 ` Leon Romanovsky
2021-05-12 12:23 ` Marciniszyn, Mike
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210525131358.GU1002214@nvidia.com \
--to=jgg@nvidia.com \
--cc=dennis.dalessandro@cornelisnetworks.com \
--cc=dledford@redhat.com \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=mike.marciniszyn@cornelisnetworks.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).