Linux-Fsdevel Archive on lore.kernel.org
 help / color / Atom feed
From: Jerome Glisse <jglisse@redhat.com>
To: John Hubbard <jhubbard@nvidia.com>
Cc: john.hubbard@gmail.com, Matthew Wilcox <willy@infradead.org>,
	Michal Hocko <mhocko@kernel.org>,
	Christopher Lameter <cl@linux.com>,
	Jason Gunthorpe <jgg@ziepe.ca>,
	Dan Williams <dan.j.williams@intel.com>, Jan Kara <jack@suse.cz>,
	Al Viro <viro@zeniv.linux.org.uk>,
	linux-mm@kvack.org, LKML <linux-kernel@vger.kernel.org>,
	linux-rdma <linux-rdma@vger.kernel.org>,
	linux-fsdevel@vger.kernel.org,
	Christian Benvenuti <benve@cisco.com>,
	Dennis Dalessandro <dennis.dalessandro@intel.com>,
	Doug Ledford <dledford@redhat.com>,
	Mike Marciniszyn <mike.marciniszyn@intel.com>
Subject: Re: [PATCH 0/4] get_user_pages*() and RDMA: first steps
Date: Fri, 28 Sep 2018 17:49:37 -0400
Message-ID: <20180928214934.GA3265@redhat.com> (raw)
In-Reply-To: <4c884529-e2ff-3808-9763-eb0e71f5a616@nvidia.com>

On Fri, Sep 28, 2018 at 12:06:12PM -0700, John Hubbard wrote:
> On 9/28/18 8:29 AM, Jerome Glisse wrote:
> > On Thu, Sep 27, 2018 at 10:39:45PM -0700, john.hubbard@gmail.com wrote:
> >> From: John Hubbard <jhubbard@nvidia.com>
> >>
> >> Hi,
> >>
> >> This short series prepares for eventually fixing the problem described
> >> in [1], and is following a plan listed in [2].
> >>
> >> I'd like to get the first two patches into the -mm tree.
> >>
> >> Patch 1, although not technically critical to do now, is still nice to have,
> >> because it's already been reviewed by Jan, and it's just one more thing on the
> >> long TODO list here, that is ready to be checked off.
> >>
> >> Patch 2 is required in order to allow me (and others, if I'm lucky) to start
> >> submitting changes to convert all of the callsites of get_user_pages*() and
> >> put_page().  I think this will work a lot better than trying to maintain a
> >> massive patchset and submitting all at once.
> >>
> >> Patch 3 converts infiniband drivers: put_page() --> put_user_page(). I picked
> >> a fairly small and easy example.
> >>
> >> Patch 4 converts a small driver from put_page() --> release_user_pages(). This
> >> could just as easily have been done as a change from put_page() to
> >> put_user_page(). The reason I did it this way is that this provides a small and
> >> simple caller of the new release_user_pages() routine. I wanted both of the
> >> new routines, even though just placeholders, to have callers.
> >>
> >> Once these are all in, then the floodgates can open up to convert the large
> >> number of get_user_pages*() callsites.
> >>
> >> [1] https://lwn.net/Articles/753027/ : "The Trouble with get_user_pages()"
> >>
> >> [2] https://lkml.kernel.org/r/20180709080554.21931-1-jhubbard@nvidia.com
> >>     Proposed steps for fixing get_user_pages() + DMA problems.
> >>
> > 
> > So the solution is to wait (possibly for days, months, years) that the
> > RDMA or GPU which did GUP and do not have mmu notifier, release the page
> > (or put_user_page()) ?
> > 
> > This sounds bads. Like i said during LSF/MM there is no way to properly
> > fix hardware that can not be preempted/invalidated ... most GPU are fine.
> > Few RDMA are fine, most can not ...
> > 
> 
> Hi Jerome,
> 
> Personally, I'm think that this particular design is the best one I've seen
> so far, but if other, better designs show up, than let's do those instead, sure.
> 
> I guess your main concern is that this might take longer than other approaches.
> 
> As for time frame, perhaps I made it sound worse than it really is. I have patches
> staged already for all of the simpler call sites, and for about half of the more
> complicated ones. The core solution in mm is not large, and we've gone through a 
> few discussion threads about it back in July or so, so it shouldn't take too long
> to perfect it.
> 
> So it may be a few months to get it all reviewed and submitted, but I don't
> see "years" by any stretch.

Bit of miss-comprehention there :) By month, years, i am talking about
the time it will take for some user to release the pin they have on the
page. Not the time to push something upstream.

AFAICT RDMA driver do not have any upper bound on how long they can hold
a page reference and thus your solution can leave one CPU core stuck for
as long as the pin is active. Worst case might lead to all CPU core waiting
for something that might never happen.

> 
> 
> > If it is just about fixing the set_page_dirty() bug then just looking at
> > refcount versus mapcount should already tell you if you can remove the
> > buffer head from the page or not. Which would fix the bug without complex
> > changes (i still like the put_user_page just for symetry with GUP).
> > 
> 
> It's about more than that. The goal is to make it safe and correct to
> use a non-CPU device to read and write to "pinned" memory, especially when
> that memory is backed by a file system.
> 
> I recall there were objections to just narrowly fixing the set_page_dirty()
> bug, because the underlying problem is large and serious. So here we are.

Except that you can not solve that issue without proper hardware. GPU are
fine. RDMA are broken except the mellanox5 hardware which can invalidate
at anytime its page table thus allowing to write protect the page at any
time.

With the solution put forward here you can potentialy wait _forever_ for
the driver that holds a pin to drop it. This was the point i was trying to
get accross during LSF/MM. You can not fix broken hardware that decided to
use GUP to do a feature they can't reliably do because their hardware is
not capable to behave.

Because code is easier here is what i was meaning:

https://cgit.freedesktop.org/~glisse/linux/commit/?h=gup&id=a5dbc0fe7e71d347067579f13579df372ec48389
https://cgit.freedesktop.org/~glisse/linux/commit/?h=gup&id=01677bc039c791a16d5f82b3ef84917d62fac826

Cheers,
J�r�me

  reply index

Thread overview: 29+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-09-28  5:39 john.hubbard
2018-09-28  5:39 ` [PATCH 1/4] mm: get_user_pages: consolidate error handling john.hubbard
2018-09-28  5:39 ` [PATCH 3/4] infiniband/mm: convert to the new put_user_page() call john.hubbard
2018-09-28 15:39   ` Jason Gunthorpe
2018-09-29  3:12     ` John Hubbard
2018-09-29 16:21       ` Matthew Wilcox
2018-09-29 19:19         ` Jason Gunthorpe
2018-10-01 12:50         ` Christoph Hellwig
2018-10-01 15:29           ` Matthew Wilcox
2018-10-01 15:51             ` Christoph Hellwig
2018-10-01 14:35       ` Dennis Dalessandro
2018-10-03  5:40         ` John Hubbard
2018-10-03 16:27       ` Jan Kara
2018-10-03 23:19         ` John Hubbard
2018-09-28  5:39 ` [PATCH 2/4] mm: introduce put_user_page(), placeholder version john.hubbard
2018-10-03 16:22   ` Jan Kara
2018-10-03 23:23     ` John Hubbard
2018-09-28  5:39 ` [PATCH 4/4] goldfish_pipe/mm: convert to the new release_user_pages() call john.hubbard
2018-09-28 15:29 ` [PATCH 0/4] get_user_pages*() and RDMA: first steps Jerome Glisse
2018-09-28 19:06   ` John Hubbard
2018-09-28 21:49     ` Jerome Glisse [this message]
2018-09-29  2:28       ` John Hubbard
2018-09-29  8:46         ` Jerome Glisse
2018-10-01  6:11           ` Dave Chinner
2018-10-01 12:47             ` Christoph Hellwig
2018-10-02  1:14               ` Dave Chinner
2018-10-03 16:21                 ` Jan Kara
2018-10-01 15:31             ` Jason Gunthorpe
2018-10-03 16:08           ` Jan Kara

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180928214934.GA3265@redhat.com \
    --to=jglisse@redhat.com \
    --cc=benve@cisco.com \
    --cc=cl@linux.com \
    --cc=dan.j.williams@intel.com \
    --cc=dennis.dalessandro@intel.com \
    --cc=dledford@redhat.com \
    --cc=jack@suse.cz \
    --cc=jgg@ziepe.ca \
    --cc=jhubbard@nvidia.com \
    --cc=john.hubbard@gmail.com \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=mhocko@kernel.org \
    --cc=mike.marciniszyn@intel.com \
    --cc=viro@zeniv.linux.org.uk \
    --cc=willy@infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-Fsdevel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-fsdevel/0 linux-fsdevel/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-fsdevel linux-fsdevel/ https://lore.kernel.org/linux-fsdevel \
		linux-fsdevel@vger.kernel.org
	public-inbox-index linux-fsdevel

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-fsdevel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git