Re: [LSFMM] RDMA data corruption potential during FS writeback

From: Jason Gunthorpe <jgg@ziepe.ca>
To: John Hubbard <jhubbard@nvidia.com>
Cc: Dan Williams <dan.j.williams@intel.com>,
	Christopher Lameter <cl@linux.com>,
	linux-rdma <linux-rdma@vger.kernel.org>,
	Linux MM <linux-mm@kvack.org>, Michal Hocko <mhocko@kernel.org>
Subject: Re: [LSFMM] RDMA data corruption potential during FS writeback
Date: Fri, 18 May 2018 21:24:00 -0600	[thread overview]
Message-ID: <20180519032400.GA12517@ziepe.ca> (raw)
In-Reply-To: <c8861cbb-5b2e-d6e2-9c89-66c5c92181e6@nvidia.com>

On Fri, May 18, 2018 at 07:33:41PM -0700, John Hubbard wrote:
> On 05/18/2018 01:23 PM, Dan Williams wrote:
> > On Fri, May 18, 2018 at 10:36 AM, Jason Gunthorpe <jgg@ziepe.ca> wrote:
> >> On Fri, May 18, 2018 at 04:47:48PM +0000, Christopher Lameter wrote:
> >>> On Fri, 18 May 2018, Jason Gunthorpe wrote:
> >>>
> >>>
> >>> The newcomer here is RDMA. The FS side is the mainstream use case and has
> >>> been there since Unix learned to do paging.
> >>
> >> Well, it has been this way for 12 years, so it isn't that new.
> >>
> >> Honestly it sounds like get_user_pages is just a broken Linux
> >> API??
> >>
> >> Nothing can use it to write to pages because the FS could explode -
> >> RDMA makes it particularly easy to trigger this due to the longer time
> >> windows, but presumably any get_user_pages could generate a race and
> >> hit this? Is that right?
> 
> +1, and I am now super-interested in this conversation, because
> after tracking down a kernel BUG to this classic mistaken pattern:
> 
>     get_user_pages (on file-backed memory from ext4)
>     ...do some DMA
>     set_pages_dirty
>     put_page(s)

Ummm, RDMA has done essentially that since 2005, since when did it
become wrong? Do you have some references? Is there some alternative?

See __ib_umem_release

> ...there is (rarely!) a backtrace from ext4, that disavows ownership of
> any such pages.

Yes, I've seen that oops with RDMA, apparently isn't actually that
rare if you tweak things just right.

I thought it was an obscure ext4 bug :(

> Because the obvious "fix" in device driver land is to use a dedicated
> buffer for DMA, and copy to the filesystem buffer, and of course I will
> get *killed* if I propose such a performance-killing approach. But a
> core kernel fix really is starting to sound attractive.

Yeah, killed is right. That idea totally cripples RDMA.

What is the point of get_user_pages FOLL_WRITE if you can't write to
and dirty the pages!?!

Jason