From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from bombadil.infradead.org ([198.137.202.133]:39826 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726889AbeJEJsw (ORCPT ); Fri, 5 Oct 2018 05:48:52 -0400 Date: Thu, 4 Oct 2018 19:52:12 -0700 From: Matthew Wilcox To: Dan Williams Cc: Jan Kara , linux-fsdevel , linux-nvdimm , Barret Rhoden Subject: Re: [PATCH] dax: Fix deadlock in dax_lock_mapping_entry() Message-ID: <20181005025212.GA12446@bombadil.infradead.org> References: <20180927112332.3649-1-jack@suse.cz> <20180927132843.GA19006@bombadil.infradead.org> <20180927134129.GB8800@quack2.suse.cz> <20181004162748.GI28384@quack2.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: linux-fsdevel-owner@vger.kernel.org List-ID: On Thu, Oct 04, 2018 at 06:57:52PM -0700, Dan Williams wrote: > On Thu, Oct 4, 2018 at 9:27 AM Jan Kara wrote: > > > > On Thu 27-09-18 11:22:22, Dan Williams wrote: > > > On Thu, Sep 27, 2018 at 6:41 AM Jan Kara wrote: > > > > > > > > On Thu 27-09-18 06:28:43, Matthew Wilcox wrote: > > > > > On Thu, Sep 27, 2018 at 01:23:32PM +0200, Jan Kara wrote: > > > > > > When dax_lock_mapping_entry() has to sleep to obtain entry lock, it will > > > > > > fail to unlock mapping->i_pages spinlock and thus immediately deadlock > > > > > > against itself when retrying to grab the entry lock again. Fix the > > > > > > problem by unlocking mapping->i_pages before retrying. > > > > > > > > > > It seems weird that xfstests doesn't provoke this ... > > > > > > > > The function currently gets called only from mm/memory-failure.c. And yes, > > > > we are lacking DAX hwpoison error tests in fstests... > > > > > > I have an item on my backlog to port the ndctl unit test that does > > > memory_failure() injection vs ext4 over to fstests. That said I've > > > been investigating a deadlock on ext4 caused by this test. When I saw > > > this patch I hoped it was root cause, but the test is still failing > > > for me. Vishal is able to pass the test on his system, so the failure > > > mode is timing dependent. I'm running this patch on top of -rc5 and > > > still seeing the following deadlock. > > > > I went through the code but I don't see where the problem could be. How can > > I run that test? Is KVM enough or do I need hardware with AEP dimms? > > KVM is enough... however, I have found a hack that makes the test pass: > > diff --git a/mm/filemap.c b/mm/filemap.c > index 52517f28e6f4..d7f035b1846e 100644 > --- a/mm/filemap.c > +++ b/mm/filemap.c > @@ -1668,6 +1668,9 @@ unsigned find_get_entries(struct address_space *mapping, > goto repeat; > } > export: > + if (iter.index < start) > + continue; > + > indices[ret] = iter.index; > entries[ret] = page; > if (++ret == nr_entries) > > Is this a radix bug? I would never expect: > > radix_tree_for_each_slot(slot, &mapping->i_pages, &iter, start) > > ...to return entries with an index < start. Without that change above > we loop forever because dax_layout_busy_page() can't make forward > progress. I'll dig into the radix code tomorrow, but maybe someone > else will be me to it overnight. If 'start' is within a 2MB entry, iter.index can absolutely be less than start. I forget exactly what the radix tree code does, but I think it returns index set to the canonical/base index of the entry.