RE: Hang / zombie process from Xarray page-fault conversion (bisected)

["Barror, Robert" <robert.barror@intel.com>]

Hi Guys,

"> It's limited to xfs, no failure on ext4 to date", this is incorrect. I have been able to reproduce this issue with ext4. In order to do that, I need to run the full test (on both pmems in the system) and not the half test (only 1 pmem) that I use for inducing the hang under XFS. The test also runs considerably longer before failing with ext4 than XFS.

Thx bob


-----Original Message-----
From: Dave Chinner [mailto:david@fromorbit.com] 
Sent: Monday, March 11, 2019 9:38 PM
To: Williams, Dan J <dan.j.williams@intel.com>
Cc: Matthew Wilcox <willy@infradead.org>; Linux MM <linux-mm@kvack.org>; linux-nvdimm <linux-nvdimm@lists.01.org>; linux-fsdevel <linux-fsdevel@vger.kernel.org>; Barror, Robert <robert.barror@intel.com>
Subject: Re: Hang / zombie process from Xarray page-fault conversion (bisected)

On Mon, Mar 11, 2019 at 08:35:05PM -0700, Dan Williams wrote:
> On Mon, Mar 11, 2019 at 8:10 AM Matthew Wilcox <willy@infradead.org> wrote:
> >
> > On Thu, Mar 07, 2019 at 10:16:17PM -0800, Dan Williams wrote:
> > > Hi Willy,
> > >
> > > We're seeing a case where RocksDB hangs and becomes defunct when 
> > > trying to kill the process. v4.19 succeeds and v4.20 fails. Robert 
> > > was able to bisect this to commit b15cd800682f "dax: Convert page 
> > > fault handlers to XArray".
> > >
> > > I see some direct usage of xa_index and wonder if there are some 
> > > more pmd fixups to do?
> > >
> > > Other thoughts?
> >
> > I don't see why killing a process would have much to do with PMD 
> > misalignment.  The symptoms (hanging on a signal) smell much more 
> > like leaving a locked entry in the tree.  Is this easy to reproduce?  
> > Can you get /proc/$pid/stack for a hung task?
> 
> It's fairly easy to reproduce, I'll see if I can package up all the 
> dependencies into something that fails in a VM.
> 
> It's limited to xfs, no failure on ext4 to date.
> 
> The hung process appears to be:
> 
>      kworker/53:1-xfs-sync/pmem0

That's completely internal to XFS. Every 30s the work is triggered and it either does a log flush (if the fs is active) or it syncs the superblock to clean the log and idle the filesystem. It has nothing to do with user processes, and I don't see why killing a process has any effect on what it does...

> ...and then the rest of the database processes grind to a halt from there.
> 
> Robert was kind enough to capture /proc/$pid/stack, but nothing interesting:
> 
> [<0>] worker_thread+0xb2/0x380
> [<0>] kthread+0x112/0x130
> [<0>] ret_from_fork+0x1f/0x40
> [<0>] 0xffffffffffffffff

Much more useful would be:

# echo w > /proc/sysrq-trigger

And post the entire output of dmesg.

Cheers,

Dave.
--
Dave Chinner
david@fromorbit.com