From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from corp-mailer.zoner.com ([217.198.120.77]:47247 "EHLO corp-mailer.zoner.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750839AbdA0U7J (ORCPT ); Fri, 27 Jan 2017 15:59:09 -0500 Subject: Re: Quota-enabled XFS hangs during mount References: <20161103013153.GH9920@dastard> <7993e9b8-6eb8-6a0d-aa72-01346cca1b63@zoner.cz> <20161103204049.GA28177@dastard> <43ca55d0-6762-d54f-5ba9-a83f9c1988f6@zoner.cz> <20170123134452.GA33287@bfoster.bfoster> <5b41d19b-1a0d-2b74-a633-30a5f6d2f14a@zoner.cz> <20170125221739.GA33995@bfoster.bfoster> <30d56003-a517-f6f0-d188-d0ada5a9fbb7@zoner.cz> <20170126191254.GB39683@bfoster.bfoster> <27c2f5aa-517d-22b5-b55f-f0ceb277e9a3@zoner.cz> <20170127170734.GA49571@bfoster.bfoster> From: Martin Svec Message-ID: <5dc542b4-1fc7-2b67-0911-76914edf725e@zoner.cz> Date: Fri, 27 Jan 2017 21:49:27 +0100 MIME-Version: 1.0 In-Reply-To: <20170127170734.GA49571@bfoster.bfoster> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Sender: linux-xfs-owner@vger.kernel.org List-ID: List-Id: xfs To: Brian Foster Cc: Dave Chinner , linux-xfs@vger.kernel.org Dne 27.1.2017 v 18:07 Brian Foster napsal(a): > On Fri, Jan 27, 2017 at 02:06:45PM +0100, Martin Svec wrote: >> Dne 26.1.2017 v 20:12 Brian Foster napsal(a): >>> On Thu, Jan 26, 2017 at 06:46:42PM +0100, Martin Svec wrote: >>>> Hello, >>>> >>>> Dne 25.1.2017 v 23:17 Brian Foster napsal(a): >>>>> On Tue, Jan 24, 2017 at 02:17:36PM +0100, Martin Svec wrote: >>>>>> Hello, >>>>>> >>>>>> Dne 23.1.2017 v 14:44 Brian Foster napsal(a): >>>>>>> On Mon, Jan 23, 2017 at 10:44:20AM +0100, Martin Svec wrote: >>>>>>>> Hello Dave, >>>>>>>> >>>>>>>> Any updates on this? It's a bit annoying to workaround the bug by increasing RAM just because of the >>>>>>>> initial quotacheck. >>>>>>>> >>>>>>> Note that Dave is away on a bit of an extended vacation[1]. It looks >>>>>>> like he was in the process of fishing through the code to spot any >>>>>>> potential problems related to quotacheck+reclaim. I see you've cc'd him >>>>>>> directly so we'll see if we get a response wrt to if he got anywhere >>>>>>> with that... >>>>>>> >>>>>>> Skimming back through this thread, it looks like we have an issue where >>>>>>> quota check is not quite reliable in the event of reclaim, and you >>>>>>> appear to be reproducing this due to a probably unique combination of >>>>>>> large inode count and low memory. >>>>>>> >>>>>>> Is my understanding correct that you've reproduced this on more recent >>>>>>> kernels than the original report? >>>>>> Yes, I repeated the tests using 4.9.3 kernel on another VM where we hit this issue. >>>>>> >>>>>> Configuration: >>>>>> * vSphere 5.5 virtual machine, 2 vCPUs, virtual disks residing on iSCSI VMFS datastore >>>>>> * Debian Jessie 64 bit webserver, vanilla kernel 4.9.3 >>>>>> * 180 GB XFS data disk mounted as /www >>>>>> >>>>>> Quotacheck behavior depends on assigned RAM: >>>>>> * 2 or less GiB: mount /www leads to a storm of OOM kills including shell, ttys etc., so the system >>>>>> becomes unusable. >>>>>> * 3 GiB: mount /www task hangs in the same way as I reported in earlier in this thread. >>>>>> * 4 or more GiB: mount /www succeeds. >>>>>> >>>>> I was able to reproduce the quotacheck OOM situation on latest kernels. >>>>> This problem actually looks like a regression as of commit 17c12bcd3 >>>>> ("xfs: when replaying bmap operations, don't let unlinked inodes get >>>>> reaped"), but I don't think that patch is the core problem. That patch >>>>> pulled up setting MS_ACTIVE on the superblock from after XFS runs >>>>> quotacheck to before it (for other reasons), which has a side effect of >>>>> causing inodes to be placed onto the lru once they are released. Before >>>>> this change, all inodes were immediately marked for reclaim once >>>>> released from quotacheck because the superblock had not been set active. >>>>> >>>>> The problem here is first that quotacheck issues a bulkstat and thus >>>>> grabs and releases every inode in the fs. The quotacheck occurs at mount >>>>> time, which means we still hold the s_umount lock and thus the shrinker >>>>> cannot run even though it is registered. Therefore, we basically just >>>>> populate the lru until we've consumed too much memory and blow up. >>>>> >>>>> I think the solution here is to preserve the quotacheck behavior prior >>>>> to commit 17c12bcd3 via something like the following: >>>>> >>>>> --- a/fs/xfs/xfs_qm.c >>>>> +++ b/fs/xfs/xfs_qm.c >>>>> @@ -1177,7 +1177,7 @@ xfs_qm_dqusage_adjust( >>>>> * the case in all other instances. It's OK that we do this because >>>>> * quotacheck is done only at mount time. >>>>> */ >>>>> - error = xfs_iget(mp, NULL, ino, 0, XFS_ILOCK_EXCL, &ip); >>>>> + error = xfs_iget(mp, NULL, ino, XFS_IGET_DONTCACHE, XFS_ILOCK_EXCL, &ip); >>>>> if (error) { >>>>> *res = BULKSTAT_RV_NOTHING; >>>>> return error; >>>>> >>>>> ... which allows quotacheck to run as normal in my quick tests. Could >>>>> you try this on your more recent kernel tests and see whether you still >>>>> reproduce any problems? >>>> The above patch fixes OOM issues and reduces overall memory consumption during quotacheck. However, >>>> it does not fix the original xfs_qm_flush_one() freezing. I'm still able to reproduce it with 1 GB >>>> of RAM or lower. Tested with 4.9.5 kernel. >>>> >>> Ok, thanks. I'll get that fix posted shortly. >>> >>> I hadn't tried reducing RAM any further. I dropped my vm down to 1GB and >>> I don't reproduce a hang. If I drop to 512MB, the mount actually crashes >>> due to what looks like the problem that djwong just fixed[1]. >>> >>> With that one liner applied, it does look like I've hit a mount hang in >>> the quotacheck path. Note that I'm also running into OOM issues again >>> though, probably due to legitimately not having enough RAM for this vm. >>> Anyways, I'll see if I can dig anything out of that... >>> >>> FWIW, this is all on the latest for-next (4.10.0-rc5). >>> >>> [1] http://www.spinics.net/lists/linux-xfs/msg03869.html >>> >>>> If it makes sense to you, I can rsync the whole filesystem to a new XFS volume and repeat the tests. >>>> At least, that could tell us if the problem depends on a particular state of on-disk metadata >>>> structures or it's a general property of the given filesystem tree. >>>> >>> That couldn't hurt, thanks. >>> >> Well, after rsync to a fresh non-resized XFS volume, I still hit the mount hang with 1GB RAM. >> > The problem looks like a race between dquot reclaim and quotacheck. The > high level sequence of events is as follows: > > - During quotacheck, xfs_qm_dqiterate() walks the physical dquot > buffers and queues them to the delwri queue. > - Next, kswapd kicks in and attempts to reclaim a dquot that is backed > by a buffer on the quotacheck delwri queue. xfs_qm_dquot_isolate() > acquires the flush lock and attempts to queue to the reclaim delwri > queue. This silently fails because the buffer is already queued. > > From this point forward, the dquot flush lock is not going to be > released until the buffer is submitted for I/O and completed via > quotacheck. > - Quotacheck continues on to the xfs_qm_flush_one() pass, hits the > dquot in question and waits on the flush lock to issue the flush of > the recalculated values. *deadlock* > > There are at least a few ways to deal with this. We could do something > granular to fix up the reclaim path to check whether the buffer is > already queued or something of that nature before we actually invoke the > flush. I think this is effectively pointless, however, because the first > part of quotacheck walks and queues all physical dquot buffers anyways. > > In other words, I think dquot reclaim during quotacheck should probably > be bypassed. Given that, we could either adjust when the shrinker is > registered until after quotacheck or set a flag somewhere to cause dquot > reclaim to back out when quotacheck is running. I opted for something > like the latter. Care to test the appended patch? > > Note that I think this does mean that you could still have low memory > issues if you happen to have a lot of quotas defined.. > Looks good, no more hangs with 1 GB. Thank you, Brian. If I further reduce RAM to 512 MB, mount succeeds too but multiple "BUG: Bad page state in process mount" errors are reported. Is it one of the expected low memory issues? Martin