* [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim @ 2016-10-14 12:27 Chris Mason 2016-10-15 22:34 ` Dave Chinner 2019-06-14 12:58 ` Amir Goldstein 0 siblings, 2 replies; 24+ messages in thread From: Chris Mason @ 2016-10-14 12:27 UTC (permalink / raw) To: Dave Chinner, linux-xfs Hi Dave, This is part of a series of patches we're growing to fix a perf regression on a few straggler tiers that are still on v3.10. In this case, hadoop had to switch back to v3.10 because v4.x is as much as 15% slower on recent kernels. Between v3.10 and v4.x, kswapd is less effective overall. This leads more and more procs to get bogged down in direct reclaim Using SYNC_WAIT in xfs_reclaim_inodes_ag(). Since slab shrinking happens very early in direct reclaim, we've seen systems with 130GB of ram where hundreds of procs are stuck on the xfs slab shrinker fighting to walk a slab 900MB in size. They'd have better luck moving on to the page cache instead. Also, we're going into direct reclaim much more often than we should because kswapd is getting stuck on XFS inode locks and writeback. Dropping the SYNC_WAIT means that kswapd can move on to other things and let the async worker threads get kicked to work on the inodes. We're still working on the series, and this is only compile tested on current Linus git. I'm working out some better simulations for the hadoop workload to stuff into Mel's tests. Numbers from prod take roughly 3 days to stabilize, so I haven't isolated this patch from the rest of the series. Unpatched v4.x our base allocation stall rate goes up to as much as 200-300/sec, averaging 70/sec. The series I'm finalizing gets that number down to < 1 /sec. Omar Sandoval did some digging and found you added the SYNC_WAIT in response to a workload I sent ages ago. I tried to make this OOM with fsmark creating empty files, and it has been soaking in memory constrained workloads in production for almost two weeks. Signed-off-by: Chris Mason <clm@fb.com> --- fs/xfs/xfs_icache.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c index bf2d607..63938fb 100644 --- a/fs/xfs/xfs_icache.c +++ b/fs/xfs/xfs_icache.c @@ -1195,7 +1195,7 @@ xfs_reclaim_inodes_nr( xfs_reclaim_work_queue(mp); xfs_ail_push_all(mp->m_ail); - return xfs_reclaim_inodes_ag(mp, SYNC_TRYLOCK | SYNC_WAIT, &nr_to_scan); + return xfs_reclaim_inodes_ag(mp, SYNC_TRYLOCK, &nr_to_scan); } /* -- 2.9.3 ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-10-14 12:27 [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim Chris Mason @ 2016-10-15 22:34 ` Dave Chinner 2016-10-17 0:24 ` Chris Mason 2019-06-14 12:58 ` Amir Goldstein 1 sibling, 1 reply; 24+ messages in thread From: Dave Chinner @ 2016-10-15 22:34 UTC (permalink / raw) To: Chris Mason; +Cc: linux-xfs On Fri, Oct 14, 2016 at 08:27:24AM -0400, Chris Mason wrote: > > Hi Dave, > > This is part of a series of patches we're growing to fix a perf > regression on a few straggler tiers that are still on v3.10. In this > case, hadoop had to switch back to v3.10 because v4.x is as much as 15% > slower on recent kernels. > > Between v3.10 and v4.x, kswapd is less effective overall. This leads > more and more procs to get bogged down in direct reclaim Using SYNC_WAIT > in xfs_reclaim_inodes_ag(). > > Since slab shrinking happens very early in direct reclaim, we've seen > systems with 130GB of ram where hundreds of procs are stuck on the xfs > slab shrinker fighting to walk a slab 900MB in size. They'd have better > luck moving on to the page cache instead. We've already scanned the page cache for direct reclaim by the time we get to running the shrinkers. Indeed, the amount of work the shrinkers do is directly controlled by the amount of work done scanning the page cache beforehand.... > Also, we're going into direct reclaim much more often than we should > because kswapd is getting stuck on XFS inode locks and writeback. Where and what locks, exactly? > Dropping the SYNC_WAIT means that kswapd can move on to other things and > let the async worker threads get kicked to work on the inodes. Correct me if I'm wrong, but we introduced this shrinker behaviour long before 3.10. I think the /explicit/ SYNC_WAIT was added some time around 3.0 when I added the background async reclaim, but IIRC the synchronous reclaim behaviour predates that by quite a bit. IOWs, XFS shrinkers have the same blocking behaviour in 3.10 as they do in current 4.x kernels. Hence if you're getting problems with excessive blocking during reclaim on more recent 4.x kernels, then it's more likely there is a change in memory reclaim balance in the vmscan code that drives the shrinkers - that has definitely changed between 3.10 and 4.x, but the XFS shrinker behaviour has not. What XFS is doing is not wrong - the synchrnous behaviour is the primary memory reclaim feedback mechanism that prevents reclaim from trashing the working set of clean inodes when under memory pressure. It's also the choke point where we prevent lots of concurrent threads from trying to do reclaim at once, contending on locks and inodes and causing catastrophic IO breakdown because such reclaim results in random IO patterns for inode writeback instead of nice clean ascending offset ordered IO. This happens to be exactly the situation you are describing - hundreds of processes all trying to run inode reclaim concurrently, which if we don't block will repaid trash the clean inode cache leaving only the dirty inodes around. Hence to remove the SYNC_WAIT simply breaks the overload prevention and throttling feedback mechanisms the XFS inode shrinker uses to maintain performance under severe memory pressure. > We're still working on the series, and this is only compile tested on > current Linus git. I'm working out some better simulations for the > hadoop workload to stuff into Mel's tests. Numbers from prod take > roughly 3 days to stabilize, so I haven't isolated this patch from the rest > of the series. > > Unpatched v4.x our base allocation stall rate goes up to as much as > 200-300/sec, averaging 70/sec. The series I'm finalizing gets that > number down to < 1 /sec. That doesn't mean you've 'fixed' the problem. AFAICT you're just disconnecting memory allocation from the rate at which inode reclaim can clean and reclaim inodes. Yes, direct reclaim stalls will go away, but that doesn't speed up inode reclaim, nor does it do anything to prevent random inode writeback patterns. With the amount of memory your machines have, removing SYNC_WAIT will simply hide the problem by leaving the inodes dirty until journal pressure forces the inode to be cleaned and then the background reclaimer can free it. In general, that doesn't work particularly well because journals can index more dirty objects that can be easily cached in memory, and that's why we have to drive IO and throttle direct reclaim. > Omar Sandoval did some digging and found you added the SYNC_WAIT in > response to a workload I sent ages ago. That commit (which I had to go find because you helpfully didn't quote it) was a7b339f1b869 ("xfs: introduce background inode reclaim work") introduced asynchronous background reclaim work and so made the reclaim function able to handle both async and sync reclaim. To maintain the direct reclaim throttling behaviour of the shrinker, that function now needed to be told to be sycnhronous, hence the addtion of the SYNC_WAIT. We didn't introduce sync reclaim with this commit in 2.6.39(!), we've had that behaviour since well before that. Hence if the analysis performed stopped at this point in history, it was flawed. > I tried to make this OOM with > fsmark creating empty files, and it has been soaking in memory > constrained workloads in production for almost two weeks. That workload reclaims mostly from the background worker because a) the majority of allocations are GFP_NOFS and so the inode shrinker doesn't run very often, b) inode allocation is sequential so inode writeback optimises into large sequential writes, not slow random writes, c) it generates significant journal pressure so drives inode writeback through tail-pushing the journal, and d) sequential, ascending order inode allocation and inode writeback optimisations means reclaim rarely comes across dirty inodes because it reclaims inodes in the same order they are allocated and cleaned. I'm sure there's a solution to whatever problem is occurring at FB, but you're going to need to start by describing the problems and documenting the analysis so we can understand what the problem is before we start discussion potential solutions. It seems that the problem is excessive direct reclaim concurrency and XFS throttling concurrency back to the maximum the underlying filesystem can optimise sanely, but nobody has explained why that is happening on 4.x and not 3.10. i.e. We need to clearly understand the root cause of the problem before we discuss potential solutions. Changes to long standing (and unchanged) behaviours might end up being the fix, but it's not something we'll do without first understanding the problem or exploring other potential solutions... Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-10-15 22:34 ` Dave Chinner @ 2016-10-17 0:24 ` Chris Mason 2016-10-17 1:52 ` Dave Chinner 0 siblings, 1 reply; 24+ messages in thread From: Chris Mason @ 2016-10-17 0:24 UTC (permalink / raw) To: Dave Chinner; +Cc: linux-xfs On Sun, Oct 16, 2016 at 09:34:54AM +1100, Dave Chinner wrote: >On Fri, Oct 14, 2016 at 08:27:24AM -0400, Chris Mason wrote: >> >> Hi Dave, >> >> This is part of a series of patches we're growing to fix a perf >> regression on a few straggler tiers that are still on v3.10. In this >> case, hadoop had to switch back to v3.10 because v4.x is as much as 15% >> slower on recent kernels. >> >> Between v3.10 and v4.x, kswapd is less effective overall. This leads >> more and more procs to get bogged down in direct reclaim Using SYNC_WAIT >> in xfs_reclaim_inodes_ag(). >> >> Since slab shrinking happens very early in direct reclaim, we've seen >> systems with 130GB of ram where hundreds of procs are stuck on the xfs >> slab shrinker fighting to walk a slab 900MB in size. They'd have better >> luck moving on to the page cache instead. > >We've already scanned the page cache for direct reclaim by the time >we get to running the shrinkers. Indeed, the amount of work the >shrinkers do is directly controlled by the amount of work done >scanning the page cache beforehand.... > >> Also, we're going into direct reclaim much more often than we should >> because kswapd is getting stuck on XFS inode locks and writeback. > >Where and what locks, exactly? This is from v4.0, because all of my newer hosts are trying a variety of patched kernels. But the traces were very similar on newer kernels: # cat /proc/282/stack [<ffffffff812ea2cd>] xfs_buf_submit_wait+0xbd/0x1d0 [<ffffffff812ea6e4>] xfs_bwrite+0x24/0x60 [<ffffffff812f18a4>] xfs_reclaim_inode+0x304/0x320 [<ffffffff812f1b17>] xfs_reclaim_inodes_ag+0x257/0x370 [<ffffffff812f2613>] xfs_reclaim_inodes_nr+0x33/0x40 [<ffffffff81300fb9>] xfs_fs_free_cached_objects+0x19/0x20 [<ffffffff811bb13b>] super_cache_scan+0x18b/0x190 [<ffffffff8115acc6>] shrink_slab.part.40+0x1f6/0x380 [<ffffffff8115e9da>] shrink_zone+0x30a/0x320 [<ffffffff8115f94f>] kswapd+0x51f/0x9e0 [<ffffffff810886b2>] kthread+0xd2/0xf0 [<ffffffff81770d88>] ret_from_fork+0x58/0x90 [<ffffffffffffffff>] 0xffffffffffffffff This one hurts the most. While kswapd is waiting for IO, all the other reclaim he might have been doing is backing up. The other common path is the pag->pag_ici_reclaim_lock lock in xfs_reclaim_inodes_ag. It goes through the trylock loop, didn't free enough, and then waits on the locks for real. > >> Dropping the SYNC_WAIT means that kswapd can move on to other things and >> let the async worker threads get kicked to work on the inodes. > >Correct me if I'm wrong, but we introduced this shrinker behaviour >long before 3.10. I think the /explicit/ SYNC_WAIT was added some >time around 3.0 when I added the background async reclaim, but IIRC >the synchronous reclaim behaviour predates that by quite a bit. With this workload, it can take as much as a week to verify a given change really makes it better. So, I'm much more focused on a high level view of making the MM in current kernels more efficient than I am in digging through each commit between v3.10 and v4.x that may be related. Once I've nailed down a good list of fixes, I'll start making some tests to make sure they keep fixing things in the future. I'm starting with obvious things: direct reclaim is happening because kswapd isn't processing the lists fast enough because he's stuck in D state. This lead to two changes. This patch, and lowering the dirty ratios to make sure background writeout can keep up and kswapd doesn't have to call pageout(). XFS is also limiting the direct reclaim speed of all the other slabs. We have 15 drives, each with its own filesystem. But end result of the current system is to bottleneck behind whichever FS is slowest at any given moment. > >IOWs, XFS shrinkers have the same blocking behaviour in 3.10 as they >do in current 4.x kernels. Hence if you're getting problems with >excessive blocking during reclaim on more recent 4.x kernels, then >it's more likely there is a change in memory reclaim balance in the >vmscan code that drives the shrinkers - that has definitely changed >between 3.10 and 4.x, but the XFS shrinker behaviour has not. Correct, I got down this path assuming that commits between v3.10 and 4.x changed how often we're calling the xfs shrinker, even though the shrinker was the same. > >What XFS is doing is not wrong - the synchrnous behaviour is the >primary memory reclaim feedback mechanism that prevents reclaim from >trashing the working set of clean inodes when under memory pressure. >It's also the choke point where we prevent lots of concurrent >threads from trying to do reclaim at once, contending on locks >and inodes and causing catastrophic IO breakdown because such >reclaim results in random IO patterns for inode writeback instead of >nice clean ascending offset ordered IO. It's also blocking kswapd (and all the other procs directly calling shrinkers) on IO. Either IO it directly issues or IO run by other procs. This causes the conditions that make all the threads want to do reclaim at once. >That commit (which I had to go find because you helpfully didn't >quote it) was a7b339f1b869 ("xfs: introduce background inode reclaim >work") introduced asynchronous background reclaim work and so made >the reclaim function able to handle both async and sync reclaim. To >maintain the direct reclaim throttling behaviour of the shrinker, >that function now needed to be told to be sycnhronous, hence the >addtion of the SYNC_WAIT. We didn't introduce sync >reclaim with this commit in 2.6.39(!), we've had that behaviour >since well before that. Hence if the analysis performed stopped at >this point in history, it was flawed. This was an RFC because I was RFCing. We're stuffing kswapd behind synchronous IO, and limiting the rate at which we can reclaim pages on the system. I'm happy to toss unpatched kernels into the workload and gather stats to hep us nail down good behaviour, but really I'm asking what those stats might be. What we have now is a single synchronous shrinker taking the box over. Nothing happens until XFS gets its inodes down the pipe, even when there are a considerable number of other freeable pages on the box. There are probably a series of other optimizations we can be making in the MM code around when the shrinkers are called, and how well it deals with the constant churn from this workload. I want to try those too, but we're stuck right now on this one spot. An over all description of the hadoop workload: Lots of java threads, spanning 15 filesystems, 4T each. Each thread runs an unpredictable amount of disk and network IO and uses massive amounts of CPU. The threads last for unpredictable amounts of time. These boxes have ~130GB of ram and two sockets of CPUs (12 cores per-cpu, HT enabled). The files themselves are relatively large and are mostly streaming reads/writes. -chris ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-10-17 0:24 ` Chris Mason @ 2016-10-17 1:52 ` Dave Chinner 2016-10-17 13:30 ` Chris Mason 0 siblings, 1 reply; 24+ messages in thread From: Dave Chinner @ 2016-10-17 1:52 UTC (permalink / raw) To: Chris Mason; +Cc: linux-xfs On Sun, Oct 16, 2016 at 08:24:33PM -0400, Chris Mason wrote: > On Sun, Oct 16, 2016 at 09:34:54AM +1100, Dave Chinner wrote: > >On Fri, Oct 14, 2016 at 08:27:24AM -0400, Chris Mason wrote: > >> > >>Hi Dave, > >> > >>This is part of a series of patches we're growing to fix a perf > >>regression on a few straggler tiers that are still on v3.10. In this > >>case, hadoop had to switch back to v3.10 because v4.x is as much as 15% > >>slower on recent kernels. > >> > >>Between v3.10 and v4.x, kswapd is less effective overall. This leads > >>more and more procs to get bogged down in direct reclaim Using SYNC_WAIT > >>in xfs_reclaim_inodes_ag(). > >> > >>Since slab shrinking happens very early in direct reclaim, we've seen > >>systems with 130GB of ram where hundreds of procs are stuck on the xfs > >>slab shrinker fighting to walk a slab 900MB in size. They'd have better > >>luck moving on to the page cache instead. > > > >We've already scanned the page cache for direct reclaim by the time > >we get to running the shrinkers. Indeed, the amount of work the > >shrinkers do is directly controlled by the amount of work done > >scanning the page cache beforehand.... > > > >>Also, we're going into direct reclaim much more often than we should > >>because kswapd is getting stuck on XFS inode locks and writeback. > > > >Where and what locks, exactly? > > This is from v4.0, because all of my newer hosts are trying a > variety of patched kernels. But the traces were very similar on > newer kernels: > > # cat /proc/282/stack > [<ffffffff812ea2cd>] xfs_buf_submit_wait+0xbd/0x1d0 > [<ffffffff812ea6e4>] xfs_bwrite+0x24/0x60 > [<ffffffff812f18a4>] xfs_reclaim_inode+0x304/0x320 > [<ffffffff812f1b17>] xfs_reclaim_inodes_ag+0x257/0x370 > [<ffffffff812f2613>] xfs_reclaim_inodes_nr+0x33/0x40 > [<ffffffff81300fb9>] xfs_fs_free_cached_objects+0x19/0x20 > [<ffffffff811bb13b>] super_cache_scan+0x18b/0x190 > [<ffffffff8115acc6>] shrink_slab.part.40+0x1f6/0x380 > [<ffffffff8115e9da>] shrink_zone+0x30a/0x320 > [<ffffffff8115f94f>] kswapd+0x51f/0x9e0 > [<ffffffff810886b2>] kthread+0xd2/0xf0 > [<ffffffff81770d88>] ret_from_fork+0x58/0x90 > [<ffffffffffffffff>] 0xffffffffffffffff > > This one hurts the most. While kswapd is waiting for IO, all the > other reclaim he might have been doing is backing up. Which says two things: the journal tail pushing nor the background inode reclaim threads are keeping up with dirty inode writeback demand. Without knowing why that is occurring, we cannot solve the problem. > The other common path is the pag->pag_ici_reclaim_lock lock in > xfs_reclaim_inodes_ag. It goes through the trylock loop, didn't > free enough, and then waits on the locks for real. Which is the "prevent hundreds of threads from all issuing inode writeback concurrently" throttling. Working as designed. > XFS is also limiting the direct reclaim speed of all the other > slabs. We have 15 drives, each with its own filesystem. But end > result of the current system is to bottleneck behind whichever FS is > slowest at any given moment. So why is the filesystem slow in 4.0 and not slow at all in 3.10? And how does a 4.8 kernel compare, given there were major changes to the mm/ subsystem in this release? i.e. are you chasing a mm/ problem that has already been solved? > >What XFS is doing is not wrong - the synchrnous behaviour is the > >primary memory reclaim feedback mechanism that prevents reclaim from > >trashing the working set of clean inodes when under memory pressure. > >It's also the choke point where we prevent lots of concurrent > >threads from trying to do reclaim at once, contending on locks > >and inodes and causing catastrophic IO breakdown because such > >reclaim results in random IO patterns for inode writeback instead of > >nice clean ascending offset ordered IO. > > It's also blocking kswapd (and all the other procs directly calling > shrinkers) on IO. Either IO it directly issues or IO run by other > procs. This causes the conditions that make all the threads want to > do reclaim at once. Yup, that's what it's meant to do. As I keep repeating - this behaviour is indicative of /some other problem/ occurring. If inode writeback and reclaim is occurring efficiently and correctly, then kswapd will not throttle like this because it will never block on IO. And, by design, when we throttle kswapd we effectively throttle direct reclaim, too. > >That commit (which I had to go find because you helpfully didn't > >quote it) was a7b339f1b869 ("xfs: introduce background inode reclaim > >work") introduced asynchronous background reclaim work and so made > >the reclaim function able to handle both async and sync reclaim. To > >maintain the direct reclaim throttling behaviour of the shrinker, > >that function now needed to be told to be sycnhronous, hence the > >addtion of the SYNC_WAIT. We didn't introduce sync > >reclaim with this commit in 2.6.39(!), we've had that behaviour > >since well before that. Hence if the analysis performed stopped at > >this point in history, it was flawed. > > This was an RFC because I was RFCing. We're stuffing kswapd behind > synchronous IO, and limiting the rate at which we can reclaim pages > on the system. I'm happy to toss unpatched kernels into the > workload and gather stats to hep us nail down good behaviour, but > really I'm asking what those stats might be. I can't suggest anything right now, because you haven't given me system/workload details or concrete analysis to base any suggestions on. > What we have now is a single synchronous shrinker taking the box > over. Nothing happens until XFS gets its inodes down the pipe, even > when there are a considerable number of other freeable pages on the > box. > > There are probably a series of other optimizations we can be making > in the MM code around when the shrinkers are called, and how well it > deals with the constant churn from this workload. I want to try > those too, but we're stuck right now on this one spot. > > An over all description of the hadoop workload: > > Lots of java threads, spanning 15 filesystems, 4T each. http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F Things like number of allocation groups, journal size, what the physical storage is, IO scheduler (or lack of), etc, are all important here. > Each thread runs an unpredictable amount of disk and network IO and > uses massive amounts of CPU. The threads last for unpredictable > amounts of time. These boxes have ~130GB of ram and two sockets of > CPUs (12 cores per-cpu, HT enabled). The files themselves are > relatively large and are mostly streaming reads/writes. Again, details. meminfo, iostat, etc, details of how memory usage and IO patterns change when everything backs up on inode reclaim, etc are /really important/ here. I need to be able to reproduce a similar memory imbalance myself to be able to test any solution we come up with. Here's some immediate questions I have from the workload description: - If the workload is mostly large files and streaming reads and writes, then why are there so many inodes that need writeback that reclaim is getting stuck on them? - Why aren't the inodes getting written back regularly via periodic log work (e.g. via the xfs_ail_push_all() call that occur every 30s)? - Is there so much data IO that metadata IO is being starved? - Why does it take a week to manifest - when does the system go out of balance and is there anything in userspace that changes behaviour that might trigger it? - Is there a behavioural step-change in the workload of some threads? - are you running at near ENOSPC and so maybe hitting some filesystem fragmentation level that causes the seek load to slowly increase until there's no IOPS left in the storage? Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-10-17 1:52 ` Dave Chinner @ 2016-10-17 13:30 ` Chris Mason 2016-10-17 22:30 ` Dave Chinner 0 siblings, 1 reply; 24+ messages in thread From: Chris Mason @ 2016-10-17 13:30 UTC (permalink / raw) To: Dave Chinner; +Cc: linux-xfs On 10/16/2016 09:52 PM, Dave Chinner wrote: > On Sun, Oct 16, 2016 at 08:24:33PM -0400, Chris Mason wrote: >> On Sun, Oct 16, 2016 at 09:34:54AM +1100, Dave Chinner wrote: >>> On Fri, Oct 14, 2016 at 08:27:24AM -0400, Chris Mason wrote: >>>> >>>> Hi Dave, >>>> >>>> This is part of a series of patches we're growing to fix a perf >>>> regression on a few straggler tiers that are still on v3.10. In this >>>> case, hadoop had to switch back to v3.10 because v4.x is as much as 15% >>>> slower on recent kernels. >>>> >>>> Between v3.10 and v4.x, kswapd is less effective overall. This leads >>>> more and more procs to get bogged down in direct reclaim Using SYNC_WAIT >>>> in xfs_reclaim_inodes_ag(). >>>> >>>> Since slab shrinking happens very early in direct reclaim, we've seen >>>> systems with 130GB of ram where hundreds of procs are stuck on the xfs >>>> slab shrinker fighting to walk a slab 900MB in size. They'd have better >>>> luck moving on to the page cache instead. >>> >>> We've already scanned the page cache for direct reclaim by the time >>> we get to running the shrinkers. Indeed, the amount of work the >>> shrinkers do is directly controlled by the amount of work done >>> scanning the page cache beforehand.... >>> >>>> Also, we're going into direct reclaim much more often than we should >>>> because kswapd is getting stuck on XFS inode locks and writeback. >>> >>> Where and what locks, exactly? >> >> This is from v4.0, because all of my newer hosts are trying a >> variety of patched kernels. But the traces were very similar on >> newer kernels: >> >> # cat /proc/282/stack >> [<ffffffff812ea2cd>] xfs_buf_submit_wait+0xbd/0x1d0 >> [<ffffffff812ea6e4>] xfs_bwrite+0x24/0x60 >> [<ffffffff812f18a4>] xfs_reclaim_inode+0x304/0x320 >> [<ffffffff812f1b17>] xfs_reclaim_inodes_ag+0x257/0x370 >> [<ffffffff812f2613>] xfs_reclaim_inodes_nr+0x33/0x40 >> [<ffffffff81300fb9>] xfs_fs_free_cached_objects+0x19/0x20 >> [<ffffffff811bb13b>] super_cache_scan+0x18b/0x190 >> [<ffffffff8115acc6>] shrink_slab.part.40+0x1f6/0x380 >> [<ffffffff8115e9da>] shrink_zone+0x30a/0x320 >> [<ffffffff8115f94f>] kswapd+0x51f/0x9e0 >> [<ffffffff810886b2>] kthread+0xd2/0xf0 >> [<ffffffff81770d88>] ret_from_fork+0x58/0x90 >> [<ffffffffffffffff>] 0xffffffffffffffff >> >> This one hurts the most. While kswapd is waiting for IO, all the >> other reclaim he might have been doing is backing up. > > Which says two things: the journal tail pushing nor the background > inode reclaim threads are keeping up with dirty inode writeback > demand. Without knowing why that is occurring, we cannot solve the > problem. > >> The other common path is the pag->pag_ici_reclaim_lock lock in >> xfs_reclaim_inodes_ag. It goes through the trylock loop, didn't >> free enough, and then waits on the locks for real. > > Which is the "prevent hundreds of threads from all issuing inode > writeback concurrently" throttling. Working as designed. Ok, I think we're just not on the same page about how kswapd is designed. So instead of worrying about some crusty old kernel, lets talk about that for a minute. I'm not trying to explain kswapd to you, just putting what I'm seeing from the shrinker in terms of how kswapd deals with dirty pages: LRUs try to keep the dirty pages away from kswapd in hopes that background writeback will clean them instead of kswapd. When system memory pressure gets bad enough, kswapd will call pageout(). This includes a check for congested bdis where it will skip the IO because it doesn't want to wait on busy resources. The main throttling mechanism is to slow down the creation of new dirty pages via balance_dirty_pages(). IO is avoided from inside kswapd because there's only one kswapd per-numa node. It is trying to take a global view of the freeable memory in the node, instead of focusing on any one individual page. Shrinkers are a little different because while individual shrinkers have a definition of dirty, the general concept doesn't. kswapd calls into the shrinkers to ask them to be smaller. With dirty pages, kswapd will start IO but not wait on it. With the xfs shrinker, kswapd does synchronous IO to write a single inode in xfs_buf_submit_wait(). With congested BDIs, kswapd will skip the IO and wait for progress after running through a good chunk of pages. With the xfs shrinker, kswapd will synchronously wait for progress on a single FS, even if there are dozens of other filesystems around. For the xfs shrinker, the mechanism to throttle new dirty inodes on a single FS is stalling every process in the system in direct reclaim? > >> XFS is also limiting the direct reclaim speed of all the other >> slabs. We have 15 drives, each with its own filesystem. But end >> result of the current system is to bottleneck behind whichever FS is >> slowest at any given moment. > > So why is the filesystem slow in 4.0 and not slow at all in 3.10? > It's not that v3.10 is fast. It's just faster. v4.x is faster in a bunch of other ways, but this one part of v3.10 isn't slowing down the system as much as this one part of v4.x > And how does a 4.8 kernel compare, given there were major changes to > the mm/ subsystem in this release? i.e. are you chasing a mm/ > problem that has already been solved? We don't think it's already solved in v4.8, but we're setting up a test to confirm that. I'm working on a better simulation of the parts we're tripping over so I can model this outside of production. I definitely agree that something is wrong in MM land too, we have to clamp down on the dirty ratios much more than we should to keep kswapd from calling pageout(). We can dive into workload specifics too, but I'd rather do that against simulations where I can try individual experiments more quickly. The reason it takes me a week to get hard numbers is because the workload is very inconsistent. The only way to get a good comparison is to put the test kernel on roughly 30 machines and then average major metrics over a period of days. Just installing the kernel takes almost a day because I can only reboot one machine every 20 minutes. -chris ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-10-17 13:30 ` Chris Mason @ 2016-10-17 22:30 ` Dave Chinner 2016-10-17 23:20 ` Chris Mason 0 siblings, 1 reply; 24+ messages in thread From: Dave Chinner @ 2016-10-17 22:30 UTC (permalink / raw) To: Chris Mason; +Cc: linux-xfs On Mon, Oct 17, 2016 at 09:30:05AM -0400, Chris Mason wrote: > On 10/16/2016 09:52 PM, Dave Chinner wrote: > >On Sun, Oct 16, 2016 at 08:24:33PM -0400, Chris Mason wrote: > >>On Sun, Oct 16, 2016 at 09:34:54AM +1100, Dave Chinner wrote: > >>This one hurts the most. While kswapd is waiting for IO, all the > >>other reclaim he might have been doing is backing up. > > > >Which says two things: the journal tail pushing nor the background > >inode reclaim threads are keeping up with dirty inode writeback > >demand. Without knowing why that is occurring, we cannot solve the > >problem. > > > >>The other common path is the pag->pag_ici_reclaim_lock lock in > >>xfs_reclaim_inodes_ag. It goes through the trylock loop, didn't > >>free enough, and then waits on the locks for real. > > > >Which is the "prevent hundreds of threads from all issuing inode > >writeback concurrently" throttling. Working as designed. > > Ok, I think we're just not on the same page about how kswapd is > designed. Chris, I understand perfectly well what kswapd is and how it is supposed to work. I also understand how shrinkers work and how they are supposed to interact with page reclaim - my dirty paws are all over the shrinker infrastructure (context specific shrinkers, NUMA, GFP_NOFS anti-windup mechanisms, cgroup, etc). > The main throttling mechanism is to slow down the creation of new > dirty pages via balance_dirty_pages(). I'm also the guy who architected the IO-less dirty page throttling infrastructure so there's not much you can teach me about that, either. Indeed, the XFS behaviour that you want to remove implements a similar (but much more complex) feedback mechanism as the IO-less dirty throttle. > IO is avoided from inside kswapd because there's only one kswapd > per-numa node. It is trying to take a global view of the freeable > memory in the node, instead of focusing on any one individual page. kswapd is also the "get out of gaol free" thread for reclaim when the memory pressure is entirely filesystem bound and so direct reclaim skips all filesystem reclaim because GFP_NOFS is being asserted. This happens a lot in XFS..... The result of this is that only in kswapd context (i.e. GFP_KERNEL, PF_KSWAPD shrinker context) can we do the things necessary to /guarantee forwards reclaim progress/. That means kswapd might sometimes be slow, but if we don't allow the shrinker to block from time to time then there's every chance that reclaim will not make progress. > Shrinkers are a little different because while individual shrinkers > have a definition of dirty, the general concept doesn't. kswapd > calls into the shrinkers to ask them to be smaller. Well, that's not exactly what the shrinkers are being asked to do. Speaking as the guy who designed the current shrinker API, shrinkers are being asked to /scan their subsystem/ and take actions that will /allow/ memory to be reclaimed. Not all shrinkers sit in front of slab caches - some sit on hardware related pools and trigger early garbage collection of completed commands and queues. e.g. the DRI subsystem shrinkers. Shrinkers are for more than just slab caches these days - they are a general "memory pressure notification" mechanism and need to be thought of as such rather than a traditional "slab cache shrinker". IOWs, the actions that shrinkers take may not directly free memory, but they may result in memory becoming reclaimable in the near future e.g. writing back dirty inodes doesn't free memory - it can actually create memory demand - but it does then allow the inodes and their backing buffers to be reclaimed once the inodes are clean. To allow such generic implementations to exist, shrinkers are allowed to block just like the page cache reclaim is allowed to block. Blocking should be as limited as possible, but it is allowed as it may be necessary to guarantee progress. The difference here is that page cache reclaim has a far more context in which to make decisions on whether to block or not. Shrinkers have a gfp mask, and nothing else. i.e shrinkers are not given enough context by the mm subsystem to make smart decisions on how much they are allowed to block. e.g. GFP_NOFS means no filesystem shrinker can run, even though the memory allocation may be coming from a different fs and there's no possibility of a deadlock. I've been talking about this with the mm/ guys and what we can do differently to pass more context to the shrinkers (e.g. task based reclaim context structures rather than GFP_* flags passed on the stack) so we can be smarter in the shrinkers about what we can and can't do. > With dirty pages, kswapd will start IO but not wait on it. > With the xfs shrinker, kswapd does synchronous IO to write a single > inode in xfs_buf_submit_wait(). That's because there is /no other throttling mechanism/ for shrinker controlled slab caches. We can't throttle at allocation time because we have no mechanism for either counting or limiting the number of dirty objects in a slab caches, like we do the page cache. We have /limited/ control via the size of the filesystem journal (which is why I've been asking for that information!), but realistically the only solid, reliable method we have to prevent excessive dirty inode accumulation in large memory machines with multiple filesystems and slow disks is to throttle memory allocation to the rate at which we can reclaim inodes. This prevents the sorts of situations we were regularly seeing 7-8 years ago where a freeze (for a snapshot) or unmount could take /hours/ because we have built up hundreds of thousands of dirty inodes in cache over a period of days because the slow SATA RAID6 array can only do 50 write IOPS..... i.e. there are many, many good reasons for why XFS treats inode reclaim the way it does. I don't expect you to understand all the issues that we are preventing by throttling memory allocation like this, but I do expect you to respect the fact it's done for very good reasons. > With congested BDIs, kswapd will skip the IO and wait for progress > after running through a good chunk of pages. With the xfs shrinker, > kswapd will synchronously wait for progress on a single FS, even if > there are dozens of other filesystems around. Yes, that can happen but, again, this behaviour indicates something else is wrong and the system is out of whack in some way. In the normal case, kswapd will only block on IO long enough for the other other async threads run through inode writeback and clean inode reclaim as efficiently as possible. By the time that the shrinker wakes up after blocking on an IO, it should either have clean inodes to reclaim or nothing more to do. What you are reporting is equivalent to having pageout() run and do all the writeback (badly) instead of the bdi flusher threads doing all the writeback (efficiently). pageout() is a /worst case/ behaviour we try very hard to avoid and when it occurs it is generally indicative of some other problem or imbalance. Same goes here for the inode shrinker. > For the xfs shrinker, the mechanism to throttle new dirty inodes on > a single FS is stalling every process in the system in direct > reclaim? We're not throttling dirty inodes - we are throttling memory allocation because that's the only hammer we have to prevent excessive buildup of dirty of inodes that have already been allocated. We can't do that throttling when we dirty inodes because that happens in transaction context holding locked objects and blocking there waiting on inode writeback progress will cause journal deadlocks.... This stuff is way more complex than just "have cache, will shrink". > We don't think it's already solved in v4.8, but we're setting up a > test to confirm that. I'm working on a better simulation of the > parts we're tripping over so I can model this outside of production. > I definitely agree that something is wrong in MM land too, we have > to clamp down on the dirty ratios much more than we should to keep > kswapd from calling pageout(). Having pageout() run is pretty indicative of instantaneous memory demand being significantly higher than the IO throughput of the storage subsystem and the only reclaimable memory in the system being dirty filesystem caches. i.e. background writeback is not keeping up memory demand and dirtying rates for some reason. The general rule of thumb is that if getting pageout() is occurring then the IO subsystem is about to die a horrible death of random IO. If you're taking great lengths to avoid pageout() from being called, then it's no surprise to me that your workload is, instead, triggering the equivalent "oh shit, we're in real trouble here" behaviour in XFS inode cache reclaim. I also wonder, after turning down the dirty ratios, if you've done other typical writeback tuning tweaks like speeding up XFS's periodic metadata writeback to clean inodes faster in the absence of journal pressure. It's detailed information like this that I've been asking for - there's good reasons I've asked for that information, and any further discussion will just be a waste of my time without all the details I've already asked for. > We can dive into workload specifics too, but I'd rather do that > against simulations where I can try individual experiments more > quickly. Yup, I think you need to come up with a reproducable test case you can share.... Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-10-17 22:30 ` Dave Chinner @ 2016-10-17 23:20 ` Chris Mason 2016-10-18 2:03 ` Dave Chinner 0 siblings, 1 reply; 24+ messages in thread From: Chris Mason @ 2016-10-17 23:20 UTC (permalink / raw) To: Dave Chinner; +Cc: linux-xfs On 10/17/2016 06:30 PM, Dave Chinner wrote: > On Mon, Oct 17, 2016 at 09:30:05AM -0400, Chris Mason wrote: >> On 10/16/2016 09:52 PM, Dave Chinner wrote: >>> On Sun, Oct 16, 2016 at 08:24:33PM -0400, Chris Mason wrote: >>>> On Sun, Oct 16, 2016 at 09:34:54AM +1100, Dave Chinner wrote: >>>> This one hurts the most. While kswapd is waiting for IO, all the >>>> other reclaim he might have been doing is backing up. >>> >>> Which says two things: the journal tail pushing nor the background >>> inode reclaim threads are keeping up with dirty inode writeback >>> demand. Without knowing why that is occurring, we cannot solve the >>> problem. >>> >>>> The other common path is the pag->pag_ici_reclaim_lock lock in >>>> xfs_reclaim_inodes_ag. It goes through the trylock loop, didn't >>>> free enough, and then waits on the locks for real. >>> >>> Which is the "prevent hundreds of threads from all issuing inode >>> writeback concurrently" throttling. Working as designed. >> >> Ok, I think we're just not on the same page about how kswapd is >> designed. > > Chris, I understand perfectly well what kswapd is and how it is > supposed to work. I also understand how shrinkers work and how they > are supposed to interact with page reclaim - my dirty paws are all > over the shrinker infrastructure (context specific shrinkers, > NUMA, GFP_NOFS anti-windup mechanisms, cgroup, etc). Dave, I'm just trying break down the conversation to a series common ground vocabulary that I know you understand. I'll come back with a workload in .c form. You can run the workload and decide for yourself if the shrinkers are incorrectly bottlencecking the system. > >> The main throttling mechanism is to slow down the creation of new >> dirty pages via balance_dirty_pages(). > > I'm also the guy who architected the IO-less dirty page throttling > infrastructure so there's not much you can teach me about that, > either. Indeed, the XFS behaviour that you want to remove implements > a similar (but much more complex) feedback mechanism as the IO-less > dirty throttle. > >> IO is avoided from inside kswapd because there's only one kswapd >> per-numa node. It is trying to take a global view of the freeable >> memory in the node, instead of focusing on any one individual page. > > kswapd is also the "get out of gaol free" thread for reclaim when > the memory pressure is entirely filesystem bound and so direct > reclaim skips all filesystem reclaim because GFP_NOFS is being > asserted. This happens a lot in XFS..... > > The result of this is that only in kswapd context (i.e. GFP_KERNEL, > PF_KSWAPD shrinker context) can we do the things necessary to > /guarantee forwards reclaim progress/. That means kswapd might > sometimes be slow, but if we don't allow the shrinker to block from > time to time then there's every chance that reclaim will not make > progress. > Sure, if the shrinker blocked from time to time, I wouldn't be sending this email. >> Shrinkers are a little different because while individual shrinkers >> have a definition of dirty, the general concept doesn't. kswapd >> calls into the shrinkers to ask them to be smaller. > > Well, that's not exactly what the shrinkers are being asked to do. > Speaking as the guy who designed the current shrinker API, shrinkers > are being asked to /scan their subsystem/ and take actions that will > /allow/ memory to be reclaimed. Not all shrinkers sit in front of > slab caches - some sit on hardware related pools and trigger early > garbage collection of completed commands and queues. e.g. the DRI > subsystem shrinkers. Shrinkers are for more than just slab caches > these days - they are a general "memory pressure notification" > mechanism and need to be thought of as such rather than a traditional > "slab cache shrinker". Yes, which is why its so important that a single individual shrinker not block all the other shrinkers from happening. > > IOWs, the actions that shrinkers take may not directly free memory, > but they may result in memory becoming reclaimable in the near > future e.g. writing back dirty inodes doesn't free memory - it can > actually create memory demand - but it does then allow the inodes > and their backing buffers to be reclaimed once the inodes are clean. > > To allow such generic implementations to exist, shrinkers are > allowed to block just like the page cache reclaim is allowed to > block. Blocking should be as limited as possible, but it is allowed > as it may be necessary to guarantee progress. > > The difference here is that page cache reclaim has a far more > context in which to make decisions on whether to block or not. > Shrinkers have a gfp mask, and nothing else. i.e shrinkers are not > given enough context by the mm subsystem to make smart decisions on > how much they are allowed to block. e.g. GFP_NOFS means no > filesystem shrinker can run, even though the memory allocation may > be coming from a different fs and there's no possibility of a > deadlock. > > I've been talking about this with the mm/ guys and what we can do > differently to pass more context to the shrinkers (e.g. task based > reclaim context structures rather than GFP_* flags passed on the > stack) so we can be smarter in the shrinkers about what we can and > can't do. > >> With dirty pages, kswapd will start IO but not wait on it. >> With the xfs shrinker, kswapd does synchronous IO to write a single >> inode in xfs_buf_submit_wait(). > > That's because there is /no other throttling mechanism/ for shrinker > controlled slab caches. We can't throttle at allocation time because > we have no mechanism for either counting or limiting the number of > dirty objects in a slab caches, like we do the page cache. We have > /limited/ control via the size of the filesystem journal (which is > why I've been asking for that information!), but realistically the > only solid, reliable method we have to prevent excessive dirty inode > accumulation in large memory machines with multiple filesystems and > slow disks is to throttle memory allocation to the rate at which we > can reclaim inodes. > > This prevents the sorts of situations we were regularly seeing 7-8 > years ago where a freeze (for a snapshot) or unmount could take > /hours/ because we have built up hundreds of thousands of dirty > inodes in cache over a period of days because the slow SATA RAID6 > array can only do 50 write IOPS..... > > i.e. there are many, many good reasons for why XFS treats inode > reclaim the way it does. I don't expect you to understand all the > issues that we are preventing by throttling memory allocation like > this, but I do expect you to respect the fact it's done for very > good reasons. > >> With congested BDIs, kswapd will skip the IO and wait for progress >> after running through a good chunk of pages. With the xfs shrinker, >> kswapd will synchronously wait for progress on a single FS, even if >> there are dozens of other filesystems around. > > Yes, that can happen but, again, this behaviour indicates something > else is wrong and the system is out of whack in some way. In the > normal case, kswapd will only block on IO long enough for the other > other async threads run through inode writeback and clean inode > reclaim as efficiently as possible. By the time that the shrinker > wakes up after blocking on an IO, it should either have clean inodes > to reclaim or nothing more to do. > > What you are reporting is equivalent to having pageout() run and do > all the writeback (badly) instead of the bdi flusher threads doing > all the writeback (efficiently). pageout() is a /worst case/ > behaviour we try very hard to avoid and when it occurs it is > generally indicative of some other problem or imbalance. Same goes > here for the inode shrinker. Yes! But the big difference is that pageout() already has a backoff for congestion. The xfs shrinker doesn't. > >> For the xfs shrinker, the mechanism to throttle new dirty inodes on >> a single FS is stalling every process in the system in direct >> reclaim? > > We're not throttling dirty inodes - we are throttling memory > allocation because that's the only hammer we have to prevent > excessive buildup of dirty of inodes that have already been > allocated. > > We can't do that throttling when we dirty inodes because that > happens in transaction context holding locked objects and blocking > there waiting on inode writeback progress will cause journal > deadlocks.... > > This stuff is way more complex than just "have cache, will shrink". Actually we can easily throttle after common metadata operations that dirty inodes. There aren't many, and the throttling can be done after the transaction locks are dropped. > >> We don't think it's already solved in v4.8, but we're setting up a >> test to confirm that. I'm working on a better simulation of the >> parts we're tripping over so I can model this outside of production. >> I definitely agree that something is wrong in MM land too, we have >> to clamp down on the dirty ratios much more than we should to keep >> kswapd from calling pageout(). > > Having pageout() run is pretty indicative of instantaneous memory > demand being significantly higher than the IO throughput of the > storage subsystem and the only reclaimable memory in the system > being dirty filesystem caches. i.e. background writeback is not > keeping up memory demand and dirtying rates for some reason. The > general rule of thumb is that if getting pageout() is occurring then > the IO subsystem is about to die a horrible death of random IO. > > If you're taking great lengths to avoid pageout() from being called, > then it's no surprise to me that your workload is, instead, > triggering the equivalent "oh shit, we're in real trouble here" > behaviour in XFS inode cache reclaim. I also wonder, after turning > down the dirty ratios, if you've done other typical writeback tuning > tweaks like speeding up XFS's periodic metadata writeback to clean > inodes faster in the absence of journal pressure. No, we haven't. I'm trying really hard to avoid the need for 50 billion tunables when the shrinkers are so clearly doing the wrong thing. > > It's detailed information like this that I've been asking for - > there's good reasons I've asked for that information, and any > further discussion will just be a waste of my time without all the > details I've already asked for. I'll send reproduction workloads. It'll take a few days to nail them down, but it'll be much easier to talk about than all of hadoop. -chris ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-10-17 23:20 ` Chris Mason @ 2016-10-18 2:03 ` Dave Chinner 2016-11-14 1:00 ` Chris Mason 0 siblings, 1 reply; 24+ messages in thread From: Dave Chinner @ 2016-10-18 2:03 UTC (permalink / raw) To: Chris Mason; +Cc: linux-xfs On Mon, Oct 17, 2016 at 07:20:56PM -0400, Chris Mason wrote: > On 10/17/2016 06:30 PM, Dave Chinner wrote: > >On Mon, Oct 17, 2016 at 09:30:05AM -0400, Chris Mason wrote: > >What you are reporting is equivalent to having pageout() run and do > >all the writeback (badly) instead of the bdi flusher threads doing > >all the writeback (efficiently). pageout() is a /worst case/ > >behaviour we try very hard to avoid and when it occurs it is > >generally indicative of some other problem or imbalance. Same goes > >here for the inode shrinker. > > Yes! But the big difference is that pageout() already has a backoff > for congestion. The xfs shrinker doesn't. pageout() is only ever called from kswapd context for file pages. Hence applications hitting direct reclaim really hard will never call pageout() directly - they'll skip over it and end up calling congestion_wait() instead and at that point the page cache dirty throttle and background writeback should take over. This, however, breaks down when there are hundreds of direct reclaimers because the pressure put on direct reclaim can exceed the amount of cleaning work that can be done by the background threads in the maximum congestion backoff period and there are other caches that require IO to clean. Direct reclaim then has no clean page cache pages to clean on each LRU scan, so it effectively then transfers that excess pressure to the shrinkers that require IO to reclaim. If a shrinker hits a similar "reclaim pressure > background cleaning rate" threshold, then it will end up directly blocking on IO congestion, exactly as you are describing. Both direct reclaim and kswapd can get stuck in this becase shrinkers - unlike pageout() - are called from direct reclaim as well as kswapd. i.e. Shrinkers are exposed to unbound direct reclaim pressure, pageout() writeback isn't. Hence shrinkers need to handle unbound incoming concurrency without killing IO patterns, without trashing the working set of objects it controls, and it has to - somehow - adequately throttle reclaim rates in times of pressure overload. Right now the XFS code uses IO submission to do that throttling. Shrinkers have no higher layer throttling we can rely on here. Blocking on congestion during IO submission is effectively no different to calling congestion_wait() in the shrinker itself after skipping a bunch of dirty inodes that we can't write because of congestion. While we can do this, it doesn't change the fact that shrinkers that do IO need to block callers to adequately control the reclaim pressure being directed at them. If the XFS background metadata writeback threads are doing their work properly, shrinker reclaim should not be blocking on dirty inodes. However, for all I know right now the problem could be that the background reclaimer is *working too well* and so leaving only dirty inodes for the shrinkers to act on..... IOWs, what we need to do *first* is to work out why there is so much blocking occuring - we need to find the /root cause of the blocking problem/ and once we've found that we can discuss potential solutions. [snip] > >If you're taking great lengths to avoid pageout() from being called, > >then it's no surprise to me that your workload is, instead, > >triggering the equivalent "oh shit, we're in real trouble here" > >behaviour in XFS inode cache reclaim. I also wonder, after turning > >down the dirty ratios, if you've done other typical writeback tuning > >tweaks like speeding up XFS's periodic metadata writeback to clean > >inodes faster in the absence of journal pressure. > > No, we haven't. I'm trying really hard to avoid the need for 50 > billion tunables when the shrinkers are so clearly doing the wrong > thing. XFS has *1* tunable that can change the behaviour of metadata writeback. Please try it. Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-10-18 2:03 ` Dave Chinner @ 2016-11-14 1:00 ` Chris Mason 2016-11-14 7:27 ` Dave Chinner 0 siblings, 1 reply; 24+ messages in thread From: Chris Mason @ 2016-11-14 1:00 UTC (permalink / raw) To: Dave Chinner; +Cc: linux-xfs On Tue, Oct 18, 2016 at 01:03:24PM +1100, Dave Chinner wrote: [ Long stalls from xfs_reclaim_inodes_ag ] >XFS has *1* tunable that can change the behaviour of metadata >writeback. Please try it. [ weeks go by, so this email is insanely long ] Testing all of this was slow going because two of the three test boxes I had with the hadoop configuration starting having hardware problems. The good news is that while I was adjusting the benchmark, we lined up access to a bunch of duplicate boxes, so I can now try ~20 different configurations in parallel. My rough benchmark is here: git://git.kernel.org/pub/scm/linux/kernel/git/mason/simoop.git The command line I ended up using was: simoop -t 512 -m 190 -M 128 -C 28 -r 60000 -f 70 -T 20 -R1 -W 1 -i 60 -w 300 -D 2 /hammer/* I'll break down how I came up with that command line and the end of the email. This isn't trying to be like hadoop, it's more trying to create the same VM pressures that hadoop does. It's missing all of the network traffic, which is at least as much of the workload as disk. The benchmark has a few parts: Fill the drives to ~60-70% full. This working set is reused run to run Read things Write things Burn CPU -- it's important that kswapd have to compete for CPU slices Allocate things I'm pretending to be java here, so anything that could be done with a one thread was done with 10 threads for good luck. Stage one is to create a working set of files across all the drives. We use 15 drives, and each drive has its own filesystem. On my current boxes, the drives are 8TB each, but production can be as small as 4TB. The multiple FS part of the benchmark is important to reproduce what we see, but I doubt you need all 15. If you want to try this out, I'm happy to match your config and help come up with a command line that reproduces for me. While I was reproducing the problem, I was trying to avoid the one part of hadoop that bothers me the most. It has one thread per filesystem that runs du on the whole FS every 10 minutes or so. There are patches in upstream hadoop to use df instead, but ugh. I think what's happening is that when the du thread kicks in, we get a lot of inodes that are tossed soon after. This in turn means XFS_ICI_RECLAIM_TAG is set on a lot of ags, which makes it easier to trigger my stalls. In hopes of simulating a workload less dumb, my simoop runs periodically stat all the files on only two of the 15 drives. The idea is that a mixed workload is slowing the system down to the slowest drive. We find the allocation stalls counters in /proc/vmstat roughly matches poor performance overall. So, the benchmark reads that from time to time includes it in the output. Some example results, plain v4.8 kernel. This run had xfssyncd tuning left at the default, but I did crank it down last week without any real change in the numbers. The bad news is it takes the runs a long time to stabilize, I run at least an hour. v4.8 ___ Run time: 3623 seconds Read latency (p50: 21,594,112) (p95: 31,621,120) (p99: 36,896,768) Write latency (p50: 19,169,280) (p95: 31,096,832) (p99: 37,158,912) Allocation latency (p50: 12,992,512) (p95: 28,934,144) (p99: 36,110,336) work rate = 9.12/sec (avg 8.77/sec) (p50: 8.91) (p95: 9.11) (p99: 9.99) alloc stall rate = 44.98/sec (avg: 41.22) (p50: 39.56) (p95: 41.20) (p99: 50.96) In general, we care most about the p99 of the latencies. Once you average things over the fleet, most users see the p99 often enough that it really hurts. The latency times here are in usec, but read/write latencies are the time it took for N threads to start, do IO and return. work rate is a made up number, and basically summarizes the combination of reading, writing, allocating and hogging the CPU. For that, we'd use the avg or p50 number to measure production. Comparing with the patch: ___ Run time: 3614 seconds Read latency (p50: 18,644,992) (p95: 29,720,576) (p99: 34,930,688) Write latency (p50: 12,173,312) (p95: 22,839,296) (p99: 29,851,648) Allocation latency (p50: 9,322,496) (p95: 25,395,200) (p99: 30,834,688) work rate = 10.82/sec (avg 11.16/sec) (p50: 10.90) (p95: 11.14) (p99: 11.30) alloc stall rate = 28.05/sec (avg: 19.52) (p50: 23.56) (p95: 31.48) (p99: 36.12) I also had a bpf probe running alongside the test to measure how long we're spending in xfs_reclaim_inodes_ag(). argdist.py comes with the bcc iovisor: https://github.com/iovisor/bcc. I ran: argdist.py -i 600 -H 'r::xfs_reclaim_inodes_ag():u64:$latency:$latency < 56137438953472' (v4.9-rc fixes a bug where the latencies go negative, so you have to ignore crazy numbers from v4.8 or older). Unpatched: $latency : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 0 | | 128 -> 255 : 0 | | 256 -> 511 : 0 | | 512 -> 1023 : 0 | | 1024 -> 2047 : 0 | | 2048 -> 4095 : 1 | | 4096 -> 8191 : 0 | | 8192 -> 16383 : 6 | | 16384 -> 32767 : 27 | | 32768 -> 65535 : 35 | | 65536 -> 131071 : 80 | | 131072 -> 262143 : 249 |** | 262144 -> 524287 : 701 |******* | 524288 -> 1048575 : 1761 |********************| 1048576 -> 2097151 : 634 |******* | 2097152 -> 4194303 : 28 | | 4194304 -> 8388607 : 0 | | 8388608 -> 16777215 : 0 | | 16777216 -> 33554431 : 0 | | 33554432 -> 67108863 : 0 | | 67108864 -> 134217727 : 0 | | 134217728 -> 268435455 : 0 | | 268435456 -> 536870911 : 0 | | 536870912 -> 1073741823 : 43 | | 1073741824 -> 2147483647 : 23 | | 2147483648 -> 4294967295 : 23 | | 4294967296 -> 8589934591 : 25 | | 8589934592 -> 17179869183 : 24 | | 17179869184 -> 34359738367 : 2 | | 34359738368 -> 68719476735 : 6 | | -i 600 means to collect data for 600 seconds and then print the histogram and reset. The latencies are in nanoseconds, so v4.8 had six calls that lasted between 34 and 68 seconds. Same stats for v4.8-patched: [04:54:22] $latency : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 0 | | 128 -> 255 : 0 | | 256 -> 511 : 0 | | 512 -> 1023 : 0 | | 1024 -> 2047 : 1 | | 2048 -> 4095 : 8 | | 4096 -> 8191 : 9 | | 8192 -> 16383 : 23 | | 16384 -> 32767 : 48 | | 32768 -> 65535 : 73 | | 65536 -> 131071 : 164 | | 131072 -> 262143 : 245 | | 262144 -> 524287 : 393 | | 524288 -> 1048575 : 857 |* | 1048576 -> 2097151 : 14980 |********************| 2097152 -> 4194303 : 287 | | 4194304 -> 8388607 : 1 | | 8388608 -> 16777215 : 2 | | 16777216 -> 33554431 : 11 | | 33554432 -> 67108863 : 10 | | 67108864 -> 134217727 : 8 | | 134217728 -> 268435455 : 0 | | 268435456 -> 536870911 : 0 | | 536870912 -> 1073741823 : 152 | | There are consistently more calls in the histogram on the patched kernel, and consistently huge stalls (30 seconds plus) in the unpatched kernel. I've already written a small novel here, but I'll talk about where I got the command line for simoop: simoop -t 512 -m 190 -M 128 -C 28 -r 60000 -f 70 -T 20 -R1 -W 1 -i 60 -w 300 -D 2 /hammer/* The box has 130GB of ram, two sockets (14 cores each) the overall goals are: 95-99% user CPU 200MB-500MB/s reads and writes Filesystems 60% full Allocation stalls between 30 and 50/sec Allocation stalls over 100 tend to oom the box eventually. These are the most important knobs: -t 512 worker threads -m 190 is the per-thread working set ram (in MB). It's allocated once per thread and stays in use the whole time. -M 128 is the allocation we time for latencies, it's freed almost right away. -C 28 threads that just burn cpus. The rest tune how much IO is done (-R, -W) and how many subthreads do the IO (-T). -w 300 gives us 300 seconds of warm up time before we reset the stats for the long haul. -D 2 are the du threads. I'm sorry. I have a few more answers for why v3.10 in prod has so many fewer allocation stalls than v4.0+. It's partially because some slabs have gotten a little bigger. It's also because netlink started using larger order allocations (this is an even better story than the du threads), and because our stacks jumped up to 16K contiguous. My goal in posting all of this is to help find a way to put proper throttling on the slab so we don't have to do synchronous waiting and IO in the shrinker. As far as I can tell, simoop roughly mirrors the benefits I get from my one liner in production, but this is v0.000001 so there is a lot of room for improvement. -chris ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-11-14 1:00 ` Chris Mason @ 2016-11-14 7:27 ` Dave Chinner 2016-11-14 20:56 ` Chris Mason 0 siblings, 1 reply; 24+ messages in thread From: Dave Chinner @ 2016-11-14 7:27 UTC (permalink / raw) To: Chris Mason; +Cc: linux-xfs On Sun, Nov 13, 2016 at 08:00:04PM -0500, Chris Mason wrote: > On Tue, Oct 18, 2016 at 01:03:24PM +1100, Dave Chinner wrote: > > [ Long stalls from xfs_reclaim_inodes_ag ] > > >XFS has *1* tunable that can change the behaviour of metadata > >writeback. Please try it. > > [ weeks go by, so this email is insanely long ] > > Testing all of this was slow going because two of the three test > boxes I had with the hadoop configuration starting having hardware > problems. The good news is that while I was adjusting the > benchmark, we lined up access to a bunch of duplicate boxes, so I > can now try ~20 different configurations in parallel. > > My rough benchmark is here: > > git://git.kernel.org/pub/scm/linux/kernel/git/mason/simoop.git > > The command line I ended up using was: > > simoop -t 512 -m 190 -M 128 -C 28 -r 60000 -f 70 -T 20 -R1 -W 1 -i > 60 -w 300 -D 2 /hammer/* There's a lightly tested patch below that should do the trick. After 5 minutes on a modified simoop cli on a single filesystem, 4.9-rc4+for-next: $ ./simoop -t 128 -m 50 -M 128 -C 14 -r 60000 -f 2 -T 20 -R1 -W 1 -i 60 -w 300 -D 2 /mnt/scratch .... Run time: 300 seconds Read latency (p50: 3,174,400) (p95: 4,530,176) (p99: 18,055,168) Write latency (p50: 14,991,360) (p95: 28,672,000) (p99: 33,325,056) Allocation latency (p50: 1,771,520) (p95: 17,530,880) (p99: 23,756,800) work rate = 4.75/sec (avg 5.24/sec) (p50: 5.79) (p95: 6.99) (p99: 6.99) alloc stall rate = 94.42/sec (avg: 51.63) (p50: 51.60) (p95: 59.12) (p99: 59.12) With the patch below: Run time: 300 seconds Read latency (p50: 3,043,328) (p95: 3,649,536) (p99: 4,710,400) Write latency (p50: 21,004,288) (p95: 27,557,888) (p99: 29,130,752) Allocation latency (p50: 280,064) (p95: 680,960) (p99: 863,232) work rate = 4.08/sec (avg 4.76/sec) (p50: 5.39) (p95: 6.93) (p99: 6.93) alloc stall rate = 0.08/sec (avg: 0.02) (p50: 0.00) (p95: 0.01) (p99: 0.01) Stall rate went to zero and stayed there at the 120s mark of the warmup. Note the p99 difference for read and allocation latency, too. I'll post some graphs tomorrow from my live PCP telemetry that demonstrate the difference in behaviour better than any words... Cheers, Dave. -- Dave Chinner david@fromorbit.com xfs: reduce blocking in inode reclaim From: Dave Chinner <dchinner@redhat.com> When we already have maximum reclaim concurrency going on, stop the caller from doing any more reclaim rather than blocking them. Instead, transfer the scanning work the next context that gets access to the reclaim locks. Further, make sure not to block kswapd ireclaim on IO or locks. This means it will always be able to move on to reclaim something else rather than blocking and preventing reclaim from other filesytsems. Finally, change the blocking reclaim cases to follow the reclaim cursor so that we don't restart from a position where inodes may curentl be under IO from a previous flush. This means we cycle through all inodes int eh AG before revisting indoes that may now be clean for reclaim. This change does not appear to cause overall performance regression with the pure dirty inode cache load (such as fsmark inode creation) and inode traversal worklaods (find, rm -rf) on inodes collections much larger than can be cached in memory. The reclaim pattern is noticably lumpier, so work still to be done. Under initial simoop testing, p99 read latencies are down by 70%, p99 allocation latency is down by over 90% and allocation stalls completely go away. Signed-off-by: Dave Chinner <dchinner@redhat.com> --- fs/super.c | 8 +++++++- fs/xfs/xfs_icache.c | 59 +++++++++++++++++++++++++++++++++++++++++++------------- 2 files changed, 53 insertions(+), 14 deletions(-) diff --git a/fs/super.c b/fs/super.c index c183835566c1..9fff5630b12d 100644 --- a/fs/super.c +++ b/fs/super.c @@ -102,8 +102,14 @@ static unsigned long super_cache_scan(struct shrinker *shrink, freed += prune_icache_sb(sb, sc); if (fs_objects) { + unsigned long ret; + sc->nr_to_scan = fs_objects + 1; - freed += sb->s_op->free_cached_objects(sb, sc); + ret = sb->s_op->free_cached_objects(sb, sc); + if (ret == SHRINK_STOP) + freed = SHRINK_STOP; + else + freed += ret; } up_read(&sb->s_umount); diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c index 9c3e5c6ddf20..65c0f79f3edc 100644 --- a/fs/xfs/xfs_icache.c +++ b/fs/xfs/xfs_icache.c @@ -975,7 +975,7 @@ xfs_reclaim_inode( error = 0; xfs_ilock(ip, XFS_ILOCK_EXCL); if (!xfs_iflock_nowait(ip)) { - if (!(sync_mode & SYNC_WAIT)) + if (sync_mode & SYNC_TRYLOCK) goto out; xfs_iflock(ip); } @@ -987,7 +987,7 @@ xfs_reclaim_inode( goto reclaim; } if (xfs_ipincount(ip)) { - if (!(sync_mode & SYNC_WAIT)) + if (sync_mode & SYNC_TRYLOCK) goto out_ifunlock; xfs_iunpin_wait(ip); } @@ -1103,7 +1103,7 @@ xfs_reclaim_inode( * then a shut down during filesystem unmount reclaim walk leak all the * unreclaimed inodes. */ -STATIC int +STATIC long xfs_reclaim_inodes_ag( struct xfs_mount *mp, int flags, @@ -1113,18 +1113,22 @@ xfs_reclaim_inodes_ag( int error = 0; int last_error = 0; xfs_agnumber_t ag; - int trylock = flags & SYNC_TRYLOCK; + int trylock; int skipped; + int dirty_ags; restart: + trylock = flags & SYNC_TRYLOCK; ag = 0; skipped = 0; + dirty_ags = 0; while ((pag = xfs_perag_get_tag(mp, ag, XFS_ICI_RECLAIM_TAG))) { unsigned long first_index = 0; int done = 0; int nr_found = 0; ag = pag->pag_agno + 1; + dirty_ags++; if (trylock) { if (!mutex_trylock(&pag->pag_ici_reclaim_lock)) { @@ -1132,10 +1136,16 @@ xfs_reclaim_inodes_ag( xfs_perag_put(pag); continue; } - first_index = pag->pag_ici_reclaim_cursor; } else mutex_lock(&pag->pag_ici_reclaim_lock); + /* + * Always start from the last scanned inode so that we don't + * block on inodes that a previous iteration just flushed. + * Iterate over the entire inode range before coming back to + * skipped/dirty inodes. + */ + first_index = pag->pag_ici_reclaim_cursor; do { struct xfs_inode *batch[XFS_LOOKUP_BATCH]; int i; @@ -1201,23 +1211,31 @@ xfs_reclaim_inodes_ag( } while (nr_found && !done && *nr_to_scan > 0); - if (trylock && !done) - pag->pag_ici_reclaim_cursor = first_index; - else - pag->pag_ici_reclaim_cursor = 0; + if (done) + first_index = 0; + pag->pag_ici_reclaim_cursor = first_index; mutex_unlock(&pag->pag_ici_reclaim_lock); xfs_perag_put(pag); } /* - * if we skipped any AG, and we still have scan count remaining, do + * If we skipped all AGs because they are locked, we've reached maximum + * reclaim concurrency. At this point there is no point in having more + * attempts to shrink the cache from this context. Tell the shrinker to + * stop and defer the reclaim work till later. + */ + if (skipped && skipped == dirty_ags) + return SHRINK_STOP; + + /* + * If we skipped any AG, and we still have scan count remaining, do * another pass this time using blocking reclaim semantics (i.e * waiting on the reclaim locks and ignoring the reclaim cursors). This * ensure that when we get more reclaimers than AGs we block rather * than spin trying to execute reclaim. */ if (skipped && (flags & SYNC_WAIT) && *nr_to_scan > 0) { - trylock = 0; + flags &= ~SYNC_TRYLOCK; goto restart; } return last_error; @@ -1229,8 +1247,12 @@ xfs_reclaim_inodes( int mode) { int nr_to_scan = INT_MAX; + long ret; - return xfs_reclaim_inodes_ag(mp, mode, &nr_to_scan); + ret = xfs_reclaim_inodes_ag(mp, mode, &nr_to_scan); + if (ret == SHRINK_STOP) + return 0; + return ret; } /* @@ -1241,17 +1263,28 @@ xfs_reclaim_inodes( * reclaim of inodes. That means if we come across dirty inodes, we wait for * them to be cleaned, which we hope will not be very long due to the * background walker having already kicked the IO off on those dirty inodes. + * + * Also, treat kswapd specially - we really want it to run asynchronously and + * not block on dirty inodes, unlike direct reclaim that we can tolerate + * blocking and some amount of IO latency. If we start to overload the reclaim + * subsystem with too many direct reclaimers, it will start returning + * SHRINK_STOP to tell the mm subsystem to defer the work rather than continuing + * to call us and forcing us to block. */ long xfs_reclaim_inodes_nr( struct xfs_mount *mp, int nr_to_scan) { + int flags = SYNC_TRYLOCK; + /* kick background reclaimer and push the AIL */ xfs_reclaim_work_queue(mp); xfs_ail_push_all(mp->m_ail); - return xfs_reclaim_inodes_ag(mp, SYNC_TRYLOCK | SYNC_WAIT, &nr_to_scan); + if (!current_is_kswapd()) + flags |= SYNC_WAIT; + return xfs_reclaim_inodes_ag(mp, flags, &nr_to_scan); } /* ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-11-14 7:27 ` Dave Chinner @ 2016-11-14 20:56 ` Chris Mason 2016-11-14 23:58 ` Dave Chinner 0 siblings, 1 reply; 24+ messages in thread From: Chris Mason @ 2016-11-14 20:56 UTC (permalink / raw) To: Dave Chinner; +Cc: linux-xfs On 11/14/2016 02:27 AM, Dave Chinner wrote: > On Sun, Nov 13, 2016 at 08:00:04PM -0500, Chris Mason wrote: >> On Tue, Oct 18, 2016 at 01:03:24PM +1100, Dave Chinner wrote: >> >> [ Long stalls from xfs_reclaim_inodes_ag ] >> >>> XFS has *1* tunable that can change the behaviour of metadata >>> writeback. Please try it. >> >> [ weeks go by, so this email is insanely long ] >> >> Testing all of this was slow going because two of the three test >> boxes I had with the hadoop configuration starting having hardware >> problems. The good news is that while I was adjusting the >> benchmark, we lined up access to a bunch of duplicate boxes, so I >> can now try ~20 different configurations in parallel. >> >> My rough benchmark is here: >> >> git://git.kernel.org/pub/scm/linux/kernel/git/mason/simoop.git >> >> The command line I ended up using was: >> >> simoop -t 512 -m 190 -M 128 -C 28 -r 60000 -f 70 -T 20 -R1 -W 1 -i >> 60 -w 300 -D 2 /hammer/* > > There's a lightly tested patch below that should do the trick. > > After 5 minutes on a modified simoop cli on a single filesystem, > 4.9-rc4+for-next: > > $ ./simoop -t 128 -m 50 -M 128 -C 14 -r 60000 -f 2 -T 20 -R1 -W 1 -i 60 -w 300 -D 2 /mnt/scratch > .... > Run time: 300 seconds > Read latency (p50: 3,174,400) (p95: 4,530,176) (p99: 18,055,168) > Write latency (p50: 14,991,360) (p95: 28,672,000) (p99: 33,325,056) > Allocation latency (p50: 1,771,520) (p95: 17,530,880) (p99: 23,756,800) > work rate = 4.75/sec (avg 5.24/sec) (p50: 5.79) (p95: 6.99) (p99: 6.99) > alloc stall rate = 94.42/sec (avg: 51.63) (p50: 51.60) (p95: 59.12) (p99: 59.12) > > With the patch below: > > Run time: 300 seconds > Read latency (p50: 3,043,328) (p95: 3,649,536) (p99: 4,710,400) > Write latency (p50: 21,004,288) (p95: 27,557,888) (p99: 29,130,752) > Allocation latency (p50: 280,064) (p95: 680,960) (p99: 863,232) > work rate = 4.08/sec (avg 4.76/sec) (p50: 5.39) (p95: 6.93) (p99: 6.93) > alloc stall rate = 0.08/sec (avg: 0.02) (p50: 0.00) (p95: 0.01) (p99: 0.01) > > Stall rate went to zero and stayed there at the 120s mark of the > warmup. Note the p99 difference for read and allocation latency, > too. > > I'll post some graphs tomorrow from my live PCP telemetry that > demonstrate the difference in behaviour better than any words... Thanks Dave, this is definitely better. But at least for the multi-disk setup, it's not quite there yet. Your patch: ___ Run time: 15535 seconds Read latency (p50: 22,708,224) (p95: 34,668,544) (p99: 41,746,432) Write latency (p50: 21,200,896) (p95: 34,799,616) (p99: 41,877,504) Allocation latency (p50: 11,550,720) (p95: 31,424,512) (p99: 39,518,208) work rate = 7.48/sec (avg 8.41/sec) (p50: 8.69) (p95: 9.57) (p99: 9.87) alloc stall rate = 14.08/sec (avg: 14.85) (p50: 15.74) (p95: 19.74) (p99: 22.04) Original patch: ___ Run time: 15474 seconds Read latency (p50: 20,414,464) (p95: 29,786,112) (p99: 34,275,328) Write latency (p50: 15,155,200) (p95: 25,591,808) (p99: 31,621,120) Allocation latency (p50: 7,675,904) (p95: 22,970,368) (p99: 29,523,968) work rate = 8.33/sec (avg 10.54/sec) (p50: 10.54) (p95: 10.58) (p99: 10.58) alloc stall rate = 37.08/sec (avg: 21.73) (p50: 23.16) (p95: 24.68) (p99: 25.00) v4.8 ___ Run time: 15492 seconds Read latency (p50: 22,642,688) (p95: 35,848,192) (p99: 43,712,512) Write latency (p50: 21,200,896) (p95: 35,454,976) (p99: 43,450,368) Allocation latency (p50: 12,599,296) (p95: 34,144,256) (p99: 41,615,360) work rate = 9.77/sec (avg 8.15/sec) (p50: 8.37) (p95: 9.29) (p99: 9.55) alloc stall rate = 8.33/sec (avg: 33.65) (p50: 34.52) (p95: 37.40) (p99: 37.96) One thing that might have been too far buried in my email yesterday. The read/write latencies include the time to start threads, that's not just IO in there. I've had this running all day, but the differences stabilized after 5-10 minutes. Everyone's p99s trend higher as the day goes on, but the percentage difference stays about the same. I think the difference between mine and yours is we didn't quite get the allocation stalls down to zero, so making tasks wait for the shrinker shows up in the end numbers. For your patch, the stalls from xfs_reclaim_inodes_ag() were about the same as the unpatched kernel yesterday. We still had long tails in the 30 second+ category. I did a trace on vmscan:mm_vmscan_direct_reclaim_begin, and 91% of the allocation stalls were: order=0 may_writepage=1 gfp_flags=GFP_HIGHUSER_MOVABLE|__GFP_ZERO classzone_idx=3 These are all page faults, either during read() syscalls or when simoop was touching all the pages in its mmap()'d area (-M from the cmdline) One detail I didn't give yesterday was these are all using deadline as the IO scheduler. -chris ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-11-14 20:56 ` Chris Mason @ 2016-11-14 23:58 ` Dave Chinner 2016-11-15 3:09 ` Chris Mason 2016-11-15 5:54 ` Dave Chinner 0 siblings, 2 replies; 24+ messages in thread From: Dave Chinner @ 2016-11-14 23:58 UTC (permalink / raw) To: Chris Mason; +Cc: linux-xfs [-- Attachment #1: Type: text/plain, Size: 11106 bytes --] On Mon, Nov 14, 2016 at 03:56:14PM -0500, Chris Mason wrote: > On 11/14/2016 02:27 AM, Dave Chinner wrote: > >On Sun, Nov 13, 2016 at 08:00:04PM -0500, Chris Mason wrote: > >>On Tue, Oct 18, 2016 at 01:03:24PM +1100, Dave Chinner wrote: > >> > >>[ Long stalls from xfs_reclaim_inodes_ag ] > >> > >>>XFS has *1* tunable that can change the behaviour of metadata > >>>writeback. Please try it. > >> > >>[ weeks go by, so this email is insanely long ] > >> > >>Testing all of this was slow going because two of the three test > >>boxes I had with the hadoop configuration starting having hardware > >>problems. The good news is that while I was adjusting the > >>benchmark, we lined up access to a bunch of duplicate boxes, so I > >>can now try ~20 different configurations in parallel. > >> > >>My rough benchmark is here: > >> > >>git://git.kernel.org/pub/scm/linux/kernel/git/mason/simoop.git > >> > >>The command line I ended up using was: > >> > >>simoop -t 512 -m 190 -M 128 -C 28 -r 60000 -f 70 -T 20 -R1 -W 1 -i > >>60 -w 300 -D 2 /hammer/* > > > >There's a lightly tested patch below that should do the trick. > > > >After 5 minutes on a modified simoop cli on a single filesystem, > >4.9-rc4+for-next: > > > >$ ./simoop -t 128 -m 50 -M 128 -C 14 -r 60000 -f 2 -T 20 -R1 -W 1 -i 60 -w 300 -D 2 /mnt/scratch > >.... > >Run time: 300 seconds > >Read latency (p50: 3,174,400) (p95: 4,530,176) (p99: 18,055,168) > >Write latency (p50: 14,991,360) (p95: 28,672,000) (p99: 33,325,056) > >Allocation latency (p50: 1,771,520) (p95: 17,530,880) (p99: 23,756,800) > >work rate = 4.75/sec (avg 5.24/sec) (p50: 5.79) (p95: 6.99) (p99: 6.99) > >alloc stall rate = 94.42/sec (avg: 51.63) (p50: 51.60) (p95: 59.12) (p99: 59.12) > > > >With the patch below: > > > >Run time: 300 seconds > >Read latency (p50: 3,043,328) (p95: 3,649,536) (p99: 4,710,400) > >Write latency (p50: 21,004,288) (p95: 27,557,888) (p99: 29,130,752) > >Allocation latency (p50: 280,064) (p95: 680,960) (p99: 863,232) > >work rate = 4.08/sec (avg 4.76/sec) (p50: 5.39) (p95: 6.93) (p99: 6.93) > >alloc stall rate = 0.08/sec (avg: 0.02) (p50: 0.00) (p95: 0.01) (p99: 0.01) > > > >Stall rate went to zero and stayed there at the 120s mark of the > >warmup. Note the p99 difference for read and allocation latency, > >too. > > > >I'll post some graphs tomorrow from my live PCP telemetry that > >demonstrate the difference in behaviour better than any words... > > Thanks Dave, this is definitely better. But at least for the > multi-disk setup, it's not quite there yet. I can see why now I've run the test for half an hour to get a decent idea of steady state behaviour. I'll come back to that.... > Your patch: > ___ > Run time: 15535 seconds > Read latency (p50: 22,708,224) (p95: 34,668,544) (p99: 41,746,432) > Write latency (p50: 21,200,896) (p95: 34,799,616) (p99: 41,877,504) > Allocation latency (p50: 11,550,720) (p95: 31,424,512) (p99: 39,518,208) > work rate = 7.48/sec (avg 8.41/sec) (p50: 8.69) (p95: 9.57) (p99: 9.87) > alloc stall rate = 14.08/sec (avg: 14.85) (p50: 15.74) (p95: 19.74) > (p99: 22.04) > > Original patch: > ___ > Run time: 15474 seconds > Read latency (p50: 20,414,464) (p95: 29,786,112) (p99: 34,275,328) > Write latency (p50: 15,155,200) (p95: 25,591,808) (p99: 31,621,120) > Allocation latency (p50: 7,675,904) (p95: 22,970,368) (p99: 29,523,968) > work rate = 8.33/sec (avg 10.54/sec) (p50: 10.54) (p95: 10.58) (p99: 10.58) > alloc stall rate = 37.08/sec (avg: 21.73) (p50: 23.16) (p95: 24.68) > (p99: 25.00) Ok, so stall rate is down, but the latencies are still garbage because stalls are happening. > I've had this running all day, but the differences stabilized after > 5-10 minutes. Everyone's p99s trend higher as the day goes on, but > the percentage difference stays about the same. Right - they stabilise after the first few stalls because they dominate the p99 numbers. For example, with my patch, I saw the first stall event between 1080 and 1140 seconds - you can see this clearly in the 3rd chart: Run time: 1080 seconds Read latency (p50: 3,035,136) (p95: 4,284,416) (p99: 5,349,376) Write latency (p50: 27,623,424) (p95: 31,031,296) (p99: 45,154,304) Allocation latency (p50: 240,384) (p95: 709,632) (p99: 1,308,672) work rate = 3.75/sec (avg 3.71/sec) (p50: 3.71) (p95: 4.04) (p99: 4.04) alloc stall rate = 0.65/sec (avg: 0.04) (p50: 0.00) (p95: 0.03) (p99: 0.03) du thread is done /mnt/scratch ___ Run time: 1140 seconds Read latency (p50: 3,035,136) (p95: 4,415,488) (p99: 6,119,424) Write latency (p50: 27,557,888) (p95: 31,490,048) (p99: 45,285,376) Allocation latency (p50: 247,552) (p95: 1,497,088) (p99: 19,496,960) work rate = 3.68/sec (avg 3.71/sec) (p50: 3.71) (p95: 4.04) (p99: 4.04) alloc stall rate = 1.65/sec (avg: 0.12) (p50: 0.00) (p95: 0.12) (p99: 0.12) du thread is running /mnt/scratch So a single "stall event" blows out the p99 latencies really badly. This is probably the single most important revelation about this so far... > I think the difference between mine and yours is we didn't quite get > the allocation stalls down to zero, so making tasks wait for the > shrinker shows up in the end numbers. Right, but so far we haven't answered the obvious question: what triggers the stall events? > For your patch, the stalls from xfs_reclaim_inodes_ag() were about > the same as the unpatched kernel yesterday. We still had long tails > in the 30 second+ category. Because when a stall happens, it's doing more than just trashing the inode cache. The 4 charts I've attached in a tarball are: Chart 1: unpatched kernel, inode/dentry/xfs_inode cache usage Chart 2: unpatched kernel, memory usage Chart 3: unpatched kernel, inode/dentry/xfs_inode cache usage Chart 4: unpatched kernel, memory usage These are taken with a 5s sample period, so we're looking at overall trends, not fine grained behaviour. Things to note: - discontinuities in the graphs indicate an unresponsive machine and samples were lost. Good sign that the machine is stalling on memory allocation - dentry/vfs inode counts come from /proc/sys/fs/{dentry,inode}_state - memory usage chart: "other" is userspace and kernel memory other than the page cache ("cached"). - cached + other + free = 16GB (all memory) - "slab" is a subset of "other", but is split out above 16GB so i can easily see salb cache memory usage - "dirty" is a subset of "cached", but is split out so I can see how much of the page cache is dirty. Now, with that in mind, let's compare the graphs. Let's start with the inode/dentry/xfs_inode cache usage. We can see that the unpatched kernel has rapid rises and falls in cache footprint. The vertical falling edges are evidence of allocation stalls. You can see that it's somewhat regular - the lines stay roughly level, then we see a deep, sharp valley where the caches are completely freed and then immediately repopulate. This generally co-incides with periodic "du scan" starting - it adds new memory pressure, tipping the kernel into memory reclaim, which immediately stalls and trashes the cache. Given that there is now lots of memory free, the du runs at several thousand IOPS (SSDs, remember) and immediately refills the cache and finishes, producing no more memory pressure. The cache then sits at the "full" level until du starts again and we fall off the cliff again. If we switch to the memory usage graph for the unpatched kernel (chart 2), we can see that allocation stalls trash more than just the slab caches - they clean all memory and free all the page cache, too. i.e. it completely trashes any working set that we might have. This is not caused by the shrinker - it has no effect on page cache reclaim. It appears that page reclaim code is, for some reason, agressively freeing the page cache for multiple seconds, even after there is already noticable free memory. This is probably an issue with there being too much direct reclaim stalling due to pressure (e.g. on dirty pages), and then when the cork is pulled we've got lots of threads that then run a full direct reclaim scan and that empties the caches... /me has always said unbound concurrency in memory reclaim and IO writeback is a bad thing because you cannot control and limit behaviour in overload situations like this. Anyway, lets look at the patched kernel inode cache behaviour - chart 3. The first thing to note is that there is a very different pattern - there's a very regular triangular waveform. The trough occurs when the du thread starts, and the peak occurs when the du thread stops. IOWs, the inode cache is no longer retaining single use inodes in the cache for extended periods. The waveform is neat and regular, and it's very clear where the allocation stall at ~1100s occurred - there's a reclaim cliff. If we look at chart 4 right now - the patched kernel memory usage - we can see there is a big dip in the page cache, too. So when an allocation stall event occurs, the bad behaviour is still there. However, the way the system responds to a stall event is very different - it doesn't immediately refill the inode cache. It goes back to the triangle waveform until the next stall occurs. What is particularly interesting in chart 3 is the vfs.inodes.count and xfs_inode slab metrics (red and pink lines respectively). The active xfs_inode slab objects starts off below the vfs inode count, then over time 6 cycles xfs_inode count slowly increases above th vfs inode count. Then we get an allocation stall event, and it returns the xfs_inode count to less than the vfs inode count. The second stall even occurs at a similar point. The xfs_inode > vfs inode count values indicate that we've got XFs inodes in the reclaim state and that reclaim of XFs inodes is not keeping up with the VFS inode cache reclaim rate. This is indicative of either dirty inodes being reclaimed, or we are not putting enough pressure on the XFS inode cache and so it's putting additional pressure on other parts of memory reclaim. This change in balance seems to correlate to allocation stalls being triggered, so I'm going to try to remove that imbalance and see if that avoids the allocation stalls. IOWs, I'm not using latency stats to guide me - they tell me nothing useful about what causes the latency. The charts tell me when there is a reclaim issue - I can see it clear as anything. If the graphs are smooth, maintain a consistent relationship to each other and behave nicely when there are step changes in behaviour then everything else will take care of itself. It's easy to get lost in creating fancy histograms with shiny new tools like ebpf/bcc, but this is mostly wasted effort when it comes to finding the cause of the problem. Latency histograms don't help us understand why the latencies occur, nor do they help track down the cause. The charts I have show that there is an obvious, one-off, large scale memory reclaim event that is causing the stalls and bad latencies. Finding the trigger for that large scale event is what matters, not pretty histograms of it's effect... Cheers, Dave. -- Dave Chinner david@fromorbit.com [-- Attachment #2: charts.tar.gz --] [-- Type: application/gzip, Size: 81821 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-11-14 23:58 ` Dave Chinner @ 2016-11-15 3:09 ` Chris Mason 2016-11-15 5:54 ` Dave Chinner 1 sibling, 0 replies; 24+ messages in thread From: Chris Mason @ 2016-11-15 3:09 UTC (permalink / raw) To: Dave Chinner; +Cc: linux-xfs On 11/14/2016 06:58 PM, Dave Chinner wrote: > On Mon, Nov 14, 2016 at 03:56:14PM -0500, Chris Mason wrote: >> On 11/14/2016 02:27 AM, Dave Chinner wrote: >>> On Sun, Nov 13, 2016 at 08:00:04PM -0500, Chris Mason wrote: >>>> On Tue, Oct 18, 2016 at 01:03:24PM +1100, Dave Chinner wrote: >>>> >>>> [ Long stalls from xfs_reclaim_inodes_ag ] >>>> >>>>> XFS has *1* tunable that can change the behaviour of metadata >>>>> writeback. Please try it. >>>> >>>> [ weeks go by, so this email is insanely long ] >>>> >>>> Testing all of this was slow going because two of the three test >>>> boxes I had with the hadoop configuration starting having hardware >>>> problems. The good news is that while I was adjusting the >>>> benchmark, we lined up access to a bunch of duplicate boxes, so I >>>> can now try ~20 different configurations in parallel. >>>> >>>> My rough benchmark is here: >>>> >>>> git://git.kernel.org/pub/scm/linux/kernel/git/mason/simoop.git >>>> >>>> The command line I ended up using was: >>>> >>>> simoop -t 512 -m 190 -M 128 -C 28 -r 60000 -f 70 -T 20 -R1 -W 1 -i >>>> 60 -w 300 -D 2 /hammer/* >>> >>> There's a lightly tested patch below that should do the trick. >>> >>> After 5 minutes on a modified simoop cli on a single filesystem, >>> 4.9-rc4+for-next: >>> >>> $ ./simoop -t 128 -m 50 -M 128 -C 14 -r 60000 -f 2 -T 20 -R1 -W 1 -i 60 -w 300 -D 2 /mnt/scratch >>> .... >>> Run time: 300 seconds >>> Read latency (p50: 3,174,400) (p95: 4,530,176) (p99: 18,055,168) >>> Write latency (p50: 14,991,360) (p95: 28,672,000) (p99: 33,325,056) >>> Allocation latency (p50: 1,771,520) (p95: 17,530,880) (p99: 23,756,800) >>> work rate = 4.75/sec (avg 5.24/sec) (p50: 5.79) (p95: 6.99) (p99: 6.99) >>> alloc stall rate = 94.42/sec (avg: 51.63) (p50: 51.60) (p95: 59.12) (p99: 59.12) >>> >>> With the patch below: >>> >>> Run time: 300 seconds >>> Read latency (p50: 3,043,328) (p95: 3,649,536) (p99: 4,710,400) >>> Write latency (p50: 21,004,288) (p95: 27,557,888) (p99: 29,130,752) >>> Allocation latency (p50: 280,064) (p95: 680,960) (p99: 863,232) >>> work rate = 4.08/sec (avg 4.76/sec) (p50: 5.39) (p95: 6.93) (p99: 6.93) >>> alloc stall rate = 0.08/sec (avg: 0.02) (p50: 0.00) (p95: 0.01) (p99: 0.01) >>> >>> Stall rate went to zero and stayed there at the 120s mark of the >>> warmup. Note the p99 difference for read and allocation latency, >>> too. >>> >>> I'll post some graphs tomorrow from my live PCP telemetry that >>> demonstrate the difference in behaviour better than any words... >> >> Thanks Dave, this is definitely better. But at least for the >> multi-disk setup, it's not quite there yet. > > I can see why now I've run the test for half an hour to get a decent > idea of steady state behaviour. I'll come back to that.... > >> Your patch: >> ___ >> Run time: 15535 seconds >> Read latency (p50: 22,708,224) (p95: 34,668,544) (p99: 41,746,432) >> Write latency (p50: 21,200,896) (p95: 34,799,616) (p99: 41,877,504) >> Allocation latency (p50: 11,550,720) (p95: 31,424,512) (p99: 39,518,208) >> work rate = 7.48/sec (avg 8.41/sec) (p50: 8.69) (p95: 9.57) (p99: 9.87) >> alloc stall rate = 14.08/sec (avg: 14.85) (p50: 15.74) (p95: 19.74) >> (p99: 22.04) >> >> Original patch: >> ___ >> Run time: 15474 seconds >> Read latency (p50: 20,414,464) (p95: 29,786,112) (p99: 34,275,328) >> Write latency (p50: 15,155,200) (p95: 25,591,808) (p99: 31,621,120) >> Allocation latency (p50: 7,675,904) (p95: 22,970,368) (p99: 29,523,968) >> work rate = 8.33/sec (avg 10.54/sec) (p50: 10.54) (p95: 10.58) (p99: 10.58) >> alloc stall rate = 37.08/sec (avg: 21.73) (p50: 23.16) (p95: 24.68) >> (p99: 25.00) > > Ok, so stall rate is down, but the latencies are still garbage > because stalls are happening. > >> I've had this running all day, but the differences stabilized after >> 5-10 minutes. Everyone's p99s trend higher as the day goes on, but >> the percentage difference stays about the same. > > Right - they stabilise after the first few stalls because they > dominate the p99 numbers. For example, with my patch, I saw the > first stall event between 1080 and 1140 seconds - you can see this > clearly in the 3rd chart: Yes, I went back and forth a bunch of times between zeroing out the p50/p95/p99 periodically or leaving them to gather for the whole run. In general for production we average results over a whole day, so I ended up letting them gather. > > Run time: 1080 seconds > Read latency (p50: 3,035,136) (p95: 4,284,416) (p99: 5,349,376) > Write latency (p50: 27,623,424) (p95: 31,031,296) (p99: 45,154,304) > Allocation latency (p50: 240,384) (p95: 709,632) (p99: 1,308,672) > work rate = 3.75/sec (avg 3.71/sec) (p50: 3.71) (p95: 4.04) (p99: 4.04) > alloc stall rate = 0.65/sec (avg: 0.04) (p50: 0.00) (p95: 0.03) (p99: 0.03) > du thread is done /mnt/scratch > ___ > Run time: 1140 seconds > Read latency (p50: 3,035,136) (p95: 4,415,488) (p99: 6,119,424) > Write latency (p50: 27,557,888) (p95: 31,490,048) (p99: 45,285,376) > Allocation latency (p50: 247,552) (p95: 1,497,088) (p99: 19,496,960) > work rate = 3.68/sec (avg 3.71/sec) (p50: 3.71) (p95: 4.04) (p99: 4.04) > alloc stall rate = 1.65/sec (avg: 0.12) (p50: 0.00) (p95: 0.12) (p99: 0.12) > du thread is running /mnt/scratch > > So a single "stall event" blows out the p99 latencies really badly. > This is probably the single most important revelation about this > so far... Depending on the workload, an allocation stall rate lower than 5 is fine, but some tiers notice anything > 0. If we gave hadoop stalls at 0, they would almost certainly thank us for all the hard work, and then add more load. > >> I think the difference between mine and yours is we didn't quite get >> the allocation stalls down to zero, so making tasks wait for the >> shrinker shows up in the end numbers. > > Right, but so far we haven't answered the obvious question: what > triggers the stall events? > >> For your patch, the stalls from xfs_reclaim_inodes_ag() were about >> the same as the unpatched kernel yesterday. We still had long tails >> in the 30 second+ category. > > Because when a stall happens, it's doing more than just trashing the > inode cache. The 4 charts I've attached in a tarball are: We have variations for these charts in prod, but my test boxes aren't wired into the collection system for some of the stats yet. We usually don't have the granularity of individual slabs, but it can be added if needed. Many proc files do large order allocations when you open them, so they can skew the results. For the next merge window, I have a patch to make seq_buf_alloc() fall back into vmalloc without doing direct reclaim. The page cache on my boxes in vmstat stays relatively constant. I think the huge amount of ram on these machines is soaking up the slop. But I do have a more regular series of stalls (20/sec) than you do, and if I lightened up the workload a bit, I might see your peaks and valleys. The size of the XFS inode cache does vary pretty dramatically, but that's not too surprising with the du threads (sorry). What I found in prod is that generally the allocation stalls are tied to kswapd being in the D state. Whenever they spike, he spits out a stack trace of being stuck <somewhere>. Once it wasn't stuck in slab shrinkers, the next step was being stuck in writepage(). Johannes had a patch to change how eager kswapd was to writepage(). While that gets bashed out upstream, I just dropped our dirty ratios down to saner levels. This brought our stalls down from ~10-100/sec to <1-2. It's not perfect, our write iops and read iops both went up, but the application visible performance was higher so it's good enough for now. We also saw cases where kswapd was starved for CPU, or just sitting around and waiting for congested block devices to make progress. The other comment I'll make is this workload is all spindles. We definitely have flash based workloads hit by this too, but not to the extent these huge drives are. I definitely agree about the value of the graphs in addition to the bpf probes. The bpf histogram answer the question of "is it fixed yet" while the graphs help understand the rest of what's going on. The biggest place bpf helps is being able to say: show me all the stack traces whenever kswapd is stuck longer than X usec From the iovisor git repo, you can try: offcputime.py -m 3000 -p 365 60 where 365 is the pid of one of your kswapds. The answer it spits out is a mixture of writepage() and cond_resched(). Bumping the prio of kswapd and limiting dirty pages cuts my stall down from 20/sec to much lower. We'll have to see what this looks like in the morning, but going back to the first parts of the run with my one liner patch: ~600 seconds alloc stall rate = 32.08/sec (avg: 22.36) (p50: 22.36) (p95: 24.12) (p99: 24.12) vs same kernel but with dirty ratios lower and kswapd prio bumped: alloc stall rate = 3.85/sec (avg: 6.50) (p50: 7.07) (p95: 9.85) (p99: 9.85) Thanks for looking at this. I'm happy to try patches and dig up analysis from specific areas where you want to see stats from real workloads. -chris ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-11-14 23:58 ` Dave Chinner 2016-11-15 3:09 ` Chris Mason @ 2016-11-15 5:54 ` Dave Chinner 2016-11-15 19:00 ` Chris Mason 1 sibling, 1 reply; 24+ messages in thread From: Dave Chinner @ 2016-11-15 5:54 UTC (permalink / raw) To: Chris Mason; +Cc: linux-xfs On Tue, Nov 15, 2016 at 10:58:01AM +1100, Dave Chinner wrote: > On Mon, Nov 14, 2016 at 03:56:14PM -0500, Chris Mason wrote: ..... > So a single "stall event" blows out the p99 latencies really badly. > This is probably the single most important revelation about this > so far... > > > I think the difference between mine and yours is we didn't quite get > > the allocation stalls down to zero, so making tasks wait for the > > shrinker shows up in the end numbers. > > Right, but so far we haven't answered the obvious question: what > triggers the stall events? Our prime suspicion in all this has been that blocking on dirty inodes has been preventing the XFS inode cache shrinker from making progress. That does not appear to be the case at all here. From a half hour sample of my local workload: Inode Clustering xs_iflush_count....... 20119 <<<<<< xs_icluster_flushcnt.. 20000 xs_icluster_flushinode 440411 Vnode Statistics vn_active............. 130903 vn_alloc.............. 0 vn_get................ 0 vn_hold............... 0 vn_rele............... 1217355 vn_reclaim............ 1217355 <<<<<<< vn_remove............. 1217355 There have been 1.2 million inodes reclaimed from the cache, but there have only been 20,000 dirty inode buffer writes. Yes, that's written 440,000 dirty inodes - the inode write clustering is capturing about 22 inodes per write - but the inode writeback load is minimal at about 10 IO/s. XFS inode reclaim is not blocking significantly on dirty inodes. ..... > What is particularly interesting in chart 3 is the vfs.inodes.count > and xfs_inode slab metrics (red and pink lines respectively). The > active xfs_inode slab objects starts off below the vfs inode count, > then over time 6 cycles xfs_inode count slowly increases above th > vfs inode count. Then we get an allocation stall event, and it > returns the xfs_inode count to less than the vfs inode count. The > second stall even occurs at a similar point. Ok, this behaviour is not relevant. It's a result of inode cache scan cycling behaviour that I added where skipped inodes are not immediately revisited by SYNC_WAIT scans. Ok, the more I look into this, the less I think this is an XFS inode cache shrinker blocking problem. These gigantic "allocation stall" are a complete and utter mess at the vmscan level. I caught a trace in a vanilla 4.9-rc4 kernel where a stall event trashed 8GB of page cache and ~60,000 inodes in about 2s flat. This has nothing to do with XFS shrinkers blocking, and everything to do with the mm subsystem going apeshit. Let's start here at 8501.5s, where kswapd suddenly kicks in and starts scanning - The first invocation of the XFs shrinker for node3 reports: kswapd3-861 [015] 8501.546421: mm_shrink_slab_start: super_cache_scan+0x0 0xffff88023abfc4c0: nid: 3 objects to shrink 552 gfp_flags GFP_KERNEL pgs_scanned 45 lru_pgs 198175 cache items 9197 delta 4 total_scan 556 the page cache has ~200k pages, and the inode cache 9000 objects. kswapd didn't scan many pages, didn't add much delta to the shrinker. kswapd keeps running, not finding any pages to reclaim and escalting priority, running the shrinker on each escalation: kswapd3-861 [015] 8501.548073: mm_vmscan_lru_shrink_inactive: nid=3 nr_scanned=22 nr_reclaimed=22 priority=8 flags=RECLAIM_WB_FILE|RECLAIM_WB_ASYNC kswapd3-861 [015] 8501.548090: mm_shrink_slab_start: super_cache_scan+0x0 0xffff88023abfc4c0: nid: 3 objects to shrink 613 gfp_flags GFP_KERNEL pgs_scanned 726 lru_pgs 197537 cache items 9197 delta 67 total_scan 680 By priority 8, we can see the pages scanned and the shrinker delta is going up, but no significant reclaim progress has been made. Priority 7: kswapd3-861 [015] 8501.549347: mm_shrink_slab_start: super_cache_scan+0x0 0xffff88023abfc4c0: nid: 3 objects to shrink 680 gfp_flags GFP_KERNEL pgs_scanned 1448 lru_pgs 196821 cache items 9197 delta 135 total_scan 815 Still no shrinker work, and kswapd stops here, having reached it's thresholds. Shortly after it starts again kswapd3-861 [015] 8501.571629: mm_vmscan_kswapd_wake: nid=3 zid=3 order=0 Priority winds up to 8: kswapd3-861 [015] 8501.572660: mm_shrink_slab_start: super_cache_scan+0x0 0xffff88023abfc4c0: nid: 3 objects to shrink 876 gfp_flags GFP_KERNEL pgs_scanned 718 lru_pgs 195330 cache items 9197 delta 67 total_scan 943 But still not over the shrinker threshold. At priorty 7, it kicks over the batch threshold and we free 326 cached objects: kswapd3-861 [015] 8501.573640: mm_shrink_slab_start: super_cache_scan+0x0 0xffff88023abfc4c0: nid: 3 objects to shrink 943 gfp_flags GFP_KERNEL pgs_scanned 1430 lru_pgs 194619 cache items 9197 delta 135 total_scan 1078 kswapd3-861 [015] 8501.577927: mm_shrink_slab_end: super_cache_scan+0x0 0xffff88023abfc4c0: nid: 3 unused scan count 943 new scan count 54 total_scan 54 last shrinker return val 362 Again, kswapd stops here, to be woken about 10ms later, winds up to priority 7, ends with lru_pgs 192292 cache items 8779 kswapd3-861 [015] 8501.586501: mm_vmscan_kswapd_wake: nid=3 zid=3 order=0 So after another 10 wakeups and P7 reclaims, we are at: kswapd3-861 [015] 8501.711927: mm_shrink_slab_start: super_cache_scan+0x0 0xffff88023abfc4c0: nid: 3 objects to shrink 845 gfp_flags GFP_KERNEL pgs_scanned 1220 lru_pgs 167629 cache items 8400 delta 122 total_scan 967 This pattern continues, freeing pages and winding up priority: kswapd3-861 [015] 8501.763802: mm_vmscan_lru_isolate: isolate_mode=0 classzone=3 order=0 nr_requested=32 nr_scanned=32 nr_taken=32 file=1 kswapd3-861 [015] 8501.799755: mm_vmscan_lru_shrink_inactive: nid=3 nr_scanned=32 nr_reclaimed=32 priority=5 flags=RECLAIM_WB_FILE|RECLAIM_WB_ASYNC kswapd3-861 [015] 8501.899717: mm_vmscan_lru_shrink_inactive: nid=3 nr_scanned=32 nr_reclaimed=32 priority=4 flags=RECLAIM_WB_FILE|RECLAIM_WB_ASYNC by the time we get to priority 4, we've lost half the page cache and a thid of the inode cache: lru_pgs 105911 cache items 6802. kswapd keeps being started and winding up to priority 4, where after another 200ms: kswapd3-861 [015] 8502.093850: mm_vmscan_lru_shrink_inactive: nid=3 nr_scanned=32 nr_reclaimed=32 priority=4 flags=RECLAIM_WB_FILE|RECLAIM_WB_ASYNC .... kswapd3-861 [015] 8502.096649: mm_shrink_slab_start: super_cache_scan+0x0 0xffff88023abfc4c0: nid: 3 objects to shrink 850 gfp_flags GFP_KERNEL pgs_scanned 3104 lru_pgs 75733 cache items 4596 delta 376 total_scan 1226 This inode shirnker invocation blocks for the first time for: kswapd3-861 [015] 8502.423233: mm_shrink_slab_end: super_cache_scan+0x0 0xffff88023abfc4c0: nid: 3 unused scan count 850 new scan count 309 total_scan 202 last shrinker return val 628 About 300ms because it's the cache is now really small and we were asked to scan more than a quarter of it. At this point we've lost 60% of the page cache and half the inode cache. kswapd continues to be run and wind up the reclaim pressure. It's at priority 5 next time the XFS inode cache shrinker is run: kswapd3-861 [015] 8502.441241: mm_shrink_slab_start: super_cache_scan+0x0 0xffff88023abfc4c0: nid: 3 objects to shrink 920 gfp_flags GFP_KERNEL pgs_scanned 1754 lru_pgs 66784 cache items 4417 delta 232 total_scan 1152 kswapd3-861 [015] 8502.706469: mm_shrink_slab_end: super_cache_scan+0x0 0xffff88023abfc4c0: nid: 3 unused scan count 920 new scan count 437 total_scan 128 last shrinker return val 668 Again, we block for 250ms. kswapd then winds up to prior 4 but doesn't quite trigger the shrinker. It then runs again, triggers the shrinker at prior 6, blocks for 400ms, now down to: 8502.815198: lru_pgs 60245 cache items 4078 kwapd continues, winds up to prio 4, triggers shrinker again, blocks for 100ms, and by this time we're at lru_pgs 50169 cache items 2972. We've lost 75% of the page cache and 70% of the inode cache. kswapd is woken again immediately, runs to prio 5, no blocking, no changes in cache footprint. Then: kswapd3-861 [015] 8503.318575: mm_vmscan_kswapd_wake: nid=3 zid=3 order=0 .... kswapd3-861 [015] 8503.327057: mm_vmscan_lru_shrink_inactive: nid=3 nr_scanned=32 nr_reclaimed=32 priority=3 flags=RECLAIM_WB_FILE|RECLAIM_WB_ASYNC We ramp up to priority 3 and by this stage we've done serious damage to the inode caches: kswapd3-861 [015] 8503.330165: mm_shrink_slab_start: super_cache_scan+0x0 0xffff88023abfc4c0: nid: 3 objects to shrink 72 gfp_flags GFP_KERNEL pgs_scanned 3104 lru_pgs 43054 cache items 2007 delta 289 total_scan 361 And we keep scanning at prio 3 putting serious pressure on the page cache, and then it happens: kswapd3-861 [015] 8503.333975: mm_shrink_slab_start: super_cache_scan+0x0 0xffff88023abfc4c0: nid: 3 objects to shrink 361 gfp_flags GFP_KERNEL pgs_scanned 3104 lru_pgs 39969 cache items 2067 delta 321 total_scan 682 .... kswapd3-861 [015] 8503.337100: mm_shrink_slab_start: super_cache_scan+0x0 0xffff88023abfc4c0: nid: 3 objects to shrink 682 gfp_flags GFP_KERNEL pgs_scanned 3104 lru_pgs 36888 cache items 2661 delta 447 total_scan 1129 .... kswapd3-861 [015] 8503.344760: mm_vmscan_lru_shrink_inactive: nid=3 nr_scanned=32 nr_reclaimed=0 priority=3 flags=RECLAIM_WB_FILE|RECLAIM_WB_ASYNC kswapd3-861 [015] 8503.344766: mm_vmscan_lru_isolate: isolate_mode=0 classzone=3 order=0 nr_requested=32 nr_scanned=32 nr_taken=32 file=1 kswapd3-861 [015] 8503.344788: mm_vmscan_writepage: page=0xffffea00003415a5 pfn=3413413 flags=RECLAIM_WB_FILE|RECLAIM_WB_ASYNC kswapd3-861 [015] 8503.344793: mm_vmscan_writepage: page=0xffffea00003bcd44 pfn=3919172 flags=RECLAIM_WB_FILE|RECLAIM_WB_ASYNC Yup, memory reclaim starts dirty page writeback. From this point, there are ~10,000 pages written by memory reclaim over the next 700ms. This is how close we go to kicking the OOM - we hit prio 1: kswapd3-861 [015] 8503.935138: mm_vmscan_lru_shrink_inactive: nid=3 nr_scanned=32 nr_reclaimed=32 priority=1 flags=RECLAIM_WB_FILE|RECLAIM_WB_ASYNC The XFS inode shrinker blocking plays no significant part in this series of events. yes, it contributes to /reclaim latency/, but it is not the cause of the catastrophic breakdown that results in kswapd emptying the page cache and the slab caches to accomodate the memory demand coming from userspace. We can fix the blocking problems with the XFS shrinker, but it's not the shrinker's job to stop this overload situation from happening. Indeed, I just ran the workload with my patch and captured an alloc stall in the same manner with the same tracing. It has the same "kswapd keeps being run and escalating reclaim until there's nothing left to reclaim" behaviour. kswapd never blocks in the XFS inode shrinker now, so the allocation latencies are all from direct reclaim doing work, which is exactly as it should be. The fact that we are seeing dirty page writeback from kswapd indicates that the memory pressure this workload generates from userspace is not being adequately throttled in throttle_direct_reclaim() to allow dirty writeback to be done in an efficient and timely manner. The memory reclaim throttling needs to back off more in overload situations like this - we need to slow down the incoming demand to the reclaim rate rather than just increasing pressure and hoping that kswapd doesn't burn up in a ball of OOM.... Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-11-15 5:54 ` Dave Chinner @ 2016-11-15 19:00 ` Chris Mason 2016-11-16 1:30 ` Dave Chinner 0 siblings, 1 reply; 24+ messages in thread From: Chris Mason @ 2016-11-15 19:00 UTC (permalink / raw) To: Dave Chinner; +Cc: linux-xfs On 11/15/2016 12:54 AM, Dave Chinner wrote: > On Tue, Nov 15, 2016 at 10:58:01AM +1100, Dave Chinner wrote: >> On Mon, Nov 14, 2016 at 03:56:14PM -0500, Chris Mason wrote: > ..... >> So a single "stall event" blows out the p99 latencies really badly. >> This is probably the single most important revelation about this >> so far... >> >>> I think the difference between mine and yours is we didn't quite get >>> the allocation stalls down to zero, so making tasks wait for the >>> shrinker shows up in the end numbers. >> >> Right, but so far we haven't answered the obvious question: what >> triggers the stall events? > > Our prime suspicion in all this has been that blocking on dirty > inodes has been preventing the XFS inode cache shrinker from making > progress. That does not appear to be the case at all here. From > a half hour sample of my local workload: > > > Inode Clustering > xs_iflush_count....... 20119 <<<<<< > xs_icluster_flushcnt.. 20000 > xs_icluster_flushinode 440411 > Vnode Statistics > vn_active............. 130903 > vn_alloc.............. 0 > vn_get................ 0 > vn_hold............... 0 > vn_rele............... 1217355 > vn_reclaim............ 1217355 <<<<<<< > vn_remove............. 1217355 > > There have been 1.2 million inodes reclaimed from the cache, but > there have only been 20,000 dirty inode buffer writes. Yes, that's > written 440,000 dirty inodes - the inode write clustering is > capturing about 22 inodes per write - but the inode writeback load > is minimal at about 10 IO/s. XFS inode reclaim is not blocking > significantly on dirty inodes. I think our machines are different enough that we're not seeing the same problems. Or at least we're seeing different sides of the problem. We have 130GB of ram and on average about 300-500MB of XFS slab, total across all 15 filesystems. Your inodes are small and cuddly, and I'd rather have more than less. I see more with simoop than we see in prod, but either way its a reasonable percentage of system ram considering the horrible things being done. Both patched (yours or mine) and unpatched, XFS inode reclaim is keeping up. With my patch in place, tracing during simoop does show more kswapd prio=1 scanning than unpatched, so I'm clearly stretching the limits a little more. But we've got 30+ days of uptime in prod on almost 60 machines. The oom rate is roughly in line with v3.10, and miles better than v4.0. One other difference is that I have 15 filesystems being shrunk in series, so my chances of hitting a long stall are much higher. If I let bpf tracing on shrink_slab() latencies run for 600 seconds, I see v4.8 called shrink_slab roughly 43,000 times. It spent a total of 244 wall clock seconds in the shrinkers. Giving us an average of .0056 seconds per call. But 67 of those calls consumed 243 of those seconds. The other second of wall time was spread over the other 42,000+ calls. On your machine, you can see the VM wreck your page cache and inode caches during the run. This isn't too surprising because the pages are really use once. We read a random file, we write a random file, and we should expect these pages to be gone before we ever use them again. The real page cache working set of the load is: number of workers * number of IO threads * 1MB or so for good streaming. I've got simoop cranked a little harder than you, so my allocation stalls are in more of a steady state (20/second). lru_pgs tends to shuffle between 12K and 13K per node for most of any tracing runs. [ tracing analysis ] > > The XFS inode shrinker blocking plays no significant part in this > series of events. yes, it contributes to /reclaim latency/, but it > is not the cause of the catastrophic breakdown that results in > kswapd emptying the page cache and the slab caches to accomodate the > memory demand coming from userspace. We can fix the blocking > problems with the XFS shrinker, but it's not the shrinker's job to > stop this overload situation from happening. My bigger concern with the blocking in the shrinker was more around the pile up of processes arguing about how to free a relatively small amount of ram. The source of the overload for us is almost always going to be the users, and any little bit of capacity we give them back will get absorbed with added load. > > Indeed, I just ran the workload with my patch and captured an alloc > stall in the same manner with the same tracing. It has the same > "kswapd keeps being run and escalating reclaim until there's nothing > left to reclaim" behaviour. kswapd never blocks in the XFS inode > shrinker now, so the allocation latencies are all from direct > reclaim doing work, which is exactly as it should be. > > The fact that we are seeing dirty page writeback from kswapd > indicates that the memory pressure this workload generates from > userspace is not being adequately throttled in > throttle_direct_reclaim() to allow dirty writeback to be done in an > efficient and timely manner. The memory reclaim throttling needs to > back off more in overload situations like this - we need to slow > down the incoming demand to the reclaim rate rather than just > increasing pressure and hoping that kswapd doesn't burn up in a ball > of OOM.... Johannes was addressing the dirty writeback from kswapd. His first patch didn't make as big a difference as we hoped, but I've changed around simoop a bunch since then. We'll try again. -chris ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-11-15 19:00 ` Chris Mason @ 2016-11-16 1:30 ` Dave Chinner 2016-11-16 3:03 ` Chris Mason 0 siblings, 1 reply; 24+ messages in thread From: Dave Chinner @ 2016-11-16 1:30 UTC (permalink / raw) To: Chris Mason; +Cc: linux-xfs On Tue, Nov 15, 2016 at 02:00:47PM -0500, Chris Mason wrote: > On 11/15/2016 12:54 AM, Dave Chinner wrote: > >On Tue, Nov 15, 2016 at 10:58:01AM +1100, Dave Chinner wrote: > >>On Mon, Nov 14, 2016 at 03:56:14PM -0500, Chris Mason wrote: > >There have been 1.2 million inodes reclaimed from the cache, but > >there have only been 20,000 dirty inode buffer writes. Yes, that's > >written 440,000 dirty inodes - the inode write clustering is > >capturing about 22 inodes per write - but the inode writeback load > >is minimal at about 10 IO/s. XFS inode reclaim is not blocking > >significantly on dirty inodes. > > I think our machines are different enough that we're not seeing the > same problems. Or at least we're seeing different sides of the > problem. > > We have 130GB of ram and on average about 300-500MB of XFS slab, > total across all 15 filesystems. Your inodes are small and cuddly, > and I'd rather have more than less. I see more with simoop than we > see in prod, but either way its a reasonable percentage of system > ram considering the horrible things being done. So I'm running on 16GB RAM and have 100-150MB of XFS slab. Percentage wise, the inode cache is a larger portion of memory than in your machines. I can increase the number of files to increase it further, but I don't think that will change anything. > Both patched (yours or mine) and unpatched, XFS inode reclaim is > keeping up. With my patch in place, tracing during simoop does > show more kswapd prio=1 scanning than unpatched, so I'm clearly > stretching the limits a little more. But we've got 30+ days of > uptime in prod on almost 60 machines. The oom rate is roughly in > line with v3.10, and miles better than v4.0. IOWs, you have a workaround that keeps your production systems running. That's fine for your machines that are running this load, but it's not working well for any of the other other loads I've looked at. That is, removing the throttling from the XFS inode shrinker causes instability and adverse reclaim of the inode cache in situations where the maintaining a working set in memory is required for performance. Indeed, one of the things I noticed with the simoops workload running the shrinker patches is that it no longer kept either the inode cache or the XFS metadata cache in memory long enough for the du to run without requiring IO. i.e. the caches no longer maintained the working set of objects needed to optimise a regular operation and the du scans took a lot longer. That's why on the vanilla kernels the inode cache footprint went through steep sided valleys - reclaim would trash the inode cache, but the metadata cache stayed intact and so all the inodes were imemdiately pulled from there again and populated back into the inode cache. With the patches to remove the XFS shrinker blocking, the pressure was moved to other caches like the metadata cache, and so the clean inode buffers were reclaimed instead. Hence when the inodes were reclaimed, IO was necessary to re-read the inodes during the du scan, and hence the cache growth was also slow. That's what removing the blocking from the shrinker causes the overall work rate to go down - it results in the cache not maintaining a working set of inodes and so increases the IO load and that then slows everything down. There's secondary and tertiary effects all over the place that, and from the XFS POV this is a catch-22. The shrinker blocking has been put in place to control the impact of unbound reclaim concurrency on the working set caches need to maintain to sustain acceptible performance. This blocking, however, is causing issues with latency under your workload. If we remove the shrinker blocking to address the FB allocation latency issue, then we screw up the cached working set balance for every other XFS user out there and we'll end up making things worse for many of XFS users. Quite frankly, if I have to choose between these two things, then I'm not going to change the shrinker implementation. FB can maintain their own fixes until such a point in time that the underlying reclaim problem that requires the XFS shrinker to block has been fully addressed and then we can change the XFS shrinker to work well in all situations. > >The XFS inode shrinker blocking plays no significant part in this > >series of events. yes, it contributes to /reclaim latency/, but it > >is not the cause of the catastrophic breakdown that results in > >kswapd emptying the page cache and the slab caches to accomodate the > >memory demand coming from userspace. We can fix the blocking > >problems with the XFS shrinker, but it's not the shrinker's job to > >stop this overload situation from happening. > > My bigger concern with the blocking in the shrinker was more around > the pile up of processes arguing about how to free a relatively > small amount of ram. This is not a shrinker problem, though. The shrinkers should be completely isolated from allocation demand concurrency. The fact is that they aren't isolated from it, and we have to deal with that as best we can. IOWs, this is a direct reclaim architecture problem. i.e. it presents unbound concurrency to the shrinkers and then requires them to "behave nicely" when the mm subsystem starts saying "I don't care that you're already dealing with 200 other concurrent calls from me - fucking well free everything for me now!". Controlling and limiting the unbound concurrency of reclaim and isolating shrinkers from the incoming demand is the only way we can sanely keep both the reclaim latency to a minimum and maintain a decent working set in the caches under extreme memory pressure. We obviously cannot do both in a shrinker implementation, so we really need a high level re-architecting here... > The source of the overload for us is almost > always going to be the users, and any little bit of capacity we give > them back will get absorbed with added load. Exactly why we need to re-acrhitect reclaim. because if we don't, then the users will simply increase the load until it reclaim breaks down through whatever band-aid we've added to hide the last problem... Put simply: reclaim algorithms should not change just because there are more processes demanding memory - increased demand should simply mean that the processes demanding memory should /wait longer/. Right now they end up waiting longer by adding load and concurrency to the reclaim subsystems, and somewhere in those reclaim subsystems we end up blocking to try to avoid catastrophic degradations. This is exactly analogous to the IO-less dirty page throttling situation we battled with for years. We had an architecture where we had direct submission of IO that throttling in the block layer on request queues. When we had tens to hundreds of processes all doing this, the IO patterns randomised, throughput tanked completely and applications saw extremely non-deterministic long-tail latencies during write() operations. We fixed this by decoupling incoming process dirty page throttling from the mechanism of cleaning of dirty pages. We now have a queue of incoming processes that wait in turn for a number of pages to be cleaned, and when that threshold is cleaned by the background flusher threads, they are woken and on they go. it's efficient, reliable, predictable and, above all, is completely workload independent. We haven't had a "system is completely unresponsive because I did a large write" problem since we made this architectural change - we solved the catastrophic overload problem one and for all.(*) Direct memory reclaim is doing exactly what the old dirty page throttle did - it is taking direct action and relying the underlying reclaim mechanisms to throttle overload situations. Just like the request queue throttling in the old dirty page code, the memory reclaim subsystem is unable to behave sanely when large amounts of concurrent pressure is put on it. The throttling happens too late, too unpredictably, and too randomly for it to be controllable and stable. And the result of that is that application see non-deterministic long-tail latencies once memory reclaim starts. We've already got background reclaim threads - kswapd - and there are already hooks for throttling direct reclaim (throttle_direct_reclaim()). The problem is that direct reclaim throttling only kicks in once we are very near to low memory limits, so it doesn't prevent concurency and load from being presented to the underlying reclaim mechanism until it's already too late. IMO, direct reclaim should be replaced with a queuing mechanism and deferral to kswapd to clean pages. Every time kswapd completes a batch of freeing, it can check if it's freed enough to allow the head of the queue to make progress. If it has, then it can walk down the queue waking processes until all the pages it just freed have been accounted for. If we want to be truly fair, this queuing should occur at the allocation entry points, not the direct reclaim entry point. i.e if we are in a reclaim situation, go sit in the queue until you're told we have memory for you and then run allocation. Then we can design page scanning and shrinkers for maximum efficiency, to be fully non-blocking, and to never have to directly issue or wait for IO completion. They can all feed back reclaim state to a central backoff mechanism which can sleep to alleviate situations where reclaim cannot be done without blocking. This allows us to constrain reclaim to a well controlled set of background threads that we can scale according to observed need. We know that this model works - IO-less dirty page throttling has been a spectacular success. We now just take it for granted that the thottling works because it self tunes to the underlying storage characteristics and rarely, if ever, does the wrong thing. The same cannot be said about memory reclaim behaviour.... > >The fact that we are seeing dirty page writeback from kswapd > >indicates that the memory pressure this workload generates from > >userspace is not being adequately throttled in > >throttle_direct_reclaim() to allow dirty writeback to be done in an > >efficient and timely manner. The memory reclaim throttling needs to > >back off more in overload situations like this - we need to slow > >down the incoming demand to the reclaim rate rather than just > >increasing pressure and hoping that kswapd doesn't burn up in a ball > >of OOM.... > > Johannes was addressing the dirty writeback from kswapd. His first > patch didn't make as big a difference as we hoped, but I've changed > around simoop a bunch since then. We'll try again. We need an architectural change - bandaids aren't going to solve the problem... Cheers, Dave. (*) Yes, I'm aware of Jen's block throttling patches - that's fixing an IO scheduling issue to avoid long read latencies due to background writeback being /too efficient/ at cleaning pages when we're driving the system really hard. IOWs, it's a good problem to have because it's a result of things working too well under load... -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-11-16 1:30 ` Dave Chinner @ 2016-11-16 3:03 ` Chris Mason 2016-11-16 23:31 ` Dave Chinner 2016-11-17 0:47 ` Dave Chinner 0 siblings, 2 replies; 24+ messages in thread From: Chris Mason @ 2016-11-16 3:03 UTC (permalink / raw) To: Dave Chinner; +Cc: linux-xfs On Wed, Nov 16, 2016 at 12:30:09PM +1100, Dave Chinner wrote: >On Tue, Nov 15, 2016 at 02:00:47PM -0500, Chris Mason wrote: >> On 11/15/2016 12:54 AM, Dave Chinner wrote: >> >On Tue, Nov 15, 2016 at 10:58:01AM +1100, Dave Chinner wrote: >> >>On Mon, Nov 14, 2016 at 03:56:14PM -0500, Chris Mason wrote: >> >There have been 1.2 million inodes reclaimed from the cache, but >> >there have only been 20,000 dirty inode buffer writes. Yes, that's >> >written 440,000 dirty inodes - the inode write clustering is >> >capturing about 22 inodes per write - but the inode writeback load >> >is minimal at about 10 IO/s. XFS inode reclaim is not blocking >> >significantly on dirty inodes. >> >> I think our machines are different enough that we're not seeing the >> same problems. Or at least we're seeing different sides of the >> problem. >> >> We have 130GB of ram and on average about 300-500MB of XFS slab, >> total across all 15 filesystems. Your inodes are small and cuddly, >> and I'd rather have more than less. I see more with simoop than we >> see in prod, but either way its a reasonable percentage of system >> ram considering the horrible things being done. > >So I'm running on 16GB RAM and have 100-150MB of XFS slab. >Percentage wise, the inode cache is a larger portion of memory than >in your machines. I can increase the number of files to increase it >further, but I don't think that will change anything. I think the way to see what I'm seeing would be to drop the number of IO threads (-T) and bump both -m and -M. Basically less inode working set and more memory working set. >> Both patched (yours or mine) and unpatched, XFS inode reclaim is >> keeping up. With my patch in place, tracing during simoop does >> show more kswapd prio=1 scanning than unpatched, so I'm clearly >> stretching the limits a little more. But we've got 30+ days of >> uptime in prod on almost 60 machines. The oom rate is roughly in >> line with v3.10, and miles better than v4.0. > >IOWs, you have a workaround that keeps your production systems >running. That's fine for your machines that are running this load, >but it's not working well for any of the other other loads I've >looked at. That is, removing the throttling from the XFS inode >shrinker causes instability and adverse reclaim of the inode cache >in situations where the maintaining a working set in memory is >required for performance. We agree on all of this much more than not. Josef has spent a lot of time recently on shrinkers (w/btrfs but the ideas are similar), and I'm wrapping duct tape around workloads until the overall architecture is less fragile. Using slab for metadata in an FS like btrfs where dirty metadata is almost unbounded is a huge challenge in the current framework. Ext4 is moving to dramatically bigger logs, so it would eventually have the same problems. > >Indeed, one of the things I noticed with the simoops workload >running the shrinker patches is that it no longer kept either the >inode cache or the XFS metadata cache in memory long enough for the >du to run without requiring IO. i.e. the caches no longer maintained >the working set of objects needed to optimise a regular operation >and the du scans took a lot longer. With simoop, du is supposed to do IO. It's crazy to expect to be able to scan all the inodes on a huge FS (or 15 of them) and keep it all in cache along with everything else hadoop does. I completely agree there are cases where having the working set in ram is valid, just simoop isn't one ;) > >That's why on the vanilla kernels the inode cache footprint went >through steep sided valleys - reclaim would trash the inode cache, >but the metadata cache stayed intact and so all the inodes were >imemdiately pulled from there again and populated back into the >inode cache. With the patches to remove the XFS shrinker blocking, >the pressure was moved to other caches like the metadata cache, and >so the clean inode buffers were reclaimed instead. Hence when the >inodes were reclaimed, IO was necessary to re-read the inodes during >the du scan, and hence the cache growth was also slow. > >That's what removing the blocking from the shrinker causes the >overall work rate to go down - it results in the cache not >maintaining a working set of inodes and so increases the IO load and >that then slows everything down. At least on my machines, it made the overall work rate go up. Both simoop and prod are 10-15% faster. We have one other workload (gluster) where I have no idea if it'll help or hurt, but it'll probably be January before I have benchmark numbers from them. I think it'll help, they do have more of a real working set in page cache, but it still breaks down to random IO over time. [ snipping out large chunks, lots to agree with in here ] >We fixed this by decoupling incoming process dirty page throttling >from the mechanism of cleaning of dirty pages. We now have a queue >of incoming processes that wait in turn for a number of pages to be >cleaned, and when that threshold is cleaned by the background >flusher threads, they are woken and on they go. it's efficient, >reliable, predictable and, above all, is completely workload >independent. We haven't had a "system is completely unresponsive >because I did a large write" problem since we made this >architectural change - we solved the catastrophic overload problem >one and for all.(*) (*) Agree Jens' patches are pushing io scheduling help higher up the stack. It's a big win, but not directly for reclaim. > >Direct memory reclaim is doing exactly what the old dirty page >throttle did - it is taking direct action and relying the underlying >reclaim mechanisms to throttle overload situations. Just like the >request queue throttling in the old dirty page code, the memory >reclaim subsystem is unable to behave sanely when large amounts of >concurrent pressure is put on it. The throttling happens too late, >too unpredictably, and too randomly for it to be controllable and >stable. And the result of that is that application see >non-deterministic long-tail latencies once memory reclaim starts. > >We've already got background reclaim threads - kswapd - and there >are already hooks for throttling direct reclaim >(throttle_direct_reclaim()). The problem is that direct reclaim >throttling only kicks in once we are very near to low memory limits, >so it doesn't prevent concurency and load from being presented to >the underlying reclaim mechanism until it's already too late. > >IMO, direct reclaim should be replaced with a queuing mechanism and >deferral to kswapd to clean pages. Every time kswapd completes a >batch of freeing, it can check if it's freed enough to allow the >head of the queue to make progress. If it has, then it can walk down >the queue waking processes until all the pages it just freed have >been accounted for. > >If we want to be truly fair, this queuing should occur at the >allocation entry points, not the direct reclaim entry point. i.e if >we are in a reclaim situation, go sit in the queue until you're told >we have memory for you and then run allocation. > >Then we can design page scanning and shrinkers for maximum >efficiency, to be fully non-blocking, and to never have to directly >issue or wait for IO completion. They can all feed back reclaim >state to a central backoff mechanism which can sleep to alleviate >situations where reclaim cannot be done without blocking. This >allows us to constrain reclaim to a well controlled set of >background threads that we can scale according to observed need. > Can't argue here. The middle ground today is Josef's LRU ideas so that slab reclaim has hopes of doing the most useful work instead of just writing things and hoping for the best. It can either be a band-aid or a building block depending on how you look at it, but it can help either way. Moving forward, I think I can manage to carry the one line patch in code that hasn't measurably changed in years. We'll get it tested in a variety of workloads and come back with more benchmarks for the great slab rework coming soon to a v5.x kernel near you. -chris ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-11-16 3:03 ` Chris Mason @ 2016-11-16 23:31 ` Dave Chinner 2016-11-17 0:27 ` Chris Mason 2016-11-17 0:47 ` Dave Chinner 1 sibling, 1 reply; 24+ messages in thread From: Dave Chinner @ 2016-11-16 23:31 UTC (permalink / raw) To: Chris Mason; +Cc: linux-xfs On Tue, Nov 15, 2016 at 10:03:52PM -0500, Chris Mason wrote: > On Wed, Nov 16, 2016 at 12:30:09PM +1100, Dave Chinner wrote: > >On Tue, Nov 15, 2016 at 02:00:47PM -0500, Chris Mason wrote: > >>On 11/15/2016 12:54 AM, Dave Chinner wrote: > >>>On Tue, Nov 15, 2016 at 10:58:01AM +1100, Dave Chinner wrote: > >>>>On Mon, Nov 14, 2016 at 03:56:14PM -0500, Chris Mason wrote: > >>>There have been 1.2 million inodes reclaimed from the cache, but > >>>there have only been 20,000 dirty inode buffer writes. Yes, that's > >>>written 440,000 dirty inodes - the inode write clustering is > >>>capturing about 22 inodes per write - but the inode writeback load > >>>is minimal at about 10 IO/s. XFS inode reclaim is not blocking > >>>significantly on dirty inodes. > >> > >>I think our machines are different enough that we're not seeing the > >>same problems. Or at least we're seeing different sides of the > >>problem. > >> > >>We have 130GB of ram and on average about 300-500MB of XFS slab, > >>total across all 15 filesystems. Your inodes are small and cuddly, > >>and I'd rather have more than less. I see more with simoop than we > >>see in prod, but either way its a reasonable percentage of system > >>ram considering the horrible things being done. > > > >So I'm running on 16GB RAM and have 100-150MB of XFS slab. > >Percentage wise, the inode cache is a larger portion of memory than > >in your machines. I can increase the number of files to increase it > >further, but I don't think that will change anything. > > I think the way to see what I'm seeing would be to drop the number > of IO threads (-T) and bump both -m and -M. Basically less inode > working set and more memory working set. If I increase m/M by any non-trivial amount, the test OOMs within a couple of minutes of starting even after cutting the number of IO threads in half. I've managed to increase -m by 10% without OOM - I'll keep trying to increase this part of the load as much as I can as I refine the patchset I have. > >>Both patched (yours or mine) and unpatched, XFS inode reclaim is > >>keeping up. With my patch in place, tracing during simoop does > >>show more kswapd prio=1 scanning than unpatched, so I'm clearly > >>stretching the limits a little more. But we've got 30+ days of > >>uptime in prod on almost 60 machines. The oom rate is roughly in > >>line with v3.10, and miles better than v4.0. > > > >IOWs, you have a workaround that keeps your production systems > >running. That's fine for your machines that are running this load, > >but it's not working well for any of the other other loads I've > >looked at. That is, removing the throttling from the XFS inode > >shrinker causes instability and adverse reclaim of the inode cache > >in situations where the maintaining a working set in memory is > >required for performance. > > We agree on all of this much more than not. Josef has spent a lot > of time recently on shrinkers (w/btrfs but the ideas are similar), > and I'm wrapping duct tape around workloads until the overall > architecture is less fragile. > > Using slab for metadata in an FS like btrfs where dirty metadata is > almost unbounded is a huge challenge in the current framework. Ext4 > is moving to dramatically bigger logs, so it would eventually have > the same problems. Your 8TB XFS filesystems will be using 2GB logs (unless mkfs settings were tweaked manually), so there's a huge amount of metadata 15x8TB XFS filesystems can pin in memory, too... > >Indeed, one of the things I noticed with the simoops workload > >running the shrinker patches is that it no longer kept either the > >inode cache or the XFS metadata cache in memory long enough for the > >du to run without requiring IO. i.e. the caches no longer maintained > >the working set of objects needed to optimise a regular operation > >and the du scans took a lot longer. > > With simoop, du is supposed to do IO. It's crazy to expect to be > able to scan all the inodes on a huge FS (or 15 of them) and keep it > all in cache along with everything else hadoop does. I completely > agree there are cases where having the working set in ram is valid, > just simoop isn't one ;) Sure, I was just pointing out that even simoop was seeing signficant changes in cache residency as a result of this change.... > >That's what removing the blocking from the shrinker causes the > >overall work rate to go down - it results in the cache not > >maintaining a working set of inodes and so increases the IO load and > >that then slows everything down. > > At least on my machines, it made the overall work rate go up. Both > simoop and prod are 10-15% faster. Ok, I'll see if I can tune the workload here to behave more like this.... Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-11-16 23:31 ` Dave Chinner @ 2016-11-17 0:27 ` Chris Mason 2016-11-17 1:00 ` Dave Chinner 0 siblings, 1 reply; 24+ messages in thread From: Chris Mason @ 2016-11-17 0:27 UTC (permalink / raw) To: Dave Chinner; +Cc: linux-xfs On Thu, Nov 17, 2016 at 10:31:36AM +1100, Dave Chinner wrote: >On Tue, Nov 15, 2016 at 10:03:52PM -0500, Chris Mason wrote: >> On Wed, Nov 16, 2016 at 12:30:09PM +1100, Dave Chinner wrote: >> >On Tue, Nov 15, 2016 at 02:00:47PM -0500, Chris Mason wrote: >> >>On 11/15/2016 12:54 AM, Dave Chinner wrote: >> >>>On Tue, Nov 15, 2016 at 10:58:01AM +1100, Dave Chinner wrote: >> >>>>On Mon, Nov 14, 2016 at 03:56:14PM -0500, Chris Mason wrote: >> >>>There have been 1.2 million inodes reclaimed from the cache, but >> >>>there have only been 20,000 dirty inode buffer writes. Yes, that's >> >>>written 440,000 dirty inodes - the inode write clustering is >> >>>capturing about 22 inodes per write - but the inode writeback load >> >>>is minimal at about 10 IO/s. XFS inode reclaim is not blocking >> >>>significantly on dirty inodes. >> >> >> >>I think our machines are different enough that we're not seeing the >> >>same problems. Or at least we're seeing different sides of the >> >>problem. >> >> >> >>We have 130GB of ram and on average about 300-500MB of XFS slab, >> >>total across all 15 filesystems. Your inodes are small and cuddly, >> >>and I'd rather have more than less. I see more with simoop than we >> >>see in prod, but either way its a reasonable percentage of system >> >>ram considering the horrible things being done. >> > >> >So I'm running on 16GB RAM and have 100-150MB of XFS slab. >> >Percentage wise, the inode cache is a larger portion of memory than >> >in your machines. I can increase the number of files to increase it >> >further, but I don't think that will change anything. >> >> I think the way to see what I'm seeing would be to drop the number >> of IO threads (-T) and bump both -m and -M. Basically less inode >> working set and more memory working set. > >If I increase m/M by any non-trivial amount, the test OOMs within a >couple of minutes of starting even after cutting the number of IO >threads in half. I've managed to increase -m by 10% without OOM - >I'll keep trying to increase this part of the load as much as I >can as I refine the patchset I have. Gotcha. -m is long lasting, allocated once at the start of the run and stays around for ever. It basically soaks up ram. -M is allocated once per work loop, and it should be where the stalls really hit. I'll peel off a flash machine tomorrow and find a command line that matches my results so far. What kind of flash are you using? I can choose between modern nvme or something more crusty. >> With simoop, du is supposed to do IO. It's crazy to expect to be >> able to scan all the inodes on a huge FS (or 15 of them) and keep it >> all in cache along with everything else hadoop does. I completely >> agree there are cases where having the working set in ram is valid, >> just simoop isn't one ;) > >Sure, I was just pointing out that even simoop was seeing signficant >changes in cache residency as a result of this change.... Yeah, one of the problems with simoop is it should actually go faster if we empty all the caches every time. It only needs enough dirty pages pages around for efficient IO. I should add a page-reuse-after-N-seconds mode so that it notices if some jerk tries a patch that tosses all the pages. It won't make it any less effective for pretending to be hadoop, and it'll catch some mistakes I'm likely to make. > >> >That's what removing the blocking from the shrinker causes the >> >overall work rate to go down - it results in the cache not >> >maintaining a working set of inodes and so increases the IO load and >> >that then slows everything down. >> >> At least on my machines, it made the overall work rate go up. Both >> simoop and prod are 10-15% faster. > >Ok, I'll see if I can tune the workload here to behave more like >this.... What direction do you have in mind for your current patches? Many tiers have shadows where we can put experimental code without feeling bad if machines crash or data is lost. I'm happy to line up tests if you want data from specific workloads. -chris ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-11-17 0:27 ` Chris Mason @ 2016-11-17 1:00 ` Dave Chinner 0 siblings, 0 replies; 24+ messages in thread From: Dave Chinner @ 2016-11-17 1:00 UTC (permalink / raw) To: Chris Mason; +Cc: linux-xfs On Wed, Nov 16, 2016 at 07:27:27PM -0500, Chris Mason wrote: > On Thu, Nov 17, 2016 at 10:31:36AM +1100, Dave Chinner wrote: > >>>So I'm running on 16GB RAM and have 100-150MB of XFS slab. > >>>Percentage wise, the inode cache is a larger portion of memory than > >>>in your machines. I can increase the number of files to increase it > >>>further, but I don't think that will change anything. > >> > >>I think the way to see what I'm seeing would be to drop the number > >>of IO threads (-T) and bump both -m and -M. Basically less inode > >>working set and more memory working set. > > > >If I increase m/M by any non-trivial amount, the test OOMs within a > >couple of minutes of starting even after cutting the number of IO > >threads in half. I've managed to increase -m by 10% without OOM - > >I'll keep trying to increase this part of the load as much as I > >can as I refine the patchset I have. > > Gotcha. -m is long lasting, allocated once at the start of the run > and stays around for ever. It basically soaks up ram. -M is > allocated once per work loop, and it should be where the stalls > really hit. I'll peel off a flash machine tomorrow and find a > command line that matches my results so far. > > What kind of flash are you using? I can choose between modern nvme > or something more crusty. Crusty old stuff - a pair of EVO 840s in HW-RAID0 behind 512MB of BBWC. Read rates peak at ~150MB/s, write rates sustain at about 75MB/s. I'm testing on a 200GB filesystem, configured as: mkfs.xfs -f -dagcount=8,size=200g /dev/vdc The backing file for /dev/vdc is fully preallocated and linear, accessed via virtio/direct IO, so it's no different to accessing the real block device.... > >>>That's what removing the blocking from the shrinker causes the > >>>overall work rate to go down - it results in the cache not > >>>maintaining a working set of inodes and so increases the IO load and > >>>that then slows everything down. > >> > >>At least on my machines, it made the overall work rate go up. Both > >>simoop and prod are 10-15% faster. > > > >Ok, I'll see if I can tune the workload here to behave more like > >this.... > > What direction do you have in mind for your current patches? Many > tiers have shadows where we can put experimental code without > feeling bad if machines crash or data is lost. I'm happy to line up > tests if you want data from specific workloads. Right now I have kswapd as fully non-blocking - even more so that your one line patch because reclaim can (and does) still block on inode locks with SYNC_TRYLOCK set. I don't see any problems with doing this. I'm still trying to work out what to do with direct reclaim - it's clearly the source of the worst allocation latency problems, and it also seems to be the contributing factor to the obnoxious kswapd FFE behaviour. There's a couple more variations I want to try to see if I can make it block less severely, but what I do in the short term here is largely dependent on the effect on other benchmarks and loads.... Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-11-16 3:03 ` Chris Mason 2016-11-16 23:31 ` Dave Chinner @ 2016-11-17 0:47 ` Dave Chinner 2016-11-17 1:07 ` Chris Mason 1 sibling, 1 reply; 24+ messages in thread From: Dave Chinner @ 2016-11-17 0:47 UTC (permalink / raw) To: Chris Mason; +Cc: linux-xfs On Tue, Nov 15, 2016 at 10:03:52PM -0500, Chris Mason wrote: > Moving forward, I think I can manage to carry the one line patch in > code that hasn't measurably changed in years. We'll get it tested > in a variety of workloads and come back with more benchmarks for the > great slab rework coming soon to a v5.x kernel near you. FWIW, I just tested your one-liner against my simoops config here, and by comparing the behaviour to my patchset that still allows direct reclaim to block on dirty inodes, it would appear that all the allocation latency I'm seeing here is from direct reclaim. So I went looking at the direct reclaim throttle with the intent to hack it to throttle earlier. It throttles based on watermarks, so I figured Id just hack them to be larger to trigger direct reclaim throttling earlier. And then I found this recent addition: https://patchwork.kernel.org/patch/8426381/ +============================================================= + +watermark_scale_factor: + +This factor controls the aggressiveness of kswapd. It defines the +amount of memory left in a node/system before kswapd is woken up and +how much memory needs to be free before kswapd goes back to sleep. + +The unit is in fractions of 10,000. The default value of 10 means the +distances between watermarks are 0.1% of the available memory in the +node/system. The maximum value is 1000, or 10% of memory. + +A high rate of threads entering direct reclaim (allocstall) or kswapd +going to sleep prematurely (kswapd_low_wmark_hit_quickly) can indicate +that the number of free pages kswapd maintains for latency reasons is +too small for the allocation bursts occurring in the system. This knob +can then be used to tune kswapd aggressiveness accordingly. + The /exact hack/ I was thinking of was committed about 6 months ago and added "support for ever more" /proc file: commit 795ae7a0de6b834a0cc202aa55c190ef81496665 Author: Johannes Weiner <hannes@cmpxchg.org> Date: Thu Mar 17 14:19:14 2016 -0700 mm: scale kswapd watermarks in proportion to memory What's painfully obvious, though, is that even when I wind it up to it's full threshold (10% memory), it does not prevent direct reclaim from being entered and causing excessive latencies when it blocks. This is despite the fact that simoops is now running with a big free memory reserve (3-3.5GB of free memory on my machine as the page cache now only consumes ~4GB instead of 7-8GB). And, while harder to trigger, kswapd still goes on the "free fucking everything" rampages that trigger page writeback from kswapd and empty both the page cache and the slab caches. The only difference now is that it does this /without triggering the allocstall counter/.... So it's seems that just upping the direct reclaim throttle point isn't a sufficient workaround for the "too much direct reclaim" problem here... Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-11-17 0:47 ` Dave Chinner @ 2016-11-17 1:07 ` Chris Mason 2016-11-17 3:39 ` Dave Chinner 0 siblings, 1 reply; 24+ messages in thread From: Chris Mason @ 2016-11-17 1:07 UTC (permalink / raw) To: Dave Chinner; +Cc: linux-xfs On Thu, Nov 17, 2016 at 11:47:45AM +1100, Dave Chinner wrote: >On Tue, Nov 15, 2016 at 10:03:52PM -0500, Chris Mason wrote: >> Moving forward, I think I can manage to carry the one line patch in >> code that hasn't measurably changed in years. We'll get it tested >> in a variety of workloads and come back with more benchmarks for the >> great slab rework coming soon to a v5.x kernel near you. > >FWIW, I just tested your one-liner against my simoops config here, >and by comparing the behaviour to my patchset that still allows >direct reclaim to block on dirty inodes, it would appear that all >the allocation latency I'm seeing here is from direct reclaim. Meaning that your allocation latencies are constant regardless of if we're waiting in the xfs shrinker? > >So I went looking at the direct reclaim throttle with the intent to >hack it to throttle earlier. It throttles based on watermarks, so >I figured Id just hack them to be larger to trigger direct reclaim >throttling earlier. And then I found this recent addition: > >https://patchwork.kernel.org/patch/8426381/ > >+============================================================= >+ >+watermark_scale_factor: >+ >+This factor controls the aggressiveness of kswapd. It defines the >+amount of memory left in a node/system before kswapd is woken up and >+how much memory needs to be free before kswapd goes back to sleep. >+ >+The unit is in fractions of 10,000. The default value of 10 means the >+distances between watermarks are 0.1% of the available memory in the >+node/system. The maximum value is 1000, or 10% of memory. >+ >+A high rate of threads entering direct reclaim (allocstall) or kswapd >+going to sleep prematurely (kswapd_low_wmark_hit_quickly) can indicate >+that the number of free pages kswapd maintains for latency reasons is >+too small for the allocation bursts occurring in the system. This knob >+can then be used to tune kswapd aggressiveness accordingly. >+ > >The /exact hack/ I was thinking of was committed about 6 months >ago and added "support for ever more" /proc file: Yeah, Johannes spent a bunch of time looking at kswapd in a few places it was causing trouble here. > >commit 795ae7a0de6b834a0cc202aa55c190ef81496665 >Author: Johannes Weiner <hannes@cmpxchg.org> >Date: Thu Mar 17 14:19:14 2016 -0700 > > mm: scale kswapd watermarks in proportion to memory > > >What's painfully obvious, though, is that even when I wind it up to >it's full threshold (10% memory), it does not prevent direct reclaim >from being entered and causing excessive latencies when it blocks. >This is despite the fact that simoops is now running with a big free >memory reserve (3-3.5GB of free memory on my machine as the page >cache now only consumes ~4GB instead of 7-8GB). Huh, I'll try to reproduce that. It might be NUMA imbalance or just that simoop is so bursty that we're blowing past that 3.5GB. > >And, while harder to trigger, kswapd still goes on the "free fucking >everything" rampages that trigger page writeback from kswapd and >empty both the page cache and the slab caches. The only difference >now is that it does this /without triggering the allocstall >counter/.... > >So it's seems that just upping the direct reclaim throttle point >isn't a sufficient workaround for the "too much direct reclaim" >problem here... > >Cheers, > >Dave. >-- >Dave Chinner >david@fromorbit.com ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-11-17 1:07 ` Chris Mason @ 2016-11-17 3:39 ` Dave Chinner 0 siblings, 0 replies; 24+ messages in thread From: Dave Chinner @ 2016-11-17 3:39 UTC (permalink / raw) To: Chris Mason; +Cc: linux-xfs On Wed, Nov 16, 2016 at 08:07:28PM -0500, Chris Mason wrote: > On Thu, Nov 17, 2016 at 11:47:45AM +1100, Dave Chinner wrote: > >On Tue, Nov 15, 2016 at 10:03:52PM -0500, Chris Mason wrote: > >>Moving forward, I think I can manage to carry the one line patch in > >>code that hasn't measurably changed in years. We'll get it tested > >>in a variety of workloads and come back with more benchmarks for the > >>great slab rework coming soon to a v5.x kernel near you. > > > >FWIW, I just tested your one-liner against my simoops config here, > >and by comparing the behaviour to my patchset that still allows > >direct reclaim to block on dirty inodes, it would appear that all > >the allocation latency I'm seeing here is from direct reclaim. > > Meaning that your allocation latencies are constant regardless of if > we're waiting in the xfs shrinker? No, what I mean is that all the big p99 latencies are a result of blocking in direct reclaim, not blocking kswapd. i.e. fully non-blocking kswapd + blocking direct reclaim == big bad p99 latencies vs non-blocking kswapd + direct reclaim == no big latencies. It also /appears/ that the bad FFE kswapd behaviour is closely correlated to the long blocking latencies in direct reclaim, though I haven't been able to confirm this hypothesis yet. > >commit 795ae7a0de6b834a0cc202aa55c190ef81496665 > >Author: Johannes Weiner <hannes@cmpxchg.org> > >Date: Thu Mar 17 14:19:14 2016 -0700 > > > > mm: scale kswapd watermarks in proportion to memory > > > > > >What's painfully obvious, though, is that even when I wind it up to > >it's full threshold (10% memory), it does not prevent direct reclaim > >from being entered and causing excessive latencies when it blocks. > >This is despite the fact that simoops is now running with a big free > >memory reserve (3-3.5GB of free memory on my machine as the page > >cache now only consumes ~4GB instead of 7-8GB). > > Huh, I'll try to reproduce that. It might be NUMA imbalance or just > that simoop is so bursty that we're blowing past that 3.5GB. It's probably blowing through it, but regardless of this there's more serious problems with this approach. I originally turned up the watermarks a few seconds after starting simoops and everythign was fine. However, when I stopped and tried to restart simoops, it *always* fails in a few seconds with either: .... Creating working files done creating working files du thread is running /mnt/scratch du thread is done /mnt/scratch error 11 from pthread_create $ or .... Creating working files done creating working files du thread is running /mnt/scratch du thread is done /mnt/scratch mmap: Cannot allocate memory $ I couldn't start simoops again until I backed out the watermark tuning, and then it started straight away. IOWs, screwing with the watermarks to try to avoid direct reclaim appears to make userspace randomly fail with ENOMEM problems when there are large reserves still available. So AFAICT this doesn't fix the problems that I've been seeing and instead creates a bunch of new ones. Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim 2016-10-14 12:27 [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim Chris Mason 2016-10-15 22:34 ` Dave Chinner @ 2019-06-14 12:58 ` Amir Goldstein 1 sibling, 0 replies; 24+ messages in thread From: Amir Goldstein @ 2019-06-14 12:58 UTC (permalink / raw) To: Chris Mason; +Cc: Dave Chinner, linux-xfs, Brian Foster On Fri, Oct 14, 2016 at 3:36 PM Chris Mason <clm@fb.com> wrote: > > > Hi Dave, > > This is part of a series of patches we're growing to fix a perf > regression on a few straggler tiers that are still on v3.10. In this > case, hadoop had to switch back to v3.10 because v4.x is as much as 15% > slower on recent kernels. > > Between v3.10 and v4.x, kswapd is less effective overall. This leads > more and more procs to get bogged down in direct reclaim Using SYNC_WAIT > in xfs_reclaim_inodes_ag(). > > Since slab shrinking happens very early in direct reclaim, we've seen > systems with 130GB of ram where hundreds of procs are stuck on the xfs > slab shrinker fighting to walk a slab 900MB in size. They'd have better > luck moving on to the page cache instead. > > Also, we're going into direct reclaim much more often than we should > because kswapd is getting stuck on XFS inode locks and writeback. > Dropping the SYNC_WAIT means that kswapd can move on to other things and > let the async worker threads get kicked to work on the inodes. > > We're still working on the series, and this is only compile tested on > current Linus git. I'm working out some better simulations for the > hadoop workload to stuff into Mel's tests. Numbers from prod take > roughly 3 days to stabilize, so I haven't isolated this patch from the rest > of the series. > > Unpatched v4.x our base allocation stall rate goes up to as much as > 200-300/sec, averaging 70/sec. The series I'm finalizing gets that > number down to < 1 /sec. > > Omar Sandoval did some digging and found you added the SYNC_WAIT in > response to a workload I sent ages ago. I tried to make this OOM with > fsmark creating empty files, and it has been soaking in memory > constrained workloads in production for almost two weeks. > > Signed-off-by: Chris Mason <clm@fb.com> > --- > fs/xfs/xfs_icache.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c > index bf2d607..63938fb 100644 > --- a/fs/xfs/xfs_icache.c > +++ b/fs/xfs/xfs_icache.c > @@ -1195,7 +1195,7 @@ xfs_reclaim_inodes_nr( > xfs_reclaim_work_queue(mp); > xfs_ail_push_all(mp->m_ail); > > - return xfs_reclaim_inodes_ag(mp, SYNC_TRYLOCK | SYNC_WAIT, &nr_to_scan); > + return xfs_reclaim_inodes_ag(mp, SYNC_TRYLOCK, &nr_to_scan); > } > > /* > -- Hi Chris, We've being seeing memory allocation stalls on some v4.9.y production systems involving direct reclaim of xfs inodes. I saw a similar issue was reported again here: https://bugzilla.kernel.org/show_bug.cgi?id=192981 I couldn't find any resolution to the reported issue in upstream commits, so I wonder, does Facebook still carry this patch? Or was there a proper fix and I missed it? Thanks, Amir. ^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2019-06-14 12:59 UTC | newest] Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-10-14 12:27 [PATCH RFC] xfs: drop SYNC_WAIT from xfs_reclaim_inodes_ag during slab reclaim Chris Mason 2016-10-15 22:34 ` Dave Chinner 2016-10-17 0:24 ` Chris Mason 2016-10-17 1:52 ` Dave Chinner 2016-10-17 13:30 ` Chris Mason 2016-10-17 22:30 ` Dave Chinner 2016-10-17 23:20 ` Chris Mason 2016-10-18 2:03 ` Dave Chinner 2016-11-14 1:00 ` Chris Mason 2016-11-14 7:27 ` Dave Chinner 2016-11-14 20:56 ` Chris Mason 2016-11-14 23:58 ` Dave Chinner 2016-11-15 3:09 ` Chris Mason 2016-11-15 5:54 ` Dave Chinner 2016-11-15 19:00 ` Chris Mason 2016-11-16 1:30 ` Dave Chinner 2016-11-16 3:03 ` Chris Mason 2016-11-16 23:31 ` Dave Chinner 2016-11-17 0:27 ` Chris Mason 2016-11-17 1:00 ` Dave Chinner 2016-11-17 0:47 ` Dave Chinner 2016-11-17 1:07 ` Chris Mason 2016-11-17 3:39 ` Dave Chinner 2019-06-14 12:58 ` Amir Goldstein
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.