Re: Performance issue with recently_deleted() /no_journal with huge directories

[Jan Kara <jack@suse.cz>]

Hi!

On Thu 27-08-20 12:09:21, James Scriven (jamscriv) wrote:
> Hi, I'm working on migrating a workload from kernel 2.6 to 4.18 (REHL6 to
> RHEL8).
> 
> The use case is a build farm that has a basic workflow of:
> 
> 1) rm -rf a large directory tree (about 2M files ~ 200GB) to free some space
> 2) download and extract a large tarbar (about 2M files ~ 200GB)
> 3) perform a build in the extracted directory tree
> Repeat...
> 
> We've being using an ext4 filesystem with no journal for maximum
> performance with great success. We're not very concerned about losing
> data, but do want some persistence, which is why we don't just use tmpfs
> for this. We'll keep a number of these large workspaces around as long as
> space permits, and delete the oldest (step 1) just before starting a new
> one (step 2). 
> 
> When migrating to this newer kernel, we are seeing performance
> degradation when we expand the tar, which I suspect is caused by inode
> allocation trying to find an unused inode that has not been used too
> recently. Since we have 2M deleted inodes that *have* been recently
> deleted, every one of the new 2M inodes has to search through all 2M of
> the deleted ones (or something to that approximation - my full
> understanding of the ext4 code is limited).
> 
> The simple testcase below shows the issue. My question is, is this edge
> case already understood? Is there a good way to re-gain this lost
> performance? Adding a "sync + drop_caches", or a sufficiently long sleep,
> between steps 1 and 2 will work around the issue, but is not ideal.

So from the tests below it isn't obvious to me that the recently_deleted()
logic is the culprit of the problem. But it is quite possible. We have
somewhat changed the logic in commit d05466b27b19 "ext4: avoid ENOSPC when
avoiding to reuse recently deleted inodes" so now we just reuse recently
deleted inode if we cannot find any better in the current group. That should
significantly reduce the cost of searching for free inode in your usecase
so you might give that change a try...

								Honza

> # each iteration of the test case the number of recently deleted inodes increases and performance degrades.
> 
> $ uname -a
> Linux sjc-asr-bm-470 4.18.0-147.3.1.el8_1.x86_64 #1 SMP Wed Nov 27 01:11:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> $ sync; echo 3 | sudo tee /proc/sys/vm/drop_caches; for x in {1..10}; do rm -rf dirtree; mkdir dirtree; time mkdir dirtree/{1..50000}; done
> 3
> 
> real    0m1.796s
> user    0m0.041s
> sys     0m1.528s
> 
> real    0m3.280s
> user    0m0.035s
> sys     0m3.235s
> 
> real    0m4.329s
> user    0m0.035s
> sys     0m4.279s
> 
> real    0m6.033s
> user    0m0.032s
> sys     0m5.988s
> 
> real    0m7.303s
> user    0m0.041s
> sys     0m7.246s
> 
> real    0m7.874s
> user    0m0.032s
> sys     0m7.826s
> 
> real    0m9.376s
> user    0m0.036s
> sys     0m9.323s
> 
> real    0m9.979s
> user    0m0.052s
> sys     0m9.910s
> 
> real    0m9.808s
> user    0m0.037s
> sys     0m9.749s
> 
> real    0m9.067s
> user    0m0.038s
> sys     0m9.011s
> 
> 
> 
> 
> $ uname -a
> Linux sjc-asr-bm-100 2.6.32-754.17.1.el6.x86_64 #1 SMP Thu Jun 20 11:47:12 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
> $ sync; echo 3 | sudo tee /proc/sys/vm/drop_caches; for x in {1..10}; do rm -rf dirtree; mkdir dirtree; time mkdir dirtree/{1..50000}; done
> 3
> 
> real    0m0.724s
> user    0m0.031s
> sys     0m0.693s
> 
> real    0m0.762s
> user    0m0.041s
> sys     0m0.721s
> 
> real    0m0.717s
> user    0m0.043s
> sys     0m0.674s
> 
> real    0m0.712s
> user    0m0.037s
> sys     0m0.675s
> 
> real    0m0.749s
> user    0m0.036s
> sys     0m0.712s
> 
> real    0m0.710s
> user    0m0.040s
> sys     0m0.670s
> 
> real    0m0.746s
> user    0m0.038s
> sys     0m0.707s
> 
> real    0m0.715s
> user    0m0.034s
> sys     0m0.680s
> 
> real    0m0.747s
> user    0m0.040s
> sys     0m0.707s
> 
> real    0m0.732s
> user    0m0.042s
> sys     0m0.690s
> 
> 
> 
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR