Re: [regression] 5.15 kernel triggering 100x more inode evictions

[Bruno Damasceno Freire <bdamasceno@hotmail.com.br>]

___Updated test results

These results were obtained with the script I've made to research this regression [1].
[1] https://github.com/bdamascen0/s3e


Main results

The regression was reproduced on:
-several different 5.15 kernels versions across several different distros.
-all 5.15 kernels that I have tried on.
-the 5.15.0-rc1 kernel from the opensuse tumbleweed comunity repository.
-the 5.15.12 vanilla kernel from the official opensuse tumbleweed repository.

The regression could not be reproduced on kernels versions other than the 5.15.

The vanilla kernel test was suggested by Thorsten Leemhuis to make sure downstream custom patches aren't causing the symptoms.
The vanilla kernel test result shows the exact same pattern verified on downstream kernels and fully validates the regression.


General test results for the 5.15 kernel series (x86_64)

opensuse tumbleweed ----- kernel 5.15.12 --- vanilla --- (kvm)
... 250 files - zstd:         13327 ms @inode_evictions: 31375
... 250 files - lzo:          13361 ms @inode_evictions: 31375
... 250 files - uncompressed:  1204 ms @inode_evictions: 499
opensuse tumbleweed ----- kernel 5.15.0-rc1-1.g8787773 - (kvm)
... 250 files - zstd:         13875 ms @inode_evictions: 31375
... 250 files - lzo:          15351 ms @inode_evictions: 31375
... 250 files - uncompressed:  1231 ms @inode_evictions: 499
opensuse tumbleweed ----- kernel 5.15.12----------------------
... 250 files - zstd:         12500 ms @inode_evictions: 31375
... 250 files - lzo:          12327 ms @inode_evictions: 31375
... 250 files - uncompressed:  1482 ms @inode_evictions: 499
debian bookworm --------- kernel 5.15.0-3 - (5.15.15) -- (kvm)
... 250 files - zstd:         12343 ms @inode_evictions: 31375
... 250 files - lzo:          14028 ms @inode_evictions: 31375
... 250 files - uncompressed:  1092 ms @inode_evictions: 499
Zenwalk 15.0 Skywalker ---kernel 5.15.19 --------------- (kvm)
... 250 files - zstd:         14374 ms @inode_evictions: -
... 250 files - lzo:          14163 ms @inode_evictions: -
... 250 files - uncompressed:  2173 ms @inode_evictions: -
ubuntu jammy jellyfish -- kernel 5.15.0.23 - (5.15.27) - (kvm) 
... 250 files - zstd:         17521 ms @inode_evictions: 31375
... 250 files - lzo:          17114 ms @inode_evictions: 31375
... 250 files - uncompressed:  1138 ms @inode_evictions: 499


General test results for other kernels (x86_64)

opensuse leap 15.3 ------ kernel 5.3.18-150300.59.54----------
... 250 files - zstd:           668 ms @inode_evictions: 251
... 250 files - lzo:            693 ms @inode_evictions: 251
... 250 files - uncompressed:   661 ms @inode_evictions: 252
opensuse leap 15.4 beta - kernel 5.14.21-150400.11 ----- (kvm)
... 250 files - zstd:           811 ms @inode_evictions: 251
... 250 files - lzo:            912 ms @inode_evictions: 251
... 250 files - uncompressed:   993 ms @inode_evictions: 251
opensuse tumbleweed ----- kernel 5.14.14 --------------- (kvm)
... 250 files - zstd:           888 ms @inode_evictions: 251
... 250 files - lzo:           1063 ms @inode_evictions: 251
... 250 files - uncompressed:   778 ms @inode_evictions: 251
opensuse tumbleweed ----- kernel 5.16.14----------------------
... 250 files - zstd:          1398 ms @inode_evictions: 250
... 250 files - lzo:           1323 ms @inode_evictions: 250
... 250 files - uncompressed:  1365 ms @inode_evictions: 250


Load test results (x86_64):

opensuse leap 15.4 beta - kernel 5.14.21-150400.11 ----- (kvm)
...   50 files - zstd:            261 ms @inode_evictions: 51
...   50 files - lzo:             256 ms @inode_evictions: 51
...   50 files - uncompressed:    317 ms @inode_evictions: 51
...  100 files - zstd:            450 ms @inode_evictions: 101
...  100 files - lzo:             461 ms @inode_evictions: 101
...  100 files - uncompressed:    471 ms @inode_evictions: 101
...  150 files - zstd:            618 ms @inode_evictions: 151
...  150 files - lzo:             624 ms @inode_evictions: 151
...  150 files - uncompressed:    612 ms @inode_evictions: 151
...  200 files - zstd:            822 ms @inode_evictions: 201
...  200 files - lzo:             933 ms @inode_evictions: 201
...  200 files - uncompressed:    747 ms @inode_evictions: 201
...  250 files - zstd:           1128 ms @inode_evictions: 251
...  250 files - lzo:             974 ms @inode_evictions: 251
...  250 files - uncompressed:    936 ms @inode_evictions: 251
... 1000 files - zstd:           3517 ms @inode_evictions: 1001
... 1000 files - lzo:            4373 ms @inode_evictions: 1001
... 1000 files - uncompressed:   3797 ms @inode_evictions: 1001
ubuntu jammy jellyfish -- kernel 5.15.0.23 - (5.15.27) - (kvm) 
...   50 files - zstd:            424 ms @inode_evictions: 1275
...   50 files - lzo:             423 ms @inode_evictions: 1275
...   50 files - uncompressed:    207 ms @inode_evictions: 99
...  100 files - zstd:           1744 ms @inode_evictions: 5050
...  100 files - lzo:            1838 ms @inode_evictions: 5050
...  100 files - uncompressed:    373 ms @inode_evictions: 199
...  150 files - zstd:           4785 ms @inode_evictions: 11325
...  150 files - lzo:            4660 ms @inode_evictions: 11325
...  150 files - uncompressed:    689 ms @inode_evictions: 299
...  200 files - zstd:           9763 ms @inode_evictions: 20100
...  200 files - lzo:           10106 ms @inode_evictions: 20100
...  200 files - uncompressed:    938 ms @inode_evictions: 399
...  250 files - zstd:          17550 ms @inode_evictions: 31375
...  250 files - lzo:           17337 ms @inode_evictions: 31375
...  250 files - uncompressed:   1373 ms @inode_evictions: 499
... 1000 files - zstd:         143614 ms @inode_evictions: 101132
... 1000 files - lzo:          146724 ms @inode_evictions: 100314
... 1000 files - uncompressed:   7735 ms @inode_evictions: 1999


Load test results comparisson for compressed files (x86_64):
ubuntu jammy jellyfish - compared to - opensuse leap 15.4 beta

50   files gives aprox.  1.6 x more time and aprox.  25 x more inode evictions 
100  files gives aprox.  3.8 x more time and aprox.  50 x more inode evictions 
150  files gives aprox.  7.4 x more time and aprox.  75 x more inode evictions 
200  files gives aprox. 10.8 x more time and aprox. 100 x more inode evictions 
250  files gives aprox. 15.5 x more time and aprox. 125 x more inode evictions 
1000 files gives aprox. 33.5 x more time and aprox. 100 x more inode evictions 


Load test results comparisson for uncompressed files (x86_64):
ubuntu jammy jellyfish - compared to - opensuse leap 15.4 beta

50   files gives aprox. 0.6 x more time and aprox. 2 x more inode evictions 
100  files gives aprox. 0.8 x more time and aprox. 2 x more inode evictions 
150  files gives aprox. 1.1 x more time and aprox. 2 x more inode evictions 
200  files gives aprox. 1.2 x more time and aprox. 2 x more inode evictions 
250  files gives aprox. 1.4 x more time and aprox. 2 x more inode evictions 
1000 files gives aprox. 2.0 x more time and aprox. 2 x more inode evictions 


CPU usage results:
The regression causes significant CPU usage by the kernel.

ubuntu jammy jellyfish -- kernel  5.15.0.23 (5.15.27) - (kvm) 
... 1000 files - zstd:         137841 ms
    real    2m17,881s
    user    0m 1,704s
    sys     2m11,937s
... 1000 files - lzo:          135456 ms
    real    2m15,478s
    user    0m 1,805s
    sys     2m 9,758s
... 1000 files - uncompressed:   7496 ms
    real    0m 7,517s
    user    0m 1,386s
    sys     0m 4,899s


Test system specification:
host: AMD FX-8370E 8 cores / 8GB RAM / ssd
guests (kvm): 2 cores / 2G RAM / ssd
test storage medium: RAM disk block device (host and guest)


TIA, Bruno