Hello,

Our nightly performance testing found a performance regression when we rebased
our devel tree onto v5.16-rc.  This took me a few days to bisect down, but this
patch

7fd7a9e0caba ("sched/fair: Trigger nohz.next_balance updates when a CPU goes NOHZ-idle")

is the one that introduces the regression.  My performance testing box is a 2
socket, with a model name "Intel(R) Xeon(R) Bronze 3204 CPU @ 1.90GHz", for a
total of 12 cpu's reported in cpuinfo.  It has 128gib of RAM, and these perf
tests are being run against a SSD and spinning rust device, but the regression
is consistent across both configurations.  You can see the historical graph of
the completion latencies for this specific run

http://toxicpanda.com/performance/emptyfiles500k_write_clat_ns_p99.png

Or for something a little more braindead (untar firefox) you can see a increase
in the runtime

http://toxicpanda.com/performance/untarfirefox_elapsed.png

These two tests are single threaded, the regression doesn't appear to affect
multi-threaded tests.  For a simple reproducer you can simply download a tarball
of the firefox sources and untar it onto a clean btrfs file system.  The time
before and after this commit goes up ~1-2 seconds on my machine.  For a less
simple test you can create a clean btrfs file system and run

fio --name emptyfiles500k --create_on_open=1 --nrfiles=31250 --readwrite=write \
	--readwrite=write --ioengine=filecreate --fallocate=none --filesize=4k \
	--openfiles=1 --alloc-size 98304 --allrandrepeat=1 --randseed=12345 \
	--directory <mount point>

And you are looking for the "Write clat ns p99" metric.  You'll see a 5-10%
increase in the latency time.  If you want to run our tests directly it's
relatively easy to setup, you can clone the fsperf repo

https://github.com/josefbacik/fsperf

Then in the fsperf directory edit the local.cfg and add

[main]
directory=/mnt/test

[btrfs]
device=/dev/sdc
iosched=none
mkfs=mkfs.btrfs -f
mount=mount -o noatime

And then run the following on the baseline kernel

./fsperf -p regression -c btrfs -n 10 emptyfiles500k

This will run the test 10 times and save the results to the database.  Then you
can boot into your changed kernel and runn

./fsperf -p regrssion -c btrfs -n 10 -t emptyfiles500k

This will run the test 10 times and take the average and compare it to the
baseline and print out the values, you'll see the increase latency values there.

I can reproduce this at will, if you want to just throw patches at me I'm happy
to run it and let you know what happens.  I'm attaching my .config as well in
case that is needed, but the HZ and PREEMPT settings are

CONFIG_NO_HZ_COMMON=y
CONFIG_NO_HZ_FULL=y
CONFIG_NO_HZ=y
CONFIG_HZ_1000=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
CONFIG_PREEMPT_DYNAMIC=y
CONFIG_PREEMPT_RCU=y
CONFIG_HAVE_PREEMPT_DYNAMIC=y
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_DEBUG_PREEMPT=y

Thanks,

Josef