The problem is illustrated by the attached graphs.  These graphs on the
odd numbered pages show the frequency of each core measures at every clock
tick.  At each measurement there is a small bar representing 4ms of the
color associated with the frequency.  The percentages shown are thus not
entirely accurate, because the frequency could change within those 4ms and
we would not observe that.

The first graph, 5.9schedutil_yeti, is the normal behavior of schedutil
running.  The application mostly uses the second highest turbo mode, which
is the appropriate one given that there are around 5 active cores most of
the time.  I traced power:cpu_frequency, which is the event that occurs
when the OS requests a change of frequency.  This happens around 5400
times.

The second graph, 5.15-schedutil_yeti, is the latest version of Linus's
tree.  The cores are almost always at the lowest frequency.  There are no
occurrences of the power:cpu_frequency event.

The third graph, 5.9schedutil_after_yeti, it what happens when I reboot
into 5.9 after having changed to passive mode in 5.15.  The number of
power:cpu_frequency drops to around 1100.  The proper turbo mode is
actually used sometimes, but much less than in the first graph.  More than
half of the time, an active core is at the lowest frequency.

This application (avrora from the DaCapo benchmarks) is continually
stopping and starting, both for very short intervals.  This may discourage
the hardware from raising the frequency of its own volition.  I also tried
a simple spin loop (for(;;);) with the 5.15 rc version, and it does go to
the highest frequency as one would expect.  But there are again no
power:cpu_frequency events.

julia