I have an LVS/DR cluster of 10 machines that receive similar traffic
via a round-robin strategy. These machines run Debian Lenny with
2.6.26, and consistently have a 15-minute load average between 4-12
depending on the time of day.

Upgrading any one of these machines to a newer kernel compiled with
NO_HZ=y causes the reported load average to drop significantly. Here,
fe3 and fe12 are running 3.2.4:

        fe3:    0.48    0.53    0.55
        fe4:    6.73    5.59    5.11
        fe5:    5.93    5.29    5.60
        fe6:    6.20    5.79    6.08
        fe7:    8.32    5.65    5.05
        fe8:    6.34    5.85    5.93
        fe9:    5.80    5.46    5.53
       fe10:    5.49    4.91    5.03
       fe11:    6.60    6.11    6.10
       fe12:    0.39    0.54    0.46

The newly reported load average is much lower than the other machines
performing equivalent work, and does not match cpu usage numbers
reported by vmstat.

Originally, I attempted to upgrade to 2.6.32 (used by lenny-backports
and squeeze). In 2.6.32, load averages are misreported even when using
NO_HZ=n. This bug was fixed in 2.6.34-rc4 (74f5187ac8: sched: Cure
load average vs NO_HZ woes).

The NO_HZ=y case was supposed to be fixed in 2.6.37-rc5 (0f004f5a69:
sched: Cure more NO_HZ load average woes), with the commit stating
"behaviour between CONFIG_NO_HZ=[yn] should be equivalent". In my
environment, however, kernels after this patch was introduced still
misreport load averages when compiled with NO_HZ=y.

Here's a list of the kernel versions I've tried (more details in
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=620297):

  Correct Load Average
    2.6.26.25
    2.6.32.55-620297patch (CONFIG_NO_HZ=n)

  Incorrect Load Average
    2.6.32-bpo.5-amd64
    2.6.32.55
    2.6.32.55-620297patch
    2.6.32.55-620297patch (nohz=off)
    2.6.37-rc5-cure-more
    2.6.39.4
    3.2.2
    3.2.4

Also worth nothing is that fe12 is much newer than fe3, which should
help rule out hardware as a cause of this bug.

    fe3: Dell PowerEdge 2950 w/ Xeon(R) CPU           L5420  @ 2.50GHz
  fe12: Dell PowerEdge R710 w/ Xeon(R) CPU           E5645  @ 2.40GHz

I've attached dmesg output from fe12 booting up on 3.2.4. I am happy
to provide any other information that would be useful, and would
appreciate any advice on patches to try or ways to narrow the bug down
further.

  Aman