/proc/stat information incorrect

* /proc/stat information incorrect
@ 2012-04-16  8:04 Bas van der Oest
  2012-04-16 12:36 ` Martin Schwidefsky
  0 siblings, 1 reply; 5+ messages in thread
From: Bas van der Oest @ 2012-04-16  8:04 UTC (permalink / raw)
  To: linux-kernel

Hi,

I am trying to figure out what my benchmark system is spending CPU
time on by looking at the /proc/stat information provided by the
kernel.
I manually map my benchmark application (in this case an
Ethernet-benchmark) to CPU4-7 and the interrupts are assigned to CPU5.
I am running kernel version 3.2.8 on a 8-core nehalem architecture CPU
(2* Intel E5540).
The interrupts are generated by a 10Gb Ethernet adapter (Intel AT2).
I benchmark for a ten second period and take the /proc/stat
information before and after this period.

The /proc/stat information looks like this:
before:
cpu     10384   0       136128  14417335        15115   6       21414
0       0       0
cpu0    371     0       1136    1808846         14606   3       398 0
     0       0
cpu1    238     0       451     1827606         126     0       50 0
    0       0
cpu2    89      0       345     1828021         39      0       15 0
    0       0
cpu3    314     0       524     1827629         26      0       10 0
    0       0
cpu4    268     0       3008    1804530         106     2       1827 0
      0       0
cpu5    2320    0       33947   1767454         99      0       18941
0       0       0
cpu6    3429    0       48610   1776287         70      0       76 0
    0       0
cpu7    3352    0       48105   1776957         41      0       95 0
    0       0
after:
cpu     10389   0       136562  14425011        15115   6       21467
0       0       0
cpu0    371     0       1136    1809913         14606   3       398 0
     0       0
cpu1    238     0       452     1828676         126     0       50 0
    0       0
cpu2    89      0       345     1829092         39      0       15 0
    0       0
cpu3    315     0       524     1828699         26      0       10 0
    0       0
cpu4    269     0       3102    1805504         106     2       1827 0
      0       0
cpu5    2322    0       34039   1767989         99      0       18993
0       0       0
cpu6    3429    0       48692   1777274         70      0       76 0
    0       0
cpu7    3353    0       48270   1777862         41      0       95 0
    0       0

After dropping some zero columns and taking the difference between the
statistics the reformatted result is:
       user    nice    system  idle    iowait  irq     softirq sum
cpu     5       0       434     7676    0       0       53      8168
cpu0    0       0       0       1067    0       0       0       1067
cpu1    0       0       1       1070    0       0       0       1071
cpu2    0       0       0       1071    0       0       0       1071
cpu3    1       0       0       1070    0       0       0       1071
cpu4    1       0       94      974     0       0       0       1069
cpu5    2       0       92      535     0       0       52      681
cpu6    0       0       82      987     0       0       0       1069
cpu7    1       0       165     905     0       0       0       1071

I added a sum column which totals the time spent in the different
modes. The above table now shows how long each CPU was in what mode
for how long.
Now I am wondering how it is possible that CPU5 has spent much less
time than all the other CPUs. I expected that all CPUs spent around
the same time (10s). This time includes idle time so this is not
related to the difference in active/idle CPUs.

I know for a fact that this effect is related to which CPU is handling
my IRQs; this effect happens to all CPUs if I map the interrupts to
that particular CPU.
I looked up the scheduler's statistics handling in the kernel source
but was not able to find any cause for the above mentioned effect.

Can anyone reproduce this behaviour?
Does anyone know where/what might be the cause of this?

Regards,

Bas

^ permalink raw reply	[flat|nested] 5+ messages in thread