[PATCH 0/2] Credit2: fix per-socket runqueue setup

* [PATCH 0/2] Credit2: fix per-socket runqueue setup
@ 2014-08-22 17:15 Dario Faggioli
  2014-08-22 17:15 ` [PATCH 1/2] x86: during boot, anticipate identifying the boot cpu Dario Faggioli
                   ` (2 more replies)
  0 siblings, 3 replies; 14+ messages in thread
From: Dario Faggioli @ 2014-08-22 17:15 UTC (permalink / raw)
  To: xen-devel; +Cc: george.dunlap, Andrew Cooper, keir, Jan Beulich

Hi everyone,

Code is already there in the credit2 scheduler, for setting up one runqueue per
each socket found in the system, despite the fact that comments in
sched_credit2.c still report this (well, something like this )to be a TODO
item.

However, such code is not functional. In fact, on the following host:

cpu_topology           :
cpu:    core    socket     node
  0:       0        1        0
  1:       0        1        0
  2:       1        1        0
  3:       1        1        0
  4:       2        1        0
  5:       2        1        0
  6:       8        1        0
  7:       8        1        0
  8:       9        1        0
  9:       9        1        0
 10:      10        1        0
 11:      10        1        0
 12:       0        0        1
 13:       0        0        1
 14:       1        0        1
 15:       1        0        1
 16:       2        0        1
 17:       2        0        1
 18:       8        0        1
 19:       8        0        1
 20:       9        0        1
 21:       9        0        1
 22:      10        0        1
 23:      10        0        1
numa_info              :
node:    memsize    memfree    distances
   0:     18432      17488      10,20
   1:     19200      17932      20,10

What I see during boot and scheduler initialization is as follows (for all the
24 pCPUs):

(XEN) init_pcpu: Strange, cpu 1 already initialized!
(XEN) init_pcpu: Strange, cpu 2 already initialized!
...
(XEN) init_pcpu: Strange, cpu 23 already initialized!

Also, once the system booted, here's what happened:
root@tg03:~# xl dmesg |grep -i runqueue
(XEN) Adding cpu 0 to runqueue 0
(XEN)  First cpu on runqueue, activating
(XEN) Adding cpu 1 to runqueue 0
(XEN) Adding cpu 2 to runqueue 0
...
(XEN) Adding cpu 23 to runqueue 0

I.e., only one runqueue was created, despite the host having 2 sockets, and all
the pCPUs have been assigned to it.

This series fixes that by doing the following three things:
- anticipate the identification of the topology of the boot CPU to happen
  _before_ the scheduler's initialization. This is necessary to make sure that,
  when credit2 will try to figure out on which socket CPU 0 it, it will find the
  correct info (patch 1);
- properly initialize the cpuinfo_x86 data structure, and more specifically the
  phys_proc_id field, that hosts the socket, to something invalid (-1). In fact,
  having that field in all elements set to 0 would induce credit2 to think that
  the pCPU have already been initialized, and that all are on socket 0 (patch 1); 
- use the info made available by early boot CPU identification in credit2
  (patch 2).

After this series is applied, no more "Strange, cpu X already initialized!" can
be found in the logs. Also, here's how the scheduler is being setup:

root@tg03:~# xl dmesg |grep -i runqueue
(XEN) Adding cpu 0 to runqueue 1
(XEN)  First cpu on runqueue, activating
(XEN) Adding cpu 1 to runqueue 1
(XEN) Adding cpu 2 to runqueue 1
(XEN) Adding cpu 3 to runqueue 1
(XEN) Adding cpu 4 to runqueue 1
(XEN) Adding cpu 5 to runqueue 1
(XEN) Adding cpu 6 to runqueue 1
(XEN) Adding cpu 7 to runqueue 1
(XEN) Adding cpu 8 to runqueue 1
(XEN) Adding cpu 9 to runqueue 1
(XEN) Adding cpu 10 to runqueue 1
(XEN) Adding cpu 11 to runqueue 1
(XEN) Adding cpu 12 to runqueue 0
(XEN)  First cpu on runqueue, activating
(XEN) Adding cpu 13 to runqueue 0
(XEN) Adding cpu 14 to runqueue 0
(XEN) Adding cpu 15 to runqueue 0
(XEN) Adding cpu 16 to runqueue 0
(XEN) Adding cpu 17 to runqueue 0
(XEN) Adding cpu 18 to runqueue 0
(XEN) Adding cpu 19 to runqueue 0
(XEN) Adding cpu 20 to runqueue 0
(XEN) Adding cpu 21 to runqueue 0
(XEN) Adding cpu 22 to runqueue 0
(XEN) Adding cpu 23 to runqueue 0

Which makes a lot more sense. :-)

Regards,
Dario

---
Dario Faggioli (2):
      x86: during boot, anticipate identifying the boot cpu
      sched: credit2: use boot CPU info for CPU #0

 xen/arch/x86/setup.c            |    8 ++++++--
 xen/arch/x86/smpboot.c          |    3 ++-
 xen/common/sched_credit2.c      |   12 +++++-------
 xen/include/asm-x86/processor.h |    6 ++++--
 4 files changed, 17 insertions(+), 12 deletions(-)

--
s happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

^ permalink raw reply	[flat|nested] 14+ messages in thread