Re: Xen on ARM IRQ latency and scheduler overhead

From: Julien Grall <julien.grall@arm.com>
To: Dario Faggioli <dario.faggioli@citrix.com>,
	Stefano Stabellini <sstabellini@kernel.org>
Cc: george.dunlap@eu.citrix.com, edgar.iglesias@xilinx.com,
	nd@arm.com, xen-devel@lists.xen.org
Subject: Re: Xen on ARM IRQ latency and scheduler overhead
Date: Fri, 17 Feb 2017 19:34:29 +0000	[thread overview]
Message-ID: <3672df49-6353-5d03-ab0b-1143ffdf0822@arm.com> (raw)
In-Reply-To: <1487329361.6732.76.camel@citrix.com>

Hi,

On 02/17/2017 11:02 AM, Dario Faggioli wrote:
> Just very quickly...
>
> On Thu, 2017-02-16 at 15:07 -0800, Stefano Stabellini wrote:
>> (XEN) Active queues: 1
>> (XEN)   default-weight     = 256
>> (XEN) Runqueue 0:
>> (XEN)   ncpus              = 4
>> (XEN)   cpus               = 0-3
>> (XEN)   max_weight         = 256
>> (XEN)   instload           = 1
>> (XEN)   aveload            = 3208 (~1%)
>> (XEN) l(XEN)    idlers: 00000000,00000000,00000000,0000000a
>> a(XEN)  tickled: 00000000,00000000,00000000,00000000
>> t(XEN)  fully idle cores: 00000000,00000000,00000000,0000000a
>> e(XEN) Domain info:
>> n(XEN)  Domain: 0 w 256 v 4
>> c(XEN)    1: [0.0] flags=2 cpu=0 credit=10500000 [w=256] load=3170 (~1%)
>> (XEN)     2: y[0.1] flags=0 cpu=1  credit=10500000 [w=256]( load=131072 (~50%)
>> (XEN)     3: n[0.2] flags=0 cpu=2s credit=10500000 [w=256]) load=131072 (~50%):
>> (XEN)     4:  [0.3] flags=0 cpu=3m credit=10500000 [w=256]a load=131072 (~50%)x
>>
> Status of vcpus 2, 3 and 4 is a bit weird. I'll think about it.
>
>> =(XEN)  Domain: 1 w 256 v 1
>> 1(XEN)    5: 1[1.0] flags=2 cpu=2 credit=9713074 [w=256] load=56 (~0%)
>> (XEN) Runqueue info:
>> 6(XEN) runqueue 0:
>> 9(XEN) CPUs info:
>> 0(XEN) CPU[00]   runq=0, sibling=00000000,00000000,00000000,00000001, wcore=00000000,00000000,00000000,00000001
>>
> This tells me that nr_cpu_ids is very big (I think it tells it is 128,
> i.e., ARM default), which means cpumask_t-s are huge.
>
> What does `xl info' says. On my (x86) test box, it's like this:
>
>  ...
>  nr_cpus                : 16
>  max_cpu_id             : 63
>  ...
>
> (and I have NR_CPUS=256, i.e., x86 the default).
>
> Cpumasks being bigger also means cpumask operation being slower, and
> this matters quite a bit in Credit2, because we use cpumasks a lot (but
> also in Credit1, because we use cpumasks a little less than in Credit2,
> but still quite a bit).
>
> Isn't there a way, on ARM, to figure out online that you're not going
> to have 128 cpus in the platform?

It is just we never set nr_cpu_ids on ARM :/. There was a patch on the 
ML a while ago [1] but never got applied.

Stefano, I think the patch is still valid. Could you apply it?

It would probably be worth to do the benchmark again with this patch 
applied.

Cheers,

[1] https://patchwork.kernel.org/patch/8177261/

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel