Re: [Xen-users] Changing netback tx interrupts affinity on Dom0

Re: [Xen-users] Changing netback tx interrupts affinity on Dom0

[Jintack Lim]

[-- Attachment #1.1: Type: text/plain, Size: 2121 bytes --]

Hi,

I was trying to set irq affinity by writing a value
to /proc/irq/<irq>/smp_affinity,
but smp_affinity value was not changed at all.
Ian suggested to take this to the devel list.
I'm working on Xen4.5, ARMv8.

Thanks,
Jintack

On Thu, Sep 3, 2015 at 11:34 AM, Ian Campbell <ian.campbell@citrix.com>
wrote:

> On Thu, 2015-09-03 at 10:59 -0400, Jintack Lim wrote:
> > Hi,
> >
> > While I was running Apache server,
> > I found that one of Dom0 vcpu is running 100% to handle irqs,
> > and those irqs are set to be processed only on that specific vcpu.
> >
> > Referring to this document,
> > http://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide
> > I tried to change smp_affinity by writing a value to /proc/irq/<irq
> > -no>/smp_affinity,
> > however the smp_affinity value was not changed.
> >
> > I'm working on Xen 4.5 on ARMv8,
> > and the irq is netback tx interrupt.
> >
> > # cat /proc/irq/106/smp_affinity
> > 1
> >
> > # cat /proc/interrupts
> > ...
> > 106:      53849          0          0          0   xen-dyn-event
> > vif1.1-q0-tx
> > 107:          1          0          0          0   xen-dyn-event
> > vif1.1-q0-rx
> > 108:      61460          0          0          0   xen-dyn-event
> > vif1.1-q1-tx
> > 109:          1          0          0          0   xen-dyn-event
> > vif1.1-q1-rx
> > 110:      67118          0          0          0   xen-dyn-event
> > vif1.1-q2-tx
> > 111:          1          0          0          0   xen-dyn-event
> > vif1.1-q2-rx
> > 112:      58273          0          0          0   xen-dyn-event
> > vif1.1-q3-tx
> > 113:          1          0          0          0   xen-dyn-event
> > vif1.1-q3-rx
> > ...
> >
> > What would be the way to change smp_affinity?
>
> It should be via /proc/irq/<irq>/smp_affinity as you've tried, I asked
> Stefano and he things this was already implemented in 4.5 even. That it is
> not happening would be a bug, I think.
>
> I'd suggest you take this to the devel list as a bug.
>
> > and where is the affinity set initially for netback tx interrupts?
>
> Not, sure, I think it might just be generic IRQ code.
>
> Ian.
>
>

[-- Attachment #1.2: Type: text/html, Size: 3459 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-users] Changing netback tx interrupts affinity on Dom0

[Jintack Lim]

[-- Attachment #1.1: Type: text/plain, Size: 2387 bytes --]

On Thu, Sep 3, 2015 at 12:06 PM, Jintack Lim <jintack@cs.columbia.edu>
wrote:

> Hi,
>
> I was trying to set irq affinity by writing a value
> to /proc/irq/<irq>/smp_affinity,
> but smp_affinity value was not changed at all.
> Ian suggested to take this to the devel list.
> I'm working on Xen4.5, ARMv8.
>

I checked that setting smp_affinity works on Xen4.5, x86.
So, probably this is an ARM specific issue.


>
> Thanks,
> Jintack
>
> On Thu, Sep 3, 2015 at 11:34 AM, Ian Campbell <ian.campbell@citrix.com>
> wrote:
>
>> On Thu, 2015-09-03 at 10:59 -0400, Jintack Lim wrote:
>> > Hi,
>> >
>> > While I was running Apache server,
>> > I found that one of Dom0 vcpu is running 100% to handle irqs,
>> > and those irqs are set to be processed only on that specific vcpu.
>> >
>> > Referring to this document,
>> > http://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide
>> > I tried to change smp_affinity by writing a value to /proc/irq/<irq
>> > -no>/smp_affinity,
>> > however the smp_affinity value was not changed.
>> >
>> > I'm working on Xen 4.5 on ARMv8,
>> > and the irq is netback tx interrupt.
>> >
>> > # cat /proc/irq/106/smp_affinity
>> > 1
>> >
>> > # cat /proc/interrupts
>> > ...
>> > 106:      53849          0          0          0   xen-dyn-event
>> > vif1.1-q0-tx
>> > 107:          1          0          0          0   xen-dyn-event
>> > vif1.1-q0-rx
>> > 108:      61460          0          0          0   xen-dyn-event
>> > vif1.1-q1-tx
>> > 109:          1          0          0          0   xen-dyn-event
>> > vif1.1-q1-rx
>> > 110:      67118          0          0          0   xen-dyn-event
>> > vif1.1-q2-tx
>> > 111:          1          0          0          0   xen-dyn-event
>> > vif1.1-q2-rx
>> > 112:      58273          0          0          0   xen-dyn-event
>> > vif1.1-q3-tx
>> > 113:          1          0          0          0   xen-dyn-event
>> > vif1.1-q3-rx
>> > ...
>> >
>> > What would be the way to change smp_affinity?
>>
>> It should be via /proc/irq/<irq>/smp_affinity as you've tried, I asked
>> Stefano and he things this was already implemented in 4.5 even. That it is
>> not happening would be a bug, I think.
>>
>> I'd suggest you take this to the devel list as a bug.
>>
>> > and where is the affinity set initially for netback tx interrupts?
>>
>> Not, sure, I think it might just be generic IRQ code.
>>
>> Ian.
>>
>>
>

[-- Attachment #1.2: Type: text/html, Size: 4300 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-users] Changing netback tx interrupts affinity on Dom0

[Julien Grall]

On 03/09/15 17:06, Jintack Lim wrote:
> Hi,

Hi Jintack,

> On Thu, Sep 3, 2015 at 11:34 AM, Ian Campbell <ian.campbell@citrix.com
> <mailto:ian.campbell@citrix.com>> wrote:
> 
>     On Thu, 2015-09-03 at 10:59 -0400, Jintack Lim wrote:
>     > Hi,
>     >
>     > While I was running Apache server,
>     > I found that one of Dom0 vcpu is running 100% to handle irqs,
>     > and those irqs are set to be processed only on that specific vcpu.
>     >
>     > Referring to this document,
>     > http://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide
>     > I tried to change smp_affinity by writing a value to /proc/irq/<irq
>     > -no>/smp_affinity,
>     > however the smp_affinity value was not changed.
>     >
>     > I'm working on Xen 4.5 on ARMv8,
>     > and the irq is netback tx interrupt.
>     >
>     > # cat /proc/irq/106/smp_affinity
>     > 1
>     >
>     > # cat /proc/interrupts
>     > ...
>     > 106:      53849          0          0          0   xen-dyn-event
>     > vif1.1-q0-tx
>     > 107:          1          0          0          0   xen-dyn-event
>     > vif1.1-q0-rx
>     > 108:      61460          0          0          0   xen-dyn-event
>     > vif1.1-q1-tx
>     > 109:          1          0          0          0   xen-dyn-event
>     > vif1.1-q1-rx
>     > 110:      67118          0          0          0   xen-dyn-event
>     > vif1.1-q2-tx
>     > 111:          1          0          0          0   xen-dyn-event
>     > vif1.1-q2-rx
>     > 112:      58273          0          0          0   xen-dyn-event
>     > vif1.1-q3-tx
>     > 113:          1          0          0          0   xen-dyn-event
>     > vif1.1-q3-rx
>     > ...
>     >
>     > What would be the way to change smp_affinity?

All those interrupts are in-fine event channels. This is a bug in Linux
which I sent a fix a month ago and it's queued in xentip for Linux 4.3 [1]

The event channel rebinding was not working because we don't have vector
callback on ARM and therefore the driver was only allowing event channel
to be routed on VCPU0.

commit 4a5b69464e51f4a8dd432e8c2a1468630df1a53c
Author: Julien Grall <julien.grall@citrix.com>
Date:   Tue Jul 28 10:10:42 2015 +0100

xen/events: Support event channel rebind on ARM
Currently, the event channel rebind code is gated with the presence of
the vector callback.

The virtual interrupt controller on ARM has the concept of per-CPU
interrupt (PPI) which allow us to support per-VCPU event
channel.Therefore there is no need of vector callback for ARM.

Xen is already using a free PPI to notify the guest VCPU of an event.
Furthermore, the xen code initialization in Linux (see
arch/arm/xen/enlighten.c) is requesting correctly a per-CPU IRQ.

Introduce new helper xen_support_evtchn_rebind to allow architecture
decide whether rebind an event is support or not. It will always return
true on ARM and keep the same behavior on x86.

This is also allow us to drop the usage of xen_have_vector_callback
entirely in the ARM code.

Signed-off-by: Julien Grall <julien.grall@citrix.com>
Signed-off-by: David Vrabel <david.vrabel@citrix.com>

Regards,

[1] https://lkml.org/lkml/2015/7/28/205

-- 
Julien Grall

Re: [Xen-users] Changing netback tx interrupts affinity on Dom0

[Jintack Lim]

[-- Attachment #1.1: Type: text/plain, Size: 3949 bytes --]

Thanks Julien.

On Thu, Sep 3, 2015 at 2:21 PM, Julien Grall <julien.grall@citrix.com>
wrote:

> On 03/09/15 17:06, Jintack Lim wrote:
> > Hi,
>
> Hi Jintack,
>
> > On Thu, Sep 3, 2015 at 11:34 AM, Ian Campbell <ian.campbell@citrix.com
> > <mailto:ian.campbell@citrix.com>> wrote:
> >
> >     On Thu, 2015-09-03 at 10:59 -0400, Jintack Lim wrote:
> >     > Hi,
> >     >
> >     > While I was running Apache server,
> >     > I found that one of Dom0 vcpu is running 100% to handle irqs,
> >     > and those irqs are set to be processed only on that specific vcpu.
> >     >
> >     > Referring to this document,
> >     > http://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide
> >     > I tried to change smp_affinity by writing a value to /proc/irq/<irq
> >     > -no>/smp_affinity,
> >     > however the smp_affinity value was not changed.
> >     >
> >     > I'm working on Xen 4.5 on ARMv8,
> >     > and the irq is netback tx interrupt.
> >     >
> >     > # cat /proc/irq/106/smp_affinity
> >     > 1
> >     >
> >     > # cat /proc/interrupts
> >     > ...
> >     > 106:      53849          0          0          0   xen-dyn-event
> >     > vif1.1-q0-tx
> >     > 107:          1          0          0          0   xen-dyn-event
> >     > vif1.1-q0-rx
> >     > 108:      61460          0          0          0   xen-dyn-event
> >     > vif1.1-q1-tx
> >     > 109:          1          0          0          0   xen-dyn-event
> >     > vif1.1-q1-rx
> >     > 110:      67118          0          0          0   xen-dyn-event
> >     > vif1.1-q2-tx
> >     > 111:          1          0          0          0   xen-dyn-event
> >     > vif1.1-q2-rx
> >     > 112:      58273          0          0          0   xen-dyn-event
> >     > vif1.1-q3-tx
> >     > 113:          1          0          0          0   xen-dyn-event
> >     > vif1.1-q3-rx
> >     > ...
> >     >
> >     > What would be the way to change smp_affinity?
>
> All those interrupts are in-fine event channels. This is a bug in Linux
> which I sent a fix a month ago and it's queued in xentip for Linux 4.3 [1]
>
> The event channel rebinding was not working because we don't have vector
> callback on ARM and therefore the driver was only allowing event channel
> to be routed on VCPU0.
>
>
After applying this patch, I was able to set smp_affinity.

In addition, the default value of smp_affinity for netback tx interrupts on
Dom0 becomes "f" (I have four vcpus on Dom0).
What I meant by 'netback tx interrupts' is vif1.1-q<num>-tx
This is slightly different from x86, because netback tx interrupts are
pinned to separate vcpus on x86.
I'm not sure this question is appropriate for devel-list,
but why the policy to set smp_affinity for netback tx interrupts are
different on ARM and x86?


> commit 4a5b69464e51f4a8dd432e8c2a1468630df1a53c
> Author: Julien Grall <julien.grall@citrix.com>
> Date:   Tue Jul 28 10:10:42 2015 +0100
>
> xen/events: Support event channel rebind on ARM
> Currently, the event channel rebind code is gated with the presence of
> the vector callback.
>
> The virtual interrupt controller on ARM has the concept of per-CPU
> interrupt (PPI) which allow us to support per-VCPU event
> channel.Therefore there is no need of vector callback for ARM.
>
> Xen is already using a free PPI to notify the guest VCPU of an event.
> Furthermore, the xen code initialization in Linux (see
> arch/arm/xen/enlighten.c) is requesting correctly a per-CPU IRQ.
>
> Introduce new helper xen_support_evtchn_rebind to allow architecture
> decide whether rebind an event is support or not. It will always return
> true on ARM and keep the same behavior on x86.
>
> This is also allow us to drop the usage of xen_have_vector_callback
> entirely in the ARM code.
>
> Signed-off-by: Julien Grall <julien.grall@citrix.com>
> Signed-off-by: David Vrabel <david.vrabel@citrix.com>
>
> Regards,
>
> [1] https://lkml.org/lkml/2015/7/28/205
>
> --
> Julien Grall
>
>

[-- Attachment #1.2: Type: text/html, Size: 7067 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-users] Changing netback tx interrupts affinity on Dom0

[Julien Grall]

Hi,

On 03/09/15 22:12, Jintack Lim wrote:
> On Thu, Sep 3, 2015 at 2:21 PM, Julien Grall <julien.grall@citrix.com
> <mailto:julien.grall@citrix.com>> wrote:
> 
>     On 03/09/15 17:06, Jintack Lim wrote:
>     > Hi,
> 
>     Hi Jintack,
> 
>     > On Thu, Sep 3, 2015 at 11:34 AM, Ian Campbell <ian.campbell@citrix.com <mailto:ian.campbell@citrix.com>
>     > <mailto:ian.campbell@citrix.com <mailto:ian.campbell@citrix.com>>>
>     wrote:
>     >
>     >     On Thu, 2015-09-03 at 10:59 -0400, Jintack Lim wrote:
>     >     > Hi,
>     >     >
>     >     > While I was running Apache server,
>     >     > I found that one of Dom0 vcpu is running 100% to handle irqs,
>     >     > and those irqs are set to be processed only on that specific
>     vcpu.
>     >     >
>     >     > Referring to this document,
>     >     >
>     http://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide
>     >     > I tried to change smp_affinity by writing a value to
>     /proc/irq/<irq
>     >     > -no>/smp_affinity,
>     >     > however the smp_affinity value was not changed.
>     >     >
>     >     > I'm working on Xen 4.5 on ARMv8,
>     >     > and the irq is netback tx interrupt.
>     >     >
>     >     > # cat /proc/irq/106/smp_affinity
>     >     > 1
>     >     >
>     >     > # cat /proc/interrupts
>     >     > ...
>     >     > 106:      53849          0          0          0   xen-dyn-event
>     >     > vif1.1-q0-tx
>     >     > 107:          1          0          0          0   xen-dyn-event
>     >     > vif1.1-q0-rx
>     >     > 108:      61460          0          0          0   xen-dyn-event
>     >     > vif1.1-q1-tx
>     >     > 109:          1          0          0          0   xen-dyn-event
>     >     > vif1.1-q1-rx
>     >     > 110:      67118          0          0          0   xen-dyn-event
>     >     > vif1.1-q2-tx
>     >     > 111:          1          0          0          0   xen-dyn-event
>     >     > vif1.1-q2-rx
>     >     > 112:      58273          0          0          0   xen-dyn-event
>     >     > vif1.1-q3-tx
>     >     > 113:          1          0          0          0   xen-dyn-event
>     >     > vif1.1-q3-rx
>     >     > ...
>     >     >
>     >     > What would be the way to change smp_affinity?
> 
>     All those interrupts are in-fine event channels. This is a bug in Linux
>     which I sent a fix a month ago and it's queued in xentip for Linux
>     4.3 [1]
> 
>     The event channel rebinding was not working because we don't have vector
>     callback on ARM and therefore the driver was only allowing event channel
>     to be routed on VCPU0.
> 
> 
> After applying this patch, I was able to set smp_affinity.
> 
> In addition, the default value of smp_affinity for netback tx interrupts
> on Dom0 becomes "f" (I have four vcpus on Dom0).
> What I meant by 'netback tx interrupts' is vif1.1-q<num>-tx
> This is slightly different from x86, because netback tx interrupts are
> pinned to separate vcpus on x86.
> I'm not sure this question is appropriate for devel-list,
> but why the policy to set smp_affinity for netback tx interrupts are
> different on ARM and x86?

I'm not sure why. By default on both ARM and x86 the IRQ code is setting
the affinity to all the vCPU.

Note that you are likely using PV for x86 DOM0 which is different than
HVM so it may be possible that we differ for setting the affinity.

I've CCed David who was working on the event channel code. It may have
more insights.

Regards,

-- 
Julien Grall