From mboxrd@z Thu Jan  1 00:00:00 1970
From: Julien Grall <julien.grall@citrix.com>
Subject: Re: [Xen-users] Changing netback tx interrupts affinity
 on Dom0
Date: Fri, 4 Sep 2015 14:28:18 +0100
Message-ID: <55E99C72.3030503@citrix.com>
References: <CAHyh4xj3+si5+TzSGJeN3Wp0VE8MnPV30BFc_b7c+uuW+o+2mw@mail.gmail.com>
	<1441294496.26292.372.camel@citrix.com>
	<CAHyh4xggXVtw06P_ESgir-20dSgJeXwU4tgD0NQNWirz9V3kSQ@mail.gmail.com>
	<55E88FB7.6020004@citrix.com>
	<CAHyh4xjjiSpx29=NOZ+CTzCr996kdmWPqQquSPNb8n-+M7bzpg@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Return-path: <xen-devel-bounces@lists.xen.org>
In-Reply-To: <CAHyh4xjjiSpx29=NOZ+CTzCr996kdmWPqQquSPNb8n-+M7bzpg@mail.gmail.com>
List-Unsubscribe: <http://lists.xen.org/cgi-bin/mailman/options/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xen.org>
List-Help: <mailto:xen-devel-request@lists.xen.org?subject=help>
List-Subscribe: <http://lists.xen.org/cgi-bin/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xen.org?subject=subscribe>
Sender: xen-devel-bounces@lists.xen.org
Errors-To: xen-devel-bounces@lists.xen.org
To: Jintack Lim <jintack@cs.columbia.edu>
Cc: David Vrabel <david.vrabel@citrix.com>, Ian Campbell <ian.campbell@citrix.com>, xen-devel <xen-devel@lists.xen.org>
List-Id: xen-devel@lists.xenproject.org

Hi,

On 03/09/15 22:12, Jintack Lim wrote:
> On Thu, Sep 3, 2015 at 2:21 PM, Julien Grall <julien.grall@citrix.com
> <mailto:julien.grall@citrix.com>> wrote:
> 
>     On 03/09/15 17:06, Jintack Lim wrote:
>     > Hi,
> 
>     Hi Jintack,
> 
>     > On Thu, Sep 3, 2015 at 11:34 AM, Ian Campbell <ian.campbell@citrix.com <mailto:ian.campbell@citrix.com>
>     > <mailto:ian.campbell@citrix.com <mailto:ian.campbell@citrix.com>>>
>     wrote:
>     >
>     >     On Thu, 2015-09-03 at 10:59 -0400, Jintack Lim wrote:
>     >     > Hi,
>     >     >
>     >     > While I was running Apache server,
>     >     > I found that one of Dom0 vcpu is running 100% to handle irqs,
>     >     > and those irqs are set to be processed only on that specific
>     vcpu.
>     >     >
>     >     > Referring to this document,
>     >     >
>     http://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide
>     >     > I tried to change smp_affinity by writing a value to
>     /proc/irq/<irq
>     >     > -no>/smp_affinity,
>     >     > however the smp_affinity value was not changed.
>     >     >
>     >     > I'm working on Xen 4.5 on ARMv8,
>     >     > and the irq is netback tx interrupt.
>     >     >
>     >     > # cat /proc/irq/106/smp_affinity
>     >     > 1
>     >     >
>     >     > # cat /proc/interrupts
>     >     > ...
>     >     > 106:      53849          0          0          0   xen-dyn-event
>     >     > vif1.1-q0-tx
>     >     > 107:          1          0          0          0   xen-dyn-event
>     >     > vif1.1-q0-rx
>     >     > 108:      61460          0          0          0   xen-dyn-event
>     >     > vif1.1-q1-tx
>     >     > 109:          1          0          0          0   xen-dyn-event
>     >     > vif1.1-q1-rx
>     >     > 110:      67118          0          0          0   xen-dyn-event
>     >     > vif1.1-q2-tx
>     >     > 111:          1          0          0          0   xen-dyn-event
>     >     > vif1.1-q2-rx
>     >     > 112:      58273          0          0          0   xen-dyn-event
>     >     > vif1.1-q3-tx
>     >     > 113:          1          0          0          0   xen-dyn-event
>     >     > vif1.1-q3-rx
>     >     > ...
>     >     >
>     >     > What would be the way to change smp_affinity?
> 
>     All those interrupts are in-fine event channels. This is a bug in Linux
>     which I sent a fix a month ago and it's queued in xentip for Linux
>     4.3 [1]
> 
>     The event channel rebinding was not working because we don't have vector
>     callback on ARM and therefore the driver was only allowing event channel
>     to be routed on VCPU0.
> 
> 
> After applying this patch, I was able to set smp_affinity.
> 
> In addition, the default value of smp_affinity for netback tx interrupts
> on Dom0 becomes "f" (I have four vcpus on Dom0).
> What I meant by 'netback tx interrupts' is vif1.1-q<num>-tx
> This is slightly different from x86, because netback tx interrupts are
> pinned to separate vcpus on x86.
> I'm not sure this question is appropriate for devel-list,
> but why the policy to set smp_affinity for netback tx interrupts are
> different on ARM and x86?

I'm not sure why. By default on both ARM and x86 the IRQ code is setting
the affinity to all the vCPU.

Note that you are likely using PV for x86 DOM0 which is different than
HVM so it may be possible that we differ for setting the affinity.

I've CCed David who was working on the event channel code. It may have
more insights.

Regards,

-- 
Julien Grall