vif processing is unbalance

vif processing is unbalance

[xu cong]

[-- Attachment #1.1: Type: text/plain, Size: 542 bytes --]

Hi All,

I found the vif processing are always done on the first vcpu in dom0. I
give dom0 2 vcpus pinned to two dedicated cores respectively. The affinity
of each domU's vif-rx/tx is 3 (/proc/irq/**/smp_affinity is 3). However,
all vif processing are still only done on the vcpu0. This can be observed
by reading top output (%si utilization) and /proc/interrupts. My test bed
is Xen4.2.2, linux 3.12 is used as both dom0 kernel and domU kernel.

Is there a method to balance the vif processing among all vcpus in dom0?
Thanks

Regards,
Cong

[-- Attachment #1.2: Type: text/html, Size: 648 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: vif processing is unbalance

[Wei Liu]

On Mon, Apr 21, 2014 at 02:24:48PM -0400, xu cong wrote:
> Hi All,
> 
> I found the vif processing are always done on the first vcpu in dom0. I
> give dom0 2 vcpus pinned to two dedicated cores respectively. The affinity
> of each domU's vif-rx/tx is 3 (/proc/irq/**/smp_affinity is 3). However,
> all vif processing are still only done on the vcpu0. This can be observed
> by reading top output (%si utilization) and /proc/interrupts. My test bed
> is Xen4.2.2, linux 3.12 is used as both dom0 kernel and domU kernel.
> 
> Is there a method to balance the vif processing among all vcpus in dom0?
> Thanks
> 

You probably need to either manually bind IRQ to a specific CPU or use
irqbalance.

> Regards,
> Cong

Re: vif processing is unbalance

[xu cong]

[-- Attachment #1.1: Type: text/plain, Size: 1009 bytes --]

irqbalance does not work for vif IRQ. Manually bind IRQ works, but it is
not a elegant method. Would you mind telling me which code trigger this IRQ
to vif-tx/rx? Maybe I can try tuning it.  Thanks.


2014-04-21 14:29 GMT-04:00 Wei Liu <wei.liu2@citrix.com>:

> On Mon, Apr 21, 2014 at 02:24:48PM -0400, xu cong wrote:
> > Hi All,
> >
> > I found the vif processing are always done on the first vcpu in dom0. I
> > give dom0 2 vcpus pinned to two dedicated cores respectively. The
> affinity
> > of each domU's vif-rx/tx is 3 (/proc/irq/**/smp_affinity is 3). However,
> > all vif processing are still only done on the vcpu0. This can be observed
> > by reading top output (%si utilization) and /proc/interrupts. My test bed
> > is Xen4.2.2, linux 3.12 is used as both dom0 kernel and domU kernel.
> >
> > Is there a method to balance the vif processing among all vcpus in dom0?
> > Thanks
> >
>
> You probably need to either manually bind IRQ to a specific CPU or use
> irqbalance.
>
> > Regards,
> > Cong
>

[-- Attachment #1.2: Type: text/html, Size: 1464 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: vif processing is unbalance

[Wei Liu]

On Mon, Apr 21, 2014 at 02:45:38PM -0400, xu cong wrote:
> irqbalance does not work for vif IRQ. Manually bind IRQ works, but it is
> not a elegant method. Would you mind telling me which code trigger this IRQ
> to vif-tx/rx? Maybe I can try tuning it.  Thanks.
> 

You probably need to have newer version of irqbalance. I fixed it long
time ago. Not sure if it still works though -- I have not used it for a
long time.

Wei.

> 
> 2014-04-21 14:29 GMT-04:00 Wei Liu <wei.liu2@citrix.com>:
> 
> > On Mon, Apr 21, 2014 at 02:24:48PM -0400, xu cong wrote:
> > > Hi All,
> > >
> > > I found the vif processing are always done on the first vcpu in dom0. I
> > > give dom0 2 vcpus pinned to two dedicated cores respectively. The
> > affinity
> > > of each domU's vif-rx/tx is 3 (/proc/irq/**/smp_affinity is 3). However,
> > > all vif processing are still only done on the vcpu0. This can be observed
> > > by reading top output (%si utilization) and /proc/interrupts. My test bed
> > > is Xen4.2.2, linux 3.12 is used as both dom0 kernel and domU kernel.
> > >
> > > Is there a method to balance the vif processing among all vcpus in dom0?
> > > Thanks
> > >
> >
> > You probably need to either manually bind IRQ to a specific CPU or use
> > irqbalance.
> >
> > > Regards,
> > > Cong
> >

Re: vif processing is unbalance

[xu cong]

[-- Attachment #1.1: Type: text/plain, Size: 1533 bytes --]

Thanks. Would you mind sharing your customized irqbalance? I can generate a
new version based on your code.


2014-04-21 14:49 GMT-04:00 Wei Liu <wei.liu2@citrix.com>:

> On Mon, Apr 21, 2014 at 02:45:38PM -0400, xu cong wrote:
> > irqbalance does not work for vif IRQ. Manually bind IRQ works, but it is
> > not a elegant method. Would you mind telling me which code trigger this
> IRQ
> > to vif-tx/rx? Maybe I can try tuning it.  Thanks.
> >
>
> You probably need to have newer version of irqbalance. I fixed it long
> time ago. Not sure if it still works though -- I have not used it for a
> long time.
>
> Wei.
>
> >
> > 2014-04-21 14:29 GMT-04:00 Wei Liu <wei.liu2@citrix.com>:
> >
> > > On Mon, Apr 21, 2014 at 02:24:48PM -0400, xu cong wrote:
> > > > Hi All,
> > > >
> > > > I found the vif processing are always done on the first vcpu in
> dom0. I
> > > > give dom0 2 vcpus pinned to two dedicated cores respectively. The
> > > affinity
> > > > of each domU's vif-rx/tx is 3 (/proc/irq/**/smp_affinity is 3).
> However,
> > > > all vif processing are still only done on the vcpu0. This can be
> observed
> > > > by reading top output (%si utilization) and /proc/interrupts. My
> test bed
> > > > is Xen4.2.2, linux 3.12 is used as both dom0 kernel and domU kernel.
> > > >
> > > > Is there a method to balance the vif processing among all vcpus in
> dom0?
> > > > Thanks
> > > >
> > >
> > > You probably need to either manually bind IRQ to a specific CPU or use
> > > irqbalance.
> > >
> > > > Regards,
> > > > Cong
> > >
>

[-- Attachment #1.2: Type: text/html, Size: 2300 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: vif processing is unbalance

[Wei Liu]

On Mon, Apr 21, 2014 at 03:01:20PM -0400, xu cong wrote:
> Thanks. Would you mind sharing your customized irqbalance? I can generate a
> new version based on your code.
> 

It's not customized. The fix is in upstream irqbalance. We always work
with upstream code.

But irqbalance site on Google code is down somehow. I have no idea what
happened...

Wei.

Re: vif processing is unbalance

[xu cong]

[-- Attachment #1.1: Type: text/plain, Size: 553 bytes --]

Got it. Is it possible to send the vif irq to a specified vcpu via
modifying the source code of netback? Thanks.


2014-04-21 16:36 GMT-04:00 Wei Liu <wei.liu2@citrix.com>:

> On Mon, Apr 21, 2014 at 03:01:20PM -0400, xu cong wrote:
> > Thanks. Would you mind sharing your customized irqbalance? I can
> generate a
> > new version based on your code.
> >
>
> It's not customized. The fix is in upstream irqbalance. We always work
> with upstream code.
>
> But irqbalance site on Google code is down somehow. I have no idea what
> happened...
>
> Wei.
>

[-- Attachment #1.2: Type: text/html, Size: 977 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: vif processing is unbalance

[Wei Liu]

On Tue, Apr 22, 2014 at 12:14:48AM -0400, xu cong wrote:
> Got it. Is it possible to send the vif irq to a specified vcpu via
> modifying the source code of netback? Thanks.
> 

I think it is possible, but I would not recommend doing that. It would
be better you use userspace program, so that you can have more flexible
strategy.

Wei.

> 
> 2014-04-21 16:36 GMT-04:00 Wei Liu <wei.liu2@citrix.com>:
> 
> > On Mon, Apr 21, 2014 at 03:01:20PM -0400, xu cong wrote:
> > > Thanks. Would you mind sharing your customized irqbalance? I can
> > generate a
> > > new version based on your code.
> > >
> >
> > It's not customized. The fix is in upstream irqbalance. We always work
> > with upstream code.
> >
> > But irqbalance site on Google code is down somehow. I have no idea what
> > happened...
> >
> > Wei.
> >

Re: vif processing is unbalance

[xu cong]

[-- Attachment #1.1: Type: text/plain, Size: 1402 bytes --]

Thanks for your suggestion. I am doing research in this area. I hope to
send the vif irq to the vcpu running the corresponding vif kernel thread,
just like the linux NIC RPS/RFS. Dynamically changing the IRQ affinity in
userspace can not provide such a fine granularity IRQ scheduling, because
vif kernel threads are floating among vcpus. If you can give me some hints
that which functions or codes involving IRQ issuing in netback, I will be
very appreciated. Thanks very much.


2014-04-22 4:34 GMT-04:00 Wei Liu <wei.liu2@citrix.com>:

> On Tue, Apr 22, 2014 at 12:14:48AM -0400, xu cong wrote:
> > Got it. Is it possible to send the vif irq to a specified vcpu via
> > modifying the source code of netback? Thanks.
> >
>
> I think it is possible, but I would not recommend doing that. It would
> be better you use userspace program, so that you can have more flexible
> strategy.
>
> Wei.
>
> >
> > 2014-04-21 16:36 GMT-04:00 Wei Liu <wei.liu2@citrix.com>:
> >
> > > On Mon, Apr 21, 2014 at 03:01:20PM -0400, xu cong wrote:
> > > > Thanks. Would you mind sharing your customized irqbalance? I can
> > > generate a
> > > > new version based on your code.
> > > >
> > >
> > > It's not customized. The fix is in upstream irqbalance. We always work
> > > with upstream code.
> > >
> > > But irqbalance site on Google code is down somehow. I have no idea what
> > > happened...
> > >
> > > Wei.
> > >
>

[-- Attachment #1.2: Type: text/html, Size: 2080 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: vif processing is unbalance

[Ian Campbell]

On Tue, 2014-04-22 at 10:14 -0400, xu cong wrote:
> Thanks for your suggestion. I am doing research in this area. I hope
> to send the vif irq to the vcpu running the corresponding vif kernel
> thread, just like the linux NIC RPS/RFS. Dynamically changing the IRQ
> affinity in userspace can not provide such a fine granularity IRQ
> scheduling, because vif kernel threads are floating among vcpus. If
> you can give me some hints that which functions or codes involving IRQ
> issuing in netback,

As requested in [0] Please use the tools you have available, such as
search engines and grep to try and answer these questions for yourself
before asking on the list.

[0] http://wiki.xen.org/wiki/Asking_Xen_Devel_Questions

Please also do not top post.

Ian.

Re: vif processing is unbalance

[Wei Liu]

On Tue, Apr 22, 2014 at 10:14:58AM -0400, xu cong wrote:
> Thanks for your suggestion. I am doing research in this area. I hope to
> send the vif irq to the vcpu running the corresponding vif kernel thread,
> just like the linux NIC RPS/RFS. Dynamically changing the IRQ affinity in
> userspace can not provide such a fine granularity IRQ scheduling, because
> vif kernel threads are floating among vcpus. If you can give me some hints
> that which functions or codes involving IRQ issuing in netback, I will be
> very appreciated. Thanks very much.
> 
> 

drivers/net/xen-netback/{interface,xen_netback}.c