From mboxrd@z Thu Jan 1 00:00:00 1970 From: Mirela Simonovic Subject: Re: [PATCH 0/7] xen/arm: CPU hotplug fixes Date: Wed, 11 Apr 2018 18:37:52 +0200 Message-ID: References: <20180411131927.27083-1-mirela.simonovic@aggios.com> <61809e61-d4be-9000-2d5c-2e8b7770d525@aggios.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8754947827285500206==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" To: Julien Grall Cc: "Edgar E. Iglesias" , Stefano Stabellini , Davorin Mista , Xen Devel List-Id: xen-devel@lists.xenproject.org --===============8754947827285500206== Content-Type: multipart/alternative; boundary="001a114abe80b0253f056995422a" --001a114abe80b0253f056995422a Content-Type: text/plain; charset="UTF-8" Hi Julien, On Wed, Apr 11, 2018 at 6:02 PM, Julien Grall wrote: > Hi, > > On 11/04/18 16:58, Mirela Simonovic wrote: > >> On 04/11/2018 05:07 PM, Julien Grall wrote: >> >>> On 11/04/18 14:19, Mirela Simonovic wrote: >>> >> Migrating interrupts when turning off a CPU already works. However, when >> a CPU is turned back on there is no interrupt migration back to the >> hotplugged CPU - all interrupts will remain routed to the CPU#0. >> Patch 7/7 fixes this >> > > What do you mean by all interrupts? Interrupts routed to guest will always > follow the vCPU. So are you sure they are going to be migrated when that > vCPU is paused/off? > > Just to make sure we're on the same page - this is about hotplugging physical CPUs. Hotplugging vCPUs using virtual PSCI CPU_OFF interface is already implemented and unrelated to this series. Assuming that system has 2 pCPUs by 'all interrupts' I mean interrupts that were targeted to the pCPU#0 and pCPU#1 prior to doing any hotplug. For example, if a guest is pinned to pCPU#1 an interrupt of a device it owns will be targeted to pCPU#1. When pCPU#1 is turned off that interrupt will be migrated to pCPU#0. pCPU#0 finalizes the suspend and receives wake-up interrupts. However, when CPU#1 is turned back on that interrupt will remain targeted to the CPU#0, which I assumed is wrong. The scenario described here is also how I tested this. > Can you give the path in Xen doing that? > > Sure, here is a backtrace (dumped on the CPU being turned off): 0 0x2603dc arch_move_irqs(): vgic.c, line 309 1 0x22ee58 sched_move_irqs()+20: schedule.c, line 303 2 0x2318e8 cpu_disable_scheduler()+1000: schedule.c, line 586 3 0x2318e8 cpu_disable_scheduler()+1000: schedule.c, line 586 4 0x25aff8 __cpu_disable()+96: smpboot.c, line 386 5 0x201608 take_cpu_down()+52: cpu.c, line 75 6 0x23426c stopmachine_action()+188: stop_machine.c, line 159 7 0x235858 do_tasklet_work()+176: tasklet.c, line 94 8 0x235c80 do_tasklet()+104: tasklet.c, line 126 9 0x24daec idle_loop()+144: domain.c, line 72 10 0x25b1f8 start_secondary()+404: smpboot.c, line 368 > >> >>>> Calls to enable/disable_nonboot_cpus() functions currently don't exist >>>> in >>>> Xen ARM code. This will be added with the suspend to RAM support for >>>> ARM. >>>> >>> >>> What would be the way to test that series? >>> >>> >> I tested/developed the series on Xilinx Zynq UltraScale+ MPSoC (ZCU102 >> board), but how would others test that - I don't know. We will have to ask >> Xilinx for detailed answer to this question. >> > > I am not sure to understand your answer. What I asked, is how did you did > turned off the CPU? Did you hack Xen or do you rely on another series? > > I rely on the suspend to RAM series that will be submitted after this one is done. Not complete suspend to RAM support is needed in order to do the testing. Essentially, I have triggered suspend to RAM from Dom0 and the minimum in Xen is to have the suspend support for secondary CPUs (calls to disable/enable_nonboot_cpus). > If the former, it would be nice to get the code you used. If the latter, > then having a hack patch to test that code would be nice. Ideally, you want > to plug that in the SYSCTL interface for out-of-box testing. > > Ok, I have never used that but I'll try to figure it out. I may come up with additional questions. Thanks, Mirela > Cheers, > > -- > Julien Grall > --001a114abe80b0253f056995422a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Julien,

On Wed, Apr 11, 2018 at 6:02 PM, Julien Grall &= lt;julien.grall@a= rm.com> wrote:
Hi,

On 11/04/18 16:58, Mirela Simonovic wrote:
On 04/11/2018 05:07 PM, Julien Grall wrote:
On 11/04/18 14:19, Mirela Simonovic wrote:
 Migrating interrupts when turning off a CPU already works. However, when a = CPU is turned back on there is no interrupt migration back to the hotplugge= d CPU - all interrupts will remain routed to the CPU#0.
Patch 7/7 fixes this

What do you mean by all interrupts? Interrupts routed to guest will always = follow the vCPU. So are you sure they are going to be migrated when that vC= PU is paused/off?


Just to make sure we're on the sam= e page - this is about hotplugging physical CPUs. Hotplugging vCPUs using v= irtual PSCI CPU_OFF interface is already implemented and unrelated to this = series.

Assuming that system has 2 pCPUs by 'a= ll interrupts' I mean interrupts that were targeted to the pCPU#0 and p= CPU#1 prior to doing any hotplug.

For example, if = a guest is pinned to pCPU#1 an interrupt of a device it owns will be target= ed to pCPU#1.
When pCPU#1 is turned off that interrupt will be mi= grated to pCPU#0. pCPU#0 finalizes the suspend and receives wake-up interru= pts. However, when CPU#1 is turned back on that interrupt will remain targe= ted to the CPU#0, which I assumed is wrong.
The scenario describe= d here is also how I tested this.

=C2=A0
Can you give the path in Xen doing that?


Sure, here is a backtrace (dump= ed on the CPU being turned off):
=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0=C2=A0
=C2=A0 =C2=A0 0=C2=A0 0x2603dc arch_move_irqs(): vg= ic.c, line 309
=C2=A0 =C2=A0 1=C2=A0 0x22ee58 sched_move_irqs()+2= 0: schedule.c, line 303
=C2=A0 =C2=A0 2=C2=A0 0x2318e8 cpu_disabl= e_scheduler()+1000: schedule.c, line 586
=C2=A0 =C2=A0 3=C2=A0 0x= 2318e8 cpu_disable_scheduler()+1000: schedule.c, line 586
=C2=A0 = =C2=A0 4=C2=A0 0x25aff8 __cpu_disable()+96: smpboot.c, line 386
= =C2=A0 =C2=A0 5=C2=A0 0x201608 take_cpu_down()+52: cpu.c, line 75
=C2=A0 =C2=A0 6=C2=A0 0x23426c stopmachine_action()+188: stop_machine.c, l= ine 159
=C2=A0 =C2=A0 7=C2=A0 0x235858 do_tasklet_work()+176: tas= klet.c, line 94
=C2=A0 =C2=A0 8=C2=A0 0x235c80 do_tasklet()+104: = tasklet.c, line 126
=C2=A0 =C2=A0 9=C2=A0 0x24daec idle_loop()+14= 4: domain.c, line 72
=C2=A0 =C2=A010=C2=A0 0x25b1f8 start_seconda= ry()+404: smpboot.c, line 368

=C2=A0


Calls to enable/disable_nonboot_cpus() functions currently don't exist = in
Xen ARM code. This will be added with the suspend to RAM support for ARM.

What would be the way to test that series?


I tested/developed the series on Xilinx Zynq UltraScale+ MPSoC (ZCU102 boar= d), but how would others test that - I don't know. We will have to ask = Xilinx for detailed answer to this question.

 I am not sure to understand your answer. What I asked, is how did you did t= urned off the CPU? Did you hack Xen or do you rely on another series?


I rely on the suspend to RAM series th= at will be submitted after this one is done. Not complete suspend to RAM su= pport is needed in order to do the testing.
Essentially, I have t= riggered suspend to RAM from Dom0 and the minimum in Xen is to have the sus= pend support for secondary CPUs (calls to disable/enable_nonboot_cpus).
=C2=A0
If the former, it would be nice to get the code you used. If the latter, th= en having a hack patch to test that code would be nice. Ideally, you want t= o plug that in the SYSCTL interface for out-of-box testing.


Ok, I have never used that but I'l= l try to figure it out. I may come up with additional questions.
=
Thanks,
Mirela

=C2=A0
Cheers,

--
Julien Grall

--001a114abe80b0253f056995422a-- --===============8754947827285500206== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVucHJvamVjdC5vcmcKaHR0cHM6Ly9saXN0 cy54ZW5wcm9qZWN0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3hlbi1kZXZlbA== --===============8754947827285500206==--