All of lore.kernel.org
 help / color / mirror / Atom feed
* VCPU hotplug on KVM/ARM
@ 2018-02-27  9:31 bthakur
  2018-02-27 10:47 ` Christoffer Dall
  0 siblings, 1 reply; 32+ messages in thread
From: bthakur @ 2018-02-27  9:31 UTC (permalink / raw)
  To: kvmarm


[-- Attachment #1.1: Type: text/plain, Size: 684 bytes --]

Hi,

 

I hope it is the right forum to post my query.

 

I am currently looking at the possibility of adding a new VCPU to a running
guest VM in KVM/ARM. I see that currently, it is not allowed to add a new
VCPU to a guest VM, if it is already initialized. The first check in
kvm_arch_vcpu_create() returns failure if it is already initialized.

 

There was some work done in QEMU to add support for VCPU hotplug:
https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html

 

But I am looking at the KVM side for enabling adding a new VCPU. If you can
point me to any relevant work/resources, which I can refer to then it will
help me.

 

Thanks.

 

Regards,

Bhupinder


[-- Attachment #1.2: Type: text/html, Size: 2826 bytes --]

[-- Attachment #2: Type: text/plain, Size: 151 bytes --]

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: VCPU hotplug on KVM/ARM
  2018-02-27  9:31 VCPU hotplug on KVM/ARM bthakur
@ 2018-02-27 10:47 ` Christoffer Dall
  2018-02-27 12:04   ` bthakur
  0 siblings, 1 reply; 32+ messages in thread
From: Christoffer Dall @ 2018-02-27 10:47 UTC (permalink / raw)
  To: bthakur; +Cc: kvmarm

Hi Bhupinder,

On Tue, Feb 27, 2018 at 03:01:17PM +0530, bthakur@codeaurora.org wrote:
> I hope it is the right forum to post my query.
> 
>  
> 
> I am currently looking at the possibility of adding a new VCPU to a running
> guest VM in KVM/ARM. I see that currently, it is not allowed to add a new
> VCPU to a guest VM, if it is already initialized. The first check in
> kvm_arch_vcpu_create() returns failure if it is already initialized.
> 

This would require a major rework of a lot of logic surrounding the GIC
and other parts of KVM initialization.

>  
> 
> There was some work done in QEMU to add support for VCPU hotplug:
> https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html
> 
>  
> 
> But I am looking at the KVM side for enabling adding a new VCPU. If you can
> point me to any relevant work/resources, which I can refer to then it will
> help me.
> 

I don't have any specific pointers, but I was always told that the way
we were going to do CPU hotplug would be to instantiate a large number
of VCPUs, and hotplug would be equivalent to turning on a VCPU which was
previously powered off.

Is this not still a feasible solution?

How does VCPU hotplug work on x86?

Thanks,
-Christoffer

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: VCPU hotplug on KVM/ARM
  2018-02-27 10:47 ` Christoffer Dall
@ 2018-02-27 12:04   ` bthakur
  2018-02-27 12:46     ` Christoffer Dall
  0 siblings, 1 reply; 32+ messages in thread
From: bthakur @ 2018-02-27 12:04 UTC (permalink / raw)
  To: Christoffer Dall; +Cc: Christoffer Dall, kvmarm

Hi Christoffer,

Thanks for your reply.

On 2018-02-27 16:17, Christoffer Dall wrote:
> Hi Bhupinder,
> 
> On Tue, Feb 27, 2018 at 03:01:17PM +0530, bthakur@codeaurora.org wrote:
>> I hope it is the right forum to post my query.
>> 
>> 
>> 
>> I am currently looking at the possibility of adding a new VCPU to a 
>> running
>> guest VM in KVM/ARM. I see that currently, it is not allowed to add a 
>> new
>> VCPU to a guest VM, if it is already initialized. The first check in
>> kvm_arch_vcpu_create() returns failure if it is already initialized.
>> 
> 
> This would require a major rework of a lot of logic surrounding the GIC
> and other parts of KVM initialization.
> 
>> 
>> 
>> There was some work done in QEMU to add support for VCPU hotplug:
>> https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html
>> 
>> 
>> 
>> But I am looking at the KVM side for enabling adding a new VCPU. If 
>> you can
>> point me to any relevant work/resources, which I can refer to then it 
>> will
>> help me.
>> 
> 
> I don't have any specific pointers, but I was always told that the way
> we were going to do CPU hotplug would be to instantiate a large number
> of VCPUs, and hotplug would be equivalent to turning on a VCPU which 
> was
> previously powered off.
> 
> Is this not still a feasible solution?
It should be a feasible solution provided the guest VM is not able to 
control the onlining/offlining of VCPUs. It should be controlled by the 
Host.

> 
> How does VCPU hotplug work on x86?
On x86, you can add a vcpu through libvirt setvcpu command and it shows 
up in the guest VM as a new CPU if you do lscpu.

> 
> Thanks,
> -Christoffer

Regards,
Bhupinder

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: VCPU hotplug on KVM/ARM
  2018-02-27 12:04   ` bthakur
@ 2018-02-27 12:46     ` Christoffer Dall
  2018-02-27 13:21         ` Andrew Jones
  0 siblings, 1 reply; 32+ messages in thread
From: Christoffer Dall @ 2018-02-27 12:46 UTC (permalink / raw)
  To: bthakur; +Cc: Christoffer Dall, Christoffer Dall, kvmarm

On Tue, Feb 27, 2018 at 05:34:28PM +0530, bthakur@codeaurora.org wrote:
> Hi Christoffer,
> 
> Thanks for your reply.
> 
> On 2018-02-27 16:17, Christoffer Dall wrote:
> >Hi Bhupinder,
> >
> >On Tue, Feb 27, 2018 at 03:01:17PM +0530, bthakur@codeaurora.org wrote:
> >>I hope it is the right forum to post my query.
> >>
> >>
> >>
> >>I am currently looking at the possibility of adding a new VCPU to a
> >>running
> >>guest VM in KVM/ARM. I see that currently, it is not allowed to add a
> >>new
> >>VCPU to a guest VM, if it is already initialized. The first check in
> >>kvm_arch_vcpu_create() returns failure if it is already initialized.
> >>
> >
> >This would require a major rework of a lot of logic surrounding the GIC
> >and other parts of KVM initialization.
> >
> >>
> >>
> >>There was some work done in QEMU to add support for VCPU hotplug:
> >>https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html
> >>
> >>
> >>
> >>But I am looking at the KVM side for enabling adding a new VCPU. If you
> >>can
> >>point me to any relevant work/resources, which I can refer to then it
> >>will
> >>help me.
> >>
> >
> >I don't have any specific pointers, but I was always told that the way
> >we were going to do CPU hotplug would be to instantiate a large number
> >of VCPUs, and hotplug would be equivalent to turning on a VCPU which was
> >previously powered off.
> >
> >Is this not still a feasible solution?
> It should be a feasible solution provided the guest VM is not able to
> control the onlining/offlining of VCPUs. It should be controlled by the
> Host.
> 

KVM could simply refuse to turn on some of the CPUs unless given
permission from host userspace.

> >
> >How does VCPU hotplug work on x86?
> On x86, you can add a vcpu through libvirt setvcpu command and it shows up
> in the guest VM as a new CPU if you do lscpu.
> 

Sure, but what is the mechanism, does x86 qemu actually call
KVM_CREATE_VCPU, or is this also a question of turning on already
created vcpus ?

Thanks,
-Christoffer

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
  2018-02-27 12:46     ` Christoffer Dall
@ 2018-02-27 13:21         ` Andrew Jones
  0 siblings, 0 replies; 32+ messages in thread
From: Andrew Jones @ 2018-02-27 13:21 UTC (permalink / raw)
  To: Christoffer Dall
  Cc: bthakur, Christoffer Dall, Christoffer Dall, kvmarm, imammedo,
	qemu-devel, qemu-arm

On Tue, Feb 27, 2018 at 01:46:04PM +0100, Christoffer Dall wrote:
> On Tue, Feb 27, 2018 at 05:34:28PM +0530, bthakur@codeaurora.org wrote:
> > Hi Christoffer,
> > 
> > Thanks for your reply.
> > 
> > On 2018-02-27 16:17, Christoffer Dall wrote:
> > >Hi Bhupinder,
> > >
> > >On Tue, Feb 27, 2018 at 03:01:17PM +0530, bthakur@codeaurora.org wrote:
> > >>I hope it is the right forum to post my query.
> > >>
> > >>
> > >>
> > >>I am currently looking at the possibility of adding a new VCPU to a
> > >>running
> > >>guest VM in KVM/ARM. I see that currently, it is not allowed to add a
> > >>new
> > >>VCPU to a guest VM, if it is already initialized. The first check in
> > >>kvm_arch_vcpu_create() returns failure if it is already initialized.
> > >>
> > >
> > >This would require a major rework of a lot of logic surrounding the GIC
> > >and other parts of KVM initialization.
> > >
> > >>
> > >>
> > >>There was some work done in QEMU to add support for VCPU hotplug:
> > >>https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html
> > >>
> > >>
> > >>
> > >>But I am looking at the KVM side for enabling adding a new VCPU. If you
> > >>can
> > >>point me to any relevant work/resources, which I can refer to then it
> > >>will
> > >>help me.
> > >>
> > >
> > >I don't have any specific pointers, but I was always told that the way
> > >we were going to do CPU hotplug would be to instantiate a large number
> > >of VCPUs, and hotplug would be equivalent to turning on a VCPU which was
> > >previously powered off.
> > >
> > >Is this not still a feasible solution?
> > It should be a feasible solution provided the guest VM is not able to
> > control the onlining/offlining of VCPUs. It should be controlled by the
> > Host.
> > 
> 
> KVM could simply refuse to turn on some of the CPUs unless given
> permission from host userspace.
> 
> > >
> > >How does VCPU hotplug work on x86?
> > On x86, you can add a vcpu through libvirt setvcpu command and it shows up
> > in the guest VM as a new CPU if you do lscpu.
> > 
> 
> Sure, but what is the mechanism, does x86 qemu actually call
> KVM_CREATE_VCPU, or is this also a question of turning on already
> created vcpus ?
>

CC'ing Igor and qemu-devel

drew

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: VCPU hotplug on KVM/ARM
@ 2018-02-27 13:21         ` Andrew Jones
  0 siblings, 0 replies; 32+ messages in thread
From: Andrew Jones @ 2018-02-27 13:21 UTC (permalink / raw)
  To: Christoffer Dall
  Cc: Christoffer Dall, qemu-devel, Christoffer Dall, qemu-arm,
	imammedo, kvmarm

On Tue, Feb 27, 2018 at 01:46:04PM +0100, Christoffer Dall wrote:
> On Tue, Feb 27, 2018 at 05:34:28PM +0530, bthakur@codeaurora.org wrote:
> > Hi Christoffer,
> > 
> > Thanks for your reply.
> > 
> > On 2018-02-27 16:17, Christoffer Dall wrote:
> > >Hi Bhupinder,
> > >
> > >On Tue, Feb 27, 2018 at 03:01:17PM +0530, bthakur@codeaurora.org wrote:
> > >>I hope it is the right forum to post my query.
> > >>
> > >>
> > >>
> > >>I am currently looking at the possibility of adding a new VCPU to a
> > >>running
> > >>guest VM in KVM/ARM. I see that currently, it is not allowed to add a
> > >>new
> > >>VCPU to a guest VM, if it is already initialized. The first check in
> > >>kvm_arch_vcpu_create() returns failure if it is already initialized.
> > >>
> > >
> > >This would require a major rework of a lot of logic surrounding the GIC
> > >and other parts of KVM initialization.
> > >
> > >>
> > >>
> > >>There was some work done in QEMU to add support for VCPU hotplug:
> > >>https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html
> > >>
> > >>
> > >>
> > >>But I am looking at the KVM side for enabling adding a new VCPU. If you
> > >>can
> > >>point me to any relevant work/resources, which I can refer to then it
> > >>will
> > >>help me.
> > >>
> > >
> > >I don't have any specific pointers, but I was always told that the way
> > >we were going to do CPU hotplug would be to instantiate a large number
> > >of VCPUs, and hotplug would be equivalent to turning on a VCPU which was
> > >previously powered off.
> > >
> > >Is this not still a feasible solution?
> > It should be a feasible solution provided the guest VM is not able to
> > control the onlining/offlining of VCPUs. It should be controlled by the
> > Host.
> > 
> 
> KVM could simply refuse to turn on some of the CPUs unless given
> permission from host userspace.
> 
> > >
> > >How does VCPU hotplug work on x86?
> > On x86, you can add a vcpu through libvirt setvcpu command and it shows up
> > in the guest VM as a new CPU if you do lscpu.
> > 
> 
> Sure, but what is the mechanism, does x86 qemu actually call
> KVM_CREATE_VCPU, or is this also a question of turning on already
> created vcpus ?
>

CC'ing Igor and qemu-devel

drew

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
  2018-02-27 13:21         ` Andrew Jones
@ 2018-03-01  9:50           ` Igor Mammedov
  -1 siblings, 0 replies; 32+ messages in thread
From: Igor Mammedov @ 2018-03-01  9:50 UTC (permalink / raw)
  To: Andrew Jones
  Cc: Christoffer Dall, bthakur, Christoffer Dall, qemu-devel,
	Christoffer Dall, qemu-arm, kvmarm

On Tue, 27 Feb 2018 14:21:31 +0100
Andrew Jones <drjones@redhat.com> wrote:

> On Tue, Feb 27, 2018 at 01:46:04PM +0100, Christoffer Dall wrote:
> > On Tue, Feb 27, 2018 at 05:34:28PM +0530, bthakur@codeaurora.org wrote:  
> > > Hi Christoffer,
> > > 
> > > Thanks for your reply.
> > > 
> > > On 2018-02-27 16:17, Christoffer Dall wrote:  
> > > >Hi Bhupinder,
> > > >
> > > >On Tue, Feb 27, 2018 at 03:01:17PM +0530, bthakur@codeaurora.org wrote:  
> > > >>I hope it is the right forum to post my query.
> > > >>
> > > >>
> > > >>
> > > >>I am currently looking at the possibility of adding a new VCPU to a
> > > >>running
> > > >>guest VM in KVM/ARM. I see that currently, it is not allowed to add a
> > > >>new
> > > >>VCPU to a guest VM, if it is already initialized. The first check in
> > > >>kvm_arch_vcpu_create() returns failure if it is already initialized.
> > > >>  
> > > >
> > > >This would require a major rework of a lot of logic surrounding the GIC
> > > >and other parts of KVM initialization.
> > > >  
> > > >>
> > > >>
> > > >>There was some work done in QEMU to add support for VCPU hotplug:
> > > >>https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html
> > > >>
> > > >>
> > > >>
> > > >>But I am looking at the KVM side for enabling adding a new VCPU. If you
> > > >>can
> > > >>point me to any relevant work/resources, which I can refer to then it
> > > >>will
> > > >>help me.
> > > >>  
> > > >
> > > >I don't have any specific pointers, but I was always told that the way
> > > >we were going to do CPU hotplug would be to instantiate a large number
> > > >of VCPUs, and hotplug would be equivalent to turning on a VCPU which was
> > > >previously powered off.
> > > >
> > > >Is this not still a feasible solution?  
> > > It should be a feasible solution provided the guest VM is not able to
> > > control the onlining/offlining of VCPUs. It should be controlled by the
> > > Host.
> > >   
> > 
> > KVM could simply refuse to turn on some of the CPUs unless given
> > permission from host userspace.
> >   
> > > >
> > > >How does VCPU hotplug work on x86?  
> > > On x86, you can add a vcpu through libvirt setvcpu command and it shows up
> > > in the guest VM as a new CPU if you do lscpu.
> > >   
> > 
> > Sure, but what is the mechanism, does x86 qemu actually call
> > KVM_CREATE_VCPU, or is this also a question of turning on already
> > created vcpus ?
In QEMU on x86 (and I think ppc, s390 as well), we create vCPUs on demand.

It would be nice if ARM would be able to do that too,
so that it could take advantage of the same code.

 
> CC'ing Igor and qemu-devel
> 
> drew
> 

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
@ 2018-03-01  9:50           ` Igor Mammedov
  0 siblings, 0 replies; 32+ messages in thread
From: Igor Mammedov @ 2018-03-01  9:50 UTC (permalink / raw)
  To: Andrew Jones
  Cc: Christoffer Dall, qemu-devel, Christoffer Dall, qemu-arm, kvmarm

On Tue, 27 Feb 2018 14:21:31 +0100
Andrew Jones <drjones@redhat.com> wrote:

> On Tue, Feb 27, 2018 at 01:46:04PM +0100, Christoffer Dall wrote:
> > On Tue, Feb 27, 2018 at 05:34:28PM +0530, bthakur@codeaurora.org wrote:  
> > > Hi Christoffer,
> > > 
> > > Thanks for your reply.
> > > 
> > > On 2018-02-27 16:17, Christoffer Dall wrote:  
> > > >Hi Bhupinder,
> > > >
> > > >On Tue, Feb 27, 2018 at 03:01:17PM +0530, bthakur@codeaurora.org wrote:  
> > > >>I hope it is the right forum to post my query.
> > > >>
> > > >>
> > > >>
> > > >>I am currently looking at the possibility of adding a new VCPU to a
> > > >>running
> > > >>guest VM in KVM/ARM. I see that currently, it is not allowed to add a
> > > >>new
> > > >>VCPU to a guest VM, if it is already initialized. The first check in
> > > >>kvm_arch_vcpu_create() returns failure if it is already initialized.
> > > >>  
> > > >
> > > >This would require a major rework of a lot of logic surrounding the GIC
> > > >and other parts of KVM initialization.
> > > >  
> > > >>
> > > >>
> > > >>There was some work done in QEMU to add support for VCPU hotplug:
> > > >>https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html
> > > >>
> > > >>
> > > >>
> > > >>But I am looking at the KVM side for enabling adding a new VCPU. If you
> > > >>can
> > > >>point me to any relevant work/resources, which I can refer to then it
> > > >>will
> > > >>help me.
> > > >>  
> > > >
> > > >I don't have any specific pointers, but I was always told that the way
> > > >we were going to do CPU hotplug would be to instantiate a large number
> > > >of VCPUs, and hotplug would be equivalent to turning on a VCPU which was
> > > >previously powered off.
> > > >
> > > >Is this not still a feasible solution?  
> > > It should be a feasible solution provided the guest VM is not able to
> > > control the onlining/offlining of VCPUs. It should be controlled by the
> > > Host.
> > >   
> > 
> > KVM could simply refuse to turn on some of the CPUs unless given
> > permission from host userspace.
> >   
> > > >
> > > >How does VCPU hotplug work on x86?  
> > > On x86, you can add a vcpu through libvirt setvcpu command and it shows up
> > > in the guest VM as a new CPU if you do lscpu.
> > >   
> > 
> > Sure, but what is the mechanism, does x86 qemu actually call
> > KVM_CREATE_VCPU, or is this also a question of turning on already
> > created vcpus ?
In QEMU on x86 (and I think ppc, s390 as well), we create vCPUs on demand.

It would be nice if ARM would be able to do that too,
so that it could take advantage of the same code.

 
> CC'ing Igor and qemu-devel
> 
> drew
> 

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] [Qemu-arm]  VCPU hotplug on KVM/ARM
  2018-03-01  9:50           ` Igor Mammedov
@ 2018-03-01 10:05             ` Peter Maydell
  -1 siblings, 0 replies; 32+ messages in thread
From: Peter Maydell @ 2018-03-01 10:05 UTC (permalink / raw)
  To: Igor Mammedov
  Cc: Andrew Jones, bthakur, Christoffer Dall, Christoffer Dall,
	QEMU Developers, Christoffer Dall, qemu-arm, kvmarm

On 1 March 2018 at 09:50, Igor Mammedov <imammedo@redhat.com> wrote:
> In QEMU on x86 (and I think ppc, s390 as well), we create vCPUs on demand.
>
> It would be nice if ARM would be able to do that too,
> so that it could take advantage of the same code.

It's not clear to me how that would work, given that for
instance the interrupt controller wants to know up-front
how many CPUs it has to deal with.

thanks
-- PMM

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-arm] [Qemu-devel] VCPU hotplug on KVM/ARM
@ 2018-03-01 10:05             ` Peter Maydell
  0 siblings, 0 replies; 32+ messages in thread
From: Peter Maydell @ 2018-03-01 10:05 UTC (permalink / raw)
  To: Igor Mammedov
  Cc: Christoffer Dall, QEMU Developers, Christoffer Dall, qemu-arm, kvmarm

On 1 March 2018 at 09:50, Igor Mammedov <imammedo@redhat.com> wrote:
> In QEMU on x86 (and I think ppc, s390 as well), we create vCPUs on demand.
>
> It would be nice if ARM would be able to do that too,
> so that it could take advantage of the same code.

It's not clear to me how that would work, given that for
instance the interrupt controller wants to know up-front
how many CPUs it has to deal with.

thanks
-- PMM

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] [Qemu-arm] VCPU hotplug on KVM/ARM
  2018-03-01 10:05             ` [Qemu-arm] [Qemu-devel] " Peter Maydell
@ 2018-03-01 13:32               ` David Hildenbrand
  -1 siblings, 0 replies; 32+ messages in thread
From: David Hildenbrand @ 2018-03-01 13:32 UTC (permalink / raw)
  To: Peter Maydell, Igor Mammedov
  Cc: bthakur, Christoffer Dall, Andrew Jones, Christoffer Dall,
	QEMU Developers, Christoffer Dall, qemu-arm, kvmarm

On 01.03.2018 11:05, Peter Maydell wrote:
> On 1 March 2018 at 09:50, Igor Mammedov <imammedo@redhat.com> wrote:
>> In QEMU on x86 (and I think ppc, s390 as well), we create vCPUs on demand.>>
>> It would be nice if ARM would be able to do that too,
>> so that it could take advantage of the same code.
> 
> It's not clear to me how that would work, given that for
> instance the interrupt controller wants to know up-front
> how many CPUs it has to deal with.
>

So how is cpu hotplug handled in HW? Or doesn't it even exist there?

(we have max_cpus for the interrupt controller, but not sure if that is
what we want)

> thanks
> -- PMM
> 
-- 

Thanks,

David / dhildenb

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] [Qemu-arm] VCPU hotplug on KVM/ARM
@ 2018-03-01 13:32               ` David Hildenbrand
  0 siblings, 0 replies; 32+ messages in thread
From: David Hildenbrand @ 2018-03-01 13:32 UTC (permalink / raw)
  To: Peter Maydell, Igor Mammedov
  Cc: Christoffer Dall, QEMU Developers, Christoffer Dall, qemu-arm, kvmarm

On 01.03.2018 11:05, Peter Maydell wrote:
> On 1 March 2018 at 09:50, Igor Mammedov <imammedo@redhat.com> wrote:
>> In QEMU on x86 (and I think ppc, s390 as well), we create vCPUs on demand.>>
>> It would be nice if ARM would be able to do that too,
>> so that it could take advantage of the same code.
> 
> It's not clear to me how that would work, given that for
> instance the interrupt controller wants to know up-front
> how many CPUs it has to deal with.
>

So how is cpu hotplug handled in HW? Or doesn't it even exist there?

(we have max_cpus for the interrupt controller, but not sure if that is
what we want)

> thanks
> -- PMM
> 
-- 

Thanks,

David / dhildenb

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] [Qemu-arm] VCPU hotplug on KVM/ARM
  2018-03-01 13:32               ` David Hildenbrand
@ 2018-03-07 12:47                 ` Marc Zyngier
  -1 siblings, 0 replies; 32+ messages in thread
From: Marc Zyngier @ 2018-03-07 12:47 UTC (permalink / raw)
  To: David Hildenbrand, Peter Maydell, Igor Mammedov
  Cc: Christoffer Dall, QEMU Developers, Christoffer Dall, qemu-arm, kvmarm

On 01/03/18 13:32, David Hildenbrand wrote:
> On 01.03.2018 11:05, Peter Maydell wrote:
>> On 1 March 2018 at 09:50, Igor Mammedov <imammedo@redhat.com> wrote:
>>> In QEMU on x86 (and I think ppc, s390 as well), we create vCPUs on demand.>>
>>> It would be nice if ARM would be able to do that too,
>>> so that it could take advantage of the same code.
>>
>> It's not clear to me how that would work, given that for
>> instance the interrupt controller wants to know up-front
>> how many CPUs it has to deal with.
>>
> 
> So how is cpu hotplug handled in HW? Or doesn't it even exist there?

I don't know of any physical system offering that facility.

> (we have max_cpus for the interrupt controller, but not sure if that is
> what we want)
We'd need something along those lines. Each CPU has a notional point to
point link to the interrupt controller (to the redistributor, to be
precise), and this entity must pre-exist.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] [Qemu-arm] VCPU hotplug on KVM/ARM
@ 2018-03-07 12:47                 ` Marc Zyngier
  0 siblings, 0 replies; 32+ messages in thread
From: Marc Zyngier @ 2018-03-07 12:47 UTC (permalink / raw)
  To: David Hildenbrand, Peter Maydell, Igor Mammedov
  Cc: Christoffer Dall, Christoffer Dall, QEMU Developers, qemu-arm, kvmarm

On 01/03/18 13:32, David Hildenbrand wrote:
> On 01.03.2018 11:05, Peter Maydell wrote:
>> On 1 March 2018 at 09:50, Igor Mammedov <imammedo@redhat.com> wrote:
>>> In QEMU on x86 (and I think ppc, s390 as well), we create vCPUs on demand.>>
>>> It would be nice if ARM would be able to do that too,
>>> so that it could take advantage of the same code.
>>
>> It's not clear to me how that would work, given that for
>> instance the interrupt controller wants to know up-front
>> how many CPUs it has to deal with.
>>
> 
> So how is cpu hotplug handled in HW? Or doesn't it even exist there?

I don't know of any physical system offering that facility.

> (we have max_cpus for the interrupt controller, but not sure if that is
> what we want)
We'd need something along those lines. Each CPU has a notional point to
point link to the interrupt controller (to the redistributor, to be
precise), and this entity must pre-exist.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
  2018-02-27 13:21         ` Andrew Jones
@ 2018-07-24 18:35           ` Maran Wilson
  -1 siblings, 0 replies; 32+ messages in thread
From: Maran Wilson @ 2018-07-24 18:35 UTC (permalink / raw)
  To: kvmarm, bthakur
  Cc: Andrew Jones, Christoffer Dall, Christoffer Dall, qemu-devel,
	Christoffer Dall, qemu-arm, imammedo, peter.maydell, david,
	marc.zyngier

It's been a few months since this email thread died off. Has anyone 
started working on a potential solution that would allow VCPU hotplug on 
KVM/ARM ? Or is this a project that is still waiting for an owner who 
has the time and inclination to get started?

Thanks,
-Maran

On 2/27/2018 5:21 AM, Andrew Jones wrote:
> On Tue, Feb 27, 2018 at 01:46:04PM +0100, Christoffer Dall wrote:
>> On Tue, Feb 27, 2018 at 05:34:28PM +0530, bthakur@codeaurora.org wrote:
>>> Hi Christoffer,
>>>
>>> Thanks for your reply.
>>>
>>> On 2018-02-27 16:17, Christoffer Dall wrote:
>>>> Hi Bhupinder,
>>>>
>>>> On Tue, Feb 27, 2018 at 03:01:17PM +0530, bthakur@codeaurora.org wrote:
>>>>> I hope it is the right forum to post my query.
>>>>>
>>>>>
>>>>>
>>>>> I am currently looking at the possibility of adding a new VCPU to a
>>>>> running
>>>>> guest VM in KVM/ARM. I see that currently, it is not allowed to add a
>>>>> new
>>>>> VCPU to a guest VM, if it is already initialized. The first check in
>>>>> kvm_arch_vcpu_create() returns failure if it is already initialized.
>>>>>
>>>> This would require a major rework of a lot of logic surrounding the GIC
>>>> and other parts of KVM initialization.
>>>>
>>>>>
>>>>> There was some work done in QEMU to add support for VCPU hotplug:
>>>>> https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html
>>>>>
>>>>>
>>>>>
>>>>> But I am looking at the KVM side for enabling adding a new VCPU. If you
>>>>> can
>>>>> point me to any relevant work/resources, which I can refer to then it
>>>>> will
>>>>> help me.
>>>>>
>>>> I don't have any specific pointers, but I was always told that the way
>>>> we were going to do CPU hotplug would be to instantiate a large number
>>>> of VCPUs, and hotplug would be equivalent to turning on a VCPU which was
>>>> previously powered off.
>>>>
>>>> Is this not still a feasible solution?
>>> It should be a feasible solution provided the guest VM is not able to
>>> control the onlining/offlining of VCPUs. It should be controlled by the
>>> Host.
>>>
>> KVM could simply refuse to turn on some of the CPUs unless given
>> permission from host userspace.
>>
>>>> How does VCPU hotplug work on x86?
>>> On x86, you can add a vcpu through libvirt setvcpu command and it shows up
>>> in the guest VM as a new CPU if you do lscpu.
>>>
>> Sure, but what is the mechanism, does x86 qemu actually call
>> KVM_CREATE_VCPU, or is this also a question of turning on already
>> created vcpus ?
>>
> CC'ing Igor and qemu-devel
>
> drew
> _______________________________________________
> kvmarm mailing list
> kvmarm@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: VCPU hotplug on KVM/ARM
@ 2018-07-24 18:35           ` Maran Wilson
  0 siblings, 0 replies; 32+ messages in thread
From: Maran Wilson @ 2018-07-24 18:35 UTC (permalink / raw)
  To: kvmarm, bthakur
  Cc: Christoffer Dall, Christoffer Dall, marc.zyngier, david,
	qemu-devel, Christoffer Dall, qemu-arm, imammedo

It's been a few months since this email thread died off. Has anyone 
started working on a potential solution that would allow VCPU hotplug on 
KVM/ARM ? Or is this a project that is still waiting for an owner who 
has the time and inclination to get started?

Thanks,
-Maran

On 2/27/2018 5:21 AM, Andrew Jones wrote:
> On Tue, Feb 27, 2018 at 01:46:04PM +0100, Christoffer Dall wrote:
>> On Tue, Feb 27, 2018 at 05:34:28PM +0530, bthakur@codeaurora.org wrote:
>>> Hi Christoffer,
>>>
>>> Thanks for your reply.
>>>
>>> On 2018-02-27 16:17, Christoffer Dall wrote:
>>>> Hi Bhupinder,
>>>>
>>>> On Tue, Feb 27, 2018 at 03:01:17PM +0530, bthakur@codeaurora.org wrote:
>>>>> I hope it is the right forum to post my query.
>>>>>
>>>>>
>>>>>
>>>>> I am currently looking at the possibility of adding a new VCPU to a
>>>>> running
>>>>> guest VM in KVM/ARM. I see that currently, it is not allowed to add a
>>>>> new
>>>>> VCPU to a guest VM, if it is already initialized. The first check in
>>>>> kvm_arch_vcpu_create() returns failure if it is already initialized.
>>>>>
>>>> This would require a major rework of a lot of logic surrounding the GIC
>>>> and other parts of KVM initialization.
>>>>
>>>>>
>>>>> There was some work done in QEMU to add support for VCPU hotplug:
>>>>> https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html
>>>>>
>>>>>
>>>>>
>>>>> But I am looking at the KVM side for enabling adding a new VCPU. If you
>>>>> can
>>>>> point me to any relevant work/resources, which I can refer to then it
>>>>> will
>>>>> help me.
>>>>>
>>>> I don't have any specific pointers, but I was always told that the way
>>>> we were going to do CPU hotplug would be to instantiate a large number
>>>> of VCPUs, and hotplug would be equivalent to turning on a VCPU which was
>>>> previously powered off.
>>>>
>>>> Is this not still a feasible solution?
>>> It should be a feasible solution provided the guest VM is not able to
>>> control the onlining/offlining of VCPUs. It should be controlled by the
>>> Host.
>>>
>> KVM could simply refuse to turn on some of the CPUs unless given
>> permission from host userspace.
>>
>>>> How does VCPU hotplug work on x86?
>>> On x86, you can add a vcpu through libvirt setvcpu command and it shows up
>>> in the guest VM as a new CPU if you do lscpu.
>>>
>> Sure, but what is the mechanism, does x86 qemu actually call
>> KVM_CREATE_VCPU, or is this also a question of turning on already
>> created vcpus ?
>>
> CC'ing Igor and qemu-devel
>
> drew
> _______________________________________________
> kvmarm mailing list
> kvmarm@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
  2018-07-24 18:35           ` Maran Wilson
@ 2018-07-25  6:45             ` Igor Mammedov
  -1 siblings, 0 replies; 32+ messages in thread
From: Igor Mammedov @ 2018-07-25  6:45 UTC (permalink / raw)
  To: Maran Wilson
  Cc: kvmarm, bthakur, Christoffer Dall, Andrew Jones,
	Christoffer Dall, peter.maydell, marc.zyngier, david, qemu-devel,
	Christoffer Dall, qemu-arm

On Tue, 24 Jul 2018 11:35:31 -0700
Maran Wilson <maran.wilson@oracle.com> wrote:

> It's been a few months since this email thread died off. Has anyone 
> started working on a potential solution that would allow VCPU hotplug on 
> KVM/ARM ? Or is this a project that is still waiting for an owner who 
> has the time and inclination to get started?
I'm working on QEMU side of it as time allows.

I can guide or share task if are you interested in helping out with it?


> Thanks,
> -Maran
> 
> On 2/27/2018 5:21 AM, Andrew Jones wrote:
> > On Tue, Feb 27, 2018 at 01:46:04PM +0100, Christoffer Dall wrote:  
> >> On Tue, Feb 27, 2018 at 05:34:28PM +0530, bthakur@codeaurora.org wrote:  
> >>> Hi Christoffer,
> >>>
> >>> Thanks for your reply.
> >>>
> >>> On 2018-02-27 16:17, Christoffer Dall wrote:  
> >>>> Hi Bhupinder,
> >>>>
> >>>> On Tue, Feb 27, 2018 at 03:01:17PM +0530, bthakur@codeaurora.org wrote:  
> >>>>> I hope it is the right forum to post my query.
> >>>>>
> >>>>>
> >>>>>
> >>>>> I am currently looking at the possibility of adding a new VCPU to a
> >>>>> running
> >>>>> guest VM in KVM/ARM. I see that currently, it is not allowed to add a
> >>>>> new
> >>>>> VCPU to a guest VM, if it is already initialized. The first check in
> >>>>> kvm_arch_vcpu_create() returns failure if it is already initialized.
> >>>>>  
> >>>> This would require a major rework of a lot of logic surrounding the GIC
> >>>> and other parts of KVM initialization.
> >>>>  
> >>>>>
> >>>>> There was some work done in QEMU to add support for VCPU hotplug:
> >>>>> https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html
> >>>>>
> >>>>>
> >>>>>
> >>>>> But I am looking at the KVM side for enabling adding a new VCPU. If you
> >>>>> can
> >>>>> point me to any relevant work/resources, which I can refer to then it
> >>>>> will
> >>>>> help me.
> >>>>>  
> >>>> I don't have any specific pointers, but I was always told that the way
> >>>> we were going to do CPU hotplug would be to instantiate a large number
> >>>> of VCPUs, and hotplug would be equivalent to turning on a VCPU which was
> >>>> previously powered off.
> >>>>
> >>>> Is this not still a feasible solution?  
> >>> It should be a feasible solution provided the guest VM is not able to
> >>> control the onlining/offlining of VCPUs. It should be controlled by the
> >>> Host.
> >>>  
> >> KVM could simply refuse to turn on some of the CPUs unless given
> >> permission from host userspace.
> >>  
> >>>> How does VCPU hotplug work on x86?  
> >>> On x86, you can add a vcpu through libvirt setvcpu command and it shows up
> >>> in the guest VM as a new CPU if you do lscpu.
> >>>  
> >> Sure, but what is the mechanism, does x86 qemu actually call
> >> KVM_CREATE_VCPU, or is this also a question of turning on already
> >> created vcpus ?
> >>  
> > CC'ing Igor and qemu-devel
> >
> > drew
> > _______________________________________________
> > kvmarm mailing list
> > kvmarm@lists.cs.columbia.edu
> > https://lists.cs.columbia.edu/mailman/listinfo/kvmarm  
> 
> 

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
@ 2018-07-25  6:45             ` Igor Mammedov
  0 siblings, 0 replies; 32+ messages in thread
From: Igor Mammedov @ 2018-07-25  6:45 UTC (permalink / raw)
  To: Maran Wilson
  Cc: Christoffer Dall, Christoffer Dall, marc.zyngier, david,
	qemu-devel, Christoffer Dall, qemu-arm, kvmarm

On Tue, 24 Jul 2018 11:35:31 -0700
Maran Wilson <maran.wilson@oracle.com> wrote:

> It's been a few months since this email thread died off. Has anyone 
> started working on a potential solution that would allow VCPU hotplug on 
> KVM/ARM ? Or is this a project that is still waiting for an owner who 
> has the time and inclination to get started?
I'm working on QEMU side of it as time allows.

I can guide or share task if are you interested in helping out with it?


> Thanks,
> -Maran
> 
> On 2/27/2018 5:21 AM, Andrew Jones wrote:
> > On Tue, Feb 27, 2018 at 01:46:04PM +0100, Christoffer Dall wrote:  
> >> On Tue, Feb 27, 2018 at 05:34:28PM +0530, bthakur@codeaurora.org wrote:  
> >>> Hi Christoffer,
> >>>
> >>> Thanks for your reply.
> >>>
> >>> On 2018-02-27 16:17, Christoffer Dall wrote:  
> >>>> Hi Bhupinder,
> >>>>
> >>>> On Tue, Feb 27, 2018 at 03:01:17PM +0530, bthakur@codeaurora.org wrote:  
> >>>>> I hope it is the right forum to post my query.
> >>>>>
> >>>>>
> >>>>>
> >>>>> I am currently looking at the possibility of adding a new VCPU to a
> >>>>> running
> >>>>> guest VM in KVM/ARM. I see that currently, it is not allowed to add a
> >>>>> new
> >>>>> VCPU to a guest VM, if it is already initialized. The first check in
> >>>>> kvm_arch_vcpu_create() returns failure if it is already initialized.
> >>>>>  
> >>>> This would require a major rework of a lot of logic surrounding the GIC
> >>>> and other parts of KVM initialization.
> >>>>  
> >>>>>
> >>>>> There was some work done in QEMU to add support for VCPU hotplug:
> >>>>> https://lists.gnu.org/archive/html/qemu-arm/2017-05/msg00404.html
> >>>>>
> >>>>>
> >>>>>
> >>>>> But I am looking at the KVM side for enabling adding a new VCPU. If you
> >>>>> can
> >>>>> point me to any relevant work/resources, which I can refer to then it
> >>>>> will
> >>>>> help me.
> >>>>>  
> >>>> I don't have any specific pointers, but I was always told that the way
> >>>> we were going to do CPU hotplug would be to instantiate a large number
> >>>> of VCPUs, and hotplug would be equivalent to turning on a VCPU which was
> >>>> previously powered off.
> >>>>
> >>>> Is this not still a feasible solution?  
> >>> It should be a feasible solution provided the guest VM is not able to
> >>> control the onlining/offlining of VCPUs. It should be controlled by the
> >>> Host.
> >>>  
> >> KVM could simply refuse to turn on some of the CPUs unless given
> >> permission from host userspace.
> >>  
> >>>> How does VCPU hotplug work on x86?  
> >>> On x86, you can add a vcpu through libvirt setvcpu command and it shows up
> >>> in the guest VM as a new CPU if you do lscpu.
> >>>  
> >> Sure, but what is the mechanism, does x86 qemu actually call
> >> KVM_CREATE_VCPU, or is this also a question of turning on already
> >> created vcpus ?
> >>  
> > CC'ing Igor and qemu-devel
> >
> > drew
> > _______________________________________________
> > kvmarm mailing list
> > kvmarm@lists.cs.columbia.edu
> > https://lists.cs.columbia.edu/mailman/listinfo/kvmarm  
> 
> 

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
  2018-07-24 18:35           ` Maran Wilson
@ 2018-07-25 10:40             ` Marc Zyngier
  -1 siblings, 0 replies; 32+ messages in thread
From: Marc Zyngier @ 2018-07-25 10:40 UTC (permalink / raw)
  To: Maran Wilson, kvmarm, bthakur
  Cc: Andrew Jones, Christoffer Dall, Christoffer Dall, qemu-devel,
	Christoffer Dall, qemu-arm, imammedo, peter.maydell, david

On 24/07/18 19:35, Maran Wilson wrote:
> It's been a few months since this email thread died off. Has anyone 
> started working on a potential solution that would allow VCPU hotplug on 
> KVM/ARM ? Or is this a project that is still waiting for an owner who 
> has the time and inclination to get started?

This is typically a project for someone who would have this particular
itch to scratch, and who has a demonstrable need for this functionality.

Work wise, it would have to include adding physical CPU hotplug support
to the arm64 kernel as a precondition, before worrying about doing it in
KVM.

For KVM itself, particular area of interests would be:
- Making GICv3 redistributors magically appear in the IPA space
- Live resizing of GICv3 structures
- Dynamic allocation of MPIDR, and mapping with vcpu_id

This should keep someone busy for a good couple of weeks (give or take a
few months).

That being said, I'd rather see support in QEMU first, creating all the
vcpu/redistributors upfront, and signalling the hotplug event via the
virtual firmware. And then post some numbers to show that creating all
the vcpus upfront is not acceptable.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: VCPU hotplug on KVM/ARM
@ 2018-07-25 10:40             ` Marc Zyngier
  0 siblings, 0 replies; 32+ messages in thread
From: Marc Zyngier @ 2018-07-25 10:40 UTC (permalink / raw)
  To: Maran Wilson, kvmarm, bthakur
  Cc: Christoffer Dall, Christoffer Dall, david, qemu-devel,
	Christoffer Dall, qemu-arm, imammedo

On 24/07/18 19:35, Maran Wilson wrote:
> It's been a few months since this email thread died off. Has anyone 
> started working on a potential solution that would allow VCPU hotplug on 
> KVM/ARM ? Or is this a project that is still waiting for an owner who 
> has the time and inclination to get started?

This is typically a project for someone who would have this particular
itch to scratch, and who has a demonstrable need for this functionality.

Work wise, it would have to include adding physical CPU hotplug support
to the arm64 kernel as a precondition, before worrying about doing it in
KVM.

For KVM itself, particular area of interests would be:
- Making GICv3 redistributors magically appear in the IPA space
- Live resizing of GICv3 structures
- Dynamic allocation of MPIDR, and mapping with vcpu_id

This should keep someone busy for a good couple of weeks (give or take a
few months).

That being said, I'd rather see support in QEMU first, creating all the
vcpu/redistributors upfront, and signalling the hotplug event via the
virtual firmware. And then post some numbers to show that creating all
the vcpus upfront is not acceptable.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
  2018-07-25 10:40             ` Marc Zyngier
@ 2018-07-25 12:28               ` Andrew Jones
  -1 siblings, 0 replies; 32+ messages in thread
From: Andrew Jones @ 2018-07-25 12:28 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: Maran Wilson, kvmarm, bthakur, Christoffer Dall,
	Christoffer Dall, peter.maydell, david, qemu-devel,
	Christoffer Dall, qemu-arm, imammedo

On Wed, Jul 25, 2018 at 11:40:54AM +0100, Marc Zyngier wrote:
> On 24/07/18 19:35, Maran Wilson wrote:
> > It's been a few months since this email thread died off. Has anyone 
> > started working on a potential solution that would allow VCPU hotplug on 
> > KVM/ARM ? Or is this a project that is still waiting for an owner who 
> > has the time and inclination to get started?
> 
> This is typically a project for someone who would have this particular
> itch to scratch, and who has a demonstrable need for this functionality.
> 
> Work wise, it would have to include adding physical CPU hotplug support
> to the arm64 kernel as a precondition, before worrying about doing it in
> KVM.
> 
> For KVM itself, particular area of interests would be:
> - Making GICv3 redistributors magically appear in the IPA space
> - Live resizing of GICv3 structures
> - Dynamic allocation of MPIDR, and mapping with vcpu_id

I have CPU topology description patches on the QEMU list now[*]. A next
step for me is to this MPIDR work. I probably won't get to it until the
end of August though.

[*] http://lists.nongnu.org/archive/html/qemu-devel/2018-07/msg01168.html

> 
> This should keep someone busy for a good couple of weeks (give or take a
> few months).

:-)

> 
> That being said, I'd rather see support in QEMU first, creating all the
> vcpu/redistributors upfront, and signalling the hotplug event via the
> virtual firmware. And then post some numbers to show that creating all
> the vcpus upfront is not acceptable.

I think the upfront allocation, allocating all possible cpus, but only
activating all present cpus, was the planned approach. What were the
concerns about that approach? Just vcpu memory overhead for too many
overly ambitious VM configs?

Thanks,
drew

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
@ 2018-07-25 12:28               ` Andrew Jones
  0 siblings, 0 replies; 32+ messages in thread
From: Andrew Jones @ 2018-07-25 12:28 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: Christoffer Dall, Christoffer Dall, david, qemu-devel,
	Christoffer Dall, qemu-arm, imammedo, kvmarm

On Wed, Jul 25, 2018 at 11:40:54AM +0100, Marc Zyngier wrote:
> On 24/07/18 19:35, Maran Wilson wrote:
> > It's been a few months since this email thread died off. Has anyone 
> > started working on a potential solution that would allow VCPU hotplug on 
> > KVM/ARM ? Or is this a project that is still waiting for an owner who 
> > has the time and inclination to get started?
> 
> This is typically a project for someone who would have this particular
> itch to scratch, and who has a demonstrable need for this functionality.
> 
> Work wise, it would have to include adding physical CPU hotplug support
> to the arm64 kernel as a precondition, before worrying about doing it in
> KVM.
> 
> For KVM itself, particular area of interests would be:
> - Making GICv3 redistributors magically appear in the IPA space
> - Live resizing of GICv3 structures
> - Dynamic allocation of MPIDR, and mapping with vcpu_id

I have CPU topology description patches on the QEMU list now[*]. A next
step for me is to this MPIDR work. I probably won't get to it until the
end of August though.

[*] http://lists.nongnu.org/archive/html/qemu-devel/2018-07/msg01168.html

> 
> This should keep someone busy for a good couple of weeks (give or take a
> few months).

:-)

> 
> That being said, I'd rather see support in QEMU first, creating all the
> vcpu/redistributors upfront, and signalling the hotplug event via the
> virtual firmware. And then post some numbers to show that creating all
> the vcpus upfront is not acceptable.

I think the upfront allocation, allocating all possible cpus, but only
activating all present cpus, was the planned approach. What were the
concerns about that approach? Just vcpu memory overhead for too many
overly ambitious VM configs?

Thanks,
drew

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
  2018-07-25 12:28               ` Andrew Jones
@ 2018-07-25 13:07                 ` Marc Zyngier
  -1 siblings, 0 replies; 32+ messages in thread
From: Marc Zyngier @ 2018-07-25 13:07 UTC (permalink / raw)
  To: Andrew Jones
  Cc: Maran Wilson, kvmarm, bthakur, Christoffer Dall,
	Christoffer Dall, peter.maydell, david, qemu-devel,
	Christoffer Dall, qemu-arm, imammedo

On 25/07/18 13:28, Andrew Jones wrote:
> On Wed, Jul 25, 2018 at 11:40:54AM +0100, Marc Zyngier wrote:
>> On 24/07/18 19:35, Maran Wilson wrote:
>>> It's been a few months since this email thread died off. Has anyone 
>>> started working on a potential solution that would allow VCPU hotplug on 
>>> KVM/ARM ? Or is this a project that is still waiting for an owner who 
>>> has the time and inclination to get started?
>>
>> This is typically a project for someone who would have this particular
>> itch to scratch, and who has a demonstrable need for this functionality.
>>
>> Work wise, it would have to include adding physical CPU hotplug support
>> to the arm64 kernel as a precondition, before worrying about doing it in
>> KVM.
>>
>> For KVM itself, particular area of interests would be:
>> - Making GICv3 redistributors magically appear in the IPA space
>> - Live resizing of GICv3 structures
>> - Dynamic allocation of MPIDR, and mapping with vcpu_id
> 
> I have CPU topology description patches on the QEMU list now[*]. A next
> step for me is to this MPIDR work. I probably won't get to it until the
> end of August though.
> 
> [*] http://lists.nongnu.org/archive/html/qemu-devel/2018-07/msg01168.html
> 
>>
>> This should keep someone busy for a good couple of weeks (give or take a
>> few months).
> 
> :-)
> 
>>
>> That being said, I'd rather see support in QEMU first, creating all the
>> vcpu/redistributors upfront, and signalling the hotplug event via the
>> virtual firmware. And then post some numbers to show that creating all
>> the vcpus upfront is not acceptable.
> 
> I think the upfront allocation, allocating all possible cpus, but only
> activating all present cpus, was the planned approach. What were the
> concerns about that approach? Just vcpu memory overhead for too many
> overly ambitious VM configs?

I don't have any ARM-specific concern about that, and I think this is
the right approach. It has the good property of not requiring much
change in the kernel (other than actually supporting CPU hotplug).

vcpu memory overhead is a generic concern though, and not only for ARM.
We currently allow up to 512 vcpus per VM, which looks like a lot, but
really isn't. If we're to allow this to be bumped up significantly, we
should start accounting the vcpu-related memory against the user's
allowance...

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
@ 2018-07-25 13:07                 ` Marc Zyngier
  0 siblings, 0 replies; 32+ messages in thread
From: Marc Zyngier @ 2018-07-25 13:07 UTC (permalink / raw)
  To: Andrew Jones
  Cc: Christoffer Dall, Christoffer Dall, david, qemu-devel,
	Christoffer Dall, qemu-arm, imammedo, kvmarm

On 25/07/18 13:28, Andrew Jones wrote:
> On Wed, Jul 25, 2018 at 11:40:54AM +0100, Marc Zyngier wrote:
>> On 24/07/18 19:35, Maran Wilson wrote:
>>> It's been a few months since this email thread died off. Has anyone 
>>> started working on a potential solution that would allow VCPU hotplug on 
>>> KVM/ARM ? Or is this a project that is still waiting for an owner who 
>>> has the time and inclination to get started?
>>
>> This is typically a project for someone who would have this particular
>> itch to scratch, and who has a demonstrable need for this functionality.
>>
>> Work wise, it would have to include adding physical CPU hotplug support
>> to the arm64 kernel as a precondition, before worrying about doing it in
>> KVM.
>>
>> For KVM itself, particular area of interests would be:
>> - Making GICv3 redistributors magically appear in the IPA space
>> - Live resizing of GICv3 structures
>> - Dynamic allocation of MPIDR, and mapping with vcpu_id
> 
> I have CPU topology description patches on the QEMU list now[*]. A next
> step for me is to this MPIDR work. I probably won't get to it until the
> end of August though.
> 
> [*] http://lists.nongnu.org/archive/html/qemu-devel/2018-07/msg01168.html
> 
>>
>> This should keep someone busy for a good couple of weeks (give or take a
>> few months).
> 
> :-)
> 
>>
>> That being said, I'd rather see support in QEMU first, creating all the
>> vcpu/redistributors upfront, and signalling the hotplug event via the
>> virtual firmware. And then post some numbers to show that creating all
>> the vcpus upfront is not acceptable.
> 
> I think the upfront allocation, allocating all possible cpus, but only
> activating all present cpus, was the planned approach. What were the
> concerns about that approach? Just vcpu memory overhead for too many
> overly ambitious VM configs?

I don't have any ARM-specific concern about that, and I think this is
the right approach. It has the good property of not requiring much
change in the kernel (other than actually supporting CPU hotplug).

vcpu memory overhead is a generic concern though, and not only for ARM.
We currently allow up to 512 vcpus per VM, which looks like a lot, but
really isn't. If we're to allow this to be bumped up significantly, we
should start accounting the vcpu-related memory against the user's
allowance...

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
  2018-07-25 13:07                 ` Marc Zyngier
@ 2018-07-25 17:26                   ` Maran Wilson
  -1 siblings, 0 replies; 32+ messages in thread
From: Maran Wilson @ 2018-07-25 17:26 UTC (permalink / raw)
  To: Marc Zyngier, Andrew Jones, imammedo, kvmarm
  Cc: bthakur, Christoffer Dall, Christoffer Dall, peter.maydell,
	david, qemu-devel, Christoffer Dall, qemu-arm

Thanks everyone. It sounds like there is consensus around how best to 
proceed (at a high level at least).

Since Igor has already gotten started, I'll coordinate with him offline 
to see where I can jump in.

Thanks,
-Maran

On 7/25/2018 6:07 AM, Marc Zyngier wrote:
> On 25/07/18 13:28, Andrew Jones wrote:
>> On Wed, Jul 25, 2018 at 11:40:54AM +0100, Marc Zyngier wrote:
>>> On 24/07/18 19:35, Maran Wilson wrote:
>>>> It's been a few months since this email thread died off. Has anyone
>>>> started working on a potential solution that would allow VCPU hotplug on
>>>> KVM/ARM ? Or is this a project that is still waiting for an owner who
>>>> has the time and inclination to get started?
>>> This is typically a project for someone who would have this particular
>>> itch to scratch, and who has a demonstrable need for this functionality.
>>>
>>> Work wise, it would have to include adding physical CPU hotplug support
>>> to the arm64 kernel as a precondition, before worrying about doing it in
>>> KVM.
>>>
>>> For KVM itself, particular area of interests would be:
>>> - Making GICv3 redistributors magically appear in the IPA space
>>> - Live resizing of GICv3 structures
>>> - Dynamic allocation of MPIDR, and mapping with vcpu_id
>> I have CPU topology description patches on the QEMU list now[*]. A next
>> step for me is to this MPIDR work. I probably won't get to it until the
>> end of August though.
>>
>> [*] http://lists.nongnu.org/archive/html/qemu-devel/2018-07/msg01168.html
>>
>>> This should keep someone busy for a good couple of weeks (give or take a
>>> few months).
>> :-)
>>
>>> That being said, I'd rather see support in QEMU first, creating all the
>>> vcpu/redistributors upfront, and signalling the hotplug event via the
>>> virtual firmware. And then post some numbers to show that creating all
>>> the vcpus upfront is not acceptable.
>> I think the upfront allocation, allocating all possible cpus, but only
>> activating all present cpus, was the planned approach. What were the
>> concerns about that approach? Just vcpu memory overhead for too many
>> overly ambitious VM configs?
> I don't have any ARM-specific concern about that, and I think this is
> the right approach. It has the good property of not requiring much
> change in the kernel (other than actually supporting CPU hotplug).
>
> vcpu memory overhead is a generic concern though, and not only for ARM.
> We currently allow up to 512 vcpus per VM, which looks like a lot, but
> really isn't. If we're to allow this to be bumped up significantly, we
> should start accounting the vcpu-related memory against the user's
> allowance...
>
> Thanks,
>
> 	M.

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
@ 2018-07-25 17:26                   ` Maran Wilson
  0 siblings, 0 replies; 32+ messages in thread
From: Maran Wilson @ 2018-07-25 17:26 UTC (permalink / raw)
  To: Marc Zyngier, Andrew Jones, imammedo, kvmarm
  Cc: Christoffer Dall, Christoffer Dall, david, qemu-devel,
	Christoffer Dall, qemu-arm

Thanks everyone. It sounds like there is consensus around how best to 
proceed (at a high level at least).

Since Igor has already gotten started, I'll coordinate with him offline 
to see where I can jump in.

Thanks,
-Maran

On 7/25/2018 6:07 AM, Marc Zyngier wrote:
> On 25/07/18 13:28, Andrew Jones wrote:
>> On Wed, Jul 25, 2018 at 11:40:54AM +0100, Marc Zyngier wrote:
>>> On 24/07/18 19:35, Maran Wilson wrote:
>>>> It's been a few months since this email thread died off. Has anyone
>>>> started working on a potential solution that would allow VCPU hotplug on
>>>> KVM/ARM ? Or is this a project that is still waiting for an owner who
>>>> has the time and inclination to get started?
>>> This is typically a project for someone who would have this particular
>>> itch to scratch, and who has a demonstrable need for this functionality.
>>>
>>> Work wise, it would have to include adding physical CPU hotplug support
>>> to the arm64 kernel as a precondition, before worrying about doing it in
>>> KVM.
>>>
>>> For KVM itself, particular area of interests would be:
>>> - Making GICv3 redistributors magically appear in the IPA space
>>> - Live resizing of GICv3 structures
>>> - Dynamic allocation of MPIDR, and mapping with vcpu_id
>> I have CPU topology description patches on the QEMU list now[*]. A next
>> step for me is to this MPIDR work. I probably won't get to it until the
>> end of August though.
>>
>> [*] http://lists.nongnu.org/archive/html/qemu-devel/2018-07/msg01168.html
>>
>>> This should keep someone busy for a good couple of weeks (give or take a
>>> few months).
>> :-)
>>
>>> That being said, I'd rather see support in QEMU first, creating all the
>>> vcpu/redistributors upfront, and signalling the hotplug event via the
>>> virtual firmware. And then post some numbers to show that creating all
>>> the vcpus upfront is not acceptable.
>> I think the upfront allocation, allocating all possible cpus, but only
>> activating all present cpus, was the planned approach. What were the
>> concerns about that approach? Just vcpu memory overhead for too many
>> overly ambitious VM configs?
> I don't have any ARM-specific concern about that, and I think this is
> the right approach. It has the good property of not requiring much
> change in the kernel (other than actually supporting CPU hotplug).
>
> vcpu memory overhead is a generic concern though, and not only for ARM.
> We currently allow up to 512 vcpus per VM, which looks like a lot, but
> really isn't. If we're to allow this to be bumped up significantly, we
> should start accounting the vcpu-related memory against the user's
> allowance...
>
> Thanks,
>
> 	M.

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
  2018-07-25 13:07                 ` Marc Zyngier
@ 2018-07-31 10:27                   ` Igor Mammedov
  -1 siblings, 0 replies; 32+ messages in thread
From: Igor Mammedov @ 2018-07-31 10:27 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: Andrew Jones, Maran Wilson, kvmarm, bthakur, Christoffer Dall,
	Christoffer Dall, peter.maydell, david, qemu-devel,
	Christoffer Dall, qemu-arm, David Gibson, cohuck, borntraeger

On Wed, 25 Jul 2018 14:07:12 +0100
Marc Zyngier <marc.zyngier@arm.com> wrote:

> On 25/07/18 13:28, Andrew Jones wrote:
> > On Wed, Jul 25, 2018 at 11:40:54AM +0100, Marc Zyngier wrote:  
> >> On 24/07/18 19:35, Maran Wilson wrote:  
> >>> It's been a few months since this email thread died off. Has anyone 
> >>> started working on a potential solution that would allow VCPU hotplug on 
> >>> KVM/ARM ? Or is this a project that is still waiting for an owner who 
> >>> has the time and inclination to get started?  
> >>
> >> This is typically a project for someone who would have this particular
> >> itch to scratch, and who has a demonstrable need for this functionality.
> >>
> >> Work wise, it would have to include adding physical CPU hotplug support
> >> to the arm64 kernel as a precondition, before worrying about doing it in
> >> KVM.
> >>
> >> For KVM itself, particular area of interests would be:
> >> - Making GICv3 redistributors magically appear in the IPA space
> >> - Live resizing of GICv3 structures
> >> - Dynamic allocation of MPIDR, and mapping with vcpu_id  
> > 
> > I have CPU topology description patches on the QEMU list now[*]. A next
> > step for me is to this MPIDR work. I probably won't get to it until the
> > end of August though.
> > 
> > [*] http://lists.nongnu.org/archive/html/qemu-devel/2018-07/msg01168.html
> >   
> >>
> >> This should keep someone busy for a good couple of weeks (give or take a
> >> few months).  
> > 
> > :-)
> >   
> >>
> >> That being said, I'd rather see support in QEMU first, creating all the
> >> vcpu/redistributors upfront, and signalling the hotplug event via the
> >> virtual firmware. And then post some numbers to show that creating all
> >> the vcpus upfront is not acceptable.  
> > 
> > I think the upfront allocation, allocating all possible cpus, but only
> > activating all present cpus, was the planned approach. What were the
> > concerns about that approach? Just vcpu memory overhead for too many
> > overly ambitious VM configs?  
> 
> I don't have any ARM-specific concern about that, and I think this is
> the right approach. It has the good property of not requiring much
> change in the kernel (other than actually supporting CPU hotplug).
for x86 we allocate VCPUs dynamically (both QEMU and KVM)
CCing ppc/s390 folks as I don't recall how it's implemented there.

but we do not delete vcpus in KVM after they were created
(as it deemed to be too complicated), we are just deleting QEMU part
of it and keep kvm's vcpu for reuse with future hotplug.

> vcpu memory overhead is a generic concern though, and not only for ARM.
> We currently allow up to 512 vcpus per VM, which looks like a lot, but
> really isn't. If we're to allow this to be bumped up significantly, we
> should start accounting the vcpu-related memory against the user's
> allowance...
> 
> Thanks,
> 
> 	M.

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
@ 2018-07-31 10:27                   ` Igor Mammedov
  0 siblings, 0 replies; 32+ messages in thread
From: Igor Mammedov @ 2018-07-31 10:27 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: Christoffer Dall, cohuck, Christoffer Dall, david, qemu-devel,
	borntraeger, Christoffer Dall, qemu-arm, kvmarm, David Gibson

On Wed, 25 Jul 2018 14:07:12 +0100
Marc Zyngier <marc.zyngier@arm.com> wrote:

> On 25/07/18 13:28, Andrew Jones wrote:
> > On Wed, Jul 25, 2018 at 11:40:54AM +0100, Marc Zyngier wrote:  
> >> On 24/07/18 19:35, Maran Wilson wrote:  
> >>> It's been a few months since this email thread died off. Has anyone 
> >>> started working on a potential solution that would allow VCPU hotplug on 
> >>> KVM/ARM ? Or is this a project that is still waiting for an owner who 
> >>> has the time and inclination to get started?  
> >>
> >> This is typically a project for someone who would have this particular
> >> itch to scratch, and who has a demonstrable need for this functionality.
> >>
> >> Work wise, it would have to include adding physical CPU hotplug support
> >> to the arm64 kernel as a precondition, before worrying about doing it in
> >> KVM.
> >>
> >> For KVM itself, particular area of interests would be:
> >> - Making GICv3 redistributors magically appear in the IPA space
> >> - Live resizing of GICv3 structures
> >> - Dynamic allocation of MPIDR, and mapping with vcpu_id  
> > 
> > I have CPU topology description patches on the QEMU list now[*]. A next
> > step for me is to this MPIDR work. I probably won't get to it until the
> > end of August though.
> > 
> > [*] http://lists.nongnu.org/archive/html/qemu-devel/2018-07/msg01168.html
> >   
> >>
> >> This should keep someone busy for a good couple of weeks (give or take a
> >> few months).  
> > 
> > :-)
> >   
> >>
> >> That being said, I'd rather see support in QEMU first, creating all the
> >> vcpu/redistributors upfront, and signalling the hotplug event via the
> >> virtual firmware. And then post some numbers to show that creating all
> >> the vcpus upfront is not acceptable.  
> > 
> > I think the upfront allocation, allocating all possible cpus, but only
> > activating all present cpus, was the planned approach. What were the
> > concerns about that approach? Just vcpu memory overhead for too many
> > overly ambitious VM configs?  
> 
> I don't have any ARM-specific concern about that, and I think this is
> the right approach. It has the good property of not requiring much
> change in the kernel (other than actually supporting CPU hotplug).
for x86 we allocate VCPUs dynamically (both QEMU and KVM)
CCing ppc/s390 folks as I don't recall how it's implemented there.

but we do not delete vcpus in KVM after they were created
(as it deemed to be too complicated), we are just deleting QEMU part
of it and keep kvm's vcpu for reuse with future hotplug.

> vcpu memory overhead is a generic concern though, and not only for ARM.
> We currently allow up to 512 vcpus per VM, which looks like a lot, but
> really isn't. If we're to allow this to be bumped up significantly, we
> should start accounting the vcpu-related memory against the user's
> allowance...
> 
> Thanks,
> 
> 	M.

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
  2018-07-31 10:27                   ` Igor Mammedov
@ 2018-07-31 10:57                     ` David Hildenbrand
  -1 siblings, 0 replies; 32+ messages in thread
From: David Hildenbrand @ 2018-07-31 10:57 UTC (permalink / raw)
  To: Igor Mammedov, Marc Zyngier
  Cc: Andrew Jones, Maran Wilson, kvmarm, bthakur, Christoffer Dall,
	Christoffer Dall, peter.maydell, qemu-devel, Christoffer Dall,
	qemu-arm, David Gibson, cohuck, borntraeger

On 31.07.2018 12:27, Igor Mammedov wrote:
> On Wed, 25 Jul 2018 14:07:12 +0100
> Marc Zyngier <marc.zyngier@arm.com> wrote:
> 
>> On 25/07/18 13:28, Andrew Jones wrote:
>>> On Wed, Jul 25, 2018 at 11:40:54AM +0100, Marc Zyngier wrote:  
>>>> On 24/07/18 19:35, Maran Wilson wrote:  
>>>>> It's been a few months since this email thread died off. Has anyone 
>>>>> started working on a potential solution that would allow VCPU hotplug on 
>>>>> KVM/ARM ? Or is this a project that is still waiting for an owner who 
>>>>> has the time and inclination to get started?  
>>>>
>>>> This is typically a project for someone who would have this particular
>>>> itch to scratch, and who has a demonstrable need for this functionality.
>>>>
>>>> Work wise, it would have to include adding physical CPU hotplug support
>>>> to the arm64 kernel as a precondition, before worrying about doing it in
>>>> KVM.
>>>>
>>>> For KVM itself, particular area of interests would be:
>>>> - Making GICv3 redistributors magically appear in the IPA space
>>>> - Live resizing of GICv3 structures
>>>> - Dynamic allocation of MPIDR, and mapping with vcpu_id  
>>>
>>> I have CPU topology description patches on the QEMU list now[*]. A next
>>> step for me is to this MPIDR work. I probably won't get to it until the
>>> end of August though.
>>>
>>> [*] http://lists.nongnu.org/archive/html/qemu-devel/2018-07/msg01168.html
>>>   
>>>>
>>>> This should keep someone busy for a good couple of weeks (give or take a
>>>> few months).  
>>>
>>> :-)
>>>   
>>>>
>>>> That being said, I'd rather see support in QEMU first, creating all the
>>>> vcpu/redistributors upfront, and signalling the hotplug event via the
>>>> virtual firmware. And then post some numbers to show that creating all
>>>> the vcpus upfront is not acceptable.  
>>>
>>> I think the upfront allocation, allocating all possible cpus, but only
>>> activating all present cpus, was the planned approach. What were the
>>> concerns about that approach? Just vcpu memory overhead for too many
>>> overly ambitious VM configs?  
>>
>> I don't have any ARM-specific concern about that, and I think this is
>> the right approach. It has the good property of not requiring much
>> change in the kernel (other than actually supporting CPU hotplug).
> for x86 we allocate VCPUs dynamically (both QEMU and KVM)
> CCing ppc/s390 folks as I don't recall how it's implemented there.

s390x: also handled that way. Dynamically allocated.

Unplug: not supported by the architecture and fenced.

I remember a discussion where people said dynamically creating/deleting
VCPUs (and therefore threads) is preferred, because then there is no way
on earth a malicious guest could make use of such a CPU (in case there
would be a subtle BUG in QEMU).

-- 

Thanks,

David / dhildenb

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
@ 2018-07-31 10:57                     ` David Hildenbrand
  0 siblings, 0 replies; 32+ messages in thread
From: David Hildenbrand @ 2018-07-31 10:57 UTC (permalink / raw)
  To: Igor Mammedov, Marc Zyngier
  Cc: Christoffer Dall, Christoffer Dall, cohuck, qemu-devel,
	borntraeger, Christoffer Dall, qemu-arm, kvmarm, David Gibson

On 31.07.2018 12:27, Igor Mammedov wrote:
> On Wed, 25 Jul 2018 14:07:12 +0100
> Marc Zyngier <marc.zyngier@arm.com> wrote:
> 
>> On 25/07/18 13:28, Andrew Jones wrote:
>>> On Wed, Jul 25, 2018 at 11:40:54AM +0100, Marc Zyngier wrote:  
>>>> On 24/07/18 19:35, Maran Wilson wrote:  
>>>>> It's been a few months since this email thread died off. Has anyone 
>>>>> started working on a potential solution that would allow VCPU hotplug on 
>>>>> KVM/ARM ? Or is this a project that is still waiting for an owner who 
>>>>> has the time and inclination to get started?  
>>>>
>>>> This is typically a project for someone who would have this particular
>>>> itch to scratch, and who has a demonstrable need for this functionality.
>>>>
>>>> Work wise, it would have to include adding physical CPU hotplug support
>>>> to the arm64 kernel as a precondition, before worrying about doing it in
>>>> KVM.
>>>>
>>>> For KVM itself, particular area of interests would be:
>>>> - Making GICv3 redistributors magically appear in the IPA space
>>>> - Live resizing of GICv3 structures
>>>> - Dynamic allocation of MPIDR, and mapping with vcpu_id  
>>>
>>> I have CPU topology description patches on the QEMU list now[*]. A next
>>> step for me is to this MPIDR work. I probably won't get to it until the
>>> end of August though.
>>>
>>> [*] http://lists.nongnu.org/archive/html/qemu-devel/2018-07/msg01168.html
>>>   
>>>>
>>>> This should keep someone busy for a good couple of weeks (give or take a
>>>> few months).  
>>>
>>> :-)
>>>   
>>>>
>>>> That being said, I'd rather see support in QEMU first, creating all the
>>>> vcpu/redistributors upfront, and signalling the hotplug event via the
>>>> virtual firmware. And then post some numbers to show that creating all
>>>> the vcpus upfront is not acceptable.  
>>>
>>> I think the upfront allocation, allocating all possible cpus, but only
>>> activating all present cpus, was the planned approach. What were the
>>> concerns about that approach? Just vcpu memory overhead for too many
>>> overly ambitious VM configs?  
>>
>> I don't have any ARM-specific concern about that, and I think this is
>> the right approach. It has the good property of not requiring much
>> change in the kernel (other than actually supporting CPU hotplug).
> for x86 we allocate VCPUs dynamically (both QEMU and KVM)
> CCing ppc/s390 folks as I don't recall how it's implemented there.

s390x: also handled that way. Dynamically allocated.

Unplug: not supported by the architecture and fenced.

I remember a discussion where people said dynamically creating/deleting
VCPUs (and therefore threads) is preferred, because then there is no way
on earth a malicious guest could make use of such a CPU (in case there
would be a subtle BUG in QEMU).

-- 

Thanks,

David / dhildenb

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: [Qemu-devel] VCPU hotplug on KVM/ARM
  2018-07-31 10:27                   ` Igor Mammedov
@ 2018-08-01  8:09                     ` Bharata B Rao
  -1 siblings, 0 replies; 32+ messages in thread
From: Bharata B Rao @ 2018-08-01  8:09 UTC (permalink / raw)
  To: Igor Mammedov
  Cc: Marc Zyngier, bthakur, Christoffer Dall, Andrew Jones, cohuck,
	Christoffer Dall, Maran Wilson, Peter Maydell, david, qemu-devel,
	borntraeger, Christoffer Dall, qemu-arm, kvmarm, David Gibson

On Tue, Jul 31, 2018 at 3:57 PM, Igor Mammedov <imammedo@redhat.com> wrote:

> On Wed, 25 Jul 2018 14:07:12 +0100
> Marc Zyngier <marc.zyngier@arm.com> wrote:
>
> > On 25/07/18 13:28, Andrew Jones wrote:
> > > On Wed, Jul 25, 2018 at 11:40:54AM +0100, Marc Zyngier wrote:
> > >> On 24/07/18 19:35, Maran Wilson wrote:
> > >>> It's been a few months since this email thread died off. Has anyone
> > >>> started working on a potential solution that would allow VCPU
> hotplug on
> > >>> KVM/ARM ? Or is this a project that is still waiting for an owner
> who
> > >>> has the time and inclination to get started?
> > >>
> > >> This is typically a project for someone who would have this particular
> > >> itch to scratch, and who has a demonstrable need for this
> functionality.
> > >>
> > >> Work wise, it would have to include adding physical CPU hotplug
> support
> > >> to the arm64 kernel as a precondition, before worrying about doing it
> in
> > >> KVM.
> > >>
> > >> For KVM itself, particular area of interests would be:
> > >> - Making GICv3 redistributors magically appear in the IPA space
> > >> - Live resizing of GICv3 structures
> > >> - Dynamic allocation of MPIDR, and mapping with vcpu_id
> > >
> > > I have CPU topology description patches on the QEMU list now[*]. A next
> > > step for me is to this MPIDR work. I probably won't get to it until the
> > > end of August though.
> > >
> > > [*] http://lists.nongnu.org/archive/html/qemu-devel/2018-
> 07/msg01168.html
> > >
> > >>
> > >> This should keep someone busy for a good couple of weeks (give or
> take a
> > >> few months).
> > >
> > > :-)
> > >
> > >>
> > >> That being said, I'd rather see support in QEMU first, creating all
> the
> > >> vcpu/redistributors upfront, and signalling the hotplug event via the
> > >> virtual firmware. And then post some numbers to show that creating all
> > >> the vcpus upfront is not acceptable.
> > >
> > > I think the upfront allocation, allocating all possible cpus, but only
> > > activating all present cpus, was the planned approach. What were the
> > > concerns about that approach? Just vcpu memory overhead for too many
> > > overly ambitious VM configs?
> >
> > I don't have any ARM-specific concern about that, and I think this is
> > the right approach. It has the good property of not requiring much
> > change in the kernel (other than actually supporting CPU hotplug).
> for x86 we allocate VCPUs dynamically (both QEMU and KVM)
> CCing ppc/s390 folks as I don't recall how it's implemented there.
>
> but we do not delete vcpus in KVM after they were created
> (as it deemed to be too complicated), we are just deleting QEMU part
> of it and keep kvm's vcpu for reuse with future hotplug.
>

Same with PPC, we too dynamically create vcpus and during unplug keep the
KVM's vcpus for reuse.

Regards,
Bharata.

^ permalink raw reply	[flat|nested] 32+ messages in thread

* Re: VCPU hotplug on KVM/ARM
@ 2018-08-01  8:09                     ` Bharata B Rao
  0 siblings, 0 replies; 32+ messages in thread
From: Bharata B Rao @ 2018-08-01  8:09 UTC (permalink / raw)
  To: Igor Mammedov
  Cc: bthakur, Christoffer Dall, Andrew Jones, Christoffer Dall,
	Maran Wilson, Marc Zyngier, cohuck, david, qemu-devel,
	borntraeger, Christoffer Dall, qemu-arm, Peter Maydell, kvmarm,
	David Gibson

On Tue, Jul 31, 2018 at 3:57 PM, Igor Mammedov <imammedo@redhat.com> wrote:

> On Wed, 25 Jul 2018 14:07:12 +0100
> Marc Zyngier <marc.zyngier@arm.com> wrote:
>
> > On 25/07/18 13:28, Andrew Jones wrote:
> > > On Wed, Jul 25, 2018 at 11:40:54AM +0100, Marc Zyngier wrote:
> > >> On 24/07/18 19:35, Maran Wilson wrote:
> > >>> It's been a few months since this email thread died off. Has anyone
> > >>> started working on a potential solution that would allow VCPU
> hotplug on
> > >>> KVM/ARM ? Or is this a project that is still waiting for an owner
> who
> > >>> has the time and inclination to get started?
> > >>
> > >> This is typically a project for someone who would have this particular
> > >> itch to scratch, and who has a demonstrable need for this
> functionality.
> > >>
> > >> Work wise, it would have to include adding physical CPU hotplug
> support
> > >> to the arm64 kernel as a precondition, before worrying about doing it
> in
> > >> KVM.
> > >>
> > >> For KVM itself, particular area of interests would be:
> > >> - Making GICv3 redistributors magically appear in the IPA space
> > >> - Live resizing of GICv3 structures
> > >> - Dynamic allocation of MPIDR, and mapping with vcpu_id
> > >
> > > I have CPU topology description patches on the QEMU list now[*]. A next
> > > step for me is to this MPIDR work. I probably won't get to it until the
> > > end of August though.
> > >
> > > [*] http://lists.nongnu.org/archive/html/qemu-devel/2018-
> 07/msg01168.html
> > >
> > >>
> > >> This should keep someone busy for a good couple of weeks (give or
> take a
> > >> few months).
> > >
> > > :-)
> > >
> > >>
> > >> That being said, I'd rather see support in QEMU first, creating all
> the
> > >> vcpu/redistributors upfront, and signalling the hotplug event via the
> > >> virtual firmware. And then post some numbers to show that creating all
> > >> the vcpus upfront is not acceptable.
> > >
> > > I think the upfront allocation, allocating all possible cpus, but only
> > > activating all present cpus, was the planned approach. What were the
> > > concerns about that approach? Just vcpu memory overhead for too many
> > > overly ambitious VM configs?
> >
> > I don't have any ARM-specific concern about that, and I think this is
> > the right approach. It has the good property of not requiring much
> > change in the kernel (other than actually supporting CPU hotplug).
> for x86 we allocate VCPUs dynamically (both QEMU and KVM)
> CCing ppc/s390 folks as I don't recall how it's implemented there.
>
> but we do not delete vcpus in KVM after they were created
> (as it deemed to be too complicated), we are just deleting QEMU part
> of it and keep kvm's vcpu for reuse with future hotplug.
>

Same with PPC, we too dynamically create vcpus and during unplug keep the
KVM's vcpus for reuse.

Regards,
Bharata.

^ permalink raw reply	[flat|nested] 32+ messages in thread

end of thread, other threads:[~2018-08-01  8:09 UTC | newest]

Thread overview: 32+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-27  9:31 VCPU hotplug on KVM/ARM bthakur
2018-02-27 10:47 ` Christoffer Dall
2018-02-27 12:04   ` bthakur
2018-02-27 12:46     ` Christoffer Dall
2018-02-27 13:21       ` [Qemu-devel] " Andrew Jones
2018-02-27 13:21         ` Andrew Jones
2018-03-01  9:50         ` [Qemu-devel] " Igor Mammedov
2018-03-01  9:50           ` Igor Mammedov
2018-03-01 10:05           ` [Qemu-devel] [Qemu-arm] " Peter Maydell
2018-03-01 10:05             ` [Qemu-arm] [Qemu-devel] " Peter Maydell
2018-03-01 13:32             ` [Qemu-devel] [Qemu-arm] " David Hildenbrand
2018-03-01 13:32               ` David Hildenbrand
2018-03-07 12:47               ` Marc Zyngier
2018-03-07 12:47                 ` Marc Zyngier
2018-07-24 18:35         ` [Qemu-devel] " Maran Wilson
2018-07-24 18:35           ` Maran Wilson
2018-07-25  6:45           ` [Qemu-devel] " Igor Mammedov
2018-07-25  6:45             ` Igor Mammedov
2018-07-25 10:40           ` Marc Zyngier
2018-07-25 10:40             ` Marc Zyngier
2018-07-25 12:28             ` [Qemu-devel] " Andrew Jones
2018-07-25 12:28               ` Andrew Jones
2018-07-25 13:07               ` Marc Zyngier
2018-07-25 13:07                 ` Marc Zyngier
2018-07-25 17:26                 ` Maran Wilson
2018-07-25 17:26                   ` Maran Wilson
2018-07-31 10:27                 ` Igor Mammedov
2018-07-31 10:27                   ` Igor Mammedov
2018-07-31 10:57                   ` David Hildenbrand
2018-07-31 10:57                     ` David Hildenbrand
2018-08-01  8:09                   ` Bharata B Rao
2018-08-01  8:09                     ` Bharata B Rao

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.