KVM Live migration with GICv3

KVM Live migration with GICv3

[Vijay Kilari]

Hi,

   I have prototyped Live migration with GICv3.
For this I have made following changes

1) Save and Restore of GICv3 registers in QEMU.
    - For GICv2, QEMU is saving/restoring GICD, GICC  registers. For GICv3,
     we have to save/restore GICD, GICR and ICC registers.
     However ICC registers are system registers which cannot be
accessed @ EL0 level (SRE=1). So these ICC registers should be
accessed as mmio registers by QEMU, for this we have to add ioctl to
access ICC @ EL1 level similar to GICC registers of GICv2.

2) KVM ioctls in kernel provides only 32-bit register access to GIC
registers, where
   as some registers in GICD/GICR requires 64-bit register access. I
propose to use mmio.flag to specify 32/64 bit access.

3) KVM ioctls to access ICC registers for GICv3

Please provide your initial feedback. Let me know if some of this
issues are already fixed

I am attending KVM-forum next week @ Seattle. We can discuss there as well.

Regards
Vijay

RE: KVM Live migration with GICv3

[Pavel Fedin]

 Hello!

>    I have prototyped Live migration with GICv3.

 This is interesting of course, i will need this for my project too and i was also going to take a look at this later.

> For this I have made following changes
> 
> 1) Save and Restore of GICv3 registers in QEMU.
>     - For GICv2, QEMU is saving/restoring GICD, GICC  registers. For GICv3,
>      we have to save/restore GICD, GICR and ICC registers.
>      However ICC registers are system registers which cannot be
> accessed @ EL0 level (SRE=1). So these ICC registers should be
> accessed as mmio registers by QEMU, for this we have to add ioctl to
> access ICC @ EL1 level similar to GICC registers of GICv2.

 Sorry, your description is a bit fuzzy. What ioctl's did you have to add and why?
 The kernel already has KVM_GET_ONE_REG and KVM_SET_ONE_REG ioctl's. And they can be used to access anything inside the VM, including system registers. I believe you would need only to implement handlers for GICv3 registers.

> Please provide your initial feedback.

 Would be nice to take a look at patches.

> I am attending KVM-forum next week @ Seattle. We can discuss there as well.

 I would love to go there, and my company wanted to send me there. But it appears to be extremely difficult for a tech specialist from Russia to get a US visa. They are still considering me and the time is running out, so i'm unlikely to attend, unfortunately.

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

Re: KVM Live migration with GICv3

[Vijay Kilari]

On Tue, Aug 11, 2015 at 12:11 PM, Pavel Fedin <p.fedin@samsung.com> wrote:
>  Hello!
>
>>    I have prototyped Live migration with GICv3.
>
>  This is interesting of course, i will need this for my project too and i was also going to take a look at this later.
>
>> For this I have made following changes
>>
>> 1) Save and Restore of GICv3 registers in QEMU.
>>     - For GICv2, QEMU is saving/restoring GICD, GICC  registers. For GICv3,
>>      we have to save/restore GICD, GICR and ICC registers.
>>      However ICC registers are system registers which cannot be
>> accessed @ EL0 level (SRE=1). So these ICC registers should be
>> accessed as mmio registers by QEMU, for this we have to add ioctl to
>> access ICC @ EL1 level similar to GICC registers of GICv2.
>
>  Sorry, your description is a bit fuzzy. What ioctl's did you have to add and why?
>  The kernel already has KVM_GET_ONE_REG and KVM_SET_ONE_REG ioctl's. And they can be used to access anything inside the VM, including system registers. I believe you would need only to implement handlers for GICv3 registers.

I mean CPU interface registers GICC* defined for vgicv2 under
"vgic_cpu_ranges[]" in
virt/kvm/arm/vgic-v2-emul.c needs to be saved and restored for vgicv3 as well.
So to  access vgic_cpu_ranges[] for vgicv3, we need to register device
for CPU registers space
similar to gicv2 in QEMU for gicv3.

-----------------------------------------------------------------------------------------
    memory_region_init_reservation(&s->cpuiomem[0], OBJECT(s),
                                   "kvm-gic_cpu", 0x1000);
    sysbus_init_mmio(sbd, &s->cpuiomem[0]);
    kvm_arm_register_device(&s->cpuiomem[0],
                            (KVM_ARM_DEVICE_VGIC_V3 << KVM_ARM_DEVICE_ID_SHIFT)
                            | KVM_VGIC_V3_ADDR_TYPE_CPU,
                            KVM_DEV_ARM_VGIC_GRP_ADDR,
                            KVM_VGIC_V3_ADDR_TYPE_CPU,
                            s->dev_fd);
-------------------------------------------------------------------------------------------

Anyway, I will send RFC patches.

>
>> Please provide your initial feedback.
>
>  Would be nice to take a look at patches.
>
>> I am attending KVM-forum next week @ Seattle. We can discuss there as well.
>
>  I would love to go there, and my company wanted to send me there. But it appears to be extremely difficult for a tech specialist from Russia to get a US visa. They are still considering me and the time is running out, so i'm unlikely to attend, unfortunately.
>
> Kind regards,
> Pavel Fedin
> Expert Engineer
> Samsung Electronics Research center Russia
>
>

RE: KVM Live migration with GICv3

[Pavel Fedin]

 Hello!

> I mean CPU interface registers GICC* defined for vgicv2 under
> "vgic_cpu_ranges[]" in
> virt/kvm/arm/vgic-v2-emul.c needs to be saved and restored for vgicv3 as well.
> So to  access vgic_cpu_ranges[] for vgicv3, we need to register device
> for CPU registers space
> similar to gicv2 in QEMU for gicv3.

 This thing is optional and can be missing. It is for backwards compatibility only, it allows to run GICv2 guests on GICv3. You should not use this API for accessing system registers.

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

Re: KVM Live migration with GICv3

[Marc Zyngier]

On 11/08/15 16:24, Pavel Fedin wrote:
>  Hello!
> 
>> I mean CPU interface registers GICC* defined for vgicv2 under
>> "vgic_cpu_ranges[]" in
>> virt/kvm/arm/vgic-v2-emul.c needs to be saved and restored for vgicv3 as well.
>> So to  access vgic_cpu_ranges[] for vgicv3, we need to register device
>> for CPU registers space
>> similar to gicv2 in QEMU for gicv3.
> 
> This thing is optional and can be missing. It is for backwards
> compatibility only, it allows to run GICv2 guests on GICv3. You
> should not use this API for accessing system registers.

Even more than that.

The KVM emulation of GICv3 doesn't have *any* GICv2 compatibility. And
that's on purpose. So trying to save the GICC registers should give you
an error, because these registers *do not exist* as far as a guest is
concerned.

If you plan to do anything for GICv3, you should only deal with with the
system register version of the CPU interface.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: KVM Live migration with GICv3

[Peter Maydell]

On 12 August 2015 at 17:17, Marc Zyngier <marc.zyngier@arm.com> wrote:
> If you plan to do anything for GICv3, you should only deal with with the
> system register version of the CPU interface.

Also, please make sure that you think about (and in terms of)
the underlying state inside the GIC, which is closely
related to but not necessarily identical to "contents of
all the registers". Migration cares about saving and restoring
state.

-- PMM

Re: KVM Live migration with GICv3

[Marc Zyngier]

On 11/08/15 06:25, Vijay Kilari wrote:
> Hi,
> 
>    I have prototyped Live migration with GICv3.
> For this I have made following changes
> 
> 1) Save and Restore of GICv3 registers in QEMU.
>     - For GICv2, QEMU is saving/restoring GICD, GICC  registers. For GICv3,
>      we have to save/restore GICD, GICR and ICC registers.
>      However ICC registers are system registers which cannot be
> accessed @ EL0 level (SRE=1). So these ICC registers should be
> accessed as mmio registers by QEMU, for this we have to add ioctl to
> access ICC @ EL1 level similar to GICC registers of GICv2.

I've already replied to this. The ICC_* registers *must* be exposed as
system registers. There is no GICC_* state to save. And I don't get your
EL0 access thing.

> 
> 2) KVM ioctls in kernel provides only 32-bit register access to GIC
> registers, where
>    as some registers in GICD/GICR requires 64-bit register access. I
> propose to use mmio.flag to specify 32/64 bit access.

I don't believe the MMIO registers *require* 64bit access. The spec says:

"For the GITS_*, GICD_* and GICR_* registers, the upper 32 bits and the
lower 32 bits can be accessed independently, unless the register
requires a 64 bit access."

and as far as I can see, no register mandates a 64bit access.

> 
> 3) KVM ioctls to access ICC registers for GICv3
> 
> Please provide your initial feedback. Let me know if some of this
> issues are already fixed
> 
> I am attending KVM-forum next week @ Seattle. We can discuss there as well.

Feel free to come and talk to us.

	M.
-- 
Jazz is not dead. It just smells funny...

RE: KVM Live migration with GICv3

[Pavel Fedin]

 Hello! Where are you? I'd like to take a look at your code.

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia


> -----Original Message-----
> From: Vijay Kilari [mailto:vijay.kilari@gmail.com]
> Sent: Tuesday, August 11, 2015 6:16 PM
> To: Pavel Fedin
> Cc: kvmarm@lists.cs.columbia.edu; Christoffer Dall; Alex Bennée
> Subject: Re: KVM Live migration with GICv3
> 
> On Tue, Aug 11, 2015 at 12:11 PM, Pavel Fedin <p.fedin@samsung.com> wrote:
> >  Hello!
> >
> >>    I have prototyped Live migration with GICv3.
> >
> >  This is interesting of course, i will need this for my project too and i was also going to take a look at
> this later.
> >
> >> For this I have made following changes
> >>
> >> 1) Save and Restore of GICv3 registers in QEMU.
> >>     - For GICv2, QEMU is saving/restoring GICD, GICC  registers. For GICv3,
> >>      we have to save/restore GICD, GICR and ICC registers.
> >>      However ICC registers are system registers which cannot be
> >> accessed @ EL0 level (SRE=1). So these ICC registers should be
> >> accessed as mmio registers by QEMU, for this we have to add ioctl to
> >> access ICC @ EL1 level similar to GICC registers of GICv2.
> >
> >  Sorry, your description is a bit fuzzy. What ioctl's did you have to add and why?
> >  The kernel already has KVM_GET_ONE_REG and KVM_SET_ONE_REG ioctl's. And they can be used
> to access anything inside the VM, including system registers. I believe you would need only to
> implement handlers for GICv3 registers.
> 
> I mean CPU interface registers GICC* defined for vgicv2 under
> "vgic_cpu_ranges[]" in
> virt/kvm/arm/vgic-v2-emul.c needs to be saved and restored for vgicv3 as well.
> So to  access vgic_cpu_ranges[] for vgicv3, we need to register device
> for CPU registers space
> similar to gicv2 in QEMU for gicv3.
> 
> -----------------------------------------------------------------------------------------
>     memory_region_init_reservation(&s->cpuiomem[0], OBJECT(s),
>                                    "kvm-gic_cpu", 0x1000);
>     sysbus_init_mmio(sbd, &s->cpuiomem[0]);
>     kvm_arm_register_device(&s->cpuiomem[0],
>                             (KVM_ARM_DEVICE_VGIC_V3 << KVM_ARM_DEVICE_ID_SHIFT)
>                             | KVM_VGIC_V3_ADDR_TYPE_CPU,
>                             KVM_DEV_ARM_VGIC_GRP_ADDR,
>                             KVM_VGIC_V3_ADDR_TYPE_CPU,
>                             s->dev_fd);
> -------------------------------------------------------------------------------------------
> 
> Anyway, I will send RFC patches.
> 
> >
> >> Please provide your initial feedback.
> >
> >  Would be nice to take a look at patches.
> >
> >> I am attending KVM-forum next week @ Seattle. We can discuss there as well.
> >
> >  I would love to go there, and my company wanted to send me there. But it appears to be extremely
> difficult for a tech specialist from Russia to get a US visa. They are still considering me and the time is
> running out, so i'm unlikely to attend, unfortunately.
> >
> > Kind regards,
> > Pavel Fedin
> > Expert Engineer
> > Samsung Electronics Research center Russia
> >
> >

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM Live migration with GICv3

[Vijay Kilari]

On Tue, Aug 25, 2015 at 4:57 PM, Pavel Fedin <p.fedin@samsung.com> wrote:
>  Hello! Where are you? I'd like to take a look at your code.

   Sorry,  I am occupied with other tasks for next 2 weeks, After that
I will look into this.

>
> Kind regards,
> Pavel Fedin
> Expert Engineer
> Samsung Electronics Research center Russia
>
>
>> -----Original Message-----
>> From: Vijay Kilari [mailto:vijay.kilari@gmail.com]
>> Sent: Tuesday, August 11, 2015 6:16 PM
>> To: Pavel Fedin
>> Cc: kvmarm@lists.cs.columbia.edu; Christoffer Dall; Alex Bennée
>> Subject: Re: KVM Live migration with GICv3
>>
>> On Tue, Aug 11, 2015 at 12:11 PM, Pavel Fedin <p.fedin@samsung.com> wrote:
>> >  Hello!
>> >
>> >>    I have prototyped Live migration with GICv3.
>> >
>> >  This is interesting of course, i will need this for my project too and i was also going to take a look at
>> this later.
>> >
>> >> For this I have made following changes
>> >>
>> >> 1) Save and Restore of GICv3 registers in QEMU.
>> >>     - For GICv2, QEMU is saving/restoring GICD, GICC  registers. For GICv3,
>> >>      we have to save/restore GICD, GICR and ICC registers.
>> >>      However ICC registers are system registers which cannot be
>> >> accessed @ EL0 level (SRE=1). So these ICC registers should be
>> >> accessed as mmio registers by QEMU, for this we have to add ioctl to
>> >> access ICC @ EL1 level similar to GICC registers of GICv2.
>> >
>> >  Sorry, your description is a bit fuzzy. What ioctl's did you have to add and why?
>> >  The kernel already has KVM_GET_ONE_REG and KVM_SET_ONE_REG ioctl's. And they can be used
>> to access anything inside the VM, including system registers. I believe you would need only to
>> implement handlers for GICv3 registers.
>>
>> I mean CPU interface registers GICC* defined for vgicv2 under
>> "vgic_cpu_ranges[]" in
>> virt/kvm/arm/vgic-v2-emul.c needs to be saved and restored for vgicv3 as well.
>> So to  access vgic_cpu_ranges[] for vgicv3, we need to register device
>> for CPU registers space
>> similar to gicv2 in QEMU for gicv3.
>>
>> -----------------------------------------------------------------------------------------
>>     memory_region_init_reservation(&s->cpuiomem[0], OBJECT(s),
>>                                    "kvm-gic_cpu", 0x1000);
>>     sysbus_init_mmio(sbd, &s->cpuiomem[0]);
>>     kvm_arm_register_device(&s->cpuiomem[0],
>>                             (KVM_ARM_DEVICE_VGIC_V3 << KVM_ARM_DEVICE_ID_SHIFT)
>>                             | KVM_VGIC_V3_ADDR_TYPE_CPU,
>>                             KVM_DEV_ARM_VGIC_GRP_ADDR,
>>                             KVM_VGIC_V3_ADDR_TYPE_CPU,
>>                             s->dev_fd);
>> -------------------------------------------------------------------------------------------
>>
>> Anyway, I will send RFC patches.
>>
>> >
>> >> Please provide your initial feedback.
>> >
>> >  Would be nice to take a look at patches.
>> >
>> >> I am attending KVM-forum next week @ Seattle. We can discuss there as well.
>> >
>> >  I would love to go there, and my company wanted to send me there. But it appears to be extremely
>> difficult for a tech specialist from Russia to get a US visa. They are still considering me and the time is
>> running out, so i'm unlikely to attend, unfortunately.
>> >
>> > Kind regards,
>> > Pavel Fedin
>> > Expert Engineer
>> > Samsung Electronics Research center Russia
>> >
>> >
>
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm