[Qemu-devel] [PATCH 1/2] Fix real mode guest migration

[Qemu-devel] [PATCH 1/2] Fix real mode guest migration

[Orit Wasserman]

Older KVM versions save CS dpl value to an invalid value for real mode guests
(0x3). This patch detect this situation when loading CPU state and set all the
segments dpl to zero.
This will allow migration from older KVM on host without unrestricted guest
to hosts with restricted guest support.
For example migration from a Penryn host (with kernel 2.6.32) to
a Westmere host.

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
---
 target-i386/machine.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/target-i386/machine.c b/target-i386/machine.c
index 3659db9..7e95829 100644
--- a/target-i386/machine.c
+++ b/target-i386/machine.c
@@ -260,6 +260,24 @@ static int cpu_post_load(void *opaque, int version_id)
     CPUX86State *env = &cpu->env;
     int i;
 
+    /*
+      Real mode guest segments register DPL should be zero.
+      Older KVM version were setting it worngly.
+      Fixing it will allow live migration from such host that don't have
+      restricted guest support to an host with unrestricted guest support
+      (otherwise the migration will fail with invalid guest state
+      error).
+    */
+    if (!(env->cr[0] & CR0_PE_MASK) &&
+         (env->segs[R_CS].flags >> DESC_DPL_SHIFT & 3) != 0) {
+        env->segs[R_CS].flags &= ~(env->segs[R_CS].flags & DESC_DPL_MASK);
+        env->segs[R_DS].flags &= ~(env->segs[R_DS].flags & DESC_DPL_MASK);
+        env->segs[R_ES].flags &= ~(env->segs[R_ES].flags & DESC_DPL_MASK);
+        env->segs[R_FS].flags &= ~(env->segs[R_FS].flags & DESC_DPL_MASK);
+        env->segs[R_GS].flags &= ~(env->segs[R_GS].flags & DESC_DPL_MASK);
+        env->segs[R_SS].flags &= ~(env->segs[R_SS].flags & DESC_DPL_MASK);
+    }
+
     /* XXX: restore FPU round state */
     env->fpstt = (env->fpus_vmstate >> 11) & 7;
     env->fpus = env->fpus_vmstate & ~0x3800;
-- 
1.8.1.4

[Qemu-devel] [PATCH 2/2] Fix real mode guest segments dpl value in savevm

[Orit Wasserman]

Older KVM version put invalid value in the segments registers dpl field for
real mode guests (0x3).
This breaks migration from those hosts to hosts with unrestricted guest support.
We detect it by checking CS dpl value for real mode guest and fix the dpl values
of all the segment registers.

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
---
 target-i386/machine.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/target-i386/machine.c b/target-i386/machine.c
index 7e95829..4e39ca9 100644
--- a/target-i386/machine.c
+++ b/target-i386/machine.c
@@ -252,6 +252,24 @@ static void cpu_pre_save(void *opaque)
     }
 
     env->fpregs_format_vmstate = 0;
+
+    /*
+      Real mode guest segments register DPL should be zero.
+      Older KVM version were setting it worngly.
+      Fixing it will allow live migration to host with unrestricted guest
+      support (otherwise the migration will fail with invalid guest state
+      error).
+    */
+    if (!(env->cr[0] & CR0_PE_MASK) &&
+         (env->segs[R_CS].flags >> DESC_DPL_SHIFT & 3) != 0) {
+        env->segs[R_CS].flags &= ~(env->segs[R_CS].flags & DESC_DPL_MASK);
+        env->segs[R_DS].flags &= ~(env->segs[R_DS].flags & DESC_DPL_MASK);
+        env->segs[R_ES].flags &= ~(env->segs[R_ES].flags & DESC_DPL_MASK);
+        env->segs[R_FS].flags &= ~(env->segs[R_FS].flags & DESC_DPL_MASK);
+        env->segs[R_GS].flags &= ~(env->segs[R_GS].flags & DESC_DPL_MASK);
+        env->segs[R_SS].flags &= ~(env->segs[R_SS].flags & DESC_DPL_MASK);
+    }
+
 }
 
 static int cpu_post_load(void *opaque, int version_id)
-- 
1.8.1.4

Re: [Qemu-devel] [PATCH 1/2] Fix real mode guest migration

[Paolo Bonzini]

Il 22/07/2013 08:49, Orit Wasserman ha scritto:
> Older KVM versions save CS dpl value to an invalid value for real mode guests
> (0x3). This patch detect this situation when loading CPU state and set all the
> segments dpl to zero.
> This will allow migration from older KVM on host without unrestricted guest
> to hosts with restricted guest support.
> For example migration from a Penryn host (with kernel 2.6.32) to
> a Westmere host.
> 
> Signed-off-by: Orit Wasserman <owasserm@redhat.com>
> ---
>  target-i386/machine.c | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
> 
> diff --git a/target-i386/machine.c b/target-i386/machine.c
> index 3659db9..7e95829 100644
> --- a/target-i386/machine.c
> +++ b/target-i386/machine.c
> @@ -260,6 +260,24 @@ static int cpu_post_load(void *opaque, int version_id)
>      CPUX86State *env = &cpu->env;
>      int i;
>  
> +    /*
> +      Real mode guest segments register DPL should be zero.
> +      Older KVM version were setting it worngly.
> +      Fixing it will allow live migration from such host that don't have
> +      restricted guest support to an host with unrestricted guest support
> +      (otherwise the migration will fail with invalid guest state
> +      error).
> +    */

Coding standard asks for *s on every line.

As discussed offlist, I would prefer to have this in the kernel since
that's where the bug is.  Gleb disagrees.

We need to find a third person who mediates...  Anthony, Eduardo, what
do you think?

Paolo

> +    if (!(env->cr[0] & CR0_PE_MASK) &&
> +         (env->segs[R_CS].flags >> DESC_DPL_SHIFT & 3) != 0) {
> +        env->segs[R_CS].flags &= ~(env->segs[R_CS].flags & DESC_DPL_MASK);
> +        env->segs[R_DS].flags &= ~(env->segs[R_DS].flags & DESC_DPL_MASK);
> +        env->segs[R_ES].flags &= ~(env->segs[R_ES].flags & DESC_DPL_MASK);
> +        env->segs[R_FS].flags &= ~(env->segs[R_FS].flags & DESC_DPL_MASK);
> +        env->segs[R_GS].flags &= ~(env->segs[R_GS].flags & DESC_DPL_MASK);
> +        env->segs[R_SS].flags &= ~(env->segs[R_SS].flags & DESC_DPL_MASK);
> +    }
> +
>      /* XXX: restore FPU round state */
>      env->fpstt = (env->fpus_vmstate >> 11) & 7;
>      env->fpus = env->fpus_vmstate & ~0x3800;
> 

Re: [Qemu-devel] [PATCH 1/2] Fix real mode guest migration

[Gleb Natapov]

On Mon, Jul 22, 2013 at 11:49:25AM +0200, Paolo Bonzini wrote:
> Il 22/07/2013 08:49, Orit Wasserman ha scritto:
> > Older KVM versions save CS dpl value to an invalid value for real mode guests
> > (0x3). This patch detect this situation when loading CPU state and set all the
> > segments dpl to zero.
> > This will allow migration from older KVM on host without unrestricted guest
> > to hosts with restricted guest support.
> > For example migration from a Penryn host (with kernel 2.6.32) to
> > a Westmere host.
> > 
> > Signed-off-by: Orit Wasserman <owasserm@redhat.com>
> > ---
> >  target-i386/machine.c | 18 ++++++++++++++++++
> >  1 file changed, 18 insertions(+)
> > 
> > diff --git a/target-i386/machine.c b/target-i386/machine.c
> > index 3659db9..7e95829 100644
> > --- a/target-i386/machine.c
> > +++ b/target-i386/machine.c
> > @@ -260,6 +260,24 @@ static int cpu_post_load(void *opaque, int version_id)
> >      CPUX86State *env = &cpu->env;
> >      int i;
> >  
> > +    /*
> > +      Real mode guest segments register DPL should be zero.
> > +      Older KVM version were setting it worngly.
> > +      Fixing it will allow live migration from such host that don't have
> > +      restricted guest support to an host with unrestricted guest support
> > +      (otherwise the migration will fail with invalid guest state
> > +      error).
> > +    */
> 
> Coding standard asks for *s on every line.
> 
> As discussed offlist, I would prefer to have this in the kernel since
> that's where the bug is.  Gleb disagrees.
> 
This workaround also has to be in userspace if one day someone will
want to implement kvm->tcg migration (nothing impossible here). Also
userspace is the only place where the workaround can be isolated to the
migration code. All this, and the fact that no kernel upgrade is required
to propagate the fix speaks strongly in favor of this patches. For me,
the mere fact that the code is in userspace and no kernel changes
needed, but result is the same is enough.

> We need to find a third person who mediates...  Anthony, Eduardo, what
> do you think?
> 
> Paolo
> 
> > +    if (!(env->cr[0] & CR0_PE_MASK) &&
> > +         (env->segs[R_CS].flags >> DESC_DPL_SHIFT & 3) != 0) {
> > +        env->segs[R_CS].flags &= ~(env->segs[R_CS].flags & DESC_DPL_MASK);
> > +        env->segs[R_DS].flags &= ~(env->segs[R_DS].flags & DESC_DPL_MASK);
> > +        env->segs[R_ES].flags &= ~(env->segs[R_ES].flags & DESC_DPL_MASK);
> > +        env->segs[R_FS].flags &= ~(env->segs[R_FS].flags & DESC_DPL_MASK);
> > +        env->segs[R_GS].flags &= ~(env->segs[R_GS].flags & DESC_DPL_MASK);
> > +        env->segs[R_SS].flags &= ~(env->segs[R_SS].flags & DESC_DPL_MASK);
> > +    }
> > +
> >      /* XXX: restore FPU round state */
> >      env->fpstt = (env->fpus_vmstate >> 11) & 7;
> >      env->fpus = env->fpus_vmstate & ~0x3800;
> > 

--
			Gleb.

Re: [Qemu-devel] [PATCH 1/2] Fix real mode guest migration

[Orit Wasserman]

On 07/22/2013 12:49 PM, Paolo Bonzini wrote:
> Il 22/07/2013 08:49, Orit Wasserman ha scritto:
>> Older KVM versions save CS dpl value to an invalid value for real mode guests
>> (0x3). This patch detect this situation when loading CPU state and set all the
>> segments dpl to zero.
>> This will allow migration from older KVM on host without unrestricted guest
>> to hosts with restricted guest support.
>> For example migration from a Penryn host (with kernel 2.6.32) to
>> a Westmere host.
>>
>> Signed-off-by: Orit Wasserman <owasserm@redhat.com>
>> ---
>>  target-i386/machine.c | 18 ++++++++++++++++++
>>  1 file changed, 18 insertions(+)
>>
>> diff --git a/target-i386/machine.c b/target-i386/machine.c
>> index 3659db9..7e95829 100644
>> --- a/target-i386/machine.c
>> +++ b/target-i386/machine.c
>> @@ -260,6 +260,24 @@ static int cpu_post_load(void *opaque, int version_id)
>>      CPUX86State *env = &cpu->env;
>>      int i;
>>  
>> +    /*
>> +      Real mode guest segments register DPL should be zero.
>> +      Older KVM version were setting it worngly.
>> +      Fixing it will allow live migration from such host that don't have
>> +      restricted guest support to an host with unrestricted guest support
>> +      (otherwise the migration will fail with invalid guest state
>> +      error).
>> +    */
> 
> Coding standard asks for *s on every line.
I will fix it in v2.
By the way checkpatch.pl didn't send a warning about this it will be nice to add.

> 
> As discussed offlist, I would prefer to have this in the kernel since
> that's where the bug is.  Gleb disagrees.
> 

As this is a migration bug I prefer to fix it in the migration code only.
Fixing it in the kernel will effect all scenarios that load the segments,
in userspace it is only in load VM code.

Orit
> We need to find a third person who mediates...  Anthony, Eduardo, what
> do you think?
> 
> Paolo
> 
>> +    if (!(env->cr[0] & CR0_PE_MASK) &&
>> +         (env->segs[R_CS].flags >> DESC_DPL_SHIFT & 3) != 0) {
>> +        env->segs[R_CS].flags &= ~(env->segs[R_CS].flags & DESC_DPL_MASK);
>> +        env->segs[R_DS].flags &= ~(env->segs[R_DS].flags & DESC_DPL_MASK);
>> +        env->segs[R_ES].flags &= ~(env->segs[R_ES].flags & DESC_DPL_MASK);
>> +        env->segs[R_FS].flags &= ~(env->segs[R_FS].flags & DESC_DPL_MASK);
>> +        env->segs[R_GS].flags &= ~(env->segs[R_GS].flags & DESC_DPL_MASK);
>> +        env->segs[R_SS].flags &= ~(env->segs[R_SS].flags & DESC_DPL_MASK);
>> +    }
>> +
>>      /* XXX: restore FPU round state */
>>      env->fpstt = (env->fpus_vmstate >> 11) & 7;
>>      env->fpus = env->fpus_vmstate & ~0x3800;
>>
> 

Re: [Qemu-devel] [PATCH 1/2] Fix real mode guest migration

[Paolo Bonzini]

Il 22/07/2013 12:10, Orit Wasserman ha scritto:
> > As discussed offlist, I would prefer to have this in the kernel since
> > that's where the bug is.  Gleb disagrees.
> 
> As this is a migration bug I prefer to fix it in the migration code only.
> Fixing it in the kernel will effect all scenarios that load the segments,
> in userspace it is only in load VM code.

Ok, here's the third person.  Sorry for not thinking of you in the first
place. :)

Paolo

Re: [Qemu-devel] [PATCH 1/2] Fix real mode guest migration

[Andreas Färber]

Am 22.07.2013 11:49, schrieb Paolo Bonzini:
> Il 22/07/2013 08:49, Orit Wasserman ha scritto:
>> Older KVM versions save CS dpl value to an invalid value for real mode guests
>> (0x3). This patch detect this situation when loading CPU state and set all the
>> segments dpl to zero.
>> This will allow migration from older KVM on host without unrestricted guest
>> to hosts with restricted guest support.
>> For example migration from a Penryn host (with kernel 2.6.32) to
>> a Westmere host.
>>
>> Signed-off-by: Orit Wasserman <owasserm@redhat.com>
>> ---
>>  target-i386/machine.c | 18 ++++++++++++++++++
>>  1 file changed, 18 insertions(+)
>>
>> diff --git a/target-i386/machine.c b/target-i386/machine.c
>> index 3659db9..7e95829 100644
>> --- a/target-i386/machine.c
>> +++ b/target-i386/machine.c
>> @@ -260,6 +260,24 @@ static int cpu_post_load(void *opaque, int version_id)
>>      CPUX86State *env = &cpu->env;
>>      int i;
>>  
>> +    /*
>> +      Real mode guest segments register DPL should be zero.
>> +      Older KVM version were setting it worngly.
>> +      Fixing it will allow live migration from such host that don't have
>> +      restricted guest support to an host with unrestricted guest support
>> +      (otherwise the migration will fail with invalid guest state
>> +      error).
>> +    */
> 
> Coding standard asks for *s on every line.
> 
> As discussed offlist, I would prefer to have this in the kernel since
> that's where the bug is.  Gleb disagrees.
> 
> We need to find a third person who mediates...  Anthony, Eduardo, what
> do you think?

Having the code here does not look wrong to me, to enforce a consistent
state inside QEMU.

However I wonder what happens without this patch on Westmere? Might it
make sense to sanitize or at least "assert" (whatever the kernel
equivalent is ;)) in the ioctl setting X86CPU state to the vCPU that the
incoming values will be valid for the host CPU? And optionally in QEMU's
KVM code for the reverse direction, cpu_synchronize_state(), to cope
with older kernels?

Regards,
Andreas

>> +    if (!(env->cr[0] & CR0_PE_MASK) &&
>> +         (env->segs[R_CS].flags >> DESC_DPL_SHIFT & 3) != 0) {
>> +        env->segs[R_CS].flags &= ~(env->segs[R_CS].flags & DESC_DPL_MASK);
>> +        env->segs[R_DS].flags &= ~(env->segs[R_DS].flags & DESC_DPL_MASK);
>> +        env->segs[R_ES].flags &= ~(env->segs[R_ES].flags & DESC_DPL_MASK);
>> +        env->segs[R_FS].flags &= ~(env->segs[R_FS].flags & DESC_DPL_MASK);
>> +        env->segs[R_GS].flags &= ~(env->segs[R_GS].flags & DESC_DPL_MASK);
>> +        env->segs[R_SS].flags &= ~(env->segs[R_SS].flags & DESC_DPL_MASK);
>> +    }
>> +
>>      /* XXX: restore FPU round state */
>>      env->fpstt = (env->fpus_vmstate >> 11) & 7;
>>      env->fpus = env->fpus_vmstate & ~0x3800;

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 1/2] Fix real mode guest migration

[Juan Quintela]

Orit Wasserman <owasserm@redhat.com> wrote:
> Older KVM versions save CS dpl value to an invalid value for real mode guests
> (0x3). This patch detect this situation when loading CPU state and set all the
> segments dpl to zero.
> This will allow migration from older KVM on host without unrestricted guest
> to hosts with restricted guest support.
> For example migration from a Penryn host (with kernel 2.6.32) to
> a Westmere host.
>
> Signed-off-by: Orit Wasserman <owasserm@redhat.com>
> ---
>  target-i386/machine.c | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
>
> diff --git a/target-i386/machine.c b/target-i386/machine.c
> index 3659db9..7e95829 100644
> --- a/target-i386/machine.c
> +++ b/target-i386/machine.c
> @@ -260,6 +260,24 @@ static int cpu_post_load(void *opaque, int version_id)
>      CPUX86State *env = &cpu->env;
>      int i;
>  
> +    /*
> +      Real mode guest segments register DPL should be zero.
> +      Older KVM version were setting it worngly.

s/worngly/wrongly/

on both patches

> +      Fixing it will allow live migration from such host that don't have
> +      restricted guest support to an host with unrestricted guest support
> +      (otherwise the migration will fail with invalid guest state
> +      error).
> +    */
> +    if (!(env->cr[0] & CR0_PE_MASK) &&
> +         (env->segs[R_CS].flags >> DESC_DPL_SHIFT & 3) != 0) {
> +        env->segs[R_CS].flags &= ~(env->segs[R_CS].flags & DESC_DPL_MASK);
> +        env->segs[R_DS].flags &= ~(env->segs[R_DS].flags & DESC_DPL_MASK);
> +        env->segs[R_ES].flags &= ~(env->segs[R_ES].flags & DESC_DPL_MASK);
> +        env->segs[R_FS].flags &= ~(env->segs[R_FS].flags & DESC_DPL_MASK);
> +        env->segs[R_GS].flags &= ~(env->segs[R_GS].flags & DESC_DPL_MASK);
> +        env->segs[R_SS].flags &= ~(env->segs[R_SS].flags & DESC_DPL_MASK);
> +    }
> +
>      /* XXX: restore FPU round state */
>      env->fpstt = (env->fpus_vmstate >> 11) & 7;
>      env->fpus = env->fpus_vmstate & ~0x3800;

Re: [Qemu-devel] [PATCH 1/2] Fix real mode guest migration

[Gleb Natapov]

On Mon, Jul 22, 2013 at 12:33:31PM +0200, Andreas Färber wrote:
> Am 22.07.2013 11:49, schrieb Paolo Bonzini:
> > Il 22/07/2013 08:49, Orit Wasserman ha scritto:
> >> Older KVM versions save CS dpl value to an invalid value for real mode guests
> >> (0x3). This patch detect this situation when loading CPU state and set all the
> >> segments dpl to zero.
> >> This will allow migration from older KVM on host without unrestricted guest
> >> to hosts with restricted guest support.
> >> For example migration from a Penryn host (with kernel 2.6.32) to
> >> a Westmere host.
> >>
> >> Signed-off-by: Orit Wasserman <owasserm@redhat.com>
> >> ---
> >>  target-i386/machine.c | 18 ++++++++++++++++++
> >>  1 file changed, 18 insertions(+)
> >>
> >> diff --git a/target-i386/machine.c b/target-i386/machine.c
> >> index 3659db9..7e95829 100644
> >> --- a/target-i386/machine.c
> >> +++ b/target-i386/machine.c
> >> @@ -260,6 +260,24 @@ static int cpu_post_load(void *opaque, int version_id)
> >>      CPUX86State *env = &cpu->env;
> >>      int i;
> >>  
> >> +    /*
> >> +      Real mode guest segments register DPL should be zero.
> >> +      Older KVM version were setting it worngly.
> >> +      Fixing it will allow live migration from such host that don't have
> >> +      restricted guest support to an host with unrestricted guest support
> >> +      (otherwise the migration will fail with invalid guest state
> >> +      error).
> >> +    */
> > 
> > Coding standard asks for *s on every line.
> > 
> > As discussed offlist, I would prefer to have this in the kernel since
> > that's where the bug is.  Gleb disagrees.
> > 
> > We need to find a third person who mediates...  Anthony, Eduardo, what
> > do you think?
> 
> Having the code here does not look wrong to me, to enforce a consistent
> state inside QEMU.
> 
> However I wonder what happens without this patch on Westmere? Might it
Guest entry fails (but it is very hard to hit in practise since
migration has to happen while guest is in real mode).

> make sense to sanitize or at least "assert" (whatever the kernel
> equivalent is ;)) in the ioctl setting X86CPU state to the vCPU that the
> incoming values will be valid for the host CPU? And optionally in QEMU's
In a perfect world kvm would have return an error if wrong state is
loaded, but in reality the number of checks that have to be done is huge
and it is too late to add it now since this breaks ABI: old userspace
will start to get errors (and make it right from the first time is
practically impossible and each bug fix would have broke ABI too :)).

--
			Gleb.

Re: [Qemu-devel] [PATCH 1/2] Fix real mode guest migration

[Orit Wasserman]

On 07/22/2013 01:33 PM, Andreas Färber wrote:
> Am 22.07.2013 11:49, schrieb Paolo Bonzini:
>> Il 22/07/2013 08:49, Orit Wasserman ha scritto:
>>> Older KVM versions save CS dpl value to an invalid value for real mode guests
>>> (0x3). This patch detect this situation when loading CPU state and set all the
>>> segments dpl to zero.
>>> This will allow migration from older KVM on host without unrestricted guest
>>> to hosts with restricted guest support.
>>> For example migration from a Penryn host (with kernel 2.6.32) to
>>> a Westmere host.
>>>
>>> Signed-off-by: Orit Wasserman <owasserm@redhat.com>
>>> ---
>>>  target-i386/machine.c | 18 ++++++++++++++++++
>>>  1 file changed, 18 insertions(+)
>>>
>>> diff --git a/target-i386/machine.c b/target-i386/machine.c
>>> index 3659db9..7e95829 100644
>>> --- a/target-i386/machine.c
>>> +++ b/target-i386/machine.c
>>> @@ -260,6 +260,24 @@ static int cpu_post_load(void *opaque, int version_id)
>>>      CPUX86State *env = &cpu->env;
>>>      int i;
>>>  
>>> +    /*
>>> +      Real mode guest segments register DPL should be zero.
>>> +      Older KVM version were setting it worngly.
>>> +      Fixing it will allow live migration from such host that don't have
>>> +      restricted guest support to an host with unrestricted guest support
>>> +      (otherwise the migration will fail with invalid guest state
>>> +      error).
>>> +    */
>>
>> Coding standard asks for *s on every line.
>>
>> As discussed offlist, I would prefer to have this in the kernel since
>> that's where the bug is.  Gleb disagrees.
>>
>> We need to find a third person who mediates...  Anthony, Eduardo, what
>> do you think?
> 
> Having the code here does not look wrong to me, to enforce a consistent
> state inside QEMU.
> 
> However I wonder what happens without this patch on Westmere? Might it
> make sense to sanitize or at least "assert" (whatever the kernel
> equivalent is ;)) in the ioctl setting X86CPU state to the vCPU that the
> incoming values will be valid for the host CPU? And optionally in QEMU's
> KVM code for the reverse direction, cpu_synchronize_state(), to cope
> with older kernels?
> 

Without the patch we get "kvm: unhandled exit 80000021" error in incoming
migration or loadvm. This is a KVM error (kernel) which translates to invalid
guest state.This happens only in migration of a real mode guest.

The problem in fixing the values in cpu_synchronize_state is that the function
is called in many places in the code. 
As real mode code is very complex (Gleb can attest to that) I prefer a fix that
has a very limited scope like fixing it in the cpu_post_load and cpu_pre_save
function that are only used in savevm and live migration.

Orit

> Regards,
> Andreas
> 
>>> +    if (!(env->cr[0] & CR0_PE_MASK) &&
>>> +         (env->segs[R_CS].flags >> DESC_DPL_SHIFT & 3) != 0) {
>>> +        env->segs[R_CS].flags &= ~(env->segs[R_CS].flags & DESC_DPL_MASK);
>>> +        env->segs[R_DS].flags &= ~(env->segs[R_DS].flags & DESC_DPL_MASK);
>>> +        env->segs[R_ES].flags &= ~(env->segs[R_ES].flags & DESC_DPL_MASK);
>>> +        env->segs[R_FS].flags &= ~(env->segs[R_FS].flags & DESC_DPL_MASK);
>>> +        env->segs[R_GS].flags &= ~(env->segs[R_GS].flags & DESC_DPL_MASK);
>>> +        env->segs[R_SS].flags &= ~(env->segs[R_SS].flags & DESC_DPL_MASK);
>>> +    }
>>> +
>>>      /* XXX: restore FPU round state */
>>>      env->fpstt = (env->fpus_vmstate >> 11) & 7;
>>>      env->fpus = env->fpus_vmstate & ~0x3800;
> 

Re: [Qemu-devel] [PATCH 1/2] Fix real mode guest migration

[Juan Quintela]

Orit Wasserman <owasserm@redhat.com> wrote:
> On 07/22/2013 01:33 PM, Andreas Färber wrote:
>> Am 22.07.2013 11:49, schrieb Paolo Bonzini:
>>> Il 22/07/2013 08:49, Orit Wasserman ha scritto:
>>>> Older KVM versions save CS dpl value to an invalid value for real mode guests
>>>> (0x3). This patch detect this situation when loading CPU state and set all the
>>>> segments dpl to zero.
>>>> This will allow migration from older KVM on host without unrestricted guest
>>>> to hosts with restricted guest support.
>>>> For example migration from a Penryn host (with kernel 2.6.32) to
>>>> a Westmere host.
>>>>
>>>> Signed-off-by: Orit Wasserman <owasserm@redhat.com>
>>>> ---
>>>>  target-i386/machine.c | 18 ++++++++++++++++++
>>>>  1 file changed, 18 insertions(+)
>>>>
>>>> diff --git a/target-i386/machine.c b/target-i386/machine.c
>>>> index 3659db9..7e95829 100644
>>>> --- a/target-i386/machine.c
>>>> +++ b/target-i386/machine.c
>>>> @@ -260,6 +260,24 @@ static int cpu_post_load(void *opaque, int version_id)
>>>>      CPUX86State *env = &cpu->env;
>>>>      int i;
>>>>  
>>>> +    /*
>>>> +      Real mode guest segments register DPL should be zero.
>>>> +      Older KVM version were setting it worngly.
>>>> +      Fixing it will allow live migration from such host that don't have
>>>> +      restricted guest support to an host with unrestricted guest support
>>>> +      (otherwise the migration will fail with invalid guest state
>>>> +      error).
>>>> +    */
>>>
>>> Coding standard asks for *s on every line.
>>>
>>> As discussed offlist, I would prefer to have this in the kernel since
>>> that's where the bug is.  Gleb disagrees.
>>>
>>> We need to find a third person who mediates...  Anthony, Eduardo, what
>>> do you think?
>> 
>> Having the code here does not look wrong to me, to enforce a consistent
>> state inside QEMU.
>> 
>> However I wonder what happens without this patch on Westmere? Might it
>> make sense to sanitize or at least "assert" (whatever the kernel
>> equivalent is ;)) in the ioctl setting X86CPU state to the vCPU that the
>> incoming values will be valid for the host CPU? And optionally in QEMU's
>> KVM code for the reverse direction, cpu_synchronize_state(), to cope
>> with older kernels?
>> 
>
> Without the patch we get "kvm: unhandled exit 80000021" error in incoming
> migration or loadvm. This is a KVM error (kernel) which translates to invalid
> guest state.This happens only in migration of a real mode guest.
>
> The problem in fixing the values in cpu_synchronize_state is that the function
> is called in many places in the code. 
> As real mode code is very complex (Gleb can attest to that) I prefer a fix that
> has a very limited scope like fixing it in the cpu_post_load and cpu_pre_save
> function that are only used in savevm and live migration.

I fully agree with this approach. So far,  the problem only happens with
migration.  This fix the case if we have new qemu.  If we have old qemu,
we got the same problem that we had before.

And as Gleb said,  checking for all possible problems on kvm is
imposible,  as they are too many,  and we would break abi.

So,  I preffer this approach,  for what is worth.

Later,  Juan.

Re: [Qemu-devel] [PATCH 1/2] Fix real mode guest migration

[Eduardo Habkost]

On Mon, Jul 22, 2013 at 12:14:25PM +0200, Paolo Bonzini wrote:
> Il 22/07/2013 12:10, Orit Wasserman ha scritto:
> > > As discussed offlist, I would prefer to have this in the kernel since
> > > that's where the bug is.  Gleb disagrees.
> > 
> > As this is a migration bug I prefer to fix it in the migration code only.
> > Fixing it in the kernel will effect all scenarios that load the segments,
> > in userspace it is only in load VM code.
> 
> Ok, here's the third person.  Sorry for not thinking of you in the first
> place. :)

I'm coming late to the discussion, but: I fullly agree with Orit and
Gleb, especially about patch 1/2.

Patch 2/2 is not strictly necessary if the destination host kernel gets
fixed (latest kernels are already fixed), but it is nice to have it, to
not require a kernel upgrade.

-- 
Eduardo

Re: [Qemu-devel] [PATCH 1/2] Fix real mode guest migration

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 886 bytes --]

On 07/22/2013 12:49 AM, Orit Wasserman wrote:
> Older KVM versions save CS dpl value to an invalid value for real mode guests
> (0x3). This patch detect this situation when loading CPU state and set all the
> segments dpl to zero.
> This will allow migration from older KVM on host without unrestricted guest
> to hosts with restricted guest support.
> For example migration from a Penryn host (with kernel 2.6.32) to
> a Westmere host.

> +    /*
> +      Real mode guest segments register DPL should be zero.
> +      Older KVM version were setting it worngly.

s/worngly/wrongly/

> +      Fixing it will allow live migration from such host that don't have
> +      restricted guest support to an host with unrestricted guest support

s/an host/a host/

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 621 bytes --]

Re: [Qemu-devel] [PATCH 1/2] Fix real mode guest migration

[Anthony Liguori]

Paolo Bonzini <pbonzini@redhat.com> writes:

> Il 22/07/2013 08:49, Orit Wasserman ha scritto:
>> Older KVM versions save CS dpl value to an invalid value for real mode guests
>> (0x3). This patch detect this situation when loading CPU state and set all the
>> segments dpl to zero.
>> This will allow migration from older KVM on host without unrestricted guest
>> to hosts with restricted guest support.
>> For example migration from a Penryn host (with kernel 2.6.32) to
>> a Westmere host.
>> 
>> Signed-off-by: Orit Wasserman <owasserm@redhat.com>
>> ---
>>  target-i386/machine.c | 18 ++++++++++++++++++
>>  1 file changed, 18 insertions(+)
>> 
>> diff --git a/target-i386/machine.c b/target-i386/machine.c
>> index 3659db9..7e95829 100644
>> --- a/target-i386/machine.c
>> +++ b/target-i386/machine.c
>> @@ -260,6 +260,24 @@ static int cpu_post_load(void *opaque, int version_id)
>>      CPUX86State *env = &cpu->env;
>>      int i;
>>  
>> +    /*
>> +      Real mode guest segments register DPL should be zero.
>> +      Older KVM version were setting it worngly.
>> +      Fixing it will allow live migration from such host that don't have
>> +      restricted guest support to an host with unrestricted guest support
>> +      (otherwise the migration will fail with invalid guest state
>> +      error).
>> +    */
>
> Coding standard asks for *s on every line.
>
> As discussed offlist, I would prefer to have this in the kernel since
> that's where the bug is.  Gleb disagrees.
>
> We need to find a third person who mediates...  Anthony, Eduardo, what
> do you think?

I can rationalize the post_load part.  We may get garbage state via live
migration, we fix it up.

The pre_save part troubles me though. Is the kernel actively returning
invalid space to userspace?  If so, this needs to be fixed.  That's
clearly a bug.

Or is this expected to work around an old kernel?  If so, is the kernel
advertising that it now handles this state correctly via a capability?
If not, it probably should.

Regards,

Anthony Liguori

>
> Paolo
>
>> +    if (!(env->cr[0] & CR0_PE_MASK) &&
>> +         (env->segs[R_CS].flags >> DESC_DPL_SHIFT & 3) != 0) {
>> +        env->segs[R_CS].flags &= ~(env->segs[R_CS].flags & DESC_DPL_MASK);
>> +        env->segs[R_DS].flags &= ~(env->segs[R_DS].flags & DESC_DPL_MASK);
>> +        env->segs[R_ES].flags &= ~(env->segs[R_ES].flags & DESC_DPL_MASK);
>> +        env->segs[R_FS].flags &= ~(env->segs[R_FS].flags & DESC_DPL_MASK);
>> +        env->segs[R_GS].flags &= ~(env->segs[R_GS].flags & DESC_DPL_MASK);
>> +        env->segs[R_SS].flags &= ~(env->segs[R_SS].flags & DESC_DPL_MASK);
>> +    }
>> +
>>      /* XXX: restore FPU round state */
>>      env->fpstt = (env->fpus_vmstate >> 11) & 7;
>>      env->fpus = env->fpus_vmstate & ~0x3800;
>> 

Re: [Qemu-devel] [PATCH 1/2] Fix real mode guest migration

[Gleb Natapov]

On Mon, Jul 22, 2013 at 12:50:23PM -0500, Anthony Liguori wrote:
> Paolo Bonzini <pbonzini@redhat.com> writes:
> 
> > Il 22/07/2013 08:49, Orit Wasserman ha scritto:
> >> Older KVM versions save CS dpl value to an invalid value for real mode guests
> >> (0x3). This patch detect this situation when loading CPU state and set all the
> >> segments dpl to zero.
> >> This will allow migration from older KVM on host without unrestricted guest
> >> to hosts with restricted guest support.
> >> For example migration from a Penryn host (with kernel 2.6.32) to
> >> a Westmere host.
> >> 
> >> Signed-off-by: Orit Wasserman <owasserm@redhat.com>
> >> ---
> >>  target-i386/machine.c | 18 ++++++++++++++++++
> >>  1 file changed, 18 insertions(+)
> >> 
> >> diff --git a/target-i386/machine.c b/target-i386/machine.c
> >> index 3659db9..7e95829 100644
> >> --- a/target-i386/machine.c
> >> +++ b/target-i386/machine.c
> >> @@ -260,6 +260,24 @@ static int cpu_post_load(void *opaque, int version_id)
> >>      CPUX86State *env = &cpu->env;
> >>      int i;
> >>  
> >> +    /*
> >> +      Real mode guest segments register DPL should be zero.
> >> +      Older KVM version were setting it worngly.
> >> +      Fixing it will allow live migration from such host that don't have
> >> +      restricted guest support to an host with unrestricted guest support
> >> +      (otherwise the migration will fail with invalid guest state
> >> +      error).
> >> +    */
> >
> > Coding standard asks for *s on every line.
> >
> > As discussed offlist, I would prefer to have this in the kernel since
> > that's where the bug is.  Gleb disagrees.
> >
> > We need to find a third person who mediates...  Anthony, Eduardo, what
> > do you think?
> 
> I can rationalize the post_load part.  We may get garbage state via live
> migration, we fix it up.
> 
> The pre_save part troubles me though. Is the kernel actively returning
> invalid space to userspace?  If so, this needs to be fixed.  That's
> clearly a bug.
> 
It is fixed in recent kernels.

> Or is this expected to work around an old kernel?  If so, is the kernel
> advertising that it now handles this state correctly via a capability?
> If not, it probably should.
> 
It is to late to do it now, but the if() here identifies the incorrect
state very accurately.

> Regards,
> 
> Anthony Liguori
> 
> >
> > Paolo
> >
> >> +    if (!(env->cr[0] & CR0_PE_MASK) &&
> >> +         (env->segs[R_CS].flags >> DESC_DPL_SHIFT & 3) != 0) {
> >> +        env->segs[R_CS].flags &= ~(env->segs[R_CS].flags & DESC_DPL_MASK);
> >> +        env->segs[R_DS].flags &= ~(env->segs[R_DS].flags & DESC_DPL_MASK);
> >> +        env->segs[R_ES].flags &= ~(env->segs[R_ES].flags & DESC_DPL_MASK);
> >> +        env->segs[R_FS].flags &= ~(env->segs[R_FS].flags & DESC_DPL_MASK);
> >> +        env->segs[R_GS].flags &= ~(env->segs[R_GS].flags & DESC_DPL_MASK);
> >> +        env->segs[R_SS].flags &= ~(env->segs[R_SS].flags & DESC_DPL_MASK);
> >> +    }
> >> +
> >>      /* XXX: restore FPU round state */
> >>      env->fpstt = (env->fpus_vmstate >> 11) & 7;
> >>      env->fpus = env->fpus_vmstate & ~0x3800;
> >> 
> 

--
			Gleb.