[Qemu-devel] [PATCH] pseries: Fix compat_pvr on reset

[Qemu-devel] [PATCH] pseries: Fix compat_pvr on reset

[Laurent Vivier]

If we a migrate P8 machine to a P9 machine, the migration fails on
destination with:

  error while loading state for instance 0x1 of device 'cpu'
  load of migration failed: Operation not permitted

This is caused because the compat_pvr field is only present for the first
CPU.
Originally, spapr_machine_reset() calls ppc_set_compat() to set the value
max_compat_pvr for the first cpu and this was propagated to all CPUs by
spapr_cpu_reset().  Now, as spapr_cpu_reset() is called before that, the
value is not propagated to all CPUs and the migration fails.

To fix that, propagate the new value to all CPUs in spapr_machine_reset().

Fixes: 25c9780d38d4 ("spapr: Reset CAS & IRQ subsystem after devices")
Signed-off-by: Laurent Vivier <lvivier@redhat.com>
---
 hw/ppc/spapr.c          | 8 +++++++-
 hw/ppc/spapr_cpu_core.c | 2 ++
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index baedadf20b8c..d063312a3b2a 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -1752,7 +1752,13 @@ static void spapr_machine_reset(MachineState *machine)
         spapr_ovec_cleanup(spapr->ov5_cas);
         spapr->ov5_cas = spapr_ovec_new();
 
-        ppc_set_compat(first_ppc_cpu, spapr->max_compat_pvr, &error_fatal);
+        /*
+         * reset compat_pvr for all CPUs
+         * as qemu_devices_reset() is called before this,
+         * it can't be propagated by spapr_cpu_reset()
+         * from the first CPU to all the others
+         */
+        ppc_set_compat_all(spapr->max_compat_pvr, &error_fatal);
     }
 
     /*
diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
index bf47fbdf6f7f..45e2f2747ffc 100644
--- a/hw/ppc/spapr_cpu_core.c
+++ b/hw/ppc/spapr_cpu_core.c
@@ -43,6 +43,8 @@ static void spapr_cpu_reset(void *opaque)
 
     /* Set compatibility mode to match the boot CPU, which was either set
      * by the machine reset code or by CAS. This should never fail.
+     * At startup the value is already set for all the CPUs
+     * but we need this when we hotplug a new CPU
      */
     ppc_set_compat(cpu, POWERPC_CPU(first_cpu)->compat_pvr, &error_abort);
 
-- 
2.21.0

Re: [Qemu-devel] [PATCH] pseries: Fix compat_pvr on reset

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2597 bytes --]

On Mon, Aug 26, 2019 at 11:08:12AM +0200, Laurent Vivier wrote:
> If we a migrate P8 machine to a P9 machine, the migration fails on
> destination with:
> 
>   error while loading state for instance 0x1 of device 'cpu'
>   load of migration failed: Operation not permitted
> 
> This is caused because the compat_pvr field is only present for the first
> CPU.
> Originally, spapr_machine_reset() calls ppc_set_compat() to set the value
> max_compat_pvr for the first cpu and this was propagated to all CPUs by
> spapr_cpu_reset().  Now, as spapr_cpu_reset() is called before that, the
> value is not propagated to all CPUs and the migration fails.
> 
> To fix that, propagate the new value to all CPUs in spapr_machine_reset().
> 
> Fixes: 25c9780d38d4 ("spapr: Reset CAS & IRQ subsystem after devices")
> Signed-off-by: Laurent Vivier <lvivier@redhat.com>

Applied to ppc-for-4.2, thanks.

> ---
>  hw/ppc/spapr.c          | 8 +++++++-
>  hw/ppc/spapr_cpu_core.c | 2 ++
>  2 files changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index baedadf20b8c..d063312a3b2a 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -1752,7 +1752,13 @@ static void spapr_machine_reset(MachineState *machine)
>          spapr_ovec_cleanup(spapr->ov5_cas);
>          spapr->ov5_cas = spapr_ovec_new();
>  
> -        ppc_set_compat(first_ppc_cpu, spapr->max_compat_pvr, &error_fatal);
> +        /*
> +         * reset compat_pvr for all CPUs
> +         * as qemu_devices_reset() is called before this,
> +         * it can't be propagated by spapr_cpu_reset()
> +         * from the first CPU to all the others
> +         */
> +        ppc_set_compat_all(spapr->max_compat_pvr, &error_fatal);
>      }
>  
>      /*
> diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
> index bf47fbdf6f7f..45e2f2747ffc 100644
> --- a/hw/ppc/spapr_cpu_core.c
> +++ b/hw/ppc/spapr_cpu_core.c
> @@ -43,6 +43,8 @@ static void spapr_cpu_reset(void *opaque)
>  
>      /* Set compatibility mode to match the boot CPU, which was either set
>       * by the machine reset code or by CAS. This should never fail.
> +     * At startup the value is already set for all the CPUs
> +     * but we need this when we hotplug a new CPU
>       */
>      ppc_set_compat(cpu, POWERPC_CPU(first_cpu)->compat_pvr, &error_abort);
>  

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] [PATCH] pseries: Fix compat_pvr on reset

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2597 bytes --]

On Mon, Aug 26, 2019 at 11:08:12AM +0200, Laurent Vivier wrote:
> If we a migrate P8 machine to a P9 machine, the migration fails on
> destination with:
> 
>   error while loading state for instance 0x1 of device 'cpu'
>   load of migration failed: Operation not permitted
> 
> This is caused because the compat_pvr field is only present for the first
> CPU.
> Originally, spapr_machine_reset() calls ppc_set_compat() to set the value
> max_compat_pvr for the first cpu and this was propagated to all CPUs by
> spapr_cpu_reset().  Now, as spapr_cpu_reset() is called before that, the
> value is not propagated to all CPUs and the migration fails.
> 
> To fix that, propagate the new value to all CPUs in spapr_machine_reset().
> 
> Fixes: 25c9780d38d4 ("spapr: Reset CAS & IRQ subsystem after devices")
> Signed-off-by: Laurent Vivier <lvivier@redhat.com>

Applied to ppc-for-4.2, thanks.

> ---
>  hw/ppc/spapr.c          | 8 +++++++-
>  hw/ppc/spapr_cpu_core.c | 2 ++
>  2 files changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index baedadf20b8c..d063312a3b2a 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -1752,7 +1752,13 @@ static void spapr_machine_reset(MachineState *machine)
>          spapr_ovec_cleanup(spapr->ov5_cas);
>          spapr->ov5_cas = spapr_ovec_new();
>  
> -        ppc_set_compat(first_ppc_cpu, spapr->max_compat_pvr, &error_fatal);
> +        /*
> +         * reset compat_pvr for all CPUs
> +         * as qemu_devices_reset() is called before this,
> +         * it can't be propagated by spapr_cpu_reset()
> +         * from the first CPU to all the others
> +         */
> +        ppc_set_compat_all(spapr->max_compat_pvr, &error_fatal);
>      }
>  
>      /*
> diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
> index bf47fbdf6f7f..45e2f2747ffc 100644
> --- a/hw/ppc/spapr_cpu_core.c
> +++ b/hw/ppc/spapr_cpu_core.c
> @@ -43,6 +43,8 @@ static void spapr_cpu_reset(void *opaque)
>  
>      /* Set compatibility mode to match the boot CPU, which was either set
>       * by the machine reset code or by CAS. This should never fail.
> +     * At startup the value is already set for all the CPUs
> +     * but we need this when we hotplug a new CPU
>       */
>      ppc_set_compat(cpu, POWERPC_CPU(first_cpu)->compat_pvr, &error_abort);
>  

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] [PATCH] pseries: Fix compat_pvr on reset

[Greg Kurz]

On Mon, 26 Aug 2019 11:08:12 +0200
Laurent Vivier <lvivier@redhat.com> wrote:

> If we a migrate P8 machine to a P9 machine, the migration fails on
> destination with:
> 
>   error while loading state for instance 0x1 of device 'cpu'
>   load of migration failed: Operation not permitted
> 
> This is caused because the compat_pvr field is only present for the first
> CPU.
> Originally, spapr_machine_reset() calls ppc_set_compat() to set the value
> max_compat_pvr for the first cpu and this was propagated to all CPUs by
> spapr_cpu_reset().  Now, as spapr_cpu_reset() is called before that, the
> value is not propagated to all CPUs and the migration fails.
> 
> To fix that, propagate the new value to all CPUs in spapr_machine_reset().
> 

Yeah, the assumption that compat_pvr would be set for the boot CPU before
device reset was rather fragile. It makes a lot of sense to do this explicitly
from the core machine code.

Reviewed-by: Greg Kurz <groug@kaod.org>

And now, ppc_set_compat() ends up being called twice for every CPU at
machine reset. It isn't a great performance penalty but I think the
case of hotplugged CPUs could be better handled by calling ppc_set_compat()
from spapr_core_plug(). This would also be cleaner to have the compat_pvr
stuff to be handled in spapr.c only rather than in two separate files IMHO.

I'll send a patch.

> Fixes: 25c9780d38d4 ("spapr: Reset CAS & IRQ subsystem after devices")
> Signed-off-by: Laurent Vivier <lvivier@redhat.com>
> ---
>  hw/ppc/spapr.c          | 8 +++++++-
>  hw/ppc/spapr_cpu_core.c | 2 ++
>  2 files changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index baedadf20b8c..d063312a3b2a 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -1752,7 +1752,13 @@ static void spapr_machine_reset(MachineState *machine)
>          spapr_ovec_cleanup(spapr->ov5_cas);
>          spapr->ov5_cas = spapr_ovec_new();
>  
> -        ppc_set_compat(first_ppc_cpu, spapr->max_compat_pvr, &error_fatal);
> +        /*
> +         * reset compat_pvr for all CPUs
> +         * as qemu_devices_reset() is called before this,
> +         * it can't be propagated by spapr_cpu_reset()
> +         * from the first CPU to all the others
> +         */
> +        ppc_set_compat_all(spapr->max_compat_pvr, &error_fatal);
>      }
>  
>      /*
> diff --git a/hw/ppc/spapr_cpu_core.c b/hw/ppc/spapr_cpu_core.c
> index bf47fbdf6f7f..45e2f2747ffc 100644
> --- a/hw/ppc/spapr_cpu_core.c
> +++ b/hw/ppc/spapr_cpu_core.c
> @@ -43,6 +43,8 @@ static void spapr_cpu_reset(void *opaque)
>  
>      /* Set compatibility mode to match the boot CPU, which was either set
>       * by the machine reset code or by CAS. This should never fail.
> +     * At startup the value is already set for all the CPUs
> +     * but we need this when we hotplug a new CPU
>       */
>      ppc_set_compat(cpu, POWERPC_CPU(first_cpu)->compat_pvr, &error_abort);
>