* [Qemu-devel] Live Migration between machines with different processor ids @ 2018-08-21 6:39 Mjaggi Oss 2018-08-23 11:18 ` [Qemu-devel] [Query] " Jaggi, Manish 0 siblings, 1 reply; 20+ messages in thread From: Mjaggi Oss @ 2018-08-21 6:39 UTC (permalink / raw) To: qemu-devel; +Cc: manish.jaggi Live Migration between machines with different processorIds VM Migration between machines with different processorId values throws an error in qemu/kvm. Though this check is appropriate but is overkill in cases where the two machines are of same SoC/arch family and have exactly similar core/gic but delta could be in other parts of Soc which have no effect on VM operation. There could be two ways to address this issue by ignoring the comparison of processorIDs and so need feedback from the community on this. a) Maintain a whitelist in qemu: This will be a set of all processorIds which are compatible and migration can happen between any of the machines with the Ids from this set. This set can be statically built within qemu binary. b) Provide an extra option with migrate command migrate tcp:<ip>:<port>:<dest_processor_id> This is to fake the src_processor_id as dest_processor_id, so the qemu running on destination machine will not complain. The overhead with this approach is that the destination machines Id need to be known beforehand. If there is some better way… please suggest. Thanks, Manish ^ permalink raw reply [flat|nested] 20+ messages in thread
* [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-08-21 6:39 [Qemu-devel] Live Migration between machines with different processor ids Mjaggi Oss @ 2018-08-23 11:18 ` Jaggi, Manish 2018-08-23 14:29 ` Juan Quintela 0 siblings, 1 reply; 20+ messages in thread From: Jaggi, Manish @ 2018-08-23 11:18 UTC (permalink / raw) To: Anthony Liguori, Juan Quintela, qemu-devel; +Cc: Jaggi, Manish Hi, Posting again with my cavium ID and CCing relevant folks Live Migration between machines with different processorIds VM Migration between machines with different processorId values throws error in qemu/kvm. Though this check is appropriate but is overkill where two machines are of same SoC/arch family and have same core/gic but delta could be in other parts of Soc which have no effect on VM operation. There could be two ways to address this issue by ignoring the comparison of processorIDs and so need feedback from the community on this. a) Maintain a whitelist in qemu: This will be a set of all processorIds which are compatible and migration can happen between any of the machines with the Ids from this set. This set can be statically built within qemu binary. b) Provide an extra option with migrate command migrate tcp:<ip>:<port>:<dest_processor_id> This is to fake the src_processor_id as dest_processor_id, so the qemu running on destination machine will not complain. The overhead with this approach is that the destination machines Id need to be known beforehand. If there is some better way… please suggest. Thanks, Manish ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-08-23 11:18 ` [Qemu-devel] [Query] " Jaggi, Manish @ 2018-08-23 14:29 ` Juan Quintela 2018-08-24 9:24 ` Jaggi, Manish 0 siblings, 1 reply; 20+ messages in thread From: Juan Quintela @ 2018-08-23 14:29 UTC (permalink / raw) To: Jaggi, Manish; +Cc: Anthony Liguori, qemu-devel "Jaggi, Manish" <Manish.Jaggi@cavium.com> wrote: > Hi, Hi [Note that I was confused about what do you mean with problems with processorID. There is no processorID on the migration stream, so I didn't understood what you were talking about. Until I realized that you were trying to migrate from different cpu types] > Posting again with my cavium ID and CCing relevant folks It will be good to give What architecture are we talking about? MIPS, ARM, anything else? Why? Because we do this continously on x86_64 world. How do we do this? We emulate the _processor_ capabilities, so "in general" you can always migrate from a processor to another with a superset of the features. If you look at the ouput of: qemu-system-x86_64 -cpu ? You can see that we have lots of cpu types that we emulate and cpuid (features really). Migration intel<->amd is tricky. But from "intel with less features" to "intel with more features" (or the same with AMD) it is a common thing to do. Once told that, it is a lot of work, simple things like that processors run at different clock speeds imply that you need to be careful during migration with timers and anything that depends on frequencies. I don't know enough about other architectures to know how to do it, or how feasible is. > Live Migration between machines with different processorIds > > VM Migration between machines with different processorId values throws > error in qemu/kvm. Though this check is appropriate but is overkill where > two machines are of same SoC/arch family and have same core/gic but > delta could be in other parts of Soc which have no effect on VM > operation. Then you need to do the whole process of: Lets call both processors A1 and A2. You need to do the whole process of: a- defining cpu A1 b- make sure that when you run qemu/kvm on processor A2, the features/behaviours that the guest sees. This is not trivial at all. c- when migration comes, you can see that you need to adjust to whatever is the architecture of the destination. > There could be two ways to address this issue by ignoring the > comparison of processorIDs and so need feedback from the > community on this. > > a) Maintain a whitelist in qemu: > > This will be a set of all processorIds which are compatible and migration can > happen between any of the machines with the Ids from this set. This set can > be statically built within qemu binary. In general, I preffer whitelists over blacklists. > b) Provide an extra option with migrate command > > migrate tcp:<ip>:<port>:<dest_processor_id> > > This is to fake the src_processor_id as dest_processor_id, so the qemu running > on destination machine will not complain. The overhead with this approach is > that the destination machines Id need to be known beforehand. Please, don't even think about this: a- migration commands are architecture agnostic b- in general it is _much_, _much_ easier to fix things on destination that on source. > If there is some better way… please suggest. Look at how it is done on x86_64. But be aware that "doing it right" takes a lot of work. To give you one idea: - upstream, i.e. qemu, "warantee" that migration of: qemu-X -M machine-type-X -> qemu-Y -M machine-type-X works when X < Y. - downstream (i.e. redhat on my case, but I am sure that others also "suffer" this) allow also: qemu-Y -M machine-type-X -> qemu-X -M machine-type-X (Y > X) in general it is a very complicated problem, so we limit _what_ you can do. Basically we only support our machine-types, do a lot of testing, and are very careful when we add new features. I.e. be preparred to do a lot of testing and a lot of fixing. I am sorry to not be able to tell you that this is an easy problem. Later, Juan. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-08-23 14:29 ` Juan Quintela @ 2018-08-24 9:24 ` Jaggi, Manish 2018-08-28 17:27 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 20+ messages in thread From: Jaggi, Manish @ 2018-08-24 9:24 UTC (permalink / raw) To: Juan Quintela, qemu-devel; +Cc: Jaggi, Manish, Anthony Liguori > On 23-Aug-2018, at 7:59 PM, Juan Quintela <quintela@redhat.com> wrote: > > External Email > > "Jaggi, Manish" <Manish.Jaggi@cavium.com> wrote: >> Hi, > > Hi > > [Note that I was confused about what do you mean with problems with > processorID. There is no processorID on the migration stream, so I > didn't understood what you were talking about. Until I realized that > you were trying to migrate from different cpu types] > >> Posting again with my cavium ID and CCing relevant folks > > It will be good to give What architecture are we talking about? MIPS, > ARM, anything else? > arm64 > Why? Because we do this continously on x86_64 world. How do we do > this? We emulate the _processor_ capabilities, so "in general" you can > always migrate from a processor to another with a superset of the > features. If you look at the ouput of: > > qemu-system-x86_64 -cpu ? > > You can see that we have lots of cpu types that we emulate and cpuid > (features really). Migration intel<->amd is tricky. But from "intel > with less features" to "intel with more features" (or the same with AMD) > it is a common thing to do. Once told that, it is a lot of work, simple > things like that processors run at different clock speeds imply that you > need to be careful during migration with timers and anything that > depends on frequencies. > > I don't know enough about other architectures to know how to do it, or > how feasible is. For arm64 qemu/kvm throws an error when processorID does not match. > >> Live Migration between machines with different processorIds >> >> VM Migration between machines with different processorId values throws >> error in qemu/kvm. Though this check is appropriate but is overkill where >> two machines are of same SoC/arch family and have same core/gic but >> delta could be in other parts of Soc which have no effect on VM >> operation. > > Then you need to do the whole process of: > > Lets call both processors A1 and A2. You need to do the whole process > of: > > a- defining cpu A1 > b- make sure that when you run qemu/kvm on processor A2, the > features/behaviours that the guest sees. This is not trivial at > all. > c- when migration comes, you can see that you need to adjust to whatever > is the architecture of the destination. > >> There could be two ways to address this issue by ignoring the >> comparison of processorIDs and so need feedback from the >> community on this. >> >> a) Maintain a whitelist in qemu: >> >> This will be a set of all processorIds which are compatible and migration can >> happen between any of the machines with the Ids from this set. This set can >> be statically built within qemu binary. > > In general, I preffer whitelists over blacklists. > >> b) Provide an extra option with migrate command >> >> migrate tcp:<ip>:<port>:<dest_processor_id> >> >> This is to fake the src_processor_id as dest_processor_id, so the qemu running >> on destination machine will not complain. The overhead with this approach is >> that the destination machines Id need to be known beforehand. > > Please, don't even think about this: > a- migration commands are architecture agnostic > b- in general it is _much_, _much_ easier to fix things on destination > that on source. > >> If there is some better way… please suggest. > > Look at how it is done on x86_64. But be aware that "doing it right" > takes a lot of work. To give you one idea: > - upstream, i.e. qemu, "warantee" that migration of: > qemu-X -M machine-type-X -> qemu-Y -M machine-type-X > works when X < Y. > > - downstream (i.e. redhat on my case, but I am sure that others also > "suffer" this) allow also: > > qemu-Y -M machine-type-X -> qemu-X -M machine-type-X (Y > X) > > in general it is a very complicated problem, so we limit _what_ you > can do. Basically we only support our machine-types, do a lot of > testing, and are very careful when we add new features. I.e. be > preparred to do a lot of testing and a lot of fixing. At this point I am targeting a simpler case where Machine A1 and A2 has a core from the same SoC family. For example Cavium ThunderX2 Core incremental versions which has identical core and GIC and may have some errata fixes. In that case Y=X since migration only takes care of PV devices. In that case a whitelist could be an easier option? How to provide the whitelist to qemu in a platform agnostic way? - I will look into intel model as you have suggested, does intel keeps a whitelist or masks off some bits of processorID How does intel does it - is providing a -mirate-compat-whitelist <file> option for arm only looks good? this option can be added in A1/A2 qemu command, so it would be upstream / downstream agnostic. > > I am sorry to not be able to tell you that this is an easy problem. > > Later, Juan. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-08-24 9:24 ` Jaggi, Manish @ 2018-08-28 17:27 ` Dr. David Alan Gilbert 2018-08-29 12:40 ` Jaggi, Manish 0 siblings, 1 reply; 20+ messages in thread From: Dr. David Alan Gilbert @ 2018-08-28 17:27 UTC (permalink / raw) To: Jaggi, Manish Cc: Juan Quintela, eric.auger, drjones, peter.maydell@linaro.org qemu-devel@nongnu.org, Anthony Liguori (Cc'ing in Eric, Drew, and Peter for ARM stuff) * Jaggi, Manish (Manish.Jaggi@cavium.com) wrote: > > > > On 23-Aug-2018, at 7:59 PM, Juan Quintela <quintela@redhat.com> wrote: > > > > External Email > > > > "Jaggi, Manish" <Manish.Jaggi@cavium.com> wrote: > >> Hi, > > > > Hi > > > > [Note that I was confused about what do you mean with problems with > > processorID. There is no processorID on the migration stream, so I > > didn't understood what you were talking about. Until I realized that > > you were trying to migrate from different cpu types] > > > >> Posting again with my cavium ID and CCing relevant folks > > > > It will be good to give What architecture are we talking about? MIPS, > > ARM, anything else? > > > arm64 > > > Why? Because we do this continously on x86_64 world. How do we do > > this? We emulate the _processor_ capabilities, so "in general" you can > > always migrate from a processor to another with a superset of the > > features. If you look at the ouput of: > > > > qemu-system-x86_64 -cpu ? > > > > You can see that we have lots of cpu types that we emulate and cpuid > > (features really). Migration intel<->amd is tricky. But from "intel > > with less features" to "intel with more features" (or the same with AMD) > > it is a common thing to do. Once told that, it is a lot of work, simple > > things like that processors run at different clock speeds imply that you > > need to be careful during migration with timers and anything that > > depends on frequencies. > > > > I don't know enough about other architectures to know how to do it, or > > how feasible is. > > For arm64 qemu/kvm throws an error when processorID does not match. > > > >> Live Migration between machines with different processorIds > >> > >> VM Migration between machines with different processorId values throws > >> error in qemu/kvm. Though this check is appropriate but is overkill where > >> two machines are of same SoC/arch family and have same core/gic but > >> delta could be in other parts of Soc which have no effect on VM > >> operation. > > > > Then you need to do the whole process of: > > > > Lets call both processors A1 and A2. You need to do the whole process > > of: > > > > a- defining cpu A1 > > b- make sure that when you run qemu/kvm on processor A2, the > > features/behaviours that the guest sees. This is not trivial at > > all. > > c- when migration comes, you can see that you need to adjust to whatever > > is the architecture of the destination. > > > >> There could be two ways to address this issue by ignoring the > >> comparison of processorIDs and so need feedback from the > >> community on this. > >> > >> a) Maintain a whitelist in qemu: > >> > >> This will be a set of all processorIds which are compatible and migration can > >> happen between any of the machines with the Ids from this set. This set can > >> be statically built within qemu binary. > > > > In general, I preffer whitelists over blacklists. > > > >> b) Provide an extra option with migrate command > >> > >> migrate tcp:<ip>:<port>:<dest_processor_id> > >> > >> This is to fake the src_processor_id as dest_processor_id, so the qemu running > >> on destination machine will not complain. The overhead with this approach is > >> that the destination machines Id need to be known beforehand. > > > > Please, don't even think about this: > > a- migration commands are architecture agnostic > > b- in general it is _much_, _much_ easier to fix things on destination > > that on source. > > > >> If there is some better way… please suggest. > > > > Look at how it is done on x86_64. But be aware that "doing it right" > > takes a lot of work. To give you one idea: > > - upstream, i.e. qemu, "warantee" that migration of: > > qemu-X -M machine-type-X -> qemu-Y -M machine-type-X > > works when X < Y. > > > > - downstream (i.e. redhat on my case, but I am sure that others also > > "suffer" this) allow also: > > > > qemu-Y -M machine-type-X -> qemu-X -M machine-type-X (Y > X) > > > > in general it is a very complicated problem, so we limit _what_ you > > can do. Basically we only support our machine-types, do a lot of > > testing, and are very careful when we add new features. I.e. be > > preparred to do a lot of testing and a lot of fixing. > > At this point I am targeting a simpler case where Machine A1 and A2 has a core from the same SoC family. > For example Cavium ThunderX2 Core incremental versions which has identical core and GIC and may have some errata fixes. > In that case Y=X since migration only takes care of PV devices. > > In that case a whitelist could be an easier option? > > How to provide the whitelist to qemu in a platform agnostic way? > - I will look into intel model as you have suggested, does intel keeps a whitelist or masks off some bits of processorID > How does intel does it Purely based on features rather than IDs. If it's an Intel processor and it's got that set of CPU features migration to it will normally work. (There are some gotcha's that we hit from time to time, but the basic idea holds) Dave > - is providing a -mirate-compat-whitelist <file> option for arm only looks good? > this option can be added in A1/A2 qemu command, so it would be upstream / downstream agnostic. > > > > I am sorry to not be able to tell you that this is an easy problem. > > > > Later, Juan. > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-08-28 17:27 ` Dr. David Alan Gilbert @ 2018-08-29 12:40 ` Jaggi, Manish 2018-08-29 13:16 ` Andrew Jones 2018-08-29 13:58 ` Dr. David Alan Gilbert 0 siblings, 2 replies; 20+ messages in thread From: Jaggi, Manish @ 2018-08-29 12:40 UTC (permalink / raw) To: Dr. David Alan Gilbert Cc: Jaggi, Manish, Juan Quintela, Auger Eric, drjones, peter.maydell@linaro.org qemu-devel@nongnu.org, Anthony Liguori > On 28-Aug-2018, at 10:57 PM, Dr. David Alan Gilbert <dgilbert@redhat.com> wrote: > > External Email > > (Cc'ing in Eric, Drew, and Peter for ARM stuff) > Thanks, > * Jaggi, Manish (Manish.Jaggi@cavium.com) wrote: >> >> >>> On 23-Aug-2018, at 7:59 PM, Juan Quintela <quintela@redhat.com> wrote: >>> >>> External Email >>> >>> "Jaggi, Manish" <Manish.Jaggi@cavium.com> wrote: >>>> Hi, >>> >>> Hi >>> >>> [Note that I was confused about what do you mean with problems with >>> processorID. There is no processorID on the migration stream, so I >>> didn't understood what you were talking about. Until I realized that >>> you were trying to migrate from different cpu types] >>> >>>> Posting again with my cavium ID and CCing relevant folks >>> >>> It will be good to give What architecture are we talking about? MIPS, >>> ARM, anything else? >>> >> arm64 >> >>> Why? Because we do this continously on x86_64 world. How do we do >>> this? We emulate the _processor_ capabilities, so "in general" you can >>> always migrate from a processor to another with a superset of the >>> features. If you look at the ouput of: >>> >>> qemu-system-x86_64 -cpu ? >>> >>> You can see that we have lots of cpu types that we emulate and cpuid >>> (features really). Migration intel<->amd is tricky. But from "intel >>> with less features" to "intel with more features" (or the same with AMD) >>> it is a common thing to do. Once told that, it is a lot of work, simple >>> things like that processors run at different clock speeds imply that you >>> need to be careful during migration with timers and anything that >>> depends on frequencies. >>> >>> I don't know enough about other architectures to know how to do it, or >>> how feasible is. >> >> For arm64 qemu/kvm throws an error when processorID does not match. >>> >>>> Live Migration between machines with different processorIds >>>> >>>> VM Migration between machines with different processorId values throws >>>> error in qemu/kvm. Though this check is appropriate but is overkill where >>>> two machines are of same SoC/arch family and have same core/gic but >>>> delta could be in other parts of Soc which have no effect on VM >>>> operation. >>> >>> Then you need to do the whole process of: >>> >>> Lets call both processors A1 and A2. You need to do the whole process >>> of: >>> >>> a- defining cpu A1 >>> b- make sure that when you run qemu/kvm on processor A2, the >>> features/behaviours that the guest sees. This is not trivial at >>> all. >>> c- when migration comes, you can see that you need to adjust to whatever >>> is the architecture of the destination. >>> >>>> There could be two ways to address this issue by ignoring the >>>> comparison of processorIDs and so need feedback from the >>>> community on this. >>>> >>>> a) Maintain a whitelist in qemu: >>>> >>>> This will be a set of all processorIds which are compatible and migration can >>>> happen between any of the machines with the Ids from this set. This set can >>>> be statically built within qemu binary. >>> >>> In general, I preffer whitelists over blacklists. >>> >>>> b) Provide an extra option with migrate command >>>> >>>> migrate tcp:<ip>:<port>:<dest_processor_id> >>>> >>>> This is to fake the src_processor_id as dest_processor_id, so the qemu running >>>> on destination machine will not complain. The overhead with this approach is >>>> that the destination machines Id need to be known beforehand. >>> >>> Please, don't even think about this: >>> a- migration commands are architecture agnostic >>> b- in general it is _much_, _much_ easier to fix things on destination >>> that on source. >>> >>>> If there is some better way… please suggest. >>> >>> Look at how it is done on x86_64. But be aware that "doing it right" >>> takes a lot of work. To give you one idea: >>> - upstream, i.e. qemu, "warantee" that migration of: >>> qemu-X -M machine-type-X -> qemu-Y -M machine-type-X >>> works when X < Y. >>> >>> - downstream (i.e. redhat on my case, but I am sure that others also >>> "suffer" this) allow also: >>> >>> qemu-Y -M machine-type-X -> qemu-X -M machine-type-X (Y > X) >>> >>> in general it is a very complicated problem, so we limit _what_ you >>> can do. Basically we only support our machine-types, do a lot of >>> testing, and are very careful when we add new features. I.e. be >>> preparred to do a lot of testing and a lot of fixing. >> >> At this point I am targeting a simpler case where Machine A1 and A2 has a core from the same SoC family. >> For example Cavium ThunderX2 Core incremental versions which has identical core and GIC and may have some errata fixes. >> In that case Y=X since migration only takes care of PV devices. >> >> In that case a whitelist could be an easier option? >> >> How to provide the whitelist to qemu in a platform agnostic way? >> - I will look into intel model as you have suggested, does intel keeps a whitelist or masks off some bits of processorID >> How does intel does it > > Purely based on features rather than IDs. > > If it's an Intel processor and it's got that set of CPU features > migration to it will normally work. > (There are some gotcha's that we hit from time to time, but > the basic idea holds) > Just to add what happens in ARM64 case, qemu running on Machine A sends cpu state information to Machine B. This state contains MIDR value, and so Processor ID value is compared in KVM and not in qemu (correcting myself). IIRC, Peter/Eric please point if there is something incorrect in the below flow... (Machine B) target/arm/machine.c: cpu_post_load() - updates cpu->cpreg_values[i] : which includes MIDR (processor ID register) - calls write_list_to_kvmstate(cpu, KVM_PUT_FULL_STATE) target/arm/kvm.c: write_list_to_kvmstate - calls => kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &r); => and it eventually lands up IIRC in Linux code in => arch/arm64/kvm/sys_regs.c : set_invariant_sys_reg(u64 id, void __user *uaddr) /* This is what we mean by invariant: you can't change it. */ if (r->val != val) return -EINVAL; Note: MIDR_EL1 is invariant register. result: Migration fails on Machine B. A few points: - qemu on arm64 is invoked with -machine virt and -cpu as host. So we don't explicitly define which cpu. - In case Machine A and Machine B have almost same Core and the delta may-not have any effect on qemu operation, migration should work by just looking into whitelist. whitelist can be given as a parameter for qemu on machine B. qemu-system-aarch64 -whitelist <ids separated by commas> (This is my proposal) - So in cpu_post_load (Machine B) qemu can lookup whitelist and replace the MIDR with the one at Machine B. Sounds good? - Juan raised a point about clock speed, I am not sure it will have any effect on arm since qemu is run with -cpu host param. I could be wrong here, Peter/Eric can you please correct me... -Thanks Manish > Dave >> - is providing a -mirate-compat-whitelist <file> option for arm only looks good? >> this option can be added in A1/A2 qemu command, so it would be upstream / downstream agnostic. > >>> >>> I am sorry to not be able to tell you that this is an easy problem. >>> >>> Later, Juan. >> > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-08-29 12:40 ` Jaggi, Manish @ 2018-08-29 13:16 ` Andrew Jones 2018-08-31 9:52 ` Jaggi, Manish 2018-08-29 13:58 ` Dr. David Alan Gilbert 1 sibling, 1 reply; 20+ messages in thread From: Andrew Jones @ 2018-08-29 13:16 UTC (permalink / raw) To: Jaggi, Manish Cc: Dr. David Alan Gilbert, Juan Quintela, Auger Eric, peter.maydell@linaro.org qemu-devel@nongnu.org, Anthony Liguori On Wed, Aug 29, 2018 at 12:40:08PM +0000, Jaggi, Manish wrote: > > > > On 28-Aug-2018, at 10:57 PM, Dr. David Alan Gilbert <dgilbert@redhat.com> wrote: > > > > External Email > > > > (Cc'ing in Eric, Drew, and Peter for ARM stuff) > > > Thanks, > > * Jaggi, Manish (Manish.Jaggi@cavium.com) wrote: > >> > >> > >>> On 23-Aug-2018, at 7:59 PM, Juan Quintela <quintela@redhat.com> wrote: > >>> > >>> External Email > >>> > >>> "Jaggi, Manish" <Manish.Jaggi@cavium.com> wrote: > >>>> Hi, > >>> > >>> Hi > >>> > >>> [Note that I was confused about what do you mean with problems with > >>> processorID. There is no processorID on the migration stream, so I > >>> didn't understood what you were talking about. Until I realized that > >>> you were trying to migrate from different cpu types] > >>> > >>>> Posting again with my cavium ID and CCing relevant folks > >>> > >>> It will be good to give What architecture are we talking about? MIPS, > >>> ARM, anything else? > >>> > >> arm64 > >> > >>> Why? Because we do this continously on x86_64 world. How do we do > >>> this? We emulate the _processor_ capabilities, so "in general" you can > >>> always migrate from a processor to another with a superset of the > >>> features. If you look at the ouput of: > >>> > >>> qemu-system-x86_64 -cpu ? > >>> > >>> You can see that we have lots of cpu types that we emulate and cpuid > >>> (features really). Migration intel<->amd is tricky. But from "intel > >>> with less features" to "intel with more features" (or the same with AMD) > >>> it is a common thing to do. Once told that, it is a lot of work, simple > >>> things like that processors run at different clock speeds imply that you > >>> need to be careful during migration with timers and anything that > >>> depends on frequencies. > >>> > >>> I don't know enough about other architectures to know how to do it, or > >>> how feasible is. > >> > >> For arm64 qemu/kvm throws an error when processorID does not match. > >>> > >>>> Live Migration between machines with different processorIds > >>>> > >>>> VM Migration between machines with different processorId values throws > >>>> error in qemu/kvm. Though this check is appropriate but is overkill where > >>>> two machines are of same SoC/arch family and have same core/gic but > >>>> delta could be in other parts of Soc which have no effect on VM > >>>> operation. > >>> > >>> Then you need to do the whole process of: > >>> > >>> Lets call both processors A1 and A2. You need to do the whole process > >>> of: > >>> > >>> a- defining cpu A1 > >>> b- make sure that when you run qemu/kvm on processor A2, the > >>> features/behaviours that the guest sees. This is not trivial at > >>> all. > >>> c- when migration comes, you can see that you need to adjust to whatever > >>> is the architecture of the destination. > >>> > >>>> There could be two ways to address this issue by ignoring the > >>>> comparison of processorIDs and so need feedback from the > >>>> community on this. > >>>> > >>>> a) Maintain a whitelist in qemu: > >>>> > >>>> This will be a set of all processorIds which are compatible and migration can > >>>> happen between any of the machines with the Ids from this set. This set can > >>>> be statically built within qemu binary. > >>> > >>> In general, I preffer whitelists over blacklists. > >>> > >>>> b) Provide an extra option with migrate command > >>>> > >>>> migrate tcp:<ip>:<port>:<dest_processor_id> > >>>> > >>>> This is to fake the src_processor_id as dest_processor_id, so the qemu running > >>>> on destination machine will not complain. The overhead with this approach is > >>>> that the destination machines Id need to be known beforehand. > >>> > >>> Please, don't even think about this: > >>> a- migration commands are architecture agnostic > >>> b- in general it is _much_, _much_ easier to fix things on destination > >>> that on source. > >>> > >>>> If there is some better way… please suggest. > >>> > >>> Look at how it is done on x86_64. But be aware that "doing it right" > >>> takes a lot of work. To give you one idea: > >>> - upstream, i.e. qemu, "warantee" that migration of: > >>> qemu-X -M machine-type-X -> qemu-Y -M machine-type-X > >>> works when X < Y. > >>> > >>> - downstream (i.e. redhat on my case, but I am sure that others also > >>> "suffer" this) allow also: > >>> > >>> qemu-Y -M machine-type-X -> qemu-X -M machine-type-X (Y > X) > >>> > >>> in general it is a very complicated problem, so we limit _what_ you > >>> can do. Basically we only support our machine-types, do a lot of > >>> testing, and are very careful when we add new features. I.e. be > >>> preparred to do a lot of testing and a lot of fixing. > >> > >> At this point I am targeting a simpler case where Machine A1 and A2 has a core from the same SoC family. > >> For example Cavium ThunderX2 Core incremental versions which has identical core and GIC and may have some errata fixes. That may or may not be a simple case. What happens when the minor revision that contains an erratum fix not only stops requiring a guest kernel workaround to be used, but actually causes the guest kernel to break when that workaround is attempted? > >> In that case Y=X since migration only takes care of PV devices. > >> > >> In that case a whitelist could be an easier option? > >> > >> How to provide the whitelist to qemu in a platform agnostic way? > >> - I will look into intel model as you have suggested, does intel keeps a whitelist or masks off some bits of processorID > >> How does intel does it > > > > Purely based on features rather than IDs. x86 processors have a stable core, i.e. it's pretty rare to have an erratum in a new processor that can't easily be dealt with by turning off its corresponding feature, i.e. masking off some CPUID feature bits. AArch64 processors have ID registers (similar to CPUID feature leaves), so if there was an erratum in one of the features advertised by its ID registers, then that erratum could be dealt with the same way. However, AArch64 processors also have plenty of errata in their core. The only way to detect when those errata require workarounds is by MIDR. > > > > If it's an Intel processor and it's got that set of CPU features > > migration to it will normally work. > > (There are some gotcha's that we hit from time to time, but > > the basic idea holds) > > > > Just to add what happens in ARM64 case, qemu running on Machine A sends cpu state information to Machine B. > This state contains MIDR value, and so Processor ID value is compared in KVM and not in qemu (correcting myself). > > IIRC, Peter/Eric please point if there is something incorrect in the below flow... > > (Machine B) > target/arm/machine.c: cpu_post_load() > - updates cpu->cpreg_values[i] : which includes MIDR (processor ID register) > > - calls write_list_to_kvmstate(cpu, KVM_PUT_FULL_STATE) > > target/arm/kvm.c: write_list_to_kvmstate > - calls => kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &r); > > => and it eventually lands up IIRC in Linux code in > > => arch/arm64/kvm/sys_regs.c : set_invariant_sys_reg(u64 id, void __user *uaddr) > /* This is what we mean by invariant: you can't change it. */ > if (r->val != val) > return -EINVAL; > Note: MIDR_EL1 is invariant register. > result: Migration fails on Machine B. > > A few points: > - qemu on arm64 is invoked with -machine virt and -cpu as host. So we don't explicitly define which cpu. > > - In case Machine A and Machine B have almost same Core and the delta may-not have any effect on qemu operation, migration should work by just looking into whitelist. > whitelist can be given as a parameter for qemu on machine B. > > qemu-system-aarch64 -whitelist <ids separated by commas> > > (This is my proposal) > > - So in cpu_post_load (Machine B) qemu can lookup whitelist and replace the MIDR with the one at Machine B. > Sounds good? It shouldn't be necessary. With '-cpu host' QEMU should probably just read all the ID registers from the host first, updating the guest's copy to match the destination host's registers (we're using '-cpu host', the registers should match the host - including MIDR.) If a user chooses to migrate a guest that is using '-cpu host', then they need to know what they are doing. If a whitelist of close-enough processors is possible to create, then that whitelist should be managed and used at a higher layer in the virt stack, not down in QEMU. For example, openstack can determine destination candidates using whatever policy it wants, including a close- enough processor whitelist. So, I propose blindly updating all invariant registers when migrating a '-cpu host' guest and leaving it to the user to do these migrations at their own risk (when migrating to a truly identical host, the blind update will not change anything. So it would be no worse than what we do today.) One side note is that we're starting to give QEMU control over what optional processor features are available to the guest, e.g. SVE. So before blindly updating all ID registers we'd want to inform KVM of the guest configuration in order for KVM to return appropriate ID register values. Thanks, drew > > - Juan raised a point about clock speed, I am not sure it will have any effect on arm since qemu is run with -cpu host param. > I could be wrong here, Peter/Eric can you please correct me... > > -Thanks > Manish > > > > > Dave > >> - is providing a -mirate-compat-whitelist <file> option for arm only looks good? > >> this option can be added in A1/A2 qemu command, so it would be upstream / downstream agnostic. > > > >>> > >>> I am sorry to not be able to tell you that this is an easy problem. > >>> > >>> Later, Juan. > >> > > -- > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-08-29 13:16 ` Andrew Jones @ 2018-08-31 9:52 ` Jaggi, Manish 2018-08-31 11:11 ` Andrew Jones 0 siblings, 1 reply; 20+ messages in thread From: Jaggi, Manish @ 2018-08-31 9:52 UTC (permalink / raw) To: Andrew Jones Cc: Jaggi, Manish, Dr. David Alan Gilbert, Juan Quintela, Auger Eric, peter.maydell@linaro.org qemu-devel@nongnu.org, Anthony Liguori On 29-Aug-2018, at 6:46 PM, Andrew Jones <drjones@redhat.com<mailto:drjones@redhat.com>> wrote: External Email On Wed, Aug 29, 2018 at 12:40:08PM +0000, Jaggi, Manish wrote: On 28-Aug-2018, at 10:57 PM, Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>> wrote: External Email (Cc'ing in Eric, Drew, and Peter for ARM stuff) Thanks, * Jaggi, Manish (Manish.Jaggi@cavium.com<mailto:Manish.Jaggi@cavium.com>) wrote: On 23-Aug-2018, at 7:59 PM, Juan Quintela <quintela@redhat.com<mailto:quintela@redhat.com>> wrote: External Email "Jaggi, Manish" <Manish.Jaggi@cavium.com<mailto:Manish.Jaggi@cavium.com>> wrote: Hi, Hi [Note that I was confused about what do you mean with problems with processorID. There is no processorID on the migration stream, so I didn't understood what you were talking about. Until I realized that you were trying to migrate from different cpu types] Posting again with my cavium ID and CCing relevant folks It will be good to give What architecture are we talking about? MIPS, ARM, anything else? arm64 Why? Because we do this continously on x86_64 world. How do we do this? We emulate the _processor_ capabilities, so "in general" you can always migrate from a processor to another with a superset of the features. If you look at the ouput of: qemu-system-x86_64 -cpu ? You can see that we have lots of cpu types that we emulate and cpuid (features really). Migration intel<->amd is tricky. But from "intel with less features" to "intel with more features" (or the same with AMD) it is a common thing to do. Once told that, it is a lot of work, simple things like that processors run at different clock speeds imply that you need to be careful during migration with timers and anything that depends on frequencies. I don't know enough about other architectures to know how to do it, or how feasible is. For arm64 qemu/kvm throws an error when processorID does not match. Live Migration between machines with different processorIds VM Migration between machines with different processorId values throws error in qemu/kvm. Though this check is appropriate but is overkill where two machines are of same SoC/arch family and have same core/gic but delta could be in other parts of Soc which have no effect on VM operation. Then you need to do the whole process of: Lets call both processors A1 and A2. You need to do the whole process of: a- defining cpu A1 b- make sure that when you run qemu/kvm on processor A2, the features/behaviours that the guest sees. This is not trivial at all. c- when migration comes, you can see that you need to adjust to whatever is the architecture of the destination. There could be two ways to address this issue by ignoring the comparison of processorIDs and so need feedback from the community on this. a) Maintain a whitelist in qemu: This will be a set of all processorIds which are compatible and migration can happen between any of the machines with the Ids from this set. This set can be statically built within qemu binary. In general, I preffer whitelists over blacklists. b) Provide an extra option with migrate command migrate tcp:<ip>:<port>:<dest_processor_id> This is to fake the src_processor_id as dest_processor_id, so the qemu running on destination machine will not complain. The overhead with this approach is that the destination machines Id need to be known beforehand. Please, don't even think about this: a- migration commands are architecture agnostic b- in general it is _much_, _much_ easier to fix things on destination that on source. If there is some better way… please suggest. Look at how it is done on x86_64. But be aware that "doing it right" takes a lot of work. To give you one idea: - upstream, i.e. qemu, "warantee" that migration of: qemu-X -M machine-type-X -> qemu-Y -M machine-type-X works when X < Y. - downstream (i.e. redhat on my case, but I am sure that others also "suffer" this) allow also: qemu-Y -M machine-type-X -> qemu-X -M machine-type-X (Y > X) in general it is a very complicated problem, so we limit _what_ you can do. Basically we only support our machine-types, do a lot of testing, and are very careful when we add new features. I.e. be preparred to do a lot of testing and a lot of fixing. At this point I am targeting a simpler case where Machine A1 and A2 has a core from the same SoC family. For example Cavium ThunderX2 Core incremental versions which has identical core and GIC and may have some errata fixes. That may or may not be a simple case. What happens when the minor revision that contains an erratum fix not only stops requiring a guest kernel workaround to be used, but actually causes the guest kernel to break when that workaround is attempted? Peter/Eric, Are you aware of any such case in Arm64? In that case Y=X since migration only takes care of PV devices. In that case a whitelist could be an easier option? How to provide the whitelist to qemu in a platform agnostic way? - I will look into intel model as you have suggested, does intel keeps a whitelist or masks off some bits of processorID How does intel does it Purely based on features rather than IDs. x86 processors have a stable core, i.e. it's pretty rare to have an erratum in a new processor that can't easily be dealt with by turning off its corresponding feature, i.e. masking off some CPUID feature bits. AArch64 processors have ID registers (similar to CPUID feature leaves), so if there was an erratum in one of the features advertised by its ID registers, then that erratum could be dealt with the same way. However, AArch64 processors also have plenty of errata in their core. The only way to detect when those errata require workarounds is by MIDR. If it's an Intel processor and it's got that set of CPU features migration to it will normally work. (There are some gotcha's that we hit from time to time, but the basic idea holds) Just to add what happens in ARM64 case, qemu running on Machine A sends cpu state information to Machine B. This state contains MIDR value, and so Processor ID value is compared in KVM and not in qemu (correcting myself). IIRC, Peter/Eric please point if there is something incorrect in the below flow... (Machine B) target/arm/machine.c: cpu_post_load() - updates cpu->cpreg_values[i] : which includes MIDR (processor ID register) - calls write_list_to_kvmstate(cpu, KVM_PUT_FULL_STATE) target/arm/kvm.c: write_list_to_kvmstate - calls => kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &r); => and it eventually lands up IIRC in Linux code in => arch/arm64/kvm/sys_regs.c : set_invariant_sys_reg(u64 id, void __user *uaddr) /* This is what we mean by invariant: you can't change it. */ if (r->val != val) return -EINVAL; Note: MIDR_EL1 is invariant register. result: Migration fails on Machine B. A few points: - qemu on arm64 is invoked with -machine virt and -cpu as host. So we don't explicitly define which cpu. - In case Machine A and Machine B have almost same Core and the delta may-not have any effect on qemu operation, migration should work by just looking into whitelist. whitelist can be given as a parameter for qemu on machine B. qemu-system-aarch64 -whitelist <ids separated by commas> (This is my proposal) - So in cpu_post_load (Machine B) qemu can lookup whitelist and replace the MIDR with the one at Machine B. Sounds good? It shouldn't be necessary. With '-cpu host' QEMU should probably just read all the ID registers from the host first, updating the guest's copy to match the destination host's registers (we're using '-cpu host', the registers should match the host - including MIDR.) If a user chooses to migrate a guest that is using '-cpu host', then they need to know what they are doing. If a whitelist of close-enough processors is possible to create, then that whitelist should be managed and used at a higher layer in the virt stack, not down in QEMU. How would qemu know to that it has to patch? Could not understand your point here. IIUC qemu needs some parameter for this For example, openstack can determine destination candidates using whatever policy it wants, including a close- enough processor whitelist. So, I propose blindly updating all invariant registers when migrating a '-cpu host' guest and leaving it to the user to do these migrations at their own risk yes good point updating all invariant is better, for the case where user is aware that host and destination have same cpu arch. I can prepare a RFC patch but cpu_post_load will need some flag to replace these values., (when migrating to a truly identical host, the blind update will not change anything. So it would be no worse than what we do today.) One side note is that we're starting to give QEMU control over what optional processor features are available to the guest, e.g. SVE. So before blindly updating all ID registers we'd want to inform KVM of the guest configuration in order for KVM to return appropriate ID register values. Not sure how to handle this. Thanks, drew - Juan raised a point about clock speed, I am not sure it will have any effect on arm since qemu is run with -cpu host param. I could be wrong here, Peter/Eric can you please correct me... -Thanks Manish Dave - is providing a -mirate-compat-whitelist <file> option for arm only looks good? this option can be added in A1/A2 qemu command, so it would be upstream / downstream agnostic. I am sorry to not be able to tell you that this is an easy problem. Later, Juan. -- Dr. David Alan Gilbert / dgilbert@redhat.com<mailto:dgilbert@redhat.com> / Manchester, UK ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-08-31 9:52 ` Jaggi, Manish @ 2018-08-31 11:11 ` Andrew Jones 2018-09-04 9:16 ` Jaggi, Manish 0 siblings, 1 reply; 20+ messages in thread From: Andrew Jones @ 2018-08-31 11:11 UTC (permalink / raw) To: Jaggi, Manish Cc: Anthony Liguori, Juan Quintela, peter.maydell@linaro.org qemu-devel@nongnu.org, Dr. David Alan Gilbert, Auger Eric Manish, Your mail reader doesn't appear to be providing any quoting, so I'm not sure I found all your replies. I've tried to fix up the quoting in order to reply here, but you should look into fixing your reader. On Fri, Aug 31, 2018 at 09:52:12AM +0000, Jaggi, Manish wrote: > On bleh-bleh, Andrew Jones wrote: > > On bleh-bleh, Jaggi, Manish wrote: > > > - So in cpu_post_load (Machine B) qemu can lookup whitelist and replace the MIDR with the one at Machine B. > > > Sounds good? > > > > > It shouldn't be necessary. With '-cpu host' QEMU should probably just read > > all the ID registers from the host first, updating the guest's copy to > > match the destination host's registers (we're using '-cpu host', the > > registers should match the host - including MIDR.) If a user chooses to > > migrate a guest that is using '-cpu host', then they need to know what > > they are doing. If a whitelist of close-enough processors is possible to > > create, then that whitelist should be managed and used at a higher layer > > in the virt stack, not down in QEMU. > > How would qemu know to that it has to patch? Could not understand your point here. > IIUC qemu needs some parameter for this QEMU would unconditionally update the guest's view of the invariant registers after migration, when '-cpu host' is used. There's no need for a parameter, as the update is unconditional. > > > For example, openstack can determine > > destination candidates using whatever policy it wants, including a close- > > enough processor whitelist. > > > > So, I propose blindly updating all invariant registers when migrating > > a '-cpu host' guest and leaving it to the user to do these migrations > > at their own risk > > > yes good point updating all invariant is better, for the case where user is aware that host and destination have same cpu arch. > I can prepare a RFC patch but cpu_post_load will need some flag to replace these values., I'm not sure what you mean by a flag, but I'll review the RFC when you post it. > > > > (when migrating to a truly identical host, the blind > > update will not change anything. So it would be no worse than what we > > do today.) One side note is that we're starting to give QEMU control > > over what optional processor features are available to the guest, e.g. > > SVE. So before blindly updating all ID registers we'd want to inform > > KVM of the guest configuration in order for KVM to return appropriate > > ID register values. > > > Not sure how to handle this. I think the sequence should look something like this: 1) Guest running on Host A with processor a 2) Stop guest and save its state for migration 3) Migrate guest to Host B with processor b (b is "close enough" to a) 4) Restore guest state after migration If guest is running with '-cpu host' 4.a) Inform KVM of any configuration that impacts invariant registers 4.b) Update the guest's view of all invariant registers to match the host EndIf 5) Run guest 4.a and 4.b require new code both in QEMU and KVM. 4.a may require a new KVM API, unless the existing API can be leveraged. The definition of "close enough" is left to the users and/or higher layers of the Virt stack. Thanks, drew ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-08-31 11:11 ` Andrew Jones @ 2018-09-04 9:16 ` Jaggi, Manish 2018-09-04 9:54 ` Andrew Jones 0 siblings, 1 reply; 20+ messages in thread From: Jaggi, Manish @ 2018-09-04 9:16 UTC (permalink / raw) To: Andrew Jones, Juan Quintela, Dr. David Alan Gilbert Cc: Jaggi, Manish, Anthony Liguori, peter.maydell@linaro.org qemu-devel@nongnu.org, Auger Eric > On 31-Aug-2018, at 4:41 PM, Andrew Jones <drjones@redhat.com> wrote: > > External Email > > Manish, > > Your mail reader doesn't appear to be providing any quoting, so I'm not > sure I found all your replies. I've tried to fix up the quoting in order > to reply here, but you should look into fixing your reader. > Oh. I am using Mail on MacOSx, not sure what is wrong here... > On Fri, Aug 31, 2018 at 09:52:12AM +0000, Jaggi, Manish wrote: >> On bleh-bleh, Andrew Jones wrote: >>> On bleh-bleh, Jaggi, Manish wrote: >>>> - So in cpu_post_load (Machine B) qemu can lookup whitelist and replace the MIDR with the one at Machine B. >>>> Sounds good? >>>> >>> It shouldn't be necessary. With '-cpu host' QEMU should probably just read >>> all the ID registers from the host first, updating the guest's copy to >>> match the destination host's registers (we're using '-cpu host', the >>> registers should match the host - including MIDR.) If a user chooses to >>> migrate a guest that is using '-cpu host', then they need to know what >>> they are doing. If a whitelist of close-enough processors is possible to >>> create, then that whitelist should be managed and used at a higher layer >>> in the virt stack, not down in QEMU. >> >> How would qemu know to that it has to patch? Could not understand your point here. >> IIUC qemu needs some parameter for this > > QEMU would unconditionally update the guest's view of the invariant > registers after migration, when '-cpu host' is used. There's no need > for a parameter, as the update is unconditional. > >> >>> For example, openstack can determine >>> destination candidates using whatever policy it wants, including a close- >>> enough processor whitelist. >>> >>> So, I propose blindly updating all invariant registers when migrating >>> a '-cpu host' guest and leaving it to the user to do these migrations >>> at their own risk >>> >> yes good point updating all invariant is better, for the case where user is aware that host and destination have same cpu arch. >> I can prepare a RFC patch but cpu_post_load will need some flag to replace these values., > > I'm not sure what you mean by a flag, but I'll review the RFC when you > post it. > >>> >>> (when migrating to a truly identical host, the blind >>> update will not change anything. So it would be no worse than what we >>> do today.) One side note is that we're starting to give QEMU control >>> over what optional processor features are available to the guest, e.g. >>> SVE. So before blindly updating all ID registers we'd want to inform >>> KVM of the guest configuration in order for KVM to return appropriate >>> ID register values. >>> >> Not sure how to handle this. > > I think the sequence should look something like this: > > 1) Guest running on Host A with processor a > 2) Stop guest and save its state for migration > 3) Migrate guest to Host B with processor b (b is "close enough" to a) > 4) Restore guest state after migration > If guest is running with '-cpu host' > 4.a) Inform KVM of any configuration that impacts invariant registers > 4.b) Update the guest's view of all invariant registers to match the > host > EndIf > 5) Run guest > > 4.a and 4.b require new code both in QEMU and KVM. 4.a may require a new > KVM API, unless the existing API can be leveraged. > > The definition of "close enough" is left to the users and/or higher layers > of the Virt stack. > Thanks for detailing the sequence. I got another comment from Juan and David which is not to use -cpu host, see below "I really think that the right approach here is not using -cpu host. You do the full work, create a model as David says, and be sure that you car run that model on both cpus. It is a lot of work, but it is the only way to make sure that this is going to work long term.” Not using -cpu host is orthogonal to the sequence we have been discussing in this thread. Use something like -cpu cortex-a57 (this however does not work so far) This would avoid close-enough definition, but would need substantial work. So which approach should be taken here, whats your take... > Thanks, > drew ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-09-04 9:16 ` Jaggi, Manish @ 2018-09-04 9:54 ` Andrew Jones 2018-09-04 10:27 ` Juan Quintela ` (2 more replies) 0 siblings, 3 replies; 20+ messages in thread From: Andrew Jones @ 2018-09-04 9:54 UTC (permalink / raw) To: Jaggi, Manish Cc: Juan Quintela, Dr. David Alan Gilbert, Auger Eric, Anthony Liguori, peter.maydell@linaro.org qemu-devel@nongnu.org On Tue, Sep 04, 2018 at 09:16:58AM +0000, Jaggi, Manish wrote: > > On 31-Aug-2018, at 4:41 PM, Andrew Jones <drjones@redhat.com> wrote: > > I think the sequence should look something like this: > > > > 1) Guest running on Host A with processor a > > 2) Stop guest and save its state for migration > > 3) Migrate guest to Host B with processor b (b is "close enough" to a) > > 4) Restore guest state after migration > > If guest is running with '-cpu host' > > 4.a) Inform KVM of any configuration that impacts invariant registers > > 4.b) Update the guest's view of all invariant registers to match the > > host > > EndIf > > 5) Run guest > > > > 4.a and 4.b require new code both in QEMU and KVM. 4.a may require a new > > KVM API, unless the existing API can be leveraged. > > > > The definition of "close enough" is left to the users and/or higher layers > > of the Virt stack. > > > > Thanks for detailing the sequence. > I got another comment from Juan and David which is not to use -cpu host, see below > > "I really think that the right approach here is not using -cpu host. You > do the full work, create a model as David says, and be sure that you car > run that model on both cpus. It is a lot of work, but it is the only > way to make sure that this is going to work long term.” > > Not using -cpu host is orthogonal to the sequence we have been discussing in this thread. > Use something like -cpu cortex-a57 (this however does not work so far) > This would avoid close-enough definition, but would need substantial work. > > So which approach should be taken here, whats your take... > Inventing a base-AArch64 cpu model that can then be extended with optional features is a nice way to extend the migratability of a guest, however it's hard to do because of errata. Since errata workarounds are enabled per MIDR, then we'd need to invent our own MIDR and also some way to communicate which errata we want to enable, possibly through some paravirt mechanism or through some implementation defined system registers that KVM would need to reserve and define. That's not just a ton of work for the entire virt stack (not just KVM and QEMU, but also all the layers above), but it's possible that it won't be useful in the end anyway. There's risk that enabling just one erratum workaround would restrict the guest to hosts of the exact same type anyway. For each erratum that needs to be enabled, the probability of enabling an incompatible one goes up, so it may not be likely to do much better than '-cpu host' in the end. I'm afraid that until errata are primarily showing up in optional CPU features that can simply be disabled for the workaround, that we're stuck with '-cpu host'. I'd be happy to discuss it more though. In short, I'd go with the proposal above, for now, with possibly one change. libvirt folk (Andrea Bolognani and Pino Toscano) suggest that the guest invariant register updating on the destination host only be done if the user opts-in to it. This is because right now if a user tries to migrate to a host that is not 100% identical the migration will fail, which makes the "mistake" clear. If we silently change the behavior to allow it, then what could have been a mistake, because the hosts aren't actually "close enough", may go unnoticed. I'm not 100% sure we need another user opt-in flag to be set, though, as I think the '-cpu host' indicates the user expects the VCPU to look like the host CPU, and even after migration that expectation should be met. Simply, users that migrate '-cpu host' VMs need to know what they're doing. Thanks, drew ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-09-04 9:54 ` Andrew Jones @ 2018-09-04 10:27 ` Juan Quintela 2018-09-04 10:32 ` Dr. David Alan Gilbert 2018-09-05 11:46 ` Jaggi, Manish 2 siblings, 0 replies; 20+ messages in thread From: Juan Quintela @ 2018-09-04 10:27 UTC (permalink / raw) To: Andrew Jones Cc: Jaggi, Manish, Dr. David Alan Gilbert, Auger Eric, peter.maydell@linaro.org qemu-devel@nongnu.org Andrew Jones <drjones@redhat.com> wrote: > On Tue, Sep 04, 2018 at 09:16:58AM +0000, Jaggi, Manish wrote: >> So which approach should be taken here, whats your take... >> [ Remoning Anthony form CC. Address don't exist anymore ] > Inventing a base-AArch64 cpu model that can then be extended with optional > features is a nice way to extend the migratability of a guest, however > it's hard to do because of errata. Since errata workarounds are enabled > per MIDR, then we'd need to invent our own MIDR and also some way to > communicate which errata we want to enable, possibly through some paravirt > mechanism or through some implementation defined system registers that > KVM would need to reserve and define. > > That's not just a ton of work for the entire virt stack (not just KVM and > QEMU, but also all the layers above), but it's possible that it won't be > useful in the end anyway. There's risk that enabling just one erratum > workaround would restrict the guest to hosts of the exact same type > anyway. For each erratum that needs to be enabled, the probability of > enabling an incompatible one goes up, so it may not be likely to do much > better than '-cpu host' in the end. I'm afraid that until errata are > primarily showing up in optional CPU features that can simply be disabled > for the workaround, that we're stuck with '-cpu host'. I'd be happy to > discuss it more though. Then, we are basically at the point when we can only migrate to the exact same processor, no? > In short, I'd go with the proposal above, for now, with possibly one > change. libvirt folk (Andrea Bolognani and Pino Toscano) suggest that > the guest invariant register updating on the destination host only be > done if the user opts-in to it. This is because right now if a user > tries to migrate to a host that is not 100% identical the migration > will fail, which makes the "mistake" clear. If we silently change the > behavior to allow it, then what could have been a mistake, because > the hosts aren't actually "close enough", may go unnoticed. I'm not > 100% sure we need another user opt-in flag to be set, though, as I > think the '-cpu host' indicates the user expects the VCPU to look > like the host CPU, and even after migration that expectation should be > met. Simply, users that migrate '-cpu host' VMs need to know what they're > doing. I don't know really what to say here: - on the one hand, not creating the proper cpu types is going to bite us, big time, later. - on the other hand, it appears that cpu compatibility is not so "strong", "nice", or whatever do you want to call it on ARM land. Why I am so worried? Because we have spent lots (and I mean lots) of time on x86_64 when we forgot to enable/disable/indicate that one cpu has a new MSR/feature. Normal problem is that it only happens when customer is using that particular feature. It has been the case for us that to reporduce the problem we had to ping-pong migration several hundred times between two different cpus until we found _why_ it failed. So, I am pretty sure that: a- doing it right is a lot of work now. b- doing it fast now is a lot of work now, and much more work later. Later, Juan. ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-09-04 9:54 ` Andrew Jones 2018-09-04 10:27 ` Juan Quintela @ 2018-09-04 10:32 ` Dr. David Alan Gilbert 2018-09-04 12:17 ` Peter Maydell 2018-09-05 11:46 ` Jaggi, Manish 2 siblings, 1 reply; 20+ messages in thread From: Dr. David Alan Gilbert @ 2018-09-04 10:32 UTC (permalink / raw) To: Andrew Jones Cc: Jaggi, Manish, Juan Quintela, Auger Eric, peter.maydell@linaro.org qemu-devel@nongnu.org * Andrew Jones (drjones@redhat.com) wrote: > On Tue, Sep 04, 2018 at 09:16:58AM +0000, Jaggi, Manish wrote: > > > On 31-Aug-2018, at 4:41 PM, Andrew Jones <drjones@redhat.com> wrote: > > > I think the sequence should look something like this: > > > > > > 1) Guest running on Host A with processor a > > > 2) Stop guest and save its state for migration > > > 3) Migrate guest to Host B with processor b (b is "close enough" to a) > > > 4) Restore guest state after migration > > > If guest is running with '-cpu host' > > > 4.a) Inform KVM of any configuration that impacts invariant registers > > > 4.b) Update the guest's view of all invariant registers to match the > > > host > > > EndIf > > > 5) Run guest > > > > > > 4.a and 4.b require new code both in QEMU and KVM. 4.a may require a new > > > KVM API, unless the existing API can be leveraged. > > > > > > The definition of "close enough" is left to the users and/or higher layers > > > of the Virt stack. > > > > > > > Thanks for detailing the sequence. > > I got another comment from Juan and David which is not to use -cpu host, see below > > > > "I really think that the right approach here is not using -cpu host. You > > do the full work, create a model as David says, and be sure that you car > > run that model on both cpus. It is a lot of work, but it is the only > > way to make sure that this is going to work long term.” > > > > Not using -cpu host is orthogonal to the sequence we have been discussing in this thread. > > Use something like -cpu cortex-a57 (this however does not work so far) > > This would avoid close-enough definition, but would need substantial work. > > > > So which approach should be taken here, whats your take... > > > > Inventing a base-AArch64 cpu model that can then be extended with optional > features is a nice way to extend the migratability of a guest, however > it's hard to do because of errata. Since errata workarounds are enabled > per MIDR, then we'd need to invent our own MIDR and also some way to > communicate which errata we want to enable, possibly through some paravirt > mechanism or through some implementation defined system registers that > KVM would need to reserve and define. > > That's not just a ton of work for the entire virt stack (not just KVM and > QEMU, but also all the layers above), but it's possible that it won't be > useful in the end anyway. There's risk that enabling just one erratum > workaround would restrict the guest to hosts of the exact same type > anyway. For each erratum that needs to be enabled, the probability of > enabling an incompatible one goes up, so it may not be likely to do much > better than '-cpu host' in the end. I'm afraid that until errata are > primarily showing up in optional CPU features that can simply be disabled > for the workaround, that we're stuck with '-cpu host'. I'd be happy to > discuss it more though. > > In short, I'd go with the proposal above, for now, with possibly one > change. libvirt folk (Andrea Bolognani and Pino Toscano) suggest that > the guest invariant register updating on the destination host only be > done if the user opts-in to it. This is because right now if a user > tries to migrate to a host that is not 100% identical the migration > will fail, which makes the "mistake" clear. If we silently change the > behavior to allow it, then what could have been a mistake, because > the hosts aren't actually "close enough", may go unnoticed. I'm not > 100% sure we need another user opt-in flag to be set, though, as I > think the '-cpu host' indicates the user expects the VCPU to look > like the host CPU, and even after migration that expectation should be > met. Simply, users that migrate '-cpu host' VMs need to know what they're > doing. The problem here is that in the x86 world we've said: 'don't use -cpu host if you're using it be careful and you need to know what you're doing' on ARM it's looking like the default; that means lots more people will use it who don't know what they're doing. At the very least you need to log cases like this (to stderr) when you detect them on the destination, so that when we get the innevitable 'guest crashed after migration' reports we can see the problem in the log. Dave > Thanks, > drew -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-09-04 10:32 ` Dr. David Alan Gilbert @ 2018-09-04 12:17 ` Peter Maydell 0 siblings, 0 replies; 20+ messages in thread From: Peter Maydell @ 2018-09-04 12:17 UTC (permalink / raw) To: Dr. David Alan Gilbert Cc: Andrew Jones, Auger Eric, Juan Quintela, peter.maydell@linaro.org qemu-devel@nongnu.org, Jaggi, Manish On 4 September 2018 at 11:32, Dr. David Alan Gilbert <dgilbert@redhat.com> wrote: > * Andrew Jones (drjones@redhat.com) wrote: >> Simply, users that migrate '-cpu host' VMs need to know what they're >> doing. > > The problem here is that in the x86 world we've said: > 'don't use -cpu host > if you're using it be careful and you need to know what you're doing' > > on ARM it's looking like the default; that means lots more people will > use it who don't know what they're doing. It is also the case that on Arm migration of -cpu host means you need to know what you're doing. But also, we don't support any kind of cross-cpu-type migration or "present cpu type A to the guest when running cpu type B", so a lot of the things you can do on x86 just aren't possible (yet) on Arm. The vague plan has been that we should fill in the gaps generally in line with what x86 does. But as Andrew says the errata-workaround-enablement issue is a bit tricky. thanks -- PMM ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-09-04 9:54 ` Andrew Jones 2018-09-04 10:27 ` Juan Quintela 2018-09-04 10:32 ` Dr. David Alan Gilbert @ 2018-09-05 11:46 ` Jaggi, Manish 2018-09-05 12:20 ` Andrew Jones 2 siblings, 1 reply; 20+ messages in thread From: Jaggi, Manish @ 2018-09-05 11:46 UTC (permalink / raw) To: Andrew Jones Cc: Jaggi, Manish, Juan Quintela, Dr. David Alan Gilbert, Auger Eric, peter.maydell@linaro.org qemu-devel@nongnu.org > On 04-Sep-2018, at 3:24 PM, Andrew Jones <drjones@redhat.com> wrote: > > External Email > > On Tue, Sep 04, 2018 at 09:16:58AM +0000, Jaggi, Manish wrote: >>> On 31-Aug-2018, at 4:41 PM, Andrew Jones <drjones@redhat.com> wrote: >>> I think the sequence should look something like this: >>> >>> 1) Guest running on Host A with processor a >>> 2) Stop guest and save its state for migration >>> 3) Migrate guest to Host B with processor b (b is "close enough" to a) >>> 4) Restore guest state after migration >>> If guest is running with '-cpu host' >>> 4.a) Inform KVM of any configuration that impacts invariant registers >>> 4.b) Update the guest's view of all invariant registers to match the >>> host >>> EndIf >>> 5) Run guest >>> >>> 4.a and 4.b require new code both in QEMU and KVM. 4.a may require a new >>> KVM API, unless the existing API can be leveraged. >>> >>> The definition of "close enough" is left to the users and/or higher layers >>> of the Virt stack. >>> >> >> Thanks for detailing the sequence. >> I got another comment from Juan and David which is not to use -cpu host, see below >> >> "I really think that the right approach here is not using -cpu host. You >> do the full work, create a model as David says, and be sure that you car >> run that model on both cpus. It is a lot of work, but it is the only >> way to make sure that this is going to work long term.” >> >> Not using -cpu host is orthogonal to the sequence we have been discussing in this thread. >> Use something like -cpu cortex-a57 (this however does not work so far) >> This would avoid close-enough definition, but would need substantial work. >> >> So which approach should be taken here, whats your take... >> > > Inventing a base-AArch64 cpu model that can then be extended with optional > features is a nice way to extend the migratability of a guest, however > it's hard to do because of errata. Since errata workarounds are enabled > per MIDR, then we'd need to invent our own MIDR and also some way to > communicate which errata we want to enable, possibly through some paravirt > mechanism or through some implementation defined system registers that > KVM would need to reserve and define. > > That's not just a ton of work for the entire virt stack (not just KVM and > QEMU, but also all the layers above), but it's possible that it won't be > useful in the end anyway. There's risk that enabling just one erratum > workaround would restrict the guest to hosts of the exact same type > anyway. For each erratum that needs to be enabled, the probability of > enabling an incompatible one goes up, so it may not be likely to do much > better than '-cpu host' in the end. I'm afraid that until errata are > primarily showing up in optional CPU features that can simply be disabled > for the workaround, that we're stuck with '-cpu host'. I'd be happy to > discuss it more though. > > In short, I'd go with the proposal above, for now, with possibly one > change. libvirt folk (Andrea Bolognani and Pino Toscano) suggest that > the guest invariant register updating on the destination host only be > done if the user opts-in to it. This is because right now if a user > tries to migrate to a host that is not 100% identical the migration > will fail, which makes the "mistake" clear. If we silently change the > behavior to allow it, then what could have been a mistake, because > the hosts aren't actually "close enough", may go unnoticed. I'm not > 100% sure we need another user opt-in flag to be set, though, as I > think the '-cpu host' indicates the user expects the VCPU to look > like the host CPU, and even after migration that expectation should be > met. Simply, users that migrate '-cpu host' VMs need to know what they're > doing. > Followup to the sequence you have outlined, I have added below pseudocode, This is only for processors which differ in invariant registers like MIDR but have no underlying CPU/GIC changes or any errata that affects guest kernel. And user opts in for it. (not sending full patch just to check if the logic is ok..) (a) Changes in KVM: - Introducing a specific error code (KVM_EINVARIANT) in case of invariant writes. This should not change anything to API SET_ONE_REG KVM API. Not sure which is the best place to put the define… I have added in include/uapi/linux/kvm_para.h. diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 22fbbdbece3c..c8a4fbe8a8bb 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1111,7 +1111,7 @@ static int __set_id_reg(const struct sys_reg_desc *rd, void __user *uaddr, /* This is what we mean by invariant: you can't change it. */ if (val != read_id_reg(rd, raz)) - return -EINVAL; + return -KVM_EINVARIANT; return 0; } diff --git a/include/uapi/linux/kvm_para.h b/include/uapi/linux/kvm_para.h index 6c0ce49931e5..3a49a321d0df 100644 --- a/include/uapi/linux/kvm_para.h +++ b/include/uapi/linux/kvm_para.h @@ -17,6 +17,7 @@ #define KVM_E2BIG E2BIG #define KVM_EPERM EPERM #define KVM_EOPNOTSUPP 95 +#define KVM_EINVARIANT 96 #define KVM_HC_VAPIC_POLL_IRQ 1 #define KVM_HC_MMU_OP 2 (b) Changes in Qemu code 1. Handling of new error code, which would update guest state with hosts invariant reg values. diff --git a/target/arm/kvm.c b/target/arm/kvm.c index 65f867d569..0cf14323a2 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -452,7 +452,15 @@ bool write_list_to_kvmstate(ARMCPU *cpu, int level) abort(); } ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &r); - if (ret) { + if (ignore_invariant && (ret == -KVM_EINVARIANT)) { + /* Update Guest invariant to match with migrated host regs*/ + ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &r); + if (ret) + ok = false; + else + cpu->cpreg_values[i] = r.addr; + } + else if (ret) { /* We might fail for "unknown register" and also for * "you tried to set a register which is constant with * a different value from what it actually contains". 2. ignore_invariant is the flag I was referring to which is what you also mentioned opt-in. This can be supplied as a command line parameter to qemu on Machine B. PS: I will add code to put warning logs as suggested by Dave. -manish > Thanks, > drew ^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-09-05 11:46 ` Jaggi, Manish @ 2018-09-05 12:20 ` Andrew Jones 2018-09-05 12:42 ` Jaggi, Manish 0 siblings, 1 reply; 20+ messages in thread From: Andrew Jones @ 2018-09-05 12:20 UTC (permalink / raw) To: Jaggi, Manish Cc: Juan Quintela, Dr. David Alan Gilbert, Auger Eric, peter.maydell@linaro.org qemu-devel@nongnu.org On Wed, Sep 05, 2018 at 11:46:11AM +0000, Jaggi, Manish wrote: > (a) Changes in KVM: > > - Introducing a specific error code (KVM_EINVARIANT) in case of invariant writes. > This should not change anything to API SET_ONE_REG KVM API. > Not sure which is the best place to put the define… > I have added in include/uapi/linux/kvm_para.h. > > diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c > index 22fbbdbece3c..c8a4fbe8a8bb 100644 > --- a/arch/arm64/kvm/sys_regs.c > +++ b/arch/arm64/kvm/sys_regs.c > @@ -1111,7 +1111,7 @@ static int __set_id_reg(const struct sys_reg_desc *rd, void __user *uaddr, > > /* This is what we mean by invariant: you can't change it. */ > if (val != read_id_reg(rd, raz)) > - return -EINVAL; > + return -KVM_EINVARIANT; > > return 0; > } > diff --git a/include/uapi/linux/kvm_para.h b/include/uapi/linux/kvm_para.h > index 6c0ce49931e5..3a49a321d0df 100644 > --- a/include/uapi/linux/kvm_para.h > +++ b/include/uapi/linux/kvm_para.h > @@ -17,6 +17,7 @@ > #define KVM_E2BIG E2BIG > #define KVM_EPERM EPERM > #define KVM_EOPNOTSUPP 95 > +#define KVM_EINVARIANT 96 > > #define KVM_HC_VAPIC_POLL_IRQ 1 > #define KVM_HC_MMU_OP 2 > > (b) Changes in Qemu code > > 1. Handling of new error code, which would update > guest state with hosts invariant reg values. > > diff --git a/target/arm/kvm.c b/target/arm/kvm.c > index 65f867d569..0cf14323a2 100644 > --- a/target/arm/kvm.c > +++ b/target/arm/kvm.c > @@ -452,7 +452,15 @@ bool write_list_to_kvmstate(ARMCPU *cpu, int level) > abort(); > } > ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &r); > - if (ret) { > + if (ignore_invariant && (ret == -KVM_EINVARIANT)) { > + /* Update Guest invariant to match with migrated host regs*/ > + ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &r); > + if (ret) > + ok = false; > + else > + cpu->cpreg_values[i] = r.addr; > + } > + else if (ret) { > /* We might fail for "unknown register" and also for > * "you tried to set a register which is constant with > * a different value from what it actually contains". > > 2. ignore_invariant is the flag I was referring to which is what you also > mentioned opt-in. This can be supplied as a command line parameter to qemu on Machine B. The same libvirt folk also suggest that a QMP command be provided that allows the selection of this ignore-invariant mode (in addition to or instead of the command line parameter.) Otherwise a guest that has already started without the parameter will not be able to migrate to a "close enough" host - even if it's decided later that it would be OK to do so. > > PS: I will add code to put warning logs as suggested by Dave. Yeah, I like that idea too. This approach looks good to me. Let's see what maintainers say when they see the patch submission. Thanks, drew ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-09-05 12:20 ` Andrew Jones @ 2018-09-05 12:42 ` Jaggi, Manish 2018-09-05 13:17 ` Andrew Jones 0 siblings, 1 reply; 20+ messages in thread From: Jaggi, Manish @ 2018-09-05 12:42 UTC (permalink / raw) To: Andrew Jones Cc: Jaggi, Manish, Juan Quintela, Dr. David Alan Gilbert, Auger Eric, peter.maydell@linaro.org qemu-devel@nongnu.org > On 05-Sep-2018, at 5:50 PM, Andrew Jones <drjones@redhat.com> wrote: > > External Email > > On Wed, Sep 05, 2018 at 11:46:11AM +0000, Jaggi, Manish wrote: >> (a) Changes in KVM: >> >> - Introducing a specific error code (KVM_EINVARIANT) in case of invariant writes. >> This should not change anything to API SET_ONE_REG KVM API. >> Not sure which is the best place to put the define… >> I have added in include/uapi/linux/kvm_para.h. >> >> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c >> index 22fbbdbece3c..c8a4fbe8a8bb 100644 >> --- a/arch/arm64/kvm/sys_regs.c >> +++ b/arch/arm64/kvm/sys_regs.c >> @@ -1111,7 +1111,7 @@ static int __set_id_reg(const struct sys_reg_desc *rd, void __user *uaddr, >> >> /* This is what we mean by invariant: you can't change it. */ >> if (val != read_id_reg(rd, raz)) >> - return -EINVAL; >> + return -KVM_EINVARIANT; >> >> return 0; >> } >> diff --git a/include/uapi/linux/kvm_para.h b/include/uapi/linux/kvm_para.h >> index 6c0ce49931e5..3a49a321d0df 100644 >> --- a/include/uapi/linux/kvm_para.h >> +++ b/include/uapi/linux/kvm_para.h >> @@ -17,6 +17,7 @@ >> #define KVM_E2BIG E2BIG >> #define KVM_EPERM EPERM >> #define KVM_EOPNOTSUPP 95 >> +#define KVM_EINVARIANT 96 >> >> #define KVM_HC_VAPIC_POLL_IRQ 1 >> #define KVM_HC_MMU_OP 2 >> >> (b) Changes in Qemu code >> >> 1. Handling of new error code, which would update >> guest state with hosts invariant reg values. >> >> diff --git a/target/arm/kvm.c b/target/arm/kvm.c >> index 65f867d569..0cf14323a2 100644 >> --- a/target/arm/kvm.c >> +++ b/target/arm/kvm.c >> @@ -452,7 +452,15 @@ bool write_list_to_kvmstate(ARMCPU *cpu, int level) >> abort(); >> } >> ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &r); >> - if (ret) { >> + if (ignore_invariant && (ret == -KVM_EINVARIANT)) { >> + /* Update Guest invariant to match with migrated host regs*/ >> + ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &r); >> + if (ret) >> + ok = false; >> + else >> + cpu->cpreg_values[i] = r.addr; >> + } >> + else if (ret) { >> /* We might fail for "unknown register" and also for >> * "you tried to set a register which is constant with >> * a different value from what it actually contains". >> >> 2. ignore_invariant is the flag I was referring to which is what you also >> mentioned opt-in. This can be supplied as a command line parameter to qemu on Machine B. > > The same libvirt folk also suggest that a QMP command be provided that > allows the selection of this ignore-invariant mode (in addition to or > instead of the command line parameter.) Otherwise a guest that has > already started without the parameter will not be able to migrate to > a "close enough” host ignore_invariant: This can be supplied as a command line parameter to qemu on Machine B. Machine A VM has to be migrated to B. > - even if it's decided later that it would be OK > to do so. > >> >> PS: I will add code to put warning logs as suggested by Dave. > > Yeah, I like that idea too. > > This approach looks good to me. Let's see what maintainers say when they > see the patch submission. Cool. Thanks. > > Thanks, > drew ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-09-05 12:42 ` Jaggi, Manish @ 2018-09-05 13:17 ` Andrew Jones 0 siblings, 0 replies; 20+ messages in thread From: Andrew Jones @ 2018-09-05 13:17 UTC (permalink / raw) To: Jaggi, Manish Cc: peter.maydell@linaro.org qemu-devel@nongnu.org, Auger Eric, Juan Quintela, Dr. David Alan Gilbert On Wed, Sep 05, 2018 at 12:42:05PM +0000, Jaggi, Manish wrote: > > > > On 05-Sep-2018, at 5:50 PM, Andrew Jones <drjones@redhat.com> wrote: > > > > External Email > > > > On Wed, Sep 05, 2018 at 11:46:11AM +0000, Jaggi, Manish wrote: > >> (a) Changes in KVM: > >> > >> - Introducing a specific error code (KVM_EINVARIANT) in case of invariant writes. > >> This should not change anything to API SET_ONE_REG KVM API. > >> Not sure which is the best place to put the define… > >> I have added in include/uapi/linux/kvm_para.h. > >> > >> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c > >> index 22fbbdbece3c..c8a4fbe8a8bb 100644 > >> --- a/arch/arm64/kvm/sys_regs.c > >> +++ b/arch/arm64/kvm/sys_regs.c > >> @@ -1111,7 +1111,7 @@ static int __set_id_reg(const struct sys_reg_desc *rd, void __user *uaddr, > >> > >> /* This is what we mean by invariant: you can't change it. */ > >> if (val != read_id_reg(rd, raz)) > >> - return -EINVAL; > >> + return -KVM_EINVARIANT; > >> > >> return 0; > >> } > >> diff --git a/include/uapi/linux/kvm_para.h b/include/uapi/linux/kvm_para.h > >> index 6c0ce49931e5..3a49a321d0df 100644 > >> --- a/include/uapi/linux/kvm_para.h > >> +++ b/include/uapi/linux/kvm_para.h > >> @@ -17,6 +17,7 @@ > >> #define KVM_E2BIG E2BIG > >> #define KVM_EPERM EPERM > >> #define KVM_EOPNOTSUPP 95 > >> +#define KVM_EINVARIANT 96 > >> > >> #define KVM_HC_VAPIC_POLL_IRQ 1 > >> #define KVM_HC_MMU_OP 2 > >> > >> (b) Changes in Qemu code > >> > >> 1. Handling of new error code, which would update > >> guest state with hosts invariant reg values. > >> > >> diff --git a/target/arm/kvm.c b/target/arm/kvm.c > >> index 65f867d569..0cf14323a2 100644 > >> --- a/target/arm/kvm.c > >> +++ b/target/arm/kvm.c > >> @@ -452,7 +452,15 @@ bool write_list_to_kvmstate(ARMCPU *cpu, int level) > >> abort(); > >> } > >> ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &r); > >> - if (ret) { > >> + if (ignore_invariant && (ret == -KVM_EINVARIANT)) { > >> + /* Update Guest invariant to match with migrated host regs*/ > >> + ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &r); > >> + if (ret) > >> + ok = false; > >> + else > >> + cpu->cpreg_values[i] = r.addr; > >> + } > >> + else if (ret) { > >> /* We might fail for "unknown register" and also for > >> * "you tried to set a register which is constant with > >> * a different value from what it actually contains". > >> > >> 2. ignore_invariant is the flag I was referring to which is what you also > >> mentioned opt-in. This can be supplied as a command line parameter to qemu on Machine B. > > > > The same libvirt folk also suggest that a QMP command be provided that > > allows the selection of this ignore-invariant mode (in addition to or > > instead of the command line parameter.) Otherwise a guest that has > > already started without the parameter will not be able to migrate to > > a "close enough” host > > ignore_invariant: This can be supplied as a command line parameter to qemu on Machine B. > Machine A VM has to be migrated to B. OK, I just confirmed with libvirt people that there's no need for the QMP command in this case, as the source host doesn't require any knowledge of it. The command line parameter (machine property?) must be discoverable by libvirt, though, so it can know whether or not the destination host's QEMU supports it. Thanks, drew ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-08-29 12:40 ` Jaggi, Manish 2018-08-29 13:16 ` Andrew Jones @ 2018-08-29 13:58 ` Dr. David Alan Gilbert 2018-08-31 9:41 ` Juan Quintela 1 sibling, 1 reply; 20+ messages in thread From: Dr. David Alan Gilbert @ 2018-08-29 13:58 UTC (permalink / raw) To: Jaggi, Manish Cc: Juan Quintela, Auger Eric, drjones, peter.maydell@linaro.org qemu-devel@nongnu.org, Anthony Liguori * Jaggi, Manish (Manish.Jaggi@cavium.com) wrote: > Just to add what happens in ARM64 case, qemu running on Machine A sends cpu state information to Machine B. > This state contains MIDR value, and so Processor ID value is compared in KVM and not in qemu (correcting myself). > > IIRC, Peter/Eric please point if there is something incorrect in the below flow... > > (Machine B) > target/arm/machine.c: cpu_post_load() > - updates cpu->cpreg_values[i] : which includes MIDR (processor ID register) > > - calls write_list_to_kvmstate(cpu, KVM_PUT_FULL_STATE) > > target/arm/kvm.c: write_list_to_kvmstate > - calls => kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &r); > > => and it eventually lands up IIRC in Linux code in > > => arch/arm64/kvm/sys_regs.c : set_invariant_sys_reg(u64 id, void __user *uaddr) > /* This is what we mean by invariant: you can't change it. */ > if (r->val != val) > return -EINVAL; > Note: MIDR_EL1 is invariant register. > result: Migration fails on Machine B. > > A few points: > - qemu on arm64 is invoked with -machine virt and -cpu as host. So we don't explicitly define which cpu. Note that even on x86 we don't guarantee as much about '-cpu host', what we expect to work for migration is that if you pick a '-cpu amodel' and both hosts support the feature flags required by 'amodel' then it should work. > - In case Machine A and Machine B have almost same Core and the delta may-not have any effect on qemu operation, migration should work by just looking into whitelist. > whitelist can be given as a parameter for qemu on machine B. > > qemu-system-aarch64 -whitelist <ids separated by commas> > > (This is my proposal) > > - So in cpu_post_load (Machine B) qemu can lookup whitelist and replace the MIDR with the one at Machine B. > Sounds good? > > - Juan raised a point about clock speed, I am not sure it will have any effect on arm since qemu is run with -cpu host param. > I could be wrong here, Peter/Eric can you please correct me... Clock speed is only really a problem for things like timestamp counters; some cores let you scale them; for those that don't then yes it's a bit odd. Dave > -Thanks > Manish > > > > > Dave > >> - is providing a -mirate-compat-whitelist <file> option for arm only looks good? > >> this option can be added in A1/A2 qemu command, so it would be upstream / downstream agnostic. > > > >>> > >>> I am sorry to not be able to tell you that this is an easy problem. > >>> > >>> Later, Juan. > >> > > -- > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [Qemu-devel] [Query] Live Migration between machines with different processor ids 2018-08-29 13:58 ` Dr. David Alan Gilbert @ 2018-08-31 9:41 ` Juan Quintela 0 siblings, 0 replies; 20+ messages in thread From: Juan Quintela @ 2018-08-31 9:41 UTC (permalink / raw) To: Dr. David Alan Gilbert Cc: Jaggi, Manish, Auger Eric, drjones, peter.maydell@linaro.org qemu-devel@nongnu.org, Anthony Liguori "Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote: > * Jaggi, Manish (Manish.Jaggi@cavium.com) wrote: > >> Just to add what happens in ARM64 case, qemu running on Machine A sends cpu state information to Machine B. >> This state contains MIDR value, and so Processor ID value is compared in KVM and not in qemu (correcting myself). >> >> IIRC, Peter/Eric please point if there is something incorrect in the below flow... >> >> (Machine B) >> target/arm/machine.c: cpu_post_load() >> - updates cpu->cpreg_values[i] : which includes MIDR (processor ID register) >> >> - calls write_list_to_kvmstate(cpu, KVM_PUT_FULL_STATE) >> >> target/arm/kvm.c: write_list_to_kvmstate >> - calls => kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &r); >> >> => and it eventually lands up IIRC in Linux code in >> >> => arch/arm64/kvm/sys_regs.c : set_invariant_sys_reg(u64 id, void __user *uaddr) >> /* This is what we mean by invariant: you can't change it. */ >> if (r->val != val) >> return -EINVAL; >> Note: MIDR_EL1 is invariant register. >> result: Migration fails on Machine B. >> >> A few points: >> - qemu on arm64 is invoked with -machine virt and -cpu as host. So we don't explicitly define which cpu. > > Note that even on x86 we don't guarantee as much about '-cpu host', what > we expect to work for migration is that if you pick a '-cpu amodel' > and both hosts support the feature flags required by 'amodel' then it > should work. I really think that the right approach here is not using -cpu host. You do the full work, create a model as David says, and be sure that you car run that model on both cpus. It is a lot of work, but it is the only way to make sure that this is going to work long term. >> - In case Machine A and Machine B have almost same Core and the >> delta may-not have any effect on qemu operation, migration should >> work by just looking into whitelist. >> whitelist can be given as a parameter for qemu on machine B. >> >> qemu-system-aarch64 -whitelist <ids separated by commas> >> >> (This is my proposal) >> >> - So in cpu_post_load (Machine B) qemu can lookup whitelist and >> replace the MIDR with the one at Machine B. >> Sounds good? >> >> - Juan raised a point about clock speed, I am not sure it will have >> any effect on arm since qemu is run with -cpu host param. >> I could be wrong here, Peter/Eric can you please correct me... > > Clock speed is only really a problem for things like timestamp counters; > some cores let you scale them; for those that don't then yes it's a bit > odd. This was just one example that I thought on top of my head. Other thing that I remember was that at some point, if you migrated to a cpu with hyperthreading, you lost half of your performance counters. I still think that you need to create a proper model, and work from there. Later, Juan. ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2018-09-05 13:17 UTC | newest] Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-08-21 6:39 [Qemu-devel] Live Migration between machines with different processor ids Mjaggi Oss 2018-08-23 11:18 ` [Qemu-devel] [Query] " Jaggi, Manish 2018-08-23 14:29 ` Juan Quintela 2018-08-24 9:24 ` Jaggi, Manish 2018-08-28 17:27 ` Dr. David Alan Gilbert 2018-08-29 12:40 ` Jaggi, Manish 2018-08-29 13:16 ` Andrew Jones 2018-08-31 9:52 ` Jaggi, Manish 2018-08-31 11:11 ` Andrew Jones 2018-09-04 9:16 ` Jaggi, Manish 2018-09-04 9:54 ` Andrew Jones 2018-09-04 10:27 ` Juan Quintela 2018-09-04 10:32 ` Dr. David Alan Gilbert 2018-09-04 12:17 ` Peter Maydell 2018-09-05 11:46 ` Jaggi, Manish 2018-09-05 12:20 ` Andrew Jones 2018-09-05 12:42 ` Jaggi, Manish 2018-09-05 13:17 ` Andrew Jones 2018-08-29 13:58 ` Dr. David Alan Gilbert 2018-08-31 9:41 ` Juan Quintela
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.