* Re: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support [not found] <MN2PR18MB26869A6CA4E67558324F655CC5C70@MN2PR18MB2686.namprd18.prod.outlook.com> @ 2020-04-03 8:02 ` Marc Zyngier 2020-04-11 4:10 ` Zengtao (B) 0 siblings, 1 reply; 12+ messages in thread From: Marc Zyngier @ 2020-04-03 8:02 UTC (permalink / raw) To: George Cherian Cc: kvmarm, Ganapatrao Kulkarni, kvm, suzuki.poulose, andre.przywara, christoffer.dall, Dave.Martin, james.morse, linux-arm-kernel, Anil Kumar Reddy H, alexandru.elisei, jintack, julien.thierry.kdev Hi George, On 2020-04-03 08:27, George Cherian wrote: > Hi Marc, > > On 2/11/20 9:48 AM, Marc Zyngier wrote: >> This is a major rework of the NV series that I posted over 6 months >> ago[1], and a lot has changed since then: >> >> - Early ARMv8.4-NV support >> - ARMv8.4-TTL support in host and guest >> - ARMv8.5-GTG support in host and guest >> - Lots of comments addressed after the review >> - Rebased on v5.6-rc1 >> - Way too many patches >> >> In my defence, the whole of the NV code is still smaller that the >> 32bit KVM/arm code I'm about to remove, so I feel less bad inflicting >> this on everyone! ;-) >> >> >From a functionality perspective, you can expect a L2 guest to work, >> but don't even think of L3, as we only partially emulate the >> ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, PMU, >> as well as anything that would require a Stage-1 PTW. What we want to >> achieve is that with NV disabled, there is no performance overhead and >> no regression. >> >> The series is roughly divided in 5 parts: exception handling, memory >> virtualization, interrupts and timers for ARMv8.3, followed by the >> ARMv8.4 support. There are of course some dependencies, but you'll >> hopefully get the gist of it. >> >> For the most courageous of you, I've put out a branch[2]. Of course, >> you'll need some userspace. Andre maintains a hacked version of >> kvmtool[3] that takes a --nested option, allowing the guest to be >> started at EL2. You can run the whole stack in the Foundation >> model. Don't be in a hurry ;-). >> > The full series was tested on both Foundation model as well as Marvell > ThunderX3 > Emulation Platform. > Basic boot testing done for Guest Hypervisor and Guest Guest. > > Tested-by: George Cherian <george.cherian@marvell.com> Thanks for having given this a go. However, without more details, it is pretty hard to find out what you have tested. What sort of guest have you booted, with what configuration, what workloads did you run in the L2 guests and what are the architectural features that TX3 implements? The last point is specially important, as the NV architecture spans two major revisions of the architecture and affects tons of other extensions that are themselves optional. Without any detail on that front, I have no idea what the coverage of your testing is. Thanks, M. -- Jazz is not dead. It just smells funny... _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support 2020-04-03 8:02 ` [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support Marc Zyngier @ 2020-04-11 4:10 ` Zengtao (B) 2020-04-11 9:23 ` Marc Zyngier 0 siblings, 1 reply; 12+ messages in thread From: Zengtao (B) @ 2020-04-11 4:10 UTC (permalink / raw) To: Marc Zyngier, George Cherian Cc: kvmarm, Ganapatrao Kulkarni, kvm, suzuki.poulose, andre.przywara, christoffer.dall, Dave.Martin, james.morse, linux-arm-kernel, Anil Kumar Reddy H, alexandru.elisei, jintack, julien.thierry.kdev Hi Marc: Since it's a very large patch series, I want to test it on my platform which don't support nv, and want to make sure if this patch series affects the existed virtualization functions or not. Any suggestion about the test focus? Thanks. > -----Original Message----- > From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] > On Behalf Of Marc Zyngier > Sent: Friday, April 03, 2020 4:02 PM > To: George Cherian > Cc: Dave.Martin@arm.com; alexandru.elisei@arm.com; > andre.przywara@arm.com; christoffer.dall@arm.com; > james.morse@arm.com; jintack@cs.columbia.edu; > julien.thierry.kdev@gmail.com; kvm@vger.kernel.org; > kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org; > suzuki.poulose@arm.com; Anil Kumar Reddy H; Ganapatrao Kulkarni > Subject: Re: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested > Virtualization support > > Hi George, > > On 2020-04-03 08:27, George Cherian wrote: > > Hi Marc, > > > > On 2/11/20 9:48 AM, Marc Zyngier wrote: > >> This is a major rework of the NV series that I posted over 6 months > >> ago[1], and a lot has changed since then: > >> > >> - Early ARMv8.4-NV support > >> - ARMv8.4-TTL support in host and guest > >> - ARMv8.5-GTG support in host and guest > >> - Lots of comments addressed after the review > >> - Rebased on v5.6-rc1 > >> - Way too many patches > >> > >> In my defence, the whole of the NV code is still smaller that the > >> 32bit KVM/arm code I'm about to remove, so I feel less bad inflicting > >> this on everyone! ;-) > >> > >> >From a functionality perspective, you can expect a L2 guest to work, > >> but don't even think of L3, as we only partially emulate the > >> ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, > PMU, > >> as well as anything that would require a Stage-1 PTW. What we want > to > >> achieve is that with NV disabled, there is no performance overhead > and > >> no regression. > >> > >> The series is roughly divided in 5 parts: exception handling, memory > >> virtualization, interrupts and timers for ARMv8.3, followed by the > >> ARMv8.4 support. There are of course some dependencies, but you'll > >> hopefully get the gist of it. > >> > >> For the most courageous of you, I've put out a branch[2]. Of course, > >> you'll need some userspace. Andre maintains a hacked version of > >> kvmtool[3] that takes a --nested option, allowing the guest to be > >> started at EL2. You can run the whole stack in the Foundation > >> model. Don't be in a hurry ;-). > >> > > The full series was tested on both Foundation model as well as Marvell > > ThunderX3 > > Emulation Platform. > > Basic boot testing done for Guest Hypervisor and Guest Guest. > > > > Tested-by: George Cherian <george.cherian@marvell.com> > > Thanks for having given this a go. > > However, without more details, it is pretty hard to find out what you > have tested. > What sort of guest have you booted, with what configuration, what > workloads did you > run in the L2 guests and what are the architectural features that TX3 > implements? > > The last point is specially important, as the NV architecture spans two > major > revisions of the architecture and affects tons of other extensions that > are > themselves optional. Without any detail on that front, I have no idea > what the > coverage of your testing is. > > Thanks, > > M. > -- > Jazz is not dead. It just smells funny... _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support 2020-04-11 4:10 ` Zengtao (B) @ 2020-04-11 9:23 ` Marc Zyngier 2020-04-16 1:38 ` Zengtao (B) 0 siblings, 1 reply; 12+ messages in thread From: Marc Zyngier @ 2020-04-11 9:23 UTC (permalink / raw) To: Zengtao (B) Cc: kvmarm, Ganapatrao Kulkarni, kvm, suzuki.poulose, andre.przywara, christoffer.dall, jintack, George Cherian, james.morse, linux-arm-kernel, Anil Kumar Reddy H, alexandru.elisei, Dave.Martin, julien.thierry.kdev Hi Zengtao, On Sat, 11 Apr 2020 05:10:05 +0100, "Zengtao (B)" <prime.zeng@hisilicon.com> wrote: > > Hi Marc: > > Since it's a very large patch series, I want to test it on my platform > which don't support nv, and want to make sure if this patch series > affects the existed virtualization functions or not. > > Any suggestion about the test focus? Not really. Given that the NV patches affect absolutely every architectural parts of KVM/arm64, everything needs careful testing. But more than testing, it needs reviewing. Thanks, M. -- Jazz is not dead, it just smells funny. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support 2020-04-11 9:23 ` Marc Zyngier @ 2020-04-16 1:38 ` Zengtao (B) 2020-04-16 18:22 ` Marc Zyngier 2020-04-17 15:11 ` Alexandru Elisei 0 siblings, 2 replies; 12+ messages in thread From: Zengtao (B) @ 2020-04-16 1:38 UTC (permalink / raw) To: Marc Zyngier Cc: kvmarm, Ganapatrao Kulkarni, kvm, suzuki.poulose, andre.przywara, christoffer.dall, jintack, George Cherian, james.morse, linux-arm-kernel, Anil Kumar Reddy H, alexandru.elisei, Dave.Martin, julien.thierry.kdev Hi Marc: Got it. Really a bit patch set :) BTW, I have done a basic kvm unit test git://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git And I find that after apply the patch KVM: arm64: VNCR-ize ELR_EL1, The psci test failed for some reason, I can't understand why, this is only the test result.(find the patch by git bisect + kvm test) My platform: Hisilicon D06 board. Linux kernel: Linux 5.6-rc6 + nv patches(some rebases) Could you help to take a look? Thanks Zengtao > -----Original Message----- > From: Marc Zyngier [mailto:maz@kernel.org] > Sent: Saturday, April 11, 2020 5:24 PM > To: Zengtao (B) > Cc: George Cherian; Dave.Martin@arm.com; alexandru.elisei@arm.com; > andre.przywara@arm.com; christoffer.dall@arm.com; > james.morse@arm.com; jintack@cs.columbia.edu; > julien.thierry.kdev@gmail.com; kvm@vger.kernel.org; > kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org; > suzuki.poulose@arm.com; Anil Kumar Reddy H; Ganapatrao Kulkarni > Subject: Re: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested > Virtualization support > > Hi Zengtao, > > On Sat, 11 Apr 2020 05:10:05 +0100, > "Zengtao (B)" <prime.zeng@hisilicon.com> wrote: > > > > Hi Marc: > > > > Since it's a very large patch series, I want to test it on my platform > > which don't support nv, and want to make sure if this patch series > > affects the existed virtualization functions or not. > > > > Any suggestion about the test focus? > > Not really. Given that the NV patches affect absolutely every > architectural parts of KVM/arm64, everything needs careful > testing. But more than testing, it needs reviewing. > > Thanks, > > M. > > -- > Jazz is not dead, it just smells funny. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support 2020-04-16 1:38 ` Zengtao (B) @ 2020-04-16 18:22 ` Marc Zyngier 2020-04-17 15:06 ` Marc Zyngier 2020-04-17 15:11 ` Alexandru Elisei 1 sibling, 1 reply; 12+ messages in thread From: Marc Zyngier @ 2020-04-16 18:22 UTC (permalink / raw) To: Zengtao (B) Cc: kvmarm, Ganapatrao Kulkarni, kvm, suzuki.poulose, andre.przywara, christoffer.dall, jintack, George Cherian, james.morse, linux-arm-kernel, Anil Kumar Reddy H, alexandru.elisei, Dave.Martin, julien.thierry.kdev Hi Zengtao, On 2020-04-16 02:38, Zengtao (B) wrote: > Hi Marc: > > Got it. > Really a bit patch set :) Well, yeah... ;-) > > BTW, I have done a basic kvm unit test > git://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git > And I find that after apply the patch KVM: arm64: VNCR-ize ELR_EL1, > The psci test failed for some reason, I can't understand why, this > is only the test result.(find the patch by git bisect + kvm test) That it is that mechanical, we should be able to quickly nail that one. > My platform: Hisilicon D06 board. > Linux kernel: Linux 5.6-rc6 + nv patches(some rebases) > Could you help to take a look? I'll have a look tomorrow. I'm in the middle of refactoring the series for 5.7, and things have changed quite a bit. Hopefully this isn't a VHE vs non-VHE issue. Thanks, M. -- Jazz is not dead. It just smells funny... _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support 2020-04-16 18:22 ` Marc Zyngier @ 2020-04-17 15:06 ` Marc Zyngier 2020-04-18 2:49 ` Zengtao (B) 0 siblings, 1 reply; 12+ messages in thread From: Marc Zyngier @ 2020-04-17 15:06 UTC (permalink / raw) To: Zengtao (B) Cc: kvmarm, Ganapatrao Kulkarni, kvm, suzuki.poulose, andre.przywara, christoffer.dall, jintack, George Cherian, james.morse, linux-arm-kernel, Anil Kumar Reddy H, alexandru.elisei, Dave.Martin, julien.thierry.kdev On Thu, 16 Apr 2020 19:22:21 +0100 Marc Zyngier <maz@kernel.org> wrote: > Hi Zengtao, > > On 2020-04-16 02:38, Zengtao (B) wrote: > > Hi Marc: > > > > Got it. > > Really a bit patch set :) > > Well, yeah... ;-) > > > > > BTW, I have done a basic kvm unit test > > git://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git > > And I find that after apply the patch KVM: arm64: VNCR-ize ELR_EL1, > > The psci test failed for some reason, I can't understand why, this > > is only the test result.(find the patch by git bisect + kvm test) > > That it is that mechanical, we should be able to quickly nail that one. > > > My platform: Hisilicon D06 board. > > Linux kernel: Linux 5.6-rc6 + nv patches(some rebases) > > Could you help to take a look? > > I'll have a look tomorrow. I'm in the middle of refactoring the series > for 5.7, and things have changed quite a bit. Hopefully this isn't a VHE > vs non-VHE issue. So I've repeatedly tried with the current state of the NV patches[1], on both an ARMv8.0 system (Seattle) and an ARMv8.2 pile of putrid junk (vim3l). PSCI is pretty happy, although I can only test with at most 8 vcpus (GICv2 gets in the way). Can you please: - post the detailed error by running the PSCI unit test on its own - test with the current state of the patches Thanks, M. [1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/nv-5.7-rc1-WIP -- Jazz is not dead. It just smells funny... _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support 2020-04-17 15:06 ` Marc Zyngier @ 2020-04-18 2:49 ` Zengtao (B) 2020-04-20 14:03 ` Marc Zyngier 0 siblings, 1 reply; 12+ messages in thread From: Zengtao (B) @ 2020-04-18 2:49 UTC (permalink / raw) To: Marc Zyngier Cc: kvmarm, Ganapatrao Kulkarni, kvm, suzuki.poulose, andre.przywara, christoffer.dall, jintack, George Cherian, james.morse, linux-arm-kernel, Anil Kumar Reddy H, alexandru.elisei, Dave.Martin, julien.thierry.kdev -----Original Message----- > From: Marc Zyngier [mailto:maz@kernel.org] > Sent: Friday, April 17, 2020 11:06 PM > To: Zengtao (B) > Cc: George Cherian; Dave.Martin@arm.com; alexandru.elisei@arm.com; > andre.przywara@arm.com; christoffer.dall@arm.com; > james.morse@arm.com; jintack@cs.columbia.edu; > julien.thierry.kdev@gmail.com; kvm@vger.kernel.org; > kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org; > suzuki.poulose@arm.com; Anil Kumar Reddy H; Ganapatrao Kulkarni > Subject: Re: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested > Virtualization support > > On Thu, 16 Apr 2020 19:22:21 +0100 > Marc Zyngier <maz@kernel.org> wrote: > > > Hi Zengtao, > > > > On 2020-04-16 02:38, Zengtao (B) wrote: > > > Hi Marc: > > > > > > Got it. > > > Really a bit patch set :) > > > > Well, yeah... ;-) > > > > > > > > BTW, I have done a basic kvm unit test > > > git://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git > > > And I find that after apply the patch KVM: arm64: VNCR-ize ELR_EL1, > > > The psci test failed for some reason, I can't understand why, this > > > is only the test result.(find the patch by git bisect + kvm test) > > > > That it is that mechanical, we should be able to quickly nail that one. > > > > > My platform: Hisilicon D06 board. > > > Linux kernel: Linux 5.6-rc6 + nv patches(some rebases) > > > Could you help to take a look? > > > > I'll have a look tomorrow. I'm in the middle of refactoring the series > > for 5.7, and things have changed quite a bit. Hopefully this isn't a VHE > > vs non-VHE issue. > > So I've repeatedly tried with the current state of the NV patches[1], > on both an ARMv8.0 system (Seattle) and an ARMv8.2 pile of putrid junk > (vim3l). PSCI is pretty happy, although I can only test with at most 8 > vcpus (GICv2 gets in the way). > > Can you please: > > - post the detailed error by running the PSCI unit test on its own I tried to trace the error, and I found in kernel function kvm_mpidr_to_vcpu, casually, mpidr returns zero and we can't get the expected vcpu, and psci test failed due to this. And as I mentioned in my last before, the psci error is introduced by the patch KVM: arm64: VNCR-ize ELR_EL1.(Only test result) Maybe you have to try tens of times to reproduce. :) Deep into the patch itself, I don't find any connection between the patch and the issue. > - test with the current state of the patches I test with your nv-5.7-rc1-WIP branch and latest kvm_unit_test, the error still exist. Thanks. zengtao > > Thanks, > > M. > > [1] > https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/l > og/?h=kvm-arm64/nv-5.7-rc1-WIP > -- > Jazz is not dead. It just smells funny... _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support 2020-04-18 2:49 ` Zengtao (B) @ 2020-04-20 14:03 ` Marc Zyngier 2020-04-21 2:22 ` Zengtao (B) 0 siblings, 1 reply; 12+ messages in thread From: Marc Zyngier @ 2020-04-20 14:03 UTC (permalink / raw) To: Zengtao (B) Cc: kvmarm, Ganapatrao Kulkarni, kvm, suzuki.poulose, andre.przywara, christoffer.dall, jintack, George Cherian, james.morse, linux-arm-kernel, Anil Kumar Reddy H, alexandru.elisei, Dave.Martin, julien.thierry.kdev On 2020-04-18 03:49, Zengtao (B) wrote: > -----Original Message----- >> From: Marc Zyngier [mailto:maz@kernel.org] >> Sent: Friday, April 17, 2020 11:06 PM >> To: Zengtao (B) >> Cc: George Cherian; Dave.Martin@arm.com; alexandru.elisei@arm.com; >> andre.przywara@arm.com; christoffer.dall@arm.com; >> james.morse@arm.com; jintack@cs.columbia.edu; >> julien.thierry.kdev@gmail.com; kvm@vger.kernel.org; >> kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org; >> suzuki.poulose@arm.com; Anil Kumar Reddy H; Ganapatrao Kulkarni >> Subject: Re: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested >> Virtualization support >> >> On Thu, 16 Apr 2020 19:22:21 +0100 >> Marc Zyngier <maz@kernel.org> wrote: >> >> > Hi Zengtao, >> > >> > On 2020-04-16 02:38, Zengtao (B) wrote: >> > > Hi Marc: >> > > >> > > Got it. >> > > Really a bit patch set :) >> > >> > Well, yeah... ;-) >> > >> > > >> > > BTW, I have done a basic kvm unit test >> > > git://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git >> > > And I find that after apply the patch KVM: arm64: VNCR-ize ELR_EL1, >> > > The psci test failed for some reason, I can't understand why, this >> > > is only the test result.(find the patch by git bisect + kvm test) >> > >> > That it is that mechanical, we should be able to quickly nail that one. >> > >> > > My platform: Hisilicon D06 board. >> > > Linux kernel: Linux 5.6-rc6 + nv patches(some rebases) >> > > Could you help to take a look? >> > >> > I'll have a look tomorrow. I'm in the middle of refactoring the series >> > for 5.7, and things have changed quite a bit. Hopefully this isn't a VHE >> > vs non-VHE issue. >> >> So I've repeatedly tried with the current state of the NV patches[1], >> on both an ARMv8.0 system (Seattle) and an ARMv8.2 pile of putrid junk >> (vim3l). PSCI is pretty happy, although I can only test with at most 8 >> vcpus (GICv2 gets in the way). >> >> Can you please: >> >> - post the detailed error by running the PSCI unit test on its own > I tried to trace the error, and I found in kernel function > kvm_mpidr_to_vcpu, > casually, mpidr returns zero and we can't get the expected vcpu, and > psci > test failed due to this. Can you post the exact error message from the unit test? > And as I mentioned in my last before, the psci error is introduced by > the > patch KVM: arm64: VNCR-ize ELR_EL1.(Only test result) > Maybe you have to try tens of times to reproduce. :) > Deep into the patch itself, I don't find any connection between the > patch > and the issue. Me neither, and I haven't managed to reproduce your issue. >> - test with the current state of the patches > I test with your nv-5.7-rc1-WIP branch and latest kvm_unit_test, the > error still exist. How many vcpus do you create with this PSCI test? Thanks, M. -- Jazz is not dead. It just smells funny... _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 12+ messages in thread
* RE: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support 2020-04-20 14:03 ` Marc Zyngier @ 2020-04-21 2:22 ` Zengtao (B) 0 siblings, 0 replies; 12+ messages in thread From: Zengtao (B) @ 2020-04-21 2:22 UTC (permalink / raw) To: Marc Zyngier Cc: kvmarm, Ganapatrao Kulkarni, kvm, suzuki.poulose, andre.przywara, christoffer.dall, jintack, George Cherian, james.morse, linux-arm-kernel, Anil Kumar Reddy H, alexandru.elisei, Dave.Martin, julien.thierry.kdev > -----Original Message----- > From: kvm-owner@vger.kernel.org [mailto:kvm-owner@vger.kernel.org] > On Behalf Of Marc Zyngier > Sent: Monday, April 20, 2020 10:03 PM > To: Zengtao (B) > Cc: George Cherian; Dave.Martin@arm.com; alexandru.elisei@arm.com; > andre.przywara@arm.com; christoffer.dall@arm.com; > james.morse@arm.com; jintack@cs.columbia.edu; > julien.thierry.kdev@gmail.com; kvm@vger.kernel.org; > kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org; > suzuki.poulose@arm.com; Anil Kumar Reddy H; Ganapatrao Kulkarni > Subject: Re: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested > Virtualization support > > On 2020-04-18 03:49, Zengtao (B) wrote: > > -----Original Message----- > >> From: Marc Zyngier [mailto:maz@kernel.org] > >> Sent: Friday, April 17, 2020 11:06 PM > >> To: Zengtao (B) > >> Cc: George Cherian; Dave.Martin@arm.com; > alexandru.elisei@arm.com; > >> andre.przywara@arm.com; christoffer.dall@arm.com; > >> james.morse@arm.com; jintack@cs.columbia.edu; > >> julien.thierry.kdev@gmail.com; kvm@vger.kernel.org; > >> kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org; > >> suzuki.poulose@arm.com; Anil Kumar Reddy H; Ganapatrao Kulkarni > >> Subject: Re: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested > >> Virtualization support > >> > >> On Thu, 16 Apr 2020 19:22:21 +0100 > >> Marc Zyngier <maz@kernel.org> wrote: > >> > >> > Hi Zengtao, > >> > > >> > On 2020-04-16 02:38, Zengtao (B) wrote: > >> > > Hi Marc: > >> > > > >> > > Got it. > >> > > Really a bit patch set :) > >> > > >> > Well, yeah... ;-) > >> > > >> > > > >> > > BTW, I have done a basic kvm unit test > >> > > git://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git > >> > > And I find that after apply the patch KVM: arm64: VNCR-ize > ELR_EL1, > >> > > The psci test failed for some reason, I can't understand why, this > >> > > is only the test result.(find the patch by git bisect + kvm test) > >> > > >> > That it is that mechanical, we should be able to quickly nail that > one. > >> > > >> > > My platform: Hisilicon D06 board. > >> > > Linux kernel: Linux 5.6-rc6 + nv patches(some rebases) > >> > > Could you help to take a look? > >> > > >> > I'll have a look tomorrow. I'm in the middle of refactoring the series > >> > for 5.7, and things have changed quite a bit. Hopefully this isn't a > VHE > >> > vs non-VHE issue. > >> > >> So I've repeatedly tried with the current state of the NV patches[1], > >> on both an ARMv8.0 system (Seattle) and an ARMv8.2 pile of putrid > junk > >> (vim3l). PSCI is pretty happy, although I can only test with at most 8 > >> vcpus (GICv2 gets in the way). > >> > >> Can you please: > >> > >> - post the detailed error by running the PSCI unit test on its own > > I tried to trace the error, and I found in kernel function > > kvm_mpidr_to_vcpu, > > casually, mpidr returns zero and we can't get the expected vcpu, and > > psci > > test failed due to this. > > Can you post the exact error message from the unit test? > Some debug code added as follow(virt/kvm/arm/arm.c): unsigned long saved_mpidr[256]; static void dump_saved_mpidr(struct kvm *kvm, unsigned long mpidr) { struct kvm_vcpu *vcpu; int i; printk("target mpidr:%lx\n", mpidr); kvm_for_each_vcpu(i, vcpu, kvm) { printk("saved_mpidr:%lx latest mpidr:%lx\n", saved_mpidr[i], kvm_vcpu_get_mpidr_aff(vcpu)); } } struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr) { struct kvm_vcpu *vcpu; int i; mpidr &= MPIDR_HWID_BITMASK; kvm_for_each_vcpu(i, vcpu, kvm) { saved_mpidr[i] = kvm_vcpu_get_mpidr_aff(vcpu); if (mpidr == saved_mpidr[i]) return vcpu; } dump_saved_mpidr(kvm, mpidr); return NULL; } error log: [root@localhost test]# ./psci BUILD_HEAD=b16df9ee timeout -k 1s --foreground 90s /sbin/qemu-system-aarch64 -nodefaults -machine virt,gic-version=host,accel=kvm -cpu host -device virtio-serial-device -device virtconsole,chardev=ctd -chardev testdev,id=ctd -device pci-testdev -display none -serial stdio -kernel /tmp/tmp.QDQH5cBotg -smp 128 # -initrd /tmp/tmp.uW5pheTN1jINFO: psci: PSCI version 1.0 PASS: psci: invalid-function PASS: psci: affinity-info-on PASS: psci: affinity-info-off [ 218.891944] target mpidr:1 [ 218.894645] saved_mpidr:0 latest mpidr:0 [ 218.898569] saved_mpidr:0 latest mpidr:1 [ 218.902481] saved_mpidr:2 latest mpidr:2 [ 218.906393] saved_mpidr:3 latest mpidr:3 [ 218.910308] saved_mpidr:4 latest mpidr:4 [ 218.914223] saved_mpidr:5 latest mpidr:5 [ 218.918138] saved_mpidr:6 latest mpidr:6 [ 218.922051] saved_mpidr:7 latest mpidr:7 [ 218.925965] saved_mpidr:8 latest mpidr:8 [ 218.929878] saved_mpidr:9 latest mpidr:9 [ 218.933788] saved_mpidr:a latest mpidr:a [ 218.937703] saved_mpidr:b latest mpidr:b [ 218.941618] saved_mpidr:c latest mpidr:c [ 218.945533] saved_mpidr:d latest mpidr:d [ 218.949445] saved_mpidr:e latest mpidr:e [ 218.953359] saved_mpidr:f latest mpidr:f [ 218.957274] saved_mpidr:100 latest mpidr:100 [ 218.961536] saved_mpidr:101 latest mpidr:101 [ 218.965798] saved_mpidr:102 latest mpidr:102 [ 218.970059] saved_mpidr:103 latest mpidr:103 [ 218.974319] saved_mpidr:104 latest mpidr:104 [ 218.978584] saved_mpidr:105 latest mpidr:105 [ 218.982842] saved_mpidr:106 latest mpidr:106 [ 218.987105] saved_mpidr:107 latest mpidr:107 [ 218.991362] saved_mpidr:108 latest mpidr:108 [ 218.995624] saved_mpidr:109 latest mpidr:109 [ 218.999881] saved_mpidr:10a latest mpidr:10a [ 219.004133] saved_mpidr:10b latest mpidr:10b [ 219.008389] saved_mpidr:10c latest mpidr:10c [ 219.012643] saved_mpidr:10d latest mpidr:10d [ 219.016898] saved_mpidr:10e latest mpidr:10e [ 219.021152] saved_mpidr:10f latest mpidr:10f [ 219.025408] saved_mpidr:200 latest mpidr:200 [ 219.029664] saved_mpidr:201 latest mpidr:201 [ 219.033919] saved_mpidr:202 latest mpidr:202 [ 219.038171] saved_mpidr:203 latest mpidr:203 [ 219.042426] saved_mpidr:204 latest mpidr:204 [ 219.046681] saved_mpidr:205 latest mpidr:205 [ 219.050934] saved_mpidr:206 latest mpidr:206 [ 219.055188] saved_mpidr:207 latest mpidr:207 [ 219.059442] saved_mpidr:208 latest mpidr:208 [ 219.063698] saved_mpidr:209 latest mpidr:209 [ 219.067952] saved_mpidr:20a latest mpidr:20a [ 219.072206] saved_mpidr:20b latest mpidr:20b [ 219.076463] saved_mpidr:20c latest mpidr:20c [ 219.080722] saved_mpidr:20d latest mpidr:20d [ 219.084976] saved_mpidr:20e latest mpidr:20e [ 219.089229] saved_mpidr:20f latest mpidr:20f [ 219.093483] saved_mpidr:300 latest mpidr:300 [ 219.097738] saved_mpidr:301 latest mpidr:301 [ 219.101994] saved_mpidr:302 latest mpidr:302 [ 219.106250] saved_mpidr:303 latest mpidr:303 [ 219.110504] saved_mpidr:304 latest mpidr:304 [ 219.114756] saved_mpidr:305 latest mpidr:305 [ 219.119010] saved_mpidr:306 latest mpidr:306 [ 219.123267] saved_mpidr:307 latest mpidr:307 [ 219.127521] saved_mpidr:308 latest mpidr:308 [ 219.131779] saved_mpidr:309 latest mpidr:309 [ 219.136034] saved_mpidr:30a latest mpidr:30a [ 219.140289] saved_mpidr:30b latest mpidr:30b [ 219.144544] saved_mpidr:30c latest mpidr:30c [ 219.148798] saved_mpidr:30d latest mpidr:30d [ 219.153054] saved_mpidr:30e latest mpidr:30e [ 219.157308] saved_mpidr:30f latest mpidr:30f [ 219.161562] saved_mpidr:400 latest mpidr:400 [ 219.165817] saved_mpidr:401 latest mpidr:401 [ 219.170071] saved_mpidr:402 latest mpidr:402 [ 219.174324] saved_mpidr:403 latest mpidr:403 [ 219.178578] saved_mpidr:404 latest mpidr:404 [ 219.182834] saved_mpidr:405 latest mpidr:405 [ 219.187093] saved_mpidr:406 latest mpidr:406 [ 219.191350] saved_mpidr:407 latest mpidr:407 [ 219.195609] saved_mpidr:408 latest mpidr:408 [ 219.199861] saved_mpidr:409 latest mpidr:409 [ 219.204117] saved_mpidr:40a latest mpidr:40a [ 219.208371] saved_mpidr:40b latest mpidr:40b [ 219.212625] saved_mpidr:40c latest mpidr:40c [ 219.216881] saved_mpidr:40d latest mpidr:40d [ 219.221133] saved_mpidr:40e latest mpidr:40e [ 219.225388] saved_mpidr:40f latest mpidr:40f [ 219.229642] saved_mpidr:500 latest mpidr:500 [ 219.233898] saved_mpidr:501 latest mpidr:501 [ 219.238155] saved_mpidr:502 latest mpidr:502 [ 219.242411] saved_mpidr:503 latest mpidr:503 [ 219.246665] saved_mpidr:504 latest mpidr:504 [ 219.250920] saved_mpidr:505 latest mpidr:505 [ 219.255175] saved_mpidr:506 latest mpidr:506 [ 219.259428] saved_mpidr:507 latest mpidr:507 [ 219.263680] saved_mpidr:508 latest mpidr:508 [ 219.267935] saved_mpidr:509 latest mpidr:509 [ 219.272189] saved_mpidr:50a latest mpidr:50a [ 219.276444] saved_mpidr:50b latest mpidr:50b [ 219.280699] saved_mpidr:50c latest mpidr:50c [ 219.284957] saved_mpidr:50d latest mpidr:50d [ 219.289212] saved_mpidr:50e latest mpidr:50e [ 219.293470] saved_mpidr:50f latest mpidr:50f [ 219.297725] saved_mpidr:600 latest mpidr:600 [ 219.301982] saved_mpidr:601 latest mpidr:601 [ 219.306237] saved_mpidr:602 latest mpidr:602 [ 219.310493] saved_mpidr:603 latest mpidr:603 [ 219.314749] saved_mpidr:604 latest mpidr:604 [ 219.319003] saved_mpidr:605 latest mpidr:605 [ 219.323258] saved_mpidr:606 latest mpidr:606 [ 219.327511] saved_mpidr:607 latest mpidr:607 [ 219.331766] saved_mpidr:608 latest mpidr:608 [ 219.336020] saved_mpidr:609 latest mpidr:609 [ 219.340276] saved_mpidr:60a latest mpidr:60a [ 219.344532] saved_mpidr:60b latest mpidr:60b [ 219.348783] saved_mpidr:60c latest mpidr:60c [ 219.353036] saved_mpidr:60d latest mpidr:60d [ 219.357290] saved_mpidr:60e latest mpidr:60e [ 219.361545] saved_mpidr:60f latest mpidr:60f [ 219.365799] saved_mpidr:700 latest mpidr:700 [ 219.370054] saved_mpidr:701 latest mpidr:701 [ 219.374308] saved_mpidr:702 latest mpidr:702 [ 219.378560] saved_mpidr:703 latest mpidr:703 [ 219.382815] saved_mpidr:704 latest mpidr:704 [ 219.387072] saved_mpidr:705 latest mpidr:705 [ 219.391326] saved_mpidr:706 latest mpidr:706 [ 219.395582] saved_mpidr:707 latest mpidr:707 [ 219.399840] saved_mpidr:708 latest mpidr:708 [ 219.404098] saved_mpidr:709 latest mpidr:709 [ 219.408351] saved_mpidr:70a latest mpidr:70a [ 219.412605] saved_mpidr:70b latest mpidr:70b [ 219.416860] saved_mpidr:70c latest mpidr:70c [ 219.421114] saved_mpidr:70d latest mpidr:70d [ 219.425370] saved_mpidr:70e latest mpidr:70e [ 219.429622] saved_mpidr:70f latest mpidr:70f INFO: psci: unexpected cpu_on return value: caller=CPU117, ret=-2 FAIL: psci: cpu-on SUMMARY: 4 tests, 1 unexpected failures FAIL psci (4 tests, 1 unexpected failures) > > And as I mentioned in my last before, the psci error is introduced by > > the > > patch KVM: arm64: VNCR-ize ELR_EL1.(Only test result) > > Maybe you have to try tens of times to reproduce. :) > > Deep into the patch itself, I don't find any connection between the > > patch > > and the issue. > > Me neither, and I haven't managed to reproduce your issue. > > >> - test with the current state of the patches > > I test with your nv-5.7-rc1-WIP branch and latest kvm_unit_test, the > > error still exist. > > How many vcpus do you create with this PSCI test? > 128. > Thanks, > > M. > -- > Jazz is not dead. It just smells funny... _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support 2020-04-16 1:38 ` Zengtao (B) 2020-04-16 18:22 ` Marc Zyngier @ 2020-04-17 15:11 ` Alexandru Elisei 1 sibling, 0 replies; 12+ messages in thread From: Alexandru Elisei @ 2020-04-17 15:11 UTC (permalink / raw) To: Zengtao (B), Marc Zyngier Cc: kvmarm, Ganapatrao Kulkarni, kvm, suzuki.poulose, andre.przywara, christoffer.dall, Dave.Martin, George Cherian, james.morse, linux-arm-kernel, Anil Kumar Reddy H, jintack, julien.thierry.kdev Hi, On 4/16/20 2:38 AM, Zengtao (B) wrote: > Hi Marc: > > Got it. > Really a bit patch set :) > > BTW, I have done a basic kvm unit test > git://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git > And I find that after apply the patch KVM: arm64: VNCR-ize ELR_EL1, > The psci test failed for some reason, I can't understand why, this > is only the test result.(find the patch by git bisect + kvm test) Just a heads-up, a couple of fixes to kvm-unit-tests were merged last week, and among them was one touching the psci code, e14e6ba56f6e ("arm/arm64: psci: Don't run C code without stack or vectors"). It might be worth checking that you have it, and if you don't, if applying it fixes the issue. Thanks, Alex > > My platform: Hisilicon D06 board. > Linux kernel: Linux 5.6-rc6 + nv patches(some rebases) > Could you help to take a look? > > Thanks > Zengtao > >> -----Original Message----- >> From: Marc Zyngier [mailto:maz@kernel.org] >> Sent: Saturday, April 11, 2020 5:24 PM >> To: Zengtao (B) >> Cc: George Cherian; Dave.Martin@arm.com; alexandru.elisei@arm.com; >> andre.przywara@arm.com; christoffer.dall@arm.com; >> james.morse@arm.com; jintack@cs.columbia.edu; >> julien.thierry.kdev@gmail.com; kvm@vger.kernel.org; >> kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org; >> suzuki.poulose@arm.com; Anil Kumar Reddy H; Ganapatrao Kulkarni >> Subject: Re: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested >> Virtualization support >> >> Hi Zengtao, >> >> On Sat, 11 Apr 2020 05:10:05 +0100, >> "Zengtao (B)" <prime.zeng@hisilicon.com> wrote: >>> Hi Marc: >>> >>> Since it's a very large patch series, I want to test it on my platform >>> which don't support nv, and want to make sure if this patch series >>> affects the existed virtualization functions or not. >>> >>> Any suggestion about the test focus? >> Not really. Given that the NV patches affect absolutely every >> architectural parts of KVM/arm64, everything needs careful >> testing. But more than testing, it needs reviewing. >> >> Thanks, >> >> M. >> >> -- >> Jazz is not dead, it just smells funny. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 12+ messages in thread
[parent not found: <MN2PR18MB2686F84659A6A994ECDF8031C5C70@MN2PR18MB2686.namprd18.prod.outlook.com>]
* Re: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support [not found] <MN2PR18MB2686F84659A6A994ECDF8031C5C70@MN2PR18MB2686.namprd18.prod.outlook.com> @ 2020-04-03 9:49 ` Marc Zyngier 0 siblings, 0 replies; 12+ messages in thread From: Marc Zyngier @ 2020-04-03 9:49 UTC (permalink / raw) To: George Cherian Cc: kvmarm, Ganapatrao Kulkarni, kvm, suzuki.poulose, andre.przywara, christoffer.dall, Dave.Martin, james.morse, linux-arm-kernel, Anil Kumar Reddy H, alexandru.elisei, jintack, julien.thierry.kdev On 2020-04-03 10:36, George Cherian wrote: >> -----Original Message----- >> From: Marc Zyngier <maz@kernel.org> >> Sent: Friday, April 3, 2020 1:32 PM >> To: George Cherian <gcherian@marvell.com> >> Cc: Dave.Martin@arm.com; alexandru.elisei@arm.com; >> andre.przywara@arm.com; christoffer.dall@arm.com; >> james.morse@arm.com; jintack@cs.columbia.edu; >> julien.thierry.kdev@gmail.com; kvm@vger.kernel.org; >> kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org; >> suzuki.poulose@arm.com; Anil Kumar Reddy H <areddy3@marvell.com>; >> Ganapatrao Kulkarni <gkulkarni@marvell.com> >> Subject: Re: [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested >> Virtualization support >> >> >> ---------------------------------------------------------------------- >> Hi George, >> >> On 2020-04-03 08:27, George Cherian wrote: >> > Hi Marc, >> > >> > On 2/11/20 9:48 AM, Marc Zyngier wrote: >> >> This is a major rework of the NV series that I posted over 6 months >> >> ago[1], and a lot has changed since then: >> >> >> >> - Early ARMv8.4-NV support >> >> - ARMv8.4-TTL support in host and guest >> >> - ARMv8.5-GTG support in host and guest >> >> - Lots of comments addressed after the review >> >> - Rebased on v5.6-rc1 >> >> - Way too many patches >> >> >> >> In my defence, the whole of the NV code is still smaller that the >> >> 32bit KVM/arm code I'm about to remove, so I feel less bad inflicting >> >> this on everyone! ;-) >> >> >> >> >From a functionality perspective, you can expect a L2 guest to work, >> >> but don't even think of L3, as we only partially emulate the >> >> ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, >> >> PMU, as well as anything that would require a Stage-1 PTW. What we >> >> want to achieve is that with NV disabled, there is no performance >> >> overhead and no regression. >> >> >> >> The series is roughly divided in 5 parts: exception handling, memory >> >> virtualization, interrupts and timers for ARMv8.3, followed by the >> >> ARMv8.4 support. There are of course some dependencies, but you'll >> >> hopefully get the gist of it. >> >> >> >> For the most courageous of you, I've put out a branch[2]. Of course, >> >> you'll need some userspace. Andre maintains a hacked version of >> >> kvmtool[3] that takes a --nested option, allowing the guest to be >> >> started at EL2. You can run the whole stack in the Foundation model. >> >> Don't be in a hurry ;-). >> >> >> > The full series was tested on both Foundation model as well as Marvell >> > ThunderX3 >> > Emulation Platform. >> > Basic boot testing done for Guest Hypervisor and Guest Guest. >> > >> > Tested-by: George Cherian <george.cherian@marvell.com> >> >> Thanks for having given this a go. >> >> However, without more details, it is pretty hard to find out what you >> have >> tested. >> What sort of guest have you booted, with what configuration, what >> workloads did you run in the L2 guests and what are the architectural >> features that TX3 implements? >> > > We have tried the following configurations and tests (GH - Guest > Hypervisor GG- Guest Guest). > 1 - configuration: Host:8cpus/4GB Mem, GH:4vcpus/3GB, GG: 2vcpus/2GB > Ran hackbench and Large Malloc tests (1GB allocations) across HOST,GH > and GG. > > 2 - configuration: Host:8cpus/4GB Mem, GH:1vcpu/3GB, GG: 1vcpu/2GB > Ran hackbench and Large Malloc tests across HOST,GH and GG. Host: > > We used QEMU for all these testing. Interesting. So you have added NV support to QEMU? Please be aware that the userspace side is pretty incomplete and subject to changes. > TX3 implements v8.4 Enhanced Nested Virtualization Support. Hmm. Ok. This doesn't really answer my question in terms of what other features the CPU has that would be affected by NV, but I guess that's all we'll get at this point. Thanks, M. > >> The last point is specially important, as the NV architecture spans >> two major >> revisions of the architecture and affects tons of other extensions >> that are >> themselves optional. Without any detail on that front, I have no idea >> what >> the coverage of your testing is. >> >> Thanks, >> >> M. >> -- >> Jazz is not dead. It just smells funny... -- Jazz is not dead. It just smells funny... _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support @ 2020-02-11 17:48 Marc Zyngier 0 siblings, 0 replies; 12+ messages in thread From: Marc Zyngier @ 2020-02-11 17:48 UTC (permalink / raw) To: linux-arm-kernel, kvmarm, kvm Cc: Suzuki K Poulose, Andre Przywara, Christoffer Dall, Dave Martin, James Morse, Alexandru Elisei, Jintack Lim, Julien Thierry This is a major rework of the NV series that I posted over 6 months ago[1], and a lot has changed since then: - Early ARMv8.4-NV support - ARMv8.4-TTL support in host and guest - ARMv8.5-GTG support in host and guest - Lots of comments addressed after the review - Rebased on v5.6-rc1 - Way too many patches In my defence, the whole of the NV code is still smaller that the 32bit KVM/arm code I'm about to remove, so I feel less bad inflicting this on everyone! ;-) From a functionality perspective, you can expect a L2 guest to work, but don't even think of L3, as we only partially emulate the ARMv8.{3,4}-NV extensions themselves. Same thing for vgic, debug, PMU, as well as anything that would require a Stage-1 PTW. What we want to achieve is that with NV disabled, there is no performance overhead and no regression. The series is roughly divided in 5 parts: exception handling, memory virtualization, interrupts and timers for ARMv8.3, followed by the ARMv8.4 support. There are of course some dependencies, but you'll hopefully get the gist of it. For the most courageous of you, I've put out a branch[2]. Of course, you'll need some userspace. Andre maintains a hacked version of kvmtool[3] that takes a --nested option, allowing the guest to be started at EL2. You can run the whole stack in the Foundation model. Don't be in a hurry ;-). [1] https://lore.kernel.org/r/20190621093843.220980-1-marc.zyngier@arm.com [2] git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git kvm-arm64/nv-5.6-rc1 [3] git://linux-arm.org/kvmtool.git nv/nv-wip-5.2-rc5 Andre Przywara (3): KVM: arm64: nv: Save/Restore vEL2 sysregs KVM: arm64: nv: Handle traps for timer _EL02 and _EL2 sysregs accessors KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ Christoffer Dall (17): KVM: arm64: nv: Introduce nested virtualization VCPU feature KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state KVM: arm64: nv: Handle trapped ERET from virtual EL2 KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2 KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2 changes KVM: arm/arm64: nv: Factor out stage 2 page table data from struct kvm KVM: arm64: nv: Implement nested Stage-2 page table walk logic KVM: arm64: nv: Handle shadow stage 2 page faults KVM: arm64: nv: Unmap/flush shadow stage 2 page tables KVM: arm64: nv: arch_timer: Support hyp timer emulation KVM: arm64: nv: vgic-v3: Take cpu_if pointer directly instead of vcpu KVM: arm64: nv: vgic: Emulate the HW bit in software KVM: arm64: nv: Add nested GICv3 tracepoints KVM: arm64: nv: Sync nested timer state with ARMv8.4 Jintack Lim (19): arm64: Add ARM64_HAS_NESTED_VIRT cpufeature KVM: arm64: nv: Add EL2 system registers to vcpu context KVM: arm64: nv: Support virtual EL2 exceptions KVM: arm64: nv: Inject HVC exceptions to the virtual EL2 KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from virtual EL2 KVM: arm64: nv: Trap CPACR_EL1 access in virtual EL2 KVM: arm64: nv: Handle PSCI call via smc from the guest KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP,FPEN} settings KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2 KVM: arm64: nv: Configure HCR_EL2 for nested virtualization KVM: arm64: nv: Introduce sys_reg_desc.forward_trap KVM: arm64: nv: Set a handler for the system instruction traps KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2 KVM: arm64: nv: Trap and emulate TLBI instructions from virtual EL2 KVM: arm64: nv: Nested GICv3 Support Marc Zyngier (55): KVM: arm64: Move __load_guest_stage2 to kvm_mmu.h KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values KVM: arm64: nv: Add EL2->EL1 translation helpers KVM: arm64: nv: Refactor vcpu_{read,write}_sys_reg KVM: arm64: nv: Handle virtual EL2 registers in vcpu_read/write_sys_reg() KVM: arm64: nv: Handle SPSR_EL2 specially KVM: arm64: nv: Handle HCR_EL2.E2H specially KVM: arm64: nv: Forward debug traps to the nested guest KVM: arm64: nv: Filter out unsupported features from ID regs KVM: arm64: nv: Hide RAS from nested guests KVM: arm64: nv: Use ARMv8.5-GTG to advertise supported Stage-2 page sizes KVM: arm64: Check advertised Stage-2 page size capability KVM: arm64: nv: Support multiple nested Stage-2 mmu structures KVM: arm64: nv: Move last_vcpu_ran to be per s2 mmu KVM: arm64: nv: Fold guest's HCR_EL2 configuration into the host's KVM: arm64: nv: Propagate CNTVOFF_EL2 to the virtual EL1 timer KVM: arm64: nv: Load timer before the GIC KVM: arm64: nv: Implement maintenance interrupt forwarding arm64: KVM: nv: Add handling of EL2-specific timer registers arm64: KVM: nv: Honor SCTLR_EL2.SPAN on entering vEL2 arm64: KVM: nv: Handle SCTLR_EL2 RES0/RES1 bits arm64: KVM: nv: Restrict S2 RD/WR permissions to match the guest's arm64: KVM: nv: Allow userspace to request KVM_ARM_VCPU_NESTED_VIRT arm64: Detect the ARMv8.4 TTL feature arm64: KVM: nv: Add handling of ARMv8.4-TTL TLB invalidation arm64: KVM: nv: Invalidate TLBs based on shadow S2 TTL-like information arm64: KVM: nv: Tag shadow S2 entries with nested level arm64: Add SW reserved PTE/PMD bits arm64: Add level-hinted TLB invalidation helper arm64: KVM: Add a level hint to __kvm_tlb_flush_vmid_ipa arm64: KVM: Use TTL hint in when invalidating stage-2 translations arm64: KVM: nv: Add include containing the VNCR_EL2 offsets KVM: arm64: Introduce accessor for ctxt->sys_reg KVM: arm64: sysreg: Use ctxt_sys_reg() instead of raw sys_regs access KVM: arm64: sve: Use __vcpu_sys_reg() instead of raw sys_regs access KVM: arm64: pauth: Use ctxt_sys_reg() instead of raw sys_regs access KVM: arm64: debug: Use ctxt_sys_reg() instead of raw sys_regs access KVM: arm64: Add missing reset handlers for PMU emulation KVM: arm64: nv: Move sysreg reset check to boot time KVM: arm64: Map VNCR-capable registers to a separate page KVM: arm64: nv: Move nested vgic state into the sysreg file KVM: arm64: Use accessors for timer ctl/cval/offset KVM: arm64: Add VNCR-capable timer accessors for arm64 KVM: arm64: Make struct kvm_regs userspace-only KVM: arm64: VNCR-ize ELR_EL1 KVM: arm64: VNCR-ize SP_EL1 KVM: arm64: Disintegrate SPSR array KVM: arm64: aarch32: Use __vcpu_sys_reg() instead of raw sys_regs access KVM: arm64: VNCR-ize SPSR_EL1 KVM: arm64: Add ARMv8.4 Enhanced Nested Virt cpufeature KVM: arm64: nv: Synchronize PSTATE early on exit KVM: arm64: nv: Allocate VNCR page when required KVM: arm64: nv: Enable ARMv8.4-NV support KVM: arm64: nv: Fast-track 'InHost' exception returns KVM: arm64: nv: Fast-track EL1 TLBIs for VHE guests .../admin-guide/kernel-parameters.txt | 4 + .../virt/kvm/devices/arm-vgic-v3.txt | 8 + arch/arm/include/asm/kvm_asm.h | 6 +- arch/arm/include/asm/kvm_emulate.h | 3 + arch/arm/include/asm/kvm_host.h | 32 +- arch/arm/include/asm/kvm_hyp.h | 12 +- arch/arm/include/asm/kvm_mmu.h | 90 +- arch/arm/include/asm/kvm_nested.h | 11 + arch/arm/include/asm/stage2_pgtable.h | 9 + arch/arm/include/uapi/asm/kvm.h | 1 + arch/arm/kvm/hyp/switch.c | 11 +- arch/arm/kvm/hyp/tlb.c | 18 +- arch/arm64/include/asm/cpucaps.h | 5 +- arch/arm64/include/asm/esr.h | 6 + arch/arm64/include/asm/kvm_arm.h | 28 +- arch/arm64/include/asm/kvm_asm.h | 10 +- arch/arm64/include/asm/kvm_coproc.h | 2 +- arch/arm64/include/asm/kvm_emulate.h | 181 +- arch/arm64/include/asm/kvm_host.h | 212 ++- arch/arm64/include/asm/kvm_hyp.h | 32 +- arch/arm64/include/asm/kvm_mmu.h | 62 +- arch/arm64/include/asm/kvm_nested.h | 94 + arch/arm64/include/asm/pgtable-hwdef.h | 2 + arch/arm64/include/asm/stage2_pgtable.h | 9 + arch/arm64/include/asm/sysreg.h | 126 +- arch/arm64/include/asm/tlbflush.h | 30 + arch/arm64/include/asm/vncr_mapping.h | 73 + arch/arm64/include/uapi/asm/kvm.h | 2 + arch/arm64/kernel/asm-offsets.c | 3 +- arch/arm64/kernel/cpufeature.c | 55 + arch/arm64/kvm/Makefile | 5 + arch/arm64/kvm/emulate-nested.c | 205 +++ arch/arm64/kvm/fpsimd.c | 6 +- arch/arm64/kvm/guest.c | 85 +- arch/arm64/kvm/handle_exit.c | 98 +- arch/arm64/kvm/hyp/Makefile | 1 + arch/arm64/kvm/hyp/at.c | 231 +++ arch/arm64/kvm/hyp/debug-sr.c | 18 +- arch/arm64/kvm/hyp/entry.S | 3 +- arch/arm64/kvm/hyp/switch.c | 241 ++- arch/arm64/kvm/hyp/sysreg-sr.c | 338 +++- arch/arm64/kvm/hyp/tlb.c | 134 +- arch/arm64/kvm/inject_fault.c | 12 - arch/arm64/kvm/nested.c | 899 ++++++++++ arch/arm64/kvm/regmap.c | 37 +- arch/arm64/kvm/reset.c | 72 +- arch/arm64/kvm/sys_regs.c | 1523 +++++++++++++++-- arch/arm64/kvm/sys_regs.h | 6 + arch/arm64/kvm/trace.h | 56 + include/kvm/arm_arch_timer.h | 9 +- include/kvm/arm_vgic.h | 21 +- virt/kvm/arm/arch_timer.c | 271 ++- virt/kvm/arm/arch_timer_nested.c | 95 + virt/kvm/arm/arm.c | 72 +- virt/kvm/arm/hyp/vgic-v3-sr.c | 35 +- virt/kvm/arm/mmio.c | 14 +- virt/kvm/arm/mmu.c | 458 +++-- virt/kvm/arm/trace.h | 6 +- virt/kvm/arm/vgic/vgic-init.c | 30 + virt/kvm/arm/vgic/vgic-kvm-device.c | 22 + virt/kvm/arm/vgic/vgic-nested-trace.h | 137 ++ virt/kvm/arm/vgic/vgic-v2.c | 10 +- virt/kvm/arm/vgic/vgic-v3-nested.c | 240 +++ virt/kvm/arm/vgic/vgic-v3.c | 51 +- virt/kvm/arm/vgic/vgic.c | 74 +- virt/kvm/arm/vgic/vgic.h | 10 + 66 files changed, 5925 insertions(+), 737 deletions(-) create mode 100644 arch/arm/include/asm/kvm_nested.h create mode 100644 arch/arm64/include/asm/kvm_nested.h create mode 100644 arch/arm64/include/asm/vncr_mapping.h create mode 100644 arch/arm64/kvm/emulate-nested.c create mode 100644 arch/arm64/kvm/hyp/at.c create mode 100644 arch/arm64/kvm/nested.c create mode 100644 virt/kvm/arm/arch_timer_nested.c create mode 100644 virt/kvm/arm/vgic/vgic-nested-trace.h create mode 100644 virt/kvm/arm/vgic/vgic-v3-nested.c -- 2.20.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2020-04-21 2:23 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <MN2PR18MB26869A6CA4E67558324F655CC5C70@MN2PR18MB2686.namprd18.prod.outlook.com> 2020-04-03 8:02 ` [PATCH v2 00/94] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support Marc Zyngier 2020-04-11 4:10 ` Zengtao (B) 2020-04-11 9:23 ` Marc Zyngier 2020-04-16 1:38 ` Zengtao (B) 2020-04-16 18:22 ` Marc Zyngier 2020-04-17 15:06 ` Marc Zyngier 2020-04-18 2:49 ` Zengtao (B) 2020-04-20 14:03 ` Marc Zyngier 2020-04-21 2:22 ` Zengtao (B) 2020-04-17 15:11 ` Alexandru Elisei [not found] <MN2PR18MB2686F84659A6A994ECDF8031C5C70@MN2PR18MB2686.namprd18.prod.outlook.com> 2020-04-03 9:49 ` Marc Zyngier 2020-02-11 17:48 Marc Zyngier
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).