Any way to disable KVM VHE extension?

Any way to disable KVM VHE extension?

[Qu Wenruo]

Hi,

Recently I'm playing around the Nvidia Xavier AGX board, which has VHE 
extension support.

In theory, considering the CPU and memory, it should be pretty powerful 
compared to boards like RPI CM4.

But to my surprise, KVM runs pretty poor on Xavier.

Just booting the edk2 firmware could take over 10s, and 20s to fully 
boot the kernel.
Even my VM on RPI CM4 has way faster boot time, even just running on 
PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.

This is definitely out of my expectation, I double checked to be sure 
that it's running in KVM mode.

But further digging shows that, since Xavier AGX CPU supports VHE, kvm 
is running in VHE mode other than HYP mode on CM4.

Is there anyway to manually disable VHE mode to test the more common HYP 
mode on Xavier?

BTW, this is the dmesg related to KVM on Xavier, running v5.13 upstream 
kernel, with 64K page size:
[    0.852357] kvm [1]: IPA Size Limit: 40 bits
[    0.857378] kvm [1]: vgic interrupt IRQ9
[    0.862122] kvm: pmu event creation failed -2
[    0.866734] kvm [1]: VHE mode initialized successfully

While on CM4, the host runs v5.12.10 upstream kernel (with downstream 
dtb), with 4K page size:
[    1.276818] kvm [1]: IPA Size Limit: 44 bits
[    1.278425] kvm [1]: vgic interrupt IRQ9
[    1.278620] kvm [1]: Hyp mode initialized successfully

Could it be the PAGE size causing problem?

Thanks,
Qu

Any way to disable KVM VHE extension?

[Qu Wenruo]

Hi,

Recently I'm playing around the Nvidia Xavier AGX board, which has VHE 
extension support.

In theory, considering the CPU and memory, it should be pretty powerful 
compared to boards like RPI CM4.

But to my surprise, KVM runs pretty poor on Xavier.

Just booting the edk2 firmware could take over 10s, and 20s to fully 
boot the kernel.
Even my VM on RPI CM4 has way faster boot time, even just running on 
PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.

This is definitely out of my expectation, I double checked to be sure 
that it's running in KVM mode.

But further digging shows that, since Xavier AGX CPU supports VHE, kvm 
is running in VHE mode other than HYP mode on CM4.

Is there anyway to manually disable VHE mode to test the more common HYP 
mode on Xavier?

BTW, this is the dmesg related to KVM on Xavier, running v5.13 upstream 
kernel, with 64K page size:
[    0.852357] kvm [1]: IPA Size Limit: 40 bits
[    0.857378] kvm [1]: vgic interrupt IRQ9
[    0.862122] kvm: pmu event creation failed -2
[    0.866734] kvm [1]: VHE mode initialized successfully

While on CM4, the host runs v5.12.10 upstream kernel (with downstream 
dtb), with 4K page size:
[    1.276818] kvm [1]: IPA Size Limit: 44 bits
[    1.278425] kvm [1]: vgic interrupt IRQ9
[    1.278620] kvm [1]: Hyp mode initialized successfully

Could it be the PAGE size causing problem?

Thanks,
Qu

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Any way to disable KVM VHE extension?

[Qu Wenruo]

Hi,

Recently I'm playing around the Nvidia Xavier AGX board, which has VHE 
extension support.

In theory, considering the CPU and memory, it should be pretty powerful 
compared to boards like RPI CM4.

But to my surprise, KVM runs pretty poor on Xavier.

Just booting the edk2 firmware could take over 10s, and 20s to fully 
boot the kernel.
Even my VM on RPI CM4 has way faster boot time, even just running on 
PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.

This is definitely out of my expectation, I double checked to be sure 
that it's running in KVM mode.

But further digging shows that, since Xavier AGX CPU supports VHE, kvm 
is running in VHE mode other than HYP mode on CM4.

Is there anyway to manually disable VHE mode to test the more common HYP 
mode on Xavier?

BTW, this is the dmesg related to KVM on Xavier, running v5.13 upstream 
kernel, with 64K page size:
[    0.852357] kvm [1]: IPA Size Limit: 40 bits
[    0.857378] kvm [1]: vgic interrupt IRQ9
[    0.862122] kvm: pmu event creation failed -2
[    0.866734] kvm [1]: VHE mode initialized successfully

While on CM4, the host runs v5.12.10 upstream kernel (with downstream 
dtb), with 4K page size:
[    1.276818] kvm [1]: IPA Size Limit: 44 bits
[    1.278425] kvm [1]: vgic interrupt IRQ9
[    1.278620] kvm [1]: Hyp mode initialized successfully

Could it be the PAGE size causing problem?

Thanks,
Qu


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Any way to disable KVM VHE extension?

[Qu Wenruo]

On 2021/7/15 下午4:55, Qu Wenruo wrote:
> Hi,
> 
> Recently I'm playing around the Nvidia Xavier AGX board, which has VHE 
> extension support.
> 
> In theory, considering the CPU and memory, it should be pretty powerful 
> compared to boards like RPI CM4.
> 
> But to my surprise, KVM runs pretty poor on Xavier.
> 
> Just booting the edk2 firmware could take over 10s, and 20s to fully 
> boot the kernel.
> Even my VM on RPI CM4 has way faster boot time, even just running on 
> PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
> 
> This is definitely out of my expectation, I double checked to be sure 
> that it's running in KVM mode.
> 
> But further digging shows that, since Xavier AGX CPU supports VHE, kvm 
> is running in VHE mode other than HYP mode on CM4.
> 
> Is there anyway to manually disable VHE mode to test the more common HYP 
> mode on Xavier?
> 
> BTW, this is the dmesg related to KVM on Xavier, running v5.13 upstream 
> kernel, with 64K page size:
> [    0.852357] kvm [1]: IPA Size Limit: 40 bits
> [    0.857378] kvm [1]: vgic interrupt IRQ9
> [    0.862122] kvm: pmu event creation failed -2
> [    0.866734] kvm [1]: VHE mode initialized successfully

Wait, the kernel I'm currently running on Xavier is still using 4K page 
size, just like CM4.

Thus it should not be a problem in page size.

Thanks,
Qu
> 
> While on CM4, the host runs v5.12.10 upstream kernel (with downstream 
> dtb), with 4K page size:
> [    1.276818] kvm [1]: IPA Size Limit: 44 bits
> [    1.278425] kvm [1]: vgic interrupt IRQ9
> [    1.278620] kvm [1]: Hyp mode initialized successfully
> 
> Could it be the PAGE size causing problem?
> 
> Thanks,
> Qu

Re: Any way to disable KVM VHE extension?

[Qu Wenruo]

On 2021/7/15 下午4:55, Qu Wenruo wrote:
> Hi,
> 
> Recently I'm playing around the Nvidia Xavier AGX board, which has VHE 
> extension support.
> 
> In theory, considering the CPU and memory, it should be pretty powerful 
> compared to boards like RPI CM4.
> 
> But to my surprise, KVM runs pretty poor on Xavier.
> 
> Just booting the edk2 firmware could take over 10s, and 20s to fully 
> boot the kernel.
> Even my VM on RPI CM4 has way faster boot time, even just running on 
> PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
> 
> This is definitely out of my expectation, I double checked to be sure 
> that it's running in KVM mode.
> 
> But further digging shows that, since Xavier AGX CPU supports VHE, kvm 
> is running in VHE mode other than HYP mode on CM4.
> 
> Is there anyway to manually disable VHE mode to test the more common HYP 
> mode on Xavier?
> 
> BTW, this is the dmesg related to KVM on Xavier, running v5.13 upstream 
> kernel, with 64K page size:
> [    0.852357] kvm [1]: IPA Size Limit: 40 bits
> [    0.857378] kvm [1]: vgic interrupt IRQ9
> [    0.862122] kvm: pmu event creation failed -2
> [    0.866734] kvm [1]: VHE mode initialized successfully

Wait, the kernel I'm currently running on Xavier is still using 4K page 
size, just like CM4.

Thus it should not be a problem in page size.

Thanks,
Qu
> 
> While on CM4, the host runs v5.12.10 upstream kernel (with downstream 
> dtb), with 4K page size:
> [    1.276818] kvm [1]: IPA Size Limit: 44 bits
> [    1.278425] kvm [1]: vgic interrupt IRQ9
> [    1.278620] kvm [1]: Hyp mode initialized successfully
> 
> Could it be the PAGE size causing problem?
> 
> Thanks,
> Qu

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Any way to disable KVM VHE extension?

[Qu Wenruo]

On 2021/7/15 下午4:55, Qu Wenruo wrote:
> Hi,
> 
> Recently I'm playing around the Nvidia Xavier AGX board, which has VHE 
> extension support.
> 
> In theory, considering the CPU and memory, it should be pretty powerful 
> compared to boards like RPI CM4.
> 
> But to my surprise, KVM runs pretty poor on Xavier.
> 
> Just booting the edk2 firmware could take over 10s, and 20s to fully 
> boot the kernel.
> Even my VM on RPI CM4 has way faster boot time, even just running on 
> PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
> 
> This is definitely out of my expectation, I double checked to be sure 
> that it's running in KVM mode.
> 
> But further digging shows that, since Xavier AGX CPU supports VHE, kvm 
> is running in VHE mode other than HYP mode on CM4.
> 
> Is there anyway to manually disable VHE mode to test the more common HYP 
> mode on Xavier?
> 
> BTW, this is the dmesg related to KVM on Xavier, running v5.13 upstream 
> kernel, with 64K page size:
> [    0.852357] kvm [1]: IPA Size Limit: 40 bits
> [    0.857378] kvm [1]: vgic interrupt IRQ9
> [    0.862122] kvm: pmu event creation failed -2
> [    0.866734] kvm [1]: VHE mode initialized successfully

Wait, the kernel I'm currently running on Xavier is still using 4K page 
size, just like CM4.

Thus it should not be a problem in page size.

Thanks,
Qu
> 
> While on CM4, the host runs v5.12.10 upstream kernel (with downstream 
> dtb), with 4K page size:
> [    1.276818] kvm [1]: IPA Size Limit: 44 bits
> [    1.278425] kvm [1]: vgic interrupt IRQ9
> [    1.278620] kvm [1]: Hyp mode initialized successfully
> 
> Could it be the PAGE size causing problem?
> 
> Thanks,
> Qu


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Any way to disable KVM VHE extension?

[Robin Murphy]

On 2021-07-15 09:55, Qu Wenruo wrote:
> Hi,
> 
> Recently I'm playing around the Nvidia Xavier AGX board, which has VHE 
> extension support.
> 
> In theory, considering the CPU and memory, it should be pretty powerful 
> compared to boards like RPI CM4.
> 
> But to my surprise, KVM runs pretty poor on Xavier.
> 
> Just booting the edk2 firmware could take over 10s, and 20s to fully 
> boot the kernel.
> Even my VM on RPI CM4 has way faster boot time, even just running on 
> PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
> 
> This is definitely out of my expectation, I double checked to be sure 
> that it's running in KVM mode.
> 
> But further digging shows that, since Xavier AGX CPU supports VHE, kvm 
> is running in VHE mode other than HYP mode on CM4.
> 
> Is there anyway to manually disable VHE mode to test the more common HYP 
> mode on Xavier?

According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its 
low-level equivalent "id_aa64mmfr1.vh=0") on the command line should do 
that.

However I'd imagine the discrepancy is likely to be something more 
fundamental to the wildly different microarchitectures. There's 
certainly no harm in giving non-VHE a go for comparison, but I wouldn't 
be surprised if it turns out even slower...

Robin.

> BTW, this is the dmesg related to KVM on Xavier, running v5.13 upstream 
> kernel, with 64K page size:
> [    0.852357] kvm [1]: IPA Size Limit: 40 bits
> [    0.857378] kvm [1]: vgic interrupt IRQ9
> [    0.862122] kvm: pmu event creation failed -2
> [    0.866734] kvm [1]: VHE mode initialized successfully
> 
> While on CM4, the host runs v5.12.10 upstream kernel (with downstream 
> dtb), with 4K page size:
> [    1.276818] kvm [1]: IPA Size Limit: 44 bits
> [    1.278425] kvm [1]: vgic interrupt IRQ9
> [    1.278620] kvm [1]: Hyp mode initialized successfully
> 
> Could it be the PAGE size causing problem?
> 
> Thanks,
> Qu
> 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Any way to disable KVM VHE extension?

[Robin Murphy]

On 2021-07-15 09:55, Qu Wenruo wrote:
> Hi,
> 
> Recently I'm playing around the Nvidia Xavier AGX board, which has VHE 
> extension support.
> 
> In theory, considering the CPU and memory, it should be pretty powerful 
> compared to boards like RPI CM4.
> 
> But to my surprise, KVM runs pretty poor on Xavier.
> 
> Just booting the edk2 firmware could take over 10s, and 20s to fully 
> boot the kernel.
> Even my VM on RPI CM4 has way faster boot time, even just running on 
> PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
> 
> This is definitely out of my expectation, I double checked to be sure 
> that it's running in KVM mode.
> 
> But further digging shows that, since Xavier AGX CPU supports VHE, kvm 
> is running in VHE mode other than HYP mode on CM4.
> 
> Is there anyway to manually disable VHE mode to test the more common HYP 
> mode on Xavier?

According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its 
low-level equivalent "id_aa64mmfr1.vh=0") on the command line should do 
that.

However I'd imagine the discrepancy is likely to be something more 
fundamental to the wildly different microarchitectures. There's 
certainly no harm in giving non-VHE a go for comparison, but I wouldn't 
be surprised if it turns out even slower...

Robin.

> BTW, this is the dmesg related to KVM on Xavier, running v5.13 upstream 
> kernel, with 64K page size:
> [    0.852357] kvm [1]: IPA Size Limit: 40 bits
> [    0.857378] kvm [1]: vgic interrupt IRQ9
> [    0.862122] kvm: pmu event creation failed -2
> [    0.866734] kvm [1]: VHE mode initialized successfully
> 
> While on CM4, the host runs v5.12.10 upstream kernel (with downstream 
> dtb), with 4K page size:
> [    1.276818] kvm [1]: IPA Size Limit: 44 bits
> [    1.278425] kvm [1]: vgic interrupt IRQ9
> [    1.278620] kvm [1]: Hyp mode initialized successfully
> 
> Could it be the PAGE size causing problem?
> 
> Thanks,
> Qu
> 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Any way to disable KVM VHE extension?

[Robin Murphy]

On 2021-07-15 09:55, Qu Wenruo wrote:
> Hi,
> 
> Recently I'm playing around the Nvidia Xavier AGX board, which has VHE 
> extension support.
> 
> In theory, considering the CPU and memory, it should be pretty powerful 
> compared to boards like RPI CM4.
> 
> But to my surprise, KVM runs pretty poor on Xavier.
> 
> Just booting the edk2 firmware could take over 10s, and 20s to fully 
> boot the kernel.
> Even my VM on RPI CM4 has way faster boot time, even just running on 
> PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
> 
> This is definitely out of my expectation, I double checked to be sure 
> that it's running in KVM mode.
> 
> But further digging shows that, since Xavier AGX CPU supports VHE, kvm 
> is running in VHE mode other than HYP mode on CM4.
> 
> Is there anyway to manually disable VHE mode to test the more common HYP 
> mode on Xavier?

According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its 
low-level equivalent "id_aa64mmfr1.vh=0") on the command line should do 
that.

However I'd imagine the discrepancy is likely to be something more 
fundamental to the wildly different microarchitectures. There's 
certainly no harm in giving non-VHE a go for comparison, but I wouldn't 
be surprised if it turns out even slower...

Robin.

> BTW, this is the dmesg related to KVM on Xavier, running v5.13 upstream 
> kernel, with 64K page size:
> [    0.852357] kvm [1]: IPA Size Limit: 40 bits
> [    0.857378] kvm [1]: vgic interrupt IRQ9
> [    0.862122] kvm: pmu event creation failed -2
> [    0.866734] kvm [1]: VHE mode initialized successfully
> 
> While on CM4, the host runs v5.12.10 upstream kernel (with downstream 
> dtb), with 4K page size:
> [    1.276818] kvm [1]: IPA Size Limit: 44 bits
> [    1.278425] kvm [1]: vgic interrupt IRQ9
> [    1.278620] kvm [1]: Hyp mode initialized successfully
> 
> Could it be the PAGE size causing problem?
> 
> Thanks,
> Qu
> 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Any way to disable KVM VHE extension?

[Qu Wenruo]

On 2021/7/15 下午5:28, Robin Murphy wrote:
> On 2021-07-15 09:55, Qu Wenruo wrote:
>> Hi,
>>
>> Recently I'm playing around the Nvidia Xavier AGX board, which has VHE 
>> extension support.
>>
>> In theory, considering the CPU and memory, it should be pretty 
>> powerful compared to boards like RPI CM4.
>>
>> But to my surprise, KVM runs pretty poor on Xavier.
>>
>> Just booting the edk2 firmware could take over 10s, and 20s to fully 
>> boot the kernel.
>> Even my VM on RPI CM4 has way faster boot time, even just running on 
>> PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
>>
>> This is definitely out of my expectation, I double checked to be sure 
>> that it's running in KVM mode.
>>
>> But further digging shows that, since Xavier AGX CPU supports VHE, kvm 
>> is running in VHE mode other than HYP mode on CM4.
>>
>> Is there anyway to manually disable VHE mode to test the more common 
>> HYP mode on Xavier?
> 
> According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its 
> low-level equivalent "id_aa64mmfr1.vh=0") on the command line should do 
> that.

Thanks for this one, I stupidly only searched modinfo of kvm, and didn't 
even bother to search arch/arm64/kvm...

> 
> However I'd imagine the discrepancy is likely to be something more 
> fundamental to the wildly different microarchitectures. There's 
> certainly no harm in giving non-VHE a go for comparison, but I wouldn't 
> be surprised if it turns out even slower...

You're totally right, with nvhe mode, it's still the same slow speed.

BTW, what did you mean by the "wildly different microarch"?
Is ARMv8.2 arch that different from ARMv8 of RPI4?

And any extra methods I could try to explore the reason of the slowness?

At least RPI CM4 is beyond my expectation and is working pretty fine.

Thanks,
Qu

> 
> Robin.
> 
>> BTW, this is the dmesg related to KVM on Xavier, running v5.13 
>> upstream kernel, with 64K page size:
>> [    0.852357] kvm [1]: IPA Size Limit: 40 bits
>> [    0.857378] kvm [1]: vgic interrupt IRQ9
>> [    0.862122] kvm: pmu event creation failed -2
>> [    0.866734] kvm [1]: VHE mode initialized successfully
>>
>> While on CM4, the host runs v5.12.10 upstream kernel (with downstream 
>> dtb), with 4K page size:
>> [    1.276818] kvm [1]: IPA Size Limit: 44 bits
>> [    1.278425] kvm [1]: vgic interrupt IRQ9
>> [    1.278620] kvm [1]: Hyp mode initialized successfully
>>
>> Could it be the PAGE size causing problem?
>>
>> Thanks,
>> Qu
>>
>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: Any way to disable KVM VHE extension?

[Qu Wenruo]

On 2021/7/15 下午5:28, Robin Murphy wrote:
> On 2021-07-15 09:55, Qu Wenruo wrote:
>> Hi,
>>
>> Recently I'm playing around the Nvidia Xavier AGX board, which has VHE 
>> extension support.
>>
>> In theory, considering the CPU and memory, it should be pretty 
>> powerful compared to boards like RPI CM4.
>>
>> But to my surprise, KVM runs pretty poor on Xavier.
>>
>> Just booting the edk2 firmware could take over 10s, and 20s to fully 
>> boot the kernel.
>> Even my VM on RPI CM4 has way faster boot time, even just running on 
>> PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
>>
>> This is definitely out of my expectation, I double checked to be sure 
>> that it's running in KVM mode.
>>
>> But further digging shows that, since Xavier AGX CPU supports VHE, kvm 
>> is running in VHE mode other than HYP mode on CM4.
>>
>> Is there anyway to manually disable VHE mode to test the more common 
>> HYP mode on Xavier?
> 
> According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its 
> low-level equivalent "id_aa64mmfr1.vh=0") on the command line should do 
> that.

Thanks for this one, I stupidly only searched modinfo of kvm, and didn't 
even bother to search arch/arm64/kvm...

> 
> However I'd imagine the discrepancy is likely to be something more 
> fundamental to the wildly different microarchitectures. There's 
> certainly no harm in giving non-VHE a go for comparison, but I wouldn't 
> be surprised if it turns out even slower...

You're totally right, with nvhe mode, it's still the same slow speed.

BTW, what did you mean by the "wildly different microarch"?
Is ARMv8.2 arch that different from ARMv8 of RPI4?

And any extra methods I could try to explore the reason of the slowness?

At least RPI CM4 is beyond my expectation and is working pretty fine.

Thanks,
Qu

> 
> Robin.
> 
>> BTW, this is the dmesg related to KVM on Xavier, running v5.13 
>> upstream kernel, with 64K page size:
>> [    0.852357] kvm [1]: IPA Size Limit: 40 bits
>> [    0.857378] kvm [1]: vgic interrupt IRQ9
>> [    0.862122] kvm: pmu event creation failed -2
>> [    0.866734] kvm [1]: VHE mode initialized successfully
>>
>> While on CM4, the host runs v5.12.10 upstream kernel (with downstream 
>> dtb), with 4K page size:
>> [    1.276818] kvm [1]: IPA Size Limit: 44 bits
>> [    1.278425] kvm [1]: vgic interrupt IRQ9
>> [    1.278620] kvm [1]: Hyp mode initialized successfully
>>
>> Could it be the PAGE size causing problem?
>>
>> Thanks,
>> Qu
>>
>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Any way to disable KVM VHE extension?

[Qu Wenruo]

On 2021/7/15 下午5:28, Robin Murphy wrote:
> On 2021-07-15 09:55, Qu Wenruo wrote:
>> Hi,
>>
>> Recently I'm playing around the Nvidia Xavier AGX board, which has VHE 
>> extension support.
>>
>> In theory, considering the CPU and memory, it should be pretty 
>> powerful compared to boards like RPI CM4.
>>
>> But to my surprise, KVM runs pretty poor on Xavier.
>>
>> Just booting the edk2 firmware could take over 10s, and 20s to fully 
>> boot the kernel.
>> Even my VM on RPI CM4 has way faster boot time, even just running on 
>> PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
>>
>> This is definitely out of my expectation, I double checked to be sure 
>> that it's running in KVM mode.
>>
>> But further digging shows that, since Xavier AGX CPU supports VHE, kvm 
>> is running in VHE mode other than HYP mode on CM4.
>>
>> Is there anyway to manually disable VHE mode to test the more common 
>> HYP mode on Xavier?
> 
> According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its 
> low-level equivalent "id_aa64mmfr1.vh=0") on the command line should do 
> that.

Thanks for this one, I stupidly only searched modinfo of kvm, and didn't 
even bother to search arch/arm64/kvm...

> 
> However I'd imagine the discrepancy is likely to be something more 
> fundamental to the wildly different microarchitectures. There's 
> certainly no harm in giving non-VHE a go for comparison, but I wouldn't 
> be surprised if it turns out even slower...

You're totally right, with nvhe mode, it's still the same slow speed.

BTW, what did you mean by the "wildly different microarch"?
Is ARMv8.2 arch that different from ARMv8 of RPI4?

And any extra methods I could try to explore the reason of the slowness?

At least RPI CM4 is beyond my expectation and is working pretty fine.

Thanks,
Qu

> 
> Robin.
> 
>> BTW, this is the dmesg related to KVM on Xavier, running v5.13 
>> upstream kernel, with 64K page size:
>> [    0.852357] kvm [1]: IPA Size Limit: 40 bits
>> [    0.857378] kvm [1]: vgic interrupt IRQ9
>> [    0.862122] kvm: pmu event creation failed -2
>> [    0.866734] kvm [1]: VHE mode initialized successfully
>>
>> While on CM4, the host runs v5.12.10 upstream kernel (with downstream 
>> dtb), with 4K page size:
>> [    1.276818] kvm [1]: IPA Size Limit: 44 bits
>> [    1.278425] kvm [1]: vgic interrupt IRQ9
>> [    1.278620] kvm [1]: Hyp mode initialized successfully
>>
>> Could it be the PAGE size causing problem?
>>
>> Thanks,
>> Qu
>>
>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Any way to disable KVM VHE extension?

[Robin Murphy]

On 2021-07-15 10:44, Qu Wenruo wrote:
> 
> 
> On 2021/7/15 下午5:28, Robin Murphy wrote:
>> On 2021-07-15 09:55, Qu Wenruo wrote:
>>> Hi,
>>>
>>> Recently I'm playing around the Nvidia Xavier AGX board, which has 
>>> VHE extension support.
>>>
>>> In theory, considering the CPU and memory, it should be pretty 
>>> powerful compared to boards like RPI CM4.
>>>
>>> But to my surprise, KVM runs pretty poor on Xavier.
>>>
>>> Just booting the edk2 firmware could take over 10s, and 20s to fully 
>>> boot the kernel.
>>> Even my VM on RPI CM4 has way faster boot time, even just running on 
>>> PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
>>>
>>> This is definitely out of my expectation, I double checked to be sure 
>>> that it's running in KVM mode.
>>>
>>> But further digging shows that, since Xavier AGX CPU supports VHE, 
>>> kvm is running in VHE mode other than HYP mode on CM4.
>>>
>>> Is there anyway to manually disable VHE mode to test the more common 
>>> HYP mode on Xavier?
>>
>> According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its 
>> low-level equivalent "id_aa64mmfr1.vh=0") on the command line should 
>> do that.
> 
> Thanks for this one, I stupidly only searched modinfo of kvm, and didn't 
> even bother to search arch/arm64/kvm...
> 
>>
>> However I'd imagine the discrepancy is likely to be something more 
>> fundamental to the wildly different microarchitectures. There's 
>> certainly no harm in giving non-VHE a go for comparison, but I 
>> wouldn't be surprised if it turns out even slower...
> 
> You're totally right, with nvhe mode, it's still the same slow speed.
> 
> BTW, what did you mean by the "wildly different microarch"?
> Is ARMv8.2 arch that different from ARMv8 of RPI4?

I don't mean Armv8.x architectural features, I mean the actual 
implementation of NVIDIA's Carmel core is very, very different from 
Cortex-A72 or indeed our newer v8.2 Cortex-A designs.

> And any extra methods I could try to explore the reason of the slowness?

I guess the first check would be whether you're trapping and exiting the 
VM significantly more. I believe there are stats somewhere, but I don't 
know exactly where, sorry - I know very little about actually *using* KVM :)

If it's not that, then it might just be that EDK2 is doing a lot of 
cache maintenance or system register modification or some other 
operation that happens to be slower on Carmel compared to Cortex-A72.

Robin.

> At least RPI CM4 is beyond my expectation and is working pretty fine.
> 
> Thanks,
> Qu
> 
>>
>> Robin.
>>
>>> BTW, this is the dmesg related to KVM on Xavier, running v5.13 
>>> upstream kernel, with 64K page size:
>>> [    0.852357] kvm [1]: IPA Size Limit: 40 bits
>>> [    0.857378] kvm [1]: vgic interrupt IRQ9
>>> [    0.862122] kvm: pmu event creation failed -2
>>> [    0.866734] kvm [1]: VHE mode initialized successfully
>>>
>>> While on CM4, the host runs v5.12.10 upstream kernel (with downstream 
>>> dtb), with 4K page size:
>>> [    1.276818] kvm [1]: IPA Size Limit: 44 bits
>>> [    1.278425] kvm [1]: vgic interrupt IRQ9
>>> [    1.278620] kvm [1]: Hyp mode initialized successfully
>>>
>>> Could it be the PAGE size causing problem?
>>>
>>> Thanks,
>>> Qu
>>>
>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>
> 

Re: Any way to disable KVM VHE extension?

[Robin Murphy]

On 2021-07-15 10:44, Qu Wenruo wrote:
> 
> 
> On 2021/7/15 下午5:28, Robin Murphy wrote:
>> On 2021-07-15 09:55, Qu Wenruo wrote:
>>> Hi,
>>>
>>> Recently I'm playing around the Nvidia Xavier AGX board, which has 
>>> VHE extension support.
>>>
>>> In theory, considering the CPU and memory, it should be pretty 
>>> powerful compared to boards like RPI CM4.
>>>
>>> But to my surprise, KVM runs pretty poor on Xavier.
>>>
>>> Just booting the edk2 firmware could take over 10s, and 20s to fully 
>>> boot the kernel.
>>> Even my VM on RPI CM4 has way faster boot time, even just running on 
>>> PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
>>>
>>> This is definitely out of my expectation, I double checked to be sure 
>>> that it's running in KVM mode.
>>>
>>> But further digging shows that, since Xavier AGX CPU supports VHE, 
>>> kvm is running in VHE mode other than HYP mode on CM4.
>>>
>>> Is there anyway to manually disable VHE mode to test the more common 
>>> HYP mode on Xavier?
>>
>> According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its 
>> low-level equivalent "id_aa64mmfr1.vh=0") on the command line should 
>> do that.
> 
> Thanks for this one, I stupidly only searched modinfo of kvm, and didn't 
> even bother to search arch/arm64/kvm...
> 
>>
>> However I'd imagine the discrepancy is likely to be something more 
>> fundamental to the wildly different microarchitectures. There's 
>> certainly no harm in giving non-VHE a go for comparison, but I 
>> wouldn't be surprised if it turns out even slower...
> 
> You're totally right, with nvhe mode, it's still the same slow speed.
> 
> BTW, what did you mean by the "wildly different microarch"?
> Is ARMv8.2 arch that different from ARMv8 of RPI4?

I don't mean Armv8.x architectural features, I mean the actual 
implementation of NVIDIA's Carmel core is very, very different from 
Cortex-A72 or indeed our newer v8.2 Cortex-A designs.

> And any extra methods I could try to explore the reason of the slowness?

I guess the first check would be whether you're trapping and exiting the 
VM significantly more. I believe there are stats somewhere, but I don't 
know exactly where, sorry - I know very little about actually *using* KVM :)

If it's not that, then it might just be that EDK2 is doing a lot of 
cache maintenance or system register modification or some other 
operation that happens to be slower on Carmel compared to Cortex-A72.

Robin.

> At least RPI CM4 is beyond my expectation and is working pretty fine.
> 
> Thanks,
> Qu
> 
>>
>> Robin.
>>
>>> BTW, this is the dmesg related to KVM on Xavier, running v5.13 
>>> upstream kernel, with 64K page size:
>>> [    0.852357] kvm [1]: IPA Size Limit: 40 bits
>>> [    0.857378] kvm [1]: vgic interrupt IRQ9
>>> [    0.862122] kvm: pmu event creation failed -2
>>> [    0.866734] kvm [1]: VHE mode initialized successfully
>>>
>>> While on CM4, the host runs v5.12.10 upstream kernel (with downstream 
>>> dtb), with 4K page size:
>>> [    1.276818] kvm [1]: IPA Size Limit: 44 bits
>>> [    1.278425] kvm [1]: vgic interrupt IRQ9
>>> [    1.278620] kvm [1]: Hyp mode initialized successfully
>>>
>>> Could it be the PAGE size causing problem?
>>>
>>> Thanks,
>>> Qu
>>>
>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>
> 
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Any way to disable KVM VHE extension?

[Robin Murphy]

On 2021-07-15 10:44, Qu Wenruo wrote:
> 
> 
> On 2021/7/15 下午5:28, Robin Murphy wrote:
>> On 2021-07-15 09:55, Qu Wenruo wrote:
>>> Hi,
>>>
>>> Recently I'm playing around the Nvidia Xavier AGX board, which has 
>>> VHE extension support.
>>>
>>> In theory, considering the CPU and memory, it should be pretty 
>>> powerful compared to boards like RPI CM4.
>>>
>>> But to my surprise, KVM runs pretty poor on Xavier.
>>>
>>> Just booting the edk2 firmware could take over 10s, and 20s to fully 
>>> boot the kernel.
>>> Even my VM on RPI CM4 has way faster boot time, even just running on 
>>> PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
>>>
>>> This is definitely out of my expectation, I double checked to be sure 
>>> that it's running in KVM mode.
>>>
>>> But further digging shows that, since Xavier AGX CPU supports VHE, 
>>> kvm is running in VHE mode other than HYP mode on CM4.
>>>
>>> Is there anyway to manually disable VHE mode to test the more common 
>>> HYP mode on Xavier?
>>
>> According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its 
>> low-level equivalent "id_aa64mmfr1.vh=0") on the command line should 
>> do that.
> 
> Thanks for this one, I stupidly only searched modinfo of kvm, and didn't 
> even bother to search arch/arm64/kvm...
> 
>>
>> However I'd imagine the discrepancy is likely to be something more 
>> fundamental to the wildly different microarchitectures. There's 
>> certainly no harm in giving non-VHE a go for comparison, but I 
>> wouldn't be surprised if it turns out even slower...
> 
> You're totally right, with nvhe mode, it's still the same slow speed.
> 
> BTW, what did you mean by the "wildly different microarch"?
> Is ARMv8.2 arch that different from ARMv8 of RPI4?

I don't mean Armv8.x architectural features, I mean the actual 
implementation of NVIDIA's Carmel core is very, very different from 
Cortex-A72 or indeed our newer v8.2 Cortex-A designs.

> And any extra methods I could try to explore the reason of the slowness?

I guess the first check would be whether you're trapping and exiting the 
VM significantly more. I believe there are stats somewhere, but I don't 
know exactly where, sorry - I know very little about actually *using* KVM :)

If it's not that, then it might just be that EDK2 is doing a lot of 
cache maintenance or system register modification or some other 
operation that happens to be slower on Carmel compared to Cortex-A72.

Robin.

> At least RPI CM4 is beyond my expectation and is working pretty fine.
> 
> Thanks,
> Qu
> 
>>
>> Robin.
>>
>>> BTW, this is the dmesg related to KVM on Xavier, running v5.13 
>>> upstream kernel, with 64K page size:
>>> [    0.852357] kvm [1]: IPA Size Limit: 40 bits
>>> [    0.857378] kvm [1]: vgic interrupt IRQ9
>>> [    0.862122] kvm: pmu event creation failed -2
>>> [    0.866734] kvm [1]: VHE mode initialized successfully
>>>
>>> While on CM4, the host runs v5.12.10 upstream kernel (with downstream 
>>> dtb), with 4K page size:
>>> [    1.276818] kvm [1]: IPA Size Limit: 44 bits
>>> [    1.278425] kvm [1]: vgic interrupt IRQ9
>>> [    1.278620] kvm [1]: Hyp mode initialized successfully
>>>
>>> Could it be the PAGE size causing problem?
>>>
>>> Thanks,
>>> Qu
>>>
>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Any way to disable KVM VHE extension?

[Mark Rutland]

On Thu, Jul 15, 2021 at 11:00:42AM +0100, Robin Murphy wrote:
> On 2021-07-15 10:44, Qu Wenruo wrote:
> > 
> > 
> > On 2021/7/15 下午5:28, Robin Murphy wrote:
> > > On 2021-07-15 09:55, Qu Wenruo wrote:
> > > > Hi,
> > > > 
> > > > Recently I'm playing around the Nvidia Xavier AGX board, which
> > > > has VHE extension support.
> > > > 
> > > > In theory, considering the CPU and memory, it should be pretty
> > > > powerful compared to boards like RPI CM4.
> > > > 
> > > > But to my surprise, KVM runs pretty poor on Xavier.
> > > > 
> > > > Just booting the edk2 firmware could take over 10s, and 20s to
> > > > fully boot the kernel.
> > > > Even my VM on RPI CM4 has way faster boot time, even just
> > > > running on PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
> > > > 
> > > > This is definitely out of my expectation, I double checked to be
> > > > sure that it's running in KVM mode.
> > > > 
> > > > But further digging shows that, since Xavier AGX CPU supports
> > > > VHE, kvm is running in VHE mode other than HYP mode on CM4.
> > > > 
> > > > Is there anyway to manually disable VHE mode to test the more
> > > > common HYP mode on Xavier?
> > > 
> > > According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its
> > > low-level equivalent "id_aa64mmfr1.vh=0") on the command line should
> > > do that.
> > 
> > Thanks for this one, I stupidly only searched modinfo of kvm, and didn't
> > even bother to search arch/arm64/kvm...
> > 
> > > 
> > > However I'd imagine the discrepancy is likely to be something more
> > > fundamental to the wildly different microarchitectures. There's
> > > certainly no harm in giving non-VHE a go for comparison, but I
> > > wouldn't be surprised if it turns out even slower...
> > 
> > You're totally right, with nvhe mode, it's still the same slow speed.
> > 
> > BTW, what did you mean by the "wildly different microarch"?
> > Is ARMv8.2 arch that different from ARMv8 of RPI4?
> 
> I don't mean Armv8.x architectural features, I mean the actual
> implementation of NVIDIA's Carmel core is very, very different from
> Cortex-A72 or indeed our newer v8.2 Cortex-A designs.
> 
> > And any extra methods I could try to explore the reason of the slowness?
> 
> I guess the first check would be whether you're trapping and exiting the VM
> significantly more. I believe there are stats somewhere, but I don't know
> exactly where, sorry - I know very little about actually *using* KVM :)
> 
> If it's not that, then it might just be that EDK2 is doing a lot of cache
> maintenance or system register modification or some other operation that
> happens to be slower on Carmel compared to Cortex-A72.

It would also be worthchecking tha the CPUs are running at the speed you
expect, in e.g. case the lack of a DVFS driver means they're running
slow, and this just happens to be more noticeable in a VM.

You can estimate that by using `perf stat` on the host on a busy loop,
and looking what the cpu-cycles count implies.

Thanks,
Mark.

Re: Any way to disable KVM VHE extension?

[Mark Rutland]

On Thu, Jul 15, 2021 at 11:00:42AM +0100, Robin Murphy wrote:
> On 2021-07-15 10:44, Qu Wenruo wrote:
> > 
> > 
> > On 2021/7/15 下午5:28, Robin Murphy wrote:
> > > On 2021-07-15 09:55, Qu Wenruo wrote:
> > > > Hi,
> > > > 
> > > > Recently I'm playing around the Nvidia Xavier AGX board, which
> > > > has VHE extension support.
> > > > 
> > > > In theory, considering the CPU and memory, it should be pretty
> > > > powerful compared to boards like RPI CM4.
> > > > 
> > > > But to my surprise, KVM runs pretty poor on Xavier.
> > > > 
> > > > Just booting the edk2 firmware could take over 10s, and 20s to
> > > > fully boot the kernel.
> > > > Even my VM on RPI CM4 has way faster boot time, even just
> > > > running on PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
> > > > 
> > > > This is definitely out of my expectation, I double checked to be
> > > > sure that it's running in KVM mode.
> > > > 
> > > > But further digging shows that, since Xavier AGX CPU supports
> > > > VHE, kvm is running in VHE mode other than HYP mode on CM4.
> > > > 
> > > > Is there anyway to manually disable VHE mode to test the more
> > > > common HYP mode on Xavier?
> > > 
> > > According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its
> > > low-level equivalent "id_aa64mmfr1.vh=0") on the command line should
> > > do that.
> > 
> > Thanks for this one, I stupidly only searched modinfo of kvm, and didn't
> > even bother to search arch/arm64/kvm...
> > 
> > > 
> > > However I'd imagine the discrepancy is likely to be something more
> > > fundamental to the wildly different microarchitectures. There's
> > > certainly no harm in giving non-VHE a go for comparison, but I
> > > wouldn't be surprised if it turns out even slower...
> > 
> > You're totally right, with nvhe mode, it's still the same slow speed.
> > 
> > BTW, what did you mean by the "wildly different microarch"?
> > Is ARMv8.2 arch that different from ARMv8 of RPI4?
> 
> I don't mean Armv8.x architectural features, I mean the actual
> implementation of NVIDIA's Carmel core is very, very different from
> Cortex-A72 or indeed our newer v8.2 Cortex-A designs.
> 
> > And any extra methods I could try to explore the reason of the slowness?
> 
> I guess the first check would be whether you're trapping and exiting the VM
> significantly more. I believe there are stats somewhere, but I don't know
> exactly where, sorry - I know very little about actually *using* KVM :)
> 
> If it's not that, then it might just be that EDK2 is doing a lot of cache
> maintenance or system register modification or some other operation that
> happens to be slower on Carmel compared to Cortex-A72.

It would also be worthchecking tha the CPUs are running at the speed you
expect, in e.g. case the lack of a DVFS driver means they're running
slow, and this just happens to be more noticeable in a VM.

You can estimate that by using `perf stat` on the host on a busy loop,
and looking what the cpu-cycles count implies.

Thanks,
Mark.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Any way to disable KVM VHE extension?

[Mark Rutland]

On Thu, Jul 15, 2021 at 11:00:42AM +0100, Robin Murphy wrote:
> On 2021-07-15 10:44, Qu Wenruo wrote:
> > 
> > 
> > On 2021/7/15 下午5:28, Robin Murphy wrote:
> > > On 2021-07-15 09:55, Qu Wenruo wrote:
> > > > Hi,
> > > > 
> > > > Recently I'm playing around the Nvidia Xavier AGX board, which
> > > > has VHE extension support.
> > > > 
> > > > In theory, considering the CPU and memory, it should be pretty
> > > > powerful compared to boards like RPI CM4.
> > > > 
> > > > But to my surprise, KVM runs pretty poor on Xavier.
> > > > 
> > > > Just booting the edk2 firmware could take over 10s, and 20s to
> > > > fully boot the kernel.
> > > > Even my VM on RPI CM4 has way faster boot time, even just
> > > > running on PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
> > > > 
> > > > This is definitely out of my expectation, I double checked to be
> > > > sure that it's running in KVM mode.
> > > > 
> > > > But further digging shows that, since Xavier AGX CPU supports
> > > > VHE, kvm is running in VHE mode other than HYP mode on CM4.
> > > > 
> > > > Is there anyway to manually disable VHE mode to test the more
> > > > common HYP mode on Xavier?
> > > 
> > > According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its
> > > low-level equivalent "id_aa64mmfr1.vh=0") on the command line should
> > > do that.
> > 
> > Thanks for this one, I stupidly only searched modinfo of kvm, and didn't
> > even bother to search arch/arm64/kvm...
> > 
> > > 
> > > However I'd imagine the discrepancy is likely to be something more
> > > fundamental to the wildly different microarchitectures. There's
> > > certainly no harm in giving non-VHE a go for comparison, but I
> > > wouldn't be surprised if it turns out even slower...
> > 
> > You're totally right, with nvhe mode, it's still the same slow speed.
> > 
> > BTW, what did you mean by the "wildly different microarch"?
> > Is ARMv8.2 arch that different from ARMv8 of RPI4?
> 
> I don't mean Armv8.x architectural features, I mean the actual
> implementation of NVIDIA's Carmel core is very, very different from
> Cortex-A72 or indeed our newer v8.2 Cortex-A designs.
> 
> > And any extra methods I could try to explore the reason of the slowness?
> 
> I guess the first check would be whether you're trapping and exiting the VM
> significantly more. I believe there are stats somewhere, but I don't know
> exactly where, sorry - I know very little about actually *using* KVM :)
> 
> If it's not that, then it might just be that EDK2 is doing a lot of cache
> maintenance or system register modification or some other operation that
> happens to be slower on Carmel compared to Cortex-A72.

It would also be worthchecking tha the CPUs are running at the speed you
expect, in e.g. case the lack of a DVFS driver means they're running
slow, and this just happens to be more noticeable in a VM.

You can estimate that by using `perf stat` on the host on a busy loop,
and looking what the cpu-cycles count implies.

Thanks,
Mark.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Any way to disable KVM VHE extension?

[Marc Zyngier]

On Thu, 15 Jul 2021 10:44:32 +0100,
Qu Wenruo <wqu@suse.com> wrote:
> 
> 
> 
> On 2021/7/15 下午5:28, Robin Murphy wrote:
> > On 2021-07-15 09:55, Qu Wenruo wrote:
> >> Hi,
> >> 
> >> Recently I'm playing around the Nvidia Xavier AGX board, which has
> >> VHE extension support.
> >> 
> >> In theory, considering the CPU and memory, it should be pretty
> >> powerful compared to boards like RPI CM4.
> >> 
> >> But to my surprise, KVM runs pretty poor on Xavier.
> >> 
> >> Just booting the edk2 firmware could take over 10s, and 20s to
> >> fully boot the kernel.
> >> Even my VM on RPI CM4 has way faster boot time, even just running
> >> on PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
> >> 
> >> This is definitely out of my expectation, I double checked to be
> >> sure that it's running in KVM mode.
> >> 
> >> But further digging shows that, since Xavier AGX CPU supports VHE,
> >> kvm is running in VHE mode other than HYP mode on CM4.
> >> 
> >> Is there anyway to manually disable VHE mode to test the more
> >> common HYP mode on Xavier?
> > 
> > According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its
> > low-level equivalent "id_aa64mmfr1.vh=0") on the command line should
> > do that.
> 
> Thanks for this one, I stupidly only searched modinfo of kvm, and
> didn't even bother to search arch/arm64/kvm...
> 
> > 
> > However I'd imagine the discrepancy is likely to be something more
> > fundamental to the wildly different microarchitectures. There's
> > certainly no harm in giving non-VHE a go for comparison, but I
> > wouldn't be surprised if it turns out even slower...
> 
> You're totally right, with nvhe mode, it's still the same slow speed.

My experience with Denver (Nvidia's previous core) is that it is
horribly slow when running KVM. I guess that the JIT-like microarch
fares poorly with exceptions and save-restore operations.

> BTW, what did you mean by the "wildly different microarch"?
> Is ARMv8.2 arch that different from ARMv8 of RPI4?
> 
> And any extra methods I could try to explore the reason of the slowness?
> 
> At least RPI CM4 is beyond my expectation and is working pretty fine.
> 
> Thanks,
> Qu
> 
> > 
> > Robin.
> > 
> >> BTW, this is the dmesg related to KVM on Xavier, running v5.13
> >> upstream kernel, with 64K page size:
> >> [    0.852357] kvm [1]: IPA Size Limit: 40 bits
> >> [    0.857378] kvm [1]: vgic interrupt IRQ9
> >> [    0.862122] kvm: pmu event creation failed -2

And this isn't going to help finding out the bottleneck, as the kernel
doesn't find a PMU. On Denver, once the PMU is enabled, profiling
anything makes the whole thing even slower. At which point, I just
parked the board and forgot about it.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: Any way to disable KVM VHE extension?

[Marc Zyngier]

On Thu, 15 Jul 2021 10:44:32 +0100,
Qu Wenruo <wqu@suse.com> wrote:
> 
> 
> 
> On 2021/7/15 下午5:28, Robin Murphy wrote:
> > On 2021-07-15 09:55, Qu Wenruo wrote:
> >> Hi,
> >> 
> >> Recently I'm playing around the Nvidia Xavier AGX board, which has
> >> VHE extension support.
> >> 
> >> In theory, considering the CPU and memory, it should be pretty
> >> powerful compared to boards like RPI CM4.
> >> 
> >> But to my surprise, KVM runs pretty poor on Xavier.
> >> 
> >> Just booting the edk2 firmware could take over 10s, and 20s to
> >> fully boot the kernel.
> >> Even my VM on RPI CM4 has way faster boot time, even just running
> >> on PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
> >> 
> >> This is definitely out of my expectation, I double checked to be
> >> sure that it's running in KVM mode.
> >> 
> >> But further digging shows that, since Xavier AGX CPU supports VHE,
> >> kvm is running in VHE mode other than HYP mode on CM4.
> >> 
> >> Is there anyway to manually disable VHE mode to test the more
> >> common HYP mode on Xavier?
> > 
> > According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its
> > low-level equivalent "id_aa64mmfr1.vh=0") on the command line should
> > do that.
> 
> Thanks for this one, I stupidly only searched modinfo of kvm, and
> didn't even bother to search arch/arm64/kvm...
> 
> > 
> > However I'd imagine the discrepancy is likely to be something more
> > fundamental to the wildly different microarchitectures. There's
> > certainly no harm in giving non-VHE a go for comparison, but I
> > wouldn't be surprised if it turns out even slower...
> 
> You're totally right, with nvhe mode, it's still the same slow speed.

My experience with Denver (Nvidia's previous core) is that it is
horribly slow when running KVM. I guess that the JIT-like microarch
fares poorly with exceptions and save-restore operations.

> BTW, what did you mean by the "wildly different microarch"?
> Is ARMv8.2 arch that different from ARMv8 of RPI4?
> 
> And any extra methods I could try to explore the reason of the slowness?
> 
> At least RPI CM4 is beyond my expectation and is working pretty fine.
> 
> Thanks,
> Qu
> 
> > 
> > Robin.
> > 
> >> BTW, this is the dmesg related to KVM on Xavier, running v5.13
> >> upstream kernel, with 64K page size:
> >> [    0.852357] kvm [1]: IPA Size Limit: 40 bits
> >> [    0.857378] kvm [1]: vgic interrupt IRQ9
> >> [    0.862122] kvm: pmu event creation failed -2

And this isn't going to help finding out the bottleneck, as the kernel
doesn't find a PMU. On Denver, once the PMU is enabled, profiling
anything makes the whole thing even slower. At which point, I just
parked the board and forgot about it.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Any way to disable KVM VHE extension?

[Marc Zyngier]

On Thu, 15 Jul 2021 10:44:32 +0100,
Qu Wenruo <wqu@suse.com> wrote:
> 
> 
> 
> On 2021/7/15 下午5:28, Robin Murphy wrote:
> > On 2021-07-15 09:55, Qu Wenruo wrote:
> >> Hi,
> >> 
> >> Recently I'm playing around the Nvidia Xavier AGX board, which has
> >> VHE extension support.
> >> 
> >> In theory, considering the CPU and memory, it should be pretty
> >> powerful compared to boards like RPI CM4.
> >> 
> >> But to my surprise, KVM runs pretty poor on Xavier.
> >> 
> >> Just booting the edk2 firmware could take over 10s, and 20s to
> >> fully boot the kernel.
> >> Even my VM on RPI CM4 has way faster boot time, even just running
> >> on PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core.
> >> 
> >> This is definitely out of my expectation, I double checked to be
> >> sure that it's running in KVM mode.
> >> 
> >> But further digging shows that, since Xavier AGX CPU supports VHE,
> >> kvm is running in VHE mode other than HYP mode on CM4.
> >> 
> >> Is there anyway to manually disable VHE mode to test the more
> >> common HYP mode on Xavier?
> > 
> > According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its
> > low-level equivalent "id_aa64mmfr1.vh=0") on the command line should
> > do that.
> 
> Thanks for this one, I stupidly only searched modinfo of kvm, and
> didn't even bother to search arch/arm64/kvm...
> 
> > 
> > However I'd imagine the discrepancy is likely to be something more
> > fundamental to the wildly different microarchitectures. There's
> > certainly no harm in giving non-VHE a go for comparison, but I
> > wouldn't be surprised if it turns out even slower...
> 
> You're totally right, with nvhe mode, it's still the same slow speed.

My experience with Denver (Nvidia's previous core) is that it is
horribly slow when running KVM. I guess that the JIT-like microarch
fares poorly with exceptions and save-restore operations.

> BTW, what did you mean by the "wildly different microarch"?
> Is ARMv8.2 arch that different from ARMv8 of RPI4?
> 
> And any extra methods I could try to explore the reason of the slowness?
> 
> At least RPI CM4 is beyond my expectation and is working pretty fine.
> 
> Thanks,
> Qu
> 
> > 
> > Robin.
> > 
> >> BTW, this is the dmesg related to KVM on Xavier, running v5.13
> >> upstream kernel, with 64K page size:
> >> [    0.852357] kvm [1]: IPA Size Limit: 40 bits
> >> [    0.857378] kvm [1]: vgic interrupt IRQ9
> >> [    0.862122] kvm: pmu event creation failed -2

And this isn't going to help finding out the bottleneck, as the kernel
doesn't find a PMU. On Denver, once the PMU is enabled, profiling
anything makes the whole thing even slower. At which point, I just
parked the board and forgot about it.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel