Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Peter Maydell]

KVM on big.little setups is a kernel-level question really; I've
cc'd the kvmarm list.

On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2021/12/13 21:17, Michal Prívozník wrote:
> > On 12/11/21 02:58, Qu Wenruo wrote:
> >> Hi,
> >>
> >> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4,
> >> with upstream kernels.
> >>
> >> For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG
> >> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
> >> brings quite some troubles for VMs.
> >>
> >> In short, without proper cpuset to bind the VM to either all A72 cores
> >> or all A55 cores, the VM will mostly fail to boot.
> >>
> >> Currently the working xml is:
> >>
> >>    <vcpu placement='static' cpuset='4-5'>2</vcpu>
> >>    <cpu mode='host-passthrough' check='none'/>
> >>
> >> But even with vcpupin, pinning each vcpu to each physical core, VM will
> >> mostly fail to start up due to vcpu initialization failed with -EINVAL.
> >>
> >>
> >> This brings a problem, in theory RK3399 SoC should out-perform BCM2711
> >> in multi-core performance, but if a VM can only be bind to either A72 or
> >> A55 cores, then the performance is no longer competitive against
> >> BCM2711, wasting the PCIE 2.0 x4 capacity.
> >>
> >> I guess with projects like Asahi Linux making progress, there will be
> >> more and more such problems.
> >>
> >> Any clue on how to properly pass all physical CPU cores to VM for
> >> little.BIG setup?
> >>
> >
> > I have never met big.LITTLE but my understanding was that those big
> > cores are compatible with little ones and the only difference is that
> > the big ones are shut off if there's no demand (to save energy) leaving
> > only the little ones running.
>
> The big ones are not disabled AFAIK.
>
> And even changing the CPU model to A53 (the little ones), it still fails
> to boot, thus it looks like A72 is not really able to emulate A53 cores?
>
> >
> > Anyway, this is likely too high level forum and I'd ask QEMU developers:
> >
> > https://www.qemu.org/support/
>
> That's indeed the case, adding qemu to the CC list.
>
> And I found an existing bug report:
> https://bugs.linaro.org/show_bug.cgi?id=1443

(This bug tracking system was essentially abandoned years ago; the
status of leftover bugs within it isn't indicative of anything.)

> But I still didn't get the point why the 1:1 CPU-to-vcpu mapping still
> doesn't work.

-- PMM
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Marc Zyngier]

On Mon, 13 Dec 2021 16:06:14 +0000,
Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> KVM on big.little setups is a kernel-level question really; I've
> cc'd the kvmarm list.

Thanks Peter for throwing us under the big-little bus! ;-)

> 
> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >
> >
> >
> > On 2021/12/13 21:17, Michal Prívozník wrote:
> > > On 12/11/21 02:58, Qu Wenruo wrote:
> > >> Hi,
> > >>
> > >> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4,
> > >> with upstream kernels.
> > >>
> > >> For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG
> > >> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
> > >> brings quite some troubles for VMs.
> > >>
> > >> In short, without proper cpuset to bind the VM to either all A72 cores
> > >> or all A55 cores, the VM will mostly fail to boot.

s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
though of it makes me sick).

> > >>
> > >> Currently the working xml is:
> > >>
> > >>    <vcpu placement='static' cpuset='4-5'>2</vcpu>
> > >>    <cpu mode='host-passthrough' check='none'/>
> > >>
> > >> But even with vcpupin, pinning each vcpu to each physical core, VM will
> > >> mostly fail to start up due to vcpu initialization failed with -EINVAL.

Disclaimer: I know nothing about libvirt (and no, I don't want to
know! ;-).

However, for things to be reliable, you need to taskset the whole QEMU
process to the CPU type you intend to use. That's because, AFAICT,
QEMU will snapshot the system registers outside of the vcpu threads,
and attempt to use the result to configure the actual vcpu threads. If
they happen to run on different CPU types, the sysregs will differ in
incompatible ways and an error will be returned. This may or may not
be a bug, I don't know (I see it as a feature).

If you are annoyed with this behaviour, you can always use a different
VMM that won't care about such difference (crosvm or kvmtool, to name
a few). However, the guest will be able to observe the migration from
one cpu type to another. This may or may not affect your guest's
behaviour.

I personally find the QEMU behaviour reasonable. KVM/arm64 make little
effort to support BL virtualisation as design choice (I value my
sanity), and userspace is still in control of the placement.

> > >> This brings a problem, in theory RK3399 SoC should out-perform BCM2711
> > >> in multi-core performance, but if a VM can only be bind to either A72 or
> > >> A55 cores, then the performance is no longer competitive against
> > >> BCM2711, wasting the PCIE 2.0 x4 capacity.

Vote with your money. If you too think that BL systems are utter crap,
do not buy them! Or treat them as 'two systems in one', which is what
I do. From that angle, this is of great value! ;-)

> > >> I guess with projects like Asahi Linux making progress, there will be
> > >> more and more such problems.

Well, not more than any other big-little system. They suffer from
similar issues, plus those resulting from not fully implementing the
ARM architecture. They are however more consistent in their feature
set than the ARM implementations ever were.

> > >>
> > >> Any clue on how to properly pass all physical CPU cores to VM for
> > >> little.BIG setup?
> > >>
> > >
> > > I have never met big.LITTLE but my understanding was that those big
> > > cores are compatible with little ones and the only difference is that
> > > the big ones are shut off if there's no demand (to save energy) leaving
> > > only the little ones running.

No. They are all notionally running. It is the scheduler that places
tasks (such as a vcpu) on a 'convenient' core, where 'convenient'
depends on the scheduling policy.

HTH,

	M.

-- 
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Qu Wenruo]

On 2021/12/14 00:49, Marc Zyngier wrote:
> On Mon, 13 Dec 2021 16:06:14 +0000,
> Peter Maydell <peter.maydell@linaro.org> wrote:
>>
>> KVM on big.little setups is a kernel-level question really; I've
>> cc'd the kvmarm list.
>
> Thanks Peter for throwing us under the big-little bus! ;-)
>
>>
>> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>>
>>> On 2021/12/13 21:17, Michal Prívozník wrote:
>>>> On 12/11/21 02:58, Qu Wenruo wrote:
>>>>> Hi,
>>>>>
>>>>> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4,
>>>>> with upstream kernels.
>>>>>
>>>>> For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG
>>>>> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
>>>>> brings quite some troubles for VMs.
>>>>>
>>>>> In short, without proper cpuset to bind the VM to either all A72 cores
>>>>> or all A55 cores, the VM will mostly fail to boot.
>
> s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
> though of it makes me sick).
>
>>>>>
>>>>> Currently the working xml is:
>>>>>
>>>>>     <vcpu placement='static' cpuset='4-5'>2</vcpu>
>>>>>     <cpu mode='host-passthrough' check='none'/>
>>>>>
>>>>> But even with vcpupin, pinning each vcpu to each physical core, VM will
>>>>> mostly fail to start up due to vcpu initialization failed with -EINVAL.
>
> Disclaimer: I know nothing about libvirt (and no, I don't want to
> know! ;-).
>
> However, for things to be reliable, you need to taskset the whole QEMU
> process to the CPU type you intend to use.

Yep, that's what I'm doing.

> That's because, AFAICT,
> QEMU will snapshot the system registers outside of the vcpu threads,
> and attempt to use the result to configure the actual vcpu threads. If
> they happen to run on different CPU types, the sysregs will differ in
> incompatible ways and an error will be returned. This may or may not
> be a bug, I don't know (I see it as a feature).

Then this brings another question.

If we can pin each vCPU to each physical core (both little and big),
then as long as the registers are per-vCPU based, it should be able to
pass both big and little cores to the VM.

Yeah, I totally understand this screw up the scheduling, but that's at
least what (some insane) users want (just like me).

>
> If you are annoyed with this behaviour, you can always use a different
> VMM that won't care about such difference (crosvm or kvmtool, to name
> a few).

Sounds pretty interesting, a new world but without libvirt...

> However, the guest will be able to observe the migration from
> one cpu type to another. This may or may not affect your guest's
> behaviour.

Not sure if it's possible to pin each vCPU thread to each core, but let
me try.

>
> I personally find the QEMU behaviour reasonable. KVM/arm64 make little
> effort to support BL virtualisation as design choice (I value my
> sanity), and userspace is still in control of the placement.
>
>>>>> This brings a problem, in theory RK3399 SoC should out-perform BCM2711
>>>>> in multi-core performance, but if a VM can only be bind to either A72 or
>>>>> A55 cores, then the performance is no longer competitive against
>>>>> BCM2711, wasting the PCIE 2.0 x4 capacity.
>
> Vote with your money. If you too think that BL systems are utter crap,
> do not buy them! Or treat them as 'two systems in one', which is what
> I do. From that angle, this is of great value! ;-)

I guess I'm setting my expectation too high for rk3399, just seeing its
multi-thread perf beating RPI4 and has better IO doesn't mean it's a
perfect fit for VM.

Hopes rk3588 could change it.

For now I guess overclocking the big core to 2.2G is what I can do to
grab more performance from the board.

Thanks for your detailed reason and new advices!
Qu

>
>>>>> I guess with projects like Asahi Linux making progress, there will be
>>>>> more and more such problems.
>
> Well, not more than any other big-little system. They suffer from
> similar issues, plus those resulting from not fully implementing the
> ARM architecture. They are however more consistent in their feature
> set than the ARM implementations ever were.
>
>>>>>
>>>>> Any clue on how to properly pass all physical CPU cores to VM for
>>>>> little.BIG setup?
>>>>>
>>>>
>>>> I have never met big.LITTLE but my understanding was that those big
>>>> cores are compatible with little ones and the only difference is that
>>>> the big ones are shut off if there's no demand (to save energy) leaving
>>>> only the little ones running.
>
> No. They are all notionally running. It is the scheduler that places
> tasks (such as a vcpu) on a 'convenient' core, where 'convenient'
> depends on the scheduling policy.
>
> HTH,
>
> 	M.
>
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Michal Prívozník]

On 12/14/21 01:41, Qu Wenruo wrote:
> 
> 
> On 2021/12/14 00:49, Marc Zyngier wrote:
>> On Mon, 13 Dec 2021 16:06:14 +0000,
>> Peter Maydell <peter.maydell@linaro.org> wrote:
>>>
>>> KVM on big.little setups is a kernel-level question really; I've
>>> cc'd the kvmarm list.
>>
>> Thanks Peter for throwing us under the big-little bus! ;-)
>>
>>>
>>> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>
>>>>
>>>>
>>>> On 2021/12/13 21:17, Michal Prívozník wrote:
>>>>> On 12/11/21 02:58, Qu Wenruo wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI
>>>>>> CM4,
>>>>>> with upstream kernels.
>>>>>>
>>>>>> For RPI CM4 its mostly smooth sail, but on RK3399 due to its
>>>>>> little.BIG
>>>>>> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
>>>>>> brings quite some troubles for VMs.
>>>>>>
>>>>>> In short, without proper cpuset to bind the VM to either all A72
>>>>>> cores
>>>>>> or all A55 cores, the VM will mostly fail to boot.
>>
>> s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
>> though of it makes me sick).
>>
>>>>>>
>>>>>> Currently the working xml is:
>>>>>>
>>>>>>     <vcpu placement='static' cpuset='4-5'>2</vcpu>
>>>>>>     <cpu mode='host-passthrough' check='none'/>
>>>>>>
>>>>>> But even with vcpupin, pinning each vcpu to each physical core, VM
>>>>>> will
>>>>>> mostly fail to start up due to vcpu initialization failed with
>>>>>> -EINVAL.
>>
>> Disclaimer: I know nothing about libvirt (and no, I don't want to
>> know! ;-).
>>
>> However, for things to be reliable, you need to taskset the whole QEMU
>> process to the CPU type you intend to use.
> 
> Yep, that's what I'm doing.
> 
>> That's because, AFAICT,
>> QEMU will snapshot the system registers outside of the vcpu threads,
>> and attempt to use the result to configure the actual vcpu threads. If
>> they happen to run on different CPU types, the sysregs will differ in
>> incompatible ways and an error will be returned. This may or may not
>> be a bug, I don't know (I see it as a feature).
> 
> Then this brings another question.
> 
> If we can pin each vCPU to each physical core (both little and big),
> then as long as the registers are per-vCPU based, it should be able to
> pass both big and little cores to the VM.
> 
> Yeah, I totally understand this screw up the scheduling, but that's at
> least what (some insane) users want (just like me).
> 
>>
>> If you are annoyed with this behaviour, you can always use a different
>> VMM that won't care about such difference (crosvm or kvmtool, to name
>> a few).
> 
> Sounds pretty interesting, a new world but without libvirt...
> 
>> However, the guest will be able to observe the migration from
>> one cpu type to another. This may or may not affect your guest's
>> behaviour.
> 
> Not sure if it's possible to pin each vCPU thread to each core, but let
> me try.
> 

Sure it is, for instance:

<cputune>
    <vcpupin vcpu="0" cpuset="1-4,^2"/>
    <vcpupin vcpu="1" cpuset="0,1"/>
    <vcpupin vcpu="2" cpuset="2,3"/>
    <vcpupin vcpu="3" cpuset="0,4"/>
    <emulatorpin cpuset="1-3"/>
    <iothreadpin iothread="1" cpuset="5,6"/>
    <iothreadpin iothread="2" cpuset="7,8"/>
</cputune>

pins vCPU#0 onto host CPUs 1-4, excluding 2; vCPU#1 onto host CPUs 0-1
and so on. You can also pin emulator (QEMU) and its iothreads. It's
documented here:

https://libvirt.org/formatdomain.html#cpu-tuning

Michal

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Michal Prívozník]

On 12/13/21 17:49, Marc Zyngier wrote:
> On Mon, 13 Dec 2021 16:06:14 +0000,
> Peter Maydell <peter.maydell@linaro.org> wrote:
>>
>> KVM on big.little setups is a kernel-level question really; I've
>> cc'd the kvmarm list.
> 
> Thanks Peter for throwing us under the big-little bus! ;-)
> 
>>
>> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>>
>>> On 2021/12/13 21:17, Michal Prívozník wrote:
>>>> On 12/11/21 02:58, Qu Wenruo wrote:
>>>>> Hi,
>>>>>
>>>>> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4,
>>>>> with upstream kernels.
>>>>>
>>>>> For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG
>>>>> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
>>>>> brings quite some troubles for VMs.
>>>>>
>>>>> In short, without proper cpuset to bind the VM to either all A72 cores
>>>>> or all A55 cores, the VM will mostly fail to boot.
> 
> s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
> though of it makes me sick).
> 
>>>>>
>>>>> Currently the working xml is:
>>>>>
>>>>>    <vcpu placement='static' cpuset='4-5'>2</vcpu>
>>>>>    <cpu mode='host-passthrough' check='none'/>
>>>>>
>>>>> But even with vcpupin, pinning each vcpu to each physical core, VM will
>>>>> mostly fail to start up due to vcpu initialization failed with -EINVAL.
> 
> Disclaimer: I know nothing about libvirt (and no, I don't want to
> know! ;-).
> 
> However, for things to be reliable, you need to taskset the whole QEMU
> process to the CPU type you intend to use. That's because, AFAICT,
> QEMU will snapshot the system registers outside of the vcpu threads,
> and attempt to use the result to configure the actual vcpu threads. If
> they happen to run on different CPU types, the sysregs will differ in
> incompatible ways and an error will be returned. This may or may not
> be a bug, I don't know (I see it as a feature).
> 

Oh, I had no idea that QEMU does this. Anyway, QEMU spawns a thread for
each vCPU which is perfect, because then libvirt can use CGroups to
restrict each vCPU individually onto a subset of physical CPU(s). And in
fact, libvirt has this feature for many years. I always viewed it as a
performance tuning, but TIL.

Michal

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Qu Wenruo]

On 2021/12/14 15:53, Michal Prívozník wrote:
> On 12/14/21 01:41, Qu Wenruo wrote:
>>
>>
>> On 2021/12/14 00:49, Marc Zyngier wrote:
>>> On Mon, 13 Dec 2021 16:06:14 +0000,
>>> Peter Maydell <peter.maydell@linaro.org> wrote:
>>>>
>>>> KVM on big.little setups is a kernel-level question really; I've
>>>> cc'd the kvmarm list.
>>>
>>> Thanks Peter for throwing us under the big-little bus! ;-)
>>>
>>>>
>>>> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 2021/12/13 21:17, Michal Prívozník wrote:
>>>>>> On 12/11/21 02:58, Qu Wenruo wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI
>>>>>>> CM4,
>>>>>>> with upstream kernels.
>>>>>>>
>>>>>>> For RPI CM4 its mostly smooth sail, but on RK3399 due to its
>>>>>>> little.BIG
>>>>>>> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
>>>>>>> brings quite some troubles for VMs.
>>>>>>>
>>>>>>> In short, without proper cpuset to bind the VM to either all A72
>>>>>>> cores
>>>>>>> or all A55 cores, the VM will mostly fail to boot.
>>>
>>> s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
>>> though of it makes me sick).
>>>
>>>>>>>
>>>>>>> Currently the working xml is:
>>>>>>>
>>>>>>>      <vcpu placement='static' cpuset='4-5'>2</vcpu>
>>>>>>>      <cpu mode='host-passthrough' check='none'/>
>>>>>>>
>>>>>>> But even with vcpupin, pinning each vcpu to each physical core, VM
>>>>>>> will
>>>>>>> mostly fail to start up due to vcpu initialization failed with
>>>>>>> -EINVAL.
>>>
>>> Disclaimer: I know nothing about libvirt (and no, I don't want to
>>> know! ;-).
>>>
>>> However, for things to be reliable, you need to taskset the whole QEMU
>>> process to the CPU type you intend to use.
>>
>> Yep, that's what I'm doing.
>>
>>> That's because, AFAICT,
>>> QEMU will snapshot the system registers outside of the vcpu threads,
>>> and attempt to use the result to configure the actual vcpu threads. If
>>> they happen to run on different CPU types, the sysregs will differ in
>>> incompatible ways and an error will be returned. This may or may not
>>> be a bug, I don't know (I see it as a feature).
>>
>> Then this brings another question.
>>
>> If we can pin each vCPU to each physical core (both little and big),
>> then as long as the registers are per-vCPU based, it should be able to
>> pass both big and little cores to the VM.
>>
>> Yeah, I totally understand this screw up the scheduling, but that's at
>> least what (some insane) users want (just like me).
>>
>>>
>>> If you are annoyed with this behaviour, you can always use a different
>>> VMM that won't care about such difference (crosvm or kvmtool, to name
>>> a few).
>>
>> Sounds pretty interesting, a new world but without libvirt...
>>
>>> However, the guest will be able to observe the migration from
>>> one cpu type to another. This may or may not affect your guest's
>>> behaviour.
>>
>> Not sure if it's possible to pin each vCPU thread to each core, but let
>> me try.
>>
>
> Sure it is, for instance:
>
> <cputune>
>      <vcpupin vcpu="0" cpuset="1-4,^2"/>
>      <vcpupin vcpu="1" cpuset="0,1"/>
>      <vcpupin vcpu="2" cpuset="2,3"/>
>      <vcpupin vcpu="3" cpuset="0,4"/>
>      <emulatorpin cpuset="1-3"/>
>      <iothreadpin iothread="1" cpuset="5,6"/>
>      <iothreadpin iothread="2" cpuset="7,8"/>
> </cputune>

That's what I have already tried before.
I pinned vcpu 0-6 to physical core 0-6, and still no reliable boot up.

And that's why I'm asking here.

Thanks,
Qu

>
> pins vCPU#0 onto host CPUs 1-4, excluding 2; vCPU#1 onto host CPUs 0-1
> and so on. You can also pin emulator (QEMU) and its iothreads. It's
> documented here:
>
> https://libvirt.org/formatdomain.html#cpu-tuning
>
> Michal
>
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Marc Zyngier]

On Tue, 14 Dec 2021 00:41:01 +0000,
Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> 
> 
> 
> On 2021/12/14 00:49, Marc Zyngier wrote:
> > On Mon, 13 Dec 2021 16:06:14 +0000,
> > Peter Maydell <peter.maydell@linaro.org> wrote:
> >> 
> >> KVM on big.little setups is a kernel-level question really; I've
> >> cc'd the kvmarm list.
> > 
> > Thanks Peter for throwing us under the big-little bus! ;-)
> > 
> >> 
> >> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>> 
> >>> 
> >>> 
> >>> On 2021/12/13 21:17, Michal Prívozník wrote:
> >>>> On 12/11/21 02:58, Qu Wenruo wrote:
> >>>>> Hi,
> >>>>> 
> >>>>> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4,
> >>>>> with upstream kernels.
> >>>>> 
> >>>>> For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG
> >>>>> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
> >>>>> brings quite some troubles for VMs.
> >>>>> 
> >>>>> In short, without proper cpuset to bind the VM to either all A72 cores
> >>>>> or all A55 cores, the VM will mostly fail to boot.
> > 
> > s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
> > though of it makes me sick).
> > 
> >>>>> 
> >>>>> Currently the working xml is:
> >>>>> 
> >>>>>     <vcpu placement='static' cpuset='4-5'>2</vcpu>
> >>>>>     <cpu mode='host-passthrough' check='none'/>
> >>>>> 
> >>>>> But even with vcpupin, pinning each vcpu to each physical core, VM will
> >>>>> mostly fail to start up due to vcpu initialization failed with -EINVAL.
> > 
> > Disclaimer: I know nothing about libvirt (and no, I don't want to
> > know! ;-).
> > 
> > However, for things to be reliable, you need to taskset the whole QEMU
> > process to the CPU type you intend to use.
> 
> Yep, that's what I'm doing.

Are you sure? The xml directive above seem to only apply to the vcpus,
and no other QEMU thread.

> > That's because, AFAICT,
> > QEMU will snapshot the system registers outside of the vcpu threads,
> > and attempt to use the result to configure the actual vcpu threads. If
> > they happen to run on different CPU types, the sysregs will differ in
> > incompatible ways and an error will be returned. This may or may not
> > be a bug, I don't know (I see it as a feature).
> 
> Then this brings another question.
> 
> If we can pin each vCPU to each physical core (both little and big),
> then as long as the registers are per-vCPU based, it should be able to
> pass both big and little cores to the VM.

Absolutely. But that's not how QEMU works. It assumes that it can
restore the *same* registers to all the vcpus. Which of course doesn't
work (we don't allow you to change MIDR_EL1, for a start).

> Yeah, I totally understand this screw up the scheduling, but that's at
> least what (some insane) users want (just like me).

That's fine, we all have our own use cases.

> 
> > 
> > If you are annoyed with this behaviour, you can always use a different
> > VMM that won't care about such difference (crosvm or kvmtool, to name
> > a few).
> 
> Sounds pretty interesting, a new world but without libvirt...
>
> > However, the guest will be able to observe the migration from
> > one cpu type to another. This may or may not affect your guest's
> > behaviour.
> 
> Not sure if it's possible to pin each vCPU thread to each core, but let
> me try.

Again: the problem isn't the vcpu threads, but the dummy VM that QEMU
creates to snapshot the vcpu registers.

> > I personally find the QEMU behaviour reasonable. KVM/arm64 make little
> > effort to support BL virtualisation as design choice (I value my
> > sanity), and userspace is still in control of the placement.
> > 
> >>>>> This brings a problem, in theory RK3399 SoC should out-perform BCM2711
> >>>>> in multi-core performance, but if a VM can only be bind to either A72 or
> >>>>> A55 cores, then the performance is no longer competitive against
> >>>>> BCM2711, wasting the PCIE 2.0 x4 capacity.
> > 
> > Vote with your money. If you too think that BL systems are utter crap,
> > do not buy them! Or treat them as 'two systems in one', which is what
> > I do. From that angle, this is of great value! ;-)
> 
> I guess I'm setting my expectation too high for rk3399, just seeing its
> multi-thread perf beating RPI4 and has better IO doesn't mean it's a
> perfect fit for VM.

I find my own rk3399 perfectly adequate with QEMU.

HTH,

	M.

-- 
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Marc Zyngier]

On Tue, 14 Dec 2021 08:16:40 +0000,
Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> 
> 
> 
> On 2021/12/14 15:53, Michal Prívozník wrote:
> > On 12/14/21 01:41, Qu Wenruo wrote:
> >> 
> >> 
> >> On 2021/12/14 00:49, Marc Zyngier wrote:
> >>> On Mon, 13 Dec 2021 16:06:14 +0000,
> >>> Peter Maydell <peter.maydell@linaro.org> wrote:
> >>>> 
> >>>> KVM on big.little setups is a kernel-level question really; I've
> >>>> cc'd the kvmarm list.
> >>> 
> >>> Thanks Peter for throwing us under the big-little bus! ;-)
> >>> 
> >>>> 
> >>>> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >>>>> 
> >>>>> 
> >>>>> 
> >>>>> On 2021/12/13 21:17, Michal Prívozník wrote:
> >>>>>> On 12/11/21 02:58, Qu Wenruo wrote:
> >>>>>>> Hi,
> >>>>>>> 
> >>>>>>> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI
> >>>>>>> CM4,
> >>>>>>> with upstream kernels.
> >>>>>>> 
> >>>>>>> For RPI CM4 its mostly smooth sail, but on RK3399 due to its
> >>>>>>> little.BIG
> >>>>>>> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
> >>>>>>> brings quite some troubles for VMs.
> >>>>>>> 
> >>>>>>> In short, without proper cpuset to bind the VM to either all A72
> >>>>>>> cores
> >>>>>>> or all A55 cores, the VM will mostly fail to boot.
> >>> 
> >>> s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
> >>> though of it makes me sick).
> >>> 
> >>>>>>> 
> >>>>>>> Currently the working xml is:
> >>>>>>> 
> >>>>>>>      <vcpu placement='static' cpuset='4-5'>2</vcpu>
> >>>>>>>      <cpu mode='host-passthrough' check='none'/>
> >>>>>>> 
> >>>>>>> But even with vcpupin, pinning each vcpu to each physical core, VM
> >>>>>>> will
> >>>>>>> mostly fail to start up due to vcpu initialization failed with
> >>>>>>> -EINVAL.
> >>> 
> >>> Disclaimer: I know nothing about libvirt (and no, I don't want to
> >>> know! ;-).
> >>> 
> >>> However, for things to be reliable, you need to taskset the whole QEMU
> >>> process to the CPU type you intend to use.
> >> 
> >> Yep, that's what I'm doing.
> >> 
> >>> That's because, AFAICT,
> >>> QEMU will snapshot the system registers outside of the vcpu threads,
> >>> and attempt to use the result to configure the actual vcpu threads. If
> >>> they happen to run on different CPU types, the sysregs will differ in
> >>> incompatible ways and an error will be returned. This may or may not
> >>> be a bug, I don't know (I see it as a feature).
> >> 
> >> Then this brings another question.
> >> 
> >> If we can pin each vCPU to each physical core (both little and big),
> >> then as long as the registers are per-vCPU based, it should be able to
> >> pass both big and little cores to the VM.
> >> 
> >> Yeah, I totally understand this screw up the scheduling, but that's at
> >> least what (some insane) users want (just like me).
> >> 
> >>> 
> >>> If you are annoyed with this behaviour, you can always use a different
> >>> VMM that won't care about such difference (crosvm or kvmtool, to name
> >>> a few).
> >> 
> >> Sounds pretty interesting, a new world but without libvirt...
> >> 
> >>> However, the guest will be able to observe the migration from
> >>> one cpu type to another. This may or may not affect your guest's
> >>> behaviour.
> >> 
> >> Not sure if it's possible to pin each vCPU thread to each core, but let
> >> me try.
> >> 
> > 
> > Sure it is, for instance:
> > 
> > <cputune>
> >      <vcpupin vcpu="0" cpuset="1-4,^2"/>
> >      <vcpupin vcpu="1" cpuset="0,1"/>
> >      <vcpupin vcpu="2" cpuset="2,3"/>
> >      <vcpupin vcpu="3" cpuset="0,4"/>
> >      <emulatorpin cpuset="1-3"/>
> >      <iothreadpin iothread="1" cpuset="5,6"/>
> >      <iothreadpin iothread="2" cpuset="7,8"/>
> > </cputune>
> 
> That's what I have already tried before.
> I pinned vcpu 0-6 to physical core 0-6, and still no reliable boot up.
> 
> And that's why I'm asking here.

You are still missing the point of how QEMU works:

- QEMU creates a dummy VM with a single vcpu. This can happen on *any*
  CPU.
- It snapshots the sysregs for this vcpu, and keep them for later
- It then destroy this VM
- QEMU then creates the full VM, with all the vcpus
- Each vcpu gets initialised with the state saved earlier. If any vcpu
  is initialised on a physical CPU of a different type from the one
  that has been used for the dummy VM, you lose, as we cannot restore
  some of the registers such as MIDR_EL1 (and other registers that KVM
  considers as invariant).

To fix this, you need to change QEMU's notion of a template VM, or
change KVM's notion of invariant registers. The former is quite hard,
and the later breaks a ton of things for guests, such as errata
workarounds.

The best workaround is to taskset the QEMU process (and I really mean
the process, not individual threads) to an homogeneous set of CPUs and
be done with it.

	M.

-- 
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Peter Maydell]

On Tue, 14 Dec 2021 at 09:52, Marc Zyngier <maz@kernel.org> wrote:
> You are still missing the point of how QEMU works:
>
> - QEMU creates a dummy VM with a single vcpu. This can happen on *any*
>   CPU.
> - It snapshots the sysregs for this vcpu, and keep them for later
> - It then destroy this VM
> - QEMU then creates the full VM, with all the vcpus
> - Each vcpu gets initialised with the state saved earlier. If any vcpu
>   is initialised on a physical CPU of a different type from the one
>   that has been used for the dummy VM, you lose, as we cannot restore
>   some of the registers such as MIDR_EL1 (and other registers that KVM
>   considers as invariant).

Put another way, QEMU's "-cpu host" is exactly one thing
(the "create a dummy VM and snapshot" steps above are where
QEMU defines what "-cpu host" means), and we have an implicit
assumption that the VM must only have one kind of guest CPU,
and not be heterogenous.

> The best workaround is to taskset the QEMU process (and I really mean
> the process, not individual threads) to an homogeneous set of CPUs and
> be done with it.

Agreed. I suspect that often the 'little' CPUs are sufficiently
low-power to probably not be worth giving to the VM anyway.

Side note: if you *do* give a guest both big and little CPUs
using kvmtool or something similar, does the guest kernel get
enough information to schedule tasks properly to both kinds of
CPU, or does it just assume they're all the same and happily
put performance-requiring tasks on the little CPUs ?

-- PMM
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Qu Wenruo]

On 2021/12/14 17:52, Marc Zyngier wrote:
> On Tue, 14 Dec 2021 08:16:40 +0000,
> Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2021/12/14 15:53, Michal Prívozník wrote:
>>> On 12/14/21 01:41, Qu Wenruo wrote:
>>>>
>>>>
>>>> On 2021/12/14 00:49, Marc Zyngier wrote:
>>>>> On Mon, 13 Dec 2021 16:06:14 +0000,
>>>>> Peter Maydell <peter.maydell@linaro.org> wrote:
>>>>>>
>>>>>> KVM on big.little setups is a kernel-level question really; I've
>>>>>> cc'd the kvmarm list.
>>>>>
>>>>> Thanks Peter for throwing us under the big-little bus! ;-)
>>>>>
>>>>>>
>>>>>> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 2021/12/13 21:17, Michal Prívozník wrote:
>>>>>>>> On 12/11/21 02:58, Qu Wenruo wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI
>>>>>>>>> CM4,
>>>>>>>>> with upstream kernels.
>>>>>>>>>
>>>>>>>>> For RPI CM4 its mostly smooth sail, but on RK3399 due to its
>>>>>>>>> little.BIG
>>>>>>>>> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
>>>>>>>>> brings quite some troubles for VMs.
>>>>>>>>>
>>>>>>>>> In short, without proper cpuset to bind the VM to either all A72
>>>>>>>>> cores
>>>>>>>>> or all A55 cores, the VM will mostly fail to boot.
>>>>>
>>>>> s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
>>>>> though of it makes me sick).
>>>>>
>>>>>>>>>
>>>>>>>>> Currently the working xml is:
>>>>>>>>>
>>>>>>>>>       <vcpu placement='static' cpuset='4-5'>2</vcpu>
>>>>>>>>>       <cpu mode='host-passthrough' check='none'/>
>>>>>>>>>
>>>>>>>>> But even with vcpupin, pinning each vcpu to each physical core, VM
>>>>>>>>> will
>>>>>>>>> mostly fail to start up due to vcpu initialization failed with
>>>>>>>>> -EINVAL.
>>>>>
>>>>> Disclaimer: I know nothing about libvirt (and no, I don't want to
>>>>> know! ;-).
>>>>>
>>>>> However, for things to be reliable, you need to taskset the whole QEMU
>>>>> process to the CPU type you intend to use.
>>>>
>>>> Yep, that's what I'm doing.
>>>>
>>>>> That's because, AFAICT,
>>>>> QEMU will snapshot the system registers outside of the vcpu threads,
>>>>> and attempt to use the result to configure the actual vcpu threads. If
>>>>> they happen to run on different CPU types, the sysregs will differ in
>>>>> incompatible ways and an error will be returned. This may or may not
>>>>> be a bug, I don't know (I see it as a feature).
>>>>
>>>> Then this brings another question.
>>>>
>>>> If we can pin each vCPU to each physical core (both little and big),
>>>> then as long as the registers are per-vCPU based, it should be able to
>>>> pass both big and little cores to the VM.
>>>>
>>>> Yeah, I totally understand this screw up the scheduling, but that's at
>>>> least what (some insane) users want (just like me).
>>>>
>>>>>
>>>>> If you are annoyed with this behaviour, you can always use a different
>>>>> VMM that won't care about such difference (crosvm or kvmtool, to name
>>>>> a few).
>>>>
>>>> Sounds pretty interesting, a new world but without libvirt...
>>>>
>>>>> However, the guest will be able to observe the migration from
>>>>> one cpu type to another. This may or may not affect your guest's
>>>>> behaviour.
>>>>
>>>> Not sure if it's possible to pin each vCPU thread to each core, but let
>>>> me try.
>>>>
>>>
>>> Sure it is, for instance:
>>>
>>> <cputune>
>>>       <vcpupin vcpu="0" cpuset="1-4,^2"/>
>>>       <vcpupin vcpu="1" cpuset="0,1"/>
>>>       <vcpupin vcpu="2" cpuset="2,3"/>
>>>       <vcpupin vcpu="3" cpuset="0,4"/>
>>>       <emulatorpin cpuset="1-3"/>
>>>       <iothreadpin iothread="1" cpuset="5,6"/>
>>>       <iothreadpin iothread="2" cpuset="7,8"/>
>>> </cputune>
>>
>> That's what I have already tried before.
>> I pinned vcpu 0-6 to physical core 0-6, and still no reliable boot up.
>>
>> And that's why I'm asking here.
>
> You are still missing the point of how QEMU works:
>
> - QEMU creates a dummy VM with a single vcpu. This can happen on *any*
>    CPU.

This is the main point that I missed.

Thanks very much for point this out.

> - It snapshots the sysregs for this vcpu, and keep them for later
> - It then destroy this VM
> - QEMU then creates the full VM, with all the vcpus
> - Each vcpu gets initialised with the state saved earlier. If any vcpu
>    is initialised on a physical CPU of a different type from the one
>    that has been used for the dummy VM, you lose, as we cannot restore
>    some of the registers such as MIDR_EL1 (and other registers that KVM
>    considers as invariant).
>
> To fix this, you need to change QEMU's notion of a template VM, or
> change KVM's notion of invariant registers. The former is quite hard,
> and the later breaks a ton of things for guests, such as errata
> workarounds.

> The best workaround is to taskset the QEMU process (and I really mean
> the process, not individual threads) to an homogeneous set of CPUs and
> be done with it.

Yeah, that's why the cpuset way is working, as it seems also limiting
the initial "temporary" VM creating to specified CPUs.

Just curious, is there some defined common VM related registers that can
be restore on all cores? (At least for A53 + A72 case).

If completely no, then virtualization is really not even targeted for
BIG.little designs I guess.

Thanks,
Qu

>
> 	M.
>
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Marc Zyngier]

On Tue, 14 Dec 2021 10:08:29 +0000,
Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> On Tue, 14 Dec 2021 at 09:52, Marc Zyngier <maz@kernel.org> wrote:
> > The best workaround is to taskset the QEMU process (and I really mean
> > the process, not individual threads) to an homogeneous set of CPUs and
> > be done with it.
> 
> Agreed. I suspect that often the 'little' CPUs are sufficiently
> low-power to probably not be worth giving to the VM anyway.

They are actually pretty beefy, in most cases. If you are I/O bound,
the small cores are the right target. This email is brought to you via
a couple of A55 CPUs running a VM and acting as my home gateway. If
'the internet' was getting slow, I'd have heard about it! ;-)

> Side note: if you *do* give a guest both big and little CPUs
> using kvmtool or something similar, does the guest kernel get
> enough information to schedule tasks properly to both kinds of
> CPU, or does it just assume they're all the same and happily
> put performance-requiring tasks on the little CPUs ?

Upstream kvmtool doesn't make any effort to do this, but I had a
series to perform such placement in the past. Once you know which
physical CPUs you are placing the vcpus on, you can extract the
individual CPU capacity[1] and expose it in the guest's DT as a
cpu-specific 'capacity-dmips-mhz' property. A Linux guest will take
the hint and do the right thing.

	M.

[1] /sys/devices/system/cpu/cpu*/cpu_capacity

-- 
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Daniel P. Berrangé]

On Tue, Dec 14, 2021 at 09:34:18AM +0000, Marc Zyngier wrote:
> On Tue, 14 Dec 2021 00:41:01 +0000,
> Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > 
> > 
> > 
> > On 2021/12/14 00:49, Marc Zyngier wrote:
> > > On Mon, 13 Dec 2021 16:06:14 +0000,
> > > Peter Maydell <peter.maydell@linaro.org> wrote:
> > >> 
> > >> KVM on big.little setups is a kernel-level question really; I've
> > >> cc'd the kvmarm list.
> > > 
> > > Thanks Peter for throwing us under the big-little bus! ;-)
> > > 
> > >> 
> > >> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > >>> 
> > >>> 
> > >>> 
> > >>> On 2021/12/13 21:17, Michal Prívozník wrote:
> > >>>> On 12/11/21 02:58, Qu Wenruo wrote:
> > >>>>> Hi,
> > >>>>> 
> > >>>>> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4,
> > >>>>> with upstream kernels.
> > >>>>> 
> > >>>>> For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG
> > >>>>> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
> > >>>>> brings quite some troubles for VMs.
> > >>>>> 
> > >>>>> In short, without proper cpuset to bind the VM to either all A72 cores
> > >>>>> or all A55 cores, the VM will mostly fail to boot.
> > > 
> > > s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
> > > though of it makes me sick).
> > > 
> > >>>>> 
> > >>>>> Currently the working xml is:
> > >>>>> 
> > >>>>>     <vcpu placement='static' cpuset='4-5'>2</vcpu>
> > >>>>>     <cpu mode='host-passthrough' check='none'/>
> > >>>>> 
> > >>>>> But even with vcpupin, pinning each vcpu to each physical core, VM will
> > >>>>> mostly fail to start up due to vcpu initialization failed with -EINVAL.
> > > 
> > > Disclaimer: I know nothing about libvirt (and no, I don't want to
> > > know! ;-).
> > > 
> > > However, for things to be reliable, you need to taskset the whole QEMU
> > > process to the CPU type you intend to use.
> > 
> > Yep, that's what I'm doing.
> 
> Are you sure? The xml directive above seem to only apply to the vcpus,
> and no other QEMU thread.

For historical reasons this XML element is a bit misleadingly named.

With the config

   <vcpu placement='static' cpuset='4-5'>2</vcpu>

the 'cpuset' applies to the QEMU process as a whole - its vCPUs,
I/O threads and any other emulator threads.

There is a separate config for setting per-VCPU binding which was
illustrated elsewhere in this thread.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Qu Wenruo]

On 2021/12/14 18:36, Daniel P. Berrangé wrote:
> On Tue, Dec 14, 2021 at 09:34:18AM +0000, Marc Zyngier wrote:
>> On Tue, 14 Dec 2021 00:41:01 +0000,
>> Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>>
>>> On 2021/12/14 00:49, Marc Zyngier wrote:
>>>> On Mon, 13 Dec 2021 16:06:14 +0000,
>>>> Peter Maydell <peter.maydell@linaro.org> wrote:
>>>>>
>>>>> KVM on big.little setups is a kernel-level question really; I've
>>>>> cc'd the kvmarm list.
>>>>
>>>> Thanks Peter for throwing us under the big-little bus! ;-)
>>>>
>>>>>
>>>>> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2021/12/13 21:17, Michal Prívozník wrote:
>>>>>>> On 12/11/21 02:58, Qu Wenruo wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4,
>>>>>>>> with upstream kernels.
>>>>>>>>
>>>>>>>> For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG
>>>>>>>> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
>>>>>>>> brings quite some troubles for VMs.
>>>>>>>>
>>>>>>>> In short, without proper cpuset to bind the VM to either all A72 cores
>>>>>>>> or all A55 cores, the VM will mostly fail to boot.
>>>>
>>>> s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
>>>> though of it makes me sick).
>>>>
>>>>>>>>
>>>>>>>> Currently the working xml is:
>>>>>>>>
>>>>>>>>      <vcpu placement='static' cpuset='4-5'>2</vcpu>
>>>>>>>>      <cpu mode='host-passthrough' check='none'/>
>>>>>>>>
>>>>>>>> But even with vcpupin, pinning each vcpu to each physical core, VM will
>>>>>>>> mostly fail to start up due to vcpu initialization failed with -EINVAL.
>>>>
>>>> Disclaimer: I know nothing about libvirt (and no, I don't want to
>>>> know! ;-).
>>>>
>>>> However, for things to be reliable, you need to taskset the whole QEMU
>>>> process to the CPU type you intend to use.
>>>
>>> Yep, that's what I'm doing.
>>
>> Are you sure? The xml directive above seem to only apply to the vcpus,
>> and no other QEMU thread.
>
> For historical reasons this XML element is a bit misleadingly named.
>
> With the config
>
>     <vcpu placement='static' cpuset='4-5'>2</vcpu>
>
> the 'cpuset' applies to the QEMU process as a whole - its vCPUs,
> I/O threads and any other emulator threads.
>
> There is a separate config for setting per-VCPU binding which was
> illustrated elsewhere in this thread.

Which also means, I can put the io threads to A53 cores freeing up the
A72 cores more.

And is there any plan to deprecate the old "cpuset" key of vcpu element,
and recommend to use "vcpupin" element?

Thanks,
Qu

>
> Regards,
> Daniel
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Marc Zyngier]

On Tue, 14 Dec 2021 10:21:40 +0000,
Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> On 2021/12/14 17:52, Marc Zyngier wrote:
> > The best workaround is to taskset the QEMU process (and I really mean
> > the process, not individual threads) to an homogeneous set of CPUs and
> > be done with it.
> 
> Yeah, that's why the cpuset way is working, as it seems also limiting
> the initial "temporary" VM creating to specified CPUs.
> 
> Just curious, is there some defined common VM related registers that can
> be restore on all cores? (At least for A53 + A72 case).

Most of the registers are common, and most of the feature registers
are actually massaged by KVM to make them look homogeneous if even the
HW isn't. There are however a few registers that need to be exposed to
the guest verbatim, and MIDR_EL1 is the most important one, as it
carries the core 'identity', which an operating system will absolutely
need to implement critical workarounds (and there are no shortage of
them on both A53 and A72).

> If completely no, then virtualization is really not even targeted for
> BIG.little designs I guess.

If your use of virtualisation is to be completely abstracted from the
underlying HW, then you are right, that doesn't really work at all.
Not by design, but because all implementations have embarrassing warts
that need some sort of workarounds.

	M.

-- 
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Daniel P. Berrangé]

On Tue, Dec 14, 2021 at 06:59:12PM +0800, Qu Wenruo wrote:
> 
> 
> On 2021/12/14 18:36, Daniel P. Berrangé wrote:
> > On Tue, Dec 14, 2021 at 09:34:18AM +0000, Marc Zyngier wrote:
> > > On Tue, 14 Dec 2021 00:41:01 +0000,
> > > Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > > > 
> > > > 
> > > > 
> > > > On 2021/12/14 00:49, Marc Zyngier wrote:
> > > > > On Mon, 13 Dec 2021 16:06:14 +0000,
> > > > > Peter Maydell <peter.maydell@linaro.org> wrote:
> > > > > > 
> > > > > > KVM on big.little setups is a kernel-level question really; I've
> > > > > > cc'd the kvmarm list.
> > > > > 
> > > > > Thanks Peter for throwing us under the big-little bus! ;-)
> > > > > 
> > > > > > 
> > > > > > On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > On 2021/12/13 21:17, Michal Prívozník wrote:
> > > > > > > > On 12/11/21 02:58, Qu Wenruo wrote:
> > > > > > > > > Hi,
> > > > > > > > > 
> > > > > > > > > Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4,
> > > > > > > > > with upstream kernels.
> > > > > > > > > 
> > > > > > > > > For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG
> > > > > > > > > setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
> > > > > > > > > brings quite some troubles for VMs.
> > > > > > > > > 
> > > > > > > > > In short, without proper cpuset to bind the VM to either all A72 cores
> > > > > > > > > or all A55 cores, the VM will mostly fail to boot.
> > > > > 
> > > > > s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
> > > > > though of it makes me sick).
> > > > > 
> > > > > > > > > 
> > > > > > > > > Currently the working xml is:
> > > > > > > > > 
> > > > > > > > >      <vcpu placement='static' cpuset='4-5'>2</vcpu>
> > > > > > > > >      <cpu mode='host-passthrough' check='none'/>
> > > > > > > > > 
> > > > > > > > > But even with vcpupin, pinning each vcpu to each physical core, VM will
> > > > > > > > > mostly fail to start up due to vcpu initialization failed with -EINVAL.
> > > > > 
> > > > > Disclaimer: I know nothing about libvirt (and no, I don't want to
> > > > > know! ;-).
> > > > > 
> > > > > However, for things to be reliable, you need to taskset the whole QEMU
> > > > > process to the CPU type you intend to use.
> > > > 
> > > > Yep, that's what I'm doing.
> > > 
> > > Are you sure? The xml directive above seem to only apply to the vcpus,
> > > and no other QEMU thread.
> > 
> > For historical reasons this XML element is a bit misleadingly named.
> > 
> > With the config
> > 
> >     <vcpu placement='static' cpuset='4-5'>2</vcpu>
> > 
> > the 'cpuset' applies to the QEMU process as a whole - its vCPUs,
> > I/O threads and any other emulator threads.
> > 
> > There is a separate config for setting per-VCPU binding which was
> > illustrated elsewhere in this thread.
> 
> Which also means, I can put the io threads to A53 cores freeing up the
> A72 cores more.
> 
> And is there any plan to deprecate the old "cpuset" key of vcpu element,
> and recommend to use "vcpupin" element?

No, they're complementary as they're operating at different levels
and not every scenario needs this fine grained level.
In fact if you just use 'vcpupin' and don't provide 'cpuset', then
internally treats it as if 'cpuset' was the union of all 'vcpupin'
bitsets.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm