Re: [PATCH v1 5/6] hw/arm/virt: Enable backup bitmap for dirty ring

[Gavin Shan <gshan@redhat.com>]

On 2/23/23 2:54 AM, Peter Maydell wrote:
> On Wed, 22 Feb 2023 at 04:36, Gavin Shan <gshan@redhat.com> wrote:
>>
>> On 2/22/23 3:27 AM, Peter Maydell wrote:
>>> Why does this need to be board-specific code? Is there
>>> some way we can just do the right thing automatically?
>>> Why does the GIC/ITS matter?
>>>
>>> The kernel should already know whether we have asked it
>>> to do something that needs this extra extension, so
>>> I think we ought to be able in the generic "enable the
>>> dirty ring" code say "if the kernel says we need this
>>> extra thing, also enable this extra thing". Or if that's
>>> too early, we can do the extra part in a generic hook a
>>> bit later.
>>>
>>> In the future there might be other things, presumably,
>>> that need the backup bitmap, so it would be more future
>>> proof not to need to also change QEMU to add extra
>>> logic checks that duplicate the logic the kernel already has.
>>>
>>
>> When the dirty ring is enabled, a per-vcpu buffer is used to track the dirty pages.
>> The prerequisite to use the per-vcpu buffer is existing running VCPU context. There
>> are two cases where no running VCPU context exists and the backup bitmap extension
>> is needed, as we know until now: (a) save/restore GICv3 tables; (b) save/restore ITS
>> tables; These two cases are related to KVM device "kvm-arm-gicv3" and "arm-its-kvm",
>> which are only needed by virt machine at present. So we needn't the backup bitmap
>> extension for other boards.
> 
> But we might have to for other boards we add later. We shouldn't
> put code in per-board if it's not really board specific.
> 
> Moreover, I think "we need the backup bitmap if the kernel is
> using its GICv3 or ITS implementation" is a kernel implementation
> detail. It seems to me that it would be cleaner if QEMU didn't
> have to hardcode "we happen to know that these are the situations
> when we need to do that". A better API would be "ask the kernel
> 'do we need this?' and enable it if it says 'yes'". The kernel
> knows what its implementations of ITS and GICv3 (and perhaps
> future in-kernel memory-using devices) require, after all.
> 

Well, As we know so far, the backup bitmap extension is only required by 'kvm-arm-gicv3'
and 'arm-its-kvm' device. Those two devices are only used by virt machine at present.
So it's a board specific requirement. I'm not sure about the future. We may need to
enable the extension for other devices and other boards. That time, the requirement
isn't board specific any more. However, we're uncertain for the future.

In order to cover the future requirement, the extension is needed by other boards,
the best way I can figure out is to enable the extension in generic path in kvm_init()
if the extension is supported by the host kernel. In this way, the unnecessary overhead
is introduced for those boards where 'kvm-arm-vgic3' and 'arm-its-kvm' aren't used.
The overhead should be very small and acceptable. Note that the host kernel don't know
if 'kvm-arm-vgic3' or 'arm-its-kvm' device is needed by the board in kvm_init(), which
is the generic path.

The 'kvm-arm-vgic3' and 'arm-its-kvm' devices are created in machvirt_init(), where
the memory slots are also added. Prior to the function, host kernel doesn't know if
the extension is needed by QEMU. It means we have to enable the extension in machvirt_init(),
which is exactly what we're doing. The difference is QEMU decides to enable the extension
instead of being told to enable it by host kernel. Host kernel doesn't have the answer to
"Hey host kernel, do we need to enable the extension" until machvirt_init() where the devices
are created. Besides, machvirt_init() isn't the generic path if we want to enable the extension
for all possible boards. Further more, the extension can't be enabled if memory slots have been
added.

In summary, the best way I can figure out is to enable the extension in kvm_init() if it
has been supported by host kernel, to cover all possible boards for future cases. Otherwise,
we keep what we're doing to enable the extension in machvirt_init(). Please let me know your
thoughts, Peter :)

Thanks,
Gavin