Re: [PATCH] util: NUMA aware memory preallocation

From: David Hildenbrand <david@redhat.com>
To: "Daniel P. Berrangé" <berrange@redhat.com>
Cc: Michal Privoznik <mprivozn@redhat.com>, qemu-devel@nongnu.org
Subject: Re: [PATCH] util: NUMA aware memory preallocation
Date: Wed, 11 May 2022 12:03:24 +0200	[thread overview]
Message-ID: <a11ca582-3d17-b064-6736-bd66ed5a5ec4@redhat.com> (raw)
In-Reply-To: <YnuDONrdbMcJT08p@redhat.com>

On 11.05.22 11:34, Daniel P. Berrangé wrote:
> On Wed, May 11, 2022 at 11:31:23AM +0200, David Hildenbrand wrote:
>>>> Long story short, management application has no way of learning
>>>> TIDs of allocator threads so it can't make them run NUMA aware.
>>>
>>> This feels like the key issue. The preallocation threads are
>>> invisible to libvirt, regardless of whether we're doing coldplug
>>> or hotplug of memory-backends. Indeed the threads are invisible
>>> to all of QEMU, except the memory backend code.
>>>
>>> Conceptually we need 1 or more explicit worker threads, that we
>>> can assign CPU affinity to, and then QEMU can place jobs on them.
>>> I/O threads serve this role, but limited to blockdev work. We
>>> need a generalization of I/O threads, for arbitrary jobs that
>>> QEMU might want to farm out to specific numa nodes.
>>
>> At least the "-object iothread" thingy can already be used for actions
>> outside of blockdev. virtio-balloon uses one for free page hinting.
> 
> Ah that's good to know, so my idea probably isn't so much work as
> I thought it might be.

I guess we'd have to create a bunch of iothreads on the command line and
then feed them as an array to the memory backend we want to create. We
could then forward the threads to a new variant of os_mem_prealloc().

We could

a) Allocate new iothreads for each memory backend we create. Hm, that
might be suboptimal, we could end up with many iothreads.

b) Reuse iothreads and have separate sets of iothreads per NUMA node.
Assign them to a node once.

c) Reuse iothreads and reassign them to NUMA nodes on demand.

However, I'm not sure what the semantics are when having multiple
backends referencing the iothreads ...

Regarding c) and alternative  I raised is that we could simply
preallocate the threads in QEMU internally on machine creation, and let
libvirt reassign them to the fitting NUMA node whenever
coldplugging/hotplugging a new memory backend.

-- 
Thanks,

David / dhildenb