Re: [Qemu-devel] [PATCH 2/2] Add monitor command mem-nodes

[Wanlong Gao <gaowanlong@cn.fujitsu.com>]

On 06/11/2013 09:40 PM, Eduardo Habkost wrote:
> On Tue, Jun 11, 2013 at 03:22:13PM +0800, Wanlong Gao wrote:
>> On 06/05/2013 09:46 PM, Eduardo Habkost wrote:
>>> On Wed, Jun 05, 2013 at 11:58:25AM +0800, Wanlong Gao wrote:
>>>> Add monitor command mem-nodes to show the huge mapped
>>>> memory nodes locations.
>>>>
>>>
>>> This is for machine consumption, so we need a QMP command.
>>>
>>>> (qemu) info mem-nodes
>>>> /proc/14132/fd/13: 00002aaaaac00000-00002aaaeac00000: node0
>>>> /proc/14132/fd/13: 00002aaaeac00000-00002aab2ac00000: node1
>>>> /proc/14132/fd/14: 00002aab2ac00000-00002aab2b000000: node0
>>>> /proc/14132/fd/14: 00002aab2b000000-00002aab2b400000: node1
>>>
>>> Are node0/node1 _host_ nodes?
>>>
>>> How do I know what's the _guest_ address/node corresponding to each
>>> file/range above?
>>>
>>> What I am really looking for is:
>>>
>>>  * The correspondence between guest (virtual) NUMA nodes and guest
>>>    physical address ranges (it could be provided by the QMP version of
>>>    "info numa")
>>
>> AFAIK, the guest NUMA nodes and guest physical address ranges are set
>> by seabios, we can't get this information from QEMU,
> 
> QEMU _has_ to know about it, otherwise we would never be able to know
> which virtual addresses inside the QEMU process (or offsets inside the
> backing files) belong to which virtual NUMA node.

Nope, if I'm right, actually it's linear except that there are holes in
the physical address spaces. So we can know which node the guest virtual
address is included just by each numa node size. It's enough for us if we
can provide a QMP interface from QEMU to let external tools like libvirt
set the host memory binding polices according to the QMP interface, and
we can also provide the QEMU command line option to be able to set host
bindings through command line options before we start QEMU process.

> 
> (After all, the NUMA wiring is a hardware feature, not something that
> the BIOS can decide)

But this is ACPI table which wrote by seabios now. AFAIK, there is no
unified idea about moving this part to QEMU with the QEMU interfaces
for seabios removed or just stay it there.


> 
> 
>> and I think this
>> information is useless for pinning memory range to host.
> 
> Well, we have to somehow identify each region of guest memory when
> deciding how to pin it. How would you identify it without using guest
> physical addresses? Guest physical addresses are more meaningful than
> the QEMU virtual addresses your patch exposes (that are meaningless
> outside QEMU).

As I mentioned above, we can know this just by the guest node memory size,
and can set the host bindings by treating this sizes as offsets.
And I think we only need to set the host memory binding polices to each
guest numa nodes. It's unnecessary to set polices to each region as you
said.

> 
> 
> 
>>>  * The correspondence between guest physical address ranges and ranges
>>>    inside the mapped files (so external tools could set the policy on
>>>    those files instead of requiring QEMU to set it directly)
>>>
>>> I understand that your use case may require additional information and
>>> additional interfaces. But if we provide the information above we will
>>> allow external components set the policy on the hugetlbfs files before
>>> we add new interfaces required for your use case.
>>
>> But the file backed memory is not good for the host which has many
>> virtual machines, in this situation, we can't handle anon THP yet.
> 
> I don't understand what you mean, here. What prevents someone from using
> file-backed memory with multiple virtual machines?

While if we use hugetlbfs backed memory, we should know how many virtual machines,
how much memory each vm will use, then reserve these pages for them. And even
should reserve more pages for external tools(numactl) to set memory polices.
Even the memory reservation also has it's own memory policies. It's very hard
to control it to what we want to set.


> 
>>
>> And as I mentioned, the cross numa node access performance regression
>> is caused by pci-passthrough, it's a very long time bug, we should
>> back port the host memory pinning patch to old QEMU to resolve this performance
>> problem, too.
> 
> If it's a regression, what's the last version of QEMU where the bug
> wasn't present?
> 

 As QEMU doesn't support host memory binding, I think
this was present since we support guest NUMA, and the pci-passthrough made
it even worse.


Thanks,
Wanlong Gao