Re: [Qemu-devel] [PATCH 2/2] Add monitor command mem-nodes

[Eduardo Habkost <ehabkost@redhat.com>]

On Thu, Jun 13, 2013 at 08:04:00PM -0500, Anthony Liguori wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
> > On Wed, Jun 05, 2013 at 07:57:42AM -0500, Anthony Liguori wrote:
> >> Wanlong Gao <gaowanlong@cn.fujitsu.com> writes:
> >> 
> >> > Add monitor command mem-nodes to show the huge mapped
> >> > memory nodes locations.
> >> >
> >> > (qemu) info mem-nodes
> >> > /proc/14132/fd/13: 00002aaaaac00000-00002aaaeac00000: node0
> >> > /proc/14132/fd/13: 00002aaaeac00000-00002aab2ac00000: node1
> >> > /proc/14132/fd/14: 00002aab2ac00000-00002aab2b000000: node0
> >> > /proc/14132/fd/14: 00002aab2b000000-00002aab2b400000: node1
> >> 
> >> This creates an ABI that we don't currently support.  Memory hotplug or
> >> a variety of things can break this mapping and then we'd have to provide
> >> an interface to describe that the mapping was broken.
> >
> > What do you mean by "breaking this mapping", exactly? Would the backing
> > file of existing guest RAM ever change? (It would require a memory copy
> > from one file to another, why would QEMU ever do that?)
> 
> Memory hot-add will change the mapping.  hot-remove (if ever
> implemented) would break it.

So, would the backing-file/offset of existing guest RAM ever change? (It
would require a memory copy from one file to another, why would QEMU
ever do that?)


[...]
> >
> > Does THP work with tmpfs, already?
> 
> No.

OK, that's a real problem.


> > If it does, people who doesn't want
> > hugetlbfs and want numa tuning to work with THP could just use tmpfs for
> > -mem-path.
> >
> >> 
> >> I had hoped that we would get proper userspace interfaces for describing
> >> memory groups but that appears to have stalled out.
> >
> > I would love to have it. But while we don't have it, sharing the
> > tmpfs/hugetlbfs backing files seem to work just fine as a mechanism to
> > let other tools manipulate guest memory policy.  We just need to let
> > external tools know where the backing files are.
> 
> Is this meant for numad?  Wouldn't you want numad to work without
> hugetlbfs?
> 
> You have to preallocate pages to hugetlbfs.  It's very difficult to use
> in practice.

If you don't want hugetlbfs you could use tmpfs, and set the policy on
the tmpfs files. What I am asking is: why do we need to ask the kernel
folks for interfaces to define and set policies on memory groups if we
can (in theory) do the exactly same using tmpfs and hugetlbfs files?

(But the fact that THP doesn't work with tmpfs is a real problem, as I
said above)

> >> 
> >> Does anyone know if this is still on the table?
> >> 
> >> If we can't get a proper kernel interface, then perhaps we need to add
> >> full libnuma support but that would really be unfortunate...
> >
> > Why isn't the "info mem-nodes" solution (I mean: not this version, but a
> > proper QMP version that exposes all the information we need) an
> > option?
> 
> We're exposing internal QEMU information (the HVA -> GPA mapping) as an
> external stable interface.

I never wanted to expose the HVA -> GPA mapping. What I want to expose
is:

 * The virtual-NUMA-node -> GPA-range mapping
 * The GPA -> mem-path file/offset mapping

(Alternatively, a simple virtual-NUMA-node -> mem-path file/offset
mapping would be enough, too)

We could even replace "mem-path file/offset mapping" with "memory
groups", if the kernel already had interfaces to deal with memory
groups.

-- 
Eduardo