From mboxrd@z Thu Jan 1 00:00:00 1970 From: Bharata B Rao Subject: Re: [Qemu-devel] [RFC PATCH] Exporting Guest RAM information for NUMA binding Date: Mon, 21 Nov 2011 21:30:01 +0530 Message-ID: <20111121160001.GB3602@in.ibm.com> References: <20111029184502.GH11038@in.ibm.com> <7816C401-9BE5-48A9-8BA9-4CDAD1B39FC8@suse.de> <20111108173304.GA14486@sequoia.sous-sol.org> <20111121150054.GA3602@in.ibm.com> <1321889126.28118.5.camel@twins> Reply-To: bharata@linux.vnet.ibm.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Andrea Arcangeli , kvm list , Alexander Graf , qemu-devel Developers , Chris Wright , dipankar@in.ibm.com, Vaidyanathan S To: Peter Zijlstra Return-path: Received: from e28smtp02.in.ibm.com ([122.248.162.2]:38733 "EHLO e28smtp02.in.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751163Ab1KUQAY (ORCPT ); Mon, 21 Nov 2011 11:00:24 -0500 Received: from /spool/local by e28smtp02.in.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 21 Nov 2011 21:30:19 +0530 Received: from d28av02.in.ibm.com (d28av02.in.ibm.com [9.184.220.64]) by d28relay05.in.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id pALG09pi4260028 for ; Mon, 21 Nov 2011 21:30:10 +0530 Received: from d28av02.in.ibm.com (loopback [127.0.0.1]) by d28av02.in.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id pALG08oP031684 for ; Tue, 22 Nov 2011 03:00:09 +1100 Content-Disposition: inline In-Reply-To: <1321889126.28118.5.camel@twins> Sender: kvm-owner@vger.kernel.org List-ID: On Mon, Nov 21, 2011 at 04:25:26PM +0100, Peter Zijlstra wrote: > On Mon, 2011-11-21 at 20:48 +0530, Bharata B Rao wrote: > > > I looked at Peter's recent work in this area. > > (https://lkml.org/lkml/2011/11/17/204) > > > > It introduces two interfaces: > > > > 1. ms_tbind() to bind a thread to a memsched(*) group > > 2. ms_mbind() to bind a memory region to memsched group > > > > I assume the 2nd interface could be used by QEMU to create > > memsched groups for each of guest NUMA node memory regions. > > No, you would need both, you'll need to group vcpu threads _and_ some > vaddress space together. > > I understood QEMU currently uses a single big anonymous mmap() to > allocate the guest memory, using this you could either use multiple or > carve up the big alloc into virtual nodes by assigning different parts > to different ms groups. > > Example: suppose you want to create a 2 node guest with 8 vcpus, create > 2 ms groups, each with 4 vcpu threads and assign half the total guest > mmap to either. > > > In the past, Anthony has said that NUMA binding should be done from outside > > of QEMU (http://www.kerneltrap.org/mailarchive/linux-kvm/2010/8/31/6267041) > > If you want to expose a sense of virtual NUMA to your guest you really > have no choice there. The only thing you can do externally is run whole > VMs inside one particular node. > > > Though that was in a different context, may be we should re-look at that > > and see if QEMU still sticks to that. I know its a bit early, but if needed > > we should ask Peter to consider extending ms_mbind() to take a tid parameter > > too instead of working on current task by default. > > Uh, what for? ms_mbind() works on the current process, not task. In the original post of this mail thread, I proposed a way to export guest RAM ranges (Guest Physical Address-GPA) and their corresponding host host virtual mappings (Host Virtual Address-HVA) from QEMU (via QEMU monitor). The idea was to use this GPA to HVA mappings from tools like libvirt to bind specific parts of the guest RAM to different host nodes. This needed an extension to existing mbind() to allow binding memory of a process(QEMU) from a different process(libvirt). This was needed since we wanted to do all this from libvirt. Hence I was coming from that background when I asked for extending ms_mbind() to take a tid parameter. If QEMU community thinks that NUMA binding should all be done from outside of QEMU, it is needed, otherwise what you have should be sufficient. Regards, Bharata.