From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eduardo Habkost Subject: Re: [libvirt] TSC frequency configuration & invtsc migration (was Re: [PATCH 4/4] kvm: Allow migration with invtsc) Date: Thu, 5 Jan 2017 10:48:57 -0200 Message-ID: <20170105124857.GQ3315@thinpad.lan.raisama.net> References: <1482866480-26208-1-git-send-email-ehabkost@redhat.com> <1482866480-26208-5-git-send-email-ehabkost@redhat.com> <20170104115656.GB14961@amt.cnet> <20170104133916.GG3315@thinpad.lan.raisama.net> <20170104195917.GM3315@thinpad.lan.raisama.net> <20170104222623.GA21789@amt.cnet> <20170105013631.GO3315@thinpad.lan.raisama.net> <20170105104830.GB6299@amt.cnet> <20170105121950.GP3315@thinpad.lan.raisama.net> <20170105123351.GF3292@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Haozhong Zhang , kvm@vger.kernel.org, libvir-list@redhat.com, Marcelo Tosatti , qemu-devel@nongnu.org, Paolo Bonzini To: "Daniel P. Berrange" Return-path: Content-Disposition: inline In-Reply-To: <20170105123351.GF3292@redhat.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+gceq-qemu-devel=gmane.org@nongnu.org Sender: "Qemu-devel" List-Id: kvm.vger.kernel.org On Thu, Jan 05, 2017 at 12:33:51PM +0000, Daniel P. Berrange wrote: > On Thu, Jan 05, 2017 at 10:19:50AM -0200, Eduardo Habkost wrote: > > On Thu, Jan 05, 2017 at 08:48:32AM -0200, Marcelo Tosatti wrote: > > > On Wed, Jan 04, 2017 at 11:36:31PM -0200, Eduardo Habkost wrote: [...] > > > > Scenario 2: > > > > > > > > Host A has TSC scaling, host B doesn't have TSC scaling. We want > > > > to be able to start the VM on host A, and migrate to B. In this > > > > case, the only possible solution is to use B's frequency when > > > > starting the VM. The QEMU process doesn't have enough information > > > > to make that decision. > > > > > > That is a good point. But again, its a special case and > > > should be supported by -cpu xxx,tsc-frequency=zzzz. > > > > > > However, for the vast majority of 99.999% cases, the issue > > > can be handled entirely in QEMU, without libvirt involvement, > > > and without adding extra steps to the management software. > > > > I agree it should cover most cases. The only problem here is that > > it can break migration in unexpected ways. > > > > Then my point is: assuming that libvirt will prefer to require > > explicit TSC frequency configuration to enable invtsc migration > > (instead of getting unpredictable migration compatibility), is > > the added complexity to migration code worth the effort, if > > choosing an explicit frequency is safer and more predictable? I > > believe this is where we disagree. > > I believe that if libvirt detects that QEMU supports the 'tsc-frequency' > option, then libvirt should set it by default in the XML, if not already > set by the mgmt app. That way, libvirt can validate TSC freq comapt > for migration before it even launches QEMU in the target host. If you do this unconditionally, you have another problem: if tsc-frequency is set explicitly, migration is only possible if TSC frequency of the destination matches[1], or if TSC scaling is supported by the destination. It's a good idea to set a TSC frequency only if invtsc is enabled explicitly in the config. [1] Currently the frequency needs to match exactly. That's a separate issue: we should probably add a knob to allow a slight variation in TSC frequency (e.g. <1% difference). -- Eduardo From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56109) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cP7Te-0001Fp-2f for qemu-devel@nongnu.org; Thu, 05 Jan 2017 07:49:06 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cP7TZ-0003Kl-49 for qemu-devel@nongnu.org; Thu, 05 Jan 2017 07:49:06 -0500 Received: from mx1.redhat.com ([209.132.183.28]:52966) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cP7TY-0003Kc-Rw for qemu-devel@nongnu.org; Thu, 05 Jan 2017 07:49:01 -0500 Date: Thu, 5 Jan 2017 10:48:57 -0200 From: Eduardo Habkost Message-ID: <20170105124857.GQ3315@thinpad.lan.raisama.net> References: <1482866480-26208-1-git-send-email-ehabkost@redhat.com> <1482866480-26208-5-git-send-email-ehabkost@redhat.com> <20170104115656.GB14961@amt.cnet> <20170104133916.GG3315@thinpad.lan.raisama.net> <20170104195917.GM3315@thinpad.lan.raisama.net> <20170104222623.GA21789@amt.cnet> <20170105013631.GO3315@thinpad.lan.raisama.net> <20170105104830.GB6299@amt.cnet> <20170105121950.GP3315@thinpad.lan.raisama.net> <20170105123351.GF3292@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170105123351.GF3292@redhat.com> Subject: Re: [Qemu-devel] [libvirt] TSC frequency configuration & invtsc migration (was Re: [PATCH 4/4] kvm: Allow migration with invtsc) List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Daniel P. Berrange" Cc: Marcelo Tosatti , libvir-list@redhat.com, Paolo Bonzini , qemu-devel@nongnu.org, kvm@vger.kernel.org, Haozhong Zhang On Thu, Jan 05, 2017 at 12:33:51PM +0000, Daniel P. Berrange wrote: > On Thu, Jan 05, 2017 at 10:19:50AM -0200, Eduardo Habkost wrote: > > On Thu, Jan 05, 2017 at 08:48:32AM -0200, Marcelo Tosatti wrote: > > > On Wed, Jan 04, 2017 at 11:36:31PM -0200, Eduardo Habkost wrote: [...] > > > > Scenario 2: > > > > > > > > Host A has TSC scaling, host B doesn't have TSC scaling. We want > > > > to be able to start the VM on host A, and migrate to B. In this > > > > case, the only possible solution is to use B's frequency when > > > > starting the VM. The QEMU process doesn't have enough information > > > > to make that decision. > > > > > > That is a good point. But again, its a special case and > > > should be supported by -cpu xxx,tsc-frequency=zzzz. > > > > > > However, for the vast majority of 99.999% cases, the issue > > > can be handled entirely in QEMU, without libvirt involvement, > > > and without adding extra steps to the management software. > > > > I agree it should cover most cases. The only problem here is that > > it can break migration in unexpected ways. > > > > Then my point is: assuming that libvirt will prefer to require > > explicit TSC frequency configuration to enable invtsc migration > > (instead of getting unpredictable migration compatibility), is > > the added complexity to migration code worth the effort, if > > choosing an explicit frequency is safer and more predictable? I > > believe this is where we disagree. > > I believe that if libvirt detects that QEMU supports the 'tsc-frequency' > option, then libvirt should set it by default in the XML, if not already > set by the mgmt app. That way, libvirt can validate TSC freq comapt > for migration before it even launches QEMU in the target host. If you do this unconditionally, you have another problem: if tsc-frequency is set explicitly, migration is only possible if TSC frequency of the destination matches[1], or if TSC scaling is supported by the destination. It's a good idea to set a TSC frequency only if invtsc is enabled explicitly in the config. [1] Currently the frequency needs to match exactly. That's a separate issue: we should probably add a knob to allow a slight variation in TSC frequency (e.g. <1% difference). -- Eduardo