From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1752454AbdKMKqk (ORCPT <rfc822;w@1wt.eu>);
        Mon, 13 Nov 2017 05:46:40 -0500
Received: from bombadil.infradead.org ([65.50.211.133]:47011 "EHLO
        bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1751464AbdKMKqj (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Mon, 13 Nov 2017 05:46:39 -0500
Date: Mon, 13 Nov 2017 11:46:34 +0100
From: Peter Zijlstra <peterz@infradead.org>
To: Wanpeng Li <kernellwp@gmail.com>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        kvm <kvm@vger.kernel.org>, Paolo Bonzini <pbonzini@redhat.com>,
        Radim Kr??m???? <rkrcmar@redhat.com>,
        Wanpeng Li <wanpeng.li@hotmail.com>
Subject: Re: [PATCH v4 2/4] KVM: X86: Add paravirt remote TLB flush
Message-ID: <20171113104634.62cmjalnj73scfnl@hirez.programming.kicks-ass.net>
References: <1510533206-9821-1-git-send-email-wanpeng.li@hotmail.com>
 <1510533206-9821-3-git-send-email-wanpeng.li@hotmail.com>
 <20171113080403.ici7nt5r4b5fpxa6@hirez.programming.kicks-ass.net>
 <CANRm+Cw0x4dFUrvCYFeeCHRibd--0ZVmEcr-4aoKiMiOh9TKUA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CANRm+Cw0x4dFUrvCYFeeCHRibd--0ZVmEcr-4aoKiMiOh9TKUA@mail.gmail.com>
User-Agent: NeoMutt/20170609 (1.8.3)
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Nov 13, 2017 at 04:26:57PM +0800, Wanpeng Li wrote:
> 2017-11-13 16:04 GMT+08:00 Peter Zijlstra <peterz@infradead.org>:

> > So if at this point a vCPU gets preempted we'll still spin-wait for it,
> > which is sub-optimal.
> >
> > I think we can come up with something to get around that 'problem' if
> > indeed it is a problem. But we can easily do that as follow up patches.
> > Just let me know if you think its worth spending more time on.
> 
> You can post your idea, it is always smart. :) Then we can evaluate
> the complexity and gains.

I'm not sure I have a fully baked idea just yet, but the general idea
would be something like:

 - switch (back) to a dedicated TLB invalidate IPI

 - introduce KVM_VCPU_IPI_PENDING

 - change flush_tlb_others() into something like:

   for_each_cpu(cpu, flushmask) {
	 src = &per_cpu(steal_time, cpu);
	 state = READ_ONCE(src->preempted);
	 do {
		 if (state & KVM_VCPU_PREEMPTED) {
			 if (try_cmpxchg(&src->preempted, &state,
						 state | KVM_VCPU_SHOULD_FLUSH)) {
				 __cpumask_clear_cpu(cpu, flushmask);
				 break;
			 }
		 }
	 } while (!try_cmpxchg(&src->preempted, &state,
				 state | KVM_VCPU_IPI_PENDING));
   }

   apic->send_IPI_mask(flushmask, CALL_TLB_VECTOR);

   for_each_cpu(cpu, flushmask) {
	 src = &per_cpu(steal_time, cpu);
	 smp_cond_load_acquire(&src->preempted, !(VAL & KVM_VCPU_IPI_PENDING);
   }


 - have the TLB invalidate handler do something like:

   state = READ_ONCE(src->preempted);
   if (!(state & KVM_VCPU_IPI_PENDING))
	   return;

   local_flush_tlb();

   do {
   } while (!try_cmpxchg(&src->preempted, &state,
			 state & ~KVM_VCPU_IPI_PENDING));

 - then at VMEXIT time do something like:

   state = READ_ONCE(src->preempted);
   do {
	if (!(state & KVM_VCPU_IPI_PENDING))
		break;
   } while (!try_cmpxchg(&src->preempted, state,
			 (state & ~KVM_VCPU_IPI_PENDING) |
			 KVM_VCPU_SHOULD_FLUSH));

   and clear any possible pending TLB_VECTOR in the guest state to avoid
   raising that IPI spuriously on enter again.


This way the preemption will clear the IPI_PENDING and the
flush_others() wait loop will terminate.