From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752443AbcDWXHA (ORCPT ); Sat, 23 Apr 2016 19:07:00 -0400 Received: from mail-ob0-f194.google.com ([209.85.214.194]:36139 "EHLO mail-ob0-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751986AbcDWXG6 (ORCPT ); Sat, 23 Apr 2016 19:06:58 -0400 MIME-Version: 1.0 In-Reply-To: <20160422091233.26c23c34@redhat.com> References: <20160404164607.09e306fa@redhat.com> <1459803623.6219.28.camel@redhat.com> <57035899.6080609@gmail.com> <20160405084046.21f68608@redhat.com> <20160422091233.26c23c34@redhat.com> Date: Sun, 24 Apr 2016 07:06:57 +0800 Message-ID: Subject: Re: [PATCH] kvm: x86: make lapic hrtimer pinned From: Wanpeng Li To: Luiz Capitulino Cc: Yang Zhang , Rik van Riel , kvm , "linux-kernel@vger.kernel.org" , Paolo Bonzini , Radim Krcmar , Marcelo Tosatti , Bandan Das Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 2016-04-22 21:12 GMT+08:00 Luiz Capitulino : > On Fri, 22 Apr 2016 07:12:51 +0800 > Wanpeng Li wrote: > >> 2016-04-05 20:40 GMT+08:00 Luiz Capitulino : >> > On Tue, 5 Apr 2016 14:18:01 +0800 >> > Yang Zhang wrote: >> > >> >> On 2016/4/5 5:00, Rik van Riel wrote: >> >> > On Mon, 2016-04-04 at 16:46 -0400, Luiz Capitulino wrote: >> >> >> When a vCPU runs on a nohz_full core, the hrtimer used by >> >> >> the lapic emulation code can be migrated to another core. >> >> >> When this happens, it's possible to observe milisecond >> >> >> latency when delivering timer IRQs to KVM guests. >> >> >> >> >> >> The huge latency is mainly due to the fact that >> >> >> apic_timer_fn() expects to run during a kvm exit. It >> >> >> sets KVM_REQ_PENDING_TIMER and let it be handled on kvm >> >> >> entry. However, if the timer fires on a different core, >> >> >> we have to wait until the next kvm exit for the guest >> >> >> to see KVM_REQ_PENDING_TIMER set. >> >> >> >> >> >> This problem became visible after commit 9642d18ee. This >> >> >> commit changed the timer migration code to always attempt >> >> >> to migrate timers away from nohz_full cores. While it's >> >> >> discussable if this is correct/desirable (I don't think >> >> >> it is), it's clear that the lapic emulation code has >> >> >> a requirement on firing the hrtimer in the same core >> >> >> where it was started. This is achieved by making the >> >> >> hrtimer pinned. >> >> > >> >> > Given that delivering a timer to a guest seems to >> >> > involve trapping from the guest to the host, anyway, >> >> > I don't see a downside to your patch. >> >> > >> >> > If that is ever changed (eg. allowing delivery of >> >> > a timer interrupt to a VCPU without trapping to the >> >> > host), we may want to revisit this. >> >> >> >> >> >> Posted interrupt helps in this case. Currently, KVM doesn't use PI for >> >> lapic timer is due to same affinity for lapic timer and VCPU. Now, we >> >> can change to use PI for lapic timer. The only concern is what's >> >> frequency of timer migration in upstream Linux? If it is frequently, >> >> will it bring additional cost? >> > >> > I can't answer this questions. >> > >> >> BTW, in what case the migration of timers during VCPU scheduling will fail? >> > >> > For hrtimers (which is the lapic emulation case), it only succeeds if >> > the destination core has a hrtimer expiring before the hrtimer being >> > migrated. >> >> Interesting, did you figure out why this happen? Actually the clock >> event device will be reprogrammed if the expire time of the new >> enqueued hrtimer is earlier than the left most(earliest expire time) >> hrtimer in hrtimer rb tree. > > Unless the code has changed very recently, what you describe is > what happens when queueing a hrtimer in the same core. Migrating a > hrtimer to a different core is a different case. You are right! Regards, Wanpeng Li