From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD6D6C433F5 for ; Wed, 2 Mar 2022 07:29:11 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 1CEAB49EC2; Wed, 2 Mar 2022 02:29:11 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, body has been altered) header.i=@kernel.org Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vTNJ2EJog9Sh; Wed, 2 Mar 2022 02:29:08 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id C836049EC0; Wed, 2 Mar 2022 02:29:08 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 6EFAA49EBD for ; Wed, 2 Mar 2022 02:29:07 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GsR1hPMMhzhi for ; Wed, 2 Mar 2022 02:29:02 -0500 (EST) Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id A880F49EBC for ; Wed, 2 Mar 2022 02:29:02 -0500 (EST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3564661929; Wed, 2 Mar 2022 07:29:01 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 991A3C004E1; Wed, 2 Mar 2022 07:29:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1646206140; bh=jWx+LGSjHYVPOLqQ2ixdypgnUdVNer561Qll5V5i7BQ=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=tXNaR/6M0E2MuSbHYp4aD8AEs9SCS2Q4Ai/cwXyNqOPD3+WvNsmummpIWaJG3bJLt JRRkfeNlFC7WYhW2Ss5R2uoJgct8RWFGj5zLe6JxXi/LeP4YldlctBSYRZQ5byXSy7 q2uGmxt4xqb1fylGEXvvq833f5fT1CGSt0hY4zYh/YmZzmm19jfXBPmszcy+dpZOpf yTh8WaQIs99wPRUKGNVRXHnAgFeo3jez5MCQA6G/KiSdFhT1tdA9zaSV1CCRe6qKPm yH4PmYChMFxIzB+qQroQHf6ggbs5JDc1+Pgr25sYlkvT8uIu0GsoxOS/15o01NOiEV oDY0Fyg4SpGpA== Received: from ip-185-104-136-29.ptr.icomera.net ([185.104.136.29] helo=billy-the-mountain.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nPJPu-00BbX8-2J; Wed, 02 Mar 2022 07:28:58 +0000 Date: Wed, 02 Mar 2022 07:28:56 +0000 Message-ID: <87wnhc6cef.wl-maz@kernel.org> From: Marc Zyngier To: Eugene Huang Subject: Re: Timer delays in VM In-Reply-To: References: <667c9f084b2d38725369de60daef6d58@misterjones.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.104.136.29 X-SA-Exim-Rcpt-To: eugeneh@nvidia.com, kvmarm@lists.cs.columbia.edu X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Cc: "kvmarm@lists.cs.columbia.edu" X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Tue, 01 Mar 2022 19:03:33 +0000, Eugene Huang wrote: > > > > * Does this timer rely on kvm timer irq injection? > > > > Yes. A timer interrupt is always injected in SW. But the timer interrupt can > > either come from the HW timer itself (the VM was running while the timer > > expired), or from a SW timer that KVM as setup if the guest was blocked on > > WFI. > > Here for arm64, EL1Virtual Timer is used. EL1 Virtual Timer is > a HW timer, correct? There is an armvtimer implementation in QEMU > 6.1+. Does this armvtimer make a difference? KVM only deals with the EL1 timers (both physical and virtual). I guess that by 'armvtimer', you mean libvirt's front-end for the stolen time feature to expose to the guest how wall clock and CPU time diverge (i.e. it isn't a timer at all, but a dynamic correction for it). > > > * What can be any possible causes for the timer delay? Are there > > > some locking mechanisms which can cause the delay? > > > > This completely depend on how loaded your host is, the respective priorities > > of the various processes, and a million of other things. > > This is no different from the same userspace running on the host. > > It also depends on the *guest* kernel, by the way. > > Our guest kernel is 5.4. How is the *guest* kernel involved? > Can you give an example? Do you have suggestions on the guest kernel > version as well. It is the guest kernel that programs the timer, and KVM isn't involved at all, specially on your HW (direct access to both timers on VHE-capable systems). > > > * What parameters can tune this timer? > > > > None. You may want to check whether the delay is observed when the VM > > has hit WFI or not. > > Yes, delay is observed after vm_exit because of WFx (not sure > WFI or WFE) but only when on a different vCPU in the same VM some > workload is started. Let me see if I understand what you mean: - vcpu-0 is running your timer test, everything is fine - vcpu-1 starts some other workload, and this affects the timer test on the other vcpu Is that correct? It so, this would tend to indicate that both vcpu share some physical resources such as a physical CPU. How do you run your VM? Also, please work out whether you exit because of a blocking WFI or WFE, as they are indicative of different guest behaviour. > Since we pin that workload to its own vCPU, in > theory, it should not affect the timing of another vCPU. Why not? a vcpu is just a host thread, and if they share a physical CPU at some point, there is a knock-on effect. > > You also don't mention what host kernel version you are running. > > In general, please try and reproduce the issue using the latest kernel version > > (5.16 at the moment). Please also indicate what HW you are using. > > Tried 5.15 and 5.4 kernels. Both have the issue. Do you think > 5.16 can make a difference? The HW is an Ampere Altra system. Unlikely. The Altra is a mostly sane system, as long as you make sure that VMs don't migrate across sockets (at which point it becomes laughably bad). Nothing to do with KVM though. Are these kernels compiled from scratch? Or are they whatever the distro ships? Same question for the guest. Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm