From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9EF5C32796 for ; Tue, 23 Aug 2022 20:43:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234156AbiHWUnb (ORCPT ); Tue, 23 Aug 2022 16:43:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51244 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234036AbiHWUnH (ORCPT ); Tue, 23 Aug 2022 16:43:07 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BDE6B64C5; Tue, 23 Aug 2022 13:35:33 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 3CADCB81F3B; Tue, 23 Aug 2022 20:35:32 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E1F85C433D7; Tue, 23 Aug 2022 20:35:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1661286931; bh=0JNK622xWgO/a9K2BWJzsLD+0hXBjzLzKVy98NParO4=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=H6sJwrSDAGGP51crSveoiDT/qRVGGPOt/wIzeSIv0HjAmTS88tXrmCfkeFkG1Jekk t9klrKPWCizqfa3R1G+i5fVMKc65sMDQL4Vavg2QHzN4WCUoxMAyOCAsGMXFdfancV u0fQK06VgJaHtzinJ0QcUQucPds/aLc5zvNHBYdTqziY8YL6dxLOB1UBDOa8iEwVFe peZI1qOoU/LPxWonuZ7bclgwVgXf+azagqSkQ1ZX7OY1SAlYUjeuohe2ILhCmv2olG K9ScYIc9GY38dAIni7CybxbgfdXtj7MsxVX9ntIelXPPPMJwVqzAQ6bBeIB8ma44Al NMSVwhtwoHzVw== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1oQacS-005I5Q-K9; Tue, 23 Aug 2022 21:35:28 +0100 Date: Tue, 23 Aug 2022 21:35:27 +0100 Message-ID: <87a67uwve8.wl-maz@kernel.org> From: Marc Zyngier To: Oliver Upton Cc: Gavin Shan , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, peterx@redhat.com, pbonzini@redhat.com, corbet@lwn.net, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org, shuah@kernel.org, seanjc@google.com, drjones@redhat.com, dmatlack@google.com, bgardon@google.com, ricarkol@google.com, zhenyzha@redhat.com, shan.gavin@gmail.com Subject: Re: [PATCH v1 1/5] KVM: arm64: Enable ring-based dirty memory tracking In-Reply-To: References: <20220819005601.198436-1-gshan@redhat.com> <20220819005601.198436-2-gshan@redhat.com> <87lerkwtm5.wl-maz@kernel.org> <41fb5a1f-29a9-e6bb-9fab-4c83a2a8fce5@redhat.com> <87fshovtu0.wl-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: oliver.upton@linux.dev, gshan@redhat.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, peterx@redhat.com, pbonzini@redhat.com, corbet@lwn.net, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org, shuah@kernel.org, seanjc@google.com, drjones@redhat.com, dmatlack@google.com, bgardon@google.com, ricarkol@google.com, zhenyzha@redhat.com, shan.gavin@gmail.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 23 Aug 2022 15:44:42 +0100, Oliver Upton wrote: > > On Mon, Aug 22, 2022 at 10:42:15PM +0100, Marc Zyngier wrote: > > Hi Gavin, > > > > On Mon, 22 Aug 2022 02:58:20 +0100, > > Gavin Shan wrote: > > > > > > Hi Marc, > > > > > > On 8/19/22 6:00 PM, Marc Zyngier wrote: > > > > On Fri, 19 Aug 2022 01:55:57 +0100, > > > > Gavin Shan wrote: > > > >> > > > >> The ring-based dirty memory tracking has been available and enabled > > > >> on x86 for a while. The feature is beneficial when the number of > > > >> dirty pages is small in a checkpointing system or live migration > > > >> scenario. More details can be found from fb04a1eddb1a ("KVM: X86: > > > >> Implement ring-based dirty memory tracking"). > > > >> > > > >> This enables the ring-based dirty memory tracking on ARM64. It's > > > >> notable that no extra reserved ring entries are needed on ARM64 > > > >> because the huge pages are always split into base pages when page > > > >> dirty tracking is enabled. > > > > > > > > Can you please elaborate on this? Adding a per-CPU ring of course > > > > results in extra memory allocation, so there must be a subtle > > > > x86-specific detail that I'm not aware of... > > > > > > > > > > Sure. I guess it's helpful to explain how it works in next revision. > > > Something like below: > > > > > > This enables the ring-based dirty memory tracking on ARM64. The feature > > > is enabled by CONFIG_HAVE_KVM_DIRTY_RING, detected and enabled by > > > CONFIG_HAVE_KVM_DIRTY_RING. A ring buffer is created on every vcpu and > > > each entry is described by 'struct kvm_dirty_gfn'. The ring buffer is > > > pushed by host when page becomes dirty and pulled by userspace. A vcpu > > > exit is forced when the ring buffer becomes full. The ring buffers on > > > all vcpus can be reset by ioctl command KVM_RESET_DIRTY_RINGS. > > > > > > Yes, I think so. Adding a per-CPU ring results in extra memory allocation. > > > However, it's avoiding synchronization among multiple vcpus when dirty > > > pages happen on multiple vcpus. More discussion can be found from [1] > > > > Oh, I totally buy the relaxation of the synchronisation (though I > > doubt this will have any visible effect until we have something like > > Oliver's patches to allow parallel faulting). > > > > Heh, yeah I need to get that out the door. I'll also note that Gavin's > changes are still relevant without that series, as we do write unprotect > in parallel at PTE granularity after commit f783ef1c0e82 ("KVM: arm64: > Add fast path to handle permission relaxation during dirty logging"). Ah, true. Now if only someone could explain how the whole producer-consumer thing works without a trace of a barrier, that'd be great... Thanks, M. -- Without deviation from the norm, progress is not possible.