From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F929C433EC for ; Mon, 27 Jul 2020 17:12:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 41C44206E7 for ; Mon, 27 Jul 2020 17:12:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="lQ1Z9K59" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 41C44206E7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D19B46B0003; Mon, 27 Jul 2020 13:12:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CC3166B0006; Mon, 27 Jul 2020 13:12:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B93466B0007; Mon, 27 Jul 2020 13:12:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0197.hostedemail.com [216.40.44.197]) by kanga.kvack.org (Postfix) with ESMTP id A5F276B0003 for ; Mon, 27 Jul 2020 13:12:56 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 5DE2A3625 for ; Mon, 27 Jul 2020 17:12:56 +0000 (UTC) X-FDA: 77084500752.14.team81_4d1050b26f62 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 2263E18227CE4 for ; Mon, 27 Jul 2020 17:12:38 +0000 (UTC) X-HE-Tag: team81_4d1050b26f62 X-Filterd-Recvd-Size: 5069 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Mon, 27 Jul 2020 17:12:37 +0000 (UTC) Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 9B562206E7; Mon, 27 Jul 2020 17:12:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1595869956; bh=WogOJT1SpSbDm9BIS5mq+VMEccrmGTd/fbEvNfTWPgM=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=lQ1Z9K59C8/AlkD1/eM/JwEqvlO7ySlctuRPoQqZ36nRK1Dy7JlBuq/54o4cIjEcD cbn9dbymYVSeLmrss7+oAwKbKw0LEVhdVjYi0OVwO3tnBP+DzAKy8Wn2N0a0srie7z DjhgZiDTdIYE3B13+xlYN3P1+D7OOQalV77YMcwg= Received: from disco-boy.misterjones.org ([51.254.78.96] helo=www.loen.fr) by disco-boy.misterjones.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1k06fy-00FNOn-Va; Mon, 27 Jul 2020 18:12:35 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Date: Mon, 27 Jul 2020 18:12:34 +0100 From: Marc Zyngier To: Zhenyu Ye Cc: james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org, steven.price@arm.com, mark.rutland@arm.com, ascull@google.com, kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, arm@kernel.org, xiexiangyou@huawei.com Subject: Re: [RESEND RFC PATCH v1] arm64: kvm: flush tlbs by range in unmap_stage2_range function In-Reply-To: References: <20200724134315.805-1-yezhenyu2@huawei.com> <5d54c860b3b4e7a98e4d53397e6424ae@kernel.org> User-Agent: Roundcube Webmail/1.4.5 Message-ID: X-Sender: maz@kernel.org X-SA-Exim-Connect-IP: 51.254.78.96 X-SA-Exim-Rcpt-To: yezhenyu2@huawei.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org, steven.price@arm.com, mark.rutland@arm.com, ascull@google.com, kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, arm@kernel.org, xiexiangyou@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-Rspamd-Queue-Id: 2263E18227CE4 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Zhenyu, On 2020-07-27 15:51, Zhenyu Ye wrote: > Hi Marc, >=20 > On 2020/7/26 1:40, Marc Zyngier wrote: >> On 2020-07-24 14:43, Zhenyu Ye wrote: >>> Now in unmap_stage2_range(), we flush tlbs one by one just after the >>> corresponding pages cleared.=C2=A0 However, this may cause some=20 >>> performance >>> problems when the unmap range is very large (such as when the vm >>> migration rollback, this may cause vm downtime too loog). >>=20 >> You keep resending this patch, but you don't give any numbers >> that would back your assertion. >=20 > I have tested the downtime of vm migration rollback on arm64, and found > the downtime could even take up to 7s. Then I traced the cost of > unmap_stage2_range() and found it could take a maximum of 1.2s. The > vm configuration is as follows (with high memory pressure, the dirty > rate is about 500MB/s): >=20 > 192 > 48 > > > > > This means nothing to me, I'm afraid. >=20 > After this patch applied, the cost of unmap_stage2_range() can reduce=20 > to > 16ms, and VM downtime can be less than 1s. >=20 > The following figure shows a clear comparison: >=20 > | vm downtime | cost of unmap_stage2_range() > --------------+--------------+---------------------------------- > before change | 7s | 1200 ms > after change | 1s | 16 ms > --------------+--------------+---------------------------------- I don't see how you turn a 1.184s reduction into a 6s gain. Surely there is more to it than what you posted. >>> + >>> +=C2=A0=C2=A0=C2=A0 if ((end - start) >=3D 512 << (PAGE_SHIFT - 12)) = { >>> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __tlbi(vmalls12e1is); >>=20 >> And what is this magic value based on? You don't even mention in the >> commit log that you are taking this shortcut. >>=20 >=20 >=20 > If the page num is bigger than 512, flush all tlbs of this vm to avoid > soft lock-ups on large TLB flushing ranges. Just like what the > flush_tlb_range() does. I'm not sure this is applicable here, and it doesn't mean this is as good on other systems. Thanks, M. --=20 Jazz is not dead. It just smells funny...