From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 616A6C43334 for ; Thu, 14 Jul 2022 04:52:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B3C3C94017A; Thu, 14 Jul 2022 00:51:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AEBB4940134; Thu, 14 Jul 2022 00:51:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B44894017A; Thu, 14 Jul 2022 00:51:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 88218940134 for ; Thu, 14 Jul 2022 00:51:59 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id CF95B33EC0 for ; Thu, 14 Jul 2022 04:51:58 +0000 (UTC) X-FDA: 79684483116.10.BCDCD26 Received: from mail-ed1-f50.google.com (mail-ed1-f50.google.com [209.85.208.50]) by imf19.hostedemail.com (Postfix) with ESMTP id 6A6131A0071 for ; Thu, 14 Jul 2022 04:51:58 +0000 (UTC) Received: by mail-ed1-f50.google.com with SMTP id x91so945576ede.1 for ; Wed, 13 Jul 2022 21:51:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=hW7IbWPmYdn2d1NNnyMMYZo7sUWhfXMc3EeZopJnd9s=; b=ixpskF3es1ZmyzgNugOOlVGXEXU01wH8x+O+9c0Hn53VnayTkfg7B02WDNC5tIrO7l iEwg3ca5x42mE2uIM8nqQUrXVwXLpT6ULvvEDEhDoQovcbWbHLnoAwOv78ubyxnHhjrc b+F58wCwdRfZkdz4PEAGwhDlQaXN17a/dAqjH7lRr9IZR6Oxl15yDJPqx5DZqhd1LDca m3K9L9Ce8HJHQ3yTYC00bD8m8HCeZVO9kN8Rn4LXDKPCB1TSqsZxNf0zJQAMyjXyiJ7g cpQu1lJzObpQ/9Off5+y4JJmm1YPuTUAuigqbDyhvyv4ipfIih1IQ2OiwKFqjfHfMEIL 9Opw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=hW7IbWPmYdn2d1NNnyMMYZo7sUWhfXMc3EeZopJnd9s=; b=LU6wmo9+dcc9xKqUT83Bq4LECHOiVcx8nA9Fk7PxLJew2Ihjf6qBAjmUJHWkYr7COb ZqjfH63NoOdwJZf7zmFedAJ1pzkHw8Di9GT84pI1Ykq3cLm/+9N26FgItZyCM/50/T/9 XpiRvJ+gCe/4U4i+D0dhQzjAxRm4uocN6ljvDZXVE5ewtosDQxtdMOGSZ7mbncShLzZS sCeONgmsnYcZ+u7pZpRBR4/WDv/9OGby9AN8u+B9DedcFkCFdUbgwl0baOfi16xche39 bXR8Rk6j4GJybGrSifhf1wPPZiDJiwaTRSCqBG9wj9/TSZQqFvhKiKlxuON4cZpxh25m ZCsQ== X-Gm-Message-State: AJIora/+ktGB3u46T1GOIIEp8Y6OUuNIZTwa32ggylvys5auYCwMQ6v1 3o0WtyRmChLmC2jkYZr/M0pjqaQzYHkjqf7F8cE= X-Google-Smtp-Source: AGRyM1sCekI8nbaSkEz0wXp62jC4NmfsPr1d7cEeKiuuhPUbnQAKxU8DeX6rgQmQqaqEEOVW9HUjdoQGawmviSj4Nuo= X-Received: by 2002:aa7:db9a:0:b0:43a:76bf:5401 with SMTP id u26-20020aa7db9a000000b0043a76bf5401mr9638511edt.244.1657774317014; Wed, 13 Jul 2022 21:51:57 -0700 (PDT) MIME-Version: 1.0 References: <20220711034615.482895-1-21cnbao@gmail.com> <24f5e25b-3946-b92a-975b-c34688005398@linux.alibaba.com> In-Reply-To: <24f5e25b-3946-b92a-975b-c34688005398@linux.alibaba.com> From: Barry Song <21cnbao@gmail.com> Date: Thu, 14 Jul 2022 16:51:45 +1200 Message-ID: Subject: Re: [PATCH v2 0/4] mm: arm64: bring up BATCHED_UNMAP_TLB_FLUSH To: xhao@linux.alibaba.com Cc: Andrew Morton , Linux-MM , LAK , x86 , Catalin Marinas , Will Deacon , Linux Doc Mailing List , Jonathan Corbet , Arnd Bergmann , LKML , Darren Hart , Yicong Yang , huzhanyuan@oppo.com, =?UTF-8?B?5p2O5Z+56ZSLKHdpbmsp?= , =?UTF-8?B?5byg6K+X5piOKFNpbW9uIFpoYW5nKQ==?= , =?UTF-8?B?6YOt5YGl?= , real mz , linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org Content-Type: text/plain; charset="UTF-8" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657774318; a=rsa-sha256; cv=none; b=h+m+KKMCk3zD7qNLZBLLNECCt6JL7NFAr+geYr5VI/HA8KwyvGYWSYdABGaieUIGinVSF2 itL/PKBVnPMCHmCy5Vxly/Zv56Ej8QiusynikK0YdwOG3+O5u3vivYI5nCjcmCfjvuODik bO0PiSjhAtbeYxI3oXYOIk3qkUhsdKg= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ixpskF3e; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.208.50 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657774318; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hW7IbWPmYdn2d1NNnyMMYZo7sUWhfXMc3EeZopJnd9s=; b=QqTf33s/cTUdNSFkAYwI7ghcg4OtwuihdDVVNjNLCnbvw4duajytCBg6iHxmTk6YoCd6ZB wf8zaaNG75s+iggLENNT6UBTplkf198aMEH2te2S3bRayX5Bu4hKOMWZMv6ijMWhUWb9Dy U3k7FQ1E9NNeYNQlVG/j12xHW6l3yAw= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 6A6131A0071 Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ixpskF3e; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.208.50 as permitted sender) smtp.mailfrom=21cnbao@gmail.com X-Stat-Signature: aw8rmu8k46oo41rxam41r8r88g4sxuir X-Rspam-User: X-HE-Tag: 1657774318-883046 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Jul 14, 2022 at 3:29 PM Xin Hao wrote: > > Hi barry. > > I do some test on Kunpeng arm64 machine use Unixbench. > > The test result as below. > > One core, we can see the performance improvement above +30%. I am really pleased to see the 30%+ improvement on unixbench on single core. > ./Run -c 1 -i 1 shell1 > w/o > System Benchmarks Partial Index BASELINE RESULT INDEX > Shell Scripts (1 concurrent) 42.4 5481.0 1292.7 > ======== > System Benchmarks Index Score (Partial Only) 1292.7 > > w/ > System Benchmarks Partial Index BASELINE RESULT INDEX > Shell Scripts (1 concurrent) 42.4 6974.6 1645.0 > ======== > System Benchmarks Index Score (Partial Only) 1645.0 > > > But with whole cores, there have little performance degradation above -5% That is sad as we might get more concurrency between mprotect(), madvise(), mremap(), zap_pte_range() and the deferred tlbi. > > ./Run -c 96 -i 1 shell1 > w/o > Shell Scripts (1 concurrent) 80765.5 lpm (60.0 s, 1 > samples) > System Benchmarks Partial Index BASELINE RESULT INDEX > Shell Scripts (1 concurrent) 42.4 80765.5 19048.5 > ======== > System Benchmarks Index Score (Partial Only) 19048.5 > > w > Shell Scripts (1 concurrent) 76333.6 lpm (60.0 s, 1 > samples) > System Benchmarks Partial Index BASELINE RESULT INDEX > Shell Scripts (1 concurrent) 42.4 76333.6 18003.2 > ======== > System Benchmarks Index Score (Partial Only) 18003.2 > > ---------------------------------------------------------------------------------------------- > > > After discuss with you, and do some changes in the patch. > > ndex a52381a680db..1ecba81f1277 100644 > --- a/mm/rmap.c > +++ b/mm/rmap.c > @@ -727,7 +727,11 @@ void flush_tlb_batched_pending(struct mm_struct *mm) > int flushed = batch >> TLB_FLUSH_BATCH_FLUSHED_SHIFT; > > if (pending != flushed) { > +#ifdef CONFIG_ARCH_HAS_MM_CPUMASK > flush_tlb_mm(mm); > +#else > + dsb(ish); > +#endif > i was guessing the problem might be flush_tlb_batched_pending() so i asked you to change this to verify my guess. /* > * If the new TLB flushing is pending during flushing, leave > * mm->tlb_flush_batched as is, to avoid losing flushing. > > there have a performance improvement with whole cores, above +30% But I don't think it is a proper patch. There is no guarantee the cpu calling flush_tlb_batched_pending is exactly the cpu sending the deferred tlbi. so the solution is unsafe. But since this temporary code can bring the 30%+ performance improvement back for high concurrency, we have huge potential to finally make it. Unfortunately I don't have an arm64 server to debug on this. I only have 8 cores which are unlikely to reproduce regression which happens in high concurrency with 96 parallel tasks. So I'd ask if @yicong or someone else working on kunpeng or other arm64 servers is able to actually debug and figure out a proper patch for this, then add the patch as 5/5 into this series? > > ./Run -c 96 -i 1 shell1 > 96 CPUs in system; running 96 parallel copies of tests > > Shell Scripts (1 concurrent) 109229.0 lpm (60.0 s, 1 samples) > System Benchmarks Partial Index BASELINE RESULT INDEX > Shell Scripts (1 concurrent) 42.4 109229.0 25761.6 > ======== > System Benchmarks Index Score (Partial Only) 25761.6 > > > Tested-by: Xin Hao Thanks for your testing! > > Looking forward to your next version patch. > > On 7/11/22 11:46 AM, Barry Song wrote: > > Though ARM64 has the hardware to do tlb shootdown, the hardware > > broadcasting is not free. > > A simplest micro benchmark shows even on snapdragon 888 with only > > 8 cores, the overhead for ptep_clear_flush is huge even for paging > > out one page mapped by only one process: > > 5.36% a.out [kernel.kallsyms] [k] ptep_clear_flush > > > > While pages are mapped by multiple processes or HW has more CPUs, > > the cost should become even higher due to the bad scalability of > > tlb shootdown. > > > > The same benchmark can result in 16.99% CPU consumption on ARM64 > > server with around 100 cores according to Yicong's test on patch > > 4/4. > > > > This patchset leverages the existing BATCHED_UNMAP_TLB_FLUSH by > > 1. only send tlbi instructions in the first stage - > > arch_tlbbatch_add_mm() > > 2. wait for the completion of tlbi by dsb while doing tlbbatch > > sync in arch_tlbbatch_flush() > > My testing on snapdragon shows the overhead of ptep_clear_flush > > is removed by the patchset. The micro benchmark becomes 5% faster > > even for one page mapped by single process on snapdragon 888. > > > > > > -v2: > > 1. Collected Yicong's test result on kunpeng920 ARM64 server; > > 2. Removed the redundant vma parameter in arch_tlbbatch_add_mm() > > according to the comments of Peter Zijlstra and Dave Hansen > > 3. Added ARCH_HAS_MM_CPUMASK rather than checking if mm_cpumask > > is empty according to the comments of Nadav Amit > > > > Thanks, Yicong, Peter, Dave and Nadav for your testing or reviewing > > , and comments. > > > > -v1: > > https://lore.kernel.org/lkml/20220707125242.425242-1-21cnbao@gmail.com/ > > > > Barry Song (4): > > Revert "Documentation/features: mark BATCHED_UNMAP_TLB_FLUSH doesn't > > apply to ARM64" > > mm: rmap: Allow platforms without mm_cpumask to defer TLB flush > > mm: rmap: Extend tlbbatch APIs to fit new platforms > > arm64: support batched/deferred tlb shootdown during page reclamation > > > > Documentation/features/arch-support.txt | 1 - > > .../features/vm/TLB/arch-support.txt | 2 +- > > arch/arm/Kconfig | 1 + > > arch/arm64/Kconfig | 1 + > > arch/arm64/include/asm/tlbbatch.h | 12 ++++++++++ > > arch/arm64/include/asm/tlbflush.h | 23 +++++++++++++++++-- > > arch/loongarch/Kconfig | 1 + > > arch/mips/Kconfig | 1 + > > arch/openrisc/Kconfig | 1 + > > arch/powerpc/Kconfig | 1 + > > arch/riscv/Kconfig | 1 + > > arch/s390/Kconfig | 1 + > > arch/um/Kconfig | 1 + > > arch/x86/Kconfig | 1 + > > arch/x86/include/asm/tlbflush.h | 3 ++- > > mm/Kconfig | 3 +++ > > mm/rmap.c | 14 +++++++---- > > 17 files changed, 59 insertions(+), 9 deletions(-) > > create mode 100644 arch/arm64/include/asm/tlbbatch.h > > > -- > Best Regards! > Xin Hao > Thanks Barry