From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21336CCA483 for ; Sat, 23 Jul 2022 09:22:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A27096B0071; Sat, 23 Jul 2022 05:22:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D5ED6B0073; Sat, 23 Jul 2022 05:22:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 89E7C6B0074; Sat, 23 Jul 2022 05:22:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7DE546B0071 for ; Sat, 23 Jul 2022 05:22:56 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 5C87F140DA2 for ; Sat, 23 Jul 2022 09:22:56 +0000 (UTC) X-FDA: 79717825152.26.1F784AB Received: from out30-44.freemail.mail.aliyun.com (out30-44.freemail.mail.aliyun.com [115.124.30.44]) by imf02.hostedemail.com (Postfix) with ESMTP id 5361F80063 for ; Sat, 23 Jul 2022 09:22:54 +0000 (UTC) X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046059;MF=xhao@linux.alibaba.com;NM=1;PH=DS;RN=25;SR=0;TI=SMTPD_---0VK94Ta0_1658568166; Received: from B-X3VXMD6M-2058.local(mailfrom:xhao@linux.alibaba.com fp:SMTPD_---0VK94Ta0_1658568166) by smtp.aliyun-inc.com; Sat, 23 Jul 2022 17:22:48 +0800 From: xhao@linux.alibaba.com Reply-To: xhao@linux.alibaba.com Subject: Re: [PATCH v2 0/4] mm: arm64: bring up BATCHED_UNMAP_TLB_FLUSH To: Barry Song <21cnbao@gmail.com>, Yicong Yang Cc: Andrew Morton , Linux-MM , LAK , x86 , Catalin Marinas , Will Deacon , Linux Doc Mailing List , Jonathan Corbet , Arnd Bergmann , LKML , Darren Hart , huzhanyuan@oppo.com, =?UTF-8?B?5p2O5Z+56ZSLKHdpbmsp?= , =?UTF-8?B?5byg6K+X5piOKFNpbW9uIFpoYW5nKQ==?= , =?UTF-8?B?6YOt5YGl?= , real mz , linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, Yicong Yang , "tiantao (H)" References: <20220711034615.482895-1-21cnbao@gmail.com> <24f5e25b-3946-b92a-975b-c34688005398@linux.alibaba.com> <8e603deb-7023-5de5-c958-8911971aec24@huawei.com> Message-ID: <3ac4b1a3-8067-3edb-be4f-326e2a4943ed@linux.alibaba.com> Date: Sat, 23 Jul 2022 17:22:45 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658568176; a=rsa-sha256; cv=none; b=Cq7FNCVyfa25mXWL1c8RLx1VAJTL33PRZeb1BINyobKFmnWcTuFSm89xj+1+Fa6HO4jTMA +aXYoHUztL84K0fymNafqS6RyYKexTQSNR56h0ekb+DQqBsMXo3Lp7hasMMpEqvbLlZJYr Ezsjd5ahn7ndGL2YK1ke6a32j1R8iDM= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf02.hostedemail.com: domain of xhao@linux.alibaba.com designates 115.124.30.44 as permitted sender) smtp.mailfrom=xhao@linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658568176; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A0uiFaHfxYJMfxD/XkIxHAn+ImW2CAbRhVQs6uAADXQ=; b=AvkYOQb/kHyvWJ3sidvClLr2tmzF1XZV6Hf5A5B5OuDhtC5XaE5Xk3crLncUJG63Y7b0AP R1Ah/t30LF9VWWIsSmWcgs28MtPhgIzcrfKsttgllDhoNy0MYw747Bb2bAOgLxG/ESfdBQ OmtZra72OnSn60SUNDz7EP+4adzlQP8= X-Rspamd-Queue-Id: 5361F80063 Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf02.hostedemail.com: domain of xhao@linux.alibaba.com designates 115.124.30.44 as permitted sender) smtp.mailfrom=xhao@linux.alibaba.com X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: 3du1s51izy56f9hnfo4ufkjuiki3dd9b X-HE-Tag: 1658568174-84408 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 7/20/22 7:18 PM, Barry Song wrote: > On Tue, Jul 19, 2022 at 1:28 AM Yicong Yang wrote: >> On 2022/7/14 12:51, Barry Song wrote: >>> On Thu, Jul 14, 2022 at 3:29 PM Xin Hao wrote: >>>> Hi barry. >>>> >>>> I do some test on Kunpeng arm64 machine use Unixbench. >>>> >>>> The test result as below. >>>> >>>> One core, we can see the performance improvement above +30%. >>> I am really pleased to see the 30%+ improvement on unixbench on single core. >>> >>>> ./Run -c 1 -i 1 shell1 >>>> w/o >>>> System Benchmarks Partial Index BASELINE RESULT INDEX >>>> Shell Scripts (1 concurrent) 42.4 5481.0 1292.7 >>>> ======== >>>> System Benchmarks Index Score (Partial Only) 1292.7 >>>> >>>> w/ >>>> System Benchmarks Partial Index BASELINE RESULT INDEX >>>> Shell Scripts (1 concurrent) 42.4 6974.6 1645.0 >>>> ======== >>>> System Benchmarks Index Score (Partial Only) 1645.0 >>>> >>>> >>>> But with whole cores, there have little performance degradation above -5% >>> That is sad as we might get more concurrency between mprotect(), madvise(), >>> mremap(), zap_pte_range() and the deferred tlbi. >>> >>>> ./Run -c 96 -i 1 shell1 >>>> w/o >>>> Shell Scripts (1 concurrent) 80765.5 lpm (60.0 s, 1 >>>> samples) >>>> System Benchmarks Partial Index BASELINE RESULT INDEX >>>> Shell Scripts (1 concurrent) 42.4 80765.5 19048.5 >>>> ======== >>>> System Benchmarks Index Score (Partial Only) 19048.5 >>>> >>>> w >>>> Shell Scripts (1 concurrent) 76333.6 lpm (60.0 s, 1 >>>> samples) >>>> System Benchmarks Partial Index BASELINE RESULT INDEX >>>> Shell Scripts (1 concurrent) 42.4 76333.6 18003.2 >>>> ======== >>>> System Benchmarks Index Score (Partial Only) 18003.2 >>>> >>>> ---------------------------------------------------------------------------------------------- >>>> >>>> >>>> After discuss with you, and do some changes in the patch. >>>> >>>> ndex a52381a680db..1ecba81f1277 100644 >>>> --- a/mm/rmap.c >>>> +++ b/mm/rmap.c >>>> @@ -727,7 +727,11 @@ void flush_tlb_batched_pending(struct mm_struct *mm) >>>> int flushed = batch >> TLB_FLUSH_BATCH_FLUSHED_SHIFT; >>>> >>>> if (pending != flushed) { >>>> +#ifdef CONFIG_ARCH_HAS_MM_CPUMASK >>>> flush_tlb_mm(mm); >>>> +#else >>>> + dsb(ish); >>>> +#endif >>>> >>> i was guessing the problem might be flush_tlb_batched_pending() >>> so i asked you to change this to verify my guess. >>> >> flush_tlb_batched_pending() looks like the critical path for this issue then the code >> above can mitigate this. >> >> I cannot reproduce this on a 2P 128C Kunpeng920 server. The kernel is based on the >> v5.19-rc6 and unixbench of version 5.1.3. The result of `./Run -c 128 -i 1 shell1` is: >> iter-1 iter-2 iter-3 >> w/o 17708.1 17637.1 17630.1 >> w 17766.0 17752.3 17861.7 >> >> And flush_tlb_batched_pending()isn't the hot spot with the patch: >> 7.00% sh [kernel.kallsyms] [k] ptep_clear_flush >> 4.17% sh [kernel.kallsyms] [k] ptep_set_access_flags >> 2.43% multi.sh [kernel.kallsyms] [k] ptep_clear_flush >> 1.98% sh [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore >> 1.69% sh [kernel.kallsyms] [k] next_uptodate_page >> 1.66% sort [kernel.kallsyms] [k] ptep_clear_flush >> 1.56% multi.sh [kernel.kallsyms] [k] ptep_set_access_flags >> 1.27% sh [kernel.kallsyms] [k] page_counter_cancel >> 1.11% sh [kernel.kallsyms] [k] page_remove_rmap >> 1.06% sh [kernel.kallsyms] [k] perf_event_alloc >> >> Hi Xin Hao, >> >> I'm not sure the test setup as well as the config is same with yours. (96C vs 128C >> should not be the reason I think). Did you check that the 5% is a fluctuation or >> not? It'll be helpful if more information provided for reproducing this issue. >> >> Thanks. > I guess that is because "./Run -c 1 -i 1 shell1" isn't an application > stressed on > memory. Hi Xin, in what kinds of configurations can we reproduce your test > result? Oh, my fault, I do the test is not based on the lastest upstream kernel, there maybe some impact here, i will do a new test on the lastest kernel. > As I suppose tlbbatch will mainly affect the performance of user scenarios > which require memory page-out/page-in like reclaiming file/anon pages. > "./Run -c 1 -i 1 shell1" on a system with sufficient free memory won't be > affected by tlbbatch at all, I believe. > > Thanks > Barry -- Best Regards! Xin Hao