From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 698F5C433FE for ; Tue, 1 Nov 2022 14:49:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC3546B0072; Tue, 1 Nov 2022 10:49:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C4C7F6B0073; Tue, 1 Nov 2022 10:49:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B14646B0074; Tue, 1 Nov 2022 10:49:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A3E226B0072 for ; Tue, 1 Nov 2022 10:49:25 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5CF64C0F37 for ; Tue, 1 Nov 2022 14:49:25 +0000 (UTC) X-FDA: 80085156690.11.E9F97FF Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by imf20.hostedemail.com (Postfix) with ESMTP id BD5B11C0018 for ; Tue, 1 Nov 2022 14:49:22 +0000 (UTC) Received: from frapeml500005.china.huawei.com (unknown [172.18.147.200]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4N1tCR2Rgzz67bjw; Tue, 1 Nov 2022 22:47:15 +0800 (CST) Received: from lhrpeml500004.china.huawei.com (7.191.163.9) by frapeml500005.china.huawei.com (7.182.85.13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Tue, 1 Nov 2022 15:49:18 +0100 Received: from [10.202.227.118] (10.202.227.118) by lhrpeml500004.china.huawei.com (7.191.163.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Tue, 1 Nov 2022 14:49:17 +0000 Message-ID: <520d44e0-b7a9-b841-047a-d2707f3df3fe@huawei.com> Date: Tue, 1 Nov 2022 14:49:17 +0000 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.13.0 Subject: Re: [RFC 0/6] migrate_pages(): batch TLB flushing To: haoxin , Huang Ying CC: , Andrew Morton , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , "Matthew Wilcox" , , , <21cnbao@gmail.com>, References: <20220921060616.73086-1-ying.huang@intel.com> <393d6318-aa38-01ed-6ad8-f9eac89bf0fc@linux.alibaba.com> From: Hesham Almatary In-Reply-To: <393d6318-aa38-01ed-6ad8-f9eac89bf0fc@linux.alibaba.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Originating-IP: [10.202.227.118] X-ClientProxiedBy: lhrpeml500003.china.huawei.com (7.191.162.67) To lhrpeml500004.china.huawei.com (7.191.163.9) X-CFilter-Loop: Reflected ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667314164; a=rsa-sha256; cv=none; b=5KyuYgUFFxsdWMPSNdRevq/9BZO4hOOcDiLYiTLjksDBNo5ntr0RNlnVD7eLJxS7GfwHAQ 2rcD+SIsyzlJ5ZemjvyDSpsT0aR+K7weAFV73nOFAu4tiKHTOPBdOEL6yKfLfTfNFpRyXq BFFQ+G7EpRBwkIzsJwy/jsPD8GzRI8Y= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf20.hostedemail.com: domain of hesham.almatary@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=hesham.almatary@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667314164; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ofLUOPmZFtEKwAVqZACU4zpSTR3bAo4XhprvWvp8jDg=; b=6IrVXVZBsFakCEcHOyLxUjDE06NBgRAid+LlMSe9uThVkVPa4jjRKT2ZX0feXyq+RiQ6NK dHQi8Z/wFqbykFIvUfs+NK3Dm1DPSTQGmSjnQ87OH68gaK1+3A7iJtmFiz0FjR5eMKFbBU 9Rb63Cp6yL83R6uLAD+/x7D4NAf13Hc= X-Rspamd-Server: rspam02 X-Rspam-User: Authentication-Results: imf20.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf20.hostedemail.com: domain of hesham.almatary@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=hesham.almatary@huawei.com X-Stat-Signature: xdm9n7nyjzxjz1j5uosiyo5jcjsa7zsp X-Rspamd-Queue-Id: BD5B11C0018 X-HE-Tag: 1667314162-356980 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 9/27/2022 12:21 PM, haoxin wrote: > Hi, Huang > > 在 2022/9/21 下午2:06, Huang Ying 写道: >> From: "Huang, Ying" >> >> Now, migrate_pages() migrate pages one by one, like the fake code as >> follows, >> >>    for each page >>      unmap >>      flush TLB >>      copy >>      restore map >> >> If multiple pages are passed to migrate_pages(), there are >> opportunities to batch the TLB flushing and copying.  That is, we can >> change the code to something as follows, >> >>    for each page >>      unmap >>    for each page >>      flush TLB >>    for each page >>      copy >>    for each page >>      restore map >> >> The total number of TLB flushing IPI can be reduced considerably.  And >> we may use some hardware accelerator such as DSA to accelerate the >> page copying. >> >> So in this patch, we refactor the migrate_pages() implementation and >> implement the TLB flushing batching.  Base on this, hardware >> accelerated page copying can be implemented. >> >> If too many pages are passed to migrate_pages(), in the naive batched >> implementation, we may unmap too many pages at the same time. The >> possibility for a task to wait for the migrated pages to be mapped >> again increases.  So the latency may be hurt.  To deal with this >> issue, the max number of pages be unmapped in batch is restricted to >> no more than HPAGE_PMD_NR.  That is, the influence is at the same >> level of THP migration. >> >> We use the following test to measure the performance impact of the >> patchset, >> >> On a 2-socket Intel server, >> >>   - Run pmbench memory accessing benchmark >> >>   - Run `migratepages` to migrate pages of pmbench between node 0 and >>     node 1 back and forth. >> > As the pmbench can not run on arm64 machine, so i use lmbench instead. > I test case like this:  (i am not sure whether it is reasonable, but it seems > worked) > ./bw_mem -N10000 10000m rd & > time migratepages pid node0 node1 > FYI, I have ported pmbench to AArch64 [1]. The project seems to be abandoned on bitbucket, I wonder if it makes sense to fork it elsewhere and push the pending PRs there. [1] https://bitbucket.org/jisooy/pmbench/pull-requests/5 > o/patch w/patch > real    0m0.035s      real    0m0.024s > user    0m0.000s      user    0m0.000s > sys    0m0.035s        sys    0m0.024s > > the migratepages time is reduced above 32%. > > But there has a problem, i see the batch flush is called by > migrate_pages_batch >     try_to_unmap_flush >         arch_tlbbatch_flush(&tlb_ubc->arch); // there batch flush really work. > > But in arm64, the arch_tlbbatch_flush are not supported, becasue it not > support CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH yet. > > So, the tlb batch flush means no any flush is did, it is a empty func. > > Maybe this patch can help solve this problem. > https://lore.kernel.org/linux-arm-kernel/20220921084302.43631-1-yangyicong@huawei.com/T/ > > > > > > >