From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADEE2C43217 for ; Wed, 2 Nov 2022 03:15:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED9816B0071; Tue, 1 Nov 2022 23:15:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E89DB6B0072; Tue, 1 Nov 2022 23:15:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D51B96B0073; Tue, 1 Nov 2022 23:15:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C27B56B0071 for ; Tue, 1 Nov 2022 23:15:31 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 97DC380206 for ; Wed, 2 Nov 2022 03:15:31 +0000 (UTC) X-FDA: 80087036862.10.C7D7401 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by imf28.hostedemail.com (Postfix) with ESMTP id 9E1FAC0003 for ; Wed, 2 Nov 2022 03:15:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1667358930; x=1698894930; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version:content-transfer-encoding; bh=m48/xeNHXwikdkMS7hl7niHQHFI4b3Rpl1Ik4bOU1Mg=; b=hd6y8eDpIBcdA9eSwy7ZqTZzdfATRX3PxF/PGxgwQj6nbOOkyCA1AQF4 yV+zFj4NQ/2hpjVfBrXq8O7qm0SkAGksFZGvSP0SiPbyMXBQJf6bV0f6q OBWMrOw7ZYQAov/gUypLnWCK0xTo8hBx4CUb9OoP1UAVCEQVPKlQvFjZe 58n16WgiC35XTdRdcYYJ73UuiU5QxapzCeyX7Uym7yDWVYo8XbYnHxyr7 kH962Hi/KDRMkUpmeBCuzG9pkdRAAeJsALLF7I/+YE+3AmmQZ6SBRaj06 XitmygzEhgQTWVVBxLSkragm4tWz0a2HgwrZRzQ0wEp0Fh/fGA9d8LjWr A==; X-IronPort-AV: E=McAfee;i="6500,9779,10518"; a="310405285" X-IronPort-AV: E=Sophos;i="5.95,232,1661842800"; d="scan'208";a="310405285" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2022 20:15:28 -0700 X-IronPort-AV: E=McAfee;i="6500,9779,10518"; a="703106234" X-IronPort-AV: E=Sophos;i="5.95,232,1661842800"; d="scan'208";a="703106234" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Nov 2022 20:15:25 -0700 From: "Huang, Ying" To: Hesham Almatary Cc: haoxin , , Andrew Morton , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , "Matthew Wilcox" , , , <21cnbao@gmail.com>, Subject: Re: [RFC 0/6] migrate_pages(): batch TLB flushing References: <20220921060616.73086-1-ying.huang@intel.com> <393d6318-aa38-01ed-6ad8-f9eac89bf0fc@linux.alibaba.com> <520d44e0-b7a9-b841-047a-d2707f3df3fe@huawei.com> Date: Wed, 02 Nov 2022 11:14:41 +0800 In-Reply-To: <520d44e0-b7a9-b841-047a-d2707f3df3fe@huawei.com> (Hesham Almatary's message of "Tue, 1 Nov 2022 14:49:17 +0000") Message-ID: <87h6ziavz2.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii Content-Transfer-Encoding: 8bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667358931; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2/xAUnRlDUGDdd0egyC/0KDhAvRAlAjHaxlLaZz75sc=; b=NYsjHdTIjIyTK1MDduX/cnW1xDnCWHHEpsIo1diZceVceQileEZSVHmZZV0iv8YMfEEsUX v0u28TmhZW5yi5IxECdIwE7cgHhh61iLG42FcAuxDbDQq9gZFFr0vFoNZfZE9r7Il6RCek vYlpFvTPPZPvJ77A8GEwa/kVe2fVC5Y= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=hd6y8eDp; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf28.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667358931; a=rsa-sha256; cv=none; b=8Ry1uJGYbrLR3OZ4IijGuuuu4DxRm2oeI+k/ubABGXfkmaYHJMyzubxVk3R3C1t2wspYff ZP10/O0wg/xofSqD2sfQRfDxwiKf6p6wZyBKBXLxoPPj2kHUoHO8cfn2vkVBpigbM1bgrt IRaNcmrVtpUogwP1Wk1MC+auKSRX7Y8= Authentication-Results: imf28.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=hd6y8eDp; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf28.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=ying.huang@intel.com X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 9E1FAC0003 X-Stat-Signature: bben9nb1gjwk6mp9wcs14y8xtrj93ab1 X-HE-Tag: 1667358930-101151 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hesham Almatary writes: > On 9/27/2022 12:21 PM, haoxin wrote: >> Hi, Huang >> >> ( 2022/9/21 H2:06, Huang Ying S: >>> From: "Huang, Ying" >>> >>> Now, migrate_pages() migrate pages one by one, like the fake code as >>> follows, >>> >>>   for each page >>>    unmap >>>    flush TLB >>>    copy >>>    restore map >>> >>> If multiple pages are passed to migrate_pages(), there are >>> opportunities to batch the TLB flushing and copying. That is, we can >>> change the code to something as follows, >>> >>>   for each page >>>    unmap >>>   for each page >>>    flush TLB >>>   for each page >>>    copy >>>   for each page >>>    restore map >>> >>> The total number of TLB flushing IPI can be reduced considerably. And >>> we may use some hardware accelerator such as DSA to accelerate the >>> page copying. >>> >>> So in this patch, we refactor the migrate_pages() implementation and >>> implement the TLB flushing batching. Base on this, hardware >>> accelerated page copying can be implemented. >>> >>> If too many pages are passed to migrate_pages(), in the naive batched >>> implementation, we may unmap too many pages at the same time. The >>> possibility for a task to wait for the migrated pages to be mapped >>> again increases. So the latency may be hurt. To deal with this >>> issue, the max number of pages be unmapped in batch is restricted to >>> no more than HPAGE_PMD_NR. That is, the influence is at the same >>> level of THP migration. >>> >>> We use the following test to measure the performance impact of the >>> patchset, >>> >>> On a 2-socket Intel server, >>> >>> - Run pmbench memory accessing benchmark >>> >>> - Run `migratepages` to migrate pages of pmbench between node 0 and >>>   node 1 back and forth. >>> >> As the pmbench can not run on arm64 machine, so i use lmbench instead. >> I test case like this: (i am not sure whether it is reasonable, >> but it seems worked) >> ./bw_mem -N10000 10000m rd & >> time migratepages pid node0 node1 >> > FYI, I have ported pmbench to AArch64 [1]. The project seems to be > abandoned on bitbucket, > > I wonder if it makes sense to fork it elsewhere and push the pending PRs there. > > > [1] https://bitbucket.org/jisooy/pmbench/pull-requests/5 Maybe try to contact the original author with email firstly? Best Regards, Huang, Ying >> o/patch w/patch >> real  0m0.035s   real  0m0.024s >> user  0m0.000s   user  0m0.000s >> sys  0m0.035s    sys  0m0.024s >> >> the migratepages time is reduced above 32%. >> >> But there has a problem, i see the batch flush is called by >> migrate_pages_batch >>   try_to_unmap_flush >>     arch_tlbbatch_flush(&tlb_ubc->arch); // there batch flush really work. >> >> But in arm64, the arch_tlbbatch_flush are not supported, becasue it >> not support CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH yet. >> >> So, the tlb batch flush means no any flush is did, it is a empty func. >> >> Maybe this patch can help solve this problem. >> https://lore.kernel.org/linux-arm-kernel/20220921084302.43631-1-yangyicong@huawei.com/T/ >> >> >> >> >> >>