From: "Huang, Ying" <ying.huang@intel.com>
To: Hesham Almatary <hesham.almatary@huawei.com>
Cc: haoxin <xhao@linux.alibaba.com>, <linux-kernel@vger.kernel.org>,
Andrew Morton <akpm@linux-foundation.org>,
Zi Yan <ziy@nvidia.com>, Yang Shi <shy828301@gmail.com>,
Baolin Wang <baolin.wang@linux.alibaba.com>,
Oscar Salvador <osalvador@suse.de>,
"Matthew Wilcox" <willy@infradead.org>,
<yangyicong@hisilicon.com>, <v-songbaohua@oppo.com>,
<21cnbao@gmail.com>, <linux-mm@kvack.org>
Subject: Re: [RFC 0/6] migrate_pages(): batch TLB flushing
Date: Wed, 02 Nov 2022 11:14:41 +0800 [thread overview]
Message-ID: <87h6ziavz2.fsf@yhuang6-desk2.ccr.corp.intel.com> (raw)
In-Reply-To: <520d44e0-b7a9-b841-047a-d2707f3df3fe@huawei.com> (Hesham Almatary's message of "Tue, 1 Nov 2022 14:49:17 +0000")
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=ascii, Size: 3105 bytes --]
Hesham Almatary <hesham.almatary@huawei.com> writes:
> On 9/27/2022 12:21 PM, haoxin wrote:
>> Hi, Huang
>>
>> ( 2022/9/21 \vH2:06, Huang Ying S:
>>> From: "Huang, Ying" <ying.huang@intel.com>
>>>
>>> Now, migrate_pages() migrate pages one by one, like the fake code as
>>> follows,
>>>
>>> for each page
>>> unmap
>>> flush TLB
>>> copy
>>> restore map
>>>
>>> If multiple pages are passed to migrate_pages(), there are
>>> opportunities to batch the TLB flushing and copying. That is, we can
>>> change the code to something as follows,
>>>
>>> for each page
>>> unmap
>>> for each page
>>> flush TLB
>>> for each page
>>> copy
>>> for each page
>>> restore map
>>>
>>> The total number of TLB flushing IPI can be reduced considerably. And
>>> we may use some hardware accelerator such as DSA to accelerate the
>>> page copying.
>>>
>>> So in this patch, we refactor the migrate_pages() implementation and
>>> implement the TLB flushing batching. Base on this, hardware
>>> accelerated page copying can be implemented.
>>>
>>> If too many pages are passed to migrate_pages(), in the naive batched
>>> implementation, we may unmap too many pages at the same time. The
>>> possibility for a task to wait for the migrated pages to be mapped
>>> again increases. So the latency may be hurt. To deal with this
>>> issue, the max number of pages be unmapped in batch is restricted to
>>> no more than HPAGE_PMD_NR. That is, the influence is at the same
>>> level of THP migration.
>>>
>>> We use the following test to measure the performance impact of the
>>> patchset,
>>>
>>> On a 2-socket Intel server,
>>>
>>> - Run pmbench memory accessing benchmark
>>>
>>> - Run `migratepages` to migrate pages of pmbench between node 0 and
>>> node 1 back and forth.
>>>
>> As the pmbench can not run on arm64 machine, so i use lmbench instead.
>> I test case like this: (i am not sure whether it is reasonable,
>> but it seems worked)
>> ./bw_mem -N10000 10000m rd &
>> time migratepages pid node0 node1
>>
> FYI, I have ported pmbench to AArch64 [1]. The project seems to be
> abandoned on bitbucket,
>
> I wonder if it makes sense to fork it elsewhere and push the pending PRs there.
>
>
> [1] https://bitbucket.org/jisooy/pmbench/pull-requests/5
Maybe try to contact the original author with email firstly?
Best Regards,
Huang, Ying
>> o/patch w/patch
>> real 0m0.035s real 0m0.024s
>> user 0m0.000s user 0m0.000s
>> sys 0m0.035s sys 0m0.024s
>>
>> the migratepages time is reduced above 32%.
>>
>> But there has a problem, i see the batch flush is called by
>> migrate_pages_batch
>> try_to_unmap_flush
>> arch_tlbbatch_flush(&tlb_ubc->arch); // there batch flush really work.
>>
>> But in arm64, the arch_tlbbatch_flush are not supported, becasue it
>> not support CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH yet.
>>
>> So, the tlb batch flush means no any flush is did, it is a empty func.
>>
>> Maybe this patch can help solve this problem.
>> https://lore.kernel.org/linux-arm-kernel/20220921084302.43631-1-yangyicong@huawei.com/T/
>>
>>
>>
>>
>>
>>
next prev parent reply other threads:[~2022-11-02 3:15 UTC|newest]
Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-21 6:06 [RFC 0/6] migrate_pages(): batch TLB flushing Huang Ying
2022-09-21 6:06 ` [RFC 1/6] mm/migrate_pages: separate huge page and normal pages migration Huang Ying
2022-09-21 15:55 ` Zi Yan
2022-09-22 1:14 ` Huang, Ying
2022-09-22 6:03 ` Baolin Wang
2022-09-22 6:22 ` Huang, Ying
2022-09-21 6:06 ` [RFC 2/6] mm/migrate_pages: split unmap_and_move() to _unmap() and _move() Huang Ying
2022-09-21 16:08 ` Zi Yan
2022-09-22 1:15 ` Huang, Ying
2022-09-22 6:36 ` Baolin Wang
2022-09-26 9:28 ` Alistair Popple
2022-09-26 18:06 ` Yang Shi
2022-09-27 0:02 ` Alistair Popple
2022-09-27 1:51 ` Huang, Ying
2022-09-27 20:34 ` John Hubbard
2022-09-27 20:57 ` Yang Shi
2022-09-28 0:59 ` Alistair Popple
2022-09-28 1:41 ` Huang, Ying
2022-09-28 1:44 ` John Hubbard
2022-09-28 1:49 ` Yang Shi
2022-09-28 1:56 ` John Hubbard
2022-09-28 2:14 ` Yang Shi
2022-09-28 2:57 ` John Hubbard
2022-09-28 3:25 ` Yang Shi
2022-09-28 3:39 ` Yang Shi
2022-09-27 20:56 ` Yang Shi
2022-09-27 20:54 ` Yang Shi
2022-09-21 6:06 ` [RFC 3/6] mm/migrate_pages: restrict number of pages to migrate in batch Huang Ying
2022-09-21 16:10 ` Zi Yan
2022-09-21 16:15 ` Zi Yan
2022-09-22 1:15 ` Huang, Ying
2022-09-21 6:06 ` [RFC 4/6] mm/migrate_pages: batch _unmap and _move Huang Ying
2022-09-21 6:06 ` [RFC 5/6] mm/migrate_pages: share more code between " Huang Ying
2022-09-21 6:06 ` [RFC 6/6] mm/migrate_pages: batch flushing TLB Huang Ying
2022-09-21 15:47 ` [RFC 0/6] migrate_pages(): batch TLB flushing Zi Yan
2022-09-22 1:45 ` Huang, Ying
2022-09-22 3:47 ` haoxin
2022-09-22 4:36 ` Huang, Ying
2022-09-22 12:50 ` Bharata B Rao
2022-09-23 7:52 ` Huang, Ying
2022-09-27 10:46 ` Bharata B Rao
2022-09-28 1:46 ` Huang, Ying
2022-09-26 9:11 ` Alistair Popple
2022-09-27 11:21 ` haoxin
2022-09-28 2:01 ` Huang, Ying
2022-09-28 3:33 ` haoxin
2022-09-28 4:53 ` Huang, Ying
2022-11-01 14:49 ` Hesham Almatary
2022-11-02 3:14 ` Huang, Ying [this message]
2022-11-02 14:13 ` Hesham Almatary
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87h6ziavz2.fsf@yhuang6-desk2.ccr.corp.intel.com \
--to=ying.huang@intel.com \
--cc=21cnbao@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=baolin.wang@linux.alibaba.com \
--cc=hesham.almatary@huawei.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=osalvador@suse.de \
--cc=shy828301@gmail.com \
--cc=v-songbaohua@oppo.com \
--cc=willy@infradead.org \
--cc=xhao@linux.alibaba.com \
--cc=yangyicong@hisilicon.com \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).