From: Kalesh Singh <kaleshsingh@google.com>
To: unlisted-recipients:; (no To-header on input)
Cc: surenb@google.com, minchan@google.com, joelaf@google.com,
lokeshgidra@google.com, kaleshsingh@google.com,
kernel-team@android.com,
Catalin Marinas <catalin.marinas@arm.com>,
Will Deacon <will@kernel.org>,
Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
x86@kernel.org, "H. Peter Anvin" <hpa@zytor.com>,
Andrew Morton <akpm@linux-foundation.org>,
Shuah Khan <shuah@kernel.org>, Kees Cook <keescook@chromium.org>,
"Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
Peter Zijlstra <peterz@infradead.org>,
Masahiro Yamada <masahiroy@kernel.org>,
Arnd Bergmann <arnd@arndb.de>,
Sami Tolvanen <samitolvanen@google.com>,
Frederic Weisbecker <frederic@kernel.org>,
Krzysztof Kozlowski <krzk@kernel.org>,
Hassan Naveed <hnaveed@wavecomp.com>,
Christian Brauner <christian.brauner@ubuntu.com>,
Mark Rutland <mark.rutland@arm.com>,
Mike Rapoport <rppt@kernel.org>, Gavin Shan <gshan@redhat.com>,
Dave Martin <Dave.Martin@arm.com>,
Mark Brown <broonie@kernel.org>, Jia He <justin.he@arm.com>,
Zhenyu Ye <yezhenyu2@huawei.com>,
John Hubbard <jhubbard@nvidia.com>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Zi Yan <ziy@nvidia.com>, Ram Pai <linuxram@us.ibm.com>,
Ralph Campbell <rcampbell@nvidia.com>,
Mina Almasry <almasrymina@google.com>,
"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>,
Sandipan Das <sandipan@linux.ibm.com>,
Dave Hansen <dave.hansen@intel.com>,
Jason Gunthorpe <jgg@ziepe.ca>,
Masami Hiramatsu <mhiramat@kernel.org>,
Brian Geffon <bgeffon@google.com>,
Kamalesh Babulal <kamalesh@linux.vnet.ibm.com>,
SeongJae Park <sjpark@amazon.de>,
linux-kernel@vger.kernel.org,
linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org,
linux-kselftest@vger.kernel.org
Subject: [PATCH v3 4/5] arm64: mremap speedup - Enable HAVE_MOVE_PUD
Date: Mon, 5 Oct 2020 15:40:07 +0000 [thread overview]
Message-ID: <20201005154017.474722-5-kaleshsingh@google.com> (raw)
In-Reply-To: <20201005154017.474722-1-kaleshsingh@google.com>
HAVE_MOVE_PUD enables remapping pages at the PUD level if both the
source and destination addresses are PUD-aligned.
With HAVE_MOVE_PUD enabled it can be inferred that there is approximately
a 19x improvement in performance on arm64. (See data below).
------- Test Results ---------
The following results were obtained using a 5.4 kernel, by remapping
a PUD-aligned, 1GB sized region to a PUD-aligned destination.
The results from 10 iterations of the test are given below:
Total mremap times for 1GB data on arm64. All times are in nanoseconds.
Control HAVE_MOVE_PUD
1247761 74271
1219896 46771
1094792 59687
1227760 48385
1043698 76666
1101771 50365
1159896 52500
1143594 75261
1025833 61354
1078125 48697
1134312.6 59395.7 <-- Mean time in nanoseconds
A 1GB mremap completion time drops from ~1.1 milliseconds
to ~59 microseconds on arm64. (~19x speed up).
Signed-off-by: Kalesh Singh <kaleshsingh@google.com>
---
Change in v3:
- Add set_pud_at() macro - Used by move_normal_pud().
arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/pgtable.h | 1 +
2 files changed, 2 insertions(+)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 844d089668e3..4d521f0a5863 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -122,6 +122,7 @@ config ARM64
select HANDLE_DOMAIN_IRQ
select HARDIRQS_SW_RESEND
select HAVE_MOVE_PMD
+ select HAVE_MOVE_PUD
select HAVE_PCI
select HAVE_ACPI_APEI if (ACPI && EFI)
select HAVE_ALIGNED_STRUCT_PAGE if SLUB
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index d5d3fbe73953..8848125e3024 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -415,6 +415,7 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd)
#define pfn_pud(pfn,prot) __pud(__phys_to_pud_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot))
#define set_pmd_at(mm, addr, pmdp, pmd) set_pte_at(mm, addr, (pte_t *)pmdp, pmd_pte(pmd))
+#define set_pud_at(mm, addr, pudp, pud) set_pte_at(mm, addr, (pte_t *)pudp, pud_pte(pud))
#define __p4d_to_phys(p4d) __pte_to_phys(p4d_pte(p4d))
#define __phys_to_p4d_val(phys) __phys_to_pte_val(phys)
--
2.28.0.806.g8561365e88-goog
WARNING: multiple messages have this Message-ID (diff)
From: Kalesh Singh <kaleshsingh@google.com>
Cc: surenb@google.com, minchan@google.com, joelaf@google.com,
lokeshgidra@google.com, kaleshsingh@google.com,
kernel-team@android.com,
Catalin Marinas <catalin.marinas@arm.com>,
Will Deacon <will@kernel.org>,
Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
x86@kernel.org, "H. Peter Anvin" <hpa@zytor.com>,
Andrew Morton <akpm@linux-foundation.org>,
Shuah Khan <shuah@kernel.org>, Kees Cook <keescook@chromium.org>,
"Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
Peter Zijlstra <peterz@infradead.org>,
Masahiro Yamada <masahiroy@kernel.org>,
Arnd Bergmann <arnd@arndb.de>,
Sami Tolvanen <samitolvanen@google.com>,
Frederic Weisbecker <frederic@kernel.org>,
Krzysztof Kozlowski <krzk@kernel.org>,
Hassan Naveed <hnaveed@wavecomp.com>,
Christian Brauner <christian.brauner@ubuntu.com>,
Mark Rutland <mark.rutland@arm.com>,
Mike Rapoport <rppt@kernel.org>, Gavin Shan <gshan@redhat.com>,
Dave Martin <Dave.Martin@arm.com>,
Mark Brown <broonie@kernel.org>, Jia He <justin.he@arm.com>,
Zhenyu Ye <yezhenyu2@huawei.com>,
John Hubbard <jhubbard@nvidia.com>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Zi Yan <ziy@nvidia.com>, Ram Pai <linuxram@us.ibm.com>,
Ralph Campbell <rcampbell@nvidia.com>,
Mina Almasry <almasrymina@google.com>,
"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>,
Sandipan Das <sandipan@linux.ibm.com>,
Dave Hansen <dave.hansen@intel.com>,
Jason Gunthorpe <jgg@ziepe.ca>,
Masami Hiramatsu <mhiramat@kernel.org>,
Brian Geffon <bgeffon@google.com>,
Kamalesh Babulal <kamalesh@linux.vnet.ibm.com>,
SeongJae Park <sjpark@amazon.de>,
linux-kernel@vger.kernel.org,
linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org,
linux-kselftest@vger.kernel.org
Subject: [PATCH v3 4/5] arm64: mremap speedup - Enable HAVE_MOVE_PUD
Date: Mon, 5 Oct 2020 15:40:07 +0000 [thread overview]
Message-ID: <20201005154017.474722-5-kaleshsingh@google.com> (raw)
In-Reply-To: <20201005154017.474722-1-kaleshsingh@google.com>
HAVE_MOVE_PUD enables remapping pages at the PUD level if both the
source and destination addresses are PUD-aligned.
With HAVE_MOVE_PUD enabled it can be inferred that there is approximately
a 19x improvement in performance on arm64. (See data below).
------- Test Results ---------
The following results were obtained using a 5.4 kernel, by remapping
a PUD-aligned, 1GB sized region to a PUD-aligned destination.
The results from 10 iterations of the test are given below:
Total mremap times for 1GB data on arm64. All times are in nanoseconds.
Control HAVE_MOVE_PUD
1247761 74271
1219896 46771
1094792 59687
1227760 48385
1043698 76666
1101771 50365
1159896 52500
1143594 75261
1025833 61354
1078125 48697
1134312.6 59395.7 <-- Mean time in nanoseconds
A 1GB mremap completion time drops from ~1.1 milliseconds
to ~59 microseconds on arm64. (~19x speed up).
Signed-off-by: Kalesh Singh <kaleshsingh@google.com>
---
Change in v3:
- Add set_pud_at() macro - Used by move_normal_pud().
arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/pgtable.h | 1 +
2 files changed, 2 insertions(+)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 844d089668e3..4d521f0a5863 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -122,6 +122,7 @@ config ARM64
select HANDLE_DOMAIN_IRQ
select HARDIRQS_SW_RESEND
select HAVE_MOVE_PMD
+ select HAVE_MOVE_PUD
select HAVE_PCI
select HAVE_ACPI_APEI if (ACPI && EFI)
select HAVE_ALIGNED_STRUCT_PAGE if SLUB
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index d5d3fbe73953..8848125e3024 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -415,6 +415,7 @@ static inline pmd_t pmd_mkdevmap(pmd_t pmd)
#define pfn_pud(pfn,prot) __pud(__phys_to_pud_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot))
#define set_pmd_at(mm, addr, pmdp, pmd) set_pte_at(mm, addr, (pte_t *)pmdp, pmd_pte(pmd))
+#define set_pud_at(mm, addr, pudp, pud) set_pte_at(mm, addr, (pte_t *)pudp, pud_pte(pud))
#define __p4d_to_phys(p4d) __pte_to_phys(p4d_pte(p4d))
#define __phys_to_p4d_val(phys) __phys_to_pte_val(phys)
--
2.28.0.806.g8561365e88-goog
next prev parent reply other threads:[~2020-10-05 15:41 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-05 15:40 [PATCH v3 0/5] Speed up mremap on large regions Kalesh Singh
2020-10-05 15:40 ` Kalesh Singh
2020-10-05 15:40 ` Kalesh Singh
2020-10-05 15:40 ` [PATCH v3 1/5] kselftests: vm: Add mremap tests Kalesh Singh
2020-10-05 15:40 ` Kalesh Singh
2020-10-05 15:40 ` Kalesh Singh
2020-10-05 15:40 ` [PATCH v3 2/5] arm64: mremap speedup - Enable HAVE_MOVE_PMD Kalesh Singh
2020-10-05 15:40 ` Kalesh Singh
2020-10-05 15:40 ` Kalesh Singh
2020-10-05 15:40 ` [PATCH v3 3/5] mm: Speedup mremap on 1GB or larger regions Kalesh Singh
2020-10-05 15:40 ` Kalesh Singh
2020-10-05 15:40 ` Kalesh Singh
2020-10-05 15:40 ` Kalesh Singh [this message]
2020-10-05 15:40 ` [PATCH v3 4/5] arm64: mremap speedup - Enable HAVE_MOVE_PUD Kalesh Singh
2020-10-05 15:40 ` [PATCH v3 5/5] x86: " Kalesh Singh
2020-10-05 15:40 ` Kalesh Singh
2020-10-05 15:40 ` Kalesh Singh
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201005154017.474722-5-kaleshsingh@google.com \
--to=kaleshsingh@google.com \
--cc=Dave.Martin@arm.com \
--cc=akpm@linux-foundation.org \
--cc=almasrymina@google.com \
--cc=aneesh.kumar@linux.ibm.com \
--cc=arnd@arndb.de \
--cc=bgeffon@google.com \
--cc=bp@alien8.de \
--cc=broonie@kernel.org \
--cc=catalin.marinas@arm.com \
--cc=christian.brauner@ubuntu.com \
--cc=dave.hansen@intel.com \
--cc=frederic@kernel.org \
--cc=gregkh@linuxfoundation.org \
--cc=gshan@redhat.com \
--cc=hnaveed@wavecomp.com \
--cc=hpa@zytor.com \
--cc=jgg@ziepe.ca \
--cc=jhubbard@nvidia.com \
--cc=joelaf@google.com \
--cc=justin.he@arm.com \
--cc=kamalesh@linux.vnet.ibm.com \
--cc=keescook@chromium.org \
--cc=kernel-team@android.com \
--cc=kirill.shutemov@linux.intel.com \
--cc=krzk@kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linuxram@us.ibm.com \
--cc=lokeshgidra@google.com \
--cc=mark.rutland@arm.com \
--cc=masahiroy@kernel.org \
--cc=mhiramat@kernel.org \
--cc=minchan@google.com \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=rcampbell@nvidia.com \
--cc=rppt@kernel.org \
--cc=samitolvanen@google.com \
--cc=sandipan@linux.ibm.com \
--cc=shuah@kernel.org \
--cc=sjpark@amazon.de \
--cc=surenb@google.com \
--cc=tglx@linutronix.de \
--cc=will@kernel.org \
--cc=x86@kernel.org \
--cc=yezhenyu2@huawei.com \
--cc=ziy@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.